G. KIELHOFNER (2016), Research in Occupationnal Therapy, Methods of Inquiry For Enhancing Practice, FA Davis Company

00Keilhofner(F)-FM 5/19/06 4:52 PM Page i
RESEARCH IN
OCCUPATIONAL
THERAPY
Methods of Inquiry
for Enhancing Practice
00Keilhofner(F)-FM 5/19/06 4:52 PM Page ii
This page has been left intentionally blank.

00Keilhofner(F)-FM 5/19/06 4:52 PM Page iii
Gary Kielhofner, DrPH, OTR/L, FAOTA

Professor and Wade-Meyer Chair
Department of Occupational Therapy
College of Applied Health Sciences
University of Illinois at Chicago
RESEARCH IN
OCCUPATIONAL
THERAPY
Methods of Inquiry
for Enhancing Practice
00Keilhofner(F)-FM 5/20/06 5:28 PM Page iv
iv Header
F. A. Davis Company
1915 Arch Street
Philadelphia, PA 19103
www.fadavis.com
Copyright © 2006 by F. A. Davis Company
All rights reserved. This book is protected by copyright. No part of it may be reproduced,
stored in a retrieval system, or transmitted in any form or by any means, electronic, mechani-
cal photocopying, recording, or otherwise, without written permission from the publisher.
Printed in the United States of America
Last digit indicates print number: 10 9 8 7 6 5 4 3 2 1
Acquisitions Editor: Christa A. Fratantoro

Manager of Content Development: Deborah J. Thorp
Developmental Editor: Peg Waltner
Design Manager: Carolyn O’Brien
As new scientific information becomes available through basic and clinical research, recom-
mended treatments and drug therapies undergo changes. The authors and publisher have done
everything possible to make this book accurate, up to date, and in accord with accepted stan-
dards at the time of publication. The authors, editors, and publisher are not responsible for
errors or omissions or for consequences from application of the book, and make no warranty,
expressed or implied, in regard to the contents of the book. Any practice described in this
book should be applied by the reader in accordance with professional standards of care used
in regard to the unique circumstances that may apply in each situation. The reader is advised
always to check product information (package inserts) for changes and new information
regarding dose and contraindications before administering any drug. Caution is especially
urged when using new or infrequently ordered drugs.
Library of Congress Cataloging-in-Publication Data
Kielhofner, Gary
Research in occupational therapy : methods of inquiry for enhancing practice /
Gary Kielhofner.
p. cm.
Includes bibliographical references and index.
ISBN 13: 978-0-8036-1525-0
ISBN 10: 0-8036-1525-6
1. Occupational therapy—Research. I. Title.
RM735.K54 2006
615.8′515—dc22
2006041013
Authorization to photocopy items for internal or personal use, or the internal or personal use
of specific clients, is granted by F. A. Davis Company for users registered with the Copyright
Clearance Center (CCC) Transactional Reporting Service, provided that the fee of $.10 per
copy is paid directly to CCC, 222 Rosewood Drive, Danvers, MA 01923. For those organiza-
tions that have been granted a photocopy license by CCC, a separate system of payment has
been arranged. The fee code for users of the Transactional Reporting Service is: 8036-
1042/03 0 + $.10
00Keilhofner(F)-FM 5/19/06 4:52 PM Page v
Header v
This book is dedicated to:
A. Jean Ayres,
for role-modeling the vocation of a scientist in
the field
Mary Reilly,
for providing timeless insights into the nature
of knowledge and discovery in occupational
therapy
Noomi Katz,
for demonstrating open and persistent inquiry
into a complex clinical problem
Anne Mosey,
for logically critiquing how scholarship should
advance the practice of occupational therapy
Elizabeth Yerxa,
for demonstrating how to love the questions
posed by our field
And to all others who have passed on to a

current generation of scholars the passion,
persistance, and perspectives necessary to
scholarship in occupational therapy
00Keilhofner(F)-FM 5/19/06 4:52 PM Page vi

00Keilhofner(F)-FM 5/19/06 4:52 PM Page vii
C O N T R I B U T O R S
Beatriz C. Abreu, PhD, OTR, FAOTA Susan Corr, DipCOT, MPhil, DIPMed ED,
Clinical Professor PhD
Department of Occupational Therapy Reader in Occupational Science
School of Allied Health Sciences School of Health
University of Texas Medical Branch; The University of Northampton
Director of Occupational Therapy Park Campus,
Transitional Learning Center at Galveston Northampton, England
Galveston, Texas
Marian Arbesman, PhD, OTR/L Wendy J. Coster, PhD, OTR/L, FAOTA

Clinical Assistant Professor Associate Professor and Chair
Department of Rehabilitation Science Department of Occupational Therapy and
University at Buffalo; President, Arbesldeas, Inc. Rehabilitation Counseling
Consultant, AOTA Evidence-Based Literature Sargent College of Health & Rehabilitation
Review Project Sciences
Buffalo, New York Boston University
Boston, Massachusetts
Nancy A. Baker, ScD, OTR/L
Assistant Professor Anne Cusick, BAppSc(OT), MA, PhD
School of Health and Rehabilitation Sciences College of Science and Health
University of Pittsburgh University of Western Sydney
Pittsburgh, Pennsylvania Penrith South, Australia
Brent Braveman, PhD, OTR/L, FAOTA
Clinical Associate Professor Jean Crosetto Deitz, PhD, OTR/L, FAOTA
Director of Professional Education Professor & Graduate Program Coordinator
Department of Occupational Therapy Department of Rehabilitation Medicine
College of Applied Health Sciences University of Washington
University of Illinois at Chicago Seattle, Washington
Chicago, Illinois
Anne E. Dickerson, PhD, OTR/L, FAOTA
Lisa Castle, MBA, OTR/L
Professor and Chair
Director of Occupational Therapy
University of Illinois Medical Center at Chicago
East Carolina University; Co-Director of
Chicago, Illinois
Research for Older Adult Driver Initiative
Sherrilene Classen, PhD, MPH, OTR/L (ROADI)
Assistant Professor Editor of Occupational Therapy in Health Care
University of Florida Greenville, North Carolina
College of Public Health and Health Professions
Gainesville, Florida M. G. Dieter, MLIS, MBA
Clinical Assistant Professor, Health Informatics
Mary A. Corcoran, PhD, OTR, FAOTA Department of Biomedical and Health
Research Professor Information Sciences
The George Washington University University of Illinois at Chicago
Department of Health Care Sciences Chicago, Illinois
School of Medicine and Health Sciences
Washington, DC;
Professor Heather Dillaway, PhD
Department of Occupational Therapy Assistant Professor, Sociology
Shenandoah University Wayne State University
Winchester VA Detroit, Michigan
vii
00Keilhofner(F)-FM 5/19/06 4:52 PM Page viii
viii Contributors
Claire-Jehanne Dubouloz, OT(c), PhD Patricia Heyn, PhD

Associate Professor Senior Research Instructor
Associate Dean of the Faculty of Health Sciences Division of Geriatric Medicine/School of
Director of the School of Rehabilitation Sciences Medicine
University of Ottawa University of Colorado Health Sciences Center
Ontario, Canada Denver, Colorado
Edward A. S. Duncan, PhD, BSc(Hons), Frederick J. Kviz, PhD

Dip.CBT, SROT Professor
Postdoctoral Research Fellow Community Health Sciences
Nursing, Midwifery and Allied Health School of Public Health
Professions Research Unit University of Illinois at Chicago
The University of Stirling Chicago, Illinois
Stirling, Scotland
Clinical Specialist Occupational Therapist
Mark R. Luborsky, PhD
Occupational Therapy Department
Professor of Anthropology and Gerontology
The State Hospital
Director of Aging and Health Disparities Research
Carstairs, Scotland
Institute of Gerontology
Wayne State University
Mary Egan, PhD, OT(C) Detroit, Michigan
Associate Professor
School of Rehabilitation Sciences
Cathy Lysack, PhD, OT(C)
University of Ottawa
Associate Professor, Occupational Therapy and
Ontario, Canada
Gerontology
Wayne State University
Marcia Finlayson, PhD, OT(C), OTR/L Detroit, Michigan
Associate Professor
Annie McCluskey, PhD, MA, DipCOT
College of Applied Health Sciences
Senior Lecturer in Occupational Therapy
School of Exercise & Health Sciences
Chicago, Illinois
University of Western Sydney
Penrith South, Australia
Gail Fisher, MPA, OTR/L
Associate Department Head for Administration
Jane Melton, MSc, DipCOT (UK)
Clinical Associate Professor
Lead Occupational Therapist for Practice
Development & Research
Gloucestershire Partnership NHS Trust
Chicago, Illinois
The Charlton Lane Centre
Cheltenham, England
Kirsty Forsyth, PhD, OTR
Senior Lecturer
Jaime Phillip Muñoz, PhD, OTR, FAOTA
Occupational Therapy Department
Assistant Professor
Queen Margaret University College
Edinburgh, Scotland
Duquesne University
Pittsburgh, Pennsylvania
Ellie Fossey, DipCOT (UK), MSC
Senior Lecturer - Postgraduate Courses
David L. Nelson, PhD, OTR, FAOTA
Coordinator
Professor of Occupational Therapy
School of Occupational Therapy
College of Health Sciences
La Trobe University;
Medical University of Ohio at Toledo
PhD Candidate
Toledo, Ohio
School of Social Work
The University of Melbourne
Victoria, Australia
00Keilhofner(F)-FM 5/19/06 4:52 PM Page ix
Contributors ix
Kenneth J. Ottenbacher, PhD, OTR/L Pimjai Sudsawad, ScD, OTR

Professor and Director Assistant Professor
Division of Rehabilitation Sciences Occupational Therapy Program
University of Texas Medical Branch Department of Kinesiology
Galveston, Texas University of Wisconsin-Madison
Madison, Wisconsin
Sue Parkinson, BA, DipCOT
Practice Development Advisor for OTs Lynn Summerfield-Mann, DipCOT, SROT,
Derbyshire Mental Health Services NHS Trust MSc
Derby, England Doctoral Student
Principal Lecturer in Occupational Therapy
Department of Allied Health Professions
Amy Paul-Ward, PhD Faculty of Health & Social Care
Assistant Professor London South Bank University
Department of Occupational Therapy London, England
Florida International University
Miami, Florida
Renée R. Taylor, PhD
Associate Professor
Nadine Peacock, PhD Department of Occupational Therapy
Associate Professor University of Illinois at Chicago
Community Health Sciences Chicago, Illinois
School of Public Health
Adjunct Associate Professor Machiko R. Tomita, PhD
Department of Anthropology Clinical Associate Professor
University of Illinois at Chicago Department of Rehabilitation Science
Chicago, Illinois Director, Aging and Technology Projects
School of Public Health and Health Professions
Geneviève Pépin, PhD, OTR University at Buffalo
Adjunct Professor Buffalo, New York
Occupational Therapy Program
Rehabilitation Department Hector W. H. Tsang, PhD, OTR
Laval University Associate Professor
Quebec, Canada Department of Rehabilitation Sciences
The Hong Kong Polytechnic University
Hong Kong
Mick Robson, DipCOT
Formerly a Senior Research Practitioner within
the Central and North West Toni Van Denend, MS, OTR/L
London Mental Health NHS Trust RehabWorks
(CNWL)/UK Centre of Outcomes Research Staff Therapist
and Education (UKCORE) Partnership Hunt Valley
London, England Maryland
Yolanda Suarez-Balcazar, PhD Craig A. Velozo, PhD, OTR

Associate Professor Associate Professor and Associate Chair
Department of Occupational Therapy Department of Occupational Therapy
Associate Director, Center for Capacity Building College of Public Health and Health Professions;
for Minorities with Disabilities Research Research Health Scientist
University of Illinois at Chicago North Florida\South Georgia Veterans Health
Chicago, Illinois System
University of Florida
Gainesville, Florida
00Keilhofner(F)-FM 5/19/06 4:52 PM Page x
x Contributors
Elizabeth White, PhD, DipCOT Don E. Workman, PhD

Head of Research and Development Executive Director
College of Occupational Therapists Office for the Protection of Research Subjects
London, England Northwestern University
Chicago, Illinois
Suzie Willis, DipCOT, SROT, MSC, MA
Head Research Practitioner Yow-Wu B. Wu, PhD
Central and Northwest London Mental Health Associate Professor
Trust School of Nursing
Tamarind Centre University at Buffalo
Park Royal Centre for Mental Health Buffalo, New York
London, England
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xi
R E V I E W E R S
Catherine Emery MS, OTR/L, BCN Rosemary M. Lysaght, PhD, OTR/L

Assistant Professor Assistant Professor
Occupational Therapy Program University of Utah
Alvernia College Salt Lake City, Utah
Lancaster, Pennsylvania
Maralynne D. Mitcham, PhD, OTR/L, FAOTA
Karen Funk, OTD, OTR Professor & Director
Assistant Professor Occupational Therapy Educational Program
University of Texas, El Paso College of Health Professions
El Paso, Texas Medical University of South Carolina
Charleston, South Carolina
Heather A. Gallew, OTD, OTR/L
Assistant Professor of Occupational Therapy Carol Reed, EdD, OTR, FAOTA
Xavier University Professor & Chair
Cincinnati, Ohio Occupational Therapy Department
Nova Southeastern University
Robert W. Gibson, MSOTR/L Plantation, Florida
Research Associate
University of Florida Pat Sample, PhD
Gainesville, Florida Associate Professor
Occupational Therapy
Stacie Lynn Iken, PhD, MS, OTR/L Colorado State University
Occupational Therapy Program Director & Fort Collins, Colorado
Associate Professor
University of Mary Barbara A. Boyt Schell, PhD, OTR/L, FAOTA
Bismarck, North Dakota Professor & Chair
Panelpha L. Kyler, MA, OT, FAOTA Brenau University
Public Health Analyst & Adjunct Faculty Genetic Athens, Georgia
Services Branch
Maternal & Child Health Bureau Karen Sladyk, PhD, OTR, FAOTA
Health Resources & Services Administration Professor & Chair
Department of Health & Human Services Bay Path College
Towson University Longmeadow, Massachusetts
Bethesda, Maryland
Janet H. Watts, PhD, OTR, CRC
Jennie Q. Lou, M.D., M.Sc., OTR Associate Professor & Director of Post-
Associate Professor of Public Health & Associate Professional Graduate Studies
Professor of Occupational Therapy Virginia Commonwealth University
Nova Southeastern University Richmond, Virginia
Ft. Lauderdale, Florida
Shirley A. Wells, MPH, OTR, FAOTA
Ferol Menks Ludwig, PhD, OTR, FAOTA, Assistant Professor
GCG University of Texas
Professor & Director of Doctoral Program Brownsville, Texas
Nova Southeastern University
Ft. Lauderdale, Florida
xi
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xii

00Keilhofner(F)-FM 5/19/06 4:52 PM Page xiii
P R E F A C E
Those who enter the profession of occupational “a systematic search for and validation of
therapy come with a desire to enable people to par- knowledge about issues of importance…”
ticipate more fully in everyday life. The profes- Polit and Hungler (1999, p 3)
sion’s mission is practical and humanistic. Not
“a structured process of investigating facts
surprisingly, the mention of research in occupa-
and theories and exploring connections”
tional therapy can engender expectations of an
Portney and Watkins (2000, p 4)
uninteresting and technical topic of limited rele-
vance. This is not surprising. Research is often “Multiple, systematic strategies to generate
written about as though it were mysterious and knowledge about human behavior, human
quite apart from practical therapeutic work. As a experience and human environments in
consequence, discussions of research can some- which the thought and action processes of
times make it appear inaccessible and remote from the researcher are clearly specified so that
occupational therapy practice. they are logical, understandable, con-
Investigators, such as those who have con- firmable, and useful”. Depoy and Gitlin
tributed to this book, do research because they find (1998, p 6)
the process of discovery
exhilarating and because Taken together, these
they see how it enhances definitions emphasize a
Investigators, such as those number of important
occupational therapy
practice. Thus, in this who have contributed to this points. First, research is
systematic, principled,
text, we have sought to book, do research because structured, and logical.
underscore two themes.
First, we have sought to they find the process of dis- Second, it involves a
process of investigation
illustrate how research is covery exhilarating and or obtaining evidence in
a creative process of dis-
covering and sharing because they see how it order to generate knowl-
new information. We enhances occupational ther- edge, and test theories
have aimed to demon- Third, research aims to
strate that research is not
apy practice. study issues of impor-
only comprehensible but tance and solve practical
also engaging. Second, problems. These charac-
we have made specific efforts to demonstrate how teristics of research have been our point of depar-
research is both essential to and can support and ture and are emphasized throughout the text.
improve occupational therapy practice. We sought
to accomplish this not only through constant use of
occupational therapy research examples, but also SCOPE, CONTENT, AND
through addressing issues of how research con- ORGANIZATION OF THE
tributes to professional knowledge and how
research can be used by members of the profes- BOOK
sion. This book discusses contemporary research meth-
ods and tools. It aims to comprehensively cover the
range research relevant to occupational therapy.
DEFINITIONS OF RESEARCH The book is designed to build an appreciation of
the kinds of perspectives, strategies and specific
One can readily find a number of definitions of
tools that are used by researchers. It should also
research as it pertains to the field of health care.
allow the reader to become a competent consumer
For instance, research has been defined as:
of most types of research. Finally, it includes the
“ a systematic and principled way of necessary information to prepare one for being a
obtaining evidence (data, information) for competent collaborator in research.
solving heath care problems” Polgar and Because the book is broad in scope, many top-
Thomas, S.A. (2000, p 3) ics are covered at the most basic level. Therefore,
xiii
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xiv
xiv Preface
many chapters will provide the reader with many lying quantitative research, namely; how investiga-
additional resources for understanding research tors quantify or measure the phenomena they
reports and for participating in research. study. The rigor of all quantitative investigation
Deciding how to organize and sequence the rests on how adequately the researcher has opera-
chapters in a text such as this is no small chal- tionalized the variables of interest. Thus, the quan-
lenge; everyone consulted offered up differing tification of research variables must be carefully
opinions. Our strategy has been to organize the approached. Three chapters address this issue from
book into nine fairly small sections. Each section the different perspectives of classical test theory,
is designed to stand alone; however, some sections contemporary item response theory, and a critical
do rely on material discussed in other sections and reflection on the aims, value, and limits of quan-
the reader will profit from referring to those other tification.
chapters. Within each section, the chapters are The fourth section completes coverage of quan-
generally best read in sequence. titative research by discussing the major statistical
Importantly, the sequence of the sections and methods used in such research. In addition to the
the chapters within the sections is not intended to overview of statistics provided by the chapters,
convey anything about importance. To a large this section includes tables that provide, at a
extent, sequence was driven by how we felt the glance: summaries of the questions various statis-
underlying logical with which we sought to tics answer, requirements for appropriate applica-
explain research would best unfold. tion of the statistic, and how the numbers
Reading this book can be approached in a vari- generated by the statistic should be interpreted.
ety of ways. Most readers will find it helpful to Section five covers qualitative research meth-
complete section one first, since it discusses some ods. These chapters provide an overview of the
of the broadest and most basic topics related to logic and methods of qualitative research, and dis-
research in occupational therapy. Beyond this first cuss how qualitative data are collected, managed
section, the sequence can depend on the reader’s and analyzed. This section aims to provide the
interest or the organization for a course or course- reader with an understanding of the logic and
work for which to book is being used. Below each range of qualitative methods, the various purposes
section is described to give a sense of its purpose to which qualitative research is put, and the major
and contents. methodological procedures involved in doing qual-
The first section introduces the reader to the itative research.
nature and scope of research and its place in occu- The sixth section covers research designs that
pational therapy. Chapters address why research is do not neatly fit into either the broad quantitative
needed, philosophical underpinnings of research, or qualitative categories. Two chapters cover
the range of research methods, the basic process research approaches that achieve insights through
common to all forms of research, and the various systematically eliciting opinions or perspectives. A
roles and responsibilities that members of the pro- third chapter discusses how qualitative and quanti-
fession should and can take on related to research. tative research methods can be combined in a rig-
By the end of the section, the reader should have a orous way.
very broad idea of aims of research, the assump- Section seven covers in detail the various steps
tions, principles and concerns that guide research, and procedures the investigators complete when
and the different ways that research can be con- they conduct research. The first chapter provides
ducted. Additionally, the reader should appreciate an overview of the main steps of research which
what the results of research have to do with the are discussed in more detail in subsequent chap-
profession and what members of the profession ters. These include: searching the literature; identi-
should have to do with research. fying research questions; designing the study;
The second section covers major forms of assuring the research is ethically sound; obtaining
quantitative research designs (i.e., exploratory, funding for the research; securing samples; col-
group comparison, survey, retrospective, longitu- lecting, managing and analyzing data; disseminat-
dinal, and single subject) These chapters aim to ing findings; and writing a research report. This
provide an overview of both the logic and methods section aims to discuss the research process in
that belong to each design. Readers should glean such a way that the reader appreciates all the ele-
a substantial understanding of what types of ments that go into research and so that a first-time
research questions quantitative designs can researcher can systematically anticipate and plan
address and how such designs answer these ques- for what is involved in a study.
tions. Section eight discusses how research methods
Section three addresses a specific issue under- can specifically be used to enhance occupational
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xv
Preface xv
therapy practice. It includes illustrations of how best suited their discussion. Because of its classic
research methods can be applied to needs assess- use certain key concepts still use the term, subject,
ment and to program development and evaluation. in the phraseology (e.g., between-subjects vari-
Additionally this section features three chapters ance or single-subject design). In such instances
that address participatory authors have retained
research approaches. classical terminology to
These approaches repre- We hope the book conveys avoid confusion. Other
sent a promising way to the deep commitment that all authors preferred use of
conduct research that the contemporary terms,
is grounded in and
who contributed feel toward such as participants,
addresses the issues and the research process. stakeholders or partners
perspectives of occupa- for their discussion.
tional therapy practition-
ers and consumers.
Section nine discusses evidence based practice. CONCLUSION
The chapters in this section cover the definition, Writing this book was a challenging enterprise for
history and purpose and implementation of evi- all involved and reading it will be no less chal-
dence-based practice. This section also addresses lenging. Nonetheless, the reader will find that the
how practice can effectively be changed in chapter authors are not only knowledgeable and
response to evidence and how evidence can be gen- passionate about their topic, but made substantial
erated in ways that are more relevant to practice. effort to help the reader comprehend the topic. We
hope the book conveys the deep commitment that
A NOTE ON TERMINOLOGY all who contributed feel toward the research
process. Most of all, we hope that the reader will
Convention about how to refer to persons who are share some of that commitment after having
being studied in an investigation has changed over digested this volume.
the years. The classical term, subject, connoted Gary Kielhofner
that individuals were objects of inquiry and/
or were subjected to experimental procedures. REFERENCES
Contemporary researchers generally prefer the DePoy, E. & Gitlin, L. (1998) Introduction to Research:
term, participant. This terminology stresses the Understanding and Applying Multiple Strategies. St.
Louis: C.V. Mosby.
extent to which people who are studied freely con-
tribute time, personal information, and effort to a Polgar, S. & Thomas, S.A. (2000). Introduction to
study. Still other researchers, who emphasize the Research in the Health Sciences. Edinburgh: Churchill
importance of involving people who are being Livingston.
studied as partners in the research enterprise, use
Polit, D.F. & Hungler, B.P. (1999). Nursing Research:
terms such as partners or stakeholders to refer to Principles and Methods. Philadelphia: Lippincott.
individuals whose circumstances are being investi-
gated. Portney, L.G. & Watkins, M.P. (2000). Foundations of
Throughout this text the authors of chapters Clinical Research: Applications to Practice. (2nd Ed)
Upper Saddle River, New Jersey: Prentice-Hall.
were given freedom to use the terminology which
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xvi

00Keilhofner(F)-FM 5/19/06 4:52 PM Page xvii
A C K N O W L E D G M E N T S
Scholarship of Practice
No book of this size and scope comes into exis- Classen, Susan Doble, Regina Doherty, Edward
tence without the contributions of many people. Duncan, Catherine Emery, John Fleming, Ellie
First of all, I am grateful for the privilege of work- Fossey, Linnea Franits, Karen Funk, Linda
ing as editor with the esteemed colleagues from Gabriel, Heather A. Gallew, Kilian Garvey,
throughout the world who contributed chapters to Robert W. Gibson, Stacie Lynn Iken, Panelpha
this text. Each brought unique qualities to the L. Kyler, Lori Letts, Jennie Q. Lou, Ferol
chapters they created, writing from the informed Menks Ludwig, Rosemary M. Lysaght, Nancy
perspectives of experience and expertise. In the MacRae, Maralynne Mitcham, Maggie Nicol,
course of editing their work, I learned a great deal. Jane O’Brien, Genèvieve Pépin, Carol Reed,
Pat Sample, Dianne F. Simons, Karen Sladyk,
I owe a debt of gratitude for the commitment and Roger Smith, Barry Trentham, Kerryellen
the many hours of careful work that my team at Vroman, Janet H. Watts, Shirley A. Wells, Don
UIC put into this project. Research assistants and Workman, and Hon Yuen. Thanks are also
graduate students, Jenny Fox-Manaster, Annie owing to Marcia Ciol, who created many
Ploszaj, and Jessica Kramer provided outstanding graphs for figures in Chapter 11, Single Subject
leadership that helped guide this manuscript to Research.
fruition at various stages. They had able editorial
help of their peers: Mona Youngblood, Stefanie
Conaway, Sun-Wook Lee, and Kathleen Kramer. This book would not have come into existence but
My assistant, Felicia Walters, ably oversaw the for the friendship and persistent badgering of
process of acquiring photographs, while post-doc- Christa Fratantoro and Margaret Biblis. Margaret
toral fellow Patty Bowyer gave helpful feedback convinced me that the book was a good idea and
throughout the process. artfully talked me into taking it on, despite my ini-
tial hesitations. Christa was there at every stage
We relied heavily on external reviewers to improve and willing to do whatever it took to make this a
the clarity and content of the book. Their guidance great book. Deborah Thorp provided able editorial
and feedback were instrumental to improving the guidance and the final look and layout of the text
final quality of this text. I am grateful to the fol- is a reflection of her hard work.
lowing persons who reviewed the book at its vari-
ous stages of development: Over the years, F.A. Davis has supported projects
that began as ideas without much precedent,
Julie Bass-Haugen, Bette R. Bonder, Barbara including my very first text. I am grateful for their
A. Boyt Schell, Jane Case-Smith, Sherrilene ongoing faith in and support of me.
xvii
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xviii

00Keilhofner(F)-FM 5/19/06 4:52 PM Page xix
T A B L E O F C O N T E N T S
SECTION 1 Chapter 11 Single-Subject

Research, 140
The Nature and Scope of Inquiry in Jean Crosetto Deitz
a Practice Profession
Chapter 1 The Necessity of Research in a
Profession, 1 SECTION 3
Gary Kielhofner Quantitative Measurement
Chapter 2 The Aim of Research: Chapter 12 Developing and
Philosophical Foundations of Scientific Evaluating Quantitative Data Collection
Inquiry, 10 Instruments, 155
Gary Kielhofner Gary Kielhofner
Chapter 3 The Range of Research, 20 Chapter 13 Objective Measurement:
Gary Kielhofner • Ellie Fossey The Influence of Item Response Theory on
Chapter 4 Characteristics of Sound Inquiry Research and Practice, 177
and the Research Process, 36 Craig A. Velozo • Kirsty Forsyth • Gary
Gary Kielhofner Kielhofner
Chapter 5 Professional Responsibility and Chapter 14 Evaluating the Use of Assess-

Roles in Research, 46 ments in Practice and Research, 201
Anne Cusick • Gary Kielhofner Wendy J. Coster
SECTION 2 SECTION 4
Quantitative Designs Statistical Analysis
Chapter 6 Descriptive Quantitative Chapter 15 Making Meaning From
Designs, 58 Numbers: Measurements and Descriptive
Gary Kielhofner Statistics, 213
Machiko R. Tomita
Chapter 7 Group Comparison Studies:
Quantitative Research Designs, 65 Chapter 16 Concepts of Inferential Statis-
David L. Nelson tics: From Sample to Population, 232
Machiko R. Tomita
Chapter 8 Survey Research
Design, 91 Chapter 17 Methods of Analysis: From
Kirsty Forsyth • Frederick J. Kviz Univariate to Multivariate Statistics, 243
Machiko R. Tomita
Chapter 9 Study Designs for Secondary
Analysis of Existing Data, 110 Chapter 18 Meta-Analysis, 281
Sherrilene Classen Kenneth J. Ottenbacher • Patricia Heyn
• Beatriz C. Abreu
Chapter 10 Longitudinal Research:
Design and Analysis, 127 Appendix A Statistical Reference
Yow-Wu B. Wu • Kirsty Forsyth Tables, 299
• Gary Kielhofner Machiko R. Tomita
xix
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xx
xx Table of Contents
Appendix B Summary of Common Chapter 27 Searching the

Research Statistics, 314 Literature, 437
Gary Kielhofner • Machiko R. Tomita M. G. Dieter • Gary Kielhofner
Chapter 28 Generating Research

SECTION 5 Questions, Design, and Methods, 452
Qualitative Methods Marian Arbesman • Hector W. H. Tsang
Chapter 19 Overview of Qualitative Chapter 29 Integrity and Ethics in

Research, 326 the Conduct of Human Subject
Mark R. Luborsky • Cathy Lysack Research, 468
Don E. Workman • Gary Kielhofner
Chapter 20 Gathering Qualitative
Data, 341 Chapter 30 Obtaining Funding for
Cathy Lysack • Mark R. Luborsky • Heather Research, 486
Dillaway Renée R. Taylor • Yolanda Suarez-Balcazar
• Geneviève Pépin • Elizabeth White
Chapter 21 Contemporary Tools for
Managing and Analyzing Qualitative Chapter 31 Securing Samples for Effective
Data, 358 Research Across Research Designs, 515
Nadine Peacock • Amy Paul-Ward Anne E. Dickerson
Chapter 22 Qualitative Approaches to Chapter 32 Collecting Data, 530

Interpreting and Reporting Data, 372 Renée R. Taylor • Gary Kielhofner
Heather Dillaway • Cathy Lysack
• Mark R. Luborsky Chapter 33 Data Management, 548
Marcia Finlayson • Toni Van Denend
SECTION 6 Chapter 34 Disseminating

Other Design Approaches Research, 565
Gary Kielhofner • Ellie Fossey • Renée R. Taylor
Chapter 23 Exploring Perceptions About
Services Using Q Methodology, 389 Chapter 35 Writing a Research
Susan Corr Report, 578
Chapter 24 The Nature and Use of
Consensus Methodology in Practice, 401
Edward A. S. Duncan SECTION 8
Chapter 25 Using Mixed Methods Designs Enhancing Practice Through Inquiry
to Study Therapy and Its Outcomes, 411 Chapter 36 Assessing Need for
Mary A. Corcoran Services, 591
Marcia Finlayson
SECTION 7 Chapter 37 Using Research to Develop
Conducting Inquiry and Evaluate Programs of Service, 607
Brent Braveman • Yolanda Suarez-Balcazar
Chapter 26 Organizing the Components of • Gary Kielhofner • Renée R. Taylor
Inquiry Together: Planning and Imple-
menting a Coherent Study and Research Chapter 38 Participatory Research in
Tradition, 420 Occupational Therapy, 620
Gary Kielhofner • Marcia Finlayson Renée R. Taylor • Yolanda Suarez-Balcazar
• Renée R. Taylor • Kirsty Forsyth • Gary Kielhofner
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xxi
Table of Contents xxi
Chapter 39 Building Culturally Competent Chapter 42 Analyzing Evidence for

Community–University Partnerships for Practice, 663
Occupational Therapy Scholarship, 632 Nancy A. Baker
Yolanda Suarez-Balcazar • Jaime Phillip Muñoz
• Gail Fisher
Chapter 43 Managing Change and Barriers
to Evidence-Based Practice, 685
Chapter 40 Participatory Research for the Annie McCluskey
Development and Evaluation of Occupa-
tional Therapy: Researcher–Practitioner Chapter 44 Creating Outcome
Collaboration, 643 Research Relevant for Evidence-Based
Gary Kielhofner • Lisa Castle • Claire-Jehanne
Practice, 697
Pimjai Sudsawad
Dubouloz • Mary Egan • Kirsty Forsyth • Jane
Melton • Sue Parkinson • Mick Robson • Lynn Index 707
Summerfield-Mann • Renée R. Taylor • Suzie
Willis
SECTION 9
Evidence-Based Practice
Chapter 41 Definition, Evolution, and
Implementation of Evidence-Based Practice
in Occupational Therapy, 656
Pimjai Sudsawad
00Keilhofner(F)-FM 5/19/06 4:52 PM Page xxii

01Kielhofner(F)-01 5/5/06 3:40 PM Page 1
S E C T I O N 1
The Nature and Scope of Inquiry in a Practice Profession
C H A P T E R 1
The Necessity of Research in a Profession

Gary Kielhofner
A. Jean Ayres provided occupational therapy to Murray, 2002). Armed with empirical support for
children with learning disabilities (Figure 1.1). She the existence of certain patterns of sensory pro-
observed that some of these children had difficulty cessing problems and with a conceptualization of
processing sensory information from their bodies those problems, she refined strategies of interven-
and from the environment. She also noted that these tion. Her own research and hundreds of studies by
sensory processing problems appeared related to others over several decades have served to improve
problems in motor and academic learning. Spurred understanding of sensory integration problems, to
by her observations, Ayres began developing a con- develop and refine a battery of tests designed to
ceptualization of how the brain organizes and inter- help therapists identify and understand sensory
prets sensory information (Ayres, 1979). She used integrative impairments, and to test the effective-
this theory, referred to as Sensory Integration, to ness of interventions designed to remediate them
understand the functional difficulties found in some (Walker, 2004).
children with learning disabilities. Lucy Jane Miller began research developing
Ayres went on to conduct research on large norm-referenced standardized scales for preschool
samples of children to describe the different types children after working as an occupational therapist
of sensory problems that certain children exhib- in the Head Start program (Figure 1.2). During that
ited. As part of this research, she constructed tests time she realized that most of the children who
to study the behavioral manifestations of sensory later demonstrated problems were “missed” by tra-
processing problems. Findings from a series of ditional screening tests. She was awarded federal
nation-wide studies comparing normal children funding to standardize the Miller Assessment for
and children with problems processing sensory Preschoolers and to develop the FirstSTEP, a
information eventually identified patterns of sen- shorter screening tool (Miller, 1982, 1988, 1993).
sory integrative impairment (Bundy, Lane, & Dr. Miller later developed and standardized the
Toddler and Infant Motor Evaluation and made a
large step forward in intelligence testing when she
co-authored the Leiter International Performance
Scale, which provides a nonmotor, nonverbal alter-
native to traditional IQ scales (Miller, 1988, 1982;
Roid & Miller, 1997).
Dr. Miller used tests as a way to identify and
evaluate children with sensory processing prob-
lems and developmental delays. Since 1995,
Dr. Miller and her team have run a full-time pro-
gram of research investigating the underlying
mechanisms of sensory dysfunction and evaluating
the effectiveness of occupational therapy for chil-
dren with these disorders (McIntosh, Miller, Shyu,
& Hagerman, 1999; Miller et al., 1999). This
research has included diverse investigations such
as quantitative studies of the sympathetic nervous
system based on descriptions of children’s “fight or
Figure 1.1 Dr. A. Jean Ayres. (Reprinted with flight” responses and qualitative studies of parental
permission from Kielhofner, G. Conceptual hopes for therapy outcomes.
foundations of occupational therapy, 3rd ed. Dr. Laura Gitlin, an applied research sociolo-
Philadelphia: FA Davis, 2004, p. 47.) gist, has a long-standing passion for developing
2 Section 1 The Nature and Scope of Inquiry in a Practice Profession
been a natural fit. Since 1988, in partnership with

her occupational therapy colleagues, Dr. Gitlin has
received federal and foundation funding to study
the adaptive processes of older adults, the role and
benefits of adaptive equipment and home environ-
mental modification for older people with func-
tional impairments, the quality of life of persons
with dementia living at home, and interventions
to enable older adults with functional difficul-
ties to age in place at home (Gitlin & Corcoran, in
press; Gitlin, Luborsky, & Schemm, 1998; Gitlin,
Swenson Miller, & Boyce, 1999). Her recent study
on physical frailty has demonstrated the effective-
ness of a combined occupational therapy and phys-
ical therapy intervention in reducing difficulties
with activities of daily living.
In addition, Dr. Gitlin and her team have been
investigating the most effective approaches to sup-
porting families to manage dementia-related care
challenges (Gitlin, Schinfeld, Winter, Corcoran,
Figure 1.2 Dr. Lucy Jane Miller. & Hauck, 2002). This research has involved devel-
oping various measures such as a standardized
and testing innovative health and human services assessment of the physical features of home envi-
that can make a difference in the lives of persons ronments that support or hinder functionality
with disabilities and their families (Figure 1.3). (Home Environmental Assessment Protocol
Working out of the traditions of Kurt Lewin’s field [HEAP]), a tool to measure the type and range of
theory and M. Powell Lawton’s competence- care strategies families use (Task Management
environmental press framework in aging research, Strategy Index [TMSI]), and a tool to evaluate the
her collaboration with occupational therapy has strength of the therapeutic engagement process
with families (Therapeutic Engagement Index
[TEI]) (Gitlin et al. 2002).
This program of research has yielded an effec-
tive intervention with families, the Environmental
Skill-building Program (ESP). It has also gener-
ated other related approaches that enhance the
quality of life of both persons with dementia and
their families (Gitlin, et al., 2003). Currently, Dr.
Gitlin and her team are working on ways of trans-
lating these evidence-based interventions into
reimbursable services.
Kerstin Tham, a Swedish occupational thera-
pist, became interested in a particular problem
some of her clients with a cerebrovascular accident
(CVA) experienced (Figure 1.4). This problem,
unilateral neglect, meant that, following a CVA,
people no longer recognized half of their own bod-
ies nor did they perceive half of the world. As a
consequence, they neglected these regions of the
self and the world, washing only one side of the
body, eating only the food on one half of the plate,
and so on.
A great deal of research has been published
about the problem of neglect. Moreover, a number
of training approaches have been developed to
Figure 1.3 Dr. Laura N. Gitlin. help people with neglect overcome the problem.
Chapter 1 The Necessity of Research in a Profession 3
Figure 1.4 Dr. Kerstin Tham (left) and student Ann-Helen Patomella.
However, these approaches have not been shown In a subsequent investigation, Dr. Tham and a
to be very successful in improving function. colleague went on to examine how behaviors of
Dr. Tham became convinced that the research other people influenced the experiences and
that had described unilat- behavior of the person
eral neglect had one large with neglect (Tham &
flaw. It always examined Without the development of Kielhofner, 2003). She
how neglect appeared a research base to refine continues this line of
from the outside—that is, research, which offers a
how it appeared to clini- and provide evidence about new approach to under-
cians and researchers. It the value of its practice, standing and providing
never asked the person services to persons with
with CVA how it was to occupational therapy unilateral neglect. She
experience neglect. So, simply will not survive, and her doctoral students
she decided to undertake are now also examining
research that would de- much less thrive, as a the experience of persons
scribe neglect phenome- health profession. having other types of
nologically (i.e., from the perceptual cognitive im-
point of view of the per- pairments after acquired
son who had it), with the aim of better providing brain damage (Erikson, Karlsson, Söderström, &
services for it. Tham, 2004; Lampinen, & Tham, 2003).
In a qualitative study in which she observed and Each of these four researchers has engaged
interviewed four women over an extended period in widely different programs of occupational ther-
of time, she and her colleagues came to provide apy research. They used different methods on very
some startling insights into the nature of unilateral different populations and sample sizes. Nonethe-
neglect (Tham, Borell, & Gustavsson, 2000). For less, the work of each of these investigators stands
instance, they found that people with neglect as an example of why research is important to
believed that the neglected body parts were not the field overall. That is, their research has pro-
their own or not attached to their bodies. Their vided:
research described a natural course of discovery in
which persons with neglect came to understand • Knowledge and know-how that guides therapists
that they had neglect and were able to make sense in their everyday work, and
of their strange and chaotic experiences of the self • Evidence that assures others about the impact of
and the world. occupational therapy services.
The Profession’s It seems clear that as administrators and

policymakers render decision about how
Research Obligation health care providers are used and reim-
bursed, those disciplines with objective evi-
Every health profession asks its clients and the dence of their effectiveness and efficiency
public to have a level of confidence in the worth of will have a competitive advantage. (p.197)
its services. To justify that confidence, the profes- He concludes, quite correctly, that research is
sion must enable its members to offer high-quality an economic imperative for the profession:
services which will benefit the client. Thus, when
healthcare professionals provide service to clients, Without the development of a research base
the knowledge and skills they use should be “justi- to refine and provide evidence about the
fied in terms of a systematic and shared body of value of its practice, occupational therapy
professional knowledge” (Polgar & Thomas, 2000, simply will not survive, much less thrive, as
p. 3). This knowledge includes the theory that a health profession (Christiansen, 1983)
informs practice and the tools and procedures that
are used in practice. Research is the means by
which the profession generates evidence to test and Evidence-Based Practice
validate its theories and to examine and demon-
strate the utility of its practice tools and procedures. The obligation of the profession to conduct
Therefore, every profession has an ongoing obliga- research that refines and validates its knowledge
tion to undertake systematic and sustained research base is paralleled by an obligation of individual
(Portney & Watkins, 2000). therapists to engage in evidence-based practice
(Taylor, 2000). Accordingly, whenever possible,
practitioners should select intervention strategies
The Necessity of Research and tools that have been empirically demonstrated
to be effective (Eakin, 1997). Evidence-based
for Professional practitioners integrate their own expertise with the
Recognition and Support best available research evidence. The next section
briefly examines some of the ways that research
provides evidence for practice.
The existence of occupational therapy depends on
societal support. This support ranges from subsi-
dizing educational programs that prepare occupa-
tional therapists to reimbursing occupational
How Research
therapists for their services. Societal support for Supports Practice
healthcare professions cannot be assumed. For
instance, those who make public policy and decide Research supports practice in many different ways
what healthcare services are needed increasingly including:
rely on scientific evidence to decide where limited
• Generating foundational knowledge used by
public and private resources should be directed. As
therapists,
a result, research is increasingly necessary to
• Providing evidence about the need for occupa-
ensure that resources will be available to support
tional therapy services,
the profession. Along these lines Christiansen
• Developing and testing theory that underlies
(1983) notes:
practice, and
• Generating findings about the process and out-
comes of therapy.
“The goal of research in an applied discipline In the section below we examine each of these
such as occupational therapy is to enhance and ways that research generates knowledge to support
refine the knowledge base relevant to practice and advance practice
so that consumers receive the best available
treatment.” (Ottenbacher, 1987, p. 4) Generating Foundational Knowledge
“It is incumbent upon a developing profession to
insure that its steadily growing body of Much of the background information that occupa-
knowledge be verified through research…” tional therapists use on a daily basis would not
(West, 1981, p. 9) exist without research. Often, a long history of
investigation is behind what has become taken-for-
granted knowledge. Knowledge of musculoskele- Together these studies indicated that persons
tal anatomy, neuronal transmission, the milestones with HIV/AIDS would potentially benefit from an
the characterize child development, the nature of individualized intervention designed to help them
personality, and the etiology and prognosis of dis- achieve independent living and employment as
eases have resulted from thousands of studies. they envisioned it. These studies provided a foun-
Over decades, investigators examined these dation on which to propose a study of such an
phenomena, providing accounts that were subse- occupational therapy intervention which is in
quently verified or corrected by others. In time, process as the book goes to press (Paul-Ward,
this knowledge was accumulated and refined until Braveman, Kielhofner, & Levin, 2005).
it became part of the repository of knowledge that
informs occupational therapy practice. This Developing and Testing Occupational
knowledge is ordinarily generated by persons who
Therapy Theory that Explains Practice
are not occupational therapists, but their research
is, nonetheless, important to occupational therapy Every profession makes use of theories that under-
practice. lie and explain its practice. By definition, the expla-
nations offered by a theory are always tentative. By
Providing Evidence of the Need testing these explanations, research allows theory
to be corrected and refined so that it provides
for Occupational Therapy Services
increasingly useful explanations for practice. As
Without clear identification of need, one can nei- will be discussed in the next chapter, ideas about
ther decide what services to provide nor accurately how research refines and tests theory have evolved
evaluate the value of any service. Needs assessment over the centuries, but research remains the pri-
research determines what clients require to achieve mary tool by which a theory can be improved.
some basic standard of health or to improve their In professions, theories are ordinarily organized
situation (Witkin & Altschuld, 1995). It focuses on into frameworks or models that are used to guide
identifying gaps between clients’ desires and their practice. These theories explain problems that ther-
situations (Altschuld & Witkin, 2000). apists address and to justify approaches to solving
Needs assessment is particularly important in them that are used in therapy. Consequently, the
identifying the nature and consequences of new testing and refinement of such theories through
types of disabilities or new circumstances affect- research contributes to advancing practice. Thera-
ing persons with disabilities, and in identifying pists should always judge and place their confi-
problems not previously recognized or understood. dence in the explanations provided by any theory
For example, studies recently indicated that human in relation to the extent to which that theory has
immunodeficiency virus/acquired immunodefi- been tested and developed by research.
ciency syndrome (HIV/AIDS) increasingly affects A wide range of research can be used to test and
individuals from disadvantaged minority popula- develop theory. No single study can ever test all
tions, and individuals with histories of mental ill- aspects of a theory. The following kinds of studies
ness, substance abuse, poverty, limited education, are ordinarily used to examine and develop theory:
and limited work experience (Centers for Disease
• Studies that aim to verify the accuracy of the
Control and Prevention [CDC], 2000; Karon,
concepts. These types of studies ask whether
Felming, Steketee, & De Cock, 2001; Kates,
there is evidence to support the way the concept
Sorian, Crowley, & Summers, 2002). Research has
describes and/or explains some phenomena.
also shown that while newer drug therapies have
• Studies that ask whether there are relation-
lowered AIDS mortality, the chronic disabling
ships between phenomena as specified by the
aspects of the disease and numerous associated
theory.
conditions continue to pose challenges for those
• Studies that compare different groups of partici-
affected (CDC, 2000). Many people with HIV/
pants on concepts that the theory offers to
AIDS struggle to overcome personal, financial,
explain the differences between those groups.
and social challenges that impact their desire to
• Studies that examine the potential of the theory
live independently and return to the workforce
to predict what will happen.
(McReynolds & Garske, 2001). Moreover, despite
these general characteristics of the AIDS popula- Over time, as the evidence accumulates across
tion, a needs assessment study also demonstrated such studies, informed judgments can be made
that individuals’ perceptions of needs differed by about the accuracy and completeness of a theory.
race, ethnicity, and gender (Sankar & Luborsky, Findings from such research ordinarily lead to
2003). alterations in the theory that allow it to offer more
accurate explanations. Since the theories used in Studies that Test Assessments
occupational therapy typically seek to explain Used in Therapy
problems that therapists encounter in practice and
to explain how therapists go about solving those A number of interrelated forms of inquiry are used
problems, these types of studies directly inform to develop and test assessments used in the field;
practice. the aim of such research is to ensure the depend-
The following is an example of research that ability of those methods (Benson & Schell, 1997).
tests theory with implications for practice. Dependable assessments are reliable—that is, they
Occupational therapy practice with persons who yield consistent information in different circum-
have central nervous system damage has been stances, at different times, with different clients,
guided by a theory of how people control move- and when different therapists administer them. A
ment, the motor control model. Toward the end of dependable information-gathering method must
the 20th century, this model, which previously saw also be valid—that is, it must provide the informa-
the control of movement as being directed exclu- tion it is intended to provide. Typical of studies
sively by the brain, began to change. A new con- examining whether an assessment is valid are
ceptualization (Mathiowetz & Bass-Haugen, 1994, those that:
2002) argued that movement is a result of the inter- • Ask experts whether the content of an assessment
action of the human nervous system, the muscu- is coherent and representative of what is intended
loskeletal system, and the environment. This to be gathered (i.e., content validity).
theory emphasized the importance of the task • Analyze the items that make up an assessment to
being done and of the environment (e.g., the determine whether they coalesce to capture the
objects used) in influencing how a person moves. trait they aim to measure.
The implication of this theory was that the tasks • Ask whether the assessment correlates with
chosen and the objects used in therapy would have measures of concepts that are expected to concur
an impact on recovery of coordinated movement. and whether it diverges from those with which no
Occupational therapists conducted research that relationship is expected.
illustrated clearly that the nature of the task being • Determine whether they can differentiate
done and the environment do affect the quality between different groups of people.
of movement (Lin, Wu, & Trombly, 1998;
Mathiowetz & Bass- Haugen, 1994; Wu, Trombly, In addition to studies that examine reliability
& Lin, 1994). These and other studies (Ma and and validity of assessments, there are studies that
Trombly, 2002; Trombly & Ma, 2002) now pro- examine the clinical utility of assessments. Such
vide evidence that tasks involving meaningful studies may ask therapists and/or clients whether
objects and goal-oriented activities positively they find the assessments informative and useful
influence performance and motor learning. for identifying problems and making decisions
about theory. The development of any assessment
Providing Evidence About the Nature ordinarily involves a series of studies that con-
tribute to the ongoing improvement of the assess-
and Outcomes of Therapy ment over time.
Many types of studies examine aspects of occupa-
tional therapy practice and its outcomes. The most
typical of these studies are those that:
Studies of Clinical Reasoning
Occupational therapists work with clients to iden-
• Are undertaken to develop and test assessments
tify their clients’ problems and choose a course of
that are used in practice,
action to do something about them. This process is
• Examine the clinical reasoning of therapists
referred to as clinical reasoning (Rogers, 1983;
when they are making decisions about therapy,
Schon, 1983). An important area of research in
• Determine the kinds of outcomes that result from
occupational therapy has been investigations that
therapy,
examine clinical reasoning.
• Examine the process of therapy, asking what
One of the most influential studies of clinical
goes on in therapy, or
reasoning by Mattingly and Flemming (1994)
• Use participatory methods to investigate and
identified different types of reasoning that charac-
improve services in a specific context.
terized occupational therapy practice. Their
Below we examine each of these kinds of stud- research has served as a framework for under-
ies in more detail. standing how occupational therapists make sense
of and take action with reference to their clients’ assigned to occupational therapy. This study
problems and challenges in therapy. showed how the meanings of therapy intended by
therapists were often not received by or in concert
Outcomes Research with the client’s meaning. The study findings
underscored the importance of therapists having
Outcomes research is concerned with the results of knowledge of the client’s narrative and organizing
occupational therapy. Investigations that examine therapy as a series of events that enter into that nar-
the outcomes of occupational therapy services rative. Such studies of the process of therapy pro-
include: vide important information about how therapy can
be improved to better meet client needs.
• Investigations of a specific intervention strategy
or technique,
• Studies of comprehensive occupational therapy Participatory Research
programs, and A new and rapidly growing approach to investiga-
• Inquiries that examine the contribution of occu- tion is participatory research. This approach
pational therapy to an interdisciplinary program involves researchers, therapists, and clients doing
of services (Kielhofner, Hammel, Helfrich, research together to develop and test occupational
Finlayson, & Taylor, 2004). therapy services. Participatory research embraces
The study of occupational therapy techniques the idea of partnership in which all the constituents
and approaches helps refine understanding of these work together and share power and responsibility
discrete elements of practice. This type of research to investigate, improve, and determine the out-
examines outcomes specific to an intended inter- comes of service. It also involves innovation in
vention. Such studies may also seek to determine which new services are created to respond to prob-
the relative impact of different techniques or lems that are mutually identified by researchers
approaches (e.g., comparisons between individual and therapists/clients.
versus group interventions). This type of research is especially useful for
Studies of comprehensive occupational therapy contributing knowledge that practitioners can
programs ask whether an entire package of serv- readily use and that consumers will find relevant to
ices produces a desired outcome. Such studies typ- their needs. A recent example of this kind of study
ically examine the impact of services on such involved developing and evaluating a consumer-
outcomes as independent living, employment, or driven program for individuals with chronic
enhanced school performance. A well-known fatigue syndrome. This program provided clients
example of this type of research is a study by Clark with an opportunity to learn self-advocacy skills;
et al. (1997), which documented the positive out- improve quality of life, functional capacity, and
comes of an occupational therapy program for well coping skills; acquire resources; and achieve
elderly persons. Finally, studies that examine the increased quality of life (Taylor, 2004).
impact of interdisciplinary services can also docu-
ment the impact of the occupational therapy com-
ponent of such services. Conclusion
This chapter introduced the necessity of research
Inquiry Into the Processes of Therapy
for occupational therapy. We saw that research
It is important not only to understand whether gives clients and the public reason to have con-
interventions work, but also to know why they fidence in occupational therapy. We also saw that
work or do not work. Studies that examine the it provides the rationale for administrators and pol-
impact of interventions increasingly focus on icymakers to support the occupational therapy
identifying the underlying mechanism of change services.
(Gitlin et al., 2000). Often an important prelude The chapter also examined how research cre-
to designing intervention outcome studies is to ates the knowledge and know-how that therapists
examine what goes in therapy in order to improve use in practice by testing theory and practice.
upon services before they are more formally Figure 1.5 summarizes the dynamic relationship
tested. between theory, research, and practice. As it illus-
An example is a study by Helfrich and trates, each of these key elements of the profession
Kielhofner (1994) that examined how client occu- has an influence on the other. Theory and research
pational narratives influenced the meaning clients evidence guides practice. Practice raises problems
Explains phenomena addressed P

Theory in practice and provides a
rationale for practice. R
A
C
T
Yields
findings that Generates clinical puzzles
I
lead to
theory and
addressed by theory and research. C
that examine E
and refine
existing
theory.
Generates evidence about the effectiveness of

Research specific practices.
Figure 1.5 The relationship of theory, research, and practice.
and questions to be addressed in theory and Christiansen, C. (1983). An economic imperative. The
research. Research tests theory and practice, pro- Occupational Therapy Journal of Research, 3 (4),
195–198.
viding information about their validity and utility, Clark, F., Azen, S. P., Zemke, R., Jackson, J., Carlson,
respectively. M., Mandel, D., et al. (1997). Occupational therapy
The remainder of this text explains the nature, for independent-living older adults: A randomized
scope, design, methods, and processes that make controlled trial. Journal of the American Medical
Association, 278, 1321–1326.
up research. It also illustrates the wide range of
Eakin, P. (1997). The Casson Memorial Lecture 1997:
tools that researchers use for their inquiries. Shifting the balance—Evidence based practice. The
Throughout the text, as one encounters multiple British Journal of Occupational Therapy, 60 (7),
discussions of how research is done, it is important 290–294.
not to lose sight of why it is done. One would do Erikson, A., Karlsson, G., Söderström, M., & Tham,
K. (2004). A training apartment with electronic aids
well to remember Yerxa’s (1987, p. 415) observa- to daily living: Lived experiences of persons with brain
tion that “Research is essential to achieving our damage. American Journal of Occupational Therapy,
aspirations for our patients and our hopes and 58, 261–271.
dreams for our profession.” Gitlin, L. N. & Corcoran, M. (in press). An occupational
therapy guide to helping caregivers of persons with
dementia: The home environment skill-building pro-
REFERENCES gram. American Occupational Therapy Association.
Altschuld, J. W., & Witkin, B. R. (2000). From needs Gitlin, L. N., Corcoran, M., Martindale-Adams, J., Malone,
assessment to action: Transforming needs into solution M. A., Stevens, A., & Winter, L. (2000). Identifying
strategies. Thousands Oak, CA: SAGE Publications. mechanisms of action: Why and how does intervention
Ayres, A. J. (1979). Sensory integration and the child. Los work? In R. Schulz (Ed.). Handbook of dementia care
Angeles: Western Psychological Services. giving: Evidence-based interventions for family care-
Benson J., & Schell, B. A. (1997). Measurement theory: givers. New York: Springer.
Application to occupational and physical therapy. Gitlin, L. N., Luborsky, M., & Schemm, R. L. (1998).
In J. Van Deusen, & D. Brunt (Eds.), Assessment in Emerging concerns of older stroke patients about
Occupational Therapy and Physical Therapy. assistive device use. The Gerontologist, 3 (2), 169–180.
Philadelphia: W. B. Saunders. Gitlin, L. N., Schinfeld, S., Winter, L., Corcoran, M., &
Bundy, A. C., Lane, S. J., & Murray, E. A. (2002). (Eds.), Hauck, W. (2002). Evaluating home environments
Sensory integration: Theory and practice (2nd ed., of person with dementia: Interrater reliability and
pp. 169–198). Philadelphia: F. A. Davis. validity of the home environmental assessment
Centers for Disease Control and Prevention (CDC). (2000). protocol (HEAP). Disability and Rehabilitation, 24,
HIV/AIDS Surveillance Supplemental Report, 7 (1). 59–71.
Atlanta: U.S. Department of Health and Human Gitlin, L. N., Swenson Miller, K., & Boyce, A. (1999).
Services, Centers for Disease Control and Prevention. Bathroom modifications for frail elderly renters: out-
comes of a community-based program. Technology Miller, L. J., McIntosh, D. N., McGrath, J., Shyu, V.,
and Disability, 10, 141–149. Lampe, M., Taylor, A. K., et al. (1999). Electrodermal
Gitlin, L. N., Winter, L., Corcoran, M., Dennis, M., responses to sensory stimuli in individuals with fragile
Schinfeld, S., & Hauck, W. (2003). Effects of the X syndrome: A preliminary report. American Journal
Home Environmental Skill-building Program on the of Medical Genetics, 83 (4), 268–279.
caregiver-care recipient dyad: Six-month outcomes Ottenbacher, K. J. (1987). A scholarly rite of passage.
from the Philadelphia REACH initiative. The Occupational Therapy Journal of Research, 7 (1), 3–5.
Gerontologist, 43 (4), 532–546. Paul-Ward, A., Braveman, B., Kielhofner, G., & Levin, M.
Gitlin, L. N., Winter, L., Dennis, M., Corcoran, M., (2005). Developing employment services for individu-
Schinfeld, S., & Hauck, W. (2002). Strategies used als with HIV/AIDS: Participatory action strategies at
by families to simplify tasks for individuals with work. Journal of Vocational Rehabilitation, 22, 85–93.
Alzheimer’s disease and related disorders: Psycho- Polgar, S., & Thomas, S. A. (2000). Introduction to
metric analysis of the task management strategy index research in the health sciences. Edinburgh: Churchill
(TMSI). The Gerontologist, 42, 61–69. Livingston.
Helfrich, C. & Kielhofner, G. (1994). Volitional narratives Portney, L. G., & Watkins, M. P. (2000). Foundations of
and the meaning of therapy. The American Journal of clinical research: Application to practice (2nd ed.).
Occupational Therapy, 48, 319–326. Upper Saddle River, NJ: Prentice-Hall.
Karon, J. M., Fleming, P. L., Steketee, R. W., & De Cock, Rogers, J. C. (1983). Eleanor Clarke Slagle Lectureship—
K. M. (2001). HIV in the United States at the turn of 1983; Clinical reasoning: The ethics, science, and art.
the century: An epidemic in transition. American American Journal of Occupational Therapy, 37 (9),
Journal of Public Health, 91 (7), 1060–1068. 601–616.
Kates, J. R., Sorian, J. S., Crowley, T. A., & Summers, Roid, G. H., & Miller, L. J. (1997). Leiter International
T. A. (2002). Critical policy challenges in the third Performance Scale Revised. Wood Dale, IL: Stoelting
decade of the HIV/AIDS epidemic. American Journal Co.
of Public Health, 92 (7), 1060–1063. Sankar, A., & Luborsky, M. (2003). Developing a commu-
Kielhofner, G., Hammel, J., Helfrich, C., Finlayson, M., nity-based definition of needs for persons living with
& Taylor, R. R. (2004). Studying practice and its out- chronic HIV. Human Organization, 62 (2), 153–165.
comes: A conceptual approach. American Journal of Schon, D. (1983). The reflective practitioner: How profes-
Occupational Therapy, 58, 15–23. sionals think in action. New York: Basic Books.
Lampinen, J., & Tham, K. (2003). Interaction with the Taylor, M. C. (2000). Evidence-based practice for occupa-
physical environment in everyday occupation after tional therapists. Oxford, UK: Blackwell Science.
stroke: A phenomenological study of visuospatial Taylor, R. (2004). Quality of life and symptom severity in
agnosia. Scandinavian Journal of Occupational individuals with chronic fatigue syndrome; findings
Therapy, 10, 147–156. form a randomized clinical trial. American Journal of
Lin, K. C., Wu, C. Y., & Trombly, C. A. (1998). Effects Occupational Therapy 58 (1), 35–43.
of task goal on movement kinematics and line bisection Tham, K., Borell, L., & Gustavsson, A. (2000). The dis-
performance in adults without disabilities. American covery of disability: A phenomenological study of uni-
Journal of Occupational Therapy, 52, 179–187. lateral neglect. American Journal of Occupational
Ma, H., & Trombly, C. A. (2002). A synthesis of the Therapy, 54, 398–406.
effects of occupational therapy for persons with stroke, Tham, K., & Kielhofner, G. (2003). Impact of the social
part II: Remediation of impairments. American Journal environment on occupational experience and per-
of Occupational Therapy, 56, 260–274. formance among persons with unilateral neglect.
Mathiowetz, V., & Bass-Haugen, J. (1994). Motor behavior American Journal of Occupational Therapy, 57 (4),
research: Implications for therapeutic approaches to 403–412.
central nervous system dysfunction. American Journal Trombly, C. A., & Ma, H. (2002). A synthesis of the
of Occupational Therapy, 48, 733–745. effects of occupational therapy for persons with
Mathiowetz, V., & Bass-Haugen, J. (2002). Assessing abili- stroke, part I: Restoration of roles, tasks, and activities.
ties and capacities: Motor behavior. In C. A. Trombly American Journal of Occupational Therapy, 56,
& M. V. Radomski (Eds.), Occupational therapy for 250–259.
physical dysfunction (5th ed., pp. 137–158). Baltimore: Walker, K. F. (2004). A. Jean Ayres. In K. F. Walker & F.
Lippincott Williams & Wilkins. M. Ludwig (Eds.), Perspectives on theory for the prac-
Mattingly, C., & Flemming, M. (1994). Clinical reaso- tice of occupational therapy (3rd ed., pp. 145–236).
ning: Forms of inquiry in a therapeutic practice. Austin, TX: Pro-ed.
Philadelphia: F. A. Davis. West, W. (1981). A journal of research in occupational
McIntosh, D. N., Miller, L. J., Shyu, V., & Hagerman, therapy: The response, the responsibility. Occupational
R. (1999). Sensory-modulation disruption, electro- Therapy Journal of Research, 1 (1), 7–12.
dermal responses, and functional behaviors. Deve- Witkin, B. R., & Altschuld, J. W. (1995). Planning and
lopmental Medicine and Child Neurology, conducting needs assessments: A practical guide.
41, 608–615. The Psychological Corporation. Thousand Oaks, CA: SAGE Publications.
McReynolds, C. J., & Garske, G. G. (2001). Current issues Wu, C. Y., & Trombly, C. A., & Lin, K. C. (1994). The
in HIV disease and AIDS: Implications for health and relationship between occupational form and occupa-
rehabilitation professionals. Work, 17, 117–124. tional performance: A kinematic perspective. American
Miller, L. J. (1988, 1982). Miller assessment for preschool- Journal of Occupational Therapy, 48, 679–687.
ers. San Antonio, TX:The Psychological Corporation. Yerxa, E. J. (1987). Research: The key to the development
Miller, L. J. (1993). FirstSTEP (screening test for evaluat- of occupational therapy as an academic discipline.
ing preschoolers): Manual. San Antonio, TX: The American Journal of Occupational Therapy, 41 (7),
Psychological Corporation. 415–419
C H A P T E R 2
The Aim of Research: Philosophical

Foundations of Scientific Inquiry
Gary Kielhofner
Although the philosophical underpinnings of stud- As will be seen, each of these periods offers a
ies are rarely discussed in research publications, different understanding of the aims and conse-
they are always lurking in the background. quences of conducting inquiry.
Consider, for instance, the following contrasts. There is no attempt here to provide a compre-
Some researchers will take pains to demonstrate hensive discussion of the philosophy of science.
how subjectivity and personal bias were eliminated Rather, the aims are to capture some key ideas and
from their studies, while others will detail how their highlight the epistemological stances that are
personal histories and subjective experiences likely to be implied in the range of research found
shaped and informed their investigation. Some in occupational therapy.
researchers will be concerned with what their find-
ings tell us about the nature of reality, while others
focus instead on helping the reader understand how
reality was experienced by those who where stud-
Classicism: Origins of
ied. Some investigators carefully distance them- the Scientific Method
selves from subjects, while other investigators
immerse themselves in the lives of those they study. Aristotle is generally considered the first philoso-
Still other investigators invite those they study to be pher of science in the Western world inasmuch
equal partners in the research enterprise. as he formulated the foundations of empiricism. He
Such differences in proposed that the world
the conduct of research could be understood
are not simply incidental Although the philosophical by detailed observa-
to the methods used. tion combined with a
They reflect fundamen-
underpinnings of studies are systematizing of those
tally different philosoph- rarely discussed in research observations. To this
ical stances on reality, publications, they are always end, Aristotle outlined
objectivity, and human research as a process of
knowing. Anyone who lurking in the background. going from observations
participates in research of the natural world (i.e.,
or wishes to be a con- the objects and events
sumer of research should appreciate the philo- that we perceive through the senses) to explanation,
sophic underpinnings that shape the fundamental and then back to observations (Losee, 1993). The
attitudes and beliefs of those who conducted the first part of this process, induction, involves creat-
research. In the end, these may be as important and ing explanations (i.e., theory) from specific obser-
consequential as the researcher’s adherence to vations. In the second phase, deduction, one
accepted methods and protocols (Kaplan, 1964). derives from the theory specific statements or pre-
This chapter examines the philosophical foun- dictions that could be examined to see if they bear
dations of scientific inquiry and illustrates how up under observation of the natural world. When
ideas about inquiry and the knowledge it creates statements deduced from a theory were shown to
have evolved. This historical account will examine correspond to what was observed, one had proof of
four periods of the philosophy of science: the veracity of the theory. This induction–deduc-
tion process (Figure 2.1) is still at the core of all
• Classicism, research, though different types of research view it
• Modernism, differently, emphasize one or another of the phases,
• Critical Modernism, and and have different ideas about the role of the inves-
• Postmodernism. tigator in the process.
Chapter 2 The Aim of Research: Philosophical Foundations of Scientific Inquiry 11
Induction Theory
Deduction
Observations of the natural world
Figure 2.1 Aristotle’s inductive–deductive method.
The Search for Truth larger theory following strict logical principles.
However, the inductive phase was problematic. It
Aristotle and other early philosophers of science required something more than logic. Induction
were fundamentally concerned with separating sci- involved an intuitive leap. In arriving at explana-
entific knowing from the fallibility of ordinary tions, a scholar had to invent first principles that
knowing. Thus, they focused on how to ensure the were the foundation of the explanation that any
truthfulness of knowledge generated through the theory provided. These first principles could only
inductive–deductive cycle (Klee, 1997; Losee, be assumed to be true, since they could not be
1993). This truth, they believed, depended on logic. proved. Confronted with this problem, Aristotle
They reasoned that if pure logic was used to con- opted for a faith in their truth. He justified this faith
nect the natural world to scientific knowledge, the by arguing that the natural world presented itself in
latter could be demonstrated to be true. Thus, they a fashion that explicated its most basic laws. In
critically examined how logic was used in both the other words, the first principles were self-evident.
inductive and deductive phases of research. In the late 15th and early 16th centuries,
For Aristotle and many who followed, the Galileo criticized Aristotle’s self-evident first prin-
deductive stage readily conformed to rigorous ciples as being too metaphysical and therefore
logic—that is, the specific statements that were unscientific. Galileo sought, instead, to ground sci-
tested through research could be deduced from the entific explanations in the “obvious truth” of math-
ematical descriptions that perfectly fit the natural
Theory and Its Components world (Klee, 1997; Losee, 1993). He and his con-
temporaries reasoned that the correspondence
between mathematical concepts and the natural
Theory is a network of explanations; it provides
concepts that label and describe phenomena and world could not be coincidental, and therefore rep-
postulates that specify relationships between con- resented an obvious truth.
cepts. Concepts describe, define, and provide a Nonetheless, like Aristotle, Galileo was ulti-
specific way of seeing and thinking about some mately confronted with the fact that the process of
entity, quality, or process. For example, the con- induction involved intuition. He made extensive
cept of “strength” refers to a characteristic of use of imagining phenomena that he could not
muscles (i.e., their ability to produce tension for observe (e.g., free fall in a vacuum) to arrive at
maintaining postural control and for moving explanations. In the end this imaginative process
body parts). Exercise is a concept that refers to a also required him to use certain unavoidable non-
process (i.e., the use of muscles to produce force
mathematical assumptions to construct his expla-
against resistance).
Postulates posit relationships between con- nations. Thus, a complete logical connection
cepts, asserting how the characteristics or between the natural world and theory was still not
processes to which concepts refer are organized achieved for the inductive process.
or put together. An example of such a postulate Descartes, a contemporary of Galileo, was not
is: exercise increases the ability of muscles to convinced that mathematical correspondence was
produce force. When several concepts and sufficient to constitute the truth of theoretical first
postulates are linked together, they constitute principles. He sought to resolve the problem by
a whole network of explanations that make up doubting all potential first principles in search for
a given theory those that were beyond doubt (Watson, 2002). His
search for principles beyond all doubt resulted in
the famous dictum: Cogito, ergo sum. (I think, Modernism and Progressive Science
therefore I am.) Like Aristotle and Galileo,
Descartes also was unable to avoid the fact that he The philosophical ideas referred to as “logical
made intuitive leaps in his inductive reasoning. He positivism” contained an important idea of
referred to these intuitive leaps as using analogies progress that is also a cornerstone of modernism.
(e.g., inferring planetary movement had to be cir- Born out of 18th century Enlightenment, mod-
cular based on observations of other naturally ernism included not only faith in science as a
occurring phenomena such as whirlpools). While method but also a belief that true human progress
would result from science. That is, science was
he sought to defend the logic of analogical think-
expected to continually improve the human con-
ing, like those before him, Descartes was unable to dition as knowledge accumulated and was used
reduce induction to pure logic. It remained a cre- to the betterment of society and its institutions.
ative act that went beyond logical thought. Thus, modernism optimistically sought “univer-
In the end, philosophers of science were unable sal human emancipation through mobilization of
to avoid the conclusion that induction involved the powers of technology, science, and reason
more than pure logic. Even today, it is understood (Harvey, 1990, p. 41).
that induction is an intuitive and creative process.
Moreover, they were also unable to resolve the
problem that the first principles that were generated
through induction could not be proven. Early the other theoretical principles that could be
philosophers attempted to justify the first principles deduced from the first principles were labeled
that resulted from induction on the grounds of self- theorems),
evidence, mathematical correspondence, and truth • Specifying how the theorems were correlated
beyond doubt. No matter how well argued these with the empirical world so that they could be
justifications, the bottom line was that they each systematically tested, and
demanded some type of belief that went beyond • Testing the theorems through observation of the
logic. That intuition and faith remained unavoid- natural world.
able parts of induction eventually led philosophers
Although this approach admitted that the first
to search for truth in the deductive phase.
principles could not be proved, they had to yield
theorems that did not contradict the natural world.
Modernism: From Absolute Hence, first principles that yielded incorrect theo-
rems would ultimately be understood to be false
Truth to Positivistic Science and, therefore, could be eliminated (Cohen, 1958).
A critical turning point in the philosophy of sci- Logical Positivism

ence was ushered in by Newton and his contempo-
This approach to research meant that theories had
raries in the 17th century (Cohen, 1958; Klee,
to be contingent and subject to revision as the evi-
1997; Losee, 1993). These early researchers
dence generated in research required it. Thus, the
replaced the concern for absolute truth with con-
approach of Newton and his contemporaries did
cern for how to correct errors in knowledge. They
not seek to claim that their theories were true.
envisioned science as a process of testing and ver-
Rather, they asserted that any theory was a possi-
ification of the theory created through inductive
ble, but not necessarily infallible, explanation. A
reasoning. Newton accepted the idea of intuition
theory’s plausibility depended on whether state-
and creativity in generating theory as a necessary
ments that could be logically deduced from it held
process, since he made extensive use of such
up in the natural world. If observations of the nat-
thinking in his own theorizing. Thus, in order to
ural world did not bear out what was logically
establish confidence in his theories, Newton
deduced from a theory (i.e., were disproved), then
focused on the deductive phase.
the theory would ultimately be rejected. Within
To specify how error could be identified in the-
this framework, while truth was not immediately at
ory through testing, he outlined an axiomatic
hand, scientists could make progress toward it.
method (Cohen & Smith, 2002). It contained three
Research could systematically identify what was
steps:
false through empirical testing. What survived the
• Identifying within the theory those first princi- test of time and evidence would be increasingly
ples that could not be deduced from any others closer to the truth. This view that research allowed
that were, therefore, ultimately not provable theory to progress toward truth came to be known
(these first principles were labeled axioms and as logical positivism.
Subsequent philosophers of science in the dependability of theories relied on their logical

logical positivist tradition focused on improving interconnectedness.
the logical rigor of methods through which In the 19th century, John Stuart Mill continued
researchers induced theory from observation and the search to identify rigorously logical procedures
then went about testing the fit of theoretical expla- for arriving at theoretical explanations (Skorupski,
nations with the empirical world. The work of 1998). His greatest contributions were the further
these philosophers of science represents a complex systemization of the logical process of induction.
and often subtly nuanced discussion of how He developed four laws that governed theorizing
empiricism was possible. Below we examine about causation:
only a few highlights of the kinds of contributions
• Agreement,
that major philosophers made to refine the under-
• Concomitant variation,
standing of how knowledge was generated and
• Residues, and
refined.
• Difference (Skorupski, 1998).
In the 18th century, two of the most important
philosophers refining ideas about the logical According to the method of agreement, when a
processes that linked the theoretical explanation to researcher observed that a certain variable pre-
the observable world were David Hume and ceded another variable under a number of circum-
Immanuel Kant (Klee, 1997; Losee, 1993). They stances, it was reasonable to assert the probability
represent two distinct schools of thought. Thus, that the first variable caused the second. In con-
each emphasized different aspects of the empirical comitant variation, causation could be asserted
method. based on the observation that two variables are
Hume believed that all human understanding noted to vary together and in the same direction
began with sensations of the world and that all (e.g., more of one is accompanied with more of
human knowledge comes from experience another). In the method of residues, a causal rela-
(Jenkins, 1992). Thus, for him, research was the tionship between one variable and another could
systematic means of building up knowledge from be made when the causal relationships among
experience of the world. Kant disagreed with other variables are removed and the relationship of
Hume’s assertion that all human knowledge was one variable preceding another variable remains.
built up incrementally through experience (Kant, Mill considered the method of difference to be
1996). Rather, his thesis of “Transcendental the most important. In this instance, causation is
Idealism” argued that the human mind played an asserted when the presence of the first variable is
active role in how humans experience and make followed by the second and when, in the first vari-
sense of the world. Nonetheless, he emphasized, able’s absence, the second does not occur. The
along with Hume, that objective knowledge of the method of difference led to the use of experiments
natural world was possible. in which presence of a presumed causable variable
Hume refined the logic by which one could is systematically manipulated.
explain the experienced world and test the veracity In the 20th century, philosophers of science
of those explanations (Jenkins, 1992). He differen- continued to construct a logical framework of sci-
tiated between two types of scientific statements: ence. Two important philosophers, Hempel and
(1) relational statements that derived their truth Oppenheim, exemplify the efforts of this period
from their internally consistent logic, irrespective (Klee, 1997; Losee, 1993). They underscored the
of any relationship to the empirical world and (2) importance of theory being able to predict phe-
matter-of-fact statements whose truth could be nomena. For one to have confidence in an expla-
claimed only by reference to empirical data. Hume nation, it had to be capable of generating
was primarily concerned with how matter-of-fact predictions that could be investigated through
statements could be tested and disproved if they research. Their work focused on how one logically
were inaccurate and he helped to refine the logic of derived observable predictive statements from the-
this process. oretical explanations.
Because Kant focused on how the human mind Nagel (1949) built on their work by specifying
ordered understanding of the world, his primary the difference between theoretical and observa-
contributions are in the area of theory construction. tional statements. Observational statements, he
He argued that logic provided the ultimate rules argued, did not go beyond what was actually
according to which theories could be constructed. observed, and were therefore not “contaminated”
He argued for the importance of any new theory by theory. This was an important point since, if the
being able to be incorporated logically into older actual test of a theory is to be objective, then the
empirically established ones. In his view, the statement that is being tested cannot be “biased”
by the theory. It has to stand apart from the theory These two conclusions basically undermine the
and merely link the theory logically to something whole idea of a progressive, self-correcting sci-
that can be observed in order to determine whether ence that incrementally eliminates error and
or not the theory accurately predicts the phenom- thereby moves toward truth. Instead, these argu-
ena under question. ments point out that a theory, at best, represents
one possible explanation of the events it addresses
The Critique of Positivism (Hesse, 1970; Scriven, 1970).
Logical positivism underscored the importance of
deriving hypotheses logically from theory so that
they could be tested by research and, when incor- Critical Modernism:
rect, shown to be false. However, a major problem
that arose was whether a hypothesis, much less a Rethinking the Role
theory, could actually be demonstrated to be false.
The whole foundation of logical positivism was
of Empiricism
based on the falsifiabilty of hypotheses that
Criticisms of logical positivism ushered in an
allowed research to correct theories.
important new direction in the understanding of
The popular notion was that a crucial experi-
science. This perspective has been labeled critical
ment could be designed for each hypothesis that
modernism (Midgely, 2003). As already noted, an
would, once and for all, decide whether it was
earlier shift had redirected the ideal of science as
false. However, the idea of the crucial experiment
achieving necessary truth toward a conception of
came to be strongly criticized. For instance,
science as progressing toward truth through self-
Grunbaum (1970) argued that no hypothesis
correcting empiricism. Despite their differences,
derived from a theory could be as sufficiently iso-
both of these views ultimately sought to identify
lated from the theory as to provide an absolute test
logical principles and procedures that would
of the theory. This was, in part, because the mean-
emancipate science from the fallibility of ordinary
ing of any hypothesis was not contained solely in
human thinking and knowing.
the statement of that hypothesis but also in the
However, the more philosophers attempted to
entire matrix of concepts from which the hypothe-
isolate the logic of science from other psychologi-
sis was deduced. This means that the understand-
cal processes (e.g., intuition and creativity), the
ing of the hypothesis, including what evidence
more apparent it became that this was not possible.
could constitute its falsity, also depends on the the-
The logical principles that were once considered to
ory from which it is derived.
specify the very nature of science came to be
Therefore, evidence that contradicts a particular
understood as only one property of science. While
hypothesis can easily be explained away. A con-
logic is necessary, it is not sufficient for doing
venient shift in the sense of the hypothesis will
research. Within this new framework the role of
suffice to protect the theory from which the
intuition in induction, the nonprovability of first
hypothesis was derived. Grunbaum’s argument led
principles, and the embeddedness of observations
to the conclusion that there could be no logic of
within their theoretical contexts were looked at
proof and disproof external to any theory. Rather,
anew.
any proposed test of a theory depends on the the-
Take for instance, what White has to say
ory for its sensibility. For instance, Hesse (1970)
about the problem of intuition in the process of
pointed out that all observational terms contained
induction. He notes that the process of “reducing
in hypotheses are theory-dependent. That is, their
concrete experience to artificial abstractions, or to
meaning cannot stand apart from the theory.
put it more precisely, the act of substituting
Therefore, any attempt to capture the empirical
concepts, ‘free inventions of the human intel-
world in the language of a hypothesis irrevocably
lect’….for concrete experiences of the senses…is
commits the researcher to the theory that makes
the very essence of sciencing” (White, 1938, p.
sense of the hypothesis in the first place.
372).
These were not small problems for logical pos-
Moreover, philosophers of science came to see
itivism. If the very observational terms necessary
the incorrigibility of the first principles upon
to generate evidence about a theory are themselves
which all theories must be based, not as a funda-
reliant on the theory for their meaning, then:
mental problem, but as an important clue about the
• A theory can never truly be shown to be false. nature of science. That is, if the most abstract com-
• Evidence cannot be used to show that one theory ponents of a theory cannot be shown to be
is better than another (Hesse, 1970; Scriven, grounded in the empirical world, it is because the-
1970). ory imparts meaning on, rather than extracts mean-
ing from, the natural world. Theory is a creation of How Theories Change
the human mind that makes sense of the world.
The creative process by which researchers origi- Kuhn also provided an account of how theories
nate ideas to make sense of their observations were eventually rejected and replaced with newer
of the natural world is as much a part of the theories. It was not, as supposed in the logical pos-
research process as the logic by which researchers itivist tradition, the result of critical experiments
link the theoretical with the observed (Bronowski, that proved the theory wrong. Instead, theories
1978). were replaced when they were no longer fruitful
These critics of logical positivism identified for generating insights, yielding new problems
flaws in its assertion that research would advance for analysis and/or when they no longer were
theory toward truth. Moreover, they were able to effective in making sense of new observations. In
give creative and meaning-making processes a essence, theories were abandoned not because
place in the research process. However, if research they were shown to be untrue, but because they
does not advance knowledge toward truth, what simply ran out of steam. They were replaced with
then is its purpose? Exactly how does research newer theories that were more effective in generat-
advance knowledge? The answers to these ques- ing insights and problems to investigate and that
tions have been greatly influenced by Thomas made sense of things the old theory could not
Kuhn’s (1962) historical analyses of how research decipher.
was conducted in physical sciences. His work Kuhn also underscored the importance of the
refined understanding of what researchers do when scientific community sharing a common theoreti-
they empirically test theory. Along with other crit- cal perspective (what he refers to as the paradigm).
ics of positivism, Kuhn (1977) argues that when This common understanding refers not only to
investigators collect data they do not directly test the explanations that are given by the theory but
that theory, since “the scientist must premise cur- also to:
rent theory as the rules of his game” (p. 270). He • What problems are considered most important to
further notes that all theories “can be modified by investigate and
a variety of ad hoc adjustments without ceasing to • What techniques or approaches to collecting and
be, in their main lines, the same theories” (Kuhn, analyzing data are considered acceptable or
1977, p. 281). So, instead of testing theory, evi- preferable.
dence generated in research allows the theory to be
adjusted to better fit whatever phenomena it is Toulmin (1961) agrees with Kuhn, noting that
designed to explain. theories both select what the researcher will see as
Kuhn’s insights point out that research does not problems to be investigated and also impart partic-
prove or disprove the theory. However, it does ular meanings to the phenomena that are studied.
improve the theory. Research serves as the basis These problems are not the only possible problems
for generating theories and, thereafter, can be used and the meanings are not the only meanings that
to enhance the fit of that theory to the natural might be brought to bear on the world. However,
world. Said another way, research allows investi- they are the ones that make sense to the commu-
gators to advance the particular understanding nity of researchers at the time. Without them, the
of the phenomena that a given theory offers. As research process could not take place.
noted earlier, theory serves as a way to impart The perspectives, norms, rules, and so forth that
meaning on observations made in research. Once guide how research is done in a particular arena are
in place, theory serves as a schema for guiding the not absolute. Rather they constitute the “culture”
systematic observation of the world. Finally, of the research community. They are binding
because theory explains the world in a particular because:
way, it leads to investigations that refine that expla- • They represent a world view that is shared by all
nation. All of these processes result in the accu- those doing research in a particular area. This
mulation of knowledge. Studies thus add to the allows them to makes sense of what they are
stockpile of information related to any theoretical studying and to make sense of the very process
system. by which they go about studying it.
Over time the knowledge accumulated through • Anyone wanting to be taken seriously as a
research does appear somewhat like the progres- researcher in the area must conform to them.
sion of knowledge envisioned by the logical posi-
tivists—with one important difference. Instead of Kuhn’s work underscores, then, that the scien-
progressing toward truth, theories progress by tific community needs theories or paradigms as
becoming better at the particular way they make contexts for doing research. They give the commu-
sense of the world. nity of researchers a shared way of making sense
of empirical objects and events so that they can be ecological problems and the failure of modern sci-
pondered in a particular way. As White (1938) ence to address the needs of oppressed groups,
notes, theories are arbitrary points of view from women, ethnic minorities, and members of third
which scientists examine and consider reality. world countries).
Theories are inescapable intellectual spaces within Lyotard further argues that scientific disciplines
which researchers conduct their research. are like games with their own rules, boundaries,
and permissible moves. Importantly, what is per-
missible in the game is determined by power struc-
Postmodernism ture of any particular branch of science. Foucault
(1970), in particular, emphasizes this relation
Postmodernism represents the most recent set of between power and knowledge, arguing that the
ideas in the philosophy of science. It is not a coher- production of knowledge is closely tied to social
ent single argument, but rather a set of loosely control and domination. His work provides a basis
related themes. Postmodernists are particularly for many postmodern critiques of how science
critical of the logical positivist perspective in more serves to perpetuate oppression.
extreme ways than the critical modernists (Harvey, As a result of Lyotard’s and Foucault’s work,
1990). The critique of modernism discussed postmodernists are particularly critical of any
earlier pointed out how broad theories that they
it is impossible to disen- see as forms of meta-
tangle the language of a Theories give the community narrative. They argue
hypothesis from the theo- of researchers a shared way that these metanarratives
retical system in which privilege certain groups
it is embedded. The of making sense of empirical and certain perspectives,
philosopher, Wittgenstein objects and events so that while they have no more
(1953) went even further, validity than any other
asserting that language they can be pondered in a “story” that might be nar-
constructs reality. His particular way. rated. Postmodernists
argument leads to the emphasize the right of
conclusion that, since groups to have their own
language determines what humans perceive, sci- voice and speak for their own reality (Harvey,
ence cannot escape its own linguistic blinders. In 1990). For this reason, postmodern thinking has
other words, the very language of science deter- been used by scholars whose work has champi-
mines what the scientist can come to know. oned disenfranchised groups (e.g., women’s stud-
Wittgenstein is generally attributed with begin- ies [Weedon, 1987] and disability studies [Rioux,
ning what has come to be known as “social 1997]).
constructivism,” a standpoint that pervades post- In the end, most postmodernists paint a negative
modern thought. It asserts, in essence, that all view of science. They not only discount the
knowledge, including scientific knowledge, is methodological claims made by the logical posi-
socially constructed and, therefore, relative. In this tivists, but also call into question the value of much
postmodern framework scientific knowledge is no of the information science has created. The ulti-
more privileged than any other source of knowl- mate conclusion of postmodernism is that “there
edge. It is the particular perspective of a particular can be no universals, that absolute truth is an
group of people who have a particular purpose in illusion” (Midgley, 2003, p. 48). Moreover, post-
mind. modernists have critiqued science as being ideolog-
Lyotard’s (1979) critique of the state of scien- ically biased, tied to power structures, and
tific knowledge directly assaults the positivist ultimately contributing to oppression by replacing
approach of modernism. He argues that science is local knowledge with falsely claimed universal
a form of metanarrative. According to Lyotard, the knowledge. Consequently, postmodernists seek to
scientific metanarrative claims that science is a “promote variety, undermine certainty, and pro-
project of human enlightenment based on logic mote local, critical thought” (Midgley, 2003, p. 55).
and empiricism and that promises to create a uni-
fied understanding of the word, work for the good
Critiques of Postmodernism
of all, and improve the human condition. Lyotard
and other postmodernists point out the many fail- A number of severe critiques of postmodernism
ures of science in this regard (e.g., contributions of exist (Harvey, 1990; Midgley, 2003). Some of
science to the atrocities of modern warfare and these are directed at apparent self-contradictory
arguments within postmodernism. For example, process that would progress toward truth. Later,
the most frequently cited contradiction within post- critical modernists challenged this view and argued
modernism is that, while it disparages grand theo- that theories cannot be corrected by research but
ries (metanarratives), it proposes a grand theory only improved as explanatory systems. Finally,
that is supposed to supersede all previous theories. postmodernists called into question all of science
Or conversely, if one accepts the proposition that as a privileged form of knowing, suggesting that no
no universal claims about knowledge are true, then universal knowledge can be achieved. Table 2.1
one has to reject the postmodernist claim that all illustrates these four periods in the philosophy of
knowledge is socially constructed. Postmodernists science and their major arguments concerning the-
typically admit that there are ironic and self- ory, empiricism, and scientific knowledge. It repre-
contradictory elements of the postmodern argu- sents not only the evolution of ideas about
ment, but they dismiss criticisms of this aspect of research, but also a continuum of perspectives,
postmodernism as a misplaced concern for logic some of which are embraced, either implicitly or
and coherence. explicitly, by researchers today.
The most salient criticism of postmodernism, For instance, most quantitative research
however, is that while it has successfully pointed employs logical methods developed in the logical
out some of the limitations and failures of mod- positivist tradition. Many qualitative researchers
ernism, it has not offered any alternative (Midgley, embrace ideas represented in critical modernism
2003). In fact, extreme postmodernists argue that and postmodernism. Every study draws in some
any attempt at creating universal knowledge is way upon the kinds of ideas that have been devel-
essentially pointless. oped in the philosophy of science.
What, then, are the lessons that researchers can
take from the continuum of philosophical perspec-
Conclusion tives discussed in this chapter? First of all, few
today would embrace the classic idea that science
This chapter documented the history of thought in produces truth. However, with the exception of
the philosophy of science. It illustrated how extreme postmodernists, most would agree that
research was classically conceived as a means of science produces potentially useful knowledge.
attaining truth based on self-evident principles and Representing the view of critical modernists, Kuhn
proof. This perspective was replaced by the logical (1977) ultimately concluded that scientific efforts
positivist view of science as a self-correcting should be judged on their utility—that is, “the con-
Table 2.1 A Continuum of Ideas in the Philosophy of Science

Classicism Modernism Critical Modernism Postmodernism
The nature Theory is built on first Theory is a logical Theory is a product of Theory is a meta-
of theory principles that are system that creative imagina- narrative that
self-evident (i.e., explains and can tion that enables falsely claims
revealed by the predict events in scientists to appre- privileged status
world). the world. ciate the world in a over other possible
particular way. narratives.
The role of Theory can be Theory can be Theory can be Empirical testing
empiricism proved by disproved through improved by proves nothing;
deducing empirical tests of empirical testing. it only reinforces
empirically logically derived claims to power/
demonstrable hypotheses. legitimacy.
statements.
View of Scientific knowledge Scientific knowledge Scientific knowledge All knowledge,
scientific represents truth. is tentative but is one possible including scientific
knowledge can be made version of the knowledge, is
increasingly true world, which socially con-
over time. must be critically structed/relative.
judged for its
consequences.
crete technical results achievable by those who Regarding Theory

practice within each theory” (p. 339). Kuhn’s com-
ments suggest that scientific efforts in occupa- • Theories are human creations that seek to impart
tional therapy should be judged on their ability to meaning on the world.
help therapists effectively solve the kinds of prob- • First principles or underlying assumptions
lems their clients face. are unavoidable and untestable parts of any
Postmodernism is generally not useful as a theory.
philosophical premise for doing research. • Theories always represent one way of explaining
Nonetheless, postmodernism can be useful as a or making sense of things.
critical stance from which to judge scientific • While theories cannot be disproved, their ability
efforts. In particular, it is useful in calling attention to explain the natural world can be improved
to how science can be shaped by ideology, power, through research.
and interest. • The ultimate worth of any theory is its ability to
One of the most relevant examples for occupa- generate solutions to practical problems.
tional therapy are disability studies that use the • It is not possible to undertake research, no matter
postmodern social constructivist argument as a how open-ended and free of presuppositions,
basis for critiquing much existing research on dis- without some underlying theory and first princi-
ability. As Rioux (1997) points out, the various ples, even if they are not made explicit. Thus,
research efforts to classify, count, and study rela- whether a researcher is using only a handful of
tionships among variables associated with disabil- loosely connected assumptions and concepts or a
ity appear to be objective and scientific. However, complex theory, some conceptual context is nec-
this science is informed by an ideology about the essary to any research.
nature of disability that focuses on disability as an
individual deviation from norms. Importantly, the Regarding the Research Process
understanding of disability that has resulted from
this approach is at variance with how people with • Research is part of an inductive–deductive
disabilities experience their situation. Thus, the process in which theory is derived from and tied
dominant modern understanding of disability is a back to the world through empiricism.
social construction, not a fact. Scholars in disabil- • Logic is necessary to connect the concepts and
ity studies have called for the voices of disabled propositions that make up a theory with each
persons to be added to the scientific discourse other and to connect them with the things in the
about disability in order correct this prevailing world to which they refer.
misunderstanding of disability (Scotch, 2001). • All research is embedded in theory (whether or
A second important lesson to be taken from not it is made explicit). The theory is what makes
postmodernism is the need to contextualize knowl- sense of the phenomena examined, the scientific
edge in the circumstances of its production—that problems addressed, and the way those problems
is, to place any claims to knowledge in the context are solved.
within which the knowledge was generated. A • Research does not advance theory toward truth,
number of investigators, especially those involved but instead improves the way that any theory
in qualitative research, where the investigator’s makes sense of the world.
interpretation is a major analytic tool, do carefully
context their research efforts in their personal
and other relevant histories. By doing so, the inves-
Regarding Researchers
tigator gives the reader an additional perspective • Researchers always impart meaning on what they
from which to understand and judge the research observed by creating theories.
findings. • Investigators bring to bear all their characteristics
While the ideas about the nature of research in (including personal history, training, theoretical
the philosophy of science have changed dramati- understandings, and assumptions) on the re-
cally over the centuries, each era has offered cer- search process.
tain principles, ideas, and understandings that are • Researchers are part of a social community
useful to keep in mind regarding research. While that shares a perspective that makes sense
they might be formulated in a number of ways, this of what is studied as well as related norms
chapter concludes with the following key insights and rules that set out what should be studied and
and guidelines. how.
Regarding the Impact of Research Kuhn, T. (1962). The structure of scientific revolutions.
Chicago: University of Chicago Press.
• Research is not inherently value-free or benign. Kuhn, T. (1977). The essential tension. Chicago: University
• Research can be tied to particular ideologies and of Chicago Press.
Losee, J. (1993). A historical introduction to the philoso-
used to reinforce power structures and to disen- phy of science (3rd ed.). New York: Oxford University
franchise or oppress groups. Press.
• Research can be used for positive ends and by Lyotard, J. (1979). The postmodern condition: A report on
advancing understanding and prediction of cer- knowledge. Minneapolis: University of Minnesota Press.
tain phenomena, it can inform practical action. Midgley, G. (2003). Five sketches of post-modernism:
implications for systems thinking and operational
research. International Journal of Organizational
REFERENCES Transformation and Social Change. 1, 47–62.
Bronowski, J. (1978). The origins of knowledge and imagi- Nagel, E. (1949). The meaning of reduction in the natural
nation. New Haven: Yale University Press. sciences. In R. C. Stauffer (Ed.), Science & civilization
Cohen, I. B. (Ed.). (1958). Isaac Newton’s papers & letters (pp. 327–338). Madison: University of Wisconsin Press.
on natural philosophy. Cambridge, MA: Harvard Rioux, M. H. (1997). Disability: The place of judgment in
University Press. a world of fact. Journal of Intellectual Disability
Cohen I. B., & Smith, G. (Eds.). (2002). The Cambridge Research, 41, 102–111.
companion to Newton. Cambridge, UK: Cambridge Robson, J. M. (1996) Collected works of John Stuart Mill.
University Press. New York: Routledge.
Foucault, M. (1970). The order of things: An archeology of Scotch, R. K. (2001). From good will to civil rights:
the human sciences. New York: Pantheon. Transforming federal disability policy (2nd ed.).
Grunbaum, A. (1970). Can we ascertain the falsity of a sci- Philadelphia: Temple University Press.
entific hypothesis? In E. Nagel, S. Bromberger, & A. Scriven, M. (1970). Explanations, predictions and laws. In
Grunbaum (Eds.), Observations and theory in science H. Feigl & G. Maxwell (Eds.), Minnesota studies in the
Baltimore: John Hopkins University Press. philosophy of science. Minneapolis: University of
Harvey, D. (1990). The condition of postmodernity. Minnesota Press.
Cambridge, MA: Blackwell. Skorupski, J. (Ed.). (1998). The Cambridge companion to
Hesse, M. (1970). Is there an independent observation lan- John Stuart Mill. Cambridge, UK: Cambridge
guage? In R. G. Colodny (Ed.). The nature and func- University Press.
tion of scientific theories. Pittsburgh: University of Toulmin, S. (1961). Foresight and understanding: An
Pittsburgh Press. inquiry into the aims of science. Bloomington, IN:
Jenkins, J. (1992). Understanding Hume. Edinburgh: University of Indiana Press.
Edinburgh University Press. Watson, R. (2002). Cogito ergo sum: The life of Rene
Kant, I. (1996). Critique of pure reason (W. S. Pluhar, Decartes. Boston: Godine Press.
Trans.). Indianapolis: Hackett. Weedon, C. (1987). Feminist practice and poststructuralist
Kaplan, A. (1964). The conduct of inquiry: Methodology theory. Cambridge, MA: Blackwell.
for behavioral science. San Francisco: Chandler. White, L. (1938). Science is sciencing. Philosophy of
Klee, R. (1997). Introduction to the philosophy of science: Science, 5, 369–389.
Cutting nature at its seams. New York: Oxford Wittgenstein, L. (1953). Philosophical investigations.
University Press. Oxford: Basil Blackwell.
C H A P T E R 3
The Range of Research

Gary Kielhofner • Ellie Fossey
Research studies are almost as varied as they are were the first to develop sophisticated research
numerous. Even within a specific field such as methods; they were followed by the biological sci-
occupational therapy, there is considerable diver- ences. As a result, approaches to doing research
sity of investigations. For example, studies may dif- were first developed in the context of studying the
fer along such dimensions as: physical world (Johnson, 1975). Later, fields that
studied human behavior (e.g., psychology and
• The sample size, or number of study participants
sociology) emulated perspectives and methods that
(from one to hundreds or thousands),
were already flourishing in the physical and life
• What participants are asked to do (being observed
sciences.
versus undergoing complex interventions),
• How information is gathered (following partici-
pants in their ordinary activities and context ver- Assumptions About the
sus taking measurements in a laboratory setting), Phenomena Under Inquiry
and Fundamental to any inquiry are its assumptions
• How the data are analyzed (identifying underly- about the nature of the phenomena being studied
ing narrative themes versus computing statistical (Neuman, 1994). The quantitative tradition is
analyses). grounded in the assumption that there is one,
One way to appreciate this diversity of research objective reality that is stable over time and across
is to examine different ways it is classified. In this situations. So, for example, the laws of gravity are
chapter, we consider three ways that research is presumed to remain the same over the centuries
differentiated: by method, by design, and by pur- and to apply equally throughout the universe.
pose (Table 3.1). Since all the approaches to Similarly, in the biological domain, it is assumed
research that we discuss here are detailed in this that the laws of kinematics that govern how the
book, the aim of this chapter is to give the reader a body produces movement apply equally when one
broad appreciation of the range of research that is an adolescent and when one is older and that
will be discussed later. they are the same across people. Therefore, the
aim of quantitative methods is to discover the rules
or laws underlying the objective world as a basis
Research Methods: for scientific prediction and control (Guba &
Qualitative and Quantitative Lincoln, 1994).
One of the most common distinctions is that

between qualitative and quantitative research Table 3.1 Typical Ways of Categorizing
methods. Terminology suggests these methods dif-
Research
fer by the presence or absence of quantification.
However, it is important to note that these two Research methods Qualitative methods
broad categories of research are also distinguished Quantitative methods
by important philosophical differences (Crotty, Research designs Experimental and quasi-
1998). Hence, we describe the origins of qualita- experimental studies
tive or quantitative research and their differing Single subject designs
Field studies and natura-
assumptions, approaches to rigor, and research
listic observation
focus, as well as examining how they gather, ana- Surveys
lyze, and interpret data. Psychometric studies
Quantitative Methods Research purposes Basic

Applied
Quantitative research originated in the physical Transformative
sciences. Fields such as astronomy and physics
Chapter 3 The Range of Research 21
Research in occupational therapy began to develop to the dominant upper extremity of patients with
in earnest in the mid-20th century. At that time multiple sclerosis would improve their coordina-
occupational therapy practice was dominated by tion. The rationale was that the addition of weight
an approach that emulated medicine’s emphasis would mitigate patients’ intention tremors and thus
on scientific methods developed in the physical increase coordination. In this study of 39 patients
and life sciences, such as chemistry and biology in a physical rehabilitation program, the subjects
(Kielhofner, 2004). Not surprisingly, the research were tested with and without a weighted cuff using
that began to appear around this time was quantita- the MRM Test (used in this study as the measure
tive. The following two examples of research, of coordination). The results of the study failed
reported in the American Journal of Occupational to support the hypothesis that the addition of a
Therapy, are characteristic of the period. weight would improve coordination. In fact, the
Drussell (1959) reported a descriptive study opposite was observed; subject scored significantly
that asked whether the industrial work perform- lower when wearing the weighted cuffs. This
ance of adults with cerebral palsy was related to author concluded, then, that the addition of the
their manual dexterity, as measured by the cuff slowed the speed of movement, negatively
Minnesota Rate of Manipulation Test (MRM). The affecting coordination.
MRM is a standardized measure of manual dexter- The characteristics of these two studies, quan-
ity originally used for testing workers’ ability to tification of the variables under study through use
perform semiskilled factory operations. Work per- of standardized measures, use of experimental con-
formance was measured with a widely used indus- ditions in the second study, and statistical analyses
trial measure, the Service Descriptive Rating (descriptive in the first study; inferential in the sec-
Scale. In this study both tests were administered to ond study) are hallmarks of quantitative research.
32 adults with cerebral palsy who were enrolled in Since these studies were conducted, the use of
an adult vocational training program. The results more complex experimental designs, including
of the study indicated that the two measures were pre- and post- intervention testing, randomization
positively correlated. This finding was interpreted of study participants, and test development, has
as indicating that the MRM could be a valuable developed in occupational therapy. Nevertheless,
tool in assessing vocational potential for this the underlying logic of the research designs used
population. in these two studies is similar to that of contempo-
Cooke (1958) reported results of an experimen- rary quantitative research in occupational therapy.
tal study that investigated whether adding a weight
This assumption of a stable world governed by subjective error or bias. This concern for scientific
timeless laws is also applied to the study of human objectivity permeates quantitative research and
behavior. For example, early in the field of psycho- its methodological procedures. Thus, in quantita-
logical research, behav- tive research the investi-
iorism sought to discover gator ordinarily employs
the laws that governed …the aim of quantitative standardized and prede-
how positive or negative methods is to discover the termined designs and
consequences influenced procedures (Seltiz,
behavior. It was expected rules or laws underlying the Wrightsman, & Cook,
that these laws would objective world as a basis 1976). Because quantita-
allow the articulation of tive researchers aim to
causal models to predict for scientific prediction and discover invariant laws
behavior across people control. that apply across situa-
and across the course of tions, they are also con-
development. Thus this cerned with being able to
tradition of psychological research was predicated generalize from the sample studied in the particular
on a view of human behavior as observable, meas- investigation to a larger population that the samples
urable, and predictable (Rice & Ezzy, 1999). represent. Thus, the use of standard sampling pro-
cedures aims to ensure the generalizability of the
Rigor: An Emphasis on Objectivity findings.
Quantitative methods place importance on the sci-
A Focus on Theory Testing
entist maintaining an objective stance. The quanti-
tative researcher must take great care to see the Because of how they view the world, quantitative
outside world as it is by avoiding the influence of researchers tend to focus on:
• Creating theory that explains the laws governing researchers encountered foreign languages, per-
the phenomena under study, and spectives, and practices, they recognized that
• Testing that theory to verify and refine it through behavior reflected rules specific to the social and
research (Filstead, 1979). cultural context (Denzin, 1971; Edgerton &
Langess, 1974; Johnson, 1975; Pelto & Pelto,
Thus, translating abstract theory into specific
1978; Schutz, 1954). Qualitative methods devel-
answerable questions and concrete observable
oped in response to the recognition that the every-
hypotheses (statements of expected observations
day world that people experience is a valid focus
based on the theory) is key to quantitative research
of inquiry (Crotty, 1998; Neuman, 1994). Thus,
(Filstead, 1979).
qualitative research traditions generated a unique
set of assumptions about, and approaches to study-
Data Representation and Analysis: ing, the phenomena they investigated.
Quantification and Statistics Qualitative research is generally divided into
Quantification involves transforming observa- ethnographic, phenomenological, and narrative
tions into numerical data for manipulation using inquiry approaches, each of which represents a
statistics. The researcher assigns numbers in a somewhat different standpoint. Ethnography
systematic, objective way to capture some essen- emphasizes the societal and cultural context that
tial characteristic of the variables under study. shapes meaning and behavior. Phenomenology
Quantification can include enumeration based focuses on how people experience and make sense
on assignment to differential categories, determi- of their immediate worlds, and narrative inquiry
nation of degree (ranking or rating), or determina- seeks to understand how people construct storied
tion of amount (measurement). There are many accounts of their and others’ lives and of shared
different procedures and tools for analyzing data events (Rice & Ezzy, 1999).
statistically, dependent on the type of data and the
nature of the research question and design. Assumptions About the
However, in general, statistical analyses are Phenomena Under Study
used to:
Early qualitative researchers justified their
• Characterize or describe aggregated data (e.g., to approach by emphasizing the differences between
indicate frequency, variability, averages), and the phenomena they studied and the approach of
• To draw inferences about the data collected (e.g., the traditional physical scientists. For instance,
to determine whether the average value of a char- Schutz (1954) argues:
acteristic differs across two groups).
The world of nature, as explored by the
Findings are presented in the form of statistics natural scientist, does not “mean” any-
that characterize the sample and the population thing to the molecules, atoms and electrons
it represents, answer questions, and/or test hypo- therein. The observational field of the
theses. social scientist, however, namely the social
reality, has a specific meaning and rele-
Qualitative Methods vance structure for the human beings liv-
ing, acting, and thinking, therein. (p. 266)
Qualitative methods are typically used in research
that aims to describe and explain persons’ experi- Qualitative researchers asserted that human
ences, actions and interactions, and social con- behavior was guided by meaning, not objective
texts. Qualitative research is an umbrella term for laws (Blumer, 1969; Lofland, 1976; Pelto & Pelto,
a range of methodologies originating from the 1978). Moreover, qualitative researchers recognize
fields of anthropology, sociology, philosophy, and not one single human reality but, instead, multiple
psychology. Today, they are widely used in the realities reflected in the different meaning struc-
health sciences. Many researchers in occupational tures of different groups.
therapy have embraced these methodologies to Qualitative researchers have also stressed that
study occupation and practice issues, viewing individual and collective actions require those
them as congruent with the profession’s philo- involved to assess and interpret the ongoing and
sophical orientation (Hammell, 2002). typical situations that confront them (Blumer,
The perspectives and methods of qualitative 1969; Lincoln & Guba, 1985; Rice & Ezzy, 1999).
research originally grew out of the challenges of Thus, unlike the phenomena studied by the physi-
studying groups of people who were dramatically cal and life sciences that are governed by invariant
different from the investigator. Because qualitative and timeless laws, the phenomena studied in qual-
Qualitative research began to appear in occupa- the participants tended not to be future oriented;
tional therapy literature during the 1980s. At they did not expect things to be change, nor did
that time, there was a resurgence of interest in they make plans for achieving change in their
ideas about occupation, its meanings and signifi- lives. Hence, he argues among other things that
cance for health, on which occupational therapy the participants:
practice was founded (Kielhofner, 2004). This
…have ceased to become in the sense of the
led occupational therapists to seek relevant
dominant culture, and from their own point of
research designs for exploring the meanings
view, they are off the career time track. They
and contexts of people’s everyday lives, occupa-
are, in a sense, “frozen in time.” (Kielhofner,
tions, and experiences of illness, disability, or
1979, p. 163)
therapy, and to argue for the use of qualitative
designs in occupational therapy (Kielhofner, Another feature of how these study participants
1982a, 1982b; Krefting, 1989; Yerxa, 1991). experienced their lives uniquely was that unlike
Early examples of qualitative research published many other members of American culture, they
in occupational therapy most commonly used had a surplus of time and a deficiency of things to
ethnographic designs, originating in anthropologi- do that would fill up their time. As a result, they
cal fieldwork methods, of which the following is did not experience long periods waiting for events
an example. to occur with the impatience or frustration that
This study examined the daily life experiences characterized the investigators’ reactions. Rather,
of 69 adults with developmental delay who were waiting was something that helped to fill time.
discharged from state hospitals to residential facili- These and other findings pointed out that these
ties as part of the deinstitutionalization movement. adults approached the organization of their daily
In this study, the project team (anthropologists, activities and their lives in a radically different
sociologists and clinicians followed the study par- way from mainstream American culture
ticipants over a 3-year period, participating with (Kielhofner, 1981).
them in their daily life events in the five residential This study highlights the emphasis of ethno-
facilities where they lived. Researchers recorded graphic research on illuminating the social and
observational data in field notes, conducted ongo- cultural context of human action and its meaning.
ing open-ended interviews with the residents, and It also illustrates the use of this type of research
videotaped them. Analysis of the data form this in examining how changes in health policy and
field study resulted in several publications services may impact persons. Since this study
(Bercovici, 1983; Goode, 1983; Kielhofner, 1979, was conducted, qualitative research in occupa-
1981). Kielhofner (1979) reported how the partici- tional therapy has diversified, using phenomeno-
pants experienced and organized their behavior in logical, narrative, and, more recently, participatory
time. He described how the participants did not approaches. It has also expanded in focus to
progress through the usual life events that tend to explore occupational therapists’ clinical reasoning
demark maturation (e.g., graduating high school, and practice issues in many settings, as well as the
marriage, parenthood). Rather, their lives were everyday lives and occupations of clients of occu-
largely unchanged over time with the result that pational therapy services.
itative research are seen as dynamic, relational, participate with them in the activities under study,
and embedded in a particular context. in order to achieve an insider (or emic) under-
standing.
Rigor: An Emphasis on Unlike quantitative researchers who sought to
Understanding and Representing maintain an objective stance on the phenomena
they studied, qualitative researchers aim to
Subjective Reality
immerse themselves in the subjective reality of the
Since qualitative researchers seek to understand persons whom they study (DePoy & Gitlin, 1998;
the actions of people, they must know the everyday Pelto & Pelto, 1978; Rice & Ezzy, 1999).
meaning and contexts that inform and shape those Qualitative researchers also reflect on their own
actions (Strauss & Corbin, 1990). Qualitative personal reactions in the research setting to gain
researchers are concerned with accurately captur- better access to how study participants experience
ing research participants’ subjective meanings, their reality (Denzin, 1971). The greatest threat to
actions, and perceptions of their social contexts rigor in qualitative research is that researchers may
(Popay, Rogers, & Williams, 1998). Consequently, erroneously substitute their own meaning for the
qualitative researchers use methods to actively meanings of those they are studying, creating ficti-
engage their study participants in dialogue, and tious, and thus invalid, findings.
A Focus on Authenticity allow extensive quotation of participants’ own

and Groundedness words, detailed notes that describe events and
actions, written and visual documents, and record-
Qualitative research aims to “illuminate the sub- ings are typical data. These data provide a rich
jective meaning, actions and context of those being source for the qualitative investigator, but they also
researched” (Popay, Rogers, & Williams, 1998, p. pose a challenge to coherently and concisely pres-
345). Thus, central to the quality of qualitative ent findings.
research is: Qualitative findings are represented as textual
• “Whether participants’ perspectives have been descriptions or narratives. Consequently analysis
genuinely represented in the research (authentic- involves translating a wealth of detailed qualitative
ity), and information to a textual account. Importantly, this
• Whether the findings are coherent in the sense account or narrative must authentically describe
that they ‘fit’ the data and social context from the phenomena being studied. That is, it must pre-
which they were derived” (Fossey, Harvey, serve for the reader the same essence of what was
McDermott, & Davidson, 2002, p. 723). studied.
Qualitative data analysis requires the researcher
Qualitative researchers ordinarily begin their to explore the meanings, patterns, or connections
inquiry, like quantitative among data. This process
researchers, with guiding involves the researcher’s
theoretical concepts and …in qualitative research, own thought, reflection,
questions. However, they the resulting abstractions, and intuition. There are
formulate broad ques- many different qualitative
tions, rather than nar- or theory developed, are procedures and tools for
rowly defined questions grounded in the participants analyzing qualitative data.
or specific hypotheses. They share the common
As data are gathered and experience and social feature of progressively
inform these broad ques- context. exploring the data, and
tions, they are refined, comparing and contrasting
leading to more focused different parts of the data
sampling and information-gathering. Thus, quali- in order to evolve a more sophisticated under-
tative research is flexible, emergent, and respon- standing (Tesch, 1990).
sive to the study setting, data, and its analysis. The Often data gathering and data analysis occur
participants and their social context shape the iteratively, with each influencing the other. Since
kinds of information gathered, and the themes and the researcher seeks to generate findings that are
explanations that emerge in the study. clearly grounded in participants’ viewpoints, vari-
This means that, in qualitative research, the ous safeguards are built into the analytic process.
resulting abstractions, or theory developed, are For example, qualitative research requires ade-
grounded in the participants’ experiences and quate sampling of information sources (i.e., peo-
social contexts. Valid representation centers on the ple, places, events, types of data) so as to develop
transformation of the meanings, perspectives, and a full description of the phenomenon being studied
behaviors of those studied into theoretical abstrac- (Rice & Ezzy, 1999). In addition, qualitative re-
tions. These abstractions must authentically repre- searchers typically return to the participants to
sent how those studied experience and organize seek their feedback as to whether the findings gen-
their world (Rice & Ezzy, 1999). erated truly characterize their experiences.
Presentations of findings must enable the reader
Data Representation and Analysis: to appreciate the phenomena studied, and to gain
Textual Description and Theory insights into how they are experienced by the par-
ticipants. One way of accomplishing this is
Grounded in Data
through “thick description” (Geertz, 1973). Thick
Data collection in qualitative research focuses on description refers to a sufficiently detailed depic-
gaining understanding of the phenomena under tion, drawn from the raw data, of people’s experi-
study as they are experienced by the participants. ences, actions, and situations to convey the layers
This means that researchers strive to preserve the of personal and contextual meanings that inform
ways in which participants characterize their expe- them (Denzin, 1971). For this reason, qualitative
riences and actions. Audiotaped interviews that findings are generally presented with substantial
quotes, verbatim field notes, and other data that

help point to the essence of the phenomena that the
Research Designs: An
researcher is attempting to characterize. Overview of Common
Basic Designs
The Qualitative–Quantitative Distinction
Table 3.2 illustrates how quantitative and qualita- Research can also differ by its basic design.
tive methods differ across the dimensions dis- Design refers to the fundamental strategy or plan
cussed above. One way of thinking about these of how the research will be structured. Research
two methods is that quantitative research tends to designs each have their own inherent logic. While
emphasize the deductive logical phase, focusing the intention here is not to exhaustively list all
on the testing of theory, whereas the qualitative research designs, we will cover the most common
tradition tends to emphasize the inductive phase, designs found in occupational therapy investiga-
generating theory out of the careful observation of tions. They include:
the nature situation (DePoy & Gitlin, 1998). These • Experimental and quasi-experimental studies,
two modes of inquiry are also distinguished by dif- • Single subject studies,
ferent assumptions and approaches to achieving • Field studies and naturalistic observation,
scientific rigor (Crotty, 1998; Fossey, Harvey, • Survey studies, and
McDermott, & Davidson, 2002). • Psychometric studies.
Despite their differences, these research meth-
ods have long been integrated in the social sci- Experimental and
ences. More recently, they are used together in
health and human and services research (DePoy &
Quasi-experimental Studies
Gitlin, 1998). Increasingly, occupational therapy Experimentation grew out of research in the life
researchers understand and draw upon both meth- sciences and uses quantitative research methods.
ods. As noted earlier, quantitative methods have The basic characteristic of all experimental
their origins in the study of physical phenomena. research is that the investigator manipulates an
In occupational therapy, whenever the object of independent variable, the antecedent variable that
inquiry is the structure and function of the human is expected to produce an effect on a dependent
body, quantitative methods are the appropriate variable. These designs aim to provide evidence
research approach to use. Quantitative methods that the independent variable is the cause of
are also used to study such things as cognitive changes or differences in the dependent variable.
functioning, sense of mastery, life satisfaction, Experimental and quasi-experimental designs
adaptation, playfulness, and coping. When the are specific blueprints for how to conduct an
aim of research is to compare different groups on experiment (Campbell & Stanley, 1963). The fun-
these variables, study their relationships, or to damental aim of experimentation is to control, as
determine whether intervention changes them, much as possible, extraneous influences that might
then quantitative research is the method of lead to an incorrect conclusion about the influence
choice. of the independent variable on the dependent vari-
Qualitative research is better suited to the study able. In a true experiment, two or more groups of
of subjective experience, meaning, and the subjec- participants are subjected to different independent
tive and contextual aspects of human action and variables. A fundamental characteristic of these
interaction (Guba & Lincoln, 1994; Neuman, research designs is the inclusion of a control group
1994). Qualitative methods are especially appropri- as a basis for comparison with the experimental
ate for studies that seek to understand individuals’ group, which undergoes the condition of primary
and groups’ subjective experiences of impairment, interest (such as an occupational therapy interven-
occupation, and daily life, as well as to explore tion) in the study.
social and cultural factors that influence these A simple example of an experiment in occupa-
experiences. Consequently, there are many issues tional therapy is a study in which one group of per-
about which occupational therapy researchers are sons receives therapy and a second group does not.
interested to learn (e.g., how clients experience In such a study, the researcher measures some
therapy, the thinking process behind therapists’ characteristic (dependent variable), such as the
decisions about their clients, what it is like to expe- independent self-care performance. The aim of the
rience a particular disability) for which qualitative experiment would be to attribute any differences in
research is the method of choice. self-care independence (dependent variable)
Table 3.2 Key Differences Between Qualitative and Quantitative Research Methods
Characteristic Quantitative Research Tradition Qualitative Research Tradition
Origin Physical and life sciences Study of people different from the
investigator (anthropology, philosophy,
sociology)
Assumptions Objective reality contains stable Social reality is dynamic, contexted, and
preexisting patterns or order that governed by local meanings.
can be discovered
Aims To discover natural laws that enable To understand social life and describe how
prediction or control of events people construct social meaning
Approach to rigor Maintain objectivity Authentically represent the viewpoints of
those studied.
Data presentation Numbers (statistics) Textual, “thick” descriptions in language of
participants
Data analysis Describes variables and their Identifies meaning, patterns, and
relationships and tests hypotheses connections among data; describes
in order to test theory experience/social scene; produces
theory “grounded” in the data.
between the two groups to the independent vari- services to persons living in the same house was
able (receipt of occupational therapy services). likely to create other situations that would bias the
However, in this case, the dependent variable results. For example, if a person in one group
might be influenced by the initial level of func- shared information and resources he or she
tioning of participants. If one group were generally received from services with a roommate who was
better functioning than the other group at the not receiving those services, it would contaminate
beginning of the experiment, then this difference the findings. Consequently, for this study, a quasi-
in functioning could account for differences in experimental design was chosen. All residents in
self-care independence. We might find this differ- one setting receive the same services and are com-
ence between the two groups whether or not they pared with residents of another setting who receive
received occupational therapy services. Thus, in different services. This type of design opens the
experimental studies subjects are randomly experiment to alternative explanations for any dif-
assigned to the two different conditions to achieve ferences in independence or employment found
equivalent groups. other than the services received, such as group per-
Quasi-experimental designs follow the same sonality, types of people in each house, and house
logic as experimental designs; they typically staff. However, it was the most rigorous design
include a control group, or one or more compari- practicable in this context. Thus, despite their lim-
son groups, and experimental manipulation of the itations, quasi-experimental designs are valuable
independent variable of interest, but lack the when the constraints in health or human service
degree of rigor found in true experiments (Cook & environments mean random assignment is not
Campbell, 1979). For example, a study in which appropriate, ethical, or feasible (De Poy & Gitlin,
participants are not randomly assigned to groups 1998).
would be considered quasi-experimental.
Occupational therapy researchers sometimes
Single-Subject Studies
undertake less rigorous quasi-experimental
research because true experiments research can be Experimental designs rely on comparisons of aver-
difficult to undertake in real-life contexts. For ages in groups. Individual variation in response to
example, one of the authors is involved of an an intervention is not a focus of such studies. For
investigation of the effects of occupational therapy that reason, practitioners sometimes find large
services, compared to a less intensive standard group experiments to have limited relevance to
educational intervention, on independent living decision-making about what services or strategies
and employment (dependent variables). In this would be best for an individual client. Single-
study, services are delivered to residents in the subject designs follow the logic of experimenta-
facilities where they live. Random assignment was tion, but examine the impact of interventions on
not feasible because delivering different types of single subjects who serve as their own controls.
Examples of Experimental and Quasi-experimental Designs
A study examining the value of combined recommendations, and facilitated minor home
sensory integration and perceptual–motor treat- modifications. A monthly falls calendar was used
ment for children with developmental coordination to ascertain falls over a 12-month follow-up
disorder (DCD) provides an example of a quasi- period. The intervention was effective among those
experimental design. Davidson and Williams (n ⫽ 206) who reported one or more falls in the
(2000) assessed 37 children with DCD, using year prior to the study: at 12 months the risk of
the Movement ABC and the Beery-Buktenica falling and of recurrent falls and the likelihood of
Developmental Test of Visual–Motor Integration, being admitted to hospital were all significantly
before and after they had received combined sen- lower in the occupational therapy intervention
sory integration and perceptual–motor treatment group than in the control group. The authors con-
for 10 weeks. The children were also reassessed cluded that home visits by occupational therapists
with the same measures at 1-year follow-up. The can prevent falls among older people at increased
authors reported results that indicated these chil- risk of falling. Further, this effect may not be
dren made statistically significant but relatively caused by home modifications alone, given only
small gains in fine motor skills and visual–motor about 50% of recommended home modifications
integration. They concluded that 10 weeks o were in place at a 12-month follow-up visit.
therapy may be ineffective to achieve improve- Both these studies were designed to examine
ments in motor skills and motor integration the effectiveness of an occupational therapy inter-
for children with developmental coordination vention. They each involved taking measurements
disorder. for the outcome variables of interest pre- and post-
In contrast, an experimental design was used in intervention, both also included a 1-year follow-
a study to determine whether occupational thera- up. The principal differences between them lie in
pist home visits targeted at environmental hazards the inclusion of a control group, and random
reduce the risk of falls in community-based frail assignment of participants to this and the experi-
older adults in Sydney, Australia (Cumming et al., mental group. This allowed the researchers
1999). The sample comprised 530 older adults (Cumming et al., 1999) to determine whether those
(mean age 77 years), recruited from a hospital who received occupational therapy intervention
prior to discharge and randomly assigned to a con- benefited compared to those others who receive no
trol group or to occupational therapy intervention intervention (control group), and to have greater
(experimental group). The latter group received a confidence that detected benefits are attributable to
post-discharge home visit by an experienced the intervention. As can be seen, it was also a sub-
occupational therapist, who assessed the home stantially more ambitious undertaking to conduct
for environmental hazards, made home safety an experimental study of this type.
These designs permit a controlled experimental an experimental logic and thus are not to be con-
approach within which to observe single subjects fused with qualitative studies that may involve a
under ongoing treatment conditions in clinical set- single participant. Both types of studies are char-
tings (Portney, & Watkins, 2000). acterized by a sample of one, but their underlying
Single-subject designs generally involve two logic is different. Qualitative research that includes
major strategies that allow the subject to represent only one study participant follows the logic of
both a control and an experimental condition(s): qualitative methodology. In this instance, the judg-
ment is made that one participant is of sufficient
• Gathering baseline data over time during which
interest or adequately characterizes the phenom-
the experimental condition is absent and then
ena under question. Thus, additional participants
gathering data over time during which the exper-
are not necessary to inform the intent of the study.
imental condition is present, and
• Gathering data during alternating periods in
which the experimental condition is present or Field Studies and Naturalistic
withdrawn. Observation
Quantitative data are gathered on the dependent Field studies and naturalistic observation are forms
variable during the different experimental and con- of research that take place in the actual settings.
trol phases, and are analyzed both visually and Investigators study events as they happen and per-
using statistics designed for single-subject experi- sons in their natural context. Both qualitative and
mentation. qualitative research methods make use of this type
As noted above, single-subject designs follow of design.
Example of a Single-Subject Experimental Design
To investigate the effectiveness of a new series. Graphic and statistical interrupted time-
somatosensory retraining program to improve series analyses show clearly the changes between
tactile and proprioceptive discrimination in the phases in each case. Improvements were clini-
adults with sensory loss due to stroke, Carey, cally significant, with all participants reporting
Matyas, and Oke (1993) designed a study involv- improved performance comparable to that of
ing two series of four single-subject experiments. their other hand. These effects were maintained
In the first series involving four medically stable in follow-up tests.
adults with stroke, data were gathered using the Sensory training was a relatively new area of
Tactile Discrimination Test on 10 occasions investigation and the population of adults with sen-
within each phase: the baseline (prior to training) sory loss post-stroke potentially diverse. Given
and intervention (somatosensory retraining), these factors, the authors chose a single subject
and then a follow-up at an interval similar to experimental design to allow individual responses
the time taken to complete the baseline and to this intervention to be systematically examined.
intervention phases. The second series involved Often papers reporting single-subject experiments
another four stroke participants in a similar will report results on several subjects, as Carey et
procedure, except that both the Tactile al. (1993) did. These are considered replications
Discrimination Test and the Proprioceptive since the results from each single subject’s experi-
Discrimination Test were administered in this ment are analyzed separately.
In qualitative field studies, investigators seek to Survey Studies

gain an insider’s view of the phenomena under
study through intensive and extended immersion. Survey studies are nonexperimental designs under-
Investigators ordinarily collect data in multiple taken to investigate the characteristics of a defined
ways (e.g., gathering documents and artifacts, population (Depoy, & Gitlin, 1998; Portney, &
informal interviewing and observation) over some Watkins, 2000). They are often conducted with
extended period of time. Researchers also use their large samples. Survey studies are used to investi-
growing appreciation of the phenomena under gate such things as conditions or needs within a
study to continuously evolve the kinds of data col- defined community, or the extent of disease or dis-
lected, the methods for acquiring data, and who is ability in a population. Generally, survey research
sought out as a source of data. aims to randomly select the sample so that the
Naturalistic observation refers to quantitative findings can be generalized to the population from
research that takes place in natural settings. Such which the sample was chosen.
research aims to study the phenomena “undistur- The most common form of survey research
bed” by laboratory conditions or experimental pro- is implemented through the use of mailed ques-
cedures. For example, naturalistic observation can tionnaires. More recently, the Internet has been
be used to study specific behaviors as they occur in used as a method of questionnaire distribution.
classrooms, hospitals, or nursing homes. In natura- Also, surveys can be conducted through Web-
listic observation studies the observer seeks to based survey sites, to which selected subjects are
make “unbiased” observations of how events or directed. Questionnaires are usually designed to
behaviors actually take place. The investigator does gather quantitative data, although open-ended
not participate in the events under study, but rather questions may be asked to elicit qualitative
seeks to be as unobtrusive as possible. responses that are used to supplement quantitative
Naturalistic observations generally seek to findings.
determine the kinds of behaviors that occur, their Other survey research methods include tele-
frequency, the conditions under which they occur, phone and face-to-face interviews. When surveys
and so forth. For instance, the investigator may use follow the logic of quantitative methods research,
a time sampling approach in which observations the investigator uses a structured interview proto-
are recorded at specific time intervals, those inter- col so that all the participants respond to the same
vals being chosen randomly or according to some standardized questions. In qualitative surveys, the
logical schema. Data are ordinarily collected using investigator is more likely to use an interview
some kind of coding procedure, determined prior guide that allows participants to influence the
to beginning the research, which enables the direction of the interview, but also emphasizes
behavioral observations to be recorded in a manner strategies for probing, which seek to elicit the
that can be enumerated. respondents’ perspective.
Examples of Naturalistic Field Studies
Field studies are conducted in the natural context ties in the context of their everyday lives. The
and thus are particularly suited to examining how study participants were four Swedish women
some behavior or natural process occurs, such as with left hemiparesis and unilateral neglect
how mothers facilitate their children’s play at following stroke. To explore their experiences of
home, or how persons discover what they are able unilateral neglect, Tham and colleagues inter-
to do in daily life when their capacities have been viewed these women five to seven times over a
altered by trauma or disease. The following two 16-week period during their rehabilitation, includ-
studies exemplify qualitative field studies in occu- ing 4 weeks of an occupational therapy interven-
pational therapy. tion that utilized meaningful everyday occupations
The study by Pierce (2000) was designed to as means to improve awareness of disabilities
develop a theoretical description of the develop- (Tham, Ginsburg, Fisher, & Tegner, 2001). A
mental progressions in how 1- to 18-month-old phenomenological method was used to create a
infants play within the home, using intensive lon- description of how these women learned to live
gitudinal observation and grounded theory meth- with unilateral neglect. The findings describe a
ods of analysis (Glaser & Strauss, 1967). Pierce discovery process in which the participants moved
(2000) gathered extensive monthly data in the from experiencing their disabilities as new and
homes of 18 American mothers and infants over a unfamiliar, to beginning to discover and under-
period of 18 months, including written play obser- stand the consequences of unilateral neglect
vations, multiple interviews with the mothers, and during performance of everyday activities, and
many hours of videotapes of the infants at play then learning to handle the experience of neglect
with the usual objects and spaces in their home in everyday life situations. The authors suggest
environments. One reported aspect of this study increased understanding was a prerequisite to
focused on how mothers manage the spaces and being able to use strategies to compensate for dis-
objects in their homes to create and support their ability. They concluded therefore that through par-
infants’ play (Pierce, 2000; Pierce & Marshall, ticipating in meaningful occupational situations,
2004). These findings describe the everyday tasks the participants gradually discovered their disabili-
of mothers in selecting playthings for their infants ties and began to recapture the left half of their
(e.g., household implements, clothing, commercial worlds.
toys), as well as organizing their spatial arrange- The research designs used in these two field
ment to support and control the infants’ play in the studies illustrate different traditions of qualitative
home. Pierce (2000) concluded that this study research. Nevertheless, several of their characteris-
makes visible the behind-the-scenes work of moth- tics typify much qualitative research: extended
ers in supporting infant play. engagement with participants in a natural setting;
In a naturalistic study of a different kind, data collection that involves extensive interaction
Tham, Borell, and Gustavsson (2000) designed a between the researchers and study participants;
phenomenological study to investigate how adults and the presentation of findings as a textual
with unilateral neglect experienced their disabili- description.
Psychometric Studies before it is developed, as well as to examine its

clinical utility.
Psychometric studies are specifically designed to Validity refers to whether an instrument meas-
investigate the properties of clinical assessment ures what it is intended to measure. Because instru-
tools, or data collection instruments, intended for ments are designed to operationalize an underlying
use in research. Strictly speaking, this type of concept or construct, this aspect is often referred to
research is aimed at determining the validity and as construct validity. There are many methods of
reliability of these instruments. Following quanti- determining validity. These include, for instance,
tative logic, instruments with known validity and concurrent validity and predictive validity.
reliability provide objective measurement of the Concurrent validity follows the logic that an instru-
variables under study. Research with the primary ment designed to capture a variable should show
purpose of the development and evaluation of an association with another variable that is theoret-
clinical assessment tools may also include ically expected to be related to it. Predictive valid-
research questions that address their practical sig- ity asks whether a measure of some characteristic
nificance in clinical contexts. Psychometric (e.g., ability to perform activities of daily living) is
research is largely quantitative, although qualita- able to predict some future outcome, such as
tive methods are sometimes used to determine the whether a person is able to perform those activities
kinds of content that should go into an assessment with or without assistance. Thus studies designed
A Survey Study Example
Survey research designs have frequently been in were engaging in a modest amount of evidence-
used in occupational therapy to investigate profes- based practice: they occasionally accessed
sional attitudes, knowledge, and practices. In this research information through from various
study, Dysart and Tomlin (2002) aimed to estimate sources, with the majority (57%) implementing
the prevalence of evidence-based practice among one to five new research-based treatment plans in
occupational therapists in the United States. They the past year. However, less experienced respon-
designed a questionnaire to obtain demographics, dents more frequently believed that research
information about current evidence-based practice conclusions translated into treatment plans for
use, and factors related to its use. The question- individual clients than those with 15 or more
naire was mailed to 400 randomly selected clini- years of clinical experience. Time available at
cally practicing American Occupational Therapy work to access research information, high continu-
Association members, and 209 completed the ing education costs, weak research analysis skills,
questionnaire for a 58% response rate. The authors and placing higher value on clinical experience
analyzed the survey data using descriptive and than on research were reported barriers to research
inferential statistics to respectively describe the utilization. The authors suggested mitigating
distribution of opinions and associations between these barriers may increase evidence-based prac-
demographic and evidence-based practice vari- tice use among clinically practicing occupational
ables. Their findings suggested that therapists therapists.
to test expected associations, or predictions, pro- Basic Research

vide evidence on behalf of the validity of an
assessment tool or data collection instrument. Basic research includes investigations undertaken
Reliability refers to whether a given instrument for the purposes of generating evidence about
provides stable information across different cir- some phenomena or testing theory about some
cumstances. Thus studies designed to test reliabil- phenomena (Depoy & Gitlin, 1998; Portney &
ity might examine whether a given instrument is Watkins, 2000). Basic research is undertaken for
reliable, for instance, when multiple raters use the the sake of generating new knowledge without
instrument to gather data and when data are gath- direct concern for its applicability or practical sig-
ered on more than one occasion (referred to as nificance. The full range of research methods and
inter-rater and test–retest reliability, respectively). designs previously described may be used in basic
Psychometric studies in occupational therapy research, although traditionally, basic research
have used a range of research designs. Examples emphasized the importance of value-free science
include the development of observation-based per- that was disinterested in questions of application in
formance measures, such as the Assessment of order to avoid undue bias. It was thought that basic
Motor and Process Skills (Fisher, 1997; see also science would inform practice by identifying the
http://www.ampsintl.com/) and interview-based underlying laws that governed phenomena and
tools, such as the Canadian Occupational Perform- thus providing the logic for professions that
ance Measure (COPM) (Carswell et al., 2004) and applied that knowledge (Schon, 1983). This
the Occupational Performance History Interview- approach has been criticized by some scholars
II (Kielhofner et al., 2004), the development of who argue that basic science knowledge does not
which is summarized in the feature box titled translate readily into practice (Peloquin, 2002;
Instrument Development in Occupational Therapy. Thompson, 2001).
Throughout much of its history, occupational
therapy has relied on basic research conducted by
Research Purposes: Applied, other disciplines to inform practice. For instance,
research that identified the anatomy of the muscu-
Basic, and Transformative loskeletal system and the physiology of nerve con-
duction are two examples of information generated
Research can be differentiated according to its from basic research in the fields of anatomy and
underlying purpose: basic, applied, or transforma- physiology that form part of the foundation of
tive. Basic, applied, and transformative types of occupational therapy knowledge. More recently,
research each have different stances on how infor- some occupational therapists have argued for the
mation generated from research informs practice, development of a basic science concerned with the
so we also draw attention to these differences in study of occupation, referred to as occupational
relation to occupational therapy. science. Its proposed purpose was to generate
Instrument Development in Occupational Therapy
Psychometric research is an ongoing process that and depression. Following the logic that one’s
usually represents a series of studies leading to occupational life history represents strengths
progressive refinement of an assessment or meas- and weaknesses for future adaptation, Henry,
ure (Benson & Schell, 1997). The following is a Tohen, Coster, and Tickle-Degnen (1994) found
description of some of the early studies that exam- that the OPHI (administered with young adults
ined the reliability, validity, and clinical utility of during hospitalization for a first psychotic episode)
the Occupational Performance History Interview predicted psychosocial functioning and sympto-
II (OPHI-II) (Kielhofner et al., 2004), an occupa- matic recovery 6 months after discharge, thus pro-
tional therapy interview with an accompanying viding evidence of its predictive validity. Fossey’s
rating scale. (1996) and Neistadt’s (1995) studies provided evi-
The OPHI was originally studied with a sample dence that therapists perceived the OPHI to be a
of 154 occupational therapy clients from psychia- useful assessment in a range of contexts. Studies
try, physical disabilities, and gerontology practice such as these provide cumulative evidence about
in the United States and Canada (Kielhofner & an instrument’s reliability, validity, and clinical
Henry, 1988). This investigation found that the utility.
raw total score obtained from the rating scale Beyond the studies reported above, subsequent
had only marginally acceptable inter-rater reli- studies lead to further revision of the interview
ability and test–retest reliability. A second study itself to improve the kind of data collected and
(Kielhofner, Henry, Walens, & Rogers, 1991) creation of new rating scales to accompany it
sought to improve the reliability of the OPHI rat- (Kielhofner & Mallinson, 1995; Mallinson,
ing scale by developing more specific guidelines Kielhofner, & Mattingly, 1996; Mallinson,
for conducting the interview and completing the Mahaffey, & Kielhofner, 1998). The OPHI-II rat-
rating scale. The scale was found to be acceptably ing scales were eventually shown to be more reli-
stable in this study. able and valid than the previous scale (Kielhofner,
Lynch and Bridle (1993) examined the Mallinson, Forsyth & Lai, 2001), and several
concurrent validity of the OPHI, finding moderate recent studies have illustrated the kind of informa-
correlations between OPHI raw scores and meas- tion about people’s occupational lives gained using
ures of depression and pain in persons with trau- the OPHI-II interview and its clinical usefulness
matic spinal cord injury. This was an expected (Bravemann & Helfrich, 2001; Chaffey & Fossey,
relationship since other studies had shown that 2004; Goldstein, Kielhofner & Paul-Ward, 2004;
function in everyday life is associated with pain Gray & Fossey, 2003).
explanations of humans as occupational beings questions concerning occupational therapy prac-

(Yerxa et al., 1989). Like that of other basic tice for mothers or the children with identified
research, the role of occupational science was occupational dysfunction, ill health, or disability.
envisioned as describing, explaining, and predict- However, it provides information about the kinds
ing events as part of the search for knowledge and of questions that occupational therapists could ask
truth (Primeau, Clark, & Pierce, 1989). Mosey about facilitating play at home, both in practice
(1992a, 1993) questioned the legitimacy of a basic and future applied research with mothers of very
science in occupational therapy on the grounds young children with identified occupational dys-
that the allocation of human and other resources to function, ill health, or disability.
basic inquiry would detract from badly needed
applied inquiry. Its proponents, nevertheless, argue
that occupational science will likely influence how
Applied Research
occupational therapists perceive and approach Investigations that seek to solve some practical
their work (Zemke & Clark, 1996). problem, or to generate information specifically to
Basic research may vary in how closely it inform practice, are referred to as applied research
relates to practical problems and practice issues, (Depoy & Gitlin, 1998; Portney, & Watkins,
on which applied research focuses. To illustrate, 2000). Historically, applied research was most
Pierce’s (2000) study (see related feature box) often undertaken by professions to address their
offers insights into the occupation-related tasks practical concerns and held less status in academic
and routines of American mothers facilitating play circles where basic science was considered to be
with very young children at home. This study’s most rigorous because it was deemed to be value-
purpose is consistent with basic research: to gener- free and thereby not biased by the practical con-
ate a theory of infant play development within the cerns or less rigorous conditions under which
home. Consequently, the study did not address applied research was conducted. This viewpoint
has changed; most scientists now recognize the Examples of transformative research are rela-
importance of applied research. tively new in occupational therapy (Letts, 2003).
Many important research problems or questions The most common form of research with a trans-
generated in health and human service environ- formative purpose in health care and in occupa-
ments are applied in nature. Applied research gen- tional therapy is participatory research. Some
erally seeks to investigate the merits of practice common features of participatory types of research
strategies (e.g., assessments and interventions). In are that it:
occupational therapy, applied research address
• Is always grounded in a practical context,
issues such as:
• Involves people not simply as data sources but as
• Whether an assessment used in practice provides partners in the research process,
dependable and useful information to guide prac- • Emphasizes power sharing between the resear-
tice, chers and local stakeholders (e.g., therapists and
• How therapists reason in the context of prac- clients), and
tice, and • Is action-oriented, focusing on making change in
• What outcomes are achieved by providing partic- the practice setting and on examining the impact
ular services as part of therapy. of that change from the perspectives of those
who are most influenced by it.
Applied research is often viewed as particularly
important for achieving external credibility (i.e., Transformative research is newer than either
influencing those who make policy and economic basic or applied research. Transformative research
decisions that impact on the delivery of occupa- calls for embedding the research process in the
tional therapy services). Indeed, Mosey (1992b) practice setting and giving stakeholders (e.g.,
argued that this type of practitioners and clients)
research is critical to a voice in shaping the
occupational therapy Research can take on many research process. It aims
since it provides infor- to alter and empirically
mation about the value
different forms, each with examine services, while
of what the profession its own aims, approach to empowering the stake-
does. However, practi- holders and embedding
tioners have critiqued
generating knowledge, and change processes within
applied research for test- strategies for achieving rigor. the context to which they
ing practice strategies are relevant. In this way,
under ideal conditions it attempts to combine
that cannot be reproduced in practice (Dubouloz, research, education, and action or to link theory
Egan, Vallerand, & Von Zweck, 1999; Dysart & (knowing) and practice (doing) (Rice & Ezzy,
Tomlin, 2002). Applied research in occupational 1999).
therapy ranges from psychometric studies, to qual- On the face of it, such research has special rel-
itative investigations of the therapy process, to evance to practitioners and clients in fields such as
controlled experiments that compare different occupational therapy since it is much more directly
therapeutic approaches (see examples in feature driven by their agendas and aimed at having a pos-
boxes in this chapter). itive impact on their circumstances (Crist &
Kielhofner, 2005). Proponents argue that research
Transformative Research grounded in and directly helping to evaluate prac-
tice in natural contexts should be given high prior-
Transformative research refers to inquiry that is ity in the field.
designed specifically to bring about change in some
practical situation, or a specific context. Its empha-
sis is on transforming social realities so that peo- Conclusion
ple’s lives are improved. Transformative research
aims to foster self-reflection, mutual learning, par- This chapter provided an overview of the range
ticipation, and empowerment (Letts, 2003; Reason of research that one is likely to encounter in
1994; Wadsworth & Epstein, 1998). Hence, this occupational therapy. It examined three different
type of research has been used to enable groups of ways of differentiating research: method, design,
people who are in some way marginalized, depri- and purpose. The aim was to be illustrative rather
ved, or oppressed to bring about change in their than exhaustive in examining variation in research.
lives and communities (Rice & Ezzy, 1999). It should be obvious from the discussion that
research can take on many different forms, each A study of the effectiveness of a combined sensory
with its own aims, approach to generating knowl- integration and perceptual-motor intervention. British
Journal of Occupational Therapy, 63, 495–499.
edge, and strategies for achieving rigor Denzin, N. (1971). The logic of naturalistic inquiry. Social
In this discussion, we also mentioned some of Forces, 50, 166–182.
the debates about the relative merits of differing DePoy, E., & Gitlin, L. N. (1998). Introduction to
types of research. Such debates are ongoing, and research: Understanding and applying multiple
strategies (2nd ed.). St. Louis: C. V. Mosby.
rightfully so. It is important to also recognize that,
Drussell, R. D. (1959). Relationship of Minnesota rate of
while researchers of particular traditions have pre- manipulation test with the industrial work performance
viously been more divided in their allegiance to of the adult cerebral palsied. American Journal of
and use of one or another approach to research, it Occupational Therapy, 13, 93–105.
is not uncommon for contemporary research to use Dubouloz, C., Egan, M., Vallerand, J., & VonZweck,
C. (1999). Occupational therapists’ perceptions of
multiple methods and to incorporate more than one evidence based practice. American Journal of
purpose in a single study. Occupational Therapy, 53, 445–453.
Dysart, A. M., & Tomlin, G. S. (2002). Factors related to
evidence-based practice among US occupational ther-
REFERENCES apy clinicians. American Journal of Occupational
Benson, J., & Schell, B. A. (1997). Measurement theory: Therapy, 56, 275–284.
Application to occupational and physical therapy. Edgerton, R., & Langess, L. (1974). Methods and styles in
In J. Van Deusen, & D. Brunt (Eds.), Assessment in the study of culture. San Francisco: Chandler and
occupational therapy and physical therapy (pp. 3–24). Sharp.
Philadelphia: W. B. Saunders. Filstead, W. J. (1979). Qualitative methods: A needed per-
Bercovici, S. (1983). Barriers to normalization. Baltimore: spective in evaluation research. In T. Cook, D.
University Park Press. Campbell, & C. Reichart (Eds.), Qualitative and quan-
Blumer, H. (1969). Symbolic interactionism. Englewood titative methods in evaluation research (pp. 33–48).
Cliffs, NJ: Prentice-Hall. Beverly Hills: SAGE Publications.
Braveman, B., & Helfrich, C. A. (2001). Occupational Fisher, A. G. (1997). Multifaceted measurement of daily
identity: Exploring the narratives of three men living life task performance: Conceptualizing a test of instru-
with AIDS. Journal of Occupational Science, 8, 25–31. mental ADL and validating the addition of personal
Campbell, D. T., & Stanley, J. C. (1963). Experimental and ADL tasks. Physical Medicine and Rehabilitation:
quasi-experimental designs for research. Chicago: State of the Art Reviews, 11, 289–303.
McNally & Co. Fossey, E. (1996). Using the Occupational Performance
Carey, L. M., Matyas, T. A., & Oke, L. E. (1993). Sensory History Interview (OPHI): Therapists’ reflections.
loss in stroke patients: effective training of tactile and British Journal of Occupational Therapy, 59(5),
proprioceptive discrimination. Archives of Physical 223–228.
Medicine and Rehabilitation, 74, 602–611. Fossey, E., Harvey, C. A., McDermott, F., & Davidson,
Carswell, A., McColl, M. A., Baptiste, S., Law, L. (2002). Understanding and evaluating qualitative
M., Polatajko, H., & Pollock, N. (2004). The Canadian research. Australian & New Zealand Journal of
Occupational Performance Measure: A research and Psychiatry, 36, 717–732.
clinical literature review. Canadian Journal of Geertz, C. (1973). Thick description: Toward an interpre-
Occupational Therapy, 71(4), 210–222. tive theory of cultures. In C. Geetrz (Ed.), The interpre-
Chaffey, L., & Fossey, E. (2004). Caring and daily life: tation of culture: Selected essays (pp. 3–30). New York:
Occupational experiences of women living with sons Basic Books.
diagnosed with schizophrenia. Australian Occupational Glaser, B., & Strauss, A. (1967). The discovery of
Therapy Journal, 51(4), 199–207. grounded theory. New York: Aldine.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimen- Goldstein, K., Kielhofner, G., & Paul-Ward, A. (2004).
tation: Design and analysis issues for field settings. Occupational narratives and the therapeutic process.
Boston: Houghton Mifflin. Australian Occupational Therapy Journal, 51,
Cooke, D. M. C. (1958). The effect of resistance on multi- 119–124.
ple sclerosis patients with intention tremor. American Goode, D. (1983). Who is Bobby? Idealogy and method in
Journal of Occupational Therapy, 12(2), 89–92. the discovery of a Down’s syndrome person’s compe-
Crist, P., & Kielhofner, G. (2005). The scholarship of prac- tence. In G. Kielhofner (Ed.), Health through occupa-
tice: Academic and practice collaborations for promot- tion: Theory & practice in occupational therapy (pp.
ing occupational therapy. Binghamton, NY: Hayworth 237–255). Philadelphia: F. A. Davis.
Press. Gray, M., & Fossey, E. (2003). The illness experiences and
Crotty, M. (1998). The foundations of social research: occupations of people with chronic fatigue syndrome.
Meaning and perspective in the research process. Australian Occupational Therapy Journal, 50,
Crows Nest, Australia: Allen & Unwin. 127–136.
Cumming, R. G., Thomas, M., Szonyi, G., Salkeld, G., Guba, E. G., & Lincoln, Y. S. (1994). Competing paradigms
O’Neill, E., Westbury, C., et al. (1999). Home visits in qualitative research. In N. K. Denzin & Lincoln, Y. S.
by an occupational therapist for assessment and modifi- (Eds.), Handbook of qualitative research (pp. 105–117).
cation of environmental hazards: A randomized trial of Thousand Oaks, CA: SAGE Publications.
falls prevention. Journal of the American Geriatrics Hammell, K. W. (2002). Informing client-centered practice
Society, 47, 1397–1402. through qualitative inquiry: Evaluating the quality of
Davidson, T., & Williams, B. (2000). Occupational therapy qualitative research. British Journal of Occupational
for children with developmental coordination disorder: Therapy, 65, 175–184.
Henry, A., Tohen, M., Coster, W., & Tickle-Degnen, L. American Journal of Occupational Therapy, 50,
(1994). Predicting psychosocial functioning and symp- 338–346.
tomatic recovery of adolescents and young adults fol- Mallinson, T., Mahaffey, L., & Kielhofner, G. (1998). The
lowing a first psychotic episode. Paper presented at the Occupational Performance History Interview: Evidence
Joint Annual Conference of the American Occupational for three underlying constructs of occupational adapta-
Therapy Association and the Canadian Association of tion (rev.). Canadian Journal of Occupational Therapy,
Occupational Therapists, Boston. 65, 219–228.
Johnson, J. (1975). Doing field-research. New York: Free Mosey, A. C. (1992a). Applied scientific inquiry in the
Press. health professions: An epistemological orientation.
Kielhofner, G. (1979). The temporal dimension in the lives Rockville, MD: American Occupational Therapy
of retarded adults: A problem of interaction and inter- Association.
vention. American Journal of Occupational Therapy, Mosey, A. C. (1992b). The issue is—Partition of occupa-
33, 161–168. tional science and occupational therapy. American
Kielhofner, G. (1981). An ethnographic study of deinstitu- Journal of Occupational Therapy, 46, 851–853.
tionalized adults: Their community settings and daily Mosey, A. C. (1993). The issue is—Partition of occupa-
life experiences. Occupational Therapy Journal of tional science and occupational therapy: Sorting out
Research, 1, 125–142. some issues. American Journal of Occupational
Kielhofner, G. (1982a). Qualitative research: Part One— Therapy, 47, 751–754.
Paradigmatic grounds and issues of reliability and Neistadt, M. (1995). Methods of assessing client’s priori-
validity. Occupational Therapy Journal of Research, ties: A survey of adult physical dysfunction settings.
2(2), 67–79. American Journal of Occupational Therapy, 49,
Kielhofner, G. (1982b). Qualitative research: Part Two— 420–436.
Methodological approaches and relevance to occupa- Neuman, W. L. (1994). Social research methods: Qualita-
tional therapy. Occupational Therapy Journal of tive and quantitative approaches. Needham Heights,
Research, 2(2), 67–79. MA: Allyn and Bacon.
Kielhofner, G. (2004). Conceptual foundations of occupa- Peloquin, S. M. (2002). Confluence: Moving forward with
tional therapy (3rd ed.). Philadelphia: F. A. Davis. affective strength. American Journal of Occupational
Kielhofner, G. & Henry, A. (1988). Development and Therapy, 56, 69–77.
investigation of the Occupational Performance History Pelto, P., & Pelto, G. (1978). Anthropological research:
Interview. American Journal of Occupational Therapy, The structure of inquiry. New York: Cambridge
42, 489–498. University Press.
Kielhofner, G., Henry, A., Walens, D., & Rogers, S. Pierce, D. (2000). Maternal management of the home as a
(1991). A generalizability study of the Occupational developmental play space for infants and toddlers.
Performance History Interview. Occupational Therapy American Journal of Occupational Therapy, 54,
Journal of Research, 11, 292–306. 290–299.
Kielhofner, G., & Mallinson, T. (1995). Gathering narrative Pierce, D., & Marshall, A. (2004). Maternal management
data through interviews: Empirical observations and of the home space and time to facilitate infant/toddler
suggested guidelines. Scandinavian Journal of play and development. In S. Esdaile & J. Olson (Eds.),
Occupational Therapy, 2, 63–68. Mothering occupations: Challenge, agency and partici-
Kielhofner, G., Mallinson, T., Crawford, C., Nowak, M., pation (pp. 73–94). Philadelphia: F. A. Davis.
Rigby, M., Henry, A., & Walens, D. (2004). Occupa- Popay, J., Rogers, A., & Williams, G. (1998). Rationale
tional Performance History Interview II (OPHI-II) and standards for the systematic review of qualitative
(Version 2.1). Model of Human Occupation literature in health services research. Qualitative Health
Clearinghouse, Department of Occupational Therapy, Research, 81, 341–351.
College of Applied Health Sciences, University of Portney, L. G., & Watkins, M. P. (2000). Foundations of
Illinois at Chicago, Chicago, IL. clinical research: Applications to practice (2nd ed.).
Kielhofner, G., Mallinson, T., Forsyth, K., & Lai, J. S. Upper Saddle River, NJ: Prentice-Hall.
(2001). Psychometric properties of the second version Primeau, L. A., Clark, F., & Pierce, D. (1989).
of the Occupational Performance History Interview. Occupational therapy alone has looked upon occupa-
American Journal of Occupational Therapy, 55, tion: Future applications of occupational science to
260–267. pediatric occupational therapy. Occupational Therapy
Krefting, L. (1989). Disability ethnography: A method- in Health Care, 6, 19–32.
ological approach for occupational therapy research. Reason, P. (1994). Three approaches to participatory
Canadian Journal of Occupational Therapy, inquiry. In N. K. Denzin & Lin Y. S. Lincoln. (Eds.),
56, 61–66. Handbook of qualitative research (pp. 324–339).
Letts, L. (2003). Occupational therapy and participatory Thousand Oaks, CA: SAGE Publications.
research: A partnership worth pursuing. American Rice, P. L. & Ezzy, D. (1999). Qualitative research meth-
Journal of Occupational Therapy, 57(1), 77–87. ods, a health focus. Melbourne: Oxford University
Lincoln, Y. S., Guba, E. G. (1985). Naturalistic inquiry. Press.
Thousand Oaks, CA: SAGE Publications. Schutz, A. (1954). Concept and theory formation in the
Lofland, J. (1976). Doing social life. New York: John social sciences. Journal of Philosophy, 51, 266–267.
Wiley & Sons. Schon, D. (1983). The reflective practitioner. New York:
Lynch, K., & Bridle, M. (1993). Construct validity of the Basic Books.
Occupational Performance History Interview. Seltiz, C., Wrightsman, L., & Cook, S. (1976). Research
Occupational Therapy Journal of Research, 13, methods in social relations. New York: Holt, Rinehart
231–240. & Winston.
Mallinson, T., Kielhofner, G., & Mattingly, C. (1996). Strauss, A., & Corbin, J. (1990). Basics of qualitative
Metaphor and meaning in a clinical interview. research. Thousand Oaks, CA: SAGE Publications.
Tesch, R. (1990). Qualitative research: analysis types and services. Systemic Practice and Action Research, 11,
software tools. New York: Palmer Press. 353–379.
Tham, K., Borell, L., & Gustavsson, A. (2000). The dis- Yerxa, E. J. (1991). National speaking: Seeking a relevant,
covery of disability: a phenomenological study of uni- ethical, and realistic way of knowledge for occupa-
lateral neglect. American Journal of Occupational tional therapy. American Journal of Occupational
Therapy. 54, 398–406. Therapy, 45, 199–204.
Tham, K., Ginsburg, E., Fisher, A. G., & Tegner, R. (2001). Yerxa, E. J., Clark, F., Frank, G., Jackson, J., Parham, D.,
Training to improve awareness of disabilities in clients Pierce, D., et al. (1989). An introduction to occupa-
with unilateral neglect. American Journal of Occupa- tional science: A foundation for occupational therapy in
tional Therapy, 55, 46–54. the 21st century. Occupational Therapy in Health Care,
Thompson, N. (2001). Theory and practice in human serv- 6, 1–17.
ices. Maidenhead: Open University Press. Zemke, R., & Clark, F (1996). Occupational Science: The
Wadsworth, Y., & Epstein, M. (1998). Building in dialogue evolving discipline. Philadelphia: F. A. Davis.
between consumers and staff in acute mental health
C H A P T E R 4
Characteristics of Sound Inquiry

and the Research Process
Gary Kielhofner
Despite their varying methods, designs, and pur- over factors that could bias or confound the
poses, all research studies share some common fea- results. In a classic text, Campbell and Stanley
tures and procedures. The aim of this chapter is (1963) outlined the threats to validity in experi-
to overview those elements that characterize all mental research and detailed a series of experi-
research. First, accepted characteristics of good mental and quasi-experimental designs that to
research are identified and examined. Second, the various extents controlled for these threats.
activities that ordinarily make up the research Today, when investigators use one of these exper-
process from the time it is planned until its com- imental or quasi-experimental designs, members
pletion are identified. of the research community readily recognize the
logic of their design and the extent of rigor it pro-
vided in the experiment. The confidence that is
Characteristics of Research placed in the findings corresponds to the rigor of
the design used.
The following characteristics are hallmarks of all • In survey research, investigators are concerned
research (Crotty, 1998; DePoy & Gitlin, 1998; that the sample accurately represents the popula-
Polgar & Thomas, 2000; Polit & Hungler, 1999; tion of interest (Rea & Parker, 1997). To ensure
Portney & Watkins, 2000; Stein & Cutler, 1996): representativeness, investigators must identify
the population and sample from it so as to ensure
• Rigor,
that those asked to participate in the study char-
• A scientific/scholarly attitude of skepticism and
acterize the total population. Then, the investiga-
empiricism,
tor must engage in a series of steps to ensure that
• Logic, and
as many of those asked to participate actually do.
• Communality.
Finally, the investigator must ask whether there is
Each of these is interrelated with the others. any evidence that those who respond to the sur-
Moreover, quality research exhibits all of these vey are systematically different from those who
characteristics. did not respond.
• In qualitative research investigators follow pro-
Rigor in Research cedures to ensure that they have penetrated
and comprehended how the people they are
Research is distinguished from ordinary searching
studying think about, choose, and experience
for knowledge by its degree of rigor. The concept
their actions (Rice & Ezzy, 1999; Strauss &
of rigor means that investigators carefully follow
Corbin, 1990). Since these procedures cannot be
rules, procedures, and techniques that have been
pre-standardized as in experimental or survey
developed and agreed upon by the scientific com-
research, investigators maintain a record of the
munity as providing confidence in the information
natural history of the study that documents how
generated by research (Neuman, 1994; Thompson,
insights and new questions arose from the data,
2001).
how decisions were made to gather new informa-
These rules, procedures, and techniques that
tion, how the participants in the study were
make up rigorous research are quite varied and
selected, how the data were coded, and what
specific to the research questions, the methods and
procedures were used to extract themes and
design of the research, and the topic or phenomena
meaning from the data. When researchers have
under investigation. The following are three exam-
formulated their findings, they return to those
ples of how investigators achieve rigor in research:
they have studied and share these findings to ver-
• In experimental research, investigators use stan- ify that they have authentically captured their
dardized research designs that provide control experience.
Chapter 4 Characteristics of Sound Inquiry and the Research Process 37
These are but a sampling of the many approa- The Scientific/Scholarly Attitude:
ches and procedures that investigators use to ensure Skepticism and Empiricism
rigor in research. In each of these instances, the
rules and procedures that the investigators follow All research is characterized by a scientific or
have been worked out over time by communities scholarly attitude that incorporates two elements:
of investigators who sought to improve the sound- • Skepticism and
ness of their research. These rules and procedures • Empiricism (Polit & Hungler, 1999; Portney &
have been shared, scrutinized, discussed, and Watkins, 2000).
debated in the scientific community. Eventually,
through consensus, they have become prevailing Skepticism is the idea that any assertion of
standards by which investigators do research and knowledge should be open to doubt, further analy-
by which completed research is judged (Crotty, sis, and criticism. Skepticism is important because
1998). it prevents prematurely accepting something as
As the previous examples illustrate, rigor accurate. Continual questioning of proposed
requires investigators to be thoroughly aware of knowledge allows the scientific community to
the accepted rules and procedures for the type of ensure that claims to knowledge are not taken as
research they plan to undertake. Investigators must accurate unless they survive constant scrutiny over
also be cognizant of unique approaches that are time (Thompson, 2001).
typically used by other researchers who conduct The scientific or scholarly attitude also
research on the specific topic of the investigation. demands proof rather than opinion. Empiricism
This second type of information is often critical for means that scientific knowledge emerges from and
knowing how to effectively apply rules and proce- is tested by observation and experience (Portney,
dures in a given context or with the uniquely chal- & Watkins, 2000). Thus, all research involves gen-
lenging characteristics of the study topic and/or erating data. Data are information about the world
subjects. While the formal rules and procedures for gathered through observation, listening, asking,
rigorous research can be found in methodological and other forms of acquiring or extracting infor-
texts, the specific processes that are used in a given mation from a situation, event, or person.
area of investigation are often shared through var- Data are valued over opinion because the latter
ious forms of communication including published is prone to error and personal bias. However, data
reports of research, scientific presentations, and are gathered systematically so that the influence of
direct communication among investigators in a error and bias are minimized (Benson & Schell,
particular area. 1997). Moreover, it is common that more than a
With this kind of procedural knowledge as single opinion will be held about any topic.
background, investigators must form sound judg- Consequently, the most systematic way of decid-
ments about what rules, procedures, and techniques ing between or among differing perspectives is by
(referred to as research design and methods) will be asking how well they bear up under scrutiny.
used in their studies to generate knowledge in Empiricism, then, includes the notion that scien-
which the scientific community can have the most tists can refine what they know by consistently
confidence (Seltiz, Wrightsman, & Cook, 1976). checking it against the world.
When implementing their research, investigators
are further responsible to ensure adherence to these Logical Reasoning
chosen methods and, in some stances, to make
additional informed decisions about methods as the Another cornerstone of research is logic.
research unfolds (Lincoln & Guba, 1985). Importantly, logic is used to systematically link
As can be seen, research is meticulous, knowledge to what the knowledge is supposed to
detailed, and reflective. Investigators strive to explain. This occurs through the process of induc-
achieve the highest degree of rigor possible in a tive and deductive reasoning (DePoy & Gitlin,
given study so as to optimize the confidence that 1998). As pointed out in Chapter 2, the logic of
can be placed in the information generated. inductive and deductive reasoning has been a topic
Finally, researchers are obligated to honestly let of constant dialogue, debate, and refinement in the
others know what their procedures were, any prob- scientific community.
lems that were encountered, and what limitations Inductive reasoning, or induction, involves
of rigor are inherent in the study. With this infor- making generalizations from specific observations.
mation, research consumers know how much con- For instance, suppose over the course of a week, an
fidence to place in the findings. occupational therapy investigator observes on sev-
eral occasions clients who suddenly become aware general propositions or theory by induction from
of another client or a therapist observing them. The data, followed by deduction of specific statements
researcher also notices that on these occasions, that can be verified or dismissed by data, is the
clients appear to increase their efforts. From these most basic process by which scientific knowledge
observations, the investigator might arrive at the is advanced.
general assertion that clients feel socially obligated Different research approaches emphasize either
to make the most of therapy and that there sense of the inductive or the deductive phase. For example,
obligation is strengthened when clients feel in experimental research, hypotheses deduced from
socially visible. Creating such a generalization is theories are tested. This deductive–empirical
an example of the process of induction. approach is typical of quantitative research. On the
As the example illustrates, induction involves other hand, qualitative research tends to emphasize
synthesizing information to generate a theory or an the inductive phase. Rich data gathered from par-
explanation about observed patterns. Induction ticipation, observation, and interviews with people
makes sense of observations by identifying them are used to generate new insights, concepts, or the-
as belonging to a larger class of phenomena that ories. While different research traditions may
exhibit an underlying pattern or meaning. emphasize one or another aspect of the data–induc-
Such a generalization is plausible, but, of tion–generalization–deduction–data cycle, all
course, it is not yet tested or verified. To test it, an research ultimately makes use of both elements.
investigator would have to use deductive reason-
ing—that is, to logically derive from this general-
Communality
ization statements that reference observable
phenomena. The following are some examples: Research occurs within a community of scientists
who consider how research in a given area should
• If asked whether they feel clients in therapy
be conducted, who scrutinize individual studies,
should put forth all the effort they can, clients
and who collectively arrive at judgments about
will respond affirmatively.
what conclusions should be drawn about a body of
• If observed doing therapy alone and in the pres-
research findings (Crotty, 1998; DePoy & Gitlin,
ence of other clients, clients would demonstrate
1998; Polgar & Thomas, 2000; Polit & Hungler,
more effort when in the presence of others.
1999; Portney & Watkins, 2000; Stein & Cutler,
• While engaged in a therapy session, clients’
1996). Every study is submitted to a public process
efforts would increase or decrease according to
in which both the knowledge acquired and the
whether the therapist was paying attention to them
means of acquiring that knowledge are laid bare
or not.
for others to scrutinize, criticize, and replicate.
Each of these statements could be used as a test Research is most typically made public through
of the veracity of the larger generalization created scientific presentations or posters at conferences
through induction. This is because these observa- or meetings and through publication in scientific
tional statements were logically derived. That is, it journals. A review process (typically done by
was deduced that, if persons feel social pressure to anonymous peers) ensures that the study to be pre-
put forth effort in therapy, and if being observed sented or published meets a basic threshold of
increases social pressure, then each of these state- rigor.
ments should be true. If they are shown not to be Once a study has been presented and/or pub-
true, then the investiga- lished, others in the sci-
tor would have to aban- entific community have
don or revise the general Every study is submitted to the opportunity to scru-
statement from which a public process in which tinize and criticize it.
they were derived. Criticism of existing
Induction and deduc- both the knowledge acquired studies is also a public
tion allow the investiga- and the means of acquiring process that occurs in
tor to go back and forth presentations or publica-
between explanations
that knowledge are laid tions. In fact, a very typi-
and observations. As bare for others to scrutinize, cal prelude to presenting
shown in Figure 4.1, criticize, and replicate. the findings of any study
observations are the is to point out both the
basis on which scientists findings and limitations
induce generalizations and test statements deduced of previous investigations, arguing how the current
from those generalizations. The cycle of creating study both builds and improves upon them.
Reviewing the literature

• Searching, obtaining, and analyzing the literature
to identify what is known, gaps in knowledge, and
how scientists go about generating knowledge in a
given area or on a given topic.
Identifying the questions

Disseminating findings as
to be answered that also
scientific presentations,
guide the research.
posters, articles, and in
consumer-friendly formats.
Selecting methods to answer the questions:

Background • Design-how the research will be
Interpreting results to Considerations: structured.
generate findings. • Rigor • Selecting the sample-determining how
• Logistics many people, who will be in the study,
• Available how they will be recruited, and what
Organizing and Resources they will do.
analyzing data. • Determining data collection and
analysis.
Implementing any procedures

and collecting data.
Obtaining ethical approval to ensure that

any subject risk is warranted, and to en- Writing the research plan that includes
sure that subjects make an informed and the literature review, research methods,
free choice and are protected from harm. a management plan, and a plan for
resources and budget.
Figure 4.1 The research process.
Finally, it is common practice for scientists researchers not only learn from each other the
to replicate a published study (often improving findings generated by research, they also learn
on some aspect of it or doing it under different from others’ mistakes and inventions. Innovations
conditions) to determine whether they achieve in research methods generated by one investigator
the same results. For this reason, when a study are routinely used by other investigators to
is offered to the scientific community, it is impor- improve their own research. In this way, the scien-
tant that it is explained sufficiently so that tific community tends to advance together, with
others can understand exactly what was done and each investigator learning from the experiences of
how the conclusions of the study were generated others.
(verifiability) and so that others can repeat the
study to see if the same results are obtained
(replicability).
Researchers build upon and place new knowl-
Research as a Set of
edge in the context of existing knowledge gener- Interrelated Activities
ated by the scientific community. All research is
informed by what has gone before. This is why the As illustrated in Figure 4.1, all research involves
report of a study always begins with a review of the following key activities (DePoy & Gitlin,
existing literature. The literature review serves to 1998; Polgar, & Thomas, 2000; Polit & Hungler,
situate the study in the context of what the scien- 1999; Portney & Watkins, 2000; Stein & Cutler,
tific community already knows. Importantly, 1996):
A B
Figure 4.2 (A, B) Kim Eberhardt, MS, OTR/L, a research coordinator with the Spinal Cord Injury
Program at the Rehabilitation Institute of Chicago conducts a literature review to identify a research
question.
• Reviewing the literature, • What kind of information has been generated on

• Identifying the research questions, the topic, and
• Selecting the research methods, • What kind of research methods investigators
• Writing a research plan, have used to generate that information.
• Obtaining ethical review,
Thus, the literature review serves to justify or
• Implementing research procedures and collecting
provide a rationale for the research since it allows
data,
the investigator to decide what knowledge is lack-
• Organizing and analyzing data,
ing and how to best go about generating it.
• Interpreting results to generate findings, and
Reviewing the literature first requires a system-
• Disseminating findings.
atic literature search. This search is ordinarily
These activities are listed in the sequence that accomplished through a variety of means. Today
investigators generally follow when planning and the most common method is using Web-based
implementing a study. However, it is often the case searching and search engines that retrieve publica-
that an investigator moves back and forth between tions based on key words. Another common
these activities. For example, all research begins method is a manual search that involves examining
with a review of the literature. Nonetheless, at the the references of existing publications to see what
end of the study when the findings are being pre- publications previous investigators have consid-
pared for dissemination, an investigator would be ered important.
remiss without going back to the literature to see if Once one has exhaustively identified and
anything new has been published in the interim. obtained the relevant literature, it is necessary to
In the following sections, each of these activi- analyze it. When analyzing the literature, an inves-
ties are discussed. They are briefly examined to tigator asks the following kinds of questions:
provide a general orientation to the research
• What is known about this topic?
process. Later chapters will significantly expand
• What theories are used to explain it?
each of these topics.
• What methods are used to generate this knowl-
edge?
Reviewing the Literature • What are the strengths and weaknesses of these
methods?
The aim of reviewing the literature is to identify,
• What gaps exist in knowledge about this topic?
evaluate, and understand the existing published
theory and research on a given topic. By doing a By applying these kinds of questions to the
thorough literature review, investigators learn: existing literature as a whole, the investigator
arrives at judgments that allow one to identify the wanted to know whether different factors motivate
research questions and select the methods. A well persons of different ages, then a quantitative sur-
done literature review makes it clear why the study vey is needed.
is important and needed, and provides important
information concerning how it should be under- Deciding the Research Methodology
taken.
Deciding the methodology of any study involves
making decisions about the research design, sam-
Identifying Research Questions ple, data collection, and analysis. In approaching
A key step in any study is deciding what question these decisions the investigator must first deter-
the research will seek to answer. Researchers often mine whether to draw from one or both of the
begin with broad questions and then narrow them. quantitative and qualitative traditions of research.
Creating the research question is closely tied to the Each tradition will be suited to answer certain
literature review, as it requires the investigator to types of research questions and not others.
identify what is not currently known that the inves- Deciding the design of the research involves
tigation will address. In addition to reviewing several interrelated decisions. First, the investiga-
the literature, investigators generally consult with tor has to decide the overall approach of the study.
other researchers in the area to make sure the This involves such broad considerations as
question they are developing is warranted and whether the investigation will be:
useful.
• A field study in which the investigator becomes
Identifying the research question also involves
immersed in the phenomena under question,
selecting a theoretical approach that frames the
• An experimental study in which the investigator
question. For example, an investigator wants to
will control different conditions in which sub-
examine the broad area of what factors increased
jects will be examined, or
motivation of clients to participate in therapy. To
• A survey study in which participants will
formulate a question, the investigator has to begin
respond to a written questionnaire.
with some kind of theoretical idea about what con-
stitutes motivation. This process can be fairly As noted earlier, when making such decisions,
straightforward if there is a single well-formulated the investigator is deciding on the best overall
theory that characterizes a particular research approach to answer the type of question the study
topic. In some other cases, there may be compet- is seeking to answer.
ing theories, so that one has to select from among Once the broad decision about research design
them. In still other instances there may not be a is made, other types of decisions must also be
clear theoretical approach, so that one has to iden- made to refine the study design. For example, in an
tify a theoretical approach that might be appropri- experimental study, the investigator will need to
ate to the topic. Even if an investigator does not decide whether random assignment is feasible and
specify a theory, certain assumptions and concepts what will constitute the different conditions to be
will be implicit in how any question is posed. compared. In a survey study, the investigator will
Thus, research is more logical and transparent have to decide whether subjects will participate in
when the theory is made explicit. the study only once, or whether they will be con-
How the research question is formulated deter- tacted multiple times over a longer period. In a
mines much of the rest of the study. In particular, qualitative study, the investigator will need to
the question asked shapes the selection of the decide, for instance, whether it will be an extended
research methodology. For example, if an investi- participant observation or a brief series of key
gator formulates a question about motivation that informant interviews. In designing any given
asks what factors influence motivation, then the study, investigators will often consider a number
design of the study will be largely descriptive and of possibilities. It is not uncommon to redesign a
could involve either qualitative or quantitative study several times while planning. All of these
approaches. On the other hand, if an investigator design decisions affect the rigor of the study as
wants to know whether an intervention improves well as the resources that will be needed to carry
motivation, then a quantitative design such as a out the study.
control group study or a single subject study will The investigator must also decide who will par-
be required. If an investigator wants to know ticipate in the study. This decision involves what
unique factors that motivate individuals in a par- characteristics the subjects will have (e.g., age,
ticular context, a qualitative field study is likely to how they will be recruited and chosen, and how
be the design of choice. Finally, if an investigator many will be included and what they will be asked
to do in the study). This aspect is referred to as request funding for the research. When the pro-
sampling. posal is used to request funding, it is referred to as
The final aspect of deciding the research meth- a grant proposal.
ods is determining how data will be collected and The research plan includes the elements already
analyzed. The answers to the study questions discussed (i.e., the literature review and the
depend on the quality of data collected and how research methods). Another component of the
they are analyzed. Once again the choices may research proposal is a management plan. It
range widely. For example, data may be collected includes a description of the major tasks necessary
with open-ended methods that evolve over the to complete the project, who will do them, and
course of the research or they may be collected when they will be done. Along with the manage-
through the use of highly standardized proce- ment plan, larger studies include an organizational
dures and are stable from subject to subject and plan that includes all the personnel involved in the
over time. No matter what type of data are col- study, what their responsibilities will be, and how
lected, investigators are concerned with their they will relate to each other.
dependability. A final component of a research plan is consid-
The approach of data analysis also depends on eration of the necessary resources for conducting
whether the study is qualitative and/or quantita- the research. These include, for instance, space and
tive. Qualitative data are ordinarily coded and clas- equipment, personnel, and supplies. A budget is
sified and then thematically analyzed. Qualitative also typically prepared for a research plan and is
data analysis tends to unfold as the research always required for a grant proposal, since it forms
unfolds (Hammell, Carpenter, & Dyck, 2000). the rationale for the funds that are being requested.
Often data analysis is both shaped by and shapes The budget covers necessary costs of the research
the kind of data that gets collected. Quantitative for such things as personnel to help carry out the
data analysis involves the use of statistics. research, supplies necessary for the research, and
Statistics may be used to describe what was found so on.
or to draw inferences about it (e.g., to decide
whether two groups differ on a characteristic or Obtaining Ethical Review
whether one variable predicts another).
How an investigator goes about making deci- Studies involving human beings as subjects
sions about study design and methods also undergo ethical review, which is a process
depends on the overall type of research being designed to:
done. In most quantitative research, most if not all
• Protect subjects from any harm,
of the decisions will be made before the research
• Ensure that subjects’ effort and any risk involved
commences. However, in qualitative research,
is warranted by the study’s importance, and
some of the decisions may be made before the
• Ensure subjects freely give informed consent to
research begins but others will depend on and
participate.
respond to how the research unfolds.
Persons who are not directly involved in the
research conduct a review of the proposed study to
Writing the Research Proposal
make sure it meets these ethical standards.
In most instances, investigators will write up a Institutions in which research is routinely con-
plan of the research in order to organize the plan- ducted maintain ethics boards (sometimes called
ning process and to communicate the plan to Institutional Review Boards), whose purpose is to
others. A research proposal is a document that review and approve research. Obtaining ethical
details the need and rationale for the study ques- approval is ordinarily the last step before begin-
tion, describes the anticipated methods that will be ning implementation of a study.
used to address the question, and provides the
anticipated logistics and necessary resources for Implementing Research Procedures
conducting the research.
and Collecting Data
The proposal serves first to organize the
research plan in the investigator’s mind and as a Implementation of a study can vary dramatically
kind of blueprint for later implementing the with the nature of the research. For example,
research. A proposal is also typically used to implementing a qualitative field study may involve
secure approval (e.g., administrative approval or months of participation with subjects in their natu-
supervisory approval when the investigator is in ral context, during which time the investigator
training). In other instances, the proposal is used to takes field notes, records interviews, and collects
documents. On the other hand, implementing brochures, and mailings. Once subjects have indi-
a controlled study comparing groups receiving cated an interest, the investigator must adequately
different therapy approaches may involve assign- inform the potential participants about the study
ing subjects randomly to the two intervention con- and obtain and document informed consent (a
ditions, collecting baseline data, providing the process in which potential subjects learn about the
interventions, documenting their conformity to the purpose of the study and what is being asked of
intervention protocol, and collecting post-inter- them as participants and subsequently decide
vention data. Implementing a survey study may whether or not to participate).
require obtaining a random sample of subjects When the research involves a single point of
with addresses, mailing the survey instrument, and contact with subjects, as in a survey, consent and
doing follow up mailings to ensure the highest data collection may occur at the same time.
possible response rate. However, many studies require participants to be
Research designed to develop assessments may involved over a period of time and thus require
involve a series of sequential steps (Benson & careful attention to subject retention. Once again, a
Schell, 1997). For example, the investigator may number of strategies, such as subject reimburse-
begin with collecting qualitative information from ment, ongoing contact, and messages to thank and
clients and/or therapists to ascertain what kind of remind, are used to maintain interest and involve-
information should be included in the assessment. ment with the study.
Then, once a prototype is designed, a pilot study An important consideration in all research is to
may be conducted to obtain systematic feedback make sure that the necessary data are collected.
from those who used or experienced the assess- Once again, concerns about data collection will
ment. Following this, the assessment may be depend on the design of the research. In qualitative
revised and then data will be collected and ana- research, the investigator is concerned that each
lyzed to examine whether they have the properties topic has been saturated (i.e., that enough data
of a sound assessment. Next, revision of the have been collected from enough different circum-
assessment may be followed by further data col- stances to ensure that a particular topic is fully
lection and analysis to determine whether it has informed). In survey research, as noted above, the
improved psychometrically. researcher will be concerned with getting
Depending on the type of research being con- responses from as many persons as possible from
ducted, the implementation may be either emer- the sample chosen. In experimental research, the
gent or highly structured. For instance, qualitative investigator will be careful to avoid missing data
field studies generally begin with an overall ques- from the pre- and post-group conditions.
tion and plan, but the investigation will be guided No matter what the type of research, the inves-
by what happens and what is found in the field. tigator must always be vigilant during the research
The selection of what to observe, whom to inter- implementation to make sure that the research
view, and what types of data to collect will be unfolds in ways that optimize confidence in the
shaped by the unfolding understanding of the topic findings. Understanding the logic behind the
under investigation. In this case, rigor depends on research design allows the investigator to deter-
the investigator’s careful attention to the situation mine how to respond to the inevitable unexpected
under study and strategic development of research circumstances that occur in implementing
strategies to faithfully create an understanding of research.
the situation that reflects how those in the situation
experience it (Lincoln & Guba, 1985).
Managing and Analyzing Data
In contrast, a controlled experimental study
requires strict adherence to protocols defined After data are collected, managing, storing, and
before the implementation begins. In this case, the analyzing the data is the next important step.
investigator will seek to avoid the interference of During this step, it is important to monitor data
natural conditions on the study process and will collection to make sure it is being carried out as
carefully document any aberration from the plan of planned and that the accumulated data are compre-
the research. Here, rigor is achieved through strict hensive. Thus, data are routinely monitored and
adherence to protocol. logged in as they are collected. Next, data must be
A pivotal aspect of implementing any study is prepared for the analytic process—that is, trans-
recruiting and retaining the participants or subjects formed into a format appropriate to either qualita-
necessary for the study. Subjects can be recruited tive and/or quantitative analysis. For qualitative
for studies using a variety of approaches such as analysis, most data are typically in the form of text
presentations that invite participation, fliers or or narrative and are usually entered into a qualita-
tive software package. Quantitative data are trans- goes about interpreting the data collected in
ferred to an electronic database appropriate to the research and generating the findings is quite vari-
software that will be used for statistical analysis. able depending on the type of research and the
Another important aspect of data management is to topic of inquiry.
ensure that data are secured so that they will not be In some research, the process is quite formal.
accidentally destroyed and to ensure that only For instance, in an experiment where hypotheses
members of the research team have access to the (statements of expected results) have been postu-
data in order to protect the confidentiality of the lated, the major task will be to decide whether the
study subjects. data analysis supports or requires one to reject the
Data analysis involves manipulating the data in veracity of the hypothesis. In qualitative research,
order to answer the research question. As noted the investigator must ponder the themes and pat-
earlier, qualitative data analysis typically involves terns in the data to generate insights into their
coding and sorting the data in order to identify key meaning (Hammell, Carpenter, & Dyck, 2000).
themes that will make up the findings, whereas This can be a highly creative process in which the
quantitative data analysis involves computing investigator’s theoretical background as well as
descriptive and inferential statistics. Data analysis knowledge of the subjects and their life contexts
is a complex process and there are a large number comes into play.
of qualitative and quantitative approaches to ana- In addition to the major work of generating an
lyzing data as will be discussed in detail later in understanding or assigning meaning to the patterns
this text. in the data, the process of interpreting results also
requires the investigator to skeptically examine the
Interpreting Results and data for alternative explanations to the ones being
Generating Findings pursued, and to make sure there are not problems
in the data set (e.g., missing or incomplete data, or
Perhaps the most critical and exciting aspect of an unexpected pattern in the data that affects how
any research is the process of making sense of pat- it can be statistically analyzed) that need to be con-
terns in the data and transforming them into a sidered or corrected in the analysis. Finally, the
coherent set of findings to be shared with other investigator has to carefully consider the degree of
members of the scientific community. As with confidence that should be assigned to the findings.
other aspects of the research process, how one This aspect involves several considerations includ-
ing such things as how persuasive the data patterns
are in supporting the conclusions being drawn, and
the limitations of the research methods used to
generate the findings.
Increasingly, investigators conduct this aspect
of the research in public. For example, they may
present preliminary findings and seek feedback
from peers. They may ask consultants with spe-
cialized expertise in the analytical methods (e.g.,
statistics) to give their opinions about the data and
their meaning. They may share their findings with
other persons doing research in the same area to
gain their insights.
Dissemination
No study is complete until it has been formally
shared with other members of the scientific com-
munity and other interested constituencies. As
noted earlier, investigators typically disseminate
their findings through refereed presentations,
posters, and published papers. In some instances,
the investigator may write a book to disseminate
Figure 4.3 A graduate student checks research findings.
data entered into the database to prepare for The purpose of disseminating research findings
analysis. in the scientific/scholarly community is twofold.
First, one is sharing the Understanding research

information generated No study is complete until and, in particular, being
through the research able to conduct research,
with others, thus con- it has been formally shared requires detailed knowl-
tributing to what is with other members of edge of all the elements
known about the phe- that were only touched
nomena under study. A the scientific community on in this chapter.
second and equally and other interested Subsequent sections of
important element is this book will offer this
that by making the constituencies. kind of specific informa-
research public, other tion.
scientists/scholars can
examine the work to determine its rigor and, there-
fore, form opinions about how dependable the REFERENCES
findings are.
Benson J., & Schell, B. A. (1997). Measurement Theory:
It has become increasingly important to share Application to occupational and physical therapy. In J.
research results with nonscientific groups who are Van Deusen & D. Brunt (Eds.), Assessment in occupa-
affected by the research (e.g., persons whose situ- tional therapy and physical therapy. Philadelphia: W.
ations or conditions were studied in the research). B. Saunders.
In the current information age, consumers are Campbell, D. T., & Stanley, J. C. (1963). Experimental and
quasi-experimental designs for research. Chicago:
increasingly desirous of accessing information McNally & Co.
generated by research themselves in order to make Crotty, M. (1998). The foundations of social research:
informed judgments. For this reason, many inves- Meaning and perspective in the research process.
tigators also seek to disseminate their findings in Crows Nest, Australia: Allen & Unwin.
DePoy, E. ,& Gitlin, L. (1998). Introduction to research:
formats that are more user-friendly to those out- Understanding and applying multiple strategies. St.
side of the scientific/scholarly community. Such Louis: C. V. Mosby.
formats include nonscientific articles, books, Web Hammell, K. W., Carpenter, C., & Dyck, I. (2000). Using
sites, videos, and brochures. qualitative research: A practical introduction for occu-
pational and physical therapists. Edinburgh: Churchill
Livingston.
The Full Circle of Research Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry.
Thousand Oaks, CA: SAGE Publications.
At the point research findings are disseminated, Neuman, W. L. (1994). Social research methods:
the research process has come full circle. Each Qualitative and quantitative approaches. Needham
investigation began as an examination of the liter- Heights, MA: Allyn and Bacon.
Polgar, S. & Thomas, S. A. (2000). Introduction to
ature to find out what was known about a topic research in the health sciences. Edinburgh: Churchill
and how such knowledge was generated. Once Livingston.
published, the research becomes part of that body Polit, D. F., & Hungler, B. P. (1999). Nursing research:
of literature. By culminating the research pro- Principles and methods. Philadelphia: Lippincott.
cess with publication, investigators link their Portney, L. G., & Watkins, M. P. (2000). Foundations of
clinical research: Applications to practice (2nd ed.)
work back to the community of scientists/scholars Upper Saddle River, NJ: Prentice-Hall.
working in a particular area. Without publication, Rea, L., & Parker, R. (1997). Designing and conducting
the research, for all practical purposes, does not survey research: A comprehensive guide. San
exist. Francisco: Jossey-Bass.
Rice, P. L., & Ezzy, D. (1999). Qualitative research meth-
ods, a health focus. Melbourne: Oxford University
Press.
Conclusion Seltiz, C., Wrightsman, L., & Cook, S. (1976). Research
methods in social relations. New York: Holt, Rinehart
& Winston.
This chapter discussed the elements that character- Stein, F., & Cutler, S. (1996). Clinical research in allied
ize all forms of research. It examined characteris- health and special education. San Diego: Singular
tic hallmarks of good research and overviewed the Press.
Strauss, A., & Corbin, J. (1990). Basics of qualitative
ordinary activities that make up the research research. Thousand Oaks, CA: SAGE Publications.
process. This chapter aimed to give the reader a Thompson, N. (2001). Theory and practice in human serv-
broad sense of what research involves. ices. Maidenhead: Open University Press.
C H A P T E R 5
Professional Responsibility
and Roles in Research
Anne Cusick • Gary Kielhofner
Mary smiles to the audience as she accepts the impact on their professional and personal lives.
award for her contribution to occupational therapy Moreover, their decisions about creating and con-
research. Over a 40-year career, her achievements suming research evidence affect the profession as
include published books and papers reflecting a whole.
studies funded by large competitive grants. This chapter explores professional responsibil-
Lingson rushes from his part-time clinical job ity and roles in conducting and consuming inquiry.
to college to teach a class, across town to pick up It examines ways in which occupational therapists
his children, and then back to the university cere- might view research and make choices about
monial hall. He wonders if he will ever see a day research roles in professional life. The chapter
when he does not feel tired. Donning unfamiliar describes knowledge and skills central to various
clothes, he proceeds into the hall to receive his research roles, and it illustrates some strategies
doctorate and postdoctoral fellowship. that can be used to develop them.
Hannah completes the final editing of her proj-
ect investigating an aspect of practice: “I’ll never
do that again: I’ll just read other folk’s research
and concentrate on putting that into practice” she
Responsibility, Uncertainty,
thinks. and Research
Ellamal goes to work early as she has done for
20 years. She approaches some new clients, invit- Being a professional brings with it responsibilities,
ing them to participate in a study, fills out screen- and ethical practice is one of them. Evidence
ing forms, and administers informed consent from research can be used to inform practice deci-
before sending their in- sions, guide therapeutic
formation on to the Pri- processes, and provide
ncipal Investigator. Then Evidence from research can relevant information
she starts her day’s work. about how to interpret
Carlos and Maria be used to inform practice outcomes of service
meet at a café and spread decisions, guide therapeutic (Cusick, 2001a). When
out their papers. Going therapists use research
through piles of data and
processes, and provide evidence in their clinical
background articles, they relevant information about decisions, they can better
spend hours working know what to do, with
on a critical review of
how to interpret outcomes whom, when, why, and
research papers, prepar- of service. how best to do it. They
ing a presentation for can also be more ac-
therapist peers in which countable as they are
they will recommend changes to practice. aware of potential and actual outcomes of their
Christa tells the student she is supervising that service (Barlow, Hayes, & Nelson, 1984; Cusick,
she’s never felt the need to read research articles 2001b). Occupational therapists, thus, have an eth-
since her on-the-job experience has taught her all ical responsibility to be aware of research and to
she needs to know about what works. engage with it.
The occupational therapists in these scenarios Occupational therapists also have a profes-
each hold a personal view about the importance of sional responsibility to use research to help
research in their profession. They have made enhance the quality of their clinical decisions. If
choices about whether and how to support, use, or problems addressed by therapists in practice were
produce research to enhance occupational therapy straightforward, and if issues and answers were
practice. These choices and judgments have an certain, solutions could be provided by unthinking
Chapter 5 Professional Responsibility and Roles in Research 47
implementation of protocol. There would be no • Research producer,

need at all for the training, responsibility, discre- • Research collaborator,
tion, expertise, and autonomy of professionals. • Research consumer, or
Professionals operate within uncertainty in the • Research advocate.
many complex and high-impact decisions they
make every day. Professionals must therefore find
Research Producers
ways to negotiate the uncertainty that comes with
practice (Charles, 2001; Dew, Dowell, McLeod, Research producers can be academics, practitioner-
Collings, & Bushnell, 2005; Mullavey-O’Byrne & researchers, and students who actively engage in
West, 2001). In the past, professional authority or research in university, clinical, and community set-
expertise was considered sufficient (Basmajian, tings (Barlow et al., 1984; Cusick, 2000, 2001c;
1975). However, in the 21st century, decision- Hinojosa, 2003; Murray & Lawrence, 2000;
making in professional practice needs to be backed Polatajko & MacKinnon, 1987). Research produc-
by rigorous and transparent evidence. In recogni- ers develop high levels of research expertise. They
tion of this demand for evidence, occupational design and lead investigations, develop teams of
therapy research productivity has steadily dedicated staff, and bring together research
increased (Majnemer et al., 2001; Paul, Liu, & resources to produce new knowledge. Research
Ottenbacher, 2002). Developments in evidence- producers generate published papers, conference
based practice have provided further opportunities presentations, books, and sometimes multimedia or
for scientific knowledge to be integrated with creative works. These resource products are widely
expert opinion in professional judgments. When disseminated and critically reviewed prior to and
used intelligently, evidence-based approaches sys- after release.
tematically use a variety of forms of information Some research producers choose this role early
including research, therapist, and client opinion to in their careers; others take a different route.
make meaningful and relevant decisions (Bury & Occupational therapists who select research pro-
Mead, 1998; Dawes et al., 1999; Taylor, 2000). ducer roles must commit themselves to the highest
This approach means therapists can responsibly level of precision and rigor. For this, they must
exercise professional discretion in practice within complete specialized research training at the doc-
known limits of their information base. toral and postdoctoral levels (Paul, 2001). Doctoral
Therapists need to be discerning about practice training emphasizes advanced knowledge and
priorities, and research information can help skills in both theoretical and methodological
them do this. When resources are scarce, diffi- domains. It culminates in the production of a dis-
cult decisions must be made about how best to use sertation that requires conception, implementation,
them. A research orientation to practice helps ther- and documentation of a major study or study series.
apists deal with the inter-professional competition, Dissertation research is completed under the
budget cuts, insurance denials, and other chal- supervision of a committee of seasoned resear-
lenges that can emanate from limited resources. chers who guide doctoral candidates in their
Further, when professionals have a research orien- research and who ultimately judge whether or not
tation, they can respond to challenges about one’s the research is sufficiently rigorous and important.
knowledge or service values as opportunities to Often, the doctoral dissertation process culminates
rationally examine evidence rather than interpret- with a “defense,” in which the doctoral candidate
ing criticism as personal affronts. presents their work, answers critical questions, and
responds to probing comments about the research
from supervisory committees and sometimes a
Research Roles public audience. The defense reflects the very pub-
lic nature of the research enterprise and serves as a
Individual therapists can engage with research in a way to evaluate the individual’s readiness to
variety of ways. For instance, they can do original engage in the public process of science as dis-
research, collaborate in studies, or engage in read- cussed in Chapter 34.
ing and critiquing research with peers. Therapists While the doctoral degree is increasingly com-
can take on one or more of the following roles mon in occupational therapy, postdoctoral training
(summarized in Table 5-1) (American Occupatio- is still rare. In mature research fields, postdoctoral
nal Therapy Foundation, 1983; Barlow et al., training is required before one enters into a fully
1984; Cusick, 1994, 2000, 2001b, 2001c; Cusick, independent research role. Postdoctoral trainees
Franklin, & Rotem, 1999; Murray & Lawrence, work alongside accomplished researchers and/or
2000): within a research team/laboratory. They are
advanced “apprentices” who learn more sophisti- able collaborator. In other instances, therapists
cated analytical techniques, and develop specific may bring skills they learned in a thesis project or
expertise in an area of study, a theoretical domain, research courses taken as part of professional or
or a specialized kind of inquiry. They also typi- post-professional education. Sometimes therapists
cally gain experience in grant writing and publica- will receive specialized training that enables them
tion during postdoctoral training. to implement an intervention being studied or col-
lect data in a reliable and valid manner. Often this
Research Collaborators training is provided in preparation for or as part of
therapists’ involvement in a particular study. Thus,
Collaboration can occur in a range of ways and, therapists who collaborate learn important skills
although a working knowledge of research is and knowledge through the research process itself.
always required, not all collaboration needs to be
at the level of a research producer. Examples of
collaborative activity that do not involve leading Research Consumers
research include: All therapists should use research to inform their
• Being subjects/participants in a study by answer- practice. Applying research information in practice
ing surveys or participating in focus groups, has been called “consuming research” (AOTF,
• Referring and screening clients for studies, 1983), “research utilization” (Brown & Rodger,
• Collecting data for an investigation, 1999; Craik & Rappolt, 2003), and being “research
• Implementing services that are being tested in an sensitive” (Cusick, 2001a). It is part of being a
intervention study, “reflective practitioner” (Fleming & Mattingly,
• Serving on the advisory board of a funded 1994; Parham, 1987). More recently, therapists
research grant, have been encouraged to become evidence-based
• Helping investigators negotiate the politics and practitioners, routinely consuming and using
administrative processes of research in a clinical research in practice (Cusick & McCluskey, 2000;
site, and Taylor, 2000).
• Identifying a research question and helping inter- The preparation for research consumer roles
pret the results of a study of practice. varies depending on how one goes about it. Most
professional programs in occupational therapy
Collaboration is the most common research provide at least the basic knowledge required to
involvement by therapists in clinical settings intelligently read a research report. Critical
(Majnemer et al., 2001). Without therapists who appraisal (such as that discussed in Chapter 42)
are willing to train for study requirements, to requires more advanced skills to formulate a ques-
implement protocols, to volunteer time, and to tion, identify appropriate literature, and even cal-
maintain quality records, many clinical studies culate some basic statistics that summarize the
could not be completed. Research collaboration is information available in research literature.
critical for the field. Critical appraisal skills are taught in some occupa-
Collaboration needs careful negotiation, plan- tional therapy programs and are often available
ning, good communication, and relationships of through continuing education. These skills are
trust (Brown, 1994; Cusick, 1994). Chapter 40 dis- likely to become a requirement of professional
cusses challenges and opportunities that arise in competency in the future.
collaboration between practitioners. Among other While all therapists should consume, use, and
things, expectations relating to requirements, apply research evidence to inform practice, some
authorship, and intellectual property need to be will take their role as critical consumers one step
negotiated and clear. Depending on the intellectual further. They will publicize knowledge gaps in
contribution made to the design, interpretation, practice as well as the information they find that
and writing up of the study, collaborators may or fills those gaps. They do this through conference
may not be considered “co-investigators” or “co- papers, letters to editors, discussion papers, and
authors.” (See Chapter 29 for a discussion of critically appraised topics. This form of critical
authorship guidelines.) consumerism is important both because it stimu-
Preparation and expertise required for research lates debate and further investigation, and because
collaboration vary widely depending on the nature it contributes knowledge to others. The prolifera-
of the research endeavor. In many instances, the tion of occupational therapy resources such as crit-
professional education and experience of occupa- ically appraised topics in journals, Internet sites,
tional therapists are all that is needed to be a valu- and discussion boards are examples of the growing
Postdoctoral Training
Dr. Patricia Bowyer is an occupational therapist Dr. Bowyer is also collaborating to complete a
who recently earned a doctoral degree. Dr. qualitative study on the impact of Enabling Self
Bowyer’s goal is to do clinically relevant research Determination, an occupational therapy service
related to the Model of Human Occupation using a program for persons with HIV/AIDS. The focus
range of research methodologies. She came to the of this inquiry is to understand how staff in transi-
University of Illinois at Chicago, where she is tional living facilities viewed the impact of these
completing a postdoctoral fellowship in the occu- new occupational therapy services on the clients
pational therapy department. and the facility.
In addition to taking two advanced courses in During her postdoctoral training, Dr. Bowyer
theory and statistics, she is working on a number will also gain experience reviewing grants and
of research projects that will give her experience in developing grant-writing expertise through
developing assessments and studying service out- participating in grant submissions. She also
comes. For instance, she is currently working on has opportunities to write and submit papers
developing a pediatric assessment. In this project, for publication. By working full time on a
she has sought input from a national pool of clini- range of research projects and processes, she
cians and she will go on to collect data internation- is developing advanced skills for the role of a
ally that will be used to examine the research producer.
psychometrics of this tool.
Postdoctoral fellow, Dr. Bowyer (center) discusses concepts related to her

research with several graduate students.
importance of this type of research consumer role identifying research priorities, and lobbying pro-
and contribution. fessional leaders to “do something” to help thera-
pists working on practice problems. They work as
research advocates by providing the momentum
Research Advocates
and support for research even though they do not
Some therapists support research by finding produce it themselves. The following are some
knowledge gaps, generating relevant questions, examples of research advocacy:
Research Collaborator
Heidi Waldinger Fischer is a research occupational

therapist working at the Rehabilitation Institute of
Chicago (RIC). When Heidi graduated from the
University of Illinois at Chicago, she went to work
at RIC as a staff therapist providing services on
inpatient and outpatient units to individuals with
spinal cord injury, traumatic brain injury, stroke,
and orthopedic conditions. During that time, Heidi
became more and more aware that the need for
evidence-based practice was growing. She was
also concerned about the lack of evidence to guide
many aspects of her practice.
After 3 years of working as a staff therapist,
she was offered the opportunity to assist engineers
in the Sensory Motor Performance Program to
explore the potential of rehabilitation robotics for
the upper extremity following stroke. Heidi saw
this as an opportunity to contribute toward the
building of a more substantial evidence base. For
the past 2 years, she has assisted in the
Neuromuscular Hand Rehabilitation and Arm
Guide Laboratories of the Sensory Motor
Performance Program at RIC. Her current research
experience focuses on stroke rehabilitation. She
works on projects aimed at understanding underly- Heidi Waldinger Fischer (left), a research occupa-
ing mechanisms of impairment, developing indi- tional therapist working at the Rehabilitation
vidualized treatment protocols, and development Institute of Chicago (RIC), works with a participant
and use of robotic devices to improve upper in a study of upper extremities utilizing rehabilitation
robotics.
extremity function. Her responsibilities include
participation in the collection and analysis of data
related to muscle activity, muscle stimulation, ciplinary colleagues with a unique understanding
brain imaging, and kinematics. Most recently, she of a person’s occupational performance. This
also has experience working at RIC’s Center for knowledge directly impacts the way robotic
Rehabilitation Outcomes Research assisting in the devices are developed and guides her fellow
development and implementation of a physical researchers in maintaining a client-centered focus
activity promotion program for persons with to their work. Heidi hopes her work will not only
arthritis. As an occupational therapist, Heidi is able serve to supplement the much needed evidence to
to offer a clinical perspective to motor control and inform practice, but also will ultimately improve
robotics research. She is able provide her interdis- the quality of life of those individuals she serves.
• Being involved in a local or national professional through emotional support, praise, and recogni-
association that funds or otherwise supports tion to create research supportive cultures, and
research, • Responding to agency invitations to comment on
• Lobbying policymakers/legislators to improve research priorities or goals.
access to research databases, funds, and profes-
Often persons who advocate research have no
sional development for occupational therapists,
specialized training or background in research.
• Asking employers to provide Internet access to
Their primary qualification is that they appreciate
therapists at work so they can consult easily
the importance of research to the vitality of the
accessible databases for “just in time” informa-
profession.
tion,
• Donating money to various causes and projects
Research Role Contributions
or to organizations that fund occupational ther-
apy research, Every research role makes valuable contributions
• Encouraging colleagues involved in research to the profession. Without research producers,
Practitioner-Researcher
Natasha Lannin has been an occupational therapist

for just over 10 years; she has mostly worked in
the area of neurological rehabilitation. As a new
graduate working in rural and remote settings, she
sought out research information on the Internet to
inform her practice decisions and support her
choice of clinical evaluations. She attended and
later presented at regional and national confer-
ences. She did this to ensure that she stayed up to
date with practice trends and received feedback on
her practice through peer review. Natasha also
made regular personal contacts with leading practi-
tioners and researchers at conferences and devel-
oped a lively e-mail network that helped overcome
her geographical isolation. As part of her post-
professional development, Natasha completed a
postgraduate diploma in case management by dis-
tance education. This diploma helped broaden her
practice skill base at the same time as focusing her Natasha Lannin (left), practicing therapist and doc-
interest on research methods that might be used to toral student at the University of Western Sydney,
solve clinical problems. as part of her doctoral studies applies a splint to a
Following a challenge by hospital colleagues to client in a clinical setting.
her use of splinting intervention for people who
had had a stroke, she critically appraised the litera-
ture and identified a significant knowledge gap. that she needed the kind of expertise and resources
Identifying this knowledge gap was a turning point that could only be obtained through a graduate
in her career. She decided to address it by becom- degree program. By this time, Natasha had moved
ing involved in research. She identified clinical to the city, and had a supervisory team and wide
experts and researchers who were willing to assist network of hospital clinicians ready and willing to
“long-distance” in the design of a study. She support her work by acting as recruiters. Natasha
engaged in individual study to learn more about obtained a scholarship for her research work, but
research methods. She sought out supportive inter- continued in a part time practice role, which per-
disciplinary networks in the hospital and across the mitted time for research and family responsibili-
region by phone. ties. Natasha Lannin has now completed her
With these resources, she began the long research degree, received a national award for her
process of study design, ethical approval, and research work, and published several clinical stud-
implementation of a small randomized controlled ies that address questions and issues she faced as a
trial. After the study was published in the Archives therapist. These include instrument studies, out-
of Physical Medicine, queries from her occupa- come studies, critical appraisals, and systematic
tional therapy colleagues made it clear that there reviews. Practitioner-researchers like Natasha
were further questions in need of answers. Lannin keep occupational therapy research focused
Next, Natasha decided to pursue a research on the difficult, complex, and urgent questions of
degree. It had become increasingly apparent to her day-to-day practice.
there is no original knowledge base. Without col- however, everyone can contribute to the profession
laborators, clinical studies could not be imple- becoming evidence-based (Cusick, 2003; Rodger,
mented. Without supporters, resources needed for Mickan, Tooth & Strong, 2003).
research or research dissemination and uptake are
not available. Without therapists to advocate for
research and guide research questions and priori- Taking on Research Roles
ties, research will not be relevant. Without thera-
pists to use or consume research in their service for Choosing and developing a research role unfolds
the public, the quality of practice suffers. All ther- much as any other life role (Cusick, 2001c) (Table
apists can help support inquiry-based practice in 5.1). Role development is part of the “social self”
some way. Not everyone will conduct research; (Blumer, 1969), and developing a “researching
Academic/Practitioner Collaborators
Collaboration between academics and practitioners series of research questions emerged, and a long-
is commonplace in clinical research. Each person term research relationship was born. Their collabo-
brings different role skills, knowledge, resources, rative projects have included evaluations of
and attributes to the inquiry process. Iona Novak occupational therapy home program interventions,
(left) is an occupational therapist with specialist instrumentation studies, and multidisciplinary
practice expertise in cerebral palsy. She has intervention outcome trials. Iona’s clinical contacts
worked for many years in a specialist cerebral resulted in ongoing research collaborations involv-
palsy service with her professional roles ranging ing physicians, other therapists, people with cere-
from direct care therapist, to institution-wide roles bral palsy, and families. Anne’s research contacts
facilitating professional development. Most resulted in collaborations with statisticians,
recently, she has become manager of research and research therapists, students, research assistants,
innovation activities in her institution. This role and administrators. Iona has completed her first
involves participation and leadership in national research postgraduate qualification with Anne (a
and international cerebral palsy forums. Master of Science Honors Degree) and is now an
Anne Cusick (right) is an occupational therapy industry sponsored PhD candidate working with
academic who has worked for more than 20 years Anne on a randomized controlled trial investigat-
in a university doing research, teaching, and ing occupational therapy intervention efficacy with
administration following her years as a clinician. children who have cerebral palsy. This clinical trial
Anne’s inquiry has focused primarily on issues involves research leadership of a multidisciplinary
related to practitioner research. More recently, she team, working with industry sponsorship, interna-
has become involved in clinical outcome studies, tional trial registration, and management of project
particularly randomized controlled trials. staff. The scope and logistics of this study are
Iona and Anne have been working together on made possible by Iona’s and Anne’s collaboration.
research projects for more than 5 years, and their Their partnership has involved not only adminis-
plans extend well into the next 5 years. They knew trative negotiation, transparency, and accountabil-
each other by reputation prior to meeting at a con- ity, but also a genuine relationship of trust and
ference where they discussed issues relating to respect driven by a shared vision of the research
outcome evaluation. From that conversation, a project benefits for people with cerebral palsy.
Occupational therapy academic, Dr. Anne Cusick (right) collaborates with

practitioner-researcher Iona Novak (PhD candidate).
52
Research-Resistant Practitioners
Despite the widely acknowledged importance Ironically, therapists who resist using research
of research to the profession, there are individuals information place themselves at risk of having
in the field who are “research resistant.” These their hard-earned expertise dismissed by interdisci-
therapists adopt fearful, hostile, or neglectful posi- plinary colleagues, administrators, and others.
tions toward information generated by research. Sometimes therapists feel intimidated by
Such therapists have clinical skills based on expe- research since their own professional training did
rience, but they have no way to ensure that their not include research or treated it in a cursory man-
practice is not limited or based on outdated infor- ner. However, even those who have limited
mation. They may also be guided primarily by research training or understanding have choices.
personal frames of reference and world views For example:
outside professional approaches (Cusick, 2001d). • They can choose to use research-based clinical
Therapists may be research resistant because guidelines that are presented in practical user-
they actively devalue research or, by default, friendly ways,
because they have chosen to do nothing. • They can adopt service recommendations of
Research resistance is not consistent with managers or supervisors who use research infor-
professional responsibility. mation, or
Therapists who are research-resistant may be • They can be open to new ideas that students or
concerned that their experience and expertise is colleagues bring that might be founded in
being de-valued by the increasing attention being research and ask questions.
given to research information. However, scientific
In the future, being research-resistant may not
knowledge is viewed as complementary to knowl-
be a role choice available. There are likely to be
edge gained through experience and the latter is
credentialing or employment conditions that
valued in the context of evidence-based practice
require therapists to have skills for and to use
(Richardson, 2001; Titchen & Ersser, 2001).
research information for practice.
self” is an intensely social process (Magoon & charged to develop and justify new services, a
Holland, 1984). One must be socialized to research therapist becomes engaged with evidence about
just as with other professional roles (Lewis & service outcomes),
Robinson, 2003). The process of identifying, • Inspired by meeting an investigator (e.g., at a
reflecting on, and constructing a research role is conference or at a university),
called “role taking” (Turner, 1956, 1990). Role • Fueled by personal desires (e.g., enriching one’s
taking occurs as one observes and comes to value work experience, becoming more credible and
different roles, tries out role behaviors, gets feed- able to justify decisions with good quality infor-
back, and eventually finds a good fit between the mation, or achieving the autonomy and flexibility
self and a role (Biddle, 1979). that research roles can bring with them),
Most investigation and discussion of research • Based on a drive to achieve enhanced status and
role development has occurred in relation to participate in higher status groups where one can
research producers (Cusick, 2000, 2001c, 2001d), have an influence on important issues in occupa-
but the processes of role development can be tional therapy (e.g., gaining national research
equally applied to other research roles. It starts grants that provide resources to study therapy,
with the particular stance a therapist takes toward influencing policy decisions, developing advan-
research. That stance is predisposed by personal ced educational opportunities for occupational
biography and context (Cusick, 2000, 2001c, therapists, and creating new services for an
2001d). underserved population),
An occupational therapist’s research standpoint • Based on a sense of obligation to contribute to
may be: research as part of professional identity,
• Fueled by a sense of generosity and gratitude
• Sparked by an event such as meeting someone or
toward the profession coupled with a desire to
experiencing a difficult situation, (e.g., deciding
“give something back,” or
to become more involved in research because of
• A strategic response to opportunities.
funding cuts or insurance denials that referenced
a lack of evidence about occupational therapy), Most therapists in research-related roles will
• Incidental to daily obligations, (e.g., when identify that a combination of these factors
Table 5.1 Research Roles and Related Training, Education, Expertise, and Activities
Role Typical Education/Training Expertise Activities
Research • Doctorate • Advanced knowledge of • Identify original research
producer • Postdoctoral training theoretical and questions and
research methods design/methods for
answering them.
• Secure funding for research.
• Supervise/oversee the
implementation of research.
• Work with interdisciplinary
collaborators, statistical
consultants and others with
specific research expertise.
• Prepare research reports for
presentation or publication.
Research • Professional training • Practice • Serve as subjects/
collaborator • Specialized training • Knowledge participants.
related to study • Specialized knowledge/ • Refer and screen potential
involvement skills subjects.
• For particular studies • Collect data.
• Implement services that are
being tested.
• Provide clinical
advise/expertise.
• Help negotiate the politics
and administrative
processes.
• Help identify a research
question and interpret
results.
Research • Professional training • Practice expertise to • Read research and use
consumers • Training in critical appraise relevance of findings to guide practice.
appraisal skills findings • Identify practice knowledge
• Skills in evidence-based gaps.
practice/ critical • Complete and present/publish
appraisal of research critical appraisals.
Research • Appreciation of the value • Administrative/ • Serve in professional
advocates of research that can Organizational associations that fund/
accrue from training expertise support research.
and/or experience • Personal commitment to • Lobby policymakers &
supporting research legislators.
• Ask employers for resources.
• Donate money
encouraging/supporting
colleagues involved in
research.
• Comment on research
priorities or goals.
sparked their interest and then influenced their construct research roles that suit them and their
own research role taking. life/work context. Building one’s own research
role requires:
Strategies for Research Roles • Exposure to role models and role alternatives,
A research standpoint is only the beginning. • Opportunities to reflect on and try out role behav-
Therapists need to then identify, reflect on, and iors,
• Opportunities to get feedback, and Depending on the role one chooses, the neces-
• Opportunities to evaluate new standpoints and sary skills may be readily learned or they may take
experiences. years of training. For example, many of the techni-
cal research skills required for research consumer
Exposure provides opportunities to identify
roles such as evidence-based practice are taught in
what particular research roles look like. It can hap-
professional entry level courses or continuing edu-
pen through encounters in occupational therapy
cation. On the other hand, as noted earlier, learning
education programs, at conferences, on the job,
to be a research producer requires earning a doc-
and socially. Strategies for enhancing one’s first-
toral degree and gaining postdoctoral training
hand role exposure include:
and/or experience.
• Attending colloquia by visiting speakers, In addition to research-specific skills, success-
• Joining journal clubs, ful researchers in occupational therapy must also:
• Attending professional association meetings, • Accept responsibility and maximize their own
• Seeking supervision or field work placements autonomy,
where research is taking place, • Clearly articulate the values that underpin their
• Volunteering to participate in a research project, work,
and • Engage in forward planning by setting priorities,
• Being employed as a research assistant. deadlines, and goals,
• Integrate and carefully schedule various activi-
One can also gain exposure through written
ties, and
profiles, biographies, seminars, and other media
• Understand the system in which they work and
that describe not only the technical aspects of
manage colleagues, gatekeepers, and people of
research roles, but also its social dimensions and
influence to get resources (Cusick, 2000, 2001c,
personal processes. This type of information is
2001d).
useful to consider the kinds of attributes required
for various research roles and for learning how Research-specific and the above mentioned
people balance home and work responsibilities and general inquiry skills are facilitated through train-
negotiate the obligations, demands, and politics of ing opportunities that provide realistic and respect-
research involvement. It is also important to learn ful feedback, and that couple clear expectations for
about the impact one can have through research performance with high degrees of autonomy
involvement and the personal satisfaction that can (AOTF, 1983; Bland & Ruffin, 1992; Cusick,
accrue from it. 2000; Magoon & Holland, 1984; Murray &
Taking on a research role is an important and Lawrence, 2000; Paul, Stein, Ottenbacher, & Liu,
conscious decision. Consequently, opportunities to 2002; Pelz & Andrews, 1976). This type of train-
figuratively try on potential research roles are use- ing and feedback often begins with the student role
ful. Therapists need to be able to not only think wherein students learn whether or not their litera-
about a particular research role “out there,” but ture review, critically appraised topic, or research
also think about themselves in relation to the role. methods assignment met standards and expecta-
Conversations with trusted mentors, supervisors, tions. It will assuredly continue for the developing
managers, friends, and colleagues permit thinking researcher when submitting conference papers,
out loud and provide different views of what is journal articles, grants, or requests for release time
required for various roles. Such conversations can to be involved in research. Over time, these
offer realistic feedback on one’s capacity, encour- repeated experiences are opportunities for learn-
age different scenario planning, identify practical ing, role feedback, and, if positive, research role
constraints and implications of particular role validation. Therapists therefore don’t just “do”
choices, and provide a “reality check.” research’ they “become” research producers, con-
Once particular research roles are selected, sumers, advocates, or collaborators through a
occupational therapists need opportunities to try reflective process and social process (Cusick,
out different research role behaviors, and to 2001c, 2001d; Young, 2004). They think about
acquire knowledge and technical skills for their desired roles, select and try on research role behav-
preferred role. Depending on the role, technical iors, get feedback on their relative role attainment,
knowledge and skills may range from reading and consider whether or not the role feels worth-
research in an informed way, to developing while. This self-reflective process involves a con-
research questions, preparing designs and proto- tinual internal dialogue with oneself about the
cols, to collecting and analyzing data, to deciding emerging and changing research role. That dia-
how to apply published findings to practice. logue is more meaningful when conversations with
trusted friends, mentors, or colleagues can provide in practice. Each of the therapists at the beginning
opportunities to “think out loud” and, in doing so, of this chapter had a particular approach to his or
further refine one’s research standpoint, role taking her practice and a demonstrable impact on the pro-
choices, and views about the worth of the research fession. Each had a different research standpoint
enterprise in one’s life. and each had his or her own story concerning his
or her research role. What will your story be?
Conclusion REFERENCES
American Occupational Therapy Foundation (1983).
In this chapter, we have stressed that research Research competencies for clinicians and educators.
involvement is a professional responsibility. We American Journal of Occupational Therapy, 37, 44–46.
Barlow, D. H., Hayes, S. C., & Nelson, R. O. (1984). The
indicated that occupational therapists make con-
scientist-practitioner: Research and accountability in
scious choices about research roles, which include clinical and educational settings. New York: Pergamon
those of advocate, consumer, collaborator, or pro- Press.
ducer of research. Strategies of research role Basmajian, J. V. (1975). Research or retrench: The rehabili-
development were also discussed. tation professions challenged. Physical Therapy, 55,
607–610.
While this chapter focused on individual Biddle, B. J. (1979). Role theory: Expectations, identities
research role development, no one can be success- and behaviors. New York: Academic Press.
ful in a research role when functioning only as an Bland, C. J., & Ruffin, M. T. (1992). Characteristics of a
individual. Research roles emerge only in relation productive research environment: Literature review.
Academic Medicine, 67, 385–397.
to others. Research involvement of any form is an
Blumer, H. (1969). Symbolic interactionism: Perspective
intensely social process. Whether involvement is and method. Englewood Cliffs, NJ: Prentice-Hall.
as a leader or member of a research team, joining Brown, G. T. (1994). Collaborative research between clini-
a journal club, serving on a editorial board, having cians and academics: Necessary conditions, advantages
one’s grant reviewed by a panel, or presenting a and potential difficulties. Australian Occupational
Therapy Journal, 41, 19–26.
paper at a conference, one is always engaged with Brown, G. T., & Rodger, S. (1999). Research utilization
others. Even the process of publishing a research models: Frameworks for implementing evidence-based
paper, which can take weeks of private writing occupational therapy practice. Occupational Therapy
time, ends up in a highly public process. International, 6, 1–23.
Bury, T., & Mead, J. (Eds.). (1998). Evidence-based health
At one time, research was viewed as a solo
care: A practical guide for therapists. Oxford:
enterprise. The ideal of the independent investiga- Butterworth-Heinemann.
tor was viewed as the epitome of science. Today, Charles, C. (2001). The meaning(s) of uncertainty in treat-
however, it is recognized that the best research is ment decision-making. In J. Higgs & A. Titchen (Eds.),
created by teams that include diverse members Professional practice in health, education and the cre-
ative arts. Oxford, UK: Blackwell Science, pp. 62–71.
who can bring their different expertise and per- Craik, J., & Rappolt, S. (2003). Theory of research utiliza-
spectives to the research enterprise. Participatory tion enhancement: A model for occupational therapy.
methods (such as those described in Chapters 38, Canadian Journal of Occupational Therapy, 70,
39, and 40) typically bring researchers together 266–275.
Cusick, A. (1994). Collaborative research: Rhetoric or real-
with practitioners, consumers, and community
ity? Australian Occupational Therapy Journal, 41,
members, all of whom function as integral partners 49–54.
in the research process. Research is also becoming Cusick, A. (2000). Practitioner-researchers in occupational
increasingly interdisciplinary in nature. This therapy. Australian Occupational Therapy Journal, 47,
means that occupational therapists must be pre- 11–27.
Cusick, A. (2001a). The research sensitive practitioner. In
pared to extend their research involvement to those J. Higgs & A. Titchen (Eds.), Professional practice in
beyond the profession. health, education and the creative arts. Oxford:
Research roles and responsibility are key fea- Blackwell Science, pp. 125–135.
tures of professional life. They underpin ethical Cusick, A. (2001b). 2001 Sylvia Docker Lecture: OZ OT
EBP 21C: Australian occupational therapy, evidence-
practice and provide ways to enhance practice
based practice and the 21st century. Australian Occu-
quality and to be accountable in addition to the pational Therapy Journal, 48, 102–117.
broader endeavor of contributing to the develop- Cusick, A. (2001c). The experience of practitioner-
ment of the profession. Research involvement is researchers in occupational therapy. American Journal
part of a community of effort in which people col- of Occupational Therapy 55, 9–18.
Cusick, A. (2001d). Personal frames of reference in profes-
laborate together to advocate for, create, critique, sional practice. In J. Higgs & A. Titchen (Eds.),
and make use of research evidence for the better- Practice knowledge and expertise in the health profes-
ment of the profession and the people we service sions. Oxford: Butterworth-Heineman, pp. 91–95.
Cusick, A. (2003). Clinical research: A room of one’s own. Parham, D. (1987). Toward professionalism: The reflective
Australian Occupational Therapy Journal, 50(1), therapist. American Journal of Occupational Therapy,
44–47. 41, 555–561.
Cusick, A., Franklin, A., & Rotem, A. (1999). Meanings of Paul, S. (2001). Postdoctoral training for new doctoral
‘research’ and ‘researcher’—the clinician perspective. graduates: Taking a step beyond a doctorate. American
Occupational Therapy Journal of Research, 19, Journal of Occupational Therapy, 55, 227–229.
101–125. Paul, S., Liu, Y., & Ottenbacher, K. J. (2002). Research
Cusick, A., & McCluskey, A. (2000). Becoming an productivity among occupational therapy faculty mem-
evidence-based practitioner through professional bers in the United States. American Journal of
development. Australian Occupational Therapy Occupational Therapy, 56, 331–334.
Journal, 47, 159–170. Paul, S., Stein, F., Ottenbacher, K. J., & Liu, Y. (2002). The
Dawes, M., Davies, P., Gray, A., Mant, K., Seers, K., & role of mentoring on research productivity among
Snowball, R. (1999). Evidence-based practice: A occupational therapy faculty. Occupational Therapy
primer for health care professionals. Edinburgh: International, 9, 24–40.
Churchill Livingstone. Pelz, D. C., & Andrews, F. M. (1976). Scientists in organi-
Dew, K., Dowell, A., McLeod, D., Collings, S., & zations productive climates for research and develop-
Bushnell, J. (2005). This glorious twilight zone of ment (rev. ed.). Ann Arbor, MI: Institute for Social
uncertainty: Mental health consultancies in general Research, University of Michigan.
practice in New Zealand. Social Science in Medicine Polatajko, H., & MacKinnon, J. (1987). Occupational ther-
(online March 2005, Science Direct). apy graduate education: The scientist practitioner
Fleming, M., & Mattingly, C. (1994). Action and inquiry: model. Canadian Journal of Occupational Therapy,
Reasoned action and active reasoning. In C. Mattingly 54, 119–138.
and M. H. Fleming (Eds.), Clinical reasoning: Forms Richardson, B. (2001). Professionalisation and profes-
of inquiry in a therapeutic practice. Philadelphia: F. A. sional craft knowledge. In J. Higgs & A. Titchen
Davis, pp. 316–342. (Eds.), Practice knowledge and expertise in the
Hinojosa, J. (2003). Therapist or scientist—How do these health professions. Oxford: Butterworth-Heineman,
roles differ? American Journal of Occupational pp. 42–47.
Therapy, 57, 225–226. Rodger, S., Mickan, S., Tooth, L., & Strong, J. (2003).
Lewis, S. J., & Robinson, J. W. (2003). Role model identi- Clinical research: Room for all? Australian Occupatio-
fication by medical radiation science practitioners: A nal Therapy Journal, 50, 40–43.
pilot study. Radiography, 9, 13–21. Taylor, M. C. (2000). Evidence-based practice for occupa-
Magoon, T. M., & Holland, J. L. (1984). Research training tional therapists. Oxford: Blackwell Science.
and supervision. In S. D. Brown & R. W. Lent (Eds.), Titchen, A., & Ersser, S. J. (2001). The nature of profes-
Handbook of counselling psychology. New York: John sional craft knowledge. In J. Higgs & A. Titchen
Wiley & Sons. (Eds.), Practice knowledge and expertise in the health
Majnemer, A., Desrosiers, J., Gauthier, J., Dutil, E., professions. Oxford: Butterworth-Heineman, pp.
Robichaud, L., Rousseau, J., & Herbert, L. (2001). 35–41.
Involvement of occupational therapy departments in Turner, R. H. (1956). Role taking, role standpoint, and
research: A provincial survey. Canadian Journal of reference group behaviour. Abridged from American
Occupational Therapy, 68, 272–279. Journal of Sociology, 61, 316–328. In B. J. Biddle
Mullavey-O’Byrne, C., & West, S. (2001). Practising with- & E. J. Thomas (Eds.), Role theory: Concepts and
out uncertainty: Providing health care in an uncertain research. New York: John Wiley & Sons, pp.
world. In J. Higgs & A. Titchen (Eds.), Professional 151–158.
practice in health, education and the creative arts. Turner, R. H. (1990). Role change. Annual Review of
Oxford, UK: Blackwell Science, pp. 49–61. Sociology, 16, 87-110.
Murray, L., & Lawrence, B. (2000). Practitioner-based Young, A. F. (2004). Becoming a practitioner-researcher: A
enquiry: Principles for post-graduate research. personal journey. British Journal of Occupational
London: Falmer Press. Therapy, 67, 369–371.
06kielhofner(F)-06 5/5/06 3:43 PM Page 58
S E C T I O N 2
Quantitative Designs
C H A P T E R 6
Descriptive Quantitative Designs

Gary Kielhofner
Descriptive research depicts naturally occurring There are two types of descriptive research:
events or characteristics of research participants
• Univariate research in which data are collected
(e.g., behaviors, attitudes, and other attributes)
on a single variable or a series of single variables
(DePoy & Gitlin, 1998; Polit & Hungler, 1999;
and then characterized with descriptive statistics,
Portney & Watkins, 2000). Descriptive research is
and
common in occupational therapy and serves a
• Correlational research in which relationships
number of purposes. It should also be noted that
between two or more variables are examined.
descriptive research is often the aim, or one of the
aims, of retrospective designs that use existing These two types of descriptive studies and their
databases, as discussed in Chapter 9 or survey uses in occupational therapy research are pre-
research, as discussed in Chapter 8. Strictly speak- sented in this chapter
ing, descriptive research also includes studies
that compare groups when there is not experi-
menter manipulation of an independent variable. Univariate Descriptive Studies
However, since the logic of group comparisons
is similar across a range of experimental and Univariate investigations are typically used to:
descriptive designs, descriptive group comparison • Characterize the sample or circumstances that
studies are discussed in Chapter 7. The aim of this make up any study,
chapter is to address other forms of descriptive • Characterize a problem or phenomena,
research. • Document incidence and prevalence of health
In many situations, too little is known to under- related conditions,
take studies in which independent variables are • Establish norms,
manipulated and their effects on dependent vari- • Document developmental phenomena, and
ables are observed. Descriptive research often • Document case studies.
takes advantage of naturally occurring events or
available information in order to generate new Chapter 15 provides an overview of descriptive
insights through induc- univariate statistics that
tive processes. Conse- are used for these types
quently, descriptive Descriptive research often of investigations. These
investigations often serve takes advantage of naturally univariate descriptions
an exploratory purpose. are ordinarily in the form
Such studies typically occurring events or available of frequencies, central
lead to greater under- information in order to gen- tendencies (e.g., mean)
standing of phenomena, and dispersion (e.g.,
and the resulting concep- erate new insights through range or standard devia-
tualizations are then later inductive processes. tion). Below, the nature
tested through more rig- and purposes of these
orous research designs. types of descriptive
Sometimes basic descriptive information is needed research are discussed.
in order to indicate norms, trends, needs, and cir-
cumstances that inform and guide practice. Finally, Characterizing a Study Sample
descriptive research is a component of all research
and Characteristics
studies, since it is used to characterize subjects and
other relevant circumstances that surround the No matter how complex the design of a study, all
research. investigations include components of basic
Chapter 6 Descriptive Quantitative Designs 59
descriptive design through which the participants tions. When such studies describe ways that
and other relevant circumstances in the study are individuals with disabilities adapt, they are helpful
characterized. Basic descriptive data are always as guidelines for practitioners. The first feature
collected, analyzed, and reported in any research. box that follows contains an example of such a
This is true for qualitative research as well, in study.
which basic quantitative information is typically
used along with qualitative information to charac- Incidence and Prevalence Studies
terize the study participants.
Essential to any study report is information Another important purpose of descriptive studies
about the subjects of a study on key demographic is to document the incidence and prevalence of
variables (e.g., age, sex, race or ethnicity, diagno- health-related conditions. Studies that address
sis, type of impairment). In addition to character- this purpose are referred to as epidemiological
izing research participants, it is also important to studies. Although incidence and prevalence studies
characterize other conditions that make up a study. are usually conducted by public health and
For example, in a study of intervention outcomes, medical researchers, their results are widely used
the investigator would describe the types of serv- in occupational therapy. In addition, occupational
ices provided, the length of time over which the therapy researchers are increasingly contribut-
service was provided, the frequency of service ing to and conducting incidence and prevalence
provision, and how much service in total was studies that focus on functional aspects of impair-
provided. ment.
Prevalence is a proportion of a total population
who have a particular health-related condition
Characterizing a Phenomenon or Problem (Polit & Hungler, 1999; Portney & Watkins, 2000).
Descriptive studies in occupational therapy often Prevalence (P) is calculated as:
serve the purpose of illuminating some phenomena
or circumstance that is of interest to the field. For Number of observed cases of a
instance, Bar-Shalita, Goldstand, Hahn-Markowitz, condition at a given time point or
and Parush (2005) studied the response patterns during a given interval
of typical 3- and 4-year-old Israeli children to tac- P
tile and vestibular stimulation. This study was Total population at risk
undertaken to generate basic knowledge about
the response patterns of typically developing When prevalence is examined at a specific time
children to sensory stimuli. The investigators point, it is referred to as point prevalence. When it
sought to describe sensory response patterns and is calculated based on observed cases during a
to determine whether they changed from age 3 to time period (e.g., over a year), it is referred to as
age 4. period prevalence (Portney & Watkins, 2000).
The study results indicated that these children Prevalence is calculated without reference to the
were neither hypo- nor hyper-responsive to tactile onset of the condition; thus, it aims to characterize
and vestibular stimuli. By providing evidence what proportion of a population has a condition at
that typically developing children do not swing a point or period of time without consideration of
between hyper- and hypo-responsivity, this study when the condition began. The feature box on
provided a framework for identifying children prevalence rates for chronic fatigue syndrome
whose sensory responsiveness is atypical. The presents an example of a prevalence study.
investigators also found no evidence of change in Incidence is concerned with how many persons
responsiveness from the 3- to 4-year-olds. This have onset of a condition during a given span of
finding suggested that children’s patterns of time. Incidence refers to the number of new cases
response to tactile and vestibular stimulation have of a disease or disability in a population during
stabilized by age 3. As this investigation illustrates, the specified time period. It can be calculated as
descriptive research that characterizes typical phe- either cumulative incidence or incidence rate
nomena, such as sensory responsiveness, can be (MacMahon & Trichopoulos, 1996).
very useful in occupational therapy. Cumulative incidence is calculated as:
In addition, descriptive studies are often used
Number of observed new cases
in occupational therapy to characterize functional
during a specified period
aspects of a disability. Such investigations are
CI
helpful in understanding what types of occupa-
Total population at risk
tional challenges are faced by certain popula-
Liedberg, Hesselstrand, and Henriksson (2004) population data. For example, the authors found
used a diary method to collect data on the time that the women in this study spent more time in
use and activity patterns of women with long- self-care than women in the general population.
term pain (diagnosed with fibromyalgia). The The study also included an intervention component
women were asked to fill in a diary for 7 con- for all women (i.e., they were supported to exam-
secutive days and again 3 months later over 4 ine the results of the analysis of their initial diary
consecutive days. In the diary they noted what they entries and to identify goals for changing their
were doing during each time period, where they daily activity patterns). More than half of the
were, with whom they were doing the activity, women were successful in achieving some of their
whether they had physical problems, and their long-term, comprehensive goals by the second
mood. The investigators then coded the diary period of data collection.
entries in a number of ways. For example, they This descriptive study provided a detailed por-
coded activities as belonging to one of seven trayal of these women’s daily lives. Such descrip-
spheres of activity (care of self, care of others, tive findings are helpful for understanding how
household care, recreation, travel, food procure- persons with chronic pain often organize and expe-
ment or preparation, and gainful employment). rience activities in their lives. The findings also
Within these general categories the activities point to the value of this kind of data collection
were further sorted into more detailed activity cat- and analysis as a tool for educating clients in reha-
egories (e.g., sweep, vacuum, scrub, and dust as bilitation, and as a framework for helping clients
categories of household care). This study yielded think about how they will accommodate and cope
data that could be compared with existing Swedish with their condition.
Prevalence Rates for Chronic Fatigue Syndrome
Chronic fatigue syndrome (CFS) is an ambiguous explain a participant’s symptoms. Descriptive

and debilitating condition. Initially, it was consid- statistics that included univariate and multivariate
ered a rare syndrome and it was believed to occur approaches were used to estimate the overall
only among Caucasian, affluent persons who were prevalence of CFS and the prevalence of CFS
“overachievers.” Early attempts to estimate the according to sex, ethnicity, age, and socioeco-
prevalence of CFS utilized physician referrals nomic status. Results produced an estimate of
as a means by which to count individuals with the CFS as occurring in approximately 0.42% of the
diagnosis. The problem with this approach was sample. This translates to a prevalence estimate
that many physicians either did not believe in the that CFS occurs in 4.2 per thousand persons.
existence of CFS or were unfamiliar with the Moreover, when the prevalence was broken down
diagnostic criteria. In addition, prior studies neg- according to sociodemographic categories, CFS
lected to account for the large proportion of people was found to be highest in women, in members
within the United States that do not have access to of numerical minority groups, and in individuals
medical care and do not have the opportunity to of lower socioeconomic status.
receive an evaluation and diagnosis. Findings from this study challenged prior per-
Jason et al. (1999) conducted a study to esti- ceptions of CFS as a rare disorder that affected
mate the prevalence of CFS in a randomly select- only Caucasian upper-class women. This study
ed, community-based sample that was free of exemplifies the potential that descriptive epidemi-
biases imposed by access to medical care and ological studies have in changing existing knowl-
attitudes about CFS held by referring physicians. edge and attitudes about health conditions and
To estimate the CFS prevalence, researchers made the people who experience them. When the results
telephone contact with a sample of 18,675 adults of this study were replicated by a similar method-
residing in Chicago. These individuals completed a ology, the Centers for Disease Control and
brief telephone interview about fatigue. Those who Prevention revised their official estimates of preva-
reported symptoms that matched CFS diagnostic lence rates. Moreover, the official recognition that
criteria (screened positives) and a non-fatigued this was not a rare disease led to increased federal
control group (screened negatives) received a com- funding to address the needs of this population
plete medical and psychiatric evaluation. The pur- and increased awareness of health providers that
pose of the evaluation was to rule out other it was a prevalent disorder and that it occurred
medical and psychiatric conditions that might among minority populations
60
Incidence rate (IR) is calculated as: past 20 years, the incidence of the disease has
changed. That is, HIV increasingly affects women,
Number of observed new cases during
individuals from minority populations, and indi-
a specified period
viduals with histories of mental illness and sub-
IR
stance abuse (CDC, 2000; Karon, Fleming,
Total person-time
Steketee, & DeCock, 2001; Kates, Sorian,
In the formula for IR, the denominator is calcu- Crowley, & Summers, 2002; Orenstein 2002).
lated as person-periods. For example, if a condi- Findings on the incidence of new cases thus help
tion was studied over a 10-year period and 100 identify which populations are at greatest risk.
persons were enrolled, the total possible person-
years are 1,000 (100 persons observed over 10 Normative Research
years). However, if two persons died and were no
Investigators undertake normative research in
longer in the sample after 5 years, they would con-
order to establish usual or average values for spe-
tribute only 10 person-years (5 years x 2 persons)
cific variables. This type of research is helpful for
to the formula, reducing the denominator by 10
identifying and characterizing performance prob-
person-years. Unlike cumulative incidence, which
lems (i.e., knowing how extreme a person’s devia-
assumes all subjects are at risk during the entire
tion from a norm is) and doing treatment planning.
period studied, incidence rate characterizes the
As Portney and Watkins (2000, p. 171) note, norms
number of new cases as a proportion of persons
are “often used as a basis for prescribing corrective
who were actually at risk during each of identified
intervention or predicting future performance.” For
segments of the total period studied (e.g., for each
example, research that led to the description of
year of a 10-year period).
what is normal strength for different populations is
Prevalence and incidence rates are important to
used routinely in occupational therapy practice.
providing both an understanding of the magnitude
These types of norms serve as a basis for evaluat-
of a health-related condition (i.e., how common or
ing performance of a given individual by compar-
prevalent it is) and risk (i.e., what are the chances
ing that individual with what is typical for
of someone incurring a given health-related condi-
someone with the same characteristics (e.g., age
tion). For example, in 2003, the Centers for
and sex). Research that aims to establish norms
Disease Control and Prevention (CDC) estimated
must be particularly careful to avoid sampling
that there were between 800,000 and 900,000 peo-
bias. For this reason, normative studies use random
ple living with human immunodeficiency virus
samples and ordinarily rely on large sample sizes.
(HIV) in the United States. Calculated as a pro-
portion of the population, these existing cases
Developmental Research
would represent prevalence of HIV in the United
States. At that time, the CDC also estimated that Developmental research seeks to describe “pat-
approximately 40,000 new (HIV) infections occur terns of growth or change over time within
annually in the United States (CDC, 2004). When selected segments of a population” (Portney &
calculated as a proportion of the total U.S. popula- Watkins, 2000, p. 14). Such studies may describe
tion, these cases would represent the cumulative patterns of change that characterize typical growth
incidence of HIV infection. The prevalence of HIV and development. Developmental research has
infection provides researchers and healthcare been important in documenting the course of
providers an understanding of the magnitude of the acquiring functional abilities in childhood (e.g.,
problem of HIV infection in the United States, crawling, walking, talking, grasping) as well as the
since it provides information about both the functional changes associated with aging.
absolute number of cases and the proportion of the Other developmental investigations seek to
total population affected by the condition. The characterize the course of disease or disability over
incidence estimate provides an understanding of time (e.g., the course of functional recovery from a
risk. traumatic event or the course of functional decline
Incidence and prevalence are ordinarily calcu- in degenerative conditions).
lated for subpopulations as well (e.g., different Developmental research usually involves a
ages, males versus females), since the occurrence cohort design in which a sample of participants is
of new and existing cases of conditions is not followed over time and repeated measures are
evenly distributed across the population. For taken at certain intervals in order to describe how
example, early studies of HIV in the United States the variable(s) under study have changed (Stein &
indicated that males who engaged in homosexual Cutler, 1996). Sometimes developmental research
activity were at highest risk. However, over the is accomplished through a cross-sectional design
62 Section 2 Quantitative Designs
in which the investigator collects data a one time case in which occupational therapy services based
point from a sample that is stratified into different on concepts of motor control were used to improve
age groups (e.g., children who are 6, 12, 18, and 24 mobility and community access in an adult with
months of age or older adults who are 60–64, ataxia. The study reported baseline information
65–69, 70–74, 75–79, etc.). Observed differences from the Functional Index Measure (FIM™) and
between the sample in each age strata are then then reported occupational therapy techniques of
attributed to the process of development or aging. “adapted positioning, orthotic prescription,
Cohort studies have the advantage of eliminating adapted movement patterns, and assistive technol-
effects of sample bias on observed changes, but ogy” that were used to address client-centered
they are subject to the influence of historical events functional goals (Gillen, 2002, p. 465). The study
that are unrelated to the course of development. reported positive changes in the client’s FIM score
Cross-sectional designs avoid the latter problem following the intervention.
but are prone to cohort effects (i.e., effects that are
unrelated to age and are due to some circumstance
unique to a particular age cohort). In both cases, Correlational Research
random sampling is critical in order to generalize
from the sample to the population the study is Correlational research aims to demonstrate rela-
intended to characterize. tionships between variables under study (Portney
& Watkins, 2000). Correlation refers to an interre-
lationship or association between two variables
Descriptive Case Studies
(i.e., the tendency for variation in one variable to
Descriptive case studies are in-depth descriptions be either directly or inversely related to variation
of the experiences or behaviors of a particular indi- in another variable). This type of research is some-
vidual or a series of individuals. Case studies are times referred to as exploratory, since it is fre-
most typically undertaken to describe some new quently undertaken to identify whether specified
phenomena or to document a client’s response to a variables are related or to determine which vari-
new intervention. While many case studies in ables are related in a multivariable study.
occupational therapy are qualitative in nature, Correlational studies also can provide evidence
quantitative case studies can also be useful in char- that is consistent or inconsistent with causal asser-
acterizing how an individual has responded fol- tions and thus they can provide important evidence
lowing a traumatic event and/or intervention. for developing theoretical propositions that assert
Descriptive case studies differ from the single- causal relationships between variables (see the fol-
subject designs described in Chapter 11 in that lowing feature box). An important limitation of
there is no experimental manipulation of an inde- correlational studies with reference to making
pendent variable. Rather, the investigator docu- causal assertions is that, without experimental con-
ments variables as they naturally occur or trol, there is no way to rule out the possibility that
documents an intervention and what happens fol- two variables might be related by virtue of their
lowing it (but without the comparison to the association with a third, unobserved variable.
absence of the intervention or to another interven- Correlational studies can serve as helpful first
tion). Case studies are most valuable when the steps in sorting out causal relationships (i.e., if two
information reported is comprehensive. Thus, variables are not correlated, then there is no reason
investigators undertaking a case study will either to anticipate or test for a causal relationship
attempt to provide data on several variables of between them). Many correlational studies serve
interest or provide repeated measures of some as precursors to experimental studies that exert
variable over time. experimental control over independent and
While such case studies are readily related to dependent variables in order to draw inferences
practice, they are one of the least rigorous forms of about causation.
research since they lack control and generalizabil- While many correlational studies are concerned
ity (Portney & Watkins, 2000). For this reason, a with relationships between pairs of variables,
series of, or several, case studies are often used as investigators are often interested to know the rela-
the basis from which to generalize concepts which tionship between several variables that are hypoth-
then lead to more controlled studies. esized or presumed to have a causal relationship
Case studies can be particularly useful for with a dependent variable. For example, many
investigating new interventions or interventions studies have examined factors that influence return
that require substantial individualization or trial to work following an injury. In this case, whether
and error. For example, Gillen (2002) reported a or not a person returns to work is the dependent
Correlational studies are often useful for testing sation, independent of the level of depression.
hypothesized relationships derived from theory. Another example is a study by Peterson et al.
For example, the Model of Human Occupation (1999) that examined the relationship between
(Kielhofner, 2002) argues that volition leads to personal causation (feelings of efficacy related
choices about occupational participation. Based on to falling) and the pattern of occupational
this theoretical argument, it can be hypothesized engagement in 270 older adults. They found, as
that volitional traits should be associated with expected, that lower personal causation (i.e., lower
observed patterns of occupational participation. falls self-efficacy) was related to reduced partici-
Correlational studies have been implemented to pation in leisure and social occupations. These
test this proposed relationship. two studies thus provided evidence in support
For example, Neville-Jan (1994) examined the of the proposition that volitional traits (in this
correlation between volition and patterns of occu- case, personal causation) influence the choices
pation among 100 individuals with varying degrees persons make about engaging in occupations. As
of depression. Her study found, as hypothesized, a noted in this chapter, however, these correlations
relationship between the adaptiveness of the sub- are consistent with but cannot be taken as proof of
jects’ routines and a measure of their personal cau- causation.
variable of interest and a study may examine the REFERENCES

influence of number of supposed causal or predic- Bar-Shalita, T., Goldstand, S., Hahn-Markowitz, J., &
tive variables (e.g., age, education, extent of Parush, S. (2005). Typical children’s repsonsivity pat-
impairment, personality, worker identity, and pre- terns of the tactile and vestibular systems. American
vious work history) on return to work. This type of Journal of Occupational Therapy, 59, 148–156.
Centers for Disease Control and Prevention. (CDC).
study ordinarily uses regression analysis (dis- (2000). HIV/AIDS Surveillance Supplemental Report
cussed in Chapter 17). The aim of such studies is (Vol. 7). Atlanta: U.S. Department of Health and
to identify the amount of variance in the dependent Human Services, Centers for Disease Control and
variable that is explained by a set of predictor vari- Prevention.
Centers for Disease Control and Prevention (CDC). (2004).
ables. HIV/AIDS Surveillance Report, 2003 (Vol. 15). Atlanta:
In correlational research, there may or may not U.S. Department of Health and Human Services,
be a temporal sequence between the hypothesized Centers for Disease Control and Prevention.
predictor variables and the dependent variable of DePoy, E., & Gitlin, L. N. (1998). Introduction to
interest. Sometimes correlational studies are retro- research: Understanding and applying multiple strate-
gies (2nd ed.). St. Louis: C. V. Mosby.
spective and the information on all variables is col- Gillen, G. (2002). Improving mobility and community
lected simultaneously. However, in prospective access in an adult with ataxia. American Journal of
studies data on the predictor variables are collected Occupational Therapy, 56, 462–465.
first and the dependent variable is observed at a Jason, L. A., Richman, J. A., Rademaker, A. W., Jordan, K.
M., Plioplys, A. V., Taylor, R. R., et al. (1999). A com-
later time or at several later time points. Obviously, munity-based study of chronic fatigue syndrome.
when there is temporal sequence, inferences about Archives of Internal Medicine, 159, 2129–2137.
causal relationships have greater weight. Karon, J. M., Fleming, P. L., Steketee, R. W., & DeCock,
K. M. (2001). HIV in the United States at the turn of
the century: An epidemic in transition. American
Conclusion Journal of Public Health, 91(7), 1060–1068.

Kates, J., Sorian, R., Crowley, J. S., & Summers, T. A.
(2002). Critical policy challenges in the third decade of
Descriptive research is common in occupational the HIV/AIDS epidemic. American Journal of Public
Health, 92(7), 1060–1063.
therapy. As noted in this chapter, it serves a num- Kielhofner, G. (2002). A model of human occupation:
ber of important purposes, not the least of which is Theory and application (3rd ed.). Baltimore: Williams
leading to more sophisticated research designs & Wilkins.
such as those discussed in the following chapters. Liedberg, G., Hesselstrand, M., & Henriksson, C. M.
This chapter focused on descriptive research in (2004). Time use and activity patterns in women with
long-term pain. Scandinavian Journal of Occupational
which investigators obtained observational data Therapy, 11, 26–35.
directly from subjects. Other types of studies, MacMahon, B., & Trichopoulos, D. (1996). Epidemio-
including retrospective studies that examine exist- logy: Principles & methods (2nd ed.). Boston: Little,
ing data sets and survey research, also sometimes Brown.
Neville-Jan, A. (1994). The relationship of volition to
employ designs whose purpose is descriptive. adaptive occupational behavior among individuals with
When that is the case, the underlying logic of these varying degrees of depression. Occupational Therapy
studies is the same as described in this chapter. in Mental Health, 12(4), 1–18.
Orenstein, R. (2002). Presenting syndromes of human Polit, D. F., & Hungler, B. P. (1999). Nursing research:
immunodeficiency virus. Mayo Clinic Proceedings, Principles and methods. Philadelphia: Lippincott.
77(10), 1097–1102. Portney, L. G., & Watkins, M. P. (2000). Foundations of
Peterson, E., Howland, J., Kielhofner, G., Lachman, M. E., clinical research: Applications to practice (2nd ed.).
Assmann, S., Cote, J., & Jette, A. (1999). Falls self- Upper Saddle River, NJ: Prentice-Hall.
efficacy and occupational adaptation among elders. Stein, F., & Cutler, S. (1996). Clinical research in allied
Physical & Occupational Therapy in Geriatrics, health and special education. San Diego: Singular
16(1/2), 1–16. Press.
C H A P T E R 7
Group Comparison Studies:

Quantitative Research Designs
David L. Nelson
Designs that compare groups are among the most

common in quantitative research. Group compari-
Basic Experiments
son designs range from true experiments to Basic experimental design is the most rigorous
comparisons between naturally occurring groups, type of group comparison study (Campbell &
and from elegantly simple designs to complex Stanley, 1963; Kerlinger, 1986). This type of
combinations of different kinds of interventions, design will be examined by starting with a defini-
outcomes, and controls. Interestingly, although tion of and then explaining each part of the defini-
some of the simplest designs have the best claim to tion using examples.
validity and utility, relatively complex designs are
appropriate for certain research questions. Definition of a Basic Experiment
Experimental designs provide evidence about
probable causality. An experiment tests the proba- • There is one sample that is drawn representa-
bility that an intervention has an effect on an out- tively from one population.
come, or the probability that one intervention has • There is one categorical independent variable.
a different effect on an outcome than another inter- • Study participants are randomly assigned to as
vention. Geniuses such as Thorndike and many groups as there are conditions to the inde-
Woodward (1901); Fisher (1935); and Amberson, pendent variable.
McMahon, and Pinner (1931) integrated con- • The independent variable is administered as
cepts such as randomization and controls with planned.
probabilistic statistics, and the result is a rational • Potentially confounding variables are minimized,
and logical basis for if–then statements. When and otherwise uncontrollable events are equally
if–then statements are drawn from theory or from likely across the groups.
clinical observations and transformed into tes- • There is one dependent variable on which all
table hypotheses, science advances. Embedded in subjects are measured or categorized.
this chapter are procedures to enhance validity • The experimental hypothesis tested is the proba-
as well as practical discussions about the assu- bility of a causal effect of the independent vari-
mptions that must be made in quantitative able on the dependent variable within the
research. population.
Nonexperimental group comparison designs
(sometimes called quasi-experimental designs) Writing about group designs is a humbling expe-
lack the degree of control achieved by experimen- rience. The methodological literature concerning
tal designs. In some cases, these kinds of designs group research designs is so broad and so deep
can lay the groundwork for subsequent experi- that every statement in this chapter could be fol-
mental investigations. In other cases, where lowed by 20 reference citations. As an alterna-
the researcher needs to know about the differ- tive, this chapter cites only key references
thought to be of the most direct benefit to the
ences between different kinds of people, true reader. Those interested in more in-depth treat-
experimentation is impossible. The researcher ments of group research designs are referred the
must build other kinds of controls into these list of publications in the “Additional Resources”
designs. section at the end of the chapter. Both the pri-
This chapter begins with a discussion of exper- mary references and this additional bibliography
imental designs and proceeds through other types are sources from which I have learned and con-
of group comparison designs. A basic premise is tinue to learn, and collectively they support the
that no design is best for all research questions; statements made in the chapter.
each has its advantages and disadvantages.
In the following sections, each element is able when the researcher examines its probable
examined in detail. effects on an outcome (called a dependent vari-
able). The independent variable is the possible
cause, and the dependent variable is the possible
Explaining the Definition effect. In an occupational therapy study, designed
There Is One Sample that Is Drawn to test whether an occupationally embedded exer-
Representatively from One Population. cise protocol produces a different outcome from a
rote exercise protocol, one condition of the cate-
A population is any set of people (or in some gorical independent variable might be stirring
cases, animals) who share common features. cookie batter and another condition might be the
“Americans 70 years of age or older residing in same kind of stirring but with no batter. The
extended care facilities (ECFs)” is a population. dependent variable could be the number of times
Similarly, “persons with Parkinson’s disease (stage that the circular motion is completed.
II through IV)” is another population, and “6-year-
old children with autism (but not mental retarda-
tion) attending special education classes” also
Study Participants Are Randomly
makes up a population. Assigned to as Many Groups
A sample consists of the persons (participants as There Are Conditions to the
or subjects) who are in the study. The sample is a Independent Variable.
subset of some population. A study sample could Study participants are assigned by the researcher
consist of 120 persons 70 years of age or older, into groups, each of which receives one condition
residing in extended care facilities (ECFs) in the of the independent variable. For example, if there
United States. This subset of 120 persons is drawn are 120 study participants and if there are two con-
from the entire population (hundreds of thousands ditions of the independent variable, 60 persons
of people) who are older than 70 years of age and could be assigned to one condition and 60 to the
reside in American ECFs. It is critical that the per- other condition. For an experiment with four con-
sons who make up the sample in an experiment are ditions to the independent variable and with N
representative of the population from which they (overall sample size) ⫽ 120, 30 persons could be
are drawn. Research procedures for making sure assigned to each group.
that the sample is representative of the population As in the previous examples, a one-to-one ratio
are described in Chapter 31. is usually used in assigning participants to groups,
but not always. For example, a researcher might
There Is One Categorical have a good reason for assigning more participants
Independent Variable. to one group than to another. Regardless of ratio,
randomization is absolutely essential. In a one-to-
There are two kinds of variables: categorical and one ratio, randomization means that every partici-
continuous. The conditions of a categorical vari- pant in the sample will have an equal chance of
able are qualitatively different from each other, being in each of the groups. Procedures for ran-
whereas the conditions of a continuous variable domization are presented in Chapter 31.
are quantitatively different from each other. Type Randomization ensures that there is no bias that
of intervention is an example of a categorical vari- favors one group over the other. If randomization
able. Neurodevelopmental therapy is one type of is done properly, then the rules of chance make it
intervention, and biomechanically based therapy is probable that the groups are equivalent to each
a different kind of intervention. Each of these other at the beginning of the experiment.
interventions could be a condition of a categorical Each of the conditions of the independent vari-
variable; in this case the categorical variable would able might consist of alternative, previously
have two, qualitatively different conditions. untested interventions or stimuli (e.g., comparing a
Another example of a categorical variable could parallel group task to a project group task). A sec-
have three conditions: authoritarian group struc-
ture, laissez-faire group structure, and democratic
group structure.
In contrast, continuous variables involve quan- A common error is to say “independent vari-
titative, ordered relationships. Measured height is ables” when referring to the multiple conditions
a continuous variable. The score on a sensory inte- of a single independent variable. Remember: a
gration test is also a continuous variable. variable must vary.
A categorical variable is an independent vari-
Chapter 7 Group Comparison Studies: Quantitative Research Designs 67
participant in the rote exercise condition to “make

In her 2000 Eleanor Clarke Slagle lecture, Dr.
Margo Holm reasoned that embedding exercise cookies.” Also, a subject in the image-based con-
within everyday tasks (what I term occupationally dition might become distracted, fail to think about
embedded exercise) is one of the few ideas in stirring cookies, and just do the stirring as a rote
occupational therapy that has been supported by a exercise. Both of these situations exemplify bleed-
systematic review of multiple experiments. Dr. ing: a participant assigned to one condition actu-
Holm (2000) cites this area of inquiry as an ally experiences some or all of the comparison
example of the highest level of research evidence. condition.
In addition to bleeding, there are other prob-
lems of fidelity. Consider an experiment whereby
ond option is to compare an intervention to a con- each participant is supposed to attend four spe-
trol condition (i.e., the absence of any research- cially designed family education sessions (versus a
induced intervention or stimulus). For example, usual-care condition) and some subjects arrive late
the control group in a study of an occupation- and miss part of the sessions. Here the problem is
based wellness program for middle-aged, over- that some of the subjects simply are not experienc-
weight male professors could be given no special ing what they are supposed to be experiencing. A
intervention, as a way of finding out if the wellness problem of delivery occurs, for instance, when the
program is effective. interventionist fails to spend the required time with
A third option is to compare an intervention or the participant, or forgets to administer all of the
stimulus to an attention-placebo condition. A protocol (e.g., leaving out a component of one of
placebo condition in drug studies involves admin- the family education sessions). A problem of
istration of a nonactive substance (e.g., a pill) that receipt is a failure of the participant to pay atten-
looks, smells, and tastes the same as the active tion to the therapist’s instructions (e.g., a subject is
drug in the study. An attention-placebo condition present at the educational session but is distracted
involves administration of human contact that because of a problem at work). A problem of
mimics the intervention under study, without enactment is a failure of participants to do some-
involving the essence of what is under study. For thing they are supposed to do (e.g., following the
example, in a study of neurodevelopmental ther- advice given in the educational session).
apy (NDT), the children in the attention-placebo Documentation of fidelity is especially important
condition receive the same amount of touching and in studies that involve complex interventions
time with an adult as the children receiving NDT. administered over long periods of time, and in
A technique to ensure equality of attention is yok- studies involving the participant’s generalization
ing, whereby each person in the attention-placebo of learning from one context to application in
group receives the same amount of attention as a another.
specific person in the intervention group. With ref-
erence to the previous NDT study example, this Potentially Confounding Variables
technique would mean that each child in the Are Minimized, and Otherwise
attention-placebo condition would receive the Uncontrollable Events Are Equally
same amount of touching and time as a specific Likely Across the Groups.
child in the NDT group. Fourth, a common type of
comparison condition involves usual care or stan- In an ideal experimental world, the independent
dard care, which is compared to some new type of variable is the only thing affecting the participants
intervention. For ethical reasons, subjects in any of while they are in the experiment. But in the real
these different types of comparison conditions
often receive the intervention condition (if found I (your chapter author) first learned about inter-
effective) after the experiment is completed. vention fidelity from Dr. Laura Gitlin (please see
selected references under her name), at a meeting
The Independent Variable Is of the Center for Research Outcomes and
Administered as Planned. Education (CORE) on a sunny day in Chicago.
As attested by the smiles of the participants,
A study is said to have fidelity if there is docu- learning about experimental research can be fun.
mentation that the independent variable is admin- Dr. Virgil Mathiowetz, who was also in atten-
istered just as planned (Moncher & Prinz, 1991). dance that day, and I later wrote about fidelity
Using the earlier example of a study of occupa- (Nelson & Mathiowetz, 2004). Virgil wrote the
tionally embedded exercise, a problem of fidelity better parts of the paper.
occurs if the research assistant by mistake tells a
world, there are many potentially confounding research, other types of blinding may sometimes
variables (things that could affect the participants be used:
in addition to the conditions of the independent
• The person conducting the randomization can be
variable). Potentially confounding variables are
masked from the subsequent randomization
also sometimes called confounds, extraneous vari-
sequence,
ables, nuisance variables, and sources of error. For
• The person measuring the dependent variable
example, in a study comparing occupationally
(see below) can be masked from knowing who
embedded exercise to rote exercise in older nurs-
has received which condition, and
ing home residents, a nursing aide could interrupt
• The statistician can be masked from knowing
the participant while stirring the cookie dough or
who has received which condition.
the older person could lose attentiveness because
of poor sleep the night before. Therefore, one’s
experimental design incorporates practical strate- There Is One Dependent Variable
gies for minimizing the chances of potentially con- on Which All subjects Are Measured
founding variables. For example, the researcher or Categorized.
can explain the need for privacy to the staff; select
a quiet, out-of-the-way room for the study; and put The dependent variable is the criterion used to
up a do-not-disturb sign. compare the conditions of the independent variable
Despite any researcher’s best efforts, there are to each other. All participants are measured or cat-
always events that cannot be controlled. This is egorized on the dependent variable. For example,
especially true of intervention studies taking for an independent variable involving a compari-
place over many months. For example, a study of son between occupationally embedded exercise
the effects of a multisite, school-year-long, occu- and rote exercise, the dependent variable might be
pation-based fitness program for overweight chil- the number of exercise repetitions (a continuous
dren (versus an attention-placebo control variable that can be measured). Alternatively, the
condition) involves potentially confounding vari- dependent variable in a special skin protection pro-
ables such as differing school-based practices, dif- gram for wheelchair users (compared to a usual-
fering levels of family participation, family and care condition) might be the presence or absence of
community events (positive as well as negative), skin breakdown (a categorical variable).
differing exposures to mass media, differing The method used to collect data on the depend-
opportunities for sports participation, changes ent variable must be valid and reliable. Experi-
resulting from growing older, and illnesses that menters often try to improve reliability and validity
may occur. Experimental design cannot eliminate through a variety of strategies:
all these potentially confounding variables, and
• Measuring each participant several times so that
there are practical limitations even on monitoring
the dependent variable is the mean,
them. The researcher’s goal is that these poten-
• Providing training that exceeds the minimum
tially confounding variables are equally likely to
specified in the measurement protocol,
occur in each of the comparison groups. In other
• Using multiple, simultaneous raters or judges in
words, the researcher tries to ensure that bias does
measuring or judging the dependent variable, and
not occur (where one of the conditions is system-
• Masking (see above) the measurers/judges.
atically favored). Other terms used to refer to a
biasing confounding variable are artifact and In selecting a dependent variable for a study,
sources of systematic error. the researcher has to be especially concerned about
To eliminate certain artifacts, researchers fre- its responsiveness. Responsiveness (also called
quently use masking (also called blinding). In a sensitivity in the nonmedical literature) is the
“double-blind” study, neither the participants (one capacity of the dependent variable to show small
blind) nor the researcher (the other blind) is aware but meaningful increments of change over time.
of which groups receive which condition of the For example, a nonresponsive measure might be a
independent variable. This is usually done through three-point scale for measuring self-care: inde-
use of a placebo. However, total double-blinds are pendent, partly dependent, and dependent. A
impossible in most occupational therapy studies, research participant can make much meaningful
because the participant is usually aware of what is progress in taking care of oneself, but the person’s
happening and the administrators of the independ- progress does not show up in the measure. For
ent variable (often occupational therapists) are example, if a female participant starts out with the
professionally responsible for being aware of the ability to do very little self-care and ends with the
service they are providing. In occupational therapy ability to take care of herself except for one or two
The term sensitivity is a good example of a

The Experimental Hypothesis
research term meaning two totally different Tested Is the Probability of a Causal
things. In the medical literature, sensitivity refers Effect of the Independent Variable
to predictive validity of a diagnostic test. In the on the Dependent Variable Within
literature of psychology, sensitivity often refers the Population.
to the capacity of a measurement to reflect
change in participants over time (what is called The first thing about this definition to notice is the
responsiveness in the medical literature). word prediction. It is a hallmark of quantitative
research that all the variables under study and their
possible relationships, one way or the other, are
things, her rating would not change since she is determined in advance. A hypothesis is a prediction
still partly dependent. about the relationship between variables within a
Another kind of nonresponsive measure is one population. The end result of an experiment can
that has ceiling or floor effects. A ceiling effect have only two possibilities: support for this hypoth-
occurs where the sample’s scores are already so esis, or a lack of support. Consider the following
high on the scale that little improvement is possi- experimental hypothesis: Occupationally embed-
ble. A floor effect occurs when the opposite is the ded exercise elicits more exercise repetitions than
case. For example, adolescents with conduct disor- rote exercise in nursing home residents aged 65
ders are unlikely to demonstrate a reduction in years or more. Even though the present tense (elic-
aggressive episodes in a one-day study if the typi- its, not will elicit) is used, a prediction is implied,
cal participant engages in only a few aggressive because the hypothesis is stated before the
incidents per week. research starts. The independent variable has two
Ideally, the experimenter–researcher uses a conditions: occupationally embedded exercise ver-
method of capturing the dependent variable that is sus a control condition. The dependent variable
recognized as the criterion standard (sometimes consists of exercise repetitions. And the sample is
called the gold standard) in the field. Thus in drawn from the population of nursing home resi-
selecting a data collection procedure for the dents 65 years of age or older. The type of rela-
dependent variable, the investigator asks if there is tionship between the variables under study is the
a consensus in the field hypothesized effect of
concerning the best way the independent variable
to measure or categorize Type I error involves report- on the dependent vari-
that variable. For exam- able. Probable causality
ing a relationship when there is involved: the inde-
ple, 5-year survival rate
is a criterion standard for really is no relationship. pendent variable is the
interventions addressing hypothesized cause, and
certain types of cancers. the dependent variable
However, occupational therapy researchers fre- is the hypothesized effect. Quantitative research
quently address research questions that have not deals with probability, not with certainty or
been studied adequately in the past. An excellent truth. See Chapter 17 on statistics to find out
way to address this problem is for the designer of how probability is used to assess research out-
an experiment to ask experts in the field to confirm comes.
the validity of a dependent variable. At the same The logic of the basic experiment is as
time, the investigator can ask the experts to iden- follows:
tify how much of a difference between comparison
groups at the end of the study is required to be • If groups assigned to conditions of the independ-
meaningful. For example, the investigator might ent variable are probably equal at the start of the
ask whether a 10% reduction in rehospitalization experiment (the chance processes of randomiza-
rates is a meaningful difference in a study of tion ensure probable equivalence),
interventions for participants with chronic schizo- • If groups are treated the same except for the inde-
phrenia. pendent variable, and
Knowing how much change one can expect to • If there are differences between groups at the end
see in the dependent variable can be factored into of the experiment, then the differences were
a decision about the number of subjects needed in probably caused by the independent variable act-
the study. There is a calculation called power ing on the dependent variable. This is the basis
analysis that allows one to make this determina- for experimental design’s claim to study probable
tion. It is discussed in detail in Chapter 17. causality.
S S
R X O R X O
R O R Y O
S = Start R Z O
R = Randomization R 'O
X = Condition of independent variable
O = Observation (measurement or categorization used S = Start
for dependent variable) R = Randomization
Figure 7.1 Basic experimental design, with true X = An intervention
Y = A different intervention or usual care
control group, one intervention, and no pretest.
Z = A different intervention, usual care, or attention-
After the sample is randomized (R) into two placebo
groups, one group receives an intervention O = Observation (measurement or categorization used
(X) while the control group does not. Both for dependent variable)
groups are measured or categorized (O) in
the same way. *The symbolic format for these Figure 7.3 Basic experimental design (pretest–
figures is adapted from Campbell and Stanley posttest) comparing three conditions of the
(1963). independent variable to each other and to a
control group. After the sample is randomized
(R) into four groups, three groups receive X, Y,
Z respectively while the fourth receives nothing.
The following is another experimental hypoth- All groups are measured or categorized (O) in
esis: In nursing home residents 65 years of age or the same way.
older, there are differences among occupationally
embedded exercise, imagery-based exercise, and
rote exercise in terms of exercise repetitions. Here,
the independent variable has three conditions that Evaluating the Validity of
are compared with each other. This is also an
example of a nondirectional hypothesis. A nondi- Group Comparison Designs
rectional hypothesis does not predict which condi-
tion will end up with the superior outcome; it Type I Error
simply predicts that one of the conditions will be
superior. Still implied is probable causality. In con- Type I error involves reporting a relationship when
trast, a directional hypothesis is clear as to which there really is no relationship (Rosenthal &
group is predicted to have higher scores (or higher Rosnow, 1991). The reader of the research report
proportions in the case of categorical variables). can never be absolutely certain that a type I error
Figures 7.1 to 7.3 illustrate three basic experimen- has been made. However, the reader of the
tal designs. research report may reason that weaknesses in the
research design make the chances of a type I error
unacceptably large.
To understand this point, consider the following
S nonexperimental research design that is highly
R X O
prone to type I error: the pretest–posttest no-
R Y O
control group design (Figure 7.4). In this design, a
single group of participants receive a test, an inter-
S = Start
R = Randomization vention, and another test after the intervention.
X = Condition of independent variable Consider the statement: Manipulation of the spine
Y = A different intervention, usual care, or attention-
placebo
O = Observation (measurement or categorization used Campbell, D. T., & Stanley, J. C. (1963).
for dependent variable)
Experimental and quasi-experimental designs for
Figure 7.2 Basic experimental design comparing research. Chicago: Rand McNally is the classic
two conditions of the independent variable, and reference concerning type I error and how to pre-
no pretest. After the sample is randomized (R) vent it. It is a remarkably brief book filled with
into two groups, one group receives an inter- an intense logic that many a graduate student has
vention (X) while the other group receives (a) a practically memorized. It is highly recommended
different intervention, (b) usual care, or (c) an by this author as a resource for understanding
attention-placebo (Y). Both groups are meas- group comparison designs.
ured or categorized (O) in the same way.
decreases pain in adult males with acute lumbar S

pain secondary to lifting-related injuries. X O
The researcher following this design simply: Y O
• Selects persons with acute lumbar pain second- S = Start

ary to lifting-related injuries, X = An intervention
• Measures the level of pain (a pretest), Y = A different intervention, usual care, or attention-
placebo
• Administers a series of manipulation interven-
O = Observation (measurement or categorization used
tions, for dependent variable)
• Measures the level of pain once again (a
posttest), and Figure 7.5 Nonrandomized comparison group
• Compares the pretest to the posttest. design (no pretest). The subjects in one natu-
rally occurring group receive one intervention
If pain decreases significantly from the pretest (X), and the subjects in a different naturally
to the posttest, the researcher concludes that the occurring group receive a different intervention.
spinal manipulation is effective. The problem with A measurement or categorization is then done
this researcher’s logic is that the design does not (O). It is important to note that this design is not
protect against type I error: it is quite likely (no recommended except for pilot testing in
one knows exactly how likely) that the manipula- advance of future research.
tion has nothing to do with the improvement in
pain levels. The following potentially confounding wrist range of motion for persons with carpal tun-
variables unrelated to manipulation may have nel syndrome. Instead of randomly assigning par-
caused the improvement: ticipants to groups, the researcher assigns one
naturally occurring group (e.g., patients at one
• Acutely ill persons might recover without any hand clinic) to receive technique A and assigns
intervention through natural healing. another naturally occurring group (e.g., patients in
• The research participants might have received a different hand clinic) to receive technique B. At
other interventions between the pretest and the the end of 2 weeks of being splinted, the patients
posttest. in the first clinic who received technique A actu-
• There might have been something about being ally have greater wrist range of motion than the
tested twice on the pain scale that resulted in patients in the other clinic who received technique
lower scores the second time around. B. Consequently, the researcher reports the superi-
There are many possible explanations for the ority of splinting technique A.
observed change, and so the bottom line is that we However, the chances of a type I error are unac-
do not know anything more than we knew before ceptably high. The problem is that the patients at
the research study. Possible sources of type I error the first clinic might have been different from, or
are not controlled in this study design. nonequivalent to, the patients in the second clinic
Another nonexperimental research design often (even before splinting, they might have been less
highly at risk for a type I error is the nonrandom- severely disabled). Also, they might have had spe-
ized control group design (Figure 7.5). Consider cial opportunities for progress, based on the qual-
the following hypothesis: Splinting technique A is ity of the clinic’s staff or on demographic factors,
superior to splinting technique B in improving or they might have had less risk for re-injury. Even
if the researcher uses a pretest–posttest nonran-
domized control group design and finds that the
S two groups do not differ on the baseline, there is
O X O still a high chance of a type I error. Patients at the
first clinic might have a greater potential for
S = Start change than the patients at the second clinic. For
X = An intervention example, it is possible that the range- of-motion
O = Observation (measurement or categorization used scores are equal at pretest even though the patients
for dependent variable) at the first clinic have relatively recent, acute
Figure 7.4 Pretest–posttest design with no con-
injuries whereas the patients at the second clinic
trol or comparison group. After a pretest, an have chronic injuries unlikely to change in a short
intervention is given, to be followed by a period of time.
posttest. It is important to note that this design The basic experiment was invented to reduce
is not recommended except for pilot testing in the chances of a type I error to some small, con-
advance of future research. trollable amount. Randomization ensures that the
comparison groups are probably equivalent at the • Selecting 20 clients for the study,
outset of the study. A comparison condition bal- • Randomizing them to groups,
ances potentially confounding variables (thereby • Making sure of high fidelity within groups and
preventing systematic bias). For example, both no bleeding between groups,
groups in a basic experiment are equally likely to • Ensuring that the measurer remains masked (i.e.,
be exposed to the same processes of healing, mat- unaware of the intervention condition of the par-
uration, extraneous interventions, repeated testing, ticipants), and
and other factors. The basic experiment was devel- • Following the plan for statistical analysis.
oped within the tradition of scientific skepticism: Following the logic discussed earlier, this
the biggest danger to scientific development is to design allows little chance of a type I error,
report a relationship when one does not exist. because neither group is favored by biasing vari-
ables. However, the chances of a type II error are
Type II Error so great that it is very unlikely that a difference
will be found between groups even if the type of
Type II error is the failure to find and report a rela-
intervention under study is actually effective. This
tionship when the relationship actually exists
is because:
(Rosenthal & Rosnow, 1991). Like type I error, the
reader of a research report can never be certain that • The intervention is not intensive enough to pro-
a type II error has occurred. But certain research duce a large effect,
designs and situations are prone to increasing the • The number of subjects in each group is small,
risks of type II error to unacceptable levels. and
Recommendations by Ottenbacher (1995) and • The quality of the measure of the dependent vari-
Ottenbacher and Maas (1998) for identifying risks able is unknown and may not be reliable or sen-
of type II error and for how to avoid type II error sitive.
are particularly recommended for those who wish To decrease the chances of a type II error, the
to understand more about this issue. researcher can:
Sources of type II error include:
• Increase the sample size by extending the study
• A small sample size, over time or by finding additional sites for the
• A subtle (yet real) independent variable, research,
• Much dispersion on the dependent variable • Increase the effect of the independent variable by
within groups: providing therapy more often per week and over
• Dispersion due to individual differences a longer period of time, and
among subjects, or • Decrease dispersion due to measurement error by
• Dispersion due to measurement error, and using a test of organizational ability that has been
• A stringent or nonrobust statistical test of the demonstrated to be accurate (reliable) yet sensi-
hypothesis. tive to change (responsive) in past studies.
Although the basic experiment provides the In fact, all three strategies are warranted in this
best protection against type I error, it sometimes case to avoid type II error.
provides poor protection against type II error. There are several strategies to decrease disper-
Consider a study with the following hypothesis: an sion on the dependent variable that is due to
occupational therapy pro- individual differences.
gram will increase the The most straightfor-
organizational skills of Type II error is the failure ward way to decrease
clients with traumatic to find and report a dispersion is to select
brain injury (vs. an atten- participants who are
tion-control condition). relationship when the homogeneous (similar to
The occupational therapy relationship actually exists. each other, as opposed
program takes place once to heterogeneous). In
a week for 4 weeks, and our example of persons
the measurement of organizational ability involves with brain injury, the researcher could decide in
a nonstandardized rating system whereby a thera- advance that research participants are included
pist makes judgments concerning the client’s orga- only if at a similar stage of recovery.
nizational ability while shopping from a list in the Another way to decrease dispersion is to use a
supermarket. The researcher has ensured a lack of pretest–posttest experimental design, as opposed
bias by: to a posttest-only experimental design. Figures 7.6
S Sometimes it is impossible to do a pretest. For

R O X O example, in a study of occupationally embedded
R O O exercise, it is impossible to do a pretest of repeti-
tions because the repetitions can be counted only
S = Start in the simultaneous context of the independent
R = Randomization variable (e.g., while actually stirring). In this case,
some other variable can be measured in advance
for dependent variable) that is probably associated with the dependent
variable. This kind of variable is called a covariate.
Figure 7.6 Basic experimental design (pretest– For example, in a study in which the dependent
posttest), with true control group and one inter- variable involves stirring repetitions, a likely
vention. After the sample is randomized (R) into covariate is grip strength. Individual differences
two groups, both groups are pretested (O to reflecting grip strength can be controlled through
left) in the same way. Next, one group receives
analysis of covariance. The point here is that a
an intervention (X) while the control group does
not. Finally both groups are posttested (O to covariate, properly chosen in advance of the study,
right). can reduce individual differences and thereby
reduce dispersion on the dependent variable. The
result is less of a chance of a type II error, making
it possible to demonstrate that the independent
and 7.7 illustrate two such designs. To use the variable probably makes a real difference. Figures
same example again, a pretest of organization abil- 7.8 and 7.9 (on p. 74) illustrate two such designs.
ity administered before the independent variable Yet another strategy to decrease individual dif-
would permit statistical procedures that control for ferences is to use a randomized matched subjects
individual differences in the final analysis. The design (Figure 7.10 on p. 74); it is called a ran-
dependent variable is adjusted, correcting for the domized matched pairs design where there are two
individual differences. This is done in an unbiased conditions of the independent variable. This design
way, so that neither group being compared to each is also referred to as a type of randomized block
other receives a special advantage. Several differ- design. In this instance, participants are matched to
ent statistical procedures are capable of making each other in advance on some relevant variable,
this adjustment. and then they are randomly assigned to groups
(each of which receives a different condition of the
independent variable). Consider the hypothesis:
community-dwelling persons with dementia who
S
R O X O receive added home-based occupational therapy
are less likely to enter extended care facilities than
R O Y O
community-dwelling persons with dementia
R O Z O
R O O
S
R C X O
S = Start
R = Randomization R C Y O
X = An intervention
Y = A different intervention, usual care, or attention- S = Start
placebo R = Randomization
Z = A different intervention, usual care, or attention- C = Covariate(s) reflecting variable(s) associated with
placebo the dependent variable
O = Observation (measurement or categorization used X = Condition of independent variable
for dependent variable) Y = A different intervention, usual care, or attention-
placebo
Figure 7.7 Basic experimental design (pretest– O = Observation (measurement or categorization used
posttest) comparing three conditions of the for dependent variable)
independent variable to each other and to a
control group. After the sample is randomized Figure 7.8 Experimental design comparing two
(R) into four groups, groups are pretested (O conditions of the independent variable, with
to left) in the same way. Next, three groups planned covariate(s). After the sample is ran-
receive X, Y, Z respectively while the fourth domized (R) into two groups, the covariate (C)
receives nothing. Finally, all groups are meas- is measured. Next, each group receives inter-
ured or categorized (O to right) in the same ventions X or Y. Both groups are measured or
way. categorized (O) in the same way.
S S
R X O R X O
C M
R Y O R Y O
S = Start S = Start
R = Randomization M = Matching
C = Covariate reflecting variable associated with the R = Randomization
dependent variable X = Condition of independent variable
X = An intervention Y = A different intervention, usual care, or attention-
Y = A different intervention, usual care, or attention- placebo
placebo O = Observation (measurement or categorization used
O = Observation (measurement or categorization for dependent variable)
used for dependent variable)
Figure 7.10 Matched subjects (matched pairs)
Figure 7.9 Experimental design comparing two experimental design comparing two conditions
conditions of the independent variable, with a of the independent variable. First, subjects are
planned covariate available prior to randomiza- paired based on some relevant similarity.
tion. After the sample is randomized (R), each Randomization (R) is done in blocks of two
group receives X or Y. Both groups are meas- (randomized block design). Next, one group
ured or categorized (O) in the same way. receives X or Y. Both groups are measured or
categorized (O) in the same way.
receiving usual care. Participants can be paired to
each other in terms of level of dementia, and then preventing type I error. But special adjustments
randomly assigned to either the occupational ther- must often be made to prevent type II error. Table
apy condition or to usual care. The statistical 7.1 is a way of depicting how likely a type II error
analysis for this design involves comparisons of is in occupational therapy research in comparison
persons who are similar to each other, thereby to research in some other fields. But we are not
reducing unsystematic error due to individual dif- alone. Researchers in psychotherapy, special
ferences. education, and many branches of medicine must
In summary of our discussion of type I and type overcome the same disadvantages. Statistical pre-
II error, the basic experiment is a great means of sentations in Chapter 17 discuss the importance of
Table 7.1 Inherent Dangers of Type II Error in Occupational Therapy Research

Common Causes of Type II Error Laboratory Animal Studies Occupational Therapy Studies
Small sample size Animals are readily available at Persons with disabilities are small
reasonable cost. minorities of the population, often with
major health and financial problems.
Dispersion due to Healthy animals are highly Persons with disabilities vary from each
individual differences homogeneous (often other more than the general popu-
genetically matched). lation; all persons respond somewhat
differently to occupational forms.
Dispersion due to Measurement systems in most Most measurement systems in
measurement error animal research is highly occupational therapy generate high
precise. levels of error.
Dispersion due to random Highly controlled environments are People’s lives are full of chance events
events typical. that cannot be eliminated in
multisession studies.
Robustness of the Biological interventions are often A single short-term occupation usually
independent variable powerful; subtle effects can be has little long-term effects; ethical
investigated after robust effects considerations preclude risky
are demonstrated. interventions.
Powerful statistical Biostatisticians are familiar with Much occupational therapy research in
procedures techniques to reduce the past has not employed
unsystematic error. sophisticated procedures to reduce
unsystematic error.
reporting the effect size of a comparison, not just takes place (as much as possible) under everyday
the p value. What is important to note here is that conditions. Instead of studying perception in a
an experiment reporting a substantial but statisti- controlled lab, it can be studied in the participant’s
cally nonsignificant difference must be interpreted home. The problem in this instance is that natura-
with great caution: the chances of a type II error listic settings involve potentially confounding vari-
might be high. ables (the ring of the telephone or the voices of
others in the home) that threaten to cause problems
of internal validity. For this reason, careful investi-
External Validity gators may gather preliminary evidence under con-
trolled laboratory conditions, and later in
External validity has to do with generalizability, subsequent experiments show that the effect also
whether the results of a research project can be can be demonstrated in the everyday world. As for
applied to the nonresearch environment (the real artificiality of the dependent variable, unobtrusive
world out there) (Campbell & Stanley, 1963). measurements can sometimes be used. For exam-
Unfortunately, it is possible that an experiment can ple, there are motion detection systems that do not
avoid making type I and type II errors (i.e., it could involve strapping objects to participants’ arms.
have internal validity) and yet have no external Unrepresentativeness of the sample can be
validity. For example, a laboratory study of how addressed in two ways: subject selection proce-
people interact under experimental conditions dures and dealing with study dropouts. The basic
might have excellent internal validity but might idea of this strategy is to ensure that the partici-
not reflect how people interact when not under pants in the study do not represent some special
close examination. subpopulation that responds differently to the
External validity can be threatened by two main independent variable from the population that is
factors: supposedly under study. Examples of unrepresen-
tative samples are studies supposedly of U.S. nurs-
• Artificiality in the experimental environment
ing home residents where the sample is 80% male,
(things are so different that subjects behave dif-
or studies of children with autism where the sam-
ferently from how they do in everyday life), and
ple is 80% female. A proper experimental write-up
• Unrepresentative samples (a mismatch exists
details the relevant demographic characteristics of
between the intended population and the actual
the sample so that the reader can decide to whom
sample that is studied).
the study can be generalized. There is nothing
Artificiality can apply to the independent vari- wrong with a study of male nursing home residents
able or the dependent variable. Lab coats, unusual as long as the researcher does not say that the
settings, excessively complex informed consent results can be generalized to all nursing home res-
procedures, and unusual equipment can threaten idents.
external validity. For example, in studies of per- Dropouts are particularly likely in long-term
ception, participants asked to discriminate among experiments: participants might lose interest over
carefully projected images might make discrimina- time, become ill in ways that are not relevant to
tions that they cannot the experiment, or move
make in the blooming away. The danger to
and buzzing confusion of External validity has to do external validity is that
everyday life, where they with generalizability, whether the participants who
focus on individually drop out might be differ-
meaningful things. An the results of a research ent from the participants
example of artificiality in project can be applied to the who remain in the study
measuring the dependent (e.g., dropping out might
variable can be seen in nonresearch environment be a way of avoiding
an electrogoniometer (the real world out there). unpleasant side effects
strapped onto the partici- of the intervention under
pant’s arm, to measure study, or dropping out
elbow flexion. The participant might reach differ- might reflect special frailty). Strategies to prevent
ently when wearing this bulky contraption (e.g., he dropouts include careful explanation of the study
or she might lean forward more than usual, thereby in advance, frequent positive communications over
decreasing elbow flexion artificially). the course of the study, and due consideration for
To decrease artificiality, the researcher can the inconveniences participants experience (e.g.,
strive to design the independent variable so that it lost time, transportation, etc.). The experimental
plan should include procedures for recording and S

reporting any dropouts; ideally dropouts can be R X O X O X O
compared to non-dropouts to see if differences R Y O Y O Y O
exist.
S = Start
R = Randomization
Variations on X = An intervention
Randomized Designs placebo
We have already discussed three variations on
basic posttest-only experimental design: Figure 7.11 Experimental design (no pretest)
comparing two conditions of the independent
• Pretest–posttest experimental design, variable with repeated observations (repeated
• Use of a covariate to adjust for individual differ- measures). After the sample is randomized (R)
ences, and into two groups, each group receives X or Y.
• Randomized matched subjects design. Both groups are observed (O) in the same way.
Observations (O) occurring before the desig-
There are other variations that can be helpful, nated primary end point are called interim
given particular research problems. They are dis- measures. Observations (O) occurring after the
cussed below. primary endpoint are called post–posttests.
Interim Repeated Measures, Post–Post intervention to reduce falls in community-dwelling

Tests, and Long-Term Follow-up Tests older persons (versus an attention-control condi-
tion), the Institutional Review Board wants to
Sometimes it is desirable to measure the outcome rule out the possibility that the daily walking
repeatedly over the course of an experiment might actually increase the rate of falling. These
(Fleiss, 1986). For example, in a study in which are unique statistical procedures for analyzing the
sensory integrative therapy is administered to chil- dependent variable when there are repeated meas-
dren with learning disabilities over a full year ures. The more statistical tests that are done, the
(compared to a true control group), the researcher more likely it is that some of those tests appear to
might measure school achievement each quarter. be statistically significant by chance alone.
In this way, the researcher can gain insight as to The difference between post-posttests and
quarter-by-quarter rates of change. It is possible long-term follow-up tests is that long-term follow-
that there might be little difference between groups up test occurs after a period of no intervention. To
at the end of the first quarter, but a large difference take the example of the year-long program of sen-
at the end of the second quarter. This provides sory integration, a long-term follow-up test can
important information concerning the duration take place a year after the conclusion of interven-
required for this intervention to have an effect. The tion and the posttest. The long-term follow-up
researcher using repeated measures should be clear test sheds light as to whether the effects of the
in advance as to the primary (most important) end- intervention are still detectable a year after the
point, the measurement that will be used to test the withdrawal of the intervention. Figure 7.12 illus-
main hypothesis of the study. For example, in the trates an experimental design with long-term fol-
study of sensory integration, the primary endpoint low ups.
might be the final quarter’s measurement of school
performance, at the end of the school year. In this Multiple Dependent Variables
case, the earlier quarter-by-quarter measures are Tested in Reference to the Same
interim measures. However, if the researcher des- Independent Variable
ignates the second-quarter measurement of school
performance as the primary endpoint, then the Researchers frequently want to know whether an
third- and fourth-quarter measurements are called intervention affects several dependent variables
post–posttests. Figure 7.11 illustrates a repeated (Stevens, 1986). For example, in a study of the
measures design. effects of added levels of occupational therapy in
Sometimes the organizations responsible for comparison to usual care in subacute rehabilitation
the protection of human subjects require interim patients, the researcher might want to know if the
measurements (Friedman, Furberg, & DeMets, added sessions affect three separate outcomes:
1998). For example, in a study of a daily-walking objectively measured self-care, discharge out-
S S
R O X O O O R X OPQ
R O Y O O O R OPQ
R O Y O O O
S = Start
R = Randomization
S = Start X = An intervention
R = Randomization O = Observation (a dependent variable)
P = Observation (another dependent variable)
Q = Observation (yet another dependent variable)
placebo
Z = A different intervention, usual care, or attention- Figure 7.13 Experimental design (posttest-only),
placebo with three dependent variables (O, P, Q) and a
O = Observation (measurement or categorization used true control group. After the sample is random-
ized (R) into two groups, one group receives an
Figure 7.12 Experimental design (pretest– intervention (X) while the control group does
posttest) comparing three conditions of the not. Both groups are measured or categorized
independent variable with repeated observa- (O, P, Q) in the same way.
tions (repeated measures) and two long-term
follow-ups. After the sample is randomized (R)
into three groups, each group is pretested. the independent variables, to be discussed later in
Next, each group receives X, Y, or Z. Groups the section on crossover designs.
are observed (O) in the same way immediately Another problem encountered by having sev-
after the intervention and at two additional
eral outcomes is multiplicity: the increase in type
points in time.
I error due to multiple statistical tests on the
same participants. The more tests that are done,
come, and patients’ self-reports of goal achieve- by chance alone it is likely that some of those
ment. There is a statistically testable hypothesis for tests will appear to be statistically significant.
each of the dependent variables, and there is also a Unless special care is taken, the chances of a
way to do a single test of type I error increase
the effectiveness of the for multiple statistical
intervention across all The more statistical tests tests, especially if these
three dependent variables that are done, the more tests are done on the
(multivariate analysis of same participants. For
variance). Figure 7.13 likely it is that some of instance one might have
illustrates an experimen- those tests appear to be 20 dependent variables,
tal design with three if one wishes to look at
dependent variables. statistically significant by each aspect of self-care
The advantage of hav- chance alone. (tooth-brushing, hair-
ing multiple dependent combing, buttoning,
variables is obvious. It is etc.) If each of these 20
great to know about all three outcomes without variables is tested independently at the .05 level,
having to do three studies. A possible problem, the chances are that at least 1 of the 20 tests will
however, is that measurement of the first depend- be found significant even if the independent vari-
ent variable has an effect on the second dependent able had no effect at all (a type I error). Therefore
variable. A person could have fatigue after the first investigators use special corrective procedures
measurement, or the first measurement might sen- discussed in Chapter 17. It is also good practice
sitize the person to the second measurement (a that a researcher interested in multiple outcomes
measurement of vestibular ability might artificially clearly identify and justify a primary dependent
inflate a subsequent measure of alertness). A variable and discriminate in statistical procedures
design strategy to deal with this problem is coun- between analysis of the primary dependent vari-
terbalancing of the dependent variables (Figure able and secondary dependent variables.
7.14 on p. 78), where the different variables are
experienced in different sequences, so each
Mechanisms of Change
dependent variable sometimes occurs early in the
testing protocol and sometimes late. Note that this A special argument in favor of multiple dependent
counterbalancing of the dependent variables variables occurs when the researcher theorizes that
should not be confused with counterbalancing of the intervention under study works in a chain reac-
S S
R X OPQ R OP X OP X OP
R OPQ R OP Y OP Y OP
R X PQO S= Start
R= Randomization
R PQO X= Condition of independent variable
R X QOP Y= A different intervention, usual care, or
attention-placebo
R QOP O = Observation (primary dependent variable)
P = Observation (measurement indicating theory-
S = Start based mechanism of change)
R = Randomization
X = An intervention
Figure 7.15 Experimental design comparing two
O = Observation (a dependent variable) interventions, with pretest and interim repeated
P = Observation (another dependent variable) measure on primary dependent variable (O) as
Q = Observation (yet another dependent variable) well as on a measure indicating a theory-based
mechanism of change (P).
Figure 7.14 Experimental design (posttest-only),
with true control group and three dependent
variables (O, P, Q) that are counterbalanced the researcher could document that the therapists
according to a Latin Square. After the sample is
administering the two types of intervention actu-
randomized (R) into six groups, three groups
receives an intervention (X) while three groups ally follow the intended protocol, with no bleeding
serve as controls. All groups are measured or from condition to condition and with adherence to
categorized on O, P, and Q, but in different “doses” (i.e., amount of intervention) called for in
sequences, so that two groups receive O first, the protocol. Another example is to compare sta-
two receive first, and two receive Q first. The tistically the amount of time spent with subjects in
alternative is to assign subjects to all possible an attention-control group to the amount of time
sequences of the three measurements (which spent with intervention subjects. The researcher in
would result in 12 groups). this instance generates a methodological hypothe-
sis (a test of the validity of the research proce-
tion style, in which first one aspect of the person is dures).
affected, which in turn influences some other
aspect (Gitlin et al., 2000). For example, consider Completely Randomized Factorial Designs
a comparison between a client-centered approach Completely randomized factorial designs have
to occupational therapy and an impairment- more than one independent variable; otherwise
reduction approach, in which the primary depend- they resemble basic experiments. For example,
ent variable is functional outcome. The researcher a researcher may want to test the effects of a
may theorize that the reason the former approach home safety program on falls prevention in older,
is superior is that it increases the patient’s sense of community-dwelling persons, while also testing
volition. Therefore, the researcher not only meas-
ures functional outcome but also measures voli-
tion. Depending on the researcher’s theory,
S
volition might be measured at the midpoint of the R O X Px X O
study, at the end of the study when functional out-
R O Y Py Y O
come is measured, or at multiple points. The meas-
urement of hypothesized mechanisms of change S = Start
strengthens the interpretation of the results and R = Randomization
contributes to theory confirmation. Figure 7.15 X = An intervention
illustrates such a design. Y = A different intervention
O = Observation (primary dependent variable)
Px = Measure of fidelity of intervention X
Tests of Fidelity Py = Measure of fidelity of intervention Y
A desirable feature of research design is to conduct
Figure 7.16 Experimental design (pretest–
quantitative tests for intervention fidelity posttest) comparing two interventions, with an
(Lichstein, Riedel, & Grieve, 1994). Figure 7.16 interim measure of the degree to which the
illustrates this type of design. In the comparison conditions of the independent variable were
between the client-centered and impairment- administered as called for by the research pro-
reduction approaches to therapy (discussed above), tocol (Px and Py ).
the effects of a lower extremity strengthening pro- S

gram on falls prevention in the same population. In R O XY1 O
a completely randomized factorial design, the R O XY2 O
researcher randomly assigns participants to one of R O X O
four groups:
R O Y1 O
• A group that receives the lower extremity R O Y2 O
strengthening program only,
R O O
• A group that receives the home safety program
only,
• A group that receives both interventions, and S = Start
• A group that receives attention-control only. R = Randomization
X = An intervention (one of two conditions of an
In factorial design, both interventions can be independent variable-the other condition is
tested, and a special bonus is that the interaction of control)
Y1 = An intervention of type Y (one of three conditions
the two interventions can be studied. An interac-
of an independent variable- the other conditions
tion occurs when the effects of one intervention are Y2 and control)
depend on (i.e., augment or decrease) the effects of Y2 = An intervention of type Y (one of three conditions
the other intervention. For example, one kind of of an independent variable- the other conditions
interaction is that the home safety program is are Y1 and control)
effective only when combined with the lower O = Observation (a dependent variable)
extremity strengthening program, such that the Figure 7.17 Completely randomized 2 ⫻ 3 facto-
two interventions together are more effective rial design (pretest–posttest), with true control
than the lower extremity strengthening program condition. After randomization (R) and the
alone. pretest (O), one group receives a combination
When two independent variables each have two of two interventions (X and Y1); the next group
conditions (as in our falls prevention example), we receives a combination of X and Y2; three
call it a 2 ⫻ 2 factorial design. If one of the inde- groups receive a single intervention (respec-
pendent variables has three conditions (e.g., lower tively X, Y1, or Y2); and one group serves as a
true control. The posttest follows.
extremity weight training versus Tai Chi training
vs. attention-placebo) and the other has two condi-
tions, then we call it a 2 ⫻ 3 factorial design organismic variable, a nonmanipulated variable, or
(Figure 7.17). In this case, subjects are randomly a variable consisting of preexisting conditions.
assigned to six groups. If there are three independ- Basically, this kind of variable cannot be randomly
ent variables (let us say that we are adding a assigned (e.g., you cannot randomly assign some
vision-training program to the 2 ⫻ 3 design, then people to the older group and others to the younger
we call it a 2 ⫻ 2 ⫻ 3 factorial design. Here par- group). Sex is a commonly studied organismic
ticipants are randomly assigned to 12 groups. variable. Consider a study or men and women
Some interesting interactions can be studied in wherein the randomized independent variable is a
such an instance. Still another factoral design (2 ⫻ parallel group (in which each person completes a
2 ⫻ 2) is shown in Figure 7.18 on p. 80. Another task in the presence of others) versus a project
advantage of factorial designs is that statistical group (in which each person works together on a
analysis (through analysis of variance) often shared, common project). The dependent variable
reduces error, thus tending to prevent type II error. in this study is a measure of nonverbal socializa-
The main problems with factorial design are that tion (e.g., the frequency that participants make eye
many more subjects are needed to fill up all those contact). The researcher recruits an equal number
groups, and there are many more things that can go of men and women. A positive design feature is to
wrong in a complex factorial design than in a rela- ensure that the men and women match up well to
tively straightforward basic experiment. each other on potentially confounding variables,
such as age and socioeconomic status. Next, the
researcher assigns half the men to the parallel con-
Randomized Treatments by Levels Design dition and half to the project condition, and pro-
A different kind of factorial design involves a ran- ceeds to assign half the women to the parallel
domized independent variable along with another condition and half to the project condition. This
independent variable that reflects two or more design permits the study of the interaction of sex
types of persons (Figure 7.19 on p. 80). This kind and parallel/ project group status. For example, it
of independent variable is sometimes called an could be found that women interact nonverbally
S S
R C X1Y1Z1 O
R O X O
R C X2Y1Z1 O T1 M
R O Y O
R C X1Y2Z1 O
R O X O
R C X2Y2Z1 O T2 M
R O Y O
R C X1Y1Z2 O
R C X2Y1Z2 O T1 = One type of person (e.g., persons with a specific
health problem, or persons of one gender)
R C X1Y2Z2 O
T2 = A different type of person (e.g., persons with a
R C X2Y2Z2 O different health problem/persons with no health
problem, or persons of the other gender)
S = Start S = Start
R = Randomization M = Matching of the two types of persons on potential-
C = Covariate (a variable associated with the depend- ly confounding variables
ent variable) R = Randomization
X1 = An intervention of type X (one of two conditions X = An intervention
of an independent variable- the other condition is Y = A different intervention
X 2) O = Observation (a dependent variable)
X2 = An intervention of type X (one of two conditions
of an independent variable- the other condition is Figure 7.19 Randomized treatments by levels
X 1) design (2 ⫻ 2) (pretest–posttest with matching
Y1 = An intervention of type Y (one of two conditions on potentially confounding variables). There are
of an independent variable- the other condition is two types of people before the start of the
Y 2) research (T1 and T2). The two types of people
Y2 = An intervention of type Y (one of two conditions are matched (M) in relevant ways. The first type
of an independent variable- the other condition is of persons are then randomly assigned to one
Y 1) of two interventions, and the second type of
Z1 = An intervention of type Z (one of two conditions persons are also assigned to one of the two
of an independent variable- the other condition is interventions.
Z 2)
Z2 = An intervention of type Z (one of two conditions
of an independent variable- the other condition is Randomized Controlled Trials (RCTs)
Z 1)
O = Observation (a dependent variable) A randomized controlled trial (sometimes called a
Figure 7.18 Completely randomized 2 ⫻ 2 ⫻ 2 clinical trial) is an experiment wherein an impor-
factorial design (posttest-only with covariate). tant health outcome is the dependent variable, a
After randomization (R) and measurement of clinical intervention is part of the independent
the covariate (C), the eight groups receive all variable, and research participants are recruited
possible combinations of the three types of and randomly assigned over time as they become
interventions, each of which has two conditions. available (Friedman, Furberg, & DeMets, 1998).
The posttest follows. Many of the examples used already in this chapter
reflect hypotheses that could be studied by RCTs
more when working in a parallel situation, (e.g., falls prevention studies, the effects of various
whereas men interact nonverbally more in a proj- occupational therapy interventions on functional
ect situation. outcomes). However, not all the examples dis-
The following are other examples of random- cussed in this chapter dealt with outcomes; several
ized treatments by levels design: dealt with short-term effects of theoretical interest
to occupational therapy (e.g., the effects of occu-
• Studying the effects of an intervention (vs. usual pationally embedded exercise on exercise repeti-
care) in persons with left hemiparesis in compar- tions, or the comparison of parallel versus project
ison to persons with right hemiparesis, groups in terms of nonverbal socialization). These
• Comparing the effectiveness of an educational theory-based studies of short-term effects are
strategy (vs. typical classroom strategy) in first- experiments but not RCTs. These non-RCT exper-
year versus second-year occupational therapy iments add to the theoretical base of occupational
students, and therapy and occupational therapy models of prac-
• Studying the effects of an intervention (vs. tice, but they do not directly test whether occupa-
a usual care condition) at multiple sites, where tional therapy produces health outcomes or not.
by subjects are randomly assigned within each Randomized controlled trials reflect a tradition
site. of experimentation that developed in medicine and
pharmacology. A major problem faced in drug out- not considered). On the other hand, studies of
come studies that is not so much of a problem in effectiveness test whether the intervention works
short-term, non-RCT experimentation is the fact in typical clinical conditions.
that dropouts can threaten the validity of results. A special issue related to dropouts involves a
Another problem in studying long-term outcomes choice between intention-to-treat analysis and per-
is that bias can easily be introduced if there are not protocol analysis (Hollis & Campbell, 1999). In
special procedures for masking randomization intention-to-treat analysis, dropouts are sought out
codes. For example, if a research assistant is for outcomes testing even if they discontinued the
screening a particularly weak patient who never- intervention to which they were assigned, and even
theless meets the criteria for inclusion, and if the if they ended up experiencing the opposing inter-
research assistant knows that the randomization vention (the other condition of the independent
code indicates that the next person accepted into variable). Part of the rationale for intention-to-treat
the study is assigned to the research assistant’s pre- analysis is that participants who drop out or choose
ferred intervention, the temptation (conscious or the opposite intervention might do so because of
unconscious) is for the research assistant to reject adverse side effects brought about by the interven-
the patient from the study. Another feature of tion to which they were originally assigned.
RCTs is that outcomes are often categorical (e.g., Advocates of intention-to-treat analysis argue
life or death) rather than measurable; therefore, that the clinician and the patient need to know the
biostatisticians have paid particular attention to likelihood that a particular intervention will be
branches of statistics dealing with categorical out- effective in advance of a prescription. In contrast,
comes (e.g., survival rates). per-protocol analysis excludes dropouts, with the
Authors within the RCT tradition often use dif- rationale that inclusion of dropouts only causes
ferent terms from other experimenters (e.g., in random error and increases the chances of a type II
psychology, agriculture, or sociology). Instead of error. Currently, intention-to-treat analysis tends to
saying independent variable, they often say inter- be the favored methodology, with the possibility of
ventions. Instead of saying that participants are a secondary test on a per-protocol basis after the
randomly assigned to conditions of the independ- primary test.
ent variable, they often say that subjects are ran- Much effort has been devoted to the improve-
domly assigned to arms. Instead of saying ment of RCT design and RCT reporting. A result
dependent variable, they often say outcome. In the of this effort is the CONSORT (Consolidated
RCT literature, a distinction has been made Standards of Reporting Trials) Statement. The cur-
between studies of efficacy and studies of effec- rent version of CONSORT provides a checklist of
tiveness. Efficacy deals with the study of an inter- essential items that should be included in an RCT
vention under nearly ideal conditions (where (Table 7.2), and a diagram for documenting the
random error is highly controlled, where interven- flow of participants through a trial (Figure 7.20 on
tionists have special training, and where costs are p. 83) (Moher, Shulz, & Altman, 2001).
Table 7.2 CONSORT Criteria for Randomized Control Trials

Paper section topic Item Description
Title and abstract 1 How participants were allocated to interventions (e.g., “random allocation,”
“randomized,” or “randomly assigned”).
Introduction 2 Scientific background and explanation of rationale.
Method
Participants 3 Eligibility criteria for participants and the settings and locations where the
data were collected.
Interventions 4 Precise details of the interventions intended for each group and how and
when they were actually administered.
Objectives 5 Specific objectives and hypotheses.
Outcomes 6 Clearly defined primary and secondary outcome measures and, when
applicable, any methods used to enhance the quality of measurements
(e.g., multiple observations, training of assessors).
(continued)
Table 7.2 CONSORT Criteria for Randomized Control Trials (continued)

Paper section topic Item Description
Sample size 7 How sample size was determined and, when applicable, explanation of
any interim analyses and stopping rules.
Randomization: 8 Method used to generate the random allocation sequence, including
Sequence generation details of any restriction (e.g., blocking, stratification).
Randomization: 9 Method used to implement the random allocation sequence (e.g.,
Allocation concealment numbered containers or central telephone), clarifying whether the
sequence was concealed until interventions were assigned.
Randomization: 10 Who generated the allocation sequence, who enrolled participants, and
Implementation who assigned participants to their groups.
Blinding (masking) 11 Whether or not participants, those administering the interventions, and
those assessing the outcomes were blinded to group assignment.
When relevant, how the success of blinding was evaluated.
Statistical methods 12 Statistical methods used to compare groups for primary outcome(s);
Methods for additional analyses, such as subgroup analyses and
adjusted analyses.
Results
Participant flow 13 Flow of participants through each stage (a diagram is strongly
recommended). Specifically, for each group report the numbers of
participants randomly assigned, receiving intended treatment,
completing the study protocol, and analyzed for the primary outcome.
Describe protocol deviations from study as planned, together with
reasons.
Recruitment 14 Dates defining the periods of recruitment and follow-up.
Baseline data 15 Baseline demographic and clinical characteristics of each group.
Numbers analyzed 16 Number of participants (denominator) in each group included in each
analysis and whether the analysis was by “intention-to-treat.” State the
results in absolute numbers when feasible (e.g., 10/20, not 50%).
Outcomes and estimation 17 For each primary and secondary outcome, a summary of results for each
group, and the estimated effect size and its precision (e.g., 95%
confidence interval).
Ancillary analyses 18 Address multiplicity by reporting any other analyses performed, including
subgroup analyses and adjusted analyses, indicating those
prespecified and those exploratory.
Adverse events 19 All important adverse events or side effects in each intervention group.
Discussion
Interpretation 20 Interpretation of the results, taking into account study hypotheses, sources
of potential bias or imprecision, and the dangers associated with
multiplicity of analyses and outcomes.
Generalizability 21 Generalizability (external validity) of the trial findings.
Overall evidence 22 General interpretation of the results in the context of current evidence.
Cluster Randomized Controlled Trials tions on acute rehabilitation patients with hip frac-
ture (vs. usual care), 20 rehabilitation hospitals can
A cluster randomized controlled trial (Figure 7.21 be involved. Ten are randomly assigned to the spe-
on p. 84) is a special kind of RCT, in which clini- cial home evaluation condition, with the other 10
cal sites are randomly assigned to arms (conditions assigned to usual care. Perhaps each hospital can
of the independent variable), as opposed to ran- supply 15 patients, and all patients at a given site
domly assigning individual participants (Friedman, are treated the same because they are in the same
Furberg, & DeMets, 1998). For example, in a study experimental condition. Sometimes this is called a
of the effects of intensive, repeated home evalua- nested design (participants are “nested” together
Assessed for
eligibility
Excluded
Enrollment
Not meeting inclusion
criteria
Refused to
participate
Other reasons
Randomization
Allocated to intervention Allocated to intervention

Allocation
Received allocated inter- Received allocated inter-

vention vention
Did not receive allocated Did not receive allocated

intervention (give reasons) intervention (give reasons)
Follow-up
Lost to follow up Lost to follow up

(give reasons) (give reasons)
Discontinued intervention Discontinued intervention

Analysis
Analyzed Analyzed
Excluded from analysis Excluded from analysis
Figure 7.20 CONSORT flow diagram for reporting randomized con-

trolled trials.
within each site). One advantage of this design is Crossover Design (Also Called
the prevention of bleeding from one condition to Counterbalanced Design)
the other (e.g., when patients see their roommates
getting special treatment). Another advantage is A crossover design starts off like a basic experi-
that differing skill levels and possible biases of ment, in which participants are randomly assigned
those administering the interventions can be to as many groups as there are conditions of
assumed to be more balanced across conditions the independent variable. But each group then goes
than is the case when interventionists observe each on to experience both conditions of the independ-
other in close quarters. The main disadvantages are ent variable (Chow & Lui, 1998). In the case of
complexity, expanded training, fidelity issues, and two conditions of the independent variable, one
ethical compliance within the rules and cultures of randomly assigned group receives condition X
many different organizations (Spilker, 1991). first, is measured on the dependent variable, then
Another disadvantage is a loss of power in the sta- receives condition Y, and then is measured again.
tistical analysis, caused by within-site similarity. It The other randomly assigned group receives con-
is important to distinguish this design from the dition Y first, then X. If Y consistently leads to
multisite design discussed above in the section on different effects of the dependent variable, regard-
randomized treatments by levels design, in which less of order, it is concluded that the indepen-
participants at each site are randomized (as dent variable is probably responsible for the
opposed to being nested, as in this design). difference.
S of order in a crossover design addresses all these

RS O X O causes of type I error. If participants score higher
RS O X O when experiencing one intervention than another,
RS O X O regardless of order of presentation, factors such as
Eight RS O X O warming up, learning, fatigue, and boredom prob-
Sites ably cannot account for the results.
RS O Y O
The advantage of crossover design is the reduc-
RS O Y O
tion of the chances of a type II error. Dispersion
RS O Y O due to individual differences is controlled because
RS O Y O each participant is compared to oneself. In addi-
tion, two measurements per person increase statis-
S = Start tical power in comparison to designs in which each
RS = Randomization by site (each site and all persons person is measured once. The disadvantage of this
at that site have an equal chance of being design is that a sequence effect or a carryover
assigned to X or Y effect can prevent a clear interpretation that one
X = An intervention administered to each subject at intervention is superior, regardless of order. For
the site
Y = A different intervention administered to each sub-
example, the researcher must be hesitant to con-
ject at the site clude that the special seating system is superior if
O = Observation of each person (measurement or cat- productivity was approximately equal between
egorization used for dependent variable) conditions for the group that experienced the spe-
Figure 7.21 Randomized cluster design, com- cial seating system second. Crossover design is
paring two interventions at eight research sites. therefore not recommended for studies of inter-
Four research sites are randomly assigned (Rs) ventions that are hypothesized to lead to enduring
to one condition (X) , and the other four sites changes within the person. Crossover designs are
are assigned to the other condition (Y). also not recommended for studies in which the
dependent variable involves certain kinds of learn-
ing (in which participants tend to do better the sec-
Consider the hypothesis: A specially designed ond time through a problem). On the other hand,
wheelchair seat will increase work productivity in this design is particularly appropriate for studies of
adults with cerebral palsy and mental retardation assistive technology and compensatory methods,
(in comparison to an off-the-shelf, standard sling- and for the study of stimuli that have short-term
seat wheelchair). Half the participants are ran- effects (e.g., effects on mood or arousal).
domly assigned to experiencing the special seating If there are three or more conditions to the inde-
system first, and then experiencing the standard pendent variable, counterbalanced design offers
seating system. The other half are randomly two options: the Latin Square (Figure 7.22) or ran-
assigned to the standard system first, and then to dom assignment to all possible sequences (Figure
the special system. If productivity is greater for 7.23). Consider the hypothesis of comparing occu-
both groups when seated in the special system, the pationally embedded exercise (coded O), imagery-
directional hypothesis is supported. based exercise (coded I), and rote exercise (coded
This design controls against type I error R). One third of the sample is randomly assigned
through counterbalancing the conditions of the
independent variable. A faulty design not control-
ling for type I error is the administration of one S
condition of the independent variable to all partic- R X O Y O Z O
ipants first, and then the administration of the other R Y O Z O X O
condition second. The difference on the dependent R Z O X O Y O
variable might be due to many factors other than
the independent variable. For example, partici-
S = Start
pants might have scored high on the second condi- R = Randomization
tion because they were warmed up on the first, or X = An intervention
because they learned what they have to do to score Y = A different intervention
well on the dependent variable. On the other hand, Z = Another different intervention
participants might have scored low on the second O = Observation (primary dependent variable)
condition because of fatigue or boredom. These Figure 7.22 Randomized counterbalanced
are potentially biasing confounding variables. In design (Latin Square), with an independent
contrast to this faulty design, the counterbalancing variable consisting of three conditions (X, Y, Z).
S S
R X O Y O Z O O X O
R Y O Z O X O O Y O
R Z O X O Y O
S = Start
R X O Z O Y O X = Intervention
R Y O X O Z O O = Observation (measurement or categorization used
R Z O Y O X O
Figure 7.24 Nonrandomized comparison group
S = Start design (pretest–posttest, with true control
R = Randomization group). After a pretest and before the posttest
X = An intervention
(O), the subjects in one naturally occurring
Y = A different intervention
group receive an intervention (X), and the sub-
Z = Another different intervention
O = Observation (primary dependent variable)
jects in a different naturally occurring group
receive nothing. It is important to note that this
Figure 7.23 Randomized counterbalanced design is not recommended except for pilot
design (fully randomized), with an independent testing in advance of future research.
variable consisting of three conditions (X, Y, Z).
room). Confounding variables might account for
any difference found on the dependent variable.
to an O–I–R sequence; another one third is Hence this design is often called the nonequivalent
assigned to I–R–O, and the rest of the sample is group design.
assigned to R-O-I. This is a Latin Square. Note that A variation on this kind of design is the waiting
each of the interventions occurs first once, second list control group design, in which those at the top
once, and third once. The Latin square uses three of a waiting list for an intervention are assigned
of the six possible sequences; other possible to an intervention and those at the bottom of the
sequences are I–O–R, O–R–I, and R–I–O. The list serve as the control group. After the comple-
alternative counterbalancing strategy is to ran- tion of the study, those at the bottom of the list
domly assign the sample into six groups, each of receive the intervention, so this design is favored
which experiences one of the six possible for humanitarian reasons. Another advantage of
sequences. this design is that it permits the study of expensive
interventions (e.g., home modifications) without
Group Designs Not the researcher having to fund the intervention
because all the persons on the waiting list will ulti-
Involving Randomization mately receive the intervention. However, as with
other nonrandomized designs, it is possible that
the people at the bottom of the list are systemati-
Nonrandomized Comparison Group Design
cally different from those at the top (e.g., those at
The nonrandomized comparison group design the top might be more knowledgeable on how to
(Figure 7.24) is also called a nonrandomized trial work the system, more resourceful, and/or more
when the dependent variable is a valued health out- assertive).
come. When type I error was discussed earlier, An important design strategy when conducting
nonrandomized comparison group designs were nonrandomized designs is to match the groups at
described as similar to experiments, but using con- the outset on potentially confounding variables as
venient preexisting groupings as the method of illustrated in Figure 7.25 on p. 86. Consider the
assigning participants to conditions of the inde- hypothesis that an occupational therapy handwrit-
pendent variable, as opposed to randomization. ing program increases legibility in first-grade chil-
For example, classrooms of children make up con- dren identified as educationally at-risk because of
venient preexisting groups, so that all the children poverty. The school principal and teachers wel-
in two classrooms can be assigned to one condition come the research and offer four classrooms of
and all the children in two different classrooms can children, but only if classes are not disrupted by
be assigned to the other condition. The problem is randomization. The researcher gathers data on
that the children in two sets of classrooms might exact age and standardized educational scores.
be systematically different from each other (e.g., Then the researcher makes sure that the children in
due to class placement procedures, such as group- the two classrooms to receive the intervention are
ing the academically talented children in one class- matched to the children in the other two class-
S S
M X O T1 MO
M Y O T2 MO
T3 MO
S = Start
M = Matching on the dependent variable or some
variable associated with the dependent variable T1 = One type of person (e.g., persons with a specific
X = Intervention health problem)
Y = A different intervention T2 = A different type of person (e.g., persons with a
O = Observation (measurement or categorization used different health problem)
for dependent variable) T3 = Another different type of person (e.g., persons
with a different health problem, or persons with no
Figure 7.25 Nonrandomized comparison group health problem)
design, with matching. The subjects in one nat- S = Start
M = Matching of the two types of persons on
urally occurring group are matched to the sub-
potentially confounding variables
jects in another group, either on the dependent O = Observation (a dependent variable)
variable (O) or on a variable associated with the
dependent variable. Each group then receives Figure 7.26 Cross-sectional design (three condi-
X or Y. Although the matching procedure tions), with matching. The nonmanipulated inde-
improves the design somewhat over other non- pendent variable (three types of persons to be
randomized designs, it is always possible that compared) preexists the research. The three
some relevant variable has not been matched. types of persons are matched (M) on potentially
It is important to note that this design is not rec- confounding variables, and immediately
ommended except for pilot testing in advance observed (O) for possible differences.
of future research.
rooms. Means and dispersions of age and stan- tion of pilot data to justify a much more expensive
dardized educational scores as well as proportions randomized controlled trial.
of sex are approximately equal between children
receiving the intervention and those not receiving Cross-Sectional Design (Immediate
it. This process of matching necessarily involves
Ex Post Facto Comparisons)
the exclusion of some children at the extremes of
the ranges who are preventing an overall match. The purpose of this design (Figure 7.26) is to com-
Ideally, the matching and exclusion are done by pare different types of persons in terms of some
someone masked to the hypothesis and to knowl- immediately measurable dependent variable. For
edge of which group will receive which condition example, the researcher might want to compare
of the independent variable (dealing only with children with cerebral palsy to children without a
numbers). Despite all this, it is still possible known disorder in terms of spatial perception. The
that the two groups are systematically different researcher wants to know if children with cerebral
from each other in some way that the researcher palsy have a special problem with spatial percep-
did not or could not measure. For example, one of tion. Another example is the cross-sectional devel-
the classroom teachers might emphasize penman- opmental study, where children of different age
ship in ways that the other classroom teachers groups (let us say 36-month-olds, 42-month-olds,
do not. and 48-month-olds) are compared in terms of
An alternative strategy to matching is the use of attention to task. A third example is to compare
special statistical procedures designed to correct persons with left hemiplegia to persons with right
for initial differences between groups. Here, hemiplegia in terms of standing balance.
potentially confounding variables must be iden- In this design, the researcher does not manipu-
tified, measured, and entered into statistical cal- late the assignment of subjects to groups, as in
culations. These procedures are controversial experiments. This kind of independent variable is
(Huitema, 1980). On the one hand, it can be argued sometimes called an organismic variable or a vari-
that appropriate statistical control permits a able consisting of preexisting conditions. Because
researcher to make an end-of-study claim that an the independent variable took place in the past
independent variable probably affected the before the researcher came along, the Latin phrase
dependent variable. However, others place little ex post facto is also used to describe this design.
confidence in the results of nonrandomized com- Cross-sectional designs are needed to answer
parison designs and claim that the write-ups of questions concerning the special characteristics of
these designs should not infer probable causality. disability groups. The problem, of course, is that a
Perhaps the best use of this design is in the collec- difference on the dependent variable might well be
due to some confounding variable (a type I error). S

For example, a difference in spatial perception E or no E T1 M OE
between children with cerebral palsy and non- E or no E T2 M OE
disabled children might be due to factors that have
nothing to do with cerebral palsy (e.g., parental E = An event hypothesized to be a risk factor
skill, access to older siblings, etc.). As with other T1 = A type of person with a specific health problem
nonrandomized designs, cross-sectional designs T2 = A type of person without the health problem
S = Start
require careful matching on potentially confound- M = Matching of the two types of persons on poten-
ing variables (i.e., matching means, dispersions, tially confounding variables
and proportions). Usually a sample from the rela- OE = Observation (dependent variable) of the
tively rare population (e.g., the disability group) is hypothesized risk factor through retrospective
selected in some representative way, and then documentation
members of the comparison group (e.g., matched Figure 7.27 Case-control design investigating a
nondisabled controls) are assembled. Once again, single risk factor in a single health condition (in
masking the matcher is a positive strategy. The comparison to a matched group of persons
other strategy employed with this and all other without the health condition). A group with a
nonrandomized designs is to use statistical meth- health problem (T1) is identified and matched
ods to make adjustments on the dependent variable (M) to a group without the health problem (T2).
when comparison groups differ in relevant ways. It The researcher documents (OE) whether or not
each group experienced a specific risk factor
is important to remind ourselves that this strategy
event (E) prior to the development of the health
has vigorous adherents as well as vigorous critics. problem in T1.
In the field of human development, aspects of
cross-sectional designs are frequently combined
with longitudinal approaches (in which the same events unfold. As with many other kinds of non-
sample is repeatedly measured at planned intervals randomized designs, research questions can be
over time) (Rosenthal & Rosnow, 1991). The gen- studied in humans that are unethical or impossible
eral idea is that the researcher can have a relatively to study through random assignment. However, in
high degree of confidence if the cross-sectional addition to the possibility that the comparison
and the longitudinal approaches confirm each groups might be different from each other in other
other. The longitudinal approach eliminates the ways beside the dependent variable, retrospective
possible bias likely in comparing different popula- studies frequently depend on unreliable data. For
tions, and the cross-sectional approach eliminates example, one common way to collect data con-
the possible bias of repeated testing. The same cerning past events is to interview the participant.
logic could be applied to studies of the progression As a general rule, self-reports become less accu-
of specific disabilities. rate as distance in time increases. In addition, the
interviewer may be biased, consciously or uncon-
sciously, in the way that questions are asked to the
Case-Control Design (Case-Referent participant. Another type of measurement error
Design or Case Comparison Design) occurs when using archival information (e.g., med-
ical records). In prospective studies (involving
This design (Figure 7.27) was developed in the measurement of future events), a plan can be for-
field of epidemiology. The purpose of the design to mulated to collect data that includes special train-
find out if some specific variable from the past dis- ing for raters and tests of inter-rater independent
criminates between people who today have a disagreement. This is impossible in retrospective
ease (or disability) and people who do not. The studies. Another common problem with case-
classic example is to investigate past levels of control studies involves multiplicity, in which the
smoking habits of persons with lung cancer and researcher investigates a host of past events, some
matched controls who do not have lung cancer. An of which might discriminate between currently
example that is more relevant to occupational ther- assembled groups by pure chance. Special statisti-
apy is to compare newly admitted patients with hip cal procedures are needed to deal with specificity
fracture to matched controls in terms of past inju- as well as the possibility of complex interactions.
rious falls requiring medical care. Because the data
collected refers to past events, case-control designs Multigroup Cohort Design (Cohort
are termed retrospective studies. Analytic Study)
The advantage of this design is the relative ease
with which possible causal or risk factors can be In this design (Figure 7.28 on p. 88), the researcher
explored, without waiting for years to see how matches a sample that has a hypothesized risk fac-
S mate not only the rate of the outcome but also the
T1 M O O O differential rate in comparison to a control group.
T2 M O O O This controls for the possibility that the outcome
will develop whether or not the risk factor is pres-
ent. Another weak alternative to the multigroup
T1 = Persons with a specific risk factor cohort study is the use of previously collected
T2 = Perspns without the risk factor
S = Start demographic data on the healthy population as a
M = Matching of the two types of persons on poten- control condition, as opposed to using matched
tially confounding variables controls. Problems with previously collected
O = Observation (a dependent variable) demographic data are: (a) the data were collected
Figure 7.28 Multigroup cohort design (two condi- under different circumstances by different data
tions), with matching and longitudinal follow-ups. collectors and (b) the life experiences of a prospec-
Persons with (T1) and without (T2) a possible tive cohort are different from archival information
risk factor are matched on potentially confound- collected at a different point in time.
ing variables. Repeated observations (O) over
time indicate whether or not the hypothesized
risk factor leads to the health problem. Conclusion
In summary, choice of a research design depends
tor to a sample that does not. Then the researcher on five factors: prevention of type I error; preven-
picks a future point in time (or a series of points if tion of type II error; external validity; the resources
using repeated measures) to see if the hypothe- available to the researcher; and the theoretical or
sized risk factor truly predicts the disease or dis- clinical importance of the research question. If the
ability. A classic example in epidemiology is to see researcher wishes to study the effects of a power-
if high levels of cholesterol in middle age predict ful independent variable (i.e., one that produces a
heart disease in old age. An example more relevant large effect in the dependent variable) and if gen-
to occupational therapy is to investigate if mild left erous resources are available, the basic posttest-
hemiplegia predicts automobile accidents in com- only experimental design (perhaps configured as a
parison to matched cohorts of (a) persons with randomized controlled trial) provides the strongest
mild right hemiplegia and (b) healthy controls. evidence. However, the researcher is often
This design is similar to nonrandomized com- unaware of how powerful the independent variable
parison group design. Indeed, some authors use the is until years of study have passed. Therefore, the
term cohort design for what we earlier defined as desire for protection against type II error (which is
nonrandomized comparison group design. A dif- almost always more likely in the best of designs
ference between designs is that multigroup cohort than type I error) enhances the attractiveness of
design deals with the study of risk factors not alternative randomized research designs. Each of
under the researcher’s control and resulting in poor these has its advantages and disadvantages,
outcomes, whereas nonrandomized comparison depending on the research question and resources
group design deals with interventions under the available. Though questionable in terms of type I
control of the researcher and hypothesized to have error, nonrandomized designs are important as
positive outcomes. Given this logic, a study of cost-efficient pilot studies. Nonrandomized
whether brain injury predicts psychiatric disorders designs can also address research questions of spe-
(at higher rates than controls) is a multigroup cial importance to the field of occupational therapy
cohort design, whereas a study of a special inter- and to the understanding of persons with disability.
vention designed to prevent psychiatric disorders
in persons with brain injury is a nonrandomized
comparison group design if participants are
The introduction to group comparison designs in
assigned to conditions in some way other than ran- this chapter encourages occupational therapy stu-
domization. dents and practitioners to learn more about spe-
Prospective multigroup cohort studies gener- cific designs for specific research questions. The
ally have stronger claims to validity than retro- author’s intent will be served if this chapter
spective studies because measurements can be results in some eager young colleague doing
planned and possible confounding variables can be experimental research that advances occupa-
monitored. Multigroup cohort studies are stronger tional therapy knowledge and helps the profes-
than single cohort studies because the presence of sion to better serve our clients and students.
a matched control group makes it possible to esti-
No single design is best for all circumstances. Stevens, J. (1986). Applied multivariate statistics for the
Hopefully this chapter provides an introduction to social sciences. Hillsdale, NJ: Lawrence Erlbaum.
Thorndike, E. L., & Woodworth, R. S. (1901). The influ-
the advantages and disadvantages of the main ence of improvement in one mental function upon the
group comparison designs. efficiency of other functions. Psychological Review, 8,
247–261, 384–395, 553–564.
REFERENCES
Amberson, J. B., McMahon, B. T., & Pinner, M. (1931). A
clinical trial of sanocrysin in pulmonary tuberculosis.
RESOURCES
American Review of Tuberculosis, 24, 401–435. http://consort-statement.org/
Campbell, D. T., & Stanley, J. C. (1963). Experimental and OTseeker.com
quasi-experimental designs for research. Chicago: Altman, D. G., Schulz, K. F., Moher, D., Egger, M.,
Rand McNally. Davidoff, F., Elbourne, D., et al. (2001). The revised
Chow, S. C., & Lui, J. (1998). Design and analysis of clin- CONSORT statement for reporting randomized trials:
ical trials. New York: John Wiley & Sons. Explanation and elaboration. Retrieved September 7,
Fisher, R. A. (1935). The design of experiments. London: 2002, from http:/consort-statement.org/newene.html
Oliver and Boyd. Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D.,
Fleiss, J. L. (1986). The design and analysis of clinical Olkin, I., et al. (1996). Improving the quality of report-
experiments. New York: John Wiley & Sons. ing of randomized controlled trials: The CONSORT
Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1998). statement. Journal of the American Medical
Fundamentals of clinical trials (3rd ed.). New York: Association, 276, 637–639.
Springer-Verlag. Berk, P. D., & Sacks, H. S. (1999). Assessing the quality of
Gitlin, L. N., Corcoran, M., Martindale-Adams, J., Malone, randomized control trials: Quality of design is not the
C., Stevens, A., & Winter, L. (2000). Identifying mech- only relevant variable. Hepatology, 30, 1332–1334.
anisms of action: Why and how does intervention Bork, C. E. (1993). Research in physical therapy.
work? In R. Schulz (Ed.), Handbook on dementia care- Philadelphia: Lippincott.
giving: Evidence-based interventions for family care- Cadman, D., Law, M., DeMatteo, C., Walter, S.,
givers. New York: Springer. Rosenbaum, P., & Russell, D. (1989). Evaluation of
Hollis, S., & Campbell, F. (1999). What is meant by inten- treatment in occupational therapy: Part 2. Practical
tion to treat analysis? Survey of published randomised issues in conducting clinical trials. Canadian Journal
controlled trials. British Medical Journal, 319, of Occupational Therapy, 56, 243–247.
670–674. Cohen, J. (1988). Statistical power analysis for the behav-
Holm, M. B. (2000). Our mandate for the new millennium: ioral sciences (2nd ed.). Hillsdale, NJ: Lawrence
Evidence-based practice, 2000 Eleanor Clark Slagle Erlbaum.
Lecture. American Journal of Occupational Therapy, Cooper, H., & Hedges, L. V. (Eds.). (1994). The handbook
54, 575–585. of research synthesis. New York: Russell Sage
Huitema, B. E. (1980). The analysis of covariance and Foundation.
alternatives. New York: John Wiley & Sons. Dhingra, V., Chittock, D. R., & Ronco, J. J. (2002).
Kerlinger, F. N. (1986). Foundations of behavioral Assessing methodological quality of clinical trials in
research (3rd ed.). New York: Harcourt Brace. sepsis: Searching for the right tool. Critical Care
Lichstein, K. L., Riedel, B. W., & Grieve, R. (1994). Fair Medicine, 30, 487–488.
tests of clinical trials: A treatment implementation Donner, A., & Klar, N. (2000). Design and analysis of
model. Advances in Behaviour Research and Therapy, cluster randomization trials in health research.
16, 1–29. London: Arnold.
Moher, D., Schulz, K. F., & Altman, D. G. (2001). The Egger, M., Juni, P., & Bartlett, C. (2001). Value of flow
CONSORT statement: Revised recommendations for diagrams in reports of randomized controlled trials.
improving the quality of reports of parallel-group ran- Journal of the American Medical Association, 285,
domized trials. Annals of Internal Medicine, 134, 1996–1999.
657–662. Everitt, B. S., & Pickles, A. (1999). Statistical aspects of
Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in the design and analysis of clinical trials. Singapore:
outcome studies. Clinical Psychology Review, 11, Regal Press.
247–266. Hennekens, C. H. (1999). Clinical trials in cardiovascular
Nelson, D. L., & Mathiowetz, V. (2004). Randomized con- disease: A companion to Braunwald’s heart disease.
trolled trials to investigate occupational therapy Philadelphia: W. B. Saunders.
research questions. American Journal of Occupational Hinkle, D. E., Wiersma, W., & Jurs, S. G. (1998). Applied
Therapy, 58, 24–34. statistics for the behavioral sciences (4th ed.). Boston:
Ottenbacher, K. J. (1995). Why rehabilitation research does Houghton Mifflin.
not work (as well as we think it should). Archives of Huwiler-Muntener, K., Juni, P., Junker, C., & Egger, M.
Physical Medicine and Rehabilitation, 76, 123–129. (2002). Quality of reporting of randomized trials as a
Ottenbacher, K. J., & Maas, F. (1998). How to detect measure of methodological quality. Journal of the
effects: Statistical power and evidence-based practice American Medical Association, 287, 2801–2804.
in occupational therapy research. American Journal of Kazdin, A. E. (2003). Research design in clinical psychol-
Occupational Therapy, 53(2), 181–188. ogy (3rd ed.). Boston: Allyn & Bacon.
Rosenthal, R., & Rosnow, R. L. (1991). Essentials of Law, M., Cadman, D., Rosenbaum, P., Russell, D.,
behavioral research: Methods and data analysis (2nd DeMatteo, C., & Walter, S. (1989). Evaluation of treat-
ed.). New York: McGraw-Hill. ment in occupational therapy. Part 1. Methodology
Spilker, B. (1991). Guide to clinical trials. Philadelphia: issues in conducting clinical trials. Canadian Journal
Lippincott Williams & Wilkins. of Occupational Therapy, 56, 236–242.
McFadden, E. (1998). Management of data in clinical Prien, R. F., & Robinson, D. S. (Eds.). (1994). Clinical
trials. New York: John Wiley & Sons. evaluation of psychotropic drugs: Principles and
Moher, D., Jones, A. J., & Lepage, L. (2001). Use of guidelines. New York: Raven Press.
CONSORT statement and quality of reports of random- Reisch, J. S., Tyson, J. E., & Mize, S. G. (1989). Aid to the
ized trials: A comparative before-and after evaluation. evaluation of therapeutic studies. Pediatrics, 84,
Journal of the American Medical Association, 285, 815–827.
1992–1995. Rennie, D. (2001). CONSORT revised: Improving the
Montori, V. M., & Guyatt, G. (2001). Intention-to-treat reporting of randomized trials. Journal of the American
principle. Canadian Medical Association Journal, 165, Medical Association, 285, 2006–2007.
1339–1341. Schulz, K. F., Chambers, I., Hayes, R. J., & Altman,
Neter, J., Wasserman, W., & Kutner, M. H. (1985). Applied D. G. (1995). Empirical evidence of bias. Dimen-
linear statistical models (2nd ed.). Homewood, IL: sions of methodological quality associated with
Richard D. Irwin. estimates of treatment effects in controlled trials.
Olkin, I. (1995). Meta-analysis: Reconciling the results of Journal of the American Medical Association, 273,
independent studies. Statistics in Medicine, 14, 408–412.
457–472. Schwartz, D., Flamant, R., & Lellouch, J. (1980). Clinical
Ottenbacher, K. J. (1998). A quantitative evaluation of mul- Trials. London: Academic Press.
tiplicity in epidemiology and public health research. Yeaton, W. H., & Sechrest, L. (1981). Critical dimen-
American Journal of Epidemiology, 147, 615–619. sions in the choice and maintenance of successful
Portney, L. G., & Watkins, M. P. (2000). Foundations of treatments: Strength, integrity, and effectiveness.
clinical research: Applications to practice (2nd ed.). Journal of Consulting and Clinical Psychology,
Upper Saddle Rive, NJ: Prentice-Hall Health. 49, 156–167.
C H A P T E R 8
Survey Research Design

Kirsty Forsyth • Frederick J. Kviz
Survey research is a method of inquiry character- much as possible, as discussed throughout the
ized by collecting data using structured questions chapter.2
to elicit self-reported information from a sample of Surveys collect data using self-administered
people (Aday, 1996; DePoy & Gitlin, 1998). questionnaires, telephone interviews, or face-to-
Surveys are characterized by these key dimensions: face interviews. Self-administered questionnaires
may be mailed, administered online, or distributed
• Identifying the population of interest and appro- and collected at convenient points of contact such
priately sampling that population, as in schools, workplaces, clinics, or hospitals.
• Identifying the research aims and question and This chapter focuses on:
generating survey questions to systematically
gather the necessary information, and • How to choose the specific survey data gathering
• Developing statistical estimates that can be gen- method,
eralized to the population under study. • How to build the questionnaire/interview,
• How to administer the survey (including sam-
The main advantages of survey research are pling), and
that investigators can: • Preparation for data analysis.
• Reach a large number of respondents1 with rela-
tively minimal expenditure,
• Collect data on numerous variables, and Choosing Data
• Perform statistical manipulation during data
analysis that permits multiple uses of the data set Gathering Methods
(Rea & Parker, 1997).
As noted in the preceding section, survey research
There are two main uses questionnaires, (administered directly, by
factors that can influence mail, and online), and
the rigor of survey interviews (telephone
research (Fowler, 2002). Survey research is a method and face-to-face) to col-
The first is potential of inquiry characterized lect data. Quantitative
nonresponse bias (i.e., survey interviews are
respondents selected for by collecting data using quite distinct from inter-
the sample who elect not structured questions to elicit views used in qualitative
to respond). These sec- studies. In a survey, the
ond is potential response
self-reported information interview questions are
bias, which may result from a sample of people. fixed, and the inter-
from factors such as: viewer seeks to adminis-
ter the interview in an
• Respondents being unable to recall information objective manner following the interview protocol.
accurately,
• Respondents interpreting the meaning of a ques-
tion differently than the meaning intended by the 2Another important consideration in survey research is gov-
researcher, or ernmental regulations affecting the privacy of patients. This

• Response choices that do not accurately express varies, of course, by country. In the United States the Health
respondents’ experiences or opinions. Insurance Portability and Accountability Act (HIPAA) pri-
vacy rule has requirements which must be followed in med-
When designing survey research, investigators ical research. Medical research that falls under HIPAA
must take care to reduce these two forms of bias as regulations conform to these rules. While these rules govern
all forms of research involving patients, they can have par-
ticular implications for survey research. A useful source of
1The term “respondent” refers to people responding to the information on how HIPAA regulations affect research is the
survey. National Institutes of Health Web site.
Figure 8.1 A research assistant stuffs envelopes in preparation for a

mailed survey.
When choosing a survey data gathering A specific disadvantage of mailed question-

method, the advantages and disadvantages of each naires is that the response rate tends to be lower
method need to be considered. These are discussed than that for interview methods. Moreover, many
below and illustrated in Table 8.1. follow-ups may be required to obtain an accept-
able response rate. In addition, mailing and return
Mailed Questionnaires of questionnaires, along with any necessary fol-
low-ups to secure sufficient respondents, can be
Traditionally, the most common method of collect- time consuming (Abramson & Abramson, 1999).
ing survey data has involved the dissemination of Mail surveys usually take 3 to 4 months for
printed questionnaires through the mail to a sam- respondents to complete and return the question-
ple of respondents. Respondents are asked to com- naires in response to the three mailings (initial plus
plete the questionnaire on their own and return it to two follow-ups) that are typically required. Later
the researchers. One advantage of the mailed ques- in the chapter, strategies for follow-up are dis-
tionnaire is its relatively low cost (Abramson & cussed in more detail.
Abramson, 1999). Another advantage over inter-
views is that the questionnaire can be completed at
Directly Administered Questionnaires
the respondent’s convenience with no time con-
straint. Finally, since there is no personal contact Directly administered questionnaires have the
with an interviewer, respondent anonymity may be same advantages as mailed questionnaires, with
better preserved and potential interviewer bias is the exception that the investigator may ask for the
not a factor (Rea & Parker, 1997). questionnaire to be completed within a given time
A potential disadvantage of all questionnaires is frame. Direct administration of questionnaires has
that they require literacy skills. When studying cer- the additional advantage that it does not take the
tain disadvantaged or intellectually impaired popu- long period of time required for mailed surveys.
lations, researchers may have to deal with a lack of Finally, it is the least costly survey method.
or limited literacy. Moreover, the lack of interac- The major disadvantage of direct administra-
tion with the respondent means that any parts of the tion is sampling. Since direct administration
questionnaire that are misunderstood for any rea- requires a clustering of respondents in a specific
son cannot be clarified by the data collector. physical setting, the usual strategies for obtaining
08kielhofner(F)-08
Table 8.1 Advantages and Disadvantages of Different Methods of Data Collection in Survey Research
Questionnaires Interviews
Advantages • Relatively low cost • Greater flexibility to probe for more detail, and administer more complex
• Respondent anonymity may be better preserved questionnaires
5/20/06
• Interviewer bias is not a factor • Ensure the integrity of the questionnaire
Disadvantages • Response rate tends to be lower • Expensive (personnel and training costs)
• Any confusion about the questions cannot be clarified
• Requires literacy skills
5:32 PM
Mail Direct Online Telephone Face-to-face
Advantages • Can be completed at • Take less time • Fast • Potentially short data • Ideal for contacting hard-
the respondent’s • No mailing cost • Web-based collection period to-reach populations
Page 93
convenience • Administration can incorporate • Usually cost less • Reduces/eliminates
• No time constraint features that paper • Afford more perceived missing data
questionnaires cannot anonymity
• Data can be directly imported • Easier to sample a large
for analysis geographical area
Disadvantages • Can be time consuming • Limits sampling • Only people with computers or • Less interviewer control • Cost of travel
strategies computer skill can be • Limited ability to support • Longer data collection
• Many follow-ups may • Less flexibility in contacted questionnaires with visual period
be required time frame • Raises concerns over privacy aids • Interviewer can be a
and anonymity • Only people with telephones source of bias
can be contacted • Concerns about per-
• Opportunity to establish sonal safety of the
credibility is more limited interviewers and lack of
respondent anonymity
93
a representative sample are not possible. Thus, as interviewers can probe for more detail and
directly administered questionnaires are most use- administer more complex questionnaires. Inter-
ful when the setting where the data are collected is viewers can also maintain the integrity of the
the focus of the study and/or the persons in that ordering of the questions (whereas someone com-
setting are the population of interest. However, this pleting a questionnaire may not complete it in
is often not the case. Consequently, surveys done sequence resulting in bias) (Aday, 1996).
through direct administration are limited in their A unique advantage of telephone interviews is
generalizability. Direct administration of question- a potentially short data collection period. In some
naires is most often used to pilot (and thus refine) situations; data can be gathered and processed
the questionnaire, or to collect pilot data for a within several days. While they are more expen-
larger study that will use appropriate sampling sive than mailed questionnaires (i.e., they involve
strategies. personnel and training costs), telephone interviews
usually cost less and afford more perceived
Online Questionnaires anonymity than face-to-face interviews. In particu-
lar, telephone interviews (along with mailed ques-
While online methods of administering question-
tionnaires) offer an advantage over face-to-face
naires are relatively new, they are increasingly
interviews when the sample is distributed over a
used. There are two main methods of online
large geographical area (Rea & Parker, 1997).
administration (Bowers, 1999; Bradley, 1999;
A disadvantage of phone interviews is that the
Ramos, Sedivi, & Sweet 1998; Sheehan & Hoy,
interviewer has less control than in a face-to-face
1999). The first is an e-mail survey. In this method,
interview. The interviewer’s opportunity to estab-
the questionnaire is contained in the body of the
lish credibility is more limited over the phone.
e-mail or in a document that is sent with the e-mail
Moreover, respondents can put down the phone at
as an attachment. The respondent is requested to
anytime. A disadvantage that affects sampling is
complete the survey and return it (as an e-mail or
that only people with telephones can be contacted.
an attached completed document depending on
Thus, in some studies, constituents of the popula-
how the survey was sent). The second method of
tion of interest may be missed. Finally, there is
online administration is posted on the World Wide
limited ability to support questionnaires with
Web. Respondents are sent a request to complete
visual aids that can clarify questions in the inter-
the survey by e-mail with a link to the URL for the
view (Abramson & Abramson, 1999).
survey.
The advantages of both means of online admin-
istration are that they are fast. The Web-based Face-to-Face Interviews
administration has the potential to incorporate fea-
Survey data can be collected in person by trained
tures not available on a printed questionnaire such
interviewers. These interviews are typically con-
as pull-down menus and check-boxes. An addi-
ducted at the respondent’s residence. However,
tional benefit of the Web-based survey is that data
other sites such as schools, workplaces, or clinics
can typically be exported directly to a statistical
may also be used. This method has many of the
analysis package, bypassing the need to do manual
same strengths as phone interviews. Face-to-face
data entry.
interviews are ideal for contacting hard-to-reach
The main disadvantage of this method of ques-
populations, for example, homeless or incarcerated
tionnaire administration is that it requires mini-
criminal offenders and they are usually completed
mally access and knowledge of computer use on
with little or no missing data.
the part of the respondent. In the case of Web-
Some disadvantages of face-to-face interviews
based administration, somewhat more skill on the
are the high costs (for travel as well as personnel
part of the respondent and substantial technologi-
and training), and the time involved in traveling for
cal skill on the part of the researcher is required.
and doing the data collection period. Despite the
For these reasons, there can be particular problems
potential advantages of face-to-face interviews, the
of sampling or nonresponse bias. There can also be
interviewer can also be a source of bias. For exam-
substantial concerns about privacy and anonymity
ple, the face-to-face encounter may influence the
of the respondent.
respondent to seek approval from the interviewer.
Moreover, sex, age, or cultural differences may
Telephone Interviews
bias the respondent. Finally, face-to-face inter-
Some surveys collect data through interviews con- views can raise concerns about personal safety of
ducted via telephone by trained interviewers. All the interviewers and lack of anonymity for the
interviews have the advantage of greater flexibility respondent (Abramson & Abramson, 1999).
Chapter 8 Survey Research Design 95
Selecting and Combining Data burden and survey costs (Abramson & Abramson,
Collection Methods in Surveys 1999).
The variables chosen for a survey study depend
In making a decision about which data collection on the aim and research question(s) of the study.
method to use in survey research, there are multi- Some survey research aims primarily to describe
ple considerations. In the end, the investigator particular phenomena. For example, the periodic
must select the method that best suits the research survey conducted by the National Board for the
question, the population under study, and the Certification of Occupational Therapists (NBCOT)
available resources and constraints for the study. In in the United States (National Board for Certi-
some studies, a combination of methods may be fication in Occupational Therapy, 2004) aims to
used to lower costs and improve data quality. For characterize the practice of entry-level occupa-
example, telephone interviews may be conducted tional therapists in order to guide the development
with persons who do not respond to a mailed ques- of the certification exam. This survey is guided by
tionnaire. Or, a respondent to a face-to-face inter- questions concerning what theory or models thera-
view may be asked to complete and return by mail pists use, what assessments they use, the types of
a self-administered questionnaire that requires interventions they do, and so on. In this survey, the
consulting household records or other household/ variables of interest were those that characterized
family members (Hoyle, Harris, & Judd, 2002; entry-level practice.
Siemiatycki, 1979). Other surveys aim to determine whether rela-
tionships exist between variables. In this case, the
study questions will ask whether and how vari-
Building the ables under study covary (i.e., correlate), or
whether a number of variables can be shown to
Questionnaire/Interview account for variability in a selected variable.
Each of the variables should be clearly defined
At the heart of the survey process is development both conceptually and operationally. Part of the
of the questionnaire or interview schedule (i.e., the process of clarifying each variable is identifying
written form or guide to be used by the inter- the appropriate level of data desired, (i.e., nominal,
viewer). Many of the principles for constructing ordinal, ratio, interval, as discussed in Chapter 13)
the survey data collection method are the same and corresponding format for collecting that level
whether it is a questionnaire or an interview. of data. For example, in the NBCOT survey, most
Therefore the following discussion pertains to both of the variables were nominal, such as which
unless specific points are made about one or the model of practice or which assessment was used.
other. The usual steps in building an interview or The survey asked for this information by asking
questionnaire are: therapist to name the models they used. The same
type of information might also have been asked in
• Defining and clarifying the survey variables, order to obtain ordinal data. For instance:
• Formulating the questions,
How frequently do you use the
• Formatting the questionnaire or interview sched-
biomechanical model in your practice?
ule, and
Rarely . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
• Piloting and revising the questionnaire/interview
Often . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
schedule.
Always . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3
These steps are discussed below. Deciding the variables under study and the
level of data to be collected is a critical step. It not
Defining and Clarifying Survey Variables only affects the overall quality of the study but also
determines the type of analysis that can be carried
The first step in building the survey is to identify out with the data. Hence, when doing this step,
key variables to be measured. This can be done investigators should anticipate the statistical analy-
through a review of the literature and/or interacting ses they plan to undertake.
(e.g., via focus groups or unstructured interviews)
with the target population to gain a full under-
standing of the issues to be studied. There is not an
Formulating Questions
ideal number of variables for a survey study. The Once the variables have been identified, questions
rule, “as many as necessary and as few as possi- are formulated to elicit data on those variables. For
ble,” is sound advice so as to minimize respondent a complex variable, it may be necessary to develop
Figure 8.2 Phone interviews are a frequently used method of data

collection in research.
several questions. Moreover, when several ques- was adequate?” Breaking these double-barreled
tions are asked about a variable in order to form a questions into separate questions would be more
scale, the variable can be captured with greater straightforward. For instance: “How many times
reliability and validity (see Chapters 12 and 13). did you come to occupational therapy?” followed
Questions should: by, “Do you think that was adequate?”
Sometimes a filter question is employed to find
• Have face validity, that is, they should reflect
out if the respondent has knowledge/experience of
what the investigator wants to know and be obvi-
an issue before asking him or her to answer more
ous in meaning for the respondents,
specific questions about the issue (Hoyle et al.,
• Ask about things for which the respondent can be
2002). For example, “Do you remember receiving
expected to know the answer, and
occupational therapy during your hospitalization?”
• Be clear and unambiguous, user friendly, and
If the respondent answers “yes” to this question,
not be offensive (Bradburn, Sudman, & Wansik,
then it would be appropriate to ask about the expe-
2004).
rience of occupational therapy. If not, then there
Questions must not contain assumptions that is no point in asking further about occupational
might confuse a respondent or introduce potential therapy.
bias. For example, the question, “What was your Questions may be asked in two general forms:
experience of occupational therapy during your closed questions or open questions. A closed ques-
last hospital admission?” assumes that the respon- tion provides specific response choices and there-
dent recalls which of the services received during fore allows for more uniformity in response and
the hospitalization were occupational therapy. simplifies the analysis. The following is an exam-
Complex and lengthy sentences are particularly ple of a closed question:
likely to be misunderstood by respondents. Thus,
Which of the following factors is the most
questions should be short and simple (Converse &
important in your documentation?
Presser, 1986). Questions that have two compo-
Identification of clients occupational needs . . .1
nents should be avoided; the following is an exam-
Communication to the interdisciplinary team . .2
ple of such a question: “Recalling how many times
Capturing the client’s narrative . . . . . . . . . . . .3
you came to occupational therapy, do you think it
Also, by presenting response choices, the views, more weight is given to formatting the writ-
researcher is providing the same frame of refer- ten questionnaire for interviewer convenience and
ence to all respondents. to allow the interviewer
Closed questions provide to easily record the
options to be considered The design of the question- responses. Sometimes
and therefore can act as a naire needs to allow for for telephone inter-
memory prompt (Hoyle views, the interviewer
et al., 2002). There is, making the task of reading may use a computer-
however, concern that questions, following instruc- based questionnaire that
they may force people to allows the data to be
choose among alterna- tions, and recording answers entered directly to the
tives that are predeter- as easy as possible. computer. This method
mined by the researcher avoids the extra step of
instead of answering in data entry that occurs
their own words (Converse & Presser, 1986). when responses are recorded by the interviewer on
The following is an example of an open ques- paper.
tion:
What factors influence the content Formatting Principles
of your documentation? There are a number of overall formatting princi-
Open-ended questions, such as this one, are ples that should guide questionnaire construction
useful for eliciting a more detailed narrative (Table 8.2). The first is to develop a clear, attrac-
response. While answers to this type of open- tive front cover. It should contain:
ended question may provide rich information, they • The title of the study,
may also be challenging to categorize for analysis • Directions for completion of the survey, and
(Converse & Presser, 1986). • The name of the financial sponsor and/or institu-
Investigators sometimes use open-ended tion of the principle investigator (Dillman, 1978).
questions initially with a small pilot study sample
of the population to generate the closed questions Respondents are more likely to trust a known
for the main survey (Schuman & Presser, 1981). institution rather than a named individual whom
This approach helps to generate closed questions they do not know. Adding a picture or illustration
with options that are understandable to the respon- can be informative and add interest. The back
dents. cover of the questionnaire should not have any
Often researchers use both types of questions in questions on it. It should contain only an invitation
a survey. A common approach is to give some for further comments, a statement of thanks,
response choices and then provide an “other instructions for returning the completed question-
(please specify)” category. For example: naire, and a mailing address for the survey.
Having an uncluttered appearance to the survey
Which of the following do you use to assess a is paramount. Using extra paper is preferable to
client’s occupational participation? condensing the questionnaire into fewer pages,
Formal outcome measure . . . . . . . . . . . . . . . .1 which can lead to confusion and errors (Salant &
Informal interview with the client . . . . . . . . . .2 Dillman, 1994). For clarity of reading, 12-point
Informal observation of the client . . . . . . . . . .3 type in a standard font (e.g., Arial or Times New
Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Roman) is preferable. For a questionnaire that will
Please specify be mailed to respondents who may have lowered
vision, such as elderly persons, a larger type size,
such as 14-point, should be used.
Formatting the Questionnaire
Thorough instructions must be provided so that
The design of the questionnaire needs to allow for it is clear how the interviewer or respondent should
making the task of reading questions, following indicate their response to a question (e.g., “circle
instructions, and recording answers as easy as pos- the number of your answer”), directions for com-
sible. This section should be read with Figures 8.3 pleting the questionnaire should be distinguished
and 8.4 in mind. Mailed questionnaires must be from the questions by using special typographic
clear and attractive to complete. They require for- formatting for emphasis, such as italics, bold,
matting that ensures ease of use and accuracy of parentheses, or brackets. Caution is urged when
response. With telephone and face-to-face inter- using all capital letters; this format is difficult to
SPN__________{Private}
Interviewer ID_________
ROYAL INFIRMARY OCCUPATIONAL THERAPY PROGRAM FOLLOW-UP

(Telephone interview)
Hello, my name is __________ and I'm calling from the Royal Infirmary
occupational therapy department. May I please speak with __________? We
are interviewing previous clients of the Royal Infirmary occupational therapy
service to learn more about how occupational therapy can help people to get
back to doing everyday activities following discharge from the Royal Infirmary.
We are calling you because the occupational therapy records show that you
have difficulty doing everyday activities, and we would like to ask you some
questions about your clinic experiences, health, and everyday activities.
1. In general, how would you describe your health at this time? Would you
say it is. . .
Very poor, ...................................................... 1

Poor, .............................................................. 2
Good, or.......................................................... 3
Very good ...................................................... 4
Don't know ..................................................... 8
2. Compared to other people about your age, how would you describe your
health in general? Would you say it is . . .
Worse than average, ..................................... 1

About average, or .......................................... 2
Better than average ....................................... 3
Don't know ..................................................... 8
3. If 1 is not important and 5 is very important, how important is it for some-

one your age to be able to do everyday activities independently?
Not Very
Important Important
1 2 3 4 5 8
Figure 8.3 Format of a telephone survey.
read when applied to more than a few words. “What is your age?” rather than, “Age?” The ques-
There need to be very clear instructions about tions should all start at the left margin. All
negotiating “skip patterns,” in situations in which a response choices should be indented and all start at
certain subset of questions may not apply to some the same place. Leader dots can be helpful for
respondents (e.g., because they apply only to one visually linking the response choices with the
sex or a particular age group). Figure 8.4, question numerical codes. All the parts of the same question
4 illustrates how a skip pattern can be indicated in and its response choices should be on the same
a questionnaire. page, never split between two pages. Response
Each question should be assigned a number categories should be presented in a vertical list for-
sequentially throughout the questionnaire. The mat rather than a horizontal format (Bradburn et
questions should be written out in full rather than al., 2004). Each response choice should be
using a one-word variable label, for example, assigned a numerical code that will be circled by
SPN__________{Private}
ROYAL INFIRMARY OCCUPATIONAL THERAPY SERVICE

SURVEY OF HEALTH AND EVERYDAY ACTIVITY BEHAVIOR
1. In general, how would you describe your health at this time?
Very poor, ...................................................... 1

Poor, .............................................................. 2
Good, or......................................................... 3
Very good ...................................................... 4
2. Compared to other people about your age, how would you describe your
health in general?
Worse than average, ..................................... 1

About average, or .......................................... 2
Better than average ....................................... 3
3. What was your main reason for attending occupational therapy?
Difficulty doing self care ................................ 1

Difficulty doing work tasks ............................. 2
Difficulty doing leisure activities ..................... 3
Other (Please specify) ................................. 4
4. Do you feel satisfied with your occupational therapy experience?
Yes...............................1
No................................ 2
Figure 8.4 Format of a mailed survey.
the respondent or interviewer to record each questions that have obvious relevance to the
response. These numerical codes are likely to yield topic of the survey (Dillman, 1978). Following
fewer mistakes than using check boxes when cod- these few introductory questions, the main
ing and processing the completed questionnaire study questions are presented. Usually issues
for data entry (Dillman, 1978). related to a respondent’s beliefs, behaviors, and
Creating a vertical flow by aligning response attitudes are explored related to the study topic.
codes along the right-hand margin helps to reduce Topically related subjects should be clustered
the number of errors. Vertical flow and a generous together.
use of spacing and indentation give the survey an Some respondents regard questions about their
uncluttered, user-friendly appearance. For a series demographic background as invasive, and the rele-
of items that share the same root question and the vance of such questions to the study topic is often
same response choices, an efficient format is illus- not apparent if they are asked early in a question-
trated by Figure 8.5, question 19. naire. Therefore, questions about the respondent’s
demographic background are often placed at the
end of the questionnaire, except where it is neces-
Sequencing Questions
sary to screen a respondent’s eligibility to answer
The sequence of questions is also important. a particular group of subsequent questions (e.g.,
Every questionnaire should start with a few easy only questions about recent therapy outcomes
Table 8.2 General Principles of Survey “funnel” principle, which is used frequently, starts
Formatting with general questions followed by ones that
become increasingly more specific (Hoyle et al.,
• The front cover should be clear and attractive.
• The back cover of the questionnaire should not
2002). It is also important to avoid a potential
have any questions. question sequence effect (Tourangeau & Rasinski,
• The survey should have an uncluttered 1988). That is, one should avoid asking a question
appearance. sequence in which a previous question will likely
• A 12-point type font is preferable. bias the response to the next question.
• Questions need an assigned number sequentially.
• Clear instructions need to be provided. Formatting Questions for
• Questions need to be written out in full rather than
one word.
Different Scales
• The questions should all start at the left margin. By their nature, nominal and ordinal scales have
• All the parts of a question should be on the same finite categories. For example, a question using a
page.
nominal scale is:
• Response categories should have a vertical
response format. What is your sex?
• Response choices should have a numerical code Male . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
to be circled. Female . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
• Space generously to avoid a cluttered look.
The respondent can respond only by choosing
one of the two finite categories.
would be asked of respondents who have received Nominal categories may be randomly listed to
therapy recently). eliminate the possibility of any sequencing effect.
It is helpful to have smooth transitions between Other alternatives for nominal categories are to list
topics. This may be achieved by presenting ques- them in descending order, starting with the ones
tions in a chronological order when appropriate, that are likely to be chosen most frequently, or to
by using section headings, or by inserting a brief list them alphabetically.
statement introducing a group of questions about a An ordinal scale asks the respondent to choose
new topic. This is especially important for sensi- one of a number of finite categories. The following
tive questions that may provoke embarrassment, is an example of an ordinal scale that asks the
be viewed as private or personal, or ask about ille- respondent to choose one of three ratings on an
gal behaviors. Respondents will be more likely to ordinal scale of importance.
respond to such questions if there is an appropriate How important is your relationship with
context given for the questions and the respondent your client to the outcomes of therapy?
can see the relevance of the questions to the study Not so important . . . . . . . . . . . . . . . . . . . . . . . 1
purpose. Important . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Within a topical area of the survey, one should Extremely important . . . . . . . . . . . . . . . . . . . . 3
structure the questions in a logical sequence. The
19. What kind of occupational therapist support would you find helpful?
Very Somewhat Not at all

helpful helpful helpful
a. Self-help materials.................. 1 2 3 8
b. A lifestyle class....................... 1 2 3 8
c. An activity support group........ 1 2 3 8
d. One-to-one activity coaching.. 1 2 3 8
e. Something else (specify)....... 1 2 3 8
Figure 8.5 A sample question from an occupational therapy survey.

12. The following are some statements about how occupational therapists
(OT) might deal with clients who are having difficulty doing everyday
activities. For each statement, please indicate if you strongly disagree,
disagree, agree, or strongly agree.
Strongly Strongly
Disagree Disagree Agree Agree
a. My OT should advise me how

to engage in daily activity......... 1 2 3 4
Figure 8.6 An example of
Likert scales from a survey of b. My OT should teach me how
occupational therapy practice. to engage in daily activity......... 1 2 3 4
Ordinal responses should be listed in logical and respond to the questions. Piloting can also
order. Listing ordinal responses from lowest to determine in an interview-based survey whether
highest (as in the example above) not only makes the interviewers will be able to convey the ques-
them clearer, but also avoids the need to reverse tioning format as it is written.
the coding scheme for the analysis. Within a pilot study, investigators can gather
Responses to an interval scale may be assigned data to evaluate the intended survey instrument.
to an infinite number of possible points or to spec- Three of the most common ways of gathering such
ified ranges. Deciding on categories for interval data are focus groups, field pretesting, and individ-
scales involves judgment. For example, if asking ual interviews (Presser et al., 2004). They are dis-
respondents about their age, it is generally fine to cussed below.
ask how many years old one is, since people can
easily retrieve their age from memory. However, if Focus Groups
the question is about income, people may have
more difficulty recalling the exact amount and thus Every researcher’s views can be widened by sys-
the use of ranges such as $30,000 to $35,000 may tematic discussions within a focus group (Stewart
be preferable. The decision about using actual & Shamdasani, 1990). Focus groups ordinarily
amounts versus ranges depends on consideration involve a small group of persons who represent the
of this factor along with how important it is to have range of characteristics expected in the study sam-
exact information versus information character- ple. The investigator guides the group through
ized by a range. When using ranges, one should a discussion that aims to elicit every member’s
use boundaries that conform to traditional rounded opinion.
breaking points. For example in asking for years of Focus groups can be used during the initial
experience it is better to ask: “0–4 years, 5–9 planning of the investigation to help define the key
years” rather than “0–4.5 years, 4.6–9.5 years.” study. They can also be used to evaluate the ques-
Scaled response mechanisms, such as Likert tions once they have been developed into the ques-
scales, have response choices to elicit opinions. tionnaire. Focus groups can provide information
Likert scales are bipolar, ranging from the most about the complexity of what is being asked
negative point at one end of a continuum to the and how people understand the terms in the ques-
most positive point at the opposite end, for exam- tionnaire. In some instances, focus groups can
ple, using an “agree/disagree” continuum. The involve a question-by-question review of the ques-
questions should be focused on one issue or tionnaire.
domain. The response choices should be balanced,
with an equal number of similar points ranging Field Pretesting
from low to high, for example, “strongly agree,”
For telephone or face-to-face interviews, investiga-
“agree,” “disagree,” and “strongly disagree” (e.g.,
tors ordinarily will conduct a small number of
Figure 8.6, question 12).
interviews (usually about 15 to 30), with people
who are similar to those who will be respondents
Piloting the Questionnaire in the planned survey. For mailed questionnaires,
Piloting of the questionnaire is essential before the respondents who are from the study population are
main study. By doing a pilot study, an investigator asked to complete the questionnaire. There can be
can find out if respondents can understand the some debriefing questions at the end of the ques-
questions and if they can reasonably understand tionnaire asking for feedback on such factors as
what was unclear or confusing. Field pretests also the target population’s characteristics (parameters).
provide critical information on the practical A census is a special case, in which survey meth-
aspects of administering the survey tool that allows ods are used to collect data from or about every
unforeseen problems to be addressed before doing member of a population, that is, a 100% sample.
the larger study (Fowler, 1995). A valuable tool for The main reason for collecting data from a
assessing these aspects is to conduct a debriefing sample instead of from an entire population is that
interview with pretest interviewers. it is much less expensive. Also, when available
funding is fixed, resources can be allocated to col-
Debriefing Interview lect information about more variables than would
be possible in a survey of the entire population.
A debriefing interview with respondents about
Another important reason for sampling is that it
their understanding of questions can be helpful
usually reduces the data collection period, making
(Lessler & Tourangeau, 1989). The focus of the
the study findings available for dissemination
interview is to find out the respondent’s reactions
and application much sooner. Finally, in cases in
to the questions. The debriefing interview involves
which gaining access to population members
three steps:
requires special strategies (e.g., negotiating access
1. Asking the questions or providing the question- to worksites, or screening randomly composed
naire, numbers for a telephone interview survey), the
2. Allowing the respondent to answer the ques- strategy can be done more effectively and effi-
tionnaire or interview, and ciently with a sample than with the entire popula-
3. Asking the respondent what was going though tion.
his/her mind during the process. Some survey samples are selected by nonprob-
ability methods (e.g., selecting respondents who
An investigator may also ask respondents to: are convenient or who have volunteered). Selecting
• Paraphrase their understanding of the questions, a sample using such procedures, which depend on
• Define the terms used in the interview, and/or subjective judgments (by the respondents or by the
• Identify any confusion or concern. researcher) about who should be included, may
result in a sample that
Debriefing interviews constitutes an unrepresen-
are very helpful in identi- A hallmark of a well tative (biased) population
fying how the respondent subgroup. Although such
experiences the question- designed survey study is a sample may be useful
naire or interview process that the respondents are for an exploratory or pilot
and whether anything study about a new topic
needs to be changed to selected randomly from the
or new population, great
improve the accuracy of target population. caution must be exercised
the data they yield. in generalizing from a
nonprobability sample to
a target population. There are no systematic meth-
Implementing a Survey Study ods to account for possible selection bias. Also,
standard statistical methods (e.g., confidence inter-
The procedures used in implementing a survey vals) that are based on a random sampling theoret-
study will have a large impact on the rigor of the ical model cannot be applied appropriately to data
investigation (Fowler, 2002). Three key factors from a nonprobability sample.
that influence study rigor are: Consequently, a hallmark of a well designed
survey study is that the respondents are selected
• Sampling,
randomly from the target population. Random
• Response rates, and
(probability) sampling, in conjunction with good
• How the survey is carried out.
questionnaire design and data collection proce-
Each is discussed in the following sections. dures, provides the foundation on which reliable
estimates of population characteristics may be
derived from sample data.
Sampling Strategies
In random sampling, the selection of each
As noted at the beginning of this chapter, most sur- respondent is independent of the selection of any
vey data are collected from a sample of a relatively and all others. Thus, for example, a person would
small number of members of a target population. not be included automatically in a sample because
These sample data are used to make estimates of someone else in his or her household (e.g., spouse)
is selected. Each population member has a unique, number from 1 to N (N ⫽ the number of elements).
independent chance of being selected. That chance Then n (n ⫽ the desired sample size) elements
must be greater than zero, and it must be known or are identified to be in the sample by referring to a
it must be possible to calculate it. Although it often random number source (such as a random number
is desirable that the probability of selection is the table, or a random number generator on a calcula-
same for each population member, this is not an tor or computer), from which the researcher selects
essential aspect of random sampling. In cases in n unique numbers corresponding to elements on
which the probability of selection varies (e.g., in a the sampling frame. When sampling without
disproportionate stratified sampling design), the replacement, once an element’s number is
researcher must calculate weights to adjust for this selected, it is set aside and not used again, that is,
in the analysis. it is ignored if it is derived more than once from
The basic model underlying sampling theory the random number source.
and inferential statistics is simple random sam- In many cases, the sampling process is greatly
pling with replacement (i.e., returning the respon- simplified by using systematic random selection,
dent selected to the population after sampling so which is used commonly in survey research (Levy
that he or she has an equal change of being & Lemeshow, 2003). Again, every element on the
selected subsequently) (Cochran & Williams, frame is assigned a unique number from 1 to N.
1973). In common survey practice, however, it is Next, a selection interval (k) is calculated by divid-
not practical or desirable to collect data from the ing the population size by the sample size: k ⫽ N/n.
same population members more than once. For example, to select a sample of 200 elements
Therefore, virtually all survey samples are selected from a population of 1600, k ⫽ 1600/200 ⫽ 8.
without replacement. The first step in selecting any Then, after selecting a random starting point in the
random sample is to obtain or compile a list of all interval from 1 to k (1 to 8 in this example), the
(or as near as possible) of the members of the tar- researcher selects that first element from the sam-
get population (e.g., all the members of the British pling frame and every k element thereafter. Thus, if
College of Occupational Therapists). This list is the random starting point is 4, using a selection
called the sampling frame. In simple random sam- interval ⫽ 8, the sample will consist of elements on
pling without replacement, every element (popula- the sampling frame that previously were assigned
tion member) on the frame is assigned a unique the numbers 4, 12, 20, 28 …1596 (Figure 8.7).
Step 1: Assign sequential numbers to each element in the sampling frame.
Step 2: Select a sampling interval.
Selection Interval (k ) = Population Size/Sample Size
So for a sample size of 200 from a population of 1,600
k = 1,600/200 = 8
Step 3: Select a random starting point.
In this example the random starting point is 4.
Step 4: Select every 8th element in sampling frame.
4 Random Start Point

]8
12
]8
20
]8
28 and so on. . .
Figure 8.7 The steps for systematic random selection.

Before using systematic random selection, ipation, nonresponse bias depends on the extent to
however, the sampling frame must be assessed for which nonrespondents differ systematically from
any preexisting periodic arrangement of elements respondents in terms of their characteristics on
that might coincide with the selection interval. key variables under study (Fowler, 2002; Groves,
This is because a random procedure is used only 1989; Kviz, 1998). In such cases, the absence of
once, when the first selection point is identified. nonrespondents from the analysis may cause sur-
Thereafter, all the other elements selected in the vey estimates of population parameters to be lower
sample are identified by their relative position or higher than their true value. For example, if per-
on the sampling frame. For example, if for sons who engage in risky behaviors are less will-
some reason every eighth person listed on the sam- ing than others to participate in a survey about
pling frame in our example (starting with person health risks, then the estimates of the prevalence of
number 4) happens to be a certain sex or in a those behaviors based on data from the survey
certain occupational category, the sample may be sample will be too low (Tourangeau & Smith,
biased. Fortunately, in most studies potential 1996).
periodicity bias may be avoided effectively by Availability is another source of nonresponse,
first arranging the sampling frame in a random especially in telephone and face-to-face interview
or near random order, usually alphabetically by surveys. If the data collection times are within
surname. working times then it will be challenging to gather
A stratified random sample may be selected data on a working population. Less educated peo-
using information that is already available about ple and those older than 65 years of age are less
the population members to divide them into sub- willing to be interviewed in a random-digit tele-
groups (strata) that are of special interest for the phone procedure (Groves & Couper, 1998). Even
study purpose. For example, a sample of occupa- with a relatively high response rate, there still is a
tional therapists might be stratified according to potential for nonresponse bias if the reason for
length of employment, which would be available nonresponse is related strongly to the survey topic.
from employee records. Then a separate random Therefore, it is important that efforts are put into
sample would be selected from within each of the reducing nonresponse and/or comparing any sys-
length of employment strata. For example, three tematic difference between respondents and nonre-
samples might be selected, one each from among spondents.
those employed less than 1 year, 1 to 4 years, and Reducing nonresponse rates in telephone and
5 years or longer. The main advantages of stratifi- face-to-face interview surveys involves ensuring
cation is that it ensures that each subgroup will be access and gaining cooperation (Groves & Couper,
included appropriately in the sample and there will 1998). Access can be increased by making multi-
be more stability (less sampling error) across all ple calls and by ensuring these happen at varied
possible samples of the same size. Both of these times including evenings and weekends. An aver-
aspects contribute toward obtaining more precise age of 10 calls is usually made before deciding that
estimates of population parameters from the sam- the person being called is a nonrespondent. Having
ple statistics. Several strategies may be used to interviewers who have flexible schedules and can
decide how to construct strata and to determine the make appointments at the respondents’ conven-
number of elements to select from each stratum. In ience enhances access to the sample. Cooperation
some cases, stratification involves giving popula- can be elicited by an advanced letter clearly stating
tion members unequal probabilities of selection. the purpose of the study and the content of the
Accordingly, the researcher must adjust for this in interview. Being clear at the start of the interview
the analysis by weighting the data (Cochran & of the purpose of the questions and reassuring the
Williams, 1973). respondents their information is important to the
outcome of the research will support cooperation.
Training interviewers who understand the impor-
Response Rate
tance of good response rates and who can handle
An important aspect of the quality of the data col- challenging questions with sensitivity is essential.
lected from a representative sample is how suc- Reducing nonresponse in a mailed survey involves
cessful a survey is in obtaining cooperation from developing a well presented questionnaire and
the persons selected into the sample. As noted at sending multiple mailings to nonrespondents.
the beginning of the chapter, failure to collect data Three mailings of the questionnaire and a cover
from a high percentage of the sample is a major letter are recommended about one month apart
source of potential survey error, called nonre- (Dillman, 2000). If questionnaires are mailed to a
sponse bias. In addition to the degree of nonpartic- general population without appropriate follow-up
techniques, the response rate is likely to be less Carrying Out the Survey
than 50% (Edwards et al., 2002; Heberlein &
Baumgartner, 1978). Face-to-face interviews afford the opportunity for
The response rate is an important mechanism an advance letter that informs the potential respon-
for assessing the potential for nonresponse bias. In dent of the details of the study. Telephone random
general, the response rate is calculated as the num- digit dialing is always a cold contact. However,
ber of sample members from whom a completed other telephone surveys may afford the opportu-
questionnaire is collected, divided by the number nity to send a letter prior to the call to increase
of sample members who are eligible to participate response rates. Mail surveys can send an advance
in a particular survey, expressed as a percentage letter that explains a questionnaire will be mailed
(Kviz, 1977). However, special considerations to respondents shortly. Cover letters are critical to
often must be taken into account depending on response rates of mailed survey (Salant & Dillman,
various aspects of any particular survey. Although 1994). It is the only opportunity the researcher has
there still is no generally accepted standard for to anticipate and deal with the respondents’ ques-
computing response rates, many survey profes- tions and concerns about participating in the sur-
sionals have adapted the guidelines set forth by vey. The cover letter should be printed on
the American Association for Public Opinion letterhead stationery, and have the mailing date
Research (2004), which are available from the and the name and address of the anticipated
organization’s Web site along with a response rate respondent. It should include at a minimum why
calculator that may be downloaded free at the study is important, why the respondent’s
http://www.aapor.org. answers are important, assurance of confidentiality
Response rates vary widely across the various of their answers, and whom to contact if they have
survey data collection modes, and even across questions about the survey. The letter should con-
studies using the same mode. In general, response clude with a thank you and be signed with an orig-
rates are highest and quite similar for the two inter- inal signature whenever possible.
view survey methods, with response rates tending Telephone and face-to-face interviews obtain
to be about 5 percent higher for face-to-face inter- higher cooperation rates in the evenings and week-
views than for telephone interviews. Response ends, which is when most respondents are likely to
rates generally are lower for mailed questionnaires be available for interview (Weeks, Kulka, &
(Aday, 1996). A review by Goyder (1985) found Pierson, 1987). Interview schedules need to be
that response rates for mailed questionnaires developed to reflect when the sample can be
using the strategies we recommend here range accessed. Interviewers generally are instructed to
from about 30 percent to as high as 80 percent or allow a phone to ring up to 10 rings to allow
higher, with the 60 to 70 percent range being respondents sufficient time to pick up the phone.
regarded by some survey experts as a realistic Leaving a message on an answering machine or
average expectation for a well implemented mail voice-mail system usually is not advised unless
survey. Response rates for “typical” face-to-face this will be the last attempt to contact a respon-
interview surveys range from about 70 to 90 per- dent. It is best for the interviewer to make direct
cent or higher (Goyder, 1985). Czaja and Blair contact. Few respondents return a call in response
(2004) suggest the following response rates may to a message left by an interviewer.
be expected for most typical surveys: 65% to 95% All forms of survey need appropriate follow-up
for face-to-face interviews, 60% to 90% for tele- procedures to maximize response rates. The
phone interviews, and 45% to 75% for mailed majority of face-to-face interviews are carried out
questionnaires. within six contact attempts (Kalsbeek, Botman,
There is no agreed-upon standard minimum for Massey, & Lui, 1994). The number of follow-ups
an acceptable response rate. Some have suggested is dependent on cost. A careful cost–benefit analy-
50% to 60% as adequate (Rea & Parker, 1997) sis should be completed before deciding how
while others have suggested above 70% is accept- many repeat calls the interviewers make before
able (Fowler, 2002). Of course, as discussed ear- assigning a nonresponse status to a sample unit.
lier, in addition to the response rate, any It is more cost effective to complete contacts to
assessment of potential nonresponse bias must nonrespondents in telephone interview surveys. If
consider whether and to what extent nonres- there is a busy signal, for example, it has been rec-
pondents are likely to differ from respondents in ommended to call back a maximum of three times,
terms of key variables under study. However, even 3 minutes apart (Survey Research Laboratory,
with a 90% response rate there may be some non- 1987). When the sample is selected from a list, a
response bias. minimum of 10 call backs at different times on dif-
Table 8.3 Procedure for Administration of Mailed Surveys

Timeframe Action
Before survey is sent Mail a personalized, advanced letter to everyone in the sample.
Round 1 Mail personalized cover letter with more detail of the study, a
questionnaire and a stamped, addressed return envelope.
4–8 days later Send a follow-up postcard or letter to thank everyone who has
responded and ask those who have not done so to respond.
Round 2 To nonrespondents send a new personalized cover letter,
4 weeks after first questionnaire was sent replacement questionnaire, and stamped self-addressed
envelope.
Round 3: To nonrespondents send a new personalized cover letter,
8 weeks after first questionnaire was sent out replacement questionnaire and stamped address envelope.
End data collection
12 weeks after first questionnaire was sent out
ferent days is usually completed to nonrespon- approved by an ethical board (see Chapter 29) that
dents. When working with a random digit dialing judges the procedure to be warranted in light of
sample, where there is a chance for more nonresi- the potential respondents’ rights to consent and to
dential telephone numbers to be included in the confidentiality.
sample, most survey research organizations
attempt a minimum of about 15 calls.
Mailed surveys require a systematic approach Preparing for Data Analysis
to administration to ensure acceptable response
rates (Table 8.3). Follow-up mailings to everyone The formal process of gathering survey data has
in the sample (even those who have responded) are been outlined earlier. The data this process gener-
not efficient and may confuse or irritate those who ates must now be presented in an understandable
have already responded. Targeted repeated admin- format. During the planning stage of the research,
istrations can be achieved if there is a way of iden- the researcher should know how the data will be
tifying those who have not responded. This can be analyzed to meet the objectives of the study. This
achieved if a sample point number (SPN) is placed process of thinking forward to the analysis phase
at the top right hand corner of the questionnaire. early on can often identify and allow correction of
This number corresponds to the respondent’s entry gaps in the data that will be obtained through the
in the sampling frame, which should be kept in a questionnaire.
locked physical file and/or in a password-protected Survey data are usually entered into a computer
computer file to protect respondent confidentiality. data file for statistical analysis. Each statistical
It is good practice to tell people what the number program has different criteria in how the data
is for in the covering letter. If it is important that should be formatted. To facilitate computer pro-
the respondent not be identifiable even to the cessing and analysis, codes must be assigned to all
researcher then a postcard procedure can be responses on a completed questionnaire. There
employed (Fowler, 2002), whereby a postcard should be clear rules as to which numbers are
bearing the respondent’s SPN is enclosed with the assigned to which answers on the survey. A code
questionnaire along with a request for the respon- book should be developed that clearly indicates
dent to mail it separately when he or she mails which codes are reflective of which questionnaire
back the completed questionnaire. The text of the answers and which column the questionnaire can
postcard states: “Dear researcher, I am sending this be found in the electronic dataset (Figure 8.8).
postcard at the same time that I am sending my If the coding is complex, detailed coding
completed questionnaire. Since the questionnaire instructions will need to be developed. These need
is completely anonymous (the SPN is not recorded to be clear in order to ensure coding reliability.
on the questionnaire in this case) this will let you Codes need to be assigned to missing data to iden-
know I have returned my questionnaire.” This pro- tify if the respondent refused to answer a question,
cedure maintains anonymity while enabling the left a response blank inadvertently, a question was
researcher to track who has responded to the sur- not applicable to the respondent, or the respondent
vey. Whatever procedure is used, it must be did not know the information requested by a ques-
Variables in Export Sequence

Column Location Content of Column
1-3: subject identifier (001-999) (3 numeric code)
4-6: repeat code (001-999) (3 numeric code)
7-8: translation used (1 character code/1 numeric code)

missing data: 09
9-11: country in which data was collected code (3 character

code) missing data: 009
12-13: age (2 numeric code)

missing data:99
14: gender (M or F)
missing data: 9
15-17: nationality (3 character code)

missing data: 009
18: ethnicity (single numeric code)

missing data: 9
19: years of education (single numeric code)

missing data: 9
20-21: degree earned (2 character code)

missing data: 09
22-23: employment status (2 character code)

missing data: 09
24-25: living situation (2 character code)

missing data: 09
26: independence in occupational behavior (1 character

code) missing data: 9
27-28: major disabling condition category (2 character code)

missing data: 09
29-32: specific major disabling condition (4 numeric code)

missing data: 0009
Figure 8.8 An example of the demographic information section of a

code book.
tion. Codes need to be generated for open ques- methods include using a well developed interface
tions, or for “other” choices, where the responses between the computer and the data entry person-
were not predictable. The researcher identifies cat- nel. Computer programs (e.g., ACCESS, SPSS,
egories that emerge as themes in the answers to the SAS, Epi-Info) now can develop data entry screens
open question. to ease the process of entering data. These pro-
Quality control procedures include having grams can be set to accept only a certain range of
trained coders, independently checking each codes in a particular field, thereby reducing error
coder’s work, and coders should write notes about codes in a field. The data can be entered twice in
codes of which they are not sure so that they can two different files and then the files can be corre-
be checked by the supervisor. Other quality control lated to identify potential errors in data entry. This
can be expensive and therefore a 10% random DePoy, E., & Gitlin, L. (1998). Introduction to research:
sample can be entered in twice to check accuracy. Understanding and applying multiple strategies. St.
Louis: Mosby.
The rate of error from data entry should be less Dillman, D. A. (1978). Mail and telephone surveys: The
than 1% (Fowler, 2002). total design method. New York: John Wiley & Sons.
Precoded forms that can be scanned electroni- Dillman, D. A. (2000) Mail and Internet surveys: The tai-
cally are becoming used more often. This option lored design method. New York: John Wiley & Sons.
Edwards, P., Roberts, I., Clarke, M., DiGuiseppi, C.,
may not be financially viable at present for smaller
Pratap, S., Wentz, R., et al. (2002). Increasing response
surveys; however, this may become an option in rates to postal questionnaires: Systematic review.
the future with the reduction in cost of the equip- British Medical Journal, 324(7347), 1183–1191.
ment. Current options are discussed in Bloom and Fowler, F. (1995). Improving survey questions. Applied
Lyberg (1998). Also, as noted earlier, survey social research methods series (Vol. 38). Thousand
Oaks, CA: Sage.
researchers are increasingly exploring effective Fowler, F. (2002). Survey research methods. Applied social
ways of collecting survey data using the Internet research methods series (Vol. 1). Thousand Oaks, CA:
(Couper, 2000; Dillman, 2000) Sage.
Goyder, J. (1985). Face-to-face interviews and mailed
questionnaires: The net difference in response rate.
Conclusion Public Opinion Quarterly, 49, (2, summer), 234–252.
Groves, R., & Couper, M. (1998). Nonresponse in house-
hold interview surveys. New York: John Wiley & Sons.
Surveys allow for the systematic collection of Groves, R. M. (1989). Survey errors and survey costs. New
information from a sample of people to generate York: John Wiley & Sons.
an understanding of the population from which the Heberlein, T., & Baumgartner, R. (1978). Factors affecting
response rates to mailed questionnaires: A quantitative
sample was drawn. Completing a rigorous survey
analysis of the published literature. American
study requires careful attention to building the sur- Sociological Review, 43, 447–462.
vey, administering the survey, and processing the Hoyle, R. H., Harris, M. J., & Judd, C. M. (2002).
data. If these procedures are followed, the sum- Research methods in social relations (7th ed.). Pacific
mary statistics can be generalized to the population Grove, CA: Wadsworth.
Kalsbeek, W., Botman, S., Massey, J., & Lui, P. (1994).
under study, which is the aim of survey research. Cost efficiency and the number of allowable call
attempts in the National Health Interview Survey.
REFERENCES Journal of Official Statistics, 10, 133–152.
Abramson, J., & Abramson, Z. (1999). Survey methods in Kviz, F. J. (1977). Toward a standard definition of response
community medicine (5th ed.). New York: Churchill rate. Public Opinion Quarterly, 41, 265–267.
Livingstone. Kviz, F. J. (1998). Nonresponse in sample surveys. In T.
Aday, L. (1996). Designing and conducting health surveys. Cotton & P. Armitage (Eds.), Encyclopedia of biostatis-
San Francisco: Jossey-Bass. tics. Chichester, UK: John Wiley & Sons.
American Association for Public Opinion Research (2004). Lessler, J.,Tourangeau, R., & Salter W. (1989, May).
Standard definitions: Final dispositions of case codes Questionnaire design research in the cognitive research
and outcome rates for surveys. Retrieved in March laboratory. Vital and Health Statistics Series 6, No.1;
2005, from http://www.aapor.org. DHHS Publication No. PHS-89-1076. Washington,
Bloom, E., & Lyberg, L. (1998). Scanning and optical DC: US Government Printing Office.
character recognition in survey organizations. In M. Levy, P. S., & Lemeshow, S. (2003). Sampling of popula-
Couper, R.P. Baker, J Bethlehem, C.Z.F. Clark, J. tions: Methods and applications. New York: John
Martin, W. Nicholls, & J.M. O’Reilly (Eds.), Computer Wiley & Sons.
assisted information collection (pp. 449–520). New National Board for Certification in Occupational Therapy.
York: John Wiley & Sons. (2004). A practice analysis study of entry-level occupa-
Bowers, D. K. (1999). FAQs on online research. Marketing tional therapist registered and certified occupational
Research, 10(1), 45–48. therapy assistant practice. OTJR, Occupation,
Bradburn, N. M., Sudman, S., & Wansik, B. (2004). Asking Participation and Health, Spring, Volume 24, Suppl. 1.
questions. San Francisco: Jossey-Bass. Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin,
Bradley, N. (1999). Sampling for Internet Surveys: An J., Rothgeb, J. M., et al. (2004). Methods for testing
examination of respondent selection for Internet and evaluating survey questions. Public Opinion
research. Journal of the Market Research Society, Quarterly, 68 (Spring), 109–130.
41(4), 387–395. Ramos, M., Sedivi, B. M., & Sweet, E. M. (1998).
Cochran, W.G., & Williams, G. (1973). Sampling tech- Computerized self-administered questionnaires. In M.
niques. New York: John Wiley & Sons. P. Couper, R. P. Baker, J. Bethlehem, C. Z. E. Clark, J.
Converse, J., & Presser, S. (1986). Survey questions: Martin, W. L. Nichols, & J. M. O’Reilly (Eds.),
Handcrafting the standardized questionnaire (2nd ed.). Computer assisted survey information collection (pp.
Beverly Hills, CA: Sage. 389–408). New York: John Wiley & Sons.
Couper, M. P. (2000) Web Surveys: A review of issues and Rea, L., & Parker, R. (1997). Designing and conducting
approaches. Public Opinion Quarterly, 64, 464–494. survey research: A comprehensive guide. San
Czaja, R., & Blair, J. (2004). Designing surveys: A guide to Francisco: Jossey-Bass.
decisions and procedures. Thousand Oaks, CA: Pine Salant, P., & Dillman, D. (1994). How to conduct your own
Forge Press. survey. New York: John Wiley & Sons.
Schuman, H., & Presser, S. (1981). Questions and answers Sage Publications. The survey kit. A collection of brief,
in attitudes surveys. New York: Academic Press. applied books by various authors, about survey
Sheehan, K. B., & Hoy, M. G. (1999). Using e-mail to sur- research methods. Information is available at
vey Internet users in the United States: Methodology http://www.sagepub.com. NOTE: Search for
and Assessment. Journal of Computer Mediated “survey kit.”
Communication, 4 (3). Available: U.S. Office of Management and Budget. Revisions to the
http://www.ascusc.org/jcmc/vol4/issue3/sheehan.html. Standards for the Classification of Federal Data on
Siemiatycki, J. (1979). A comparison of mail, telephone, Race and Ethnicity, (October 30, 1997). Available on-
and home interview strategies for Household Health line at http://www.whitehouse.gov/omb/fedreg/omb-
Surveys. American Journal of Public Health, 69 (3), dir15.html.
238–245.
Stewart, D., & Shamdasani, P. (1990). Focus groups. Articles
Newbury Park, CA: Sage. Couper, M. P., Traugott, M. W., & Lamias, M. J. (2001).
Survey Research Laboratory, University of Illinois. (1987). Web survey design and administration. Public Opinion
Chicago area general population survey on AIDS (SRL Quarterly, 65, 230–253.
No. 606), Interviewer Manual. Chicago: Author. Schaeffer, N. C., & Presser, S. (2003). The science of
Tourangeau, R., & Rasinski, K. (1988). Cognitive asking questions. Annual Review of Sociology, 29,
processes underlying context effects in attitude meas- 65–88.
urement. Psychological Bulletin, 103, 299–314. Schwarz, N. (1999). Self-reports: How the questions shape
Tourangeau, R., & Smith, T. W. (1996). Asking sensitive the answers. American Psychologist, 54(2), 93–105.
questions: The impact of data collection mode, ques- Books
tion format, and question content. Public Opinion Couper, M. P., & Nicholls, II, W. L. (1998). The history
Quarterly, 60, 275–304. and development of computer assisted survey informa-
Weeks, M., Kulka, R., & Pierson, S. (1987). Optimal call tion collection methods. In M. P. Couper, R. P. Baker,
scheduling for a telephone survey. Public Opinion J. Bethlehem, C. Z. F. Clark, J. Martin, W. L. Nicholls
Quarterly, 51, 540–549. II, et al. (Eds.), Computer assisted survey information
collection (pp. 1–21). New York: John Wiley & Sons.
RESOURCES DeVellis, R. F. (2003). Scale development: Theory and
applications. Newbury Park, CA: SAGE Publications.
Web Sites Fink, A. (1995). How to ask survey questions. Thousand
American Association for Public Opinion Research Oaks, CA: Sage.
(AAPOR). Best practices for survey and public opinion Fowler, F. J., Jr. (1998). Design and evaluation of survey
research. 2002. Available at AAPOR Web site: questions. In L.Bickman & D. J. Rog (Eds.), Handbook
http://www.aapor.org/default.asp?page⫽survey_meth- of applied social research methods (pp. 343-–374).
ods/standards_and_best_practices. Thousand Oaks, CA: SAGE Publications.
American Association for Public Opinion Research Harkness, J. A., van de Vijver, F.J. R., & Mohler, P. P.
(AAPOR). Code of professional ethics and practices. (2002). Cross-cultural survey methods. San Francisco:
2002. Available at AAPOR Web site: Jossey-Bass.
http://www.aapor.org/default.asp?page⫽survey_meth- Presser, S., Rothgeb, J. M., Couper, M. P., Lessler, J. T.,
ods/standards_and_best_practices/code_for_profes- Martin, E., Martin, J., et al. (Eds.) (2004). Methods for
sional_ethics_and_practices testing and evaluating survey questionnaires. New
American Association for Public Opinion Research York: John Wiley & Sons.
(AAPOR). Response rate calculator, available at: Schwarz, N., & Sudman, S. (Eds.) (1996). Answering ques-
http://www.aapor.org/default.asp?page⫽survey_meth- tions: Methodology for determining cognitive and com-
ods/response_rate_calculator. A free copy of an Excel municative processes in survey research. San
spreadsheet (AAPOR Outcome Rate Calculator Version Francisco: Jossey-Bass.
2.1) may be downloaded to facilitate computing rates Sudman, S., Bradburn, N. M., & Schwarz, N. (1996).
according to the methods described in AAPOR’s docu- Thinking about answers: The application of cognitive
ment, “Standard Definitions: Final Dispositions of Case processes to survey methodology. San Francisco:
Codes and Outcome Rates for Surveys.” Jossey-Bass.
American Statistical Association, Survey Research Tourangeau, R., Rips, L. J., & Rasinski, K. (2000) The psy-
Methods Section. Proceedings. Online proceedings of chology of survey response. New York: Cambridge
the American Statistical Association Survey Research University Press.
Methods Section from 1978 to present. Also includes Willis, G. B. (2005). Cognitive interviewing: A tool for
papers from the Joint Statistical Meetings and some improving questionnaire design. Thousand Oaks, CA:
papers from the American Association of Public SAGE Publications.
Opinion Research meetings. More than 3000 papers in *Note: Many universities have expertise in survey design
all. http://www.amstat.org/sections/srms/proceedings/ and can be consulted when creating a survey.
C H A P T E R 9
Study Designs for Secondary

Analysis of Existing Data
Sherrilene Classen
Secondary analysis involves the use of data that

were gathered for previous research studies or
Definition of
other primary purposes (e.g., medical records, Secondary Analysis
databases from surveys). The main purpose of
using this type of research is to describe or explore Secondary analysis involves using available infor-
phenomena that may lead the researcher to ask mation. As noted earlier, this information may be
more specific questions, to generate hypotheses, or available in existing records that were generated
to engage in prospective research (gathering of for purposes of documenting health care. Another
data at some point in the future). important source of data for secondary analysis is
For example, consider a researcher who wants existing databases (i.e., data that were originally
to determine if there is a relationship between med- collected for another study and are now available
ication use and functional status of older adults. for further analysis). When using records for sec-
Such a researcher could begin to answer this ondary analysis, the investigator observes, records,
question by exploring the medication and func- classifies, counts, and analyzes this existing infor-
tional status data from existing records in a nursing mation. In the case of existing databases, data are
home or rehabilitation facility. If the researcher already organized into a format for analysis, so the
found negative associations between sedatives and investigator chooses what data to access and how
cognitive functioning, it would be possible to ask to analyze those.
more specific questions (e.g., why is there a
negative relationship between sedatives and cogni-
tive functioning, or what is the relationship
between sedatives and instrumental activities of
Designs for
daily living [ADL] functioning in older adults?). Secondary Analysis
The researcher may then choose to engage pros-
pectively in a follow-up study to answer these Although the data already exist in both circum-
questions. stances, it is important that the researcher consid-
Secondary analysis is a valid mode of inquiry, ers the design that will be used to guide how data
widely used in such fields as public health and are extracted from existing records or databases
nursing. It is also beginning to be used in occupa- and how they will be analyzed. Secondary analysis
tional therapy. This method holds many advan- includes both descriptive and analytical designs.
tages. For example, it is cost effective, decreases Descriptive designs depict the distribution and fre-
subject burden, and is useful for exploring new quency of health conditions. Their purpose is to
areas of research. Nonetheless, many pitfalls exist identify populations at high risk for developing a
and investigators must plan for managing and condition and to formulate hypotheses for further
acknowledging those. investigation. For example, using data from a
This chapter defines secondary analysis and health survey, an investigator may wish to describe
describes how retrospective analysis and the use of the health status of people in a community. After
existing databases fit within this type of design. analyzing the data, the researcher may identify
The definitions, advantages, limitations, and appli- high-risk groups and plan to examine the risks fur-
cation of this type of research are outlined. The ther in follow-up studies.
chapter also describes different datasets, how they Analytical designs are concerned with identify-
can be accessed, their purposes, and how to man- ing or measuring the effects of risk factors or health
age their limitations. conditions, and with testing hypotheses. The com-
Chapter 9 Study Designs for Secondary Analysis of Existing Data 111
Hypothesis
(inclusion of
comparison group)
Yes No
Analytic Descriptive
Figure 9.1 Decision tree for retro- Case-Control Cohort Cross-Sectional Cross-Sectional
Study Study Study
spective study designs.
mon types of analytical studies are cross-sectional, In cross-sectional studies, investigators usually
case control, and cohort. Figure 9.1 presents a deci- examine the relationship between a health out-
sion tree for designs that involve secondary analy- come and some other variables. All the variables
sis of existing data. If the researcher wants to test a under study were captured at the same point
hypothesis, the analytical path will be followed, in time and thus provide a “snapshot” of the sub-
resulting in a choice among case control, cohort, or jects under study. In this type of study, the most
analytical cross-sectional studies. If no hypothesis common measure of association used is called the
is involved and the researcher wants to describe the odds ratio (OR) as explained in the feature box
population, a descriptive cross-sectional design is below.
appropriate. The odds ratio indicates the direction (positive
or negative) and the size of the effect of those
exposed to birth anoxias who were likely to show
Cross-Sectional Design
sensory integrative disorder. To interpret the odds
Cross-sectional studies are also sometimes referred ratio adequately one needs to calculate a confi-
to as prevalence studies, when the aim is to exam- dence interval. One may say that this proposed
ine the extent to which a health behavior or condi- exposure acts as a significant risk to disease if the
tion exists among some group (e.g., the prevalence odds ratio is greater than one (as it is in this case),
of exercise behavior among physically disabled and the lower bound of the confidence interval
persons or the prevalence of acquired immunodefi- does not go below 1. This chapter does not address
ciency syndrome [AIDS] among intravenous drug the actual way of calculating odds ratios, as this
users). procedure is described in any statistics textbook, or
The Odds Ratio
The odds ratio is the ratio of two odds, which is an If such a study involved 100 subjects it might
estimate of risk. For example, consider the follow- result in the following data:
ing notation for the distribution of a binary expo-
sure (exposed or not exposed) and a binary health Outcome: Sensory integrative disorder
outcome (sensory integrative disorder/no sensory Yes No
disintegrative disorder) in a population.
Exposure: Birth anoxia Yes 43 7
Outcome: Sensory integrative disorder No 2 48
Yes No
Exposure: Birth anoxia Yes a b In this case the odds ratio is 43 ⫻ 48/ 7 ⫻ 2 ⫽
No c d 2,064/14 ⫽ 147.43. The odds ratio indicates that
someone who has been exposed to birth anoxia is
147 times more likely to have sensory integrative
The odds ratio in the example above is (a ⫻ d) / disorder, than someone not exposed to birth
(b ⫻ c) anoxia.
may be obtained by using a statistical software • Multiple factors can be evaluated for a single
program such as SPSS or SAS. disease.
The major advantages of the cross-sectional
Limitations of this design are:
design are that it:
• Various forms of bias are inherent in the study
• Is relatively easy to do,
design,
• Does not require a long time to execute,
• Information on risk factors and confounding
• Is usually less expensive than other secondary
variables may be difficult to identify,
analyses, and
• Difficult to assess whether health condition
• Provides a snapshot of the population.
causes exposure to a factor or the factor causes
The limitations of this type of study are that it: the health condition, and
• Difficult to identify and assemble case group
• Is not appropriate for studying rare health condi- representatives and identify a control group
tions or those with a short duration, (Mausner & Kramer, 1985).
• Cannot distinguish the temporal sequence of
The following is an example of a case control
cause and effect,
study. An occupational therapy researcher wants to
• May falsely classify a person with a health prob-
know what activity risk factors are associated with
lem in remission as not having the problem
repetitive motion injuries in the upper extremity.
(because the design only provides a snapshot of a
From clinics serving such persons in a given state,
person’s status at a point in time), and
the researcher obtains reports on subjects who
• Is not suitable to examine rare outcomes, or out-
have been diagnosed with repetitive motion
comes of short duration (Mausner & Kramer,
injuries (cases). In addition, the researcher solicits
1985).
records of people without repetitive motion
The following is an example of a cross-sectional injuries (controls) from general practitioner clinics
study. An occupational therapy researcher wants to in the same state. To determine the most important
determine the relationship between medications in risk factors associated with upper extremity pain,
the senior adult population and their activities of the researcher identifies the different activity expo-
daily living status. To answer this question, the sures associated with these cases and controls.
investigator analyzes an existing dataset containing
information on medication management and on the Cohort Design
functional status of a cohort of older adults. After
analysis, the descriptive statistics will provide Cohort studies may be prospective or retrospec-
information about the associations between the tive. The prospective cohort is described in
explanatory variable (types of medication taken) Chapter 7. The retrospective cohort, which is the
and the outcome variable (independence in activi- version of this design used in secondary analysis,
ties of daily living). The researcher may formulate is also called a longitudinal, incidence, follow-up,
a hypothesis from these associations that will facil- or historical cohort study. The incidence of health
itate further investigation. outcomes is determined in persons who were
exposed versus those who were not exposed to a
certain factor or factors. The major advantages of
Case-Control Design this design are that:
Also called a case-history study, case control
• The study factor (independent variable) is obser-
designs represent a true retrospective study and are
ved on each subject before the health outcome is
described in detail in Chapter 7. The researcher
observed,
compares cases with controls and determines the
• Bias, for example, recall bias, is excluded in the
outcome of interest by looking backwards in time
exposure as the researcher observes the develop-
to determine which factors may lead to a particular
ment of an outcome based not on the memory of
outcome. The risk estimate is also expressed as an
the subjects, but on the exposure, and
odds ratio. The major advantages of this design are:
• One can study additional associations with the
• It is quick and inexpensive compared to other outcome.
designs,
Limitations of this study design include:
• It is appropriate for evaluating rare health out-
comes or those with long latency periods (i.e., a • The attrition or drop-out of subjects, and
long time between a precipitating factor and the • Changes in criteria, practice standards, or meth-
outcome of interest), and ods occurring over time (Mausner & Kramer,
1985). (For example, a researcher studies the The three major retrospective studies—cross-sec-
association between multiple medication use and tional, case-control, and cohort—were described
functional limitations in a group of older adults, with examples pertaining to occupational therapy
using a 10-year-old database and realizes that practice. Table 9.1 summarizes the strengths and
many of the older subjects died or “dropped out,” weaknesses of these designs.
which greatly affects the sample size at the end
of the 10-year period. In addition, some of the
medications that were used as standard practice
10 years ago are not being used today.) Managing Bias
The following is an example of a retrospective and Confounding
cohort study. From an existing dataset, an occupa-
tional therapy researcher examines the association In retrospective analysis, the researcher does not
between medications (exposure) used by senior have control over the data because they were gen-
adults and safe driving (outcome). The researcher erated in the past. Consequently, the researcher
identifies age, race, sex, and comorbidities as must be aware of two existing sources of error that
explanatory variables. She targets senior adults, age may distort findings of the study:
65 and older, and examines their historical records
• Bias and
retrospectively for 5 years. Annually and at the end
• Confounding (Mausner & Kramer, 1985).
of the 5-year period she determines the association
between medication use and safe driving.
Bias
Summary Bias, which is systematic rather than random error,
This section discussed secondary analysis designs may occur during any stage of the study: during
consisting of descriptive and analytical designs. the design and conduct, in the analysis, and in the
Table 9.1 Summary of Advantages and Disadvantages of Retrospective Studies

Type of Study Advantages Disadvantages
Cross-sectional • Is relatively easy to do. • Not appropriate for rare health conditions or
• Can be done quickly. those with a short duration.
• Provides a snapshot of the • Cannot determine temporal sequence of
population. cause and effect.
• Is usually less expensive than • Because the design only provides a snapshot
other designs for secondary of a person’s status, a person with a health
analyses. problem in remission may be falsely
classified as not having the problem.
Case-control • Quick and inexpensive compared • Various forms of bias are inherent in the
to other designs. study design.
• Appropriate for evaluating rare • Information on risk factors and confounding
health outcomes or those with variables may be difficult to identify.
long latency periods. • Difficult to assess whether health condition
• Multiple factors can be evaluated causes exposure to a factor or the factor
for a single disease. causes the health condition.
• Difficult to identify and assemble case group
representatives and identify control group.
Cohort (historical) • The study factor is observed on • Problems of attrition.
each subject before the outcome • Changes over time in criteria and methods.
is observed.
• Lack of bias in exposure.
• Yields relative risk and incidence
rates, which are measures of
absolute risk.
• Can study additional health
outcome associations.
interpretation of the study results (Sackett, 1979). more involved impairments). If this were so, the
Two major sources of bias are: researcher may potentially not capture the informa-
tion of “more involved clients” and underestimate
• Selection bias and
the extent to which assistive devices and adaptive
• Information bias.
equipment are used by nursing home residents.
The following is an example of a selection bias This same example can be used to illustrate
in retrospective analysis. Consider a study in which information bias. Nursing home records may
an investigator wants to do retrospective analysis of contain information about assistive devices and
nursing home records to examine assistive device adaptive equipment that clients were provided.
and adaptive equipment utilization. In such a study However, this information may not be an accurate
clients who did not give consent to have their estimate of utilization if more devices and equip-
records accessed may differ systematically from ment were provided to residents than were used by
those who did give consent (e.g., they may have residents. Table 9.2 presents sources of selection
Table 9.2 Bias: Explanation, Impact, and Ways to Avoid or Manage in Retrospective Studies
Bias Explanation Impact Ways to Avoid or Manage
Selection bias
Prevalence/incidence Results from inclusion of The use of prevalent Use incident cases, or
bias (Neyman’s prevalent or existing cases can lead to an prevalent cases with a
bias) cases rather than overrepresentation of short interval between
incident or new cases. cases of long duration, the diagnosis and onset
because those who of disease/disability.
have rapidly cured, or
died, have a lower
probability of being
included.
Self-selection or Refers to the characteristics May increase or decrease Can be minimized in select-
membership of an individual that may the risk estimate. ing groups that are
bias consciously or uncon- closely related in terms
sciously affect member- of activity, occupations,
ship in a group. environment, etc.
Loss to follow-up or Most relevant in cohort May increase or decrease Compare loss group to
withdrawal bias studies and refers to the the risk estimate follow-up group on
completeness of follow- depending on whether baseline data to
up or rate of withdrawal, or not the outcome determine important
which may be different experience is different differences.
in the exposed vs. among those lost.
nonexposed.
Berkson’s bias Relevant in case-control May increase or decrease Use controls, without the
studies, refers to the risk estimate. outcome under investi-
selective factors that gation, that are equal to
lead hospital cases and the cases. For example,
controls to be different if hospital cases are
from what is seen in the used, then choose
general population. hospital controls (not
healthy controls) as a
comparison group.
Information bias
Recall bias Differences, generally Increase the risk Validate exposure informa-
among cases and estimate. tion from independent
controls, in accuracy or source; use hospital
completeness of recall controls and/or more
to memory of prior than one control group.
events or experiences.
(continued)
Table 9.2 Bias: Explanation, Impact, and Ways to Avoid or Manage in Retrospective Studies (continued)
Bias Explanation Impact Ways to Avoid or Manage
Interviewer bias The interviewer is subconsciously Increase the risk Theoretically by blinding the
or consciously gathering estimate. interviewer to the
selective information. For case/control status; use
example, an interviewer may of trained interviewers
probe for more information and standardized
about exposure among cases, interview form; re-
because of the expectation of interview a sample of
exposure. study subjects.
Family information Most relevant in case-control Increase the risk Validate disease and
bias studies. Cases may be more estimate. exposure status; use
aware of family history of controls with another
disease or risk factors disease state.
because the occurrence of
disease has stimulated discus-
sion of past history.
and information bias, their impact, and ways to illness. Each individual who was not employed for
manage these biases in retrospective designs. the last 5 years might be matched on the basis of
sex, age, and education level if these were not vari-
Confounding ables of interest and are likely to be confounding
variables.
Confounding is the mixing of the effects of extra- Matching by frequency occurs by first knowing
neous variables with the variables under study. An or estimating the expected number of cases within
extraneous variable may fully or partially account each level of the confounding variable or variables,
for the apparent association between variables such as white females aged 20 to 24, white females
under study. Alternatively, it may mask or hide an aged 25 to 29, white females 30 to 34, and so forth.
underlying true association. A variable must meet The appropriate number of controls for each
two criteria for being considered a confounding stratum is then selected from the pool of poten-
variable: tial controls until the number needed in each stra-
• It must be a risk factor for the outcome under tum is achieved (Kelsey, Whittemore, Evans, &
study, or associated with the outcome, but not Thompson, 1996).
necessary causal, and Stratification involves partitioning the data
• It must be associated with the exposure under according to the levels of the confounder. For
study, but not a consequence of the exposure. example, consider a study of physically disabled
persons in which the study focus is the relationship
Confounding can be controlled in the design between extent of physical impairment and
through restriction or matching. It can be con- employment. If a comorbid condition (e.g., affec-
trolled in the study analysis through stratification, tive disorder) is a confounder of employment out-
or multivariate analysis, or both. comes, then one may stratify the information
Restriction allows one to narrow the ranges of according to whether or not the subjects had a
values of potential confounders. For example, diagnosis of depression, bipolar disorder, or no
when sex, age, and race are expected to be con- affective disorder. Controlling for more than two
founding factors, an investigator might restrict a or three confounding variables is usually not rec-
study sample to white men, ages 35 to 40. While ommended.
such subject restriction would control for age, race, Finally, multivariate analysis is the construction
and sex, the results would be restricted to a narrow of mathematical models to simultaneously
sample. Matching by pairs occurs where one or describe the effect of exposure and other factors
more controls are chosen for each case based on that may be confounding. Chapter 17 covers mul-
specific characteristics or range of variables. For tivariate analyses and how they are used. The fol-
example, consider a retrospective study that aims to lowing is an example of using multivariate
determine factors that account for difficulty analysis to control for a confounding factor. Using
remaining employed among persons with mental an existing workman’s compensation database, a
researcher wants to examine the incidence of • Estimates of selected characteristics of the entire
carpal tunnel syndrome in computer operators. population surveyed (e.g., how many people in
The researcher identifies possible confounding the United States have a disability), or
factors (e.g., certain kinds of leisure time activi- • A specific subset of the population (e.g., the pro-
ties) that may contribute to the carpal tunnel con- portion of working-age men with disabilities
dition and decides to control for leisure time who have college degrees).
activities in the analysis.
Aggregate data are frequently used to describe
trends over specific time periods (e.g., how
Existing Databases: the unemployment rate for people with AIDS
has changed from year to year since 1995). One
Advantages and limitation of these prepared aggregate data is that
Disadvantages to
they might not use all the variables of interest to
the user. Examples of aggregate data sources
Occupational Therapy include:
Researchers • Census tract statistics,

• Centers for Disease Control and Prevention
(CDC) records of reportable diseases by geo-
An existing health database is a set of information
graphic areas, and
collected at some point in the past for the purpose
• Hospital data reported by diagnosis-related
of shedding light on health conditions (Miller,
group (DRG) or conditions.
1982). Secondary analysis of existing data is
receiving increased attention in health care. Individual level data are those for which separate
Shrinking grant dollars, pressure to produce replic- information is available for each person or member
able findings, and a growing interest in outcomes of a list of individuals. The most common sources
investigation have all contributed to an increasing of individual level data include:
use of existing data for multiple research purposes.
Secondary data analysis may offer the practitioner • Administrative (e.g., claims data),
and researcher ready access to large datasets with • Operational (e.g., rehabilitation or discharge
multiple variables. reports),
• Analytic (e.g., cost reports),
• Patient medical records, and
Types of Secondary Data • Primary data collected in previous studies.
Two types of secondary data exist: Existing databases can be obtained from a
• Aggregate and variety of population-based (e.g., National Center
• Individual. for Health Statistics), state (e.g., State Health
Department), regional (e.g., County Health
Aggregate data are summary-level data, the Department), and local resources (e.g., a local
result of applying statisti- rehabilitation facility)
cal analyses to micro- (Freburger & Konrad,
level datasets. Analysis Shrinking grant dollars, 2002). Routinely collected
may entail summing the pressure to produce replic- data, for example, Na-
number of each type of tional Center for Health
response for a given variable findings, and a grow- Statistics data, are often
able, then applying addi- ing interest in outcomes used to provide descrip-
tional calculations such as tive statistics underlying
weighting and estimation investigation have all con- the health condition and
of sampling error. These tributed to an increasing an indication of the fre-
procedures are meant to quency of the occurrence
provide reliable infer- use of existing data for mul- of a health condition. Such
ences about an entire pop- tiple research purposes. data also serve to provide
ulation based on data leads concerning the etiol-
collected from the sam- ogy (causes) of outcomes
ple, or set of samples, surveyed (Disability or conditions, and to generate hypotheses (Mainous
Statistics Center, 2005). Aggregate data provide: & Hueston, 1997).
The following example illustrates the use of an Prevention (CDC); and data on a variety of health
existing database. An occupational therapy conditions reported in the CDC’s Morbidity and
researcher wishes to study the relationship between Mortality Weekly Report (MMWR).
travel patterns (e.g., night vs. day driving, distances
Researchers may also obtain data by abstracting
driven, frequency of trips taken, and destinations of
and tabulating information from records that have
trips) of older adult drivers and crashes. Using data
been kept primarily for other purposes, such as
from the National Travel Household Survey, the
hospital records, occupational therapy records,
researcher examines the association between expo-
industrial records, insurance records, university
sure variables (age, race, sex, and geographic
student health records, and Armed Forces and
region) and explanatory variables (type of car, med-
Veterans Administration records. Prior to spending
ical status, use of medications, comorbidities,
time on abstracting and tabulating information
socioeconomic status) on the outcome variable,
from records, researchers should enquire if these
crashes. The researcher identifies descriptive infor-
records are available in computerized form (Pabst,
mation and associations that will help explain the
2001). Table 9.3 presents a summary of data types
relationship between risk factors and crashes.
useful for determining some aspects of health
Some data are readily available, in that they are
conditions.
routinely abstracted, tabulated, and published, or
Use of existing databases may produce a
can easily be obtained in nonpublished forms upon
cost–benefit advantage (Castle, 2003). Although
request. Included in this category are:
the researcher will have to invest time in defining,
• Surveys from the National Centers for Health collaborating with the dataset owners, or abstract-
Statistics, and ing and tabulating the data, dollars in collecting the
• Data from the Centers for Disease Control and data will be saved. Researchers, who oftentimes
Table 9.3 Summary of Data Sets Useful for Determining Aspects of Health Status
Category of Data Importance Examples of Variables
Community health data Provide a profile of the Mortality rates, disability rates; incidence
community health necessary and prevalence; morbidity rates;
for development of goals and unemployment rates.
objectives.
Health system data Has limited impact on health Utilization data (admissions, discharges,
status. Data for utilizing client demographics, accessibility);
process evaluation and cost facility data (institutional and human
benefit analysis. resources); cost data (primary care,
indirect, clinic maintenance, personnel,
and insurance costs).
Lifestyle data National data, e.g., the Behavior Lifestyle components defined as behavioral
Risk Factor Surveillance habits, or risk-taking behavior, e.g.,
System (BRFSS) provides smoking, obesity, exercise, substance
useful data for identifying risk use and marital patterns.
factors and risk groups.
Human biology data Provide identification of target All demographic characteristics (age, sex,
groups or subgroups at risk, racial/ethnic groups, occupation); genetic
or for promotion, prevention, risk information.
or intervention activities.
Environmental data Provide identification of risk Air, water, soil conditions; climate, general
factors, confounders or the environmental quality; housing data;
identification of subgroups. occupational data.
Socioeconomic data Provide data for identification of Income, distribution of wealth, employment
“at risk” groups and to status, poverty levels; living environ-
examine subgroups of the ment; level of education; social order;
community. physical and mental health; social
status; recreation and leisure.
work with limited funding, may benefit from using Purposes, Nature, and Sources
existing datasets. of Existing Data
For example, an occupational therapy resear-
cher is ultimately interested in developing an Existing data may be used to examine estimates of
occupation-based health promotion program to such variables as:
curtail or prevent depression in senior adults. To • Hospitalizations,
begin, the investigator wants to examine, in persons • Frequency of visits to rehabilitation settings,
with chronic illnesses, the association between • Impairment and disability,
activity restriction (exposures) and depression (out- • Measures of cost,
come). Confounding variables are age, sex, race, • Risk factors,
and geographic area. The researcher decides to use • Socioeconomic indicators, and
the National Health Interview Survey, a U.S. sur- • Occupational indicators.
vey with respondent data from approximately
130,000 people from 50,000 households. In this Such data can be found in national surveys,
survey, individuals are asked to report conditions surveillance programs, and vital statistics from a
that resulted in activity restrictions occurring close variety of Web-based resources. In addition, such
to the time of the interview and they also report information may be obtained from organizations
depressive symptoms. This database, while it has (e.g., hospitals, insurance companies and state
certain limitations in the quality of the data, can health departments). Table 9.4 presents data
provide important findings about the relationship of sources, URL addresses, and a description of a
activity restriction and depression across a large range of data sources.
and diverse sample. The researcher would be able Modern databases may incorporate linked data-
to formulate a hypothesis from the findings of this bases, which provide population-based, statewide,
exploratory data analysis to further examine inac- computerized, and probabilistic data. For example,
tivity and depression in the senior adult population. The National Highway Traffic Safety Administra-
Table 9.4 Sources of Existing Data: Access to National Databases

Data Source and URL Address Description
Centers for Disease Control and Prevention The Centers for Disease Control and Prevention (CDC) is
http://www.cdc.gov the lead federal agency for protecting the health and
safety of people, providing credible information to
enhance health decisions, and promoting health. Many
datasets for developing and applying disease prevention
and control, environmental health, and health promotion
and education activities are available from the CDC.
Some are discussed below.
National Center for Health Statistics (NCHS) Some NCHS data systems and surveys are ongoing annual
http://www.cdc.gov/nchs/express.htm systems while others are conducted periodically. NCHS
has two major types of data systems:
• Systems based on populations, containing data col-
lected through personal interviews or examinations
and
• Systems based on records, containing data collected
from vital and medical records.
Healthy Women: State trends in health and This site contains tables that describe the health of people
mortality http://www.cdc.gov/nchs/ in each state by sex, race, and age. Currently, mortality
healthywomen.htm tables and health behavior and risk factor tables can be
accessed by downloading a free data dissemination
software called Beyond 20/20®.
WISQARS: Web-based Injury Statistics Query WISQARSTM (Web-based Injury Statistics Query and
and Reporting System http://www.cdc.gov/ Reporting System) is an interactive database system
ncipc/wisqars/ that provides customized reports of injury-related data.
These include fatal and non-fatal injury reports, leading
causes of nonfatal injuries and years of potential life lost
(continued)
Table 9.4 Sources of Existing Data: Access to National Databases (continued)

National Center for Chronic Disease CDC’s National Center for Chronic Disease Prevention and
Prevention and Health Promotion Health Promotion’s mission is to prevent and control
http://www.cdc.gov/nccdphp chronic diseases. The center conducts studies to better
understand the causes of these diseases, supports
programs to promote healthy behaviors, and monitors the
health of the nation through surveys.
CDC: Behavioral Risk Factor Surveillance The BRFSS, the world’s largest telephone survey, tracks
System (BRFSS) http://www.cdc. health risks in the United States. Information from the
gov/brfss/ survey is used to improve the health of the American
people. The following subsets of data are available:
Prevalence Data: Information on health risks in your state
and in the nation; Trends Data: Data of selected risk factors
in a state or in the nation; SMART Selected Metropolitan/
Metropolitan Area Risk Trends: Information on health risks
for selected local areas; BRFSS maps illustrating health
risks at national, state, and local levels. Historical
Questions: The historical record of BRFSS Survey
Questions by category.
CDC WONDER http://wonder.cdc.gov/ WONDER provides a single point of access to a wide variety
of reports and numeric public health data. These include
data on health practice and prevention, injury prevention,
and environmental and occupational health.
National Health Information Center The National Health Information Center (NHIC) is a health
(NHIC) http://www.health.gov/nhic/ information referral service. NHIC puts health professionals
and consumers who have health questions in touch with
those organizations that are best able to provide answers.
NHIC was established in 1979 by the Office of Disease
Prevention and Health Promotion (ODPHP), Office of
Public Health and Science, and the Office of the Secretary,
U.S. Department of Health and Human Services.
U.S. Census Bureau The Census Bureau has extensive data at the U.S. level and
http://www.census.gov/ substantial data for states and counties, somewhat less for
cities. At present they release data for the very smallest
areas (census tracts, block groups, and blocks) only once
a decade after tabulating the results of the Population and
Housing Census. The home page has key links (People,
Business, Geography) to help you. Subjects A to Z gives
you a large list of topics through which to search. You can
find the population of a state, county, or city in FedStats.
National Health Examination Survey Reports on data collected through direct examination, testing
http://webapp.icpsr.umich.edu/ and measurement of a national sample of the civilian
cocoon/NACDA-SERIES/00197.xml noninstitutionalized U.S. population
National Survey of Family Growth Reports based on data collected in periodic surveys of a
http://www.nichd.nih.gov/about/cpr/ nationwide probability sample of women 15–44 years of
dbs/res_national5.htm age.
National Hospital Discharge Survey Data are obtained from sampling records of short-term
http://www. cdc.gov/nchs/about/major/ general and specialty hospitals in the United States Within
hdasd/nhdsdes.htm each sampled hospital, discharges are randomly sampled
from daily listing sheets and information on diagnoses,
surgical procedures, and length of stay are abstracted.
The National Ambulatory Medical Survey This survey includes nonfederally employed physicians
http://www.cdc.gov/nchs/about/major/ sampled from the American Medical Association and the
ahcd/namcsdes.htm American Osteopathic Association, in office-based patient
care practices. During a random week in a year, the
physician-patient visit is recorded in terms of the diagnosis,
treatment, disposition and length of visit.
(continued)
119
Table 9.4 Sources of Existing Data: Access to National Databases (continued)

National Health Care Survey Data collection from healthcare settings such as hospitals, outpatient
http://www.cdc.gov/nchs/ clinics, emergency rooms, ambulatory care facilities, outpatient and
nhcs.htm free standing surgical units, health agencies, hospitals, and
community-based long-term care facilities. A Patient Follow-up
Component is included, providing information on outcomes of patient
care and subsequent use of care through periodic contacts with
patients or patient’s families.
The National Health and Nutrition Provides data from physical examinations, clinical and laboratory tests,
Examination Survey I and questionnaires on a sample of non-institutionalized civilian
http://www.cdc.gov/nchs/ population of the U.S. NHANES I was a prevalence study
nhanes.htm undertaken during 1971–1975, and included about 20,000
individuals in the age range 1–74 years. It was designed to measure
overall health status, with emphasis on nutritional status, dental
health, skin problems, and eye conditions. Detailed information from
a subset of individuals is available on chronic lung disease; disabling
arthritis, cardiovascular disease; hearing level; health-care needs;
and general well-being.
The National Health and Nutrition Provides data from 20,000 individuals age 6 months to 74 years, carried
Examination Survey II http:// out during 1976–1980. This prevalence study was designed to
www.cdc.gov/nchs/nhanes.htm permit some assessment of changes in the population’s nutritional
status, and certain other variables over time. Health and nutritional
status of three Hispanic groups were included in the Hispanic Health
and Nutritional Survey (HHANES) carried out in 1982–1984.
The National Health and Nutrition Carried out in 1988–1994, this survey includes data on persons 2
Examination Survey III http:// months of age and older with no upper age limit. All participants
www.cdc.gov/nchs/nhanes.htm undergo physical examination, body measurements, and a dietary
interview.
The Longitudinal Study of Aging Group of surveys based on the Supplement on Aging to the 1984
http://www.cdc.gov/nchs/ National Health Interview Survey. Data were obtained on the family
products/elec_prods/subject/ structure, frequency of contacts with children, housing, community
lsoa.htm and social support, occupation and retirement, ability to perform
work-related functions, conditions and impairment, functional
limitations, and providers of help for those activities. Information was
obtained from 160,000 people aged 55 years and older. Sub
samples of participants have been interviewed in subsequent years,
mainly in order to measure changes in various social and health-
related variables over time.
National Highway Traffic Safety Statewide, population-based, computerized data related to motor
Administration: Crash Outcome vehicle crashes for two calendar years These include crash data
Data Evaluation System collected by police on the scene, EMS collected by ENTs who
(CODES) http://www.nhtsa.dot. provide treatment en route and Emergency department/hospital data
gov/people/perform/trafrecords/ collected by hospital and outpatient medical staff. Probabilistic
pages/codes/codes.htm linkage of the crash, hospital and either EMS or emergency
department data are possible so that persons involved and injured in
a motor vehicle crash can be tracked from the scene through the
healthcare system. Linkage also makes it possible to evaluate the
medical and financial outcome for specific event, vehicle and person
characteristics.
National Highway Traffic Safety Fatality information derived from FARS includes motor vehicle traffic
Administration: Fatality crashes that result in the death of an occupant of a vehicle or a non-
Analysis Reporting System motorist within 30 days of the crash. FARS contains data on all fatal
(FARS) http://www-nrd.nhtsa. traffic crashes within the 50 states, the District of Columbia, and
dot.gov/departments/nrd-30/ Puerto Rico. The data system was conceived, designed, and
ncsa/fars.html developed by the National Center for Statistical Analysis (NCSA) to
assist the traffic safety community in identifying traffic safety
problems, developing and implementing vehicle and driver
countermeasures, and evaluating motor vehicle safety standards
and highway safety initiatives.
120
tion’s (NHTSA, 2004) CODES (Crash Outcome determine the type, duration, and frequency of
Data Evaluation System) database provides crash services needed and delivered over a period of
data (collected by police at the scene), Emergency time, and to track the rehabilitation outcomes,
Management System (EMS) data (collected by including discharge destinations and follow-up
emergency medical technicians [EMTs] who pro- status. Such information may be used to develop
vide treatment at the scene), and en route- staffing models, marketing plans, and budgets for
emergency department (ED) and hospital data the following fiscal year.
(collected by physicians, nurses, and others who Other computerized information may reveal
provide treatment in the ED, hospital, or outpatient information from subpopulations of hospitalized
setting). Probabilistic linkage exists among the patients and produce comparisons of characteris-
crash, hospital, and EMS or ED data so that per- tics of subpopulations. Statistical applications,
sons involved and injured in a motor vehicle crash assisting the user to analyze the data, are often inte-
can be tracked from the scene of the accident grated into such a system. For example, based on a
through the healthcare system. Linkage makes it description of the subpopulation’s distribution,
possible to evaluate a variety of variables such as stratified by age, days of therapy required, and peo-
the following: ple power required, an occupational therapy resear-
cher could determine the most important service
• Demographic variables associated with increased
needs and intervention areas for this subpopulation.
risk for injury,
Studies using industrial records, can examine
• Vehicle registration, driver license, and citation
whether the environment is contributing to an
data,
occupational health condition. For example, an
• Roadway, vehicle, or human behavior character-
occupational therapy researcher could use industry
istics,
data to determine the incidence and prevalence of
• Medical insurance, medical conditions, mental
work-related injuries, and the most common risk
health, and substance abuse data,
factors associated with them. Such information
• Injury patterns by type of roadway and geo-
may be foundational for developing an injury pre-
graphic location, and
vention and education program.
• Safety needs at the community level.
People enrolled in health insurance groups have
The availability of large datasets, the potential a financial incentive to obtain all of their care
of accessing existing data through collaboration through the plan. Thus data from such groups can
with organizations, and capabilities that modern provide opportunity for researchers to link records
databases afford provide substantial opportunities of earlier health conditions, treatment, and medica-
for the occupational therapist to engage in research. tions, to records of later health conditions. Often
computerized records are available over time, mak-
Examples of Using Existing ing them potentially very useful for longitudinal
Databases to Shed Light on Practice, analysis. For example, an occupational therapy
researcher could determine, in the senior adult
Service Needs, and Outcomes
population, the psychogeriatric medications most
Existing data may be used to shed light on prac- associated with falls leading to injuries, and the
tice, service needs, and outcomes. This section dis- subsequent cost of rehabilitation services. After
cusses some existing databases and provides operationalizing each variable of importance, the
examples of how these databases could be used by researcher could request these data from the health
occupational therapy researchers and practitioners. insurance group. The insurance company could
Computerized data systems used in hospitals, provide de-identified data (data stripped of any per-
rehabilitation clinics, outpatient, and home health sonal identifiers) to the researcher who may now
settings make it readily possible to obtain clinical perform the analysis and obtain the results.
and health information from a patient. These data Two major sources of mental health data exist
include administrative information, laboratory and on the prevalence of psychiatric disorders in the
diagnostic tests, medications and blood products general U.S. population. The National Institute of
received, admitting and discharge diagnoses, pro- Mental Health Epidemiologic Catchment Area
cedures performed, and treatment received (ECA) Program provides data on the incidence,
(Iezzoni, 2002). Within a few minutes the user can prevalence, and other characteristics of several psy-
identify patients with various combinations of chiatric disorders in about 18,000 community resi-
attributes, and display or download the associated dents. The Division of Biometry, National Institute
information. For example, a researcher could use of Mental Health, collects hospital admission sta-
computerized data from a rehabilitation clinic to tistics throughout the country and issues reports
(Kelsey et al., 1996). Using this type of data, an develop a clinical assessment tool for activities of
occupational therapy researcher could conduct a daily function. He or she downloads the NHIS
study of the incidence, prevalence, and type of dataset, a text file, from the NCHS Web site and
psychiatric disorders in this community, as back- uses Statistical Package for the Social Sciences
ground to developing a community-based program. (SPSS) software input statements to convert the
The therapist could also learn about other impor- downloaded text file into a SPSS file. He then
tant characteristics of the population, including selects the variables of interest based on ability to
risk factors, needs, and likely insurance availabil- perform various daily living tasks. The missing
ity to reimbursement for occupational therapy serv- data, and subsequent reduction in sample size, is
ices. the biggest problem for this analysis. However, by
The National Institute of Occupational Safety using existing data, he or she could retrieve and
and Health (NIOSH, 2005) publishes documents start developing an assessment tool ready for psy-
on frequency of diseases of presumed occupational chometric evaluation.
etiology in a variety of industries and on a vari- Accessing person level data from known
ety of etiologic agents found in the occupational sources (e.g., obtaining existing data from a
environment. An occupational therapy researcher research organization or another researcher) ordi-
could use data from NIOSH to determine areas of narily involves a number of steps including:
potential intervention for prevention of occupa-
• Negotiating administrative approval for access,
tional injuries. Identified risk factors could be used
• Undergoing an Institutional Review Board (IRB)
for the development of a workplace health educa-
protocol (i.e., human subjects ethical review),
tion and promotional program.
and
• Assuring compliance with any privacy regula-
tions (e.g., in the United States complying with
Utilizing an Existing Database Health Information Portability and Privacy Act
[HIPAA]).
Utilizing an existing database requires an investi-
Another point of access may be achieved by
gator to know how to access the database, what its
contacting private organizations, such as a rehabil-
protocols are, how to sort through the database, and
itation hospital or insurance company. In this case
what challenges to expect. This section overviews
the researcher will also have to complete the steps
these issues.
noted earlier. Moreover, the investigator would
need to learn and follow the policies of the organ-
Accessing and Retrieving Data ization, often having to pay a fee for obtaining the
In many cases a researcher may access data from abstracted strip data.
national organizations via the Internet. The Accessing and retrieving the data can present a
researcher must follow the published guidelines for number of challenges. Becoming familiar with re-
data access and data manipulation; these may vary trieval policies and procedures can sometimes
considerably with each dataset. Some datasets can pose technical problems. To address these chal-
be downloaded or installed to the researcher’s own lenges, a researcher may:
personal computer and used with an appropriate • Seek formal training provided by the dataset
software package. To load the data and translate keeper,
them into a form that can be read by the software, • Rely on the resources of the primary data analyst,
executable files are often included with the data. • Seek assistance from a computer information
Other public domain files can be accessed through specialist,
the Internet only by using specific query language • Obtain abstracted data (data ready for use based
that is available on the Web site where the data are on specific variables that the researcher require),
housed. or
For example, the National Center of Health • Utilize a statistical analyst to help with data re-
Statistics (NCHS) provides public information on trieval and analyses.
health status by compiling statistical information
from medical records, by conducting medical Compliance with Privacy
examinations, examining birth and death reg-
and Ethical Protocols
istries, and conducting various health surveys.
NCHS releases this information in a variety of The use of patient-identifiable data in research is
datasets. An occupational therapy researcher uses subject to increasingly complex regulation. More-
the National Health Interview Survey (NHIS) to over, it may vary by country and by organization.
While this section briefly overviews some of the protocol for this process must be developed.
steps required to comply with privacy and ethical Personnel who will obtain data must be trained and
protocols, investigators will need to become famil- periodically checked for accuracy. For example, if
iar with what is required in each setting. a study is using medical records, personnel may
Ordinarily, when analyzing existing data, an need to review individual records to locate and
investigator may apply for exempt IRB approval code data, and enter the data into a computer pro-
since no new data are collected. When using data gram for analysis. A procedure for how data will be
from another researcher, approval must be obtained located, coded, and entered will need to be devel-
by submitting a letter to the review committee out- oped and personnel will need to be trained and
lining the proposed secondary analysis study and checked for accuracy.
the keeper of the original data. The investigator
doing the secondary analysis is bound by the same
confidentiality and privacy restrictions as the pri- The Challenges of Using
mary investigator. Ordinarily, the primary princi-
pal investigator and secondary researcher sign a Existing Databases
document of agreement. This document outlines There are a number of limitations associated with
the rights, responsibilities, and obligations of the using existing databases (Brown & Semradek,
primary principal investigator and secondary 1992; Clarke & Cossette, 2000; Pollack, 1999).
researcher. This document usually describes the The following are some key limitations or prob-
data that were accessed (e.g., interviews, demo- lems:
graphic data), method of access (i.e., via computer
software), and provisions for reference citations • It is often difficult to know all the details about
in publications and presentations. The primary the secondary data. Information on the source
principal investigator and secondary researcher data (i.e., the original dataset, its purpose, func-
also sign a Certificate of Confidentiality as assur- tion and use), instruments, and procedures used
ance that the primary principal investigator for data collection are often limited, missing, or
will protect the confidentiality and privacy of the not easily found.
subjects. • Secondary analysis does not afford the investiga-
When patient-identifiable information is needed tor complete control over operationalizing vari-
to link data from two sources for analysis, the ables. Consequently, there may not be congruence
process is increasingly more complex (Malfroy, between the investigator’s conceptual definition
Llewelyn, Johnson, & Williamson, 2004). The of a variable and what is available in the dataset.
process of establishing a study may involve obtain- There may be a lack of congruence between the
ing ethical research and development and data pro- conceptual definition in the secondary study and
tection approval, including application to the the operational definition used in an original
Patient Information Advisory Group, set up under study. It is also possible that the unit of analysis
Section 60 of the Health and Social Care Act, 2001. may not be the same (e.g., data collected from
people cannot be translated to family-level data).
• The original study may lack a theoretical frame-
Obtaining Data
work, challenging the secondary investigator’s
Once the important data approval issues have been use of a conceptual framework for the secondary
decided, the researcher can consider how the data analysis.
will be obtained and moved to the selected data • There may be threats to reliability including
storage medium. Steps need to be taken to ensure problems with the instruments (e.g., unknown
that data are handled (i.e., retrieved and stored) a psychometric properties), poor training and
minimum number of times. Ideally, data should be supervision of the data collectors, and inconsis-
handled only once (Pollack, 1999). tent data collection procedures.
Each additional time data are accessed before • There may be threats to validity such as the
they are put into the storage medium, there is a instruments not fitting the concepts, large amount
chance that they can be damaged or changed. Soft- of data being missing, confusion about coding
ware packages, such as BrainMaker (2004), may missing data, or inaccuracy in data entry.
enable the researcher to combine data that had • There may be selection and information bias in
been previously stored in smaller files and put into the data that are difficult to detect or manage.
a large data set for analysis. • The secondary researcher has no choice in the
When data are being obtained from a medium selection of the instruments. Instruments used
that requires data conversion into study variables, a in the study may not be the best matches for
measuring the variables under study, yielding Consider the Database, the Study
measurement concerns (Strickland, 1997). Methodology, and the Study Conclusions
• Data may be out of date owing to changes in case
definitions, classifications of diagnoses, and Motheral and colleagues (2003) developed a retro-
changes in clinical practice. For example, the spective research checklist of 27 questions to
CDC has revised the guide consideration of
case definition of AIDS the database, the study
three times. Similarly, Retrospective research and methodology, and the
what was called a cere- study conclusions. This
bral hemorrhage at the
analysis offer many uses checklist covers a wide
beginning of the decade and much promise for occu- range of issues, includ-
was tending to be called ing relevance, reliability
pational therapy practice, and validity, data link-
cerebral thrombosis at
the end of the decade service, and research. ages, eligibility determi-
(Kelsey et al., 1996). nation, research design,
• The investigator lacks treatment effects, sample
control in conceiving, generating, and recording selection, censoring, variable definitions, resource
the dataset. valuation, statistical analysis, generalizability, and
• Data may be incomplete, erroneous, or mislead- data interpretation. Use of this checklist can be
ing. For example, U.S. hospitals are usually reim- very helpful in evaluating and using a database for
bursed according to the Diagnostic Related Group retrospective analysis.
System (DRG); thus, the diagnosis may be influ-
enced by reimbursement considerations. Select the Most Appropriate Database
These limitations illustrate the potential pitfalls to Answer the Research Questions
associated with using existing databases. The next
The properties of the database can either facilitate
section describes necessary strategies to limit or
or hinder addressing the intended research ques-
eliminate these pitfalls.
tions. For example, in considering a database to
examine the determinants of unsafe driving among
Dealing with the Challenges senior adults, the most desirable database would be
of Existing Databases routinely collected, up-to-date, and linked; it would
This section discusses nine general strategies for provide population-based, statewide computerized
managing some pitfalls associated with use of ex- and probabilistic data pertaining to motor vehicle
isting databases (Brown & Semradek, 1992; Clarke crashes. The CODES database is an example of
& Cosette, 2000; Pollack, 1999). such data. Figure 9.2 represents a flow diagram of
CODES linked data. From this diagram one can
Consider the Research Question easily see how environmental information (e.g.,
description of the injury event) is linked to medi-
The variables contained within each dataset are a cal (e.g., inpatient rehab) and claims (e.g., HMO)
function of the original purpose for the compilation information, affording the researcher with opportu-
of the data. In using data for a secondary analysis, nities to follow the client from the scene of the
the issues of variable conceptualization, operation, accident through discharge from the medical or
and availability become paramount. Before work- rehabilitation setting.
ing with a dataset, the researcher needs to generate:
• Carefully conceptualized research problem and Obtain a Thorough
question, and Description of the Data
• Identified variables required for the investigation.
Many archives provide a codebook, or data diction-
Then the investigator must ask how well the ary, for each dataset. The codebook contains the
problem, question, and variables match the way list of variables as well as information on the
existing data were defined and collected. Delving sampling design, instruments, and procedures.
into an existing dataset without first constructing Some archives of single datasets offered by indi-
the conceptual foundation and methodologi- vidual researchers may not provide all this infor-
cal approach for the inquiry increases the poten- mation. In this case, the secondary investigator
tial of introducing threats to both reliability and should contact the primary researcher to request the
validity. codebook, copies of instruments and instrument
Environmental Information
EMS
Hospital
Emergency Inpatient Long-Term
Injury Event
Department Rehab Care
Figure 9.2 Flow diagram explain- Nonhospital Outpatient

ing the CODES linked data.
Physician Office Death Certificate
(Reprinted with permission from Clinic
National Highway Transportation HMO
Safety Administration [NHTSA]). Claims
descriptions, written protocols, and instructions save time and money. They can provide database
to data collectors. In some cases, the secondary construction; database management, data monitor-
researcher may wish to speak with the data collec- ing, data cleaning, and reduction; data dictionary
tor(s) to better understand the context of the study development; and data table construction.
and to elicit any problems that arose during data
collection. Manage Missing Data
If missing data are nonrandom, steps should be
Choose a Timely Data Source
taken to determine whether the data could be
Timeliness can be an issue when there is a long recovered, as might happen if a certain set of files
time interval between the original study and the did not link during a data merge (Kneipp &
secondary study. It may be necessary for the sec- McIntosch, 2001). The most commonly used meth-
ondary researcher to choose another dataset that ods are to:
has less time-sensitive data or that was collected
more recently. • Delete those subjects for whom there are missing
data (listwise deletion),
• Delete those subjects only when the variable with
Talk to Program Staff missing data is used in an analysis (pairwise
Program staff may help in abstracting the data. deletion), and
Although most national organizations, for example, • Impute some value to replace the missing data, or
the Census Bureau, provide data for free on the by using regression (McCleary, 2002).
Internet, investigators sometimes need to access the
data in a format not available over the Internet. In
those cases, they may be able to get the data with
help of the program staff (and, perhaps, at some
Conclusion
additional cost).
Retrospective research and analysis offer many
uses and much promise for occupational therapy
Training practice, service, and research. By carefully evalu-
Some organizations provide extensive on-site ating datasets before choosing one, researchers can
training in use of the dataset. While such training capitalize on the benefits, and at the same time,
can be very valuable, the researcher may have to minimize the potential pitfalls of secondary analy-
incur travel and training costs to become educated ses. Any inquiry using existing data must be care-
in the use of the dataset. fully designed to optimize integrity, reliability, and
validity of the study. The research question, data-
base, study methodology, data description, timeli-
Use a Computer Information Specialist
ness, use of support, managing data entry errors,
With continued advancements in technology, and missing data are all important concepts in
researchers may want to work with a computer ensuring scientific rigor. Once practitioners and
scientist or analyst to optimize effectiveness and researchers fully recognize the challenges inherent
efficiency in data retrieval, storage, management, in a secondary analysis, rich opportunities exist for
and analysis. These professionals have skills and studying health-related questions of relevance to
knowledge that may ease the task of analysis, and occupational therapy.
REFERENCES NIOSH: Databases and information resources. Retrieved

April 2005 from http://www.cdc.gov/niosh/about.html.
BrainMaker. California Scientific software. Accessed Pabst, M. K. (2001). Methodological considerations: Using
September 5, 2004 from http://www.calsci.com/ large data sets. Outcomes Management in Nursing
tsLargeDataSets.html. Practice, 5, 6–10.
Brown, J. S., & Semradek, J. (1992). Secondary data on Pollack, C. D. (1999). Methodological considerations with
health-related subjects: Major sources, uses, and limita- secondary analyses. Outcomes Management Nursing
tions. Public Health Nursing, 9, 162–171. Practice, 3, 47–52.
Castle, J. E. (2003). Maximizing research opportunities: Sackett, D. L. (1979). Bias in analytical research. Journal
Secondary data analysis. Journal of Neuroscience of Chronic Disability, 32, 51–63.
Nursing, 35, 287–290. Strickland, O. L. (1997). Measurement concerns when
Clarke, S. P., & Cossette, S. (2000). Secondary analysis: using existing databases. Journal of Nursing
Theoretical, methodological, and practical considera- Measurement, 5, 115–117.
tions. Canadian Journal of Nursing Research, 32,
109–129.
Disability Statistics Center. Finding disability data on the RESOURCES
web. Retrieved May 5, 2005 from http://dsc.ucsf.edu/
main.php?name⫽finding_data Best, A .E. (1999). Secondary databases and their use in
Freburger, J. K., & Konrad, T. R. (2002). The use of fed- outcomes research: A review of the area resource file
eral and state databases to conduct health services and the Healthcare Cost and Utilization Project.
research related to physical and occupational therapy. Journal of Medical Systems, 23, 175–181.
Archives of Physical Medicine and Rehabilitation, 83, Byar, P. B. (1980). Why databases should not replace ran-
837–845. domized clinical trials. Biometrics, 36, 337–342.
Iezzoni, L. I. (2002). Using administrative data to study Estabrooks, C. A., & Romyn, D. M. (1995). Data shar-
persons with disabilities. The Milbank Quarterly, 80, ing in nursing research: Advantages and challenges.
347–379. Canadian Journal of Nursing Research, 27, 77–88.
Kelsey, J. L., Whittemore, A. S., Evans, A. S., & Feinstein, A. R. (1984). Current problems and future chal-
Thompson, W. D. (1996). Methods in observational lenges in randomized clinical trials. Circulation, 70,
epidemiology (2nd ed.). New York: Oxford University 767–774.
Press. Greenland, S., & Finkle, W. D. (1995). A critical look at
Kneipp, S. M., & McIntosh, M. (2001). Handling missing methods for handling missing covariates in epidemio-
data in nursing research with multiple imputation. logical regression analyses. American Journal of
Nursing Research, 50, 384–389. Epidemiology, 142, 1255–1264.
Mainous, A. G., III, & Hueston, W. J. (1997). Using other Jacobson, A. F., Hamilton, P., & Galloway, J. (1993).
people’s data: The ins and outs of secondary data Obtaining and evaluating data sets for secondary analy-
analysis. Family Medicine, 29, 568–571. sis in nursing research. Western Journal of Nursing
Malfroy, M., Llewelyn, C. A., Johnson, T., & Williamson, Research, 15, 483–494.
L. M. (2004). Using patient-identifiable data for epi- Kashner, T. M. (1998). Agreement between administrative
demiological research. Transfusion Medicine, 14, files and written medical records: A case of the
275–279. Department of Veterans Affairs. Medical Care, 6,
Mausner J. S., & Kramer, S. (1985). Epidemiology: An 1324–1336.
introductory text. Philadelphia: W. B. Saunders. Kneipp, S. M., & Yarandi, H. N. (2002). Complex sam-
McCleary, L. (2002). Using multiple imputation for analy- pling designs and statistical issues in secondary
sis of incomplete data in clinical research. Nursing analysis. Western Journal of Nursing Research, 24,
Research, 51, 339–343. 552–566.
Miller, J. D. (1982). Secondary analysis and science educa- Kraft, M. R. (2003). Database research. Science of
tion research. Journal of Research in Science Teaching, Nursing, 20, 43–44.
19, 719–725. Landerman, L. R., Land, K. C., & Pieper, C. F. (1997). An
Moriarty, H. J., Deatrick, J. A., Mahon M. M., Feetham, S. empirical evaluation of the predictive mean matching
L., Carroll, R. M., Shepard, M. P., & Orsi, A. J. (1999). method of imputing missing values. Sociological
Issues to consider when choosing and using large Methods & Research, 26, 3–33.
national databases for research of families. Western Last, M. (2001). A dictionary of epidemiology (4th ed.).
Journal of Nursing Research, 21, 143–153. New York: Oxford University Press.
Motheral, B., Brooks, J., Clark, M. A., Crown, W. H., Rotnitzky, A., & Wypij, D. (1994). A note on the bias of
Davey, P., Hutchins, D., Martin, B. C., & Stang, estimators with missing data. Biometrics, 50,
P. (2003). A checklist for retrospective database stud- 1163–1170.
ies—report of the ISPOR Task Force on Retrospective Szabo, V., & Strang, V. R. (1997). Secondary analysis of
Databases. Value Health, 6, 90–97. qualitative data. Advances in Nursing Science, 20,
National Highway Traffic Safety Administration (NHTSA). 66–74.
Crash Outcome Data Evaluation System (CODES). Victora, C. G., Habicht, J., & Bryce, J. (2004). Evidence-
Retrieved September 10, 2004 from http://www-nrd. based public health: Moving beyond randomized trials.
nhtsa.dot.gov/departments/nrd30/ncsa/CODES.html American Journal of Public Health, 94, 400–405.
C H A P T E R 1 0
Longitudinal Research: Design

and Analysis
Yow-Wu B. Wu • Kirsty Forsyth • Gary Kielhofner
Longitudinal research is useful for investigators

who want to gain a detailed understanding of pat- Spector and Takada (1991) describe an analysis
based on 2,500 residents in 80 nursing homes in
terns of stability and change in occupational partic-
Rhode Island. Multivariate models were used to
ipation over time. It is also valuable for developing estimate which aspects of care were associated
a coherent explanation of how and why the occu- with resident outcomes after controlling for resi-
pational change has occurred. dent characteristics. Outcomes, measured over a
Longitudinal research involves collecting data 6-month period, included death, functional
over time using repeated measures and completing decline, and functional improvement. Results
a longitudinal analysis of the data. Longitudinal suggest that higher staff levels and lower staff
research has focused on understanding people’s turnover were related to functional improvement.
engagement in activities of daily living and is, Facilities with high catheter use, low rates of
therefore, relevant to occupational therapists skin care, and low participation in organized
activities were associated with negative out-
(Sonn, Grimby, & Svanborg, 1996; Spector &
comes. Facilities with few private-pay residents
Takada, 1991). were also associated with negative outcomes.
The purpose of this chapter is to provide a
basic understanding of the longitudinal study
methodology (design and analysis), and opportuni-
ties for applying this approach in occupational ther-
are referred to as dependent variables. In some lon-
apy research. The chapter covers: gitudinal designs, investigators make inferences
about causal links between intervention variables
• What constitutes longitudinal research,
and dependent variables related to time.
• Longitudinal research designs, and
Since this type of research includes different
• Selected techniques in analyzing longitudinal
designs and analytical methods, it is difficult to
data.
give a definition that applies to all situations. The
This should be viewed as an introductory dis- aim of longitudinal research is to study the phe-
cussion of longitudinal nomena in terms of time-
research; only the basic related constancy and
concepts of designs and The defining characteristic of change. For example, lon-
selected number of ana- longitudinal research is that gitudinal research would
lytical techniques are be the method of choice
discussed. Readers who data are collected at multiple if one wanted to know
are interested in more time points. what happens to older
detailed discussions persons’ engagement in
should consult materi- everyday activities over
als in the Resources list. time after they enter a nursing home. An appropri-
ate longitudinal design to answer this question
The Nature of could involve measuring the numbers of residents’
everyday activity at admission, and at 1, 3, 6, 9,
Longitudinal Research and 12 months after admission. The analysis of
these data would focus on identifying change or
The defining characteristic of longitudinal research stability of these everyday activity measures, and
is that data are collected at multiple time points. if data were collected on other variables, the study
In longitudinal studies, the variables that are might also aim to identify what causes this change
examined as to their stability or change over time or stability.
Intra-individual and • What is the direction of the change? (e.g.,

Inter-individual Variability increase or decrease in number of life roles over
time?)
Baltes and Nesselroade (1979) point out that lon- • Is the change linear or nonlinear? (e.g., do life
gitudinal research is focused on studying both: roles decrease for a period and then plateau?)
• Intra-individual differences (i.e., differences • Is the change consistent or does it fluctuate from
within each participant over time), and time to time? (e.g., are the number of life roles
• Inter-individual patterns (i.e., differences in the highly variable across time according to the
pattern of change or stability across participants health status of the person at the time of meas-
in the study). urement?)
For example, consider a study in which the The following are examples of questions about
investigator collected information on the number the relationship between predictors and the pattern
of life roles a person reports every 6 months for 5 of change:
years starting from the age of 65 to 70. In such a
study, the researcher would be interested in finding • Is the pattern of life role change predicted by the
out whether there is a change in number of life health status of the person?
roles across the 5 (65 to 70) years. This would be • Do men and women show different patterns of
intra-individual change. The investigator would change in life roles over time?
also want to know whether the patterns of these
changes vary from person to person. This would be To answer this type of question, investigators
inter-individual change. would test hypotheses about the variables that
In the study just mentioned, the investigator account for differences in patterns of change over
would also likely want to know the reasons for any time.
differences in the pattern of change between indi-
viduals. Keeves (1988) identified three major sys-
tems that affect the change and stability in human Longitudinal
development:
Research Designs
• Biological factors,
• Environmental factors, and All longitudinal research designs are characterized
• Planned learning or interventions. by the following two features:
With regard to the example, the pattern of life
• They must have a measure of time (i.e., observed
role change may vary from person to person
change or stability in variables must be associ-
because of biological factors such as whether or
ated with time), and
not the person had a chronic illness. Similarly, the
• Observations must be made at more than two
pattern of life role change may differ because of
time points to identify the impact of time on tra-
different opportunities in participants’ families and
jectories.
neighborhoods (environmental factors). Finally,
differences in number of life roles may be due Longitudinal research designs can vary as to
to whether the older adults participated in a whether the time intervals at which data are col-
lifestyle and wellness program offered by an occu- lected are fixed (i.e., every participant must be
pational therapist (planned learning and interven- observed or interviewed at a fixed time point such
tion factors). as 3, 6, and 9 months) or flexible (i.e., data can be
gathered at flexible time points). The advantage of
Types of Longitudinal Questions flexible time points is, for example, that if a par-
According to Singer and Willett (2003), there are ticipant was supposed to be interviewed at 3
two types of questions that can be asked about months and was quite ill at that time, the data can
change over time: be gathered at either 3.5 or 4 months. Whether data
are collected at fixed or flexible time points has
• Descriptive questions, and
implications for how the data can be analyzed (i.e.,
• Questions that examine relationships between
using fixed models vs. more flexible random or
predictors and patterns of change.
mixed models for analyzing data). Finally, an
Descriptive questions aim to characterize each important consideration in all longitudinal designs
individual’s pattern of change over time. They is that in longitudinal research, missing data are
might ask: inevitable.
Chapter 10 Longitudinal Research: Design and Analysis 129
Five Longitudinal Research Designs Table 10.1 Simultaneous Cross-Sectional

Study Layout
Kessler and Greenberg (1981) identify five differ-
ent types of longitudinal research designs: Different Age Data
• Simultaneous cross-sectional studies, groups Samples* Collection Time
• Trend studies, 60 1 01/04
• Time series studies, 62 2 01/04
• Intervention studies, and
64 3 01/04
• Panel studies.
66 4 01/04
Each of these designs has its special focus in 68 5 01/04
answering different types of questions. In this sec-
70 6 01/04
tion, we briefly introduce these designs and dis-
cuss their strengths and weaknesses. We also 72 7 01/04
address the questions that are associated with these 74 8 01/04
designs. 76 9 01/04
78 10 01/04
Simultaneous Cross-Sectional Studies 80 11 01/04
In a simultaneous cross-sectional study, investiga-

*Each different number represents a different
tors collect data at one time point using samples sample set
of participants representing different age groups.
The purpose of this design is to investigate
whether there is an age group difference on certain
which represents a different time point. The sec-
variables. Notably, time in these studies is a func-
ond factor is how to select representative samples
tion of the different ages of participants in each
from the specified age groups. This requires
sample. Consequently, this design is unique among
identification of the population from which the
longitudinal designs in the way the investigator
sample will be drawn. If the target population
operationalizes the metric of time. Rather than col-
is all persons older than 65 years of age in the
lecting data across time, it collects data from sam-
United Kingdom, then one might generate the
ples that represent a span of time.
sample from the government register of retired
A researcher who plans to use this design needs
people.
to consider:
If the intent of the study was to examine the
• How many participants are required in each impact of other variables, such as sex and race on
group for the analysis, and time spent in self-care activities, then the sampling
• How samples can be selected to represent a strategy might stratify the sample according to
defined population. these variables. See Chapter 31 for a more in-depth
discussion of stratification.
In addition, there are also typical considera- This type of design is based on the assumption
tions about how to best sample in order to answer that the chronological age is related to the depend-
the substantive question. For example, consider an ent variable. The advantages of the simultaneous
occupational therapy researcher who wishes to cross-sectional design are:
know if the amount of time spent in self-care activ-
ities is a function of age in persons 60 to 80 years • It is easy and economical to execute because data
old. There are two major factors to consider in this are collected at one time point, and
type of longitudinal study design. The first is how • Environmental confounding factors are reduced
to determine the age group. For instance, should because they are all measured at the same time.
the investigator select the age groups by consecu-
The following are requirements of this type of
tive years, by every other year, or every 5 years?
design (Keeves, 1988):
This is a substantive issue that must be determined
by theory, previous research, or clinical evidence. • The age samples must be drawn from the same
An investigator may decide to gather informa- common population, and
tion every other year. Table 10.1 illustrates such a • The influencing factors and their effects must
design. It is a typical simultaneous cross-sectional remain constant across the time span during
longitudinal design, in that all data are collected which the different age samples have been
simultaneously from different samples, each of exposed to the factors.
If these requirements are not met, the findings Table 10.2 Trend Studies Layout
will be confounded by other factors unrelated to
age. For example, if the samples selected in 60- Age Group Sample* Data Collection Times
year-old group are predominantly African males, 10–14 1 1994
the 62-year-old group are predominantly white
10–14 2 1995
females and the 64-year-old group are predomi-
nantly Asian males and females, then the first 10–14 3 1996
assumption would not be met. If all the samples 10–14 4 1997
were drawn randomly from the same common 10–14 5 1998
population, one would expect sex and race to be 10–14 6 1999
equalized across the groups. With the kind of sam- 10–14 7 2000
ple described above, the factors that influence the
10–14 8 2001
time spent in self-care activities could very well
differ by race and sex. Therefore, any conclusion 10–14 9 2002
about the effect of aging on self-care activities 10–14 10 2003
would be misleading, since it is confounded by
race and sex. *Each different number represents a different
The primary weakness of this design is that it sample set.
does not provide data on the change pattern of
individual participants, since it does not follow
individuals over time. If individual pattern of have to collect data spanning a 10-year period,
change is a major concern of the study question, selecting a sample of 10- to 14-year-olds that rep-
this is not an appropriate design to use (Singer and resent the target population for each time period.
Willett, 2003). Another weakness is that each time Thus, the participants would be different each time
period is represented by a different sample; these but their mean age would be the similar. Table 10.2
samples may differ on characteristics other than provides a layout of this design.
age. Taking the preceding example, one would Investigators can use this type of design not
expect that not all those in the 60-year-old group only to describe the trend, but also to explain
are likely to live until 80. Thus, the sample of those observed trends. For example, adolescents in pub-
who are 80 may differ from those in the 60-year- lic versus private schools could be sampled and the
old age group who will not survive to 80 because effects of type of school enrollment on leisure and
of significant biological or other differences. schoolwork could be examined. Such a study
Nonetheless, these potential differences are might find, for instance, that in one of these groups
unknown. study time increased while it remained constant in
A further weakness in this type of study is that the other. In addition, this type of study can be
the factors that impact the dependent variable may used to examine the impact of historical variables
not be the same across different samples. Since we on the trend. For example, the introduction of
do not use the same participants at different times, school testing during the 10-year period could be
caution must be taken if we assume the variable examined to see if it had an effect on the depend-
that has influenced the self-care activities will be ent variables.
the same across different times. The strength of this approach is that data for
this type of study can be obtained either retrospec-
tively (which is economical) or prospectively. A
Trend Studies
retrospective application of this design would
In trend study designs, investigators collect data require the investigator to sample persons who
from samples of participants representing the same were 10 to 14 years of age during the targeted
age group at different time points. Therefore, the years. An obvious limitation of this type of design
pool of participants selected to represent each time is that it asks subject to recall their leisure and
point are different, although they are from the school work involvement. Notably, the period
same age group. The purpose of this type of study since the behavior occurred will differ across the
is to examine trends in certain phenomenon as time different year samples (e.g., the 2003 group would
progresses. For example, consider an investigator be asked to recall behavior of a year ago, while the
who wants to know whether the amount of time 1995 group would be asked to recall what they did
spent in leisure and in school work has changed in a decade earlier. A prospective application of this
the decade from 1994 to 2003 for children ages 10 design would require the investigator to conduct
to 14 years old. In this case, the investigator would the study over a 10-year period, sampling 10- to
14-year-olds each year. An obvious limitation of The following are advantages of time series
this design is the amount of time required. studies:
A requirement of this design is that the envi-
• This design can directly identify intra-individual
ronmental factors (with the exception of any that
constancy and change through time.
are measured) are the same across all different
• It is possible to examine whether or not patterns
times. Otherwise, if there are unknown environ-
over time (trends) are homogeneous across indi-
mental factors, they might be confounded with the
viduals.
effect of time.
• It is possible to examine relationship of change
In addition to the limitations noted in the pre-
patterns to demographic and other variables
ceding paragraphs, the major limitation of the
(including experimental/intervention variables).
trend study design is that this type of design does
• It is possible to use advanced statistical tech-
not allow the investigator to explain any trends
niques to make causal inferences (von Eye,
based on biological, environmental, or interven-
1985).
tion variables (Keeves, 1988). One possible excep-
tion is the natural occurrence of an event in a An example of a time series design that exam-
prospective study that can be examined for its ines relationships between change patterns and
effect on the dependent variable. demographic variables is a study of whether pat-
terns of recovery following a stroke differ between
Time Series and Intervention Studies sexes or between persons of different initial health
status.
In time series studies, investigators collect data In time series studies, a researcher can also add
from the same participants at different time points. intervention at a selected time point (Keeves,
The purpose of this type of study is to observe the 1988). This can be used as a one group design, in
pattern of changes within (intra-individual) and which the researcher is interested in comparing the
across (inter-individual) individuals. Time series changes on the dependent variable before and after
designs can be used both for nonexperimental sit- the intervention. The design can also be used with
uations and experimental purposes. The latter are two or more groups. For example, both a one-
intervention studies. A researcher who uses a time group and a two-group time series design could be
series design selects one or more sample(s) of par- used to examine the impact of occupational ther-
ticipants and follows these participants, gathering apy on people who have had a stroke. In a one-
data from time to time. Table 10.3 illustrates a time group time series design, persons with stroke
series study in which data are first collected from could be measured for functional ability every
the subjects when they are age 45 and each year month beginning in the 6th month post-stroke.
thereafter for a total of 10 years. An occupational therapy intervention could
be offered to all the participants in a study follow-
ing the 8th month observation and three more
Table 10.3 Time Series Study Layout observations could be completed. This type of
time series design follows the same logic as single-
Different Age Data subject designs as discussed in Chapter 11. A two-
Group Sample* Collection Times group design might involve an occupational
therapy intervention immediately following the
45 1 1995
stroke for one group that would be compared
46 1 1996 to a control group that did not receive the inter-
47 1 1997 vention or that received a control intervention.
48 1 1998 In this case, data might be collected on both
49 1 1999 groups at 6, 12, 18, and 24 months post-interven-
50 1 2000
tion. In this instance, the logic of the design is the
same as control group designs, which are dis-
51 1 2001
cussed in Chapter 7.
52 1 2002 An important design consideration in time
53 1 2003 series studies is whether the sequence of observa-
54 1 2004 tions is adequate to examine the process of change.
In designing a study, one needs to consider both
*Numbers representing the sample are the same
how long and how frequently it is necessary to
since data are collected from the same subjects gather data in order to capture the relevant trends
each year. in the dependent variable.
There are three shortcomings to using time design, each row corresponds to time series
series studies: design, and each column corresponds to a cross-
sectional design.
1. Repeated measurement can introduce a meas-
Panel studies ordinarily yield more information
urement effect (i.e., the act of collecting data
than trend and cross-sectional studies because the
once may influence the participant’s score on
same groups of participants are measured repeat-
the subsequent occasion of data collection.) For
edly. In a time series study, observations are usu-
example, when the dependent variable is meas-
ally taken on a single entity at a relatively large
ured with a performance test, there may be prac-
number of time points. In a panel study, data are
tice effect (i.e., taking the test improves the
collected from many participants, but ordinarily at
ability being tested).
relatively fewer points in time than in a time series
2. There is a greater tendency for missing data
study (Markus, 1979).
because of participant attrition.
In a panel study, investigators can examine the
3. Time series studies are costly.
simultaneous influence of time, cohort (C), time of
measurement (T), and age (A) (Schaie, 1965);
Panel Studies these are noted on Figure 10.1. Consequently, a
The term “panel” refers to a sample of participants strength of this design is that researchers can
involved in the context of a longitudinal study. explore many different factors that may influence
Keeves (1998) described a panel study design as a the observed patterns in dependent variables. For
series of different samples (panels) that are initi- example, a researcher is able to examine the inter-
ated at different times; data are collected from action effects of A C, A T, and C T. For
each sample at the same time intervals. The longi- example, A C is used to examine if the pattern
tudinal panel study is often referred to as a multi- of change differs among six different dates of birth
ple cohort design, since it investigates multiple cohorts. It will answer the question of whether the
cohorts on multiple occasions (Bijleveld et al., growth pattern of 1990 is the same as the growth
1998). Figure 10.1 illustrates a panel study in patterns of 1992, 1994……. and 2000. The A T
which data are collected from six sets (panels) of is used to examine if there is any difference among
subjects beginning at age 1 and every other year different age groups. For example, we can com-
thereafter until age 9. As illustrated in the figure, pare to see if year 1 versus year 3 makes any dif-
the first cohort of participants begins in 1991 when ference. The C T is used to examine the
children are age 1, and data are collected from this differences among the same ages across different
cohort every 2 years until 1999 when they are age years of birth. The challenge of this design is that
9. The second cohort starts from 1993 when they it is very time consuming, costly, and subject to
are 1 year of age and follows the same pattern of high attrition rates. Unless there is a strong finan-
data collection until 2001. A total of six cohorts cial support, this design is not easy to implement.
are included following this pattern. Close exami- For example, one challenge of this design is to
nation of Figure 10.1 shows that the diagonals extend the length of the study sufficiently to dif-
under chronological age correspond to a trend ferentiate cohort and age (Keeves, 1988).
Date of
Chronological age at time of measurement (A)
Birth (C)
1990 1 3 5 7 9
1992 1 3 5 7 9
1994 1 3 5 7 9
1996 1 3 5 7 9
1998 1 3 5 7 9
2000 1 3 5 7 9
Time of Figure 10.1 Panel design showing

Measurement '91 '93 '95 '97 '99 '01 '03 '05 '07 '09
ages of 5-year cohorts measured
(T)
at 2-year intervals.
Summary different approaches to sampling and to opera-

tionalizing time. Each has its own strengths and
The previous sections discussed major longitudi- weaknesses. Table 10.4 summarizes these designs
nal study designs. Each of these designs combines and their strengths and weaknesses.
Table 10.4. A Comparison of Longitudinal Study Designs

Design Description Strengths Weaknesses
Simultaneous Data are collected at one • Easy and economical to • Does not provide data
cross-sectional time point, using different execute because there is on the change pattern
samples of participants only one time of data of individual subjects.
representing different collection. • Each time period is
age groups. • Environmental confounding represented by a
factors are reduced since different sample;
they are all measured at samples may differ on
the same time. characteristics other
than age.
• Factors that impact the
dependent variable
may not be the same
across different
cohorts.
Trend studies Data are collected on • Can be implemented either • Does not allow
samples of participants retrospectively (which is explanation of trends
representing the same very economical), or based on biological,
age group at different prospectively. environmental, or
time points. intervention variables.
• Prospective: requires
substantial time.
• Retrospective: can
introduce bias due to
differential recall
requirements.
Time series studies Data are collected from the • Can directly identify intra- • Repeated measurement
same participants at individual constancy and can introduce a
different time points. change through time and measurement effect.
individuals serve as their • There is a greater
own controls. tendency for missing
• Can examine whether or data due to participant
not patterns over time attrition.
(trends) are homoge- • Time series studies are
neous across individuals. costly.
• Can examine relationship of
change patterns to
demographic and other
variables (including
experimental/intervention
variables).
• Can use advanced
statistical techniques to
make causal inferences.
Panel studies A series of different samples • Can explore many different • Panel studies are very
(panels) are initiated at factors that may influence time consuming,
different times; data are the observed patterns in costly, and subject to
collected from each dependent variables. high attrition rates
sample at the same time leading to missing
intervals. data.
Statistical Analysis of Both descriptive statistics and explanatory sta-

tistics can be used to answer the research question
Longitudinal Studies of this design. Descriptive statistics present the
means and standard deviations of each time meas-
Two primary functions in analyzing longitudinal ure and plot the findings. This visual presentation
data are: will demonstrate the impact of time on the out-
come variable. Both ANOVA and regression
• Descriptive and analyses can be used to study the relationship
• Explanatory. between other variables and the trend.
Since this is not a within-subject design, there
Descriptive analysis is used to characterize the
is no guarantee that the participants for the sample
dependent variable at different time points and to
representing each year will be homogeneous.
identify patterns of the dependent variable over
Therefore, the more heterogeneous participants’
time. Explanatory analysis is used to explore the
demographic characteristics are across samples,
relationship of observed patterns with other vari-
the more caution one must exercise in interpreting
ables. This section introduces selected analytical
trends as due to time. Table 10.2 presents an exam-
approaches for analyzing longitudinal data. For
ple of a trend study in which children ages 10 to 14
purposes of illustration, we use previously men-
years were sampled across a 10-year period to
tioned designs as examples for introducing differ-
examine changes in time spent in leisure and
ent analytical techniques. Our discussion focuses
schoolwork. If, in such a study, there was a sub-
only on continuous dependent variables, since
stantially different composition of participants
those who are interested in analyzing dichoto-
according to sex or race across the samples for
mous, categorical, or count-dependent variables in
each year represented, these differences might
longitudinal data are referred to readings in the
account for observed changes or be confounded
Resources list.
with the influence of time.
Analysis for Simultaneous Statistical Analysis for Time Series Studies
Cross-sectional Studies
Time series studies are within subject designs,
Consider the example of a researcher who wishes which mean that data are collected on each partic-
to know if the amount of time spent in self-care ipant at more than one time point. In these designs,
activities is a function of age in people 60 to 80 data from each time point are interrelated to data
years old. In the study example provided in Table collected at other time points. For instance, a per-
10.1, the investigator collected data at one time son’s score on a functional measure at one time
point with 11 samples each of different age groups, point in a time series study is likely to be corre-
each 2 years apart (60, 62, 64,…78, 80). lated with subsequent scores and so on. Therefore,
For descriptive analysis, this researcher would multilevel data analysis (sometimes called hierar-
calculate the means and standard deviations for chical linear models or mixed models) can be used
each of these 11 groups. The investigator could for data analysis.
also make a line chart to examine if the amount of In multilevel data analysis for longitudinal
time spent in self-care activities has changed from data, each individual person will be used as a unit
age 60 to age 80. These analyses describe the data, of analysis at the first level. Time is treated as a
but do not test whether the means are significantly predictor to model the growth pattern of the
different from each other. dependent variable. Thus each individual will have
The investigator can ask not only if there is any an intercept and regression coefficient(s). The
difference among these age groups in time spent in researcher can then study if these intercepts and
self-care, but also which age groups make the dif- regression coefficients vary among different indi-
ferences. Answers to these questions can be found viduals. If they do, the next step is to model these
by using a one-way analysis of variance (ANOVA) intercepts and regression coefficients to find out
(as described in Chapter 17 with post hoc tests what factors contribute to these variations. We
because in this design, independent samples repre- will demonstrate how to use this multilevel data
sent each age group). analysis technique in the “Analytic Examples”
section.
Analysis of Trend Studies
For trend analysis, the question to be answered by
Statistical Analysis for Panel Studies
the analysis is whether time has any impact on the Panel data studies include both within and between
outcome for a specific age group. designs, each participant is measured at at least
more than two time points, and each participant

One example of a study that utilized a repeated-
belongs to one cohort. Usually, panel data studies
measures design is a follow-up study of long-
have fewer numbers of time points than time series term quality of life outcomes for individuals with
studies. In our example, each cohort has five time chronic fatigue syndrome (Taylor, Thanawala,
points in the panel study and there are ten time Shiraishi, & Schoeny, in press). Researchers used
points in the time series study. a within-subjects, repeated-measures cohort
In panel studies, we can use either ANOVA design to evaluate the long-term effects of an
with repeated measures, fixed model ANOVA, or integrative rehabilitation program. Twenty-three
multilevel data analysis. The choice depends on participants with chronic fatigue syndrome
the researcher’s focus of interests as well as how attended eight sessions of an illness-management
the data were collected. If the researcher is inter- group followed by 7 months of individualized
peer counseling that took place once per week
ested in comparing the differences among time
for 30 minutes. Quality of life was measured five
points within the same cohort, ANOVA with times (i.e., at baseline, following the group
repeated measures can be used to achieve the goal. phase, following the one-on-one phase, and 4
If the interest is focusing on the pattern of changes and 12 months following program completion).
among persons or groups, multilevel data analysis A within-subjects repeated measures ANOVA
would be a better choice. Multilevel analysis is revealed significant increases in overall quality of
more flexible in terms of using random time inter- life for up to 1 year following program comple-
vals. For example, in ANOVA with repeated meas- tion (F (4, 21) 23.5, p 0.001). Although
ures, the time intervals are fixed to all individuals; definitive conclusions about program efficacy
however, there is no such limitation for multilevel were limited by the lack of follow-up data on a
control group (necessitating a within-subjects
data analysis.
design), findings suggested that the program may
have led to improvement in quality of life for up
Analytical Example to 1 year following program completion.
The following example is based on a fictitious data

set. We assumed that 20 participants were ran-
domly assigned to an experimental group and 20 to patterns of changes are presented in Figures 10.2
a control group. The participants in the experimen- and 10.3.
tal group received an intervention treatment that Figure 10.2 presents the ACIS measures of 20
is designed to enhance clients’ communication participants in the experimental group. The major-
and interaction skills. Each participant’s skills ity of participants have lower measures on the ini-
were measured four times by observation using tial ACIS, and they have a significant improvement
the Assessment of Communication and Interaction 3 months later, except two participants whose
Skills (ACIS) (Forsyth, Lai, & Kielhofner, 1999; measures drop from initial to 3 months. Almost all
Forsyth, Salamy, Simon, & Kielhofner, 1998), (except one) subjects had a small drop from 3 to
before intervention (initial) and 3 months, 6.5 6.5 months and later on. Communication and inter-
months, and 12 months after intervention. The action skills stabilized between 6.5 and 12 months,
108
96
84
72
60
48
36
24
Overall Pattern
12
0
Initial 3 Months 6.5 Months 12 Months
Figure 10.2 Communication and interaction skill scores for the experi-
mental group.
108
96
84
72
60
48
36
24
Overall Pattern
12
0
Figure 10.3 Communication and interaction skill scores for the control
group.
as there are no obvious changes between these two timepoints. Since time is a factor nested within
time points. each individual, it is called a within-subject factor.
In the control group (see Figure 10.3), four par- Whether or not the participants received the
ticipants significantly dropped from initial to 3 occupational therapy intervention is called a
months. Three participants showed major improve- between-subjects factor. Consequently, this is a
ments from initial to 3 months. Other participants one within-factor and one between-factor ANOVA
stayed about the same from baseline to 12 months. with repeated measures design. Three major ques-
Approximately four to five participants in each tions can be asked by this analysis:
group had very different initial ACIS measures
1. Is there a time effect on participants’ ACIS
when compared to the remaining group members.
measures? This question asks whether there is a
These people also showed a different pattern of
difference on the ACIS measure at the four dif-
change compared to the remaining members of the
ferent time points, regardless of whether the
group. Descriptive statistics for these two groups
participant received the intervention.
are presented in Table 10.5.
2. Is there a group mean difference (average of
four time points) on ACIS measures between
ANOVA with Repeated Measures experimental and control groups?
3. Is there an interaction effect between time and
We will demonstrate the ANOVA with repeated
intervention?
measures approach because it is a frequently used
method in analyzing longitudinal data. Basic con- Usually, the second question is not important to
cepts of ANOVA are covered in Chapter 17, so we a researcher in a longitudinal study because the
will not repeat them here. In our interaction investigator is more interested in the time effect
and communication intervention example, the and interaction effect between time and interven-
dependent variable is collected at four different tion. It is more useful to know if the intervention
Table 10.5 ACIS Means and Standard Deviations for Experimental and Control Group
at Four Different Times
Experimental 60.06 81.84 79.20 76.80
(8.88) (7.92) (7.32) (7.20)
Control 59.94 58.26 55.80 52.98
(7.86) (7.56) (7.32) (6.12)
Overall 60.00 70.08 67.50 64.86
(8.28) (14.16) (13.86) (13.74)
Numbers in parentheses are standard deviations.

resulted in a difference of communication skills multilevel analysis approach focuses on the

over time. If the interaction effect is significant, it changes of the trajectories instead of comparing
means we cannot conclude (as would be stated in the mean differences across time points. Multilevel
the null hypothesis) that the pattern of change is longitudinal data analysis is based on the concept
the same between experimental and control that there is a relationship between the time factor
groups. and the outcome variable and that this relationship
Results of this hypothetical study show that the can be either linear or nonlinear. This relationship
time factor and intervention factor have an interac- is reflected by a rate of change that indicates the
tion effect on the ACIS measures because the pat- relationship between the outcome variable and
terns of change between intervention and control time. The rate of change may differ from person to
groups are different (see Figure 10.2). The experi- person, and multilevel analysis uses this rate of
mental group showed an increase in communica- change as a dependent variable, and further
tion skills after receiving the intervention and then explores what factors have a relationship with this
declined slightly before stabilizing. In contrast, the rate of change.
control group (see Figure 10.3) that did not receive The researcher can examine variations within
the intervention showed an overall decline in com- and between individuals. There are three advan-
munication skills over time. tages of using this approach:
As shown in Table 10.5, before the interven-
tion, the ACIS measures do not differ significantly 1. This approach has less stringent restrictions
between experimental and control groups. Three concerning the characteristics of data being ana-
months after the intervention, the ACIS measures lyzed, compared to univariate ANOVA with
of the experimental group are significantly repeated measures. For example, the time inter-
improved; however, the control group decreases val is not required to be fixed, which means that
slightly compared to its initial ACIS measure. Both a subject does not have to be measured at certain
groups have shown a similar stable rate of decreas- fixed time points.
ing from 3 months on. 2. The parameter estimations are more precise
ANOVA results shown in Table 10.6 indicated because of using more advanced estimating
that both a time effect (initial to 3 months, 3 methods.
months to 6.5 months, and 6.5 months to 12 3. Missing data in first level are not an obstacle of
months, p .001), and a time and intervention using this approach.
interaction effect (initial to 3 months, p .01)
By examining the four time points in the ACIS
exist. The interaction effect exists only at the ini-
data example, we find a trend that the relationship
tial and 3-month data collection points and is not
between time and the ACIS score for experimental
significant for the remaining data collection points.
group is not merely a straight line (linear model),
but the trend of ACIS has one bend. This is the
Multilevel Data Analysis
reason why we add a quadratic component in the
Multilevel data analysis uses a different approach regression model and treat it as a nonlinear model.
to analyzing longitudinal data from that of the tra- In the experimental group, the improvement is dra-
ditional ANOVA with repeated measures. The matic from initial to 3 months, then it drops slowly
Table 10.6 Findings on ANOVA with Repeated Measures Testing Within-Subjects’ Contrasts
SS df MS F p
Time Initial–3 months 4,052.16 1 34.3 .001
3 months–6.5 months 261.90 1 129.4 .001
6.5 months–12 months 280.44 1 157.4 .001
Time Group Initial–3 months 5,498.09 1 46.5 .001
3 months–6.5 months .40 1 .2 .657
6.5 months–12 months 1.90 1 1.1 .309
Error Initial–3 months 4,490.64 38 118.17
3 months–6.5 months 76.68 38 2.02
6.5 months–12 months 67.68 38 1.78
Table 10.7 Final Estimation of Fixed Effects and Variance Components

Standard Approximate
Fixed Effect Coefficient Error t-ratio df p-value
Final estimation of fixed effects:
Status around 6 month

Control 56.64 1.72 32.95 38 .001
ExpControl 25.54 2.49 10.24 38 .001
Linear slope
Control –.60 .17 –3.45 38 0.002
ExpControl 2.21 .30 7.45 38 .001
Quadratic slope
Control .006 .04 0.16 38 0.871
ExpControl –.39 .06 –6.86 38 .001
Final estimation of variance components:

Standard Variance
Random Effect Deviation Component df Chi-square p-value
Status around 6 months 7.45 55.57 38 182.28 .001
Linear slope .80 .64 38 110.42 .001
Quadratic slope .16 .02 38 62.43 0.008
Level 1 4.62 21.31
from 3 months to 12 months. In the control group, data analysis. The estimated components in this
several people dropped very fast from initial to 3 level data analysis include:
months; other ACIS scores were stable from initial
to 3 months, and then dropped at a minimal but • The intercept, and
consistent rate. Based on this information, it is • The regression coefficients related to linear and
more appropriate to model the growth trajectories nonlinear (quadratic, cubic, or other types) com-
in a nonlinear fashion. ponents.
This involves estimating each person’s commu-
Each participant will have an intercept, a regres-
nication status at around 6 months after beginning
sion coefficient for the linear part, and a quadratic
this study (referred to as the intercept), linear term
part in our example. If the pattern of change is a
of growth (expressed as positive or negative
straight line, then we do not need to model the
regression coefficient), and quadratic term of
quadratic component. The second step of analysis
growth (expressed as an accelerating or deacceler-
is to treat these three estimated variables (intercept,
ating regression coefficient). These estimated val-
regression coefficients for linear and quadratic
ues of the intercept, regression coefficient for the
components) as three dependent variables in the
linear term, and regression coefficient for the
second-level analysis. We then use coded variables
quadratic term are then used as dependent vari-
to indicate if the participant belongs to the experi-
ables to compare the difference between experi-
mental or control group as a predictor.
mental and control groups on second-level data
Based on the aforementioned logic, we will be
analysis.1
able to obtain the following information in Table
In multilevel longitudinal data analysis, one
10.7. The communication skill measure, around 6
first models the growth trajectories in the first-level
months of the study, for the control group is 56.64
1Multilevel data analysis is also known as mixed effects and for the experimental group is 25.54 higher
models. There are different software programs that can be than for the control group. The linear growth tra-
used to analyze such data, among them, the most frequently
jectories for the control group is decreasing by .60
used are Hierarchical Linear Models (HLM), ML-Win
(Multi-level and SAS for Windows). This example is based per month; however, for the experimental group it
on the results of HLM. increases 1.61 (2.21.60) per month. The differ-
ence between the control and experimental groups Forsyth, K., Salamy, M., Simon, S., Kielhofner, G. (1998).
is significantly different (p .001). For the control A user’s guide to the assessment of communication and
interaction skills (ACIS). Bethesda: The American
group, the quadratic slope is .006, which is not sig- Occupational Therapy Association.
nificantly different from 0. This means that for the Keeves, J. P. (1988). Longitudinal research methods. In J.
control group the communication skill decreasing P. Keeves (Ed.), Educational research, methodology,
rate is kept consistent from month to month. For and measurement: An international handbook. Oxford:
Pergamon Press.
the experimental group, the communication skill
Kessler, R. C., & Greenberg, D. F. (1981). Linear panel
measure improves at beginning. However, when analysis: Models of quantitative change. New York:
time progresses, this growth rate slows down by Academic Press.
–.38 (–.39 .006) per month. The differences of Markus, G. B. (1979). Analyzing panel data. London:
quadratic slopes between experimental and control Sage.
Schaie, K. W. (1965). A general model for the study of
groups are also statistically significant (p .001). developmental problems. Psychological Bulletin, 64,
By examining the lower part of Table 10.7, we 92–107.
found that the residual variances of the second Singer, J. D., & Willett, J. B. (2003). Applied longitudinal
level three variables (intercept, regression coeffi- data analysis: Modeling change and event occurrence.
New York: Oxford University Press.
cients for linear and quadratic components) are
Sonn, U., Grimby, G., & Svanborg, A. (1996). Activities of
still significantly different from 0 after we model daily living studies longitudinally between 70 and 76
the intervention variable. This means that the years of age. Disability and Rehabilitation, 18(2),
researcher can still explore other predictors in 91–100.
addition to the intervention to explain the varia- Spector, W. D., & Takada, H. A. (1991). Characteristics of
nursing homes that affect resident outcomes. Journal of
tions among individuals, such as if sex or age Aging and Health, 3(4), 427–454.
make a difference or not. Taylor, R. R., Thanawala, S. G., Shiraishi, Y., & Schoeny,
M. E. (In press). Long-term outcomes of an integrative
rehabilitation program on quality of life: A follow-up.
Conclusion Journal of Psychosomatic Research.
von Eye, A. (1985). Longitudinal research methods. In T.
Huaén & T. N. Postlethwaite, (Eds.), The international
This chapter introduced longitudinal methods. We encyclopedia of education (Vol. 5, pp. 3140–3152).
noted that longitudinal research involves different Oxford and New York: Pergamon.
designs and analytical techniques. Since each
design has its purpose, strengths, and weaknesses, RESOURCES
researchers need to know specifically what they For Longitudinal Designs
want to study to determine the most appropriate Menard, S. (2002). Longitudinal research. Thousand Oaks,
design and analytical technique to use. CA: SAGE Publications.
Ruspini, E. (2002). Introduction to longitudinal research.
London: Routledge.
REFERENCES Schulsinger, F., Mednick, S. A., & Knop, J. (1981) (Ed).
Baltes, P. B., & Nesselroade, J. R. (1979). History and Longitudinal research: Methods and uses in behavioral
rationale of longitudinal research. In J. R. Nesselroade, science. Boston: Martinus Nijhoff.
& P.B. Baltes (Eds.), Longitudinal research in the study
of behavior and development. New York: Academic
For Longitudinal Analysis
Lindsey, J. K. (1993). Models for repeated measurements.
Press.
Oxford and New York: Oxford University Press..
Bijleveld, C. C. J. H., van der Kamp, L. J. Th., Mooijaart,
Little, T. D., Schnabel, K. U., & Baumert, J. (2000).
A., van der Kloot, W. A., van der Leeden, R., & van
Modeling longitudinal and multilevel data: Practical
der Burg, E. (1998). Longitudinal data analysis:
issues, applied approaches, and specific examples.
Designs, models and methods. London: SAGE
Mahwah, NJ: Lawrence Erlbaum.
Publications.
For Web sites of longitudinal studies: Ruspini (2002).
Forsyth, K., Lai, J., & Kielhofner, G. (1999). The assess-
ment of communication and interaction skills (ACIS): Panel design
Measurement properties. British Journal of Hsiao, C. (1986). Analysis of panel data. New York:
Occupational Therapy, 62(2), 69–74. Cambridge University Press.
C H A P T E R 1 1
Single-Subject Research
Jean Crosetto Deitz
A therapist working in a technology center wanted which the therapist is working with an individual
to know if her client, an individual with a high- with a low-incidence diagnosis or impairment. A
level spinal cord injury, would be more successful further advantage of single-subject research is that
in terms of text entry using system A as opposed to it does not require the withholding of treatment
system B. A second therapist, working with four from a no-treatment control group. Because each
individuals who were participant serves as his or
institutionalized with her own control, the treat-
chronic depression, Single-subject research ment typically is withheld
wanted to know if these methods are useful for from the participant for
individuals would initi- one or more periods of
ate conversation more answering questions regard- time and then it is insti-
in their group therapy ing the effectiveness of spe- tuted or reinstituted. Also,
session if it followed a because only one or a
pet therapy session. cific interventions for specific small number of individu-
These are the types of individuals by providing als are studied, the finan-
questions that therapists cial and time demands are
continually ask, and
experimental control and by realistic for practice set-
both of these questions contributing to clear and pre- tings. Even though only
can be addressed using one or a small number of
single-subject research
cise clinical documentation. individuals are studied,
methods. According to the findings from single-
Backman, Harris, Chisholm, and Monette (1997), subject research can be used to inform practice and
these methods “can inform and illuminate clinical to justify and inform larger scale investigations
practice” and “provide concrete data to validate (Ottenbacher, 1990). Information learned in the
existing theories in rehabilitation as well as formu- process of single-subject research can be used in
late new ones” (p. 1143). designing more costly group experimental studies
Single-subject research, sometimes referred to involving numerous participants and multiple
as single-system research, is based on within-sub- sites.
ject performance, with the unit of study being one
person or a single group that is considered collec-
tively. Each participant serves as his or her own Common Single-Subject
control and there are repeated measurements over
time of the same dependent variable or variables.
Research Designs
Also, while other factors are held constant, there
A simple notation system is used for single-subject
is systematic application, withdrawal, and some-
research designs where A represents baseline; B
times variation of the intervention (independent
represents the intervention phase; and C and all
variable).
other letters represent additional interventions or
Single-subject research methods are useful for
conditions.
answering questions regarding the effectiveness of
specific interventions for specific individuals by
The A–B Design and Variations
providing experimental control and by contribut-
ing to clear and precise clinical documentation. The design upon which all others are built is the
They are especially suited to occupational therapy A–B design, where A indicates a baseline phase
research because groups of persons with similar and B indicates a treatment (or intervention) phase.
characteristics are not required, and thus these This can be exemplified by looking at both the first
methods are appropriate for use when the therapist baseline phase (A) and the treatment phase (B) in
has one, or at most, a few clients with whom a par- Figure 11.2 in the feature box for the therapy ball
ticular intervention is employed, or in situations in study. This displays the data for Emily for percent-
Chapter 11 Single-Subject Research 141
age of intervals seated. The vertical axis of the for seating during language arts. Note that the per-
graph indicates the percentage of intervals seated centage of intervals seated increased substantially
and the horizontal axis indicates the days. During during the treatment phase. If the study had
the first baseline phase (A), Emily experienced 12 stopped after data were collected for only a base-
days of sitting on classroom chairs as usual and line phase and an intervention phase, this would
data were systematically kept on percentage of have been an A–B design.
intervals seated. Note that Emily’s percentages of A common variation of the A–B design is the
intervals seated were consistently below 60%. A–B–C successive intervention design. With this
Following the novelty week, phase B (interven- design, a second intervention is introduced in the
tion) was started and Emily used the therapy ball C phase. For example, the therapist studying the
The Therapy Ball Study
Example of a Single-Subject Research Study

Designed to Answer a Practice Question
An interdisciplinary group wanted to know if the
use of therapy balls as classroom seating devices
affected the behavior of students with attention
deficit hyperactivity disorder (ADHD) (Schilling,
Washington, Billingsley, & Deitz, 2003). Their
first research question was, “What effect does
using therapy balls as chairs have on in-seat
behavior?” (Schilling et al., 2003, p. 535). The
convenience sample for the study consisted of
three children (two males and one female) with
ADHD and average intelligence. The three
children were from the same fourth-grade
public school classroom and, during language
arts, all demonstrated out-of-seat behavior
requiring repeated teacher verbal and/or
physical prompts.
The dependent variable, “in-seat behavior,” was
measured whereby following each 10-second obser-
vation a rater scored the participant’s behavior as
either “out-of-seat” or “in-seat” (Schilling et al.,
2003). Each participant was observed for five 2-
minute periods during language arts, for a total of
60 observations per session. Data systematically
were collected for “in-seat behavior” for each of the
participants individually. During language arts, the
three participants and all other class members expe-
rienced (1) a series of days during which chairs
were used for seating (first baseline phase); (2) a Figure 11.1 A study participant is observed
1-week novelty period during which balls were used working while using a therapy ball as a chair.
for seating; (3) a series of days during which balls
were used for seating (first intervention phase); (4) a participant’s medication for ADHD was held con-
series of days during which chairs were used for stant in terms of type and dosage.
seating (second baseline phase); and (5) last, a Throughout all baseline and intervention phases,
series of days during which balls were used for even though all of the children in the classroom par-
seating (second intervention phase). Throughout the ticipated in the intervention, data were collected
study, efforts were made to hold all factors constant only for the three children with ADHD. Data were
that could potentially have influenced the results of graphed and examined for each of the three partici-
the study. For example, the teacher was the same pants individually. Refer to Figure 11.2 for an
each day, data were collected at the same time each example of graphed data for percentage of intervals
day, and, throughout the duration of the study, each seated for Emily (pseudonym).
(continued)
The Therapy Ball Study (continued)
The data indicates that Emily was “in-seat” a increased when the intervention was reinstituted.
higher percentage of intervals when using the ther- The graphs for the other two participants were
apy ball for seating during language arts than when similar to the graph depicted in Figure 11.2, thus
using a chair for seating. For Emily, confidence in providing additional support for the use of therapy
the effect of the intervention is supported since her balls for classroom seating for fourth-grade students
“in-seat behavior” increased immediately following with ADHD. Studies such as the one just described
institution of the intervention, dropped markedly support evidence-based practice.
when the intervention was removed, and again
A B A B
100
Percentage of Intervals Seated
80
Novelty Week
60
In-seat
40
20
Emily
0
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Mon
Calendar Days
Figure 11.2 Graph of Emily’s “in-seat behavior.” Connected data points represent consecutive
days within the same week. Variability in the number of data points was the result of a non-
school day or absence from class. (Copyright 2003 by the American Occupational Therapy
Association, Inc., Reprinted with permission.)
effects of the alternative seating devices might in which the goal is stepwise increases in accu-
have chosen to introduce the ball for seating in racy (e.g., quality of letters written), frequency
the B phase and an air (e.g., number of repeti-
cushion for seating in tions completed), dura-
the C phase in an attempt Multiple-baseline designs tion (e.g., length of
to see if one device was exercise session), latency
more effective than the
require repeated measures (e.g., time it takes to
other. Another variation of at least three baseline begin to respond after a
of the A–B design is question is asked), and
the A–B–C changing-
conditions that typically are magnitude (e.g., pounds
criterion design, which implemented concurrently of weight lifted) (Hart-
is characterized by hav- with each successive mann & Hall, 1976).
ing three or more phases Consider a client who
with the criterion for baseline being longer than seldom or never exer-
success changing se- the previous one. Multiple- cises and is enrolled in a
quentially from one health promotion pro-
intervention phase to baseline designs can be (a) gram because of health
the next. This design is across behaviors; (b) across concerns. The baseline
suited to interventions phase (A) might involve
that are modified in a participants; or (c) across recording the number of
stepwise manner since settings. continuous minutes the
the criterion for success individual walks on a
is changed incrementally treadmill at 1.5 miles per
with each successive intervention phase (Hart- hour. During the first intervention phase (B) the
mann & Hall, 1976). It is appropriate for situations criterion for success might be 30 continuous min-
utes of treadmill walking at 1.5 miles per hour; the therapy ball likely resulted in an increase in
during the second intervention phase (C) the crite- “in-seat behavior” for Emily during language arts.
rion for success might be 30 continuous minutes of
treadmill walking at 2.0 miles per hour; and during
the third intervention phase (D) the criterion for Multiple-Baseline Designs
success might be 30 continuous minutes of tread- Multiple-baseline designs, the next category of
mill walking at 2.5 miles per hour. This could con- designs, require repeated measures of at least three
tinue until the desired rate of walking was baseline conditions that typically are implemented
achieved. The changing criterion design also is concurrently with each successive baseline being
useful in situations in which stepwise decreases in longer than the previous one. Multiple-baseline
specific behaviors are desired. designs can be (a) across behaviors; (b) across par-
With both the A–B and A–B–C designs, no ticipants; or (c) across settings.
causal statements can be made. In the therapy ball
study, if data had been collected only for an initial Multiple-Baseline Design
baseline phase and a treatment phase (see Figure
Across Behaviors
11.2 located in the feature box on the therapy ball
study), the therapist would not have known if some In a multiple-baseline design across behaviors, the
factor other than the introduction of the therapy same treatment variable is applied sequentially to
ball for classroom seating resulted in Emily’s separate behaviors in a single participant. Consider
increase in “in-seat behavior.” For example, at the the hypothetical example of an adult with demen-
beginning of the intervention phase, the language tia who frequently displays three antisocial behav-
arts assignments might have changed to a topic of iors: swearing, door slamming, and screaming. The
greater interest to Emily and that change, rather therapist is interested in knowing whether or not
than the therapy ball, might have influenced her intervention is successful in reducing or elimi-
Emily’s “in-seat behavior.” Therefore, this design nating the frequency of occurrence of these behav-
is subject to threats to internal validity. iors. For 5 days, during a 2-hour socialization
group the researcher collects baseline data on these
Withdrawal Designs behaviors, making no change in intervention
(Figure 11.3 on p. 144). On the sixth day, the
The A–B–A or withdrawal design has stronger researcher introduces the intervention, thus starting
internal validity than the A–B and A–B–C designs. the treatment phase (B) for the first behavior
This design consists of a minimum of three phases: (swearing). The researcher makes no change in the
baseline (A), intervention (B), and baseline (A) or treatment program for door slamming and scream-
conversely B–A–B. However, it can extend to ing. These two behaviors remain in baseline (A).
include more phases, with the A–B–A–B being one After 10 days the researcher initiates the same
of the most common. The latter has ethical appeal intervention for door slamming, and, after 15 days,
in situations in which the intervention is effective she initiates this same intervention for screaming.
since you do not end by withdrawing the interven- If the researcher can demonstrate a change across
tion. Withdrawal designs are exemplified by the all three behaviors following the institution of the
therapy ball study in which the therapy ball for intervention, this provides support for the effec-
seating was removed after the intervention phase tiveness of the intervention in decreasing antisocial
and data were again collected under baseline con- behaviors in the adult studied. This exemplifies a
ditions and then the intervention was reinstituted multiple-baseline design across behaviors.
(see Figure 11.2 in the therapy ball study featured
earlier in this chapter). Because this design Multiple-Baseline Design
involves a return to baseline, it is most appropriate Across Participants
for behaviors that are reversible (likely to return to
the original baseline levels when intervention is With a multiple-baseline design across partici-
withdrawn). This is a true experimental design in pants, one behavior is treated sequentially across
the sense that causal inferences can be made matched participants. For example, if you had
related to the participant or participants studied. three men with limited grip strength, you might
For example, in the therapy ball study, because the institute a specific intervention for the first on the
percentage of “in-seat behavior” increased during 8th day, for the second on the 16th day, and for the
the intervention phase and then returned to the third on the 26th day. Figure 11.4 on p. 145 dis-
original baseline levels during the second baseline plays hypothetical data for such a study.
phase and then increased when the intervention A variation of the multiple-baseline design
was reinstituted, it is possible to say that the use of across participants is the nonconcurrent multiple-
20 (A) (B)
Frequency
15
10
Swearing
5
0
0 5 10 15 20 25
Days
20 (A) (B)
Frequency
15
Door Slamming
10
0
0 5 10 15 20 25
Days
20 (A) (B)
Frequency
15
10
Screaming
0
0 5 10 15 20 25
Days
Figure 11.3 Multiple-baseline design across behaviors—example of

graphed data.
baseline design across individuals (Watson & each of the three participants, the researcher would
Workman, 1981), whereby different individuals randomly select without replacement one of the
are studied at different times, a feature that makes baseline lengths. See Figure 11.5 on p. 146 for a
it ideal for clinical settings where it often is imprac- graphic display of hypothetical data. Since baseline
tical to start multiple clients in a study simul- data collection typically is continued until a stable
taneously. With this design, the researcher prede- pattern emerges, Watson and Workman (1981) rec-
termines baseline lengths and randomly assigns ommend dropping a participant if his or her base-
these to participants as they become available. For line data do not achieve stability within the
example, in a nonconcurrent multiple baseline predetermined time for the participant’s baseline.
design across three individuals, the researcher Because each of the participants in a nonconcurrent
might choose baseline lengths of 6 days, 10 days, multiple baseline study starts the intervention at a
and 13 days. The first participant might start the different randomly determined time, some control
study on April 1st, the second might start on April is provided for other variables that could result in
13th, and the last might start on April 26th. For desired changes in the target behavior.
80 (A) (B)
60 Participant 1
Pounds 40
20
0
0 10 20 30 40 50 60
Days
80 (A) (B)
60 Participant 2
Pounds
40
20
0
0 10 20 30 40 50 60
Days
80 (A) (B)
60 Participant 3
Pounds
40
20
0
0 10 20 30 40 50 60
Days
Figure 11.4 Multiple-baseline design across participants—example of

graphed data.
Multiple-Baseline Design line data in the other two settings. After 3 more
Across Settings days, she introduces the intervention in the second
setting (the cafeteria), while continuing to collect
With a multiple-baseline design across settings, baseline data in the third setting. Last, after 3 more
the same behavior or behaviors are studied in sev- days, she introduces the intervention in the third
eral independent settings. Consider the child with setting (the playground). Refer to Figure 11.6 on
an autism spectrum disorder who is in an inclusive p. 147 for a graph of hypothetical data.
school setting. This child repeatedly interrupts. He With multiple-baseline designs, intervention
does this in the classroom, in the cafeteria, and on effectiveness is demonstrated if a desired change
the playground. The therapist collects baseline in level, trend, or variability occurs only when the
data in all three settings for 5 days. Then, she tries intervention is introduced. In addition, the change
an intervention in the classroom (the first setting), in performance should be maintained throughout
while simultaneously continuing to collect base- the intervention phase.
Participant 1
(A) (B)
Participant 2
(A) (B)
Participant 3
(A) (B)
0 10 20 30 40 50
Days of Study
Figure 11.5 Nonconcurrent multiple-baseline design across participants—example of
graphed data.
Multiple-baseline designs have three major The primary weakness of multiple-baseline

strengths. The first relates to internal validity. designs is that they require more data collection
Because the intervention is started at a different time because of the staggered starting times for the
time for each individual, behavior, or setting, these intervention phases. Because of this, some behav-
designs help to rule out the internal validity threat iors or participants are required to remain in the
of history as described by Campbell and Stanley baseline phase for long periods of time, which may
(1963). For more detail regarding internal and prove to be problematic. For example, in the first
external validity, refer to Chapter 7. Also, because hypothetical study involving an adult with demen-
results of research using these designs can show tia (see Figure 11.3), the last behavior (screaming)
that change is effected in multiple individuals, was allowed to continue for 15 days prior to the
behaviors, or settings, support is provided for the institution of the intervention.
demonstration of causal relationships. The second
strength of multiple-baseline designs is that they Alternating-Treatments Design
require no reversal or withdrawal of the interven-
tion, a characteristic that makes them appealing The alternating-treatments design and/or minor
and practical for research when discontinuing ther- variations of it also have been termed the multi-
apy is contraindicated. The third strength of multielement baseline design, the randomization
ple-baseline designs is that they are useful when design, and the multiple schedule design (Barlow
behaviors are not likely to be reversible. Often, & Hayes, 1979). These designs can be used to
therapists expect that an intervention will cause a compare the effects of intervention and no inter-
difference that will be maintained even when the vention, or they can be used to compare the effects
intervention is withdrawn. For example, once a of two or more distinct interventions. This can
therapist has taught a client who has had a spinal extend to comparing the effectiveness of two or
cord injury to dress independently, the therapist more different therapists or the effectiveness of
expects the client will be able to dress independ- providing a specific intervention at one time of day
ently, even when therapy is withdrawn. This makes versus another. In all cases, these designs involve
it inappropriate to use the withdrawal design dis- the fast alternation of two or more different inter-
cussed earlier, because with this design, interven- ventions or conditions. Though they typically have
tion effectiveness is demonstrated when the a baseline phase, it is not essential. These designs
behavior returns to the original baseline level fol- are not appropriate when behavior is expected to
lowing withdrawal of the intervention. take time to change, when effects are expected to
20 (A) (B)
Frequency
15 Classroom
10
0
0 5 10 15 20
Days
20 (A) (B)
Frequency
15
Cafeteria
10
0
0 5 10 15 20
Days
20 (A) (B)
Frequency
15 Playground
10
0
Figure 11.6 Multiple-baseline design across 0 5 10 15 20
settings—example of graphed data. Days
be cumulative, or when multiple treatment inter- Because alternating-treatments designs require

ference (described in Chapter 7) is anticipated. interventions that will produce immediate and dis-
Alternating-treatments designs are useful in sit- tinct changes in behavior, they are suited to inde-
uations when change is expected in the dependent pendent variables that can be instituted and
variable over time because of factors such as a dis- removed quickly and whose effects typically are
ease process (e.g., rheumatoid arthritis) or a natu- immediate. Thus, they are suited to studying the
ral recovery process (e.g., strokes, burns). With effects of independent variables such as splints,
other designs, such factors could result in changes orthoses, positioning devices, and assistive tech-
in the dependent variable that could compromise nology. This design also has been used when
interpretation of study results. However, with the studying the effects of interventions on dependent
alternating-treatments design, effects from these variables such as self-report measures, overt
changes largely are controlled because each inter- behavior exhibited during specific interventions,
vention is instituted within a short period of and exercise repetitions. For example, Melchert-
time, typically 1 or 2 days. See the feature box McKearnan, Deitz, Engel, and White (2000) stud-
titled Example (Hypothetical) Study Using an ied the effects of play activities versus rote
Alternating-Treatments Design to further illustrate exercise for children during the acute phase fol-
the use of this design. lowing burn injuries. Outcome measures were
number of repetitions of therapeutic exercise com- condition or where ethically it is difficult to justify
pleted, number and type of overt distress behaviors baseline data collection.
displayed, scores on self-report scales of pain The primary weakness of the alternating-
intensity, and self-report of overall enjoyment of treatments design stems from its vulnerability to a
the activity. The alternating-treatments design was validity threat relating to the influence of an inter-
chosen for this study because it was expected that vention on an adjacent intervention (multiple treat-
there would be changes in the children as a result ment interference). As a partial control for this
of the recovery process and it was expected that threat, all variables that could potentially influence
the effect of the intervention on each of the out- the results of the study should be counterbalanced.
come variables would be immediate. For example, if a therapist was trying to determine
The alternating-treatments design has three pri- which of two electric feeders was best for his client
mary strengths. First, it does not require a lengthy relative to time to complete a meal, and feeder A
withdrawal of the intervention, which may result always was used at lunch and feeder B always
in a reversal of therapeutic gain. Second, it often was used at dinner, it might be that the client did
requires less time for a comparison to be made more poorly with feeder B because he strained
because a second baseline is not required. Third, his neck when using feeder A at lunch and this
with this design, it is possible to proceed without a influenced his use of feeder B. Randomizing the
formal baseline phase. This is useful in situations order of use of the feeders, though it would not
where there is no practical or meaningful baseline eliminate order effects, would allow for order
Example (Hypothetical) Study Using an per hour were collected for five sessions. The
Alternating-Treatments Design woman did not wear the splint during this phase.
Phase 2 began after the woman wore the splint
A woman’s productivity on an assembly line is for a 1-week accommodation period. During the
compromised because of arthritis. The therapist 16 days of phase 2, the order of the two conditions
wants to know if use of a specific splint will result (with and without splint) was counterbalanced by
in an increase in productivity. The independent random assignment without replacement.
variable is the splint and the dependent variable is Therefore, for 8 of the 16 days in phase 2, the
the number of assembly units completed per hour. woman wore the splint at work; for the other 8
See Figure 11.7 for a graph of hypothetical data, days she did not use the splint. In both conditions,
noting that data for both the intervention and non- data were systematically collected on number of
intervention conditions are plotted on the same assembly units completed per hour. As depicted in
graph. the graph, the woman’s work productivity
During the first phase of the study, baseline improved, thereby providing support for the use of
data on the number of assembly units completed the splint in the work setting for this woman.
60
1-week accommodation period
50
Assembly Units Completed
40
30
20
10 No Splint
Splint
0
Monday Monday Monday Monday Monday
Figure 11.7 Alternating-treatments design—example of graphed data.
effects to be identified if they did exist (Hains & such as every 5 or 10 seconds, or uses videotapes
Baer, 1989). of recorded sessions with beeps superimposed
on the tapes. The data collector typically records
Variations responses on a recording sheet such as the one
presented in Figure 11.8 on p. 150, which was
The research designs described are only a sample
designed for use for 10 minutes of data collection
of the possibilities. The literature is replete with
with 5-second intervals. For each interval, the re-
creative examples of design variations. For exam-
searcher circles either an “I” for “in-seat” or an
ple, Wacker and colleagues (1990) advocated for
“O” for “out of seat.” This approach of making the
the sequential alternating-treatments design, a
same mark (a circle) regardless of whether “in
combination of the alternating-treatments and
seat” or “out of seat” is scored, is important in
multiple-baseline designs, and others have sug-
situations where two data collectors, in close
gested extending the withdrawal design to include
proximity, must collect data simultaneously for
more phases and multiple intervention conditions
reliability checks. Otherwise, the subtle sounds of
(e.g., A–B–A–C–A–D–A–C–A–D–A–B–A).
making an “I” or an “O” might inadvertently pro-
vide a cue to the other data collector, possibly
Definition of Variables biasing results.
and Collection of Data

Similar to group research, dependent variables
Data Reporting and Analysis
have to be operationally defined and data collection
methods that are replicable and reliable must be
Data Graphs
used. In addition to using physiologic measures Typically, with single-subject research, data for
(e.g., oxygen saturation), strength or endurance each variable for each participant or system are
measures (e.g., pounds of grip strength, time spent graphed with the dependent variable on the y-axis
exercising at a given level), frequency measures and time (e.g., days, weeks) on the x-axis. Vertical
(e.g., number of bites of food taken without lines indicate phase changes, and lines connect
spilling during a meal), and self-report measures data points reflecting consecutive days. See Figure
(e.g., level of pain, level of satisfaction), interval- 11.2 for an example. When graphing data for one
recording techniques often are used. There are variable for more than one participant, the scale for
three common interval-recording techniques: each graph should be the same to facilitate com-
momentary time sampling, partial interval record- parisons across graphs. For example, if one partic-
ing, and whole interval recording (Harrop & ipant’s data ranged from 3 to 29, and another’s
Daniels, 1986; Richards, Taylor, Ramasamy, & ranged from 20 to 58, both graphs should start
Richards, 1999). With momentary time sampling, a at 0 and extend to at least 60. This facilitates
response is recorded if it occurs precisely at a pre- visual comparison across participants. Carr and
determined moment. For example, consider a study Burkholder (1998) described the process for creat-
in which “in seat behavior” was the dependent ing graphs for single-subject research using
variable. Using momentary time sampling, a data Microsoft Excel TM.
collector, listening to a tape of beeps recorded at 5- Researchers graph data on either equal interval
second intervals, would record whether or not the graph paper or the standard behavior chart (some-
child was “in seat” at the moment of each beep. By times referred to as six-cycle graph paper). The
contrast, with partial-interval recording, a response primary benefit of the former is that it is easily
is scored if it occurs in any part of the interval. For understood. Benefits of the standard behavior
example, in the previous example, the child would chart are that behaviors with extremely high or
be scored as “in seat” if she was in her seat during low rates can be recorded on the chart and the
the first 2 seconds of the 5-second interval and out graph progresses in semilog units thus facilitating
of her seat during the last 3 seconds of that inter- the estimation of linear trends in the data (Carr &
val. With whole-interval recording, the child would Williams, 1982).
have to be in her seat during the full 5-second inter-
val in order to be scored as “in seat.”
Phase Lengths
Typically, with interval-recording techniques,
either the data collector uses headphones to hear Though the researcher estimates phase lengths
a tape of beeps recorded at regular intervals, prior to implementation of the study, typically the
Data Recording Sheet for In-Seat/Out-of-Seat Study
Instructions: For each interval, circle either "I" for "in-seat" or "O" for "out of seat."
min. 1 I O I O I O I O I O I O I O I O I O I O I O I O
KEY
I = In seat
O = Out of seat
Figure 11.8 Sample recording sheet for interval recording (momentary time sampling,
whole-interval recording, or partial-interval recording).
actual length of each phase is determined during involves looking for a change in level, trend, or
the course of the study with data collection within variability between phases when treatment is insti-
a phase continuing until a clear pattern emerges. In tuted or withdrawn (Ottenbacher & York, 1984;
determining estimates for phase lengths prior to Wolery & Harris, 1982). See Figure 11.9 for
study implementation, several factors should be potential patterns of data reflecting no change and
taken into account. First, the researcher should change.
decide whether or not he or she intends to use sta- In some cases, incorporating additional des-
tistical analyses because these often necessitate a criptive information into graphs can be used to
specified minimum number of data points in each augment visual data analyses. For example, the
condition in order to meet the required assumptions split-middle technique can be used to describe data
for their use. Second, in situations where change is and predict outcomes given the rate of change. The
expected due to factors such as development or a accuracy of these predictions depends on the num-
disease process, the researcher should consider ber of data points on which the prediction is based
making the lengths of baseline and intervention and on how far into the future a prediction is being
phases comparable. This facilitates the visual made. The split-middle technique also can be used
analyses of the resulting data. Third, the researcher to facilitate the examination and comparison of
should consider the variability in the expected trends in two or more phases.
data and the magnitude of the expected change. In Using the split-middle technique, once data for
cases where high variability and/or small (but a phase are plotted on a graph, a celeration line
important) changes are expected, longer phases are reflecting the direction and rate of change is deter-
advised. mined. For details concerning the process for
using the split-middle technique and creating cel-
eration lines refer to Kazdin (1982), Barlow and
Visual Analysis
Hersen (1984), or Ottenbacher (1986). Though
Visual analysis of graphically presented data these strategies typically are used to facilitate
to infer conclusions about cause and effect visual analyses, statistical change can be evaluated
No Change No Change No Change
Change in Level Change in Trend Change in Variability

Figure 11.9 Patterns of data reflecting no change and change.
by applying a binomial test to determine whether Statistical Significance

or not a significant proportion of data points in
an intervention phase fall above or below the If the researcher using single-subject methods plans
celeration line projected from baseline. For com- to determine statistical significance for a study to
putational details refer to the writings of White supplement visual analysis, the researcher needs to
(1972), Barlow and Hersen (1984), or Ottenbacher design the study to meet the necessary assump-
(1986). tions. One of the best and most underutilized statis-
Another example of descriptive information tical tests appropriate for use in single-subject
that can be incorporated into graphs to augment research is the randomization test (Edgington,
visual analyses is the use of a dashed line across 1980, 1995; Onghena & Edgington, 1994; Todman
each phase demarcating the mean for that phase. & Dugard, 2001; Wampold & Worsham, 1986). It
Though useful in some situations, this approach is a conservative way to evaluate shifts in time-
can contribute to misinterpretations of the data series data and is ideal because it is robust to serial
when the data reflect either an upward or down- dependency (i.e., correlation among successive
ward trend, when there are few data points within responses in one individual) and systematic trends
a phase, and when data points within a phase are in the data and it does not require a normal distri-
highly variable. For example, relative to the for- bution. It focuses exclusively on the data available,
mer, it is possible to have similar means in adja- evaluating the probability of the actual outcome
cent A and B phases, thus suggesting no effect when compared to the set of all possible outcomes.
from the intervention. However, if there was a If a researcher plans to use randomization tests
steady increase in an undesirable behavior to analyze data from single-subject research, it is
(upward trend) in the A phase and a steady important that the researcher randomly assign
decrease in the undesirable behavior (downward experimental conditions (Edgington, 1980, 1995).
trend) in the B phase, comparison of the means In doing this, the researcher should consider the
could lead to the erroneous conclusion of no dif- need to control for systematic trends over the
ference between the two phases. See Figure 11.10 course of the study that may be attributable to
on p. 152 for a hypothetical example. developmental or recovery processes.
10
8
Frequency
6
Mean = 5 Mean = 5
0 Figure 11.10 Example of mis-

0 1 2 3 4 5 6 7 8 9 10 11 leading use of comparison of
Days means across phases.
Incorporation of Social target children and all other children in the class
and the teacher filled out social validity question-
Validation Procedures naires. Second, at the conclusion of the study, all
into the Design were instructed to write about what they “liked”
and “didn’t like” about using balls for classroom
seating. Last, evidence of social validity was
According to Schwartz and Baer (1991), the pur- obtained by observing the teacher’s choices fol-
pose of assessing social validity is to evaluate the lowing the completion of the study. She continued
acceptability or viability of an intervention. to use therapy balls for seating for the children
Typically this is accomplished through use of one with ADHD and she ordered additional balls for
or more questionnaires directed to the research other students, thus supporting the social validity
participants and others in their environments (e.g., of the intervention.
family members, teachers). Social validation is
important since it is possible for an intervention to
result in desirable changes in the dependent vari-
able, but still be identified by research participants The Issue of Generality
or other stakeholders as being unacceptable. For
example, results from a research study may indi- Though a well-designed and implemented single-
cate that use of a specific type of splint decreases subject study completed with one individual has
daily reports of pain strong internal validity,
and increases range of it is weak relative to
motion. However, in a Social validation is important external validity (dis-
social validation ques- since it is possible for an cussed in Chapter 7).
tionnaire, the research Thus, though we may
participant may indicate intervention to result in desir- have confidence in the
that the splint limits able changes in the depend- findings for the individ-
function and is cosmeti- ual studied, we do not
cally unacceptable and ent variable, but still be know the extent to which
therefore would not be identified by research partici- these findings general-
chosen for use. ize to other comparable
Social validation pro- pants or other stakeholders individuals. Generality
cedures were used in as being unacceptable. in single-subject re-
the therapy ball study search is achieved only
described in the feature through replication with
box that appeared earlier in this chapter (Schilling other individuals, in other settings, and with other
et al., 2003). This involved three steps. First, the therapists. For example, in the therapy ball study
Questions to Answer When Designing a Single-Subject Research Study
(Though the following questions are listed in an to the intervention as intended and described
order, in practice, the process is iterative and may (intervention fidelity)?
vary in sequence.) • What is (are) the dependent variable(s)?
• How will I ensure that my measurements are
• What is the purpose of my study and why is it reliable?
important? • What is the design of my study and what is my
• What is my research question? rationale for selecting that design? (Do I plan
• What literature do I need to review in order to to use a test for statistical significance, and if
understand the topic, the measurement issues, the so, how will that influence the design of my
research design, etc.? study?)
• Who are my participants and how will I select • How will I graph and analyze my data?
them? • How will I assess social validity?
• What is the independent variable (intervention • What are the strengths and limitations of my pro-
strategy)? posed study?
• How will I ensure that the intervention imple- • What ethical factors should I address and what
mented in my study is consistently exact and true are the Institutional Review Board requirements?
described previously, two other children, in addi- successful with different clients. These methods
tion to Emily, were studied. Because results for are congruent with responsible occupational ther-
all three children were similar, some support is apy practice in that they emphasize clearly articu-
provided for the gener- lating desired outcomes,
ality of the findings. defining intervention and
Additional replications Both the process of single- measurement strategies,
by other researchers, in and collecting data relia-
other settings, and with subject research and the bility over time. Both the
different children would products of single-subject process of single-subject
further increase general- research and the products
ity if comparable out-
research support evidence- of single-subject research
comes continued to be based practice. support evidence-based
achieved. practice. Questions for
Kazdin (1982) distin- the clinician/researcher
guished between direct replication that involves to answer when designing a single subject research
systematically applying the same procedures study appear in the following feature box.
across different, but similar, individuals and sys-
tematic replication that involves repeating the REFERENCES
study with systematic variations. An example of Backman, C., Harris, S. R., Chisholm, J., & Monette, A. D.
the latter would be to design and conduct a repli- (1997). Single-subject research in rehabilitation: A
cation study using different types of participants review of studies using AB, withdrawal, multiple base-
(e.g., children with autism instead of children with line, and alternating treatments designs. Archives of
Physical Medicine and Rehabilitation, 78, 1145–1153.
attention deficit hyperactivity disorder [ADHD]). Barlow, D. H., & Hayes, S. C. (1979). Alternating treat-
Ideally, in a systematic replication study, only one ments design: One strategy for comparing the effects of
variable is changed at a time. Systematic replica- two treatments in a single subject. Journal of Applied
tions expand understanding regarding with whom Behavior Analysis, 12, 199–210.
Barlow, D. H., & Hersen, M. (1984). Single case experi-
and under what conditions the intervention is mental designs strategies for studying behavior change
effective. (2nd ed.). New York: Pergamon Press.
Campbell, D., & Stanley, J. (1963). Experimental and
quasi-experimental designs for research. Chicago:
Rand McNally.
Conclusion Carr, J. E., & Burkholder, E. O. (1998). Creating single-
subject design graphs with Microsoft ExcelTM. Journal
of Applied Behavior Analysis, 31, 245–251.
Single-subject research methods are useful in Carr, B. S., & Williams, M. (1982). Analysis of therapeutic
addressing many important questions related to the techniques through use of the standard behavior chart.
extent to which different intervention strategies are Physical Therapy, 52, 177–183.
Edgington, E. S. (1980). Random assignment and statisti- Schilling, D. L., Washington, K., Billingsley, F., & Deitz, J.
cal tests for one-subject experiments. Behavioral (2003). Classroom seating for children with attention
Assessment, 2, 19–28. deficit hyperactivity disorder: Therapy balls versus
Edgington E.S. (1995). Single subject randomization tests. chairs. The American Journal of Occupational Therapy,
In E. S. Edgington (Ed.), Randomization tests (3rd ed., 57, 534–541.
pp. 263–301). New York: Marcel Dekker. Schwartz, I. S., & Baer, D. M. (1991). Social validity
Hains, H., & Baer, D. M. (1989). Interaction effects in assessments: Is current practice state of the art? Journal
multielement designs: Inevitable, desirable, and ignor- of Applied Behavior Analysis, 24, 189–204.
able. Journal of Applied Behavior Analysis, 22, 57–69. Todman, J. B., & Dugard, P. (2001). Single-case and
Harrop, A., & Daniels, M. (1986). Methods of time sam- small-n experimental designs a practical guide to
pling: A reappraisal of momentary time sampling and randomization tests. Mahwah, NJ: Lawrence Erlbaum
interval recording. Journal of Applied Behavior Associates.
Anlaysis, 19, 73–77. Wacker, D., McMahon, C., Steege, M., Berg, W., Sasso,
Hartmann, D. P., & Hall, R. V. (1976). The changing crite- G., & Melloy, K. (1990). Applications of a sequential
rion design. Journal of Applied Behavior Analysis, 9, alternating treatments design. Journal of Applied
527–532. Behavior Analysis, 23, 333–339.
Kazdin, A. E. (1982). Single-case research designs: Wampold, B. E., & Worsham, N. L. (1986).
Methods for clinical and applied settings. New York: Randomization tests for multiple-baseline designs.
Oxford University Press. Behavioral Assessment, 8, 135–143.
Melchert-McKearnan, K., Deitz, J., Engel, J., & White, O. Watson, P. J., & Workman, E. A. (1981). The non-concur-
(2000). Children with burn injuries; Purposeful activity rent multiple baseline across-individuals design: An
versus rote exercise. The American Journal of extension of the traditional multiple baseline design.
Occupational Therapy, 54, 381–390. Journal of Behavior Therapy and Experimental
Onghena, P., & Edgington, E. S. (1994). Randomization Psychiatry, 12, 257–259.
tests for restricted alternating treatments designs. White, O. R. (1972). A manual for the calculation and use
Behaviour Research & Therapy, 32, 783–786. of the median slope – a technique of progress estima-
Ottenbacher, K. (1986). Evaluating clinical change: tion and prediction in the single case. Eugene, OR:
Strategies for occupational and physical therapists. University of Oregon, Regional Resource Center for
Baltimore: Williams & Wilkins. Handicapped Children.
Ottenbacher, K. (1990). Clinically relevant designs for Wolery, M., & Harris, S. (1982). Interpreting results of
rehabilitation research: The idiographic model. single-subject research designs. Physical Therapy, 62,
American Journal of Physical Medicine & 445–452.
Rehabilitation, 69, 286–292.
Ottenbacher, K., & York, J. (1984). Strategies for evaluat-
ing clinical change: Implications for practice and RESOURCES
research. The American Journal of Occupational
Therapy, 38, 647–659. Selecting Target Behaviors
Richards, S. B., Taylor, R. L., Ramasamy, R., & Richards, Kazdin, A. E. (1985). Selection of target behaviors: The
R. Y. (1999). Single subject research applications in relationship of the treatment focus to clinical dysfunc-
educational and clinical settings. San Diego: Singular. tion. Behavioral Assessment, 7, 33–47.
S E C T I O N 3
Quantitative Measurement
C H A P T E R 1 2
Developing and Evaluating Quantitative

Data Collection Instruments
Gary Kielhofner
Both everyday practice and research in occupa- coordinated than others). However, everyday pow-
tional therapy require the use of sound quantitativeers of observation are not very precise or reliable.
data collection instruments. These instruments can For example, when two persons have similar
include such things as: strength, it may be difficult to say who is stronger.
Moreover, if two different people are asked to
• Self-report forms,
judge who of a small group of individuals is the
• Interviews with rating scales,
most coordinated, they are likely to arrive at dif-
• Observational checklists or rating scales,
ferent judgments. This kind of imprecision and
• Calibrated measurement devices, and
inaccuracy of judgment is unacceptable in research
• Tests.
and clinical practice. Both situations require that
Chapter 32 includes a more detailed discussion occupational therapists make much more precise
of the various type of data collection procedures judgments than are possible through everyday
and instruments. The purpose of this chapter is to powers of observation.
examine the concepts and methods that underlie Quantitative instruments seek to achieve
the development of assessments and the criteria accuracy and consistency by translating infor-
that may be applied to examine the quality of a mation about some aspect of a person into
data collection assessment. numbers (Cronbach, 1990; Guilford, 1979). This
This chapter mainly process of measurement
discusses an approach to is classically defined as a
quantitative instrument This process of measure- rule-bound procedure by
development and analy- ment is classically defined which one assigns num-
sis referred to as classi- bers to variables in order
cal test theory (CTT) as a rule-bound procedure to quantify some charac-
(Hambleton, & Jones, by which one assigns num- teristic.1
1993; Nunally, 1978). All measurement req-
Chapter 13 discusses a bers to variables in order to uires a specific procedure
more recent and com- quantify some characteristic. and instrumentation that
plementary approach, allows quantification of
item response theory the characteristic of inter-
(IRT). Further, Chapter 14 discusses an additional est. For example, therapists measure muscle
set of considerations for choosing or developing an strength using instruments that quantify strength
instrument. as an amount of pressure generated or an amount
of weight lifted. Similarly, coordination can be
quantified by transforming the observed speed and
Quantifying Information accuracy of standardized task performance into a
score.
Many of the things that occupational therapists
1 As discussed in Chapter 13, item response theory
seek to measure can be directly observed and
approaches use a more restrictive definition of measure-
judged in a commonsense way. For example, one ment, but this chapter follows the more classic definition.
can readily recognize strength and coordination The IRT definition is referred to as objective measurement
(i.e., some persons are obviously stronger or more to differentiate these two approaches.
156 Section 3 Quantitative Measurement
Figure 12.1 Sound measures are essential to gather data on clients and research participants.
Two Key Elements of what strength is (e.g., muscularity or ability to

demonstrate strength in performance) and has
of Measurement some way to observe that idea (e.g., looking at
who appears more muscular or watching to see
Returning to the earlier example, the problems who lifts more, or wins an arm wrestling contest).
that occur in everyday judgments about who is These two elements tend to be implicit in
stronger or more coordinated are linked to the fact everyday judgments leading to the kinds of inac-
that: curacy or disagreement noted earlier. Conse-
quently, they are made explicit in measurement.
• Strength or coordination might mean different That is, underlying all measurement are two essen-
things to different persons, and tial steps:
• Each person may have a different procedure or
criteria for arriving at judgments about strength
• Definition of the construct to be measured, and
or coordination.
• Operationalizaiton of the construct through for-
For example, one person may think of strength mal instrumentation.
as an observable muscle mass. Another person
may think of strength in terms of some perform-
ance (e.g., winning an arm wrestling contest). In
Constructs and Their Operationalization
each instance, the person who seeks to make a All variables that are measured must first be
judgment about strength is working with an idea abstractly conceptualized as constructs. For exam-
Chapter 12 Developing and Evaluating Quantitative Data Collection Instruments 157
ple, consider the phenomena “movement.” If one so on) rely upon the development of unique new
wishes to measure some aspect of movement, one forms of measurement. For the most part it is these
must first conceptualize or define the specific char- latter forms of measurement that are discussed in
acteristic of movement that one wishes to measure. this chapter.
For example, the measurement of movement could
involve such things as: Scales of Measurement
• Freedom of movement, The rules of measurement reflect the basic scales
• Speed of movement, and of measurement. That is, numbers may be used to:
• Efficiency of movement.
• Differentiate one characteristic from another,
Once one has chosen an element of movement • Indicate order from less to more of a characteris-
for measurement (e.g., freedom of movement) then tic,
one must clearly define or conceptualize what will • Indicate an amount of a characteristic on a con-
be measured. The most common measure of the tinuum from less to more, and
freedom of movement is joint range of motion, • Indicate an absolute amount of a characteristic.
which is defined as the movement about the axis
These purposes correspond to the nominal,
of a joint. Joint range is further specified as active
ordinal, interval, and ratio level scales. Each of
and passive. Active range of motion refers to
these scales of measurement has a specific pur-
the range of movement about the axis of a joint
pose, meaning of the number, and rules that gov-
that a person can produce using his or her own
ern how numbers are assigned and how they can be
strength.
mathematically manipulated as summarized in
Once a very specific definition of a construct is
Table 12.1.
developed, then a procedure and/or instrument for
operationalizing that construct can be developed.
In the case of active range of motion, the proce- Nominal Scales
dure involves asking a client or research partici- Nominal scales are used to classify characteristics
pant to move the part of the body of interest and such as sex. In this case numbers are used to iden-
then to apply an instrument that operationalizes the tify a specific category (e.g., 1 ⫽ female; 2 ⫽
movement and represents it in terms of numbers. male). In nominal scales numbers have no other
In the case of a goniometer, movement about the meaning than identifying the category or charac-
axis of a joint is operationalized as degrees of a teristic to which a person belongs. The basic rule
circle (i.e., 0 to 360 degrees). underlying nominal scales is that each category
If one wished to measure another aspect of must be exclusive of the other categories. The only
movement, such as efficiency of movement, then a mathematical operation that is allowed with nomi-
specific construct would need to be defined and an nal scales is that they can be counted. Thus,
appropriate method of operationalizing that con- research variables based on nominal characteris-
struct developed. Consequently, all measurement tics are usually represented as frequency counts
begins with identifying the construct that one and proportions derived from those counts (e.g., 20
intends to measure and proceeds to a specific males and 80 females ⫽ 1/5 male).
method of operationalizing that construct.
Ordinal Scales
The Rule-Bound Nature Ordinal scales are used to classify ranked cate-
gories. A typical example of ranking in occupa-
of Measurement tional therapy is degree of dependence and
independence (e.g., 1 ⫽ totally independent, 2 ⫽
Measurement is rule bound in that there are spe- needs minimal assistance, 3 ⫽ needs moderate
cific rules or laws that govern how numbers can be assistance, 4 ⫽ needs maximal assistance, 5 ⫽
used to stand for some quality of the construct that totally dependent). In ordinal scales, the number
is being measured. Physical measurements of refers to a rank order. Using the example above, 2
human traits (e.g., height, weight, strength, and is the second most independent rating.
range of motion) build upon physical measures Importantly, in ordinal scales the intervals
that have been developed for characterizing a between ranks are not necessarily the same. That
whole range of objects. Other human traits (e.g., is, the difference between “totally independent”
abilities, attitudes, aspects of the personality, and and “needs minimal assistance” may not be the
Table 12.1 Scales of Measurement

Type of Purpose Meaning Requirement Possible Mathematical Examples
Scale of Scale of Numbers Manipulation
Nominal Classification Identify a category Mutual Counting (i.e., Sex, ethnicity/
exclusivity compilation of race, religion,
frequencies) diagnosis
Ordinal Ranking (i.e., Indicate rank order Mutual Strictly speaking, Degree of
position within exclusivity/ same as independ-
a distribution Ordinality nominal. In ence, grade
categories) practice, often of muscle
used as if they strength (i.e.,
are interval good, fair,
poor)
Interval Represents Indicate position Mutual Can be added and Pounds of
continuum of a on a continuum exclusivity, subtracted pressure
characteristic partitioned into ordinality, generated as
using equal- equal unit and equiv- a measure of
interval units intervals alency of strength
units
Ratio Indicates amount Indicate absolute Mutual All mathematical Height, weight
amount (zero exclusivity, and statistical
equals total ordinality, operations
absence of equivalency
characteristic of units, and
measured) absolute
zero point
same as the distance between “needs moderate traits, ordinal scales are the most commonly used
assistance” and “needs maximal assistance.” This scales.
means that the numbers used in ordinal scales do
not represent an amount. Rather they are categori-
Interval Scales
cal labels that indicate ranking within a distribu-
tion of categories. Interval scales demonstrate equal distances (i.e.,
Strictly speaking, then, ordinal scales are intervals) between the units of measurement.
descriptive scales like categorical scales and the Interval scales represent the continuum of a char-
numbers used are not true quantities. Thus, mathe- acteristic (from less to more) using equal interval
matical operations to which they can be subjected units. They allow the investigator or practitioners
in the strictest sense are the same as nominal vari- to determine relative difference. For example, on
ables. While it is common to calculate such num- an interval score the difference between 2 and 3 is
bers as an average rank or a change score that the same as the difference between 3 and 4, 4 and
involves mathematical operations (i.e., addition, 5, and so on.
subtraction, multiplication, and division), the Interval scales do not indicate absolute amount
resulting numbers are not “meaningful as true or magnitude of a characteristic because they do
quantities” (Portney & Watkins, 2000, p. 55). not have a true zero point indicating the absence of
Thus, when ordinal scores are subjected to these any of the characteristics. It should be noted that
mathematical operations they are treated “as if” while some interval scales do have a zero point
they had the properties of interval scales. This (along with plus and minus values), these are arbi-
common practice is considered controversial by trarily zero points without meaning. A true zero
some researchers for reasons discussed in Chapter point must represent the absence of the character-
13. To an extent the widespread treatment of istic being measured.
ordinal data as if it were interval data reflects Importantly, interval scales are additive since
the fact that in occupational therapy, like other dis- the intervals between numbers are the same. Thus,
ciplines that seek to measure a range of human total scores (addition) and change scores (subtrac-
tion) can be calculated. Because interval scales can When the error term is known, the true score
be subjected to a number of mathematical opera- is defined as the observed score minus the error
tions without violating the underlying rules, they score or:
are preferable to ordinal scales. This is why there T⫽X–E
is increasing interest in the IRT methods discussed
Thus T is the score that will be assigned to an
in Chapter 13 that convert ordinal scale data into
individual or that an individual would achieve on
interval scales.
an instrument if the instrument were error-free.
Ratio Scales Types of Measurement Error

Ratio scales demonstrate equal distances between There are two types of errors that can occur when
units of measure and they also have an absolute applying instruments:
zero point. Therefore, they indicate absolute • Systematic error, and
amounts of the characteristics measured. Unlike • Random error.
interval scales, numbers from ratio level scales can
be interpreted as ratios; for example, someone who Systematic errors are consistent or predictable
is 6 feet tall can be said to be twice as tall as some- errors; they occur when an instrument misesti-
one who is 3 feet tall. Moreover, all forms of math- mates the true score by a consistent amount and in
ematical and statistical operations are permissible the same direction (too low or too high).2 Random
with ratio scales. An example of ratio scales used error occurs by chance and is, thus, unpredictable.
in occupational therapy is strength measures (e.g., Typical sources of error on a measurement
a dynamometer) where the zero point represents a instrument include:
total lack of strength (i.e., inability to generate any • The instrument itself,
pressure). • The individual who is administering the instru-
ment (i.e., rater or tester error), and
• Fluctuations in the characteristic measured.
Measurement Error A variety of instrument factors contribute to
measurement error. These include such things as:
Returning again to the example of estimating
strength and coordination, the underlying problem • Problems in the conceptualization of the con-
was inaccuracy of judgment. The aim of measure- struct being measured and how it is translated
ment is to achieve the most accurate judgment into an instrument,
possible. Theoretically, there is always some error • Lack of precision that requires the person admin-
present in measurement, but measurement seeks to istering or taking the instrument to estimate or
minimize the amount of measurement error. To guess,
think about and minimize error, classical test • Complexity or lack of clarity in how the instru-
theory uses two concepts. The first concept is ment is to be used, and
true score, which refers to the actual quality or • Ambiguity that leads to differential interpretation
amount of the underlying characteristic measured of items or tests that are part of the instrument.
that a person has. The second concept is the
observed score, which refers to the number that Strategies for Reducing
the observer assigns to the individual using an
instrument. Measurement Error
In classical test theory, the observed score is
considered to be a function of two factors: There are a number of ways that developers and
users of instruments seek to reduce measurement
• The true score, and error. The most common are:
• The error of measurement (Hambleton & Jones, • Standardization of the instrument,
1993).
2Classical test theory ordinarily conceptualizes systematic
This relationship is represented by the equa- error as a problem of validity (Portney & Watkins, 2000).
tion: However, one of the most common sources of systematic
error is inter-rater disagreement due to differences in sever-
X⫽T⫹E ity/leniency. This form of systematic error has been largely
overlooked because the statistical methods used to examine
where X ⫽ bserved score; T ⫽ true score; E ⫽ reliability do not account for systematic covariation coupled
error. with differences in severity/leniency.
• Methods of informing, training, and ensuring • Knows the administration protocol, and
accuracy of raters, and • Understands the content of the instrument.
• Taking repeated measures.
Standardization of instruments helps to reduce
variability in how raters administer a test. Other
Standardization approaches to minimizing rater or tester error are:
One of the most important and effective ways to • Providing detailed instrument instructions or an
reduce the measurement error of an instrument is administration manual,
to standardize it. Standardization refers to specify- • Training, and
ing a process or protocol for administering the • Credentialing those who will administer an
assessment. Instruments may involve varying instrument (credentialing ordinarily involves
degrees of standardization, depending on the both formal training and some kind of practical
nature of the instrument. The following are exam- test or other demonstration that the person is
ples. Tests such as the Minnesota Rate of competent to administer the instrument).
Manipulation Test (Lafayette Instruments, 1969)
or the Sensory Integration and Praxis Tests (Ayres, Repeated Measurement
1989) require a specific administration protocol In instances in which the characteristic that the
along with a standard test kit. Observational instru- instrument seeks to measure tends to fluctuate,
ments such as the Assessment of Motor and testers typically take multiple measurements in
Process Skills (Fisher, 1993) may require stan- order to note the range and central tendency of the
dardized situations while allowing a certain variability. In this way the practitioner or investi-
amount of discretion on the part of the admin- gator can avoid using only an extreme or unusual
istrator. Semistructured clinical interviews such as score and thus misestimating the true or more
the Occupational Circumstances Interview and usual value of the characteristic being measured.
Rating Scales (Forsyth et al., 2005) may allow sub- Regression toward the mean (the tendency for
stantial flexibility in how the interview is con- extreme scores to be followed with scores that are
ducted but require the therapist to complete a more average) makes taking repeated measures a
standardized rating scale. Self-administered instru- good strategy for reducing error. Use of this strat-
ments such as the Child Occupational Self egy depends, of course, on how feasible it is to
Assessment (Keller, Kafkes, Basu, Federico, & take multiple measures and on how much the fluc-
Kielhofner, 2005) rely on the structure of the tuation affects the precision of the measure.
paper-and-pencil form and clear instructions and
guidance on the part of the therapist administering
the assessment. Instrument Reliability
In each of these instances, the developer of the
instrument considered and often completed many Reliability refers to the property of consistency in
trials to determine what procedures would opti- a measure. Implied in the definition of reliability is
mize gathering comprehensive and stable informa- that any difference in the score obtained (e.g., from
tion. In the case of testing sensory–motor abilities, time to time or from different individuals) should
a set of specific highly structured motor tasks was be due to true differences in the underlying char-
considered optimal. In the case of doing an inter- acteristic and not due to error. Reliability, then,
view, flexibility to respond to clients and make reflects the extent to which an instrument is free
them feel comfortable was considered optimal for from sources of error (Cronbach, 1990)
gathering the personal information for which the
interview asks. The Reliability Coefficient
Reliability is expressed as a ratio of the variance of
Reducing Rater or Tester Error the true score to the total variance observed on an
instrument, or:
Rater or tester error is ordinarily caused by such
factors as mistakes, guessing, or variability in test T
ᎏ
administration or circumstances. This source of T⫹E
error is minimized by ensuring that the person
using the instrument: where T ⫽ true score variance and E ⫽ error
variance.
• Understands the construct the instrument is The value of this ratio is referred to as the reli-
designed to measure, ability coefficient. It ranges from 0.0 to 1.0; 1.0,
which indicates there is no error. The larger the vidual receives on the second administration. For
error, the more the reliability coefficient will devi- this reason, investigators generally want to include
ate from a perfect coefficient of 1.0. a period that is:
The reliability coefficient is interpreted as the
• Long enough to erase the effects of memory or
“proportion of observed variance that is attributed
practice, yet
to true score variance” (Bensen & Schell, 1997, p.
• Not so long as to result in a genuine change in the
5). Thus, for example, a reliability coefficient of
underlying characteristic that will be confounded
.90 is considered to be an estimate that 90% of the
with error.
variance observed can be attributed to variance in
the characteristic measured (true score) as opposed Moreover, when reporting findings on test–
to error variance. When the reliability of assess- retest reliability, investigators should indicate both
ments is being investigated, reliability coefficients the time interval between administrations and any
are calculated as correlation coefficients, which rationale for whether the underlying trait is
are discussed in the following sections and expected to change during that period.
explained in detail in Chapter 17. Test–retest reliability correlations are calcu-
lated based on the two administrations of the
instrument; the time 1 score is the first variable and
Empirical Evaluation of the time 2 score is the second variable. The
Pearson Product Moment correlation is typically
Instrument Reliability used for test–retest reliability. Generally, correla-
tions (r-values) above .60 for longer time intervals,
When instruments are being developed or evalu- and higher values for shorter intervals, are consid-
ated for particular use, their reliability is empiri- ered evidence of reliability. In the end, interpreta-
cally investigated. In the context of classical test tion of the statistic should be based on theoretical
theory, reliability of an instrument is empirically expectations. If there is no reason to suspect that
assessed using the following methods: the underlying characteristic changed, then a
• Test–retest or stability, higher correlation will be expected.
• Split-half reliability, The following is an example of test–retest reli-
• Alternate forms reliability, and ability. Doble, Fisk, Lewis, and Rockwood (1999)
• Internal consistency. examined the test–retest reliability of the
Assessment of Motor and Process Skills (AMPS)
Test–Retest or Stability (Fisher, 1993). They administered the AMPS to a
sample of 55 elderly adults and then reassessed
One of the most common ways of determining them within 1 to 10 days, calculating Pearson
whether an instrument provides consistent results Product Moment Correlations. The two adminis-
is to administer the same instrument on two differ- trations of the AMPS were highly correlated (i.e.,
ent occasions. When readministering an instru- Motor r ⫽ .88 and Process r ⫽ .86), providing evi-
ment to assess reliability, there is no empirical way dence of good test–retest reliability.
to separate differences that are due to:
• Changes in the underlying trait, and Split-half Reliability
• Changes that are due to error.
Split-half reliability is a technique most often used
Therefore, consideration has to be given to how when testing reliability of questionnaires. It is pre-
likely it is that the underlying trait will change or ferred since the alternative way to test reliability is
has changed in the period between administra- to readminister the entire questionnaire. If it is
tions. This is a consideration in both choosing the readministered too soon, reliability may be overes-
period of time between administrations and in timated because of memory (i.e., the respondents
interpreting the statistical results. fill it out based on how they recall having filled it
Choosing the period for readministration also out before). If it is readministered too far apart,
requires consideration of possible effects of the then the underlying characteristic may have chan-
first administration on subsequent administration. ged, leading to an underestimation of reliability
For example, memory can inflate agreement if the since true score change is confounded with error.
subject recalls and repeats the responses given on To avoid these problems, investigators divide
an instrument when taking the instrument a second the items into two smaller questionnaires (usually
time. In tests of ability, the practice effect of taking by dividing it into odd and even items, or first
the test the first time may inflate the score the indi- half–last half) and then correlating the scores
obtained from the two halves of the instrument. In It should be noted that internal consistency and
this case, the Spearman–Brown prophecy statistic construct validity, which is discussed later, are
is typically used. The correlation (r) should be .80 closely related. However, internal consistency is
or higher. considered an issue of reliability because if many
items measure the underlying construct and some
Alternate Forms or Equivalency Reliability do not, the latter will be adding error to the instru-
ment. Consequently, internal consistency is gener-
For some instruments, it is important to be able to ally taken as evidence of both reliability and
administer different versions of the instrument. For validity.
example, national certification tests or aptitude Internal consistency is typically assessed using
tests use different combinations of items to avoid Cronbach’s coefficient alpha (␣) (Cronbach,
cheating and also to ensure that the item pool 1951). Alpha is the average of all split-half relia-
reflects contemporary material (Benson & Schell, bilities for the items that make up the instrument.
1997). In other cases, investigators are concerned It can be used with both dichotomous and ordinal
with different administration formats of an instru- scales. Like other correlations, alpha ranges from
ment (e.g., an instrument that can be administered 0.0 to 1.0 and the larger the alpha coefficient, the
as a paper-and-pencil checklist or a card sort pro- stronger the inter-correlation among items and,
cedure). In these instances, it is important that dif- therefore, homogeneity of the scale as a whole.
ferent versions, or forms, of an instrument provide Generally, alpha values that approach .90 are indi-
consistent results. cations of high homogeneity. Since alpha is
Alternate forms reliability involves administra- affected by the number of items, longer scales will
tion of the alternative forms to subjects at the same tend to generate higher coefficients. While alpha
time. In order to avoid the effects of sequence gives an indication of overall consistency, it does
(e.g., fatigue), the order in which the forms are not provide information about which items may be
administered may be counterbalanced (i.e., half inconsistent and, thereby, contributing error to the
the subjects take one instrument first and the other instrument.
half take the other instrument first), or the items Another approach to examining internal consis-
that make up the two forms may be integrated ran- tency or homogeneity is item-to-total correlations.
domly or in alternating sequence in a single test. In this method, each item is correlated to the total
Alternate forms reliability is assessed using the test score. Pearson Product Moment correlations
Pearson Product Moment Correlation. It is gener- are used unless items are dichotomous in which
ally accepted that the correlation (r) should be .80 case a point-biserial correlation coefficient is used.
or higher. Generally, authors suggest that item-total correla-
tions should yield correlations between .70 and .90
Internal Consistency (Streiner & Normal, 1995). The advantage of item-
total correlations over alpha is that they allow an
Investigators often examine whether the items that instrument developer to identify individual items
make up an instrument covary or correlate with that may be inconsistent with the total score and,
each other. This property is often referred to as thereby, contributing error to the instrument.
homogeneity.3 This is tested by asking whether the
items covary. For example, if the items on a scale
of cognition all reliably measure cognition, then a Evaluating Instrument Reliability
person with low cognition would tend to score The choice of which approach(s) to use for evalu-
lower on all the items and a person with high cog- ating reliability depends on what sources of meas-
nition would tend to score higher on the items. If urement error are relevant to the measurement
this is the case across a sample, then the items will instrument. For example, if an instrument targets a
demonstrate consistent variance. characteristic that is somewhat variable from time
3Although
to time (e.g., mood state or fatigue), test-retest is
internal consistency is generally discussed as a not a very good estimate of stability since there
measure of reliability, it overlaps with validity. That is, the
homogeneity of items indicates the items converge or work is no way to empirically sort out what variability is
together to measure the same underlying construct. In item due to error versus fluctuation in the underlying
response theory, this property is known as unidimensional- trait. In this case, split-half reliability is a more rel-
ity (see Chapter 13). In contrast to simply asking whether evant approach. On the other hand, if an instru-
the items covary, this method asks instead whether the
observed scores of items fit with expectations based on the
ment measures a characteristic that is relatively
item’s placement on the continuum of the characteristic stable, then test–retest is a relevant form of relia-
being measured. bility to examine.
In cases in which different items or administra- race, age) may impact how a person behaves. For
tion formats are used in versions of an instrument example, in responding to an interview, clients
(e.g., the national certification examination taken who perceive the interviewer to be more able to
by occupational therapists) the approach of choice understand their situation may give more complete
is equivalency or parallel forms. Thus, in develop- information or note problems that would be with-
ing or assessing evidence about reliability of an held from another interviewer who is not perceived
instrument, consideration needs to be given to to be as capable of understanding. Moreover, a
what the instrument measures and how the instru- client may seek to create a particular impression
ment is intended to be used. on the part of the interviewer based on perceived
It is important to recognize that reliability characteristics of the interviewer.
assessment based on classical test theory is sample
dependent (Hambleton & Jones, 1993). This Rater Bias
means that the obtained reliability coefficient will
Rater bias may occur when the rater translates the
differ from sample to sample, largely because of
information obtained into a classification or rating.
differences in the variability of different samples
In this situation, any number of characteristics of
drawn from the same population. For this reason,
the rater may introduce error. Rater characteristics
Benson and Schell (1997) recommend that when-
that might result in error include demographic
ever an instrument is used within a study, reliabil-
characteristics, experience, training, or theoretical
ity evidence should be reported for that sample. In
orientation of the rater. Rater demographics might
practice, it is often the case that, if an instrument is
result in error, for example, when the rater shares
used with a group for which there has previously
characteristics with the person rated and overem-
been reported reliability data, investigators will
pathizes or judges the person too harshly or with
make reference to this previous research, but not
too much leniency because of personal experience.
reevaluate reliability in the context of the study.
Differences in experience, training, and theoretical
However, it is common practice and highly desir-
orientation may result in raters bringing different
able to report reliability findings when an investi-
perspectives or understandings to the instrument,
gation uses an instrument with a population that
thereby introducing error.
differs from previous research or when some
aspect of the administration varies from previously
reported studies.
Assessing Inter-rater Reliability
The extent of rater bias is assessed through inves-
Rater/Observer Effects on Reliability tigations of inter-rater reliability. Inter-rater relia-
bility is typically studied by having two or more
In occupational therapy many types of instruments raters observe the same clients, either directly or
require the administrator to complete a checklist, through videotape or audiotape. The inter-rater
form, or rating scale based on information gath- reliability coefficient that is used depends on the
ered about a client based on observation, testing, nature of rating that is done.
or interview. As noted earlier, the rater or observer For example, when raters classify clients on
can be a source of error and certain strategies can characteristics, investigators sometimes calculate
be used to reduce rater error. Rater or observer percent agreement. While percent agreement can
error is also assessed empirically. For this purpose, provide some information about rater agreement, it
there are two sources of observer/rater error that tends to inflate agreement when fewer categories
are typically examined: are used. For example, when there are only two
• The biasing effects of observer presence or categories, raters will agree 50% of the time just
observer characteristics, and by chance. Thus, 75% agreement represents only a
• Rater bias. 25% agreement above chance. For this reason, a
more accurate estimate of agreement takes chance
Observer Presence or Characteristics agreement into consideration. Thus investigators
use the kappa statistic (Cohen, 1960), which cor-
In many instances of observation, interview, or rects for chance. When there is concern about
testing, the rater’s presence (and characteristics) magnitude of disagreement a weighted kappa sta-
may have in impact on the behavior or information tistic is used, since kappa assumes all disagree-
provided by the client. For instance, clients who ments are of equal weight or importance. For
are aware of being observed may alter their behav- example, if raters were classifying client mood
ior to influence the observer’s conclusion. In addi- states as depressed, anxious, happy, or tranquil,
tion, the characteristics of an observer (e.g., sex, there would be more concern about two raters who
judged a client to be happy and depressed, respec- An important limitation of correlation coeffi-
tively than if they rated the client depressed and cients in estimating rater agreement is systematic
anxious, respectively. The latter is a smaller disagreement, which occurs when there are differ-
amount of disagreement than the former. In this ences in rater severity or leniency. For example, on
instance a weighted kappa (Cohen, 1968) can be a 10-point rating scale, two raters may consistently
used to consider the extent of disagreements; it give different ratings that covary as shown in Table
would distinguish between the two types of dis- 12.2. While the two raters are always in disagree-
agreements given in the example above. ment about the appropriate rating, their disagree-
The kappa statistic is influenced by sample ment is systematic (i.e., rater 2 always gives a
size, subject variability, and the number of cate- higher score that is 3 points higher than rater 1’s
gories used, so it must be interpreted with care. score). For this reason, investigators increasingly
Fleiss (1981) provides the following guidelines for estimate inter-rater reliability with more powerful
interpretation of kappa values: statistics than these traditional correlations.
• ⬎ 0.75 ⫽ excellent agreement,
Generalizabilty Theory and
• 0.40 to 0.75 ⫽ fair to good agreement, and
• ⬍0.40 ⫽ poor agreement. Intraclass Correlations
The following is an example of the use of the Alternatives to traditional correlational approaches
kappa statistic. Clemson, Fitzgerald, Heard, and to estimating inter-rater reliability include Item
Cumming (1999) examined the inter-rater reliabil- Response Theory (discussed in Chapter 13) and
ity of the Westmead Home Safety Assessment the variance components approach of generaliz-
(WeHSA), a checklist of categories of potential ability theory (Benson & Schell, 1997). The latter
fall hazards. Since this is a dichotomous assess- approach allows for the estimation of multiple
ment (i.e., items are assessed as to whether or not sources of error in addition to the random error
they present a potential falls hazard), kappa is an considered in classical test theory. Moreover, clas-
appropriate statistic for assessing inter-rater relia- sical approaches to assessing reliability estimate
bility. reliability only by examining one source of error at
A previous study had shown inadequate relia- a time, whereas the variance components approach
bility for a third of the items. Therefore the instru- estimates reliability while accounting for several
ment was revised to clarify items and a manual and different sources of error simultaneously. Thus,
a training program was developed. In this study, this method can be used to calculate the reliability
pairs of therapists, who were trained in the use of of an observed score in estimating the true score by
the assessment, completed normally scheduled partitioning error due to several factors (referred to
home visits during which one therapist adminis- as facets) such as variations in the raters and test-
tered, the other observed, and both independently ing conditions, alternate forms, and administration
scored the WeHSA. Based on evaluations of 21 at different times. This approach uses Analysis of
homes of clients who were referred for home mod- Variance (ANOVA) (discussed in Chapter 17) to
ification assessment, falls risk management and estimate sources of variation and their interactions.
other functional reasons, kappa statistics were cal- A reliability coefficient, called the intraclass corre-
culated. The investigator reported that: lation coefficient (ICC), can be computed. Unlike
• 52 of the items received kappa values greater

than 0.75, Table 12.2 A Hypothetical Illustration
• 48 of the items received kappa values between of Systematic Disagreement Due to
0.40 and 0.75, and
Differences in Rater Severity/Leniency
• None of the items received kappa values lower
than 0.40. Observation Rater 1 Rater 2
The results of this study indicated that the 1 2 5
instrument’s inter-rater reliability had improved 2 3 6
and met recommended criteria for reliability. 3 5 8
When ordinal scales are used, the most appro-
4 1 4
priate measure of agreement is a nonparametric
correlation coefficient, the Spearman rho. For 5 6 9
interval and ratio scales the parametric, Pearson 6 4 7
Product Moment Correlation is typically used.
other reliability estimates it is not sample depend- How one goes about generating or assessing the
ent; therefore the components approach is also evidence of validity depends on both the underly-
referred to as generalizability theory. Another ing trait is seeks to measure and the intended use
important value of this statistic is that it reflects the of the assessment. Generally, the following are
extent of agreement between raters including sys- indices that are used to develop and assess validity
tematic disagreement which is not reflected in of an instrument:
other estimates of reliability (Ottenbacher &
• Face validity,
Tomchek, 1993).
• Content validity,
• Criterion validity, and
• Construct validity.
Validity
Each is discussed and illustrated below.
Validity means that measure derived from an
instrument represents the underlying construct that Face Validity
the instrument is designed to measure. Strictly
speaking, an instrument is never validated. Rather, Face validity means that an instrument has the
investigators seek to validate an interpretation of appearance of measuring an underlying construct.
the scores the instrument yields (Benson & Schell, For example, if an assessment is designed to meas-
1997; Cronbach, 1971; Nunally, 1978). This dis- ure an attitude about leisure (i.e., how important
tinction underscores the fact that all measurement leisure is), and the items all are made up of state-
instruments are used to make inferences about a ments about leisure, the instrument can be said to
specific characteristic of a person (or group in the have face validity. Face validity is, however, the
case of community or group assessments). It is the weakest evidence of validity. For example, con-
validity of that inference that is ultimately of consider the following statements about leisure:
cern. Thus, when an instrument is said to have
• I always try to make time for leisure activities.
validity, it means that the interpretation of the
• I often feel there is not enough time to do the
measurement that is made with the instrument is
things I enjoy.
correct in its meaning.
• Doing leisure activities helps one achieve relax-
As noted earlier, all instruments are designed to
ation and refreshment.
measure some abstract characteristic or construct.
• Engaging in leisure always enhances my mood.
A construct is a theoretical creation and so it is
important to demonstrate the usefulness of the On the face of it, the items all ask about leisure
construct for explanation and for practice (e.g., and, arguably, reflect how much a person values
making sense of a client’s behavior, identifying a leisure. However, the second item may reflect
client’s problems or strengths, predicting future more about how much a person works or fulfills
functioning). Inevitably, concerns for validity also other non-leisure obligations than how much
interrelate with the intended use and demonstrated leisure is important to that person. Similarly, the
utility of the results last item may reflect
obtained from an instru- whether a person is
ment. The validity of an instrument depressed instead of
Validity is not an all- how much leisure is val-
or-nothing property of an is demonstrated by the ued.
instrument, but rather a accumulation of several Consequently, face
matter of degree (Benson validity alone is insuffi-
& Schell, 1997). The
types of evidence produced cient to demonstrate the
validity of an instrument over many studies. validity of an instru-
is demonstrated by the ment. However, it is
accumulation of several often a good place to
types of evidence produced over many studies. start when one is trying to generate or create items
An instrument should be judged by a body of to make up an instrument. Moreover, face validity
evidence that provides or fails to provide sup- can be helpful in deciding whether or not to use an
port for its validity. Moreover, ongoing research instrument. If the items that make up an instrument
should continue to provide evidence about the are not, on the face of it, relevant or meaningful to
validity of an instrument long after it is published what one aims to measure or to the intended audi-
and in use. ence, then the instrument is not likely to be valid
for that purpose. Finally, in the absence of any prehensiveness and focus. In a third round they
other evidence about the validity of an instrument, may be asked to generate or evaluate specific items
face validity is the minimum criterion that one that reflect the content.
should apply when deciding whether to use an The following is an example of how one group
instrument. However, since there is no formal way of instrument developers approached the issue of
of evaluating face validity, it is ultimately an infor- content validity. In developing the Volitional
mal process. Consequently, two experts may Questionnaire (de las Heras, Geist, Kielhofner, &
review the same instrument and one might say it Li, 2003), investigators first identified the broad
has face validity and the other may not. construct of volition that includes a person’s
thoughts and feelings about personal causation
Content Validity (effectiveness and capacity), interests, and values.
Moreover, included in the definition of the con-
Content validity refers to the adequacy with which struct was that volition was manifested across
an instrument captures the domain or universe of a continuum of exploratory, competency, and
the construct it aims to measure (Nunally, 1978). achievement motivation. This instrument was
The constructs that instruments are intended to designed to capture volition as it is manifested in
measure inevitably include a range of content. For behavior. Thus, to operationalize the construct of
example, self-care includes such content as brush- volition the authors had to specify ways that a per-
ing teeth, bathing, and dressing. When developing son demonstrated a sense of capacity and efficacy,
or judging an instrument, one must ask whether the interest, and value or meaningfulness in action
universe of content represented by the underlying across the continuum from exploration to compe-
construct is adequately reflected in the instrument. tency to achievement. Items were generated based
In addition to the concern about whether all the on clinical experience and using feedback from a
relevant content is included in an instrument, con- panel of experts who used the concept of volition
tent validity is also concerned that irrelevant in practice (Chern, Kielhofner, de las Heras, &
content be excluded from the instrument. So, for Magalhaes, 1996).
example, an assessment of self-care should not The resulting items that were designed to cap-
have items that reflect socializing with friends or ture the universe of volition are shown in Table
performance at work, since these occupations are 12.3. Later research, using Item Response Theory,
not part of the occupation of self-care. identified that the items all belonged to the con-
Content validation requires one to conceptually struct of volition and represented a continuum
define the domain that is being measured and spec- from exploration to competency (Li & Kielhofner,
ify how this domain is to be operationally defined 2004).
(i.e., making concrete the elements of the concep- One important consideration in identifying the
tual definition) (Benson & Schell, 1997). Only universe or domain of a construct is how broadly it
then can one determine that the items that make up is defined since it affects the degree of inference
an instrument adequately represent the universe of that someone using the instrument must make.
the construct. Borg and Gall (1983) note that a low-inference
Assessing content validity often begins with construct is one that is readily or easily observed
developing a set of specifications about what and that requires only limited judgment on the part
domains make up the construct of interest. This of the observer. In contrast, a high-inference item
can be done by: may involve a series of events or behaviors and/or
one that requires the observer to assemble different
• Reviewing relevant literature,
aspects of a client’s behavior into a judgment.
• Reviewing existing instruments that target the
High-inference items tend to produce less reliable
construct to see what content is included, and
observation. Therefore, low- inference items have
• Seeking the opinions of an expert panel (i.e., a
the virtue of making observational scales more
group of individuals who have in-depth knowl-
reliable. This is often the case when the trait of
edge or expertise concerning the domain of
interest is relatively concrete.
interest).
However, in occupational therapy there are
Sometimes expert panels are consulted in sev- variables of interest that are more difficult to trans-
eral rounds as an instrument is being developed. late into low-inference items. The Volitional
For example, in the first round members of the Questionnaire, which was described earlier, is an
panel may be asked to brainstorm content that example of this situation. The developers of this
should be included. In the second round they may instrument wanted to measure volition from
be asked to examine a list of content for its com- observing the behavior of individuals who could
Table 12.3 Items that Make Up the Finally, when developing a questionnaire or
Volitional Questionnaire self-report, it is often important to identify the
domain from the perspective of persons who will
ACHIEVEMENT
be responding to the instrument. For example, if
Seeks challenges. the intention of a self-report assessment is to cap-
Seeks additional
ture environmental barriers to disabled persons’
responsibilities.
Invests additional
independent living, content validity would require
energy/emotion/attention. that it include all the things that persons with dis-
Pursues an activity abilities encounter as barriers. In this case, using
to completion/ focus groups of persons with disabilities to gener-
accomplishment. ate ideas about and to evaluate the content of such
an instrument would be advisable.
COMPETENCY
Tries to solve problems.
Shows pride. Criterion Validity
Tries to correct
mistakes/failures. Unlike face and content validity, criterion validity
Indicates goals. involves collecting objective evidence about the
Stays engaged. validity of an assessment. Criterion validity refers
to the ability of an instrument to produce results
EXPLORATION
that concur with or predict a known criterion
Shows that an activity is special
instrument or known variable. As the definition
or significant.
Tries new things.
implies, criterion validity includes two types of
Initiates actions/tasks. evidence:
Shows curiosity.
• Concurrent validity, and
Shows preferences.
• Predictive validity.
When assessing criterion validity, it is impor-
not self-report their volitional thoughts and feel- tant to select a criterion instrument that is recog-
ings. Making inferences about motivation based on nized and demonstrated to have good reliability
behavior requires a level of abstraction and judg- and validity. Often such an instrument is referred
ment. Consequently, the instrument’s developers to as the “gold standard” instrument (i.e., an
had to create a detailed manual with definitions and instrument that is widely recognized and empiri-
examples of each item. Moreover, it is important cally demonstrated to be a reliable and valid meas-
that the person using the Volitional Questionnaire ure of the intended construct). Benson and Schell
have a theoretical understanding of volition, the (1997) also recommend and provide a formula for
construct that the instrument seeks to measure. estimating the upper bound (highest possible
As this example illustrates, it is important in value) of a validity coefficient. If it is too low, then
high-inference instruments that the observer/rater they recommend improving one of the two meas-
have adequate background and understanding of ures or selecting a different criterion measure.
the intended construct and how it is operational- They also provide a related formula for estimating
ized. Sometimes, this need can be addressed how high a validity coefficient would be if the two
though a detailed user’s manual. Moreover, in measures were perfectly correlated; if the value of
some instances, training in how to use the instru- this estimate is too low, then they recommend
ment is also desirable or necessary (Benson & choosing a different criterion measure.
Schell, 1997). It also seems that in these types of
instruments, the user must have both a solid theo- Concurrent Validity
retical background and commitment to applying
theory in practice. Concurrent validity refers to evidence that the
Assessing content validity is inherently a sub- instrument under development or investigation
jective and conceptual process that involves con- concurs or covaries with the result of another
sideration of the various domains that make up a instrument that is known to measure the intended
construct. Sometimes these domains are specified construct or with another criterion. Concurrent
by consensus, as in the case of self-care. Other validity is often the method of choice when there
times, the domains must be identified from a par- is an existing “gold standard” instrument. One
ticular theoretical perspective, as in the earlier may ask, if such an instrument exists, why develop
examples of the Volitional Questionnaire a new assessment? There may be a variety of
reasons. For example, the existing instrument(s) construct. In reality, construct validity is the ulti-
may be too lengthy or costly to administer regu- mate objective of all forms empirically assessing
larly. Moreover, the existing assessments may validity, but the process of empirically assessing
demand capabilities for participation that the construct validity involves a series of studies that
intended clients do not possess. Finally, the exist- provide cumulative evidence. Construct validity is
ing instruments may simply not be practical for ultimately concerned with the underlying con-
use in the situation for which the new instrument is struct that the instrument targets. Construct valid-
intended. ity is crucial when the “interpretation to be made
Concurrent validity is assessed by administer- of the scores implies an explanation of the behav-
ing the new assessment that is under development ior or trait” (Bensen & Schell, 1997, p. 11).
or investigation at the same time as the criterion As noted earlier, the idea of a construct is a the-
instrument or variable and then calculating a cor- oretical conceptualization. As such, it is tied to a
relation. If the two instruments are found to be network of explanatory ideas that make sense of
highly correlated then there is evidence of concur- the trait. This network of explanation is founda-
rent validity. For exam- tional to how construct
ple, Sudsawad, Trombly, validation is accom-
Henderson, and Tickle- A well articulated theory that plished. It includes not
Degnen (2000) studied only a clear definition of
the relationship between explains a construct and its the construct but also
the Evaluation Tool of relationship to other con- how it is related to other
Children’s Handwriting constructs. A well articu-
(ETCH) and teachers’ structs allows for a stronger lated theory that explains
perceptions of handwrit- approach to validation. a construct and its rela-
ing legibility (the crite- tionship to other con-
rion variable) using a structs allows for a
questionnaire that asked about the student’s hand- stronger approach to validation.
writing performance in the classroom. Contrary to Construct validity testing is sometimes refer-
expectations, there was no significant relationship red to as hypothesis driven, since studies that
between the ETCH and teacher questionnaire provide evidence of construct validity test hypo-
scores in legibility or task-specific legibility. The theses that are based on theoretical assertions
findings of this study brought into question about the construct and its relationship to other
whether the ETCH validly measures handwriting variables.
legibility. There are several approaches to demonstrating
construct validity; the most common include:
Predictive Validity
• Known groups method,
Predictive validity involves evidence that a meas- • Convergent and divergent methods, and
ure is a predictor of a future criterion. Assessment • Factor analytic methods.
of predictive validity is achieved by administering
the instrument under question first and then col- Each is discussed below.
lecting data on the criterion variable at a later time.
For example, if an assessment is designed to cap- Known Groups Method
ture a client’s ability for independent living or
The known groups method involves identifying
return to work, then the appropriate criteria would
subjects who are demonstrated to differ on the
be whether the person is living independently or
characteristic the instrument aims to measure. So,
employed at a future date.
for example, if an instrument is designed to meas-
Predictive validity is often challenging to
ure capacity of independent living, it might be
demonstrate because it requires a longitudinal
administered to people who are living in nursing
study. All other forms of validity and reliability
homes and those living independently in the com-
testing can essentially be done through simultane-
munity. In this instance, the instrument should pro-
ous data collection. Nonetheless, it can be power-
duce different scores for the two groups of persons
ful evidence of validity.
(i.e., document differences known to exist in the
two different groups).
Construct Validity
With the known groups method, it is also com-
Construct validity refers to the capacity of an mon to perform a discriminant analysis (see
instrument to measure the intended underlying Chapter 17) to evaluate the ability of the meas-
ure(s) derived from the instrument to correctly tics that are closer and more distant conceptually
classify the subjects into their known groups. from the intended construct and asking whether
Discriminant analysis is a form of regression the strength of association is related.
analysis in which independent variables (in this Campbell and Fiske (1959) proposed a combi-
case, test scores) and categorical dependent vari- nation of convergent and divergent validity for
ables (group membership) are analyzed. Based assessing validity; it is referred to as the multi-
on an equation generated by the discriminant trait–multimethod approach. According to this
analysis, individuals are assigned to groups in the approach an investigator would examine two or
analysis. If the measure accurately discriminates more traits using two or more instruments (meth-
subjects into known groups, there is evidence of ods) for measuring each trait. For example, if one
validity. is developing an instrument to measure problem-
solving, it should correlate strongly with another
Convergent and Divergent Methods test of problem-solving and moderately with a test
of attention. Similarly, two tests of attention should
Assessment of convergent and divergent validity correlate more strongly than tests of different
involves theoretically derived comparisons. traits such as a test of attention and a test of prob-
Convergence is the principle that two measures lem solving. The logic is that the two tests
intended to capture the same underlying trait designed to measure the same thing should be
should be highly correlated. Convergence obvi- more correlated than two tests designed to measure
ously overlaps with the concept of concurrent concepts whose functions overlap as illustrated in
validity. Implied in the concept of concurrent Figure 12.2.
validity is that the association between two meas-
ures of the same construct should be demonstrated
Factor Analytic Method
across different circumstances of place, sample,
and time. When investigators believe that a construct is or
Divergence (or discriminant validity) is the may be multidimensional, factor analysis (see
principle that tests designed to measure different Chapter 17) is sometimes used to empirically
traits should show patterns of association that dis- demonstrate the dimensions of the construct.
criminate between the traits. Thus, for example Factor analysis examines a set of items that make
two measures of unrelated traits such as attitudes up an instrument and determines whether there are
toward leisure and motor capacity should be unre- one or more clusters of items.
lated. Similarly, measures of associated but not Sachs and Josman (2003) used factor analysis
identical constructs should be moderately related. to study the Activity Card Sort (ACS), a standard-
Thus, in using convergent and divergent methods, ized assessment that aims to measure the amount
investigators examine how closely the results of an and level of involvement in various activities. The
instrument correlate with measures of characteris- ACS requires persons to sort cards depicting peo-
Problem- Problem- Attention

solving solving
Problem-
solving
Attention Problem- Attention

solving
Figure 12.2 An illustration of
the multitrait–multimethod Attention
matrix: Expected pattern of
concurrence–divergence
based on theoretical associ-
ations of problem-solving
and attention.
Strong Correlation Moderate Correlation
ple engaged in real-life activities into categories. Summary

The investigators administered the ACS to 184
participants (53 students and 131 elderly persons). As noted earlier, evidence about construct validity
Factor analysis revealed five factors (i.e., demand- is ordinarily reflected in a number of the types of
ing leisure, instrumental activities of daily living, construct validity just discussed. The feature box
maintenance, leisure, and social recreation) for below includes an example of an investigation
students and four factors (i.e., instrumental activi- designed to provide evidence about construct valid-
ties of daily living, leisure, demanding leisure, and ity. The study reported in this feature box is one of
maintenance) for older persons based on 60 pic- a series of studies that ultimately contribute to the
tures. The results of this study indicate an impor- construct validity of the instrument in question.
tant feature of factor analysis (i.e., its sample
dependency). Factors from an instrument identi- Interrelationship of
fied for one group cannot be assumed to be the
same for another group. Moreover, factors should Reliability and Validity
be previously hypothesized based on the underly-
ing theory. Exploratory factor analysis that looks The reliability and validity of instruments are
for patterns after the fact does not provide evi- interrelated. If an instrument is unreliable because
dence of validity. of random measurement error, it cannot be valid.
A Study of Construct Validity
Koning and Magill-Evans (2001) studied the con- of general social skills (which does not measure
struct validity of the Child and Adolescent Social social perception). This instrument can be used
Perception Measure (CASP) (Magill-Evans, by a parent, a teacher, or completed via self-
Koning, Cameron-Sadava, & Manyk, 1995), an report. Because it measures a construct (general
instrument that measures the ability to use nonver- social skills) that is related to (but not identical
bal cues to identify the emotions of others. This to) the construct of social perception measured
instrument involves 10 videotaped scenes that by the CASP, moderate correlations were pre-
depict situations that children and adolescents fre- dicted. Correlations between the ES and ESC and
quently encounter (with verbal content removed). the parent, child, and teacher ratings on the SSRS
After viewing each scene, the persons being tested ranged from.34 to .63, as expected. The pattern of
are asked to identify the emotions portrayed by correlations (stronger association for parent and
each of the characters and to note which cues they teacher than for students) was as expected, since
used to identify the emotions. The instrument gen- children with social skills problems tend to have
erates two scores: an emotion score (ES) that difficulty admitting their problems with peers.
reflects the ability to correctly identify emotions Similarly, moderate correlations were predicted
and a nonverbal cues score (ESC) that reflects the with IQ (i.e., 59 for both the ES and ESC).
ability to correctly identify the cues that were pres- Correlations with three scores from the Clini-
ent in the scene for inferring emotions. cal Evaluation of Language Fundamentals-Revised
In this study, the authors used a known groups (CELF-R), a standardized evaluation of expressive
approach. Participants were 32 adolescent males and receptive language skills (a construct not
who had social skills deficits consistent with the expected to be highly related to social perception),
diagnosis of Asperger’s Disorder and 29 controls ranged from .29 to .40. Correlations of the CASP
who were matched on sex, age, and intelligence with scores obtained from the Child Behavior
quotient (IQ). The mean score for both the ES and Checklist (CBCL), a standardized questionnaire in
ESC scores on the CASP was higher for the con- which the teacher or parent report the frequency of
trol group than the group with social skills deficits problem behaviors, ranged from .38 to .57. Since
(p ⬍ .001). The investigators also used discrimi- some problem behaviors are related to difficulties
nant analysis to determine how well the two CASP in reading social cues, these somewhat higher cor-
scores together could discriminate the subjects into relations were also consistent with expectations.
the two groups. The found that 96.9% of the chil- This study illustrates a systematic approach to
dren with social skills deficits and 86.2% of the construct validity for the CASP. The investigators
controls were correctly classified in their respec- used a known groups method, along with conver-
tive groups. gent and divergent methods. They demonstrated
The investigators further examined the cor- patterns of difference/discrimination and associa-
relation between the CASP and the Social Skills tion that fit based on theoretical arguments about
Rating System (SSRS), a standardized assessment the construct under study.
However, a reliable instrument is not, de facto, Other Psychometric Properties

valid. That is, an instrument might produce consis-
tent results but the items may not be targeted to the
intended variable; in this instance the items would Reliability and validity are fundamental to any
consistently measure something other than the good instrument. However, since instruments are
intended construct. used in practice and research to measure traits in
In the end, a desirable instrument achieves con- ways that influence clinical and research decisions,
sistent results that clearly target the underlying a good instrument must have other properties that
construct the instrument is designed to measure. include:
Factors that make for a weak instrument generally • Precision,
affect both reliability and validity. For example, if • Sensitivity and specificity,
an instrument includes several items that are • Criterion or norm referencing, and
ambiguous, the instrument will have high error • Standardized scores and standard error of meas-
(unreliability) and will fail to consistently target urement.
the underlying characteristic of interest (invalid-
ity). Efforts to enhance any instrument ordinarily Each of these psychometric properties is dis-
target both reliability and validity simultaneously. cussed below.
Creating and Evaluating Precision

Evidence of Reliability Precision refers to the exactness of a measure
(i.e., the extent of its ability to discriminate differ-
and Validity ing amounts of a variable). The more precise a
measure is, the finer its ability to discriminate
As noted earlier, validity is always a matter of between different amounts. For example, a ruler
degree that is reflected in the extent of accumulated that is calibrated only to 1/8 of an inch cannot
evidence about an instrument. Moreover, the more accurately discriminate smaller amounts. A ruler
variability in evidence (e.g., across types of valid- that is calibrated to 1/16th of an inch is more sensi-
ity, populations, and situations), the more confi- tive in measuring distance and, thus, has more pre-
dence one can have in the validity of a measure. cision. Similarly, a scale that measures only pounds
Deciding how to go about validating an instru- is less precise than a scale that measures pounds
ment or judging the evidence on behalf of an and ounces. The precision of physical measures
instrument depends in part on the intended use of is determined by the fineness of the calibration
the instrument. For example, Law (1987) argues they use.
that different approaches are relevant depending All other things being equal, the precision of
on whether the purpose of an instrument is measures of psychological or behavioral traits
descriptive, predictive, or evaluative. If the pur- (e.g., measures that are based on self-reports or
pose is descriptive, then evidence of content and rating scales) is generally related to the number of
construct validation is most relevant. If the pur- items on the scale. If each item is conceptualized
pose is predictive, then content and criterion- as an “estimate” of the underlying trait, precision
related validity are important to assess. If the increases with each estimate. Of, course other fac-
purpose is evaluative, content and construct valid- tors affect precision including the error associated
ity is most important. with each item and how well the items are targeted
In the end, each instrument developer must pur- to the person being measured (see Chapter 13 for a
sue a logic and strategy that provides evidence discussion of this issue).
connected to: Because more items increase precision, it
would seem logical to compose instruments of as
• The nature of intended construct, many items as possible. However, lengthy instru-
• The purpose to which the instrument will be put, ments are often not feasible in either practice or
• The population(s) with whom the instrument is research. Moreover, when instruments are too
intended to be used, and lengthy, error can be increased because of such
• The types of circumstances in which the instru- factors as fatigue. Thus, in constructing an assess-
ment will be applied. ment a balance must be struck between enhancing
When all these factors are reflected in the evi- precision through increased items and practical
dence of validity, one can have a degree of confi- issues of instrumentation (e.g., time, cost demands
dence in the validity of the instrument. on the client or subject).
Precision is closely linked to the ability of an to detect the presence of a problem when it does
instrument to measure change (sometimes referred not exist, nor does one want to incorrectly decide a
to as responsiveness to change) as well as to power problem is not present.
and effect size (discussed in Chapter 16). For In establishing the cutoff, increasing sensitivity
instance, if a scale is calibrated to the nearest will also likely reduce specificity. In other words,
pound only and a child grows by gaining 7 ounces, if a higher score means a problem is present and
no change will be detected. In choosing or devel- the cutoff is set at a lower level, then sensitivity is
oping an instrument for research or practice, one increased (persons with the problem will likely be
will need to consider how much change is relevant detected) but at the same time specificity is proba-
or important to detect and select an instrument bly reduced (some persons without the condition
with the appropriate level of sensitivity. For exam- will be judged to have it). If the cutoff is set higher
ple, if one were studying the influence of an inter- specificity can be increased (fewer or no persons
vention on the growth rates of premature infants, a will be incorrectly judged to have the problem),
scale calibrated to ounces might be very important. but sensitivity will be decreased (some persons
On the other hand, a study of weight loss among with the problem may not be detected). The extent
obese adults would not require the same level of to which this is an issue depends on how precise an
sensitivity. assessment is, but since all instruments have lim-
There is no standard way to assess the precision ited precision and some error, one has to decide in
of an instrument since desired precision is always which direction it is better to err.
a function of the purpose to which an instrument is
being applied. One common approach is to read- Norm Referencing and
minister an assessment following a time interval or
Criterion Referencing
circumstance that is expected to produce a change
and determine through statistical testing whether a The aim of any instrument is to allow the user to
change in the mean score is detected. Another make judgments. Two typical ways that judgments
solution is to calculate an effect size (see Chapter are formed are through norm referencing or crite-
16) in the same circumstances. rion referencing. Criterion referencing refers to
making judgments or interpretations of a score
with reference to what is considered adequate or
Sensitivity and Specificity
acceptable performance (Portney & Watkins,
When instruments are used to determine the exis- 2000). In occupational therapy, criterion may be
tence of a problem or the need for therapy, sensi- typically linked to such factors as performance
tivity and specificity are concerns. Sensitivity (i.e., adequacy for completing a task), participation
refers to the ability of an instrument to detect the (i.e., ability to partake in a occupation such as
presence of a problem or condition when it is pres- work or leisure or live independently in the com-
ent. Specificity is the ability of an instrument to munity), and quality of life or well being (e.g.,
produce negative results when the problem or con- adequate feelings of efficacy).
dition is not present. Sometimes a criterion is clearly linked to the
Most occupational therapy instruments do not content of a scale (e.g., if a scale includes all the
result in simple dichotomous decisions about the abilities that are necessary to drive safely, then
presence or absence of a problem. Rather, they passing all the items may be the criterion). In other
typically produce continuous data that illustrate cases criteria are tied to a level of knowledge or
where the individual falls on a continuum of less to ability such as passing a certain percentage of
more of the trait measured. Instruments require a items on a test. Criterion referencing works best
cutoff score when they are used to make a clinical when it is clearly linked to a relevant outcome. In
decision about whether a problem is present (or some instances it will be necessary to gather
sufficiently present to warrant intervention or empirical data on the measure and compare it to
some other decisions such as discharge place- the criterion in order to determine the logic for the
ment). criterion.
A cutoff score is the point beyond which a per- Norms are summarized data from the sample
son is determined to have a problem, or to be on which an instrument is developed (Benson &
unable to perform an activity or achieve a level of Schell, 1997). Norms are created by administering
independence. In establishing a cutoff score, one an instrument in a standardized way and then com-
has to take into consideration the sensitivity and bining and summarizing the scores obtained from
specificity of the cutoff. That is, one does not want all those on whom data were collected. Norms
should be created by testing a large sample that whom norms have been developed for the instru-
meets a profile (e.g., that proportionally represents ment. The t-scores are computed by setting the
all those with whom the instrument is intended to mean at 50 and the standard deviation at 10. They
be used). allow a person’s raw score to be converted to a per-
Norms are usually reported in terms of means, centile for comparison to normative data.
standard deviations, percentiles, and standard If any instrument was applied to an individual
scores. It is important to recognize that norms are an infinite number of times, it is reasonable to
sample dependent, so that it is important that the assume that the scores obtained would vary some-
sample on which norms are based is thoroughly what each time. Theoretically, the distribution of
described (Benson & Schell, 1997). When consid- these scores would resemble a normal curve (see
ering the usefulness of norms one should consider Chapter 15 for discussion of the normal curve).
whether the sample on which norms are based is Moreover, the mean of all these measures would be
sufficiently similar to the intended population. For equal to the person’s true score (i.e., the actual
instance, one might ask whether the ages, cultural amount of the variable or characteristics in the per-
background, and sex of the sample are the same as son being measured) and the scores would be
in the target population. evenly distributed on either side of this mean with
Considerable misunderstanding and contro- fewer and fewer obtained scores the more extreme
versy exists over the use of norm-versus criterion- the deviation from the mean. The more accurate or
referenced tests. Proponents of norm-referencing error free a measure, the more closely the obtained
argue that norms provide important points of scores will be distributed around the mean.
reference from which to judge whether or not a The standard deviation of all the measurement
person’s performance or characteristic is of con- errors is referred to as the standard error of meas-
cern or warrants intervention. Critics of norm- urement (SEM). The SEM for a given instrument is
referencing point out that norms often fail to take estimated from a sample of subjects using the fol-
into consideration variability in development and lowing formula:
performance and they argue that use of norms
unfairly devalues or stigmatizes persons who are SEM ⫽ sx 兹苶
1 – rxx
different from the average. Proponents of crite-
rion-referencing point out that criteria are more where sx ⫽ the standard deviation of the
“objective” in that they link judgments to some- observed scores and rxx ⫽ the reliability coefficient
thing that is rational and not biased by a preference for the instrument (e.g., test–retest or inter-rater
for the typical. Critics of criterion referencing reliability) (Portney & Watkins, 2000).
point out that a criterion can sometimes be arbi- The SEM can be interpreted according to the
trarily imposed. In the end, the use of both norm characteristics of the normal curve. That is, there is
and criterion referencing should be done with a 68% chance that an individual’s true score falls
clear attention to the purposes for which they within ⫾ 1 SEM of the obtained score and a 95%
are being used and the consequences of making chance that the true score falls within ⫾ 2 SEM of
judgments. the obtained score. An important limitation of the
In clinical practice, these considerations must SEM is that its interpretation is dependent on the
be individualized for a client. For example, identi- type of reliability coefficient that is used to com-
fying a child’s deviation from developmental pute it. So, for example if it is based on test–retest
motor or cognitive norms may be important to reliability the SEM reflects the error expected
determining the need for early intervention serv- based on readministration; it will not take into con-
ices for one child with developmental disability. sideration error due to rater bias.
On the other hand, some developmental norms
may be much less relevant than criteria for making
judgments about treatment intervention with a Conclusion
child who is severely developmentally delayed.
This chapter provided an overview of the main
Standardized Scores and Standard issues of concern in the development and evalua-
tion of measurement instruments from the perspec-
Error of Measurement
tive of classical test theory. Entire books are
Standardized scores (i.e., t-scores) are sometimes devoted to detailed discussions of the concepts
developed for instrument scores to facilitate com- and methods of measurement and instrument con-
parison of an individual’s score to that of others on struction. Therefore, this chapter is best taken as an
orientation to these issues and as a guide for exam- For the investigator who is seeking to develop
ining literature that reports the development and an assessment, the references cited in this chapter
empirical study of measurement instruments. and the resources listed below are a good begin-
Moreover, the next two chapters will add addi- ning. In addition, the feature box includes a dis-
tional considerations for evaluating and developing cussion of the general steps involved in developing
measures. and empirically assessing instruments.
The Process of Instrument Development
Instrument development is an ongoing process • Specify the underlying construct. Good meas-
that, arguably, has no clear endpoint. Basically, an urement begins with a clear articulation of the
instrument should be developed and judged fol- underlying construct an instrument is designed to
lowing a stream of evidence about its reliability measure. Instrument developers focus not only
and validity. Different authors have recommended on clearly articulating the trait or variable the
the sequence of steps that should go into the con- instrument targets, but also on whether this trait
struction and investigation of instruments (Benson is homogeneous or multifaceted.
& Clark, 1982; Benson & Hagtvet, 1996). • Create a plan of how the construct will be
Nonetheless, there is no standard process for operationalized. This step involves identifying
developing assessments. The following is a recom- the range or universe of content that will be
mendation of steps that can be followed based on included. If a multidimensional construct is
the literature and the experience of this author in intended, each subconstruct needs to be clearly
developing a number of instruments. Since this defined and its content identified. If the instru-
chapter focuses on classical test theory, these steps ment targets a single homogeneous construct, the
will mainly refer to the methods that belong to that content within this domain needs to be identified.
approach. Other approaches to instrument develop- During this stage, it is also helpful to think about
ment are discussed in Chapter 13 and are followed what represents the continuum of the construct.
by many instrument developers. It should be noted Particular attention should be paid to the upper
that the steps of instrument development noted and lower ends of the characteristics to be meas-
below are not strictly linear. Often, instrument ured to ensure that the content covers the whole
developers will move back and forth between steps range of variation in the characteristic (construct)
as an instrument is being refined. to be measured.
• Identify the need for the instrument. A large During this stage it is often useful to consult
number of instruments both within and relevant with experts (i.e., with persons who have knowl-
to occupational therapy exist. Thus, before edge and experience relevant to the intended con-
embarking on the development of an instrument, struct, purpose, and population of the instrument).
one should first identify the need for the assess- Focus groups with such experts (including clients
ment. Typical reasons for developing an assess- for whom the instrument is intended) can be help-
ment are the lack of adequate assessments for the ful in brainstorming appropriate content.
intended construct and the inapplicability of • Decide the format of the instrument. This step
existing assessments for the intended population is linked to identifying the purpose of the instru-
or context of administration. This step obviously ment. It involves deciding the actual mechanism
requires that one comprehensively review the lit- by which information will be gathered and trans-
erature to identify existing instruments and their lated into numbers. Some typical choices are self-
properties. report forms or card sorts that will be scored,
• Identify the purpose and the intended popula- observational instruments that use categorical or
tion. Measurement instruments are tools for ordinal rating forms, interviews accompanied by
accomplishing particular aims with a particular rating forms, tests of various capacities, and so
group of clients or subjects. One should clearly on. This step should also give consideration to the
outline the purpose of the instrument, including population for which the instrument is intended to
what type of information it is intended to gener- make sure the format of the instrument is suited
ate and for what ends the information will be to their age, capacity, and so
used. In addition, one needs to consider the forth.
intended population in terms of such factors as An important consideration at this stage is
age, cultural background, and what types of how the instrument will be administered and by
impairment may be present. This information is whom. If it is intended to be a self-administered
important to determining both the content and the instrument, then consideration has to be given to
format of the instrument. the persons who will be taking it (e.g., their
(continued)
The Process of Instrument Development (continued)
reading level and their perspective). If it is to be • At the right level of difficulty,

administered by a tester or observer, consideration • Understandable,
needs to be given to how much knowledge and • Applicable to the circumstance of the
skill will be assumed. respondent, and
During this stage, developers consider details of • Reflect what they consider important about
the instrument. For example, if the instrument is a the thing being measured.
rating form one needs to consider whether the When focus groups are not feasible, it may be
items will be rated as dichotomous (yes/no) or on useful to approach a series of individuals for their
an ordinal scale. If an ordinal rating is used, then opinions on these issues.
developers must consider how many rating points • Develop supporting materials. This step
should be included. This decision is influenced by involves clearly delineating and writing out
considering conceptually how many values or instructions, procedures, meanings of items, and
points are important to identify. For example, so on. Usually, at this stage one will create an
in creating a rating of self-care independence, administration guide or manual to accompany the
investigators have generally sought to discriminate instrument.
between levels of assistance that are needed so • Pilot the instrument. During this stage, a first
that a three-point scale (e.g., independent, needs version of the intended instrument should be
assistance, dependent ) is considered insufficient applied as it is intended to be used and with
to discriminate the levels of independence that are the administrators and/or population for which
of clinical relevance. it is intended. Pilot testing almost inevitably
Another consideration is variability. Scales results in the identification of one or more of the
generally provide more variability when there are following:
more rating points. One exception, however, is • Unforeseen difficulties in administration,
that, sometimes, scales with middle points (e.g., • Problems with content (e.g., too difficult or
a five-point scale) provide less variability than irrelevant),
scales without a neutral rating (e.g., a four-point • Ambiguity in items,
scale) since the latter forces persons who other- • Difficulty with test instructions or proce-
wise would overuse the neutral rating to make dures, and
discriminations. A final consideration is how fine • Difficulties in using the categories or rating
a discrimination the respondent or user can make. scale.
Providing a seven-point scale to clients who do • Revise the instrument and develop supporting
not make such fine discriminations in their judg- materials. Feedback from the pilot testing typi-
ments will only add error to the instrument. cally results in identification of items that need
• Develop items. Once the target construct and to be clarified or revised and the need to more
format for the instrument are clearly defined and clearly spell out instructions, procedures, and
operationalized, the developer will go about cre- meanings of items, and so on. Usually the pilot
ating the actual items that make up the question- will result in revision of the administration guide
naire, test, and rating scale. This process is or manual. Depending on the extent of revision
critical since an instrument will only be as good needed, one may repeat the pilot phase and once
as the items of which it is constructed. Develop- again revise. Some instrument developers who
ment of items often involves: use a participatory research process (see Chapters
• Asking for feedback from a panel of experts on 38, 39, and 40) use an iterative process in which
initial items. This helps to ensure that the con- the instrument is successively piloted and revised
tent targets the intended variable (when experts with therapists and/or consumers acting as peers
are knowledgeable about the variable of inter- in the instrument development process.
est) and that the content is relevant to practi- • Empirically assess reliability and validity in
cal/clinical considerations (when experts are successive stages. This step is actually a series of
experienced practitioners). This can be done in steps or studies that progressively assess the
a focus group format or through a survey. Often instrument. As noted in this chapter, how one
it is helpful to create a survey of intended items goes about this depends on the nature of the
and ask experts to provide feedback on several instrument being developed. One should develop
dimensions of each intended item (e.g., clarity, a plan of the sequence of questions that are to be
relevance, importance). asked about reliability and validity. Generally, it
• Doing focus groups with persons with whom is useful to start with smaller samples that are
the instrument will be used and/or with practi- necessary for the statistics to be performed, since
tioners who will use the assessment. This is early studies often result in further revision of the
often done to ensure the relevance of the items instrument. Once the instrument is finalized,
to those for whom it is intended. Relevance larger samples necessary for creating norms or
may mean that the items are: identifying criterion (cutoffs) can be studied.
175
REFERENCES Guilford, J. (1979). Psychometric methods (2nd ed.). New

York: McGraw-Hill.
Ayres, A. J. (1989). Sensory integration and praxis texts Hambleton, R. K., & Jones, R. W. (1993). Comparison of
(manual) . Los Angeles: Western Psychological classical test theory and item response theory and their
Services. applications to test development. Educational Measure-
Benson, J., & Cark, F. (1982). A guide for instrument ment: Issues and Practice, 12, 38–47.
development and validation. American Journal of Keller, J., Kafkes, A., Basu, S., Federico, J., & Kielhofner,
Occupational Therapy, 36, 789–800. G. (2005) The Child Occupational Self-Assessment
Benson, J., & Hagtvet, K. (1996). The interplay between (COSA) (version 2.1). Model of Human Occupation
design and data analysis in the measurement of coping. Clearinghouse, Department of Occupational Therapy,
In M. Zeidner & N. Endler (Eds.), Handbook of coping College of Applied Health Sciences, University of
(pp. 83–106). New York: John Wiley & Sons. Illinois at Chicago, Chicago, IL.
Benson J., & Schell, B. A. (1997). Measurement theory: Koning, C., & Magill-Evans, J. (2001). Validation of the
Application to occupational and physical therapy. In J. child and adolescent social perception. Occupational
Van Deusen & D. Brunt (Eds.), Assessment in occupa- Therapy Journal of Research, 21, 41–67.
tional therapy and physical therapy (pp. 3–24). Lafayette Instruments (1969). The Complete Minnesota
Philadelphia: W. B. Saunders. Dexterity Test. Examiner’s manual. Lafayette Instru-
Borg, W., & Gall, M. (1983). Educational research: An ments, Lafayette Instrument Co., Lafayette, IN.
introduction (4th ed.). New York: Longman. Law, M. (1987). Measurement in occupational therapy:
Campbell, D. T., & Fiske, D. W. (1959). Convergent and Scientific criteria for evaluation. Canadian Journal of
discriminant validation for the multitrait-multimethod Occupational Therapy, 54(3), 133–138.
matrix. Psychological Bulletin, 56, 81–105. Li, Y., & Kielhofner, G. (2004). Psychometric properties of
Chern, J., Kielhofner, G., de las Heras, C., & Magalhaes, the Volitional Questionnaire. Israeli Journal of Occu-
L. (1996). The volitional questionnaire: Psychometric pational Therapy, 13, 85–98.
development and practical use. American Journal of Magill-Evans, J., Koning, C., Cameron-Sadava, A, &
Occupational Therapy, 50, 516–525. Manyk, K. (1995). The child and adolescent social per-
Clemson, L., Fitzgerald, M. H., Heard, R., & Cumming, ception measure. Journal of Nonverbal Behavior, 19,
R. G. (1999) Inter-rater reliability of a home fall haz- 151–169.
ards assessment tool. Occupational Therapy Journal Nunally, J. C. (1978). Psychometric theory (2nd ed.). New
of Research, 19, 83–100 York: McGraw-Hill.
Cohen, J. (1960) Coefficient of agreement for nominal Ottenbacher, K. J., & Tomchek, S. D. (1993). Measurement
scales. Educational and Psychological Measurement, in rehabilitation research: Consistency versus consen-
20, 37–46. sus. In C. V. Granger & G. E. Gresham (Eds.), Physical
Cohen, J. (1968). Weighted kappa: nominal scale agree- medicine and rehabilitation clinics of North America:
ment with provision for scaled disagreement or partial New developments in functional assessment (pp.
credit. Psychological Bulletin, 70, 213–220. 463–473). Philadelphia: W. B. Saunders.
Cronbach, L. J. (1951). Coefficient alpha and the internal Portney, L. G., & Watkins, M. P. (2000). Foundations of
structure of tests. Psychometrika, 16, 297–334. clinical research: Applications to practice (2nd ed.).
Cronbach, L. J. (1971). Test validation. In R. L. Thorndike Upper Saddle River, NJ: Prentice-Hall.
(Ed.), Educational measurement (2nd ed., pp. 443–507). Sachs, D., & Josman, N. (2003). The activity card sort: A
Washington, DC: American Council on Education. factor analysis. OTJR: Occupation, Participation and
Cronbach, L. J. (1990). Essentials of psychological testing Health, 23, 165–174.
(5th ed.). New York: Harper and Row. Streiner, D. L., & Normal, G. R. (1995). Health measure-
de las Heras, C. G., Geist, R., Kielhofner, G., & Li, Y. ment scales: A practical guide to their development
(2003). The Volitional Questionnaire (version 4.0). and use (2nd ed.). New York: Oxford University Press.
Model of Human Occupation Clearinghouse, Sudsawad, P., Trombly, C. A., Henderson, A., & Tickle-
Department of Occupational Therapy, College of Degnen, L. (2000). The relationship between the evalu-
Applied Health Sciences, University of Illinois at ation tool of children’s handwriting and teachers’
Chicago, Chicago, IL. perceptions of handwriting legibility. American Journal
Doble, S., Fisk, J. D., Lewis, N., & Rockwood, K. (1999). of Occupational Therapy, 55, 518–523.
Test–retest reliability of the assessment of motor and
process skills in elderly adults. Occupational Therapy
Journal of Research, 19, 203–219. RESOURCES
Fisher, A. G. (1993). The assessment if IADL motor skills: Benson, J., & Cark, F. (1982). A guide for instrument
An application of many faceted Rasch analysis. development and validation. American Journal of
American Journal of Occupational Therapy, 47, Occupational Therapy, 36, 789–800.
319–329. Cronbach, L. J. (1990). Essentials of psychological testing
Fleiss, J. L. (1981). Statistical methods for rates and pro- (5th ed.). New York: Harper and Row.
portions. New York: John Wiley & Sons. Nunally, J., & Bernstein I. H. (1994). Psychometric theory
Forsyth, K., Deshpande, S., Kielhofner, G., Henriksson, C., (3rd ed.). New York: McGraw-Hill.
Haglund, L., Olson, L., Skinner, S., & Kulkarni, S. Thorndike, R. L., & Hagen, E. (1990). Measurement and
(2005). The Occupational Circumstances Assessment evaluation in psychology and education (5th ed.). New
Interview and Rating Scale (version 4.0). Model of York: John Wiley & Sons.
Human Occupation Clearinghouse, Department of Van Deusen J., & Brunt D. (Eds.) (1997). Assessment in
Occupational Therapy, College of Applied Health occupational therapy and physical therapy. Philadel-
Sciences, University of Illinois at Chicago, Chicago, IL. phia: W. B. Saunders.
C H A P T E R 1 3
Objective Measurement: The Influence

of Item Response Theory on
Research and Practice
Craig A. Velozo • Kirsty Forsyth • Gary Kielhofner
As presented in Chapter 12, standardization, relia- reveals that many present and future instruments
bility, and validity are the cornerstone of develop- can emulate the precision of measures used in the
ing sound instruments in occupational therapy and physical sciences.
health care. Standardization in administering and Accurate, consistent, and meaningful measure-
scoring instruments is critical to ensure that scores ment is possible only with standardized, reliable,
obtained from different raters or administrations of and valid instruments. The purpose of this chapter
an instrument are com- is to build upon and go
parable. The consistency beyond the traditional
of results, as established Accurate, consistent, and psychometric analyses
by reliability testing, that are discussed in
provides researchers and meaningful measurement Chapter 12. The chapter
therapists with the confi- is possible only with introduces concepts and
dence that the scores examples of objective
from assessments will be
standardized, reliable, and measurement. It also dis-
consistent at different valid instruments. cuss methodologies that
times and across differ- can provide additional
ent raters. Furthermore, value to existing instru-
studies of validity, especially construct validity, ments and provide a framework for developing
provide evidence that instruments measure what new measures with features not attainable via tra-
they intend to measure. ditional psychometrics. For reasons that will
Standardization, reliability, and validity of become apparent later in this chapter, the numbers
instruments are important since the correctness of attached to the majority of occupational therapy
judgments depends on them. Decisions of whether and healthcare assessments are referred to here
a client needs a given service, has improved, or as scores. Only those numbers that have the attrib-
should be discharged are typical judgments thera- utes of objective measurement are referred to as
pists make in everyday practice. Occupational measures.
therapy researchers make judgments about
whether a tested intervention works, or whether
one group differs from another on variables of Comparison of
interest. For such judgments to be sound, they
must depend upon standardized, reliable, and valid
Measures to Scores
assessment instrumentation.
One way to clarify the differences between objec-
In spite of the well-established psychometric
tive measures and the present-day “scores” used
procedures upon which occupational therapy and
in occupational therapy is to compare how basic-
healthcare measures have been built, these assess-
science measures distance with how occupatio-
ments have always lacked the degree of accuracy
nal therapy attempts to rate or score self-care.
found in measurement in the basic sciences.
Table 13.1 on p. 178 presents a summary of this
Obviously, many of the constructs assessed in
comparison.
occupational therapy (e.g., functional ability, bur-
den of care, volition) are intrinsically more global
Equal Intervals
or abstract than those found in the physical sci-
ences (e.g., distance, temperature, and weight). On all measures of distance, markings indicating
Nonetheless, examining occupational therapy con- increments of distance have equal intervals. For
structs from an objective measurement perspective instance, markings of inches or centimeters have
Table 13.1 Comparison of Measures to Scores

Ruler (Measure) ADL (Score)
Equal intervals—Markings on ruler represent Unequal intervals—Ratings do not necessarily represent
equal intervals. equal intervals.
Efficiency—Use only the aspect of the ruler that Inefficiency—All items of the assessment are
is most relevant to the object being measured. administered, independent of the ability of the
individual.
Precision—Once the distance of an object is Imprecision—No logical method to achieve precision.
estimated, attention is paid to the finer
markings of the ruler to gain precision.
Transparency—Distance can be translated Nontransparency—No convenient, reliable ways to
across numerous instruments and converted translate scores between assessments.
across different scales.
equal distances along the entire length of the meas- ing. That is, when measuring an object that is
urement instrument (e.g., the distance between 3 81/2 inches wide, one uses only the markings close
and 4 inches is the same as the distance between 1 to 81/2 inches. One does not have to test the object
and 2 inches, 5 and 6 inches, and so on). In con- against the markings at 2, 3, 4, 5 inches and so on.
trast, the markings for scores on activities of daily In contrast to using a ruler, the generating a score
living (ADL) assessments do not have this charac- for ADL is not particularly efficient. The great
teristic of equal intervals. The ratings of the items majority of ADL instruments require that the client
(or questions) of an assessment may appear to be be scored on all items of the instrument. That is,
equidistant since one often rates them with num- even if an individual is capable of walking up
bers (e.g., 1 ⫽ total dependence assist, 2 ⫽ needs stairs, according to standardized procedures, the
maximal assist, 3 ⫽ moderate assist, 4 ⫽ minimal assessor is required to evaluate whether or not that
assist, and 5 ⫽ independent). Nonetheless, on individual is capable of transferring to a toilet seat
this type of rating it cannot be guaranteed that the or in or out of a tub. Deriving a score from the
intervals between numbers are all the same. For instrument requires that all of the items of the
example, is the interval between total dependence instrument be rated so that they can be totaled.
and maximum assistance equivalent to the interval
between minimum assistance and independence? Precision
When studies have examined this type of ques-
Not only is a ruler efficient, but even the crudest of
tion, answer is almost invariably, no. Even when
rulers is quite precise and can be used according to
rating scales are labeled with percentages (e.g.,
the level of precision desired. A quick observation
25%, 50%, 75% independent) there is no guaran-
may give an estimated width of the object (e.g.,
tee that the distances between these ratings are
81/2 inches) that may be adequate for many pur-
equivalent.
poses. However, if desired, one can more closely
observe with the ruler to more precisely measure
Efficiency the width of an object. By lining the finer markings
The application of the ruler1 (e.g., a 12-inch ruler, on the ruler to the edge of the object one might
a yardstick, or a tape measure) to the task of meas- arrive at the conclusion that 8 and 7/16 th inches is
urement is easily learned and thus can be done the more accurate measurement of its width.
with acceptable efficiency by virtually everyone. Unfortunately, with traditional ADL instru-
To measure the width of a small object, we do not ments, there is no analogous method of achieving
have to use the entire ruler, just the part of the ruler greater precision in measuring ADL status.
that is relevant to the object that is being measur- Independent of the ability level of an individual,
all items of the instrument must be used to obtain
a score. This means that many items that provide
1This example of measurement is taken from continued little information on the person’s ADL ability (i.e.,
American use of the English system of measurement. We are
aware that in many other parts of the world the metric sys-
items well below and/or well above the persons’
tem is used for everyday measurement. The example we ability level) are used to measure ADL ability. This
give can be easily extrapolated to this form of measurement. approach sacrifices both efficiency and precision.
Chapter 13 Objective Measurement: The Influence of Item Response Theory 179
Transparency ures using Rasch and IRT methodologies (Linacre,

Heinemann, Wright, Granger, & Hamilton, 1994;
Measures of distance are “transparent” (Fisher, Marino & Graves, 2004).
1997). That is, the measures exist independently of
the instruments used to generate them. When pro-
vided with a particular measure of distance (e.g.,
Scores Are Not Measures
inch, centimeter, yard, meter, or mile), there is no Scores from most occupational therapy and health-
need to indicate the instrument used to attain that care instruments produce frequency counts, not
measurement in order to understand what the measures. Total values obtained from these types
measure means. Furthermore, translating scores of instruments represent the number of items the
across scales (e.g., inches to centimeters) is com- client passed (i.e., in yes/no or pass/fail instru-
monplace and easily ments) or a total of the
accomplished by placing number of items with
both scales on the same While all measures are a rating of 1, 2, 3, 4,
ruler or with simple con- etc. Scores or frequency
version formulas.
numbers, all numbers are counts produced from
The assessment of not measures. most assessments used
ADL is not so transpar- by occupational thera-
ent. The meaning of pists fail to reflect the
scores is tied to the instrument used to generate the qualities of objective measurement and they fail to
scores. Scores from one scale, such as the adequately capture the underlying construct that
Functional Independence Measure (FIM™) are the instruments intend to represent.
not easily translated into other ADL scales such as Ordinal ratings produced by scores are not the
the Barthel Index. This limitation prevents moni- same as interval ratings produced by measures. A
toring clients across settings that use different measure is a number with which arithmetic and
ADL instruments (e.g., inpatient rehabilitation and linear statistics can be accomplished (Wright &
skilled nursing facilities) and similarly makes it Linacre, 1989). That is, even the simplest form
difficult to compare findings across studies that of mathematics requires interval values in which
use different instruments. distances between consecutive ratings are equal
(Merbitz, Morris, & Grip, 1989; Wright &
Linacre, 1989). Scores fail to achieve interval-
Limitation of Scores based measurement.
For example, on the FIM, item ratings indicat-
Versus Measures ing that total dependence is 0%, maximum assis-
tance is 25%, moderate assistance is 50%, and so
The terms measure and measurement are widely forth suggest that the adjacent ratings are equidis-
used to refer to occupational therapy and health- tance from each other. However, these labels do
care assessments that yield numerical values. not ensure that the distances between ratings are
However, such usage is not entirely accurate. equal as is the case on an interval scale. Figure
Although all measures are numbers, all numbers 13.1 on p. 180 shows a true interval scale and the
are not measures (Wright, 1997b). It is common to typical type of ordinal scale used for rating ADL;
assign numbers to information gathered in practice as shown the latter usually has unequal intervals
and research, especially when therapists or clients across the ratings. On such a scale, there is no
are asked to rate or judge information. In doing so, mathematical guarantee that an improvement from
it is also common to ignore that one has made a total dependence to maximum assistance equals an
major conceptual jump when attaching numbers to improvement from modified independence to total
the qualitative statements used to characterize independence. Without the guarantee that the dis-
observations. tances between the numbers produced by instru-
Because of its familiarity to many occupational ments are interval, analyses from scores can lead
therapists, the FIM is used throughout this chapter to misinterpretations. For example, if improve-
to demonstrate the limitations of using scores ment from total dependence to maximum assis-
instead of measures in occupational therapy and tance actually represents a larger change than an
health care. This is not intended to be an indict- improvement from moderate assistance to mini-
ment of the FIM. The FIM shows sound psycho- mum assistance, ordinal scores would underesti-
metric qualities and efforts have been made to mate improvements made by more severely
convert scores produced by the FIM to true meas- impaired clients.
Distribution of Ratings on an Interval Scale
1 2 3 4 5 6 7
Typical Distribution of Ratings on the Type of Ordinal Scale Used to

Rate ADL Assessments
1 2 2 33 4 4 5 6 6 7
Total Maximum Moderate Minimum Supervision Modified Total
Dependence Assistance Assistance Assistance Independence Independence
Figure 13.1 Rating scale comparison: Interval rating scale versus ordinal ratings.
There are other problems in how scores repre- present, there are more than 85 assessments of
sent characteristics of clients or subjects. For ADL (McHorney, 2003). Ironically, while the
example, on the FIM, the lowest score, 18, repre- items included in these assessments overlap con-
sents getting a 1 (total dependence) on all 18 items siderably (e.g., almost all assessments include
and the highest score, 126, reflects getting a 7 items that represent eating, dressing, grooming/
(independence without equipment) on all 18 items. hygiene, etc.), instrument developers and users
A score of 18 on the FIM implies the lowest pos- seem to have incessant debates on which assess-
sible functional ability and score of 126 implies ment is the best and there have been few efforts to
the highest possible functional ability. create methods to translate scores between the dif-
These implications are incorrect. The lowest ferent instruments.
score of 18 does not differentiate a range of abili- Test dependency leads to an almost total lack of
ties (e.g., the individual who responds to only pin- communication in healthcare research and out-
prick and the individual who only has the ability to comes. Researchers often explain their failure to
bring a face cloth to his/her face). These two indi- reproduce findings from previous studies as a
viduals would both would get the same, lowest result of the inability to compare findings from dif-
score on the FIM while they actually have differ- ferent assessments. Furthermore, test dependency
ent ability levels. Similarly, a score of 126 does not prevents the monitoring of patients across health-
reflect highest possible functional ability. This care settings that use different instruments. For
score would be assigned only to a certain range of example, it is difficult, if not impossible, to empir-
individuals: those individuals with only minor ically determine if a patient who is transferred
physical and cognitive abilities, to individuals who from a rehabilitation inpatient facility (which uses
are fully functional in all their life roles. Thus, the the FIM as an outcome measure) deteriorates,
FIM would fail to differentiate these individuals. maintains, or improves after transferring to a
As with the FIM, scores produced by most instru- skilled nursing facility (which use the Minimum
ments used by occupational therapists fail to ade- Data Set [MDS] as an outcome measure).
quately represent the true range of ADL abilities.
These are ways in which the FIM and other assess- Scores May Lack Unidimensionality
ments fail to achieve the requirements of objective
An essential characteristic of measurement is the
measurement.
quality of unidimensionality. That is, a measure
should define a single, unidimensional variable
Scores Are Test Dependent
(Wright & Linacre, 1989). For example, speed,
Inherent in producing scores instead of measures is weight, and length are all unidimensional con-
test dependency. That is, instruments with different structs; a score that reflected both weight and
items, number of items, and/or rating scales gener- length would be confusing. How such confusion
ate different total scores. Test dependency leads to arises from multidimensional instruments in occu-
the almost endless development of assessments pational therapy can be exemplified with the FIM.
that are intended to measure the same construct. At While it is common practice to use a total score on
the FIM, studies suggest that the FIM total score ditional assessments. Below some of the key con-
reflects both a motor and cognition-communication cepts to this framework are noted.
component (Linacre et al., 1994). Without separat-
ing these two constructs, improvement in an FIM The Measure Is Separate From the
score could reflect several combinations of events Instrument That Generates the Measure
(e.g., improvement in both motor and cognition,
improvement in motor but not in cognition, or First, it must be recognized that the measure is
improvement in cognition but not motor abilities). separate from the instrument used to generate the
Since a score does not reflect which of these or measure. Wright (1997b) describes the measure
other combinations of factors it is based on, it is as a theoretical construct and the unit of measure-
difficult to interpret a score. ment as a perfect idea. For example, in the physi-
Wright and Linacre (1989) note that no assess- cal sciences, distance is a theoretical construct.
ment can ever achieve perfect unidimensionality. Measuring with rulers is so commonplace in daily
This is even true for the physical sciences. life that we often do not recognize that our instru-
Nonetheless, Wright and Linacre (1989) argue that ment, the ruler, only estimates the perfect idea of
the ideal of unidimensionality must be approxi- distance.
mated in a measure. While there are many distances that are practi-
cal and measured on a daily basis, measured dis-
tances can range from the fraction of a micron that
Objective Measurement separates two electrons to the span between solar
systems that are light years apart. Moreover, there
in Occupational Therapy are literally thousands devices for measuring dis-
and Health Care tance ranging from micrometers that measure very
small distances to astronomical instruments that
capture distances in terms of the speed of light.
The Institute of Objective Measurement (2004) Despite the almost infinite ways of measuring dis-
defines objective measurement as follows: tance, it is relatively easy to compare or transpose
Objective measurement is the repetition of the units produced by all of these instruments. This
a unit amount that maintains its size, within is true even when the unit of measurement is based
an allowable range of error, no matter on a different scale. Often, measurement instru-
which instrument is used to measure the ments place different units of measurement on the
variable of interest and no matter who or same instrument (e.g., inches and centimeters on a
what relevant person or thing is measured. ruler). Otherwise, simple formulas or tables can be
(http://www.rasch.org/define.htm) readily used to translate measures across different
scales.
This definition points out several characteristics Once one recognizes that instruments and the
of measurement to which occupational therapy measures they represent are not one in the same,
instruments should aspire. The phrase repetition of one can take advantage of the pragmatic features
a unit amount that maintains its size refers to of objective measurement. For example, ADL, like
equality of intervals between units on the instru- distance, can be thought to be a theoretical con-
ment. It underscores the importance of having struct, a perfect idea. While the FIM represents
interval-based, not ordinal-based values as dis- possible ADL activities, these activities are lim-
cussed earlier. As stated in the definition, the unit ited. ADL can theoretically range from the ability
of measure should be independent of the instru- of lifting a facecloth to one’s face to fulfilling a
ment is used to measure the variable of interest. daily life role, such as being a worker. As there are
This means that measures must be test free (i.e., different rulers to measure the theoretical construct
that different instruments should be able to gener- of distance, different ADL instruments also should
ate comparable measures). Finally, the phrase no be able measure ADL. Some ADL measures could
matter who or what relevant person or thing is measure low levels of ADL ability while others
measured refers to measures being “sample free,” could measure higher levels of ADL ability.
or independent from the person or object being Furthermore, the theoretical construct of ADL
measured. connects the different ADL instruments. This
Creating measures in occupational therapy and opens up the possibility of translating measures
health care requires a different conceptual frame- generated by different ADL instruments and
work than that which has been used to develop tra- scales. Theoretically, like the measurement of dis-
tance, someday the ADL construct could become assist” on combing hair than achieving a “minimal
transparent. That is, it would be possible to refer to assist” rating on bathing. Said another way,
an amount of ADL ability independent of the bathing requires more ADL ability than hair brush-
measurement instrument used to generate the ADL ing. Figure 13.2 illustrates what the ADL assess-
ability measure. ment likely appears like when compared to the
theoretical continuum of ADL independence.
Items Represent Differing Amounts Viewing an instrument’s items in a hierarchical
arrangement, representing more or less of a con-
of the Intended Construct
struct, offers tremendous advantages in measuring
When using occupational therapy and healthcare constructs in occupational therapy and health care.
measures, it is important to recognize that the First, if a person is capable of challenging tasks,
questions or items on instruments do not represent there is a high probability that he or she will be
the same amount of the construct being measured. successful at accomplishing less challenging tasks.
When measuring distance, different parts of the In our example (Figure 13.2), if an individual is
ruler represent different amounts of distance. For capable of bathing or dressing, there is a high
example, objects that reach the 12-inch mark on a probability that he or she will be successful at
ruler are obviously longer than objects that reach brushing teeth or combing hair. For the technical
the 2-, 3-, or 4-inch mark. Measures in occupa- purpose of measuring ADL, it would be unneces-
tional therapy and health care can similarly be sary to assess lower amounts of the construct (e.g.,
conceptualized as representing more or less of the brushing teeth and combing) if the individual is
construct being measured. For example, ADL capable of performing items that represent greater
tasks represent different levels of complexity and amounts of the construct (e.g., dressing or
demand. That is, bathing and dressing arguably bathing). It should be noted, that this is in no way
involve more motor activity and more steps than to suggest that practitioners should limit their eval-
brushing teeth or combing hair. If this is so, it uation of relevant ADL activities. Nonetheless, for
should be easier to achieve a rating of “minimal purposes of measurement, it should be unneces-
High ADL Ability
Dressing
6
Bathing
5
4
3
Brushing teeth
2
Combing hair
1
Figure 13.2 Items on a hypotheti-

Actual Position Theoretical cal ADL independence instru-
of ADL Measurable ment compared to the
Independence Continuum of ADL Low ADL Ability theoretical continuum of ADL
Items and Ratings Independence independence.
Person
B
Easy Assessment
D1 D8 Score 6
Person
B
Hard Assessment
D1 D8
Score 2
Figure 13.3 Demonstration of scores for an individual (B) when taking an easy assess-
ment versus a hard assessment using CTT. D1 – D8 represents items of different diffi-
culty. One point is achieved for every item that B gets correct. While the individual’s
ability does not change across tests, his/her score is dependent on the test taken.
sary to assess all possible ADL activities in order When error is defined,2 the true score (T) is defined
to measure a client’s ADL ability. as the observed score (X) minus the error score.
As described earlier, precision in measuring
distance typically proceeds first with an estimate Test Dependency
of the length of an object followed by a focus on
The serious limitation of this model is that scores
the finer markings of the ruler to increase preci-
obtained with CTT are test dependent. That is, a
sion. Once items from occupational therapy and
respondent’s true score is dependent on the ques-
healthcare instruments are identified as represent-
tions or tasks of particular assessment on which
ing more or less of the construct they intend to
they are tested (Hambleton & Jones, 1993). On
measure, a process analogous to measuring dis-
more challenging assessments, respondents will
tance can take place. For example, in measuring
get lower observed scores and on easier assess-
ADL, once it is determined that an individual has
ments they will get higher observed scores (Figure
the ability to accomplish activities such as bathing,
13.3). Test dependency leads to challenges in
items of similar difficulty (e.g., dressing) or items
translating scores across similar assessments.
representing finer levels of the item (e.g., amount
of assistance, amount of time, etc.) can be asked to
Reliability
more finely discriminate the ADL abilities of this
individual. This procedure, in addition to being While internal consistency, test–retest, and inter-
efficient (e.g., avoiding questions far above or far rater reliability are necessary they are not suffi-
below the ADL ability of the individual), has the cient for objective measurement. All reliability
potential of achieving precision by focusing more statistics are sample dependent and instrument
detailed questions at the ability level of the dependent. For example, internal consistency, as
individual. determined by Cronbach alpha levels, is not only
dependent on the inter-correlations among items
but by the number of items (i.e., more items pro-
Classical Test Theory duce higher alpha levels) (Streiner & Norman,
1989).
Many of the limitations faced in achieving objec- Reliability statistics also have other limitations.
tive measurement in occupational therapy and Inter-rater reliability at times can mask the quali-
health care is a function of the mathematical theo- ties of a measure. For instance, instruments with
ries that traditionally underlie instrument develop- ceiling or floor effects can show high test–retest
ment and psychometrics. Test development in reliability despite the instruments’ failure to differ-
education and health care has been based on clas- entiate clients in the ceiling or floor. Reliability
sical test theory (CTT). The classical test model is 2Three assumptions are necessary: (1) true scores and error
a simple linear relationship that links the observed scores are uncorrelated, (2) average error score for a popu-
score (X) to the sum of the true score (T) and error lation is zero, and (3) error scores on parallel tests are uncor-
score (E), X ⫽ T ⫹ E (Hambleton & Jones, 1993). related.
statistics also are limited in their ability to reveal and health care, they are not required for objective
covariance in rater agreement. For example, raters measurement. For example, in the physical sci-
may show high correlations across their ratings ences the quality of the measurement of distance is
and therefore have high inter-rater agreement, but not dependent on norms. In fact, one can readily
they may systematically differ in the severity of see the absurdity of having to generate norms for
their ratings (e.g., one rater consistently rate heights or widths for all objects that could be
patients two points higher than another rater). potentially measured.
Systematic difference in ratings can have serious Thurstone (1928) notes that a measurement
implications in the scoring of individuals, that is, instrument must not be seriously affected in its
more lenient raters providing consistently higher measuring function by the object (or person) it is
ratings than more severe raters. measuring. Once an instrument has established
measurement qualities, such as precision, it can be
Validity used to produce norms if they are desired and use-
ful for some purpose. Although norms have prag-
Similar critiques can be levied on evaluating
matic utility (e.g., to determine if an item or person
instruments on the basis of validity criterion. While
being measured is within normal limits or if a child
predictive validity is commonly used as a means to
has reached a milestone) they are not an essential
evaluate the measurement qualities of instruments
feature of objective measurement.
in occupational therapy and health care, this
methodology would be highly suspect in evaluat-
ing measures in the physical sciences. For exam-
ple, to validate measures of distance (e.g., length of
Rasch Measurement
an object) as to whether or not they would predict and Item Response Theory
a single event, for example, whether an object
would fit through a doorway, would seem absurd. The solution to object measurement lies in item
Measures of distance have a myriad of uses beyond response theory (IRT) models and methodologies.3
this simple objective. In contrast to CTT, IRT focuses on the item rather
Furthermore, the prediction of any event is than the test. Three IRT models are prevalent in
never simply a function of the instrument under the literature and are described as the one-, two-,
study. For example, using the FIM to predict length and three-parameter models. All models relate
of stay (LOS) at first glance may seem logical person ability to one or more parameters. The one-
(reflecting the importance of ADL function to parameter model, often referred to as the Rasch
return home), but in reality, there are many factors model, includes item difficulty. The two-parameter
that influence LOS and there can be complex inter- includes both item difficulty and item discrimina-
relationships between these factors (e.g., secondary tion. Finally, the three-parameter model includes
conditions, exacerbation of a medical condition, item difficulty, item discrimination, and guessing.
medical capitation for inpatient services, etc.). In The focus of the remainder of this chapter is on the
summary, validity, like reliability, is not sufficient one-parameter, or Rasch model. In addition to this
for determining measurement quality. The exclu- being the most basic of the IRT models, from a
sive focus on concurrent, discriminant, and predic- technical perspective, it most closely reflects the
tive validity (all which follow the same logic) tenants of objective measurement as outlined
may actually obscure more fundamental qualities above.
of objective measurement such as efficiency and
precision. Rasch Model
In 1953, the Danish mathematician George Rasch
Norms (1960) generated solutions to Thurstone’s meas-
One of the psychometric characteristics often con- urement requirements with the simple relationship
sidered critical for the acceptance of a healthcare of person ability (B), item difficulty (D), and the
instrument is the availability of norms. That is, the probability of a correct solution (P). The most
quality of an instrument is considered to depend on basic version of the Rasch formula is presented
whether or not its values are connected to averages below (Velozo, Kielhofner, & Lai, 1999).
of a particular population (e.g., children of a given 3
Fredrick Lord is often credited as the father of IRT, with his
age, males or females, individuals with a particular observation that ability scores, in contrast to true scores,
impairment, and so on). While normative samples are more fundamental to measurement since they are test
can be extremely useful in occupational therapy independent (Hambleton & Jones, 1993; Lord, 1953).
Fundamental Objective Measurement
Wright (1997a,b) notes that much of the critical • Sample free calibration—a measuring instrument
literature on fundamental, objective measure- must not be seriously affected in its measuring
ment has been ignored by classic test theory. function by the object being measured
Nonetheless, its beginnings can be traced to (Thurstone, 1928).
Normal Campbell’s (1920) deduction that funda- • Test-free measurement—it should not be required
mental measurement requires the possibility of to submit every subject to the whole range of the
physical “concatenation,” like the joining of sticks scale (Thurstone, 1926).
to concatenate length or piling of bricks to con-
catenate weight (Campbell, 1920; Wright, 1997a). In addition to Thustone’s requirements for
Louis Thurstone, in the late 1920s and early 1930s measurement, another critical development
outlined the following requirements of useful was Guttman’s (1950) concept of “conjoint
measurement: transitivity.” That is, the concept of a total raw
score can be clarified only by specifying the
• Unidimensionality—the universal characteristic response to every item or question in a test or
of all measurement is that it describes only one assessment. He proposed that if a person endorses
attribute of the object measured (Thurstone, or passes a challenging (hard) item, he/she
1931). should endorse all less challenging (easy) items.
• Linearity—measurement implies a linear contin- While this concept is critical to objective measure-
uum, for example, the qualitative variations of ment, Rasch (1960) replaced the Guttman’s deter-
scholastic achievement must be forced into a ministic model (i.e., the requirement that all less
linear scale (Thurstone & Chave, 1929). challenging items must be “passed” and all more
• Abstraction—the linear continuum implied in all challenging items must be “failed”) with a proba-
measurement is an abstraction, not the measuring bilistic model of passing and failing (i.e., very
device (Thurstone, 1931). easy items have a high probability of being
• Invariance—the unit of measurement can be passed, items at the persons ability level have a
repeated without modification in the different 50% probability of being passed, and very chal-
parts of the measurement continuum (Thurstone, lenging items have a high probability of being
1931). failed).
Log [Pni/1 ⫺ Pni] ⫽ Bn ⫺ Di vidual’s ADL ability is low relative to “brushing

where hair,” he or she will have a high probability of fail-
Pni ⫽ probability of person n passing item i, ing that item. When an individual’s ability matches
1 ⫺ Pni ⫽ probability of person n failing item difficulty, there is a 50% probability of pass-
item i, ing the item.
Bn ⫽ ability of person n, and The relation of person ability to item difficulty
Di ⫽ difficulty of item i. is an accurate representation of the process of
The left side of the formula represents the lin- measuring, even in the basic sciences. For exam-
ear transformation of the probability of passing a ple, if we consider parts of the ruler as “items” of
particular item divided by the probability of failing different difficulty, we can replicate the operation
a particular item. The logarithmic transformation of the Rasch formula. When measuring an object
is critical in the creation of linear, interval-based approximately 31/2 inches in width, the object has a
measures. “high probability of passing” (i.e., exceeding) a
The core of the Rasch formula is the relation- marking of 1 inch, 2 inches, and 3 inches. That
ship of person ability (Bn) to item difficulty (Di). object also has a “high probability of failing” (i.e.,
The attractiveness of this simple formula is that it falling short of) markings of 6 inches, 5 inches,
captures the fundamental element of all testing, the and 4 inches. As one attempts to determine the
relationship of a person to the assessment item. exact width of an object, one must have less
That is, the most basic description of any assess- confidence in the correctness of our measurement.
ment situation is whether or not the person is suc- That is, one can be confident that the object is at
cessful in passing the item that he or she is least 1, 2, or 3 inches in width, and not 4, 5, or 6
attempting. Consider for instance, the ADL item inches. However, one cannot be as confident that
“brushing hair.” If an individual’s ADL ability is the object is exactly 31/2 inches. It could be 3 and
15
high relative to “brushing hair,” he or she will have /32 inches or 3 and 17/32 inches. Ironically, as one
a high probability of passing that item. If the indi- approaches the calibrations closer to the true width
of the object, one has a 50% probability of being raters, multifaceted Rasch analysis can determine
exactly correct. the relative severity of raters and correct for this
It is interesting to note that the above measure- severity in determining a person’s true measure.
ment process typifies the strategy a therapist may
use when interviewing new clients about their
Rasch Measurement Model Statistics
functional ability. First questions may be off target.
That is one may ask about ADL tasks that are far Rasch measurement statistics, while having some
above or far below the person’s ADL ability level. analogs in traditional psychometrics, differ owing
If one receives affirmative answers to the questions to their focus on analysis at the person and item
asked about ADL independence, then one moves level versus the test level (Table 13.2). In contrast
on to more difficult ADL tasks and ask about them. to traditional psychometrics, Rasch provides
If one receives negative answers, one moves to detailed information on rating scales, items, per-
easier ADL tasks. Eventually, as when one is ask- sons, and other factors (called facets) such as rater
ing about ADL tasks that are nearer the actual abil- severity. These statistics provide an analysis of
ities of the individual, one starts getting less clear critical objective measurement features of assess-
answers, such as “I think I can do that.” Once the ments.
ADL tasks one asks about are at the ability level Rasch analysis first converts ordinal data gener-
of the individual, there is a 50% probability of the ated by dichotomous or rating scale data into inter-
individual answering similar ADL questions in val measures called log equivalent units (logits)
either the affirmative or negative. (Merbitz, Morris, & Grip, 1989; Wright &
Slightly more advanced Rasch formulas can Linacre, 1989). Logits are based on the natural
take into account more complicated assessment log of getting a correct response and have the char-
situations. For example, instead of the dichoto- acteristic of interval data by retaining the same
mous situations described above, rating scales mathematical length throughout the continuum of
can be analyzed. Rasch the variable (Linacre &
rating scale models allow Wright 1989; Smith,
consideration of more The core of the Rasch for- 1992). The logit scale is
detailed information mula is the relationship of usually set at the average
about the level of inde- difficulty level of the
pendence (i.e., 1 ⫽ person ability (Bn) to item items of the particular
unable to do, 2 ⫽ needs difficulty (Di). The attractive- assessment. Person abil-
maximal assistance, 3 ⫽ ity, item difficulty, and
needs minimal assis- ness of this simple formula facets such as rater
tance, 4 ⫽ independent). is that it captures the funda- severity are all expressed
To analyze this situation, in the same unit of meas-
as an alternative to inves-
mental element of all testing, urement, the logit.
tigating the probabilities the relationship of a person Person measures, or
of a person passing or to the assessment item. calibrations, are gener-
failing an item, the for- ated by estimating the
mula determines for position of each person
individuals of different abilities the probability of assessed on the continuum from less to more of the
passing or failing an item at a particular rating underlying construct. Persons with higher meas-
(e.g., whether or not the individual with low abil- ures are those persons with more of the construct
ity passes combing hair at the level of needing being measured and vice versa. In the case of an
minimal assistance). ADL scale, a person’s calibration is the measure of
Similarly, many-faceted Rasch formulas can how much ADL ability the person has.
take into account rater severity. That is, while Item measures, or calibrations, indicate how
raters may be consistent in their ratings, there is a much of the underlying construct an item repre-
high likelihood that some raters have a tendency to sents. The order in which items of a scale are cali-
be more severe or more lenient than other raters. brated is also important for assessing the validity
With CTT scoring methods, clients that are rated of a scale. Items with higher calibrations should be
by more severe raters will get lower scores while those that are expected to represent more of the
clients rated by less severe raters will get higher construct measured. Similarly, items calibrated
scores, independent of their true ability levels. By lower should be those expected to represent less of
comparing the probabilities of ratings across the characteristic. For example, on an ADL scale
Table 13.2 Rasch Statistics*

Comparable Explanation of
Statistic Name Traditional Statistic Rasch Statistic Desired Values
Person Measure Person total raw score Logistic transformation of the Can range from ⫺' to ⫹'. No
person’s raw score particular desired values
(logit)—represents person though the distribution of
ability person measures should
overlap the distribution of
item measures (see
person–item match below)
Item Measure p-value—proportion of Logistic transformation of the Can range from ⫺' to ⫹'. No
the sample that p-value (logit) particular desired values
passed the item though the distribution of
item measures should
overlap the distribution of
person measures
Item Item discrimination Proportional to the slope of Rasch model holds item
Discrimination index—difference the function of the proba- discrimination constant at
between the bility of passing an item 1.0
proportion of high relative to person ability. Two- and three-parameter
and low scores The Rasch model holds models—high positive
passing an item this constant for all items values (low positive and
while two- and three- negative values should be
parameter models of concern) (Hambleton &
calculate a value for each Jones, 1993, p. 45)
item
Item or Person Fit For item fit—point Mean square residual MnSq—ideal is 1.0.
biserial (MnSq)—ratio of Values ⬎ 1.0 indicate
- Infit–inlier-sensitive correlation— observed scores to unpredictability (unmodeled
or information- correlation of the expected scores— noise, data underfitting the
weighted fit item to rest of amount of distortion of measurement model)
the test the measurement system Values ⬍ 1.0 indicate over
- Outfit–outlier- No analog for person Standardized mean square predictability (Redundancy,
sensitive fit fit. (ZSTD) —conversion of data overfitting the
MnSq to ␣ normally measurement model)
distributed standardized Recommended ranges for
score surveys is 0.6–1.4 and for
clinical assessments is
0.5–1.7 (Wright &
Linacre,1994)
ZSTD—ideal value is 0.0. Less
than 0 indicates too
predictable; greater than 0.0
indicates to erratic.
Acceptable ranges are
between –2.0 and 2.0. Note
high n-sizes result in
inflated ZSTD
Reliability Cronbach alpha or Separation reliability—ratio of Similar in estimate to Cronbach
Kuder-Richardson unbiased sample alpha or K-R 20—
20 (K-R 20)—an standard deviation to satisfactory values are
indication of average standard error of between 7.0 and 8.0 (Bland
internal the test (Fisher, 1993) & Altman, 1997)
consistency
analogous to doing
multiple split-half
reliability tests
(continued)
187
Table 13.2 Rasch Statistics* (continued)

Comparable Explanation of
Statistic Name Traditional Statistic Rasch Statistic Desired Values
Person Separation Standard deviation of Person separation—person In general, person separation
and Person distribution of raw spread divided by error over 2. Technically, person
Strata scores Person strata—number of strata over 2 means the
statistically distinct strata instrument divides the
⫽ (4Gp ⫹ 1)/3, where Gp sample into at least two
⫽ person separation statistically distinct strata
(Wright & Masters, 1982,
p. 106)
Person-Item Match Proportion of Comparison of person Typical distribution
individuals passing measure distribution to comparisons (e.g., visual
or failing items item measure distribution comparison of distributions,
person and item means
within 2 standard
deviations)
*Adapted from Fisher (1993).
we would expect dressing to be calibrated higher weighted fit, is more sensitive to the pattern of
than combing hair. responses targeted on the items or persons. For
Since items and persons are calibrated on the example, persons showing high infit are demon-
same continuum, one can determine whether items strating an unexpected pattern of responses on
are appropriately targeted to the levels of the trait items that are most reflective of their ability level.
represented by the client under study. In general, Outfit, or outlier-sensitive fit, is more sensitive to
an instrument should have items whose mean is responses far from the persons or items. For exam-
near the mean of the clients. Furthermore, the ple, a person showing high outfit is demonstrating
items should spread out so that they cover the an unexpected pattern on items that are easy or
range of variation represented by the clients, and hard for the individual.
thus avoid ceiling and floor effects. A scale with- Fit mean-square (MnSq) values represent the
out higher-level items to measure higher-level per- observed variance from the data divided by the
sons and lower level items to measure lower level expected variance estimated by the Rasch mathe-
persons will have ceiling and/or floor effects (i.e., matical model. The ideal MnSq ratio is 1.0,
everyone above or below a certain level will get whereby the observed variance from the data
the same score despite differences between them equals the expected variance from the Rasch
in the construct being measured). model. MnSq illustrates the extent to which the
Rater measures, or calibrations, indicate how item, person, or rater fit the expectations of the
severe or lenient a rater is when assigning scores model. MnSq above 1.0 indicates the item, person,
on the scale. A rater with a higher calibration is or rater is increasingly erratic (i.e., observed
more severe in assigning ratings and a rater with a variance is higher than expected variance) while
lower calibration is more lenient in assigning rat- MnSq below 1.0 represents data that is overly pre-
ings. Ideally, differences in raters’ severity should dictable. High MnSq are taken to indicate misfit
be small, so that the measure a person receives is (i.e., that an item was not a valid indicator of a con-
not greatly impacted by who did the rating. When struct, that a client was not validly measured,
this is not the case, rater calibrations can be used or that a rater was not using the scale in a valid
to correct person measures for rater severity manner).
(Fisher, 1993; Forsyth, Lai, & Kielhofner, 1999; Low MnSq below 0.5–0.6 indicates that the
Lai, Velozo, & Linacre, 1997). item, person, or rater shows too little variability or
The most critical statistic in Rasch analysis is that the scoring pattern is too deterministic. Wright
the fit statistic (Haley, McHorney, & Ware, 1994; and Linacre (1994), recommend MnSq ranges for
Wright & Stone, 1979). Fit statistics are generated surveys to be 0.6–1.4 and for clinical observations
for persons, items, and raters. Rasch analytic pro- to be 0.5–1.7. High-fit statistics are generally of
grams usually separate fit statistics into infit and more concern because they represent a greater
outfit. Infit, or inlier-sensitive/information- threat to the validity of the assessment.
Fit statistics can be useful in identifying a vari- lead to pragmatic solutions to measurement chal-
ety of aberrations in the assessment process. High lenges in health care.
person fit may indicate a survey respondent who is
providing false information. It also can be diag- Analyzing Existing
nostic; for example, a person performing unex-
Healthcare Instruments
pectedly poorly on an ADL item which for most
persons is easy (e.g., eating) may be indicative of Many researchers have begun to use IRT method-
a particular problem (e.g., difficulty with swallow- ologies to reexamine instruments that were origi-
ing). Item misfit can be an indication of a poorly nally developed following CTT. For instance, IRT
worded item or can be an indication of multidi- methodologies have been used to investigate the
mensionality in an assessment.4 Individual erratic item-level characteristics of existing instruments.
items can be removed from an assessment. A rater Rasch analyses of the FIM (Linacre et al., 1994),
showing high fit statistics may be an indication Patient Evaluation Conference System (PECS;
that he or she is misinterpreting the scoring crite- Silverstein, Fisher, Kilgore, Harley, & Harvey,
rion. This could lead to removing that rater’s data 1991) and the Level of Rehabilitation Scale
from a dataset. It also can mean that the rater (LORS-III; Velozo, Magalhaes, Pan, & Leiter,
requires further training to use the assessment 1995) have produced parallel results that provide
correctly. new insights into these assessments.
Rasch statistical programs also produce statis- Rasch analysis item fit statistics indicate that
tics that are analogous to those used in traditional the items of these instruments are multidimen-
psychometrics. Person separation reliability is sional. That is, the items of global functional status
analogous to Cronbach’s alpha (Bland & Altman, instruments appear to represent more than one
1997) and similarly serves as a measure of internal underlying latent trait. For example, Velozo and
consistency. Person separation represents the num- colleagues (1995) showed that when all the items
ber of levels into which subjects and raters are of LORS are analyzed together, items representing
classified by their calibrations. When persons are ADL (e.g., eating, grooming, dressing, etc.) show
well separated, the scale effectively discriminates good fit statistics while items representing the cog-
different levels of the trait being measured. When nition/communication (e.g., memory, expression,
applied to persons, the separation statistic repre- problem solving, etc.) showed unacceptably high
sents the sensitivity of the scale to detect differ- fit statistics. When the items representing each of
ences in subjects. On the other hand, rater these constructs are analyzed separately, they show
separation represents the extent of rater bias due to acceptable fit statistics. These findings and find-
severity/leniency. ings similarly found in the Rasch analysis of the
FIM (Linacre et al., 1994) led both research groups
to conclude that these global functional status
Applications of Item instruments measure two unique constructs. The
Response Theory in above findings have now led to the practice of
scoring FIM ADL and cognition/communication
Health Care separately when reporting clinical outcomes (e.g.,
Bates & Stineman, 2000).
IRT and the associated methodologies provide a Item calibrations produced from the analysis
revolutionary framework for the investigation of also provided insights on how well these instru-
existing instruments and development of new ments were measuring the samples under study. As
instruments. This capability emerges from the abil- noted earler, in addition to producing both item dif-
ity to analyze at the item level instead of the test ficulty measures and person ability measures,
level. While CTT methodologies provide general Rasch analysis places these measures on the same
indications of problems or limitations of existing linear continuum. This item person relationship
instruments, IRT methodologies provide more reveals not only the presence of ceiling or floor
detailed information on the potential cause of the effects, but also the relationship of instrument
limitation. In combination with clinical experi- items to these effects. This is illustrated on Figure
ence, analysis of instruments at the item level can 13.4, which shows both items and person measures
on the same graph for the LORS. Velozo and col-
4Fit
leagues (1995) used this methodology to investi-
statistics alone are not adequate in identifying multidi-
mensionality. Even when all items fit the Rasch model, fac- gate the effectiveness of the ADL items of the
tor analysis can reveal multiple constructs (Linacre, 1998; LORS in measuring patients at admission and
Smith, 2002). discharge. An analysis of 3,056 rehabilitation inpa-
ADMISSION DISCHARGE
7 PERSONS MAP OF ITEMS 8 PERSONS MAP OF ITEMS

<more> | <rare> <more> | <rare>
.# .############
. S
6 . 29 patients 7
T .######
. .
. .
996 patients
5 . 6 .
. .#######
.
.
.## .####
4 5 .
.# M
. S .####
.## .###
.## .
3 .##### 4 .###
.###### T .##
.##### .
.######## .####
2 .###### 3 .###
.######### M .###
.######## S . S
.######## .##
1 .########## 11-V12-MOBILITY-RN 2 .## T
.######## .#
.####### 1-V1-ADL-DRESSING-N V-V5-ADL-WASHING-N .#
.####### M 6-V6-ADL-WASHING-T 7-V7-ADL-TOILETING-N .#
0 .############ 12-V13-MOBILITY-PT 2-V2-ADL-DRESSING-T 1 .#
.##### S 8-V8-ADL-TOILETING-T .# 5-V5-ADL-WASHING-N
.###### .# S 6-V6-ADL-WASHING-T
11-V12-MOBILITY-RN
.##### 3-V3-ADL-TOILETING-T . 1-V1-ADL-DRESSING-N 7-V7-ADL-TOILETING-N
-1 #### S 0 #
.#### .# T 12-V13-MOBILITY-PT 2-V2-ADL-DRESSING-T
.####### . M 8-V8-ADL-TOILETING-T
.### .
-2 .### 4-V4-ADL-GROOMING-T . 3-V3-ADL-GROOMING-N
9-V9-ADL-FEEDING-N -1
.## T T .
.## . S
.## .
.##
-3 .## 10-V10-FEEDING-T -2 . 4-V4-ADL-GROOMING-T 9-V9-ADL-FEEDING-N
.# .
.# .
.# . T 10-V10-V10-ADL-FEEDING-T
-4 . -3 .
. .
. .
. .
-5 . -4 .
. .
.
. .
-6 -5 .
.
. .
.
-7 -6
. .
-8 -7
.#
.
<less> | <frequ> <less> | <frequ>
EACH '#' IS 17. EACH '#' IS 17.
Figure 13.4 Person ability–item difficulty match at admission and discharge for the Level of
Rehabilitation Scale–III. (Adapted from Velozo et al., 1995.)
tients with admission and discharge measures showed a distinct pattern. Items representing feed-
revealed that while there was no ceiling effect at ing and grooming, were easy and items represent-
admission (see left-hand side of Figure 13.4), 32% ing washing (bathing) and mobility (locomotion)
of the inpatients were in the ceiling (i.e., above the were the most difficult. Therefore, elimination of
highest calibrated items) at discharge (see right- the ceiling effect at discharge could likely be
hand side of Figure 13.4). That is, 32% of the inpa- achieved by including more challenging ADL
tients received the maximum measure on the items such as carrying items while walking or
instrument. Even more dramatic were the ceiling climbing several flights of stairs.
effects for the cognition/communication scales of The comparisons of item difficulties to person
the LORS demonstrating 35.5% of the patients in abilities also provide some evidence why global
the ceiling at admission and 49.9% of the patients functional measures would not be effective in
in the ceiling at discharge. measuring outcomes for outpatient rehabilitation.
Further, item-based analyses of the LORS pro- Individuals discharged from inpatient rehabilita-
vided insights on how to eliminate these effects. tion are likely to be competent with basic ADLs.
The ADL item difficulty hierarchy for the LORS This does not suggest that these individuals are
fully rehabilitated. More challenging items, such environmental challenge to the items, for example,
as more strenuous locomotor tasks or instrumental “climbing well-lit stairs,” “climbing poorly lit
activities of daily living would more likely differ- stairs,” carrying bundles up well-lit stairs,” carry-
entiate persons discharged from inpatient services. ing bundles up poorly lit stairs.”
Furthermore, these findings would suggest the Rasch analysis of 106 community dwelling
need for further rehabilitation. respondents revealed only 1% of the sample in the
floor of the instrument, and 7.5% in the ceiling.
In general, the resultant item calibrations verified
Development of New Instruments
the construct validity of the instrument. The easi-
While IRT models are useful in analyzing existing est items in the instrument were “getting in/out of
healthcare instruments, their true value may be bed,” “get on/off toilet,” and “get dressed” while
in the development of new instruments. Well- the most challenging items were “walk outside
known assessments used by occupational thera- when it was icy,” “carry bundles up poorly lit
pists that have been developed though Rasch stairs,” and “use a step stool to reach in kitchen
analysis include the: Assessment of Motor and cabinet.” Furthermore, as hypothesized, common
Process Skills (Fisher, 1993), Assessment of items with differing environmental challenges cal-
Communication and Interaction Skills (Forsyth, ibrated in the expected hierarchy (e.g., carrying
Salamy, Simon, & Kielhofner, 1998), Pediatric bundles up poorly lit stairs was more difficult than
Evaluation of Disability Inventory (Haley, Coster, carrying bundles up well-lit stairs).
Ludlow, Haltiwanger & Andrellos, 1992), Occu- In addition to providing evidence of construct
pational Performance History Interview (OPHI) validity, the ability to place person measures and
(Kielhofner et al., 2004), and Worker Role Inter- item calibrations on the same linear continuum
view (Braveman et al., 2005). provides a critical link between the qualitative con-
In contrast to instruments that have been devel- tent of the instrument and the measures produced
oped using CTT models, instruments developed by the instrument. Following Rasch analysis, each
using IRT models are more likely to incorporate person ability measure becomes connected with
item difficulty when measuring individuals. By an item calibration. Velozo and Peterson (2001)
doing this, researchers are able to test the underly- demonstrated this connection between person
ing theory or logic of the item hierarchy, thus crit- measures and the qualitative content of the UIC
ically examining the validity of the instrument in Fear of Falling Measure. For example, the average
measuring the underlying construct. Instruments measure of the sample they studied was associated
developed to systematically reflect an item hierar- with being moderately worried about standing on a
chy are less likely to demonstrate ceiling and floor moving bus and carrying objects up well-lit stairs.
effects and are more likely to be sensitive in detect- These investigators postulated that this connection
ing differences between those being assessed. Most between the quantitative values could be used to
important of all, Rasch analysis provides detailed determine clinical significance, i.e., an individual
information that can provide insights into how to showing adequate improvement on the fear of
improve the instruments under development. falling scale so that they were “not fearful about
For example, Velozo and Peterson (2001) used walking on a crowded street” could be a criterion
Rasch measurement as a basis to develop the for an individual living in an urban setting.
University of Illinois (UIC) Fear of Falling
Measure. The purpose of this measure was to iden-
tify and differentiate fear of falling among commu-
Innovative Measurement
nity dwelling elderly. Velozo and Peterson (2001) Methodologies Based on
used Stone’s (1997) method of item development, Item Response Theory
whereby fear of falling was expressed as a variable
In addition to its utility as a means of developing
representing a range of items that would elicit fear
assessments, the use of the Rasch measurement
in the most fearful individuals (e.g., “getting out of
model has also opened up other possibilities for
bed,” “getting on/off a toilet”) to items that would
advancing measurement methodologies. Below we
elicit fear only in the least fearful individuals (e.g.,
discuss three of these advances:
“using an escalator” and “walking outside when it
was icy”). Furthermore, middle-difficulty items • The development of keyforms for instantaneous
were developed such as “walking on a crowded measurement and quality control,
street” and “climbing stairs.” More subtle degrees • Computer adapted testing, and
of item difficulty were generated by incorporating • Linking existing measures.
Development of Keyforms for cal presentation the essential steps of data collec-
Instantaneous Measurement tion and measurement construction” (Linacre,
and Quality Control 1997, p. 316). While the keyform does require
Rasch analysis (commonly done by the instrument
Despite the advantages of Rasch-based measures developer), once the keyform is created, Rasch-
in generating interval measures versus ordinal based interval measures can be generated directly
scores, use of the former in everyday occupational from the form without further computer analysis.
therapy practice is still limited. One obvious bar- The first example of keyforms for an occupational
rier is that raw data must be converted into Rasch- therapy instrument was developed for the second
based interval measures. With few exceptions version of the Occupational Performance History
(e.g., the Assessment of Motor and Process Skills, Interview (OPHI-II) (Kielhofner et al., 2004).
Fisher, 1993), methods of generating interval Prior to developing the keyforms, research using
measures via computer scoring are not readily the Rasch measurement model supported the con-
available to practitioners. Therefore, even when clusion that the three OPHI-II scales were reliable
using assessments that were originally developed and valid measures. The OPHI-II keyforms allow
using the Rasch measurement model, practitioners generation of instantaneous measures from raw
ordinarily do not have available means to generate rating scale data, while also exercising intuitive
Rasch-based interval measures. quality control of the data obtained (Kielhofner,
Linacre (1997) first proposed an alternative to Dobria, Forsyth, & Basu, 2005; Kielhofner,
using computers to obtain interval measures. He Mallinson, Forsyth, & Lai, 2001).
introduced the idea of a keyform, that is, a paper- The basis of a keyform is that there is a relation-
and-pencil method that “combine into one graphi- ship between raw scores and interval measures.
Completing the OPHI-II Keyforms
Below are the instructions for completing the also useful to circle the numbers in order to
OPHI-II Keyforms. The reader should refer to inspect what the pattern reveals. There are two
the keyform for the Occupational Identity Scale things that can be readily discovered. First, if
(Figure 13.5). The process for completing all someone’s pattern of ratings are all or mostly
three OPHI-II keyforms is identical. at the extreme (i.e., all or mostly 1’s or 4’s),
then the person has likely demonstrated a
When All Items Are Rated: ceiling or floor effect—that is, he or she has
either more or less identity then this scale
1. Turn the OPHI Key on its side (landscape format). could measure.
2. Record the ratings in the leftmost column
marked “rate client here.” Obtaining a Measure With Missing Data
3. Calculate the sum of the ratings and record on
the total score line. 1. If you have not scored all the items, then the
4. Turn the form back upright (portrait format). procedure is as follows:
5. Locate the total score you obtained in the first 2. Turn the OPHI Key on its side (landscape for-
(left) column of the box at the right, marked mat).
total score. 3. Circle all the items you DID rate.
6. Now, look at the corresponding numbers in the 4. Draw a line across the body of the key form, by
two following columns. The first number (mid- eye, through the average of the ratings.
dle column) is the client measure—it is based on 5. The line will intersect a client measure and
a 100-point scale where 0 is the least amount of corresponding standard error, as shown in
occupational identity and 100 is the most occu- example 2, in the box immediately below the
pational identity that can be captured with this key.
rating scale. The second number is the standard 6. Turn the form back upright (portrait format).
error of that the measure. 7. Look at the numbers intersected by your line in
7. Record the client measure and standard error the left box, immediately below the key. The left
numbers in this box onto the lines titled client column represents client measure, and the right
measurement and standard error. column standard error.
8. Record the numbers intersected by the line you
Examining the Pattern of Scores drew onto the lines titled client measurement
In addition to adding up the scores and obtaining and standard error.
the client measure from the raw total score, it is
Raw scores can be used to estimate interval meas- Computerized Adaptive Testing (CAT)
ure in two ways. When all items are rated, the sum
of the raw scores estimates the measure. At first One of the most dramatic advances in modern
blush, this statement may seem to contradict the measurement is the combination of IRT with com-
challenges identified with adding up raw scores. puter technology to develop computerized adap-
However, if a person measured fits the expectations tive tests (CAT). As proposed by Thurstone (1926)
of the Rasch model, in a Rasch-based instrument and described earlier in this chapter, a distinct
such as the OPHI-II, raw scores reflect the position characteristic of measurement is that it eliminates
of items on the unidimensional continuum. the need to use all the items of an instrument to
Increasingly higher sums of the raw scores are derive a measure. The relevance of this feature of
achieved through higher ratings on increasingly measurement in health care is demonstrated in
higher-level items. Thus, when the person’s ratings Figure 13.6 on p. 195. Portrayed is a continuum of
conform to that expected pattern, the sum of the physical function represented by “easy” items such
raw score ratings can be readily converted to linear as “sitting balance,” “grooming,” and “upper
measures via a keyform table. extremity dressing” on the left of the continuum,
An important step in completing the keyforms and “difficult” items such as “jogging,” “biking 10
is to visually determine whether the pattern of rat- miles,” and “running 10 miles” to the far right.
ings conforms to these expectations. Referring to Logically, individuals of different abilities should
Figure 13.5 on p. 194, the OPHI Identity Scale be tested with different items along the continuum.
keyform, the expectation is that there is a higher For example, an individual with severe multiple
probability in getting higher ratings on the lower- sclerosis would be likely be tested with less chal-
level items (e.g., “Made occupational choices lenging items such as “sitting balance,” and an
(past)”) and lower ratings on the higher-level items individual with a moderate level cerebrovascular
(“Has personal goals and projects”). If the ratings accident tested with more challenging items such
violate these expectations (e.g., unexpectedly low as “toilet transfers” and “ambulation. An individ-
ratings on a lower-level items or unexpectedly ual with mild osteoarthritis would likely be tested
high ratings on a higher-level item), a judgment with even more challenging items such as “jog-
must be made about whether the anomalous pat- ging” and “biking 10 miles.”
tern means the assessment was valid for the client While IRT statistics can provide the item
in question. If the judgment is that the pattern is calibrations to identify the items that are most
not valid, all of the client’s data can be disregarded appropriate for an individual of a particular ability,
or just the anomalous rating could be treated as paper-and-pencil instruments are not a practical
missing data in deriving a measure. When the rat- means of administering these items. Personal com-
ings are as expected, they can be summed to derive puters, on the other hand, readily provide the tech-
a total raw score for which the keyform will pro- nology to selectively administer items to an
vide a corresponding person measure along with individual depending on the answers or perform-
its standard error. When some items are not rated a ance on items previously administered to the indi-
total raw score cannot be calculated. However, the vidual. An algorithm for the administration of
remaining items can be used to estimate the person items via a computerized adaptive test (CAT) is
measure. In this case, a visual representation of the presented in Figure 13.7 on p. 195. The algorithm
relationship between raw scores and measures is starts with setting an initial estimated person
used (see Figure 13.5). The keyforms combine measure and then items are directed to an individ-
both these features together. ual that are close in calibration to the last cali-
Keyforms, therefore, allow therapists to com- brated person measure. The stopping rule can be
bine the steps of data collection, quality control, based on a confidence interval or amount of preci-
and measurement into a single process. By elimi- sion set by the administrator. Once the desired con-
nating the necessity to computer score every fidence interval is achieved, the respondent
assessment, keyforms provide the therapist with receives a measure for the construct and the algo-
Rasch-based interval measures and provide a rithm gets initiated for the next construct to be
means of deriving measures when there is missing measured. The program stops when all constructs
data. While keyforms are available only for two have been measured.
occupational therapy assessments as of this writing Extensive development of CATs took place in
(keyforms are also available for the Occupational the 1980s (Martin, McBride, & Weiss, 1983;
Self-Assessment, Baron et al., 2006), their avail- Olsen, Maynes, Slawson, & Ho, 1986; Weiss,
ability and use is likely to grow because of their 1985), and by 1994 a variety of CATs were being
ease in deriving interval measures. administered in the field of education. In spite of
Rate Client
Here
194
Occupational Identity Key
Has personal goals
and projects 3 1. . . . . . . . . . . 1 2 3 4 . . . . . .4
13Kielhofner(F)-13
Identifies a desired
3 1. . . . . . . . . 1 2 3 4 . . . . . . . .4
occupational lifestyle
Expects success 2 1. . . . . . . . . . . 1 2 3 4 . . . . . .4
5/5/06
Accepts responsibility 3 1. . . . . . . .1 2 3 4 . . . . . . . . .4
Appraises abilities
and limitations 3 1. . . . . . . . 1 2 3 4 . . . . . . . .4
3:47 PM
Has commitments
and values 3 1. . . . . .1 2 3 4. . . . . . . . . . . 4
Recognizes identity
? 1. . . . . . . . . 1 2 3 4 . . . . . . . .4
and obligations
Has interests 1. . . . . . . . . 1 2 3 4 . . . . . . . .4
Page 194
3
Felt effective (past) 1 1. . . . . . 1 2 3 4. . . . . . . . . . .4
Found meaning and
satisfaction in lifestyle (past) 1 1. . . . . . . 1 2 3 4 . . . . . . . . .4
Measure client here. Circle ratings and draw line
Made occupational
1 1. . . . . . . . 1 2 3 4 . . . . . . . . .4
choices (past)
2
8
4
6
0
Cl
12
18
52
58
82
88
92
98
14
16
54
56
84
86
94
96
22
28
32
38
42
48
62
68
72
78
24
26
34
36
44
46
64
66
74
76
90
20
40
50
60
80
70
10
30
Total Score
100
ien
tM
ea
6
8
3
3
6
5
5
5
5
5
5
4
4
4
4
4
4
3
4
4
4
4
4
4
4
4
4
4
8
5
su
13
13
St re
an
da
rd
Er
ror
To
43
15
16
18
19
20
22
23
24
25
26
27
28
29
30
31
33
35
36
37
38
40
41
42
44
11
12
13
14
17
21
32
34
39
ta lS
co
re
Client Measure 42
0
9
91
21
23
27
29
31
34
36
38
39
41
43
45
47
49
52
58
64
67
70
73
78
81
85
15
18
26
33
55
61
75
Cl
100
ien
tM
ea
su
8
4
4
4
4
4
3
3
3
4
4
4
4
4
4
4
5
5
4
4
4
5
5
6
8
6
5
4
3
5
5
4
Standard Error St
13
an re
13
da
rd
Er
ror
Figure 13.5 OPHI-II Keyform.

INJURED
INPATIENT WITH
WEEKEND
MODERATE CVA
ATHLETE
Grooming Toilet Transfers Climb Stairs Biking 10 Miles
Sitting UE Dressing Ambulation Jogging Running

Balance 10 Miles
INPATIENT OUTPATIENT
WITH WITH MILD
SEVERE MS OSTEOARTHRITIS
Figure 13.6 Hypothetical continuum of function with associated items in hierarchical order
of difficulty from left to right. Framed boxes relate where on the continuum each of these
clients could be located and the items most appropriate for testing these clients.
the relative widespread use of CAT in educational development in the area of rehabilitation and
testing, its use in health care is in its infancy. Cella disability. Haley and his colleagues are presently
and his colleagues have developed CATs to moni- field testing their Activity Measure for Post Acute
tor quality of life outcomes in oncology (Webster, Care (AM-PAC) (Haley, Coster, Andres, &
Cella, & Yost, 2003) and Ware and colleagues have Kosinski, 2004) and Velozo and his colleagues are
developed a CAT to assess the impact of headaches presently field testing the ICF Activity Measure, a
(Ware, Cella, & Yost, 2003; Ware et al., 2003). self-report assessment of physical disability
At present, we are aware of two CATs being (ICFmeasure.com).
1. Begin with initial score

estimate.
2. Select and present 3. Score response

optimal scale item.
No
4. Estimate confidence
5. Is stopping
interval.
rule satisfied?
Yes
7. End of
battery? 8. Administer next
6. End scale assessment. construct.
Yes No
9. Stop
Figure 13.7 Algorithm for computer adaptive testing. (Adapted from Wainer et al., 2000.)
The development of the ICF Activity Measure tem. Standard apache web server and PHP server
was funded by the Department of Education, side scripting technology was used to design Web
National Institute of Disability and Rehabilitation pages which present individual questions to the
(H133G000227). Using the International Classi- respondent via a standard Web browser (e.g., Mic-
fication of Functioning, Disability, and Health rosoft Explorer, Netscape, Mozilla). Data storage
(ICF) as a theoretical framework, an initial item capability is being designed using PostgreSQL.
bank was developed that focused on movement The system is upward scalable (e.g., questions can
challenges for individuals with upper extremity be added, deleted, and changed) and the adminis-
injuries, lower extremity injuries, back pain, or trator can select a number administration criterion
spinal cord injury. The bank was edited on the basis (e.g., initial item calibration, amount of random-
of focus groups and interviewing of individuals ness in selecting the initial item) and exit criterion
with disabilities as well as a professional advisory (e.g., confidence interval, maximum number of
panel. questions). Since the CAT is administered using a
A paper-and-pencil field test of the item bank standard Web browser, it can be accessed globally
was completed with 408 participants across the via the Internet. Figure 13.8 presents a page of the
four diagnostic groups to produce the data for CAT as it is presented to a respondent. The investi-
Rasch analysis. Item fit statistics were used to gators are using touch screen laptop computers to
identify erratic items and principle components administer the CAT, allowing respondents to use a
analysis was used to identify separate constructs pencil-like probe or their finger to select an answer
within the item bank. The result was an item bank instead of a computer mouse.
of 264 questions representing six constructs: (1) Since computerized adaptive testing is rela-
positioning/transfers, (2) gross upper extremity, tively new in health care, there are few studies pre-
(3) fine hand, (4) walking and climbing, (5) wheel- senting its advantages and feasibility. Initial
chairs/scooters, and (6) self-care activities. findings are promising. Haley and colleagues
The computerized aspect of the ICF Activity (2004) with CAT simulations of their 101-item
Measure was built using unix/linux operating sys- AM-PAC have shown that measures derived of
Figure 13.8 ICFmeasure.com Web page.

four- or five-item CAT presentations correlate with data from each instrument in separate Rasch analy-
measures derived from the total item bank (101 ses while “presetting,” or anchoring, items at the
items) between .90 and .95, and measures derived cocalibrated item measures.
from 10 item CAT presentations correlate between Fisher and colleagues (1995) were the first to
.96 and .98 with measures derived from the total use common-sample equating to link two global
item bank. These findings suggest abbreviated measures of function, the FIM and PECS. Using
CAT-based assessments can be administered with- the methodology described above, they showed
out losing the accuracy of administering full tests. that the 13 FIM and 22 PECS ADL/motor items
This decrease in respondent burden should be could be scaled together in a 35-item instrument.
welcomed by patients and study participants and The authors found that separate FIM and PECS
should represent considerable cost-savings in measures for 54 rehabilitation patients correlated
administering instruments as part of clinical and .91 with each other and correlated .94 with the
research protocols. cocalibrated values produced by Rasch analysis.
Furthermore, these authors demonstrated that
Linking Existing Measures either instrument’s ratings were easily and quickly
converted into the others via a table that used a
A third potential application of IRT methodologies common unit of measurement, which they referred
is the capability of linking healthcare measures to as the “rehabit.” This common unit of measure-
(Velozo et al., 1999). As presented earlier in this ment allows the translation of scores from one
chapter, a measure is not confined to a specific instrument to another. Since the results of Rasch
scale or specific set of items, but instead can be analysis are sample free, these tables can be used
presented by large banks of items that represent for all future and past instrument-to-instrument
the measure. Considering that as many as 85 score conversions.
instruments measuring functional ability exist, it is A number of more recent studies support the
possible that these individual instruments do not potential of linking functional measures. Recently,
each measure a unique aspect of functional ability Smith and Taylor (2004) replicated the FIM-PECS
but are simply subsets of items representing the linking with a more substantial sample of 500
unidimensional construct of functional ability. patients with similar results. Fisher, Eubanks, and
Examples from the physical sciences (e.g., the Marier (1997) used common-sample equating to
ability to easily convert across the many instru- link the 10 physical function items of the Medical
ments used to measure time), and Thurstone’s Outcome Scale (MOS) SF-36 (the PF-10) and
(1926) requirement that measures must be “test the Louisiana State University Health Status
free,” imply that similar measures in health care Instrument. Difficulty estimates for a subset of
can be linked. The possibility of linking measures similar items from the two instruments correlated
of functional ability was underscored when the at .95, again indicating that the items from the two
difficulty order of ADL/motor items on the FIM, scales were working together to measure the same
Patient Conference Evaluation Conference System construct. McHorney and Cohen (2000) applied a
(PECS) and Level of Rehabilitation Scale-III two-parameter IRT model to 206 physical func-
(LORS-III) were found to be virtually identical tioning items (through 71 common items across
(Linacre et al., 1994; Silverstein et al. 1991; samples), and in a similar study McHorney (2002)
Velozo et al., 1995). These initial findings pro- linked 39 physical functioning items (through 16
vided evidence that the scales were measuring the common items) from three modules of the Asset
same underlying functional ability, despite differ- and Health Dynamics Among the Oldest Old
ences in item definitions and rating methodology. study. Both studies demonstrated successful
While the above studies suggest that translation linking of item banks through sets of common
across functional measures is feasible, actual items, allowing placement of all items on a com-
instrument linking can be accomplished through mon metric.
common-sample equating (Fisher, Harvey, Taylor, Preliminary studies by Velozo and colleagues
Kilgore, & Kelley, 1995). For this methodology, are designed to investigate the validity of linking
the items from both instruments are calibrated on a functional measures. Using the FIM and MDS
common sample of persons (i.e., subjects must be with common sample data on 290 veterans, these
assessed with both instruments). Data from both investigators created a crosswalk between the
instruments are “co-calibrated” in a single Rasch measures using the Rasch common-person linking
analysis, producing item measures using the same method described above. Using an independent
scaling unit. Comparisons across the two instru- sample of 2,600 veterans, Velozo and colleagues
ments can then be accomplished by analyzing the compared the converted FIM and converted MDS
raw scores to actual FIM and MDS raw scores. Sciences, University of Illinois at Chicago, Chicago,
Distributions of converted and actual scores did IL.
Campbell, N R. (1920). Physics: The elements. London:
not differ significantly and showed moderate cor- Cambridge University Press.
relations (.73–.74). While the strength of these cor- Fisher, A. G. (1993). The assessment if IADL motor skills:
relations may not support using converted scores An application of many faceted Rasch analysis.
to monitor individual patients, these preliminary American Journal of Occupational Therapy, 47,
319–329.
findings suggest that that converted scores may be
Fisher, W. P. (1997). Physical disability construct conver-
feasible for population-based studies (e.g., predic- gence across instruments: Towards a universal metric.
tive studies). Journal of Outcome Measurement, 1(2), 87–113.
Fisher, W. P., Eubanks R. L., & Marier, R. L. (1997).
Equating the MOS SF36 and the LSU HSI physical
Conclusion
functioning scales. Journal of Outcome Measurement,
1, 329–362.
Fisher, W. P., Harvey, R. F., Taylor, P., Kilgore, K. M., &
Kelley, C. K. (1995). Rehabits: A common language of
While classical test theory has produced numerous
functional assessment. Archives of Physical Medicine
useful assessments, it does not address aspects of and Rehabilitation, 76, 113–122.
objective measurement. This chapter provided an Forsyth, K., Lai, J., & Kielhofner, G. (1999). The Assess-
overview of item response theory and its potential ment of Communication and Interaction Skills (ACIS):
for objective measurement in occupational ther- Measurement properties. British Journal of Occupatio-
nal Therapy, 62(2), 69–74.
apy. The chapter discussed how this new method- Forsyth, K., Salamy, M., Simon, S., & Kielhofner, G.
ology provides tools to identify and correct (1998). The Assessment of Communication and
limitations of existing instruments and to develop Interaction Skill (ACIS) (version 4.0). Model of Human
new measures that meet the requirements of objec- Occupation Clearinghouse, Department of Occupatio-
nal Therapy, College of Applied Health Sciences,
tive measurement.
University of Illinois at Chicago, Chicago, IL.
The chapter also discussed emerging applica- Guttman, L. (1950). The basis for scalogram analysis.
tions of IRT in instantaneous measurement and In Stouffer et al. Measurement and prediction (Vol. 4,
computer adaptive testing. These approaches allow pp. 60–90). Princeton, NJ: Princeton University
for new levels of efficiency and precision in occu- Press.
Haley, S. M., Coster, W. J., Andres, P. L., Kosinski, M., &
pational therapy and healthcare measurement. Ni, P. (2004). Score comparability of short forms and
Finally, the chapter discussed the process of link- computerized adaptive testing: Simulation study with
ing measures that could allow the comparison of the activity measure for post-acute care. Archives of
results from different studies and tracking out- Physical Medicine and Rehabilitation, 85, 661–666.
Haley, S. M., Coster, W. J., Ludlow, L., Haltiwanger, J., &
comes of clients across the continuum of care.
Andrellos, P. (1992). Pediatric Evaluation of Disability
While the use of these modern measurement tech- Inventory (PEDI). Boston: Boston University Center
niques are in their infancy in occupational therapy, for Rehabilitation Effectiveness.
they promise to have great value in improving how Haley, S. M., McHorney, C. A., & Ware, J. E., Jr. (1994).
our clients are measured for clinical and research Evaluation of the MOS SF-36 physical functioning
scale (PF-10): I. Unidimensionality and reproducibility
purposes. of the Rasch item scale. Journal of Clinical
Epidemiology, 47, 671–684.
Hambleton, R. K., & Jones, R. W. (1993). Comparison of
REFERENCES classical test theory and item response theory and their
Baron, K., Kielhofner, G., Iyenger, A., Goldhammer, V., & applications to test development. Educational
Wolenski, J. (2006). The Occupational Self Assessment Measurement: Issues and Practice, 12, 38–47.
(OSA) (version 2.2). Model of Human Occupation Institute of Objective Measurement (2000). Definition of
Clearinghouse, Department of Occupational Therapy, objective measurement. Retrieved September 3, 2004
College of Applied Health Sciences, University of from http://www.rasch.org/define.htm.
Illinois at Chicago, Chicago, IL. Kielhofner G., Dobria, L., Forsyth K., & Basu, S. (2005).
Bates, B. E., & Stineman, M. G. (2000). Outcome indica- The construction of keyforms for obtaining instanta-
tors for stroke: Application of an algorithm treatment neous measures from the occupational performance
across the continuum of post acute rehabilitation serv- history interview rating scales. Occupational Therapy
ices. Archives of Physical Medicine and Rehabilitation, Journal of Research: Occupation, Participation and
81, 1468–1478. Health, 25(1), 1–10.
Bland, J. M., & Altman, D. G. (1997). Statistics notes: Kielhofner, G., Mallinson, T., Crawford, C., Nowak, M.,
Cronbach’s alpha. British Medical Journal, 314, 572. Rigby, M., Henry, A. & Walens, D. (2004). Occupatio-
Braveman, B., Robson, M., Velozo, C., Kielhofner, G., nal Performance History Interview II (OPHI-II) (ver-
Fisher, G., Forsyth, K., & Kerschbaum, J. (2005). sion 2.1). Model of Human Occupation Clearinghouse,
Worker Role Interview (WRI) (Version 10.0). Model Department of Occupational Therapy, College of
of Human Occupation Clearinghouse, Department Applied Health Sciences, University of Illinois at
of Occupational Therapy, College of Applied Health Chicago, Chicago, IL.
Kielhofner, G., Mallinson, T., Forsyth, K., & Lai, J-S. Smith, E. V., Jr. (2002). Detecting and evaluating the
(2001). Psychometric properties of the second version impact of multidimensionality using item fit statistics
of the Occupational Performance History Interview and principal component analysis of residuals. Journal
(OPHI-II). American Journal of Occupational Therapy, of Applied Measurement, 3, 205–231.
55, 260–267. Smith, R. M. (1992). Applications of Rasch measurement.
Lai, J-S., Velozo, C. A., & Linacre, J. M. (1997). Adjusting Maple Grove, MN: JAM Press.
for rater severity in an unlinked functional independ- Smith, R. M., & Taylor, P. (2004). Equating rehabilitation
ence measure database: An application of the many outcome scales: Developing common metrics. Journal
facets Rasch model. In R. M. Smith (Ed.), Physical of Applied Measurement, 5, 229–242.
Medicine and Rehabilitation. State of Arts Reviews, Stone, M. H. (1997). Steps in item construction. Rasch
11, 325–332. Measurement Transactions, 11, 559.
Linacre, J. M. (1997). Instantaneous measurement and Streiner, D. L., & Norman, G. R. (1989). Health measure-
diagnosis. Physical Medicine and Rehabilitation, 11, ment scales: A practical guide to their development
315–324. and use. New York: Oxford University Press.
Linacre, J. M. (1998). Detecting multidimensionality: Thurstone, L. L. (1926). The scoring of individual perform-
Which residual data-types work best? Journal of ance. Journal of Educational Psychology, 17, 446–457.
Outcomes Measurement, 2, 266–283. Thurstone, L. L. (1928). Attitudes can be measured.
Linacre, J. M., Heinemann, A. W., Wright, B. D., Granger, American Journal of Sociology, 23, 529–554.
C. V., & Hamilton B. B. (1994). The structure and sta- Thurstone, L. L. (1931). Measurement of social attitudes.
bility of the Functional Independence Measure. Journal of Abnormal and Social Psychology, 26,
Archives of Physical Medicine and Rehabilitation, 75, 249–269.
127–132. Thurstone, L. L., & Chave, E. J. (1929). The measurement
Linacre, J. M., & Wright, B. D. (1989). The “length” of a of attitude. Chicago: University of Chicago Press.
logit. Rasch Measurement Transactions, 3(1), 3–5. Velozo, C. A., Kielhofner, G., & Lai, J-S. (1999). The use
Lord, F. M. (1953). The relation of test score to the trait of Rasch analysis to produce scale-free measurement of
underlying the test. Educational and Psychological functional ability. American Journal of Occupational
Measurement, 13, 517–548. Therapy, 53, 83–90.
Marino, R. J., & Graves, D. E. (2004). Metric properties of Velozo, C. A., Magalhaes, L. C., Pan, A-W., & Leiter, P.
the ASIA motor score: subscales improve correlation (1995). Functional scale discrimination at admission
with functional activities. Archives of Physical and discharge: Rasch analysis of the Level of Rehabili-
Medicine and Rehabilitation, 85, 1804–1810. tation Scale–III. Archives of Physical Medicine and
Martin, J. T., McBride, J. R., & Weiss, D. J. (1983). Rehabilitation, 76, 706–712.
Reliability and validity of adaptive and conventional Velozo, C. A., & Peterson, E. W. (2001). Developing
tests in a military recruit population. (Research Report meaningful fear of falling measures for community
83-1). Minneapolis: University of Minnesota. dwelling elderly. American Journal of Physical
McHorney, C. A. (2002). Use of item response theory to Medicine and Rehabilitation, 80, 662–673.
link 3 modules of functional status items from the Wainer, H., Dorans, N. J., Eignor, D., et al. (2000).
Asset and Health Dynamics among Oldest Old Study. Computerized adaptive testing: A primer (2nd ed.).
Archives of Physical Medicine and Rehabilitation, 83, Mahwah, NJ: Lawrence Erlbaum Associates.
383–394. Ware, J. E., Jr, Kosinski, M., Bjorner, J. B., Bayliss, M. S.,
McHorney, C. A. (2003). Ten recommendations for Batenhorst, A., Dahlof, C. G., Tepper, S., & Dowson,
advancing patient-centered outcomes measurement for A. (2003). Applications of computerized adaptive test-
older persons. Annals of Internal Medicine, 139, ing (CAT) to the assessment of headache impact.
403–409. Quality of Life Research, 12, 935–952.
McHorney, C. A., & Cohen, A. S. (2000). Equating health Webster, K., Cella, D., &Yost, K. (2003). The Functional
status measures with item response theory: Illustrations Assessment of Chronic Illness Therapy (FACIT) meas-
with functional status items. Medical Care, 38 (Suppl. urement system: Properties, applications, and interpre-
II), 1143–1159. tation. Health Quality of Life Outcomes, 1, 79.
Merbitz, C., Morris, J., & Grip, J. C. (1989). Ordinal scales Weiss, D. J. (1985). Adaptive testing by computer. Journal
and foundations of misinference. Archives of Physical of Consulting and Clinical Psychology, 53, 774–789.
Medicine and Rehabilitation, 70, 308–312. Wright, B. D. (1997a). A history of social science meas-
Olsen, J. B., Maynes, D. D., Slawson, D., & Ho, K. 1986). urement. MESA Memo 62, http://www.rasch.org/
Comparison and equating of paper-administered, memo62.htm.
computer-administered, and computerized adaptive Wright, B. D. (1997b). Fundamental measurement for out-
tests of achievement. Paper presented at the annual come evaluation. Physical Medicine and Rehabilita-
meeting of the American Educational Research tion: State of the Art Reviews, 11 (2), 261–288.
Association, San Francisco. Wright, B. D., & Linacre, J. M. (1989). Observations are
Rasch, G. (1960). Probabilistic models for some intelli- always ordinal; measurements, however, must be inter-
gence and attainment tests [Danish Institute of val. Archives of Physical Medicine and Rehabilitation,
Educational Research 1960, University of Chicago 70, 857–860.
Press 1980, MESA Press 1993]. Chicago: MESA Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-
Press. square fit values. Rasch Measurement Transactions,
Silverstein, B., Fisher, W. P, Kilgore, K. M., Harley, J. P., 8, 3.
& Harvey, R. F (1991). Applying psychometric criteria Wright, B. D., & Masters, G. (1982). Rating scale
to functional assessment in medical rehabilitation: II. analysis. Chicago: MESA Press.
Defining interval measures. Archives of Physical Wright, B. D., & Stone, M. H. (1979). Best test design.
Medicine and Rehabilitation, 73, S3–S23. Chicago: MESA Press.
RESOURCES Linacre, J. M. (2004). WINSTEPS Rasch measurement

computer program. http://winsteps.com
Books
This book is an excellent introductory/intermediary text- This software can be used to run the Rasch, 1-parameter
book about Rasch measurement theory and applications model to analyze additional facets such as rater sever-
of Rasch analysis: ity. A free student/evaluation version of the software
Bond, T. G., & Fox, C. M. (2001). Applying the Rasch called MINIFAC can be downloaded from the same
model: Fundamental measurement in the human sci- site:
ences. Mahwah, NJ: Lawrence Erlbaum. Linacre, J. M. (2001). FACETS Rasch measurement com-
puter program. http://winsteps.com
This brief manual provides explanations of WINSTEPS
and FACETS software output: Web Sites
Smith, R.M. (2003). Rasch measurement models: This Web site provides extensive information, resources
Interpreting WINSTEPS and FACETS output. Maple and access to software on the Rasch measurement
Grove, MN: Journal of Applied Measurement Press. model and Rasch analysis. Of particular note is the
Software search engine of this site that provides access to the
This software can be used to run the Rasch, 1-parameter Rasch Measurement Transactions – extensive short and
model to analyze dichotomous and rating scale data A mid-size reports on the many aspects of Rasch theory
free student/evaluation version of the software called and analysis:
MINISTEPS can be downloaded from the same site: Institute of Objective Measurement, http://www.rasch.org/
C H A P T E R 1 4
Evaluating the Use of Assessments

in Practice and Research
Wendy J. Coster
Typical research texts spend considerable time performing activities of daily living [ADLs]),
addressing research design issues such as sample seven different cutoff scores had been used to dif-
size and power, the degree to which threats to ferentiate favorable and unfavorable outcomes. As
internal validity such as rater bias or inappropri- the authors note, in a comment that could be
ately matched comparison groups have been con- applied to many areas of rehabilitation research,
trolled, or the appropriate selection of statistical “Clearly, when there is so much variability in the
analysis methods. In comparison, content related methods used for outcome assessment … it is dif-
to questions of measure validity is often quite lim- ficult to know what to make of the results”
ited. While research designs typically are sub- (Duncan et al., 2000, p. 1435). This uncertainty
jected to very critical evaluation, critique of the affects not only the investigators, but also practi-
choice of measures and interpretation of results tioners searching for sound evidence to guide clin-
from their use is often surprisingly superficial. Yet, ical decision-making and consumers seeking
the usefulness of the entire clinical research enter- information about effective intervention options.
prise rests on the extent to which the measures The purpose of this chapter is to provide a
used to examine whether the experimental group framework for practitioners and clinical researchers
has “improved” or whether persons who received for thinking about the choice of assessment instru-
occupational therapy “function” better in their ments and interpretation of evidence from clinical
homes really do measure what they say they do. research and practice. It is organized around a
An example from the stroke literature illus- series of important questions that need to be asked
trates this point (Duncan, Lai, & Keighly, 2000). and answered in all assessment situations.
Stroke is a major health event that often results in
long-term disability; thus considerable research
has been devoted to examining the extent to which
rehabilitation can decrease the likelihood or extent
What Assumptions Are
of disability or improve the degree or rate of recov- Reflected in This Measure?
ery of function. In this study, the authors examined
the impact of using different measures and differ- The very process of using an assessment from
ent cutoff points to define successful recovery after which one derives a score rests on an important
stroke. The results indicated that the percentage of assumption, which is that functional skills, capa-
both the treatment and placebo groups considered bility, emotional experience, or whatever construct
“recovered” varied quite significantly depending is the focus of the assessment can be quantified.
on which measures and cutoff points were used. When we observe behavior in the natural environ-
For example, shifting the definition of “recovered” ment, it typically appears fluid and seamless, but
by one level on the modified Rankin Scale (a com- when we use a scale such as the Functional
monly used disability severity measure) (Rankin, Independence Measure (FIM) (Guide for the uni-
1957) changed the percentage of those classified as form data set for medical rehabilitation, 1997) we
recovered from ⬍ 25% to 53.8%, a substantial dif- accept for the moment that meaningful differences
ference. Thus, a study that used the more lenient between people in their dressing, bathing, or walk-
cutoff to classify someone as recovered would ing can be described using one of seven distinct
report more successful results than one using the categories of assistance.
more conservative cutoff. Similarly, when we ask a parent to complete
A related paper (Duncan, Jorgensen, & Wade, the Functional Skills section of the Pediatric
2000) reported that, across 27 drug trials for treat- Evaluation of Disability Inventory (PEDI) (Haley,
ment of stroke that used the Barthel Index (an Coster, Ludlow, Haltiwanger, & Andrellos, 1992),
index that measures need for assistance while we accept for the moment that there is a clear
Figure 14.1 Dr. Coster chats

with colleagues following a
presentation on assessments
at the annual American
Association (AOTA) meeting.
(observable) difference between children who are that this standard performance context necessarily
and are not capable of “using a fork well” and that reveals more about the person’s abilities than his
this difference can be represented by a score of 1 or her performance in a different context, for
or 0. This assumption is necessary in order to con- example, in one that is more familiar. That is
duct any kind of quantitative measurement of separate issue, which we will discuss later in the
behavior. However, like all assumptions, the plau- chapter.
sibility of its application in a particular measure-
ment situation should be carefully evaluated.
A second assumption is that a sample of a per- What Social Forces Have
son’s behavior, thoughts, or opinions taken at one
point in time can serve as a legitimate representa-
Affected the Design and
tion of his or her “true” situation or experience. Use of This Assessment?
This assumption rests on the interesting proposi-
tion, strongly reflected in Western culture, that Other chapters in this text address in depth a vari-
there is such a thing as the person’s “true” situation ety of social and cultural influences that shape the
or experience. In other words, an investigator who inquiry process. To the extent that “knowing” in
had the correct methods would be able to pinpoint the quantitative tradition involves information
the person’s “true” level of function, understand- derived from assessments, these influences also
ing, judgment, and so forth. Some form of this must be considered here. Consider how the out-
assumption is also necessary in order to engage in comes of intervention methods are currently eval-
measurement. However, assessment developers uated in physical rehabilitation practice and
and users may vary in how they interpret the mean- research. A large proportion of studies use meas-
ing of information derived from a particular sam- ures of body function/body structure or impair-
ple of behavior. ment, such as strength, range of motion, or
Until the past decade or so, users of measures endurance. Why? The reasons are complex; how-
of skill performance or ability (both clinicians and ever, at least one factor is the continuing influence
researchers) accepted that administration under of the medical model paradigm, which assigns a
standard, controlled conditions yielded the best higher scientific value to physical or biological
approximation of the person’s “true” capability. indicators that can be measured precisely using
This assumption confounds two important but dis- well-calibrated equipment. Within this paradigm,
tinct issues. By having persons perform under it has been traditionally assumed that improved
standard conditions we do help ensure that the function will follow directly from reduced impair-
scores for each person were obtained under rea- ment. Therefore, demonstration of change in
sonably similar conditions, which is necessary impairment is considered sufficient evidence of a
for a fair comparison. However, it does not follow meaningful outcome and additional measures of
Chapter 14 Evaluating the Use of Assessments in Practice and Research 203
function are not necessary. However, function, as (Coster, Deeney, Haltiwanger, & Haley, 1998) or
measured by activity and participation, is a com- the Stroke Impact Scale (Duncan et al., 1999,
plex phenomenon, and the relationship between 2003). Ideally, the authors provide a definition of
discrete impairments and function in daily life is this construct (and, possibly, its relevant dimen-
not straightforward. sions) and cite theoretical and research work
A second large group of studies use measures to support the validity of their definition in a
of basic ADL performance, in particular the manual or in published papers. However, the items
Barthel (Mahoney & Barthel, 1965) and the FIM and measurement scales constitute the actual
(Guide for the UDS, 1997), to examine outcomes. operational definition of the construct. Any infer-
Both of these instru- ences that practitioners or
ments measure function researchers want to draw
in terms of the amount Validity is not an attribute of from results of using the
of assistance needed. the test itself, but rather it instrument are restricted
Why? Again, the reasons to those that can be drawn
are very complex; how- is evidence that supports appropriately from this
ever at least one factor is interpretation or application operational definition.
the influence of reim- Therefore, it is important
bursement systems in (i.e., the meaning) of the to examine that definition
the United States. ADLs test scores. independent of what the
such as eating, toileting, authors describe.
and bathing must be per- For example, consider
formed regularly in order to regain and/or maintain the Barthel index used in the studies of stroke out-
physical health. Therefore, inability to complete comes described earlier. The Barthel index was
these activities without help (sometimes referred used by researchers as a measure of functional
to as the burden of care) has significant cost impli- recovery, and inferences about rates of recovery
cations for payers. In contrast, the health implica- were made based on these scores. The Barthel’s
tions of inability to do grocery shopping or to get operational definition of recovery is that the person
together with friends are not as immediate or obvi- can complete a specific set of basic ADLs with
ous. Cost is a legitimate concern; however, it is minimal to no assistance (depending on the cutoff
important to question whether the extent of score that is used). However, this is only one of
improvements in basic ADL performance should many possible operational definitions of the con-
be the primary scientific criterion for evaluation of struct of functional recovery. The person who no
rehabilitation effectiveness (Wade, 2003). longer needs physical help with ADLs, but still
cannot do the grocery shopping, return to work, or
drive a car may not describe himself as “recov-
What Is Really ered.” (See Lai, Perera, Duncan, & Bode, 2003;
Being Measured? Lai, Studenski, Duncan, & Perera, 2002 for an
example). Thus, when reviewing research results,
or reporting results from an assessment, it is criti-
Assessments are often named according to the cally important to know the item content of the
major construct they are thought to measure, for assessment in order to set the appropriate bound-
example, the School Function Assessment (SFA) aries around conclusions.
It is also important to be familiar with the scale
used to quantify degrees of the construct being
measured because this scale is also part of the
What social forces affected the design and use operational definition. For example, asking the per-
of this assessment?
Who (what group of users) was the measure
son if he or she needs physical help to get dressed
intended for? in the morning is not the same as asking if he or she
What purpose was the measure designed for? has difficulty getting dressed (e.g., Laditka &
How has this purpose affected the content and Jenkins, 2001). A person who gets the maximum
design of the assessment? score on an assistance scale (does not need help)
What values are reflected in this measure? may not achieve the maximum score on a difficulty
Are these values and purpose consistent with my scale; according to one scale he is “recovered,”
purpose and values? while on the other scale he is experiencing “con-
tinuing limitations.” In studies in which we have
compared items with similar content from different What Is the Model of the
functional assessment instruments, we have found Underlying Construct?
repeatedly that these different operational defini-
tions may result in very different pictures of the As noted earlier, the authors of an assessment
person’s function (Coster et al., 2004; Coster, should provide a definition and conceptual model
Haley, Ludlow, Andres, & Ni, 2004). of the attribute they are trying to measure. In the
behavioral sciences, a theoretical model is typically
cited to support the construction of the assessment;
What Evidence Supports however, it is not uncommon in the rehabilitation
sciences to find that an assessment was designed
the Congruence Between based simply on pragmatic considerations; for exa-
the Conceptual Model and mple, the items selected are a set of “typical daily
activities.” A theoretical model lends itself to test-
the Actual Measure? ing through research because it contains a series of
propositions about how certain elements of per-
When we are examining what an assessment actu- formance should relate to other elements, or how
ally measures, we are considering questions performance on the assessment should differ
of validity (see Chapters 12 and 13 for further dis- according to age, functional status, or other factors.
cussions of validity). Validity is not an attribute of By testing these propositions, data are obtained that
the test itself, but rather it is evidence that supports either do or do not support the interpretation of test
interpretation or application (i.e., the meaning) of scores in the way the authors proposed. For exam-
the test scores (Messick, 1995; Morgan, Gliner, & ple, the function/disablement model that guided the
Harmon, 2001). Because interpretations may vary development of the School Function Assessment
according to who is being assessed, or the context (SFA) (Coster et al., 1998) proposes that functional
in which he or she is being assessed, or the pur- performance of students with disabilities is context
pose of conducting the assessment, validity exam- dependent. If this proposition is correct, then rat-
ination is an ongoing process from which evidence ings on the SFA Participation Scale should vary
is accumulated over time. Two related areas of across school settings. This proposition was sup-
validity investigation are discussed here: ported by findings from the standardization data,
which indicated that performance did vary signi-
• Investigations that support interpretation of the ficantly across settings and, furthermore, that there
assessment as a valid indicator of the underlying was a hierarchy of settings, with students likely
construct, and to participate more in some settings than others
• Investigations of the extent to which scores reflect (Mancini & Coster, 2004).
factors or processes that are irrelevant to the con-
struct.
Do Subscales Represent Distinct
Dimensions of the Construct?
What does this assessment actually measure?
Many assessments used by occupational therapy
What domain do the authors identify as the focus practitioners include subscales for which individ-
of the measure? ual scores can be obtained, as well as overall sum-
What is this definition based on? mary scores. This kind of structure implies that the
Does this definition derive from a theoretical construct being measured has several distinguish-
or a conceptual model? able dimensions and evidence should be provided
Is there evidence to support this model? to support this proposition. Factor analysis is used
What is the operational definition of the con- frequently to test this aspect of the model, by look-
struct, as defined by the items and rating or ing to see if the items really do group themselves
scoring system? into separate factors that correspond to the sub-
Are there aspects of the construct that are
not represented by this operational
scales of the test (e.g., Jette et al, 2002; Ronen,
definition? Streiner, Rosenbaum, & the Canadian Pediatric
Does this operational definition cover the phe- Epilepsy Network, 2003). If they do not, this result
nomenon that is important for my current contradicts the authors’ model and suggests that
purpose? the subscale scores do not reflect different dimen-
Are the authors’ conclusions appropriate, based sions of the attribute, ability, or skill, or that the
on this operational definition? dimensions are different from what the authors
identified. This information is important for users
of the assessment, because it has implications for

Some readers may have noticed that I have not
the appropriate interpretation of subscale scores.
used terminology they have encountered in other
Without evidence to support the distinctiveness of discussions of assessment (see Chapter 12), such
the subscales, interpreting differences among sub- as content validity, discriminant validity, or con-
scale scores as describing a profile of strengths and current validity (Gregory, 2000; Morgan, et al.,
weaknesses has no sound basis. 2001). These terms really all refer to different
On the other hand, if the assessment provides a ways of obtaining evidence related to construct
meaningful summary score by averaging or sum- validity, which is the overarching concern we
ming subscale scores, then evidence should be pro- have been examining, rather than to separate cat-
vided to support this aspect of the model as well egories of validity. From this perspective, content
(e.g., see Young, Williams, Yoshida, & Wright, validity refers to evidence that the assessment
adequately covers the relevant domains of the
2000). In this case, the authors are proposing that
construct (content coverage) and that all of the
even though there may be some differences across content is relevant to the construct (content rele-
subscales, there will also be meaningful consis- vance). Discriminant validity refers to evidence
tency across them that can be represented by a that the assessment differentiates between groups
summary score. Various kinds of analyses may be who should behave or perform differently on a
conducted to provide this evidence. For example, measure of this particular construct. Concurrent
correlations may be conducted between subscale or criterion validity refers to evidence gathered
scores and summary scores to show the extent of by examining the relation between the assess-
association between them. Although it is unlikely ment and other measures administered to the
that all subscale scores would be equally corre- same persons at a similar point in time. If the
assessment in question is a good measure of its
lated with the summary score, major discrepancies
intended construct, then scores from the measure
in the size of the correlations may indicate that should correlate with scores from other measures
it is dominated by one or more subscales, or, of that construct (convergence). It is also impor-
conversely, that the summary score does not accu- tant to demonstrate that the assessment does not
rately reflect the performance, ability, or experi- correlate with other measures of different con-
ence measured by that subscale. Factor analysis structs (divergence). Although this evidence is
results may also be examined as evidence. If there obtained by administering concurrent measures,
is very little overlap in the factors (i.e., each item the results speak to the meaning of the scores
loads on one and only one factor), this weakens the obtained from each, in other words, to construct
basis for aggregating these factors in a summary validity.
score. More recently, scale dimensionality has
been examined using the IRT methods discussed in
Chapter 13 (for examples, see Duncan et al., 2003;
pretation as well (see Baum, Perlmutter, & Dunn,
Young et al., 2000).
2001). Of course, the definition of “irrelevant”
These validity questions are complex and are
depends on the construct of interest and the
not resolved by a single study. The developer of an
population being assessed. If one is interested in
assessment may have examined the factor structure
measuring elders’ knowledge about community
in a sample of typically developing children, but
resources, then factors such as a survey’s small
that does not ensure that the conclusions can
print size and advanced reading level may intro-
be applied to a sample of children with cerebral
duce irrelevant variance in responses. On the other
palsy or Down syndrome. Many clinical condi-
hand, if one is assessing functional literacy, then
tions affect one area of performance abilities more
print size may be highly relevant to determining the
than another, which can result in very different
limits of what the person can and cannot do in nat-
profiles across subscales of complex tests by dif-
ural settings. Practitioners can apply their knowl-
ferent groups, and may yield summary scores
edge about the clients they work with to identify
whose meaning is very unclear.
these threats to valid interpretation of assessment
scores, whether they are in a research report or in a
Are Scores Affected by Irrelevant Factors? clinical assessment report.
The example just described is one where there is
substantial construct-irrelevant variance (Messick,
1995), meaning that too much variance in scores Whose Voice Is Represented?
may be due to factors that are not part of the con-
struct. There are many other types of irrelevant A researcher or practitioner selecting an assess-
variance that may affect the validity of score inter- ment must also think about what source of infor-
Thus, when we are evaluating evidence to

What evidence supports the congruence incorporate into practice decisions, we want to
between the conceptual model and the
look at where that evidence comes from (Brown &
actual measure?
What methods were used to group items into Gordon, 2004). We need to think about: Whose
scales? Are these methods based in evidence? voice are we hearing? Is that the appropriate
What evidence supports the distinctions among voice? Are we using the correct vehicle to bring us
subscales and/or the creation of an aggregate that voice? These questions have been asked with
(summary) score? much more frequency in recent years. As dis-
Was this evidence derived from a population sim- cussed below, there are some excellent examples
ilar to mine? If not, how might differences in the literature that illustrate the importance of,
between these populations affect my applica- for example, making more effort to include the
tion of these results? consumer’s voice in the design of instruments or
Are there characteristics of my client group that
including the voices of children and others with
could be sources of construct-irrelevant vari-
ance on this measure? more limited communication skills rather than
What evidence has been presented to support substitute proxy respondents
validity of the measure as an indicator of the Meyers and Andresen (2000) have critiqued the
underlying construct? Does this evidence design of instruments and data collection methods
apply to my client population? from a universal design perspective. They note that
population survey research (the kind of research,
for example, used to identify the health status or
service needs for a city, state, or nation’s popula-
mation is appropriate for the question being asked. tion) often relies on random digit dialing methods,
The source should be congruent with the definition with specific restrictions on number of rings
or model of the construct, as well as with the pur- allowed before terminating a call and against leav-
pose of the assessment. For example, in her discus- ing messages. What impact does this methodology
sion of participation as defined in the International (which is quite rigorous in the traditional sense)
Classification of Functioning, Disability, and have on obtaining the perspective of persons with
Health (World Health Organization, 2001) Law mobility limitations (who may take a longer time
(2002) notes that a person’s preferences for and sat- to reach the phone), or of persons who communi-
isfaction with engagement in occupations are cate through assistive technology?
important dimensions of this construct. To be con- Other limitations may be less conspicuous. For
sistent with this definition, the only appropriate example, many health status questionnaires ask the
respondent for assessment of these aspects of par- respondents to make statements about their general
ticipation is the person him- or herself. health over a specific time period, such as “in the
However, an assessment also reflects the devel- past 12 months” or “in the past month.” How will a
oper’s idea of what’s important to measure about person with a variable condition such as multiple
another person. Therefore, decisions are not just sclerosis or Parkinson’s disease, whose ability to
conceptually based, but are also value based. Often, engage in various life activities may have varied
the values behind these decisions are implicit and dramatically over the period in question, respond to
may go unexamined for a long period of time. For such an instrument? Or, how should a middle-aged
example, the child development literature contains man who sustained a spinal cord injury as an ado-
many studies that describe themselves as reporting lescent respond to a question that asks: “As a result
on parents’ effects on development, or parents’ of your health, have you been limited in your abil-
perceptions of their children’s behavior, or parents’ ity to complete your tasks at work?” This question
beliefs. A closer examination of the studies reveals probably was written by someone without a dis-
that most or all of the parents providing this infor- ability whose definition of health conditions
mation are mothers, and that fathers (even in two includes all variations from “typical,” including the
parent families) often are not included. This design mobility limitations that follow spinal cord injury.
decision may have been made for practical reasons That may not be the perspective of the person
(e.g., fathers were less available during working with the disability, who may reserve the term
hours than mothers; the project did not have suffi- health condition for more acute physiological con-
cient funds to interview both parents). However, ditions such as flu or pressure sores, and may view
the value implicit in the studies’ titles and descrip- limitations at work as resulting from inadequate
tions is that mothers’ voices are more important to accommodations in that environment. Finally, if
listen to, and that they can speak for both parents’ the highest item on a scale (i.e., the item that would
experience. identify those with the greatest health) is the ability
to walk a mile, how will the scale give voice to the Ronen, Rosenbaum, & Law, 1999). As has been
experiences of those who may have skied, white- reported for adults, children’s own perceptions of
water-rafted, or traveled extensively without ever their function, distress, or difficulties may differ
walking a mile? significantly from the perceptions of adult proxies
Researchers who have begun the measurement (e.g., Achenbach, McConaughy, & Howell, 1987;
development process by asking consumers about Ronen et al., 2003) and reports of children’s func-
their perspectives often tion may also differ as a
hear different issues than function of the context in
what is represented on If the highest item on a which they are observed
available instruments. scale (i.e.,, the item that (Haley, Coster, & Binda-
For example, Laliberte- Sundberg, 1994).
Rudman, Yu, Scott, and would identify those with the Not surprisingly, pop-
Pajouhandeh (2000) con- greatest health) is the ability ulations with develop-
ducted a qualitative study mental or intellectual
using focus groups on the to walk a mile, how will the disabilities have also
perspectives of persons scale give voice to the expe- been excluded frequently
with psychiatric disabili- from having a “voice,”
ties of what constitutes
riences of those who may often because persons
quality of life. The par- have skied, white-water- with disabilities were
ticipants identified key rafted, or traveled exten- assumed to be unable
themes of managing to report reliably on
time, connecting and sively without ever walking their own experience.
belonging, and making a mile? However, this assump-
choices and having con- tion is being challenged
trol, each of which had increasingly as new, bet-
several subthemes. The authors compared the ter adapted methods are used for the design of
themes identified by the participants to the content measures (e.g., see Cummins, 1997).
of existing quality of life instruments and identi-
fied a number of gaps. Interestingly, not only were
some areas identified as important by the par- How Do We Know if the
ticipants with disabilities not represented in the
instruments, but even when certain content was
Information From Our
included, the instrument often did not emphasize Measures Is Trustworthy?
the aspects that were important to the participants.
For example, many instruments ask about the As noted earlier, our use of assessments rests on
number of friends a person has, or how frequently the assumption that the sample of behavior we
the person interacts with others, but the partici- gather at one point in time can be treated as a legit-
pants indicated that the important element to them imate representation of a person’s situation or
was the quality of the interactions they had and experience. Reliability data provide one source of
whether it enhanced their sense of connectedness evidence regarding the validity of this assumption
and belonging. Other authors have also questioned (Gliner, Morgan, & Harmon, 2001), specifically
the use of quantity as an appropriate indicator of whether the information obtained at one point in
the quality of a person’s social life (Dijkers, time or from one particular source is likely to be
Whiteneck, & El-Jaroudi, 2000). replicated on another occasion (test–retest) or
One group whose voice was heard very little when provided by a different source (inter-rater).
until recently is children. Instead, we have heard As discussed in Chapters 12 and 17, the most com-
the voices of concerned adults—parents, teachers, mon form for estimates of reliability is the correla-
clinicians—who reported on the activities and par- tion coefficient (r).
ticipation of the children. However, studies have
shown that by age 7 and up children can respond to Whose voice is represented?
many of the typical response formats used in self- Whose perspective guided the selection of con-
report instruments (e.g., Juniper et al., 1996), that tent and the scoring metric of the instrument?
they can reliably report on their own functional per- Who is the respondent? If someone other than
formance (e.g., Young, Yoshida, Williams, Bom- the client is the respondent, is this an appro-
bardier, & Wright, 1995), and that they can identify priate source of information for the purpose?
meaningful dimensions of their quality of life (e.g.,
Reliability is a feature of a measurement, not of impairment information) and performance in the

an instrument; thus it is more appropriate to talk occupational therapy clinic were less concordant
about score reliability rather than test reliability. with in-home performance.
This distinction is often overlooked in the litera-
ture, but is important for practitioners to keep in Could the Change on This Measure
mind. A study of an instrument’s test–retest relia-
bility is actually an examination of the repro-
Be Due to Measurement Error?
ducibility of the scores under specific conditions. Standard reliability estimates provide evidence
The practitioner must decide whether the condi- about reliability of scores for a group. This evi-
tions in which he or she wants to use the instru- dence is very useful when evaluating a test for pos-
ment are similar enough to those conditions that sible use; however practitioners may be less
the estimate can be generalized. For example, if an concerned with scores for a group than with the
instrument has been tested only with healthy reliability of the score of a particular individual.
middle-aged adults, would scores be equally reli- That is, we need an estimate of the likely consis-
able when the instrument was administered to tency of that individual’s score, which would let us
young men with traumatic brain injury? If the know how confident we should be that the score
assessment is typically administered by having the we obtained is a good indicator of the individual’s
person fill out the survey independently, will the “true” ability or performance level. The standard
scores be equally reliable if the information is error of measurement (SEM) (Portney & Watkins,
gathered by having a clinician ask the questions in 2000) is used for this purpose.
face-to-face interview? The research to answer For assessments developed using classical test
these questions may not be available, in which theory (CTT), the standard error of measurement
case practitioners will need to use their clinical is calculated from a reliability coefficient and
expertise to think these questions through carefully applies to an entire scale. However, one of the
before making a decision. (For some provocative advantages of assessments developed using IRT
commentary on reliability, see Rothstein, 2001, methods (discussed in Chapter 13) is that they
2002, 2003). yield separate standard error estimates for each
possible score on the scale, which are therefore
Should We Always Expect more precise. The SEM indicates how much vari-
ability is expected around a given summary score
Consistency in Scores?
and is used to construct confidence intervals
Inter-respondent (or inter-rater) comparisons raise around scores. Use of these confidence intervals
additional questions. First, for survey measures or cautions us that our measures are not perfect and
judgment-based assessments such as the PEDI that the best we can do is identify the range within
(Haley et al., 1992), is client–therapist (or clini- which the person’s score is likely to fall.
cian–parent, or teacher–parent) agreement really Thus, when using a cutoff score to make a diag-
a reliability estimate? If the ratings are different, nostic or eligibility decision, we should be sure
does this mean that one of the respondents is that the person’s score is not only below the cutoff
wrong or unreliable? Not necessarily. It may be point, but is below the lower confidence interval.
that each person is responding based on different If we are examining whether a person has made
information or perspectives. For example, using significant progress, we should be sure that the
meta-analysis to examine agreement between amount of improvement on the reassessment
respondents on children’s behavioral and emotional extends beyond the upper confidence interval of
problems, Achenbach, McConaughy, and Howell the initial score. Otherwise, we are exaggerating
(1998) found that the mean correlation between
pairs of informants varied depending on whether
they usually saw the child in the same situation or How do we know if the information from our
measures is trustworthy?
not. Rogers et al. (2003) examined agreement
How has reliability been examined and what
between self and proxy reports, and in-hospital and were the results?
in-home observed performance of ADL and IADL Has reliability been examined with clients/condi-
activities by community-dwelling older women. tions similar to mine?
They concluded that cognitively intact respondents What is the standard error of measurement? How
provided a reasonably accurate indication of the wide (how many points) is the confidence
functional tasks they could do at home, and that interval around an individual score?
clinical judgment (estimates based on client
the accuracy of our instruments and, potentially, scale is not sensitive to this degree of change in per-
drawing inaccurate conclusions about the effects formance.
of intervention.
Do Lower Scores Always Mean
That No Gains Were Made?
If Reassessment Shows
“No Change,” Does That Reassessment with norm-referenced tests is com-
mon practice (sometimes even a requirement)
Mean the Intervention within some settings (e.g., schools). It is not un-
common to find that children with disabilities
Didn’t Work? obtain lower scores on these tests as they get older,
which may be misinterpreted as a lack of overall
In practice and in research, assessments may be progress or even as a worsening of the condition
used for a variety of purposes, including describing (see Garwood, 1982). This interpretation is valid
a person’s current status (e.g., cognitive develop- only in the very narrow sense that the child is not
ment; motor skills), identifying deficit or clinical performing the items expected for his age group.
disorder, or predicting the likelihood of a future Norm-referenced scores reflect a comparison to
outcome. Another major use of assessment in same-age peers; thus, the performance or behavior
health and rehabilitation is to examine whether on which the scores are based changes as chil-
change has occurred over time as a result of inter- dren’s development proceeds. A very young child
vention, natural development, or recovery pro- with a developmental disorder may acquire many
cesses (Kirshner & Guyatt, 1985). This particular of the basic skills measured for that age group and,
application of assessments requires support from therefore, obtain an average or “slightly delayed”
another type of evidence, which is evidence that score. However, if the child’s pace of skill acquisi-
the instrument is sensitive enough to detect mean- tion does not match the very rapid pace that is typ-
ingful change over time. Evaluation of that evi- ical of nondisabled children during the toddler and
dence requires application of clinical expertise as preschool period, then, at the next assessment, the
well as knowledge about the relevant research comparison-based score will drop. The child may
methods. have continued to make steady progress; however,
it wasn’t enough to obtain an “average” score.
Can This Measure Capture the Criterion-referenced scores are better suited to
indicate whether progress has occurred in situa-
Kind of Change That Is Likely?
tions in which one doesn’t expect “catch-up” with
When considering the use of an assessment to a comparison group (Haley, Coster, & Ludlow,
measure change over time, one of the first ques- 1991). These types of scores, which may be desig-
tions that must be asked is how much change is ned for a specific age group but are not norm-
likely in the situation one will be examining. If a referenced, provide a clearer indication of whether
practitioner is planning to measure changes in the person being assessed has continued to move
function in a student with cerebral palsy receiving toward the positive end of the continuum repre-
in-class intervention, what degree and type of sented by the measure, for example, functional
change is reasonable to expect over the course of a skill acquisition (see the PEDI, Haley et al., 1992;
school year (based on experience or research liter- and SFA, Coster et al., 1998 for examples).
ature)? Are the items and scoring system of the
assessment designed to record those changes? To Can This Measure Capture
illustrate, suppose items on my test are scored with
Meaningful Change?
a 4-point scale, and that, in order to obtain a rating
higher than “0,” the student must perform the activ- An instrument’s ability to detect change (sensitiv-
ity described with no physical help. If the student ity), however, does not guarantee that the change
I am working with has moderate cerebral palsy, detected has clinical or “real life” meaning, which
and my intervention is focused on enabling the stu- is referred to as responsiveness. Responsiveness
dent to complete some part of each activity on his requires evidence that changes on the measure
own, so that physical help is needed only for relate to some external criterion of usefulness,
the final, more difficult parts, then, even if the stu- value, or meaning. There are two important issues
dent achieves this amount of progress on all the to consider. The first involves the distinction
items, his summary score will not change. The between statistical and clinical differences. Given
a large enough sample, even a small difference on

Is this measure responsive to change in the
a measure (e.g., change in scores from the first to
clients and context I am interested in?
the second test administration, or the difference Is the method for scoring items likely to detect
between two groups’ scores) may achieve statisti- the degrees or types of change seen in my
cal significance. This result only reflects the prob- clients over the usual assessment period?
ability that the average amount of change detected Is there evidence that summary scores are
is greater than might be expected due to chance sensitive to changes in clients like mine?
variation. The result says nothing about whether Is there evidence about how much change
this amount of change is large enough to have any on the measure is needed to be clinically
sort of clinical or “real life” impact or meaning. To meaningful?
illustrate, in certain circumstances changes of a
few degrees in range of motion may be statistically
significant, but the likelihood that this amount of
change altered the person’s ability to accomplish can be answered by the application of formulas or
daily activities, i.e., that it is clinically significant, concrete rules because they require careful analy-
is probably slim. sis and synthesis of multiple factors including the
Responsiveness studies are needed to determine question of interest, the clients being assessed and
whether the amount of change on a measure has the conditions of assessment, the features of the
value or meaning instrument itself, and the
(Liang, 2000). The basic interactions among all of
design of such studies is An instrument’s ability to these. A thorough evalu-
a comparison of change ation of these questions
measured on the assess-
detect change…does not also requires integration
ment to some other, guarantee that the change of knowledge derived
valid indicator of change detected has clinical or from clinical experience
in the attribute. with knowledge about
Unfortunately, there are “real-life” meaning. measurement so that our
no real “gold standards” conclusions are guided
we can use for this pur- by what we know from
pose in rehabilitation, so most studies use a combi- our interactions with the real people whose lives
nation of other methods to judge change such as may be affected by the numbers applied to them by
client or clinician visual analog ratings of how our assessments.
much overall change has occurred or performance
observations by raters who do not know the client’s REFERENCES
score on the assessment (i.e., “masked” or Achenbach, T. M., McConaughy, S. H., & Howell, C. T.
“blinded” raters) (e.g., Bedell, Haley, Coster, & (1987). Child/adolescent behavioral and emotional
Smith, 2002). These data are also sometimes used problems: Implications for cross-informant correla-
to calculate the “minimal clinically important dif- tions. Psychological Bulletin, 101, 213–232.
ference (MCID), which is the minimum amount of Baum, C., Perlmuttter, M., & Dunn, W. (2001).
Establishing the integrity of measurement data: Identi-
change on the assessment (e.g., number of points) fying impairments that can limit occupational perform-
needed before an externally observable difference ance and threaten the validity of assessments. In M.
is detected (see Iyer, Haley, Watkins, & Dumas, Law, C. Baum, & W. Dunn (Eds.), Measuring occupa-
2003, for an example). The MCID is particularly tional performance (pp. 43–56). Thorofare, NJ: Slack.
Bedell, G. M., Haley, S. M., Coster, W. J., & Smith, K. W.
useful information for the practitioner who is try- (2002). Developing a responsive measure of change for
ing to translate assessment scores into “real-life” pediatric brain injury rehabilitation. Brain Injury, 16,
implications. 659–671.
Brown, M., & Gordon, W. A. (2004). Empowerment in
measurement: “Muscle,” “voice,” and subjective quality
Conclusion of life as a gold standard. Archives of Physical
Medicine and Rehabilitation, 85 (Suppl. 2), S13–S20.
Coster, W. J., Deeney, T., Haltiwanger, J., & Haley, S. M.
This chapter has discussed a series of important (1998). School function assessment. San Antonio, TX:
questions to ask about the assessments used in The Psychological Corporation/Therapy Skill Builders.
practice and reported in clinical research. They all, Coster, W. J., Haley, S. M., Andres, P., Ludlow, L., Bond,
in some way, concern validity, that is, the meaning T., & Ni, P. (2004). Refining the conceptual basis for
rehabilitation outcome measurement: Personal care
that can justifiably be assigned to numbers gener- and instrumental activities domain. Medical Care,
ated by our instruments. None of these questions 42 (Suppl. l), I-62–I-72.
Coster, W. J., Haley, S. M., Ludlow, L. H., Andres, P. L., & Laditka, S. B., & Jenkins, C. L. (2001). Difficulty or
Ni, P. S. (2004). Development of an applied cognition dependency? Effects of measurement scales on disabil-
scale for rehabilitation outcomes measurement. ity prevalence among older Americans. Journal of
Archives of Physical Medicine and Rehabilitation, 85, Health & Social Policy, 13(3), 1–15.
2030–2035. Lai, S. M., Perera, S., Duncan, P. W., & Bode, R. (2003).
Cummins, R. A. (1997). Self-rated quality of life scales for Physical and social functioning after stroke: Com-
people with an intellectual disability: A review. Journal parison of the Stroke Impact Scale and Short Form-36.
of Applied Research in Intellectual Disabilities, 10, Stroke, 34, 488–493.
199–216. Lai, S. M., Studenski, S., Duncan, P. W., & Perera, S.
Dijkers, M. P., Whiteneck, G., & El-Jaroudi, R. (2000). (2002). Persisting consequences of stroke measured by
Measures of social outcomes in disability research. the Stroke Impact Scale. Stroke, 33, 1840–1844.
Archives of Physical Medicine & Rehabilitation, 81 Laliberte-Rudman, D., Yu, B., Scott, E., Pajouhandeh, P.
(Suppl 2), S63–S80. (2000). Exploration of the perspectives of persons with
Duncan, P. W., Jorgensen, H. S., & Wade, D. T. (2000). schizophrenia regarding quality of life. American
Outcome measures in acute stroke trials: A systematic Journal of Occupational Therapy, 54, 137–147.
review and some recommendations to improve practice. Law, M. (2002). Participation in the occupations of every-
Stroke, 31, 1429–1438 day life. American Journal of Occupational Therapy,
Duncan, P. W., Lai, S. M., Bode, R. K., Perera, S., DeRosa, 56, 640–649.
J. & the GAIN Americas Investigators. (2003). Stroke Liang, M. H. (2000). Longitudinal construct validity:
Impact Scale-16: A brief assessment of physical func- establishment of clinical meaning in patient evaluative
tion. Neurology, 60, 291–296. instruments. Medical Care, 38, 84–90.
Duncan, P. W., Lai, S. M., & Keighley, J. (2000). Defining Mahoney, F. I., & Barthel, D. W. (1965). Functional evalu-
post-stroke recovery: implications for design and inter- ation: The Barthel Index. Maryland State Medical
pretation of drug trials. Neuropharmacology, 39, Journal, 14, 61–65.
835–841. Mancini, M. C., & Coster, W. J. (2004). Functional pre-
Duncan, P. W., Wallace, D., Lai, S. M., Johnson, D., dictors of school participation by children with dis-
Embretson, S., Laster, L. J. (1999). The Stroke Impact abilities. Occupational Therapy International, 11,
Scale Version 2.0: Evaluation of reliability, validity, and 12–25.
sensitivity to change. Stroke, 30, 2131–2140. Messick, S. (1995). Validity of psychological assessment:
Garwood, S. G. (1982). (Mis) use of developmental scales Validation of inferences from person’s responses and
in program evaluation. Topics in Early Childhood performances as scientific inquiry into score meaning.
Special Education, 1(4), 61–69. American Psychologist, 50, 741–749.
Gliner, J. A , Morgan, G. A., & Harmon, R. J. (2001). Meyers, A. R., & Andresen, E. M. (2000). Enabling our
Measurement reliability. Journal of the American instruments: Accommodation, universal design, and
Academy of Child & Adolescent Psychiatry, 40(4), access to participation in research. Archives of Physical
486–488. Medicine & Rehabilitation, 81 (Suppl. 2), S5–S9.
Gregory, R. J. (2000). Psychological testing (3rd ed.). Morgan, G. A., Gliner, J. A., & Harmon, R. J. (2001).
Boston: Allyn & Bacon. Guide for the uniform data set Measurement validity. Journal of the American Acad-
for medical rehabilitation (Including the FIM instru- emy of Child & Adolescent Psychiatry, 40(6), 729–731.
ment), Version 5.1 (1997). Buffalo: State University of Portney, L. G., & Watkins, P. (2000). Foundations of clini-
New York at Buffalo. cal research (2nd ed.). Upper Saddle River, NJ:
Haley, S. M., Coster, W. J., & Binda-Sundberg, K. (1994). Prentice-Hall.
Measuring physical disablement: The contextual chal- Rankin, J. (1957). Cerebral vascular accidents in patients
lenge. Physical Therapy, 74, 443–451. over the age of 60. II. Prognosis. Scottish Medical
Haley, S. M., Coster, W. J., & Ludlow, L. H. (1991). Pedia- Journal, 2, 200–215.
tric functional outcome measures. In K. Jaffe (Ed.), Rogers, J. C., Holm, M. B., Beach, S., Schulz, R., Cipriani,
Pediatric Rehabilitation. Physical Medicine & Rehabi- J., Fox, A., & Starz, T. W. (2003). Concordance of four
litation Clinics of North America, 2(4), 689–723. methods of disability assessment using performance in
Haley, S. M., Coster, W. J., Ludlow, L., Haltiwanger, J., & the home as the criterion method. Arthritis &
Andrellos, P. (1992). Pediatric Evaluation of Disability Rheumatism, 49, 640–647.
Inventory (PEDI). Boston, MA: Boston University Ronen, G. M., Rosenbaum, P., & Law, M. (1999). Health-
Center for Rehabilitation Effectiveness. related quality of life in childhood epilepsy: the results
Iyer, L. V., Haley, S. M., Watkins, M. P., & Dumas, H. M. of children participating in identifying the components.
(2003). Establishing minimal clinically important dif- Developmental Medicine and Child Neurology, 41,
ferences for scores on the Pediatric Evaluation of 554–559.
Disability Inventory for inpatient rehabilitation. Ronen, G. M., Streiner, D. L., Rosenbaum, P., & the
Physical Therapy, 83, 888–898. Canadian Pediatric Epilepsy Network. (2003). Health-
Jette, A. M., Haley, S. M., Coster, W. J., Kooyoomjian, J. related quality of life in children with epilepsy:
T., Levenson, S., Heeren, T. et al. (2002). Late Life Development and validation of self-report and parent
Function and Disability Instrument: I. Development proxy measures. Epilepsia, 44, 598–612.
and evaluation of the disability component. Journal of Rothstein, J. M. (2001). Sick and tired of reliability?
Gerontology: Medical Sciences, 57A, M209–M216. Physical Therapy, 81, 774–775.
Juniper, E. F., Guyatt, G. H., Feeny, D. H., et al. (1996). Rothstein, J. M. (2002). Switching from autopilot. Physical
Measuring quality of life in children with asthma. Therapy, 82, 542–543.
Quality of Life Research, 5, 35–46. Rothstein, J. M. (2003). Living with error. Physical
Kirshner, B., & Guyatt, G. H. (1985). A methodological Therapy, 83, 422–423.
framework for assessment of health and disease. Wade, D. T. (2003). Outcome measures for clinical rehabil-
Journal of Chronic Disease, 38, 27–36. itation trials: Impairment, function, quality of life, or
value? American Journal of Physical Medicine and assessment and outcome measures for the rehabi-
Rehabilitation, 82 (10, Suppl.), S26–S31. litation health professional. Gaithersburg, MD
World Health Organization (2001). International Classifi- Aspen.
cation of Functioning, Disability, and Health. Geneva, *Finch, E., Brooks, D., Stratford, P. W., Mayo, N. E.
Switzerland: Author. (2002). Physical rehabilitation outcome measures: A
Young, N. L., Williams, J. I., Yoshida, K. K., & Wright, J. guide to enhanced clinical decision making (2nd ed.)
G. (2000). Measurement properties of the Activities (with CD-ROM). BC Decker: Hamilton, Ontario.
Scale for Kids. Journal of Clinical Epidemiology, 53, Law, M., Baum, C., & Dunn, W. (2001). Measuring occu-
125–137. pational performance: supporting best practice in
Young, N. L., Yoshida, K. K., Williams, J. I., Bombardier, occupational therapy. Thorofare, NJ: Slack.
C., & Wright, J. (1995). The role of children in report- Law, M., King, G., MacKinnon, E., Russell, D., Murphy,
ing their physical disability. Archives of Physical C., & Hurley, P. (1999). All about outcomes (CD-
Medicine & Rehabilitation, 76, 913–918. ROM). Thorofare, NJ: Slack.
*These resources include actual assessments.
RESOURCES Structured Review Forms for Evaluating
Listings and Samples of Assessments Assessments
Asher, I. E. (1996). Occupational therapy assessment M. Law (Ed.), (2002). Evidence-based rehabilitation: a
tools: An annotated index (2nd ed.). Bethesda, MD: guide to practice. Thorofare, NJ: Slack.
American Occupational Therapy Association. Forms are also available on the CanChild web site:
*Dittmar, S. S., & Gresham, G. E. (1997). Functional http://www.fhs.mcmaster.ca/canchild
S E C T I O N 4
Statistical Analysis
C H A P T E R 1 5
Making Meaning From Numbers:

Measurements and Descriptive Statistics
Machiko R. Tomita
Numbers are tools for making meaning. They are User-friendly computer software now handles this
the basic tools of mathematics and represent for- task. Moreover, for complicated statistics, most
mal ways of characterizing and making sense of investigators collaborate with statisticians and bio-
the world. Statistics, a branch of mathematics, statisticians.
involves systematically converting observations It is important for both consumers and users of
into numbers, and then using those numbers to statistics to have a good understanding of:
characterize and to draw inferences about the
• Which statistics should be used to answer
world. Because the procedures for translating
research questions or test hypotheses,
observations into numbers and for manipulating
• How the statistic goes about reaching conclu-
numbers to answer questions and make decisions
sions (i.e., what is the underlying logic of the sta-
are formal and systematic, they are especially
tistic), and
useful in achieving an objective approach to con-
• How statistical results should be interpreted.
ducting science. The purpose of this chapter is to
introduce the use of statistics in research and to For these reasons, this and subsequent chapters
overview basic concepts of statistics. dealing with statistics will emphasize the concep-
Researchers in occupational therapy are always tual formulas of statistical tests, not the calculation
a bit discouraged when they hear therapists or formulas. Moreover, the discussions will focus on
students comment that, when reading research the underlying logic of each statistic, what kinds of
articles, they skip the statistical analysis section, or information the statistic gives, and how to make
find it difficult to understand. Nonetheless, it is sense of the statistic’s results.
recognized that many who are drawn to occupa- A good basic grasp of statistics is necessary for
tional therapy because of its humanistic and even simple quantitative research. When one
pragmatic orientation find mathematics and grasps the underlying logic of statistical analysis,
statistics both distant and obscure. The intent of it is possible to design better research, to ask bet-
this and subsequent chapters is to illustrate that ter questions, and to formulate correct hypotheses.
statistics are simply pragmatic tools and that Understanding the basics of statistics is also essen-
their use can help us understand phenomena and tial to being able to read research articles critically
make important decisions that address humanistic and to draw conclusions about what the research
ends. findings mean for one’s practice.
Statistics are not only central tools of quantita-
tive science; they are also important for informing
What Is Statistics?
decision making in occupational therapy practice.
That is, statistical analysis is also the basis on Simply said, statistics is a method for generating
which therapists may decide such things as whether the best information from available data. Statistics
one practice approach is better than another, how is a branch of applied mathematics that deals with
much confidence to have in the results of an assess- the collection, description, and interpretation of
ment, whether a client’s performance is sufficient quantitative data, and with the use of probability
for a particular functional task, and how long a theory, estimates population characteristics from
therapist may need to continue an intervention to samples. In a quantitative study, the investigator
achieve certain results. ordinarily wants to draw conclusions about all the
For the new researcher who uses statistics it is people who share certain characteristics. However,
no longer necessary to calculate difficult math. it is usually impossible to study all of them, so
214 Section 4 Statistical Analysis
investigators study only a selected subgroup of necessary for answering the question. Data consist
these people. of variables; a variable is a characteristic being
The larger group of people about whom an measured that varies among the persons, objects,
investigator wants to draw conclusions is called the or events being studied. For example, age is a
population and the measures of their characteristics variable because it can have many values. If age
are called parameters. The subgroup of people who is measured as “Young” or “Old,” one can say the
are actually studied is called the sample, and the age has two levels: young and old. Functional
measures of their characteristics are called statis- ability is a variable because it may vary from
tics. In most studies, the population parameters are “Completely Independent” to “Completely
unknown; therefore, they must be estimated from Dependent.” If an investigator introduces two
the sample statistics. When investigators objec- types of occupational therapy interventions and
tively collect some type of information from the compared them with a control group, then the
sample (i.e., data), they can apply a statistical pro- intervention is a variable and it has three levels:
cedure to the data to estimate the population “occupational therapy intervention 1, occupational
parameter. therapy intervention 2, and no intervention.”
Although the levels of all variables described in
the preceding paragraph use words, they can be
How Do Investigators Use Statistics?
expressed as numerals. For instance, it is possible
Statistics are used in three ways. The first is to to assign a “1” for “Young” and a “2” for “Old.” In
describe information of the sample meaningfully this case, the numerals are used to indicate to
and efficiently by using smaller sets of data. For which category of young or old the person
this, one uses descriptive statistics, and one can belongs. Similarly, numbers could be assigned to
typically find them in a demographic table or a indicate where a person falls on the continuum
summary table of most published quantitative from completely independent to completely
research articles. The second is to generalize the dependent. Therefore, measurement is the process
observed information of the sample to the popula- of assigning numerals to observed people, objects,
tion with a degree of confidence. For this one uses or events according to a set of rules for the purpose
inferential statistics, and this process is explained of understanding them objectively without ambi-
in Chapters 16 and 17. The third is to identify asso- guity. These rules designate how numbers are to be
ciations, relations, and differences among the sets assigned, reflecting amount or values and units of
of observations, and inferential statistics are also measurements.
used for this purpose. Various statistical methods
for this purpose are described in Chapter 17. The
primary purpose of quantitative research is to esti-
Types of Measurement Scales
mate the population parameters; therefore, the The rules that define how numbers are assigned to
investigators report the sample statistics that observed people, objects, or events determine the
the generalization is based on and the accuracy levels (scale) of measurement. There are four lev-
of the sample that represents the value of the pop- els, or scales, of measurement: nominal, ordinal,
ulation parameter. interval, and ratio. Whenever a variable is meas-
ured it requires using one of the four scales.
Among the four measurement scales, nominal and
Measurement Scales ordinal scales are called categorical scales since
they use only discrete numbers. A discrete variable
can be described only in whole units for data col-
What Is Measurement?
lection. Examples are the number of children in a
Investigators collect data from the sample for a family, the number of times a person has been hos-
research study to answer the research question. pitalized, and the number of mistakes a person
Therefore, data should provide the information makes on a test of cognition, which are expressed
as 1, 2, 3, etc. When categorical scales are used to
The word statistics originally came form the

phrase “lecture about state affairs,” meaning the The traditional classification of levels of meas-
analysis of data about the state. In the early 19th urement was developed by Stevens (1946) and is
century it became the collection and classifica- still widely used: nominal, ordinal, interval, and
tion of data. ratio.
Chapter 15 Making Meaning From Numbers: Measurements and Descriptive Statistics 215
measure a dependent variable, nonparametric sta- Most demographic tables in research studies
tistics are used. include variables measured by a nominal scale.
Interval and ratio scales are called continuous Table 15.1 illustrates hypothetical nominal data for
scales; they can use discrete numbers or continu- the characteristics of two groups of assistive tech-
ous numbers. A continuous variable can take on nology workshop participants. The variables meas-
any value along a continuum within a defined ured by a nominal scale in this table are sex, race,
range. For instance, a person’s weight could be and group affiliation.
60.455 kg and blood pressure can be 120.051. When a variable is a construct that is an abstract
Parametric statistics are used for a dependent vari- variable such as pain, physical functioning level,
able measured by a continuous scale. and cognition, a series of nominal level items may
be used together to provide a measure of the vari-
able. In this case, each item may have a dichoto-
Nominal Scale
mous choice to indicate the presence or absence of
A nominal scale is a categorical scale and is the some characteristic. Each item is thus measured by
lowest scale of measurement used to classify vari- a nominal scale. However, to interpret the variable,
ables. Although numerals are assigned for each a total score based on all items is calculated. The
category, they have no numerical values in that resulting total score is ordinarily considered to be
they “name” the characteristic but do not suggest an interval scale because of its wide range of
an order, an amount, or value. For example, inves- scores. Examples are the Observed Tasks of Daily
tigators may code types of disability as (1) physi- Living-Revised (OTDL-R) (Diehl, Marsiske,
cal, (2) cognitive, and (3) mental, or assign male ⫽ Horgas, & Saczynski, 1998) and Sickness Impact
1 and female ⫽ 2, but the numbers are merely Profile (Gilson et al., 1975). The OTDR-L is a
labels and it does not make sense, for example, to behavioral measure of everyday competence that
add these numbers, or to calculate an average. requires a client to perform a number of observ-
Finally, nominal data cannot be converted to any able actions in response to a question by the tester.
other scales whereas other scales can be converted Scores are recorded as correct or incorrect. Then a
to nominal scales. score of 1 is given to a correct response and 0 to an
Further examples of nominal scales are type incorrect response; the possible total score range
of occupation (work, play, or self-care), race of 0 through 26.
(African-American, Caucasian, Hispanic, Native Sometimes instruments use weighted items to
American, etc.), work status (full time, part time, arrive at a total score; weighting allows items to
retired, etc.), and the dichotomous answer, “Yes” contribute different amounts to the total score
or “No.” Nominal scale categories have to be based on some criterion such as their importance,
mutually exclusive so that no person or object can severity, or difficulty. For example, The Sickness
be assigned to more than one category. Impact Profile (SIP)-Dysfunction consists of 45
Table 15.1 Characteristics of Assistive Technology Workshop Participants for Each Group (N ⫽ 37)
Group A Group B
n % n %
Sex
Men 4 19 4 25
Women 17 81 12 75
Race
Caucasian 8 38 9 56
African-American 5 24 1 6
Hispanic 8 38 6 38
Mean SD Mean SD
Other variables
Age 56.0 (10.0) 44.7 (8.5)
Years since onset 10.7 (3.3) 12.1 (6.3)
Fatigue Severity Scale Score 5.3 (.9) 5.3 (.9)
items that use dichotomous scales to determine a the Center for Epidemiology Study for Depression
physical functioning level. A client will answer (CES-D) (Gilson et al., 1975). This scale requires
yes/no for each item and items have been assigned frequency of the feelings experienced during the
various weights to indicate the severity of the past week. It consists of 20 items, and for each item
physical problem. For example, for the statement there are four choices: Rarely (less than 1 day),
“I am getting around only within one building” if Some of the time (1 to 2 days), Moderately (3 to 4
a subject answers “Yes” the score of 86 is given, days), and Mostly (5 to 7 days). These choices
and if the response is “No” the score of 0 is given. have rankings assigned of 0, 1, 2, and 3 respec-
Moreover, for the statement “I walk shorter dis- tively. To interpret the depression level, the total
tances or stop to rest often” the “Yes” response scores of 20 items will be calculated, producing
weighs 48. The total weighted scores for all 45 rankings 0 to 60. The total score is considered to
items divided by all weights together determine be continuous (interval), thus allowing arithmetic
the physical functioning level. calculations.
Other investigators prefer not to consider such
Ordinal Scale scales as interval in nature since they are com-
posed of only ordinal rankings. A method called
An ordinal scale requires one to rank-order the cat- Rasch analysis that converts items that were meas-
egories. This is considered the second lowest scale ured by an ordinal scale to interval scale (Bond &
and the data measured using an ordinal scale can Fox, 2001) is discussed in Chapter 13. Increas-
be converted to that of a nominal scale. Data are ingly, investigators prefer to use Rasch methods to
organized into adjacent categories in a linear fash- convert scores based on ranked items to true inter-
ion, ascendant or descendant. Two examples are val scales rather than assuming they “approxi-
health status ranked as (1) Very Poor, (2) Poor, (3) mate” an interval scale. The Functional Index
Fair, (4) Good, and (5) Very Good, and pain levels Measure (FIM) that measures Activities of Daily
ranked as (1) No pain, (2) Mild, (3) Moderate, and Living (ADL) is an example of an instrument cre-
(4) Severe (Jette, 1980). When numbers are used to ated using Rasch analysis (Wright & Linacre,
indicate ranking, they refer only to the relative 1989).The FIM uses the following ordinal scale for
position of the category along the continuum. The each of 18 items: (7) complete independence, (6)
distances between the ranks are not necessarily modified independence, (5) supervision, (4) mini-
equal. Using the example of the pain level, the dis- mal assist, (3) moderate assist, (2) maximal assist,
tance between (1) No pain and (2) Mild may not be and (1) total assist (Center for Functional
the same as the distance between (3) Moderate and Assessment Research/Uniform Data System for
(4) Severe. Investigators often tabulate ranking of Medical Rehabilitation, 1994). Rasch analysis
an item and report the mean rank. For instance, converts the ordinal data generated from complet-
Mitcham, Lancaster, and Stone (2002) reported a ing the scale to an interval measure. However, in
study of the effectiveness of occupational therapy practice, many persons continue to use the raw
faculty development workshops. In this study, 106 total score to avoid the necessity of computer
participants ranked eight elements of the workshop analysis of the raw scores to generate a true
on a 10-point scale. For example, the mean rank interval measure. Linacre (1995, 1997) developed
for teaching skills of leader was 9.1 (the highest an alternative to computer scoring for obtaining
score); handouts, 8.4; and opportunity for career measures. He introduced the idea of a keyform—
development, 7.7 (the lowest score). a paper-and-pencil method that generates inter-
Strictly speaking, if a dependent variable is val measures from the ordinal raw data of the
measured by an ordinal scale, the data should be FIM (Linacre, 1997). More recently Kielhofner,
analyzed by nonparametric statistics. However, Dobria, Forsyth, and Basu (2004) used this
there are research studies in which data gathered same methodology to create keyforms for the
by ordinal scales were treated as interval scale Occupational Performance History Interview—
data, and parametric statistics were applied. When 2nd version (OPHI-II). These keyforms allow
the intervals between adjacent ranks can be occupational therapists to obtain instantaneous
assumed reasonably equal and there are many measures from raw rating scale data. The avail-
items within the variable, the ranking of each item ability of such methods for achieving interval
can be summed to produce many rankings. In this scores from raw data may make it less common in
case, many investigators will consider the total the future that investigators will treat data gener-
score to “approximate” a continuous (interval) ated from ordinal rankings as “approximately”
scale. An example is a depression instrument called interval.
Nonetheless, there have been long statistical assumptions are met, parametric statistics can be
debates on appropriateness of treating ordinal data applied. Some examples of ratio level scales are
as interval data (Guildford, 1954; Kerlinger, 1973; height, weight, age, range of motion, distance,
Nunally, 1978; Merbitz, Morris, & Grip, 1989; strength, blood pressure, duration of exercise,
Velleman & Wilkinson, 1993). Portney and handwriting speed, amount of food consumed, and
Watkins (2000) take a stand that investigators number of falls. In Table 15.1, “Age” and “Years
should treat ordinal data as such, but if the investi- since onset” are measured by ratio scales. Vari-
gator can make a reasonable argument that the ables measured by interval and ratio scales are
ordinal scale approximates an interval scale, the generally called continuous variables. However,
error involved is defensible. strictly speaking, among variables measured by
ratio or interval scales, there are two types of vari-
ables: continuous variables (e.g., duration of exer-
Interval Scale
cise) and discrete variables (e.g., number of falls).
An interval scale is the second highest measure- All nominal and ordinal scales are discrete vari-
ment scale; therefore, the data measured by an ables.
interval scale can be converted to a nominal or It is useful to remember that if possible, inves-
an ordinal scale. The distance between any two tigators should use a higher level measurement
adjacent units of measurements or intervals is the scale to measure people, objects, and events. The
same, but not their proportionate magnitude, since higher levels of measurement can always be con-
there is no meaningful zero point. Zero can be verted to lower levels of measurements, if neces-
used on interval scales, but it does not mean sary, but the converse is not possible.
nonexistence or absence. A good example is tem- For instance, an investigator who collects data
perature, either in Fahrenheit or Centigrade. Zero on age using a nominal scale such as “old” and
degrees Centigrade is 32 “young” will never be
degrees Fahrenheit, and able to categorize them
zero degrees Fahrenheit The higher levels of meas- further or recategorize
is negative 17.8 degrees urement can always be them. If on the other
Centigrade. Zero is hand, the investigator
merely a line on the converted to lower levels uses a ratio scale (i.e.,
thermometer. Therefore, of measurements, if neces- years of age) this data
temperature is an interval could be converted to
scale and does not indi- sary, but the converse is ordinal scales (e.g.,
cate absolute magnitude not possible. below 10, 11–20, 21–30,
of an attribute. Within etc.) or to nominal scales
the Fahrenheit or Centi- (young versus old). As a
grade temperature scale, the distance between 21 final point, although researchers use measurement
and 31 degrees is the same as that between 61 and scales to measure both independent and dependent
71 degrees. However, it is incorrect to say 60 variables, the type of scale used to measure the
degrees is three times hotter than 20 degrees, dependent variable, not the independent variable,
since zero does not represent the absence of is one of the determining factors for the use of
temperature. A dependent variable measured parametric versus non-parametric statistics.
using an interval scale can be analyzed using
parametric statistics if other statistical assumptions Types of Statistics in Relation
are met. to Measurement Scales
Ratio Scale The scale that measures the dependent variable
The ratio scale is the highest measurement scale together with other statistical assumptions deter-
and the data gathered using a ratio scale can be mines which of two types of statistics are used:
converted to other lower measurement scales. It is parametric statistics versus nonparametric statis-
a scale in which any two adjacent values are the tics. Here, without explaining much detail, it can
same distance apart (just like an interval scale) and be pointed out that many researchers prefer to use
in which there is a true zero point (unlike an inter- parametric statistics over nonparametric statistics
val scale). All arithmetic computations are possi- for two reasons:
ble with a ratio scale and if the dependent variable • With use of parametric statistics, a researcher can
is measured by a ratio scale, and if other statistical be more confident to say one group’s characteris-
tics are significantly different from the others, Frequency Distribution

and
• There are more parametric statistics than non- Data summarized using a frequency distribution
parametric statistics, so in that sense, parametrictypically present individual percentages and cumu-
statistics represent a wider range of tools for lative percentages as well. The notation of fre-
answering research questions. quency is (f). Table 15.2 is a Statistical Package for
the Social Sciences (SPSS) (version 12.0, 2003)
The details of these statistics are explained in printout of frequency distribution of the total num-
the following chapters. Figure 15.1 summarizes ber of medications taken among 72 home-based
the relationship between use of types of statistics frail elderly. The capital N indicates the total sam-
and measurement scales. ple size and lowercase n indicates the sample size
of each subcategory.
The first column contains the number of med-
Descriptive Statistics ications taken, the second column is number or
frequency of people who reported the number of
When data are collected, the initial form of the medications taken, the third column is percentage
data is called raw data. To understand characteris- of the frequency of the total sample size, and the
tics of people, objects, last column is a cumula-
and events, the raw data tive percentage up to the
must be organized. Even The type of scale used to particular value. Note in
though the main purpose measure the dependent this data no one took 2 or
of performing statistical 11 medications; there-
analyses is to infer popu- variable, not the independ- fore 2 and 11 are absent
lation characteristics that ent variable, is one of the in the first column.
are likely to be the nature The same data can be
of entire people for your
determining factors for the grouped into classes in
inquiry (which involves use of parametric versus which each class repre-
inferential statistics), it is sents a unique range of
important to summarize
non-parametric statistics. scores within the distri-
the sample characteris- bution. The classes are
tics on which the inferences are based. This mutually exclusive or there are no overlapping val-
involves descriptive statistics. Presentation of ues and are exhaustive within the range of scores
descriptive statistics includes graphs and fre- obtained. If the range of each class is two, the
quency distribution tables as well as measures of grouped frequency distribution of Table 15.2 will
central tendency and variability. form five classes (0–2, 3–5, 6–8, 9–11, and 12–14)
Measurement Scales Descriptive Inferential*

Statistics Statistics
Ratio Scale
Continuous Central Tendency Parametric
Scale Interval Scale Variability Statistics
Ordinal Scale
Categorical Frequency Nonparametric
Scale Statistics
Nominal Scale
*Type of inferential statistics to be used for analysis is determined by:

• Type of measurement scale used for the dependent variable, Figure 15.1 Relationship be-
• Type of distribution of scores of the dependent variable, and tween measurement scales and
• Homogeneity of groups in terms of dependent variable. types of statistics for descriptive
In addition, a random selection or assignments of sample is an assumption. and inferential statistics.
Table 15.2 Frequency Distribution of Total Number of Medications Taken Among Home-Based Frail
Elderly (N ⫽ 72)
n Frequency Percentage Valid Percentage Cumulative Percentage
Valid 0 6 8.3 8.3 8.3
1 2 2.8 2.8 11.1
3 9 12.5 12.5 23.6
4 9 12.5 12.5 36.1
5 12 16.7 16.7 52.8
6 5 6.9 6.9 59.7
7 7 9.7 9.7 69.4
8 8 11.1 11.1 80.6
9 6 8.3 8.3 88.9
10 4 5.6 5.6 94.4
12 2 2.8 2.8 97.2
13 1 1.4 1.4 98.6
14 1 1.4 1.4 100.0
Total 72 100.0 100.0
and the frequencies are 8, 30, 20, 10, and 4, respec- graph as shown in Figure 15.2a. The x-axis (hori-
tively. When the intervals of the classes are the zontal) is the number of medications taken and
same, the comparisons of classes are easy. the y-axis (vertical) is the measured variable (i.e.,
Investigators will typically report intervals of frequencies of persons who take the number of
classes when individual classes contain a few or medications). The advantage of this graphical
zero frequency. In this case classes are clustered to presentation is the obvious efficiency in identify-
reveal a more meaningful interpretation of the ing the shape and characteristics of distribution.
data. For example, in the study titled, The largest number of elders (n ⫽ 12) took five
“Development of a standardized instrument to medications and the fewest number of elders (n ⫽
access computer task performance,” (Dumont, 0) took 2 or 11 medications.
Vincent, & Mazer, 2002) the table of sample char- The pie graph, an alternative to the bar graph, is
acters consists only of frequencies and percent- a circle that has been partitioned into percentage
ages. Some of the types of information included distribution and presented in Figure 15.2b. For
are age in five classes (17–19, 20–29, 30–39, detailed information on constructing a pie chart,
40–49, and 50⫹), occupation in three categories refer to Wallgren, Wallgren, Persson, Jorner, and
(homemaker or retired, student, and employed), Haaland (1996).
frequency of computer use (rarely, occasion-
ally, and regularly), touch-typing method (yes Graphical Presentation of Frequency
and no).
Data for Continuous Variables
Graphical Presentation of Frequency There are several ways to present frequency data
for continuous variables. They are a histogram, a
Data for Discrete Variables
frequency polygon, and the stem-and-leaf plot. A
There are several ways to graphically present a histogram is a bar graph without space between
frequency distribution of data. Since the number of bars. A frequency polygon is a line plot, where
medications a person takes is a whole number, it is each point on the line represents frequency or per-
a discrete value such as three or five medications. centage. Figure 15.3 is a SPSS (2003) printout of a
Discrete variables are presented using a bar graph frequency polygon of grip strength by sex of frail
or chart. A space separates each bar to emphasize elders in kilograms. In this figure, the two lines
the noncontinuous nature of the data. All nominal represent females and males.
and ordinal data and some interval and ratio data The stem-and-leaf plot is most useful for pre-
use the bar graph. For instance, the information senting the pattern of distribution of a continuous
contained in Table 15.2 can be presented in a bar variable. There are two parts in this plot: stem and
14 A.
12
10
2
Count
0
.00 3.00 5.00 7.00 9.00 12.00 14.00
1.00 4.00 6.00 8.00 10.00 13.00
Number of Medications Taken
14.00 B.
13.00
12.00 .00
10.00 1.00
9.00
3.00
8.00
4.00
Figure 15.2 (a) Bar graph of

7.00 frequency of medications
taken. (b) Pie chart of fre-
6.00 5.00 quency of medications taken.
leaf. Table 15.3 shows raw data of age of veterans Mean

with spinal cord injury and its stem-and-leaf plot.
The stem in this plot is segmented by every 10 The mean is the sum of individual scores (X)
years and the leaf represents additional ages. divided by the number of sample size (n). The
Examples of interpretation of this plot are the fol- symbol used to represent a population mean is the
lowing: there are two veterans with spinal cord Greek letter, ␮ (read as mu) and a sample mean is
injury who are in the teens, 18 and 19 years old. represented by 苶
X or M. The formula for calculating
Therefore, in the stem of 10, 8 and 9 are listed in the sample mean from raw data is
the leaf. Similarly, there are six veterans who are
in the 40s. Since the leaf contains two zeros, two ΣX
X⫽ ᎏ
苶 n
veterans are 40 years old, and others are 42, 44, 45,
and 47 years old.
where Σ (read as sigma) indicates summation. If
the values of X are 2, 4, 5, 6, 3, 7, 4, and 10, the
Measures of Central Tendency sum of all values is 41 and dividing it by the num-
When a variable is measured using an interval or ber of cases, 8, equals 5.125. The mean is the most
ratio scale, measures of central tendency or aver- useful average because it is most reliable among
ages are often used to describe the typical nature of measures of central tendency. When samples are
the data. There are three indices: the mean, the drawn randomly and repeatedly from the popula-
median, and the mode. tion, the means of those samples would fluctuate
40
30
Frequency
20
10
Sex
Female
Male
0
0 7 13 19 25 31 37 50 62
Figure 15.3 Frequency poly- 3 10 16 22 28 34 40 55 80
gon for grip strength by sex
(N ⫽ 319). Grip Strength for Dominant Hand
less than the mode or median. The mean can be scores is even, there is no single middle score. In
used for arithmetic calculations and is used most that case, the median is calculated by taking the
often for statistical analyses. average of the two middle scores. Therefore, the
4⫹5
Median median for the value is 4.5 (⫽ ᎏ ). When
2
The median is the middle position of measurement there is an extreme score in a dataset, the median
in a set of ranked scores. Therefore, the values is a better index than the mean because the median
should be arranged in an ascending or descending is not influenced by the score. In the preceding
order. Using the same preceding example, the val- example, if 100 replaces the value 10, the new
ues are rearranged: 2, 3, 4, 4, 5, 6, 7, and 10. The mean will be 16.375 owing to the extreme score of
midpoint is between 4 and 5. When the number of 100, but the median is still 4.5.
Table 15.3 Age of Veterans with Spinal Cord Injury (Age 18–50): Raw Data and Stem-and-Leaf Plot
Raw Data in Chronological Order Stem-and-Leaf Plot
18 24 27 29 30 32 34 37 Stem Leaf
19 25 27 29 30 32 34 37 10 89
20 25 27 29 30 32 34 37 20 012233344444555556677
21 25 27 29 30 32 35 38 20 777777778888888999999
22 25 27 29 30 32 35 38 20 99999
22 25 28 29 30 32 35 39 30 000000000001111122222
23 26 28 29 30 33 35 39 30 223333334444555555666
23 26 28 29 31 33 35 40 30 77778899
23 27 28 29 31 33 35 40 40 002457
24 27 28 30 31 33 36 42
24 27 28 30 31 33 36 44
24 27 28 30 31 33 36 45
24 27 29 30 32 34 37 47
Mode the median is in the center, and the mean is toward

the tail (Figure 15.4c). The median is always in the
The mode is the score that occurs most frequently middle between the mean and the mode, and the
in a dataset. In the example discussed in the previ- mean is pulled toward the tail. Therefore, if two
ous paragraph, the mode is 4. In the set, there is central tendencies are reported, one can identify
only one mode; therefore, the data is unimodal. If the skewness of the data distribution.
there are two modes, the distribution is considered
bimodal. Some distributions of variables do not Measures of Variability
have a mode.
To understand the sample characteristics, in addi-
Location of Central Tendency tion to the distribution and the central tendency,
it is important to understand variability. Variability
in Various Distributions
is the dispersion of values of any variable in a
Locations of central tendencies are affected by the dataset. For example, consider a study that aims to
distribution of values in the data. When the distri- identify the difference in effectiveness in teaching
bution of values is normal, the mean, median, and methods for statistics: in class A the instructor
mode are all in the center of the distribution of the used a conventional teaching style only, and in
values (Figure 15.4a). A normal distribution is uni- class B the instructor used an interactive method.
modal, and both sides of the mean have symmetri- The scores for the final exam in statistics in class
cal shapes (a more detailed discussion of normal A (n ⫽ 7) were: 98, 72, 85, 88, 65, 93, and 79.
distribution occurs later in this chapter). When the Therefore, the mean was 82.9 and the median was
distribution is skewed to positive (positively 85. In class B (n ⫽ 7) the scores were: 89, 86, 79,
skewed distribution), the tail is toward the right 78, 92, 71, and 85; therefore, the mean was also
side of the graph. In this type of distribution, the 82.9 and the median was 85 as well. Since there is
mode is located in the far left among the three cen- no mode in both classes, values of the central ten-
tral tendencies, the median is in the middle, and dency in these classes are identical. Therefore,
the mean is toward the tail (Figure 15.4b). When both methods appear to be equal in their effective-
the distribution is skewed to negative (negatively ness. However, scores in class A are more widely
skewed distribution), the tail is toward the left side spread than the scores in class B; therefore, there
of the graph, the mode is located in the far right, is a difference in variability. There are five meas-
A. Normal Distribution
Mean
Median
B. Positively Skewed Distribution Mode
C. Negatively Skewed Distribution
Mode Mean
Median
Mode Mean
Median
Figure 15.4 Types of distribution and location of the central tendency. (A) Normal distribution.
(B) Positively skewed distribution. (C) Negatively skewed distribution.
ures to describe variability: range, percentiles, Variance

variance, standard deviation, and coefficient of
variation. Variance is the variability of the individual cases
away from the mean. It is one of the most impor-
tant concepts for statistical analysis, such as
Range Analysis of Variance (ANOVA). A group with a
larger variance has scores that are more dispersed
The range is the difference between the highest
or scattered (i.e., is less homogeneous) than a
and lowest values in the dataset. Using the preced-
group with a smaller variance. The sample vari-
ing example, the range for class A is 98 ⫺ 65 ⫽
ance is symbolized as s2. The population variance
33, and the range for class B is 92 ⫺ 71 ⫽ 21.
is noted as ␴2 and read as sigma squared. The for-
Therefore, the range in class A is larger than that
mula to identify the sample variance is the sum of
of class B, indicating more variability in scores
the squared deviation from the mean scores
of class A. Since range is merely the difference
divided by the degrees of freedom (i.e., the number
between two extreme values and ignores other val-
of cases minus 1):
ues, it is a rough descriptive measure of variability.
Many research papers report the highest or maxi- Σ(X ⫺ X苶)2
s2 ⫽ ᎏᎏ ⫽ SS/df
mum value and the lowest or minimum value, N⫺1
rather than providing the range.
The numerator is called sum of squares and its
abbreviation is SS. The deviation from the mean is
Percentiles squared because otherwise the sum of deviations
from the mean becomes zero. The denominator is
Percentile is a value above and below which a cer-
called degrees of freedom and is denoted as df.
tain percentage of values in a distribution fall and
The degrees of freedom are identified as the sam-
describes a score in relation to other scores in a
ple size minus 1. If one is dealing with the popula-
distribution. Percentiles are symbolized by the let-
tion, the denominator is N but when one is dealing
ter P, with a subscript indicating the percentage
with a sample, N–1 provides a more accurate value
below the score value. Therefore, P75 refers to the
of variability.
75th percentile, which indicates that the particular
The process to calculate variance is shown in
value is higher than 75% of values in the dataset.
Table 15.4 using the final exam scores from the
Quartiles divide a distribution into four equal
statistics classes that were presented in an earlier
parts, or quarters. The 25th percentile is called the
example found in this chapter. For class A, the
first quartile, the 50th percentile the second quar-
variance is 135.81 while for class B, it is 52.48,
tile, and the 75th percentile the third quartile. The
indicating class A is more diverse in the scores
most commonly used interpercentile measure is
than class B. Class B is more homogeneous than
the interquartile range (IQR), which is the distance
class A.
between the 25th and 75th percentiles. In the pre-
vious example, arranging the scores from the min- Standard Deviation
imum to the maximum in class A, the scores are:
65, 72, 79, 85, 88, 93, 98. The 50th percentile is Standard deviation is the most widely used meas-
the median, 85. There are three values below 85 ure of variability. It indicates the average variabil-
(65, 72, and 79). Their median is 72. Their median ity of the individual scores from the mean.
of three values above 85 (88, 93, and 98) is 93. Therefore, the greater the standard deviation, the
Therefore, the interquartile range is 72 to 93. For greater the variability in a dataset. The standard
class B, the interquartile range is 78 to 89. This deviation is symbolized as SD, Sd, sd, or s for a
indicates that class B is less variable or more sample. The sample SD is defined as the square
homogeneous than class A. These interquartile root of variance:
冪莦 Σ(X ⫺ X苶)2
ranges, like the median, are not sensitive to
extreme values and if a distribution is severely s ⫽ 兹苶
S2 ⫽ ᎏᎏ
n⫺1
skewed, it is an appropriate method to indicate
variability. The standard deviation of population is sym-
When comparing two very different subgroup bolized as ␴ (read as sigma) and the denominator
characteristics, one eliminates the participants that in the square root is N, the sample size. The advan-
fall in the interquartile range and instead compares tage of the standard deviation over the variance is
the participants below the first quartile and above that the standard deviation has the same unit of
the 3rd quartile. measurement as the mean, since it is the square
Table 15.4 Process of Calculating Variance for Final Exam Scores in Statistics Classes
Deviation Squares of the Standard
from the Deviation from Sum of Deviation
冪莦 Σ
Mean the Mean Squares Variance (X ⫺ X苶)2
Raw Score (X ⫺ X苶) (X ⫺ X苶)2 Σ(X ⫺ X苶)2 (X ⫺ X苶)2/n-1 ᎏᎏ n-1
For Class A
98 98 ⫺ 82.9 ⫽ 15.1 228.01
72 72 ⫺ 82.9 ⫽ ⫺ 10.9 118.81
85 85 ⫺ 82.9 ⫽ 2.1 4.41
88 88 ⫺ 82.9 ⫽ 5.1 26.01
65 65 ⫺ 82.9 ⫽ ⫺ 17.9 320.41
93 93 ⫺ 82.9 ⫽ 10.1 102.01
79 79 ⫺ 82.9 ⫽ ⫺ 3.9 15.21
SS and S2 814.87 814.87/6 ⫽ 兹135.8
苶1苶 ⫽ 11.76
135.81
For Class B
89 89 ⫺ 82.9 ⫽ 6.1 37.21
86 86 ⫺ 82.9 ⫽ 3.1 9.61
79 79 ⫺ 82.9 ⫽ ⫺ 3.9 15.21
78 78 ⫺ 82.9 ⫽ ⫺ 4.9 24.01
92 92 ⫺ 82.9 ⫽ ⫺ 9.1 82.81
71 71 ⫺ 82.9 ⫽ ⫺ 11.9 141.61
85 85 ⫺ 82.0 ⫽ 2.1 4.41
SS and S2 314.87 314.87/6 ⫽ 兹52.48
苶 ⫽ 7.24
52.48
root of the squared value (variance). Using the deviation is large, it indicates heterogeneity among
example of final scores in two statistics classes the group. Standard deviation is a continuous
described earlier, the standard deviation for class A value. When reporting a standard deviation next to
is 兹苶 135.81 ⫽ 11.76 and that for class B is a mean in descriptive statistics, one uses 1 standard
deviation. Using the example above, 2 standard
兹52.48
苶 ⫽ 7.24 (see Table 15.4). Therefore, class deviations for class A is 23.52. The standard devi-
B is more homogeneous than class A in terms of ation is an important statistic in its own right and
final scores. is also used as the basis of other statistics such as
The standard deviation correlations, standard
of sample data should be errors, and z-scores.
reported with the mean The standard deviation of In the study titled
so that the data can be the sample data should be “Impact of pediatric
summarized according to rehabilitation services
both central tendency and reported with the mean so on children’s functio-
variability. that the data can be summa- nal outcomes” (Chen,
Many studies report Heinemann, Bode,
them in one of the two rized according to both cen- Granger, & Mallinson,
following formats: 82.9 tral tendency and variability. 2004), three summaries
⫾ 11.76 or 82.9 (11.76). of descriptive statistics
The value of standard are presented. One is a
deviation is always positive but if it is zero, it summary table titled “Sample characteristics” in
means there are no variations in distribution, or all frequencies and percentages (Table 15.5) and the
values are the same. When the value of standard second one is a summary table titled “Descriptive
Table 15.5 Sample Characteristics ical therapy, speech pathology, and psychology)
for five types of diagnoses (cerebral palsy, non-
n (%)a traumatic brain injuries, traumatic brain injuries,
Genderb major multiple trauma, and other impairments).
Male 465 (57) This type of visual presentation allows efficient
Female 346 (43) simultaneous comparisons for two independent
Race variables (in the study mentioned earlier, they are
Caucasian 522 (64) types of therapy and types of diagnoses).
African-American 182 (22)
Coefficient of Variation
Hispanic-American 65 (8)
Admit From The coefficient of variation (CV) is an indicator of
Home 255 (31) variability that allows comparisons of different
Acute—Own 151 (19) variables. The CV is the ratio of the standard devi-
Acute—Other 380 (47) ation to the mean, expressed as a percentage:
Pre-Rehabilitation Setting
SD
Home 773 (95) CV ⫽ ( ᎏ ) ⫻ 100
Acute—Own 7 (1)
苶
X
Acute—Other 8 (1) The CV is independent of the units of measure-
Discharge Setting ment.
Home 745 (91) In a study to develop a standardized instrument
Nursing Home 4 (.5) to assess computer task performance (Dumont et
Rehabilitation Facilities 18 (2) al., 2002), the authors described the sample size,
Acute Units 23 (3) the mean, the standard deviation, the range using
Discharge Live With the minimum and maximum scores, and the coef-
Two Parents 314 (39) ficient of variation for the impaired group and the
One Parent 199 (24)
nonimpaired group in terms of tasks involved in a
keyboard and a mouse. For keyboard use, the CV
Relatives 37 (5)
for the nonimpaired group is constantly lower
Foster Care 12 (2)
(32.8–68.5) than the CV for the impaired group
Other (or missing) 252 (31) (40.9–92.2), indicating the impaired group is more
Primary Payer variable in physical capacity than the nonimpaired
Medicaid 263 (32) group. It should be noted that the CV cannot be
Commercial 286 (35) used when the variable mean is a negative number,
HMO 216 (27) since CV is expressed as a percentage.
Private Pay 16 (2)
Graphical Presentation of Central
N ⫽ 814 Tendency and Variability
aNote. Total percentages may not add up to 100 due
to “other” or “missing” Box Plot

bThree participants are missing; percentages are
A box plot, or a box-and-whiskers plot, is a graph-
rounded to nearest whole number
ical presentation that uses descriptive statistics
From Table 2 in Chen, C.C., Heinemann, A.W.,
Bode, R.K., Granger, C.V. & Mallinson T. (2004).
based on percentiles. It employs the median, the
Impact of pediatric rehabilitation services on interquartile range (IQR), and the minimum and
children’s functional outcomes. American the maximum values in the dataset. The box plot is
Journal of Occupational Therapy, 58, 44–53, illustrated in Figure 15.6. The length of box corre-
with permission. sponds to the IQR (P75– P25); that is, the box
begins with the 25th percentile and ends with the
75th percentile. Then, it determines the extreme
statistics by impairment groups” in means, stan- outlying values by multiplying the IQR (P75– P25)
dard deviations, and ranges (Table 15.6). The last value by 3. Individual scores that are more than
summary is a graph titled, “Mean therapy units three times the IQR from the upper and lower
(15-minute) by impairment” in multiple bars edges of the box are the extreme outlying values
(Figure 15.5). The length of therapy is compared and are denoted by a symbol, E. Minor outlying
among four therapies (occupational therapy, phys- values are determined by multiplying the IQR by
Table 15.6 Descriptive Statistics by Impairment Group

Impairment1
CP NTBI TBI MMT Other Total
(n⫽91) (n⫽114) (n⫽336) (n⫽57) (n⫽216) (N⫽814)
Mean (SD)
Range
Age (in months) 87 (48) 104 (62) 133 (63) 157 (55) 137 (63) 126 (64)
16–222 12–237 12–220 21–231 12–239 12–239
Onset Days2 845 (1554) 96 (415) 35 (171) 29 (60) 199 (768) 178 (723)
0–6782 0–3978 0–2780 1–407 0–5207 0–6782
Length of Stay 27 (16) 30 (22) 33 (26) 40 (31) 28 (21) 31 (24)
5–85 5–123 5–145 7–119 5–120 5–145
Raw FIM Scores3

Mean (SD)
Admission Total FIM 41 (20) 41 (26) 40 (25) 44 (24) 58 (26) 45 (26)

Discharge Total FIM 56 (25) 62 (37) 84 (30) 90 (21) 82 (31) 78 (32)
FIM Gain 15 (14) 22 (23) 44 (26) 45 (22) 24 (17) 32 (25)
FIM Efficiency .60 (.65) 1.08 (1.26) 2.11 (1.72) 1.79 (1.33) 1.24 (1.26) 1.54 (1.52)
Rasch-Transformed Measures
Mean (SD)
Admission Self-Care 18 (19) 21 (22) 22 (22) 24 (20) 34 (20) 25 (22)

Discharge Self-Care 36 (18) 39 (28) 56 (24) 57 (13) 52 (21) 50 (24)
Self-Care Gain 18 (18) 17 (19) 33 (21) 33 (18) 18 (15) 25 (20)
Admission Mobility 14 (11) 21 (18) 23 (16) 19 (13) 26 (17) 22 (16)
Discharge Mobility 24 (14) 39 (27) 55 (22) 52 (19) 45 (20) 47 (23)
Mobility Gain 15 (11) 18 (18) 32 (18) 33 (18) 20 (14) 25 (18)
Admission Cognition 52 (26) 31 (26) 31 (24) 45 (31) 65 (31) 43 (31)
Discharge Cognition 54 (25) 44 (29) 57 (21) 71 (18) 74 (27) 60 (26)
Cognition Gain 2 (16) 13 (18) 26 (20) 26 (20) 9 (14) 17 (19)
1CP ⫽ Cerebral Palsy, NTBI ⫽ Nontraumatic Brain Injuries, TBI ⫽ Traumatic Brain Injuries, MMT ⫽ Major
Multiple Trauma, Other ⫽ Other impairments
2Interquartile range of onset (time from diagnosis of impairment to rehabilitation admission): CP 3–1525 days,
NTBI 1–38 days, TBI 6–19 days, MMT 7–23 days, Other 8–42 days
3FIM ⫽ WeeFIM® (Functional Independence Measure of Children) (UDSMR, 1993)
From Table 3 in Chen, C.C., Heinemann, A.W., Bode, R.K., Granger, C.V. & Mallinson.T. (2004). Impact of
pediatric rehabilitation services on children’s functional outcomes. American Journal of Occupational
Therapy, 58, 44–53, with permission.
1.5. Individual scores between 1.5 times the IRQ 72 ⫺ 21 ⫻ 3). The minor outlying values should
and three times the IQR away from the edges of be between the values of 124.5 (⫽ 93 ⫹ 21 ⫻ 1.5)
the box are minor outlying values and indicated by and 156 or between the values of 40.5 (⫽ 72 ⫺ 21
O. Finally, whiskers of the box indicate the maxi- ⫻ 1.5) and 9. In the example, no values fall into
mum and the minimum values that are not minor the minor or extreme outlying value ranges. The
or extreme outlying values. largest value that is not an outlier is 98 and the
Using as an example the final scores for class A smallest value that is not an outlier is 65.
discussed previously, the 25th percentile is 72 and A box plot is useful for comparisons among
75th percentile is 93; therefore the IQR is 21. The several groups. Gagné and Hoppers (2003), in the
extreme outlying values should be over the value study titled “The effects of collaborative goal-
of 156 (⫽ 93 ⫹ 21 ⫻ 3) or under the value of 9 (⫽ focused occupational therapy on self-care skills: a
160
140
120
Number of (15-minute) Units

100
80
60
Figure 15.5 Mean therapy
units (15 minute) by impair- OT
ment. (From Figure 1 in 40
Chen, C.C., Heinemann, PT
A.W., Bode, R.K., Granger, 20
C.V., & Mallison, T. (2004). SP
Impact of pediatric rehabi-
litation services on children’s 0 PS
functional outcomes. The CP NTBI TBI MMT Other
American Journal of
Occupational Therapy,
58(1), 44–53. Reprinted Mean Therapy Units (15-Minute) by Impairment.
with Permission from The N = 814
American Journal of CP = Cerebral Palsy, NTBI = Nontraumatic Brain Injuries, TBI = Traumatic
Occupational Therapy.) Brain Injuries, MMT = Major Multiple Trauma, Other = Other impairments
o Minor outlier
Largest value that is not an outlier
75th percentile
50th percentile (median)
25th percentile
Smallest value that is not an outlier
E Extreme outlier
Figure 15.6 Box plot.
227
pilot study,” presented 12 box plots, two plots

(experimental and control groups) each for six
Sample Distributions
activities of daily living (ADLs) (eating, groom- A sample distribution not only describes the distri-
ing, bathing, upper-body dressing, lower body bution of data characteristics but also determines
dressing, and toileting) (Gagné & Hoppers, 2003) the type of inferential statistics (parametric versus
(Figure 15.7). For visual comparisons of the nonparametric statistics) one should use. Fre-
change in FIM scores across the ADLs, the unit of quency distributions of samples can be categorized
scale (vertical axis) needs to be considered. In their into normal, positively skewed, and negatively
presentations, for eating, grooming, and upper skewed distributions. Whichever shape it takes, a
body dressing, the unit was .5 points; for bathing distribution is a ranking, from lowest to highest, of
and lower body dressing the unit was 1.0; and for the values of a variable and the resulting pattern of
toileting it was 2.0. In this study, the main purpose measures or scores. Often these are plotted on a
of the presentation was to compare the treatment graph.
and control groups within, not across, the ADL If something in a sample is measured that is
tasks. If the purpose of the comparisons is to iden- representative of the population and has a suffi-
tify which tasks show more changes than other cient size, the results tend to form a distribution
tasks, the unit of the dependent variable (Changes curve that is similar to a normal (or a bell-shaped)
in FIM Scores) for each graph should be the same curve. Most scores will fall around the mean, and
for all graphs. the frequency will be less as distance from the
Eating Grooming Bathing

0.0 0.5 1.0 1.5 2.0 2.5 3.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0
5
]
p = 0.244 p = 0.157
.......
Changes in FIM Scores

p = 0.292
4
] . ...
]. ...
]. ...
2
]... .......
]... ..
]... ..
]...
Experimental Control Experimental Control Experimental Control
Upper Body Dressing Lower Body Dressing Toileting

0.0 0.5 1.0 1.5 2.0 2.5 3.0
6
]
]
4
]
......
...... .......
... .......
p = 0.016
p = 0.241
p = 0.051
3
4
2
] ...
2
1
] ......
]....... .....
]....... ...
]....... .
]....
0
0
-1
Experimental Control Experimental Control Experimental Control
Participants' gains in self-care scores on the FIM between measurements at admission and 2-week follow-up.
Note. The heavy horizontal bar represents the median change. The box represents the middle 50% of the change
data. The dashed lines represent the range of the data within 1.5 times the interquartile range. The stand-alone
bar represents possible outliers.
Figure 15.7 Participants’ gains in self-care scores on the FIM between measurements at admis-
sion and 2-week follow-up. (From Figure 1 in Gagne, D. E., & Hoppes, S. (2003). The effects of
collaborative goal-functional occupational therapy on self-care skills: A pilot study. The American
Journal of Occupational Therapy, 57(2), 215–219. Reprinted with permission from The American
Journal of Occupational Therapy.)
mean increases, with relatively few extreme

The normal distribution was originally studied by
scores. The normal distribution is important for DeMoivre (1667–1754), who was curious about
two reasons: its use in predicting the probabilities in gam-
• Many of inferential statistics called parametric bling. The first person to apply the normal distri-
bution to social data was Adolph Quetelet
statistics assume that the populations are nor-
(1796–1874).He collected data on the chest
mally distributed, and measurements of Scottish soldiers, and the
• The normal curve is a probability distribution heights of French soldiers, and found that they
that helps to determine the likelihood of occur- were normally distributed. His conclusion was
rence of the group difference by chance. that the mean was nature’s ideal, and data on
either side of the mean were a deviation from
Normal Distributions nature’s ideal. Although his conclusion was
absurd, he nonetheless represented normal
Normal distributions are continuous distributions distribution in a real-life setting.
with bell-shaped (normal) frequency curves that
are unimodal and symmetrical. There are many
different normal distributions, one for every possi-
ble combination of mean and standard deviation. A deviation is 34.13% or 0.3413. The area between
normal distribution is sometimes called Gaussian –1 standard deviation and 1 standard deviation (⫾
distribution. 1 sd) is 68.26% or 0.6826. Similarly, (⫾ 2 sd) will
cover 95.44% and (⫾ 3 sd), 99.74% of the area
Normal Standard Distribution under the curve.
Sometimes investigators report scores in terms of
standard deviation units (i.e., standardized scores z-Score
or z-scores). The distribution of these scores is A z-score (a lowercase z is used for this statistic) is
called the normal standard distribution (Z-distribu- the most commonly used standard score. It gives
tion). It is a normal distribution that has a mean the distance in standard deviation units from the
equal to 0 and a standard deviation equal to 1. The mean. It is calculated by dividing the deviation of
horizontal axis of this distribution is expressed an individual score from the mean by the standard
using standard deviation units. The mean, mode, deviation:
and median should equal zero; therefore, 0 has the X⫺X 苶
highest peak and both sizes of 0 are symmetrical. z⫽ ᎏ
Theoretically, there are no boundaries to the curve, s
that is, scores potentially exist with infinite values. A z-score of –2.0 means the particular score is
Therefore, the tails of the curve will never touch two standard deviations below the mean of 0. z-
the baseline. The normal standard distribution is scores are useful for comparing several measures
illustrated in Figure 15.8. that have different means and standard deviations.
Since these properties are constant or standard, The following is an example; it is illustrated in
one can determine the proportion of the area Figure 15.9. If a child’s weight is 29 kg where the
under the curve by the standard deviations in a mean is 25 kg and the standard deviation is 3.8 kg,
normal distribution. If the area under the curve is then the z-score is 29 – 25/3.8 ⫽ 1.05. If one refers
100%, the area between 0 (mean) and 1 standard to Appendix A, Table A [areas under the normal
68.26%
2.14% 34.13% 34.13% 2.14%

13.59% 13.59%
Figure 15.8 Normal standard
distributions. -4 -3 -2 -1 0 1 2 3 4
z = 1.05
85.31%
-4 -3 -2 -1 0 1 2 3 4
25 28.8 29
kg kg kg Figure 15.9 z-score for 29 kg.
curve (z)], the area between 0 and the z ⫽ 1.05 is REFERENCES

0.3531 and the area above z ⫽ 1.05 is 0.1469. Bond, T. G., & Fox, C. M. (2001). Applying the Rasch
From this information one can calculate that model. Mahwah, NJ: Lawrence Erlbaum.
85.31% (0.3531 ⫹ 0.50 ⫽ 0.8531 or 1.00 – 0.1469 Center for Functional Assessment Research/Uniform Data
System for Medical Rehabilitation (1994). Guide for
⫽ 0.8531) of children are lighter than the child’s
the use of the uniform data set for medical rehabilita-
weight of 29 kg and 14.69% of children are heav- tion. Buffalo, NY: Center for Functional Assessment
ier than the child. Moreover, if the sample size is Research.
200, one could further calculate that about 171 Chen, C. C., Heinemann, A. W., Bode, R. K., Granger, C.
children (200 ⫻ 0.8531) were lighter than the V., & Mallinson, T. (2004). Impact of pediatric rehabil-
itation services on children’s functional outcomes. The
child and 29 are heavier (200 ⫻ 0.1469 ⫽ 29 or American Journal of Occupational Therapy, 58 (1),
200 – 171 ⫽ 29). 44–53.
Diehl, M., Marsiske, M., Horgas, A. L., & Saczynski, J.
(1998). Psychometric properties of the revised
Conclusion observed tasks of daily living (OTRL-R). Poster session
presented at the 51st Annual Scientific Meeting of the
Deontological Society of America, Philadelphia.
This chapter discussed how researchers measure Dumont, C., Vincent, C., & Mazer, B. (2002).
people, objects, and events when conducting quan- Development of a standardized instrument to assess
titative research. It also discussed how measure- computer task performance. The American Journal of
Occupational Therapy, 56(1), 60–68.
ment scales and descriptive statistics are the basis Gagné, D. E., & Hoppers, S. (2003). The effects of collab-
of statistics for all quantitative inquiry. In occupa- orative goal-focused occupational therapy on self-care
tional therapy, many quantitative studies aim to skills: A pilot study. The American Journal of
identify relationships among variables or are Occupational Therapy, 57(2), 215–219.
Gilson, B., Gilson, M., Bergner, R., Bobbit, S., Kressel,
experiments designed to identify causal relation- W., Pollard, W., et al. (1975). Sickness Impact Profile:
ships. For those types of studies, the investigator Development of an outcome measure of health care.
uses inferential statistics. Chapter 16 introduces American Journal of Public Health, 65, 1304–1325.
inferential statistics. Glass, G. V., & Hopkins, K. D. (1996). Statistical methods
in education and psychology (3rd ed.). Boston: Allyn
and Bacon.
Guildford, J. (1954). Psychometric methods (2nd ed.). New
Parametric statistics assume that data are nor- York: McGraw-Hill.
mally distributed in the population being studied. Jette, A. (1980). Functional Status Index: Reliability of
chronic disease evaluation instrument. Archives of
Therefore, if the sample distribution is skewed, Physical Medicine and Rehabilitation, 61, 395–401.
even after a random sampling, a transformation Kerlinger, F. N. (1973). Foundations of behavioral
of the data is usually done if one does not wish research. New York: Holt, Rinehart & Winston.
to use nonparametric statistics. The transformed Kielhofner, G., Dobria, L., Forsyth, K., & Basu, S. (2004).
scales are not in the same metric as the original; Construction of keyforms for obtaining instantaneous
therefore, interpretations should be made with measures from the Occupational Performance History
caution. On the other hand, Glass and Hopkins Interview rating scales. Occupation and Participation:
(1996) argue that many parametric methods will An Occupational Therapy Journal of Research, 25,
work even if their assumptions are violated. 23–32.
Linacre, J. M. (1995). KeyFIM—Self-measuring score
form. Rasch Measurement Transactions, 9, 453–454.
Linacre, J. M. (1997, June). Instantaneous measurement Wright, B. D., & Linacre, J. M. (1989). Observations are
and diagnosis. Physical Medicine and Rehabilitation: always ordinal; measurements, however, must be inter-
State of the Art Reviews, 11(2), 315–324. val. Archives of Physical Medicine and Rehabilitation,
Merbitz, C., Morris, J., & Grip, J. C. (1989). Ordinal scales 70, 857–860.
and foundations of misinference. Archives of Physical
Medicine and Rehabilitation, 70, 308–312.
Mitcham, M. D., Lancaster, C. J., & Stone, B. M. (2002). RESOURCES
Evaluating the effectiveness of occupational therapy
faculty development workshops. The American Journal For a Web site on definition of statistics, go to
of Occupational Therapy, 56 (3), 335–339. http://www.wordiq.com/definition/Statistics.
Nunally, J. C. (1978). Psychometric theory (2nd ed.).New For information to draw a pie chart, refer Wallgren, A.,
York: McGraw-Hill. Wallgren B., Persson, R., Jorner, U., & Haaland, J.
Portney, L. G., & Watkins, M. (2000). Foundations of clini- (1996). Graphing statistics & data. Thousand Oaks,
cal research (2nd ed.). Upper Saddle River, NJ: CA: SAGE Publications.
Prentice-Hall. To understand Rasch Analysis, refer Bond, T. G. & Fox,
Statistical Package for the Social Sciences (SPSS) (2003). C.M. (2001). Applying the Rasch model. Mahwah, NJ:
Version 12.0 [software]. Chicago: SPSS, Inc. Lawrence Erlbaum Associates..
Stevens, S. S. (1946). On the theory of scales of measure- For a Web site on normal distribution, go to http://cnx.rice.
ment. Science, 103, 677–680. edu/content/m11164/latest/
Velleman, P. F., & Wilkinson, L. (1993). Nominal, ordinal, For detailed information on data transformation, read:
interval and ratio typologies are misleading. American Tabachnic, B. G., & Fidell, L. S. (1996). Using multivari-
Statistician, 47, 65–72. ate statistics (3rd ed.). New York: Harper Collins
Wallgren, A., Wallgren, B., Persson, R., Jorner, U., & College.
Haaland, J. (1996). Graphing statistics & data. Glass, G. V., & Hopkins, K. D. (1996) Statistical methods
Thousand Oaks, CA: SAGE Publications. in education and psychology. Boston: Allyn and Bacon.
C H A P T E R 1 6
Concepts of Inferential Statistics:

From Sample to Population
Machiko R. Tomita
Inferential Statistics program designed to enhance their motivation and

habits show increased activity levels after the pro-
Investigators conduct quantitative research with gram. While this conclusion predicts that 80% of
the aim of generalizing their results from the sam- people with chronic mental illness who receive
ple studied to the population that sample was such a program will demonstrate increased activity
selected to represent. levels, it does not mean
The population is usu- that every person has an
ally a theoretical con- Probability is the likelihood 80% chance of increased
cept in research. For activity level. Twenty
instance, if an investiga- that a particular event will percent of the population
tor wants to measure the occur in the long run. (for unknown reasons)
effectiveness of a sen- will not increase activity
sory stimulation versus
Therefore, the probability is level following such a
a play-oriented inter- predictive in that it reflects program. Whether a sin-
vention for children in gle individual belongs to
the United States with
what should happen over the 80% or the 20% cate-
developmental disabili- many instances, but not gory is not known.
ties whose ages are Although the proba-
necessarily what will happen bility of interest is always
between 3 and 8 years,
the population of inter- for any given event. population probability, it
est would be all children is unknown. Therefore,
with the aforementioned the sample probabilities
characteristics. However, it is impossible to access are used to estimate population parameters. Values
that entire population and test two types of occu- of statistical probability range from 1.0 (always) to
pational therapies on them. Therefore, the investi- 0 (never). They cannot take on a negative value.
gator must infer the population parameter from the The probability of an event, denoted by P(Event), is
results obtained from the sample. given by:
Inferential statistics are used to estimate popu-
Number of observations for
lation parameters based on data from a sample.
which the event occurs
The accuracy of the estimation depends on the
P(Event)
extent to which the sample is representative of the
Total number of observations
population. In estimating population parameters
from a sample, two statistical concepts are used: Consider a sample of 100 frail elders who are
probability and sampling error. living at home. In this sample of 100 persons, four
are placed in a nursing home and five die over a
Probability 3-month period. Therefore, the sample probability
that frail elders will be placed in a nursing home in
Probability is the likelihood that a particular event 3 months is P(Institutionalization) 4/100 .04 (or 4%),
will occur in the long run. Therefore, the probabil- and the probability that frail elders will die is
ity is predictive in that it reflects what should hap- P(Death) 5/100 .05 (or 5%). The probability
pen over many instances, but not necessarily what that frail elders survive and live at home is P(Survival)
will happen for any given event. 91/100 .91(or 91%). These three events are
For example, an occupational therapist con- mutually exclusive and complementary events
cludes from many studies that 80% of clients with because they cannot occur at the same time and
chronic mental illness who receive a structured because they represent all possible outcomes.
Chapter 16 Concepts of Inferential Statistics: From Sample to Population 233
Therefore, the sum of their probabilities will statistic will occur by sampling error or by chance
always equal 1.00 (or 100%). These probabilities alone. Traditionally, psychosocial and health
are good estimates of population probabilities if researchers often use a cutoff point of p ≤ .05 (5%)
samples were selected at random. and biomedical investigators tend to use a cutoff of
Suppose, for instance, an investigator wanted to p ≤ .01 (1%). Even though investigators do random
test the effectiveness of an occupational therapy sampling from the population, the sample mean
intervention on fatigue among individuals with and sample standard deviation are not exactly the
various levels of chronic fatigue syndrome. If the same as the values of population. The tendency for
investigator chooses only study participants whose sample values to be different from values of popu-
symptoms were mild and lation is called sampling
assigned them to the error. The characteristics
treatment and control Sampling error cannot be of a larger sample are
groups, the results of the avoided but its magnitude likely to be closer to those
study could not be gener- of the population than that
alized to the population can be reduced by increas- of a smaller sample. This
as a whole. While they ing sample size. is because the larger the
could be generalized to sample, the less likely it is
persons with mild symp- that an unusual occur-
toms they may not reflect what the outcome would rence will bias the entire sample. For instance, con-
be with persons with severe symptoms. Therefore, sider a population in which three out of every 1000
the result of such a study would not be a true persons have an unusual characteristic. If by
reflection of the population (which includes per- chance one of the people with that characteristic is
sons with both mild and severe symptoms) selected, that subject will have a much larger bias-
because bias was introduced in the process of ing influence on a sample of 20 than on a sample of
sampling. Consequently, the estimation of popula- 100. Sampling error cannot be avoided but its mag-
tion characteristics from sample data is based on nitude can be reduced by increasing sample size.
the assumption that the samples are randomly
chosen and are valid representatives of the popula- Sampling Distribution
tion. Below is a brief overview of random sam-
pling; a more detailed discussion can be found in When many samples are drawn from a population,
Chapter 31. the means of these samples tend to be normally
Two criteria for random sampling are: distributed. The larger the number of samples, the
more the distribution of their means approximates
• Every person, object, or event must have an equal the normal curve. The distribution of means from
chance to be chosen, and many samples is called a sampling distribution
• Each choice must be independent of other (the distribution of raw data is called a sample dis-
choice. tribution). A sample size of 30 or more usually will
The notion of independence means that the result in a sampling distribution of the mean that
occurrence of one event does not influence the is very close to a normal distribution (Vaughan,
probability of another event. For instance, in a 1998). Sampling error of the mean is the difference
study of children with developmental problems, if between the sample mean and the population
an investigator randomly chooses a child with mean, and this difference happens by chance. The
autism, that choice does not affect the probability Central Limit Theorem explains why sampling
of choosing another child with the same condition. error is smaller with a large sample than with a
small sample and why one can use the normal dis-
Sampling Error tribution to study a wide variety of statistical prob-
lems. The Central Limit Theorem consists of three
In research, investigators use probability not only
statements:
to decide how well the sample data estimates the
population parameter, but also to determine if the 1. The mean of the sampling distribution of means
experiment or intervention differences are likely to is equal to the mean of the population from
be representative of population differences or if which the samples were drawn.
they could have happened by chance. For most sta- 2. The variance of the sampling distribution of
tistics this probability is represented as the p-value means is equal to the variance of the population
(presented as a lowercase p). For example, p from which the samples were drawn divided by
0.04 means that there is a 4% probability that the the size of the samples.
3. If the original population is distributed normally 95%CI 5 (1.96) (1.58)

(i.e., it is bell shaped), the sampling distribution 5 3.10
of means will also be normal. If the original 95%CI 1.90 to 8.10
population is not normally distributed, the sam-
pling distribution of means will increasingly One can be 95% confident that the population
approximate a normal distribution as sample mean falls between 1.90 and 8.10. Therefore, there
size increases (i.e., when increasingly large is a 5% chance that the population mean is not
samples are drawn) (Kallenberg, 1997). included in the interval.
If one wants to be confident 99% of the time,
thus, allowing a 1% risk, one uses z 2.596. For
Sampling Error of the Mean the mean 5 and the standard error of the mean
Drawing on the second statement, the variance of 1.58, the limits of a 99% CI are:
the sampling distribution can be expressed as s2/n 99%CI 5 (2.596) (1.58)
(variance divided by the sample size). More fre- 5 4.10
quently, investigators use the standard deviation of 99%CI 0.90 to 9.10
the sampling distribution which is called the stan-
dard error of the mean. It is calculated as: The confidence interval becomes wider (a dif-
ference of 6.20 to 8.20) with an increased confi-
SX s/
n dence level (95% to 99%). Clinical research tends
to use the confidence interval more often than the
Therefore, the standard error of the mean indi- point estimate.
cates how much, on average, the sample mean dif- When the sample size is smaller than 30, use of
fers from the population mean. If the standard the standard normal curve is not considered an
deviation is 5 and the sample size is 10, then the adequate representation and an alternate sampling
standard error of the mean is 1.58; with the same distribution, called the t-distribution, should be
standard deviation and an increased sample size of used. The t-distribution is unimodal and symmetri-
30, the standard error of the mean is 0.91. As this cal but flatter and wider at the tails than the normal
example demonstrates, the larger the sample size, curve. In order to calculate the CI, instead of the z-
the smaller the difference between the sample score, the t-value is used. To identify the t-value,
mean and the population mean. Both the sample degrees of freedom (df), that is n – 1, is used.
mean and the standard error of the mean can allow (Refer to Appendix A, Table B). Therefore, the for-
us to estimate the population characteristics. mula to calculate confidence intervals for small
sample sizes is:
Confidence Intervals CI X (t) Sx

There are two ways to use the sampling distribu-
tion to estimate population parameters. One is a Use of a 95% confidence interval is illustrated
point estimate using a sample mean (X ) to estimate in a study comparing typically developing children
a population mean (). The other is to use an inter- with children who have Asperger’s syndrome
val estimate, which specifies an interval within reported by Dunn, Smith-Myles, and Orr (2002).
which the population mean (parameter) is In their report, the authors provide descriptive sta-
expected to fall. A confidence interval (CI) is a tistics for sections and factors of the sensory pro-
range of scores with specific boundaries or confi- file for typically developing children and children
dence limits that should contain the population with Asperger’s syndrome. These statistics include
mean. The degree of confidence is expressed as a the mean, the standard deviation, and the 95% CI.
probability percentage, such as a 95% or 99% CI, For 13 of the 14 sections and for all nine factors,
meaning one is 95% or 99% sure that the popula- the higher limit of the 95% CI for the children with
tion mean will fall within the interval. The formula
to obtain the confidence limits is:
Most researchers in health care use 5% for the
CI
X (Z) Sx alpha level. It is mainly due to tradition that
Ronald Fisher, the influential statistician at that
For 95% confidence interval, z 1.96 (from time, started in 1926 and even now it is used by
Appendix A, Table A). many journals as the general standard (Moore,
For the mean 5 and the standard error of the 1991).
mean 1.58, the limits of a 95% CI are:
Asperger syndrome was lower than the lower limit

of the 95% CI for the typically developing chil-
Hypothesis Testing
dren. This finding indicated that the two groups In addition to estimating population parameters,
have separate characteristics when measured by inferential statistics are used to answer questions
the sensory profile. regarding the relationships among variables such
The information provided by the CI is useful as comparisons of group means, proportions, cor-
especially when two mean values are close to each relations, and associations.
other. Using an example from Table 16.1, the mean
of “Items indicating threshold for response” for Null Hypothesis and
typical children was 13.53 (SD 0.32) and that Alternative Hypothesis
for children with Asperger syndrome is 11.08 (SD
0.40). In this study, the investigators wanted to Since sampling error is unavoidable, its effect on a
ask: “Can 11.08 be considered typical?” Since the study must always be taken into consideration. For
95% CI for the typical children was 12.89 and example, if in a study comparing two different
14.16, the score of 11.08 falls outside that CI. occupational therapy treatment approaches there
Thus, the answer is “No.” Since this type of ques- are no true differences in the dependent variable,
tion is frequently asked in clinical research, the one, nevertheless, would expect to find some dif-
confidence interval is widely used. ferences solely due to chance (i.e., sampling error).
Table 16.1 Means, Standard Deviations, and 95% Confidence Intervals for Groups on Each Section
Section Typical M (SD) Asperger M (SD) Typical CI Asperger CI
Auditory processing 34.28 (0.63) 23.73 (0.78) 33.03–35.52 22.18–25.28
Visual processing 38.80 (0.71) 31.50 (0.89) 37.37–40.23 29.73–33.27
Vestibular processing 52.53 (0.73) 44.54 (0.90) 51.08–53.98 42.74–46.34
Touch processing 83.50 (1.22) 61.31 (1.52) 81.06–85.94 58.28–64.33
Multisensory processing 30.88 (0.50) 22.81 (0.62) 29.88–31.87 21.58–24.04
Oral sensory processing 54.85 (1.18) 42.65 (1.46) 52.50–57.21 39.73–45.58
Sensory processing related to 42.28 (0.89) 31.27 (1.11) 40.49–44.06 29.05–33.49
endurance/tone
Modulation related to body 46.48 (0.70) 38.92 (0.87) 45.08–47.87 37.19–40.66
position and movement
Modulation of movement 27.45 (0.56) 20.00 (0.69) 26.34–28.56 18.62–21.38
affecting activity level
Modulation of sensory input 18.43 (0.36) 12.46 (0.45) 17.71–19.14 11.57–13.35
affecting emotional
responses and activity level
Modulation of visual input 17.45 (2.21) 16.65 (2.74) 13.04–21.86 11.19–22.12
affecting emotional
responses and activity level
Emotional/social responses 72.03 (1.23) 49.81 (1.52) 69.57–74.48 46.68–52.85
Behavioral outcomes of 26.13 (0.48) 16.12 (0.59) 25.18–27.07 14.94–17.29
sensory processing
Items indicating thresholds for 13.53 (0.32) 11.08 (0.40) 12.89–14.16 10.29–11.87
response
Note. CI 95% confidence interval. Lower scores indicate poorer performance; that is, the children engage in
the difficult behaviors more often (always 1,never 5). Children without disabilities (typical) have a low
rate of the behaviors on the Sensory Profile; fewer behaviors yield a higher score.
From Table 3 in Dunn, W., Smith Myles, B., & Orr, S. (2002). Sensory processing issues associated with
Asperger Syndrome: A preliminary investigation, American Journal of Occupational Therapy, 56, 97–106,
with permission.
Therefore, one needs to be able to determine when to substantiate the effect. One can never “prove”
differences found in a study reflect the true differ- the existence or nonexistence of true differences
ences in the population under study. The mecha- because one cannot test all members of the
nism for determining this is called a process of population.
hypothesis testing. Since research involves a sample drawn from
Sorting out whether an observed difference in the population to which the investigator wants to
the dependent variable between the two groups has generalize the results, it is only possible to reject or
occurred by chance due to sampling error is the not reject the hypothesis based on a chosen level of
concern of the statistical hypothesis. The statistical confidence (typically a 5% or 1% chance of error).
hypothesis is also called the null hypothesis (noted When reporting the results of a study,
as Ho). The null hypothesis states that the observed researchers usually present the alternative (or
difference between the groups is due to chance research) hypothesis or the research question. For
(i.e., that there is no true difference between the example, King, Thomas, and Rice (2003, p. 517)
groups). In a study com- stated the following
paring different treatment alternative hypothesis:
approaches, the investi- One can never prove the “There will be a differ-
gator is really asking existence or nonexistence of ence in range of shoul-
whether one approach is der motion and quality
more effective than the the true differences because of movement….. when
other (e.g., whether the researchers cannot simply wearing the orthosis
mean of the dependent compared to the free
variable for one group test all possible cases hand condition.” Other
will be different than (members of a population). investigators present a
the mean for the other research question such
group). This research as “Is there a difference
question is represented by the alternative hypothe- between learning the functional living skill of
sis (also called the research hypothesis), and it is cooking for people with serious and persistent
the opposite of the null hypothesis. By either schizophrenia when it is taught in a clinic or in
rejecting or not rejecting the null hypothesis, the their homes?” (Duncombe, 2004, p. 272). Either
investigator, in effect, accepts or rejects the alter- way, the null hypothesis (which is typically not
native (research) hypothesis. stated in a research report) is implied.
The null hypothesis is concerned with the pop-
ulation parameter, so the mean is expressed using Types of Alternative Hypothesis
a Greek letter. The null hypothesis can be stated in
An alternative (research) hypothesis can be stated
one of the following two formats:
as either directional or nondirectional. The two
• The population mean of group A and the popula- examples given in the previous paragraph are both
tion mean of group B are the same, or nondirectional hypotheses because they do not
• The population mean of group A minus the pop- state which group is expected to obtain a higher
ulation mean of group B is equal to zero. dependent variable mean. If, on the other hand, the
These two options are written as follows: alternative hypothesis indicates that one group
• H0: A B or mean will be higher than the other, it is a direc-
• H0: A – B 0 tional hypothesis. An example of a directional
hypothesis is, “…the program would lead to an
Usually, the researcher’s intention is to reject overall reduction in symptom severity and
the null hypothesis. By rejecting the null hypothe- improvements in quality of life over time for indi-
sis, the investigator concludes that it is unlikely viduals in the program as compared to controls…”
that the observed difference occurred by chance; (Taylor, 2004, p. 35). In this case, the difference
therefore the researcher concludes that there is a being compared is a difference in amount of
significant effect. When the null hypothesis is not change as opposed to a sample mean but the prin-
rejected, the investigator concludes that the ciple is the same.
observed difference is probably due to chance and, The notation for a nondirectional hypothesis is:
thus, that the difference is not significant. When a
H1: A ≠ B or H1: A – B ≠ 0 for a,
difference is not significant, it does not necessarily
mean that there is no true effect, or no difference. whereas the notation for directional hypotheses
Rather, it means that the evidence is too weak are either:
H1: A B or H1: A – B 0 or If one accepts the null hypothesis when it is

true, concluding there is no difference when in
H1: A B or H1: A – B 0 . actuality there is no difference, one made a right
depending on the direction the mean is decision. If one accepts a null hypothesis that is
expected to move toward. false (i.e., conclude there is no difference when
Although most research reports will state that there is an actual difference), a type II error is
the (research) hypothesis was accepted, statisti- made. Figure 16.1 describes the relationship
cally the investigators are dealing only with the between the two types of null hypothesis and two
null hypotheses, not alternative hypotheses. types of error.
(That’s why a null hypothesis is called a statistical
hypothesis.) That is, they have either rejected or Type I Error
not rejected the null hypothesis. The probability of committing a type I error is
called alpha () level. The level of .05 or .01 is
Types of Errors in Testing Hypothesis usually used and it indicates the maximum amount
of type I error that can be committed if one rejects
When one draws a sample from which to infer the a true null hypothesis, 5% or 1%, respectively.
population parameter, there is always sampling If is .05, it means that one will accept up to a 5%
error. Therefore, there is chance of falsely reject-
always a possibility of ing the null hypothesis.
making a wrong inference. Although most research Alpha is the level
Since one either rejects or of significance that de-
does not reject the null
reports will state that the
termines whether the
hypothesis when drawing (research) hypothesis was observed difference is
a conclusion, each conclu- accepted, statistically the in- due to sampling error or
sion can be either right or real and is denoted as p
wrong, allowing for four vestigators are dealing only (lowercase). If p .24,
possible outcomes as with the null hypotheses, not it means that there is a
shown in Table 16.2. 24% probability that the
Drawing the wrong alternative hypotheses. That difference occurred by
conclusion is called an is, they have either rejected chance, or there is a
error of inference. There 24% chance of commit-
are two possible types of or not rejected the null ting a type I error if the
error: type I error and type hypothesis. researcher decides to
II error. If one rejects the reject the null hypothe-
null hypothesis and states sis. However, when one
there is a significant difference when in actuality has set .05 (5%) a priori, 24% is beyond the
the difference does not exist and the null hypothe- set 5%; in this instance one cannot reject the null
sis is true, concluding there is a difference, one has hypothesis and must accept it instead, concluding
made a type I error. In this situation, the difference that there is no difference. On the other hand, if
was due to chance. This type of error should be p .03, one would reject the null hypothesis, con-
avoided because often it is a serious mistake. cluding there is a significance difference.
Table 16.2 Four Possible Decision Outcomes in Null Hypothesis Testing

Results of Null Hypothesis (H0)
Testing H0 (There is no difference between groups)
True False
Reject Ho Type I error () Correct decision

Statistical power (1-)
(There is difference between groups.)
Accept Ho Correct decision Type II error ()
(There is no difference between groups)
Distribution: The null hypothesis

is true.
Type I Error

Distribution: The null

hypothesis is false.
Type II Error
Statistical

Power
Figure 16.1 Two types of null

Acceptance Region Rejection Region
hypotheses and type I and
type II error.
Interpretation of a p-value requires caution. It is The solution to decreasing the chances of type
a common mistake, when .05, to say a smaller II error is to increase statistical power (i.e., the
p-value (e.g., p . 001) probability that a test
is “more significant” will lead to rejection of
than a larger p-value The magnitude of p is not an the null hypothesis).
(e.g., p .01). The mag- Statistical power is rep-
nitude of p is not an indi-
indicator of the degree of resented as 1 – . The
cator of the degree of validity of the alternative power of .80 is consid-
validity of the alternative hypothesis. Once the p-value ered reasonable because
hypothesis. Once the p- 80% of the time an inves-
value is judged against is judged against the set tigator would correctly
the set level, the deci- level, the decision is dichoto- identify a statistical dif-
sion is dichotomous: ference and reject the
either yes, significant or mous: either yes, significant null hypothesis (Cohen,
no, not significant. or no, not significant. 1988). Conventionally,
then, is set at .20 or
Type II Error 20%.
If an investigator does not reject the null hypothe- Statistical Power and Effect Size
sis when it is false, a type II error is committed.
This type of error is of great concern in clinical There are four factors that influence the statistical
research. For example, in a study whose aim was power:
to determine whether an occupational therapy
• Significance level (),
intervention was better than a control condition, a
• Variance of data (s2),
type II error would mean concluding that the inter-
• Sample size, and
vention does not produce better outcomes when, in
• Effect size.
fact, it does.
A type II error is denoted by beta (). Thus the The effect size is an indicator of magnitude of
value of represents the likelihood that one will the observed difference in the dependent variable.
be unable to identify real differences (however, Said another way, it is the impact made by the
there is no direct mathematical relationship independent variable and if the impact is strong,
between type I and type II error). Decreasing the the effect size will be large. So, for example, con-
likelihood of type II error by increasing would sider a study that was designed to determine the
only increase the chance of type I error. extent to which a new occupational therapy inter-
vention was more effective than a traditional inter- doing this is comparing independent variables that
vention for increasing functional capacity. In this are more clearly differentiated from each other so
instance, the effect size would refer to the amount that a larger difference of the dependent variable is
of difference observed between the two groups in more likely to be observed.
the functional outcomes.
The first method to increase the power is to
Power Analysis
increase the significance level () for example,
from .05 to .10. Then will be decreased and that One of the largest concerns when planning any
will increase the power. However, as noted earlier, study is determining the necessary sample size.
doing so increases the chance of type I error (from Investigators use power analysis to estimate the
5% to 10%). This is always undesirable since type sample size needed to obtain a desired level of
I error is often a serious mistake to commit. power. This should be done before one finalizes
The second method to increase power is to the study plan.
decrease the dependent variable’s variance within To determine the sample size, one first decides
groups. Achieving a reduction in within group a significance level (usually either .01 or .05) and
variance can be accomplished by selecting a more desired power (usually .80). Effect size should be
homogeneous sample that will show less variabil- calculated on the dependent variable that is the
ity in the dependent variable. It can also be focus of the primary study question or hypothesis.
achieved by selecting a more accurate measure of Effect size should be calculated based on the
the dependent variable. Since observed variance is past studies or preliminary data, using the appro-
a function of true variance plus error variance, priate statistical analysis for the study. (For calcu-
increasing within-group homogeneity and meas- lation of effect size for various statistics, see
urement accuracy are both ways of reducing Chapter 17.)
observed variance. Power analyses can also be used when a study’s
The third method to increase the power is to results are not statistically significant. In this
increase a sample size. When small samples are case, it may be that a type II error was committed.
used, power is reduced. Increasing sample size is This can be determined based on the significance
the only way to increase power once a study level, observed effect size, and sample size.
design has been finalized (i.e., when the independ- Nonsignificant findings may have occurred because
ent and dependent variable and the measure of of a small sample size and not because of the
the dependent variable have already been deter- true effectiveness of the treatment (effect size).
mined). Suppose observed effect size was d .40 at
The last method used to increase power is to .05, and the sample size in each group was 20.
increase effect size (often called clinical signifi- Then the power is only .23. With the same effect
cance). There are different ways to calculate effect size, if the sample size in each group was 100, the
size. One common way to calculate the effect size power becomes .80. In this instance, even though
between two groups is to use a d-index. d is calcu- results are not statistically significant, it is worth
lated as the difference between the two means (or reporting the medium effect size with the hope that
the mean difference) divided by the common stan- a similar study will be conduced in the future with
dard deviation (i.e., the square root of the mean of a larger sample size. Therefore, whenever results
the two groups’ variance when sample size is the are not significant, the effect size should be
same for the two groups). If the mean difference is reported.
a half size of the common standard deviation, then
d .50 and it is interpreted as medium effect size.
Critical Value
The effect size of d .30 is considered small and
d .80 is large. Since effect size is the degree to To find the difference between the means of two
which the null hypothesis is false, the large effect groups, one uses a test statistic called a t-test to
size means more power. calculate the t-value. Then, one identifies whether
In a study comparing two different treatment or not the null hypothesis can be rejected at a given
approaches, increasing effect size means widening level (e.g., ≤ .05) using a t-distribution. In
the observed difference between the two groups’ doing this, one identifies the critical value (i.e., the
dependent variable means. One of the most com- value determining the critical region [5%] in a
mon ways of doing this is increasing the length or sampling distribution) by looking it up on a t-dis-
frequency of the interventions when the treatment tribution table (Appendix A, Table B). The critical
that is expected to produce better outcomes region is the area wherein one can reject the null
improves with greater intensity. Another means of hypothesis.
To find the critical value for t-tests, one must

Degrees of freedom (df) is the number of values
first identify the degrees of freedom (df). The df that are free to vary when the sum is predeter-
indicates how much data are used to calculate a mined. For example, when there are five numbers
particular statistic. For t-tests, df (n1 – 1) (n2 in a group that sums up to 15, the first four num-
– 1), where n1 and n2 are sample sizes for each bers can vary but once the first four numbers are
group. For instance, if one has 15 subjects in each determined, the last number does not have free-
group, the df is 28. When 1 ≤ .05 (the subscript, dom to vary. Therefore, df 4.
1, indicates the hypothesis is directional and one-
tailed test is used), the critical value is 1.701 from
the t-distribution table. A t-test can be one-tailed One-Tailed and Two-Tailed Tests
(for when there is a directional hypothesis) or two-
tailed (for when there is not a directional hypothe- The word “tail” refers to the end of the probability
sis), as will be discussed in more detail later. If the curve. Many researchers use a one-tailed test of
calculated t-value is 1.90, it is larger than the crit- significance when a research hypothesis is direc-
ical value 1.701; therefore, it is within the rejection tional and a two-tailed test for a nondirectional
region (of the null hypothesis). Then one can say research hypothesis. When researchers use a one-
the difference between the two groups is signifi- tailed test there is more power than two-tailed test
cant. If the t-value is smaller than the critical value, as illustrated in Figure 16.2. To return to the previ-
one cannot reject the null hypothesis. Therefore, ous example, a directional hypothesis indicated the
the difference between two groups is not consid- use of a one-tailed test ( ≤ .05), with df 28.
ered significant. This explanation is presented in According to Appendix A, Table B, the critical
Figure 16.2a and b. The step-by-step process of value for a two-tailed test is 2.048. Since the cal-
hypothesis testing is described later in this chapter. culated t-value, 1.90, is smaller than the critical
A)
5%
1.701 1.90
Critical Value Calculated Value
Acceptance Region Rejection Region
B)
2.5% 2.5%
1.90 2.048
Calculated Value Critical Value
Figure 16.2 (A) Critical
value and critical region:
One-tailed test. (B) Critical
Rejection Acceptance Region Rejection
Region Region
value and critical region:
Two-tailed test.
value for a two-tailed t-test, the researcher cannot the null hypothesis, saying the mean differ-
reject the null hypothesis. With a two-tailed test, ence was significant. (When using statistical
the 5% of rejection region is divided into two software, if the p-value of the calculated value
regions of 2.5% each. In order to reject the null is equal to or less than .05 ( ≤ .05), then one
hypothesis when the alternative hypothesis is can reject the null hypothesis.)
nondirectional (two-tailed), the calculated value Step 5. When the results do not allow one to
needs to be much larger than the calculated value reject the null hypothesis (i.e., the critical
for the directional (one-tailed) hypothesis. Thus the value is greater than the calculated t-value), it
one-tailed test has more power than two-tailed test. is a good idea to calculate effect size to deter-
mine the strength of the effectiveness of inter-
vention and the level of type II error ().
Steps of Hypothesis Testing When the sample size is very large (for exam-
Given the previous discussions, the following are ple, 2500 subjects), the result may be statistically
recommended steps for hypothesis testing after significant (p ≤ .05) even if the effect size is small
stating the alternative (research) hypothesis, and (e.g., d 10). Consequently, one must be aware
collecting data. of what statistical significance means. It simply
Step 1. Decide the alpha level such as ≤ .05. means that the observed difference is rare (i.e.,
Step 2. Use an appropriate test statistic such as a occurring 5 out of 100 times) when the null hypoth-
t-test and identify a calculated value, a t-value. esis is true. In occupational therapy research, the
Step 3. Identify the critical value on a t- more important significance is clinical significance
distribution table and compare it with the and that is identified by the effect size. For exam-
calculated t-value. (If one is using statistical ple, when reporting their findings from a study
software, it will calculate the p-value for of the effects of an energy conservation course
the calculated t-value, such as p . 234 or on fatigue impact for persons with progressive
p .000.) Multiple Sclerosis, Vanage, Gilbertson, and
Step 4. Decide whether to reject the null Mathiowetz, (2003) reported effect size regardless
hypothesis (H0) or not. If the calculated value the significance of t-tests for subcategories and
is larger than the critical value, one can reject total score of the Fatigue Impact Scale (Table 16.3).
Table 16.3 Means and Standard Deviations of Difference Scores Between Various Assessment
Timesa, One-Sample t Tests for All Subjects (N 37b) and Cohen’s d Effect Sizes.
FIS M (SD) df t d
Pre-EC Course vs. Post-EC Course #1
Cognitive 4.0 (6.8) 36 3.6* .82
Physical 4.2 (7.9) 36 3.2* .75
Psychosocial 7.5 (12.7) 36 3.6* .83
Total 15.7 (25.0) 36 3.8* .89
Post-EC Course #1 vs. Post-EC Course #2
Cognitive
.4 (7.2) 27
.3
.08
Physical 1.0 (8.1) 27 .6 .17
Psychosocial 1.0 (13.3) 27 .4 .11
Total 2.1 (23.7) 27 .5 .13
aPre-EC Course scores were those recorded prior to the course. Post-EC Course #1 scores were recorded
immediately following the completion of the course. Post-EC Course #2 scores were recorded 8 weeks after
the completion of the course.
bn 28 Available for Post-EC Course #2.
*P .01.
FIS Fatigue Impact Scale

EC Energy conservation
From Table 5 in Vanage, S.M., Gilbertson, K.K., & Mathiowetz, V. (2002). Effects of an energy conservation
course on fatigue impact for persons with progressive multiple sclerosis. American Journal of Occupational
Therapy, 56, 315-323, with permission.
The differences between the pretests and the Kallenberg, O. (1997). Foundations of modern probability.
posttests resulted in large effect sizes (d is close to New York: Springer-Verlag.
King, S., Thomas, J. J., & Rice, M. S. (2003). The immedi-
or larger than .80) across categories, indicating the ate and short-term effects of a wrist extension orthosis
occupational therapy intervention had a definite on upper-extremity kinematics and range of shoulder
and effective impact. The report of the effect sizes motion. The American Journal of Occupational
made the study more meaningful than only stating Therapy, 57(5), 517–524.
Moore, D. S. (1991). Statistics: Concepts and controver-
the statistical significance. Reporting the effect size
sies (3rd ed.). New York: W. H. Freeman.
is the only way to indicate the clinical significance Taylor, R. R. (2004). Quality of life and symptom severity
for occupational therapy intervention studies. for individuals with chronic fatigue syndrome: Findings
from a randomized clinical trail. The American Journal
of Occupational Therapy, 58(1), 35–43.
Conclusion Vange, S. M, Gilbertson, K. K., & Mathiowetz, V. (2003).
Effects of an energy conservation course on fatigue
impact for persons with progressive multiple sclerosis.
This chapter discussed the estimation of popula- The American Journal of Occupational Therapy, 57(3),
tion characteristics based on the sample statistics. 315–323.
It concluded with discussions of statistical power Vaughan, E. D. (1998). Statistics: Tools for understanding
and effect size, two important statistics for occu- data in the behavioral sciences. Upper Saddle River,
NJ: Prentice-Hall.
pational therapy research and practice. The next
chapter discusses inferential statistics in two parts: RESOURCES
univariate analysis and multivariate analysis.
For relationship between confidence interval and p-value,
read:
REFERENCES Munro, B. H. (2001). Statistical methods for health care
Cohen, J. (1988). Statistical power analysis for the behav- research (4th ed.) Philadelphia: Lippincott.
ioral science (2nd ed.). Hillsdale, NJ: Lawrence For sampling distribution, refer to:
Erlbaum. Kallenberg, O. (1997). Foundations of modern probabil-
Duncombe, L. W. (2004). Comparing leaning of cooking in ity. New York: Springer-Verlag.
home and clinic for people with schizophrenia. The For power analysis, refer to:
American Journal of Occupational Therapy, 58 (3), Cohen, J. (1988). Statistical power analysis for the
272–278. behavioral science (2nd ed.). Hillsdale, NJ: Lawrence
Dunn, W..Smith Myles, B., & Orr, S. (2002). Sensory pro- Erlbaum Associates.
cessing issues associated with Asperger syndrome: A For Central Limit Theorem, visit:
preliminary investigation. The American Journal of http://mathworld.wolfram.com/CentralLimitTheorem.
Occupational Therapy, 56(1), 97–106. html.
C H A P T E R 1 7
Methods of Analysis: From Univariate

to Multivariate Statistics
Machiko R. Tomita
This chapter discusses inferential statistics, which If the sample size is large (N 30), those
are statistical techniques used to draw conclusions assumptions are usually met. Ideally, samples are
about a population on the basis of data describing randomly selected or assigned to groups. When the
a sample. These statistical procedures can be cate- data do not present a normal distribution, the vari-
gorized into: ables are measured using a nominal or ordinal
• Parametric statistics, and scale, and/or they are not homogeneous across the
• Nonparametric statistics. groups, in general, nonparametric statistics will be
used. When the sample size is small, data charac-
The statistics discussed in this chapter can also teristics are such that one may have to use non-
be categorized into the following approaches: parametric statistics.
• Univariate, Nonparametric tests, in usual instances, have
• Bivariate, less power for finding statistical significance than
• Multivariable, and parametric tests. However, with very small sample
• Multivariate approaches. sizes (six or less), the power of nonparametric tests
becomes equal to that of parametric tests.
Finally, these statistics can be differentiated by
the use of either independent samples or correlated
samples. This chapter considers all these distinc- Independent and
tions and explains the statistics characterized by
these categories. Throughout the chapter, the for- Correlated Samples
mulas for these statistics are given. In most
instances, these statistics are calculated in statisti- Both nonparametric and parametric statistics can
cal packages. Nonetheless, the formulas are given be further categorized to statistics for independent
here, and examples of how they are calculated are samples and those for dependent or correlated
provided, as they help develop a conceptual under- samples. This is important since statistical proce-
standing of the statistics. dures are different for independent samples and
correlated or dependent samples.
Samples are independent when they are not
Parametric and related to one another in terms of the dependent
Nonparametric Statistics variable. For example, consider an investigator
studying the effects of an occupational therapy
program for people with chronic mental illness.
Parametric statistics are for estimating population
This investigator could compare the outcomes of a
parameters (characteristics) and for testing
treatment group that receives occupational therapy
hypotheses based on population parameters.
to those of a control group that does not receive
Parametric statistics can be used if the following
therapy. In this case, the samples are independent
statistical assumptions are met:
because the people in the two groups are mutually
• The sample is normally distributed (normal exclusive and, thus, the outcome measures taken
distribution), on the two groups will not be related to each
• The dependent variables are measured using an other. On the other hand, the investigator might
interval or ratio scale, and investigate the effects of the occupational therapy
• The variance of the dependent variable is the program by measuring the dependent variable
same across all levels of the independent vari- before and after the therapy on the same group.
able. That is, the groups in the sample are homo- Since the investigator is comparing data collected
geneous (referred to as homogeneity of variance from the same sample, it is called dependent or
assumption). correlated.
This notion of independent and correlated sam- • Multivariate statistics, strictly speaking, deal
ples is sometimes not so clear. For example, con- with more than one dependent variable simulta-
sider a study in which an investigator is testing neously.
strength in different shoulders (right extension or
Table 17.1 summarizes the statistical methods
left extension). In this case, the shoulders can be
presented in this chapter according to the classifi-
considered independent groups, even though they
cations just discussed.
belong to the same person, since the strength in
one shoulder is not affected by the other. On the
other hand, consider a study that aims to determine
parents’ attitudes about an affordable price range Nonparametric Statistics:
for a child’s powered wheelchair. In this case, the Chi-Square
opinion of the mother and father are based on the
same financial resources. Therefore, they can be There are many nonparametric statistics but in this
considered a correlated sample. section only Chi-square (read as kai) will be intro-
It is inappropriate to mix a dependent (corre- duced. Other types of nonparametric statistics are
lated) sample and an independent sample in a sta- explained in the following sections together with
tistical analysis. For example, take a study the parametric statistics they parallel.
designed to identify whether parents of nondis- Chi-square can be used when:
abled children differ in their opinions about appro-
priate toys from parents of children with physical • There is only one categorical variable with more
disabilities. It would be incorrect to include two than one level (group), or
mothers of two children and both parents of a third • There are two categorical variables.
child in the same sample. This is because the toy For the former, goodness of fit tests are used,
preference of the mother and father of the third and, for the latter, tests of independence are used.
child are likely correlated, while those of the moth- Either way, Chi-square is the difference
ers of the different children are not. To include between the expected and observed frequencies
both parents of the third child in this sample means and is denoted as 2. The formula to calculate Chi-
that they are counted as two different opinions, just square values is:
as the two opinions of the mothers of the first and
second child. Although it is possible to analyze (O E)2
both a correlated sample and an independent sam- 2 Σ
E
ple in one study using a mixed design, it is still
required that independent samples are analyzed by where O represents the observed frequency and
statistics for independent samples and the corre- E represents expected frequency.
lated samples analyzed by statistics for correlated The Chi-square value is the sum of squared dif-
samples (see repeated measures analysis of vari- ference between the observed and expected values
ance [ANOVA] with a between-factor discussed divided by the expected values. To use Chi-square,
later in this chapter). categories should be mutually exclusive and
exhaustive.
Univariate, Bivariate, Goodness of Fit Test

(a Nonparametric Test)
Multivariable, and The goodness of fit test compares an observed
Multivariate Analyses frequency with a uniform, known, or normal
distribution.
Another way of classifying statistics is based on
the number of variables in the study. This classifi- Uniform Distribution
cation is as follows:
A uniform distribution expects the same frequency
• Univariate statistics deal with one dependent counts across all categories. For example, suppose
variable and one independent variable, an investigator wishes to identify the number of
• Bivariate statistics handle two variables regard- fall occurrences across three time periods (morn-
less of direction of influence in the analysis, ing, afternoon, and evening) among older psychi-
• Multivariable statistics manage one dependent atric inpatients. The null hypothesis is that an
and several independent variables together, and equal number of falls will occur across the three
Chapter 17 Methods of Analysis: From Univariate to Multivariate Statistics 245
Table 17.1 Types of Inferential Statistics (Introduced in this chapter)

Types of Statistics Independent Sample Correlated Sample
Frequency analysis
Nonparametric (only) Chi-square McNemar test
Comparison of two group means
Parametric Independent-test Paried t-test
Nonparametric Mann-Whitney U test Wilcoxon Signed rank test
The sign test
Comparision of more than two group means
One dependent variable and one independent variable
Parametric One-way ANOVA Repeated measures ANOVA
Nonparametric Kruskal-Wallis one-way ANVOA Friedman test
One dependent variable and more than one independent variables
Parametric (only) Two-way ANOVA Repeated measures ANOVA
Multi-way ANOVA (more than 2)
Mixed design Same as independent sample
ANCOVA Same as independent sample
More than one dependent variables and one or more independent variables
Parametric (only) MANOVA Same as independent sample
MANCOVA Same as independent sample
Associaion between variables
Two variables
Parametric Pearson correlation Same as independent sample
Nonparametric Spearman rank order correlation Same as independent sample
More than two variables
Parametric (only) Multiple correlation Same as independent sample
Predication
One criterion and one predictor variable
Parametric (only) Simple linear regression Same as independent sample
One criterion and more than one predictor variable
Parametric (only) Multiple regression Same as independent sample
Logistic regression Same as independent sample
Classification
Parametric (only) Exploratory factor analysis Same as independent sample
Causal relationship establishment
Parametric (only) Path analysis Same as independent sample
Confirmedly factor analysis Same as independent sample
time periods. For example, if a total of 90 falls (see Appendix A, Table C). To use this table, one
were observed, one would assume that 30 falls looks up the degrees of freedom and selects the set
occurred in each period; thus, that is the expected (or significance) level to see the critical value for
value. If the observed falls occurred 0, 60, and 30 2. In this example, for there to be a significant dif-
times in each period, then the Chi-square value ference between categories, the 2 value must be
would be: greater than the critical value of 5.99. The calcu-
lated value of 60 is larger than 5.99; therefore, one
␹2 (0 30)2/30 (60 30)2/30 finds that the observed value is significantly differ-
(30 30)2/30 60 ent from the expected value and concludes that the
number of falls varied across the periods. The next
In a uniform distribution, degrees of freedom step is to identify which category is contributing to
(df) are the number of categories (k) minus 1. the significant difference. For that, standardized
Therefore, in this example, df 3 1 2. If is residuals (SR) should be calculated. SR is deter-
set at .05, then the critical value is (.05)2(2) 5.99 mined by the absolute value of the difference
between the observed and expected values divided lated and the sum of calculated values for the eight
by the square root of the expected value: categories will be the 2 value. In this case, the
degrees of freedom are (k – 3) or 8 – 3; therefore,
⏐O E⏐
Standardized Residual df 5. Three degrees of freedom are subtracted
because in addition to the usual 1 degree of free-
E dom, two additional degrees of freedom are lost
Residuals close to or greater than 2.00 or because of the known mean score and the known
smaller than –2.00 are considered important standard distribution in the normal distribution
(Haberman, 1984). Returning to the example, the (Snedecor & Cochran, 1991).
standardized residuals for the first and second
zones are 5.477 for both. Therefore, zones 1 and
2 are both contributing to the significant value
Tests of Independence
of 2. (Nonparametric Tests)
The more common use of Chi-square in occupa-
Known Distribution tional therapy research is for tests of independ-
When the distribution within a population or ence. These tests are used to examine the
underlying population is known, then the observed association of two categorical variables. When
frequency counts can be compared against this data are arranged in a two-way matrix, it is called
known distribution. For example, if the number of a contingency table or a cross-tabulation. For this
people with disabilities in several age categories in test, the null hypothesis is: the two variables are
the United States is known, and one wants to com- independent (i.e., not associated). Therefore, if the
pare that with those in a particular state, the former resulting Chi-square value is statistically signifi-
frequencies will serve as expected values and the cant, one concludes that the two variables are asso-
latter, observed values. The calculation method is ciated. Usually the rows (R) are for the dependent
exactly the same as for the uniform distribution, variable and the columns (C) are for the independ-
and the degrees of freedom is k – 1. If the different variable.
ence is significant, then the sample distribution is Consider an investigation of whether being a
different from the known distribution. veteran with combat experience is associated with
a diagnosis of depression. In the survey of 80
Normal Distribution adults with 40 combat-exposed veterans, 16 veter-
ans are diagnosed with depression. Among the 40
The goodness of fit test can be used to determine if non-veterans, 8 are diagnosed as depressed. The
the sample distribution is significantly different null hypothesis states that veteran status is
from a normal distribution by dividing the normal independent of the presence of a depressive diag-
distribution into eight sections. From the Z- nosis. As shown in Table 17.2, observed frequency
distribution (see Appendix A, Table A), one knows of Cell A (Veterans who are depressed) is 16;
the area under the curve is 1.00 or 100%. The area Cell B (Non-Veterans who are depressed), 8; Cell
between Z 0 and Z 1 SD is .3413; 1 SD and 2 C (Veterans who are not depressed), 24; and
SD, .1359; 2 SD and 3 SD, .0215; and more than 3 Cell D (Non-Veterans who are not depressed),
SD, .0013. Since the shape is symmetrical, the 32. Expected values are calculated using the
negative side of the mean has the same area total frequency counts for each row and column.
respectively. The expected frequency can be deter- The formula for the expected frequency (E) is:
mined by multiplying the total sample size by the
area. To make the calculation simple, if the sample fR fC
size is 100, the expected value between 0 and 1 E N
would be 34.13, between 1 and 2 it would be
13.59, and so on. Therefore, the observed frequen- where fR indicates the total frequency of
cies should be classified into the eight sections and the row, and fC indicates the total frequency of the
the Chi-square value can be determined using the column.
formula. For example, if 50 cases were observed in In the example above, the expected value of
the area of Z 0 and Z 1, the difference veterans who are depressed (Cell A) is E 40
between the observed and the expected value 24/ 80 12. Once one finds an expected value of
would be 50 – 34.13 15.87. Then it will be a cell in a 2 2 table, other expected values can
squared (15.87 15. 87 251.86), and divided be easily identified by subtracting the expected
by 34.13 (251.86/34.13 7.38). In the same man- value from the total in the row or the column.
ner, the remaining of seven sections can be calcu- Thus, expected values of veterans who are not
Table 17.2 Chi-Square for Veteran’s Status and Depression (N 80)

Veterans Non-veterans Total
Depressed (A) (B) 24
O 16 O8
E (40 24)/80 12 E 24 – 12 12
SR ⏐16 –12⏐/ 12
1.15 SR ⏐8 – 12⏐/ 12
1.15
Not depressed (C) (D)
O 24 O 32 56
E 40 – 12 28 56 – 28 28
SR ⏐24 – 28⏐/ 28
0.76 SR ⏐32 – 28⏐/ 28
0.76
Total 40 40 80
SR Standardized residual.
depressed are found by subtracting 12 from the cell categories if possible. If it is not possible,
total of the row, 40, E 40 12 28. The Yates’ correlation for continuity can be used, but it
expected frequency of non-veterans who are has less power. If the table is 2 2 (two by two),
depressed is identified by E 24 12 12. Non- the Fisher Exact test should be used. Both are dis-
veterans who are not depressed are found by sub- cussed later in this chapter.
tracting 8 from the total of the row, 40; therefore,
32. Below those values are plugged in the Chi- McNemar Test for Correlated
square formula: Samples (a Nonparametric Test)
(16 12)2 (24 28)2 When the sample is correlated and the dependent
␹ 2 variable has only two levels, such as Yes and No,
12 28
then the table is 2 2 and the McNemar test can
(8 12)2 (32 28)2 be used. This is often used for pre- and posttests on
3.82 the same sample to identify if the change is signif-
12 28 icant or not.
This yields a Chi-square value of 3.82. Degrees The calculation formula is (A and D are loca-
of freedom are obtained by (R 1) x (C 1). tions of cells. See Table 17.2 to see how cells are
Therefore, (2 1) (2 1) 1. From Appendix labeled):
A, Table C, the critical value of (0.5)2(1) 3.84. (A D)2
Since 2 is always positive (defined by the squared 2 with df 1
(A D)
term divided by a positive value), the Chi-square
distribution is always a one-tailed test and the sig-
nificance value should be estimated based on a
Power Analysis
one-tailed test. Chapter 16 discussed the importance of examining
In this instance, the calculated value is smaller the effect size. For a contingency table, the effect
than the critical value at p = 05.1 Therefore, the size (ω, omega) for 2 2 and larger numbers of
investigator cannot reject the null hypothesis and cell categories are obtained respectively by:
must conclude that the two variables are not asso-

ciated. 2 2

, or
( q 1)
When there is a small sample size (an expected N N (q 1)
frequency is less than 1 in each cell, and less than
20% of the cells have observed frequencies of less where q the number of rows or columns,
than 5), one should collapse the table to have fewer whichever is smaller.
The interpretation of the effect size is:
1The α level is the probability or p-value one is willing to
accept as the amount of error, therefore, α = .05 is the same
• ω .10 is small,
as p = .05. If α is set at 5%, in order to reject the null hypoth- • ω .30 is medium, and
esis, the p-value has to be equal to or smaller than p = .05. • ω .50 is large.
Using the example of veterans and depression, test (both of these tests use an F-statistic). When
the F-value is significant (p .05), it means that

3.82
2 3.82,
, and the effect size is .22, two group variances are considered dissimilar, and
80 thus the homogeneity assumption has not been
which is small to medium. By referring to met. In this case, the formula for the t-test must be
Appendix A, Table G, one can see that the power adjusted (many statistical software packages will
is approximately .50. To increase the power to con- perform this calculation automatically.) If the F-
ventional .80, with the effect size, one needs a value is not significant, it means that the two group
sample size of about 200. variances are considered similar or homogeneous.
The formula for the t-test will not have to be
Comparisons of Two
adjusted.
The formula for the t-test below is used with
Group Means: t-Tests groups that have similar or homogenous variance:
1
X X2
This section deals with statistics to compare two t
group means. When comparing of two group Sx x
1 2
means, the null hypothesis is: with df N – 2, where N is the total sample
H0: 1 2 size or (n1 – 1) (n2 – 1), and where n1 and n2 are
the sample sizes for two groups.
Complementing the null hypothesis is the alter- In this equation, the numerator is the difference
native (research) hypothesis: between the two group means. The denominator is
called the standard error of the difference between
H1: 1, 2 (for nondirectional hypothesis), or the means. This indicates how much the difference
H1: 12 or 1 2 (for directional between the mean of the sample is different than
hypothesis) that of the population. It is calculated as follows:

These hypotheses are tested, for indepen- S2P S2P
S x x
n1
n2 where
dent samples, by using one of the following two 1 2
statistics:
• The parametric independent t-test, or S21 (n1 1) S22 (n2 1)
• The nonparametric Mann-Whitney U-test. S2P n1 n2 2
For dependent or correlated samples, investiga- The t-test for groups with unequal variances is:
tors use:
1
X X2
• The parametric paired t-test, t

• The nonparametric Wilcoxon Signed Rank Test,
s21 s22
or
• The nonparametric Sign test. n1 n2
Each is discussed below. In both forms of the t-test, the calculated value
is compared to the critical values in Appendix A,
Table B.
Independent t-Test (a Parametric Test)
When t-tests are performed in a statistical pack-
Independent t-tests are used to compare two group age, output is generated that includes all the steps
means of independent samples. Because the t-test and information noted above. For example, a com-
is a parametric test, one must ensure that the nor- parison of Instrumental Activities of Daily Living
mality assumption of the data is met and the (IADL) scores between elders who remained at
dependent variable is measured by either an inter- home and those who were institutionalized was
val or a ratio scale. Since the t-test is robust, the made using the Statistical Package for the Social
assumption of homogeneity is not really an issue, Sciences (SPSS). The output from this analysis is
but the calculation methods of the t-value vary shown in Tables 17.3a and 3b.
depending on whether the two groups are homoge- Table 17.3a indicates that there are 234 elders
neous or not. The test of homogeneity can be per- who were still living in their own home and 42 eld-
formed using either a Levene’s test or a Bartlett’s ers who were institutionalized. For those living at
Table 17.3 Independent t-Test for IADL Scores for Home Living and Institutionalized Elders:
SPSS Output
(a) Descriptive Statistics

FINAL N Mean Std. Deviation Std. Error Mean
IADL total score Living at own home 234 9.91 3.265 .213
Institutionalized 42 5.38 3.682 .568
(b) Independent t-Test

Levene’s Test for t-test for Equality of Means
Equality of Variances
Std. 95% Confidence
Sig. Mean Error Interval of the
F Sig. t df (two-tailed) Difference Difference Difference
Lower Upper
Equal 2.980 .085 8.122 274 .000 4.53 .558 3.435 5.633
variances
assumed
Equal 7.469 53.204 .000 4.53 .607 3.316 5.751
variances
not assumed
home, the mean of the total IADL scores was 9.91 If the dependent variable is measured on a ratio
(SD 3.265). For those who went to a nursing or interval scale, the raw scores have to be con-
home, the mean was 5.38 (SD 3.682). As shown verted to rank orders across both groups (not within
in Table 17.3b, Levene’s test indicated that two a group) and the U statistic is calculated as follows:
groups were homogeneous (p .05). Therefore,
one examines the statistics given in the row labeled n1 (n1 1)
U1= R1 and
“Equal variances assumed.” The t-value of 8.122 2
was determined by the Mean Difference (4.53)
over the Standard Error Difference (.558). The n2 (n1 1)
degrees of freedom (df) (274) were calculated by U2 R2 2
(234 – 1) (42 – 1). The t-value was significant at
p .001. The 95% confidence interval of the dif- where n1 is the smaller sample size, n2 is the
ference is between 3.435 and 5.633. The SPSS larger sample size, and R1 and R2 are the sum of
printout shows the probability of p .000 (see ranks for each group.
Table 17.3b) using a two-tailed test (nondirectional The smaller of these values (U1 or U2) is desig-
hypothesis). If an alternative hypothesis is direc- nated to be the calculated U value. This calculated
tional, one should use a one-tailed test. If that is value should be smaller than the critical value to be
the case, one can divide the p-value by 2. For statistically significant. This is opposite to what is
example, if p .064 for a two-tailed test, the p- done in parametric statistics. For parametric statis-
value would be .032 for a one-tailed test. tics, the calculated value should be larger than the
critical value to be statistically significant. When
Mann-Whitney U-Test the sample size is larger than 25, U is converted to
z and tested against the Z-distribution. The article
(a Nonparametric Test)
by Parush, Winokur, Goldstand, and Miller (2002)
The Mann-Whitney U-test (also called Wilcoxon used the Mann-Whitney U for testing their rank
rank-sum test) is a powerful nonparametric test order data. Since they had a sample size of 30, they
that is used in place of the independent t-test when used z-scores for reporting. Most statistical soft-
the data do not meet the statistical assumptions ware packages automatically convert the raw
required for parametric statistics. scores to rank orders when using the Mann-
Whitney U-test. This is available in many statisti- (r . 893) between the two scores that is signifi-
cal software packages. cant (p.001), one can use a paired t-test. The t-
value of 6.301 was calculated by the mean
Paired t-Test (a Parametric Test) difference (5.339) over the standard error of the
mean (.84739) and it is significant (p.001). The
For dependent or correlated samples, one uses a
degrees of freedom (df) are 58 (59 – 1). The con-
paired t-test to analyze the difference scores (d) or
clusion is that the FIM Motor score declined by
change scores within each pair. The formula to cal-
about 5.3 and the decline was significant over 4
culate a paired t-test is:
years.
d
t sd The Sign Test and Wilcoxon Signed

Rank Test (Nonparametric Tests)
where d is the mean of the difference scores and
sd is the standard error of the difference scores. There are two nonparametric tests that can be used
in place of a paired t-test. They are:
The degrees of freedom are n – 1, where n rep-
resents the number of pairs of scores. Tables 17.4a, • The Sign Test, and
b, and c contain an SPSS printout of changes in • The Wilcoxon Signed Rank Test.
two FIM scores in a single sample of 59 people
over a period of 4 years. The Sign Test is used when changes in the
Table 17.4a shows that the mean of the FIM direction of the data are of primary interest (i.e.,
Motor scores of 59 people was about 76.4, but 4 pluses and minuses, or data changing in either a
years later it became about 71.0. Therefore, the positive or a negative direction). Suppose an inves-
difference between the two mean scores is almost tigator wanted to find out what effect an occupa-
5.3 (Table 17.4c). For the paired t-test, the statisti- tional therapy intervention had on some behavior
cal assumption is whether there is significant cor- (e.g., using proper body mechanics during activ-
relation or not between the first or pretest scores ity) among 15 people. An increase of such behav-
and the second or posttest scores. Since Table iors can be recorded as “,” a decrease as “,”
17.4b tells that there is a very high correlation and no changes as “0.” The null hypothesis is that
Table 17.4 Paired t-Test for Changes in FIM Scores: SPSS Output

Mean N Std. Deviation Std. Error Mean
Pair 1 FIMYEAR1 76.3559 59 14.34040 1.86696
FIMYEAR5 71.0169 59 13.58180 1.76820
(b) Paired Samples Correlation

N Correlation Sig.
Pair 1 FIMYEAR1 and 59 .893 .000
FIMYEAR5
(c) Paired-test
Sig. (two-
Paired Differences t df tailed)
95% Confidence
Std. Std. Error Interval of the
Mean Deviation Mean Difference
Lower Upper
Pair 1 FIM YEAR1
FIM YEAR5 5.3390 6.50891 .84739 3.6428 7.0352 6.301 58 .000
the number of occurrences of negative and positive Effect sizes are often reported in the occupa-
change are equal. The alternative hypothesis is that tional therapy literature. Although in this chapter
there are more positive changes than negative and the previous chapter, power analyses devel-
changes. Suppose there were 10 positive and 3 oped by Cohen (1988) are described, some occu-
negative changes, ignoring two people who had a pational therapy researchers use the effect size
sign of “0.” One takes the smaller number of recommended by Stratford, Binkley, and Riddle
changes (i.e., 3). From Appendix A, Table L, the (1996). When a paired t-test is used, the latter pro-
Probabilities Associated with Values of X in the duces a more conservative effect size than
Binomial Test, the probability of having three pairs Cohen’s. Power tables for the t-tests are presented
indicating a negative change among 13 pairs is p in Appendix A, Table H.
.046. Therefore, the investigator would reject the
null hypothesis and conclude that there are more
positive changes than negative behavioral changes Comparison of More Than
after exposure to the intervention. This test should
be used only when the Wilcoxon Signed Rank Test Two Group Means (ANOVA)
cannot be used for most instances.
The Wilcoxon Signed Rank Test is similar to When comparing three or more group means (or
the Sign test, but it uses the T (capital T) statistic levels of a single independent variable), investiga-
that identifies the relative magnitude of differences tors use the analysis of variance (ANOVA). For the
and the direction of changes. The T will be the ANOVA, groups represent levels of the independ-
smaller sum of ranks of scores for one direction of ent variable, and the scores for each group on an
change. Both Wilcoxon Signed Rank Test and Sign outcome are the values for the dependent variable.
test are available in a statistical software SPSS. For example, consider a study in which an investi-
gator compares the quality of life of individuals
Power Analysis with chronic fatigue syndrome according to time
since onset (more than 10 years since onset,
To calculate an effect size for t-tests, one uses a d- between 5 and 10 years, and less than 5 years). In
index. This is the most commonly referred effect this instance, quality of life is the dependent vari-
size because it is easy to understand. It examines able and the independent variable is time (duration
the mean difference in terms of the standard devi- since the onset can be broken down into three
ation. For example, if the mean difference is half groups and therefore has three levels). In this case,
the value of the standard deviation, it is considered only one independent variable is being tested; thus,
to have a medium effect size. the proper name for this test is called a One-way
For independent t-tests with equal variances ANOVA. It is possible to use an F-test (i.e., a one-
assumed, the formula is: way ANOVA) when there are only two groups.
1
X X2 This will produce the same results as a t-test (i.e.,
d F t2).
s It is rare and not methodologically rigorous
For independent t-tests with equal variances not for researchers to use multiple t-tests instead of
assumed, it is: an ANOVA to compare more than two group
means. When one sets .05, and the compar-
1
X X2 isons are repeated using multiple t-tests, there is
d a potential cumulative type I error. This accumu-

S21 S22 lated error can be calculated, using the following
formula:
2
1 (1 )e
For paired t-tests, the formula is
where e is the number of possible comparisons.
d For example, if there are four levels in the inde-
sd
d 2 pendent variable, then six comparisons should be
made (Groups A vs. B, A vs. C, A vs. D, B vs. C,
B vs. D, and C vs. D). When one uses t-tests for six
The interpretation of the effect size index is:
comparisons, the type I error would be .26. Thus,
• Small is d .20, the cumulative error is 26% (not the original 5%).
• Medium is d .50, and In order to keep the type I error at a 5% level,
• Large is d .80. investigators use the F-statistic. When investiga-
tors use an ANOVA, and an F-value is statistically intervention they receive. In ANOVA, the variance
significant, a multiple comparisons test or a post- is called mean square (MS). As discussed in
hoc test is typically used thereafter. Commonly, Chapter 16 variance is the ratio of the sum
these tests are used to identify which pairs within of squares (SS) and degrees of freedom. The fol-
the three or more groups being compared are sig- lowing is an example of a One-way ANOVA
nificantly different from the determined minimum and compares visual motor integration scores
mean difference. among three groups. As shown in Table 17.5a, the
Because it is a parametric test, the statistical group means are 4.58, 7.89, and 8.22. The grand
assumptions for an ANOVA are that the sample mean (the mean of all cases) is 6.90. The concep-
groups are randomly drawn from a normally dis- tual formula to derive the F ratio is illustrated
tributed population with equal variances among Table 17.5b.
the groups. When the sample sizes for each group The total SS (the last row in Table 17.5b) is the
are similar, minor violations of normality and sum of squared differences between an individual
homogeneity of variance do not seriously affect score and the grand mean. It is expressed as:
the population estimate. However, if the sample
sizes are different, and gross violations of homo- SS total Σ(X
XG) 2
geneity of variance are present, this violation may
increase the chances of a type I error. In that case, where X denotes each individual score and XG
a nonparametric test would be used. is the grand mean.
The between-groups SS is the sum of the sam-
One-Way ANOVA (A Parametric Test) ple size in the group multiplied by squared differ-
ences between the group mean and the grand
The term “one-way” indicates that there is one mean.
independent variable, or a factor, with three or
more levels or groups. If there are two independent SS between Σn (X
J –
XG) 2
variables involved in one analysis, it is called a
two-way or two-factor analysis of variance and it where n is the sample size for the group, and
XJ
is one of the multivariable tests that will be is the group mean.
explained in the following section. ANOVA The within-groups SS is the Sum of Squared
involves only one dependent variable. differences between the individual score and the
The null hypothesis of a one-way ANOVA is group mean.
that all group means are equal. It is denoted as:
SS within Σ(X –
XJ) 2
H0: 1 2 3 …… k
The degrees of freedom for the total sample
where (mu) is the population mean, and k is
are: (N – 1); for between-groups are: (k – 1); and
the level or group.
for within-groups are (N – 1) – (k – 1), N – k, or k(n
The alternative hypothesis can vary. For exam-
– 1). The MS is determined by the sum of squares
ple, it can be:
divided by the degrees of freedom (SS/df) for each
H1 : 1 2 3 , or (1 2) (3 4) source. Finally, the F- value is derived by dividing
the between-groups MS by the within-groups MS
The F-test is used to determine how much of (MSbetween /MSwithin).
the total variability in the sample is explained by As shown in Table 17.5c, the resulting F-value
the differences among group means (between- is 32.371. The degrees of freedom for the total
groups) and the variability among subjects in each sample is one less than the total number, so dftotal
group (within-groups). Therefore, the F ratio is 30 – 1 29. For the between-groups, it is one
determined by comparing the between groups vari- less than the number of groups (k – 1), thus, dfb
ances to the within groups variances. The former 3 – 1 2. Finally, for the within-groups or error,
variability is explained by the independent vari- the degrees of freedom is the total sample size
able, meaning the differences in the group mean minus the number of groups (N – k), or dfe 30 –
are due to the independent variable or intervention. 3 27. The degrees of freedom (df) for the F-
The latter variability is unexplained (by the inde- value include both dfb and dfe. Then the critical
pendent variable) and is, therefore, referred to as value of F can be obtained from Table for Critical
error variance. The assumption is that individual Values of F at .01 in Appendix A, Table D. If
scores in each group are the same due to the same the level has been set at .05, looking at df of 2
Table 17.5 One-Way ANOVA for Independent Samples: Visual Motor Integration for Three Groups
(N 30)
(a) Raw Scores, Group Means, and Grand Mean

Group 1 (n 10) Group 2 (n 10) Group 3 (n 10) Total (N 30)
Raw score 2.50 7.80 7.00
3.50 5.80 6.50
4.70 6.80 8.30
2.90 7.50 7.80
5.80 8.80 9.20
6.30 9.00 9.90
4.80 7.90 8.00
5.20 8.50 7.40
4.90 8.00 9.60
5.20 8.80 8.50
Mean 4.58 7.89 8.22 6.90 (Grand mean)
(b) Conceptual Formula

Source of Variance Sum of Squares df Mean Square (SS/df) F
Between groups Σn(Xj XG)2 k–1 SS (between group)/k – 1 MS (between group)
MS (within group)
Within groups Σ(X Xj)2 k (n – 1) SS (within group)/k(n – 1)
or N – k
Total Σ(X XG)2 N–1
(c) ANOVA Summary Table

Between groups 81.05 2 40.52 32.37
Within groups 33.80 27 1.25
Total 114.85 29
and 27, the critical value of F is 3.35. Since the Multiple Comparison Tests
calculated value is 32.37, and larger than the criti- for One-Way ANOVA
cal value, one would conclude that there is a sig-
nificant difference among the three group means. Multiple comparison tests are classified as:
However, one does not yet know which group dif- • Post-hoc (done with an ANOVA procedure), and
ference is contributing to the significant F. In fact, • A priori test (planned comparisons).
an F-test is known as an omnibus test for this very
reason. It is an overall test that specifies whether
Post-hoc Tests
there are significant differences but it does not
specify what kinds of differences exist among Post-hoc tests are completed after an investigator
which groups. conducts a one-way ANOVA and finds a signifi-
Therefore, post-hoc tests are used to identify cant F-value. These tests are done in the following
which differences between the two mean score are way. First, the groups are arranged in the order of
contributing the significant F. They are called the size of the mean. Second, the difference
post-hoc analyses because they will be performed between these two means is obtained, and third,
once an ANOVA has been completed and it has the difference is compared with a minimum signif-
revealed significant results. icant difference (MSD). If the absolute difference
between the two group means is equal to or greater

MSe
than the minimum significant difference, then the MSD q
difference is considered significant. The ways to n
identify the minimum significant difference vary
depending on the methods, but there are only two where q is the critical value of the studentized
types of error rates that different methods use: range statistic, MSe is mean square for error or
within-groups from the ANOVA output summary
• Per comparison error rate, and table (see Table 17.5c), and n is a sample size in
• Familywise error rate. each group.
From Table 17.5a, the means of Groups 1, 2,
The former (per comparison) uses a 5% or a 1%
and 3 are 4.58, 7.89, and 8.22, respectively. The
margin for a type I error for each single compari-
difference between Groups 1 and 2 is (7.89 – 4.58
son. The latter (familywise) sets a type I error of
) 3.31, that of 2 and 3 is (8.22 – 7.89 ) 0.33,
5% or 1% for all comparisons in one experiment.
and that of 1 and 3 is (8.22 – 4.58 ) 3.64. Using
The latter may be the more conservative and pre-
.05, and MSe from Table 17.5c,
ferred approach for multiple comparisons for the
same reason discussed earlier (i.e., performing

1.25
multiple comparisons to answer a single study MSD 3.51 1.24
question increases the margin for a type I error). 10
One of the most widely used post-hoc tests is
Tukey’s honestly significant difference (HSD) test. Therefore, the differences between Groups 1
This approach uses a familywise error rate. Its and 2 (3.31) and Groups 1 and 3 (3.64) are larger
minimum significant difference is identified by: than the MSD (1.24) and are significant, but the
Table 17.6 Multiple Comparisons: SPSS Output
(a) Tukey’s Honestly Significant Difference

(I) (J) Mean Std. 95% Confidence
Group Group Difference (I–J) Error Sig. Interval
Lower Bound Upper Bound
1.00 2.00 3.31000(*) .50038 .000 4.5506 2.0694

3.00 3.64000(*) .50038 .000 4.8806 2.3994
2.00 1.00 3.31000(*) .50038 .000 2.0694 4.5506
3.00 .33000 .50038 .789 1.5706 .9106
3.00 1.00 3.64000(*) .50038 .000 2.3994 4.8806
2.00 .33000 .50038 .789 .9106 1.5706
*The mean difference is significant at the .05 level.
(b) Scheffé Comparison**

(I) (J) Mean Std. 95% Confidence
GROUP GROUP Difference (I–J) Error Sig. Interval
Lower Bound Upper Bound
1.00 2.00 3.3100(*) .50038 .000 4.6060 2.0140
3.00 3.6400(*) .50038 .000 4.9360 2.3440
2.00 1.00 3.3100(*) .50038 .000 2.0140 4.6060
3.00 .3300 .50038 .806 1.6260 .9660
3.00 1.00 3.6400(*) .50038 .000 2.3440 4.9360
2.00 .3300 .50038 .806 .9660 1.6260
*The mean difference is significant at the .05 level.

**Dependent variable: SCORE.
difference between Groups 2 and 3 (0.33) is not Kruskal-Wallis One-Way ANOVA

significant. (a Nonparametric Test)
Many other post-hoc tests are available in sta-
tistical software. Table 17.6a shows the SPSS out- When it is not possible to use a parametric test for
put of a post-hoc analysis of the data of three or more group comparisons (such as when
visual–motor integration (which was presented the measurement is made using an ordinal scale,
Table 17.5a), using Tukey’s HSD. The results indi- the groups are not normally distributed, or group
cate that Group 1 is significantly different from variance is not homogeneous) investigators use the
Group 2 and Group 3 (both p = .001) but Group 2 Kruskal-Wallis nonparametric procedure. If this
and Group 3 are not (p .789). test is used in a study in which the measurements
Another frequently used post-hoc procedure is are done using a ratio or interval data, the data
the Scheffé comparison. This is a more rigorous must be converted to ordinal rank data. For exam-
method than Tukey’s HSD. This method also uses ple, if there were 15 cases in total, the ranks would
a familywise error rate. The calculation formula be between 1 and 15. In the case of ties, one would
for the Scheffé comparison is: use the mean ranks. For example, if two cases
scored the same after the case ranked 2, they

2MSe would be ranked 3.5 by dividing the sum of rank-
MSD (k

1)F
n ings 3 and 4 by 2.
The Kruskal-Wallis H statistics is calculated as
Using the results of MSe from the Visual–Motor follows:
Integration example shown in Table 17.5c, and F is 12 R2
the critical value of F from Appendix A, Table D H
N (N 1)
Σ
n
3(N 1)
for 2 and 27 degrees of freedom at
where N is the number of total cases, n is the
MSD
(3 1)3.35 2 1.25
1.29
10
number of cases in each individual group, and R2
is the squared sum of ranks for each group.
H is distributed as Chi-square with df k – 1.
Using this method, although the results are the Therefore the critical value is obtained from
same in this particular example, the MSD is larger, Appendix A, Table C. The numbers 12 and 3 are
requiring a larger mean difference. An SPSS print- treated as constants in the equation. Suppose the
out of this post-hoc test is presented in Table 17.6b. data in Table 17.5a are analyzed using the Kruskal-
A discrepancy between the results of the F-test Wallis test; then the total ranks are 56.5, 197.0, and
and post-hoc tests sometimes occurs because two 211.5 for each group. Therefore, H 18.939. The
different statistical procedures are used. Even if the critical value for degrees of freedom 2 at
F-value is significant, there may be no significant .05 is 5.99. Since the H-value is larger than the
differences between pairs or vice versa. In these critical value, one concludes that there is a signifi-
instances, investigators may report both results. cant difference for at least one of the pair.
Planned Comparisons
Multiple Comparisons for
In some cases, investigators will decide which the Kruskal-Wallis ANOVA
pairs should be compared before data collection
takes place. This is called an a priori or planned To calculate multiple comparisons for the Kruskal-
comparison. In this instance, even if the F-value is Wallis ANOVA, investigators estimate the pairwise
not significant, the specific comparison will take R
place because prior to the data analyses, the inves- difference among the mean rankings: R n . The
tigator decided that the comparisons should be
number of pair wise comparisons is determined by
done. For example, the investigator may be mainly
k (k – 1)/2. If there are three groups, three compar-
interested in knowing the difference between
isons should be made. This is calculated by 3(3 –
Group 2 and Group 3 in the preceding example.
1)/2 and it tests Group 1 vs. Group 2, Group 2 vs.
Then, the statistical procedure is an independent t-
Group 3 and Group 3 vs. Group 1. The computa-
test. The results of an a priori test are usually pre-
tional formula for the post-hoc procedures is:
sented together with the results of an omnibus test

or the F-test. For multiple comparisons, the N(N 1) 1 1
Bonferroni t-test (also called Dunn’s multiple ⏐R
1
R2⏐ ≥ z ( )
12 n1 n2
comparison procedure) is used (see later section).
where N is the total number of cases and n1 and pendent samples. However, the choice of statisti-
n2 are the number of cases in each group of com- cal procedures should be based on a study
parisons. The z-score at .05 is obtained from design and whether subjects are independent or
Table 17.7 in this chapter. Using the example from correlated rather than on issues involving statisti-
the previous section (Table 17.5a), the mean ranks cal power.
are 4.58, 7.89, and 8.22. The difference between The repeated measures ANOVA does not
Groups 1 and 2 is 3.31; Groups 1 and 3, 3.64, and require the statistical assumption of homogeneity
Groups 2 and 3, 0.33. To find the significant dif- among groups because it involves only one group.
ference, these mean rank differences should be Rather, it requires that the variances of the score
larger than difference among each group are similar and cor-
related. This is called the assumption of sphericity.

30(30 1) 1 1 Statistical software packages will present the
2.394 ( ) 9.425
12 10 10 assumption of sphericity. If this assumption is vio-
lated, the test will have an inflated type I error rate
Therefore, one concludes that none of the group and significant differences may be found when
rank means are significantly different. none actually exists.
The repeated measures ANOVA uses the F-
Repeated Measures ANOVA ratio. The numerator of the F-ratio is the variance
(a Parametric Test) between treatment groups. The denominator of the
F-ratio is the error variance. A repeated measures
Study designs in which subjects are correlated and
design removes individual differences from the
measurements are taken more than two times are
within-group variance (in one-way ANOVA)
called repeated measures designs or within-
because the same people are in all treatments.
subjects designs. The statistical test that is used for
Expressed numerically, the error term is the resid-
these designs is a repeated measures ANOVA. The
ual of the total variances minus the sample vari-
statistical advantage of using a repeated measures
ance (the error term is smaller than the error term
ANOVA over a series of one-way ANOVAs is that
of the one-way ANOVA). The conceptual formula
the repeated measures ANOVA reduces the error
is presented in Table 17.8a.
variance (i.e., variability that is not due to the inde-
One can illustrate the repeated measures
pendent variable or the treatment effect). This is
ANOVA using the same example of visual motor
because the repeated measures ANOVA controls
integration that was used above for the one-way
for individual differences by subtracting the vari-
ANOVA. To do so, one must assume that the three
ances of individual differences from within-group
groups are the three measurements over time for
variances in one-way ANOVA. This reduced
the same persons (rather than three different
error variance results in a larger F-ratio. Thus,
groups of subjects as was assumed earlier when
the single-factor repeated measures ANOVA is
presenting the one-way ANOVA in Table 17.5a).
more powerful than one-way ANOVA with inde-
In this case, the calculation is presented in Table
17.8b.
Table 17.7 Critical Values of z for Multiple The conceptual formula for F is MSbetween/
Comparison with H Statistics and Xr2 MSerror, presented in Table 17.8a, and they are as
follows:
Number of Comparisons z at .05
1 1.960 SStotal (X X G)2
2 2.241 SSbetween n(X j XG)2
3 2.394 SSsubject k(Xs XG)2
4 2.498 SSerror SStotal – SSbetween – SSsubject
5 2.576
6 2.638 where X is an individual score, Xj is the group
7 2.690 mean, XG is the grand mean, XS and is the mean of
8 2.734 each person across interventions.
Using the same data presented in Table 17.5a,
Table 17.8b shows the calculated values in the
Adapted from Table AII of Siegel, S., & Castellan,
N.J. (1988). Nonparametric statistics for the ANOVA summary table. The degrees of freedom
behavioral sciences (2nd ed.). New York: for between-group variances are k – 1 2 and that
McGraw-Hill, with permission. for error is (n – 1) (K – 1) (10 – 1) (3 – 1) 18.
Table 17.8 Repeated Measures ANOVA
(a) Conceptual Formula

Between groups Σn(Xj
XG)2 k1 SS (between MS (between group)
group)/k 1 MS (error)
Subject Σk(Xs XG)2 (n 1)
Error Total SS Between group (n 1) X SS (error)/(n 1)
SS Subject SS (k 1) (k 1)
Total Σ(X XG)2 N1
(b) ANOVA Summary Table: Constructed from SPSS Output

Between groups 81.049 2 40.524 98.245
Subject 26.376 9 2.931
Error 7.425 18 .412
Total 114.85 29
The mean square (MS) for each group is calculated Friedman test would be appropriate, owing to the
by SS/df; therefore, MS between is 81.049/2 small sample size. It uses rank orders within a sub-
40.524 and MS error is 7.425/18 .412, thus F = ject. In this instance the ranks range between 1 and
40.524/.412 = 98.245. The F-value for the 3 for all subjects (i.e., when they had the most,
repeated measures ANOVA is much larger than the least, and intermediate levels of fatigue). When
F for the one-way ANOVA (32.37). there is a tie, ranks are averaged. Then, ranks are
summed within each column (measurement/time).
Multiple Comparison Tests for The null hypothesis is that the rank sums for all
Repeated Measures ANOVA measurements are the same. If the null hypothesis
is rejected, then at least one pair of measures will
After finding the significant F, the investigator is show a difference.
often interested in knowing when the significant The formula to calculate the Friedman
change occurred. In this instance, one uses a (chisquare r) value is:
Bonferroni correction. This method reduces the
type I error by dividing the level by the number 12
of comparisons. For example, in the above case X2r
nk (k 1)
ΣR2 3n(k 1)
with three pair comparisons, the alpha level
becomes .05/3 .0167. To use this method, the where n is the number of subjects and k is the
investigator performs three paired t-tests and each number of measurements/times (groups), and R2 is
p-value is compared with .0167. When a computer the squared sum of rankings in each group.
statistical package is used, this multiple compari- The critical value for Chi-square r is found in
son procedure is called contrast. The results show the Chi-square r distribution with k minus 1
that Group 1 is significantly different from Groups degrees of freedom. The critical values are pre-
2 and 3. sented in Table 17.7 in this chapter. The post-hoc
test uses a familywise error rate. The formula to
Friedman Two-Way Analysis of Variance determine the minimum significant difference
of Ranks Test (a Nonparametric Test) (MSD) for all pair differences is:

The nonparametric test that is commonly used in nk(k 1)
⏐R1 R2⏐≥ z
place of the repeated measures ANOVA is the 6
Friedman test. If an investigator measures three
fatigue levels of five persons with multiple sclero- where R1 and R2 are the rank sums, n is the
sis before exercise (time 1), right after exercise number of subjects, k is the number of measure-
(time 2), and 1 hour after exercise (time 3), the ments within a person, and the z is from Table 17.7.
Power Analysis groups factor and a between-groups factor. Also,

simple cell effects are discussed to compliment
For the ANOVA, the effect size index is f and it is omnibus factorial analyses.
defined by:

SS b
Two-Way ANOVA
f
SS e A two-way analysis of variance (ANOVA) or a
two-factor ANOVA involves two independent vari-
where SSb is the sum of squares of the ables (i.e., A and B). In this analysis, there are three
between-groups and SSe is the sum of squares possible questions:
of the error that are obtained from the summary
• What is the main effect of variable A, independ-
table of ANOVA. Using the data provided in Table
ent of B,
• What is the main effect of variable B, independ-

81.05
Table 5c, f is
33.80 1.55.
ent of A, and
• What is the joint effect or interaction of variables
A and B?
The interpretation of f is:
The main effects are effects of two separate
• Small f .10, independent variables.
• Medium f .25, and A two-way ANOVA will be illustrated using a
• Large f .40. hypothetical study of the dependent variable,
instrumental activities of daily living (IADLs)
Therefore, 1.55 is very large. An effect size can
(scale score range 0–14). The two independent
be very large such as more than 1.0. The f index can
variables are: (A) living status (two levels, i.e., liv-
be applied to independent samples and repeated
ing alone versus with someone), and (B) sex (two
measures ANOVA also. The sample sizes needed
levels). The means for each cell and the means for
for ANOVA with various effect sizes for the power
levels of the main effects, called marginal means,
of .80 at .05 are found in Appendix A, Table I.
are presented in Table 17.9. The standard devia-
tions for each are presented in parentheses. In a
two-way ANOVA, one compares marginal means,
Multifactorial Analysis not the means in cells for the main effects.
of Variance: Comparison
Therefore, for sex, one compares 7.6 for males
with 8.5 for females. For living status, one com-
of Group Means for Two or pares 9.4 for living alone with 6.9 for living with
someone. In addition to main effects, one exam-
More Independent Variables ines combined effects (interaction effects) of levels
of independent variables on a dependent variable.
When there are more than two independent vari- Using the mean values in four cells from the data
ables and one dependent variable in an analysis, in Table 17.9, Figure 17.1a shows that there is no
and when the purpose of the analysis is to compare interaction because the two lines (Living Alone
group means, one uses a multifactorial design or and Living with Someone) are not crossing and are
multiway ANOVA. This is called a multivariable almost parallel. Figure 17.1b indicates an interac-
approach and there are no corresponding nonpara- tion.
metric statistics. An interaction effect occurs when the relation-
This section focuses only on the two-way ship between two variables differs depending on
ANOVA and mixed design where there is a within- the value of a third variable.
Table 17.9 Total IADL Score by Sex and Living Status Categories (N 758)
Sex Living Alone Living With Someone Total (Marginal Means)
Male 9.6 (2.9) 6.0 (4.1) 7.6 (4.0)
Female 9.4 (2.8) 6.9 (4.0) 8.5 (3.5)
Total (marginal means) 9.4 (2.8) 6.9 (4.0) 8.3 (3.6)
A)
IADL scores
Living alone
10
9 Living with someone
Male Female
B)
IADL scores
Male
10
9 Female
Figure 17.1 Types of Interaction.

(A) No interaction. (B) Possible Living Living with
interaction. alone someone
Using the example above, the null hypotheses live alone, and A2B2 indicates females who live
are: with someone.
The statistical assumptions are that the inde-
1. H0: A1 A2
pendent variable must be comprised of mutually
2. H0: B1 B2
exclusive groups and the dependent variable must
3. H0: A1B1 A1B2 A2B1 A2B2
be normally distributed and demonstrate homo-
Where A1 is male, A2 is female, B1 is living geneity of variance across groups.
alone, B2 is living with someone, A1B1 denotes A hypothetical summary table for a two-way
males who live alone, A1B2 denotes males who ANOVA is shown in Table 17.10. This table is
live with someone, A2B1 indicates females who based on an analysis of the impact of two inde-
Table 17.10 Hypothetical Summary Table for Two-Way ANOVA

Sex (M or F) 120.98 1 120.98 10.37
Living regions 105.03 2 52.515 4.50
Interaction 128.37 2 64.185 5.5
Error 8770.53 752 11.67
Total 9910.64 757
pendent variables, region in which a person lives Appendix A, Table D. For sex, the critical value is
(.01) F (1,752) 6.63 and that for living regions and
(East, West, and South) and sex (male versus
female) on daily activity level. The level was set interaction is (.01) F (2, 752) 4.61. Therefore, the
at .01. main effect of living regions is not significant, but
From the degrees of freedom in this table, one the main effect of sex and the interaction effect
can tell there are 758 subjects in the study (df total (living regions x sex) are significant. Since the
N – 1 757), sex has two levels (df sex k – 1 interaction effect is significant, Figures 17.1c and
1), and living regions have three levels 17.1d show almost and complete crossing lines.
(df living regions k – 1 2). The degrees of freedom In two-way ANOVA, before examining the two
for the interaction is 1 2 2 and that of error is main effects, one should examine the interaction
obtained by df total – df sex – df living regions –df interaction effect. If it is not significant, then one can report
757 – 1 – 2 – 2 752. All mean square values the main effects as shown in the results. If the
are determined by dividing the sum of squares for interaction is significant, the next step is to exam-
each effect by its associated degrees of freedom. ine the type of interaction, which can be ordinal or
The F-ratio for sex is determined by MS sex/MS error disordinal. In an ordinal interaction, one effect is
120.98/11.67 10.37. The F-ratio for living not affected by the other effects. Figure 17.1c
regions is found by 52.515/11.67 4.50. The F- shows an ordinal interaction in that regardless of
ratio for the interaction is calculated by living regions, males are more active than females.
64.185/11.67 5.5. Note that the error term is the Therefore, lines for male and female do not cross.
same for all main effects and interaction effects. The rank order of activities for East is one for
The critical value for each effect is obtained from males (more active) and two for females. This
C)
Male
5
4 Female
East West South
D)
East
5
West
4 South
Male Female Figure 17.1 (C) Ordinal. (D) Disordinal.

order is the same for West and South; therefore, it effect for time (a within-factor) was significant but
is called ordinal interaction, and one can say that the main effect for intervention, the between-
males are always active than females. For the con- factor, was not. Then, pre- and posttest scores for
clusion, one can look at the F-value of sex, a main the treatment group were compared using only
effect. paired t-tests. The results of t-values, p-values, and
In contrast, Figure 17.1d shows a disordinal effect sizes (d-index) were reported. One of the
interaction, presenting crossed lines. Among dependent variables, visual–motor control, showed
males, residents in the East are the most active, significant interaction effects (p .004). The
then residents in the South. Residents in the West pretest scores of the treatment and control groups
are most inactive. This order is not the same for were 9.14 and 15.44, respectively, and the posttest
females. Females living in South are most active, scores were 11.25 and 16.67, respectively. Note
followed by West, and then East. When the order that the interactions were ordinal (the control
of the dependent variable scores for levels of one group always scores higher than the treatment
independent variable is not the same across lev- group and the posttest scores are always higher
els/groups of another independent variable in the than the pretest scores). Therefore, both main
X-axis, a disordinal interaction occurs. When it effects were considered valid (p .111 for the
happens, it is meaningless to compare mean scores within-factor and p .715 for the between-factor).
across the first independent variable (Living Since the main interest of the research was the
regions). The F-value for the main effect is often effect of intervention, pre- and posttest scores of
not significant. visual–motor control for only the treatment group
The next step is to examine males and females were analyzed, using a paired t-test. The result was
separately. These separate effects are called simple statistically significant (p .039) with a moderate
effects. Since six comparisons are made using effect size (d .58).
independent t-tests, the type I error must be con- There is no comparable nonparametric test for
trolled; using a Bonferroni correction. The level multifactor analyses such as two-way ANOVAs,
can be set to .05/6 .0083. mixed designs, and multiway ANOVAs. The
ANOVA is robust, and thus minor violations of
statistical assumptions might not alter the results
Mixed Design
significantly. However, the same does not apply to
When a single experiment involves at least the repeated measures ANOVA.
one independent factor (between-groups) and one Since a mixed design reflects the combination
repeated factor (within-groups), the design is of the two statistical methods, when dealing with a
called a mixed design and it is analyzed using small sample size, one may not be able to use a
a repeated measures ANOVA with a between- repeated measures ANOVA with a between-factor.
factor. A mixed design is the most often used One solution is to conduct separate analyses for
design in occupational therapy experimental the between-factor and the within-factor.
research studies.
This design combines a one-way ANOVA for
Simple Cell Effect
the between-factor and a repeated-measures
ANOVA for the within-factor. For example, in a Because of their omnibus nature, significant main
study designed to identify the effects of school- effects for a two-way ANOVA and a mixed design
based occupational therapy services on students’ simply indicate that at least one pair of marginal
handwriting (Case-Smith, 2002) a mixed design means is significantly different. Using the example
was used. The within-factor was time (i.e., pre- of Figure 17.1c, the ANOVA compares marginal
and posttest scores). The between-factor was inter- means of Male (activity mean is 3) and Female
vention (i.e., students who received an occupa- (activity mean is 1.5), and indicate that the F-value
tional therapy intervention and a control group that for sex is significant. Since this analysis compares
did not receive the intervention). There were three only two different levels (Male and Female), there
separate dependent (outcome) variables: handwrit- is no post-hoc procedure. However, the investiga-
ing legibility, speed, and associated performance tor may be interested in knowing the difference
components. For all components of legibility between Male and Female in East and West. In this
scores, posttest scores were higher than pretest case, the next step is to examine the simple cell
scores regardless of occupational therapy interven- effects, such as the difference between the mean of
tion. Also, the control group always scored higher Male (4) and that of Female (1) for East and the
than the treatment group. Therefore, there were no mean of Male (2) and that of Female (1.5) for
significant interaction effects (p .054). The main West, using independent t-tests. It is best to deter-
mine the intention of particular simple cell analy- are 1.00 or –1.00, and if there is no correlation, the
sis prior to conducting an ANOVA. This process is correlation coefficient is 0. In some instances, two
called an a priori test or a planned comparison. variables have a curvilinear relationship, such as a
Actually, in such a case, one may not even be U-shape or an arch-shape. This occurs when one
interested in conducting an F-test to find overall variable follows the pattern of another variable up
effects. Therefore, regardless of the significance of until a certain point and then it reverses direction.
the F-value, one would conduct a priori tests. This For example, half of variable X might have a
procedure involves a Bonferroni correction or strong positive correlation with variable Y, but the
Dunn’s multiple comparison procedure. It requires other half of variable X might have a strong nega-
one to set the level by dividing the set level by tive correlation with variable Y. In this case, the
the number of comparisons. For example, if one correlation coefficient will be close to 0. This
sets the level as .05 and wants to compare three means there is no linear correlation but has a curvi-
sets of means, then the altered level would be linear relationship. To find the nature of the rela-
.05/2 .025. While this approach reduces the like- tionship between the two variables, it is best if one
lihood of the type 1 error, it does make it more dif- plots them.
ficult to find significant results.
Pearson Product Moment Correlation
Correlation Definitional Formula
The Pearson Product Moment correlation coeffi-
Correlation is the extent to which two or more
cient is the most often used measure of correlation.
variables are related to one another, and does not
The symbol for the coefficient is r and it is used
necessarily mean that one causes another although
when both variables are normally distributed and
often the causal relationship may be obvious, such
measured using an interval or ratio scale. It is
as age up to 20 years or so and height. Height does
called “product-moment” because it is calculated
not cause age, but as a boy becomes older, he
by multiplying the z-scores of the two variables to
becomes taller; therefore, age is a cause of height.
get their “product” and then calculating the aver-
Nevertheless, the purpose of correlation analysis is
age, which is called a “moment,” of these products.
to identify the association between two variables,
The conceptual formula is:
not to identify a causal relationship. Height and
weight often have a high correlation, which means, Σ ZX ZY
if a person is tall, he or she tends to be heavier than r N
a short person. In this case, the two variables are
denoted by X and Y (height and weight, respec- where z (X X )/SD and N is the number of
tively) and when the variable (X) increases one pairs. The degrees of freedom are N – 2.
unit, the variable (Y) also increases. This is The second formula is:
referred to a bivariate correlation (or zero-order
correlation), since the relationship between only r Covariance/SDxSDY
two variables is of concern. There is another type where covariance is a measure of the joint vari-
of correlation, such as a multiple correlation, ances of two variables and calculated by:
where the relationship between one dependent
variable and several independent variables is Σ(X X)(Y Y)

examined. It is discussed in the next section. In this N1
section two correlations are explained:
and SDxSDY is the product of the standard devi-
• Pearson (or Pearson’s) correlation, in which both
ation of the two variables.
variables should be measured by a continuous
A correlation can be easily identified by creat-
scale, and
ing a scattergram or scatter plot. In Figures 17.2a
• Spearman (or Spearman’s) rho in which variables
through 17.2d, various patterns of correlation are
are measured by rank orders.
shown. The first graph (17.2a) presents a perfect
positive correlation (r 1); the second graph (17.2
Correlation Coefficient
b) is a perfect negative correlation (r –1); the
Pearson Correlation uses the correlation coeffi- third (17.2c) shows a moderate relationship (r
cient to express the strength of the relationship .60); and the fourth (17.2d) illustrates a pattern in
between two variables. The strongest correlations which there is no correlation (r 0).
A) B)
C) D)
Figure 17.2 Various patterns of correlation. (A) Perfect positive correlation. (B) Perfect neg-
ative correlation (r –1.0). (C) Moderate correlation (r .60). (D) No correlation (r 0).
Correlation Matrix older they are, the lower the BMI score. The low-
est correlation is r –.026 between age and num-
When an investigator uses many variables in one ber of illnesses. In this matrix there is a negligible
study, it is often useful to identify bivariate corre- correlation between age and number of illnesses.
lations among them. Then, one uses a correlation
matrix such as the one shown in Table 17.11. In
this matrix, there are five variables: age, number of Strength and Significance of
illnesses, Body Mass Index (BMI), physical dis- the Correlation Coefficient
ability level, and Mini Mental State Exam
The interpretation of the correlation coefficient is
(MMSE) score. As shown in the table, the correla-
the strength of the relationship. The following is
tion of each variable with itself is 1. The correla-
how values are typically interpreted:
tion coefficients on either side of the diagonal are
the same. For example, the correlation between • 0–.20 suggests a negligible correlation,
age and MMSE (–.240) is found in two places: one • .20–.40 is a low correlation,
in the bottom of the first column and the other in • .40–.60 is a moderate correlation,
the top of the last column. Therefore, one can look • .60–.80 is a high correlation, and
at either at the upper or the lower side of the matrix • .80–1.00 is a very high correlation.
that is separated by ones. The highest correlation
in the matrix is r –.290 between age and BMI. Other researchers interpret the values of 0–.25
This correlation is interpreted to mean that the as little or no relationship .25–.50 as fair, .50–.75
as moderate or good, and .75 and above are good considered large. Sample sizes needed for r at
to excellent. The interpretation often is based on .05 with power .80 can be found in Appendix A,
the area of study and it reflects the typical range of Table J.
correlations demonstrated in that field.
The significance of the correlation coefficient is Spearman Rank
expressed as the probability (p). Therefore, if it Correlation Coefficient
is set as .05, then the interpretation is that it is
either significant or not. The p-value should not be When the correlation is between two rank-ordered
interpreted as reflecting the strength of the rela- variables, the nonparametric statistic, called
tionship. Critical values of r are found in Appendix Spearman rank correlation, or Spearman’s rho (rs),
A, Table F. is used. To calculate rs, one ranks the scores within
Correlation is very sensitive to the sample size. the variable X and Y separately. A rank of 1 is
Therefore, if one has a very large sample size such given to the smallest value and ties are given the
as in Table 17.11 (N 389), even if a correlation average of their ranks. Then the difference
coefficient is negligible such as r .145, the p- between the rank X and the rank Y(d) is calculated
value is significant at .01. In this case, the cor- and squared (d2) for each pair. The computational
relation is significant; but its magnitude is formula is:
negligible and does not indicate a meaningful
6Σd2
association between the two variables. The coeffi- rs 1
cient of determination (r-square) accurately n(n2 1)
describes the strength of a bivariate correlation. It
where n is the number of pairs. As most non-
represents the percentage of variance of one vari-
parametric statistics, this has less power than
able that is explained by the variance of the other
Pearson correlation.
variable in a bivariate correlation.
Effect Size of Pearson Correlation

Regression
The effect size for Pearson correlation is simple
because r is the same as the effect size. When r Regression is a statistical technique used to ask
.10, it is small, r .30 is medium, and r .50 is whether it is possible to predict some variables by
Table 17.11 Correlation Matrix: SPSS Printout

Number Physical
Age of Illness BMI Disability Level MMSE
Age Pearson Correlation 1 .026 .290(**) .145(**) .240(**)
Sig. (two-tailed) . .613 .000 .004 .000
N 389 389 389 389 389
Number of Pearson Correlation .026 1 .128(*) .287(**) .069
illness Sig. (two-tailed) .613 . .011 .000 .172
N 389 389 389 389 389
BMI Pearson Correlation .290(**) .128(*) 1 .038 .158(**)
Sig. (two-tailed) .000 .011 . .461 .002
N 389 389 389 389 389
Physical disability Pearson Correlation .145(**) .287(**) .038 1 .213(**)
level Sig. (two-tailed) .004 .000 .461 . .000
N 389 389 389 389 389
MMSE Pearson Correlation .240(**) .069 .158(**) .213(**) 1
Sig. (two-tailed) .000 .172 .002 .000 .
N 389 389 389 389 389
**Correlation is significant at the 0.01 level (2-tailed).

*Correlation is significant at the 0.05 level (2-tailed).
knowing other variables. It is used to answer ques- The calculation of the regression equation starts
tions, such as can a parent’s stress level (Y) be pre- with that of b.
dicted based on the infant’s length of stay in a
hospital (X)? If there is a strong correlation b the ratio between the standard deviation
between X and Y, the prediction would be easier of Y to that of X, SDy /SDx
than if the correlation is weak. bX
aY
In this case, infant’s length of hospital stay is
the independent variable (X) and the parent’s stress Therefore, the observed value, Y is calculated
level is the dependent variable (Y). In regression by a bX residual or error, but typically, the
analysis, some researchers use the term “a predic- regression equation is expressed as Y (not Y or Ŷ)
tor” for the independent variable and “a criterion a bX.
variable” or an “outcome variable” for the depend- For statistical inferences about the regression
ent variable. equation (i.e., to verify that the relationship
To answer the question of whether one variable between X and Y did not occur by chance), one
predicts another, one can use a simple linear analyzes the variance of regression. The null
regression. If there are several independent vari- hypothesis is Ho: b 0. It means there is no slope
ables, then one needs to use multiple regression. In and the best bet to predict the Y score is based on
this section, in addition to these two linear regres- the mean of Y for any value of X.
sion analyses, logistic regression is also discussed.
Logistic regression is a type of regression tech- The Assumptions for Regression Analysis
nique in which the dependent variable is measured
by nominal scale. In addition, analysis of covari- In regression analysis, for any values of X, a ran-
ance (ANCOVA) is discussed. ANCOVA uses a dom distribution of Y scores exists. Therefore, the-
combination of ANOVA and regression. oretically, the mean of each distribution of Y lies
on the regression line. Each distribution is normal
Simple Linear Regression and its standard deviation is homogeneous. These
statistical assumptions are illustrated in Figure
Simple linear regression assumes that two vari- 17.3 on p. 266. This figure illustrates a theoretical
ables are linearly correlated and the line that best regression line, which theoretically contains a
describes the relationship is called a regression series of normal distributions running across it.
line. To determine this line, one uses a regression The mean of each normal distribution is shown to
analysis. The regression line is expressed by the fall directly onto the regression line.
regression equation: In regression analysis, these assumptions can
be examined by plotting residuals (the difference
Y a bX
between observed and predicted scores). Usually
where Y or Ŷ is the predicted value of Y, a is the the residuals are plotted on the Y-axis and the pre-
Y-intercept (the value of Y when X is 0), and b is dicted scores are on the X-axis. When the linear
the slope of the line (the rate of change in Y for regression model is a good fit, the residuals will be
each unit change in X). The variable a is the randomly dispersed around Y 0 with similar
regression constant and b is the regression coeffi- width above and below the horizontal line (Figure
cient. When b is positive, the Y line goes higher as 17.4 on p. 266).
X increases, indicating a positive correlation. If the plot of residuals deviates from this pat-
When b is negative, the Y line goes lower as X tern, a transformation of the data may be neces-
increases, indicating a negative correlation. If a is sary. Transformations use a constant to covert
3, b is 2, and X is 5, then Y 3 (2)(5) 13. the data to a more normal distribution. For trans-
The regression line is called the line of best fit, formation methods, read Tabachnick and Fidell
because it is the line that best describes (fits) the (1996).
pattern of the data. The errors determined by the In regression, outliers should be examined. If
differences between observed values and predicted one determines that the outlier is due to a data
values (Y – Y) are called residuals. The regression entry error, it can be eliminated. Some researchers
line is the line for which the residuals are smallest. consider the score an outlier if it is beyond three
The squared term of residuals is used to avoid the standard deviations. However, if it is a true score,
negative values; therefore, the smallest sum of one has to consider whether it should be discarded
squared residuals is the line of best fit. The method or not. Since outliers can greatly influence the
to identify this best fit line is called the method of results of a regression, they can distort the overall
least squares. picture of how well X predicts Y.
Figure 17.3 A statistical assump-

tion of linear regression: Normal
distribution of predicted scores
and their means being on the lin-
ear regression.
Accuracy of Prediction prediction of the regression equations and is calcu-

lated by:
To examine the accuracy of prediction (how well X
Σ(Y Y )2

predicts Y), one examines the coefficient of deter-
SEE
mination (r-square for simple linear regression and (n 2)
R-square for multiple regression and the standard
error of the estimate (SEE). The coefficient of where Σ (Y Y )2 is the sum of the square
determination is a measure of proportion: the residuals and n is the number of pairs of scores.
percentage of the total variance in the dependent
variable that can be explained by the independent Analysis of Variance for Regression
variable; therefore, indicating the accuracy of
There are three variance components in analysis of
prediction based on the independent variable. The
variance for regression:
correlation of r .30 means r2 .09; that is,
9% of the variances of Y is explained by the vari- • The sum of squared distance between the mean
ances of X. On the other hand, 1 – r2 is the ) and the observed scores (Y) are
of all Y scores (Y
unknown variances; therefore, the higher the val- total SS (SStotal),
ues of coefficient of determination, the more accu- • The sum of squared distance between the mean
rate of the prediction based on the independent of all Y scores (Y ) and the predicted scores (Y′)
variable. are explained by regression SS (SSreg), and
The standard deviation of the distribution • The sum of squared distance between observed
of errors is called the standard error of the esti- score (Y) and predicted value (Y′) is residual SS
mate. The SEE indicates the average error of (SSres).
Residuals
Figure 17.4 Residuals indicat-

ing the assumption of linear
Predicted Scores regression is being met.
Observed
value
Error
Total
Regression
Figure 17.5 Components of vari-

ances in regression.
These components are illustrated in Figure 17.5 In Table 17.13a, the R is the multiple correla-
and summarized in Table 17.12 (Analysis of tion coefficient (i.e., the correlation between all
Variance Table for Regression). independent variables altogether and the depend-
An F-distribution is used for testing the signif- ent variable). In simple linear regression analysis,
icance of the relationship between X and Y but not there is only one independent variable; therefore, R
the strength between the two variables. The inter- is the same as r, and r .290 in Table 17.13a. The
pretation of the strength relies on R2 (multiple cor- next column, R-square, indicates the shared vari-
relation) or r2 for a simple linear regression. ance and explains how much of the variance of the
The definitional formula to derive F value is as dependent variable is explained by that of the
follows: independent variables (8.4%). Table 17.13b shows
that F 35.506 and it is significant at p .001,
SS total Σ(Y Y
)2 indicating that the regression line is a good fit
(Appendix A, Tables D and E).
SS regression Σ(Y′ Y
)2 As shown in Table 17.13c, a t-test is applied to
SS residual SS total – SS regression find out whether b (the regression coefficient and
B in the SPSS printout) is significantly different
where Y is the observed value, Y is the mean of from 0 degrees (angle of the slope) or not. If it is
all Y values, and Y′ is the predicted Y value. close to 0, the regression line will be horizontal or
Degrees of freedom (df) for total is N – 1, df for parallel to the X-axis, indicating there is no corre-
regression is the number of independent variables lation between the two variables. Since t (–5.959)
( 1 for a linear regression), and df for residual is is significant, it can be interpreted that the value of
dftotal – dfregression. Mean Square (MS) is obtained Y can be predicted by value of X. In this case the
by dividing SS by df and the F-value is regression equation (Y a bX) is:
MSregression /MSresidual. Y 48.722 .268X.
Using the example of correlation between Age
and BMI in Table 17.11, a regression analysis was Since b (B in SPSS printout) is negative, it is
performed. The results of the SPSS output are indicated that as people become older, the BMI
shown in Tables 17.13 a, b, and c on p. 268. score will be less. This is true since the sample is
Table 17.12 Conceptual Definitions of Analysis of Variance Table for Regression

Sources of Variance SS df MS F-value
Regression Σ (Y )2
Y No. of IV SS/df MS reg./MS residual
Residual Σ(Y Y )2 (N 1)No. of IV SS/df
Total Σ(Y Y )2 N1
Table 17.13 Regression Analysis: SPSS Printout
(a) Model Summary

Model R R2 Adjusted R2 Std. Error of the Estimate
1 .290* .084 .082 7.10929
*Predictors: (Constant), age.
(b) ANOVA Table for Regression*

Model Sum of Squares df Mean Square F Sig.
1 Regression 1794.526 1 1794.526 35.506 .000**
Residual 19559.746 387 50.542
Total 21354.272 388
*Dependent variable: BMI.

**Predictors: (Constant), age.
(c) Coefficients*
Unstandardized Standardized
Model Coefficients Coefficients T Sig.
B Std. Error Beta
1 (Constant) 48.722 3.427 14.216 .000

dem1b .268 .045 .290 5.959 .000
*Dependent variable: BMI.
65 years old or over. For example, if a person is 70 The power can be obtained from the Appendix
years old, his/her BMI can be predicted as 48.722 A, Table K. For the detailed power analysis, con-
.268 70 29.962. sult Cohen’s (1998).
Power Analysis for Linear Regression Multiple Regression

For a simple regression analysis and multiple The true benefit of using regression is best realized
regression analysis, the effect size is calculated in use of multiple regression. The main purpose of
using f 2. The calculation formula for the effect multiple regression is to predict the value of a
size for the example above uses multiple correla- dependent variable based on several independent
tion and it is calculated by: variables. Other purposes of regression analysis
are to:
R2
f 2 2 • Identify the strength of association (R2) between
1R
the dependent variable and several independent
The interpretation of the effect size index is: variables together, and
• Small is f 2 .02, • Identify variables that significantly contribute to
• Medium is f 2 .15, and predicting the values of the dependent variable.
• Large is f 2 .35 In addition to the statistical assumptions for
To obtain the statistical power for regression simple linear regression, one more thing that has to
analyses, one more step is necessary, using λ be considered for multiple regression is muticolin-
(lambda). The lambda is calculated by: earity among independent variables in a regression
equation. Multicolinearity means that the inde-
R2 (the number of independent pendent variables in the regression equations are
λ 2 variables + the number of df highly correlated. If those variables have correla-
1R
for the error variance + 1) tion of .80 or higher, the one that has a lower
unique correlation with the dependent variable A)

should be removed from the equation.
There are three major methods in selecting IV1 Variances explained
variables for a multiple regression: the standard by IV1
DV
method, the hierarchical method, and the stepwise
method. Usually, one wants to find the small-
est group of variables that will account for the Variances explained
greatest proportion of variance in the dependent IV2 by IV2
variable.
Standard Method B)
In the standard method, all independent variables IV1 Variances explained

are entered into the regression equation at once. In by IV1
this instance, if two independent variables overlap DV
in accounting for the variance of the dependent
variable, as indicated by the unshaded variances in
Figure 17.6a, the effects of neither of these inde- Variances explained
IV2
by IV2
pendent variables are not accounted for. This is
illustrated in Figure 17.6a.
C)
Hierarchical Method
This method is best illustrated through an example. IV1 Variances explained
by IV1
Suppose one wants to know whether depression is DV
a predictor of assistive device use. In this case,
there are already some variables that are known to
affect assistive device use (i.e., fixed factors known Variances explained
IV2 by IV2
to influence the dependent variable), such as dis-
ability levels, age, socioeconomic status, and med-
ications taken. Therefore, in a hierarchical Figure 17.6 Regression methods. (A) Standard
regression, one wants to know whether depression method: Shared Variances by IV1 and IV2 are
makes a difference over and above these fixed fac- not accounted for by either one of the indepe-
tors. In such a case, one may enter the fixed factors ndent variables. (B) Hierarchical method: The
first (so they can explain the most variance of the investigator decided to enter IV1 into the reg-
dependent variable) and add depression last. The ression equation first before IV2, so the shared
variables may be entered one at a time or in sub- variance among the DV, IV1, and IV2 are
sets, but there should always be a theoretical explained by IV1. In this way, the investigator
tests whether IV2 can explain any additional
rationale for their order of entry. Within each sub- variance in the DV over and above the effects
set, variables may be entered in standard format as of IV1. (C) Stepwise method: The first variable
a group, or in a stepwise fashion (see the next sec- entered into the regression equation was
tion). Tomita, Mann, Fraas, and Stanton (2004) decided statistically (rather than theoretically)
used this approach in predicting the number of use because it had the highest correlation with the
of assistive devices to address physical disabilities dependent variable. Therefore, the variance
among home-based frail elders. After adjusting the that is shared by the DV, IV1, and IV2 is
effects for physical disability, number of medica- explained by IV2.
tions taken, race, living regions, living arrange-
ment, and education, depression was an important
and second variables. This is illustrated in Figure
predictor for assistive device use.
17.6b.
In hierarchical regression, the variance in the
dependent variable are considered to be explained
Stepwise Method
by the independent variable that was entered into
the regression equation prior to the other inde- The stepwise method is generally used to identify
pendent variables. Thus, when two variables over- important predictors when the field of study is still
lap in accounting for the variance of the dependent exploratory (i.e., an investigator is not testing a
variable, the first variable entered receives credit highly specific, theoretically grounded hypothe-
for predicting the variance shared between the first sis). Stepwise selection is a combination of for-
ward and backward procedures. With the stepwise correlation with the dependent variable when a
solution, variables are entered using a method stepwise method is used, and
called the forward solution. They are assessed at • The standardized beta weight (), .222 is larger
each step using the backward method to determine than the gender standardized beta, .183 (the sign
whether their contribution is still significant, given is ignored) for the cardiovascular diagnosis. In
the effect of other variables already in the equation the SPSS output is called beta but the regres-
(see Figure 17.6c). Dudek-Shriber (2004) used this sion coefficient (b) is also a beta weight.
approach to identify predictors among parent and
Absolute standardized beta weight (ignoring
infant characteristics for parent’s stress level in the
the positive or negative sign) is used to compare
neonatal intensive care unit. One of the dependent
the magnitude of contribution in accounting for the
variables was total stress. The most important pre-
variances of the dependent variable. It ranges 0 (no
dictor was infant cardiovascular diagnosis (b
contribution) to 1 (total contribution). The squared
.504) and the second predictor was gender of the
multiple correlation R2 indicates how much the
parent (b –.337), the variance accounted for
variance of the dependent variable is explained by
was 7.4% (R2 = .074), and corrected variance was
the variances of these two independent variables
6.3% (Adjusted R2 = .063). The Standard Error of
(i.e., cardiovascular diagnosis and gender of par-
Estimate was also reported (SEE = .709). In addi-
ent). R2 .074, which is 7.4%.
tion, the author reported the standardized beta ()
The adjusted multiple correlation represents R2
to identify which independent variables con-
that is corrected due to a chance occurrence.
tributed more to explaining the variable of the
Therefore, the adjusted R2 reflects a more accurate
dependent variable. The figures and their interpre-
figure for the population, and some researchers
tations are in Table 17.14.
prefer reporting the adjusted R2 to R2. The standard
In Table 17.14, there are two predictors in the
error of the estimate (SEE) indicates the accuracy
regression equation (presence or absence of car-
of the equation model for the prediction. A smaller
diovascular diagnosis, and gender of parent, such
SEE is indicative of higher accuracy.
as mother or father). The intercept (a) is 2.660, the
The stepwise solution is less popular today.
regression coefficient for cardiovascular diagnosis
Some of the reasons are: R- square values tend to
is .504, that for gender is –.337. The author coded
be inflated, regression coefficients for some vari-
1 for “Having cardiovascular diagnosis” and 0 for
ables are inflated, and therefore, the possibility of
“Not having the diagnosis,” and 1 for “Father” and
making a type I error expands dramatically with
0 for “Mother.”
increased numbers of independent variables. To
The equation model to predict total means
avoid this, some statisticians say that this method
stress occurrence is 2.660 .504 (cardiovascular
requires a larger sample size than hierarchical
diagnosis 1 or 0) – .337(0 mother or 1
methods. However, more serious problem is that p-
father). Therefore, the stress level that is found for
values sometimes do not have the proper meaning
the mother who has an infant with cardiovascular
and its correction is very difficult and increasing
diagnosis is 3.164. The presence of cardiovascular
the sample size does not correct this problem. The
diagnosis is a more important predictor than the
preferable solution may be combine stepwise and
gender of the parent because:
hierarchical approaches by entering a block of
• The former was entered into the model (regres- variables to the regression equation hierarchically
sion equation) first due to the highest bivariate but use a stepwise regression within each block.
Table 17.14 Final Regression Model to Predict Parent’s Total Means Stress Occurrence
The Most The Second Standard
Constant Contributing Variable Most Contributing Adjusted Error of
Source a b () variable b () R2 R2 Estimate
Total 2.660 Infant Sex .074 .063 .709
Means Cardiovascular .337
Stress Diagnosis (.183)
Occurrence .504 (.222)
Adapted from Table 7 in Dudek-Shriber, L. (2004). Parent stress in the neonatal intensive care unit and the
influence of parent and infant characteristics. The Journal of American Occupational Therapy, 58(5), p. 516, with
permission.
Analysis of Covariance In a study of the effects of assistive technology

and environmental interventions in maintaining
The main purpose of analysis of covariance independence and reducing home care costs for the
(ANCOVA) is to control for extraneous factors frail elderly Mann, Ottenbacher, Fraas, Tomita,
(i.e., independent variables that are not a part of and Granger (1999) used an ANCOVA to control
the theoretical model) that may influence the for the difference in baseline (pretest) scores on
dependent variable. By doing so, one hopes that the FIM motor scale between the treatment and
the variances of the dependent variable can be control groups. After the intervention (i.e., provid-
explained by that of the independent variables that ing necessary assistive devices and home modifi-
are of particular interest. ANCOVA is a combina- cations), the 18th month follow-up found that the
tion of ANOVA and linear regression. By remov- difference between the two groups was significant
ing the effect of the unwanted variables (called at p .01 (71.6 and 66.4, respectively). The
covariates) that are affecting the variability of the process of adjusting the covariate (pretest scores in
dependent variable prior to computing the group the example above), is illustrated in Figures 17.7a
means of the dependent variables, one can achieve and b using hypothetical data. For the treatment
a more valid explanation of the relationship bet- group, pretest FIM Motor score was 70.83, and for
ween the theoretically relevant dependent and the control group it was 70.83. For posttest, the
independent variables. treatment group was 81.25 and the control group
A)
= Post-Test Score
Y = 61.404 + .28 X
Treatment Group
80
81.25
80.94
75 Y = 18.651 + .81 X
Control Group
70
= Covariate
70 75 80 Pretest Score
70.83 76.88
B)
= Post-Test Score
Y = 61.734 + .275 X
Treatment Group
83.67
80
Figure 17.7 Process of Y = 17.708 + .822 X

ANCOVA. (A) Regression 79.23 Control Group
lines explaining the rela- 75
tionship between the intial
FIM motor score (covariate)
and the follow-up FIM
Motor Score (dependent 70
variable). (B) Adjusted
means for follow-up FIM
motor score, based on ini- = Covariate
tial FIM motor score as 70 75 80 Pretest Score
covariate. 74.38
was 80.94. The difference of posttest scores for the The probability that an individual belongs to
two groups is statistically not significant (p the target group uses the logit, and is computed by:
.904).
First, the regression lines for both groups ez
(where the X-axis is the covariate, pretest FIM Probability
1 ez
Motor score, and the Y-axis is the dependent vari-
able, FIM Motor scores posttest) are drawn (see where e is a log of 1 ( 2.718).
Figure 17.7a). The regression equation for the Therefore, the difference between multiple
treatment group is Y 61.404 .28X and that for regression and logistic regression is that the linear
the control group is Y 18.651 .81X. The mean portion of the equation (a b1X1 b2X2 b3X3….
score for X for the treatment group is 70.83 and bkXk) is not the end purpose, but is used to iden-
that for the control group is 76.88. To eliminate the tify the odds of belonging in one of categories.
difference, one statistically equated the two groups Logistic regression does not minimize the error
on the pretest scores, using the mean of the total by the least square method that is used for linear
sample (74.38). One assigned this value as the regressions. Instead, logistic regression estimates
mean pretest FIM Motor score for each group, and the maximum likelihood that results in the accu-
used the regression lines to predict the mean score rate prediction of group membership.
for the posttest FIM Motor score ( dependent This approach is illustrated in a study that
variable). By moving the treatment group up by aimed to predict home-based living (the target
3.55 ( 74.38 – 70.83) and moving the control group) or institutional living (the reference group)
group down by 2.50 ( 76.88 – 74.38), one artifi- within a year among 276 home-based frail elders,
cially made the initial scores equal. Then, the based on their Mini Mental State Exam (MMSE)
mean of the posttest (Y ) for the treatment group total score and Body Mass Index (BMI). Table
became 83.67 and that for the control group, 17.15a shows the classification table for commu-
79.23. These are called adjusted means. This nity living and 17.15b shows logistic regression
process is illustrated in Figure 17.7b. The adjusted results for selected important predictors.
mean score for the treatment group is higher than In Table 17.15a, step 1 is based only on the
the actual follow-up score of FIM Motor, and that MMSE total score. As seen in the table, this score
for the control group is lower than the actual fol- predicts 97.4% of home-based living and 40.5% of
low-up of FIM Motor, thus, making a larger differ- institutionalization accurately. The overall accu-
ence. The result is the significant difference racy is 88.8%. Step 2 is based on the MMSE total
between posttest scores of the two groups (p score and the BMI. Although the accuracy for
.007). This process may work in the opposite home-based living is the same, for the nursing
direction, resulting in no significant difference home, the classification accuracy increased
between the two groups. by 2.4%, resulting in 89.1% of overall accuracy.
The uniqueness of logistic regression involves
Logistic Regression the use of the odds ratio. The odds ratio indicates
how much more likely it is that an individual
In multiple regression, the dependent variable
belongs to the target group than the reference
should be measured using a continuous variable. In
group.
logistic regression, the dependent variable is a
If the odds ratio is 1, then it does not predict
dichotomous outcome, such as “Yes” for presence
classification at all. If the odds ratio is greater than
and “No” for absence of a particular condition.
1, the change of one unit in the independent vari-
The independent variable can be categorical or
able increases the likelihood of being classified in
continuous. Therefore, logistic regression allows
the target group than in the reference group. In an
us to predict the probability (i.e., likelihood of
SPSS printout, the odds ratio is designated as Exp
“Yes”) of an event occurring (target group) based
B. In Table 17.15b for MMSE, odds is .744, indi-
on several independent variables. For that one uses
cating if a person decreases one unit of MMSE (1
the logarithm of the odds:
of 30), the person is .744 times more likely to be
Z a b1X1 b2X2 b3X3…. bkXk living at home than being institutionalized.
Likewise, if a person exhibits one unit decrease in
where Z is the natural logarithm of the odds, BMI, then the person is .890 times more likely to
called a logit; a is a constant; and b is the regres- live at home rather than in an institution. When the
sion coefficient or a slope. (Note: this is the usual coding (0 and 1) is flipped for the dependent vari-
linear regression equation.) able, the same data would show that the odds for
Table 17.15 Logistic Regression: SPSS Printout
(a) Classification Table

Observed Predicted
NHHOME Percentage Correct
1.00 2.00
Step 1 NHHOME 1.00 17 25 40.5

Home Based 2.00 6 228 97.4
Overall percentage 88.8
Step 2 NHHOME 1.00 18 24 42.9
Home Based 2.00 6 228 97.4
Overall percentage 89.1
The cut value is .500
(b) Variables in the Equation

B SE Wald df Sig. Exp(B)
Step 1* MMSE
.292 .046 40.474 1 .000 .74 7
TOTAL
Constant 5.961 1.217 24.001 1 .000 387.936
Step 2** BMI .117 .040 8.708 1 .003 .890
MMSE
.296 .048 37.304 1 .000 .744
TOTAL
Constant 9.299 1.788 27.059 1 .000 10931.603
*Variable(s) entered on step 1: MMSE TOTAL.

**Variable(s) entered on step 2: BMI.
the MMSE is 1.344 and that for the BMI is 1.124. more popular than discriminant analysis where the
They should be interpreted that when the MMSE statistical assumptions are more rigid than for
increases one unit, the person is 1.344 times more logistic regression.
likely to be living at home than being institutional-
ized and when the BIM increases one unit, they are
1.124 times more likely to be living at home than Multivariate Analysis
being institutionalized. Among frail elders, a lower
BMI is a predictor for institutionalization. This Multivariate analysis is conducted when there is
study used a binary logistic regression. However, more than one dependent variable in the analysis.
logistic regression can also be used when the In this section, factor analysis, path analysis, and
dependent variable has more than two categories multivariate analysis of variance (MANOVA) are
(i.e., multinominal categories). This is referred to explained.
as multinomial logistic regression. Many resear-
chers prefer binary or dichotomous outcomes
Factor Analysis
because the interpretation using odds ratios is
clearer than in multinominal logistic regression. The purpose of factor analysis is to find structure
Since logistic regression can be used for a among data by reducing it. It is a statistical tech-
multinominal function, this approach is becoming nique applied to a single group of variables to
determine which combinations of variables in the in a theoretical solution uncontaminated by unique

group form unique and coherent subsets that are and error variability, FA is the choice, while one
relatively independent from each other. Factor wants an empirical summary of the data set, PCA
analysis is often used to establish construct valid- is the choice.
ity of an instrument, meaning one creates or vali- In this chapter, FA is explained, using the
dates an instrument to describe its construct, such Functional Index Measure (FIM) as an example.
as depression, self esteem, quality of life, and so Tables 17.16 a, b, and c show some of the steps
forth, by identifying underlying different concepts involved in factor analysis. In this example, FIM 1
or factors. Factor analysis does not compare group is self-care, FIM 2 is sphincter control, FIM 3 is
means, such as ANOVA, identify the correlation transfer, FIM 4 is locomotion, FIM 5 is communi-
between two variables, such as Pearson correla- cation, and FIM 6 is social cognition.
tion, or predict values of the dependent variable The first step in a factor analysis is based on the
based on several independent variables, such as observed correlation matrix, meaning the matrix
multiple regression or logistic regression. Instead, created by observed variables. This correlation
this analysis produces factors that consist of a matrix was explained in the earlier section in this
grouping of variables that are highly correlated chapter. Then, the correlation matrix is used to
with each other but are poorly correlated with identify the factors of the data by the process
other factors. There are two types of factor analy- called extraction. In the beginning, the number of
sis: exploratory and confirmatory. In exploratory factor extracted is the same as the number of vari-
factor analysis that is this section’s focus, one ables. An eigenvalue reflects the amount of the
seeks to describe and summarize data by grouping total variance accounted for by each factor. Then,
together variables that are correlated, and it is the number of factors are reduced by using the
usually performed in the early stage of research. eigenvalue cutoff that is conventionally 1.0. The
The results are often used to generate hypotheses first factor in the SPSS printout in Table 17.16a
about underlying processes. On the other hand, accounts for the largest variance in the data
confirmatory factor analysis is a much more (61.754%).The second factor accounts for the sec-
sophisticated technique, and is used in the advan- ond largest variance from the remaining variance
ced stages of research to test a theory about latent (17.037%). The factor analysis presented in Table
processes. 17.16a shows only two factors, and these explain
There are two major different methods for fac- 78.791% of the total variance. The factor matrix in
tor analysis: principal component analysis (PCA) Table 17.16b shows the factor loadings for each of
and factor analysis (FA). These two approaches the six FIM variables and their correlations with
share major procedures, and even problems. For each of the two identified factors (i.e., factors 1
example, both PCA and FA take steps in the fol- and 2). Factor loadings greater than .30 are gener-
lowing order: ally considered to be indicative of some degree of
relationship. Two of the factor loadings for the
1. Select and measure a set of variables,
FIM scores for both of the factors are above .30,
2. Prepare the correlation matrix,
and it is not clear as to which factors some vari-
3. Extract a set of factors from the correlation
ables such as FIM 5 and 6 belong to.
matrix,
The next step is to maximize the likelihood that
4. Determine the number of factors, rotating the
each variable relates highly to only one factor (i.e.,
factors to increase interpretability, and
that the FIM variables that show factor loadings of
5. Interpret the results.
.30 or above for factor 1 will be different than the
The problems are discussed later in this section. FIM variables that show factor loadings of .30 or
The difference between PCA and FA is that PCA above for factor 2). This is accomplished by rotat-
produces components and FA produces factors. In ing the X- and Y-axes. Although there are several
PCA, all variance in the observed variable is ana- ways to rotate, the most commonly used method is
lyzed, while in FA, only shared variance, called varimax rotation. A varimax rotation means that
covariance or communality, is analyzed. The goal the two axes stay as perpendicular to each other as
of PCA is to extract maximum variance from a possible (orthogonal) as they are rotated. The
data set with a few independent components, while results of this rotation are shown in Table 17.16c.
the goal of FA is to reproduce the correlation Following rotation, it is now much clearer that
matrix with independent factors. Very often, both FIM variables 1 to 4 belong to factor 1 and vari-
components and factors are referred as factors. The ables FIM 5 and 6 belong to factor 2. As evident in
choice between PCA and FA depends on many table 16c, FIM variables 1 and 2 also have factor
things, but as a rule of thumb, if one is interested loadings above .30 in factor 2. However, because
Table 17.16 Factor Analysis: SPSS Printout
(a) Total Variance Explained:

Extraction Sums of Rotation Sums of
Factor Initial Eigenvalues Squared Loadings Squared Loadings
% of Cumulative % of Cumulative % of Cumulative
Total Variance % Total Variance % Total Variance %
1 3.906 65.093 65.093 3.705 61.754 61.754 2.653 44.216 44.216

2 1.166 19.427 84.520 1.022 17.037 78.791 2.074 34.574 78.791
3 .461 7.689 92.209
4 .245 4.088 96.297
5 .135 2.243 98.539
6 .088 1.461 100.000
Extraction method: Principal Axis Factoring.
(b) Factor Matrix*

Factor
1 2
FIM1 .904 .199

FIM2 .698 .073
FIM3 .843 .397
FIM4 .709 .429
FIM5 .779 .557
FIM6 .762 .572
Extraction method: Principal Axis Factoring.

*Two factors extracted. Eight iterations required.
(c) Rotated Factor Matrix*

Factor
1 2
FIM1 .829 .412

FIM2 .590 .380
FIM3 .906 .219
FIM4 .821 .110
FIM5 .259 .922
FIM6 .236 .923
Extraction method: Principal Axis Factoring. Rotation method: Varimax with Kaiser Normalization.
*Rotation converged in three iterations.
the factor loadings for FIM 1 and 2 are higher in factor 2). This judgment is made by placing the
factor 1, the investigator decided that FIM vari- variable in the factor for which it has the higher of
ables 1 and 2 should remain a part of factor 1. In the two factor loadings. In our example (Table
cases like these when the same variable has a fac- 17.16c), the factor loading for FIM 1 and 2 are
tor loading above .30 for more than one factor, an higher in factor 1 (.829 and .590, respectively)
investigator must make a judgment call regarding than in factor 2 (.412 and .380, respectively) so the
where to place the variable (e.g., factor 1 versus variables are placed in factor 1.
Once an investigator has identified factors There are a number of theoretical and statistical
within the group of variables being examined, the assumptions that need to be considered in a path
investigator usually gives the factors unique names analysis. Theoretically, the dependent variable and
or labels. These labels should reflect the unique the independent variable should be correlated, the
characteristics of the cohesive variables that com- dependent variable should occur later than the
prise the factor or component. Using our example, independent variable, and the relationship between
one sees that the FIM variables that comprise the the two variables exists even with the presence of
first factor (component 1) include FIM 1 to 4. another variable. The statistical assumptions are in
Since all of these share motor characteristics, this addition to those of regression analysis, so the flow
factor was named “FIM Motor.” FIM 5 and 6 com- of causation in the model is unidirectional and the
prise factor 2 and it was named “FIM Cognition.” measurement error is zero.
As to the required sample size, there are various In a path analysis, there are direct and indirect
opinions. Munro (2001) states that a ratio of at effects. The direct effects are drawn using straight
least 10 subjects for each variable is desirable to lines connecting two variables and indicating the
generalize from the sample to the population. cause and effect by an arrow head. Indirect effects
Knapp and Brown (1995) argue that only three are indicated by lines not connecting two variables.
subjects are required for each variable. However, All variables are either endogenous or exogenous.
this estimate is on the low side and likely to Endogenous variables are ones that are being influ-
increase the chances of type 1 error, particularly in enced by other variables in the model. One needs
a factor analysis in which issues of multicollinear- to apply a regression analysis for every endoge-
ity are involved. In terms of total sample size, nous variable. Exogenous variables are ones that
Tabachnic and Fidell (1996) say at least 300 sub- are independent of any influence.
jects are required to perform a factor analysis. In the study titled, “After rehabilitation: an 18-
When there are high loadings such as .80, only 150 month follow-up of elderly inner-city women”
cases may be required, according to Guadagnoli (Lysack, Neufeld, Mast, MacNeill, & Lichtenberg,
and Velicer (1988). 2003), the authors presented a path diagram. The
Since factor analysis can be subjective and modified path diagram is presented in Figure 17.8.
judgmental owing to many choices of extraction In the diagram, there are two endogenous vari-
methods and rotation methods, one should consult ables (IADL and living status) and three exoge-
an experienced statistician to perform a factor nous variables (dementia, comorbidity, and ADL).
analysis. Factor analysis and principal components Their sample consisted of 125 older women who
analysis should not be confused with confirmatory were living alone prior to their hospitalization, and
factor analysis, which is an application of struc- who were then followed up 18 months after dis-
tural equation modeling (SEM). Confirmatory fac- charge. The path coefficients, seen adjacent to the
tor analysis is used when there is a theory about lines connecting the four figures, are the same as
factors. Further explanation of confirmatory factor the standardized regression coefficients (beta
analysis is beyond the scope of this chapter. For a weights) described in regression analyses. Based
more detailed discussion, readers are referred to on the model, the dementia is the most influential
Tabachnick and Fidell (1996). variable for IADL. ADL has a positive relationship
with IADLs, comorbidity has a negative relation-
ship with IADLs, and IADLs have a negative rela-
Path Analysis
tionship with the status of living alone (0 living
Path analysis is a procedure that is associated with alone and 1 not living alone). A higher level of
multiple regression. It is a diagram-based statistical IADL is indicative of being able to live alone. In
approach used to test the strength and direction of this diagram, all paths are direct paths, and it is an
relationships among several variables, and it overidentified model in that there are more meas-
enables an investigator to identify direct and indi- ured variables than the number of paths. If the
rect causal pathways among the variables. To use a numbers of variables and paths are the same, it is
path analysis, all variables should be measured on called an identified model, and if the number of
a continuous (i.e., interval or ratio) scale. For the paths is more than that of variables, it is an under-
sample size, it is recommended that there are 30 identified model.
subjects per independent variable to be general- Originally, direct paths from all four exogenous
ized. Path analysis can be used with cross-sectional variables (depression was also integrated) to IADL
or longitudinal study designs, but because it tests were included. However, the path from depression
causal relationships, it is best used for longitudinal to IADL was statistically not significant; therefore,
designs. it was removed after the analysis. Until recently,
Dementia
.40
-.23 Not living

Figure 17.8 Path diagram for liv- IADL alone after
Comorbidity -.59 discharge
ing alone status 18 months
after discharge. (Adapted from
Lysack, Neufeld, Mast,
MacNeill & Lichtenberg, 2003. .23
The American Journal of
Occupational Therapy, 57(3), ADL
p. 303, with permission.)
investigators used regression analysis and manual ables (number of illnesses and physical disability
calculations for path models, but currently a num- level), and the Pearson correlation between the two
ber of software packages are available, such as dependent variables is r .326. Vectors for home-
AMOS (produced by the manufacturers of SPSS). based elders are written as V 1 (6.2, 27.6) and
For a more complete understanding of path analy- that for nursing home residents are V 2 (5.8,
sis, readers are referred to Munro (2001). 39.5). The vector values are shown in Table 17.17a
on p. 278 in the column labeled “Mean.” When the
Multivariate Analysis of Variance two groups are drawn, there are center points in
these groups that are called group centroids. They
Multivariate analysis of variance (MANOVA) is
represent the intersection of the means for both
similar to an ANOVA, but unlike an ANOVA,
dependent variables that are the spatial location of
which is designed to manage only one dependent
the mean vector (Figure 17.9). The goal of the
variable, MANOVA allows an investigator to
MANOVA is to determine if there is a significant
examine outcomes for more than one dependent
difference among the group centroids. In
variable. MANOVA is used when there are several
MANOVA, the total variance in the sample is
dependent variables that are theoretically and sta-
divided into parts that represent between-groups
tistically correlated. When comparing group
and error, variability is measured against centroids
means, one considers the overall effect of the inde-
while in ANOVA, variability is measured against
pendent variable on all dependent variables
group means.
). If a score for
together; this is called a vector (V
The null hypothesis for a MANOVA is:
one dependent variable changes based on an inde-
pendent variable (e.g., an intervention), then the 1
H0: V V 2 …….V
k
scores of the other dependent variables should also
change. Suppose, for example, that there are two Table 17.17b shows multivariate test results.
groups (elders who are living at home and elders Since there are two groups, Hotelling’s Trace (T2)
living in nursing homes) and two dependent vari- should be used. Other statistics such as Pillai’s
Home-based elders
7
Number of Illnesses
Nursing home resident
3
Figure 17.9 Representation of
physical disability level and num-
ber of illnesses in nursing home
20 25 30 35 40 45
residents and home-based elders
for logistic regression. Physical Disability Level
Table 17.17 MANOVA: SPSS Printout

Living Status Mean Std. Deviation N
Number of illnesses Home-based 6.2051 2.77338 234
Nursing home 5.8333 3.39236 42
Total 6.1486 2.87238 276
Physical disability level
Home-based 27.6130 14.02267 234
Nursing home 39.4981 16.72233 42
Total 29.4216 15.05276 276
(b) Multivariate Tests*

Effect Value F Hypothesis df Error df Sig.
Intercept Pillai’s Trace .790 513.568** 2.000 273.000 .000
Wilks’ Lambda .210 513.568** 2.000 273.000 .000
Hotelling’s Trace 3.762 513.568** 2.000 273.000 .000
Roy’s Largest Root 3.762 513.568** 2.000 273.000 .000
Living status Pillai’s Trace .102 15.568** 2.000 273.000 .000
Wilks’ Lambda .898 15.568** 2.000 273.000 .000
Hotelling’s Trace .114 15.568** 2.000 273.000 .000
Roy’s Largest Root .114 15.568** 2.000 273.000 .000
*Design: Intercept FINAL.

**Exact statistic.
(c) Tests of Between-Subjects Effects

Dependent Type III Sum Mean
Source Variable of Squares df Square F Sig.
Corrected model Number of illnesses 4.922* 1 4.922 .596 .441
Physical disability 5029.931** 1 5029.931 24.060 .000
Intercept Number of illnesses 5160.574 1 5160.574 624.561 .000
Physical disability 160377.932 1 160377.932 767.156 .000
Living status Number of illnesses 4.922 1 4.922 .596 .441
Physical disability 5029.931 1 5029.931 24.060 .000
Error Number of illnesses 2263.987 274 8.263
Physical disability 57281.093 274 209.055
Total Number of illnesses 12703.000 276
Physical disability 301224.961 276
Corrected Total Number of illnesses 2268.909 275
Physical disability 62311.024 275
*R2 .002 (Adjusted R2 .001).

**R2 .081 (Adjusted R2 .077).
278
Trace, Wilk’s Lambda, and Roy’s Largest Root ables. For more information on this topic, readers
should be used when more than two groups are are referred to Tabachnick and Fidell (1996).
compared. Since it is significant (p .001), the
two vectors are significantly different. However,
when two dependent variables are explained sepa-
rately, it produces a slightly different result.
Conclusion
When the MANOVA demonstrates a significant
This chapter discussed major inferential statistics
effect, follow-up analyses are usually based on
that are frequently used in occupational therapy
either univariate ANOVA or discriminant analysis.
research. The basic logic of each of the statistical
The latter may be preferable because it is still a
approaches was discussed and illustrated. While
multivariate analysis. The univariate analyses are
this discussion should serve as an introduction to
produced by SPSS MANOVA automatically and
understanding these statistics, there are often many
presented in Table 17.17c. The number of illnesses
subtleties involved in their use. Thus, investigators
is not significantly different for the two groups,
are referred to more detailed treatments of the
but the physical disability levels are different.
sophisticated statistics.
Nursing home residents are considerably more
disabled.
Some researchers use MANOVA instead of REFERENCES
repeated measures ANOVA with one between- Case-Smith, J. (2002). Effectiveness of school-based occu-
factor (mixed design) because a repeated measures pational therapy intervention on handwriting. The
American Journal of Occupational Therapy, 56(1),
ANOVA has more rigid statistical assumptions
17–33.
than a MANOVA. If the measurements are done Cohen, J. (1988). Statistical power analysis for the behav-
three times (pre-, post-, and follow-up), one can ioral sciences (2nd ed). Hillsdale, NJ: Lawrence
conduct a repeated measure ANOVA with three Erlbaum.
levels and one-between factor, or one can use a Dudek-Shriber, L. (2004). Parent stress in the neonatal
intensive care unit and the influence of parent and
MANOVA with three dependent variables with infant characteristics. The American Journal of
one-between factor. However, many researchers Occupational Therapy, 58 (5), 509–520.
use a repeated-measures ANOVA because it is Dunn, W., Smith-Myles, B., & Orr, S. (2002). Sensory
more conservative and interpretable. For a processing issues associated with Asperger syndrome:
A preliminary investigation. The American Journal of
MANOVA, a large sample size is required. A sam-
Occupational Therapy, 56 (1), 97–102,
ple size of 20 in the smallest cell in a univariate Guadagnoli, E., & Velicer, W. F. (1988). Relation of sam-
case may be adequate. ple size to the stability of component patters. Psycho-
In a study titled “Sensory processing issues logical Bulletin, 103, 265–275.
associated with Asperger syndrome: A preliminary Haberman, S. J. (1984). The analysis of residuals in cross-
classified tables. Biometrics, 29, 205–220.
investigation” (Dunn, Smith-Myles, & Orr, 2002), Knapp, T. R. & Brown, J. K. (1995). Ten measurement
42 children with Asperger syndrome and 42 commandments that often should be broken. Research
children without the syndrome were compared on in Nursing & Health, 18, 465–469.
both section and factor scores of the instrument, Lysack, C., Neufeld, W., Mast, B., MacNeill, S., &
Lichtenberg, P. (1993). After rehabilitation: An 18-
called Sensory Profile, using MANOVA. In the
month follow-up of elderly inner-city women. The
MANOVA results tables, the authors reported F- Journal of American Occupational Therapy, 57(3),
value, p-value, effect size (2), and power. Among 298–306.
14 sections, 13 were statistically significant and all Mann, W. C., Ottenbacher, K., Fraas, L., Tomita, M., &
9 factors were also significant, indicating that chil- Granger, C. (1999). Effectiveness of assistive technol-
ogy and environmental interventions in maintaining
dren with Asperger’s syndrome have clearly differ- independence and reducing home care costs for the
ent sensory processing patterns from peers without frail elderly. The Archives of Family Medicine, 8,
the syndrome. 210–217.
Finally, when one wants to control for interven- Munro, B. H. (2001). Statistical methods for health care
research (4th ed.). Philadelphia: Lippincott Williams &
ing variables, a MANCOVA will be used. A MAN-
Wilkins.
COVA is similar to ANCOVA in that it combines Parush, S., Winokur, M., Goldstand, S., & Miller, L. J.
MANOVA and regression analysis, and that treats (2002). Prediction of school performance using the
one or more of the independent variables as covari- Miller Assessment for Preschoolers (MAP): Validity
ates in the analysis. Simply stated, a MANCOVA study. The American Journal of Occupational Therapy,
56(5), 547–555.
allows an investigator to consider the effects of Siegel, S., & Castellan, N. J. (1988). Nonparametric statis-
multiple independent and dependent variables in a tics for behavioral science (2nd ed.). New York:
single analysis, controlling for intervening vari- McGraw-Hill.
Snedecor, G. W., & Cochran, W. G. (1991). Statistical RESOURCES

methods (8th ed.). Ames, A: Iowa Sate University Press.
Stratford, P. W., Binkley, J. M., & Riddle, D. L. (1996). Web site that does statistical calculations including power
Heath status measures: analyses, visit http://members.aol.com/johnp71/javas-
Strategies and analytic methods for assessing change tat.html
scores. Physical Therapy, 76, 1109–1123. For history of F-test, go to World of Visual Statistics
Tabachnick, B., & Fidell, L. (1996). Using multivariate (2004). Retrospect and prospectus. http://www.visual-
statistics (3rd ed.). New York: HarperCollins. statistics.net/
Tomita, M. R., Mann, W. C., Fraas, L., & Stanton, K. For the list of publication of effect size and power analy-
(2004). Predictors of the use of assistive devices that ses, go to http://www.kuleuven.ac.be/psystat/power-
address physical impairment among community-based training/References.pdf
frail elders. Journal of Applied Gerontology, 23(2), For review of statistical power analysis software, go to
141–155. http://www.zoology.ubc.ca/~krebs/power.html
C H A P T E R 1 8
Meta-Analysis
Kenneth J. Ottenbacher • Patricia Heyn • Beatriz C. Abreu
The dramatic expansion of research in health care meaningful, or if the outcomes measured are not
over the past 20 years is well documented (Institute sufficiently similar, or if the studies reviewed are
of Medicine, 2001). Primary studies published in qualitative in nature. If the results of the individual
the healthcare research literature routinely recom- studies are statistically combined to produce an
mend further investigation of the topic so that the overall quantitative outcome, this is usually called
findings can be corroborated. These calls for addi- a meta-analysis. The remainder of this chapter
tional research are based on the belief that scien- describes the procedures used to conduct a meta-
tific knowledge should be cumulative. Ideally, analysis. These same procedures, with the excep-
cumulative scientific findings lead to valid knowl- tion of computing a common statistical metric
edge that can be integrated into practice to improve (effect size), are used to conduct a systematic
health-related outcomes. In the current climate of review. Therefore, they can be taken as relevant to
accountability and evidence-based practice, this the review of qualitative research publications.
cumulative approach to determining scientific There are several systems for defining the
knowledge is extremely powerful (Abreu, 2003). strongest or best evidence for integrating research
Evidence-based practice emerged in the 1990s findings with clinical practice. Some disciplines,
and continues to provide a strong incentive for such as nursing, have developed variations of the
increased research designed to refine and guide levels of evidence hierarchy currently used by the
practice in clinical fields including occupational Center for Evidence-Based Medicine. The vast
therapy (Evidence-Based Medicine Working majority of the different “levels of evidence”
Group, 1992; Tickle-Degnen, 1998). Chapter 41 of frameworks used in healthcare research rate meta-
this text provides a detailed description of the his- analyses or systematic reviews as providing the
tory and methods of evidence-based practice. The best or highest level of research evidence.
focus of evidence-based practice over the past 10 Why are the results from meta-analyses studies
years has been to develop strategies to translate considered the highest level of evidence in making
research findings into information that can be used evidence-based decisions to improve clinical prac-
to justify and improve clinical decision-making tice? Traditional quantitative studies are based on
(Sackett, Straus, Richardson, Rosenberg, & a model of hypothesis testing using statistical tests.
Haynes, 2000). A sophisticated system of evaluat- The results of these tests provide estimates of the
ing quantitative studies has been established to probability that a research hypothesis is valid and
identify those investigations that provide the best should be supported. Whenever a researcher con-
evidence for determining treatment effectiveness. ducts a statistical test and makes a decision about
The system is often referred to as “levels of evi- whether to accept or reject the null hypothesis,
dence,” and provides a hierarchy of research there is a chance of making an error. In a typical
designs and grades of evidence. Figure 18.1 research study, the investigator may report a statis-
includes an overview of the levels of evidence hier- tical value followed by p ⬍ .05. This means the
archy currently used by the Center for Evidence- researcher has found a statistically significant result
Based Medicine (Sackett et al., 2000; University and will reject the null hypothesis and accept the
Health Network, 2004). Inspection of the figure research hypothesis (i.e., that there is a difference
indicates that the strongest level of evidence is a between the groups or conditions compared). In
systematic review or meta-analysis of randomized making this decision, the researcher is usually cor-
trials. The terms meta-analysis and systematic rect, but there is also a probability of being wrong.
review are often used interchangeably, but there is The probability of being wrong in this case is 5%
an important difference. A systematic review does (p ⬍ .05). The fact that even the best controlled and
not need to contain a statistical synthesis of the most rigorously conducted study may come to an
results from the included studies. This might be incorrect conclusion (type I or type II error) is why
impossible if the designs of the studies are too dif- replication and corroboration of research findings
ferent for a statistical average of their results to be are so important.
Evidence-based practice research

design rating system
1
Levels I Grades
2
II
3
III
IV
4
V
5
Levels of Evidence Based on Research Design
a. Systematic review (meta-analyses) of randomized trials
with narrow confidence intervals
Level 1*
b. Individual randomized trial with narrow confidence
intervals
a. Systematic review (meta-analyses) of homogenous
cohort studies
Level 2
b. Individual cohort studies and low-quality randomized trials
(e.g. trials with <80% follow-up)
a. Systematic review (meta-analyses) of homogenous
Level 3 case-control studies
b. Individual case-control studies
Level 4 Case series and poor quality cohort and case-control studies
Level 5 Expert opinion without explicit critical appraisal, or based in

physiological or bench research.
*Developed by Centers for Evidence-Based Medicine (http://www.cebm.net/). Not all
grades are shown for each Level of Evidence.
Figure 18.1 A description of levels of evidence hierarchy used in

evidence-based clinical medicine.
The details of statistical hypothesis testing are duces the highest level or best evidence in making
not the topic of this chapter, but are discussed else- evidence-based decisions.
where (see Chapter 17). The important point is that The capacity to synthesize multiple research
evidence from one study is not sufficient to draw studies is at the heart of the evidence-based prac-
conclusions about the outcomes of a health inter- tice movement and philosophy. Narrative reviews
vention. To determine of aggregated investiga-
the effectiveness of a tions have traditionally
treatment, or the valid- To determine the effective- been used to synthesize
ity of a scientific find- information from individ-
ing, multiple studies are ness of a treatment, or the ual quantitative research
required that address the validity of a scientific finding, studies. These narrative
same research question reviews have been shown
and produce consistent multiple studies are required to be biased, subjective,
findings. Meta-analysis which address the same and often lead to conflict-
is a systematic method ing conclusions (Glass,
of combining the results
research question and pro- McGaw, & Smith, 1981;
of multiple individual duce consistent findings. Whitehead, 2002). For
research studies to example, two narrative
answer a scientific ques- reviews were published
tion. The ability to obtain a statistical consensus in Neurology in 1989 examining the effectiveness
across many studies is why meta-analysis proof stroke rehabilitation programs. One paper was
Chapter 18 Meta-Analysis 283
Contrasting Approaches to Summarizing

Information from Individual Research Studies
Study 1 Study 2 Study 3 Study 4

p>.05 with p>.05 with p<.05 with p<.05 with
small sample medium sample medium sample large sample
Narrative Literature Review: Meta-Analysis Review:
Review based on informal Review based on systematically

process in which studies are defined criteria determined prior
selected and examined in an to conducting the meta-analysis.
undefined way and findings are Outcomes are converted to
frequently integrated based on common metric (effect size) that
comparing tests of statistical allows comparison across similar
significance. The result is often studies. The result is consensus
confusion across multiple studies. based on integrated information.
Figure 18.2 Schematic diagram of meta-analysis process including

comparison of narrative review and meta-analysis methods.
titled “Focused stroke rehabilitation programs Cooper (1989) argues that integrating separate
improve outcomes” (Reding & McDowell, 1989) research projects involves scientific inference. He
and the second article was titled “Focused stroke has conceptualized meta-analysis as a unique type
rehabilitation programs do not improve outcomes” of research endeavor with a series of distinct steps.
(Dobkin, 1989). Both papers were written by In the remainder of this chapter we present:
respected researchers and appeared in a prestigious
journal. From the titles there was a clear difference • Steps involved in conducting a meta-analysis,
in the conclusions. These two narrative reviews • An example of a meta-analysis relevant to occu-
generated confusion rather than consensus. pational therapy, and
The subjective and biased outcomes of narra- • Advantages and disadvantages of meta-analysis
tive reviews led to the development of methodolo- as a research method in occupational therapy.
gies and techniques in the 1970s to systematically
integrate findings from many individual research
studies (Glass et al., 1981; Rosenthal, 1978). The Steps in Meta-Analysis
procedures associated with meta-analysis are
designed to treat the review process as a unique Meta-analysis typically follows the same steps
type of research endeavor that produces an objec- used in a primary research design. The investigator
tive quantitative synthesis of research results first defines the research question or problem, the
(Cooper & Hedges, 1994; Lipsey & Wilson, 2001). inclusion criteria, the selection method, and the
The procedures provide a mechanism for investi- data collection process. Sample selection in meta-
gating variation in study characteristics such as analysis consists of applying procedures for locat-
sampling, design procedures, and type and number ing studies that meet the specified criteria for
of dependent and independent variables (Cooper, inclusion. Data are collected from studies in two
1998). Variance in these variables is then related to ways:
study outcome. This type of comparison is not pos-
sible in traditional literature reviews based on nar- • Major study features are coded according to the
rative descriptions of published studies. Figure objectives of the review, and
18.2 presents a comparison of the characteristics • Study outcomes are transformed to a common
of narrative reviews of the research literature and metric called an effect size (ES), so that they can
meta-analysis. be compared.
Table 18.1 Steps Involved in Planning, Conducting, and Reporting a Meta-Analysis

Review of Published Research
Steps Description
1. Problem formulation A topic that can be addressed by meta-analysis is selected followed by a
formulation of a research question relevant to the topic of interest. It
includes identification of the problem and formation of the research
questions.
2. Data collection A comprehensive, sensitive, and extensive search strategy is developed
to compile possible reports. Key words and variables determine
sources of potentially relevant reports. The inclusion criteria are
defined and eligibility criteria are set for the meta-analysis. Construct
definitions to distinguish relevant from irrelevant studies.
3. Data evaluation and coding Includes three areas:
1. Study Quality Assessment: Assessment of the methodological quality
and validity of the studies.
2. Data Identification and Quantification: Outcome variables are
identified and extracted into a coding system. The group’s contrasts
and effect sizes calculations are performed.
3. Characteristics of Interest: General information regarding study/trial
design should be standardized into a coding system for further
analysis, such as treatments and sample characteristics.
4. Analysis and interpretation It includes two areas:
1. Statistical Procedures: Compiling the data for quantitative synthesis
and summary effect sizes by using appropriate methods and
effects models (random or fixed) should be clearly stated. Effect
models should explore the sources of variation if variability i
present (e.g., differences in study quality, participants, treatments,
or outcomes).
2. Interpretation of Results: Translating the results with caution and
reanalyzing the data due to the robustness or uncertainty of results is
desired. Results could be uncertain due to imperfections in original
study reports or missing data.
5. Reporting the results Key aspects of all of the above stages should be clearly stated in the final
report in order to allow replication and critical appraisal of the meta-
analysis. The process of reporting should be rigorous and explicit.
Methodological limitations of both original studies and the meta-
analysis should be highlighted. Implications for future research should
be included.
Effect size is the name given to a family of ied. In its most basic form, the research problem
indices that measure the magnitude of a treatment includes definitions of relevant variables and a
effect. Unlike significance tests, these indices are rationale for relating the variables to one another.
independent of sample size. They are calculated by Two levels of definition are commonly associated
taking the difference between control and experi- with variables included in the problem formation
mental group means and dividing that difference step; these levels are conceptual and operational.
by the standard deviation of the scores of both The independent variable or treatment must
groups combined. It is also called delta or d. first be defined conceptually. Both primary
The procedures necessary to conduct a meta- researchers and research reviewers must choose a
analysis study are summarized in Table 18.1. conceptual definition and a degree of abstractness
for variables contained in the problem. The con-
ceptual definitions employed by reviewers using
Problem Formation quantitative methods tend to be rather broad to
The first task in any research endeavor is to iden- include all relevant instances of a particular con-
tify the focus or formulate the problem to be stud- struct. For instance, in the example described ear-
lier the problem question might be stated as: “Do relevant to the research problem or question. As
rehabilitation programs improve functional out- discussed in Chapter 27, the most popular and effi-
come for persons who have had a stroke?” cient method is the online computer or Internet
After the variables have been identified concep- search. The computer search is a time-saving tech-
tually, they must be linked to empirical reality. This nique that allows the reviewer to exhaustively scan
is accomplished by the formulation of an opera- several retrieval sources at a rapid rate. The fact
tional definition that relates the concept under that different electronic resources contain different
study to concrete events. The operational definition journals and document indicators or descriptors is
allows the investigator to determine whether a con- a limitation of online databases. Bibliographic
cept is present in a particular situation. searches using the same terms as descriptors or
Primary research usually involves only one or keywords may produce different results depending
two operational definitions of the same construct. on the database searched.
In contrast, meta-analysis reviews may involve A related retrieval approach is to employ a
many empirical versions of a concept. For instance, descendency search using the Science or Social
in a meta-analysis of sensory integration, the vari- Science Citation Indexes (Web of Science).
able “sensory integration therapy” may be opera- Because the citation indexes are primarily organ-
tionalized in a number of ways across the studies ized by author (not topic), they are most useful
that are reviewed. when particular investigators or research papers
As a consequence of the variety of operational are closely associated with an area of investiga-
definitions of a construct, the evidence retrieved by tion. In contrast to the descendency approach, a
reviewers typically contains more method-gener- reviewer may employ the ancestry method in
ated variance than evidence collected as primary which the reviewer retrieves information by track-
data. The fact that a particular concept or construct ing citations from one study or research report to
may be operationally defined in a variety of ways another. Most reviewers are aware of several stud-
across different studies is referred to as operational ies related to their problem before they formally
multiplicity. The presence of multiple operational- begin the literature search. These studies provide
izations in a meta-analysis represents an important bibliographies that cite earlier research reports that
source of variance. Two reviewers using an identi- may be related to the topic under study. The most
cal label for a construct may employ different informal method of retrieving research reports
operational definitions. For example, in a meta- occurs when researchers who are working in a par-
analysis on the effectiveness of rehabilitation for ticular area exchange reprints and study informa-
stroke described later in this chapter, the authors tion (Crane, 1969).
looked at the impact of rehabilitation programs for The investigator conducting a meta-analysis
stroke on functional outcomes. Functional out- samples completed studies. Reviewers may
comes were operationally defined as measures of attempt to retrieve an entire population of studies
activities of daily living (ADLs), length of stay rather than draw a representative sample of the
(LOS), language/cognitive tests, and motor/reflex studies on a particular topic. The multiple methods
tests. Other investigators have included physiolog- of study retrieval may affect the results of various
ical measures such as muscle strength, sensation, reviews. Research reports available may differ
and/or proprioception as functional outcomes. A from one source to another. Two reviewers who
meta-analysis that operationally defines functional use different retrieval techniques to locate studies
outcomes as only ADLs will produce different may end up with different sets of research reports
results than a meta-analysis that operationally and potentially different evidence. Diversity in
defines functional outcomes as including measures information retrieval methods represents a proce-
of ADLs, LOS, and language/cognitive and motor/ dural variation that may affect review conclusions.
reflex tests. Therefore, reviewers should specify the various
In the problem formation stage, it is important retrieval sources or methods they employed just as
for the researcher to provide a clearly stated a primary researcher would report the procedures
research question. This question must include used to select subjects to include in their sample.
detailed operational descriptions of how the con-
cepts will be defined. This, in turn, determines
which studies will be included in the meta-analysis. Data Evaluation and Coding
Each research report retrieved is examined care-
Data Collection
fully to determine whether it meets certain prede-
Research reviewers have multiple methods at their termined criteria. These criteria are formulated to
disposal to identify and retrieve research studies eliminate studies:
• Not related to the problem under investigation, or The Cochrane Collaboration

• Not meeting particular operational parameters.
Reviewers may differ in the identification of The Cochrane Collaboration (2004) was started
in 1993 and is the largest organization in the
criteria for evaluating the relevancy of a particular
world engaged in the production and mainte-
research report. For instance, some reviewers may nance of systematic reviews and meta-analyses
decide to include only studies published in peer- related to health care. There are 50 Cochrane
reviewed journals. Other investigators may attempt Collaboration Review Groups responsible for
to be more inclusive and obtain research reports reviews within particular areas of health care.
presented at professional meetings or in nonpub- The groups review and evaluate published meta-
lished sources such as master’s theses or doctoral analyses from existing journals. They synthesize
dissertations. this evidence and then present it in a standard
Decisions about which studies to include structured way. The reviews and summaries are
depend upon: available online as part of the Cochrane Library
(http://www.cochrane.org/). The Cochrane
• The availability of research reports, Collaboration only includes randomized trials in
• How many studies there are in total, its database of clinical or therapeutic outcome
• How many are published, studies.
• The frequency and quality of research designs
used, and
• The research question that is being examined
(Petitti, 2000). study design on outcomes. In some research areas,
the design is associated with a higher probability
In making decisions regarding which studies to of achieving a positive or negative outcome
include two important questions must be (Colditz, Miller, & Mosteller, 1989). Generally,
addressed: the poorer or lower level designs (see Figure 18.1)
are associated with a higher likelihood of a posi-
• What are the criteria for choosing studies?
tive outcome. Other studies have found no rela-
• What are the implications for a particular selec-
tionship between design characteristics, such as
tion strategy?
random assignment of subjects, and study outcome
An area that requires special attention in mak- (Concato, Shah, & Horwitz, 2000). The researcher
ing an evaluation decision is the type of research conducting a meta-analysis must make a decision
design used in the study (Lipsey & Wilson, 2001). regarding which approach to use in evaluating
There is considerable disagreement in the litera- whether to include a study in the final group of
ture on meta-analysis regarding this issue. One articles to be examined.
approach widely used in the biomedical research Once the evaluation decisions have been clearly
literature is to include only reports published in specified, information is extracted from the indi-
peer-reviewed journals that are based on random- vidual studies. In a primary research study, data are
ized clinical trials (RCTs). The argument support- collected from the individual subjects participating
ing this approach is that these studies use designs in the investigation. In a meta-analysis, information
that reduce bias and have the greatest ability to is gathered by coding the characteristics of the indi-
ensure that any effects are the result of the treat- vidual studies. Systematic coding frames are devel-
ment (independent variable) as opposed to some oped to record information and outcomes from
uncontrolled or unknown factors (Light & each individual study. An example of a coding
Pillemer, 1984). In addition, studies appearing in frame used in a recent meta-analysis examining the
scientific journals have undergone a rigorous effects of exercise programs on physical, behav-
review process and this helps ensure the validity ioral, and social outcomes in older adults with
and accuracy of the findings. dementia or Alzheimer’s disease (Heyn, Abreu, &
In contrast, Glass and colleagues (1981) sug- Ottenbacher, 2004) is in Figure 18.3.
gest that all potentially relevant studies be included The accuracy and reliability of the coding pro-
in the meta-analysis and that no prior judgments cess and information recorded on the coding frame
about the quality of a study design be made. They must be determined. This is accomplished by hav-
argue that the question of study “quality” and how ing more than one rater record information from
quality variables impact on the outcome of a study the study and then comparing their independent
can be addressed using the appropriate quantitative ratings.
reviewing methodology. Previous research has The reliability and accuracy of collecting infor-
reported mixed results regarding the impact of mation from the primary studies included in a
(continued)
Figure 18.3 An example of a coding frame used in a recent meta-analysis examining the
effects of exercise programs on physical, behavioral, and social outcomes in older adults
with dementia or Alzheimer’s disease (Heyn, Abreu, & Ottenbacher, 2004).
287
Figure 18.3 (Continued)
288
Figure 18.3 (Continued)
meta-analysis is improved when the definitions for the introduction, demographics, and methodology
treatments, outcomes, study design, subject char- sections of the article without knowledge of the
acteristics, and other variables are clearly and findings. The results and discussion section of the
completely described. article are rated independently so that they will not
Another potential problem in extracting infor- be influenced by the knowledge of the study design
mation from studies is coder bias. Strategies have and type of subjects participating or by knowledge
been developed to help reduce coder bias including of the investigators or institution where the
training and pilot testing the coding forms. One research was conducted. A comprehensive discus-
method to reduce bias is to have examiners code sion regarding how to evaluate coding decisions
and reduce error and bias are available in the liter- measure as the “degree to which the null hypothe-
ature (Orwin, 1994; Stock, 1994). sis is false.” Cohen developed and cataloged effect
size measures appropriate for use with most types
of research design and statistical analysis. Three
Analysis and Interpretation
types of effect sizes are used in most meta-analyses
The traditional criterion for gauging the impor- conducted in occupational therapy research:
tance of quantitative research findings is statisti-
• The standardized mean difference,
cal significance. Significance testing is strongly
• The odds ratio, and
influenced by the size of the sample and various
• The correlation coefficient.
authorities have questioned its continued use in
empirical research (Carver, 1978). Significance Table 18.2 contains a brief description of each
testing, which compares an observed relation to of these measures of effect size.
the chance of no relation, becomes less informa- There are other types of effect size measures
tive as evidence supporting a phenomenon accu- available to investigators conducting a meta-analy-
mulates (Hunter & sis and these effect sizes
Schmidt, 1994). are described in many
The question turns The reliability and accuracy excellent publications
from whether a treatment available on meta-analy-
effect exists to how much
of collecting information from sis (Cooper & Hedges,
of an effect exists. Effect the primary studies included 1994; Petitti, 2000). The
size measures can play an in a meta-analysis is standardized mean dif-
important role in deter- ference or d-index is
mining the degree to improved when the defini- among the most widely
which a treatment exerts tions for treatments, out- used effect size in meta-
an influence on different analysis studies reported
outcomes. As noted ear- comes, study design, subject in occupational therapy.
lier, the use of a sum- characteristics, and other For example, Vargas and
mary statistical measure Camilli (1999) used d-
such as an effect size is variables are clearly and indexes to compare the
what distinguishes meta- completely described. effects of sensory inte-
analysis from systematic gration therapy to con-
reviews. ditions in which the
Two primary methods were originally advo- subjects either received no therapy or a comparison
cated to statistically integrate results across multi- therapy. The d-index is a number that tells how far
ple studies: apart two group means are in terms of their com-
mon standard deviation. If a d-index equals 0.3, it
• Combining probabilities by adding z-scores and
indicates that 3/10 of a standard deviation separates
• Explaining variation in study effect sizes.
the average person in the two groups being com-
The technique of combining probabilities, pared. This effect size transforms the results from
referred to as the Stouffer Method, is easy to com- any two-group comparison into a standardized met-
pute when probability levels are reported. How- ric, regardless of the original measurement scales.
ever, the method of combining probabilities has The d-index can be computed from t and F
two major inadequacies: ratios when means and standard deviations are not
reported in an article. Friedman (1968) has pro-
• Studies with significant p-levels are more likely vided formulas and a rationale for transforming t
to be published than are studies with nonsignifi- and F values to d-indexes. In cases where t and F
cant p-levels. ratios are not reported, they may be estimated from
• The technique does not tap the wealth of infor- the significance level and sample size. When non-
mation contained in the variation of results found parametric statistics or percentiles are reported,
in multiple studies. effect sizes can be computed using procedures
As a consequence, the method of combining described by Glass and others (Glass et al., 1981;
probabilities is rarely used in current meta-analy- Hedges & Olkin, 1985).
sis investigations. Cohen (1988) presents several measures of dis-
Quantitative procedures capable of uncovering tribution overlap meant to enhance the interpre-
systematic variation in study outcomes were pio- tability of effect size indexes. The overlap measure
neered by Cohen (1988). He defines an effect size most often employed in meta-analysis is called U3.
Table 18.2 Description of Effect Size Measures Commonly Used

in Occupational Therapy Meta-Analysis Studies
Effect Size Formula Description Range
d-Index The d-index represents a standardized group for 0.20–0.49 small
苶G1 ⫺ X
X 苶G2 a contrast continuous measure. It is 0.50–0.79 medium
苶S ⫽ ᎏᎏ
E苶 commonly used for designs where two groups
Spooled ⬎ 0.80 large
or conditions are compared. X represents
mean of groups and Spooled is the pooled
standard deviation (some situations use
control group standard deviation).
Sometimes referred to as “g.”
The odds ratio is based on a 2 ⫻ 2 contingency 1.50⫺2.49 small
ad
Odds ratio* 苶S⫽ ᎏ
E苶 table, such as the one below. The odds ratio is 2.50⫺4.29 medium
bc the odds of success in the treatment group
⬎ 4.30 large
relative to the odds of success in the control
group.
r-index Represents the strength of association between 0.10–0.24 small

two continuous measures. Generally reported 0.25–0.39 medium
苶
E苶
S⫽ r
directly as “r” (the Pearson product moment
⬎ 0.40 large
coefficient).
*2 ⫻ 2 Table used to compute odds ratio.
Frequencies
Success Failure
Treatment group a b
Control group c d
The U3 value indicates the percentage of the popu- lapping distributions for two groups of subjects
lation with the smaller mean (generally control being compared in a meta-analysis investigation.
groups) that is exceeded by the average person in The U3 associated with a d-index of 0.30 is 61.8.
the population with the higher mean (generally This means that the average performance of sub-
treatment groups). Figure 18.4 presents the over- jects in the higher meaned (treatment) groups is
-index = 0.62
Control Group Treatment Group

Distribution Distribution
Figure 18.4 Overlapping distributions of effect sizes from the control

group and treatment group. The U3 associated with a d-index of 0.62
is 0.73. The U3 value indicates that the average performance of sub-
jects in the higher meaned (treatment) groups is “better” than 73% of
the subjects in the lower meaned (control or comparison) groups not
receiving the intervention.
“better” than 61.8% of the subjects in the lower provided by previous reviews on the same or
meaned (control or comparison) groups not receiv- related topics. In most cases, the reviewer chooses
ing the particular intervention or independent a format that is convenient for the particular review
variable. problem. Cooper (1989) proposed that reviewers
Once the type of effect size to be used has been employing quantitative methods report their results
determined, there are several other analysis con- using a format similar to that employed by investi-
cerns that the investigator must address. One of the gators conducting primary research studies. The
most important is dealing with potential sampling results of a meta-analysis should include Introduc-
bias and error in the studies examined. It is gener- tion, Methods, Results, and Discussion sections.
ally accepted that effect sizes should be weighted In the Introduction, the investigator identifies
based on the number of participants included in the the problem under review and discusses the results
individual study. This is because large samples of previous research and traditional literature
will produce more accurate population estimates. reviews. The need for a meta-analysis is estab-
For example, a d-index or r-value based on a sam- lished in the Introduction. The Methods section
ple of 500 subjects will give a more precise esti- describes the procedures the reviewer used to
mate of the true population effect size than a retrieve research reports and the sources searched.
sample of 50 subjects. Effect sizes from the stud- Information on the number of studies selected, cri-
ies included in the meta-analysis should reflect this teria used for inclusion, and coding of studies (i.e.,
fact. Various sample size weighting systems have what information was extracted from each research
been developed for different effect size measures. report) is also included in this section of the meta-
Another sampling related concern is the degree analysis. The Results section includes the findings
of variability present in the sample of effect size of the quantitative synthesis and should contain
values included in the meta-analysis study. This is information on the type of effect size used, how it
tested by what is called a homogeneity analysis. was computed, the basic unit of analysis, and the
The sample of effect size values generated by a actual statistical outcomes. Finally, the Discussion
meta-analysis will vary due to normal sampling section allows the reviewer to summarize the find-
error. The researcher wants to know if the amount ings, compare them to previous narrative reviews
of variability in the effect sizes is greater than and primary research studies, and suggest areas in
would be expected by chance (sampling error). need of further investigation.
The homogeneity analysis asks the question: Is the Table 18.3 lists a number of meta-analyses that
observed variance in effect sizes significantly dif- have been published on topics directly relevant to
ferent from that expected by sampling error? If the occupational therapy. These studies provide exam-
answer is “No,” then some statisticians would ples of how to conduct and report the findings
argue the analysis should stop, since sampling of meta-analyses in the professional and research
error is the simplest explanation for why effect literature.
sizes differ. If the answer is “Yes,” that is, the vari-
ance in effect size values is larger than would be
expected from sampling error, then the investigator An Example Relevant to
begins to examine whether study characteristics,
such as research design are associated with differ-
ences in effect sizes. The sophistication of statisti-
The narrative reviews regarding stroke rehabilita-
cal procedures to examine this variation has
tion referred to in the introduction were published
increased dramatically over the past decade and is
in 1989 and did not include meta-analysis proce-
beyond the scope of this introductory chapter.
dures. They led to confusion regarding the effec-
Information on statistical methods used in meta-
tiveness of stroke rehabilitation programs. In 1993,
analysis to test the homogeneity of effect sizes is
a meta-analysis was published in Neurology titled
provided in a number of excellent sources (Cooper
“Results of clinical trials in stroke rehabilitation
& Hedges, 1994; Hedges & Olkin, 1985).
research” (Ottenbacher & Jannell, 1993). This
meta-analysis involved 36 clinical trails examining
Reporting Results
stroke rehabilitation and included a total of 3,717
Cooper (1989) observed that reviewers using tradi- patients. The overall quantitative results suggested
tional narrative methods have no formal guidelines that the average patient in the treatment group
describing how to structure the final report of their receiving a focused program of stroke rehabilita-
findings (see Figure 18.2). Narrative reviewers tion had a better outcome than 65.5 percent of the
have traditionally followed informal guidelines patients in the control groups not receiving reha-
Table 18.3 Occupational Therapy Related Meta-Analysis Articles

Year Author(s) Topic
1985 Ottenbacher & Peterson Vestibular stimulation research
1986 Cusick, A. Research in occupational therapy
1993 Ottenbacher & Jannell Clinical trials in stroke
1996 Carlson et al. Occupational therapy for older patients
1997 Lin, Wu, Tickle-Degnen, & Coster Occupational embedded exercise
1998 Wu Context and CVA
1998 Tickle-Degnen Collaborative treatment
1999 Vargas & Camilli Sensory integration treatment
2000 Sudsawad Kinesthetic training and handwriting in children
2000 Dennis & Rebeiro Occupational therapy and mental health
2002 Horowitz Geriatric rehabilitation
2002 Reid et al. Seated mobility devices and performance
of users and caregivers
2002 Trombly & Ma Occupational therapy for stroke, part 1
2002 Ma & Trombly Occupational therapy for stroke, part 2
2002 Deane et al. Paramedical therapies and Parkinson’s disease
2002 Steultjens et al. Occupational therapy for rheumatoid arthritis
2003 Mulligan Sensory integration and children
2003 Handy et al. Upper limb function after stroke
2003a Steultjens et al. Occupational therapy for multiple sclerosis
2003b Steultjens et al. Occupational therapy for stroke (cognition)
2004 Steultjens et al. Occupational therapy for children with cerebral palsy
2004 Kielhofner et al. Outcomes research
bilitation (d-index of 0.40). The 65.5% represents was found related to the type of research design.
an example of the U3 value described earlier. The type of research design was coded in each
All primary research studies included in the study as experimental, quasi-experimental, or pre-
meta-analysis involved a comparison between a experimental. There was a significant difference in
group of persons who received a focused program the average effect size (d-index) based on whether
of rehabilitation following a stroke and a group the outcome measure was blindly recorded. Blind
that received standard medical care on a neurolog- recording means that the persons collecting the
ical or medical unit within the hospital. The out- outcome information did not know if the individ-
come measures in this meta-analysis were ual subject was a member of the treatment or
categorized as ADLs, visual/perceptual, language/ control group. The impact of blind recording, how-
cognition, length of stay, motor/reflex, and other. ever, was only found for research designs in which
The consistency and accuracy of the coding for all subjects were not randomly assigned to a treatment
study characteristics were examined by having or control group (pre- and quasi-experimental
three raters independently review and complete the designs). Figure 18.5 displays the interaction
coding form for 20 randomly selected studies. The between research design and how the outcome
intraclass correlation coefficient values ranged measures were recorded.
from 0.77 to 1.00, indicating good to excellent In contrast to the narrative literature reviews
agreement for the items on the coding form used in published in 1989 (Dobkin, 1989; Reding &
the analysis. McDowell, 1989) that reported conflicting findings
The meta-analysis found the largest effect size regarding the effectiveness of stroke rehabilitation
(d-index) for measures labeled as ADL and the programs, the stroke rehabilitation meta-analysis
smallest average effect size for language/cognition illustrates how quantitative reviewing procedures
measures. An important variation in study results can produce consensus by systematically combin-
0.8 groups (treatment and control groups) when a dif-

ference actually exists. Statistical power is directly
0.7 influenced by sample size; the larger the sample
size, the greater the statistical power when all other
0.6 factors remain the same. Statistical power is also
associated with type II errors. A type II error
Effect Size
0.5
occurs when a researcher mistakenly rejects a null
0.4 Blind hypothesis (indicating no difference between
groups) although a difference actually does exist
0.3 Not Blind (see Chapter 17).
Meta-analysis increases statistical power by
0.2
increasing the sample size used in making com-
0.1 parisons between groups. For example, in the
meta-analysis examining the effectiveness of
0 stroke rehabilitation the combined sample size for
Pre Quasi True the 36 studies was 3,717. This sample size is much
larger than the sample size included in any indi-
Type of Experimental Design vidual primary research study and ensures that the
results generated from the meta-analysis will not
Figure 18.5 Interaction between type of research be a type II error.
design and method of recording outcome
measure (blind versus not blind). (Reprinted
The second advantage is the ability to deter-
with permission from Ottenbacher, K.J., & mine the magnitude of a treatment effect. This
Jannell, S.J. [1993]. The results of clinical trials advantage is related to the distinction between
in stroke rehabilitation research. Neurology, findings that are statistically significant versus
50, 37–44.) findings that are clinically or practically important.
Statistical significance is an all-or-none determina-
tion based on a probability level, usually p ⬍ .05.
ing the results of multiple studies. This meta-analy- Effect sizes, in contrast, allow for a range of inter-
sis also illustrates how study design characteristics pretations. The interpretations require judgments
can be examined in unique ways not possible in a on the part of the researcher or reader. As implied
single or primary research investigation. The meta- earlier, statistical significance is closely related to
analysis on stroke rehabilitation outcomes found an sample size, such that with large sample sizes
interaction between blind recording of the outcome small differences between groups may be statisti-
measures and type of study design. This informa- cally significant. These small differences may have
tion is important to the interpretation of existing limited practical importance. In contrast, large dif-
research investigations in this area, but is also im- ferences that are practically important may not be
portant to researchers planning future studies. statistically significant if the sample size is small.
Measures of effect size provide a more direct
indication of treatment impact. Effect size meas-
Advantages and Limitations ures can be converted to percent differences using
of Meta-Analysis indexes such as the U3 value described earlier. For
example, the d-index of 0.40 reported for the
stroke rehabilitation meta-analysis described ear-
Light and Pillemer (1984) have identified four spe- lier can be converted to a U3 value of 65.5%
cific advantages in the use of meta-analysis to inte- (Cohen, 1988). This U3 value means that the aver-
grate studies. These advantages include: age person in the treatment groups receiving stroke
• Increased statistical power, rehabilitation did 15.5% better than the average
• Obtaining an estimate of the magnitude of exper- person in the control or comparison group who
imental effects, received standard medical care. Using this infor-
• Greater insight into the nature of relationships mation, the reader is able to make a judgment
among variables, and regarding whether a 15.5% increase in perform-
• The ability to objectively explore contradictions ance is clinically or practically important.
in a group of studies. The third advantage of meta-analysis is the
ability to better understand the relationship
Statistical power is related to the sensitivity or between study subject characteristics and study
ability of a study to find a true difference between outcome. Since multiple studies are included in a
meta-analysis, there is greater variability in study tiple studies may create an illusion of statistical
design and subject characteristics than are found objectivity that is not justified by the data obtained
in an individual primary from the review. Related
research study. This vari- to the issue of statistical
ability allows the re- Meta-analysis is particularly precision is the fact
searcher conducting a relevant to disciplines such that multiple hypotheses
meta-analysis to exam- tests may be included in
ine questions that cannot as occupational therapy that a single research report
be asked in a primary are in the early stages of and effect sizes gener-
study. The meta-analysis ated from these multiple
on stroke rehabilitation
developing theory and hypotheses are not inde-
provides an example of research to support clinical pendent data points. This
such a question. As noted introduces the problem
in the previous section,
practice. of non-independence,
an interaction was found which may affect the
between the type of research design (per-experi- results of inferential statistical procedures used to
mental, quasi-experimental, and true-experimen- analyze the data. The role of inferential statistical
tal) and how the outcome measures were recorded procedures in the data analysis stage of meta-
(blind recording versus not blind) (see Figure analysis is controversial and beyond the scope of
18.5). This relationship could not have been found this introductory chapter (Lipsey & Wilson, 2001).
in a primary research study.
The final advantage is the potential to explore
Qualitative Methods and Meta-Analysis
contradictions across a series of primary research
studies. Meta-analysis allows the investigator to Researchers using qualitative methods have pro-
systematically examine moderating or confounding posed applying the concepts associated with sys-
variables that may explain contradictory outcomes tematic reviews to synthesize findings from
in primary studies. For example, if age is a variable multiple qualitative studies (Paterson, Thorne,
that impacts the outcome of interest so that a treat- Canam, & Jillings, 2001). Upshur (2001) suggested
ment is more likely to work for subjects older than that there are two different definitions of evidence
60 years of age than for those younger than age 60, from the qualitative perspective. The first narrative
this can be detected in a meta-analysis; a meta- evidence he describes as primarily Qualita-
analysis may help explain why previous studies tive/Personal and the second as Qualitative/Gene-
including younger subjects produce results differ- ral. Qualitative/Personal evidence is concrete,
ent or contradictory to studies using the same treat- particular, and historical while Qualitative/General
ment, but that include older subjects. evidence is historical and social.
Meta-analysis procedures also have limitations Occupational therapists have embraced qualita-
that must be recognized and acknowledged. Meta- tive methodologies in search of a better under-
analysis methods contain aspects of both art and standing of the social and personal aspects of heath
science, as does all research. The science is care (Clark, 1993; Krefting & Krefting, 1991), in
revealed in the systematic application and defini- conducting individual studies; however, we were
tion of a research approach related to literature unable to find examples of systematic reviews of
reviewing. The art refers to the judgments that qualitative research in the occupational therapy
need to be made in the application of the proce- literature. Examples do exist in the broader health-
dures. Like all research methods, meta-analysis care literature (Clemmens, 2003; McCormick,
involves assumptions that must be made explicit, Rodney, & Varcoe, 2003; Varcoe, Rodney, &
and if these assumptions are not clear to the user or McCormick, 2003; Sandelowski, Lambe &
reader, misleading conclusions may occur. Barroso, 2004).
The ability of meta-analysis procedures to test Qualitative systematic reviews are a new
certain interactions or relationships contained research methodology and the procedures and tech-
within aggregated studies does not mean that all nology are not well developed (Schreiber, Crooks,
problems of conceptualization or methodological & Stern, 1997). Qualitative research has informed
artifact can be resolved in this manner. As in eval- healthcare practitioners for decades, it deepens our
uating a single study, alternate conceptualizations understanding of human experience and of phe-
of included treatment variables may rival the nomena that illustrate that experience (Morse,
one offered. In addition, some readers may judge 1997). Standards for assessing the rigor of qualita-
that the quantitative synthesis of results from mul- tive research methods for conducting and dissemi-
nating systematic reviews need to be developed and Cohen, J. (1988). Statistical power analysis for the behav-
tested (Davies & Dodd, 2002; Whittemore, Chase, ioral sciences (2nd ed.). Hillsdale, NJ: Lawrence
Erlbaum.
& Mandle, 2001). There is also a lack of agreement Colditz, G. A., Miller, J. N., & Mosteller, F. (1989). How
regarding how to assess the quality and usefulness study design affects outcomes in comparisons of ther-
of qualitative studies for evidence-based practice apy, Part 1: Medical. Statistics in Medicine, 8,
(Marks, 1999; Morse, 1997; Morse, Swanson, & 441–454.
Concato, J., Shah, N., & Horwitz, R. I. (2000).
Kuzel, 2001; Paterson et al, 2001).
Randomized, controlled trials, Observational studies,
Practitioners traditionally combine the art and and the hierarchy of research designs. The New
science of occupational therapy to determine the England Journal of Medicine, 342, 1887–1892.
quality of clinical care. This integrative perspec- Cooper, H. M. (1989). Integrating research: A guide for lit-
tive requires the use of qualitative studies to erature reviews. Newbury Park, CA: SAGE
Publications.
improve our understanding of occupational ther- Cooper, H. M. (1998). Synthesizing research: A guide for
apy and occupational science (Giacomini, 2001). literature reviews. Thousand Oaks, CA: SAGE
The contributions to occupational therapy evi- Publications.
dence are complemented by both methodologies; Cooper, H. M., & Hedges, L. V. (1994). The handbook of
research synthesis. New York: Russell Sage Foundation.
they both can enhance client and caretaker empow-
Crane, D. (1969). Social structure in a group of scientists:
erment and the quality of occupational therapy and A test of the invisible college hypothesis. American
occupational science. One of the challenges facing Sociological Review, 34, 335–352.
researchers in occupational therapy is to collec- Cusick, A. (1986). Research in occupational therapy: Meta-
tively integrate the information from multiple analysis. Australian Occupational Therapy Journal, 33,
142–147.
qualitative investigations. Davies, D., & Dodd, J. (2002). Qualitative research and the
In spite of the limitations and challenges cited question of rigor. Qualitative Health Research, 12(2),
earlier, meta-analysis represents a significant 279–289.
advance over the traditional narrative methods Deane, K. H., Ellis-Hill, C., Jones, D., Whurr, R., Ben
Shlomo, Y., Playford, E. D., & Clarke, C. E. (2002).
of reviewing quantitative research. Meta-analysis
Systematic review of paramedical therapies for
is particularly relevant to disciplines such as occu- Parkinson’s disease. Movement Disorders, 17,
pational therapy that are in the early stages of 984–991.
developing theory and research to support clinical Dennis, D. M., & Rebeiro, K. L. (2000). Occupational
practice. The use of meta-analysis represents an therapy in pediatric mental health: Do we practice what
we preach? Occupational Therapy in Mental Health,
important shift in scientific thinking in which the 16, 5–25.
literature review is conceptualized as a form of sci- Dobkin, B. H. (1989). Focused stroke rehabilitation pro-
entific inquiry in its own right. The continued evo- grams do not improve outcome. Archives of Neurology,
lution and application of meta-analysis should 46, 701–703.
Evidence-Based Medicine Working Group. (1992).
help researchers establish a scientifically respected
Evidence-based medicine: A new approach to teaching
foundation for evidence-based practice in occupa- the practice of medicine. Journal of the American
tional therapy. Medical Association, 268, 2420–2425.
Friedman, H. (1968). Magnitude of experimental effect and
a table for its rapid estimation. Psychological Bulletin,
REFERENCES 70, 245–251.
Abreu, B. C. (2003). Evidence-based practice. In G. L. Giacomini, M. K. (2001). The rocky road: Qualitative
McCormack, E. Jaffe, & M. Goodman-Lavey (Eds.), research as evidence. ACP Journal Club, 134 (1),
The occupational therapy manager (4th ed.) (pp. A11–A13.
351–373). Bethesda, MD: AOTA Press. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-
Carlson, M., Fanchiang, S. P., Zemke, R., & Clark, F. analysis in social research. Beverly Hills, CA: SAGE
(1996). A meta-analysis of the effectiveness of occupa- Publications.
tional therapy for older persons. American Journal of Handy, J., Salinas, S., Blanchard, S. A., & Aitken, M. J.
Occupational Therapy, 50, 89–98. (2003). Meta-analysis examining the effectiveness of
Carver, R. P. (1978). The case against statistical signifi- electrical stimulation in improving functional use of the
cance testing. Harvard Educational Review, 48, upper limb in stroke patients. Physical and
378–399. Occupational Therapy in Geriatrics, 21, 67–78.
Clark, F. (1993). Occupation embedded in a real life: Hedges, L. V., & Olkin, I. (1985). Statistical methods for
Interweaving occupational science and occupational meta-analysis. Orlando, FL: Academic Press.
therapy. 1993 Eleanor Clarke Slagle Lecture. American Heyn, P., Abreu, B. C., & Ottenbacher, K. J. (2004). The
Journal of Occupational Therapy, 47(12), 1067–1078. effects of exercise training on elderly persons with cog-
Clemmens, D. (2003). Adolescent motherhood: A meta- nitive impairment and dementia: A meta-analysis.
synthesis of qualitative studies. The American Journal Archives of Physical Medicine and Rehabilitation, 85,
of Maternal/Child Nursing, 28(2), 93–99. 1694–1704.
Cochrane Reviews. (2004). Cochrane Collaboration Horowitz, B. P. (2002). Rehabilitation utilization in New
Website [Online]. Retrieved November 15, 2005 from York state: Implications for geriatric rehabilitation in
http://www.cochrane.org 2015. Topics in Geriatric Rehabilitation, 17, 78–89.
Hunter, J. E., & Schmidt, F. L. (1994). Correcting for rehabilitation programs improve outcome. Archives of
sources of artificial variation across studies. In H. Neurology, 46, 700–701.
M.Cooper & L. V. Hedges (Eds.), The handbook of Reid, D., Laliberte-Rudman, D., & Hebert, D. (2002).
research synthesis (pp. 323–336). New York: Russell Impact of wheeled seated mobility devices on adult
Sage Foundation. users’ and their caregivers’ occupational performance:
Institute of Medicine. (2001). Crossing the quality chasm: A critical literature review. Canadian Journal of
A new health system for the 21st century. Washington, Occupational Therapy, 69, 261–280.
DC: National Academy Press. Rosenthal, R. (1978). Combining results of independent
Kielhofner, G., Hammel, J., Finlayson, M., Helfrich, C., & studies. Psychological Bulletin, 85, 185–193.
Taylor, R. R. (2004). Documenting outcomes of occu- Sackett, D. L., Straus, S., Richardson, S., Rosenberg, W.,
pational therapy: the center for outcomes research and & Haynes, R. B. (2000). Evidence-based medicine:
education. American Journal of Occupational Therapy, How to practice and teach EBM (2nd ed.) Edinburgh,
58, 15–23. UK: Churchill Livingstone.
Krefting, L., & Krefting, D. (1991). Leisure activities after Sandelowski, M., Lambe, C., & Barroso, J. (2004). Stigma
a stroke: An ethnographic approach. American Journal in HIV-positive women. Journal of Nursing
of Occupational Therapy, 45(5), 429–436. Scholarship, 36(2), 122–128.
Light, R. J., & Pillemer, D. B. (1984). Summing up: The Schreiber, R., Crooks, D., & Stern, P. N. (1997). Qualita-
science of reviewing research. Cambridge, MA: tive meta-analysis. In J. M. Morse (Ed.), Completing a
Harvard University Press. qualitative project: Details and dialogue (pp. 311–326).
Lin, K., Wu, C., Tickle-Degnen, L., & Coster, W. (1997). Thousand Oaks, CA: SAGE Publications.
Enhancing occupational performance through occu- Steultjens, E. M., Dekker, J., Bouter, L. M., Cardol, M.,
pationally embedded exercise: A meta-analytic van de Nes, J. C., & van den Ende, C. H. (2003a).
review. Occupational Therapy Journal of Research, Occupational therapy for multiple sclerosis. Cochrane
17, 25–47. Database Systematic Reviews, CD003608.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta- Steultjens, E. M., Dekker, J., Bouter, L. M., van de Nes, J.
analysis (Vol. 49) Thousand Oaks, CA: SAGE C., Cup, E. H., & van den Ende, C. H. (2003b).
Publications. Occupational therapy for stroke patients: A systematic
Ma, H. I., & Trombly, C. A. (2002). A synthesis of the review. Stroke, 34, 676–687.
effects of occupational therapy for persons with stroke, Steultjens, E. M., Dekker, J., Bouter, L. M., van de Nes, J.
Part II: Remediation of impairments. American Journal C., Lambregts, B. L., & van den Ende, C. H. (2004).
of Occupational Therapy, 56, 260–274. Occupational therapy for children with cerebral palsy:
Marks, S. (1999). Qualitative Studies. In A. McKibbon A systematic review. Clinical Rehabilitation, 18, 1–14.
(with Eady, A. & Marks, S.). (Eds.), PDQ evidence- Steultjens, E. M., Dekker, J., Bouter, L. M., van
based principles and practice (pp. 187–204). Hamilton, Schaardenburg, D., van Kuyk, M. A., & van den Ende,
Ontario: B. C. Decker. C. H. (2002). Occupational therapy for rheumatoid
McCormick, J., Rodney, P., & Varcoe, C. (2003). arthritis: A systematic review. Arthritis Rheumatism,
Reinterpretations across studies: An approach to 47, 672–685.
meta-analysis. Qualitative Health Research, 13(7), Stock, W. A. (1994). Systematic coding for research syn-
933–944. thesis. In H. M. Cooper & L. V. Hedges (Eds.), The
Morse, J. M. (Ed.). (1997). Completing a qualitative proj- handbook of research synthesis (pp. 125–138). New
ect: Details and dialogue. Thousand Oaks, CA: SAGE York: Russell Sage Foundation.
Publications. Sudsawad, P. (2000). The effect of kinesthetic training on
Morse, J. M., Swanson, J. M., & Kuzel, A. J. (2001). The handwriting performance in grade one children with
nature of qualitative evidence. Thousand Oaks, CA: handwriting difficulties. [Dissertation Abstract: 2000-
SAGE Publications. 95010-344]. Dissertation Abstracts International:
Mulligan, S. (2003). Examination of the evidence for occu- Section B: The Sciences & Engineering, 60 (11-B),
pational therapy using a sensory integration framework 5472. US: University Microfilms International.
with children: Part one. Sensory Integration Special Tickle-Degnen, L. (1998). Communicating with clients
Interest Section Quarterly, 26, 1–4. about treatment outcomes: The use of meta-analytic
Orwin, R. G. (1994). Evaluating coding decisions. In H. evidence in collaborative treatment planning.
M.Cooper & L. V. Hedges (Eds.), The handbook of American Journal of Occupational Therapy, 52,
research synthesis (pp. 139–162). New York: Russell 526–530.
Sage Foundation. Trombly, C. A., & Ma, H. I. (2002). A synthesis of the
Ottenbacher, K. J., & Jannell, S. (1993). The results of effects of occupational therapy for persons with stroke,
clinical trials in stroke rehabilitation research. Archives Part I: Restoration of roles, tasks, and activities.
of Neurology, 50, 37–44. American Journal of Occupational Therapy, 56,
Ottenbacher, K. J., & Petersen, P. (1985). A meta-analysis 250–259.
of applied vestibular stimulation research. Physical and University Health Network. (2004) Center for Evidence-
Occupational Therapy in Pediatrics, 5, 119–134. Based Medicine [Online]. Available at: http://www.
Paterson, B. L., Thorne, S. E., Canam, C., & Jillings, C. cebm.utoronto.ca/
(2001). Meta-study of qualitative health research: A Upshur, R. E. G. (2001). The status of qualitative research
practical guide to meta-analysis and meta-synthesis as evidence. In J. M. Morse, J. M. Swanson & A. J.
(vol. 3). Thousand Oaks, CA: SAGE Publications. Kuzel (Eds.), The nature of qualitative evidence
Petitti, D. B. (2000). Meta-analysis, decision analysis, and (pp. 5–26). Thousand Oaks, CA: SAGE Publications.
cost-effectiveness analysis: Methods for quantitative Varcoe, C., Rodney, P., & McCormick, J. (2003). Health
synthesis in medicine. New York: Oxford University care relationships in context: An analysis of three
Press. ethnographies. Qualitative Health Research, 13(7),
Reding, M. J., & McDowell, F. H. (1989). Focused stroke 957–973.
Vargas, S., & Camilli, G. (1999). A meta-analysis of Wu, C. Y. (1998). Effects of context on movement kine-
research on sensory integration treatment. American matics in adults with and without cerebral vascular
Journal of Occupational Therapy, 53, 189–198. accident. [Dissertation Abstract: 1998-95002-215].
Whitehead, A. (2002). Meta-analysis of controlled clinical Dissertation Abstracts International: Section B: The
trials. Chichester: John Wiley & Sons. Sciences & Engineering. 58 (7-B), 3593. US:
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001). University Microfilms International.
Validity in qualitative research. Qualitative Health
Research, 11(4), 522–537.
18Kielhofner(F)-App-A 5/5/06 5:26 PM Page 299
A P P E N D I X A
Statistical Reference Tables

Machiko R. Tomita
FPO
Table A Areas Under the Normal Curve Between Mean and z-Score
Area Between Area Between Area Between
z 0 and z z 0 and z z 0 and z
0.00 .0000 0.25 .0987 0.50 .1915
0.01 .0040 0.26 .1026 0.51 .1950
0.02 .0080 0.27 .1064 0.52 .1985
0.03 .0120 0.28 .1103 0.53 .2019
0.04 .0160 0.29 .1141 0.54 .2054
0.05 .0199 0.30 .1179 0.55 .2088
0.06 .0239 0.31 .1217 0.56 .2123
0.07 .0279 0.32 .1255 0.57 .2157
0.08 .0319 0.33 .1293 0.58 .2190
0.09 .0359 0.34 .1331 0.59 .2224
0.10 .0398 0.35 .1368 0.60 .2257
0.11 .0438 0.36 .1406 0.61 .2291
0.12 .0478 0.37 .1443 0.62 .2324
0.13 .0517 0.38 .1480 0.63 .2357
0.14 .0557 0.39 .1517 0.64 .2389
0.15 .0596 0.40 .1554 0.65 .2422
0.16 .0636 0.41 .1591 0.66 .2454
0.17 .0675 0.42 .1628 0.67 .2486
0.18 .0714 0.43 .1664 0.68 .2517
0.19 .0753 0.44 .1700 0.69 .2549
0.20 .0793 0.45 .1736 0.70 .2580
0.21 .0832 0.46 .1772 0.71 .2611
0.22 .0871 0.47 .1808 0.72 .2642
0.23 .0910 0.48 .1844 0.73 .2673
0.24 .0948 0.49 .1879 0.74 .2704
(continued)
Table A Areas Under the Normal Curve Between Mean and z-Score (continued)

0.75 .2734 1.23 .3907 1.70 .4554
0.76 .2764 1.24 .3925 1.71 .4564
0.77 .2794 1.25 .3944 1.72 .4573
0.78 .2823 1.26 .3962 1.73 .4582
0.79 .2852 1.27 .3980 1.74 .4591
0.80 .2881 1.28 .3997 1.75 .4599
0.81 .2910 1.29 .4015 1.751 .4600
0.82 .2939 1.30 .4032 1.76 .4608
0.83 .2967 1.31 .4049 1.77 .4616
0.84 .2995 1.32 .4066 1.78 .4625
0.85 .3023 1.33 .4082 1.79 .4633
0.86 .3051 1.34 .4099 1.80 .4641
0.87 .3078 1.35 .4115 1.81 .4649
0.88 .3106 1.36 .4131 1.82 .4656
0.89 .3133 1.37 .4147 1.83 .4664
0.90 .3159 1.38 .4162 1.84 .4671
0.91 .3186 1.39 .4177 1.85 .4678
0.92 .3212 1.40 .4192 1.86 .4686
0.93 .3238 1.41 .4207 1.87 .4693
0.94 .3264 1.42 .4222 1.88 .4699
0.95 .3289 1.43 .4236 1.881 .4700
0.96 .3315 1.44 .4251 1.89 .4706
0.97 .3340 1.45 .4265 1.90 .4713
0.98 .3365 1.46 .4279 1.91 .4719
0.99 .3389 1.47 .4292 1.92 .4726
1.00 .3413 1.48 .4306 1.93 .4732
1.01 .3438 1.49 .4319 1.94 .4738
1.02 .3461 1.50 .4332 1.95 .4744
1.03 .3485 1.51 .4345 1.96 .4750
1.04 .3508 1.52 .4357 1.97 .4756
1.05 .3531 1.53 .4370 1.98 .4761
1.06 .3554 1.54 .4382 1.99 .4767
1.07 .3577 1.55 .4394 2.00 .4772
1.08 .3599 1.56 .4406 2.01 .4778
1.09 .3621 1.57 .4418 2.02 .4783
1.10 .3643 1.58 .4429 2.03 .4788
1.11 .3665 1.59 .4441 2.04 .4793
1.12 .3686 1.60 .4452 2.05 .4798
1.13 .3708 1.61 .4463 2.054 .4800
1.14 .3729 1.62 .4474 2.06 .4803
1.15 .3749 1.63 .4484 2.07 .4808
1.16 .3770 1.64 .4495 2.08 .4812
1.17 .3790 1.6451 .4500 2.09 .4817
1.18 .3810 1.65 .4505 2.10 .4821
1.19 .3830 1.66 .4515 2.11 .4826
1.20 .3849 1.67 .4525 2.12 .4830
1.21 .3869 1.68 .4535 2.13 .4834
1.22 .3888 1.69 .4545 2.14 .4838
1
z1.645. Area between 0-z .4500 indicates the area above z .05.
300
Table A Areas Under the Normal Curve Between Mean and z-Score (continued)

2.15 .4842 2.58 .4951 3.03 .4988
2.16 .4846 2.59 .4952 3.04 .4988
2.17 .4850 2.60 .4953 3.05 .49886
2.18 .4854 2.61 .4955 3.06 .49889
2.19 .4857 2.62 .4956 3.07 .49893
2.20 .4861 2.63 .4957 3.08 .49896
2.21 .4864 2.64 .4959 3.093 .49900
2.22 .4868 2.65 .4960 3.10 .49903
2.23 .4871 2.66 .4961 3.11 .49906
2.24 .4875 2.67 .4962 3.12 .49910
2.25 .4878 2.68 .4963 3.13 .49913
2.26 .4881 2.69 .4964 3.14 .49916
2.27 .4884 2.70 .4965 3.15 .49918
2.28 .4887 2.71 .4966 3.16 .49921
2.29 .4890 2.72 .4967 3.17 .49924
2.30 .4893 2.73 .4968 3.18 .49926
2.31 .4896 2.74 .4969 3.19 .49929
2.32 .4898 2.75 .4970 3.20 .49931
2.3262 .4900 2.76 .4971 3.21 .49934
2.33 .4901 2.77 .4972 3.22 .49936
2.34 .4904 2.78 .4973 3.23 .49938
2.35 .4906 2.79 .4974 3.24 .49940
2.36 .4909 2.80 .4974 3.25 .49942
2.37 .4911 2.81 .4975 3.26 .49944
2.38 .4913 2.82 .4976 3.27 .49946
2.39 .4916 2.83 .4977 3.28 .49948
2.40 .4918 2.84 .4977 3.29 .49950
2.41 .4920 2.85 .4978 3.30 .49951
2.42 .4922 2.86 .4979 3.31 .49953
2.43 .4925 2.87 .4979 3.32 .49955
2.44 .4927 2.88 .4980 3.33 .49957
2.45 .4929 2.89 .4981 3.34 .49958
2.46 .4931 2.90 .4981 3.35 .49960
2.47 .4932 2.91 .4982 3.36 .49961
2.48 .4934 2.92 .4982 3.37 .49962
2.49 .4936 2.93 .4983 3.38 .49964
2.50 .4938 2.94 .4984 3.39 .49965
2.51 .4940 2.95 .4984 3.40 .49966
2.52 .4941 2.96 .4985 3.45 .49972
2.53 .4943 2.97 .4985 3.50 .49977
2.54 .4945 2.98 .4986 3.60 .49984
2.55 .4946 2.99 .4986 3.70 .49989
2.56 .4948 3.00 .4987 3.80 .49993
2.57 .4949 3.01 .4987 3.90 .49995
2.576 .4950 3.02 .4987 4.00 .49997
2z2.326. Area between 0-z .4900 indicates the area above z .01.
3z3.09. Area between 0-z .4990 indicates the area above z .001.
301
1 2 2
1- 1-
t t t
Table B Critical Values of t

df 1 0.10 0.05 0.025
1 3.078 6.314 12.706
2 1.886 2.920 4.303
3 1.638 2.353 3.182
4 1.533 2.132 2.776
5 1.476 2.015 2.571
6 1.440 1.943 2.447
7 1.415 1.895 2.365
8 1.397 1.860 2.306
9 1.383 1.833 2.262
10 1.372 1.812 2.228
11 1.363 1.796 2.201
12 1.356 1.782 2.179
13 1.350 1.771 2.160
14 1.345 1.761 2.145
15 1.341 1.753 2.131
16 1.337 1.746 2.120
17 1.333 1.740 2.110
18 1.330 1.734 2.101
19 1.328 1.729 2.093
20 1.325 1.725 2.086
21 1.323 1.721 2.080
22 1.321 1.717 2.074
23 1.319 1.714 2.069
24 1.318 1.711 2.064
25 1.316 1.708 2.060
26 1.315 1.706 2.056
27 1.314 1.703 2.052
28 1.313 1.701 2.048
29 1.311 1.699 2.045
30 1.310 1.697 2.042
40 1.303 1.684 2.021
60 1.296 1.671 2.000
120 1.289 1.658 1.980
∞ 1.282 1.645 1.960
2 .20 .10 .05
For independent t-tests df (n1 1) (n 1)

For paired t-tests df n 1
Calculated value must be greater than or equal to critical value to reject Ho.
Source: Adapted from Table 12 in Pearson and Hartley (Eds.) (1970). Biometrika tables for statisticians.
New York: Cambridge University Press, with permission of Biometrika Trustees.
302
Table B Critical Values of t (continued)

df 1 0.01 0.005 0.0025 0.0005
1 31.821 63.657 127.322 636.590
2 6.965 9.925 14.089 31.598
3 4.541 5.841 7.453 12.924
4 3.747 4.604 5.598 8.610
5 3.365 4.032 4.773 6.869
6 3.143 3.707 4.317 5.959
7 2.998 3.499 4.029 5.408
8 2.896 3.355 3.833 5.041
9 2.821 3.250 3.690 4.781
10 2.764 3.169 3.581 4.587
11 2.718 3.106 3.497 4.437
12 2.681 3.055 3.428 4.318
13 2.650 3.012 3.372 4.221
14 2.624 2.977 3.326 4.140
15 2.602 2.947 3.286 4.073
16 2.583 2.921 3.252 4.015
17 2.567 2.898 3.222 3.965
18 2.552 2.878 3.197 3.922
19 2.539 2.861 3.174 3.883
20 2.528 2.845 3.153 3.849
21 2.518 2.831 3.135 3.819
22 2.508 2.819 3.119 3.792
23 2.500 2.807 3.104 3.768
24 2.492 2.797 3.091 3.745
25 2.485 2.787 3.078 3.725
26 2.479 2.779 3.067 3.707
27 2.473 2.771 3.057 3.690
28 2.467 2.763 3.047 3.674
29 2.462 2.756 3.038 3.659
30 2.457 2.750 3.030 3.646
40 2.423 2.704 2.971 3.551
60 2.390 2.660 2.915 3.460
120 2.358 2.617 2.860 3.373
∞ 2.326 2.576 2.807 3.291
2 .02 .01 .005 .001
For independent t-tests df (n1 1) (n 1)

For paired t-tests df n 1
Source: Adapted from Table 12 in Pearson and Hartley (Eds.) (1970). Biometrika tables for statisticians.
303

1-
2
Table C Critical Values of Chi-square (2)

df 1 .05 .025 .01 .005 .001
1 3.84 5.02 6.64 7.88 10.83
2 5.99 7.38 9.21 10.60 13.82
3 7.82 9.35 11.35 12.84 16.27
4 9.49 11.14 13.28 14.86 18.47
5 11.07 12.83 15.09 16.75 20.52
6 12.59 14.45 16.81 18.55 22.46
7 14.07 16.01 18.48 20.28 24.32
8 15.51 17.53 20.09 21.96 26.13
9 16.92 19.03 21.67 23.59 27.88
10 18.31 20.48 23.21 25.19 29.59
11 19.68 21.92 24.73 26.76 31.26
12 21.03 23.34 26.22 28.30 32.91
13 22.36 24.74 27.69 29.82 34.53
14 23.69 26.12 29.14 31.32 36.12
15 25.00 27.49 30.58 32.80 37.70
16 26.30 28.85 32.00 34.27 39.25
17 27.59 30.19 33.41 35.72 40.79
18 28.87 31.53 34.81 37.16 42.31
19 30.14 32.85 36.19 38.58 43.82
20 31.41 34.17 37.57 40.00 45.32
21 32.67 35.48 38.93 41.40 46.80
22 33.92 36.78 40.29 42.80 48.27
23 35.17 38.06 41.64 44.18 49.73
24 36.42 39.36 42.98 45.56 51.18
25 37.65 40.65 44.31 46.93 52.62
26 38.89 41.92 45.64 48.29 54.05
27 40.11 43.19 46.96 49.65 55.47
28 41.34 44.46 48.28 50.99 56.89
29 42.56 45.72 49.59 52.34 58.30
30 43.77 46.98 50.89 53.67 59.70
40 55.76 59.34 63.69 66.77 73.40
50 67.51 71.42 76.15 79.49 86.66
For one-sample test, df k 1. For two sample test, df (R 1) (C 1) where R is number of cells in
row and C is number of cells in column. Calculated value must be greater than or equal to critical value
to reject H0.
Source: Adapted from Table 8 in Pearson and Hartley (Eds.) (1970). Biometrika Tables for Statisticians.
New York: Cambridge University Press, with permission of the Biometrika Trustees.
304
18Kielhofner(F)-App-A
Table D Critical Values of F, at .05
df for Between-Groups
df for
Error 1 2 3 4 5 6 7 8 9 10 12 15 20 30 60 ∞
1 161.4 199.5 215.7 224.6 230.2 234.0 236.8 238.9 240.5 241.9 243.9 245.9 248.0 250.1 252.2 254.3
5/5/06
2 18.51 19.00 19.16 19.25 19.30 19.33 19.35 19.37 19.38 19.40 19.41 19.43 19.45 19.46 19.48 19.50
3 10.13 9.55 9.28 9.12 9.01 8.64 8.89 8.85 8.81 8.79 8.74 8.70 8.66 8.62 8.57 8.53
4 7.71 6.94 6.59 6.39 6.26 6.16 6.09 6.04 6.00 5.96 5.91 5.86 5.80 5.75 5.69 5.63
5 6.61 5.79 5.41 5.19 5.05 4.95 4.88 4.82 4.77 4.74 4.68 4.62 4.56 4.50 4.43 4.36
5:26 PM
6 5.99 5.14 4.76 4.53 4.39 4.28 4.21 4.15 4.10 4.06 4.00 3.94 3.87 3.81 3.74 3.67
7 5.59 4.74 4.35 4.12 3.97 3.87 3.79 3.73 3.68 3.64 3.57 3.51 3.44 3.38 3.30 3.23
8 5.32 4.46 4.07 3.84 3.69 3.58 3.50 3.44 3.39 3.35 3.28 3.22 3.15 3.08 3.01 2.93
9 5.12 4.26 3.86 3.63 3.48 3.37 3.29 3.23 3.18 3.14 3.07 3.01 2.94 2.86 2.79 2.71
10 4.96 4.10 3.71 3.48 3.33 3.22 3.14 3.07 3.02 2.98 2.91 2.85 2.77 2.70 2.62 2.54
Page 305
11 4.84 3.98 3.59 3.36 3.20 3.09 3.01 2.95 2.90 2.85 2.79 2.72 2.65 2.57 2.49 2.40
12 4.75 3.89 3.49 3.26 3.11 3.00 2.91 2.85 2.80 2.75 2.69 2.62 2.54 2.47 2.38 2.30
13 4.67 3.81 3.41 3.18 3.03 2.92 2.83 2.77 2.71 2.67 2.60 2.53 2.46 2.38 2.30 2.21
14 4.60 3.74 3.34 3.11 2.96 2.85 2.76 2.70 2.65 2.60 2.53 2.46 2.39 2.31 2.22 2.13
15 4.54 3.68 3.29 3.06 2.90 2.79 2.71 2.64 2.59 2.54 2.48 2.40 2.33 2.25 2.16 2.07
16 4.49 3.63 3.24 3.01 2.85 2.74 2.66 2.59 2.54 2.49 2.42 2.35 2.28 2.19 2.11 2.01
17 4.45 3.59 3.20 2.96 2.81 2.70 2.61 2.55 2.49 2.45 2.38 2.31 2.23 2.15 2.06 1.96
18 4.41 3.55 3.16 2.93 2.77 2.66 2.58 2.51 2.46 2.41 2.34 2.27 2.19 2.11 2.02 1.92
19 4.38 3.52 3.13 2.90 2.74 2.63 2.54 2.48 2.42 2.38 2.31 2.23 2.16 2.07 1.98 1.88
20 4.35 3.49 3.10 2.87 2.71 2.60 2.51 2.45 2.39 2.35 2.28 2.20 2.12 2.04 1.95 1.84
21 4.32 3.47 3.07 2.84 2.68 2.57 2.49 2.42 2.37 2.32 2.25 2.18 2.10 2.01 1.92 1.81
22 4.30 3.44 3.05 2.82 2.66 2.55 2.46 2.40 2.34 2.30 2.23 2.15 2.07 1.98 1.89 1.78
23 4.28 3.42 3.03 2.80 2.64 2.53 2.44 2.37 2.32 2.27 2.20 2.13 2.05 1.96 1.86 1.76
24 4.26 3.40 3.01 2.78 2.62 2.51 2.42 2.36 2.30 2.25 2.18 2.11 2.03 1.94 1.84 1.73
25 4.24 3.39 2.99 2.76 2.60 2.49 2.40 2.34 2.28 2.24 2.16 2.09 2.01 1.92 1.82 1.71
26 4.23 3.37 2.98 2.74 2.59 2.47 2.39 2.32 2.27 2.22 2.15 2.07 1.99 1.90 1.80 1.69
27 4.21 3.35 2.96 2.73 2.57 2.46 2.37 2.31 2.25 2.20 2.13 2.06 1.97 1.88 1.79 1.67
28 4.20 3.34 2.95 2.71 2.56 2.45 3.26 2.29 2.24 2.19 2.12 2.04 1.96 1.87 1.77 1.65
29 4.18 3.33 2.93 2.79 2.55 2.43 2.35 2.28 2.122 2.18 2.10 2.03 1.94 1.85 1.75 1.64
30 4.17 3.32 2.92 2.69 2.53 2.42 2.33 2.27 2.21 2.16 2.09 2.01 1.93 1.84 1.74 1.62
40 4.08 3.23 2.84 2.61 2.45 2.34 2.25 2.18 2.12 2.08 2.00 1.92 1.84 1.74 1.64 1.51
60 4.00 3.15 2.76 2.53 2.37 2.25 2.17 2.10 2.04 1.99 1.92 1.84 1.75 1.65 1.53 1.39
120 3.92 3.07 2.68 2.45 2.29 2.17 2.09 2.02 1.96 1.91 1.83 1.75 1.66 1.55 1.43 1.25
∞ 3.84 3.00 2.60 2.37 2.21 2.10 2.01 1.94 1.88 1.83 1.75 1.67 1.57 1.46 1.32 1.00
305
Table E Critical Values of F, .01
306
df for Between-Groups
df for
Error 1 2 3 4 5 6 7 8 9 10 12 15 20 30 60 ∞
1 4052 5000 5403 5625 5764 5859 5928 5982 6022 6056 6106 6157 6209 6261 6313 6366
2 98.50 99.00 99.17 99.25 99.30 99.33 99.36 99.37 99.39 99.40 99.42 99.43 99.45 99.47 99.48 99.50
5/5/06
3 34.12 30.82 29.46 28.71 28.24 27.91 27.67 27.49 27.35 27.33 27.05 26.87 26.69 26.50 26.32 26.13
4 21.20 18.00 16.69 15.98 15.52 15.21 14.98 14.80 14.66 14.55 14.37 14.20 14.02 13.84 13.65 13.46
5 16.26 13.27 12.06 11.39 10.97 10.67 10.46 10.29 10.16 10.05 9.89 9.72 9.55 9.38 9.20 9.02
6 13.75 10.92 9.78 9.15 8.75 8.47 8.26 8.10 7.98 7.87 7.72 7.56 7.40 7.23 7.06 6.88
7 12.25 9.55 8.45 7.85 7.46 7.19 6.99 6.84 6.72 6.62 6.47 6.31 6.16 5.99 5.82 5.65
5:26 PM
8 11.26 8.65 7.59 7.01 6.63 6.37 6.18 6.03 5.91 5.81 5.67 5.52 5.36 5.20 5.03 4.86
9 10.56 8.02 6.99 6.42 6.06 5.80 5.61 5.47 5.35 5.26 5.11 4.96 4.81 4.65 4.48 4.31
10 10.04 7.56 6.55 5.99 5.64 5.39 5.20 5.06 4.94 4.85 4.71 4.56 4.41 4.25 4.08 3.91
11 9.65 7.21 6.22 5.67 5.32 5.07 4.89 4.74 4.63 4.54 4.40 4.25 4.10 3.94 3.78 3.60
Page 306
12 9.33 6.93 5.95 5.41 5.06 4.82 4.64 4.50 4.39 4.30 4.16 4.01 3.86 3.70 3.54 3.36
13 9.07 6.70 5.74 5.21 4.86 4.62 4.44 4.30 4.19 4.10 3.96 3.82 3.66 3.51 3.34 3.17
14 8.86 6.51 5.56 5.04 4.69 4.46 4.28 4.14 4.03 3.94 3.80 3.66 3.51 3.35 3.18 3.00
15 8.68 6.36 5.42 4.89 4.56 4.32 4.14 4.00 3.89 3.80 3.67 3.52 3.37 3.21 3.05 2.87
16 8.53 6.23 5.29 4.77 4.44 4.20 4.03 3.89 3.78 3.69 3.55 3.41 3.26 3.10 2.93 2.75
17 8.40 6.11 5.18 4.67 4.34 4.10 3.93 3.79 3.68 3.59 3.46 3.31 3.16 3.00 2.83 2.65
18 8.29 6.01 5.09 4.58 4.25 4.01 3.84 3.71 3.60 3.51 3.37 3.23 3.08 2.92 2.75 2.57
19 8.18 5.93 5.01 4.50 4.17 3.94 3.77 3.63 3.52 3.43 3.30 3.15 3.00 2.84 2.67 2.49
20 8.10 5.85 4.94 4.43 4.10 3.87 3.70 3.56 3.46 3.37 3.23 3.09 2.94 2.78 2.61 2.42
21 8.02 5.78 4.87 4.37 4.04 3.81 3.64 3.51 3.40 3.31 3.17 3.03 2.88 2.72 2.55 2.36
22 7.95 5.72 4.82 4.31 3.99 3.76 3.59 3.45 3.35 3.26 3.12 2.98 2.83 2.67 2.50 2.31
23 7.88 5.66 4.76 4.26 3.94 3.71 3.54 3.41 3.30 3.21 3.07 2.93 2.78 2.62 2.45 2.26
24 7.82 5.61 4.72 4.22 3.90 3.67 3.50 3.36 3.26 3.17 3.03 2.89 2.74 2.58 2.40 2.21
25 7.77 5.57 4.68 4.18 3.85 3.63 3.46 3.32 3.22 3.13 2.99 2.85 2.70 2.54 2.36 2.17
26 7.72 5.53 4.64 4.14 3.82 3.59 3.42 3.29 3.18 3.09 2.96 2.81 2.66 2.50 2.33 2.13
27 7.68 5.49 4.60 4.11 3.78 3.56 3.39 3.26 3.15 3.06 2.93 2.78 2.63 2.47 2.29 2.10
28 7.64 5.45 4.57 4.07 3.75 3.53 3.36 3.23 3.12 3.03 2.90 2.75 2.60 2.44 2.26 2.06
29 7.60 5.42 4.54 4.04 3.73 3.50 3.33 3.20 3.09 3.00 2.87 2.73 2.57 2.41 2.23 2.03
30 7.56 5.39 4.51 4.02 3.70 3.47 3.30 3.17 3.07 2.98 2.84 2.70 2.55 2.39 2.21 2.01
40 7.31 5.18 4.31 3.83 3.51 3.29 3.12 2.99 2.89 2.80 2.66 2.52 2.37 2.20 2.02 1.80
60 7.08 4.98 4.13 3.65 3.34 3.12 2.95 2.82 2.72 2.63 2.50 2.35 2.20 2.03 1.84 1.60
120 6.85 4.79 3.95 3.48 3.17 2.96 2.79 2.66 2.56 2.47 2.34 2.19 2.03 1.86 1.66 1.38
∞ 6.63 4.61 3.78 3.32 3.02 2.80 2.64 2.51 2.41 2.32 2.18 2.04 1.88 1.70 1.47 1.00
Adapted from Table 18 in Pearson and Hartley (Eds.) (1970). Biometrika Tables for Statisticians. New York: Cambridge University Press, with permission of Biometrika Trustees.
Table F Critical Values of r

df 1 .05 .025 .01 .005 .0005
1 .988 .997 .9995 .9999 .9999
2 .900 .950 .980 .990 .999
3 .805 .878 .934 .959 .991
4 .729 .811 .882 .917 .974
5 .669 .755 .833 .875 .951
6 .622 .707 .789 .834 .925
7 .582 .666 .750 .798 .898
8 .549 .632 .716 .765 .872
9 .521 .602 .685 .735 .847
10 .497 .576 .658 .708 .823
11 .476 .553 .634 .684 .801
12 .458 .532 .612 .661 .780
13 .441 .514 .592 .641 .760
14 .426 .497 .574 .623 .742
15 .412 .482 .558 .606 .725
16 .400 .468 .543 .590 .708
17 .389 .456 .529 .575 .693
18 .378 .444 .516 .561 .679
19 .369 .433 .503 .549 .665
20 .323 .381 .445 .487 .597
25 .360 .423 .492 .537 .652
30 .296 .349 .409 .449 .554
35 .275 .325 .381 .418 .519
40 .257 .304 .358 .393 .490
45 .243 .288 .338 .372 .465
50 .231 .273 .322 .354 .443
60 .211 .250 .295 .325 .408
70 .195 .232 .274 .302 .380
80 .183 .217 .257 .283 .357
90 .173 .205 .242 .267 .338
100 .164 .195 .230 .254 .321
2 .10 .05 .02 .01 .001
df n 2.
Source: Adapted from Table 13 in Pearson and Hartley (Eds.) (1970). Biometrika Tables for Statisticians.
307
Table G Power of the Chi-Square Test at ⴝ .05 for df ⴝ 1

Effect size (ω)
N .10 .20 .30 .40 .50 .60 .70 .80 .90
10 .062 .097 .158 .244 .353 .475 .600 .716 .812

15 .067 .117 .213 .341 .491 .642 .774 .873 .937
20 .073 .146 .269 .432 .609 .765 .879 .947 .981
25 .078 .170 .323 .516 .705 .851 .938 .979 .995
30 .085 .195 .376 .591 .782 .908 .970 .992 .999
35 .091 .220 .427 .658 .841 .944 .985 .997
40 .096 .244 .475 .716 .885 .967 .993 .999
45 .102 .269 .521 .765 .9184 .981 .997
50 .108 .293 .564 .807 .942 .988 .999
60 .121 .341 .642 .873 .972 .996
70 .133 .387 .709 .917 .9869 .999
80 .145 .432 .765 .947 .994
90 .157 .475 .812 .967 .9973
100 .170 .516 .851 .979 .999
120 .194 .591 .907 .9923
140 .219 .658 .944 .997
160 .243 .716 .967 .999
180 .268 .765 .981
200 .293 .807 .990
300 .410 .934
Power of the Chi-Square Test at = .05 for df 2

Effect size (ω)
N .10 .20 .30 .40 .50 .60 .70 .80 .90
10 .058 .081 .124 .188 .274 .378 .495 .614 .723

15 .061 .098 .164 .264 .392 .537 .678 .799 .887
20 .065 .115 .207 .341 .504 .669 .807 .904 .959
25 .069 .133 .250 .415 .603 .771 .890 .957 .986
30 .073 .151 .293 .487 .688 .846 .940 .982 .996
35 .077 .169 .336 .553 .758 .899 .968 .993 .999
40 .081 .188 .378 .614 .815 .935 .984 .997
45 .085 .207 .420 .669 .861 .959 .992 .999
50 .090 .226 .46 .718 .896 .975 .996
60 .098 .264 .537 .79 .944 .991 .999
70 .107 .302 .606 .859 .971 .997
80 .115 .341 .669 .904 .985 .999
90 .124 .378 .723 .935 .993
100 .133 .415 .771 .957 .997
120 .151 .487 .846 .982 .999
140 .169 .553 .899 .993
160 .188 .614 .935 .997
180 .207 .669 .959 .999
200 .226 .718 .975
300 .322 .883 .998
308
Power of the Chi-Square Test at α = .05 for df = 3

Effect size (ω)
N .10 .20 .30 .40 .50 .60 .70 .80 .90
10 .056 .075 .109 .161 .233 .325 .433 .548 .660

15 .059 .088 .142 .225 .338 .472 .613 .742 .845
20 .062 .102 .176 .291 .441 .603 .752 .865 .937
25 .065 .116 .213 .359 .537 .711 .848 .934 .977
30 .068 .130 .250 .424 .623 .796 .911 .970 .992
35 .071 .145 .287 .488 .698 .859 .950 .987 .997
40 .075 .161 .325 .548 .761 .905 .972 .994 .999
45 .078 .176 .363 .603 .814 .937 .985 .998
50 .081 .192 .400 .654 .856 .959 .992 .999
60 .088 .225 .472 .742 .917 .983 .998
70 .095 .258 .540 .812 .954 .994
80 .102 .291 .603 .865 .975 .998
90 .109 .325 .660 .905 .987 .999
100 .116 .359 .711 .934 .993
120 .130 .424 .796 .970 .998
140 .145 .488 .859 .987
160 .161 .548 .905 .994
180 .176 .603 .937 .998
200 .192 .654 .959 .999
300 .275 .840 .996
Power of the Chi-Square Test at α = .05 for df = 4

Effect size (ω)
N .10 .20 .30 .40 .50 .60 .70 .80 .90
10 .055 .071 .099 .144 .207 .290 .389 .499 .611

15 .058 .082 .128 .200 .301 .426 .563 .696 .808
20 .060 .093 .158 .259 .396 .553 .706 .831 .916
25 .063 .106 .189 .320 .488 .664 .812 .912 .966
30 .065 .118 .222 .381 .573 .754 .884 .957 .987
35 .068 .131 .255 .441 .649 .824 .931 .980 .996
40 .071 .144 .290 .499 .716 .877 .961 .991 .999
45 .073 .158 .324 .553 .773 .916 .978 .996
50 .076 .172 .358 .605 .820 .943 .988 .998
60 .082 .200 .426 .696 .891 .975 .997
70 .088 .229 .492 .771 .936 .990 .999
80 .093 .259 .553 .831 .964 .996
90 .099 .290 .611 .877 .980 .999
100 .106 .320 .664 .912 .989
120 .118 .381 .754 .957 .997
140 .131 .441 .824 .980
160 .144 .499 .877 .991
180 .158 .553 .916 .996
200 .172 .605 .943 .998
300 .244 .802 .994
Source: For permission to reprint, contact T & T Pacific Communications, Buffalo, NY or the author of this
Appendix, Machiko R. Tomita.
309
Table H Power of the t-Test for α1 = .05

Effect size (d)
n .10 .20 .30 .40 .50 .60 .70 .80 1.00 1.20 1.40
8 .073 .103 .142 .189 .244 .308 .377 .451 .601 .738 .845
9 .075 .108 .150 .203 .265 .335 .412 .492 .650 .785 .884
10 .076 .112 .159 .217 .285 .362 .445 .530 .694 .825 .914
11 .078 .117 .167 .230 .304 .388 .476 .566 .732 .858 .936
12 .080 .121 .176 .243 .324 .413 .507 .600 .767 .885 .953
13 .081 .125 .184 .257 .342 .437 .536 .632 .797 .908 .966
14 .083 .129 .192 .270 .361 .461 .563 .661 .824 .926 .975
15 .084 .133 .200 .282 .379 .483 .589 .689 .848 .941 .982
16 .086 .137 .207 .295 .396 .505 .614 .714 .868 .953 .987
17 .087 .141 .215 .307 .414 .527 .638 .738 .886 .962 .991
18 .088 .145 .223 .320 .431 .547 .660 .760 .902 .970 .993
19 .090 .149 .230 .332 .447 .567 .681 .781 .916 .976 .995
20 .091 .153 .238 .344 .463 .587 .702 .799 .928 .981 .997
30 .103 .190 .310 .455 .606 .743 .850 .922 .985 .998 1.000
40 .115 .224 .376 .551 .716 .845 .928 .971 .997 1.000 1.000
50 .125 .257 .438 .634 .799 .909 .966 .990 1.000 1.000 1.000
100 .174 .407 .681 .880 .970 .995 .999 1.000 1.000 1.000 1.000
200 .259 .637 .911 .991 1.000 1.000 1.000 1.000 1.000 1.000 1.000
Power of the t-Test for α2 = .05

Effect size (d)
n .10 .20 .30 .40 .50 .60 .70 .80 1.00 1.20 1.40
8 .0532 .0631 .0803 .1054 .1393 .1828 .2360 .2983 .4440 .5991 .7386
9 .0538 .0655 .0857 .1154 .1555 .2066 .2684 .3398 .5009 .6621 .7961
10 .0544 .0678 .0912 .1255 .1717 .2302 .3003 .3798 .5532 .7163 .8416
11 .0550 .0702 .0967 .1356 .1879 .2537 .3315 .4184 .6011 .7626 .8774
12 .0555 .0726 .1022 .1458 .2041 .2769 .3619 .4553 .6447 .8020 .9055
13 .0561 .0750 .1078 .1560 .2202 .2998 .3916 .4906 .6842 .8353 .9274
14 .0567 .0774 .1133 .1662 .2363 .3224 .4205 .5243 .7200 .8634 .9444
15 .0573 .0797 .1189 .1763 .2523 .3447 .4484 .5563 .7522 .8870 .9575
16 .0579 .0821 .1245 .1865 .2682 .3666 .4755 .5866 .7810 .9067 .9676
17 .0584 .0845 .1301 .1967 .2840 .3881 .5017 .6183 .8069 .9232 .9754
18 .0590 .0869 .1357 .2069 .2996 .4091 .5269 .6424 .8300 .9369 .9814
19 .0596 .0893 .1413 .2171 .3151 .4298 .5512 .6679 .8506 .9487 .9859
20 .0602 .0918 .1470 .2272 .3305 .4500 .5746 .6919 .8689 .9577 .9894
30 .0660 .1161 .2035 .3267 .4742 .6258 .7595 .8613 .9668 .9948 .9994
40 .0719 .1407 .2597 .4204 .5966 .7546 .8710 .9417 .9924 .9994 1.000
50 .0778 .1654 .3149 .5062 .6963 .8438 .9336 .9767 .9984 .9999 1.000
100 .1066 .2888 .5594 .8041 .9410 .9884 .9985 .9999 1.000 1.000 1.000
200 .1685 .5136 .8494 .9790 .9988 1.000 1.000 1.000 1.000 1.000 1.000
Source: For permission to reprint, contact T & T Pacific Communications, Buffalo, NY or the author of this
Appendix, Machiko R. Tomita.
310
Table I Sample Sizes Needed for the ANOVA at = .05 and Power = .80
Effect size (f )
df 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
1 1571 394 176 100 64 45 34 26 21 17 15 12 11 10 9 8 7 6 6
5/5/06
2 1286 323 144 82 53 37 28 22 17 14 12 10 9 8 7 7 6 6 5
3 1092 274 123 70 45 32 24 19 15 12 11 9 8 7 6 6 5 5 5
4 956 240 108 61 40 28 21 16 13 11 9 8 7 6 6 5 5 5 4
5:26 PM
5 857 215 96 55 36 25 19 15 12 10 9 7 7 6 5 5 4 4 4
6 780 196 88 50 33 23 17 14 11 9 8 7 6 5 5 5 4 4 4
7 719 181 81 46 30 21 16 13 10 9 7 6 6 5 5 4 4 4 3
8 669 168 76 43 28 20 15 12 10 8 7 6 5 5 4 4 4 4 3
Page 311
9 627 158 71 40 26 19 14 11 9 8 7 6 5 5 4 4 4 3 3
10 592 149 67 38 25 18 13 11 9 7 6 5 5 4 4 4 3 3 3
Source: For permission to reprint, contact T & T Pacific Communications, Buffalo, NY or the author of this Appendix, Machiko R. Tomita.
Table J Sample Sizes Needed for Correlation at 1 .05 and Power .80
Effect size (f )
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
2472.897 618.116 274.618 154.371 98.688 68.413 50.130 38.233 30.047 24.159 19.769 16.394 13.729 11.572 9.783 8.261 6.925 5.696 4.459
Sample Sizes Needed for Correlation at 2 .05 and Power .80
Effect size (f )
0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95
3138.315 783.639 347.570 194.925 124.248 85.831 62.639 47.559 37.190 29.741 24.197 19.943 16.592 13.888 11.655 9.762 8.110 6.603 5.101
311
Table K Power of the F-Test for Regression Analysis at α = .05
312
Lambda (λ)
K dfres 2 4 6 8 10 12 14 16 18 20 24 28 32 36 40
1 20 .268 .473 .640 .763 .848 .906 .942 .966 .980 .988 .996 .999 1.00 1.00 1.00
50 .283 .500 .670 .792 .873 .924 .956 .975 .986 .992 .998 .999 1.00 1.00 1.00
5/5/06
150 .290 .511 .682 .802 .881 .931 .961 .978 .988 .994 .998 1.00 1.00 1.00 1.00
2 20 .195 .355 .506 .634 .737 .816 .873 .915 .944 .963 .985 .994 .998 .999 1.00
50 .214 .393 .555 .688 .788 .861 .911 .944 .966 .979 .993 .998 .999 1.00 1.00
150 .222 .408 .575 .708 .807 .876 .923 .953 .972 .984 .995 .998 1.00 1.00 1.00
5:26 PM
3 20 .159 .288 .418 .589 .644 .732 .801 .856 .897 .927 .965 .984 .993 .997 .999
50 .179 .332 .481 .613 .721 .805 .867 .911 .942 .962 .985 .994 .998 .999 1.00
150 .188 .350 .506 .641 .749 .829 .887 .927 .954 .972 .990 .996 .999 1.00 1.00
Page 312
4 20 .137 .242 .354 .464 .565 .653 .729 .791 .842 .881 .936 .966 .983 .992 .996
50 .158 .290 .427 .554 .664 .754 .824 .877 .916 .943 .975 .990 .996 .999 .999
150 .167 .310 .456 .589 .700 .788 .854 .902 .935 .958 .983 .994 .998 .999 1.00
5 20 .121 .209 .304 .402 .495 .581 .658 .725 .781 .828 .897 .940 .966 .982 .990
50 .143 .260 .385 .506 .615 .707 .783 .842 .888 .921 .963 .984 .993 .997 .999
150 .152 .281 .417 .546 .658 .750 .822 .876 .916 .944 .976 .991 .996 .999 1.00
10 20 .081 .118 .160 .205 .253 .302 .352 .401 .450 .500 .585 .663 .730 .787 .833
50 .105 .177 .259 .348 .438 .525 .605 .677 .740 .794 .876 .928 .960 .779 .989
150 .116 .202 .302 .407 .510 .605 .689 .760 .818 .865 .929 .965 .984 .993 .997
15 20 .062 .075 .088 .102 .116 .131 .146 .161 .177 .192 .224 .257 .289 .321 .353
50 .089 .138 .195 .260 .328 .398 .468 .536 .599 .658 .758 .835 .892 .931 .957
150 .100 .165 .243 .329 .418 .505 .588 .663 .729 .785 .872 .927 .961 .980 .990
20 20 ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____ ____
50 .079 .114 .156 .202 .253 .307 .363 .419 .474 .528 .629 .716 .789 .846 .891
150 .090 .143 .207 .278 .355 .433 .509 .582 .650 .711 .810 .882 .929 .960 .978
Table L Probabilities Associated with Values of x in the Binomial Test
x
n 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
4 .062 .312 .688 .938 — — — — — — — — — — — — —

5 .031 .188 .500 .812 .969 — — — — — — — — — — — —
5/5/06
6 .016 .109 .344 .656 .891 .984 — — — — — — — — — — —
7 .008 .062 .227 .500 .773 .938 .992 — — — — — — — — — —
8 .004 .035 .145 .363 .637 .855 .965 .996 — — — — — — — — —
9 .002 .020 .090 .254 .500 .746 .910 .980 .998 — — — — — — — —
5:26 PM
10 .001 .011 .055 .172 .377 .623 .828 .945 .989 .999 — — — — — — —
11 — .006 .033 .113 .274 .500 .726 .887 .967 .994 — — — — — — —

12 — .003 .019 .073 .194 .387 .613 .806 .927 .981 .997 — — — — — —
Page 313
13 — .002 .011 .046 .133 .291 .500 .709 .867 .954 .989 .998 — — — — —
14 — .001 .006 .029 .090 .212 .395 .605 .788 .910 .971 .994 .999 — — — —
15 — — .004 .018 .059 .151 .304 .500 .696 .849 .941 .982 .996 — — — —
16 — — .002 .011 .038 .105 .227 .402 .598 .773 .895 .962 .989 .998 — — —
17 — — .001 .006 .025 .072 .166 .315 .500 .685 .834 .928 .975 .994 .999 — —
18 — — .001 .004 .015 .048 .119 .240 .407 .593 .760 .881 .952 .985 .996 .999 —
19 — — — .002 .010 .032 .084 .180 .324 .500 .676 .850 .916 .968 .990 .998 —
20 — — — .001 .006 .021 .058 .132 .252 .412 .588 .748 .868 .942 .979 .994 .999
21 — — — .001 .004 .013 .039 .095 .192 .332 .500 .668 .808 .902 .961 .987 .996
22 — — — — .002 .008 .026 .067 .143 .262 .416 .584 .738 .857 .933 .974 .992
23 — — — — .001 .005 .017 .047 .105 .202 .339 .500 .661 .798 .895 .953 .983
24 — — — — .001 .003 .011 .032 .076 .154 .271 .419 .581 .729 .846 .924 .968
25 — — — — — .002 .007 .022 .054 .115 .212 .345 .500 .655 .788 .885 .946
26 — — — — — .001 .005 .014 .038 .084 .163 .279 .423 .577 .721 .837 .916
27 — — — — — .001 .003 .010 .026 .061 .124 .221 .351 .500 .649 .779 .876
28 — — — — — — .002 .006 .018 .044 .092 .172 .286 .425 .575 .714 .828
29 — — — — — — .001 .004 .012 .031 .068 .132 .229 .356 .500 .644 .771
30 — — — — — — .001 .003 .008 .021 .049 .100 .181 .292 .428 .572 .708
313
Tabled probabilities are for one-tailed tests. Double values in table for a two-tailed test.
Adapted from Table D in Siegel S, Castellan NJ: Nonparametric Statistics for the Behavioral Sciences (2nd ed.). New York: McGraw Hill, 1988, with permission.
18Kielhofner(F)-App-B 5/5/06 3:53 PM Page 314
A P P E N D I X B
Summary of Common Research Statistics

Gary Kielhofner • Machiko R. Tomita
• Analysis of Covariance (ANCOVA)

• Chi-Square
• Cronbach’s Coefficient Alpha ()
• Factor Analysis (Exploratory Factor Analysis)
• Friedman Test
• Hierarchical Linear Models (HLM)
• Independent t-Test
• Intraclass Correlation Coefficient
• Kappa
• Kruskal-Wallis (One-Way ANOVA)
• Logistic Regression
• Mann-Whitney U-test (also called Wilcoxon Rank-Sum Test)
• McNemar Test
• Multifactor Analysis (Two-Way ANOVA, Three-Way ANOVA)
• Multivariate Analysis of Variance (MANOVA)
• Multiple Regression
• One-Way Analysis of Variance (ANOVA)
• Paired t-Test
• Path Analysis
• Pearson Product Moment Correlation
• Rasch Analysis
• Repeated Measures (ANOVA)
• Sign Test
• Simple Linear Regression
• Spearman Rank Correlation Coefficient
• Wilcoxon Signed Rank Test
18Kielhofner(F)-App-B
Purpose of the Symbols and Numeric Things to Consider When Where the Statistic
Statistic and How Values Reported for this Using/Drawing Conclusions is Discussed in
Statistic It Analyzes Data Statistic and What They Mean from this Statistic this Text
Analysis of Covariance Tests the main and F, which is the test of Assumptions include: Chapter 17, Section on
(ANCOVA) interaction effects of significance of the main and • Interval or ratio level dependent variable Analysis of Covariance
5/5/06
categorical variables on a interaction effect, of a single • Limited number of covariates
continuous dependent interval dependent and one • Low measurement error of the covariate
variable, controlling for or more independent • Covariates linearly related
the effects of other variables. • Homogeneity of covariate regression
variables. If p-value is smaller than the set coefficients
3:53 PM
level, at least two group • No covariate outliers
means are significantly • No high multicollinearity of the covariates
different after adjusted to • Independence of the error term
covariates. • Independent variables orthogonal to
Page 315
covariates
• Homogeneity of variances
• Normal distribution within groups
Chi-Square Two purposes: 2, which is the sum of the Chapter 17,
squared difference in Section on Nonparametric
observed and expected Statistics
scores over the expected
score. This concept applies
for both types of Chi-square.
• Goodness-of-fit-test: Goodness-of-fit-test: If the Goodness-of-fit test compares observed
Identifies whether one p-value is smaller than the frequency with an expected (uniform or
sample distribution set level, the observed normal distribution) or known distribution.
(frequencies) differs from frequencies are different from
an expected/known the expected/known
distribution. frequencies.
• Test of independence: Test of independence: if the Tests of independence are used to examine
Measures whether two p-value is smaller than the the independence of two categorical
categorical variables have set level, the two variables variables.
a relationship. are not independent or they
are associated.
315
316
Cronbach’s Coefficient Asks whether the items that , which like any correlation Alpha is the average of all split half reliabilities Chapter 12, Section on
Alpha () make up a scale are coefficient ranges from 0.0 to for the items that make up the instrument. Internal Consistency
homogeneous. 1.0 and indicates the strength Alpha values that approach .90 are indications
of relationship; in this case it of high homogeneity.
5/5/06
indicates the extent to which Alpha is affected by the number of items; so,
all items are interrelated or longer scales will tend to generate higher
constitute a homogeneous coefficients
instrument
3:53 PM
Factor Analysis (Exploratory Reduces items into fewer Eigenvalues, which reflect the There are two major methods for factor Chapter 17, Section on
Factor Analysis) sets of the similar amount of variance analysis: Multivariate Analysis
concepts (i.e., factors). accounted for by each factor. • Factor analysis, and
Factor loadings, which represent • Principal component analysis.
Page 316
the correlation between each Exploratory factor analysis should not be
item and each factor. confused with confirmatory factor analysis,
Rotations are done to identify which is an application of structural
meaningful factors that equation modeling (SEM).
include highly correlated Usually the factor loading of .30 or less is not
items to the factor. meaningful.
Friedman Test Compares the rank orders r2 which reflects the difference This test is a nonparametric alternative to the Chapter 17, Section on
taken more than two among three or more group repeated measures ANOVA. The raw data Comparison of More
times for correlated rank orders. are converted into rank orders. than Two Group Means
samples. If the p-value is smaller than the
set level, at least one pair
of rank orders is significantly
different.
A post-hoc test will follow to
identify which pairs of mean
ranks are significant.
Hierarchical Linear Models Examine the growth Symbols are not always the Users must be cautious in setting up models Chapter 17
(HLM) trajectories (i.e., change same across the published as well as interpreting the results.
5/5/06
in dependent variable literature. Second level sample size has influence on the
over time) and study Interpretation of intercepts and power of the test.
factors that may influence regression coefficients follow
these trajectories. the same principles used in
Nested type of data will be
3:53 PM
regression analysis.
analyzed in two levels:
• First-level data will use
time as predictor variable
to estimate individual
Page 317
growth trajectories. The
estimated intercepts and
regression coefficients can
be used as dependent
variables in the second
model.
• Second-level data can be
modeled and uses the
individual as the unit of
analysis.
Independent t-Test Compares two group means t, which reflects the difference The t-test is preceded by a test for Chapter 17, Section on
when the samples are between the two group homogeneity (either Bartlett’s test or Comparisons of Two
independent. means divided by the Levine’s test) and if the two groups are not Group Means
standard error of the homogeneous, the formula for the t-test
difference. must be adjusted; this is done automatically
If the p-value is smaller than the in many statistical packages.
set , the two group means Used for either a directional hypothesis or a
are significantly different. nondirectional hypothesis but is adjusted
accordingly.
Intraclass Correlation Calculates the reliability of ICC, which is a correlation Estimates reliability while taking into Chapter 12, Section on
Coefficient an observed score in ranging from 0.0–1.0 consideration error due to several factors Generalizability Theory
estimating the true score. such as variations in the raters and testing and Intraclass
conditions, alternate forms, and Coefficients
administration at different times.
317
318
Kappa Examines extent of rater K, which indicates strength of Guidelines for interpretation of kappa values: Chapter 12, Section on
agreement with relationship; values range • 0.75 excellent agreement Assessing Inter-rater
categorical instruments; from 0.0 to 1.0 • 0.40–0.75 fair to good agreement Reliability
attenuates for chance • 0.40 poor agreement
5/5/06
agreement. Is influenced by sample size, subject variability
and the number of categories used; must
be interpreted with care.
3:53 PM
Kruskal-Wallis (One-Way Compares rank orders H, which is distributed as Chi- This test is a nonparametric alternative to the Chapter 17, Section on
ANOVA) among three or more square with df k – 1. one-way ANOVA. Used when dependent Comparison of More
groups for independent If the p-value is smaller than the variable is measured with an ordinal scale, than Two Group Means
samples. set level, at least two group or when assumptions are not met if
means are significantly measured with an interval/ratio scale.
Page 318
different. Then, a post-hoc
test to identify which pairs of
means are significantly
different will follow.
Logistic Regression Examines the effect of Z, which is the logarithm of the Logistic regression estimates the maximum Chapter 17, Section on
several independent odds. The odds ratio is how likelihood that results in the accurate Logistic Regression
variables (categorical or likely it is that an individual prediction of group membership.
continuous) on a belongs to the target group Need fairly large number of cases, and avoid
categorical dependent rather than the reference multicollinearity and outliers.
variable. group. There are direct and sequential logistic
If the p-value for each regression regressions.
coefficient is smaller than the
set level, the relationship
between the two variables is
not likely to be due to
chance.
Classification accuracy indicates
the rate of accurately
predicted frequencies to
actual observed frequencies.
Mann-Whitney U-test (also Measures difference of rank U score for each group is Used in place of the independent t-test when Chapter 17, Section on
called Wilcoxon Rank- orders between two calculated and the smaller of the data do not conform to all or most of Comparisons of Two
5/5/06
Sum Test) groups. the two values (i.e., calcu- the normality assumptions required for Group Means
lated U value) is compared to parametric statistics.
a critical value. When the If the dependent variable is measured on a
calculated value is smaller ratio or interval scale, the raw scores have
to be converted to rank orders across both
3:53 PM
than the critical value for
the set level, it is statisti- groups.
cally significant (i.e., p-value
is smaller than the set
level).
Page 319
A p-value smaller than the set
level indicates a significant
difference in rank orders
between the two groups.
McNemar Test Compares correlated 2, which reflects the This test is used when: Chapter 17, Section on
frequency distributions independence of frequencies. • The samples are correlated, and Nonparametric
(i.e., pre- and post- If the p-value is smaller than • The dependent variable has only two levels, Statistics
measures) to see if there the set level, there is a such as Yes and No.
is a significant difference. significant change or
difference.
Multifactor Analysis Tests the main effects (of F, which is the ratio of between- Used for independent samples. Chapter 17, Section on
(Two-Way ANOVA, more than one subjects variance to within- To interpret the results, the interaction effects Multifactorial Analysis of
Three-Way ANOVA, etc.) independent variable) subjects variance. should be analyzed first. If significant, the Variance
and interaction effects If the p-value is smaller than the type of interaction should be identified
(among independent set level, the independent (ordinal vs. disordinal). If the interaction is
variables) on the variable or interaction has disordinal, the variables causing the
dependent variable. significant effects on the interaction should be examined separately.
dependent variable.
Post-hoc analysis will follow to
identify which independent
variables/interaction account
for differences.
319
320
Multivariate Analysis of Tests whether there is an F for multivariate test which Assumptions: Chapter 17, Section on
Variance (MANOVA) overall effect of the reflects the level of • Multivariate normal distribution, Multivariate Analysis
independent variable(s) association among all • Linear relationships among all pairs of
on all dependent dependent variables and the dependent variables, and
5/5/06
variables together. main effect(s) and interaction • Homogeneity of variances and regression.
effect(s). • Multicollinearity and outliers should be
F for between-subjects tests avoided.
which reflects the level of
3:53 PM
association between each of
the dependent variables and
each of the independent
variables and interaction
Page 320
effects.
If the p-value is smaller than the
set level, the effect of
independent variable or
interaction is significant.
The post-hoc analysis is
performed by either one-way
ANOVA or Discriminant
Analysis.
Multiple Regression Examines ability of several R 2, which is the strength of the Assumptions: Chapter 17, Section on
independent variables to association between the • Same as for simple linear regression, Regression
predict the value of a dependent variable and all • No multicollinearity (independent variables
dependent variable. independent variables in the are highly correlated) or outliers.
model. There are three major methods of selecting
F, which reflects the ratio variables for multiple regression:
between the variance (Mean • Standard (all independent variables entered
Square) of predicted values at once),
and the variance (Mean • Hierarchical (independent variables entered
Square) of error. one at a time or in subsets based on a
If the p-value is smaller than the theoretical rationale) and
set level, it tells that the • Stepwise method.
relationship between the two Usually the smallest group of variables that will
variables is not likely to be account for the greatest proportion of vari-
the result of chance. ance in the dependent variable is the goal.
One-Way Analysis of Compares group dependent F, which is the ratio of between- Assumes that the samples are randomly drawn Chapter 17, Section on
Variance (ANOVA) variable means for three group variance to within-group from a normally distributed population with Comparisons of More
5/5/06
or more groups variance. equal variances among the groups. If the than Two Group Means
representing different If the p-value is smaller than the set statistical assumptions are not met, then
levels of a single level, it means at least two Kruskal-Wallis nonparametric test should
independent variable. group means are significantly be used.
3:53 PM
different. Then, a post-hoc
multiple comparison test will
follow to determine differences
between means.
Paired t-Test Compares two group means t, which is obtained by dividing the The paired t-test assumes a significant Chapter 17, Section on
Page 321
when the samples are mean of the difference in scores correlation between two dependent Comparisons of Two
dependent/correlated by the standard error of the variables; if they are not correlated, an Group Means
(e.g., when comparing difference in scores. independent t-test may be used.
pre- and posttests on the If the p-value is smaller than the set
same sample). level, there is a significant
change or difference.
Path Analysis A diagram-based statistical Direct and indirect effects. Direct Assumptions: Chapter 17, Section on
procedure used to test effects are drawn using straight • The dependent variable and the independent Multivariate Analysis
the strength and direction lines connecting two variables variable should be correlated,
of relationships among and indicating the cause and • The dependent variable should occur later
several variables as well effect by an arrowhead. Indirect than the independent variable in time, and
as identifying direct and effects are indicated by lines not • The relationship between the two variables
causal pathways among connecting two variables. All exists even in the presence of another
the variables. variables are endogenous (ones variable.
that are influenced by other
variables) and/or exogenous
(ones that are independent of any
influence).
r, which indicates a correlation
between two variables.
p y, x, which indicates a path
coefficient from the variable x to
the variable y.
If the p-value is smaller than the set
321
level, the path will be exhibited.

322
Pearson Product Moment Measures the strength of the r, which represents the strength The r values are generally interpreted as Chapter 17, Section on
Correlation association between two of association, and can range follows: Correlation
continuous variables. from 0 (no association) to • 0–.2 negligible correlation
1.0 (perfect association), • .2–.4 low
5/5/06
and can be positive • .4–.6 moderate
(indicating a parallel • .6–.8 high
relationship) or negative • .8–1.0 very high
(indicating an inverse The p-values for this statistic are sensitive to
3:53 PM
relationship). sample sizes.
p-value which indicates the
likelihood the observed
relationship occurred by
Page 322
chance
Rasch Analysis Answers the following Mean Square Fit Statistic A sample of 30 is considered a minimum. Chapter 13
questions: (MnSq): This statistic does not require random
• Do items on an instrument • 1.0 is the desired/ideal mean sampling.
fit or conform to an square; scores above 1.0 When multiple raters are examined, they must
underlying unidimensional indicate proportion of error or be linked via a common subject or subjects.
construct? noise (unexpected variability), Ordinary norm for interpreting the Mean
• Does a subject’s pattern of e.g., 1.2 20% error or noise. Square (MnSq) is that values between .7
scores fit with the • Scores lower than 1.0 indicate and 1.4 are considered acceptable for a
underlying unidimensional less variation than expected. clinical assessment.
construct? • Standardized Mean Square MnSq 1.4 with a Zstd 2.0 means:
• Do raters use the (Zstd) significance of MnSq; • Item is either poorly defined or doesn’t fit
instrument in a way that Zstd 2.0 .05 level of (belong) to the same construct as the other
reflects the underlying significance Zstd 2.0 items.
unidimensional construct? indicates .05 level of • Subject’s pattern of scores was unexpected
(This question is not asked significance. (may mean subject was not validly
when the took is a self- Separation index value indicates: assessed).
report, since there is no • The number of strata into • Rater’s pattern of scores was unexpected
separate rater or which subjects were separated (indicates rater is not using the scale in a
assessor.) by the instrument. valid way).
• The number of strata into MnSq 0.7 with a Zstd 2.0 means:
which items on the scale are • Item is too invariant, doesn’t add much
separated. information about the true ability/
characteristic of the subjects.
Rasch Analysis (continued) • How well spread out are • The number of strata into • Item may overlap with another item.
the items? (i.e. what range which raters are separated. • Subject tended to get same scores on all
5/5/06
of the characteristic do Note: also gives a coefficient items irrespective of how much of the trait the
they measure?) that indicates the reliability item represented.
• How many groups does the with which items, subjects • Rater tended to give same ratings to subjects
instrument separate and raters are divided into irrespective of where the subject fell on the
3:53 PM
subjects into? (i.e., how their strata scale.
sensitive is it?) Item, person, and rater
• How much of the calibration: usually given in
characteristic measured is positive and negative
represented by an item? numbers called logits; this is
Page 323
(Where does this item fall an interval level measure
on a continuum of less to created through log
more of the construct?) transformation; hence logits
• How much of this trait does refers to “log linear units.” The
a given person have? meaning of the numbers is
• How much difference is specific to the assessment
there in the severity/ being studied. (Note
leniency of raters? sometimes authors convert
the logits to a 1–100 scale to
make it easier to deal with
the logit values.)
Interpreted as:
• Where the item falls on the
continuum from less to more of
the trait measured.
• Where this person falls on the
continuum of less to more of
the trait measured.
• How severe or lenient a rater is
compared to other raters.
323
324
Repeated Measures Compares changes or F, which is the ratio between Since the sample is correlated, the statistical Chapter 17, Section on
(ANOVA) differences among three Mean Square of between- assumption of homogeneity among groups Comparison of More
or more measures or factors (group variance) is not applicable. However, the similarity than Two Group Means
groups in correlated and Mean Square of within- and correlation among variance of change
5/5/06
samples. factors (error variance). scores should be tested. This is the
If the p-values is smaller than assumption of sphericity.
the set level, at least two
group measures are
3:53 PM
significantly different or
changes in at least two
measures are significantly
different. Then, identify which
Page 324
pairs of means are
significantly different.
Sign Test Tests whether there are Z, which reflects the ratio of the Used when changes in the direction of the data Chapter 17, Section on
significant changes in difference between the are of primary interest (e.g., data changing Comparisons of Two
directions (positive or number of fewer changes in either a positive or a negative direction). Group Means
negative) of a dependent and a half of the sample size This statistic provides the lowest level of
variable. to a half of the square root of statistical information. If possible, Wilcoxon
the sample size. signed rank test should be used.
The probability (p-value) of the
occurrence of the result is
derived from the calculated z;
if the p-value is smaller than
the set level there is
significant change in
direction.
Simple Linear Regression Asks whether a R2 ( r2), which reflects the The dependent variable should be measured Chapter 17, Section on
dependent variable variance of the dependent by either interval or ratio scale. Regression
5/5/06
can be predicted from variable explained by the Assumes that two variables are linearly
an independent variance of the independent correlated.
variable. variable. Sensitive to outliers.
F, which reflects the ratio between
3:53 PM
the variance (Mean Square) of
predicted values and the
variance (Mean Square) of error.
If the p-value is smaller than the set
level, the relationship between
Page 325
the two variables is not likely to
be the results of chance.
Spearman Rank Correlation Measures the strength of rs, which represents the strength of The rs values reflect the strength of the Chapter 17, Section on
Coefficient the association association. It ranges from –1 to relationship between two variables and are Correlation
between two ranked
1. generally interpreted as follows:
variables. • 0–.2 negligible correlation
• .2–.4 low
• .4–.6 moderate
• .6–.8 high
• .8–1.0 very high
This test is a nonparametric alternative to
Pearson correlation. Used when variables
are measured by ordinal scales.
Wilcoxon Signed Rank Test Identifies the relative T, which is the smaller sum of the This is a nonparametric alternative to the
magnitude of rank of scores for one direction paired t-test when the sample is dependent. Chapter 17, Section on
differences and the of change. Used when the dependent variable is Comparisons of Two
direction of the change If the p-value is smaller than the set measured with an ordinal scale, or when Group Means
for correlated samples. level, there is a significant groups are not normally distributed if
change or difference measured with an interval/ratio scale.
325
S E C T I O N 5
Qualitative Methods
C H A P T E R 1 9
Overview of Qualitative Research

Mark R. Luborsky • Cathy Lysack
The aim of this chapter is to provide a broad others while growing up as human beings and
overview of qualitative research. It considers the during professional training (deductive processes).
nature and uses of qualitative research, discussing Investigators also keep their eyes open to see in
its place and importance in scholarship in general new ways and build up new ideas (inductive pro-
and occupational therapy in particular. This chap- cesses).
ter also introduces the major epistemologies or tra- Both approaches are needed. Scholars continu-
ditions of thought within qualitative research. ally move between studying things in terms of what
is already known to be true about the world and
studying things from a fresh point of view. The lat-
The Nature of and Need ter is especially important when received wisdoms
for Qualitative Research seem not to apply to a new situation or cannot
explain why things go poorly in certain circum-
stances.
Research is, simply, asking informed questions.
The hallmark of all research is curiosity about the
The Nature of Qualitative Research
nature and functioning of the world including its
peoples, and working to develop some generaliza- Qualitative study methods are needed when resear-
tions about this world. In this quest for understand- chers ask certain kinds of questions. Qualitative
ing, investigators make use of existing concepts research is a broad term for approaches to develop-
and knowledge including wisdoms received from ing new knowledge that have as their main goal the
Figure 19.1 Dr. Lysack reflects

upon questions arising from past
research on the experience of liv-
ing with spinal cord injury and
identifies new qualitative research
questions for future exploration.
Chapter 19 Overview of Qualitative Research 327
naturalistic discovery, identification, and descrip- ations where there is insufficient prior work. In
tion of basic features of the worlds people live in this case, qualitative research is often the first step
and their experiences of those worlds. Qualitative for investigating a particular phenomenon. In addi-
methods are used when it’s important to learn more tion, it is often the case that certain phenomena,
about the kinds of features that are present, and experiences, or processes are inadequately cap-
about what are the salient contents and meanings tured by the preconceived concepts and predefined
of a phenomenon. Simply stated, qualitative meth- tools. This is the second major reason for qualita-
ods are well suited to the task of discovering what tive research.
needs to be measured or described, and how to Many phenomena in occupational therapy are
measure it. best studied with qualitative methods. For exam-
ple, an investigator who wishes to learn about the
Questions Addressed evolution of occupational therapists’ acquisition of
by Qualitative Research professional competencies could count words or
behaviors, or collect data on constructs such as
Occupational therapy contributes outstanding independence, judgment, reasoning, and control.
questions to the wider scientific research enter- While such a strategy would answer certain ques-
prise. These include, for example, questions about tions, it would likely not capture the dynamic
the impact of disability on occupation, meaningful range of issues, dilemmas, and struggles that real
activities, and habits. As a relatively young field, occupational therapists grapple with as they seek
its scholars are continuing to develop the tools for to design and implement therapies and interven-
answering these questions. Qualitative research is tions to facilitate the recovery and well-being of
emerging as an important tool for answering many their clients. A qualitative, narrative approach
of the questions generated in the field. would be required to discover these aspects of pro-
First, qualitative methods enable discovery of fessional development. Narratives discovered in
the basic form of salient things to measure in situ- qualitative research reflect the lived experience of
therapists. They reveal layers of intention, emo-
tion, and meaning, including complex contradic-
Judge a man by his questions not his tions that fixed questions may miss. The following
answers. are some additional examples of questions that
require qualitative research.
—Voltaire
Consider the question of what factors affect
Science is the belief in the ignorance of the adherence to adaptive device use. Research shows
experts. that the answers to this question reside in the inter-
—Richard Feynman, American physicist, face between the device and the person, the social
1985 Nobel Prize winner settings of device use, and even within the person’s
ideals and expectations with respect to the per-
These two scholars, widely separated in ceived benefits afforded by the device. Research
time, embody the basic scientific stance of also shows that adaptation of environments can
active questioning and critical disbelief in taken-
dramatically alter the need for assistive devices.
for-granted or official normative descriptions
and explanations. Conducting qualitative research in naturalistic set-
Good research is determined by what and tings like people’s homes and communities has
whose questions are asked. A clearly focused been essential to discovering these factors (Gitlin,
important question that leads to new knowledge Luborsky, & Schemm, 1998; Luborsky, 1997).
(if only to dissuade us from the complacency of Consider the question of what factors affect
taken for granted ideas) is more crucial than the outcomes in stroke rehabilitation. In a series of
methods investigators use. Just labeling or nam- studies, researchers have shown that rehabilitation
ing things by applying existing knowledge is not after stroke is sometimes devalued by healthcare
research, even if it leads to useful interventions professionals, in contrast to the views of stroke
and treatments. Such an approach only applies
patients (Becker & Kaufman, 1995; Kaufman &
what is accepted as knowledge instead of chal-
lenging it. Using an established disease classifi- Becker, 1986). For patients, rehabilitation repre-
cation (“nosology”) or behavioral checklist only sented a hopeful opportunity for recovery if they
assigns a place in an already established univer- worked hard enough, which resulted in feeling let
sal framework. Discovery is needed to establish down when full recovery did not occur. On the
new knowledge; it is not enough to merely apply other hand, professionals’ views were dominated
an existing label or idea to something observed. by the idea that the potential to influence the ill-
ness trajectory is quite limited. As a result, reha-
328 Section 5 Qualitative Methods
bilitation professionals in stroke settings tend to

divide patients into two categories: rehabilitation
The Features of
candidates and geriatric care patients. This practice Qualitative Research
is founded on culturally based assumptions about
aging and notions of appropriate rehabilitation for All qualitative research is characterized by several
older people which ultimately serve to limit costs. aims. It seeks to discover:
Tham and Kielhofner (2003) studied older women
with stroke focusing on unilateral neglect. They • The insider’s (emic) view and compare it to the
found that how the women experienced neglect observer’s outsider (etic) view,
differed from professional conceptions of neglect. • Meanings, symbols, beliefs, and values in the
Consequently, the strategies that were most helpful language of the participants,
in assisting these women to manage daily life were • The multiple perspectives of persons, groups, and
not those that emanated from professional concep- organizations across the spectrum of positions in
tions of neglect treatment. Rather, as the women’s a social setting or culture, including those at the
experience of neglect changed over time, an evolv- margins of society, not just the center, and
ing set of strategies helped them reclaim and • Features of the worlds of everyday lived experi-
occupy the neglected half of the world. Tham and ence.
Kielhofner concluded that occupational therapy While these aims are common to all forms of
interventions for persons with neglect as a conse- qualitative inquiry, there are different epistemolo-
quence of stroke could become more effective by gies or traditions of thought within qualitative
systematically incorporating the kinds of strategies research. These are considered next.
identified in this study.
Consider the question of what it is like for
clients and families to live every day with the phys-
ical and social consequences of chronic illness and Major Epistemologies or
disability. Understanding what that is like from
their perspectives can provide invaluable insights.
Traditions of Thought in
Examples of qualitative research that provide such Qualitative Research
understanding are abundant. For example, some
research has centered on observations of a particu- Terms such as philosophical traditions and episte-
lar person or group, such as persons with cancer, mology may seem abstract and unrelated to daily
autism, or mental retardation (c.f., Langness & life. Yet everyone already has a philosophy and
Levine, 1986). In other instances, the researcher is already does philosophy when thinking about and
simultaneously the author and the subject of the addressing questions about life, meaning, society,
study, as in the case of Murphy’s (1987) classic, and morality. Philosophies, albeit at times uncon-
The Body Silent, which powerfully illustrates liv- sciously, underlay everyone’s approaches to think-
ing with a disabling and terminal disease. ing and acting.
Edgerton’s (1967, 1993) Cloak of Competence Qualitative research traditions are somewhat
reports research that showed how the stigma of akin to social traditions. They are gestalt ways of
mental retardation pushed individuals to hide and seeing the world; they define events, practices,
even deny their cognitive handicap in attempts to activities, ideals, and goals. Just as deeply held
pass as normal in society, and thus escape the cultural values and beliefs shape the expression of
social inspection and surveillance that accompa- peoples’ lives, the perspectives of qualitative
nied their disability. Edgerton also revealed the research traditions shape how researchers within
unexpected skills, resources, and insights of his those traditions see and act on the world in their
study participants, dispelling myths about the pub- studies.
lic’s sense of their incompetence. Finlayson’s
(2004) qualitative research illuminated the per- The Diversity of Qualitative
spective of adults with multiple sclerosis, describ-
Research Epistemologies
ing the challenges they encounter and fear as they
enter older age. These are only a few illustrations Each epistemological approach presents a particu-
of qualitative research that provided an insider’s lar philosophical stance that directs investigators to
viewpoint on illness and disability. Such research certain questions and ideas about what counts as
offers important insights that can shape and answers. Each stance involves contrasting ideas
improve occupational therapy practice. not only about how to go down the path to new
The Norms of Interest in Qualitative Research
There are two kinds of norms. Norms derived by and rehabilitation classifications are powerful and
statistical analyses of standardized measures enable the control or management of impairment,
describe the central tendency of a phenomena the fixing of serious injuries, and the prevention of
defined by researchers. In clinical practice, a disease. Nonetheless, the next era of medical sci-
patient’s test results can be normal, meaning they ence and discovery needs methods and tools to
fall within probabilistically defined normal cate- better address patients’ continuing lives within the
gories. Social norms are the expected standards social fabric of daily life in the community. For
and ideals by which people orient their actions. example, current quantitative research on the diag-
Social norms both pattern social action and are nosis of depression focuses on asking patients
used to evaluate right and wrong, and admonish about the presence, duration, and effect of feelings
deviations from the norm. These norms also of sadness and despair. To extend this knowledge
operate powerfully in the clinical situation, base, qualitative research must also be undertaken
though they are most often implicit and unac- to seek to learn about individual and group con-
knowledged. It is this latter kind of norms that cepts of depression. Such research would ask
qualitative researchers seek to discover. whether the way lay persons understand depres-
Consider the point of a master clinician when sion is different from that of researchers and pro-
he wrote, fessionals. It would seek to understand their beliefs
about the nature, causes, and natural course of
It happened the other morning on rounds, as depression. It would ask about their views of what
it often does, that while I was carefully aus- should be done about depression. No doubt such
caltating a patient’s chest, he began to ask me research will yield unexpected insights. These
a question. ‘Quiet,’ I said, ‘I can’t hear you insights will likely have direct and, perhaps, dra-
while I’m listening’ (Baron, 1985, p. 606). matic practice implications.
He tells us how trained clinicians efficiently Occupational therapy has already recognized
ignore the thinking person while listening for signs that the nosology it inherited from medicine and
of anatomical parts. When clinical practitioners rehabilitation is too limiting. Recognition of the
hear a patient’s words, they often select only those limitations of a narrow biomedical focus led Reilly
that are relevant to a medical diagnostic model of (1962), Kielhofner (2001), and others (Zemke &
norms. They typically do not attend to the social Clark, 1996) to attempt to develop a more integra-
norms, meanings, concerns, and ways of interpret- tive conceptual framework for the profession that
ing experiences. In contrast, patients typically do more properly includes human occupations, mean-
not separate out the physical from the social dis- ingful activities, and personal values. Occupational
turbances. therapists increasingly are examining the embod-
Qualitative research aspires to learn the entire ied experiences of patients as socially and not
range of features characterizing a patient’s experi- only physically functioning beings and as intend-
ence and build up a picture and evaluate how ing and feeling persons. The notion of embodi-
existing trait systems (like those used in clinical ment has at its core, the idea that substance and
practice) are useful, when they miss important spirit are inseparable (Csordas, 1994; Kielhofner,
phenomena, and even when they do harm. Medical 1995).
knowledge, but also about procedures. Occupatio-

the destination. The rules nal therapy researchers
of each epistemological The rules of each epistemo- can benefit from learn-
tradition directly shape ing the different stances
the type and style of
logical tradition directly that have shaped qualita-
research questions asked shape the type and style of tive research episte-
as well as the specific mologies. People who
procedures used.
research questions asked adopt one stance are led
It is beyond the scope as well as the specific to investigate particular
of this chapter to con- procedures used. problems that can be
sider which stance best answered within the per-
addresses a problem. spective of that stance.
Instead, the aim is to provide an appreciation of Those who adopt a different stance will be directed
each stance, its basic principles, definition, and to other problems.
The Place of Qualitative Research in Science with quantitative designs (Taylor, Braveman, &
Hammel, 2004), it most often employs qualitative
Because research in the behavioral sciences research strategies. Moreover, because PAR
and health care was historically dominated by involves important ideals and principles about how
quantities methods, for many decades qualitative knowledge should be generated and used, it is
research was misunderstood and viewed as less viewed by many qualitative researchers as a par-
rigorous. Fortunately, this circumstance has ticular epistemological tradition. This section
changed in most sectors of science. In many provides a brief overview of PAR in qualitative
fields of knowledge there is a very complemen-
research; Chapters 38 to 40 in this text provide
tary and interdependent relationship between
quantitative and qualitative approaches. In occu- detailed examinations of participatory research
pational therapy, as in other fields, scholars are and its application in occupational therapy.
increasingly trained in both qualitative and quan- PAR combines both research efforts and active
titative traditions (although they may specialize intervention within a single project. PAR is strik-
in one or the other).There is growing recognition ing for how the research goal is formulated and
that rigorous criteria for qualitative data exist. pursued: it involves multiple stakeholders, includ-
These criteria are recognized increasingly, in ing research participants, institutional representa-
top tier medical journals (e.g., British Medical tives (e.g., teachers, doctors, service providers),
Journal; Mays & Pope 1995a–c, 2000), and and researchers as equal partners. In addition to
research funding agencies (National Institutes
concerns for research rigor, PAR adds concern
of Health 2001; Ragin, Nagel, & White, 2004).
There has also been a rapid expansion of spe- with creating community trust and a sense of own-
cialty journals and book series attending to ership of the project and findings. The people who
qualitative research. It has been estimated that conduct the study and the procedures, and forms of
upwards of several hundred journals exists the findings and dissemination are collaboratively
across disciplines that publish qualitative planned and conducted in order to ensure that the
work (Wark, 1992). results both represent all the stakeholders and they
are able to trust the processes by which it was
developed.
Today in qualitative research, major epistemo-
logical traditions include: Conducting a PAR Project
• Participatory action research (PAR), The aphorism, “Look, think, act,” (Stringer, 1999)
• Critical theory, sums up the main features of PAR and highlights
• Ethnography, its socially engaged stance. When looking, investi-
• Phenomenology, and gators collect information to discover, define, and
• Grounded theory. describe a phenomenon or setting. When think-
The sections that follow will provide a sense of ing, they explore by interpreting, analyzing, and
what each tradition posits and what it offers. explaining. Finally, when acting, investigators
develop, implement, and evaluate a purposeful
Participatory Action Research (PAR) plan formulated to meet a local need or change
something in the context. These three steps pro-
Participatory Action Research (PAR) is an vide PAR collaborators with a script of explicit
approach to research used to confront pressing orientations and goals. Looking, thinking, and act-
social problems. Although PAR can be combined ing can be repeated, iteratively, throughout the
research process as new information, agendas, and
questions emerge from the input and shared expe-
The Nature of Epistemology riences of all the PAR partners.
Overall, the structure of PAR proceeds in the
Epistemology refers to the broad arena of philos- following steps. A project begins with the initial
ophy concerned with the nature and scope of
identification of a problem first by the researchers
knowledge. Epistemology asks such questions as:
What does it mean to know (the truth), and what or the participants, or both together. Then, collab-
is the nature of truth? What kinds of things can orative discussion and negotiations among all
be known? Can we believe in knowledge that is stakeholders serves to refine the sense of the prob-
outside the evidence of our senses, such as the lem. Next, the research partners review what is
lived experiences of others or events of the past? already known and published about the issue
What are the limits of self-knowledge? and/or attempts to address a problem. Afterwards,
they work to redefine the problem more clearly
and formulate an agenda for change. Next, the Resources on Participatory Action Research
methods for research and evaluation are selected.
Then, the partners implement the change, and col- In the United States, the federal Agency for
lect and analyze data to evaluate their efforts. Health Quality Research (AHQR) has published
Finally, the results are prepared and disseminated the results of a consensus conference and litera-
with recommendations for wider audiences. ture meta-analyses; it confirmed the positive out-
Generally, the investigators will return to refining comes of PAR, and summarized key design
the problem, goals, and procedures as they make criteria (Viswanathan et al., 2004). PAR has been
vital to the success of projects spanning from
use of the emerging knowledge and experiences
patient-centered mental health and substance
including evaluation of the changes. abuse treatment to community-based health and
in many primary health care and international
Contributions of Participatory development projects, including the community-
Action Research based rehabilitation movement (Lysack &
Kaufert, 1994). Several excellent studies using
Interest in PAR has grown rapidly. In part, this is the PAR approach are featured in recent issues of
because the collaborative design incorporates core the American Journal of Public Health (AJPH,
progressive social ideals. PAR seeks to ensure that 2003, August & September), Hart and Bond
the interests of all the partners are on an equal (1996), Stringer (1999), a report commissioned
footing; it is often described as “democratic” by the AHRQ (Viswanathan et al., 2004), and
(Gitlin, Lyons, & Kolodner, 1994). Its popularity is from Minkler and colleagues (2003). One of the
also owing to the fact that the results of PAR are most comprehensive reviews of the history,
more likely to be trusted by consumers and profes- development, and international uses of PAR is
Koning and Martin (1996). An excellent text
sionals and are more likely to be used. Thus, the published by the American Psychological Associ-
end result of PAR is not only new knowledge, but ation with occupational therapy contributors dis-
also changes in practices, organizations, and in cusses the use of PAR in community contexts
governmental, legal, economic, or social rules. It is (Jason et al., 2004). Finally, a new occupational
also socially progressive since it is designed to therapy text (Crist & Kielhofner, 2005) illustrates
enhance the life of individuals and communities. the use of PAR in advancing practice.
While PAR is relatively new to occupational ther-
apy, a number of projects using this approach have
demonstrated how services can be improved and
their experiences and reactions are part of the pro-
how therapists and clients can be empowered to
cess of gaining insight into the people and settings
achieve desired ends (Forsyth, Summerfield-
studied. What distinguishes ethnographic fieldwork
Mann, & Kielhofner, 2005; Taylor et al., 2004).
from other methods conducted in field settings is a
quest for the naturally occurring language, insider’s
Ethnography viewpoint and values, and cultural patterns.
Ethnography is a research approach that aims to The term, ethnography, embraces a wide range
discover and describe the point of view of a people of approaches that share an interest in learning:
or social scene. Ethnography is a dynamic tradi-
• The patterns in how a people define and view the
tion with a long history. From its early days in the
world,
late 1800s it was defined by a fieldwork tradition.
• Habitual patterns and ways of life,
Investigators, such as Franz Boas (1966), who first
• Categories of thought,
studied the Eskimo (Inuit) or Malinowski who
• Symbols and meanings,
studied the natives in the South Pacific Islands,
• Kinds of social relationships, and
lived among the people they studied and immersed
• Systems of moral goals, values, and social struc-
themselves in the settings and events in those
tures.
places. These investigators systematically learned
and spoke the local language; they observed and It strives to gain an insider’s view of the social
described the material lives, activities, structures scene (Spradley 1979, 1980). Geertz (1973)
of social life, relationships, and cultural beliefs summed up the ethnographic task as figuring out
that they observed. By conducting such fieldwork what those under study think they are up to. This
to learn directly from the natives about their view- aim contrasts with trying to force fit a description
points, these early ethnographers developed a of those under study into the language categories
method of first-hand discovery. and values held by the researcher.
Ethnography continues to evolve. For example, Ethnography is distinctive in several ways.
ethnographers now see researchers as instruments; Among these are its aims to:
• Describe the insider’s view, categories of lan- Conducting Ethnographic Research

guage, thought, rules for behavior and relation-
ships, and symbols, Ethnographic research is conducted in a series of
• Conduct studies in the natural settings of inform- basic phases:
ants’ lives by immersion and participation, • Preparation and entry,
• Regard participants as informants who help to • Immersion using participation and observation,
direct and interpret the topic of study, verify or • Exit, and
refute conclusions, and • Writing up.
• Focus on exploring the particulars of the specific
setting in time (historical, life course, develop- Study participants are properly called “inform-
mental), people, and place (physical and cultur- ants” as they inform and teach about their life and
ally constructed). community; they are not “subjects” controlled by
the researcher.
Ethnography also aims to build, validate, and The start of immersion can take several forms,
refute generalizations about human society. While from preparation by reading in an archive or pub-
ethnography focuses on the detailed case of a par- lications, to entry into the field site. As an outside
ticular culture or setting, it does so with an eye to participant, the ethnographer begins to learn the
proposing larger patterns of human life by using language and folk categories, how to ask and
what was already learned about the beliefs and answer questions, the history, kinds, and structure
structures of other societies. Generalizations grow of relationships, behavioral expectations, social
from the accruing record of each society which and life values, and life as defined by the par-
reveals recurrent patterns of similarity and differ- ticipants. Next, with continued immersion and
ence across systems of cultural values, relation- increasing insider knowledge, the questions reach
ships, and symbols. General theories about the into deeper realms of cultural values and philoso-
sociocultural life of humans are built and const- phy; they explore diversities in beliefs, and indi-
antly case tested by “the ethnographic veto,” a term vidual and group histories. Comparisons between
for the use of the rich record of empirical ethno- observed actions and events and the informants’
graphic studies of societies to provide counter expressed beliefs and social rules become possible
examples to a theory or over-simplification. For with extended time in the field. Keeping an ongo-
example, to provide a newer model of stigma (the ing field journal is used to monitor accruing
social labeling of a person as undesirably different) insights and highlight gaps and questions. Lastly,
researchers used a comparative global perspective ethnographers exit the setting to begin summariz-
to reveal institutional practices that covertly per- ing and interpreting their field data, but now at a
petuate stigma (Das, 2001; Link & Phelan, 2001). distance. The distance allows them to mentally
Ethnography’s insider view also helps to dispel compare insider and outsider viewpoints to
myths. For example, economic development explore and analyze the fieldwork data.
experts engaged to help subsistence peasant The data collection toolkit for ethnography fea-
farmers living high in the Andes Mountains tures:
believed the poverty was due to idleness and
under-employment. These experts wanted to get • Direct interview and observation of people,
more out of those they labelled as lazy peasants events, and artifacts,
because they saw few were laboring to dig or work • Personal participation in the ongoing routine and
the fields, tasks associated with busy productivity special events of social life, and
to the economists’ industrial models. Ethnogra- • Interpreting the stories, symbols, and objects in
phers used time allocation cultural methods the field site.
(Gross, 1984) to observe and ask peasants what Nowadays the traditional handwritten journal
they did, when and why. Their findings showed notebooks and maps for collecting data, are
that the crops required seasonal effort, not constant replaced by a range of technology including digi-
intense work. Moreover, of equal importance was tal audio or video recordings, GPS mapping, and
that people must be posted to stand watch over the computer software for taking field notes, indexing,
fields to protect the grain harvests from predation and analyzing of observations and interpretations.
by birds and animals. Thus, crop watching was a
very important economic routine. When under- Contributions of Ethnography
stood and counted as productive activity (not idle-
ness), the employment rate was 98% (Brush, Since ethnography can provide systematic data on
1977). people’s own perceptions, meanings, expectations
and needs, and structures for action, it is a power- taken concurrently during the project rather than in
ful tool for the health and social service fields. a sequence. Researchers will use data collection
Ethnography is used to study topics that range methods such as narratives, focused interviews,
from the social structure and value systems in hos- informal discussion, participant observation, and
pitals and rehabilitation facilities to patterns of field notes. Sample sizes often are not large, usu-
practitioner-patient interactions. Such research has ally in the range from 20 to 50 participants at the
provided important insights to problems and prac- most.
tices. For example, ethnographic studies of the cul- What distinguishes the grounded theory method
ture of nursing homes have shown that: from other qualitative methods is the structured
formalized process for data collection and theory
• Disruptive behavior by residents on a Alzheimer’s
development. It specifies a continuous interplay
care unit may be tied to the timing of nursing shift
between data collection and interpretation leading
changes (e.g., during meals) (Stafford, 2003),
to a generalization that is project-specific for that
• Incontinence is defined and managed by nursing
data (Strauss & Corbin, 1990). The work of evolv-
home staff as a wetness problem requiring dia-
ing a theory from the data is one of constant com-
pers instead of a potentially reversible medical
parison by which each new piece of information (a
condition that can be treated (Schnelle et al.,
belief, explanation for an event, experience, sym-
1989), and
bol or relationship) is compared to each other piece
• Malnutrition is a major problem in American
of data as it is gathered.
nursing homes (Kayser-Jones et al., 2003).
Investigators code the conditions, actions,
Without such sustained ethnographic field re- strategies for interactions, and outcomes observed.
search, these important health issues would never Should the information be similar to other already
have been identified. existing information, it is assigned to that category
The ethnographic tradition shares with occupa- and labeled, or coded with the descriptive word or
tional therapy the desire to work to learn, not pre- phrase for that group. On the other hand, if the idea
define, people’s own meaningful desired habits, or phenomenon does not fit a category already cre-
values, and actual life settings as well as perceived ated, then a new one is created.
challenges and resources. Perhaps this is why occu- Over time a set of categories emerge that are
pational therapists have looked to and welcomed refined and confirmed and the resulting set of
ethnographic insights offered by anthropology categories is used to make a more abstract gener-
researchers, and reciprocally, anthropologists have alization (i.e., the grounded theory) to explain
benefited from the insights of occupational therapy the phenomenon. The published reports from
clinicians to guide and frame their own work on grounded theory studies follow a similar design;
disability and rehabilitation (c.f., Mattingly, 1998). they usually would not contain a detailed literature
review (preexisting theory and data). The report is
Grounded Theory a descriptive discussion of the structured proce-
dures followed and findings.
Grounded theory is an inductive method designed
to construct theory from qualitative data (Glaser & Contributions of Grounded Theory
Strauss, 1967). It does so by following a defined set
of procedures for data collection but without direc- Grounded theory studies have the potential to offer
tion from existing constructs or theory about the important understanding of how people live with
phenomena. The term grounded refers to the aim to the challenges of illness and disability. For exam-
have the theory emerge from, or be grounded in, ple, the goal of a grounded theory study by
the data. The grounded theory approach seeks to Clements, Copeland, and Loftus (1990) was to
ensure that the theory derives from the experiences learn how parents coped with the adversity of liv-
of those on whom the study is focused. In this ing and caring for a child with a chronic illness.
instance, the theory is a generalization about the This study included focused interviews with thirty
empirical data. That is, the investigator seeks to families who attended a clinic where the children
explain the data from the specific study, not to pro- were treated. They found heightened challenges
pose a conceptual or philosophical model. and resource needs at the critical changes in the
child’s condition. A grounded theory developed by
Conducting a Grounded Theory Study the researchers focused specifically on the phe-
nomena described by participants in that study.
The procedures for conducting a grounded theory The theory they proposed was that the specific
study can be described individually, but are under- ways of coping developed by a family with a
chronically ill child attempts to meet the needs of Qualitative Research Is Not Only for Pilot
all the family members. A balance or equilibrium or Preliminary Research
is reached if there are resources, but that can not be
maintained if the demands rise or the support Although qualitative research is often used as
changes. the first or exploratory investigation in a new
area, it is also used to address problems that
Critical Theory other methods have not been able to unravel in
a well researched area. For example, research
Critical theory researchers take the stance that has shown that persons’ appraisals of their own
knowledge (and theory) is not universal and health are one of the most powerful predictors
absolute. Instead, it starts with the basic premise of disability and death. For example, persons
that social reality is embedded in and constructed who rate their health as fair or poor have a three
in specific historical times and places and that it times greater likelihood of death than those who
is produced and reproduced by people. Simply rate it as good or excellent. These findings
stated, then, adherents of critical theory see social emerged from the analyses of many large-scale
secondary datasets on health services utilization.
reality and knowledge as relative to particular peo-
Yet, extensive replication in multiple interna-
ple and times. They argue that multiple social real- tional studies has revealed little about what peo-
ities exist distributed across the various segments ple have in mind or how they go about reasoning
of society and groups. Critical theory overlaps on their way to such self-labels for their health.
with the post-modern philosophical perspectives Consequently, epidemiologists have recently
discussed in Chapter 2. turned to a variety of qualitative interview meth-
Critical theorists contrast starkly with “posi- ods to learn first hand what was in the minds of
tivists” who assume there is but one objective real- individuals as they provided these self ratings.
ity which can be captured and measured by Qualitative findings are beginning to shed light
instruments that are independent of the observer. on this issue. For example, Idler and Benyamini
(1997) reported that persons making self-rated
Positivist studies generally attempt to test theory,
health judgments are influenced not only by fac-
in an attempt to increase the predictive under- tors associated with their physical body and its
standing of phenomena. On the other hand, critical maladies, but rather, include social aspects of
theory’s purpose is to enlighten people and make the impact of their illnesses and disabilities too.
them critically aware. Luborsky (2005) has found that self-appraisals
The goal in critical theory research is to bring to and interpretations of health include complex
light or become aware, or (i.e., critical) rather than belief systems related to the perceived moral
passively acting according to the reigning sociopo- consequences associated with labeling yourself
litical structures and settings that shape ways of as ‘good’ or bad’. The social fall-out associated
thinking. In this context, the term, critical, does not with functional independence and the meaning
associated with participation in roles, activities,
mean to demean or ridicule. Rather, it means to
and settings, exert an overwhelmingly powerful
pose questions. The aim of critical theory is posi- influence on how individuals rate their health,
tive social and political transformation, including well-being, and overall life quality. None of these
reducing social injustices. Thus, it focuses on taken evaluations are or ever can be predicted by blood
for granted ways of thinking, insights gathered tests, medical diagnoses, or the severity of one’s
through heightened awareness of the diversity, and illness or injury. Similarly, research has shown
inequalities afflicting many segments of society. that injury severity is not a useful predictor of
More fundamentally, critical theorists view human either long term physical functioning or social
inaction in the face of social injustices as resulting participation and community integration (Dijkers,
from domination by the status quo. 1997, 1998; Lysack, Zafonte, Neufeld, &
Dijkers, 2001; Mossey & Shapero, 1982).
Critical theory is reflected in the work of a wide
range of scholars from Marx and Hegel to
Foucault and Derrida. One of the best known pro-
ponents of critical theory, Habermas (1988), along
with others, argue that scientific and philosophical Critical theory is less familiar to occupational
constructs are enmeshed in and serve to recreate therapy than some other qualitative epistemologi-
wider social-historical patterns.1 cal traditions. Nonetheless, there are some exam-
ples of occupational therapy research that are
1More general critical theory programs are sometimes also directly informed by a critical theory perspective,
described as critical research (Mishler, 1986) or analytic
induction (Strauss & Corbin, 1990). Cogent reviews of the such as the work of Whiteford and Wright-St. Clair
limits of critical theory are also available (Hammersley, (2004). A more critically informed occupational
1992; Honneth, 1991). therapy will be reflective about the nature of the
research questions it asks, their historical origins, age relates to health, morbidity, and mortality have
and the forms for answers and solutions it allows. shaped the scientific portrait of what constitutes
To the extent that occupational therapy is centrally normal aging. While these intertwined social influ-
concerned with practices that are “client-centered” ences have always been present, they are only now
(Law, 1998; Townsend, Langille, & Ripley, 2003) coming to be understood (Dannefer & Sell, 1988;
and morally concerned with their clients’ social Longino, 1990).
positions in society, they must be attentive to the In many respects, this is because it is difficult to
diversity in their clients’ desired goals and the be reflective about the times in which one lives. It
methods by which occupational therapy practition- is only afterwards, with the benefit of hindsight
ers can fully address their rehabilitation needs that investigators can see a historical period more
within that broader social context. clearly. For example, consider how clothing styles
One limitation of this approach to research that and fashions, tastes in music, and design of auto-
is particularly relevant to occupational therapy is mobiles evolve over a period of years. The same is
that in critical theory, the individual is not concep- true for less visible attributes of a historical period.
tualized as a willful autonomous person. Instead, People during different decades hold very different
individuals are viewed as social rule or norm attitudes and values than generations before, and
bound; they are defined simply as the sum of fam- afterwards.
ily, work and community roles. A case in point is the public’s attitudes toward
people with disabilities. These attitudes have
Conducting a Critical Theory Study changed substantially over the last several decades.
For examples, disabled persons are no longer sys-
No formalized procedures exist for critical theory,
tematically segregated in asylums and institutions.
compared to those for grounded theory. But, a gen-
Moreover, while the status of persons with disabil-
eralized approach can be outlined (Luborsky &
ity can still be much improved, there are disability
Sankar, 1993). The critical theory approach rests
rights laws, protections against disability discrimi-
on the systematic pursuit of a set of clearly articu-
nation in the workplace, more accessible buildings,
lated questions.
and more public visibility and acceptance of dis-
There are four general components. First, a
abled persons.
clear definition of a key concept or problem is pre-
Occupational therapists are in a position to uti-
sented. Second, a description of how the construct
lize the results of critical theory research to
is currently conceptualized is stated, and that for-
enhance their interventions and positively impact
mulation’s place in the continuing (past to present)
the occupational well-being of their clients. For
thought on the issue is summarized, often as a lit-
example, occupational therapists are more aware of
erature review. Third, the current definition is cri-
the disproportionate prevalence of disability among
tiqued to reveal gaps and limitations in the
economically disadvantaged people (House, Kess-
concept/problem formulation’s ability to explain
ler, Herzog, 1990; Townsend & Wilcock, 2003) and
the phenomena it focuses on, and other problems
the exclusion of oppressed persons from opportuni-
that it does not highlight. Notably, one would ask
ties or resources needed to engage in meaningful
in what ways the problems defined for study are
activity (Kronenberg, Simo-Algado, & Pollard,
consonant with the wider sociopolitical climate of
2005; Whiteford & Wright-St. Clair, 2004). The
that time, and in what ways they implicitly
promise of critical theory for occupational therapy
embody visions for continued re-creation of the
is in examining and documenting underlying
existing social organization and values for human
assumptions that reflect power imbalances and
life. Fourth, the researcher conducts research and
social injustices.
presents data that is informed by the analytical and
historical critique. The form of results for critical
theory research is new data as well as new ques- Phenomenology
tions and analytic frames for thought.
Phenomenology is both a way of doing research
(method) and a way of questioning and conceptu-
The Contributions of Critical Theory
alizing thought (philosophy). Like critical theory,
Critical theory helps reveal how each culture and phenomenology is a complex and multifaceted phi-
group has its own definitions for familiar scientific losophy that is not easily characterized. Moustakas
categories such as health, illness, ethnicity, family, (1994) explains, “The understanding of meaningful
or self. It points out that such familiar categories concrete relations implicit in the original descrip-
are not universal. For example, historically unrec- tion of experience in the context of a particular sit-
ognized ethnic and social class differences in how uation is the primary target of phenomenological
knowledge” (p. 14). Schwandt (1997) reminds us rience with other people (e.g., body language, art,
that phenomenology “rejects scientific realism and music, etc.). Nonetheless, it is not easy for one per-
the accompanying view that the empirical sciences son to comprehend or appreciate another person’s
have a privileged position” (p. 114) in identifying experience in exactly the same way that one does.
and explaining features in our world. Therefore, a critical element in phenomenological
At its most basic level, this approach focuses on studies is the skill of the researchers in identifying
the everyday life-world and gives great attention to an appropriate source of information about the
the careful description of how the ordinary is expe- experience they are choosing for study, a topic
rienced and expressed in the consciousness of indi- which will be addressed shortly.
viduals. This requires that phenomenology rests on There is a second major contrast with other
an assumption that there is a structure and essence qualitative approaches worthy of notice. In terms
to personal experience that can be communicated of the researcher, phenomenological research dif-
to others in a systematic way, often using narra- fers from ethnographic and grounded theory. In the
tives. The primary question that is asked from this latter forms of qualitative research, the meanings
perspective is: what is the meaning of one’s expe- emerge through a back and forth unfolding explo-
rience and how does one interpret it? ration between the researcher and the researched,
When William James appraised kinds of mental and are then further interpreted and explained
activity in the stream of consciousness (including during data analyses. In phenomenology, no struc-
their embodiment and their dependence on habit), ture or framework is imposed on the data by the
he was practicing a form of phenomenology researcher. Rather, the researcher must find and
(James, 1967). So too are all analytic philosophers (re)present the experiences in the form they are
of the mind. Still, the discipline of phenomenology expressed. This can be very challenging and con-
as we know it today is largely due to Edmund siderable investigator effort must be devoted
Husserl (2001) who launched the modern day to the choice of sample to ensure that it can pro-
movement with his seminal work, Logical vide the fundamental insights into the experience
Investigations. For Husserl, phenomenology inte- that is the target of the investigation (Luborsky,
grates a kind of psychology with a kind of logic. It 1994b; Luborsky & Rubinstein, 1995).
is psychological in the sense that it describes and For example, if a phenomenological study is
analyzes types of subjective mental activity or designed to understand the experience of surviving
experience, and it is logical in the sense that it a hip fracture and returning to normal life in the
describes and analyzes the objective contents of community, then it would be imperative that an
consciousness (i.e., experience). Other famous informed participant is selected for interview and
phenomenologists are Heidegger, Sartre, and perhaps even observation. Importantly, qualitative
Merleau-Ponty. Each hold different conceptions of researchers must remember that in a study about
phenomenology however, and use different meth- the experience of hip fracture, the unit of study
ods to study human experience. The Encyclopedia would be the individual with this injury, but the
of Phenomenology (Embree, 1997) is an excellent focus of research enterprise, or the units of analy-
reference that comprehensively details the features sis would be the experiences of hip fracture and
of seven separate forms of phenomenology, includ- the experiences of reconnecting to a personally
ing these prominent theorists. meaningful life. The intent of a phenomenological
Phenomenology contrasts with other qualitative researcher in such a study would be to gain under-
research approaches in its stance toward the standing of what it is like to live with an altered
informant and the researcher. Phenomenological body that limits mobility, and perhaps makes other
research regards the sense of lived experiences and people think of one as old. Phenomenological
meanings as fully knowable only by those who researchers would also want to know how these
share the experience. That is, in ordinary life, peo- experiences shape the person’s sense of them-
ple are somewhat limited in their ability to grasp selves as a full adult person (Luborsky, 1994a).
and intuit the meanings of lived experiences of As stated earlier, one’s choice of informant is
other individuals. This is because the meanings of critical in phenomenology. Sampling in this tradi-
experience must be transmitted and filtered from tion, as should be clear by now, must be purposive
the person who has the experience to the other per- and theoretically driven, in an effort to maximize
sons who wish to understand that experience the range of experiential phenomenology of people
(Luborsky, 1994a, 1994b, 1995). Of course, things with the experience of interest (Karlsson, 1993).
can be lost in translation! Sampling in this tradition must also build on what
There are many forms of expressive media is already known (Bertaux, 1981; Glaser & Strauss,
beyond words that are used to communicate expe- 1967; Luborsky & Rubinstein, 1995). If phenome-
nological research is to be successful in capturing philosophy. The outcome of analysis is a theoreti-

what particular experiences are like, it must gather cal statement responding to the research question.
their data from informed samples with first-hand The statement is validated by examples of the data,
access to the experience of interest. Only then can often direct quotes from the subjects.
the investigation gain access and insight into the The procedures for phenomenological research
informant/participant’s experiences. involve biographical story telling and informal dis-
cussion, with encouragement to reflect on at least
Conducting a Phenomenological Study the four aspects of experience outlined above.
Researchers listen for and inquire about the body,
Phenomenological researchers conceptualize the time, place, and settings of the phenomena, but the
person and the environment as a whole. A central informant is entirely in charge of directing the nar-
tenet of this perspective ratives and story telling.
is that the only reliable In one example, occupa-
source of information to Just as quantitative tional therapy researchers
answer questions about researchers select from a aimed to characterize the
personal experience is experiential features of
the person with the toolkit of methods and engagement in creative
experience, him or her- frameworks, qualitative activity as therapy for
self. Since each individ- elderly people with ter-
ual has his or her own researchers have available a minal illness (la Cour,
unique reality, the task wide range of methods from Josephsson, & Luborsky,
of the phenomenologist 2005). Using extended
is to engage in lengthy which to select those best discussions about the
discussions with partici- suited to their questions, the projects undertaken by
pants about their experi- older adults in Sweden,
ences and then to locate
data required to answer the
the researchers found how
and summarize common questions, and the forms of creative activity served as
themes in their expres- analyses suited to that data. a medium that enabled
sions of experiences that creation of connections to
convey a central essen- wider culture and daily
tial meaning. To begin to achieve this goal, four or life which countered some of the more serious
more aspects of human experience need to be social consequences of terminal illness, such as
explored. These include: isolation. The creation of connections to life expe-
rience in this study embodied three features: a gen-
• The lived space (spatiality),
erous perceptive environment as the foundation for
• The lived body (corporeal embodied experience),
meaningful activity, the creations as an unfolding
• Lived social relationships (relationality), and
evolving liberating process; and a reaching beyond
• Lived time (temporality).
the present for possible meaning horizons. The
Thus, phenomenology is simultaneously holis- findings showed that creative activity fosters con-
tic and also relativistic to the particular experi- nections to meanings as an active person, even in
ences and situations of each person. the face of uncertain life-threatening illness.
Understanding human experience also requires
that the person with the experience must self- Contributions of Phenomenological
interpret these experiences for the researcher, and
Research
then the researcher must further interpret the indi-
vidual’s explanation provided by the person. Thus, As the previous example illustrated, powerful
the method of phenomenology is one of interactive occupational therapy relevant insights are provided
dialogue and exchange, as the researcher seeks to by this perspective. For example, Hasselkus (1998)
know what the experience is like. used a phenomenological approach to illuminate
Typically, the data in phenomenological studies the daily experiences of day-staff who cared for
are collected by in-depth conversations in which Alzheimer’s patients on a dementia unit. Studies
the researcher and the informant are fully interac- like this that deeply probe the experiences of care
tive. Analysis begins when the first data are col- providers, be they professionals, family members,
lected. This analysis will guide decisions related to or others, are essential for the profession so that we
further data collection. The meanings attached to can see how our interventions best fit to support the
the data are expressed within phenomenological efforts of others.
A recent phenomenology method formulated by the multiple kinds of stances and goals available to
Karlsson (1993) is gaining use among occupational qualitative researchers. Chapters 20 to 22 provide
therapy researchers. For example, a compelling use more details of the nuts and bolts of undertaking
of this phenomenological method is reported by qualitative research. They will illustrate in more
Tham, Borell, and Gustavsson (2000). In this study practical ways how to implement the different
of the experience of unilateral neglect post-stroke, frameworks outlined in this chapter.
research findings strongly supported the value
of occupational therapy and clearly illustrate REFERENCES
the links between research and effective practice. American Journal of Public Health. (2003, August). This
The following passage from the study illustrates issue focuses on public health advocacy and includes
the kind of insights that phenomenological research an article on challenges and strategies to obtain funding
for Community-Based Participatory Research.
seeks to achieve (Tham & Kielhofner, 2003): American Journal of Public Health. (2003, September).
The study demonstrated that the partici- This issue focuses on the built environment and health.
The article “Jemez Pueblo: Built and social-cultural
pants needed to experience and ultimately environments and health within a rural American
come to a practical recognition of their Indian community in the Southwest” describes a study
own impairments and the consequences of that used participatory research to uncover sociocultu-
those impairments during occupational ral and environmental factors that indicate capacity for
improving health.
performance, before they learned to handle
Baron, R. J. (1985). An introduction to medical phenome-
them in everyday life. However, because nology. Annals of Internal Medicine, 103, 606–610.
neglect represents a particular life-world Becker, G., & Kaufman, S. (1995). Managing an uncertain
experience, people with neglect had to illness trajectory after stroke: Patients’ and physicians’
learn to stand outside their own experi- views of stroke. Medical Anthropology Quarterly, 9,
165–187.
ence. Specifically, because unilateral neg- Bertaux, D. (Ed.) (1981). Biography and society: The life
lect is not directly experienced by the history approach in the social sciences. Beverly Hills,
person who has it (e.g., the left half of the CA: SAGE Publications.
world is not “felt” to be absent from per- Boas, F. (1966). Kwakiutl ethnography. Chicago, IL &
London, England: University of Chicago Press &
ception), persons with neglect must embark
Toronto, Ontario: University of Toronto Press.
on a discovery process. This discovery Brush, S. (1977). Myth of the idle peasant: Employment in
process involves coming to understand that a subsistence economy. In R. Halperin & J. Dow
there exists a half of the world that is not (Eds.), Peasant livelihood (pp. 60–78). New York: St.
part of their life-world. Once they can com- Martin’s Press.
Clements D., Copeland L., & Loftus M. (1990). Critical
prehend the existence of a part of the world times for families with a chronically ill child. Pediatric
that is outside their experience, they could Nursing, 16(2), 157–161.
begin to manage the consequences of their Crist, P., & Kielhofner, G. (2005). The scholarship of prac-
own unilateral neglect during occupational tice: Academic and practice collaborations for promot-
ing occupational therapy. Binghamton, NY: Hayworth
performance (p. 404).
Press.
Csordas, T. (1994). Embodiment and experience: The exis-
tential ground of culture and self. Cambridge, UK:
Conclusion Cambridge University Press.
Dannefer, D., & Sell, R. (1988). Age structure, the life
course and “aged heterogeneity”: Prospects for
By now it should be clear that in qualitative research and theory. Comprehensive Gerontology [B],
research there is no single approach. As Patton 2(1), 1–10.
(1990) suggests, investigators need to use the Das, V. (2001). Stigma, contagion, defect: Issues in the
anthropology of public health. Presented at “Stigma
methods that are most appropriate for the research and Global Health: Developing a Research Agenda.”
questions they confront. Just as quantitative An International Conference, September 5–7, 2001,
researchers select from a toolkit of methods and Bethesda, Maryland. http://www.stigmaconference.
frameworks, qualitative researchers have available nih.gov/papers.html (accessed June 20, 2005).
Dijkers, M. (1997). Quality of life after spinal cord injury:
a wide range of methods from which to select those
a meta-analysis of the effects of disablement compo-
best suited to their questions, the data required to nents. Spinal Cord, 35, 829–840.
answer the questions, and the forms of analyses Dijkers, M. (1998). Community integration: conceptual
suited to that data. issues and measurement approaches in rehabilitation
The reader should not expect to be able to research. Topics in Spinal Cord Injury Rehabilitation,
4, 1–15.
neatly define and argue the merits and limits of Edgerton, R. B. (1967, 1993). The cloak of competence:
each epistemological approach discussed in this Stigma in the lives of the mentally retarded. Berkeley,
chapter. It is sufficient to have begun to appreciate CA: University of California Press.
Embree, L. (Ed.) (1997). Encyclopedia of phenomenology. Kaufman, S., & Becker, G. (1986). Stroke: Health care on
Dordrecht and Boston: Kluwer Academic. the periphery. Social Science and Medicine, 22,
Finlayson, M. (2004). Concerns about the future among 983–989.
older adults with multiple sclerosis. American Journal Kayser-Jones J., Schell, E., Lyons, W., Kris, A., Chan, J.,
of Occupational Therapy, 58, 54–63. & Beard, R. (2003). Factors that influence end-of-life
Forsyth, K., Summerfield-Mann, L., & Kielhofner, G. care in nursing homes: The physical environment, inad-
(2005). A scholarship of practice: Making occupation- equate staffing, and lack of supervision. Gerontologist,
focused, theory-driven, evidence-based practice a real- 43(2), 76–84.
ity. British Journal of Occupational Therapy, 68, Kielhofner, G. (1995). A meditation on the use of hands.
261–268 Scandinavian Journal of Occupational Therapy, 2,
Geertz, C. (1973). The interpretation of cultures. New 153–166.
York: Basic Books. Kielhofner, G. (2001). A model of human occupation
Gitlin, L., Luborsky, M., & Schemm, R. (1998). Emerging (3rd ed.). Philadelphia, PA: Lippincott, Williams &
concerns of older stroke patients about assistive device Wilkins.
use. The Gerontologist, 38(2), 169–180. Koning, De K., & Martin, M. (1996). Participatory
Gitlin, L., Lyons, K. J., & Kolodner, E. (1994). A model to research in health: Issues and experiences. London:
build collaborative research or educational teams of Zen Books.
health professionals in gerontology. Educational Kronenberg, F., Simo-Algado, S., & Pollard, N. (2005).
Gerontology, 20(1), 15–34. Occupational therapy without borders. UK: Churchill
Glaser, B., & Strauss, A. (1967). The discovery of Livingstone.
grounded theory: Strategies for qualitative research. la Cour, K., Josephsson, S., & Luborsky, M. (2005).
Chicago: Aldine. Creating connections to life during life-threatening ill-
Gross, D. (1984). Time allocation: A tool for the study of ness: Creative activity experienced by elderly people
cultural behavior. Annual Review of Anthropology, 13, and occupational therapists. Scandinavian Journal of
519–558. Occupational Therapy, 12(3), 98-109.
Habermas, J. (1988). The logic of the social sciences. Langness, L., & Levine, G. (Eds.) (1986). Culture and
Cambridge, MA: MIT Press. retardation. Dordrecht: Reidel.
Hammersley, M. (1992). What’s wrong with ethnography? Link, B., & Phelan, J. (2001). Conceptualizing stigma.
Methodological explorations. London, England: Annual Review of Sociology, 27, 363–385.
Routledge. Luborsky, M. (1994a). The cultural adversity of physical
Hart, E., & Bond, M. (1996). Making sense of action disability: Erosion of full adult personhood. Journal of
research through the use of a typology. Journal of Aging Studies, 8(3), 239–253.
Advanced Nursing, 23(1), 152–159. Luborsky, M. (1994b). The identification and analysis of
Hasselkus, B. R. (1998). Occupation and well-being in themes and patterns. In J. Gubrium & A. Sankar (Eds.),
dementia: The experience of day-care staff. The Qualitative methods in aging research (pp. 189–210).
American Journal of Occupational Therapy, 52, Thousand Oaks, CA: SAGE Publications.
423–434. Luborsky, M. (1995). The process of self-report of impair-
Honneth, A. (1991). The critique of power: reflective ment in clinical research. Social Science & Medicine,
stages in a critical social theory. Cambridge, MA: MIT 40(11), 1447–1459.
Press. Luborsky, M. (1997). Attuning assessment to the client:
House, J., Kessler, R., & Herzog, A. (1990). Age, socioe- Recent advances in theory and methodology.
conomic status, and health. Milbank Quarterly, 68(3), Generations, 21(1), 10–16.
383–411. Luborsky, M., & Rubinstein, R. (1995). Sampling in quali-
Husserl, E., (2001). Logical investigations (vols. 1 and 2), tative research: Rationales, issues, and methods.
Trans. J. N. Findlay. Ed. with translation corrections Research on Aging, 17(1), 89–113.
and with a new Introduction by Dermot Moran. With a Luborsky, M., & Sankar, A. (1993). Extending the critical
new Preface by Michael Dummett. London and New gerontology perspective: Cultural dimensions. Geronto-
York: Routledge. A new and revised edition of the orig- logist, 33(4), 440–444.
inal English translation by J. N. Findlay. London: Lysack, C., & Kaufert, J. (1994). Comparing the origins
Routledge & Kegan Paul, 1970. From the second edi- and ideologies of the independent living movement and
tion of the German. First edition, 1900–01; second edi- community based rehabilitation. International Journal
tion, 1913, 1920. of Rehabilitation Research, 17, 231–240.
Idler, E., & Benyamini, Y. (1997). Self-rated health and Lysack, C., Zafonte, C., Neufeld, S., & Dijkers, M. (2001).
mortality: A review of twenty-seven community stud- Self-care independence after spinal cord injury:
ies. Journal of Health and Social Behavior, 38(1), Patient and therapist expectations and real life per-
21–37. formance. Journal of Spinal Cord Medicine, 24(4),
James, H. Jr. (1967). The Letters of William James 257–265.
(Boston: Atlantic Monthly Press, 1920). I:190 quoted Mattingly, C. (1998). Healing dramas and clinical plots:
in J. McDermott (Ed.), The writings of William James The narrative structure of experience. Cambridge, UK:
(p. xi). New York: Random House. Cambridge University Press.
Jason, L. A., Keys, C. B., Suarez-Balcazar, Y., Taylor, R. Mays, N., & Pope, C. (2000). Qualitative research in health
R., Durlak, J., Davis M. & Isenberg, D. (Eds.) (2004). care: Assessing quality in qualitative research. British
Participatory community research: Theories and meth- Medical Journal, 320, 50–52.
ods in action. Washington, DC: American Psychologi- Mays, N., & Pope, C. (1995a). Observational methods in
cal Association. health care settings. British Medical Journal, 311,
Karlsson, G. (1993). Psychological qualitative research 182–184.
from a phenomenological perspective. Stockholm: Mays, N., & Pope, C. (1995b). Reaching the parts other
Almqvist & Wiksell International. methods cannot reach: An introduction to qualitative
methods in health and health services research. British encounters with nursing home culture. Sante Fe, NM:
Medical Journal, 311, 42–45. School of American Research Press.
Mays, N., & Pope, C. (1995c). Rigour and qualitative Strauss, A, & Corbin, J. (1990). Basics of qualitative
research. British Medical Journal, 311,109–112. research: Grounded theory procedures and techniques.
Minkler, M., Blackwell, A., Thompson, M., & Tamir, H. Newbury Park, CA: SAGE Publications.
(2003). Community-based participatory research: Stringer, E. (1999). Action research (2nd ed.). Thousand
Implications for public health funding. American Oaks, CA: SAGE Publications.
Journal of Public Health, 93(8), 1210–1213. Taylor, R. R., Braveman, B., & Hammel, J. (2004). Deve-
Mishler, E. (1986). Research interviewing. Cambridge, loping and evaluating community services through
MA: Harvard University Press. participatory action research: Two case examples.
Mossey, J., & Shapiro, E. (1982). Self rated health: A American Journal of Occupational Therapy, 58,
predictor of mortality among the elderly. American 73–82.
Journal of Public Health, 72, 800–808. Tham, K., Borell, L., & Gustavsson, A. (2000). The dis-
Moustakas, C. (1994). Phenomenological research covery of disability of disability: A phenomenological
methods. Thousand Oaks, CA: SAGE Publications. study of unilateral neglect. American Journal of
Murphy, R. (1987). The body silent. New York: Henry Holt. Occupational Therapy, 54, 398–406
National Institutes of Health (2001). Qualitative methods Tham, K., & Kielhofner G. (2003). Impact of the social
in health research: Opportunities and considerations in environment on occupational experience and perform-
application and review. Office of Behavioral and Social ance among persons with unilateral neglect. The
Sciences Research, National Institutes of Health. American Journal of Occupational Therapy, 57,
Bethesda, MD. NIH Publication No. 02-5046. 403–412.
(http://obssr.od.nih.gov/Publications/Qualitative.PDF) Townsend, E. A., & Wilcock, A. A. (2003). Occupational
Patton, M. (1990). Qualitative evaluation and research justice. In C. Christiansen & E. Townsend (Eds.),
methods. Thousand Oaks: SAGE Publications. Introduction to occupation (pp. 243–273). Thorofare,
Ragin, C., Nagel, J., & White, P. (Eds.) (2004). Workshop NJ: Prentice-Hall.
on the scientific foundations of qualitative research. Viswanathan, M., Ammerman, A., Eng, E., Gartlehner, G.,
National Science Foundation. Arlington, Virginia. Lohr, K., Griffith, D., Rhodes, S., Samuel-Hodge, C.,
Document Number: nsf04219. (also, http://www.nsf. Maty, S., Lux, L., Webb, L., Sutton, S., Swinson, T,
gov/pubs/2004/nsf04219/start.htm accessed May 3 Jackman, A, & Whitener, L. (2004). Community-based
2005). participatory research: assessing the evidence.
Reilly, M. (1962). Eleanor Clark Slagle Lecture: Occupa- Summary, Evidence Report/Technology Assessment
tional therapy can be one of the great ideas of 20th No. 99 (Prepared by RTI–University of North Carolina
century medicine. American Journal of Occupational Evidence-based Practice Center under Contract No.
Therapy, 16, 1–9. 290-02- 0016). AHRQ Publication 04-E022-1.
Schnelle, J. F., Traughber, B., Sowell, V. A., Newman, D. Rockville, MD: Agency for Healthcare Research
R., Petrilli, C. O., & Ory, M. (1989). Treatment of uri- and Quality. http://www.ahrq.gov/clinic/evrptpdfs.
nary incontinence in nursing home patients. A behavior htm#cbpr.
management approach for nursing home staff. Journal Wark, L. (1992). Qualitative research journals. The
of the American Geriatrics Society, 37, 1051–1057. Qualitative Report, 1 (4): (pages not numbered)
Schwandt, T. (1997). Qualitative inquiry. Thousand Oaks, (http://www.nova.edu/ssss/QR/QR1-4/wark.html)
CA: SAGE Publications. Whiteford, G., & Wright St. Clair, V. (2004). (Eds.)
Spradley, J. (1979). The ethnographic interview. New York: Occupation and context in practice. Sydney: Churchill
Holt, Rinehart and Winston. Livingstone.
Spradley, J. (1980). Participant observation. New York. Zemke, R., & Clark, F. (Eds.) (1996). Occupational sci-
Holt, Rinehart and Winston. ence: The evolving discipline. Philadelphia: F. A.
Stafford, P. (Ed.) (2003). Gray areas: Ethnographic Davis.
C H A P T E R 2 0
Gathering Qualitative Data

Cathy Lysack • Mark R. Luborsky • Heather Dillaway
Qualitative research has become an increasingly narrative analysis, symbolic interactionism, con-
important mode of inquiry for occupational ther- tent analysis, discourse analysis, critical theory,
apy. Long dominated by techniques borrowed from semiotics, and action research, to name a few.
the experimental sci- Each of these qualita-
ences, occupational ther- tive approaches is dif-
apy has embraced many Irrespective of its particular ferent with respect to the
of the methods of the disciplinary roots, qualitative type of research designs
social sciences, particu- commonly used, the
larly from anthropology
studies ask questions that methods used to gather
and sociology, as an are rarely specifiable as data, data analysis ap-
alternate means of study- proaches, and the form in
ing and understanding
conventional hypotheses, as which study findings are
social phenomena of rel- is the case in quantitative disseminated. In spite of
evance to the profession. research. their differences, there
While valuable in its are some common char-
own right, quantitative acteristics and proce-
research differs substantially from qualitative dures for the conduct of qualitative research and
research. Quantitative research describes phenom- several philosophical assumptions that are shared
ena using a single universal standardized language among them. Most basically, all qualitative
(numbers), and investigator-defined topics and researchers are intrigued with the complexity of
issues. Based on statistical theories of probability, social interactions as expressed in daily life. This
it seeks to discover and describe such things as sta- interest takes them into natural settings as opposed
tistically defined norms and central tendencies and to laboratories. This is why qualitative research is
averages in the distribution of the data. also sometimes called naturalistic inquiry.
In contrast, qualitative methods use the local Rossman and Rallis (1998) have summarized the
varieties of languages and words of the partici- core features of qualitative research and qualitative
pants in the social settings studied. These methods researchers. Although they originally outlined
work to discover and describe participant-defined eight features, we believe the following five are
topics of concern and socioculturally constructed unique to qualitative work. Qualitative work is:
worlds within which individuals pursue meaning-
ful actions. In short, qualitative methods seek to • Naturalistic,
discover and describe socioculturally constructed • Emergent and evolving, rather than prefigured,
worldviews, values, and sociocultural norms and • Fundamentally interpretive,
how these are instilled, enacted, reinforced, or • Characterized by a holistic view of social phe-
resisted and changed in everyday life. Qualitative nomenon, and
researchers go to the field to study the topic in the • Sensitive to the influence of investigators on the
natural settings where people socially interact. study and its findings.
While it would be tidy if the phrase “qualitative
research” denoted one single methodological In its broadest sense, qualitative research must
approach, unfortunately this is not the case. There be understood as the systematic study of social
are many different forms of qualitative research, phenomena. Irrespective of its particular discipli-
depending on the discipline and its assumptions nary roots, qualitative studies ask questions that
about what counts as knowledge and how it is gen- are rarely specifiable as conventional hypotheses,
erated. Probably the best known approaches in the as is the case in quantitative research. The ultimate
qualitative tradition come from anthropology and goal of qualitative research is to go beyond the
sociology. These disciplines provide a wide array what of research to explain the why and how. This
of tools and approaches with which to gather qual- empirical pursuit requires concepts and tools that
itative data including ethnography, life histories, yield such information and approaches.
Major Data Gathering The personal reflections gained from participa-

tion in the research setting, as well as the obser-
Strategies vations made, are systematically recorded and
included as an essential part of the study data.
The following are four main methods for gathering Thus, simultaneously, the researcher using partici-
qualitative data: pation as a method is challenged to recognize and
record how his or her presence in the field may or
• Participation in the setting, may not be influencing study participants’ behav-
• Direct observation, iors in the field. Major efforts to track the impact
• In-depth interviewing, and of the presence of the investigator in the field are a
• Analyzing documents and objects. feature of this methodology. Journals and field-
All of these methods fundamentally reflect the notes, recorded both during and after episodes of
core qualities of qualitative research as identified data collection, are a core source of study data.
by Rossman and Rallis (1998). This chapter pro-
vides a review and discussion of these four meth- Observation
ods, highlighting their relative strengths and
weaknesses. Given the key role of the qualitative Observation requires careful watching, listening,
researcher as a data collection instrument, the role and recording of events, behaviors, and objects in
and stance of the qualitative researcher are also the social setting chosen for study. The observation
critically examined. record, frequently referred to as field notes, is com-
posed of detailed nonjudgmental, concrete descrip-
tions of what has been observed. For studies
Participation
relying exclusively on observation, the researcher
Participation is both an overall approach to inquiry makes no special effort to have a particular role
and a data gathering method. To some degree all other than simply being an unobtrusive observer.
qualitative research has a participatory element. Unlike participation methods, observational
Participation demands first-hand involvement in methods do not ask the researcher to become
the social world chosen for study. Qualitative actively involved. Rather, the researcher takes a
researchers work to gain access to modes of under- relatively outsider role. Observational studies of
standing and to approximate as closely as possible in-patient rehabilitation units are one example of
the lived realities of those studied, knowing that this type of study. Of course, without other sources
they will rarely (some would say never) truly expe- of data, the meaning of observations can only be
rience the situation exactly as their participants do. inferred. That is why, when setting, interactions
The purpose of engaging in participation is not to among persons, and their personal views are
become one of the group studied, but rather to use important, qualitative studies include both obser-
what is seen and heard to help identify what is vation and participation.
important to learn about, and to discover issues Participant observation is a more active data
and events that might not be obvious to an outsider gathering strategy and combines elements of both
who is using only preconceived ideas. approaches. In this method, the investigator estab-
Ideally, as with the traditions of cultural anthro- lishes and sustains a many-sided relationship with
pology and qualitative sociology, researchers will a social phenomenon in its natural setting for the
spend considerable time in the field. Depending on purpose of developing a scientific understanding
the study purpose and the size of the research proj- of that phenomenon. Participant observation thus
ect, this could mean anywhere from several weeks allows qualitative researchers to see how things
to a year or more participating in the routine activ- really are (observation) and also check in (partici-
ities of the group and setting. This immersion pation) with knowledgeable insiders who can con-
experience offers the qualitative researcher the firm, or not, the researchers’ emergent insights,
opportunity to learn directly from his or her own understandings, and explanations as they experi-
experience. The implicit premise is that a truer ence the the social phenomenon first hand.
understanding of the phenomenon studied will be There can be several degrees of participant
obtained. This approach requires the researcher to observation, sometimes in sequence. In its earliest
be very self-aware and reflective, not only observ- stages, the qualitative researcher using a participant
ing the actions and behaviors of others, but also observation approach enters the field with a broad
appreciating and recording his or her own personal topic of interest but no predetermined categories or
thoughts, experiences, and reactions. strict observational checklists. At this stage, the
Chapter 20 Gathering Qualitative Data 343
investigator is intent on identifying and describing • Development of useful observational measures,

the actions of the participants in pursuit of mean- • Accurate documentation of observations in dis-
ingful goals, values, and ideals. Noting these pat- tracting conditions,
terns and reviewing them systematically over time • Managing requests to align oneself with one per-
leads to the development and use of more highly son or group versus another,
specified observational checklists and perhaps even • Simultaneously participating with and recording
some direct questions to key participants in the observations of study participants at the same
setting. time as experiencing the phenomenon oneself
The ultimate goal of the qualitative researcher and recording it in a complex and ever-changing
using participant observation is to understand and environment, and
explain the sociocultural values, norms, and expec- • Documenting additional fieldnotes, questions,
tations that underlie the personal beliefs and quandaries, and complexities after each data col-
individual actions observed. As time in the field lection episode.
proceeds, it is possible to understand more fully
both what one is observing and what it all means. It must be recognized that qualitative studies
As stated at the beginning of this chapter, qualita- utilizing observation and participation have both
tive research is responsive to the changing condi- strengths and weakness. Their greatest strength is
tions of the research setting and the topic studied. that they provide very rich and detailed data in
Thus, as new data are obtained and synthesized, the settings and situations in which subjects are
iterative process of uncovering and then confirming observed. While other methods such as in-depth
emergent understandings (that is, analysis) begins. interviews would contribute information from one
Many qualitative studies use observational individual’s point of view, observation and partici-
methods, at least to some degree. For example, pation allow the investigator to study interactions
when conducting an in-depth interview, a qualita- between multiple persons and between persons in
tive researcher is engaged in a shared social scene specific physical and/or social environments.
with customs and expectations about how to inter- Furthermore, observation and participation meth-
act, be respectful, and reciprocate. The nature of ods are necessary when individual interviews are
the social interaction is part of the data collection not possible because of the limited capacity of the
and the data to record. Also, investigators are atten- participants. For example, some research partici-
tive to the participants’ body language and affect— pants may not be able to provide full and complete
not only their words. In whatever ways observation information about their experiences owing to the
is used, it is demanding on the part of the resear- nature of their disability or health condition. When
cher. Observational researchers confront psycho- this is the case, alternate or at least supplemental
logical discomfort and fatigue, unexpected ethical data gathering strategies are necessary. Clearly,
dilemmas, and may even expose themselves to participation and observation studies are very time-
unanticipated danger. In addition, interviewers are and labor-intensive. These types of studies also
required to responsibly and as completely as possi- require a great deal of advance preparation, includ-
ble record the social happenings around them at the ing, at times, special permissions to allow access to
same time as they are trying to find the big picture the setting and group members of interest. In addi-
analytically. This is a real challenge since studying tion, the presence of the researcher in both partici-
human beings in their natural environments (e.g., pation and observation studies can lead study
in home settings, rehabilitation centers, schools, participants to alter their behavior in order to “look
etc.) by definition implies complexity and a fast- good” in the eyes of the researcher or provide what
paced and ever-changing social scene. they perceive to to be as “the right answer” (i.e.,
The following are key activities of qualitative social desirability bias or Hawthorne effect). This
researchers using participation and observation: can pose a serious threat to the validity of study
findings.
• “Gaining entry” to the setting, Qualitative researchers must be aware of the
• Negotiationg and establishing a social identity in potential for study participants to influence the
the setting, events that transpire during data collection as well
• Sustained engagement in the research setting as the potential for those influences to impact the
over time and learning how to maintain good study findings. Fortunately, while individuals
relationships in that community or group, being studied may be conscious of the presence
• Active and genuine involvement with group of the researcher and consciously edit or restrict
members, their more extreme or controversial opinions and
An Example of Research in Which Both Observation and Participation Were Key Data Collection Methods
Siporin and Lysack (2004) were interested to know how the self-perceived quality of life (QOL) of women
with developmental disabilities working in supported employment differed from those working in sheltered
workshops. In this study, the principal investigator (Siporin) devoted weeks of inconspicuous observational
data collection in the places of employment of her research participants. In the daytime, she accompanied
her research participants as they worked in small enclaves as housekeepers in local hotels and as food
preparation assistants in fast-food restaurants. In the evenings, she observed their leisure activities with
their family and friends.
This design choice provided invaluable data about the structures and routines that characterized her sub-
jects’ work and home life. For example, the pace of work, chain of authority at work, and the complexity
of numerous pieces of equipment in the workplace were far more evident to the investigator after observa-
tion than before. So were the stresses these women experienced because of unreliable transportation.
Frequent but unpredictable disruptions in bus transportation wreaked havoc with the official work schedule
to which they were supposed to adhere. This caused tensions and occasionally heated arguments and cer-
tainly reduced the more positive attitudes these women might have held toward the supported employment
experience. A lack of available safe transportation also curtailed participants’ leisure activities (e.g., shop-
ping, going out to movies, bowling).
Other observations in the home setting provided essential data with which to understand these women’s
QOL. These data painted a picture about the love, respect, and acceptance these women enjoyed and what
specific roles and responsibilities they assumed within their families. These data provided a valuable contrast
or counterpoint against which to compare the workplace observations. Participant observation data in this
study revealed how agency policies and governmental regulations impacted participants’ QOL. For example,
it was learned that if a supported employment worker actually improved her skills to the extent that she was
promoted and received a pay raise, she would likely become Medicaid ineligible, thus putting in jeopardy
insurance coverage for significant medical expenses including, for example, hospital care, durable medical
equipment, prescription medications, and eyeglasses. This was not something these women (or their families)
were prepared to do. Thus, despite their wishes and efforts to become more self-sufficient, the women with
developmental disabilities stopped short of any “success” that would put their Medicaid eligibility at risk.
Without the method of participant observation, these policies and regulations would not have been identified;
nor would their consequences on the participants’ perceived QOL be fully understood.
actions, over time study participants will become most detailed information possible about the topic
less concerned about the presence of the investiga- at hand. Interviews vary with respect to their a pri-
tor and reveal their “true” thoughts and behaviors. ori structure and in the latitude the interviewee has
As with all data collections methods, however, the in responding to questions.
investigator faces trade-offs with every method- Generally speaking though, in-depth interviews
ological choice. Table 20.1 summarizes the stren- can be developed along a continuum. At one end of
gths and weaknesses of participant observation. the continuum is the most open-ended and unstruc-
The most successful qualitative researchers will tured conversational approach, in which the inter-
consider all aspects of their topic, its purpose, and view proceeds more like a casual visit. Another,
the research setting they are in, and then choose slightly more directed approach includes having
their research methods accordingly. some prepared but still unstructured topics about
which to inquire. A semistructured interview pro-
vides even more structure by using a combination
In-Depth Interviews
of fixed-response and open-ended questions. At
In-depth interviews are most often conducted the far other end of the continuum is the structured
in face-to-face situations with one individual, interview, in which the questions and response cat-
although they can be conducted by telephone and egories are virtually all predetermined.
in a group situation (see focus group interviews Irrespective of their specific form, data gath-
below). The goal of the in-depth interview is to ered using in-depth interviewing methods are typ-
delve deeply into a particular event, issue, or con- ically recorded using audiotapes and written notes,
text. Interviews differ from participation and and sometimes even video recording when the
observation primarily in the nature of the interac- study purpose demands. Audio and video record-
tion. In the in-depth interview, the purpose is to ing is used to increase the amount of data available
probe the ideas of the interviewees and obtain the for later analyses and to provide verification of
Table 20.1 Participant Observation: Relative Strengths and Limitations

Strengths Weaknesses
“Rich description,” that is, detailed information about Few opportunities to probe the specific meanings of
the social phenomenon studied. participants’ actions and behaviors if data are not
Very natural (i.e., valid) data are obtained in these supplemented by other methods such as
normal (versus laboratory) settings. interviews.
Only method that permits study of people’s actual Responses may be influenced by social desirability
behaviors (versus merely their attitudes and bias.
beliefs) in a particular physical and social Can be complex to analyze these data without other
environment. data to confirm the observed and/or investigator
May be the only form of data collection possible when experienced data.
study participants themselves cannot be Often requires advance planning, special
interviewed because of their disability. permissions, and more detailed IRB/ethics
proposals before approval for the study is granted.
data accuracy. For example, audiotapes can be to reveal more specific information about his or
used to create data ranging from a paraphrase or her personal experiences and circumstances. Other
summary of a conversation all the way to micro- appropriate probes might be: “Can you tell me
scopically detailed data on tone and speed con- more about your reasons for this answer?” and
tours, including the length of pauses and speed of “Could you give an example?” Most importantly,
talking. Verbatim transcripts of interviews can be however, the unstructured or conversational inter-
analyzed at a later point in the study, at varying view conveys the attitude that the participant’s
amounts of detail. views are valuable and useful, and the task of the
researcher is to capture these views as completely
Unstructured Interviews and accurately as possible.
Unstructured interviews resemble guided conver- Semistructured and Structured Interviews
sations. Such interviews typically include a rela-
tively short list of “grand tour” general questions These types of interviews permit modest to maxi-
(sometimes referred to as an “interview guide”), mum investigator control over the design and
and interviewers generally respect how the inter- sequence of research questions. In structured inter-
viewee frames and structures their responses. For views, interviewers are trained to ask each question
example, an unstructured interview about the ade- precisely as written. This does not mean, however,
quacy of home care received by a recently dis- that the interview consists only of fixed-response
charged stroke patient may begin with a general questions. All interviews, whether unstructured or
question such as, “How is your home care going structured, can include both fixed-response and
since you got out of the hospital?” The answer pro- open-ended questions. For example, a qualitative
vided may be brief or lengthy, but no matter what interview focused on the meaning of disability
the interviewee says, the interviewer accepts the might include a fixed-response question such as,
words used and explanations as offered. Still, when “Do you think of yourself as the same person you
appropriate, some gentle probe style follow-up were before your injury?” for which the response
questions may be used to delve more deeply into set could be limited to “yes” versus “no.” The
interviewees’ initial responses, seeking examples, response categories could also be categorical and
explanations, and rationales for expressed beliefs ordered, for example, “Yes, just the same,” Yes,
and behaviors. For example, an appropriate probe somewhat the same,” “No, not really the same,”
after a question such as, “Have you encountered and “No, absolutely not the same at all.” Finally,
any unexpected surprises with your home care?” the question could be entirely open-ended. As their
might be something like “Could you describe one name suggests, semistructured interviews typi-
or two of these surprises?” Optimally, probes like cally include a combination of fixed-response and
this are value neutral and function simply to elict open-ended questions with a variety of response
more information. They are not meant to direct the categories, some of which border on the kinds
respondent toward any particular topic or value of response categories typical of surveys. Struc-
judgment, but rather to encourage the respondent tured interviews provide the least interviewer
flexibility, as all of the questions are predetermined

and the interviewer is encouraged not to deviate
at all from the prescribed interview protocol. Table
20.2 summarizes the strengths and weaknesses
of fixed-response versus open-ended interview
questions.
Interviewing has strengths and limitations.
Interviews involve personal interaction and coop-
eration; trust and rapport between interviewer and
interviewee are essential. Interviewees may not be
comfortable sharing all that the interviewer hopes
to explore, or may be unaware of specific facts and
experiences that are relevent to the study purpose
but are not revealed in the interview.
Lack of skill and training on the part of the
qualitative researcher may also lead to poorly
developed questions, and inadequate interviewer
training may lead to inadequate probing and fol-
low-up questions during the interview itself. To be
successful, qualitative interviewers must work to
develop superb listening skills and be skillful at
personal interaction, question framing, and gentle
probing for elaboration. One of the great benefits
Figure 20.1 Research team members practice of interviews is the volume of detailed data that
conducting a semistructured interview using an can be obtained. However, even optimal interview
interview guide. data are time consuming to analyze, and depend-
Table 20.2 Fixed-Response Versus Open-Ended Interview Questions:

Relative Strengths and Limitations
Strengths Limitations
Fixed-response questions
Quick to ask and answer. Unclear if the respondent under-
Large cohort of data can be obtained stands the question as the
in a short time. researcher intended.
By forcing “one best answer” a respon- Relevant information may not be
dent’s basic position is clarified. collected.
Responses can be more easily Responses may be influenced
compared across groups. by social desirability bias.
Statistical analysis can be conducted
on numerical data.
Open-ended questions
Respondents’ interpretation of the Can be very time consuming to
question is more obvious. both gather data and analyze it.
Issues of importance to the participant Respondents may not wish to
are more likely to be identified and reveal personal, sensitive,
described. or provocative information.
There is sufficient time and interviewer Data from open-ended questions
awareness to record nonverbal are not easily comparable
behaviors and emotional res- across groups.
ponses (e.g., tears, anger, Without well-trained interviewers,
confusion, etc.). too much of the data gathered
With a skilled interviewer, sensitive may be “off-topic” and not
topics can be more easily probed adequately address the study
and explored. aims.
ing on the expertise of the interviewer, too much of standings or experiences (true or not) that make it
these data will not be central to the study aims and harder to get an informant/participant to fully ver-
thus may not illuminate the study topic at hand. balize and explain experiences or beliefs that are
Transcribing and analyzing less relevent data is “obvious” or taken for granted. Qualitative investi-
costly and time consuming. gators conducting interviews with these special
Finally, there is the issue of the quality of inter- populations will need to address a wide array of
view data. When interviewing is the sole method considerations such as these in the institutional
of qualitative data collection, the qualitative review board (IRB)/ethics approval process that is
researcher obtains the interviewees’ perspectives virtually always required in advance of approvals
on events and issues. This is usually absolutely to conduct research studies sponsored by universi-
appropriate. However, if the study aims require ties, whether these studies take place in a reha-
more objective confirmation of events and issues, bilitation facility, or a person’s home, school, or
the qualitative investigator may need to triangulate workplace, and so forth.
his or her interview data with data gathered The following are key activities of qualitative
through other methods. The process of triangula- researchers before and during in-depth interviews:
tion (discussed further below) provides an addi-
• Identifying an appropriate study sample,
tional methodological check on the validity and
• Logging communication during recruitment
reliability of study data.
efforts,
In certain circumstances, qualitative researchers
• Establishing comfort, rapport, and trust with
will have to consider a wider array of factors when
interviewees,
planning and carrying out interviews. This is true
• Developing the interview guide (conversational
when interviewing special or vulnerable popula-
style) or explicit interview protocol (structured
tions, for example, children or persons with spe-
style interviews),
cific types of disabilities. For example, interviews
• Skillful listening and question asking,
with children require greater care during the
• Judging appropriately when and how and if to
construction of interview questions to ensure they
probe and pursue interesting turns in the inter-
are not too complex for a child to understand.
view (e.g., disclosure of unexpected information,
Depending on the topic, questions may also pose
controversial or provocative opinions and state-
unique risks to the child. For example, particular
ments, etc.),
interview questions about severe burns they
• Presentation of the self as a competent and skill-
sustained in a traumatic house fire, or injuries sus-
ful interviewer,
tained in a serious car accident, may create emo-
• Note taking comprehensively while astutely
tional upset, fear, or even psychological distress,
questioning and listening, and
depending on circumstances associated with those
• Writing-up summaries and notes about what the
events. Similarly, interviews with persons with spe-
interviewee said as well as reflective personal
cific types of physical and cognitive disabilties
observations.
demand special consideration.
Qualitative interviewing in these contexts Table 20.3 summarizes the strengths and weak-
requires not only more careful interview item con- nesses of qualitative interviewing. In addition,
struction in advance of the interview, but also train- there are two other kinds of interviewing that
ing to prepare the interviewer for a range of deserve brief mention owing to their frequency
expected and unexpected challenges during data of use and potential to contribute unique data
collection itself. For example, persons with stroke to a qualitative research project. These are focus
may have comprehension and speaking difficulties group interviews and key informant interviews.
and it may be necessary to move from more open- Table 20.4 summarizes their strengths and weak-
ended to more fixed-response style questions. In nesses.
addition, the choice of interviewer is a considera-
tion, even if the answer is not. Sensitive topics
Focus Group Interviews
about sexual function or victimization, for exam-
ple, clearly require attention to who can best make The focus group interview emerged from con-
the interaction comfortable, such as a same-sex or sumer research in the 1950s. Consumer research
same-age person. Alternatively, choosing someone showed that people tended to make decisions
with the same ethnic background may not always within a group context, and therefore, to under-
provide better data. In studies of minorities there stand consumer behavior, consumer preferences
may be unstated assumptions about shared under- needed to be studied in a group setting. The same
Table 20.3 Qualitative Interviewing: Relative Strengths and Limitations

Strengths Weaknesses
The researcher can gather detailed information Lack of efficiency: It takes much more time to gather
on the topic of interest. This method also data from individuals than from groups.
permits exploration of additional topics All data are “self-reported.”
generated by individuals’ responses. There is no opportunity to confirm or disconfirm the
Optimal confidentiality. personal values, attitudes, and beliefs or the
Assuming expert interviewers who establish a related background and events related by
respectful and trusting relationship, more participants, unless triangulation is used.
personal information is revealed in Vast amounts of data are generated by interviews
interviews than may be the case in focus and they are very costly to transcribe.
group situations. Data analysis is costly and time-consuming, even
with excellent data.
principle holds in occupational therapy research. optimal context to gather data can sometimes be a
Occupational therapists recognize that many context in which numerous and varied perspectives
aspects of decision-making, especially decisions can be heard at the same time. This is what makes
related to health and disability, are made with fam- the focus group a popular method of gathering
ily members and other valued persons. Thus, the qualitative data.
Table 20.4 Focus Group and Key Informant Interviews: Relative Strengths and Limitations
Strengths Limitations
Focus groups
Efficiency: The researcher can gather The number of questions asked must
data from multiple persons instead be minimized since it takes so
of only one. much more time to gather multiple
Because of the dynamic interactions responses. With six people in a
across group members, there is the one-hour focus group, an inter-
potential for contradictory opinions viewer would have difficulty asking
and not only consensual views to any more than 10 questions.
be shared. Responses may be overly “sanitized,”
More valid data: Group interactions that is, negatively influenced by
tend to hone in on the most salient social desirability bias.
issues, therefore making it relatively Difficult to control:
easy to identify a relatively Unexpected diversions, interperso-
consistent shared view among nal conflicts, etc. can distract
interviewees. group members from its purpose.
Skilled facilitators are essential to
quality data.
Difficult to take notes with so much
going on.
Limited ability to protect confiden-
tiality.
Key informant interviews
Insight: Key informants are well Often requires a referral from a
informed and can shed consid- prestigious and respected “others”
erable light on the history and before access is achieved.
policies of groups and organiza- Requires great skill and effort to
tions.They may be elites or simply “manage” the egos and
ordinary informants who are identi- personalities of these important
fied for more intense in-depth leaders and spokespersons.
follow-up.
Focus group interviews are conducted with a Key Informant Interviews

small group of people on a specific topic. Typically
four to six people participate in the interview, Key informants are used in at least two ways. In
which lasts about 1 to 2 hours (although focus one form, “elite” interviews are conducted with
groups can include eight to twelve participants, or individuals considered to be influential, prominent,
even more). Participants in focus group interviews and/or well-informed people in an organization or
are usually a homogeneous group who are selected community. Key informants (elites) are specifi-
because of their knowledge about the study topic. cally selected for interview on the basis of their
Participants in focus groups get to hear each oth- expertise in areas relevant to the research. In
ers’ responses and contribute their own responses another approach, key informants are selected
in light of what others have said. Often the ques- from the larger sample for more extended and in-
tions in a focus group are deceptively simple. The depth discussion.
aim is to promote the focus group participants’ Key informant interviewing has many advan-
expressions of their views through the creation of tages. Valuable information can be gained owing to
a supportive environment. the positions these persons hold in social, political,
As with other methods, the focus group has or administrative realms. Key informants can also
countervailing strengths and weaknesses. To a provide an overall view of the organization and its
great extent, the advantages of the focus group activities, policies, past history, and future plans
interview are the disadvantages of the individual from a particular perspective. The disadvantage of
interview and vice versa. The primary advantage key informant interviewing is that is often difficult
of focus group interviews is their efficiency. A to gain access to this group. They are often very
great deal of information can be gathered quickly. busy people and difficult to contact, especially ini-
In addition, focus groups provide the opportunity tially. The interviewer may have to rely on an
for data to emerge as a result of the dynamic inter- introduction or recommendation of another elite to
actions between group members. However, it is a gain access/entry to the study setting. Another dis-
common misperception that focus groups are only, advantage of interviewing key informants or elites
or best, for learning about shared opinions or main is that the interviewer may have to radically adapt
themes. Further, focus group interviews have high the interview to suit the wishes and predilections
face validity because they are conducted in a natu- of the person interviewed. Although this is a possi-
ral social setting and under more relaxed circum- bility with all individual interviewing, elites
stances than individual interviews. may be especially bright, thoughtful, and articulate
The downside, of course, is their management. and will resent poorly conceived or ill-phrased
Particularly strong personalities can dominant the questions.
focus group and some individuals may not have an Well practiced at public meetings and persua-
opportunity to express their views, especially if sive arguments, key informants may desire an
they are in disagreement with a dominant member active interplay with their interviewer and be
of the group. Optimal data from focus groups unhappy and uncooperative if the interviewer is
require a skilled facilitator so that there are oppor- not superbly prepared or not capable of a pleasing
tunities for all members to participate and con- intellectual exchange (see Table 20.4). Thus, con-
tribute. Because the facilitator/interviewer has less ducting key informant interviews can put a consid-
control over what is discussed in the focus group, erable strain on the interviewer, who must
the interview can result in lost time as irrelevent demonstrate competence in inquiring about the
and dead-end issues are discussed. It may also be subject matter at hand, and, at some level, be pre-
difficult to manage the group conversation at the pared to entertain his or her interviewee with
same time as recording people’s opinions on the shrewd questioning, and even sharp debate. This
topic at hand, which is why a focus group con- hard work can pay off, however, as elites are typi-
venor and a note-taker are often used. Finally, cally intelligent and quick-thinking people and
data gathered using this method can be very diffi- are completely at home in the realm of ideas, poli-
cult to analyze because context is essential to cies, and generalizaitons.
understanding individual comments and it is sim-
ply not possible to delve into the background con-
Written Documents and Material Objects
text for every opinion offered without the entire Researchers may choose to gather their data using
focus group grinding to a halt. The feature box on sources of already existing information, or sec-
the next page illustrates the usefulness of focus ondary data. Analysis of available documents (as
group interviews. opposed to primary data newly collected by the
An Example of Focus Group Interviews
Dillaway and a research team from the Barbara Ann Karmanos Cancer Institute in Detroit recently con-
ducted a study about African-American men and prostate cancer (Dillaway et al., 2005). The purposes
of this study were threefold:
1. To explore African-American men’s awareness of prostate cancer,
2. To determine the barriers to their participation in prostate cancer research trials, and
3. To begin to develop new ideas for recruitment strategies to encourage more African-American men to
participate in research trials.
In the pilot stage of the research, focus group interviews were used since research team members wanted
to make sure that they were exploring a large range of potential barriers to recruitment and were unsure
what recruitment strategies would increase African-American men’s participation in research.
Not wishing to assume they already knew the answers to these questions, focus groups were selected
so the investigators could see how African-American men themselves conversed about these topics. Eight
focus groups with 61 African-American men were conducted. Focus group participants highlighted a range
of barriers to their participation in research trials, many of which were expected. For instance, in every
focus group, participants highlighted their mistrust of doctors and researchers as a reason for their reluc-
tance to participate, sometimes referring to the infamous Tuskegee experiment as a reason for this mistrust.
In that experiment, which is discussed in detail in Chapter 29, African-American men with syphillis were
not informed that they were taking part in a research study on the long-term consequences of untreated
syphillis.
Since the investigators were well aware of the historical injustices perpetrated on this minority popula-
tion in the name of health research, they expected participants to highlight such barriers. Yet, in the course
of conversation, African-American men also highlighted reasons why they would participate in research
trials. For instance, in all eight focus groups, participants discussed how important it was for African-
American men to “step up” and be models for future generations. They also discussed how important it
was to keep themselves healthy so that they could ensure their ability to finish raising their children.
Finally, they highlighted the fact that, in order to remedy the lack of knowledge about African-American
men’s health, they knew they had to overcome their mistrust of research studies and participate in research,
if only to make certain that their children would be healthier than they were.
As a result of the focus groups, the research team learned that individual participants had a higher aware-
ness of prostate cancer and a greater desire for more research than anyone of the team expected. In addition,
while the researchers expected participants to highlight barriers to participation in research, they did not
expect participants to highlight so many reasons why they would be willing to participate. Because most
existing research only highlights the barriers to African-Americans’ participation in research, the researchers
did not even ask any questions about why individuals might want to participate in research trials. Thus, if
the researchers had not conducted focus groups, a method of data collection in which new meanings can
be highlighted within group conversation, they may not have realized that individual African-American men
have many reasons to participate in prostate cancer research and, more specifically, may not have realized
that African-American men’s connections to their families and to racialized communities could directly
facilitate individuals’ participation in research trials. These data would not have been identified without
the focus group method.
researcher such as interview transcripts) can ized analytic approach called content analysis. The
include diaries and personal journals, historical raw material of content analysis may be any form
documents, minutes of meetings, Web sites, adver- of communication or text, although written forms
tisements, annual reports, newspapers, magazines, are most common. Historically, content analysis
or political speeches. Analysis can also utilize emphasized systematic, objective, and quantitative
materials or cultural objects and artifacts. counts and descriptions of content derived from
Researchers who use as their primary or sole researcher-developed categories. Today, content
method a review of documents or studies of inani- analysis can be exclusively numeric or exclusively
mate objects can be described as using an unobtru- interpretive—largely dependent on the theoretical
sive methodology. Observation, described earlier, traditions dominant within the researcher’s disci-
is also an unobtrusive methodology. It is consid- pline. For example, quantitative political scientists
ered as such because there is minimal investigator might rely exclusively on numerical counts of
disruption to the study participants and setting. words in a political speech and use the evidence of
The use of documents often entails a special- the amount of particular forms of speech to argue
that a particular politician holds a particular view. munity integration after spinal cord injury, Lysack
In contrast, an anthropologist or a historian will be and Luborsky’s (2004) research team is analyzing
far more interested in the meaning of the words drawings made by its research participants depict-
conveyed by the text than by the number of times ing the meaning of this injury. While in many
a phrase is spoken. cases these drawings are only crude sketches, the
One can easily imagine an occupational therapy visual representations offer a medium by which to
researcher designing a study using methods of express and represent experiences and ideas not
document review and content analysis. For exam- readily put into words. Drawing allows partici-
ple, if a researcher was interested in understanding pants a nonverbal way to “tap into” and express
the meanings of mobility aids such as walkers, deeper feelings. Interviewers on this project are
crutches, and wheelchairs to adults with mobility trained to ask questions after the drawing is com-
disabilities, they could design a study in which the pleted to elicit responses about how and why the
family photographs of persons who have lived drawings were generated as they were. The draw-
with mobility impairments all of their lives were ings themselves, coupled with the participants’
reviewed and analyzed. The photographs would responses, shed considerable light on topics
likely reveal a number of insights including how including the level of responsibility for the injury
the devices were commonly used, and what activi- felt by participants; the degree of resentment and
ties these devices were helpful in facilitating. hostility aimed at those who have provided, or cur-
The qualitative researcher in this type of study rently provide, medical treatment and care; and in
would also note who was in the photographs, in a somewhat more abstract way, the existential
what locales the activities occurred, and whether place these persons occupy when they assume the
there was any evidence of attempting to hide the label “disabled person” in an able-bodied world.
mobility devices, perhaps related to embarrass- As stated earlier, methods that use written doc-
ment, shame, or stigma. Of course, a study that uments, visual materials, and cultural objects have
supplemented this analysis with individual inter- their advantages. For example, they are usually
views would result in potentially more useful data more quickly moved through the human subjects
than relying on the photographs alone. When mul- IRB/ethical review process because they may be
tiple methods of data collection are used in the regarded as posing less “risk” of harm to the par-
same study, the approach is called triangulation. ticipant. On the downside, without supplementa-
This is one way of increasing the rigor of a quali- tion with other data gathering methods, it may be
tative study, a technique that is discussed in more difficult for the qualitative researcher to clarify the
detail below. meanings of these materials and objects to those
Reviews of material and cultural objects are not who possess them or are influenced by them. As
restricted to those provided by individuals in one- mentioned previously, triangulation of data
on-one situations. Brochures, descriptions of pro- sources and data methods are two ways of increas-
gram services, and historical documents developed ing the methodological rigor of a qualitative study,
by organizations, health programs, or social move- issues that are discussed below.
ments can be studied too. For example, in a study The following are key activities of qualitative
described in Lysack and Kaufert (1999), Lysack researchers engaged in document reviews and
reviewed books, promotional, and educational analysis of material and cultural objects:
materials used by activists within the independent
• Selecting appropiate documents or objects,
living movement and professional proponents of
• Gaining permissions to observe and study them,
international community-based rehabilitation. The
• Selecting and using analytic methods best suited
goal of this research was to understand how con-
to the kind of data, and
sumer organizations and professional and policy
• Identifying additional means by which to con-
bodies used the language and imagery of “commu-
firm their meanings.
nity” to guide their disability-related educational
and rehabilitation activities. Lysack used discourse
analysis in this study, a specific method by which
special attention is paid to the process of spoken
and unspoken communication. This method
Ways of Strengthening the
revealed dramatic tensions between the service Quality of Qualitative Data
delivery models of the two groups, tensions that
were directly linked to the fundamental views held In all research, the question must continuously be
about the meaning of community. asked: “How trustworthy are these data? The trust-
In another example, an ongoing study of com- worthiness of qualitative data, or how sure we can
be that the data are accurate and reflect social, Managing Bias in Qualitative Research
cultural, and lived reality, is essential since the
goal of the qualitative researcher is to capture and Bias is a type of prejudiced consideration or
communicate experiences, meanings, and social judgment. Several types of bias can negatively
situations. impact qualitative studies: (1) Over-reliance on
Several important methodological actions can accessible research participants or favoring more
be undertaken to enhance data trustworthiness, dramatic events and statements involving
including: research participants and the context of study; (2)
biasing effects produced by the presence of the
• Interviewer training, investigator in the research site, that is, the
• Prolonged engagement in the field, Hawthorne effect; and (3) biases stemming from
• Reflexivity, the influence of the participants and the research
• Triangulation, site on the investigator. In all of these situations,
• Stakeholder checks, and the investigator may be biased if he or she is
unaware of the social influences that various
• Audit trails.
players in the research enterprise exert and are
These methods can be employed whether the subject to. In qualitative research, biases must be
data are collected via observation, participation, recognized and accounted for. Qualitative investi-
interviews, or review of existing documents, mate- gators have an onus to report on the reasons for
rials, and objects. having chosen their particular topic for study,
their design choices, as well as decisions about
sample and methods. All need to be transparent
Interviewer Training so that the reader of the study can judge for
The importance of interviewer training should not themselves the quality of the results.
be underestimated. The time and financial costs of
interviews are great. Thus, significant time spent
training interviewers is a critical investment to
ensure that data are obtained most efficiently, must also learn to practice a quiet awareness of the
without compromising quality. trade-off between patience and efficiency.
Interviewer training includes technical skills on
how to ask the interview questions and the use of Prolonged Engagement in the Field
follow-up probes to elicit more detailed explana-
tions. It also includes training on appropriate Prolonged engagement is the phrase used to des-
behaviors needed to gain entry into the research cribe the period of time spent in the field observ-
context. The latter includes the interpersonal ing the phenomenon of interest. The amount of
behaviors with research participants that occur time in the field varies, depending on:
during subject recruitment and data collection.
• The nature of the inquiry and its scope,
Interviewers must also be carefully trained to
• The design of the study,
ensure a consistent style of data collection across
• The time available to the investigator, and
research participants. This requires an element of
• The time available to the research participants
standardization in question asking and probing. At
themselves.
the same time, however, the interviewer needs to
be flexible and responsive As a rule of thumb,
when the need arises. however, data collec-
These are skills that Trustworthiness in qualitative tion continues in the
can be learned. For exam- field until saturation is
ple, mock interviews research is crucial since the reached. Saturation is
under supervision are an primary contribution of quali- the point in the data col-
excellent way to learn to lection period when the
ask questions skillfully, tative research is to capture researcher is gaining lit-
to listen carefully, and to and convey the experiences, tle or no new informa-
pose appropriate follow- tion. Since the criteria
up questions. Interviewers
meanings, and events are data-based and not
especially must learn how encountered in the field. time-based, there is no
to facilitate a somewhat fixed standard duration
conversational style dur- of time in the field.
ing the interview at the same time that they com- Investigators need to evaluate their data for signs of
municate respect for the research participant. They diminishing gains or saturation. When investiga-
tors are no longer adding new insights, or no longer method alone. The purpose of triangulation is to
puzzled by what they observe and are able to pre- validate a particular finding. Triangulation of data
dict what is going to happen next, saturation is methods increases the chances that the conclusions
likely being reached. This means that it is nearing reached are better able to represent the whole set
the time to leave the field and begin writing up the of relevant features, or are “true” due to the com-
results of the observations in the field. plementary strengths of the respective methods.
While triangulation of methods is the most com-
Reflexivity mon form of triangulation in qualitative inquiry,
there are other forms of triangulation, including tri-
Reflexivity refers to a deliberate and systematic
angulation of data gatherers. Triangulation of data
process of self-examination. It involves a continu-
gatherers refers to the use of two or more individu-
ous cycle of:
als who have independently observed and recorded
• Seeking insights from inward reflection on the their own field notes of a phenomenon. Data qual-
experiences of working in the outside world, and ity is enhanced by comparing their observations
• Looking back at what is being learned outside in afterwards, and resolving differences through dis-
light of the inner experience. cussion. The same process can be undertaken dur-
ing the data analysis stage of a research project.
This process is necessary in qualitative research Triangulation in this sense refers to the use of mul-
because the investigator will encounter a wide tiple persons to do the data analysis. Sometimes
array of thoughts, feelings, and reactions to people this is called peer debriefing. Irrespective of its
and events in the course of data collection. While title, the process is one of multiple investigators
such feelings and reactions are not a major concern simultaneously but independently engaging in the
of quantitative researchers, they are of great analytic process.
importance to qualitative researchers. Peer debriefing is very valuable in qualitative
First, these data are important in and of them- inquiry because it provides a means by which
selves. Especially in observational and participa- areas of disagreement and controversy are high-
tion studies, these are the only data that will be lighted. Again, as in data collection, the use of
collected and analyzed. Second, the reactions and multiple analysts provides a mechanism for con-
views of the qualitative researcher—both gleaned trary views to arise and receive careful review.
as a direct result of participation in the study and Peer debriefing may be the only way for opposing
brought to the study in the form of preexisting atti- opinions to be heard and contrary explanations for
tudes and values—have the potential to color the phenomenon aired. The use of peer debriefing
data collection and analyses processes. Qualita- sometimes leads to additional data collection as
tive researchers acknowledge that this sort of influ- the need to clarify conflicting data and conflicting
ence is real and has the potential to influence study views becomes apparent. The entire process
findings. strengthens the legitimacy of the final version of
They also know that while the “bias” cannot, study findings. It is an important point of departure
and in fact, should not be eliminated, it is impor- from more standardized forms of data collection
tant to identify it and examine its influence on that rely on a standardized fixed set of measures
emerging interpretations. Personal diaries are fre- conducted identically with each participant.
quently used by qualitative investigators to note
these attitudes, feelings, and reactions. Later, the
diaries themselves become a source of data and are Stakeholder Checks
an important check on the development of research
conclusions. Also called member checking, stakeholder checks
is the process whereby the investigators check out
their assumptions and emerging interpretations
Triangulation
about the data with the original stakeholders who
Triangulation is another technique used to increase provided the information. This is vitally important.
the accuracy (or trustworthiness) of data gathered. Not only does it ensure accuracy of the facts and
Triangulation refers to the use of two or more information gathered in the study, but it also helps
strategies to collect and/or interpret or analyze ensure that the investigator’s conclusions make
information. For example, in a single study trian- sense from the perspective of the persons who
gulation may mean using interviews to learn what experienced those events.
people say to investigators about using a wheel- As with prolonged engagement in the field,
chair and observation to see what they actually do there is no magic amount of stakeholder check-
and tell to others, instead of relying on either one ing that is “right” for every study. In very small
studies, it may be possible to return the interview ruin even the best developed scientific design.
transcript to every person interviewed for checks Thus, entering and exiting relationships with par-
on accuracy of transcription and to return drafts of ticipants at the field site requires careful and full
emerging research findings to the original partici- consideration of practical details.
pants too. In a large study, however, this is usually Adequate preparation is necessary to ensure
not possible and sometimes only a small portion of that all members of the research team (subject
the data (commonly about 20%) are returned to the recruiters, interviewers, etc.) present themselves
original stakeholders for this sort of check. and treat others professionally. This includes, for
example, one’s style of dress and behavior, since
Audit Trail appearance and verbal communication send very
important messages to the research participants
An audit trail is a systematically maintained set of about the importance of the study, and the investi-
documentation, typically including: gator’s respect for them as human beings. A rea-
• All data generated in the study, sonable rule of thumb is to dress conservatively but
• Explanations of all concepts and models that professionally. It is important to be attentive to the
shaped the study design, values and style of the individuals and organiza-
• Explanations of procedures used in data collections where the data are gathered. This can require
tion and analysis, “dressing up” in studies conducted in more formal
• Notes about technical aspects of data collection settings, as well as “dressing down” in studies with
and analysis as well as decisions taken through- teenagers or where circumstances dictate.
out the study to refine data collection procedures The hazards associated with being both too for-
and interpretations, mal and too informal can be overlooked by quali-
• Personal notes and reflections, and tative researchers. When project staff misjudge the
• Copies of all instruments and interview protocols impression required to be taken seriously in the
used to collect study data. research setting, the entire study can be put in
peril. In those cases, organizations will be impos-
The audit trail can be used by the researcher as sible to penetrate and informants impossible to
a means of managing record-keeping and encour- recruit for the simple reason that participants per-
aging reflexivity about a project and its goals. It ceive the research is not sufficiently “in tune” with
also permits a third-party examiner to review all them to reward the project with their participation.
aspects of the conduct of a qualitative study and If research participants do not accept the legiti-
attest to the use of dependable procedures. In this macy of the researchers or do not take the research
way, the audit trail functions as a means of relia- endeavor seriously, they may not participate, or,
bility checking on both the procedures and conclu- alternatively, provide only very limited data.
sions of a study. Ongoing effort is needed to remind project staff of
the importance of appearance and professionalism
Project-Based Methods and to take care to represent the project to others
appropriately. Only in this way can the highest
Project-based methods include daily operational
quality of data be collected.
procedures that help to minimize errors in all
Gaining entry to the field of a qualitative study
processes related to data collection, data storage,
can be difficult and challenging for other reasons.
and data management. Chapter 33 addresses these
For example, an organization or group may have
more technical and procedural aspects of actions
had a negative experience with a previous project.
taken to ensure optimal data quality and data secu-
For example, in the study of community integra-
rity in detail. However, two final aspects of quali-
tion after spinal cord injury described earlier,
tative data gathering should be reviewed, namely
research staff realized that potential study recruits
entry and exit from the research setting.
had several misperceptions about what participa-
tion in the study really meant. They learned, for
Entering the Field Study Site
example, that for some recruits spinal cord
Textbooks on qualitative research do not often research meant being “stuck with needles” or
address the basic issues of entry and exit from the “strapped to a machine.” Interest and willing-
field, especially the relationships that must be cre- ness to participate increased significantly, once it
ated and then ended with the participants/inform- was understood that the study involved being
ants and stakeholders who support access to the interviewed and no invasive procedures would be
research setting. Neglecting these processes can used.
Another reason that gaining entry into the field difficult, they may drastically restrict the informa-
can be difficult is because the gatekeepers to the tion they share. Superficial data is the result.
research setting are important and busy people. On the other hand, the interviewer may cause
They may view research participation as less poor quality data. An interviewer who is perceived
important than many of their other activities. In to be overly comfortable or overly intrusive and
such a situation, gaining access will require con- who creates an unwelcome sense of familiarity
siderable persistance, tact, and possibly assistance with participants, can have negative consequences.
from someone who is respected in the setting. At These interviewer problems can best be avoided by
this preliminary level, gaining access requires the thorough staff training, clearly written recruitment
investigator to negotiate with those who control materials, and prepared interview scripts and
access to the research setting, or access to the doc- responses to frequently asked qestions to use ver-
uments or objects that wish to be studied. The batim when answering the most common queries
investigator must also balance the needs of the of prospective research recruits. With complete
study against the concerns of the host group. descriptions of the study purpose, information
Where the investigator expects cooperation, about the kinds of the questions that will be asked,
gaining entry may be largely a matter of establish- and honest evaluations about the time and effort
ing trust and rapport. A mainstay of the qualitative required of participants provided before data col-
investigator is saying something like: “I’m here lection, most problems of data collection can be
because I would like to understand X better and avoided.
because we believe your opinions and experiences Collecting qualitative data can be challeng-
will help us to learn more about Y and help to ing for reasons beyond the practical issues
improve Z”. When access is expected to be more reviewed above. A qualitative interviewer or qual-
difficult, often the best tack is through the known itative participant-observer is always at some risk
sponsor approach. In this approach, the qualitative of being pulled into the lives of study participants
researcher is vouched for by an already familiar (Dillaway, 2002). This pull can come in the form
and respected person. If truly trusted and respec- of requests for assistance and advice, and emo-
ted, the qualitative researcher can rely on his or her tional support, for example. For occupational ther-
introduction to facilitate entry and to facilitate if or apists who are also researchers, the dual role of
when unanticipated bumps arise in the data collec- clinician and researcher can be especially chal-
tion process over time. lenging. For example, in the study of community
A useful technique in participant observation integration after spinal cord injury mentioned ear-
studies in particular is to begin observations/field- lier, office staff and interviewers were frequently
work at the same time that new members begin the asked for advice on the following: where to obtain
activity of study interest or join the group. By better home care services, where to socialize in
using timing to one’s advantage, investigators can order to meet someone of the opposite sex, how to
minimize the disparity between their level of find a better medical specialist, where to buy an
knowledge and that of the study participants. adapted motor vehicle, how to obtain funding
A closely related challenge is obtaining high- to return to school, how to find research projects
quality data once in the field (Dillaway, 2002). that paid more, and how to find a better job. Of
Although achieving excellent data quality is not course, these requests for advice were predicated
directy linked to gaining access to the research on the reasonable assumption that the staff pos-
site, it too is a topic not often addressed in qualita- sessed a higher degree of expertise in some of
tive methods textbooks. If measures are not taken these areas than participants did themselves,
to ensure smooth operations, data quality will suf- although this is not always true. When researchers
fer. As mentioned earlier, challenges to data occur are also clinicians, they may find themselves in
when participation in the research is difficult for conflict about which hat they are wearing and
participants, for whatever reason. This may be due what ethical and practical obligations they have to
to research questions that participants consider too both the project and the research participants.
private, provocative, or controversial. In situations Thus, qualitative researchers should be aware that
like this, participants may be too embarrassed to while they must develop sufficient rapport and
reveal their true attitudes and describe their expe- closeness to research participants to elicit useful
riences in full detail, even to a well-trained and research data, there will be times when the rela-
empathetic interviewer. When participants feel the tionship becomes too close with the potential for
questions they are answering are too personal or negative consequences to participant and resear-
intimate or find the topics studied too emotionally cher alike.
To be clear, it is commonplace and expected sive symptoms or suicidal thoughts. Although

that qualitative researchers will give back to their these instances are infrequent, the qualitative
study participants in at least a modest way. The researcher must be knowlegeable and prepared to
provision of helpful answers, advice, and assis- exercise skill and resourcefulness. On such occa-
tance are some immediate ways of doing this and sions, qualitative researchers must be appropri-
providing the reciprocity that researchers feel they ately emphathetic, but also be prepared to offer
owe their research participants given their generous referrals to professionals for counselling and to
contributions of data. Reciprocating research par- relevent agencies for more tangible resources.
ticipation (e.g., simple assistance, thank you cards, Confronting these types of challenging interper-
educational handouts) can become substantial and sonal circumstances repeatedly and over a sus-
frequent, placing more serious demands on the tained time frame can lead to fatigue, stress, and
qualitative researcher. When this occurs, the psy- potential burnout. Thus, the principal investigator
chological resources of the interviewer/researcher must address such circumstances at their earliest
can be stretched and even ethically compromised. evidence because they not only adversely affect
In some instances, it can drain the financial, social, the health and well-being of research staff, but also
and tangible resources of the entire project. compromise optimal data quality.
In such situations, team meetings and discus-
sions with the principal investigator of the study
are imperative to ensure guidelines are set and fol-
Leaving (Exiting) the Field Study Site
lowed to clarify what does and does not constitute Lofland and Lofland (1995) report that leaving the
appropriate action. These guidelines will involve field is one of the most difficult aspects of research
issues of practicality but will also bear directly on for the qualitative researcher. Leaving the field
a variety of ethical issues and responsibilities, not reminds the qualitative researcher of the unequal
only to research participants, but also to project power relationships that exist between the
staff, the university, and even the agency that researcher and the researched. Not uncommonly,
funded the research (Stern, 2005, November). leaving creates feelings of guilt since the resear-
Regular debriefing sessions with project peers, cher walks away with valued data and the partici-
particularly after difficult data collection episodes, pants appear to receive little or nothing at all. Of
are essential to provide emotional support to data course, this is not completely true. The give backs
gatherers who can and do face surprising and mentioned above including small measures of
stressful events during qualitative interviews. assistance and advice can have a real and lasting
When the interviews take place in the homes impact on some study participants. Even sending
and communities of the research participants, the out simple thank you cards might be appreciated.
range of unexpected events can be rather remark- Beyond these tokens of appreciation, it is impor-
able. These surprises can inlcude such diverse tant to realize that many informants welcome the
things as insect infestations, angry family mem- opportunity to not be seen as helpless, but rather
bers, unhealthy pets, and instances of negligent to be seen as valued and useful to others through
care or abuse. Thus, thorough staff supervision and their role as research participants. In the end, how-
leadership by the principal investigator of the ever, all researchers must say thank you and good-
study is essential to provide guidance in dealing bye.
with such situations. This guidance is necessary to One important reward to research participants
ensure that the actions of the staff are: is a methodologically rigorous study that yields
important new findings that are disseminated to
• As scientifically sound as possible,
audiences where changes and improvements can
• Ethically appropriate,
be made. In this way, although the benefits of the
• Adequate to protect the research staff and the
research do not accrue directly to the individuals
participants in the encounter, and
who originally contributed the data, and they are
• In compliance with legal requirements (e.g.,
hardly immediate, there are meaningful benefits to
reporting abuse and neglect).
others who share the same health condition, social
Since qualitative research can involve deep circumstances, or experience. For all these rea-
investigations of social experiences and meanings sons, qualitative reserch increasingly involves uti-
on personal and emotionally difficult topics there lization of participatory approaches (see Chapters
will be occasions when qualitative researchers 38 to 40) that seek in the course of the study to
encounter a study participant who becomes upset empower participants to effect desired changes in
or tearful, or more rarely, expresses severe depres- their own circumstances.
Conclusion developmental disabilities. American Journal of

Occupational Therapy, 58 (4), 455–465.
Stern, E. (2005, November). American Occupational
As noted at the outset, qualitative research is a nat- Therapy Foundation and American Occupational
uralistic, emergent and evolving, and interpretive Therapy Association’s “Promoting Integrity in the
Next Generation of Researchers: A Curriculum for
endeavor (Rossman & Rallis, 1998). Qualitative Responsible Conduct of Research in Occupational
data collection reflects these characteristics. In part, Therapy (Web-based curriculum).” Retrieved
it means that the data collection process is inti- November 28, 2005 from http://www.aotf.org/html/
mately interwoven with the analysis and interpreta- rcrgraduateeducation.shtml
tion processes discussed in Chapters 21 and 22. RESOURCES
Morover, qualitative researchers view social phe-
nomena holistically, and are sensitive to their own Web Sites Relevant to Gathering
influence on the study and its findings (Rossman & Qualitative Research
http://www.socialresearchmethods.net/kb/
Rallis, 1998). These two elements always guide http://www.vanguard.edu/faculty/dratcliff/index.cfm?
how the data collection strategies discussed in this doc_id⫽4258
chapter are undertaken in a given study. Recommended Readings for
Gathering Qualitative Research
Creswell, J. C. (1998). Qualitative inquiry and research
REFERENCES design: Choosing among five traditions. Thousand
Dillaway, H. E. (2002). Menopause in social context: Oaks, CA: SAGE Publications.
Women’s experiences of reproductive aging in the Denzin, N. K., & Lincoln, Y. S. (2000). Handbook of qual-
United States. Unpublished doctoral dissertation, itative research (2nd ed.). Thousand Oaks, CA: SAGE
Michigan State University, East Lansing, MI. Publications.
Dillaway, H., Stengle, W., Miree, C., St. Onge, K., Berry- Gubrium, J. F., & Holstein, J. A. (Eds.)(2002). Handbook
Bobovski, L., White, J., King, S., & Brown, D. (2005). of interview research: Context and method. Thousand
Community as paradox: Understanding both barriers Oaks, CA: SAGE Publications.
and incentives to African American men’s participation Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry.
in prostate cancer prevention trials. Manuscript sub- Beverly Hills, CA: SAGE Publications.
mitted for publication. Lofland, J., & Lofland, L. (1995). Analyzing social set-
Lofland, J., & Lofland, L. (1995). Analyzing social settings: A guide to qualitative analysis and observation.
tings: A guide to qualitative analysis and observation. Belmont, CA: Wadsworth.
Belmont, CA: Wadsworth. Marshall, C., & Rossman, G. B. (Eds.) (1999). Designing
Lysack, C., & Kaufert, J. (1999). Disabled consumers’ qualitative research. Thousand Oaks, CA: SAGE
perspectives on provision of community rehabilitation Publications.
services. Canadian Journal of Rehabilitation, 12 (3), Shaffir, W. B., & Stebbins, R. A. (Eds.) (1991). Experienc-
157–166. ing fieldwork: An inside view of qualitative research.
Lysack, C., & Luborsky, M. (2004). Community living Newbury Park, CA: SAGE Publications.
after spinal cord injury: Models and outcomes. (R01 Spradley, J. (1979). The ethnographic interview. New York:
#1HD43378, funded by NIH / NICHD / NCMRR). Holt, Rinehart and Winston.
Rossman, G. B., & Rallis, S. F. (1998). Learning in the Qualitative Methods In Health Research: Opportunities
field: An introduction to qualitative research. Thousand and Considerations In Application and Review. Office
Oaks, CA: SAGE. of Behavioral and Social Sciences Research National
Siporin, S., & Lysack, C. (2004). Quality of life and sup- Institutes of Health. http://obssr.od.nih.gov/
ported employment: A case study of three women with Publications/Qualitative.PDF
21Keihlofner(F)-21 5/5/06 3:56 PM Page 358
C H A P T E R 2 1
Contemporary Tools for Managing

and Analyzing Qualitative Data
Nadine Peacock • Amy Paul-Ward
In any research endeavor, “raw” data must be The aim of this chapter is to identify key ele-
transformed into coherent, believable, and mean- ments shared by a range of approaches to qualita-
ingful findings. In qualitative studies, this transfor- tive data analyses, and to describe some of the
mation requires that the investigator make a varied software programs and other types of tech-
number of strategic decisions about data manage- nological support (such as transcription and edit-
ment and analysis, including whether to use spe- ing systems) that are useful for managing and
cialized technology. These decisions ordinarily analyzing qualitative data. This chapter will not
require consideration of the following factors: provide an exhaustive list or review of software
programs, as there are a number of publications
• The amount of time and funds allocated to the and Internet resources that do this quite nicely (see
analysis process, Resources).
• The volume and structure of the data,
• One’s comfort level with computer use and learn-
ing new software, and Common Features of Qualitative Research
• Epistemological stances guiding one’s research.
Though qualitative research approaches are many
Qualitative studies can range from those that and varied as discussed in Chapter 19, they tend to
are very exploratory to those designed to test or share certain commonalities. First, qualitative
confirm hypotheses or findings from prior studies tend to have a less formal, less structured
research. They can be performed by an individual purpose. A qualitative study is often built around
investigator using participant observation in what Mason (1996) calls an intellectual puzzle—
extended ethnographic fieldwork, or by a team that is, a general question about a social phenome-
doing rapid assessment with structured check lists non that the investigator hopes to understand
and interview guides. The data can take many better. This puzzle can then be expanded into a
forms including: number of more specific research questions, which
are usually best addressed by observing, listening,
• Loosely structured, lengthy narratives derived
and interpreting rather than measuring and testing.
from interviewers,
Thus, qualitative research tends to be more induc-
• Short-answer responses to open-ended survey
tive than deductive, and leans more toward theory
questions,
building than theory testing.
• Field notes taken by participant observers,
A corollary to the inductive nature of the
• Sound or video recordings, and
research is the blurring of boundaries among the
• Secondary data (e.g., documents, brochures,
tasks of data collection, data management, and
minutes of meetings) created by persons other
data analysis. In other words, qualitative research
than the investigators.
is an iterative process, wherein some data are col-
The data from a given study may range from a lected, interim analyses are performed, and
handful of documents to hundreds of them. research instruments are modified before further
Management and analysis of qualitative data can data are collected and analyzed. Because of the
be done by an individual researcher or by a team, fluid, nonlinear nature of this process, it becomes
and can take a low-technology approach such as particularly important to establish an audit trail, in
reading through data and making note of passages which study design, data collection and manage-
that illustrate useful themes, or a high-technology ment procedures, and analytic decisions are care-
approach that makes use of sophisticated and spe- fully documented.
cialized software. Finally, analysis and interpreta- Another key feature of qualitative data analysis
tion of qualitative data can reflect a broad spectrum is the creation of meta-data (i.e., data about data).
of epistemological stances, from the strongly pos- These are new text and/or graphic products created
itivist to constructivist and interpretivist. by the researcher to represent key themes, con-
Chapter 21 Contemporary Tools for Managing and Analyzing Qualitative Data 359
structs, and relationships that emerge from and are through their normal activities in the residential
applied to a body of qualitative data. As such, they facilities and the neighborhoods where they lived.
are tools for as well as products of data analysis. They took special note of:
The following are examples of meta-data in quali-
tative research: • Interactions and conversations between the
adults with developmental disabilities and staff
• Memos, in the facilities,
• Codes, • Interactions between these adults and persons in
• Data matrixes, the neighborhood,
• Concept maps, and • The daily routines of the participants, and
• Case summaries. • The perspectives of participants on their every-
day lives.
These are discussed in more detail in this
chapter. They also conducted interviews with these
adults with developmental disabilities and with
staff in the facilities. Observations were recorded
Management and Analysis— as handwritten field notes, which were later typed
up with additional descriptive detail. Interviews
General Principles were audiotaped and later transcribed into type-
written documents. Occasionally, events or inter-
One might think of coding and the production of views were video-recorded; scripts of some of
other meta-data as features of computer-assisted these videotapes were produced that included
qualitative data analysis. However, it is important not only transcribed records of what was said, but
to realize that qualitative analysis software simply also descriptions of behaviors that took place.
adds a layer of technology to a process that was a The researchers also routinely included the fol-
well-established component of qualitative inquiry lowing in their expanded notes:
for decades before the widespread use of comput-
ers for these purposes. • Parenthetical comments and explanations that
To illustrate this point, consider a study under- help fill in the picture of what was going on dur-
taken at the University of California at Los ing the observations, and
Angeles (UCLA) in the 1970s to investigate the • Reflexive notes on the investigators’ reactions in
experiences of adults with developmental dis- the field and their role in generating and inter-
abilities who were deinstitutionalized from state preting the data.
hospitals into commu-
nity residential facilities. The activities de-
This study was funded Qualitative research is an scribed so far (typing and
through the Maternal and expanding field notes
Child Health Division of
iterative process, wherein and transcribing inter-
the U.S. Public Health some data are collected, views) are data manage-
Service and undertaken ment tasks as well as
interim analyses are per- early steps in analysis,
by a team of anthropolo-
gists, sociologists, and formed, and research instru- whereby the investiga-
occupational therapists. ments are modified before tors began to extract
The investigators in this meaning from and make
study were interested in further data are collected sense of their qualitative
how successfully these and analyzed. data.
adults integrated into There are other im-
community life, what portant data management
types of occupational routines they established, tasks that involved identifying the body of products
and how the organization structures and practices that constituted the investigators’ data. In this
of the residential settings affected their lives instance the data included not only typewritten in-
(Bercovici, 1983; Goode, 1983; Kielhofner, 1979, formation from participant observation, interviews,
1981; Kielhofner & Takata, 1980). and videotaping, but also:
In this study, participant observation was
selected as the key method of data collection. The • Documents produced in the residential facility
occupational therapists and social scientists (e.g., mission statements, policies, notices,
accompanied and observed these adults going advertising brochures and reports),
• Medical records of some of the developmentally • Analytic memos, which were reflective notes
disabled adults, and through which the investigators began to organize
• Photographic and video records of events. their thoughts about the data. These were the
raw materials for later interpretations and find-
Decisions had to be made about where and how
ings.
these varied types of data were stored, who had
access to them, and what role they played in In this study, because several investigators
addressing the intellectual puzzle and research worked together they routinely presented and
questions. discussed the conceptual
Once the investigators labels and analytic mem-
had begun to amass Qualitative analysis software os. These discussions led
typed field notes, inter- simply adds a layer of tech- to the identification and
view transcriptions, and elaboration of important
other documents, they nology to a process that emerging themes which
were ready to delve was a well-established com- were captured in addi-
further into the analytic tional analytic notes.
process. Recall that desk- ponent of qualitative inquiry They also guided the
top computers were not for decades before the focus of future data gath-
widely available in the ering sessions, in that
1970s when this study widespread use of comput- special attention could
took place. Although ers for these purposes. be paid to data that were
mainframe computers relevant to these emerg-
were commonly used in ing concepts and themes.
the social and behavioral sciences, their use for Clearly, the active reading and working discus-
qualitative data analysis at this time was rather sions described here served as early stages of data
rare. How then did the systematic management and analysis.
analysis of this data set proceed?
First, the investigators made multiple photo- Conceptual Labels and Index Codes
copies of each set of typed field notes and tran-
As these investigators proceeded with multiple
scripts. An original copy with complete provenance
readings through the data, descriptive conceptual
information (such as the date and location the
labels tended to become more systematic and
observation or interview took place, who gathered
consolidated, eventually culminating in a fairly
the data, who transcribed the tape, etc.) was kept in
comprehensive set of what can be called index
a reference file that was arranged chronologically
codes. Such codes function much like the index to
and by site. The photocopies of the data were
a book, in that they indicate places in the text
working copies that team members marked up in a
where one can retrieve information on a particular
process of active reading of the data—a key early
topic.
step in the analysis process.
Codes in this study included labels for different
As the investigators first read through the data,
types of social behaviors (e.g., approaching, avoid-
they routinely made free-form margin notes, com-
ing, staring, demeaning, reproaching, punishing);
menting on things that struck them as important
cognitive/affective states (embarrassment, frustra-
and noting consistencies or contradictions in dif-
tion, anger, fear); subjective experiences (e.g.,
ferent observations or interviews. This process was
helplessness, agency, being “frozen in time”); and
facilitated by leaving a wide margin when notes
issues over which there was often misunderstand-
were typed, so that there was ample room for such
ing between mainstream culture and those who
secondary note-taking. When more room was
were the focus of study (e.g., privacy, personal
needed, notes were stapled to the pages to which
space, scheduling time). The early creation of
they pertained. On examining these notes, one
codes such as “stigmatizing behavior” and “time
would notice that they fell into two broad cate-
use” reflects the fact that the study was conceptu-
gories (Figure 21.1):
ally grounded in the field of occupational therapy
• Conceptual labels, which were words or short (e.g., Kielhofner, 1977), and that it built on previ-
phrases that serve as a kind of tag for segments of ous research on deinstitutionalization (e.g.,
text, categorically describing information the Edgerton, 1971).
segments contained. These would eventually Index codes then can be derived from findings
evolve into a list of index codes used to label and of prior studies or from an initial conceptual
retrieve segments of text that relate to a common framework, and additional codes can be added as
theme (see the next heading). new concepts emerge from the data. As codes are
Fieldnotes Site: Picadilly July 12, 1978 Participant Observer: Gary Kielhofner
Daniel and Michelle had brought their guitars, so

naturally Doris was asked to play. Doris began to
play and sing with Michelle accompanying her both
playing guitar and singing. I was sitting on top of a
Note, there is a picnic table a few feet away and Tim sat beside me.
legitimizing affect From where I sat I could see not only Doris and
that those who "appear Doreen, Bill, Shereen, Jess and the UCLA folks,
normal" have in being but also the many kids in the background who had
associated with those noticed us and were curiously looking on. Finally
who have one brave soul (a young man who appeared 10
obvious physical years old) approached the corner of the area and
characteristics that lead unobu unobtrusively watched.
to stigma. He appeared almost mesmerized by the whole scene
which included: a toothless lady with funny red
Physical features that skin and very obese, wearing a red wig playing a
make stigma obvious guitar and singing her soul out; Buddy, a 50 year
old with Down Syndrome whose appearance
approximates a buddha figure, Doreen whose
appeances approximate Doris's but who appears
older in a black wig. Everyone else could possibly
pass for "normal" on appearances alone. Probably
the presence of these folks and the 5 UCLA people
make the whole scene a little lss less threatening.
After the first young man had ventured into "our
territory" more and more children began to
approach. Those who were younger held
Note; an interesting expressions of curiousity, fear and intensity. They
feature of the age appeared nervous and cautious but inexorably
differences is that drawn toward the scene. Two little girls looked at
children appear to Buddy (A downs-syndrome phenotype) whispered
become increasingly to each other, giggled with their hands over their
aware with age that it is mouths, stopped, stared for a while and so forth
okay (perhaps over and over. Bobby seemed unaware of their
normative) to make fun attention; he did not appear to look at them. Most
of people who are of the others focused on Doris and Doreen (who
different and their sense were doing the singing and guiart guitar playing).
of shame in responding Responses were obviously age graded. The younger
to stigma seems to ones in the group seemed not to know what to make
correspondingly, of it. The few who were older recognized that it was
disappear a spectactle and laughed obviously, with only mild
attempts to hide their response such as turning away
attracting "spectators" sometimes when they laughed harder.
Something about their laughter was interesting- it
was nervous laughter, not the open enjoyable
laughter of a harmless joke, but laughter which
betrayed some sort of ambivalence about the whole
sene scene.
Personal reflection: I
Most of the Picadilly people were unaffected by the
was uncomfortable
presence of the children. Doris and Doreen seemed
initially and then
to definitely like them as an audience. Jim is was
became aware of this. Let
laughing about their presence and I think he saw
me elaborate. The feeling
them as just an audience to the music. Tim,
I had was one of "guilt
however, got very nervous from the beginning.
by association" There
More than anyone else from Picadilly, so I think
was this automatic
feeling of self Tim knew what was going on. he turned to me with
protectiveness that arose a mortified look on his face and said he was
with the thought that the going to leave the area and join Nancy and Sheryl
children would "think and Lolita who were playing tennis. I asked him if
ther was something what was happening was making him nervous and
wrong with me too" he said yes. I knew that Tim knew and that he
When I realized this I clearly wished not to accrue guilt by association.
was ashamed of the Later when Lisa, and the others joined us from the
feeling tennis area, Tim came along. I noted later that he
was singing along with the music and appeared
Figure 21.1 Field notes from a much more comfortable and there was definitely
Differential awareness
less tension on in the air.
1970s UCLA study with index of others reaction to
labels and analytic notes. stigma
361
developed, a code directory or codebook is con- a quite cumbersome way of finding all the passages
structed, which includes operational definitions, as containing information on time-related behaviors.
well as inclusion and exclusion criteria—instruc- The investigators chose instead to physically
tions about circumstances under which the code group together all the text segments dealing with
should or should not be applied (MacQueen, the topic of interest. Any passage labeled with the
McLellan, Kay, & Milstein, 1998). Code defini- relevant code was physically cut from a photocopy
tions tend to evolve over time, meaning that with of the original page, tagged with information on its
increasing use, the definitions and the inclusion source location, and placed along with all other
and exclusion criteria become more precise. Codes such text segments in a folder as shown in Figure
that represent broad themes may be broken down 21.2. Typically multiple copies of a coded page
to component parts, and conversely codes that are needed to be made, since more than one code was
conceptually very closely related may be merged. often associated with a given text passage, and
Investigators should keep careful records of when therefore the passage would be filed in more than
codes are created, split or merged, and when their one folder.
definitions or application criteria are refined. (This By flipping through any folder, the investiga-
is one component of the “audit trail” discussed tors were able to read, grouped together, all the
previously). Investigators may need to go back and passages in the data where participants discuss
recode earlier coded portions of the data set in these issues. Passages gathered from different
light of evolving code criteria. parts of the data set (thus “de-contextualized”)
In the UCLA study, the investigators eventually were used to build an explanation about various
developed a fairly exhaustive indexing system, ways the participants behaved with reference to
consisting of a comprehensive list of concepts time, and how this behavior was related to their
grouped together into various themes. For exam- life experiences and perspectives, and to the organ-
ple, one emergent theme was “temporal adapta- ization of the settings where they lived.
tion” (Kielhofner, 1979). This pertains to how the Of course it was also important to preserve
deinstitutionalized adults experienced time, and information on where in the document a passage
the ways in which their unique experience of time was originally located, since appreciating its full
differed from the mainstream culture. Within this meaning often required viewing it in the context of
broad conceptual theme of temporality were the surrounding text. Thus, de-contextualizing and
nested subcategories, such as: re-contextualizing of data are important compo-
nents of the qualitative analysis process.
• Time-related behaviors (e.g., waiting, dealing
As data accumulated in the folders representing
with appointments, filling time),
various concepts, the reading and processing of
• Temporal perspectives (e.g., how people talked
that material sometimes resulted in the contents of
about the future), and
a folder being divided into two or more component
• Temporal misunderstandings (i.e., problematic
concepts. At other times, two folders were col-
interactions that emanated from non-normative
lapsed into one. In this way, the coding scheme
views of time held by the participants).
was continuously being refined as the data analy-
These and other index codes were written in the sis unfolded.
margin of clean copies of the typed field notes.
Recall that a main purpose of coding is to allow
Memos
the investigators to retrieve all relevant passages on
a given theme so that their content can be further Recall that the margin notes contain not only index
explored. The index code “Time-related Beha- labels, but also longer annotations we referred to
viors” is used to indicate where in the texts one can as memos. Rather than serving primarily an index-
find participants’ comments about making appoint- ing function, memos are methodological and ana-
ments, waiting, and so forth. To conduct a more in- lytic notes from which explanations and findings
depth analysis of this content area, the investigators are built. While these memos may start as notes in
needed to find all passages labeled with this the margin, they commonly become longer and
code, and look in detail at the selected passages. more complex as analysis proceeds.
This could prove difficult, given that any single Over time, the investigators in the UCLA study
page of data (field notes, transcribed interview, or started recording these memos in a larger format,
other document) can have numerous codes and using separate sheets of paper that were filed along
margin notes, and a single passage might be coded with text passages in topic folders. Eventually,
for more than one concept. Looking through hun- these memos were integrated together into work-
dreds of pages of codes and margin notes would be ing drafts of the analysis. These working papers
He appeared almost mesmerized by the

Physical whole scene which included: a toothless lady
features
with funny red skin and very obese, wearing
that
make
a red wig playing a guitar and singing her
stigma soul out; Buddy, a 50 year old with Down
obvious Syndrome whose appearance approximates a Physical features
buddha figure, Doreen whose appeances that make stigma
obvious
approximate Doris's but who appears older in
After the first young man had ventured into "our

"attracting territory" more and more children began to
spectators"
approach. Those who were younger held
expressions of curiousity, fear and intensity. They
appeared nervous and cautious but inexorably
Spectators
drawn toward the scene. Two little girls looked at
Buddy (A downs-syndrome phenotype)
whispered to each other, giggled with their hands
over their mouths, stopped, stared for a while and
so forth over and over
Figure 21.2 An example of “decontextualizing” data by grouping text passages related

to a common theme.
were shared in meetings of the research team data analysis. These are summarized in Table 21.1.
which led to discussion, critique, and new insights. These steps, of course, primarily refer to the more
These developing ideas were also recorded as “procedural” aspects of data analysis. How one
memos. Individual team members or small groups actually goes about the more “conceptual” aspects
had primary responsibility for generating explana- of data analysis depends on the particular form of
tions on a given theme, but the group process qualitative research one is doing. These conceptual
allowed for other team members to write memos aspects of data analysis are covered in detail in
and share their thoughts on the topic. In this way Chapter 22.
the analysis was organic and collaborative, unfold-
ing in constant interaction with the data.
Quality Assurance in
Interdependence of Codes and Memos
Qualitative Data Analysis
While indexing is fairly simple and mechanical,
and memo-writing is more complex and analytic, Methods for ensuring the quality or trustworthi-
they are quite interdependent and coordinated. As ness of qualitative data were introduced in Chapter
analysis proceeds, the coding scheme may be elab- 20. These methods include interviewer training,
orated, and perhaps made hierarchical, with some prolonged engagement in the field, reflexivity, tri-
index terms reflecting conceptual domains. These angulation, stakeholder checks, and audit trails.
domains serve as cover terms for more detailed Since qualitative inquiry is an iterative process
analytic categories, which commonly emerge after involving successive rounds of data collection and
the investigator spends time intellectually process- interim analyses, these methods are as relevant to
ing the data and writing memos. The coding data analysis as to data collection.
scheme may be used to develop a concept map in
which relationships among categories are defined Coding and Retrieving
and graphically illustrated.
There are additional quality assurance measures
that are specific to data analysis, particularly to the
Summary
process of coding. The creation of a code directory
This section provided an overview of some of the with clear operational definitions is an important
major steps or processes involved in qualitative first step in ensuring that the coding process is rig-
Table 21.1 Qualitative Data Analysis Procedures and Their Functions

Analysis Procedure Function
Active reading The process by which investigators immerse themselves in their
qualitative data, reading and re-reading while making marginal
notes about important themes and constructs
Index codes/conceptual labels Words or short phrases that serve as a kind of tag for segments of
text, categorically describing information the segments contain
Analytic memos Reflective notes through which the investigators began to organize
their thoughts about the data
Code directory A list of all codes with clear operational definitions to ensure that the
coding process is rigorous and reliable
orous and reliable. Particularly when multiple in- a means of ensuring reliability of the coding pro-
vestigators are coding, analyzing, and/or interpret- cess. This begins the same way—with the redun-
ing the data, it is important that these activities dant coding of texts by two or more analysts.
take place in a systematic and consistent way. Differences are identified and discussed. Eventu-
This is not to say that all investigators work- ally, the parties reconcile their differences and
ing on a project should arrive at the same interpre- agree on one set of codes for the document. Many
tations and explanations of qualitative data. qualitative researchers use only this approach, and
Indeed, the possibility that investigators will bring reject any use of statistical measures, seeing those
different perspectives to analysis (in other words, as a needless and even inappropriate since bor-
investigator triangulation) is an important reason rowed from positivist, quantitative paradigms.
for conducting collaborative research. However, The authors maintain that both approaches are
the mechanics of analytic tasks such as coding and useful and important. When code-by-code agree-
retrieval of text segments must be systematic and ment statistics are collected and scrutinized, the
consistent across researchers and documents. investigators can detect patterns in which types of
Otherwise, the results are simply a series of inde- codes are more or less easily agreed on. For exam-
pendent, individually conducted analyses rather ple, they may find that codes that represent affec-
than a coordinated research effort. tive states are much more difficult to apply
consistently than those that capture overt behav-
Inter-Rater Reliability iors. Discussions in which coders explain their
rationale for applying codes in a particular way
There are a number of steps investigators can take
can lead not only to reconciliation, but also to
to assess and improve inter-rater reliability in the
more careful crafting of operational definitions
application of codes. This generally involves some
that will make future disagreements less likely.
form of redundant coding, where two researchers
This can be documented by assessing agreement
code the same text passages and compare their
statistics before reconciliation, and then again
results. One can then quantitatively assess the level
using a new subset of the data after code criteria
of agreement between the coders by calculating
have been refined.
some kind of agreement statistic. The simplest
approach is to calculate the proportion of instances
in which the two coders agree on the coding of
given text passages. This method, however, can
overestimate agreement for codes that are com-
Computer-Assisted Qualitative
monly applied, because some degree of agreement Data Analysis (CAQDAS)
by pure chance is likely. An alternative approach is
to calculate a statistic such as kappa, an agreement This chapter has so far presented a very basic view
statistic discussed in Chapter 12 that takes chance of the qualitative data analysis process. Though the
agreement into account (Carey, Morgan, & low-technology example of qualitative data analy-
Oxtoby, 1996). A separate kappa statistic is calcu- sis has not been discussed in its full complexity,
lated for each code, and a range of kappa values this is nonetheless a good place to pause, review
can be reported for the entire set of codes. what the ULCA investigators did, and begin to talk
In contrast to this quantitative approach, many about high-technology or computer-assisted ver-
researchers prefer discussion and reconciliation as sions of these activities.
Table 21.2 Hard/Analog-to-Digital Data Conversion for Computer-Assisted Analysis

Hard/Analog Digital
Text Handwritten or typed field notes, Word-processing files, spreadsheet data,
observation records, pencil- text files, pdf files, etc.
and-paper surveys, brochures/
small media
Sound Audiotape recordings Word-processing files; digital recordings
Images Photographs, drawings, etc. Scanned images, pdf files, Web pages, etc.
Image and sound Videotape, film Digital film, streaming video/sound files,
multimedia Web pages, etc.
Recall that the first thing that was done to facil- need to be converted from a word processing for-
itate systematic analysis was the conversion of mat to a “text only” file. However, the latest ver-
data from one format (handwritten field notes) to sions of most QDA software programs can directly
another (multiple copies of typed and expanded utilize word processing files, most typically rich
notes). Likewise, computer-assisted qualitative text (.RTF) files, which maintain their formatting
data analysis almost always requires a preparatory features across multiple software programs and
step of data conversion. In this case, the conversion platforms. Once files are imported into the QDA
is from data stored in a “hard” or analog form to a program, a variety of data management and analy-
digital form. For example, verbatim transcripts of sis tasks can be performed. Some key tasks are
audiotaped interviews are generally prepared by described below, including data storage and man-
typing them in a word processing program. Other agement, tag-and-retrieval functions, memo writ-
examples of such conversions are listed in Table ing, theory building, and quality assurance. Ways
21.2 (read further for discussion of technologies that software programs support team-based analy-
that permit the automation or elimination of consis and mixed-method approaches are also
version steps). described.
If the UCLA study data were being analyzed in
2005 rather than the 1970s, how might the work Data Storage and Management
proceed in a way that takes advantage of advances
Recall that in the low-technology type of analysis
in computer technology? First of all, the
described in the preceding text, a clean copy of all
observers’ handwritten notes would likely be typed
typed documents is kept in a master file organized
in a word processing program such as Microsoft
on some selected criteria. For example, the project
Word or WordPerfect, and then imported into a
data might include transcripts of interviews, filed
qualitative data analysis (QDA) software program.
sequentially by their unique alpha-numeric ID
The first commercially available QDA program,
numbers, which carry information about the par-
called The Ethnograph®, was introduced in the
ticular data gathering event. For example, the ID
early 1980s. Since then, the selection has greatly
number D022677_3M indicates that this interview
expanded, with products such as N6® (originally
was done in a facility labeled “Facility D” on the
marketed under the name NUD.IST®), NVivo®,
26th of February, 1977. It was the third interview
and ATLAS.ti® being among the most popular.
done at that site on that day, and the respondent
Newer on the scene are AnSWR® and EZ-Text®,
was a male. No other interview could have exactly
programs that were developed by the Centers for
that constellation of attributes; thus the ID number
Disease Control and Prevention (CDC), and that
is unique. Transcripts can be filed by a primary cri-
have the distinct advantage of being distributed
terion (i.e., the setting), and within that grouping
free to the public.1
they can be ordered by a secondary criterion (i.e.,
Depending on the program used, the data may
the date of the interview).
1When illustrating computer-assisted qualitative data analy-
sis in this chapter, examples from the ATLAS.ti program are
Labeling Data in QDA Systems
used in most cases. This should not be considered an How does computer-assisted data storage and
endorsement of a particular product over others, but is used
simply because this is the program most familiar to the management differ from the process described
authors. Where we know of significant differences between above? One major advantage in terms of computer
programs, these are pointed out. applications is that one can easily use the software
to categorize the data files in multiple ways. For In N6, the minimal “text unit” must be defined
example, the data may be entered and retrieved by in advance as a line, a sentence or a paragraph,
any number of criteria such as sex, age, ethnic though the boundaries of a segment can be modi-
group, or disability status. fied after creation (this reflects the influence on
program developers of classic Grounded Theory
Text-Based Searches methodology, with its signature line-by-line
coding). In all of these programs, text segments
All the major QDA programs allow for text-based
can overlap with or be embedded within other
searches, in which words or strings of text can be
segments.
searched for in the original data. Most also have
QDA programs also include one or more search
the capability to conduct word counts—calcula-
tools, which constitute the “retrieving” part of tag-
tion of the frequency with which different words
and-retrieval. These tools allow the investigator to
appear in the text. Some programs have more spe-
retrieve all text segments labeled with a particular
cialized features for conducting various types of
code, so that passages on a given topic or theme
quantitative content analysis.
can be grouped and viewed together, just as in the
topic folders in the “low-tech” example. The QDA
Codebook Development programs can go much further though, allowing
All major QDA programs support the creation of a for complex search commands that combine two
list of codes to be applied to textual data. The pro- or more codes, or other search criteria. This can be
grams vary, however, in the extent to which full particularly useful with large data sets, when
codebook development is supported. In ATLAS.ti, searching on a code that is heavily used may result
the investigator creates codes that are displayed in in a very large volume of retrieved data, not all of
a drop-down list. Code definitions, instructions for which may be relevant to the question at hand. In
application, examples, etc. must either be stored in such cases, the search can be narrowed by retriev-
a generic “comment” window attached to each ing only a subset of the coded segments, either by
code, or in a separate document such as a spread- combining one code with another, or by imposing
sheet or word processing file. N6 is similar, with other restrictions on what is retrieved.
each code having a “description” window in which
code definitions, application criteria, etc. can be Boolean Connectors. A common way to perform
recorded. CDC’s AnSWR has more fully devel- these more complex searches is by using Boolean
oped codebook features, with separate fields for connectors (e.g., “and,” “or,” “not”) to combine
the code name, a brief definition, a full definition, codes. For example, the investigators might be
inclusion criteria, exclusion criteria, and examples. interested in examining accounts of disability-
related stigma in the community, but only when
Tag-and-Retrieval the stigma involves some kind of discriminatory
behavior. The search string “stigma AND discrim-
Recall that in the “low-tech” coding example, ination” would retrieve the desired material,
investigators labeled passages of text with index while ignoring accounts of stigma not associated
codes, after which like-labeled passages were with discrimination, as well as discrimination
extracted and placed together in a folder. QDA linked to race or ethnicity but not disability.
software uses the same logic, but accomplishes the Chapter 27 has a more extended discussion of
tasks with much greater efficiency by using hyper- Boolean logic and how it can be used to search for
text connections, whereby noncontiguous text or information.
images are electronically linked to one another.
Codes stored in a code list can be linked electron- Semantic Connectors. Complex searches can
ically to segments of text. The text passage is also be conducted using semantic connectors.
selected and the relevant code or codes are Recall from the discussion above that codes
attached through keystroke commands, drop-down reflecting broad conceptual domains can serve as
menus, buttons, or a “drag-and-drop” function. cover terms for more detailed analytic categories.
This is the “tagging” part of tag-and-retrieval. In In QDA software programs, semantic relationships
ATLAS.ti and AnSWR, text passages of any size, can be created that indicate hierarchical or other
from a single character to the entire document, can types of relationships between codes. For example,
be highlighted and designated as a “text segment” “waiting,” “filling time,” and “being stuck in time”
(in AnSWR) or a “quotation” (in ATLAS.ti) (see can be defined as types of “temporal experience.”
the feature box on QDA Software Coding). “Temporal experience” therefore serves as a parent
term for the other three, and the three types of grouping of related codes without defining a
experiences are related to one another as sibling semantic relationship. In ATLAS.ti, codes can be
terms. This semantic relationship is illustrated in grouped together in units called “families,” which
Figure 21.3. can also be used as search terms. Families of other
Since these semantic relationships are specified objects, such as documents and memos, can also
in the software, one can issue a simple search com- be used in searches.
mand that says in effect “retrieve any passage
coded with ‘Temporal experience’ or any of its Proximity Connectors. A third kind of complex
daughter terms.” Some programs also allow the search uses proximity connectors, which analyze
QDA Software Coding Example—Enabling Self-Determination
Investigators at the University of Illinois at employment. Data are being analyzed with the
Chicago are carrying out a project to develop and assistance of ATLAS.ti, one of the QDA software
test a new model of independent living services for programs described in this chapter. In the screen
persons living with HIV/AIDS (Kielhofner & shot below, one can see how the QDA program
Braveman, 2001). The project is based on a partic- permits the linking of index codes and memos to
ipatory research model that recognizes the impor- text segments in a manner that parallels the “low-
tance of input and collaboration from clients and tech” method described earlier.
facility staff for achieving program goals. In an Investigators on the ESD project are also using
early project activity, investigators conducted a ATLAS.ti’s theory-building functions to develop
series of focus group discussions with clients and conceptual models that are grounded in the qualita-
staff of service organizations, in which participants tive data. The next page provides an example of a
identified barriers to community living and Network View showing code–code relationships.
(continued)
QDA Software Coding Example—Enabling Self-Determination (continued)
instance of discrimination may be reported. This

Temporal
co-occurrence would not be picked up using
Experience
semantic connectors because the “stigma” and
“discrimination” codes are not attached to a single
text segment. Using proximity connectors, how-
ever, one could search for all “stigma” segments
that are adjacent to (or overlapping or embedded
Filling Being stuck within) a segment coded with “discrimination.”
Waiting
time in time
Memo-Writing
= Parent relationship All major QDA programs have a mechanism for
writing notes and memos, through which the inves-
= Sibling relationship tigators can record impressions about the data,
methodological insights, developing interpreta-
Figure 21.3 An illustration of semantic
connectors. tions and theory, and so forth. ATLAS.ti has two
mechanisms for such annotations. Memos are one
of the four major objects in the program (along
spatial relationships between coded segments. with documents, codes, and quotations). These
Continuing with the preceding example, imagine objects have a special status, and can be ordered,
that the data contain a passage describing a stigma- sorted, and displayed in special ways. “Comments”
tizing event in the community. This account itself are a more general category of annotation, and
may have had no overt reference to discrimination, serve as a kind of “sticky note” that can be attached
but in the text segment immediately following it, an to any object, including the entire project, a docu-
ment, a code, or a “family.” In N6, memos can be another called HyperResearch® provide the most
attached to both documents and codes. Comments sophisticated handling of audio and video files.
on particular passages of text are recorded as There are also specialized programs specifically
“annotations,” which are imbedded in the text designed for audio and video editing, some of
when it is viewed within the program. which include tag-and-retrieval functions (e.g.,
Transana®).
Theory Building
Support for Mixed-Method Analyses
The most popular QDA programs go beyond sim-
ple tag-and-retrieval and memoing functions, and For those interested in incorporating quantitative
support theory-building. Theory-building func- methods into their qualitative studies, tools are
tions allow the construction of explanations that available in some QDA software programs to sup-
are grounded in the qualitative data, and include port a variety of approaches including calculating
mechanisms for defining relationships among data and graphing code frequencies, creating code-by-
elements, including original text, codes, memos, document tables and other data matrixes, exporting
and documents. ATLAS.ti has a tool called a data to statistical analysis or spreadsheet pro-
Network Editor, which is a graphic space in which grams, and quantitative content analysis. Beyond
data and meta-data elements can be imported and basic functions, however, specialized programs
linked to one another through semantic relation- may be needed. For example, programs such as
ships, as described earlier. These “network” views Wordstat®, Alceste®, and Concordance® are spe-
can evolve into formal analytic products such as cialized content analysis programs, used for tasks
conceptual models or concept maps. In ATLAS.ti like concordance and cluster analyses of word
these connections can take the form of hypertext distributions within texts. Programs such as
links between text segments in a project, even if AnthroPak® are used for analysis of data gathered
they are not in the same document. For example, a through systematic elicitation techniques used to
staff member in a residential facility may comment explore cultural domains. Examples are free lists
that it is all right to let oneself into a resident’s (in which a respondent is asked to list items in a
room unannounced since “they don’t really care domain, such as types of discrimination), pile sorts
about privacy.” Other passages in the dataset could (in which respondents sort a set of printed cards or
be linked to this statement, with a semantic rela- items into piles from which taxonomies are built),
tionship such as “supports” or “contradicts.” and paired comparisons (in which items in a
domain are ranked along some dimension by hav-
ing respondents compare pairs of items in the
Teamwork
domain). Complex quantitative procedures that
All of the major QDA software programs provide often build on qualitative research are also sup-
some level of support for team-based analysis. ported by specialized software. For example, deci-
Login names can be assigned to multiple users sion analysis is a kind of modeling exercise that
working on the same project so that their contribu- attempts to predict behavioral choices under spec-
tions can be tracked. Users can be assigned differ- ified circumstances. Preliminary models are gener-
ent levels of access, so that only those in ally inductively constructed based on exploratory
supervisory positions have the ability to alter or qualitative data, then refined based on new data
delete data. There are “bundling” procedures to that focuses more specifically on decision criteria
facilitate moving files between users, computers, for the behavior of interest. Models are iteratively
and networks. Partial analyses performed by dif- tested and revised until an acceptably predictive
ferent researchers can be combined through merge model is reached. Another example is the
procedures. The CDC programs (AnSWR and EZ- Q-method, which is discussed in Chapter 23.
Text) include a tool for calculating Cohen’s kappa,
the inter-rater reliability statistic discussed in the
preceding text. Other Tools and Software
Multimedia Capabilities that Facilitate QDA
Several QDA programs allow for importing and With new technological advances, some of the ana-
coding not only text files, but also graphic files, log-to-digital conversion steps described earlier
HTML Web pages, and digital audio and video can be automated or even eliminated. Optical scan-
files. Of the general QDA programs, ATLAS.ti and ning can quickly convert written or typed material
to machine-readable text or image files. Voice- As Weitzman (2000) notes, there are software
recognition software programs can be used to con- tools to support a wide variety of research and
vert taped interviews to text files. Audio and video analysis methods and not just those that are posi-
data can be directly recorded in digital format with tivist or quasi-positivist in nature. Moreover, there
digital recorders and cam- is no single best program
eras. Interview responses to meet all needs and no
may be typed directly into To make the best use of program will do the data
a computer by the inves- QDA software programs, analysis for an investiga-
tigator, or even by the tor. To make the best
respondent, as is done qualitative researchers must use of QDA software
with Audio CASI technol- first and foremost under- programs, qualitative re-
ogy, an interview method searchers must first and
in which respondents use stand the fundamentals of foremost understand the
a computer keyboard to qualitative inquiry. fundamentals of qualita-
enter responses to prere- tive inquiry. They must
corded questions. Audio take time to think about
CASI is most often used with closed-ended ques- and articulate their intellectual puzzles, research
tionnaires (often in order to deal with literacy as questions, and the type of inquiry needed to answer
well as privacy issues), but can be used with open- those questions. Finally, the researcher should take
ended questions as well. advantage of the considerable volume of resources
available to help select an appropriate program.
Some of those resources are listed at the end of this
Conclusion chapter.
Although increasingly common, the use of com- REFERENCES

puters has not been universally embraced by Bercovici, S. (1983). Barriers to normalization. Baltimore:
qualitative researchers. Some simply have well- University Park Press.
established work habits using manual techniques, Carey, J. W., Morgan, M., & Oxtoby, M. J. (1996).
and see no need to change them. There may be Intercoder agreement in analysis of responses to open-
ended interview questions: Examples from tuberculosis
concern among some that the programs will be too research. Cultural Anthropology Methods 8(3), 1–5.
difficult to learn or not worth the effort for the type Edgerton, R. (1971). The cloak of competence: Stigma in
of analysis planned. Some, particularly those the lives of the mentally retarded. Berkeley: University
working within participatory research (as dis- of California Press.
cussed in Chapter 38), may fear that the computers Goode, D. (1983). Who is Bobby? In G. Kielhofner (Ed.),
Health through occupation: Theory & practice in occu-
distance the researchers too much from the voices pational therapy. Philadelphia: F. A. Davis.
and lived experiences of research participants and Kielhofner, G. (1977). Temporal adaptation: A conceptual
make it more difficult for the latter to participate in framework for occupational therapy. American Journal
the data analysis. Still others have a basic philo- of Occupational Therapy, 31, 235–242.
Kielhofner, G. (1979). The temporal dimension in the lives
sophical opposition to using technological tools of retarded adults: A problem of interaction and inter-
associated with positivist paradigms that they see vention. American Journal of Occupational Therapy,
as antithetical to the spirit of qualitative research. 33, 161–168.
Some of these concerns are not without merit. Kielhofner, G. (1981). An ethnographic study of deinstitu-
For very small projects it may arguably not be tionalized adults: Their community settings and daily
life experiences. Occupational Therapy Journal of
worth a researcher’s while to learn a complex Research, 1, 125–142.
QDA program, when the same results could be Kielhofner, G., & Braveman, B. (2001). Enabling Self
achieved with low-technology approaches. Some Determination for people living with AIDS.
researchers are tempted to use software inappro- Department of Occupational Therapy, University of
Illinois at Chicago. Grant proposal submitted to and
priately, for example, performing individual-level funded by the National Institute of Disability and
statistical analyses on focus group data. Some Rehabilitation Research, U.S. Department of Education
investigators become so wrapped up in their cod- (H133G020217-3).
ing schemes and diagrams that they fail to stay Kielhofner, G., & Takata, N. (1980). A study of mentally
close to the texts that are the true heart of their retarded persons: Applied research in occupational
therapy. American Journal of Occupational Therapy,
research findings. Such examples notwithstanding, 34, 252–258.
computer-assisted analysis is appropriate and use- MacQueen, K. M., McLellan, E., Kay, K., & Milstein,
ful for many types of qualitative research. B. (1998). Codebook development for team-based
qualitative analysis. Cultural Anthropology Methods, Miles, M. B., & Huberman, A. M. (1994). Qualitative data
10(2), 31–36. analysis (2nd ed.). Thousand Oaks, CA: SAGE
Mason, J. (1996). Qualitative researching. Thousand Oaks, Publications.
CA: SAGE Publications. Weitzman, E. A., & Miles, M. B. (1995). Computer pro-
Weitzman, E. A. (2000). Software and qualitative research. grams for qualitative data analysis: A software source-
In N. K. Denzin & Y.S. Lincoln (Eds.), Handbook of book. Thousand Oaks, CA: SAGE Publications.
qualitative research (2nd ed.) (pp. 803–820). Thousand Web Sites—General
Oaks: SAGE Publications. CAQDAS Networking Project—a project housed at the
University of Surrey in the U.K., and funded to support
RESOURCES the use of software in qualitative data analysis.
The past decade has seen a dramatic proliferation of books, http://caqdas.soc.surrey.ac.uk/
journals, Web sites, and listservs dealing with various Web Sites—Product-Specific
aspects of qualitative research. The following is a small AnSWR, EZ-Text: CDC, http://www.cdc.gov/ hiv/
selection of resources the reader might find useful for software.htm
learning about computer-assisted qualitative data analy- ANTHROPAC: Analytic Technologies, http://www.
sis. Though several of the books contain helpful advice analytictech.com/
on selecting software programs, developments in these ATLAS.ti: Scientific Software Development, http://www.
products occur so quickly that it is best to consult atlasti.com/
Internet resources that are more frequently updated for Ethnograph: Qualis Research Assoc., http://www.qualisre-
help with selecting a program. The Web sites listed search.com/
below contain links to many software sites in addition Hyper-Research: ResearchWare, http://www.researchware.
to those listed here. The software sites in turn contain com/
links to product-specific listservs. N6, NVivo: QSR International, http://www.qsrinterna-
Books tional.com/
Denzin, N. K., & Lincoln, Y.S. (2000). Handbook of quali- Listservs
tative research (2nd ed.). Thousand Oaks, CA: SAGE QUALRS-L—Qualitative Research for the Human
Publications. Sciences, University of Georgia: http://www.listserv.
Glaser, B., & Strauss, A. (1967). The discovery of uga.edu/archives/qualrs-l.html
grounded theory: Strategies for qualitative research. ATLAS.ti Mailing List: http://www.atlasti.com/maillist.
Chicago: Aldine. shtml
Lofland, J., & Lofland, L. (1995). Analyzing social settings QSR Forum (N6, NVivo): http://www.qsrinternational.
(3rd ed.). Belmont, CA: Wadsworth. com/resources/qsrforum/qsr_forum.htm
C H A P T E R 2 2
Qualitative Approaches to Interpreting

and Reporting Data
Heather Dillaway • Cathy Lysack • Mark R. Luborsky
There are many ways in which qualitative approa- (from ethnographic field notes, for example), inter-
ches to data analysis are unique. This chapter out- views, or documents (Miles & Huberman, 1994).
lines some of the basic purposes and stages of Reading through observational field notes, inter-
qualitative analysis. It also discusses how qualita- view transcripts, or examining documents, the
tive analyses can be reported to a broader audience researcher looks for:
after they are completed. • Patterns or common expressions of people’s per-
While there are many different qualitative ceptions or understandings of their world,
approaches to analysis, this chapter does not deal • The meanings they attach to aspects of the set-
with the specific nuances between them. Rather, it tings in which they live or the behaviors in which
covers the broad themes and major characteristics they engage,
of qualitative analysis and reporting. It aims to • The reasons why people think particular things,
provide an understanding of the general philoso- • The ways in which people account for or come to
phies behind and steps within any type of qualita- particular actions, or
tive data analysis or reporting strategy. Interested • How they organize their day-to-day situations.
readers may wish to explore grounded theory,
narrative analysis, phenomenological analysis, dis- At base, then, qualitative analysis is about un-
course analysis, semiotic analysis, and other spe- derstanding meanings, processes, people and their
cific data analysis strategies themselves after thoughts and actions, through the interpretation of
reading this chapter. people’s words.
This first section of this chapter defines the While qualitative researchers easily could
qualitative approach to interpreting or analyzing reduce participants’ words to numbers, it is impor-
data, and discusses some major differences bet- tant that they do not in most cases. The reasons for
ween qualitative data analysis and quantitative data this can be found in the purposes of qualitative
analysis. It identified what makes qualitative data research. That is, qualitative researchers aim to
analysis unique. The sections that follow explain describe and explore a particular topic, group, or
the process of completing qualitative analysis and culture. Qualitative research is useful when a par-
reporting qualitative data once analysis is com- ticular topic or group has not been explored before,
plete. The chapter concludes with reminders about but it can also be used when much is already
qualitative approaches to analysis and reporting. known about a topic and deeper understanding and
explanation is needed to fully comprehend the
phenomenon at hand. In both situations, people’s
The Qualitative Approach to presentations or constructions of their own worlds
Data Analysis: The Uniqueness are very important since the central concern of the
researcher is to describe and gain new insights.
of Qualitative Data Analysis Quantitative analysis addresses cause-and-
effect, that is, how one independent variable or
In broad terms, the qualitative approach to inter- multiple independent variables may affect a
preting and analyzing data has to do with interdependent variable. The goal, then, of quantitative
preting words, not numbers.1 The words that are research is often more about explanation than
analyzed are either in the form of observations description. In this latter case, numerical analyses
become fruitful as they can help the researcher
1There are some qualitative researchers who analyze images explain or predict relationships between two vari-
(moving or still) as their data, but we do not deal with this
form of qualitative analysis in this chapter. In fact, we ables. But because qualitative researchers aim to
restrict most of our discussion to qualitative interviewing describe and explore and understand, they need to
and observations. preserve people’s voices and the context that sur-
Chapter 22 Qualitative Approaches to Interpreting and Reporting Data 373
rounds their voices. Ultimately this means that patient characteristics (e.g., sociodemographic,
qualitative researchers preserve the original format psychosocial) or health behaviors (e.g., drug use,
of their data (or stay as close to the original as pos-
risk profile) in order to determine the statistical
sible) so that readers of their analyses can see or relationship between these factors and patient
experience for themselves as much of the partici- medication adherence, where adherence is gener-
pants’ world as possible. ally specified as a percent (e.g., 90% adherent). In
the literature, adherence has been variably opera-
tionalized as the number of pills or doses taken
An Inductive Process
over a specified period of time, numbers of days
Qualitative analysis is deeply inductive. This is a with no missed doses, or patient self-report of per-
characteristic of several steps and stages in the cent adherence (e.g., “how adherent have you been
overall qualitative research enterprise. Thus, it was in the last month?”) and so on. Although many
also emphasized in Chapter 20, which discussed patient characteristics and health behaviors have
the gathering of qualita- been shown to be statisti-
tive data. The idea cally correlated with
behind an inductive In broad terms, the qualita- adherence, their explana-
approach is to allow tory value is low (only
research insights to tive approach to analysis small amounts of the vari-
emerge over the course centers on the meanings, ance in patient adherence
of the study without is explained) and there-
being limited to pre-
actions, and values embed- fore it remains difficult to
defined factors or pre- ded in social life. accurately predict who
determined theories. will not comply, based on
Miles and Huberman patient characteristics.
(1994) suggest that the researcher “attempts to A qualitative approach to medication adher-
capture data on the perceptions of local actors ence, in contrast, focuses on the understandings,
from the inside, through a process of deep atten- meanings, and experiences of living with a particu-
tiveness, of empathetic understanding…, and of lar disease and adhering to medications. An exam-
suspending or ‘bracketing’ preconceptions about ple is a study of adherence to HIV anti-retroviral
the topic under discussion” (p. 6). Emerson, Fretz, medications among African Americans in Detroit.
and Shaw talk about this same process as being One finding is that patients hold a different under-
one of “ethnographic participation.” For these standing of how to be adherent than their doctors
authors, qualitative research requires the investiga- do. Patient understanding of the notion of being
tor to achieve a “deeper immersion in others’ completely adherent often means “managing all
worlds in order to grasp what they experience as aspects of one’s illness the best one can given the
meaningful and important” (1995, p. 2). Mishler circumstances,” which may mean skipping doses
(1979) has described this process as being more of the prescribed medications (Sankar, Neufeld, &
fundamental still. He wrote that any meaning that Luborsky, 2005)!
is extracted out of the interactions between the Thus, the power of qualitative research in this
researcher and the researched is fundamentally case is to explicate the variety of meanings of
social, and cannot exist without the context from adherence held by patients and uncover reasons for
which it was generated. Mishler argues the patients’ behaviors in relation to compliance with
researcher and the researched are inextricably therapeutic medication regimens. As evident from
intertwined: the phenomenon studied cannot even this example, qualitative studies focus on uncover-
have objective characteristics independent of the ing how specific behaviors like taking medications
observer’s perspectives and methods. Therefore, a are understood by patients in the context of their
researcher using an inductive naturalistic approach daily lives. This type of research is a powerful way
is not only centrally concerned with the identifica- to interrogate taken for granted assumptions about
tion of key themes and findings, but also concerned what constructs, like adherence, really mean on the
with epistemology, or what counts as knowledge. ground in real people’s lives.
The following example illustrates some of
these fundamental differences between quantita-
The Goals of Description and Explanation
tive and qualitative research. In the area of adher-
ence to medications, quantitative approaches Another significant way in which qualitative
have largely been used to measure a small set of research differs from quantitative research is in its
ultimate goal. The goal of qualitative analysis is Without this very systematic work to interview
not to come up with conclusions that could be gen- individuals with HIV and follow their logic, these
eralized to populations. In many cases, the qualita- insights and understandings would not be uncov-
tive researcher’s goal is simply to describe his or ered. This type of research is clearly essential in
her particular sample and/or the small group that the areas of health-related behaviors and services.
he or she studied. This description begins an often In keeping with its purpose, qualitative
new and in-depth understanding of the topics researchers depend on nonprobability purposive
under study or people under study, and often spurs sampling procedures to recruit the most informa-
on larger, future research projects that could be tive people possible to illuminate the topic of inter-
more population-based. In other instances, as in est. Again, the value of qualitative research lies in
the example on HIV medication adherence des- the “thick description” (Geertz, 2001), or deep
cribed earlier, the goal is to gain deeper insight understanding gained. Depth of understanding and
into human behavior across a larger sample, in an proximity to people’s worlds cannot be found in
effort to achieve full understanding across a wide more structured, deductive, quantitative, popula-
range of study subjects. tion-based research.
As another related example from the same
study, it has long been observed that many indi-
viduals with HIV are diagnosed only when they
Data Collection and Analysis Intertwined
are very ill and hospitalized, and this has been Another extremely important characteristic of
attributed to a failure of public health education to qualitative analysis is that it begins during data
properly teach people about HIV risk groups and collection and then continues after data collection
behaviors. Qualitative research asks the “why” is completed. Thus there is a “zigzag” or back-and-
questions such as, “Why do some people wait so forth process of data gathering and analysis, espe-
long to get tested for HIV?” Answers such as, cially in the early stages. At times, especially in
“because I didn’t think I was at risk,” generate fol- grounded theory studies, data analysis might also
low-up questions such as, “Why didn’t you think be interrelated with sampling and with the choice
you were at risk?” Results show that individuals of setting for the research, as researchers try to find
have learned the health messages (i.e., they know people or settings that will garner them more infor-
what risky behavior is and precautions needed to mation about particular themes arising in early
avoid HIV infection), but they don’t fit the risk cat- field notes, journals, and interview transcripts.
egories defined by the experts (e.g., injection drug In qualitative research, several steps in the
users, gay) (Neufeld, Berry, & Sankar, 2005). research process (i.e., sampling, recruitment, entry
Contributions of Qualitative Research to Understanding People’s Worlds
Qualitative research is particularly valuable in It Crazy (1981, 1985) and revealed how our soci-
providing an understanding of the unique circum- ety and even medical professionals stigmatize and
stance of different groups of people. For instance, discriminate against this group. Howard Becker
we would never know as much about how poor and colleagues (1961, 1991) Boys in White, is
communities share resources (e.g., cars, childcare, another classic ethnography in book form. In
money, etc.), if it were not for ethnographies such this study, Becker and his team conducted inter-
as All Our Kin by Carol Stack (1974). Although views with a cohort of medical students as they
this research is not generalizable in the quantitative moved through their medical school training. This
sense, it is nonetheless very valuable if our goal is research focused on the cultural rites of passage
to understand the role of kin and community and rituals of professional indoctrination that
resource networks among poor, racially disadvan- transform medical students into doctors. Another
taged groups. There are many other seminal contri- excellent ethnography that focuses on cultural
butions of qualitative research of relevance to the practices is Everyone Here Spoke Sign Language
health sciences and to occupational therapy. (Groce, 1985). This book is the result of a study of
Many of these qualitative studies are ethno- the islanders on Martha’s Vineyard and how they
graphic in nature and represent relatively pro- worked to include those with congenital deafness.
longed periods of research time in the field. When All of these examples enter the life worlds of
this is the case, the reporting of the study often social and cultural groups and explore their prac-
takes the form of a book. For example, Sue Estroff tices and rituals that help explain how and why
documented the lives of the mentally ill in Making things come to be the way they are.
into the field, data collection, analysis, etc.) may that most, if not all, of these second round ques-
be intertwined. For example, in qualitative studies, tions are impossible to identify and plan for in
interviewers might realize early in data collection advance. For example, in responding to questions
that a particular theme is arising, and may vary the about their medication adherence, many patients
main questions and probes that they use in later reported their number of missed doses (during
interviews so that they can explore this theme some specified period of time), but later in the
more fully. In her study of menopausal women, interview or in “side-talk” revealed that the mean-
Dillaway (2005) discovered in early in-depth inter- ing of a missed dose may not be straightforward
views that women did not feel “old” upon this life (e.g., “I missed that dose because I did not take it
stage. Because she began analysis of these early until later”). As a result, a new set of questions
interviews before data collection was over she was about the meaning of a missed dose was developed
able to explore the reasons why women felt this and given to patients and their clinicians. Results
way overall, finding that many women distin- revealed large variability in the understanding of
guished between reproductive aging and other “missed dose,” and what to do about missed doses,
types of aging. That is, according to her later inter- among both patients and clinicians and substantial
viewees, menopause was positive because it differences between the two groups (Sankar,
allowed women to let go of a function that they Nevedal, Neufeld, & Luborsky, 2005). The results
were ready to give up, whereas other aging condi- have implications for the measurement of self-
tions (e.g., losing eyesight, Alzheimer’s disease, reported adherence, for patient–doctor communi-
etc.) meant the loss of a function they desired to cation, and for medication adherence itself.
keep (Dillaway, 2005). This theme was fully devel-
oped only during data analysis because data collec-
tion strategies were altered to explore menopausal
women’s perceptions more deeply through probing
Steps in the Analytic
and follow-up questions. Process: Carrying Out
In another example, recent studies have found
that older adults may not adopt a mobility aid such Qualitative Data Analysis
as a scooter or a walker just because their occupa-
tional therapist tells them it will make them safer While this section focuses on analysis and not on
and more independent. Issues like their sense of data collection, the reader should keep in mind that
being seen and treated as a disabled person and the two processes are not separate, and thus what
“less” than a full adult person actually weigh more was discussed in Chapter 20 should be kept in
heavily on their decision to acquire and utilize a mind. A key to good qualitative analysis is that it
mobility aid than many people, including thera- is not a time-bound process, and that data collec-
pists, realize (Gitlin, Luborsky, & Schemm, 1998; tion and analysis are not partitioned from each
Luborsky, 1997). The fact that individuals’ mean- other. At base, there is a reflexivity that develops
ings of mobility aids differ from occupational ther- between data collection and analysis in qualitative
apists’ perceptions is something that might be research. And this reflexivity specifically creates
discovered only through a back-and-forth relation- the ability to achieve new and deeper understand-
ship between data collection and analysis. In early ings of particular perceptions, behaviors, and ways
data collection, an occupational therapy researcher of human living.
might not be inclined to ask older individuals
about their own perceptions and, rather, may begin
The Importance of Data Processing
by asking about their contact with therapists and
their recommendations. Yet, if analysis starts dur- Data processing must occur for data analysis to
ing data collection, a researcher could quickly pick begin; in some ways it is the first step one takes in
up on the fact that they might not be exploring this interpreting and reporting data. Chapter 21 cov-
issue in full, and that individuals’ perceptions of ered many of the mechanics of data processing, so
mobility aids may be more important than the rec- they are briefly noted here. Data processing
ommendations they are given by their occupa- includes making permanent records of interviews,
tional therapist. field notes, or other documents that count as data
Finally, it is useful to return once more to the (Rubin & Rubin, 2005). For example, data pro-
HIV adherence study described above, since it cessing might be turning hand-written field notes
very ably demonstrates how the results obtained into typed computer notes or transcribing audio-
from one set of qualitative research questions drive recorded (and/or video-recorded) data from in-
an entirely new set of questions. It should be noted depth interviews. Transcribing is the process by
Rigor, Flexibility, Challenge, and Reward in Qualitative Analysis
Qualitative analysis—and qualitative research in Qualitative researchers follow particular steps

general—should be thought of as both a flexible and verify their findings. At the same time,
and rigorous process. What qualitative researchers they adapt to the data that is arising in front of
select to analyze in their data is not set in stone, them.
and how qualitative researchers interpret these data Qualitative data analysis is a complex process
can depend on many things (e.g., existing litera- that can be both time-consuming and exhausting;
ture, the research questions or research problem he thus, it can be challenging. In qualitative research,
or she begins the project with, the kinds of themes and largely because data collection does not occur
that arise in early interviews or early analytic in controlled situations, data collection and espe-
notes, or a combination of all of these). Yet, at the cially data analysis will take considerable energy
same time, there are particular systematic steps and time. The important thing to remember is that,
that a qualitative researcher goes through in order at the same time as qualitative analysis can be
to arrive at a final interpretation of the data that challenging and time-consuming, it is a rewarding
can be reported. process as one sees participants’ voices and lives
While qualitative research is often touted come into focus as one shares these voices with a
for its flexibility and subjectivity, it is as impor- larger audience. Those who engage in qualitative
tant to think about how qualitative research is research will acknowledge the difficulties that oth-
simultaneously precise, detailed, and valid or ers have in qualitative analysis, but these difficul-
trustworthy because of the process that resear- ties do remain somewhat hidden from those who
chers undertake in data collection and analysis. do not engage in this research.
which one listens to taped recordings of inter- gain a deep understanding of the themes that might
views, and types verbatim what is said during the be arising out of the sample. With early data pro-
interview so that the original data is preserved. cessing, one makes the best use of the strengths of
Many researchers also take handwritten notes dur- the qualitative approach to interpreting and ana-
ing in-depth interviews or as part of textual analy- lyzing data.
sis in addition to audio recordings, so those notes Regardless of the fact that data has been pro-
would need to be typed as well. Data processing cessed, when starting one’s analysis, one should be
might also include making paper copies of all orig- looking at and interpreting collected data in its
inal data files stored in a computer so that original original form. Thus transcriptions and typed notes
data is not lost during data collection, data analy- should be as precise as possible. Certain types of
sis, or afterwards. qualitative analysis in particular (e.g., phenome-
When analysis starts, one’s analytic work nology, narrative analysis, or semiotic analysis)
should take place with or on a duplicate copy of require that processed data mirror exactly what
the original data so there is no chance to ruin the was said in an interview or what was typed in an
original data; this also allows one to backtrack and original field note, but all qualitative analysis
start an analysis over if need be. At base, numerous should be initiated with data in as close to its orig-
attempts should be made to preserve data in its inal format as possible. When and how an inter-
original format in multiple copies (and stored in viewee laughs during a conversation, for example,
multiple, safe locations) so that the researcher can may be important in telling the researcher about
always find and access it. This is particularly true their feelings of nervousness when discussing
of computer files—backing up data means not only about a particular topic or about the level of trust
peace of mind but also assurance against corrupted the researcher has gained during the interview.
files, computers that crash and ruined files, stolen How quickly an interviewee answers a particular
or damaged computers, and the like. interview question may tell the researcher how
Just as importantly, data processing should at strongly that person feels about a particular topic.
least begin alongside data collection, so that For those using semiotics or a linguistic approach
important themes in early interviews are not lost to data analyses, even more detailed attention to
and the benefits of the reflexive, back-and-forth the level of words and even syllables is under-
process between data collection and analysis can taken. Thus, interview transcriptions, depending
be captured. The earlier data processing is com- on the question, might even record laughs, hesita-
pleted, the earlier one can begin the analysis tions, interruptions, emphases, and other speech
process. Moreover, the earlier one starts the analy- patterns as much as possible, alongside the actual
sis process, the more likely one is able to grasp and words of the conversation.
Riessman (1993) also notes that when an inter- these thoughts may occur at random times (e.g., in
viewee or informant appears to be going off on a the shower, lying in bed at night, driving). Thus,
side topic or tangent in conversation, one must keeping a journal or a set of informal computer
make every effort to follow and keep records of notes that can be accessed easily at all times is a
that side conversation, because later on a way of documenting such thoughts, no matter
researcher might realize that conversation was an when they occur. These informal notes or journals
entry into a story that is relevant for and answers become important sources for ideas of how to ana-
one’s research questions. Particular care should be lyze and report about the pile of data that one
taken during data processing so that all of these accumulates over time. This informal data source
data are completely and systematically captured so can make the analysis stage less cumbersome,
that original data is maintained. especially at the beginning.
As noted in Chapters 20 and 21, researchers can
and do expand upon their primary data (e.g., hand- Next Steps After Accumulating
written field notes, audiotaped interview data) by a Pile of Processed Data
keeping informal notes, memos, or a field journal
during data collection. Notes could be about Rubin and Rubin (2005) suggest that it is helpful
important ideas that interviewees or participants to think about data sources being broken down into
are suggesting about the topic or their culture or data units, or blocks of information that will even-
their experiences, or their behavior, as well as gen- tually be examined together. Investigators must
eral ideas that the researcher begins to have about begin their analysis by figuring out what blocks of
what they are observing and hearing and would information are appropriate to analyze. These
like to analyze as complex themes later in analysis blocks of information could be:
(after they are out of the field). They might also • Stories that interviewees tell that span several
record key quotes from interviews or conversa- typed pages,
tions that researchers overhear that might be • A paragraph of typed text from an interview or
important in the data analyses later on. Notes field note in which an individual describes a par-
could detail researcher’s personal assessments or ticular encounter with a therapist, or discussion
opinions about how interviews or field observa- of a particular health condition and its meaning,
tions are going and the ways in which data collec- or
tion needs to be altered in order to secure a deep • A phrase or term used over and over by intervie-
understanding of the topic/group at hand. In addi- wees or informants or members of a particular
tion, notes include one’s reactions to being in the organization, clinical team, or profession.
field if one is doing participant observation, or
feelings about the course of any type of data col- The characteristic of these blocks of informa-
lection experience. tion is that they signify a key concept or theme.
The purpose of this extra data can be twofold; Thus, initial analyses consist of multiple readings
they can: of one’s data sources, to look for the blocks of
information that might be worth analysis. Once the
• Highlight key events during observations or researcher finds a block of information that seems
interviews and thereby serve as a check on one’s to emerge as important, then she or he might look
primary data source, and/or to see if a similar block of information exists in a
• Serve as the first step to data analysis by getting second data source, whether that data source is an
one’s main ideas out on paper to propel more individual or a large group of individuals. Then,
advanced analytic thinking about the study topic. the researcher begins a process of comparing and
For these reasons, all qualitative researchers contrasting across data sources, to see whether this
should keep some sort of informal notes or a jour- data unit is important across a particular sample, as
nal as they engage in qualitative data collection. well as how and why it is important to the people
These notes or journal entries will help in the interviewed or observed.
interpretation and documentation of one’s research
Four Steps in the Data Analytic Process
experiences and one’s data (Emerson, Fretz, &
Shaw, 1995; Rubin & Rubin, 2005). They can be The analytic or interpretive process is discussed as
typed or handwritten, but optimally, should be four separate steps in this section. What follows is
made immediately after one leaves the research only one, albeit a common approach to data analy-
setting for the day. One may also have thoughts sis. While data analysis frequently does begin with
about how data collection is proceeding, or about a singular data source as discussed below, it can
the types of patterns arising in interviews, and also begin with responses to an entire set of inter-
Qualitative Results versus Qualitative Interpretation
Qualitative results are not useful until they are or confirming existing knowledge or by con-
interpreted in order to gauge their credibility, tributing new information.
importance, and meaning. The term “analysis” The following are recommended strategies for
is avoided by some qualitative researchers because interpreting results:
it is used to refer to quantitative standardized 1. Tie the findings into one or more major contem-
approaches. Thus, some qualitative researchers porary theories related to the research topic. For
prefer to describe all the tasks of extracting example, Deppen-Wood, Luborsky, and Scheer
meaning from the data as “interpretation” to (1997) have studied and written about post-
better characterize their meaning-centered, polio syndrome. In this case study research, the
naturalistic, interpretive paradigm. Nonetheless investigators interpreted the results of one older
qualitative researchers are encouraged to be African-American woman’s life-long accommo-
aware of the two separate stages and goals in dations to mobility disability. The authors con-
the research process. In multidisciplinary col- sidered the multiple jeopardy hypotheses, a well
laboration, the rigor and value of qualitative known theory that suggests that disadvantage
research are reinforced when investigators can create a cumulative burden. Specifically, in
clearly describe these steps separately, however this case, the multiple jeopardy hypothesis sug-
they are labeled. gests that visible minorities, women, and per-
Thus, the following distinctions are made: sons with disabilities are more disadvantaged
• The results of analyses are simply what an inves- than nondisabled white men. The authors of
tigator finds out or produces by conducting pro- this study assessed this hypothesis in their
cedures for processing and analyzing the data analysis but through careful review of their
(e.g., those procedures discussed in Chapter 21). data ultimately rejected this idea, in favor of an
Hence, results are just an intermediate, internal alternative interpretation. In this way, the authors
step in the research process and have no inher- framed the results of their study within a larger
ent meaning or value. For example, listing three context and body of literature and theory. To
themes does not locate those findings within the find out more about this case, one should read
larger framework of what we already know. It the case study and see why this woman pre-
also does not reveal anything about the cred- ferred to use a cane instead of a walker in public
ibility of these particular results. even though it provided much lower objective
• Interpreting the results establishes the value or functional mobility.
importance of the data analytic results. Only 2. Connect the analysis to a substantive or con-
after the analytic results are interpreted can they troversial issue; that is, compare the results of
be evaluated to determine if, how, and how much the qualitative analysis to those of prominent
they contribute new knowledge. researchers on the topic or link the analysis
Qualitative researchers interpret results by eval- to a high-profile issue or controversial debate in
uating the resulting product of the analytic proce- the field of study. For example, in rehabilitation
dures in at least two ways: research it has long been believed that the dura-
tion, intensity, and focus of physical rehabilita-
• First, investigators gauge the results in two ways, tion is the primary cause of functional recovery.
in order to be clear about how credible they are. This has been generally accepted in stroke and
This is done in light of internal issues concerning orthopedic rehabilitation. More recently, how-
the “epistemological” status (see Chapter 20) of ever, researchers have found that the presence
the data to gauge its strengths and limitations of key social relationships may be as critical
(e.g., trustworthiness, validity, reliability, sample if not more critical to long-term functional
adequacy). In addition, with an eye to how credi- independence and community integration after
ble the data are, investigators assess the findings stroke, hip fracture, and a wide array of other
in light of what was already known before start- physical impairments as well (Dijkers, 1999;
ing the research, and answer the question of how Glass, Matchar, Belyea, & Feussner, 1993;
the results add new knowledge by contradicting Magaziner et al., 2000).
view questions across a group of study partici- Step One: Formal Analysis or
pants. Furthermore, when the latter is the case, Interpretation of One Data Source
“codes and coding” refer to an analytic category
and not merely the words of one single study par- Formal analysis or interpretation of qualitative
ticipant. These distinctions are drawn below. data often begins with one data source. This may
be one interview or one field note, for example. case analysis, in that one is doing all one can to
The researcher reads through this data source mul- master an understanding of a particular interview
tiple times, gaining comfort with what it includes. transcription or a set of observational field notes.
Rubin and Rubin (2005) explain how, in this step, If, on the other hand, the research is less
one gains recognition of the concepts, events, feel- exploratory and more conceptual (i.e., theory and
ings, and behaviors recorded, as well as the pat- literature driven), qualitative investigators may uti-
terns within that particular data source. For lize more semistructured interviews, and as a result
instance, an investigator may ask whether an inter- be in a position to focus on a clearly defined set of
viewee continually referred back to family history, data provided in response to a clearly delineated
situating the disability experience within the con- subset of interview questions. In this case the sin-
text of the extended family, or whether the inform- gle data source is somewhat more broadly con-
ant characterized the disability experience as strued. When this is the case, the researchers
owing very little to the extended family. undertake a systematic process of comparisons
Once researchers are familiar with the initial across study participants simultaneously. They are
data source, they should begin coding it. That is, also engaging in a deeper level of explanatory
investigators should come up with brief terms that coding. In essence, the researchers are looking
can be written in the margins or recorded within a beyond the words and descriptions of their inter-
computer program that summarize blocks of infor- view subjects to understand more fundamental
mation in that data source. Schwandt (1997) conceptualizations of how people experience what
defines coding as “a procedure that disaggregates they do.
the data, breaks it down into manageable segments Geertz (2001) has referred to this process as
and identifies or names those segments” (p. 16). It inscribing social discourse; it is a deliberate act to
is impossible to undertake coding without at least ensure that interactional details are fully captured.
some conceptual structure in mind. Nonetheless, This level of detail in both the collection of data in
coding can be under- fieldnotes and in the
taken in a more descrip- interpretive process of
tive mode or in a more It is very easy to be data analysis is demand-
analytic mode, depending, but holds the poten-
ing on the level of inter- mechanical in the coding tial to explain categories
pretation involved. process but much more of meaning in a way that
The most trouble- more superficial attention
some tendency in the act
difficult to address the theo- to words alone cannot.
of coding according to retical understandings that Irrespective of the
Strauss (1987) is to code are involved in understand- stance taken, it is gener-
too much at the descrip- ally accepted that one
tive level instead of coding social phenomena. should code everything
ing for the purposes of in the data source that is
explaining or develop- relevant to one’s research
ing an understanding of what is going on. This questions or research problem. However, this does
understanding is the heart of data analysis for not mean that one will code everything within a
thoughtful, reflective, and critical qualitative data source (Rubin & Rubin, 2005). One must
researchers. It is very easy to be mechanical in the make a great many decisions about what data units
coding process but much more difficult to address are related to one’s initial goals in the research
the theoretical understandings that are involved in project, and whether/how particular data units
understanding social phenomena. Returning to the address the research questions one sets out to
example alluded to briefly above, descriptive cod- answer. Certainly, some research questions take on
ing could thus take place every time a person refers greater priority, as is the case when a particular
to his or her family history during an interview, for manuscript on a specific topic is planned. Thus, at
instance, “family history” (even “fam. hist.” or times, coding must be strategically undertaken in
“FH”) could be recorded or “coded” next to that response to practicalities such as available time
data unit. At this descriptive level, the goal is to and energy.
find a way to track similar blocks of information Once data are coded, important and related
through a coding system, finding brief terms that blocks of information are easy to find and access
categorize multiple areas of text. Miles and for further analysis and for reporting. Codes essen-
Huberman (1994) refer to this process as within- tially flag patterns or themes that rest within the
data. By defining and labeling one’s codes, and units or pattern among data units) for further
through the acting of assigning codes to their data, analysis. How one selects a particular code/pattern
qualitative researchers begin to identify salient in the data for this second step of analysis is vari-
analytical connections among data units (even able. Researchers could select a particular code to
within the same data source). Thus, regardless of analyze based on their reading of literature on their
whether coding is undertaken on an informant by topic of study, or based on the research questions
informant basis, or is done across subjects simul- with which they began their project. However,
taneously at a more conceptual level, coding must qualitative researchers need to know about past
be done with great care since it forms the bedrock research but not let it dictate the specific data units
of subsequent interpretations. they’ll analyze. Over-reliance on past literature
For example, an investigator may wish to know and preconceived research questions can constrict
how interviewees understand and describe a par- the ways in which one is reading the data source
ticular health problem such as hip fracture in old as a whole and/or how one interprets parti-
age. The investigator may further be interested to cular blocks of information (Emerson, Fretz, &
determine whether particular symptoms of a hip Shaw, 1995; Miles & Huberman, 1994; Riessman,
fracture are more salient than others or whether 1993).
certain meanings of having a hip fracture manifest Thus, investigators should pay attention to
over and over again in the same data source. To codes/patterns that simply emerge out of the data
address these concerns, a specific code should be source themselves, upon the investigators’ critical
created for each issue to detail what is important reflection. For example, Siporin and Lysack (2004)
about each data unit. A researcher may go back studied the work settings of women with develop-
and recode this data source at a later point, but mental disabilities. They relied on prolonged
these initial analytic steps are essential to estab- engagement in the field and in-depth interviews to
lishing a systematic and theoretically informed illuminate how different the experience of working
analytic strategy and for selecting particular data in a sheltered workshop was from working in more
units to explore further. community-based supported employment pro-
grams. This study relied on prolonged periods of
Step Two: Selecting a Particular observation in the workplace of clients to show
Code for Further Analysis how factors in the physical and social environment
operated to enhance or limit the quality of life per-
After the researcher completes the coding of one ceived by the clients.
particular data source, the next step is to select a Specifically, the investigators learned that the
particular code (and thus, a particular set of data type of work characteristic of sheltered workshops
as opposed to supported employment settings
accounted for only a small amount of the clients’
subjective quality of life. This finding contradicted
the investigators’ expectations that type of work
would matter much more. Other factors such as the
personalities of agency staff and life coaches, and
the types of social opportunities linked to each dis-
tinctive type of worksite, turned out to be more
powerful influences on clients’ expressed quality
of life. If these investigators had not paid attention
to codes they attached to these other aspects of the
work environment they may have overlooked a
range of personally meaningful interactions with
agency staff and fun opportunities outside of work
as key contributors to self-rated quality of life.
Other things may also influence what analytic
strategy an investigator chooses to pursue. For
instance, the way in which current media is fram-
ing an issue/topic/cultural group may be important
Figure 22.1 Members of a qualitative research in shaping one’s analytic strategy. A qualitative
team discuss their coding and analysis of an researcher engaged in a study on aging may note
interview transcript. that aging is sometimes characterized negatively
by the mainstream media, and will, in response, ily history led directly to his or her own health
develop and assign one or more of codes that condition. Therefore, such an interviewee might
speaks to this issue. On the other hand, ideas from characterize the impact of family history as nega-
one’s informal notes or field journal may generate tive. However, a second interviewee might also
some new codes. There are numerous ways in talk about family history, but consider his or her
which researchers select the codes and data units own health problems to be an anomaly. In this sec-
they analyze. Qualitative investigators must be ond data source, then, the interviewee may have
aware and open to a range of different strategies characterized family history more positively. In
when developing their approach to data analysis this example, the comparison of similarly coded
and coding. text across two data sources leads to a greater
An important strategy for simplifying the com- understanding of how individuals attach impor-
plex task of data analysis is picking only one tance or meaning to their family histories. It may
code to analyze at a time. This strategy can be par- become clearer as one compares and contrasts sim-
ticularly useful advice for novice qualitative ilarly coded text that, while importance is placed
researchers. Because qualitative data include on family history across interviewees, the particu-
many different patterns and themes a good strategy lar meanings they attach to family history vary. In
for the novice is to keep one’s attention on one pat- such an instance, the codes may be adjusted to fit
tern in the data at a time. There is always time to the new understanding of these data units as they
go back and analyze other sets of data units at are compared and contrasted. This is a repeated
other times. As will be discussed later, the pro- process, as one compares and contrasts each data
cess of qualitative analysis is never completely source to others until one has analyzed all cases in
over. Thus, researchers might as well dedicate a sample (Miles & Huberman, 1994).
themselves to just one specific analytic task and The decisions one makes during initial coding
carry it through to completion before undertaking and coding revisions shape what a researcher will
another. be able to conclude at the end of the analysis.
Thus, the process of coding and then comparing,
Step Three: Compare and Contrast Data contrasting, and revising one’s understanding of
Sources Utilizing a Particular Code codes is key to the analytic or interpretive process.
One’s emphasis should be on either finding ways
After one has read and coded one particular data of relating coded blocks of data together (similari-
source and has selected one code or pattern to ties) or finding differences in the way data units
highlight in further analysis, the next step is to uphold a particular code, so that the nuances
compare and contrast data sources utilizing a par- underneath a code can be brought to the surface.
ticular code. Miles and Huberman (1994) call this
step, cross-case analysis. In this step one asks Step Four: Drawing Some General
whether it is possible to:
Conclusions About What a Coding
• Move to the next data source and find similar Strategy or Arising Data Pattern Means
ideas or topics, and
In a final step of analysis, investigators stop spe-
• Code that second data source in similar ways
cific cross-case comparisons and zoom out, so to
(i.e., using the same code selected in Step Two).
speak, on what has been found thus far. In this
In this step, the researcher is essentially clarify- step, the investigator will draw some general con-
ing what is meant by a particular data unit or coded clusions about what a coding strategy or arising
block of text (Rubin & Rubin, 2005). As one com- data pattern means. This allows both the consis-
pares blocks of information in the first and second tencies and differences among a particular set
data source, one begins to understand the similar of data units to be described and understood.
and different ways interviewees or field notes dis- Therefore, this step is about arriving at a kind of
cuss a particular topic. Thus the information stored final synthesis (Rubin & Rubin, 2005) or a the-
within each code is analyzed for nuances and the matic narrative (Emerson, Fretz, & Shaw, 1995)
meaning of the original code becomes clearer than that illuminates the theme or pattern that has been
it was in previous steps. recognized or identified in the cross-case analyses
Going back to the example of how people completed in Step Three.
might talk about the role of their family history in The goal of this step is to develop a way to refer
explicating their disability experience, the resear- to the theme or pattern, and the subthemes and
cher might find that one interviewee felt that fam- subpatterns beneath it (if one has created subcodes
underneath a main code, which is often the case). further and broader implications that might have
This is very much a reflective analytic step, one in meaning outside of any particular research sample.
which the researcher must step back and find a While this is a final step in analysis, it should be
way to evaluate all data units underneath a specif- acknowledged that an investigator may return again
ically coded theme. The key questions2 one needs and again (often in response to reviewers’, editors’,
to answer in this step are: and particular audiences’ comments) to clarify
what is meant by particular data units explained by
• What am I going to call this theme as I move into
particular themes. Investigators may also add fur-
a reporting stage?
ther nuances to the understanding of how data units
• How am I going to define the nuances I found
fit (or do not fit) together at a later date.
within it as I looked at different data units and in
It is also important to note that researchers typ-
different data sources?
ically move back and forth among these major
• What are the examples of variations under the
steps as they attempt to make sense of the data
umbrella of this coded theme, and how can I
units relevant to their research question or research
explain them? Why do these variations matter?
problem. In the grounded theory tradition, this iter-
• What are examples of the commonalities across
ative process is called the constant comparative
all data units I have labeled with a given theme,
method (Glaser & Strauss, 1967).
and how do I explain them? Why do they matter?
Irrespective of the specific approach, just as
• What is the significance of this theme overall for
data collection and analysis can be a back-and-
my research topic, and how might my analysis of
forth process, so too is the analysis process. Inves-
this particular topic/theme add to previous litera-
tigators will move back and forth between looking
ture on this subject?
at individual data sources and multiple data
If one’s coding strategy and previous levels of sources, and the tentative interpretations attached
analysis can hold as one answers these questions, to the data. Only at the point when researchers are
then it is probably the case that one has arrived at finding no more new interpretations or no new
what resembles a complete analysis. Perhaps the similarities and differences should they move to
most important question in the above list is the last conclude that saturation has been reached, and
one. A researcher must think about the broader sig- begin to write up the findings. When one can
nificance of the analysis just completed, and draw a small set of conclusions about the interpre-
whether or not there are general implications of tations that help to answer one’s initial research
what one has learned. Thus, one should ask how questions (and that would be endorsed by the peo-
far the thematic analysis (and thus codes/coding ple studied as reflecting their views) the analysis is
strategies one has developed) might extend. complete.
For example, Lysack and Seipke (2002) found
that oldest-old women often defined “aging well” Enhancing Rigor Through the Steps
differently than policymakers and healthcare of Qualitative Data Analysis
providers. They defined their well-being in terms
of whether they could still complete gendered Emerson, Fretz, and Shaw (1995) urge qualitative
household tasks rather than in terms of established researchers to utilize a variety of organizational
health and independence scales used by therapists strategies to enhance the rigor of their analyses.
or policymakers to determine need for services. These authors and others (Strauss, 1987) devote
While individuals’ definitions of aging well were entire chapters to strategies undertaken in the field
interesting as findings in and of themselves, Lysack to capture scene depictions, to communicate dia-
and Seipke (2002) found their results had policy logue and interpersonal exchanges, to elucidate
and healthcare implications. Moreover, they sug- indigenous meanings with and without verbal data,
gested that traditional notions of successful aging to develop and analyze integrative memos, and
and independence among the elderly need to be many other tasks in the qualitative data collection
rethought at the national as well as local level. and analysis process. All such measures contribute
Thus, while qualitative research is not generaliz- to achieving greater quality in the end product of
able in that it cannot speak to a heterogeneous qualitative research.
population’s attitudes or behaviors, qualitative There are various ways to check oneself during
analyses can still hint at ideas that must be explored the analysis process, or verify the patterns or
themes one is finding in the original data. First,
2Thislist of questions is based loosely on a list of questions some researchers have multiple coders and/or ana-
found in Rubin & Rubin (2005: 216). lysts on their research projects to ensure that they
are reading data units in similar and valid ways. tant, and will require different kinds of informa-
This is a check on the reliability of the emerging tion. For instance, if one has a goal of influencing
interpretations. Second, qualitative researchers policy for the developmentally disabled, one needs
may return copies of their initial analyses to their to think critically about the policy implications in
interviewees or informants in the field, to assess the final steps of analysis and make them under-
how closely they have captured what participants’ standable and believable to policymakers who are
voices and worlds are. This is called member typically not well versed in academic literature or
checking in some qualitative traditions. Third, one research methodology. Presentations to consumers
can look to previous literature to evaluate whether also take on a very different shape. On the other
one’s initial results make sense (although this does hand, if one has an academic audience, it will be
not mean that one’s findings need to mirror the more important to spend time reporting on method-
results of past studies!). Fourth, one might reflect ologic details, and linking findings to the results of
on whether, intuitively, initial findings make sense. prior studies. Thus, the audience of one’s report
Finally, within the analysis process itself, if one’s will determine how the analysis is communicated.
coding holds up across multiple data sources and Another important issue to think about, as one
back-and-forth processes of within-case and cross- begins the reporting process, is how best to retain
case analysis, then one probably has some reason- participants’ voices and/or the original data as one
able grounds to conclude that the coding strategy communicates themes in the data. As noted earlier,
undertaken and larger analytic strategy are valu- one of the key goals of qualitative research is to
able and the results generated from the qualitative report what a group’s thoughts, behaviors, and
data analysis are worth reporting. lives look like from the inside (Miles & Huberman,
1994). To stay close to this goal, the researcher
must maintain a commitment to reporting original
How and Where to data. This means, for instance, reporting partici-
Report Findings pants’ actual words from interview conversations
or verbatim field notes so that readers can immerse
themselves in the original research setting along-
Researchers should make the transition from side the researcher.
analysis to reporting by thinking about why they Since the qualitative researcher is the research
initiated their projects and what their goals were. instrument, great care must be taken to include
For instance, qualitative research may be under- both what is observed or heard, alongside what is
taken with: thought and interpreted by the researcher. This
• A purely basic science goal of adding to existing means that, within a report, examples of original
knowledge on the topic or group, or data must be provided along with interpretations of
• A more applied goal of coming up with ways to these data. Thus, while a researcher may provide
make a particular groups’ lives better in some an interpretation of how to read or make sense of
way. the data, the actual data is also in the report for
readers to interpret on their own.
Investigators also need to consider what form
the results need to be in, in order for one to achieve
their goals. This may include, for instance:
How to Go About Reporting the Analysis
Typically qualitative researchers select a broad
• A peer-reviewed journal article,
pattern or theme to introduce in their report. This
• A book or other lengthy manuscript,
theme is directly related to the codes that are ana-
• A policy brief,
lyzed first in one data source, then across data
• An oral presentation,
sources. How one presents this theme is usually
• A conference poster,
related to how one answers the questions presented
• A consumer magazine or newspaper article, and/
under Step Four of the analysis. That is, one must
or
first name and define what the finding or theme is.
• A TV interview.
This is often about defining one’s coding strategy.
Based on the initial goals, one should also After one presents and summarizes a theme, one
determine the audience—that is, the group of peo- should present an example of original data.
ple who will be reading and reacting to the find- For instance, in reporting the quality of nursing
ings. Different audiences will find different care in their report, Williams and Irurita (1998) dis-
presentations of the analysis more or less impor- cuss how nurses initiate rapport with their patients;
“initiating rapport” was a code they defined during brings the reader into the description as much as
the analysis stage of their research. In their report, possible. The presentation of a theme or coding
they first define this concept: “Rapport was estab- strategy begins broadly, and then becomes more
lished by informal, social communication that specific. This pattern of reporting qualitative data
enabled the nurse and the patient to get to know is widely accepted and well understood as an ade-
each other as persons” (p. 38). Then they present quate way to present complex analyses and, at the
an example, stating: “One of the nurses inter- same time, remain as close as possible to partici-
viewed described this interaction: pants’ voices.
The novice researcher should be mindful of
‘Just by introducing yourself, by chatting
presenting a clear and complete description rather
along as you’re doing things … with the
than presenting every theme. This means present-
patient. Asking them … questions about
ing very few themes or findings in each report. The
themselves…like ‘how are you feeling
definition and discussion of qualitative analyses is
about being in hospital? How are you feel-
a lengthy process:
ing about the operation tomorrow?’ And
then they’ll sort of give you a clue…and • The definition and discussion of the finding,
actually then tell you how they’re feeling • The presentation of original data,
about things…just general chit chat… • The presentation of nuances within that finding,
(Nurse).’ (Williams & Irurita, 1998, p. 38) and
• The presentation of more original data (to pro-
The sequence used by Williams and Irurita vide evidence of these nuances).
(1998) to describe and elaborate on the data pat-
terns they analyzed is an effective format for Thus, there is often not enough room within
reporting qualitative analyses. This format consists any one written document or oral presentation to
of: communicate more than one or a very few themes
or findings. Thus, qualitative researchers should
• A name or label for the category, keep reports simple so that they can report specific
• The authors’ description of the meaning attached findings in full, following the format of starting
to the category, and with a broad theme and then defining its specific
• A quotation from the raw text or original data to nuances.
elaborate on the meaning of the category and to Regardless of the type of reporting qualitative
show the type of text coded into the category. researchers produce (e.g., poster, oral presentation,
published paper or written report), they must stand
After presenting this sequence of information, ready to explain the qualitative approach and its
the researcher goes on to present the nuances benefits, as many questions about it are always
underneath each thematic category. This includes, raised by quantitative researchers. Thus, when
perhaps, presenting the ways in which particular making a report of qualitative analyses and/or
data sources grouped together under that category responding to audience members’ or readers’ ques-
or theme during analysis. If one found, for exam- tions, one should be ready to discuss the following
ple, that the way in which interviewees talked issues:
about family history varied depending on whether
or not they felt family factors determined a current • Why one chose to do qualitative over quantitative
health condition, one could present first those who research,
felt positively about their family history (defining • The unique benefits of qualitative data, or why
this group and then presenting examples of origi- one might engage in qualitative research,
nal data), and subsequently present those who felt • The rigor of one’s qualitative data collection and
negatively about the same. Then one might specu- analysis strategies and the specific steps one
late about the reasons why there were differences completed to arrive at results, and
in the meanings attached to family histories. • The implications and/or significance of one’s
The presentation of nuances underneath a results—that is, how one’s results can be used to
theme or code illustrates the rigorous process by promote future research, advocacy efforts, or
which the researcher arrived at conclusions about policy efforts, even if one’s analyses are based
the important aspects of their data, and also helps on a limited sample. In other words, why do
shape a deeper understanding of the topic at hand. one’s results matter even if they are not general-
By presenting nuances underneath a theme, one izable and/or based on a random sample and
shows how thorough the analysis has been and numerical data?
Common Ways to Report Qualitative Data: A Basic Guide
Presenting Qualitative Analyses in Poster Form should be confronted with information about basic
results or findings—that is, a presentation of the
Poster presentations can be made easily in soft- broad themes found within the analysis. A reader
ware programs such as PowerPoint or Microsoft of the poster should be presented with the name
Word. Although various styles and formats for and definition of the theme/code and then original
organizing one poster are possible, typically, the data examples to back up the definitions. As read-
poster layout should permit the reader to move ers move to the right-hand side of the poster, they
from left to right as they view the poster, as if he should then see the presentation of the specific
or she were reading a book. Thus, on the left, one nuances within the findings, or ways in which par-
should start with a small introduction of one’s ticular groups of one sample or groups of data
topic/study, and some brief information about the sources allowed for a deeper understanding of that
research methods undertaken. As a reader moves theme/code. The following is an illustration of how
in front of the middle area of the poster, she or he a poster might be organized:
Title of Poster
Author & Affiliation
Introduction Definition of Broad Definition of Specific

(perhaps 1-2 Theme/Code Nuances
paragraphs)
Methods Examples of Original Examples of Original

(perhaps 1-2 Data Data
paragraphs and
charts/graphs)
Some Basic Conclusions
One variation to this basic example is to place one’s presentation. Novice researchers should
the key research question and the main study find- query prior conference participants on facts like
ings and conclusions in the top center of the poster. these to ensure that their presentation “hits the
This can be eye-catching and effective. Another mark” with respect to the audience’s needs.
variation might be undertaken to address particular Typically, 10 to 20 minutes are allotted for an
needs of one’s audience. That is, if this is a poster oral presentation at a scholarly or professional
presented in an academic setting, one might want conference or other public forum, followed by 5 to
to present more details on prior research literature 10 minutes of audience questions. The researcher
on the topic on the left-hand side of the poster, should determine the allocated length of time and
after the introductory section and before the pres- assure their presentation is of appropriate length.
entation of methods. If this poster is for an advo- Oral presentations of qualitative data analyses are
cacy organization’s event or for a group of often quite difficult to complete in a short amount
healthcare providers or policymakers, it might be of time. Introducing the topic and discussing the
less important to provide the results of prior research methods are needed, but you must allo-
research and instead highlight more salient take- cate more time to describe your findings and their
home messages for policy decision-makers. contribution. Moreover, since one often needs to
read examples of original data out loud (or at least
Oral Presentations of Qualitative Analyses summarize what pieces of original data suggest)
for the audience, time is quickly spent. Thus, to
In presenting one’s results orally, one would usu- skillfully complete an oral presentation of qualita-
ally report the same data as in a poster format. The tive analyses, one must prepare for the specific
only variation may be based on one’s audience, time allotted and present few findings. Keeping
and this is very similar to our discussion of poster things simple and presenting as few themes/find-
presentations: academic audiences are accustomed ings as possible is extremely important. One
to hearing more about past literature than policy should expect that in a 10-minute presentation,
audiences or consumer/lay audiences. Thus, the for example, one might be able to present only
researcher should know both the disciplinary back- two or three examples of original data to illustrate
grounds of her or his audience and the level of a broad theme and then may only be able to pres-
interest they might have in the components of ent one or two nuances under that theme.
(continued)
385
Common Ways to Report Qualitative Data: A Basic Guide (continued)
Formal Written Reports of Qualitative • A findings section, detailing one or a few

Analyses for Academic Audiences broad findings and all the specific nuances
found within these findings. For both broad
When completing a thesis dissertation, or a funded and specific sets of findings, numerous exam-
report, or when publishing in an academic forum, ples of original data should be presented and
one can present a fuller picture of the topic under explained in full,
study, the past literature on the topic, a broad • A discussion and/or conclusion section, summa-
description of one’s research methods, and more rizing what was found in the results and the
detail on actual findings. In a formal written implications of the findings. Attention may be
report, there is also considerably more emphasis paid to avenues for future research, especially
on formalizing conclusions and the implications based on particular limitations of the current
of completed analyses. Thus, this type of report study,
is the most complete version of a qualitative • Any tables, charts, graphs, or other appendices
analysis. The standard format for a formal written that make the discussion of methods or findings
report, especially for an academic audience is as clearer to the reader, and
follows: • References to past research on the topic.
• An introduction to the topic or group under The above sections may vary in length,
study, including a discussion of the research depending on the audience for the written report.
purpose and the relevance of the research, If one follows this standard format, there is a
• A literature review or “background” section,
greater chance that one’s qualitative research and
illustrating past research on the topic, including a
discussion of any gaps in this past literature that analytic procedures will be considered credible
might be filled by the current analyses, or believable. This is particularly important when
• A methods section that details the sample, data presenting qualitative analyses to audiences who
collection and processing procedures, analytic value numerical results more than original data,
strategies, and potential biases/limitation of the or people’s voices, or who value explanation over
research, description and exploration.
If one prepares statements or answers in connections (i.e., a reporting strategy). Qualitative

response to these issues before finalizing the reporting then involves sharing information about
report, one can guarantee a better reception of the the research process and about both broad and spe-
qualitative analyses and conclusions. Laypersons cific information found in the research.
both value quantitative research for the autonomy Qualitative analysis and reporting are ongoing
of numbers, while they value qualitative research as well. While one may come to the end of a cycle
for its sense of authenticity of real people. Thus, of analysis or the end of a particular report, there
qualitative researchers need to be prepared to deal are always more data to analyze and more analyses
with these societal views in public forums. to report. The researcher should be ready to con-
tinue analyzing the same data sources over and
over again for more and deeper insights and expla-
Conclusion nations. Qualitative researchers should also be
ready to go back and analyze the very same coded
This chapter illustrated how both qualitative analy- themes and patterns, for there is always more one
sis and reporting of qualitative research findings can discover about previously coded data units.
are multilayered enterprises. Investigators start Each time the researcher starts a particular analy-
with a particular data unit in a particular data sis or reporting cycle, she or he makes choices
source and then connect it to other data units in about what to concentrate on, which, ultimately,
other data sources during a complex, flexible yet means some analysis and reporting is left for later.
rigorous, analytic process. Once connections are This means, then, that the conclusions qualita-
found (i.e., patterns or themes across multiple tive researchers discuss are partial, and always
interviews or observations), investigators think warrant further exploration. Thus, researchers
about what those patterns mean and create a way should always be thinking about how they can
to not only understand, but also to talk about those extend the data they have with just a bit more data
analysis. One should think about ways in which to Aging, disability and ethnicity. In J. Sokolovsky (Ed.),
push the boundaries of interpretation and under- Cultural context of aging: Worldwide perspectives (2nd
ed., pp. 443–451). New York: Bergin & Garvey.
standing to facilitate insights that are needed to Dijkers, M. (1999). Correlates of life satisfaction among
answer the range of study questions. This may persons with spinal cord injury. Archives of Physical
include a search for ways of broadening the ana- Medicine and Rehabilitation, 80(8), 867–876.
lytic strategy but it can also mean finding ways to Dillaway, H. E. (2005). Menopause is the “good old”:
Women’s thoughts about reproductive aging. Gender
be more specific and insightful. Finally, because
& Society, 19(3), 398–417.
data collection and analysis should always be Emerson, R. M., Fretz, R. I., & Shaw, L. (1995). Writing
reflexive and iterative, one should never stop think- ethnographic fieldnotes. Chicago: University of
ing about the next steps that could be taken in data Chicago Press.
collection. Such reflection can lay the groundwork Estroff, S. (1981, 1985). Making it crazy: An ethnography
of psychiatric clients in an American community.
for yet another new phase of the research project. Berkeley, CA: University of California Press.
While this chapter discussed the major steps or Geertz, C. (2001). Thick description: Toward an interpre-
processes that all qualitative researchers complete tive theory of culture. In R. M. Emerson (Ed.),
in order to interpret and report qualitative data, Contemporary field research (2nd ed., pp. 55–75).
Long Grove, IL: Waveland Press.
there are necessarily and inevitably variations in
Gitlin, L., Luborsky, M., & Schemm, R. (1998). Emerging
how each researcher moves through the analysis concerns of older stroke patients about assistive device
and reporting processes. By no means does every- use. The Gerontologist, 38(2), 169–180.
one analyze or report their data in the same way. Glaser, B., & Strauss, A. (1967). The discovery of
Nor should they. As highlighted many times, a grounded theory: Strategies for qualitative research.
Chicago: Aldine Press.
benefit to qualitative research is the flexibility that Glass, T. A., Matchar, D. B., Belyea, M., & Feussner, J. R.
characterizes it. Thus, while this chapter detailed (1993). Impact of social support on outcome in first
particular ways in which qualitative analyses and stroke. Stroke, 24, 64–70.
reports can be completed, they should be seen only Groce, N. (1985). Everyone here spoke sign language:
Hereditary deafness on Martha’s Vineyard. Cambridge,
as a guide. They should not be taken as a prescrip-
MA: Harvard University Press.
tion for how to complete qualitative analysis and Luborsky, M. (1997). Attuning assessment to the client:
reporting. Recent advances in theory and methodology.
Finally, qualitative research is time-consuming Generations, 21(1), 10–16.
and conceptually demanding. The goals of qualita- Lysack, C., & Seipke, H. (2002). Communicating the
occupational self: A qualitative study of oldest-old
tive research—to describe, to explore, to present American women. Scandinavian Journal of Occupa-
people’s voices—can be realized only if indivi- tional Therapy, 9, 130–139.
duals are dedicated to the complexities involved Magaziner, J., Hawkes, W., Hebel, J. R., Zimmerman, S. I.,
in data collection, analysis, and reporting. Resea- Fox, K. M., Dolan, M., Felsenthal, G., & Kenzora, J.
(2000). Recovery from hip fracture in eight areas of
rchers should be mindful of this during the arduous
function. The Journals of Gerontology: Series A.
process of qualitative analysis and reporting. Biological Sciences and Medical Sciences, 55(9),
Qualitative research has contributed under- 498M 507.
standing about the wide range of factors that con- Miles, M. B., & Huberman, A. M. (1994). Qualitative
tribute to individuals’ health and disability, about data analysis: An expanded sourcebook (2nd ed.).
Thousand Oaks, CA: SAGE Publications.
individuals’ use of particular therapeutic treat- Mishler, E. G. (1979). Meaning in context: Is there
ments, and about the processes by which occupa- any other kind? Harvard Educational Review, 49,
tional therapists facilitate health and wellness 1–19.
through their interventions. Consequently, the ini- Neufeld, S., Berry, R., & Sankar, S. (2005). Pathways to
HIV diagnosis among African Americans. Manuscript
tiation of more qualitative research in occupa-
submitted for publication.
tional therapy settings is critical. While there have Riessman, C. K. (1993). Narrative analysis. Thousand
been important qualitative studies in recent years Oaks, CA: SAGE Publications.
in occupational therapy, there is a need for much Rubin, H., & Rubin, I. (2005). Qualitative interviewing:
more description and exploration of both clients’ The art of hearing data (2nd ed.). Thousand Oaks,
CA: SAGE Publications.
conditions, and of both clients’ and therapists’ atti- Sankar, A., Neufeld, S., & Luborsky, M. (2005a). The
tudes and behaviors. meaning of adherence to HIV anti-retroviral medica-
tions. Manuscript submitted for publication.
Sankar, A., Nevedal, D., Neufeld, S., & Luborsky, M.
REFERENCES (2005). What is a missed dose? Implications for con-
Becker, H. S., Geer, B., Hughes, E. C., Strauss, A. L. struct validity and patient adherence. Manuscript sub-
(1961, 1991). Boys in white: Student culture in medical mitted for publication.
school. Somerset, NJ: Transaction. Schwandt, T. A. (1997). Qualitative inquiry: A dictionary
Deppen-Wood, M., Luborsky, M., & Scheer, J. (1997). of terms. Thousand Oaks, CA: SAGE Publications.
Siporin, S., & Lysack, C. (2004). Quality of life and sup- Strauss, A. (1987). Qualitative analysis for social scien-
ported employment: A case study of three women with tists. Cambridge, UK: Cambridge University Press.
developmental disabilities. American Journal of Williams, A. M., & Irurita, I. F. (1998). Therapeutically
Occupational Therapy, 58(4), 455–465. conducive relationships between nurses and patients:
Stack, C. (1974). All our kin: Strategies for survival in a An important component of quality nursing care.
Black community. New York: Harper & Row. Australian Journal of Advanced Nursing, 16(2), 36–44.
S E C T I O N 6
Other Design Approaches
C H A P T E R 2 3
Exploring Perceptions About Services

Using Q Methodology
Susan Corr
Q methodology (also known as operant subjectiv- ice users’1 perspectives of occupational therapy
ity) provides a complementary approach to quan- (Corr, Neill, & Turner, 2005). Both of these stud-
titative and qualitative ies will be discussed
methods. It maintains the later to illustrate the
subjectivity of partici- Q methodology allows indi- processes and use of Q
pants and has a rigorous methodology in service
and objective process of viduals to measure them- evaluation. In addition,
doing so (Smith, 2001). selves rather than being this chapter outlines the
It is a tool for exploring theoretical assumptions
and generating a greater
measured by researchers of the method, the pro-
understanding of peo- (Smith, 2001). cesses involved and
ple’s perspectives and gives an indication of its
beliefs. Q methodology potential use in estab-
uses a unique method of data collection and statis- lishing views relating to occupational therapy
tical analysis techniques (Brown, 1996). It has been services.
used in the United States, and to a lesser extent in
History of Q Methodology
Britain, by a broad range of researchers, including
psychologists, social scientists, educators, and
political scientists. The method is increasingly
William Stephenson first introduced Q methodol-
being used in healthcare studies. The following are
ogy in the 1930s (Stephenson, 1935; Wigger &
some examples:
Mrtek, 1994). He had the “desire to understand
• Exploring stress and coping strategies in commu- what made the individual person unique rather
nity psychiatric nurses (Leary et al., 1995), than what characteristics could be found across
• Understanding reasons for pharmacy students large populations of individuals” (Wigger &
entering this profession (Wigger and Mrtek, Mrtek, 1994, p. 9). In developing both the concepts
1994), and techniques of Q methodology, Stephenson
• Explaining how patients with irritable bowel drew from both quantitative and qualitative
syndrome understand the nature and causality of research traditions. Q methodology allows individ-
their illness (Stenner, Dancey, & Watts, 2000), uals to measure themselves rather than being
• Assessing nurses’ job satisfaction (Chinnis, measured by researchers (Smith, 2001). How indi-
Summers, Doerr, Paulson, & Davis, 2001), and viduals felt about issues was the focus of his con-
• Identifying priorities and barriers to primary cern and the method he developed.
health care in post-conflict Serbia (Nelson et al., One of Stephenson’s students, Steve Brown,
2003). was instrumental in providing a breadth of expert-
ise about Q methodology. His publications, con-
The use of Q methodology by occupational ference presentations, email discussion list, and
therapists is just developing. For example, one seminars (see Resources section) have increased
study used Q methodology as an aspect of an eval- awareness of Q methodology.
uation of a day service for adults post-stroke (Corr, 1
Service user is the preferred term for clients or consumers
Phillips, & Capdevila, 2003). Another used the of occupational therapy in the United Kingdom where this
method as part of an investigation of former serv- study was conducted.
390 Section 6 Other Design Approaches
Theoretical Bases and 23.1 indicates the differences between these for
ease of comparison. Although Q methodology like
Assumptions many qualitative approaches, aims to explore sub-
jective phenomena from the individual’s own per-
Q methodology is based on Stephenson’s (1978) spective, a significant difference is that in Q
view of subjectivity and his belief that an objective methodology individuals are given specific state-
account of this viewpoint can be obtained through ments to which they are asked to respond
an operant process. It draws on quantitative and (Cordingley et al., 1997). In some cases these state-
qualitative approaches for its data collection, ments may originate from qualitative approaches
analysis, and interpreta- such as semi- or unstruc-
tion processes. tured interviews. A sec-
The key to Q methodology is ond difference is the fact
that in Q methodology,
Subjectivity that the self is central to all the viewpoints (factors)
The key to Q methodol- else (Brown 1972). that emerge from the data
ogy is that the self is cen- are derived statistically
tral to all else (Brown, (Cordingley et al., 1997).
1972). Stephenson con- This process of data
sidered subjectivity to be analysis is more akin to
a point of view that the Through Q methodology, an that of quantitative stud-
individuals were willing expression of feeling or opin- ies and therefore limits
to say, either to them- researcher bias. The
selves or to others ion is identified through con- similarity with qualita-
(Smith, 2001). He argued crete behavior. tive approaches emerges
that since the individual again when considering
holds the subjective the interpretation of the
viewpoint, he or she must be the center of the analyzed data. The researcher needs to describe and
investigation (Brown, 1972). interpret the emerging viewpoints. Although Q
methodology allows each individual to reflect his
Operant Subjectivity or her own viewpoint through how he or she
Q methodology establishes participants’ view- arranges the statements, the factor analysis process
points through individuals undertaking an operant allows researchers to view clusters or patterns of
procedure (behavior) (i.e., sorting statements that responses to uncover perceptions even within small
relate to the concept under investigation. Stephen- groups (Donner, 2001; Robinson, Popovich,
son considered this process to produce an expres- Gustafson, & Fraser, 2003). Researchers may
sion of the participants’ subjectivity (Wigger & describe typical patterns of perceptions rather than
Mrtek, 1994). According to Brown (1980), behav- average perceptions and will name or label distinct
ioral and social scientists have assumed that feel- points of view and provide some interpretation of
ings, preferences, and thoughts are traits that can these (Robinson et al., 2003).
be measured only indirectly by tests and scales.
However, through Q methodology, an expression Steps to Conducting Q Methodology
of feeling or opinion is identified through concrete Q methodology has the following four distinct
behavior. This process can be measured and stud- phases, each of which is outlined in this chapter:
ied in a scientific manner, allowing a construct to
develop (Smith, 2001). Thus, participants’ view- • Developing the Q-sort pack,
points lead to theoretical interpretations of the • Administering the Q-sort,
emerging constructs (Cordingley, Webb, & Hillier, • Factor analyzing the data, and
1997). Brown (1980) acknowledges that the sub- • Interpreting factors.
jective opinions identified in a Q methodology
study are not provable, but they can be shown to Developing a Q-Sort Pack
have structure and form.
The first, and according to Donner (2001) the most
Comparison with Qualitative challenging, step in Q methodology is the devel-
opment of the Q-sort pack. As in traditional survey
and Quantitative Methods techniques, discussed in Chapter 8, this early step
As previously noted, Q methodology draws on involves creating items or statements that will be
both qualitative and quantitative approaches. Table used to examine the topic of investigation (Chinnis
Chapter 23 Exploring Perceptions About Services Using Q Methodology 391
items. Therefore, statements need to “be plausible

Q methodology can be used to:
competitors with one another, such that some par-
• Make discoveries but not predictions. ticipants may be attracted to them and others dis-
• Examine emerging factors of the phenomenon inclined to choose them” (Donner, 2001, p. 27).
of interest. The statements should also be similar in style (e.g.,
• Examine life as lived from the standpoint of the
either as full sentences or just phrases). These
person living it.
• Identify characteristics of individuals who share statements should be clear and without double
common viewpoints. negatives.
• Measure arrays of attitudes at a certain point in
time. Two Case Examples
• Measure attitudinal changes over time.
• Study subjective preferences, sentiments, In a study conducted by Corr et al. (2003) to estab-
ideals, the nature of beauty, tastes. lish perceptions of benefit from a day service for
• Understand individuals as complex, holistic young adults with stroke, group interviews were
beings. carried out to generate statements of perceived
• Discover unexpected characteristics. benefits of the service. Groups consisted of some
• Discover what was previously unknown as of the users of the service, their caregivers, volun-
opposed to testing a theory. teers, fund holders, and those who referred clients
(Brown, 1972, 1996; Cordingley et al., 1997; to the service. The literature, especially that relat-
Dennis, 1986; Mrtek et al. 1996; Smith, 2001; ing to the provision of day services, was also
Wigger & Mrtek, 1994). explored to generate statements. Finally the ser-
vice’s own documentation that outlines its aims
was considered and used as a source of statements.
et al., 2001). The Q-sort pack ideally consists of An advantage of creating statements in this man-
between 40 and 80 statements related to the ner is that the preconceived biases of the
research topic (Dennis, researcher can be mini-
1986). These statements mized, since the state-
can be generated from The factor analysis process ments come from the
several sources including participants themselves
preliminary interviews
allows researchers to view (Chinnis et al., 2001).
with potential partici- clusters or patterns of The statements were
pants, relevant literature then checked for dupli-
or any sources that pro-
responses to uncover cation to eliminate over-
vide information, and perceptions. lap. Finally statements
opinions connected to were examined to make
the study (Brown, 1996; sure they focused on the
Dennis, 1986; Wigger and Mrtek, 1994). intended topic. For instance, those describing the
According to Barbosa, Willoughby, Rosenberg, process of the service as opposed to the benefits
and Mrtek (1998), there should be enough state- were removed.
ments with a variety of opinions to cater to the The second study aimed to explore former serv-
widely different subjective feelings the partici- ice users’ views of occupational therapy (Corr,
pants may have about the research topic. Donner Neill, & Turner, 2005). Creek (2003) originally
(2001) suggests that the statements can be conducted an ethnographic study to develop a con-
assumed to be simply a subset of the possible conceptual framework about the nature and purpose of
cepts that may be important to the issue at hand. occupational therapy. The document she produced,
The statements do not have to be lengthy “Occupational therapy as a complex intervention”
(Donner, 2001). Pictures or images, simple (Creek, 2003), was used to generate the statements
phrases, or single words can be used and may be for this study. In total 32 statements were gener-
more appropriate for certain populations such as ated and an example of these are presented in
children. Donner (2001) suggests that to create a Figure 23.1.
good set of statements researchers should try to
choose statements that have distinct meanings and
Administrating a Q Sort
are not merely repetitive, overlapping, or the exact
inverse of each other. He also recommends avoid- The Q-sort pack consists of a pack of cards with
ing statements that are so extreme that all partici- each statement on a separate card. Participants are
pants could be expected to either agree or disagree asked to sort the various statements according to a
with them, to the exclusion of prioritizing other specific condition of instruction given by the
Table 23.1 Comparison of Q Methodology with Qualitative and Quantitative Methods

Issue Quantitative Methods Q Methodology Qualitative Methods
Purpose Identify changes or Identify ranges of viewpoints. Explore individual
characteristics in large experiences.
populations.
Constructs being Researcher identifies Researcher provides the broad Constructs are usually
explored these at the outset. framework; the process unknown and they
enables the constructs to emerge through the
emerge. research process.
Population Large populations are The population needs to rep- A small population is
usually required. resent those who have a used, and may even
view on the topic in ques- just be one individual.
tions (all stakeholders).
Between 20 and 40 are
common population sizes.
Tools Measurable items such Statements (or similar Free-flow conversation
as interval score and materials) that reflect the via interviews.
objective measure- research topic.
ment scales.
Data collection Objective measuring and Statements are sorted by Interviews are using
recording of data individuals using an ordinal unstructured or
usually in a numerical scale. semistructured.
format.
Analysis Uses descriptive and Uses a statistical process Content or discourse
inferential statistics to (factor analysis) to identity analysis used seeking
look for trends and range of viewpoints and meaning units and
comparisons. differences and similarities themes.
of views.
Interpretation Reports results as The researcher tells the story The researcher interprets
statistical findings of the emerging viewpoints the data but may use
indicating levels of and explains differences techniques such as
significance. and similarities. member checking to
aid this.
researcher (Cordingley et al., 1997). The most gory. For example, referring to Figure 23.2, the
usual conditions involve sorting according to the participant would place two statements in the
degree of agreement they place on each of the extreme “most disagree” box, and the extreme
statements, from “most disagree” through “neu- “most agree” box and six statements in the neutral
tral” to “most agree” (Brown, 1996; Dennis, 1986; box and so on. This “flattened” normal distribution
Leary et al., 1995). It can also be “most unlike” to curve, that is, a curve that allows more points on
“most like” or other such conditions (Smith, the extremes, is usually used in Q methodology
2001). Stainton Rogers (1995) suggests that it is (Dennis, 1986).
easier to start by separating the statements under In using this grid, important statements, with
relevant headings such as “disagree,” “neutral,” which the participant has strongly agreed and
and “agree” before sorting them more discrimi- strongly disagreed, should first be placed at the
nately along the continuum. Prior to administering extreme ends of the distribution pattern. State-
the Q-sort pack, the researcher needs to develop a ments that hold less salience for the participant
sorting grid (i.e., a grid layout to indicate the shape would be found near the midpoint of the grid
of the distribution). There are usually an odd num- (Mrtek, Tafesse, & Wigger, 1996). Participants are
ber of column values in order to allow for a middle therefore forced into prioritizing which statements
neutral column (Donner, 2001). Figure 23.2 shows they wish to use to express their strongest attitudes
the grid for the Corr et al. (2005) study, which used or feelings (Cordingley et al., 1997). Even though
32 statements. From this grid it will be clear how this approach does force the distribution, the num-
many statements should be placed in each cate- ber of ways in which a sample can be sorted is still
1. It provides an opportunity for me to retrain for work.
2. The occupational therapist provides group activities for me to participate in.
3. Helps me to cope in my community.
4. I have to work in partnership with my occupational therapist.
5. During treatment the activities are adjusted so that they remain challenging and interesting.
6. Enables me to remain independent.
7. Teaches me new skills and helps me to relearn old ones.
8. Looks at all areas of need from looking after myself to my hobbies and my employment.
9. Provides adaptations to my home.
10. Occupational therapists offer support to my caregivers as well as to me.

Figure 23.1 Example of the statements. [Reprinted with permission from Corr, S., Neill, G.,
& Turner, A. (2005). Comparing occupational therapy definition and consumers’ experi-
ences: A Q-methodology study. British Journal of Occupational Therapy, 68(8), 338-346.]
huge. All statements can be moved about and study’s aim and question, rather than on the basis
exchanged until the participant is completely satis- of their representativeness of a larger population
fied with his or her choices, as there are no right or (Chinnis et al., 2001; Cordingley et al., 1997).
wrong answers (Brown, 1980; Stainton Rogers, Since the researcher aims to establish a variety of
1995). perspectives, a key issue in choosing participants
The sample size for Q methodology can be is to ensure that diverse selections of viewpoints
small, as the study concerns the viewpoints of par- are included (Cordingley et al., 1997).
ticipants rather than the character traits of a spe- Sampling can be illustrated by referring back to
cific population that the participants are chosen to the two studies previously discussed. In the first
represent (Mrtek et al., 1996). Therefore, as in instance, 37 participants took part in the day serv-
qualitative research, individuals are chosen to par- ice evaluation study (Corr et al., 2003). Nineteen
ticipate in a study based on their relevance to the had used the service while the remaining 18 were
Most disagree Most agree
-4 -3 -2 -1 0 +1 +2 +3 +4
(2) (2)
(3) (3)
(4) (4) (4) (4)
(6)
Figure 23.2 Sorting grid. [Reprinted with permission from Corr, S.

Neill, G., & Turner, A. (2005). Comparing occupational therapy defini-
tion and consumers’ experiences: A Q-methodology study. British
Journal of Occupational Therapy, 68(8), 338-346.]
other stakeholders of the service such as caregivers World Wide Web (see Resources section). The
of users, volunteers, and fund-holders. In the sec- rank-ordering pattern (Q-sort) of each participant
ond instance, 16 former is entered into the statis-
users were selected to tical package. It is then
give their views on occu- In this methodology it is the factor analyzed for its
pational therapy (Corr et meaning using principal
al., 2005). viewpoints that are important component analysis and
The Q-sort pack can rather than the number of varimax rotation (Mrtek
be administered in group et al., 1996). Chapter 17
or individual settings.
participants who held those provides a discussion of
The participants require viewpoints. factor analysis. The goal
enough space to sort the of factor analysis is to
statements, visually view find the underlying fac-
the entire sort (to make adjustments), and then tors that summarize the pattern of correlations
record their answers without disturbing the sort among the Q-sorts undertaken by the participants
(Donner, 2001). It takes on average 20 or 25 min- (Cordingley et al., 1997).
utes for a sort of 30 statements. In addition, partic- The first stage of the analysis involves correlat-
ipants can be invited to add any comments they ing each person’s Q-sort with all the other Q-sorts,
wish regarding the statements and sorting process. resulting in a matrix of correlation coefficients
These should be recorded by the researcher and (Cordingley et al., 1997). The correlation coeffi-
noted as complementary data for use in the inter- cients indicate the extent to which pairs of Q-sorts
pretation of the factors. resemble or are different from each other. If all
participants sort the statements similarly, there will
be a high correlation coefficient and only one fac-
Analyzing the Data
tor (viewpoint) will be identified.
Analysis of the data gathered by Q methodology Factor analysis searches for groups of Q-sorts
can be carried out by a dedicated Q package such which, on the basis of their correlations, appear to
as PQMethod, which can be downloaded from the go together as a view or factor. Participants are
Figure 23.3 A study participant (right) decides how much she

agrees or disagrees with statements about occupational therapy
while Dr. Corr (left) notes any additional comments made and
final positions of the statements.
grouped on the same factor when their Q-sorts are ing to Stephenson (1983) results in meaning or
similar, that is, they have sorted the same state- explanation and understanding of the factors. “Q
ments that they most agree with and most disagree factors point to the necessity for insights, hunches
with (Dennis, 1986). Principal component analysis and guesses supported subsequently by facts”
is used to provide eigenvalues for each factor (Stephenson, 1998, p. 73).
(eigenvalues are an indication of the proportion of As shown in the feature box below, six factors
variance explained by each factor) (Kline, 1994). emerged in the study of views of occupational
This information contributes to the decision mak- therapy (Corr et al., 2005). The ‘story’ for each fac-
ing on the number of factors to identify. Factors tor was generated noting the placement of the
with an eigenvalue greater than one explain more “most agree,” and “most disagree” statements and
variance and therefore the maximum number of additional post-sort comments made by the partic-
factors taken to the next stage (rotation) usually ipants. For example, it can be seen from Figure
corresponds to the number of initial factors with 23.3 that the “story” for this factor emerges from
eigenvalues greater than one (Donner, 2001). statements 20, 32, 28, 31, and 18 as well as 9, 13,
Varimax rotation clarifies the structure of the fac- 22, 1, and 2. This factor placed emphasis on occu-
tors by maximizing the variance between each of pational therapy as being beneficial for improving
the factors (Donner, 2001). It allows the researcher individuals’ sense of self and adjusting to their dis-
to identify those sorts that load cleanly on a single ability and not about providing adaptations or
factor. assisting with returning to work. However, placing
Usually, more than one but less than seven fac- statement 22 in the –3 column raised questions
tors (viewpoints) are identified in Q methodology about the consistency of the participants’ experi-
studies (Dennis, 1986). In this methodology, it is ences, as on the one hand they feel supported in
the viewpoints that are important rather than the adjusting to life after stroke while on the other they
number of participants who held those viewpoints. suggested that they are not supported in dealing
Once the factors are identified, Q-sorts are con- with the emotional and psychological aspects of
structed in the arrangement of these views to facil- their stroke. The study found similarities in what
itate interpretation. Figure 23.4 is the reconstructed the participants perceived as benefits of occupa-
sort of one factor for the study conducted by Corr tional therapy based on their experiences, com-
et al. (2005). The researcher is then in a position to pared to the professional view as noted in the
establish statistically how each individual’s Q-sort definition document by Creek (2003). The study
relates (or loads) on to these identified factors noted the importance of the individual within
(Cordingley et al., 1997). occupational therapy intervention, as a unique
individual, for whom occupational therapy assists
Interpreting Findings to retain his or her sense of self, and enables him
or her to adjust to disability and to participate in
The final stage of a Q methodological study is to society.
interpret the factors that have emerged. This is
achieved by studying the analysis output and by
looking at the reconstructed sorts representing each Potential Application in
factor (Chinnis et al., 2001). The analysis output, as
well as presenting the sorts for each factor, also Occupational Therapy
allows comparisons to be made between the posi-
tioning of statements in each factor. It highlights Q methodology has the potential for use by occu-
contentious and consensus statements so that both pational therapists as “the range of topics which
differences and similarities can be noted. can be studied using this technique is almost
At this stage often the factors are given labels
that best describe the patterns of statements in the
given factor (Corr, 2001). The researcher may Factors identified as views of service of occupa-
use the additional interview data collected when tional therapy services:
the individual sorted the statements, or he or she Factor 1: Improved sense of self
may go back to the participants with the descrip- Factor 2: The importance of being heard
tions of the factors to confirm the interpretation Factor 3: Practical assistance
(Cordingley et al., 1997). It is up to the researcher Factor 4: Maintaining autonomy
to infer the meaning of the factors (Donner, 2001). Factor 5: Desire for involvement
Factor 6: General benefits
However, theory, previous research, and cultural
knowledge may aid interpretation (Stainton (Corr et al., 2005)
Rogers, 1995). Interpretation of the factors accord-
23Kielhofner(F)-23
Editor's review
Author's review
396
(if needed)
Initials
19. I have to play
an active role in
x
my treatment.
1525_23_03.eps
1/4/05
5/5/06
i i
ISBN #
23. I am able to
1525
Document name
k if
freely express
Date
Ch
my feelings.
3:57 PM
29. Enables me
Kate Margeson
25. Occupational 7. Teaches me 21. Gives me a 6. Enables me
therapists help new skills and to pursue my sense of to remain
me to sort out my helps me relearn leisure activities achievement. independent.
F23_3
Kielhofner
finanaces and old ones. and hobbies.
Page 396
benefits.
Author
Fig. #
Artist
22. Helps me to 4. I have to work 15. Provides 24. I am listened 14. I am able to 17. Helps me feel 28. Improves my
deal with the in partnership equipment to to rather than told choose the goals more able to quality of life.
emotional and with my enable to remain what to do. of my treatment. participate in
psychological occupational independent. social activities.
effects of my therapist.
stroke.
9. Provides 1. It provides an 16. I am able to 11. I am regularly 10. Occupational 12. The 30. Uses 31. It helps me to 20. I am treated
adaptations to opportunity for choose activities asked how I feel therapists offer information given activities to help adjust to my as an individual
my home. me to retrain for that are about my support to my to me is easy to me overcome my disabilities. rather than as
work. important and treatment. carers as well as understand and problems. "just another
meaningful to to me. helps me make patient."
me. decisions.
13. Carers are 2. The 8. Looks at all 3. Helps me to 27. Occupational 5. During 26. Occupational 18. Helps me to 32.
involved in my occupational areas of need cope in my therapy focuses treatment the therapy treatment gain a sense of Occupational
treatment. therapist from looking after community. on activities that activities are helps me to carry identity. therapy is
provides group myself to my help me look after adjusted so that out everyday concerned with
activities for me hobbies and myself, i.e., they remain activities in my how my stroke
to participate in. employment. washing and challenging and home. has affected
dressing. interesting. me physically.
-4 -3 -2 -1 0 +1 +2 +3 +4
Least beneficial Neutral Most beneficial
Figure 23.4 Factor 1 Improved sense of self. [Reprinted with permission form Corr, S., Neill, G., & Turner, A. (2005). Comparing occupational therapy
definition and consumers’ experiences: A Q-methodology study. British Journal of Occupational Therapy, 68(8), 338-346.]
unlimited” (Stainton Rogers, 1995, p. 180).

Wigger and Mrtek (1994) believe that whenever Factors Identified as Views of Users of Stroke
subjective matters are at the center of the research Day Service
question, Q methodology can be used. It would Factor 1: New experiences
help identify attitudes to illness and disability from Factor 2: Feeling valued
a client’s perspective and has already been used to Factor 3: Social recovery
attempt to explain health and illness (Stainton Factor 4: Security
Rogers, 1991). Factor 5: Prevents isolation
So far, occupational therapists have used Q Factor 6: General recovery
methodology mainly in service evaluation. Service
evaluation is considered key in providing effective Factors Identified as Views of Nonusers
and efficient services (Salmon, 2003) while Factor 1: Psychological gains
Murphy, Dingwall, Greatbatch, Parker, and Factor 2: Social confidence
Watson (1998) consider it to be “an essential part Factor 3: Encourages communication
of any intervention or program of action in health” Factor 4: Respite for carers
(p. 215). Q methodology enables all relevant stake- Factor 5: Sense of purpose
holders to participate in service evaluation.
(Corr et al., 2003)
However, historically it is the service users who
have often been excluded from contributing to the
development and change of services.
The aim of the day service evaluated by Corr et benefits, what changes occurred, and the aim of the
al. (2003) was to offer individuals between 18 and service. These findings allowed the stakeholders to
55 years of age who had a stroke the opportunity review the service aims and therefore clarify
to identify and pursue meaningful and realistic whether the perceived benefits match the aim.
opportunities within the community. The purpose They also enable the stakeholders to establish if
of the evaluation was to establish any benefits from new aims need to be set and to understand the var-
attending the service including any perceived ben- ious perceptions of benefits.
efits to inform future planning and provision with The factors and defining statements for the fac-
respect to long-term care post-stroke (Corr et al., tors in a service evaluation, such as the example
2003; Corr, Phillips, & Walker, 2004). In addition above, could be used to create a routine evaluation
to the Q methodology study, a randomized tool such as a Likert scale or questionnaire. The Q
crossover design study was used to measure methodology study identified the relevant concepts
change objectively following attendance at the through small-sample analysis. However, this can
service for 6 months (Corr et al., 2004). The find- be useful in the development of questionnaires for
ings suggest that attending the service increased administration to larger samples (Brown, 2002).
occupational performance and satisfaction with This previous example indicates the use of Q
performance. However, there was no evidence that methodology in establishing perceptions of benefit
depression and anxiety were reduced, that quality to services. Q methodology could also be used to
of life and self-concept were improved, or that identify clients’ attitudes both at the beginning and
there was increased participation in community end of intervention. An additional use is as a single
activities. Many unmet needs were also identified. case study, in which the individual may be asked to
The Q methodology study generated a range of sort the statements under a range of conditions of
viewpoints with eleven factors identified, six from instruction (Smith, 2001). For example, an indi-
the service users and five from the nonusers (see vidual may be asked to sort according to how he
feature box above). These factors suggested that felt when he was first referred to occupational ther-
the service provided new experiences, enabled apy, how he feels now, and how he anticipates he
individuals to feel valued, aided social recovery, will feel in the future.
provided security, prevented isolation, contributed
to psychological gains, enhanced social confi-
dence, encouraged communication, provided Strengths and Limitations
respite for caregivers, and engendered a sense of
purpose.
Strengths
The findings from both aspects of the evalua-
tion suggested that the service provided some sup- There are a number of benefits to using Q method-
port to the individuals post-stroke although there ology. Most importantly, it allows people to
were differences between what were perceived as express their own views (Corr, 2001). Turner
(2002) suggests that one element of the philosophy quantify and analyze such information and allows
of occupational therapy is that “people are indi- differences as well as similarities between view-
viduals of worth and inherently different from points to be easily identified (Cordingley et al.,
each other” (p. 5). If used with clients, occupa- 1997; Mrtek et al., 1996).
tional therapy researchers could establish the dif- Another more recent advantage related to Q
ferent views that individuals hold. Q methodology methodology is the availability of free computer
provides the opportunity for service users to have software such as PQMethod. This can perform the
an active role in service evaluation, which is complex calculations associated with factor analy-
important (Martin, 1986). sis and speedily identify the factors.
Sorting the Q pack is a novel administration
method that requires the active participation of the
Limitations
research participant. This, although not unique, is
unusual in the research process. As a result of this Q methodology has several limitations, related
active participation, it is rare to have missing data mainly to the process involved. Time is required for
and undecided responses (Dennis, 1986). Donner each participant to sort the statements; this amount
(2001) has noted that participants want to see their of time can add up depending on the number of
opinions translated into factors and quantified. participants in the study. Also, explaining the
Also the ranking of the statements during the process to the participants can be time-consuming,
Q-sort requires participants to make fine discrim- as the instructions are comprehensive and partici-
ina tions they otherwise might not make (Dennis, pants may need to be shown how to proceed
1986). In other research methods, such as Likert (Dennis, 1986). It is necessary to invest this time if
scales, participants are asked to indicate their lev- participants are to represent their perspectives
els of agreement on a range of statements. accurately and adequately. In the pilot for the day
However, the advantage of Q methodology over service study, those users of the service who had
one such as Likert is that participants have to iden- comprehension problems had some difficulties
tify their level of agreement with a statement in sorting the Q pack. Also, all service users took far
relation to all the other statements (Donner, 2001). longer than caregivers and volunteers and appeared
It also allows researchers to clarify the range of to tire during the process. As a result, a smaller (33
constructs present in viewpoints on the research statements) separate Q-sort pack and grid was
topic (Barbosa et al., 1998). developed for the service users to complete.
In Q methodology the content validity of each A second limitation is that it is not designed to
of the statements is derived from the rank order in show how many people in a study population have
which they are placed and the vicinity to other a specific viewpoint (Mrtek et al., 1996). Also, as
statements as determined by the participant Barbosa et al. (1998) point out, “no claim is made
(Wigger & Mrtek, 1994). The face validity of Q that other viewpoints do not exist in the broader
methodology relates to the degree of satisfaction a population” (p. 1039). In essence, Q methodology
participant feels about how accurately his or her cannot answer such questions so alternative meth-
ranking of the statements reflects his or her per- ods should be considered. It is worth bearing in
sonal feelings (Barbosa et al., 1998). Dennis mind that a Q methodology study can provide the
(1986) suggests that the data tend to be highly reli- concepts for a larger questionnaire study that can
able and this is supported by the studies of allow inferences concerning the population to be
Fairweather (1981), who found test–retest correla- made.
tions greater than .90 for short intervals and Validity can be compromised if the participants
Kerlinger (1972), who found correlations of .81 did not comprehend the Q-sort task, leading to a
for an 11-month period. misrepresentation of his or her views (Dennis,
Another positive aspect of Q methodology is 1986). Alternatively, there is the potential for par-
that only a small number of participants are needed ticipants to make mechanical rather than concep-
(Mrtek et al., 1996). As few as a dozen participants tual choices to complete the process, particularly
may be used, while it is quite rare to have more if they find the process tiring, which may be the
than 100 participants in a study (Donner, 2001). Q case with some client groups (Dennis, 1986).
methodology also reveals how many different Participants have identified the forced nature of the
viewpoints are present among the group of partic- sorting process as a difficulty (Chinnis et al.,
ipants (Mrtek et al., 1996). It is like other qualita- 2001). However, it ensures that participants sys-
tive methods in that it generates qualitative data. tematically think about all of the statements in
However, it is different in that it provides a way to relation to each other and makes them identify the
most strongly felt issues. It is important that the Kline, P. (1994). An easy guide to factor analysis. London:
researcher clearly explains the instructions. Routledge.
Leary, J., Gallagher, T., Carson, J., Fagin, L., Bartlett,
Another limitation is the fact that the meanings H., & Brown, D. (1995). Stress and coping strategies
are given to factors by the researchers and therein community psychiatric nurses: A Q-methodological
fore could be influenced by researcher bias study. Journal of Advanced Nursing, 21, 230–237.
(Barbosa et al., 1998). However, the analysis pro- Martin, E. (1986). Consumer evaluation of human ser-
vices. Social Policy and Administration, 20(3),
cess aids objective reporting by indicating which
185–200.
statements define the factors. Mrtek, R., Tafesse, E., & Wigger, U. (1996). Q-
methodology and subjective research. Journal
of Social and Administrative Pharmacy, 13(2),
REFERENCES 54–64.
Barbosa, J., Willoughby, P., Rosenberg, C., & Mrtek, R. Murphy, E., Dingwall, R., Greatbatch, D., Parker, S.,
(1998). Statistical Methodology: VII. Q-Methodology, & Watson, P. (1998). Qualitative research methods in
a structural analytic approach to medical subjectivity. health technology assessment: A review of the litera-
Academic Emergency Medicine, 5, 1032–1040. ture. Health Technology Assessment, 2(16).
Brown, S. (1972). A fundamental incommensurability Nelson, B., Simic, S., Beste, L., Vukovic, D., Bjegovic, V.,
between objectivity and subjectivity. In S. Brown & D. & Van Rooyen, M. (2003). Multimodal assessment of
Brenner (Eds.), Science, psychology and communica- the primary healthcare system of Serbia: A model for
tion (pp. 57–94). New York: Teachers College Press. evaluating post-conflict health systems. Prehospital and
Brown, S. (1980). Political subjectivity. Applications of Disaster Medicine, 18(1), 6–13.
Q methodology in political science. New Haven: Yale Robinson, T., Popovich, M., Gustafson, R., & Fraser, C.
University Press. (2003). Older adults’ perceptions of offensive senior
Brown, S. (1996). Q methodology and qualitative research. stereotypes in magazine advertisements: Results of a
Qualitative Health Research, 6(4), 561–567. Q method analysis. Educational Gerontology, 29,
Brown, S. (2002). Q technique and questionnaires. Operant 503–519.
Subjectivity, 25, 117–126. Salmon, N. (2003). Service evaluation and the service user:
Chinnis, A., Summers, D., Doerr, C., Paulson, D., & Davis, A pluralistic solution. British Journal of Occupational
S. (2001). Q methodology. A new way of assessing Therapy, 66(7), 311–317.
employee satisfaction. Journal of Nursing Smith, N. (2001). Current systems in psychology: History,
Administration, 31(5), 252–259. theory, research and application. Belmont CA:
Cordingley, L., Webb, C., & Hillier, V. (1997). Q method- Wadsworth/Thomson Learning.
ology. Qualitative data analysis. Nurse Researcher, Stainton Rogers, W. (1991). Exploring health and illness.
4(3), 31–45. An exploration of diversity. New York: Harvester
Corr, S. (2001). An introduction to Q methodology, a Wheatsheaf.
research technique. British Journal of Occupational Stainton Rogers, R. (1995). Q methodology. In J. Smith &
Therapy, 64(6), 293–297. R. Harre & L. VanLangenhove (Eds.), Rethinking meth-
Corr, S., Neill, G., & Turner, A. (2005). Comparing occu- ods in psychology (pp. 178–207). Newbury Park, CA:
pational therapy definition and consumers experiences: SAGE Publications.
A Q-methodology study. British Journal of Stenner, P., Dancey, C., & Watts, S. (2000). The under-
Occupational Therapy. 68(8), 338–346. standing of their illness amongst people with irritable
Corr, S., Phillips, C., & Capdevila, R. (2003). Using Q bowel syndrome: A Q methodological study. Social
methodology to evaluate a day service for younger adult Science and Medicine, 51, 439–452.
stroke survivors. Operant Subjectivity, 27(1), 1–23. Stephenson, W. (1935). Technique of factor analysis.
Corr, S., Phillips, C., & Walker, M. (2004). Evaluation of a Nature, 136, 297.
pilot service designed to provide support following Stephenson, W. (1978). Concourse theory of communica-
stroke: A randomized cross-over design study. Clinical tion. Communication, 3, 21–20.
Rehabilitation, 18, 69–75. Stephenson, W. (1983). After interpretation. Operant
Creek, J. (2003). Occupational therapy defined as a com- Subjectivity, 6(3), 73–103.
plex intervention. London: College of Occupational Stephenson, W. (1998). Quantum theory media research:
Therapy. II. Intentionality and acculturation. Operant
Dennis, K. (1986). Q methodology: Relevance and applica- Subjectivity, 21, 73–91.
tion to nursing research. Advances in Nursing Science, Turner, A. (2002). History and philosophy of occupational
8(3), 6–17. therapy. In A. Turner & M. Foster & S. Johnson (Eds.),
Donner, J. (2001). Using Q-sorts in participatory Occupational therapy and physical dysfunction: princi-
processes: an introduction to the methodology. In R. ples, skills and practice (5th ed., pp. 3–24). Edinburgh:
Krueger, M. Casey, J. Donner, S. Kirsch, & J. Maack Churchill Livingstone.
(Eds.), Social analysis. Selected tools and techniques Wigger, U., & Mrtek, R. (1994). Use of Q-technique to
(vol. 36, pp. 24–49). Washington: World Bank. examine attitudes of entering pharmacy students toward
Fairweather, J. (1981). Reliability and validity of Q- their profession. American Journal of Pharmaceutical
method results: Some empirical evidence. Operant Education, 58, 8–15.
Subjectivity, 5, 2–16.
Kerlinger, F. (1972). Q methodology in behavioral
research. In S. Brown & D. Brenner (Eds.), Science,
RESOURCES
psychology and communication (pp. 3–38). New York: The following are a range of resources available to anyone
Teachers College Press. interested in Q methodology.
Web-Based Material Subjectivity. Details of subscribing are on the Q

There is a Web site for Q methodology that contains a method Web site: http://www.qmethod.org
breadth of useful information for those interested in Conferences
this method. It can be accessed at: http://www. The International Society for the Scientific Study of
qmethod.org Subjectivity holds an annual meeting where research
An e-mail discussion list is an active discussion forum reg- and theoretical papers are presented. For details check
ularly used by experts and novices alike to share ideas, the Q methodology Web site: http://www.qmethod.org
recent publications, and solutions to problems. To join
the list, send the command “subscribe Q-method Textbooks
⬍your name⬎” (without quotations or brackets) to A range of Q references are now available including those
Listserv@listerv.kent.edu in this chapter’s reference list. Two textbooks that are
useful “manuals” are:
Computer Software Brown, S. R. (1980) Political subjectivity: Applications of
PQMethod is a factor analytic program, which is freely Q methodology in political science. New Haven: Yale
available on the Internet and is in the public domain. It University Press.
was developed by Peter Schmolck, and is a software This somewhat weighty text covers all the steps and princi-
program specifically designed for Q methodology stud- ples in undertaking a Q methodology study with exam-
ies. This software can be downloaded for free at the ples drawn from political science.
following Web site: http://www.12.unibw-muenchen. McKeown, B. F., & Thomas, D. B. (1988). Q methodology.
de/p41bsmk/qmethod/ Newbury Park, CA: SAGE Publications.
Journals This is a slim paperback that is user-friendly and contains
Operant Subjectivity is the quarterly journal of the the essentials of Q methodology.
International Society for the Scientific Study of
24Keilhofner(F)-24 5/5/06 3:58 PM Page 401
C H A P T E R 2 4
The Nature and Use of Consensus

Methodology in Practice
Edward A. S. Duncan
Consensus methodology is the use of a structured their research (Hammersley, 1992; Kirk & Miller,
approach to arrive at a single statement, or set of 1986). Rather, as Murphy, Dingwall, Greatbatch,
statements, that all participants accept; or to iden- Parker, and Watson (1998a) argue, “…the objec-
tify any central ten- tive should be the search
dency and spread of for knowledge about
opinion regarding an While the findings of consen- which we can be reason-
issue (Murphy et al., sus studies are, to a certain ably confident. Such con-
1998). Consensus meth- fidence will be based
ods can make important extent, constructions of par- upon judgments about the
contributions to the evi- ticipants’ own experiences, credibility and plausibil-
dence base of occupa- ity of knowledge claims”
tional therapy and are
these representations can be (p. 69). While the find-
increasingly used to valid and relevant to the ings of consensus studies
assist in the develop- population being studied. are, to a certain extent,
ment of clinical guide- constructions of partici-
lines for practice. This pants’ own experiences,
chapter provides an overview of consensus meth- these representations can be valid and relevant to
ods and addresses what is important to consider the population being studied.
when selecting and using a consensus method- Consensus methods are frequently employed
ology. to explore an area where there is a lack of empiri-
cal knowledge (Pope & Mays, 2000) or where con-
sumer participation in the development of a clinical
The Philosophical Basis of program is sought (Twible, 1992). Increasingly, the
expert opinion of participants in consensus resea-
Consensus Methodology: rch is recognized as a valid form of developing evi-
Subtle Realism dence-based guidelines (Harbour & Miller, 2001).
Consensus approaches comprise three separate Defining Features of

Consensus Methodology
methodologies:
• Delphi studies,
• Nominal group techniques (NGTs), and While each of the consensus approaches differs in
• Consensus conferences. its methods, all share four common themes (Pope
Each of these three methodologies is presented & Mays, 2000):
and discussed later in the chapter. Consensus • Anonymity,
approaches embrace a mixed method philosophy • Iteration (i.e., the use of “rounds” that offer par-
and adopt a subtle realist perspective of knowledge ticipants the opportunity to change their minds as
(Duncan, 2004; Hammersley, 1992; Kirk and the perspectives of other group members become
Miller, 1986). Subtle realists assert that all re- known),
search involves subjective perceptions and obser- • Controlled feedback (i.e., the researcher shares
vations and concede that different methods will the distribution of the group’s response with the
produce different pictures of the participant(s) participants, and
being studied (Pope & Mays, 2000). The subtle • Statistical and qualitative analysis (i.e., each study
realist understands that researchers cannot claim to contains statistical measures of agreement as well
have absolute certainty regarding the findings of as a more qualitative analysis of the findings).
Table 24.1 A Comparison of Consensus Methods in Practice

Delphi Methodology Nominal Group Technique Consensus Conferences
Participant Location Remote Local Local
Time Scale Different rounds are Approximately 60–90 minutes. 1/ –1

2 Day
spread over weeks.
Participant identifi- Participant identification is not Participant identification not
Anonymity cation and study anonymous, but study item anonymous and
item prioritization. prioritization is. participant perspectives
are also public.
Variable, dependent Group sizes of between 6–9 Panel size of 10–15 people
on topic. Larger participants are recom- plus relevant conference
Optimal Size samples may be mended. delegates.
more open to high Several groups can be run and
attrition rates. summative assessment occur.
Variable. Statistical or Both statistical and qualitative. A consensus statement on
descriptive analysis Analysis can cease following an the issues raised is pre-
Analysis of Findings is possible accord- individual group or various pared and is formally pre-
ing to the study’s groups can be combined for a sented to relevant bodies
design. summative analysis. and organizations.
Delphi Methodology • Explore and expose assumptions and views,

• Gain the expertise of a larger number of people
The Delphi study is an iterative, multistage form of than possible through face-to-face gatherings,
survey designed to systematically gain, collate, • Educate the respondent group as to the consensus
and aggregate expert opinion and form a group or diversity of their opinion, and
consensus on a particular issue (Hasson, Keeney, • Eliminate time-consuming but irrelevant discus-
& McKenna, 2000; Love, 1997). It is designed to: sion (Strauss & Ziegler, 1975).
• Create an environment in which each subject is Background

anonymous—this enables people to share ideas
The Delphi technique was developed by Olaf
and opinions, which they may not feel free to
Helmer-Hirschberg for the Rand Corporation.
share if they are aware of other, possibly more
During the 1960s, the methodology was further
dominant, opinions,
developed within the scientific community as a
method of predicting future trends. Today, Delphi
studies have become an accepted methodology for
the development of consensus within health care
research (Cantrill, Sibbald, & Buetow, 1996; Love,
1997; Salmond, 1994). While Delphi studies have
traditionally been paper-based, they are increas-
Anonymity Iteration ingly carried out in an electronic format (Duncan
et al., 2004; Hasson et al., 2000; Jones & Hunter,
2000). This approach has the advantage of allow-
ing for swift communication between the resear-
cher and correspondent. However, it does require
participants to be computer literate.
Statistical and
Controlled Qualitative Methodological Considerations
Feedback Analysis
Researchers, through time, have slightly modified
the Delphi technique according to their own
requirements. However, the basic format has
remained unchanged and is outlined below. Within
each of the stages various factors that impact on
Figure 24.1 Features of consensus methodology. the methods integrity must be considered.
Chapter 24 The Nature and Use of Consensus Methodology in Practice 403
Identification and Recruitment generating new information and greater consensus

of Group Members (Ludwig, 1997).
One of the benefits of Delphi methodology is the
removal of geographical limitations encountered in Construction and Distribution
other consensus methods (Jones & Hunter, 2000). of Round One
Sample sizes of Delphi studies range broadly and Within a classical Delphi study, the first round
should be viewed in light of the subject area under consists of an open question aimed at generating
scrutiny. Samples tend to be smaller in a special- ideas or statements for the participants to rank in
ized area, while within a more general area there subsequent rounds (Gibson, 1998). Some studies
are generally greater numbers of participants. with large samples restrict the number of state-
While large sample sizes may appear attractive, ments that can be generated in order to obtain a
they tend to result in larger attrition rates in each manageable amount of data (Schmidt, 1997).
round of the study, which can bias the final analy- Other studies have adapted the standard Delphi
sis (Reid, 1988). process to include preexisting information for the
To gain a meaningful consensus in an area, it is purposes of ranking in round one (Duffield, 1993;
important to address the question to the correct Jerkins & Smith, 1994). However, this potentially
audience. Participants in some Delphi studies introduces bias and limits participant options
should be those who are affected by the decision. (Hassan & Barnett, 2002).
In other Delphi studies it is vital that participants Responses are then collated and grouped
are experts in their area (Sweigert & Schabacker, together so that where several different terms are
1974). Whatever the approach, the sampling used to closely describe the same issue, the resear-
method should be purposive or criterion based cher groups them together in one universal descrip-
(Hasson et al., 2000). Purposive sampling assumes tion (Hassan & Barnett, 2002). Where terms are
that the researcher’s knowledge of the popula- similar but with a different nuance, they are left
tion can be used to choose the participants who separate. Hassan and Barnett (2002) emphasize the
will be approached to participate in the study (Polit importance of verifying the process of groupings of
& Hungler, 1997). However, such a sampling statements to ensure that the data are fairly repre-
methodology is open to both researcher and partic- sented. Verifying is carried out by direct question-
ipant bias (Hasson et al., 2000). An alternative ing of the participants about the research to ensure
method of purposive sampling is to employ a third they have adequately understood what each term is
party, who has knowledge of the population, but communicating. Some researchers omit infre-
does not have a personal investment in the quently occurring items from future rounds of their
research. Delphi studies (Green, Jones, Huges, & Williams,
Hasson et al. (2000) highlight the importance 1999; Whitman, 1990). However, such an approach
of carefully approaching and informing potential may introduce bias to the results.
participants so that they are well prepared to con-
sent and to maintain participation in the study.
McKenna (1994) advocates that the initial contact Construction and Distribution
should be face-to-face in order to increase res- of Round Two
ponse rates. However, this is obviously impractical In the second round, the synthesized findings of the
if the sample is internationally dispersed and may first round are presented to each participant in a
raise ethical issues relating to consent. Whitman manner that allows each item to be ranked. While
(1990) recommends that written information be ranking of items is a key component of the Delphi
given at initial contact and accompany the first process, there is no standardized method of ranking
round of the study, as an effective way to increase (Love, 1997; Reid, 1988). Some typical approaches
response rates. Reminder letters should be emplo- are to request that participants rank the findings
yed to attempt to increase response rates (Hasson according to a fixed Likert scale or to use categori-
et al., 2000). cal coding such as “Not important,” “Slightly Im-
portant,” “Important,” and “Very Important.”
Development, Analysis, and Presentation
of Delphi Study Findings Collation of Results and Round Three
The number of rounds of data collection during a The third round begins with providing the partici-
Delphi study is not fixed. However, more than pants the collated results of the ratings given to
three rounds have been found to be ineffective in statements in round two. The manner in which the
results are presented will depend largely on the rat- result in a difference of more than 6%. Crisp,
ing methods employed in round two. Pelletier, Duffield, Adams, and Nagy (1997) ques-
Typically, within Delphi studies, only norma- tion the use of percentages and state that stability
tive information (e.g., the mean ranking or average of ratings over time is a better indicator of consen-
rating) is fed back to participants. However, it has sus. However, such an approach diverges from the
been argued that providing each participant’s original methodology and has not been widely
rationale for giving each rating can enhance the employed.
responses and make the process more meaningful
(Murphy et al., 1998b). Furthermore, providing Anonymity
purely normative data increases the tendency of
participants to bow to group pressure (Kaplan & Reviews of the Delphi procedure highlight
Miller, 1987). anonymity as one of its central positive features
(Beech, 1999; Cantrill et al., 1996a; Jones &
Achievement of Group Consensus/Possible Hunter, 2000). Anonymity is not complete since
Further Rounds to Gain Consensus the researcher must be aware of who has and has
not responded in order to pursue nonrespondents
There is little academic agreement regarding and encourage their participation within the study.
appropriate cutoff points for consensus in Delphi The term “quasi-anonymity” has been used to
studies. Williams and Webb (1994) state that con- describe the situation within a Delphi study in
sensus can be defined only when there is 100% which respondents will be known to the researcher
agreement among participants. Arguing that 100% and perhaps even to each other, but their individual
agreement is often unrealistic, Love (1997) pro- responses and opinions will remain known to the
posed 73% as the figure representing group con- researcher alone.
sensus. Other studies have selected figures such as
51% (Loughlin & Moore, 1979), 70% (Sumison,
1998), and 80% (Green et al., 1999). Ultimately, Nominal Group Technique
the percentage chosen will depend on the sample
size; one participant disagreeing in a sample size Nominal Group Technique (NGT) is an evaluative
of 100 will result in a difference of 1%, while one methodology that uses a structured group activity,
participant disagreeing in a sample size of 15 will designed to elicit the views of group members on
A Delphi Survey of Best Practice Occupational Therapy for Parkinson’s Disease

in the United Kingdom (Deane et al., 2003)
Deane et al. (2003) present a Delphi study that round data) and returned by 87% (n ⫽ 153) of
aimed to provide evidence for designing a best participants. Before beginning the study, it was
practice statement for occupational therapists work- agreed that the criteria for consensus would be
ing with people who have in the area Parkinson’s 80%. Consensus was reached on 82% (n ⫽ 27)
disease. The study was also used to provide guid- of items (a total of 33 items were ranked in round
ance for future randomized control trials of inter- two). The top 10 items in which participants
ventions for this population. Two hundred and strongly agreed that the specific expertise of occu-
forty-two individuals were invited to participate in pational therapists should focus on improving on
the study and 69% (n ⫽ 168) responded. maintaining function were:
As the Delphi study was a follow-up study of
an earlier survey, the initial round of the research Eating and drinking (97%)
was formed from 30 practice statements and asked Domestic and kitchen skills (97%)
to rate these on a five-point scale. This procedure Washing and dressing (96%)
represents a modification of traditional Delphi Home safety (93%)
studies that begin with the brainstorming of priori- Work activities (92%)
ties prior to rating. However, a brainstorming of Leisure activities (90%)
practice statements had recently been carried out Confidence (89%)
by the authors and further statements were added Cognitive skills (86%)
on participants’ suggestion. Transfers (85%)
The second round was sent out to 176 partici- Integration into society (78%)
pants (as some had missed the deadline for first-
Figure 24.2 A researcher uses a nominal group technique with occu-

pational therapy program graduates as part of a program evaluation.
a specific topic (Lloyd-Jones, Fowell, & Fligh, ommends that the number of participants for each
1999). nominal group should range from six to nine,
while Carney, McIntosh, and Worth (1996) suggest
Background that groups with fewer than five participants may
be mildly threatening. Ultimately, as Cantrill and
NGT was developed in the United States during
colleagues (1996) state, “…the size of the eventual
the 1960s (Van de Ven & Delbacq, 1972). Initially
panel is determined by pragmatic considerations”
it was applied to government services, education,
(p. 69).
and industry. Later, it became incorporated into the
While Murphy and colleagues (1998b) argue
research repertoire of health care research, where
that the personal characteristics of participants
the method was predominantly used for examining
have little effect on the outcome, they suggest each
the appropriateness of clinical interventions, prac-
nominal group panel should be interpreted in light
tice development, education, and priority setting
of key characteristics such as the profession(s)
(Cantrill et al., 1996; Pope & Mays, 2000).
represented (Cantrill et al., 1996; Murphy et al.,
Methodological Considerations 1998a). Hall (1983) suggests that heterogeneous
groups, rather than homogeneous groups, generate
NGT has been compared with focus group a greater degree of high consensus outcomes.
methodology discussed in Chapter 20. However, in Regardless of the composition of the group, each
contrast to the open discussion used in focus participant should be selected on the basis of his
groups, NGT tends to be strictly controlled and or her expertise in the area (Pope & Mays, 2000).
discussion is restricted to later stages of the data In health care studies, experienced clinicians are
collection (Cantrill et al., 1996). Furthermore, frequently viewed as experts (Cantrill et al., 1996)
NGT focuses on the study of a single topic rather along with clients, who are viewed as experts of
than the range of ideas commonly associated with their own experience (Hares, Spencer, &
focus group methodology (Pope & Mays, 2000). Gallagher, 1992).
Several methodological issues must be consid- The level of response from participants partak-
ered when undertaking a nominal group study: ing in a nominal group study will directly affect
the reliability of the study’s results. Not everyone
• Participant characteristics,
who is invited to participate in a nominal group
• Response rates, and
study will consent (Cantrill et al., 1996). One of
• Levels of validity and reliability (Cantrill et al.,
the main disadvantages of NGT methodology is
1996; Pope & Mays, 2000).
that participants must agree to come to a central
Optimal size and composition of groups should meeting location, leading to a high level of non-
be governed by the environment and purpose of response (Claxton, Ritchie, & Zaichkowsky,
the study (Cantrill et al., 1996). Hall (1983) rec- 1980). Lack of respondent interest is the most
important factor influencing non-participation analysis. On the other hand, identification

(Goyder, 1982; Heberlein & Baumgartner, 1978). of dimensions is done for the purpose of
Nominal groups can be biased, as they are likely to providing a typology of themes i.e. group-
contain a disproportionate number of enthusiasts ing of themes that relate to a general prob-
(Cantrill et al., 1996). lem area” (p. 311).
No specific assessment of reliability and valid-
The process of indexing (or ranking) themes is
ity of nominal group studies has been developed.
carried out by aggregating the scores assigned to
Cantrill et al. (1996) suggest that the reliability of
each statement. Therefore, themes that had been
nominal group techniques can be considered by
highlighted in the majority of sessions and rated
comparing the level of agreement of decisions
highly receive a greater score than another theme
between two or more groups. Twible (1992) has
that occurred less frequently and was rated lower
developed a twofold level of analysis of nominal
(Claxton et al., 1980). The aggregation of state-
group data, which reviews each group’s individual
ments to provide ranking of results is deemed as
ratings and combines the groups’ responses in a
one of the advantages of the methodology
form of summative analysis. The validity of a
(Claxton et al., 1980). This advantage is, however,
nominal group technique is also difficult to assess
tempered by the limitations of ordinal data.
(Cantrill et al., 1996). Pope and Mays (2000) point
out that agreement may reflect a consensus of
ignorance, rather than wisdom. It is certainly
important to appraise and report the validity and
Consensus Conferences
reliability of each NGT study. A consensus conference is a public hearing lasting
3 days. It is chaired with an audience. It involves
Analysis and Presentation active participation of 10 to 15 laypersons who are
of Nominal Group Results sometimes referred to as the jury or the panel and
an equivalent number of experts, who may be from
Frequently, studies that use nominal group metho- different disciplines and/or represent different
dologies use only one group for data collection viewpoints within a discipline. The purpose of a
(Crabb, Simpson, Hall, Beck, & Willard, 1981; consensus conference is to develop an informed
Horton, 1980; Justice & Jang, 1990; Lloyd-Jones debate and report on a topic in which there is little
et al., 1999; McClusky, 2000; Sloan, 1999). Miller, developed knowledge. The debate is often devel-
Shewchuk, Eilliot, and Richards (2000) used two oped using six to seven main questions to stimu-
groups: Each group comprised differing experts late debate (Agersnap, 1992).
(clinicians and patients) and was individually ana-
lyzed and then compared with the other. Two Background
papers report data collection using several NGT
groups and combining the results to form a sum- Consensus conferences originated in the United
mative analysis (Claxton et al., 1980; Twible, States. They aim to provide a means by which
1992). members of specific professional groups and more
Claxton et al. (1980) describe a four-stage broadly society become purposefully involved in
process to analyzing the results of a series of nom- influencing important decisions.
inal group interviews:
Methodological Considerations
• Categorization of initial statements into themes, The consensus conference engenders debate by
• Calculation of a score or index reflecting the means of a broadly based group of selected repre-
importance of each theme, sentatives. Experts present a variety of views on a
• Ranking of themes according to their index, and subject and are open to questioning by the confer-
• Regrouping of themes to form major dimensions. ence. Finally, the conference prepares a consensus
Claxton et al. (1980) differentiate between statement on the issues raised and this statement is
statements, themes, and dimensions in the follow- formally presented to the relevant bodies and
ing manner: organizations.
The conference is helped in its work by a mod-
“…the purpose of identifying [themes] is to erator who ensures that the conference flows and
aggregate across NGT sessions statements functions efficiently. Background documentation
that express essentially the same idea, a is frequently sent to conference delegates regard-
process conceptually similar to content ing the focus of the conference so that delegates
Research Priorities in Forensic Occupational Therapy (Duncan et al., 2003)
Duncan et al. (2003) employed a Nominal Group sampling method, highlights the potential for par-
Technique (NGT) as part of a an exercise for ticipant bias in such a methodology. Despite
establishing research priorities for occupational this admitted weakness, the NGT study identified
therapists working with individuals who have the following six clear research priorities for
mental illnesses or intellectual disabilities and practice:
are being cared for in secure (forensic) settings.
In this case the NGT was selected as an appropri- • Outcome measurement
ate consensus method. Since the data were gath- • Evaluating interventions of effectiveness
ered on a convenience sample of occupational • An examination of the efficacy of life-skills
therapists attending a national forensic occupa- training
tional therapy conference, use of a method • The development of specific forensic risk
that could be completed in one session was assessments
advantageous. • A multicentered study of AMPS
All delegates (n ⫽ 110) were invited to attend • Validation of current assessments in practice
the NGT session; however, only eight elected to These results correlated closely with those of a
participate in the session. Participants undertook a larger survey study that examined the same issue.
classic NGT process of idea generation, recording Comparing the findings from these two studies
of ideas, discussion of ideas, and individual with different methodologies provided important
ranking of each individual’s top six topics. The insights into research priorities in forensic occupa-
small number, coupled with the convenience tional therapy.
come prepared for discussion. The experts should experts in the area (Fink, Kosecoff, Chassin, &
be given clear guidelines regarding their role in the Brook, 1984; Jones & Hunter, 2000; Lomas, 1991).
proceedings. The technical and administrative Murphy et al. (1998b) highlight that the definition
requirements of such an event should not be under- of an expert depends on the perspective being
estimated as considerable input is required to syn- sought. For instance, clinicians are expert in the
thesize the proceedings and produce a definitive delivery of interventions; researchers bring scien-
consensus conference statement by its conclusion. tific expertise; and clients have a unique expertise
in experiencing the impact of interventions.
Methodological Considerations Group Composition: Heterogeneity

in Consensus Development vs. Homogeneity
Methodology When developing a consensus group, it is impor-
tant to consider whether the characteristics of
A wide variety of factors can influence the plan- group participants should be similar (homoge-
ning, development, and analysis of consensus stud- neous) or mixed (heterogeneous). The following
ies. Several of these factors are discussed below. characteristics are typically considered:
• Age,
Selection of Consensus • Occupation,
Development Methodology • Cultural background,
None of the three methods is innately superior to • Abilities and expertise,
the others. Rather, the selection of an appropriate • Status, and
consensus method depends on a variety of factors • Mix of initial opinions (Murphy et al., 1998b).
such as participant availability, geographical dis-
Neither heterogeneous nor homogeneous
persion, resources, and the overall aim of the con-
groups are considered superior. Rather, Cantrill et
sensus study.
al. (1996) suggest that the “…composition of both
Delphi and Nominal Group methodologies should
Participant Selection be governed by the purpose of the investigation”
The main consideration in the selection of partici- (p. 69). Consensus conferences by definition
pants for consensus methods is their credibility as include two distinct groups and, within each,
consideration is given to issues of heterogeneity or the quality of group decisions” (p. 67). Cantrill et
homogeneity. al. (1996) argue that consensus approaches are par-
ticularly suitable when individual judgments must
be elicited and combined in order to form a rigor-
Social Environment
ous understanding that cannot be made by a single
Time pressure and mood are two social environ- person or random grouping of people.
mental factors that have been highlighted as affect- Nonetheless, authors caution against over-
ing the outcome of group decision-making. Karau reliance on the findings of consensus approaches
and Kelly (1992) found and question the nature
that when time pressure of consensus (Lomax &
was high, members’ ini- When rigorously applied, McLeman, 1984; Pope &
tial preferences had a consensus methods offer Mays, 2000). Consensus
greater influence than the approaches reflect issues
overall group response. the potential to develop an on which the majority of
With moderate time pres- expert panel evidence base the groups agree. How-
sure the groups became ever, agreement is affec-
more focused and the on which to develop best ted by the particular
quality of the output practice guidelines and methodology being ap-
increased. Isen and plied which can skew the
Means (1983) demon- provide the foundation for picture of the group res-
strated that participants future research. ponse, concealing strong
who are more positive in minority disagreement
mood engaged less thor- (Carney et al., 1996).
oughly in a group process than participants who Despite these drawbacks, consensus approaches
were neutral in mood. have several advantages (Lomax & McLeman,
1984):
Characteristics of Group Facilitator
• They limit the influence and potential bias of the
Wortman, Vinokur, and Sechrest (1988) found that researcher,
a facilitative chairperson was crucial to a success- • They avoid pressure on group participants to
ful consensus conference. Clawson, Bostrom, and conform to other members’ opinions,
Anson (1993) suggest that the role of a group facil- • They incorporate the advantages of both qualita-
itator includes: tive and quantitative techniques, and
• Providing structure, • They are economic and efficient methods for
• Maintaining the agenda, gathering data.
• Managing conflict, and
• Creating a positive environment.
Summary
Murphy et al. (1998b) state that the group facil-
itator will likely play a key role in each consensus Consensus approaches to research are increasingly
method. being used in health care research. They provide
important methods for systematically structuring
expert opinion in areas where the existing evidence
Limitations of Consensus base is sparse. However, these methodologies have
Methodology been criticized (Sackman, 1975) and even propo-
nents of the approach have recognized its limita-
tions (Carney et al., 1996; Lloyd-Jones et al.,
Consensus approaches contribute to the develop-
1999; Lomax & McLeman, 1984; Twible, 1992).
ment of knowledge in health research, particularly
Pope and Mays (2000) have recommended that
in under-researched areas or where the geographi-
when carrying out studies using consensus
cal spread of experts restricts other methods of data
methodology, the following criteria be used to
collection (Harbour & Miller, 2001; Murphy et al.,
examine the validity and relevance of the research:
1998b). Cantrill et al. (1996) suggest that “…when
properly used, [consensus] techniques are powerful • The emphasis should be on a clear justification
tools for increasing a group’s capacity to generate for using such methods,
critical ideas, understand problems and improve • The use of sound methodology, including the
selection of experts and a precise definition of Goyder, J. C. (1982). Further evidence on factors affecting
target levels of consensus, should be described in response rates to mailed questionnaires. American
Sociology Review, 47, 550–553.
detail, Green, B., Jones, M., Hughes, D., & Williams, A. (1999).
• The findings should be presented in an appropri- Applying the Delphi technique in a study of GPs’
ate and accessible manner, and information requirements. Health and Social Care in
• The relevance of the findings to the topic area the Community, 7(3), 198–205.
Hall, R. S. (1983). The Nominal Group Technique for
should be clearly articulated.
planning and problem solving. Journal of
When rigorously applied, consensus methods Biocommunication, 10(2), 24–27.
Hammersley, M. (1992). What’s wrong with ethnography?
offer the potential to develop an expert panel evi-
London: Routledge.
dence base on which to develop best practice Harbour, R., & Miller, J. (2001). A new system for grading
guidelines and provide the foundation for future recommendations in evidence based guidelines. British
research. Medical Journal, 323, 334–336.
Hares, T., Spencer, J., & Gallagher, M. (1992). Diabetes
care: Who are the experts? Quality in Health Care, 1,
REFERENCES 219–224.
Agersnap, T. (1992). Consensus Conferences for Hassan, T. B., & Barnett, D. B. (2002). Delphi type
Technology Assessment. In: Technology and democ- methodology to develop consensus on the future design
racy. The use and impact of technology assessment in of EMS systems in the United Kingdom. Emergency
Europe. Proceedings of the 3rd European Congress on Medicine Journal, 19(2), 155–159.
Technology Assessment, Copenhagen, November 4–7, Hasson, F., Keeney, S., & McKenna, H. (2000). Research
1992. guidelines for the Delphi survey technique. Journal of
Beech, B. (1999). Go the extra mile-use the Delphi Tech- Advanced Nursing, 32(4), 1008–1015.
nique. Journal of Nurse Management, 7(5), 281–288. Heberlein, T. A., & Baumgartner, R. (1978). Factors affect-
Cantrill, J. A., Sibbald, B., & Buetow, S. (1996). The ing response rates to mailed questionnaires: A quantita-
Delphi and nominal group techniques in health services tive analysis of the published literature. American
research. International Journal of Pharmacy Practice, Sociology Review, 43, 447–462.
4(2), 67–74. Horton, J. N. (1980). Nominal group technique. A method
Carney, O., McIntosh, J., & Worth, A. (1996). The use of of decision-making by committee. Anaesthesia, 35(8),
the Nominal Group Technique in research with com- 811–814.
munity nurses. Journal of Advanced Nursing, 23(5), Isen, A. M., & Means, B. (1983). The influence of positive
1024–1029. affect on decision making strategy. Social Cognition, 2,
Claxton, J. D., Ritchie, J. R., & Zaichkowsky, J. (1980). 18–31.
The Nominal Group Technique: Its potential for con- Jerkins, D., & Smith, T., (1994). Applying Delphi method-
sumer research. Journal of Consumer Research, 7(3), ology in family therapy research. Contemporary Family
308–313. Therapy, 15, 411–430.
Clawson, V. Q., Bostrom, R. P., & Anson, R. (1993). The Jones, J., & Hunter, D. (2000). Using the Delphi and
role of the facilitator in computer supported meetings. Nominal Group Technique in health services research.
Small Group Research, 24, 547–565. In C. Pope & N. Mays (Eds.), Qualitative research in
Crabb, L., Simpson, R., Hall, D., Beck, J. & Willard, D. healthcare (2nd ed.). London: BMJ Publishing Group.
(1981). Effective decision-making method for dentists: Justice, J., & Jang, R. (1990). Tapping employee insights
The nominal group technique. General Dentistry, with the Nominal Group Technique. American
29(2), 129–132. Pharmacy, 30(10), 43–45.
Crisp, J., Pelletier, D., Duffield, C., Adams, A., & Nagy, Kaplan, M. F., & Miller, C. E. (1987). Group decision
S. (1997). The Delphi method? Nursing Research, making and normative vs. informational influence:
46(2), 116–118. Effects of type of issue and assigned decision rule.
Deane, K. H. O., Ellis-Hill, C., Dekker, K., Davies, P., & Journal of Personal and Social Psychology, 28,
Clarke, C. E. (2003). A Delphi survey of best practice 542–571.
occupational therapy for Parkinson’s disease in the Karau, S. J., & Kelly, J. R. (1992). The effects of time
United Kingdom. British Journal of Occupational scarcity and time abundance on group performance
Therapy, 66(6), 247–254. quality and interaction process. Journal of Experi-
Duffield, C. (1993). The Delphi technique: A comparison mental and Social Psychology, 28, 542–571.
of results obtained using two expert panels. Internatio- Kirk, J., & Miller, M. (1986). Reliability and validity
nal Journal of Nursing Studies, 30(3), 227–237. in qualitative research. Beverly Hills, CA: SAGE
Duncan, E. A. S., & Nicol, M. M. (2004). Subtle realism Publications.
and occupational therapy: An alternative approach to Lloyd-Jones, G., Fowell, S., & Bligh, J. G. (1999). The use
knowledge generation and evaluation. The British of the nominal group technique as an evaluative tool in
Journal of Occupational Therapy, 67(10), 453–456. medical undergraduate education. Medical Education,
Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984). 33(1), 8–13.
Consensus methods: Characteristics and guidelines for Lomas, J. (1991). Words without action? The produc-
use. American Journal of Public Health, 74, 979–983. tion, dissemination, and impact of consensus recom-
Gibson, J. M. (1998). Using the Delphi technique to iden- mendations. Annual Review of Public Health, 12,
tify the content and context of nurses’ continuing pro- 41–65.
fessional development needs. Journal of Clinical Lomax, P., & McLeman, P. (1984). The uses and abuses
Nursing, 7(5), 451–459. of Nominal Group Technique in polytechnic course
evaluation. Studies in Higher Education, 9(2), American Educational Research Association. Chicago.
183–190. (ERIC Document Reproduction Service No. ED 091
Loughlin, K. G., & Moore, L. F. (1979). Using Delphi to 415)
achieve congruent objectives and activities in a pedi- Twible, R. L. (1992). Consumer participation in planning
atrics department. Journal of Medical Education, health promotion programmes: A case study using the
54(2), 101–106. nominal group technique. Australian Occupational
Love, C. (1997). A Delphi study examining standards for Therapy Journal, 39(2), 13–18.
patient handling. Nursing Standard, 11(45), 34–38. Van de Ven, A. H., & Delbacq, A. L. (1972). The nominal
Ludwig, B. (1997). Predicting the future: Have you consid- group as a research instrument for exploratory health
ered the Delphi methodology? Journal of Extension, studies. American Journal of Public Health, 62,
35. Retrieved March 18, 2002, from http://www.joe. 337–342.
org/joe/1997october/tt2.html Whitman, N. I. (1990). The Delphi technique as an alterna-
McClusky, A. (2000). Collaborative Curriculum develop- tive for committee meetings. Journal of Nursing
ment: Clinicians views on the neurology content of a Education, 29(8), 377–379.
new occupational therapy course. Australian Williams, P. L., & Webb, C. (1994). The Delphi technique:
Occupational Therapy Journal, 47(1), 1–10. A methodological discussion. Journal of Advanced
McKenna, H. P. (1994). The Delphi technique: A worth- Nursing, 19(1), 180–186.
while research approach for nursing? Journal of Wortman, P. M., Vinokur, A., & Sechrest, L. (1988). Do
Advanced Nursing, 19(6), 1221–1225. consensus conferences work? A process of evaluation
Miller, D., Shewchuk, R., Elliot, T. R. & Richards, S. of the NIH consensus development programme.
(2000). Nominal group technique: A process for identi- Journal of Health Politics Policy and Law, 13,
fying diabetes self-care issues among patients and care- 469–498.
givers. Diabetes Educator, 26(2), 305–310.
Murphy, E., Dingwall, R., Greatbatch, D., Parker, S., & RESOURCES
Watson, P. (1998a). Qualitative research methods in The following resources are useful for people interested in
health technology assessment: A review of the litera- conducting a consensus study:
ture. Health Technology Assessment, 2(16), vii–260.
Murphy, M. K., Black, N. A., Lamping, D. L., McKee, General
C. M., Sanderson, C. F. B. J., & Marteau, T. (1998b). Murphy, M. K., Black, N. A., Lamping, D. L., McKee, C.
Consensus development methods and their use in clini- M., Sanderson, C. F. B. J., & Marteau, T. (1998).
cal guideline development. Health Technology Consensus development methods and their use in clini-
Assessment, 2(3). cal guideline development. Health Technology
Polit, D. F., & Hungler, B. P. (1997). Essentials of nursing Assessment, 2(3).
research: Methods, appraisal and utilization. New
York: Lippincott, Williams & Wilkins. Nominal Group Information
Pope, C., & Mays, N. (2000). Qualitative research in Claxton, J. D., Ritchie, J. R., & Zaichkowsky, J. (1980).
health care (2nd ed.). London: British Medical Journal The Nominal Group Technique: Its potential for con-
Publishing Group. sumer research. Journal of Consumer Research, 7(3),
Reid, N. (1988). The Delphi technique: Its contribution to 308–313.
the evaluation of professional practice. In R. Ellis Dunham, R. B. (1998). Nominal Group Technique: A users
(Ed.), Professional competence and quality assurance guide.
in the caring professions (pp. 230–262). London: HYPERLINK “http://instruction.bus.wisc.edu/obdemo/
Chapman & Hall. readings/ngt.html” (28 April 2005).
Sackman, H. (1975). Delphi critique. Boston, MA: Delphi Study Information
Lexington Books. Greatorex, J., & Dexter, T. (2000). An accessible analytical
Salmond, S. W. (1994). Orthopaedic nursing research approach for investigating what happens between
priorities: A Delphi study. Orthopaedic. Nurse, 13(2), rounds of a Delphi study. Journal of Advanced
31–45. Nursing, 32(4), 1016–1024.
Schmidt, R. C. (1997). Managing Delphi surveys using Hasson, F., Keeney, S., & McKenna, H. (2000). Research
non-parametric statistical techniques. Decision guidelines for the Delphi survey technique. Journal of
Sciences, 28, 763–774. Advanced Nursing, 32(4), 1008–1015.
Sloan, G. (1999). Good characteristics of a clinical super-
visor: A community mental health nurse perspective. Consensus Study Information
Journal of Advanced Nursing, 30(3), 713–722. Anderson, I., & Jæger, B. (1999). Scenario Workshops and
Strauss, A. L., & Ziegler, L. H. (1975). The Delphi tech- consensus conferences: Towards more democratic deci-
nique and its uses in social science research. The sion making. Science and Public Policy, 26(5),
Journal of Creative Behaviour, 9(4), 253–259. 331–340.
Sumsion, T. (1998). The Delphi technique: An adaptive Slocum H (2005) Participatory Methods Toolkit: A practi-
research tool. The British Journal of Occupational tioner’s manual. Method: Consensus Conference. King
Therapy, 61(4), 153–156. Baudouin Foundation & Flemish Institute for Science
Sweigert, R. L., & Schabacker, W. H. (1974). The Delphi and Technology Assessment. Belgium. Retrieved on
technique: How well does it work in setting educational November 29, 2005 from http://www.kbs-rb.be/files/db/
goals? Paper presented at the Annual Meeting of the EN/PUB_1540_Toolkit_4_ConsensusConference.pdf
C H A P T E R 2 5
Using Mixed Methods Designs to Study

Therapy and Its Outcomes
Mary A. Corcoran
Historically, only two major research traditions dures. Prominent methodologists have called for
have been widely recognized in the global scien- conceptualizing mixed methods as a third tradition
tific community. These traditions are known by separate but equal to qualitative and quantitative
a number of terms, including post-positivist, traditions. They further support its development as
experimental, and quantitative as opposed to a set of rigorous research tools for addressing
constructionist, naturalistic, and qualitative. The complex questions.
quantitative–qualitative dichotomy may be most The term mixed methods refers to a research
familiar since these terms differentiate the types design that integrates elements of both qualitative
of data primarily collected within each tradition. and quantitative methods so that the strengths of
Chapter 3 discusses these two traditions and points each are emphasized.1 However, Tashakkori and
out their differing epistemological modes (i.e., dif- Teddlie (2003) posit that mixed methods are more
ferent underlying assumptions, focus, design, and that just combinations of qualitative and quantita-
methods of data gathering and analysis). tive procedures. Because qualitative and quantita-
For years, scientists tive procedures stem
have informally com- from two separate (and
bined designs or meth- Prominent methodologists in many ways, opposing)
ods within or from both have called for conceptualiz- epistemologies, they must
traditions. Usually, they be combined as inter-
did so by conducting a ing mixed methods as a dependent but separate
sidebar or secondary third tradition separate but procedures during data
study within a larger collection. This is usually
one. These combinations equal to qualitative and accomplished by estab-
are most often intended quantitative traditions. lishing one of the tradi-
to generate in-depth tions (either qualitative or
information about the quantitative) as the core
experiences or opinions of a group of people who method. The underlying assumptions of the core
have participated in an experimental study. method are prioritized and reflected in the study
However, mixed methods as a legitimate design purpose, methodological decisions, and overall
approach began to gain wide attention in the 1980s analytic approach. In implementing the portion of
after Denzin (1978) introduced the concept of tri- the study that is based on a secondary method, the
angulation. Triangulation refers to the practice of investigator must guard against violating any of
gathering data from a number of different sources the assumptions of the core method while still
for the purposes of detaching the method of meas- maintaining the integrity of the secondary method.
urement from the phenomena being measured. For This requires careful planning and solid under-
example, if a social behavior emerges in an inter- standing of both traditions represented. During
view and is observed in action, this finding may be data analysis, the core method continues to dictate
regarded as particularly salient because two differ- the overall approach, although data from the sec-
ent data sources independently confirm it. The ondary method may be transformed and integrated
concept of triangulation fits well with a new para- with data from the core method.
digm of pragmatism that was emerging and taking This relatively new approach to scientific
root at the same time (Howe, 1988). inquiry is growing in popularity owing to the flex-
Pragmatism holds that qualitative and quantita-
1Mixed methods should be distinguished from mixed
tive traditions are compatible and can be success-
designs, which are consistent with a quantitative tradition
fully combined in the same design. However, the
and involves “factorial designs in which the number of
mixed methods studies that followed were a levels of the factors are not the same for all factors” (Vogt,
hodgepodge of typologies, definitions, and proce- 1993, p 140).
Randomized study to test the effectiveness of a new

behavioral intervention
In-depth interviews to determine

how intervention was used by
several different groups
Figure 25.1 Illustration of a mixed methods design.
ibility it affords investigators. For example, a and that faces the challenge of blending many bod-
mixed methodologist has the ability to combine ies of knowledge in the dynamic concept of occu-
strict controls necessary for generalization with pation. Therefore, the purpose of this chapter is to
in-depth examinations of the study participants’ introduce current thinking about mixed methods as
experiences or perceptions on a particular topic. a third methodological tradition with unique
Figure 25.1 illustrates this combination of con- nomenclature, principles, designs, and procedures,
trolled quantitative design with a qualitative natu- and to apply that tradition to occupational therapy.
ralistic design, graphically using a typical mixed The timing is perfect because Tashakkori and
method approach. In this figure, traditional quanti- Teddlie (2003) have recently published the first
tative methods, such as experimental and quasi- Handbook of Mixed Methods, upon which this
experimental designs, can be conceptualized as the chapter draws heavily.
wide, shallow box. This box in Figure 25.1 repre-
sents the broad scope of experimental studies
regarding a very narrow topic. The narrow but Mixed Methods Nomenclature
deep triangle represents the in-depth but highly
focused approach of qualitative studies. In Figure
and Typologies
25.1, a primarily quantitative study to test the The overarching typology of research designs that
effectiveness of a new behavioral intervention also includes mixed methods is known as multiple
contains in-depth interviews that allow the investi- methods designs (Tashakkori & Teddlie, 2003).
gator to understand more about the experiences of Multiple methods designs refer to use of two or
those individuals who participated in this efficacy more data collection strategies or methods for a
study. The investigator may want to hear from sev- given research question. Multiple methods designs
eral types of participants, including those who can be further divided into two subcategories,
were able to incorporate behavioral changes in multimethod designs and mixed methods designs,
their lives, those who had difficulty doing so, and which are defined below (Tashakkori & Teddlie,
those who were unable to make changes called for 2003).
in the intervention. While investigators using such
an approach can answer the question, “What is the
Multimethod Designs
effect of the intervention on X?”, they can also
gather information about how easy or difficult it Multimethod designs incorporate two or more data
was for participants to actually use the behavioral collection techniques within only one tradition
strategies from the intervention. This informa- (qualitative or quantitative) (Taskakkori & Teddlie,
tion can then be used to guide analysis, interpret 2003). For example, a study question regarding the
study findings, and refine the intervention in future effectiveness of an occupational therapy interven-
trials. tion may be best answered through a quantitative
There are many ways to merge quantitative and tradition, such as a randomized two-group design.
qualitative procedures in new and unique ways. As However, an investigator using a multimethod
research questions become more complex, mixed approach may decide to triangulate his data with
methods may emerge as a principal tradition in two different surveys to measure the dependent
social science in years to come (Tashakkori & variable, independence in self-care. As shown in
Teddlie, 2003). Mixed methods are particularly Figure 25.2, the dependent variable is measured by
relevant for occupational therapy, a profession a self-report survey from the study participant and
with firm foundations in a number of disciplines a proxy report from a caregiver.
Chapter 25 Using Mixed Methods Designs to Study Therapy and Its Outcomes 413
investigator may suspect that an underlying cul-

Tashakkori and Teddlie (2003) make the obser-
tural issue is mediating the effect of the interven-
vation that quantitative research designs have
enjoyed a long tradition of commonly under- tion being tested. This investigator hypothesizes
stood and well-defined terms. This has provided that the participants’ culturally based definition of
the quantitative tradition with a common lan- disability shapes the level at which they will enact
guage as a basis for developing and describing the intervention procedures being studied.
methodologies. The qualitative research tradition A mixed methods design could be used in this
has been working toward a common lexicon only case. To implement such a design, the investigator
for the past two decades but during that time has might conduct an ethnography subsequent to the
made great strides in identifying and defining field experiment to develop a better understanding
key concepts. of the relationship between definition of disability
Mixed methods, as a tradition in its “adoles-
and self-care actions. By comparing Figures 25.1
cence” (Tashakkori and Teddlie, 2003), has only
begun to consider whether a common language and 25.2, one can see that the former is an illustra-
is needed, and if so, what system of terms and tion of a mixed methods design (using methods
definitions should be adopted. Tashakkori and from both the qualitative and quantitative tradi-
Teddlie (2003) provide a Glossary of terms in tions), while the latter illustrates a multimethod
their Handbook of Mixed Methods that has design (using more than one method to collect data
been defined through consensus of several within a single quantitative design).
leading authors in the field and does not have Other authors have recommended more com-
alternative definitions. The serious student of plex typologies. For instance, Newman, Ridenour,
mixed methods should be familiar with this Newman, and DeMarco (2003) suggest that the
new language.
typology be organized by research purpose rather
than design type. Interested readers are encour-
aged to consult other texts for additional ways of
systematically classifying mixed methods designs,
Multimethod designs are powerful approaches including Newman and Benz (1998), Tashakkori
to complex and nuanced research questions and an and Teddlie (1998, 2003), Creswell (2003), and
important strategy in the qualitative tradition for Greene & Caracelli (1997).
ensuring trustworthiness. However, because multi-
method designs do not combine more than one
research tradition, as do mixed methods designs, Principles of Mixed
competing philosophies and underlying assump-
tions are not an issue. Therefore, the remainder of
Methods Designs
this chapter is devoted to discussing the unique
Morse (2003) warns strongly against using the
methodological associated with mixed methods
“muddling method” of combining models (p. 189),
designs.
which involves simply tossing together methods
and models without adequate consideration for
Mixed Methods Designs issues of validity. For example, the investigator
In mixed methods designs, qualitative and quanti- who combines ethnography with a field experi-
tative traditions are used simultaneously or con- ment must avoid changing the intervention mid-
secutively in the methods section (Tashakkori & stream to reflect what he has learned from the
Teddlie, 2003). Extending the above example, the participants, and runs the risk of being unable to
Participant
Measure of
Self-Care
Baseline Status
and Eligibility Intervention
Criteria
Caregiver
Measure of
Self-Care
Figure 25.2 Illustration of a quantitative multimethod design.

maintain an unbiased and objective status. Neither and qualitative traditions. This is obviously a proj-
of these actions would be a problem in a qualita- ect with a discovery basis, so an inductive drive is
tive tradition, but would introduce threats to valid- appropriate.
ity in the quantitative portion of the study.
As with all traditions of research, decisions Adhere to the Methodological
must be made based on the conceptual framework, Assumptions of the Core Method
purpose, and research question(s) of the study.
This may be even more important when combining Respecting the integrity of methods (i.e., do not
qualitative and quantitative traditions, which have violate the assumptions, philosophical founda-
diametrically opposed philosophies on several tions, and procedures of the core method) seems
fundamental points, including the shape of the like a simple message when introducing other sec-
research process (linear versus spiral), role ondary methods. In actual practice, respecting
of the investigator (objective versus subjective), methodological integrity requires continually link-
and type of logic used (deductive versus induc- ing the unique philosophical foundations with the
tive). If not carefully planned, an investigator mayresearch question(s) and maintaining one method
find that a number of threats to validity have been as dominant over the other. When decisions are
introduced in the process of mixing methods and made throughout the project, methodological
models. integrity is always among the first considerations,
but an investigator using a mixed methods design
cannot stop there. The investigator must begin with
Principles of Mixed Methods Designs consideration for the integrity of the core method
Four principles are offered by Morse (2003) for a and then consider the implications of each decision
mixed methods design, and are discussed below. for the integrity of the secondary method(s).
Morse (2003) suggests keeping the core and sec-
Recognize the Theoretical ondary methods clear by using capital letters when
referring to the core method (such as QUAL ⫹
Drive of the Project
quan to denote a study that is primarily qualitative
Research projects fall broadly into two types of with a secondary quantitative component).
purpose: discovery or testing (i.e., inductive or For example, an investigator is interested in
deductive). Morse uses the term theoretical drive knowing how a school-based intervention affects
of a project to refer to whether the main reasoning both self-confidence and legibility of handwriting.
process required for the purpose is inductive or The investigator decides to test intervention effects
deductive (1991). The investigator must remain on the dependent variables in two ways, directly
clear as to the type of inquiry and reasoning through a measure of self-confidence and hand-
process driving the project and how each compo- writing samples of the participants, and indirectly
nent fits the whole. through a proxy open-ended interview with the
In a mixed methods study, both the quantitative teachers on these variables (self-confidence and
and qualitative components are introduced. legibility). The type of mixed design would
However, there can be only one theoretical therefore be written so as to designate that the
drive, either inductive or study is quantitative with
deductive. The following a deductive theoretical
example is used to illus- If not carefully planned, an drive plus an additional
trate these decisions in investigator may find that a qualitative method (proxy
action. An investigator is open-ended interview)
interested in describing number of threats to validity used simultaneously
the ways people with have been introduced in the (i.e., QUAN ⫹ qual). If
traumatic brain injury the secondary qualitative
(TBI) handle informa- process of mixing methods investigation were to be
tion about their diagno- and models. conducted sequentially
sis and medical history the design would be writ-
on the job. Do they dis- ten as: QUAN → qual,
close their head injury, and if so, to whom and such as in the case of interviewing only those indi-
how? If they do not disclose, what barriers to they viduals who dropped out of an experimental study.
perceive as keeping them from doing so? The For an in-depth discussion of this principle and its
investigator plans to use both questionnaires and use in several multiple method studies, see Morse
qualitative interviews drawing on both quantitative (2003).
Now, the investigator remains clear as to the datasets must be approached carefully, however, to
order of each of the methods (quantitative and avoid violating the integrity of the core method.
qualitative) and can make decisions that maintain First, information about how to convert datasets
the integrity of the core method. For example, the will be presented, followed by an example of the
investigator using teachers’ interviews to supple- decision making process involved when conduct-
ment direct observation and handwriting samples ing the actual analysis.
should gather the qualitative information at the Converting Datasets. In a primarily inductive
same time data are collected from the children. study, quantitative data from the secondary method
Even though it would be logistically easier to get should be “qualitized”—collected quantitative
all the teachers together once in a focus group for data types converted to narratives that can be ana-
the purposes of collecting information on all the lyzed qualitatively (Tashakkori & Teddlie, 2003,
study participants, to do so would introduce a pro- p. 9). This is the approach used in the example
blem of comparing two different timeframes for above when ordinal data were treated as a “code”
measuring the dependent variables. for sorting and examining information about disclo-
Recognize the Role of the sure attitudes. In the qualitized dataset, entire par-
ticipant interviews are coded in terms of a quality
Imported Models to the Project
(extent to which the physical impairment is obvi-
The core model of the project will determine the ous), which allows an investigator to sort all inter-
reasoning process of the entire project, but the sec- view data according to whether the individual had
ondary model must be understood to either supple- obvious physical impairments or not. Conversely, in
ment or inform the core model (Morse, 2003). a primarily deductive study, qualitative data from
Using the earlier examples of a study of disclo- the secondary method can be “quantitized”—that
sure in the workplace among individuals with TBI, is, qualitative data types are converted into categor-
suppose the investigator notices midway through ical or ordinal numerical codes that can be statisti-
the investigation that individuals who appear to cally analyzed (Tashakkori & Teddlie, 2003, p. 9).
have an obvious residual physical impairment An example can be taken from the handwriting
appear less reluctant to disclose. The investigator study mentioned earlier in which the investigator
thus decided to pursue the heuristic that persons can code the teachers’ open-ended interviews in
with less obvious impairment are more inclined terms of high or low self-confidence. A numerical
to “pass” as nondisabled. However, this requires code is assigned to the teachers’ reports (1 ⫽
some kind of quantification of the extent to which teacher reports child has high self-confidence; 2 ⫽
a visible physical impairment is present. The teacher reports child has low self-confidence) and
investigator chooses a simple ordinal rating of: the data are entered into a statistical software pro-
gram for analysis. Converting data not only serves
• No obvious physical impairment,
to reduce the number of datasets that must be han-
• Marginally obvious physical impairment, and
dled, but also integrates the datasets.
• Obvious physical impairment.
Maintaining Methodological Integrity When
The purpose of this secondary method is to fur- Working With Converted Datasets. In a mixed
ther explore what is being discovered about indi- methods study, an investigator must adhere to the
viduals’ perceptions regarding disclosure on the methodological assumptions of the core method
job. The project would be designated as QUAL → when deciding how to approach analysis. The
quan with an inductive drive and a subsequent col- implications of these decisions can be subtle, as
lection of quantitative data. illustrated in the TBI study mentioned previously.
In that study, the investigator has ordinal data for
Work with as Few Datasets as Possible
a subset of participants (remember, he realized
This principle refers to converting the datasets to midway through the study that some participants
forms that are consistent with the core method, may disclose based on how obvious their physical
when feasible. Converting datasets may not always impairments were), which he has qualitized. As a
make the most sense methodologically, especially result, for this subset of the sample, the investiga-
in sequential designs (secondary methods imple- tor can sort narrative information according to how
mented subsequent to the core method) described obvious the physical impairments are and analyze
below. However, in concurrent designs (core and the data to describe how visibility of physical
secondary methods implemented simultaneously) impairment interacts with the decision to disclose
converting datasets can be a powerful way to and other factors that influence thoughts about
approach analysis. That said, analysis of converted disclosure.
On the other hand, the investigator could decide Sequential Exploratory Design
to keep the ordinal data on the degree of residual
physical impairment as numeric and quantitize the A sequential exploratory design is identical to the
information on whether or not the person discloses one above (sequential explanatory design) except
(a dichotomous quantitative variable represented the sequence is reversed. The investigator first col-
by 0 ⫽ does not disclose and 1 ⫽ discloses) in lects in-depth and nuanced information about a
order to conduct a chi-square analysis. However, to phenomenon using a qualitative tradition followed
do so could easily violate assumptions of the in- by collection and analysis of quantitative data.
ductive theoretical drive by handling data analysis Again, the two models are analyzed and inter-
as though the purpose was deductive (i.e., testing preted separately. An example of a study using this
the hypothesis of whether variable x is related to type of design is a qualitative project to determine
variable y). An inductive approach seeks to under- the meaning of caregiving for spouses of individu-
stand more about how differing levels of physical als with dementia, followed by development and
impairment interact with other factors to affect the testing of a survey based on the results of the qual-
way persons talk about disclosing, not to test a itative core. The qualitative component informs the
hypothesis about a specific relationship. Further, interpretation of the psychometric findings of the
it is doubtful that number of informants in a survey, including the solution that best fits the fac-
qualitative study would be large enough to ade- tor analysis.
quately power a statistical test of the differences,
so assumptions of even the supplemental compo- Sequential Transformative Design
nent are violated. As with the two sequential designs described in
the preceding text, there are two separate and
subsequent phases of data collection and analy-
Design Types in Mixed sis (Creswell et al., 2003). However, in this in-
Method Designs stance, rather than the first model being the core
model, either model, qualitative or quantitative,
can be used as the core model. Moreover, the pur-
Creswell, Clark, Gutmann, and Hanson (2003)
pose of a sequential transformative design is to use
identify six major designs in mixed methods re-
a clearly identified theoretical perspective to direct
search. These six designs can be organized into
the research question toward change in policy,
two larger categories:
action, or ideology (Creswell et al., 2003). An
• Sequential designs, and example of a transformative design is an evalua-
• Concurrent designs. tion component of a service program with the main
purpose being feedback for improvement in the
Designs categorized as sequential introduce a service.
secondary method subsequent to the core method.
Designs in the concurrent category include sec-
ondary methods that are used simultaneously with
Concurrent Triangulation Design
the core method (Creswell et al., 2003). Each is Creswell et al. (2003) proposes that a concurrent
described in more detail below and summarized in triangulation design is the most familiar of all six
Table 25.1. types; it is often the design that comes to mind
when the term “mixed methods” is raised. As in all
concurrent designs, both qualitative and quantita-
Sequential Explanatory Design
tive data are gathered simultaneously, and results
In a sequential explanatory design, an investigator are validated by virtue of having been confirmed
first collects and analyzes core quantitative data through multiple data collection techniques.
used to explain or predict phenomena. This is fol- Neither tradition is designated as core or second-
lowed by collection and analysis of in-depth infor- ary, which frees the investigator to pursue interest-
mation through the use of a qualitative tradition. ing developments as they occur. The disadvantage
The two approaches are analyzed separately. is the need to make decisions that maintain the
An example is a survey design to describe leisure methodological integrity of both traditions simul-
performance patterns of adults who have survived taneously. Data from all sources are integrated in
a stroke that finds community access to be an the analysis phase of the study, when feasible. An
identified issue, followed by in-depth interview- example of a concurrent triangulation design is a
ing of these individuals to fully describe their study that compared a self-report of caregiving
experiences. strategies with an observation of caregiving in
Table 25.1 Summary of Mixed Methods Designs

Name Notation Description
Sequential explanatory QUAN → qual Explanation or prediction followed by in-depth
design description
Sequential exploratory QUAL → quan In-depth description followed by explanation or
design prediction
Sequential transformative QUAN → qual First-phase core method used to direct second-
design or phase change in policy or action
QUAL → quan
Concurrent triangulation Qual ⫹ quan Neither tradition is designated as “core” or
design or “secondary.” Data from each are collected
simultaneously.
Quan ⫹ qual
Concurrent nested QUAL ⫹ quan Either tradition is designated as core. Data from each
design or are collected simultaneously.
QUAN ⫹ qual
Concurrent transformative Either of the concurrent Data collected through use of both traditions
design notations above simultaneously. May or may not have a designated
core method. Purpose is to direct a change in
policy or action.
action (Corcoran, 2004a). In this study, caregivers Concurrent Nested Design

rated themselves on the Task Management Strategy
Index (TMSI) (Gitlin et al., 2002) which recorded A concurrent nested design differs from a triangu-
the caregivers’ report of frequency with which spe- lation design only in terms of the predominance of
cific strategies are used, such as “placing all items one tradition over another. In a concurrent nested
where they can be seen.” Caregivers were then design, the primary tradition determines how data
videotaped conducting a daily care task in which from the secondary tradition will be handled.
strategies on the TMSI may have been used. Creswell et al. (2003) propose that a nested
During a replay of the videotape, caregivers talked design can serve many purposes. Two different tra-
about their use of strategies in comparison to those ditions may be used to answer two different but
reported on the TMSI, and these interviews were related questions. Other studies may wish to meas-
recorded and transcribed. During data analysis, the ure aspects of the same phenomena at different
TMSI data were qualitized and interviews were levels. For instance, managerial focus groups may
coded according to the frequency of caregiver use be used for in-depth understanding of personnel
of strategies (above or below median for the sam- practices, but a survey is a better choice to describe
ple). The investigator then was able to sort accord- workers’ agreement with these practices. One very
ing to frequency of strategy use and examine both common use of concurrent nested designs is seen
the videotapes and interview data for these two in quantitative studies that illustrate a particularly
groups. In addition, the sample size was large salient finding with a case study.
enough that the investigator could also conduct a One example of concurrent nested design is
chi-square test to examine the relationship between particularly important for occupational therapy.
use of strategies and overall approach to care Like many professions, occupational therapy is
(quantitized data that emerged from the interviews challenged to validate practice with efficacy stud-
and videotapes). This provided an opportunity to ies and has been making good progress in doing
examine the data from multiple perspectives and to so. However, too little attention is paid to measur-
triangulate use of caregiving strategies from three ing the level at which the tested intervention is
sources, one quantitative (TMSI) and two qualita- actually delivered to study subjects as designed.
tive (videotape and follow-up interview). This tri- Several authors have promoted the use of treat-
angulation procedure served to strengthen the ment implementation, or treatment fidelity meas-
trustworthiness of the study by corroborating data ures (Burgio et al., 2001; Lichstein, Riedel, &
from several sources (Creswell, 1998). Grieve, 1994). Concurrent nested designs are very
useful for devising strategies that collect valid nested or triangulated design, but the overall pur-
treatment implementation data for the purposes of pose is to promote change in the entity being stud-
tracking the actual delivery, receipt, and enactment ied. Participatory action research is usually based
of an intervention as it was originally planned. on a concurrent transformative design as inves-
Many issues can develop in an intervention study tigators use the methods necessary to give all
that threaten to change the treatment actually being stakeholders a voice in identifying the problem,
tested. Problems with poorly defined protocols and developing a solution, and evaluating the outcome
lack of continual monitoring can result in a differ- of implementing the solution.
ent form of the original intervention being deliv-
ered by each interventionist, and even changing
over time as the interventionists gain more experi- Conclusion
ence. Thus, each subject receives a different ver-
sion of the original intervention plan. Further, In this chapter, mixed methods has been examined
subjects may actually receive and enact different as a flexible, yet rigorous approach to complex
versions of the intervention depending on their study problems that defy study with more tradi-
interpretation of what they are being told or shown. tional approaches. Mixed methods are used to
The result is that the investigator has little idea of strengthen the study and compensate for the weak-
what was actually being tested. nesses inherent in designs from both qualitative
A concurrent nested design was used to meas- and quantitative traditions. Topics studied as part
ure and enhance treatment fidelity in the Resources of occupational therapy seem well suited to the
for Enhancing Alzheimer’s Caregiver Health design types (sequential and concurrent) described
(REACH), a large-scale, multisite study involving in this chapter. Further, use of mixed methods is
family caregivers of individuals with dementia recommended as one way to ensure the treatment
(Schulz, Gallagher-Thompson, Haley, & Czaja, fidelity of occupational therapy interventions.
2000; Wisniewski et al., 1999). In that study, each
member of the research team was tested periodi- REFERENCES
cally for knowledge of relevant procedures (includ- Burgio, L., Corcoran, M. A., Lichstein, K. L., Nichols, L.,
ing those delivering the intervention), team Czaja, S., Gallagher-Thompson, D. E., Bourgeois, M.,
meetings were analyzed for level of understanding Stevens, A., & Ory, M. (2001). Judging outcomes in
regarding the intervention, and each interventionist psychosocial interventions for dementia caregivers: The
problem of treatment implementation. The
was evaluated on-site frequently to monitor adher-
Gerontologist 41, 481–489.
ence to the protocol. In addition, each study site Corcoran, M. A. (2004a). Understanding the care typolo-
was assessed in terms of the accuracy of the inter- gies of individuals providing care for a spouse with
vention manual and use of handouts to ensure sub- dementia. American Occupational Therapy Association,
jects actually received the intervention as designed. annual meeting, Minneapolis, MN.
Corcoran, M. A. (2004b). Caregiving styles and strategies.
Finally, subject enactment of the intervention was In K. Dokar (Ed.), Living with grief: Alzheimer’s dis-
assessed using a satisfaction survey that asked ease. Washington, DC: Hospice Foundation of
specifically the extent to which each component of America.
the intervention was used (Burgio et al., 2001). Creswell, J. W. (1998). Qualitative inquiry and research
design: Choosing among five traditions. Thousand
Although time-consuming and associated with
Oaks, CA: SAGE Publications.
some additional costs, the treatment fidelity Creswell, J. W. (2003). Research Design: Qualitative,
approach which used a concurrent nested design Quantitative, and Mixed Methods Approaches (2nd
was vital to accurately interpreting and replicating Ed). Thousand Oaks, CA: SAGE Publications.
the REACH intervention. Creswell, J. W., Clark, V. P., Gutmann, M. L., & Hanson,
W. E. (2003). Advanced mixed methods research
designs. In A. Tashakkori & C. Teddlie (Eds.),
Concurrent Transformative Design Handbook of mixed methods in social and behavioral
research (pp. 209–240). Thousand Oaks, CA: SAGE
As with a sequential transformative design, the Publications.
Denzin, N. K. (1978). The logic of naturalistic inquiry. In
purpose of a concurrent transformative design is
N. K. Denzin (Ed.), Sociological methods: A source-
use of a theoretical perspective to enact change in book. New York: McGraw-Hill.
a group or organization (Creswell et al., 2003, Gitlin, L. N., Winter, L., Dennis, M., Corcoran, M.,
p. 230). Choices about the predominant tradition Schinfeld, S., & Hauck, W. (2002). Strategies used by
and whether methods are nested or triangulated are families to simplify tasks for individuals with
Alzheimer’s disease and related disorders:
made based on the degree to which the theoretical Psychometric analysis of the task management strategy
perspective is facilitated. Thus, a transformative index. The Gerontologist, 42, 61–69.
design may take on the characteristics of either a Greene, J. C., & Caracelli, V. J. (1997). Advances in
Mixed-Method Evaluation: The Challenges and Handbook of dementia caregiving interventions. New
Benefits of Integrating Diverse Paradigms (New York: Springer.
Directions for Evaluation, No. 74). San Francisco: Tashakkori, A., & Teddlie, C. (1998). Mixed methodology:
Jossey-Bass. Combining the qualitative and quantitative approaches
Howe, K. R. (1988). Against the quantitative-qualitative (Applied Social Research Methods, No. 46). Thousand
incompatibility thesis or dogmas die hard. Educational Oaks, CA: SAGE Publications.
Researcher, 17, 10–16. Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed
Lichstein, K.. L., Riedel, B. W., & Grieve, R. (1994). Fair methods in social and behavioral research. Thousand
tests of clinical trials: A treatment implementation Oaks, CA: SAGE Publications.
model. Advances in Behavior Research and Therapy, Vogt, P. W. (1993). Dictionary of statistics and methodol-
16, 1–29. ogy. Thousand Oaks, CA: SAGE Publications.
Morse, J.M. (1991). Approaches to qualitative-quantitiative Wisniewski, S., Belle, S., Coon, D., Marcus, S., Ory, M.,
metholdological triangulation. Nursing Research, Burgio, L., Burns, R., & Schulz, R. (1999). The
40(2), 120–123. resources of enhancing Alzheimer’s caregiver health
Morse, J. M. (2003). Principles of mixed methods and (REACH) project design and baseline characteristics.
multimethod research design. In A. Tashakkori & C. Psychology and Aging, 18, 375–384.
Teddlie (Eds.), Handbook of mixed methods in social
and behavioral research (pp.189–208). Thousand Oaks,
CA: SAGE Publications.
RESOURCES
Newman, I., & Benz, C. R. (1998). Qualitative-quantita- DePoy, E., & Gitlin, L. N. (2005). Introduction to
tive research methodology: Exploring the interactive research: Understanding and applying multiple strate-
continuum. Carbondale, IL: Southern Illinois gies (3rd ed.). St. Louis: C. V. Mosby.
University Press. Newman, I., & Benz, C. R. (1998). Qualitative-quantita-
Newman, I., Ridenour, C. S., Newman, C., & DeMarco, tive research methodology: Exploring the interactive
G. M. P. (2003). A typology of research purposes and continuum. Carbondale, IL: Southern Illinois
its relationship to mixed methods. In A. Tashakkori & University Press.
C. Teddlie (Eds.), Handbook of mixed methods in Schwandt, T. A. (2001). Dictionary of qualitative inquiry
social and behavioral research (pp. 167–188). (2nd ed.). Thousand Oaks, CA: SAGE Publications.
Thousand Oaks, CA: SAGE Publications. Tashakkori, A., & Teddlie, C. (2003). Handbook of mixed
Schulz, R., Gallagher-Thompson, D. E., Haley, W., & methods in social and behavioral research. Thousand
Czaja, S. (2000). Understanding the intervention Oaks, CA: SAGE Publications.
process: A theoretical/conceptual framework for inter- Vogt, P. W. (1993). Dictionary of statistics and methodol-
vention approaches to caregiving. In R. Schulz (Ed.), ogy. Thousand Oaks, CA: SAGE Publications.
S E C T I O N 7
Conducting Inquiry
C H A P T E R 2 6
Organizing the Components of Inquiry

Together: Planning and Implementing a
Coherent Study and Research Tradition
Gary Kielhofner • Marcia Finlayson • Renée R. Taylor
Implementing high-quality research is a multifac-

eted and challenging process. It requires planning,
Traditions of Inquiry
resources, careful implementation, documentation, Conducting high-quality research is the outcome
and storage of data, analysis, and dissemination of of a developmental process (Case-Smith, 1999) in
findings in presentations and publications. Even a which a variety of research strategies are inte-
modest research project will take upwards of a year grated over time into a program of research. The
to implement from beginning to end. Typically, following are some key elements of this process:
publication requires a year or more from the time a
paper is submitted until it appears in the literature. • Establishing a theoretical framework that frames
Therefore, conducting a quality study from the research,
planning to publication is ordinarily no less than a • Refining the methods used in the research so that
2-year commitment. Most research projects extend they become more sophisticated and suited to the
much longer, with 3 to 5 years being the typical topic under study,
funding period for implementing major federally • Establishing a track record in a defined area of
funded research projects, excluding dissemination research (through regular publication and dis-
efforts after the project is officially closed. Pub- semination of findings),
lication of a typical research article culminates a • Developing ongoing relationships with research
process that began years before with the initial collaborators and sites, and
planning of the research. By the time findings are • Securing funding to support the research.
in print, a great deal of time and effort has been When viewing research from a developmental
expended. perspective, the most productive context for re-
Research involves commitment over a substan- search is a tradition of inquiry (Hammel, Finlayson,
tial period of time. Before embarking on a research Kielhofner, Helfrich, & Peterson, 2002; Helfrich,
project (even if it involves collaborating with one Finlayson, & Lysack, in press; Kielhofner,
component of a study) one should have an appre- Hammel, Helfrich, Finlayson, & Taylor, 2004;
ciation of what the overall process entails. Taking Taylor, Fisher & Kielhofner, 2005) A tradition of
responsibility for full collaboration or leadership inquiry ordinarily encompasses the following
in research requires a firm grasp of everything elements:
involved, something that takes training, mentor-
ing from a seasoned investigator, and substantial • A line of inquiry defined by theoretical, substan-
experience. tive, and/or methodological interest, and
The purpose of this chapter is to provide an • A consistent stream of funding that supports the
overview of the major processes that are required research.
to organize and complete a study from beginning A tradition of inquiry is ordinarily led by (a)
to end. The steps in research that are overviewed in principal investigator(s) and co-investigators and it
this chapter are covered in detail in the subsequent involves the sustained effort of a team of research
chapters. Our aim is to give the big picture of what staff, including graduate students who are part of
a study entails from the time it is planned until the university-based research teams, community
findings are disseminated. In addition, this chapter members, consulting or collaborating researcher
places the individual research study in the larger from other organizations, and external agency per-
context of a tradition of inquiry. sonnel who are involved in the research.
Chapter 26 Planning and Implementing a Coherent Study and Research Tradition 421
A research tradition involves pursuing a line Each of these processes is discussed separately
of inquiry over an extended period of time, so below. However, it should be noted that they are
that those involved are continuously improving not necessarily sequential steps. Rather, as shown
their skills and expertise. in Figure 26.1, they are
Within traditions of re- part of an interactive
search, a given study A research tradition involves process with each ele-
never really stands alone. pursuing a line of inquiry ment helping to refine
As part of a tradition of the other. For instance,
research each study picks over an extended period of one may begin with a
up where a previous time, so that those involved very broad research
study has left off, gener- question, which guides
ating new findings and are continuously improving the literature review. As
new questions that inform their skills and expertise. one explores the litera-
and shape future lines of ture, the research ques-
inquiry. A tradition of tion may focus or go off
research takes many years to establish and typi- in a somewhat different direction. Similarly, as one
cally extends over the entire career of a researcher. examines the methodological implications of ask-
ing a question, it may become apparent that the
question is too broad or otherwise needs to be
The Need for Planning altered.
All studies, no matter the size and scope of the The Literature Search
research, require a strong organizational plan. The A literature review involves the systematic and
planning process includes: replicable process that identifies, evaluates, and
• Searching the literature, interprets an existing body of recorded work on a
• Identifying research questions, and particular topic or question. When one is planning
• Deciding the research methodology. an inquiry, the literature review serves two key
objectives that include:
The Research Tradition as a Context • Identifying the current level of theory and knowl-
for Learning Research edge development, as well as gaps and areas of
inconsistency, and
Research traditions are ideal learning contexts • Identifying the research methods that are com-
for graduate students who function within them monly used, and their strengths and limitations.
as research apprentices (Hammel et al., 2002).
Students learn research by participating in actual By serving these objectives, the literature
research procedures. Moreover, they typically review allows the researcher to develop a sound
have an opportunity to participate in informal rationale for his or her current project that is
interactions and to see how research actually grounded in and builds upon existing knowledge.
unfolds. They gain an appreciation of what is When planning a study, an often unappreciated
being learned through the research and the secondary outcome of a good literature review is
inherent logic of the process. the identification of experts in the topic area that
The research tradition is also an ideal frame- could be approached for consultation if that is
work wherein students can select research topics,
deemed to be appropriate or necessary as the pro-
and complete masters’ theses and doctoral dis-
sertations. Rather than being expected to identify ject develops.
and pursue “original and independent” research, A good literature review will be planned ahead
students working within such traditions engage of time, and guided by a specific question or series
in research that emanates from the ongoing of questions. The search itself will be done in a
research tradition. In this way, the directions for systematic and methodical manner, taking into
the research and the questions or hypotheses to account the searching nuances and strategies rele-
be examined emanate from the previous round of vant to the particular search method that is being
research. Moreover, within traditions of research used. There are four basic methods of searching
students learn not only from their research men- the literature:
tors, but also from project staff and peer students
who have complementary knowledge and skills • Using electronic bibliographic databases such as
working. MEDLINE, CINAHL, PSYCHinfo, ERIC, Sci-
ence Citation Index, the Agency for Healthcare
422 Section 7 Conducting Inquiry
Formulating the Research

Reviewing the Literature: Question:
Identifying what is known about Deciding what the study will be
the topic and what gaps in designed to reveal
knowledge exist
Deciding the Research Methods:

• Design
• Sampling
• Data collection and analysis
Figure 26.1 Mutually interactive components of planning a study.
Research and Quality Clinical Guidelines and or by other people. The initial steps in conducting
Evidence Reports, British Medical Journal’s a good literature review include:
Clinical Evidence, and the Cochrane Database of
Systematic Reviews, AgeLine, and many others, • Writing a clear and focused question,
• Manually searching through specific journals, • Identifying the key concepts and relationships
either hard-copy or through online tables of con- embedded within the question,
tents, for articles that are of interest, • Identifying the best search method or methods to
• Reviewing articles listed in the reference lists of use to address the question (e.g., electronic bibli-
articles that were previously located on the topic ographic databases and hand searching),
of interest, and • Initiating the search, working through one
• Searching the World Wide Web. method at a time, and
• Refining the search as needed, documenting
It is important to realize that searching an elec- changes in the question, terms, or method being
tronic bibliographic database and searching the used.
WWW is not the same thing. Databases are com-
pilations of published research, scholarly articles, When searching the literature, one should be
books, government reports, newspaper articles, open to:
and so forth. There are many different databases, • The work of researchers in other disciplines, as
each with its own focus, purpose, and primary they may have interesting new approaches or
audience. For example, CINAHL indexes both ideas that will inform the work that is being
peer-reviewed and “gray” literature (e.g., disserta- planned, and
tions) from nursing and allied health, and focuses • Literature that is more than 8 to 10 years old.
on serving this audience. Most electronic biblio-
graphic databases are international in scope, and as Some people consider reading older literature
a result, will provide publications that are written irrelevant, because it is out of date and potentially
in English as well as other languages. In compari- inaccurate. However, limiting searches to the most
son, the WWW is a network of documents that are recently available work risks that important, clas-
made available through the Internet. While some sic pieces may be missed. Having a solid under-
search engines can do complex searches, there are standing of classic works is often integral to
no standard search terms to use or thesauruses to developing and refining ideas. Finally, reviewing
assist when relevant materials are not being identi- the literature is not a once-only activity. Keeping
fied. Doing a WWW search may result in locating up with the literature and new developments in a
a relevant article, but many irrelevant materials field is a critical part of building and organizing a
will be found as well. In addition, it is often diffi- program of inquiry.
cult to evaluate the credibility of a Web site, given
that there are no restrictions as to who can post
Identifying Research Questions
materials on the Web.
Regardless of the method used, a good litera- All research sets out to generate new knowledge
ture review will be clearly documented, step by that fills a current gap. Such questions may have
step, so that it is reproducible on subsequent days their origins in clinical experience. They might
emerge from the literature. Or, they may have their nificance, and timeliness of the research ques-
origins in the findings of previous research. Articu- tion/topic. Depending upon the objectives and
lating and refining the research question involves scope of the study, this may involve:
identifying: • Talking to people who have done research in the
area in order to receive their input about how to
• What it is that is not currently known, and
best formulate the problem (this can range from
• What the investigation will address.
getting direct supervision or consultation from
Research questions generally start out broad expert researchers in the topic area),
and are narrowed over time. For instance, an inves- • Discussing the topic with practitioners or
tigator might begin with the following types of consumers in order to make sure the question
questions: has relevance and significance to contemporary
practice,
• Why do so many persons who are hospitalized • Obtaining information from public policymak-
with serious mental illness tend to be rehospital- ers and potential grant funding agencies regard-
ized? ing their perspective on the significance,
• What kinds of characteristics predict which per- relevance, and timeliness of a given research
sons will be more successful following rehabili- question (within the field of occupational ther-
tation? apy, these may include members of public advi-
• What is the personal experience of persons fol- sory boards, clinical organizations, hospital
lowing a cerebrovascular accident? administrators, self-help or advocacy-based
• What differentiates clients who tend to be moti- organizations, and governmental officials that
vated to get the most out of therapy and those provide funding for research and/or services),
who lack motivation? and
The process of formulating a research question • Presenting an early version of the question to
involves going from such broad formulations of get feedback from others.
the question to something that is much more spe- Developing a research question should be a
cific and can be addressed in a single study. For public process that benefits from the input of peo-
example, the first question noted above might be ple who know something about the problem from a
narrowed into one of the following questions: research and/or practical perspective. Consulting
• Is functional level related to the frequency of with such people enhances the likelihood that the
hospitalizations over a 3-year period? study will address an important and relevant ques-
• Do persons who have family support have a tion. Participatory approaches to research (see
lower rate of hospitalization in a year than those Chapters 38 to 40) stress the importance of involv-
without family support? ing stakeholders who will be influenced by or who
would be expected to be consumers of the research.
As these examples illustrate, choosing a Carefully deciding the research question is
research question means that one must select an worth all the effort and time one can give it, since
aspect of the broader it influences all subse-
question being studied quent decisions and pro-
that is manageable. All research sets out to gen- cedures and ultimately
Every research question shapes the worth of the
has costs in terms of re-
erate new knowledge that study. Moreover, choos-
sources and time needed fills a current gap. ing a research question
to generate a developing involves creativity. Some
answer. Thus one should of the best research is
formulate the research question with an eye toward based on figuring out a new way to approach a
what will be feasible in the study being planned. problem or finding a different way to ask the ques-
Formulating a research question usually tion others have asked. Taking time to consider
involves the following steps: information from the literature and from others’
perspectives is a wise investment for any research
• Reviewing the literature (as noted above) in order
project.
to identify what is already known about the topic,
what type of questions have been asked in previ-
ous research on the topic, and how people have
Deciding the Research Methodology
gone about asking those questions, and In deciding the methodology, one first needs to
• Consulting with others about the relevance, sig- decide whether the study will be quantitative, qual-
itative, or a combination of these two methodolog- lowing are some examples of the types of more
ical approaches. Once one has decided the broad detailed questions that will be asked in further
type of methodology the study will employ, there delineating the study design:
are four critical methodological decisions:
• In the case of a longitudinal survey, how many
• Design, times will participants be asked to respond and at
• Sampling, what time points?
• Data collection, and • If the study is an experimental study, how many
• Analysis. different experimental conditions will be com-
Each is briefly examined below. Although dis- pared and what will be the control condition?
cussed separately, these four aspects of the meth- What is the feasibility of assigning subjects to
ods must be considered together. conditions randomly?
• If the study is qualitative and if it uses key
informants, how will these be selected?
Design
Most often, designing the study involves a
Choosing a research design requires the investiga- number of iterative stages in which one:
tor to decide how the research will be structured to
answer the research question. Ordinarily, selecting • Examines how other investigators have appro-
the study design involves several layers of decision ached the question one plans to address,
making. As noted earlier, one first decides whether • Creatively considers ways to address the ques-
the overall design will be qualitative or quantita- tion,
tive or combine facets of both. Then, one must • Compares ideas about the design to expected
select the details of the design. For quantitative resources and research conditions to make sure
studies one has to consider whether the overall the design is realistic, and
design will be: • Seeks consultation and feedback from other
investigators and from persons who can provide
• A descriptive one in which quantitative data are information about logistics.
collected to answer a broad question that charac-
terizes a selected group or that explores relation- This same iterative process also applies to deci-
ships between variables, sions about sampling, data collection, and analysis
• An experimental design (e.g., clinical trial) in since they are all interrelated.
which a variable will be manipulated to ascertain
its effect on participants, or Sampling
• A prospective longitudinal or follow-up design in
Sampling refers to decisions about who will par-
which data on a variable will be collected over
ticipate in the study and how they will best repre-
time to determine how it changes naturally or in
sent the larger population of interest. Deciding
response to some event.
how to sample is a fairly complex process involv-
In the case of a qualitative study, the following ing several parameters. It involves deciding:
types of broad design options are considered:
• Who are appropriate potential subjects for the
• Whether the study will involve naturalistic obser- study (inclusion/exclusion criteria),
vation or participant observation, • How participants will be chosen for the study
• The number of different settings that will be (selection),
examined in the study, and • How many participants will be required for the
• How data collection and analysis will be coordi- study (sample size),
nated over time. • What will be required of participants (participant
burden), and
After such first level decisions are made then
• How participants will be allocated to different
more detailed decisions must be made about the
conditions (assignment) if there are different
study design. At each level, the considerations that
conditions or situations on which subjects are
guide the design choices represent consideration
compared.
of issues of rigor as they are conceptualized in the
qualitative and/or quantitative traditions along Inclusion/exclusion criteria specify who will be
with practical logistic considerations (e.g., what is included in the study and who will not. Parti-
possible to do in the study given constraints of cipants are included based on the question to be
resources, contexts, and other factors that influ- asked. That is, if one wants to know about the
ence the realistic parameters of choice). The fol- experience of persons following cerebrovascular
accident (CVA), then inclusion criteria might be ethical; that is, what is asked of subjects has to be
persons who have experienced a CVA in the past 6 weighed against the potential benefit of the study
weeks. In occupational therapy studies, factors that to the participants and to society in general. If the
are generally considered in inclusion criteria are participant burden cannot be justified by the poten-
the diagnosis/impairment, ethnicity, and age. tial benefit then the study cannot be considered
Exclusion criteria are generally decided on the ethically warranted. Second, the more that is asked
basis of eliminating factors that may confound the of subject, the more difficult it may be to recruit
research findings and excluding those who would and retain subjects in the study.
not be capable of participating in the research
protocol. Returning to the preceding example, an Data Collection and Analysis
exclusion factor to eliminate confounding vari-
ables might be having another physical or psychi- The answer(s) to the research question(s) of the
atric disability since the experience of a second study are dependent on the quality of data collected
disability would be difficult to sort out from the and how they are analyzed. Decisions about these
experience of the CVA. If the same study involved two aspects of the study methods go hand in hand.
extensive interviewing of participants about their First of all, the type of data collected and the type
experiences, then one exclusionary criterion would of analysis will depend on whether the study is
be persons with expressive aphasia. qualitative and/or quantitative in nature. Qualitative
The method of selecting participants for a study data are ordinarily collected through observations
depends on the study method and design. For or interviews that are recorded through field notes,
example, a qualitative method generally uses a audiotapes, or videotapes. Quantitative data are
purposive sampling strategy that seeks out persons collected through observation, interview, and self-
who are knowledgeable about the topic under report forms that are generally recorded on paper or
study (i.e., key informants). In contrast, a large- on computer.
scale survey may seek a sample that is randomly Investigators are always concerned with the
selected from the defined population so that find- dependability of data, although this issue is con-
ings can be generalized. Subject assignment is a sidered differently within the qualitative and
concern when the study is an experiment. Random quantitative traditions. Briefly stated, qualitative
assignment is the most rigorous strategy in this sit- concerns include such things as whether the expe-
uation to equalize the distribution of factors that rience and circumstance of those studied is faith-
might confound a research question across the fully represented while quantitative concerns are
groups that are being studied. ordinarily couched in terms of data reliability (i.e.,
Sample sizes can vary widely and depend on stability over time, method of data collection, and
the design of the study. Typically, qualitative re- data collector) and validity (assurance that the data
search involves fewer participants from whom a represent the intended concept). Most strategies for
great deal of information is collected over time. ensuring data dependability in qualitative research
Some quantitative designs may involve large sam- are dependent on how the investigator goes about
ples of participants who are contacted only once gathering the data. Quantitative data dependability
(e.g., to complete a written survey or respond to a may be based on previous research that has demon-
structured interview), and in others (e.g., longitu- strated reliability and validity of the preexisting,
dinal studies) subjects may be contacted several standardized data collection tools that are chosen
times and evaluated using measures of the same for the study. In cases where this type of research
variables over a period of time. In some cases the is not available on the data collection method cho-
sample size will be determined by how many sub- sen or where a data collection method has to be
jects are necessary to achieve a statistically signif- created for the study, data on the reliability and/or
icant finding. Experimental and longitudinal validity of the instrument may be gathered as part
studies also have to account for attrition, or the of the study itself or in a previous pilot study.
likelihood that some subjects will drop out of the Data analysis also depends on whether the
study, relocate, die, or go missing for some other study is qualitative and/or quantitative. Qualitative
reason. In this case, a power analysis (as discussed data analysis ordinarily involves coding and classi-
in Chapter 17) may be done to decide the number fying data to identify various themes that make up
of subjects that are necessary. the findings. Qualitative data analysis tends to be
Participant burden is generally defined as how more “organic.” That is, the analysis is partly
much effort is required of the subject or how much shaped by the research question and partly shaped
the subject will be subjected to. Participant burden by the nature of the data itself. Moreover, data col-
is a concern for two reasons. The first reason is lection and analysis in qualitative research is often
a spiral process in which analysis of initial data Descriptive statistics, such as measures of skew-
shapes the kind of data that are collected next and ness, kurtosis, variability, and visual analysis of a
the new data result in further analyses, and so on. scatterplot, can be used to determine whether data
Quantitative data analysis involves descriptive and for a given group of individuals are normally dis-
inferential statistics. How the data will be analyzed tributed. If data are not normally distributed or if
depends on the type of data collected (nominal, the sample size is small, then nonparametric statis-
ordinal, ratio, or interval), the research question, tics can be used.
the sample size, and how the data are distributed. Generally speaking, inferential statistics can be
Descriptive statistics can serve a number of subclassified in terms of univariate and multi-
purposes. For example, they can be used to variate statistics. Univariate statistics are used to
describe the sociodemographic characteristics analyze a single dependent variable. Multivariate
(e.g., sex, age range, and ethnicity) of a sample or statistics are used to test the effects of one or more
to describe how a sample has scored or performed independent variables on more than one outcome
on a given measure. For example, consider a variable. Considering the study example above, if
researcher who has obtained a sample of 60 a researcher wanted to compare the group of
women, 30 of whom have had a C-3 to C-6 spinal women with spinal cord injury to the group of
cord injury and 30 of whom have had a severe women with CVA in terms of fatigue severity and
CVA. If the researcher is interested in describing hypothesized that the women with CVA would
fatigue severity in the two groups of individuals report more severe fatigue levels, the researcher
from a quantitative perspective, the researcher would first check if the fatigue severity scores are
might utilize a measure of fatigue severity that pro- normally distributed in both groups and then likely
vides a continuous fatigue score ranging from 0 use a t-test to compare the mean fatigue severity
(low) to 30 (high) and then provide statistics that scores between the two groups.
show the variation and central tendency of fatigue
in each of the two groups.
Integrating the Planning Components
Descriptive statistics are most commonly
reported in terms of frequency and percentage data As noted earlier, the three components of planning
(for categorical variables) or in terms of means and a study (literature review, formulating the ques-
standard deviations (for continuous variables). tion, and deciding the methods) are interrelated
Depending on the research question, other descrip- and influence each other. The kind of question one
tive statistics that may be presented include medi- formulates will have implications for the literature
ans, ranges, or modes. Descriptive statistics can search and research methodology. Moreover, the
also be presented visually in graphs such as his- state of the literature and available methodological
tograms, scatterplots, or boxplots. resources for implementing research will influence
Statistics from the study sample can be com- the types of questions that can or should be asked.
pared to those from other sample groups within Consider, for example, the question posed ear-
the same study, or to statistics from a normative lier about the personal experience following cere-
reference group. If a researcher is interested in bral vascular accident. Such a question will have
comparing scores between two groups, inferential certain implications for the research design
statistics can be used. Inferential statistics allow depending on the state of the literature. If the
the researcher to test a given hypothesis—or an- literature indicates there is little to no information
swer a research question that is comparative in to answer this question, then a qualitative,
nature, or probes the efficacy of a given interven- exploratory design, or a quantitative pilot study
tion. In this case, the sample size and how the data that gathers information from a small sample is a
are distributed must be considered. good starting point. When such studies already
For example, a direct comparison of means exist, they may provide sufficient information for
between the group of women with spinal cord creating a large-scale survey research design. Such
injury and the group with CVA assumes that the a study might enlist a larger sample of persons to
data from the two study groups have adequate vari- see how much variability there is in their experi-
ability and are normally distributed. Whether data ence and then identify what demographic factors
are distributed normally will, in part, be based on are associated with different experiences.
the sample size. Larger sample sizes (*25 sub- Similarly, if one explores the question about
jects) are more likely to be normally distributed what accounts for recidivism and finds out little is
than smaller sample sizes. However, sample size known, then a qualitative research study may be
alone does not always predict normal distribution. indicated. However, if one finds out there is a large
literature and quite a bit is known about a given tant to think ahead about the funds and other
topic, such as what causes rehospitalization in per- resources that may be required to implement the
sons with mental illness, then the investigator may research.
decide that the question needs to be reformulated One important resource for any study is space.
to ask: “Can the rate of rehospitalization be Even if the study is done in a natural setting, there
reduced by offering follow-up services?” This type needs to be a place where the investigator can store
of question is best answered by a control group the data for the study and work on data analysis.
design in which some persons receive the service Most studies require space for such things as:
and others do not. • Housing research staff and equipment necessary
for the research, and
Writing the Research Plan • Conducting any procedures that are part of the
As one develops the research questions, completes research (e.g., experimental procedures, staff
the literature review, and plans the methodology of meetings, interviews with subjects).
a study, it is typical that a research plan is written It is important that space not only be adequate
up. Writing up the plan helps to organize and in size for the study but also that it can ensure con-
record one’s thinking processes. A research plan is fidentiality of subjects (e.g., private rooms for data
ordinarily required before one can go on to imple- collection and storage) and appropriate execution
ment research, since it is the basis for individuals of any research procedure (e.g., quiet space for
or bodies who approve research (e.g., the research testing). In most cases research is conducted in
advisor in the case of research undertaken by a existing space, so planning for the research
student or trainee, the ethical review board that involves requesting and securing permission for
approves the study on ethical grounds when human use of the space (exclusive or shared as needed) for
subjects are involved). Writing a good research the duration of the research. Space is ordinarily
plan not only helps one to systematically think provided by the agency or organization sponsoring
about the study, it also provides some of the basic the research, but sometimes it is necessary to rent
material that will be used later in writing a report space for a study, in which case it can become part
of the research findings. of the budget.
A research plan ordinarily includes: Other resources for research ordinarily are pur-
• An introduction that indicates the general nature chased specifically for the study. Therefore they
of the study and the rationale, or why it is impor- fall under the process of planning and securing a
tant, budget for the research. In any research project the
• A review of the literature that indicates what is typical costs are:
already known in the area and that demonstrates • Personnel,
the need for the question of the study to be asked, • Equipment,
• A statement of the overall aim(s) or objectives of • Supplies,
the study, • Subject reimbursement,
• An indication of the specific questions to be • Travel, and
answered and/or hypotheses to be tested, and • Subcontracts.
• A description of the planned methodology of the
The kinds of personnel who make up a research
study including design, sampling strategy, data
project are discussed below. Depending on the set-
collection, and data analysis.
ting and size of the research project, some or all of
The actual format and length of the research the salaries of personnel involved in the study may
plan will depend on a number of factors including be paid. Projects that require several persons ordi-
the type of research planned and the intended audi- narily are supported by grants. Grants generally
ence for the plan. pay:
Planning and Budgeting • Part of the salary of some persons who have other
responsibilities,
for Necessary Resources
• The full salary of persons who are employed
Every study has financial implications. The cost of solely for purposes of conducting the research,
studies may range from small investigations that and
cost a few hundred dollars to large funded studies • Consultant or subcontract fees to people who are
that require millions of dollars to implement. No performing specific, limited tasks related to the
matter what size the planned research, it is impor- study.
Equipment generally refers to objects that are pices the research will be conducted. Intramural
durable while supplies refer to things that con- grants are generally smaller than extramural
sumed by use (e.g., paper, office items). However, grants. The decision about funding may be made
different granting agencies may specify different by an administrator or a committee who reviews
ways of deciding what is equipment and what are that grant proposal and determines whether it mer-
supplies. Within occupational therapy research, its funding. Intramural proposals may or may not
typical equipment needed for a study may range be competitive (i.e., when several grant proposals
from motor and cognitive testing materials, to as- are considered simultaneously and only the best
sistive devices, to therapeutic tools or toys, to ones are funded).
splints, to highly specialized instruments that meas- Extramural funding is provided by an agency
ure muscle strength, physical endurance, range of outside the one where the researcher is employed
motion, or pain sensitivity. For qualitative research or where the research will take place. There are
items such as tape recorders, transcription ma- two basic types of extramural funding agencies:
chines or video recorders may be needed. Other private foundations and public (state or federal)
typical research equipment includes computers, fax agencies. Both types of agencies generally have
machines, printers, telephones, and items used to published priorities indicating the type of research
enter and store data such as file cabinets. Supplies they wish to fund and both types of agencies typi-
generally encompass office supplies, printing sup- cally have deadlines for submitting research pro-
plies, study brochures, postage, stationery, and posals since their funding is generally competitive.
software packages. The largest funder of research in the United States
Many studies, particularly those that do not is the federal government. Since most agencies
offer a service, reimburse subjects for the time, fund only a small proportion of the proposals they
effort, or discomfort associated with participa- receive, a typical researcher will have to prepare
tion. The extent of reimbursement depends on substantially more grant proposals than projects
what the subjects are being asked to do. Subject that are actually funded.
reimbursement can be used as an “incentive” for A grant always includes a research plan. It also
subjects, but ethical concerns require that the typically includes the other following elements:
amount of reimbursement not be so much that
subjects feel compelled to participate in research • A management plan that includes discussion of
in which they would not otherwise chose to parti- the personnel necessary to implement the study,
cipate. the objectives and tasks necessary to conduct the
Travel can include any costs incurred in con- study, and a timeline for their completion,
ducting or reporting findings for a study. Travel • A budget indicating the necessary resources and
may be the cost of investigators or data collectors justifying the proposed expenditures (most or all
traveling to subjects’ homes. On the other hand, of which the granting agency will be asked to
travel may include the cost of reimbursing subjects provide and some of which may be provided by
to travel to the research site. The cost of investiga- the institution that houses the grant),
tors traveling to scientific and/or professional con- • An evaluation plan that indicates how the
ferences to present the research findings is also overall quality of the proposed project will be
generally considered part of the cost of doing assessed,
research. • Evidence that ethical approval has been obtained
Subcontracts are chunks of larger budgets that or is being sought (when human subjects are
can be conceived as “sub-budgets.” Based on a involved), and
contractual agreement with a collaborator from a • Appendices that include such things as letters of
different institution, subcontracts are generally commitment from individuals and organizations
used to fund staff and research activities that occur who will be essential to the study implementa-
outside of the main institution that houses the tion, copies of instruments or procedures that
research. will be used to collect and analyze data, and pro-
tocols describing procedures that will make up
Writing a Grant experimental and control conditions in experi-
mental studies.
The most typical way of obtaining resources for a
study is to prepare a grant proposal. Proposals may Depending on the granting agency and the
be intramural or extramural. An intramural pro- nature of the study, a grant application may be only
posal is one submitted to the agency in which the a few pages long or may contain several hundred
investigator is employed and/or under whose aus- pages of material.
Management Plan • An explanation of the research, what it will

address, why the research is needed/justified, and
Every research project requires a management how the research will be done, especially how
plan that outlines: subjects will be recruited, selected, and what will
• The major tasks necessary to complete the be required of them,
project, • An explanation of how subjects will be informed
• Who is responsible to accomplish those tasks, about the study and how their consent will be
and obtained, and
• When the tasks will be completed. • Letters that will be used to obtain/document sub-
ject informed consent.
Generally the management plan is organized
around the major objectives of the study, breaking Investigators cannot begin the process of
down the tasks that are necessary to achieve those recruiting subjects until ethical approval is ob-
objectives. Table 26.1 shows a section of a man- tained. This process can take several weeks or
agement plan from a research project, Pathways to months depending on the institution and the nature
Self Determination (H133-G020-217). This con- of the study, so it should be set in motion as soon
trol group study examines the effectiveness of an as the investigator has finalized the plan of the
occupational therapy intervention in helping par- research.
ticipants achieve independent living and employ-
ment. Since research projects unfold over long Personnel (Staff) and Organizational Plan
periods of time, the management plan is a useful
way to monitor progress to make sure that critical Some research projects are simple enough to be
tasks are being done on time. conducted by a single person, without the support
of additional staff. However, most research is suffi-
Obtaining Ethical Review ciently complex to require a diversity of research-
related skills and the efforts of several personnel.
Any study that involves human beings who partic- To get the work done in the most accurate and effi-
ipate in the research as subjects should undergo cient manner, it is often necessary to take a close
ethical review. In most countries it is required look at the tasks that need to be completed, the
before a study funded by the government can be skills necessary to complete these tasks well, and
undertaken. Moreover, institutions that routinely then identify what type of person can achieve them
receive government support for research are the best.
required to have all their human research ethically An array of different tasks is required for any
reviewed whether or not it is funded. given study. There is a research plan or proposal to
Ethical review is a process that is designed to be written, including the analyses sections. There
protect those who participate as subjects from any may be a grant application to be completed, in-
harm and to ensure that their effort and any risk cluding the budget and budget narrative. An IRB
involved is warranted by the importance of the application must be prepared and submitted if
study. A fundamental principle governing ethical human subjects are to be involved. Data must be
conduct of research involving human beings is collected and managed in an appropriate and con-
informed consent. The elements of informed consistent way, and then they must be analyzed, inter-
sent are: preted, and prepared for dissemination.
Many people may be involved in such a series
• That the potential subject understands the pur-
of efforts. Table 26.2 summarizes some of the key
pose of the research, what is involved in being a
players on a research team, and their typical roles
subject of the study, and whether there are any
and responsibilities. Not every project will need
potential risks to being involved, and
all of these players. On the other hand, some proj-
• That permission to participate is given freely
ects may need multiple individuals within any of
without any form of coercion.
these categories. Regardless of size or setting of a
Institutions in which research is routinely con- research project, the same basic tasks must be
ducted (e.g., universities and large medical or accomplished. Consequently, it is important when
rehabilitation centers) maintain ethical boards planning a program of inquiry to think carefully
[institutional review boards (IRBs) in the United about what needs to be accomplished, and the best
States] that review and approve research as meet- ways to get the work done within the time frame
ing ethical standards. Investigators must complete and financial resources available.
IRB applications that typically include: Although having multiple people working
Table 26.1 Example Section of a Management Plan Excerpted from a Grant Entitled
“Pathways to Self-Determination”
PD-Project Director CoPD-CoProject Director OT-Occupational Therapist
RS Research Specialist PM-Peer Mentor RA-Research Assistant
Aim 3: Rigorously study the impact of the ESD and its sustainability by implementing a three-group comparison study
combined with Participatory Action Research
Primary Timeline
Goal Objective Responsibility (Yr/Mo)
3.1 Design system 3.1a Determine mechanism for providing data CoPD 1/1–1/2
for data entry and to RA for entry; check that data
storage. arecomplete/clean.
3.1b Monitor data collection, entry, and storage RS 1/2–3/1
on an ongoing basis to ensure complete
and timely data collection and storage;
conduct bi–monthly reviews.
3.2 Train staff in use 3.2a Train program staff and research assistant PD 1/1–1/2
of all data collec- in quantitative and qualitative data
tion tools and collection.
methods of 3.2 b Collect demographic and disability data RS 1/2, 1/8, 2/2
reporting; collect for both baseline and outcomes.
data. 3.2c Develop system for following clients who RS 1/1–1/3
move out of the facilities to prevent missing
data.
3.3 Develop 3.3a Establish formal mechanisms for program CoPD 1/1–1/11
feedback systems clients, facility staff and leadership, and
for key employers to give feedback and
stakeholders for suggestions for program improvements.
continuous 3.3b Routinely collect data from all key RS 1/1–3/10
program stakeholders and report to grant team at
improvement. biweekly meeting.
3.4 Analyze data to 3.4a Hold biweekly meetings of grant staff to PD 1/1–3/1
allow for program review data collection systems, monitor
modification and program fidelity, and identify variances in
continuous program component implementation.
improvement. 3.4b Meet quarterly to review evaluation PD 1/3 and
findings with staff at the three facilities. quarterly
3.4c Produce semiannual and final reports with CoPD 1/7, 2/1, 2/7,
findings on project implementation and 3/1, 3/7,
outcomes. Review with Advisory Council, 3/12
staff, and leadership at the three facilities.
3.5 Identify and inte- 3.5a Identify and implement strategies for PD 1/3–3/10
grate changes program improvement, measure effect, and
and assess document results to aid with program
impact on replication and program fidelity.
outcomes.
3.6 Evaluate 3.6a Include facility staff in the development of PD 1/2–3/11
effectiveness of the evaluation plan to empower participants.
staff intervention 3.6b Identify measures of program fidelity. PD 1/1–1/2
to maintain 3.6c Develop data collection methods to RS 1/1–2/1
fidelity of program measure of program effectiveness and
services. program fidelity.
3.6d Evaluate services provided by facility staff. RS 2/2–3/1
430
Table 26.2 Research Team Roles and Responsibilities

Team Member Typical Roles and Responsibilities
Principal investigator • Develops questions and research design.
• Takes ultimate responsibility for the project, including:
• Writing proposal,
• Obtaining IRB approvals,
• Obtaining funds,
• Developing or selecting instruments and equipment,
• Hiring and training project coordinator and other staff,
• Supervising staff,
• Monitoring progress on all aspects of the project,
• Confirming accuracy of analysis, and
• Disseminating findings.
Co-principal • Works closely with the PI, often taking responsibility for a specific aspect
investigator of the project.
Project coordinator • Works closely with the PI and Co-PI, under their supervision.
• Fulfills a range of administrative and human resource responsibilities,
including but not exclusive to:
• Participating in grant writing,
• Managing the day-to-day aspects of the budget (e.g., invoices,
payments, etc.),
• Providing day-to-day supervision and coordination of research
assistants,
• Coordinating data collection, entry, cleaning, and checking, and
• Participating in dissemination.
Research specialist • Often a more senior member of the research team, and one with
specialized skills or experience directly relevant to the project.
• Often the primary data collector and/or interventionist, particularly when
specialized skills are needed.
Research assistants • Nature of responsibilities vary with project but are often the primary data
collectors when specialized skills are not required (i.e., can be trained
specifically for the project), e.g., interviewers, raters, testers.
• May be involved in literature reviews, data coding and checking, data
entry.
• Exact nature and extent of their duties will depend on previous
experience and education.
Statistician • Can be a consultant or primary member of the team.
• Typically works collaboratively with the PI, starting at the time of the
study planning.
• May take a role in data management, if other members of the team do
not have the necessary skills.
Administrative assistant • Common with large projects, particularly in center grants and other
multiproject efforts.
• Fulfills roles of receptionist and secretary; sometimes will be involved in
coordinating data collection efforts.
Database manager • Important for large projects, particularly ones that have multiple sources
of data or are receiving data from multiple sites.
• Maintains all computer files, and is often involved in running specific
analyses.
• Will have specialized computer and analytic skills (e.g., SAS).
Data entry specialists • Convert raw data into electronic format.
431
together on a study can be invigorating and greatly • Policy and procedure manuals that outline all
enhance research productivity, supervising and aspects of the team’s operations from contacting
coordinating research staff presents challenges. To research participants, to entering and backing up
manage these challenges, it is important to draw on data, and
knowledge from individuals who specialize in • Regular team meetings to review progress and
human resources management and supervision. plan strategies to address challenges, and to
Some of the basic tools that can be used to facili- celebrate achievements and keep team members
tate research team operations and the process of motivated.
supervising staff include:
Recruiting and Retaining
• Clear job descriptions that outline responsibili-
ties, lines of authority, and required skills (the
Subjects (Participants)
feature box below provides an example of a The success of every research project hinges on
research assistant job description), being able to recruit and retain adequate numbers
• Organizational charts, particularly for large and of participants for the study. Since these subjects
dispersed teams (Figure 26.2), are volunteers who rarely receive much incentive
Job Description: Research Assistant
Project Title: Aging with Multiple Sclerosis: a. Receive and organize return letters of individ-
Unmet Needs in the Great Lakes Region uals willing to participate under the direction
Principal Investigator: Marcia Finlayson, PhD, of the project coordinator.
OTR/L b. Send letters to people chosen to participate to
notify them of approximate date/time of inter-
Name and Title of Direct Supervisor: Toni Van view call.
Denend, Project Coordinator 3. Assist in preparation of telephone interview
Summary of Job Character: The purpose of this guides for use.
position is to provide administrative and research a. Participate in the pilot testing of the revised
support to Dr. Finlayson for the MS Health Care telephone interview guide for understandabil-
Delivery & Policy Research Grant. The research ity and time using advisory group members
assistant will work under the direction of the and/or other volunteers.
Project Coordinator. Duties will include, but will b. Transfer interview guide to SPSS Data Entry
not be exclusive to providing clerical support, Builder.
assisting with data collection and entry, conducting c. Pilot test interview guide using SPSS Data
basic statistical analyses, and communicating with Entry Builder to confirm data entry process
other collaborators. The research assistant will be and make any necessary revisions.
expected to work with minimal to moderate super- 4. Assist in telephone interviews with people with
vision, to meet deadlines, and to actively con- MS.
tribute to the efforts of the research team. a. Contact selected participants and conduct tele-
Under the direction of the principal investigator phone interview.
and/or the project coordinator, the research assis- b. Complete data entry directly into SPSS Data
tant will: Entry Builder program.
5. Assist in sample selection of caregivers.
1. Assist in any logistical arrangements for the a. Maintain list of individual caregiver
focus groups, as requested. names and contact information provided
a. Assist in the setup of office systems to manage by people with MS via the telephone
focus group documents and data. interviews.
b. Make telephone calls and send confirmations b. Send letters to people chosen to participate to
regarding focus group locations. notify them of the study and upcoming tele-
c. Assist with notifications to focus group partic- phone contact.
ipants. 6. Assist in telephone interviews with caregivers.
d. Prepare audiotapes for the transcriptionist a. Contact selected participants and conduct tele-
after the focus groups are completed. phone interview.
e. Assist in coding focus group transcripts, as b. Complete data entry directly into SPSS Data
requested. Entry Builder program.
2. Assist in the setup and maintenance of office 7. Conduct literature reviews for dissemination
systems to manage the sampling frame and the activities.
sampling process for both the older adult inter- 8. Perform other duties as assigned.
views and the caregiver interviews.
larger population of interest. Ensuring represen-

Principal tativeness involves a certain level of expertise and
Investigator
knowledge about the sociodemographic and
impairment-related characteristics of a given popu-
Statistician lation. Though clinical experience is helpful, it
cannot always account for the wide ethnic and
Project socioeconomic composition of a given sample.
Coordinator Moreover, clinical samples by definition do not
include individuals without physical or economic
access to health care because of low income, lack
of insurance, or the nature and severity of the
impairment. For these reasons, it is also important
Research Research
to gather knowledge about a population of interest
Specialist Assistant
from published epidemiological studies that
describe the prevalence and incidence of a given
Figure 26.2 Simple organizational chart.
condition and can include information about its
geographic, racial/ethnic, sex, age, and socioeco-
beyond the opportunity to contribute to science, it nomic distribution, among other variables.
can be a challenge to secure adequate numbers.
Subjects can be recruited for studies using a vari- Creating Links to Sample Sources
ety of approaches. This step involves building relationships with the
Within occupational therapy, samples are most gatekeepers of the sample. Gatekeepers may take
commonly recruited from inpatient and outpatient many forms and may include referring physicians,
clinics that house occupational therapy clients, administrators of local outpatient rehabilitations
from physicians and other healthcare providers, centers, or presidents of self-help organizations.
from schools, and from self-help and advocacy- The stronger the relationships with these sources
based organizations to which individuals with and the more incentive they have for allowing their
impairments and chronic illness are members. clients or constituents to participate in the study,
Subjects can also be recruited through advertise- the more likely they will be to refer or allow access
ments about the study in newsletters, local cable to participants.
TV stations and newspapers, special interest
magazines, and from special interest Web sites. Developing Recruitment Materials
However, placing advertisements in some of these
Great care should be taken in developing recruit-
venues can be very costly. Recruiting subjects for
ment materials, such as brochures and posters, that
a study generally involves creating a recruitment
are clear, inviting, physically attractive, and in
plan. This plan involves five major steps:
alternative, accessible formats (e.g., different lan-
• Locating the sample, guages). These materials should emphasize the
• Ensuring sample representativeness, benefits of the study and they should mention that
• Creating links to sample sources, incentives or payments will be available, if appli-
• Developing recruitment materials, and cable.
• Funding recruitment efforts (recruitment materi-
als and participant payment or incentive funds). Funding Recruitment Efforts
Recruitment is enhanced if there are sufficient eco-
Locating the Sample nomic resources to provide quality recruitment
Developing a recruitment plan involves determin- materials and incentives for participants. Incentives
ing the most likely places where the intended sam- can range from direct cash payment to tokens such
ple exists. It also requires determining what is as gift certificates, raffle tickets, study pens, mouse
required in order to come into contact with the pads, cups, candy, or stickers. Funding for these
sample. Finally, it necessitates determining how efforts is generally provided by the supporting
best to make potential subjects aware of the study. institution or the supporting grant.
Ensuring Sample Representativeness Collecting, Managing, Storing,

and Analyzing Data
Ensuring that the sample is representative is
important as a means of increasing the likelihood Research involves not only the actual process of
of generalizing findings from a given study to a collecting data, but also managing and storing
Statement of a Sampling Plan
The following is an excerpt from a federal grant viduals who are most severely disabled from her
proposal that describes plans to generate a sample roster of individuals seeking services. In addition,
of adults with chronic fatigue syndrome (CFS) for all of our study advertisements will emphasize the
a randomized clinical trial. It illustrates the kind benefits of the intervention to individuals with CFS
of detailed considerations that go into planning who are most severely disabled. Once referrals
for a study sample. to the study are made, we will continue the recruit-
To ensure that the sample is demographically ing of severely disabled individuals with CFS by
representative of the general population of individ- highlighting issues related to easy accessibility
uals with CFS, we will recruit participants so that to the intervention and by emphasizing the incen-
at least 53% of individuals in our sample are from tive value of the project, the provision of resource
numerical minority backgrounds, and at least 72% funds to support the opportunity to gain access
of individuals are women. Consistent with our find- to important resources (e.g., personal assistants,
ings from a prior epidemiological study, we will vocational rehabilitation evaluation and train-
also ensure that people of all ages over 18 years are ing). We will emphasize that the intervention is
represented, and that all levels of socioeconomic designed to accommodate people with CFS who
status are included, particularly those with the most use wheelchairs and/or are too ill to leave home
limited resources. We will make concentrated via usual modes of car, taxi, or regular public
efforts to ensure that as many participants who are transportation.
ethnic minorities are recruited as possible using the We will also add incentive value and make the
following strategies: (1) our collaborating organiza- intervention accessible to severely disabled indi-
tion has a database of members that includes basic viduals with CFS by informing prospective par-
demographic information and research interest. We ticipants of our intention to provide the following
will actively recruit members who identified them- alternative transportation options in order to pro-
selves on their initial membership application as vide transportation to and from the intervention:
being interested in research participation, and who (1) renting wheelchairs to facilitate transportation
identify as an ethnic minority; and (2) we will to and from the CIL; (2) utilizing Chicago Transit
access our existing network of connections with Authority Special Services, which provides low-
Chicago-area physicians who treat individuals with cost, door-to-door van service transportation with
CFS and practice in ethnically diverse neighbor- appointment and has the capacity to transport
hoods and we will ask them to refer ethnic minor- ambulatory individuals as well as individuals
ity participants with CFS to the project. In general, who use motorized scooters, motorized wheel-
we will also recruit participants by advertising the chairs, and manual push wheelchairs; (3) utilizing
study in the newsletter produced by the collaborat- taxi services operative within the Chicago area
ing association, in newsletters produced by larger that accommodate standard wheelchairs and offer
national self-help organizations, and on listserves reduced fares with voucher for individuals with
and Web sites dedicated to CFS. disabilities; or (4) utilizing private medical trans-
In addition to these issues, we will also ensure portation companies (e.g., “Dial O Ride”) that
that the sample is representative in terms of degree provide vans equipped to accommodate motorized
of disability of participants. Given that middle and wheelchairs with tilt-back features. When neces-
higher functioning individuals may be more likely sary, we will use grant resource funds to support
to participate than lower functioning individuals the cost of these transportation options. In addition
with CFS, we will actively recruit a sufficient to these options, individuals too ill to attend group
number of lower functioning individuals by using on any given day, but able to talk on the phone,
a variety of approaches. First, we will target lower will be encouraged to participate in the group
functioning individuals by asking referring physi- via teleconference speaker hook-up. Our col-
cians to refer their most disabled individuals with laborating organization already owns this resource
CFS to the intervention, and by asking the presi- and is willing to provide it during the illness-
dent of our collaborating organization to refer indi- management groups.
what is collected. Together, collecting, managing, • Following through on the use of these tools
storing, and analyzing data is referred to as data and systems, refining and adding to them as the
management. It involves: project evolves over time.
• Documenting how all forms of data will be col- The purpose of data management is to:
lected, handled, stored, and prepared for use,
• Developing tools and systems to operationalize • Ensure reliability of the information collected,
the documentation, and • Maintain data security,
• Facilitate research team communications, and Communication with Funding Agency

• Facilitate the preparation of the final reports. (Progress and Final Reports)
Unfortunately, data management is a skill that Funders, whether intramural or extramural, want
most researchers learn through their own mistakes, to know how a project is progressing as expected.
which is why mentorship by senior investigators Consequently, they will almost always provide a
can save new researchers a great deal of time, and detailed schedule and a set of instructions about
strengthen the rigor of their work. what reports to provide to them, and when. These
instructions will include what information to
Dealing with Typical Challenges include in the report, including updates on expen-
in Study Implementation ditures. It is critical that reports are completed on
time, following the funder’s specifications. Failure
As with any long-term undertaking, research to do so can sometimes result in a revocation of
projects are prone to encounter challenges. Three funding.
of the most typical challenges are:
• Participant recruitment and retention, Dissemination
• Collaborative agencies, and
• Study staff changes. No study is completed until it has been formally
Sometimes, despite the best intentions, plans shared with other members of the scientific com-
for participant recruitment are unsuccessful, or munity and other interested constituencies. The
come up short of the numbers needed to complete usual vehicles for sharing research findings are:
the work. In these situations, it is often necessary • Presentations and posters at scientific/profes-
to reexamine the research questions, the recruit- sional conference,
ment strategies, and the data analysis plan and • Published papers in refereed journals, and
make revisions. These revisions will almost always • Consumer-friendly dissemination products such
need to be approved by the funding agency and the as educational videos, resource manuals, book-
IRB before they can be implemented. To circum- lets, and curricula.
vent this challenge, it is best to plan recruitment
strategies to take into account the worst-case sce- Each of these venues of dissemination is briefly
nario. It is better to have too many potential par- discussed below. Chapter 34 provides a detailed
ticipants than not enough. discussion of dissemination strategies.
Challenges to research implementation can also
arise when there are changes at collaborative agen-
Presentations and Posters
cies, such as changes in their organization or Presenting a study before peer scientists and/or
staffing that make it difficult to maintain their professionals is an important form of sharing find-
commitment to the project. To address this possi- ings. Because presentations are face to face and
bility, it is always wise to maintain close contacts involve opportunity for discussion of the findings,
with collaborators so that if problems are emerg- they can also provide an opportunity to help the
ing, the researcher is not investigator think about
caught off-guard. the implications of the
Given the length of No study is completed study. Presentations and
many research projects, poster sessions are gen-
staff change is inevitable,
until it has been formally erally refereed—that is,
particularly in a univer- shared with other members one has to submit an
sity environment where of the scientific community abstract describing the
graduate students play a presentation and then
large role as research and other interested reviewers decide which
team members. Building constituencies. of the abstracts will be
in incentives to keep selected for presentation.
staff—for example, offer- This selection process is
ing continuing education resources—can often designed to ensure that the presentations/posters
help maintain a stable staff. Barring this, ensuring meet some minimal criteria of scientific rigor.
the project, tasks, and details of the study are Depending on the type of conference, the referees
well documented will aid in any transitions that a may or may not be aware of the presenter’s iden-
project must endure. tity when abstracts are reviewed.
Publication These include consumers, their family members,

and practitioners who are interested in using the
Refereed papers are by definition submitted to a procedures and tools that were developed or tested
journal editor for blind review (i.e., review by per- as part of the research process. Typical venues for
sons who are unknown to the author and to whom dissemination include presentations or publica-
the author’s identity is unknown). The process of tions that are focused on the practical implications
blind review is designed to minimize bias and to of the research findings, Web sites, and manuals.
maximize the degree of honesty in the feedback
process. Scientific journals have published guide-
lines that specify the kinds of papers that are Conclusion
appropriate for the journal and that specify how
the papers are to be submitted (electronic versus As noted at the beginning of the chapter, the
hard copy, number of copies, forms to be submit- process of planning and implementing a study is
ted with the article). Papers that are not judged to long and complicated. Needless to say, it requires
meet adequate standards or the purpose of the jour- commitment, foresight, flexibility, and persever-
nal may be returned rejected. Depending on the ance. Planning is key, and this process can be made
journal, rates of acceptance may vary from a high significantly easier by connecting with more sen-
proportion of submitted manuscripts to a very ior investigators, using co-investigators and collab-
small percentage. It is also common for a paper to orators, and being realistic about what can be
be returned with requests for substantial revision accomplished. Like research itself, the process of
in which case the paper is typically re-reviewed by doing research is developmental, with each new
the original reviewers. Generally, even when sub- project building on the experiences and learning
mitted papers are accepted for publication, revi- that took place during the previous project.
sions are required before the paper goes into press.
Different journals have different lead times for a REFERENCES
journal to appear once accepted; these may be as Case-Smith, J. (1999). Developing a research career:
short as a couple of months or longer than a year Advice from occupational therapy researchers. The
depending on the journal. American Journal of Occupational Therapy, 53(1),
44–50.
The process of submitting papers for publica- Hammel, J., Finlayson, M., Kielhofner, G., Helfrich, C., &
tion requires a degree of patience and willingness Peterson, E. (2002). Educating scholars of practice: An
to accept and use feedback. As challenging as the approach to preparing tomorrow’s researchers.
review process can be at times, it is necessary in Occupational Therapy in Health Care, 15(1/2),
157–176.
order to help ensure the quality of papers that are
Helfrich, C. A., Finlayson, M., & Lysack, C. (in press).
published. Journals range in the degree of rigor Using a mentoring community to build programs of
they expect of papers and consequently some jour- research: Lessons learned and recommendations from
nals are much harder to publish in than others. participating in a mentoring community. Journal of
Allied Health.
Kielhofner, G., Hammel, J., Helfrich, C., Finlayson, M., &
Consumer-Focused Dissemination Taylor, R. R. (2004). Studying practice and its out-
comes: A conceptual approach. The American Journal
While presentations and publications are impor- of Occupational Therapy, 58, 15–23.
tant ways to disseminate research findings to sci- Taylor, R. R., Fisher, G. & Kielhofner, G. (2005)
Synthesizing research, education, and practice accord-
entific and professional peers, there is increasing ing to the scholarship of practice model: Two faculty
emphasis on sharing findings to other constituen- examples. Occupational Therapy in Health Care, 19,
cies who might benefit from the information. 95–106.
C H A P T E R 2 7
Searching the Literature

M. G. Dieter • Gary Kielhofner
Retrieving available information relevant to a topic solving, planning research, finding evidence to
is vital to both conducting and being a consumer of guide practice, and personal knowledge develop-
research. Two factors are rapidly transforming how ment. Information needs generally fall into two
investigators and practitioners search for knowl- categories:
edge. The first is an explosion of available infor-
• Ready-reference, and
mation. For instance, in the year 2002, nearly
• Subject searching.
five exabytes of information were newly created
for storage in print, film, magnetic, and optical
Ready-Reference
media. This new information would fill the
equivalent of 500,000 new libraries equal in size Ready-reference information requests are typically
to the Library of Congress print collection (Lyman very specific, and usually resolved through
& Varian, 2003). The a closed-ended search
second factor is how process. For example,
it is currently stored and Information literacy is the an occupational therapist
made accessible. In the ability to articulate informa- may wish to know the
past, searching for infor- following kinds of things
mation mostly involved tion needs, and subse- about a particular dis-
accessing printed mate- quently locate, retrieve, and ease: its prevalence, eti-
rial stored in libraries. ology, prognosis, func-
Over the last few de- critically evaluate of informational implications, and
cades information has tion for appropriate use so on. In this instance,
become increasing avail- the IR task is to identify
able through information
(Association of College and an appropriate informa-
and communications te- Research Libraries [ACRL], tion resource and to
chnology (ICT). 2000). access the information
While the exponen- (e.g., a text, Web site,
tial growth and ready and/or recent review
availability of information is a substantial article that contains the desired information).
resource, it also means that one can easily be lost Ready-reference tends to be relatively straightfor-
in too much information or fail to find all the infor- ward.
mation one needs. Effective identification and use
of information requires a level of literacy.
Information literacy is the ability to articulate
Subject Searching
one’s information needs, and subsequently locate, Subject searching is an iterative and ongoing infor-
retrieve, and critically evaluate information for its mation-seeking process. Subject searches use a
intended use (Association of College and Research querying method to transform abstract topic con-
Libraries [ACRL], 2000; Jenson, 2004; Saranto & cepts into text search queries. These queries are
Hovenga, 2004; Swanson, 2004). This chapter inputted into the interface of online information
overviews what is involved in, and provides retrieval tools, such as a bibliographic citation
resources to enable readers to improve their infor- database search engine like PubMed or a Web
mation literacy. search engine like Google. The resultant informa-
tion retrieval is a set of information items (“hits”)
that have been identified by the search query, and
Online Information Retrieval organized or prioritized in some fashion.
Importantly, when one analyzes the retrieved
All information retrieval (IR) activities are driven set items, it is common that one will refine the sub-
by information needs. Occupational therapists ject. This, then, necessitates more search queries.
need information for purposes such as problem- Consequently, searching is essentially a sifting or
filtering process intended to produce a core set of

information items that are identified by a succes-
Online Access to Published
sion of specific search queries. Information
Searching allows one to integrate information
into knowledge. This can take the form of an There are a number of possible ways to access the
evidence-based search (as discussed in Chapter variety of information resources available online.
42). The purpose of a search is to exhaustively One important source of access is through institu-
identify all the information relevant to a topic. In tions that have access to online information
some cases, the search may also put certain param- resources, such as Web portals. An alternative is
eters on the kind of information one wishes to Web access through local public or community
retrieve. For example, one may wish to limit infor- college libraries. Generally, there are three chan-
mation sources to published journal articles. nels to online information retrieval that provide
access to published information resources:
The Process of Information Retrieval • Online public access catalogs (OPACs),
Most IR involves a combination of two broad • Bibliographic citation databases, and
search strategies, browsing and querying (Taylor, • The World Wide Web (WWW or Web).
2004). Each is discussed below:
In the following sections we focus mainly on
Browsing strategies for subject query searching involving
bibliographic citation databases, since these are
There are basically two browsing methods: the major sources for finding research publications
• A structured approach, and used in evidence-based practice and in the litera-
• A serendipitous approach (Taylor, 2004). ture review step of the research process.
Structured browsing involves a preplanned Online Public Access Catalogs

organization of a list of topics. These lists often
take the form of a hierarchy in which there is a Libraries and other repositories of information tra-
more general list of topics. Then, within each of ditionally used catalogs as a point of access for
these topics a sublist of more specific topics. In locating their materials. Originally in print format
this approach, one follows a directed path to the as a book, catalogs later evolved into card catalog
information sought. and microfilm formats. At present, both individual
The serendipitous approach is more random- libraries and networks of libraries use an electronic
ized. In this approach, the searcher travels a non- database of their contents commonly known as an
linear path to information. For example, one might online public access catalog (OPAC). Consortia of
randomly follow an unfolding series of hyperlinks OPACs may be networked to provide a wider range
on the World Wide Web, picking up useful infor- of information-locating possibilities. Although
mation along the way. OPACs usually have subject searching functional-
ity, OPAC searches tend to be more close-ended,
Querying seeking known items at a particular site, for exam-
ple, a book by a specific author, or a title. OPAC
A querying method of information retrieval
records include bibliographic descriptions of phys-
involves text phrase matching or keyword matching
ical items such as books, pamphlets, and periodi-
(Taylor, 2004). Phrase matching specifies a partic-
cals, as well as electronically formatted materials.
ular string of text that is used to reduce the quantity
of retrieved items by increasing specificity. For
example, an occupational therapist seeking to find
appropriate assessments to use in a pediatric setting
Bibliographic Citation
may begin with the phrase “occupational therapy Databases
assessment,” proceed to “occupational therapy
pediatric assessment,” and then to “pediatric occu- Bibliographic citation databases allow online
pational therapy observational assessment.” access to the bibliographic records. Like OPACs,
Keyword searching generally involves using electronic citation databases group information
one or more words to retrieve information. content in certain ways, including by general sub-
Importantly, keywords can be logically combined ject. One example is the National Library of
and uniquely related to each other in search Medicine’s (NLM) Medline bibliographic citation
queries. This type of search strategy is discussed database, which offers access to predominantly
later in the chapter. biomedical information content. Unlike OPACs,
Chapter 27 Searching the Literature 439
the information items constituting a bibliographic • The particular characteristics of the information
citation database are not limited to resources that resources one chooses,
are located at a particular site or repository. • One’s IR skills,
Nonetheless, they may include information about • Available time, and
institutions that own a particular book or journal, • Resources to cover costs.
for instance.
Much of the information content represented by For subject querying bibliographic citation
bibliographic citation databases has undergone bases, there are two kinds of search processes:
some kind of review process that ensures its accu- • Concept expansion, and
racy, originality, authority, rigor, and so on. For the • Concept contraction.
kinds of information that occupational therapists
generally seek, this review process ordinarily These are often combined in tandem as com-
involves peer review, which is discussed in plementary iterative processes.
Chapter 29. Concept expansion refers to broadening a
Bibliographic citation databases such as search topic to include all its relevant dimensions.
Medline, CINAHL, and the Cochrane Database Concept expansion often originates from a specific
of Systematic Reviews item of information item,
index research reports, or a small set of such
practice guidelines, lit- Ultimately, the only way to items. For example, one
erature reviews, edito- improve online searching might receive a few spe-
rial comments, letters cific research papers from
from subscribers, and skills is by continuing to a colleague, or find a pre-
other information into actively perform and refine viously completed litera-
content groupings. What ture review on a topic
database one can use searches, and to evaluate of interest. Alternatively,
and what is available in information retrieval. one might find a key
the database depends on source by browsing the
where one is searching Web and finding a Web
from (e.g., through a university or a public library). site with a number of bibliographic references on a
Each institution has particular subscription topic. In each of these instances, these resources
arrangements that may provide access to cita- are the initial context for conceptual expansion.
tions only, or access to the full text of the publica- The goal of the complementary process, con-
tions. traction, is to focus the retrieval into a more con-
ceptually consistent, relevant, and manageable set
The WWW of information. As noted earlier, one frequently
The World Wide Web or WWW has made possible alternates expansion and contraction techniques to
a wide variety of electronic publications. Unlike modify the developing search concept, resulting in
bibliographic citation databases, the Web currently a search process that is dynamic and iterative.
lacks standardized indexing. While streamlining In practice, expansion and contraction proc-
the processes of publication and dissemination, the esses fine-tune the relationship between recall (the
WWW also bypasses traditional processes for comprehensiveness or proportion of available
evaluating the quality of the information included. information items available that are retrieved) and
Web search engines, along with WWW directories precision (i.e., the proportion of retrieved items
(e.g., Yahoo) that categorize Web information con- that are relevant). In general, expansion strategies
tent, are a means of retrieving a wide variety of are aimed toward maximizing recall, while the
content published on the Web. goal of contraction strategies is precision.
For example, a search process may approach a
Subject Searching in Bibliographic figure of 100% recall, meaning that one has
Citation Databases retrieved nearly every journal article in the biblio-
graphic citation database that has any relevance to
Subject Search Strategies one’s defined topic. However, if the topic for
which one is searching is a well-researched area,
Selecting an online IR subject searching strategy
one may retrieve more journal articles than one has
depends on many factors, including:
resources to review and evaluate. Among them
• The scope and depth of one’s information needs, would be some that are extremely relevant to one’s
• The availability of information resources, topic and others that are not particularly useful. To
narrow the number of articles to those most rele- • What (am I looking for?),
vant, one would use contraction strategies that • Where (can I find it?),
maximize precision. This would generate fewer • How (do I access and retrieve it?), and
articles that more specifically address one’s • How Well: (does it satisfy my information
defined topic. requirements?) (Figure 27.1).
Precision is desirable, but can also be problem-
atic if it results in too few articles and other
resources. For instance, if one searches in an area Step One: What?
of occupational therapy research for which there is
more limited research, a precise search could One must first identify one’s information needs.
result in too few articles. Thus, one can use expan- Among other factors, this typically includes con-
sion and contraction strategies to help identify just sideration of:
how specific the existing knowledge is, and to
identify the right “pool” of articles and other • Why the information is necessary,
resources to help one develop knowledge of a • For whom/what it is intended,
given topic. • The format required (e.g., refereed journal arti-
Many online bibliographic citation database cles), and
search engines and search interfaces have intrinsic • The time frame.
search features and syntaxes. It is useful to read all This first step is crucial and not as simple as it
Help screens, and print them out for future refer- may appear. How one initially chooses to define a
ence. The terms and concepts introduced in this subject influences:
chapter are found generally in most advanced
search tools, and can save much time in defining a • Later search decisions,
search. • Selection of which information items to include
(e.g., articles, books, reports), and
Conducting Online Searches • Examination and analysis of the information one
retrieves (Rieh, 2002).
in Bibliographic Citation The ultimate goal of this step is to be clear
Databases about what one needs to retrieve so that it can be
translated into search terms.
This section offers a generalized online IR frame-
work for subject querying. The overall method
involves making choices and decisions that lead to Step Two: Where?
a manageable number of resources that can be
examined, evaluated, retained or discarded, and Searching would be greatly simplified if there
used. An effective search strategy first requires that were a single comprehensive information resource
one: that could be searched. Unfortunately, this is not
yet reality. Bates (1989) originally described the
• Transform concepts (topics) into words that will process of information-seeking strategy as
be used to search, “berrypicking,” since searchers typically go from
• Logically connect these words by specifying
relationships between them, and
• Format them into search syntax appropriate for
the bibliographic citation database being What?
searched.
Next, one inputs the search queries into a data-
base search engine interface to retrieve sets of
information items. Then, one evaluates the infor- How Well? Where?
mation retrieved to ascertain if it is sufficiently rel-
evant, detailed, and focused to the topic for which
one was searching. The process is repeated to
refine the search until the retrieved information
items meet one’s needs. How?
The overall search process can be guided by
four key questions: Figure 27.1 Search processes.
resource to resource in search of relevant informa- portals will also suggest Internet Web sites that
tion items. may have useful information content. For instance,
A good search process is characterized by mak- Figure 27.3 shows UIC Library Internet Resources
ing good predictions of where relevant information Page for Occupational Therapy.
will be and using retrieved information to further Some institutions use metasearch engines that
define and articulate the search concept and trans- allow federated searching of multiple information
late it into further search strategies. When making resources. For example, Figure 27.4 shows the UIC
decisions about information resources, one should Library’s qUICsearch. It allows one to input a sin-
seek advice from knowledgeable peers, professors, gle set of search terms into the search interface that
mentors, colleagues, and reference librarians. will be sent to many different information sources
General knowledge sources such as dictionaries, (mainly bibliographic citation databases) to
encyclopedias, and textbooks often are valuable as retrieve a set of appropriate information items from
a point of entry into unfamiliar subject areas. each of them. These multiple sets can be combined,
Frequently, universities have created Web informa- ordered, and sorted in a number of ways.
tion portals where information resources are In general, selecting the appropriate informa-
organized alphabetically or by subject grouping tion source depends on one’s topic. This makes it
lists, often with appropriate thumbnail descriptions difficult to apply a general rule for selecting appro-
of the scope of each resource’s information con- priate information sources. For the purposes of this
tent. Figure 27.2 shows a page of the Web infor- chapter, we have reduced the range of choices to
mation portal found on the Web site of the several bibliographic citation databases that are
University of Illinois at Chicago (UIC). clearly relevant to occupational therapy, for exam-
In addition to providing access points to biblio- ple, the Cochrane, Medline, and CINAHL biblio-
graphic citation databases, some Web information graphic citation databases. These are not the only
Figure 27.2 UIC library subject page for occupational therapy databases.
Figure 27.3 UIC library Internet resources page for occupational therapy.
citation databases that an occupational therapist citation index database, such as the Thompson
might wish to use, but they are generally a good Corporation’s Institute for Scientific Information
place to begin. (ISI) Web of Knowledge. Using this tool, one can
Once one selects a bibliographic citation data- look forward from the original item to identify
base, search queries retrieve citations (i.e., the publications that have subsequently cited it in their
author, title, date of publishing, journal, etc.) of own references.
relevant information items. The citation may or
may not be linked to an abstract or to the full text
of the information item itself.
Step Three: How?
Citation Searching
The decisions that searchers make during this
Bibliographic citation analysis is another approach stage are guided by the first two steps of the
to subject searching. Citation searching means process. In this process, one must:
looking at the bibliographic citations or references
at the end of articles. It offers another way to • Articulate and expand the topic into appropriate
expand a search from a single article, chapter, search terms using synonyms,
book, or web resource, or from a few such re- • Generate a search query defining logical rela-
sources. Such references represent the author’s or tionships between the search terms,
Web creator’s efforts to identify important writings • Apply appropriate search syntax to the search
from the work of earlier authors. A searcher can query as needed to expand and contract search
than ready-reference search these cited resources. retrieval in terms of precision and recall, and
A prospective (forward-looking) form of cita- • Analyze information content to evaluate how
tion searching requires access to a bibliographic well it corresponds to one’s intended topic.
Figure 27.4 UIC’s qUICsearch metasearch engine.
As shown in Figure 27.5, these activities are Keywords

repeated in a cyclical process to refine the search
process. These activities are elaborated in the dis- Some bibliographic citation database search
cussion that follows. engines allow one to enter keyword search terms
by providing a text box with a drop-down label
Search Terms specifying each text entry as belonging to one or
more of the database’s fields. These keywords are
The process of selecting search terms involves a used to find information items in one of two ways.
process in which one uses information uncovered First, bibliographic citation databases sometimes
from the search to derive new search terms that index information items by selecting several key
yield the intended information (Dalrymple, 2001). terms that describe the general scope of its content.
Search terms are items used in search queries In this instance, articles and other information
that connect the intended topic to text in informa- items that have been assigned those keywords are
tion items that are retrieved. One continuously retrieved. Second, keywords may be used to iden-
identifies new search terms to use to expand tify articles or other information items for which
or contract the retrieval, improving recall and the keyword occurred in the title, author, subject,
precision, respectively. For example, using syn- abstract fields, or full text.
onyms for search terms is a common way to
expand a search and, thus, maximize recall. When Controlled Language
this results in irrelevant information being
retrieved, one will subsequently use contraction Some databases have features that allow one to
strategies to improve precision. The most common retrieve specific kinds of information items. For
search terms used are keywords and controlled example, the contents in a database may be organ-
language. ized according to a structured classification sys-
sion or contraction strategies that maximize either

Identify Search the recall or precision. Both kinds of strategies can
Terms
be incorporated into a single search query through
the use of nesting, described in the section below
on Boolean logic. One’s competence in post-
Analyze coordinating the logic of information retrieval pro-
Content for Apply Search cesses will enhance the meaning and relevance of
Correspondance Logic what one is able to retrieve.
to Intended
Topic Search query logic is arguably the most effec-
tive means of applying expansion and contraction
search strategies to maximize retrieval recall and
precision. When one configures precoordinated
Apply Search controlled language search terms using post-
Syntax to coordinated search logic one increases the likeli-
Query hood of retrieving exactly the information one
needs. This process is known as set searching.
Figure 27.5 The search process.
Boolean Logic
tem. In this case, one can use controlled language The search logic that is commonly used in biblio-
terms to specify a more direct pathway to the graphic citation databases is called Boolean logic,
information items one wants to retrieve. However, after 19th century English mathematician George
effective use of controlled language search terms Boole (Kluegel, 2001). Using Boolean logic, one
requires one to understand how an information can connect search terms, allowing more control
resource’s content has been organized. Controlled over the information retrieved than when using
language searches result in increased precision individual search terms. While it may seem intim-
when the correct subject headings are used. idating at first, it is well worth learning to increase
In some databases content is already organized the precision of searching.
(precoordinated) into controlled language classifi- Following the four-step search process de-
cation systems that use subject headings, thesauri, scribed in this chapter, one enters search queries
and ontologies as means of classifying and locat- using syntax appropriate to the bibliographic cita-
ing content. In addition, controlled language sys- tion database that is being used. The search engine
tems provide cross-references and links between then retrieves a set of information items that fit the
related terms to assist one in finding the appropri- requirements specified in the search query.
ate subject heading. Boolean relationships specify the inclusion or
The National Library of Medicine’s Medline exclusion of content that matches the text of search
Medical Subject Headings system, (MeSH), and terms in search queries, as illustrated graphically
the Cumulative Index of Nursing and Allied Health in Figure 27.6. Each circle represents an informa-
Literature (CINAHL) Subject Headings are exam- tion retrieval set that matches a text search term;
ples of controlled languages that are relevant for each pair of circles represents the use of a different
occupational therapists’ information needs. Both Boolean search operator to logically relate two
are hierarchical arrangements of subject headings. search terms, changing what is retrieved. There are
Medline’s content is predominately biomedical three main Boolean operators: AND, OR, and
and clinical medicine, while CINAHL’s is related NOT.
to nursing and allied health. Both systems provide
Boolean AND. The Boolean AND logical condi-
scope notes that explain the use of each subject
tion creates an exclusive set of information items
heading, its position in the hierarchy, and the his-
that match the specific text of both search terms.
tory of its use. Some bibliographic citation data-
As shown in the left diagram in Figure 27.6, the
base search interfaces, like Ovid, map natural
relationship between “dogs AND cats” defines a
language terms to the correct subject headings, or
retrieval set that includes only articles that are
present a list of near matches from which searchers
about both dogs and cats. By specifying dogs AND
can select a subject heading.
cats, one would not retrieve articles that are only
about dogs or only about cats. As show in Figure
The Search Query 27.6, the shaded area representing the intersection
Skill in specifying logical relationships between of the two sets defines a set of articles mentioning
search terms enhances one’s ability to use expan- both dogs and cats in their information content. As
Dogs Cats Dogs Cats Dogs Cats dogs, pets, and health. Among many other kinds of
articles irrelevant to the intended topic would be
those about pets that are not cats or dogs, and those
about the health of humans.
AND OR NOT Search Syntax

Figure 27.6 A Venn diagram of Boolean search In general, each bibliographic citation database
logic. search interface has its own syntax or format rules
specifying operations on search terms and queries.
this example illustrates, the Boolean AND opera- One should rely upon printouts of Help screens if
tor can be used in contraction search strategies to one is unfamiliar with the commands. However,
increase the precision in retrieval sets. some generic syntax functions and commands that
Boolean OR. The Boolean OR logical condition are common to many online information retrieval
creates a set of all information items that match the resources can be used to expand or contract
specific text of either of the search terms. As retrieval.
shown in the middle diagram in Figure 27.6, by
Case Sensitivity
specifying “dogs OR cats” one will retrieve arti-
cles about dogs, articles about cats, and articles In general, most bibliographic citation database
about both dogs and cats. In the diagram, the search engines are not case sensitive. The default
shaded area represents the union of the two sets, a condition is typically lowercase, and the retrieval
set of articles mentioning dogs, cats, and both dogs includes lower- and uppercase occurrences.
and cats. As this example illustrates, the Boolean
OR operator can be used in expansion search Truncation
strategies to increase recall.
This is a useful search operator for expanding
Boolean NOT. The Boolean NOT logical condi- retrieval to account for multiple forms of a term, or
tion creates an exclusive set of all information when the exact spelling of the term is uncertain.
items that match only the specific text of one of the The most frequent types of truncation functional-
search terms. As shown the right diagram in Figure ity are termed “right-hand” and “left-hand”; most
27.6, specifying “cats NOT dogs” defines a research engines offer right-hand (the end of the
trieval set that includes articles about cats only. In term). The specific operator varies, but often the
this instance, any articles about cats that also con- “?”, “#”, or “*” character is used for truncation. An
tain information about dogs would be eliminated example of right-hand truncation is the search term
from the search set. In this case, the shaded area “librar*.” The retrieval would include items with
represents elimination of the dog set from the cat “library,” “libraries,” “library’s,” “librarian,”
set; that is, articles about dogs, and articles about “librarians,” and “librarianship.” Left-hand trunca-
both cats and dogs are eliminated, leaving articles tion is situated at the leading end of a search term,
about cats only. The Boolean NOT operator can be and is used infrequently. Middle truncation func-
used in contraction search strategies to increase tionality is less commonly available than right-
precision. hand truncation, but more so than left-hand
truncation, for example, “wom#n” used for
Nesting “woman” or “women.”
Nesting allows Boolean search term relationships Phrases
to be combined with each other in a search query
to improve control over the set of retrieval items. Some bibliographic citation databases allow
Parentheses are used to separate the search term phrase searching. Phrase searching searches for an
relationships. For example, a nested statement exact string of text. This increases precision and
would be “((cats OR dogs) AND pets) AND reduces retrieval. For example, a search for
health.” This search query would result in the “reduce tactile defensiveness” or “increase range
retrieval of items about the health of cats and/or of motion” would retrieve only information items
dogs that are pets. It is important to remember that in which those specific phrases occurred.
leading and closing parentheses must enclose each
Proximity
logical set. One can readily see how much more
precision is achieved by using this kind of search Proximity searching (also called adjacency or posi-
logic than simply searching for the key terms cats, tional searching) is a useful tool for specifying the
relationship of terms by their nearness to each other research question: “What statistical methods might
in the text of the information item. For example, be used to describe the outcomes of occupational
one might search for items about health care work- therapy interventions for persons with stroke?”
ers, and could search using the phrase “health care The matrix rows show search terms related with
workers.” This strategy might eliminate a number the Boolean “AND,” which would contract the
of useful information items that do not use the search, facilitating precision. The columns relate
exact phrase “health care workers,” such as “work- terms with the Boolean “OR,” which would
ers in health care” or “workers employed by health expand the concept, facilitating recall.
care providers.” A proximity search operator allows Ideally one would employ an information
one to expand the search, increasing the recall to resource that allows set searching, allowing one to
include terms that are near each other in an article use search terms related by Boolean logic. For the
or other information item. Some bibliographic cita- purpose of demonstration, Figure 27.8 illustrates
tion database search engines use “adj,” “n,” or “w” and explains an example of a single search query
to specify proximity. The order of the search terms that combines the matrix search terms according to
often specifies the order of occurrence in the text. the specified Boolean logic, using a generic syn-
More powerful search engines can specify the tax.
number of words that may occur between the One should note that in the examples in Figure
terms in the item, or their order of occurrence. 27.8, lowercase and truncation functionalities were
Proximity can be used as an expansion or contrac- used to expand the terms further. In addition, posi-
tion strategy. tional syntax was substituted for phrase matching
as a means of fine-tuning contraction. Truncation
Analyzing Information Content Evaluatively also reduced the number of search terms required
When new information items are retrieved, one for entry by accounting for variants of several
must evaluate them in terms of their relevancy to terms (e.g., rehab* for rehabilitation, rehabilitate).
the topic for which one is searching. Also, one may In bibliographic citation databases such as
use the retrieved information as feedback that can CINAHL that use controlled language, there is
lead to broadening or narrowing of one’s intended pre-coordination of terms like “occupational ther-
topic. This reflects the fact that the search process apy,” which eliminates the need for positional syn-
is iterative, with each step informing the next. tax. Many variations of the query shown in Figure
27.8 are possible for querying the research ques-
Sample Query tion, as is true for all online subject search queries.
Prior to searching online, a good way to begin is
Post-Qualification and Limiting
by diagramming the terms one intends to use along
with their logical relationships in a matrix. By At any point in the retrieval process, searchers can
doing this, one is constructing a nested search choose to further focus a retrieval set by limiting
query from search terms that best represent the the retrieval to a certain time period, type of publi-
topic for which one is searching. cation, language, or source. Typically, this is done
Figure 27.7 shows how a searcher might begin near the final steps of the process. Searchers can
the process on paper with a search matrix for the also post-qualify terms by specifying occurrence
CONTRACTION
Statistics Method Outcome(s) Occupational Intervention(s) Stroke

EXPANSION
Therapy
OR
Cerebral-
Statistical Methodology Results OT Rehabilitation vascular
accident
OR
Description Quality of After-care Therapy CVA
Analysis life
AND AND AND AND AND
Figure 27.7 Initial topic expansion and contraction using Boolean relationships of search
terms for the research question: “What statistical methods are used to describe the out-
comes of occupational therapy interventions for persons with stroke?”
Note: Assume that "*" is used as a right-hand truncation symbol for 0 to n

characters; and "w" is a proximity operator, where an optional number can be
used to specify positional distance in number of words, e.g., the use of "w2" to
specify the positional arrangement as shown in the example below. A logical
representation of the relationships in the matrix could appear as below. In this
case each "AND" separates text representing a column of the matrix.
(statistic* OR (descript* OR describe*)) AND (method* OR analy*) AND
(outcome* OR result* OR (quality* w2)) AND ((occupational w therapy) OR
ot OR (aftercare OR (after w care)) AND (intervent* OR rehab* OR therap*).
This query contains the following elements:

■ (statistic* OR (descript* OR describe*)) Inclusive expansion of column 1
of the matrix
■ AND Used to contract the retrieval
■ (method* OR analy*) Inclusive expansion of column 2 of the matrix
■ (outcome* OR result* OR (quality w2 life)) Inclusive expansion of column
3 of the matrix
■ ((occupational w therapy) OR ot OR (aftercare OR (after w care)) Inclu-
sive expansion of column 4 of the matrix
■ AND Used to contract the retrieval by limiting inclusion
■ (intervent* OR rehab* OR therap*) Inclusive expansion of column 5 of
the matrix.
Figure 27.8 A sample search query derived from Figure 27.7.
in a particular database field, such as the title. This will vary by the purpose for which the information
would reduce the amount of retrieval. was retrieved. For example, in conducting a gen-
eral literature review to refine a research question,
Step Four: How Well? one may include a wide range of studies that
employ different research methods. For a meta-
analysis (see Chapter 18) one may have very strict
There are two levels of analysis at this stage; they criteria related to sampling, the nature of the inde-
concern: pendent variable, presence of a control group, and
• How well the information retrieved fulfills one’s so forth.
needs, and
• The reliability of the information items. Searching the World Wide Web
Although published literature often undergoes Web Search Engines
some form of a pre-publication review process
Web search engines must comb a vast information
(e.g., peer review), it is important for searchers to
space that grows larger every day. Unlike compar-
develop their criteria for critical thinking skills.
atively small, well organized, and well structured
For example, one may establish the criteria that all
bibliographic citation databases, the WWW lacks a
the articles one wants must employ random sam-
systematic schema of organization. Therefore,
pling of subjects or that they have a minimum sam-
search engines used for searching the WWW have
ple size.
several distinguishing characteristics. Kluegel
Critical evaluation requires that one select,
(2001) has suggested that Web search engines are
weigh, and apply appropriate criteria to the infor-
in reality comprised of three parts:
mation retrieved. For purposes of critically evalu-
ating reports of research, the various discussions of • The crawler,
methodological rigor in this text are a useful • The index, and
source of criteria. In the end, the criteria one uses • The search engine itself.
The search engine crawler is a software tool tively improving the search recall. Examples of
that continually traverses the WWW looking for metasearch engines are Vivisimo (http://vivisimo.
new information content and for information con- com/), metacrawler (http://metacrawler.com/), and
tent changes. Information that the crawler locates DogPile (http://www.DogPile.com/).
is stored in an index. The search engine searches
the index for query matches to terms that one Web Directories
enters. The way in which specific Web search
engines identify, select, and prioritize information Web directories are organized lists of WWW
content for retrieval display is commonly propri- information content. The typical organizational
etary knowledge, often patented by software structure is hierarchical. Unlike Web search
designers. The most important thing to remember engines, Web directories do not use crawlers to
is that no one Web search tool has yet indexed or seek out or refresh information content, relying
catalogued the entire WWW. An example of a Web instead on Web authors to register their Web page
search engine is Google (http://www.google.com/), or Web site with the directory. Most use a search
one of the most extensively used search engines. engine to find a suitable subject point of access for
searchers to subsequently browse from by clicking
Web Search Tips. Like searching bibliographic links. An example of a Web directory is Yahoo
databases, it is important to apply logical strategies (http://www.yahoo.com/), a well known Web
for searching the Web. Here are some general con- information portal, providing links to many differ-
siderations for using Web search engines to search ent kinds of information and commercial sites.
the WWW:
• Utilize the Help function to identify search func-
tionality and proper syntax.
Information Resources
• Use topic-appropriate keywords. Relevant to Occupational
• Add more search terms to reduce retrieval.
• Check spelling; although some Web search tools Therapy
may prompt for spelling errors, not all search
engines have this function (this tip applies gener- Ultimately, a wide range of resources are poten-
ally to all search queries). tially relevant to occupational therapy. In this sec-
• Look for an Advanced interface that gives more tion, we discuss some of those most frequently
control to the searcher. used by occupational therapists.
• Use phase searching to increase specificity and
reduce retrieval size. Medline
• Check for Boolean and nesting functionality; Medline is a bibliographic citation database with
when entering multiple search terms, the default more than 13 million references from 4800 bio-
connector is usually “AND.” Check if “⫹” and medical journals and magazines covering the
“⫺” can be used to force inclusion or exclusion period from 1966 to present (plus some articles
of terms, respectively. published between 1963 and present). This data-
• Look for word stemming or truncation function- base can be searched free of charge using the
ality to increase recall. National Library of Medicine (NLM) PubMed
• Use the text search function in your browser to system at http://pubmed.gov. It is searchable using
find text in the pages you select for examination; keywords or the (NLM) controlled language,
hit ctrl-f to open the search text box. MeSH (National Library of Medicine, 2004). The
Metasearch Engines MeSH hierarchical tree structure includes “occu-
pational therapy” as a precoordinated terminal
Metasearch engines provide a single search inter- subject heading under the subject heading
face for a cluster of individual search engines. The “Rehabilitation,” as shown in Figure 27.9. There
search terms one enters are transmitted to each of are approximately 4,000 information items with an
the component search engines. The resultant MeSH major topic focus of “occupational therapy”
retrieval is usually categorized according to the in Medline. The NLM provides free access to
component search tool. Additional processing of Medline via its Web-based PubMed search inter-
the information retrieval may eliminate multiple face. Medline does not provide full text access
occurrences and organize the content in some fash- online to information items, but specific institu-
ion. Because individual search engines index only tions may offer links to electronic versions of jour-
portions of the WWW, metasearch tools are a way nal articles indexed in Medline as part of publisher
to expand the scope of information retrieval, effec- contractual arrangements.
Figure 27.9 PubMed MeSH subject heading note for “Occupational Therapy.”
CINAHL The Cochrane Library Databases

The organizational structure of the CINAHL bibli- The Cochrane Collaboration prepares and dissem-
ographic citation database resembles Medline, but inates systematic reviews of health care inter-
the focus of its information content lies in the ventions, published in the Cochrane Library, that
fields of allied health. In addition to identifying focuses mainly on controlled trials (Guyatt,
“Occupational Therapy” as a CINAHL subject Rennie, Evidence-Based Medicine Working
heading, there are nearly a dozen expanded OT Group, & American Medical Association, 2002).
variant subject headings that facilitate improved The main value of the Cochrane Library databases
information retrieval selectivity. Altogether, con- to occupational therapy is their potential as a re-
trolled language terms with occupational therapy source to guide evidence-based practice. There are
major subject headings translate into approxi- three main sections:
mately 6,000 information items that are relevant to
• The Cochrane Database of Systematic Reviews
occupational therapy. In view of this, CINAHL
(CDSR), which contains reports on systematic
should be considered a principal information
reviews and protocols,
resource for occupational therapy. Unlike Medline,
• The Database of Reviews of Effectiveness
CINAHL is a proprietary information resource that
(DARE), which are searchable systematic
requires institutional access. Moreover, unlike
reviews published outside Cochrane, and
Medline, CINAHL publication types include a
• The Cochrane Controlled Trials Registry
number of formats other than journal articles,
(CCTR), which includes economic and health
some with full text content. For the most part, links
technology assessments (Guyatt et al., 2002).
to electronic journal content are usually provided
by institutions’ contractual arrangements with Some institutions package the Cochrane data-
journal publishers. bases with ACP Journal Club, a publication of the
American College of Physicians, and Evidence- apy journals (published in English) as well as
Based Medicine, a joint publication with the journals related to the field. This database in-
British Medical Journal Group. The main focus of cludes only bibliographic information and the
Cochrane is clinical medicine. Nonetheless, at the abstract when there is one. The Wilma L. West
time this chapter was being written, a simple (WLW) Library, which is part of the American
search using the keyword “occupational therapy” Occupational Therapy Foundation, owns a copy
in the All EBM Reviews—Cochrane DSR, ACP of all the material indexed in OT SEARCH. If one
Journal Club, DARE, and CCTR database resulted is unable to otherwise locate material found
in the retrieval of nearly 500 information items. through OT SEARCH, one can contact the WLW
Unlike Medline and CINAHL, Cochrane does not library to arrange an interlibrary loan or a photo-
use a controlled language for selective information copy (for a nominal charge) where copyright
retrieval. In addition to searchable bibliographic restrictions do not prohibit it. OT SEARCH is
citation databases discussed in this section, Figure available on a subscription basis only. More infor-
27.10 lists a limited number of occupational ther- mation on the database can be found at http://
apy journals and Web sites that may provide value www.aotf.org.
as information resources.
Gray Literature Relevant
OT SEARCH to Occupational Therapy
OT SEARCH is a bibliographic database that Gray literature refers to information that is not
includes literature of occupational therapy and published or available in usual formats such as
related subjects. It includes monographs, proceed- journals. Examples of gray literature are abstracts
ings, reports, doctoral dissertations, master’s of conference papers, dissertations or unpublished
theses, and most international occupational ther- theses, and unpublished reports. Such literature
Occupational Therapy Journals and Web Sites:
Journals and Periodicals
Web Sites
• American Occupational Therapy Association (AOTA) http://www.aota.org/
• American Occupational Therapy Foundation (AOTF) http://www.aotf.org/
• Canadian Association of Occupational Therapists http://www.caot.ca/
• Illinois Occupational Therapy Association http://www.ilota.org/
• National Board for Certificaltion in Occupational Therapy (NBCOT)
http://www.nbcot.org
• British Association of Occupational Therapists/College of Occupational
Therapists http://www.cot.co.uk
• OTSeeker http://www.otseeker.com/
Figure 27.10 Some occupational therapy journals and Web sites.

may contain vital and specific information. Some to actively perform and refine searches, and to
organizations, such as the United Kingdom evaluate information retrieval.
College of Occupational Therapists, London
(http://www.cot.co.uk), have developed specialist REFERENCES
libraries of gray literature that are available to Association of College and Research Libraries (ACRL).
review on request. The Internet significantly aug- (2000). Information literacy competency standards for
higher education. Retrieved August 9, 2004 from
mented the availability of gray literature. However, http://www.ala.org/ala/acrl/acrlstandards/informationlit-
not all of this type information is electronically eracycompetency.htm
available. Bates, M. J. (1989). The design of browsing and berrypick-
ing techniques Retrieved August 10, 2004 from
Specific Content Web Sites http://www.gseis.ucla.edu/faculty/bates/berrypicking.
html
A number of Web sites are available that can pro- Dalrymple, P. W. (2001). Bibliographic control, organiza-
tion of information, and search strategies. In R. E.
vide access to various sources of information Bopp, & L. C. Smith (Eds.), Reference and information
(including publications) related to a specific topic. services: An introduction (3rd ed., pp. 69–96).
One example is the Web site for Sensory Englewood, CO; Libraries Unlimited.
Integration International (http://home.earthlink. Guyatt, G., Rennie, D., Evidence-Based Medicine Working
net/~sensoryint/), a nonprofit corporation related Group, & American Medical Association. (2002).
Users’ guides to the medical literature: A manual for
to the sensory integration model of practice. The evidence-based clinical practice (JAMA & archives
Web site lists publications relevant to sensory inte- journals) Chicago: AMA Press.
gration and provides a newsletter. Another exam- Jenson, J. D. (2004). It’s the information age, so where’s
ple is the Model of Human Occupation (MOHO) the information? College Teaching, 52(3), 107–112.
Kluegel, K. M. (2001). Electronic resources for reference.
Clearinghouse Web site (http://www.moho.uic. In R. E. Bopp & L. C. Smith (Eds.), Reference and
edu), which not only maintains an up-to-date bib- information services: An introduction (3rd ed.).
liography of publications related to the model, but Englewood, Colorado: Libraries Unlimited.
also has a search engine that can be used to iden- Lyman, I., & Varian, H. R. (2003) How much information?
tify published materials related to specific topics 2003 Executive summary Retrieved August 10, 2004
from http://www.sims.berkeley.edu/research/projects/
such as MOHO-based assessments and interven- how-much-info-2003/execsum.html
tions in specific practice areas. National Library of Medicine. (2004). FAQ: Finding med-
ical information in MEDLINE. Retrieved September 8,
2004 from http://www.nlm.nih.gov/services/usemed-
Conclusion line.html
Rieh, S. Y. (2002). Judgment of information quality and
cognitive authority in the web. Journal of the American
Skill in online searching is fundamental to infor- Society for Information Science and Technology, 53(2),
mation literacy, and a valuable asset for profes- 145–161.
Saranto, K., & Hovenga, E. J. S. (2004). Information liter-
sional development and practice. Occupational
acy—what it is about?: literature review of the concept
therapists seeking to improve information literacy and the context. International Journal of Medical
skills should begin by identifying a core set of use- Informatics, 73(6), 503–513.
ful occupational therapy information resources, Swanson, T. A. (2004). A radical step: Implementing a crit-
and then concentrate on mastering the intricacies ical information literacy model. Libraries and the
Academy, 4(2), 259–273.
of the search tools or interfaces required to access Taylor, A. G. (2004). The organization of information (2nd
their information content. Ultimately, the only way ed.). Library and Information Science Text Series.
to improve online searching skills is by continuing Westport, CT: Libraries Unlimited.
C H A P T E R 2 8
Generating Research Questions,

Design, and Methods
Marian Arbesman • Hector W. H. Tsang
The research question is a statement of the specific describes what the study will accomplish and what
knowledge that is being sought in the study. will be the value of the study.
Generating this question is the most important step A research hypothesis states the expected
of the entire research results of quantitative
process, as this directs study. Not all quantitative
the course of the subse- Research problems are studies will have stated
quent study, particularly hypotheses. Most com-
the design and methods
broadly defined gaps in monly, hypotheses are
of the investigation knowledge, while research stated when the analysis
(Bordage & Dawson, questions are narrow, will involve inferential
2003). Not surprisingly, statistics or when the
generating the question, focused, and specific ques- study is comparing a
design, and methods is tions intended to generate dependent variable across
one of the most chal- two or more groups.
lenging aspects of the knowledge to help close More specific discussions
research process. In this the gap. of hypotheses are found
chapter we examine key in Chapters 7 and 17.
aspects of this process The research design
and discuss the role of creativity in developing and methods refers to the specific strategies that
sound research questions, design, and methods. the investigator will use to answer the question or
questions that guide the research (or to test a
hypothesis where a hypothesis is included).
The Research Problem, Decisions about design and methods begin broad
and become more detailed as the specific investi-
Question, Purpose, gation is planned. Generally an investigator first
decides whether a study will be quantitative or
Hypothesis, Design, qualitative, or whether it will combine methods
and Methods from these two approaches. Then the investigator
makes a decision about the specific type of
research design such as whether the study will be:
Investigators usually begin the research process by
identifying their main area of interest. After this, • A mailed survey to a national sample,
they proceed to clarify a research problem, which • A participant observational study of a specific
is a gap in current knowledge within a specific setting,
topic area. Once the research problem is identified, • A single-subject design replicated across three
it provides the groundwork for defining specific clients,
research questions that can be answered in the • A series of focus groups with providers and con-
research study. Research problems are broadly sumers of service in a type of setting, or
defined gaps in knowledge, while research ques- • An experimental study with two independent
tions are narrow, focused, and specific questions variables composed of two different intervention
intended to generate knowledge to help close the approaches.
gap.
After delineating the research questions, one Once the decision is made about the study
clarifies the research purpose. The statement of design, the investigator will proceed to more spe-
research purpose should be precise and concise. It cific decisions and tasks such as selecting the data
Chapter 28 Generating Research Questions, Design, and Methods 453
collection instruments, determining the sample for als with severe mental illness. As a practitioner-
the study, planning the specific analysis and how it researcher, one of the authors had used both of
will be done, and determining logistical aspects of these approaches separately. Thus, the question
the study such as timing, costs, space, and equip- arose as to whether vocational outcomes would be
ment. Importantly these decisions are often itera- better if these two interventions were combined.
tive. That is, as one attempts to operationalize a This led to an investigation that examined an inte-
research question into specific methods and design, grated supported employment model that com-
new insights are gained that may lead to revision of bined both approaches (Tsang, 2003).
the research question. Sometimes as one proceeds
to plan the specific methods, it becomes apparent Occupational Therapy Theory
that the research design will not adequately answer
the question or that another question needs to be A number of conceptual models or theoretical
added to the study. Thus, while an investigator gen- frameworks have been developed in occupational
erally proceeds from the question to the design and therapy. Their theory often serves as the basis for
methods, it is not unusual to work back and forth developing a research question. Theories explain
between these tasks. phenomena and predict what things will happen
given certain circumstances. Researchers can use
theories to logically deduce expected observations
Uncovering the Research or to make predictions. Research questions derived
in this manner can be used to guide studies that test
Problem and Question occupational therapy theory. An example of how
theory directs research is provided later in the case
There are many ways an investigator can uncover illustrations.
a research problem and question. Later in this
chapter, we discuss a process for identifying the Directions of Professional
research question and translating it into a design
and methods for the study. Below are some key Organizations and Funding Agencies
sources of information that can help lead to identi- Local, state, and national professional organiza-
fication of a research question. tions (both for occupational therapy and for other
fields) develop strategic plans that may incorporate
Clinical Experience short- and long-term research goals. By examining
the goals and policy statements, whether or not
Clinical experience is a good medium from which they are related to research, one may discover an
to identify a research problem and related ques- approach that can lead to a viable research prob-
tions. For instance, therapists often encounter lem and question.
problems such as not having an assessment for a For example, in recent years, the aging popula-
specific purpose or not knowing which services tion of Hong Kong has increased. In response to
will best help a given client group. Therapists rou- this trend, the Hong Kong Occupational Therapy
tinely raise questions about clinical phenomena Association has put a great deal of emphasis on
such as: research and development of services for the eld-
• Is an assessment reliable or valid? erly population. A study on psychosocial function-
• Is a treatment approach or modality effective? ing of elderly people in Hong Kong was developed
• What accounts for different treatment outcomes in response to this development (Tsang, Cheang,
within a defined population? Tong, & Tse, 2004).
• Are there ways to improve the treatment outcome Another important consideration in the genera-
for a particular group? tion of research questions is consideration of fund-
ing priorities. As discussed in Chapter 30, funding
These types of clinical questions can be agencies often have mission statements or indica-
tremendously helpful to uncover relevant research tions of their overall funding priorities. Moreover,
questions. some funding agencies routinely generate requests
The following is an example of how a research for proposals that indicate research problems they
question emerged from practice. Psychosocial would like to see addressed in grant proposals.
skills training and a supported employment are two Paying attention to what funding agencies consider
intervention approaches that have been typically as priorities for research can be helpful not only in
used to improve vocational outcomes of individu- generating ideas, but also in identifying research
problems and questions for which there is likely to shows examples of research questions reflecting
be funding support. quantitative and qualitative approaches.
Similarly, every research question implies a
Literature Review research design best suited to answer it. For exam-
ple, if the research question asks about the char-
Reading the literature is key to developing any
acteristics of a particular group, a descriptive
research question. There are many ways that one
research design will be appropriate. If the question
can find gaps in knowledge that will lead to a
asks about whether two groups experiencing dif-
research question. For example, reviewing a study
ferent treatments would have different outcomes,
involving a specific population or context may trig-
an experimental design is required (Portney &
ger the question as to whether a described approach
Watkins, 2000).
can be applied to different types of clients or in
Different research questions also require differ-
another setting. Or, one may recognize inconsis-
ing amounts and kinds of resources to answer
tencies of findings across several studies that point
them. Identifying available resources is important
out the need for further research efforts to account
in designing your research. Questions should be
for these inconsistencies.
formulated with an understanding of the kinds of
funding, space, instruments, personnel, access to
participants, and other resources that will be nec-
Narrowing Down the essary to answer the question.
Research Focus As discussed in Chapter 29, the consideration
of participants’ rights needs to be foremost in the
development of any research protocol. Ethical
After identifying the general problem and question
issues and privacy regulations such as the Health
for the study, one must refine the question to
Insurance Portability and Accountability Act in the
become practical and manageable. As noted earlier,
U.S. (HIPAA, 2004) may constrain the kind of
the research question will serve as a framework
questions that can be asked.
working out the design and methods of the study.
Since the research question implies many elements
of the actual study, one must think forward to these
when developing the research question. The Role of Creativity
in Developing Research
How one frames the research question will, first
of all, determine whether a quantitative or qualita-
tive approach is most appropriate to answer the
research. As discussed in previous chapters these
Questions, Design, and
two traditions of research involve very different Methods
assumptions about the research process and
include different approaches to assuring the The research process can appear to be the antithe-
dependability and validity of findings. Table 28.1 sis of creativity. Nonetheless, creativity can be a
Table 28.1 Examples of Quantitative and Qualitative Research Questions

Quantitative Research Questions Qualitative Research Questions
Will children identified with tactile defensiveness What is the experience of children with tactile
assigned to 6 weeks of therapy based on defensiveness when they participate in
sensory integration show greater reduction of occupational therapy based on sensory
their tactile defensiveness when compared to integration principles?
children assigned to receive play-oriented
therapy?
Is higher confidence for returning to work related to What is the perspective on working among persons
an increase in job-seeking behavior among with AIDS whose work has been interrupted,
persons with AIDS? and what are the range of concerns they
express about returning to work?
Will persons living in nursing homes report fewer What happens to the occupational roles of persons
occupational roles than a matched samples of when they enter a nursing home?
persons living in their homes?
The Creative Process and Brainstorming
Descriptions of the creative process fall into two • Creativity incorporates a disciplined evaluation
camps. The first describes creative problem solving component to determine the utility of any idea
as taking place within the unconscious, involving and thus converge on the idea or ideas that have
incubation of an idea followed by sudden illumina- the greatest usefulness.
tion that produces an “Aha!” or “Eureka!” (Simon,
1981). The second sees creativity as dependent on
a conscious, systematic process and asks, “How
Brainstorming
can we develop strategies that enable us to come The best known of the divergent tools is “brain-
up with more creative and innovative research storming,” a term coined by Osborn (1979).
ideas?” In time, researchers may have their share Brainstorming requires deferring judgment while
of spontaneous inspirations but most of the creative striving for a large number of ideas including
process in research involves the conscious, system- those that are free-wheeling and wild along with
atic use of strategies that support creativity. conservative ones. There are many ways to engage
in brainstorming related to research. When doing
Definitions of Creativity individual brainstorming, the process of coming
up with new ideas can be augmented by using the
Ackoff and Vergara (1988) and Sternberg and Forced Fitting tool (i.e., spending a minute or two
Lubart (1999) point out that: looking at objects that have no obvious link to the
• Creativity is the process of producing something problem that one is working on) (Isaksen et al.,
that is both original and valuable, and 2000). Brainstorming in groups can enable mem-
• The development of a creative idea is a process bers to piggyback or build on the ideas of others.
that can be developed and enhanced, not just Evidence indicates that electronic brainstorming
something that occurs by chance. (i.e., requests for ideas sent out by email requests)
can successfully generate new ideas (Dennis &
Creativity also involves two complementary Valacich, 1993). This process may be highly effec-
divergent and convergent processes: tive if one is working as part of a research team
that is spread out geographically or when one
• Openness to new ideas combined with efforts to wants to consult with international experts in a
generate as many divergent ideas as possible particular area.
(e.g., brainstorming), and
powerful tool in developing the research question, Components and Stages

design, and methods. There is a long history of of the CPS Process
creativity in the scientific and social science
research literature (Bohn & Peat, 1987; Popper, Originally developed by Osborn (1979), the CPS
1959; Root-Bernstein, 1997) process has been refined into six stages that make
up three main components (Figure 28.1) (Isaksen,
The Creative Problem- Dorval, & Treffinger, 2000; Lewin & Reed, 1998).
Solving (CPS) Process Each stage of the process incorporates tools that
The Creative Problem-Solving Process (CPS) is a facilitate divergent or convergent thinking. Applied
specific disciplined approach to creativity that can to the task of generating a research question, diver-
enhance one’s ability to generate creative ideas gent thinking tools allow one to generate many
and solutions (Noller, 1977). It is a way of looking ideas and to speculate about possibilities related to
at problems that combines: a given research topic. Convergent tools enable one
to select from the ideas and possibilities generated
• The generation of many new ideas, and and to formulate a focused research topic and gen-
• The critical thinking needed to decide whether or erate an appropriate research question.
not a given idea will work. In the sections that follow, we apply the CPS
It also provides the tools to determine how a process to the task of generating research questions
specific idea can be put into practice. In this section and deciding on the methods for addressing the
we examine how the CPS process can be applied to question or questions that are finally selected. The
the task of generating a research question and feature box on the next page titled “An Example of
selecting the research design and methods. the Creative Problem-Solving Process” illustrates
• Constructing Opportunities (looking for identifying some

sort of gap(s) that needs to be filled)
• Exploring Data (learning what is known and what is
unknown and how investigators go about framing and
answering research questions)
• Framing Problems (refining the research problem so it
can be solved)
• Brainstorming ideas for questions, design, and methods

and selecting from them those that have the most potential.
• Developing Solutions (consideration of how the research

will be conducted and how all the tasks will be
accomplished).
• Building Acceptance (recognizing the limitations of a
particular study while still clearly identifying its value).
Figure 28.1 The creative problem solving process applied to generat-

ing a research questions and translating them into research plans.
An Example of the Creative Problem-Solving Process
The following is a hypothetical example of an By engaging in these activities, our hypotheti-

occupational therapy researcher who is generally cal occupational therapy researcher may identify a
interested in the topic of head injury. particular interest in acute care and early rehabili-
tation services. It would also become apparent
Understanding the Challenge that the main services in this area are focused on
cognitive–perceptual problems and on problems of
Constructing Opportunities motor coordination. Furthermore, input from prac-
During this first phase, an occupational therapy titioners and from reading would identify that
researcher interested in head injury might: there is a relatively new, motor control approach in
this area for which there are not yet detailed
• Read some key chapters in occupational therapy assessment procedures or interventions. Thus, as a
texts to identify what are the main types of serv- result of this phase, the researcher decides that the
ices offered, and what theories guide services in lack of assessments and intervention protocols
this area; related to motor control represents important gaps
• Identify one or more settings where occupational in knowledge.
therapists are providing services to persons with
head injury in order to observe and discuss inter- Exploring Data
vention with therapists; Having identified these potential gaps in knowl-
• Attend workshops in which experts discuss prac- edge, the investigator would next begin to gather
tice and research in the area, and/or systematic information by identifying and review-
• Begin to read some studies related to occupa- ing the major texts and research articles in the area
tional therapy and head injury in order to identify of motor control. This review would identify, for
different approaches to research in this area. instance, that most research in this area has
(continued)
456
An Example of the Creative Problem-Solving Process (continued)
focused on controlled laboratory experiments that to research. However, several problems are identi-
examine theoretical ideas behind the motor control fied related to this option. First of all, such a study
theory. For instance, these studies have shown that is costly and given that there is little research in the
involving a person in a real life task produces dif- area, it may be difficult to secure funding. Second,
ferent quality movements than having the person the motor control approach is based on “individual-
attempt to perform movements outside a meaning- izing” the intervention for each client, which
ful task. It also reveals that there is very little re- would make it challenging to define a single inde-
search that has applied this approach to an actual pendent variable for the experimental group. For
clinical situation. Discussions with clinicians show this reason the investigator begins to focus on the
that they are particularly interested in research that idea of using a single-subject design with a few
would demonstrate the impact of this approach on clients. Consequently, the following first draft of
outcomes. the research question is formulated:
However, exploring data can also involve con-
sulting experts (clinical and research) in the topic Will the introduction of motor control services
one is exploring. demonstrate an improvement in a client’s
functional capacities?
Framing Problems
After considering the literature and feedback from Developing Solutions and Building Acceptance
practitioners, the researcher decides to frame the Once the investigator has reached the tentative
problem in the following statement: research question noted above, developing solu-
tions to addressing the question would involve
Many practitioners use an approach to serv- the following kinds of considerations:
ice based on concepts of motor control.
Where there is substantial basic science evi- • Where is a head injury service where one could
dence to support the theory underlying this undertake the research (i.e., where there would
approach, there is limited evidence about its be administrative support and therapists’ willing-
impact on improving functional outcomes of ness to participate in the study)?
persons with head injury. This study will aim • What kind of single subject design is feasible in
to generate such knowledge. the specific head injury service?
• What single subject design would provide the
Generating Ideas most rigorous findings?
At this stage, the investigator goes back to the liter- • Will the study design be replicated across several
ature. Since there are few studies of the motor con- clients?
trol approach in practice, the investigator decides • What kind of functional measure could be used
to examine ways that other researchers have stud- in the context of the study (i.e., a measure that
ied the outcomes of other types of occupational could be easily administered several times since
therapy interventions. This literature identifies a single-subject designs require repeated meas-
number of group studies that examine the impact ures)?
of different interventions on outcomes. Also the
As the answers to these and other detailed
investigator finds some examples of single-subject
questions become apparent, the researcher would
designs that have been used to examine the impact
further refine both the study design and the ques-
of interventions. The investigator also contacts two
tion (and possibly a hypothesis) to operationalize
investigators in the area of motor control via e-mail
the question in an experimental context. As these
and exchanges ideas with them. Finally, the investi-
decisions are made to refine the study question and
gator engages in a brown bag discussion over
design, the investigator and others (e.g., practition-
lunch with a group of practitioners working in the
ers) who may have been involved in earlier stages
area of head injury who use the motor control
of planning the research will have to recognize and
approach. As a result of this process, several alter-
accept that the study is appropriate given context,
native ideas for research are identified including an
resources, the state of existing knowledge, and
experimental study that compares the motor con-
other factors.
trol approach to another more traditional approach
this process by using a hypothetical example of a • Constructing Opportunities,

researcher who is interested in occupational ther- • Exploring Data, and
apy for persons with head injury. • Framing Problems.
Understanding the Challenge. The initial com- Constructing Opportunities. The first stage, con-
ponent of the process, Understanding the Chal- structing opportunities, refers to the process
lenge, consists of three stages: of identifying potential areas where gaps in
knowledge may exist. This stage may involve, for of the research literature in a particular area is
instance, talking to clinicians or researchers about important. It will identify how other researchers
a topic to identify potential areas where knowledge are going about solving the problem (i.e., what
is needed. It can involve initial reading of the liter- types of questions they address, what design and
ature, attending lectures, workshops, or colloquia methods they use in their investigations).
in the area where one is potentially interested. It If the problem one has generated is an applied
may involve observing practice in the area of inter- problem for which the investigation is likely to
est. At this stage one aims to identify a range of involve practitioners and clients and for which one
potentially fruitful avenues for research based on expects the research results to have practical sig-
gaps in current knowledge (i.e., the potential nificance, it is important to discuss possible solu-
research problem). tions with these stakeholders. Informal meetings
and discussions or formal focus groups can be
Exploring Data. Once one or more potential gaps
used to achieve this end.
in knowledge are identified, one needs to get a bet-
After generating a range of ideas about how the
ter handle on the nature and scope of problem that
research problem might be addressed (i.e., types of
might be addressed. To do this one needs to gather
questions that might be asked and designs and
systematic information regarding the problem one
methods that might be used to answer these ques-
is considering. This stage is called exploring data.
tions), one must engage in a focusing phase.
It is crucial to the development of a research ques-
Focusing involves selecting ideas that have the
tion since it is here that one learns what is already
most potential for solving the problem. At this
known about the potential problem. Exploring the
stage, investigators will typically begin to draft a
data always involves a review of the literature,
specific research question and outline the antici-
which is discussed in more detail later. However,
pated design and methods. These will be subject to
exploring data can also involve consulting experts
refinement in the stages that follow.
(clinical and research) in the topic one is explor-
ing. The aim of this step is to develop a compre- Planning for Action: Developing Solutions and
hensive understanding of an area that focuses on: Building Acceptance. Once a potential research
question has been identified to address the research
• What knowledge is already available,
problem, it is time to put energy into refining and
• How questions are ordinarily addressed and
developing a workable research protocol. This is
framed in this area of study,
the component of planning for action. There are
• Where existing knowledge leaves off, and
two stages in this component:
• What gaps in knowledge are considered most
important by those involved. • Developing Solutions, and
• Building Acceptance.
This step can often result in generating a num-
ber of possible problems that could be addressed in This step of developing solutions involves
a study, as well as different approaches that might designing the actual research protocol. It involves
be taken to address them. consideration of how the research will be con-
ducted (design and methods) and how the study
Framing Problems. The next step, framing prob-
will be implemented (logistics). The logistic fac-
lems, involves refining the issue that will be
tors include such details as:
addressed in the study in such a way as to make
clear exactly what the problem is that will be • Establishing deadlines for all the necessary steps,
addressed. The process that results in this third • Planning for how to compensate participants, and
stage is critical to planning any study, since it is • Determining how data will be entered into sys-
impossible to come up with an effective solution to tems for data analysis.
a research problem if the problem is not clearly
Thinking through these details allows one to
identified and translated into a workable research
determine the feasibility of the study. It also allows
question.
one to make modifications in plans so the research
Generating Ideas. After a problem statement is will be feasible.
selected, one moves on to the next CPS component While the notion of building acceptance may
(and stage) known as generating ideas. At this not intuitively seem necessary for the process of
stage, investigators first seek to generate a range of planning research, it is an important step. Often
ideas concerning how one might go about address- researchers (individuals and research teams) begin
ing the problem in a study. Once again, a review with high aspirations for the research process that
must be tempered in light of logistics, available • Have adequate support from mentors, collabora-
resources, and the nature of the knowledge in the tors, and institutions involved,
area. Inevitably, designing a study involves mak- • Able to be completed, and
ing compromises between the ideal study and the • A satisfying process for those involved.
one that is possible given actual circumstances and
resources. In part, acceptance means realizing the
limits of the study one will undertake while still Tools for Decision-Making
clearly identifying what will be valuable about the A variety of tools have been developed to system-
study. When a team is involved in the research, this atize the kind of decision-making that underlies
process is important to generating “buy in” for the planning a study. One tool that uses criteria
study. to make a systematic choice is the Evaluation
Matrix (Isaksen et al., 2000; Lewin & Reed, 1998).
Choosing the Best The first step of the Evaluation Matrix is to make
a grid with rows and columns. Along the columns,
Research Question, one would put all the criteria, and the potential
Design, and Methods choices would be put along the rows. Using a
scale of 1 to 5, with 1 being poor to 5 being excel-
lent, one then rates each option under considera-
Early in planning a study, one may be left with tion. By comparing each option across the criteria,
more ideas than one could complete in a typical one can better understand the implications of
lifetime. Consequently, one needs to select the idea each choice. Table 28.2 illustrates a matrix that
or ideas that best fit a particular situation. Develo- might facilitate the decision making when consid-
ping criteria helps one to clarify which research ering three different design options for studying
questions, design, and methods might be the best outcomes of occupational therapy services based
to pursue. These criteria may include whether the on motor control concepts in the area of head
study question will be: injury.
• Important, In using the Evaluation Matrix, it is important
• Answerable, to evaluate each criterion across the choices. This
• Feasible based on skill, background, time, money, is done to avoid a potential halo effect that could
available participants, and additional resources, occur if one has a favorite option. When the grid is
Table 28.2 An Evaluation Matrix for Considering Different Design Options for a Study of Functional
Outcomes of Occupational Therapy Services Based on Motor Control for Persons with Head Injury
Survey of Therapists Two-Group Experiment Single-subject Design
About Their Comparing Motor to Examine Impact of
Perceptions of the Control to Other Motor Control on
Value of Control Approaches Functional Outcomes
Approaches Across Three Subjects
Can study be accom- 4 1 5
plished with a small
seed grant (i.e., $5,000)?
Can study be completed in 5 2 5
12 months?
Will study contribute 2 5 4
rigorous evidence about
impact of motor control func-
tional outcomes?
Can study be done on 1 3 5
local rehabilitation unit with
an average of 45 clients
and 3 therapists?
Note: 1 = poor fit; 5 = excellent fit.

complete and all the criteria have been evaluated, ered in terms of the quantity and quality of the
one can have a general picture of the value of each literature.
choice.
A careful decision-making process can often
Finding the Right Literature
leave one with few or no valuable potential choi-
ces. In this situation, another valuable tool is the There is no specific formula for how much litera-
Advantages, Limitations, and Overcome the Limi- ture review is necessary. The expectation for the
tations (AL-O) process (Lewin & Reed, 1998). By depth and breadth of the review will vary based on
taking each choice and generating its advantages the topic and the type of study one is planning. A
and limitations, one has the ability to see the review of the literature may be somewhat limited,
choice through new eyes. Through the generation for example, if the area is clearly defined and
of ideas to overcome the limitations to a given research on the topic is quite focused. On the other
research question and design one can creatively hand, if one is examining a new area or an area
problem solve how to make a given option more where extensive work has already been done, it
viable. In a sense, the AL-O process helps one to may be necessary to do a very large literature
move directly to planning the research protocol review. In the former case, one may need to explore
since one is able to see what will support and what a number of related areas to the new area to gener-
will be a barrier to a given research option. ate ideas about how to approach the new area cho-
sen for the study. In the latter case one would need
to do a literature review of sufficient depth to make
The Literature Review sure one is not asking a question for which there
already is substantial evidence.
One should continuously review the literature as In all situations, beginning the search with a
the research question, design, and methods are rather broad approach to the topic will allow the
being developed. Understanding background infor- researcher to get a picture of how the topic is
mation, past research, and theoretical concepts help understood by other researchers. An example of
to make sure that the correct research question is this can be seen in the area of falls in the elderly.
being formulated. While it may be relatively easy At the start of searching in this area one might
for the researcher to know where to begin the broadly examine risk factors for all elderly per-
search for background information, it is often diffi- sons. As one’s understanding of the area deepens,
cult to know that enough literature has been cov- one would be aware of the different risk factors
Table 28.3 Checklist of Things to Consider When Planning the Design and Methods
of a Research Study.
Dates for Task—
Task Completed
Deadlines
Select appropriate research type (qualitative or quantitative).
Select type of study design.
Identify the study group.
Formulate a plan to select study participants.
Obtain approval to carry out research at organization, if necessary.
Organize appropriate resources (money, support, time, knowledge) to
carry out study.
Apply for ethical approval of study.
Complete thorough literature review that can demonstrate importance
and need of proposed study.
State research problem/hypothesis in format approved by
department/research committee.
Complete thorough description of methods section that can describe
research process to research group and other readers.
Develop timeline for study.
among community-based falls, falls to hospital- • Asking how appropriate a given source is to the
ized elderly, and falls that take place in a nursing selected topic.
home. With this framework in place, one may then
limit further searching to a more specific area such The quality of an article, book, or chapter is
as falls that take place in nursing homes. based on many factors. For example, if an assess-
Another factor that determines the type of liter- ment tool is involved in a study, is the reliability
ature review is one’s own understanding of the and validity of the instrument discussed? If a study
research topic. The novice researcher will need to examines the effectiveness of a given intervention,
spend time uncovering background information to are the reported results consistent with the type of
make sure that all areas are covered. While this study design? It is not possible to exhaustively list
process may involve following up on what appear all the things one might consider in evaluating an
to be many unimportant paths, it is necessary to article. Several sources of information in this text
keep digging a bit deeper to make sure that a are helpful to this end. First of all, the chapters dis-
strong web of understanding of the topic has devel- cussing various methods provide the rationale and
oped. For the more seasoned researcher, the litera- the criteria for rigor in studies. Second, the discus-
ture review will most likely focus on providing an sions of evidence-based practice in Chapter 41
update on the latest information. It will also serve also discuss criteria that can be used to analyze
to challenge experienced researchers to broaden research reports.
and deepen their understanding of the research Deciding the relevance of each piece of litera-
question/hypothesis so that he or she does not rely ture to the research topic can be somewhat com-
only on past comprehension of the topic. plicated by the fact that the research topic often
It is important to examine literature from a vari- changes in scope as the literature review process
ety of sources, and, in most instances, to examine unfolds. For example, one might begin with a very
references that span more than a single discipline. broad topic, in which case a wide range of articles
Following along on the example of falls in the eld- are relevant. As the literature is reviewed and the
erly, one would find that the literature spans occu- topic narrowed, a smaller subset of articles will be
pational therapy, physical therapy, social work, identified as relevant. In some instances, the
exercise physiology, public health, and nursing lit- research topic expands, necessitating that an ini-
erature. tially narrow search be broadened.
Finding the appropriate literature for a litera- When reviewing research literature, it is a good
ture review involves a process of searching the lit- idea to characterize each article by:
erature. Using appropriate search strategies can
• Summarizing the research question, and
ensure that one’s literature review has sufficient
• Describing key elements of the methods, results,
breadth and depth. The process of searching the lit-
conclusions, and implications.
erature is described in depth in Chapter 27.
Doing so not only clarifies the important points
Synthesizing the Discovered Literature of an article, but also creates a record to which one
Synthesizing the discovered literature requires two can return when writing up the literature review.
interdependent steps: There is no substitute for carefully examining
each piece of literature included in the literature
• Critically analyzing each article, book, or chapter review. Reading and reviewing an article more
found, and than once may allow the reader to see more of
• Figuring out how to present the information the limitations and strengths of the article. It will
from the literature in a coherent and influential also permit the researcher to more easily make
manner. the determination if an article fits the research
question.
Staying focused on the task of refining the
In some cases, writing the research question out
research question helps to focus and integrate these
and having it available when deciding if an article
two steps. The evolving research question should
fits may be enough to move the process along. For
drive the critical analysis and development of the
some researchers, or for some types of research
framework for the presentation of the information.
questions, it may be necessary to create a schema-
Critical analysis of the literature involves:
tic or map of the research question. The map will
• Assessing the quality of an article (e.g., compre- look at relationships in the question, what is
hensiveness in a review article, or rigor in a known and what is not known. It can also include
research article), and the relationship to the theoretical framework. A
schematic map allows the researcher to figure out should lead naturally to the choice of research
what is needed and will enable the individual to question, design, and methods.
understand the parameters of the research question
and to easily answer the question “Does it fit
or not?”. Figure 28.2 is an example of a schematic
map used for the development of a research Case Illustrations: Identifying
question.
Another way to create a structure for informa-
the Research Question,
tion is to use a table format to summarize the Design, and Methods
results found in the most important supporting lit-
erature. This framework can then be used to write
the literature review. Table 28.4 provides an exam-
Case One: Studying Attitudes Using
ple of how the articles to be included in a literature Conjoint Analysis
review can be summarized in tabular form. Constructing Opportunities
Whether the researcher is involved in quantitative
or qualitative research, the two main goals of a lit- The study of public attitudes toward disability,
erature review are to be accurate and to be persua- especially regarding people with severe mental
sive. If one is accurate but not persuasive, one will illness, has been an important research topic in
be left with a literature review that is technically the occupational therapy and related literature.
good but does not convince the reader that the Researchers have been particularly interested in
topic has merit. If the review is fervent but without the attitudes of occupational therapy professionals
substance, the same readers will have the impres- and students toward people with disabilities. My
sion that the author was attempting to get by on his personal experience as an occupational therapist
or her passion. A good literature review should and the findings of previous studies indicated that
lead one down a well-structured path that con- negative attitudes toward people with disabilities
vinces the reader of both the completeness of one’s affect the successful rehabilitation, integration,
examination of the literature and the merit of the and independence of people with disabilities
research question. Finally, the literature review (Antonak & Livneh, 1988).
Characteristics
of nursing home
General residents
background
on aging
Visual Fall
changes in prevention
aging programs
Visual Nursing Fear of

attention/ home falls falling
processing
changes Visual
with aging attention/
processing
program Risk
factors for
falls in a
nursing
home
General
risk
factors for
falls
Figure 28.2 Schematic of literature review for research question: Does

participation in a visual attention/visual processing program reduce
falls in nursing home residents?
Table 28.4 Example of Table for Summarizing Articles to Be Used in a Literature Review
Author/Year/ Section Design/ Intervention and
of Literature Review Study Objectives Participants Measures Used Results
Owsley & To examine the Cross-sectional Survey as part of Lower scores on visual
McGwin, 2004 association 342 older prospective study on attention/processing
Visual attention/ between adults (aged mobility. speed were
processing visual 55–85) living Measured visual significantly related
changes with attention/ independently attention/processing to poorer scores on
aging processing in the speed, performance performance mobility,
speed and community. mobility, self-reported after adjustment for
mobility in Recruited from measures of falls, fall age, sex, race,
older adults. primary eye efficacy, mobility/ education, number of
care practices. balance, and chronic conditions,
physical activity, cognitive status,
demographics, depressive
health, and func- symptoms, visual
tional information. acuity, and contrast
sensitivity (P =.04).
Becker et al., To evaluate the Randomized con- Staff and resident The incidence density
2003 effectiveness trolled trial education on fall of falls for the
Fall prevention of a multi- (randomized by prevention, advice on intervention group
programs faceted fall clusters) 1981 environmental was slightly more
prevention nursing home adaptations, than half of that for
program. residents, 60 progressive balance the control group.
years of age and resistive training, There was no
and older. and hip protectors. difference between
Number of falls and both groups for
fractures. number of fractures.
Exploring Data and Framing Problems from and collaborate with other disciplines. At one
point, I discussed my concerns regarding the limi-
My personal interest in this area prompted me to tations of the traditional methods of studying
systematically review the literature, which attitudes with a professor from rehabilitation psy-
included Science Citation Index, Social Sciences chology at the University of Wisconsin–Madison
Citation Index, CINAHL, and MEDLINE. Results who was interested in the study of public attitudes
show that many researchers have questioned the on individuals with mental illness. He introduced
utility of using overt and obtrusive direct methods me to an indirect measure market research tech-
to measure attitudes using the traditional self- nique called “conjoint analysis.” The virtue of con-
report and interview methods, especially when the joint analysis is that it asks the respondent to make
targeted attitude referent is emotionally loaded and choices in the same fashion as the consumer pre-
socially sensitive. The conscious or unconscious sumably does—by trading off features, one against
mechanisms of the respondent may interfere and another. Compared to the commonly used rating
alter his or her attitudes so as to conform to pre- and ranking methods, the innovative conjoint
vailing norms and socially sanctioned beliefs analysis is less obtrusive, and its task design simu-
(Livneh & Antonak, 1994). These concerns are rel- lates real-life considerations more closely. It pro-
evant in measuring public attitudes on issues vides attitude/preference scores that are more
related to the community integration of people realistic, less abstract, less declarative, and less
with severe mental illness. While I was fully aware tainted by social desirability (Shamir & Shamir,
of the limitations of the current techniques of 1995). As an indirect method to evaluate attitudes
measuring attitudes, I did not have a strategy that as relative to the combinations of characteristics of
could overcome this barrier. persons with disabilities, conjoint analysis also has
the advantage of being less affected by social desir-
Generating Ideas
ability than the traditional methods of assessment.
My personal experience of having a breakthrough This enhances the validity of research results from
in research on several occasions has been to learn well-educated healthcare professionals who were
the subjects of interests in this study (Shamir & The next step was to transfer the above into a
Shamir, 1995). Knowing its uniqueness and essen- workable research protocol. The main considera-
tial features, we brainstormed its potential uses in tions were the sample and sampling procedure, the
the field of rehabilitation. I had an intuition that it instruments to be used, the procedure for data col-
had great potential to be used to study attitudes of lection, and how the data should be analyzed to
disabilities in occupational therapy. To use the ter- answer the research questions (Tsang, Chan, &
minology of CPS, I identified new ideas and devel- Chan, 2004).
oped a new way of framing the traditional problem.
Case Two: Family Burdens
Developing Solutions and of the Mentally Ill Offenders
Building Acceptance Constructing Opportunities
With this creative idea in my mind, I performed a Another illustration of the process of developing a
systematic literature review again to see if there research question involved the development of the-
were previous attempts to use this innovative oretical model using a qualitative research design.
approach to study public attitudes of disabilities, Based on my past clinical experiences and dis-
and I was delighted to see that there were no such cussion with some experienced researchers in
studies. This confirmed that this new idea was the field, it came to my notice that a number of
original. My research question became: “Is con- research projects had been devoted to the study of
joint analysis applicable to the study of public atti- needs, responsibilities, and intervention programs
tudes toward people with disabilities?” of families and caregivers of psychiatric clients
To explore the application of conjoint analysis because of the emphasis on deinstitutionalization
in occupational therapy research, we first identi- and community integration. As I had worked as an
fied the scope and objectives of the study. We occupational therapist in a forensic psychiatric set-
planned to employ the conjoint analysis procedure ting for a number of years, I was fully aware that
to examine factors influencing occupational ther- the responsibilities for families when looking after
apy students’ context-specific attitudes toward a client with a criminal history were even heavier
people with disabilities. The relationships between and more complicated than for those without a
sex of the occupational therapy students and their criminal history.
preferences for people with disabilities as well as
the occupational therapy curricula effects on atti-
Exploring Data
tudes were also examined. Factors influencing
context-specific attitudes of occupational therapy Studies on identifying family needs and responsi-
students toward persons with disabilities were bilities for mentally ill offenders have been very
examined by studying their preferences for placing limited, however, and I recognized that this could
a residential rehabilitation facility in their own be a new area of research. I first completed a com-
neighborhood. This context was chosen as comprehensive literature review on family stress and
munity integration of people with disabilities is studies on family needs for psychiatric clients with
still a controversial and sensitive issue in Hong a forensic history with particular emphasis on
Kong (Cheung, 1990). the methodological problems of available studies.
A quantitative research design was employed Since research in this area was limited, I felt that
for the study (Heppner, Kivlighan, & Wampold, building a theoretical model could act as a concep-
1999) as it included research strategies that tual guide for subsequent studies to identify bur-
enabled the investigator to describe the occurrence dens of families of mentally ill offenders. Based on
of variables, the underlying dimension in a set of the literature review and my own experience, I
variables, and the relationship between or among postulated a model (Figure 28.3) suggesting that
variables. Specifically, passive research designs the core source of stress of families of the mentally
(e.g., ex post facto designs and multiple regres- ill offenders appeared to be the criminal offense
sion) were considered to examine complex rela- itself. A secondary source of stress and burden
tionships among variables. In this study, conjoint were the events associated with the offense,
analysis (a nonparametric multiple regression) was including the court proceedings; dealing with the
used to determine the effect of disability-specific police and the media; and admission to a special
and other demographic variables on attitudes hospital. The crime and the secondary events in
toward people with disabilities and an ex post facto turn add to the severity of the burdens to the fami-
design was used to determine the sex and curricula lies along the four classical dimensions (Tsang,
effects on attitudes. Pearson, & Yuen, 2002).
Features of a Good Research Question
The following are elements of a good research • The question should be operational, that is, it
question: should not be too abstract and should refer to
actual variables under investigation.
• The question should clearly state what is being
described, compared, or contrasted in the study. The table below illustrates research questions
• The question should correspond to the type of that conform or fail to conform to these elements.
research method and design that will be used
(e.g., type of study, sample, planned analysis).
Strong Research Questions Weak Research Questions

Do therapists who use the Occupational Performance Is the Occupational Performance History
History Interview view it as providing information Interview a good assessment?
helpful to understanding the client, treatment
planning, and discharge planning?
Are lower scores on the Functional Index Measure at Does functional level determine where patients
discharge related to type of discharge setting (home, are discharged to?
nursing home, assisted living) in persons with
cerebrovascular accident?
Will young adults (18–30 years) diagnosed with bipo- Would an occupational therapy follow-up pro-
lar disorder attending a monthly occupational therapy gram reduce rehospitalization?
follow-up program following discharge have less
rehospitalization rates in the 12 months post-discharge
than a control group that does not attend this follow-
up program?
What factors do occupational therapy clients with How do clients with AIDS think occupational
AIDS enrolled in a vocational rehabilitation program therapy helps them return to work?
find most helpful to their return to work?
Symptom-specific burden
Court proceedings
Criminal
Emotional offense Social
Dealing with + Dealing
burden policemen with media burden
Mental
illness
Admission to a
special hospital
Financial burden
Figure 28.3 Stress and burden for families of forensic psychiatric clients.
465
Figure 28.4 Dr. Tsang (right, holding the microphone) discusses a

collaborator research with researchers of other disciplines from
Chicago, Illinois and Beijing, China.
Framing Problems pants and the results were published in a peer-

reviewed journal (Pearson & Tsang, 2004).
With the theoretical model, we worked out the
research questions. What were the experiences and
needs of families of forensic psychiatric patients in
Hong Kong with particular respect to their contact Conclusion
with the police, courts, media, and service pro-
viders? What suggestions for service provisions This chapter discussed the process of generating a
could be made after the needs of the families had research question, and deciding design and meth-
been identified (Figure 28.4)? ods. We emphasized the extent to which this is a
creative, if disciplined, process. While there is no
Generating Ideas and Developing Solutions. single pathway to developing a sound and doable
My collaborators and I brainstormed the different research question and to designing rigorous
methodologies that we might use in answering the research to answer it, one important rule of thumb
above questions. Based on the fact that this would is to employ the process of divergent and conver-
likely be a pioneering and exploratory study, we gent thinking. The former is useful for generating
decided to use a qualitative approach. In-depth opportunities, options, and ideas. The latter is nec-
interviews were planned to be conducted among essary for choosing between these various ele-
sufficient numbers of participants using a semi- ments in making good decisions that lead to a
structured interview schedule. Another problem sound and valuable study.
we faced at that time was whether there was an
appropriate assessment tool. The Relative Assess-
REFERENCES
ment Interview was found to be appropriate. How-
ever, modification was needed to include extra (2004). The Health Insurance Portability and Accountabi-
lity Act.
items related to the forensic nature of the patients. Ackoff, R. L., & Vergara, E. (1988). Creativity in problem
We therefore again read the literature on research solving and planning. In R. L. Kuhn (Ed.),Handbook
methods involved in scale development and modi- for creative and innovative managers (pp. 77–89).
fication. We eventually developed the Chinese New York: McGraw-Hill.
Antonak, R., & Livneh, P. (1988). The measurement of
Relative Assessment Interview for our own pur- attitudes toward people with disabilities—Methods,
poses (Tsang & Pearson, 2001). The interviews psychometric and scales. Springfield, IL: Charles
were successfully completed among 23 partici- C Thomas.
Bohn, D., & Peat, P. (1987). Science, order and creativity. Tsang, H. W. H., Chan, F., & Chan, C. C. H. (2004).
Toronto: Bantam. Factors influencing occupational therapy students’ atti-
Bordage, G., & Dawson, B. (2003). Experimental study tudes toward persons with disabilities: A conjoint
design and grant writing in eight steps and 28 ques- analysis. American Journal of Occupational Therapy,
tions. Medical Education, 37(4), 376–385. 58(4), 426–434.
Cheung, F. (1990). People against the mentally ill: Tsang, H. W. H., Cheang, C. T. K., Tong, B. Y. M., & Tse,
Community opposition to residential treatment facili- S. S. L. (2004). Psychosocial functioning of Chinese
ties. Community Mental Health Journal, 26, 205–212. older people with chronic physical illness.
Dennis, A., & Valacich, J. (1993). Computer brainstorms: International Journal of Therapy and Rehabilitation,
More heads are better than one. Journal of Applied 11(3), 99–105.
Psychology, 78(4), 531–537. Tsang, H. W. H., & Pearson, V. (2001). A work-related
Heppner, P. P., Kivlighan, D. M., & Wampold, B. E. social skills training for people with schizophrenia in
(1999). Research design in counseling. Belmont, Hong Kong. Schizophrenia Bulletin, 27(1), 139–148.
CA: Wadsworth. Tsang, H. V. H., Pearson, V., & Yuen, C. H. (2002). Family
Isaksen, S. K., Dorval, B., & Treffinger, D. (2000). needs and burdens of the mentally ill offenders.
Creative approaches to problem solving. Dubuque, International Journal of Rehabilitation Research,
IA: Kendall/Hunt. 25(1), 25–32.
Lewin, J., & Reed, C. (1998). Creative problem solving
in occupational therapy. Philadelphia: Lippincott. RESOURCES
Livneh, H., & Antonak, R. (1994). Indirect methods of
attitude measurement: Reply to Linkowski and Yuder. Computational Scientific Discovery
Rehabilitation Education, 8, 144–148. Computational scientific discovery (Swanson &
Noller, R. (1977). Scratching the surface of creative prob- Smalheiser, 1997) was developed specifically to facili-
lem solving. Buffalo, NY: DOK Publishers. tate the development of hypotheses across specialties
Osborn, A. (1979). Applied imagination. New York: that typically are not linked together. Arrowsmith
Charles Scribner’s Sons. (Smalheiser & Swanson, 1998), a form of computa-
Pearson, V., & Tsang, H. (2004). Duty, burden, & ambiva- tional scientific discovery, uses interactive software and
lence: Families of forensic psychiatric patients in Hong database strategies to uncover potential links that had
Kong. International Journal of Law and Psychiatry, 27, not been identified before. In a regular search, for
361–374. example, one may find no references when combining
Popper, K. (1959). The logic of scientific discovery. New two disparate areas. Through the use of Arrowsmith,
York: Harper & Row. one is able to retrieve the keyword terms that the two
Portney, L., & Watkins, M. (2000). Foundations of clinical disparate areas have in common. By using the filters
research—Applications to practice. Upper Saddle available through Arrowsmith, one can then reduce the
River, NJ: Prentice-Hall. number of potential choices for research. Arrowsmith is
Root-Bernstein, R. (1997). Discovering. Cambridge, MA: available on the Web at http://arrowsmith.psych.uic.
Harvard University Press. edu/arrowsmith_uic/index.html
Shamir, M., & Shamir, J. (1995). Competing values in pub- Creativity-Based Information Resources (CBIR)
lic opinion: A conjoint analysis. Political Behavior, 17, Creativity Based Information Resources (CBIR) is a litera-
107–133. ture database maintained by the International Center
Simon, H. (1981). The psychology of scientific problem for Studies in Creativity at Buffalo State College, State
solving. In R. Tweney, M. Doherty & C. Mynatt (Eds.), University of New York. The database contains more
Scientific thinking. NewYork: Columbia University than 12,000 annotated references of works focusing on
Press. creativity, and includes articles on research related to
Smalheiser, N., & Swanson, D. (1998). Using ARROW- creativity. CBIR is available on the Web at
SMITH: A computer-assisted approach to formulating http://www.buffalostate.edu/orgs/cbir.
and assessing scientific hypotheses. Computer Methods
and Programs in Biomedicine, 57, 149–153. Creativity for Life
Sternberg, R., & Lubart, T. (1996). Investing in creativity. Creativity for Life.com is a Web site containing articles and
American Psychologist, 51, 677–688. links related to creativity and creative thinking. While
Sternberg, R., & Lubart, T. (1999). The concept of creativ- not specifically dealing with the research process, the
ity: Prospects and paradigms. In R. Sternberg (Ed.), included links and articles contain information on
Handbook of creativity (pp. 3–15). Cambridge, UK: enhancing writing and improving creativity. This Web
Cambridge University Press. site can be found at http://www.creativityforlife.com.
Swanson, D., & Smalheiser, N. (1997). An interactive sys- Innovation Tools
tem for finding complementary literatures: A stimulus Innovation Tools is a catalog of creativity resources includ-
to scientific discovery. Artificial Intelligence, 91, ing tools, links, and articles. While this Web site is
183–203. focused on business innovation, the tools that are dis-
Tsang, H. W. H. (2003). Augmenting vocational outcomes cussed are directly transferable to the research process.
of supported employment with social skills training. Innovation Tools can be found on the Web at
Journal of Rehabilitation, 69(3), 25–30. http://www.innovationtools.com.
C H A P T E R 2 9
Integrity and Ethics in the Conduct

of Human Subject Research
Don E. Workman • Gary Kielhofner
Overview of Issues of integrity and ethics are complex. There are situa-
tions when it is impossible to perform research
Integrity and Ethical Conduct without impinging upon one or more of the ethical
Surrounding Research principles, in which case the investigator may need
to develop additional safeguards to protect the sub-
jects of the research. For instance, one of the basic
This chapter summarizes issues related to research principles of research ethics requires that subjects
integrity and ethics. Conducting research in an eth- be enrolled in research only when they have pro-
ical manner requires an investigator to develop vided fully informed consent (this is included
knowledge beyond a common-sense understand- under the principle of respect for persons) (The
ing of moral issues. In this chapter, we cover the National Commission for the Protection of Human
basic ethical principles that should guide the con- Subjects of Biomedical and Behavioral Research,
duct of research. 1979). However, to answer important questions
Moreover, in much of the world today, investi- about clinical interventions, some occupational
gators doing research that involves human subjects therapy research needs to be performed with
come under the jurisdiction of national principles persons whose impairments prevent them from
and regulations that govern the ethical conduct fully understanding the research and who,
of research. When dis- therefore, have a lim-
cussing such govern- ited capacity to give in-
mental regulations1 we Ethical conduct in research formed consent. In such
focus mainly on the must go beyond compliance cases, the investigator
United States in this may be required to
chapter. However, since with the letter of the law and, obtain permission on
these regulations reflect instead, involve behavior behalf of the patient
degrees of international from a spouse or other
consensus about research consistent with a knowledge- family member before
ethics, they will parallel able awareness of research enrolling the patient as a
many aspects of how eth- subject of research. In
ical conduct of research ethics and an underlying still other situations, it
is managed throughout spirit of integrity. may simply not be ethi-
much of the world. The cally permissible to
structure of this chapter enroll the subjects at all
is partly based on an Office of Research Integrity (for instance, when the research represents more
publication entitled “Introduction to the than minimal risk) (HHS Working Group on the
Responsible Conduct of Research” (Steneck, NBAC Report, n.d.).
2004). In the end, ethical research depends on the
It is important to note at the outset of this chap- knowledge and integrity of the investigator.
ter that the application of the general principles of Everyone who undertakes an investigation
assumes a moral responsibility to abide by com-
1
In the United States, researchers who are not federally monly accepted ethical standards. In this chapter
funded, or who conduct their research in institutions that we discuss compliance with regulatory principles
have no federal funding, may not be obligated at the time and procedures that enforce these ethical stan-
this chapter is being written to learn and abide by these dards. However, it is important to underscore that
rules, but should be aware there is a movement underway at
the federal level to require all research, regardless of
ethical conduct in research must go beyond com-
whether there is federal funding involved, to abide by these pliance with the letter of the law and, instead,
same standards. involve behavior consistent with a knowledgeable
Chapter 29 Integrity and Ethics in the Conduct of Human Subject Research 469
awareness of research ethics and an underlying ORI has and follows a number of federal poli-
spirit of integrity. cies for resolving allegations of misconduct. The
following is their definition of misconduct:
Regulation of Research Ethics
Misconduct in Science means fabrication,
and Integrity
falsification, plagiarism, or other practices
To ensure that there is a reasonably objective that seriously deviate from those that are
review of the ethical issues related to human sub- commonly accepted within the scientific
ject research, many institutions worldwide require community for proposing, conducting, or
that research plans be reviewed and approved by reporting research. It does not include hon-
an ethical review board (known in the United est error or honest differences in interpre-
States as an Institutional Review Board [IRB]) tations or judgments of data. [Cited from
prior to the initiation of the research study. This 42 CFR 50.102]
requirement was codified in U.S. federal regula-
tions in 1981 when 17 federal agencies agreed Definitions and examples of fabrication, falsifi-
to a common set of rules to govern research on cation, and plagiarism are included in the feature
human subjects that they funded or over which box titled Fabrication, Falsification, and Plagia-
they provided oversight (Office for Human rism. These constitute the primary issues that fall
Research Protection, 2001). It is also reflected in under research misconduct, but it is important to
the national regulations of many countries note that the definition also includes other prac-
throughout the world. The role and function of eth- tices that seriously deviate from established scien-
ical review/IRB boards is discussed later in this tific norms. Thus researchers in occupational
chapter. therapy need to be familiar with established scien-
Occupational therapy research may fall under tific norms, both within and outside the field of
the jurisdiction of specific federal regulations or occupational therapy.
it may not be specifically governed by any regu-
lations, depending on the institution or country The Relationship Between Ethical
where the research is conducted. Anyone conduct-
Conduct and Regulatory Compliance
ing research is responsible to conduct the research
ethically and with integrity, whether or not specific or Noncompliance
regulations pertain to the study. Moreover, when Often an inherent tension arises concerning com-
there are regulations they must be followed and it pliance and noncompliance with regulations. It is
is the responsibility of the investigator to find out easy to feel offended when one is told that one has
what the rules and procedures are. “broken a rule” (consider the average response to a
policeman when someone has been pulled over for
Policies Governing Research Integrity speeding). Higher stages of moral development,
however, involve guiding one’s behavior by ethical
In the United States, the Office of Research
principles rather than rules. This allows one, at
Integrity (ORI) (see the feature box titled Inter-
times, to understand that the morally right thing to
national Regulations Governing Research) is
do may indeed be inconsistent with an established
responsible for both the promotion of the responsi-
rule.
ble conduct of research (RCR) and the resolution
In watching Les Miserables, one readily identi-
of allegations of research misconduct. The Public
fies with the hero who has been imprisoned for 20
Health Service has identified nine core areas that
years because he broke into a house and stole a loaf
need to be addressed for investigators to know how
of bread for his sister’s starving child. Nonetheless,
to conduct research responsibly:
one would also concede that laws against stealing
• Data acquisition, management, sharing and are generally just and there should be consequences
ownership, for individuals who break into houses and steal
• Conflict of interest and commitment, from others.
• Human subjects, The regulations regarding research ethics
• Animal welfare, and research integrity are not different. Knowing
• Research misconduct, the regulations is helpful for understanding the
• Publication practices and responsible authorship, broad parameters for acceptable conduct. Ethical
• Mentor/trainee responsibilities, research behavior is usually achieved by being
• Peer review, and compliant with those regulations. Nonetheless, as
• Collaborative science. already noted, investigators should think beyond
International Regulations Governing Research
There is growing international consensus con- the local IRBs just as the local IRB provides
erning ethical principles that govern research. oversight of the human subject studies that are
For instance, the Declaration of Helsinki (http:// underway.
www.wma.net/e/policy/b3.htm) sets forth the
principles regarding the ethical conduct of human Canada
subject research originally adopted by the World
This Tri-Council Policy Statement: Ethical Con-
Medical Association in 1964. The current docu-
duct for Research Involving Humans describes the
ment, adopted in 2000, includes 32 principles
policies of the Medical Research Council (MRC),
and has two subsequent clarifications.
the Natural Sciences and Engineering Research
Nonetheless, most countries have their own
Council (NSERC), and the Social Sciences and
specific guidelines or regulations that govern
Humanities Research Council (SSHRC). The
ethical conduct of research and investigators
document replaces SSHRC’s Ethics Guidelines
must be familiar and comply with them. Below
for Research with Human Subjects, MRC’s
is information on Australia, the United States,
Guidelines on Research Involving Humans, and
Canada, and the United Kingdom
MRC’s Guidelines for Research on Somatic Cell
Australia Gene Therapy in Humans. The Councils will
consider funding (or continued funding) only
National Statement on Ethical Conduct in Re- to individuals and institutions that certify com-
search Involving Humans (National Health and pliance with this policy regarding research
Medical Research Council, 1999) (available from: involving human subjects.
http://www.nhmrc.gov.au/issues/researchethics. This joint policy expresses the continuing
htm) provides guidelines made in accordance commitment by the three Councils to the people
with the National Health and Medical Research of Canada, to promote the ethical conduct of
Council Act 1992, to which research involving research involving human subjects. This commit-
humans must conform. This document: ment was first expressed in the publication of
guidelines in the late 1970s. Work on the joint
• Sets out the ethical principles and values that policy was started by formation of the Tri-Council
should govern research involving humans, Working Group in 1994. The Councils published
• Provides guidance about how research should three documents prepared by the Working Group:
be designed and conducted so as to conform an Issues Paper in November 1994, a Discussion
to these principles, and Draft in May 1996, and its Final Report (Code of
• Outlines the procedures for consideration and Ethical Conduct for Research Involving Humans)
approval of all such research by Human in July 1997. Each of these documents stimulated
Research Ethics Committees (HRECs). extensive discussion in the academic community.
All research involving humans must be con- The present Policy Statement was prepared by the
ducted in accordance with the principles contained Councils by revision of the Working Group’s Final
in this statement. Report in the light of consultations between mid-
1997 and May 1998.
United States
United Kingdom
All research that is funded through the U.S.
Public Health Service (PHS) falls under the In the United Kingdom, the Department of Health
jurisdiction of the Office for Research Integrity has produced a framework outlining the principles
(http://ori.dhhs.gov/). PHS-funded research of all research within its remit that is undertaken in
must be conducted in compliance with the fed- health and social care. The Research Governance
eral regulations at Title 42 of the Code of Fed- Framework for Health and Social Care (2005, 2nd
eral Regulations (CFR), Part 50 and Subpart A ed.) is available at: http://www.dh.gov.uk/Policy
(http://ori.dhhs.gov/misconduct/reg_subpart_a. AndGuidance/ResearchAndDevelopment/Research
shtml). In addition, all research involving human And DevelopmentAZ/ResearchGovernance/fs/en
subjects that is funded through the Department Research Governance, which applies to all
of Health and Human Services (HHS), which is those who undertake or participate in research,
part of the Public Health Service, is subject to and sets out standards to improve research and
the oversight of the Office for Human Subject safeguard the public. Requirements include:
Protections (OHRP) (http://www.hhs.gov/ohrp/). • Independent review of research proposals to
OHRP is responsible for ensuring that HHS- ensure ethical standards are met,
funded research is conducted in accordance with • Scientific review by independent experts,
the federal regulations at Title 45, Part 46 of the • Informed consent and safeguards for patient data,
Code of Federal Regulations (CFR). OHRP is • Clear and accessible dissemination strategies, and
the federal office that provides oversight over • Provision of a quality research culture.
470
regulations to a thorough consideration of ethical furthering of a culture of compliance and respect

issues that inevitably arise in doing research. among researchers.
Moreover, investigators should seek to become as
knowledgeable about ethical issues as they are in
other aspects of the conduct of research. Research Integrity
For the sake of summarizing the key elements of
The Benefits of Responsible Conduct
research integrity in this chapter, we present a
The benefits of responsible conduct of research go summary of the basic tenets of research integrity
far beyond avoiding the negative consequences of as elucidated by Steneck (2004). These tenets are
noncompliance. Researchers usually are internally available for review or downloading on the Web
motivated to do things right, and a familiarity with site of the Office of Research Integrity (http://
the principles of the responsible conduct of ori.dhhs.gov/documents/rcrintro.pdf). It is worth
research allows researchers to proceed with confi- noting that in the introduction to this helpful book-
dence that they are abiding by accepted standards let, the Director of the Office of Research Integrity
of ethical conduct. Moreover the training of pro- (ORI) clearly indicates that the teaching and appli-
fessionals and researchers in RCR will increase the cation of these principles will vary for different
common knowledge base and will contribute to the professions.
Fabrication, Falsification, and Plagiarism
The key aspects of scientific misconduct include without giving appropriate credit [cited from 42
fabrication, falsification, and plagiarism. Each is CFR 93.103 (b)]. An occupational therapist has
defined and exemplified below. been working on a paper and discusses it with a
Fabrication: Fabrication is making up data colleague from nursing. This colleague points
or results and recording or reporting them. [cited out that a paper addressing the issue has been pub-
from 42 CFR 93.103 (a)]. For example, an occupa- lished in the nursing literature and provides a copy.
tional therapy graduate student has been told by The occupational therapist used the fundamental
his or her advisor that a minimum of 30 subjects ideas for and structures the occupational therapy
are required for a thesis project he or she is under- paper much the same as the nursing paper. When
taking. The student obtained data on 29 subjects the occupational therapy paper is published it has
and is facing a deadline that would prevent grad- no reference to the nursing paper. This would be
uation on time. Under the pressure of time, the an example of plagiarism and it constitutes scien-
student makes up a 30th subject, entering that tific misconduct.
subject into the database for analysis. In each of the three hypothetical instances of
Falsification: Falsification is manipulating scientific misconduct given above, it is likely that
research materials, equipment, or processes, or those involved will have “good reasons” for their
changing or omitting data or results such that behavior. For instance, the student who fabricates
the research is not accurately represented in a subject may reason that 1 subject out of 30
the research record [cited from 42 CFR 93. won’t really change the findings. The qualitative
103 (b)]. Two occupational therapy researchers investigators are convinced that their conceptual
involved in a qualitative study have data provid- argument is mainly correct. They reason that men-
ing support of a conceptual argument they have tioning contradictory data will only undermine
previously published in the literature. They are others’ confidence in what is basically a sound
discussing the data in a research team meeting conceptual argument. The writer reasons that,
when a research assistant points out a number since the original work was published in another
of instances in the fieldnotes that call into ques- field, it doesn’t matter if the author is not cited.
tion their argument. Nonetheless, the investiga- However, in each instance, the reasoning masks
tors decide to complete an article based only on the fact that each of these hypothetical persons
the data that support their argument. They not gained something from their misconduct. An
only ignore the contravening data but also do investigator must always be careful to ask whether
not mention its existence in the research report. personal gain is influencing a decision that
This is an instance of falsification and it consti- involves an ethical issue. Moreover, decisions
tutes scientific misconduct. should always be guided by the ethical principles
Plagiarism: Plagiarism is the appropriation of involved and not by personal, logistic, political,
another person’s ideas, processes, results, or words or other considerations.
The Price of Irresponsible Conduct interest. In the case of research that is sponsored
(funded) by a private corporation, there is usually
When researchers receiving federal funding are a contract that specifies the terms of the funding
investigated and judged to have engaged in that is provided and that clarifies the ownership of
research misconduct, they can bring significant data resides with the private corporation who
penalties and consequences upon themselves and retains this right in hopes of applying it for com-
their host institutions. These can include: mercial use. Philanthropic organizations may also
• Requirements that federal funding be returned fund research, and they may either retain rights or
(paid back), give them away depending on their interests.
• Additional penalties and fines to the individual Finally, there are student research projects and
investigator and to the institution for not main- clinician-initiated studies where the ownership
taining regulatory compliance, interests in the data are not clearly articulated or
• Institutional sanctions such as halting of all fed- understood. It is important for researchers to
eral funding of research, or possibly even stop- understand the nature of the agreements that pro-
ping all ongoing human subjects research
vide funding for the research as well as any appli-
(regardless of the source of funding) until the
institution is “brought into compliance.” cable policies in the settings where the research is
conducted in order to know whether or not they
There are agencies that “blacklist” investiga- have the right to publish those data.
tors so that they are prevented from participating
in future research.
Accepted Practices
For the results of research to be of value, it is essen-
tial that the data that are gathered are reliable. There
Data Acquisition, Management, is no one way to ensure the reliability of the data,
but the responsible investigator will use acceptable
Sharing, and Ownership
standards within his or her field of research to
In the practice of science, data collection and stor- ensure the careful collection of accurate informa-
age are crucial activities. Given the advent of the tion. In addition, the investigator must understand
personal computer, and the proliferation of easy (or consult with others who understand) statistical
ways to replicate, share, and store electronic methods adequately to ensure that they are using an
copies of data (laptop computers, handheld storage appropriate strategy for the analysis of the data.
devices, flash memory cards, and e-mail to name a Although this point is covered elsewhere in this text
few), there are many new mechanisms that can be as an issue of scientific rigor, it is important to
used to facilitate the acquisition, management, and understand that following accepted scientific prac-
sharing of data. While these streamline many tice is also an ethical mandate.
aspects of research, they also require additional The investigator must also understand what lev-
care in how data are stored, managed, and shared. els of authorization might be required to collect
In conducting research, one or many people some forms of data. For instance, under the
may be collecting the data, but this does not nec- “Privacy Rule” (the rule many in the United States
essarily infer rights of ownership over that data. refer to by the legislation that required it—Health
There may be important limits to what one can eth- Information Privacy and Access Act or HIPAA),
ically do with the data that are collected. For individuals who are collecting protected health
instance, when research is funded through a grant information for research purposes in a setting such
by the federal government, the research institution as a clinic or hospital will be required to obtain the
(e.g., university or hospital) is assigned ownership written authorization of the patient/subject, or a
rights to the data gathered under that research. The data use agreement from the clinic or hospital, or a
research institution is held accountable for ensur- waiver of authorization from the IRB or Privacy
ing the integrity of the data that are collected. In Boar (Department of Health and Human Service,
this case neither the individual researcher nor the n.d.).
government have immediate ownership rights to Finally, the investigator must ensure that data
the data—they belong primarily to the research are properly protected so as to ensure the integrity
institution that received the award (e.g., grant). of the data. This often means using appropriate fil-
In other cases, the federal or state government ing strategies, including security measures such as
may fund research through a contract, in which locked file cabinets within locked offices, and
case the data are usually required to be “delivered” password protected files on computer disks or
to the government which then retains ownership other electronic storage devices. In many circum-
stances data must be retained for a number of years • Who is entitled to access and use the data that are
after they are published, or after the funding period collected by the trainee?
is over. Specific data retention policies for the • How will the data be stored, and by whom, after
institution where data is gathered and stored the trainee has completed his training?
should be consulted before data (or signed consent
documents or authorizations) are destroyed. It is helpful for a research laboratory to con-
sider development of standard operating proce-
Paper and Electronic Storage dures governing the nature of the mentor/trainee
relationship, and for standardizing authorship/pub-
Investigators frequently store their raw data on
lication practices.
paper documents called case report forms. This
The trainee also has responsibilities in the men-
information may subsequently be summarized in
toring relationship. These include conscientious
computer files for subsequent statistical analysis
conduct that is consistent with research protocols,
or for creating charts or graphs. The investigator
and other local institutional requirements. In the
should maintain the “source documents” as well as
mentoring relationship, like in other aspects of
the computer files for some time after the results of
academic training, trainees are responsible to
the research are published or the grant or contract
understand the nature of their role and to seek out
has ended.
opportunities to learn from their teachers and men-
Under HIPAA, in addition to the Privacy Rule
tors, who by virtue of their qualifications and expe-
there is also a Security Rule (Centers for Medicare
rience should be in a position to be role models
and Medicaid Services, 2004) that requires that all
and resources for the trainee.
electronically stored protected health information
Many institutions or agencies funding research
(e-PHI) be safeguarded according to acceptable
or research training require trainees to undergo
standards. Each covered entity (e.g., clinic or hos-
formal training in the responsible conduct of
pital) is required to develop its own policies and
research. This requirement has led to the develop-
procedures for ensuring the security of e-PHI. The
ment of a significant number of online resources
researcher must be sure to comply with the appli-
that can be found in research institutions and on
cable policies for his or her institution.
the ORI Web site. All persons who plan to collab-
Mentor/Trainee Relationships orate in research or conduct research would be
well advised to take advantage of these educa-
One of the most important, but least standard- tional tools as well in order to consider ways they
ized mechanisms for training professional stu- can maximize their benefit from the mentoring
dents in the proper conduct of research is the relationship.
mentor–trainee relationship. Students are usually
required to have a supervisor over their research
activities, but the form and quality of the supervi- Publication Practices and
sion will vary widely from mentor to mentor, and Responsible Authorship
may be different with each trainee. The sharing of the results of research occurs in
Mentors are required to invest time and numerous contexts, and can easily be one of the
resources in their trainees. Because of their experi- more contentious issues related to research
ence base and relative power in the relationship, it integrity. Early results from experiments are often
is usually helpful for mentors to establish many of shared in laboratory meetings, local research meet-
the ground rules for the mentoring relationship. ings, and clinical conferences, as well as scientific
These might include topics such as: meetings. Later analysis of research results are fre-
• How much time will the trainee be required to quently published in scholarly journals and books.
spend on the mentor’s research? At a minimum, any communication of the
• How much direct time will the mentor spend results of research must be accurate and honest.
with the trainee providing individual or group Researchers should strive to accurately report their
supervision? methods, the results obtained, and the conclusions
• What criteria will be used to evaluate the per- they have drawn from the research. Scientific pub-
formance of the trainee? lication allows for the replication of methods, and
• What are the authorship expectations for differ- the generalization of results from the study being
ent research projects? reported. In human subject research, it allows for
• What are the standard operating procedures the generalization of the results from the experi-
regarding the conduct of the research, including mental sample to others from a larger population.
the acquisition and storage of data? The publication of findings allows others to learn
from the experiences and data collected during the ment of the prior presentation or publication. This
conduct of the research. avoids wasting resources and keeps the research
record clear. It is also important for readers to
Assignment of Due Credit know whether one is reporting the same results
Authorship, or the assignment of names to a pub- again or an independent replication of the previ-
lication or presentation, is an important aspect of ous research (whether it is the same data being
the responsible conduct of research. Persons who presented again or a new set of data). This is
made substantive contributions to the research also important when researchers conduct meta-
should be represented by inclusion in the author- analyses (see Chapter 18), since it is essential that
ship listing. This ordinarily can include: the “sample of samples” contain an accurate
accounting and that research samples not be inad-
• Persons who were instrumental in the initial con- vertently repeated.
ception and design of the study,
• Persons who were responsible for the collection Citations
and interpretation of the data, and
The appropriate and accurate citation of support-
• Those who wrote up the results or substantively
ing evidence for one’s own research is an impor-
edited the presentation before publication.
tant aspect of research integrity. When ideas, data,
Persons who play more minor roles are often or conclusions are based on other published work,
acknowledged in the publication, but not given including prior publications of the author or co-
authorship credit. Individuals should not be given authors, there must be appropriate attributions
“honorary” authorship credit by virtue of their made through inclusion of a citation. As exempli-
relationship to one or more of the authors, but fied in the feature box titled “Fabrication, Falsi-
included in the author listing only if they have fication, and Plagiarism,” not properly citing
made a substantive contribution. Open conversa- someone else’s work can be cause for allegations
tions between the parties conducting the research of plagiarism (presenting someone else’s work or
are essential regarding this topic as it is a frequent ideas as if they were your own).
area of misunderstanding.
Within an organizational unit where members Conflicts of Interest or Commitment
engage in research (e.g., a department or college),
it is a good idea to develop policies that govern There can be numerous conflicts of interest in the
authorship. These policies should reflect: research enterprise. One of these is the desire to
attain status and gratification from being the one to
• Consideration of ethical issues involved, make new information available to others.
• Protection of less powerful individuals (e.g., However, it is important in research not to make
students in relation to faculty members), and public statements about the results of the research
• Local consensus about issues of fairness and prematurely. When preliminary results are pre-
responsibility. sented, this practice is usually limited to presenta-
Deciding authorship can be a challenging issue tions at scientific meetings where the audience will
when students are involved with research under the understand and the presenter will clearly articulate
supervision of faculty advisors or within the fac- the results as being preliminary and subject to
ulty member’s research projects/teams. In such additional analysis and the scrutiny of peer review
cases, it is useful to have departmental policies that before being published for more general scientific
guide decisions about authorship (see the feature consumption.
box titled “Policies Governing Authorship,” as an In addition, there are circumstances when the
example). Finally, it should be noted that policies results are not favorable, and when the reporting of
alone can never cover all contingencies, so open those results might be harmful to the funding
and honest communication combined with fair- opportunity or career of the researcher. There are
minded negotiation is also important. This process times when it would be a clear violation of princi-
should begin before the research commences and ples of integrity to hide or suppress unexpected,
continue as persons shift responsibilities, contribu- contrary, or negative results for the sake of self-
tions, and roles. promotion or interest.
Conflict of commitment tends to address issues
such as hours spent on an outside job or other
Repetitive and Fragmentary Publication
income generation, which may prevent researchers
It is important that the same results not be pub- from having their full energy and effort for their
lished more than once without clear acknowledg- primary employment and, in particular, for dis-
Policies Governing Authorship
Developing local policies of authorship that list of contributors is excessive; therefore, author-
involve faculty–student collaboration should fol- ship should be limited to those with significant
low careful discussion that takes into consideration roles. Some activities that generally do not warrant
multiple factors including such things as ethical authorship are: commentary on a draft of a paper,
principles that guide the policy, the local organiza- one or two consultations to a project, editorial
tion culture, and how faculty and students ordinar- assistance which focuses on grammar, punctuation,
ily work together within the setting. Below is the and composition, compensated data collection
policy published in the Student Handbook of the or limited voluntary data collection and compen-
Department of Occupational Therapy at the sated statistical analysis. Such contributions are
University of Illinois (http://www.uic.edu/ahs/OT): generally noted in an acknowledgment. Authorship
should never be used as a reward for limited assis-
Policy on Joint Authorship tance to a project; it should always be based on a
negotiated significant role in the process.
An important part of the recognized mission of The following are some guidelines that should
the Department of Occupational Therapy and the be helpful in determining authorship:
University is the generation and dissemination
of knowledge. Publication is the primary process 1. The first author is someone who does all or
through which knowledge is disseminated in a pro- many of the following: initiation of the idea,
fession. Therefore, the Department expects its fac- determination of the method to be used, making
ulty to publish and encourage students to consider major decisions concerning variables and control
publication of their scholarly work. Although pub- of intervening variables, determining methods of
lication can never be a requirement for a student data reduction, making interpretation
paper or thesis, student scholarship and research is of results, assumes a major role in writing the
often of a caliber to merit publication. In many paper and assumes responsibility for communi-
cases the final product is the effort of several peo- cating between authors, with the journal editor,
ple including the student and joint-authorship will and for any revisions following review and for
be a consideration. submission of a flawless final manuscript and
Authorship connotes ownership of ideas, find- galley editing if it is used by the journal. (Note:
ings, conclusions, and so on. It is indicative of In the event that two people equally shared this
an individual’s work and intellectual contribution first level of responsibility, alphabetic order is
to a final published product. It is both criteria by the protocol for entry of names.)
which individuals can be judged for such conse- 2. The second author is someone who may do
quential processes as merit, tenure, and promotion. some of the things noted above and who typi-
Moreover, it is something for which an individual cally assists in the development of ideas,
may receive substantial recognition and career method, and instrumentation and who assists
advancement. Thus, for these several reasons in data reduction and analysis and in writing.
authorship should never be taken lightly. 3. The third author may be someone who assists
The overriding principle that should always or carries out data collection of a significant
govern inclusion of an individual as an author is portion of the data or who makes a substantial
that the person has made a significant contribu- contribution to one or more phases of the project
tion to the scholarly piece and that this contri- such as statistical analysis and interpretation.
bution was made with the explicit intention of (Note: In the event that authors other than the
sharing in the publication. Implied, then, is that first author have made equal contributions,
authorship should be determined at the beginning alphabetic order is the protocol for order of
of the process and not at the end. Also, since per- entry of names.)
sons’ roles in a particular scholarly process may 4. In the event that the original negotiated first
change while it is underway, authorship should author chooses not to assume his/her responsi-
be subject to renegotiation if an individual’s role bility to pursue publication in good faith within
becomes much greater or much less than originally 1 year of completion of the project, other per-
intended. Because application of useful informa- sons who originally negotiated to be second
tion is not only a privilege but also a responsibility, or third authors may assume this responsibility.
persons involved in a potentially publishable activ- In any case the first author’s name should be
ity should always make provisions and plans for included in the publication, although first
bringing the material to the jury process. authorship may be renegotiated.
Not all authorship is equal. Generally the first 5. Everyone whose name appears on a published
author is recognized as the senior author—i.e., the article should have the opportunity to view and
person who had major responsibility for the pub- approve the final draft unless he/she explicitly
lished contents. Authorship can be diluted if the designates the responsibility to a coauthor(s).
475
charging research obligations. Local institutions are reviewing. It is not acceptable for a reviewer to
are likely to have developed their own thresholds allow someone else to assist him or her in per-
and definitions of when an activity constitutes a forming the review, since this would be a breach of
conflict of financial interest or commitment. Both the confidentiality requirement. They also must not
the investigator and the research team members are use ideas they find in grants or manuscripts until
responsible for complying with the institutional the information is publicly available. The manu-
and federal requirements for disclosure and man- scripts or grant applications that have been re-
agement of conflicts of interest and commitment. viewed should be returned to the editors/granting
agencies after the review is completed or they
should be shredded or otherwise destroyed.
Peer Review
One of the hallmarks of scholarly publication is Collaborative Science
the practice of peer review. This process involves a
review of the planned publication by other scien- Scientific investigation is becoming increasingly
tists who are neutral in response to the publication interdisciplinary and inter-institutional. This has
and have sufficient expertise to provide a scientific led to an increase in collaborative efforts between
critique and evaluation. Part of their evaluation investigators and between scientific disciplines. It
includes judgments as to the potential value of the has also led to consortium arrangements whereby
research to the current literature. multiple institutions may share some common sci-
Peer review, like IRB review of human subject entific resources. The responsible conduct of col-
research, is intended to be an evaluative process laborative science entails attention to establishing
through which quality scientific publications are clear roles and responsibilities, as well as written
vetted. The role of peer review then is one that agreements that will satisfy institutional officials
requires honest appraisal and feedback. Journal at the various institutions.
and book editors rely on peer reviewers to provide It is helpful to be able to establish the ground
them with expert opinions with regard to potential rules for conduct and reporting of the research
publication manuscripts. Granting agencies rely on early in the collaboration. Various co-investigators
peer review to make decisions about which may share interest in common research aims, and
research proposals should be funded. Therefore, it so there should be some discussion of the proposed
is important that the peer reviewer adopt a facili- authorship arrangements in relation to the pro-
tative role. This requires putting aside personal posed conduct of and responsibility over the study.
disagreements with others in the discipline or
deferring one’s own ambitions to the sake of the Sharing of Materials and Data
scientific enterprise. Reviewers must also respect with External Collaborators
the confidentiality of the information with which
There may need to be contractual agreements,
they are provided.
including material transfer agreements, that con-
Peer reviewers are usually selected in confi-
vey ownership rights to intellectual property or
dence by the editors of a journal or book. The
materials that will change hands during the course
reviewers usually are blind to the author of the
of the study. These agreements are usually nego-
manuscript under review, and the authors of the
tiated through the grants and contracts office of
manuscript are not informed regarding the identity
a larger institution and by corporate counsel at a
of the reviewers. Reviewers are not paid for their
smaller institution.
time and efforts, but are expected to provide timely
and honest reviews in accordance with the format Special Concerns Related to HIPAA. Following
of feedback desired by the editors. Typically, edi- implementation of the HIPAA Privacy Rule, and
tors will provide reviewers’ comments back to the more recently the Security Rule, it is prudent to
author, who will then revise the manuscript to consider the HIPAA-related issues early in the
address the issues raised. The editor may also research development process to ensure adequate
notify the author that there is not support for the time to accrue the appropriate waivers, agree-
publication of that manuscript in their journal, in ments, or approval for authorization agreements. It
which case the author can consider submission of is important to remember that the institution where
the manuscript to another journal for review. It is PHI is created has a primary responsibility for
important to refrain from submitting a manuscript ensuring that there are appropriate mechanisms for
to more than one journal at a time. accessing PHI for research, especially when PHI
Peer reviewers are obligated to provide honest will be shared with co-investigators outside of the
feedback regarding the grants or manuscripts they covered entity.
Protection of Human but then considered the research important enough

that it was continued. They collected spinal fluid
Subjects in Research through nontherapeutic spinal taps, telling the
unknowing research subjects that they were being
Of all the ethical considerations in research, the provided with treatment for “bad blood.”
most compelling pertain to humans who will be In 1943, penicillin was recognized as an effec-
subjects or participants in a study. Unfortunately, tive treatment for syphilis. In the eyes of the
the impetus for contemporary ethical standards for researchers from the Public Health Service (PHS),
the protection of human research subjects is this made the cohort of patients in the Tuskegee
grounded in previous abuses of humans under the trial even more precious as they might be one of
guise of research. Two of the most well known the last cohorts of individuals with the disease who
instances are the Nazi doctors’ trials and the would be studied longitudinally (over a long
Tuskegee experiment. period of time). For this reason, the PHS re-
searchers made additional efforts to ensure that
Nazi Doctors’ Trials their research subjects remained under study. This
meant preventing them from knowing they had a
In the Nazi doctors’ trials following World War II,
disease, and preventing them from obtaining med-
the tribunal identified numerous experiments that
ical treatment for syphilis. During World War II,
had been conducted in the interest of science,
the PHS investigators managed to convince the
which represent horrific abuses of human beings.
local draft board not to enlist any of the Tuskegee
For instance, concentration camp prisoners were
subjects into the armed forces because they would
exposed to low atmospheric pressures, to simulate
have been readily diagnosed and treated.
what might happen to pilots if they were exposed
In retrospect, and from a perspective outside of
to atmospheric conditions at high altitudes.
the investigators, this is easily viewed as a morally
Individuals would lapse into coma, and sometimes
repugnant study. Investigators deceived innocent
died as the Nazi scientists established some of the
and vulnerable men into thinking they were getting
limits of human endurance. In other experiments,
some form of treatment, when they were actually
prisoners were placed naked into tubs of icy water
being denied information about a disease they had
in order to establish limits for hypothermia. They
(and were sharing with others in their community),
were then often rescued using various techniques
and were not provided with or actively prevented
that sometimes produced scalding burns and even
from receiving a treatment for their syphilis. This
death. While these experiments provided the Nazi
study was exposed by the media in 1972 and the
military with valuable information regarding the
study was finally halted in 1973. The public outcry
length of time human beings can survive in water
led to the regulations and the IRB and federal over-
at various temperatures, and information about
sight processes which make up today’s human sub-
effective methods for reviving soldiers who were
ject protection programs in the United States
partially frozen, those hapless prisoners were
(Dunn & Chadwick, 2002).
involuntarily exposed to experimental torture and
even death. The experiments by the Nazi doctors Nuremberg Code
were ethically reprehensible because they caused
harm without regard for the well-being or for the One of the earliest U.S. codes regarding the ethi-
informed consent of the subject. They also cal conduct of research was written during the
exposed individuals to unacceptably high levels of Nazi doctors’ trials. Because the trials were con-
risk without regard for their pain and suffering, ducted in Nuremberg, Germany, the 10 principles
and in ways in which the risks were not reasonable of ethical human subject research became known
in light of the benefit that might be gained from the as the Nuremberg Code. Dunn and Chadwick
information that was derived from the research. (2002, p. 16) cite from the code the following pro-
tections:
Tuskegee Experiment • Informed consent of volunteers must be obtained
When an agency of the American Public Health without coercion in any form.
Service began its study of syphilis in the African- • Human experiments should be based upon prior
American male, they selected a rural area in the animal experimentation.
south (Macon County, Alabama) with a high con- • Anticipated scientific results should justify the
centration of men who had the disease (Dunn and experiment.
Chadwick, 2002). They intended to study the nat- • Only qualified scientists should conduct medical
ural history of the disease for a period of 6 months, research.
• Physical and mental suffering and injury should research. This means that the risks are reasonable
be avoided. in light of the potential for benefit from participa-
• There should be no expectation of death or dis- tion (to the subjects themselves, or from the
abling injury from the experiment. knowledge to be gained from their participation).
The risks considered include reasonably antici-
The Principles of Belmont pated physical and mental risks, as well as physi-
cal and mental discomforts. Beneficence also
In 1979, the U.S. National Commission for the
requires the use of sound research methodology,
Protection of Human Subjects of Biomedical and
since no degree of risk is acceptable if the research
Behavioral Research published a document sum-
design is inadequate. That is, there would be little
marizing for the Department of Health, Education
likelihood for benefit from the knowledge to be
and Welfare (later to become the Department of
gained if the study is not sound, and without any
Health and Human Services) the basic ethical
anticipated benefits no risk to subjects is justified.
principles for human subject research. In the
Finally, beneficence also requires that risks
paper, they distinguish between clinical practice
must be minimized. This means that investigators
and clinical research, and they set out three basic
should take every precaution and make every effort
principles that are intended to apply to all research
to anticipate and prevent or minimize physical or
involving humans:
mental discomfort or harm that may accrue from
• Respect for persons, participation in the study.
• Beneficence, and
• Justice. Maximizing Potential for Benefit. There are
numerous ways in which the IRB and the investi-
Respect for Persons gator can design the study to maximize potential
for benefit, both for the individual subjects and
Respect for persons is the most basic of the three from the potential knowledge that may be gained
principles, and asserts that human persons must be from the research. For instance, a study might
respected in terms of their right to self-determina- increase the potential for individual subjects to
tion. Key to this principle is the notion of informed benefit by incorporating a crossover design rather
consent as a required prerequisite for most kinds of than a placebo control. In the case of the placebo
research, in particular when the research involves control, those subjects receive no therapy at all,
imposing some risks on the subjects. while in the crossover design everyone has a
The Belmont Report’s principle of respect for period of time when they are receiving the new
persons emphasizes the importance of the individ- treatment. The IRB’s insistence on sound research
ual to make choices whether or not to participate in methodology is a way of ensuring that there is a
a research study as an exercise of free will. maximum potential for benefit from the research in
Respect for persons assumes that the researcher terms of its providing generalizable knowledge.
can provide enough information, in a language
the prospective subject can understand, for the Justice
individual to provide fully informed consent to
be involved in the research. The report goes on to The principle of justice requires that the research
recognize that there are circumstances and popula- impose the burden of risk and the potential for
tions that are not able to provide fully informed benefit upon the same groups of people. It is unac-
consent, and requires that additional protections be ceptable from the perspective of the principle of
afforded to these vulnerable populations. Pregnant justice for an investigator to take advantage of a
women and fetuses are considered to be a vulnera- vulnerable population, for instance the poor, in
ble population, as are prisoners and children. order for others to reap the benefits of the research.
Consequently, ensuring justice typically involves
Beneficence examination of the composition of subjects who
have been enrolled according to such categories as
Beneficence is the principle that requires that
sex and race/ethnicity.
human subjects research minimize risk to the
greatest extent possible and maximizes the poten-
Informed Consent
tial for benefits to be gained from the research
(either for the individuals participating in the The principle of informed consent requires that
research or from the knowledge that will be prospective research subjects be given enough
gained). Risks must be reasonable in relation to information, before they choose to participate in a
the potential for benefit to be derived from the research study, that they can make an informed
decision as to whether or not they want to partici- In the case of adults who may have cognitive
pate. While there are some circumstances in which impairments (e.g., psychotic episodes or demen-
the requirement of prospective informed consent tia), the researcher will need to obtain the assent of
may be waived or altered, usually research involv- such persons to the extent they are able to provide
ing human subjects requires they provide written it and the permission of a surrogate. Ideally the sur-
informed consent (and possibly a HIPAA authori- rogate is a legally authorized agent under the local
zation for research use of protected health infor- laws. When such laws do not exist, it is important
mation) before they participate in the research. for the investigator who may be enrolling adult
Informed consent usually requires: subjects who are cognitively impaired to consult
with legal counsel to ensure that appropriate surro-
• That subjects be prospectively informed that they
gate consent is being obtained.
are being asked to participate in research,
• That their participation is voluntary, and
Documentation of Consent
• That they may choose to discontinue their partic-
ipation at any time. The documentation of consent refers to obtaining
the appropriate signature on (an) IRB-approved
Prospective subjects must also be informed of
consent or assent document(s). The original docu-
reasonably anticipated risks and discomforts from
ments are very important to keep as any audit of
participation, as well as any benefits that may be
the research study will require the investigator to
expected to result from their participation or from
produce them. Funding agencies require that these
the knowledge gained from the study. The IRB
original documents be maintained for a number of
may waive or alter the required elements of
years after the completion of the study. Investiga-
informed consent under some circumstances.
tors are usually required to provide a copy of the
The Process of Consent consent document to the subjects as an information
sheet they can take with them. Under HIPAA, if an
A central component of informed consent is a well investigator is also obtaining an authorization for
written informed consent document, and a process the research use of protected health information,
of obtaining consent that involves a careful review the investigator is required to provide subjects with
of the information in the document with ample a copy of the signed consent document.
opportunity for the prospective subject to have any Informed consent is often a complex process
questions answered. The process under which that involves a number of considerations. These
investigators inform prospective subjects about the include, for instance:
research is even more important than the document
that the subject signs. Consent conferences should • Making sure the prospective subject understands
reflect the information in the consent document, the study and the risks/benefits involved (this
but also include questions to allow the person increasingly involves not only ensuring that the
obtaining consent to be assured the potential sub- study will be explained in lay terms but indicat-
ject understands the information that is being pre- ing how the researcher will ascertain that the sub-
sented. ject has understood what was explained),
Consent usually is not a one-time event, but an • Consideration of whether prospective subjects
ongoing process through a subject’s participation have the ability to give consent,
in research. If a study goes for a prolonged period • Balancing any incentives or reimbursement for
of time, it may be prudent to revisit the consent participation to avoid coercion (e.g., giving
document, ask about willingness to continue in the financial incentives that would be difficult for
research, and offer to answer questions about some potential subjects to decline), and
ongoing participation. • Assuring freedom of consent in situations where
other obligations or roles might pressure individ-
Assent and Surrogate Permission uals to consent.
Consent refers to the process whereby a competent It is important the investigators consider and
adult gives permission for something to be done to make plans to deal with these and other considera-
his or her own person or information. One cannot tions that may be unique to a particular study when
give consent for something to be done to others. developing informed consent procedures. The fea-
Therefore, in the case of children and adolescents, ture box titled “Informed Consent Procedures,”
the regulations usually require a combination of illustrates an occupational therapy informed con-
parental permission and the active assent (not the sent procedure and the forms used in association
failure to dissent) of the children and adolescents. with it.
Ethics Committees or Institutional risk and maximize the potential for benefit from
Review Boards the research (to the subjects or from the knowledge
that will be gained). The IRB is also responsible
Most institutions that have research involvement for ensuring that adequate safeguards are in place,
have established committees that oversee issues of including the requirements for informed consent,
ethical conduct in research. In the international so that the risks to subjects are minimized to the
context these bodies are usually referred to as greatest extent possible.
Ethics Committees. In the United States research The specific operations of the IRB and the
institutions that receive federal funding for human application of the regulations (laws) will vary from
subject research are required to have an Institu- one institution to another. They also may vary
tional Review Board (IRB) that will exercise over- depending on the source of funding or the nature
sight authority over the research (Steneck, 2004). of the research. IRBs are responsible to make
In such institutions, an individual must be desig- numerous determinations about the level of risk
nated to oversee this process; that individual is related to participation in the research, the ade-
usually given a title such as Research Integrity quacy of the informed consent documents or
Officer. IRB/ethics committee review is estab- processes, the appropriateness of recruitment
lished as a mechanism whereby a group of diverse materials, as well as ensuring there are additional
individuals, with scientific and nonscientific inter- safeguards for protecting vulnerable subjects. The
ests review the research to ensure that the research IRB process may involve several series of
plan is ethical. responses to questions, modifications of the
The IRB is responsible for reviewing research research plan, recruitment materials, and informed
proposals (including grant funding applications) to consent documents before the project is approved
ensure there are adequate provisions to minimize to begin (Matthews-Lopez & Watson, 2004).
Informed Consent Procedures
The approval of Informed Consent procedures by The guardians and children will be given the
an IRB requires the researcher to: information sheet (that has been attached for
• Describe how subjects will be recruited, IRB review) and will be asked to read the
• Describe the informed consent process, and form (or have the form read to them) and to
• Submit the informed consent letter that will be ask any questions they may have before
used for approval. agreeing to be in the research.
Below is the section from an Approved They will be told that:
Protocol at the University of Illinois at Chicago for • As part of the research their child will be
a Psychometric Study of the Child Occupational asked to complete a questionnaire and to
Self-Assessment. discuss information obtained through the
questionnaire.
Recruitment and Consent Procedure • The child’s name will not be used to identify
The initial contact will be made by the them on the test form. They will not be iden-
Occupational Therapist. tified in any way in any oral or written
report of this study.
When subjects are recruited under the UIC • There are no benefits or risks involved in
collection protocol, guardians of the subjects participating in the study.
and subjects will be asked if they will be will-
ing to participate in a research study to Because this is a study involving children, both
improve the Child’s Occupational Self- the children and their parents/legal guardians are
Assessment (COSA). explained the study and asked for consent. The
procedure for asking parents/guardians is consid-
It will be explained that: ered consent since the children are considered not
Children are being asked to participate in a yet able to give full consent on their own behalf.
research study to improve the Child’s However, children’s right to refuse participation is
Occupational Self-Assessment (COSA) and respected and thus they are explained the study;
that the purpose of the assessment is to study the procedure for asking children is referred to as
how children perceive their strengths and assent. Below are the consent and assent letters
weaknesses regarding performance of daily used for the informed consent process in this
activities. study.
(continued)
Informed Consent Procedures (continued)
WRITTEN AND VERBAL ASSENT

TO PARTICIPATE IN RESEARCH
COSA STUDY
(For children ages 8-17)
My name is Gary Kielhofner.
I am asking you to take part in a research study because we are trying to learn more about the lives of
children who receive occupational therapy. We are trying to make a better questionnaire so that
occupational therapists can have better information about the children they work with, especially how
children think about their own strengths and weaknesses.
If you agree to be in this study, we will ask you to answer questions on a form.
Filling out the form will take place during your regular occupational therapy time.
There are no direct benefits to taking part in the study. However, your participation in the study will help
to develop a questionnaire that may help other kids receive better occupational therapy.
Please talk this over with your parents before you decide whether or not you want to be in the study.
We will ask your parents to give their permission for you to take part in this study. But even if your
parents say "yes" you can still decide not to do this.
If you don't want to be in this study, you don't have to participate. Remember, being in this study is up
to you and no one will be upset if you don't want to participate or even if you change your mind later
and want to stop.
You can ask any questions that you have about the study. If you have a question later that you didn't
think of now, you can call me, Gary Kielhofner, at 312-996-6901 or ask your occupational therapist.
Signing your name at the bottom means that you agree to be in this study. You and your parents will
be given a copy of this form after you have signed it.
Name of Subject Date
Signature Age Grade in School
(continued)
481
Informed Consent Procedures (continued)

Consent for Participation in Research
“Child Occupational Self Assessment”
Why am I being asked?
You are being asked if your child can participate in a research study to examine a self-rating evaluation form of occupational function.
By using information obtained through this assessment, occupational therapists will be better able to provide therapy that is responsive
to the individual needs of the child.
This study is conducted by Dr. Gary Kielhofner at the University of Illinois at Chicago. Your child has been asked to participate in the
research because he/she has been referred to Occupational Therapy and may be eligible to participate. We ask that you read this form
and ask any questions you may have before agreeing to be in the research.
You and your child’s participation in this research is voluntary. Your decision whether or not to participate will not affect your current
or future relations with the University. The occupational therapy your child receives will not be affected in any way by your decision to
participate or not. If you decide to participate, you are free to withdraw at any time without affecting that relationship.
Why is this research being done?
Objectives: The objective of this project is to refine and determine the reliability of a self-rating form, the Child Occupational Self
Assessment (COSA). The purpose of this research is to obtain information about a child’s self perceptions of his /her occupational func-
tioning.
Study Method: Data will be collected on a self rated form as part of the occupational therapy assessment procedure that your child would
normally be involved in.
Risk/Benefits: There are no expected risks or benefits from participating in this research.
What is the purpose of this research?
The purpose of this research is to obtain information about a child’s self perceptions of his /her occupational functioning. By using such
an assessment, occupational therapists will be better able to provide therapy that is responsive to the individual needs of the child.
What procedures are involved?
If you agree to be in this research, we would ask your child to do the following things:
• Complete a self-assessment form and
• Take part in a subsequent discussion with an occupational therapist concerning his/her responses.
This form will be completed during his/her ordinary time for therapy or at a time scheduled for your own convenience. Your child’s name
will not be used to identify him/her on the test form and will not be identified in any way in any oral or written report of this study.
Approximately 1,000 people may be involved in this research at the University of Illinois at Chicago.
What are the potential risks and discomforts?
There are no known risks associated with participation in this study beyond those that you would encounter in any supervised therapy
session. Most of the questions are related to your child’s ability to care for himself/herself. In the unlikely event that your child is injured
while completing the assessment, the University of Illinois at Chicago will not be responsible for providing either medical care or com-
pensation for such care as required by law.
Are there benefits to taking part in the research?
Although you will not directly benefit from participating in this study, your child’s participation will help to develop an assessment that
may be of benefit to children receiving therapy.
What about privacy and confidentiality?
The information that the research team receives will not have any information on it that will identify your child as a subject.
What if my child is injured as a result of my participation?
In the event of injury related to this research study, treatment will be made available through the University of Illinois at Chicago Hospital.
However, you or your third-party payer, if any, will be responsible for payment of this treatment. There is no compensation and/or pay-
ment for such medical treatment from the University of Illinois at Chicago for such injury, except as may be required of the University by
law. If you feel your child has been injured, you may contact the researcher, Gary Kielhofner, at (312) 996–6901.
What are the costs for participating in this research?
There are no additional research costs for which the subject would be responsible.
Will I be reimbursed for any of my expenses or paid for my child’s participation in this research?
There will be no monetary compensation for participation in the research.
Can I withdraw or be removed from the study?
You can choose whether your child will be in this study or not. You may withdraw at any time without consequence. Your child may also
refuse to answer any questions he/she doesn’t want to answer and still remain in the study. The investigator may withdraw your child
from this research if circumstances arise which warrant doing so.
Whom should I contact if I have questions?
The researcher conducting this study is Gary Kielhofner. You may ask any questions you have now. If you have questions later, you may
contact the researchers at: (312) 996–6901.
What are my child’s rights as a research subject?
If you have any questions about your rights as a research subject, you may call the Office for Protection of Research Subjects at (312)
996–1711.
Remember:
Your child’s participation in this research is completely voluntary. Your decision whether or not to participate will not affect your current
or future relations with the University. If you decide to participate, you are free to withdraw at any time without affecting that relationship.
You will be given a copy of this form for your information and to keep for your records.
Signature of Subject or Legally Authorized Representative
I have read (or someone has read to me) the above information. I have been given an opportunity to ask questions and my questions
have been answered to my satisfaction. I agree to participate in this research. I have been given a copy of this form.
Signature Date
Printed Name
Signature of Researcher Date (must be same as subject’s)
482
When research proposals are not approved, and Until the IRB has approved the study, the inves-
even if they are disapproved by the IRB, the IRB is tigator cannot proceed with the research. As noted
required to inform the investigator what changes earlier, approval sometimes involves several steps
to the research would make it approvable. After in which the IRB responds to the original or previ-
initial approval, IRBs are required to promptly ous submission with questions and required
receive and review information related to unantic- changes. Investigators should find out deadlines
ipated problems involving risks to subjects or oth- for submission of IRB documentation and the
ers, and to review and require changes or approve schedule of IRB meetings, factoring these into
any planned modifications to the research before overall deadlines and planning for a research
they are implemented (except when changes to the project.
protocol would remove subjects from risk of
immediate harm—in this case the changes can be The Role of the Investigator
implemented right away and the IRB should be and Research Team
notified and an amendment submitted as soon as
possible). Finally, the IRB is responsible for sub- The investigator and research team have the most
stantive re-review of the research no less often than direct responsibility for ensuring human subject
once every 365 days. protections are implemented and maintained as
Institutions that have IRBs generally publish approved by the IRB. The Principal Investigator
guidelines, procedures, and forms that are used to (i.e., the researcher in charge of the project) is ulti-
submit proposed research and research progress mately responsible for ensuring that the required
reports. Investigators are responsible to find out human subject protections are followed. The
and comply with these procedures and necessary Principal Investigator may delegate some of the
documentation. In some institutions, the Principal responsibilities for the conduct of the study, but
Investigator and others who are part of a research may not delegate ultimate responsibility for the
team are required to undergo training in research conduct of the research team. In addition, the
ethics before submitting research for approval. Principal Investigator is responsible for reporting
The necessary documentation for a planned any unanticipated problems involving risks to
study is submitted to the IRB. This minimally subjects or others, and for abiding by the IRB
includes: approved protocol and consent document/process.
Ultimately, it is the Principal Investigator who is
• A basic description of the study design and meth- held responsible for the conduct of the research,
ods, but each individual member of the research team is
• Justification for the research (including the also responsible for his or her own conduct. When
general benefits expected to accrue from the questions arise regarding research integrity or
study), some aspect of the research conduct in light of reg-
• The numbers of subjects/participants planned in ulations, it is the responsibility of the Principal
the study and their characteristics including Investigator and the research team to find a satis-
whether they represent vulnerable populations, factory answer.
• A description of the planned procedures used to
recruit (including inducements and reimburse-
ments) and obtain informed consent from sub-
jects (including consent forms that will be used),
• What subjects will be asked to do as participants
in the study,
• What if any benefits will accrue to subjects from
participation,
• What if any risks to subjects are involved and
how these will be minimized,
• How confidentiality will be maintained (includ-
ing how data will be stored and how long it will
be retained), and
• Who will be the personnel in the study.
Increasingly, institutions provide forms or

structured formats for submitting this information Figure 29.1 An Institutional Review Board (IRB)
to the IRB for approval. reviews a research proposal.
Research Misconduct the allegation. Finally, the institutional policy

makes provisions for reporting the findings to ORI.
As noted earlier, misconduct in science refers to
fabrication, falsification, plagiarism, or serious
deviation from commonly accepted practice for Conclusion
proposing, conducting, or reporting research.
Misconduct does not include honest error. Nor There are many policies that govern the ethical
does it include honest differences in interpretations conduct of research. These policies reflect a long
or judgments concerning research data. These cri- history of efforts to correct past abuses of human
teria for research misconduct are from the U.S. right as well as efforts to identify and attain the
regulations at Title 42 CFR 50.102. This regulation highest standards of integrity in research. Re-
also establishes the criteria for allegations of searchers and students who participate in research
research misconduct as well as for determinations are well advised to familiarize themselves with the
made after a formal inquiry or investigation into basic requirements for the responsible conduct of
the allegations. To qualify as misconduct, the research. This can be accomplished through online
behavior of the investigator or member of the browsing and through continuing education sec-
research team must represent a significant deviations at professional meetings and is often pro-
tion from commonly accepted practices, and it vided within research institutions. Some useful
must be intentional, knowing, or reckless. In its Web sites are noted at the end of this chapter in the
view, Office of Research Integrity in the U.S. con- Resources section.
siders plagiarism to include outright theft and mis- In addition to the federal regulations,
appropriation of intellectual property, as well as researchers need to be familiar with their own local
the substantial unattributed textual copying of policies and procedures. These local policies will
another’s work. It does not include authorship or often go into greater detail regarding how the
credit disputes (Office of Research Integrity, researcher needs to conduct his or her research and
2005). Finally, the allegations need to be proven by stay in compliance. Many institutions have admin-
a preponderance of evidence for such a determina- istrative offices that handle conflicts of interest,
tion to be made. Thus, there are forms of miscon- responsible conduct of research, and that coordi-
duct that may not warrant sanctions, but that nate grant and contract applications or ethical
clearly breach ethical standards. Therefore, as we approval of research. Officials in those offices and
noted earlier, conduct in research is better guided their local Web sites are both likely to be rich
by a concern for ethics and integrity than a concern sources of additional guidance.
to avoid sanction. In the end, the best way for investigators to
ensure compliance with human subject protections
Mechanisms for Resolving Allegations is for the investigator to be well informed.
Understanding the ethical principles and the regu-
Research institutions usually develop policies for latory requirements should increase a researcher’s
resolving allegations of misconduct in a timely motivation to ensure that adequate protections are
manner and through a two-step process. Typically provided. A commitment to and thorough under-
there is an inquiry phase during which the allega- standing of the ethical principles and the require-
tion is initially explored to determine whether it ments for the responsible conduct of research are
meets the institutional and federal definitions of essential to conducting research with integrity.
misconduct, and whether there seems to be evi-
dence of the alleged misconduct. If the allegation REFERENCES
appears meritorious, then the matter is referred to Centers for Medicare and Medicaid Services. (2004).
a more comprehensive investigation phase, during HIPAA Security and Electronic Signature Standards;
which the allegations and the evidence are Proposed Rule. Retrieved at the time of publication
reviewed by a larger group. During the investiga- from http://www.cms.hhs.gov/hipaa/hipaa2/regula-
tions/security/default.asp.
tion, the institution is seeking to find out whether Department of Health and Human Services. (n.d.).
or not the allegation of misconduct is true. The Protecting personal health information in research:
institutional policy must also define a person with Understanding the HIPAA Privacy Rule. Retrieved at
authority in the institution to act on the results of the time of publication from http://privacyrulean-
dresearch.nih.gov/pdf/HIPAA_Booklet_4-14-2003.pdf.
the investigation. That person can impose sanc- Dunn, C. M., & Chadwick, G. L. (2002). Protecting study
tions upon the individual if found guilty of mis- volunteers in research: A manual for investigative sites
conduct or can vindicate the person if cleared of (2nd ed.). Boston: Thomson.
HHS Working Group on the NBAC Report. (n.d.). guidance and online information resources is readily
Research involving individuals with questionable available at the OHRP Web site (http://www.hhs.
capacity to consent: Points to consider (Appendix gov/ohrp/) and the FDA Web site for regulating med-
C). Retrieved at the time of publication from ical devices (http://www.fda.gov/cdrh/index.html).
http://aspe.hhs.gov/sp/nbac/appendixc.shtml. Many academic careers have ended with a finding of
Matthews-Lopez, J. L., & Watson, K. D. (2004). research misconduct, or have been impaired because
Navigating the review process for research involving of noncompliance with other regulations. In addition,
human participants: An overview and practical guide- institutions have faced very stiff fines or sanctions that
lines [Electronic Version]. Ohio Research and Clinical have cost millions of dollars. The ORI Web site con-
Review, 14, 7–13. Retrieved at the time of publication tains informative case summaries of closed cases and
from http://www.ohiocore.org/Research/documents/ the sanctions that have been made (http://ori.dhhs.gov/
ORCR/ORCR_2004.pdf. misconduct/cases/index.shtml) The OHRP Web site
National Commission for the Protection of Human includes access to determination letters regarding
Subjects of Biomedical and Behavioral Research. findings of noncompliance (http://www.hhs.gov/ohrp/
(1979). The Belmont Report. Retrieved at the time compliance/letters/index.html). Reviewing these docu-
of publication from http://www.hhs.gov/ohrp/ ments can be quite informative for the student and
humansubjects/guidance/belmont.htm. researcher alike as we all try to better understand how
Office for Human Research Protections. (2001). Policy for to conduct research responsibly.
the Protection of Human Subjects (Basic DHHS Policy
for Protection of Human Research Subjects), Title 45
The Belmont Report
In recognition of the 25th anniversary of the Belmont
Public Welfare, Part 46, Protection Of Human Subjects,
Report, and in tribute to its important contribution
The Common Rule. Retrieved at the time of publica-
to our thinking about the ethical conduct of research,
tion from http://www.hhs.gov/ohrp/humansubjects/
there is a special Web site with historical information
guidance/45cfr46.htm.
pertaining to the report and the process of its develop-
Office of Research Integrity. (2005). Policies: ORI provides
ment (http://www.hhs.gov/ohrp/belmontArchive.
working definition of plagiarism. Retrieved at the time
html).
of publication from http://ori.dhhs.gov/policies/
plagiarism.shtml. The Declaration of Helsinki
Steneck, N. H. (2004). ORI introduction to the responsible Within the international context, the major document sum-
conduct of research. Washington, DC: U.S. Department marizing ethical principles for research is the
of Health and Human Services. Retrieved at the time of Declaration of Helsinki. The World Medical
publication from http://ori.dhhs.gov/documents/ Association initially adopted this statement of ethical
rcrintro.pdf. principles in 1964, and last amended the document in
2000. It is available online at: http://www.wma.
RESOURCES net/e/policy/b3.htm.
A study course is available to occupational therapists
U.S. Federal Research Integrity Policies entitled: “Promoting Research Integrity in the Next
There are several U.S. federal research integrity policies Generation of Occupational Therapy Researchers.” It
posted on the Office of Research Integrity (ORI) Web was developed through a contract to the American
site (including those from the Public Health Service Occupational Therapy Association and Foundation
and the National Science Foundation) (http://ori. (AOTF/AOTA) from the Office of Research Integrity
dhhs.gov/). The federal regulations regarding the con- administered by the American Association of Medical
duct of human subjects research that is funded by the Colleges. Information can be found at the AOTA Web
Department of Health and Human Services, as well as site; www.aota.org.
C H A P T E R 3 0
Obtaining Funding for Research

Renée R. Taylor • Yolanda Suarez-Balcazar • Geneviève Pépin • Elizabeth White
The most common mechanism for funding of 1. Research grants,

research is a grant. A grant award is a specified 2. Demonstration grants,
amount of money given to an investigator or to the 3. Training, educational, and professional develop-
investigator’s parent institution to undertake a ment grants, and
specific research project. Grants may also include 4. Center grants.
funding for development, educational, training,
and/ or evaluation projects that often have a Research Grants
research component. A grant application is the
document that an investigator prepares to request Research grants allow investigators to address sci-
the funding. This chapter covers the nature of grant entific questions that will contribute to knowledge
awards, what they fund, who provides them, and in a given topic area. There are numerous kinds of
how to prepare a competitive grant application. studies that research grants tend to fund, but they
can generally be classified into two major groups:
1. Basic (bench) science studies, and
The Purpose of Grant Funding 2. Applied science studies.
Basic science studies are typically narrow in
The various agencies that provide grants do so to focus. Rather than addressing a practical issue or
make possible a research project that might other- clinical problem, basic science studies provide the
wise not occur. Thus, they often cover all or most necessary knowledge and background for later
of the expenses associated with the research study. applied research. They are designed to generate
Depending on the guidelines of the granting knowledge about a particular theory or about a
agency, grant funding may be used to cover: basic diagnostic, biological, behavioral, attitudi-
nal, or emotional phenomenon. Basic science
• All or a certain percentage of team members’ studies may include, but are not limited to, epide-
salaries, miological studies, laboratory studies, and field
• Tuition waivers and small monetary stipends to observations.
graduate research assistants, Applied science studies aim to test the applica-
• Supplies and equipment that is necessary for the tion of a particular theory to a practical life prob-
conduction of the study (e.g., tests, mechanical lem. Applied science studies may be used to
devices for experiments, computers, printers, and develop new technologies or intervention appro-
relevant software), aches (e.g., rehabilitation strategies). Applied stud-
• Incentives to participants, and ies commonly involve evaluating the effectiveness
• The costs of ancillary needs such as telephone, of the application of one or more of these tech-
postage, transportation, printing, and photocopy nologies or intervention approaches. A well-known
costs. example of an applied study is a clinical outcomes
Some grants also cover indirect costs of doing study (e.g., a randomized clinical trial) in which
the research, including, for example, space rental one or more technologies or intervention appro-
or maintenance, the cost of heating and air condi- aches are compared against a control condition to
tioning, and electricity. evaluate their efficacy in addressing a given clini-
cal problem. Participatory research that seeks to
empower individuals to transform their current
skills or circumstances through education and
Major Types of Grants social action is another example of applied
research. More information about participatory
There are four major types of grants that may be research methods is provided in Chapters 38 to 40
obtained by occupational therapy researchers: of this text.
Chapter 30 Obtaining Funding for Research 487
Alternative Mechanisms of Funding Training, Educational, and Professional

Development Grants
Depending on the funding agency or sponsor,
funding can be achieved through different mech- Training and educational grants are used to support
anisms. For example, in the United States, the professionals and students to develop or extend
National Institutes of Health (NIH) uses three their knowledge or skills. These grants can be used
types of funding venues to support researchers: to support professional activities that involve train-
grants, contracts, and cooperative agreements. ing and education (e.g., conferences or symposia).
Grants differ from contracts and cooperative They can also be used to support the implementa-
agreements in that the investigator has more
tion of specialized academic programs. For exam-
influence in deciding the research topic to be
designed or developed and the accompanying ple, a training grant can be issued to a university
methodological approach. With contracts the for the purpose of supporting first-generation
government or private funder usually decides undergraduate students from underserved groups
on and selects the research that fulfills the per- who have a goal of pursuing graduate training
ceived need and then specifies detailed logisti- in occupational therapy. Similarly, a professional
cal or methodological requirements that an development grant is used to enable an individual
investigator is then asked to carry out. A coop- who is already employed in a professional capacity
erative agreement is similar to a grant, but the to begin a career in research or to further develop
awarding institute or center and the researcher research-related skills and contributions.
both have significant involvement together in
carrying out the activities of the project. Because
most funding agencies tend to utilize research Center Grants
grants as their primary funding source, this
chapter focuses on describing the process of Center grants are usually large grants; they are typ-
writing and applying for research grants. Infor- ically funded for about 5 years with the possibility
mation about contracts and cooperative agree- of competitive renewal. Center grants generally
ments can be obtained from literature provided involve several related projects that incorporate
by individual funding agencies (see http:// research, development, implementation, training,
www.npguides. org/guide/index.html for and dissemination activities. Center grants also
resources). involve identifying multiple partners and collabo-
rators, which might include several researchers
from across the country and/or multiple agencies
who have access to potential participants or who
Demonstration Grants want to participate in training and/or dissemination
The aim of a demonstration grant is to allow activities. Although center grants often involve
investigators to develop, expand, and evaluate a multiple collaborators and disciplines, they are
specific set of healthcare services, a model pro- focused and theme-related. As such, all proposed
gram, or a particular methodological approach research, development, or training activities are
(Gitlin & Lyons, 2004). For example, some occu- designed to make contributions to the advance-
pational therapy investigators use demonstration ment of knowledge, practice, and policies in a
grants to develop new assessments, interventions, specific area. Overall, center grants are unique
new ways to disseminate healthcare information, opportunities to develop state-of-the art innova-
or assistive devices. Demonstration grants allow tions and engage in multidisciplinary research, dis-
investigators to build on existing knowledge about semination, and training activities.
the efficacy of a given approach to service or See the feature box on the next page for an
programming (Gitlin & Lyons, 2004). They typi- example of a center grant.
cally involve some kind of program evaluation
component that involves elements of research.
These grants typically take place in settings seek-
ing to expand or alter their services and those
Reasons to Apply
wanting to develop new programs that can serve as for Grant Funding
models for later replication in other settings.
Demonstration grants may also be used to support In many academic settings, grant funding is vital
the evaluation and modification of ongoing pro- to the daily operation and activities of the organi-
grams in clinical, industry, educational, and com- zation. Many clinics, clinical training programs,
munity settings. and academic departments would not exist in the
The University of Illinois at Chicago Center study that otherwise would not be possible. Or,
for Capacity Building for Minorities with they allow a study to be larger in scope and greater
Disabilities Research (CCBMDR) in impact than would otherwise be possible.
Moreover, grants allow an investigator to work
This center grant was funded for 5 years for with a funding agency to produce peer-reviewed
a total amount of $ 3 million by the National research that serves public, legislative, and/or pri-
Institutes of Disability and Rehabilitation vate interests. Grants also allow an investigator to
Research (see Balcazar & Suarez-Balcazar, advance his or her career and work collaboratively
2004). The center was funded to increase the with a research team that mutually enriches and
capacity of community-based organizations supports the efficiency, productivity, and profes-
such as Centers for Independent Living, State sional development of all of its members.
Vocational Rehabilitation Agencies, and other
agencies serving minorities with disabilities
to identify services, improve their capacity
to provide culturally competent services, and How Grants Get Funded
improve their capacity to evaluate the impact
of services on the lives of participants. The cen- In most cases, grant funding occurs as a result of a
ter will also develop state-of-the-art scientific rigorous review process, discussed in a later sec-
knowledge on issues related to cultural compe- tion of this chapter. One thing is certain: Grant
tence, adaptation and development of culturally funding does not occur in a vacuum. It requires the
appropriate vocational rehabilitation assessment involvement of a number of entities, including:
tools, and the development of disability identity
among minorities. The center grant also will • One’s sponsoring university, clinic, or home
undertake five research projects. institution,
The center involves a principal investigator • Administrators within the funding agency (often
and a co-principal investigator from the Univer- referred to as project officers),
sity of Illinois at Chicago, nine research collab- • Individuals charged to review the relevance and
orators from five different states (California,
Illinois, Massachusetts, Montana, and Texas),
quality of the grant application (often referred to
and a number of community partner agencies as peer reviewers or grant reviewers), and
in each of the five states including Centers for • Individuals who take the feedback of project offi-
Independent Living, other community-based cers and reviewers and ultimately oversee the
organizations, and Vocational Rehabilitation allocation of funds within a granting agency
agencies. These collaborators will be working (e.g., agency trustees or a board of directors).
together to conduct a national conference and
then produce a state-of-the-science book and In addition to these individuals, investigators
other forms of scholarship such as publications are wise to identify experienced colleagues, men-
and professional presentations. tors, or hired reviewers to evaluate their ideas,
methods, and eventual written grant proposal
before it is submitted for formal review. When
appropriate (e.g., when using research methodolo-
gies that emphasize consumer participation and
absence of grant funding. Within the field of occu- representation), investigators should also involve
pational therapy, grants support the refinement, and include prospective research participants in
advancement, and empirical study of educa- the grant writing and initial grant review and eval-
tion, theory, assessments, technologies, and serv- uation process.
ices. Grant funding is also an indirect source of
reputation-building and publicity for an organiza-
tion and the occupational therapists involved.
For all of these reasons, grant funding is often
The Process of Grant
an expectation of occupational therapy faculty Writing and Application
members working in top research universities.
Likewise, occupational therapists working in prac- Grant writing takes time and commitment as well
tice settings may be involved in writing grants. as advance planning and preparation. In the grant
Such grants enable advancing, maintaining, and/or writing and application process, one must justify
evaluating programs of service. and plan the research, write and package the pro-
Grants also provide individual benefits to the posal to meet the requirements of the funding
investigators who receive them. They allow an agency, gather institutional support and necessary
investigator to have the resources to conduct a collaborators or consultants, submit the proposal
to the funding agency, and follow up with the have the potential to be of high impact in terms of
funding agency once the grant has been submitted. understanding, preventing, reversing, or alleviating
A successful grant includes a good idea, knowl- certain health conditions. Developing a grant pro-
edge of hot topics and current funding initiatives posal idea with potential to be of high impact
and policies, sophisticated understanding of re- involves the following considerations:
search design and methods, a good track record,
• Defining impact,
and patience.
• Taking into account policy documents and leg-
Grant writing encompasses the following steps:
islative initiatives,
• Developing an idea, • Matching your idea to the goals and priorities of
• Evaluating and negotiating with the sponsoring the funding agency, and
institution, • Building on existing contemporary scientific
• Identifying and enlisting support from co-investi- trends.
gators, consultants, and other future personnel, Often, one of the main challenges in evaluating
• Selecting the appropriate funding agency, fund- one’s idea in terms of impact involves knowing the
ing institute, and funding avenue, ideology and funding priorities of the agency to
• Knowing regulations, policies, and guidelines, which one is applying.
• Working with funding agency administrators,
• Identifying a theoretical basis for the study, Defining Impact
• Demonstrating expert knowledge of the topic
area, Some agencies consider the severity, imminence,
• Demonstrating good scholarship, and potential for reversibility of a condition in
• Conducting pilot research, determining their funding priorities and deci-
• Identifying specific aims, sions. They may consider some diseases, chronic
• Developing hypotheses, illnesses, or impairments to be more worthy of
• Choosing an appropriate and rigorous design, funding than others based on fatality rates or
• Ensuring an ethical design and methodological other characteristics of the disease or population.
approach, Other agencies may value certain methodological
• Addressing logistical issues and obstacles in data approaches over others. For example, an agency
collection up front, whose priority is to reduce and eradicate highly
• Planning analyses, prevalent diseases with high mortality rates may be
• Developing a timeline and evaluation plan, more inclined to fund medically innovative
• Developing a reasonable budget request, research that involves biomarkers and/or aspects of
• Obtaining letters of support, and the human genome over research that focuses on
• Determining where to send the grant. improving the quality of life of individuals living
with the condition. Conversely, a different agency
Although these steps are presented and dis- may be more inclined to fund research that focuses
cussed below sequentially, in reality many of them on empowerment and capacity-building for indi-
are often performed simultaneously. Some steps viduals with existing impairments. An added
will be left incomplete as others are initiated. The priority for many agencies is reducing health dis-
scramble to prepare a competitive application parities. Such agencies will tend to fund re-
often requires substantial multitasking and cross- searchers that work with participants who do not
checking between all of the steps. Moreover, the have adequate economic resources, educational
order of the steps may vary depending on the fund- and employment opportunities, and access to
ing source and the kind of competition to which healthcare. Knowing these priorities can help a
one is applying. Nonetheless, in most cases, the researcher determine how reviewers might evalu-
steps discussed below will be required for prepar- ate a proposal in terms of its overall significance
ing a grant proposal. and potential for impact.
Developing an Idea: Taking into Account Policy Documents

The Importance of Impact and Legislative Initiatives
The most critical aspect of preparing a grant pro- Being knowledgeable about politics, current
posal is developing a research idea that is signifi- events, and legislative initiatives can aid in deter-
cant and innovative enough to warrant funding. mining whether a funding idea will be considered
Agencies want to utilize their money wisely and to be important and of high impact. Knowledge of
parsimoniously. They want to be sure that studies legislative initiatives is particularly important when
it comes to obtaining funding from federal agencies Evaluating and Negotiating with
(Gitlin & Lyons, 2004). One can become familiar the Sponsoring Institution
with legislative initiatives as they are reflected on
funding agency Web sites, in government publica- Before deciding to write a grant, one must identify
tions such as the Federal Register in the United and evaluate existing resources within one’s own
States, and in funding announcements. At the institution. Sponsoring institutions often have
broadest level, one can obtain knowledge about internal rules and regulations that govern the grant
upcoming federal funding priorities by reading the submission process, many of which involve budg-
daily newspaper and watching the news, and by etary and resource issues. For example, some
watching and otherwise keeping current on issues sponsoring institutions specify a minimum require-
in healthcare policy through other media outlets. ment for indirect cost support provided by a grant-
ing agency. Conversely, to award grant funding to
Matching One’s Idea to the Goals and an investigator, some funding agencies require a
Priorities of the Funding Agency certain level of commitment of monetary or in-kind
support (e.g., a certain percentage of cost-sharing
One should always ensure that the research topic, or matching funds) from a sponsoring institution. It
population, and methodological approach reflect is important to clarify issues of resource allocation
the goals and priorities of the funding agency to and administrative rules and regulations before a
which one applies. A number of steps can be grant proposal is submitted so that all agreements
applied to effectively match one’s research idea to are in place should a grant get funded.
the agenda of the funding agency: Another important factor to consider during
• Consult relevant Web-based and other resources negotiations with one’s sponsoring institution is
(many of which are discussed in this chapter) to that grant writing takes time and commitment.
develop a preliminary working list of possible Applicants will need support in the form of time
funding agencies to approach, allocated to writing and preparing the grant pro-
• Periodically scan the Web and other resources for posal and other types of support such as access to
program announcements and/or funding agencies secretarial, research assistant, and administrative
that reflect one’s area of interest, support for such diverse things as conducting liter-
• Consult with respected peers, mentors, and pro- ature searches, gathering protocols and instru-
gram officers to receive feedback about the ments, gathering letters of cooperation, preparing a
match of funding agencies to one’s idea, and budget, and making photocopies. The sponsoring
• Refine and define one’s proposal ideas to match institution should also be willing to release the
chosen potential agencies based on the informa- applicant from other responsibilities (e.g., commit-
tion gathered. tee work, teaching) so he or she has the necessary
time to construct a strong proposal.
Building on Existing Contemporary
Identifying and Enlisting Support
Scientific Trends
from Co-Investigators, Consultants,
Most funding agencies keep relatively current in and Other Future Personnel
terms of their knowledge of methodological
approaches that are contemporary and/or on the Most grant evaluation criteria include an assess-
cutting edge of science. In addition, certain scien- ment of the strengths and credentials of the various
tific trends tend to develop and some gain a sub- members of the investigative team. Thus, selection
stantial amount of credibility and support within of one’s team members is critically important and
the research community. For example, recent requires thought and effort. Depending on the size
trends within healthcare research include: of one’s study, research team members can include:
• A focus on participatory approaches to program • Co-investigators/subcontractors,
development, service provision, and healthcare • Consultants (e.g., biostatisticians),
reform, • Grant staff,
• Interdisciplinary research (e.g., research that • Student research assistants, and
draws upon the expertise of professionals from a • Volunteers.
wide range of disciplines), and Co-investigators (commonly paid as subcon-
• Translational research (e.g., studies that incorpo- tractors unless they are housed within the same
rate a range of approaches that span the basic and institution) are research personnel critical to con-
applied concerns). ducting the study. They:
• Share responsibility for the intellectual contribu- Sometimes, funding decisions are made even
tions made to the development of the study idea, before the review process begins because the
design, and methods and analyses, and investigator has selected the wrong agency, fund-
• Collaborate with the principal investigator in ing institute, or funding avenue for the proposed
interpreting the findings and in accessing avenues project. Some of the more widely utilized fund-
for dissemination. ing sources that may be accessed by occupational
therapy researchers and their collaborators
Co-investigators are typically senior-level sci-
include:
entists with the knowledge base, technical skills,
publication history, and a scientific reputation that • Grants awarded by professional organizations,
support the central aims of the study and comple- • Grants awarded by private foundations,
ment the credentials of the principal investigator • Grants awarded by self-help organizations,
and other collaborators. Increasingly agencies • Grants and grant competitions within university
expect a research team reflecting diverse and com- settings, and
plementary disciplines. • Grants awarded by the federal government.
Consultants are of similar status and complete
similar functions as co-investigators. However, Grants Awarded by Professional
their role is often more circumscribed and their
Organizations
contributions to the overall study are proportion-
ally smaller than those of the co-investigators. Co- Grants awarded by professional organizations are
investigators and consultants are usually selected useful resources for individuals seeking to advance
before a proposal is written and they are always the profession. As such, their scope is limited to
identified in the grant proposal. Most funding projects within the singular discipline of occupa-
agencies require that they provide curricula vitae tional therapy and typically funding is provided
and written letters of support. In many cases, a sub- for tightly constructed, time-limited, and highly
contract agreement will be in place that allows for focused research studies, projects, or professional
formal budgetary relationships to be established. educational activities. Examples of organizations
Other members of the research team, including that award grants to occupational therapists
grant staff, student research assistants, and volun- are noted in the feature box titled “Professional
teers, can be named once the grant has been Organizations Funding Occupational Therapy
funded. However, some reviewers look more Research.”
favorably upon grant applications that identify key
grant staff because it leaves an impression that the Grants Awarded by Private Foundations
investigator has a stable research team. However, it
is often not possible to name the more junior-level Numerous private foundations support research,
or secondary contributors up front. developmental, and educational activities. The
Selecting collaborators involves deciding what mission and funding agenda of private foundations
intellectual and physical resources are needed to are as varied as the individual donors (e.g., Tiger
complete a study and determining who might be Woods Foundation), families (e.g., Field Family
available to meet those needs (Gitlin & Lyons, Foundation of Illinois), and private industries (e.g.,
2004). Selecting strong collaborators will not Procter & Gamble) that provide grants. Private
only increase the likelihood of a positive review, foundation grants can range from small award
but also ensure the overall success of the study. amounts to awards in excess of US$1,000,000 for
Enlisting support from collaborators early in a single application. Topics for funding generally
the grant writing process can be vital to idea devel- focus on, but are not limited to, community-based
opment. In addition, collaborators may support initiatives directed at improving communities,
activities such as grant writing, study implemen- improving education, reducing conflict and vio-
tation, and the write-up and dissemination of study lence, improving access and participation for indi-
findings. viduals with disabilities, increasing job skills and
employment for underserved, mentally ill, home-
Selecting the Appropriate Funding Agency, less, or adjudicated individuals, and reducing dis-
ease and disability and improving health outcomes
Funding Institute, and Funding Avenue
for a wide range of populations and human condi-
Being knowledgeable about the missions, values, tions. The feature box titled “Private Foundations”
and funding priorities of the different agencies that discusses three foundations that fund research rel-
fund the kind of research one intends is vital. evant to occupational therapy.
Professional Organizations Funding Occupational Therapy Research
Occupational therapy professional associations tions. Details are advertised on the Web site at
are increasingly becoming sources of funding. http://www.cot.org.uk.
Below information is contained on occupational
therapy professional bodies form the United Canadian Occupational Therapy Foundation
States, Canada, the United Kingdom, and the
World Federation of Occupational Therapy. The Canadian Occupational Therapy Foundation
Occupational therapy researchers in countries (COTF) is a nonprofit professional organization
other then those mentioned here are encouraged that works in tandem with the Canadian Associa-
to identify additional possible sources of funding tion of Occupational Therapists (CAOT) to
within their own national professional bodies. develop mechanisms for granting awards to indi-
viduals and organizations for research, scholarship,
American Occupational Therapy Foundation and publication (COTF, 2005). COTF provides
opportunities for occupational therapy researchers
The American Occupational Therapy Foundation whose aim is to address the evolving needs of the
(AOTF) is a nonprofit organization whose mission occupational therapy community in Canada. COTF
is to advance occupational therapy research spe- generates, receives, and maintains funds to support
cifically as it informs clinical practice. AOTF is a broad range of research and scholarship in the
also focused toward efforts that increase public field of occupational therapy.
understanding of occupational therapy services. With the support from CAOT and donations
In conjunction with ongoing support from the from individuals, corporations, organizations,
American Occupational Therapy Association and foundations, COTF has awarded a total of
(AOTA), AOTF has provided nearly $4,000,000 CAN$51,500 to occupational therapy researchers.
in support for research grants and projects. Cur- COTF has three different grant programs: The
rently, there are three types of research studies Research Grants (CAN$28,000), the Scholarships
that AOTF and AOTA conjointly fund: innova- Grants (CAN$22,000) and a third category,
tion studies (US$8,000 maximum award amount Other (CAN$1,500). Further details regarding
per application), impact studies (US$30,000 each program, their policy, eligibility and
maximum award amount per application), and reviewer criteria can be found at http://www.
student research grants (US$1,000 maximum cotfcanada.org.
award amount per application). A listing of 10
current occupational therapy research priorities World Federation of Occupational Therapists
of AOTF/AOTA can be found at http://www.aotf.
org. Gillette (2000) has published a 20-year history Every 2 years, the World Federation of Occupa-
of research funding in occupational therapy that tional Therapists (WFOT) reviews applications
describes the various activities and awards made for the Thelma Cardwell Foundation Award for
by AOTF and AOTA. More information about Research and Education. This award supports
grants funded by AOTF and AOTA can be found any project aiming to enhance the development
at http://www.aotf.org. of occupational therapy in any way. The award
does not support research projects for which
British Association/College of funds could be sought from governmental agencies
Occupational Therapists or other grant-giving foundations. The budget is
restricted to include only coverage for equipment,
In the United Kingdom, the professional body maintenance, or technical assistance for an already
administers a range of grants for research, educa- approved research project. Thus, WFOT funding
tion, and professional development. These are is best used to supplement other types of funding
available mainly on an annual basis, though some provided for a given study. The Thelma Cardwell
support is also available for low-cost courses that Foundation Award offers investigators a maximum
occur at short notice. In addition, travel bursaries amount of US$5,000 per application. More infor-
are offered at three points in the year. Awards arise mation about this grant award and evaluation crite-
both from restricted funds held by the College of ria used to select applications can be found at
Occupational Therapists (COT) and grants made http://www.wfot.org.
available from companies and charitable organiza-
(continued)
492
Private Foundations
Below a few private foundations from thousands in the Langeloth Foundation, readers may access the
existence around the world are described. foundation’s Web site at http://www.langeloth.org.
U.S. Foundations Canadian Foundations

The Robert Wood Johnson Foundation is the ln Canada, most grants for occupational therapy-
largest private philanthropy within the United based research come from private organizations,
States that is exclusively devoted to improving charities, and associations such as the Alzheimer
health and healthcare. This foundation has made Society of Canada, the Canadian Cystic Fibrosis
basic science and applied research grant awards Association, the Parkinson Society of Canada, the
ranging from US$1,200 to US$50,000,000. Grants Canadian Mental Health Association, or the Royal
are announced through calls for proposals that are Canadian Legion Fellowship in Gerontology.
highly specific to the goals and agenda of the pro- Other important sources of funding come from
gram that issues the call. Independent grants (i.e., each province’s own funding agencies. Regardless
unsolicited grant applications that reflect an inves- of whether they are governmental or private, each
tigator’s unique ideas) can also be funded. Some has specific research interests, a mission, and pro-
of the foundation’s interest areas are prevention cedures for submitting a research project, and poli-
and treatment of addictions, building human cies and criteria for funding. For example, in the
capital within the healthcare workforce, health province of Québec, the Fonds de recherche en
disparities, quality healthcare, and pioneering santé du Québec (http://www.frsq.gpouc.qc.ca)
research that promotes fundamental breakthroughs funds specific rehabilitation programs, such as the
in health and healthcare. More information about Reseau Provincial de recherche en Adaptation-
funding through the Robert Wood Johnson Réadaptation. The Fonds de recherche sur la
Foundation can be found at http://www.rwjf.org/ société et la culture du Québec (http://www.fqrsc.
index.jsp. gouv.ca) is another funding agency that opens
The John D. and Catherine T. MacArthur foun- doors for occupational therapists concerned with
dation is a private, independent organization dedi- social inclusion, diversity, and adaptation.
cated to promoting a lasting improvement in the
human condition. The foundation awards large United Kingdom Foundations (Charities)
(i.e., over US$1,000,000) and small (i.e., under
US$5,000) basic science and applied research. In the United Kingdom, sources of grants for occu-
More information about funding from the pational therapists arise from a wide range of char-
MacArthur Foundation can be found at ities. For instance, the Wellcome Trust is an
http://www.macarthur.org. independent charity that funds research to improve
The Jacob and Valeria Langeloth Foundation human and animal health. Two of its funding
is a private philanthropy that awards applied streams of relevance to occupational therapists are
research grants in the area of healthcare (mainly Biomedical Science and Medical Humanities. The
to hospitals and other healthcare facilities). In Wellcome Trust also funds 4-year PhD scholar-
2004, Langeloth awarded 31 grants totaling to ships in Life Sciences. Further information is
US$5,261,943. This averages to approximately available from http://www.wellcome.ac.uk/
US$169,740 per grant award. Langeloth’s mis- funding. The Joseph Rowntree Charitable Trust
sion is to support effective and creative healthcare was founded by Joseph Rowntree, a Quaker busi-
practices and policies related to recovery from ill- nessman, in 1904. This Trust makes available
ness, accident, and physical, social, or emotional grants totaling some £4m each year. Information
trauma. For more information about funding from can be obtained from http://www.jrct.org.uk.
Grants Awarded by Self-Help process. The size of grants awarded by self-help

Organizations organizations is generally commensurate with the
size of the membership and the amount of contri-
Grants awarded by self-help organizations largely butions made to a given organization. However,
serve the interests of clients who have experienced because self-help organizations are largely sup-
or are experiencing a specific illness, trauma, or ported by small donations made by clients, their
impairment, their loved ones, and their specialist loved ones, and their healthcare providers, they
healthcare providers. As such, the scope of grants tend to make smaller or more mid-sized grant
provided by self-help organizations is limited to awards. There are thousands of self-help organiza-
projects that focus on a given condition or disease tions that fund grants throughout the world. The
feature box titled “The Arthritis Foundation” high- application to be submitted elsewhere, or to initi-
lights an organization that tends to fund a larger ate a new line of research that is perhaps too novel
number of grants per annum than many others. to receive funding in a competitive environment
outside of the university setting. The amount of
Grants and Grant Competitions funding available for these grants tends to be
Within University Settings small, and usually funding is limited to only a few
proposals. Information about limited internal com-
Two general types of grant competitions occur petitions and seed grants is generally provided
within university settings: limited internal compe- through a university research office.
titions and seed grants. Limited internal competi-
tions are made available to research-oriented Grants Awarded by Governments
universities by certain kinds of funding agencies
(mostly federal agencies). These competitions Each country, and often subjurisdictions such as
involve two or more phases. The first phase occurs states or provinces, have granting programs. These
between faculty within the university to limit the vary widely by country or jurisdiction. Major gov-
number of applications from that university. Many ernmental funding bodies in the United States,
limited internal competitions accept only one or Canada, and the United Kingdom are briefly
two applications per university. The application reviewed in this chapter. Readers from other coun-
that emerges as strongest and most relevant from tries will want to investigate their national and
the internal competition is the one that is selected. local governmental sources of funding. Ordinarily,
Many university settings, particularly those information is publicly available on Web sites and
with a research focus or emphasis, offer opportu- official government publications. Nonetheless, it
nities for faculty internal to the university to com- often takes substantial time and effort to learn all
pete for small grants that are offered by the the mechanisms, rules, regulations, deadlines and
university itself. These grants are often referred to so on that are part of government funding. Before
as seed grants. In large part, they are designed to applying for a government grant, one should
provide funding (or other resources) to help an become as familiar as possible with this type of
investigator collect pilot data for a later grant information.
Federal Grants in the United States. A number
of federal granting agencies have programmatic
The Arthritis Foundation interest areas relevant to occupational therapy
researchers in the United States. This section will
The Arthritis Foundation is a multifaceted discuss those with the largest history of funding
national self-help organization that has funded occupational therapy research. The National
more than 2,100 researchers with more than Institutes of Health (NIH) is the primary federal
US$272,000,000 over the past 50 years. As the funding agency that supports medical research.
largest, private, nonprofit contributor to arthritis Most applications sent to NIH are investigator
research in the world, the Arthritis Foundation initiated. This means that the research idea (e.g.,
funds a wide spectrum of large and small basic central aims of the research, topic area, study
and applied research studies conducted by a
design, and methodology) is unsolicited and
broad range of disciplines that focuses on arthri-
tis prevention, treatment, and cure. Clinical stud- uniquely a product of the investigators’ thinking.
ies have been funded to test novel medical, The investigator must be responsible for the plan-
rehabilitation, surgical, and psychoeducational ning, direction, and execution of the project.
treatments. Epidemiological studies and health Despite the implicit intellectual liberties associ-
services/quality of care research are also impor- ated with this funding avenue, a caveat that many
tant funding priorities. In addition to studies of applicants forget is that their ideas still must incor-
various forms of arthritis, the foundation also porate ideas, aims, study populations, and methods
funds studies of related conditions, including that are considered of relevance to at least one of
systemic lupus, osteoporosis, Lyme disease, scle- the various institutes or centers within NIH.
roderma, and fibromyalgia. The foundation funds
Opportunities for grant funding are announced
investigator-initiated grants, career development
and training programs, and special targeted in a number of ways. A Program Announcement
research initiatives. More information about (PA) is a formally prepared statement in writing
grant funding through the Arthritis Foundation that invites applications in a defined area of inter-
can be located at http://www.arthritis.org/ est. A PA is not a guarantee that funds have been
research. set aside to support the defined interest area, and
applications are generally treated as being investi-
gator initiated in all other respects. In a Request for research that enhances the quality, appropriate-
Applications (RFA), NIH invites applications for a ness, and effectiveness of healthcare services and
one-time competition in a specific topic area and service access. Topical areas of research cover the
describes an institute’s initiative in a well-defined organization, financing, and delivery of healthcare
scientific area to stimulate research in an area of services, disease prevention, and the improvement
exceptionally high priority to the institute. In this of clinical healthcare practices. Most of the large
case, the RFA does guarantee that a certain amount grants that AHRQ funds involve research projects,
of funding has been set aside to support the defined demonstration projects, program evaluations, and
interest area, and it specifies up front how many dissemination activities.
awards will be made. A Request for Proposals Because priorities are based on legislation,
(RFP) is similar to an RFA except that an RFP in- policies, and public need, areas of specific interest
volves a contractual relationship between the inves- for AHRQ change regularly. AHRQ interest areas
tigator and NIH, rather than a grant. RFAs and tend to be published in the form of Requests for
RFPs are dedicated mainly to problem-oriented Proposals (RFPs), program announcements (PAs),
research efforts that focus on disease-specific ini- and notices. More information about current
tiatives, particularly in the beginning stages of AHRQ funding opportunities can be accessed at
research. http://www.ahrq.gov/fund/ongoing.htm. In addi-
NIH is comprised of a number of different insti- tion, announcements can be searched regularly by
tutes and centers, each dedicated to a specific accessing the NIH Guide for Grants and Con-
health-related topic area and mission. Some exam- tracts, which can be found at http://grants.nih.gov/
ples of NIH institutes that may be of particular rel- grants/guide/index.html.
evance to occupational therapy researchers
include: U.S. Department of Education. The U.S. Depart-
ment of Education is comprised of nine principal
• The National Institute on Child Health and offices (somewhat akin to NIH institutes). A full
Human Development (NICHD), which also listing can be found at http://www.ed.gov. Among
houses the National Center for Medical Rehabili- them, the Office of Special Education and Reha-
tation Research, bilitation Services (OSERS) is one of the offices of
• The National Institute on Aging (NIA), highest relevance to the work of occupational ther-
• The National Institute on Alcohol Abuse and apy researchers working in educational, university,
Alcoholism (NIAAA), and clinical rehabilitation settings. Within the prin-
• The National Institute on Drug Abuse (NIDA), cipal offices of the Department of Education, there
• The National Institute of Arthritis and are program offices. For example, the National
Musculoskeletal and Skin Diseases (NIAMS), Institute on Disability and Rehabilitation Research
• The National Institute of Mental Health (NIMH), (NIDRR), the Office of Special Education Pro-
and grams (OSEP), and the Rehabilitation Services
• The National Institute of Neurological Disorders Administration (RSA) are housed within OSERS.
and Stroke (NINDS), among several others (see All of NIDRR’s programmatic efforts are aimed at
http://www.nih.gov/icd for a complete listing). improving the lives of individuals with disabilities
Specific research topics that represent strong throughout the entire lifespan. Projects that are
interest areas for the NIH can be searched regu- funded by NIDRR aim to maximize the full
larly by accessing the NIH Guide for Grants and inclusion, independent living, employment, and
Contracts, which can be found at http://grants.nih. economic self-sufficiency of individuals with dis-
gov/grants/guide/index.html. abilities. OSEP supports educational, technologi-
Table 30.1, which was composed from informa- cal, and developmental initiatives for infants,
tion in several tables provided on the NIH Web site toddlers, children, and adolescents with disabili-
(http://grants1.nih.gov/grants/funding/funding_ ties. RSA focuses on funding efforts that enable
program.htm), describes some of the different individuals with disabilities to obtain employment
types of grants awarded through the NIH that are and live more independently through supports
relevant to occupational therapy researchers. such as counseling programs, medical and psycho-
logical services, job training, and other individual-
Agency for Healthcare Research and Quality. ized services.
The Agency for Healthcare Research and Quality The U.S. Department of Education’s name for
(AHRQ) is a Federal funding source within the what NIH calls an extramural grant is a discre-
U.S. Department of Health and Human Services tionary grant. A discretionary grant is an award
Public Health Service dedicated to funding that the Department has chosen to make based on
Table 30.1 Selected* Funding Awards Made by NIH

Award Type* Brief Definition
Research Grants
NIH Research Project Grant Program (R01) An R01 is typically a larger research grant award made to
http://grants1.nih.gov/grants/funding/r01.htm support a very well-defined and highly specific research
project. RO1s provide support to investigators for health-
related research and development projects that coincide
with the NIH mission. R01s can be funded for a period
of 1–5 years and can total to more than US$2,000,000
for a full 5-year funding period.
NIH Small Grant Program (R03) An R03 is a research grant award that provides limited
http://grants1.nih.gov/grants/funding/r03.htm funding for a short period of time. It can be used to
fund pilot or feasibility studies, secondary analysis of
existing data, small, self-contained research studies,
development of research methodology, or the
development of a new technology.
NIH Academic Research Enhancement Award AREA grants are small awards that support individual
(AREA Grants) (R15) biomedical and behavioral-science research projects
http://grants1.nih.gov/grants/funding/area.htm that are conducted collaboratively by faculty and
students in undergraduate institutions that are housed in
schools that have not been major recipients of other
types of NIH research grant funding.
NIH Exploratory/Developmental Research An R21 provides a limited amount of support for research
Grant Award (R21) projects, ideas, and methodologies that are excep-
http://grants1.nih.gov/grants/funding/r21.htm tionally novel, potentially groundbreaking, or innovative.
NIH Clinical Trial Planning Grant Program (R34) The R34 is a 1-year grant award that supports the
http://grants1.nih.gov/grants/funding/r34.htm development of Phase III clinical trials.
NIH Research Career Development Awards

Mentored Research Scientist Development The K01 supports 3–5 years of an intensive, supervised,
Award (K01) career development experience for an investigator
http://grants1.nih.gov/grants/guide/pa-files/ entering a new area of research in a biomedical,
PA-00-019.html behavioral, or clinical science.
International Research Scientist Development The K01–IRSDA supports U.S. postdoctoral biomedical,
Award (KO1–IRSDA) social, and behavioral scientists in newer stages of their
http://grants1.nih.gov/grants/guide/pa-files/ research careers to conduct research or extend their
PAR-04-058.html current research into developing countries.
Independent Scientist Award (K02) The K02 award provides up to 5 years of salary support for
http://grants1.nih.gov/grants/guide/pa-files/ newly independent scientists that can demonstrate a
PA-00-020.html need for a period of intensive research focus that will
enable them to expand their potential to make significant
contributions to their selected area of research.
Senior Scientist Award (K05) The K05 provides salary support for up to five years for
http://grants1.nih.gov/grants/guide/pa-files/ scientists of outstanding caliber that have demonstrated
PA-00-021.html sustained, high-level productivity and whose expertise,
research accomplishments, and contributions to the field
are critical to the mission of the particular NIH center or
institute.
Mentored Clinical Scientists Development The K08 supports specialized study for individuals with a
Award (K08) health professions doctorate that want to gain independ-
http://grants1.nih.gov/grants/guide/pa-files/ ence as a laboratory or field-based researcher.
PA-00-003.html
Mentored Patient-Oriented Research Career The K23 was designed to increase the number of clinicians
Development Award (K23) trained to conduct high-quality patient-oriented clinical
http://grants1.nih.gov/grants/guide/pa-files/ research. This area covers mechanisms of human
PA-00-004.html disease, therapeutic interventions, clinical trials, and the
development of new technologies.
(continued)
496
Table 30.1 Selected* Funding Awards Made by NIH (continued)

Award Type* Brief Definition
NIH Small Business Funding Opportunities
NIH Small Business Innovation Research The SBIR program is designed to encourage U.S.-based
Program (SBIR) small businesses to engage in research and develop-
http://grants1.nih.gov/grants/funding/sbir.htm ment activities that have an impact on health and a
potential for commercialization.
NIH Small Business Technology Transfer The STTR is similar to the SBIR program in that both
Research Program (STTR) programs seek to increase the participation of smaller
http://grants1.nih.gov/grants/funding/sbir.htm businesses in federal research and development and to
increase subsequent commercialization of technologies
developed by this program within the private sector. One
difference is that STTR program applicants are required
to formally collaborate with a research institution.
*Generally, the research awards (preceded by an “R”) are designed to provide support for well-defined research
and development projects. The research career development awards (preceded by a “K”) provide support for
new, mid-career, and senior scientists who are seeking to bring greater focus and/or knowledge. The small
business awards promote the development and private commercialization of new technologies, and to
promote collaboration between research scientists and small business owners.
A more comprehensive listing of the numerous types of grant awards offered by NIH can be found at
http://grants1.nih.gov/grants/funding/funding_program.htm. Because the types of awards offered by NIH
change periodically, readers are encouraged to access the Web site for the most updated information.
a competitive review process. Table 30.2, which aboutHTA. Second, the New and Emerging Tech-
was composed from information in several tables nologies (NEAT) programme supports applied
provided on the U.S. Department of Education research in health and social care http://www.neat-
Web site (http://www.ed.gov), describes some of programme.org.uk/. Third, the Service Delivery
the different types of grants that are awarded and Organisation (SDO) programme is a national
through the U.S. Department of Education. research programme that develops the evidence
base on the organization, management and deliv-
United Kingdom. The purpose of funding for
ery of healthcare services. Further detail on this
occupational therapy research in the United King-
programme is available from http://www.sdo.
dom is to both generate new knowledge and also to
lshtm.ac.uk.
develop research capacity. Many research funders
Within the United Kingdom, there has been
support interdisciplinary research, reflecting the
recognition of the need to strengthen the research
team approach that is essential for effective health
capacity of occupational therapists by supporting
and social care interventions. Among the most
opportunities to undertake research and develop
prestigious funding sources for health and social
research careers (Creek and Ilott, 2002; Depart-
care research in the United Kingdom are the gov-
ment of Health, 2000, 2005; Higher Education
ernment-funded research councils. The Medical
Funding Council for England, 2001; Ilott &
Research Council (MRC) and the Economic and
White, 2001; Scottish Executive 2004). As a result,
Social Research Council (ESRC) invest in funding
increased funding is available for occupational
for high-quality, world-class research.
therapy through various government agencies.
The National Health Service is another highly
This funding supports not only specific research
regarded funding avenue within the United
projects but also efforts to build research capacity.
Kingdom. Three programmes of research are sup-
ported by funding from the National Health Canada. Canada’s major federal funding agency
Service (NHS). First, the NHS Health Technology for health research is the Canadian Institute of
Assessment (HTA) Programme aims to provide Health Research (CIHR) (http://www.cihrirsc.
research information relating to the costs, effec- gc.ca). The CIHR promotes research through
tiveness, and impact of health technologies (i.e., an interdisciplinary structure made up of 13
any approach that is used to promote health, treat institutes. Its philosophy rests on networks of
illness, and improve rehabilitation or long-term researchers brought together to focus on specific
care). Further information on the HTA programme and important health issues. Therefore, the
can be found at http://www.hta.nhsweb.nhs.uk/ Institute’s structure encourages partnerships and
Table 30.2 Selected Grant Awards Made by the US Department of Education Office of Special
Education and Rehabilitation Services (OSERS)
Name of Award Program Office Brief Description
Rehabilitation Engineering National Institute on Support efforts that lead to the development
Research Centers (RERC) Disability and of methods, procedures, and devices
Rehabilitation Research that will benefit individuals with disabil-
(NIDRR) ities, particularly those with the most
severe disabilities.
Research Fellowships Program National Institute on Enable individuals to build their research
Disability and capacity. Individuals with 7 or more years
Rehabilitation Research of experience relevant to rehabilitation
(NIDRR) research are eligible for a distinguished
fellowship award. Merit fellowships are
awarded to individuals in earlier stages
of rehabilitation research careers.
State Vocational Rehabilitation Rehabilitation Services Promote the availability of skilled personnel
Unit In-Service Training Administration (RSA) to serve the needs of individuals with
disabilities through vocational rehabil-
itation, supported employment, and
independent living programs.
Field-Initiated Research National Institute on Facilitate research and development activ-
Projects Disability and ities that maximize the full inclusion,
Rehabilitation Research employment, independent living, and
(NIDRR) economic sufficiency of individuals with
disabilities.
collaboration across sectors, disciplines, and with federal and private foundation health agendas
regions. Some institutes of most relevance to occu- and funding priorities. Each year, new foundations
pational therapy include: emerge that fund research related to healthcare
while at the same time others cease to offer fund-
• Institute of Aging,
ing opportunities related to healthcare research.
• Institute of Health Services and Policy Research,
Many research-oriented universities have for-
• Institute of Human Development, Child and
mal or informal research development services or
Youth Health,
offices, which are geared toward disseminating
• Institute of Musculoskeletal Health and Arthritis,
funding opportunities like these and others to fac-
and
ulty and staff. In addition to university research
• Institute of Neurosciences, Mental Health and
development services, prospective grant applicants
Addiction.
can subscribe to a wide range of publications and
Each Institute, while focusing on a specific Web-based resources that allow access to updated
area, is open to research initiatives that range from information about funding agencies and priorities.
fundamental biomedical and clinical research to Some of these links and guidelines for accessing
research that focuses on cultural dimensions of information about private funding opportunities
health and environmental variables that affect were provided in the section of this chapter on pri-
well-being. vate foundation funding sources. For federal
resources within the United States, the Web site,
Summary http://www.grants.gov/Find, contains updated
information about many types of funding announ-
This section provided a brief overview of different cements.
types of funding agencies. These agencies repre-
sent numerous opportunities available to occupa- Knowing Regulations, Policies,
tional therapy researchers and practitioners
working in a wide range of academic, private, edu-
and Guidelines
cational, and community-based settings. These Because most funding agencies receive numerous
opportunities change continually in conjunction applications for each funding cycle, and because
grants involve (sometimes complex) financial • The administrative aspects of applying for the
arrangements between the agency and the appli- grant (i.e., how to complete required forms and
cant’s home institution, all granting agencies rely progress reports), and
heavily on policies and regulations that guide the • The review process (i.e., whether to resubmit an
application and award process and are generally application and to what extent an investigator
uniformly upheld for all applicants. should respond to certain types of feedback from
Once an investigator is funded, many agencies the reviewers).
have a number of requirements that must be met in
More information about working with project
order for the investigator to retain the funding
officers in deciding whether to resubmit an appli-
award. Depending on the agency, examples may
cation is provided later in this chapter.
include providing periodic written progress reports
that demonstrate that the research team is complet-
ing the work that they promised to complete within Identifying a Theoretical
the expected time frame, and budget monitoring or Basis for the Study
occasional audits to ensure that the research team
is not spending the award money inappropriately High-quality research aims to evaluate the rele-
or purchasing items that have not been approved vance of theory that underlies the research activi-
within the budgetary guidelines. Investigators must ties. In fact, one of the central reasons why many
take great care to follow guidelines like these or an grant proposals receive poor scores from reviewers
agency can and will discontinue funding. Grant lies in the fact that the study is not well justified
writing and proposal preparation, administrative in terms of its relationship to a larger idea or sys-
and budgetary maintenance, and the provision of tem of ideas that support the central hypothesis.
periodic progress reports are not areas in which an Theoretical justification is also a requirement for
investigator is permitted to cut corners or relax studies that focus on the development of an assess-
standards. ment, program, or other rehabilitation resource.
Regardless of the nature of the study, a central the-
ory must be closely linked and utilized to support
Working with Funding Agency the specific aims and methodology of the study.
Administrators For example, if one is designing a clinical trial
There are many different types of funding agency that tests the efficacy of a given approach to reha-
administrators. These may include, but are not lim- bilitation, it would be expected that one would
ited to: have based his or her approach to rehabilitation on
an existing or emerging theory that defines the
• Agency directors, advisory councils, and direc- mechanisms that are expected to underlie the
torial boards (whose main job is to set agency anticipated change.
funding priorities and provide ultimate oversight
over the types of grants that receive funding),
• Referral officers (whose main job is to scan the Demonstrating Expert Knowledge
titles and abstracts of grant applications and of the Topic Area
assign specific reviewers),
For many granting agencies, one of the criteria
• Scientific review administrators (whose main job
by which a proposal is evaluated includes the
is to oversee the logistical, legal, and administra-
estimated expertise of the principal investigator
tive aspects of the review process), and
and the research team. Level of expertise is typi-
• Project officers (whose main job is to guide
cally judged according to a number of variables,
applicants and grantees through the review and
including:
grant management processes).
• Number of peer-reviewed publications in the area
The type of agency administrator with whom
under study,
an investigator is likely to have the most contact is
• Quality of the journals in which the articles are
the project officer (or program officer). Project
published,
officers may be involved with a grant at all stages
• History of prior grant funding in the area under
of its development and implementation. Generally,
study,
however, a project officer can educate an investi-
• Evidence of specialized training and research
gator about:
mentorship in the area under study, and
• The agency’s funding priorities (i.e., the topic • An established area of focus and a tradition of
areas of most interest to the agency), research.
One of the most important determinates of an requires a prospective investigator new to a given
investigator’s expertise is the number and quality area of science to design a study and then obtain
of publications that the investigator and research ongoing supervision and intensive training in his or
team members have in the area under study. her proposed area from an experienced researcher
Another variable that is commonly evaluated is who serves as a career mentor for the investigator.
whether the investigator and/or other team mem- The newer investigator is expected to accomplish
bers have a history of prior grant funding and par- the same research objectives as a more experienced
ticipation on experienced research teams. A researcher with the assistance of a mentor. In addi-
history of prior grant funding coupled with a con- tion to professional development options offered
sistent stream of high-impact publications emerg- by federal agencies, some private foundations and
ing from prior grants are reasonable indicators that self-help organizations wanting to attract new
the investigator has experience successfully man- investigators into an emerging or understudied area
aging the intellectual, managerial, budgetary, and may place more value upon an investigator’s
logistical challenges involved in carrying out a demonstrated interest in the research agenda and
grant-funded study. funding priorities of the agency than upon the
One of the most basic indicators of expertise, investigator’s preestablished track record of
particularly for emerging research professionals, is research in that area. Seed funding is also offered
evidence of an area of focus and a tradition of within many university settings to assist newer
research (Taylor, Fisher, & Kielhofner, 2005). This investigators in establishing an area of focus, a tra-
most basic indicator overlaps with all of the other dition of research, and a publication record.
criteria mentioned in this section because develop-
ing an area of focus and a tradition of research nec- Demonstrating Good Scholarship
essarily involves receiving good training and
One often unspoken but critical criterion for suc-
mentorship in research and building a portfolio of
cessful grant writing is to demonstrate good schol-
evidence of ongoing research involvement in the
arship in writing and assembling the grant
area. For those just starting out, more information
proposal. Good scholarship is indicated by:
about how to establish an area of focus within
occupational therapy and build a tradition of • Organizing the proposal so that it adequately
research can be found in Kielhofner (2002), and responds to each of the proposal sections,
Kielhofner, Borrell, and Tham (2002). Before sub- • Weaving a comprehensive and up-to-date litera-
mitting any application for research, Gitlin and ture review into various sections of the proposal;
Lyons (2004) recommend that conducting a self- • Including appropriate citations of prior, high-
evaluation to ensure one is ready to assume the quality research in the area,
role of principal investigator. The points that have • Providing a well-reasoned and well-justified
been covered in this section can be used as a guide argument or rationale for the central aims and
to this kind of self-evaluation. hypotheses,
One caveat is that it is important to recognize • Presenting a meticulous and well-written docu-
that the criteria used to evaluate an investigator’s ment, and
expertise are not uniformly applied within and • Obtaining mentorship, good advice, and peer
across granting agencies. Some granting agencies reviews in advance of formal submission.
issue classes of grant awards that are designed Because good scholarship is central to success-
specifically to allow a new investigator or a clini- ful grant writing, the following section extends
cian seeking to transition into a research role to each of these points.
develop his or her research skills in a given area.
One contradiction to many of the criteria outlined Organizing the Proposal So that
in this section exists in some of the K-award fund- It Adequately Responds to Each
ing (described in Table 30.3 later in this chapter) of the Proposal Sections
that is offered through the National Institutes of
Health (NIH). As described earlier, the very pur- One of the most basic aspects of grant writing that
pose of the K-awards is to support the professional differentiates it from other forms of academic writ-
development of researchers and prospective ing is that administrators and reviewers usually
researchers at varied points within their research demand that the proposal follow a highly struc-
careers. The Mentored Research Scientist Develop- tured and organized format that is presented in
ment Award (K01) is one example of a K-award the application instructions package. In addition
that allows for newer researchers in a given area to providing a highly organized structure for the
to receive funding for their research. The K01 proposal, some funding agencies assist applicants
even further by asking them to organize the pro- describe the nature, course, prevalence, inci-
posal in sections that perfectly mirror each of the dence, and long-term impact of a given condition
criteria by which the proposal will be evaluated. on functioning.
Some agencies even provide questions that frame • Provide evidence of the potential impact of the
each section of the proposal to which the applicant study. This may be accomplished by citing estab-
is asked to respond. Most grant applications must lished unknowns or contradictions within the lit-
contain many or all of the following sections erature that need resolution.
(Gitlin & Lyons, 2004): • Provide evidence of the reliability, validity, and
feasibility of the proposed study methods. This
• Title, may be accomplished by citing studies that sup-
• Abstract, port an applicant’s plans and provisions for
• Introduction (including a literature review that recruiting and retaining an adequate number of
reflects the background and significance of the subjects, by citing studies that have utilized sim-
problem, potential impact and feasibility of the ilar methods of data collection, and by citing
study, theoretical foundation for the study, and studies that attest to the reliability and validity of
general importance and relevance of the study to the measures, data collection methods, and statis-
the scientific topic area), tical analyses to be used.
• Specific aims (i.e., goals or objectives), • Describe, explain, and provide evidence for the
• Methods (including a research, evaluation, dis- chosen theoretical foundation for the study.
semination, and/or educational plan),
• Timeline and management plan (delineating In sum, the literature review should reflect both
roles and responsibilities of each research team wide-ranging and highly specialized knowledge
member and timeframes in which the work will about the topic area proposed for study. Grant
be expected), proposals are typically criticized if the back-
• Biographical information (i.e., biographical ground information, theoretical ideas presented,
sketches and/or curriculum vitae for each mem- and rationale for the study are not well supported
ber of the research team illustrating credentials, byan abundance of accurate citations of prior
level of expertise, and capacity to carry out the studies.
study), For proposals that are written in highly com-
• A summary of the resources and qualifications of plex or controversial areas, it is not sufficient to
the applicant’s institution, cite studies that support only one side of a scien-
• A budget and budget narrative (justifying antici- tific argument. In most cases it is essential to cite
pated costs associated with the study), representatives of both sides of the argument, pro-
• References (mirroring the citations provided in vide an accurate and respectful summary of the
the text), and work on each side, and then justify why one plans
• Appendices (containing consent forms, meas- to take one side over another or explain how one’s
ures, treatment manuals and more detailed study work will attempt to resolve the controversy.
protocols, fidelity rating scales, etc.).
Including Appropriate Citations
Good scholars ensure that each of these sec-
tions is equally strong in terms of content and pres- Keeping updated and being knowledgeable about
entation. the work of important leaders and scientist-peers in
one’s area of research is an ongoing but important
process. The literature review must include not
Weaving a Comprehensive and only broad-based and highly specific studies that
Up-to-Date Literature Review into justify the problem and explain the study approach,
Various Sections of the Proposal but also the most updated and cutting-edge work of
key scientist-peers working within one’s area. In
Within any scientific tradition, it is critical to con-
some cases, important, not-yet-published prelimi-
vey knowledge of the empirical findings that form
nary findings from researchers willing to share
the background of one’s decision to develop and/or
their work privately can and should be included in
test a given concept, assessment, or intervention. A
the literature review to reflect the applicant’s
literature review should be utilized to accomplish
knowledge of evolving findings within the area
the following objectives:
closest to his or her field of study. For example,
• Establish the need for the project and the signifi- when applying for a successful grant that aimed to
cance of the problem to be addressed. For exam- estimate the rates of nonrecovery from acute infec-
ple, epidemiological research may be cited to tious mononucleosis in adolescents, the first author
included not-yet-published findings of adult rates seek reviews or opinions from mentors or
of nonrecovery from a scientist-leader within the respected peers before formally submitting the
same field of study. proposal for review. The process of advice-seeking
Findings presented by other scientists at recent and opinion-gathering should be initiated in the
conferences or scientific meetings may also be early stages of idea formulation and sustained
used to support or provide background for an appli- throughout the writing process.
cant’s proposed work. In some cases, personal
communications regarding key methodological Conducting Pilot Research
issues, study feasibility issues, or other evidence of
Many funding agencies require an investigative
communication with leaders working in the same
team to have conducted preliminary research stud-
topical area is regarded positively (though cited
ies or pilot research that provides evidence for the
work is always best). Having knowledge about the
feasibility and likelihood of success of the pro-
evolving and cutting-edge work in one’s area
posed research. The extent of pilot research or
demonstrates that an applicant is careful to remain
preliminary studies necessary depends on the
absolutely current.
requirements of the funding agency, the size of
the proposed study, the research question, and the
Providing a Well-Reasoned and extent to which the collection of pilot data is eco-
Well-Justified Argument or Rationale nomically and logistically feasible in the absence
for the Central Aims and Hypotheses of grant funding. Pilot research is traditionally
Another critical step in demonstrating good schol- defined as a trial application of some, many, or all
arship involves ensuring that one’s proposal builds of the methods that a researcher plans to utilize in
a logical justification for the central aims and a larger, anticipated study using a smaller sample
hypotheses of the study. This rationale and justifi- size. It can also include collection of data that
cation must be articulated clearly and concretely in helps demonstrate the need or value of the pro-
the proposal so that the reviewers are able to view posed project.
the study as relevant and important to the field and Thus, in grant writing, pilot research may be
link the background literature review to the aims used to:
and methods of the study. Building a rationale for • Provide evidence of need for the study or inter-
the study may, for example, involve an explicit vention,
description of gaps or questions within the existing • Identify unanticipated logistical roadblocks in
knowledge base that the study seeks to answer. data collection,
• Test aspects of the reliability or validity of
Presenting a Meticulous and administering a given measure with a new popu-
Well-Written Document lation,
Each application cycle, funding agency adminis- • Assess the feasibility of planned strategies for
trators scan hundreds of proposals to determine subject recruitment and retention, and
whether they should be accepted for review. • Determine the likelihood of finding anticipated
Subsequently, reviewers may be assigned to read results in the larger study.
or scan up to 20 to 30 proposals per meeting. For
these reasons, it is important to write one’s grant Identifying Specific Aims
proposal in a clear, well-organized, and meticulous
manner. Applicants must also ensure that their All grant proposals require applicants to identify
spelling is correct and that there are no careless specific aims. Specific aims are brief statements
typographical errors, formatting mistakes, or con- that accurately reflect the central outcomes that
fusing and half-written sentences in the proposal. will result from the study (or program). Specific
All of these errors can and do reflect poorly on the aims should cohere with any hypotheses or ques-
overall presentation of the proposal. tions that the application seeks to answer. Often,
specific aims are the first items within the proposal
Obtaining Mentorship, Good Advice, that administrators and reviewers read to get an
and Peer Reviews in Advance of overall sense of the direction and contents of the
study. As such, the content and way in which spe-
Formal Submission
cific aims are worded deserves careful attention
Even if an applicant thinks he or she has made all and often a great deal of rewriting.
possible provisions to ensure good scholarship in Generally, a single grant proposal contains
preparing a grant proposal, it is always wise to between two and five specific aims. Some studies
also have secondary aims. Specific aims should one’s home institution and the funding agency will
contain as much action-oriented language as possi- be followed. All investigators are required to com-
ble. They should be realistic and not overly ambi- plete an Institutional Review Board (IRB) applica-
tious. They should be written clearly and tion to ensure that all ethical issues have been
concisely, and they should convey the expected considered to protect to the fullest extent possible
outcomes of the study. the rights of participants.
In selecting designs and methodologies for
Developing Hypotheses grant-funded research studies, applicants must bal-
ance the demands of methodological rigor with the
In tandem with the specific aims, the central
necessity to treat research subjects in an ethical
hypotheses frame a research study and convey
manner and protect their rights to confidentiality.
anticipated outcomes. They should be supported
For example, Taylor (2004) conducted a ran-
by theory and by the preceding literature review.
domized clinical trial that examined the effective-
As mentioned in the preceding section, hypotheses
ness of a rehabilitation program on quality of life
should correspond with the specific aims. In addi-
using a sample of adults with chronic fatigue syn-
tion, hypotheses should be written in such a way
drome. Half of the sample (the treatment group)
that they may easily incorporate or reflect the
was assigned to receive the program immediately
study design and/or statistical approach that will
following recruitment and the other half (delayed-
be utilized to analyze the expected outcomes.
treatment controls) was assigned to receive the
program 1 year later. In a traditional randomized
Choosing an Appropriate clinical trial, the investigator would not inform
and Rigorous Design participants of their group assignment because
Funding agencies and review groups vary widely expectancy effects might confound study findings.
in terms of what they consider to be appropriate Specifically, delayed treatment controls would not
and rigorous designs for research. Agencies that be told they would be receiving the program 1 year
fund basic science studies and clinical research later because their knowledge that they would
tend to value traditional experimental and quasi- eventually receive the program might bias their
experimental research designs, such as random- responses before, during, and after the program.
ized controlled studies, epidemiological research, However, when the investigator developed
and prospective follow-up studies that utilize re- the study in consultation with the ethics board it
peated measures designs. was determined that all participants should be
Agencies that fund research on healthcare serv- informed of their group assignment so that those in
ices and quality, innovative program development the delayed treatment control group knew that they
and program evaluation studies, and other forms of would eventually be receiving the treatment. This
community-based research have a broader vision approach triggered criticism about the violation of
of what is considered to be an appropriate and rig- traditional randomization when the investigator
orous design for a research study. In any case, it is submitted the findings from the study for publica-
important to ensure that the design chosen matches tion. However, in this case ethics demanded a less
the central aims of the study, the resources (budget rigorous design in which all participants were
requested), and the sample size, and is likely to informed of their group assignment.
produce the expected data. When applicants are faced with ethical dilem-
For example, participatory approaches to mas in writing grant proposals, it is advisable to
research and approaches that are descriptive have consult with ethics board representatives from the
become widely utilized in community-based funding agency and from one’s home institution
research. Qualitative research methodologies have (since both must approve the proposal before it is
also been incorporated. Over the past decade, funded and conducted). In situations that are
occupational therapists and other medical and ambiguous or debatable, applicants should make it
rehabilitation scientists have witnessed the incor- explicit within the grant proposal that alternative
poration of more of these designs into medical and designs or methodologies were considered, and
rehabilitation research. then justify why one approach was chosen over
another. When complex situations arise, being
Ensuring an Ethical Design explicit about ethical dilemmas demonstrates that
an applicant has been thoughtful about these issues
and Methodological Approach
and opens the door for reviewers to support the
A fundamental aspect of grant writing involves selected approach or recommend alternatives.
ensuring that the ethical guidelines established by More detailed information and guidance about eth-
ical considerations in occupational therapy As in this case, a grant proposal should

research is provided in Chapter 29 of this text. always:
• Make any study implementation dilemma
Addressing Logistical Issues and explicit within the application and weigh the pros
Obstacles in Data Collection and cons of each approach or indicate why a
In addition to dilemmas involving competing ethi- more rigorous approach is not feasible, and
cal and methodological considerations, grant • Provide a rationale and justification for why the
applicants often encounter dilemmas involving chosen approach was selected.
logistical and implementation issues. Expert
reviewers are well aware that data collection that Planning Analyses
seeks to respond to a single research question can Most applications for research-related grants
be approached from multiple methodological per- require investigators to specify how they plan to
spectives and it can be completed in a wide range analyze the results and to describe in detail the sta-
of settings. In applications that involve complex tistical or qualitative methods that will be used.
research questions that have the potential to be The data analysis plan is typically included in the
approached using a variety of different measures, methods section. In the case of quantitative studies,
methods, and/or statistical approaches to analysis, this section of the grant application is usually writ-
reviewers often look for evidence that an applicant ten by a statistical consultant or co-investigator
is aware of and has considered the entire range of with a high proficiency in mathematics and statisti-
choices. cal methods. In many cases, it is expected that this
For example, in a proposed study that aimed to plan will be accompanied by a power analysis. A
examine outcomes of an occupational therapy pro- power analysis (see Chapter 17) is typically com-
gram for persons with human immunodeficiency pleted by a statistical consultant or by an investiga-
virus who were living in residential facilities, tor that is experienced in the use of statistics. It is
the investigators first considered a randomized used to demonstrate that the proposed sample size
clinical trial since it was ideally the most rigorous will be large enough to detect any significant
design for studying outcomes of an intervention. effects given the proposed experimental design.
However, in the grant proposal the investigators, For research studies that involve hypothesis
Kielhofner and Braveman (2001) made the follow- testing, it is essential that all of the proposed sta-
ing argument: tistical approaches represent an accurate and
Designing this study required us to deal appropriate way to test the study hypotheses. In
with the following logistic constraints. addition, each proposed statistical approach should
Within a given facility, it would not be fea- correspond with each hypothesis listed. Each
sible to assign residents to different condi- planned statistical approach should be described
tions. The model program will result in very clearly in lay terms in as much detail as pos-
changes in the milieu of the facilities and sible. Any plans for the treatment of missing data
cannot be implemented without contamina- and plans for troubleshooting other unanticipated
tion of control subjects in the same facility. complexities within the proposed dataset should
Furthermore, once the intervention starts, also be accounted for in this section.
it would be impossible for the facility to Qualitative data analysis plans should be as
return to the control condition, for similar complete as possible. However, if some of the
reasons. This rules out both a conventional analysis will depend on the unfolding research
randomized design at the level of the client process, it should be specified how decisions about
and an interrupted times series design at analysis will be made. If software for analysis is to
the level of the facility. be used, it should be noted. The analysis should
clearly reflect the study question, be embedded in
The investigators then went on to propose their the specific qualitative approach to analysis, and
nonrandom control group design, describing statis- reflect thorough efforts to maintain trustworthiness
tical techniques that would be used to attenuate of findings.
the effects of identified initial differences in the
experimental and control groups. Since reviewers Developing a Timeline
understood that the design proposed was the most
and Evaluation Plan
rigorous design that could be implemented, given
the circumstances of the study context, the grant Most grant applications require a specific timeline
was funded. of activities and when those activities are to be
Examples of Funded Projects
Occupational therapy researchers throughout the grams offered to children newborn to 7 years old
world are increasingly receiving funding for that present with global developmental delays in
research. This section contains examples of three Rehabilitation centers in the province of Québec.
funded research grants obtained by occupational This provincial study has been funded by the
therapy researchers in Scotland, Canada, and the Consortium National de Recherche sur
United States, respectively. They illustrate a range l’Intégration Sociale (CNRIS).
of funding situations including a fellowship, inter- The CNRIS is a consensus-building organiza-
disciplinary research, and research designed to test tion whose goal is to energize and foster the devel-
an occupational therapy intervention. opment of research in the fields of intellectual
disability and pervasive developmental disorders.
Developing a Research Career in the United Inspired by Quebec approaches to integration and
Kingdom Through Fellowship Funding social participation, the Consortium pursues the
following main objectives:
After being awarded his PhD, an occupational
therapist in the United Kingdom was looking for • Foster networking amongst researchers and part-
the next step in his research career. He was able to ners, as well as the development of research proj-
obtain a postdoctoral research Fellowship specifi- ects and programs in the areas targeted by its
cally designed for nurses, midwives, and allied mission,
health professionals in Scotland. The Fellowship, • Support orientations that deal with important
which is funded jointly by the Scottish National social issues for users, parents, caregivers, and
Health Service (Education for Scotland), the service managers,
Scottish Executive Health Department, and the • Foster the emergence of research that encourages
Health Foundation, provides an award of £65,000 or brings other sectors to invest in the develop-
for 2 years. It is designed to provide the highest ment, adaptation, and improvement of services to
possible quality of experience. persons with an intellectual disability or a perva-
The Fellowship has three components: sive developmental disorder,
• Encourage the dissemination of research results
1. Research development, in various formats, and
2. Research training, and • Collaborate with health and social service organi-
3. Leadership. zations in developing the ability to use research
The award includes startup funds (£10,000) to results in designing intervention programs and
cover setting up costs and the cost of pilot research management methods.
studies. The aim of the fellowship is to provide the The overarching goal of this provincial study is
next generation of principal investigators, to develop knowledge about the current state of
strengthen existing research collaborations, early intervention programs offered to children
develop new ones, and allow focused research in presenting a global developmental delay without a
Scotland to develop rapidly. diagnosis of pervasive developmental disorder or
The application process was rigorous and com- autism age newborn to 7 years old. First, the study
petitive. The first stage entailed a detailed applica- will explore the literature specific to the clientele
tion and research proposal, developed in and early intervention programs in Québec, but
collaboration with an identified research unit also throughout North America and around the
where the Fellowship was proposed to occur. world. This first step will paint the portrait of
Following short listing, the applicant was informed the situation from the point of view of the scien-
that there would be an interview and a panel pres- tific community with regard to the object of the
entation of eminent academics from a range of col- study. Then, a vast process of data collection will
laborating Universities in Scotland. take place throughout the province of Québec in
Expected outcomes from the fellowship revolve the different rehabilitation centers for children
around developing the applicant’s capacity and with intellectual disabilities. Healthcare profes-
capability as a lead occupational therapy sionals and other professionals involved with chil-
researcher. In real terms, success will be judged by dren ages from newborn to 7 years old will be
the number of publications and successful grants met via focus groups to document in details the
that he manages to secure during the 2-year period. reality as it is in these rehabilitation centers.
The second phase of the study will provide an
A Canadian Grant in Progress: The Effects of empirical and detailed view of the situation. The
Early Intervention Programs for Children with results of these two phases will be compared and
Intellectual Disabilities challenged to identify best practices and put
This multidisciplinary grant includes a Canadian forward promising trends in early intervention
occupational therapist. The grant evaluates pro- programs.
(continued)
505
Examples of Funded Projects (continued)
The team of researchers implementing this trol group comprised of individuals who will
study is made up of occupational therapists, psy- receive basic educational services. Participatory
chologists, and educators coming from three research methods are also being used to identify
different universities located in the province of potential obstacles and solutions to program imple-
Québec. It is a 2-year project funded for mentation and efficacy, and to evaluate how the
CAN$100,000. services impact upon and are viewed by the
clients. Over the 3 years, the project, which is
An American Study in Progress: Enabling Self funded for U.S. $450,000.00, will:
Determination of People with AIDS • Implement and rigorously test an innovative
model program designed to enable persons with
A project funded by the National Institute of AIDS to live and participate in the community
Disability and Rehabilitation Research is studying and to become employed,
an occupational therapy intervention for people
• Generate knowledge to refine the model program
with AIDS. In partnership with four supportive liv-
and describe best strategies for its implementa-
ing facilities in Chicago, the investigators are
tion in supportive living contexts,
implementing and studying an innovative model
• Build capacity of four supportive living facilities
program of services based on the Model of Human
to sustain this program, and
Occupation and concepts from disability studies
• Actively disseminate the model and evidence
(Kielhofner & Braveman, 2001). The study design
about its impact.
will compare outcomes of the program with a con-
implemented, a detailed evaluation plan and a outputs and outcomes (see Chapter 37 for a
description of performance indicators. Proposals description of logic models). The logic model
ordinarily include: gives the researcher an overview of the outcomes
of the project and also of the process of its imple-
• A detailed listing of all grant activities (e.g., from
mentation (or realization). Furthermore, it pro-
recruiting participants and sites to data collec-
vides a framework that can serve as a reference
tion, data analysis and dissemination, etc.),
point for the researcher to go back to and make
• When the specified activities will take place,
sure that each step is achieved or that the proper
• Who will perform them, and
adjustments are made.
• The criteria by which they will be judged suc-
cessfully competed (which is part of the evalua-
tion plan as discussed below). Developing a Reasonable Budget Request
Grant reviewers appreciate seeing a table with An important feature of any grant application is
these elements included across the duration of the the budget request and budget narrative; together
grant. they justify the applicant’s request for a certain
The evaluation plan ordinarily must include amount of funding in each budget category. The
two separate levels of evaluation: budget narrative also provides the reviewers with a
rough overview of the applicant’s thinking about
• A formative evaluation of all the activities pro-
the logistical, timing, and implementation aspects
posed in the timeline (according to criteria
of the study. In addition, the construction of the
included for successful performance as noted
budget allows for the principal investigator and
earlier), and
research team to anticipate and think through the
• A summative evaluation of the impact of the
personnel-related managerial and contractual
research study, or training activity that the grant
aspects of the study.
application has proposed.
Creating a budget upfront allows the team to
Summative evaluation of the overall project anticipate the amount of salaried release time from
usually involves specifying how one will deter- other duties that will be required by each team
mine the extent to which the aims and objectives member to implement the grant. In addition, it
proposed have been achieved and how they will be allows for subcontract agreements to be negotiated
measured. between the principal investigator’s sponsoring
Many funders now require a project logic (home) institution and any other institutions that
model to guide the evaluation process; it provides house the co-investigators in which certain study
a visual representation of project goals, inputs, duties will be carried out.
Most funding agencies provide detailed instruc- might define direct costs as the sum of all of the
tions regarding the level of detail and budgetary above categories excluding the costs of equip-
planning that is required for documentation in an ment and training stipends for students.)
application. For example, some budget narratives • Indirect costs: Indirect costs cover basic infra-
can be so detailed that they specify the estimated structure and operational costs involved in run-
number of study-related phone calls that will be ning a research study, such as office space,
made by each member of the research team, the electricity, heating, and air conditioning.
estimated length of each call, and the estimated • Total budget request: This is the sum total of the
charge for each call. Other agencies and reviewers direct and indirect costs across all years of the
may accept a more loosely written budget justifi- study.
cation provided that the applicant is requesting
Each funding agency has a different formula
what the reviewers and administrators consider to
that is applied in the calculation of direct versus
be a reasonable amount for each category. When
indirect costs. Many agencies put a cap on the per-
possible, it always behooves an applicant to seek
centage of indirect cost funding that a sponsoring
accounting consultation and assistance from an
(home) institution can request in proportion to the
individual experienced in the assembly of a grant
total direct costs of a study. Some agencies do not
budget.
allow a sponsoring institution to request any indi-
As outlined in an earlier section of this chapter
rect costs because they expect an institution to pro-
(i.e., selecting an appropriate funding agency),
vide the basic infrastructure support to operate the
funding agencies are highly diverse in terms of the
study. Under some circumstances the percentage
amount of money they are willing to provide to
of indirect costs that an institution is permitted to
support a single study or research group. The total
extract from a grant is negotiated between the
amount for a grant award can range from $500 to
funding agency and the sponsoring institution.
well over $1,000,000 to support a single study.
Despite this diversity, one characteristic that fund-
ing agencies have in common is that they all limit Obtaining Letters of Support
what they are willing to provide and they all have Many funding agencies require letters of support
regulations on how money can be spent in a to be appended to the grant application. Letters of
given category. Thus, it is always wise to consult support are formal testimonials that describe a col-
with a program officer if there are any questions laborator’s level of experience working in the
about how much money can be requested in a sin- research area, overall enthusiasm about the idea
gle category or if there are other ambiguities and/or methodological approach of the study, and
regarding the budget in the application instruc- planned role or contribution to the study. Letters of
tions. Generally, grant budgets for health-related support are obtained from collaborators who
research may be broken down into the following intend to participate directly as members of the
categories: research team; from consumers who intend to
serve in advisory capacities; and from collaborat-
• Personnel and fringe benefits,
ing sites such as community organizations, prac-
• Tuition waivers and training stipends for stu-
tice sites, or other individuals who intend to
dents,
support the study in a more peripheral way (e.g.,
• Consultants,
practitioners who have agreed to refer their
• Travel,
patients to participate as subjects in the study).
• Equipment,
Letters of support are typically obtained from co-
• Supplies,
investigators, consultants, subcontractors, referral
• Inpatient or outpatient costs,
sources, and advisory board members.
• Subcontractual costs,
• Construction costs, and
• Other costs.
Determining Where to Send the
Grant: Ensuring Appropriate Review
The costs for each category are generally added
One of the most important steps in submitting a
together in different combinations using a formula
grant for review involves determining where and to
that subdivides the total into the following three
whom to send the application. Each agency has its
categories:
own unique policies and procedures for receiving
• Total direct costs: This is defined by the sum of grants and assigning them for review. It is always
all or a certain combination of the categories best to check with a reliable representative from an
listed above. (For example, a granting agency agency (usually a program officer or higher admin-
istrator) for first-hand information about the grant Because the review procedures for U.S. federal
submission process. granting agencies tend to be more complex than
those for other granting agencies, this section
After the Grant has focuses on describing the review procedures for
federal agencies (i.e., NIH, AHRQ, and the U.S.
been Submitted Department of Education). Private industry and
foundations use many of the evaluation criteria and
The following section explains the critical review procedures that are employed by federal
processes and interactions that occur within the agencies. Thus, the general ideas provided herein
funding agency and between the program officers should be somewhat transferable for applications
and the principal investigator after a grant has been to other funding sources. However, it is important
submitted. It is important for investigators to have to keep in mind that all agencies will differ
as much information about these processes as pos- between and within themselves in terms of the
sible so that they better understand the details that evaluation criteria they designate as most impor-
contribute to a funding agency’s decision to make tant for an investigator to address for any given
a grant award. The three critical processes that will type of grant. Following application instructions
be discussed in this section include: to the letter and tailoring the proposal to each
and every aspect of the evaluation criteria is
• Review process, the most critical aspect of preparing any grant
• Grant scores and funding decisions, and proposal.
• Feedback and resubmission. Many funding agencies adhere to a peer review
process in which reviewers are selected based upon
Understanding the Review Process
their expertise in a scientific area that matches that
For most funding agencies and competitive grant of the grant proposals being reviewed. Peer
applications, the outcome of the review process is reviewers are typically selected based on a history
the most critical determinant of whether an inves- of exceptional scholarship and achievement in the
tigator’s grant proposal will receive funding. In given area. The designation of peer reviewer does
many agencies, the review process can take from not mean that the investigator knows or works with
several months to nearly a year to complete. the reviewer. In fact, many agencies have strict
During the review process, one or more individu- conflict-of-interest regulations against dual rela-
als serving as representatives for the funding tionships or research collaborations between peer
agency carefully scrutinize each application to reviewers and applicants.
determine whether it meets a set of prespecified Usually, at least three reviewers are designated
criteria for funding. Agencies differ in terms of to provide a detailed review of a grant application.
these criteria. To a certain extent, these differences These reviewers are sometimes referred to as first,
depend upon the overall mission, values, or objec- second, and third reviewers, for example. For some
tives of the funding source. Knowing as much as agencies, an additional group of as many as 20
possible about the review process, evaluation crite- other reviewers that comprise a review panel may
ria, and the individuals serving as reviewers is be asked to score and give input on a single appli-
important in preparing as competitive an applica- cation. Often, this larger group of reviewers will
tion as possible (Gitlin & Lyons, 2004). skim the applications and base their scores on the
Agencies differ in terms of the number of grant reports given by the primary review group and on
applications they receive and in terms of the num- the contents of the discussion that followed those
bers and kinds of individuals that are assigned to reports.
review a given application. For example, the U.S. Some funding agencies assemble review panels
Public Health Service, which includes the National that consist of both peer (professional) reviewers
Institutes of Health (NIH) and the Agency for and lay reviewers or consumers. Lay reviewers
Healthcare Research and Quality (AHRQ), may be members of the same community or popu-
receives and reviews approximately 40,000 grant lation from which participants in the research proj-
applications per year. Because few proposals rela- ect will be drawn (e.g., individuals with chronic
tive to this overall number are actually funded, fatigue syndrome from a wide range of work or
the application process is highly competitive. professional backgrounds). Alternatively, they
Similarly, certain offices within the U.S. Depart- may be individuals who represent the voice of an
ment of Education have been known to fund only even broader group of individuals of which the
the top 4% of discretionary grant applications for prospective participants may be a part (e.g., indi-
certain competitions. viduals with disabilities). Lay reviewers will
inevitably read the application through different Interpreting Grant Scores in Light
lenses than peer reviewers. It is important to of Funding Decisions
know upfront whether a lay reviewer will be
reviewing the grant so that one will know whether Obtaining grant funding from most agencies is a
his or her language and writing style should be tai- highly competitive process. Agencies vary widely
lored to a broader audience. Lay reviewers are usu- in their approaches to scoring grant applications.
ally charged with the same responsibilities and For example, the U.S. Department of Education
given as much power in the vote as peer reviewers scores applications such that high scores are given
regarding whether a given application should be to the strongest applications and low scores to the
funded. weakest. Conversely, NIH scores applications such
If a grant received a fundable score as a result that low scores are given to the strongest applica-
of the peer review process, some granting agencies tions and high scores are given to the weakest.
employ a second tier of reviewers that are housed Applications that score above the 50th percentile
within the agency or closely linked to the agency based on preliminary review are generally not for-
(e.g., a board of directors) to make the final fund- warded for formal review and are not scored.
ing decision. For example, within NIH the Within NIH, this process is called streamlining
National Advisory Council functions as an over- because it is more time efficient and it facilitates
sight board that consists of scientists and adminis- the more detailed review of the stronger applica-
trators. This board reviews each highly scored tions. Even though a review panel may assign a
grant that has been recommended for funding to potentially fundable score to a grant application, it
ensure that it provides adequate provisions for the is not a guarantee that the application will be
protection of human subjects and that it is consis- funded.
tent with the overall policies, values, and vision of In conjunction with the requirement for addi-
NIH. Similarly, the U.S. Department of Education tional evaluations of the overall relevance of the
has what is called a grants team that conducts an highly scored proposal to agency values and prior-
internal review of each highly scored grant to ities, many agencies have cutoff points (often rep-
ensure that the reviewer’s scoring sheets are cor- resented by percentile rankings) that determine
rectly completed and to verify that the application which of the highly scored applications will be
meets all of the requirements of the program. funded and which will not. For example, one
Criteria by which a grant application is judged agency may fund only the top 2% of applications
vary widely. In many cases, evaluation criteria for a given competition, whereas another agency
are provided in instructional format along with the may fund the top 15%. These cutoff points fluctu-
initial grant application package. Some agencies ate depending upon accounting formulas that are
have a rather rigid set of criteria by which an appli- developed by each agency. These formulas typi-
cation is judged, whereas other agencies only cally incorporate the number of applications
offer general guidelines or do not offer much detail received for each funding cycle and the amount of
in the way of evaluation criteria. Examples of money available from cycle to cycle. Many pro-
evaluation criteria set forth by the National gram officers are willing to provide investigators
Institutes of Health and the U.S. Department of with information regarding percentile funding cut-
Education are provided in Tables 30.3 and 30.4, offs once it becomes available.
respectively.
The evaluation criteria presented in Tables 30.3 Evaluating Feedback and Determining
and 30.4 are simply examples and they are not Whether to Resubmit the Proposal
entirely comprehensive or exact. Even within a
given agency, the evaluation criteria may change After having worked numerous hours to write and
depending on the nature of the competition and the assemble a grant proposal, receiving critical or
type of grant for which one is applying. Knowing negative feedback is a challenging process for
and continually evaluating one’s grant proposal even the most experienced grant writer. When a
against the published criteria for a given competi- grant application is not funded, it is not only diffi-
tion throughout the planning and writing process is cult to read feedback from reviewers, but at times
fundamental to increasing the application’s com- it is also difficult to interpret it and decide whether
petitiveness. Just as evaluation criteria vary from others consider the proposal worthy of revision
agency to agency, so do approaches to scoring a and resubmission. Some agencies, such as the U.S.
grant proposal. More information about how appli- Department of Education and NIH, make the
cations are scored and how scores are typically process of interpreting feedback somewhat easier
interpreted is provided later in this chapter. by using a two-tier system. These agencies score
Table 30.3 Typical NIH Evaluation Criterion and Questions for Competing Research Applications*
Criterion Types of Questions for Reviewers
Significance • Is the scientific problem that the application addresses important?
• Are the outcomes of the study likely to have a significant impact on existing scientific
knowledge in this area?
• How will this application advance existing knowledge, theoretical concepts, treatment
approaches, or methodologies in this area?
Approach • Is the study based on an overarching theory or conceptual framework?
• Are the theory, design, methods, and statistical analyses cohesive and well developed?
• Do they adequately reflect the specific aims and hypotheses of the study?
• Is the proposed approach feasible and methodologically rigorous?
• Does the investigator anticipate possible pitfalls or problems with the approach and
does he or she provide alternative ways of addressing those problems should they
occur?
Innovation • Does the study introduce new theoretical concepts, approaches, or methodologies?
• How does the project challenge existing paradigms or seek to revise or reformulate
existing treatments or methodologies?
Investigator • Are the investigator and his or her research team sufficiently knowledgeable,
experienced, and adequately trained to carry out the work of the proposed project?
Environment • Does the scientific environment in which the study will take place provide adequate
resources and increase the likelihood that the study will be successful?
• Does the project involve useful and relevant collaborations between agencies or
organizations?
• Does the study take advantage of unique resources or equipment within the
investigator ’s home institution?
Inclusions, budget, • Are the plans to include both men and women in the project adequate?
and protections • Are there adequate provisions for the recruitment and retention of individuals from
minority groups?
• Is the proposed budget reasonable given the amount of professional effort put forth,
logistical requirements, methods, and length of the proposed study?
• Are there adequate provisions in place for the protection of subjects participating in the
research?
*Based on evaluation guidelines for reviewers provided at http://grants.nih.gov/grants/guide/notice-files/

NOT-OD-05-002.html.
and comprehensively evaluate only the stronger Proposals worthy of revision and resubmission
applications. Written feedback from the reviewers include those that are given scores near the per-
is forwarded only to investigators who achieved centile cutoff for funding. In many cases, feedback
the higher scores. If an applicant’s proposal is will indicate that the reviewers were enthusiastic
rejected but he or she receives written feedback about the application and would encourage revi-
from an agency, it indicates that the reviewers con- sion and resubmission. Sometimes reviewers will
sidered the application worthy enough of a provide the applicant with questions to answer in
detailed discussion regarding its merits and weak- the revised proposal or suggestions about how to
nesses. address the concerns that have been raised. The
In many cases, federal agencies consider appli- program officer can often speak to the general
cations that are streamlined (or not scored) as not level of enthusiasm about an application and can
salvageable because they are limited by major assist an applicant in deciding whether to revise
flaws. Such flaws may include ideas that lack sig- and resubmit the grant application.
nificance or do not reflect the objectives of the
funding agency, flaws in the design and methods,
absence of an adequate theoretical basis, confusing Conclusion
or inconsistent aims and hypotheses, poor overall
scholarship, and ethical or logistic problems This chapter provided an orientation to the main
(Gitlin & Lyons, 2004). kinds of funding opportunities for occupational
Table 30.4 Examples* of U.S. Department of Education General Evaluation

Criteria for Grant Reviews
Criterion Questions
Need for project • Is the problem to be addressed of sufficient magnitude or severity?
• How does the proposed project meet the need for services, identify gaps or
weaknesses in existing services, and address those gaps?
• How will the proposed project prepare personnel for fields in which shortages have
been demonstrated?
Significance • What is the national significance and likely impact of the proposed project in terms of
improving employment?
• How significant is the problem to be addressed by the project?
• What is the potential contribution of the project to increased knowledge or
understanding of rehabilitation or educational problems, issues, or effective strategies?
• What is the likelihood that the project will result in system change or improvement?
• How likely is the project to contribute to the development and advancement of theory,
knowledge, and practices in the field of study?
• How replicable will the program be in a variety of settings and how generalizable will
the findings be?
• To what extent will the proposed project yield findings or products that will be utilized
by other agencies or organizations.
• How likely will the proposed project build local capacity to provide, improve, or expand
services?
• Will the results be disseminated in ways that will enable others to use the information
or strategies?
Quality of the • To what extent are the goals, objectives and predicted outcomes clearly specified and
project design measurable?
• Is the project based upon a specific and rigorous research design?
• Does the design reflect up-to-date knowledge from research and effective practice?
• Is the design appropriate to the needs of the target population and is it likely to
address those needs?
• Is there a high-quality conceptual framework underlying the proposed research or
demonstration activities?
• Do the proposed activities add to a coherent and sustained program of research,
training, or development in the field?
• Do they add substantially to an ongoing line of inquiry?
• Is the proposed design accompanied by a thorough, high-quality review of the
literature, a quality plan for research activities/project implementation, and the use of
appropriate theoretical and methodological tools to ensure successful achievement of
the project objectives?
• Will the design lead to replication of project activities or strategies? Are proposed
development efforts accompanied by adequate quality controls and repeated testing of
products?
• Will the project build capacity and yield results that will extend beyond the funding
period?
• Does the proposed project represent an exceptional approach for meeting the priorities
established for the competition and/or the statutory purposes and requirements?
• To what extent will the project be coordinated with related efforts and establish
linkages with appropriate community, state, and federal resources and organizations
providing services to the target population?
• Does the project encourage consumer involvement?
• Are performance feedback and continuous improvement integral to the design of the
project?
• What is the quality of the methodology to be employed in this project?
Quality of • Are there strategies for ensuring equal access and treatment for eligible project
Project services participants that are members of groups that have traditionally been underrepresented
based on race, color, national origin, sex, age, or disability?
• Are the services to be provided by the project appropriate to the needs of the intended
recipients or beneficiaries of those services?
• Do entities that are to be served by any proposed technical assistance project
demonstrate support for the project?
(continued)
511
Table 30.4 Examples* of U.S. Department of Education General Evaluation

Criteria for Grant Reviews (continued)
Criterion Questions
• Do the services to be provided reflect up-to-date knowledge from research and effective
practice?
• What is the likely impact of the services to be provided on the intended recipients?
• To what extent are training or professional development services to be provided by the
proposed project of sufficient quality, intensity, and duration to lead to improvements in
practice among recipients of those services?
• Will the training or professional development services alleviate the personnel shortages
that have been identified?
• Will the project lead to improvements in the academic achievement of students as
measured against rigorous standards?
• Will the project lead to improvements in the skills necessary to gain employment or
build capacity for independent living?
• To what extent will the project involve the collaboration of appropriate partners for
maximizing the effectiveness of services?
• To what extent are the services to be provided focused on those with greatest needs?
Quality of project • Will the investigator encourage applications for project staff positions from persons who
personnel are members of groups that have traditionally been underrepresented based on race,
color, national origin, sex, age, or disability?
• How qualified, trained, and experienced are the investigators, key project personnel,
consultants, and subcontractors?
Adequacy of • Are the facilities, equipment, supplies, and other resources from the applicant
resources organization adequate?
• Has each partner demonstrated commitment to the implementation and success of the
project?
• Is the budget adequate to support the proposed project?
• Are the proposed costs reasonable in relation to the objectives, design, potential
significance and benefit, and number of persons to be served?
• Is there potential for continued support of the project by appropriate entities after federal
funding ends?
• Is there a potential for the incorporation of project purposes, activities, or benefits into
the ongoing program of the agency after the funding period?
Quality of the • Is the management plan adequate to achieve the objectives of the project on time and
management within the budget?
plan • Does the management plan include clearly defined responsibilities, timelines, and
milestones for accomplishing project tasks?
• Are the procedures for ensuring feedback and continuous improvement in project
operations adequate?
• Are the mechanisms for ensuring high-quality products and services from the project
adequate?
• Are time commitments from the investigators and other project personnel adequate to
meet the objectives?
• How will the applicant ensure that a diversity of perspectives are brought to bear in the
operation of the proposed project, including those of parents, teachers, the business
community, other disciplines and consumers?
Quality of the • Are the methods of evaluation thorough, feasible, appropriate to the context within
project which the project operates, and appropriate to the goals, objectives, and outcomes of
evaluation the project?
• Do the methods of evaluation provide for examining the effectiveness of project
implementation strategies?
• Do the methods of evaluation include the use of objective performance measures that
are clearly related to the intended outcomes of the project and will they produce
quantitative and qualitative data to the greatest extent possible?
• Will project evaluation methods provide timeline guidance for quality assurance?
• Will the evaluation provide guidance about effective strategies suitable for replication or
testing in other settings?
*Based on guidelines adapted from evaluation guidelines for reviewers provided by the Education Department
General Administrative Regulations, Part 75, Subpart D at http://www.ed.gov/policy/fund/reg/edgarReg/edlite-
part75d.html. A specific competition will typically use a subset of the types of questions listed on this table.
512
therapy researchers. The authors explained the Kielhofner, G., & Braveman, B. (2001). Enabling self
steps involved in writing and applying for grants determination for people living with AIDS. Department
of Occupational Therapy, University of Illinois at
that fund research studies and other research- Chicago. Grant proposal submitted to and funded by
related programs and initiatives. Twenty steps that the National Institute of Disability and Rehabilitation
the authors considered necessary for the attain- Research, U.S. Department of Education
ment of grant funding were reviewed. The authors (H133G020217-3).
Scottish Executive (2004). Allied Health Professions
explained elements of grant writing that ranged
Research and Development Action Plan. Edinburgh:
from idea development to ensuring an appropriate Scottish Executive.
review once a grant has been submitted. The Taylor, R. R. (2004). Quality of life and symptom severity
authors then described the review and evaluation for individuals with chronic fatigue syndrome: Findings
process and explained how funding decisions are from a randomized clinical trial. American Journal of
Occupational Therapy, 58, 35–43.
made. The chapter also provided information on Taylor, R. R., Fisher, G., & Kielhofner, G. (2005).
how to revise and resubmit a grant proposal that Synthesizing research, education, and practice accord-
was rejected. Explanations of international funding to the scholarship of practice model: Two faculty
ing procedures and mechanisms available in the Examples. Occupational Therapy and Health Care, 19,
107–122.
United Kingdom and Canada were provided and
examples that described the process of obtaining
RESOURCES
different kinds of grant funding from different
funding sources were provided. For Grant Writing
Gitlin, L. N., & Lyons, K. J. (2004). Successful grant writ-
REFERENCES ing: Strategies for health and human service profes-
Balcazar, F., & Suarez-Balcazar, Y. (2004). Center for sionals. New York: Springer.
capacity building for minorities with disabilities Additional resources can be found at (http://.www.
research. The University of Illinois at Chicago, npguides.org/guide/index.html and http://www.cpb.
Department of Disability and Human Development and org/grants/grantwriting.html).
Department of Occupational Therapy. Grant proposal To find a potential match between a private source of
submitted to and funded by the National Institutes of funding and an idea/project:
Disability and Rehabilitation Research. United States
Canadian Occupational Therapy Foundation (2004).
Opportunities for Researchers. Retrieved December 5, • The Foundation Center: A nonprofit information clearing-
2005, from house and library that collects and disseminates informa-
http://www.cotfcanada.org/site_page.asp?pageid⫽672 tion on more than 80,000 private foundations for
Creek, J., & Ilott, I. (2002). Scoping study of occupational organizations and individuals seeking information about
therapy research and development activity in Scotland, grants. It is one of the most widely accessed search
Northern Ireland and Wales. London: College of engines used by research development personnel to
Occupational Therapists. inform investigators of available funding opportunities
Department of Health (2000). Meeting the Challenge: A and competitions. The Foundation Center can be accessed
strategy for the allied health professions. London: at http://www.fdncenter.org.
Department of Health. • Foundations On-Line: A foundation directory and search
Department of Health (2005). Research governance frame- tool that can be accessed at http://www.foundations.org
work for health and social care (2nd ed.). London: • FundingSearch.com: Provides grant funding resource
Department of Health. services for nonprofit organizations, consultants, and pro-
Gillette, N. (2000). A twenty-year history of research fund- posal writers.
ing in occupational therapy. The American Journal of • GrantsWeb: Provides links to grant resources, funding
Occupational Therapy, 54(4), 441–442. opportunities and a grants database.
Gitlin, L. N., & Lyons, K. J. (2004). Successful grant writ- Additional links and resources to granting agencies,
ing: Strategies for health and human service profes- funding announcements, and grant writing tips:
sionals. New York: Springer.
Higher Education Funding Council for England (2001). • http://www.npguides.org/guide/index.html. Nonprofit
Research in nursing and allied health professions: guides is a grant writing resource with tools and links for
Report of the Task Group 3 to HEFCE and the funding for nonprofit organizations.
Department of Health. Bristol: Higher Education • http://www.foundations.org. The Doundation Center is a
Funding Council for England. great resource for finding private and public foundations,
Ilott, I., & White, E. (2001). 2001 College of Occupational the foundation directory, and matching of topics and
Therapists’ research and development strategic vision foundations.
and action plan. British Journal of Occupational • http://fconline.fdncenter.org/ The foundation center online
Therapy, 64(6), 270–274. has a comprehensive directory of private, community, and
Kielhofner, G. (2002). UIC’s scholarship of practice. OT corporate grant makers.
Practice, 7(1), 11–12. United Kingdom
Kielhofner, G., Borell, L., & Tham, K. (2002). Preparing
scholars of practice around the world. OT Practice, In the United Kingdom, information on many funding
7(6), 13–14. opportunities for health-related research can be
obtained from the web-based resource RDInfo Information About the Grant Review Process
(http://www.rdinfo.org.uk). Support for new researchers Most funding agencies will have information on their
is a recent addition to the information provided, while a grant review process posted on their Web sites.
special edition devoted to funding opportunities for Detailed information about the review process within
allied health professionals, produced in 2003, can also NIH can be found at http://grants1.nih.gov/grants/
be downloaded. peer/peer.htm
C H A P T E R 3 1
Securing Samples for Effective Research

Across Research Designs
Anne E. Dickerson
Selecting a Sample The research approach also affects sampling.

For example, a quantitative study of the efficacy of
The research process has clearly defined steps. sensory integration would compare a group of
One of the most critical of these is securing the children randomly assigned to receive sensory
study sample. An adequate, carefully selected sam- integrative treatment to a group randomly assigned
ple, suitable to the research design, is essential to to a control group. In this instance, there would be
rigorous research. Whether undertaking quantita- concern to have a sufficiently large sample size in
tive or qualitative research, there is a process for order to achieve statistical significance. In con-
determining who should participate in the study. trast, a qualitative researcher examining efficacy
Selecting a sample determines who is studied. would be likely to select a small number of chil-
However, it is intimately tied to what is studied. It dren who could be interviewed and observed in
is important to delimit the specified group of indi- great depth. In this instance, the investigator might
viduals who will be studied according to the phe- systematically select children who are better able
nomena, or conceptual idea under investigation. to articulate their experience.
For example, if an investigation is examining the A researcher may address the research ques-
use of a training technique for increased mobility tion using a single-system design as discussed
in clients who experienced a cerebrovascular acci- in Chapter 11. Such an inquiry would require only
dent (CVA), it may be very important to limit the one subject. Alternatively, a narrative history
scope of the research to clients who had either a design could involve an in-depth interview with a
right or left CVA, since training may affect differ- child, members of his/her family, and the therapist
ently depending on the location of the CVA. On providing the SI therapy. As these examples illus-
the other hand, if the research question is on the trate, a third issue that affects sampling is the
effectiveness of the Canadian Occupational Perfor- research design.
mance Measure (Law et al., 1998), CVA clients Finally, the practical considerations such as
with both right and left CVA should be included, access to the populations under interest influence
but persons with expressive aphasia would not be the selection of subjects. In many cases, these
appropriate subjects, since they could not partici- logistic issues have a major impact on a re-
pate in the assessment. searcher’s sampling plan. For example, if a study
focuses on a population that has limited numbers
(e.g., people with amyotrophic lateral sclerosis),
Issues Affecting Selection of Subjects
the sample size will be limited by access. More
The term subject pool refers to those who are iden- commonly, the limitations of budget, time, and
tified as eligible to participate in the study. How a space lead the researcher to limit the sample size
subject pool is selected depends on several issues. or diversity. Often, sampling limitations under-
First, as mentioned previously, the subject pool mine the integrity of a study, jeopardizing its use-
clearly depends on the study question. For exam- fulness. Unfortunately, many studies are ultimately
ple, if a study question asks about the efficacy of not published because the sample size or sampling
sensory integrative (SI) therapy for children, the approach was inadequate to provide the necessary
subject pool would be children who have SI prob- rigor. While practical considerations are necessary,
lems. If the question pertained to SI effectiveness an investigator will need to be prepared to explain
with children who have tactile defensiveness, then and defend the sample and sampling strategy at the
the subject pool would be further narrowed to conclusion of the study. In the end, the researcher
those children with this particular SI problem. has to balance the research question, the approach
Thus, the specific nature of the research question to inquiry, the research design, and pragmatic con-
directly impacts the selection of subjects. siderations, to achieve an optimal sample.
What to Call Individuals Participating found that males had a higher level of self- esteem
in a Study than females. In another example, Kizony and
Katz (2002) found in a study of persons with
What should the individual who participates in stroke that significantly more women than men
the study be called? Traditionally in quantitative scored above the cutoff point of the process scale
research, the term subject is used. This term of the Assessment of Motor and Process Skills. As
denotes the ideal of objectivity and the often pas- these examples illustrate, gender can make a dif-
sive nature of participating in a quantitative ference in occupational performance and volition.
study. On the other hand, qualitative research Relevant to occupational therapy, there is evi-
often stresses the researcher’s joining with those dence that Latinos and African-Americans with
studied in their everyday lives. Thus, the term
disabilities in the United States are more likely to
participant is more typically used to refer to
those whose lives are studied. Informant is also receive fewer comprehensive services, and less
used in qualitative studies to designate the person culturally relevant services compared to white
who gives the researcher valuable information families (Belgrave & Walker, 1991; Moritsugu &
about the phenomena under study. In survey Sue, 1983; Wells & Black, 2000; Zea, Quezada, &
research, the term respondent is most frequently Belgrave, 1994).
used to refer to the individual answering the sur- Decisions about sampling should always, there-
vey. In participatory research where the investi- fore, give careful consideration to issues of gender
gation is designed to affect the lives of those and race/ethnicity. Including diverse samples and
under study, the term stakeholder is common. analyzing data within these subcategories can also
Each term reflects a different way the individual
have important implications for sample size. This
participates in the study and it is the researcher
who selects which term accurately portrays the is addressed later.
role that the individual plays in the study.
Steps to Sampling
The first step in the sampling process is defining
the population of interest (Figure 31.1). This is
Gender and Race/Ethnicity Issues ordinarily done through a literature review. A thor-
ough examination of completed research in the
Historically, large-scale studies funded by national
area of concern, will define the parameters of the
grant funding focused on majority male subjects.
Government task force reports on women’s health
indicated there was an absence of research data
on women (Schroeder & Snowe, 1994). Similar
observations were made concerning the lack of Step 1: Define the population of interest
research on minorities. As a consequence, the gov-
ernment mandated that women and minorities
must be included in clinical research studies. Now
the National Institute of Health Revitalization Act
of 1993 ensures that there is representation of
women and minorities, unless there is a clear and Step 2: Consider the unit of analysis
compelling rationale that their inclusion is inap-
propriate for the purpose of the study or health of
the subjects (Hayunga & Pinn, 2002). Although
the National Institute of Health has required
awardees of grants to furnish enrollment data by
Step 3: Develop a sampling plan
sex and race/ethnic group and substantial numbers
of women and minorities have been included as
research subjects, more emphasis on identifying
potential differences between men and women and
individuals of diversity is still needed (Hayunga &
Pinn, 2002).
As with any research question, it is important to Step 4: Implement sampling procedures
consider if gender or race factors will impact the
study. For example, in a study of persons with
spinal cord injury, Tzonichaki and Kleftara (2002) Figure 31.1 Steps to sampling.
Chapter 31 Securing Samples for Effective Research Across Research Designs 517
population that will be important for a given study.

Knowing the population of interest is the first step
Sampling in Quantitative
in defining the sample. Research
The next step is to consider the unit of analysis.
What is analyzed in the study, will determine the In quantitative research, the specifications of the
unit. In most occupational therapy research, the subjects are set before the study begins. As noted
unit of analysis is an individual (e.g., the client). earlier, defining the population is the starting point
However, the unit of analysis can be settings (e.g., in sampling. The population of the study includes
comparing long-term care facilities with rehabili- all the individuals that share the defined character-
tation centers), families, caregivers, or couples, istics of interest. Sometimes the population is the
geographical areas, or other elements. For exam- key element of the research question. For example,
ple, in comparing the efficacy of sensory integra- consider an investigator who wants to find out the
tion by evaluating performance of children, the most appropriate assessment for determining if the
unit of analysis would be individual children client with head injury is ready to return to work.
because each child’s scores would be used in the The population for the study would be individuals
data analysis. However, if the goal is to determine with brain injuries who are of working age.
the efficacy of one private pediatric setting using In contrast, if a researcher wants to validate an
sensory integration against another setting, the unit assessment, the actual focus or question of the
of analysis would not be research is the construct
the settings and groups validity or the predictive
of clients as a whole. If The degree to which the validity of the assess-
the goal of the study was selected sample represents ment tool. In this case,
to examine the impact of the population could be
therapy on families with the target population is the very broad (i.e., all per-
a disabled child, the unit degree to which the results sons who could poten-
of analysis could be the tially be assessed with
family. can be generalized to the the instrument). The pop-
Once the population population. ulation is still important
and the unit of analysis since the assessment
have been identified, the should be shown to have
third step is developing a sampling plan. This plan validity for the entire population for whom it is
outlines how the investigator will select the sam- intended. Depending on the assessment, the popu-
ple. In quantitative research, researchers should lation may be defined either broadly (e.g., any dis-
use theoretically defined methods of sampling that abled clients) or narrowly (e.g., clients with
are required for making inferences about the pop- chronic mental illness). Whatever the research
ulation of interest and that are assumed by more question, the researcher must clearly define the
powerful parametric, and even some of the non- characteristics of the population about whom the
parametric statistics. As part of the plan, the conclusions will be drawn from the study.
researcher determines the sample size or how The target population is the population to
many individuals are needed for the study. The which the researcher wants to generalize his or her
rigor of a quantitative study depends on a well intended findings. For example, if the study’s tar-
planned sampling plan that is strictly followed. get population is defined as individuals with schiz-
In qualitative studies, sampling is designed to ophrenia, it would include all individuals who
be less rigid, but it is not any less important. A key have this diagnosis. The researcher selects a subset
distinction of the qualitative design is that, of that target population for the sample. The sam-
although the qualitative researcher does outline a ple will be the subjects whom the researcher uses
sampling plan, the sample can be changed during in his or her research study. The degree to which
the study including the number, type, and descrip- the selected sample represents the target popula-
tion of the subjects to be studied. In some cases it tion is the degree to which the results can be gen-
is critical to the rigor of the study to change the eralized to the population.
boundaries of the sample. The researcher wants to ensure that the sample
The last step is to implement the sampling pro- is representative of the population in order that the
cedures. The next sections elaborate on these crit- results of the study are valid for the population
ical steps for quantitative and qualitative research whom the sample was chosen to represent. For
separately. instance, if a researcher wants to investigate an
occupational therapy intervention for CVA clients, • Functionally independent in the majority of self-
the sample must represent all CVA clients who are care and daily activities.
candidates for occupational therapy. If the study
In this same study participants were excluded if
was done in a regional hospital in a specific state,
they:
the researcher would need to defend how CVA
clients admitted to this hospital are typical of the • Did not attend at least five support group and
population. Just as importantly, the investigator energy conservation sessions,
will need to define how the individuals in the study • Had an exacerbation of symptoms,
sample were actually selected from this hospital to • Changed fatigue medication, or
maximize their representativeness of the target • Had other major illnesses, hospitalizations, or
population. The main purpose of sampling in rehabilitation during the course of the study.
quantitative research is to be able to accurately
draw conclusions about the population by studying In quantitative research, the specification of
the sample. the number of persons in the sample is established
The investigator defines the parameters of the after the design is determined. For example, if the
target population by specifying the inclusion and research study calls for a pretest/posttest compari-
exclusion criteria. The inclusion criteria are the son group design, the researcher can know that two
traits that the researcher has identified as charac- groups of subjects are required. The number of sub-
terizing the population. They serve as the criteria jects necessary will depend on the desired power as
that qualify someone as a subject or participant discussed in Chapter 7 and later in this chapter.
in the study. For example, an inclusion criterion for
the earlier example would be a diagnosis of CVA. External Validity
The exclusion criteria are the characteristics that
The purpose of getting a representative sample is to
will prohibit the subject from being an appropriate
increase the generalizability of the study. A worth-
candidate for the study. These are typically factors
while study is one whose results can be generalized
that could potentially confound the results of the
to a broader population or similar populations.
study (Portney & Watkins, 2000). In other words,
External validity relates to this generalizability
the subject may have characteristics that would
(i.e., a study whose sample allows generalization to
interfere with the interpretation of the results of a
the broader population, has greater external valid-
study, and thus need to be excluded from the study.
ity). Validity refers to the “approximate truth of
For example, if the investigation sought to exam-
propositions, inferences, or conclusions” so exter-
ine the impact of an intervention on persons
nal validity refers to the “approximate truth of con-
with CVA it might produce different outcomes
clusions that involve generalizations” (Trochim,
of the person with CVA who also had a major
2002, p. 1).
mental illness. In such a case co-morbid mental ill-
To make generalizations, the researcher must
ness might be an exclusion criterion. In some
be able to assume the characteristics of the sam-
instances, clients are excluded because they are
ple members will represent the target population.
unable to participate in the study. For example, if
Unfortunately, sampling bias occurs when individ-
the intervention and the outcome measure require
uals are selected who overrepresent or underrepre-
a participant or subject to speak a given language,
sent certain population characteristics (Portney &
those who do not speak the language would be
Watkins, 2000). A bias can be deliberate when an
excluded.
investigator purposefully includes certain kinds of
The sample pool must possess the inclusion cri-
subjects. However, even when bias is unplanned, it
teria, not possess the exclusion criteria, and be
can jeopardize a study’s external validity.
available for selection. Good inclusion and exclu-
The three major threats to external validity are
sion criteria are specific and clearly identified. For
people, place, or time (Trochim, 2002). A potential
example, Mathiowetz (2003) had the following
criticism of any study is that the study’s results
specific inclusion criteria for his study of the
occurred because there was an “unusual” group of
Fatigue Impact Scale for persons with multiple
subjects in the study. For example, if the sample for
sclerosis:
an intervention study included individuals who vol-
• A diagnosis of multiple sclerosis, unteered for the study and were highly invested in
• Age of 18 years or older, results, these volunteers may “work” beyond what
• Functional literacy, the non-volunteer or average individual would
• A Fatigue Severity Scale score of four or greater choose to do. Thus, their outcome from the inter-
(i.e., moderate to high fatigue severity), vention may not generalize to all others who might
• Living in the community, and receive the intervention.
Another threat to external validity is the place Random errors are those that happen by chance.
or location of the study. For example, if an out- For example, suppose 50 subjects for a study on
comes study occurred in an area of a city where the normal grip strength were randomly selected in a
average income is well above average, affluence geographic area. If one of these subjects happened
may influence the results. Other areas with fewer to be an Olympic shot putter, his grip strength will
resources or more environmental stressors linked skew the mean higher than the true average since
to poverty may not achieve the same benefits from there is not 1 Olympian for every 50 persons in the
an intervention because there may not be any fol- population. This type of random error is expected,
low through or resources to meet the basic require- but cannot be predicted. For this reason larger
ments of an intervention (i.e., no resources to pay numbers of subjects are preferred since they reduce
for assistive devices). random error. For example, if 500 subjects had
There is the element of time in a study (i.e., been selected for the study of grip strength, it is
when data are collected from the sample). For highly improbable another Olympian would have
example, a study on the incidence of depression been selected in which case the effect of the 1
could be affected if the study occurs immediately Olympian on skewing the mean grip strength
after the holidays when more people report depres- would be much less.
sion. Thus, the researcher must evaluate these fac- Systematic error is a serious problem for a
tors prior to starting any study in order to limit study. It represents a flaw in the sampling process,
threats to external validity with sampling or sam- which results in the subjects differing from the
pling bias. population systematically. For example, if the
investigator studying grip strength recruited sub-
Kinds of Sampling and Sampling Error jects at a men’s gym, the resulting sample would
include individuals who are likely to have greater
In quantitative research, there are two kinds of grip strength than average.
sampling: The major sources of systematic error are:
• Probability, and • Using volunteers (because those who volunteer
• Nonprobability. for any study are likely to be different from those
who refuse), and
Probability or random sampling is based on
• Using groups that are available and convenient
probability theory that is addressed later in this
(but likely to share some common characteristic
chapter. Random sampling means that each
that makes them different than the intended pop-
member or element of the population can theoreti-
ulation).
cally have an equal chance of being selected for
the sample. For example, consider a study in Avoiding these two sources of error often cre-
which the population consists of all the voters in a ates a dilemma for researchers. For example, con-
particular state. If an investigator could obtain sider an occupational therapist who wants to study
a complete voters list, then a sampling design the efficacy of sensory integration for learning dis-
could be developed that ensures that each voter abled children. If the researcher seeks volunteers
will have an equal chance of being selected for the from a clinic that evaluates and treats children
sample. using sensory integration techniques, which par-
If the population parameters are known, the ents are most likely to volunteer? Those who vol-
sampling error can be calculated. Sampling error unteer will tend to be motivated parents who want
represents the difference between the values any kind of information that might help to address
obtained by the sample and the actual values that their child’s issues.
exist in the population. However, it is very unusual Since the children with the most motivated par-
to have this information. Therefore, calculating the ents are not “typical” of all learning disabled chil-
exact sampling error is usually not possible. In this dren, systematic error will have been introduced
case, an estimated sampling error can be calcu- into the selection of subjects.
lated. Since sampling error represents the degree
to which the sample is representative of the popu-
lation, the larger the sample error, the less repre- Methods of Probability
sentative the sample is to the population and the Sampling and Assignment
lower the external validity.
Randomization is considered to be the cornerstone
Sampling error is due to:
of quantitative research. It balances both the meas-
• Random error, or ured and unmeasured characteristics that affect
• Systematic error. the outcomes of a study, allows for masking, and
Selected Examples of Quantitative Studies in Occupational Therapy

Illustrating Sampling Bias Identified by the Investigators
• The sample was composed of college students in ple (volunteers) that may be biased in their views
an introductory psychology class who got credit about AJOT and the usefulness of research.
for participation in the study. Although it was Consequently, study results may not accurately
possible that there were students with varying reflect general leadership views” (Philibert,
degrees of health, the sample was biased toward Snyder, Judd, & Windsor, 2003, p. 457).
healthy college students. “The extent to which • The participants were kindergarten students from
the participant sample of college students might one school district. “The participants did not
be representative of other participant samples is effectively represent a heterogeneous population
unknown. This leaves some uncertainty about of kindergarten students as a random sample
the generalizability of the results of the study” would have” (Daly, Kelley, & Krauss, 2003,
(p. 55). Additional research comparing healthy p. 462).
individuals with those of varying degrees of ill- • The participants were kindergarten students from
ness would need to be conducted to enhance the one school district. “Only typically developing
validity of this particular study (Reich & students were assessed; thus, the research has no
Williams, 2003). implications for learning disabled students”
• Participants were recruited from the “students, (Daly, Kelley, & Krauss, 2003, p. 462).
faculty, and staff at the university, friends, and • One hundred and twenty-nine volunteers were
family” (p. 101). Older adults were recruited recruited. There was no effect for gender on the
from a research registry developed at the univer- reaction time to the visual stimulus. “However, it
sity. The participants all lived in the Midwest and should be remarked that the relatively small num-
most in urban areas. Findings may not be gener- ber of female participants (28) did not reflect the
alized to those who reside in other parts of the actual gender distribution of older drivers” (Lee,
country and other settings. In addition, the sam- Lee, & Cammeron, 2003, p. 327).
ple of older adults was highly educated. “A more • 129 volunteers were recruited. “The participants
diverse representation of education levels in who volunteered for this study cannot be taken as
future work is needed” (Pohl, Dunn, & Brown, representative of the target population because
2003, p. 105). the sample was not randomly selected but only
• The response rate from the survey was 52%. came from some sectors of the community”
“This response rate indicates a self-selected sam- (Lee, Lee, & Cammeron, 2003, p. 327).
provides a basis for inference (Berger & Bears, who are studying groups of subjects use random
2003). In other words, it is the best method of assignment to allocate subjects to groups. The
removing selection bias (Torgerson & Roberts, principle of random assignment is the same as for
1999). As will be discussed, randomization can be sampling. However, instead of eliminating bias
used both for subject selection (sampling) and sub- that makes the sample unrepresentative of the pop-
ject assignment (allocation to different experimen- ulation, random assignment seeks to eliminate bias
tal groups). due to differences in the groups being compared.
When the population is known, methods of Thus, by randomly assigning subjects into the
probability sampling can be relatively simple and groups that make up a study, an investigator
unbiased. With simple random sampling, all the achieves groups that, according to probability the-
individuals in a defined target population have ory, are likely to be equivalent.
an equal and independent chance of being selected True randomization is often difficult or prohib-
for a sample. Simple random sampling is also itive in occupational therapy studies owing to the
known as sampling without replacement (Portney structure of the practice environment. Fortunately,
& Watkins, 2000); once a person is selected, he or randomization is also considered as referring “…
she is out of the pool and has no further chance of to a broad collection of allocation methods”
being selected. Often a table of random numbers (Berger & Bears, 2003, p. 468). For example,
or a computer-generated list of random numbers is Berger and Bears argue that, in studies in which
used to select the sample from the list of the target groups are compared, strict allocation (assign-
population. ment) methods can eliminate selection bias as
In addition to using random sampling to choose effectively as randomization. If the terms of allo-
a subject pool from a population, investigators cation are identified before identification (or
screening) of subjects, selection bias is controlled. Cluster Sampling

However, if terms of allocation are done after the
identification of subjects, then direct selection bias Another common probability sampling method is
is introduced. In other words, if it is determined cluster sampling. In cluster sampling individuals
how subjects will be assigned to particular groups are not randomly selected. Rather, groups or pro-
prior to the start of the study, then it can be argued grams are selected and every member of that group
the bias is eliminated (e.g., every other subject will or program is invited to participate in the study.
be assigned to the control group). If subjects are For example, cluster sampling may be used to
identified and then assigned to groups, direct determine the usefulness of an evaluation in outpa-
selection bias is likely (e.g., the more willing sub- tient rehabilitations centers. Centers in certain
jects might be assigned to the treatment group). states or counties may be selected and all thera-
Bias can also be introduced if the investigator pists in the centers asked to participate.
has discretion to approve or deny enrollment in the
study or has advanced knowledge of the groups. In Nonprobability Sampling Methods
addition, if alternation of assignment of subjects
(e.g., every other identified subject is assigned to When the parameters of a population are not
a particular group) is used instead of randomly known or when it is not feasible to do some type
assigning groups, the sequence becomes pre- of probability sample, nonprobability sampling is
dictable and problematic (Berger & Bears, 2003). used in quantitative research. In this instance, it is
very important to try to attain the greatest degree of
Stratified Random Sampling representation for the sample. When nonprobabil-
ity sampling is used, the investigator must:
If a study requires that certain groups be repre-
sented equally, a stratified random sample may be • Clearly define the process of the sampling,
more appropriate. Stratified sampling is similar to • Acknowledge the limitations of the sampling
a simple random sample, but the selection is from procedure, and
identified subgroups in the population. For exam- • Justify why the sampling limitations do not jeop-
ple, if a study about the occupational therapy pro- ardize the research question being answered.
fession was undertaken, the researcher may want
to ensure that both the professional and technical The sample characteristics still need to be
levels of the profession are represented. Therefore, defined clearly in terms of inclusion and exclusion
a stratified random sample of the professional level criteria.
therapists and the occupational therapy assistant
population would be separated and the appropriate Convenience Sampling
number from each group selected randomly. It is
Convenience sampling method is the most prob-
important to ensure that representation from the
lematic, yet widely used nonprobability method to
stratified categories is proportional to that cate-
obtain subjects. Convenience sampling is the use
gory’s proportion of the whole population. For
of volunteers or easily available subjects such as
example, if there were two occupational therapists
a group of students in a program or clientele in a
for every assistant in the profession then the result-
clinic. In a convenience sample, subjects are
ing sample should reflect that proportion.
enrolled as they agree to enter the study, until the
desired number is reached. While a convenience
Systematic Sampling
sample is always the weakest sampling method,
Systematic sampling is considered equivalent to the degree of appropriateness of using a conven-
random sampling as long as there is no reoccurring ience sample depends on the research question.
pattern or order in the listing (Portney & Watkins, For example, if the research question asked about
2000). The number of subjects for the sample is normal grip strength for female college students,
known and divided into the number of the popula- selecting the women in an occupational therapy
tion. Then, individuals are selected from the list by class, on the face of it, does not appear to enter a
taking every “k”th name. For example, if a list of large amount of bias. However, if the research
licensed occupational therapists from Pennsylva- question asked about average knowledge of health
nia includes 1,500 occupational therapists and the issues among college students, an occupational
researcher has decided to survey 300 therapists, therapy class might be very biased in terms of such
the research would select every fifth individual knowledge. In the latter case, using the class as a
from the comprehensive list to survey. convenience sample is much less defensible.
Purposive Sampling example, in correlational research, it is traditional

to use a minimum of 30 subjects (Gall, Borg, &
Purposive sampling is the deliberate selection of Gall, 1996). Survey research requires a minimum
individuals by the researcher based on certain of 100 in each major subgroup and 20 to 50 in each
predetermined criteria (usually stated as inclusion minor subgroup whose responses will be analyzed
and exclusion criteria) (Portney & Watkins, 2000). (Gall, Borg, & Gall, 1996). In causal–comparative
For example, if a study sought to understand the or experimental research, there should be at least
impact of being involved in a wellness group on 15 subjects in each group. However, within this
health behaviors among women, an investigator category, there are variations of appropriate num-
might seek subjects from a wellness group at a bers of subjects. For example, in crossover two-
local women’s center. period designs, the same subjects are used in the
treatment and control groups. The subjects are their
Snowball or Network Sampling own controls and therefore sample sizes for this
Snowball or network sampling is a method in can be substantially smaller than the parallel
which initially identified subjects provide names groups (Albert & Borkowf, 2002).
of others who may meet the study criteria.
Snowball sampling is used when potential subjects Statistical Power and Sample Size
are difficult or impractical to obtain and when the It has become good research practice to base deci-
intended subjects are likely to be aware of others sions about sample size on statistical power.
who share their characteristics. For example, con- Statistical power refers to the likelihood of finding
sider a researcher who wants a sample of mothers a significant difference between groups or associa-
of children with spina bifida. If the researcher has tion between variables when one exists (Albert &
some initial contacts, such mothers usually know Borkowf, 2002). The number of subjects is directly
other members of the same group through support related to the statistical power of a study. As dis-
groups and other means. They can, thus, be useful cussed in Chapter 7 and this chapter, investigators
in recruiting additional subjects. However, with can determine before a study is undertaken how
the snowball method, the sampling pool can many subjects are needed.
become biased and the researcher has no control To determine the number of subjects necessary
over who is nominated for the study. to have sufficient statistical power, four interre-
lated components need to be considered:
Quota Sampling
1. Sample size,
Quota sampling is used when different proportions 2. Effect size,
of subject types are needed so that there is appro- 3. Alpha level or level of significance, and
priate representation in the sample that may not 4. The power or the odds of observing a treatment
be attainable with purposive or convenience sam- effect when it occurs (Trochim, 2002).
pling. For instance, an investigator who wanted to
If the values of three of these components are
compare male to female occupational therapists
known, computation of the fourth factor is possi-
might use quota sampling to attain equal numbers
ble. Thus, the number of subjects needed can be
of subjects in both groups.
determined based on reasonable estimates of the
other factors. The goal is to balance these compo-
Determining Sample Size nents so that maximum level of power is available
to detect an effect if one exists, given constraints
It is important to determine the right sample size on the other components (Trochim, 2002).
for every study. The sample size will influence
many factors in the design and implementation of
Effect Size
a study, especially the costs and time involved in a
study. A general rule of thumb is that one should Effect size is the “effect of differences between two
obtain the largest sample possible. Larger samples means or the degree of relationship between two
make a study more challenging to complete and variables in the results of a study” (Stein & Cutler,
require more resources. On the other hand, if there 2000, p. 502). The smaller the effect size, the more
are too few subjects, a study is not worth under- subjects a study will need. For example, consider a
taking since any findings would be suspect. study about the changes in handwriting for chil-
Researchers have agreed on a minimum number dren who were enrolled in a 6- month long sensory
of cases needed for specific research designs. For processing program. Such a study would compare
Selected Examples of Quantitative Studies in Occupational Therapy Using

Specific Types of Sampling Strategies
Randomly Assigned including having a learning disability as defined

by the State of Washington” (Handley-More,
• Subjects needing bathing devices were chosen Deitz, Billingsley,& Coggins, 2003, p. 141).
from inpatient and outpatient services from one • The sample of low-income older adults was pur-
hospital in Hong Kong. The subjects were ran- posely selected if they met inclusion criteria of
domly assigned to the intervention group or a reporting impairments in one or more areas of
control group (Chiu & Man, 2003). the Functional Independence Measure motor sub-
• Subjects were randomly assigned to test admin- scale and indicated a need for environmental
istrators and either of the two treatment groups modifications to their home to increase perfor-
or control group. “Environments and treatment mance capacity (Stark, 2004).
schedules for both groups were matched”
(Shaffer, Jacokes, Cassily, Greenspan, Tuchman, Convenience
& Stemmer, 2001, p. 157).
• The sample was 140 participants who were
Stratified Random Sampling selected from four groups with different levels
of neurological impairment and community par-
• Stratified random sampling was used in order ticipation. There was a limitation in that the lev-
for the total sample to have “…equal numbers of els of education were not equal among the four
boys (n ⫽ 20) and girls (n ⫽ 20), 6- and 7-year groups. Education was used and controlled for
olds (n ⫽ 20 each), and to allow for equal num- as a covariate in the statistical analysis to com-
bers of right-handed (n ⫽ 32) and left-handed pensate (Goverover & Josman, 2004).
(n ⫽ 8) children in each group” (Smith-Zuzovsky • Participants were recruited through day programs
& Exner, 2004, p. 383). located in one large metropolitan area and post-
• Adolescents 12 to 18 years were recruited from a ings at two mental health centers. Participants
target population of 110,000. “The use of a sam- received a small monetary honorarium. Sampling
ple stratified by age allowed for exploration of bias may be present since more motivated and
the potential differences to emerge during adoles- socially oriented individuals may have volun-
cent development…” (Passmore, 2004, p. 66). teered (Laliberte-Rudman, Hoffman, Scott, &
• The investigators selected five states from various Renwick, 2004).
parts of the country based on their geographic • Volunteers were recruited by the investigator at
location and variety of occupational therapy pro- the acute psychiatric hospital where she was
grams. AOTA member mailing lists were employed (McNulty & Fisher, 2001).
ordered. Faculty members, students, and occupa-
tional therapy assistants were eliminated from Snowball or Network
the mailing lists. The proportion of AOTA mem-
bers in each state was determined and based on • “All were recruited via word-of-mouth”
proportions, and the surveys were mailed to (Niemeyer, Aronow, & Kasman, 2004, p. 589).
randomly selected AOTA members from the • Use of posters and brochures were used to recruit
five states (Philibert, Snyder, Judd, & Windsor, subjects. “Word of mouth and personal contacts
2003). were also used” (Clemson, Manor, & Fitzgerald,
2003, p. 109).
Purposive • Participants were recruited from several settings
including “a health club, a school employee
• “Participants were identified by their occupational retirement community, a folk dancing group, a
therapist, school psychologist, or special educa- student group, and a military base community”
tion teacher according to predetermined criteria (Dickerson & Fisher, 1997, p. 248).
handwriting proficiency in children who received tor might consider the sensitivity of the measure
the sensory processing programming to a group of of handwriting used, any pilot data or previous
children who did not receive the program. The size studies that give an indication of how much change
of the expected difference in handwriting pro- could be expected as a result of the interven-
ficiency between the two groups (the effect) would tion, and how much change might occur naturally
need to be estimated. Several considerations might in the control group as a result of maturation or
enter into this estimate. For instance, the investiga- learning.
Level of Significance Selected Examples of Quantitative Studies

in Occupational Therapy Illustrating Sample
The level of significance or “alpha” is typically set
Size Issues
at .05 in occupational therapy studies. This means
that investigators are willing to accept a 5% chance
• “The sample size precluded investigators from
that they will find an effect by chance when there
conducting a factor analysis to further establish
really is no true effect. This is known as a type I validity” (Laliberte-Rudman, Hoffman, Scott,
error (we mistakenly accept the alternative hypoth- & Renwick, 2004, p. 20).
esis when the null hypothesis is in fact correct and • With this pilot study’s small sample size, the
should be accepted). Decreasing alpha (e.g., to study was able to identify one significant differ-
.01), decreases the chances of making a Type I ence in self-care performance between the con-
error; however, it also decreases the “power” or the trol and experimental groups. However,
chances of rejecting the null when the alternative “…further research, with a greater number of
hypothesis is true. “Power is the probability of participants over longer duration, is recom-
rejecting the null hypothesis when the alternative mended in order to detect other differences that
may exist” (Gange & Hoppes, 2003, p. 218).
hypothesis is true. Power equals 1 minus the prob-
• To help determine what sample size was needed
ability of making a Type II error” (Albert & for this study, the investigators looked at similar
Borkowf, 2002, p. 179). studies and found that two in which a sample
size of 20 was needed to achieve a power of
Calculating Sample Size .91 at .05 with an effect size around 31. How-
ever, in this study, the investigators wanted to
Using expected effect size, alpha level, and power, compare males and females, thus requiring a
an appropriate sample size can be established. larger sample. “Therefore, in order to achieve
Traditionally, occupational therapy studies use .05 a power of .80 at .05 with an affect size of
as the alpha level and .80 for statistical power. The .68, a sample size of 56 was needed” (Dudek-
following are points that need to be considered in Shriber, 2004, p. 511).
calculating a necessary sample size:
• As the sample size increases, the power increases.
• If variation in outcome decreases, the power Attrition (or subject dropout) is an issue that
increases. needs to be considered. Especially for studies that
• If variation in outcome increases, the sample size involve considerable time and effort on the part of
needs to increase, the subjects, the projected sample size should take
• The power increases as the effect size increases, into account the possibility of attrition. Finally, if
and the measure used has a low reliability, the power of
• If the effect size decreases, the sample size needs tests of statistical significance is decreased and an
to increase. increase in the number of subjects is justified
(Gall, Borg, & Gall, 1996).
Researchers often consult with a statistician to
perform analyses that determine the right number
of subjects based on a power estimate. Sampling in Qualitative
Other Factors Affecting Sample Size Studies
In addition to the statistical power analysis, several Morse and Field (1995) identify two principles
other factors affect the determination of sample that guide qualitative sampling:
size including subgroup analysis, expected attri-
• Appropriateness, and
tion, and reliability of measures (Gall, Borg, &
• Adequacy.
Gall, 1996). In group comparison studies, there is
often a need to compare subgroups after the pri- Appropriateness is the identification of partici-
mary analysis is complete. For example, there pants who will best inform the researcher about the
may be a need to compare right-handers and left- phenomena under inquiry. In qualitative research,
handers within the experimental and control although the sample size is often small, the amount
group. If the subgroups do not have enough sub- of data can be substantial and expensive to collect.
jects, the analysis may not yield any significant Therefore, research must be efficient and effective
results. Thus, it is important to plan for any sub- (Meadows, 2003). The researcher must interview
group analysis prior to the start of the study in participants who are in a position to offer the most
order to plan for enough subjects. information.
Since randomization would not serve this end, • The “marginal people” are the individuals who
random selection is not considered an effective do not feel like they really belong to the group or
sampling strategy for qualitative research. The feel ambivalent about the group. Because they do
researcher theoretically should know who would not have strong loyalty, they will often talk to
be the best participant based on the needs of the outsiders and be able to give valuable insights
study. Moreover, the number of participants or about the group. Using marginal participants
informants is not as important as their amount of requires caution lest the researcher be seen as
exposure to and knowledge of the phenomena to aligned with a member whom the others believe
be studied. to be the least trust worthy.
Adequacy of the data means that enough data
will be available to provide a rich description of Most participants can only give part of the pic-
the phenomena of interest (Meadows, 2003). The ture or have only one perspective on a setting that
goal is saturation, meaning that after continued includes many perspectives. It is therefore criti-
interviewing and/or observation, no additional cally important to make sure that all perspectives
information is gained. are represented in the collection and verification of
Depending on the study, participants may be the data. Sometimes the researcher does not know
obtained from the community or formal or infor- who are the best participants. In this case, Morse
mal groups. Frequently, volunteers are sought to (1991) recommends that the researcher use sec-
participate and those who usually volunteer tend to ondary selection. This means that the researcher
be more receptive to the interviewer and readily conducts many interviews. If a participant does not
offer information (Meadows, 2003). However, it is have the information that is needed or does not
important not to select informants based just on meet the qualities of a good interviewee, the
convenience, but on what that person can offer in researcher does not use the interview in the analy-
terms of illuminating a particular concept, experi- sis. Such data is set aside for possible use in the
ence, or cultural context. future if it turns out to have some validity.
In some organizations or groups, there will be A necessary factor in acquiring good partici-
informants who are in key positions or have infor- pants for qualitative research is the amount of rap-
mation that will be more port and trust established
insightful than other between the participants
individuals in the organi- Two principles guide qualita- and researcher. This ele-
zation. It is important ment is paramount to
to identify those key tive sampling—appropriate- the success of the study.
informants and include ness and adequacy. If the key participants
them as participants. Key are not receptive to the
informants are selected researcher or the project,
on the basis of their role, knowledge or insights, they may give shallow or partial information, not
and the type of relationship they have with others disclose their true feelings, or provide invalid
in the query. However, it can be valuable to also information on the topic.
pay attention to the quiet, less verbally expressive As with all research studies, the sampling
individuals (Meadows, 2003). These individuals methods are determined by the nature of the study.
may have a different perspective and offer insight The fluid nature of the qualitative study process is
that would otherwise be ignored. also reflected in how participants are selected. This
Douglas (1976) has identified four types of process is inductive and dynamic and may change
individuals in any setting that are useful to the as the study evolves. In fact, it is common for new
qualitative researcher: participants to be sought out as a study progresses.
• The “social gadflies” are the well-liked and lively

Unit of Analysis
individuals who mix and talk to everyone in the
group, The unit of analysis for qualitative studies tends to
• The “constant observers” are the individuals who be either people-focused or structure-focused
are the longer, well-established members of the (Patton, 2002). In people-focused studies, the
group who will freely speak of the details of past researcher examines individuals and small infor-
events, mal groups such as friends, gangs, or families.
• The “everyday philosophers” who think a great Structure-focused units include projects, pro-
deal about the setting, can give insights to what is grams, organizations, or units within organizations
going on, but who are not as forthcoming, and (Patton, 2002).
Some of the most common factors that are con- is seeking to find the broadest range of experi-
sidered when determining participants for a quali- ences, information, and/or perspectives possible.
tative study are: Homogeneous selection is the opposite. In this
instance, the investigator seeks informants who
• Culture,
have the same experience. The researcher wants to
• Geographic or organizational location,
simplify the number of experiences, characteris-
• Time or event-related experience, and
tics, and/or conceptual domains under investiga-
• Personal experience of a unique condition.
tion. This strategy is used for exploring a particular
One of the most common considerations in phenomenon in-depth, rather than examining all
qualitative research is the culture of the partici- the variations of which it is an instance.
pants. For example, Bazyk, Stalnaker, Llerena, Theory-based selection is when the researcher
Ekelman, and Bazyk’s (2003) study on use of play selects only individuals who exemplify a particular
with Mayan children; in this study, how culture theoretical construct for the purpose of expanding
influenced play was a major concern. Another the current understanding of a theory. This strat-
example is a study of two undocumented immi- egy focuses on a particular concept and seeks to
grants and their child’s participation in an early explore its meaning in-depth.
intervention program (Alvarado, 2004). As these Yet another strategy involves finding confirm-
two examples illustrate, qualitative studies often ing or disconfirming cases. In this instance, the
focus on individuals who represent unique cultural investigator looks purposefully for the informant
experiences. who will support or challenge an emerging inter-
Geographic location or membership in an orga- pretation. This is a useful strategy in the later
nizational group can be used to initially define the stages of the qualitative study, when the researcher
participants for a study. Ward’s (2003) study of the begins to feel confident that the data are leading
clinical reasoning of occupational therapists work- the investigation in a specific direction. Finding
ing in community mental health is an example of informants who can confirm or disconfirm that
such a study. Other factors leading to sample selec- direction is critical for increasing the confidence of
tion may be time or event-related. For example, the analysis and expanding the understanding of
qualitative studies often focus on experiences dur- the phenomenon.
ing events such as the Depression, 9/11, or the Finally, the researcher can select cases on the
Vietnam War era (DePoy & Gitlin, 1998; Patton, criteria that they represent an extreme example of
2002). a phenomenon or that they represent the average
Personal experiences are frequently a focus of case. In each of these instances, the design or pur-
qualitative studies, particularly phenomenological pose of the study determines what the most useful
studies. Neville-Jan’s (2003) study of chronic pain strategy is.
and Kinnealey, Oliver, and Wilbarger’s (1995)
study of the experience of being an adult with sen- Determining the Number of Participants
sory defensiveness are both examples of this type
In qualitative research, there are no standards or
of research focus. In such studies, investigators
set rules for determining the “right” number of
seek out participants who experience the phenom-
subjects or informants. In fact, the number of par-
ena under study.
ticipants is less important than selecting partici-
pants who can ensure richness of information and
Strategies for Selecting Participants depth of understanding. In some instances, explor-
Selecting participants in qualitative research is ing the experiences of a very few subjects in depth,
always purposeful. That is to say, the researcher may be sufficient to thoroughly exhaust a topic. In
strategically determines who would make the best other instances, an investigator may need to con-
participants. Obviously, the best participants are tinue selecting participants to gather necessary
those who have the knowledge and are willing and information on all elements of the question.
able to share the knowledge in enough depth so as In the end, the quality of the data obtained in
to be understandable and useful to the researcher. relation to the study question drives the sampling
Nevertheless, there are specific strategies for process. When the researcher wants to explore a
selecting informants. One of the most commonly phenomenon, explain diversity, or understand vari-
used strategies is maximum variation. Maximum ation, then a larger sample is needed (Patton,
variation involves seeking individuals who have 2002). Lincoln and Guba (1985) recommend that
extremely different experiences of the phenome- the appropriate sample size be determined by the
non being studied. In this strategy, the researcher information gathered. When no new information
Selected Examples of Qualitative Studies in Occupational Therapy Using Specific Types of Sampling
Purposefully Selected Settings Maximum Variation

• The study’s aim was to investigate the use of • Participants interviewed were from many differ-
occupation with individuals with life threatening ent sites, varied in years of experience, and used
illnesses. A hospice attached to a hospital was the income for a variety of purposes. The partici-
selected (Lyons, Orozovic, Davis, & Newman, pant craft workers used a variety of media
2002). processes (Dickie, 2003).
• The study was limited to physical rehabilitation
settings because of the variability of occupational
therapy settings and physical rehabilitation repre- Homogenous
sents one of the largest areas of practice • All the participants were Caucasian females who
(Scheirton, Mu, & Lohman, 2003). graduated from the same occupational therapy
program at the same university (Scheerer, 2003).
Purposefully Selected Participants
• Participants were recruited from local Convenience
Parkinson’s disease support groups. The criteria
included individuals who would be able to hear • The participants were recruited through support
and respond verbally in a face-to-face interview. groups associated with the local chapters of the
“Purposive sampling (Lincoln & Guba, 1985) National Multiple Sclerosis Society. Interested
was used to select four participants from the pool individuals contacted the study office and were
of seven people who indicated interest” (Doyle screened to determine eligibility (Finlayson,
Lyons & Tickle-Degnen, 2003, p. 28). 2004).
• “I used purposive sampling to select three chil-
dren with physical disabilities.” (Richardson, Snowball or Network
2002, p. 298).
• Seven participants were selected from known
Purposefully Selected Process contacts and an additional participant was sug-
gested by one of the original participants (Egan
• Participants were adults with acute hand injuries & Swedersky, 2003).
who were receiving outpatient therapy. “Usual • Participants were chosen based on their reputa-
treatment protocols are followed because the tion as expert occupational therapists in commu-
intent is to document the adaptation process nity mental health as well as their ability to
as it naturally occurs” (Chan & Spencer, 2004, communicate and reflect on their practice (Ward,
p. 129). 2003).
on the study question is forthcoming from new individuals (seeking breadth) or a greater range of
subjects, then the “right” number of subjects has experiences from a smaller number of individuals
been achieved. Patton (2002) recommends that the (depth) (Patton, 2002).
design specify minimum samples based on the
expected description of the phenomena. As with Gaining Access
all other aspects of qualitative study, the sample
size will need to be flexible, fluid, and subject to Gaining access is the entry point into a qualitative
change. inquiry and affects the selection of subjects. Fre-
quently the investigator enters the setting through
Depth Versus Breadth the gatekeeper or the person in charge of the set-
ting or organization. Winning the trust of the gate-
Before and during the data collection process, a keeper through a straightforward approach or
qualitative investigator must be acutely aware of through contacts in the organization impacts the
the implications of the participant selection ability to freely select participants.
choices and be prepared to define why all partici-
pants were selected, interviewed, and/or observed.
Domain Analysis
There is generally a trade-off between breadth and
depth. The researcher needs to decide whether to The researcher makes ongoing judgments on who
explore specific experiences of a large number of to interview and/or observe based on the unfolding
research question(s) and how well the questions Berger, V.W. & Bears, J.D. (2003). When can a clinical
are being answered. Usually the researcher starts trial be called ‘randomized’? Vaccine, 21, 468–472.
Chan, J., & Spencer, J. (2004). Adaptation to hand injury:
with broad sampling. As the research progresses, An evolving experience. American Journal of
the question (and thus sampling) becomes more Occupational Therapy, 58, 128–139.
focused and narrowed. Domain analysis is the crit- Chiu, C., & Man, D. (2003). The effect of training older
ical process of selecting and adding pieces of adults with stroke to use home-based assistive devices.
Occupational Therapy Journal of Research, 24,
information through interview and observation and
113–120.
analyzing it for further discovery. The subject Clemson, L., Manor, D., & Fitzgerald, M. (2003).
selection process is strategically guided by the aim Behavioral factors contributing to older adults falling
of achieving rich data for discovery. in public places. Occupational Therapy Journal of
Research, 23, 107–117.
Daly, C. J., Kelley, G. T., & Krauss, A. (2003). Brief
Conclusion report-Relationship between visual-motor integration

and handwriting skills of children in kindergarten: A
modified replication study. American Journal of
This chapter overviewed the process of securing Occupational Therapy, 57, 459–462.
DePoy, E. & Gitlin, L. N. (1998). Introduction to research:
samples in order to do effective research. Issues Understanding applying multiple strategies. St. Louis:
affecting the selection of subjects include the study C. V. Mosby.
question, research approach and design, and prag- Dickerson, A. E. & Fisher, A. G. (1997). Psychology and
matic considerations. The steps to sampling Aging, 12, 247–254.
include defining the population through a literature Dickie, V. A. (2003). Establishing worker identity: A study
of people in craft work. American Journal of
review, considering the unit of analysis, and devel- Occupational Therapy, 57, 250–261.
oping a sampling plan. Doyle Lyons, K., & Tickle-Degnen, L. (2003). Dramaturgi-
In quantitative research the main concern of cal challenges of Parkinson’s disease. Occupational
sampling is whether the sample represents the tar- Therapy Journal of Research, 23, 27–34.
Douglas, J.D. (1976). Investigative social research:
get population, whether compared groups are Individual and team research. London: SAGE
equivalent, and whether sample size is large Publications.
enough to achieve statistically significant results. Dudek-Shriber, L. (2004). Parent stress in the neonatal
In qualitative studies, the investigator seeks partic- intensive care unit and the influence of parent and
ipants who will best inform the researcher about infant characteristics. American Journal of
the topic under inquiry and purposefully samples Egan, M., & Swedersky, J. (2003). Spirituality as experi-
until the topic is saturated. The number of partici- enced by occupational therapists in practice. American
pants is not as critical as selecting participants who Journal of Occupational Therapy, 57, 525–533.
can ensure the richness of information and depth Finlayson, M. (2004). Concerns about the future among
older adults with Multiple Sclerosis. American Journal
of understanding. of Occupational Therapy, 58, 54–63.
Careful sampling is essential to a rigorous Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational
study. Good sampling takes planning, effort, and Research, (6th ed.). White Plains, NY: Longman.
resources. In the end, it is the foundation for hav- Gange, D. E., & Hoppes, S. (2003). Brief report—the
ing confidence in the study findings. effects of collaborative goal-focused occupational ther-
apy on self-care skills: A pilot study. American Journal
of Occupational Therapy, 57, 215–219.
REFERENCES Goverover, Y, & Josman, N. (2004). Everyday problem
Albert, P. S. & Borkowf, C. B. (2002). An introduction to solving among four groups of individuals with cogni-
biostatistics: Randomizations, hypothesis testing, and tive impairments: Examination of the discriminant
sample size. In J. Gallin (Ed.), Principles and practice validity of the observed tasks of daily living-revised.
of clinical research (pp. 163–185). San Diego: Occupational Therapy Journal of Research, 24,
Academic Press. 103–112.
Alvarado, M.I. (2004). Mucho camino: The experience of Handley-More, D., Deitz, J., Billingsley, F. F., & Coggins,
two undocumented Mexican mothers participating in T. E. (2003). Facilitating written work using computer
their child’s early intervention program. American word processing and word prediction. American
Journal of Occupational Therapy, 58, 521–530. Journal of Occupational Therapy, 57, 139–151.
Bazyk, S., Stalnaker, D., Llerena, M., Ekelman, B., & Hayunga, E. G., & Pinn, V.W. (2002). NIH policy on the
Bazyk, J. (2003). Play in Mayan children. American inclusion of women and minorities as subjects in clini-
Journal of Occupational Therapy, 57, 273–283. cal research. In J. Gallin (Ed.), Principles and practice
Belgrave, F. Z., & Walker, S. (1991). Differences in reha- of clinical research (pp. 145–160). San Diego:
bilitation service utilization patterns of African Academic Press.
Americans and White Americans with disabilities. In Kinnealey, M., Oliver, B., & Wilbarger, P. (1995). A phe-
S. Walker, F. Z. Belgrave, R. Nicholls, & K. Turner nomenological study of sensory defensiveness in
(Eds.), Future frontiers in the employment of minority adults. American Journal of Occupational Therapy, 49,
persons with disabilities. Washington, DC; Howard 444–451.
University Research and Training Center. Kizony, R. & Katz, N. (2002). Relationships between cog-
nitive abilities and the process scale and skills of the Occupational Therapy Journal of Research, 23,
assessment of motor and process skills (AMPS) in 99–106.
patients with stroke. Occupational Therapy Journal Portney, L. & Watkins, M. (2000). Foundations of clinical
of Research, 22, 82–92. research: Applications to practice (2nd ed.). Upper
Laliberte-Rudman, D., Hoffman, L., Scott, E., & Saddle River, NJ: Prentice-Hall Health.
Renwick,R. (2004). Quality of life for individuals with Reich, J., & Williams, J. (2003). Exploring the properties
schizophrenia: Validating an assessment that addresses of habits and routines in daily life. Occupational
client concerns and occupational issues. Occupational Therapy Journal of Research, 23, 48–55.
Therapy Journal of Research, 24,13–21. Richardson, P. K. (2002). The school as social context:
Law, M., Baptiste, S., McColl, M. A., Polatajko, H., & Social interaction patterns of children with physical
Pollock, N. (1998). Canadian Occupational Perfor- disabilities. American Journal of Occupational
mance Measure (3rd ed.). Ottawa, Ontario: Canadian Therapy, 56, 296–304.
Association of Occupational Therapists Publications Scheerer, C. R. (2003). Perceptions of effective profes-
ACE. sional behavior feedback: Occupational therapy student
Lee, H. C., Lee, A. H., & Cammeron, D. (2003). Validation voices. American Journal of Occupational Therapy, 57,
of driving simulator by measuring the visual attention 205–214.
skill of older adult drivers. American Journal of Scheirton, L., Mu, K., & Lohman, H. (2003). Occupational
Occupational Therapy, 57, 324–328. therapists’ responses to practice errors in physical reha-
Lincoln, W. S. & Guba, E. G. (1985). Naturalistic inquiry. bilitation settings. American Journal of Occupational
Beverly Hills, CA: SAGE Publications. Therapy, 57, 307–314.
Lyons, M., Orozovic, N., Davis, J., & Newman, J. (2002). Schroeder, P. & Snowe, O. (1994). The politics of women’s
Doing-being-becoming: Occupational experiences of health. In C. Costello & A. J. Stone (Eds.), The
persons with life-threatening illnesses. American American woman 1994–95. New York: W. W. Norton .
Journal of Occupational Therapy, 56, 285–295. Shaffer, R. J., Jacokes, L. E., Cassily, J. F., Greenspan,
Mathiowetz, V. (2003). Test-retest and convergent validity S. I., Tuchman, R. F., & Stemmer, P. J. (2001). Effect
of the Fatigue Impact Scale for persons with multiple of Interactive Metronome training on children with
sclerosis. American Journal of Occupational Therapy, ADHD. American Journal of Occupational Therapy,
57, 463–467. 55, 155–162.
McNulty, M. C., & Fisher, A. G. (2001). Validity of using Smith-Zuzovsky, N., & Exner, C. E. (2004). The effect of
the Assessment of Motor and Process Skills to estimate seated positioning quality on typical 6- and 7-year old
overall home safety in persons with psychiatric condi- children’s object manipulation skills. American Journal
tions. American Journal of Occupational Therapy, 55, of Occupational Therapy, 58, 380–388.
649–655. Stark, S. (2004). Removing environmental barriers in the
Meadows, K. A. (2003). So you want to do research? 4: An homes of older adults with disabilities improves occu-
introduction to quantitative methods. British Journal of pational performance. Occupational Therapy Journal
Community Nursing, 8, 519–526. of Research, 24, 32–39.
Moritsugu, J., & Sue, S. (1983). Minority status as a stres- Stein, F. & Cutler, S. K. (2000). Clinical research in occu-
sor. In R. D. Felner, L. A. Jason, J. Moritsugu, & S. S. pational therapy (4th ed.). San Diego: Singular
Farber (Eds.), Preventive psychology: Theory, research, Publishing Group.
and practice (pp. 162–174). New York: Pergamon. Torgerson, D. J. & Roberts, C. (1999). Randomization
Morse, J. M. (1991). Strategies for sampling. In J. D. methods: Concealment. British Medical Journal, 319,
Morse (Ed.). Qualitative nursing research: A contem- 375–376.
porary dialogue (rev. ed., pp. 127–145). Newbury Park, Trochim, W. M. (2002). Research methods knowledge base
CA: SAGE Publications. (2nd ed.). Retrieved May 19, 2005 from http://www.
Morse, J. & Field, P. (1995). Qualitative research methods socialresearchmethods.net/kb/external.htm.
for health professionals. Thousand Oaks: SAGE Tzonichaki, I. & Kleftara, G. (2002). Paraplegia from
Publications. spinal cord injury: Self esteem, loneliness, and life
Neville-Jan, A. (2003). Encounters in a world of pain: An satisfaction. OTJR: Occupation, Participation, and
autoenthography. American Journal of Occupational Health, 22, 96–103.
Therapy, 57, 88–98. Ward, J. D. (2003). The nature of clinical reasoning with
Niemeyer, L. O., Aronow, H. U., & Kasman, G. S. (2004). groups: A phenomenological study of an occupational
Brief report—A pilot study to investigate shoulder therapist in community mental health. American
muscle fatigue during a sustained isometric wheelchair- Journal of Occupational Therapy, 57, 625–634.
propulsion effort using surface EMG. American Wells, S. A., & Black, R. M. (2000). Cultural competency
Journal of Occupational Therapy, 58, 587–593. for health professionals. Rockville, MD: American
Passmore, A. (2004). A measure of perceptions of Occupational Therapy Association.
generalized self-efficacy adapted for adolescents. Zea, M. C.; Quezada, T., & Belgrave F. Z. (1994). Latino
Occupational Therapy Journal of Research, 24, cultural value and their role in adjustment to disability.
64–71. Journal of Social Behavior and Personality, 9(2),
Patton, M. Q. (2002). Qualitative research and evaluation 185–200.
methods. Thousand Oaks, CA: SAGE Publications.
Philibert, D. B., Snyder, P., Judd, D., & Windsor, M. M. RESOURCES
(2003). Practitioners’ reading patterns, attitudes, and
use of research reported in occupational therapy jour- Additional Readings Beyond References
nals. American Journal of Occupational Therapy, 57, Fink, A. (1995). How to sample in surveys. Thousand
450–458. Oaks, CA: SAGE Publications.
Pohl, P., Dunn, W., & Brown, C. (2003). The role of sen- Trochim, W. (2000). The research methods knowledge base
sory processing in the everyday lives of older adults. (2nd ed.). Cincinnati, OH: Atomic Dog Publishing.
C H A P T E R 3 2
Collecting Data
Renée R. Taylor • Gary Kielhofner
All research depends on data. Data are pieces of can then be examined in isolation or compared to
information that have been gathered according those of other variables.
to specified rules and procedures to answer ques-
tions under investigation in a study (Crotty, 1998; Levels of Data
DePoy & Gitlin, 1998; Neuman, 1994; Portney &
As discussed in Chapters 12 and 15, quantitative
Watkins, 2000). Answering the research questions
data can be:
requires that information be gathered on the phe-
nomena or variables under study. In the end, the • Nominal or categorical, in which case numeric
dependability of the research findings are all labels are assigned to designate specific cate-
linked to whether the data collected are reliable gories of a given variable (e.g., determining
and valid. whether a research participant belongs to a cate-
The purpose of this chapter is to discuss the gory pertaining to sex, race, religion, or political
process of data collection within a research study. affiliation),
It begins with a brief review of how issues of data • Ordinal, which determines the rank order of a
reliability are approached in the quantitative and variable (e.g., the rating, 1 ⫽ never, 2 ⫽ some-
qualitative traditions. Then, approaches to data times, 3 ⫽ frequently, 4 ⫽ always, is a rank order
collection are discussed. Next, the steps involved of frequency),
in research data collection are identified and • Interval (also described as continuous), which is
discussed. The chapter concludes with a discus- characterized by the assignment of numbers
sion of professional and ethical issues that involve along a continuum of less to more of a variable
the treatment of research participants during data divided into equal intervals,
collection. • Ratio (also described as continuous), which dif-
fer from interval data only in that there is a true
zero point at which none of what is being meas-
Qualitative and ured exists.
Quantitative Data Approach to Data Collection
Data can be either quantitative (i.e., numeric) or Ordinarily, in quantitative research the same data
qualitative (e.g., narrative) in nature. How investi- will be collected from all subjects, or there will be
gators think about and ensure the dependability a specific plan for more in-depth data collection
(i.e., reliability and validity) of data depend on depending on subjects’ responses or scores from
whether the data are being collected within the the first phase of data collection. For example, a
quantitative and/or qualitative traditions (Neuman, study may be structured so that only clients who
1994). Each tradition is report or demonstrate the
briefly considered below. presence of a trait will be
In the end, the dependability asked to engage in more
Quantitative Data of the research findings is in-depth data collection.
In quantitative research,
Collection linked to whether the data investigators will also
Quantitative approaches collected are reliable and undertake data collection
to data collection are valid. so as to minimize the
basically concerned with amount of missing data,
judgments of category or since statistical analyses
amount (Portney & Watkins, 2000). In quantitative are most rigorous when there are no or few miss-
data collection, variables of interest are assigned a ing data points.
numeric value that reflects the category or amount Quantitative data collection that focuses on
of that variable. Numeric quantities of one variable human behavior, thought, attitude, or emotions pri-
Chapter 32 Collecting Data 531
A Note on Terminology and refining and documenting the reliability and

validity of those instruments.
Considerable differences in terminology are
While it is sometimes necessary to construct
used in the research literature to refer to different
approaches and tools for data collection. In this instruments for research, a wide array of suitable
chapter, the term data collection procedures is instruments is increasingly available that had been
used to refer to any form of data collection previously developed and studied.
regardless of whether it is quantitative or qualita- When it is necessary to create a new instrument
tive. When discussing quantitative procedures for a study, the development of that instrument is,
for data collection, we use the generic term of itself, a substantial research undertaking.
instruments, and when referring to more specific Chapters 12 and 13 provide an introduction to the
kinds of instruments, we use the terms most issues and procedures involved in developing and
often associated with them (e.g., questionnaires, assessing a valid and reliable data collection
checklists, rating scales, measures). When dis-
instrument. Moreover, in many research institu-
cussing qualitative procedures for data collection
we refer to them as strategies. Unlike the quanti- tions there are centers or laboratories that provide
tative researchers who employ preselected instru- technical assistance to investigators who need to
ments to collect data, qualitative researchers develop new data collection instruments. Inves-
use approaches that are at least in part strategi- tigators often seek the resources of such entities
cally designed to fit the research context. These when developing an instrument.
strategic approaches are used within qualitative Concerns with the dependability of quantitative
research to generate optimal data given the data are centered on reliability and validity
natural conditions encountered in the setting and (Benson & Schell, 1997). Reliability of data seeks
to respond to emerging research questions that to ensure that the accuracy of information collected
develop as the research unfolds. It is recognized
was not unduly affected by any extraneous circum-
that qualitative researchers also use tools that
may be referred to as instruments. However, stances surrounding data collection. Reliability is
for purposes of clarity in this chapter, the term typically concerned with how accuracy may be
strategies is used to refer to all qualitative data affected by circumstances of data collection (i.e.,
collection procedures. who collects it, what kind of instrument is used to
collect it, and how and when it was collected).
Validity concerns in quantitative data collection
basically ask whether the data collected actually
marily uses structured methods of collecting data represent the variable under study. Empirical
such as observational rating scales, self-report assessment of validity focuses on such factors as
questionnaires, structured interviews, and stan- the extent to which items used to quantify a vari-
dardized tests. Within occupational therapy, quan- able coalesce together and whether the instru-
titative research also involves the use of functional ment’s scores converge with measures of variables
performance data collection procedures (e.g., that are theoretically related to the variable the
motor coordination, self-care, driver safety, and instrument intends to measure.
work capacity). These data collection procedures When collecting quantitative data for a study,
ordinarily require equipment or standardized tests investigators who wish to ensure the reliability and
for collecting data. Finally, quantitative research validity of their data ordinarily:
also can involve biometric data collection, which • Select data collection methods for which there
ranges from procedures such as measuring the are published reliability and validity findings rel-
kinematics of movement to imaging techniques evant to the intended study population, and/or
that capture brain activity. • Test the reliability and validity of the data collec-
tion procedure in their own investigation.
Methodological Rigor and
There is reason to question the reliability and
Dependability of Quantitative Data
validity of data collection procedures when:
In quantitative research, investigators carefully
• There is not sufficient previous research on the
choose and justify their data collection procedures
reliability and validity of the instrument,
when planning the research. Researchers either:
• The sample under study differs from those on
• Choose instruments that have previously been which the instrument has been studied,
developed and investigated, or • An aspect of how the instrument will be used
• Spend substantial time prior to or in the early varies from its standard use or how it has been
stages of the research developing instruments used in previous research, and
• A new data collection procedure has been devel- proceeds in an iterative fashion going back and
oped specifically for the study, for which relia- forth between analysis and data collection. In this
bility and validity are unknown. instance, the data collection process is guided by
the need for data that elaborates and looks for
In these cases, investigators will ordinarily:
counter instances of emerging findings.
• Test the reliability and/or validity prior to using Qualitative research is basically concerned with
an instrument to collect data in the study, or understanding and reporting on human behavior
• Collect data within the study that simultaneously and experience while at the same time being faith-
provide evidence pertinent to the reliability or ful to how subjects experience their own behavior.
validity of the instrument. In this instance the concerns for data dependability
are whether they faithfully represent circum-
For example, consider a data collection proce-
stances as they are experienced by those under
dure (e.g., administering a self-report measure by
study (Glaser & Strauss, 1967; Lincoln & Guba,
telephone) that has been shown to be reliable and
1985; Pelto & Pelto, 1978; Strauss & Corbin,
valid with an adult sample. An investigator who
1990). This is often referred to as the trustworthi-
wishes to use it in a study of adolescents must con-
ness of data. Consequently, the concerns for relia-
sider two questions:
bility and validity of qualitative data ordinarily
• Will the self-report measure provide valid meas- center on:
ures for an adolescent sample? • Do the data faithfully represent the phenomena
• Does administering the measure by telephone as experienced by the participants?
still yield reliable data in an adolescent sample • Do the participants believe that the findings accu-
or is there something unique about adolescents rately portray their problem, experience, or situ-
that would make telephone administration an ation?
unreliable data collection procedure in a given • Are the data comprehensive such that all existing
study? perspectives on the given problem, experience, or
Under these circumstances, an investigator may situation are represented?
do a pilot study to investigate: • Do the data accurately and completely reflect the
problem/experience/situation as it exists within a
• Whether adolescents give stable responses (e.g., given environmental context?
test–retest reliability), or
• Whether obtained measures correlate with These questions are best answered in the
another means of collecting the same informa- unfolding context of the qualitative study. Thus
tion (i.e., concurrent validity). qualitative researchers ordinarily monitor, choose,
and modify their data collection strategies as the
As the example illustrates, rigor in quantitative research process unfolds.
data collection emphasizes determining the As discussed in Chapter 20, there are a number
dependability of data collection procedures before of ways that qualitative researchers seek to ensure
or at the beginning of the research process. the reliability and validity of qualitative data.
These include but are not limited to:
• Engagement over time,
Methodological Rigor and • Reflexivity,
Trustworthiness of Qualitative Data • Triangulation, and
Within the qualitative tradition, data collection • Member checking.
tends to be strategic and emergent (Glaser & Most qualitative research involves an extended
Strauss, 1967; Lincoln & Guba, 1985; Pelto & period of engagement in the field of study. This not
Pelto, 1978; Strauss & Corbin, 1990). That is, the only allows the investigator to develop trust with
investigator ordinarily begins collecting data with participants but also helps ensure the investigator
the overall research question in mind and a general gains an insider perspective. In addition, pro-
plan of data collection. Then, as data are collected longed engagement allows the researcher to judge
and the questions and preliminary findings when data collection has saturated a question (i.e.,
emerge, the investigator tailors and refines the data when little or no new information or insights are
collection process accordingly. forthcoming).
Another important influence on data collection Reflexivity is a systematic process of self-
is the ongoing discovery of who, when, and where reflection that examines insights as well as emo-
are the best sources of data. Data collection often tions and reactions to people and events that occur
during data collection. In many instances the per- The Importance of Multiple Sources
sonal reactions of the researcher also become data. of Data in Research
Triangulation means using two or more strategies
to collect data on the same phenomena (e.g., com- It is becoming increasingly important to gather
bining observation with interviewing). Triangula- data on multiple independent and/or dependent
tion can also include using two or more data variables and to collect data at multiple time
gatherers who independently collect data on the points to enhance the explanatory power of
same phenomenon. Triangulation increases the research. For example, a study designed to help
likelihood that data capture all relevant features of clients achieve employment may collect data
the phenomena under study. Member checking is a on work status, activity level, and quality of life
over three time points following the intervention
process whereby the investigator returns to the
in order to demonstrate the broad effects of the
study participants to ask whether representations intervention and to illustrate that they are sus-
of the data accurately and adequately represent tained over time. Similarly, an intervention
their experience and understanding of the phenom- designed to improve participants’ use of assistive
ena. It is a means of ensuring that the investigator’s devices to maximize independent functioning
data adequately reflect an insider view. Unlike might document the individual and combined
quantitative investigators that seek to ensure the effects of initial training and ongoing support on
reliability and validity of data collection pro- improving overall quality of life, job functioning,
cedures a priori, qualitative researchers seek to social contacts, and physical activity level.
ensure these characteristics of data collection in Moreover, research increasingly emphasizes
the collection of functional, psychological, and
situ (i.e., as part of the ongoing process of data col-
social data along with biological data in order
lection). All of these approaches and others are to study the interaction of factors across the
typically used in qualitative research to ensure the biopsychosocial continuum. For example, an
trustworthiness of data. Within a particular study, occupational therapy researcher investigating an
the combination of approaches for ensuring trust- intervention to improve feeding behavior in chil-
worthiness will depend on the study question, the dren with severe developmental delays may aim
sample, and other contextual factors. to demonstrate that intervention improves the
participants’ emotional well-being and nutritional
status while reducing family caregiving burden.
Approaches to Data Collection
Data can be collected in a wide variety of ways. standing of an unfolding or ongoing behavior,
The most common forms of data collection are: process, or other situation in real-time. Obser-
• Observation, vation is most straightforward when there is tangi-
• Interviews, ble physical evidence, outcomes, or products that
• Self-report measures, can be seen or heard.
• Standardized tests and performance measures, In some circumstances, observational methods
• Contextual/environmental assessment, are used to corroborate data that have been col-
• Focus groups and town hall meetings, lected through other assessment methods, such as
• Biometric measures, and interviews or self-report measures. In other cases,
• Document/records review. observational methods are used when subjects lack
insight or self-evaluation skills or are not able to
While many of these methods are used in both participate in an interview or provide an accurate
quantitative and qualitative research, some are self-report (e.g., infants or individuals with severe
more or less exclusively used in one approach or cognitive limitations).
the other. Observational data collection methods are also
useful in providing direct information about a vari-
able under study that is not filtered through the
Observation perceptual lens of the person under observation.
Observational approaches to data collection are For example, in occupational therapy, home visits
suited to both qualitative and quantitative studies. to assess features of an individual’s physical envi-
When the aim of research is to answer questions ronment is a more reliable and valid means of
related to performance or behavior, observation is assessing the risk for falls than gathering this
frequently the data collection method of choice. information through self-report (Clemson,
Observational data also provide a richer under- Fitzgerald, Heard, & Cumming, 1999).
Types of Information Gathered that involve interaction between the subject(s)

Through Observation and the investigator.
The information typically sought through observa- Another advantage of recording is that play-
tion includes: backs can allow the researcher to view the data at
any speed and as many times as he or she desires.
• Subjects’ characteristics or affective states (e.g., Thus it is best used in circumstances where the
whether a person is sad, anxious, agitated, calm), question under study must be rated in a very
• Behavior (e.g., what a person does in a given sit- detailed manner and a given behavior must be
uation, how a person performs a task, level of slowed down or replayed to ensure that the
endurance to activity, or whether a person shows researcher has observed and understood all aspects
signs of restlessness, inattention, hyperactivity, of the behavior correctly.
fatigue, etc.),
• Communication (e.g., what a person says to Forms of Observation
others, how a person acts toward others, or what
a person expresses to others through gestures, There are two widely known types of observation,
facial expressions), and passive observation and participant observation.
• Environmental circumstances (e.g., objects and The first and most commonly utilized form of
their arrangement in space, social conditions, observation in quantitative research is passive
safety, task demands). observation. Passive observation involves observ-
ing subjects and recording data on the variables of
The observer ordinarily watches and/or listens interest with little to no interaction with the sub-
to participants, recording the information. Methods jects, in the interest of maintaining objectivity and
of recording information include: minimizing any biasing influence on what is being
• Observational guides (i.e., highly structured observed.
printed forms or booklets that provide probes or The second general form of observation is par-
codes for various topic areas and corresponding ticipant observation; it is commonly utilized in
space to record observations). These can be used qualitative research. When using participant obser-
for both qualitative and quantitative research. vation, the investigator joins the subjects and par-
• Structured checklists (i.e., paper-and-pencil ticipates in the same discussion or activities as the
forms used primarily to indicate the presence subjects. The aim of the participatory process is for
versus absence or frequency of certain states, the investigator to gain understanding of the phe-
behaviors, or communication). These yield cate- nomena under study as experienced by the partici-
gorical data most often used in quantitative pants (Rice & Ezzy, 1999).
research.
• Quantitative rating scales (i.e., forms that assign Role of the Observer
ordinal numerical scores to a number of items
In general, the observer’s role is to capture certain
designed to represent the variable under study).
details about the subject, discern important from
The most common is the Likert or Likert-type
unimportant observational data, interpret the
scale, which uses an ordinal rating technique in
observed data accurately and in light of the envi-
which qualitative statements are used to differen-
ronmental context, and validate observations over
tiate positions along a continuum (e.g., fre-
time. In studies that utilize observation, the role of
quently ⫽ 1, sometimes ⫽ 2, never ⫽ 3).
the observer also depends largely on whether the
• Semistructured or unstructured note taking (field
study is quantitative or qualitative. In quantitative
notes). This involves taking notes based on broad
research, the aim of the data collection is ordinar-
topics or thematic areas (semistructured) or
ily for the observer to:
based on spontaneous observations (unstruc-
tured). This approach is most common to quali- • Remain as objective as possible in gathering/
tative research, although it is used in quantitative recording the data to prevent any biases in data
research to provide supportive anecdotal data. interpretation or other personal expectations
• Electronic/digital recording of data: Data can be imposed on the data by the observer.
collected using audiotape or videotape recording. • Prevent artificiality or other changes in subjects’
Recording can occur in any stage of the research behavior due to the presence of the observer. In
depending on the aims and nature of the study. circumstances where the question under study
For example, recording can be used as an inves- does not involve interaction with the investigator,
tigator’s “third eye” to gather observational data the investigator is to remain as unobtrusive and
uninvolved as possible so as to avoid contaminat- community environment (e.g., in a grocery store or

ing the observational context or influencing the on public transportation).
behavior of the subjects. In some cases, this In the field of occupational therapy, observa-
involves observing through a one-way mirror or tional data are often collected in semistructured
from a location that is outside of a subject’s settings, such as within an inpatient or outpatient
vision or awareness. clinical setting. Unlike a subject’s natural environ-
• Take precautionary measures to ensure the relia- ment, a semistructured setting introduces the fol-
bility and validity involved in recording the data, lowing structures:
particularly if the nature of the observation
• Time (i.e., length of therapy session),
involves subtle changes in behavior, rapidly
• Space (i.e., size and configuration of the therapy
changing behavior, or some other highly detailed
space),
or nuanced aspect of behavior. Precautionary
• Objects (i.e., therapeutic equipment, assessment
measures include audio taping, videotaping,
tools, assistive technologies, arts and crafts, and
and/or corroborative rating checks by an inde-
other objects within the clinical setting),
pendent rater.
• Sensory variation (i.e., different lighting, sounds,
In qualitative research, the role of the observer smells), and
usually involves a sustained level of interaction • People (i.e., therapists, support staff, administra-
with participants over an extended period, during tors, other clients) that are artificial to a subject’s
which time the researcher(s) become increasingly natural environment.
familiar with the phenomena under study. As pre-
Standardized or laboratory contexts offer the
viously noted, prolonged involvement allows for
highest degree of control over confounding fac-
building trust with participants and developing
tors. Within occupational therapy research, a stan-
a more accurate, insider understanding of the
dardized context is typically created within a
group’s perceptions and experience. Alternatively,
staged or highly structured treatment room in a
a qualitative researcher may observe without being
clinical setting or within a standardized laboratory
a direct participant in or member of the population
space. Depending on the research question, any
under study. For example, such investigators may
variety of characteristics of these settings can be
sit in on a weekly support group or they may fol-
controlled, including room temperature, lighting,
low a group of subjects around in the field observ-
sound, contents, and space configuration. In addi-
ing activities and asking questions without
tion, standardized or laboratory contexts allow use
engaging in the same activities as the subjects.
of specialized measurement devices or test situa-
In large part, the comfort level of the subjects
tions.
will depend on the interpersonal skills of the
researcher and the research questions under study. Advantages and Disadvantages of
The interpersonal skills of researchers may be Different Observational Contexts
reflected in the roles they take within the group,
the kinds of questions they ask, the tone of voice The primary advantage of collecting data within
and body language used, the comments they make a natural environment is the ability to gather
in response to obtaining information, and the data that are authentic and ecologically valid.
nature and phrasing of the questions to which they Natural environments do not introduce extraneous
seek answers. variables that might otherwise be imposed by an
artificial laboratory or clinical environment. Semi-
Observational Context structured and structured settings can raise ques-
tions of ecological validity. However, unlike a
Observation can take place in a number of contexts highly structured laboratory setting, a semistruc-
including: tured clinical context may be construed in such a
way as to simulate enough of a subject’s natural
• Natural contexts,
environment to increase ecological validity. For
• Semistructured (e.g., clinical) contexts, and
example, a researcher seeking to answer an obser-
• Standardized or laboratory contexts.
vational research question pertaining to environ-
In its purest form, qualitative research takes mental impact on attentional problems in persons
place within the natural environment or context in with schizophrenia might compare the subject’s
which subjects live and function. Natural contexts attention within three semistructured clinical con-
may include, but are not limited to, a subject’s texts that simulate aspects of everyday settings.
home, workplace, neighborhood, and/or general First the researcher might take the subject into the
dining room of the hospital during lunchtime and • Psychosocial information (e.g., available social
test the subject’s ability to follow a conversation in support, other resources and coping abilities,
a highly stimulating environment. Then the available sources of assistance within a subject’s
researcher might take the subject into the waiting social network, sources of stress or conflict
room of the outpatient rehabilitation clinic to test within a subject’s social network),
the subject’s ability to follow a conversation in a • Information about a subject’s physical living
moderately stimulating environment. Finally, the environment (e.g., safety and accessibility within
research might test the subject’s ability to follow a home, work, and community environments), and
conversation in a private therapy room. • Employment information (e.g., work history, cur-
Distinct from variation that would be inherent rent work status, work performance issues, need
in observing subjects in their natural settings, for and access to reasonable accommodation).
semistructured settings can be applied uniformly
across subjects. However, in contrast to a labora- Methods of Recording Interview Data
tory setting in which even more control can be
imposed over the level of environmental stimula- There are three general methods of interviewing:
tion, semistructured environments still introduce • Structured interviews,
certain risks for confounding the research ques- • Semistructured interviews, and
tion. The obvious advantage of standardization is • Unstructured interviews.
that it allows the researcher to control the environ-
ment across subjects. The disadvantages include Structured interviews are comprised of a set of
limitations to the generalizability to the subject’s preestablished questions that follow strict adminis-
natural environment and elicitation of responses to tration and scoring rules. They are used in quanti-
the artificiality of the environment. tative research and mostly gather ordinal and
In deciding the level of control over observa- nominal data. In many cases, the scoring of struc-
tional context, a researcher must consider a number tured interviews follows a very rigid and well-
of variables. These include, but are not limited to, defined set of rules or template. In many instances
the nature of the research question and the vulner- these interviews ask subjects to select from among
ability of the research subjects to the influence of responses provided by the interviewer. Questions
environmental variation on their behavior. Finally, that ask interviewees to give an open-ended
the researcher must consider the feasibility of con- response are generally focused and the interview
ducting the observation within different types of either records or codes the response using a stand-
environments. ard coding scheme. Structured interviews may
contain skip patterns within it that tell an inter-
viewer that he or she may skip certain questions
Interviews (Structured and based on a subject’s responses to prior questions.
Semistructured) They also may contain allowable probing ques-
Interviews typically allow a researcher to collect tions or explanations that interviewers can use
information that leads to a broader, more holistic, when subjects do not respond accurately or fully or
or more integrative view of a subject’s impairment when they do not understand a question.
or life situation. In occupational therapy research, The high level of structure is designed to mini-
interviews are most commonly utilized to obtain mize interviewer bias. The structured interview
the following types of information: also helps eliminate inaccuracies in scoring or sub-
jective interpretation of responses on the part of the
• Sociodemographic and sociocultural information interviewer. Other advantages of structured inter-
about subjects (e.g., age, ethnic identification, views include their ease of administration (particu-
educational status, annual income, and extent to larly for beginning therapists or researchers), their
which culturally diverse clients identify with and time-limited nature, and their ease in scoring.
practice health-related beliefs and behaviors that Semistructured interviews may be used in
are related to their culture of origin), quantitative research to generate data that is later
• Historical information (e.g., health history, his- coded or categorized. They are also used to gener-
tory of events leading up to the impairment), ate more narrative accounts for qualitative
• Information about a subject’s experience of his or research. These interviews use a preestablished
her current impairment and its functional conse- schedule of open-ended questions, but allow con-
quences (this may include information about a siderable flexibility in how they are administered.
subject’s volition, habits, roles, and performance Semistructured interviews typically also allow
capacity), interviewers to tailor questions and probes in order
to obtain more in depth and trustworthy informa- When interviews are used in occupational ther-
tion. When administering a semistructured inter- apy, the most common source of interview data is
view, an interviewer may pursue questioning in the subject him/herself. During an interview, a
a related area to obtain a different perspective or subject is required to respond to questions based
to shed light on the subject’s responses to the on some degree of reflection about his or her expe-
interview questions at hand. Semistructured inter- rience and/or needs. All interviews require sub-
views require a higher level of clinical judgment, jects to have the ability to reflect honestly upon
interpersonal skill, knowledge, and expertise on their impairments and experiences with a reason-
the part of the interviewer. Because scoring or able level of accuracy. Depending on the variables
coding rules are much less rigid and, in qualita- of interest in a research study, participating in an
tive research not predefined, interviewers must interview may require varying degrees of self-
decide: awareness or insight.
Depending on the nature of the interview and
• Whether a respondent’s answer to each item is
the variables under study, behavioral information
accurate, detailed, and comprehensive enough,
may also be generated during an interview. In
• What additional information is needed,
some cases, it may be factored into the overall out-
• What kinds of questions are required to probe for
come or score of the interview. Nonverbal infor-
that information, and
mation may include subjects’ behavioral and
• How to limit tangential or overly lengthy
affective responses to interview questions, their
responses.
facility in processing auditory information, and
The advantages of semistructured interviews communication/interpersonal skills.
are that they allow for better rapport-building In many circumstances, a researcher may wish
and more detailed and in-depth understanding of to obtain information about a subject through
the variables of interest. The disadvantages can reports from significant others, family, friends, or
include length of administration and greater vul- coworkers. Data provided on behalf of a subject by
nerability to interviewer bias. others who are close to the subject may be used to
Unstructured interviews are used in qualitative corroborate information provided by a subject or
research. These interviews may be guided by only to fill in informational gaps within the subject’s
a general topic or short list of topics that the inter- self-report. In some cases, it is not possible to
viewer pursues. Alternatively, the content of the obtain self-reported interview information from
interview may depend on an issue that is raised by a subject directly owing to impairment-related
a participant or a recent observation. While inves- issues. Under these circumstances, interview data
tigators conducting this type of interview are usu- from proxy sources may be helpful.
ally guided by a broad study question, they remain
open to new topics that may emerge in the inter- Role of the Interviewer
view itself.
Unstructured interviews have the advantage of Establishing rapport is fundamental to every inter-
being able to discover new information and estab- view. Depending on the preferences and reactions
lish a sense of trust (that the investigator really of the subject, achieving rapport can be a relatively
wants to understand and hear the participant’s per- straightforward process, or it can be rather lengthy
spective). They require substantial time, interview- and complex. Some individuals will respond well
ing skill, and contextual knowledge. Since, to a brief period of introductions and small talk
however, unstructured interviews ordinarily take before an explanation of interview procedures is
place as part of an ongoing qualitative study in provided. Others will prefer that the researcher
which the investigator is also a participant assume a more professional stance and explain the
observer, they benefit from the background knowl- procedures upfront without preliminary chatter.
edge of the interviewer. The interviewer’s first role is to make his or her
best estimate of the interpersonal preferences of
Sources of Interview Data any given subject and act accordingly.
Because there is a potential for the researcher to
In every interview, there are three possible sources be viewed as an authority figure in any interview
of data: situation, it is important to know or predict how the
• Verbal data provided by the subject/respondent, subject might respond given the automatic power
• Behavioral data, and differential involved in an interview situation.
• Proxy verbal data provided by significant others, Some subjects will feel uncomfortable provid-
family, friends, and coworkers. ing difficult or intimate information to a relative
stranger. In such circumstances the researcher must structs such as personality characteristics (e.g.,
do everything possible to ensure confidentiality and degree of assertiveness in relationships), attitudes
to create an atmosphere of unconditional accept- (e.g., how much value one attaches to leisure or
ance and positive regard. work), emotional states (whether one is depressed
Although the role of an interviewer is clear with or anxious), and behavior patterns (to what extent
respect to the nature of the task, the need to estab- exercise is part of an individual’s daily routine).
lish rapport, and the power differential involved, They can also ask an individual to evaluate him-
interviewers can differ in significant ways in terms or herself more directly in terms of his or her per-
of their more nuanced behavior and roles. For formance capacity (e.g., how competent one is at
example, an interviewer’s role may vary according performing a task). The most common form of a
to whether the research is quantitative or qualita- unidimensional rating scale is the Likert scale. It
tive. During an interview that seeks to obtain quan- most frequently uses a five-category ordinal rating
titative data, an interviewer may assume a more technique in which qualitative statements are used
formal role. The interviewer may ask the subject to differentiate positions on a continuum. One
to select an answer among a limited number of example of a commonly utilized self-report meas-
options, set more limits on side conversations, ure in occupational therapy research that utilizes
and/or discourage the interviewee from providing Likert scaling is the Medical Outcomes Survey
unnecessary details or extraneous information Short-Form 36 (SF-36). The eight subscales of this
when answering open-ended questions. Quali- measure each contain Likert scale items that are
tative interviews require the researcher to assume a designed to assess self-reported health-related
less structured role. quality of life and functional impairment. In addi-
tion to instruments that utilize Likert scales,
Self-Report Measures another type of self-report instrument is the seman-
tic differential (Polit & Hungler, 1999).
Self-report measures are written instruments on
which subjects are asked to record information. Semantic Differential, Q-Sort,
They typically ask the subject to self-reflect on and Visual Analogue Scales
his or her experience or needs and select the
best option from a finite number of categories The semantic differential asks the respondent to
or to provide an open-ended explanation as a rate a given concept on a series of bipolar adjec-
response. Self-report measures are typically self- tives that are used to characterize one’s reaction or
administered by subjects and responses are usually feelings (e.g., free versus constrained, dull versus
provided in writing. However, under certain cir- exciting). The Q-sort is a self-report method that
cumstances (e.g., subjects that require accommoda- encourages respondents to organize data into
tions) subjects may provide verbal responses that visual categories (e.g., adjectives written on index
the researcher records. cards are sorted into piles). During the Q-sort pro-
When asking for factual information, self- cedure, respondents are expected to sort the visual
reports may ask a subject to: data into piles that represent meaningful cate-
gories. Visual analogue scales employ a straight
• Report demographic characteristics (e.g., age, line with labels to anchor each end. Subjects are
sex, race), then asked to mark the point on the line that corre-
• Rate the severity or frequency of certain symp- sponds most closely to their experience. Visual
toms or impairments (e.g., “I am able to walk a analogue scales typically employ a line that is 100
flight of stairs with no pain” never, sometimes, mm in length so that scoring can be accomplished
frequently, always), with use of a standard ruler (Polit & Hungler,
• Respond to open-ended questions (e.g., what one 1999). Semantic differentials, Q-sorts, and visual
typically does as a specific time of the day, or analogue scales are typically used in the assess-
what types of difficulties one has performing a ment of subjective experiences or less tangible
given task), or phenomena that are sometimes difficult to describe
• Respond to dichotomous questions (e.g., yes–no verbally, such as pain or fatigue.
questions as to whether one can or does perform
a given task). Unstandardized Questionnaires
No data exist on the reliability and validity of
Self-Rating Scales
unstandardized questionnaires. They are typically
Some self-reports involve completing self-rating created by investigators for their own use in pre-
scales. Self-rating scales are used to capture con- liminary studies to gather wide-ranging informa-
tion about a novel variable or to gather information and computer aptitude. There are also a number of
about a novel population. Researchers are more considerations regarding the confidentiality and
likely to use unstandardized questionnaires when overall security of the data that must be taken into
existing measures do not address the variables of account.
interest and a new measure must be generated. For Portable approaches to computer-based admin-
example, an occupational therapy researcher inter- istration can be used to gather repeated, self-report
ested in gathering general information about prac- measurement. Subjects are typically provided with
titioners’ attitudes and knowledge about treating a a handheld personal computer or a small data
new type of disability in practice might wish to recording device. They can be programmed to cue
administer a preliminary questionnaire that con- (e.g., beep) an individual to provide his or her self-
tains items that assess a broad range of variables. report at various times throughout the day, week,
The benefit of unstandardized questionnaires or month. Studies that aim to measure an outcome
includes their potential to provide rough prelimi- variable that is subject to change periodically
nary information about a wide range of variables. throughout the day typically use this approach. For
Because of the need for standardized instruments example, if an investigator wished to measure the
to contain items that cohere with one another and effects of overall activity levels on the subjective
reflect a general construct or constructs, reliable experience of pain, subjects might be provided
and valid questionnaires tend to be more limited in with a handheld device that would cue the person
their breadth and scope. Despite this, unstandard- periodically (e.g., at fixed or random time points)
ized questionnaires carry a number of limitations. throughout the day to answer questions about what
Generally, their accepted use in research is limited the subject is currently doing and to rate pain level
to descriptive studies that are preliminary in nature on a visual analogue scale.
and aim to report general information about a sin-
gle sample. They are less frequently used in exper- Standardized Tests and
imental and clinical outcome studies.
Performance Measures
Standardized tests involve contrived cognitive or
Administration of Self-Reports
motor tasks that are administered and scored under
The most common method of administration of a strictly standardized conditions and typically
self-report measure is to provide individuals with a generate norm-referenced or criterion-referenced
written form. Forms may be given to a subject to scores. Examples of standardized tests include
be filled out in the presence of the investigator or intelligence and aptitude tests, tests of motor pro-
to be completed at the respondent’s discretion and ficiency, and cognitive performance tests. A sub-
returned later. In addition, forms may be mailed to stantial amount of research goes into developing
individuals for responses. Instructions for com- these tests.
pleting the form may be given verbally or provided Performance measures commonly involve
in writing on the form itself. When self-reports are everyday tasks (although they may be somewhat
mailed to respondents, the instructions are typi- standardized) that are observed to allow re-
cally provided in writing. searchers to measure performance in the task. For
Increasingly, self-reports are being adminis- example, the Assessment of Motor and Process
tered using computer-based technologies. For Skills (Fisher, 1993) is used to observe clients in
example, researchers who work with relatively selected activities of daily living (ADLs) and
large samples of subjects are increasingly posting instrumental activities of daily living (IADLs). It
and administering self-report measures online via measures the quality of an individual’s perform-
the World Wide Web. An advantage to online ance (i.e., motor and process skills).
administration is that it can be organized so that One advantage of standardized tests and per-
subjects’ responses are automatically downloaded formance measures is that they are widely used
and scored using a data entry and scoring program. and widely accepted as rigorous measures of out-
Depending on sample size, online administration comes. Because they are widely used, they facili-
may save costs that would otherwise be incurred tate comparison of any investigator’s findings from
through printing and postage. However, one must a given study with those of another investigator
weigh the overall cost of the computer software from a different study. Disadvantages of standard-
and programming required to develop, post, pro- ized tests and performance measures are that they
tect, and manage the survey and the data records of are sometimes too general or broad to answer the
each of the respondents. This approach requires more detailed questions that a researcher may
that subjects in the study have computer access have. In addition, not all are cross-cultural in
nature and may not be relevant to all sociodemo- Focus groups ordinarily include between 5 and
graphic and sociocultural groups. 15 participants; the aim of the group size is to
achieve a balance between ensuring that all mem-
bers have an opportunity to share their views while
Administration of Standardized Tests including enough members to represent the diver-
and Performance Measures sity of existing viewpoints (Krueger & Casey,
Standardized tests are typically administered 2000). Focus groups are used to explore people’s
under standardized conditions that are consistent perceptions and attitudes regarding topics in which
with those that were set when the test was initially the participants have some investment or stake
developed. Depending on the nature of the test, (Bernard, 1994; Krueger & Casey, 2000; Morgan
these conditions might involve having a subject sit & Spanish, 1984; Nabors, Ramos, & Weist, 2001).
at a desk within a research office to respond to a The advantages of focus groups are that data
written questionnaire or having a subject perform can be collected from several people at once and
certain behaviors or tasks in a laboratory or clini- that the interaction between members can stimu-
cal area using standard equipment. late data that the investigator might not otherwise
have gained. One of the main reasons for using
focus groups is to obtain the kind of data that
Contextual/Environmental Assessment emerge when participants interact with and modify
In occupational therapy, contextual and environ- each other’s responses. One disadvantage is that
mental assessments typically measure aspects of some persons, especially those with minority opin-
physical, social, educational, or work-related set- ions, might be discouraged from sharing their
tings in which subjects perform daily life occupa- views. Therefore the researcher aims to create a
tions. An example of a contextual assessment that milieu that encourages participants to share their
measures the extent to which an individual’s phys- perceptions and views without the need for overall
ical environment facilitates or thwarts his or her group consensus (Krueger & Casey, 2000).
occupational adaptation is The School Setting Within a single study, researchers may replicate
Interview (Hoffman, Hemmingsson, & Kielhofner, the focus group with different sets of participants
2005), a semistructured interview that measures who represent a particular constituency. Repli-
the extent to which all of a child’s educational cation of focus groups representing a particular
environments support engagement and participa- constituency can reveal common themes, trends,
tion in learning activities. Similarly, the Work and patterns (Krueger & Casey, 2000). Inves-
Environment Impact Scale (WEIS) (Moore- tigators will also sometimes conduct different
Corner, Kielhofner, & Olson, 1998) is a semistruc- focus groups composed of persons who represent
tured interview scale that evaluates features of an different constituencies in a setting (e.g., a focus
individual’s work environment as they support or group composed of staff and a focus group com-
interfere with job functioning. posed of clients in a healthcare setting). In this
Occupational therapy researchers who incorpo- instance the focus groups are designed to empha-
rate qualitative strategies may focus on the social size differences in the perceptions of different con-
and cultural aspects of a subject’s occupational stituencies.
context. For example, a researcher interested in the Focus groups are increasingly used as a method
daily routines of homeless individuals with mental of data collection in both qualitative and quantita-
illness might observe and document the effects of tive research. In quantitative research, focus
a number of contextual and environmental vari- groups often serve as a first stage of research or as
ables. a pilot study to ensure that later data collection
procedures will focus on relevant questions, reflect
the perspectives, and be understandable to sub-
Focus Groups and Town Hall Meetings jects. In qualitative research focus groups can be
A focus group is a group discussion conducted by the only or a primary means of data collection.
an investigator who serves as a moderator, guiding Focus groups are increasingly used to develop
the discussion by introducing questions, usually and evaluate health-related interventions and pro-
from a written set of questions or topics. Ordinar- grams (Heary & Hennessy, 2002; Hildebrandt,
ily, data are recorded in the form of: 1999). The following is an example. Ivanoff
(2002) used focus groups to develop an occupa-
• Audiotapes that are then transcribed, and tion-based health education program for adults
• Notes taken by the moderator or another investi- with macular degeneration. Focus groups were
gator whose role is to record information. conducted to gain an insider perspective on how
vision problems affected their daily lives. The records; insurance records; or job performance
focus group revealed both these elders’ insecurities records maintained by an individual’s employer.
regarding their daily occupations and the strategies Within health care, access to historical information
they employed to be able to perform their occupa- of this nature, particularly without direct consent
tions. Findings generated from the focus groups from the subject, is becoming increasingly difficult
informed the development of the intervention. because of the necessity to protect the confiden-
Post-intervention focus groups were used to evalu- tiality and rights of the subjects on whom the doc-
ate the program and to generate ideas for further umentation or records are based.
program development.
Biometry/Physiological Measures Planning and Implementing

Biometry and physiological measures are objec- Data Collection
tive means of assessing any range of variables that
involve physical, biological, or physiological func- The next section of this chapter reviews three steps
tioning. Examples of biometric measures of phys- generally taken by researchers to plan and imple-
ical functioning that are commonly used by ment data collection. These include:
occupational therapy researchers include measures • Selecting instruments and procedures for data
of grip strength (e.g., dynamometer), endurance collection,
(e.g., how long an individual can sustain a motor • Developing a data collection plan, and
movement), and range of motion (e.g., joint range • Selecting and preparing personnel for data col-
of motion specific to seating). Increasingly, occu- lection.
pational therapy researchers are beginning to con-
duct interdisciplinary and translational studies in Each of these steps is discussed in detail below.
which they collaborate with researchers from other The feature box titled “A Checklist for Preparing
disciplines to measure relationships between occu- for Data Collection” contains a checklist that can
pationally based variables and other biological or be used by researchers to facilitate self-evaluation
physiological variables, such as immune function, of their selected approach to data collection.
physical fitness, and/or cardiovascular functioning.
Selecting Instruments and Strategies
Document/Records Review for Data Collection
In many cases an important source of research data
One of the most important steps in the data collec-
is preexisting documents or records. In health care,
tion process involves choosing the appropriate
an important source of information is the medical
instrument and/or strategy for data collection. This
record. Medical records can be used to extract a
choice has implications for the ease and efficiency
wide range of information that might be important
of data collection and the quality of the data that
to a researcher, including, but not limited to:
are ultimately collected.
• Sociodemographic information, Generally, identification of data collection pro-
• Medical and psychiatric diagnoses, cedures requires careful and sometimes extensive
• Access to health insurance, investigation. Some useful strategies are:
• Number, reason, and nature of contacts with
• Examining the research literature and attending
medical care providers,
conference presentations to identify strategies
• Prior treatment plans,
and instruments being used by other investigators
• Treatments prescribed, provided, or recom-
in one’s topic area.
mended, and
• Corresponding with other researchers about the
• Treatment follow-through and outcomes.
state-of-the art strategies and instruments being
In addition to medical records, qualitative used in a given topic area. This allows one to have
researchers often use a range of documents as a more extensive dialogue about the strengths,
sources of data. These may include written com- limitations, and receptivity of a given procedure
munication (memos, notices, letters); artwork; or instrument. It also allows an investigator to
diaries that contain information about daily activi- ask other researchers about details such as logis-
ties, symptoms, or other information relevant to tical considerations in administration, preferred
occupational performance; brochures; policy and approaches to scoring, and issues involving the
procedure documents; educational records; legal instrument’s sensitivity to change.
A Checklist for Preparing for Data Collection
I) Has the appropriate data collection procedure • Who will be collecting which types of data?
been selected? ____ Who will be administering each of the instru-
(Check off only if criteria A–E are endorsed.) ments?
A) Is the information gathered relevant and suf- • What training and qualifications will data col-
ficient to answer the study question? ___ lectors need and by when will they be expected
B) Is the data collection approach methodologi- to have completed this training?
cally rigorous enough to adequately address • At what time point in the study will data col-
the study question? ___ lection be initiated for each instrument or type
C) Have all relevant logistical variables related of data that will need to be collected?
to the overall study design, available • What are the deadlines for completion of data
resources, and subject characteristics been collection for each data collection time point,
accounted for? ___ instrument, or procedure?
D) Are norms or other criteria for interpreting • How will data be identified and coded for data
the instrument needed and available? ___ entry? Who will enter the data, how will it be
E) Is the type of data that the instrument yields entered, and by when will it be entered? What
appropriate for the study question and will it program will be used for data entry?
need to be normalized or transformed? ___ • Who will analyze the data and how will they
II) Has a data collection plan or protocol been be analyzed?
developed? ___ • How will data be disseminated? To what audi-
(Check only if all of the following questions ences?
have been answered.) III) Are the personnel involved in data collection
• What data are to be collected? adequately trained and prepared? ___
• Who will obtain the necessary equipment, IV) Have all relevant professional and ethical con-
exam space, and instruments for data collec- siderations been met? ___
tion and by when will they be obtained?
• Consulting Web-based and published compila- • Extent to which the instrument or strategy con-
tions of assessments (see Resources at end of forms to standards of methodological rigor,
chapter). • Logistical considerations related to the overall
study design, available resources, and subject
When a tradition of research exists on a topic,
characteristics,
there are often one or more quantitative instru-
• Availability of norms or criteria for interpreting
ments that are considered gold standards for col-
the information gathered, and
lecting data on certain variables. For example, in
• Type of data the instrument or strategy yields.
rehabilitation-related outcome studies of health-
related quality of life, the Medical Outcomes Relevance and Sufficiency of Data. In selecting
Study Short-Form Survey (SF-36) (Ware & an overall approach to data collection, the first con-
Sherbourne, 1992) is frequently the measure of sideration involves whether the approach is rele-
choice, but the Quality of Life Index (Ferrans & vant to the research question on which the study is
Powers, 1992) is an equally valid and reliable based. For example, an observational approach to
instrument that measures somewhat distinct data collection should be used when one is study-
dimensions of health-related quality of life. ing a question that involves a concretely observable
phenomenon or behavior. If the research question
Criteria for Selecting Instruments involves a variable that is highly complex or diffi-
and Strategies cult to observe self-report or interview may be the
correct approach. In the case of a complex research
In choosing one instrument or strategy over
question or variable, a self-report approach may be
another, the following considerations have been
combined with an observational approach to trian-
identified as important in guiding the selection of
gulate measurement.
one’s means of data collection (Depoy & Gitlin,
A second important consideration when select-
1998):
ing an approach to data collection is whether the
• Relevance and sufficiency of the information instrument or strategy represents an accurate
gathered to answer the study question, reflection of the research question. Consider, for
example, investigators studying the outcomes of given to what strategies or instruments are most
an occupational therapy program designed to likely to give reliable and valid data given the par-
improve fine-motor skills in children or one ticular phenomena to be studied and who the study
designed to improve communication/interaction participants will be. Sometimes preliminary pilot
skills in adults. These investigators should select testing of strategies or instruments can give useful
sound instruments that specifically measure the information about what are likely to be the best
dependent variables (i.e., fine motor skills and data collection strategies in a given context.
communication/interaction skills) identified in the
Logistical Considerations. In selecting appropri-
research questions.
ate instruments and strategies for data collection,
In addition to these considerations, the way in
researchers must also consider logistical elements
which an investigator approaches the questions of
of the study. The first logistical consideration is
relevance and sufficiency of data collection
what the planned strategies and/or instruments
approaches depends on the methodology of the
require of participants in terms of their mental and
study. In quantitative studies, the primary focus of
physical performance capacity, personal time, and
relevance is whether the method of data collection
level of effort. One should also consider the impact
provides accurate data that reflect the variable(s)
of any strategies or instruments on their health and
under investigation and is suitable to the antici-
well-being. Participants should be asked to do only
pated sample. In qualitative studies, one must
what is within their capacity and absolutely neces-
ensure that the method of data collection is likely
sary to obtain necessary data. For ethical reasons,
to generate the right kind of data of sufficient qual-
any inconvenience, stress, or risk involved in data
ity, quantity, and detail to address the central ques-
collection must be clearly outweighed by the
tion under study. As noted earlier, this process of
importance of the research question.
planning the data collection strategy begins when
A second logistical consideration involves the
the study is being planned but generally continues
resources that a given data collection procedure
through its implementation in qualitative research.
would require. Human resources (i.e., time and
Methodological Rigor. The methodological rigor effort of the investigator or other study personnel)
of data collection depends on whether there is evi- required by data collection procedures is the first
dence in the literature that a quantitative instru- major resource that must be considered. The antic-
ment is valid and reliable or that a qualitative ipated number and length of different collection
strategy is trustworthy. Researchers should ask instruments or strategies should be examined in
themselves four questions in evaluating whether a light of sample constraints (e.g., the schedules,
given instrument or strategy to data collection is locations, and availability of participants).
indicated. Economic resources are also important when
selecting an appropriate data collection procedure.
• What does this evidence say about the reliability
Considerations include the level of education or
and validity of the instrument or about the trust-
specialized training data collectors need since they
worthiness of a strategy?
have salary and cost implications. Moreover, the
• Does research evidence of reliability, validity, or
availability or cost of instruments, test kits and
trustworthiness of the instrument/strategy have
forms, scoring and data management and analysis
relevance for the proposed study sample? Are
software packages, necessary space, computers,
there any reasons to expect this sample may
telephones, and other equipment should be consid-
respond differently to the procedure than those
ered. If a study will involve data collection within
with which the procedure has been studied?
the field, transportation resources and costs must
• If there is not sufficient evidence for dependabil-
also be considered.
ity of the procedure with the proposed study sam-
A final logistical consideration in data collec-
ple, then is there some precedent in the literature
tion involves the appropriateness of the data col-
for successful use of the instrument or strategy
lection procedures for any unique characteristics
with the proposed study population?
of the sample and the subject burden posed by the
• Are there any logistical aspects of the study that
data collection procedures. A particular concern in
would preclude recommended administration of
occupational therapy research is whether the
an instrument or strategy?
intended participants have impairments that affect
One should examine what strategies or instru- their ability to participate in any of the data collec-
ments have been successfully used by other tion methods to be utilized in the study. In some
researchers studying the same or similar research cases, the investigator may resolve this issue by
questions or populations. Consideration should be removing barriers (e.g., providing a sign language
interpreter or a written instrument in interview for- there will be cutoff points assigned to scores that
mat, on audiotape, or in Braille). In other cases the may indicate the presence of a problem or the
investigator will select a data collection method point beyond which a person will be incapable
that does not require skills affected by the impair- of some criterion (e.g., independent living).
ment. Instruments based on item response theory have
Subject burden refers to the effort, inconven- built-in criteria as the items serve to represent a
ience, pain, risk, and other factors that affect what hierarchy of capacities, skills, attributes, etc. For
will be required for or may be a consequence of example, the Volitional Questionnaire (de las
data collection procedures. Subject burden in- Heras, Geist, Kielhofner, & Li, 2003) measures
creases with the time involved in data collection, motivation for occupation and it can indicate the
the number of data collection instruments utilized, level of motivation as indexed by items that range
and the negative effects of any data collection pro- from low to high motivation. A person at a basic
cedure on a participant’s emotional or physical exploratory level shows interest but does not seek
well-being. out challenges. The highest level of interest would
If subject burden is unduly high in a study, it be labeled achievement motivation behavior. A
will lead to low participation rates and high dropout person’s score on this instrument indicates, then,
rates. In turn, this will limit sample size and intro- which of the items on the instrument are below and
duce sampling bias, which limits the degree to which are above the person’s level of motivation.
which the sample is representative of all individu- This kind of criterion referencing is very helpful in
als in the population under study. Researchers must interpreting a score.
constantly balance the need to obtain adequate, In qualitative research, norms and criterion
detailed, and comprehensive information against referencing are not specific considerations. How-
human and other costs involved in actually obtain- ever, investigators will be interested to know what
ing that information. information has been generated on the phenomena
under study so that the data that emerge from their
Availability of Norms or Criteria for Inter- data collection strategies can be assessed in terms
pretation. A consideration in selecting an appro- of whether it is new information or contradicts or
priate quantitative instrument for data collection confirms previously reported findings.
involves whether norm-referenced scores are avail-
able for a given instrument. For many research Type of Data the Instrument Yields. The final
questions, the availability of norms for a particular consideration in selecting an approach to data col-
instrument may not be required. For example, if a lection involves examining the type of data that an
researcher wishes to examine attitudes among instrument or strategy yields. By definition, quan-
occupational therapists regarding the importance titative instruments yield numeric data and qualita-
of therapeutic use of self to occupational therapy tive instruments typically yield written descriptive
outcomes, he or she does not need to utilize an data that can later be coded or otherwise catego-
instrument with norm-referenced scores. Alterna- rized. In quantitative studies, researchers need to
tively, if the same researcher wishes to examine the consider whether the type of numeric output is
extent of impairment in physical functioning sufficient for the kinds of analyses that will be
among adults with cardiovascular disease over needed to answer the research question (e.g., scale
time, he or she may wish to compare subjects’ of measurement, sensitivity of the instrument
baseline physical functioning scores against so that its scores will detect change). Quantitative
the instrument’s available norms for individuals researchers also need to consider whether the type
with cardiovascular disease on the physical func- of numeric output yielded by a given instrument
tioning domain. This would yield added informa- will require transformation to make it consistent
tion about the comparability of the baseline with the numeric output yielded by other instru-
physical functioning of the sample against national ments used in the study.
norms. If physical functioning findings are compa- Qualitative investigators will be concerned with
rable, findings from the researcher’s study are whether strategies are likely to provide compre-
more likely to have greater generalizability to the hensive data that will lead to saturation. For exam-
larger population of individuals with cardiovascu- ple, if an investigator is interested in how children
lar disease. experience certain types of sensory problems,
In other instances investigators may wish to use observation alone would likely be insufficient to
instruments that are criterion referenced. As dis- answer the question. Thus, interviews with the
cussed in Chapters 12 and 13, there are different children would also be required. Moreover, since
approaches to criterion referencing. In some cases children may not be as articulate as desired in
discussing the details of experience, the investiga- Selecting and Preparing Personnel
tor might want to video the child, select certain for Data Collection
scenes to view with the child, and then discuss in
order to obtain richer data. Or, the researcher may In selecting and preparing personnel for data col-
decide to do interviewing as the child is engaged in lection, a number of considerations need to be
motor behavior to discuss the experience as it taken into account. For example, in some studies
occurs. the quality of data collection is higher when data
collectors share as many characteristics in common
Developing a Data Collection with study participants as possible. Depending on
Plan or Protocol the research question and the degree of interper-
sonal interaction required in a given study, similar-
After evaluating all of the considerations involved ities in language, age, racial or ethnic background,
in selecting the appropriate procedures and instru- and disability status can facilitate rapport and trust
ments for data collection, a researcher must during data collection procedures. Other character-
develop a data collection plan or protocol. This can istics that will enhance data collection are good
often be incorporated into an overall study time- clinical judgment, interpersonal skills, observa-
line. This plan should answer the following ques- tional skills, and the ability to grasp and under-
tions: stand the theoretical foundations for the data
collection procedures.
• What data are to be collected?
In addition to these more fundamental qualities,
• Who will obtain the necessary equipment, exam
data collectors should have adequate training and
space, and instruments for data collection and by
experience required for the data collection proce-
when will they be obtained?
dure. This may include generic training and more
• Who will be collecting which types of data? Who
specialized training associated with the particular
will be administering each of the instruments?
procedure. In most instances study-specific train-
• What training and qualifications will data collec-
ing and ongoing supervision of data collectors will
tors need and by when will they be expected to
be necessary. These measures help ensure that the
have completed this training?
particular approach to data collection is fully
• At what time point in the study will data collec-
understood and implemented correctly. They also
tion be initiated for each instrument or type of
are useful to ensure that data collectors can relate
data that will need to be collected?
well with the study population.
• What are the deadlines for completion of data
collection for each data collection time point,
instrument, or procedure? Professional and Ethical
• How will data be identified and coded for data
entry? Who will enter the data, how will it be Considerations in Data
entered, and by when will it be entered? What
program will be used for data entry?
Collection
• Who will analyze the data and how will it be ana-
Professional and ethical considerations ensure the
lyzed?
safe and humane treatment of research partici-
• How will data be disseminated? To what audi-
pants. In addition to the required steps for obtain-
ences?
ing ethical approval and complying with ethical
To the greatest extent possible, all of these procedures (e.g., informed consent and risk man-
questions should be answered prior to initiating agement), researchers need to take special precau-
data collection. In qualitative research, the data tions to limit subject burden. Although most
collection plan typically evolves with the study. institutions out of which research is conducted
Nonetheless, it is equally important to begin with have internal review boards that ensure appropriate
a clear initial plan for data collection. Then, the treatment of research participants, a number of
investigator or research team should periodically more subtle issues must be considered during data
revisit and revise the plan of data collection. This collection.
may entail decisions about where to seek data, Before beginning any data collection proce-
what type of data to seek, and from whom. It may dure, adequate rapport with subjects must be
also involve multiple revisions of semistructured established. It is also the data collector’s responsi-
interviews, observational guides or checklists, and bility to maintain rapport throughout the procedure
other instruments that are commonly used in qual- to the extent that it does not distract the participant
itative research. or interfere with or confound the quality and effi-
Figure 32.1 Dr. Taylor (top picture, left) and her research team make sure data
collection procedures and responsibilities are well defined before initiating data collection.
ciency of the data collection procedures. Rapport their understanding of the limits to the researcher’s
can be maintained during data collection by role and availability in their care.
responding humanely and empathically to any
uncomfortable circumstances or difficult disclo-
sures made by the subject during data collection. Conclusion
Empathic responding involves:
This chapter reviewed the process of data collec-
• Naming, witnessing, and/or verbally acknowl-
tion. It began with an overview of how data col-
edging a difficult circumstance,
lection is viewed and approached within the
• Questioning the participant about his or her emo-
quantitative and qualitative methodological tradi-
tional or physical well-being, and
tions. It defined and evaluated a wide range of pro-
• Following through with any necessary actions to
cedures and approaches used for data collection
ensure the participant’s safety and/or physical
within these traditions. This was followed by a
and emotional well-being.
review of the three steps involved in the actual
Maintaining professional boundaries during implementation of data collection within an inves-
data collection is equally important for the pro- tigation. We concluded with a discussion of pro-
tection of research subjects. This means that fessional and ethical issues that involve the
researchers should avoid nonprofessional relation- treatment of research participants during data col-
ships with participants. Researchers should avoid lection. While this chapter overviewed the major
unplanned disclosure of highly personal informa- considerations involved in selecting and using data
tion to participants, and they should not accept collection procedures, readers who are planning
personal gifts or money from participants. Re- data collection for a study should also refer to
searchers must ensure that participants are clear in other chapters in the text that discuss qualitative
data collection and quantitative instruments in dren and families. Journal of Educational and
more detail. Psychological Consultation, 12 (3), 243–256.
Neuman, W. L. (1994). Social research methods:
Qualitative and quantitative approaches. Needham
REFERENCES Heights, MA: Allyn and Bacon.
Benson, J., & Schell, B. A. (1997). Measurement Theory: Pelto, P., & Pelto, G. (1978). Anthropological research:
Application to Occupational and Physical Therapy. In The structure of inquiry. New York: Cambridge
J. Van Deusen & D. Brunt (Eds.), Assessment in occu- University Press.
pational therapy and physical therapy (pp. 3–24). Polit, D. F., & Hungler, B. P. (1999). Nursing research:
Philadelphia, PA: W. B. Saunders. Principles and methods. Philadelphia: Lippincott.
Bernard, H. R. (1994). Research methods in anthropology: Portney, L. G., & Watkins, M. P. (2000). Foundations of
Qualitative and quantitative approaches. Thousand clinical research: Applications to practice (2nd ed.).
Oaks, CA: SAGE Publications. Upper Saddle River, NJ: Prentice-Hall.
Clemson, L., Fitzgerald, M. H., Heard, R., & Cumming, Rice, P. L., & Ezzy, D. (1999). Qualitative research meth-
R. G. (1999). Inter-rater reliability of a home fall ods, a health focus. Melbourne: Oxford University
hazards assessment tool. Occupational Therapy Press.
Journal of Research, 19, 83–100. Strauss, A., & Corbin, J. (1990). Basics of qualitative
Crotty, M. (1998). The foundations of social research: research. California: SAGE Publications.
Meaning and perspective in the research process. Ware, J. J., & Sherbourne, C. D. (1992). The MOS 36-item
Crows Nest, Australia: Allen & Unwin. short-form health survey (SF-36). Conceptual frame-
de las Heras, C. G., Geist, R., Kielhofner, G., & Li, Y. work and item selection. Medical Care, 30, 473–483.
(2003). The Volitional Questionnaire. Chicago,
Illinois: University of Illinois at Chicago.
DePoy, E., & Gitlin, L. N. (1998). Introduction to
RESOURCES
research: Understanding and applying multiple strate- There are useful publications that identify and review
gies (2nd ed.). St. Louis: C. V. Mosby. occupational therapy and related instruments for data
Ferrans, C. E., & Powers, M. J. (1992). Psychometric collection. The following are two examples:
assessment of the Quality of Life Index. Research in
Nursing and Health, 15, 29–38. • Law, M., Baum, C. & Dunn, W (2001) Measuring occu-
Fisher, A. G. (1993). The assessment if IADL motor skills: pational performance. Thoroughfare, NJ: Slack. This
An application of many faceted Rasch analysis. book covers a wide range of measures that capture data
American Journal of Occupational Therapy, 47, on such aspects of occupational performance as play,
319–329. work, activities of daily living, occupational role, and time
Glaser, B., & Strauss, A. (1967). The discovery of use. There is also a discussion of qualitative procedures
grounded theory. New York: Aldine. for obtaining data on occupational performance. This text
Heary, C. M., & Hennessy, E. (2002). The use of focus includes information about studies of reliability and valid-
group interviews in pediatric health care research. ity. Readers should be careful to search for additional
Journal of Pediatric Psychology, 27 (1), 47–57. publications beyond the publication date of this text.
Hildebrandt, E. (1999). Focus groups and vulnerable • Buros mental measurements yearbook, published by the
populations: Insights into client strengths and needs Buros Institute of Mental Measurements, the University
in complex community health care environments. of Nebraska-Lincoln, provides descriptive information,
Nursing and Health Care Perspectives, 20 (5), references, and critical reviews of tests in the areas of
256–259. personality, achievement, behavior assessment, education,
Hoffman, O. R., Hemmingsson, H., & Kielhofner, G. and science.
(2005). The School Setting Interview (Version 3.0). There are also useful Web sites for identifying instruments
Nacka, Sweden: The Swedish Association of and research on their reliability and validity.
Occupational Therapists
Ivanoff, S. D. (2002). Focus group discussions as a tool for • One such occupational therapy sight is located at:
developing a health education programme for elderly moho.uic.edu. It contains information on a number of
persons with visual impairment. Scandinavian Journal instruments related to the Model of Human Occupation.
of Occupational Therapy, 9 (1), 3–9. Visitors to the site can learn about the available instru-
Krueger, R. A., & Casey, M. A. (2000). Focus groups: ments, view copies of forms, and use a search engine
A practical guide for applied research (3rd ed.). to identify publications on reliability and validity.
Thousand Oaks: SAGE Publications. • Health and Psychosocial Instruments (HAPI) is a search-
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. able online database that contains research on published
California: SAGE Publications. and unpublished information-gathering tools that are uti-
Moore-Corner, R. A., Kielhofner, G., & Olson, L. (1998). lized in health and psychosocial research studies.
Work Environment Impact Scale. Model of Human Information on questionnaires, interview schedules, tests,
Occupation Clearinghouse. Chicago: Department of checklists, rating and other scales, coding schemes, and
Occupational Therapy, College of Applied Health projective techniques is available from 1985 onward. The
Sciences, University of Illinois at Chicago. database pertains to any medical or medically related
Morgan, D. L., & Spanish, M. T. (1984). Focus groups: A condition or treatment outcome. It contains citations to
new tool for qualitative research. Qualitative Sociology, actual test documents, bibliographic citations to journal
7, 253–270. articles that contain information about specific instru-
Nabors, L. A., Ramos, V., & Weist, M. D. (2001). Use of ments, and a catalog of commercial test publishers and
focus groups as a tool for evaluating programs for chil- their available test instruments.
C H A P T E R 3 3
Data Management
Marcia Finlayson • Toni Van Denend
The ultimate goal of any research is to produce the-scenes processes and infrastructure that allow
findings that others can trust and use. This goal is a researcher to:
the same regardless of whether the study is quanti-
• Produce high-quality information to address the
tative, and the findings will be judged on their reli-
study questions, and
ability, validity, objectivity, and generalizability, or
• Describe how and what has been done during a
whether the study is qualitative, and the findings
study accurately and comprehensively.
will be judged on their dependability, credibility,
neutrality, and transferability (Krefting, 1991). In the context of data management, data refer to
Occupational therapy researchers should always all of the pieces of information that are collected
strive to do rigorous work that will contribute to from research participants to address a study’s
the advancement of our theories and clinical prac- questions, as well as all of the information that is
tices (Kielhofner, Hammel, Finlayson, Helfrich, & gathered to monitor and manage study progress. A
Taylor, 2004). few examples of this latter type of information
Earlier chapters discussed many different fac- include:
tors that can threaten the
quality of the research. • Recruitment processes
Unfortunately, even the In the context of data man- and outcomes,
best designed study can • Progress on data collec-
agement, data refer to all tion, coding, entry and
fail if the research team
does not plan for and of the pieces of information cleaning,
address issues of data • Status of data storage
that are collected from and security, and
management. Despite
many publications and research participants to • Documentation of deci-
sions that will make it
the wealth of knowledge address a study’s questions, easier for the investiga-
about how integral data
management is to rigor- as well as all of the informa- tor to describe what
he or she has done dur-
ous investigation, its tion that is gathered to ing the course of the
importance is still some-
times underestimated.
monitor and manage study study when it is time to
Many researchers learn progress. disseminate the study
about data management findings.
by watching their men- For simplicity and
tors, talking to other researchers, picking up ideas clarity, the term data is used to refer to the infor-
at conferences, and learning from their own mis- mation collected from participants, and project
takes. information is used to refer to the other types of
The chapter provides a framework for thinking information that a data management system must
about and planning data management processes track.
and infrastructure. We also share some of our own
experiences from investigations that we have
planned and implemented, and provide tools and The Context of
Data Management
resources that can be used and modified for other
studies.
To begin, it is important to understand the overall

Basic Definitions context of data management and how it fits within
the research process. Figure 33.1 depicts the rela-
For the purposes of this chapter, data management tionship between data management and research
is defined as the logistical, reflective, and behind- design, and illustrates that the ultimate goal of data
Chapter 33 Data Management 549
management is to support the production of find- • How and where information about new recruits is
ings that are trustworthy and are useful to others. to be documented, and
The figure also illustrates that data management is • How calculations on response rates are to be
done in conjunction with design. In other words, summarized.
designs that are truly rigorous must include plan-
ning for data management as part of the process, Documenting the steps and processes of a proj-
and strong data management systems must take ect makes it easier for all parties involved to be
into account the research design (McFadden, consistent during the course of a research study,
LoPresti, Bailey, Clarke, & Wilkins, 1995). These regardless of the project’s size or the type of
two processes are inseparable in the production of research being conducted (e.g., ethnographic,
good quality data (Nyiendo, Attwood, Llyod, experimental, survey, etc.) (Antonakos, Miller, &
Ganger, & Haas, 2002). Caruso, 2002; Gassman, Owen, Kuntz, Martin, &
Finally, the figure emphasizes that good quality Amoroso, 1995).
data does not automatically mean trustworthy Using consistent processes and procedures
findings that others can use. Selecting the correct throughout a project will reduce the likelihood of
analytic techniques for making errors in the
the data that are avail- course of data collection
able, and using these The ultimate goal of data and/or its entry into ana-
techniques appropri- management is to support lytic software systems.
ately will determine the Minimizing errors at the
value of a study’s find-
the production of findings beginning and throughout
ings. Using techniques that are trustworthy and are a study will reduce prob-
appropriately means useful to others. lems later in the process
that a researcher must that could negatively
understand the type of influence the speed or
data needed for a given analytic strategy, under- accuracy of data analysis, limit the types of analy-
stand the assumptions of the technique and how ses that are possible, or increase the challenges of
these can be determined, and interpret results cor- preparing the research report (Hosking, Newhouse,
rectly.
Importance of Data High-quality, Strong data

rigorous management
Management design systems
The larger the project, the more sophisticated the

data management plan will be, but all investigators
need to plan for and implement a basic system,
regardless of the size of their study. The reasons Good data
data management is important are summarized in
the feature box on the next page and explained in
Figure 33.2.
At the most basic level, a data management
plan will facilitate and support communication
about the research study across members of a proj-
Appropriate selection
ect team, as well as between the team and external and utilization of
parties (e.g., participants, funders, colleagues, edi- analytic techniques
tors). The plan will offer guidance about how to
carry out and document all research-related proce-
dures and their consequences. For example, a data
management plan should identify:
• Which members of the research team are respon- Trustworthy and

sible for identifying and contacting potential par- useful findings
ticipants,
• What forms are to be used in this process, Figure 33.1 The context of data management.
cuss problems encountered in coding unusual par-

Data Management is important for:
ticipant responses. During these meetings, the
• Supporting a culture of consistent communica- interviewing team and principal investigator dis-
tion, cussed the problem, made a coding decision, and
• Minimizing data collection and entry errors,
then documented that decision for inclusion in the
• Facilitating data analysis and interpretation of
findings, policy and procedure manual. In this way, everyone
• Maximizing data quality by promoting internal on the team had access to the decision, and was
validity (credibility) and reliability (dependabil- able to code the data in a consistent way if and
ity) of data, when the problem occurred again. Without this
• Ensuring data security and confidentiality, process it is likely that unusual responses would
• Ensuring compliance with relevant laws, not have been consistently coded and would have
• Facilitating the preparation of the research caused problems later in the analysis phase.
report, including the identification of study Documentation of policies (rules) and proce-
strengths and limitations, and dures (explanations about how to implement the
• Facilitating data archiving and sharing.
rules) is a major aspect of a data management plan
(Gassman et al., 1995). Documentation for a study
should explain:
Bagniewska, & Hawkins, 1995; Pogash, Boehmer, • How to track what member of the research team
Forand, Dyer, & Kunselman, 2001). For example, has collected what data from which participants,
one of the data management processes used in the • When the data were collected,
Aging with MS: Unmet Needs in the Great Lakes • If any additional information needs to be col-
Region Study1 was a regular team meeting to dis- lected,
• What the status of the data currently is (e.g.,
entered or not, where), and
Good data management ensures that
• What data are missing and why (Antonakos et
everyone on the team knows what to do; al., 2002).
why, when, and how; and that they do so
consistently throughout the study.
By having all of this information at hand, a data
management system facilitates the preparation of a
comprehensive and detailed methods section of
Increases research and technical reports, as well as other
the odds
forms of dissemination (see Chapter 34). Methods
of.....
that are well documented will be transparent and
therefore will be viewed as supporting the internal
validity (credibility) and reliability (dependability)
Producing good quality data that are: of the study.
valid and reliable (quantitative)
A strong data management system also ensures
or that a research team is able to maintain the security
credible and dependable (qualitative). of the data collected from participants and demon-
strate that guidelines for the protection of human
Increases subjects have been followed. It also ensures that
the odds the research is in compliance with any other laws
of..... that are relevant to the conduct of the research
(e.g., Health Insurance Portability and Account-
ability Act2 (HIPAA), professional licensing when
delivering an intervention, etc.) (Hosking et al.,
Being able to conduct 1995; McFadden et al., 1995). It is important to
analyses and interpret data
appropriately.
remember that the human subjects committee that
approves a study is free to audit the research for
compliance at any time. Failing to follow regula-
Figure 33.2 Importance of data management. tions regarding human subjects or other relevant
1This project is based on a contract awarded to Dr. Marcia
Finlayson by the National Multiple Sclerosis Society 2HIPAA is specific to the United States, but other countries
through a Health Care Delivery and Policy Research and jurisdictions have similar laws. The focus of HIPAA is
Contract, 2002 to 2005. on the protection of personal health information.
guidelines or laws could result in a project being the number of staff involved, and the presence of
closed. more than one research site (Gassman et al., 1995;
Finally, data management will facilitate study McFadden et al., 1995). Therefore, it can be help-
and data archiving that will, in turn, make it easier ful to think about the components or parts of a data
for a research team to: management system when beginning the process.
Figure 33.3 illustrates that a strong data man-
• Compare its findings to other similar studies,
agement system, including both participant data
• Provide data to students for thesis or dissertation
and project information, includes processes and
work based on secondary analyses, and
infrastructure related to staff, and a clearly defined
• Share data with other researchers who seek to
set of policies and procedures. Furthermore, these
build on the work that has already occurred.
components are premised on a foundation of a cul-
Being able to explain to others what was done, ture of consistent communication. In the context of
and how and why will facilitate later data analyses data management, consistent communication
by members of the team or others who have an includes (Gassman et al., 1995; McFadden et al.,
interest in the topic. 1995):
Failing to address data management in a com-
• Verbal and written communication among mem-
prehensive, methodical, and consistent way will
bers of the team (e.g., principal investigator, co-
result in inconsistencies within and across staff or
investigators, research assistants, analysts) via
over time, limit the quality or detail of the methods
team meetings, e-mails, memos, etc., and
description in a published article, jeopardize data
• Documentation of individual work and decisions
quality and confidentiality, or even worse, raise
(e.g., maintenance of study notebooks, lab
questions about the internal validity or credibility
books).
of an entire study. Consequently, data management
needs to be included during the initial proposal Written documentation enables the team and
preparation stages of any research study (see others (e.g., funders, collaborators, people using
Figure 33.1). The system needs to be updated as the data for secondary analyses) to track decisions
the project evolves so that new issues are about different aspects of the project easily, and to
addressed and new decisions are documented. clearly understand the logic behind these deci-
It should be clear at this point that developing a sions. As a project unfolds, documentation will
strong data management system involves more facilitate consistency in how data management
than just recording participants’ responses to the problems are resolved (e.g., coding unusual partic-
study questions and measures. It is an administra- ipant responses) (Gassman et al., 1995).
tive process that supports rigorous research and Ultimately, without consistent communication,
involves: any data management system is at risk of crum-
bling and therefore compromising both the quality
• Documenting how all forms of data and project
of the data produced, and the ability to support the
information will be collected, handled, stored,
production of trustworthy and useful findings.
and prepared for use,
• Developing tools and systems to operationalize
the documentation, and
• Following through on the use of these tools and Strong data
systems, refining and adding to them as the proj- management
ect evolves over time. systems
Components of Data
Management Staff
Policies and
Procedures
Translating the knowledge of why data and project
information should be tracked to the actual cre-
ation of a data management system is often not
Culture of Consistent Communication
that straightforward. The process can be signifi-
cantly complicated by the breadth and depth of
what must be monitored, and will clearly be influ- Figure 33.3 Key components of a data manage-
enced by the size and scope of the research project, ment system.
Later sections of this chapter elaborate on some of responsibilities of team members, documenting
the specific tools and strategies that can be used to activities, tracking respondents, etc. To highlight
support the production of good quality data. We key issues we have taken Figure 33.3 and ex-
will also point out specific strategies that promote panded it in Figure 33.4.
consistent communication. This figure illustrates the different types of
To summarize this discussion on the compo- issues that an investigator needs to consider under
nents of data management, it is important to think both the staff component (e.g., selection of team
of a data management system as both a product as members) and the policies and procedures compo-
well as a dynamic process. When done well, it is nent (e.g., data access and security, documenta-
the infrastructure that allows the research study to tion) of the data management system. Clearly, this
be carried out as planned. When there are prob- figure is not all-inclusive, given the variability
lems with data management, even the best across projects, research designs, populations
designed and funded study can fail to address its being studied, and the size of different studies.
intended objectives. Nevertheless, experience suggests that addressing
each of the seven issues included in Figure 33.4 is
Data Management Issues key to increasing the likelihood of high-quality
data management.
to Consider and When to The specific tasks that need to be completed to
address each of the data management issues iden-
Consider Them tified in Figure 33.4 are outlined in Table 33.1. In
addition, the importance of doing each of these
Before discussing some of the specific tools that tasks and when to do them is provided in Table
can facilitate data management, we will identify a 33.1. A quick scan of the final column of Table
number of issues that need to be considered when 33.1 shows that most of the issues for data man-
developing a data management system. We have agement need to be considered during the initial
already alluded to some of these issues in our dis- proposal development. This timing is critical
cussions of the importance and components of data because human and financial resources are often
management, for example, specifying the roles and required to manage data and project information
Strong data
management
systems
Policies and
Staff
Procedures
Monitoring Training and Data collection

Selection of Data access
productivity and continuing processes and
Team Members and security
work quality education protocols
Data entry,
Documentation checking, and
cleaning protocols
Culture of Consistent Communication
Figure 33.4 Specific issues to be considered in a data management system.

33Kielhofner(F)-33
Table 33.1 Data Management Issues, Tasks, Importance, and Timing
Issues to Consider
(from Figure 33.4) Specific Tasks Importance Timing
5/5/06
• Determine what skills are needed across the members of the team • Ensures all aspects of project • Consider during proposal
(e.g., administering specific assessments, delivering interventions, are being addressed across development.
creating data files, entering data, conducting analyses). the members of the team. • Reconsider and refine at time of
• Determine the qualifications of individuals who can fulfill these skill • Enables the principal project initiation.
4:04 PM
requirements. investigator to put together a
• Clarify the roles, responsibilities, and performance expectations team that can meet all of the
Selection of team associated with specific jobs on a research team. needs within a project.
members • Outline reporting and accountability lines, which are important for staff
Page 553
monitoring.
• Develop job descriptions.
• Use job descriptions to recruit potential team members.
• Screen qualifications of potential team members.
• Select individuals who have the necessary skills and experience to meet
the project needs.
Staff
• Ensure that members of the research team understand their job

descriptions.
• Track whether staff members are doing the tasks they are assigned, in the • Ensures that the principal • Consider during proposal
way that they were trained. investigator can maintain the development.
Monitoring rigor of the project.
• Monitor staff for ongoing continuing education needs. • Begin and maintain after project
productivity
• Provide opportunities for team members to identify potential problems that is started and staff is hired.
and work
quality may later influence data quality, and solve them before it is too late. • Update as required during
• Create and implement systems to address problems in productivity and project.
work quality.
• Identify project specific training needs for staff members as a group, as • Enables the principal • Consider during proposal
well as individually (e.g., roles and responsibilities, specific job duties, investigator to contribute to development.
Training and
lines of authority and accountability). the maturation and sophis- • Begin and maintain after project
continuing
• Prepare and deliver staff training and continuing education materials. tication of a team over time. is started and staff is hired.
education
• Develop a system to keep staff up-to-date on the latest developments in • Update as required during
the field that may impact a project. project.
(continued)
553
33Kielhofner(F)-33
Table 33.1 Data Management Issues, Tasks, Importance, and Timing (continued)
554
Issues to Consider
(from Figure 33.4) Specific Tasks Importance Timing
• Design and implement a system for assigning ID codes. • Ensures compliance with • Consider at proposal
• Identify who has access to what information (e.g., budgets, participant relevant laws, rules and development.
5/5/06
information, specific data files). regulations. • Refine upon human subjects
• Provide team members with explicit guidelines for: • Protects participant review submission.
Data access confidentiality.
• How and where data and project information are to be stored, • Follow upon project initiation.
and security
• What information is to be password protected (e.g., participant contact • Modify as required during
4:04 PM
information), and project.
• What data needs to be separated (e.g., consent forms from data sheets).
• Design and implement a system for conducting data and file backups.
• Identify what project information must to be documented (e.g., recruitment • Facilitates consistent decision- • Consider at proposal
Page 554
responses, participant attrition and reasons). making. development.
• Identify where project information is to be documented (e.g., specific files and • Supports all other aspects of • Begin at project initiation.
Documentation
their locations). the data collection system. • Refine and update throughout
Policies and Procedures
• Determine when project information is to be documented (e.g., frequency). • Facilitates report preparation. project.
• Explain to members of the team the rationale for thedocumentation. • Ensures compliance with rele-
vant laws, rules and regulations.
• Explain each of the specific steps to be taken during the data collection, including: • Ensures consistency in data • Consider at proposal
• Prepare necessary scripts. collection, and therefore development.
• Prepare any anticipated protocols for addressing participants with cognitive maximizes data quality. • Reconsider and refine upon
impairments, use of proxy respondents, etc. • Ensures compliance with project initiation.
Data collection • Prepare any resource materials data collectors will require (e.g., list of relevant relevant laws, rules, and • Update and refine as new
processes phone numbers such as abuse or suicide hot lines). regulations. challenges emerge.
and protocols • Specify tasks and responsibilities for each member of the team during data
collection.
• Educate team members about how to consistently address challenging situations
that may arise during the data collection process (e.g., dealing with participants
who disclose abuse; uncovering cognitive impairment that may compromise
informed consent).
• Identify each step of the data entry and cleaning process, including: • Ensures quality data for • Consider at proposal
• Who is responsible, analysis. development.
Data entry,
• What specific tasks must be completed at what time, • Saves time at end of project by • Begin at project initiation, in
checking and
• What processes are to be used to confirm data entry, and maintaining data files conjunction with decisions about
cleaning
• How data entry errors are to be documented and resolved. throughout project. data collection.
protocols
• Develop and maintain a code book. • Refine during data collection, but
• Identify timing for data checking and cleaning. before data are being entered.
well (McFadden et al., 1995). Often, new investi- This section includes specific examples of tools
gators do not fully appreciate the amount of time that can be used in the process of managing data.
that is required to manage the everyday adminis- Table 33.2 presents the linkages between the tools
trative aspects of a study, and fail to incorporate and resources that are presented in this section,
adequate staff time in their project budget. By and the data management issues from Figure 33.4
thinking through data management, and carefully and Table 33.1.
analyzing the requirements of the project from These tools and resources are ones we have
beginning to end, an investigator will be in a better found to be particularly helpful; their use is also
position to plan the necessary resources to support supported in the existing literature on data man-
project infrastructure (McFadden et al., 1995). agement. What is presented here represents only a
sampling of what is possible, and does not include
Specific Tools and Resources all of the different tools that can be used to facili-
tate data management. The tools described are
to Facilitate Data Management intended to offer a starting point for those with lit-
tle or no experience in developing data manage-
Good data management involves a range of tools ment systems. Not everything that is presented
and processes that allow the researcher to: over the next few pages will be relevant to every
project.
• Track what has been done,
• Track what is still needs to be done, and
• Ensure that the final research report and findings
Management Plan and Project Timeline
will be recognized as products of a rigorous and The management plan and project timeline is typ-
believable process. ically prepared as part of the research proposal. A
Table 33.2 Examples of Data Management Tools and Their Correspondence

with Specific Data Management Issues
Tool, Resource or Strategy Data Management Issue Influenced by Tool, Resource or Strategy
Staff Policies and Procedures
Management plan and project • Monitoring productivity and work • Documentation

timeline quality
• Selection of team members • Data access and security
Job descriptions • Monitoring productivity and work • Data collection processes and
quality protocols
• Data entry, checking, and
cleaning protocols
Systems to track participants • Training • Data access and security
and their data including: • Monitoring productivity and work • Documentation
• Participant identifiers quality • Data collection processes and
• Master file protocols
• ID sheets
Systems for data entry and • Training • Data collection processes and
confirmation: • Selection of team members protocols
• Protocols for data coding, • Training • Data entry and checking
checking, and cleaning • Monitoring productivity and work processes and protocols
• Codebooks quality • Documentation
• Development of data
collection forms
• Training • Documentation
Team meetings • Monitoring productivity and work
quality
Backup protocol • Training • Documentation
• Data access and security
brief example is provided in Table 33.3. As a data understands his or her responsibilities. Typical
management tool, a management plan and project areas to include in a job description are:
timeline is helpful in the identifying the specific • Job title,
tasks that will need to be done over the course of • Name and title of direct supervisor,
an entire project and when these tasks need to be • Summary of job character and purpose of the
completed. Having this information will facilitate position,
the process of identifying the skills and knowledge • Qualifications and/or specific skills required, and
that will be required of the research team, which • Specific list of job duties.
will aid in their selection. Later on in the project,
the management plan and project timeline will Systems to Track Participants
provide guidance for monitoring the productivity
and Their Data
and work quality of team members, and help in
setting goals for data collection and for monitoring To obtain good quality data, it is critical that an
its progress. investigator can track participants throughout the
course of a study (Nyiendo et al., 2002; Pogash et
al., 2001). This information ensures that all infor-
Job Descriptions mation is collected from participants at the right
For a research project to operate smoothly, the times, that the data gathered from each participant
principal investigator must ensure that the people are handled correctly, and that everyone can be
who are performing the various tasks within the accounted for at the end of the study, including
project have the necessary skills and expertise to the individuals who were recruited but never
do so. Specifying these tasks and identifying the actually participated, those who dropped out, or
necessary qualifications of the individuals who those who were lost to the study (e.g., moved,
will be performing these tasks are the first steps in died, etc.).
developing job descriptions. In large projects when In the study, Addressing Concerns about Fall-
staff members are hired into a specific role, job ing Among Older Adults with Multiple Sclerosis,3
descriptions facilitate a culture of consistent com- we explained in detail the process of tracking
munication by clearly identifying the roles and participants and their data in our policy and pro-
responsibilities of each team member. For small cedure manual, including who is responsible to
projects in which one or two people play multiple 3This project is based on a grant awarded to Dr. Marcia
roles, job descriptions can ensure that all tasks are Finlayson and Ms. Elizabeth Peterson by the Retirement
addressed and assigned, and that each person Research Foundation, 2004 to 2007.
Table 33.3 Sample of a Part of a Management Plan and Project Timeline

People Measure of Task
Specific Project Tasks Time Frame Responsible Completion
Activity no. 2: Conduct focus groups with key informants
2.1. Work with collaborators to select dates, July to August, 2002 PC and RA Dates and times
times and locations for focus groups set
2.2. Work with collaborators and July to August, 2002 PI, PC, AG, and Potential
Advisory Group members to identify RA participants
potential participants for focus groups identified
2.3. Set up office systems to manage focus group July to August, 2002 PI, PC, and RA Office systems
documents and data (e.g., master file, logistics completed
task lists, transcription processes, etc.)
2.4. Contact potential participants to invite partici- August to PI, PC, possibly No. of participants
pation and provide basic information about September, 2002 AG contacted
the focus group
AG ⫽ members of advisory group; PC ⫽ project coordinator; PI ⫽ Principal investigator; RA ⫽ research

assistant.
Excerpted from the data management plan for the study: “Aging with MS: Unmet needs in the Great Lakes
Region,” Finlayson, principal investigator. Funded by the National Multiple Sclerosis Society, July 2002–June
2005, Contract no. HC0049.
complete which parts of the process, when, and A participant identifier is a tool critical to this
where. Figure 33.5 summarizes the process used process. A participant identifier is a numeric or
showing the connections between the initial alphanumeric code (e.g., A-001) that is used in
recruitment of a participant and the final data file place of the participant’s name on all documents
for analysis. pertaining to that individual. In terms of data man-
Figure 33.5 illustrates how a system for track- agement, assigning identifiers is more than simply
ing participants and their data involves a series ensuring that participant names are kept confiden-
of individual tools, all of which are used in concert tial, although that is a major reason for using iden-
to achieve the goal of good quality data. The spe- tifiers. The actual process of assigning identifiers
cific tools and processes within these systems that can facilitate the data collection and management
will be discussed in more detail in this section process by providing a consistent variable that can
include: be used to link files across the study (e.g., master
participant list, data file, follow-up files, etc.).
• Developing and assigning participant identifiers,
To illustrate how the assignment of identifiers
• Creating and maintaining a master file, and
can facilitate data collection, consider the example
• Developing and using participant ID sheets.
of the study Addressing Concerns about Falling
Developing and Assigning Among Older Adults with Multiple Sclerosis that is
provided in the feature box on the next page. In
Participant Identifiers
this project, a national sample was used for tele-
To be able to track participants and their data, it is phone interviews. To facilitate the interviewing
necessary to be able to link individual data items process, and reduce the risk that participants
to the people from whom the data were collected. would be called at inappropriate times of the day
Record participants'
Recruit Assign contact information Prepare ID
participants. identifiers. and identifier in the sheet.
master file.
Assign participants
(with ID sheet) to
data collectors.
Record all contacts and

Check data Code data Update outcome of contacts (e.g.,
coding manually. collection forms. master file. data collection complete,
refused, proxy protocol used).
Regular data checks to Correct and document

Enter data in find data entry errors data entry errors
analysis file. (e.g., run frequency (i.e., clean the data).
distributions).
Create a final analysis file and corresponding

Archive final
documentation (e.g., one that includes all derived and
analysis file.
recoded variables with their explanations).
Analyze data.
Figure 33.5 Tracking participants and their data.

(as a result of differences in time zones), informa- Creating and Maintaining a Master File
tion about the participant’s time zone was embed-
ded into the identifier. In addition, because When tracking participants and using forms and
participants were divided into two age groups for files to do so, it is important to note that documents
sampling, age group was also embedded in the containing both the participant’s name and identi-
identifier to make it easier to identify how many fier should be kept to a minimum because of the
interviews had been completed in each group. sensitivity and confidentiality of these materials.
Identifier information should be maintained in a Nevertheless, it is important in most studies to be
password-protected file. When working in larger able to link a participant identifier to information
multisite studies, assigning identifiers must be such as name, address, and/or telephone or other
considered carefully so that when the final data are means of contact. A master file is one tool to
merged, there is no possibility that identifiers are address this data management need.
duplicated. For example, one site may be assigned A basic master file typically contains partici-
to use alphanumeric codes starting with the letter pant identifying information (e.g., name, contact
“A,” while the second site may be assigned identi- information) together with the participant identi-
fiers starting with the letter “B.” fier code. Because of the sensitivity of this infor-
For therapists who are conducting research as mation, this file must be password protected and
part of their regular clinical duties, assigning iden- access to the file should be restricted to key mem-
tifiers must be done carefully to ensure that the bers of the research team. Master files are usually
research data and clinical data cannot be confused electronic spreadsheet files. As new participants
and inadvertently mixed together. Research data are recruited into a study, a participant identifier is
must be kept separate from the medical record, and assigned, and then the master file is updated (see
under separate identifiers. Figure 33.5). Depending on the project, the master
file may also include process related information
such as how many contacts were attempted with a
Assigning Identifiers—Sample Explanation participant, whether baseline or outcomes meas-
ures have been administered, or when follow-up
The Senior Research Assistant will be responsi- appointments are to be scheduled. It is important
ble for intake, entry, and assigning ID no.’s to all
eligible respondents.
to note that this type of information is not data for
analysis, but rather data to track the progress of a
• ID numbers will be assigned based on time participant through the study. Data that will be
zone and age group. used for analysis is maintained in the analysis file.
• Time zone will be the first two digits in the ID Personal contact information should never be kept
code:
• Eastern Time ⫽ ET
in the analysis file.
• Central Time ⫽ CT
• Mountain Time ⫽ MT Developing and Using
• Pacific Time ⫽ PT Participant ID Sheets
• Alaska ⫽ AT
• Hawaii ⫽ HT After recruiting participants, assigning identifiers,
• There are two age groups of interest in this and adding individuals to the master file, it is very
study, 55–64 and 65⫹. The age groups will be important to do everything possible to maintain
designated as 01 (55–64) and 02 (65⫹). These participant confidentiality. This typically means
will be the second two digits in the ID code.
that only minimally necessary information about
• The remaining part of the ID code will be the
sequential number of the respondent, e.g., 001, the participant is recorded on other data collection
002, 003, etc. Each age group is to have its own forms. In fact, actual data collection forms should
sequential numbering starting with 001. Time never include the participant’s name, but rather
zone will not be considered in the sequential only the participant identifier. A key challenge of
numbering—only age group. occupational therapy research is that data collec-
• A sample ID would therefore be: MT-02-005. tors usually need both the participant identifier
Excerpted from the Policy and Procedure Manual as well as contact information, particularly if an
from the study: “Addressing Concerns about intervention or interview is involved, and if the
Falling Among People Aging with MS” data collector must go out to the participant’s home
(Finlayson & Peterson, 2004–2007, with funding or workplace. Balancing the need to have this in-
from the Retirement Research Foundation). formation and keeping it confidential can be chal-
lenging.
A Participant ID sheet is one data management Cummings, Browner, Grady, Hearst, & Newman,
tool that can be used to address this dilemma and 2001; McFadden et al., 1995; Pogash et al., 2001;
ensure that data collectors have the pieces of infor- Portney & Watkins, 2000). Given that data coding,
mation they need to maintain their records, and at checking and cleaning are process-oriented tasks,
the same time minimize the number of places most investigators develop protocols to manage
where the participant’s name and identifier appear them. Therefore, the data management tools are the
together. A sample Participant ID sheet is provided protocols themselves. In addition to these proto-
in Figure 33.6. A major advantage of an ID sheet cols, other related data management tools and
is that it can easily be removed from the partici- strategies include codebooks and how data collec-
pants’ files when the information it contains is not tion forms are developed.
germane to the activities of the research team.
Maintaining clear protocols about how to handle Protocols for Data Coding, Checking,
the ID sheet, where it is to be stored, who can have and Cleaning
access to it, and when it is to be destroyed is very
important. Often, the human subjects’ ethics com- In quantitative research, data coding involves
mittee will have guidelines about these issues. translating participant responses into numerical
values to facilitate statistical analysis. For vari-
ables that are continuous in nature (e.g., age, years
Systems for Data Entry and Confirmation since diagnosis), responses are already numerical
While data collection is often a major focus during so coding simply involves entering a participant’s
the process of planning a study, it is equally impor- value into the analysis file. For variables that are
tant to consider how to enter and confirm these data nominal or ordinal in nature, responses must be
after the collection is completed. The issues of data assigned a numerical value. For example, consider
coding, checking, and cleaning are most often dis- a survey question that asks participants to reflect
cussed in relation to data management, and other on how they are managing today compared to a
authors have described these processes in detail year ago. Response options might be either “about
(see Aday, 1989; Hosking et al., 1995; Hulley, the same,” “worse,” or “better.” To be able to use
Aging with Multiple Sclerosis Study
ID Form for Interviewers
Date reply form received:

Date interview assigned:
Participant name:
Participant ID #:
Participant telephone #:
# of calls to obtain interview:
# of calls to complete interview:
Date interview completed:
Contact log: Please note when
contacts were made with
participant (date, time, outcome
of contact).
CONFIDENTIAL
Figure 33.6 Participant ID sheet.
these responses statistically, the answers must be vided in the feature box titled “Sample Protocol
translated into numbers, for example, “about the for Data Coding and Checking,” and one for data
same” equals 1, “worse” equals 2, and so on. This cleaning is provided in the feature box titled
process is called data coding. In qualitative “Sample Protocol for Data Cleaning.”
research, data coding involves applying labels to
sections of text. Details on qualitative data coding
Codebooks
are provided in Chapter 21.
Once data are coded, and the values for each There can be many variables in a project, and
response for each participant is entered into the keeping track of each of them is critical.
analysis file, the investigator or a designate (e.g., Codebooks are one data management tool that can
research assistant, statistician) must do the data address this need. Codebooks are relevant to both
checking. The term data checking primarily relates qualitative and quantitative research (MacQueen,
to quantitative data. Data checking involves run- McLellan, Kay, & Milstein, 1998; Nyiendo et al.,
ning frequency distributions and measures of cen- 2002; Shi, 1997). A codebook can be considered
tral tendency and variability on each of the the dictionary for the data set generated by the
individual variables in the analysis file to check for study. The key contents of a quantitative codebook
missing or out of range values. If such values are include the list of variables, their names and labels,
found, it may indicate a data entry error. Potential and the coding for the different responses.
errors can be checked against raw data, and if nec- Information in a qualitative codebook can include
essary, corrected. The process of correcting data the list of codes, a definition of each code, and a
entry errors is referred to as data cleaning. A sam- description of when and how to use it.
ple protocol for data coding and checking is pro- In quantitative research, a codebook often con-
Sample Protocol for Data Coding and Checking
1. During any of the interviews, record information follow the same procedure (i.e., single line, cor-
directly on the form being used—do not use the rection beside).
coding boxes to record information as it is being 8. Special Codes:
collected. • 777—Skip
2. All interviews should be coded by the inter- • 880—Person with MS is deceased
viewer in BLUE INK. • 888—Don’t know. If the participant is unable
3. All interviews should be coded within 24 hours to decide between two categories, code the
of the completion of the interview. response as 888 “Don’t know”.
4. Initial and date the interview guide to indicate • 899—Skipped due to the use of the cognitive
when the interview was completed, and when it impairment4 protocol
was coded. Mark initials and date on the • 990—Skipped due to the use of the proxy pro-
bottom. tocol
5. Each interviewer’s first 25 interviews should be • 997—Not applicable (e.g., in nursing home,
checked by the project coordinator for accuracy don’t ever do stairs, apartment complex mows
of coding. The Principal Investigator will check yard and does yard work)
the project coordinator’s coding accuracy. After • 998—Refusal. If the participant refuses to
these checks, the principal investigator or project answer a question or a series of questions.
coordinator will complete random coding checks • 999—Interviewer error
at regularly scheduled interviews, unless other-
wise assigned. These random checks will be Excerpted from the Policy and Procedure Manual
made of 5% to 10% of the participants from the from the study: “Aging with MS: Umet Needs in
data files. Random numbers will be applied to the Great Lakes Region” (Finlayson, 2002–2005,
the ID codes to select the interviews for check- with funding from the National Multiple Sclerosis
ing. Society, Contract no. HC0049).
6. If an error is made during the initial coding 4Depending on the nature of the participants who may be
process, the error is to be stroked out with a recruited into a study, sometimes it is necessary to develop
SINGLE LINE. Corrections are to be marked protocols for dealing with participants with certain char-
BESIDE the initial code and initialed by the acteristics. In our work, we sometimes come across indi-
team member assigned to data cleaning. viduals with cognitive impairment or individuals who
7. Any coding corrections that are made by the need assistance to complete an interview (i.e., proxy). To
team member assigned to complete the data ensure consistency across interviewers, we have specific
checking are to be done in RED INK, and are to protocols on how to proceed with these interviews.
Sample Protocol for Data Cleaning facilitate linkage and data entry and cleaning is the
format and preparation of the data collection forms
1. Interview data will be entered into SPSS’ Data (Antonakos et al., 2002; Nyiendo et al., 2002; Shi,
Builder program. For this project, use the Data 1997). For example, all of our interview guides
Builder on computer D. include the data coding for each of the response
2. The research assistants will take responsibility options (e.g., stable = 1, improving = 2, etc.) for
for entering the interview data into Data each question right on the data collection form, as
Builder. As each interview is entered, initial illustrated in the feature box titled “Examples of
and date the form in black ink to indicate that
Items from an Interview Guide.” In addition, the
it was entered. Mark initials and date on the
appropriate line on the bottom of the form. variable label that is used in the analysis file for the
3. The co-principal investigator will be responsi- question is included on the data collection form. In
ble for exporting the data file from Data this feature box, the variable labels are “Msstatus”
Builder on the first Monday of the month. and “Ablechg.” These labels are the ones that
4. The co-principal investigator will complete fre- appear in the analysis file. The variable labels and
quency distributions on the full exported file to codes on the data collection form match the ones
check for data entry errors, specifically out of in the electronic data file (e.g., SPSS, SAS). Being
range values and missing values. able to link files and forms together facilitates data
5. The co-principal investigator will give the sen- management by ensuring that data can be cross-
ior research assistant any identified errors to
checked, confirmed, and, if necessary, corrected or
check against raw data and correct as neces-
sary. updated as errors or omissions are found.
6. The co-principal investigator and senior In addition, we also include the participant’s
research assistant will maintain a log of identi- identifier on each page of each document that is
fied errors and the corrections made. The prin- connected to that person. Doing so facilitates data
cipal investigator has final authority for management by ensuring that data collection
making decisions about how to handle data forms that are unintentionally separated during the
corrections that are not obvious (i.e., not sim- data collection process can be reconnected.
ply a matter of missed entry).
Excerpted from the Policy and Procedure Manual
from the study: “Addressing Concerns about Team Meetings
Falling Among People Aging with MS”
(Finlayson & Peterson, 2004–2007, with funding Throughout this chapter, the importance of consis-
from the Retirement Research Foundation). tent communication across members of the
research team has been emphasized to ensure that
data are managed well. One important strategy for
facilitating communication is the team meeting, to
tains information on the origins of items, instruc- which we have previously alluded. Team meetings
tions on how to complete the coding, how to code are an opportunity for everyone involved in the
unusual responses, where individual data files are project to come together to discuss progress,
kept (raw data as well as electronic), who main- address problems and make decisions to solve
tains the passwords for the files, and how derived them, share achievements, and to make joint deci-
variables were developed. A derived variable is sions that influence the project.
one that is computed using other variables that To facilitate a culture of consistent communica-
already exist in the analysis file. An example of a tion, everyone on the research team should be
derived variable would be “number of ADL limita- included in a team meeting, even if it appears on
tions,” which could be derived by counting the the surface that the discussion is not directly rele-
number of individual ADL items with which each vant to his or her duties. Team meetings provide an
participant identifies requiring assistance. opportunity for all staff members to see how their
own activities and decisions influence the work of
others.
Development of Data Collection Forms
Taking minutes during team meetings is
The key to coding, checking, and cleaning is fore- advised, particularly if decisions are made regard-
thought, careful documentation, and attention to ing data collection processes, changes to data cod-
detail. Forms and files need to be linkable, and ing protocols, etc. In addition, having team
many models exist for achieving linkage (e.g., meeting minutes can make it easier for an investi-
relational databases, interface files, etc.) (Hulley et gator to summarize technical decisions at the end
al., 2001). A key strategy that can be employed to of a project.
Examples of Items from an Interview Guide that Include the Coding for Each Response,
and the Variable Names that Will Be Used in the Data File
Survey Items and Response Options Explanation

Would you say that your MS is within the last year Both of the survey items that are shown to
(read all but don’t know option): the left include the response options and the
_____ Stable (1) Msstatus__________ coding for each option (the number in the
_____ Improving (2) bracket behind the response). “Msstatus” and
_____ Deteriorating (3) “Ablechg” are the variable names that will be
_____ Variable (4) used in the data file. The line to the right of
_____ Don’t know (888) these names in the interview guide is used for
“coding” after the interview is complete. For
Thinking about your symptoms and your ability to do example, if a participant responded that in the
everyday activities, how do you think you manage now past year, his or her MS had been “stable,” a
compared to one year ago? (read all but don’t know option) “1” would be recorded in the line beside
_____ About the same (1) Ablechg__________ “Msstatus.” “1” would also be recorded in the
_____ Worse (2) data file for that respondent for this variable.
_____ Better (3)
_____ Don’t know (888)
Fear of Falling study

ID code:______________ Interview completed: Data entered: ________________
Backup Protocols process. This latter information will elaborate on

the information provided in the last column of
Nowadays, most of the information investigators Table 33.1, presented earlier in this chapter. The
retain for a research study is done so electronically. feature box titled “General Tips on Setting Up
For this reason, developing and implementing Data Files” summarizes some of these tips.
backup protocols as part of a data management
system is essential. There are two basic rules of Considerations During
backups: do them regularly and frequently
(McFadden et al., 1995). One never knows when a Proposal Development
file will become corrupted, fall victim to a com- As noted earlier, a good data management system
puter virus or worm, or when the computer hard- will be initiated during the process of preparing a
ware might fail. Therefore, planning for and doing study proposal. At the time of proposal writing,
backups of all files is an absolute necessity. decisions about staffing, project activities and
In addition to actually planning for and doing timelines, computers and software, and storage
the backups, the protocol should include directions needs (e.g., locking filing cabinets) must be made
about where the backup files are to be stored (in so that a realistic budget can be developed.
another location is always best, in case of fire), as Realistic budgets support good data management
well as instructions on how backups can be by providing the necessary financial resources to a
retrieved if that becomes necessary. project. At the time of project planning, the fol-
lowing data management tools and resources
will need to be decided upon, developed, or at least
Final Thoughts on initiated:
Data Management • Management plan and project timelines,
• Job descriptions,
So far this chapter has described the components • Equipment and supply needs (e.g., locking filing
of data management, explained why it is impor- cabinets and data management and analysis soft-
tant, outlined issues to consider, and described ware), and
some of the tools and strategies used to address • Submissions for human subjects protection
these issues. This last section provides some gen- (ethics review) and, if necessary, submissions
eral tips on data management, including what related to the use and protection of personal
things to do at different points in the planning health information (In the USA, HIPAA).
General Tips on Setting Up Data Files • Review data collection tools and ensure that their
design will facilitate accurate data entry,
Tip 1: Remember to include variables on your • Test data entry systems, including data export
data collection forms and in your data files protocols, and
that will help you describe your study later • Set up and test data storage, security, and back-
(e.g., start and end times so you can calculate up protocols.
interview length).
Tip 2: Avoid using a single code to describe all Issues During Data Collection
“missing” data. Instead, use specific codes to
detail the specifics of why data are missing— Many data management problems will first emerge
e.g., interviewer error, refused, skip, not during the actual data collection phase. Some of
asked—proxy, etc. These details can assist the them will be unexpected while others may have
statistician when making decisions about the been overlooked in the original planning. During
analysis.
this phase, most of the data management involves
Tip 3: Remember: Detail is good. Categories can
always be collapsed later, during analysis. following the protocols that were previously set,
Tip 4: When setting up your files, consider refining or adding to them as necessary, and docu-
whether you may need to link the new file to menting all decisions that are made. Specifically,
another one at some point in the future. the investigator will need to:
Include variables that will facilitate these
linkages, and determine if any variable names • Document methods and responses to each
need to be the same between the files. recruitment strategy,
Tip 5: Maintain a file documenting all of your • Document the outcome for each potential partic-
data files, where they are located, and their ipant—who is in and who is out, and why an
passwords (if electronic). individual does not participate or complete the
study,
• Update policies and procedures as new situations
emerge,
Issues During Project Initiation • Maintain and adjust (as necessary) staff monitor-
The startup phase of any project can be hectic, as ing procedures, and
many different tasks must be completed. In rela- • Document as decisions are made (e.g., how to
tion to data management, an investigator needs to code unusual responses).
review all of the data management-related materi-
als developed during the proposal phase of the Issues During Data Preparation
project, and adjust as necessary. In addition, he or and Analysis
she must: Once the data collection is completed, data entry
• Recruit and hire5 individuals who meet the qual- will need to begin. For some studies, data entry can
ifications of the positions available, and will occur simultaneously with the data col-
• Train or arrange training for staff on protection of lection process. Either way, during the time period
human subjects, use and protection of personal
health information (if necessary), and all neces-
sary aspects of their job,
• Decide what policies and procedures will be
needed, and work with staff to prepare a policy
and procedure manual,
• Set up staff monitoring procedures that can be re-
asonably followed (e.g., weekly team meetings),
• Develop systems to track participant recruitment
and outcomes, including how ID numbers will be
assigned,
5In clinical research, the principal investigator may not be
hiring staff, but rather involving therapists or others who

will be engaged in the data collection as part of their daily
duties. Even in these types of situations, investigators must
ensure that the individuals involved in the project have the
qualifications necessary to conduct the duties related to the Figure 33.7 Dr. Finlayson and team members
study. review a policy and procedure manual.
in which the data are being prepared for analysis, research. Western Journal of Nursing Research, 24 (1),
data management protocols will be in the fore- 87–100.
Gassman, J. J., Owen, W. W., Kuntz, T. E., Martin, J. P., &
front. The investigator will need to: Amoroso, W. P. (1995). Data quality assurance, moni-
toring and reporting. Controlled Clinical Trials, 16,
• Implement the data checking and cleaning proto-
104S–136S.
cols, Hosking, J. D., Newhouse, M. M., Bagniewska, A., &
• Document corrections made to the data files, Hawkins, B. S. (1995). Data collection and transcrip-
• Document the construction of derived variables, tion. Controlled Clinical Trials, 16, 66S–103S.
• Document the process of recoding variables, Hulley, S. B., Cummings, S. R., Browner, W. S., Grady, D.,
Hearst, N., & Newman, T. B. (2001). Designing clini-
• Apply weights to the data, if necessary, and doc- cal research (2nd ed.). Philadelphia, PA: Lippincott
ument this process, Williams & Wilkins.
• Transfer, if necessary, the data to the analyst, and Kielhofner, G., Hammel, J., Finlayson, M., Helfrich, C., &
• Ensure that all files are archived for later use and Taylor, R. R. (2004). Documenting outcomes of occu-
pational therapy: The Center for Outcomes Research
sharing.
and Education. American Journal of Occupational
Therapy, 58 (1), 15–23.
Krefting, L. (1991). Rigor in qualitative research: An
Conclusion assessment of trustworthiness. American Journal of
MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B.
This chapter emphasized the critical nature of data (1998). Codebook development for team based qualita-
management and how it influences the quality of tive analysis. Cultural Anthropology Methods, 10 (2),
the data that a project produces. It also emphasized 31–36.
McFadden, E. T., LoPresti, F., Bailey, L. R., Clarke, E., &
that data management involves both the informa- Wilkins, P. C. (1995). Approaches to data management.
tion gathered from participants as well as the proj- Controlled Clinical Trials, 16, 30S–65S.
ect information that facilitates overall study Nyiendo, J., Attwood, M., Llyod, C., Ganger, B., & Haas,
management. Data management is a complex set M. (2002). Data management in practice-based
research. Journal of Manipulative and Physiological
of tasks and processes. The key to doing it well is
Therapeutics, 25 (1), 49–57.
to be logical and methodical, and plan for prob- Pogash, R. M., Boehmer, S. J., Forand, P. E., Dyer, A., &
lems before they emerge. Kunselman, S. J. (2001). Data management procedures
in the asthma clinical research network. Controlled
Clinical Trials, 22, 168S–180S.
REFERENCES Portney, L., & Watkins, M. P. (2000). Foundations of clini-
Aday, L. (1989). Designing and conducting health surveys. cal research: Applications to practice (2nd ed.). Upper
San Francisco, CA: Jossey-Bass. Saddle River, NJ: Prentice-Hall.
Antonakas, C. L., Miller, J. M., & Caruso, C. C. (2002). Shi, L. (1997). Health services research methods. Clifton
Critical elements of documentation in data-based Park, NY: Delmar.
C H A P T E R 3 4
Disseminating Research
Research that is not publicly shared is incomplete. there are multiple ways that research can be dis-
Dissemination is a key step in the research process seminated. Each of these is suited to a particular
and, thus, before any investigation is undertaken, audience and purpose.
the researcher should plan ahead for how and In occupational therapy, the dissemination of
when it will be shared. This chapter discusses the research addresses the following aims:
rationale, process, and range of mechanisms for
• Making new information available to members
disseminating research.
of the profession in order to build and support
Research is undertaken for the larger benefit of
evidence-based practice,
the scientific community and the public. Research
• Making information available to scholars in
dissemination communicates new knowledge to
related disciplines who are doing research in the
these constituencies. Moreover, when disseminat-
same or related areas so that they can incorporate
ing research, the investigator describes the
and build upon one’s methods and findings,
research process so that peer scholars are able to
• Permitting criticism and replication, each of
evaluate its rigor and to replicate or build upon the
which is necessary to the refinement and further
study to advance science. Finally, dissemination
development of professional knowledge,
shares knowledge with stakeholders so it can be
• Making information available to consumers, and
used for practical ends (Mercier, Bordeleau,
• Making information available to entities and
Caron, Garcia, & Latimer, 2004).
persons who fund and/or make decisions and
There is a growing emphasis in many sectors on
policy that impact the availability and delivery of
assuring the utilization of research results. As a
occupational therapy services.
result, researchers are combining traditional con-
cerns for scientific rigor with “new outcomes plan- In planning and implementing a particular form
ning and performance measurement requirements of dissemination, one should always consider its
which focus on the purposes. Below, we
effects and utilization of briefly discuss each of
findings” (Campbell & In planning and in reporting the aims to which dis-
Schutz, 2004, p. 7). This semination can be dir-
growing emphasis on research, investigators ected. It should be kept
ensuring the practical should consider how their in mind that any particu-
impact of findings has lar act of dissemination
underscored the impor-
research can contribute to has the potential to ad-
tance of a comprehensive practice. dress more than one of
approach to dissemina- these aims.
tion. Such an approach
helps ensure that the information generated
through the research reaches the right audiences in Making New Information Available
a format that allows them to effectively use it to to Members of the Profession
change and enhance their behavior and practices.
As discussed in Chapter 41 there is a professional
obligation to base practice on evidence. Many dif-
The Nature and Role ferent forms of evidence can inform practice; how-
ever, the most rigorous is generated through
of Dissemination research. Most research conducted by occupa-
tional therapy researchers will have either direct or
Dissemination refers to the processes by which indirect implications for practice. By making their
researchers inform others within and beyond the findings available, researchers are supporting their
scholarly community about their research process practitioner peers to provide services in line with
and what they have learned from it. As we will see, evidence. In planning and in reporting research,
investigators should consider how their research similar results are obtained when studies are
can contribute to practice. Thus, a critical part of repeated by others, and to test the generalizability
research dissemination is to discuss the practice of findings to different samples and situations.
implications of the findings. Subsequent research that fails to reproduce the ear-
lier study’s findings can help to illuminate flaws
Making Information Available to that were not apparent in the original research, or
Scholars in Related Disciplines to distinguish the context-specific elements of the
findings from those that are common across set-
Occupational therapy researchers conduct their tings. When research findings are replicated by
investigations within an interdisciplinary context. others, the scientific community places more con-
First, occupational therapists mostly use research fidence in them.
methods that were developed in other fields.
Second, occupational therapy researchers are often Making Information Available to
investigating topics on which there is also interdis-
ciplinary research. Further, even when therapists
Consumers and/or Participants
are investigating topics of specific interest to the Consumers include persons who themselves or
profession, it is likely that their findings will have whose family members are receiving or may
some relevance to members of other professions receive occupational therapy services. Consumers
and disciplines. also include advocacy and lobby groups, who rep-
Making one’s research available to an interdis- resent the interests of people in the community for
ciplinary audience is an important way of partici- whom the research has relevance. Research should
pating in the discourse that advances research in a be shared with consumers whenever it can enable
particular area. Reporting findings in an interdisci- them to make informed decisions about the need
plinary context allows others to incorporate and for and likely outcomes of services. This means
build upon one’s methods and findings. It is a way that researchers should be willing to share research
of returning something to the larger scientific com- findings in ways that are accessible to consumers.
munity, from which the profession benefits. Moreover, investigators are ethically obligated to
Moreover, the reputation of occupational thera- provide information about research findings to
pists is enhanced when others encounter quality agencies and/or individuals who participate in the
research conducted by them. Finally, disseminat- research (Sieber, 1992).
ing findings in the interdisciplinary context
informs others of the profession’s particular per- Making Information Available to
spective and concepts. Funding and Decision-Making Entities
Permitting Criticism and Replication Research can serve as a means of demonstrating
the value of occupational therapy services. Such
One of the most demanding aspects of research is research has the potential of influencing persons to
submitting to public scrutiny. Anyone who has make affirmative decisions concerning reimburs-
submitted for review or published research is ing and making available occupational therapy
aware that such efforts almost assuredly generate services.
feedback concerning the limitations and flaws of
the research and its presentation. After one has
worked hard for months or years to plan and Typical Venues for
implement research, it is not easy to hear others
criticize it.
Dissemination
Nonetheless, this critique is an essential aspect
Dissemination can be thought of as falling into two
of all research. By explicating how findings were
broad audience categories: professional/scientific
generated, dissemination allows others to judge
and stakeholders. We will discuss the various
how much confidence they wish to place in the
forms of dissemination for each of these audi-
findings, given the limits and flaws in the particu-
ences.
lar research process. No investigation is ever per-
fect and it is important that others can objectively
critique and learn from it.
Professional/Scientific Audiences
Dissemination is also important so that others The most common ways of disseminating research
can understand and replicate one’s research. to professional and scientific peers occurs in the
Replications are conducted to confirm whether form of presentations and posters at conferences
Chapter 34 Disseminating Research 567
and journal publications. These venues can be build on current thinking and developments within
either peer-reviewed or non-peer-reviewed. Peer- the field of occupational therapy.
reviewed venues are more common for the presen- Each journal has its own requirements for the
tation of the research methodology and findings. format of papers, how many copies must be sub-
Non-peer-reviewed professional publications are mitted, whether electronic copies are required/per-
generally more appropriate for emphasizing the mitted, and so forth. These requirements are noted
significance of research findings for practice. in the guide to authors, which can generally be
found on the journal’s Web site (if it has one)
Peer-Reviewed Dissemination and/or in a copy of the journal itself. Journals ordi-
narily do not have deadlines. An exception is when
In most professional and virtually all scientific
a special issue of a journal is announced.
venues, one must submit the proposed poster,
Generally special issues of a journal focus on a
presentation, or journal article for peer review. The
given research topic, or method, and may involve
review process is used to ensure the quality of
one or more guest editors. These special issues are
information disseminated, and its suitability for
usually announced though a call for papers with a
the particular conference audience or journal read-
specified submission deadline.
ership.
Referees, the people who undertake reviews of
The Purpose of Reviews. Review procedures are papers submitted to journals, usually have experi-
intended to maintain the quality and standard ence in writing for publication, and represent
of conferences and aca- the broad range of profes-
demic journals. For con- sional backgrounds rele-
ference presentations Dissemination can be vant to the journal’s
and posters, this review particular field. In occu-
process focuses on the
thought as falling into two pational therapy, this
quality and suitability of broad audience categories: means referees are likely
submitted abstracts (i.e., to be occupational thera-
brief summaries of pro-
professional/scientific and pists and scholars in the
posed presentations/pos- stakeholders. field of occupational ther-
ters), and is typically apy and related fields,
undertaken by an expert who have the necessary
panel/committee responsible for selection of the publishing experience and expertise in particular
program content. practice areas and/or inquiry method to review and
In contrast, the review process for journal pub- constructively evaluate the types of work submitted
lication involves review of full-length research for publication in occupational therapy journals.
papers (also referred to as manuscripts) written
Blind Review. Many, but not all, scientific/profes-
and submitted to the journal by the researchers,
sional journals utilize blind review. Blind review
which are then reviewed by designated members
means that the review procedure is undertaken by
of the journal’s editorial board or team of referees.
the referee(s) without the author(s) of a manuscript
For occupational therapy journals, peer review
being identified to the referee(s). Blind review pro-
also ensures that published papers reflect, and
cedures are intended to foster fair evaluation of the
quality and standard of the manuscript on its mer-
Choosing Venues for Dissemination its, by minimizing the extent to which knowledge
of the authorship might influence opinions
Dissemination mechanisms should be chosen to expressed by the referee about the work, or bias
ensure that research findings are effectively com- the referee’s judgment.
municated and utilized (Patton, 1997). Thus, it is
important to consider: Consumer Review. Although less common than
review by scientific/professional peers, consumer
• Who should know about the outcomes of this
research? review is increasingly considered important and
• What information will be of most relevance to included for some dissemination venues. Consu-
each of these various audiences (e.g., individu- mer reviews offer the perspective of persons who
als, agencies, policymakers)? participated in the study, or those for whom the
• What will be the most effective mechanisms for research is intended. Consumer review is intended
sharing this information with them? to evaluate the relevance and applicability of re-
search and its presentation for consumer audiences.
AJOT Author Guidelines
The American Journal of Occupational Therapy® Brief Report. A Brief Report (8 to 10 pages)
(AJOT) is the official peer-reviewed journal of the is a short report of original research that is of a
American Occupational Therapy Association. We pilot or exploratory nature or that addresses a
welcome the submission of manuscripts that are very discrete research question and lacks broad
relevant to the study of occupation and the practice implications. References are abbreviated (10
of occupational therapy. Categories of peer- to 15).
reviewed articles include feature-length articles, Case Report. A Case Report (8 to 10 pages) is
case reports, brief reports, and issue papers. a short report of original work that focuses on a
The journal also publishes book reviews, the case example of a clinical situation. The focus can
Evidence-Based Practice Forum, Technology be on a patient or client, a family, an institution, or
and Occupation, Letters to the Editor, and the any other defined unit. The case should represent
official archival material of the Association. elements of practice that are not already repre-
AJOT uses the fifth edition of the Publication sented in the literature. References are abbreviated
Manual of the American Psychological Association (10–15).
(APA) as the style guide. Consult this manual for The Issue Is. Papers submitted for The Issue Is
style questions unless specified otherwise in this (8 to 10 pages) department are those that address
Author’s Guide. timely issues, policies, or professional trends or
One original and three copies of all manu- that express opinion that is supported by cogent
scripts should be sent to Mary A. Corcoran, PhD, argument. Papers for The Issue Is have no abstract;
OTR/L, FAOTA, Editor, 2901 Oak Shadow Drive, references are abbreviated (10–15).
Oak Hill, VA 20171, USA; ajoteditor@aol.com. Letters to the Editor. Letters discussing a recent
Manuscripts must be submitted with the AJOT article or other broad issues relative to the
authors’ explicit written assurance that they are journal are welcome. Letters should ordinarily
not simultaneously under consideration by any not exceed two double-spaced typed pages. Sub-
other publication. The journal cannot assume mission may be via e-mail or hard copy. If using
responsibility for the loss of manuscripts. e-mail, the letter should be pasted in the e-mail
message; do not send as an attachment.
1. Authors’ Responsibility Manuscripts for all categories above, except
Signatures. Before publication of any accepted Letters to the Editor, are peer reviewed.
manuscript, all authors must provide original sig- Evidence-Based Practice Forum and Technol-
natures for the statement of authorship responsibil- ogy and Occupation. Authors interested in submit-
ity, the statement of financial disclosure, and the ting manuscripts for these departments should
statement of copyright release. Signed statements contact the appropriate associate editor before
of authorship responsibility and financial disclo- submission to discuss topics and manuscript
sure must be included with the manuscript submis- preparation guidelines. Linda Tickle-Degnen is
sion (see form at end of this guide). The form the associate editor for Evidence-Based Practice
containing the copyright release statement will be (Tickle@bu.edu), and Roger O. Smith is the
mailed to the corresponding author on acceptance associate editor for Technology and Occupation
of the manuscript (after peer review). (smithro@uwm.edu).
The statement of authorship responsibility is
certification that each author has made substantial 3. Manuscript Preparation
contributions to (a) the conception and design, For format and reference style, consult the APA
acquisition of data, or analysis and interpretation style manual and recent issues of AJOT. Careful
of data; (b) drafting and revising the article; and attention to style details will expedite the peer-
(c) approval of the final version. Further, each review process.
author takes public responsibility for the work. Double-space the entire manuscript, including
Author order. The order of authors in the byline abstract, text, quotations, acknowledgments, tables,
follows APA guidelines. The principle contributor figure legends, and references. Leave 1-inch mar-
appears first, and subsequent names are in order of gins on all sides, and keep the right side unjusti-
decreasing contribution. fied. Number all pages, starting with the abstract
page. Use only standard 12-point font size. Do not
2. Categories of Articles copy pages back to back.
Feature-length article. Feature-length articles (18 Title page. The title should be short and reflect
to 20 pages) include (a) original research reports the primary focus of the paper. List three key
that focus on philosophical, theoretical, educa- words or phrases (not already in the title). List
tional, or practice topics and (b) critical reviews full names, degrees, titles, and affiliations of all
(including meta-analyses) that offer the systematic authors. Designate corresponding author and
review and critical analysis of a body of literature give full address, telephone number, fax number,
as related to occupation and occupational therapy. and e-mail address.
(continued)
568
AJOT Author Guidelines (continued)
Abstract page. A 150-word abstract is required Affix a label with name of first author, figure
for all full-length feature articles, Brief Reports, number, and arrow indicating “top” on the back
and Case Reports. Abstracts may be structured of the figure. Figures may be submitted as
(organized with subheadings Objective, Method, high-quality laser printouts, black-and-white
Results, and Conclusion) or unstructured (narrative glossy prints, or digitized electronic files.
description of the focus and key content of the Provide a caption for each figure. List all captions
paper). The abstract is page 1 of the manuscript. on one page, double-spaced. Place the figure
Acknowledgments page. The acknowledgments caption page after the tables and before the
page follows the last page of the text and precedes figures.
the reference list. Brief acknowledgments may Abbreviations. Do not use abbreviations in the
include names of persons who contributed to the title or abstract of the paper; the use of abbrevi-
research or paper but who are not authors (e.g., ations in the text should be minimal.
statistician) followed by acknowledgments of grant 4. Permissions
support. Prior presentation of the paper at a meet- Authors who wish to reprint tables, figures, or
ing should be briefly described last. lengthy quotations are responsible for obtaining
References. Follow the APA style manual for permission from the original copyright holder.
referencing, listing references in alphabetical order Letters of permission with original signatures must
starting on the page after the acknowledgments be submitted to the editor. AOTA does not reim-
and inserting authors’ surnames and year of publi- burse authors for any expense incurred when
cation for in-text citations. Personal communica- obtaining permission to reprint. The need for per-
tions or other nonretrievable citations are described mission applies to adapted tables and figures as
in the text only with name and date for a person well as exact copies.
and name, date, and address for an organization. Signed statements of permission to publish
Authors are solely responsible for the accuracy must accompany all photographs of identifiable
and completeness of their references and for cor- persons.
rect text citation. Authors must submit signed statements of permis-
The following are examples of commonly used sion from persons cited for personal communica-
reference listings: tions.
Journal article:
5. Manuscript Review
Abreu, B. C., Peloquin, S. M., & Ottenbacher, K.
Manuscripts and reviews are confidential materi-
(1998). Competence in scientific inquiry and
als. The existence of a manuscript under review is
research. American Journal of Occupational
not revealed to anyone beyond the editorial staff.
Therapy, 52, 751–759.
All submitted manuscripts are initially reviewed by
Book with corporate author and author as pub-
the editor for suitability for the journal. Suitable
lisher:
manuscripts are then sent to editorial board mem-
American Psychiatric Association. (1994).
bers or guest reviewers for peer review. The identi-
Diagnostic and statistical manual of mental
ties of the reviewers and of the authors are kept
disorders (4th ed.). Washington, DC: Author.
confidential. Initial review takes approximately 3
Book with author(s):
months; subsequent review of revisions takes less
Frank, G. (2000). Venus on wheels: Two decades
time.
of dialogue on disability, biography, and being
Authors are responsible for ensuring that a
female in America. Los Angeles: University of
blind review process can take place. Except for the
California Press.
title page, manuscripts should contain no identify-
Edited book:
ing names of specific persons or places.
Law, M. (Ed.). (1998). Client-centered occupa-
All accepted manuscripts are subject to copy-
tional therapy. Thorofare, NJ: Slack.
editing. Authors will receive a photocopy of the
For instructions on citing electronic references,
edited manuscript for review and final approval, as
see the following APA Web page:
well as reprint order forms, before publication.
www.apa.org/journals/webref.html.
The author(s) assumes final responsibility for the
Tables. Provide full titles and put each table on
content of the manuscript, including copyediting.
separate pages following the references.
Number the tables consecutively as they appear 6. Copyright and Patent
in the text. Data appearing in tables should sup- On acceptance of the manuscript, authors are
plement, not duplicate, the text. required to convey copyright ownership to AOTA.
Figures and illustrations. Submit one original and Manuscripts published in the journal are copy-
three copies of all figures or illustrations. righted by AOTA and may not be published else-
Number figures in order of mention in the text. where without permission. Permission to reprint
(continued)
569
AJOT Author Guidelines (continued)
journal material for commercial or other purposes _____ Manuscript contains no identifying
must be secured in writing from the director of names of specific persons and places within
AOTA’s Publications Department. the text
Any device, piece of equipment, splint, or other _____ Pages numbered starting with abstract on
item described with explicit directions for construc- page 1
tion in an article submitted to AJOT for publication _____ Written permissions obtained as needed
is not protected by AOTA copyright and can be pro- (photographs, personal communications, publish-
duced for commercial purposes and patented by ers of copyrighted material)
others, unless the item was already patented or _____ Labels on back of all figures
its patent is pending, at the time the article is _____ All material double-spaced (including
submitted. abstract, references, quotations, figure captions)
Checklist for Authors _____ On title page—name, full address, tele-
_____ Original and three copies of manuscript and phone and fax numbers, and e-mail address of
figures corresponding author as well as three key words
_____ Cover letter or phrases
_____ Signed authorship responsibility and financial Note. Do not send a diskette with the initial sub-
disclosure forms mission of a manuscript. A diskette will be
_____ All references checked for accuracy and com- requested if the manuscript is accepted for
pleteness and for exact match between list and text publication.
The Review Process. The process of review for feedback, which provides the rationale for the rec-
conference presentations or posters begins with a ommendation and, when revisions are asked for,
call for abstracts with a deadline. Calls for papers guidance to the author(s) of the manuscript as to
and posters generally occur several months in the kind of revisions required.
advance of the conference. Authors are notified of The journal editor considers the reviewers’
the decision about whether their paper/poster is feedback, decides on a course of action, and then
accepted well in advance of the conference. Each communicates to the author(s) whether the paper is
conference has its own rules for the length and rejected, invited to be resubmitted with revisions,
content of an abstract. Some conferences publish or accepted with minor or no revisions. The editor
abstracts of papers and posters that are selected for ordinarily shares the outcome in a detailed letter,
inclusion in a conference proceeding. Historically, which provides the rationale and spells out any
abstracts were submitted on paper. Many confer- requested revisions.
ences today use electronic formats, such as sub- Different journals have differing standards for
mission via a Web site. and rates of manuscript acceptance. For extremely
Peer review for journal articles requires sub- competitive journals, the majority of manuscripts
mission of a full manuscript that is intended for submitted will be rejected. Authors who have had
publication. The manuscript will be forwarded to articles rejected from these top tier journals often
two to three reviewers whose expertise overlaps find their manuscripts accepted by less competitive
with the methodological and/or substantive con- journals. Experienced investigators choose the
tent of the manuscript. Reviewers ordinarily are level of journal to which they originally submit
provided with detailed guidelines and forms for papers based on their assessment of the quality and
completing the review process and making recom- sophistication of their study as determined by such
mendations about the disposition of the manu- factors as design rigor, sample size, and degree of
script. Generally reviewers are asked to make one innovation.
of the following recommendations: In the case of papers that are accepted for pub-
lication, the most common experience of authors is
• Rejection,
to make fairly substantial revisions, which are then
• Invitation to make major revision and submit for
re-examined by reviewers and/or the editor.
re-review (in which case the revised manuscript
Anyone who wishes to publish research must be
is ordinarily re-reviewed by one or more of the
prepared to accept criticism and have a full meas-
original reviewers), or
ure of patience. After submitting a manuscript, one
• Acceptance pending minor or no revisions.
ordinarily waits 3 to 6 months for initial feedback.
In addition to recommending disposition of the If revisions are required, a similar period after sub-
manuscript, reviewers generally provide detailed mission of the revised manuscript lapses before the
An Example of a Research Abstract opportunities for investigators to share the process

for a Conference and results of recent research on a shorter timeline
than publication. Conferences also provide a
Title: Attending to Clients’ Stories: Use of the unique opportunity for members of the scien-
Occupational Performance History Interview tific/professional community to meet each other,
(OPHI-II) discuss research informally, make connections,
Author(s): Ellie Fossey, Karen Roberts, and and share information that can benefit future
Melanie Gray research, as well as to interact with other stake-
Abstract accepted for paper presentation at holders. The feedback that presenters receive can
the OT AUSTRALIA 20th National Conference, be helpful to preparation of a manuscript for pub-
Canberra 1999.
lication. For these reasons, investigators frequently
Abstract: Attending to clients’ life stories
is an essential part of developing positive rela- seek to present the results of their research first at
tionships with clients, and gaining an under- conferences and later in published format.
standing of their illness experiences, as well as
their life goals, and the part that therapy could Presentations. Presentations of scientific papers
play in enabling clients to achieve these goals. are usually brief (i.e., 10 to 30 minutes). Some
Ultimately, our success in designing therapeutic time is ordinarily scheduled following presenta-
interventions that are meaningful and relevant to tions for brief public discussion. Verbal presenta-
our clients depends on our ability to develop tions are primarily an oral medium so they rely on
such relationships and understanding. both the content of the presentation and its deliv-
The Occupational Performance History ery by the presenter(s).
Interview (OPHI-II) (Kielhofner et al., 2004) is a A good presentation is characterized by clarity,
revised version of the OPHI currently being conciseness, and attention to the target audience.
developed through an international collaboration,
Generally, the quality of a presentation depends
to which the authors are contributors. This inter-
view gathers information about a person’s occu- not only on the verbal content, but also on the
pational identity (self-understanding), perceived graphic representation of the research. PowerPoint
competence in occupational performance, and presentations are typical in modern conferences
his/her environment, as well as providing a and they may be supplemented with audience
framework to explore the direction, and impor- handouts. The use of these elements allows the
tant events in the person’s life story. presenter to emphasize major points, to supple-
This paper will briefly describe the OPHI-II, ment verbal presentation with visual illustration,
then present occupational narratives (stories), and provide attendees with information that goes
obtained using OPHI-II, to illustrate the richness beyond what can be presented within the time limit
of information and understanding of clients’ lives
of the presentation (Figure 34.1).
that may be gained with this tool. Stories from
people in Australian rehabilitation and commu-
Scientific Posters. Poster presentations are ordi-
nity settings will be included to discuss their
therapeutic implications, and illustrate some narily exhibited in a large room or hall that accom-
potential uses for the OPHI-II in occupational modates a number of posters simultaneously.
therapy practice. Conference attendees view and select those they
wish to read in detail. The authors are expected to
be present during scheduled poster sessions, so
they can further explain the content of their posters
in response to questions. Posters may also be avail-
author(s) receive the second round of feedback.
able for viewing outside the scheduled poster ses-
Often at this stage the author must make additional
sion, depending on the conference rules.
(usually more minor) edits and submit a manu-
The essential feature of a good poster is that the
script, which is then copyedited in most instances.
message is clear and understandable without the
Some months later, and prior to publication, the
presenter, and that it achieves a balance between
author ordinarily receives a galley proof (i.e., a
words and graphics. A range of visual techniques
facsimile of the article as it will appear in the jour-
can assist in presenting information in interesting
nal), which must be checked for accuracy and
and informative ways. These include, for instance:
returned with any corrections to the journal. A
photographs; diagrams; tables; graphs; and layout
minimum of 6 months can be expected to elapse
methods, such as flow charts and dot points.
from the time of submission until publication, and
Conference organizers typically provide spe-
more often the process takes upwards of a year.
cific guidelines about poster content, format,
Oral Presentations and Posters at Conferences. and size requirements. Familiarity with these
Professional and scientific conferences provide requirements and attention to production design,
Figure 34.1 Researchers disseminate

results in a variety of ways, including
poster presentations at national pro-
fessional conferences such as the
annual meeting of the American
Occupational Therapy Association,
through publications in journals such
as American Journal of Occupational
Therapy, and through local media such
as local news programs.
clarity of content, colors, layout, and finishing are or to present a poster at a conference involves sev-
important to maximize effectiveness. It is impor- eral considerations. Generally, oral presentations
tant to be selective—that is, to include the key are more competitive than posters, so depending
information to make the research understandable on the overall level of acceptance rates for papers
to the reader, but not to overload the poster with at a conference, one may consider the likelihood
details. of having a verbal presentation versus a poster
accepted. Poster sessions are also more likely to
Choosing a Poster or Verbal Format for Presen- accept presentations of preliminary findings and
tation. Choosing whether to orally present a paper research in process.
Posters are primarily a visual medium for shar- Invited Presentations. Conference organizers
ing information. Poster presentations provide and academic and practice organizations and asso-
graphic and textual means to illustrate studies and ciations often invite outside speakers to present
their outcomes. They are particularly valuable for information generated from research. While the
information that is best presented using schematic specific content of these presentations is not peer
diagrams and flow charts. The fact that a poster reviewed, it is ordinarily the case that speakers
can be viewed at one’s own pace and discretion (and sometimes their topics) are chosen because
may make it possible for others to better absorb the of the positive reputation of the quality of the
information than from verbal presentation. Since speaker (and the research presented). Some invited
conference poster sessions involve face-to-face presentations are associated with awards or honors
interaction, they tend to facilitate networking with and the choice of speakers and topics is highly
others interested in related research. selective. Other venues, such as a routine research
Verbally presenting a paper allows one to share colloquium or “brown bag” lunch presenta-
information with an audience in real time. For tions offered in an academic department, have
many investigators, it is the only time one shares open invitations for persons of varying levels of
research face to face with members of the scien- research accomplishment to present their work
tific and professional community. Since much of and obtain feedback. All of these venues from the
research can involve long hours of private writing most prestigious to the routine can be important
and interaction with others through written means, opportunities for sharing information about
the verbal presentation is something many re- research.
searchers value. Presentations are usually compet-
itively selected, so the fact of presenting research Continuing Education. Continuing education
gives it an air of authority. All in all, presentations provides an important means of ensuring that pro-
are a time for researchers to engage in the most fessionals and researchers remain current in their
public aspect of the scientific process. knowledge and skills. The typical vehicle for con-
tinuing education is a workshop (ranging from a
Summary: The Importance of Peer-Reviewed few hours to several days). Typically one or more
Dissemination. Dissemination through peer- persons, who are recognized experts, will organize
reviewed channels is essential to all research. Its a program of sequential topics in a specific area of
importance is linked to the fact that the peer review interest. Such workshops may provide opportuni-
process provides quality control. Presentations, ties for related studies to be presented and synthe-
posters, and published articles that have gone sized. Workshops are also an excellent venue for
through a review process have been scrutinized. discussions of the practical challenges of doing
The peer review process assures that investigators research, and the practice implications of findings.
have fully described their methods to allow evalu-
ation of rigor and replication. It also serves to Books. In some fields, it is common to report the
ensure that the claims about discovered knowledge results of research in a full book. It is not com-
made by the author(s) are warranted and that ade- monplace in occupational therapy. However, books
quate attention is paid to the limitations of the do often describe, summarize, and synthesize
study. Thus peer review often improves the quality research. For example, occupational therapy text-
of the information presented about both the books often make reference to research findings in
research process and findings to better serve those discussing practice. In addition, books that present
listening to, or reading, that information. theoretical models typically describe the kinds of
research that has been conducted to develop, apply,
Non-Peer-Reviewed Presentations and test those models.
and Publications
Professional Publications and Newsletters. A
Non-peer-reviewed venues for dissemination also variety of professional magazines and newsletters
serve important roles. These venues include: can be useful resources for sharing research find-
• Invited presentations, ings and its implications. OT Practice, published
• Continuing education, by the American Occupational Therapy Associa-
• Books, tion, is one example of such a professional maga-
• Professional publications and newsletters, zine. While it does not feature the more technical
• Nonprint materials containing information about discussions of research that appear in refereed
research findings and their implications. journals, it can be an appropriate venue to dis-
cussing how research findings can be integrated
Each of these venues is described briefly below. into practice.
Newsletters are another appropriate vehicle for • Procedures for sharing findings with the study
discussing the applied relevance of research. Some participants;
large institutional research programs and federal • Public presentations to consumer and community
agencies that fund research also produce publica- groups and collaborating organizations;
tions that share research findings. Finally, there are • Reports to government and private agencies, leg-
agencies whose purpose is to disseminate research. islative bodies, and public officials;
They produce a variety of publications that share • Web sites, targeted brochures, and other media
research. Most of these venues secure articles by for laypersons; and
inviting authors to write, accepting contributed • Releases to the popular press and media.
articles, and using in-house writers to compose
articles. If there are not published guidelines for Each of these means offers opportunities to dis-
accepting contributed papers, this information can seminate research to audiences beyond scholarly
usually be obtained from the editor. In most cases, and professional communities. They are discussed
it is up to the editor of the publication to decide below.
whether or not to include a particular topic or
paper. Sharing Information
with Research Participants
Nonprint Materials Containing Information
About Research Findings and Their Implica- There is an ethical responsibility to dissemi-
tions. With the growth of the Internet, a wide nate information about research findings and out-
range of electronic sources now contain informa- comes with research participants (Sieber, 1992).
tion related to research. These include sites spe- This responsibility is ordinarily not fulfilled by
cific to areas of research, or even specific to publication of results. Therefore, other dissemina-
individual research projects. Often Web-based dis- tion mechanisms should be considered. One such
semination can serve both professional and other mechanism for disseminating information to
stakeholder audiences. Information can also be research participants is to prepare and send a writ-
made available in different formats to suit these ten summary of the research findings to all partici-
constituencies. pants and participating agencies. This approach has
the advantages of being both low cost and time effi-
cient, so it is particularly well suited to studies with
Stakeholder Audiences large numbers of participants. Limitations of this
approach are that it does not promote discussion
Historically, the focus on research dissemination about the potential uses of the research findings,
was to the scientific and professional communities nor does it enable participants to give feedback to
through the kinds of means discussed above. There the researchers.
is increasing emphasis today on sharing research The use of technologies, such as e-mail, Web
with various stakeholder audiences. Stakeholders sites, and Internet discussion groups, have the
refer to anyone outside the scientific and profes- potential to both efficiently disseminate research
sional community who may be informed or influ- findings and enhance dialogue among researchers,
enced by the research findings either in their participants, and relevant community groups about
personal lives or in the exercise of their responsi- research. For this approach to be effective, use of
bilities. These include but are not limited to: these technologies must be widespread among the
relevant stakeholder groups. Face-to-face meetings
• Individuals and agencies who participate in the
with participants, participating agencies, and com-
research,
munity groups provide more targeted opportunities
• Consumers of health services for whom the
to discuss findings with relevant audiences and to
research has relevance,
gain feedback from them (McConnell & Kerbs,
• Officials and agencies who make decisions and
1993).
policy that might be informed by the research,
Individually tailored feedback is labor intensive,
• Entities who address needs and/or fund services
but particularly useful in research involving per-
related to the research, and
formance-based assessments and disempowered or
• The general public whose attitudes or behavior
marginalized groups (Fossey, Epstein, Findlay,
might be influenced by the research findings.
Plant, & Harvey, 2002). For example, in an
The avenues for disseminating research to these Australian study involving participants with psy-
audiences are multiple. Major ones include: chiatric disabilities, Fossey et al. (2002) developed
and evaluated a process for sharing individual feed- • What methods of information sharing are likely
back with the participants about their occupational to empower the research participants to engage
performance, based on results of the Assessment of with the material and identify its potential uses/
Motor and Process Skills (AMPS) (Fisher, 2003). implications for them?
The investigators prepared written information • What resources are available for sharing the find-
for each participant, adopting a strengths-based ings?
approach, in which the person’s occupational per-
formance strengths were described, and some ways Public Presentations to Consumer
in which these strengths could be used to overcome and Community Groups,
areas of difficulty were suggested. This written Collaborating Organizations
information was shared face to face, accompanied
by verbal explanation. Participants reported that the Research findings may be shared with consumer
focus of the feedback on strengths was helpful, groups through a variety of channels. One com-
while the verbal explanation was seen as essential monly utilized channel for presentation-based
to enabling their understanding and use of the writ- dissemination to consumers includes conferences
ten information. and conventions hosted or co-hosted by consumer
Different research traditions have approached interest groups, or by professional organizations
the issue of dissemination to stakeholders in differ- that cater to consumer interest groups. For exam-
ent ways. For example, in the quantitative tradition, ple, the National Fibromyalgia Awareness Cam-
research tends to be viewed as the researcher’s paign and Whole Health co-hosted a conference in
expertise, product, and property. Thus providing Chicago that featured the research findings of
feedback is usually viewed as a didactic process, medical and rehabilitation professionals who treat
in which the researcher gives information to individuals with fibromyalgia. Similarly, a bian-
stakeholders (Fossey et al., 2002). In contrast, nual, international conference sponsored by the
qualitative research traditions, and particularly par- American Association for Chronic Fatigue
ticipatory inquiry approaches, view knowledge Syndrome offers a “patient conference day” during
production as a shared process and findings as which research-based presentations are distinctly
shared property. Feedback then is considered inte- tailored to consumers attending the conference.
gral to the research process, being a mechanism for Presentations are selected based on their relevance
ensuring validity and sharing power in the research to patients and clinical utility. Similarly, the Inter-
relationship (Fossey et al., 2002; Guba & Lincoln, national Association of Psychosocial Rehabili-
1989; Patton, 1997). In participatory research, tation Services (IAPSRS) in the USA and The
information sharing tends to be iterative, rather Mental Health Services of Australia and New
than unidirectional. Thus, sharing information with Zealand (The MHS) respectively run integrated
the research participants is an essential part of the conferences that foster mutual dissemination and
research process. learning opportunities for consumers, care givers,
and professional groups in the mental health field.
Choosing an Approach to Dissemination A second common means by which research
findings may be disseminated to consumers is
Choosing one’s approach to dissemination with through more informal, direct presentations to
participants depends on the nature of the research consumers within their own organizational con-
process, the kind of information generated, and the texts. For example, a researcher might give a pres-
audience being addressed. The following questions entation at a self-help group meeting, weekly staff
are helpful to consider in developing mechanisms meeting, or board meeting of a consumer organi-
for sharing information about research in these zation with whom he or she is collaborating. Alter-
forums: natively, if the consumer group is not a research
• What information from this research project collaborator, research findings may be shared in a
could be helpful, or useful to the research partic- public presentation to inform the consumer/lobby
ipants? group about findings that have relevant implica-
• What methods of information sharing are likely tions (e.g., for quality of life or service delivered).
to facilitate respectful and sensitive communica- Such presentations may also serve to forge new
tion with the research participants? collaborative relationships with lobby groups and
• What methods of information sharing are most organizations. Recent advances of computer-based
likely to facilitate the research participants’ tele-health and tele-rehabilitation technologies
understanding of this research? also allow for public presentations of research
findings to be disseminated through online broad- puter disks, or CDs, are inexpensive and easy to
casting and DVDs. work with but documents must be converted to
universally acceptable formats (e.g., converting
Reports to Government and Private files to text-only formats, or using an Adobe file).
Agencies, Legislative Bodies, and They offer the advantage of being readily convert-
Public Officials ible to accessible formats for persons with some
kinds of disabilities (e.g., audio-translation by
It is not unusual for governmental agencies or bod- computer software for people with visual impair-
ies to ask for reports of research findings to use in ment). However, they are not yet as appropriate for
their deliberations as they set policies and make mass distribution as printed material because they
laws. For example, the Office of the Surgeon require consumers to have access to computers and
General may request a report on the prevalence of computer proficiency.
a given health condition so that appropriate Web sites are a common means of disseminat-
resources can be allocated to prevent. In addition, ing research-based information to consumers.
researchers can offer to make such reports avail- Though development of a Web site, or Web page,
able, even when they are not requested, as a means requires training and knowledge in Web site design
of facilitating public awareness about a key issue, and construction, once developed, Web sites are
or in an effort to advocate for increased support relatively easy to maintain and update. All con-
and funding for continued research and resources sumer-based Web sites need to be approved so
to address a given issue. that they are automatically accessible to individ-
The following is an example of dissemination uals with visual and auditory impairments. The
to a government body. The National Institutes “Bobby-approval” is perhaps the most stringent of
for Disability and Rehabilitation Research funded many available means of evaluating a Web site to
a study to examine the impact of an alternative ensure that barriers to accessibility are eliminated.
financing program for disabled persons to obtain Examples of Web site barriers to accessibility
loans to finance needed assistive technology. include audio messages displayed in the absence of
Under this study, two occupational therapy inves- a written transcript (for hearing impairment), or
tigators, Hammel and Finlayson, developed a pictures without written descriptions that can be
Web-based data management and outcomes sys- broadcast through an audio device that describes
tem (Hammel, Finlayson, & Lastowski, 2003; the contents of that picture (for visual impairment).
Finlayson & Hammel, 2003) to study the impact of Bobby approval can be accomplished by submit-
the program. They produced annual reports to ting the Web site address to the following Web site
inform U.S. congressional policymakers about the for approval: http://bobby.watchfire.com/bobby/
outcomes of their federal appropriations. The evi- html/en/index.jsp.
dence they shared with Congress was used to jus- Web sites or Web pages that are well designed
tify sustaining and expanding the program funding and easy to navigate are easily accessible by any
from $3.8 million in 2000 to $35.8 million alloca- individual with access to a computer. They also
tion in 2003. As with this example, dissemination offer the research a greater degree of visibility.
to governmental agencies can have a significant One example of a Web site that was designed for
impact in directly influencing policymaking and accessibility by individuals with chronic fatigue
systems change. syndrome was developed by the third author:
http://www.ahs.uic.edu/ahs/files/ot/bookler/CFS_
Web Sites, Targeted Brochures, Website/index.htm.
This Web site is a product of a field-initiated
and Other Media for Laypersons
research and demonstration program for individu-
Research that leads to consumer-relevant informa- als with chronic fatigue syndrome funded by the
tion and products, such as educational curricula, National Institute on Disability and Rehabilitation
resource directories, treatment tips, prevention Research. The Web site contains educational infor-
guidelines, and frequently answered questions, can mation about chronic fatigue syndrome, a copy of
be made available to the public through linkable the curriculum used as the basis for the group
Web sites, printed brochures or booklets, and com- phase of the rehabilitation program, and a copy of
puter media. Printed materials are the most easily the resource directory that contains information
accessed by individuals capable of reading print, about referrals to medical and legal professionals
but they are also one of the most expensive and and other essential information necessary for con-
time-consuming to produce and distribute. Com- sumer rights and access to employment, trans-
portation, food, and housing. Both the curriculum to the various constituencies, who have a right to
and resource directory were co-developed by know about and who could potentially benefit
research staff and participants. from awareness of the research, it is clear that sub-
stantial energy and time must be devoted to dis-
Releases to the Popular semination. Without this expenditure of effort,
research will not realize its potential value and
Press and Media
impact.
Everyone has read, heard on the radio, or seen on
television reports of recent research findings con- REFERENCES
sidered to have importance for the general public. Campbell, M. L., & Schutz, W. V. (2004). The Research
A number of strategies can be used to foster cov- Exchange (Vol. 9, No. 2). Austin, TX: National Center
for the Dissemination of Disability Research,
erage of occupational therapy research in the pop- Southwest Educational Development Laboratory.
ular media. Most medical centers, colleges, and Finlayson, M., & Hammel, J. (2003). Providing alternative
universities have marketing or “public relations” financing for assistive technology: Outcomes over 20
departments whose personnel are responsible for months. Journal of Disability Policy Studies, 14(2),
communicating with the press. Working with these 109–118.
Fisher, A. G. (2003). AMPS Assessment of Motor and
communications personnel is typically the best Process Skills, Volume 1: Development, standardization
strategy for getting media coverage of research. and administration manual (5th ed.). Fort Collins, CO:
Nonetheless, individual researchers can directly Three Star Press.
contact local TV or newspaper health writers to Fossey, E., Epstein, M., Findlay, R., Plant, G., & Harvey,
C. (2002). Creating a positive experience of research
inform them of newsworthy research findings. for people with psychiatric disabilities by sharing feed-
Sometimes an investigator can foster the potential back. Psychiatric Rehabilitation Journal, 25(4),
for media coverage by being responsive to lay 369–378.
requests for information. The following is an Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation
example. evaluation. Newbury Park, CA: SAGE Publications.
Hammel J., Finlayson M., & Lastowski, S. (2003). Using
Research that led to the development and participatory action research to create a shared assistive
investigation of a program designed to reduce fear technology alternative financing outcomes database and
of falling among community-dwelling seniors to effect social action systems change. Journal of
(Tennstedt et al., 1998) was the subject of a Disability Policy Studies, 14(2), 98–108.
Kielhofner, G., Mallinson, T., Crawford, C., Nowak, M.,
National Public Radio (NPR) broadcast of a story Rigby, M., Henry, A., & Walens, D. (2004). The
on fear of falling after one of the occupational Occupational Performance History Interview-II (ver-
therapy researchers associated with that program sion 2.1). Model of Human Occupation Clearinghouse,
responded to questions posed by a woman who Department of Occupational Therapy, College of
was concerned about her mother. Following the Applied Health Sciences, University of Illinois at
Chicago.
telephone conversation, that woman, who hap- McConnell, W. A., & Kerbs, J. J. (1993). Providing feed-
pened to be a health correspondent for NPR, back in research with human subjects. Professional
pitched the idea of a story on fear of falling to her Psychology: Research and Practice, 24(3), 266–270.
editor. Mercier, C., Bordeleau, M., Caron, J., Garcia, A., &
Latimer, E. (2004). Conditions facilitating knowledge
exchange between rehabilitation and research teams—a
study. Psychiatric Rehabilitation Journal, 28(1), 55–62.
Conclusion Patton, M. (1997). Utilization-focused evaluation. Beverly
Hills, CA: SAGE Publications.
Sieber, J. E. (1992). Planning ethically responsible
This chapter covered the nature and role of dis- research: A guide for students and internal review
semination in research, reviewing a range of boards. Newbury Park, CA: SAGE Publications.
options for disseminating research. As stated at the Tennstedt, S., Howland, J., Lachman, M., Peterson, E.,
outset, dissemination is an essential component of Kasten, L., & Jette, A. (1998). A randomized, con-
trolled trial of a group intervention to reduce fear of
any inquiry. In this age of information, the options falling and associated activity restriction in older
for sharing research are myriad. When the investi- adults. Journals of Gerontology Series B-Psychological
gator takes seriously the obligation to disseminate Sciences & Social Sciences, 53(6), 384–392.
C H A P T E R 3 5
Writing a Research Report

This chapter discusses the process of writing a uments the unfolding logic and procedures of the
research report. As noted in Chapter 34, research study, but also creates material that is eventually
can be documented in many different formats. integrated into one or more research reports that
Most common is the research article intended for flow from it.
publication in a refereed scientific/professional
journal. Consequently, this chapter is organized
around creating such a paper. At the same time, the Structure and Format
discussion will be useful to the writer who is
preparing another type of document such as a of a Research Paper
research proposal, thesis, or dissertation.1
In addition to discussing the content of a Research papers, or reports, are typically written
research paper, we also address the underlying task according to a well-known and defined structure,
of writing. Writing is absolutely necessary for which includes the following components:
research. Without it, one cannot garner the neces- • Abstract,
sary approval and resources for research. More • Introduction, purpose, and significance,
importantly, without writing, research is not shared • Literature review,
with the scientific and professional community. • Statement of the research problem,
For all practical purposes, an unpublished study • Statement of the research question and/or
never took place. Therefore, anyone who wishes to hypotheses,
do research must commit to the task of writing. • Methods (design, sample, procedures),
Investigators have a wide range of reactions to • Results (findings),
the obligation of writing. Nonetheless, writing a • Discussion, and
research report can be a satisfying culmination to • Conclusion.
the process of inquiry. It is, after all, an opportu-
nity to share one’s discovery. We briefly discuss each below. While all of
these components are similar across topics and
methods, there is variability in the style of research
When to Begin Writing

papers. Thus, while these components are gener-
ally part of a paper reporting research findings,
how they are sequenced and integrated into the
The process of writing a research report should paper can differ with the discipline, research
begin when the research is first conceived and con- method of the study, or the journal.
tinue throughout the investigation. This is a wise One should seek to identify both the targeted
approach to writing since the content of a research journal or other venue for publication and the
report mainly follows the sequence of action that is intended audience for a paper as soon as possible.
necessary to carry out a study. By writing up a Identifying the journal will provide technical
study as one proceeds through it, one not only doc- information (e.g., page limit, manuscript prepara-
tion format) necessary for preparing a final manu-
1Some graduate programs traditionally required lengthier script. Identifying the audience means becoming
theses, but it is increasingly recognized that these are not the familiar with:
standard formats in which career researchers report their
work. Consequently, more and more programs require the • Who is likely to read the manuscript,
master’s thesis presented in the standard format and length • What their level and type of knowledge is likely
of a research paper. Some programs are also following a
to be, and
model for doctoral dissertation work in which the disserta-
tion is composed of “chapters” that are each, in effect, a • What perspectives they are likely to bring to
stand-alone, publishable research report. These reports are reading the paper.
either derived from a series of studies that make up the dis-
sertation research, or represent several papers reporting dif- This information is essential not only to writing
ferent aspects of the research. a paper that will be understood by its intended
Chapter 35 Writing a Research Report 579
An informative abstract will describe the fol-

Every journal follows a particular referencing lowing elements:
and formatting guideline, often termed an edito-
rial style. Editorial style refers to the set of rules • Why the study was conducted/the importance of
that are followed to ensure that material pub- the research topic,
lished in a particular journal is presented consis- • What was done in the study,
tently. • Who the subjects/participants were,
The most common one used by occupational
• What the main results/findings were, and
therapy and related social and behavioral science
journals is the American Psychological Associa- • What the implications are for practice, theory,
tion’s APA Style. Details about its Publication and/or future research (Brown, 1996; Hocking &
Manual (American Psychological Association, Wallen, 1999).
2001), as well as information about self-teaching
When these elements are all included, the
materials for learning APA Style, and guidelines
for citing electronic media and creating effective abstract will be informative to readers and make it
visual materials can be found online at the Web clear what the benefit of reading the entire paper
site http://www.apastyle.org/. would be (Brown, 1996).
Before submitting a manuscript to a journal,
one must become knowledgeable about the edito- Introduction, Statement
rial style required by the journal and ensure that of Purpose, Significance
the manuscript submitted adheres to the format
in all its details. The introduction to a research paper provides the
context for what follows in the rest of the paper. In
other words, it sets the scene by outlining:
audience, but also to its being considered appro- • The area and topic to which the study belongs,
priate for publication by the chosen journal. • The nature of the problem or issue being
It is invaluable to read research papers from the addressed by the study, and
journal in which one intends to publish and papers • The relevance of the study to the professional
on the same research topic and using the same and/or scientific community that is assumed to be
methods as one is using. When reading, one should the audience for the paper.
pay attention to how each of the components of the
The introduction section typically serves to
research paper is written. This will help one
frame the research topic. As such, it may define
become familiar with subtle but important stylistic
and discuss key concepts/theory pivotal to under-
and technical differences that characterize differ-
standing the research topic.
ent types of research papers. Finally, reading
The purpose of the paper is generally outlined
research papers will provide examples of how the
at the end of the introduction. This statement of
components, discussed below, are best organized
purpose may be quite closely related to the aim of
and incorporated into the paper one plans to write.
the research. However, it should communicate the
aim of the paper, rather than that of the overall
The Abstract study, unless the two are exactly the same. Often,
An abstract is a summary of the key features of the however, a research paper will report only one
paper. Arguably, it is the most important part of the aspect of a larger study. The author(s) will have
paper since its purpose is to enable the reader to decided why to present the particular set of find-
decide whether to read the entire paper (Brown, ings that will be reported in the paper. It is this
1996; Hocking & Wallen, 1999). In the end, more rationale that constitutes the aim of the paper, and
people will read an abstract than will go on to read which will be more specific than the aim of the
the entire paper. overall study.
Abstracts should be written to stand alone, that The introduction will also often include a state-
is, to be understandable without a reading of the ment of the significance of the study, although this
entire paper. Moreover, the abstract should convey element can also be included as part of the conclu-
the main messages of the paper. A good abstract sion of a literature review. Wherever the signifi-
does more than provide readers with an overview cance is noted, it should address the question of
of the contents (Brown, 1996). Using phrases such why the study needed to be done. Thomas (2000)
as “the results are described” and “the implications suggests that a significance statement should be
are discussed” is insufficient. Rather, one should phrased along the following lines: “The problem
succinctly characterize the actual results and their I am studying affects lots of people in a particu-
main implications. larly unfortunate way and/or costs a lot of money”
Abstracts, like entire research reports, can vary in Abstract by Farnworth, Nitikin, and Fossey
their format according to the methods used in the (2004)
study. Below are two examples of abstracts. The
first is from a quantitative study and the second is Institutional environments are challenging
from a qualitative study. settings in which to provide rehabilitation.
This study describes the time use of a group
of inpatients, the majority diagnosed with
Abstract by Roche and Taylor (2005) schizophrenia, in a secure forensic psychiatric
unit in Australia. Time diaries, interviews,
Existing studies have shown that individuals and field notes were collected over 5 weeks.
with chronic fatigue syndrome (CFS) demon- Eight participants completed time diaries
strate functional impairment in a number for 2 consecutive days, of whom five were
of domains related to occupational participa- also interviewed using the Occupational
tion. Researchers have not yet explored Performance History Interview-II.
whether coping styles may be associated Participants’ time use was dominated by
with occupational participation in individuals personal care and leisure occupations. In gen-
with this condition. The aim of this study eral, participants were dissatisfied with their
was to examine the effects of coping styles time use, describing themselves as “bored”
on occupational participation among adults or “killing time.” Many perceived that the
with CFS. We hypothesized that occupational environment created barriers to their partici-
participation would be associated with pation in valued occupations, yet some also
coping strategies oriented toward informa- found occupations that provided solace, chal-
tion seeking and maintaining activity, and lenge, or connection with the outside world.
that this relationship would endure despite The findings indicate the importance of
individual differences in illness severity. understanding individuals’ unique occupa-
The study used a cross-sectional design to tional histories, interests, and skills to create
describe the associations between coping opportunities to engage them in relevant
and occupational participation for 47 indi- occupations that utilize personal resources,
viduals diagnosed with CFS. Findings from as part of forensic rehabilitation programs,
linear regression analysis revealed that the and the utility of the Occupational Perfor-
coping style of maintaining activity was posi- mance History Interview-II in this context.
tively associated with occupational participa- Further research exploring patient and staff
tion whereas accommodating to the illness perspectives on the challenges of occupa-
was negatively associated. Implications of tional programming in forensic settings and
the findings for continued research and clini- the longitudinal impact of such programming
cal practice in occupational therapy are dis- on inpatients’ occupational functioning,
cussed. health, and well-being is recommended.
(p. 35). In essence, then, the significance of a study argument that leads directly to the research ques-
is the underlying reason that it was important to tion and study. A literature review that simply
undertake the study. reports previous research is inadequate.
Before one begins a study, a review of the liter-
ature is undertaken to identify what is already
Literature Review
known about a topic and the methods with which
A literature review is a critical evaluation of exist- it is typically studied. The purpose of that initial
ing literature relevant to the topic under study literature review is formative—that is, it helps
(DePoy & Gitlin, 1993). A good literature review shape the development of the research plan.
informs, evaluates, and integrates relevant existing Generally, much of the literature reviewed initially
literature (Thomas, 2000). If the literature review will be incorporated into the literature review writ-
is done well, it will make apparent to the reader ten for the research report. If the report presents a
how the research was a logical next step in build- subset of the study findings relevant to a narrower
ing knowledge and/or how the research fills a crit- topic than the overall study topic, then the review
ical gap in an area of knowledge (DePoy & Gitlin, will also tend to be more narrowly focused than
1993). Thus, in writing the literature review for a the original literature review.
research paper, it is important to make sure that it Before preparing the literature review for a
is organized so as to tell a story, or provide an research paper, the investigator should update the
original literature review with any newly published • Evaluate the strengths and weakness of existing
relevant research and/or theory. If one is preparing research approaches to studying the topic, and
to report an aspect of the findings from a large • Make an argument for why any conflicting or dif-
study, the nature of particular findings may need to fering research findings exist (if they exist).
be framed with different literature than that origi-
nally reviewed in planning the study. Such a DePoy and Gitlin (1993) suggest the following
review is undertaken to appropriately context the structure for organizing a literature review:
particular findings that will be reported in the • Introduction, defining the focus and scope of the
paper. This kind of review typically is related to review,
the initial literature review, but may focus in one • Discussion of each specific concept, principle, or
area in more depth, or it may pursue a topic related theory in the current literature on the topic,
to the initial literature review that was not origi- • Brief overview of key studies, compared in par-
nally examined. Both quantitative and qualitative allel rather than serially to achieve a critical
research may produce results/findings that require appraisal of the current research,
the investigator to review a new body of literature • Integration of the work reviewed, identifying the
to prepare a report. relationships, inconsistencies among findings
The major tasks in preparing and writing a lit- across studies, controversies, and gaps in the lit-
erature review, therefore, are to locate, critically erature,
appraise, and then summarize the previous • Identification of the niche in the current knowl-
research relevant to the topic. One should address edge base that your research fills, and
the issues and controversies that are raised in the • Justification/rationale for the study and its
literature, and identify the gaps in the current design.
knowledge base that provide the rationale for the
research and its design. Brown (1996) poses seven These components are typical of many litera-
questions that should be answered in writing a ture reviews, but it should be remembered that the
review of literature for a research paper: review is designed to characterize the content of
literature that is relevant to the topic, and that
• Why is this topic important? makes sense of the chosen question and research
• What is known about the topic? methods. Thus, there are instances in which the
• What is unknown about this topic? state of the literature, or the nature of the research
• Why are some things unknown? question, may dictate a somewhat different struc-
• Why should the gaps be filled? ture. For example, if a study examines something
• Which gaps does one propose to fill and has one for which little or no previous research is reported,
chosen them? the literature review may focus on providing the
• How does one propose to fill them? rationale for the importance of the topic area or
In the literature review section of a research extrapolate from literature that is only partly
paper, it will not be possible to discuss the full related to the research question. Also, some jour-
scope and volume of literature on the research nals have formats that require literature to be suc-
topic. The aim of a literature review is not to cinctly reviewed as part of the introduction, rather
demonstrate the breadth and depth of the investi- than as a separate section. In these cases, the
gator’s knowledge of the literature. Rather the aim review may be structured differently, but still
is to address the above questions and, in so doing, needs to appraise the key literature of relevance.
to make apparent the underlying logic for under-
taking the study. Statement of Research/Scholarly Problem
Both accurate representation and critical
appraisal of the literature are important to an effec- An explicit statement of the nature of problem
tive literature review. Critical appraisal will: being addressed by the study tells the reader why
this research is important. In other words, it lets
• Identify the current trends and ways of thinking readers know why they should care about the
about the topic and how to research it, research reported in the paper. For this reason, the
• Identify the boundaries of the literature (e.g., statement of the research problem is best located
what particular populations, settings, and per- early in the paper, generally in the introduction. In
spectives were studied in the previous research?), journal formats that allow separate introduction
• Illuminate the gaps in the current knowledge and literature review sections, the problem area
base and the way in which it has developed, may be more broadly outlined in the introduction,
and the specific nature of problem more narrowly How these elements are described in a particu-
defined in the conclusion of the literature review. lar research paper will depend on the actual meth-
ods used. For example, when an investigator has
used a well-known quantitative design and com-
Statement of Research/Scholarly
mon statistical analyses, it may be sufficient sim-
Aim, Questions, and/or Hypotheses ply to name the design and analysis technique with
All research inquiry is guided by its aim and the minimal details about its use in the study. On
specific research question(s) or hypotheses being the other hand, a qualitative study in which the
addressed. Hence, a statement of the research aim methods were constructed as the study unfolded,
accompanied by the question(s) and/or hypotheses or a study with a more unconventional or innova-
allows the reader to understand the researcher’s tive research element, may require much more
intentions behind designing a study in a particular explication for the reader to grasp what was actu-
way. In research papers, this statement is typically ally done.
contained in the conclusion of the literature review
since it is supported by the reader’s conclusions Results or Findings
from appraising the literature. In some journal for-
mats, however, it is expected that the research This section of a research paper informs the reader
question(s) or hypotheses are stated as part of the about the most salient discoveries of the study.
study design. Whether one uses the term results or findings
depends to an extent on the type of study done as
well as the journal preference. As a rule, the term
Methods: Design, Procedure of results is more typically used with quantitative
Study, Sample, and Analysis studies and findings in reporting qualitative find-
The methods section of a research paper contains a ings. How the results/findings are presented also
succinct description of the research design. The depends to a large extent on the method of the
aim of this section is to provide adequate detail to study.
enable the reader to understand what was done and Quantitative study results are generally pre-
how. This part of the paper allows the reader to sented through a combination of text, statistics,
determine the degree of rigor in the study proce- summary tables, and figures. Quantitative figures
dures, and provides information that is essential provide visual representation of data such as his-
for replicating the study. tograms, plots, and charts. They are generally
This section of the paper is often challenging to organized according to the questions or hypotheses
write since the limits on page length in journals posed.
generally allow only the most key elements of the Qualitative findings are generally text-based.
research to be described. The following elements They usually incorporate depictions of themes or a
are typically included in the methods section: narrative depiction of the nature of the finding,
supplemented with evidence from the study, such
• A description of the fundamental design of the as quotes from interviewees or observations from
study, fieldnotes. The use of these data serves to illustrate
• A statement of the ethical approval and proce- a theme or demonstrating a concept in such as way
dures (for studies involving human subjects), as to deepen the reader’s appreciation or under-
• A description of the sampling procedure (whose standing. Qualitative reports may also include
participation was sought and how), tables, which usually contain textual information,
• A statement of the number of participants and or figures to illustrate a finding.
depiction of their salient characteristics (in some Tables and figures that are used to present
types of studies, this element is reported in the results/findings should be self-contained and com-
results/findings section), plementary to the text. In some cases, they contain
• A description of any experimental procedures information that is not specifically stated in the
and equipment used, text. In other cases, they amplify or serve as
• A description of the methods for data collection another way of representing findings that are dis-
and how they were used (e.g., tests, interviews, cussed in the text. Tables and figures are used
observations), along with a discussion of their when they are more efficient and succinct in pre-
suitability for the study and evidence of their senting results/findings and when they are likely to
adequacy (validity and reliability), and add to the reader’s comprehension of the results/
• An explanation of the data analysis procedures. findings.
The interpretation of results/findings also tance lies in providing the researcher with an
depends on the nature of the study being pre- opportunity to explicitly tell readers how they
sented. A rule of thumb for quantitative results is might use the information gained from this
that interpretation should not extend beyond what research for themselves in practice, or in further
the data indicate. So, interpretation serves to clar- research (Brown, 1994).
ify the findings. It should not evaluate the data. In
qualitative research, interpretation is much more Summary
interwoven into the presentation of the findings
The previous sections discussed the main elements
because the author(s) of the study are serving as
that ordinarily make up a research report. As noted
guides to allow the reader to comprehend the real-
at the outset and highlighted throughout the
ity of the phenomena studied.
discussion, these elements can be woven into a
research report in a variety of ways. What is
Discussion and Conclusion emphasized and how it is organized in a paper
The discussion and conclusion may be two sepa- depends on the type of research being reported, the
rate sections offered in sequence at the end of the methods used, and, to an extent, the topic of the
paper, or they may be presented as a single section research. As stressed earlier, there is no substitu-
that covers all the elements discussed below. The tion for becoming familiar with how articles on the
purpose of the discussion in a research paper is to research topic, using similar methods and/or pub-
consider the meaning and significance of the lished in the intended journal are composed, as
results/findings. This means both: preparation for writing one’s own research report.
• Examining the findings in relation to the study

question(s) and its purpose advanced in the liter- Selecting a Venue
ature review, and
• Considering the findings in light of the results of
for Publication
previously published studies.
Writing for publication is essentially a task of
The discussion section does not recount the communication with a specific audience.
results themselves. However, it serves to interpret Therefore, as already emphasized, knowing
or further explain the meaning or implications of one’s expected audience is crucial to effective
the findings. Moreover the discussion will typi- communication of one’s research in written form.
cally offer evaluative statements about the find- Academic and professional scholarly journals tar-
ings/results, noting the strength of the evidence or get different readerships, and so it is important to
how compelling it is. The discussion may also identify one or two suitable journals at an early
draw together or synthesize different aspects of the stage. One, then, writes so as to speak to the jour-
findings. Finally, the discussion should draw the nal’s readers.
reader’s attention to the salient similarities and dif- Selecting the wrong journal can result in the
ferences between the reported results and those rejection of a paper (see Chapter 34 for a discus-
from previous studies sion of the review process)
and to the factors. It that would otherwise be
should seek to account Writing for publication is acceptable for another
for these similarities and essentially a task of com- journal. Moreover, if one
differences. does not pay attention to
A research paper is munication with a specific the guidelines of a journal
usually concluded with audience. or the way in which arti-
some discussion of the cles are typically written
theoretical and practice for that journal, the likeli-
implications of the results, the limitations of the hood of having to make major revisions to an arti-
research, and recommended avenues for further cle is greater.
research. When it is well written, this section can
highlight the key lessons to be learned from the
Criteria for Selecting a Journal
research for the reader, guide the reader in weigh-
ing the strengths and limitations of the research, Journals differ according to their level of rigor,
and suggest an agenda and directions for further their subject matter and/or intended audience, and
research to the reader. In other words, its impor- sometimes their preference for methodology.
There is no magic formula for selecting an appro- This section describes strategies that can be
priate journal. Overall, the strategy should con- used to become and remain an effective writer.
sider two main factors. The first is whom the While writing is a creative process, it requires
author wishes to be the audience. For instance, if a deliberate and disciplined management of several
study has a strong applied focus and the author interrelated tasks (Brown, 1994). Figure 35.1 illus-
wishes it to have an trates some of the range
impact on practice, then of tasks to be managed
a professional journal is Writing research papers is a by the writer.
probably a wise choice. skill that can be learned and People often try to
If communicating with manage writing from the
interdisciplinary col- maintained through deliber- bottom up, that is, by
leagues might make ate and regular practice. focusing on the least
more impact on practice, complex tasks of writing
then an interdisciplinary shown at the bottom of
journal might be a good choice. On the other hand, Figure 35.1, such as grammar, language use, punc-
if the study contributes new knowledge to under- tuation, and so forth. However, it is more effective
standing the nature of a specific problem or diag- to take a top down approach to writing, attending
nosis, then the author may wish to consider a to the more complex tasks earlier in the writing
journal that focuses on that problem or diagnosis. process. Doing so will tend to lead to a more
The second factor that needs to be considered is coherent, readable, and engaging writing style.
the fit between one’s study and the anticipated
paper with the characteristics of the journal. To Writing Strategies
assess this fit, it can help to ask the following ques-
The following are key strategies for anyone who
tions:
wishes to become an effective research writer:
• Is the rigor of the study being reported consistent
• Setting aside and structuring time for writing,
with the level expected for this journal?
• Discovering and developing one’s own writing
• Does this journal typically publish articles on the
style and habits,
topic of this study?
• Writing for a particular audience,
• Does this journal publish studies with the kind of
• Clarifying one’s focus,
sample in this study (as defined by such factors
• Drafting, sequencing, and rewriting, and
as age or disability)?
• Seeking and using feedback.
• Does this journal publish studies that use the kind
of methods employed in this study? Each of these strategies is briefly discussed
• Does this journal emphasize applied, basic, or below.
participatory research and, if so, how does this
emphasis fit with the study? Setting Aside and Structuring Time
• Does this journal have restrictions on article
length that will affect how well the study can be Writing takes time. Most writers find that they
presented? require large blocks of time to focus on writing.
Hence, there is no substitute for structuring one’s
These questions can be answered by examining schedule to include regular writing time. These
the journal’s statement of mission, editorial state- blocks of time should be periods when one is alert
ments that indicate the special focus or emphasis and able to attend to the writing task. Scheduling
of the journal, and guidelines for manuscript sub- writing in one’s spare time, when one is tired or
mission. It is equally helpful to examine the typi- easily distracted, will only invite frustration. It can
cal content of articles in the journal. If in reading a also be helpful to allocate less optimal or smaller
selection of recent articles in that journal, one finds blocks of time for the more mundane or concrete
articles of a similar nature to the planned paper, aspects of writing (i.e., those tasks nearer the bot-
then the journal is probably a good choice. tom of Figure 35.1).
The Writing Process Discovering and Developing One’s

Own Writing Style and Habits
Writing research papers is a skill that can be While some things are essential to all writing, it is
learned and maintained through deliberate and reg- also a highly personal process. Individuals who are
ular practice. experienced and successful writers have paid
Overall Coherence
and Focus
Logical Flow of Information
Between and Within Sections
Matching Tables and Graphs with Text
Links between Headings Links between

Paragraphs Sentences
Jargon Abstract Words Grammar
Spelling Capitalization Page Layout Punctuation
Figure 35.1 The hierarchy of tasks that writers need to master in man-
aging the writing process. (Adapted from Brown [1994, p. 4].)
attention to and developed their own writing style the next task. Once again, it is important to iden-
and habits. This includes consideration of such tify and use what works best.
factors as what are optimal times and contexts for An often underestimated element is finding the
writing, and what is the best way to organize right context for writing. Scientific writing con-
resource materials. jures up images of some quiet corner in a library,
Writing requires discipline. Anyone who has but such a context does not work for everyone.
done substantial writing will admit that a key ele- Some persons need quiet space; others write effec-
ment is following through on planned time and tively with background noise or music. In the end,
making oneself focus during scheduled writing if a writer discovers ways to make the writing
time. In scheduling writing times and tasks, one process enjoyable and fulfilling, it will require less
should take into consideration what times of the effort to motivate oneself to write and the routine
day/week are best for one to concentrate on the of writing will be easier to sustain.
writing task and how long one can effectively Most of the time, we write with resources that
write. Some people write better earlier in the day, have been accumulated over time. These materials
whereas others prefer late at night. Some persons include such things as articles accumulated for
can write at a variety of times. a literature review, notes taken from reading, and
One should assess how long one can write opti- boilerplates (i.e., previously written materials
mally. Some people can write for only an hour or that are edited and incorporated into papers).
two. Others can write for an entire day. Some per- Organizing these materials in ways that optimize
sons require frequent breaks, whereas others want one’s writing style will facilitate efficient writing.
to stay focused throughout the writing. Knowing For instance, one of the authors of this chapter
and respecting one’s own writing patterns is vital. prefers to take notes in a laptop computer while
It is important to pay attention to when one gets reading and then uses those notes later when com-
the most writing done and when one has difficulty posing a chapter. This author tends to begin writ-
writing, and then to plan one’s schedule according while reading. This author also typically works
ingly. on several articles or chapters at a time and main-
It is also useful to structure one’s writing time. tains a folder for each work in process in the com-
Some people find it helpful to set short-terms goals puter. That way, notes, references, and other
for the next hour or day such as writing so many materials can be inserted in the folder whenever
pages or finishing a particular section of a paper. new information or new thoughts about a work in
Some people find it best to write for a period and progress arise. When writing, the author takes all
then review what was written, revise, and go on to these resources and weaves them together into an
A B
Figure 35.2 (A) Gary writes effectively in a variety
of places and times. One of his most productive
times for writing is during the daily commute to
and from Chicago on the train. It provides over
2 hours a day of time away from interruptions. For
him, the background noise helps him concentrate
since he is unable to write in silence. (B) Renée
writes best in a comfortable and quiet environ-
ment with large chunks of time. She needs rela-
tively frequent short breaks. Writing in front of the
fireplace in winter and in a sunny spot in the yard
during summer works well for her. (C) Ellie writes
drafts on computer at home or work, but prefers
to rework the structure of material, or to edit it on
paper. Her favorite venues for editing are her local
neighborhood cafes in Melbourne with the music
and activity of the café going on around her.
C
“organic” document that generally gets revised When collaborating on papers, it is a good idea
multiple times, often quite dramatically as it to discuss writing styles and figure out how they
evolves into the final written manuscript. can be meshed over the course of working
Another of the authors prefers to first read, sur- together. For example, one of us has found discus-
rounded by hard copies of the readings, highlight- sion, written comments, electronic editing, and
ing aspects of the readings that are important. time spent working together on the computer to
Then, the author outlines the plan of the manu- each be effective at different stages in the process
script and proceeds to write the paper progres- of writing collaboratively with colleagues. Some-
sively from beginning to end, thinking carefully times, of course, this is not possible. For instance,
about each section of the manuscript as it is being this chapter was written without any face-to-face
written and doing most of the rewriting along the discussion among the authors. Each person took
way. Once finished with a paper or manuscript, the turns working on the manuscript, which was
author seeks feedback and then generally com- shared by e-mail.
pletes one overall rewrite before the paper is fin-
ished.
Writing for a Particular Audience
These differences in writing are matters of per-
sonal style that a new writer must discover through No scientific article will be readily understood or
experimentation. Of course, writing style will also interesting to everyone. Readers bring to a written
vary somewhat according to what one is writing, piece their own training and knowledge and their
how familiar the topic is, and how tight a deadline sense of what is important. The sole purpose of
one is facing. Respect for one’s style, tempered writing is to communicate with readers. This may
with practical considerations of getting the task seem obvious, but it requires one to read one’s
done, is a wise course. writing from the imagined perspective of the
reader. Writers, of course, understand their own • Mind-mapping to distill the main message, and
writing since it is an outpouring of what they know. • Developing a working abstract as a framework
Readers, however, do not know what the writer for the first draft of a paper.
knows and, thus, the paper needs to be written and
Mind-mapping and similar techniques, such as
edited with the reader’s perspective in mind.
concept-mapping, involve generating and refining
Thus, the writing and rewriting process begins
a visual representation of the ideas to be contained
with a sense of one’s audience, including their
in a paper (Bihl-Hulme, 1985; Brown et al., 1994;
knowledge base, interests, and expectations.
Buzan & Buzan, 2000).
Writers who do not consider their intended audi-
The process begins with laying out on paper a
ence will inevitably be frustrated with rejected
detailed free-form diagram of all the facts, ideas,
papers or requests for substantial rewriting. Hence,
thoughts, questions, and linkages that could go
as previously noted, one should identify who one’s
into a paper. One then identifies the most impor-
audience is at the outset. Having done so, it is
tant parts of the mind-map by assigning priorities
important to read one’s own writing and rewriting
to the information contained in it (Brown et al.,
from the perspective of that intended audience
1994). This process allows one to more readily see
(Brown, 1994).
the relationships or connections between materi-
It takes some practice to develop this skill of
als, and to identify the material of most importance
reading your own work from another’s perspec-
to the topic. Mind-mapping also helps to define the
tive. Nonetheless, it is a skill worth cultivating. It
boundaries of a paper, letting one decide what not
can help to think of three or four specific people
to include (Brown et al., 1994).
who are potential readers of your work, and to
To further clarify the main message it is often
write and read your work with them in mind. It
helpful to generate a brief (approximately 25-word)
may also be useful to seek someone who repre-
working abstract from the mind-map. The disci-
sents that audience to give feedback, as a way of
pline of this word limit is useful to creating a clear
getting a sense of how the intended audience will
focus! This abstract should not be confused with
approach your writing. In doing so, one should pay
the abstract that is written for publication. Rather, it
careful attention not only to the implications for a
is a mission statement that guides one’s writing
given paper, but also to how the person approaches
process. The contents may, indeed, go into the final
the paper. This helps one develop the sense of what
abstract in some form, but the purpose of writing an
it means to pay attention to an audience.
abstract as part of this process is to make sure one
has clear in one’s own mind what is being written.
Clarifying One’s Focus The use of outlines is often recommended as a
tool for structuring written papers and organizing
Clarifying the main message of an intended paper
the material within them, but they can be difficult
will:
to create when one is not clear what the paper is
• Make the writing task more efficient, and going to be about. Clarifying the main message
• Result in a paper that is easier for readers to and creating a working abstract can provide the
understand (Brown, Rogers, & Pressland, 1993, foundation for creating an outline. The priorities
1994). identified in the mind-map often yield the section
headings and subheadings for an outline.
This advice sounds deceptively simple.
Typically researchers find themselves with much Finding One’s Story
more information then they can include in a single
research report. Thus, they must make challenging Mind-mapping will not work for all writers and it
decisions about what to include in their papers, probably works better for visual thinkers. Other
what to leave out, and how to organize the infor- writers may find it more helpful to think about the
mation reported. Lack of clarity about the focus paper as a story. A research paper basically
of a paper can create difficulty in making these involves telling the story of how one came up with
decisions. the research question and went about answering it
in the research process, and then revealing the
answers obtained.
Mind-Mapping and Abstracting
In this case, one needs to identify the basic plot
Based on their experiences of running workshops of the story. The plot involves a beginning, middle,
for researchers about how to write papers, Brown and end that flow together into a whole. Each part
and his colleagues (1993, 1994) recommend two of the writing contributes to the unfolding of the
strategies that are helpful in creating a clear focus: story.
Inspiration
Most writers will publicly say they don’t believe in screen or an empty piece of paper. However, that’s
inspiration, pointing out rightly that writing simply not the type of break we are suggesting. When
takes hard work. However, almost every writer will writers have spent effort over an extended time
have at least one good story about the time he or organizing volumes of materials and writing differ-
she was stumped and couldn’t figure out how to ent pieces of a paper, they have a difficult time
pull together a paper or present a particular argu- seeing the proverbial forest for the trees. At such
ment. Then, suddenly in the middle of the night, times, it can sometimes be useful to step away
in the midst of taking a shower, or while driving form the writing for a short period (from a day
home, the solution came in a flash. The writer or two to perhaps a week or so). One should not
could suddenly see how to put it all together. leave the writing so long as to let memory fade
We have heard enough such stories (and have about the contents or it will take additional work
some of our own) to assert that inspiration can to get restarted. However, stepping back from the
definitely be a part of writing. The problem with immediate writing task to reflect on the paper and
inspiration is that one can’t will it. However, there let thoughts percolate as they occur can sometimes
are some things a writer can do to enhance the be the catalyst for a new insight or inspiration.
possibility of getting the sudden vision that makes There are also ways of way of taking a break
everything clearer: without losing momentum. One way is to work on
Do your homework: Inspiration does not come some of the less intellectually demanding tasks of
to those who don’t do the basics first. If inspiration writing such as double checking one’s references
is anything, it’s the synthesis of many elements or or updating a literature search. Another way is to
pieces of information into a higher order whole. switch to writing another paper. Many investigators
However, if you have not fully immersed yourself who are working on more than a single investiga-
in the various informational components, the syn- tion or paper find that when they hit a wall writing
thesis will never come. Reading thoroughly and one paper, it can be productive to switch to another.
carefully in order to become knowledgeable and Seek someone’s opinion: Talking to someone
organizing concepts and information in some fash- about one’s paper is often a very helpful strategy.
ion are necessary foundations. Sometimes inspiration comes in the form of some
Take a break at critical times: Writing tempts solid advice. Other times, the act of discussing
many of us to take frequent breaks. Nothing one’s writing and, in particular, the problem one is
inspires us to check out the weather, see if we have trying to resolve, helps to clarify what one is really
anything in our e-mail inbox, or even clean out the having difficulty with and that, in turn, leads to
refrigerator more than staring at a blank computer identification of a solution to the writing puzzle.
To use this approach, one should write out the improve the organization of information to com-
story in as few words as possible. In a sense, it’s municate effectively with readers. This is because
like creating the abstract. However, it is a good writing is itself a learning process and the more
idea to create the story for a layperson. One can one learns about the topic, the more effectively one
imagine, for instance, how one would explain to can write about it. Successive drafts tend to get
one’s mother or to a friend what the paper is about. better because one is more knowledgeable.
This requires one to pare down everything to a The process of writing is also not linear. The
fairly simple story line. typical structure of research papers as outlined ear-
Once one is able to articulate the story in lier may constitute a sequential series of writing
straightforward terms, deciding what to include tasks. However, one does not necessarily begin at
and leave out is facilitated. Moreover, determining the beginning when writing a research paper.
the best way to chunk the successive parts of the For instance, working first on sections that are
story and link them together becomes easier. As more straightforward can help one make more
with the mind-mapping technique, one can create effective progress. The methods section is typi-
a more effective and integrated outline once one is cally one of the more straightforward to write, fol-
clear about the underlying story. lowed by results section. This is because in these
sections one is describing what one has done and
found, these being the most familiar parts of the
Drafting, Sequencing, and Revising
story one is telling.
For most authors, writing typically involves a sub- The introduction and discussion require a
stantial number of drafts to refine the content, to greater degree of abstraction and interpretation to
enhance the synthesis of ideas in a paper, and frame the story of one’s research for the reader.
Consequently, these sections may be easier to a paper for consideration for publication. Written
write at a later stage, once the main body of the comments prepared by referees for authors are
paper is complete. The introduction, in particular, intended to provide constructive feedback about
is more straightforward to write once one is clear the content, structure, and presentation of the man-
about what is being introduced uscript, and to guide the authors in subsequent
Although there are different styles of writing revision of their manuscripts. Revisions are almost
and rewriting, redrafting sections in parallel, rather always required, so being asked to make some
than sequentially, often helps to achieve better revisions should not be taken by authors as an indi-
structural and conceptual integration. It also helps cation that a manuscript is of poor quality. Authors
to ensure consistency in writing style and language are well advised to pay careful attention to the ref-
use through the paper. Every paper should be read erees’ comments and recommendations in revising
at least once as a whole with an eye toward editing a manuscript.
it to be a more integrated piece. If a journal decides not to accept a manuscript
for publication, one should carefully examine the
Seeking and Using Feedback feedback. Rejection may mean the paper does not
rise to the level of quality necessary for publica-
Seeking feedback by having others read and com- tion. It may mean that one needs to submit the
ment on drafts of a paper can be very helpful. paper to another journal. Generally, it is necessary
Writers always know more about what they want to also to revise the manuscript to address the new
say than do their readers. Consequently, when re- journal’s style and format requirements, to ensure
reading one’s own work, it is easy to make mental the writing “speaks” to the new readership, and to
assumptions and linkages that are not in the paper improve the manuscript on the basis of feedback
or apparent to the reader. Constructive feedback from the previous submission. Attending carefully
from someone else can help to identify assumed to the referees’ feedback and seeking advice, or
knowledge, missing links, and unanswered ques- coaching, from colleagues with publishing experi-
tions that a reader may have. Such feedback is ence can often be helpful at this stage.
extremely helpful for redrafting a paper.
There are different ways to get feedback on a
paper. The following are some examples:
Conclusion
• Coaching from an experienced writer in one’s
discipline/area of research (someone who knows This chapter presented the process of writing a
the topic, the intended audience and journal). report of research. It discussed both the usual con-
This type of reviewer may provide feedback tent that goes into such a paper and the process of
paragraph by paragraph on technical aspects of writing itself. As noted at the outset, many investi-
the writing, as well as on larger issues central to gators thoroughly enjoy the writing process. Even
research, such as how to articulate the rationale those who find it very enjoyable know that good
or methods. writing takes time and effort, and requires personal
• Participating in writing groups with peers. This discipline, persistence, and an openness to criti-
process is helpful for developing overall writing cism and feedback.
skills since one not only receives feedback but
also learns from reading and critiquing others’ REFERENCES
writing. Also, by mutually agreeing on deadlines
American Psychological Association (2001). Publication
for writing part or all of a manuscript, writers can manual of the American Psychological Association
help each other maintain discipline in writing. (5th ed.). Washington, DC: Author.
• Asking for feedback from colleagues with differ- Bihl-Hulme, J. (1985). Creative thinking in problem-based
ing disciplines or professional backgrounds who learning. In D. Bond (Ed.), Problem-based learning in
education for the professions (pp. 177–183). Sydney,
reflect the targeted audience. This type of feed- Australia: HERDSA.
back can be valuable in helping one step outside Brown, R. (1994). The “big picture” about managing
of familiar ways of presenting things and point writing. In O. Zuber-Skerritt & Y. Ryan (Eds.), Quality
out when papers are too full of jargon or insider in postgraduate education—Issues and processes
(pp. 90–109). London: Kogan Page.
perspectives. This type of feedback is especially
Brown, R. (1996). Key skills for writing and publishing
useful when one is aiming for publication in an research (3rd ed.). Brisbane, Australia: WriteWay
interdisciplinary journal. Consulting.
Brown, R. F., Rogers, D. J., & Pressland, A. J. (1993).
An important form of feedback also comes as Righting scientific writing: Focus on your main
part of the review process following submission of message! Rangelands Journal, 15(2), 183–189.
Brown, R. F., Rogers, D. J., & Pressland, A. J. (1994). Roche, R., & Taylor, R. R. (2005) Coping and occupational
Create a clear focus: the ‘big picture’ about writing participation in chronic fatigue syndrome. Occupa-
better research articles. American Entomologist, 40, tional Therapy Journal of Research 25, 75–83.
144–145. Thomas, S. A. (2000). How to write health sciences
Buzan, T., & Buzan, B. (2000). The mind map book papers, dissertations and theses. Edinburgh: Churchill
(3rd ed.). London: BBC Worldwide. Livingston
DePoy, E., & Gitlin, L. N. (1993). Introduction to
research: Multiple strategies for health and human
services. St. Louis: C. V. Mosby. RESOURCES
Farnworth, L., Nitikin, L. & Fossey, E. (2004). Being Brown, R. (1996). Key skills for writing and publishing
in a secure forensic psychiatry unit: Every day’s research (3rd ed.). Brisbane, Australia: WriteWay
the same, killing time or making the most of it. Consulting.
British Journal of Occupational Therapy, 67(10), Hayes, R. L. (1996). Writing for publication: solutions to
430–438. common problems. Australian Occupational Therapy
Hocking, C & Wallen, M (1999). Australian Occupational Journal, 43, 24–29.
Therapy Journal Manual for referees: Guidelines to Thomas, S. A. (2000). How to write health sciences papers,
assist referees and authors review manuscripts. dissertations and theses. Edinburgh: Churchill
Melbourne, Australia: OT AUSTRALIA. Livingston
S E C T I O N 8
Enhancing Practice Through Inquiry
C H A P T E R 3 6
Assessing Need for Services

Marcia Finlayson
Every day, occupational therapists are faced with

determining if an individual client has a problem
What Is a Needs Assessment?
or set of problems that could be remediated The term needs assessment is commonly used and
through occupational therapy services and, if so, seemingly easy to understand. At the most basic
which intervention would be the most appropriate level, it refers to the process of determining what a
to apply. Consequently, practitioners are well group of individuals, an organization, a commu-
versed in how to assess the needs of individual nity, or a population requires to achieve some basic
clients, translate these needs into intervention standard or to improve its current situation.
goals, assess progress toward goals, and determine Reviere, Berkowitz, Carter, and Ferguson (1996)
when discharge should occur. Equally important is describe a needs assessment as “a systematic and
the process of identifying the needs of broader ongoing process of providing useable and useful
groups of people in order to develop or modify information about the needs of the target popula-
services, programs, and policies and their respec- tion – to those who can and will utilize it to make
tive goals. This process is a requirement for occu- judgments about policy and programs” (p. 6).
pational therapists who are: Consequently, a needs assessment is simultane-
• Developing new areas of practice, ously a form of applied research and a political
• Expanding services within existing settings, process (Hancock & Minkler, 1997; Martí-Costa &
• Examining ways to allocate limited resources to Serano-García, 1983). Needs assessments vary
best meet the needs of a particular client base, or across four dimensions: the sophistication of the
• Trying to advocate for policy changes that will project design, the level of involvement of the
affect entire populations. stakeholders, its political orientation, and the scope
of the issue being addressed (see Figure 36.1).
To succeed in these broader types of activities,
occupational therapy practitioners and researchers
must have a basic but solid grounding in the theo- Dimension 1: Sophistication
ries and methods of needs assessment. This foun- of the Project Design
dation must include the knowledge and skills that
As a form of applied research, needs assessments
are necessary to conduct high-quality needs
range in quality and rigor, just like any other form
assessments, and the ability to work with others to
of research. This variability is reflected in the first
translate findings into action. Therefore, the objec-
dimension of a needs assessment: the sophistica-
tives of this chapter are to:
tion of the project design. To accurately inform
• Provide a theoretical overview of the concept of judgments about policies and programs, and ulti-
need, mately the allocation of resources, the ideal needs
• Examine the processes and dimensions of a assessment is one that is rigorously designed. This
needs assessment, means that it takes into account all of the key com-
• Describe and compare models and approaches to ponents of a high-quality study—clearly defined
needs assessment, questions that are grounded in theory, appropriate
• Evaluate methods commonly used in needs sampling, psychometrically sound data collection
assessments, and tools and processes, and correctly applied analytic
• Outline how needs can be translated into actions strategies. As previous authors have noted, needs
for solutions. assessments should be:
592 Section 8 Enhancing Practice Through Inquiry
Informal Sophistication of the Project Design Rigorous research
Very Limited Level of Involvement of the Stakeholders Fully participatory
Narrow, well-defined Scope of the Issue Being Addressed Broad, all encompassing
Maintains status quo Political Orientation Radical social change
Figure 36.1 Dimensions of a needs assessment.
• Systematic, services to these individuals, and the hospital that

• Empirically based, employs the therapists. It is important to realize
• Outcome oriented, and that stakeholders can also include individuals and
• Focused on solving real-world problems through organizations associated with the individuals in
the application of a variety of research method- each of these three levels. For example, additional
ologies and methods (Reviere et al., 1996; Witkin stakeholders could include families, friends, other
& Altschuld, 1995). professionals providing services (e.g., physical
therapy, social work), organizations to which
Often, though, the term needs assessment is
clients are referred on discharge (e.g., home care,
used loosely for projects that are little more than
stroke support group), and insurance companies.
informal questioning of convenient individuals
A needs assessment that is fully participatory
with little to no rigor or possibility of replication.
would seek input and direction from all three levels
As a political process, needs assessments are
of stakeholders at all points during the assessment
value laden, focus on collectives rather than indi-
process (e.g., setting the question, determining the
viduals, and raise awareness of everyday problems
methods, collecting data, etc.). This type of needs
and their causes. They seek to mobilize communi-
assessment would draw on the principles of com-
ties into action in order to influence and inform pol-
munity building (Minkler, 1997) and participatory
icymaking, infrastructure changes, and human and
action research (Reason & Bradbury, 2001). (See
financial resource management and distribution
Chapters 38–40.)
(Martí-Costa & Serano-García, 1983). Through
Involvement of stakeholders in the needs
these processes, the ultimate goal of a needs assess-
assessment process is critical to increase the odds
ment is to gather meaningful data to inform actions
of results uptake and utilization of the findings
that build on community strengths and remediate
(Green & Mercer, 2001). At the opposite end of the
any problems that are uncovered (Kretzmann &
continuum are the needs assessments that do not
McKnight, 1997; Witkin & Altschuld, 1995). As
actively involve stakeholders. In these types of
such, two additional dimensions of a needs assess-
needs assessments, an external needs assessor is
ment are the level of involvement of stakeholders
brought in to act as an expert consultant who
and its political orientation.
designs and implements the project, analyzes the
findings, and presents recommendations to the
Dimension 2: Level of Involvement
contracting group.
of Stakeholders
Stakeholders are the individuals, organizations,
and policymakers who have a role in defining the
Dimension 3: Political Orientation
issues and/or addressing the concerns that arise The political orientation of a needs assessment will
from the needs assessment process. Witkin and to a large extent depend on who initiated the
Altschuld (1995) conceptualize stakeholders at process. Some needs assessments are initiated by
three levels—individuals who are receiving serv- organizations in order to maintain the status quo,
ices, the providers of services, and organizations or alternatively, justify a decision that has already
as a whole. An example of these different levels been made (Martí-Costa & Serano-García, 1983;
may include older adults who have experienced a Witkin & Altschuld, 1995). For example, hospital
stroke, the occupational therapists who provide administrators may conduct a needs assessment
Chapter 36 Assessing Need for Services 593
that recommends a new computerized documenta- that is rigorous and can produce findings that have
tion system or a different staffing configuration on utility for informing political action. Within many
the rehabilitation units in order to improve patient disciplines, need is essentially viewed as a dis-
care. While these infrastructure changes may ben- crepancy between what an individual’s or group’s
efit patients indirectly, they often do not emerge present situation or status is and what is desired
from the issues and concerns of patients them- (Reviere et al., 1996; Witkin & Altschud, 1995).
selves. But the definition raises two key questions: What
At the opposite end of the continuum are needs is desirable? Who defines desirable?
assessments that are initiated for the purposes of Trying to answer these questions illustrates the
raising awareness of the issues within a commu- complexity of defining and then trying to assess
nity, and for promoting community building and need. Authors from the disciplines of psychology,
social change. For example, a needs assessment of sociology, philosophy, and political science
older adults who have (Bradshaw, 1972; Dill,
experienced a stroke and 1983; Doyal & Gough,
are being discharged Defining and operationaliz- 1991; Maslow, 1954;
back to the community ing need is critical to con- Thomson, 1987) suggest
may uncover issues and that the concept of need
concerns related to hous- ducting a needs assessment is complex because of
ing accessibility, lack of that is rigorous and can pro- the different ways the
social support, and inad- term is used (noun as
equate outpatient follow- duce findings that have utility well as verb), the extent
up. By documenting for informing political action. to which need is value
these issues and sharing laden, and because of its
findings with the com- links to ideas about what
munity, a needs assessment team can provide is moral and good. For example, the term need
empirical data to support social change efforts itself is used to describe basic physiological drives,
such as mobilizing community volunteers, chang- goals that are sought after (ends), as well as the
ing discharge policies, or developing a fund to strategies to achieve those goals (solutions) (Dill,
subsidize home modifications for older adults who 1983; Doyal & Gough, 1991). Furthermore, need
require home modifications after a catastrophic can be conceptualized at both an individual level
health event. as well as a collective one. These ideas are
explored below.
Dimension 4: The Scope of
the Issue Being Addressed Need as a Physiological Drive
A final dimension of a needs assessment reflects As a basic physiological drive, need is a “motiva-
the scope of the issue or problem being addressed. tional force instigated by a state of disequilibrium
Many needs assessments are narrow and well or tension set up in an organism because of a
defined in their orientation, for example, determin- particular lack” (Thomson, 1987, p. 13). Defining
ing the recreational needs of teenage members of a needs as drives is illustrated in the classic hierar-
specific community center. Other needs assess- chy described by Maslow (1954), which suggests
ments are much broader in their orientation, for that humans are motivated to first address physio-
example, determining the needs of individuals in logical needs for food and water, then safety and
the United States who are newly diagnosed with security, and finally to belong, to develop self-
multiple sclerosis. esteem, and to seek self-actualization. Examples of
needs as drives can be illustrated in the following
statements: “I need a glass of water,” “I need a
What Is It that We house,” or “I need to participate in a meaningful
activity.” Note that in all of these statements the
Are Assessing? term need is used as a verb, and “points to what
is required or desired to fill the discrepancy –
At the heart of the needs assessment process is the solutions, a means to an end” (Witkin & Altschuld,
concept of need. It is a vague term that is poorly 1995, p. 9). Yet, the discrepancy itself is not
understood and rarely defined in the research liter- explicit, nor is the specific end (e.g., becoming
ature. Nevertheless, defining and operationalizing hydrated, having shelter, being engaged in an
need is critical to conducting a needs assessment occupation).
Need as a Solution Need as Relative to the Assessor

While needs as drives is not the perspective used In addition to conceptualizing need as either goals
for the majority of needs assessments that are con- or solutions, it is also possible to consider needs
ducted by occupational therapists, using the term relative to the assessor and the method of deter-
need as a verb and conceptualizing need as a solu- mining the need. The classic typology presented
tion is very common. Take for example the situa- by Bradshaw (1972) is an example of conceptual-
tion of a community mental health agency that is izing needs from these perspectives. Bradshaw dis-
examining life skills programming for its clients. cusses four types of needs: normative, felt,
Staff members set off to conduct a needs assess- expressed, and comparative. Table 36.1 provides
ment to help its Board of Directors determine if definitions and examples of these types of needs,
clients need more occupational therapists within as well as related terms that have been used by
the scope of the agency’s service package. What is other authors.
missing in this scenario is an explicit indication of
what the actual goal might be, and a willingness to Normative Need
explore solutions other than more occupational
Normative need is defined by professionals and
therapists.
experts rather than by members of the community
Presumably, the goal of this needs assessment
themselves. Consequently, a needs assessment that
would be improved ability of the agency’s clients
uses a normative needs perspective would have lit-
to manage in the community independently.
tle to no involvement of the stakeholders. In addi-
Unfortunately, by framing the needs assessment
tion, defining needs using this approach is highly
from a solution orientation (i.e., do clients need
subject to the cultural and value biases of the
more occupational therapists?), it is unlikely that
expert and can vary greatly across experts. For
other options to achieve this goal will be identified.
example, when asked to identify the needs of
If alternatives are not identified, the Board of
young men in a homeless shelter, an occupational
Trustees cannot consider them in their decision-
therapist may identify needs related to their roles
making and resource allocation decisions.
and habits. For the same group of young men, a
For this reason, Altschuld and Witkin (2000)
social worker may identify needs related to com-
argue against conceptualizing needs as solutions
munication and social interactions. A drug and
because such an approach fails to identify the
alcohol counselor may identify the primary need
underlying issues and concerns, thereby limiting
of these young men as substance abuse rehabilita-
the opportunity to explore and examine a range of
tion. As a result, using a normative perspective
possible solutions. Yet, many needs assessments
means that the needs assessor must be vigilant
completed within health and social services con-
about the reliability of the data collection tools and
ceptualize need in this way. A classic example of
processes so that the findings can be replicated.
this type of needs assessment is the survey that
provides respondents with a list of services, and
Expressed and Felt Need
then asks them to review the list and check off
what they “need”. Such an approach risks the pos- In comparison, individuals themselves determine
sibility of identifying “needs” that do not address expressed need and felt need. Felt need refers to
the underlying issue. want, with or without actions to obtain that which
Table 36.1 Bradshaw’s Categories of Need

Bradshaw’s Need Categories Explanation/Illustration
Normative need Expert definitions
Expressed need Refers to demand for a service as measured by actual use as well as
requests for services (i.e. waiting lists).
Felt need Want
Comparative need Need based on comparisons to and equity with others. For example,
Group A receives a service, but Group B does not even though the
groups are equivalent on key characteristics. Therefore, Group B is
determined to be in need of the service.
is wanted. Expressed need refers to demand for a within the organization, community, or region in
service, either through current use or waiting lists. which the needs assessment is being conducted.
The problem with both felt and expressed need is Sociopolitical factors encompass basic beliefs
twofold. First, both frame needs in terms of solu- about what is moral or good, overriding political
tions—a problem that has already been discussed philosophies, and the nature and operations of sys-
earlier in this section. Second, these ways of defin- tems that are either leading the needs assessment
ing need cannot account for people who do not efforts, or will respond to the findings (Dill, 1983;
know about a service—one cannot want something Doyal & Gough, 1991; Shi & Singh, 2001). As
of which one is unaware. Expressed need is further such, the process, measures, outcomes, and actions
problematic because there are often individuals of a given needs assessment are fundamentally
who want a service, but who do not request it linked to either the principles and values of the
because they know it is not physically or finan- marketplace (e.g., a needs assessment to determine
cially accessible, or because they are simply tired whether to expand a private practice rehabilitation
of fighting the system. Consider for example indi- clinic) or to a commitment to the social good (e.g.,
viduals who have been consistently refused third- a needs assessment to determine the accessibility
party funding for assistive technology and simply of public housing in an urban center) (Shi &
give up asking for it. Both felt and expressed need Singh, 2001).
focus on the solution, and not necessarily the While the majority of needs assessments will
underlying issue or concern. have a primary link to one of these positions, it
may also have secondary goals and objectives in
Comparative Need the other. For example, a health management
organization may conduct a needs assessment in a
Conceptualizing need from a comparative perspec- rural area to determine the demand for an assistive
tive is relatively common in public health, and the technology clinic and the economic viability of
health and social services. Consider the situation developing one. This needs assessment would be
in which a group of older adults living in one sen- linked primarily to the principles and values of the
iors’ apartment complex (baseline group) is com- marketplace, and the organization’s need to be
pared to a group of older adults living in another economically successful. Nevertheless, it may also
complex (comparison group). If the comparison be linked to a secondary understanding that access
group does not have the same programs and serv- to assistive technologies is not equitably distrib-
ices as the baseline group, they are determined to uted between rural and urban areas, and that the
have need. Through this approach, need is framed health and quality of life of individuals with a wide
in terms of solutions (i.e., the programs and serv- range of disabilities may be negatively influenced
ices provided), which is problematic because it by the lack of such a program.
carries with it important assumptions that are often
difficult to test. Specifically, taking a comparative Needs Assessment Grounded
need approach assumes that the baseline group
actually needs the programs and services that their
in a Marketplace Philosophy
complex provides, and that they are receiving them Needs assessments that are grounded in a market-
in the correct amount. It further assumes that all of place philosophy equate need with demand, and
the baseline group’s needs are actually being met define needs in terms of solutions. In other words,
and that the programs that are being provided are they take both a felt needs and an expressed needs
the best solution to the underlying problem. conceptualization that focuses primarily on the
individuals. Individual preferences and autonomy
are viewed as key to determining need (Shi &
Identifying Needs as Singh, 2001). While there is room within a mar-
ketplace philosophy of needs assessments for
a Political Process expert definitions of needs (i.e., normative needs),
it could be argued that this approach would focus
At the beginning of this chapter, needs assessment more on the needs of the organization that would
was described as both a form of applied research deliver the service rather than the individual
and as a political process. Regardless of whether receiving it.
need is viewed as a goal or a solution, it is value Using the marketplace philosophy, needs can
laden and culturally influenced. As such, it is best be met through the free market and through a
affected by sociopolitical factors that are operating conservative approach to the development and
maintenance of health and social service policies. number of authors have pointed out, needs assess-
Services that are provided as a result of the needs ment is one of many potential tools in the
assessment are viewed as an economic good and processes of strategic and ongoing planning, qual-
are distributed based on people’s ability to pay, ity improvement, and outcomes management.
either independently or through various types of Ultimately, the results of needs assessments allow
insurance (Shi & Singh, 2001). decision makers to justify expenditures to develop
new services, or to refocus, modify, or eliminate
Needs Assessment Grounded in existing services.
In summary, the concept of need is complex
a Philosophy of the Social Good
and multilayered. The term need is used in differ-
Alternatively, needs assessments that are grounded ent ways, and these ways influence the processes
in a philosophy of the social good equate need and outcomes of a needs assessment. In addition,
with disparities across groups and inequities of the conceptualization of needs is influenced by
service supply and access, and focuses on the col- values, culture, politics, and ideas about what is
lective rather than the individual (Shi & Singh, moral and good. While the idea of “needs assess-
2001). As such, needs ment” is seemingly easy
assessments grounded in to understand, this theo-
this way have the poten- Ultimately, the results of retical overview suggests
tial to address underly- that it is much more
ing problems and issues,
needs assessments allow complex than many peo-
as well as identify poten- decision makers to justify ple realize. It would be
tial solutions. Two types expenditures to develop easy to become mired in
of need are key to this these theoretical per-
perspective: normative new services, or to refocus, spectives, and be unable
need and comparative modify, or eliminate existing to move forward to
need. From the perspec- assess the needs of a
tive of the social good, services. group of individuals, an
the purpose of conduct- organization, a commu-
ing needs assessments is nity, or a population.
to identify areas that need to be remediated in Instead, the intention of this overview is to high-
order to improve the standard of living and/or light the importance of being clear on what is
quality of life of the collective and to determine being assessed and to provide potential frame-
how to allocate services fairly. works within which current or future needs asses-
The provision of services to address the identi- sors can consider their work.
fied needs is seen as a social good (if one person is
better off, everyone is better off), and generally
involves governmental intervention whether at the
community, county, state, regional, or national Models for Approaching
level. The products of these types of needs assess-
ments guide actions that emphasize equity and
Needs Assessments
ensure that some basic set of standards is met.
While the definitions of need are multifold, and the
Actions to remediate these needs are more likely to
philosophical basis for doing needs assessments
be met through liberal or democratic policies, and
varies, the actual process of needs assessment is
social advocacy and action movements (Shi &
systematic. Over the years, a number of models or
Singh, 2001). Consequently, needs assessments
approaches to needs assessments have been pre-
grounded in this philosophy are more consistent
sented in the literature. Five important approaches
with ones that are participatory in orientation, and
include:
are directed at social change (see Figure 36.1).
• Logic models,
Summary • Three-phase model,
• Concerns report method,
These discussions suggest that need is ultimately • Community building, and
“created” by the people and organizations that are • Participatory approaches.
conducting needs assessments, particularly when
needs are conceptualized as solutions. The ques- The key features of these models are summa-
tion then becomes: Why try to measure need? As a rized in Table 36.2.
36Keilhofner(F)-36
Table 36.2 Comparison of Key Features of Different Models and Approaches to Needs Assessment
Logic Model Three-Phase Concerns Report Method Participatory Approach Community Building
Is there a clearly defined philosophical No No Yes Yes Yes
or theoretical stance?
5/5/06
Is it apparent what research approach No No Yes Yes No
fits with this model (i.e. qualitative,
quantitative, or mixed)?
4:05 PM
Are the specific research methods fitting No No Yes No No
with this model defined?
Is it clear within this model what tasks Somewhat Yes Somewhat No No
are done in what order, by whom,
Page 597
etc.? In other words, does it have a
defined structure?
Is it clear how a needs assessor would Somewhat Yes Yes Yes Somewhat
move through the process of a
needs assessment based on this
model?
Would the use of this model be resource No Maybe—depends on Maybe—depends on Yes Yes
intensive? the issues and the issues and the
the community community
What is the expected outcome of a Planning tool Action plans Task forces that develop Stronger community Stronger commu-
needs assessment using this model? solutions to commu- with ownership nity with owner-
nity concerns over issues ship over issues
597
Logic Models on the long-term goal of the project, and how the
information that is being gathered will be used.
Logic models were originally developed as a tool
to facilitate program planning and evaluation. Three-Phase Model
While logic models continue to be primarily used
The three-phase model is another tool that can be
for these purposes, they also have great utility for
used to facilitate the planning of a needs assess-
the process of planning a needs assessment (Rush
ment. This model is described in detail by Witkin
& Ogborne, 1991). Figure 36.2 presents a logic
and Altschuld (1995), and fundamentally operates
model that was developed by a group of students at
as a checklist of steps and activities that must be
the University of Illinois at Chicago for a needs
achieved over the different stages of a needs
assessment they conducted for a local suburban
assessment. The first phase of this model is the
department of public health. Although a logic
preassessment, during which:
model does not identify the philosophical stance of
the needs assessment, nor points to particular types • A planning group is established,
of methods, it does offer a concrete way of com- • The purpose of the needs assessment is defined,
municating the work of a needs assessment team to and
people outside of the immediate group. It is also an • Preliminary data gathering is completed to con-
excellent tool to keep a needs assessment focused textualize the main needs assessment.
Main Activities Focus Groups Surveys Interviews Windshield Tours

& Observations
1. Develop instruments or questionnaires 1. Select locations

2. Set up times for data collection 2. Make arrangements
Implementation 3. Recruit and sample participants 3. Conduct tours
Objectives 4. Collect data 4. Conduct observations
5. Analyze data 5. Record findings
6. Summarize data 6. Summarize findings
1. List of influencing factors Collection of tools

2. Description of values that can be used by
Outputs 3. Description of assets of community the community
4. Prioritization of identified needs another time
Short Term Increase awareness of the factors that influence whether or not
Outcome Objectives residents remain in the community as they age
Foster action among the staff of the suburban department of public health and
Intermediate Term
members of their Seniors' Coalition to increase opportunities and options that will en-
Outcome Objectives
able the residents of the community to self-determine whether they will age-in-place
The mission of the Seniors' Coalition organized by the suburban

department of public health is to assure that older adults in the
Long Term community have opportunities to:
Outcome Objectives • maintain and improve their health,
• prevent disease, injury and disability,
• promote a health community for all.
Figure 36.2 Sample logic model.

The preassessment phase is an opportunity for • Action committees are established, and
the needs assessment team to learn about the com- • A final report is disseminated throughout the
munity and its social and political context. community.
The second phase of this model is called the
assessment phase. Using information obtained One of the interesting and unique features of
during the preassessment, the needs assessment the Concerns Report Method is the survey, and
team determines the scope of their work, plans the how the way it is structured produces a prioritiza-
data collection, and determines time lines, budg- tion of community needs. The feature box on the
ets, and necessary resources. It is also during this next page provides examples of statements from a
phase that the needs assessment team actually Concerns Report Method survey that focused on
gathers data and analyzes findings. identifying older adults issues and concerns
The final phase of this model is the post- related to food acquisition and preparation. From
assessment phase, during which needs are trans- the results of these surveys, a needs index is calcu-
lated into priorities for action, potential solutions lated for each statement as follows:
are identified and compared, the needs assessment • The proportion of respondents who state that an
process is evaluated, and results are communi- item is very important is calculated,
cated. Like the logic model, the three-phase model • The proportion of respondents who state that
is a structural and planning tool that does not have they are very satisfied with the item is calculated,
a clear philosophical or theoretical stance, nor does and
it identify particular methods of data collection. • Need index ⫽ proportion very important – pro-
portion very satisfied.
Concerns Report Method
Through this strategy, the needs assessment
In comparison to the logic model and the three- team is able to provide the community with a list
phase model, the Concerns Report Method has of issues in order of priority. The range of scores
a clear grounding in theories of empowerment, on the need index can range from ⫹100, which
self-help, and community development (Ludwig-
Beymer, Blankemeier, Casa-Boyots & Suarez-
Balcazar, 1996; Schriner & Fawcett, 1988). This
grounding leads to a participatory action research
approach to needs assessments that use this model.
Methodologically, the Concerns Report Method
draws on focus groups, survey research, and ana-
lytic strategies that originate in discrepancy mod-
eling (Ludwig-Beymer et al., 1996). Through the
use of the Concerns Report Method, communities
work with the needs assessment team to identify
community strengths as well as issues and con-
cerns. Strengths are then built upon to address the
issues and concerns. The basic steps of the
Concerns Report Method are as follows:
• Focus groups are conducted to identify commu-
nity values, concerns, and priorities,
• Findings from the focus groups are used to
develop a structured survey in which each item
has an importance dimension and a satisfaction
dimension,
• The survey is administered to members of the
community,
• Data are analyzed,
• Results are shared with the community through
public meetings,
• During the public meetings, community members Figure 36.3 Dr. Finlayson conducts a focus
discuss ways to preserve and enhance community group in order to identify student wellness
strengths and address issues and concerns, needs.
Sample Statements from a Concerns Report Method Survey
I have access to transportation for shopping.

How important is this to you personally?
Very important ____________ Somewhat important ____________ Not important ____________
How satisfied are you with your own situation?
Very satisfied ____________ Somewhat satisfied____________ Not satisfied ____________
I am able to shop independently.
Very satisfied ____________ Somewhat satisfied ____________ Not satisfied ____________
I am able to prepare nutritious meals independently.
I am aware of community programs that provide nutritious meals.
indicates a very high need (i.e., the issue is very education into a single project. For the purposes of
important to everyone, but no one is very satis- a needs assessment, participatory approaches pro-
fied), to –100, which indicates a very low need or vide significant theoretical guidance in terms of
alternatively, a community strength (i.e., the issue process. For example, using a participatory ap-
is very important, yet everyone is very satisfied). A proach will require active involvement of the
score of zero on this score range indicates that stakeholders in all aspects of the research process
needs are being addressed (i.e., there is a balance). and associated decision-making. Developing
effective partnerships will be key, and the primary
Community Building and needs assessor will take the role of facilitator and
technician. Beyond these theoretical and process
Participatory Approaches
guides, taking a participatory approach does not
Both community building and participatory dictate the use of specific methods during the
approaches to needs assessment focus on social needs assessment process.
change, and draw from practices of community In community building approaches to needs
development and participatory action research. assessment, the focus is on the community and on
Both approaches are based in critical social theory, fostering development of a collective to promote
and therefore focus on the critical examination of social change. Raising critical consciousness and
rules, habits, traditions, and beliefs about an issue promoting reflection are key to the use of a com-
and how these factors impact social relationships munity building approach to needs assessment.
and structures (Lindsey & McGuinness, 1998; There are four aspects or components to this model
Lindsay, Shields, & Stajduhar 1999; Wallerstein & of needs assessment, and they include citizen
Duran, 2003). In both of these approaches to action, voluntary participation and collaborative
needs assessment, the key is community involve- problem solving, empowerment, and holistic com-
ment and community ownership over the entire munity-wide outcomes (Wallerstein & Duran,
process. 2003). Like the participatory approach to needs
In participatory approaches to needs assess- assessment, the community building approach
ment, the underlying assumption is that knowledge provides strong theoretical and process guidance,
is power, and that knowledge development within but does not dictate the use of particular methods.
a community will lead to social action (Lindsay et
al., 1999; Lindsey & McGuinness, 1998; Waller- Summary
stein & Duran, 2003). Therefore, participatory
approaches to needs assessment link social science Through this review of models and approaches
to social activism, and link research, action, and to needs assessment, it should be apparent that this
Table 36.3 Advantages and Disadvantages of the Different Models and Approaches
to Needs Assessment
Advantages of Using this Model Disadvantages of Using this Model
Logic model • Tool that can assist with planning. • Provides no theoretical guidance.
• Promotes communication among • Provides no guidance about methods.
stakeholders.
• Keeps the project focused on the long-term
outcome.
Three-phase • Clearly defined, step-by-step process. • Provides no theoretical guidance.
model • Good for novices who need checklist formats • Provides no guidance about methods.
to guide them through the needs assessment
process.
• Emphasizes use of a needs assessment
advisory committee.
Concerns report • Clearly defined, step-by-step process. • May be time and resource intensive
method • Clearly defined guiding theory. depending on the community.
• Clearly defined methods.

• Role of community explicit.
Participatory • Clearly defined guiding theory. • Time and resource intensive.
approach • Role of community explicit. • Provides no guidance about methods.
• Community controls process. • May require needs assessor to train
• Involving community members in research community members in research
process increases likelihood of ownership processes.
over findings.
Community • Clearly defined guiding theory. • Time and resource intensive.
building • Role of community explicit. • Provides no guidance about methods.
approach
• Community controls process. • Involvement of multiple constituencies
• Typically involves multiple constituencies is likely to require specialized and
(e.g., business, social services, government, experienced facilitators to negotiate
etc.). multiple viewpoints.
is an area of research that requires further method- three-phase model together with a community
ological and conceptual clarity. Each of these building approach would provide guidance to a
models and approaches offers advantages and needs assessment team in terms of theory as well
disadvantages, and these are summarized in Table as step-by-step activities.
36.3. Logic models and the three-phase models are
really structural and organizational in nature, pro-
viding guidance for process but little else. Common Data Collection
Community building and participatory approaches
provide theoretical guidance, and therefore a
Methods for Needs
philosophical stance to guide the needs assessment Assessment
process. They do not provide guidance for the
step-by-step activities of a needs assessment, nor Up to this point in the chapter, variability in needs
the methods that can and should be used. The assessments and how they are designed has been
Concerns Report Method falls in between these the message. Yet, one factor that high-quality needs
two extremes. It has a clear structure, and a step- assessments have in common is that multiple and
by-step process about what activities to complete mixed methods are used to collect the data
in which order. It also has a clear theoretical ori- (Müllersdorf & Söderback, 1998). In fact, both
entation that can guide the process conceptually. Witkin and Altschuld (1995) and Reviere et al.
Ultimately, mixing and matching these models and (1996) caution against using a single method to
approaches may have the greatest utility for a inform a needs assessment process. It is this feature
needs assessment team. For example, using the of needs assessment—multiple and mixed meth-
ods—that make this form of applied research • Development of a new policy or program,
unique as well as exciting and challenging. Using • Modification of an existing policy or program, or
multiple methods means that the research team • Modification to the delivery processes of an
often must be larger and more diverse to ensure that existing program or policy (e.g., eligibility crite-
adequate expertise is available for designing and ria, staffing, funding, etc.).
analyzing the data from each method. It also means
that preparing the findings and making recommen- To prepare recommendations that a commu-
dations may become more complicated, particu- nity can use, it is critical that the needs assessor
larly when trying to understands his or her
integrate qualitative and audience. This under-
quantitative data, and data Using multiple methods standing includes issues
from different methods means that the research such as what they want
that may contradict each to know, what they value
other. These challenges team often must be larger and think is impor-
can often be overcome by and more diverse to ensure tant, what they have the
revisiting the stakehold- ability to change/influ-
ers, refocusing on the val- that adequate expertise is ence, and how they
ues of the community, and available for designing and prefer to acquire infor-
being clear about the mation. Needs assessors
scope and purpose of what analyzing the data from that engage the commu-
is and can be addressed. each method. nity members in their
Table 36.4 summarizes work should be able to
the most commonly used develop a strong sense of
methods in needs assessments, as well as their the audience for the needs assessment findings and
advantages and disadvantages. Since many of recommendations. Developing recommendations
these methods are discussed in other chapters in that can be used by the community can be facili-
this book, they are not discussed here. Instead, tated in a number of ways.
Table 36.4 provides a cross-reference to other
chapters that address the method. In addition, ref- Developing Recommendations
erences are provided at the end of the chapter for First, the needs assessor can draft and share find-
readers who wish to learn more about the particu- ings as the needs assessment is being conducted,
lar methods described in the table. and seek feedback and interpretation from key
stakeholders as the project proceeds. Taking this
Preparing Needs Assessment action has a number of benefits. First, it provides
these individuals with the opportunity to become
Findings, Developing familiar with the findings, and begin to think about
Recommendations, possible solutions to the issues being addressed. It
may make it easier for some individuals and organ-
and Taking Action izations to be open to radical changes if they have
an opportunity to contemplate the findings well
Early in this chapter, needs assessment was before the final report is disseminated.
described as both applied research and a political Engaging stakeholders in the process of deter-
process. Although early needs assessments were mining priorities for action can also facilitate the
often simply a list of development of recom-
problems, modern needs mendations. Most needs
assessments do not stop To prepare recommenda- assessments uncover
there (Reviere et al., tions that a community can more problems and
1996). Needs assessors concerns than can be
have a responsibility to use, it is critical that the reasonably addressed.
translate their findings needs assessor understands Therefore, setting priori-
into recommendations ties for action is critical.
that the community can his or her audience. Many strategies can be
consider for action. used to set priorities, for
Typically, recommendations to address the identi- example, simple rank ordering based on the fre-
fied needs fall into one of three categories (Carter, quency with which a need was identified during
1996): the data collection process (e.g., results of a
36Keilhofner(F)-36
Table 36.4 Commonly Used Methods in Needs Assessments
Chapters in
Which Method
Advantages Disadvantages Is Discussed
5/5/06
Self-administered surveys • Can be longer (people can be interrupted and return • Low response rates generally, particularly for mail-outs. 8
(mail or in-person) later). • Biased against people with low literacy levels or visual
• Nonthreatening for sensitive information. impairments.
• Don’t have to hire or train interviewers. • Usually have to do a lot of data cleaning and editing.
4:05 PM
• Potential for broader coverage of population. • Don’t know who really filled out survey.
• Medium length of data collection period. • Questions must be simple (e.g., generally avoid skip
patterns).
Interviewer-administered • Consistent data across all subjects. • Must train interviewers extensively for good reliability. 8, 20
Page 603
surveys (telephone or • Enables statistical analysis. • Don’t get participant’s own words.
face-to-face) • Relatively quick method, generally speaking. • Choices may not fit participant’s experiences.
Semistructured and open-ended • Good when the researcher knows little about the topic. • Quality of data is dependent on the quality of the 20
interviews (including key • Allows the researcher to obtain “natural wording.” interviewer.
informant interviews) • Provides the participant with the opportunity for self- • Interviewer ability to listen and probe.
expression. • Interviewer ability to guide the discussion without
• Enables the researcher to identify relevant variables controlling it.
from the participant’s perspective. • Interviewer ability to cover all topics.
• Sensitive topics are difficult to address, risk of obtaining
socially acceptable responses only.
• Time consuming.
• Challenging to analyze.
Focus groups • Provide data from a group of people more quickly and • Need to think carefully about sampling and focus group 20
less expensively than one-on-one interviews. member mix.
• Researcher can directly interact with participants. • Responses of participants are not independent of one
• Allows for clarification, follow-up questions, nonverbal another.
cues. • Results can be biased by a dominant or opinionated
• Obtain respondents’ own words. member.
• Allows respondents to react and build on the • Summarization of results can be challenging if the
responses of other people. group members have very divergent opinions.
• Can be used with children. • Moderator can bias results.
• Can be used with people with low literacy levels.
(continued)
603
36Keilhofner(F)-36
604
Table 36.4 Commonly Used Methods in Needs Assessments (continued)
5/5/06
Chapters in
Which Method
Advantages Disadvantages Is Discussed
4:05 PM
Delphi technique • Does not require participants to meet. • Can be very time consuming, depending on the number 24
• Accommodates stakeholders with different levels of iterations.
of power in the same group.
• Allows researcher to identify and use the
experience of experts.
Page 604
Nominal group technique • Focuses on identifying priorities quickly. • Not a brainstorming technique, so issues become very 24
• Accommodates stakeholders with different levels focused very quickly.
of power in the same group. • Must get participants together.
• Good for setting priorities for action.
Secondary data (e.g., • Provides a broader perspective and larger sample • Often requires sophisticated statistical knowledge. 9, 20
administrative records, sizes. • Access can sometimes be problematic (e.g., HIPPA, cost
census figures, social • Can provide contextual information that other to buy data).
indicators) methods cannot. • Quality of findings dependent on quality of original data
• Good for determining the size and scope of a collection procedures.
problem.
Observation, including • See community in its natural form. • Requires good training and data management. 20
windshield tours • Permits descriptions of people, behaviors, settings • Analysis can be challenging.
and person–environment fit. • Researchers who are external to the environment may
• Can be qualitative or quantitative. have inadequate knowledge to interpret observations
• Can accommodate different levels of participation. accurately.
Concerns Report Survey) or Sork’s approach (Delbecq, Van deVen & Gustafson, 1975/1986)
(Witkin & Altschuld, 1995). can all be used to facilitate the discussion of possi-
Sork’s approach is based on two criteria for set- ble solutions. Townhall meetings in particular are
ting priorities: importance and feasibility. The useful for obtaining commitment from stakehold-
importance of a need can be considered in a num- ers to take action on the findings of the project, and
ber of ways, as outlined by Witkin and Altschuld to engage community members in the process of
(1995). Important needs are ones that: enacting solutions.
• Are experienced by greater numbers of people,
• If addressed, would contribute to the mission and
goals of the community, Conclusion
• Are immediate in nature, and cannot be resolved
by time, and The objectives of this chapter were to provide
• If resolved, would have additional benefits in a theoretical overview of the concept of need,
other areas. examine the processes and dimensions of a needs
assessment, describe and compare models and
Feasibility addresses the extent to which approaches to needs assessment, evaluate methods
addressing the identified needs is possible. Factors commonly used in needs assessments, and outline
that play a role in feasi- how needs can be trans-
bility include current lated into actions for
knowledge, the avail- The key to these final steps solutions. The founda-
ability of human and tion presented here, in
financial resources, and of the needs assessment addition to some extra
the commitment of the process is to engage the reading on methods,
stakeholders to make should give readers a bet-
changes to their opera-
community and listen to
ter and stronger under-
tions (e.g., change poli- what they value. It is also standing of the needs
cies, develop programs, critical to be open to a range assessment process and
change resource alloca- its importance in occupa-
tions). For occupational of possible solutions, and tional therapy research
therapists, evidence- to work with the community and practice. To close
based practice plays a this chapter and summa-
large role in determining to explore what might be rize its important points,
feasibility—what evi- possible. here is a list of proposed
dence exists that the criteria for a good quality
needs can be addressed needs assessment:
effectively?
To evaluate the importance and feasibility of a • Develop a clear conceptualization of need, focus-
list of needs identified through the needs assessing on the underlying issue rather than the poten-
ment data collection, one can use the Q-sort tial solution,
methodology or other methods to rank order items • Ground the needs assessment in a clear philo-
(Chinnis, Paulson & Davis, 2001; McKeown & sophical and theoretical stance,
Thomas, 1998). Ultimately, the goal is to identify • Actively involve stakeholders,
those needs which are both high importance and • Design a rigorous project,
highly feasible to address. For these needs, one • Use multiple data collection methods,
then moves to consider potential solutions. As • Make recommendations based on empirical evi-
already noted, recommending solutions to the dence, and
identified needs may take the form of suggesting • Ensure that plans are put in place to address the
modifications to existing programs or policies, or needs that are uncovered.
the development of new ones.
The key to these final steps of the needs assess- REFERENCES
ment process is to engage the community and lis- Altschuld, J. W., & Witkin, B. R. (2000). Setting
ten to what they value. It is also critical to be open needs-based priorities. In: From needs assessment
to a range of possible solutions, and to work with to action: Transforming needs into solution strate-
gies (pp. 99–132). Thousand Oaks, CA: SAGE
the community to explore what might be possible. Publications.
Various brainstorming techniques such as townhall Bradshaw, J. L. (1972). A taxonomy of social need. In G.
meetings and the nominal group technique McLachlan (Ed.), Problems and progress in medical
care: Essays on current research (pp. 71–82). London: ment methods in healthcare and rehabilitation. Critical
Oxford University Press. Reviews in Physical & Rehabilitation Medicine, 10,
Carter, C. (1996). Using and communicating findings. In: 57–73.
R. Reviere, S. Berkowitz, C.C. Carter, and C.G. Reason, P., & Bradbury, H. (Eds.). (2001). Handbook of
Ferguson (Eds.), Needs assessment: A creative and action research: Participatory inquiry and practice.
practical guide for social scientists (pp. 185–201). Thousand Oaks, CA: SAGE Publications.
Washington, DC: Taylor & Francis. Reviere, R., Berkowitz, S., Carter, C. C., & Ferguson, C.
Chinnis, A. S., Paulson, D. J., & Davis, S. M. (2001). G. (1996). Needs assessment: A creative and practical
Using Q methodology to assess the needs of emergency guide for social scientists. Washington, DC: Taylor &
medicine support staff employees. Journal of Francis.
Emergency Medicine, 20(2), 197–203. Rush, B., & Ogborne, A. (1991). Program logic models:
Delbecq, A. L., Van deVen, A. H., & Gustafson, D. H. Expanding their role and structure for program plan-
(1975/1986). Group techniques for program planning: ning and evaluation. The Canadian Journal of Program
A guide to nominal group and Delphi processes. Evaluation, 6(2), 95–106.
Middleton, WI: Green Briar Press. Schriner, K. F., & Fawcett, S. B. (1988). Development and
Dill, A. (1983). Defining needs, defining systems: A criti- validation of the community concerns report method.
cal analysis. The Gerontologist, 33(4), 453–460. Journal of Community Psychology, 16, 306–316.
Doyal, L., & Gough, I. (1991). A theory of human need. Shi, L., & Singh, D. (2001). Delivering health care in
New York: Guilford Press. America: A systems approach (2nd ed.), Gaithersburg,
Green, L. W., & Mercer, S. L. (2001). Can public health MD: Aspen.
researchers and agencies reconcile the push from fund- Thomson, G. (1987). Needs. London: Routledge & Kegan
ing bodies and the pull from communities? American Paul.
Journal of Public Health, 91(12),1926–1929. Wallerstein, N., & Duran, B. (2003). The conceptual, his-
Hancock, T., & Minkler, M. (1997). Community health torical and practice roots of community based partici-
assessment or healthy community assessment – Whose patory research and related participatory traditions. In
community? Whose health? Whose assessment? In M. M. Minkler & N. Wallerstein (Eds.), Community based
Minkler (Ed.), Community organizing and community participatory research for health (pp. 27–52). San
building for health (pp. 139–156). New Brunswick, NJ: Fransisco, CA: Jossey Bass.
Rutgers University Press. Witkin, B. R., & Altschuld, J. W. (1995). Planning and
Kretzmann, J., & McKnight, J. (1997). Building communi- conducting needs assessments: A practical guide.
ties from the inside out: A path toward finding and Thousand Oaks: SAGE Pubications.
mobilizing a community’s assets. Chicago, IL: ACTA
Publications.
Lindsey, E., & McGuinness, L. (1998). Significant ele-
METHODS RESOURCES
ments of community involvement in participatory Allen, S. M., Mor, V., Raveis, V., & Houts, P. (1993).
action research: Evidence from a community project. Measurement of need for assistance with daily activi-
Journal of Advanced Nursing, 28(5), 1106–1114. ties: Quantifying the influence of gender roles. Journal
Lindsey, E., Shields, L., & Stajduhar, K. (1999). Creating of Gerontology: Social Sciences, 48, S204–S211.
effective nursing partnerships: Relating community Bowling, A. (1997). Questionnaire design. In A. Bowling
development to participatory action research. Journal (Ed.), Research methods in health: Investigating health
of Advanced Nursing, 29(5), 1238–1245. and health services (pp. 242–271). Buckingham, UK:
Lugwig-Beymer, P., Blankemeier, J. R., Casas-Byots, C., & Oxford University Press.
Suarez-Balcazar, Y. (1996). Community assessment in Fowler, F.J. Jr. (2002). Survey research methods (3rd ed.).
a suburban Hispanic community: A description of Thousand Oaks, CA: SAGE Publications.
method. Journal of Transcultural Nursing, 8(1), 19–27. Gubrium, J. F., & Holstein, J. A. (Eds.). (2002). Handbook
Martí-Costa, S., & Serrano-García, I. (1983). Needs assess- of interview research. Thousand Oaks, CA: SAGE
ment and community development: An ideological per- Publications.
spective. Prevention in Human Services, 2(4), 75–88. Krueger, R. A., & Casey, M. A. (2000). Focus groups: A
Maslow, A. H. (1954). Motivation and personality. New practical guide for applied research (3rd ed.).
York: Harper & Row. Thousand Oaks: SAGE Publications.
McKeown, B., & Thomas, D. (1998). Q-methodology. Morse, J. M., & Field, P. A. (1995). Qualitative research
Thousand Oaks, CA: SAGE Publications. methods for health professionals (2nd ed.). Thousand
Minkler, M. (Ed.). (1997). Community organizing and Oaks, CA: SAGE Publications.
community building for health. New Brunswick, NJ: Weiss, R. S. (1994). Learning from strangers: The arts and
Rutgers University Press. method of qualitative interview studies. New York: The
Müllersdorf, M., & Söderback, I. (1998). Needs assess- Free Press.
C H A P T E R 3 7
Using Research to Develop and Evaluate

Programs of Service
Brent Braveman • Yolanda Suarez-Balcazar • Gary Kielhofner • Renée R. Taylor
This chapter explores the integration between

research and the development and evaluation of
Program Evaluation
occupational therapy programs. Examples are pre- Program evaluation is a process documenting the
sented throughout this chapter to illustrate this inte- impact of a newly developed or existing interven-
gration and to highlight the relationship between tion or program of services. Impact can be
the research methodologies presented in this book, assessed using process and/or data on outcomes
evidence based on research, and the processes of (indicators of change). Similar to program devel-
developing and evaluating programs of service. opment, approaches to program evaluation can
range in complexity. They may involve evalua-
Program Development tion of a single aspect of an intervention or
program with a limited number of clients or
Program development is the systematic design, evaluation of multiple related interventions with a
planning, and implementation of new services, large population in order to establish their effec-
innovations, or initiatives. It can range from rela- tiveness.
tively simple ventures such as implementing a
standard approach to care in a given setting to
complex endeavors such as the creating interven- Relationships Among Program
tions for populations facing new conditions and
significant challenges. For example, program Development and Evaluation,
development occurs when the administrator of a
rehabilitation center decides to develop and intro-
Theory, and Research
duce a new approach (e.g., sensory integration)
In some circumstances, program development and
into an existing program of occupational therapy
program evaluation are taken on as separate activ-
pediatric services. Another example of program
ities in isolation of one another. For example, this
development would include an occupational thera-
might occur when an evaluation of outcomes is
pist who collaborates with a local YWCA to
requested of a program of occupational therapy
design services for women and children who have
services that has been in existence for a long
self-identified as being victims of domestic vio-
period of time. Other circumstances might call for
lence.
the development of a new program or intervention.
Regardless of the level of complexity, the inte-
In these cases, program development is often
gration of research methods into the program
accompanied by formative evaluation (ongoing
development process can foster success. Integra-
evaluation and refinement of the process of service
ting research methods into program develop-
implementation) and followed by a summative
ment can:
evaluation (evaluation of the ultimate outcomes or
• Provide formative data that shape what services effectiveness of the program). In this chapter, pro-
are provided and how they are delivered, gram development and program evaluation are
• Provide evidence about the effectiveness of ser- treated as coexisting along a continuum that
vices, includes program development, formative evalua-
• Increase the likelihood that services will be used, tion, and summative evaluation. This approach of
sustained over time, and designed to meet the undertaking program development and program
needs of participants, and evaluation simultaneously is widely supported as
• Provide evidence about the effectiveness of the preferred approach by different fields including
underlying theory being tested. the health and social sciences.
A Scholarship of Practice research and evaluation approaches and the advan-

tages and disadvantages that come with expending
Regardless of how familiar or complex an inter- less or more time and effort. The key is to match
vention is, the process of developing and evaluat- the approach and level of effort with the desired
ing a program of service can be made easier if it is outcome and the needs of your situation. The left
guided by sound principles that connect theory and side of the figure represents “low-complexity pro-
research (both methods and the resulting evidence) grams” such as the implementation of an estab-
to practice. The relationship between theory, lished approach to care in a familiar setting to a
research, and practice has been described as a limited population (e.g., developing a cardiac
scholarship of practice (Crist & Kielhofner, 2005; rehabilitation protocol for a new cardiology pro-
Hammel, Finlayson, Kielhofner, Helfrich, & gram in a general hospital). The right side of the
Peterson, 2002; Kielhofner, 2005).The scholarship figure represents “high-complexity programs”
of practice envisions a process in which theoretical such as a two-site randomized control trial that
and empirical knowledge is brought to bear on the examined the efficacy of an energy conservation
practice problems and practice raises questions to education program for people with multiple scle-
be addressed through scholarship. Maintaining a rosis (Mathiowetz, Finlayson, Matuska, Chen, &
discourse between theory, practice and research is Lou, 2005).
an ideal framework for program development,
since programs translate theory into services and Using Research to Guide
research can guide the process and demonstrate its
impact.
Program Development
For example, Braveman and Kielhofner (2006) The primary tenets of both evidence-based prac-
describe the use of theory and research evidence tice and of a scholarship of practice are that the
to develop occupational therapy programming most effective programs are designed using the
focused on preparing persons living with human best available evidence. While other forms of evi-
immunodeficiency virus/acquired immunodefi- dence such as the testimony of experts, clinical
ciency syndrome (HIV/AIDS) to return to employ- guidelines provided from professional groups,
ment in a community-based setting in which information from Internet sites, and even one’s
occupational therapy previously had not been pro- own clinical experience are valid and sometimes
vided. Similarly, Fisher and Braveman (2006) the only evidence available, the strongest evidence
describe the use of methods commonly used in is obtained through the application of sound
research such as interviews and focus groups to research strategies, tools, and methodologies. Such
collect and organize data, information, and other strategies, tools, and methodologies are described
forms of evidence to help with the processes of throughout the other chapters of this book. While
needs assessment, program planning, and program these approaches are used to carry out research to
evaluation. develop or validate the
As noted earlier, the theories that guide pro-
range of program devel- The process of developing gramming, many of the
opment efforts and the and evaluating a program of same strategies, tools,
resulting need to evaluate and methodologies may
services can vary widely service can be made easier also be used to collect
in complexity. While less if it is guided by sound prin- and synthesize data,
complex efforts may information, and other
require less sophisticated ciples that connect theory forms of evidence during
evaluation approaches, and research. the program development
the amount and useful- process. In other words,
ness of the resulting evi- rather than associating a
dence are also limited. Naturally, more complex strategy, tool, or methodology just with the gener-
program development efforts require more sophis- ation of knowledge one should realize that they
ticated evaluation approaches and these appro- also assist with the application of knowledge.
aches may result in a greater amount of evidence The process of program development may be
with broader applications. conceptualized as a series of four steps that
In research as well as in program development include:
and evaluation there is always a trade-off between
the investment of time and effort and the value of • Needs assessment,
the results. Figure 37.1 represents a continuum of • Program planning,
Chapter 37 Using Research to Develop and Evaluate Programs of Service 609
• Few resources needed • Considerable resources needed

Lower time investment Higher time investment
Lower cost Higher cost
• Less sophisticated research skills needed • More sophisticated research skills needed
Low-Complexity Approaches High-Complexity Approaches
• Low generalizability • High generalizability

• Narrower yield of data • Broader yield of data
• Greater threat to validity and reliability of results • Less threat to validity and reliability of results
Figure 37.1 A continuum of research and evaluation approaches.
• Program implementation, and results of a needs assessment to guide the choice of

• Program evaluation (Braveman, 2001; Braveman, conceptual practice models and other elements of
Kielhofner, Belanger, Llerena, & de las Heras, an intervention designed to respond to identified
2002; Braveman, Sen, & Kielhofner, 2001). needs.
Strategies for needs assessment should be cho-
Table 37.1 lists each of these four steps across
sen based on ease of use, resources available, and
the top of the table. In addition, common research
the results they produce. Research strategies and
strategies, tools, and methodologies are listed
tools such as questionnaires, surveys, or reviews of
along the left side of the table. Each cell of the
records may help you obtain specific information
table provides an example of how the strategy, tool,
about the needs of a target population such as
or methodology could be applied in the process of
demographic characteristics and services already
developing a new program or service. The strate-
utilized. For example, an occupational therapist
gies, tools, and methods presented in Table 37.1
interested in developing occupational therapy
are not an exhaustive list, but rather represent a
services for a client population not yet served (e.g.,
sampling of such approaches. Next, each of the
adults with sensory impairment who reside in an
fours steps of program development is described in
underserved urban neighborhood) might begin by
more depth and additional examples of the use of
initiating a relatively simple needs assessment.
research strategies, tools, and methodology in the
This needs assessment might involve reviewing
program development process are provided.
existing information on the prevalence of individ-
uals with sensory impairment in the neighborhood
Needs Assessment served by the medical center and information
The first step in developing a program of service is about sensory impairment services offered by
to determine the needs of the target population that other medical centers or community-based organi-
the program addresses. Needs assessment is a sys- zations within the same neighborhood.
tematic set of procedures undertaken to make deci- Other situations might require a more system-
sions about program improvements (Witkin & atic approach to needs assessment. For example,
Altschuld, 1995). As noted in Chapter 36, it is a once preliminary data are obtained, the same
process of determining what a group of persons, a administrator might gain access to the clients with
community, or a population requires in order to sensory impairment and survey them directly to
achieve some basic standard or to improve their assess the likelihood that they would use occupa-
current situation. The needs assessment process tional therapy services. Such quantitative strate-
can range from simple to complex depending on gies can assist one to begin to understand how the
the characteristics of those who will receive serv- target population may be similar or different to
ices and the scope of planned services. In more previously researched populations. Interviews or
complicated examples, determining the needs of a focus groups may generate deeper insights on the
target population may in fact be the focus of a needs of the target population such as perceived
research effort (Witkin, 1991; Witkin & Altschuld, barriers to accessing services or desired outcomes
1995). In most cases of program development of services. Qualitative strategies such as focus
however, previous research is combined with the groups can also be useful.
37Kielhofner(F)-37
Table 37.1 Sample Applications of Research Strategies, Tools, and Methodologies Within the Steps of Program Development
610
Research Strategy, Needs Program Program

Tool or Methodology Assessment Program Planning Implementation Evaluation
Questionnaires and Identify desires and needs of Determine customer preferences, Monitor and improve staff Gather information during
surveys target populations and validate perception of needs, satisfaction, identity formative and summative
5/5/06
customers internal and and gather data to plan for opportunities for evaluation of customer
external to an organization. personnel, equipment, and continuous quality satisfaction and
other programmatic needs. improvement efforts, assessment of outcomes.
facilitate communication
4:06 PM
with key stakeholders.
Record reviews Gather demographic data, rates Establish baseline benchmarks for Determine levels of Gather necessary data for
of incidence, prevalence, and productivity and financial productivity, compliance to participation in database
service utilization. monitoring as well as accreditation or other benchmarking of customer
Page 610
assessment of outcomes. standards, and collect satisfaction, financial
trend data to help plan for performance or outcomes.
financial management and
budgeting.
Interviews or focus Identify needs and desires of Garner support of key Plan and conduct human Explore critical incidents to
groups stakeholders internal and stakeholders, identify resource functions learn about cases in which
external to the organization. roadblocks to success, validate including performance customer expectations or
the focus of your product or appraisal and staff outcomes were either
service, and collect information development. surpassed or not met.
on program competitors.
Observation Visit existing programs to learn Ensure compliance with Become more familiar with the Carry out human resource
about space needs and space accreditation and safety challenges and obstacles functions including
design, work flow, and standards and gather faced by staff in service assessment of competency
customer expectations. information to help plan delivery and communicate or performance appraisal.
continuous quality a desire for open
improvement and other communication.
evaluation systems.
Program Planning of your target population might choose to use

your service rather than another option. As one
The same strategies (and sometimes the same tools approaches the step of program implementation,
or assessments) used to assess the needs of a target these strategies also assist with planning manage-
population can be adapted and used to guide pro- ment and human resource functions such as sys-
gram planning. Program planning involves: tems for continuous quality improvement, the
• Determining the type of service to be provided, assessment of competencies, and the development
• Enumerating resources needed to provide the of job descriptions.
services,
• Characterizing the population, and Program Implementation
• Identifying how the services/program will be Program implementation involves the actual deliv-
delivered. ery of the program or services. Data collection is
Continuing with the example of using an used during program implementation to document:
evidence-based approach to plan a new program of • When the service was provided,
services for adults with sensory impairment, pro- • How many hours of services were provided,
gram planning would follow the comprehensive • Who provided the service,
needs assessment described in the preceding sec- • What type of service was provided, and
tion. Following the needs assessment, the act • How many clients were provided with services.
of program planning might involve creating a
spreadsheet in which the collected information Maintaining an activity log will help the occu-
about the needs of the pational therapist gener-
population and the infor- ate a record of program
mation about other exist- The relationship between implementation. Both
ing services within the quantitative and qualita-
research and the develop- tive approaches can be
community would be
analyzed, compared, and ment, implementation, and used to collect and ana-
discussed. In addition, evaluation of programs of lyze data, information,
findings from a literature and other forms of evi-
review of any existing service is both reciprocal dence from various
outcomes research on and continuous. sources for ongoing pro-
the efficacy of various gram evaluation. The
approaches to occupa- output of these efforts
tional therapy with individuals with various types may be used for functions such as monitoring and
of sensory impairment would be summarized, improving customer and staff satisfaction, plan-
added to the spreadsheet, and compared in order ning and conducting human resource functions
to guide the development of a conceptual model such as performance appraisals and staff develop-
of care. ment, or in continuous quality improvement
Many resources are available to help occupa- efforts.
tional therapy practitioners become skilled at find- Unfortunately, once program implementation
ing, evaluating, and integrating evidence into their begins the evidence-based strategies used in pro-
practice (Braveman, 2006; Law, 2002). Question- gram planning often lapse. However, the relation-
naires or surveys may assess how potential users ship between research and the development,
view planned services and the extent to which they implementation, and evaluation of programs of
intend to use a service if it is offered. Quantitative service should be both reciprocal and continuous.
data gained during these efforts can also help with Practitioners should continuously raise questions
planning for personnel, equipment, and space that may be answered by researchers. Researchers
needs as they may provide information related to should continuously provide new evidence that
potential volume and intensity of service utiliza- should be evaluated by practitioners to improve
tion. programs of service.
Qualitative methods including interviews,
focus groups, and observations can assist with
Program Evaluation
planning a program by uncovering potential road-
blocks to the program’s success. Such strategies The fourth step of the program development
may also be useful in learning information about process is program evaluation. While listed last,
existing services or competitors and why members program evaluation is actually a continuous
process that is integrated throughout the planning 1998; Ottenbacher, Tickle-Degnen, & Hasselkus,
and implementation of a program of service. In 2002).
this section, the different types of program evalua-
tion, the different methodological strategies used General Approaches to Program
to evaluate occupational therapy interventions, and Evaluation: Formative and Summative
a framework that includes practical steps to con-
duct program evaluation are introduced. Most generally, evaluation can be summarized
Program evaluation is the use of tools, methods, under two broad categories: formative and summa-
and skills to determine if a human service or pro- tive. Formative evaluation occurs during the initial
gram is meeting the needs of participants, if the steps of planning and implementing a program and
program is offered as planned, and if it is having reports process information. It is used to assess the
the desired impact on the lives of participants extent to which actual programming matches that
(Posavac & Carey, 2003; Stufflebeam, 2001). which was planned and the extent to which pro-
Program evaluation implies a process that con- gramming and services address the identified
tributes to the provision of quality services needs of the target population. Formative evalua-
(Posavac & Carey, 2003) and to decisions about tion often focuses on the process of documenting
improving a program or intervention (Fawcett et the delivering of services. The evaluation of
al., 1996; Suarez-Balcazar & Harper, 2003). process is designed to examine how a program or
The need for program evaluation has grown intervention is being implemented; it helps verify
quite rapidly during the last 10 years (Posavac & program implementation (Linney & Wandersman,
Carey, 2003). The demand for community-based 1996). Therefore, by using process information,
services has come with an increased demand from the evaluation can improve not only the plan for
funders for agencies and organizations to conduct services but also the implementation and delivery
evaluations of their interventions and programs, of programs. When information is gathered to doc-
especially in times of budget cuts. Philanthropic ument the implementation and delivery of a pro-
entities as well as private and public sources of gram, the purpose of the evaluation is then
funding are requesting program evaluations. It is formative evaluation.
now a common practice among funding agencies More specifically, formative evaluation, or the
to make financial support of community programs evaluation of process answers questions about the
and services contingent on evaluation of such ini- program such as, Is the program or intervention
tiatives. Program evaluation provides information being implemented as planned? For instance, an
about human services and interventions in ways occupational therapy intervention designed to
that such information can be used to improve serv- teach self-advocacy skills to family members of
ices, policies, or practices (Posavac & Carey, children with disabilities may ask the following
2003). questions to respond to process evaluation: Does
Human services staff and professionals deem the parents’ need for advocacy training match what
the evaluation of their initiatives both as a necessity they are being taught? How is the intervention
and as a challenge (Connell & Kubisch, 1998; being implemented? How many parents have been
Cousins, Donohue & Bloom, 1996). Finding the trained? What specific advocacy training strategies
appropriate measurement tools and specifying and topics are being covered? Is the training going
the right indicators can be challenging (Flora as planned?
& Grosso, 1999; Suarez-Balcazar, Orellana- Keeping detailed records of program imple-
Damacela, Portillo, Sharma, & Lanum, 2003). mentation allows for the documentation of
Nevertheless, professionals are now under pressure process. Often the information collected is
to engage in practice supported by evidence. reported in terms of outputs about program
Program evaluation supports evidence-based descriptors such as:
practice. By documenting the impact of occupa- • How many training sessions were implemented?
tional therapy interventions, practitioners will be • How many parents were trained?
engaging in practice supported by evidence. • What do parents say about the program?
Evaluation theory is rooted in outcomes-based evi-
dence models, which have gained much attention Monitoring the implementation of a program is
in Occupational Therapy literature and have major also a part of formative evaluation. This is the most
implications for research and practice (Dysart & common practice in terms of evaluation. For the
Tomlin, 2002; Holm, 2000; Kielhofner, Hammel, most part, all human services document their pro-
Finlayson, Helfrich & Taylor, 2004; Law & Baum, grams by maintaining records of the services they
provide and participants served. In fact, before • Did participants’ knowledge change as a result of
conducting an outcomes evaluation, monitoring the intervention?
the program allows for an actual record of the pro- • Which intervention produced the most impact on
gram itself and of the population being served. participants?
Another example of formative evaluation • Did participants’ conditions change as a result of
involves conducting weekly staff meetings to the intervention?
determine whether a given program of services is
being delivered as it was originally designed and Researchers have also classified outcomes in
intended. Meetings might determine whether the terms of short-term outcomes, intermediate out-
intended content of services is being delivered, comes, and long-term outcomes (Fawcett et al.,
whether services are being delivered at the antici- 1996; United Way, 2003). Short-term outcomes
pated rate, whether the speak for changes in
anticipated number of participant’s knowledge
clients are utilizing the Evaluation of outcomes and/or attitudes, interme-
services, and how staff documents the impact the diate outcomes speak
are experiencing the for changes in behavior,
process of service deliv- program or intervention is and long-term outcomes
ery. Asking clients to having on participants’ speak for changes in an
complete weekly “feed- individual’s skills/com-
back forms” that reflect
attitudes, knowledge, skills, petencies and/or condi-
how they are experienc- abilities/competencies, tion. For instance, an
ing the occupational occupational therapy in-
therapy services offered
and/or conditions. tervention designed to
is another example of a increase job related-skills
type of formative evaluation. One uses process and employment status of individuals with disabil-
information to report on a formative evaluation. ities may document the following:
Summative evaluation is also an ongoing
• Short-term outcomes: Did participants’ knowl-
process but typically focuses on the outcomes or
edge of employment resources change as a func-
effectiveness of service delivery. Summative evalu-
tion of the training?
ation provides information about the extent to
• Intermediate outcomes: Did participants’ job-
which a program achieves the objectives for which
seeking skills (e.g., writing a resume, preparing
it was developed. As with the other steps described,
for an interview) change as a function of the
both quantitative and qualitative data and informa-
intervention?
tion may be gathered using strategies, tools, or
• Long-term outcomes: How many participants
methodologies that match the evaluation question
found jobs and maintained them at 3 months? Six
and the resources available for evaluation. One
months? And 12 months after training?
example of a summative evaluation involves col-
lecting data from clients with hand injuries on To evaluate the impact of an intervention,
functional outcomes following a course of therapy. researchers need to rely on scientific methods of
Evaluation of outcomes documents the impact inquiry. The next section provides an overview of
the program or intervention is having on partici- research methods used in evaluation, from the least
pants’ attitudes, knowledge, skills, abilities/com- sophisticated design to the most sophisticated
petencies, and/or conditions. The data collected design.
through summative evaluation should help assess
the merit of a program or select among inter- Scientific Research Strategies
ventions.
Outcome evaluation answers questions such as:
in Program Evaluation
The type of strategy used to assess the outcome of
• Which intervention is most effective in produc- an intervention and the success of the program
ing changes in participants’ skills (when compar- depends on many factors, including the evaluation
ing more than one intervention)? information needs of different stakeholders,
• Did the attitudes of participants change as a resources available for the evaluation, timeline,
result of the program? and deadlines imposed by stakeholders, type of
• Did participants’ behaviors or skills/competen- program being evaluated, and the timing of the
cies change as a result of the intervention? evaluation (Gabor & Grinnell, 1994). The evalua-
tion of outcomes may go from the least sophisti- Quasi-experimental Approaches

cated design, such as single group evaluation to Evaluation
design, to the most complex, such as experimental
group evaluation design. Quasi-experimental and experimental designs
This section only briefly overviews designs that according to Posavac and Carey (2003) allow for
can be used for program evaluation. See Chapter 7 establishing cause–effect relationships and answer-
for a detailed discussion of group comparison ing the following questions:
designs that can be used in program evaluation. • Does the cause precede the effect?
Table 37.2 displays a summary of the characteris- • Does the cause covary with the effect?
tics, advantages and disadvantages of each of the
designs and methodological strategies available These designs also allow alternative explana-
for evaluating occupational therapy programs and tions of the observed effects to be ruled out. Within
interventions. quasi-experimental designs, one might use a time-
series design in which data are collected several
times across a period of time. This design allows
Single-Group Evaluation Design for control of some alterative explanations for the
According to Posavac and Carey (2003), the sim- observed effects such as maturation (Campbell &
plest form of outcome evaluation is the single- Stanley, 1963). Another potential design is the
group design, which includes one observation after nonequivalent control group design. In this design,
the intervention has taken place. For instance, in more than one comparable group is observed, at
the example of a job training program for people least one of which does not receive the interven-
with disabilities, the single-group evaluation tion. Another possibility is to combine designs
design may answer the following question: How such as the time-series and the nonequivalent con-
many individuals with disabilities who com- trol design.
pleted training are employed at 6 months after
training? Or in the example of family members Experimental Designs
learning advocacy skills, outcome evaluation may
answer how many individuals acquire advocacy Experimental designs include by definition the use
skills? of random assignment to groups. In this case, a
Although this design might be an easy way for group of similar participants is assigned to differ-
human service professionals to evaluate the pro- ent treatment and nontreatment groups randomly.
gram, it does not answer the questions of whether Although this is an optimal design in terms of
participants’ knowledge or skill would have responding to questions of causation, the reality
changed if no intervention was provided. Also, the is that human services and occupational therapy
degree of change cannot be assessed because of interventions maybe not necessarily need or
the lack of a premeasure or baseline. have the resources or support for a more sophis-
ticated type of evaluation to learn if their pro-
grams are having an impact. Experimental designs,
Pretest–Posttest design for the most part, are more costly and time con-
This design implies an observation before and suming.
after the intervention has taken place. This design
will help answer the question of whether partici- Qualitative Strategies in
pants improved or changed while receiving the Program Evaluation
intervention. For instance, in the example of advo-
Evaluators often rely on the use of qualitative
cacy skills training program for families of chil-
research strategies as a compliment or an alterna-
dren with disabilities, the occupational therapist
tive to quantitative methods. Qualitative research
could assess the level of advocacy skills and
methods may be the most appropriate program
knowledge of disability rights using a validated
evaluation methods when:
instrument before and after training. Posavac and
Carey (2003) advise that when there are clear stan- • The program has complex and multifaceted
dards for the outcome of a human service program goals,
and no participants drop out of the program that • Using empowerment and/or participatory strate-
the pretest–posttest design is an alternative that is gies,
inexpensive, simple, and might provide enough • There is a strong need to be culturally sensitive to
information for human service staff. participants, and
37Kielhofner(F)-37
Table 37.2 Program Evaluation Designs: From the Least Sophisticated to the Most Sophisticated
Design Characteristics Advantages Disadvantages
Qualitative strategies Examples: Focus groups, interviews, - Rich narrative information from the - Difficult to identify outcome indicators.
5/5/06
public forums, participant perspective of participants based on their - Difficult to generalize to other similar
observation personal experience. populations.
- Helps to understand the intervention. - Difficult to establish cause–effect
- Helps to interpret quantitative data. relationships.
4:06 PM
- Culturally sensitive to some populations.
Post-measure design One systematic - Simplest form of evaluation. - Does not show change over time.
only observation/assessment after the - Useful to follow-up on simple, discrete - Difficult to generalize to other similar
intervention behaviors (e.g., how many people have jobs populations (threats to external validity).
after 6 months of completing training). - Difficult to establish cause–effect
Page 615
relationships. (threats to internal
validity).
Pre–post measure An observation/assessment is done - Simple form of evaluation, appropriate for - Difficult to generalize to other similar
before and after the intervention simple, inexpensive, and standard populations (threats to external validity).
interventions. - Difficult to establish cause –effect
- Identifies change over time. relationships (threats to internal validity).
Quasi-experimental designs Collection of data across several time - Controls for some internal validity threats - Not appropriate for complex behaviors.
Time series intervals on a single unit of behavior (maturation).
- Use by behavior analysts.
Nonequivalent Comparison of two groups that have - Allows for comparing the target group with a - Groups are nonequivalent, which
comparison group not been selected randomly and nontreated control group. threatens external validity.
are nonequivalent but somehow
similar
Experimental design Participants are selected randomly and - Allows for cause–effect relationships. - Is expensive and complex.
assigned to control and - Control for treats to internal and external - Random assignment to groups is not that
experimental group validity. feasible in applied settings.
615
• There may be different desires or conflicts of • The program/intervention to be evaluated,

interest among key stakeholders (Posavac & • Stakeholders who need to be involved,
Carey, 2003; Suarez-Balcazar et al., 2003). • Information needs of different stakeholders,
• Timeline for the evaluation, and
Qualitative strategies include observational
• Resources needed.
methods such as participant observation and direct
observation, and other strategies such as inter- It is also important to discuss roles and expec-
views with key stakeholders, focus groups, and tations and to identify evaluation questions of
public forums. Qualitative strategies, for the most interest to different stakeholders as well as how the
part, provide rich narrative information from the data gathered are going to be utilized.
perspective of those who experience the program
or intervention. Program evaluators recommend Phase II: Developing an Outcomes
combining strategies, qualitative and quantitative, Logic Model
and using multiple informants. Different stake-
holders (e.g., participants, significant others, and A program logic model is a visual representation
program staff) can provide useful and relevant of the link between program goals, resources and
information about a program. activities, outputs, outcomes, and impact (United
Way, 1996). A program logic model is usually
Planning and Conducting an Evaluation developed in an outcomes brainstorming session
that fosters critical thinking and self-determination
The following framework for conducting a pro- about the program or service (Fetterman,
gram evaluation includes the following four Kaftarian, & Wandersman, 1996).
phases: A number of outcomes models, also referred to
• Planning the evaluation, as program logic models, have been proposed in
• Developing a program logic model, the evaluation of human service programs and
• Selecting the methodology and data collection interventions. Among these frameworks are: the
procedures, and United Way of America (1996) Outcomes Meas-
• Reporting research findings and utilizing find- urement Model; Milstein & Chapel (2002) Model
ings (Suarez-Balcazar et al., 2003). See Figure of Change; Linney and Wandersman (1996) Pre-
37.2 for phases of an evaluation. vention Plus Model; and Connell and Kubisch
(1998) and Weiss (1995) Theory of Change
Approach.
Phase I: Planning the Evaluation
The outcomes model proposed by United Way
The planning of the evaluation should begin at the of America (1996) highlights the connection
same time that the program is being developed and between program goals, inputs, activities, outputs,
planned. This is critical because the use of some and outcomes. Within this model, program goals
evaluation designs imply collecting information are specific statements of who the target popula-
about participants and about the program at the tion is and what the program is intended to
onset of the program. During this phase, the eval- achieve; inputs are defined as program resources
uation team needs to clarify the following: and context. Activities and outputs include the pro-
Plan the Evaluation Develop a Program Identify the method- Interpretation and
Logic Model ology; collect and Utilization
analyze data
- Identify program
and intervention - Outcomes - Identify evaluation - Write evaluation
components. brainstorming design. report.
- Identify key session to identify: - Sampling procedure. - Share findings with
stakeholders. program goals, - Assessment tools. stakeholders.
- Assess the strategies/ - Data collection - Disseminate
evaluability of the activities, inputs/ procedures. findings.
program. resources, outputs - Collect data. - Facilitate utilization
- Identify timeline. and outcomes. - Enter data. of findings.
- Identify evaluation - Analyze data. - Use findings to
needs. impact OT
- Conduct literature practice.
review.
Figure 37.2 Program evaluation phases.

Goals Activities Resources Outputs Outcomes
Based on evidence
Figure 37.3 Development of a logic model.
gram components and strategies that take place, evaluation report is developed, the findings are dis-
while outcome evaluation involves measuring seminated among different stakeholders. The find-
changes in participants’ knowledge, skills and ings must be interpreted carefully as decisions are
behaviors, and/or attitudes; or changes in commu- likely to be made based on the evaluation findings.
nity conditions (Figure 37.3). The use of an experimental design calls for stron-
ger recommendations, however, while a nonexper-
Phase III: Identifying the imental design may yield weak results about key
Methodology and Data Collection outcome indicators. Qualitative methods may
yield interesting and rich subjective comments
During this phase, one must select the evaluation
from participants about the program.
design that is most suitable to the problem and
Dalton, Elias, and Wandersman (2001) con-
intervention being evaluated. In addition, impor-
sider effective communication of findings impera-
tant decisions about sampling, assessment tools,
tive, given its potential impact on community
and methods to measure relevant outcome indica-
members and agencies’ subsequent actions in their
tors need to be considered. These important deci-
communities. However, working together with
sions are followed by data collection and data
agency staff, other health professionals and diverse
analysis. Most commonly, the sampling, assess-
stakeholders in interpreting and reporting findings
ment tools, and data collection strategies depend
can minimize the challenges (Harper & Salina,
on the design selected. In this decision making
2000).
process, one needs to keep in mind what is feasi-
Maximizing the use of evaluation information
ble, measurable, realistic, objective, and reliable.
is a crucial component of any evaluation effort.
Evaluation experts have called for the use of
Currently, there is an emphasis in the evaluation
multiple levels of analysis and multiple measures
field on assessing the impact of the evaluation
in the evaluation of human services. Those sup-
process on the agencies (Cousins, Donohue &
porting a more participatory and empowering
Bloom, 1996). This is a reaction to the fact that
approach to evaluation have also recommended
evaluation results have sometimes been ignored by
using both qualitative and quantitative strategies
stakeholders (Mayer, 1996; Weiss, 1995). Several
(Fetterman et al., 1996) and that the unit of analy-
authors have reported that participatory and
sis include the individual and community condi-
empowerment approaches to evaluation increase
tions (Hollister & Hill, 1995).
the likelihood of the stakeholders using the infor-
The selection of a methodology is dependent on
mation generated by the evaluation. This increase
the program to be evaluated and the population
occurs due to:
being served. It is critical that tools and methods
selected be culturally sensitive to the population of • A sense of ownership of the evaluation,
interest (Marín, 1993). For instance, researchers • Credibility and trust in the process, and
have asserted that ethnic minority individuals— • A more thorough cognitive processing of the
African-Americans and Hispanics—are more information (Cousins & Earl, 1995; Suarez-
likely to respond to one-on-one interviews and Balcazar & Harper, 2003; Wandersman et al.,
focus groups than to mail surveys (Suarez- 2004).
Balcazar et al., 2003). Similar strategies have been
Utilization should be a focus of the evaluation
reported as successful ways to collect information
throughout the entire process. At the beginning of
from people with disabilities.
the evaluation process, all stakeholders invested in
the evaluation should be asked to consider ways in
Phase IV: Reporting and Utilizing Findings
which they would use the evaluation information.
During this phase, a report needs to be produced During the planning and data collection phase,
based on the data analysis conducted. Once an stakeholders should be included in the decision
making process and should be kept informed of the self-reported opinions, practices and consequences.
preliminary information to ensure utilization. Evaluation Practice, 17, 207–226.
Cousins, J. B., & Earl, L. M. (1995). The case for partici-
Evaluation utilization should be prompted through patory evaluation: Theory, research, practices. In J. B.
a thorough discussion of findings. Cousins & L. Earl (Eds.), Participatory evaluation in
education: Studies in evaluation use and organizational
learning (pp. 3–17). London: Falmer.
Conclusion Crist, P., & Kielhofner, G. (2005) The Scholarship of
Practice: Academic and practice collaborations for
promoting occupational therapy. Binghamton, NY:
This chapter described general approaches to Hayworth Press.
developing and evaluating occupational therapy Dalton, J. H., Elias, M. J., & Wandersman, A. (2001).
programs of service. A four-step process for pro- Community psychology: Linking individuals and com-
gram development was described and the step of munities. Belmont, CA: Wadsworth/Thomson
Learning.
program evaluation was explored in further detail. Dysart, A. M., & Tomlin, G. S. (2002). Factors related to
Throughout the chapter, the value of integrating evidence-based practice among U.S. occupational ther-
research strategies, methodologies, and findings apy clinicians. American Journal of Occupational
into program development and program evaluation Therapy, 56, 275–284.
(e.g., an evidence-based approach) was empha- Fawcett, S. B., Paine-Andrews, A., Francisco, V. T.,
Schultz, J. A., Richter, K. P., Lewis, R. K., et al.
sized. Examples of the types of services that might (1996). Empowering community health initiatives
be developed, implemented, and evaluated by through evaluation. In D. Fetterman, S. Kaftarian, & A.
occupational therapy personnel were provided to Wandersman (Eds.), Empowerment evaluation:
reinforce that these efforts may occur across a con- Knowledge and tools for self-assessment and accounta-
bility (pp. 256–276). Thousand Oaks, CA: SAGE
tinuum of complexity and effort. Regardless of the Publications.
level of complexity, the tools, methodologies, and Fetterman, D. M., Kaftarian, S., & Wandersman, A. (Eds.).
approaches covered in other chapters of this book (1996). Empowerment evaluation: Knowledge and
may be utilized to foster success in program devel- tools for self-assessment and accountability. Thousand
opment and evaluation endeavors. Oaks, CA: SAGE Publications.
Fisher, G. S., & Braveman, B. (2006). Understanding
health systems. In B. Braveman (Ed.), Leading and
REFERENCES managing occupational therapy services: An evidence-
Braveman, B. (2001). Development of a community-based based approach (pp. 23–52). Philadelphia: F. A. Davis.
return to work program for people living with AIDS. In Flora, C., & Grosso, C. (1999). Mapping work and out-
B. P. Velde & P. P. Wittman (Eds.), Community occupa- comes: Participatory evaluation of the farm preserva-
tional therapy education and practice (pp. 113–132). tion advocacy network. Sociological Practice: A
New York, NY: The Haworth Press. Journal of Clinical and Applied Sociology, 1(2),
Braveman, B. (2006). Leading and managing occupational 133–155.
therapy services: An evidence-based approach. Gabor, P. A., & Grinnell, R. M., Jr. (1994). Evaluation and
Philadelphia: F. A. Davis. quality improvement in the human services. Boston:
Braveman, B., & Kielhofner, G. (2006). Developing Allyn & Bacon.
evidence-based occupational therapy programming. In Hammel, J., Finlayson, M., Kielhofner, G., Helfrich, C., &
B. Braveman (Ed.), Leading and managing occupa- Peterson, E. (2002). Educating scholars of practice: An
tional therapy services: An evidence-based approach approach to preparing tomorrow’s researchers. Occup-
(pp. 215–244). Philadelphia: F. A. Davis. ational Therapy in Health Care, 15, 157–176.
Braveman, B., Kielhofner, G., Belanger, R., Llerena, V., & Harper, G. W., & Salina, D. (2000). Building collaborative
de las Heras, C. G. (2002). Program development. In partnerships to improve community-based HIV preven-
G. Kielhofner (Ed.), The model of human occupation: tion research: The university-CBO collaborative
Theory and application (3rd ed., pp. 491–519). (UCCP) model. Journal of Prevention and Intervention
Baltimore, MD: Williams & Wilkins. in the Community, 19, 1–20.
Braveman, B., Sen, S., & Kielhofner, G. (2001). Hollister, R. G., & Hill, J. (1995). Problems in the evalua-
Community-based vocational rehabilitation pro- tion of community-wide initiatives. In J. Connell, A. C.
grams. In M. Scaffa (Ed.), Occupational therapy in Kubisch, L. B. Schorr, & C. H. Weiss (Eds.), New
community-based practice settings (pp. 139–161). approaches to evaluating community initiatives:
Philadelphia: F. A. Davis. Concepts, methods, & contexts (pp. 127–172).
Campbell, D. T., & Stanley, J. C. (1963). Experimental Washington DC: Aspen Institute.
and quasi-experimental designs for research. Chicago: Holm, M. B. (2000). Our mandate for the new millennium:
Rand-McNally. Evidence-based practice, 2000 Eleanor Clarke Slagle
Connell, J., & Kubisch, A. C. (1998). Applying a theory lecture. American Journal of Occupational Therapy,
of change approach to the evaluation of comprehensive 54, 575–585.
community initiatives: Progress, prospects, and prob- Kielhofner, G. (2005). Scholarship and Practice: Bridging
lems. In K. Fulbright-Anderson, A. C. Kubisch, & J. P. the Divide. American Journal of Occupational
Connell (Eds.), New approaches to evaluating commu- Therapy, 59, 231–239.
nity initiatives (pp. 15–44). New York: Aspen Institute. Kielhofner, G., Hammel, J., Finlayson, M., Helfrich, C., &
Cousins, J. B., Donohue, J. J., & Bloom, G. A. (1996). Taylor, R. R. (2004). Documenting outcomes of occu-
Collaborative evaluation in North America: Evaluators’ pational therapy: The center for outcomes research and
education. American Journal of Occupational Therapy, Wandersman, A., Keener, D., Snell-Johns, J., Miller, R.,
58(1), 15–23. Flaspohler, P., Livet-Dye, M., Mendez, J., Behrens, T.,
Law, M. (2002). Evidence-based rehabilitation: A guide to Bolson, B., & Robinson, L. (2004). Empowerment
practice. Thorofare, NJ: Slack. evaluation: Principles and action. In L. Jason, K. Keys,
Law, M., & Baum, C. (1998). Evidence-based occupational Y. Suarez-Balcazar, R. Taylor, M. Davis, J. Durlak, &
therapy. Canadian Journal of Occupational Therapy, D. Isenberg (Eds.), Participatory community research:
65, 131–135. Theories and methods in action (pp. 139–156).
Linney, J. A., & Wandersman, A. (1996). Empowering Washington, DC: American Psychological Association.
community groups with evaluation skills: The preven- Weiss, C.H. (1995). Nothing as practical as good theory:
tion plus III model. In D. Fetterman, S. Kaftarian & Exploring theory-based evaluation for comprehensive
A. Wandersman (Eds.), Empowerment evaluation: community initiatives for children and families. In J. P.
Knowledge and tools for self-assessment and accounta- Connell, A. Kubisch, L. Schorr, & C. H. Weiss (Eds.),
bility (pp. 256–276). Thousand Oaks, CA: SAGE New Approaches to evaluating community initiatives:
Publications. Concepts, methods, and contexts (pp. 65–92).
Marin, G. (1993). Defining culturally appropriate commu- Washington, DC: Aspen Institute.
nity interventions: Latinos/as as a case study. Journal Witkin, B. R. (1991).Setting priorities: Needs Assessment
of Community Psychology, 21, 149–161. in a time of change. In R. V. Carlson & G. Awkerman
Mathiowetz, V., Finlayson, M., Matuska, K., Chen, H. Y., (Eds.), Education planning: Concepts, strategies, and
& Lou, P. (2005). Randomized controlled trial of an practices (pp. 241–266). White Plains, NY: Longman.
energy conservation course for persons with multiple Witkin, B. R., & Altschuld, J. W. (1995). Planning and
sclerosis. Multiple Sclerosis 11(5), 592–601. conducting needs assessments: A practical guide.
Mayer, S. E. (1996). Building community capacity with Thousand Oaks, CA: SAGE Publications.
evaluation activities that empower. In D. Fetterman, S.
Kaftarian, & A. Wandersman (Eds.), Empowerment RESOURCES
evaluation: Knowledge and tools for self-assessment American Evaluation Association: Information about eval-
and accountability (pp. 256–276). Thousand Oaks, uation practices, methods, and uses through application
CA: SAGE Publications. and exploration of program evaluation.
Milstein, B., & Chapel, T. (2002). Developing a logic http://www.eval.org/
model or theory of change. Community tool box Getting to Outcomes: Methods and Tools for Planning,
(Chapter 2, section 7). Retrieved April 25, 2004, from self-evaluation and accountability: A manual that works
http://ctb.ku.edu/tools/EN/chapter_1002.htm. through 10 questions that incorporate the basic ele-
Ottenbacher, K. J., Tickle-Degnen, L., & Hasselkus, ments of program planning, implementation, evaluation
B. R. (2002). Therapists awake! The challenge and sustainability. http://www.stanford.edu~davidf/
of evidence-based occupational therapy. The empowermentevaluation.html#GTO
American Journal of Occupational Therapy, 56(3), Enhancing program Performance with Logic Models: Step
247–249. by step development of a logic model.
Posavac, E. J., & Carey, G. (2003). Program evaluation: http://www.uwex.edu/ces/Imcourse/#
Methods and case studies. Englewood Cliffs, NJ: Electronic Resources for Evaluators: This site contains
Prentice-Hall. many different links to program evaluation resources.
Stufflebeam, D. L. (2001). Evaluation models. New direc- www.itrs.usu.edu/AEA
tions for program evaluation( no. 89). San Francisco, The Promising Practice Network: Practical programmatic
CA: Jossey-Bass. and practice information to help increase the effective-
Suarez-Balcazar, Y., & Harper, G. W. (Eds.) (2003). ness and positive impact of work with children, fami-
Empowerment and participatory evaluation of commu- lies and communities.
nity interventions: Multiple benefits. New York: The http://www.promisingpractice.net/
Haworth Press. W. K .Kellogg Foundation Evaluation Handbook: You can
Suarez-Balcazar, Y., Orellana-Damacela, L., Portillo, N., get a free copy of their evaluation manual and logic
Sharma, A., & Lanum, M. (2003). Implementing model development guide. www.wkkf.org/documents/
an outcomes model in the participatory evaluation wkkfevaluationhandbook/evalhanbook.pdf
of community initiatives. In Y. Suarez-Balcazar &
G. W. Harper (Eds.), Empowerment and participa- Occupational Therapy Evidence-Based Practice
tory evaluation of community interventions: Multiple Internet Resources
benefits (pp. 5–20). New York: The Haworth Press. OTSeeker http://www.otseeker.com/
United Way of America (1996). Outcome and measurement Occupational Therapy Critically Appraised topics
resource network. Retrieved April 25, 2004, from http://www.otcats.com
http://national.unitedway.org/outcomes/. AOTA Evidence-based practice project http://www.aota.org
United Way of America (2003). Outcome measurement Center for Evidence-based Rehabilitation at McMaster
resource network. Retrieved April 25, 2004, from University http://www.fhs.mcmaster.ca/rehab/
http://national.unitedway.org/outcomes/. centre.htm
C H A P T E R 3 8
Participatory Research in
Renée R. Taylor • Yolanda Suarez-Balcazar • Kirsty Forsyth • Gary Kielhofner
Participatory approaches to research have been practice are isolated from practice settings and
applied by investigators to address a number of practitioners (Peloquin & Abreu, 1996; Thompson,
health issues ranging 2001). As a result, they
from macro-level com- may not be aware of
munity needs to specific Ultimately, the difficulties of important circumstances
needs of disabled clients applying research in practice and constraints faced by
for resource acquisi- the practitioners who are
tion and empowerment can be linked to the fact that expected to use that re-
(Balcazar, Taylor, et al., practitioners ordinarily have search (Higgs & Titchen,
2004). Recently, partici- 2001). Even when con-
patory approaches have little influence over what gets ducting applied research
been discussed and used studied and how it is studied. studies in occupational
in occupational therapy therapy, academics have
(Boyce & Lysack, 2000; largely conceived and
Cockburn & Trentham, 2002; Hammel, Finlayson, executed the research, with practitioners mostly
& Lastowski, 2003; Letts, 2003; Redick, McClain, filling secondary roles as consultants, advisors,
& Brown, 1999; Townsend, Birch, Langley, & service providers, or data collectors.
Langile, 2000). This chapter overviews participa- Not surprisingly, practitioners have expressed
tory research and discusses its rationale, principles, concerns that research findings lack relevance to
procedures, and steps. In addition, it illustrates how clinical situations, address irrelevant topics, and
participatory research can be used within occupa- fail to present findings so as to facilitate application
tional therapy. (Creek & Ilott, 2002; Dubouloz, Egan, Vallerand,
& Von Zweck, 1999; Dysart & Tomlin, 2002;
Metcalfe et al., 2001; Sudsawad, 2003). Even when
The Need for Participatory practitioners indicate that they believe that research
Research in Occupational holds value for practice, they report substantial dif-
ficulty integrating it into their practice (Dubouloz
Therapy et al., 1999; McCluskey, 2003; McCluskey &
Cusick, 2002). Ultimately, the difficulties of apply-
The need for participatory research in occupational ing research in practice can be linked to the fact
therapy is indicated by two important circum- that practitioners ordinarily have little influence
stances. The first is the gap that often exists over what gets studied and how it is studied.
between research and practice in occupational
therapy. The second is the growing call for clients’ The Need for Consumer Voice
voices in shaping the aims and content of occupa-
Within occupational therapy, the concept of client-
tional therapy services they receive.
centered practice makes the important point that
individual clients should have a voice in determin-
The Research–Practice Gap
ing their services (Law, 1998). However, outside
There can be a significant gap between what the field, disability scholars and activists are calling
research suggests and what actually occurs in for more. They argue that the disability community
professional practice (Kielhofner, 2005a). While should have a voice in the development and valida-
many factors contribute to this gap, a key factor is tion of services they receive (Fawcett, Seekins,
how the knowledge that is supposed to guide prac- Whang-Ramos, Muiu, & Suarez-Balcazar, 1987).
tice is created (Schon, 1983). Traditionally, aca- This call suggests that occupational therapy needs
demics who create theory and evidence for to go beyond client-centered practice to embrace
Chapter 38 Participatory Research in Occupational Therapy 621
a disability community-centered practice that is chapter. Many researchers integrate qualitative

informed by the collective experiences and per- research methods into participatory research.
spectives of disabled con- However, participatory
sumers of our services research is also compat-
(Kielhofner, 2005b). Frequently, disabled con- ible with rigorous exper-
Scholars from disabil- sumers observe that their imental designs (Taylor,
ity studies argue that the Braveman, & Hammel,
experience of living with individual and collective 2004). Participatory re-
a disability is, in part, a voices are missing from how search often uses a com-
function of social oppres- bination of quantitative
sion, discrimination, and
occupational therapy con- and qualitative strate-
exclusion, and that indi- strues disability and how to gies. For example, data
viduals with disabili- improve such services. collection in a study
ties have been exploited, may combine standard-
oppressed, ridiculed, ex- ized measures along
cluded, and disadvantaged by society (Fine & with interviewing key informants and conducting
Asch, 1990; Hahn, 1990; Katz, Hass, & Bailey, focus groups and public forums. Moreover, partic-
1988; Meyerson, 1990; Oliver, 1990). Moreover, ipatory research is compatible not only with
they argue that rehabilitation services, including exploratory and case study designs but also with
occupational therapy, have the potential to be quasi-experimental and experimental designs.
oppressive, stigmatizing, and largely irrelevant to Participatory research can take many forms but
the needs of individuals with disabilities the following are four key characteristics:
(Kielhofner, 2005b). This situation exists, in part,
because of the lack of control given to the con- • Participatory research is conducted in the setting
sumer in deciding what services are needed and or type of setting where the knowledge to be gen-
how they should be provided (Charlton, 1998). erated is expected to have relevance. In occupa-
Frequently, disabled consumers observe that their tional therapy this means that participatory
individual and collective voices are missing from research takes place in practice contexts,
how occupational therapy construes disability and • Participatory research involves innovation and
how to improve such services (Kielhofner, 2004). experimentation that allows inquiry to generate
new or modified services and examine how they
The Promise of Participatory work. Consequently, participatory research typi-
Research in Addressing These Needs cally combines an ongoing and reflective process
of initial investigation, ongoing input from the
Participatory research is predicated on the belief stakeholders, changes and modifications in the
that research should be committed to solving prac- services, and examination of the impact of those
tical problems, and to that end, should involve changes. This process is used to continuously
stakeholders as equal partners in the research improve services in concert with gathering evi-
process. In occupational therapy, stakeholders are dence about their impact (Reason & Bradbury,
the therapists who deliver, the clients who receive, 2001),
and the communities or groups that are affected by • Participatory research seeks to empower stake-
occupational therapy services. Hence, participa- holders by allowing them to shape the research
tory research is well suited to address the need for agenda so that their needs remain the focus of the
greater practitioner and client voice in developing research process. By definition, participatory
and studying occupational therapy service. As research aims to achieve practical outcomes
such, it promises to contribute knowledge that as they are defined by therapists, clients, and
practitioners can readily use and that consumers other stakeholders, rather than simply to address
will find relevant to their needs. problems conceptualized by the researcher
(Freire, 1970, 1993; Park, 1999). In participatory
Definitions and Approach research, stakeholders engage in many or all
aspects of the research process (Park, 1999), and
of Participatory Research • Participatory research brings stakeholders into a
mutual dialogue and cooperation with investiga-
Participatory research is an approach to doing tors to address issues of relevance to those
research that embraces certain values, perspec- involved. The researcher’s role in participatory
tives, principles, and processes outlined in this research is to become an intimate knower and
facilitator of the research process (Balcazar, when all partners are empowered to address their
Garate-Serafini, & Keys, 2004). agendas as much as possible. Moreover, when there
are conflicting agendas a process of negotiation
must occur and result in a fair compromise.
General Principles for Elden and Levin (1991) argue that participatory
Implementing Participatory research empowers those involved in three ways:
First, specific insights, new understandings, and
Research in Occupational new possibilities for addressing issues are gener-
ated, empowering those involved in the process of
Therapy discovery. Second, those who engage in participa-
tory research learn how to learn. This aspect is
Because participatory research takes place in prac- especially beneficial to practitioners or consumers
tice settings and must be responsive to the circum- who may feel insecure about themselves and their
stance of those settings and to the stakeholders who own knowledge. Third, participatory research is
are involved, it requires considerable flexibility on empowering because it often leads those involved
the part of the researcher. Consequently, there is no to change their own circumstances. True empow-
single or fixed way to conduct participatory re- erment occurs when stakeholders are increasingly
search. Nonetheless, scholars have suggested gen- able to sustain what they learned to do through the
eral principles for this type of inquiry (Balcazar, research process in order to address new needs and
Keys, Kaplan, Suarez-Balcazar, 1998; Nyden, barriers that occur in the future.
Figert, Shibley & Burrows, 1997; Selener, 1997).
Building on their recommendations we offer the Participatory Research Should Involve a
following key principles that should shape partici- Dialogical Process Among Constituents
patory research in occupational therapy.
that Leads to Critical Awareness
Stakeholders Must Be Recognized Dialogue gives researchers a more accurate under-
as Having the Capacity to Participate standing and appreciation of practice settings and
consumers’ lives. Conversely, it gives practitioners
Fully in the Research Process and clients an understanding of how inquiry works
In participatory research, researchers must recog- and can address practical problems with which they
nize stakeholders’ capacity to be involved in are concerned. Beyond the learning that occurs
research. This recognition begins with acknowl- with the exchange of information, dialogue can
edging the kind of expertise that stakeholders bring also lead those involved to reflect on their own
to the research process. For instance, practitioners assumptions, attitudes, knowledge, and behavior in
bring their accumulated experience in day-to-day a critical way that leads to new awareness.
practice as well as their local knowledge of the con- The dialogue between practitioners, consumers,
text under study. Consumers bring their intimate and researchers is facilitated by using strategies
knowledge of the problems they face in everyday such as listening sessions, focus groups, public
life as well as their own desires and aims for forums, and constant ongoing team meetings in
improving their lives. Ultimately, in participatory which all stakeholders are involved. When imple-
research, stakeholders must be viewed as co- mented in this way, participatory research can be a
researchers working in partnership with the investi- transformative and liberating process of mutual
gators. discovery.
Participatory Research Should The Research Agenda Is Shaped by the

Empower Everyone Involved Researchers, Practitioners, and Consumers
and It Aims to Address Local Social Issues
Everyone involved in participatory research has
their own agenda. Researchers typically want to Participatory research works best when the agen-
create new knowledge, generate publications, meet das of all those involved are openly discussed and
the expectations of funders, and provide practical addressed. Participatory research means that stake-
and research opportunities for students. Practi- holders other than the researchers are involved
tioners typically want to increase their skills, pro- to some significant extent in helping to identify
vide better services, and have evidence to support what gets studied, how questions are formulated,
what they do. Consumers want to improve their what kinds of data will be collected, and how
quality of life and have more control over their own data will be analyzed, shared, and used to make
lives. Participatory research is most successful change. Involving practitioners and clients in these
decisions often requires a great deal of discussion Creating a Genuine Dialogue

and negotiation between the stakeholders and the
researchers. In traditional occupational therapy Because participatory research involves bringing
scholarship, the research agenda is formulated by together people with quite different backgrounds
the researchers based on personal research agendas and perspectives, interactions can readily involve
(i.e., the desire to prove or disprove an existing mistrust and misunderstanding. A productive dia-
theory). In participatory research, the researcher’s logue requires finding common ground between
agenda is important but it must be balanced with what are often the disparate perspectives of
researchers and practitioners. Genuine dialogue
the agendas of stakeholders such as the therapists
requires:
or consumers.
• Hearing others’ perspectives in a nonjudgmen-
Participatory Research Generates tal way,
• Sharing one’s knowledge and perspectives
Knowledge that is Intended to be Used openly with others,
Utilization of findings to either inform or shape • Willingness to change one’s own perspective,
services, practices, and policies is a key feature of • Working toward a common language with others,
participatory research. Moreover, participatory • Confronting legitimate disagreements over what
is most important,
research often involves an ongoing cycle of
• Negotiating and compromising,
research informing practice and practice informing • A two-way communication style,
research. Thus, scientific knowledge and practical • An attitude of being as ready to learn as to
knowledge mutually inform each other. teach,
In participatory research, practical real-world • Recognizing that others have important knowl-
utility is considered, along with empirical rigor, as edge to contribute, and
a hallmark of good research (Higgs & Titchen, • Being ready to acknowledge diversity of
2001). Participatory research recognizes that gen- opinions.
uine knowledge arises out of efforts to achieve
desired changes or solve problems in a particular
context (Bradbury & Reason, 2001; David, Zakus, These factors include:
& Lysack, 1998). Participatory research projects • The aims and scope of the research,
are interested in practical outcomes at the local • Organizational and contextual policies,
level. In the case of occupational therapy, they ask • The culture of those involved,
such questions as: What are the specific changes in • Available resources,
practice and its impact resulting from the research • The level of readiness and desire of stakeholders
process? for participation, and
• The collaborative attitude and approach taken by
Inserting Stakeholders’ Voices the researchers.
into the Research Process Approaches to Practitioner and

As we have noted, participatory research aims to
Consumer Participation in Research
give stakeholders a genuine role in the research The particular form that the participatory research
process so that their perspectives and needs are process actually takes depends on the context of the
addressed. A key challenge of participatory research. An important factor to consider is the
research is to find ways degree of power or con-
in which practitioners trol that the stakeholders
and consumers have a In participatory research, have over the research
true voice in shaping the the researcher’s agenda is process. Danley and
questions, methods and Langer-Ellison (1999)
outcomes of the research important but it must be suggest that we can think
process (Boyce & balanced with the agendas of a continuum of power
Lysack, 2000; Suarez- held by stakeholders that
Balcazar, Harper & of stakeholders such as the spans from little power to
Lewis, 2005; Taylor et al., therapists or consumers. full power or control. At
2004). The appropriate the low end of the spec-
approach to involving trum are advisory com-
stakeholders in the research process depends on mittees, which are sometimes called participatory.
many factors. The reality in such research is that stakeholders
have some involvement but ultimately very little Sharing Power

power or authority over the research project.
On the other end of the continuum are projects Participatory research often challenges investi-
in which participants have full control over the gators to share power in unaccustomed ways.
research process, including hiring and firing Conversely, it asks therapists and clients to
authority over the professional researchers. take on responsibilities for and control over
Midpoints on the continuum include hybrid proj- matters for which they ordinarily have little or
ects in which stakeholders have high degree of con- no involvement or influence. As a result, all those
involved in participatory research have to con-
trol over the research process, but researchers are
stantly reflect on their own attitudes and behav-
responsible to outside funding agencies and thus iors with reference to issues of power. This is
retain decision-making authority over some areas. not necessarily an easy task, as often researchers
For example, Whyte’s (1991) approach involves are the ones who come in with the resources and
stakeholders in the research process “from the ini- funding and the aura of expertise of research and
tial design of the project through data gathering therapists and/or clients can be readily intimi-
and analysis, to the final conclusions and actions dated. Among other things, true power sharing
arising out of the research” (p. 7). In this approach, requires:
stakeholders become actively involved in the quest • Shared responsibility, voice, and decision-
for information and ideas to guide their future making about all aspects of the investigation,
actions. • Respect and acknowledgement of the unique
Table 38.1 was derived from a schema proposed expertise and insights of all those involved,
by Danley and Langer-Ellison (1999) and revised • Willingness to step outside their usual roles and
by Balcazar, Taylor, et al. (2004). This version, responsibilities,
which is an adaptation to occupational therapy • Identification and remediation of sources of
research, provides a means of conceptualizing the power imbalance such as money and access to
technology, and
extent of stakeholder involvement in the clinical
• Sharing resources (e.g., paying for a staff time
research process. Practitioners’ and consumers’ to devote to the project, providing participants
roles are classified on the basis of three criteria: the with stipends).
degree of control that participants have over the
research process (Litvak, Frieden, Dresden, & Doe,
1997), the extent of collaborative decision-making
between stakeholder participants and professional on alternative epistemologies that recognize, for
researchers (Turnbull & Friesen, 1997), and the example, the importance of experiential and proce-
levels of input from and commitment of partici- dural knowledge (Bradbury & Reason, 2001;
pants with the research process (Gordon, 1997). Maxwell, 1992). This means that participatory
This schema can be useful in evaluating how research values the kind of knowledge that is gen-
“participatory” a research project is. It is important erated among the participants in the study (i.e.,
to note that one end of the continuum is not auto- what they have experienced and what they have
matically superior to the other. Rather, the contin- learned to do). In this same vein, Selener (1997)
uum should be seen as a structure for thinking suggests that feeling and acting are ways of know-
about the amount of stakeholder involvement the ing that should also emerge from research. He
researcher is willing to negotiate, is good for the argues that traditional scientific methods rely
project, and is allowed by funders and other orga- exclusively on cognitive activities as a source of
nizational constituencies who affect the research knowledge. Participatory researchers typically
resources and implementation. embrace the idea that reflection and action are
important to the research process.
Knowledge-Creation and
Participatory research also values the stake-
holders’ knowledge and experiences as important
Evaluation in Participatory resources. Such subjective knowledge is viewed as
a necessary part of the process of understanding
Research any situation Elden and Levin (1991) argue that
those inside a particular context get to know more
Within participatory research, knowledge is rede- about it and have more ways of making sense
fined and judged in ways that go beyond the tradi- of their world than would be possible for any out-
tional focus on propositional, rationally deduced sider to appreciate. The best way to access such
forms of knowledge and the processes for ensuring knowledge is through dialogue, allowing individu-
rigor of such knowledge. There is a new emphasis als to share their views in a free and supportive
Table 38.1 The Continuum of Stakeholder Involvement in Participatory Research Implementation

Level of Degree of Control Typical Amount/ Degree of Commitment/
Participation Influence type of Collaboration Ownership
Nonparticipatory Practitioners and clients have Serve as implementers or None
no control or influence on recipients of services
the research process. studied or as participants
in the study.
Low Practitioner and/or client Serve as advisors to the Minimal commitment
opinions and feedback are research project. and ownership
considered and used by Provide feedback in pilot
researchers at the latter’s studies or discuss
discretion. implications of findings at
advisory board meeting.
Medium Practitioners and/or clients, Provide ongoing advice, Multiple commitments
and/or representatives of review, and consultation. to and partial
consumer groups participate Participate in discussions ownership of the
in research meetings and in leading to decisions research process
aspects of the research about the research.
implementation.
High Practitioners, clients, and/or Full partners in making key Full commitment to and
representatives of consumer research decision. equal ownership of
groups function as equal the research
partners with researchers in process with the
making all decisions about researchers
the research.
Very high Practitioners, clients, and/or Research leaders. Full commitment and
consumers groups lead the complete ownership
research with researchers of the research
assisting them. process
process. In this way, stakeholders’ perspectives can ous generalizable knowledge with concerns of
define: solving real-world problems and empowering
stakeholders in the research context (Kielhofner,
• What issues are important,
2005a). According to Elden and Levin (1991), par-
• How problems can be framed and defined,
ticipatory research involves coming to understand a
• What strategies work in the community, and
particular situation by working with those within it
• What are local challenges and barriers to change.
to develop, test, and enhance knowledge that
This kind of local knowledge is critical to the improve people’s lives. In this approach, resear-
success of participatory research and is included chers are interested in generating knowledge that
whether the primary research design is qualitative helps people learn how to better control their cir-
or quantitative. In qualitative participatory studies, cumstances. This approach implies a cycle of
the subjective experiences of stakeholders are used knowledge generation in which theory shapes prac-
throughout. In quantitative participatory studies, tice and practice shapes theory. Theory that is gen-
subjective knowledge and experiences may be used erated or enriched through participatory research
at critical points (e.g., when deciding on the con- builds on knowledge that accumulates as people
tent of services to be studied or when interpreting work together to improve understanding of a par-
the findings). ticular situation (Park, 1993).
Senge and Scharmer (2001) express a similar
A Framework for Knowledge view of participatory research, proposing that
it involves “a knowledge-creating system.” We
Generation in Participatory Research
have adapted their ideas to discuss a general model
One challenge of participatory research is how to of participatory research for occupational therapy.
balance traditional concerns for developing rigor- In our model, researchers, practitioners, and con-
Practice Capacity
Innovation Building
Creation of a new vision of Generation of practical

services and supporting know-how for therapists and
resources Researchers, consumers
Practitioners,
and Consumers
working together
Theory &
Research
Discovery and
understanding
Figure 38.1 A knowledge-creating system.
sumers work together as part of what Senge and These three components of knowledge creation
Scharmer (2001) call a “continuing cycle of creat- are collectively addressed within a single commu-
ing theory, tools and practical know-how” (p. 238). nity of people working together. As a result, con-
They further engage in three interacting domains cepts, evidence, and practice innovations are
of activity: created at the same time that practitioners’ knowl-
edge of and use of these resources is increased.
• Discovery and understanding of concepts and
In this knowledge-creating system, practitioners
data whose relevance transcends the particular
and clients are centrally involved along with
situation—that is, knowledge that is generaliz-
researchers in the process of creating knowledge.
able. This knowledge is generally the primary
Conversely researchers join practitioners and con-
objective of the researcher.
sumers in solving practice problems and innovating
• Creation of practical knowledge and capacity-
in practice. Furthermore, in a knowledge-creating
building, which enhances practitioners’ and
system there is no artificial division between creat-
clients’ awareness and capabilities. Practical
ing and assessing knowledge on the one hand and
knowledge emphasizes utilization and usefulness
applying it on the other. Finally, all stakeholders are
of the knowledge generated in addressing social
involved in co-generative learning in which their
issues of importance to consumers. Capacity
initial perspectives are altered or replaced through
building is also critical in this model because it
dialogue and co-discovery (Elden & Levin, 1991).
speaks for one of the essences of participatory
research and that is increasing skills, knowledge,
and capabilities of participants to address their
own concerns. This kind of knowledge is local The Steps of Participatory
and personal since it involves such things as
enhancing the practical know-how of occupa-
Research in Occupational
tional therapy practitioners and empowering Therapy
clients with knowledge of how to manage their
circumstances. Discussions of the process of participatory
• Practice innovation, which involves creating new research have been offered in the literature
possibilities and means for achieving them. Such (Balcazar, Keys, & Suarez-Balcazar, 2001; Fawcett
knowledge can involve rethinking practice aims et al., 2003; Selener, 1997; Suarez-Balcazar &
or methods. It involves creating or improving Harper, 2003; Taylor et al., 2004). Based on these
practical tools and approaches that work out in a ideas, we propose here a framework for the process
particular situation but that are irrelevant to other of participatory research in occupational therapy.
similar situations. Practice innovation often aims According to this framework, participatory
at creating tools that not only work in the situa- research involves six steps as depicted in Figure
tion at hand, but that can also be used in other 38.2. These steps, discussed next, are part of an
practice situations. ongoing cycle of discovery, change-making, and
evaluation of change. Although the research ordi- In participatory research, researchers and stake-
narily begins with the first step, it can begin at any holders engage in ongoing reflection aimed at
stage and may involve implementing elements of achieving a critical awareness of the problem.
different stages simultaneously. Furthermore, it is Without critical reflection, research can move for-
important to recognize that before participatory ward to solve a problem before it is fully defined
research begins and throughout its implementa- and its multiple dimensions are fully understood.
tion, there is a process of entry and building trust Participatory research seeks to confront and
wherein the researcher establishes and maintains a address the full complexity of any problem rather
true partnership with the stakeholders. than trying to isolate and study only one aspect. In
this way, participatory research is more likely to
Phase 1: Delineating the Problem Through generate solutions that actually work in the real-
Critical Reflection and Analysis life situation. Often, given the cyclical nature of
participatory research, the problem may be rede-
In traditional research, problems to be investigated fined as the research unfolds.
in the research are most often identified from care-
ful study of the literature. In participatory research,
delineating the problem to be studied also involves Phase 2: Analysis and Planning
careful attention to how stakeholders define the of the Participatory Research
problem locally and to the larger context that may
Careful and critical analysis of the problem sets
be influencing the problem. Moreover, the problem
a foundation for the next step of the process, plan-
to be addressed will emerge out of the practice
ning the research. This step involves selecting
context. In this phase it is important to ask the fol-
the research questions or hypotheses that will be
lowing questions:
answered or tested in the research. Because of the
• How does the literature shed light on the prob- previous step, the research questions or hypotheses
lem? will not be grounded only in preexisting empiri-
• How do different stakeholders view the problem? cal and theoretical knowledge. They will also
• What are contextual and environmental factors reflect key stakeholders’ perspectives on what
affecting the problem? they need to understand or know. In this phase, it
• When all these dimensions are considered is important to consider the following kinds of
together how do they frame the problem? questions:
Delineating
the
Problem
Reflection and Analysis and

Utilization Planning
Implementing Developing
Action Service Strategies
Choosing the
research design,
data collection,
and analysis
methods
Figure 38.2 The steps of participatory research.

• Given the dimensions of the problem, what do engage in choosing the research design and the
we need to know to understand or address the approach to data collection and analysis. Once
problem? again, these methods must fit the interests of stake-
• What kinds of information do we want to gener- holders and available resources (Fawcett et al,
ate about the problem and how the solutions 2003; Park, 1999). In participatory research, inves-
work? tigators and stakeholders often participate equally
• How would the information that is generated be in data collection, data analysis, and in decisions
used? as to how the success of a given strategy will be
judged. By shaping such data analysis, stakehold-
Since stakeholders are involved along with
ers have an important role in determining what
researchers, the questions generated will have not
conclusions will be drawn from the research. In
only theoretical but also utilitarian ends.
this phase, it is important to consider the following
Information is sought not only to create under-
questions:
standing of what is being studied but also to
achieve workable solutions to local problems. • Are the design, the methods for gathering data,
and the kind of data collected appropriate for
evaluating the strategies?
Phase 3: Developing Service Strategies
• Are the design, the methods for gathering data,
The third phase involves an analysis of potential and the kind of data collected appropriate to the
service strategies that can be implemented to context and the concerns of stakeholders?
understand and address the problem and its dimen- • Will the design, data collection, and analysis pro-
sions. In occupational therapy these strategies can vide information that is both credible in the sci-
include such diverse things as modifying or gener- entific community and relevant to making future
ating a new assessment or intervention, developing decisions in the research setting?
a program to meet an
unmet need, promoting This phase requires
organizational change Information is sought not achieving an important
that affects healthcare only to create understanding balance between scien-
delivery for a client tific and practical con-
group, or engaging in
of what is being studied but cerns. The design and
advocacy efforts to also to achieve workable data will not only address
change insurance legis- concerns of a larger
lation to affect a con-
solutions to local problems. scholarly community but
sumer group. Depending also provide information
on the context, choosing or implementing a solu- about whether a local problem has been effectively
tion may require systematic deliberations, complex addressed.
negotiations, and even struggles between different
perspectives or with existing power structures. Phase 5: Implementing Action
In this phase it is important to consider the fol-
lowing questions: A key element of participatory research is that it
involves action. Participatory research in occupa-
• What are potential strategies to address the prob-
tional therapy will involve implementing the
lem?
strategies that have been designed to address the
• Are the strategies appropriate given the situation
problem. As noted earlier, these strategies can
and perspectives of the stakeholders involved?
range from fairly minor modifications of services
• What are the anticipated likely consequences of
to new programs of services to large-scale changes
applying the proposed strategies or ways of
in service delivery. Whatever the action being
addressing the problem?
implemented, a key feature is that this action is
In answering these questions, the researchers undertaken in a reflective way. This means that the
and stakeholders work together to integrate exist- following kinds of questions are typically asked:
ing scientific knowledge with local knowledge.
• Are the service strategies working as anticipated?
Phase 4: Choosing the Research Design • What unexpected problems or barriers have
emerged?
and Data Collection and Analysis Methods • Have efforts to implement the innovations in
After strategies for addressing the problem have service been successful? Have they created new
been chosen, the researcher and stakeholders complications?
Depending on the design of the research, elements (Balcazar et al., 2001; Prilleltensky &
answers to these types of questions may be used to Nelson, 2002; Riger, 2001; Suarez-Balcazar et al.,
modify the services on an ongoing basis or they 2004):
will be used to provide critical information about
• Working together with others who share different
how the service strategies work.
views,
• Taking on nontraditional roles,
Phase 6: Reflection and Utilization
• “Thinking outside the box” of traditional,
In this phase, researchers and stakeholders reflect discipline-defined research,
on the strategies implemented in the research and • Balancing power, control and resources, and
the results they have achieved as indicated by the • Breaking from traditional researcher–participant
data collected and by their experiences in the relationships.
process. During this phase, questions such as the
For this reason, participatory research can be, at
following are asked.
its extremes, both frustrating and exhilarating. It
• What do the findings mean?
requires a serious commitment to the underlying
• What are their implications for future action in
vision of why participatory research is important
this setting?
for the field, and careful attention to the guidelines
• How can we use the results to improve services,
and procedures we have outlined in this chapter.
programs, and practice in general?
Chapters 39 and 40 discuss the challenges of par-
• How can the knowledge be disseminated to oth-
ticipatory research further and provide strategies
ers?
and examples of how they can be effectively
• What new questions or problems have emerged
addressed.
from taking action?
By addressing such questions, the research team
is accountable for ensuring that the research find- Conclusion
ings can be used locally and broadly with the pro-
fession to improve occupational therapy practice. This chapter overviewed the rationale and need for
participatory research in the context of occupa-
tional therapy and discussed its underlying episte-
Implementing the Six Phases mological assumptions. It also suggested ways to
think about the extent of participation by stake-
of Participatory Research holders, it offered principles, and it offered a
model to guide participatory research. In addition,
As illustrated in Figure 38.2, the phases of partici- this chapter noted some of the challenges involved
patory research are not linear. In fact, they are in participatory research.
interactive and mutually informing. Often, stages Participatory research protects the voice of con-
will be combined and implemented simultane- sumers and clients and provides a venue for
ously or iterated in small cycles. For example, it is researchers and practitioners to work together and
common that phase 5 and phase 6 are interwoven. advance both scholarship and practice. Moreover,
When this is the case, the research involves a its emphasis on innovation and reflection makes it
reflection–action–reflection cycle (Freire, 1970, particularly suited to achieve creative new
1993) in which ongoing understanding of new approaches to service. Consequently, participatory
problems and action to address those problems research has a unique role in advancing occupa-
continues as knowledge is generated in the tional therapy practice.
research process (Park, 1999).
REFERENCES
Balcazar, F. E., Keys, C. B., Kaplan, D. L., & Suarez-
Challenges of Conducting Balcazar, Y. (1998). Participatory action research and
people with disabilities: Principles and challenges.
Participatory Research Canadian Journal of Rehabilitation, 12, 105–112.
Balcazar, F. E., Keys, C. B., & Suarez-Balcazar, Y. (2001).
Empowering Latinos with disabilities to address issues
Like all forms of research, participatory research of independent living and disability rights: A capacity-
has its own unique challenges. Two of the most building approach. Journal of Prevention and
Intervention in the Community, 21(2), 53–70.
challenging aspects of participatory research are Balcazar, F. E., Garate-Serafini, T. J., & Keys, C. B.
creating a dialogue and sharing power. Participa- (2004). The need for action when conducting interven-
tory research inevitably involves the following tion research: The multiple roles of community psy-
chologists. American Journal of Community ships in disability and rehabilitation research.

Psychology, 33(3–4), 243–252. Washington, DC: U.S. Department of Education,
Balcazar, F. E., Taylor, R. R., Kielhofner, G. W., Tamley, National Institute on Disability and Rehabilitation
K., Benziger, T., Carlin, N., & Johnson, S. (2004). Research.
Participatory action research: General principles and a Hahn, H. (1990). The politics of physical difference:
study with a chronic health condition. In L.A. Jason, Disability and discrimination. In M. Nagler (Ed.),
C.B. Keys, et al. (Eds.), Participatory community Perspectives on disability (pp. 118–123). Palo Alto,
research: Theories and methods in action (pp. 17–35). CA: Health Markets Research.
Washington, DC: American Psychological Association. Hammel, J., Finlayson, M., & Lastowski, S. (2003). Using
Boyce, W., & Lysack, C. (2000). Community participation: participatory action research to create a shared assistive
Uncovering its meanings in CBR. In M. Thomas, & technology alternative financing outcomes database and
M.J. Thomas, Selected readings in community based to effect social action systems change. Journal of
rehabilitation: CBR in transition (Series 1). Bangalore: Disability Policy Studies, 14(2), 109–118.
Asia Pacific Disability Rehabilitation Journal. Higgs, J., & Titchen, A. (2001). Rethinking the practice-
Bradbury, H., & Reason, P. (2001). Conclusion: knowledge interface in an uncertain world: A model for
Broadening the bandwidth of validity: Issues and practice development. British Journal of Occupational
choice-points for improving the quality of action Therapy, 64, 526–533.
research. In P. Reason & Bradbury, H. (Eds.). Katz, I., Hass, L., & Bailey, J. (1988). Attributional
Handbook of action research: Participative inquiry and ambivalence and behavior toward people with disabili-
practice. London: SAGE Publications. ties. In H. Yuker (Ed.), Attitudes toward persons with
Charlton, J. I. (1998). Nothing about us without us. Los disabilities (pp. 47-57). New York: Springer.
Angeles: University of California Press. Kielhofner, G. (2004). Conceptual foundations of occupa-
Cockburn, L., & Trentham, B. (2002). Participatory action tional therapy (3rd ed.), Philadelphia: F. A. Davis.
research: Integrating community occupational therapy Kielhofner, G. (2005a). Scholarship and practice: Bridging
practice and research. Canadian Journal of Occupa- the divide. American Journal of Occupational Therapy,
tional Therapy, 69, 20–30. 59, 231–239.
Creek J., & Ilott I. (2002) Scoping study of occupational Kielhofner, G. (2005b). Rethinking disability and what to
therapy research and development activity in do about it: Disability studies and their implications for
Scotland, Northern Ireland and Wales. Executive occupational therapy. American Journal of Occupa-
Summary, College of Occupational Therapists, tional Therapy, 59, 487–496.
London. Law, M. (Ed). (1998). Client-centered occupational ther-
Danley, K., & Langer Ellison, M. (1999). A handbook for apy. Thorofare, NJ:Slack.
participatory action researchers. Boston: Center for Letts, L. (2003). Occupational therapy and participatory
Psychiatric Rehabilitation, Boston University. research: A partnership worth pursuing. American
David, J., Zakus, L., & Lysack, C. L. (1998). Revisiting Journal of Occupational Therapy, 57, 77–87.
community participation. Health Policy and Planning, Litvak, S., Frieden, L., Dresden, C., Doe, T. (1997).
13(1), 1–12. Empowerment, independent living research and partici-
Dubouloz, C., Egan, M., Vallerand, J., & VonZweck, patory action research. In B. Phillips-Tewey, (Ed.),
C. (1999). Occupational therapists’ perceptions of Building participatory action research partnerships in
evidence based practice. American Journal of disability and rehabilitation research. Washington, DC:
Occupational Therapy, 53, 445–453. U.S. Department of Education, National Institute on
Dysart, A. M., & Tomlin, G. S. (2002). Factors related to Disability and Rehabilitation Research.
evidence-based practice among US occupational ther- Maxwell, N. (1992). What kind of inquiry can best help us
apy clinicians. American Journal of Occupational create a good world? Science, Technology, & Human
Therapy, 56(3), 275–284. Values, 17, 205–227.
Elden, M., & Levin, M. (1991). Cogenerative learning: McCluskey, A. (2003). Occupational therapists report a low
Bringing participation into action research. In W. F. level of knowledge, skill and involvement in evidence-
Whyte (Ed), Participatory action research. Newbury based practice. Australian Occupational Therapy
Park, CA: SAGE Publications. Journal, 50(1), 3–12.
Fawcett, S., Boothroyd, R., Schultz, J., Franciasco, V. T., McCluskey, A., & Cusick, A. (2002). Strategies for intro-
Carson, V., & Bremby, R. (2003). Building capacity for ducing evidence-based practice and changing clinical
participatory evaluation within community initiatives. behavior: A manager’s toolbox. Australian Occupa-
Journal of Prevention and Intervention in the tional Therapy Journal, 49(2), 63–70.
Community, 26, 21–36. Metcalfe, C., Lewin, R., Wisher, S., Perry, S., Bannigan,
Fawcett, S. B., Seekins, T., Whang-Ramos, P., Muiu, C. & K., & Moffett, J. K. (2001). Barriers to implementing
Suarez-Balcazar, Y. (1987). Involving consumers in the evidence base in four NHS therapies: Dietitians,
decision-making. Social Policy, 13(6), 36–41. occupational therapists, physiotherapists, speech and
Fine, M., & Asch, A. (1990). Disability beyond stigma: language therapists. Physiotherapy, 87, Part 8,
Social interaction, discrimination, and activism. In M. 433–441.
Nagler (Ed.), Perspectives on disability (pp. 61–74). Meyerson, L. (1990). The social psychology of physical
Palo Alto, CA: Health Markets Research. disability: 1948–1988. In M. Nagler (Ed.), Perspectives
Freire, P. (1970). Pedagogy of the oppressed. New York: on disability (pp. 13–23). Palo Alto, CA: Health
Continuum.. Markets Research.
Freire, P. (1993). Pedagogy of the oppressed. New York: Nyden, P., Figert, A., Shibley, M., & Burrows, D. (1997).
Herder and Herder. Building community: Social science in action. Thousand
Gordon, W. A. (1997). PAR: A realistic strategy for med- Oaks: Pine Forge.
ical rehabilitation Research? In B. Phillips-Tewey, Oliver, M. (1990). The politics of disablement. London:
(Ed.). Building participatory action research partner- Macmillian.
Park, P. (1993). What is participatory research? A theoreti- Olson, B., Alvarez, J., et al. (2004). University-
cal and methodological perspective. In P. Park, M. community partnerships: A framework & case study.
Brydon-Miller, B. Hall, & T. Jackson (Eds.) Voices of In L. Jason, C. Keys, et al. (Eds.), Participatory com-
change: Participatory research in the United States and munity research: Theory and methods in action (pp.
Canada (pp. 1–19). Westport, CT: Bergin & Garvey. 105–120). Washington, DC: American Psychological
Park, P. (1999). People, knowledge, and change in partici- Association.
patory research. Management Learning, 30, 141–157. Suarez-Balcazar, Y., & Harper, G. (Eds.). (2003).
Peloquin, S. M., & Abreu, B. C. (1996). The academia and Participatory and empowerment evaluation: Multiple
clinical worlds: Shall we make meaningful connec- benefits. New York: The Haworth Press.
tions? American Journal of Occupational Therapy, Suarez-Balcazar, Y., Harper, G., & Lewis, R. (2005). An
50(7), 588–591. interactive and contextual model of Community-
Prilleltensky, I., & Nelson, G. (2002). Doing psychology University partnerships. Health Education & Behavior,
critically: Making a difference in diverse settings. 32(1), 84–101.
Basingstroke, UK: Palgrave, MacMillan. Sudsawad, P. (2003). Rehabilitation practitioners’ perspec-
Reason, P. & Bradbury, H. (2001). Handbook of action tives on research utilization for evidence-based prac-
research: Participative inquiry and practice. London: tice. Paper presented at the American Congress of
SAGE Publications. Rehabilitation Medicine conference, October 24,
Redick, A. G., McClain, L., & Brown, C. (1999). Tucson, Arizona.
Consumer empowerment through occupational therapy: Taylor, R. R., Braveman, B., & Hammel, J. (2004).
The Americans with Disabilities Act Title III. The Developing and Evaluating Community Services
American Journal of Occupational Therapy, 54, through Participatory Action Research: Two Case
207–213. Examples. American Journal of Occupational Therapy,
Riger, S. (2001). Working together: Challenges in collabo- 58, 73–82.
rative research. In M. Sullivan & J. G. Kelly (Eds.), Thompson, N. (2001). Theory and practice in human serv-
Collaborative research: University and community ices. Maidenhead: Open University Press.
partnerships (pp. 25–44). Washington, DC: APHA. Townsend, E., Birch, D. E., Langley, J., & Langille, L.
Schon, D. A. (1983). The reflective practitioner. New York, (2000). Participatory research in a mental health club-
Basic Books. house. Occupational Therapy Journal of Research,
Selener, D. (1997). Participatory action research and 20,18–44.
social change. Ithaca, NY: Cornell Participatory Action Turnbull, A. P., & Friesen, B. J. (1997). Forging collabora-
Research Network, Cornell University. tive partnerships with families in the study of disability.
Senge, P., & Scharmer, O. (2001). Community action In B. Phillips-Tewey (Ed.), Building participatory
research: Learning as a community of practi- action research partnerships in disability and rehabili-
tioners, consultants, and researchers. In P. Reason tation research. Washington, D. C.: U.S. Department of
& Bradbury, H. (Eds.), Handbook of action research: Education, National Institute on Disability and
Participative inquiry and practice. London: SAGE Rehabilitation Research.
Publications. Whyte, W. F. (1991). Participatory action research.
Suarez-Balcazar,Y., Davis, M., Ferrari, J., Nyden, P., Newbury Park, CA: SAGE Publications.
C H A P T E R 3 9
Building Culturally Competent

Community–University
Partnerships for Occupational
Therapy Scholarship
Yolanda Suarez-Balcazar • Jaime Phillip Muñoz • Gail Fisher
Occupational therapy researchers increasingly rec- erly, at-risk youth, individuals with disabilities)
ognize the need for and benefits of building part- (Suarez-Balcazar, Harper, & Lewis, 2005b).
nerships with community settings to support
scholarship (Braveman, Helfrich, & Fisher 2001;
Cockburn & Trentharn, 2002; Suarez-Balcazar,
Hammel, Helfrich, Thomas, Head-Ball, & Wilson,
Community–University
2005a). These multipurpose partnerships often Partnerships
combine research, student training and develop-
ment, and implementation and evaluation of serv- Community–university partnerships are mutually
ice delivery (Braveman et al., 2001). This chapter beneficial collaborative relationships that bring
examines factors that con- individuals from acade-
tribute to effective part- mia and the community
nerships with community Community–university together to work on com-
settings. It offers key prin- mon goals. Moreover,
ciples and a framework of partnerships are mutually participants typically
cultural competence to beneficial collaborative rela- share the risks, responsi-
guide community–univer- bilities, resources, and
sity partnerships. tionships that bring individu- rewards of their collabo-
Factors such as the als from academia and the rative efforts (Suarez-
mental health movement, Balcazar et al., 2004).
deinstitutionalization, and
community together to work Community–university
consumer self-help initia- on common goals. partnerships are critical
tives, combined with de- for research that aims to:
clining healthcare access,
governmental services, and family support, have • Apply theoretical and research-related constructs
contributed to the growth of community agencies. in a real-life setting,
There are many types of community settings; • Study varied populations and cultural contexts,
examples include: and
• Impact the health and life satisfaction of disen-
• Grass-roots advocacy organizations,
franchised populations.
• Shelters,
• Jails,
Benefits to the community partner can include:
• Social service agencies,
• Community mental health programs, and
• Access to the skills of a researcher for identify-
• Public health clinics.
ing and addressing unmet needs,
These types of agencies provide a wide range • Assistance with grant writing and accreditation,
of services, often to marginalized populations and
(e.g., low-income families, people with AIDS, • Support for program development and eval-
refugees, survivors of domestic violence, the eld- uation.
Chapter 39 Building Culturally Competent Community–University Partnerships 633
Table 39.1 Benefits of Community–University Partnerships

Benefits to the Community Benefits to the University
• Capacity building • Opportunities to advance scholarship
• Improving the success of programs • Access to research participants and sites
• Documentation of impact to inform decision • Stronger grant applications
makers and funders • Staff service as advisory board members for research
• New perspectives for needs assessment and proposals and studies
strategic planning • Capacity building to be more culturally competent
• Assistance in planning, implementing, and • Opportunity to participate in a social change process
evaluating programs and services
• Opportunities to engage in learning by doing
• Assistance in understanding problems and
• Possibility to conduct applied and participatory action
collecting data
research
• Access to researchers with the skills to examine
• Opportunity to gain knowledge about the culture of the
complex interventions
community including social norms and community
• Assistance in providing best services to clients values
• Shared ownership over products and materials • Entrée into a network of community health coalitions
developed with legitimate power and leadership
• Adoption of innovations that are developed • Collaboration with practitioners
• Practice and services based on evidence • Publications based on applied research
Thus community–university alliances can bring • Address complex social problems,

benefits to all those involved. Table 39.1 provides • Achieve mutuality,
a summary of such benefits. • Create learning communities,
• Use a participatory approach to research, and
• Advance scholarship and practice together.
Key Principles of University–
Community Partnerships Commitment to Address Complex
Social Problems
Previous literature has identified characteristics Society today faces difficult problems such as
and models of successful community–university domestic and school violence, AIDS, hunger,
partnerships (Braveman et al., 2001; Harper homelessness, unemployment, rising rates of
& Salina, 2000; Suarez-Balcazar et al., 2004; incarceration, and addiction. Solutions to these
Suarez-Balcazar et al., 2005a,b) and articulated problems are likely to be as complex and interde-
methodological strategies for Participatory Action pendent as the factors that give rise to them. Most
Research with community agencies (Balcazar,
Keys, Kaplan, & Suarez-Balcazar, 1998; Taylor, Table 39.2 Framework of Key Principles
Braveman, & Hammel, 2004). Building on this for Successful Partnerships
work, we argue that the key principles of commu-
nity–university partnerships belong to two overar- Shared Commitments Joint Responsibility
ching characteristics of successful partnerships:
• To address complex • For communication
• Shared commitment, and social problems • For recognizing,
• Joint responsibility. • To mutuality utilizing, and
• To create learning exchanging resources
Table 39.2 illustrates these key principles
communities • For addressing
which are discussed. challenges and
• To participatory
approach to research assimilating change
Shared Commitment • To advance the • For sustainability
Successful community–university partnerships Scholarship of Practice
occur when both parties share commitment to:
efforts to address these issues occur at the commu- Commitment to Create

nity level. Learning Communities
Occupational therapists in community-based
settings are beginning to demonstrate how com- Investigators should come to a partnership ready to
plex societal problems can be addressed from an learn, as well as to guide. It is useful for all to
occupational perspective (Daunhauer & Jacobs, maintain a constant exchange of perspectives and
2003; Fazio, 2001; Muñoz, Reichenbach, & flexible teacher–learner roles. Collaborative part-
Witchger Hansen, 2005). Occupational therapy nerships for occupational therapy research recog-
concepts are being used to address human prob- nize that expert knowledge resides not only within
lems that have their origins in complex social con- the profession but also in the community.
ditions (Abelenda, Kielhofner, Suarez-Balcazar, & Moreover, an interactive building of knowledge is
Kielhofner, 2005; Kielhofner, 2002; Townsend, important.
1997). Thus, occupational therapists are increas- Researchers must move from seeing a commu-
ingly able to bring their disciplinary perspectives nity as a research site to seeing a community as a
to bear on research that addresses social problems. partner. This means appreciating that the commu-
nity is a repository of rich cultural and practical
Commitment to Mutuality knowledge. For example, community partners are
typically aware of important political and organi-
Mutuality is facilitated by: zational relationships in a community. Further,
• Making time to get to know a setting and its dif- they usually hold informed perspectives on the
ferent stakeholders, needs and strengths of the community, the prevail-
• Building a common vision, ing issues, and the key players. Community part-
• Mutually setting expectations/ground rules, ners often have legitimate power and authority in
• Establishing common goals, and the community that is grounded in a sustained
• Sharing frameworks and ways of thinking about commitment to addressing the needs of the com-
issues of importance to all involved. munity as grassroots organizers or front-line serv-
ice providers. Researchers who at best have
In research involving community–university limited experience and exposure as intermittent
partnerships, a common agenda and clarification participant-observers in the community do well to
of shared values needs to be pursued from the very rely on the information, expertise, and authority of
beginning. Mutuality requires the development of their community partners.
reciprocal trust and respect for each other’s differ-
ences (Foster-Fishman, Berkowitz, Lounsbury, Commitment to a Participatory
Jacobson & Allen, 2001; Mattessich & Monsey,
Approach to Research
1992).
Decisions must be made collaboratively. Participatory research emphasizes the inclusion of
Partnerships call for flexibility when working with community partners in the earliest stages of defin-
multiple layers of decision-makers (Mattessich & ing research questions, setting research priorities,
Monsey, 1992). Moreover, community organiza- and designing research studies (Suarez-Balcazar et
tions typically have a al., 2004; Taylor et al.,
number of stakeholders 2004). In fact, in some
who must be involved in Researchers must move cases it is the commu-
decision-making. These nity who calls upon the
stakeholders range from
from seeing a community as researcher to be part of
executive directors and a research site to seeing a the partnership team. In
staff members, to com- participatory research,
munity members and
community as a research investigations are guided
leaders. Moreover, in partner. by the needs of the com-
many agencies, staff munity rather than the
turnover and work over- intellectual interests of
load may mean that one’s primary partner(s) the researcher (Braveman et al., 2001; Suarez-
change throughout the partnership (Suarez- Balcazar, 2005).
Balcazar, Orellana-Damacela, Portillo, Sharma, & Successful community–university partnerships
Lanum, 2003). Therefore, mutuality is served well recognize that social issues originate in the com-
by establishing relationships with a range of community and are better understood, analyzed, and
munity members and leaders (Braveman et al., solved when members of the community are inti-
2001). mately involved (Balcazar et al., 1998; Balcazar,
Keys, & Suarez-Balcazar, 2001; Selener, 1997). • What is communicated,

Successful partnerships also require that re- • How it is communicated, and
searchers and community partners engage in joint • The communication style and language used.
reflection and analysis. Finally, successful partner-
Communication is facilitated when goals and
ships occur when those involved use findings to
expected roles and outcomes are discussed and
support genuine change that addresses local
agreed upon and when hidden agendas or power
problems.
plays are absent (Connors & Seifer, 2000; Panet-
Commitment to Advance Raymond, 1992). Of course, goals and expecta-
tions may change throughout the process, but they
a Scholarship of Practice
need to be openly discussed and negotiated.
Community–university partnerships represent an Sensitivity to the communication mode and style
opportunity for researchers to embrace a schol- of the setting is critical. Modes refer to such com-
arship of engagement (Boyer, 1996) that both munication channels as formal memos, regularly
serves and develops a deeper understanding of the scheduled meetings, e-mail messages, phone calls,
community (Jackson & Reddick, 1999; Sanstad, and one-on-one visits (Suarez-Balcazar et al.,
Stall, Goldstein, Everett, & Brousseau, 1999). 2004). Community settings can vary in their pref-
Within occupational therapy, engaged scholar- erences or modes, and different modes can also
ship has been referred to as a Scholarship of symbolize such things as power, trust, and respect.
Practice. This perspective argues for occupational Often researchers have to adjust to the communi-
therapy research that addresses and solves prac- cation style of the community setting.
tice problems in real-life contexts (Braveman For a relationship to be successful, it is essential
et al., 2001; Crist & Kielhofner, 2005; Kielhofner, to establish open and frequent communication by
2005). The Scholarship of Practice framework providing updates, discussing issues openly, and
also emphasizes using occupational therapy sharing all information with one another as well as
theories to enhance service delivery and using the broader community membership (Mattessich
the real life challenges and circumstances of serv- & Monsey, 1992). Partners must establish regular
ice delivery to further develop theoretical con- ways of communicating in order to discuss how the
structs. partnership is unfolding, to quickly identify any
When scholars are immersed in the lived expe- issues, and to strategize how to address them.
rience of community members, they tend to
develop a greater sense of social responsibility that
leads to participating in advocacy and social Community–University Partnership Example
change efforts. In truly collaborative commu- of Shared Commitment
nity–university research partnerships, researchers
work together with community leaders, service Faculty at Duquesne University built a partner-
providers, and consumers to create knowledge not ship with a community nonprofit organization
only for the profession but also for solving local that has been dedicated to providing shelter and
problems. supportive services to homeless men and women
for over two decades (Muñoz et al., 2005). This
community–university partnership grew from a
Joint Responsibility shared vision between the executive director of
Successful community–university partnerships are the agency and an occupational therapy faculty
strong alliances where not only mutual benefit but who envisioned a strong, reciprocal partnership
also mutual responsibility occurs. Partners must that would address the comprehensive health and
wellness needs of the consumers at agency with
assume equal responsibility for:
holistic, innovative programming. In addition
• Developing adequate communication, to the Executive and Program Directors, faculty
• Recognizing, utilizing, and exchanging resources, built relationships with others whose support has
• Addressing challenges to assimilating change, ensured the success of the partnership. These
and include administrators of the homelessness and
• Ensuring sustainability of any change that is hunger programs, directors of other programs
serving homeless and incarcerated populations
implemented.
who might utilize services at the agency, and
funders who were supporting programming at the
Responsibility for Communication
shelter. Projects are chosen in collaboration with
Establishing a good communication system is at the agency staff and all partners share a commit-
the heart of authentic and successful partnerships. ment to scholarship and practice.
A communication system is concerned with:
Researchers will be most successful at develop- Moreover, resources might be differently per-
ing and sustaining an authentic relationship when ceived and available in academia and the commu-
using a nonhierarchical communication style nity. For instance, not all members of a community
(Suarez-Balcazar et al., 2005a; Harper et al., organization may have their own computers or
2004). This means avoiding using academic lan- access to the Internet. For many people with dis-
guage and jargon to impress the community. If abilities, access to accommodations such as trans-
community members feel they are being treated in portation, personal assistance, interpreting, and
condescending and paternalistic ways, their trust is captioning are critical to their active participation
impeded and they are more likely to develop in research. Communities with limited resources
resentment toward the researcher or the study may perceive research as a low priority when other
(Gills, Butler, Rose, & Bivens, 2001). Effective more fundamental needs are not being met. Many
communication is grounded in mutual respect for organizations in urban areas are struggling to
diverse viewpoints and ideas regardless of one’s attract resources and are cautious about how many
education or title. resources (e.g., time and effort) they spend on
research. Investigators must make a direct link
Responsibility for Recognizing, between their research and meeting needs of the
community and be sensitive to the resource flow
Utilizing, and Exchanging Resources
that any investigation creates. In this regard, assist-
Each of the partners brings into the relationship ing agency partners in grant writing activities
resources and strengths that need to be recognized or bringing grant-related or university-based
and valued (Connors & Seifer, 2000; Mattessich & resources to a community agency can be very
Monsey, 1992). Most typically, investigators bring helpful.
access to resources (e.g., grant funding), knowl- Time frames might also be different. Re-
edge of research design and methods, theoretical searchers often operate within longer frameworks
knowledge, and fresh perspectives. Community (such as a 3- to 5-year grant period), while com-
partners bring knowledge of the specific area or munity agencies often have briefer temporal hori-
population of interest, legitimate power and lead- zons and operate around events shaped by a variety
ership within the community, political savvy, expe- of factors such as cultural holidays, fund raising
riential information of the issues involved, and cycles, budget, or academic calendars. Unique fea-
awareness of the cultural and contextual character- tures of sites may also influence issues of timing.
istics of the setting and community (Braveman et For example, sites serving individuals without fam-
al., 2001; Suarez-Balcazar et al., 2005b). ily connections may be able to provide more access
Figure 39.1 University researchers and members of a community organization meet to

discuss the “next steps” related to findings from a previous project, resolve any con-
cerns related to the current research project, and identify areas for future collaboration.
to potential research subjects over the holidays should also be prepared to volunteer time and tal-
because they are providing expanded program- ent to the agency (e.g., serving on the board of
ming. Consequently, community–university part- directors or providing technical and professional
ners need to talk about time frames and negotiate advice when needed).
when certain activities and products are expected.
Responsibility for Addressing

Challenges and Assimilating Change Developing Cultural
An omnipresent feature in community–university Competence for University–
partnerships is change and the challenges that
come with it. Changes in personnel, governmental
Community Partnerships
regulations, and funding priorities or levels, are but for Occupational Therapy
Scholarship and Practice
a few of the challenges that either partner may
face. The challenges of partnerships have been dis-
cussed extensively in the literature (see, Connors
& Seifer, 2000; Mattessich & Monsey, 1992; Community–university partnerships for research
Riger, 2001; Suarez-Balcazar et al., 2004). Some almost always involve managing diversity. Thus,
of these challenges include: respecting and celebrating diversity is essential to
the relationship. Importantly, it requires researchers
• Managing conflicts of interest and differences in to building cultural competence (Prilleltensky,
perspectives, 2001; Suarez-Balcazar, Durlak, & Smith, 1994;
• Sustaining activities after termination of funding, Velde, Wittman, & Bamberg, 2003).
• Changing roles and redefining boundaries, There are compelling reasons for occupational
• Developing common ground, and therapy researchers to become culturally compe-
• Managing different external pressures. tent. Recent U.S. census figures suggest that ethnic
Partnerships call for flexibility and tolerance. and racial minorities are becoming numerical
Challenges should be acknowledged up front, majorities in parts of our pluralistic society (U.S.
openly discussed, and addressed. Changes in per- Census Bureau, 2000). Race, ethnicity, and socioe-
sonnel, funding, or timing may force a partnership conomic status are consistent factors in studies of
to change. A key strategy for assimilating such health disparities. A growing body of research is
change is to stay grounded in the central mission demonstrating that levels of health care, healthcare
and vision that drew the partners together initially. outcomes, and general health status are typically
When a clear and unified vision permeates think- poorer for racial and ethnic minorities and eco-
ing and planning, partners can adjust to changes nomically disadvantaged populations (Institute of
without fracturing their mutual commitment. Medicine, 2002; Mayberry et al., 1999). Re-
searchers need to address these entrenched dispar-
Responsibility for Sustainability ities, study disadvantaged populations in their own
environment, and advance the limited body of
Sustainability refers to the capacity for the part- knowledge on the impact of occupational therapy
nership to maintain itself over time. Partnerships with minority populations.
with community settings for research are not built According to Papadopoulos and Lees (2002),
overnight. They take time, a positive attitude from the development of culturally competent resear-
all those involved, and a strong commitment to chers is important because:
make it work (Suarez-Balcazar et al., 2004). From
the earliest stages of collaboration, partners should • Research and most services tend to come from a
address sustainability with an open and frank unicultural perspective,
discussion. • There is growing recognition from funders, edu-
Research partnerships with a community set- cators and health policy makers alike of the need
ting minimally need to last the duration of an to manage diversity and address inequalities in
ongoing project. Longer-term commitments can be research knowledge and service development to
developed based on common interest and success minority and disadvantaged populations,
in obtaining ongoing funding. Getting to know the • Research with minority populations is often not
program well, its constituents, the setting’s mis- integrated into subsequent research, practice and
sion, goals, constituents, and overall context well mainstream policy, and
is part of the responsibility of the researchers who • Culturally incompetent research may lead to
wish to develop a long-term relationship. They inappropriate policies, services and programs.
Community–University Partnership Example There is no universally accepted definition of

of Joint Responsibility cultural competence (Bonder, Martin, & Miracle,
2002). Several models of cultural competence
A long-term relationship has developed between have been proposed in the psychological (Cross,
the University of Illinois at Chicago and an Bazron, Dennis & Issacs, 1989; Sue, Arredondo, &
agency in Chicago that provides employment, McDavis, 1992), nursing (Campinha-Bacote,
life enrichment, advocacy, and residential serv- 2001; Leininger, 2000; Purnell & Paulanka, 2002),
ices to primarily Hispanic adults with intellec- and health professions literature (Bonder et al.,
tual disabilities. Taking joint responsibility for 2002). Collectively, these models hypothesize that
the ongoing development of this relationship has cultural competence is a multidimensional
provided the agency, clients, faculty members,
process. It involves a tripartite mix of cognitive,
students, and the occupational therapy acade-
mic program with multiple benefits. The relation- behavioral, and affective components. This tripar-
ship began with the agency requesting a clinical tite conceptualization of cultural competence is
service contract, and expanded to include the not new. In cultural psychology, Sue et al., (1992)
agency’s involvement in several research grants, organized their framework of cultural competency
a yearly commitment from the agency to host around the domains of knowledge, skills, and
community practicum students, and collaboration awareness (Arredondo et al., 1996; Sodowsky,
on a number of master’s projects. These projects, Taff, Cutkin, & Wise, 1997). In nursing, Sawyer et
which embody the Scholarship of Practice frame- al., (1995) identified knowledge, sensitivity, and
work, have addressed unmet needs expressed by collaboration as the three key characteristics of
the agency representative to the contact faculty
cultural competence. While these models have
member. For example, two students worked with
a faculty member to develop a new assessment been developed to describe competencies needed
for the agency’s group residences, which has by the practitioner, they are relevant for the
provided the agency with strategies to improve researcher as well.
their homes to promote occupational role devel- In occupational therapy, Muñoz (2002) has pro-
opment within the constraints of limited dollars posed a process-oriented model of culturally
and staff resources. A second project resulted in responsive caring that also defines three distinct
a staff development training model, with the stu- dimensions of cultural competence: knowing,
dents serving as coaches to the staff to develop doing, and becoming. This process-oriented con-
skills areas that the staff member had identified. ceptualization of culturally responsive caring is
Both projects are now being used in research
critical to the success of community–university
projects by the contact faculty member and other
occupational therapy scholars (Taylor, Fisher, & partnerships in occupational therapy and is dis-
Kielhofner, 2005). These new research grants cussed below.
will provide resources to the agency while the
community site provides access to clients for
the studies. Knowing: Generating Cultural Knowledge
Both the faculty member and the community
agency take responsibility to build the partner-
and Building Cultural Awareness for
ship by sharing resources, time, and expertise. Successful Partnerships
They each have become familiar with the culture
Culturally responsive researchers generate cultural
of the other’s environment, and have both learned
about celebrations, traditions, and priorities. The knowledge when they develop a broad knowledge
contact faculty member serves on the agency’s base that helps them understand the worldviews of
strategic planning committee and has assisted people living in diverse communities. Sue and col-
with the development of a proposal for a new leagues (1998) have argued that to be culturally
building. The community agency representative competent, one needs to possess specific informa-
serves on the advisory board for the faculty tion and knowledge about particular social groups
member’s allied health grant, assists with subject with whom one is working. Sawyer and colleagues
recruitment for research studies, and participates (1995) defined cultural knowledge as the under-
as a panelist for graduate courses. Both the con- standing of integrated systems of learned behavior
tact faculty member and the community agency
patterns in a cultural group, including how mem-
representative are committed to the sustainability
of this relationship, letting it evolve over time to bers of a group talk, think, and behave and their
address emerging needs. feelings, attitudes, and values.
Knowing involves:
• Viewing partner communities as inherently mul-

ticultural,
• Cultivating a perspective on culture that encom- • Maintaining an open and welcoming stance,
passes attributes beyond race and ethnicity, and • Reading and respectfully responding to the
• Acknowledging the differences that exist within dynamics of each encounter, and
any community. • Engaging each partner with a purposeful intent to
Generating cultural knowledge entails “reach- identify and honor the cultural lifeways of the
ing and obtaining a sound educational foundation community.
about the worldviews of different cultures”
Every partnership is a cross-cultural encounter.
(Campinha-Bacote, 2001, p. 256). Sue and Zane
The researcher approaches the partnership with a
(1987) have described this type of knowledge as
knowledge base and social norms that reflect the
cultural literacy. Cultural literacy is reflected in the
academic culture while the community partners
way that a researcher studies the culture of a com-
are seeped in a culture and knowledge base of their
munity. Culturally responsive researchers:
own. These cultural crossings sometimes compli-
• Generate culture-specific knowledge about the ment and sometimes clash. Culturally responsive
people and social groups with whom they partner, researchers often serve as cultural brokers between
• Acknowledge within-group variations in the the community partner and the wider university
community, and community.
• Make conscious efforts to determine whether Culturally responsive researchers must apply
culture-specific information is applicable to com- advocacy skills. Being an advocate or collaborat-
munity partners. ing with community partners on self-advocacy rec-
This approach to reasoning has been described ognizes the influence that socioeconomic and
by Sue (1998) as dynamic sizing. environmental conditions have on health status and
Culturally responsive researchers also build the way that political, legal, and language barriers
cultural awareness of themselves as multicultural compound these situations. Rorie, Paine, and
individuals. Awareness is enhanced by exploring Barger (1996) suggested that skills in advocacy
their cultural heritage (Tervalon & Murray-Garcia, involve moving beyond recognition that racism
1998). This self-knowledge is essential for and ethnocentrism influence health care to doing
researchers to recognize how their own cultural something to address this situation.
lenses may influence their analyses of data. Active Researchers also apply cultural skills when
reflection directed at understanding self and others they design data collection methods that respect
as cultural beings is a key aspect of building cul- the community partner’s cultural practices and per-
tural awareness. Researchers can also build cul- spectives. These skills can entail the intentional
tural awareness through a conscious process of consideration of culture when designing evalua-
examining their own biases and their capacities tion tools or intervention procedures.
and limitations for research with culturally diverse To develop the skills of a culturally responsive
populations. This willingness to examine one’s researcher one must engage culturally diverse oth-
own prejudices and to remain open to the experi- ers (Campinha-Bacote, 2001). A researcher can
ence of diversity and multiculturalism has been develop skills through direct encounters with cul-
described by Tervalon and Murray-Garcia (1998) turally diverse populations. Such encounters pro-
as cultural humility. vide a context for researchers to test the depths and
recognize the limits of their cultural knowledge
Doing: Applying Cultural Skills and and to practice skills for connecting and respond-
Engaging Culturally Diverse Others ing. Encounters are not always harmonious and
friction is often rooted in a clash of cultural values.
in Partnerships for Research
However, cultural missteps provide opportunities
Doing is conceptualized as a behavioral compo- to generate cultural knowledge and build skills.
nent of culturally responsive research. It refers to This type of learning is supported by reflective
demonstrable skills in communication and rela- processing of cultural encounters.
tionship building, advocacy, data gathering, data Spence (2001) has argued that cross-cultural
analysis, and collaborative dissemination of results encounters “have meaning prior to, in the moment,
and products. One of the primary ways that cul- and on reflection” (p. 102). Lockhart and Resnick
tural responsiveness skills are manifest is in the (1997) posit that the process of cultural compe-
interpersonal interactions that researchers employ tence begins with self-reflection. In partnerships
to forge interpersonal connections with commu- the researcher will engage in several cultural
nity partners. Researchers can establish a sense of exchanges that are likely to help shape his knowing
mutuality with community partners by: and doing in becoming more culturally competent.
Culturally responsive research requires the alism. It manifests in a desire and obligation to
integration of knowing and doing. A researcher learn and experience cultural diversity while being
must not only possess cultural knowledge and an humbled by the vastness of human multicultural-
intentional respect for varied cultural perspectives, ism (Tervalon & Murray-Garcia, 1998). Framing
but she must also develop skills and use them culturally responsiveness as exploration under-
effectively in cross cultural encounters (Brach & scores the idea that cultural competency is a
Fraser, 2000). process not a state (Campinha-Bacote, 2001). It
requires an openness to cultural understanding
Becoming: Exploring Cultural Competence where one is informed but not-knowing (Laird,
in the Context of Partnerships 1998). Not-knowing is manifest in an interper-
sonal approach that includes:
Becoming is a reflective process whereby
researchers demonstrate an intentional drive to • Active listening,
continually broaden their cultural understanding. • Intentional respect, and
Exploring multiculturalism is a purposeful act • Questioning processes that conscientiously rec-
grounded in a commitment to address culture in ognize persons as experts of their own experience
the research context. Padilla & Brown (1999) have (Anderson & Goolishian, 1992). Table 39.3 lists
characterized the process of becoming culturally strategies that facilitate developing cultural com-
competent as a journey and lifelong process. petence in the context of community–university
Becoming culturally responsive is an intentional partnerships.
endeavor. Campinha-Bacote (2001) uses the term
cultural desire to convey this sense of drive and
commitment to exploring multiculturalism. The Conclusions
term exploring is used intentionally to capture a
sense of discovery and intense reflexivity. Community–university partnerships are increas-
Becoming is a commitment toward multicultur- ingly important to scholarship. Through mutual
Table 39.3 Strategies for Developing Cultural Competence for Partnerships

Knowing Doing Becoming
• Be aware of one’s own cultural • Identify educational and training oppor- • Enact a personal commit-
heritage, cultural values, tunities that support the development ment to social justice and
attitudes, ethnic experiences, of culturally responsive practice. multiculturalism.
and biases that can influence • Seek opportunities for encounters with • Practice acceptance of
partnership processes. culturally diverse groups outside ambiguity.
• Develop understanding of health professional settings. • Remain flexible to accept
care disparities and factors that • Use a variety of interpersonal differences and change.
prevent effective care of strategies that recognize and respect • Recognize the limits of your
marginalized groups. cultural differences in communication cultural competencies and
• Become aware of the effect of styles. expertise.
racism, discrimination, and • Develop data collection methods that • Develop a high level of
oppression on the lived reflect preferred interaction styles of comfort with a broad array of
experience of marginalized the cultural group. cultural difference (race,
groups.
• Adapt services, research endeavors, ethnicity, religion, lifestyle
• Learn the culture of the and practice to meet the needs of preferences, etc.).
community, its population, and diverse populations as defined from • Be vigilant toward the
resources. within those communities. dynamics that result from
• Appreciate a variety of cultures • Challenge bias, stereotypes, and cultural differences.
and subcultures (e.g., disability discriminatory practices. • Maintain a “not-knowing”
culture).
• Participate in the traditions of the posture that intentionally
• Learn the culture of the agency agency. respects the beliefs, values,
(e.g., traditions, ways of and knowledge base of the
• Listen to the experiences of others and
communicating). community.
share experiences.
• Appreciate differences and recognize
similarities.
collaborative relationships, individuals from aca- Crist, P., & Kielhofner, G. (2005). The Scholarship of
demia and the community come together to work Practice: Academic and practice collaborations for
promoting occupational therapy. Binghamton, NY:
on a common goals. In these partnerships, knowl- Hayworth Press.
edge and practice innovations are produced in such Cross, T., Bazron, B. J., Dennis, K. W., & Issacs, M. R.
a way that both occupational therapy theory and (1989). Towards a culturally competent system of care.
occupational therapy practice benefit. Vol. I: Monograph on effective services for minority
children who are severely emotionally disturbed.
Community–university partnerships require
Washington: Georgetown University Child
researchers to let go of traditional power relation- Development Center, CASSP Technical Assistance
ships and to share commitment and responsibility. Center.
Researchers must work to achieve trust, mutual Daunhauer, L., & Jacobs, K. (2003). Occupational ther-
respect, and open communication, combined with apy’s role in preventing youth violence. Retrieved on
February 4, 2004, from
clear expectations and shared goals. Finally, inves- http://www.aota.org/members/area7/links/link18.asp?
tigators who want to engage in such partnerships PLACE⫽/members/area7/links/link18.as
must value participatory research and commit to an Fazio, L. S. (2001). Programming to support meaningful
ongoing process of becoming culturally competent. occupation and balance for the disenfranchised and
homeless. In L. S. Fazio (Ed.), Developing occupation
centered programs for the community: A workbook for
REFERENCES students and professionals (pp. 256–274). Upper
Abelenda, J., Kielhofner, G., Suarez-Balcazar, Y., & Saddle River, NJ: Prentice Hall.
Kielhofner, K. (2005). The Model of Human Foster-Fishman, P., Berkowitz, S., Lounsbury, D.,
Occupation as a Conceptual Tool for Understanding Jacobson, S., & Allen, N. (2001). Building collabora-
and Addressing Occupational Apartheid. In F. tive capacity in community coalitions: A review and
Kronenberg, S. Simo Algado, & N. Pollard (Eds.), integrative framework. American Journal of
Occupational therapy without borders (pp. 183–196). Community Psychology, 29, 241–261.
Edinburgh: Churchill Livingstone. Gills, D. C., Butler, M., Rose, A., & Bivens, S. (2001).
Anderson, H., & Goolishian, H. (1992). The client is the Collaborative research and action in communities:
expert: A not-knowing approach to therapy. In S. Partnership building in the Chicago empowerment
McNamee & K. Gergen, (Eds.), Therapy as social con- zone. In M. Sullivan & J. G. Kelly (Eds.),
struction (pp. 25–39). Newbury Park, CA: SAGE Collaborative research: University and community
Publications. partnership (pp. 25–44). Washington, DC: APHA.
Arredondo, P., Toropek, R., Brown, S. P., Jones, J., Locke, Harper, G. W., Lardon, C., Rappaport, J., Bangi, A. K.,
D. C., Sanchez, J., & Stadler, H. (1996). Contreras, R., & Pedraza, A. (2004). Community narra-
Operationalization of the multicultural counseling com- tives: The use of narrative ethnography in participatory
petencies. Journal of Multicultural Counseling and community research. In L. Jason, C. Keys, Y. Suarez-
Development, 24, 42–78. Balcazar, R. R. Taylor, M. Davis, J. Durlak, & D.
Balcazar, F., Keys, C., Kaplan, M. A., & Suarez-Balcazar, Isenberg (Eds.), Participatory community research: the-
Y. (1998). Participatory action research and people with ories and methods in action (pp. 199–217).
disabilities: Principles and challenges. Canadian Washington, DC: American Psychological Association.
Journal of Rehabilitation, 12, 105–112. Harper, G. W., & Salina, D. (2000). Building collaborative
Balcazar, F., Keys, C., & Suarez-Balcazar, Y. (2001). partnerships to improve community-based HIV preven-
Empowering Latinos with disabilities to address issues tion research: The university-CBO collaborative
of independent living and disability rights: A capacity- (UCCP) model. Journal of Prevention and Intervention
building approach. Journal of Prevention and in the Community, 19, 1–20.
Intervention in the Community, 21, 53–70. Institute of Medicine (2002). Unequal treatment: Con-
Bonder, B. R., Martin, L., & Miracle, A. (2002). Cultural fronting racial and ethnic disparities in health care.
threads in clinical contexts. Thorofare, NJ: Slack. Retrieved on April 1, 2004, from http://books.
Boyer, E. L. (1996). The scholarship of engagement. nap.edu/books/030908265X/html/
Journal of Public Service and Outreach, 1(1), 11–20. Jackson, R. S., & Reddick, B. (1999). The African
Brach, C., & Fraser, I. (2000). Can cultural competency American church and university partnerships:
reduce racial and ethnic health disparities? A review Establishing lasting collaborations. Health Education
and conceptual model. Medical Care Research and and Behavior, 26, 663–674.
Review, 57, 181–217. Kielhofner, G. (2002). Model of human occupation (3rd
Braveman, B., Helfrich, C., & Fisher, G. S. (2001). ed.). Philadelphia: Lippincott, Williams & Wilkins.
Developing and maintaining community partnerships Kielhofner, G. (2005). Scholarship and Practice: Bridging
within a ‘scholarship of practice. Occupational the divide. American Journal of Occupational Therapy,
Therapy in Health Care, 15(1/2), 109–125. 59, 231–239
Campinha-Bacote, J. (2001). A model of practice to Laird, J. (1998). Theorizing culture: Narrative ideas and
address cultural competence in rehabilitation nursing. practice principles. In M. McGoldrick (Ed.), Revision-
Rehabilitation Nursing, 26(1), 8–11. ing family therapy: Race, culture, and gender in clini-
Cockburn, L., & Trentharn, B. (2002). Participatory action cal practice (pp. 20–36). New York: Guilford Press.
research: Integrating community occupational therapy Leininger, M. (2000). Transcultural nursing: Concepts,
practice and research. Canadian Journal of theories, research & practices (3rd ed.). New York:
Occupational Therapy, 69(1), 20–30. McGraw-Hill.
Connors, K., & Seifer, S. D. (2000). Partnership perspec- Lockhart, J. S., & Resnick, L. K. (1997). Teaching cultural
tives, 1(2). San Francisco, CA: Community–Campus competence: The value of experiential learning and
Partnerships for Health. community resources. Nurse Educator, 22(3), 27–31.
Mattessich, P., & Monsey, B. (1992). Collaboration: What University–community partnerships: A framework and
makes it work. St. Paul, MN: Amherst Wilder an exemplar. In L. A. Jason, C. B. Keys, Y. Suarez-
Foundation. Balcazar, R. R. Taylor, & M. I. Davis (Eds.),
Mayberry, R. M., Mili, F., Vaid, L., Samandi, E., Ofili, Participatory community research (pp. 105–120).
M. S., McNeil, M. S., Griffith, P. A., & LaBrie, G. Washington, DC: American Psychological Association.
(1999). Racial and ethnic differences in access to med- Suarez-Balcazar, Y., Durlak, J. A., & Smith, C. (1994).
ical care: A synthesis of the literature. Menlo Park, Multicultural training practices in community psychol-
CA: The Henry J. Kaiser Family Foundation. ogy programs. American Journal of Community
Muñoz, J. (2002). Culturally responsive caring in occupa- Psychology, 22, 785–798.
tional therapy: A grounded theory. Unpublished doc- Suarez-Balcazar, Y., Hammel, J., Helfrich, C., Thomas, J.,
toral dissertation, University of Pittsburgh, Pittsburgh. Head-Ball, D., & Wilson, T. (2005a). A model of uni-
Muñoz, J. P., Reichenbach, D., & Witchger Hansen, A. versity–community partnerships for occupational ther-
(2005) Project Employ: Engineering hope and breaking apy scholarship and practice. Occupational Therapy
down barriers to homelessness. WORK - A Journal of and Health Care, 19(1/2), 47–70.
Prevention, Assessment and Rehabilitation, 25(3), Suarez-Balcazar, Y., Harper, G., & Lewis, R. (2005b). An
241–252. interactive and contextual model of Community–uni-
Padilla, R. & Brown, K. (1999). Culture and patient educa- versity partnerships. Health Education & Behavior, 32,
tion: Challenges and opportunities. Physical Therapy 84–101.
Education, 13(3), 23–33. Suarez-Balcazar, Y., Orellana-Damacela, L., Portillo, N.,
Papadopoulos, I., & Lees, S. (2002). Developing culturally Sharma, A., & Lanum, M. (2003). Implementing an
competent researchers. Journal of Advanced Nursing, outcomes model in the participatory evaluation of com-
37(3), 258–264. munity initiatives. Journal of Prevention and
Panet-Raymond, J. (1992). Partnership: Myth or reality? Intervention in the Community, 26(2), 5–25.
Community Development Journal, 27, 156–165. Sue, D. W., Arredondo, P., & McDavis, R. (1992).
Prilleltensky, I. (2001). Value-based praxis in community Multicultural counseling competencies and standards:
psychology: Moving toward social justice and social A call to the profession. Journal of Counseling and
action. American Journal of Community Psychology, Development, 70, 477–486.
29, 747–778. Sue, D. W, Carter, R. T., Casas, J. M., Fouad, N. A., Ivey,
Purnell, L. D., & Paulanka, B. J. (2002). Transcultural A. E., Jensen, M., La Fromboise, T., Manese, J. E.,
health care: A culturally competent approach. Ponterotto, J. G., & Vasquez-Nuttal, E. (1998).
Philadelphia: F. A. Davis. Multicultural counseling competencies: Individual and
Riger, S. (2001). Working together: challenges in col- organizational development. Thousand Oaks: SAGE
laborative research. In M. Sullivan & J.G. Kelly Publications.
(Eds.), Collaborative research: University and Sue, S. (1998). In search of cultural competence in psy-
community partnership (pp. 25–44). Washington, chotherapy and counseling. American Psychologist,
DC: APHA. 53(4), 440–448.
Rorie, J. L., Paine, L. L., & Barger, M. K. (1996). Primary Sue, S., & Zane, N. (1987). The role of culture and cultural
care for women: cultural competence in primary care techniques in psychotherapy: A critique and reformula-
services. Journal of Nurse-Midwifery, 41, 92–100. tion. American Psychologist, 42, 37–45.
Sanstad, K. H., Stall, R., Goldstein, E., Everett, W., & Taylor, R. R., Braveman, B., & Hammel. J. (2004).
Brousseau, R. (1999). Collaborative community Developing and evaluating community-based services
research consortium: A model for HIV prevention. through participatory action research: Two case exam-
Health Education and Behavior, 26(2), 171–184. ples. American Journal of Occupational Therapy,
Sawyer L., Regev, H., Protor, S., Nelson, M., Messias, D., 58(1), 73–82.
Barnes, D., & Meleis, A. I. (1995). Matching versus Taylor, R. R., Fisher, G., & Kielhofner, G. (2005).
cultural competence in research: Methodological con- Synthesizing research, education, and practice accord-
siderations. Research in Nursing and Health, 18, ing to the Scholarship of Practice model: Two faculty
556–567. examples. Occupational Therapy and Health Care, 19,
Selener, D. (1997). Participatory action research and 107–122.
social change. Ithaca, NY: The Cornell Participatory Tervalon, M., & Murray-Garcia, J. (1998). Cultural humil-
Action Research Network. ity versus cultural competence: A critical distinction in
Sodowsky, G. R., Taff, R. C., Cutkin, T. B., & Wise, S. L. defining physician training outcomes in multicultural
(1997). Development of multicultural counseling inven- education. Journal of Health Care for the Poor and
tory: A self-report measure of multicultural competen- Underserved, 9(2), 117–25.
cies. Journal of Counseling Psychology, 41(2), Townsend, E. (1997). Enabling occupation: A Canadian
137–148. perspective. Ottawa, ON: CAOT Publications ACE.
Spence, D. G. (2001). Prejudice, paradox, and possibility: U.S. Census Bureau (2000, January 13). Projections of the
Nursing people from cultures other than one’s own. resident population by race, Hispanic origin, and
Journal of Transcultural Nursing, 12(2), 100-106. nativity: Middle Series, 2025–2045. Washington, DC:
Suarez-Balcazar, Y. (2005). Empowerment and participa- Author. Retrieved April 20, 2004, from Population
tory evaluation of a community health Intervention: Projections Program, Population Division database
Implications for occupational therapy. Occupational on the World Wide Web: http://www.census.gov/
Therapy Journal of Research: Occupation, population/projections/nation/summary/np-t5-c.pdf
Participation, and Health, 25(4), 133–142. Velde, B., Wittman, P., & Bamberg, R. (2003). Cultural
Suarez- Balcazar, Y., Davis, M. I., Ferrari, J., Nyden, P., competence of faculty and students in a school of allied
Olson, B., Alvarez, J., Molloy, P., & Toro, P. (2004). health. Journal of Allied Health, 32(3), 189–195.
C H A P T E R 4 0
Participatory Research for the

Development and Evaluation
of Occupational Therapy:
Researcher–Practitioner Collaboration
Gary Kielhofner • Lisa Castle • Claire-Jehanne Dubouloz
Mary Egan • Kirsty Forsyth • Jane Melton • Sue Parkinson • Mick Robson
Lynn Summerfield-Mann • Renée R. Taylor • Suzie Willis
This chapter discusses researcher–practitioner col- • Feel that research evidence does not fit with their
laboration in the context of participatory research. perception of the practical situation or a particu-
The authors include both practitioners and lar client’s needs (Dubouloz et al., 1999; Dysart
researchers who have been participated in investi- & Tomlin, 2002).
gations that used a participatory approach. Thus,
As a result, it is not uncommon that experienced
this chapter reflects not only the literature, but also
occupational therapists use a wide range of tech-
first-hand experiences in implementing participa-
niques that “appear to work rather than appraising
tory research. The chapter covers:
the research evidence” (Creek, 2003, p. 27).
• The importance of participatory research for The disconnection between research and prac-
occupational therapy, tice exists partly because generating knowledge
• Principles that should underlie participatory and using knowledge are considered distinct and
research involving practitioners, and separate enterprises (Barnett, 1997; Higgs &
• Guidelines for meeting the challenges of imple- Titchen, 2001; Schon, 1983). Researchers based in
menting participatory research. academic settings conduct most occupational ther-
apy research. Moreover, since academics generally
Finally, the chapter ends with three examples have the most advanced training for research, it is
that illustrate participatory research projects in generally accepted that conducting investigation is
which therapists and researchers collaborated to their responsibility and prerogative.
develop and investigate practice. In their quest for rigor, university-based investi-
gators can be prone to overlook the circumstances
of practice for which they aim to generate evi-
The Importance of dence (Peloquin, 2002; Peloquin & Abreu, 1996).
Participatory Research Research is infrequently grounded in or guided by
practice. Even when occupational therapists are
involved in research, these practitioners serve
The majority of occupational therapists agree that
mainly as consultants, advisors, service providers,
it is a good thing to base practice on research
or data collectors. Characteristically, the researcher
(Dysart & Tomlin, 2002). However, practitioners
ultimately controls the process of investigation.
sometimes:
These circumstances are beginning to change.
• Question the relevance of the questions addressed There is an increasing recognition that the process
and findings generated by research (Dubouloz, of integrating research into practice should begin
Egan, von Zweck, & Vallerand, 1999; Sudsawad, long before the research is completed. Investiga-
2003), tors in occupational therapy are looking for ways
• Express concern that research reflects occupa- to more clearly ground their research in practice
tional therapy conducted under ideal conditions contexts. An emerging form of participatory
or with resources not readily available in practice research in occupational therapy involves investi-
(McCluskey, 2003; McCluskey & Cusick, 2002), gators and practitioners working together as part-
and ners to advance practice knowledge.
Figure 40.1 (A) Kirsty Forsyth, Suzie Willis and Lynn Summerfield
Mann (L-R) discuss plans for academic–practitioner collaboration as
part of the United Kingdom Center for Outcomes Research
(UKCORE) that fosters participatory research in England and
Scotland. (B) Clinicians Mick Robson, Janet Woodhouse, Suzie Willis,
and Kirsty Forsyth (L-R), with Central and North West London Mental
Health NHS Trust, present their experiences with practice innovation
as part of the UKCORE effort.
Principles of Participatory Stringer, 1996; Taylor, Braveman, & Hammel,

2004). The following three principles are funda-
Research for Examining mental to participatory research in occupational
Occupational Therapy therapy when the stakeholders are occupational
therapists who will use the results of the research
to shape their practice.
The idea of researcher–practitioner partnerships is
a natural application of participatory research, as 1. Therapists who ultimately use knowledge should
discussed in Chapter 38. Participatory research is be involved as equal partners in helping to gen-
an empowerment-oriented approach that involves erate and refine that knowledge.
researchers and stakeholders working together to This principle means that practitioners should
shape the research questions and methods and to be involved in research as true collaborators
conduct the research (Bradbury & Reason, 2001; with the investigators, not merely as consultants,
Chapter 40 Participatory Research for Development and Evaluation of Occupational Therapy 645
data collectors, or implementers of intervention. research involves interfacing issues of rigor that
It could be further advocated that this principle emphasize control and order with practice settings
expresses the belief that practitioners can also that are often changing and difficult to control.
become the driving force behind clinical investi- Enhancing both rigor and relevance requires a con-
gations. For example, stant balancing act.
practitioners are in an Second, participatory
excellent position to An emerging form of par- research involves the
identify areas of need ticipatory research in occu- interface of two differ-
for research as well ent perspectives. These
as relevant research pational therapy involves practice and academic
questions. investigators and practition- perspectives are each
In participatory re- characterized by differ-
search, practitioners
ers working together as ent concerns, values,
participate in making partners to advance practice time frames, and proce-
decisions about such knowledge. dures. For researchers
critical research issues and practitioners to
as the questions to be effectively collaborate,
asked and the methods to be used. Experience dialogue, compromise, and goodwill are required.
indicates that, when practitioners shape The following section considers typical chal-
research, it yields resources and evidence that is lenges that emerge in participatory research where
responsive to occupational therapists’ perspec- researchers and practitioners collaborate. It dis-
tives, work styles, and practice realities. cusses how and why these challenges arise.
2. Knowledge development should be grounded in Finally, it examines ways these challenges can be
real-life practice contexts. overcome. Importantly, these challenges can be
surmounted as will be illustrated. Nonetheless,
For findings to be usable in practice, research
participatory research will be more successful
must reflect actual practice. The surest way to
when parties enter into the collaboration aware of
reflect practice is to conduct research in practice
the types of challenges that can arise. The authors
settings. When grounded in practice settings, the
of this chapter have found participatory research to
outcomes of that research are relevant to the
be an exciting and productive form of inquiry as
actual circumstances and demands of practice.
well as one that has its own unique challenges.
3. Knowledge development should be innovative
and reflective.
The Challenge of Knowledge
Investigators and practitioners undertake partic- and Power Differentials
ipatory research to improve upon existing practice.
This means that practitioners and researchers typi- As noted in Chapter 38, participatory research
cally work together to create practice innovations requires sharing power between investigators and
and to examine how these innovations work. A stakeholders. However, this power sharing requires
reflective process in which researchers and practi- careful attention to subtle knowledge and status
tioners share their ongoing insights and perspec- differentials between researchers and therapists.
tives as the research unfolds is important to The scientific knowledge of researchers is typi-
support practice innovation. cally considered a more privileged and prestigious
form of knowledge. This discrepancy is height-
ened when the overall purpose of collaboration is
Meeting the Challenges in to conduct research. Practitioners can feel that they
don’t have skills comparable to the researcher’s,
Participatory Research: leading to an unequal initial footing. Since partici-
Achieving Effective patory research usually involves examination of
some aspect of occupational therapy services,
Researcher–Practitioner practitioners can also perceive that their own prac-
tice is being judged.
Partnerships Challenges can arise around issues of who
should control what aspects of the participatory
Participatory research that examines occupational research process. Researchers have more training
therapy practice can be challenging to undertake and experience in research methods and feel
for two primary reasons. First of all, participatory responsible for the rigor of their investigations.
tiveness of an intervention may advocate a

The following are examples of knowledge and
research design in which not all subjects receive
power issues in participatory research. In one
project involving some of the authors, the overall the intervention. In such a situation, practitioners
aim was to implement and investigate theory- may have misgivings about withholding a form of
based practice. At the beginning of the project, treatment they believe their clients need. In such a
existing practice was characterized by a lack of case both perspectives have merit, although each
clearly articulated theoretical rationale and by the emphasizes a different aspect of the situation (i.e.,
use of nonstandard “home-grown” assessments. creating more certain knowledge that can improve
It was agreed that therapists in the setting would services versus providing a group of clients with
work to articulate and implement a common the- every possible form of service.
ory and begin to use standardized assessments Differences in perspective can also affect the
that would also be used to generate data about
collaborative research process. For example,
intervention outcomes.
As part of the process of examining how the researchers whose job expectations include pub-
theory was being put into practice, the lishing often have higher priorities and tight time-
researchers suggested that practitioners would lines for writing up the results of studies.
present cases so that there could be a public dis- Practitioners who face other kinds of time demands
cussion of how the theory was working in prac- and work expectations may not share this same
tice. This process seemed relatively sense of urgency for publishing.
straightforward and benign to the researchers,
who were accustomed to public discussion and
criticism. After all, at the basis of research is the Challenges Due to Differences
assumption that knowledge is always incomplete in Work Styles and Settings
and tentative and, therefore, must be questioned,
debated, and changed. Researchers typically function in an entrepreneur-
However, the prospect of having their cases ial fashion, pursuing lines of inquiry that they are
discussed and critiqued was very threatening to passionate about, securing funding, and publishing
practitioners who were not accustomed to having in a particular area of expertise. Researchers often
their practice publicly examined. They felt that it have substantial discretion over their everyday use
was important that they had confidence in their of time, but they also must deal with deadlines and
knowledge since it was being applied with real timelines imposed by funding agencies, confer-
consequences. The idea of uncertainty and ever- ences, publishers, and the tenure and promotion
changing knowledge appeared contradictory to timetable.
the daily requirement to make decisions that
Some practitioners have highly structured
impacted clients. In this same study, some practi-
tioners spoke about feeling “brainwashed.” As workdays involving constant demands of client
one practitioner put it, she “did not want to prac- interventions with the ongoing necessity of getting
tice the same way as everyone else” or be “taken documentation and billing done. Others are con-
over” by researchers. Moreover, she saw the stantly responding to crises and must reprioritize
innovations as implying that her previous knowl- their work on a daily basis. When researchers and
edge and practice were inadequate. As can be practitioners collaborate, these differences in work
readily seen, these issues required some direct demands and work styles can result in legitimate
attention and negotiation. disagreements about priorities.
Research is a foreign activity in most practice
contexts. While researchers experience the various
Thus, researchers can be reluctant to relinquish tasks involved in designing and implementing par-
control over decisions about research methodology ticipatory research to be natural, they are not part
that affects scientific rigor. On the other hand, prac- of the practitioner’s accustomed routine. Thus,
titioners may feel that their experience gives them practitioners can experience research as increasing
a right to make the decisions about the practice their workload. Even practitioners who are ini-
issues that are addressed in the research process. tially enthusiastic can grow weary of the extra
work involved in doing research (Egan et al.,
The Challenge of Different 2004).
Many times, even when there are negotiated
Agendas and Priorities practitioner responsibilities for research, the
Practitioners and researchers sometimes have dif- demands of practice simply override research
ferent agendas and expectations for the outcomes needs. Consider, for instance, the practitioner
of their research efforts. For example, researchers faced with the choice between covering clients
who wish to make an objective study of the effec- who are part of a sick or vacationing colleague’s
caseload versus collecting data for research. These • Practice expertise, and
and other circumstances can leave clinicians feel- • Research expertise.
ing overwhelmed by the demands of research tasks
Acknowledging the value of both these forms
and researchers frustrated with the lack of progress
of expertise is essential to a successful participa-
in an investigation.
tory research project. While practitioners and
A final challenge of collaborating across prac-
investigators are likely to differ on the extent of
tice and academic work settings is the practical
expertise they have in these two areas at the outset,
matter of physical distance. Sometimes the chal-
the participatory research process provides oppor-
lenge is that academic and clinical sites are sepa-
tunity for learning from one another. Investigators
rate; other times the challenge is coordinating
become more knowledgeable about practice cir-
practitioners and academics across multiple set-
cumstances and practitioners learn more about the
tings. Thus, the usual challenges of communication
research process. Also, researchers and practition-
are further complicated by the fact that some of this
ers alike often discover that their counterparts have
communication inevitably takes place in non face-
a mixture of both types of expertise. That is,
to-face formats (e.g., Web-based communication).
researchers generate clinical insights while practi-
tioners come up with useful research strategies.
Overcoming the Challenges Clarifying the Purpose of
of Practitioner–Researcher Participatory Research
Collaboration in Participatory As noted in Chapter 38, practitioners and investi-
Research gators involved in participatory research share a
common concern for addressing practice chal-
lenges in a particular situation and for generating
Resolving differences of power, perspectives, and knowledge that can be generated in the field as a
work style are among the major challenges of par- whole. Practitioners who are involved in participa-
ticipatory research. True power sharing in partici- tory research must open up their practice to inspec-
patory research means that all participants have tion. Given this circumstance, it is critical for all to
a degree of responsibility, voice, and decision- acknowledge that the purpose of research is to
making about all aspects of the research. This examine the field’s knowledge, not therapists’ per-
means that both researchers and practitioners must sonal knowledge.
also commit to sharing what they know and to Participatory research that examines practice is
learning what the other knows. Each must respect designed to determine how the field’s prevailing
the expertise and insights of the other. Above all, knowledge works in a particular practice context.
there must be ongoing dialogue to maintain under- Moreover, it provides opportunity to generate new
standing and trust. concepts and practice innovations that can become
True dialogue means more than simply talking part of the field’s knowledge base. While this
together. Researchers and practitioners function in knowledge is expressed in how individual practi-
different worlds with different sets of concerns, tioners go about doing their work, the focus should
and different constituencies to whom they are always remain on the concepts that inform, and the
accountable. Dialogue requires that the parties dis- particular characteristics of, successful therapeutic
cover their different values and perspectives and strategies. For example, discussing how participa-
find common ground between them. Productive tory research will improve practice can trigger
dialogue requires: practitioners to worry that their practice is inade-
• Acknowledging and respecting expertise, quate. Framing such discussions as improving the
• Clarifying the purpose of participatory research, knowledge available to therapists for their practice
• Communicating honestly with sensitivity, can help avoid such misgivings.
• Suspending judgment while striving to under-
stand the perspectives of others, and Communicating Honestly with Sensitivity
• Negotiating and reformulating plans.
Everyone can readily see that dialogue works best
when all parties honestly share perspectives.
Acknowledging and Respecting Expertise However, honest communication can have unin-
In participatory research that examines practice, it tended consequences if not carefully enacted. For
is important to underscore that the research instance, direct statements of perspective have the
requires two types of expertise: potential to make those whose perspectives are dif-
ferent feel undervalued or dismissed. Similarly, laborators actively question and carefully listen in
honest feedback can exacerbate an individual’s or order to generate understanding of another’s
group’s vulnerabilities. knowledge, perspectives, and concerns. As Pelo-
Experience teaches that it is generally easier to quin and Abreu (1996) note, it is important to find
have the honest discussion among one’s research or common ground between the thought processes
practitioner peers than it is to share perspectives or that typically separate scholars and practitioners.
feedback with those from another group. Persons The following is an example of how a group of
from another group are more likely to misunder- practitioners and researchers found such common
stand or fail to appreciate what one is saying. There ground. Some of the authors have been working
is a natural tendency to avoid potentially sensitive for years to create, study, and publish standardized
discussions. However, avoiding such discussions assessments. When they began a project to instate
only ensures that gaps in perspective or beliefs will standardized assessments for documenting occupa-
be perpetuated. Consequently, all parties need to tional therapy treatment outcomes, there were frus-
put effort into recognizing the potential for infor- trated to learn that practitioners were using these
mation to be emotionally charged and to consider standardized assessments in nonstandard ways.
how it is best shared. Consistent, honest sharing of The investigators first decided to remedy the situa-
information with careful consideration to its tion by providing the therapists with more training,
impact on the receiving parties is always helpful. reasoning that these practitioners lacked under-
For example, in one participatory research pro- standing of the importance and use of standardized
ject, the primary aim was to make changes in prac- tools.
tice. While most practitioners in the study setting Unexpectedly, these efforts initially frustrated
had expressed a desire to achieve this change, one practitioners who felt that the needs and demands
working group particularly struggled with practice of practice were being overlooked. Ensuing dis-
innovation. Moreover, initial minor critiques of cussions underscored that the researchers’ con-
their work caused tension among the team mem- cerns about assessment centered on scientific
bers, indicating that their practitioners’ confidence evidence concerning validity and reliability while
was low. the practitioners’ concerns centered on ease of
Consequently, the investigators avoided provid- administration, usefulness for treatment, and
ing feedback that the group was falling behind oth- whether the interdisciplinary team valued the
ers in the setting. Instead, they provided gentle but information they yielded. Moreover, both groups
persistent nurturing and communicated sugges- were convinced that their concerns were the ones
tions for further development in the context of pos- that really mattered.
itive feedback concerning what had already been In ongoing discussions, both constituencies
accomplished. In time, this group did generate considered together how both sets of concerns
successful innovations in their practice. Moreover, might be integrated. This discussion led to a new
they felt empowered and grew in confidence. way of thinking about assessments, upon which
Importantly, these practitioners, who were initially they all agreed. That is, they all wanted a practical
wary of the researchers, came to trust them. way to dependably generate critical information
As noted earlier, participatory research often for understanding clients’ needs that the interdisci-
requires communication to take place through plinary team would understand, respect and sup-
Web-based systems. While virtual communication port. With this new common ground, they were
allows collaboration across wide distances and able work together on research that aimed to
thus makes possible participatory research that improve the assessment process.
would otherwise not be possible, the lack of con- Finding such common ground involves not only
stant face-to-face encounters requires extra atten- educating each other, but also pausing to consider
tion and effort. other ways of thinking about things. The left-hand
side of Table 40.1 illustrates some the assumptions
Suspending Judgment While Striving to that practitioners and researchers sometimes hold
about each other. Through the kind of openness,
Understand the Perspectives of Others
dialogue, and reflection espoused here, common
Because they come from different work cultures, perspectives (shown on the right-hand side) can
practitioners and researchers can sometimes mis- be generated. The unification of practitioner
construe each other’s knowledge, motives, and perspectives and academic perspectives is reflected
behaviors. Productive dialogue always begins with in a term that collaborators have found help-
an openness to leave behind preconceptions or ful, namely, practitioner-scholars. When therapists
judgments about others. It further requires that col- and researchers work together in participatory
research, they all become, in effect, practitioner- compared to a control group. Following this, it was
scholars. planned that the project team would train occupa-
tional therapists and interdisciplinary personnel
Ongoing Negotiation and indigenous to the settings to carry out the inter-
vention.
Reformulation of Plans
This plan was initially agreed on by the facili-
Participatory research, by definition, is not as lin- ties that were part of the study. Nonetheless, early
ear or neatly controlled. Because it occurs in real- in the project, it became apparent that the indige-
life settings and is designed to be responsive to the nous staff in the facilities wanted to be part of the
realities of those settings, participatory research program and help in its delivery. Moreover, the
often requires changes in aims, plans, and dead- facilities’ managers wanted all services in their
lines. The following is facility to be integrated
an example. as a whole. Thus, the
One of the authors When therapists and original plan to imple-
was involved in a feder- ment the model program
ally funded investigation researchers work together separately from other
of model occupational in participatory research, services and service per-
therapy services pro- sonnel was deemed
vided to clients living in
they all become, in effect, infeasible.
residential facilities. The practitioner-scholars. Since this project was
original plan of the government funded, the
study involved testing an investigators had to dis-
intervention with a three-group design. According cuss the changes with the funding agency and
to this plan, the model intervention was to be secure its approval. Moreover, subject recruitment
implemented in the residential settings by occu- and implementation plans had to be redrawn and
pational therapists who were part of a funded new timelines developed. Although not every par-
research team (the model program group) and ticipatory research project will involve such a
Table 40.1 The Transformation from Conflicting to Collaborative, Practitioner-Scholar,

Perspectives and Attributions
Typical Conflicting Perspectives and New Collaborative
Attributions of Researchers and Practitioners Practitioner–Scholar Perspective
Researchers Practitioners
Research is essential to good Research is often out of touch Scientific knowledge and practical
practice—i.e., it helps therapists with the “real world.” experience can be complementary:
to understand the client and Experience is the best way to • Practitioners’ experience and
know what kinds of intervention figure out what works in a expertise can inform and shape the
to implement to achieve the given context. researcher.
best results.
Client-centered practice and • Research can address practical
Practitioners undervalue theory building rapport with clients issues and shed light on practical
and research and don’t make requires flexibility that can questions.
the time to learn new by stifled be the
approaches that contemporary Practice can be systematic and
“standardization” of evidence-based while also being
evidence suggests would practice suggested by
constitute best practice. client-centered.
theory and research.
Practitioners and researchers can learn
Academics are out of touch and create practice together, share
with the real world of accountability, and work toward
practice and don’t improving and demonstrating the
understand its demands outcomes of practice.
and constraints.
Researchers’ and practitioners’
differences and similarities can be
assimilated into unique shared
dialogue to create better visions for
practice.
major change as altering the basic design of the vocational potential when they came into contact
study, minor changes are par for the course. with the service system. Occupational therapists
Participatory research results in practice inno- within a regional health organization felt that
vations. Changing practice patterns means that they had the expertise to assume this role and suc-
therapists must justify what they are doing to the cessfully negotiated with senior management to
multidisciplinary team and convince them to take on vocational evaluation. Therefore, they
accept the changes. Sometimes managers have to needed a validated assessment appropriate to
be convinced to allow therapists to change inter- capture the clients’ motivation and potential for
vention protocols or to take time to develop new work. Researchers recommended the Worker Role
skills. Often, resistance by these constituencies Interview (WRI) (Velozo, Kielhofner, & Fisher,
can cause delays, necessitate changes in plans, or 1998).
require extended negotiations and compromise. When practitioners tested the WRI, which was
For example, in one setting, where some of the originally designed for persons with acute physical
authors are involved in ongoing research, a team of impairments, they discovered that it was not well
occupational therapists began to make practice suited to clients with chronic disabilities and neg-
modifications in anticipation of eventually study- ligible work histories. At first the practitioners felt
ing outcomes. These changes involved more that the investigators lacked information about
clearly basing practice on occupational therapy their client group and were simply advancing a
theory and doing client documentation in ways favored instrument. In a meeting between the
that reflected that theory. Initially, the physicians investigators and practitioners, both groups lis-
working on the service demanded that the occupa- tened carefully to the concerns and perspectives of
tional therapist revert back to their old form of each. As a result of this discussion, a decision was
documentation, complaining that they did not made to modify the WRI so that it was more suit-
understand the new approach and language used able to psychiatric clients, and then to test the psy-
by the therapists. A great deal of tact was required chometric properties of this modified assessment.
to educate the physicians as to the reasons for the This was done in two steps.
changes and to achieve their support. The resulting First, the practitioners in the setting worked
reporting style accommodated concerns of physi- with one researcher to develop, pilot, and refine an
cians while preserving what therapists believed alternative interview format suitable to the person
was important to document about their practice. with limited work history. At the same time, other
A comprehensive strategy that seeks to antici- researchers collaborated with the practitioners to
pate problems and reactions of others, and to revise the rating scale of the WRI so that it was rel-
inform and educate those impacted by innovation evant to both acutely and chronically impaired
is often helpful. Nonetheless, comprehensive plan- clients. The result was a new and more flexible
ning and action can never avoid the emergence of version of this standardized assessment (Braveman
unforeseen problems. When these occur, flexibility et al., 2003). The feature box that follows provides
is required. the perspectives of a practitioner and an investiga-
tor involved in this project.
Summary A second innovation emerged from practition-
ers’ observations. Implementing the WRI with
The previous sections discussed some of the chal-
clients took away from time that could be used to
lenges that can arise in participatory research. It
explore other aspects of a client’s occupational
also highlighted strategies that can effectively
functioning, which were important. Therapists in
manage these challenges. What follows are three
this setting previously used the Occupational
cases that illustrate the principles and processes
Circumstances Assessment-Interview and Rating
discussed in this chapter.
Scale (OCAIRS) (Forsyth et al., 2004). The
OCAIRS is a generic assessment of occupational
Case 1: Engaging in performance and participation. Doing both the
OCAIRS and the WRI simply took too much time.
Participatory Research to Discussion between researchers and therapists
resulted in the following innovation. Since the
Modify an Assessment Process content of the OCAIRS and the WRI somewhat
overlap, it was decided to develop a single inter-
During a participatory research project involving view format that combined the content of both the
some of the authors, new legislation mandated that WRI and the OCAIRS. After doing the combined
clients with mental illness receive an evaluation of interview, therapists would complete the rating and
Engaging in Participatory Research to Modify an Assessment Process
The Practitioner’s Account issue in need of further thought and consideration

by the team. In turn, I was encouraged to consider
In 2002, I was invited to collaborate on the rede- why I was seeing things a certain way and chal-
velopment of the Worker Role Interview to make it lenge my own assumptions and prejudices. As a
more suitable for people with long-standing illness personal benefit, this allowed me to reflect on my
or disability. The original version was designed for professional experience and relate my “gut feel-
people with an acute injury. Most of my clients ings” to a theoretical framework. This process has
have not worked for many years and have long helped me to validate my experience but also to
term illness, so the suggested questions and rating extend my thinking and, I hope, understand my
scale just didn’t work in my clinical practice. clients better.
I have plenty of experience of adapting assess- Once we had agreed on the revised assess-
ments to suit my purposes, but this has always ment—following a period of testing by therapists
been done for pragmatic reasons without specific in London and Chicago—the final stage of col-
reference to any theoretical or research consi- laboration involved a wider group of people com-
derations. I like to work in an iterative way with municating by e-mail to agree on the contents of
assessments, slowly getting the feel for what the new manual. This was not simply a matter
information I need and adjusting them accord- of proofreading, because it allowed everyone to
ingly. I anticipated that working with an academic contribute to mini-debates on the final content
researcher was going to bring this way of working of the manual. I think these debates highlighted
to an abrupt halt. Orderliness and precise, clear the importance of being able to challenge each
thinking would be the order of the day. others’ assumptions and to put forward a different
My actual experience was quite different. perspective on the situation. I am sure that they
We worked very much in an equal partnership. helped to improve the final version of the manual
I brought my day to day clinical and practical so that it will satisfy important academic require-
experience to the work with all its inherent ambi- ments without forgetting the more pragmatic con-
guities. My academic colleague brought her aca- cerns of therapists.
demic expertise (and of course her own extensive
clinical experience).
The Researcher’s Account:
We agreed on a timetable for achieving each
stage of the initial development process over a 4- When I first met my practitioner colleague, he was
month period, meeting monthly and communicat- very skeptical of academics. He expected that the
ing in between by e-mail and phone. It seemed, in researchers would simply defend the existing stan-
retrospect, somewhat like taking a course of study dardized assessment in the first meeting. Instead,
with no lessons to attend, but regular assignments the researchers said in effect, “OK, you’re right!
to submit, each one building on what had already Let’s work towards making it better.” That’s when
been achieved. This approach may sound rather I saw his attitude change. His frustration turned
rigid, but I found that it gave me a great opportu- into action. We met and communicated routinely
nity to put my ideas down in writing within a over several months. I discussed the needs of
structure designed to achieve very clear goals. measurement. But most of our discussions were
My academic colleague worked with the material around clinical cases—debating them, dissecting
that I provided, returning it with her ideas and sug- them, and applying the assessment to find out if
gestions so that we worked our way step by step it would embrace the complexity of the clients’ sit-
toward a mutually agreed-upon end product. uations. He took responsibility for the working up
During this process, I had a clear sense that of the document with me and reviewing it rou-
nothing was being taken for granted with regard tinely between meetings. I made sure the measure-
to the eventual outcome, and that my input was as ment aspects of building the new scale would be
crucial to the success of the project as that of any- consistent with measurement principles. He was
body else. I felt able to ask “dumb questions,” very engaged in the process, and he said he was
knowing that I would either get an illuminating surprised by my clinical knowledge.
answer or that the question would highlight an
reporting forms for both the OCAIRS and the studied. At the time of this writing, the two assess-
WRI, thus saving substantial time. ments are routinely used and data are being col-
In both cases, innovations arose from practice lected that will allow evidence to be generated
dilemmas. A local problem was solved, and, at the about the assessments’ reliability and validity as
same time, new resources for other therapists in well as their practical utility in clinical decision-
similar situations were developed and empirically making.
Case 2: An Online dency of the researchers to teach and control and

to remove the temptation for practitioners to seek
Participatory Action quick answers. Not only did the graduate students
Research Project not have the same urge to provide answers, but
they were also seen as credible to the participants
since as recent practitioners they were aware of the
A Canadian-based project aimed to allow practic- clinical reality. In addition, these graduate students
ing occupational therapists to develop strategies were asked not to provide instruction to partici-
for enhancing their use of research findings in pants on evidence-based practice methods.
practice. The main premise of this project was the Once the recruitment of the participants and the
belief that practitioners, through their own experi- process of grouping participants were completed,
ence and knowledge, are skilled at reflecting, ana- the exploration began. Participants were keen to
lyzing, and identifying solutions grounded in start their reflection and to follow the inquiry
real-life practice contexts. During a 13-month process. Each phase of the process was facilitated
Web-mediated action research project (Egan et al., by the graduate student facilitator, keeping in mind
2004), 51 occupational therapists from eight the intention to empower the therapist participants.
Canadian provinces met online, using Stringer’s Practitioners successfully shared on-line reflections
three-step action research approach to problem on their perceptions and experiences. At the end of
resolution. The three steps of this approach are: the first phase (i.e., naming the issue), practitioners
• Naming the problem of research utilization, easily completed a collective account, accepted by
• Looking for an understanding of the use of all members of their group, that described the diffi-
research, and culties of using research in practice.
• Acting on practical strategies to be tested The second phase of the research process,
(Stringer, 1996). understanding the issue, was felt by some practi-
tioners to be a bit more complex, but still manage-
This process of inquiry structured the online able. Although more time was required, the second
communication and research activity. phase was completed. Each group created an inter-
Four virtual groups of approximately 12 occu- pretative account to explain the problems under
pational therapists each were assembled according investigation.
to four clinical interests: adult institution-based At the beginning of the third, action, phase,
care, adult community-based care, adult mental challenges to creating new strategies in the work
health care, and child health. Four academics place were articulated. Practitioners were begin-
coordinated the project. One of the debates among ning to express the need for more direction from
the research team, prior to the initiation of online the researchers as the complexity of the task
exploration, concerned how much group leaders was increasing. It was challenging to empower
should instruct practitioners about the process of practitioners as they started to feel reticent and
evidence-based practice. Academics anticipated somewhat inadequate to the task. They did not
that the practitioners would look to them for how see creating a solution as entirely within their
they should go about finding and synthesizing rights. They did not trust their own judgments and
research evidence. However, in previous research were not sufficiently reassured by the group
projects, instruction to healthcare providers on process.
evidence-based practice methods had met with In each of the two groups that were most suc-
limited success. Thus, the investigators were inter- cessful in identifying new strategies for using
ested to see the practitioners develop innovative research evidence in practice, a group member
solutions and procedures on their own. Thus, hav- emerged to take over leadership from the group
ing academics instruct practitioners on how to go facilitator. In one case, the group became frus-
about evidence-based practice was seen as poten- trated with the software used in the project and
tially stifling such creativity. formed alternate communication channels. While
The solution to this dilemma was twofold. this decreased the researchers’ access to communi-
First, the investigators decided to stay in the back- cation between the group members, it was ulti-
ground during the exploration of research utiliza- mately taken as a sign of the group’s maturity and
tion. Four post-professional graduate students ability to work toward its goals. Successful inter-
were assigned to be at the forefront of communi- nal leadership did not emerge in the two less
cations, each one facilitating one of the four successful groups. For example, in one group, dis-
groups of practitioners. It was expected that this cussions became rather circular without consensus
would be a practical method to prevent the ten- and, thus, the group’s work stalled.
While leadership and a sense of ownership son, Forsyth, & Kielhofner, 2004). This collabora-
were crucial factors in whether groups were able to tion required a constant process of educating each
create solutions to the problem of bringing other about practice or research perspectives and
research findings into practice, other factors were concerns. It also required substantial compromise
also involved. These other factors included lack and negotiation about how to proceed. Importantly,
of confidence, the challenge of using a new tech- neither those with academic roles, nor the practi-
nology (i.e., asynchronous online communica- tioners, ultimately controlled the process. Rather, it
tion), the unexpected amounts of extra time unfolded with a degree of healthy tension and ebb
required for participation, and the real difficulty in and flow of whose agenda prevailed at critical
moving from collecting information to critiquing junctures in decision-making about the developing
and synthesizing it in order to form a plan of instrument. The process of creating the MOHOST
action. was less linear than previous projects in which
Despite the different levels of success across researchers where clearly in charge. However, in
groups, almost all the practitioners reported valu- the end, our power sharing resulted in an assess-
ing the experience of being linked to other occu- ment that satisfied both academic and practical sets
pational therapists working in similar areas and of concerns. Interestingly, the story does not end
struggling to determine the best care for their here. The feature box on the next page illustrates
clients. Approximately half of the original practi- both a practitioner and a researcher perspective on
tioner participants completed the project. They this collaboration.
reported beginning to consistently use evidence- Practitioners in another setting began to use this
based methods in their work, often applying skills instrument with the dual intention of incorporating
learned in an academic setting within their work- it as part of client documentation and collecting
places for the first time. This practitioner–aca- data for a validation study. These practitioners
demic collaboration appeared to create an used an electronic medical record and found it
important bridge for them in applying skills, such cumbersome to go from the paper-and-pencil
as searching the literature and identifying a guid- instrument to creating a report in the electronic
ing theoretical model. This type of collaboration medical record. One practitioner requested to have
appears quite promising, particularly when leader- a copy of the electronic file so he could create his
ship of the group can be shifted to the practitioners own templates for reporting in the electronic med-
who then look to the academics for support rather ical record for all to be able to access. They went
than direction. through many different versions based on the
unit’s need, user friendliness, and also based on
other team members’ feedback—in terms of the
Case 3: Engaging in content and look of the report (too long, too much
Participatory Action info on the page, etc.).
During meetings involving both investigators
Research to Create and practitioners, the idea emerged to develop a
software package that incorporated the assess-
a New Screening Tool ment, allowed it to be scored electronically, and
particularly automated the process of writing a
A group of practitioners in the United Kingdom narrative note. Over time, with joint discussions
who had reviewed a number of standardized between researchers and practitioners, the soft-
assessments decided that none were routinely ware was designed to include treatment goals,
usable in the fast paced and often chaotic context interventions, and documentation of goal attain-
of acute mental health care. Consequently, they ment so that it could serve as a database for
began to develop their own assessment. These research. The software continues to be refined,
practitioners sought out academic involvement adding both more practitioner-friendly features
after they had piloted an initial version of their new and enhancing its value as a database for investi-
assessment. In the spirit of participatory research, gating practice.
the investigators did not take over the development It is important to note that the creation of the
of the assessment, but rather they joined the prac- software came out of a discussion of daily practice
titioners as partners. challenges rather than a long-term vision or plan.
For nearly 5 years so far, researchers and prac- This highlights the importance of how having a
titioners have worked together to refine and study dialogue about our daily practice challenges as
the resulting instrument, the Model of Human part of participatory research can lead to innova-
Occupation Screening Tool (MOHOST) (Parkin- tions that no one could have predicted.
Engaging in Participatory Action Research to Create a New Screening Tool
The Practitioner’s Account In participatory action research all participants

must be involved throughout the process as equal
In 1992, I read Creek’s book, Occupational partners blending the insights of the practitioner
Therapy in Mental Health (1990), and found and the expertise of the researcher. Throughout
myself identifying with her assertion that, “vague the research, the academics involved demonstrated
and inaccurate assessment leads to vague and respect for my experiential learning and this meant
imprecise treatment” (p. 82). I determined that a that I felt encouraged to maintain my own convic-
new therapist-rated outcome measure was needed tions and assert my knowledge of the needs of
for use in acute mental health settings and I took my clients and my colleagues. Their openness
the first steps toward creating an assessment that is allowed us to work together to produce someth-
now known as the Model of Human Occupation ing that would be familiar and relevant to practi-
Screening Tool (MOHOST). The first drafts were tioners, and, as a consequence, the MOHOST
rudimentary, and it was with some trepidation that has led to tangible results that have made a real
I decided to share my early work with university- difference to my practice and to other services.
based researchers. I was concerned that “home-
grown assessments” would be frowned upon, but
The Researcher’s Account
I was also convinced that some locally devised
tools could meet the specific needs of practitioners My role was ensuring that the theory was cor-
better than many generalized tools. To my delight, rectly reflected in the assessment and that we
I discovered that new ideas were warmly wel- were following measurement rules. The most
comed, and that academics in two institutions concerning issue for me about this assessment
wanted to support practitioners by matching the was the clinical request to have multiple data-
art of practice with the science of standardization. gathering methods. I could see that such an
The resulting collaboration to create a new approach would make it more usable in clinical
assessment led to my becoming involved in a par- practice and allow it to be reflective of how pra-
ticipatory research process—that is, research lead- ctitioners normally work. However, from a meas-
ing to concrete solutions for a recognized problem. urement point of view, this flexibility could be a
This kind of collaborative research serious source of error. Consequently, there were
differs from solo work because it is accom- tense times, but our commitment to the field and
plished, not first in one person’s mind, and to producing something useful that would engage
then in the other’s, but on the loom between practitioners took priority over any disagreements.
them, … The richness … lies in effectively We now get a kick out of practitioners saying how
acknowledging, and where appropriate, usable the tool is in practice. It makes all the hard
resolving differences between the partners” work worthwhile.
(Donaldson & Sanderson, 1996, p. 44).
Conclusion the prospect of having their cases publicly dis-

cussed and critiqued. Investigators worked hard to
This chapter examined the rationale and principles assure therapists that the purpose of this discussion
underlying participatory research that involves was not to critique their personal knowledge or
researchers and practitioners together. The chal- expertise and, instead, to think together about how
lenges of doing this type to best do theoretically
of research and strategies driven and evidence-
for managing these chal- When researchers and prac- based practice. When
lenges were discussed. titioners work together in such discussions took
As our three case exam- place, therapists were
ples illustrate, the unique participatory research, they pleasantly surprised to
potential of participatory are transformed and empow- learn that the researchers
research to create and affirmed their own
test practice innovations ered by the research observations and recog-
far outweighs its chal- process. nized their expertise.
lenges. Moreover, there was
The following under- opportunity to creatively
scores this point. Earlier, this chapter illustrated an discuss and arrive at decisions about further inno-
instance in which therapists were threatened by vations. Later in the research process, practitioners
collaborated with researchers to submit abstracts Graham, I. D. (2004). Enhancing research use through
and present at a national conference. Some of the on-line action research. Canadian Journal of
clinician’s cases were later incorporated into pub- Forsyth, K., Deshpande, S., Kielhofner, G., Henriksson, C.,
lished materials. Recently, there was an opportu- Haglund, L., Olson, L., Skinner, S., & Kulkarni, S.
nity in the ongoing research to have another public (2004). User’s manual for the Occupational
discussion of how therapists were using theory and Circumstances Assessment and Interview Rating Scale
(OCAIRS) (version 4.0). Model of Human Occupation
standardized assessments. There were so many
Clearinghouse, Department of Occupational Therapy,
therapists who wanted to present their practice that College of Applied Health Sciences, University of
all of them could not be accommodated! Illinois at Chicago, Chicago, IL.
This example highlights what is, perhaps, one Higgs, J., & Titchen, A. (Eds.) (2001). Practice knowledge
of the most unique things about participatory and expertise in the health professions. London:
Butterworth Heinemann.
research. Collaborators have opportunity not only McCluskey, A. (2003). Occupational therapists report a low
to work together to create and examine practice level of knowledge, skill and involvement in evidence-
innovations, but also to learn and grow profession- based practice. Australian Occupational Therapy
ally. When researchers and practitioners work Journal, 50, 3–12.
McCluskey, A., & Cusick, A. (2002). Strategies for intro-
together in participatory research, they are trans-
ducing evidence-based practice and changing clinical
formed and empowered by the research process. behaviour: A manager’s toolbox. Australian Occupa-
The old discrepancies between practitioners and tional Therapy Journal, 49, 63–70.
scholars are broken down and all involved come to Parkinson, S., Forsyth, K., & Kielhofner, G. (2004). User’s
work together as practitioner-scholars. manual for the Model of Human Occupation Screening
Tool (MOHOST) (version 1.1). London: UK CORE.
Peloquin, S. M. (2002). Confluence: Moving forward with
REFERENCES affective strength. American Journal of Occupational
Barnett, R. (1997). Higher education: A critical business. Therapy, 56(1), 69–77.
Bristol: Open University Press. Peloquin, S. M., & Abreu, B. C. (1996). The academia and
Bradbury, H., & Reason, P. (2001). Conclusion: clinical worlds; Shall we make meaningful connec-
Broadening the bandwidth of validity: Issues and tions? American Journal of Occupational Therapy, 50
choice-points for improving the quality of action (7), 588–591.
research. In P. Reason & Bradbury, H. (Eds.), Schon, D. A. (1983). The reflective practitioner: How pro-
Handbook of action research: Participative inquiry and fessionals think in action. New York: Basic Books.
practice (pp. 447–455). London: SAGE Publications. Stringer, E. T. (1996). Action research. Thousand Oaks,
Braveman, B., Robson, M., Velozo, C., Kielhofner, G., CA: SAGE Publications.
Fisher, G., Forsyth, K., & Kerschbaum, J. (2003). Sudsawad, P. (2003). Rehabilitation practitioners’ perspec-
Worker Role Interview (WRI) (version 10.0). Model of tives on research utilization for evidence-based prac-
Human Occupation Clearinghouse, Department of tice. Paper presented at the American Congress of
Occupational Therapy, College of Applied Health Rehabilitation Medicine conference, October 24,
Sciences, University of Illinois at Chicago, Chicago, Tucson, AZ.
IL. Taylor, R. R., Braveman, B., & Hammel, J. (2004).
Creek, J. (1990) Occupational therapy and mental health: Developing and evaluating community-based services
Principles, skills and practice. Edinburgh: Churchill through participatory action research: Two case exam-
Livingstone. ples American Journal of Occupational Therapy, 58,
Creek, J. (2003). Occupational therapy defined as a com- 73–82.
plex intervention, London, College of Occupational Velozo, C., Kielhofner, G., & Fisher, G. (1998). A user’s
Therapists. guide to the Worker Role Interview (version 9). Depart-
Donaldson, G. A., & Sanderson, D. A. (1996) Working ment of Occupational Therapy, University of Illinois at
together in schools. A guide for educators. Thousand Chicago.
Oaks, CA: Corwin Press.
Dubouloz, C. J., Egan, M., von Zweck, C., & Vallerand,
J. (1999). Occupational therapists’ perceptions of
RECOMMENDED READINGS
evidence-based practice. American Journal of Occupa- Crist, P., & Kielhofner, G. (2005) The Scholarship of
tional Therapy, 53, 445–453. Practice: Academic and practice collaborations for
Dysart, A. M., & Tomlin, G. S. (2002). Factors related to promoting occupational therapy. Binghamton, NY:
evidence-based practice among U.S. Occupational ther- Hayworth Press.
apy clinicians. American Journal of Occupational Harrison, M., & Forsyth, K. (2005) Developing a vision for
Therapy, 56(3), 275–284. therapists working within child and adolescent mental
Egan, M., Dubouloz, C. J., Rappolt, S., Polatajko, H., health services: Poised or paused for action? British
King, J., Vallerand, J., Craik, J., Davis, J. A., & Journal of Occupational Therapy, 68, 1–5.
S E C T I O N 9
Evidence-Based Practice
C H A P T E R 4 1
Definition, Evolution, and

Implementation of Evidence-Based
Practice in Occupational Therapy
Pimjai Sudsawad
Definitions of Evidence- More recently, Sackett, Straus, Richardson,

Rosenberg, and Haynes (2000) described EBP as
Based Practice the integration of best research evidence with clin-
ical expertise and patient values. With this updated
Evidence-based practice (EBP) is a practice version, the patient values are acknowledged as an
approach developed based on the concept of equally important and necessary ingredient in the
evidence-based medicine (EBM). There are many practice of EBP as research evidence and clinical
definitions of EBP by several authors. However, expertise.
the most widely cited definition is that by Sackett, In another widely cited definition, Gray (1997)
Rosenberg, Gray, Haynes, and Richardson (1996) described evidence-based practice as an approach
who defined evidence-based (medicine) practice as to decision-making in which the clinician uses
the conscientious, explicit, and judicious use of the best evidence available in consultation with the
current best evidence in making decisions about patient to decide upon the option that best suits
the care of individual patients. the patient. This definition stresses that the rela-
According to Sackett and colleagues (1996), tionship between clinician and patient is centrally
evidence-based practice is an integration of indi- important in clinical decision-making.
vidual clinical expertise with the best available According to Gray (1997), a clinical decision
external clinical evidence from systematic occurs based on three factors:
research. Clinical expertise refers to the profi-
• Evidence,
ciency and judgment that the individual practition-
• Values, and
ers acquire through experience. Best available
• Resources.
external clinical evidence means client-centered
research that is useful for informing practice. He characterized current healthcare decisions
It is clear from this definition that evidence- that are based principally on values and resources
based practice relies on as “opinion based deci-
practitioners’ clinical sion-making.” Gray fur-
expertise when applying Evidence-based practice ther predicted that, as the
research evidence to pressure on resources
practice. Sackett et al.
relies on practitioners’ clini- increases, decisions will
(1996) stated that neither cal expertise when apply- have to be made explic-
clinical expertise nor the itly and publicly to justify
best available external
ing research evidence to the use of the resources.
evidence alone is enough practice. Therefore, those who
for evidence-based prac- make decisions will need
tice. They believed that to be able to produce
external clinical evidence can inform but can never and describe the evidence that informed each
replace individual clinical expertise, and it is the decision.
clinical expertise that decides whether the external A third definition comes from the Canadian
evidence applies to the individual patient (i.e., Association of Occupational Therapists’ position
whether and how it matches the client’s clinical statement on evidence-based occupational therapy
state, predicaments, and preferences). (Canadian Association of Occupational Therapists,
Chapter 41 Evidence-Based Practice in Occupational Therapy 657
Association of Canadian Occupational Therapy

University Programs, Association of Canadian
Discussions of Evidence-
Occupational Therapy Regulatory Organizations, Based Practice in
& the Presidents’ Advisory Committee, 1999). It
defines evidence-based occupational therapy as the Occupational Therapy
client-centered enablement of occupation, based
on client information and a critical review of Since the introduction of EBP in occupational
relevant research, expert consensus, and experi- therapy, there continues to be discussion about its
ence. This definition of evidence-based occupa- implementation. There is an increasing recognition
tional therapy recognizes the range of sources that the implementation of evidence-based practice
and scope of evidence available to occupational is a complex process that may need some adapta-
therapists (Zimolag, French, & Paterson, 2002) tions to ensure its applicability to occupational
including: therapy.
• Research evidence, To implement EBP in occupational therapy,
• Information provided by the client for determin- the synthesis of the available evidence with
ing occupational priorities and capacities, and clinical expertise and judgment, and knowledge
• The knowledge that occupational therapists have of the values and preferences of the clients is
gained from past experience. viewed as critical (Gates & Atherton, 2001; Lee
& Miller, 2003; Lloyd, King, & Bassett, 2002;
Based upon those definitions, the essence of Rappolt, 2003). Different authors have argued that
EBP may be summarized as follows: the direct adoption of evidence-based medicine
• Evidence-based practice involves more than just (EBM) and its established prescriptive guidelines
the use of research evidence. may not adequately reflect the philosophical
• Clinical expertise is as important to evidence- beliefs and the highly contextualized and dynamic
based practice as research evidence. nature of occupational therapy (Lee & Miller,
• Client input is vital to the decision-making 2003; Tse, Blackwood, & Penman, 2001; Welch,
process in evidence-based practice. 2002).
• Healthcare decisions are also influenced by Occupational therapy authors have also ques-
available resources. tioned the strict use of the Level of Evidence
model (also based on EBM). According to this
model, the strength of research evidence is ranked
Evolution of Evidence-
based upon predetermined criteria related to the
study’s designs and characteristics (with meta-
Based Practice in analyses of randomized controlled trials usually
considered the strongest and best evidence). The
Occupational Therapy rationale for questioning this model is that the
best evidence for each circumstance may differ
Evidence-based practice (EBP) evolved from depending on the type of clinical questions
the principles of evidence-based medicine (EBM), asked, and whether the questions relate to patterns
a concept that origi- and possibility or causal-
nated in the 1980s at ity (Tickle-Degnen &
McMaster University in The need for increased Bedell, 2003; Tse et al.,
Canada (Taylor, 1997). accountability in conjunction 2001).
EBP emerged within Some authors have
healthcare and health with the spending restraints suggested qualitative
education in the 1990s in Healthcare has acceler- research methods as one
(Turner, 2001). The need of the appropriate tools
for increased accounta- ated the interest in the use to identify and address
bility, in conjunction of research evidence as clients’ priorities. They
with the spending res- argued that qualitative
traints in healthcare, has the basis for occupational methods may enable
accelerated interest in therapy practice. occupational therapists
the use of research evi- to explore the complexi-
dence as the basis for ties of clinical practice
occupational therapy practice (Law & Baum, and of living with a disability, thereby informing a
1998). client-centered, evidence-based practice perspec-
International Developments in Evidence-Based Practice
Since the introduction of EBP, there have been (Lieberman & Scheer, 2002). The overarching
several international developments for its support goal of this project is to be part of the international
in the field of occupational therapy. EBP has been effort to promote an outcome-based orientation
the main subject of prestigious lectures by several among occupational therapists (Lieberman &
prominent scholars in different countries such as Scheer, 2002). Several articles based on the
the Casson Memorial Lecture at the annual confer- evidence-based literature reviews have been pub-
ence of the College of Occupational Therapists in lished in AJOT (Baker & Tickle-Degnen, 2001;
the United Kingdom (Eakin, 1997), the Eleanor Ma & Trombly, 2002; Murphy & Tickle-Degnen,
Clarke Slagle Lecture at the annual conference of 2001; Trombly & Ma, 2002). There is also a plan
the American Occupational Therapy Association to disseminate the reviews through submissions
(Holm, 2000), and the Silvia Docker Lecture at to the National Guideline Clearinghouse which is
the annual conference of the Australian Associa- sponsored by the Agency for Healthcare Research
tion of Occupational Therapists (Cusick, 2001). and Quality, the American Medical Association,
The American Journal of Occupational Therapy and the American Association of Health Plans so
(AJOT) established the Evidence-Based Practice that the reviews will be part of the larger resources
Forum as a regular department in 1999 (Tickle- for evidence-based practice contributed by several
Degnen, 1999). Numerous articles have been pub- other practice communities (Lieberman & Scheer,
lished in this department, providing information 2002).
on different aspects of EBP. A new department In addition, AOTA has also made the evidence-
in the journal now highlights clinical scholarship based resource directory and Evidence Briefs
to enhance the links between research and practice Series available to its members through its Web
(Kielhofner, 2005). There is also an increased site. The evidence-based resource directory
drive for manuscripts that report applied research includes links to many Web sites providing infor-
findings (Corcoran, 2003). mation in many areas related to evidence-based
EBP has also been incorporated into educa- practice. The Evidence Briefs Series provides
tional requirements and professional conduct in easy-to-read summaries of articles selected from
occupational therapy. In the United States, educa- scientific literature and indexed by topic area.
tional standards require that occupational therapy Most recently, AOTA, in collaboration with
graduates be able to “provide evidence-based effec- the American Occupational Therapy Foundation
tive therapeutic intervention related to performance (AOTF), received a grant from the Agency for
areas” (Accreditation Council for Occupational Healthcare Research and Quality to host a con-
Therapy Education, 1998, requirement 5.3, Section ference on evidence based practice. The confer-
B). In addition, the occupational therapy code of ence served as an international forum to determine
ethics (American Occupational Therapy Associa- the state-of-the-art evidence-based occupational
tion, 2005), Principle 4 E. states that “occupational therapy practice and support the development of
therapy practitioners shall critically examine avail- knowledge into the field including a set of con-
able evidence so they may perform their duties on sensus guidelines for evaluating and reporting
the basis of current information.” Similarly, the research evidence, as well as a specific series
Code of ethics of the British College of Occupa- of outcomes to be accomplished internationally
tional Therapists Principle 5.4 states “Occupatio- (AOTF, 2003). The conference took place in
nal therapists shall be personally responsible for July of 2004, and the final report of this interna-
actively maintaining and developing their perso- tional conference is available for AOTA members
nal development and professional competence,” through the AOTA Web site and through a pub-
(College of Occupational Therapists, 2005) with lished article in AJOT (Coster, 2005).
a specific requirement that “Occupational therapy A Web-based evidence resource containing
personnel shall be accountable for the quality abstracts of occupational therapy-relevant ran-
of their work and base this on current guidance, domized controlled trials and systematic reviews,
research, reasoning, and the best available evi- OTSeeker, has recently been established in
dence.” Australia (Bennett et al., 2003). The OTSeeker
To address the clinicians’ and researchers’ need database currently contains over 3500 abstracts
for easily available research-grounded evidence (updated 12/20/05) and is expected to increase over
documenting the value of therapeutic interventions, time. The database was developed by a team of
the American Occupational Therapy Association occupational therapists at that University of
(AOTA) initiated a project to develop a series of Queensland and University of Western Sydney
evidence-based literature reviews of occupational with support from OT Australia, the Motor Acci-
therapy’s effectiveness with health conditions dent Authority of New South Wales, and a Center
addressed in AOTA’s practice guidelines series for Evidence-Based Physiotherapy, Australia.
658
tive of occupational therapy (Gates & Atherton, • Ensuring that informed consent was obtained in
2001; Hammell, 2001). conducting a research study (despite the pressure
There are also discussions about ethical issues to create research evidence to justify existing
related to EBP for both occupational therapy prac- practice), and
titioners and researchers. An ethical dilemma can • Being careful of biases in selection of research
occur when the client’s preference is contradicted participants.
by evidence (Roberts & Barber, 2001). Christensen
Hayes (2000) suggested that the best use of
and Lou (2001) suggested ethical considerations
resources for research should be achieved through:
for occupational therapy researchers such as:
• Conducting a limited number of large, strategi-
• Involving consumers in making research-related cally designed, significant research projects
decisions, rather than smaller, less consequential projects,
• Ensuring that there is no conflict of interest in • Replicating existing studies to confirm their find-
conducting a study due to sponsoring, ings, and
• Ensuring that disciplinary loyalty does not intro- • Conveying research findings in ways that practi-
duce potential bias into trials involving interven- tioners are able to understand, critically interpret,
tions associated with a given profession, and apply them to clinical practice.
Figure 41.1 An evidence-based practice poster session is held annually at the University of Illinois at
Chicago. Posters summarize and analyze evidence that addresses practice questions from Chicago
area clinicians.
660 Section 9 Evidence-Based Practice
Implementation Status of tinuing education, and colleagues to make clinical

decisions, over the use of research evidence
Evidence-Based Practice (Bennett et al., 2003; Dubouloz, Egan, Vallerand,
& von Zweck, 1999; Sudsawad, 2004; Sweetland
In general, occupational therapy practitioners have & Craik, 2001).
expressed positive attitudes toward an evidence- Several strategies have been proposed to allevi-
based practice approach (Bennett, Tooth et al., ate barriers and help to facilitate the adoption of
2003; Curtin & Jaramazovic, 2001; Humphris, EBP. One strategy called for occupational therapy
Littlejohn, Victor, O’Halloran, & Peacock, 2000; managers to find ways to provide time and support
Upton & Lewis, 1998). However, the implementa- for EBP in a cost-effective manner (Closs &
tion of EBP in actual practice settings seems to Lewin, 1998), and to create organizational condi-
face several barriers (Closs & Lewin, 1998; tions that promote the use of evidence-based prac-
Bennett, Tooth et al., 2003; Curtin & Jaramazovic, tice (Humphris et al., 2000). Another strategy
2001; Dysart & Tomlin, 2002; Humphris et al., involved creating initiatives to assist the imple-
2000; McCluskey, 2003; Metcalfe et al., 2001; mentation of EBP (Curtin & Jaramazovic, 2001).
Philbert, Snyder, Judd, & Windsor, 2003; There is also a call for occupational therapy pro-
Sudsawad, 2004; Sweetland & Craik, 2001; Upton grams to place more emphasis on reading and
& Lewis, 1998). From the results of these studies, interpreting research and systematic reviews to
occupational therapy practitioners indicated that help therapists overcome the obstacle of not being
they still lacked adequate knowledge and the skills able to interpret research (Gervais, Poirier, Van
necessary for evidence-based practice at present. Iterson, & Egan, 2002). Furthermore, there is a
For example, they lacked: suggestion for occupational therapy researchers to
• Information technology skills, create research evidence that is more usable for
• The ability to undertake computer literature practice (Sudsawad, 2005).
searches, There is a need to investigate the usability and
• Knowledge about electronic databases, the effectiveness of these strategies in addressing
• Sufficient evidence appraisal skills, and the barriers stated above, and it is likely that con-
• Adequate understanding of statistics. current strategies will be necessary to help move
the implementation of evidence-based practice in
Problems with the logistics of EBP implemen- occupational therapy forward. Chapters 43 and 44
tation were also apparent such as: discuss strategies that are likely to support the
• The lack of time to read research and implement change process that is necessary for the implemen-
findings due to workload pressure, tation of evidence-based practice.
• High staff turn over combined with staff short-
ages,
• The lack of organizational support, and
Conclusion
• Difficulty accessing research evidence since the Evidence-based practice is an approach that
literature is not available in one place. requires the integration of several factors in
Finally, the characteristics of research evidence decision-making including research evidence, the
can be another barrier to the implementation of practitioner’s clinical expertise, the client’s values
evidence-based practice. Practitioners find that and preferences, and available resources. From the
research evidence can be difficult to use due to EBP literature in occupational therapy, it seems
such things as: apparent that the use of evidence-based practice
continues to evolve as shown by the several debates
• Conflicting results,
and discussions on different aspects of the concep-
• Methodological problems,
tual foundation of EBP, and the appropriate ways to
• Poor generalizability,
interpret and use EBP in occupational therapy. The
• Implications for practice not being made clear,
adoption of EBP is still limited, possibly due to the
and
numerous barriers as indicated by practitioners, and
• Lack of clinical relevance of findings.
there is a need to find effective strategies to allevi-
These barriers are likely explanations for the ate those barriers in order for the implementation of
slow adoption of EBP in occupational therapy. In EBP to move forward. Subsequent chapters in this
addition, there seems to be evidence that occupa- section provide resources for the practitioner, and
tional therapy practitioners continue to favor the discuss factors that can enhance evidence-based
use of clinical experience, information from con- practice and the usability of research in practice.
REFERENCES Shifting the balance—Evidence-based practice. British

Journal of Occupational Therapy, 60, 290–294.
The Commission on Standards and Ethics of the American Gates, B., & Atherton, H. (2001). The challenge of
Occupational Therapy Association (2005). Occupa- evidence-based practice for learning disabilities.
tional therapy code of ethics. The American Journal of Learning Disability Nursing, 10, 517–522.
Occupational Therapy, 59, 639–642. Gervais, I. S., Poirier, A., Van Iterson, L. & Egan, M.
The American Occupational Therapy Foundation (2003). (2002). Attempting to use a Cochrane review:
AHRQ funds international conference on evidence- Experience of three occupational therapists. The
based occupational therapy. AOTF Connection, 10(2), American Journal of Occupational Therapy, 56,
1, 5. 110–113.
Baker, N. A., & Tickle-Degnen, L. (2001). The effective- Gray, J. A. M. (1997). Evidence based healthcare: How to
ness of physical, psychological, and functional inter- make health policy and management decisions. New
ventions in treating clients with multiple sclerosis: A York: Churchill Livingstone.
meta-analysis. The American Journal of Occupational Hammell, K. W. (2001). Using qualitative research to
Therapy, 55, 324–331. inform the client-centred evidence-based practice of
Bennett, S., Hoffmann, T., McCluskey, A., McKenna, K., occupational therapy. British Journal of Occupational
Strong, J., & Tooth, L. (2003). Introducing OTseeker Therapy, 64, 228–234.
(Occupational Therapy Systematic Evaluation of Hayes, R. L. (2000). Evidence-based occupational therapy
Evidence). A new evidence database for occupational needs strategically targeted quality research now.
therapists. The American Journal of Occupational Australian Occupational Therapy Journal, 47,
Therapy, 57, 635–638. 186–190.
Bennett, S., Tooth, L., McKenna, K., Rodger, S., Strong, J., Holm, M. B. (2000). The 2000 Eleanor Clarke Slagle
Ziviani, J., Mickan, S., & Gibson, L. (2003). Lecture. Our mandate for the new millennium:
Perceptions of evidence based practice: A survey of Evidence-based practice. The American Journal of
Australian occupational therapists. Australian Occupational Therapy, 54, 575–585.
Occupational Therapy Journal, 50, 13–22. Humphris, D., Littlejohn, P., Victor, C. O’Halloran, P., &
Canadian Association of Occupational Therapists, Peacock, J. (2000). Implementing evidence-based prac-
Association of Canadian Occupational Therapy tice: Factors that influence the use of research evidence
University Programs, Association of Canadian by occupational therapists. British Journal of
Occupational Therapy Regulatory Organizations, & the Occupational Therapy, 63, 516–522.
Presidents’ Advisory Committee (1999). Joint Position Kielhofner, G. (2005). Scholarship and Practice: Bridging
Statement on Evidence-Based Occupational Therapy. the divide. American Journal of Occupational Therapy,
Canadian Journal of Occupational Therapy, 66, 59, 231–239
267–269. Law, M., & Baum, C. (1998). Evidence-based practice
Christensen, C., & Lou, J. (2001). Evidence-based practice occupational therapy. Canadian Journal of
forum. Ethical considerations related to evidence-based Occupational Therapy, 65, 131–135.
practice. The American Journal of Occupational Lee, C. J., & Miller, L. T. (2003). The process of evidence-
Therapy, 55, 345–349. based clinical decision making in occupational therapy.
Closs, S. J., & Lewin, B. (1998). Perceived barriers to The American Journal of Occupational Therapy, 57,
research utilization: A survey of four therapies. British 473–477.
Journal of Therapy and Rehabilitation, 5, 151–155. Lieberman, D., & Scheer, J. (2002). AOTA’s evidence-
College of Occupational Therapists (2005). Code of ethics based literature review project: an overview. The
and professional conduct. London: Author. American Journal of Occupational Therapy, 56,
Corcoran, M. A. (2003). A glance back and a glimpse 344–349.
ahead. The American Journal of Occupational Therapy, Lloyd, C., King, R., & Bassett, H. (2002). Evidence-based
57, 367–368. practice in occupational therapy—why the jury is still
Coster, W. (2005). International conference on evidence- out. New Zealand Journal of Occupational Therapy,
based practice: A collaborative effort of the American 49, 10–14.
Occupational Therapy Association, the American Ma, H., & Trombly, C. A. (2002). A synthesis of the
Occupational Therapy Foundation, and the Agency for effects of occupational therapy for persons with stroke,
Health Research and Quality. The American Journal of part II: Remediation of impairments. The American
Occupational Therapy, 59, 356-358. Journal of Occupational Therapy, 56, 260–274.
Curtin, M., & Jaramazovic, E. (2001). Occupational thera- McCluskey, A. (2003). Occupational therapists report a low
pists’ views and perceptions of evidence-based practice. level of knowledge, skill and involvement in evidence-
British Journal of Occupational Therapy, 64, 214–222. based practice. Australian Occupational Therapy
Cusick, A. (2001). OZ OT EBP 21C: Australian occupa- Journal, 50, 3–12.
tional therapy, evidence-based practice and the 21st Metcalfe, C., Lewin, R., Wisher, S., Perry, S., Bannigan,
century. Australian Occupational Therapy Journal, 48, K., & Moffett, J. K. (2001). Barriers to implementing
102–117. the evidence based in for NHS therapies.
Dubouloz, C-J., Egan, M., Vallerand, J., & von Zweck, Physiotherapy, 11, 433–441.
C. (1999). Occupational therapists’ perceptions of Murphy, S., & Tickle-Degnen, L. (2001). The effectiveness
evidence-based practice. American Journal of Occupa- of occupational therapy-related treatments for persons
tional Therapy, 53, 445–453. with Parkinson’s disease: A meta-analytic review. The
Dysart, A. M., & Tomlin, G. S. (2002). Factors related to American Journal of Occupational Therapy, 55,
evidence-based practice among US occupational ther- 385–392.
apy clinicians. The American Journal of Occupational Philbert, D. B., Snyder, P., Judd, D., & Windsor, M-M.
Therapy, 56, 275–284. (2003). Practitioners’ reading patterns, attitudes, and
Eakin, P. (1997). The Casson Memorial Lecture 1997: use of research reported in occupational therapy jour-
nals. The American Journal of Occupational Therapy, evidence” as a tool for clinical decision-making. The
57, 450–458. American Journal of Occupational Therapy, 57,
Rappolt, S. (2003). The role of professional expertise in 234–237.
evidence-based occupational therapy. The American Trombly, C. A., & Ma, H. (2002). A synthesis of the
Journal of Occupational Therapy, 57, 589–593. effects of occupational therapy for persons with stroke,
Roberts, A. E. K., & Barber, G. (2001). Applying research part I: Restoration of roles, tasks, and activities. The
evidence to practice. British Journal of Occupational American Journal of Occupational Therapy, 56,
Therapy, 64, 223–227. 250–259.
Sackett, D. L., Straus, S. E., Richardson, W. S, Rosenberg, Tse, S., Blackwood, K., & Penman, M. (2001). From
W. M. C., & Haynes, R. (2000). Evidence-based rhetoric to reality: Use of randomised controlled
Medicine: How to practice and teach EBM (2nd ed.). trials in evidence-based occupational therapy.
Edinburgh; New York: Churchill Livingstone. Australian Occupational Therapy Journal, 47,
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., 181–185.
Haynes, R., & Richardson, W. S. (1996). Evidence Turner, P. (2001). Evidence-based practice and physiother-
based medicine: What it is and what it isn’t. British apy in the 1990s. Physiotherapy Theory and Practice,
Medical Journal, 312, 71–72. 17, 107–121.
Sudsawad, P. (2004). Developing a social validation model Upton, D., & Lewis, B. (1998). Clinical effectiveness and
for effective utilization of disability and rehabilitation EBP: design of a questionnaire. British Journal of
research. Project summary. Submitted to the National Therapy and Rehabilitation, 5, 647–650.
Institute of Disability and Rehabilitation Research, US Welch, A. (2002). The challenge of evidence-based prac-
Department of Education, Grant no. H133F020023. tice to occupational therapy: A literature review. The
Sudsawad, P. (2005). A conceptual framework to increase Journal of Clinical Governance, 10, 169–176.
usability of outcome research for evidence-based prac- Zimolag, U., French, N., & Paterson, M (2002).
tice. The American Journal of Occupational Therapy, Developing expert practice. Striving for professional
59, 351–355. excellence: The role of evidence-based practice and
Sweetland, J., & Craik, C. (2001). The use of evidence- professional artistry. Occupational Therapy Now, 4
based practice by occupational therapists who treat (6), 8–10.
adult stroke patients. British Journal of Occupational
Therapy, 64, 256–261.
Taylor, M. C. (1997). What is evidence-based practice? RESOURCES
British Journal of Occupational Therapy, 60, 470–474. American Occupational Therapy Association:
Tickle-Degnen, L. (1999). Evidence-based practice http://www.aota.org
forum—Organizing, evaluating, and using evidence in American Occupational Therapy Foundation: www.aotf.org
occupational therapy practice. The American Journal of Agency for Healthcare Research and Quality:
Occupational Therapy, 53, 537–539. www.ahrq.gov
Tickle-Degnen, L., & Bedell, G. (2003). Heterarchy National Guideline Clearinghouse: www.guideline.gov
and hierarchy: A critical appraisal of the “levels of OTSeeker: http://www.otseeker.com
C H A P T E R 4 2
Analyzing Evidence for Practice

Nancy A. Baker
The emergence of consumer-driven health care Developing a Question

and the advent of managed care has led to a move-
ment toward evidence-based practice (Tickle- Many practitioners do not know how to start the
Degnen, 1998). In evidence-based practice, process of evidence-based practice. The first step of
healthcare practitioners access and evaluate evidence-based practice is to develop a clear, spe-
research to identify: cific, answerable question (Law, 2002b; Sackett,
Straus, Richardson, Rosenberg, & Haynes, 2000;
• The effectiveness of interventions,
Straus & McAlister, 2000). A well-developed clin-
• The accuracy of evaluations, and
ical question serves several purposes:
• The expected prognosis of a disorder.
• It helps clearly define and refine what is relevant
This information is then used to determine the
to both the practitioners’ knowledge needs and
best practice for different health conditions.
their clients’ care needs,
Sackett and Richardson (1996) defined evidence-
• It suggests specific criteria to use for search
based practice as “the conscientious, explicit, and
strategies, and
judicious use of current best-evidence in making
• It provides a useful format for communicating
decisions about the care of individual patients.” (p.
the information to others (Sackett et al., 2000).
71). The bottom line of evidence-based practice is
that it provides practitioners with objective evi- The key to asking a clinical question is to
dence that an intervention can improve a client’s clearly specify what information is needed. For
health and well-being. instance, practitioners working with clients with
Occupational therapy practitioners are man- spasticity of the hand might want to establish the
dated by the ethics of their practice (American effectiveness of splinting for these clients. Their
Occupational Therapy Association, 2000) to initial question might be:
demonstrate that the intervention they provide not
• Is splinting useful for clients with spasticity of the
only does no harm, but also helps their client.
hand?
Nonetheless, many practitioners feel that the
acquisition and development of the skills neces- While this is definitely a question, it is not spe-
sary to be evidence-based practitioners is a formi- cific. What kind of “splinting”? What is meant by
dable task (Dubouloz, “useful,” and useful for
Egan, Vallerand, & von- what? Even the type of
Zweck, 1999; Straus & The bottom line of evidence- client is not clear. In
McAlister, 2000). While what type of “spasticity”
evidence-based practice based practice is that it is the practitioner inter-
may appear complex, provides practitioners with ested? A well-written
some simple methods clinical question con-
can be used to identify,
objective evidence that an tains specific informa-
access, and critically intervention can improve tion designed to assist
appraise evidence for a client’s health and well- a practitioner to specify
making intervention the clinical problem
decisions. This chapter being. (Richardson, Wilson,
provides a method to: Nishikawa, & Hayward,
1995).
• Develop an answerable question, Questions in evidence-based practice can be
• Find the evidence, divided into two types:
• Do a critical appraisal of the evidence, and
• Calculate effect sizes to understand how an inter- • Background questions, and
vention affects outcomes. • Foreground questions (Sackett et al., 2000).
Each type of question is useful for gathering For example, a question for splinting should
different types of knowledge. Generally novice specify clients potentially seen with this problem,
practitioners ask more background questions and such as clients post-CVA who have moderate to
more experienced therapists ask more foreground severe spasticity. The resulting P might read: In
questions as they are familiar with the disorder and clients 3 months post-CVA with moderate to severe
are trying to develop interventions to address dif- spasticity…
ferent deficits associated with the disorder (Sackett
et al., 2000). Each type of question asks for certain I—The Intervention. The I specifies a particular
types of information. intervention method, or may identify different
intervention options. The I should identify the key
Background Questions criteria of the intervention. This may include not
only the type of intervention, but also how the
Background questions provide groundwork infor- intervention will be applied, and the frequency and
mation about general aspects of the disorder such duration of the intervention. One of the key aspects
as the population involved, modes of evaluation, of naming an intervention is to use the most com-
types of intervention, or overall prognosis. They mon terminology or description so that the evi-
have two essential components: dence will be easy to access. Sometimes more than
• A question root such as who, what, where, how, one name needs to be used in a search to track
or why combined with a verb, and down the best evidence.
• A disorder or aspect of a disorder (see Table For the intervention for the clients with hand
42.1). spasticity the intervention might be specified as a
dorsal splint with digit abductors. The I portion of
While these questions do not answer a specific the PICO question about work might read: …does
intervention question, they provide a practitioner the application of a dorsal splint with digit abduc-
with a general sense of what the disorder may look tors… or it can be further refined by including the
like, act like, or what kind of general implications duration or intensity of the intervention: …does
the disorder may have. the application of a dorsal splint with digit abduc-
tors at night time for 5 days a week …
Foreground Questions
C—The Comparison Intervention. Experimen-
Many times practitioners seek evidence to support tal research usually compares two different forms
an intervention for a specific client, or for a popu- of intervention; one is the intervention of interest
lation of clients whom they see routinely. The fore- (the treatment group or experimental group) and
ground question focuses on specific knowledge one is an alternate, mock, or no intervention
about managing the care of clients with a specific condition (the control group). The C specifies the
aspect of a disorder. Foreground questions are alternate care. Defining a “control” intervention
often called PICO questions because this acronym in the clinical question will focus on whether the
spells out the key components of a specific clinical intervention will be compared to an existing
question: intervention or to receiving no intervention. In
• P—The patient or problem of interest, some studies there is no specific comparison and
• I—The intervention being considered, thus the comparison intervention can be omitted.
• C—The comparison intervention (if any), and For the splinting question, splinting will be com-
• O—The anticipated clinical outcome (Sackett et pared to daily PROM: …as compared to daily
al., 2000) PROM…
O—The Outcome. O specifies what outcomes

Specifying the PICO Question
the intervention will affect, and how the interven-
P—The Patient or Problem. The P defines the tion will affect them. Outcome improvements
criteria of the patient or population of interest and can be in activity performance, such as activi-
should include client information that may affect ties of daily living; in impairments, such as
the outcome of the intervention. While a P should speed, strength, flexibility, or endurance; or in
specify the patient, too many qualifiers will make less tangible outcomes, such as improvements
it too hard to find any evidence at all. The P defines in emotion, cognition, or social ability. The O
the criteria used to determine if the sample for the question could be: …cause a reduction
described in an article can be used with the clini- in tone and increased use of the hand as a
cian’s clients. stabilizer?
Chapter 42 Analyzing Evidence for Practice 665
Table 42.1 Background and Foreground Question Formats

Background Question
Question Root and Verb Disorder or Aspect of the Disorder
• What causes… • …Guillain-Barré syndrome?

• What is… • …the incidence of chronic low back pain?
• What are… • …the characteristics of spastic quadriplegia?
Foreground Questions (PICO)

P Problem or patient I Intervention C Comparison O Outcome
• In workers with low • …does work rehabilitation • …as compared to three • … cause a
back pain of at least or functional rehab times a week reduction in sick
3 months duration… lasting 5 days a week individualized days and
and at least 4 hours a occupational therapy… improve strength
day… and endurance?
• For a 30- to 50-year- • …will a cold suit worn for • …as compared to no • significantly
old client with severe 10 minutes prior to intervention… increase dexterity?
MS and associated therapeutic intervention…
tremors … • significantly improve
• For a middle school • …will 6 months of daily • …compared to 6 months handwriting?
boy with poor shape copying for 10 of daily 10 minutes of
handwriting minutes… handwriting practice…
Based on the points just made, the splinting

question has become quite specific:
Critical Appraisal
In clients 3 months post-CVA with moder-
of the Evidence
ate to severe spasticity, does the applica-
tion of a dorsal splint with digit abductors After completing the search and retrieval, each
at night time for 5 days a week as com- resulting article’s overall quality should be criti-
pared to daily PROM cause a reduction in cally appraised to determine if the research
tone and increased use of the hand as a methodology used is adequate to support using the
stabilizer? results to guide treatment. Critical appraisal helps
It defines the client, intervention, and outcome to identify the strength of the evidence for guiding
in a manner that allows a ready benchmark to practice. For example, if the question is about the
match any accessed literature. Table 42.1 provides effectiveness of an intervention, the highest quality
examples of some other PICO questions. research is that which accurately supports a causal
association between the intervention and the out-
come of interest.
Searching the Literature It is difficult to show a causal association,
particularly when there are complex interactions
Once an answerable question is developed, the between biological, psychological and socioe-
next step is to access conomic factors (Hulley
the evidence. Chapter et al., 2001; Newman,
27 in this book provides …critical appraisal helps to Browner, & Hulley,
a discussion of how to identify if potential threats to 2001; Rosenthal &
find literature and there Rosnow, 1991). To sup-
have been several excel- internal validity have been port a causal association
lent articles and chapters controlled through experi- between an intervention
written on the process and an outcome, all other
of searching the litera- mental design, thus prevent- plausible alternate expla-
ture and accessing the ing an artificial inflation or nations must be ruled
articles (See Resources out. The presence of
at the end of the chap- reduction of treatment effect. plausible alternate expla-
ter). nations for the effects
seen in an article is often referred to as threats to Methodological Considerations

internal validity. Thus, when reviewing interven- for Appraisal
tion studies, critical appraisal helps to identify if
potential threats to internal validity have been con- The following sections will discuss major consider-
trolled through experimental design, thus prevent- ations for critical appraisal of intervention studies.
ing an artificial inflation or reduction of treatment
The Control Group
effect (Moher et al., 1998). The feature box on the
next page provides additional information about One of the most effective methods to control for
threats to internal validity. plausible alternate explanations is a control group
There are many different methods to appraise (Portney & Watkins, 2000). A control group is a
an intervention study article quality (Ajetunmobi, set of subjects who are equivalent to the treatment
2002; GRADE Working Group, 2004; Law, group for all variables except they receive a “con-
2002a), but all appraisal methods are designed trol” treatment, such as usual care. Often, in thera-
to provide a level of confidence that the estimate peutic research studies, the control group is made
of the effect is accurate (GRADE Working up of participants who are on a wait list; in this
Group, 2004). For this chapter, the appraisal instance they receive no control intervention. This
criteria described by Sackett et al. (2000) are used, study design is somewhat weaker because the
but other authors provide variations on these improvement in outcomes seen in the treatment
criteria (see Resources for other critical appraisal group may be the result of the additional attention
methods). Also, when appraising other types experimental subjects receive, not the actual inter-
of evidence (e.g., evidence about the psycho- vention.
metric soundness of an assessment, or diagnostic After the intervention is completed, the control
ability) other criteria relevant to that type of group is compared to the treatment group statisti-
research should be used. This chapter does cally. If the control group has changed signifi-
not provide in-depth information about article cantly less than the treatment group on the
quality for articles other than those focused outcomes being measured, it is reasonable—
on intervention effects. For further information within limits of the design rigor—to attribute any
on appraising articles on diagnostic tests, instru- changes in the outcome to the intervention rather
ment psychometrics, and prognosis, see the than some other reason.
Resources. An important aspect to consider is whether ran-
The most basic critical appraisal is to identify dom allocation was used to assign subjects to the
the research design used for the study. The design treatment and control groups. Random allocation
most likely to prevent threats to internal validity is the best way to ensure that the treatment and
for intervention studies is a randomized clinical control group are equivalent, as it is important to
trial (RCT). The RCT is considered better than a avoid the confounding of group differences with
quasi-experimental study, which in turn is consid- treatment effects (Ajetunmobi, 2002). In addition,
ered better than a case-control or cohort study, the investigator determining if a person meets eli-
which are in turn considered to be better than a gibility requirements for the study should do so
correlational study (see Table 42.2 for more infor- without prior knowledge of what group the subject
mation about different research design characteris- will be assigned. Concealed allocation ensures that
tics). One hierarchy used to assess intervention subjects are retained or excluded from the study
studies as described by Moore, McQuay, and Gray without conscious or unconscious bias (Altman &
(1995) is detailed in Table 42.2, although there are Schulz, 2001) which helps to prevent an inflation
many different variations of these basic hierarchal of results (Moher et al., 1998).
levels. These hierarchies can be used to assign a Even when there is random assignment, the
rating to different intervention articles to help treatment and control groups may be significantly
ascertain the overall quality of any given article different on some measure such as age, ethnicity,
based on its design type. or a baseline outcome score. In this case investiga-
This kind of hierarchy of evidence is widely tors will often control for these differences through
accepted in medicine when examining the effec- the statistical analyses, and should mention this in
tiveness of an intervention. However, as noted in their description of the statistical analyses.
Chapter 41, some occupational therapists have
called for a more flexible approach to what is con- Intention to Treat Analysis
sidered “best” evidence depending on the type of Another safeguard of internal validity is a method
question being answered. of statistical analysis called “intention to treat”
Threats to Internal Validity
In 1979, Cook and Campbell (1979) described following table provides a review of some of these
some threats to internal validity that could occur threats as well as a description of how they can be
during the process of a study. Others have elabo- controlled.
rated on these threats to internal validity. The
Threats to Internal Validity

Threat Definition Control Example
History* Observed effect may Control group A single group (pre–post study) study
be due to events Random examined the effect of exercise on reducing
that take place assignment depression in individuals with fibromyalgia.
between baseline Isolation During the course of the study a brand new
and follow-up. drug was released that reduced depression
in those with fibromyalgia. More than half
the sample was placed on the drug. At the
end of the study there was a big, significant
difference in baseline and follow-up
measures of depression.
Maturation* Observed effect may Control group A 2-year single group (pre/post study) study
be due to Random examined the effect of fine motor
changes occurring assignment coordination training on improving
simply as a handwriting skills in 7-year-olds. Subjects
function of the received weekly treatment. At the end of 2
passage of time. years there was a big, significant difference
between the 7-year-old (baseline) and 9-
year-old (follow-up) scores.
Attrition Observed effect may Make treatment A study examined the effect of OT on the
(Mortality)* be due to the easy health of the well-elderly. They recruited
differential loss of Make visits easy 300 well-elderly and initiated a 6-month
subjects between Make measure- program of OT with 150 of them. The other
groups. ments painless 150 received no treatment. At the end of
Encourage subjects the study, 75 had dropped out of the no
to remain treatment group (attrition rate 50%) and 30
Find lost subjects out of the treatment group (20% attrition
Intention to treat rate). There was a big, significant difference
analysis between the treatment and control group.
Testing* Observed effect may Control group A single group (pre–post study) study
be due to Random examined the effect of strength training on
repeated testing assignment dexterity tested 10 subjects with the 9-hole-
causing improve- Vary tests peg test, then initiated a daily strengthening
ments in the test Limit retests program. At the end of each session, the
due to familiarity/ subjects were retested with the 9-hole peg
learning. test. At the end of 10 sessions there was a
big, significant difference between baseline
and follow-up scores on the 9-hole peg test.
Instrumen- Observed effect may Control group A single group (pre/post study) study examined
tation* be due to changes Blinding the effect of a home program in Thera-
in the instrument Frequent putty® on hand strength. Unbeknownst to
from baseline to instrument the raters, the dynamometer used to
follow-up. calibration measure strength was dropped on the floor
Instruments may be Training/certifying just prior to follow-up testing. All subjects
improperly cali- raters showed at least a 10-lb difference in
brated and/or Repeating strength at follow-up which was significantly
insensitive partic- measures different from baseline score.
ularly at the Choose sensitive
extremes of instruments
measurement.
(continued)
667
Threats to Internal Validity (continued)

Raters may be
unreliable and/or
biased.
Regression Observed effect Control group A single group (pre–post study) study
to the may be due to Random examined the effect of tai chi on balance
Mean* the tendency of assignment by recruiting 40 individuals who scored
extreme scores Multiple below the 2nd standard deviation on the
to retest nearer measurement Berg Balance Scale. They participated in
to the point of methods a 6-week program of tai chi. At follow-up,
central tendency Reliable tests all subjects have improved to within at
(mean). least one standard deviation of the mean.
There was a big, significant difference
between baseline and follow-up tests.
Selection* Observed effect Random A study examined the effect of night splinting
may be due to assignment on work related carpal tunnel syndrome.
the difference in Assess differences Those subjects who come in the morning
the type of between groups were assigned to the control group, those
people in each pre intervention. in the afternoon were assigned to the
experimental Control in statistical treatment group. After the study was
group. analysis completed, an examination of the
demographic data suggests that more of
the people in the treatment group were
currently at work, while those in the
control group were off of work. There was
no significant difference between the
groups.
Diffusion/ Treatment effect is Blinding A study examined the effect of training in
imitation of eliminated if Limit contact patient handling techniques on reports of
treatment* treatment between groups back pain. One unit in a hospital was the
involves Reinforce need not treatment group, another was the control.
informational to share During the study the investigator finds
programs and information. the treatment group handouts in the
the treatment Let both groups lunch room used by both groups. There
group shares know informa- were no significant differences between
information with tion will be groups on reports of pain at follow-up.
the control shared post-
group. study.
Compensatory Treatment effect is Blinding A study examined the effect of group
equalization eliminated if Let all know that Sensory Integration (SI) on reading in
of treatment* group receiving control group 7-year-olds. Children with SI problems
“less desirable” will receive were randomly assigned to receive SI
treatment treatment after intervention or no intervention. After 4
receives study. weeks of treatment the investigator found
additional out that the teacher was assigning those
services/ help by in the control group to extra reading
those aware of practice during the times when the
this “inequality”. treatment group was receiving SI. At the
end of the study there was no significant
difference between those receiving SI
and the control group.
(continued)
668
42Keilhofner(F)-42 5/18/06 10:53 AM Page 669
Threats to Internal Validity (continued)

Compensatory Treatment effect is Blinding A study examined the effect of modeled work
rvalry* eliminated if Let all know training on the productivity of individuals in
members of that control sheltered workshops. One group in a facility
group receiving group will was assigned to the treatment group, the other
“less desirable” receive to the control group. At the end of a year, the
treatment are treatment productivity of the control group is 25% greater
motivated to work after study. than any previous years’ productivity. There
harder than usual was no significant difference between the two
(“John Henry groups.
effect”).
Resentful Observed effect may Blinding A study examined the effect of weekend adventure
demoral- be due to the Let all know camp on short term memory of individuals with
ization* negative reaction that control TBI. The treatment group traveled to the camp
of the group group will while the control group remained in the group
receiving “less receive home over the weekend. At follow-up 1 week
desirable” treatment later, the control group’s mean memory has
treatment who after study. decreased significantly from its mean baseline
may reduce their score. There was a significant difference
level of effort. between the two groups on short-term
memory.
Placebo Observed effect may Control group A study examined the effect of ultrasound on wrist
effect** be due to the Blinding tendonitis pain. The treatment group received
subjects the clinical protocol ultrasound treatment, the
expectation that control received a sham treatment with the
the treatment will machine turned off. There was no significant
cause a change. difference between the treatment and control
groups.
Hawthorne Observed effect may Control group A study examined the effect of playing gin rummy
effect** be due to the Sham on memory and pain in elderly individuals in a
attention inherent treatment SNF. Each individual in the treatment group
in the research Blinding played gin rummy three times a week one on
process, not the one with the researcher, the control received
treatment itself. no treatment. At the end of 4 weeks the
treatment group shows significant
improvements in both memory and pain.
Halo effect** Observed effect may Blinding experi- A study examined the effect of a new form of
be due to menter dressing training on individuals with CVA. The
researchers’ investigator developed the new training and
expectations plans to market it once the research proves it
about the is better than the traditional method. Sixty
performance of subjects were randomly assigned to one of two
their subjects groups, one received the new treatment and
which may bias one received the traditional treatment. All
their judgment or evaluations were carried out by the investigator
cause them to who was aware of what group each subject
interact differently was assigned to. The outcome measures were
with subjects along the five level continuum of dependent to
depending upon independent. By the end of 2 weeks, all
group subjects in the treatment group were rated as
assignment. basically independent by the evaluator, while
only one third of the control group received a
score of independent. There was a significant
difference between the two groups.
*Cook and Campbell (1979); **Berg and Latin (1994)
669
Table 42.2 Hierarchy of Levels of Evidence by Research Design

Level Research Design
I Systematic review of research • In a systematic review, many articles about the topic are analyzed
studies (randomized and synthesized together to develop an overview of the intervention
clinical trial [RCT]) based on many different sources. Systematic reviews are considered
to be the strongest evidence because they combine the overall
conclusion of all evidence.
II Randomized clinical trial • An RCT research design includes at least one control group as well
(RCT) as the intervention group. It has random allocation of subjects to each
group. All other variables are held constant except for the intervention
which the researcher manipulates by group.
III Quasi-experimental, pre–post, • A quasi-experimental study is similar to an RCT except subjects are
cohort, and case control not randomly assigned to the treatment and control groups.
studies • A pre–post study has a single group of subjects assessed at baseline
and at intervals.
• A cohort study follows a group of subjects who do not yet have the
outcome of interest over time to see who develops the disorder. The
investigators then examine variables gathered at baseline to see
which ones are associated with those who develop the disorder.
Unlike an RCT, the investigator does not manipulate an intervention
within the groups.
• A case control study selects a group of subjects (cases) who have
the outcome of interest and matches it to a group of subjects without
the outcome (controls). The investigator looks in all the subjects’ pasts
to see what variables are associated with the outcome of interest.
IV Correlational studies of • A correlational study has a group of subjects. The investigators look
multiple sites for associations between variables and the outcome of interest. To be
a level IV, the subjects must be gathered from multiple sites.
V Correlational studies, • Qualitative studies examine thoughts and ideas by conducting in-
qualitative studies; expert depth interviews of subjects and analyzing these interviews for
opinion common themes.
• Expert opinion is a idea or belief developed by someone with
experience, but that idea does not have research to support the
conclusions.
Adapted from: Moore, McQuay, & Gray (1995).
analysis (Hollis & Campbell, 1999). In this type of Length of Follow-Up

analysis, each subject is analyzed in the group to
which he or she was assigned, regardless of Investigators must allow sufficient time for the
whether the individual received the intervention. outcomes of interest to develop fully (Sackett et
This qualification of analysis groups is important al., 2000). In some interventions the outcome will
as some subjects switch groups or dropout during be immediately evident. For example, applying
a research study. This is not an intuitive safeguard splints to improve coordination can be tested
as it would appear to make more sense to analyze almost immediately. Other times, the follow-up
subjects’ data as part of the group they attended, period may need to be weeks or months. For
not to which they were assigned. However, factors example, many stroke studies follow subjects for
such as attrition may seriously bias the results, at least 6 months to make sure the stroke has
destroying the random allocation and the equiva- resolved fully in order to understand the implica-
lency of the groups (Sackett et al., 2000). The eas- tions of the intervention over the full span of the
iest way to be sure an intention to treat analysis rehabilitation and healing process.
was used is to look in the data analysis section
Blinding
for the term “intention to treat,” although older
articles may not list that this was done (Hollis & Blinding, in which the client, investigator, asses-
Campbell, 1999). sor, or other important study personnel is unaware
of who is and who is not receiving the interven- may not be worth further assessment if better evi-
tion, is an important method of preventing bias dence is available. When there is no other available
(Schulz & Grimes, 2002). Subjects’ knowledge of evidence, the findings should be considered in
their treatment status or investigators’ expectations light of the design weaknesses. Studies with seri-
can consciously or unconsciously influence out- ous design flaws tend to systematically either
comes, or the recording of outcomes. Blinding can inflate or deflate the size of the effect on outcomes
include hiding the intervention status from: (Rosenthal & Rosnow, 1991), and therefore may
suggest that an intervention has very large effects,
• The subjects,
when in reality, the true effect is negligible. In
• Those who provide the intervention or measure
determining whether to use the results of a study,
outcomes, and
practitioners should consider:
• Those who analyze the data.
These various levels of blinding are called • The type of intervention described,
single-blinding, double-blinding, and triple-blind- • Whether the theory supporting its use is sound,
ing, respectively. Blinding is not always feasible, and
particularly where therapeutic interventions are • Whether it has been used routinely in clinical
involved as it may be impossible to hide from a practice, or is a new, “innovative” technique.
subject or provider whether he or she is receiving In cases where weaker evidence is used to sup-
or implementing an intervention (Schulz & port practice, practitioners should continue to
Grimes, 2002). search the literature in the future for new evidence.
Attrition
In studies that last for more than a few days, some Effect Size Calculation
of the research subjects may drop out or be lost
before the final outcome measures are taken, In clinical research, investigators measure the out-
called attrition. In general, if a study loses more comes that are considered most important. For
than 20% of its sample before follow-up, there is a example, an investigator examining the effect of
possibility of serious bias (Sackett et al., 2000). occupational therapy on function might choose an
For example, if a large number of the less healthy outcome measure that captures some aspect of
subjects enter a nursing home and are not meas- functional ability. Investigators analyze data on
ured at follow-up, the research will be unable to these outcomes with inferential statistics (see
describe the effect of the intervention on those Chapter 17), obtaining a p-value to determine if
with poorer health. Their loss may skew the results the results are significantly different. By conven-
toward healthier subjects. tion, a p-value of less than .05 is considered statis-
tically significant (Ajetunmobi, 2002; Portney &
Watkins, 2000). Significance, however, does not
Critical Appraisal
refer to the size or magnitude of the observed
The above criteria or criteria described in other effect. Statistically significant differences can
critical rating systems (see Resources) should be occur in situations where there is only a very small
used to critically appraise any evidence to deter- clinical effect.
mine if the article is worth further assessment and For example, Table 42.3 shows a hypothetical
use in practice. Evidence-based practice is based study examining the effect of Thera-putty® on grip
on the premise that the “best available evidence” strength. It reports the following after 6 months of
should be used (Sackett & Richardson, 1996). If daily Thera-putty®. There is a significant differ-
the research design has serious design flaws (i.e. ence between the treatment and control group for
lacking several of the above criteria) the article right and left grip strength, as represented by the
Table 42.3 Hypothetical Results of a Study Examining the Effect of Thera-putty® on Hand Strength
Treatment Group Mean Control Group Mean p-Value
Pre-intervention Post-intervention Pre-intervention Post-intervention
Grip strength right 34 lbs 37 lbs 34 lbs 35 lbs .03

Grip strength left 23 lbs 26 lbs 23 lbs 24 lbs .04
p-values of less than 0.05. This finding suggests Calculating and Interpreting
that the Thera-putty® intervention caused a change Number Needed to Treat (NNT)
in the grip strength of the treatment group. How-
ever, a closer examination shows that the hypo- Calculating NNT
thetical treatment group’s bilateral grip increased
by only 3 pounds. This is not a clinically large Number Needed to Treat (NNT) compares the
effect, particularly for 6 months of daily therapy. number of treatment group subjects (those receiv-
Significance, therefore, does not indicate the ing the intervention under study) who had a suc-
magnitude of the effect of the intervention. As dis- cessful outcome to the number of people in the
cussed in Chapter 16, an effect size refers to the control group who had a successful outcome with-
magnitude of differences found between the out- out that intervention. As illustrated on Table 42.5,
comes of persons in a treatment group and those in to calculate NNT (Bandolier, 2003; Sackett et al.,
a control group (Tickle-Degnen, 1998). Increas- 2000) one must:
ingly, studies will report effect size. This section
• Change the number of people who received the
provides information on how to quickly calculate
intervention and improved into a percentage by
effect size (when it is not provided) from available
dividing this number by the number of people
information in an article. It also discusses how to
allocated to the treatment group at the beginning
interpret an effect size.
of the study. This is the Treatment Event Rate
Outcomes usually come in one of two forms,
(TER).
frequencies and means. Frequencies are a count of
• Change the number of people who did not
the number of individuals who have had a particu-
receive the intervention and improved into a per-
lar result in each group, and are usually reported as
centage by dividing this number by the number
percentages. Dichotomous data assume an all or
of people allocated to the control group at the
nothing outcome frequency (e.g., the subjects stay
beginning of the study. This is the Control Event
at home or are placed in nursing homes; subjects
Rate (CER).
find jobs or remain unemployed). In other cases
• Take the absolute value of TER subtracted from
results are measured as continuous data (e.g., grip
CER. This is the Absolute Rate Reduction
strength or an ordinal scale such as the FIM scores
(ARR).
of 0 to 7). Results in these studies are often
• Divide 1 by the ARR.
reported as means and standard deviations.
• Round the resulting number up to the next high-
The Number Needed to Treat (NNT) is one
est whole number. This is the Number Needed to
method for determining an effect size for dichoto-
Treat (NNT).
mous data, while statistics such as d, r, and BESD
can be used to determine an effect size for contin- In the Jousset et al. (2004) article, the investi-
uous data. gators evaluated several dichotomous outcomes
In Table 42.1 one of the PICO questions was including:
focused on work rehabilitation. A search of the lit-
erature found an article by Jousset et al. (2004) that • The number of clients who subjectively rated
had adequate article quality (see critically their ability to work positively,
appraised paper [CAP] in Table 42.4). This study • The number of clients who subjectively rated that
compared the effect of an intervention called func- their physical condition was improved, and
tional restoration, which included daily occupa- • The number of clients who reported that they
tional therapy as part of a comprehensive program, increased their participation in sports and leisure
to a physical therapy intervention on reducing sick activities.
days and increasing fitness for clients with chronic
low back pain. In the study, 84 patients were ran- In all cases, the percentage of subjects in the
domly assigned either to treatment (functional treatment group who had positive outcomes was
restoration for 6 hours a day for 5 weeks, n 43) greater than the percentage of subjects in the con-
or control (3 hours a week of physical therapy for trol group who had positive outcomes. However,
5 weeks, n 41). The investigators examined the the treatment group was significantly better than
effect of each type of intervention, functional the control group only for the number of clients
restoration or physical therapy on several types of who participated in more sports and leisure activi-
outcomes: some frequency data and some continu- ties (see p-values in Table 42.5). To determine the
ous measures. In the following sections, the mag- magnitude of the clinical effect on these outcomes
nitude of the differences between these two NNT must be calculated. Table 42.5 provides the
interventions on some of these outcomes will be calculations and the NNT for each of these out-
examined using NNT, d, r, and BESD. comes.
Table 42.4 Critically Appraised Paper (CAP) Format for Jousset et al. (2004)
Clinical Bottom Line: Clients with low back pain show significant improvements in impairment and activity/
participation restrictions if they participate in any structured program (either functional rehabilitation or
3 times a week physical therapy). However, functional rehabilitation is more effective for impairment level
outcomes (strength, endurance, and flexibility) and demonstrates a trend toward being more effective and
having a larger effect size than 3 times a week physical therapy for most outcomes.
Finding the Article
Clinical population: Employed clients with chronic low back pain

Four-part question: (PICO)
In workers with low back pain of at least 3 months duration does work rehabilitation or functional rehabilitation
lasting 5 days a week and at least 4 hours a day as compared to three times a week physical therapy
cause a reduction in sick days and improve strength and endurance?
Study
Citation
Jousset, N., Fanello, S., Bontoux, L., Dubus, V., Billabert, C., Vielle, B., Roquelaure, Y., Penneau-Fontbonne,
D., & Richard, I. (2004). Effects of functional restoration versus 3 hours per week physical therapy: A
randomized controlled study. Spine, 29, 487–494.
Critical Appraisal
Threats to internal validity Present (Yes/No) Comments

Random allocation….? Yes
Concealed allocation…? ? Not clear—person assigning subjects not
specified.
Intention to treat analysis…? ? Not specified; however, statistical n
matches baseline n.
Long enough treatment/follow-up.? Yes 5 weeks of treatment, 6 months follow-up.
Blinding…? No Researchers felt standardized testing and
close communication reduced
possibility of bias.
Groups equal at baseline…? No Significantly more workers in the
treatment group had surgery,
otherwise the groups were equal.
Attrition less than 20%...? Yes
Other threats?
Evidence Level? II (based on Moore et al. (1995) hierarchy)

Subjects
Treatment group characteristics (n 43): Age—41; Sex—male 70%; on sick leave—47%; Previous
surgery—35%; Previous depression—35%; Smokers—37%
Control group characteristics (n 41): Age—39; Sex—male 63%; on sick leave—51%; Previous surgery—
15%; Previous depression—24%; Smokers—42%
Intervention/Method
Treatment group—Functional Restoration Program (FRP): Place: two different rehab centers. Duration: Six
hours per day, for 5 weeks. Consisted of group treatment including warm-up/stretching, strengthening,
aerobic activities, occupational therapy, endurance training, balneotherapy, individual interventions.
Control group—Active Individual Therapy (AIT): Place: Private physiotherapist practice. Duration: 1 hour
per day, 3 times a week, for 5 weeks contact with therapist combined with a daily 50-minute home
exercise program. First 2 weeks focused on flexibility, range of motion, and pain coping strategies.
Strengthening and functional training were then added. Cardiopulmonary was achieved through
sports activities.
(continued)
673
Table 42.4 Critically Appraised Paper (CAP) Format for Jousset et al. (2004) (continued)
Results: FRP group demonstrated significant, large improvements in participation in sports and leisure
activities (NNT 5), trunk strength (r .24), and endurance (r .29). There were no significant
differences for any other outcome, although there was a trend for those in the FRP to show more
improvement than the physical therapy group. A comparison of baseline and follow-up scores suggested
that both interventions had significant effects on both impairments and activity/participation, but the FRP
generally showed a larger effect. Selected r include: sick leave—FRP 0.50, AIT 0.41; flexibility—FRP 0.50,
AIT 0.22; lifting—FRP 0.30, AIT 0.29; endurance—FRP 0.25, AIT 0.01; pain intensity—FRP 0.38, AIT 0.12;
Activity/participation (Quebec scale)—FRP 0.37, AIT 0.25.
Additional Comments
This study was completed in France. The system for injured workers is different than for USA.
Date of Completion—September 26, 2004
Interpreting NNT • Five clients would have to participate in func-

tional restoration therapy in order for one addi-
The NNT is the number of clients who would have tional client to participate in more sports and
to be treated with the intervention studied for ben- leisure activities.
efit to occur in one additional client who otherwise
would have had an unsuccessful outcome (Katz,
Although the NNT for the first two outcomes,
2001; Moore & McQuay, 2001; Sackett et al.,
ability to work and physical conditioning, are
2000). The smaller a NNT the better the interven-
included, the results of these outcomes are non-
tion works. In the case of the Jousset et al. (2004)
significant, suggesting that they might not apply to
article, the NNT can be interpreted in the follow-
any subjects except those in the study itself.
ing fashion:
Calculating an effect size for a nonsignificant out-
• Thirteen clients would have to participate in func- come has the benefit of identifying outcomes that
tional restoration therapy in order for one addi- may have a large effect (i.e., a small NNT), but had
tional client to report improved ability to work, an insufficient number of subjects to obtain power.
• Seventeen clients would have to participate in For those nonsignificant results that have a large
functional restoration therapy in order for one effect, the practitioner may calculate the 95% con-
additional client to report improved physical fidence interval to better understand the range of
condition, and possible effect sizes associated with that outcome
Table 42.5 Calculation of NNT from Selected Outcomes from the Jousset et al. (2004)
Outcome Follow-up n Event rate ARR NNT p*
Control Treatment Control Treatment
Group group (CER) (TER)
nfollow-cont nfollow-tx 1
(baseline (baseline ⏐CERTER⏐ ARR
nbase-cont nbase-tx ARR
n 41) n 43)
33 38 1
Improved ability 33 38 .805 .884 ⏐.805.884⏐ .079 13 0.20
41 43 .079
to work
28 32 1
Improved 28 32 .683 .744 ⏐.683.744⏐ .061 17 0.42
41 43 .061
physical
condition
21 32 1
Sports and 21 32 .512 .744 ⏐.512.744⏐ .232 5 0.02
41 43 .232
leisure
activities
*Reported in the article.

How Precise Are the Effect Sizes Calculated from the Study—the 95% Confidence Interval?
In most research, the study effect size is consid- is invaluable in providing the range of possible
ered to be an estimate of the magnitude of the effect sizes and identifying the precision of the
effect of the intervention on the actual population. effect size that was obtained from the study.
However, the study effect size actually represents To calculate a 95% CI around a NNT, first cal-
an approximation of the population effect size. In culate the standard error for the ARR. (The table in
many cases, the study effect size can vary greatly this box provides the formula for the standard
from the population effect size. The 95% confi- error of ARR). This is multiplied by 1.96, and the
dence interval (95% CI) provides the range of pop- resulting number is both added and subtracted
ulation effect size values (Sackett et al., 2000; Sim from the ARR calculated for the study. The result-
& Reid, 1999), which can express the degree of ing two numbers represent the range of possible
uncertainty in relationship to that value represent- NNT that might be obtained if this study was
ing the actual population effect size. This number repeated again.
Calculation of the 95% Confidence Intervals (95% CI) Around NNT Calculated
from Selected outcomes from Jousset et al. (2004)
95% CI NNT
Outcome Event Rate Standard Error (SE) ARR 1.96(SE) (95% CI)
Control Treatment 95% CI 95% CI
(CER) (TER)
(n 41) (n 43)
SENNT
冪莦莦莦
TER(1TER)

nt
CER(1-CER)
nC
ARR
1.96(SE)
ARR
1.96(SE)
Improved
ability
to work
.81 .88 .069
冪莦莦莦
.90(1.90)

43
.88(1.88)
41
0.21 0.06 11 (5 ∞)
Improved
physical .68
condition
.74 .099
冪莦莦莦
.74(1.74) .68(1.68)

43 41
0.26 0.13 17 (4 ∞)
Sports and
leisure .51
activities
.74 .103
冪莦莦莦
.74(1.74) .51(1.51)

43 41
0.44 0.03 5 (3 34)
If the 95% CI of the ARR has a zero in the range, the overall result is considered to be nonsignificant. Thus,
when the NNT is calculated from a negative ARR, the range of possible results is reported as being from
the lower range of the 95% CI (calculated from the positive 95% CI) to infinity.
A 95% CI can also be calculated around an r. As an r has a skewed distribution, it must be translated to an r
with a normal distribution, a Fisher r. A standard error (SE) is calculated from the Fisher r using the follow-
ing formula SEr 冪莦 1

where n equals the total n of the sample. As with the SE derived for the NNT,
ntot 3
the SE derived for the r is multiplied by 1.96 and added to and subtracted from the Fisher r. The resulting
numbers, as well as the original Fisher r are translated back into a regular r.
to help determine if the intervention may be useful not possible before, then an intervention with a
to implement. In general, however, if a result is “high” NNT may be of use. For example, consider
nonsignificant, the evidence-based practitioner an OT intervention that allowed 1 out of every 10
will not calculate an effect size. young adults with psychosis to stay in the commu-
What constitutes a “low enough” NNT to jus- nity and out of a nursing home. If there was no
tify using the intervention? Unfortunately, there is alternative intervention, then this intervention
no absolute answer (Herbert, 2000). If an interven- would still be considered worth applying both
tion improves the life of a client in a way that was because it addresses an important aim (i.e., pre-
venting institutionalization) and it avoids the cost Display (BESD), which appears to be most easily
of a nursing home. However, if the intervention is understood by both clinicians and clients. The
one of many possible interventions to achieve this BESD provides intervention success rates which
aim, then an NNT of 10 may be too high. To put compare the percentage of subjects who benefited
the NNT in perspective, practitioners have to draw with the intervention (treatment group) to the per-
upon their clinical expertise and consider such centage of subjects who benefited without the inter-
issues as: vention (the control group) (Tickle-Degnen, 1998).
All three of these statistics are discussed below.
• Treatment alternatives,
• Resource availability for the intervention,
• Resource consequences of the intervention, and Calculating d
• Their clients’ values and needs.
To calculate an effect size d, subtract the mean of
Calculating and Interpreting Continuous the control group (Mc) from the mean of the treat-
ment group (MT) (Rosenthal, 1994). This differ-
Data Effect Sizes—d, r, BESD
ence score is divided by the average of the
The d statistic is essentially a standardized or z- standard deviations for the two groups (pooled
score, or the difference between two scores in stan- standard deviation) (see Table 42.6 for d formula).
dard deviation units (Portney & Watkins, 2000). It A d can be either a positive or a negative num-
is discussed in detail in Chapters 16 and 17. The d ber. Whether a positive number indicates a higher
can be transformed into a r, which is a partial cor- success rate for the treatment or control group is
relation coefficient that indicates the degree of dependent upon the direction of the outcome
association between receiving the intervention and measure. For some outcome measures a higher
having a successful outcome (Tickle-Degnen, score indicates a more positive outcome; for
1998). An r of 1 indicates that everyone in the instance, strength and range-of-motion (ROM) are
treatment group had a successful outcome while both such measures. On the other hand, for some
no one in the control group had a successful out- outcome measures a smaller score indicates a more
come. Conversely, an r of 0 indicates that there positive outcome. For example, in most pain meas-
was absolutely no difference in successful out- ures a lower score usually indicates less pain; thus
comes between the treatment and control groups. a lower score is better.
In most studies, r lies somewhere between 0 and 1. Since the control mean is subtracted from the
Both r and d can be interpreted in a clinically treatment mean, a positive outcome indicates that
useful manner, however, Rosenthal and Rubin the treatment group did better than the con-
(1982) have created a method of translating r into a trol group when a higher score indicates success.
paired statistic called the Binomial Effect Size For an outcome where a lower score indicates suc-
Table 42.6 Mean, Standard Deviation, d Calculations, d, and p-Values

for Selected Outcomes from Jousset et al. (2004)
MTMC

Treatment Control SD POOLED
M SD M SD d P**
28.748.3
No. of sick days* 28.7 44.6 48.3 66.0 0.35 0.12
55.3
35.133.7
Lifting capacity (PILE) 35.1 12.6 33.7 12.7 0.11 0.65
12.7
92.766.3
Endurance (kJ) 92.7 49.3 66.3 36.7 0.61 0.01
43.0
*An outcome in which a lower score indicates a more successful outcome.

** p-value reported in article.
PILE Progressive isoinertial lifting evaluation.
Table 42.7 Interpreting the Negative and Positive Signs Associated with d
The higher the score the more The lower the score the more
successful the outcome. successful the outcome.
d is positive. Treatment did better on the outcome. Control did better on the outcome.
d is negative. Control did better on the outcome. Treatment did better on the outcome.
cess a negative score indicates that the treatment endurance outcome, suggesting that the magnitude
did better than the control. Table 42.7 provides a of the difference for the other outcomes might be
brief summary of interpreting a positive or nega- quite different if the study was repeated with other
tive d. samples. It is also possible that the magnitude of
Jousset et al. (2004) reported on several out- effect would be different if compared to no treat-
come measures that were interval scales, and the ment. The feature box titled “Comparing Baseline
results were provided as means and standard devi- and Follow-up Results” discusses the changes in
ations. Table 42.6 provides the measures, the the magnitude of effect when comparing a new
means, standard deviations, and calculates the d treatment to no treatment. A d can also be calcu-
for each outcome. lated from the difference between pretest and
posttest scores. This method is described in the
Interpreting d feature box titled “Calculating Effect Sizes from a
Difference Score.”
The effect size d provides the size of the difference
between the means in standard deviation units. An
effect size d of 0.50 means the difference between Calculating r
the means is 1/2 standard deviation unit (Tickle- It can be useful to translate the effect size d to the
Degnen, 2001). An effect size d of 1 means that effect size r. While the d examines the difference
there is one standard deviation unit between the between the scores, the r looks at the association
means. Cohen (1988) has provided an interpreta- between a more positive outcome and participation
tive guideline for the d effect size that is shown in in treatment (Tickle-Degnen, 2001). Table 42.9
Table 42.8. provides the equation for calculating an r from a d
For the Jousset et al. (2004) (see Table 42.6), (take the square root of d squared divided by d
the functional restoration therapy in comparison to squared plus 4) and changes the d’s calculated
physical therapy had: from the Jousset (2004) article into r’s (Rosenthal,
• A negligible effect on the subjects’ lifting ability 1994).
(PILE),
• A small effect on the number of sick days, and Interpreting r
• A moderate effect on the subjects’ endurance.
The effect size r is a partial correlation coefficient;
In all cases the functional restoration group did it indicates the degree to which the independent
better on each outcome than the physical therapy measure (treatment vs. control) is associated with
group. The results were significant only for the the outcome scores. Since it is a coefficient, r can
range from -1 to 1. A negative r indicates that the
control group was more successful than the treat-
Table 42.8 Interpreting the Effect Sizes d ment group. Since the r calculation always pro-
and r duces a positive value, the sign of an r must be
assigned once the calculation is completed using
Interpretation d r
the d as a guideline (i.e., if d indicated that the
Negligible effect 0–0.19 0–0.09 experimental group had a better outcome, then r
Small effect 0.20–0.49 0.10–0.23 should be assigned a positive value; if d indicated
Moderate effect 0.50–0.79 0.24–0.37
the control group had a better outcomes, then r
should be assigned a negative value). The r can be
Large effect 0.80–0.99 0.38
interpreted something like a percent score (Tickle-
Very large effect 1.00 – Degnen, 1998). An r of .25, suggests that the treat-
ment group was 25% more successful than the
From Cohen (1988). control.
Comparing Baseline and Follow-up Results from Jousset et al. (2004)
Although the effect sizes reported here appear to appears that both physical therapy and functional
show that functional restoration intervention is not restoration are effective interventions for some
very effective, this is a misperception. Although outcomes related to chronic low back pain, though
the effect sizes for functional restoration appear functional restoration is more effective, particu-
low, it is because they were compared to another larly for endurance. What the example points out
effective intervention, physical therapy. The table is that in interpreting an effect size, one must
in this box provides an example of the effective- always take into consideration the nature of the
ness of both functional restoration and physical control condition. When an experimental interven-
therapy in improving function for clients with tion is being compared to a control condition that
chronic low back pain by calculating the effect has little or no benefit for subjects, it is easier to
sizes based on the differences between baseline show a larger effect size. If the same intervention
and follow-up scores rather than the differences is compared to a control condition that also has a
between the treatment and control group. The positive impact on the outcome variable(s), the
effect sizes are generally moderate to large (see effect size will appear smaller.
table in this box). From these effect sizes, it
Effect Size r Comparing Functional Restoration Therapy and Physical Therapy Baseline and Follow-
Up Scores for Selected Outcomes from Jousset et al. (2004)
Baseline Follow-up
M SD M SD r
No. of sick Functional 101.3 79.1 28.7 44.6 0.50

days Physical 109.8 70.4 48.3 66.0 0.41
Lifting capacity Functional 27.5 13.3 35.5 12.5 0.30
(PILE) Physical 27.5 11.6 35.0 13.0 0.29
Endurance Functional 70.2 39.0 92.7 49.3 0.25
(kJ) Physical 65.6 37.3 66.5 37.3 0.01
Calculating Effect Sizes from a Difference Score
Some articles do not provide mean scores, but dif- true effect because it controls for the differences
ference scores. The difference score is most often between the intervention and control group at
the difference between the score at baseline and at baseline.
follow-up. Difference scores can be treated like a A difference score can be created from infor-
mean to calculate the effect size d (see table in this mation available in most articles. Simply subtract
box). Subtract the control difference score from the baseline score from the follow-up score for
the intervention difference score and divide by the both the treatment and control group. Divide this
pooled standard deviation of the difference score. score by either the baseline or follow-up pooled
This effect size is often more representative of the standard deviation.
Calculating the Effect Size d from a Difference Score for Joussett et al. (2004)
MTMc

SD
Treatment Control POOLED
Mbase Mfoll Mdiff SDbase Mbase Mfoll Mdiff SDbase d
7.66.2
Lifting 27.5 35.1 7.6 13.3 27.5 33.7 6.2 11.6 0.11
12.5
capacity
(PILE) 22.5(.09)
Endurance 70.2 92.7 22.5 39.0 67.2 66.3 –0.9 37.3 0.59
38.2
(kJ)
In the case of Jousset et al. (2004), the results of calculating an effect size from the difference score are no
different than calculating an effect size from the follow-up score. This indicates that randomization worked
well in this study to achieve experimental and control groups that were equivalent.
678
Table 42.9 Transforming the Effect Size d to the Effect Size r for Selected
Outcomes from Jousset et al. (2004)
冪莦
d2
d r p**
d 4
2
冪莦
.352
No. of sick days* 0.35 0.17 0.12
.352 4
冪莦
.112
Lifting capacity (PILE) 0.11 0.05 0.65
.112 4
冪莦
.612
Endurance (kJ) 0.61 0.29 0.01
.612 4
*An outcome in which a lower score indicates a more successful outcome.

**p-value reported in article.
Like the d the r can also be classified as small, Calculating the Binomial
moderate, or large (Cohen, 1988) (see Table 42.8). Effect Size Display (BESD)
For the Jousset et al. (2004) outcomes the func-
tional rehabilitation intervention was: The BESD provides a result that is most readily
understood by many clients and helps both client
• 17% more successful than physical therapy (a and therapist to decide if the intervention should be
small effect) in decreasing the number of sick implemented (Tickle-Degnen, 1998). In the BESD,
days, the r is used to develop success rates for both the
• 5% more successful than physical therapy (a neg- treatment and control interventions, which can then
ligible effect) in increasing lifting capacity, and be compared (See Table 42.10).
• 29% more successful than physical therapy (a The treatment and control BESD can be com-
moderate effect) in increasing endurance. pared when presenting the results. In considering
the Jousset et al. (2004) article the following state-
If there is a negligible or small (although sig-
ments can be made:
nificant) effect, the case for implementing an inter-
vention is weak since the client will do almost as • Number of sick days: 58.5% of clients improved
well without the intervention. With a moderate or with functional restoration while only 41.5% of
large significant effect, the intervention is probably the clients improved with physical therapy.
worth implementing if the client or target popula- • Lifting capacity: 50.03% of clients improved
tion is similar to the treatment group and the out- with functional restoration while only 49.98% of
come is relevant. the clients improved with physical therapy.
Table 42.10 Calculating the BESD from r for Selected Outcomes from Jousset et al. (2004)
Treatment formula Control formula
(r100) (r100)
Outcome r 50 50 BESDtreatment BESDcontrol
2 2
(.17100) (.17100)
No. of sick days 0.17 50 50 58.5 41.5
2 2
(.05100) (.05100)
Lifting capacity (PILE) 0.05 50 50 50.03 49.98
2 2
(.29100) (.29100)
Endurance 0.29 50 50 64.5 35.5
2 2

• Endurance: 64.5% of clients improved with func- Sackett, Haynes, Guyatt, & Tugwell, 1991; Sackett
tional restoration while only 35.5% of the clients et al., 2000).
improved with physical therapy. The evidence-based practitioner should work
with other practitioners, both in their clinics and
The BESD defines the relative effectiveness of within their therapeutic communities, to access
both the treatment and control groups. The control and assess relevant articles. The evidence devel-
group, whether it has no intervention or some stan- oped in this process should be saved in a
dard intervention, will clinically useful manner
almost always show some so that it can be used
improvement on most Collecting and assessing evi- to guide practice. One
outcomes. The BESD dence should be an ongoing method is to summarize
allows the reader to make the information about
a determination of how
part of effective practice. the article in a critically
successful both the treat- appraised paper (CAP),
ment and control were, and determine if the suc- also referred to as a critically appraised topic
cess rate warrants using the intervention. Using the (CAT) (Suave et al., 1995). The following are stan-
Jousset et al. (2004) research results to guide prac- dard elements of a CAP that should be used when
tice, a clinician might choose functional restora- documenting the results of the process of finding,
tion for clients who are concerned about their selecting, and appraising the evidence (Law,
endurance level, but would be less likely to imple- 2002b):
ment the intervention for clients who are con-
cerned about lifting ability. Since functional • The date of completion: Evidence related to
restoration is considerably more intensive (6 hours intervention can change rapidly. The date that
a day, 5 days a week compared to 3 hours a week review was completed helps determine if the
total), the cost of the functional restoration therapy information is still relevant.
in time, manpower, and money is considerably • The clinical question: The PICO or background
larger than that for the three times a week physical question that was originally used to find the arti-
therapy intervention. Therapists and clients must cle helps structure why the literature was
determine if the extra cost is warranted by the accessed and how it was used.
expected benefit to the client. • The clinical bottom line: The meat of the review,
The effect size also provides an understanding what the evidence suggests concerning the ques-
of the overall strengths and weaknesses of the tion of interest. The clinical bottom line summa-
intervention. Compare the interpretability of these rizes the results of the critical appraisal, reports
effect sizes with the means, standard deviations, pertinent outcomes (e.g., effect sizes), and
and p-values provided in Table 42.6. Since the reports possible clinical applications of the evi-
means are on different scales, it is impossible to dence.
compare and contrast the relative effectiveness of • The article citation: A citation of the article(s)
the intervention for different types of outcomes. used to develop the CAP. When possible a copy
Once effect sizes are calculated, the means are of the original article should be kept with the
standardized, and it is possible to compare the CAP.
results to each other and to other articles. The rel- • Comments: This section provides information on
ative effectiveness of the intervention for improv- important aspects of the evidence that might oth-
ing each type of outcome becomes clear, whether erwise not fit in else where.
the impairment involves attributes such as flexibil-
Table 42.4 is an example of a CAP prepared for
ity or endurance, or a function, such as lifting.
the Jousset et al. (2004) article and provides the
basic format of a CAP.
Putting It All Together: Applying the Evidence
Building Evidence The final step in evidence-based practice is apply-
into Practice ing the evidence. There are numerous ways that
critically appraised evidence can be used in prac-
Documenting the Evidence tice. Evidence can be used to:
Collecting and assessing evidence should be an • Develop practice guidelines (Sackett et al., 2000;
ongoing part of effective practice (Law, 2002c; Tickle-Degnen, 2002),
• Develop economic analyses of different interven- Mr. X, you have had chronic low back pain
tion options (Tickle-Degnen, 2002), for 1 year now. Your physical therapy has
• Evaluate clinical performance (Sackett et al., helped some, but you continue to have
1991), trouble with home activities, and you feel
• Inform consumers of the effectiveness of an that you overall fitness level is low. I would
intervention, and like to suggest a course of therapy in which
• Shape clinicians’ choices of the most appropriate you attend daily therapy lasting 6 hours a
intervention. day. The therapy is designed to improve
your flexibility, endurance, strength, and
Evidence-based practice is not a cookie-cutter work ability. A recent study reported that
guide to intervention (Sackett & Richardson, this type of therapy was superior to a
1996). Current available best evidence must be three times a week physical therapy pro-
used judiciously by skilled practitioners in combi- gram in decreasing sick days, improving
nation with their knowledge of treatment princi- flexibility, endurance, and assisting people
ples and their overall therapeutic skills. Part of this to getting back to leisure and sports activi-
skill is matching potential evidence-based inter- ties. For example, there was a 17% greater
ventions with the needs and values of their clients. decrease in sick days for those people
The ability to understand the needs of clients and who received this type of therapy, a 29%
present evidence so clients can make informed increase in endurance, and a 17% decrease
decisions about their treatment is an important part in pain. In addition, one in five clients in
of evidence-based practice. In addition, practition- this type of intervention report improved
ers should remember that there are “consumers” ability to participate in sports and leisure
other than clients who are interested in the evi- activities.
dence about an intervention. For example, funders
and managers will also be concerned about inter- This type of statement provides the client with
vention options and choices (Tickle-Degnen, information that will help him to make an
2002). Practitioners can use evidence to justify informed decision as to whether the additional
their choices to these groups as well. Clinicians time and effort required to attend the more inten-
should be able to present the results to all three sive program is worth it.
groups, using different language and focus for
each. Tickle-Degnen (2002) suggests that when
communicating the results of evidence, practition-
ers should:
Conclusion
• Use simple, concrete, nontechnical, culturally This chapter provides an overview of some basic
neutral language, methods of applying evidence-based practice to
• Keep the information brief, treatment interventions. Using information from
• Check frequently for confusion or lack of com- this chapter should help practitioners to use the lit-
prehension, and erature to identify, assess, and implement treat-
• Suggest concrete actions related to the informa- ment evidence in their own practice. Importantly,
tion (p. 229). this chapter has not addressed how to use evidence
to evaluate the literature relating to diagnosis,
For example, a client has had several acute prognosis, or economic factors which are other
episodes of low back pain that he states has led to topics of evidence-based practice that can be
decreased participation in work, play, and home incorporated into treatment. There are several
activities. After assessment it is clear that the client excellent texts that cover these important topics
has low flexibility, endurance, and reports high (Ajetunmobi, 2002; Law, 2002c; Sackett et al.,
levels of pain. He has had several courses of 1991, 2000) and the reader is urged to explore the
physical therapy, but continues to have problems. topic of evidence-based practice more fully
The client states that he would like to miss less through these texts and other resources. Moreover,
work, improve his ability to play with his children, this chapter did not address how to use evidence
and improve his overall fitness level. The practi- from research other than studies that involve a con-
tioner working with the client believes that a trol and intervention group. While controlled stud-
course of intensive work related occupational ther- ies represent some of the most valuable sources of
apy will benefit the client, and provides him with information for evidence-based practice, many
the following information to help him make his other forms of research are also valuable. These
decision. include:
• Psychometric studies that provide information therapy: A randomized controlled study. Spine, 29,
about the dependability and utility of assess- 487–494.
Katz, D. L. (2001). Clinical epidemiology & evidence-
ments, based medicine. Thousand Oaks, CA: SAGE
• Qualitative studies that examine client experi- Publications.
ences in intervention, Law, M. (2002a). Appendix C: Critical review form for
• Needs assessment studies that point toward quantitative studies. In M. Law (Ed.), Evidence-based
rehabilitation (pp. 305–308). Thorofare, NJ: Slack.
unmet needs and/or desirable service outcomes,
Law, M. (2002b). Building evidence in practice. In M. Law
and (Ed.), Evidence-based rehabilitation (pp. 185–220).
• Participatory studies that involve consumers and/ Thorofare, NJ: Slack.
or practitioners in identifying needs and in devel- Law, M. (2002c). Evidence-based rehabilitation: A guide
oping and/or evaluating services to meet those to practice. Thorofare: NJ: Slack.
Moher, D., Pham, B., Jones, A., Cook, D. J., Jadad, A. R.,
needs. Moher, M., Tugwell, P., & Klassen, T. P. (1998). Does
quality of reports of randomised trials affect estimates
These and other approaches to research that of intervention efficacy reported in meta-analysis? The
provide evidence useful to practice are discussed Lancet, 352, 609–613.
throughout this text. Moore, A., & McQuay, H. J. (2001). What is an NNT?
Hayward Medical Group. Retrieved, 2004, from the
World Wide Web: http://www.evidence-based-
REFERENCES medicine.co.uk
Ajetunmobi, O. (2002). Making sense of critical appraisal. Moore, A., McQuay, H., & Gray, J. A. M. (1995).
London: Arnold. Evidence-based everything. Bandolier, 1 (12), 1.
Altman, D. G., & Schulz, K. F. (2001). Concealing treat- Newman, T. B., Browner, W. S., & Hulley, S. B. (2001).
ment allocation in randomised trials. British Medical Enhancing causal inferences in observational studies.
Journal, 323, 446–447. In S. B. Hulley, S. R. Cummings, W. S. Browner, D.
American Occupational Therapy Association (2000). Grady, N. Hearst, & T. B. Newman (Eds.), Designing
Occupational therapy code of ethics. American Journal clinical research (2nd ed., pp. 125–142). Philadephia,
of Occupational Therapy, 54, 614–616. PA: Lippincott, Williams, & Wilkins.
Bandolier (2003). Calculating and using NNTs. Retrieved Portney, L. G., & Watkins, M. P. (2000). Foundations of
May 26, 2004, from the World Wide Web: http://www. clinical research: Applications to practice (2nd ed.).
ebandolier.com Upper Saddle River, NJ: Prentice-Hall Health.
Berg, K. E., & Latin, R. W. (1994). Essentials of research Richardson, W. S., Wilson, M. C., Nishikawa, J.,
methods in health, physical education, exercise science, & Hayward, R. S. (1995). The well-built clinical ques-
and recreation (2nd ed.). Philadelphia: Lippincott tion: A key to evidence-based decisions. ACP Journal
Williams & Wilkins. Club, 123, A-12.
Cohen, J. (1988). Statistical power analysis for the behav- Rosenthal, R. (1994). Parametric measures of effect size.
ioral sciences (2nd ed.). Hillsdale, NJ: Lawrence In H. Cooper & L. V. Hedges (Eds.), The handbook of
Erlbaum. research synthesis (pp. 231–260). New York: Russell
Cook, T. D., & Campbell, D. T. (1979). Quasi- Sage Foundation.
experimentation: Design & analysis issues for Rosenthal, R., & Rosnow, R. L. (1991). Essentials of
field settings. Boston: Houghton Mifflin. behavioral research: Methods and data analysis (2nd
Dubouloz, C. J., Egan, M., Vallerand, J., & vonZweck, C. ed.). New York: McGraw-Hill.
(1999). Occupational therapists’ perception of Rosenthal, R., & Rubin, D. B. (1982). A simple general
evidence-based practice. American Journal of purpose display of magnitude of experimental effect.
Occupational Therapy, 53, 445–453. Journal of Educational Psychology, 74, 166–169.
GRADE Working Group. (2004). Grading quality of evi- Sackett, D. L., Haynes, R. B., Guyatt, G. H., & Tugwell, P.
dence and strength of recommendation. British Medical (1991). Clinical epidemiology (2nd ed.). Boston: Little,
Journal. Retrieved September 26, 2004, from http:// Brown and Company.
www.bmj.com Sackett, D. L., & Richardson, W. S. (1996). Evidence-
Herbert, R. D. (2000). How to estimate treatment effects based medicine: What its and what it isn’t. British
from reports of clinical trials. II. Dichotomous out- Medical Journal, 312, 71–72.
comes. Australian Journal of Physiotherapy, 46, Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg,
309–313. W., & Haynes, R. B. (2000). Evidence-based medicine
Hollis, S., & Campbell, F. (1999). What is meant by inten- (2nd ed.). New York: Churchill Livingstone.
tion to treat analysis? Survey of published randomised Schulz, K. F., & Grimes, D. A. (2002). Blinding in ran-
controlled trials. British Medical Journal, 319, domised trials: Hiding who got what. The Lancet, 359,
670–674. 696–700.
Hulley, S. B., Cummings, S. R., Browner, W. S., Grady, D., Sim, J., & Reid, N. (1999). Statistical inference by confi-
Hearst, N., & Newman, T. B. (2001). Designing clini- dence intervals: Issues of interpretation and utilization.
cal research (2nd ed.). Philadelphia: Lippincott, Physical Therapy, 79, 186–195.
Williams & Wilkins. Straus, S. E., & McAlister, F. A. (2000). Evidence-based
Jousset, N., Fanello, S., Bontoux, L., Dubus, V., Billabert, medicine: A commentary on common criticisms.
C., Vielle, B., Roquelaure, Y., Penneau-Fontbonne, Canadian Medical Association Journal, 163, 837–841.
D., & Richard, I. (2004). Effects of functional Suave, S., Lee, H. N., Meade, M. O., Lang, J. B., Faroukh,
restoration versus 3 hours per week physical M., Cook, D. J., et al. (1995). The critically appraised
topic: A practical approach to learning critical Hierarchies of Research Design

appraisal. Annals of the Royal College of Physicians Baker, N. A., & Tickle-Degnen, L. (2001). The effective-
and Surgeons of Canada, 28, 396–398. ness of physical, psychological, and functional inter-
Tickle-Degnen, L. (1998). Communicating with clients ventions in treating clients with multiple sclerosis: A
about treatment outcomes: The use of meta-analytic meta-analysis. American Journal of Occupational
evidence in collaborative treatment planning. The Therapy, 55, 324–331.
American Journal of Occupational Therapy, 52, Centre for Evidence-Based Medicine (2001). Levels
526–530. of evidence and grades of recommendation.
Tickle-Degnen, L. (2001). From the general to the specific: Retrieved January 10, 2002, from the World
Using meta-analytic reports in clinical decision mak- Wide Web: http://www.cebm.net/levels_of_
ing. Evaluation and the Health Professions, 24, evidence.asp
308–326. GRADE Working Group. (2004). Grading quality of
Tickle-Degnen, L. (2002). Communicating evidence to evidence and strength or recommendation. British
clients, managers, and funders. In M. Law (Ed.), Medical Journal, 328, 1–8. Retrieved September
Evidence-based rehabilitation (pp. 221–254). 26, 2004 from the World Wide Web: http://www.
Thorofare, NJ: Slack. bmj.com
Critically Appraising the Evidence
RESOURCES Hill, A., & Spittlehouse, C. (2001). What is critical
General Evidence-Based Practice Resources appraisal? Hayward Medical Group. Retrieved, 2004,
Agency for Healthcare Research and Quality (AHRQ) from the World Wide Web: http://www.evidence-based-
(http://www.ahcpr.gov/)—part of the Department of medicine.co.uk
Health and Human Services, this site provides informa- Khan, K. S., ter Riet, G., Popay, J., Nixon, J., & Kleijnen,
tion about evidence-based treatment as well as clinical J. (2001). Study quality assessment. Retrieved February
guidelines for healthcare professionals. 6, 2004, from the World Wide Web:
Ajetunmobi, O. (2002). Making sense of critical appraisal. http://www.york.ac.uk/inst/crd/pdf/crd4_ph5.pdf
London: Arnold. Law, M. (2002). Appendix C: Critical review form for
Bandolier (http://www.jr2.ox.ac.uk/bandolier/)— quantitative studies. Evidence-based rehabilitation:
Independent journal about evidence-based health A guide to practice (pp. 305–308). Thorofare, NJ:
care. Slack.
British Medical Journal (BMJ)—(http://www.bmj.com/) Maher, C. G., Sherrington, C., Herbert, R. D., Moseley, A.
Centre for Evidence-Based Medicine (http://www.cebm. M., & Elkins, M. (2003). Reliability of the PEDro
net/)—Describes methods of finding, evaluating and Scale for rating quality of randomized controlled trials.
using evidence. Physical Therapy, 83, 713–721.
Cochrane Collaboration (http://www.cochrane.org/ Verhagen A. P., de Vet H. C., de Bie R. A., Kessels A. G.,
index0.htm)—A subscription-based organization that Boers M., Bouter L. M., & Knipschild P. G. (1998).
produces and disseminates systematic reviews of The Delphi list: a criteria list for quality assessment of
healthcare interventions. randomized clinical trials for conducting systematic
OTseeker (www.otseeker.com)—An occupational therapy reviews developed by Delphi consensus. Journal of
systematic evaluation of evidence. Clinical Epidemiology, 51, 1235–1241.
PEDro (http://www.pedro.fhs.usyd.edu.au/index.html) — Calculating Effect Sizes
The Physiotherapy Evidence Database—a physical Bandolier. (2003). Calculating and using NNTs. Retrieved,
therapy systematic evaluation of the evidence. May 26, 2004, from the World Wide Web: www.
Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg, ebandolier.com
W., & Haynes, R. B. (2000). Evidence-based medicine Guyatt G. H., Sackett D. L., & Cook D. J. (1993). User’s
(2nd ed.). New York: Churchill Livingstone. guide to the medical literature: II. How to use an article
Sackett, D. L., Haynes, R. B., Guyatt, G. H., & Tugwell, P. about therapy or prevention: B. What were the results
(1991). Clinical epidemiology (2nd ed.). Boston: Little, and will they help me in caring for my patients?
Brown and Company. Journal of the American Medical Association, 271,
59–63.
Search Methods Herbert, R. D. (2000a). How to estimate treatment
Coster, W., & Vergara, E. (2004). Finding the resources to effects from reports of clinical trials. I: Continuous
support EBP. OT Practice, March 8, 2004, 10–15. outcomes. Australian Journal of Physiotherapy, 46,
Katz, D. L. (2001). Appendix A. In Clinical epidemiology 229–235.
& evidence-based medicine. Thousand Oaks, CA: Herbert, R. D. (2000b). How to estimate treatment effects
SAGE Publications. from reports of clinical trials. II Dichotomous out-
Lou, J. (2002). Searching for the evidence. In M. Law (ed.) comes. Australian Journal of Physiotherapy, 46,
Evidence-based rehabilitation (pp. 71–96). Thorofare, 309–313.
NJ: Slack. Rosenthal, R. (1994). Parametric measures of effect size.
Oxford Centre for Evidence-based Medicine (2001). In H. Cooper & L. V. Hedges (Eds.), The handbook of
Searching for the best evidence in clinical journals. research synthesis (pp. 231–260). New York: Russell
Retrieved January 10, 2002, from the World Wide Web: Sage Foundation.
http://www.cebm.net/searching.asp
White, H. D. (1994). Scientific communication and litera- Disseminating the Evidence
ture retrieval. In H. Cooper & L. V. Hedges (Eds.), The Centre for Evidence-Based Medicine (n.d.). The CATbank.
handbook of research synthesis (pp. 41–56). New York: Retrieved September 26, 2004 from the World Wide
Russell Sage Foundation. Web: http://www.cebm.net/cats.asp
Holm, M. B. (2001). Our mandate for the new millennium: Tickle-Degnen, L. (2001). From the general to the specific:
Evidence-based practice. OT Practice, July 2001, CE- Using meta-analytic reports in clinical decision mak-
1-CE-14. ing. Evaluation and the Health Professions, 24,
Tickle-Degnen, L. (1998). Communicating with clients 308–326.
about treatment outcomes: The use of meta-analytic Tickle-Degnen, L. (2002). Communicating evidence to
evidence in collaborative treatment planning. The clients, managers, and funders. In M. Law (Ed.),
American Journal of Occupational Therapy, 52, Evidence-based rehabilitation: A guide to practice (pp.
526–530. 221–254). Thorofare, NJ: Slack.
C H A P T E R 4 3
Managing Change and Barriers

to Evidence-Based Practice
Annie McCluskey
Previous chapters in this section have described the ological quality (Sackett, Straus, Richardson,
rationale for, and the process of, evidence-based Rosenberg, & Haynes, 2000; Straus, Richardson,
practice. This chapter examines how occupational Glasziou, & Haynes, 2005; Taylor, 2000).
therapists manage change and barriers to evidence- Nonetheless, research evidence is not intended to
based practice. Although the profession produces be used in isolation. Rather, it is combined with
good research, findings are not always used in clinical experience, clinical reasoning, knowledge
practice to improve client outcomes. Furthermore, from formal education, and information about
therapists often have difficulty discriminating client needs and values (Pollock & Rochon, 2002).
between “good” research, and research that is of Healthy debate continues about the limited rele-
poor methodological quality. Although the process vance of randomized controlled trials for practice.
of change can be slow, individuals and organiza- However, on the whole, professions such as occu-
tions need to overcome the research–practice pational therapy now accept the need to base their
gap, and make the change to evidence-based practice and teaching on sound research, much
practice. more so than they did in the past.
First, the nature of change and typical This change from experience-based practice to
responses to change are described in relation to evidence-based practice (and teaching) requires a
evidence-based practice. Next, common barriers to substantial change in skills, knowledge, attitudes
using research are examined. Finally, findings and behavior. Not surprisingly, practitioners and
from a study are used to illustrate how occupa- academics often respond with anxiety to these
tional therapists managed these barriers, what fac- expectations (Dubouloz, Egan, Vallerand, & von
tors helped, and how these therapists started to Zweck, 1999), and the degree of change required.
engage more in the process of evidence-based Therefore, it can be helpful to understand how dif-
practice over an 18-month period. ferent individuals respond to change, and the
stages a person moves through during this journey.
Individual professionals and managers who are
The Change from “Experience- better informed about change can plan ahead, and
be proactive instead of reactive (McCluskey &
Based” to “Evidence-Based” Cusick, 2002).
Practice Different Responses to Change
Until the mid-1990s, it was reasonable for practice As with the general population, therapists respond
to be based primarily on experience, hence the differently to change. Different responses to
term “experience-based” practice (Redmond, change will affect the way in which a profession
1997). When guidelines for clinical reasoning and deals with innovation, including the new emphasis
standards for the education of practitioners were on evidence-based practice. Rogers (1983) identi-
developed, research was not typically the primary fies five categories of individuals, according to
source of information. Research references were how each person responds to innovative ideas:
an optional “extra” in such standards and guide- • Innovators,
lines. • Early adopters,
Today, health professionals are encouraged to • The early majority,
use research evidence explicitly when making • The late majority, and
clinical decisions. Research is also supposed to • Laggards.
guide those who educate future practitioners.
Furthermore, professionals are expected to Innovators are ahead of the majority and make
appraise and classify studies according to method- up the smallest group. They may be isolated and
distrusted, perhaps even envied by colleagues, Contemplation is when individuals begin to

because of their uptake of a new idea such as think about changing their practice. For example, a
evidence-based practice. Early adopters are the colleague’s enthusiasm about a workshop may
respected opinion leaders in a group who typically lead an occupational therapist to think about
express interest in a new idea ahead of others. This change. Evidence-based practice will require sig-
group of opinion leaders may be useful for “mar- nificant effort such as regular visits to the library,
keting” evidence-based practice to others in the the acquisition of new skills such as learning to use
workplace or profession (Effective Health Care, electronic databases, and a commitment to contin-
1999; McCluskey & Cusick, 2002). uing professional development. At this stage of
The early majority refers to people who hold change, therapists within an organization may find
traditional views but will begin to shift their opin- it helpful to identify the pros and cons of evidence-
ions and practice when change becomes based practice.
inevitable. The early majority can help encourage The stage of preparation follows, during which
change in the late majority, or the skeptics within individuals start to learn new skills and knowledge
an organization. Typically, the late majority are in order to support the proposed change. For exam-
reluctant to accept new ideas and practices. ple, therapists will need search and appraisal skills.
Finally, the laggards accept change only when this They will also need to identify strategies for
is forced upon them, and may resist even then. implementing research findings. Therapists at this
stage might, for example, show an interest in
Readiness for Change organizing or attending an in-service on critical
appraisal of research articles.
In addition to different responses to change, there
The stage of action is when a person begins to
are also stages that characterize people confronted
implement new ideas. Therapists who proceed to
with change. The staged model of change
this stage will implement research findings by
(Prochaska & DiClemente, 1982; 1983; Prochaska,
beginning to change their work practices. This
DiClemente, & Norcross, 1992) is often used to
stage usually requires significant behavior change.
help individuals identify their readiness for change.
For instance, adoption of evidence-based practice
This model describes five stages:
can require discontinuation of treatments that have
• Precontemplation, been accepted practice for decades, and exchang-
• Contemplation, ing these for new treatment techniques.
• Preparation, Maintenance is the final stage wherein a per-
• Action, and manent change in behavior occurs. Maintenance
• Maintenance. activities relevant to evidence-based practice
might include regular searches of electronic data-
Precontemplation is when an individual has no bases such as OTseeker (http://www.otseeker.
desire or intention to change, and where there is com). Activities might also include participation in
little reflection on practice. There may be a lack of a monthly journal club (Dingle & Hooper, 2000;
awareness about the need for change. Some occu- Phillips & Glasziou, 2004; Taylor, 2000).
pational therapists have difficulty moving beyond The two final stages, action and maintenance,
this stage. They tend to feel anxious or threatened present the biggest challenge to therapists who are
by new developments such as evidence-based trying to be “evidence-based,” because it is easy to
practice (Dubouloz et al., 1999). New ways of slip back into old habits. Given typical workloads,
thinking and working are required. Routines and most therapists will need encouragement to spend
habits are disturbed by change. Fortunately, many time searching, reading, and appraising research.
practitioners are interested in acquiring new skills Working in pairs or small groups or with a mentor
and knowledge, and move on to the stage of con- may help to maintain motivation (Conroy, 1997).
templation. Presenting the findings of a search to other staff
members may also act as an incentive.
When introducing evidence-based practice, One of the most important aspects of being an
present information about responses to change evidence-based practitioner is anticipating and
and the staged model of change. Encourage planning for new challenges. For example, in a
therapists to reflect on their own response to recent research project (McCluskey, 2004;
evidence-based practice, and which stage of McCluskey & Lovarini, 2005), occupational thera-
change they have reached. pists were recruited with the primary aim of
increasing their skills and knowledge for evidence-
Chapter 43 Managing Change and Barriers to Evidence-Based Practice 687
1999). Therapists typically report that they do not

Most occupational therapists will need support
know how to identify a “good” study from a poorly
and encouragement to spend time searching,
reading, and appraising research. To help main- conducted one, nor do they know how to interpret
tain motivation, work in pairs or small groups, statistics in the results section of a paper (Metcalfe
or with a mentor. Presenting findings to other et al., 2001; Pollock, Legg, Langhorne, & Sellars,
staff can help to consolidate skills and provides 2000). Results of two surveys focusing on these
an opportunity for feedback. barriers are presented in Tables 43.1 and 43.2.
These surveys were conducted between 2000 and
2003 with occupational therapists in Australia
based practice. They were asked to identify their (McCluskey, 2003, 2004).
stage of readiness for change and to discuss their
attitudes to evidence-based practice. Next, they Survey 1, 2000
found a “buddy” or peer to work with, in order to The first survey was conducted with a convenience
promote action and maintenance. They also sample of occupational therapists. All had attended
learned about common barriers to implementing a half-day workshop on evidence-based practice at
evidence-based practice, and developed a plan of an occupational therapy conference in Sydney,
action to address their personal barriers. By explic- Australia. They completed the survey before par-
itly thinking about the change process, they ticipating in the workshop. Of the 85 therapists in
planned ahead, anticipated problems, and put attendance, 64 provided complete data on per-
strategies in place to manage these barriers. ceived barriers. In terms of readiness for change,
they were mostly in the preparation stage. They
were the early majority. Table 43.1 lists the top 10
Barriers to Evidence- barriers they identified. Most respondents identi-
Based Practice fied lack of time and a large workload or caseload
as the major barriers to adopting EBP, followed by
limited searching skills and limited critical
The two primary barriers or reasons reported by
appraisal skills.
health professionals for not using research in prac-
tice are a perceived lack of time for reading and
interpreting research (Closs & Lewin, 1998) and
Survey 2, 2002
lack of skills and knowledge when searching for The second survey was conducted 2 years later,
and appraising research literature (Dubouloz et al., with a different group of Australian occupational
Table 43.1 Perceived Barriers to Adopting Evidence-Based Practice as Reported by Australian

Occupational Therapists in May 2000 (n ⫽ 64)
Top 10 Barriers Reported n %
Lack of time 56 (87.5)
Large caseload 43 (67.2)
Limited searching skills 32 (50.0)
Limited critical appraisal skills 28 (43.7)
Difficulty accessing journals 28 (43.7)
Lack of evidence to support what occupational therapists do 26 (40.6)
Professional isolation 22 (34.4)
Limited resources and funding to support change to EBP 20 (31.2)
Difficulty accessing computer 18 (28.1)
The large volume of published research 16 (25.0)
Note: Participants were asked to choose as many barriers as they wished from the list; therefore the numbers
do not add up to 100%.
Adapted from McCluskey, A. (2003). Occupational therapists report a low level of knowledge, skill and
involvement in evidence-based practice. Australian Occupational Therapy Journal, 50(8) [Table 3]. With
permission.
Table 43.2 Perceived Barriers to Adopting Evidence-Based Practice as Reported by Australian

Occupational Therapists in 2002 (n ⫽ 114)
Pre-workshop Post-workshop Follow-up
Jan 2002 Feb 2002 October 2002
Top 10 Barriers Reported n ⫽ 114 n ⫽ 106 n ⫽ 51
n (%) n (%) n (%)
Lack of time 86 (75%) 100 (94%) 45 (88%)
Large workload/caseload 76 (67%) 79 (75%) 31 (61%)
Limited searching skills 69 (61%) 56 (53%) 12 (24%)
Limited critical appraisal skills 68 (60%) 69 (65%) 21 (41%)
Difficulty accessing journals 51 (45%) 45 (43%) 18 (35%)
The large volume of 33 (29%) 34 (32%) 7 (14%)
published research
Lack of evidence to support 31 (27%) 34 (32%) 18 (35%)
what occupational
therapists do
Professional isolation 24 (21%) 28 (26%) 7 (14%)
Limited resources and funding 23 (20%) 13 (12%) 4 (8%)
to support change to EBP
Difficulty accessing a computer 19 (17%) 15 (14%) 6 (12%)
Note: Participants were asked to choose as many barriers as they wished from the list, therefore the numbers
do not add up to 100%.
Adapted from McCluskey, A. (2004). Increasing the use of research evidence by occupational therapists [Final
report]. Penrith South, Australia: School of Exercise and Health Sciences, University of Western Sydney,
Table 4.1, p. 15. Full copy available in PDF format from http://www.otcats.com [under ‘Project Summary’].
With permission.
therapists. The sample was specially recruited for to make sense of research articles. Searching skills
their interest in EBP and willingness to attend a were seen as less of a barrier than before the work-
2-day workshop and to complete a critically shop, with respondents feeling more confident
appraised topic (CAT) as an assignment. There- about searching databases on their own.
fore, they were a self-selected and motivated Table 43.2 (right column) also indicates that
group. Most were in the preparation or action over time and with practice, a greater percentage of
stages of the change process. Once again, before therapists improved their skills. Ten months later,
attending the workshop most of this sample identi- fewer therapists felt their searching and appraisal
fied key barriers as (See Table 43.2, left column): skills were a barrier to being an evidence-based
lack of time, a large caseload, limited searching practitioner. Interestingly, the majority still identi-
skills, and limited critical appraisal skills. fied lack of time as the primary barrier. None-
Immediately after the 2-day workshop, these theless, many of these therapists had completed a
therapists were surveyed again (see Table 43.2, critically appraised topic and developed their skills.
middle column). More listed lack of time as a bar- As discussed later in this chapter, it was the way in
rier than pre-workshop. By that time, therapists which these therapists managed and reprioritized
were more aware of the work involved in being an their time that was critical to adopting evidence-
evidence-based practitioner, particularly the work based practice.
required to complete a critically appraised topic. In summary, the barriers to adopting evidence-
Similarly, more respondents identified limited crit- based practice are remarkably consistent across
ical appraisal skills as a barrier than preworkshop. groups of occupational therapists, across profes-
While therapists had learned about and practiced sions (Humphris, Littlejohns, Victor, O’Halloran,
critical appraisal during the workshop, they had & Peacock, 2000; Metcalfe et al., 2001) and coun-
become even more aware of skills they still had to tries. Lack of time, a large workload, and limited
learn. For example, they would need to learn about search and appraisal skills are perceived to be the
different research designs and statistical analyses, main problems. Yet little has been done to date to
address these barriers or problems. Furthermore, Table 43.3 Strategies for Adopting
some barriers, such as a perceived lack of time, are Evidence-Based Practice
unlikely to disappear. Therapists are unlikely to be
given, or to find, more time in their day. Instead, Strategies Subcategories
they need to reprioritize their time. Finding time for • Prioritizing activities
evidence-based • Planning ahead
Managing Barriers to
practice
Developing skills • Using evidence
Evidence-Based Practice and knowledge • Teaching EBP to others
• Seeking help
As already noted, some of the occupational thera- Staying focused • Making a commitment
• Being persistent
pists in the Australian study successfully managed
• Being motivated
the barriers and began to adopt evidence-based
practice. During the 8 months post-workshop, their
use of research evidence was monitored. For Adapted from McCluskey, A. (2004). Increasing the
use of research evidence by occupational
example, they were asked how often they con- therapists [Final report]. Penrith South, Australia:
ducted a search or engaged in appraisal. Further, School of Exercise and Health Sciences,
their level of knowledge about evidence-based University of Western Sydney.Table 5.2, page
practice and their skills were measured objectively. 33. Full copy available in PDF format from
These data were used to select a purposive sample. http://www.otcats.com [under ‘Project
Summary’]. With permission.
After 18 months, 10 of the most proactive, knowl-
edgeable, and skilled therapists were interviewed
to ascertain what factors accounted for their suc- therapists interviewed had yet reached this stage.
cess (McCluskey, 2004). Their strategies for success are now described in
more detail.
Strategies for Adopting
Evidence-Based Practice Finding Time for Evidence-Based Practice
The occupational therapists interviewed were The first strategy involved prioritizing activities
more successful than others at managing the pri- and planning ahead. Time was the major barrier to
mary barriers, lack of time and lack of skills. Three engaging in evidence-based practice for all partic-
main strategies, presented in Table 43.3, were used ipants, as indicated by one of the therapists:
to overcome barriers and adopt evidence-based
practice: I’m sure everyone finds time a big issue. It
is very difficult. Clinically, with just seeing
• Finding time for evidence-based practice, the clients here, it’s very busy. And then
• Developing skills and knowledge, and there are always loads of additional proj-
• Staying focused. ects that we’re working on, meetings and
First, participants proactively made time by pri- supervision. So, definitely it is very difficult
oritizing use of research ahead of other tasks for to find the time.
part of their week, and by planning ahead. Second,
To find time, successful therapists had to make
they proactively developed their skills and knowl-
research utilization a priority. They set time aside,
edge upon return to work, by teaching others what
both at work and after hours, for these activities.
they had learned, and getting help when this was
Less successful therapists complained about lack
needed. Third, they stayed focused and committed
of time and did not prioritize work and personal
to evidence-based practice, and found ways to
time for evidence-based practice. Some were not
maintain their motivation.
persistent in maintaining their commitment, partly
These therapists reported, and quantitative
because they and their organization did not place
data confirmed, that new skills and knowledge
a high value on activities such as searching, read-
were acquired relatively quickly as a result of
ing, and appraisal. For instance, one therapist
attending the workshop. However, finding time
observed:
to further develop their skills, and changisng pol-
icy and practice in line with new research was Here, if it doesn’t get…a little old lady out
much more difficult. Implementation took the door and back home, well then [it is not
longer—more than 12 months—and not all of the considered important].
Searching, reading, and critical appraisal were characteristic of therapists in this study who
not always considered an essential part of the started to use research in practice.
occupational therapist’s work in some organi-
zations. When these activities were less valued Developing Skills and Knowledge
than clinical “hands-on” work, therapists felt
Successful therapists developed skills and knowl-
guilty engaging in them at work, as one therapist
edge by proactively:
noted:
• Using evidence,
I found that every month I either had to
• Teaching evidence-based practice to others, and
book over that time for clinical appoint-
• Seeking help when faced with difficulties.
ments to meet the caseload demands…so
that was interesting in itself, my own atti- Lack of skills and knowledge was a barrier to
tude …rather then protecting that time and using evidence for 9 of the 10 participants. They
doing evidence-based work, I kept putting all struggled with critical appraisal and under-
it off. standing statistics in research articles. However,
successful therapists overcame these difficulties by
In some cases, therapists had to spend time out- persisting, practicing, and seeking help. The fol-
side work hours engaged in searching and lowing is one participant’s report of the impor-
appraisal. Private practitioners prioritized billable tance of practice and using newly acquired skills:
work hours ahead of searching and appraisal, since
these activities affected their income. They typi- Makes sense doesn’t it? If you allow your-
cally completed their activities outside of work self time to do something, you’ll get better
hours. Several participants felt that evidence-based at it....The penny eventually
practice had to become part of their routine work dropped…[that] with more practice [my
for it to be sustainable, with a certain number of skills and knowledge had increased].
hours being allocated per week or month; as one
Five of the 10 therapists were actively involved
therapist noted,
in journal clubs or similar research-focused activi-
We’ve got to change our culture and job ties at work, requiring them to regularly use their
descriptions…to include the time…rather skills and knowledge. One therapist noted that:
than it being something you can tack on [We] started the new journal club about a
when you’ve got a free moment. year ago…everyone has a group that
Supportive policies were already in place for [they’re] in. We meet once a month and
some participants. Successful therapists planned pick a topic, and then everyone has a cer-
ahead by booking blocks of time in their schedule. tain task to do in terms of doing the
For instance, one therapist noted: searches, or reading the articles or writing
the summary.
I have autonomy over my work practices
and was able to book in …big chunks of Those who were successful were more likely to
time…I just booked ahead…I just planned be involved in a journal club, partly because of
and booked out my [work] diary [i.e., organizational expectations, and partly because of
schedule]. routine questioning at their work. These therapists
were keen to find and use research, in order to pro-
In summary, finding time for evidence-based vide best practice to their clients. Although most
practice was difficult for all participants. Lack of therapists hoped to change their practice in
time was the major bar- response to research
rier to adopting evidence- evidence, none were yet
based practice. Most Effective time management using this routinely in
struggled to prioritize was a characteristic of thera- practice.
and plan. Successful ther- Teaching evidence-
apists managed their time pists who started to use based practice to others
by prioritizing research- research in practice. helped therapists to con-
related work ahead of solidate and practice
other tasks at certain their new skills, and
times, by scheduling time in advance in their develop confidence in their ability to use evidence.
diary/schedule, and by devoting some time outside As one therapist noted: “It was good doing the in-
work hours. Effective time management was a service…you often learn something better and
practice it more than if you’re just reading it.” The Staying Focused
more successful and active therapists were
expected to educate others in the organization The more successful occupational therapists in the
about searching and appraisal; as one therapist study used the strategy of staying focused. This
observed: “It was …pushed that we had the strategy involved:
responsibility to educate [others].” Other therapists • Making a commitment to evidence-based prac-
who were less active did not encounter the same tice,
expectations, and were less likely to feel they had • Being persistent when barriers were encountered,
the skills to educate others. and
The role of local opinion leader was one that • Being motivated about evidence-based practice.
successful therapists adopted upon their return to
work. For instance, one therapist commented that They changed their work habits and maintained
she had been “dobbed-in [i.e. nominated] to be the the changes in spite of many distractions. Their
evidence-based champion.” These therapists pro- activities were not constant. Instead, successful
vided in-services at work for other staff, and estab- therapists had periods of intense activity, followed
lished journal clubs. by periods of inactivity. However, despite periods
The third way in which successful therapists of inactivity, and barriers encountered along the
developed their skills and knowledge was by way, they did not lose sight of their goals. The first
actively seeking help from others, in person or by step was making a commitment.
phone and e-mail. This help sometimes involved a Making a commitment meant holding oneself
demonstration of searching techniques, or seeking accountable for completing activities, such as
expert advice about statistics. Librarians were a searching and appraisal. One factor that cemented
common source of help and support. Work could commitment was personal or organizational
be delegated to a librarian in some organizations, expectations that a critically appraised topic would
as one therapist explained: be completed. Making a commitment also implied
that using evidence was valued, as noted by one
To save a bit of time in the process…I therapist:
develop the clinical question and mail it
down to the librarian. She’ll do the search I suppose I had…this obligation, having
for me and send up the result. been part of the project…[You] signed up,
and you knew what you were in for. So we
Therapists found it helpful to have a buddy. A needed to finish it. But that was probably a
buddy was someone who worked with participants self-imposed obligation, because all along,
on their critically appraised topic, the project we were aware we could drop out.
assignment. A buddy helped maintain motivation,
shared the work, and sometimes supplied journal Being persistent involved hard work, and con-
articles; this was underscored by one therapist who tinuing despite failures and obstacles. It was easier
observed: for therapists to persist if they were motivated,
committed to using evidence, and had organiza-
I think the buddy system …worked really
tional support. Being motivated meant having the
well, with everyone being motivated ... to
desire and drive to finish the critically appraised
share out the jobs a little bit, bounce ideas
topic. All had been motivated initially to parti-
off each other and motivate each other. To
cipate in the study and the workshop: “I did the
also remind each other when deadlines
2-day workshop and came back very motivated
were coming up and that sort of thing. I
and very keen … and did quite a lot of work into
think that’s a great system and it helps you
my question.” However, as time progressed and
to network a little bit too.
deadlines advanced, motivation diminished for
Successful therapists, with and without bud- some of the participants interviewed. Lack of
dies, located and used experts such as librarians, or motivation was characterized by long periods of
the project outreach support person, who con- inactivity and limited time spent searching or
ducted support visits, answered e-mail questions, appraising evidence, and therefore, limited time
and helped with searching over the telephone. In spent developing or practicing skills.
summary, successful therapists in this study devel- The more successful therapists interviewed
oped skills and knowledge by using evidence in were motivated to continue using evidence because
practice regularly, by teaching evidence-based of comments made by work colleagues, friends,
practice to others, and by seeking help during and managers, and e-mails sent by the outreach
times of difficulty. support person. They were also motivated by meet-
ing deadlines for the project, such as completing If these conditions were present and positive,
and then presenting their critically appraised topic participants were more likely to progress. If these
to others. One therapist stayed focused because her conditions were absent or negative, they acted as
manager showed interest, and asked for e-mail additional barriers to change, and progress was
updates on her critically appraised topic. In sum- slower.
mary, successful therapists stayed focused on
becoming an evidence-based practitioner by mak-
ing a commitment, being persistent, and being Discussion
motivated.
This chapter has discussed barriers that limit the
use of research by occupational therapists, and
Factors and Conditions interfere with the change from “experience-based”
practice to “evidence-based” practice. The chapter
that Helped Occupational also focused on factors that allowed therapists to
Therapists to Change overcome some of these barriers. Research pre-
sented found that occupational therapists who suc-
cessfully engaged in evidence-based practice
In this Australian study, qualitative analysis identi-
reprioritized their time, proactively developed their
fied four factors or conditions that helped thera-
skills and knowledge, and stayed focused
pists to change and adopt evidence-based practice,
on answering one or more clinical questions
or conversely, that limited their progress and pre-
(McCluskey, 2004). They were in control of the
sented additional barriers. These four factors or
change process. They stopped talking about barri-
conditions (Table 43.4) were:
ers, and changed how they worked. They acknowl-
• A personal readiness for change, edged that they were intellectually ready to change
• Personal and organizational expectations that work habits, acquire new skills and knowledge,
they would apply the skills learned and teach and prioritize their time differently. Further
others, research is needed to investigate whether these
• Self-determined deadlines that pushed them experiences and strategies are similar for other
along, and cohorts of occupational therapists.
• Support within the organization, such as comput- Changing how time is spent at work is more
ers and journals, as well as encouragement from likely to occur if a person is ready to change, able
colleagues and managers. to persist, and committed (Davis, 2003). Strategies
Table 43.4 Conditions that Promoted Change to Evidence-Based Practice

Conditions Definition
Readiness for change Time ready, intellectually ready, resource ready, or skill ready. Readiness
to change work habits and allocate time to activities such as searching
and appraisal.
Personal and organizational Personal expectations of achievement. Use of evidence encouraged and
expectations expected by individuals and their organization. Managers and
supervisors were inquiring and interested, and expected new
knowledge to be applied and shared with others in the organization.
Presence of deadlines Intrinsic or extrinsic, negotiable or nonnegotiable, urgent or nonurgent. The
presence of deadlines helped initiate and stimulate further activity
levels; provided direction and focus for participants.
Availability of support Encouragement, physical resources (internet, journals, computer,
databases) financial assistance and work concessions. Support from
managers, organizations, buddies and peers.
Adapted from McCluskey, A. (2004). Increasing the use of research evidence by occupational therapists [Final
report]. Penrith South, Australia: School of Exercise and Health Sciences, University of Western Sydney.
Table 5.3, page 37. Full copy available in PDF format from http://www.otcats.com [under ‘Project Summary’].
With permission.
Developing skills and knowledge for evidence-

Strategies for managing time so that evidence-
based practice is an ongoing process (Rosenberg et
based practice can be implemented include:
al., 1998). The skills and knowledge required
• Using efficient search techniques, include learning about the process of evidence-
• Completing tasks in small manageable based practice, as discussed previously. Initially
segments, therapists need to write a clinical question, a skill
• Setting and negotiating short-term goals, ideally that most therapists learn quickly. Following this
in conjunction with a manager or supervisor,
first step, database search skills need to be devel-
• Recognizing the time and skills required to
prepare a critically appraised topic, and oped, which can be more of a challenge. This sec-
• Avoiding procrastination and extended search- ond step is more of a challenge, demanding focus
ing (both of which can be ways of avoiding the and avoidance of “busy” work (i.e., searching for
critical appraisal stage). and collecting more articles than needed).
Critical appraisal is the third step in the process
of evidence-based practice, a skill that many prac-
titioners find difficult to master (Bryar et al., 2003;
for better time use include prioritizing daily tasks McCluskey, 2003; Oswald & Bateman, 2000).
(Rogak, 1999), making plans (Dobbins & Pettman, Appraisal skills are more challenging to learn than
1998), delegating tasks to allow time for more searching, and therefore need to be thoroughly
important activities (Mancini, 1994), setting aside addressed during the professional education of
blocks of time (Dobbins occupational therapists.
& Pettman, 1998), and Introducing assignments
dividing large over- Anyone learning about early in a student’s educa-
whelming tasks into tion which demand criti-
smaller sections so they
evidence-based practice cal appraisal of published
appear more manage- should seek opportunities research may help in this
able (Kroehnert, 1999). to teach, as well as seek regard. Completion of
Other strategies include critically appraised topics
not postponing tasks help from, others. as a final year assignment
(Mackenzie, 1997). can showcase the skills of
It is also important university graduates (see
to manage behaviors that waste time. Time wasting http://www.otcats.com for examples). However,
activities include procrastination (McGuire, 2003), graduates need to be able to interpret efficacy stud-
lack of self-discipline ies (particularly random-
and inadequate plann- ized controlled trials
ing (Mackenzie, 1997), Searching, reading, and and systematic reviews),
being reactive rather critical appraisal should as well as qualitative re-
than proactive, not set- search and studies that
ting priorities, and poor no longer be considered compare tests and out-
delegation (Dobbins & activities that therapists are come measures.
Pettman, 1998). Based Managers also need
on prior literature and expected to “fit in” when to address the develop-
the recent qualitative there is “spare time.” ment of critical appraisal
study, strategies for man- skills, by supporting
aging time so that evi- graduate therapists to
dence-based practice can implemented include: attend journal clubs and interactive workshops on
evidence-based practice (McCluskey & Cusick,
• Using efficient search techniques, 2002). Local opinion leaders need to be supported
• Completing tasks in small manageable segments, and encouraged to act as role models. Experts
• Setting and negotiating short-term goals, ideally might be called in to facilitate positive first-time
in conjunction with a manager or supervisor, experiences of a journal club (Phillips & Glasziou,
• Recognizing the time and skills required to pre- 2004).
pare a critically appraised topic, and Previous studies have suggested that once skills
• Avoiding procrastination and extended searching and knowledge have been acquired, the time
(both of which can be ways of avoiding the criti- required to complete specialized activities such as
cal appraisal stage). searching databases will decrease, as practitioners
become more efficient (Bennett et al., 2003). reading, and critical appraisal should no longer be
However, skill development and progression considered activities that therapists are expected
requires practice. Outreach support appears to have “fit in” when there is “spare time.”
some effect on skills and knowledge, but regular Therefore, evidence-based practice needs to be
practice is essential to develop and maintain skills. mentioned in business plans, annual reports, orien-
Teaching colleagues about evidence-based tation program documentation, and performance
practice can also help therapists to advance their appraisals (McCluskey & Cusick, 2002). Evidence-
skills and knowledge. Teaching others can facili- based practice needs to be visible, and should be
tate self-learning, and is recognized as an effective considered an important criteria for accreditation
way of improving performance. Teaching occupa- of healthcare organizations. In the end, organiza-
tional therapy students was recently proposed as a tional culture, and how attitudes and values are
strategy for promoting use of research evidence espoused by others, particularly managers, appear
(Craik & Rappolt, 2003). These researchers found to enhance or inhibit the adoption of evidence-
that when experienced occupational therapists based practice.
taught students in clinical practice or on campus,
this teaching role consolidated their skills and ACKNOWLEDGMENT
knowledge. Anyone learning about evidence-based Sally Home and Lauren Thompson contributed to qualita-
practice should seek opportunities to teach, as well tive data collection and analysis as part of their occupa-
as seek help from, others. tional therapy undergraduate honours projects in 2003,
The number of hours in a work day and the size at the University of Western Sydney, Australia. Cate-
gories (stategies and conditions) have been further
of workloads is unlikely to change. Instead, it is
developed and refined for this chapter.
the allocation and prioritization of time, work val-
ues, and routines that need to change. Only then
REFERENCES
will time spent reading, appraising, thinking, and
Bennett, S., Tooth, L., McKenna. K., Rodger, S., Strong, J.,
changing what is actually done in practice be con-
Ziviani, J., Mickan, S., & Gibson, L. (2003).
sidered equally important to direct care and writ- Perceptions of evidence based practice: A survey of
ing reports. In most organizations, a change in the occupational therapists. Australian Occupational
way that time is valued and spent will depend Therapy Journal, 50(1), 13–22.
largely on forward thinking managers. Bryar, R. M., Closs, S. J., Baum, G., Cooke, J., Griffiths,
J., Hostick, T., Kelly, S., Knight, S., Marshall, K., &
The last two stages of evidence-based practice Thompson, D. R. (2003). The Yorkshire barriers proj-
involve implementing findings in practice, and ect: Diagnostic analysis of barriers to research utilisa-
evaluating the outcome with clients locally. As tion. International Journal of Nursing Studies, 40,
noted earlier, these appear to be the most difficult 73–84.
Closs, S. J., & Lewin, B. J. P. (1998). Perceived barriers to
stages to implement, and may require the most sup-
research utilization: A survey of four therapies. British
port. Few studies to date have focused on imple- Journal of Therapy and Rehabilitation, 5(3), 151–155.
mentation, particularly the effect of evidence-based Conroy, M. (1997). ‘Why are you doing that?’ A project to
practice on client outcomes. look for evidence of efficacy within occupational ther-
If health professionals are expected to routinely apy. British Journal of Occupational Therapy, 60,
487–490.
use evidence in practice, it seems reasonable to Craik, J., & Rappolt, S. (2003). Theory of research utiliza-
expect that time will be allocated during work time tion enhancement: A model for occupational therapy.
for searching and appraisal activities. Searching, Canadian Journal of Occupational Therapy, 70,
266–275.
Davis, N. (2003). Stealing time: Time management tech-
niques add hours to each day. Journal of the American
Health Information Management Association, 74(6),
A range of Internet resources and educational 25–28.
Web sites are available for anyone engaged Dingle, J., & Hooper, L. (2000). Establishing a journal
in teaching, and include the OTseeker site club in an occupational therapy service: One service’s
(Occupational Therapy Systematic Evalua- experience. British Journal of Occupational Therapy,
tion of Evidence, at http://www.otseeker.com), 63, 554–556.
CASP (the Critical Appraisal Skills Program Dobbins, R., & Pettman, B. O. (1998). Creating more time.
at http://www.phru.nhs.uk/casp/casp.htm), and Equal Opportunities International, 17(2), 18–27.
Dubouloz, C., Egan, M., Vallerand, J., & von Zweck,
Netting the Evidence (http://www.shef.ac.uk/ C. (1999). Occupational therapists’ perceptions of
scharr/ir/netting/). These sites contain presenta- evidence-based practice. American Journal of
tions and handouts that can be downloaded for Occupational Therapy, 53, 445–453.
teaching purposes. Effective Health Care. (1999). Getting evidence into
practice [Report]. York, England: University of York,
National Health Service Centre for Reviews and Rogers, E. M. (1983). Diffusion of innovations. New York:
Dissemination. Free Press.
Humphris, D., Littlejohns, P., Victor, C., O’Halloran, P., & Rosenberg, W. M., Deeks, J., Lusher, A., Snowball,
Peacock, J. (2000). Implementing evidence-based R., Dooley, G., & Sackett, D. (1998). Improving
practice: Factors that influence the use of research evi- searching skills and evidence retrieval. Journal of
dence by occupational therapists. British Journal of the Royal College of Physicians of London, 32(6),
Occupational Therapy, 63, 516–522. 557–563.
Kroehnert, G. (1999). Taming time and how do you eat an Sackett, D. L., Straus, S. E., Richardson, W. S., Rosenberg,
elephant? Roseville, Australia: McGraw-Hill. W., & Haynes, R. B. (2000). Evidence-based medicine:
Mackenzie, A. (1997). The time trap (3rd ed.). New York: How to practice and teach EBP (2nd ed.). Edinburgh:
American Management Association. Churchill Livingstone.
Mancini, M. (1994). Time management. New York: Strauss, S.E., Richardson, W.S., Glasziou, P., & Haynes,
McGraw-Hill. R.B. (2005). Evidence-based medicine: How to prac-
McCluskey, A. (2003). Occupational therapists report a low tice and teach evidence-based medicine (3rd ed.).
level of knowledge, skill and involvement in evidence- Edinburgh: Elsevier Churchill Livingstone.
based practice. Australian Occupational Therapy Taylor, M. C. (2000). Evidence-based practice for occupa-
Journal, 50(1), 3–12. tional therapists. Oxford: Blackwell Science.
McCluskey, A. (2004). Increasing the use of research evi-
dence by occupational therapists [Final report]. Penrith
South, Australia: School of Exercise and Health RESOURCES
Sciences, University of Western Sydney. Full copy
available in PDF format from http://www.otcats.com
The OTseeker Evidence Database (http://www.
[under ‘Project Summary’]. otseeker.com)
McCluskey, A., & Cusick, A. (2002). Strategies for OTseeker (Occupational Therapy Systematic Evaluation
introducing evidence-based practice and changing of Evidence) is a free Web-based database that supports
clinician behaviour: A manager’s toolbox. Australian evidence-based occupational therapy. Currently, the
Occupational Therapy Journal, 49, 63–70. database contains details of more than 3,000 randomized
McCluskey, A., & Lovarini, M. (2005). Providing educa- controlled trials and systematic reviews relevant to occupa-
tion on evidence-based practice improved knowledge tional therapy. These studies are preappraised and intended
but did not change behaviour: A before and after study. to help occupational therapists evaluate the validity and
BMC Medical Education, 5, 40. interpretability of current research.
McGuire, R. (2003). Successful time management. British The database has been available since March 2003, and
Medical Journal, 327, 117. at the time of writing, had received more than 260,000 vis-
Metcalfe, C., Lewin, R., Wisher, S., Perry, S., Bannigan, its. There is no subscription fee. Funds for development
K., & Moffett, J. (2001). Barriers to implementing the were provided by the Motor Accidents Authority of New
evidence base in four NHS therapies. Physiotherapy, South Wales, Australia, and OT-Australia. Funds are now
87, 433–441. being sought internationally to support ongoing mainte-
Oswald, N., & Bateman, H. (2000). Treating individuals nance, and to enable new studies to be added to the data-
according to evidence: Why do primary care practition- base.
ers do what they do? Journal of Evaluation in Clinical When visiting the site, explore diagnoses and interven-
Practice, 6(2), 139–148. tions relevant to your area of practice. At the time of writ-
Phillips, R. S., & Glasziou, P. (2004). What makes ing (May 2005), there were 470 articles listed under the
evidence-based journal clubs succeed? Evidence- category “gerontology,” 308 under “pediatrics,” 35 under
Based Medicine, 9, 36–37. “intellectual disabilities,” 477 under “mental health,” and
Pollock, A. S., Legg, L., Langhorne, P., & Sellars, C. 449 under “neurology and neuromuscular disorders.”
(2000). Barriers to achieving evidence-based stroke Entries are also categorized by intervention. At the time
rehabilitation. Clinical Rehabilitation, 14, 611–617. of writing there were 47 studies catalogued in OTseeker
Pollock, N., & Rochon, S. (2002). Becoming an evidence- relevant to “case management”; 1051 studies relevant
based practitioner. In M. Law (Ed.), Evidence-based to “consumer education,” 92 studies on “assistive tech-
rehabilitation: A guide to practice (pp. 31–46). nology/adaptive equipment,” and 342 studies on “relax-
Thorofare, NJ: Slack. ation/stress management.”
Prochaska, J. O., & DiClemente, C. C. (1982). Transtheo- The site developers, all from Australia, are Drs. Kryss
retical therapy: Toward a more integrative model of McKenna, Sally Bennett, Tammy Hoffmann, Leigh Tooth
change. Psychotherapy: Theory, Research and Practice, and Professor Jenny Strong from the University of Queens-
19(3), 276–288. land, and Dr. Annie McCluskey from the University of
Prochaska, J., & Diclemente, C. (1983). Stages and Western Sydney.
processes of self-change in smoking: Toward an inte- Additional links are provided to other databases, which
grative model of change. Journal of Consulting and contain “best evidence” on physiotherapy intervention
Clinical Psychology, 51, 390–395. (PEDro or the Physiotherapy Evidence Database, available
Prochaska, J. O., DiClemente, C. C., & Norcross, J. C. at http://www.pedro.fhs.usyd.edu.au/) and rehabilitation
(1992). In search of how people change: Applications following acquired brain impairment (PsycBITE, available
to addictive behaviours. American Psychologist, 47(9), at http://www.psycbite.com).
1102–1114.
Redmond, A. (1997). Evidence-based medicine: A blue- Occupational Therapy Critically
print for effective practice or the emperor’s new Appraised Topics (CATs)
clothes? Podiatry Management, 11, 123–126. A critically appraised topic or CAT is a short summary
Rogak, L. (1999). Smart guide to managing your time. of evidence on a topic of interest, usually focused around a
New York: John Wiley & Sons. clinical question. A CAT is like a less rigorous version of a
systematic review, summarizing and appraising the best Occupational Therapy Critically
research on a topic. Usually more than one study is Appraised Papers (CAPs)
appraised in a CAT. When a single study is appraised as The term critically appraised paper (or CAP) refers to
the “best” available evidence, the outcome is a critically the critical appraisal of an original research study, which
appraised paper (a CAP). CATs and CAPs are a method of may be qualitative or quantitative in design, or appraisal of
collating and disseminating appraisals to colleagues. CATs a systematic review. The aim is to help professionals keep
are increasingly being used as university assignments to up to date with recent advances by appraising and summa-
assess students skills in searching and appraisal. rizing the results of key research.
A free CATs Web site for occupational therapists was The Australian Occupational Therapy Journal was the
developed in 2003 and is located at http://www.otcats.com. first journal to publish CAPs relevant to occupational ther-
At the time of writing (May 2005), the site contained more apy practice, commencing a CAPs Department in June
than 30 CATs and CAPs related to occupational therapy 2003. Other journals that contain CAPs include Evidence-
interventions. The documents can be printed out and/or Based Medicine, Evidence-Based Nursing, and the
saved in PDF format. Students and therapists are invited to Australian Journal of Physiotherapy.
submit their own CAT or CAP, and a template is provided Typically a CAP includes a structured abstract, mean-
on the site for this purpose. The CATs and CAPs are not ing that the original abstract is rewritten to include details
independently peer reviewed, which is one of their limita- of the study methods, outcome measures, results, and sta-
tions. Nonetheless, the site allows therapists and students tistics. Sometimes original authors of a study are contacted
to share their completed appraisals, which may have taken for missing information or data. Comments on the method-
several weeks or months to complete. Gradually, a body of ological quality of the study and implications for practice
critical appraisals is being collected. are written as a text summary. The conclusions reached by
For more information, contact Annie McCluskey, the the appraisers may be the same or different from those
site author (a.mccluskey@uws.edu.au). reached by the original authors.
C H A P T E R 4 4
Creating Outcome Research Relevant

for Evidence-Based Practice
Pimjai Sudsawad
Research Utilization in articles for practice. Several difficulties have been

identified both in nursing and in rehabilitation
Evidence-Based Practice such as:
Evidence-based practice (EBP) is frequently • The lack of someone to help translate findings
defined as the conscientious, explicit, and judi- into practice (Champion & Leach, 1989),
cious use of current best evidence in making • Failure to make clear the implications for prac-
decisions about the care of individual patients tice in the research presentation (Funk, Cham-
(Sackett, Rosenberg, Gray, Haynes, & Richardson, pagne, Tornquist, & Wiese, 1995),
1996). Despite the general consensus on the • Difficulties with how research is communicated
importance of using EBP as a practice approach, in publications (Kajermo, Nordström, Kruse-
there has been a slow uptake of EBP in occupa- brant, & Björvell, 2000),
tional therapy (Bennett et al., 2003; Curtin & • Practitioners’ lack of skills in interpreting
Jaramazovic, 2001; Humphris, Littlejohn, Victor, research evidence (Law & Baum, 1998),
O’Halloran, & Peacock, 2000; Upton & Lewis, • Difficulty understanding statistical analyses as
1998). A number of barriers make it difficult to presented (Lynn & Moore, 1997; Parahoo, 2000),
implement EBP in real practice situations. Diffi- and
culty implementing EBP is certainly not unique to • Perceived lack of relevance, ease of application,
occupational therapy. Other rehabilitation profes- or orientation of the research literature to profes-
sionals such as physical therapy and speech–lan- sional practice (Campbell, 1996; Di Fabio, 1999;
guage pathology also faced the same issue (Meline Dubouloz et al., 1999).
& Paradiso, 2003; Metcalfe et al., 2001; Turner & With such difficulties, it is understandable how
Whitfield, 1997). practitioners would be reluctant to use research
EBP involves using research evidence, when evidence for EBP.
available, as a basis for practice decisions. How- Just because research evidence is generated and
ever, a number of studies with occupational ther- disseminated does not mean that it will be used or
apy practitioners indicate that practitioners prefer that it is usable. Snell (2003) observed that
and continue to use other sources of information researchers tend to define the problems and test the
more often than research evidence when making solutions apart from practitioners, typically speak
practice decisions (Bennett et al, 2003; Dubouloz, a different language, and have a discrepancy in
Egan, Vallerand, & von Zweck, 1999; Sudsawad, perspectives and goals. Nonetheless, most dis-
2004; Sweetland & Craik, 2001). To execute EBP cussions of EBP focus on practitioners who are
effectively, research evidence has to be used by expected to learn the skills essential to using
practitioners. Research articles in professional and research information in its current form. Little
interdisciplinary journals have been the main emphasis has been placed on researchers and their
sources of the most current evidence for occupa- role in the research application and utilization
tional therapy practitioners, particularly studies of process (Lynn & Moore, 1997).
intervention effectiveness. Research evidence that Researchers certainly share responsibility for
relates to the needs of the practice community and bridging the gap between research and practice.
is understandable to the practitioners is conceiv- Researchers need to:
ably more likely to be used that those who do not
have those characteristics. • Examine how research is being created,
Practitioners often report difficulties when • Consider its applicability and usability for prac-
attempting to use information from research tice,
• Be open to a different paradigm of creating The most influential of these elements for the
research evidence, and diffusion of an innovation is the characteristics of
• Produce outcome research that can be used for the innovation itself, which accounts for 49% to
EBP. 87% of the variance in predicting its adoption rate
(Rogers, 1995).
Ottenbacher, Barris, and Van Deusen (1986)
As identified by Rogers (2003), there are five
first raised the issues of research characteristics
characteristics of an innovation that could either
that could create impediments to occupational
facilitate or hinder the adoption of an innovation as
therapy practitioners’ use of research for practice.
delineated below:
They pointed out the problems with using group
statistics to determine treatment effectiveness • Relative advantage: The degree to which an inno-
because even if statistically significant differences vation is perceived as better than the idea it
are found, clinically relevant information cannot supersedes. The degree of relative advantage may
be inferred. They also pointed out the difficulties be measured in many terms (economic, prestige,
in duplicating the intervention strategies reported convenience, satisfaction, etc.), and the greater
in research articles within a clinical environment. the perceived relative advantage of an innova-
They called for both researchers and practitioners tion, the more rapid its rate of adoption will be.
to be concerned with the effectiveness with which • Compatibility: The degree to which an innova-
information generated through research is dissem- tion is perceived as being consistent with the
inated and incorporated into practice. Their obser- existing values, past experiences, and the needs
vations and suggestions are certainly still timely of the potential adopter. An idea that is incom-
today. patible with the values and norms of the social
In the following sections, the Diffusion of Inno- system will not be adopted as rapidly as an inno-
vations theory is introduced as the framework to vation that is compatible.
help identify desirable characteristics of research • Complexity: The degree to which an innova-
evidence that could facilitate its use for EBP. The tion is perceived as difficult to understand and
concepts of social validity, ecological validity, and use. The innovations that are readily compre-
clinical significance are presented as vehicles that hended by most potential adopters will be
can help to create those desirable characteristics adopted more rapidly while the others that are
when applied to the design and implementation perceived as more complicated will be adopted
of outcome research in occupational therapy. The more slowly.
discussion in this chapter specifically pertains • Trialability: The degree to which an innovation
to the creation of clinical outcome research that is may be experimented with on a limited basis. If
intended to guide practice decisions regarding new ideas can be tried on a small scale, it would
treatment interventions. generally be adopted more quickly.
• Observability: The degree to which the results of
an innovation are visible to others. The easier it
Identifying Desirable is for individuals to see the results of an innova-
Characteristics of tion, the more likely they are to adopt.
Research Evidence As shown in Figure 44.1, these characteristics

are factors that influence the extent to which the
innovation will be adopted and used.
Diffusion of Innovations
The Diffusion of Innovations (Rogers, 2003) is a Application of Desirable Characteristics
framework that identifies the process and influenc- of an Innovation to Research Evidence
ing factors in communicating an innovation
Research evidence fits the definition of an innova-
(defined as an idea, practice, or object that is per-
tion because it represents new ideas and/or prac-
ceived as new). This framework has been used
tice. Therefore, the Diffusion of Innovations
widely in fields such as business, marketing, and
framework can be used to examine characteristics
public health. There are four main elements in a
that, if present, would likely make research evi-
diffusion of an innovation:
dence be received more favorably by its potential
• The innovation, users. In this case, the potential users of interest are
• The communication channels, occupational therapy practitioners who would like
• Time, and to use research evidence as a basis for decision-
• The social systems. making in practice. The Diffusion of Innovations
Chapter 44 Creating Outcome Research for Evidence-Based Practice 699
Compatibility: The degree to

which an innovation is perceived
as being consistent with the
Relative existing values, past experiences, Complexity:
advantage: and the needs. The degree to
The degree which an
to which an innovation is
innovation is perceived as
perceived as difficult to
better than understand
the idea it Facilitate or and use.
supercedes. hinder the
adoption of an
innovation
Observability: The Trialability: The

degree to which degree to which an
the results of an innovation may be
innovation are experimented with
visible to others. on a limited basis.
Figure 44.1 Characteristics of an innovation influencing its adoption.
framework can be applied to research characteris- riences, and the needs of practitioners and/or
tics as indicated below. occupational therapy consumers to increase com-
patibility.
• First, research must be perceived as containing • Third, research must be presented in ways that
stronger evidence than other sources so that it is make it easy to understand and use in order to
viewed as having a relative advantage over other reduce complexity.
kinds of evidence. • Fourth, the intervention investigated must be eas-
• Second, the design and conduct of research must ily implemented in the clinical setting to increase
be consistent with the existing values, past expe- the level of trialability.
Compatibility: The design and

conduct of research is consistent
with the existing values, past
Relative experiences, and the needs of Complexity:
advantage: practitioners and/or occupational Research is
Research is therapy consumers. presented in
perceived as ways that
containing make it easy
stronger to understand
evidence and use.
than other Practitioners'
sources. use of
Research for
Evidence-Based
Practice
Observability: Trialability: The

Research outcomes intervention
demonstrate changes/ investigated is
improvement that is easily implemented
obvious to anyone, in the clinical
including practitioners. setting.
Figure 44.2 Characteristics of outcome research influencing practi-

tioner’s use of research for evidence-based practice.
• Last, the research outcomes must demonstrate Social Validity

changes/improvement with a magnitude that is
obvious to anyone, including the practitioners, to Wolf (1978), who first proposed the concept of
increase the observability. social validity, referred to it as something of social
importance. According to Wolf (1978) researchers
Based on the Diffusion of Innovations frame- should ask three questions to assess the social
work, these characteristics can be conceptualized validity of outcome research concerning goals,
as factors that influence the extent to which practi- procedures, and outcomes:
tioners will use research findings for EBP, as
shown in Figure 44.2. • Goals: Are the goals of the intervention being
Initial findings suggest that practitioners accept investigated really what society wants?
research evidence as stronger than other kinds • Procedures: Are the intervention techniques used
of evidence (Sudsawad, 2004). It appears that acceptable to the consumers, or do they cost too
occupational therapy researchers should place an much (e.g., in terms of effort, time, discomfort,
immediate focus on other characteristics to ensure ethics, or the like)?
that occupational therapy outcome research is • Outcomes: Are the consumers satisfied with the
produced such that it is most relevant to practice. intervention outcome, both with predicted
Considering the scarcity of the resources available change and with unpredicted side effects?
for occupational therapy research, it seems unwise The terms “society” and “consumers” include
to use those resources to create outcome research anyone who may be involved with and affected by
that does not lend itself to be used in practice. the intervention process and outcome including the
Creating outcome research that is usable for prac- occupational therapy clients, their caregivers, their
tice is one of the most important contributions parents and teachers (in the case of children), com-
occupational therapy researchers can make toward munity members, disability groups, and others.
evidence-based practice. There are a variety of methods for operational-
izing the social validity concept (e.g., Foster &
Mash, 1999; Hawkins, 1991; Kazdin & Matson,
Concepts and Methods 1981; Kendall & Grove, 1988; Schwartz & Baer,
1991). However, this discussion focuses on verifi-
to Fulfill the Desirable cation with consumers as a method for establishing
Characteristics of social validity.
Schwartz and Baer (1991) identified four dif-
Research Evidence ferent types of consumers who can be approached
to determine social validity:
In the following sections, the concepts of social • Direct consumers,
validity, ecological validity, and clinical sign- • Indirect consumers,
ificance are introduced and their methods • Members of the immediate community, and
discussed in relation to the five desirable charac- • Members of the extended community.
teristics of research information as delineated
earlier. Researchers can use these concepts as Direct consumers are the primary recipients of
guides to creating research that is likely to be used the intervention. Indirect consumers are those who
in EBP. purchase the intervention
These concepts should for someone else or are
be used as a supplement Creating outcome research strongly affected by the
to the usual consideration behaviors change tar-
of rigor in the design that is usable for practice is geted in the intervention,
and conduct of outcome one of the most important but they are not its recip-
research because scien- ients. Members of the
tific rigor in and of itself
contributions occupational immediate community
is not sufficient to make therapy researchers can are those who interact
research evidence usable with the direct and indi-
in practice. It is essential
make toward evidence- rect consumers on a regu-
that occupational therapy based practice. lar basis, usually through
investigators also con- close proximity during
sider other characteristics work, school, or social
to make outcome research maximally relevant for situations. The members of the extended commu-
its use in EBP. nity include those who probably do not know or
Verifying the Relevance of Research Verifying Social Appropriateness

Questions/Treatment Goals of Intervention Procedures
The following is an example of verifying the rel- The following is an example of verifying the
evance of research questions/treatment goals. social appropriateness of the intervention proce-
O’Brien et al. (2000) conducted a study to inves- dures and social importance of the outcomes.
tigate the effectiveness of an occupational ther- Schilling, Washington, Billingsley, and Deitz
apy intervention designed to improve playfulness (2003) conducted a study to investigate the
in children. In this study, information obtained effects of using therapy balls as seating on in-
from initial home visits and parents’ interview seat behaviors and legible word productivity
were used to determine the area of concern for of students with attention deficit hyperactivity
each child and set up the treatment goal (rather disorder (ADHD). They used the social validity
than determining the goal based on the methods to verify both the appropriateness of
researchers’ own opinion). the intervention and the importance of the out-
come. The researchers asked teachers, students
in the class who were classmates of the study’s
interact with the direct and indirect consumers but participants (and also using therapy balls as
seating in the classroom at the same time as the
who live in the same community. The point of this
study’s participants), and the participants them-
delineation is that a few different groups of con- selves to complete a questionnaire to determine
sumers are available to socially validate the goals, whether they observed improvement in perform-
procedures, and outcomes. To strengthen the social ance in the actual classroom setting. The teacher
validity, more than one consumer group’s opinion answered questions concerning whether she saw
and perspective may be sought at the same time. the improvement in students’ performance. The
Social validity can be implemented from the students answered questions about whether they
initial conception of the research to the conclusion saw improvement of their own performance. To
of the study. Occupational therapy researchers can verify the acceptability of the intervention, stu-
seek the appropriate study topic/research questions dents were also asked whether they preferred
using therapy balls than sitting on chairs, and
(goals) by seeking to learn about the topics of
were given an opportunity to put in writing
interest to practitioners and/or what type of evi- their opinion of sitting on balls in the classroom.
dence is needed in practice situations. In addition,
researchers can include occupational therapy
clients to help identify appropriate treatment goals
(Feature Box 44.1). treatment goals, intervention procedures, and
A study’s intervention (procedures) can be intervention outcomes when conducting research
developed such that the reality of practice is taken studies should increase compatibility—the degree
into consideration. Then, if the intervention stud- to which research information is perceived as
ied is found to be effective, practitioners can use it. being consistent with the existing values, past
The acceptability of the intervention can also be experiences, and the needs of the occupational
verified with service consumers because if con- therapy practitioners—either through addressing
sumers found the intervention unacceptable, they the consumers’ needs (which is the matter of inter-
may choose not to receive or cooperate with the est to practitioners) or taking into consideration the
intervention. practitioners’ input based on actual practice situa-
Before the intervention, researchers can verify tions, or both. In addition, investigating interven-
with the clients that the intervention outcomes tion methods that are realistic for implementation
(outcomes) chosen are important outcomes from in practice settings and acceptable by consumers
their perspective. The social validity of outcomes should increase trialability—the degree to which
can also be verified after the intervention by asking information obtained from research studies can be
clients whether they are satisfied with the out- implemented in practice settings.
comes obtained. Practitioners can also help to
identify the types of outcomes that may be relevant Ecological Validity
to their clients.
Ecological validity is related to the social impor-
Relationship Between Social Validity tance of research outcomes; it is the degree to
and Desirable Characteristics of which results obtained in controlled experimental
conditions are related to those obtained in natura-
Research Evidence
listic environments (Tupper & Cicerone, 1990).
Involving both the consumers and the practitioners What a person can do in the artificial experimental
in the process of determining research questions/ environment is not necessarily what the person
Assessing Ecological Validity should have some evidence of ecological validity.

Better yet, real-life performance should be used as
Moseley and colleagues (2004) assessed the eco- a basis to demonstrate the intervention effective-
logical validity of walking speed measurement ness in natural settings whenever possible because
after a traumatic brain injury (TBI) of three clini- there is no guarantee that a test (or a number of
cal gait tests in predicting walking performance tests) will predict performance in a natural setting.
in three natural environments (a corridor in a
brain injury rehabilitation unit, a car park of a Relationship of Ecological Validity
metropolitan shopping center, inside a metropoli-
tan shopping center). They found that for partici- to the Desirable Characteristics
pants with TBI, the agreements between the of Research Evidence
speed used in the clinical gait tests and the natu-
ral environments was poor. Sudsawad, Trombly,
Choosing study outcomes that represent actual
Henderson, and Tickle-Degnen (2001) assessed performance in real-life settings can positively
the ecological validity of a standardized hand- contribute to the use of research evidence for prac-
writing assessment and found that the level of tice. Demonstrating the effect of intervention
handwriting performance of children with various in terms of ability or performance in a natural con-
degrees of handwriting illegibility bore almost no text will help to increase compatibility of research
relationship with the children’s performance in information for occupational therapy practitioners
the classroom setting as rated by teachers.
Incorporating Real-Life Performance

does or can do in his or her everyday environment. into Outcome Studies
Similarly, the interaction between persons’ abili-
ties (or disability) and unique environmen- The following studies are examples of how real-
tal demands will yield different performances life performance can be incorporated into out-
(Sbordone, 1996). come studies. In a study of the dose–response
Choosing an ecologically valid outcome also effects of a medication for ADHD, Evans et al.
(2001) used everyday tasks in normal classroom
partly relates to the social validity aspect of the
activities as dependent measures including
outcome. Not only should the outcome measure notetaking quality, quiz and worksheet perform-
represent performance in real-life environment, it ance, written language usage and productivity,
should also reflect the performance that the clients, teacher ratings on task and disruptive behavior,
their significant others, or their peers consider to and homework completion. The researchers
be important, and with which they are satisfied believed that the measures of behavior and aca-
(Feature Box 44.3). demic performance are needed to assess medica-
Another aspect of ecological validity relates to tion response, rather than using laboratory tasks
the standardized test instruments used for assess- as proxy measures of academic functioning, as
ment of outcomes. Franzen and Wilhelm (1996) past research had showed little correspondence
between performance on such tasks and class-
indicated two general aspects of ecological valid-
room academic performance.
ity in assessment: In the study comparing the effectiveness of
• Verisimilitude, which is the similarity of the data three interventions for stuttering plus a control
collection methods in the test to tasks and skills group, Craig et al. (1996) measured the partici-
required in the free and open environment, and pants’ stuttering frequency and speech rate not
only during conversations in the clinic with the
• Veridicality, which is the extent to which test
clinician, but also in two other relevant contexts
results reflect or can predict phenomena in the including performance during a telephone con-
open environment or “real world.” versation with a family member or friend, and in
Verisimilitude will be important in the design face-to-face conversation with a family member
of assessment instruments, but once an instrument or friend in the home environment.
In another study investigating the effective-
is designed, veridicality becomes more important
ness of oral–motor sensory treatment Gisel,
(Franzen & Wilhelm, 1996). To achieve ecological Applegate-Ferrante, Benson, and Bosma (1996)
validity, a test must provide information that is rel- measured oral–motor skills through the adminis-
evant to the person’s functioning in daily life, not tration of a standardized feeding assessment dur-
simply be representative of a hypothetical con- ing observations at lunch/snack time in the
struct or even a neurological syndrome (Silver, children’s accustomed room with the person who
2000). If a standardized test is to be used as a was the child’s regular feeder.
measurement instrument in outcome research, it
because such outcomes reflect the profession’s phi- Some of the methods to demonstrate clinical
losophy of enabling clients to be able to engage in significance include:
an occupation. Presenting the outcome in everyday
terms should also reduce the perceived complexity • The use of effect sizes to demonstrate the magni-
of the information drawn from research evidence tude of change,
because such outcomes can be easily related and • Using measures of risk potency such as a odds
understood. Finally, the results of an intervention ratio, risk ratio, relative risk reduction, risk dif-
are more likely to be perceived as easily observed ference, and number needed to treat (NNT),
since outcomes were something that can be readily • Comparing both group and individual perform-
seen by normal observation as opposed to the ance to normative data (Jacobson & Traux, 1991;
abstraction of test scores; hence, increase observ- Kazdin, 1977; Kendall & Grove, 1988),
ability of research evidence. • Using meta-analysis for a pooled effect size from
several studies that is compared with normative
Clinical Significance data (Nietzel & Trull, 1998),
• Showing that the studied intervention eliminates
Clinical significance refers to the practical value or symptoms/impairments that the intervention is
the importance of the effect of an intervention, that intended to eliminate,
is, whether it makes a real difference in everyday • Demonstrating that the researched intervention
life to the clients, or to others with whom the enables clients to meet role demands, increases
clients interact (Kazdin, 1999). Statistical signifi- the level of functioning in everyday life, and/or
cance, while necessary to verify that any changes achieves change in quality of life,
observed in a study are not likely due to chance, • Incorporating subjective judgments of change by
does not ensure that such changes are meaningful the clients, their significant others, or by people
for clients’ real-life performance. Therefore, statis- who interact with the clients in their natural envi-
tical significance is an insufficient index for the ronments, and
usefulness of an intervention to improve everyday • Documenting satisfaction with the treatment
function. The fact that a treatment effect was results, including input obtained from the client,
probabilistically not “due to chance” is not ade- his or her significant others or even professionals
quately informative to the practice community who are not part of the research team (e.g. a
(Saunders, Howard, & Newman, 1988). Not sur- child’s regular occupational therapist, teacher,
prisingly, agencies and policymakers increasingly classroom assistant).
demand evidence about real-world effects of inter-
ventions in order to invest resources in them Using statistical methods to determine clinical
(Czaja & Schulz, 2003). Nonetheless, the concept significance (the first four bullet points above) is
of clinical significance has not been implemented important to improve the quality of evidence.
consistently in rehabilitation outcome research These methods, however, do not provide informa-
(Sudsawad, 2004) or in medical research (Chan, tion on the impact of the intervention on the
Man-Son-Hing, Molnar, client’s everyday life
& Laupacis, 2001). and they generate infor-
Authors have pre- Statistical significance, while mation at a level of
sented different ways to necessary to verify that any abstraction that does
demonstrate the clinical not correspond to the
significance of treatment changes observed in a study everyday world of prac-
outcomes including both are not likely due to chance, tice. Focusing on elimi-
statistical and nonstatis- nation of symptoms/
tical methods. The more
does not ensure that such impairments or norma-
suitable method that changes are meaningful for tive comparison, while
researchers can use to clients’ real-life performance. useful in some contexts,
demonstrate clinical sig- may not be applicable to
nificance for their study many areas of occupa-
will depend on many factors such as: tional therapy services. Often, the goal of occupa-
tional therapy intervention is not to eliminate
• The study area, symptoms or to promote the client’s performance
• The study sample, to that of the norms, but rather to maximize the
• The nature of the treated condition, and client’s ability to participate in everyday living as
• The nature of the expected treatment outcomes. much as possible.
Making use of client (and others with whom the • Is the topic of the study what the practitioners/
client is associated) input on the client’s perform- service consumers are interested in or need to
ance in natural contexts is a comprehensive know about?
approach. For example, assessing a client’s satis- • Is the intervention investigated in the study prac-
faction with change of his or her performance in tical enough to be implemented in practice set-
everyday activities meets the criteria for social tings if found to be effective?
validity, ecological validity, and clinical signifi- • Is the intervention procedure acceptable to the
cance. This strategy includes the consumer’s per- practitioners/service consumers?
spective and opinion, measures real-life everyday • Is the treatment outcome to be measured in the
function, and ensures that the change achieved is study what practitioners and/or consumers con-
substantial such that it is satisfactory to the client. sider to be an important outcome?
On the other hand, if the increased functioning in • Does the intervention outcome represent per-
everyday life is based only on the researcher’s formance of daily activities in natural contexts?
observation, it may meet the criteria of ecological • Does the method chosen to measure change/
validity and clinical significance, but may not have improvement demonstrate an impact on or
equally strong social validity. make a difference in the client’s everyday life
function?
Relationship of Clinical Significance The use of social validity, ecological validity,
to Desirable Characteristics of and clinical significance concepts simultaneously
Research Evidence to design and conduct outcome research, aimed to
Applying the concept of clinical significance to increase its usability for practice, was termed the
demonstrate a meaningful change in everyday Social Validation Model in a recent investigation
function can increase the compatibility of research (Sudsawad, 2004). Based on responses received
information with the ultimate goal of occupational from more than 900 practitioners in occupational
therapy intervention, which is to positively impact therapy, physical therapy, and speech–language
clients’ everyday lives. Demonstrating change in pathology, there was a strong support for the use of
ways that are related to everyday functions can the Social Validation Model in rehabilitation out-
also help to decrease the complexity of research come research. The majority of practitioners indi-
information from practitioners’ perspectives. They cated that it was either important or very important
can relate to this type of outcome as evidence of that research evidence possesses the key elements
treatment effectiveness more readily than to statis- included in the questions above. They also indi-
tical significance. Clinical significance of treat- cated that doing so would either likely or very
ment outcomes that demonstrate an impact on the likely increase their use of research information
study participants’ everyday life functioning also for practice.
increases the observability of findings.
Conclusion
Application of the Diffusion It is imperative that occupational therapy
of Innovations Framework researchers create research evidence that not only
meets the standards of scientific rigor but also has
in Occupational Therapy utility in everyday practice. Moreover, in the cur-
rent climate of limited available resources, occu-
Occupational therapy researchers can use the pational therapy researchers are well advised to
Diffusion of Innovations framework as a guide, produce outcome research that is relevant and
and the social validity, ecological validity, and applicable to occupational therapy practice. By
clinical significance as tools, to create outcome doing so, researchers will make an important con-
research that is appealing to practitioners. This tribution to the progression of evidence-based
approach has potential to bridge the gap between practice.
research and practice because it emphasizes
designing, conducting, and reporting research with
REFERENCES
the ultimate goal of creating information that is Bennett, S., Tooth, L., McKenna, K., Rodger, S., Strong, J.,
Ziviani, J., Mickan, S., & Gibson, L. (2003).
actually related to and usable in real-life practice. Perceptions of evidence based practice: A survey of
The following questions can be used to guide the Australian occupational therapists. Australian
design and implementation of outcome studies: Occupational Therapy Journal, 50, 13–22.
Campbell, T. W. (1996). Systemic therapy practice and Björvell, H. (2000). Perceptions of research utilization:
basic research. Journal of Systemic Therapy Practice, Comparisons between healthcare professionals, nursing
15(3), 15–39. students and a reference group of nurse clinicians.
Champion, V. L., & Leach, A. (1989). Variables related to Journal of Advanced Nursing, 31, 99–109.
research utilization in nursing: An empirical investiga- Kazdin, A. E. (1977). Assessing the clinical or applied
tion. Journal of Advanced Nursing, 14, 705–710. importance of behavior change to social validation.
Chan, K. B. Y., Man-Son-Hing, M., Molnar, F. J., & Behavior Modification, 1, 427–452.
Laupacis, A. (2001). How well is the clinical impor- Kazdin, A. E. (1999). The meanings and measurement of
tance of study results reported? An assessment of ran- clinical significance. Journal of Consulting and
domized controlled trials. Canadian Medical Clinical Psychology, 67, 332–339.
Association Journal, 165(9), 1197–1202. Kazdin, A. E., & Matson, J. L. (1981). Social validation in
Craig, A., Hancock, K., Chang, E., McCready, C., Shepley, mental retardation. Applied Research in Mental
A., McCaul, A., Costello, D., Harding, S., Kehren, R., Retardation, 2, 39–53.
Masel, C, & Reilly, K. (1996). A controlled clinical Kendall, P. C., & Grove, W. M. (1988). Normative compar-
trial for stuttering in persons aged 9 to 14 years. isons in therapy outcome. Behavioral Assessment, 10,
Journal of Speech and Hearing Research, 39, 808–826. 147–158.
Curtin, M., & Jaramazovic, E. (2001). Occupational thera- Law, M., & Baum, C. (1998). Evidence-based practice
pists’ views and perceptions of evidence-based practice. occupational therapy. Canadian Journal of
British Journal of Occupational Therapy, 64, 214–222. Occupational Therapy, 65, 131–135.
Czaja, S. J., & Schulz, R. (2003). Does the treatment make Lynn, M. R., & Moore, K. (1997). Research utilization by
a real difference? The measurement of clinical signifi- nurse managers: Current practices and future direc-
cance. (Translating psychosocial research into practice: tions. Seminars for Nurse Managers, 5(4), 217–223.
Methodological issues). Alzheimer’s Care Quarterly, Meline, T., & Paradiso, T. (2003). Evidence-based practice
4(3), 229–240. in schools: Evaluating research and reducing barriers.
Di Fabio, R. P. (1999). Myth of evidence-based practice. Language, Speech & Hearing Services in Schools, 34,
Journal of Orthopaedic & Sports Physical Therapy, 29, 273–283.
632–634. Metcalfe, C., Lewin, R., Wisher, S., Perry, S., Bannigan,
Dubouloz, C-J., Egan, M., Vallerand, J., & von Zweck, C. K., & Moffett, J. K. (2001). Barriers to implementing
(1999). Occupational therapists’ perceptions of evi- the evidence based in for NHS therapies.
dence-based practice, American Journal of Physiotherapy, 11, 433–441.
Occupational Therapy, 53, 445–453. Moseley, A. M., Lanzarone, S., Bosman, J. M., van Loo,
Evans, S. W., Pelham, W. E., Smith, B. H., Bukstein, O., M. A., de Bie, R. A., & Hassett, L. (2004). Ecological
Gnagy, E. M., Greiner, A. R., Altenderfer, L., & Baron- validity of walking speed assessment after traumatic
Myak, C. (2001). Dose-response effects of brain injury: A pilot study. Journal of Head Trauma
Methylphenidate on ecologically valid measures of aca- Rehabilitation, 19, 341–348.
demic performance and classroom behavior in adoles- Nietzel, M. T., & Trull, T. J. (1998). Meta-analytic
cence with ADHD. Experimental and Clinical approaches to social comparisons: A method for meas-
Psychopharmacology, 9, 163–175. uring clinical significance. Behavioral Assessment, 10,
Foster, S. L., & Mash, E. J. (1999). Assessing social valid- 159–169.
ity in clinical treatment research: Issues and proce- O’Brien, J., Coker, P., Lynn, R., Suppinger, R., Paerigen,
dures. Journal of Consulting and Clinical Psychology, T., Rabon, S., St. Aubin, M., & Ward, A. T. (2000). The
67, 308–319. impact of occupational therapy on a child’s playfulness.
Franzen, M. D., & Wilhelm, K. L. (1996). Conceptual Occupational Therapy in Health Care, 12(2/3), 39–51.
foundations of ecological validity in neuropsychologi- Ottenbacher, K. J., Barris, R., & Van Deusen, J. (1986).
cal assessment. In R. J. Sbordone & C. J. Long (Eds.), Issues related to research utilization in occupational
Ecological validity of neuropsychological testing. therapy. The American Journal of Occupational
Delray Beach, FL: GR Press/St. Lucie Press. Therapy, 40, 111–116.
Funk, S. G., Champagne, M. T., Tornquist, E. M. & Wiese, Parahoo, K. (2000). Barriers to, and facilitators of, research
R. A. (1995). Administrators’ views on barriers to utilization among nurses in Northern Ireland. Journal
research utilization. Applied Nursing Research, 8, of Advanced Nursing, 31, 89–98.
44–49. Rogers, E. M. (1995). Diffusion of innovations (4th ed.).
Gisel, E. G., Applegate-Ferrante, T., Benson, J., & Bosma, New York: The Free Press.
J. F. (1996). Oral-motor skills follow a sensory motor Rogers, E. M. (2003). Diffusion of innovations (5th ed.).
therapy in two groups of moderately dysphagic chil- New York: The Free Press.
dren with cerebral palsy. Dysphagia, 11, 59–71. Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M.,
Hawkins, R. P. (1991). Is social validity what we are inter- Haynes, R., & Richardson, W. S. (1996). Evidence
ested in? Argument for a functional approach. Journal based medicine: What it is and what it isn’t? British
of Applied Behavior Analysis, 24, 205–213. Medical Journal, 312, 71–72.
Humphris, D., Littlejohn, P., Victor, C, O’Halloran, P., & Saunders, S. M., Howard, K. I., & Newman, F. L. (1988).
Peacock, J. (2000). Implementing evidence-based prac- Evaluating the clinical significance of treatment effects:
tice: Factors that influence the use of research evidence Norms and normality. Behavioral Assessment, 10,
by occupational therapists. British Journal of 207–218.
Occupational Therapy, 63, 516–522. Sbordone, R. J. (1996). Ecological validity: Critical Issues
Jacobson, N. S., & Traux, P. (1991). Clinical Significance: for the neuropsychologist. In R. J. Sbordone & C. J.
A statistical approach to defining meaningful change in Long (Eds.), Ecological validity of neuropsychological
psychotherapy research. Journal of Consulting and testing. Delray Beach, FL: GR Press/St. Lucie Press.
Clinical Psychology, 59, 12–19. Schilling, D. L., Washington, K., Billingsley, F. F., &
Kajermo, K. N., Nordström, G., Krusebrant, Å., & Deitz, J. (2003). Classroom seating for children with
attention deficit hyperactivity disorder: Therapy balls Cicerone, The neuropsychology of everyday life:
versus chairs. The American Journal of Occupational Assessment and basic competencies. Boston: Kluwer
Therapy, 57, 534–541. Academic.
Schwartz, I. S., & Baer, D. M. (1991). Social validity Turner, P., & Whitfield, T. W. A. (1997). Physiotherapists’
assessments: Is current practice state of the art? Journal use of evidence based practice: A cross-national study.
of Applied Behavior Analysis, 24, 189–204. Physiotherapy Research International, 2, 17–29.
Silver, C. H. (2000). Ecological validity of neuropsycho- Upton, D., & Lewis, B. (1998). Clinical effectiveness and
logical assessment in childhood traumatic brain injury. EBP: Design of a questionnaire. British Journal of
Journal of Head Trauma Rehabilitation, 15, 973–988. Therapy and Rehabilitation, 5, 647–650.
Snell, M. E. (2003). Applying research to practice: The Wolf, M. M. (1978). Social validity: The case for subjec-
more pervasive problem? Research & Practice for tive measurement or how applied behavior analysis is
Persons with Severe Disabilities, 28(3), 143–147. finding its heart. Journal of Applied Behavior Analysis,
Sudsawad, P. (2004). Developing a social validation model 11, 203–214.
for effective utilization of disability and rehabilitation
research. Project summary. Submitted to the National
Institute of Disability and Rehabilitation Research, US RESOURCES
Department of Education, Grant # H133F020023. Kazdin, A. E. (1999). The meanings and measurement of
Sudsawad, P., Trombly, C. A., Henderson, A., & Tickle- clinical significance. Journal of Consulting and
Degnen, L. (2001). The relationship between the Clinical Psychology, 67, 332–339.
Evaluation of Children’s Handwriting (ETCH) and Rogers, E. M. (2003). Diffusion of Innovations (5th ed.).
teachers’ perception of handwriting legibility. The New York: The Free Press.
American Journal of Occupational Therapy, 55, Sbordone, R. J. (1996). Ecological validity: Critical issues
518–523. for the neuropsychologist. In R. J. Sbordone & C. J.
Sweetland, J., & Craik, C. (2001). The use of evidence- Long (Eds.), Ecological validity of neuropsychological
based practice by occupational therapists who treat testing. Delray Beach, FL: GR Press/St. Lucie Press.
adult stroke patients. British Journal of Occupational Wolf, M. M. (1978). Social validity: The case for subjec-
Therapy, 64, 256–261. tive measurement or how applied behavior analysis is
Tupper, D., & Cicerone, K. (1990). Introduction to the neu- finding its heart. Journal of Applied Behavior Analysis,
ropsychology of everyday life. In D. Tupper & K. 11, 203–214.
45Keilhofner(F)-index 5/18/06 8:19 PM Page 707
Index
Note: Page numbers followed by the letter f refer to figures; those followed by the letter t refer to tables.
A repeated measures, 136–137, 137t, 256–257, 257t, 324

A-B design, in single-subject research, 140–141, 142f multiple comparison tests for, 257
variation of, 141–143, 142f three-way, 319
A-B-A design, in single-subject research, 142f, 143 two-way, 258–261, 258t, 259f–260f, 259t, 319
A-B-C design, in single-subject research, 141–143, 142f Friedman test for, 257
Absolute truth, positivistic science and, 12–14 Applied research, 31–32
Abstract, of research paper, 579, 580 Aristotle, scientific philosophy of, 10, 11, 11f
Academic-practitioner collaborator, 52, 52f Arthritis Foundation, grants awarded by, 494
Accepted practices, in human subject research, 472–473 Artificiality, in long-term experimental research projects, 75
Activities of daily living (ADL). See also Instrumental activi- Asperger’s syndrome, use of confidence interval in, 234–235
ties of daily living (IADL). Assessment of Communication and Interaction Skills (ACIS),
efficiency in, 178 in experimental group, 135–136, 135f, 136f, 136t
intended construct in, items representing differing amounts Assessment process, in participatory research project, modifi-
of, 182–183, 182f cation of, 650–652
measurement of, 181 practitioner’s account, 651
Barthel’s definition of recovery in, 203 researcher’s account, 651
precision in, 178 Assessor, need relative to, 594–595
Rasch formulas in, 185–186 Attention-deficit hyperactivity disorder, therapy ball study in,
scores in, 178, 178t, 201 141–142, 141f, 142f
theoretical construct in, 181–182 Attrition, in evidence-based practice, 671
transparency in, 179 Audience
Activity Card Sort (ACS), in factor analysis, 169–170 academic, qualitative analyses reports for, 386
Administrative assistant, roles and responsibilities of, professional and scientific, research dissemination among,
431t 566–567
Administrative records, 604t specific, report writing for, 586–587
Advocates, research, 49–50, 54t stakeholder, research dissemination among, 574
Agency for Healthcare Research and Quality (AHRQ), grants Audit trail, qualitative data quality associated with, 354
funded by, 495 Australia, International Regulations governing research in,
Agendas and priorities, in participatory research, challenges of 470
different, 646 Authenticity, in qualitative research, 24
Aggregate data, 116 Authorship
Allegations, in human subject research, mechanisms for policies governing, 475
resolving, 484 responsible, in human subject research, 473–474
Alpha (␣) error, in hypothesis testing, 237–238 Awareness, cultural, in community-university partnerships,
Alternating-treatments design, in single-subject research, 638–639
146–149 Ayres, Jean, 1, 1f
example of, 148
strengths and weaknesses of, 148–149
Alternative hypotheses, statistical, 235–236 B
types of, 236–237 Background questions, in evidence-based practice, 664, 665t
American Journal of Occupational Therapy (AJOT), author Backup protocols, in data management, 562
guidelines of, 568–570 Barthel index, 201, 203
American Occupational Therapy Foundation, grants awarded Bartlett’s test, of homogeneity, 248
by, 492 Basic research, 30–31
Analog scales, visual, in data collection, 538 Behaviorism, in psychological research, 21
Analysis of covariance (ANCOVA), 315 Behaviors, multiple-baseline designs across, 143, 144f
regression and, 271–272, 271f Belmont principles, 478
Analysis of variance (ANOVA) Beneficence, in human subject research, 478
at ␣, sample sizes needed for, 311t Beta (␤) error, in hypothesis testing, 238
for regression, 266–268, 267t, 268t Between-subject factor, in repeated measures testing, 136
in statistics, 251–258 Bias
multifactorial, 258–262, 319 in data collection instrument reliability
mixed design, 261 inter-rater, 163–164, 164t
simple cell effect in, 260f, 261–262 rater, 163
multilevel data analysis and, 137–139, 138t management of
one-way, 252–253, 253t, 321 in qualitative research, 352
Kruskal-Wallis, 255, 318 in retrospective analysis, 113–115, 114t–115t
multiple comparisons for, 255–256, 256t sampling, examples illustrating, in quantitative research,
multiple comparison tests for, 253–255, 254t 520
null hypothesis of, 252 Bibliographic citation analysis, 442
708 Index
Bibliographic citation database(s), 438–440 CINAL, as information resource, 449

CINAHL, 449 Citation(s)
conduction of online searches in, 440, 440f in grant applications, 501–502
Medline, 448, 449f in human subject research, 474
OT SEARCH, 450 Citation analysis, bibliographic, 442
subject search strategies in, 439–440 Classical test theory (CTT), 183–184
World Wide Web, 439 assessments developed using, 208
Binomial effect size display, in evidence-based practice, calcu- norms in, 184
lation of, 679–680, 679t of measurement error, 159
Binomial test, values of x in, probabilities associated with, reliability of, 183–184
313t test dependency in, 183, 183f
Biometry, in data collection, 541 validity of, 184
Bivariate statistics, 244 vs. item response theory, 184
Blind review, in research dissemination, 567 Classicism, in scientific inquiry, 10–12, 11f, 17t
Blinding, in evidence-based practice, 670–671 Clinical reasoning, studies of, 6–7
Body mass index (BMI), 263, 264t, 272–273 Closed questions, formulation of, 96–97
Books, dissemination of information through, 573 Cluster randomized controlled trials. See also Randomized
Boolean connectors, in computer-assisted qualitative data controlled trials.
systems, 366 in quantitative research, 82–83, 84f
Boolean logic, in literature searches, 444–445, 445f Cluster sampling, quantitative research, 521
Bowyer, Patricia, 49, 49f Cochrane collaboration, 286
Box plot, in descriptive statistics, 225–226, 227f, 228, 228f Cochrane library databases, 449–450, 450f
Bradshaw’s categories of need, 594t Code directory, in qualitative data systems, 364t
Brainstorming, 455 Codebook
British Association/College of Occupational Therapists, grants development of, in computer-assisted qualitative data
awarded by, 492 systems, 366
Brochures, targeted, research dissemination to, 576–577 in data management, 560–561
Browsing, online, 438 Codes (coding)
Budget, in grant applications, 506–507 in data analysis, 106–107, 107f
in meta-analysis, 285–286, 287f–289f, 289–290
index, in qualitative data analysis, 360, 362, 363f, 364t
C interdependence of, 363
Canada, International Regulations governing research in, 470 Coefficient of variation, in measures of variability, 225
Canadian Government, grants awarded by, 497–498 Cohort design
Canadian Occupational Therapy Foundation, grants awarded in quantitative research, multigroup, 87–88, 88f
by, 492 in secondary data analysis, 112–113, 113t
Canadian Private Foundations, grants awarded by, 493 Co-investigator support, in grant application, 490–491
Case sensitivity, in literature searches, 445 Collaborating organizations, research dissemination among,
Case studies, descriptive, 62 575–576
Case-control design Collaboration, in research, 48, 54t
in quantitative research, 87, 87f Collaborative science, in human subject research, 476
in secondary data analysis, 112, 113t Collaborator(s)
Catalogs, public access to, online, 438 academic-practitioner, 52, 52f
Causality, probable, in experimental quantitative research, 65 in human subject research, sharing of material and data with,
Census figures, in needs assessment process, 604t 476
Center grants, 487 research, 48, 50, 50f, 51, 51f, 54t
Central limit theorem, 233–234 researcher-practitioner, 643–655. See also Participatory
Central tendency research.
graphical presentation of, 225–226, 227f, 228, 228f effective, 645–647
location of, in various distributions, 222, 222f overcoming challenges of, 647–650
measure(s) of, 220–222 selection of, grant application and, 491
mean as, 220–221 Commitment
median as, 221 in community-university partnerships, shared, 633
mode as, 222 example of, 635
Challenges in human subject research, conflicts of, 474, 476
in community-university partnerships, responsibility for, Communality, in research, 38–39
637 Communication
in high-quality research, dealing with, 435 in community-university partnerships, responsibility for,
in qualitative analysis, 376 635–636
Chi-square test, power of, 308t–309t in data management, 551
Chi-square values, 315 in participatory research, honest and sensitive, 647–648
critical, 304t Communication scores, 135–136, 135f, 136f, 136t
Friedman, calculation of, 257 Community, scientific, disseminating research findings among,
good fit test (nonparametric) and, 244–246 44–45
in nonparametric statistics, 244–248 Community building approach, to needs assessment, 597t, 600
McNemar test (nonparametric) and, 247 advantages and disadvantages of, 601t
tests of independence (nonparametric) and, 246–247, 247t Community groups, public presentation of research to,
Chronic fatigue syndrome, prevalence rates for, 60 575–576
Index 709
Community health data, 117t Conflicts of commitment, in human subject research, 474,
Community-university partnerships, 632–641 476
applying cultural skills and diversity in, 639–640 Conflicts of interest, in human subject research, 474
benefits of, 632–63, 633t Confounding, in retrospective analysis, management of,
commitment to create learning communities in, 634 115–116
commitment to mutuality in, 634 Consensus conferences
commitment to participatory research in, 634–635 background in, 406
commitment to scholarship of practice in, 635 methodological considerations in, 406–407
commitment to social problems in, 633–634 vs. other consensus methods, 402t
cultural competence in Consensus group, development of, heterogeneity vs. homo-
development of, 637–640 geneity in, 407–408
exploration of, 640, 640t Consensus methodology, 401–409
joint responsibility of, 635–637 consensus conferences in, 402t, 406–407
example of, 638 defining features of, 401, 402f
principles of, 633–637, 633t Delphi study in, 402–404, 402t
responsibility for challenges and change in, 637 development of, considerations in, 407–408
responsibility for communication in, 635–636 employment of, 401
responsibility for resources in, 636–637 group facilitator in, characteristics of, 408
responsibility for sustainability in, 637 heterogenous vs. homogenous groups in, 407–408
shared commitment in, 633 limitations of, 408
example of, 635 method selection in, 407
Comparative need, definition of, 595 nominal group technique in, 402t, 404–406, 405f
Compliance, regulatory, relationship between ethical conduct participant selection in, 407
and, 469, 471 philosophical basis of, 401
Computer-assisted qualitative data analysis (CAQDAS), Consent
364–369 documentation in, 479
codebook development in, 366 informed, 478–479
data labeling in, 365–366 procedures in, 480
hard/analog-to-digital conversion in, 365t process of, 479
memo-writing in, 368–369 Consistency, internal, of data collection instrumentation, 162
mixed-method analyses in, 369 CONSORT criteria, for randomized controlled trials, 81t–82t
multimedia capabilities in, 369 Constituents, critical awareness of, in participatory research,
storage and management of, 365–369 622
tag-and-retrieval of text in, 366–368 Construct validity, of data collection instrumentation, 168–170
Boolean connectors for, 366 convergent and divergent methods in, 169, 169f
proximity connectors for, 367–368 factor analytic method in, 169–170
semantic connectors for, 366–367, 368f known groups method in, 168–169
teamwork in, 369 study of, 170
text-based searches in, 366 Consultant support, in grant application, 491
theory building in, 369 Consumer(s)
tools and software facilitating, 369–370 in determination of social validity, 700–701
Computerized adaptive testing (CAT), in health care, 193, of research, 48–49, 54t
195–197, 195f, 196f availability of information to, 566
Computerized data systems, 121 Consumer groups, public presentation of research to, 575–576
Conceptual labels, in qualitative data analysis, 360, 362, 363f, Consumer review, in research dissemination, 567
364t Consumer voice, in participatory research, 620–621
Concerns report method, of needs assessment, 597t, 599–600 Consumer-focused dissemination, of high-quality research,
advantages and disadvantages of, 601t 436
sample statement from, 600 Content validity, of data collection instrumentation, 166–167,
Concurrent nested design, in mixed methods studies, 417–418, 167t
417t Context, assessment of, in data collection, 540
Concurrent transformative design, in mixed methods studies, Continuous scales. See Interval scale; Ratio scale.
417t, 418 Continuous variables, frequency data for, graphical presenta-
Concurrent triangulation design, in mixed methods studies, tion of, 219–220, 221f, 221t
416–417, 417t Continuum of ideas, in scientific inquiry, 17t
Concurrent validity Convenience sampling
of data collection instrumentation, 167–168 in qualitative research, example of, 527
of psychometric studies, 29 in quantitative research, 521
Conduct example of, 523
ethical. See Ethical conduct. Co-principle investigator, roles and responsibilities of, 431t
responsible, benefits of, 471 Correlation at ␣1, sample sizes needed for, 311t
Conferences Correlation at ␣2, sample sizes needed for, 311t
non–peer-reviewed dissemination at, 573 Correlation coefficient, 262–264
peer-reviewed dissemination at, 571–573, 572f definitional formula for, 262, 263f
Confidence interval intraclass, 317
calculation of, 234 Spearman rank, 264, 325
in inferential statistics, 234–235, 235t strength and significance of, 263–264
in odds ratio, 111–112 Correlation matrix, 263, 264t
710 Index
Correlational research, 62–63 entry and confirmation of, systems for, 559
Creative problem-solving (CPS) process, 455 evaluation and coding of, in meta-analysis, 284t, 285–286,
components and stages of, 455, 456f, 457–459 287f–289f, 289–290
constructing opportunities in, 456, 457–458 existing
case illustration of, 462, 464 advantages and disadvantages of, 116–122
developing solutions and building acceptances in, 457, purpose and nature of, 118, 121
458–459 sources of, 118, 118t–120t, 121
case illustrations of, 464, 466 types of, 116–118, 117t
example of, 456–457 utilization of, 122–123
exploring data in, 456–457, 458 challenges in, 123–125, 125f
case illustrations of, 463, 464, 465f choice of timely data source in, 125
framing problems in, 457, 458 computer information specialist in, 125
case illustrations of, 463, 466 for practice, service needs, and outcome, 121–122
generating ideas in, 457, 458 management of missing data in, 125
case illustrations of, 463–464, 466 methodology and conclusion considerations in, 124
understanding challenge in, 457–459 program staff help in, 125
Creativity research question considerations in, 124
definition of, 455 selection of appropriate database in, 124, 125f
role of, in identification of research questions, 454–459 thorough data description in, 124–125
Credit, assignment of, in human subject research, 474 training in, 125
Criterion referencing, with data collection instruments, frequency distribution of, 218–222, 219t
172–173 graphical presentation of
Criterion validity, of data collection instrumentation, 167 for continuous variables, 219–220, 221f, 221t
Critical awareness, of constituents, in participatory research, for discrete variables, 219, 219t, 220f
622 generating findings in, 44
Critical modernism, in scientific inquiry, 14–16, 17t in determining health status, 117t
Critical theory study, in qualitative research, 334–335 interview
conduction of, 335 methods of recording, 536–537
contributions of, 335 sources of, 537
Critical value, in statistical hypotheses, 239–240, 240f labeling of, in computer-assisted qualitative data systems,
Critically appraised paper (CAP) format, 673t 365–366
Criticism, of research, 566 management of. See Data management.
Cronbach’s coefficient alpha, 316 mental health, 121–122
Crossover (counterbalanced) design, in quantitative research, multiple sources of, importance of, 533
83–85, 84f, 85f of health insurance groups, 121
Cross-sectional design preparation of, issues during, 563–564
in quantitative research, 86–87, 86f qualitative. See Qualitative data.
in secondary data analysis, 111–112, 113t quantitative. See Quantitative data.
simultaneous, in longitudinal research, 129–130, 129t, 133t raw, 218
analysis of, 134 calculation of mean from, 220
Cultural competence, in community-university partnerships relevance and sufficiency of, 542–543
development of, 637–640 removal of, to storage medium, 123
exploration of, 640, 640t retrieval of, 122
Cultural diversity, application of, in community-university security of, as management issue, 554t
partnerships, 639–640 storage of, in high-quality research, 433–435
Cultural knowledge and awareness, in community-university vs. opinion, 37
partnerships, generation of, 638–639 Data analysis, 43–44, 44f
Cultural skills, in community-university partnerships, in high-quality research, 425–426
639–640 in qualitative research, 24. See also Qualitative data,
Cumulative incidence (CI), calculation of, 59 analysis of.
Cusick, Anne, 52, 52f in quantitative research, 44
in survey research, preparation of, 106–108, 107f
issues during, 563–564
D secondary, 110–125
d calculation, in continuous data effect sizes, 676–677, 677t bias and confounding in, management of, 113–116,
d interpretation, in continuous data effect sizes, 677, 677t 114t–115t
Data. See also Statistics. case-control design for, 112, 113t
access to, 122 cohort design for, 112–113, 113t
as management issue, 554t cross-section design for, 111–112, 113t
acquisition, management, sharing, and ownership of, in definition of, 110
human subject research, 472 designs for, 110–113, 111f, 113t
adequacy of, in qualitative research samples, 525 of existing databases, 116–122. See also Data, existing.
aggregate, 116 Data checking, 559–560
analysis of. See Data analysis. as management issue, 554t
collection of. See Data collection. sample protocol for, 560
compliance of, with privacy and ethical protocols, 122–123 Data cleaning, 559–560
computerized, 121 as management issue, 554t
definition of, 548 sample protocol for, 561
Index 711
Data coding, 559–560 internal consistency of, 162

sample protocol for, 560 norm referencing with, 172–173
Data collection, 530–546 precision of, 171–172
approach(es) to, 533–541 predictive validity of, 168
biometric/physiological measures in, 541 psychometric properties of, 171–173
contextual/environmental assessment in, 540 quantitative, 155–175
document/record review in, 541 reliability of, 160–161
focus groups in, 540–541 alternative forms or equivalency in, 162
interviews in, 536–538 empirical evaluation in, 161–165
observation in, 533–536 evaluation of, approaches to, 162–163
self-report measures in, 538–539 evidence of, creating and evaluating, 171
standardized tests and performance measures in, 539–540 generalizability theory and, 164–165
town hall meetings in, 540–541 internal consistency in, 162
as management issue, 554t inter-rater bias in, 163–164, 164t
implementation of, 43, 541–546 interrelationship of validity with, 170–171
personnel preparation in, 545 intraclass correlations and, 164–165
plan or protocol in, 545 observer presence or characteristics and, 163
selection of instruments and strategies in, 541–545 rater bias in, 163
in high-quality research, 425–426 rater/observer effects on, 163
in meta-analysis, 284t, 285 split-half, 161–162
in needs assessment process, methods of, 602, 603t–604t test-retest or stability in, 161
in qualitative research, 24–25 selection of, 541–545
in quantitative research, 22 availability of norm-referenced scores in, 544
in single-subject research, 149, 150f criteria in, 542–545
in survey research, 91–95 logistical considerations in, 543–544
advantages and disadvantages of, 93t methodological rigor in, 543
selecting and combining, 95 relevance of data in, 542–543
instruments in. See Data collection instruments. type of data yielded in, 544–545
interventional scales in, 158–159, 158t sensitivity and specificity of, 172
issues during, 563 standardized scores for, 173
key elements of measure in, 156–157 validity of, 165–170
logistical issues and obstacles in, grant applications and, 504 convergent and divergent methods of, 169, 169f
measurement error in, 159 evidence of, creating and evaluating, 171
reduction strategies for, 159–160 factor analytic method of, 169–170
standard, 173 interrelationship of reliability with, 170–171
nominal scales in, 157, 158t known groups method of, 168–169
observation in, 533–536 Data confirmation systems, 559
ordinal scales in, 157–158, 158t Data entry, as management issue, 554t
professional and ethical considerations in, 545–546 Data entry specialist, roles and responsibilities of, 431t
qualitative. See Qualitative data, gathering of. Data entry systems, 559
quantifying information in, 155 Data files, set-up tips for, 563
quantitative, 530 Data graphs, in single-subject research, 149
approach to, 530–531 Data management, 548–564
rater error in, reduction of, 160 backup protocols in, 562
ratio scales in, 158t, 159 components of, 551–552, 551f
rule-bound nature of measurement in, 157–159, 158t context of, 548–549, 549f
scales of measurement in, 157–159, 158t definition of, 548
standardization of, 160 facilitation of
tester error in, reduction of, 160 assigning participant identifiers in, 557–558
variables in, 214 backup protocols in, 562
graphical presentation of codebooks in, 560–561
continuous, 219–220, 221f, 221t coding, checking, and cleaning protocols in, 559–560
discrete, 219, 219t, 220f sample protocol of, 560, 561
Data collection checklist, 542 collection forms in, 561
Data collection forms, 561 creation of master file in, 558
Data collection instruments job descriptions in, 556
concurrent validity of, 167–168 plan and project timeline in, 555–556, 556t
construct validity of, 168–170 systems for entry and confirmation in, 559
convergent and divergent methods in, 169, 169f team meetings in, 561
factor analytic method in, 169–170 tools and resources in, 555–562, 555t
known groups method in, 168–169 tracking participants in, 556–558, 557f
study of, 170 use of participant ID sheets in, 558–559, 559f
constructs and operationalization of, 156–157 importance of, 549–551, 550f
content validity of, 166–167, 167t in high-quality research, 433–435
criterion referencing with, 172–173 issues during, 552, 553t–554t, 555
criterion validity of, 167 in data collection phase, 563
development of, process of, 174–175 in data preparation and analysis phases, 563–564
face validity of, 165–166 project initiation and, 563
712 Index
Data management (continued) in gathering qualitative data, 349–351

proposal development in, considerations during, 562 in grant applications, presentation of, 502
tools and resources in, 555–562, 555t Documentation
Data recording sheet, in single-subject research, 149, 150f as data management issue, 554t
Database(s) of consent, for human subject research, 479
bibliographic citation, 438–440. See also Bibliographic of evidence-based practice, 680
citation database(s). of policies, in data management, 550
Cochrane library, 449–450, 450f Domain analysis, in qualitative research samples, 527–528
Database manager, roles and responsibilities of, 431t Draft(s), of research paper, 588–589
Debriefing, peer, 353 Dropouts, in long-term experimental research projects, 75–76
Debriefing interview, in survey research, 102
Decision-making entities
availability of information to, 566 E
in development of research questions, design, and methods, Ecological validity, of evidence-based practice, 701–702
459–460, 459t assessment of, 702
Deduction, in research, 38 relationship between characteristics of research evidence
Definitional formula and, 702–703
for F-value, 267 Education, continuing
for Pearson Product Moment correlation coefficient, 262, as data management issue, 553t
263f dissemination of information through, 573
Delphi methodology Educational grants, 487
anonymity in, 404 Effect size (ES)
background in, 402 in evidence-based practice
construction and distribution of, 403 binomial effect size display, 679–680, 679t
development, analysis, and presentation of, findings in, 403 calculation of, 671–680, 671t
group consensus in, achievement of, 404 from different score, 678
group members in, identification and recruitment of, 403 precision of, 675
methodological considerations in, 402 d calculation, 676–677, 677t
of needs assessment process, 604t d interpretation, 677, 677t
results of, collation of, 403–404 r calculation, 677, 677t
vs. other consensus methods, 402t r interpretation, 677, 679
Demonstration grants, 486 in occupational therapy meta-analysis studies, 290, 291t,
Denominator, 223 292
Dependent variable in quantitative studies, 522–523
artificiality applied to, 75 Electronic data storage, of human subject research, 473
measuring participants against, 68–69 Empiricism
within population, probability of causal effect of independ- changing theories in, 15–16
ent variable on, 69–70, 70f in research, 37
Depth vs. breadth sampling, in qualitative research, 527 role of, 17t
Descartes, scientific philosophy of, 1112 rethinking, 14–16
Descriptive quantitative designs, 58–63 Empowerment, in participatory research, 622
correlational, 62–63 Environment, assessment of, in data collection, 540
univariate, 58–62 Environmental data, 117t
case studies of, 62 Epistemologies (traditions of thought ), in qualitative research
characterizing phenomenon in, 59 diversity of, 328–330
developmental studies in, 61–62 major, 328–338. See also specific epistemological tradition.
incidence and prevalence studies in, 59, 61 nature of, 330
normative studies in, 61 Equivalency reliability, of data collection instrumentation, 162
study sample of, 58–59 Ethical conduct, in human subject research
Developmental research, 61–62 benefits of, 471
Diffusion of innovations, in evidence-based practice, 698, Ethics Committees and Institutional Review Boards for, 480,
699f 483, 483f
application of, 704 issues of, 468–471
d-index, used in meta-analysis studies, 290, 291t, 293 Nuremberg code for, 477–478
Directory(ies) protection of participant and, 477–483
code, 364t regulation of, 469
web, 448 relationship between regulatory compliance and noncompli-
Disability, public attitudes toward, case illustrations of, ance and, 469, 471
462–464 Ethical considerations, in data collection, 545–546
Discipline, in report writing, 585 Ethical guidelines, in grant applications, 503–504
Discrete variables, graphical presentation of, frequency data Ethical protocols, existing database compliance with, 122–123
for, 219, 219t, 220f Ethical review, of research, 42, 429
Dissertation research, 47 Ethics Committees, 480, 483
Diversity, cultural, in community-university partnerships, Ethnicity issues, in research studies, 516
639–640 Ethnography, in qualitative research, 22, 331–332
Doctoral dissertation process, 47 conduction of, 332
Document(s) contributions of, 332–333
in data collection, review of, 541 Evaluation Tool of Children’s Handwriting (ETCH), 168
Index 713
Event, probability of, 232 survey 1, 2000, 687, 687t

Evidence-based practice (EBP), 656–660 survey 2, 2000, 687–689, 688t
adoption of, strategies in, 689, 689t Exclusion criteria, in quantitative research samples, 518
attrition in, 671 Expected frequency (E), in nonparametric statistics, calculation
barriers to, 687–689 of, 246
binomial effect size display in, calculation of, 679–680, 679t Experimental designs
blinding in, 670–671 in program evaluation, 614, 615t
change in in quantitative research, 65
different responses to, 685–686 basic, 65–70
factors and conditions promoting, 692, 692t definition of, 65–66
readiness for, 686–687 explanation of, 66–70
clinical significance of, 703–704 Experimental research, 25–26
relationship between characteristics of research evidence adherence to protocols in, 43
and, 704 examples of, 27
critical appraisal of, 665–671, 670t long-term, dropouts in, 75–76
control group in, 666 rigor in, 36
format of, 673–674 Expertise, in participatory research, acknowledging and
methodological considerations in, 666, 670–671 respecting, 647
definitions of, 656–657 Expressed need, definition of, 595
discussions in, 657, 659, 659f, 692–694
documentation of, 680
ecological validity of, 701–702 F
assessment of, 702 F1 ␣1, critical values of, 306t
relationship between characteristics of research evidence F1 at ␣1, critical values of, 305t
and, 702–703 Fabrication, of research, 471
effect size calculation in, 671–680, 671t Face validity, of data collection instrumentation, 165–166
from different score, 678 Face-to-face interviews
precision of, 675 advantages and disadvantages of, 93t
effect size d in approach to, 105
calculation of, 676–677, 677t in survey research, 94
interpretation of, 677, 677t Factor analysis, 316
effect size r in methods of, 274
calculation of, 677, 677t multivariate, 273–276, 275t
interpretation of, 677, 679 Factor matrix, 275t
emergence of, 281 rotated, 275t
evolution of, 657 Falsification, of research, 471
finding time for, 689–690 Family(ies), stress and burden of, of mentally ill offenders,
follow-up in, 670 case illustration of, 464, 465f, 466
implementation of, 660 Federal granting agencies, U.S., 494–495, 496t–497t
innovations in Feedback, in report writing, 589
application of desirable characteristics of, 698–700, 699f Felt need, definition of, 594–595
diffusion of, 698, 699f Field pretesting, in survey research, 101–102
application of, 704 Field research, 27–28
internal validity of, threats to, 667–669 Field study site, qualitative data quality associated with
international developments in, 658 entering of, 354–356
levels of hierarchy used in, 281, 282f, 670t exit from, 356
literature searches in, 665 prolonged engagement in, 352–353
management of, 689–692 Filter questions, formulation of, 96
number needed to treat in Fit statistics, in Rasch analysis, 187t, 188–189
calculation of, 673, 674t Fixed-response questions, vs. open-ended questions, 346t
interpretation of, 674–676 Flexibility, in qualitative analysis, 376
questions in, 663–665 Focus, in report writing, 587
background, 664, 665t Focus group(s)
PICO, 664, 665t data collection from, 540–541
specific, 664–665 in needs assessment process, 603t
research evidence in in survey research, 101
fulfillment of desirable characteristics of, 700–704 qualitative data collection from, 347–349
identification of desirable characteristics of, 698–700, strengths and weaknesses of, 348t, 349
699f Focus group interviews, example of, 350
research in, 4 Forensic occupational therapy, research priorities in, 407
creation of relevance of, 697–704 Format, of grant applications, 500–501
utilization of, 697–698 Formulating questions, in survey research, 95–97, 96f
skills and knowledge in, development of, 690–691 for different scales, 100–101, 101f
social validity of, 700–701 Foucault, scientific philosophy of, 16
relationship between characteristics of research evidence Foundational research knowledge, generation of, 4–5
and, 701 F-ratio
verification of, 701 in one-way ANOVA, 252
staying focused in, 691–692 in repeated measures ANOVA, 256
714 Index
Frequency distribution, of data, 218–222, 219t demonstrate knowledge of regulations, policies, and guide-
graphical presentation of lines in, 498–499
for continuous variables, 219–220, 221f, 221t develop hypothesis in, 503
for discrete variables, 219, 219t, 220f develop ideas in, 489–490
Friedman test, 316 develop reasonable budget request in, 506–507
for two-way ANOVA, 257 develop timeline and evaluation plan in, 504, 506
F-test ensure appropriate review in, 507–508
for regression analysis, power of, 312t ensure ethical design in, 503–504
in one-way ANOVA, 252 identify specific aims in, 502–503
Functional Index Measure (FIM), 179 identify theoretical basis for study in, 499
changes in, paired t-test for, 250t negotiate with sponsoring institution before, 490
in practice and research, 201 obtain letters of support for, 507
scores vs. measures and, limitations of, 179–180 planning analyses in, 504
Functional Index Measure (FIM) Motor Score, ANCOVA and, process of, 488–508
271–272, 271f select appropriate funding agency for, 491–498
Funding agencies. See also Grant funding. team selection before, 490–491
administrators in, working with, 499 use of pilot research in, 502
appropriate, selection of, 491–498 use of structured and organized format in, 500–501
availability of information to, 566 Grant funding. See also Funding agencies.
communication with, 435 alternative mechanisms of, 487
direction of, in identification of research questions, 453–454 decisions regarding, 509
goals and priorities of, matching grant application with, examples of, 505–506
490 process of, 488
U.S. federal, 494–495, 496t–497t purpose of, 486
F-value reasons to apply for, 487–488
definitional formula for, 267 Grant proposals, organization of, in funding applications,
in one-way ANOVA, 252, 253t 500–501
Grant review, U.S. Department of Education evaluation criteria
for, 511t–512t
G Grant scores, interpretation of, 509
Galileo, scientific philosophy of, 11 Grant writing
Gender issues, in research studies, 516 in research, 428
Generality issues, in single-subject research, 152–153 steps in, 489–508. See also Grant application(s).
Generalizability theory, of data collection instrument reliabil- Gray literature, as information resource, 450–451
ity, 164–165 Grounded theory study, in qualitative research, 333
Gitlin, Laura, 1–2, 2f conduction of, 333
Good fit test, nonparametric, 244–246 contributions of, 333–334
known distribution in, 246 Groundedness, in qualitative research, 24
normal distribution in, 246 Group comparison designs, in quantitative research
uniform distribution in, 244–246 case-control, 87, 87f
Government agencies, research dissemination to, 576 cohort analytical study of, 87–88, 88f
Governments, grants awarded by, 494–498, 496t–497t cross-sectional, 86–87, 86f
Grant(s) nonrandomized, 85–86, 85f, 86f
awarded by governments, 494–498, 496t–497t type I error in, 70–72, 71f
awarded by private foundations, 491, 493 type II error in, 72–75, 73f, 74f, 74t
awarded by professional organizations, 491, 492 validity of, 70–75
awarded by self-help organizations, 493–494 Group facilitator, in consensus methodology, characteristics of,
center, 487 408
demonstration, 486 Grunbaum, scientific philosophy of, 14
educational, 487 Guidelines, for writing grant applications, 498–499
professional development, 487
research, 486
resubmission of, 509–510 H
submission of, 508–510 H statistics, Kruskal-Wallis, calculation of, 255
feedback following, 509–510 Hand strength, effect of thera-putty on, 671–672, 671t
interpretation of scores following, 509 Hard/analog-to-digital conversion, in computer-assisted quali-
review process following, 508–509, 510t–512t tative data analysis, 365t
training, 487 Health care
types of, 486–487 application of item response theory in, 189–198
within university settings, 494 computerized adaptive testing in, 193, 195–197, 195f,
Grant application(s) 196f
address logistical issues and obstacles in, 504 objective measurement in, 181–183
build on existing scientific trends in, 490 Health care instruments
choose appropriate and rigorous design in, 503 IRT methodologies in
competing, NIH evaluation criterion and questions for, analyzing existing instruments based on, 189–191, 190f
510t application of, 189–198
demonstrate expert knowledge of topic in, 499–500 computerized adaptive testing and, 193, 195–197, 195f,
demonstrate good scholarship in, 500–502 196f
Index 715
development of new instruments based on, 191 paper and electronic data storage in, 473
innovative measurement based on, 191 participation in
instantaneous measurement and quality control based on, UIC consent form for, 482f
development of keyforms for, 192–193, 194f UIC written and verbal assent to, 481f
linking existing measures based on, 197–198 peer review in, 476
scores from, 179–180, 180f policies governing authorship in, 475
lack of unidimensionality of, 180–181 principles of Belmont in, 478
test-dependent, 180 process of consent in, 479
Health Information Privacy and Access Act (HIPAA), 472, publication practices and responsible authorship in, 473–474
473 repetitive and fragmentary publication in, 474
concerns related to, 476 research team in, role of, 483
Health insurance groups, databases of, 121 respect of persons in, 478
Health system data, 117t Tuskgee experiment in, 477
Health-related conditions, prevalence and incidence rates in, 61 Hume, David, scientific philosophy of, 13
Hempel, scientific philosophy of, 13 Hypothesis testing, 235–242
Heterogeneity, vs. homogeneity, in developing consensus alternative, 235–236
group, 407–408 types of, 236–237
Hierarchical linear models, 317 critical value in, 239–240, 240f
Hierarchical selection method, of multiple regression, 269, development of, in grant applications, 503
269f errors in, 237–238, 237t, 238f
High-quality research type I, 237–238
ensuring sample representativeness in, 433 type II, 238
team roles and responsibilities in, 431t null, 235–236. See also Null hypotheses.
Homogeneity, vs. heterogeneity, in developing consensus one-tailed and two-tailed, 240–241, 240f
group, 407–408 power analysis in, 239
Human biology data, 117t power and effect size in, factors influencing, 238–239
Human immunodeficiency virus (HIV) infection steps in, 241–242, 241t
prevalence and incidence rates of, 61
qualitative research questions in, 374
qualitative vs. quantitative research in, 374 I
Human subject research, 468–484 ID sheets, participant, development of, 558–559, 559f
accepted practices in, 472–473 Ideas
assent and surrogate permission in, 479 continuum of, in scientific inquiry, 17t
assignment of due credit in, 474 development of, in grant applications, 489–490
beneficence in, 478 Identifier(s), participant
citations in, 474 development and assignment of, 557–558
collaborative science in, 476 sample explanation of, 558
conflicts of interest or commitment in, 474, 476 Images, in computer-assisted qualitative data analysis,
data acquisition, management, sharing, and ownership in, hard/analog-to-digital conversion of, 365t
472 Impact, of grant application, importance of, 489
documentation of consent in, 479 Incidence rate (IR), calculation of, 61
ethical conduct in Inclusion criteria, in quantitative research samples, 518
benefits of, 471 Independent t-test, parametric, 248–249, 249t, 317
Ethics Committees and Institutional Review Boards for, Independent variable
480, 483, 483f administration of, planned, 67
issues of, 468–471 artificiality applied to, 75
Nuremberg code for, 477–478 categorical, 66
protection of participant and, 477–483 conditions to, participants assigned to as many groups as,
regulation of, 469 66–67
relationship between regulatory compliance and noncom- continuous, 66
pliance and, 469, 471 dependent variable and, 68–69
external collaborators in, sharing of material and data with, potentially confounding variables and, 67–68
476 within population, probability of causal effect of dependent
fabrication, falsification, and plagiarism in, 471 variable on, 69–70, 70f
informed consent for, 478–479 In-depth interviews, for gathering qualitative data, 344–345
procedures in, 480 key activities of researchers before and during, 347
integrity in, 471–476 Index codes, in qualitative data analysis, 360, 362, 363f, 364t
issues of, 468–471 interdependence of, 363
policies governing, 469 Individual data, 116
regulation of, 469 Induction, in research, 37–38
International Regulations governing, 470 Induction-deduction process, in scientific inquiry, 10, 11f
investigator in, role of, 483 Inductive process, of qualitative data analysis, 373
irresponsible conduct in, 472 Informant, in research studies, 516
justice in, 478 Information
mechanisms for resolving allegations in, 484 availability of
mentor/trainee relationships in, 473 to professionals, 565–566
misconduct associated with, 484 to scholars in related disciplines, 566
Nazi doctors’ trials in, 477 project, 548
716 Index
Information content analysis, in literature searches, 446 Interview guide, items from, examples of, 562
Information gathering, by observation, in data collection, 534 Interview questions, fixed-response vs. open-ended, 346t
Information resource, relevant to occupational therapy, Interviewer training, qualitative data quality associated with,
448–451 352
Information retrieval, online, 437–438 Interviewer-administered surveys, in needs assessment process,
process of, 438 603t
Informed consent, 478–479 Intraclass correlations, 317
procedures in, 480 in data collection instrument reliability, 164–165
Innovations, in evidence-based practice Intra-individual/inter-individual variability, in longitudinal
application of desirable characteristics and, 698–700, 699f research, 128
diffusion of, 698, 699f Investigator(s)
application of, 704 role of, in human subject research, 483
Inspiration, in report writing, 588 vigilance of, during research implementation, 43
Institution, negotiation with sponsoring, grant writing and, 490 Irresponsible conduct, in human subject research, 472
Institutional Review Board (IRB), 480, 483, 483f Issue(s)
Instrument(s) in grant applications, addressing, 504
data collection. See Data collection instruments. in needs assessment, scope of, 592f, 593
development of, in occupational therapy, 31 Item response theory (IRT)
health care. See Health care instruments. in health care, applications of, 189–198. See also Health
Instrumental activities of daily living (IADL). See also care instruments, IRT methodologies in.
Activities of daily living (ADL). in research and practice, influence of, 177–198
independent t-test for, 249t measures vs. scores in, 177–179, 178t
path analysis of, 276–277, 277f limitations of, 179–181, 180f
Integrity, in human subject research, 471–476 Rasch measurement and, 184–189, 187t–188t
issues of, 468–471 vs. classical test theory, 184
policies governing, 469
regulation of, 469
Interaction skill scores, 135–136, 135f, 136f, 136t J
International Classification of Functioning (ICF) Activity Job description(s)
Measure data management and, 556
computerized aspect of, 196, 196f research assistant, 432
development of, 196 Joint authorship policy, 475
International developments, in evidence-based practice, 658 Joint responsibility, in community-university partnerships,
International Regulations, governing human subject research, 635–637
470 example of, 638
Interquartile range (IQR), 223, 225–226 Journal(s)
Inter-rater bias, in data collection instrument reliability, peer-reviewed articles for, 570
163–164, 164t selection of, criteria in, 583–584
Inter-rater reliability, in qualitative data analysis, 364 Justice, in human subject research, 478
Interrelated activities, in research process, 39–45, 39f
Interval scale
formatting questions for, 101 K
in data collection, 158–159, 158t Kant, Immanuel, scientific philosophy of, 13
of measurement, 217 Kappa, 318
Intervention study design, in longitudinal research, 131–132 in assessment of instrument reliability, 163–164
Interview(s). See also Questionnaire/interview. Key informant interviews, for gathering qualitative data, 348t,
advantages and disadvantages of, 93t 349
debriefing, 102 Keywords, in literature searches, 443
face-to-face, 94 Knowledge
in data collection, 536–538 in community-university partnerships, cultural, 638–639
recording methods of, 536–537 in participatory research, generation of, 623, 625–626
role of interviewer in, 537–538 Knowledge of topic, demonstration of, in grant applications,
sources of, 537 499–500
in qualitative data collection Knowledge/power differentials, in participatory research, chal-
focus groups and, 347–349 lenges of, 645–646
example of, 350 Kruskal-Wallis ANOVA, one-way, 255, 318
strengths and weaknesses of, 348t, 349 multiple comparisons for, 255–256, 256t
in-depth, 344–345 Kuhn, scientific philosophy of, 15
key activities of researchers before and during, 347
key informant and, 348t, 349
semistructured and structured, 345–347, 346f, 346t, 348t L
strengths and limitations of, 348t Labeling of data, in computer-assisted qualitative data systems,
unstructured, 345 365–366
systematic approach to, 105–106 Labels, conceptual, in qualitative data analysis, 360, 362, 363f,
telephone, 94, 96f 364t
Interview data Language, controlled, in literature searches, 443–444
methods of recording, 536–537 Lannin, Natasher, 51, 51f
sources of, 537 Laws of gravity, 20
Index 717
Laws of kinetics, 20 panel studies in, 132, 132f, 133t

Learning communities, in community-university partnerships, analysis of, 134–135
commitment to, 634 simultaneous cross-sectional studies in, 129–130, 129t, 133t
Legislative bodies, research dissemination to, 576 analysis of, 134
Legislative initiatives, in grant applications, 489–490 statistical analysis of, 134–135
Letters of support, for grant application, 507 time series studies in, 131–132, 131t, 133t
Level of Rehabilitation Scale (LORS), 189–190, 190f analysis of, 134
Levene’s test, of homogeneity, 248 trend studies in, 130–131, 131, 133t
Lifestyle data, 117t analysis of, 134
Likert scale, formatting questions for, 101, 101f types of questions in, 128
Linear regression Lyotard, scientific philosophy of, 16
power analysis for, 268
simple, 265, 325
Literature review, 40–41 M
in development of research questions, 454 Mailed questionnaires
correct, 460–461 advantages and disadvantages of, 93t
synthesizing discovered, 461–462, 462f, 463t format of, 99f
in report writing, 580–581 in survey research, 92, 92f
up-to-date, in grant applications, 501 systematic approach to, 106, 106t
Literature search(es), 421–422, 437–451 Major strategy(ies), for gathering qualitative data, 342–351
bibliographic citation databases, 438–440 focus group interviews as, 347–349, 348t
conduction of online searches in, 440, 440f example of, 350
online access to published material, 438 in-depth interviews as, 344–345
online information retrieval, 437–438 key informant interviews as, 348t, 349
relevant information resource, 448–451 observation as, 342–344, 345t
CINAL, 449 participation as, 342
Cochrane library databases, 449–450, 450f semistructured and structured interviews as, 345–347, 346f,
gray literature, 450–451 346t, 348t
Medline, 448, 449f unstructured interviews as, 345
OT SEARCH, 450 written documents and material objects as, 349–351
specific content web sites, 451 Management plan, in research, 429, 430t
step 1, 440 Mann-Whitney U-test, 249–250, 319
step 2, 440–442, 441f, 442f, 443f Marketplace philosophy, needs assessment grounded in,
step 3, 442–447, 444f 595–596
analysis of information content in, 446 Master file, in data management, creation of, 558
Boolean logic in, 444–445, 445f Material objects, in gathering qualitative data, 349–351
case sensitivity in, 445 McNemar test, nonparametric, 247, 319
controlled language in, 443–444 Mean
keywords in, 443 and z-score, normal curve between, 299t–301t
nesting in, 445 as measure of central tendency, 220–221
phrases in, 445 sampling error of, 234, 235t
post-qualification and limiting in, 446–447 standard error of, calculation of, 234
proximity in, 445–446 Measure (measurement)
sample query in, 446, 446f, 447f in assignment of numerals, 214
search query in, 444 objective
search syntax in, 445 fundamental, 185
search terms in, 443 in occupational therapy and health care, 181–183
truncation in, 445 performance
step 4, 447–448 administration of, 540
metasearch engines in, 448 in data collection, 539–540
web directories in, 448 variables in, 214
web search engines in, 447–448 vs. score, 177–179, 178t
Logic, in research, 37–38, 39f efficiency of, 178
Logic model, of needs assessment, 597t, 598, 598f equal intervals in, 177–178
advantages and disadvantages of, 601t limitations of, 179–181, 180f
Logistic regression, 272–273, 273t, 318 precision in, 178
Longitudinal questions, types of, 128 transparency in, 179
Longitudinal research, 127–139 Measurement scales, 214–218. See also specific scale.
analytical example of, 135–139, 135f, 136f, 136t, 137t, in data collection instrumentation, 157–159, 158t
138t statistics in relation to, 217–218, 218f
defining characteristics of, 127 types of, 214–217
designs in, 128–133 Measures of variability, 222–225
comparison of, 133t coefficient of variation in, 225
features of, 128 graphical presentation of, 225–228
five, 129–132 percentiles in, 223
intervention studies in, 131–132 range in, 223
intra-individual and inter-individual variability in, 128 standard deviation in, 223–225, 225t, 226t, 227f
nature of, 127–128 variance in, 223, 224t
718 Index
Media, release of research findings to, 577 Multiple comparison tests, for one-way ANOVA, 253–255,
Median, as measure of central tendency, 221 254t
Medline, as information resource, 448, 449f Multiple regression, 268–270
Member checking, qualitative data quality associated with, multicolinearity in, 268
353–354 selection of variables in
Memos, in qualitative data analysis, 362–363, 364t hierarchical method for, 269, 269f
analytic, 360, 364t standard method for, 269, 269f
interdependence of, 363 stepwise method for, 269–270, 270f
Memo-writing, in computer-assisted qualitative data systems, Multiple-baseline designs, in single-subject research, 143
368–369 across behaviors, 143, 144f
Mental health data, sources of, 121–122 across participants, 143–144, 145f, 146f
Mentorship, grant applications and, 502 across settings, 145–146, 147f
Mentor/trainee relationships, in human subject research, 473 Multivariable statistics, 244
Meta-analysis, of published research, 281–296 Multivariate analysis
advantages of, 294–295 factor analysis and, 273–276, 275t
examples of, relevant to occupational therapy, 292–294, 293t MANOVA and, 277, 277f, 278t, 279, 320
limitations of, 295 path analysis and, 276–277, 277f
process of, schematic of, 283, 283f Multivariate analysis of variance (MANOVA), 277, 277f, 278t,
qualitative methods and, 295–296 279, 320
step(s) in, 283–292 null hypothesis for, 277
analysis and interpretation as, 284t, 290–292, 291f, 291t Multivariate statistics, 244
data collection as, 284t, 285 Mutuality, in community-university partnerships, commitment
data evaluation and coding as, 284t, 285–286, 287f–289f, to, 634
289–290
problem formation as, 284–285, 284t
reporting results as, 284t, 292, 293t N
Metasearch engines, 448 Nagel, scientific philosophy of, 13–14
Methodological approach, to grant writing, 503–504 Narrative inquiry, in qualitative research, 22
Mill, John Stuart, scientific philosophy of, 13 National Board for Certification of Occupational Therapists
Miller, Lucy Jane, 1, 2f (NBCOT), 95
Mind-mapping, in report writing, 587 National Center of Health Statistics (NCHS), public informa-
Mini Mental State Examination (MMSE) score, 263, 264t, tion provided by, 122
272–273 National Institute of Occupational Safety and Health (NIOSH),
Minimum significant difference (MSD) documents published by, 122
in Freidman test, calculation of, 257 National Institutes of Health (NIH)
in post-hoc tests, calculation of, 255 evaluation criterion and questions of, for competing grant
Minnesota Rate of Manipulation (MRM) test, 21 application, 510t
Misconduct, research, 484 grants funded by, 494–495, 496t–497t
Mixed methods design(s), 411–418, 412f Naturalistic field studies, 27–28
concurrent nested type of, 417–418, 417t examples of, 29
concurrent transformative type of, 417t, 418 Nazi doctors’ trials, 477
concurrent triangulation type of, 416–417, 417t Need
multimethod, 412–413, 413f as physiological drive, 593
nomenclature in, 412–413 as relative to assessor, 594–595
principle(s) of, 413–416 as solution, 594
adherence to methodological assumptions of core method Bradshaw’s categories of, 594t
as, 414–415 comparative, 595
conversion of datasets as, 415–416 expressed and felt, 594–595
recognition of role of imported models to project as, 415 identification of, as political process, 595–596
recognition of theoretical drive of project as, 414 normative, 594
quantitative and qualitative traditions used in, 413, 413f Needs assessment process, 591–605
sequential explanatory type of, 416, 417t approaches to, 596, 597t, 598–601
sequential exploratory type of, 416, 417t advantages and disadvantages of, 601t
sequential transformative type of, 416, 417t community building, 597t, 600
typologies of, 412–413 concerns report method of, 597t, 599–600
Mixed-method analyses, of computer-assisted qualitative data logic model of, 597t, 598, 598f
systems, 369 participatory, 597t, 600
Mode, as measure of central tendency, 222 three-phase model of, 597t, 598–599
Modernism, in scientific inquiry, 12–14, 17t concept of need in, 593–595
critical, 14–16, 17t Bradshaw’s categories of, 594t
Multicolinearity, in multiple regression, 268 data collection methods in, 602, 603t–604t
Multigroup cohort design, in quantitative research, 87–88, 88f developing recommendations for, 602, 605
Multilevel data analysis, of longitudinal research, 137–139, dimension 1: sophistication of project design, 591–592,
138t 592f
Multimedia capabilities, of computer-assisted qualitative data dimension 2: level of involvement of stakeholders, 592,
systems, 369 592f
Multimethod designs, in mixed methods studies, 412–413, dimension 3: political orientation, 592–593, 592f
413f dimension 4: scope of issue being addressed, 592f, 593
Index 719
grounded in marketplace philosophy, 595–596 Objectivity, in quantitative research, 21

grounded in social good philosophy, 596 Objects, material, in gathering qualitative data, 349–351
in programs of service, 609 Observation
Negotiation, in participatory research, ongoing, 649–650 in data collection, 533–536
Nesting, in literature searches, 445 contexts of, 535
Network (snowball) sampling advantages and disadvantages of, 535–536
in qualitative research, 523 forms of, 534
in quantitative research, 522, 523 role of observer and, 534–535
Newsletters, dissemination of information through, 573–574 types of information gathered through, 534
Newton, scientific philosophy of, 12 in needs assessment process, 604t
Nominal group methodology technique, 404–406 in qualitative data collection, 342–344
background in, 405 example of, 344
methodological considerations in, 405–406 key activities using, 343
results of, analysis and presentation of, 406 strengths and limitations of, 345t
vs. other consensus methods, 402t Observed score, in classical test theory, 159
Nominal group technique, of needs assessment process, 604t Observed Tasks of Daily Living-Revised (OTDL-R), 215
Nominal scale Observer presence, in data collection instrument reliability,
formatting questions for, 100 163
in data collection, 157, 158t Observer role, in data collection, 534–535
of measurement, 215–216 Occupational Circumstances Assessment-Interview and rating
variables measured by, 215t Scale (OCAIRS), 650
Noncompliance, regulatory, relationship between ethical con- Occupational Performance History Interview II (OPHI-II), 31
duct and, 469, 471 Occupational Performance History Interview II (OPHI-II)
Nonparametric statistics keyforms, 192–193
Chi-square, 244–248 Occupational therapist
goodness of fit test in, 244–246 as academic-practitioner collaborator, 52, 52f
independent and correlated samples of, 243–244 as practitioner-researcher, 51, 51f
Mann-Whitney U-test in, 249–250, 319 as research collaborator, 51, 51f
McNemar test in, 247, 319 postdoctoral training for, 49, 49f
one-way ANOVA, Kruskal-Wallis, 255, 318 professional responsibility of, use of research in, 46–47
multiple comparisons for, 255–256, 256t research-resistant, 53
power analysis in, 247–248 Occupational therapy
Sign test in, 250–251, 324 application of Q methodology in, 395, 397. See also Q
tests of independence in, 246–247, 247t methodology.
two-way ANOVA, Friedman test for, 257 assessments used in, testing of, 6
Wilcoxon signed rank test in, 251, 325 consensus methods in, 402–409. See also Consensus
Non–peer-reviewed venue, for research dissemination, methodology.
573–574 forensic, research priorities in, 407
Nonprint material, dissemination of information through, 574 information resources relevant to, 448–451
Nonprobability methods, of quantitative sampling, 521–522 measures vs. scores in, 177–179, 178t
Nonrandomized comparison group design, in quantitative limitations of, 179–181, 180f
research, 85–86, 85f, 86f meta-analysis articles relevant to, 292–294, 293t
Nonresponse rates, in survey studies, reduction of, 104 meta-analysis studies relevant to, 292–294, 293t
Norm referencing, with data collection instruments, 172–173 effect-size measures used in, 290, 291t
Normal distributions, 229 nature and outcomes of, evidence about, 6
Normal standard distribution, 229, 229f need for
Normative need, definition of, 594 assessment of, 591–605. See also Needs assessment
Normative research, 61 process.
Null hypotheses, 235–236 evidence of, 5
comparisons of two group means and, 248 objective measurement in, 181–183
decision outcomes in, 237t practice of
for MANOVA, 277 evidence-based, 4. See also Evidence-based practice
of one-way ANOVA, 252 (EBP).
types of, 238f research support of, 4–7. See also Research, in support of
Number needed to treat (NNT), in evidence-based practice practice.
calculation of, 673, 674t process of, inquiry into, 7
interpretation of, 674–676 Occupational therapy programs
Numbers development of, 607
assignment of, measurement in, 214 research as guide to, 608–609, 610t
making meaning from, 213 evaluation of, 607, 611–612
Numerator, 223 approaches to, 612–613
Nuremberg code, 477–478 experimental design in, 614, 615t
phases in, 616–618, 616f, 617f
pretest-posttest design in, 614, 615t
O qualitative strategies in, 614, 615t, 616
Objective measurement. See also Measures (measurement). quasi-experimental design in, 614, 615t
fundamental, 185 scientific research strategies in, 613–614, 615t
in occupational therapy and health care, 181–183 single-group design in, 614
720 Index
implementation of, 611 in quantitative research

needs assessment in, 609 issues affecting selection of, 515
planning of, 611 nomenclature for, 516
relationship between research and, 607–618 in research studies
scholarship of practice in, 608, 609f availability of information to, 566
Occupational therapy research. See Research entries. dropout of, 75–76
Occupational therapy scholarship, community-university part- ID sheets for, 558–559, 559f
nerships for, 632–641. See also Community-univer- identifiers of, 557–558
sity partnerships. protection of, 477–483
Occupational therapy theory. See also Theory(ies). recruitment of, 43, 432–433
development and testing of, practice explained by, 5–6 respect of, 478
Odds ratio retainment of, 432–433
interpretation of, 111–112 tracking of, 556–558, 557f
used in meta-analysis studies, 291t Participation, in gathering qualitative data, 342
Office of Research Integrity (ORI), 469, 471 example of, 344
Office of Special Education and Rehabilitation Services key activities using, 343
(OSERS), U.S., grants funded by, 495, 497, 498t Participatory action research (PAR), 330
One-tailed and two-tailed tests, in statistical hypotheses, conduction of, 330–331
240–241, 240f contributions of, 331
Online access, to published material, 438 resources on, 331
Online information retrieval, process of, 438 Participatory research, 620–629
Online project, in participatory research, 652–653 approach to, 621–622
Online questionnaires case 1: modification of assessment process, 650–652
advantages and disadvantages of, 93t practitioner’s account, 651
in survey research, 94 researcher’s account, 651
Online search strategies, by subject, 439–440 case 2: online project, 652–653
Open-ended interviews, in needs assessment process, 603t case 3: creation of new screening tool, 653
Open-ended questions practitioner’s account, 654
formulation of, 97 researcher’s account, 654
vs. fixed-response questions, 346t challenges of, 629, 645–647
Operant procedure, in Q methodology, 390 different agendas and priorities as, 646
Opinion, vs. data, 37 different work styles and settings as, 646–647
Oppenheim, scientific philosophy of, 13 knowledge and power differentials as, 645–646
Oral presentation overcoming, 647–650
at conferences commitment to, 634–635
non–peer-reviewed, 573 communication in, with honesty and sensitivity, 647–648
peer-reviewed, 571, 573 community-university partnerships in, commitment to,
of qualitative analyses, 385 634–635
Ordinal scale consumer voice in, 620–621
formatting questions for, 100–101 definition of, 621
in data collection, 157–158, 158t expertise in, acknowledging and respecting, 647
of measurement, 216–217 implementation of, principles in, 622–623
Organizational plan, for research, 429, 431t, 432, 433f importance of, 643
OT SEARCH, 450 in support of practice, 7
Outcome research knowledge-creation and evaluation in, 624–626, 626f
creation of, relevant for evidence-based practice, 697–704. need for, 620–621
See also Evidence-based practice (EBP). negotiation and reformulation of plans in, ongoing, 649–650
in support of practice, 7 perspectives of others in, suspending judgment while under-
standing, 648–649, 649t
principles of, 644–645
P promise of, 621
Paired t-test, parametric, 250, 250t, 321 purpose of, clarification of, 647
Panel study design, in longitudinal research, 132, 132f, 133t researcher-practitioner partnerships in, effective, 645–647
Paper data storage, of human subject research, 473 research-practice gap in, 620
Parametric statistics stakeholder’s voices in, 623–624, 625t
independent and correlated samples of, 243–244 step(s) in, 626–629, 627f
independent t-test in, 248–249, 249t, 317 analysis and planning as, 627–628
one-way ANOVA, 252–253, 253t critical reflection and analysis as, 627
multiple comparison tests for, 253–255, 254t data collection as, 628
paired t-test in, 250, 250t, 321 design choice as, 628
repeated measures ANOVA, 256–257, 257t developing service strategies as, 628
multiple comparison tests for, 257 implementing action as, 628–629
Participant(s) reflection and utilization as, 629
in consensus development methodology, selection of, 407 Path analysis, 321
in multiple-baseline design research, 143–144, 145f, 146f multivariate, 276–277, 277f
in qualitative research Pearson Product Moment correlation coefficient, 262–264,
appropriateness of, 524–525 322
determining number of, 526–527 definitional formula for, 262, 263f
strategies for selection of, 526 effect size of, 264
Index 721
matrix of, 263, 264t Practice and research. See also Research entries.
Spearman rank, 264 assessments in, 201–210
strength and significance of, 263–264 appropriate source of information in, 205–207
Peer debriefing, in gathering qualitative data, 353 assumptions reflected in, 201–202
Peer-review changes in, 209–210
in research dissemination, 476, 567, 570–573 conceptual model vs. actual measure in, congruence
at conferences, oral presentations and posters and, between, 204–205
571–573, 572f measurements in, 203–204
importance of, 573 social forces affecting, 202–203
process of, 570–571 trustworthiness in, 207–209
of grant applications, 502 Practices, accepted, in human subject research, 472–473
of journal articles, 570 Practitioner-researcher, 51, 51f
of scientific papers, 570–571 Precision, of data collection instrumentation, 171–172
purpose of, 567 Predictive validity
Percentiles, in measures of variability, 223 in psychometric studies, 29
Performance measurements of data collection instrumentation, 168
administration of, 540 Pretest-posttest design, in program evaluation, 614, 615t
in data collection, 539–540 Prevalence (P), calculation of, 59
Personal style and habits, in report writing, 584–586 Principle investigator, roles and responsibilities of, 431t
Personnel and organizational plan, for research, 429, 431t, 432, Principles of Belmont, 478
433f Priori test (planned comparisons), for one-way ANOVA, 255
Personnel preparation, for data collection, 545 Privacy protocols, existing database compliance with, 122–123
Phase lengths, in single-subject research, 149–150 Private agencies, research dissemination to, 576
Phenomena under study Private foundations, grants awarded by, 491, 493
in descriptive research, 59 Probability
in qualitative research, 327 in inferential statistics, 232–233
assumptions about, 22–23 in logistic regression, computation of, 272
in quantitative research, 20–21 Probability methods, of quantitative sampling, 519–521
Phenomenology, in qualitative research, 22, 335–337 Problem(s), focus and formulation of, in meta-analysis,
conduction of, 337 284–285, 284t
contributions of, 337–338 Process of consent, for human subject research, 479
Philosophical foundations, of scientific inquiry, 10–19. See Productivity, as data management issue, 553t
also Scientific inquiry, philosophical foundations of. Professional audiences, research dissemination among,
Phrases, in literature searches, 445 566–567
Physiological drive, need as, 593 Professional considerations, in data collection, 545–546
PICO questions, in evidence-based practice, 664, 665t Professional development grants, 487
specific, 664–665 Professional organizations
Pilot research, in grant applications, 502 direction of, in identification of research questions, 453–454
Piloting questionnaire, in survey research, 101 grants awarded by, 491, 492
Plagiarism, in research, 471 Professional publication(s). See also Scientific papers.
Plan(s), in participatory research, reformulation of, 649–650 dissemination of information through, 573–574
Planning analyses, in grant applications, 504 non–peer-reviewed, 573–574
Planning components, in research, integration of, 426–427 of research, 436
Policy(ies) repetitive and fragmentary, 474
governing authorship, 475 selection of, in report writing, 583–584
governing integrity, in human subject research, 469 Professional recognition, necessity of research for, 4
in grant applications, 489–490 Professional responsibility, in research, 46–56
Policy documentation, in data management, 550 Professional support, necessity of research for, 4
Political orientation, of needs assessment, 592–593, 592f Program staff, database, utilizing help of, 125
Political process, need as, identification of, 595–596 Project coordinator, roles and responsibilities of, 431t
Popular press, release of research findings to, 577 Project information, 548
Population(s), dependent and independent variables within, Project initiation, issues during, 563
probability of causal effect on, 69–70, 70f Project timeline, in data management, 555–556, 556t
Positivism, logical Project-based methods, qualitative data quality associated with,
critique of, 14 354–356
in approach to research, 12–14 Proposal(s), development of, considerations during, 562
laws governing, 13 Proximity connectors, in computer-assisted qualitative data
Postdoctoral training, 47–48, 49, 49f systems, 367–368
Poster presentation. See Scientific poster presentation. Psychiatric clients, family burdens of, case illustration of, 464,
Post-hoc tests, for one-way ANOVA, 253–255, 254t 465f, 466
Postmodernism Psychometric properties, of data collection instrumentation,
critiques of, 16–17 171–173
in scientific inquiry, 16–17, 17t Psychometric studies, 29–30
Post-qualification and limiting, in literature searches, instrument development in, 31
446–447 Public access, online, to catalogs, 438
Power analysis Public attitudes, toward disability, case illustration of, 462–464
for regression, 268 Public officials, research dissemination to, 576
statistical, 239, 247–248, 251, 258 Publication(s). See Professional publication(s).
factors influencing, 238–239 Publication practices, in human subject research, 473–474
722 Index
Published material, online access to, 438 dependability of, methodological rigor and, 532–533
Published research, replication of, 39 gathering of, 341–357
Purposive sampling major strategy(ies) for, 342–351
in qualitative research, example of, 527 focus group interviews as, 347–349, 348t
in quantitative research, 522 example of, 350
example of, 523 in-depth interviews as, 344–345
key informant interviews as, 348t, 349
observation as, 342–344, 345t
Q participation as, 342
Q methodology, 389–399 semistructured and structured interviews as, 345–347,
application of, in occupational therapy, 395, 397 346f, 346t, 348t
case examples of, 391, 393f unstructured interviews as, 345
data analysis in, 394–395, 396f written documents and material objects as, 349–351
findings in, interpretation of, 395 quality of, way(s) of strengthening, 351–356
history of, 389 audit trail as, 354
limitations of, 398–399 entering field study site as, 354–356
operant subjectivity of, 390 exiting field study site as, 356
phases in, 390–395 interviewer training as, 352
Q-sort pack in project-based methods as, 354–356
administration of, 391–394, 393f prolonged engagement in field as, 352–353
development of, 390–391 reflexivity as, 353
sample size for, 393 stakeholder checks as, 353–354
strengths of, 397–398 triangulation as, 353
subjectivity of, 390 textual description and theory grounded in, 24–25
theoretical bases of, 390–395 Qualitative data analysis (QDA) software programs, 365
use of, 389, 391 coding examples in, 367–368
in service evaluation, 397 Qualitative research, 20t, 22–25, 326–338
perceptions about, 389–399 authenticity and groundedness in, 24
vs. qualitative and quantitative approaches, 390, 392t bias in, management of, 352
Q-sort method, of data collection, 538 contributions of, to understanding people’s worlds, 374
Q-sort pack critical theory in, 334–335
administration of, 391–394, 393f conduction of, 335
development of, 390–391 contributions of, 335
Qualitative data data gathering in. See Qualitative data, gathering of.
analysis of, 358–370 epistemologies (traditions of thought) in
active reading in, 364t diversity of, 328–330
approaches to, 372–387 major, 328–338
goals of description and explanation in, 373–374 nature of, 330
inductive process in, 373 ethnographic approach to, 331–332
intertwining of data collection and analyses in, conduction of, 332
374–375 contributions of, 332–333
step(s) in, 375–383 features of, 328, 341, 358–359
1: formal analysis or interpretation of one data grounded theory in, 333
source, 378–380 conduction of, 333
2: code selection for further analysis, 380–381 contributions of, 333–334
3: compare and contrast data sources utilizing partic- meta-analysis and, 295–296
ular code, 381 meta-data in, examples of, 359
4: drawing conclusion about coding strategy or aris- nature of, 326–337
ing data pattern means, 381–382 norms of interest in, 329
after accumulation of processed data, 377 participatory action research in, 330
enhancing rigor through, 382–83 conduction of, 330–331
importance of data processing and, 375–377 contributions of, 331
code directory in, 364t resources on, 331
computer-assisted, 364–369. See also Computer-assisted phenomena under study in, 327
qualitative data (CAQDAS). assumptions about, 22–23
conceptual labels in, 360, 362, 363f, 364t phenomenology in, 22, 335–337
index codes in, 360, 362, 363f, 364t conduction of, 337
interdependence of, 363 contributions of, 337–338
memos in, 362–363, 364t questions addressed by, 327–328
interdependence of, 363 results of
principles in, 359–363, 361f interpretation of, 378
quality assurance in, 363–364 oral presentation of, 385
reporting findings in, 383–386 poster presentation of, 385
basic guide to, 385–386 reporting, 383–386
rigor, flexibility, challenge, and reward in, 376 written presentation of, 582–583
summary of, 363, 364t for academic audiences, 386
uniqueness of, 372–375 rigor in, 36, 376
Index 723
sampling in, 524–528 illustrating sample size issues, 524

adequacy of data in, 525 using specific sampling strategies, 523
appropriateness of participants in, 524–525 exclusion criteria of, 518
depth vs. breadth and, 527 external validity of, 518–519
domain analysis and, 527–528 inclusion criteria of, 518
examples of, 527 kinds of, 519
gaining access and, 527 level of significance in, 524
participants in nonprobability methods of, 521–522
appropriateness of, 524–525 parameters of target population and, 518
determining number of, 526–527 probability methods of, 519–521
strategies for selection of, 526 purposive, 522
unit of analysis in, 525–526 quota, 522
strategies of, in program evaluation, 614, 615t, 616 size of
subjective reality in, understanding and representing, 23 calculation of, 524
vs. quantitative research, 25, 26t, 373–374 determination of, 522
Quality assurance, in qualitative data analysis effect of, 522–523
coding and retrieving for, 363–364 factors affecting, 524
inter-rater reliability for, 364 statistical power and, 522
Quality control procedures, in data analysis, 107–108 snowball (network), 522
Quantification, in quantitative research, 22 stratified random, 521
Quantitative and qualitative traditions, used in mixed methods systematic, 521
designs, 413, 413f theory testing in, 21–22
Quantitative data validity of
analysis of, 44 evaluation of, 70–75
collection of, 530 external, 75–76
approach to, 530–531 type I error in, 70–72, 71f
dependability of, methodological rigor and, 531–532 type II error in, 72–75, 73f, 74f, 74t
levels of, 530 vs. qualitative research, 25, 26t, 373–374
terminology for, 531 Quasi-experimental designs
Quantitative instruments, in data collection, 155. See also Data in program evaluation, 614, 615t
collection instruments. in qualitative research, 65
Quantitative research, 20–22, 20t Quasi-experimental research studies, 26
designs in, 65–89 examples of, 27
descriptive, 58–63. See also Descriptive quantitative Querying, online, 438
designs. Questionnaire(s)
experimental, 65–70 advantages and disadvantages of, 93t
definition of, 65–66 directly administered, 92, 94
explanation of, 66–70 formatting of, 97, 98f, 99f
not involving randomization, 85–88 mailed, 92, 92f
case-control design in, 87, 87f systematic approach to, 106, 106t
cross-sectional design in, 86–87, 86f online, 94
multigroup cohort design in, 87–88, 88f piloting, 101
nonrandomized comparison group design in, 85–86, 85f, potential disadvantage of, 92
86f unstandardized, in data collection, 538–539
objectivity in, 21 volitional
phenomena under study in, assumptions about, 20–21 development of, 166
quantification and statistics in, 22 items in, 167t
randomized Questionnaire/interview. See also Interview(s).
completely randomized factorial designs in, 78–79, 79f in survey research, development of, 95–102
crossover (counterbalanced) design in, 83–85, 84f, 85f debriefing interview in, 102
interim repeated measures, post-post tests, and long-term defining and clarifying variables in, 95
follow-up tests in, 76, 76f, 77f field pretesting in, 101–102
mechanisms of change in, 77–78, 78f focus groups in, 101
multiple dependent variables and same independent vari- formatting principles in, 97–99, 99f, 100f, 100t
able in, 76–77, 77f, 78f formatting questionnaire in, 97, 98f, 99f
randomized controlled trials in, 80–81, 81t–82t, 83f formulating questions in, 95–97, 96f
cluster, 82–83, 84f for different scales, 100–101, 101f
tests of fidelity in, 78, 78f Quota sampling, in quantitative research, 522
treatment by levels design in, 79–80, 80f
variations on, 76–85
results of, written presentation of, 582 R
sampling in, 517–524, 519 r, critical values of, 307t
cluster, 521 r calculation, in continuous data effect sizes, 677, 677t
convenience, 521 r interpretation, in continuous data effect sizes, 677, 679
errors of, 519 Race issues, in research studies, 516
examples of Random error, in data collection, 159
illustrating bias in, 520 Random sampling
criteria for, 233
724 Index
Random sampling (continued) Reliability coefficient, 160–161

in quantitative research, 519–521 Repeated measures ANOVA, 324
errors of, 519 in longitudinal research, 136–137, 137t
example of, 523 parametric, 256–257, 257t
stratified, 521 multiple comparison tests for, 257
Randomized controlled trials, 80–81, 83f Replication, of research, 566
cluster, 82–83, 84f Report writing, 578–589
CONSORT criteria for, 81t–82t abstract in, 579, 580
Randomized designs, in quantitative research clarification of message in, 587
completely factorial, 78–79, 79f description of research design, study, and analysis in, 582
controlled trials of, 80–81, 81t–82t, 83f development of personal style and habits in, 584–586
crossover (counterbalanced), 83–85, 84f, 85f discussion and conclusion in, 583
interim repeated measures, post-post tests, and long-term drafting, sequencing, and revising in, 588–589
follow-up tests in, 76, 76f, 77f feedback in, seeking and using, 589
mechanisms of change in, 77–78, 78f for particular audience, 586–587
multiple dependent variables and same independent variable identification of story plot in, 587–588
in, 76–77, 77f, 78f inspiration in, 588
randomized controlled trials in, 80–81, 81t–82t, 83f introduction, statement of purpose, and significance in,
cluster, 82–83, 84f 579–580
tests of fidelity in, 78, 78f literature review in, 580–581
treatment by levels design in, 79–80, 80f mind-mapping and abstracting in, 587
variations on, 76–85 paper structure and format in, 578–583
Range, in measures of variability, 223 process of, 584–589
Rank in scale, modified, 201 results or findings in, 582–583
Rasch analysis, 322–323 selection of venue for publication in, 583–584
Rasch formulas, 184–185 setting aside and structuring time in, 584, 585f
advanced, 186 statement of scholarly aim in, 582
core of, 185 statement of scholarly problem in, 581–582
many-faceted, 186 strategies in, 584
Rasch model, 184–186 Research
measurement statistics in, 186, 187t–188t, 188–189 applied, 31–32
Rater bias, in data collection reliability, 163 basic, 30–31
Rater error, in data collection, reduction of, 160 characteristics of, 36–39
Rater/observer, effects of, on data collection reliability, 163 communality in, 38–39
Ratio scale criticism of, 566
in data collection, 158t, 159 data from, management and analysis of, 43–44, 44f. See also
of measurement, 217 Data analysis.
Raw data, 218 description of analysis of, in report writing, 582
calculation of mean from, 220 descriptive, 58–63. See also Descriptive quantitative designs.
Reading, active, in qualitative data analysis, 364t differentiation in, 20, 20t
Ready-reference, online, 437 dissemination of, 44–45, 565–577
Realism, subtle, in consensus methodology, 401 among collaborating organizations, 575–576
Reasoning, logical, in research, 37–38, 39f among consumer and community groups, 575–576
Records, in data collection, review of, 541 among research participants, 574–575
Recovery, Barthel’s operational definition of, 203 approach to, 575
Recruitment efforts, funding of, 433 nature and role of, 565–566
Recruitment materials, development of, 433 to government and private agencies, legislative bodies,
Reflexivity, qualitative data quality associated with, 353 and public officials, 576
Regression, 264–273 to popular press and media, 577
accuracy of prediction in, 266 to targeted brochures, 576–577
analysis of to web sites, 576–577
assumptions for, 265, 266f venue(s) for, 566–557
power of F-test for, 312t choice of, 567
analysis of covariance and, 271–272, 271f non–peer-reviewed, 573–574
analysis of variance and, 266–268, 267t, 268t peer-reviewed, 567, 570–573
linear professional and scientific audiences as, 566–567
power analysis for, 268 stakeholder audiences as, 574
simple, 265, 325 ethical review of, 42
logistic, 272–273, 273t, 318 experimental, 25–26, 27
multiple, 268–270, 320 field, 27–28, 29
selection of variables in, 260f, 269–270, 270f full circle of, 45
Regression equation, calculation of, 265 funding for, 486–513. See also Funding agencies; Grant
Regression line, 265 entries.
Regulations, in grant applications, demonstration of, 498–499 high-quality
Reliability collecting, managing, storing, and analyzing data in,
in psychometric studies, 30 433–435
inter-rater, in qualitative data analysis, 364 communicating with funding agency in, 435
Index 725
creating links to sample sources in, 433 quasi-experimental, 26, 27

data collection and analysis in, 425–426 range of, 20–33
dealing with challenges in, 435 relationships of therapy and practice with, 7–8, 8f
decisions regarding methodology in, 423–424 replication of, 566
developing recruitment materials in, 433 researchers in, 1–3, 1f–3f
dissemination of, 435–436 responsibility and, 46–47
consumer-focused, 436 results of, interpretation of, 44
ensuring sample representativeness in, 433 review of literature in, 40–41
funding recruitment efforts in, 433 rigor in, 36–37
grant writing in, 428 factors influencing, 91
identification of research questions in, 422–423 single-subject, 26–27, 28, 140–153. See also Single-subject
implementation of, 420 research.
integration of planning components in, 426–427 skepticism and empiricism in, 37
literature search in, 421–422 statistics in. See Statistics.
location of sample in, 433 survey, 28, 30, 91–108. See also Survey research.
management plan in, 429, 430t transformative, 32
need for planning in, 421–435 uncertainty and, 46–47
obtaining ethical review in, 429 writing up. See Report writing.
personnel and organizational plan in, 429, 431t, 432, 433f Research advocates, 49–50, 54t
planning and budgeting of resources in, 427–428 Research assistant
presentation of, 435 job description of, 432
publication of, 436 roles and responsibilities of, 431t
recruiting and retaining participants in, 432–433 Research collaborators, 48, 50, 50f, 51, 51f, 54t
sampling in, 424–425 Research consumers, 48–49, 54t
team roles and responsibilities in, 431t Research data. See Data entries.
traditions of inquiry in, 420–421, 422f Research design(s)
writing research plan in, 427 basic, 25–30
human subject, 468–484. See also Human subject research. checklist for, 460t
impact of, scientific inquiry regarding, 18–19 choice of, 424, 452
implementing procedures in, 42–43 decision-making tools in, 459–460, 459t
in evidence-based practice, 4. See also Evidence-based prac- in grant applications, 503
tice (EBP). description of, in report writing, 582
in support of practice, 4–7 descriptive. See Descriptive quantitative designs.
developing and testing theory explaining practice in, 5–6 mixed methods, 411–418. See also Mixed methods
foundational knowledge in, 4–5 design(s).
inquiry into therapy processes in, 7 quantitative, 65–89. See also under Quantitative research;
outcomes research in, 7 specific design.
participatory research in, 7 role of creativity in, 454–459
providing evidence about nature and outcomes of therapy selected study samples for, 515–517
in, 6 gender, race, and ethnicity issues in, 516
service needs in, 5 in qualitative research, 524–528. See also Qualitative
studies of clinical reasoning in, 6–7 research, sampling in.
studies testing assessments used in therapy in, 6 in quantitative research, 517–524. See also Quantitative
influence of item response theory on, 177–198. See also research, sampling in.
Item response theory. issues affecting, 515
interrelated activities in, 39–45, 39f steps in, 516–517, 516f
logical reasoning in, 37–38, 39f sophistication of, in needs assessment, 591–592, 592f
longitudinal, 127–139. See also Longitudinal research. survey, 91–108. See also under Survey research.
naturalistic observation in, 27–28, 29 Research focus, narrowing, in development of research ques-
necessity of, 1–8 tions, 454, 454t
participatory, 620–629. See also Participatory research. Research grant, 486. See also Grant entries.
philosophical foundations of, 10–19. See also Scientific Research hypothesis. See Hypothesis testing.
inquiry, philosophical foundations of. Research methodology
practice and. See Practice and research. checklist for, 460t
professional recognition and support through, 4 choice of, decision-making tools in, 459–460, 459t
professional responsibility in, 46–56 decisions regarding, 41–42, 423–424, 452–453
profession’s obligation to, 4 qualitative and quantitative, 20–25. See also Qualitative
psychometric, 29–30, 31 research; Quantitative research.
published role of creativity in, 454–459
meta-analysis of, 281–296. See also Meta-analysis, of Research paper
published research. structure and format of, 578–583
replication of, 39 abstract in, 579, 580
purposes of, 30–32 description of design, study, and analysis in, 582
qualitative. See also Qualitative research. discussion and conclusion in, 583
vs. quantitative research, 25, 26t introduction, statement of purpose, and significance in,
quantitative. See also Quantitative research. 579–580
vs. qualitative research, 25, 26t literature review in, 580–581
726 Index
Research paper (continued) methodological

procedure of study in, 582 and dependability of qualitative data, 532–533
results or findings in, 582–583 and dependability of quantitative data, 531–532
statement of scholarly aim in, 582 of data collection, 543
statement of scholarly problem in, 581–582
writing process for, 584–589. See also Report writing.
Research participants, information sharing with, 574–575 S
Research procedures, implementation of, 42–43 Sample bias, examples illustrating, in quantitative research,
Research process 520
interrelated activities in, 39–45, 39f Sample query, in literature searches, 446, 446f, 447f
scientific inquiry regarding, 18 Sample representativeness, in research, 433
Research producers, 47–48, 54t Sample size issues, in quantitative research, 524
Research proposal, written, 42, 427 Sample sources, in research
Research questions, identification of, 41, 422–423, 452–466 creation of links to, 433
case illustrations in, 462–466 location of, 433
family burdens of mentally ill offenders in, 464, 465f, 466 Sampling, in research, 424–425
studying attitudes using conjoint analysis in, 462–464 Sampling distribution, 228–230, 229f, 230f
clinical experience in, 453 in inferential statistics, 233–234
direction of professional organizations and funding agencies Sampling error, in inferential statistics, 233
in, 453–454 Sampling error of mean, in inferential statistics, 234
literature reviews in, 454 Sampling plan, statement of, 434
correct, 460–461 Sampling strategies
synthesizing discovered, 461-462, 462f, 463t in quantitative research, 523
research focus in, 454, 454t in survey study, 102–104, 103f
role of creativity in, 454–459. See also Creative problem- Scales of measurement, 214–218. See also specific scale.
solving (CSP) process. in data collection instrumentation, 157–159, 158t
tools for decision-making in, 459–460, 459t statistics in relation to, 217–218, 218f
Research resources, planning and budgeting of, 427–428 types of, 214–217
Research roles, 47–51 Scholar(s), in related disciplines, availability of information to,
choosing and developing, 51, 53–56 566
contributions of, 50–51 Scholarly aim, statement of, in report writing, 582
related training, education, expertise, and activities in, 54t Scholarly attitude, to research, 37
strategies for, 54–56 Scholarly problem statement, in report writing, 581–582
Research specialist, roles and responsibilities of, 431t Scholarship
Research team in community-university partnerships, commitment to, 635
responsibilities of, 431t in grant applications, demonstration of, 500–502
role of, 431t in programs of service, 608, 609f
in human subject research, 483 School Function Assessment (SFA), 203
selection of, grant funding and, 490–491 development of, 204
Researcher(s) Scientific audiences, research dissemination among, 566–567
in occupational therapy, 1–3, 1f–3f Scientific community, disseminating research findings among,
practitioner collaboration with, 643–655. See also 44–45
Participatory research. Scientific inquiry, philosophical foundation(s) of, 10–19
effective, 645–647 classicism in, 10–12, 11f
overcoming challenges of, 647–650 continuum of ideas in, 17t
report writing by, 578–589. See also Report writing. critical modernism in, 14–16
process of, 584–589 modernism in, 12–14
scientific inquiry regarding, 18 postmodernism in, 16–17
writing strategies of, 584 regarding research impact, 18–19
Research-practice gap, in participatory research, 620 regarding research process, 18
Research-resistant practitioners, 53 regarding researchers, 18
Resources regarding theory, 18
in community-university partnerships, responsibility for, Scientific knowledge
636–637 positivistic, from absolute truth to, 12–14. See also
planning and budgeting of, 427–428 Positivism.
Respect of persons, in human subject research, 478 view of, 17t
Respondent, in research studies, 516 Scientific message, clarification of, in report writing, 587
Response rate, in survey studies, 104–105 Scientific method, origins of, in classicism, 10–12, 11f
Responsible conduct, in human subject research, benefits of, Scientific papers. See also Professional publication(s).
471 peer-review of, 570–571
Review, of grant application, 507–508 presentation of, 571
Revisions, of research paper, 588–589 Scientific poster presentation
Reward, in qualitative analysis, 376 at conferences, peer-reviewed, 571–573, 572f
Rigor of qualitative analyses, 385
enhancement of, through qualitative data analysis steps, of research, 435
382–383 Scientific strategies, in program evaluation, 613–614, 615t
in qualitative analysis, 376 Scientific trends, in grant applications, 490
in research, 36–37 Scores
factors influencing, 91 consistency in, expectation of, 208
Index 727
from health care instruments, 179–180, 180f Skill(s), cultural, in community-university partnerships,
lack of unidimensionality of, 180–181 639–640
test-dependent, 180 Skill scores, communication and interaction, for experimental
irrelevant factors affecting, 205 group, 135–136, 135f, 136f, 136t
vs. measures, limitation of, 179–181 Snowball (network) sampling
Screening tools, in participatory research, creation of new, 653 in qualitative research, example of, 527
practitioner’s account, 654 in quantitative research, 522
researcher’s account, 654 example of, 523
Search engines, web, 447–448 Social environmental factors, in consensus methodology, 408
Search query, literature, 444 Social forces, affecting assessment in practice and research,
Search strategies, online, by subject, 439–440 202–203
Search syntax, literature, 444 Social good philosophy, needs assessment grounded in, 596
Search terms, literature, 443 Social indicators, of needs assessment process, 604t
Search tips, web, 448 Social issues
Self-administered surveys, in needs assessment process, 603t in community-university partnerships, commitment to,
Self-help organizations, grants awarded by, 493–494 633–634
Self-rating scales, in data collection, 538 in participatory research, 622–623
Self-report measures, in data collection, 538–539 Social validation procedures, incorporation of, into single-
Self-reports, administration of, in data collection, 539 subject research design, 152
Semantic connectors, in computer-assisted qualitative data sys- Social validity, of evidence-based practice, 700–701
tems, 366–367, 368f relationship between characteristics of research evidence
Semistructured interviews and, 701
in data collection, 536–538 verification of, 701
in needs assessment process, 603t Socioeconomic data, 117t
in qualitative data collection, 345–347, 346f, 346t, 348t Software, facilitating computer-assisted qualitative data analy-
Sensitivity, of data collection instrumentation, 172 sis, 369–370
Sequencing, in report writing, 588–589 Solution, need as, 594
Sequencing questions, in survey research, 99–100 Sound, in computer-assisted qualitative data analysis, hard/
Sequential explanatory design, in mixed methods studies, 416, analog-to-digital conversion of, 365t
417t Spearman rank correlation coefficient, 264, 325
Sequential exploratory design, in mixed methods studies, 416, Specificity, of data collection instrumentation, 172
417t Split-half reliability, of data collection instrumentation,
Sequential transformative design, in mixed methods studies, 161–162
416, 417t Sponsoring institution, negotiation with, grant writing and,
Service evaluation, use of Q methodology in, 397 490
Settings, multiple-baseline designs across, 145–146, 147f Stability, of data collection instrumentation, 161
Shared commitment, in community-university partnerships, Stakeholder audiences, research dissemination among, 574
633 Stakeholder checks, qualitative data quality associated with,
example of, 635 353–354
Sickness Impact Profile (SIP) Dysfunction, 215–216 Stakeholder involvement
Sign test, nonparametric, 250–251, 324 in needs assessment, 592, 592f, 602
Single-group design, in program evaluation, 614 in participatory research, 622
Single-subject research, 26–27, 140–153 continuum of, 623–624, 625t
A-B design in, 140–141, 142f in research studies, 516
variation of, 141–143, 142f Stakeholder’s voices, in participatory research, 623–624, 625t
A-B-C design in, 141–143, 142f Standard deviation, in measures of variability, 223–225, 225t,
alternating-treatments design in, 146–149 226t, 227f
example of, 148 Standard error of estimate (SEE), calculation of, 266
data analysis of, 149–151 Standard error of measurement (SEM)
data collection in, 149, 150f calculation of, 173
data graphs in, 149 of instruments, 173
designs in Standard selection method, of multiple regression, 269, 269f
common, 140–149 Standardization, in measurement error reduction, 160
incorporation of social validation procedures into, 152 Standardized scores, for data collection instrumentation, 173
questions to answer in, 153 Standardized tests
variations in, 149 administration of, 540
examples of, 28 in data collection, 539–540
issue of generality in, 152–153 Statement of purpose
multiple-baseline designs in, 143 in grant applications, 502–503
across behaviors, 143, 144f in report writing, 579–580
across participants, 143–144, 145f, 146f Statement of research, in report writing, 581–582
across settings, 145–146, 147f Statement of sampling plan, 434
phase lengths in, 149–150 Statistical reference tables, 299t–313t
statistical significance in, 151 Statistician, roles and responsibilities of, 431t
therapy ball study in, 141–142, 141f, 142f Statistics, 213–230. See also Data entries.
variables in, definition of, 149 analysis of
visual data analysis in, 150–151, 151f, 152f in longitudinal research, 134–135
withdrawal designs in, 142f, 143 in quantitative research, 22
Skepticism, in research, 37 power, 239, 247–248, 251, 258
728 Index
Statistics (continued) formatting principles in, 97–99, 99f, 100f, 100t

as tool of quantitative science, 213–214 formatting questionnaire in, 97, 98f, 99f
common, summary of, 314–325 formulating questions in, 95–97, 96f
correlation in, 262–264, 263f for different scales, 100–101, 101f
descriptive, 218–228 piloting questionnaire in, 101
central tendency and variability in, graphical presentation sequencing questions in, 99–100
of, 225–226, 227f, 228, 228f rigor in, 28, 36
frequency distribution in, 218–222, 219t factors influencing, 91
measures of variability in, 222–225, 224t study of
inferential, 232–235 carrying out survey in, 105–106, 106t
analysis of, 243–279 implementation of, 102–106
bivariate, 244 response rate in, 104–105
confidence intervals in, 234–235, 235t sampling strategies in, 102–104, 103f
hypothesis testing in, 235–242. See also Hypothesis telephone interviews in, 94, 96f
testing. Survey variables, defining and clarifying, 95
multivariable, 244 Sustainability, in community-university partnerships, responsi-
multivariate, 244 bility for, 637
nonparametric, 243. See also Nonparametric statistics. Syntax, search, in literature search, 445
parametric, 243. See also Parametric statistics. Systematic error, in data collection, 159
probability in, 232–233 Systematic sampling, in quantitative research, 521
sampling distribution in, 233–234 errors of, 519
sampling error in, 233
sampling error of mean in, 234, 235t
types of, 245t
univariate, 244 T
investigators use of, 214 Tag-and-retrieval, of text, in computer-assisted qualitative data
Rasch, 186, 187t–188t, 188–189 systems, 366–368
regression in, 264–273. See also Regression. Targeted brochures, research dissemination to, 576–577
sample distributions in, 228–230, 229f, 230f Team meetings, in data management, 561
significance of, in single-subject research, 151 Team member selection, as data management issue, 553t
users and consumers of, 213 Teamwork, in computer-assisted qualitative data systems,
Stepwise method, of selection of variables in multiple regres- 369
sion, 269–270, 270f Telephone interviews
Story plot, identification of, in report writing, 587–588 advantages and disadvantages of, 93t
Stratified random sampling, in quantitative research, 521 approach to, 105–106
example of, 523 in survey research, 94, 96f
Stroke Impact Scale, 203 survey format of, 98f
Structured interviews Tester error, in data collection, reduction of, 160
in data collection, 536–538 Test-retest, of data collection instrumentation, 161
in qualitative data collection, 345–347, 346f, 346t, 348t Tests of fidelity, in quantitative research, 78, 78f
Subject, in research studies, 516 Tests of independence, nonparametric, 246–247, 247t
Subject pool, in research studies, 515 Text, in computer-assisted qualitative data systems
Subject searching, online, 437–438 hard/analog-to-digital conversion of, 365t
strategies in, 439–440 tag-and-retrieval of, 366–368
Subjective reality, in qualitative research, 23 Text-based searches, in computer-assisted qualitative data sys-
Support letters, for grant application, 507 tems, 366
Surrogate permission, in human subject research, 479 Textual description and theory, in qualitative research, 24–25
Survey formatting, principles of, 100t Tham, Kerstin, 2–3, 3f
Survey forms Theoretical basis for study, in grant applications, identification
format of, 98f, 99f, 101f of, 499
sample question from, 100f Theory(ies)
Survey research, 28 as network of explanations, 11
advantages of, 91 changing, 15–16
data analysis of, preparation of, 106–108, 107f in generation research questions, design, and methods, 453
data collection in, 91–95 in quantitative research, testing of, 21–22
advantages and disadvantages of, 93t nature of, 17t
selecting and combining, 95 occupational therapy, developing and testing of, 5–6
design in, 91–108 scientific inquiry regarding, 18
directly administered questionnaires in, 92, 94 Theory building, in computer-assisted qualitative data systems,
examples of, 30 369
face-to-face interviews in, 94 Therapy ball study, in single-subject research, 141–142, 141f,
key dimensions in, 91 142f
mailed questionnaires in, 92, 92f Three-phase model, of needs assessment, 597t, 598–599
online questionnaires in, 94 advantages and disadvantages of, 601t
questionnaire/interview in, development of, 95–102 Time, setting aside and structuring, in report writing, 584,
debriefing interview in, 102 585f, 586
defining and clarifying variables in, 95 Time series design, in longitudinal research, 131–132, 131t,
field pretesting in, 101–102 133t
focus groups in, 101 analysis of, 134
Index 729
Timeline University setting, grants within, 494

development of, in grant applications, 504, 506 Unstructured interviews, for gathering qualitative data, 345
project, in data management, 555–556, 556t U-test, Mann-Whitney, 249–250, 319
Tools, facilitating computer-assisted qualitative data analysis, UTMB/TLC Interventions Trial Quality Form, 287f–289f
369–370
Toulmin, scientific philosophy of, 15
Town hall meetings, data collection from, 540–541
Traditions of inquiry, in research, 420–421, 422f V
Traditions of thought (epistemologies), in qualitative research Validity, in psychometric studies, 29
diversity of, 328–330 Variability, measures of. See Measures of variability.
major, 328–338. See also specific epistemological tradition. Variables, in data collection, 214, 215, 215t
nature of, 330 graphical presentation of
Training continuous, 219–220, 221f, 221t
as data management issue, 553t discrete, 219, 219t, 220f
interviewer, qualitative data quality associated with, 352 Variance, in measures of variability, 223, 224t
Training grants, 487 Verbal communication, in data management, 551
Transformative research, 32 Veridicality, in ecological validity, 702
Trend study designs, in longitudinal research, 130–131, 131, Verisimilitude, in ecological validity, 702
133t Visual analog scales, in data collection, 538
analysis of, 134 Visual analysis, of single-subject research data, 150–151, 151f,
Triangulation, qualitative data quality associated with, 353 152f
True score, in classical test theory, 159 Voice representation
Truncation, in literature searches, 445 in assessment of practice and research, 205–207
Trustworthiness, in assessment of practice and research, in participatory research
207–209 consumer, 620–621
Truth stakeholder, 623–624, 625t
absolute, positivistic science and, 12–14 Volitional questionnaire
search for, scientific philosophy in, 11–12 development of, 166
t-Test(s), 248–251 items in, 167t
critical values of, 302t–303t
for ␣1, power of, 310t
formula for, 248 W
independent (parametric), 248–249, 249t, 317 Waldinger Fischer, Heidi, 50, 50f
Mann-Whitney U-test (nonparametric), 249–250, 319 Web directories, 448
paired (parametric), 250, 250t, 321 Web search engines, 447–448
Sign test (nonparametric), 250–251 Web sites
Wilcoxon signed rank test (nonparametric), 251, 325 as information resources, 451
Tukey’s honestly significant difference (HSD) test, for one-way research dissemination to, 576–577
ANOVA, 254, 254t Well-written document, presentation of, in grant applications,
Tuskgee experiment, 477 502
Westmead Home Safety Assessment (WeHSA), inter-rater reli-
ability of, 164
U White, scientific philosophy of, 14
U value, calculation of, 249 Wilcoxon rank-sum test. See Mann-Whitney U-test.
U3 value, used in meta-analysis studies, 290–291, 291f Wilcoxon signed rank test, 251, 325
U.C.L.A study (1970), of adults with developmental disabili- Withdrawal designs, in single-subject research, 142f, 143
ties, qualitative data analysis in, 359–360, 361f, Within-subject factor, in repeated measures testing, 136, 137t
362–363 Wittgenstein, scientific philosophy of, 16
Unit of analysis, in qualitative research samples, 525–526 Work, quality of, as data management issue, 553t
United Kingdom, International Regulations governing research Work styles and settings, in participatory research, challenges
in, 470 of different, 646–647
United Kingdom Foundations, grants awarded by, 493 Worker Role Interview (WRI), 650
United Kingdom Government, grants awarded by, 497 World Federation of Occupational Therapists, grants awarded
United States, International Regulations governing research in, by, 492
470 World Wide Web (WWW), 439
United States Department of Education criteria, for grant search of, 447–448
reviews, 511t–512t Writing strategies, of researcher, 584
United States Government, grants awarded by, 494–495, Writing style and habits, of researcher, 584–586
496t–497t Written communication, in data management, 551
United States Private Foundations, grants awarded by, 493 Written reports, of qualitative analyses, for academic audi-
Univariate statistics, 244 ences, 386
University of Illinois (UIC) Center for Capacity Building for Written research proposal, 42, 427
Minorities with Disabilities Research (CCBMDR),
488
University of Illinois (UIC) consent form, for human subject
research participation, 482f Z
University of Illinois (UIC) Fear of Falling Measure, 191 z-score, 229–230, 230f
University of Illinois (UIC) written and verbal assent, to calculation of, 229
human subject research participation, 481f mean and, normal curve between, 299t–301t

G. KIELHOFNER (2016), Research in Occupationnal Therapy, Methods of Inquiry For Enhancing Practice, FA Davis Company

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

G. KIELHOFNER (2016), Research in Occupationnal Therapy, Methods of Inquiry For Enhancing Practice, FA Davis Company

Uploaded by

Copyright:

Available Formats

00Keilhofner(F)-FM 5/19/06 4:52 PM Page i

This page has been left intentionally blank.

Gary Kielhofner, DrPH, OTR/L, FAOTA

Copyright © 2006 by F. A. Davis Company

Printed in the United States of America

Last digit indicates print number: 10 9 8 7 6 5 4 3 2 1

Acquisitions Editor: Christa A. Fratantoro

Library of Congress Cataloging-in-Publication Data

This book is dedicated to:

And to all others who have passed on to a

This page has been left intentionally blank.

Marian Arbesman, PhD, OTR/L Wendy J. Coster, PhD, OTR/L, FAOTA

Claire-Jehanne Dubouloz, OT(c), PhD Patricia Heyn, PhD

Edward A. S. Duncan, PhD, BSc(Hons), Frederick J. Kviz, PhD

Kenneth J. Ottenbacher, PhD, OTR/L Pimjai Sudsawad, ScD, OTR

Yolanda Suarez-Balcazar, PhD Craig A. Velozo, PhD, OTR

Elizabeth White, PhD, DipCOT Don E. Workman, PhD

Catherine Emery MS, OTR/L, BCN Rosemary M. Lysaght, PhD, OTR/L

This page has been left intentionally blank.

This page has been left intentionally blank.

This page has been left intentionally blank.

SECTION 1 Chapter 11 Single-Subject

Chapter 5 Professional Responsibility and Chapter 14 Evaluating the Use of Assess-

Appendix B Summary of Common Chapter 27 Searching the

Chapter 28 Generating Research

Chapter 19 Overview of Qualitative Chapter 29 Integrity and Ethics in

Chapter 22 Qualitative Approaches to Chapter 32 Collecting Data, 530

SECTION 6 Chapter 34 Disseminating

Table of Contents xxi

Chapter 39 Building Culturally Competent Chapter 42 Analyzing Evidence for

This page has been left intentionally blank.

The Necessity of Research in a Profession

2 Section 1 The Nature and Scope of Inquiry in a Practice Profession

been a natural ﬁt. Since 1988, in partnership with

Chapter 1 The Necessity of Research in a Profession 3

4 Section 1 The Nature and Scope of Inquiry in a Practice Profession

The Profession’s It seems clear that as administrators and

Chapter 1 The Necessity of Research in a Profession 5

6 Section 1 The Nature and Scope of Inquiry in a Practice Profession

Chapter 1 The Necessity of Research in a Profession 7

8 Section 1 The Nature and Scope of Inquiry in a Practice Profession

Explains phenomena addressed P

Generates evidence about the effectiveness of

Figure 1.5 The relationship of theory, research, and practice.

Chapter 1 The Necessity of Research in a Profession 9

The Aim of Research: Philosophical

Chapter 2 The Aim of Research: Philosophical Foundations of Scientiﬁc Inquiry 11

Observations of the natural world

Figure 2.1 Aristotle’s inductive–deductive method.

12 Section 1 The Nature and Scope of Inquiry in a Practice Profession

A critical turning point in the philosophy of sci- Logical Positivism

Chapter 2 The Aim of Research: Philosophical Foundations of Scientiﬁc Inquiry 13

Subsequent philosophers of science in the dependability of theories relied on their logical

14 Section 1 The Nature and Scope of Inquiry in a Practice Profession

Chapter 2 The Aim of Research: Philosophical Foundations of Scientiﬁc Inquiry 15

16 Section 1 The Nature and Scope of Inquiry in a Practice Profession

Chapter 2 The Aim of Research: Philosophical Foundations of Scientiﬁc Inquiry 17

Table 2.1 A Continuum of Ideas in the Philosophy of Science

18 Section 1 The Nature and Scope of Inquiry in a Practice Profession

crete technical results achievable by those who Regarding Theory

Chapter 2 The Aim of Research: Philosophical Foundations of Scientiﬁc Inquiry 19

The Range of Research

One of the most common distinctions is that