S3
SPECIAL COMMUNICATION
A Phased Developmental Approach to Neurorehabilitation
Research: The Science of Knowledge Building
John Whyte, MD, PhD, Wayne Gordon, PhD, ABPP/Cn, Jack Nash Professor, Leslie J. Gonzalez Rothi, PhD
ABSTRACT. Whyte J, Gordon W, Rothi LJG. A phased
developmental approach to neurorehabilitation research: the
science of knowledge building. Arch Phys Med Rehabil 2009;
90(11 Suppl 1):S3-10.
A systematic and phased approach to the development of
clinical rehabilitation research is needed. Finding ways to adapt
such a phased developmental research model from the more
familiar pharmaceutical model may enhance both rehabilitation
research and the evidence base for decision making in clinical
rehabilitation.
Key Words: Rehabilitation; Research.
© 2009 by the American Congress of Rehabilitation
Medicine
As rehabilitation scientists, we must remember that reha-
bilitation research progresses through a sequence of stud-
ies with unique questions in a series that build upon each
other . . . And we must . . . be advocates for a process of
clinical trials rehabilitation research that endorses that
each phase is valuable and necessary in order to realize the
promise of what lies ahead.1(p88)
HE PURPOSE OF THIS article is to highlight a systematic
T and phased approach to the development of clinical reha-
bilitation evidence, and to consider the ways in which such a
phased developmental research model can be adapted from the
more familiar pharmaceutical model in order to enhance reha-
bilitation research and in turn the evidence base for clinical
decision making.
THE NEED FOR A PHASED APPROACH
EBM presumes that health care clinical decision making is
informed by evidence that is judged by scientific peers and
rated for its ability to reduce threats to the internal and external
validity of the findings. Unfortunately, the treatment efficacy
literature in rehabilitation is relatively small, thereby limiting
the evidence basis for clinical decision making. There may be
many possible explanations for this dearth of evidence, ranging
From the Moss Rehabilitation Research Institute, Albert Einstein Healthcare Net-
work, Philadelphia, PA (Whyte); Department of Rehabilitation Medicine, Mount
Sinai School of Medicine, New York, NY (Gordon); Brain Rehabilitation Research
Center, Veterans Affairs Medical Center, Gainesville, FL (Rothi); and Department of
Neurology, University of Florida, Gainesville, FL (Rothi).
Supported by grants from the National Institute of Disability and Rehabilitation
Research (grant no. H122B040033), United States Department of Education; Veter-
ans Affairs Rehabilitation Research and Development Center of Excellence (grant
nos. 2182 and R24 HD050836) from the National Center for Medical Rehabilitation
Research.
No commercial party having a direct financial interest in the results of the research
supporting this article has or will confer a benefit on the authors or on any organi-
zation with which the authors are associated.
Correspondence to Leslie J. Gonzalez Rothi, PhD, Brain Rehabilitation Research
Center, Malcolm Randall VAMC (151A), 1601 SW Archer Road, Gainesville, FL
32608, e-mail: gonzalj@neurology.ufl.edu. Reprints are not available from the author.
0003-9993/09/9011S-00216$36.00/0
doi:10.1016/j.apmr.2009.07.008
from limited understanding of the complexities of the process
of knowledge development that forms the basis of EBM, to
woefully inadequate funding needed to support rehabilitation
research. The development of effective rehabilitation treat-
ments would follow a strategy similar to that typically used
in the development of medical treatments involving a series
of studies that address many of the issues outlined in appendix
1.2,3 More likely in the case of rehabilitation, treatments might
bypass study of some of these issues (marked with an asterisk
in appendix 1) because they are well established by clinical
tradition even if inadequately supported by empirical research.
That is, in these instances, rehabilitation research may be more
empirically focused (“Is this well established treatment proto-
col actually effective?”), with less concern about the mecha-
nism of action and issues of safety, feasibility, treatment pro-
tocol formalization, and relevant outcome measures, because
questions about these issues may already have been addressed
within the clinical environment. Most of the issues listed in
appendix 1, however, remain crucial for targeted rehabilitation
study and in many cases need to be answered in a logical temporal
order to provide the evidence needed to guide clinical care. For
example, in medical research, issues of safety would be early
targets of study, while for rehabilitation, feasibility, in particular
whether participants could successfully complete the treatment
(which is commonly protracted), would be an early target of study.
Subsequent issues in any form of treatment research would be
efficacy and effectiveness, with each subsequent issue reliant on
resolution of prior issues in earlier targeted studies.
Given the many steps involved in the development and
evolution of effective rehabilitation treatments, it is critical to
consider where in this systematic phased developmental ap-
proach clinical efficacy and effectiveness must be established,
and where RCTs and other similarly rigorous designs are of the
greatest value. In particular, we argue that the demonstration of
practical impact of a rehabilitation treatment on patient out-
comes, using rigorous clinical trial designs, is a late phase of
study in the maturation process of rehabilitation treatment
development, and that a variety of research approaches have
value in promoting the maturation of a treatment domain lead-
ing up to the point of considering an RCT.4-9
RCTs in any biomedical field are generally expensive to
mount, are complex to administer, and usually take an extended
period to complete. RCTs in pharmaceutical development typ-
ically enroll relatively homogeneous patient/client samples,
which result in the need for multicenter trials for all but the
most common conditions. The cost of RCTs involving a reha-
bilitation intervention may exceed the cost of medically ori-
List of Abbreviations
EBM evidence-based medicine
ICF International Classification of Functioning,
Disability and Health
RCT randomized controlled trial
TBI traumatic brain injury
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
S4 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte
ented trials, because in addition to the data collection and
quality control infrastructure required by all RCTs, rehabilita-
tion trials may focus on an intervention delivered over many
hours for weeks to months by skilled clinicians. Thus, efforts to
establish treatment fidelity may require the exceptional appli-
cation of cumbersome methods to establish clinicians’ adher-
ence to the treatment protocol at multiple sites over long
periods. Additionally, there are rehabilitation treatments for
which RCTs are unlikely to be conducted because of ethical
and practical difficulties, and for which quasi-experimental
health services research designs may be the most rigorous
sources of evidence available. However, even trials of this
nature are likely to be extremely expensive and complex.
Although the inclusion and exclusion criteria may be less
restrictive, resulting in samples more similar to real world
patient populations, this greater ease of enrollment is offset by
the need for larger samples to allow for adjustment of impor-
tant confounding case mix variables. In addition, the challenges
involved in measuring the effective dose of nonpharmacologic
treatment and treatment adherence remain major methodolog-
ical challenges to implementing these designs.
Conducting expensive efficacy and effectiveness trials with-
out prior establishment of safety/feasibility/proof-of-concept
risks the outcome of a negative trial. Moreover, because effi-
cacy and effectiveness trials are ultimately empirical in nature
(that is, they deliver a yes/no answer to the efficacy question),
a negative result may provide little direction for subsequent
research. In contrast, with earlier phases of research exploring
safety/feasibility/proof-of-concept questions about theoreti-
cally motivated rehabilitation treatment approaches, even neg-
ative results might help to refine or refute the treatment
theory, thereby advancing the rehabilitation field.10 These
early phases often result in minor modifications of the
rehabilitation treatment approach in response to detailed
feedback received through closely observed delivery to
small numbers of subjects.
IMPACT OF A PREMATURE FOCUS ON CLINICAL
TRIALS REHABILITATION RESEARCH
The EBM movement has encouraged health care scientists,
clinicians, and consumers to examine the quantity and quality
of evidence supporting the range of treatments currently in use
and to develop new treatments in a climate in which the need
(and the demand) for rigorous supporting evidence is high and
resources are quite limited. Within the framework of EBM, the
RCT is viewed as the most powerful source of evidence with
which to assess treatment efficacy/effectiveness, although
RCTs have been criticized with respect to their external valid-
ity (eg, Travers et al11)—that is, their generalizability to
patients/clients and settings more heterogeneous than those
typically involved in RCTs. Additionally, as stated by Con-
cato et al in 2000, “The popular belief that only randomized,
controlled trials produce trustworthy results and that all
observational studies are misleading does a disservice to clinical
investigation . . . .”12(p1892) As applied to neurorehabilitation in
particular, intense focus on RCTs would appear to be “ . . . with-
out appreciation for the maturational process of clinical trials
needed for rehabilitation treatment innovation, evolution, and ul-
timately application to defined populations.”13(p196)
The lack of widespread acceptance of the need for a system-
atic, phased development process for rehabilitation treatment
research and, in particular, the lack of consensus on what this
developmental process should look like may have important
negative consequences that impede the progress of research.
Rehabilitation researchers, themselves influenced and pres-
sured by the cries for high quality evidence, may jump prema-
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
turely to late-stage efficacy trials with all the attendant risks.
Peer reviewers for funding agencies may also be influenced by
the values of EBM and may give poor scores to rehabilitation
research proposals that do not employ what they view as the
most robust designs, even when the goal of the proposed
studies is not to establish clinical efficacy. This may have the
unintended effect of prematurely aborting the process of
knowledge development surrounding a rehabilitation interven-
tion. Similarly, journal editors may be disinclined to publish
rehabilitation studies that describe steps of progress along a
systematic, phased development pathway because submissions
do not meet the criteria for demonstrating clinical efficacy but
instead focus on the earlier questions of safety, feasibility, and
proof-of-concept. Not publishing these studies targeting early
issues limits peer feedback to the author that encourages further
refinement of a developing rehabilitation treatment modality
and also prevents others from building on the same information
to develop more mature stages of rehabilitation research. The
barriers to funding and publication of these early studies may
encourage premature trials or slow the development of more
systematic understanding of the treatment.
Policy makers, vigilant for quality evidence of rehabilitation
treatment efficacy/effectiveness, may consider all of the pre-
liminary research phases to have produced no evidence at all
because the discoveries stagnated or because premature appli-
cations failed, rather than considering the question, “What is
the best evidence we have and where should it lead us next?”
Thus, there may be many barriers that challenge researchers as
they develop and test rehabilitation interventions.
In the discussion that follows, it is important to distinguish
the purpose of EBM from the purpose of a systematic, phased
developmental approach to rehabilitation research. Rehabilita-
tion treatment development research involves a logical se-
quence of studies designed to cultivate a treatment approach
into a formal, optimally performing intervention. These phased
rehabilitation studies address many questions, such as the fol-
lowing: How is this treatment best delivered? How are the
effects of this treatment best measured? How much of this
treatment is needed? Does this treatment appear sufficiently
promising to be moved on to a more ambitious next step of
research?
In contrast, EBM addresses a very concrete and applied
clinical question: What is the strength of evidence to guide
a clinician in selecting a rehabilitation treatment for a par-
ticular patient with a particular problem? Thus, EBM evalu-
ates the quality of the available evidence—that is, treatment
research— on a given rehabilitation intervention such that
choices made in the care of a person are informed by the
literature. Even within the EBM framework, lack of the most
rigorous evidence does not equate to complete ignorance about
what rehabilitation treatment to choose. Rather, the rehabilita-
tion clinician must still choose a treatment, but will be less
certain of the correctness of that choice.
THE PHASED DEVELOPMENTAL MODEL FOR
PHARMACEUTICAL RESEARCH
The premise that is the basis of clinical trials research is that
a medical intervention or treatment should do no harm. This
idea dates back more than 1500 years to the publication of the
Canon of Medicine in 1025 A.D.14 In this treatise, Avicenna
presented principles for testing the effectiveness of new drugs
that formed the basis of modern day clinical trials.15,16 In the
United States, this idea did not begin to gain prominence until
1938, when Congress passed the Federal Food, Drug, and
Cosmetic Act.17 The stimulus for this action was the death of
100 children from a sulfa drug that had not been tested in
 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte
humans.18 Additional impetus for clinical trials research was
the sudden occurrence of deformity in 10,000 babies in 46
countries throughout the world as a result of the inadequate
testing and misuse of thalidomide.18 This resulted in Congress
passing the Kefauver-Harris Amendments to the 1938 Federal
Food, Drug, and Cosmetic Act,17 which required new drugs to
be effective as well as safe. Thus, the law gave the Food and
Drug Administration wider regulatory authority over the test-
ing and marketing of pharmaceuticals being brought to the
market. Effectiveness was to be determined by substantial
evidence, and the only source of substantial evidence was to be
well controlled studies. It would be an understatement to say
that the passage of this law changed all science that was to
follow.
A clinical trial is “a prospective study comparing the effect
and value of interventions against a control in human sub-
jects.”19 Testing the effectiveness of a new drug involves a
sequential series of studies that are designed to address increas-
ingly complex issues toward this end.17 The process begins
with pilot safety and feasibility studies that address questions
critical to designing more advanced studies that examine the
effectiveness of the intervention, and proceeds to more clini-
cally informative stages of research.
In a phase I trial, issues of safety and tolerability are exam-
ined, generally in healthy volunteers, along with the drug’s
pharmacokinetics (what the body does to a drug) and pharma-
codynamics (the effects of a drug on the body). Issues of
dosing are examined as well because it is important to begin to
characterize the dose response of a given treatment.20 It is
important to determine early in the development of a drug how
much is needed to achieve effect, how much can be tolerated at
any given point in time, and how much is safe. A phase I trial
is the foundation for further testing of the efficacy/effectiveness
of the drug. Trials of this variety usually involve samples of
fewer than 100 participants.21
Phase II clinical trials are implemented once an intervention
has been found to be safe. Phase II trials begin to examine
whether the intervention does what it is supposed to in terms of
its basic mechanism of action (how it affects the body) and
preliminary evidence of efficacy defined by The Office of
Technology Assessment as “the probability of benefit to indi-
viduals in a defined population from a medical technology
applied for a given medical problem under ideal conditions of
use.”22 Phase II trials usually involve randomized controlled
designs with relatively limited sample sizes.
A phase III clinical trial is designed to determine the effec-
tiveness of an intervention (which is defined as “the expecta-
tion of benefit when [a treatment] is provided in a typical
fashion by typical practitioners to typical patients in typical
clinical settings and generally involves large, multi-site parallel
group RCTs.”21 (Note that there is no clear boundary between
the ending of efficacy testing and the beginning of effective-
ness studies.) Phase III trials are comparative, and may com-
pare the relative effectiveness and/or the relative toxicity of a
new drug to (1) the natural history of a disease, using a
placebo; or (2) the best available standard treatment.23 Phase
III trials are usually multicenter and involve samples that are
quite large—that is, 300 to 3000.
Phase IV trials are not required for Food and Drug Admin-
istration approval of a new drug, but consist of postmarketing
surveillance. Thus, phase IV is not a controlled research design
but a way of monitoring unexpected safety or effect differences
as the drug is introduced into less controlled settings. By the
most restrictive definitions of effectiveness, it is not until phase
IV that we get insight into the extremes of patient heterogeneity
and the fullest spectrum of the health care delivery system
S5
continuum. Indeed, periodic reports of safety concerns with
already marketed medications raise the question of whether
more formal research (but that allows for real world heteroge-
neity) should continue into phase IV.
While bringing a new drug to the market involves an evo-
lutionary process of sequential experimentation that builds on
evolving knowledge, the information gained from a phase III
trial is the standard that guides clinical practice.
ADAPTING THE MODEL TO THE FIELD OF
REHABILITATION
Rehabilitation research designed to develop or examine the
effects of an intervention differs from pharmaceutical research
to the extent that there is need for greater investment of time
and resources in the early experimental phases, and a less linear
trajectory of development. If RCTs and other efficacy and
effectiveness designs are best suited for late phases in the
development of rehabilitation treatments, it is critical to exam-
ine the earlier phases of development that set the stage for these
ultimate demonstrations of practical utility. Unfortunately, un-
like the phases of research used to develop pharmaceutical
products, few except Robey and Schultz2 have discussed within
the field of rehabilitation science what these phases should be,
and what study designs are most suitable for each phase.
Indeed, although we concur with their framework for concep-
tualizing the developmental phases of rehabilitation treatment
research,1,13 these phases are likely to be less rigidly specifiable
than those used in pharmaceutical research because rehabilita-
tion treatments may target any of the levels of the ICF, whereas
drug studies usually target only body structure and function
levels. Moreover, in biomedical trials, case mix (severity)
factors and the treatment outcome measures to be used may
emerge relatively clearly from knowledge of the disease being
treated. In contrast, rehabilitation treatment research is often
more complex because it may require additional preliminary
studies to refine the treatment dose to be provided, to revise the
treatment to accommodate confounding impairments and func-
tional limitations, and to examine relevant functional outcome
and/or case mix measures that often include psychologic and
social variables.
We endorse the systematic, phased research model for de-
velopment of rehabilitation evidence posed originally by
Robey and Schultz2 and further adapted by Rodriguez and
Rothi,1 depicted in table 1.
This systematic, phased rehabilitation treatment develop-
ment research model depicts a concept emerging from basic
discovery processes that is developed into a rehabilitation
treatment intervention and examined from proof-of-concept
through to effectiveness with respect to meaningful functional
outcomes. We propose that rehabilitation research adhering to
this evolutional process of development requires adequate fi-
nancial investment in all its phases, and journals that support
publishing not only the efficacy/effectiveness studies but also
the early research on mechanism, outcome measurement,
safety, proof-of-concept, feasibility, dosing, and schedule. In-
deed, without financial and publishing support of these early
studies, the evidence base for rehabilitation interventions will
continue to be limited. In a systematic, phased, developmental
rehabilitation research approach, like the pharmaceutical de-
velopment model, each phase asks an important but different
question; each phase requires rigorous but different scientific
methods specifically relevant to the purpose of the phase, and
each subsequent phase relies on successful resolution and com-
pletion of the prior phase’s question.1,13 Rehabilitation clinical
trials may require revision of the treatment method based on
what was learned in prior phases of study. Throughout this
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
S6 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte

Table 1: Key Aspects of Clinical Trials at Different Phases*

Phase 1 Phase II Phase III
Discovery Clinical Trial Clinical Trial Clinical Trial Health Services Research

Purpose Basic discovery Treatment innovation, Treatment efficacy Treatment Delivery method,
evolution, effectiveness societal impact
formalization
Participant Animal or Human individuals Human individuals from defined Human individuals Human individuals
model human populations (ideal conditions) from defined and institutions
and specified subpopulations population
(average of typical
conditions)
Research Within-subject and small group Small group designs, within- Multisite randomized Large group designs,
methodology experimental designs subject with replications clinical trials with within-subject with
across subjects large samples multiple replications
across subjects, and
meta-analyses

*Reprinted with permission in modified form from Rodriguez AD, Rothi LJ. Principles in conducting rehabilitation research. In: Stuss DT,
Gordon W, Robertson I, editors. Cognitive neurorehabilitation, evidence and applications. 2nd ed. New York: Cambridge Univ Pr; 2008.
p 79-90.1

research continuum, there are methodological challenges char-
acteristic of rehabilitation research. For example, unlike early
drug dosing studies that generally involve healthy volunteers,
early rehabilitation studies to refine the treatment method typ-
ically require the participation of carefully selected persons
with the disabilities to be treated. Experience-based treatments
often require the development of treatment manuals that cap-
ture the active ingredients, and adherence measures that verify
the treatment is actually being delivered. Additionally, because
of the protracted nature of the process of rehabilitation, the
simple act of completing a rehabilitation treatment trial often
requires exceptional endurance and compliance on the part of
trial participants, their caregivers (for example, in transporting
them to daily clinic visits), and the investigators. Further,
because many rehabilitation clients are not sick, participating
in a research trial is another time-consuming task that is added
to day-to-day activities and one that often competes or inter-
feres with the participant’s ongoing roles and time commit-
ments. Thus, participation often competes with life.
This maturational process of rehabilitation research, while
based on the model for pharmaceutical development, encom-
passes some unique attributes that make it more suited to the
circumstances typical of rehabilitation research. One valuable
attribute of this model is that it begins to specify the unique
purposes, designs, and methodologies of rehabilitation research
from basic science to health services research.
Laying the Groundwork: Natural History and
Measurement
Natural history and measurement research is often necessary
to lay the groundwork for any treatment research in a particular
clinical area. This research usually takes place before initiating
a phase I trial because it addresses critical issues that will affect
the design of all subsequent trials. For example, it is important
to have measures that are sensitive to the impairment or func-
tional deficit of interest, as well as measures of its real world
impact. In certain areas, these measures must be developed
before any productive treatment research can proceed. For
example, most clinicians agree that persons with TBI have
attention deficits, and many different measures of attention can
be found in the basic cognitive science literature and in use
with various clinical disorders. Yet there has been considerable
controversy about which of these measures best highlights the
clinical deficits seen after TBI and which real world measures
are sensitive to these basic attention impairments.24
Additionally, assuming the treatment target can be mea-
sured, it is important to understand its natural history postinjury
or disease. Deficits that rapidly improve during the early
postinjury period introduce a major confound into the process
of treatment assessment. This aspect of the impairment or
deficit being treated must be anticipated when selecting study
designs. If natural recovery can be modeled statistically with
precision, this may provide adequate preliminary evidence that
the treatment’s outcomes surpass the expected improvements.
In many cases, however, unexplained variance in outcome
requires the use of an untreated control group, even in the
earliest proof-of-concept research. Case mix measures related
to the treatment outcomes of interest, such as severity of the
impairment, time postinjury, coexisting impairments, and psy-
chosocial factors, are also critical to understand. Irrespective of
treatment, it is important to understand those patient or client
attributes that affect the pace and level of improvement on the
outcome measures that will be used.25 This knowledge will
shape the selection of participants in subsequent treatment
studies and will allow either assessment of the balance of
prognostic factors between randomly assigned groups or ad-
justment for minor differences that exist. Collectively, studies
that examine these issues (comorbidity, case mix) can ulti-
mately refine the choice of inclusion and exclusion criteria to
be employed in treatment studies, and will allow the calcula-
tion of necessary sample size with respect to the variance in the
measurement tools to be used.
Discovery Phase
A systematic, phased, developmental approach to rehabili-
tation treatment research (based on pharmaceutical develop-
ment models) begins with basic science, which launches the
evolutionary process of research via conceptual/mechanistic
Discovery. During Discovery, scientists conduct research at the
basic elemental level involving investigation of physiologic
and psychologic mechanisms in animals and/or humans using
within-subject designs or group comparisons. Questions might
include the following: How is the system that serves as the
target of treatment normally supported by the nervous system,
how is it disturbed by pathology, and what might the differ-
ences be? What are the principles of experience-dependent

Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009

 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte
learning (practice schedule intensity, duration, timing) that
promote and optimize lasting functional adaptation and neuro-
plasticity in animal models of chronic central nervous system
injury?26 In many cases, animal studies may be helpful in
revealing basic mechanisms of treatment impact. For example,
various schedules of practice and methods of shaping task
demands have been studied in monkeys with respect to treat-
ment for the hemiparetic arm, and these schedule and shaping
aspects may have relevance to the design of human stroke
treatment.27 Similarly, research with cats was the basis for as-
sisted walking studies in persons with spinal cord injury.28 In
other instances, however, there may be no close animal analogy
to the deficit of interest, requiring one to embark on the earliest
phases of exploration with human samples. In these instances, the
discovery phase may be based on, for example, theories of learn-
ing or our understanding of the functional organization of the
human brain.
Phase 1: Proof-of-Concept, Safety, and Feasibility
In the systematic developmental rehabilitation research
model, the goal of phase 1 is to apply newly discovered
concepts of mechanism in a rehabilitation treatment approach
and assess whether change in the targeted ICF levels is detected
(proof-of-concept). That is, what principles or mechanisms
have been uncovered in the Discovery phase, and might that
information be used to inform therapy such that performance
gains might be achieved? In this phase, treatments that have
been previously applied in animal models may be redesigned
for human use, or treatments that have been previously studied
in human laboratory settings may be adapted to clinical set-
tings. For example, one may have explored the relative impact
of 2 different motor learning approaches in humans with pa-
resis of the arm, using arbitrary planar movements within a
laboratory apparatus (eg, Masiero et al29). Clinical translation
might then involve recasting the learning principles believed to
be the active ingredients of the treatment to a range of move-
ments of clinical importance, and using more readily available
training materials. Outcome measures at this stage typically
concentrate on measures of the immediate or proximal effects
of treatment (ie, those outcomes most closely related to the
proposed treatment mechanism) for proof-of-concept.
This initial clinical translation is still exploratory because, as
mentioned, the treatment may require modification based on
the experiences of research participants. New concerns may
emerge related to the strength of the treatment, issues may arise
in how or whether the treatment can be adapted to persons with
varying baseline skill levels or confounding deficits, and a
range of practical implementation obstacles may present them-
selves. The most important products of this research phase are
(1) a potent treatment protocol that is realistic to apply to the
intended recipients by actual clinicians (and that can be dis-
seminated to future clinicians), (2) selection of sensitive out-
come measures that can detect the proximal effects of treat-
ment, and (3) preliminary evidence about the magnitude of the
treatment effect on which to base further sample size calcula-
tions. As with the Discovery phase, the choice of study design
is flexible and may be affected by factors such as chronicity
and reversibility of treatment benefits. For the purpose of
optimizing the treatment for clinical implementation, applying
it within a repeated single-subject experimental design frame-
work, when possible, allows one to examine each participant’s
course of response in detail and to modify the treatment patient
by patient in response to that feedback. Depending on how
much retooling is needed, however, a separate study may be
required to achieve the estimation of effect size of the revised
treatment. That is, if the treatment delivery is modified exten-
S7
sively up until the last few participants, then one may have
arrived at an optimal treatment, but one will not have a good
estimate of its overall impact on a range of subjects. At the
point where the treatment is no longer being modified exten-
sively, however, it is critical to distill its essential ingredients
into an exportable form such as a treatment manual or decision
algorithm in anticipation of a larger trial.
Because the goal of this phase is to provide a proof-of-
concept that the intervention works in some restricted circum-
stance, studies often include participants who are chosen based
on highly constrained inclusion criteria in order to maximize
the possibility of detecting a change in targeted behaviors that
are related to the intervention. Therefore, studies in this phase
are not designed to represent the general population to which
this treatment may eventually be applied. For example, one
might select only participants whose functional limitations are
substantially attributable to the specific impairment that is the
target of treatment.
A treatment’s hypothesized mechanism of action and the
participant’s chronicity informed by prior natural history re-
search will affect the choice of study design for early proof-
of-concept mechanistic studies. If the mechanism of action
suggests that the treatment can be effective at any point in time
in recovery (acute, postacute, chronic), as is often the case for
learning or practice-based treatments, then it may be wise to
conduct initial mechanistic studies in the chronic phase, a point
when spontaneous recovery may have more limited influence
on functional adaptation. In the absence of rapid spontaneous
recovery, pre-post designs and various forms of single-subject
experimental designs may suffice to allow estimation of the
impact of the treatment. However, if the treatment must be
implemented in the acute period, then even these early mech-
anistic studies may require a parallel group design in order to
assess whether the treated participants appear to show greater
than the expected improvement. This may be the only way to
differentiate the effects of the intervention from spontaneous
recovery. Similarly, if the treatment is anticipated to produce
reversible improvement, then a crossover design may be a
useful early test, but if the treatment effects are expected to be
long-lasting, a pre-post or parallel group design will be re-
quired. Whatever the specific design chosen for such studies,
the outcome measures in this phase should generally be se-
lected based on their proximity to the site of the treatment’s
action, and to their sensitivity to the targets of the treatment.
There is no reason to focus on more distal, ecologically mean-
ingful effects until one has identified the optimal way of
delivering the treatment and the types of subjects who are most
likely to benefit from the treatment trial.
Dose selection and safety are important areas of focus for
phase I pharmaceutical studies, and similarly, this phase in
rehabilitation research should seek to optimize the strength of
the intervention. However, the meaning of strength (or dose) is
much more complex in many rehabilitation studies because,
unlike pharmaceutical treatments, the treatment may be mod-
ified along multiple distinctive dose schedule dimensions
(appendix 2). Whereas a single phase of dose finding may
suffice in preparing for pharmaceutical trials, this concept may
need to be explored iteratively as a nonpharmacologic treat-
ment is tested, and as subgroups of patients are identified who
may respond differently to the treatment.
Safety, too, is a more complex concept in the many experi-
ence-based treatments characteristic of rehabilitation research.
For many pharmaceutical products, the expected toxicities are
related to the mechanism of therapeutic action of the drug. For
experiential rehabilitation treatments, the concept of toxicity or
unexpected consequences often fails to be addressed alto-
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
S8 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte
gether. However, when addressing unexpected consequences in
rehabilitation research, one must consider not only adverse
effects that may be directly related to the intended action of the
treatment (for example: Could intensive focus on strategic
compensation for memory impairment lead to failure to make
use of automatically stored information that would otherwise
be available? Can the early or acute introduction of aerobic
exercise impede recovery?30), but also effects that may be
much more remote (for example: Might an expectation of
treatment benefit that is not fulfilled lead to an increase in
depression or stress?). Moreover, many effortful rehabilitation
treatments such as constraint-induced movement therapy27
may induce fatigue in vulnerable neurologic populations,
which itself can have safety implications.
Phase II: Efficacy
The ultimate goal of this research phase is establishing
treatment efficacy. During efficacy studies in pharmaceutical
development, the group may be large enough to consider
multisite data collection, and the same would be likely in
rehabilitation research. Although parallel group designs are the
typical experimental design in pharmaceutical studies, an ar-
gument can be made for the use of crossover designs in some
situations. Crossover designs are particularly useful when in-
tersubject variability is much greater than intrasubject variabil-
ity, because the intersubject variability is controlled for by
allowing subjects to serve as their own controls. On the other
hand, such studies may overestimate or underestimate the treat-
ment effect when there are substantial carryover effects from
one treatment condition to another (as one would predict in
most skill learning– based interventions such as those charac-
teristic of rehabilitation, in which unlearning does not occur
immediately after the training ceases).
Whatever specific study designs are used in this phase of
rehabilitation research, the efficacy assessments require rigor-
ous control for extraneous factors such as expectancy of benefit
and effects of repeated outcome assessment. However, there is
often no close analogy to the placebo in pharmaceutical re-
search because of the difficulties inherent in concealing many
rehabilitation treatments, as well as the difficulty in designing
fully plausible treatments that lack the postulated active ingre-
dients of the treatment of interest. Determining the appropriate
control or comparison treatment in rehabilitation research is a
complex topic beyond the scope of this article, but interested
readers may consult a useful review of this topic.31
Phase III: Effectiveness
Phase III seeks to establish treatment effectiveness. On the
one hand, large multisite clinical trials in rehabilitation neces-
sarily involve a more heterogeneous set of clinicians and pa-
tients than prior stages, but on the other hand, such trials often
involve extensive training of the clinicians delivering treatment
in the study (indeed, for some studies, clinicians are hired
specifically to deliver the study treatments), tight oversight of
quality control, and relatively restricted inclusion and exclu-
sion criteria that still render the samples atypical. Nevertheless,
the goal of this phase is to establish that the treatment is
applicable to as broad a spectrum of people with the targeted
problem as possible. Meta-analyses may also be used to deter-
mine effectiveness in this sense by examining whether the
treatment effect appears comparable in multiple studies that
differ in their precise characteristics.2
Studies in this phase introduce additional rehabilitation-
specific complexities, distinctly related to the levels of impair-
ment embodied by the ICF. In the case of rehabilitation treat-
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
ments that are aimed at ameliorating impairments, there are 2
distinct ways of viewing effectiveness: (1) Does this treatment,
when applied by a wide range of clinicians to a heterogeneous
sample of patients, result in improvement of the impairment
that is the focus of treatment? and (2) Does this treatment,
when applied by a wide range of clinicians to a heterogeneous
sample of patients, result in improvement in activities and
participation goals of importance? The former is directly anal-
ogous to the transition from phase II to phase III in pharma-
ceutical research. Essentially the same rehabilitation treatment
and outcome measures are applied to a more heterogeneous
sample of patients and treatment sites. However, the second
question requires adding outcome measures that assess new
domains of function, and the relationship between the treated
impairment and those areas of function may be tenuous and
highly variable from patient to patient. Consider a hypothetical
treatment for inattentiveness and distractibility that has been
shown in a phase II trial to improve performance on specific
attention-demanding functional tasks. These findings can be
readily translated into a phase III study by enrolling a larger
number of participants and assessing the treatment’s impact on
this same set of attention-demanding tasks. However, if one
were to examine whether this treatment is effective in returning
individuals to work, unrelated obstacles to employment in
some study participants, such as paralysis or neurobehavioral
disturbances, make it clear that this would be an unrealistic
requirement for proof of effectiveness.
Phase IV: Health Services Research: Effectiveness
Beyond the Clinic
A positive outcome in the previous phase leads to the final stage
in the process of rehabilitation development research, health ser-
vices research. The purpose of health services research is to
determine impact of a treatment by encompassing research regard-
ing the structure, process, and outcome of an effective rehabilita-
tion technique or method to persons living with the consequences
of a medical condition. Impact addresses the effect an individual’s
participation in this rehabilitation program has on society in gen-
eral. Additionally studied are modes of delivery, which involve
research into how a system (health care, social services, educa-
tional, and so on) delivers an intervention. In this phase, because
the treatment is delivered by the full range of clinicians to a
broader range of patients than may have previously been studied,
one may truly get an estimate of the real world impact of the
treatment for the first time.
Because the efficacy of a treatment has already been
established by this phase of study, population studies or
large samples and no external controls are appropriate.
Cost-effectiveness compares the functional outcome of treat-
ment (ie, return to work) with the cost of providing that treatment
(ie, clinic costs). Impact can also be measured by the cost-
benefit, or comparing the cost of a treatment with the financial
contribution of the effect of that treatment on the economy.
Other important aspects of cost-benefit analysis include con-
sumer satisfaction and quality of life. The influence of delivery
and impact on public policy and legislation may also be ad-
dressed. Should other methods of treatment delivery, such as
robotics, artificial intelligence, telehealth, or computers, be
considered? The only way to answer these questions and ac-
complish successful service delivery is to conduct research that
goes beyond establishing a treatment’s efficacy to provide
evidence that supports change in current systems of delivery
and care policy.
 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte
CONCLUSIONS
We argue for a systematic, phased, developmental approach
to rehabilitation research that respects the critical importance of
all the milestones along the developmental research trajectory.
For rehabilitation research to flourish, researchers, funding
agencies, publishers of scientific journals, and health care pol-
icy makers must understand and support the value of all phases
of this maturational process.
A systematic, phased, developmental approach to rehabili-
tation research yields 2 key benefits. First, when one arrives at
the stage of conducting ambitious and expensive treatment
trials, the likelihood that they will deliver clear and useful
results (whether positive or negative) is enhanced, because one
can rely on the fact that the appropriate types of patients/clients
have been enrolled, that the outcome measures chosen were
appropriate, and that the treatment has been fully optimized.
However, it is unlikely that every rehabilitation treatment and
service will be separately subjected to this full longitudinal
developmental pathway, for reasons of ethics and feasibility as
well as cost, and herein lies the other benefit of a systematic,
phased, developmental approach to rehabilitation research. If
we are destined for some time to come to have many rehabil-
itation treatments that do not have class I efficacy evidence
available, the practicing clinician as well as the policymaker
can still reduce uncertainty in rehabilitation treatment selec-
tion by referring to scientific principles that have been
elucidated by earlier phases of study. If we face a rehabil-
itation treatment situation in which an incompletely tested
intervention that makes use of learning and adaptation is
being considered, learning principles may still offer useful
guidance in the context of remaining empirical uncertainty.
APPENDIX 1: ISSUES TO BE
ANSWERED/CLARIFIED IN
REHABILITATION RESEARCH
Formalize a treatment protocol that is repeatable*
Assess safety of the treatment*
Establish a treatment effect
Establish parameters for optimal dose and schedule of treatment
Examine probable mechanisms of treatment effect*
Identify or develop outcome measures that capture both change
and impact*
Identify the treatment’s efficacy in a population and in subgroups
of that population
Determine the treatment’s generalizability
Determine the impact of the treatment on broader, real-world
outcomes
Determine the effectiveness of the treatment as delivered in real-
world health care delivery systems to diverse patients†
Assess cost-effectiveness of the treatment
*Experimental study of these issues in some treatment methods
may not be necessary if well established by clinical tradition.
†
Efficacy (treatment impact demonstrated in a highly controlled re-
search environment) must be distinguished from effectiveness (treat-
ment impact in real world settings), but the boundary between these 2
concepts differs among authors. Some who regard large multicenter
phase III RCTs as effectiveness trials require that the treatment be
broadly effective when delivered in different settings such as different
geographic regions and different professionals. Others note that even
large phase III trials typically have highly restrictive inclusion criteria
and closely supervised treatment staff.3 What is most important is the
concept of a continuum of research that moves from efficacy to effec-
tiveness, with treatment population and settings becoming increas-
ingly diverse and control over provider training and implementation
becoming progressively weaker.
S9
APPENDIX 2: DOSING AND TIMING ATTRIBUTES
OF A RESEARCH TREATMENT
Attribute Description/Example
Dosing
Frequency Daily, twice a week, weekly
Density Minutes a session
Intensity Minutes a week
Timing
Circadian Morning or afternoon treatment
Duration Interval between first and last treatment:
1 week, 1 month
Chronicity Time relative to recovery: early or late;
acute or chronic phase
References
1. Rodriguez AD, Rothi LJ. Principles in conducting rehabilitation
research. In: Stuss DT, Gordon W, Robertson I, editors. Cognitive
neurorehabilitation, evidence and applications. 2nd ed. New York:
Cambridge Univ Pr; 2008. p 79-90.
2. Robey RR, Schultz MC. A model for conducting clinical-outcome
research: an adaptation of the standard protocol for use in apha-
siology. Aphasiology 1998;12:787-810.
3. Whyte J. Rehabilitation effectiveness: the state of the science
and a hope for the future. Am J Phys Med Rehabil 2007;86:
835-7.
4. Barrett A, Rothi L. Theoretical bases for neuropsychological
interventions. In: PJ Eslinger, editor. Neuropsychological in-
tervention: emerging treatment and management models for
neuropsychological impairments. New York: Guilford Pr;
2002. p 16-37.
5. Barrett AM, Rothi LJ. The changing view of neurorehabilitation:
a new era of optimism. J Int Neuropsychol Soc 2006;12:812-5.
6. Barrett AM, Rothi LJ. Treatment innovation in behavioral reha-
bilitation of stroke: removing limits on recovery. J Rehabil Res
Dev 2006;43:vii-x.
7. Barrett AM, Levy CE, Rothi LJ. Treatment innovation in rehabil-
itation of cognitive and motor deficits after stroke and brain injury:
physiological adjuncts. Am J Phys Med Rehabil 2007;86:423-5.
8. Barrett AM, Levy CE, Rothi LJ. Pharmaceuticals for poststroke
and brain injury rehabilitation. Am J Phys Med Rehabil 2007;86:
603-4.
9. Barrett AM, Levy CE, Rothi LJ. Poststroke and brain injury
rehabilitation treatment strategies. Am J Phys Med Rehabil 2007;
86:694-5.
10. Whyte J. A grand unified theory of rehabilitation (we wish!). The
57th John Stanley Coulter Memorial Lecture. Arch Phys Med
Rehabil 2008;89:203-9.
11. Travers J, Marsh S, Brent C, et al. External validity of randomized
controlled trials in COPD. Respir Med 2007;101:1313-20.
12. Concato J, Shah N, Horwitz RI. Randomized, controlled trials,
observational studies, and the hierarchy of research designs.
N Engl J Med 2000;342:1887-92.
13. Rothi LJ. Cognitive rehabilitation: the role of theoretical ration-
ales and respect for the maturational process needed for our
evidence. J Head Trauma Rehabil 2006;21:194-7.
14. Huff TE. The rise of early modern science: Islam, China and
West. 2nd ed. New York: Cambridge Univ Pr; 2003.
15. Tschanz DW. Arab roots of European medicine. Heart Views
2003;4.
16. Brater DC, Daly WJ. Clinical pharmacology in the middle ages:
principles that presage the 21st century. Clin Pharmacol Ther
2000;67:447-50.
17. Food and Drug Administration. Federal Food, Drug, and Cosmetic
Act (FD&C Act). 2009. Available at: http://www.fda.gov/
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
S10 KNOWLEDGE BUILDING IN NEUROREHABILITATION, Whyte
RegulatoryInformation/Legislation/FederalFoodDrugandCosmetic
ActFDCAct. Accessed July 17, 2009.
18. Bren L. The advancement of controlled trials. FDA Consumer
Magazine 2007 (March-April 2007). Available at: http://
www.qualitydigest.com/inside/fda-compliance-article/advancement-
controlled-clinical-trials.html. Accessed October 9, 2009.
19. Friedman LM, Furberg DL. Fundamentals of clinical trials. 2nd
ed. St. Louis: Mosby; 1985.
20. Ruberg SJ. Dose response studies, I: some design considerations.
J Biopharm Stat 1996;6:211-8.
21. Niebuhr BR. Handbook of clinical trial and epidemiological re-
search design. Galveston: University of Texas Medical Branch at
Galveston; 2000.
22. Office of Technology Assessment. Assessing the efficacy and
safety of medical technologies. Washington (DC): US Govern-
ment Printing Office; 1978. Report no. OTA-H-75.
23. Sylvester R, Minder CE. Introduction to the statistical aspects of planning
clinical oncologic phase III studies. Urologe A 1995;34:367-73.
24. Whyte J, Hart T, Bode RK, Malec JF. The Moss Attention Rating
Scale (MARS) for traumatic brain injury: initial psychometric
assessment. Arch Phys Med Rehabil 2003;84:268-76.
Arch Phys Med Rehabil Vol 90, Suppl 1, November 2009
25. Whyte J. Using treatment theory to refine the designs of brain
injury rehabilitation treatment effectiveness studies. J Head
Trauma Rehabil 2006;21:99-106.
26. Kleim JA, Jones TA. Principles of experience-dependent neural
plasticity: implications for rehabilitation after brain damage. J
Speech Lang Hear Res 2008;51:S225-39.
27. Taub E, Uswatte G, Pidikiti R. Constraint-induced movement
therapy: a new family of techniques with broad application to
physical rehabilitation—a clinical review. J Rehabil Res Dev
1999;36:237-51.
28. Behrman AL, Harkema SJ. Locomotor training after human spinal cord
injury: a series of case studies. Phys Ther 2000;80:688-700.
29. Masiero S, Celia A, Rosati G, Armani M. Robotic-assisted reha-
bilitation of the upper limb after acute stroke. Arch Phys Med
Rehabil 2007;88:142-9.
30. Schallert T, Fleming SM, Leasure JL, Tillerson JL, Bland ST. CNS
plasticity and assessment of forelimb sensorimotor outcome in uni-
lateral rat models of stroke, cortical ablation, parkinsonism and spinal
cord injury. Neuropharmacology 2000;39:777-87.
31. Hart T, Fann J, Novack T. The dilemma of the control condition
in experience-based cognitive and behavioural treatment research.
Neuropsychol Rehabil 2008;18:1-21.