688655

research-article2016
BDXXXX10.1177/0198742916688655Behavioral DisordersRoyer et al.

Article

Behavioral Disorders
A Systematic Review of the 2017, Vol. 42(3) 89­–107
© Hammill Institute on Disabilities 2016

Evidence Base for Instructional Reprints and permissions:

sagepub.com/journalsPermissions.nav
DOI: 10.1177/0198742916688655
https://doi.org/10.1177/0198742916688655
Choice in K–12 Settings journals.sagepub.com/home/bhd

David J. Royer, MS1, Kathleen Lynne Lane, PhD, BCBA-D1,

Emily D. Cantwell, MEd1, and Mallory L. Messenger, MEd2

Abstract
We conducted this systematic literature review to explore the current evidence base of instructional choice, a low-intensity,
teacher-delivered strategy to support academic engagement and decrease challenging behaviors. Specifically, we focused
on school-based settings, applying quality indicators (QIs) and evidence-based standards of the Council for Exceptional
Children (CEC). Included articles met five criteria: (a) independent variables included instructional choice; (b) dependent
variables included behavior (e.g., disruptive behavior, problem behavior, aggression), time on task/academic engaged time,
and/or academic performance (e.g., task initiation, completion, accuracy, fluency); (c) interventions occurred with school-age
students in traditional school settings; (d) the study followed an experimental design; and (e) the article was published in English
in a refereed journal. Twenty-five articles (26 studies) from 13 journals met inclusion criteria. Findings indicated providing
students instructional choices increased desired academic behavior while decreasing instances of disruptive behavior. Of the
26 studies, three met all QIs, with nine additional studies addressing 80% or more. Due to small participant numbers, effect
sizes, and other factors in these studies, we classified instructional choice into the CEC evidence-based category of insufficient
evidence. We conclude with a discussion of limitations and directions for future inquiry.

Keywords
choice, evidence-based practice, instructional choice, quality indicators

Since the Individuals With Disabilities Education
Improvement Act (IDEA; 2004), more and more schools have
implemented tiered systems to support academic, behavioral,
and social needs of all students (Prasse et al., 2012; Sailor,
2015). Such tiered systems include academic response to
intervention (RtI; Fuchs & Fuchs, 2006), positive behavioral
interventions and supports (PBIS; Horner & Sugai, 2015),
and comprehensive, integrated, three-tiered (Ci3T; Lane,
Oakes, & Menzies, 2014) models of prevention, all working
to intervene at the first sign of student difficulty using system-
atic data-based decision making. These graduated systems of
support generally include three levels. At Tier 1, supports
such as research- or evidence-based core curricula, effective
classroom management and organization, and effective teach-
ing strategies; school-wide expectations reinforced with
behavior-specific praise; and validated social skills curricula
(e.g., antibullying, antidrug, character building) are provided
to all students. For students needing additional support as evi-
denced, for example, by scoring at moderate or high risk on a
universal behavior screener (e.g., Student Risk Screening
Scale; Drummond, 1994) or scoring below grade level
during academic benchmarking, Tier 2 supports are avail-
able, usually within small groups or as other low-intensity
interventions. Tier 3 supports are intensive, individualized
interventions provided to the few students with the greatest
risk elements, who need more than Tier 1 and Tier 2 provi-
sions (Fairbanks, Sugai, Guardino, & Lathrop, 2007).
Within tiered systems, many educators seek low-intensity
strategies to support student academic and behavioral suc-
cess, including instructional choice, scaffolding independent
work, peer tutoring, increasing opportunities to respond, and
behavior-specific praise (Niesyn, 2009; Simonsen, Fairbanks,
Briesch, Myers, & Sugai, 2008), and student-level strategies
such as self-monitoring and behavior contracts (Lane,
Menzies, Bruhn, & Crnobori, 2011). Instructional choice is a
versatile strategy that can be incorporated into daily planning
at Tier 1 and be used as an intervention for students who need
Tier 2 support (Lane, Menzies, Ennis, & Oakes, 2015).
Instructional choice occurs when “the student is provided
with two or more options, is allowed to independently select
an option, and is provided with the selected option . . . within
1
University of Kansas, Lawrence, USA
2
Miami University, Oxford, OH, USA
Corresponding Author:
David J. Royer, University of Kansas, 1122 W. Campus Rd., 521,
Lawrence, KS, USA.
Email: david.royer@ku.edu
90 Behavioral Disorders 42(3)
naturally occurring classroom events” (Jolivette, Stichter, &
McCormick, 2002, p. 28). Dibley and Lim (1999) separated
types of choice into between-task choices (e.g., which assign-
ment to complete first, “Do you want to make a video or do
a debate to show what you learned?”) and within-task choices
(e.g., which materials to use, where to work, with whom to
work). Offering a choice of reinforcer to work toward after a
task is completed (e.g., “When you complete your assign-
ment, would you like iPad time, to run an errand for me, or to
tell the class a joke?”) is another form of choice paired with
instruction that can provide extra motivation to increase task
completion rate, task accuracy, and appropriate on-task
behavior (Lannie & Martens, 2008; Mechling, Gast, &
Cronin, 2006; Skerbetz & Kostewicz, 2015).
In theory, offering choice helps all students develop self-
esteem, self-determination, and feelings of control and inde-
pendence in life that in turn become positive behavioral
supports toward preventing problem behaviors (Shogren,
Faggella-Luby, Bae, & Wehmeyer, 2004). Historically, stu-
dents with disabilities have had limited opportunities in daily
life to make choices compared with typically developing
same-age peers (Ramsey, Jolivette, Patterson, & Kennedy,
2010; Skerbetz & Kostewicz, 2013), particularly students
with emotional and/or behavior disorders (EBD) who strug-
gle to negotiate the school context successfully (Walker,
Forness, & Lane, 2014). Providing choice-making opportu-
nities to students with disabilities equates to more chances to
increase independence and quality of life (Nota, Ferrari,
Soresi, & Wehmeyer, 2007; Rispoli et al., 2013), sense of
empowerment and self-determination (Kern, Bambara, &
Fogt, 2002; Rispoli et al., 2013; Shogren et al., 2004), and
internal locus of control (Ramsey et al., 2010; Romaniuk &
Miltenberger, 2001).
Providing students with opportunities to make choices
may also lessen reliance on punishment and extinction
(Harding, Wacker, Berg, Barretto, & Rankin, 2002). Although
the connection between opportunities for choice, higher rates
of task engagement, and lower rates of problem behavior is
not clear, evidence is growing to show choice has supple-
mentary effects not elucidated by preference alone (Skerbetz
& Kostewicz, 2013). It is possible providing choice increases
student “buy-in,” meaning students take more ownership
over self-selected assignments and tend to be more engaged,
more likely to finish tasks, and have less opportunities to
behave inappropriately. It is not known whether the act of
choosing is empowering enough to motivate increased task
engagement, whether choosing a preferred task or reinforcer
is what increases engagement, or whether it is a combination
of the two. Katz and Assor (2007) posited choice is motivat-
ing because of the element of control it presents students, but
only when selections are relevant, limited (not overwhelm-
ing in number), and culturally congruent.
Research on choice has included evaluating student pref-
erences (King & Kostewicz, 2014), determining teacher-
provided opportunities for student choice (Flowerday &
Schraw, 2000; Jolivette, Stichter, Sibilsky, Scott, & Ridgley,
2002), increasing choice making to promote self-determina-
tion (Algozzine, Browder, Karvonen, Test, & Wood, 2001),
and assessing choice as an intervention to increase academic
skills (e.g., time on task, task completion, accuracy) and/or
decrease problem behavior (Kern et al., 1998). Studies
involving choice have been conducted in the family home
(Rispoli et al., 2013), residential facility (Ramsey et al.,
2010), isolated public school classroom (Smeltzer, Graff,
Ahearn, & Libby, 2009), and university-affiliated school/
clinic for severe problem behaviors (Kern et al., 2002).
Although choice making has been studied with students of
various ages and disabilities in these more controlled set-
tings, less inquiry has been conducted examining instruc-
tional choice in typical classroom contexts within or outside
of tiered systems of supports (e.g., Skerbetz & Kostewicz,
2013).
Cordova and Lepper (1996) studied the effects of contex-
tualization, personalization, and choice on learning outcomes
for 70 fourth- and fifth-grade students in general education
using variations of a computer game to teach the order of
mathematical operations (e.g., parentheses, exponents, mul-
tiplication and division, addition and subtraction). Although
choices provided within the computer game were not rele-
vant to instruction, adding choice resulted in significantly
greater motivation and learning. Prusak (2004) also studied
choice within a general education context, examining how
choice of activities in physical education class affected vari-
ous measures of motivation for 1,110 seventh- and eighth-
grade girls in 42 classes across five schools in two districts.
Results showed the choice group had greater intrinsic moti-
vation and sense of self-regulation, and less feelings of being
extrinsically controlled (Cohen’s d ≥ 0.928). Prusak sug-
gested having activity choices increased responsibility,
allowed students to find activities more relevant to their
interests, and improved enjoyment of the physical activity.
A few studies have examined the effects of choice making
when working with students with EBD, a group of students
whose internalizing and externalizing behavior patterns
often pose substantial challenges for general and special edu-
cators alike (Walker et al., 2014). Dunlap et al. (1994) worked
with kindergarten and fifth-grade students in self-contained
classrooms, providing a menu of spelling tasks or books to
choose from. Compared with when teachers made choices
for students, student choice increased the percentage of task
engagement and completion while lowering disruptive
behavior. Cole, Davenport, Bambara, and Ager (1997)
assisted three boys in a university-affiliated laboratory school
for students with EBD and autism spectrum disorder (ASD),
providing choice of vocational tasks. Effects of choice did
not differ significantly from when preferred tasks were
assigned, but task engagement was significantly lower when
nonpreferred tasks were assigned. These results added to the
debate of whether having choice, accessing preferred tasks,
or both make more difference. Jolivette, Wehby, Canale, and
Royer et al. 91
Massey (2001) provided three 7-year-old boys the opportu-
nity to choose the order to complete three math worksheets
in a self-contained public school classroom for students with
internalizing EBD, resulting in increased appropriate behav-
ior and task engagement for two boys, whereas the third
maintained high engagement and a low rate of problem
behavior across conditions (see also Ramsey et al., 2010).
These studies demonstrated choice can be an effective strat-
egy to help students with EBD increase desirable academic
habits while decreasing disruptive behavior, likely due to
increased autonomy and control in their environment
(Jolivette et al., 2001; Ramsey et al., 2010).
Knowing instructional choice can improve outcomes for
students with EBD in controlled settings, students with ASD
at home (Rispoli et al., 2013), students with pervasive devel-
opmental disorder—not otherwise specified and fragile X
syndrome in a 1:1 instructional classroom (Smeltzer et al.,
2009), and typically developing students (Cordova & Lepper,
1996; Prusak, 2004), we sought to determine the evidence
base of choice for educators seeking low-intensity strategies
to support academic and behavioral success for all learners
(e.g., typically developing, at risk of EBD or learning dis-
abilities, with identified disabilities) in typical pre-K–12 set-
tings. We focused on traditional school settings believing
instructional choice to be a versatile practice easily applied
to daily instruction (e.g., whole class, small group, and/or
individually) in general or special education with a range of
students, including those with and at risk for EBD.
Purpose
We conducted this systematic literature review to examine the
evidence base of instructional choice for increasing academic
success and/or reducing problem behaviors in traditional pre-
K–12 educational settings. Too often strategies are either (a)
not researched with sufficient breadth or rigor to establish
them as evidence-based practices (EBPs) or (b) existing EBPs
are not adopted and implemented with fidelity (Slavin, 2008).
Teachers may benefit from professional learning on simple,
low-intensity strategies with enough empirical support to
suggest that, if implemented with integrity, they will yield
desired shifts in student performance. To this end, we
employed quality indicators (QIs) and standards set forth by
the Council for Exceptional Children (CEC; 2014) to deter-
mine whether instructional choice could be considered an
EBP for increasing academic success and/or reducing prob-
lem behaviors in traditional pre-K–12 educational settings.
Meeting rigorous methodological checks are essential to
examining individual studies and overall bodies of evidence,
both of which are necessary components to determine
whether a strategy, practice, or program is indeed an EBP.
CEC’s (2014) QI standards are highly stringent (i.e., having
to meet 100% of QIs for a study to be considered method-
ologically sound) given the complexities of actually achiev-
ing these standards when conducting school-based inquiry,
which differs markedly from research conducted in highly
controlled clinical settings (Brown, 1992). In our analyses,
we acknowledge the formidable task of researching strate-
gies in applied settings by employing weighted evaluation
criteria established by Lane, Kalberg, and Shepcaro (2009).
This approach involves considering a study methodologi-
cally sound if it meets 80% of QIs. Our research questions
were twofold.
Research Question 1: To what extent did instructional
choice studies address CEC (2014) QIs?
Research Question 2: What is the nature of the evidence
base supporting instructional choice according to CEC
(2014) guidelines, applying an 80% minimum criterion
for methodologically sound studies (Lane et al., 2009)?
Method
Search Procedures
The first author began by searching electronic databases to
identify instructional choice intervention studies with pre-
kindergarten through Grade 12 students, with the search rep-
licated with 100% reliability by the third author. Included
databases were Academic Search Complete, Education
Abstracts, Educational Resources Information Center
(ERIC), ProQuest Research Library, PsycARTICLES,
PsychINFO, and Psychology and Behavioral Sciences
Collection. Boolean search terms were used to include all
possible combinations and derivatives of choice* AND
(instruction* OR academic OR assign* OR task* OR acti-
vit*) AND (“time on task” OR “task completion” OR “aca-
demic engagement” OR “problem behavio*” OR “disruptive
behavio*” OR “destructive behavio*” OR “behavio* prob-
lem*” OR “inappropriate behavio*” OR “challenging
behavio*” OR “behavio* deficits” OR “aggressive behavio*”
OR “undesirable behavio*” OR “task initiation” OR “on
task behavio*” OR “academic completion” OR “academic
accuracy” OR “academic performance” OR “fluency”).
Article Selection
Next, we read 1,413 article titles and abstracts detected in the
first step to determine which articles should be read in full to
see whether inclusion criteria (described subsequently) were
met. Articles were excluded when the title or abstract did not
include choice making in an education context, if it was not
an intervention study, or when participants were not in a tra-
ditional pre-K–12 setting. A second reader reviewed all title
and abstract coding to assess reliability. This check resulted
in 96.74% agreement (κ = .68, 95% CI = [.59, .77], indicat-
ing substantial agreement; Cohen, 1960; Landis & Koch,
1977). We used Microsoft Excel to list all titles and abstracts,
coding each article as “1” (include) or “0” (exclude) for
meeting criteria, with disagreements resolved by another
92 Behavioral Disorders 42(3)
Figure 1.  Flow diagram illustrating search procedures and article
inclusion.
author. A total of 82 articles from 43 journals were selected
for possible inclusion (see Figure 1, search and article selec-
tion flow diagram).
Then, we read the 82 articles in full to determine which
met inclusion criteria, again having a second reader review
each article. Interrater reliability (IRR) was 82.93% (κ = .66,
95% CI = [.49, .82], indicating substantial agreement).
Disagreements arose when choice was imbedded within a
larger research design (e.g., functional assessment-based
intervention, self-monitoring, timing of choice of reinforcer),
when the research design was vague or there were no clear
method and results sections, and when the intervention set-
ting was not explicitly stated. A third reader resolved dis-
agreements, resulting in 37 articles from 13 journals meeting
inclusion criteria.
Following reading articles in full, the first, third, and
fourth authors conducted a hand search of two major univer-
sity library stacks. Journals publishing two or more included
studies (Behavioral Disorders, Education and Treatment of
Children, Journal of Applied Behavior Analysis, Journal of
Behavioral Education, and Journal of Experimental
Education) were hand searched to verify electronic search
results and identify additional articles missed by the
electronic search. A second author hand searched two of the
five journals to examine reliability. With the trend toward
online journal access, many volumes were not physically
available in either university’s library stacks and were “hand
searched” by clicking through each journal’s online version,
issue by issue. One additional article was identified in this
step (Killu, Clare, & Im, 1999), illustrating the importance of
hand searching as part of a systematic literature review. Hand
search IRR was 99.83% with one disagreement discussed
and resolved by establishing consensus (κ = .82, 95% CI =
[.69, .94], indicating near perfect agreement; Cohen, 1960;
Landis & Koch, 1977).
Next, two authors conducted independent ancestral
searches of the 38 included articles’ references (n = 862) and
identified seven potential studies for inclusion (IRR =
99.07%; κ = .96, 95% CI = [.93, .99], indicating near perfect
agreement). After obtaining and reading the full articles, we
determined none met inclusion criteria, with 100% agree-
ment between authors.
Inclusion criteria were then refined based on articles read
during search steps and our having gained an understanding
of available instructional choice literature. For example, we
initially included any study with choice of reinforcer until
reading articles, where reinforcer choice was offered after
task completion and in others, offered before tasks began.
We ultimately included studies where choice of reinforcer
was offered before a task began, as choice could then serve
as an antecedent intervention able to affect dependent vari-
ables (DVs). Of the 38 articles initially identified, 13 were
removed using updated inclusion criteria due to (a) choice of
reinforcer given after task completion, (b) choice included
option to not work on a task (e.g., take a break), (c) discrete
trial training method, (d) intervention agent was technology
(e.g., computer program offered choices), and (e) interven-
tion did not take place in a traditional school setting or was
unable to be determined. A total of 25 articles from 13 jour-
nals were retained.
Finally, we contacted the corresponding author of each
included study and the editors of the journals from which
studies were obtained, and asked whether they were aware of
any studies accepted for peer review, in press, or published
recently that would not have been found by our searches. No
additional articles were identified.
Inclusion Criteria
To be included in this review, studies had to meet five
conditions:
1.  The independent variable (IV) was some form of
instructional choice provided by an adult, defined as “the stu-
dent is provided with two or more options, is allowed to inde-
pendently select an option, and is provided with the selected
option” (Jolivette, Stichter, & McCormick, 2002, p. 28). We
Royer et al. 93
further defined choice as across-task choice (choose which task
to complete or the order to complete assigned tasks), within-
task choice (options of how to complete an assigned task), or
choice of reinforcement for when task is completed. Options to
not complete a task (e.g., take a break) or providing choice of
reinforcement after task completion were not considered to be
instructional choice. Studies where computers or other technol-
ogy devices provided choices (e.g., Cordova & Lepper, 1996)
were excluded as we were interested in choice as a low-inten-
sity strategy directly implemented by classroom teachers.
2.  The DV included one or a combination of behavior
(e.g., disruptive behavior, problem behavior, aggression),
time on task/academic engaged time, or academic perfor-
mance (e.g., task initiation, completion, accuracy, fluency).
3.  Interventions occurred with school-age students in
traditional pre-K–12 school settings ranging along the spec-
trum of services, including general education, resource class-
rooms or pull-out instruction, self-contained classrooms, and
magnet/specialty buildings. Studies conducted in residential
treatment centers (e.g., Ramsey et al., 2010), home settings
(e.g., Harding, Wacker, Berg, Winborn-Kemmerer, & Lee,
2009), or clinics resembling classroom settings (e.g., two of
the four students in Rispoli et al., 2013) were excluded along
with those using discrete trial training, as they were highly
controlled settings, varying substantially from traditional
school settings.
4.  The study used a single case or group comparison
experimental or quasi-experimental research design.
Nonexperimental (e.g., case study, descriptive study, meta-
analysis) studies on instructional choice were coded as an
article of relevance for reference.
5.  The article was published in English in a peer-
reviewed journal. Theses, dissertations, books, and chapters
were excluded from review as the rigor of their review pro-
cess could not be determined.
Coding Procedures for QIs
Articles meeting inclusion criteria were read and coded by the
first and second authors for the presence or absence of QIs of
methodologically sound interventions. We applied the eight
QIs stipulated in CEC’s (2014) Standards for Evidence-Based
Practices in Special Education related to context and setting,
participants, intervention agent, description of practice,
implementation fidelity, internal validity, outcome measures/
DVs, and data analysis. Coding procedures for each QI are
reviewed below, with the guiding principle being, “Did the
article contain enough information to enable replication?”
First and second authors compared coding, resolved discrep-
ancies (n = 45 of 728), and calculated IRR for articles and
each QI component by dividing the sum of agreements by the
number of applicable components or by 26 (number of stud-
ies; Dunlap et al., 1994, contained two studies), respectively,
times 100 to obtain a percentage. Mean IRR for QI
components was 90.55% (range = 33.33%–100%) and for
studies was 92.26% (range = 81.82%–100%). Overall,
Cohen’s κ was .87 (95% CI = [.83, .91]), indicating near per-
fect agreement (Landis & Koch, 1977). Because all four
authors were contributors to Lane, Royer, et al. (2015), a
researcher at another university familiar with using CEC
(2014) QIs agreed to be a third rater to ensure objectivity and
provided reliability for reconciled coding, which matched
100%.
Training.  The first author was initially trained in QI coding
by the second author during a summer university course in
single-case research design (SCRD). Training consisted of
reading multiple sources for QIs (i.e., CEC, 2014; Gast &
Ledford, 2014; Gersten et al., 2005; Horner et al., 2005;
Kennedy, 2005) and compiling a QI matrix for single-case
methodology (Lane, Common, Royer, & Muller, 2014) later
revised to include QIs for group comparison methodology.
We defined each QI more succinctly (see following sec-
tions) to enhance uniformity between raters and add clarity
for readers, with the goal of coding procedures minimizing
the use of inference, basing decisions on what was explicitly
published in each study. Practice articles were then coded,
results compared, and disagreements discussed and resolved.
The first and second authors, plus a researcher working on a
review for increasing opportunities to respond (Common et
al., 2016), completed additional training in spring by com-
paring independently coded SCRD articles not included in
this study with criterion set at three consecutive articles with
IRR ≥ 85%. For the final three practice SCRD articles, mean
IRR was 92.42% (range = 86.36%–95.45%). We next coded
practice group design articles in the same manner, resulting
in a mean IRR of 97.10% (range = 91.30%–100%).
QI 1.0. Context and setting.  According to CEC (2014), to meet
QI 1.1, a study had to describe critical features of context/
setting relevant to the review (e.g., geographic region, type
of school, type of classroom, physical layout). Although we
did not set a quantitative criterion as to how many features
were required to meet QI 1.1, we required sufficient detail to
allow other researchers to replicate the study. IRR for QI 1.1
was 96.15%.
QI 2.0. Participants.  The participants QI contained two com-
ponents. To meet 2.1, a study had to describe relevant partici-
pant demographics, such as grade level, age, gender, race/
ethnicity, socioeconomic status, and/or language status. We
considered this component met if at least one element was
reported. To meet 2.2, a study had to describe participant dis-
ability or risk status and method of determination (i.e., who
applied what criteria). For this review, it was not sufficient to
only state a diagnosis (e.g., emotionally disturbed, attention
deficit disorder), and when describing the method used to
establish the diagnosis, we required authors to state who
94 Behavioral Disorders 42(3)
(e.g., multidisciplinary team, family physician) applied what
specific criteria (e.g., Diagnostic and Statistical Manual of
Mental Disorders; 4th ed.; DSM-IV; American Psychiatric
Association, 1994; IDEA). For example, having a learning
disability as determined by a multidisciplinary team’s appli-
cation of district, state, or federal criteria was sufficient, but
teacher nomination to a classroom for students with behavior
concerns was not, unless quantitative information from sys-
tematic procedures enabling replication was provided. We
also considered intellectual disability (ID) sufficient when
paired with an IQ score. When all students in a classroom
were participants, we considered QI 2.2 met, as students with
disabilities were not the focus of such studies. IRR for QI 2.1
was 100% and 2.2 was 84.62%.
QI 3.0. Intervention agent. To meet QI 3.1, a study had to
describe the intervention agent’s role (e.g., teacher,
researcher, paraprofessional, parent; electronic devices were
excluded) and background (e.g., race/ethnicity, educational
background, licensure). This component was met for our
review if at least one background variable was mentioned in
addition to the agent’s role. To meet QI 3.2, a study had to (a)
describe specific training the intervention required or quali-
fications (e.g., special education teaching credential) required
to implement the intervention and (b) indicate the interven-
tion agent achieved them at some criterion. If the interven-
tion agent was the study author and designed the method and
measures (e.g., Stenhoff, Davey, & Lignugaris-Kraft, 2008),
he or she would not need to be “trained” how to implement
the intervention, so QI 3.2 was coded as met. Although offer-
ing instructional choice might first appear to some people to
not need training, in practice, we found training was required
to confirm understanding and for systematic implementation
with consistent integrity (e.g., Lane, Royer, et al., 2015). IRR
for QI 3.1 was 76.92% and 84.62% for 3.2.
QI 4.0. Description of practice. QI 4.1 required studies to
describe intervention procedures and intervention agent
actions (or cite a source where such information may be
found) with enough detail to be replicated. Description of
materials used was required to meet QI 4.2, or an appropriate
citation. IRR was 96.15% for QI 4.1 and 100% for QI 4.2.
QI 5.0. Implementation fidelity.  To meet QI 5.1, a study had to
assess and report implementation fidelity related to faithful-
ness using a direct, reliable measure such as an observation
checklist of critical intervention components (e.g., teacher
offered students a choice between X and Y; teacher allowed
students to choose order of tasks). For 5.2, studies had to
assess and report implementation fidelity related to dosage
or exposure also using a direct, reliable measure. In general,
we coded QI 5.2 met if 5.1 was met, dosage was described
(not necessarily reported by checklist), and a graph of DV
data was present. To meet QI 5.3, a study had to assess fidel-
ity “(a) regularly throughout implementation of the
intervention (e.g., beginning, middle, end of the intervention
period), and (b) for each interventionist, each setting, and
each participant or other unit of analysis” (CEC, 2014, p. 4).
To add clarity to QI 5.3, a member of the CEC workgroup
who developed the 2014 standards indicated almost any
mention of assessing implementation fidelity at different
times would suffice (B. Cook, personal communication,
March 14, 2015), in contrast to a study only mentioning
broad assessment without time points. For example, a study
that assessed 30% of sessions for implementation fidelity
would not meet 5.3, but a study that assessed 30% of each
phase for each participant would. We further clarified studies
did not have to report a measure of fidelity for each condi-
tion/phase if an aggregate measure of fidelity was reported
from sufficient time points throughout the study. IRR was
100% for QI 5.1, 96.15% for 5.2, and 88.46% for 5.3.
QI 6.0. Internal validity.  To meet QI 6.1, researchers had to
demonstrate control and systematic manipulation of the IV.
For our review, we determined this QI could only be met if
implementation fidelity (QI 5.1) was established, allowing
the possibility of experimental control (Carroll et al., 2007;
Slaughter, Hill, & Snelgrove-Clarke, 2015). To meet QI
6.2, baseline conditions had to be described, though alter-
nating treatment designs were not required to have a base-
line (though still recommended; Gast & Ledford, 2014). To
meet QI 6.3, authors had to indicate participants had no or
very limited access to intervention components (i.e., choice)
in control/comparison conditions or during baseline and
withdrawal phases. We required this to be obvious by phys-
ical location (groups at different schools) or explicitly
stated in some manner (e.g., “no choice-making opportuni-
ties were provided during baseline”; “choices were removed
during withdrawal phase”), and articles that only indicated
choices were provided during intervention phases did not
meet QI 6.3. QI 6.4 applied only to group designs, and
required researchers to describe how participants were
assigned to groups, and if not done so randomly, to describe
procedures used to ensure functional equivalence of groups
(e.g., matched on various scores and/or demographics, sta-
tistically control for differences).
SCRD studies were required to employ a design that
allowed for the possibility of three demonstrations of experi-
mental effect at three different times (e.g., A-B-A-B with-
drawal, multiple baseline, alternating treatment, changing
criterion) to meet QI 6.5. When study authors stated use of a
particular design (e.g., A-B-A-B), we confirmed graphed
results matched (e.g., was not A-B-A). To meet QI 6.6,
SCRD studies with a baseline phase were required to include
at least three baseline data points unless justification other-
wise was provided (e.g., ethical considerations, counterther-
apeutic trend). QI 6.7 was met when the SCRD controlled for
common threats to internal validity and allowed alternative
rationalizations for results to be sensibly ruled out (e.g., A-B-
A-B withdrawal, multiple baseline, alternating treatment,
Royer et al. 95
changing criterion). In this review, we again required QI 5.1
(implementation fidelity) be established as a prerequisite for
QI 6.7, as an SCRD could only control for threats to internal
validity if it was implemented with integrity. QI 6.8 was met
for group studies when overall attrition across groups was
low, which we defined as <30%, and QI 6.9 was met when
between-group attrition was low, defined as <10%, or statis-
tically controlled for. IRR scores for QIs 6.1 to 6.9 were
92.31%, 100%, 73.08%, 66.67%, 86.96%, 91.30%, 95.65%,
100%, and 100%, respectively.
QI 7.0. Outcome measures/DVs.  To meet QI 7.1, outcomes
of the intervention had to be socially valid, which for this
review could be demonstrated by a social validity ques-
tionnaire or if a strong case was made in the study’s intro-
duction or discussion. For QI 7.2, a study had to define
DVs and describe a direct, reliable measurement system
used with enough detail to allow replication (Cook et al.,
2015). To meet QI 7.3, all outcome measures had to be
reported (clearly graphed data were sufficient). To meet QI
7.4, frequency and timing of outcome measures had to be
appropriate, meaning for SCRD a minimum of three data
points per phase were collected or otherwise justified (e.g.,
ethical considerations, countertherapeutic trend) and for
group designs, assessment measures were conducted rela-
tively near the intervention chronologically. QI 7.5 was
met when reliability of outcome measures was evidenced,
such as by test–retest reliability or interobserver agree-
ment (IOA) ≥80% or κ ≥ 60%. For this review, a mean IOA
was acceptable provided no reported IOA scores fell below
60%. If the low end IOA range was consistently <70%
when using total agreement (Kennedy, 2005), QI 7.5 was
not considered met because total agreement often results in
inflated percentages as opposed to interval-by-interval
agreement (Cooper, Heron, & Heward, 2007). QI 7.6 was
applicable to group designs and met when validity was evi-
denced, such as through content or construct validity. IRR
for QIs 7.1 to 7.6 were 100%, 100%, 96.15%, 96.15%,
80.77%, and 100%, respectively.
QI 8.0. Data analysis. QI 8.1 and 8.3 applied only to group
designs, and were met when appropriate data analyses were
utilized to compare group performance and when an effect size
statistic was reported (or enough data to where an effect size
could be calculated), respectively. QI 8.2 applied to SCRD,
which required a graph of DV data from all phases allowing
visual analysis techniques (i.e., mean, level, trend, overlap,
consistency of data patterns across phases) for making conclu-
sions about experimental control. IRR for QI 8.1 and 8.2 was
100% and 33.33% for QI 8.3 (see “Discussion” section).
Evaluation Procedures for Determining EBP
After coding articles, we applied CEC (2014) criteria to
determine whether instructional choice could be categorized
as an evidence-based practice, potentially evidence-based
practice, mixed evidence, insufficient evidence, or negative
effects. Studies can be considered for EBP classifications
when they are methodologically sound, which for our review
meant meeting 80% of group or SCRD QIs. Such studies are
then classified as having positive, neutral or mixed, or nega-
tive effects based on effect sizes for group designs (e.g.,
Cohen’s d ≥ 0.25 = positive, d ≤ −0.25 = negative, −0.25 <
d < 0.25 = neutral or mixed) or for SCRD the number and
proportion of participants: With a minimum of three total
cases, effects are positive when at least 75% of cases dem-
onstrate a therapeutic change as a result of a functional rela-
tion between the IV and DVs, effects are negative if at least
75% demonstrate countertherapeutic changes, and effects
are neutral or mixed when changes occur in either direction
for less than 75% of cases. An EBP must have positive
effects in (a) two randomized group studies with 60 or more
participants, (b) four nonrandomized group studies with 120
or more participants, (c) five SCRDs with 20 or more par-
ticipants, (d) one randomized group study with 30 or more
participants plus three SCRDs with 10 or more participants,
or (e) two nonrandomized group studies with 60 or more
participants plus three SCRDs with 10 or more participants.
No studies can have negative effects and the ratio of positive
to neutral or mixed effects must be 3:1 or greater. For poten-
tially EBP, mixed evidence, and negative effects classifica-
tion requirements, please see CEC (2014). A practice is
considered for the insufficient evidence category when there
are not enough studies to meet criteria for any of the previ-
ous evidence-based categories.
After coding QIs, we applied an 80% criterion (Lane
et al., 2009) to determine which studies were methodologi-
cally sound. This method gave recognition to articles that
met most QI components by proportionally weighting
scores to contribute to a composite score. For example, QI
7.0 has five components for SCRD, so each component
contributes 20% to the total score for QI 7.0. If one com-
ponent of QI 7.0 was not met, rather than score the QI as
zero, weighted scores for remaining components contrib-
ute to a composite score of .80 (.20 + .20 + .20 + .20 = .80),
meeting the 80% weighted coding criterion. We considered
a total score of 6.40 (.80 × 8 QIs = 6.40) as indicating suf-
ficient rigor to categorize the study as methodologically
sound.
Results
This systematic review of the literature sought to determine
the extent instructional choice interventions in traditional
K–12 school settings (no pre-K studies were identified)
adhered to the eight CEC (2014) QIs, and to what degree
studies meeting QIs support instructional choice as an EBP
following CEC guidelines. Dunlap et al. (1994) conducted
two experiments, which we coded separately, bringing the
total number of studies reviewed to 26.
96 Behavioral Disorders 42(3)
QI 1.0. Context and Setting
Results revealed all studies met QI 1.0, sufficiently describ-
ing context and setting. All studies reported type of program
or classroom, and all but three studies reported type of
school. Additional descriptors were sometimes described,
with eight studies (30.77%) reporting area type (e.g., large
urban city) and 10 studies (38.46%) reporting geographic
location (see Table 1 for descriptive statistics).
QI 2.0. Participants
A large variety of participant demographics were reported
(see Table 1 for a full listing of student characteristics). All
studies met QI 2.1 participant demographics, and 15 studies
(57.69%) additionally met QI 2.2 disability status (see Figure
2). Studies met QI 2.2 when they described the disability (or
risk status) and provided a description of who applied what
criteria to make the determination. For example, Seybert,
Dunlap, and Ferro (1996) met QI 2.2 as IQ scores were
reported along with ID diagnoses; Romaniuk et al. (2002)
followed teacher nominations with functional analyses; and
Cole and Levinson (2002) conducted observations to con-
firm high rates of challenging behaviors reported by teach-
ers’ nominations. In studies where the whole class participated
(Bicard, Ervin, Bicard, & Baylot-Casey, 2012; Clifford,
1975; Myrow, 1979; Patall, Cooper, & Wynn, 2010), QI 2.0
was considered met.
QI 3.0. Intervention Agent
Seven studies (26.92%) met both components of QI 3.0.
Twelve (46.15%) sufficiently described the role and back-
ground of the intervention agent to meet QI 3.1, and eight
(30.77%) reported required training to meet QI 3.2. In most
studies (n = 18; 69.23%), intervention agents were either the
classroom teacher, paraeducator, or a school therapist,
whereas in other studies, experimenters implemented choice
interventions. In Myrow (1979), the author and graduate stu-
dent implemented the study, so training was not required as
they designed the intervention. Hua, Lee, Stansbery, and
McAfee (2014) described how teacher training used a model,
prompt, check method with procedural checklist, meeting QI
3.2. Lane, Royer, et al. (2015) provided a table of intervention
agent demographics to meet QI 3.1 and detailed the remote
training method (e.g., voiced-over presentations, quizzes,
data collection video practice, teleconferences) used to reach
criterion before intervention implementation, meeting QI 3.2.
QI 4.0. Description of Practice
All studies met QI 4.1 intervention procedures and agent
actions whereas 25 (96.15%) also met QI 4.2 materials. For
example, Smeltzer et al. (2009) described the activity prefer-
ence assessment used to identify low-preference tasks (e.g.,
dot-to-dot worksheets, math, stringing beads, pattern blocks,
wiping trays), and how tasks were presented to participants
on a student choice board for the choice condition and on a
teacher’s choice board for the yoked no-choice condition,
with each board ending in free time. Hua et al. (2014) pro-
vided detailed descriptions of materials and procedures for
each 5-min session at the end of math instruction, including
size of worksheets and paper slips used and how multiplica-
tion problems were generated.
QI 5.0. Implementation Fidelity
Overall, 11 studies (42.31%) met all components of QI 5.0.
Twelve studies (46.15%) met QI 5.1 implementation fidelity
adherence, 23 studies (88.46%) met QI 5.2 implementation
fidelity dosage, and 11 studies (42.31%) met QI 5.3 assess-
ing and reporting implementation fidelity regularly through-
out the study. Jolivette et al. (2001) videotaped all sessions
for two observers to later assess for teacher implementation
of procedures. Rispoli et al. (2013) described a procedural
task analysis for at least 30% of sessions across all condi-
tions and described frequency and length of each session
(i.e., intervention dosage). Lane, Royer, et al. (2015) met QI
5.1 with component checklists for each phase and condition,
including one for baseline and withdrawal phases to ensure
(a) baseline components remained in effect during choice
conditions and (b) addition of choice was the only change in
intervention phases.
QI 6.0. Internal Validity
Eleven studies (42.31%) met all applicable components of
QI 6.0. Thirteen studies (50.00%) met QI 6.1 having system-
atically manipulated the IV, all met QI 6.2 having described
control/comparison or baseline conditions, and 17 (65.38%)
met QI 6.3 avoiding baseline or control/comparison contami-
nation. All group design studies described assignment to
groups and met QI 6.4, whereas two (66.67%) met 6.8 and
6.9 for acceptable attrition across and between groups. For
SCRD, 21 (91.30%) met QI 6.5 having used an experimental
design that allowed for demonstration of experimental effects
at three different time points, all met QI 6.6 having included
three data points in all baseline phases, and 11 (47.83%) met
QI 6.7 with an experimental design appropriate for control-
ling for threats to internal validity.
Smeltzer et al. (2009) and Hua et al. (2014) used alternat-
ing treatment designs that do not require baseline, but each
no-choice condition was described and reported at least three
data points so QI 6.2 and 6.6 were met. For QI 6.3, avoiding
contamination of control/comparison or baseline, Cole and
Levinson (2002) used treatment integrity data to show no-
choice conditions only had directive prompts and no choice
questions; Patall et al. (2010) kept students unaware of the
treatment condition classmates were in, and directed them to
not discuss aspects of the study.
 Table 1.  Descriptive Results of Instructional Choice Intervention Studies.
QI Clifford (1975)
1.0 18 fifth-grade and 16 sixth-grade
classes from 15 school systems
in Iowa
2.0 811 students (410 boys, 401
girls), plus partial data for 46
more students; classes chosen
based on teacher volunteers
3.0 Classroom teachers; training =
instruction manual
4.0 Four formats of vocabulary study
lists with 20 words; next day
= multiple-choice quiz on half
the words
5.0 Not described
6.0 Group pre–post experimental
design with random
assignment of classes to choice
or no-choice conditions
7.0 DVs = multiple-choice vocabulary
quizzes, task-liking measure,
perceived-learning measure,
retention measure; SV not
reported
8.0 Choice decreased performance
and did not affect attitude;
only with vocabulary format
3 did students in the choice
condition score higher than
students in the no-choice
condition
97
Myrow (1979)
Nine English classes at Urbana HS,
Illinois
~200 11th- and 12th-grade students
in English; 133 completed all tasks;
teachers volunteered full classes
of students based on department
head invitation
Author and graduate student
research assistant
Six reading packet options at
11th-grade reading difficulty level;
on day 2 choice and no-choice
situations reversed
Not described
Group pre–post quasi-experimental
crossover design; high attrition
rate >30%
DVs = 25-item multiple-choice tests,
personal causation scale, affective
measure, continuing motivation
measure, study time; IOA and SV
not reported
Three of four choice conditions
scored higher than no-choice
conditions on test; significant
main effect of choice on affect,
motivation, and time spent
studying, but not on personal
causation
Dunlap et al. (1994)
Public ES, self-contained SpEd
classes for students with ED
Study 1 = Wendall & Sven, fifth
grade; Study 2 = Ahmad,
K; students referred by
homeroom teacher
Study 1 unclear; Study 2 =
behavioral consultant; no
background variables or
training described
Two 15-min independent
seatwork sessions, English or
spelling; menu of six to 10
academic activities
Not described
Wendall and Ahmad = A-B-
A-B; Sven = A-B-A; during
no-choice phases teacher
selected assignments or book
DVs = task engagement and
DB; IOA for 38%–57% of
sessions; SV not reported
Greater % of task completion
and intervals with task
engagement during choice
phases and lower % of DB
intervals
Seybert, Dunlap, and Ferro (1996)
HS in suburbs of large metropolitan
area in southeastern United States;
large vocational classroom or
courtyard on school campus
Three students with ID: Scott = 14
yo, Bob = 15 yo, Maria = 21 yo also
deaf and with a physical disability
Exp; no background variables or
training described
Two tasks 7 min each for each
session; representative materials for
four to six vocational tasks were
placed on a table, students were
asked to select one each session
Tracked exp interactions
(redirections) only
Multiple-baseline design across
participants with reversal: A-B-A
design (Bob) or A-B-A-B design
(Scott and Maria)
DVs = % of intervals with PB and task
engagement; IOA for 30%–34% of
sessions; SV not reported
Introduction of choice reduced mean
rate of problem behaviors; task
engagement increased during choice
conditions; interest and happiness
affect measures were higher or the
same in choice conditions
Cole, Davenport, Bambara,
and Ager (1997)
University-affiliated laboratory
school for students with
EBD and autism: 75
students with serious ED
and 25 with severe ID
and PB
Three boys: Abe = 12 yo,
Ben = 13 yo, Sam = 11
yo; informal observations
confirmed teacher reports
of off-task and DB
Exp; no background variables
or training described
15-min sessions with a 5-min
break between; choices
of tasks included stapling,
sealing envelopes, bagging
utensils, collating paper, and
stuffing folders with paper
Not described
Alternating treatment design;
students experienced three
conditions each day during
vocational period
DVs = task engagement, DB,
work productivity, accuracy;
32% of experimental
sessions assessed for IOA;
SV not reported
Task engagement comparable
for assigned preferred and
choice tasks; productivity
had variable results
Powell and Nelson (1997)
GE classroom during afternoon
ELA with teacher and one to
two observers and 23 students
Evan = 7 yo second-grade boy
with ADHD
Teacher; no background variables
or training described
Three assignment choices each
session included spelling lists,
silent reading, grammar and
punctuation exercises, and
others
Not described
A-B-A-B design; no academic
choices given during baseline
and withdrawal phases
DV = undesirable behavior
recorded by 10-s momentary
time sampling; IOA for 70% of
sessions; SV not reported
Undesirable behaviors decreased
during choice conditions
Vaughn and Horner (1997)
Two public ES SpEd classrooms
Four students, 7–12 yo, ID,
Down syndrome and ID,
ASD and moderate to severe
ID, Angelman syndrome and
moderate to severe ID
Teachers; no background variables
or training described
10-min sessions, either choice
between two higher preference
tasks or choice between two
lower preference tasks
Videotaped and coded on
computers
A-B-A-B design, with choice
conditions compared with those
where the teacher selected the
same tasks
DV = PB/DB; IOA for 39%–48% of
sessions; SV not reported
Mean rates of PB per minute
during choice between low-
preference tasks were lower,
with high-preference tasks,
rates of PB were variable but
relatively lower regardless of
choice or no choice
(continued)
98
Table 1.  (continued)
QI Killu, Clare, and Im (1999)
1.0 Public MS with 250 students;
self-contained SpEd classrooms
during independent spelling
work
2.0 Three boys, 12–13 yo, LD,
ID; teacher reports of
distractibility and off-task
behavior verified through
independent observation for
study inclusion
3.0 Female teacher presented
choices to students; teacher
and paraeducator reviewed
work for accuracy and
completion; training not
described
4.0 30-min sessions 4 days per week;
five spelling task options
5.0 Not described
6.0 A-B-C-D-E-F design: choice
of preferred tasks, choice
of nonpreferred, no choice
preferred, no choice
nonpreferred, no choice
preferred yoked, no choice
nonpreferred yoked
7.0 DV = task engagement; IOA for
29%–31% of sessions; SV not
reported
8.0 Higher task engagement
with high-preference tasks
regardless of choice or no
choice; minimal effects of
choice with nonpreferred tasks
Jolivette, Wehby, Canale, and
Massey (2001)
Public ES; self-contained SpEd
classroom for students with
EBD who displayed internalizing
behaviors
Three boys with ED, math level 1–2
yo below grade level, Nicky =
7.10 yo, first grade; John = 7.6 yo,
second grade; Bruce = 7.10 yo,
second grade
Female SpEd teacher; training =
restated, modeled, and practiced
delivery
First 15 min of independent math
period; chose order of three math
worksheets
All sessions videotaped and assessed
for teacher implementation of
procedures (100% for all)
Multiple-baseline design with B-A-B
withdrawal; during no-choice
sessions the teacher randomly
assigned order of worksheet
completion
DVs = task engagement, off-task,
disruption, attempted task
problems and problems correct;
IOA for 25%–29% of sessions; SV
= TARF-R
Two of three students showed
increased appropriate behaviors
and task engagement with
moderate effects with different
results for each variable
Cole and Levinson (2002)
University-affiliated laboratory
school for students with EBD
Two boys, 7 and 8 yo, one with
PDD, Tourette’s syndrome,
and ID, one with Down
syndrome, PDD, ADD,
and ID
Three female paras; 30-min
training prior to each
condition
Paras embedded choice
opportunities in daily
instructional routine task
analyses
Type of verbal prompt; no-
choice conditions = 100%
directive prompts; choice
condition sessions needed
≥80% choice questions
A-B-A-B design; during all
no-choice conditions only
directive prompts were given
DVs = % of task analysis
steps with PB, number of
task analysis steps prior to
PB, and (for Wally) % of
independently initiated steps;
SV not reported
Lower % of task analysis steps
with PB and larger number
of steps completed prior
to PB during choice; Wally
increased independently
initiated steps
Kern, Bambara, and Fogt (2002)
University-affiliated private MS for
students with severe behavioral
challenges, science classroom
Six boys, 13–14 yo, labels of severe
ED and other diagnoses
Teacher; first or second author
provided training the day before
each phase
21- to 35-min sessions during science;
group and individual choice of
activity
Type of activity, choice provided, and
high-interest activity; teacher plan
log compared with observer data
A-B-A-B design; during baseline
lesson activities teacher directed
all students to complete same
assignments same format
DVs = engagement and destructive
behavior; IOA for 29% of sessions;
SV = TARF-R and class evaluation
sheet
Mean class engagement increased 30%
first intervention; mean destructive
behavior decreased from 12%
to 0%
Romaniuk et al. (2002)
Public elementary school,
classrooms serving children
with behavior problems
Three boys and four girls, 5–
10 yo, variety of diagnoses,
range of classroom settings
Therapist; no background
variables or training
described
Choice of eight to 10 teacher-
nominated academic tasks
Not described
A-B-A-B design; functional
analysis condition that
produced highest rate of
PBs served as the first no-
choice condition
DV = % of session time with
PB (Riley’s DV = frequency
of PB because PBs were
brief); IOA for students
across 25% of sessions; SV
not reported
Mean % of PB time reduced
46.6% for four students
Carson and Eckert (2003)
Northeastern urban ES (K–6), 515
students, 53.7% White, 41.5%
Black; ~50% FRL
Three students: Tamara and
Catrina = 9 yo, female, Black;
Devin = 10 yo, male, White;
referred by teacher, verified by
three math probes
Four graduate students in school
psychology; trained in study
procedures and demonstrated
100% accuracy
One or two 15-min sessions three
times per week with 3-min
break; in phase 2, students
chose math condition
Scripted procedural protocols;
33% of sessions assessed by
audiotape = 100% integrity
Phase 1 = multielement assessment;
phase 2 = alternating treatment
design; baseline condition was
one math CBM for 2 min with no
other intervention or feedback
DV = digits computed correctly
per minute; interscorer
agreement for 100% of sessions;
SV not reported
All students chose contingent
reinforcement; for two of three
students, choice conditions
were higher than baseline but
not exp chosen condition
Daly, Garbacz, Olson, Persampieri,
and Ni (2006)
Public MS; small room next to
SpEd classroom
Two seventh-grade, 13 yo students
with EBD reading at fourth-
grade level referred by teacher;
Tina = female, Black; Jacob male,
White; assessed for inclusion
Graduate students in a school
psychology training program
with prior training in behavioral
measurement and CBM
procedures
3 days per week, 10 min of
instruction then test; chose type
and time of reading instruction
and reward
Checklist protocol for each
condition including scripted
verbal instructions; 100% of
sessions assessed by audiotape
Multiple-probe design across
tasks (reading passages); during
baseline and maintenance
students read aloud while exp
marked errors
DVs = correct read words and
errors per 30 s; IOA for 100%
of sessions by audio recording;
SV not reported
Jacob showed change in
trend; Tina change in level;
cannot separate choice from
opportunities to respond and
motivational variables
(continued)
 Table 1.  (continued)
Mechling, Gast, and Cronin
QI (2006)
1.0 Public middle school, self-
contained classroom for
students with ASD
2.0 Two boys: Jackson = 13.2 yo,
diagnosed with autism and
mild ID; Donald = 14.4 yo,
diagnosed with autism and
moderate ID
3.0 Intervention agent unclear
4.0 Two 30-min independent work
sessions per day; three work
tasks completed per session;
choice of reinforcer
5.0 Videotaped 33% of sessions
across conditions
6.0 A-B-A-B design; identical tasks
each condition; A = only
tangible reinforcers; B = video
presentation of preferred
items and choice of videos
7.0 DVs = task duration, task errors;
IOA for 33% of sessions across
all conditions; SV not reported
8.0 Mean task duration decreased
with choice in each
intervention phase
99
Lannie and Martens (2008)
Large mid-Atlantic urban district;
ES with 615 students (79% Black,
19% Latino, 0.8% White; 86.9%
FRL); fifth-grade GE during math
instruction
Two boys and three girls, fifth grade,
Black, ages 10.7 to 12.4, below
16th or 25th percentile on WJ-III
calculation subtest, on task <60%
First author and two UG
students; trained prior to study,
demonstrated proficiency with all
procedures
5- to 7-min sessions for 10 weeks;
audio cues for students to
self-monitor during math probe
completion; received chosen
reinforcer after five successful
intervals
Scripted protocols; 35% sessions
observed; 100%, 99%, 99%, 98%
TI for each student
Multiple-baseline design; during
baseline students completed
math probes at frustration level
for 5 min
DVs = digits correct per minute,
accuracy, on-task behavior; IOA
for 35% of sessions; SV = CIRP
Digits correct/minute: Willie =
+4.56, Ellen = +0, Debra and
Charles increased during self-
monitoring; accuracy results =
little evidence
Stenhoff, Davey, and Lignugaris-
Kraft (2008)
High school resource biology
classroom with 15 students
classified with LD or
behavior disorders
15-year-old male, ninth grade,
with LD
SpEd teacher (researcher)
40-min sessions; class
instruction followed by
assignment to whole class;
student given 15 s to make
choice between class
assignment or alternate
assignment
Not described
A-B-A-B design; A = no-choice
condition, B = choose
between two assignments
DVs = % of assignment
completed and accuracy
(IOA for 31%–62% of
sessions); SV = course grade
before and after
% task completion increased
87% with 75% accuracy;
reintroduction task
completion = 99% with 81%
accuracy
Smeltzer, Graff, Ahearn, and Libby
(2009)
Public elementary school in a separate
classroom used for 1:1 instruction
Three boys: Will = 8 yo, second
grade, with PDD-NOS; Dan = 6 yo,
K, with ASD; Frank = 6 yo, different
K, with Fragile X
Exp; no background variables or
training described
Activity preference assessment =
three low-preference tasks on
choice board then teacher’s board
(no-choice condition, yoked), each
ending in free time
Not described
Yoked condition alternating treatment
then concurrent operants (picked
who chose order of tasks),
followed by a three-condition
alternating treatment design
DVs = PB rate or % (IOA for
30%–33% of sessions); on-task (IOA
for 33% of sessions); duration to
complete tasks (IOA not reported);
SV not reported
On-task behavior higher with choice
for Dan, mixed for Will, Frank
= no change; rate of PB lower;
task order choice reduced task
completion time
Patall, Cooper, and Wynn
(2010)
14 classrooms in two
southeastern urban HSs;
55.5% White, 28% Black,
7% Asian, 3% Latino, 1.5%
Native American, 5% Other
207 students, 54% female,
Grades 9–12, 14
classrooms, six subjects
Preservice teachers
completing teaching
internship; received training
from first author
Homework assignments
over 4 weeks; choice
between two homework
assignments; no-choice
students yoked to choice
students and completed the
same assignment
End of unit assessment of
student-perceived choice
regarding homework
Group pre–post experimental
crossover design, randomly
assigned students to choice
or no-choice conditions by
drawing names out of a hat
DVs = intrinsic motivation
inventory, homework
assignments, unit tests;
IRR not provided; SV not
reported
Choice = predictor of interest
and enjoyment, perceived
competence, test scores,
and homework completion,
but not effort, value, or
pressure during homework
Bicard, Ervin, Bicard, and Baylot-
Casey (2012)
Southeastern parochial ES; fifth-
grade class with 20 students
average per day
21 fifth-grade students, 10 boys
and 11 girls, 10–11 yo; two
students received SpEd services
Teacher (second author) working
on master’s degree
Individual seating and group seating
choice conditions; students
chose seats at the beginning of
the week for that week
Secondary observer checked 30%
of the sessions
A-B-A-B reversal design for
individual seating and again for
group seating
DV = DB measured by tally per
student in 1-min intervals and
rate calculated for whole class;
IOA for 28% of sessions; SV not
reported
DB occurred less when teacher
selected seats; during group
seating with choice, DB
occurred >twice as often; during
individual seating choice, DB
occurred >three times as often
Rispoli et al. (2013)
Alex = resource classroom during
ELA; Dylan = self-contained
classroom; Kelly = university-
based autism clinic; Eddie =
kitchen table at home
Three boys and one girl, 5–11 yo,
with ASD
Dylan’s teacher; trained graduate
student researcher for other
participants
5-min sessions two to four times
per day, 2–4 days per week;
across-task choice = choose
from two to four activities;
within-task choice = choice
of work location, materials,
response
TI data collected 30% of sessions
using procedural task analysis
A-B-A-B alternating treatment
(across-task and within-task
choice by random assignment)
design; A = no choice
DVs: off-task, screams, aggression;
elopement, delayed echolalia,
destruction, verbal protesting;
IOA 20% sessions; SV not
reported
All participants’ rates of PB
decreased in choice phases,
across-activity choice associated
with lowest rate of PB
(continued)
100
Table 1.  (continued)
QI Skerbetz and Kostewicz (2013) Hua, Lee, Stansbery, and McAfee (2014) Lane, Royer, et al. (2015) Skerbetz and Kostewicz (2015)

1.0 Public charter school in large urban city,
general education classroom, vocabulary
component of ELA; 20 other students
present and data collector
2.0 Two boys and three girls, eighth-grade,
Black, 13 yo, six to 26 ODRs, identified as
ED or at risk of ED
3.0 Classroom teacher; no background variables
or training described
4.0 10 s to choose one of four vocabulary
assignments from a packet to complete
in 7 min
5.0 Observations videotaped and scored by
exp. using checklist; procedures were
followed 100%
6.0 A-B-A-B design, A = no choice, B = choice
condition; during baseline students given
one vocabulary assignment without choice
to complete in 7 min
7.0 IOA for 45%–55% of sessions; DVs =
task engagement/nonengagement, task
accuracy, task completion time; high SV
survey results
8.0 Average task engagement increased
15.4%; accuracy rose an average 10.2%;
completion time decreased an average
of 1.4 min
Public ES in large urban Pennsylvania district, resource room
reading and math with one SpEd teacher, one part-time
paraeducator, and 10 to 12 students; small group
Two boys and one girl, fourth-grade, White, 10 yo, identified
with SLD
Classroom SpEd teacher; researcher trained using model, prompt,
check method with procedural checklist; teacher scored 100%
before beginning the study
5-min sessions end of math; choice of math paper size; order of
conditions counterbalanced across days
Not described
Alternating treatment design for math worksheet, paper slip, and
student choice
DVs = digits correct per minute, task format selected; 33%
of assignments scored by an independent observer; SV not
reported
No obvious differences between conditions for digits correct/min
or task accuracy
Large, suburban, Midwest public ES, ~600 K–5 students; GE first-grade,
25 students, writing block, with SpEd teacher supporting students
with disabilities
One boy, one girl, first-grade, moderate risk on SRSS; Neal = 6 yo,
White, with ASD; Tina = 7 yo, Asian
General (female, 34 yo, White), special (female, 23 yo, White), and
instructional support (female, 37 yo, White) teachers trained via
distance technology
Across-task (choose order of tasks) or within-task (choose supplies,
where to work, with whom to work) choice during independent
writing
TI data collected 100% of sessions with checklists for each phase; 25%
IOA each phase
A-B-A-B alternating treatment design (across-task and within-task
choice); A = no choice, B = alternating across- and within-task choice
by random assignment
IOA calculated for at least 25% of sessions within each phase; DVs =
academic engaged time and DB; SV = IRP-15 and CIRP
Improvements in academic engaged time and reduced DB for Tina;
Neal = no functional relation (increased engagement for intervention
phase 2)
charter school in Large urban city, northeast United States, ~300
K–6 students; GE fifth-grade, 23 students, groups of four to five
during independent math review activity
Two boys and two girls, fifth-grade, 10–11 yo, one to eight ODRs,
diagnosis of either ADHD or ED
Exp and SpEd teacher; no background variables or training described
5-min warm up math computation probes with either choice
of reinforcement, reinforcement with no choice, or no
reinforcement
100% of videotaped sessions scored using an exp. checklist;
procedures followed 100%
Multielement design; baseline = no choice no reinforcer; phase 1
= independent level math, alternating no reinforcer, no choice
reinforcer, choice of reinforcer; phase 2 = instructional level
IOA for 54% of sessions for seconds of engagement and frequency
of engagement, 100% of session for accuracy of digits correct and
incorrect; SV = teacher and student surveys with high SV results
Phase 2: better separation between conditions with one student
displaying an improving trend for seconds and frequency of
engagement

Note. QI = quality indicator; HS = high school; ES = elementary school; SpEd = special education; ED = emotional disturbance; EBD = emotional and behavioral disorder; ID = intellectual disability; PB = problem behavior; GE =
general education; ELA = English language arts; K = kindergarten; yo = years old; DB = disruptive behavior; ADHD = attention deficit hyperactivity disorder; ASD = autism spectrum disorder; exp = experimenter; DV = dependent
variable; SV = social validity; IOA = interobserver agreement; MS = middle school; FRL = free or reduced-price lunch; LD = learning disability; PDD = pervasive developmental disorder; ADD = attention deficit disorder; CBM =
curriculum-based measure; TARF-R = Teacher Acceptability Rating Form–Revised (Reimers, Wacker, Cooper, & DeRaad, 1992); WJ-III = Woodcock–Johnson Tests of Achievement Third Edition (Woodcock, McGrew, & Mather, 2001);
NOS = not otherwise specified; UG = undergraduate; TI = treatment integrity; CIRP = Children’s Intervention Rating Profile (Witt & Elliott, 1985); IRR = interrater reliability; ODR = office discipline referral; SLD = specific learning
disability; SRSS = Student Risk Screening Scale (Drummond, 1994); IRP-15 = Intervention Rating Profile–15 (Witt & Elliott, 1985).
Royer et al. 101

8.0 8.0 8.0

28 7.6 8.0
8.3 7.5 8.0 7.5
27 8.0 8.0
8.2
26
8.1 7.0
25 6.8
7.6 6.7 7.0
24 6.5 6.5
7.5 6.3 7.0 7.0 7.0 6.5 6.3 6.3
23 6.3 6.3
7.4
22 6.0 6.0
7.3
21 6.0

Number of CEC (2014) quality indicators met

5.6 5.7
7.2
CEC (2014) quality indicator component

20 6.0 5.3 6.0 6.0

7.1 5.2
19 5.0
5.1
6.9 5.0
18
6.8 5.0
17
6.7 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0
16
6.6
15
6.5
14 4.0
6.4
13 4.0 4.0 4.0 4.0 4.0 4.0
6.3
12
6.2
11
6.1 3.0
10
5.3 3.0 3.0
9
5.2
8
5.1
7 2.0
4.2 QIs met: Absolute coding
6
4.1 QIs met: Weighted coding
5
3.2
4
3.1 1.0
3
2.2
2
2.1
1
1.1
0 0.0
Daly et al. (2006)
Kern et al. (2002)

Smeltzer et al. (2009)

Patall et al. (2010)
Stenhoff et al. (2008)
Jolivette et al. (2001)

Bicard et al. (2012)

Skerbetz & Kostewicz (2015)

Carson & Eckert (2003)

Skerbetz & Kostewicz (2013)

Lane, Royer et al. (2015)

Killu et al. (1999)
Powell & Nelson (1997)

Rispoli et al. (2013)

Myrow (1979)

Vaughn & Horner (1997)

Cole & Levinson (2002)

Dunlap et al. #1 (1994)
Dunlap et al. #2 (1994)
Seybert et al. (1996)

Mechling et al. (2006)

Lannie & Martens (2008)

Hua et al. (2014)

Romaniuk et al. (2002)
Clifford (1975)

Cole et al. (1997)

Study

Figure 2.  Instructional choice studies (abscissa) and CEC (2014) individual QI components met (primary ordinate; shaded cells =
component met, white cells = component not met).
Note. Secondary ordinate (right y axis) displays number of QIs met by absolute coding (triangles; 8.0 QIs required) and weighted coding (circles), which
required 6.4 QIs (80%) to be considered a methodologically sound study. Components not applicable to a study’s design are marked NA. Dunlap et al.
(1994) contained two studies. CEC = Council for Exceptional Children; QI = quality indicator.
102 Behavioral Disorders 42(3)
QI 7.0. Outcome Measures/DVs
Seventeen studies (65.38%) met all applicable components
of QI 7.0. All met QI 7.1 and 7.3, reporting or justifying
social validity and reporting outcomes for all measures,
respectively. Twenty-five studies (96.15%) met QI 7.2 for
defining and describing DVs. Twenty-three studies (88.46%)
met QI 7.4 frequency and timing of outcome measures, 20
(76.92%) met QI 7.5 reporting adequate internal/interob-
server reliability, and all group studies met QI 7.6 evidence
of validity. For QI 7.2, Lane, Royer, et al. (2015) included
examples and nonexamples of DV behaviors that helped
ensure primary and secondary data collectors were in agree-
ment. For QI 7.5, Skerbetz and Kostewicz (2015) described
how a trained observer collected IOA data for more than half
of all sessions, the method used to calculate IOA, and the
IOA percentages for each DV. The three group studies
(Clifford, 1975; Myrow, 1979; Patall et al., 2010) met QI 7.6
by describing validity coefficients for measures and demon-
strating content validity.
QI 8.0. Data Analysis
All group studies met QI 8.1 and 8.3 with appropriate data
analysis techniques and enough information to allow effect
sizes to be computed. For example, Myrow (1979) reported
means but not standard deviations, but included F statistics,
degrees of freedom, and mean squares from analysis of vari-
ance tests, allowing for effect size calculation. All SCRD
studies met QI 8.2 by including single-case graphs that dis-
tinctly indicated DV outcome data across phases and allowed
determination of intervention effects through traditional
visual analysis of mean, level, trend, overlap, and pattern.
Determination of Instructional Choice as an EBP
The second research question regarded categorizing the evi-
dence base of instructional choice using CEC’s (2014) stan-
dards. We applied weighted coding as suggested by Lane
et al. (2009), recognizing studies that met 80% of the eight
QIs as methodologically sound. Composite scores ranged
from 4.97 to 8.00, with 12 studies (46.15%) ≥6.40 and, there-
fore, eligible to consider when classifying the evidence base
of instructional choice in traditional school settings. See
Figure 2 for QI components present and weighted coding
results for each study.
When we applied CEC (2014) guidelines to classify the
12 studies meeting 80% of the QIs as having positive, neutral
or mixed, or negative effects, two of the three group studies
(one randomized, one nonrandomized) were included but
only two SCRDs met the minimum requirement of having
three or more participants (see Table 2). Rispoli et al. (2013)
did have four participants, but only two had sessions in tradi-
tional school settings, the focus of this review (as recom-
mended by CEC, 2014, our review considered only
participants who were part of the target population). Carson
and Eckert (2003) had three participants, but a functional
relation was only established for two, as one student’s alter-
nating treatment design did not have the requisite four repeti-
tions (Kratochwill et al., 2013). Two SCRDs were not eligible
for other reasons. Kern et al. (2002) had six participants, but
we were unable to determine the percentage who demon-
strated therapeutic change because data were reported in
aggregate. Lannie and Martens (2008) had four participants,
but we were unable to determine whether the functional rela-
tion between IVs and DVs was due to changes in self-moni-
toring type or due to choice of reinforcer.
We used an Excel-based effect size calculator (DeFife,
2009) to calculate effect size for the two group studies
(Myrow, 1979; Patall et al., 2010). We converted F statistics
to d = 0.113 for Myrow (1979), and used means and standard
deviations across units to calculate d = 0.154 for homework
completion rate and d = 0.190 for unit test scores for Patall
et al. (2010). According to Cohen (1988), effect sizes >0.20
could be considered small, and for this review, we set the
minimum threshold at 0.25 for a group study to be classified
as having positive effects. We, therefore, considered these
two group studies to be in the neutral or mixed effects cate-
gory (CEC, 2014).
Jolivette et al. (2001) had three participants, two of whom
had a functional relation between the choice IVs and DVs and
demonstrated a positive therapeutic trend. The third partici-
pant, however, maintained high levels of engagement and low
rates of off-task behavior across conditions, so we classified
this study as neutral or mixed effects. The final study (Bicard
et al., 2012) presented a unique case. Students were able to
choose where to sit on Mondays for the week, but on alternat-
ing weeks when the teacher assigned seats, students’ disruptive
behavior was less, classifying this study’s choice intervention
as having negative effects (see “Limitations” section).
With no eligible studies demonstrating positive effects, we
were not able to categorize instructional choice into the
mixed evidence category, and therefore concluded it fits the
insufficient evidence category based on CEC’s (2014) evi-
dence-based category specifications.
Discussion
Choice can increase motivation when a student has clear pref-
erences, appreciates the options provided, enjoys the act of
choosing, and benefits from the outcome of choosing (Patall,
2012). Despite the insufficient evidence category classifica-
tion, it bears noting the majority of study authors reported
positive participant outcomes (see Table 1, QI 8.0 for each
study’s data analysis). Looking specifically at the 12 method-
ologically sound studies, the two group studies did report posi-
tive, albeit small, effects. Jolivette et al. (2001) had positive
outcomes for all three participants but one maintained those
outcomes across phases. Five studies did not have enough
participants to contribute to the evidence base for instruc-
tional choice but reported positive outcomes for how choice
can reduce problem behavior for students with comorbid
Royer et al. 103
Table 2.  Evaluation of the Evidence Base for Instructional Choice.
 
No. of studies
Study characteristic
Total no. of participants
No. of studies eligible for effect classificationa
No. of participants in eligible studies
 
Positive
Mixed or neutral
Negative
Note. See CEC (2014) for more details. SCRD = single-case research design; CEC = Council for Exceptional Children.
a
SCRD studies require three or more participants for whom it can be determined if therapeutic or nontherapeutic trends were established.
disabilities or ASD (Cole & Levinson, 2002; Rispoli et al.,
2013); produce increased oral reading fluency rates for stu-
dents with EBD (Daly, Garbacz, Olson, Persampieri, & Ni,
2006); result in faster task completion for puzzles, math, color
by number, and sorting for students with ID (Mechling et al.,
2006); and increase academic engaged time while reducing
disruptive behavior for a first-grade general education student
(but only increased engagement in intervention phase 2 for a
student with ASD; Lane, Royer, et al., 2015). Two other meth-
odologically sound studies reported positive outcomes but
could not be considered when classifying the evidence base:
Choice increased mean engagement 30% and reduced destruc-
tive behavior from 12% to 0% for six boys with severe EBD
but data were reported in aggregate (Kern et al., 2002), and
choice increased math digits computed correctly per minute
for three fourth-grade general education students but one stu-
dent’s alternating treatment design was missing a pattern rep-
lication (Carson & Eckert, 2003).
This systematic literature review suggests the need for
better designed studies to withstand the challenges associ-
ated with conducting experimental designs in the reality of
traditional K–12 settings. More recent studies appear to be
designed to meet more QIs: One of nine (11.11%) studies
published from 1975 to 2000 met 80% of QIs, whereas 11 of
17 (64.71%) studies published after 2000 met 80% of QIs.
Although the methodological quality of the research base
appears to be improving, refinement is still needed. Some
authors have expressed concern that applying stringent QIs
may exclude meaningful research when classifying the evi-
dence base for practices and strategies (e.g., Moeller, Dattilo,
& Rusch, 2015). Indeed, the CEC (2014) requirement for
studies to meet 100% of QIs would exclude nine studies we
considered to be of high quality from being considered meth-
odologically sound.
It is a talented research team that can address this delicate
balance between experimental control and the realities of
Methodologically sound coding
Absolute Weighted
3 (11.54%) 12 (46.15%)
SCRD Group SCRD Group
25 0 48 340
 1 0  2  2
21 0 24 340
Study effect classification
0 0 0 0
0 0 1 2
1 0 1 0
day-to-day life in schools (Brown, 1992). Even with QIs in
special education available for over a decade for researchers
to use as they design studies (e.g., Gersten et al., 2005;
Horner et al., 2005), the challenge of implementing interven-
tions with fidelity remains in applied settings where clinical
control is not always possible. For example, SCRDs need
three participants with a functional relation between IVs and
DVs for EBP classification consideration. However, one of
three students in a study may become ill or move and not be
available to finish a design replication sequence, eliminating
the study from EBP classification consideration. Or, the
same study may have four participants but one is absent from
enough sessions to prevent QI 7.4 (frequency and timing)
from being met, preventing the study from being method-
ologically sound (absolute coding) even though it is other-
wise of high quality. For such reasons, weighted coding with
an 80% criterion (Lane et al., 2009) may be a useful option.
Essentially, this application allows for “partial credit” to be
given to studies, subsequently enabling a broader array of
still rigorous studies to be examined when determining
whether a strategy, practice, or program is indeed an EBP.
Although the majority of studies did not meet 80% of the
QIs, they did reflect areas of methodological strengths. For
example, though we only required studies to include one or
more background variables to meet QI 1.1 (context and set-
ting), 2.1 (participant demographics), and 3.1 (intervention
agent), some studies reported many (see Table 1), such as
tables for school characteristics, student participants, and
intervention agent background (e.g., Lane, Royer, et al.,
2015). All studies described procedures and actions suffi-
ciently for replication, with 25 (96.15%) also describing
materials used. All studies also met the requirement for hav-
ing socially important outcomes, though we hoped more
studies would have explicitly used some version of a social
validity measure. For example, Kern et al. (2002) had stu-
dents complete a class evaluation sheet daily with questions
104 Behavioral Disorders 42(3)
asking how much they enjoyed class that day, and the teacher
completed the Teacher Acceptability Rating Form–Revised
(Reimers et al., 1992). Lannie and Martens (2008) and Lane,
Royer, et al. (2015) had students complete the Children’s
Intervention Rating Profile (Witt & Elliott, 1985) and
Skerbetz and Kostewicz (2015) used teacher and student sur-
veys with high results.
It should also be noted we clarified the role of treatment
integrity in our coding. Specifically, we required QI 5.1
(implementation fidelity) be met before QI 6.1 or 6.7 (sys-
tematic manipulation of the IV and internal validity) could
be considered, and for QI 6.3, we required researchers to
explicitly state control/comparison or baseline/withdrawal
conditions were not contaminated by choice components.
Although logical and reasonable, these conditions were not
required by CEC (2014) and made these QIs more difficult to
meet. We recommend future review teams also consider
implementation fidelity as a prerequisite for these QIs, as it
must first be established the intervention was implemented
as designed.
As we consider the educational implications of this
review, we caution the reader to consider the possibility that
practices, including instructional choice, can be effective, yet
not currently classified as evidence based. Specifically, given
core QIs were introduced only 10 years ago (e.g., Gersten
et al., 2005; Horner et al., 2005), it is likely not enough high-
quality studies have been conducted to clearly demonstrate
the true effectiveness of current practices, strategies, and
programs (Cook et al., 2015). It is imperative for the scien-
tific and practitioner communities to commit to additional
inquiry in pre-K–12 classrooms to build the body of evidence
needed to determine the extent to which instructional choice
is effective in supporting various students, demonstrating
various target behaviors, in various contexts, and with the
support of various treatment agents. Studies presented in this
review provide an important starting point for examining the
utility of instructional choice as a low-intensity support. We
recommend interested readers consider the following limita-
tions and future directions before building additional evi-
dence to meet the call of determining how well instructional
choice works.
Limitations
Results of this review should be interpreted with caution
given additional studies used instructional choice but were
not included. Our primary limitation was the focus on tradi-
tional classroom settings, which excluded clinical, home,
residential schools, or other highly controlled environments
(e.g., discrete trial training; Newman, Needelman, Reinecke,
& Robek, 2002). A second limitation was our inclusion crite-
ria allowed studies where choice was not provided for each
task. For example, Bicard et al. (2012) gave students choice
of seating on Mondays, which were kept for the week. This
week-long choice was different from choice of seating for
the duration of a specific task (e.g., floor, desk, beanbag;
with a partner or alone), followed by the student returning to
teacher-selected seating. In this study, as most experienced
teachers have seen, allowing students to choose with whom
to sit was more likely to set conditions for disruptive behav-
ior. In other words, the benefits of choice did not help when
offering a choice known to negatively impact the outcome
measure. Future author teams may consider refining criteria
to include studies where choice is provided for each task or
at minimum, each day.
Third, we did not calculate effect sizes for SCRD studies.
Although not required by CEC (2014) to determine the evi-
dence base of a strategy, calculating standardized effect sizes
would aid future comparisons of findings across studies con-
taining different outcome measures (Shadish, Hedges,
Horner, & Odom, 2015). A fourth limitation was our low
IRR on QIs 3.1, 6.3, 6.4, and 8.3, which were below 80%.
We had six and seven disagreements for QIs 3.1 and 6.3,
intervention agent role/background and IV isolated from
control/baseline, for reasons including inferring the interven-
tion agent role, missing a background variable, and missing
treatment integrity data indicating the IV was isolated from
no-choice conditions. The low IRR for QIs 6.4 and 8.3 were
the results of a low base rate of group design studies included
(n = 3). For example, with one or two disagreements, IRR for
these group study QIs dropped to 2/3 or 1/3.
Finally, the broad focus of our review might be consid-
ered a limitation. That is, examining both behavioral and
academic outcomes for all learners in traditional K–12 class-
room settings may prevent generalizing review results to a
specific outcome or population of students. For example, if
we had found instructional choice to be an EBP within these
broad parameters, we could not assume it was evidence
based for specific populations.
Conclusion
The CEC (2014) expert panel set forth rigorous criteria for
methodologically sound studies, leaving flexibility within
each QI for some interpretation, recognizing the substantial
variety of studies to which QIs would be applied. We applied
and interpreted a few QIs differently when unique elements
of intervention studies were encountered, even after trainings
and check-ins to ensure accuracy of our coding (e.g., inde-
pendently coding three to four studies and discussing results
before coding the next three to four studies). Overall, this
strategy was successful. Out of 728 possible QI coding com-
ponents (26 studies × 28 QI components = 728), we had 45
discrepancies (93.82% agreement; κ = .87, 95% CI = [.83,
.91], near perfect agreement) discussed and resolved. We
encourage other systematic literature review research teams
to also design systematic training with the CEC (2014) QIs
before beginning each new review.
Given challenges researchers face implementing method-
ologically sound interventions in school settings, it was
encouraging to see practitioners implement choice interven-
tions with high procedural and treatment fidelity and
Royer et al. 105
minimal university support. For example, Lane, Royer, and
colleagues (2015) provided training to teachers in another
state via web technology to introduce the instructional choice
strategy and train primary and secondary data collectors,
allowing teachers to support each other, with university
researchers advising on when to implement phase changes
based on collected data. We believe this is a promising direc-
tion for future studies to take to help establish instructional
choice and other low-intensity strategies as EBPs teachers
can confidently implement in multiple settings with minimal
university support.
Although we anticipated being able to draw more defi-
nite conclusions about instructional choice as an EBP, we
were pleased to see high-quality studies demonstrating suc-
cess in different ways, in multiple settings and contexts,
and with various populations. The CEC (2014) guidelines
for methodologically rigorous studies are available to all,
and we encourage the research community to use them
when partnering with practitioners to design interventions.
We especially encourage focused inquiry on traditional
school settings with a variety of students in different con-
tent areas so in the end we will know how to best use
instructional choice.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect
to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, author-
ship, and/or publication of this article.
References
References marked with an asterisk indicate studies included in the
literature review.
Algozzine, B., Browder, D., Karvonen, M., Test, D. W., & Wood,
W. M. (2001). Effects of interventions to promote self-determi-
nation for individuals with disabilities. Review of Educational
Research, 71, 219–277. doi:10.3102/00346543071002219
American Psychiatric Association. (2000). Diagnostic and statisti-
cal manual of mental disorders (4th ed., text rev.). Washington,
DC: Author.
*Bicard, D. F., Ervin, A., Bicard, S. C., & Baylot-Casey, L.
(2012). Differential effects of seating arrangements on dis-
ruptive behavior of fifth grade students during independent
seatwork. Journal of Applied Behavior Analysis, 45, 407–411.
doi:10.1901/jaba.2012.45-407
Brown, A. L. (1992). Design experiments: Theoretical and meth-
odological challenges in creating complex interventions in
classroom settings. The Journal of the Learning Sciences, 2,
141–178.
Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain,
S. (2007). A conceptual framework for implementation fidel-
ity. Implementation Science, 2, Article 40. doi:10.1186/1748-
5908-2-40
*Carson, P. M., & Eckert, T. L. (2003). An experimental analy-
sis of mathematics instructional components: Examining
the effects of student-selected versus empirically-selected
interventions. Journal of Behavioral Education, 12, 35–54.
doi:10.1023/A:1022370305486
*Clifford, M. M. (1975). Affective and cognitive effects of option
in an educational setting. Journal of Experimental Education,
43(3), 1–5. doi:10.1080/00220973.1975.10806327
Cohen, J. (1960). A coefficient of agreement for nominal scales.
Educational and Psychological Measurement, 20, 37–46.
doi:10.1177/001316446002000104
Cohen, J. (1988). Statistical power analysis for the behavioral sci-
ences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
*Cole, C. L., Davenport, T. A., Bambara, L. M., & Ager, C. L.
(1997). Effects of choice and task preference on the work
performance of students with behavior problems. Behavioral
Disorders, 22, 65–74.
*Cole, C. L., & Levinson, T. R. (2002). Effects of within-activity
choices on the challenging behavior of children with severe
developmental disabilities. Journal of Positive Behavior
Interventions, 4, 29–52. doi:10.1177/109830070200400106
Common, E. A., Lane, K. L., Cantwell, E. D., Brunsting, N., Oakes,
W. P., & Bross, L. A. (2016). Teacher-delivered strategies
to increase students’ opportunities to respond: A systematic
methodological review. Manuscript in preparation.
Cook, B. G., Buysse, V., Klingner, J., Landrum, T. J., McWilliam,
R. A., Tankersley, M., & Test, D. W. (2015). CEC’s stan-
dards for classifying the evidence base of practices in special
education. Remedial and Special Education, 36, 220–234.
doi:10.1177/0741932514557271
Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied
behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson
Education.
Cordova, D. I., & Lepper, M. R. (1996). Intrinsic motivation and
the process of learning: Beneficial effects of contextualiza-
tion, personalization, and choice. Journal of Educational
Psychology, 88, 715–730. doi:10.1037/0022-0663.88.4.715
Council for Exceptional Children. (2014). Council for Exceptional
Children standards for evidence-based practices in special
education. Arlington, VA: Author.
*Daly, E. J., III, Garbacz, S. A., Olson, S. C., Persampieri,
M., & Ni, H. (2006). Improving oral reading fluency by
influencing students’ choice of instructional procedures:
An experimental analysis with two students with behav-
ioral disorders. Behavioral Interventions, 21, 13–30.
doi:10.1002/bin.208
DeFife, J. (2009). Effect size calculator and conversions. Emory
University. Retrieved from http://web.cs.dal.ca/~anwar/ds/
Excel4.xlsx
Dibley, S., & Lim, L. (1999). Providing choice making opportunities
within and between daily school routines. Journal of Behavioral
Education, 9, 117–132. doi:10.1023/A:1022888917128
Drummond, T. (1994). The Student Risk Screening Scale (SRSS).
Grants Pass, OR: Josephine County Mental Health Program.
*Dunlap, G., dePerczel, M., Clarke, S., Wilson, D., Wright, S.,
White, R., & Gomez, A. (1994). Choice making to promote
adaptive behavior for students with emotional and behavioral
challenges. Journal of Applied Behavior Analysis, 27, 505–
518. doi:10.1901/jaba.1994.27-505
Fairbanks, S., Sugai, G., Guardino, D., & Lathrop, M. (2007).
Response to intervention: Examining classroom behavior
support in second grade. Exceptional Children, 73, 288–310.
doi:10.1177/001440290707300302
106 Behavioral Disorders 42(3)
Flowerday, T., & Schraw, G. (2000). Teacher beliefs about
instructional choice: A phenomenological study. Journal of
Educational Psychology, 92, 634–645. doi:10.1037/0022-
0663.92.4.634
Fuchs, D., & Fuchs, L. S. (2006). Introduction to response to inter-
vention: What, why, and how valid is it? Reading Research
Quarterly, 41, 93–99. doi:10.1598/rrq.41.1.4
Gast, D. L., & Ledford, J. R. (Eds.). (2014). Single case research
methodology: Applications in special education and behav-
ioral sciences (2nd ed.). New York, NY: Routledge.
Gersten, R., Fuchs, L. S., Compton, D., Coyne, M., Greenwood,
C., & Innocenti, M. S. (2005). Quality indicators for
group experimental and quasi-experimental research in
special education. Exceptional Children, 71, 149–164.
doi:10.1177/001440290507100202
Harding, J. W., Wacker, D. P., Berg, W. K., Barretto, A., & Rankin,
B. (2002). Assessment and treatment of severe behavior
problems using choice-making procedures. Education and
Treatment of Children, 25, 26–46.
Harding, J. W., Wacker, D. P., Berg, W. K., Winborn-Kemmerer,
L., & Lee, J. F. (2009). Evaluation of choice allocation
between positive and negative reinforcement during func-
tional communication training with young children. Journal
of Developmental and Physical Disabilities, 21, 443–456.
doi:10.1007/s10882-009-9155-7
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom,
S., & Wolery, M. (2005). The use of single-subject
research to identify evidence-based practice in spe-
cial education. Exceptional Children, 71, 165–179.
doi:10.1177/001440290507100203
Horner, R. H., & Sugai, G. (2015). School-wide PBIS: An exam-
ple of applied behavior analysis implemented at a scale of
social importance. Behavior Analysis in Practice, 8, 80–85.
doi:10.1007/s40617-015-0045-4
*Hua, Y., Lee, D., Stansbery, S., & McAfee, J. (2014). The effects
of assignment format and choice on task completion. Journal
of Education and Learning, 3, 101–110. doi:10.5539/jel.
v3n1p101
Individuals With Disabilities Education Improvement Act of 2004,
20 U.S.C. 1400 et seq. (December 3, 2004).
Jolivette, K., Stichter, J. P., & McCormick, K. M. (2002). Making
choices—Improving behavior—Engaging in learning.
TEACHING Exceptional Children, 34(3), 24–29.
Jolivette, K., Stichter, J. P., Sibilsky, S., Scott, T. M., & Ridgley,
R. (2002). Naturally occurring opportunities for preschool chil-
dren with or without disabilities to make choices. Education
and Treatment of Children, 25, 396–414.
*Jolivette, K., Wehby, J. H., Canale, J., & Massey, N. G. (2001).
Effect of choice-making opportunities on the behavior of stu-
dents with emotional and behavioral disorders. Behavioral
Disorders, 26, 131–145.
Katz, I., & Assor, A. (2007). When choice motivates and when
it does not. Educational Psychology Review, 19, 429–442.
doi:10.1007/s10648-006-9027-y
Kennedy, C. H. (2005). Single-case designs for educational
research. Boston, MA: Pearson Education.
Kern, L., Bambara, L., & Fogt, J. (2002). Class-wide curricular mod-
ification to improve the behavior of students with emotional or
behavioral disorders. Behavioral Disorders, 27, 317–326.
*Kern, L., Vorndran, C., Hilt, A., Ringdahl, J., Adelman, B., &
Dunlap, G. (1998). Choice as an intervention to improve
behavior: A review of the literature. Journal of Behavioral
Education, 8, 151–169. doi:10.1023/A:1022831507077
*Killu, K., Clare, C. M., & Im, A. (1999). Choice vs. preference:
The effects of choice and no choice of preferred and non pre-
ferred spelling tasks on the academic behavior of students with
disabilities. Journal of Behavioral Education, 9, 239–253.
King, S. A., & Kostewicz, D. E. (2014). Choice-based stimulus
preference assessment for children with or at-risk for emotional
disturbance in educational settings. Education and Treatment
of Children, 37, 531–558. doi:10.1353/etc.2014.0026
Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin,
J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R.
(2013). Single-case intervention research design stan-
dards. Remedial and Special Education, 34, 26–38.
doi:10.1177/0741932512452794
Landis, J. R., & Koch, G. G. (1977). The measurement of observer
agreement for categorical data. Biometrics, 33, 159–174.
Lane, K. L., Common, E. A., Royer, D. J., & Muller, K. (2014).
Council for Exceptional Children 2014 group comparison
and single-case research design standards quality indicator
matrix. Unpublished tool.
Lane, K. L., Kalberg, J. R., & Shepcaro, J. C. (2009). An examina-
tion of the evidence base for function-based interventions for
students with emotional and/or behavioral disorders attending
middle and high schools. Exceptional Children, 75, 321–340.
doi:10.1177/001440290907500304
Lane, K. L., Menzies, H. M., Bruhn, A. L., & Crnobori, M. E.
(2011). Managing challenging behaviors in schools: Research-
based strategies that work. New York, NY: Guilford Press.
Lane, K. L., Menzies, H. M., Ennis, R. P., & Oakes, W. P. (2015).
Supporting behavior for school success: A step-by-step guide
to key strategies. New York, NY: Guildford Press.
Lane, K. L., Oakes, W. P., & Menzies, H. M. (2014). Comprehensive,
integrated, three-tiered models of prevention: Why does my
school—and district—need an integrated approach to meet
students’ academic, behavioral, and social needs? Preventing
School Failure: Alternative Education for Children and Youth,
58, 121–128. doi:10.1080/1045988X.2014.893977
*Lane, K. L., Royer, D. J., Messenger, M. L., Common, E. A.,
Ennis, R. P., & Cantwell, E. D. (2015). Empowering teachers
with low-intensity strategies to support academic engagement:
Implementation and effects of instructional choice for elemen-
tary students in inclusive settings. Education and Treatment of
Children, 38, 473–504. doi:10.1353/etc.2015.0013
*Lannie, A. L., & Martens, B. K. (2008). Targeting performance
dimensions in sequence according to the instructional hierar-
chy: Effects on children’ math work within a self-monitoring
program. Journal of Behavioral Education, 17, 356–375.
doi:10.1007/s10864-008-9073-2
*Mechling, L. C., Gast, D. L., & Cronin, B. A. (2006). The effects
of presenting high-preference items, paired with choice, via
computer-based video programming on task completion of stu-
dents with autism. Focus on Autism & Other Developmental
Disabilities, 21, 7–13. doi:10.1177/10883576060210010201
Moeller, J. D., Dattilo, J., & Rusch, F. (2015). Applying quality
indicators to single-case research designs used in special edu-
cation: A systematic review. Psychology in the Schools, 52,
139–153. doi:10.1002/pits.21801
*Myrow, D. L. (1979). Learner choice and task engagement.
Journal of Experimental Education, 47, 200–207. doi:10.108
0/00220973.1979.11011682
Royer et al. 107
Newman, B., Needelman, M., Reinecke, D. R., & Robek, A. (2002).
The effect of providing choices on skill acquisition and com-
peting behavior of children with autism during discrete trial
instruction. Behavioral Interventions, 17, 31–41. doi:10.1002/
bin.99
Niesyn, M. E. (2009). Strategies for success: Evidence-based
instructional practices for students with emotional and
behavioral disorders. Preventing School Failure: Alternative
Education for Children and Youth, 53, 227–234. doi:10.3200/
psfl.53.4.227-234
Nota, L., Ferrari, L., Soresi, S., & Wehmeyer, M. (2007). Self-
determination, social abilities and the quality of life of people
with intellectual disability. Journal of Intellectual Disability
Research, 51, 850–865. doi:10.1111/j.1365-2788.2006.00939.x
*Patall, E. A. (2012). The motivational complexity of choosing: A
review of theory and research. In R. M. Ryan (Ed.), The Oxford
handbook of human motivation (pp. 248–279). Oxford, UK:
Oxford University Press.
Patall, E. A., Cooper, H., & Wynn, S. R. (2010). The effectiveness
and relative importance of choice in the classroom. Journal of
Educational Psychology, 102, 896–915. doi:10.1037/a0019545
*Powell, S., & Nelson, B. (1997). Effects of choosing academic
assignments on a student with attention deficit hyperactivity
disorder. Journal of Applied Behavior Analysis, 30, 181–183.
doi:10.1901/jaba.1997.30-181
Prasse, D. P., Breunlin, R. J., Giroux, D., Hunt, J., Morrison, D., &
Thier, K. (2012). Embedding multi-tiered system of supports/
response to intervention into teacher preparation. Learning
Disabilities: A Contemporary Journal, 10, 75–93.
Prusak, K. A. (2004). Adolescent girls in physical education.
Journal of Teaching in Physical Education, 23, 19–29.
Ramsey, M. L., Jolivette, K., Patterson, D. P., & Kennedy, C.
(2010). Using choice to increase time on-task, task-completion,
and accuracy for students with emotional/behavior disorders in
a residential facility. Education and Treatment of Children, 33,
1–21. doi:10.1353/etc.0.0085
Reimers, T. M., Wacker, D. P., Cooper, L. J., & DeRaad, A. O.
(1992). Clinical evaluation of the variables associated with
treatment acceptability and their relation to compliance.
Behavioral Disorders, 18, 67–76.
*Rispoli, M., Lang, R., Neely, L., Camargo, S., Hutchins, N.,
Davenport, K., & Goodwyn, F. (2013). A comparison of
within- and across-activity choices for reducing challenging
behavior in children with autism spectrum disorders. Journal
of Behavioral Education, 22, 66–83. doi:10.1007/s10864-012-
9164-y
*Romaniuk, C., Miltenberger, R., Conyers, C., Jenner, N., Jurgens,
M., & Ringenberg, C. (2002). The influence of activity choice
on problem behaviors maintained by escape versus atten-
tion. Journal of Applied Behavior Analysis, 35, 349–362.
doi:10.1901/jaba.2002.35-349
Romaniuk, C., & Miltenberger, R. G. (2001). The influence
of preference and choice of activity on problem behavior.
Journal of Positive Behavior Interventions, 3, 152–159.
doi:10.1177/109830070100300303
Sailor, W. (2015). Advances in schoolwide inclusive school
reform. Remedial and Special Education, 36, 94–99.
doi:10.1177/0741932514555021
*Seybert, S., Dunlap, G., & Ferro, J. (1996). The effects of choice-
making on the problem behaviors of high school students with
intellectual disabilities. Journal of Behavioral Education, 6,
49–65. doi:10.1007/BF02110477
Shadish, W. R., Hedges, L. V., Horner, R. H., & Odom, S. L. (2015).
The role of between-case effect size in conducting, interpret-
ing, and summarizing single-case research. Washington, DC:
National Center for Education Research, Institute of Education
Sciences, U.S. Department of Education.
Shogren, K. A., Faggella-Luby, M. N., Bae, S. J., & Wehmeyer,
M. L. (2004). The effect of choice-making as an intervention
for problem behavior: A meta-analysis. Journal of Positive
Behavior Interventions, 6, 228–237. doi:10.1177/1098300704
0060040401
Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., & Sugai,
G. (2008). Evidence-based practices in classroom manage-
ment: Considerations for research to practice. Education and
Treatment of Children, 31, 351–380. doi:10.1353/etc.0.0007
*Skerbetz, M. D., & Kostewicz, D. E. (2013). Academic choice
for included students with emotional and behavioral dis-
orders. Preventing School Failure: Alternative Education
for Children and Youth, 57, 212–222. doi:10.1080/10459
88X.2012.701252
*Skerbetz, M. D., & Kostewicz, D. E. (2015). Consequence choice
and students with emotional and behavioral disabilities: Effects
on academic engagement. Exceptionality, 23, 14–33. doi:10.10
80/09362835.2014.986603
Slaughter, S. E., Hill, J. N., & Snelgrove-Clarke, E. (2015). What
is the extent and quality of documentation and reporting
of fidelity to implementation strategies: A scoping review.
Implementation Science, 10, Article 129. doi:10.1186/s13012-
015-0320-3
Slavin, R. E. (2008). Perspectives on evidence-based research in
education: What works? Issues in synthesizing educational
program evaluations. Educational Researcher, 37, 5–14. doi:
10.3102/0013189X08314117
*Smeltzer, S. S., Graff, R. B., Ahearn, W. H., & Libby, M. E. (2009).
Effect of choice of task sequence on responding. Research
in Autism Spectrum Disorders, 3, 734–742. doi:10.1016/j.
rasd.2009.02.002
*Stenhoff, D. M., Davey, B. J., & Lignugaris-Kraft, B. (2008). The
effects of choice on assignment completion and percent correct
by a high school student with a learning disability. Education
and Treatment of Children, 31, 203–211. doi:10.1353/
etc.0.0011
*Vaughn, B. J., & Horner, R. H. (1997). Identifying instructional
tasks that occasion problem behaviors and assessing the effects
of student versus teacher choice among these tasks. Journal
of Applied Behavior Analysis, 30, 299–312. doi:10.1901/
jaba.1997.30-299
Walker, H. M., Forness, S. R., & Lane, K. L. (2014). Design and
management of scientific research in applied school settings.
In B. Cook, M. Tankersley, & T. J. Landrum (Eds.), Advances
in learning and behavioral disabilities: Special education past,
present, and future: Perspectives from the field (Vol. 27, pp.
141–169). Bingley, UK: Emerald.
Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom
intervention strategies. In T. R. Kratochwill (Ed.), Advances
in school psychology (Vol. 4, pp. 251–288). Mahwah, NJ:
Routledge.
Woodcock, R. W., McGrew, K. S., & Mather, N. (2001). Woodcock-
Johnson III. Rolling Meadows, IL: Riverside.
Copyright of Behavioral Disorders is the property of Council for Children with Behavioral
Disorders and its content may not be copied or emailed to multiple sites or posted to a listserv
without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.