573630

research-article2015
PBIXXX10.1177/1098300715573630Journal of Positive Behavior InterventionsMaggin et al.

Literature Review
Journal of Positive Behavior Interventions

A Systematic Evidence Review of the

2015, Vol. 17(4) 197­–208
© Hammill Institute on Disabilities 2015
Reprints and permissions:
Check-In/Check-Out Program for sagepub.com/journalsPermissions.nav
DOI: 10.1177/1098300715573630

Reducing Student Challenging Behaviors jpbi.sagepub.com

Daniel M. Maggin, PhD1, Jamie Zurheide, MA1,

Kayci C. Pickett, MA1, and Sara J. Baillie, MA1

Abstract
Three-tiered models of prevention are increasingly being adopted by schools to address the behavioral needs of students.
A critical component of multitiered systems are secondary interventions used with students in need of behavioral support
but who are not candidates for individualized interventions. Despite the importance of secondary interventions, questions
remain regarding which approaches have sufficient empirical support to warrant their use. The purpose of this review
was, therefore, to examine the research underlying the Check-In/Check-Out (CICO) program, a widely used secondary
intervention, to determine the strengths, limitations, and generality of the accumulated research. The What Works
Clearinghouse (WWC) procedures for evaluating single-case and group-based research were applied with results indicating
mixed support for the program. Specifically, there were a sufficient number of single-case research studies to deem
the CICO program as evidence-based, while the group-based research had no demonstrated effects. These findings are
discussed in terms of future research on the CICO program and the broader implications for selecting and implementing
secondary interventions in school settings.

Keywords
check-in/check-out, evidence-based practice, positive behavioral interventions and supports, secondary interventions,
systematic review

The three-tiered model of prevention has become increas-
ingly prevalent in schools to address the behavioral needs of
students. These systems emphasize the use of data-based
decision-making procedures to increase the efficiency of
identifying students in need of additional behavioral sup-
ports beyond those provided to all students (Sugai & Horner,
2009). The stratification of behavioral supports into distinct
tiers allows for school personnel to allocate resources more
readily to those students with greater behavioral needs. As
such, intermediary interventions are a critical element of the
three-tiered model because these methods are used to
ensure students who are non-responsive to primary preven-
tion strategies are efficiently provided additional supports
and monitoring (Ervin, Schaughency, Matthews, Goodman,
& McGlinchey, 2007). Ideally, these secondary interven-
tions consist of a standard set of procedures and are con-
tinuously available within a school to provide eligible
students with prompt access to needed supports (Scott,
Alter, Rosenberg, & Borgmeier, 2010). Moreover, second-
ary interventions assist school personnel in distinguishing
between students in need of moderate levels of support and
those who require more intensive, individualized behavior
plans. As such, the adoption and successful implementation
of secondary interventions has important implications for
the overall strength of multitiered systems of behavioral
support and prevention.
The importance of secondary interventions to the overall
functioning of these three-tiered systems of behavioral sup-
port leads to important questions regarding which approaches
should be adopted. Broadly speaking, secondary interven-
tions represent a diverse set of procedures for addressing the
variety of behavioral challenges demonstrated by students
(Hawken, Adolphson, Macleod, & Schumann, 2009). As
such, school personnel charged with intervention selection
must weigh several factors, including the behavior or skill
being targeted and the resources available, to ensure the
adopted strategy can be successfully implemented (Mitchell,
Stormont, & Gage, 2011). Moreover, school personnel must
be cognizant of federal initiatives that emphasize the use of
1
University of Illinois at Chicago, USA
Corresponding Author:
Daniel M. Maggin, University of Illinois at Chicago, 1040 W Harrison St.,
Chicago, IL 60607, USA.
Email: dan.m.maggin@gmail.com
Action Editor: Lee Kern
198
practices with sufficient empirical support (Individuals With
Disabilities Education Act [IDEA], 2004; No Child Left
Behind [NCLB], 2002). It is therefore essential for practitio-
ners to consider the extent to which selected practices are
supported through rigorous and appropriate research meth-
ods. The present review provides an evaluation of the
research underlying check-in/check-out, a widely used sec-
ondary intervention to assist with describing the strengths,
limitations, and generality of the accumulated research for
this program.
Check-In/Check-Out
The Check-In/Check-Out (CICO) program has emerged as
a model secondary intervention for students who do not
respond to universal, preventive methods (Crone, Hawken,
& Horner, 2010). Based on the principles of contingency
management, the premise of CICO is to provide students
with more frequent and structured access to positive conse-
quences contingent on the demonstration of appropriate
behavior. The procedures of CICO have been outlined in a
program manual (Crone et al., 2010) and include five core
components including (a) the morning check-in during
which the target student meets with a member of the school
faculty and receives positive, non-contingent attention and
encouragement to meet the specified behavioral expecta-
tions; (b) the daily behavior point card, which is given to the
student during the morning check-in and provides school
personnel with a means for monitoring the extent to which
students are meeting the behavioral expectations; (c) struc-
tured teacher feedback which is provided to students
throughout the school day at regularly scheduled intervals
and is delivered through both verbal interaction and point
card ratings; (d) the afternoon check-out during which the
student’s point card is reviewed to determine the percentage
of points earned with a reward such as verbal praise or a
small tangible item delivered contingent on whether the stu-
dent met their goal; and (e) a home–school collaboration
component in which the student carries their point card
home, which is then signed by the parent or guardian. The
point card system, therefore, provides school personnel
with a structured method for giving behavioral feedback to
students and also assists school personnel in collecting data
to evaluate the effectiveness of the intervention. Moreover,
there have been numerous reports investigating the pro-
gram’s effectiveness, which have contributed to its wide-
spread use and adoption (e.g., Fairbanks, Sugai, Guardino,
& Lathrop, 2007; Filter et al., 2007; McIntosh, Campbell,
Carter, & Dickey, 2009; Simonsen, Myers, & Briere, 2011).
Given the burgeoning research on CICO, there is a need to
evaluate the evidence underlying the program to determine
whether and under what conditions the practice can be con-
sidered effective.
Three issues have emerged within the CICO research
with each having important implications for evaluating the
Journal of Positive Behavior Interventions 17(4)
evidence base. First, researchers investigating the effects of
CICO have questioned whether the student’s behavioral func-
tion serves to moderate the effects of the program (e.g.,
Hawken, MacLeod, & Rawlings, 2007). There seems to be
consensus that CICO is most appropriate for students seeking
adult or peer attention (Campbell & Anderson, 2011), although
there has yet to be a formal analysis across studies to assess
the validity of these claims. Second, there is a need to evaluate
the consistency with which the procedures and components of
the program have been implemented across studies. Variation
in the implementation of program components or low levels
of implementation fidelity would raise questions regarding
the comparability of the research. This issue is particularly
important given recent CICO research investigating the
effects of adapting, modifying, or supplementing the core pro-
cedures and components of the program (e.g., Swoszowski,
Jolivette, Fredrick, & Heflin, 2012; Turtura, Anderson, &
Boyd, 2014). Third, the CICO program is unique, in that it has
been researched using both single-case and group-based
experimental methods. Conducting parallel reviews of the
research from these methodological paradigms can be chal-
lenging because each is based on a unique set of principles
and procedures (Gast, 2010). Given the inherent methodologi-
cal differences, it is necessary to interpret the results within
the context of both the review findings and the research para-
digms used. These aspects of the CICO research base will be
addressed within the context of the current review.
Review Purpose
The purpose of this review was to evaluate the research
underlying the CICO program to determine its strengths,
limitations, and generality. The following research ques-
tions have been developed to guide the present review:
Research Question 1: Does the single-case and group-
based experimental research provide sufficient evidence
that the CICO program is effective to warrant classifica-
tion as an evidence-based practice?
Research Question 2: What student and setting charac-
teristics have been associated with successful demon-
strations of the CICO program?
Research Question 3: Have the core procedural compo-
nents of the CICO program been used consistently and
with fidelity across research studies?
Method
Study Identification Procedures
The studies for the present review were identified using
(a) an electronic database search of seven widely used
educational and social sciences databases including
Dissertation Abstracts, Educational Resources Information
Center (ERIC), ProQuest, PsychINFO, PubMed, Scopus,
Maggin et al.
and Sociological Abstracts using the two names associated
with the program: “Check-In Check-Out” and “Behavior
Education Program”; (b) an ancestral search of the refer-
ence lists of eligible studies; (c) a review of the citation
lists of relevant review articles and the program manual;
and (d) a review of the citation list by the program devel-
opers to ensure all relevant citations were identified. These
search procedures led to the identification of 30 research
reports deemed potentially eligible for review.
Eligibility Criteria
The 30 research reports identified through the initial search
were reviewed using the following five inclusion criteria: (a)
the CICO program was used regardless of whether it was
implemented with another intervention or adapted to broaden
its potential reach, (b) the study utilized an experimental or
quasi-experimental research design based on either group-
or single-case research methods, (c) the study was conducted
with students enrolled in kindergarten through 12th grade,
(d) the study was conducted in a school setting, and (e) the
outcome variable was related to problematic student behav-
ior or academic engagement. Application of these criteria
led to the removal of 8 reports that were not conducted using
experimental methods. The remaining 22 studies were then
reviewed for the present report.
Study Coding Procedures
The present review contained both single-case and group-
based research studies. Because these research paradigms
have distinct conceptual underpinnings and methodological
features, the studies from these methodologies were
reviewed and summarized separately. All coding was com-
pleted by the secondary authors with discrepancies reviewed
by the lead author who made a final determination on the
appropriate code. The procedures used to review the eligi-
ble studies for each methodological framework were based
on recommendations from the What Works Clearinghouse
(WWC). These procedures use a bifurcated process, which
includes (a) an initial evaluation of the methodological
rigor for eligible studies and (b) a subsequent assessment of
the strength of observed intervention effects for those stud-
ies deemed sufficiently rigorous. The classification as an
evidence-based practice was then made based on the num-
ber of studies meeting design standards and providing evi-
dence. For the present review, the studies providing
evidence of effectiveness were examined further to evaluate
the conditions and contexts under which CICO was found
to be effective. The specific procedures and analyses for
each methodological framework are described in the fol-
lowing sections.
Single-case research coding procedures and analyses. The 22
eligible studies included a total of 17 single-case research
199
reports with 79 student cases. Each student case was evalu-
ated on design and evidence standards independently. The
total number of studies providing evidence was then used as
the basis for evaluating the overall strength of evidence per
the WWC guideline. The specific procedures for evaluating
the single-case research are described in the following
sections.
Application of single-case design standards. The WWC sin-
gle-case design standards consist of six criteria (Kratoch-
will et al., 2010). The initial five design criteria were scored
on a dichotomous scale (i.e., present or not present). These
include the following: (a) The independent variable was
systematically manipulated with the researcher determining
when and how the conditions changed, (b) each dependent
variable was measured repeatedly over time, (c) a measure
of interobserver agreement was reported for each dependent
variable for no lesser than 20% of sessions, (d) the reported
level of interobserver agreement was equal to or greater
than 80% for percentage agreement indices, and (e) the
case had at least three attempts to demonstrate an interven-
tion effect. Failure to meet any of these criteria led to the
classification of the case as not meeting design standards.
Those cases that met these five criteria were then subjected
to the sixth design standard which relates to the number of
data points per phase. Consistent with the recommendations
of the WWC, this criterion is scored on a three-point scale
and provides the context for distinguishing between cases
that meet with and without reservations. Specifically, cases
are determined to meet design standards if there are five
or more data points per phase, meet design standards with
reservations if there are three or four data points per phase,
and not meet design standards if there are fewer than three
data points per phase.
Application of single-case evidence standards. The student
cases within each study meeting design standards with or
without reservations were then visually analyzed to evalu-
ate the strength of evidence. For the present study, visual
analyses were conducted using a protocol that prompted
coders to consider the range of data features used to assess
single-case research. Specifically, visual analysts examined
the within- and between-phase data patterns to make judg-
ments regarding intervention effects. Within-phase analyses
were used to assess the level, trend, and variability of the
data within each phase to determine whether there was a
sufficient basis to make predictions regarding future per-
formance. Between-phase analyses were used to deter-
mine whether there were discriminable differences across
phases and whether they reflected behavioral improve-
ment. Finally, visual analysts were prompted to consider
the ratio of phase contrasts with and without effects. Per
the WWC evidence standards, a ratio greater than 3:1 indi-
cates that the case provides evidence without reservations,
a ratio of 3:1 indicates that the case provides evidence with
200
reservations, and a ratio of less than 3:1 indicates that the
case provides no evidence. The visual analysts incorporated
the information pertaining to the number of data points, the
within- and between-phase data patterns, and the ratio of
effects to non-effects to determine the overall strength of
the visual evidence. The evidence standards, therefore, are
based on the traditional processes of visual analysis such as
the examination and comparison of within- and between-
phase data and specific guidelines outlined in the WWC
guidelines such as the number of data points and ratio of
effects to non-effects. Each student case was then classi-
fied as providing evidence, evidence with reservations, or
no evidence. The WWC considers an intervention to be
evidence-based if there are at least 20 student cases provid-
ing evidence with or without reservations from five differ-
ent studies conducted by three independent research teams.
The strength of these results was further examined through
computing a ratio of those cases meeting design standards
that either provided or did not provide evidence.
Quantitative evaluation of single-case research. The visual
analyses were supplemented with quantitative methods
to provide a more transparent evaluation of intervention
effects. It is important to note that there remains a lack
of consensus regarding the most appropriate statistical
methods to evaluate single-case research data (Shadish,
Rindskopf, & Hedges, 2008). As such, visual analyses
were used as the sole basis for classifying studies into evi-
dence categories and the statistical analyses were applied
only for supplemental purposes. Moreover, the unsettled
nature of statistical analyses of single-case data led to
the use of four metrics. The four metrics were applied to
data extracted using the Engague software program (M.
Mitchell, 2007). The four metrics included (a) the non-
overlap of all pairs (NAP), which provides a summary of
data overlap between all the data points within the phases
being compared, with a value of 65% or below associ-
ated with a weak effect, between 66% and 92% associated
with a moderate effect, and 93% and above considered a
strong effect (Parker & Vannest, 2009); (b) the improve-
ment rate difference (IRD), which also provides a sum-
mary of data overlap between phases with values of 70%
or above associated with a large effect, between 60%
and 70% considered moderate, and below 60% associ-
ated with small effects; (c) the Busk and Serlin (1992) no
assumptions standardized mean difference (SMD) metric,
which provides a standardized measure of the difference
between baseline and intervention data with values greater
than 2.0 considered clinically significant; and (d) the raw
data multilevel analysis (RDMA) procedures described by
Van den Noortgate and Onghena (2008), which also pro-
vides a standardized estimate of the difference between
baseline and intervention phases and is assessed with the
same interpretative guidelines used for the SMD. Overall
Journal of Positive Behavior Interventions 17(4)
estimates for each index were computed using multilevel
modeling procedures, which accounted for variability at
the student and study levels.
Group-based research coding procedures and analyses. The 22
eligible studies included a total of five group-based research
reports. Consistent with the review of single-case research,
the coding of methodological quality was followed by an
evaluation of the strength of evidence for those studies
meeting design standards with and without reservations.
The evaluation of methodological quality and estimation of
effect size metrics to assess the strength of findings are
described in the following sections.
Application of group-based design standards. The WWC
group-based design standards consist of five criteria (What
Works Clearinghouse, 2013). The initial five criteria
include (a) the random assignment of individuals to inter-
vention and control, (b) the reporting of overall and differ-
ential rates of attrition that are within acceptable limits, (c)
the demonstration that students remaining and leaving the
study are comparable on conceptually relevant variables,
(d) the demonstration that students in treatment and control
conditions are comparable on conceptually relevant vari-
ables, and (e) the use of outcome variables that are suffi-
ciently valid and reliable, and were collected using the same
procedures for both groups. Design classifications were
based on the extent to which the studies met these criteria
with those using random assignment and demonstrating low
levels of attrition considered to meet standards without res-
ervations; those studies not using random assignment but
having established group equivalence considered meeting
standards with reservations; and those studies that are non-
randomized without establishing group equivalence consid-
ered not meeting standards.
Application of group-based evidence standards. The mean
and standard deviations for the treatment and control groups
of studies meeting design criteria were extracted. This infor-
mation was then used to calculate a SMD effect size, which
is computed by subtracting the intervention group’s mean
on the dependent variable from the control group’s mean on
the dependent variable and dividing by the pooled standard
deviation of the dependent variable. The resulting index
provides an estimate of the difference between the groups
in standard deviation units. Per the WWC guidelines, effect
sizes of .25 or larger with demonstrated statistical signifi-
cance at .05 are considered substantively important. Cor-
rections for misaligned analyses and multiple outcomes
were implemented as needed and described below within
the context of the review process. The WWC considers a
practice to be evidence-based if there are two or more stud-
ies meeting standards with positive effects that are statisti-
cally significant.
Maggin et al.
Coding study characteristics. The studies meeting evidence
standards with or without reservations were then reviewed
on a variety of study characteristics to identify the condi-
tions to which the effects of CICO might apply. These char-
acteristics were coded into domains related to specific
features of the sample, setting, and program implementa-
tion. Items pertaining to the sample and setting domain
related to general demographic characteristics of the sample
(e.g., gender, age, and grade), the identified function of stu-
dent behavior, and the type of settings in which the study
was conducted. Items included within the program imple-
mentation domain related to the specific CICO components
implemented and the methods for collecting fidelity data.
These items were applied to each student case included in
single-case reports and each study for group-based reports.
Coder Reliability
Eight studies (36%) were randomly selected for recoding by
a second rater. This random selection of studies included
five single-case and three group-based studies, which repre-
sented approximately 29% and 60% of the eligible reports
from these methodologies, respectively. A series of percent-
age agreement indices were calculated to evaluate the con-
sistency of coder ratings for each criterion and domain with
agreement for single-case and group-based research reports
computed separately. The mean percentage agreement
across all items for the single-case coding protocol was 94%
with agreement on items relating to the design (M = 99%,
range = 94%-100%), evidence (M = 89%, range = 84%–
100%), and study characteristics (M = 93%, range = 87%–
100%) indicating strong agreement across coders. Overall
mean percent agreement for items of the group-based proto-
col was 96% with agreement on items relating to the design
(M = 100%, range = 100%–100%), evidence (100%, range
= 100%–100%), and study characteristics (88%, range =
82%–96%) also indicating strong agreement across coders.
Results
Evaluation of Single-Case Research Reports
Single-case design evaluation. A schematic outlining the
application of the WWC design and evidence standards is
provided in Figure 1. The 17 single-case studies eligible for
review included a total of 79 students. Forty-four (56%)
student cases were classified as not meeting design stan-
dards. The most common reasons for their exclusion related
to issues with interobserver agreement (n = 29) and not pro-
viding a sufficient number of data points per phase to meet
design standards (n = 15). This application of design stan-
dards resulted in the reduction of the original pool from 79
to 35 student cases and 17 to 9 studies eligible for evidence
review.
201
Single-case evidence evaluation. Following the application of
design standards, the graphed data were visually analyzed.
The visual analyses were conducted on each eligible depen-
dent variable reported for all student cases. Results of the
visual analyses indicated that 28 (80%) of the 35 cases
meeting design standards provided visual evidence of inter-
vention effects. As such, there was a 4:1 ratio of cases meet-
ing design standards providing visual evidence to those not.
The reason all of the cases met evidence standards with res-
ervations was either due to too few data points present
within phases or the presence of non-effects. It is also worth
noting that the seven cases not meeting visual evidence
standards did not provide any indication of harmful effects.
As such, there was no evidence that the CICO program led
to increases in problematic behavior or decreases in aca-
demic engagement. Rather, the data patterns indicated the
presence of null effects. The 28 cases meeting design stan-
dards were drawn from a total of 8 studies across 4 indepen-
dent research teams indicating that the practice can be
labeled as evidence-based according to the WWC single-
case design standards.
Single-case quantitative evaluation. The visual analyses were
augmented with the computation of quantitative indices of
treatment effect for cases providing evidence with reserva-
tions in accordance with the WWC standards. Results of
these analyses were mixed with those based on data overlap
indicating a moderate intervention effect and those based on
a more traditional statistical framework indicating a weak
effect. For problem behavior, the NAP and IRD estimates
were found to be 83% (SE = .03) and 62% (SE = .05),
respectively, with both found to be statistically significant
at the p < .05 level. The SMD for problem behavior was
found to be 1.46 (SE = .27), while the RDMA was 1.38 (SE
= .21) with both being statistically significant at the p <
.001 level. The results for academic engagement were simi-
lar with NAP and IRD estimates of 88% (SE = .02) and
74% (SE = .05), respectively, with both significant at the p
< .001 level. The SMD for academic engagement was esti-
mated at 1.77 (SE = .14) and the RDMA was estimated to
be 1.91 (SE = .67) with both significant at the p < .01 level.
These quantitative results are consistent with the visual evi-
dence evaluations indicating moderate levels of support.
Specifically, the visual analyses of baseline data confirmed
that all cases providing evidential support merited interven-
tion given the levels of problem behavior, though many also
had moderate variability at baseline. This variability would
have an effect on both the overlap and parametric statistics.
That is, the increased variability in baseline increases the
chance of overlapping data points for the overlap methods
and the error terms in the parametric statistics.
Study characteristics evaluation of single-case research. Fol-
lowing the application of the WWC evidence standards,
202 Journal of Positive Behavior Interventions 17(4)

Total Studies Eligible for Review

n = 22

Design Evaluation
A total of 17 single-case studies with
A total of 5 group-based studies were
79 student cases were eligible for
eligible for review
review

Meets Design Standards with Reservations

Meets Design Standards Does Not Meet Design Standards
3 single-case studies with 10 student cases
6 single-case studies with 25 student cases 8 single-case studies with 44 student cases
1 group-based study
1 group-based study 3 group-based studies

Evidence Evaluation

Evidence Evidence with Reservations No Evidence

0 single-case studies with 0 student cases 8 single-case studies with 28 student cases 9 single-case studies with 51 student cases

0 group-based studies 0 Group-based studies 5 group-based studies

Figure 1. Schematic overview of the What Works Clearinghouse (WWC) design and evidence standards as applied to the database
of single-case and group-based research eligible for review.
Note. Consistent with the guidelines, single-case research studies were reviewed at the case level and group-based studies reviewed at the study level.
Those studies not meeting design standards were deemed as not providing evidence in accordance with the WWC standards.

the 28 cases providing evidence with reservations were
reviewed to assess the potential generality of the research.
An overview of study characteristics is provided in Table 1.
A majority of the students for whom the intervention was
found to be effective were enrolled in a school with a
three-tiered model of behavior support (n = 22, 79%).
Moreover, 20 (71%) were found to be exposed to CICO in
general education with the other students enrolled in either
special education classrooms (n = 5, 18%) or schools (n =
3, 11%). In terms of demographics, most of the students
responding to the program were also found to be male (n
= 24, 86%) with students ranging from kindergarten
through 11th grade. With respect to disability status, sev-
eral students did not have a disability (n = 17, 61%), and
others had learning disabilities (n = 5, 18%), emotional
and behavioral disorders (n = 3, 11%), speech and lan-
guage impairment (n = 1, 4%), and two had unspecified
disabilities (7%). In addition, the function of student
behavior was identified for all the 28 students found to be
responsive. The most common function of behavior was
either teacher or peer attention (n = 20, 71%) with some
student behavior hypothesized to be maintained through
escape from challenging academic work (n = 8, 29%). For
those students with attention-maintained behaviors, the
behavior of 11 students was maintained solely through
adult attention, 4 only through peer attention, 2 through a
combination of peer and adult attention, with the remain-
ing 3 not specified. Interestingly, there were four students
in studies meeting design criteria for whom functional
assessment data indicated escape-maintained problem
behavior and who did not respond to CICO. This repre-
sents a much larger proportion of non-responding students
in studies meeting design standards than those with atten-
tion-maintained problem behavior.
Maggin et al. 203

Table 1. Overview of Study Characteristics for Single-Case Research Studies Meeting Evidence Standards With Reservations.

Student Dependent Age Primary Behavioral

Study case variable (year) Grade disability function CICO program
Campbell and Anderson Joe DB 10 N/R None PA Adapted, function-based
(2008) support
Kyle DB 10 N/R None PA Adapted, function-based
support
Campbell and Anderson Kyle AE, DB N/R 2nd LD AA Standard
(2011) Nick AE, DB N/R 5th SLI AA Standard
Paul AE, DB N/R 5th LD AA Standard
Hawken and Horner Jalen DB 13 6th LD PA Standard
(2003) Martin DB 13 6th None AA, PA Standard
Ryan DB 13 6th LD AA, PA Standard
Scott DB 12 6th None PA Standard
Mong, Johnson, and Lauren DB 8 3rd None AA Standard
Mong (2011) Andrew DB 8 3rd None AA Standard
Pam DB 9 3rd None AA Standard
Stanley DB 8 3rd None AA Standard
Swain-Bradway (2009) Donovan AE 15 10th None EDW Adapted, study skills class
Joy AE 16 11th NS EDW Adapted, study skills class
Lee AE 14 9th NS EDW Adapted, study skills class
Ricky AE 14 9th None EDW Adapted, study skills class
Travis AE 15 10th None EDW Adapted, study skills class
Swoszowski, Jolivette, Daniel DB 12 7th EBD A, NS Standard
Fredrick, and Heflin Leo DB 14 9th EBD A, NS Standard
(2010) Tyrone DB 12 6th EBD A, NS Standard
Todd, Campbell, Meyer, Chad DB N/R 1st None AA Standard
and Horner (2008) Eric DB N/R Kindergarten None AA Standard
Kendall DB N/R 2nd None AA Standard
Trevor DB N/R 3rd LD AA Standard
Turtura, Anderson, and Katie AE, DB N/R 7th None EDW Adapted, modified
Boyd (2014) expectations
Nick AE, DB N/R 6th grade None EDW Adapted, modified
expectations
Toby AE, DB N/R 8th grade None EDW Adapted, modified
expectations

Note. CICO = Check-In/Check-Out; DB = disruptive behavior; N/R = not reported; PA = peer attention; AE = academic engagement; LD = learning
disability; AA = adult attention; SLI = speech and language impairment; EDW = escape from difficult work; EBD = emotional and behavioral disorder;
A, NS = attention, not specified.

The evaluation of single-case research studies con-
cluded with an implementation analysis to determine
whether the program was consistently administered across
studies. An overview of the implementation analysis is
provided in Table 2. All eight studies with cases meeting
evidence standards with reservations reported using the
core CICO components of (a) implementing a student
check-in with an adult at the beginning of the day, (b) pro-
viding students with a daily point card with behavioral
expectations, (c) having teachers provide feedback to stu-
dents using the point card, and (d) implementing a student
check-out with an adult at the end of the day. Moreover,
seven of the studies contained information pertaining to
the extent to which the core program components were
implemented with generally positive results although only
three provided fidelity for each program component.
There was variability observed in the percentage of ses-
sions for which fidelity data were collected ranging from
nearly 35% to approximately 10% of sessions for each
participant. The procedures for collecting fidelity data
also varied across studies. Specifically, three of the studies
used direct observation, four used permanent product data,
and one used a combination of both methods. Researchers
using direct observation methods were more likely to
evaluate the accuracy of the fidelity data by reporting
interobserver agreement statistics with each demonstrat-
ing high rates of agreement across observers. Finally,
additional components were used to augment the CICO
204 Journal of Positive Behavior Interventions 17(4)

Table 2. An Overview of Implementation Fidelity Reporting and Methods for Single-Case Studies Meeting Evidence Standards With
Reservations.

Fidelity results and methods Fidelity reported for specific components

Reported Proportion Data Morning Daily Teacher Afternoon Home

Study fidelity of sessions collection IOAa check-in point card feedback check-out collaboration
Campbell and Anderson 100% 34% Direct N/R 100% 100% 100% 100% N/R
(2008) observation
Campbell and Anderson 97% 27% Direct 97% N/R N/R N/R N/R N/R
(2011) observation
Hawken and Horner 85% 17% Permanent N/R 83% 92% 92% 75% 67%
(2003) product
Mong, Johnson, and 92% 33% Bothb 100% N/R N/R N/R N/R N/R
Mong (2011)
Swain-Bradway (2009)c 28% 10% Permanent N/R N/R N/R N/R N/R N/R
product
Swoszowski, Jolivette, 94% 22% Direct 99% N/R N/R N/R N/R N/R
Fredrick, and Heflin observation
(2010)
Todd, Campbell, Meyer, N/R N/R N/R N/R N/R N/R N/R N/R N/R
and Horner (2008)
Turtura, Anderson, and 92% N/R Permanent N/R 96% 100% 93% 88% 84%
Boyd (2014) product

Note. N/R = not reported.

a
IOA = Interobserver agreement with percentages referring to reported level of interobserver agreement for fidelity checks. bUsed direct observation
for check-in and check-out and permanent product for the other procedures. cValues are estimated from available data.

program in three of the studies providing evidence with
reservations. Adaptations included the use of function-
based reinforcement for appropriate behavior in the class-
room (Campbell & Anderson, 2011), providing academic
assistance (Swain-Bradway, 2009), and modifying behav-
ioral expectations to be more specific to the student
(Turtura et al., 2014). These modifications to the standard
CICO protocol were most prevalent in studies that found
the intervention to be effective for students with escape-
maintained behaviors indicating that such adaptations
might be used to broaden the reach of the program.
Evaluation of Group-Based Research Reports
Group-based design evaluation. A schematic overview of the
application of design and evidence standards is provided in
Figure 1. The application of the WWC group-based stan-
dards revealed that four of the five studies did not use ran-
dom assignment of subjects and, therefore, were reviewed
as quasi-experimental designs. Three of these quasi-experi-
mental designs did not provide information on the pretest
comparison of students resulting in their classification as
not meeting design standards. The final quasi-experimental
design provided information pertaining to pretest means,
standard deviations, and sample sizes, allowing for analysis
(Simonsen et al., 2011). Results revealed that there were no
pretest differences between the treatment and comparison
groups leading the study to be classified as meeting design
standards with reservations. The fifth study randomly
assigned students to either the intervention or comparison
group and met all remaining criteria for group experimental
studies (Cheney et al., 2009). As a result, there were two
studies meeting the design criteria with quasi-experimental
meeting with reservations and randomized trial meeting
without reservations.
Group-based evidence evaluation. Following the application
of the design standards, the posttest effect sizes for the
studies meeting design criteria were calculated for all
behavioral outcome variables from the data available in the
research report. Before describing these results, however, it
is necessary to review the analytic procedures used for
each study. First, the authors of the qualifying quasi-exper-
imental study reported SMD effect sizes based on change
scores (Simonsen et al., 2011). The use of change scores
presents both conceptual and statistical challenges that
make it less ideal than the SMD, particularly with pretest
group equivalence established (McGaw & Glass, 1980).
Because the authors reported pretest and posttest means
and standard deviations, the more reliable and interpretable
SMD effect sizes could be readily computed and were used
for the present review. Results of these analyses revealed
that the SMD effect sizes ranged from small for social
skills ratings (d = .02, 95% CI [−0.61, 0.65]) to moderate
(d = .40, 95% CI [−0.24, 1.03]) with none of the estimates
indicating a statistically significant difference. Second, the
Maggin et al.
authors of the randomized trial reported results separately
for students who were responsive to the intervention and
those who were not (Cheney et al., 2009). The reported
means and standard deviations for these groups were,
therefore, pooled and then compared with those of the con-
trol group. Moreover, the significance tests associated with
the pretest and posttest effect sizes were further adjusted for
clustering due to the misalignment between the unit of ran-
dom assignment and the unit of analysis using the proce-
dures developed by Hedges (2007). Finally, the Benjamini
and Hochberg (1995) correction for multiple comparisons
was used because information for several outcome mea-
sures was reported in each of the qualifying studies. The
comparison of the clustered corrected probability values
with those derived from the correction for multiple com-
parisons revealed that no differences were present in the
analyses. Specifically, there were eight posttest compari-
sons reported with teacher-reported measures of internaliz-
ing behavior problems (p = .013) the only one that was
found to be below the p < .05 threshold following the clus-
tering correction. This probability value was then compared
with the recommended threshold associated with the cor-
rection for multiple comparisons, which was p < .006.
Because the probability value associated with the analyses
was larger than the one associated with the correction for
multiple comparisons, there was no evidence of an inter-
vention effect. None of the other cluster corrected estimates
were found to be below the p < .05 threshold indicating no
posttest differences between groups. Given that the group-
based research on the CICO program produced no positive
findings, the evaluation of study characteristics and imple-
mentation analysis for these studies was not conducted.
Discussion
The purpose of the present review was to describe the
strengths, limitations, and generality of the research under-
lying the CICO program. The results of the review were
mixed with several single-case studies providing evidence
with reservations and moderate visual and quantitative sup-
port, while none of the group-based studies provided evi-
dence due to a lack of rigor and results. Drawing meaningful
conclusions from these results requires careful consider-
ation of these research paradigms while also taking into
account other important characteristics of the intervention
and research. The principal findings of the review are sum-
marized in the following sections with implications for
practice subsequently described.
Summary of Principal Findings
Mixed results across research methods. The principal finding
of the present review was that the single-case research pro-
vided sufficient empirical support for the CICO program,
205
while the group-based research did not. These discrepant
findings require careful consideration of the characteristics
of both the intervention and the research methods used to
evaluate the CICO program. For instance, the single-case
research indicated that the student’s behavioral function
moderated the effectiveness of CICO (Hawken et al.,
2007). That is, students with attention-maintained problem
behaviors were more likely to respond to the standard
CICO protocol than those with escape-maintained behav-
iors. Unfortunately, this differential responding for stu-
dents based on behavioral function was not formally
investigated in either of the two group-based studies meet-
ing design standards. It is worth noting that Simonsen et al.
(2011) did establish that both groups were comprised of an
equivalent number of students with both attention- and
escape-maintained problem behaviors though differential
responding was not formally investigated. Moreover,
because both the group-based studies investigated the
effects of the standard CICO protocol, it is possible that the
effectiveness of the program was diluted due to the inclu-
sion of students with both attention- and escape-maintained
behavioral functions. In fact, descriptive group-based
research has provided some support to the importance of
considering behavioral function when using the CICO pro-
gram (e.g., McIntosh et al., 2009). Future experimental
group-based research might provide a context to verify the
role of function within the standard CICO framework with
more rigorous studies that exclusively sample students with
attention-maintained behavior problems.
Function-based adaptations. The results of the present review
seem to indicate that the standard CICO procedures tend to
be most effective for students with attention-maintained
behavioral problems. However, there was a subset of stud-
ies demonstrating that students with escape-maintained
behavior problems responded positively to the intervention
when it was adapted. For instance, Turtura et al. (2014)
made adjustments to the standard set of procedures to better
align the intervention framework for students with escape-
maintained behavioral issues. Specifically, these authors
developed behavioral expectations that directly addressed
issues of escape and avoidance. Results of this study were
positive with all students demonstrating reductions in dis-
ruptive behavior and increases in work completion. These
preliminary findings are encouraging, and although they
require additional research, might be meaningful for school
personnel seeking standard adaptations for increasing the
pool of students for whom CICO might work.
Implementation of core components. The research indicating
that adaptations of the standard CICO components might
increase the generality of the program must be considered
within the context of the extent to which those components
were implemented. As such, it is a notable strength of the
206
CICO research base that nearly all authors documented the
fidelity with which core components were implemented.
Fidelity was consistently high; however, only two of the
studies contained fidelity data for all standard components
of CICO. Parental signatures and afternoon checkout were
not implemented as consistently as other core components,
which raises questions regarding whether these components
are needed given the general positive findings of the single-
case research. There is currently insufficient information to
determine the role of these components, which might be
investigated through subsequent research. As such, all core
components should continue to be implemented until there
is sufficient information to determine which components
are necessary for success of the intervention.
Limitations
Findings of this review should be considered with the fol-
lowing limitations in mind. First, although every effort was
made to retrieve all studies pertaining to the use of the
CICO program, it is possible that some studies remained
unidentified. Second, although the use of the WWC criteria
to evaluate the methodological and evidential strength of
included studies provides some assurance that only the
most rigorous studies provided the basis for assessing the
generality of CICO, recent research has indicated that
rubrics used to evaluate methodological strength often do
not agree. As such, the results of the present review should
be viewed with the understanding that the use of other
rubrics with different criteria and scoring procedures might
lead to disparate findings (Maggin, Briesch, Chafouleas,
Ferguson, & Clark, 2014). Third, the reliance on visual
analysis to evaluate single-case research reports remains the
preferred method, although visual analyses lack transpar-
ency because the graphed data are not readily available to
the reader (Kratochwill et al., 2013). The visual analyses
were supported with supplemental quantitative metrics;
however, the absence of a reliable measure is a limitation
(Shadish et al., 2008).
Practical Implications
The results of the current investigation indicate that the
CICO program can be considered an effective secondary
intervention when used to directly address the student’s
behavioral function. In fact, the procedures seem to be
robust even outside of a multitiered system of behavioral
support when considering the function of behavior. The
capacity of the CICO program to lead to improved student
behavior irrespective of a broader system of support is
encouraging and likely relates to the general effectiveness
of contingency management programs broadly (Fisher &
Bouxsein, 2011). These are indeed positive findings, but
school personnel must consider the totality of the research
Journal of Positive Behavior Interventions 17(4)
to maximize the potential benefits of the intervention.
Specifically, school personnel should be aware that the
research consistently suggests that the standard CICO pro-
tocol is most likely to be successful for students whose
behavior is maintained through access to adult attention
(e.g., Campbell & Anderson, 2011). As such, the standard
approach to CICO is likely applicable for only a subset of
students. Considered within the context of secondary inter-
ventions, which are meant to address the needs of a majority
of students not responding to preventive supports, school
personnel should prioritize the enrollment of students with
attention-maintained behavior problems. Moreover, school
personnel should also strongly consider implementing
assessments of behavioral function within the decision-
making process to maximize the potency of the interven-
tion. Fortunately, there are assessments of behavioral
functioning that are both brief and with good psychometric
properties (March et al., 2000). The information drawn
from these assessments can also be used to fashion the
CICO program to meet the needs of students with escape-
maintained behavior problems or identify another interven-
tion that addresses these student issues. It is worth noting
that the positive findings related to adapting the CICO pro-
gram for students with escape-maintained behavior prob-
lems requires more research to fully validate these
adaptations though the early returns are encouraging.
The results of the present review revealed that none of
the group-based research contributed to the evidence base.
These results are certainly concerning given that secondary
tier interventions are meant to be implemented across stu-
dents rather than represent an intensive, individualized
intervention (Hawken et al., 2007). As discussed previ-
ously, however, there might have been sampling issues that
attenuated the potential effects of the CICO program.
Specifically, the samples within the group-based research
were generally not screened for their behavioral function,
which has been identified as an important moderating vari-
able in the single-case research. Potential reasons for the
lack of findings of the group-based research were described
earlier with particular attention paid to the need to consider
student function as a possible moderator of intervention
success. In addition, the lack of intervention effects in the
group-based research might relate to the reduced levels of
control that researchers have when using these types of
designs as compared with single-case research methods
(Gast, 2010). Research has shown that as the control over
the variables diminishes, so does the strength of the inter-
vention (Hulleman & Cordray, 2009). As such, school per-
sonnel might consider factors to increase their control over
the delivery of the intervention. For instance, the use of a
brief functional assessment prior to enrolling students into
CICO would increase control over which students would
receive the intervention. Other examples of factors under
the control of school personnel include the delivery of
Maggin et al.
teacher training, coaching, and feedback to ensure the inter-
vention is being delivered as intended or the development
of specific guidelines for the administration of the program.
The development of these structures would be designed to
increase the control of school personnel over the interven-
tion. As it currently stands, the research on CICO is promis-
ing, but additional considerations are needed to increase its
effectiveness across students.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect
to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research,
authorship, and/or publication of this article.
References
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false dis-
covery rate: A practical and powerful approach to multiple
testing. Journal of the Royal Statistical Society, 57, 289–300.
Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-
case research. In T. R. Kratochwill & J. R. Levin (Eds.),
Single-case research design and analysis: New directions
for psychology and education (pp. 187–212). Hillsdale, NJ:
Lawrence Erlbaum.
Campbell, A., & Anderson, C. M. (2008). Enhancing effects of
check-in/check-out with function based support. Behavioral
Disorders, 33, 233–245.
Campbell, A., & Anderson, C. M. (2011). Check-in/check-out:
A systematic evaluation and component analysis. Journal of
Applied Behavior Analysis, 44, 315–326.
Cheney, D. A., Stage, S. A., Hawken, L. S., Lynass, L., Mielenz,
C., & Waugh, M. (2009). A 2-year outcome study of the
check, connect, and expect intervention for students at risk
for severe behavior problems. Journal of Emotional and
Behavioral Disorders, 17, 226–243.
Crone, D. A., Hawken, L. S., & Horner, R. H. (2010). Responding
to problem behavior in schools: The Behavior Education
Program (2nd ed.). New York, NY: Guilford Press.
Ervin, R. A., Schaughency, E., Matthews, A., Goodman, S. D.,
& McGlinchey, M. T. (2007). Primary and secondary pre-
vention of behavior difficulties: Developing a data-informed
problem-solving model to guide decision making at a school-
wide level. Psychology in the Schools, 44, 7–18.
Fairbanks, S., Sugai, G., Guardino, D., & Lathrop, M. (2007).
Response to intervention: Examining classroom behavior
support in second grade. Exceptional Children, 73, 288–310.
Filter, K. J., McKenna, M. K., Benedict, E. A., Horner, R. H.,
Todd, A., & Watson, J. (2007). Check in/check out: A post
hoc evaluation of an efficient, secondary-level targeted inter-
vention for reducing problem behaviors in schools. Education
and Treatment of Children, 30, 69–84.
Fisher, W. W., & Bouxsein, K. J. (2011). Developing function-
based reinforcement procedures for problem behavior. In W.
W. Fisher, C. C. Piazza, & H. S. Roane (Eds.), Handbook
207
of applied behavior analysis (pp. 335–355). New York, NY:
Guilford Press.
Gast, D. L. (2010). Scientific research in educational and clinical set-
tings. In D. L. Gast (Ed.), Single subject research methodology
in behavioral science (pp. 20-31). New York, NY: Routledge.
Hawken, L. S., Adolphson, S. L., Macleod, K. S., & Schumann,
J. (2009). Secondary-tier interventions and supports. In W.
Sailor, G. Dunlap, G. Sugai, & R. Horner (Eds.), Handbook
of positive behavior support (pp. 395–420). New York, NY:
Springer.
Hawken, L. S., & Horner, R. H. (2003). Evaluation of a targeted
intervention within a schoolwide system of behavior support.
Journal of Behavioral Education, 12, 225–240.
Hawken, L. S., MacLeod, S. K., & Rawlings, L. (2007). Effects of
the Behavior Education Program (BEP) on office discipline
referrals of elementary school students. Journal of Positive
Behavior Interventions, 9, 94-101.
Hedges, L. V. (2007). Correcting a significance test for clustering.
Journal of Educational and Behavioral Statistics, 32, 151–179.
Hulleman, C. S., & Cordray, D. S. (2009). Moving from the
lab to the field: The role of fidelity and achieved relative
intervention strength. Journal of Research on Educational
Effectiveness, 2, 88–110.
Individuals With Disabilities Education Act (IDEA), 20 U.S.C. §
1400 (2004).
Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R.,
Odom, S. L., Rindskopf, D., & Shadish, W. R. M. (2010).
Single case designs technical documentation. Retrieved from
http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf
Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R.,
Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2013).
Single-case intervention research standards. Remedial and
Special Education, 34, 26-28.
Maggin, D. M., Briesch, A. M., Chafouleas, S. M., Ferguson, T.
D., & Clark, C. (2014). A comparison of rubrics for identify-
ing empirically supported practices with single-case research.
Journal of Behavioral Education, 23, 287–311. doi:10.1007/
s10864-013-9187-z
March, R. E., Horner, R. H., Lewis-Palmer, T., Brown, D., Crone,
D., Todd, A. W., & Carr, E. (2000). Functional assess-
ment checklist for teachers and staff (FACTS). Eugene, OR:
Educational and Community Supports.
McGaw, B., & Glass, G. V. (1980). Choice of the metric for
effect size in meta-analysis. American Educational Research
Journal, 17, 325-337.
McIntosh, K., Campbell, A. L., Carter, D. R., & Dickey, C. R.
(2009). Differential effects of a tier two behavior intervention
based on function of problem behavior. Journal of Positive
Behavior Interventions, 11, 82–93.
Mitchell, B. S., Stormont, M., & Gage, N. A. (2011). Tier two
interventions implemented within the context of a tiered pre-
vention framework. Behavioral Disorders, 36, 241–261.
Mitchell, M. (2007). Engauge digitizer: A free open-source soft-
ware to extract data points from a graphed image. Available
from http://digitizer.sourceforge.net/
Mong, M. D., Johnson, K. N., & Mong, K. W. (2011). Effects
of check-in/check out on behavioral indices and mathematics
generalization. Behavioral Disorders, 36, 225–240.
208
No Child Left Behind (NCLB) Act of 2001, Pub. L. No. 107-110,
§ 115, Stat. 1425 (2002).
Parker, R. I., & Vannest, K. (2009). An improved effect size
for single-case research: Nonoverlap of all pairs. Behavior
Therapy, 40, 357–367.
Scott, T. M., Alter, P. J., Rosenberg, M., & Borgmeier, C. (2010).
Decision-making in secondary and tertiary interventions of
school-wide systems of positive behavior support. Education
and Treatment of Children, 33, 513–535.
Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The
state of the science in the meta-analysis of single-case experi-
mental designs. Evidence-Based Communication Assessment
and Intervention, 2, 188–196.
Simonsen, B., Myers, D., & Briere, D. E. (2011). Comparing a
behavioral check-in/check-out (CICO) intervention to stan-
dard practice in an urban middle school setting using an
experimental group design. Journal of Positive Behavior
Interventions, 13, 31–48.
Sugai, G., & Horner, R. H. (2009). Responsiveness-to-intervention
and school-wide positive behavior supports: Integration of
multi-tiered system approaches. Exceptionality, 17, 223–237.
Swain-Bradway, J. L. (2009). An analysis of a secondary level inter-
vention for high school students at risk of school failure: The
Journal of Positive Behavior Interventions 17(4)
high school Behavior Education Program (Doctoral disserta-
tion, 10262). Retrieved from University of Oregon Libraries,
Eugene.
Swoszowski, N. C., Jolivette, K., Fredrick, L. D., & Heflin, L. J.
(2012). Check in/check out: Effects on students with emotional
and behavioral disorders with attention-or escape-maintained
behavior in a residential facility. Exceptionality, 20, 163–178.
Todd, A. W., Campbell, A. L., Meyer, G. G., & Horner, R. H.
(2008). The effects of a targeted intervention to reduce problem
behaviors elementary school implementation of check in–check
out. Journal of Positive Behavior Interventions, 10, 46–55.
Turtura, J. E., Anderson, C. M., & Boyd, R. J. (2014). Addressing
task avoidance in middle school students: Academic behav-
ior check-in/check-out. Journal of Positive Behavior
Interventions, 16, 159-167.
Van den Noortgate, W., & Onghena, P. (2008). A multilevel
meta-analysis of single-subject experimental design studies.
Evidence-Based Communication Assessment and Intervention,
2, 142–151.
What Works Clearinghouse. (2013, April 25). Procedures and stan-
dards handbook (Version 3.0). Washington, DC. Retrieved
from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_
procedures_v3_0_draft_standards_handbook.pdf