International Journal of Educational Research 93 (2019) 168–176

Contents lists available at ScienceDirect

International Journal of Educational Research

journal homepage: www.elsevier.com/locate/ijedures

Cultivating healing by implementing restorative practices for

T
youth: Protocol for a cluster randomized trial
⁎
Amy E. Greena, , Cathleen E. Willgingb, Kim Zamarinb, Layla M. Dehaimanc,
Patricio Ruilobad
a
Department of Psychiatry, University of California, San Diego, 9500 Gilman Drive MC 0812, La Jolla, CA 92093, USA
b
Pacific Institute for Research and Evaluation, 11720 Beltsville Drive Suite 900, Calverton, MD 20705 USA
c
School of Education, Syracuse University, 230 Huntington Hall, Syracuse, NY 13244 USA
d
3917 Camino Alameda Southwest, Albuquerque, NM, 87105, USA

A R T IC LE I N F O ABS TRA CT

Keywords: Restorative practices, non-punitive approaches to discipline focused on developing relationships

Restorative practices and healing harm, have emerged as an alternative to exclusionary disciplinary actions, such as
Implementation suspensions and expulsions. Despite widespread interest in restorative practices, there is little
School climate empirical research regarding their effectiveness. This cluster randomized controlled trial of 12
Suspension
culturally-diverse U.S. middle schools will examine whether using restorative practices with a
Expulsion
multi-faceted implementation strategy (a) reduces negative outcomes, including expulsions,
Mixed methods
suspensions, truancy, and bullying and (b) improves positive outcomes related to grade-point-
average, sense of safety, and teacher support. Secondary goals are to identify factors associated
with implementation success and conduct a cost-benefit analysis examining return on investment
from societal and government perspectives. A mixed-methods research design is employed to
address study aims.

1. Background

1.1. Significance

School disciplinary policies and procedures have a profound influence on students’ ability to remain in school (Noltemeyer, Ward,
& Mcloughlin, 2015). In recent decades, “zero-tolerance” policies have prevailed in schools across the United States (U.S.) (Daly et al.,
2016). These policies emphasize strict, rigid responses to student infractions, including removal of students from school and referrals
to juvenile justice systems (American Psychological Association Zero Tolerance Task Force, 2008), that intensify risks for dropping
out (Lamont et al., 2013). Such responses fuel the “school-to-prison pipeline” by imperiling students’ academic performance, jeo-
pardizing their future graduation, and potentially leading to detention or incarceration (Wald & Losen, 2003). Poor, racial/ethnic
minority, and special education students nationwide are excessively impacted by these policies (Annamma, Connor, & Ferri, 2013;
Annamma, Morrison, & Jackson, 2014; Gregory, Skiba, & Noguera, 2010). Despite widespread use of zero-tolerance policies, their
effectiveness is questionable and extant data point to negative impacts on school safety and adolescent wellbeing (American
Psychological Association Zero Tolerance Task Force, 2008). It is therefore imperative that other potential strategies toward

⁎
Corresponding author.
E-mail addresses: a4green@ucsd.edu (A.E. Green), cwillging@pire.org (C.E. Willging), kzamarin@pire.org (K. Zamarin),
lmdehaim@syr.edu (L.M. Dehaiman), patruiloba@hotmail.com (P. Ruiloba).

https://doi.org/10.1016/j.ijer.2018.11.005
Received 11 June 2018; Received in revised form 27 September 2018; Accepted 7 November 2018
Available online 20 November 2018
0883-0355/ © 2018 Elsevier Ltd. All rights reserved.
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

discipline be systemically evaluated.

Effective disciplinary policies are important because a school climate of insecurity and school violence (e.g., bullying, slapping,
hitting, robbery, and assault) can jeopardize learning environments and student success. Inadequate response to school violence can
cause students who cause harm to miss school due to punitive zero-tolerance policies and students who are victims of harm to miss
school out of fear for their safety. In contrast to zero-tolerance policies, responses to school rule violations based on restorative
practice approaches can facilitate learning and healing opportunities for the persons involved, foster a climate of belonging and
security for all, and keep students in the classroom to enhance educational progress (Stinchcomb, Bazemore, & Riestenberg, 2006).
Originating in criminal justice settings, restorative practice approaches to conflict resolution conceptualize infractions as violations of
people and relationships rather than just rules. The response involves eliciting understanding of the perspectives of the persons
involved, addressing an offense’s effect, and remedying the damage done to others (Drewery, 2004). Restorative practices in school
settings differ from traditional restorative justice approaches in the criminal justice system as the majority of the whole-school
community participates in supporting a wide set of strategies to proactively foster community, facilitate leadership, and build social
capital, i.e., networks of relationships based on trust, understanding, and shared values/behaviors (Wachtel, 2013). Crucially, re-
storative practices require that all parties be present and involved in relationship building and restitution, rather than removing or
providing more restrictive placements for students who may have caused harm or alienating victims from school.
Despite current interest in restorative practices in schools across the U.S. and internationally, there has been little rigorous
empirical research regarding their effectiveness in reducing negative outcomes (e.g., expulsions, suspensions, referrals to juvenile
justice) and developing safe and positive school climates. To date, there are no published studies supporting the effectiveness of
restorative practices in schools using a randomized controlled trial (RCT) design. Given the greater rates of disciplinary action and
associated negative outcomes for poor, racial/ethnic minority, and special education students, there is a need to develop methods to
adapt and implement restorative practices in ways that are contextually and culturally relevant and likely to result in better outcomes
for these groups. Examining their effectiveness in schools with high poverty, low graduation rates, and cultural diversity is crucial for
research, policy, and practice.
Scholars call for research to understand factors associated with using restorative practices with fidelity (Hurley, Guckenburg,
Persson, Fronius, & Petrosino, 2015). Implementing and sustaining new interventions in schools with fidelity involves far more than
making practice models available to staff via manuals (Lyon, Frazier, Mehta, Atkins, & Weisbach, 2011; Rogers, 2003). Common
professional education methods (e.g., workshops, seminars) are also insufficient (Owens et al., 2014). Another challenge is the
tension between adaptation and fidelity across schools, i.e., flexibility adhering to model standards (Kendall & Beidas, 2007). This
tension highlights the need for mechanisms to support use of interventions with rigor while allowing for adaptations that do not affect
core elements, such as delaying components or augmenting certain elements (e.g., adding materials). Implementation with fidelity
may require adaptations to school policies, processes, and structures. Therefore, there is a need to employ effective strategies that
consider local context in implementing restorative practices (Beets et al., 2008).

1.2. Intervention

This project will draw from implementation science and use the Dynamic Adaptation Process (DAP), a multifaceted im-
plementation strategy, to tailor a two-tiered restorative practice intervention to local school contexts (Aarons, Green et al., 2012).
Both restorative practice tiers emphasize use of affective language to foster emotional connection and reduce adversarial feelings
between students and authority figures. The first tier is used daily in classrooms and other school settings to foster relationships and
prevent conflicts. First-tier practices include Connection Circles, in which teachers lead ordered classroom conversations to enhance
relationships, rapport, and empathy. Circles may take place sequentially to reduce back-and-forth arguing and afford all a chance to
speak in an ordered fashion. Another first-tier practice is the Restorative Conversation, in which a student and a teacher/staff member
have a short, informal one-on-one meeting and make a verbal agreement about how to improve an aspect of their relationship.
The second-tier practices center on more formalized situations in which aggrieved individuals (e.g., student, teacher) addresses
the harm (e.g., talking back, bullying, name calling) directly with those who caused the harm. Such situations may include a Problem-
Solving Circle in which participants engage in an ordered circle to discuss the incident, harms caused, and reparative actions.
Restorative Agreement Meetings are another second-tier practice in which both parties make written agreements about how to improve
their relationships and achieve desired outcomes. Alternatively, Restorative Mediation includes a meeting with both parties and a
trained facilitator (e.g., a restorative practice coordinator) to help those who caused harm to understand the harms caused, determine
actions to redress them, and make written agreements to repair the harms to avoid further disciplinary action. Finally, Community
Group Conferences may involve those who caused the harm and those who were harmed, support advocates (e.g., parents), other
community members, and a trained facilitator. All parties sign an agreement detailing the actions needed for those who caused harm
to avoid further disciplinary action. For each practice, those who caused the harm can both hear about and cognize the harms of their
actions. Participants thus together explore the root of problem behaviors (i.e., social/emotional capacity to solve problems) and work
to repair the harm, while maintaining or restoring community, rather than withdrawing students from school.
School administrators, teachers, and staff (e.g. resource officers, counselors, social workers, and paraprofessionals) at all inter-
vention schools will receive training in each tier during professional development days. A full-time district-level program manager
will oversee restorative practice activities across intervention schools. Each intervention school will be assigned a specially trained
school “restorative practice coordinator.” Following their participation in a series of restorative practice trainings, the program
manager and coordinators will work closely with a local expert, who will provide ongoing in-vivo training and coaching in utilizing
restorative practices within schools and in conducting fidelity monitoring assessments of staff.

169
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

1.3. Implementation strategy

The intervention schools will use the DAP to tailor and implement the restorative practice intervention for culturally-diverse
populations in middle schools. The DAP is an iterative data-informed approach to support implementation of new interventions in
organizational- and system-level settings (Aarons, Green et al., 2012). It has four key components: initial assessment; stakeholder
engagement and training; problem solving; and outcomes feedback (Hammond, Linton, Smink, & Drew, 2007). The DAP is based on
the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework that segments implementation into four phases:
Exploration (e.g., considering new approaches to carry out restorative practices); Preparation (e.g., planning to apply practices);
Implementation (e.g., training in practices, coaching, and use of practices); and Sustainment (e.g., maintaining practices with fidelity
over time (Aarons, Hurlburt, & Horwitz, 2011). In this study, the EPIS emphasizes four levels of influence: district, school, staff, and
student. It attends to contextual factors at each level and in the larger implementation environment (e.g., policies, funding, re-
sources), and guides DAP activities by phase. The DAP also entails development and use of implementation resource teams (IRTs) at
each intervention school. An IRT is a group of 4–6 school stakeholders who work together to build implementation capacity to ensure
contextual and cultural relevance of restorative practices and to overcome barriers impeding effective utilization of these practices at
district, school, staff, and student levels. The restorative practice coordinators will recruit IRT members at each middle school and
work to keep the IRTs motivated, cohesive, and focused on goals. Preferred criteria for IRT membership include knowledge of the
school/community, time to participate, and credibility. School counselors, social workers, resource officers, teachers, (assistant)
principals or deans, and paraprofessionals will likely serve as IRT members, as well as other ancillary staff, parents, and community
members.
The restorative practices program manager and coordinators will receive a day-long training in DAP procedures, the EPIS, and
implementation and adaptation processes. The IRTs will receive a briefer training to ensure familiarity with relevant procedures/
model/processes as well, with follow-up training as needed. Additionally, the coordinators will receive training and coaching in the
use of restorative practices during the Preparation phase so that they can assist in increasing school readiness for full implementation.
During the Preparation Phase, the IRTs review data feedback reports from a multilevel school assessment analyzing (a) the extent
to which their schools possess the requisite conditions for implementing restorative practices, (b) adaptations needed in the school
context and its workforce and student body to ensure uptake, and (c) how to accomplish such adaptations. For example, school
policies that may inhibit undocumented students and families from full participation in school programs may benefit from mod-
ification to promote a sense of school belonging. Similarly, policies calling for involvement of juvenile justice or law enforcement
bodies may disincentivize participation in restorative practices. In addition, the IRTs may also consider development or adoption of
parental/family engagement strategies to form productive alliances between persons comprising the primary social support networks
of students and school personnel. Because restorative practices necessitate effective cross-cultural communication, it is important for
the IRTs to examine and enhance the extent to which their schools can respond to students and families with varying linguistic and
literacy needs. Other key concerns include culturally-based privacy concerns and taboos within Native American communities, i.e., a
coordinator engaged in restorative practices might not want to speak directly about the death of a loved one, and nontraditional
family situations requiring participation of grandparents or other respected elders in the reparative process. After identifying such
issues, the IRT would then recommend actions for the school to improve its performance in these areas.
The coordinators will guide the IRTs through a process for action planning (Centers for Disease Control & Prevention, 2012) that
a) prioritizes the implementation of restorative practice elements, b) outlines the steps needed to accomplish each goal, c) inside a
time frame, and d) determines the resources needed to accomplish each step. Actions might include raising awareness of existing
policies among stakeholder groups, training staff and students to respond to specific needs, or a combination thereof. The IRT would
outline the steps to follow through with actions. If a potential action is deemed “infeasible” at that time, the coordinator will
encourage the IRT to consider more “workable,” short-term strategies to create conditions sufficient for implementation.
Training in restorative practices with adaptation support begins and continues throughout the Implementation Phase, when IRTs
enact their action plans. In the Sustainment Phase, the IRTs examine successes, challenges, satisfaction, and fidelity and work to scale
up restorative practices found to be effective.

1.4. Research questions

The study’s primary aim is to employ a stratified cluster RCT to rigorously evaluate the effectiveness of restorative practices
implemented using the DAP in middle schools. Secondary aims are to assess factors leading to the long-term success and viability of
such initiatives and their costs and benefits. The following research questions address the study aims:

1 Do middle schools that use the DAP to implement restorative practices report increased student safety, grade point average (GPA),
teacher support, and reduced expulsions, suspensions, tardiness, and bullying compared to control schools?
2 What factors impact implementation progress and sustainment of restorative practices?
3 What are the costs and benefits of implementing restorative practices using the DAP?

170
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

2. Research plan

2.1. Study design

Twelve middle schools in a large school district in the Southwestern U.S. will be randomly assigned into intervention condition
(n = 6) or a matched control condition (n = 6) using the stratified cluster RCT design. The study’s research questions will be ex-
amined using a mixed-method research design.

2.2. Study partners

The study is driven by a Community-Academic Partnership (CAP) between the school district and academic researchers with
expertise in implementation science, school climate, cultural adaptations, and mixed methods [Pacific Institute for Research and
Evaluation (PIRE), University of California, San Diego (UCSD)]. The study advisory board (consisting of stakeholders representing the
school district, the city police and county sheriff’s offices, the county youth services center, the youth judicial system, non-profit
youth advocacy organizations, and the state child welfare agency) will provide firsthand insight into the school system and com-
munity context to enhance relevance, feasibility, and sustainment of intervention and research methods. Together, the CAP members
will conceptualize and carry out this work to achieve a sustainable implementation of restorative practices that can be systematically
evaluated for both outcome and implementation effectiveness.

2.3. Sample selection, criteria and randomization

Middle schools in a large school district in Southwest U.S. are eligible to participate (n = 27 total schools, range in size 340-1221
6-8th grade students, M = 665 students, SD = 257). Schools within the district will be approached by both district staff members and
study investigators to provide additional information about the study and request participation. We will also provide a presentation
about the study at a district meeting of all middle school principals. We will prepare a invitation letter to participate in the study,
presentations and handouts for schools that describe the study, including what will be requested of schools interested in participating.
Interested schools will submit a formal application to take part in the study. We will randomly select 12 schools from a list of
interested middle schools and use Mahalanobis distance metric matching to create pairs of similar schools (Rubin, 1980). The school
district will provide demographic and discipline data for the 12 schools for use in the randomization process. We will calculate
Mahalanobis distance from school level characteristics such as percent eligibility for free or reduced-price lunch, truancy rate, and
percentage of students in each race/ethnicity subgroup (e.g., African American, Hispanic, Native American, and White). We will sort
schools sequentially based on this score (i.e., the population centroid) and then within each pair, we will randomize schools to
intervention or control condition. We will monitor student, teacher, staff and administrator transfers between intervention and
control schools to assist with interpreting our results in the context of possible diffusion of treatments. Data for the RCT will be
analyzed using intent-to-treat analysis, with students included in the condition to which their school was initially assigned regardless
of transfers to other settings, as transfers are part of the context in which the intervention will need to work to expand districtwide. To
examine the “ideal exposure” scenario, we will also conduct a sensitivity analysis including only students who did not transfer schools
within the data collection timeframe.

2.4. Data collection and analysis

2.4.1. Research question 1: “do middle schools that use the DAP to implement restorative practices report increased student safety, grade point
average (GPA), and teacher support and reduced expulsions, suspensions, tardiness and bullying compared to control schools?”
At the student level, we will use school administrative data on demographic characteristics (race/ethnicity, free/reduced lunch
status, students in special education and classified as having a serious emotional disturbance [SED]), student truancy, GPA, rates of
in-school suspensions, out-of-school suspension, alternative educational settings, and expulsions. We will also draw on data from the
Youth Risk Behavior Surveillance System (YRBS), an anonymous statewide population-based survey that the state administers to
middle and high school students. The 2017 YRBS will offer baseline data for all schools and the 2018, 2019 and 2020 YRBS will yield
follow-up data during the Implementation Phase from the 12 schools. We will sample 100% of students at each middle school for an
average of 630 students per school for an estimated sample size of 7600 per YRBS administration. Primary outcome measures assess
bullying (“During the past 12 months, have you ever been bullied on school property?”), school safety (“During the past 30 days, on
how many days did you not go to school because you felt you would be unsafe at school or on your way to or from school?”), and
teacher support (“At my school, there is a teacher or some other adult who listens to me when I have something to say.”).
We will examine the following hypotheses with administrative data: intervention schools will have a greater reduction (1) in-
school suspensions; (2) out-of-school suspensions; (3) expulsions; (4) truancy reports, and (5) greater increases in GPA than control
schools. We will investigate the following hypotheses with YRBS data: (1) intervention schools will have a (1) greater reduction in
bullying; (2) a greater improvement in school safety than control schools; and (3) a greater improvement in teacher support than
control schools. Each analysis will examine effects for the entire population and whether there is effect modification for subgroups,
including racial/ethnic minority students, students eligible for free or reduced lunch, and students in SED programs.
The proposed effects, i.e., differential change because of the intervention, intervention effect modification for racial/ethnic
minority, free or reduced lunch, and students with SED, are assumed be small-to-medium (d = .35) in magnitude (Merrell, Gueldner,

171
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

Ross, & Isava, 2008; Wilson, Lipsey, & Derzon, 2003). The Optimal Design package was used for calculations (Raudenbush & Liu,
2000). We made the simplifying assumption that there is baseline equivalence due to random assignment, allowing us to only
consider a comparison of conditions at one follow-up observation for this power analysis. For the YRBS and school/district ad-
ministrative data analyses, given 700 students nested in 12 schools and assuming a conventional risk level of concluding a difference
when one does not exist (p = .05, two-tailed), a low-moderate level of variability among schools (ρ=.02) (Murray, 1998), and a
small-to-medium effect (d = .35), the analysis would be able to find the proposed effects 96% of the time. If it is instead assumed that
the proposed effects can be found 80% of the time, i.e., conventional 1-B = 80%, then effects as small as d = .26 could be detected.
Upon selection and randomization of the 12 schools, we will examine baseline equivalence between the intervention and control
schools and conduct an updated power analysis using the actual school sizes and level of variability.
Within this study, there is likely to be variability among our units of assignment to condition, i.e., schools, which necessitates
using models that conservatively account for these additional sources of variability (Raudenbush & Bryk, 2002). Students are nested
within school by time-period groups (e.g., 2017 observations of one school), and these repeated observations of a school are each
nested within one school. Hence, all quantitative models examining our hypotheses will use multilevel modeling (i.e. hierarchical
linear modeling) to conservatively adjust estimates for variability arising in these outcomes due to student (level 1) nesting within
school by time groups (level 2) and these repeated observations at the school level being nested within schools (level 3). Thus, all
models will be run as random intercept regressions. Sampling weight and design information will be used to appropriately weight
cases in our analyses using the multilevel pseudo maximum likelihood methods described by Asparouhov (2006) and implemented in
MPlus©. These methods confer the benefits of (a) providing more robust estimates than more conventional probability-weighted
iterative generalized least squares methods and weighted least square methods and (b) allowing for data imputation assuming data
are found to be missing at random. Our models for continuous variables (e.g. GPA, school safety, teacher support) will be run
assuming normally distributed, continuous outcomes where indicated. When descriptive analyses indicate zero-inflated count data,
zero-inflated Poisson models will be used (e.g. suspensions, expulsions, truancy). Binary outcomes (e.g. bullying) will be treated as
having a binomial distribution and use a logit link function. Our regression models (one for each measured outcomes in all hy-
potheses) will regress the outcome on a dummy contrast representing study condition; a dummy contrast representing each mod-
erator variable (racial/ethnic minority, eligible for free or reduced lunch, SED); two contrasts representing linear and quadratic, i.e.,
u-shaped, change; the orthogonal interactions between condition and change contrasts; the orthogonal interactions between mod-
erator variables, condition, and change contrasts; and all other component interactions not mentioned. These models represent
whether (a) there was differential change as a function of condition and (b) whether this differential change was more pronounced for
racial/ethnic minority students, students eligible for free or reduced lunch, and students in SED.

2.4.2. Research question 2: “what factors impact implementation progress and sustainment of restorative practices?”
Baseline data collection will involve a multilevel assessment of school, faculty/staff, and student data to identify school needs,
strengths, barriers, and readiness to implement restorative practices. For comparison purposes, we will administer these assessments
in both the intervention and control schools. The assessments will provide baseline data for the project and contribute to the IRTs’
Preparation Phase planning in the intervention schools. Data collection will include (1) small group readiness interviews/document
review (school), (2) web surveys (staff), and (3) school administrative data (student).
Small group readiness interviews consisting of an administrator, a teacher, and a staff member will occur all 12 schools.
Individual interviews will be conducted with the restorative practice coordinators. Open-ended interview questions will explore
whether necessary conditions exist for implementation and sustained use of the restorative practices and how best to optimize them.
Questions will center on attitudes toward, access to, and availability of school and community supports for these practices, current
disciplinary practices, implementation barriers, pragmatic concerns (e.g., staffing and resources), and organizational factors such as
leadership and supportive culture/climate (Aarons & Sommerfeld, 2012; Aarons et al., 2011; Aarons, Ehrhart, Farahnak, & Sklar,
2014; Aarons, Green et al., 2012). For comparison purposes, an updated version of the readiness interviews with the same sample
configuration will be undertaken with schools in the Sustainment Phase (Year 4) to compare changes in implementation milieus over
time. Documents regarding school disciplinary policies and procedures from stakeholders at each school will be collected at the time
interviews are conducted.
We will conduct annual focus groups with the IRTs to document their common and unique experiences identifying and advancing
adaptation supports to restorative practices and overall progress and satisfaction with the DAP. The objective is to better understand
local implementation milieus and the DAP’s utility in enabling IRTs to build local capacity, spearhead and implement adaptations,
and evaluate the practices over time. Each group will consist of all 4–6 IRT members per school. They will be asked 12–14 open-
ended questions, intended to prompt joint reflections on restorative practice implementation (e.g., contextual relevance, feasibility,
strategies to overcome barriers, and perceived outcomes on students, staff, and schools), factors affecting the IRT’s ability to adhere to
original action plans, reasons for diverging from these plans (e.g., staff resistance) and subsequent adaptations (e.g., staff education),
participant satisfaction, and insight into changes to school climate and disciplinary practices.
A 15- to 20-minute web survey will be administered at baseline, each year of implementation, and in the Sustainment Phase
among all school employees, including administrators, teachers, and staff in the 12 participating middle schools. The survey will
begin by assessing pertinent demographics including employment and activities performed in schools. Selected sub-scales from the
Organizational Social Context (OSC), a measure of the cultures/climates of organizations serving youth, will be used to examine staff
work attitudes and organizational stress (Aarons & Sawitzky, 2006; Aarons, Fettes, Flores, & Sommerfeld, 2009; Glisson, 2002). Work
attitudes will be assessed using two measures (1) Job Satisfaction (e.g., “How satisfied are you with your working conditions?” 10
items) and (2) Organizational Commitment (e.g., “For me this is the best of all possible organizations to work for,” 8 items), with

172
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

good reliability and validity. The subscales of Role Conflict (e.g., “Interests of my students are often replaced by bureaucratic
concerns such as paperwork,” 7 items), Role Overload (e.g., “The amount of work I have to do keeps me from doing a good job,” 7
items), and Emotional Exhaustion (e.g., “I feel emotionally drained from my work,” 6 items) create the OSC Organizational Stress
measure. School climate will be assessed using the Brief California School Climate Survey (BCSCS; 15 items) (You, O’Malley, &
Furlong, 2014). The BCSCS is adapted from California School Climate Survey, has good reliability and validity, and can be ad-
ministered to teachers, administrators, and other school staff. The BCSCS is comprised of a second-order latent factor, general school
climate, as well as two first-order latent variables, organizational supports, and relational supports. An adapted version of the
Evidence-Based Practices Attitudes Scale (Aarons, 2004) (15 items) will examine faculty/staff attitudes toward restorative practices. It
has four subscales (1) intuitive Appeal of restorative practices, (2) likelihood of adopting restorative practices given Requirements to
do so, (3) Openness to restorative practices, and (4) perceived Divergence of usual practice with restorative practices.
To measure baseline and sustainment levels of restorative practices, we will include questions at both time points to examine
frequency of staff participating in (1) connection circles, (2) restorative conversations, (3) problem-solving circles, (4) restorative
agreement meetings, (5) restorative mediations, and (6) community group conferences, and levels of perceived school support for
these practices. The district program manager and data analyst will collect administrative data on faculty and staff retention and
turnover to integrate into the analyses.
We will digitally record and transcribe all qualitative interviews and focus groups. Transcripts and written documents on school
disciplinary policies/procedures will be converted into analyzable text and imported into a password protected NVivo (QSR
International, 2012) database for efficient organization and analysis via iterative readings or codings (Patton, 2015). We will first
analyze responses/documents by “open coding” to discover themes, ideas, and issues and develop a coded index of topics addressed
in the data. We will then use “focused coding” to determine which topics arise often and which represent unusual or particular
concerns (Corbin & Strauss, 2008; Glaser & Strauss, 1967), and conduct a side-by-side comparison of key themes by school type
(intervention vs. control school) using matrices. In this staged approach to analysis, we will code transcripts, and create memos that
describe and link codes to each theme/issue. The research team will meet regularly to collectively review and interpret codes,
matrices, and overall findings derived from this analytic process.
To assess pre-implementation school characteristics, quantitative analysis of the web surveys will include descriptive analyses
(mean/SD), aggregated at the school level, for each domain and sub-scale assessed in the web survey. We will also perform de-
scriptive analyses of the administrative data collected from the schools and the YRBS. These data will be summarized in the eva-
luation and feedback reports that we will share with the IRTs in the form of graphs comparing each school’s scores on each sub-scale
to the average scores across the other 11 schools and national norms, outlining strengths and weaknesses in schools that may
influence implementation and uptake of the restorative practice intervention. The raw data, feedback, and recommended changes
will help ensure that the IRTs recognize barriers that could affect implementation and will provide impetus for the IRTs to enhance
deficit areas. For example, if the data reveal that a school has comparatively high levels of emotional exhaustion and turnover among
staff, an IRT can work to increase levels of supportive consultation during implementation. This strategy has been shown to decrease
emotional exhaustion and turnover (Aarons, Fettes et al., 2009; Aarons, Sommerfeld, Hecht, Silovsky, & Chaffin, 2009).
Implementation progress will be examined via annual focus groups conducted with IRTs at each school and the Stages of
Implementation Completion (SIC) measure. The SIC is an eight-stage measure that assesses progress of implementation using activity
completion dates and duration of activities (Chamberlain, Brown, & Saldana, 2011). The SIC is completed by coding data from
interviews, surveys, action plans, and other written documentation (e.g., phone consultation notes) concerning preset im-
plementation markers or milestones. These include (1) engagement; (2) consideration of feasibility; (3) readiness planning; (4) staff
trained; (5) fidelity monitoring process in place; (6) implementation begins; (7) ongoing implementation, consultation, and feedback;
and (8) competency. We will derive three scores from the SIC for each school: (1) number of stages completed; (2) stage duration; and
(3) proportion of activities completed in each stage. The number of stages completed is a simple count of progression through the
eight stages; the score is the last stage in which at least one activity is performed. Stage duration is the time between the date of
completion of the first activity in the stage and that of the last activity in the same stage. For schools that complete all eight stages, the
final completion date will be logged in stage eight. For schools that halt implementation at any point, the discontinuation date will be
logged in the furthest stage that the school enters. If data are summarized before a stage is complete but a school has not discontinued
implementation, the data will be censored as in a standard time-to-event or survival analysis (Kalbfleisch & Prentice, 2002). The SIC
scores indicate the level of implementation in schools over time, allowing their categorization into “low” versus “high” im-
plementation sites. These categories will be used to organize analyses of our qualitative data to determine characteristics shared
among intervention schools assigned particular scores and to describe common barriers/facilitators affecting implementation pro-
gress.
Implementation fidelity will be measured using an adapted version of the Tiered Fidelity Inventory-Restorative Practices tool
(Sprague & Tobin, 2017). The adapted fidelity inventory will be aligned with the restorative practice approach and implementation
strategy used in this study. A local expert will first coach coordinators on implementing restorative practices with fidelity and next on
the rating of practices at classroom and school levels. The coordinators will then conduct monthly fidelity ratings at both levels and
provide data to their IRTs. The school staff web-survey across all 12 middle schools will also examine how frequently staff participate
in the primary restorative practice techniques and the level of perceived school support for these practices. We will use these data to
systematically track and study comparative implementation of practices in both the intervention and control schools, as well as
between baseline and sustainment in intervention schools.

173
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

2.4.3. Research question 3: “what are the costs and benefits of implementing restorative practices using the DAP?”
Adhering to 2016 National Academies of Sciences Engineering and Medicine and its Committee on Use of Economic Evidence to
Inform Investments in Children Youth and Families (2016), the intervention cost-benefit analysis focuses on existing return on
investment from both societal and government perspectives. We will track the expenses of implementing restorative practices via the
Cost of Implementing New Strategies (COINS) cost-mapping protocol (Saldana, Chamberlain, Bradford, Campbell, & Landsverk,
2014). The SIC stages are a template for mapping implementation costs within stages and provide cost curves related to im-
plementation behavior. The three varieties of costs that must be measured within each stage to evaluate the total cost of im-
plementation are (1) direct cost of implementation service to the school; (2) indirect costs of school personnel time spent; (3)
ancillary costs that comprise the actual infrastructure investments required (e.g. new database system for tracking fidelity and school
climate). The direct costs will be measured as invoices from the restorative practices vendor for services/support provided to schools
in each stage of the SIC. Because training will be budgeted for staff across six schools, we will allocate costs proportional to number of
staff members trained. We will measure indirect costs using the COINS time/resource log modified for this project, generally basing it
on personnel, (i.e., teachers, secretary, and project management) and other costs pertaining to implementation, but not to the actual
use of restorative practices with students. Included in personnel costs are base pay, fringe benefits, and pro-rated hours in training.
Ancillary costs will be captured using a similar data collection tool as for the indirect costs. As with the indirect costs, the ancillary
cost tool will be tailored for this project.
Costs of implementation will be expressed as average cost per SIC stage for each school. We will use regression techniques to
estimate risk-adjusted cost functions, which can vary with such factors as the number of students or the time required to move
through each stage of the SIC. The dependent variable will be logged implementation cost for each stage for each school. Ranges of
costs will be imputed via regression models, where staged implementation costs, and 95% confidence intervals for those costs, will be
imputed by applying the Duan smearing estimate (Duan, 1983) to predict the levels of SIC stage costs from the logged cost regression.
The impact of behavior as rated on the SIC regarding proportion and duration on cost and the need for resource allocation will also be
evaluated. The costs mapped onto activities will be calculated in relation to individual activities as well as stages. Cost curves will be
estimated to inform the optimal rate and proportion of activity completion for schools to achieve success. Schools with higher initial
costs (e.g., due to a high duration) might be more likely to have low proportion scores during later stages. These variations will be
considered in relation to successful achievement of implementation milestones.

2.4.4. Triangulation
We will identify and summarize convergences and divergences in analyses of all data sources (Aarons, Fettes, Sommerfeld, &
Palinkas, 2012; Creswell & Creswell, 2017) integrating results through an inclusive process that values the input of a team of
stakeholders comprised of academic collaborators and school/district personnel, including the restorative practice program manager
and coordinators, to create a complete picture of DAP-driven restorative practice implementation in intervention schools over time.
First, we will consider each type of analysis on its own terms and how they differ/converge. Then we will merge data by: (a) linking
qualitative and quantitative databases; and (b) embedding one within the other so that each plays a supportive role for the other. We
will compare the results of each dataset to examine: (1) convergence (do results provide the same answer to the same question, e.g., Do
interview and document review data concur with YRBS data regarding impact of restorative practices on bullying and connectedness
to school?); (2) expansion (are unanticipated findings of one dataset explained by another, e.g., Can web survey data that suggest high
levels of emotional exhaustion in a school be explained by qualitative interview data?); and (3) complementarity (does embedding
results of the qualitative analysis in the quantitative dataset contextualize results, e.g., Does it explain variability represented by
confidence intervals or variance estimates in statistical analyses on “bullying on school ground” and “fear-based bullying”?). Data
source triangulation will also make it possible to examine the implications of infrastructure development on schools, e.g., Do school
climate measures improve in schools where in-house capacity exists to deliver restorative practices?

2.5. Timeline

Table 1 presents the four-year study timeline by phase of the DAP and school year with associated tasks for each phase.

3. Conclusion

This study aims to close gaps in research on use and effects of restorative practices in schools. Implementation is complex; this
study offers a nuanced understanding of how differing aspects of school climate relate to implementation success, fidelity, and
sustainment. The DAP offers a method for other interested schools to assess required staff, commitment, and resources to implement
restorative practices, creating a model for similar implementation elsewhere. This study also speaks to how restorative practices may
vary in culturally-diverse and economically-marginalized populations. Findings will indicate areas for adaptation for such popula-
tions and add to the limited research base on the effectiveness of restorative practices in schools, particularly in diverse student
populations, while also addressing the enormous economic and racial/ethnic disparities manifest in dropout rates (Payne & Welch,
2015).

Ethics

Ethics approval including consent to participate procedures were approved by the Pacific Institute for Research and Evaluation

174
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

Table 1
Study tasks and timeline.
Timeframe Task

Exploration & Preparation • Formalization of Community Academic Partnership

• Year 1: Spring, Summer, & Fall 2017 • Development of advisory board
• Hiring of district-wide program manager and coordinators
• Randomization of middle schools
• Creation of IRTs at intervention schools
• Baseline assessment of school, staff, and student levels
• Collect YRBS data
• Begin SIC/COINS measurement
Preparation & Implementation • IRTs prepare schools for implementation and develop action plans for adaptation and enhancing
• Year 2: Spring & Summer 2018 implementation context
• Restorative practices training for school coordinators/IRTs and 50 staff per intervention school
• Baseline assessment IRTs
• Ongoing coaching and consultation begins
• Continue SIC/COINS measurement
Implementation • Whole-school implementation begins
• Years 2-3: Fall 2018; Spring, Summer, & Fall • Ongoing coaching and consultation continues
2019 • Fidelity monitoring and review of fidelity reports by IRTs; action plan monitoring and adaptation
• Continue SIC/COINS measurement
• Collect YRBS data
• Begin and continue data analysis
Sustainment • Ongoing coaching and consultation continues
• Year 4: Spring, Summer, & Fall 2020 • Fidelity monitoring and review of fidelity reports by IRTs; action plan monitoring and adaptation
• Collect YRBS data
• Complete SIC/COINS measurement
• Planning for ongoing sustainment
• Building up local capacity to train and coach
• Undertake data analysis
• Dissemination of results
Institutional Review Board and by the participating school district’s research review committee. The authors have no conflicts of
interest to report.

Funding

This study is supported by U.S. Department of Justice Grant 2016-CK-BX-0008.

Acknowledgements

We would like to thank staff, students, and families of participating schools for their support of this research project. We would
also like to acknowledge Randy Compton and Catherine Childs of Restorative Solutions for sharing their expertise in restorative
practice implementation.

References

Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health
Services Research, 6(2), 61–74. https://doi.org/10.1023/B:MHSR.0000024351.12294.65.
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Sklar, M. (2014). Aligning leadership across systems and organizations to develop a strategic climate for evidence-
based practice implementation. Annual Review of Public Health, 35, 255–274. https://doi.org/10.1146/annurev-publhealth-032013-182447.
Aarons, G. A., Fettes, D. L., Flores, L. E., Jr, & Sommerfeld, D. H. (2009). Evidence-based practice implementation and staff emotional exhaustion in children’s services.
Behaviour Research and Therapy, 47(11), 954–960. https://doi.org/10.1016/j.brat.2009.07.006.
Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of evidence-based practice implementation and fidelity monitoring
on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology, 77(2), 270–280. https://doi.org/10.1037/a0013223.
Aarons, G. A., Fettes, D. L., Sommerfeld, D. H., & Palinkas, L. A. (2012). Mixed methods for implementation research: Application to evidence-based practice
implementation and staff turnover in community-based organizations providing child welfare services. Child Maltreatment, 17(1), 67–79. https://doi.org/10.1177/
1077559511426908.
Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., ... Chaffin, M. J. (2012). Dynamic adaptation process to implement an
evidence-based child maltreatment intervention. Implementation Science, 7, 32. https://doi.org/10.1186/1748-5908-7-32.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors.
Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7.
Aarons, G. A., & Sawitzky, A. C. (2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services,
3(1), 61–72. https://doi.org/10.1037/1541-1559.3.1.61.
Aarons, G. A., & Sommerfeld, D. H. (2012). Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. Journal
of the American Academy of Child and Adolescent Psychiatry, 51(4), 423–431. https://doi.org/10.1016/j.jaac.2012.01.018.
American Psychological Association Zero Tolerance Task Force (2008). Are zero tolerance policies effective in the schools?: An evidentiary review and re-
commendations. The American Psychologist, 63(9), 852–862. https://doi.org/10.1037/0003-066X.63.9.852.
Annamma, S., Morrison, D., & Jackson, D. (2014). Disproportionality fills in the gaps: Connections between achievement, discipline and special education in the

175
A.E. Green et al. International Journal of Educational Research 93 (2019) 168–176

school-to-prison pipeline. Berkeley Review of Education, 5(1), 53–87.

Annamma, S. A., Connor, D., & Ferri, B. (2013). Dis/ability critical race studies (DisCrit): Theorizing at the intersections of race and dis/ability. Race Ethnicity and
Education, 16(1), 1–31.
Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics—Theory and Methods, 35(3), 439–460.
Beets, M. W., Flay, B. R., Vuchinich, S., Acock, A. C., Li, K. K., & Allred, C. (2008). School climate and teachers’ beliefs and attitudes associated with implementation of
the positive action program: A diffusion of innovations model. Prevention Science, 9(4), 264–275. https://doi.org/10.1007/s11121-008-0100-2.
Centers for Disease Control and Prevention (2012). School health index: A self-assessment and planning guide. Middle school/high school version. Retrieved from Atlanta,
GAhttp://www.cdc.gov/healthyyouth/shi/pdf/MiddleHigh.pdf.
Chamberlain, P., Brown, C. H., & Saldana, L. (2011). Observational measure of implementation progress in community based settings: The stages of implementation
completion (SIC). Implementation Science, 6, 116. https://doi.org/10.1186/1748-5908-6-116.
Corbin, J., & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Thousand Oaks, CA: Sage Publications.
Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications.
Daly, B. P., Hildenbrand, A. K., Haney-Caron, E., Goldstein, N. E. S., Galloway, M., & Dematteo, D. (2016). Disrupting the school-to-prison pipeline: Strategies to reduce
the risk of school-based zero tolerance policies resulting in juvenile justice involvement. In K. Heibrun, D. DeMatteo, & N. E. S. Goldstein (Eds.). APA handbook of
psychology and juvenile justice (pp. 257–275). Washington, DC: American Psychological Association.
Drewery, W. (2004). Conferencing in schools: Punishment, restorative justice, and the productive importance of the process of conversation. Journal of Community &
Applied Social Psychology, 14(5), 332–344. https://doi.org/10.1002/casp.800.
Duan, N. (1983). Smearing estimate: A nonparametric retransformation method. Journal of the American Statistical Association, 78(383), 605–610. https://doi.org/10.
1080/01621459.1983.10478017.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New York, NY: Aldine de Gruyter.
Glisson, C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253.
Gregory, A., Skiba, R. J., & Noguera, P. A. (2010). The achievement gap and the discipline gap: Two sides of the same coin? Educational Researcher, 39(1), 59–68.
https://doi.org/10.3102/0013189x09357621.
Hammond, C., Linton, D., Smink, J., & Drew, S. (2007). Dropout risk factors and exemplary programs: A technical reportRetrieved from Clemson, SChttp://files.eric.ed.
gov/fulltext/ED497057.pdf.
Hurley, N., Guckenburg, S., Persson, H., Fronius, T., & Petrosino, A. (2015). What further research is needed on restorative justice in schools? Retrieved from San Francisco,
CAhttp://jprc.wested.org/wp-content/uploads/2015/07/Restorative_Justice_Future_Research.pdf.
Kalbfleisch, J. D., & Prentice, R. L. (2002). The statistical analysis of failure time data (2nd ed.). Hoboken, NJ: John Wiley & Sons, Inc.
Kendall, P. C., & Beidas, R. S. (2007). Smoothing the trail for dissemination of evidence-based practices for youth: Flexibility within fidelity. Professional Psychology:
Research and Practice, 38(1), 13. https://doi.org/10.1037/0735-7028.38.1.13.
Lamont, J. H., Devore, C. D., Allison, M., Ancona, R., Barnett, S. E., Gunther, R., ... Young, T. (2013). Out-of-school suspension and expulsion. Pediatrics, 131(3),
e1000–e1007. https://doi.org/10.1542/peds.2012-3932.
Lyon, A. R., Frazier, S. L., Mehta, T., Atkins, M. S., & Weisbach, J. (2011). Easier said than done: Intervention sustainability in an urban after-school program.
Administration and Policy in Mental Health, 38(6), 504–517. https://doi.org/10.1007/s10488-011-0339-y.
Merrell, K. W., Gueldner, B. A., Ross, S. W., & Isava, D. M. (2008). How effective are school bullying intervention programs? A meta-analysis of intervention research.
School Psychology Quarterly, 23(1), 26–42.
Murray, D. M. (1998). Design and analysis of group-randomized trials. New York, NY: Oxford University Press.
National Academies of Sciences Engineering and Medicine and its Committee on Use of Economic Evidence to Inform Investments in Children Youth and Families
(2016). Advancing the power of economic evidence to inform investments in children, youth, and families. Washington, D.C: The National Academies Press.
Noltemeyer, A. L., Ward, R. M., & Mcloughlin, C. (2015). Relationship between school suspension and student outcomes: A meta-analysis. School Psychology Review,
44(2), 224–240. https://doi.org/10.17105/spr-14-0008.1.
Owens, J. S., Lyon, A. R., Brandt, N. E., Warner, C. M., Nadeem, E., Spiel, C., ... Wagner, M. (2014). Implementation science in school mental health: Key constructs in a
developing research agenda. School Mental Health, 6(2), 99–111. https://doi.org/10.1007/s12310-013-9115-3.
Patton, M. Q. (2015). Qualitative research & evaluation methods: Integrating theory and methods (4th ed.). Thousand Oaks, CA: Sage Publications, Inc.
Payne, A. A., & Welch, K. (2015). Restorative justice in schools: The influence of race on restorative discipline. Youth & Society, 47(4), 539–564.
QSR International (2012). NVivo qualitative data analysis software (Version 10).
Raudenbush, S. W., & Bryk, A. (2002). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: Sage Publications, Inc.
Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5(2), 199–213. https://doi.org/10.
1037/1082-989X.5.2.199.
Rogers, E. (2003). Diffusion of innovations (5th ed.). New York, NY: Free Press.
Rubin, D. B. (1980). Bias reduction using mahalanobis-metric matching. Biometrics, 36(2), 293–298.
Saldana, L., Chamberlain, P., Bradford, W. D., Campbell, M., & Landsverk, J. (2014). The cost of implementing new strategies (COINS): A method for mapping
implementation resources using the stages of implementation completion. Children and Youth Services Review, 39, 177–182. https://doi.org/10.1016/j.childyouth.
2013.10.006.
Sprague, J., & Tobin, T. (2017). Tiered fidelity inventory–Restorative practices (TFI-RP): A tool for using restorative practices (RP) with positive behavioral interventions and
supports (PBIS). Retrieved fromUniversity of Oregon, Institute on Violence and Destructive Behavior.
Stinchcomb, J. B., Bazemore, G., & Riestenberg, N. (2006). Beyond zero tolerance: Restoring justice in secondary schools. Youth Violence and Juvenile Justice, 4(2),
123–147. https://doi.org/10.1177/1541204006286287.
Wachtel, T. (2013). Defining restorative. Retrieved from Bethlehem, PAhttp://www.iirp.edu/pdf/Defining-Restorative.pdf.
Wald, J., & Losen, D. J. (2003). Defining and redirecting a school-to-prison pipeline. New Directions for Youth Development, 2003(99), 9–15. https://doi.org/10.1002/
yd.51.
Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting
and Clinical Psychology, 71(1), 136–149.
You, S., O’Malley, M. D., & Furlong, M. J. (2014). Preliminary development of the brief California school climate survey: Dimensionality and measurement invariance
across teachers and administrators. School Effectiveness and School Improvement, 25(1), 153–173. https://doi.org/10.1080/09243453.2013.784199.

176