See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/273279401

A comparison between SRSS-IE and SSiS-PSG scores: Examining convergent

validity

Article  in  Assessment for Effective Intervention · January 2014

DOI: 10.1177/1534508414560346

CITATIONS READS

11 301

6 authors, including:

Kathleen L. Lane Wendy Peia Oakes

University of Kansas Arizona State University
232 PUBLICATIONS   6,261 CITATIONS    93 PUBLICATIONS   1,098 CITATIONS   

SEE PROFILE SEE PROFILE

Eric Alan Common Nelson Brunsting

University of Michigan-Flint Wake Forest University
39 PUBLICATIONS   338 CITATIONS    35 PUBLICATIONS   702 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

National PBIS TA Center View project

International Student Transition to US Colleges and Universities View project

All content following this page was uploaded by Wendy Peia Oakes on 16 October 2015.

The user has requested enhancement of the downloaded file.

560346
research-article2014
AEIXXX10.1177/1534508414560346Assessment for Effective InterventionLane et al.

Article
Assessment for Effective Intervention

A Comparison Between SRSS-IE and

2015, Vol. 40(2) 114­–126
© Hammill Institute on Disabilities 2014
Reprints and permissions:
SSiS-PSG Scores: Examining Convergent sagepub.com/journalsPermissions.nav
DOI: 10.1177/1534508414560346

Validity aei.sagepub.com

Kathleen Lynne Lane, PhD1, Wendy Peia Oakes, PhD2, Eric Alan Common1,
Kris Zorigian, PhD3, Nelson C. Brunsting3, and Christopher Schatschneider, PhD4

Abstract
We report findings of a validation study comparing two screening tools: the Student Risk Screening Scale–Internalizing and
Externalizing (SRSS-IE, an adapted version of the Student Risk Screening Scale) and the Social Skills Improvement System–
Performance Screening Guide (SSiS-PSG). Participants included 458 kindergarten through fifth-grade elementary students
from one school in a southeastern state. Correlation coefficients indicated statistically significant, inverse relations between
SRSS-IE scores (SRSS-IE 12 [total score], SRSS-E7 [subscale score which includes original seven items constituting the SRSS],
and SRSS-I5 [subscale scores of the five items to address internalizing behaviors]) and Prosocial Behavior, Motivation to
Learn, Reading Skills, and Math Skills subscale scores of the SSiS-PSG. Analysis of receiver operating characteristics (ROC)
curves contrasting students with significant difficulty versus adequate progress suggested the SRSS-IE12 is more accurate for
detecting Prosocial Behavior (area under the curve [AUC] = .972) and Motivation to Learn (AUC = .904) compared with
Math (AUC = .817) and Reading Skills (AUC = .805) as measured by the SSiS-PSG. Educational implications, limitations, and
future directions are offered.

Keywords
systematic screening, internalizing and externalizing behavior, emotional and behavioral disorders

Many schools in the United States are constructing multi-
tiered systems of support in an effort to become more effi-
cient and effective in (a) preventing the development of
learning and behavioral problems by offering primary pre-
vention (Tier 1) efforts to all students and (b) identifying
and assisting students with existing concerns, connecting
these students to relevant secondary (Tier 2) and/or tertiary
(Tier 3) supports according to student need (Lane, Menzies,
Oakes, & Kalberg, 2012). There are different models, with
some focusing mainly on academic performance as in
response-to-intervention (RTI; Fuchs & Fuchs, 2006) mod-
els, others focusing mostly on behavioral and social perfor-
mance as in positive behavior intervention and support
(PBIS; Sugai & Horner, 2009) models, and others address-
ing academic, behavioral, and social domains as in compre-
hensive, integrated, three-tiered (CI3T; Lane, Menzies,
Kalberg, & Oakes, 2012; McIntosh, Chard, Boland, &
Horner, 2006) models of prevention.
In each model, there are typically three levels of preven-
tion beginning with primary prevention efforts designed for
all students attending a given school. For example, all stu-
dents would access instruction addressing Common Core
State Standards (e.g., National Governors Association Center
for Best Practices [NGA Center] & the Council of Chief State
School Officers [CCSSO], 2010); learn schoolwide expecta-
tions for behavior, including opportunities to practice and
receive reinforcement for meeting expectations; and partici-
pate in a schoolwide violence prevention plan such as
Second Step Violence Prevention (Sprague et al., 2001).
Approximately 80% of the student body is anticipated to
respond to these global prevention efforts, with secondary
(e.g., small group and low intensity) and tertiary (e.g., indi-
vidualized and high-intensity) supports in place to assist stu-
dents for whom primary prevention efforts are insufficient in
academic, behavior, and/or social domains. In other words,
nonresponsiveness is expected. The key to these models is
1
University of Kansas, Lawrence, USA
2
Arizona State University, Mesa, USA
3
University of North Carolina at Chapel Hill, USA
4
Florida State University, Tallahassee, USA
Corresponding Author:
Kathleen Lynne Lane, Department of Special Education, University of
Kansas, 1122 West Campus Road, JRP Room 720, Lawrence, KS 66045,
USA.
Email: Kathleen.lane@ku.edu

Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015

Lane et al. 115
accurate detection of students who require assistance beyond
primary prevention efforts to neither overlook students in
need of additional support nor expend valuable resources
(e.g., personnel time and money) unnecessarily.
While schools more often employ academic screening
tools (e.g., AIMSweb® Reading–curriculum-based measure
[CBM] probes; Pearson Education, 2008) to benchmark
student performance, school-site and district-level leader-
ship teams are now exploring the use of behavior screening
tools to inform their decision making. For example, behav-
ior screening tools can be used to accurately detect students
with externalizing (e.g., aggressive, noncompliant) and
internalizing (e.g., excessively shy, anxious, depressed)
behavior patterns, with the latter group often overlooked if
school systems rely on more traditional approaches for
detecting students with behavior challenges (e.g., examin-
ing office discipline referral data; Bradshaw, Buckley, &
Ialongo, 2008; McIntosh, Frank, & Spaulding, 2010).
Behavior screening tools offer important data augmenting
academic screening systems to inform educational pro-
gramming and ensure students have equal access to second-
ary and tertiary supports. To illustrate, consider a third-grade
student who is struggling with reading comprehension skills
as measured by AIMSweb Reading MAZE CBM who also
demonstrates higher than average inattention and impulsiv-
ity levels as measured by the Strengths and Difficulties
Questionnaire (SDQ; Goodman, 2001). This student may
be offered additional assistance in the form of a Tier 2 read-
ing group to address his reading comprehension needs in
conjunction with a self-monitoring intervention to increase
the student’s level of engagement during the reading group.
In essence, information from behavior screening tools can
be used in conjunction with academic data to address stu-
dents’ multiple needs, offering them the necessary combi-
nations of Tier 2 or Tier 3 supports.
Several universal behavior screening tools are currently
available, including the Systematic Screening for Behavior
Disorders (SSBD; Walker & Severson, 1992), the Early
Screening Project (ESP; Walker, Severson, & Feil, 1995),
Student Risk Screening Scale (SRSS; Drummond, 1994),
the SDQ, the BASC™-2 Behavior and Emotional Screening
System (BASC-2 BESS; Kamphaus & Reynolds, 2007),
and the Social Skills Improvement System–Performance
Screening Guide (SSiS-PSG; Elliott & Gresham, 2007).
While it is beyond of the scope of this study to detail each
available screening tool, we note they vary widely in scope
and associated costs. For example, some tools are commer-
cially available and offer the benefit of corresponding inter-
vention materials (e.g., BASC-2 BESS and SSiS-PSG).
Other tools are free access, yet do not offer corresponding
intervention recommendations (e.g., SRSS and SDQ).
Ideally, all schools would have the resources to access
screening tools as well as (a) rating scales (e.g., BASC-2) to
provide additional information on students’ strengths and
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
areas of concern to inform intervention efforts and (b) evi-
dence-based strategies, practices, and programs required
within multi-tiered systems of support. Yet, the reality is
many schools do not have the resources necessary to acquire
and implement some of the commercially available tools.
For some schools, free-access tools are the best option
given budget reductions and restrictions. Therefore, it is
particularly essential we ensure all screening tools—includ-
ing free-access tools—be examined for psychometric rigor
so school teams may have confidence that the screening
tool they select is accurately identifying students for addi-
tional supports (i.e., Tiers 2 and 3).
As such, a series of studies have been conducted to deter-
mine the reliability and validity of the SRSS for use at the
elementary level to determine the accuracy of this free-
access tool in detecting students with behavioral challenges.
The SRSS is a seven-item, teacher completed screening tool
requiring approximately 10 to 15 min to rate a homeroom
class. Developed to detect students with antisocial behavior,
the tool includes the following items, each rated on a 4-point
Likert-type scale (never = 0, occasionally = 1, sometimes =
2, frequently = 3): (a) steal; (b) lie, cheat, sneak; (c) behavior
problem; (d) peer rejection; (e) low academic achievement;
(f) negative attitude; and (g) aggressive behavior. Total
scores are summed (range = 0–21, with higher scores indi-
cating higher risk) and used to classify students into one of
three risk categories established by Drummond (1994): low
(0–3), moderate (4–8), or high (9–21). Several studies have
established the reliability and validity of the SRSS at the
elementary level, including establishing convergent validity
between SRSS scores and (a) Child Behavior Checklist’s
Aggressive Behavior subscale score (Achenbach, 1991;
Drummond, Eddy, Reid, & Bank, 1994) as well as (b) SSBD
scores (Lane, Kalberg, Lambert, Crnobori, & Bruhn, 2010;
Lane, Little, et al., 2009). In terms of the SSBD, SRSS
scores were equally sensitive and specific in identifying stu-
dents with externalizing behaviors according to Stage 2
measures of the SSBD (improving chance estimates by
approximately 45%) and, to a lesser extent, those with inter-
nalizing behaviors (improving chance estimates by 30%;
Lane, Kalberg, et al., 2010; Lane, Little, et al., 2009).
Although not designed to detect students with internalizing
behaviors, it appears the SRSS holds promise in this area.
Recently, the SRSS was adapted to include additional
items to expand the tool’s original purpose and enhance detec-
tion of students with internalizing behaviors. Lane, Oakes,
et al. (2012) conducted an initial study with 2,460 elementary
students in California and Arizona of an adapted version of
the SRSS: the Student Risk Screening Scale–Internalizing
and Externalizing (SRSS-IE). Using a data analytic plan
grounded in Classical Test Theory, five of the originally tested
seven items reflecting internalizing behaviors were retained,
yielding the SRSS-IE12. These new items were (a) emotion-
ally flat; (b) shy, withdrawn; (c) sad, depressed; (d) anxious;
116 Assessment for Effective Intervention 40(2)

and (e) lonely. Findings of this first study also established Table 1.  Student Characteristics.
convergent validity of the SRSS-IE12 with two existing
Total (N = 458)
screening tools: the SDQ and the SSBD. Given the low base-
rates of students with high risk (1%–7% of the school popula- Variable and level % (n)
tion; Sugai et al., 1999) replication is important. Lane,
Gender
Menzies, Oakes, Lambert, et al. (2012) conducted two addi-
 Male 52.18 (239)
tional studies to explore the utility of the SRSS-IE with stu-  Female 47.82 (219)
dents in rural (N = 982) and urban (N = 1,079) districts, Ethnicity
offering additional evidence of the reliability and validity of  White 51.31 (235)
SRSS-IE12. Results of item level, internal consistency, and  Black 26.86 (13)
factor structure analyses supported retention of these same  Hispanic 15.94 (73)
five items. Furthermore, convergent validity was again estab-  Asian 1.53 (7)
lished between SRSS-IE12 scores as well as the two sub-   Native American 0.66 (3)
scales (SRSS-E7, seven original externalizing items; and   Mixed race 3.71 (17)
SRSS-I5, five retained internalizing items) with the SSBD. Grade level
Collectively, these findings suggested the SRSS-IE may  Kindergarten 17.69 (81)
be a reliable, valid tool with comparable accuracy to the psy-  First 21.18 (97)
chometric properties of the SDQ and SSBD in detecting stu-  Second 12.23 (56)
dents with externalizing and internalizing behavior patterns,  Third 16.81 (77)
offering schools another free-access tool requiring even less  Fourth 16.16 (74)
time than the SDQ and SSBD to complete and score. Our  Fifth 15.94 (73)
current objective is to compare the SRSS-IE (SRSS-IE12 full Special education services (%) 21.18 (97)
scale, SRSS-E7 original items, and SRSS-I5 retained inter-   Emotional disturbance 0.44 (2)
nalizing items) with the SSiS-PSG, a commercially avail-
able, relatively low-cost, user-friendly screening tool. The
SSiS-PSG is broader in scope than the SRSS-IE, developed (Externalizing subscale [SRSS-E7], Internalizing subscale
to detect academic and behavioral challenges, requires only [SRSS-I5], and Total scale [SRSS-IE12]) is equally accu-
20 to 30 min of teacher time per class, and offers the benefit rate in identifying students with significant difficulties in
of a family of accompanying tools (e.g., more detailed behav- (a) Prosocial Behavior, (b) Motivation to Learn, (c) Math
ior ratings scales) and intervention materials to support inter- Skills, and (d) Reading Skills as measured by the SSiS-
vention efforts. At the elementary level, teachers rate PSG. We hypothesized SRSS-IE scores would be inversely
homeroom students on four domains: Prosocial Behavior, related to Reading Skills, Math Skills, Prosocial Behavior,
Motivation to Learn, Reading Skills, and Math Skills using a and Motivation to Learn scores as measured by the SSiS-
five-level criterion-related performance scale. In brief, test– PSG, yielding moderately-to-highly negative correlation
retest stability has been established as has interrater reliabil- coefficients (Lane et al., 2014). We predicted the highest
ity (details to follow). In this study, we explore convergent convergent validity between SRSS-IE scores and Prosocial
validity of the SRSS-IE and SSiS-PSG scores, building upon Behavioral and Motivation to Learn scores as these are the
the work of Lane, Richards-Tutor, Oakes, and Connor (2014) most proximal constructs.
that established convergent validity of the original SRSS
scores and SSiS-PSG scores with a sample of 577 English
learners (ELs) attending a large suburban elementary school. Method
Their results suggested a negative relation between SRSS
Participants and Setting
and SSiS-PSG scores, meaning increased behavioral risk is
associated with decreased Prosocial Behavior (–0.53), Participants were 458 kindergarten through fifth-grade stu-
Motivation to Learn (–0.63), Math Skills (–0.50), and dents (239 [52.18%] boys) attending a diverse elementary
Reading Skills (–0.50) as rated by teachers. school in a southeastern state (see Table 1). The school
served students in pre-kindergarten through fifth grades;
however, the present study does not include information on
Purpose the preschool students (n = 32) due to the small sample size
This is a validation study comparing two screening tools: which would not allow for adequate exploration of conver-
the SRSS-IE and SSiS-PSG as applied with elementary-age gent validity of the two measures of interest. The diverse
students attending a diverse, southeastern, rural school. We student body was predominantly White (51.31%), with
examined the degree to which the SRSS-IE screening tool 57.51% of students qualifying for free and reduced-price

Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015

Lane et al. 117
lunches (U.S. Department of Education, 2011) and 21.18%
receiving special education services.
The Title 1 (schoolwide) school was located near a met-
ropolitan area in a southeastern state, classified as rural
fringe, meaning the school was located ≤ 5 miles from an
urbanized area. The school did not meet Adequate Yearly
Progress (AYP) benchmarks as specified in No Child Left
Behind (NCLB; 2001), having met 13 of the 17 perfor-
mance targets; a measure of the state’s progress toward
NCLB’s mandate for improving student achievement and
closing achievement gaps. Student attendance rates met the
state average of 95% as well as attendance target rates for
all student subgroups as part of AYP.
Participating teachers (n = 25, one of whom was a special
education teacher) rated each student in their homeroom
class on two measures: the SRSS-IE and SSiS-PSG (descrip-
tions to follow). Twenty-four (96.00%) teachers were female,
with 22 (88.00%) White. Teachers ranged in age from 23 to
63 years (M = 35.72, SD = 13.49). Teachers had 1 to 36 (M =
9.20, SD = 19.79) years of teaching experience, with 1 to 36
years (M = 7.71, SD = 9.98) at the current school. Twenty-
four (96.00%) teachers were certified in the subjects they
taught at the time of this study, with 18 (72.00%) having
bachelor’s and 7 (28.00%) having master’s degrees.
Procedures
After securing university and district approvals, the partici-
pating elementary school was selected for possible inclu-
sion based on the recommendation from the Associate
Superintendent and her leadership team (Director of
Elementary Education, Director of Secondary Education,
and Behavior Specialist). The district-leadership team rec-
ommended the school as it had a schoolwide positive
behavior interventions and supports (SW-PBIS) plan and
school-site leadership team in place. The district-leadership
team indicated the school faculty had the requisite skills and
structures in place to potentially benefit from the addition of
systematic screening tools.
The principal investigator (PI) contacted and met with
the principal to determine if she had interest in participating
in the current validation study, with principal expressing
interest and scheduling a meeting with teachers to allow the
PI to invite them to participate. All teachers attended this
meeting after school during March, approximately 28
weeks after the school year began. The PI explained the
purpose of the study, procedures, and time involved, and
obtained consent. All teachers (n = 25) elected to partici-
pate, completing a brief self-report demographic form (5
min) and two screening tools, the SRSS-IE (10–15 min) and
the SSiS-PSG (20–30 min). Student names were not written
on any of the screening tools. Instead, students in each class
were assigned a unique number from 1 to 25, which was
written on both screening tools and linked to a student
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
demographic form (details to follow). All teachers had a
master list numbered 1 to 25 to represent each student in
their classes. This master list was not shared with the PI nor
returned to the university. Teachers retained the master lists,
which were stored in a secure location by the principal until
the project staff returned to the school site approximately 30
days later to present findings of the study. Teachers received
results from the PI about their own students using the 1 to
25 student identification numbers.
Teachers rated each student on both screening measures
(SRSS-IE and SSiS-PSG) during this meeting. Although it
would have been optimal to counterbalance measures to
control for potential order effects, we did not do so to ensure
all teachers completed each of the screening tools correctly
during this first administration by allowing time to address
any questions on administration for each measure. Forms
were checked for accuracy and completion by project staff.
At no point were student names placed on any forms leav-
ing the school site. Before leaving the school, teachers who
completed screening tools and the demographic form were
entered into a drawing to receive a US$100 gift card. Each
teacher was assigned a random number for the drawing and
a random numbers table was used to determine the winning
teacher’s number. The teacher was selected and awarded the
gift card before the project team left the school site.
Student demographic information was obtained using
the same 1 to 25 numbering system described previously.
The school secretary completed a student demographic
form for each classroom, noting students’ gender, ethnicity,
grade, birth month, birth year, whether or not they were
receiving special education services, and if so, whether or
not the student’s eligibility category was emotionally dis-
turbed (ED) according to Individuals With Disabilities
Education Improvement Act (2004).
All de-identified data were entered into Excel spread-
sheets by project staff, with 100% checked for data-entry
reliability. Data entry errors (<1%) were reconciled.
Students had no missing SRSS-IE data. SSiS-PSG data
were mostly complete, with two students each missing one
item. Missing data were not imputed; instead, analyses
were conducted on available cases. Data analyses were con-
ducted for students in Grades kindergarten through fifth by
the PI using SAS software (Institute, 2004), with reliability
checks performed by the second author.
Measures
SRSS-IE. The SRSS-IE is the free-access measure cur-
rently under development. It is an adapted version of the
SRSS, with seven additional items measuring behaviors
characteristic of internalizing patterns. See Lane, Oakes,
et al. (2012) for a detailed description of how items were
selected using currently available evidence and tools.
Salient behaviors of internalizing behaviors were included
118 Assessment for Effective Intervention 40(2)
as new items, and the original items developed by Drum-
mond (1994) were retained. New items were as follows:
(a) emotionally flat; (b) shy, withdrawn; (c) sad, depressed;
(d) anxious; (e) obsessive-compulsive behavior; (f) lonely;
(g) self-inflicts pain. Teachers rated all 14 items (7 origi-
nal items and 7 new items) on the 4-point Likert-type scale
(never = 0, occasionally = 1, sometimes = 2, frequently =
3) used in the SRSS. Studies exploring initial evidence for
reliability and validity of the SRSS-IE at the elementary
level supported retention of five additional items (yielding
the SRSS-IE12), eliminating items: obsessive-compulsive
behavior and self-inflicts pain. Removal of these 2 items
was confirmed in subsequent studies in rural and urban
settings (Lane, Menzies, Oakes, Lambert, et al., 2012;
Lane, Oakes, et al., 2012). Results also offered convergent
validity of SRSS-IE12 scores with two established screen-
ing tools: the SDQ (Goodman, 1997; Lane, Oakes, et al.,
2012) and the SSBD (Lane, Menzies, Oakes, Lambert,
et al., 2012; Lane, Oakes, et al., 2012; Walker & Severson,
1992). In this study, we refer to the SRSS-IE as follows:
the initial 7 items constituting the SRSS as the SRSS-E7,
the retained set of 5 internalizing items as the SRSS-I5,
and the overall 12-item scale as SRSS-IE12. Internal
consistency estimates were as follows: SRSS-E7 = .85,
SRSS-I5 = .71, and SRSS-IE12 = .79.
SSiS-PSG. The SSiS-PSG is a systematic screening tool
developed to detect challenges in academic (i.e., Math Skills
and Reading Skills) and behavioral (i.e., Motivation to Learn
and Prosocial Behavior) domains. The SSiS-PSG is vali-
dated for use PK–12, offering three versions (preschool,
elementary, and secondary). For the elementary and second-
ary versions, teachers rate each student on four items using a
five-level criterion-related performance scale. On the ele-
mentary version, a score of 1 (red band) indicates students
who experience significant difficulty, 2 or 3 (yellow band)
signals students who may be experiencing moderate diffi-
culty, and 4 or 5 (green band) indicates adequate perfor-
mance. As reported in the technical manual, test–retest
reliability was calculated for 302 elementary students with
an average of 74 days between screening time points (Intra-
class Correlations: .68 Math Skills, .74 Reading Skills, .74
Motivation to Learn, .69 Prosocial Behavior; Gresham &
Elliott, 2008). Interrater reliability was calculated between
two teachers for 215 elementary students (Intraclass Corre-
lations: .68 Math Skills, .57 Reading Skills, .62 Motivation
to Learn, .55 Prosocial Behavior; Gresham & Elliott, 2008).
Experimental Design and Data Analytic Plan
We examined convergent validity of the SRSS-IE scores
using the SSiS-PSG scores as a criterion measure.
Specifically, we examined convergent validity using two
analytic approaches. First, we computed Pearson correlation
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
coefficients between the SRSS-E7 (externalizing; range =
0–21), SRSS-I5 (internalizing; range = 0–15), and
SRSS-IE12 (the reduced scale; range = 0–36, with higher
scores indicating higher levels of concern) and the four items
constituting the SSiS-PSG (Reading Skills, Math Skills,
Motivation to Learn, and Prosocial Behavior) for kindergar-
ten through fifth-grade students (see Table 2). Second, we
utilized receiver operating characteristic (ROC) curves to
evaluate accuracy (sensitivity and specificity) of the
SRSS-IE (SRSS-E7 externalizing, SRSS-I5 internalizing,
and SRSS-IE12 total scores) ratings compared with Reading
Skills, Math Skills, Motivation to Learn, and Prosocial
Behavior ratings on the SSiS-PSG (Petras, Chilcoat, Leaf,
Ialongo, & Kellam, 2004; see Table 3). ROC curves allow
for a comparison of two distributions: the distribution of
scores on the screening test for those students classified as
having a given condition (e.g., behavior challenge) against
the distribution of scores on the screening test for those that
are classified as not having a given condition. The farther
apart these distributions are from each other, the better the
screening test is able to distinguish those at risk for the given
condition from those not at risk for the given condition. One
useful index derived from ROC curves is called the AUC, or
area under the curve. The AUC can be interpreted as the
probability a randomly selected person from the group iden-
tified as having the given condition will have a higher score
on the screener than a randomly selected person from the
group that does not have the given condition. The AUC typi-
cally ranges from .50 to 1.0, with .5 indicating the screener
is operating at chance levels in detecting a problem, to 1.0,
which means the screener is operating perfectly
(Schatschneider, 2013). The first set of analyses compared
SSiS-PSG scores indicating any risk (scores = 1, 2, or 3)
compared with adequate progress (scores = 4 or 5). The sec-
ond set of analyses compared SSiS-PSG scores indicating
significant difficulty (score = 1) compared with adequate
progress (scores = 4 or 5), similar to procedures used to
explore extreme groups in earlier validation studies of the
SRSS and SSBD (Lane, Little, et al., 2009). ROC curves
based on logistic regression revealed the predictive accuracy
for all possible cutting scores: (a) 0 to 21 for the SRSS-E7
externalizing items, (b) 0 to 15 for the SRSS-I5 internalizing
items, and (c) 0 to 36 for the SRSS-IE12 full scale. For K–12
students, we hypothesized moderate-to-high, significant
negative correlations for each of the SSiS-PSG items as they
reference academic and academic enabling skill strengths.
Academic enabling skills are those needed to fully engage in
and benefit from instruction (DiPerna & Elliott, 2002) such
as Motivation to Learn and Prosocial Behaviors. In contrast,
the SRSS-IE proposes to measure antisocial and behavioral
skill-sets that impede learning—the opposite of Prosocial
Behavior. Correlations were interpreted using guidelines
specified in Kettler, Elliott, Davies, and Griffin (2012) as
informed by Cohen’s (1992) classic Power Primer as
Lane et al. 119

Table 2.  Convergent Validity: Correlation Coefficients - SRSS-E7, SRSS-I5, and SRSS-IE12 With the SSIS-PSG.

Correlation

  SRSS-E7 (M = 5.28; SD = 4.92) SRSS-I5 (M = 2.34; SD = 2.85) SRSS-IE12 (M = 7.62; SD = 6.10)

  r r r

  p value p value p value

SSiS-PSG Scale M SD n n n
Reading Skills 3.15 1.20 −.39 −.19 −.41
  <.0001 <.0001 <.0001
  457 457 457
Math Skills 3.33 1.07 −.39 −.25 −.43
  <.0001 <.0001 <.0001
  458 458 458
Motivation to Learn 3.45 1.17 −.61 −.26 −.62
  <.0001 <.0001 <.0001
  458 458 458
Prosocial Behavior 3.50 1.18 −.71 −.32 −.72
  <.0001 <.0001 <.0001
  458 458 458

Note. Correlations were interpreted using the following guidelines specified in Kettler, Elliott, Davies, and Griffin (2012): .00 to .10 were nonexistent,
.10 to .30 were small, .30 to .50 were medium, .50 to .70 were large, .70 to .90 were very large, and .90 to 1.00 were close to perfect (Hopkins, 2002;
Kettler et al., 2010). SRSS-E7 = the original seven items constituting the SRSS; SRSS-I5 = the five items retained to measure internalizing behaviors;
SRSS-IE12 = the original seven items from the SRSS developed by Drummond (1994) combined with the new five items constituting the SRSS-I5; SSiS-
PSG = Social Skills Improvement System–Performance Screening Guide (Elliott & Gresham, 2007); SRSS-IE = Student Risk Screening Scale–Internalizing
and Externalizing.

follows: .00 to .10 were nonexistent, .10 to .30 were small,
.30 to .50 were medium, .50 to .70 were large, .70 to .90
were very large, and .90 to 1.00 were close to perfect
(Hopkins, 2002; Kettler et al., 2010). We also expected the
SRSS-IE would be more predictive of Prosocial Behavior
and Motivation to Learn scores than Reading Skills and
Math Skills scores as the constructs measured by the
SRSS-IE were more closely related to social behavior and
motivation (Walker, Ramsey, & Gresham, 2004).
Results
Convergent Validity With Prosocial Behavior as
Measured by the SSiS-PSG
SSiS-PSG Prosocial Behavior scores were statistically sig-
nificantly (p < .0001) and negatively correlated with the
SRSS-E7 (r = −.71), SRSS-I5 (r = −.32), and SRSS-IE12
(r = −.72) scores, with externalizing and full scale correla-
tion coefficients of very large magnitude.
In comparing scores of students with any difficulty in
Prosocial Behavior with those making adequate progress
according to SSiS-PSG Prosocial Behavior scores, 204 stu-
dents were rated as having any difficulty (with scores of 1,
2, or 3) and 254 were making adequate progress (with
scores of 4 or 5). Findings of ROC curve analyses with
SRSS-E7 scores yielded an AUC of .845 in predicting
Prosocial Behavior (0.345 better than chance, which is
0.50). The AUC was higher for the full scale SRSS-IE12
(AUC of .859), and lower for the SRSS-I5 (AUC of .652).
In comparing scores of students with significant diffi-
culty in Prosocial Behavior with those making adequate
progress according to SSiS-PSG Prosocial Behavior scores,
33 students were rated as having significant difficulty (with
a score of 1) and 254 were making adequate progress (with
scores of 4 or 5). Findings of ROC curve analyses with
SRSS-E7 scores yielded an AUC of .956 in predicting
Prosocial Behavior (0.456 greater than chance). The AUC
was higher for the full scale SRSS-IE12 (AUC of .972), and
lower for the SRSS-I5 (AUC of .750).
Convergent Validity With Motivation to Learn as
Measured by the SSiS-PSG
SSiS-PSG Motivation to Learn scores were statistically sig-
nificantly (p < .0001) and negatively correlated with the
SRSS-E7 (r = −.61), SRSS-I5 (r = −.26), and SRSS-IE12
(r = −.62) scores, with the externalizing and full scale cor-
relation coefficients of large magnitude.
In comparing scores of students with any difficulty in
Motivation to Learn with those making adequate progress
according to SSiS-PSG Motivation to Learn scores, 207

Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015

120 Assessment for Effective Intervention 40(2)

Table 3.  Convergent Validity: Receiver Operating Characteristics - SRSS-E7, SRSS-I5, and SRSS-IE12 With the SSiS-PSG.

Student condition according to SSiS-PSG

With condition Without the condition

ROC
Target as measured by the SSiS-PSG n n SRSS-IE comparison AUC
Reading Skills
  Any difficulty vs. Adequate progress 264 193 SRSS-E7 .706
  SRSS-I5 .587
  SRSS-IE12 .705
  Significant difficulty vs. Adequate progress 51 193 SRSS-E7 .778
  SRSS-I5 .656
  SRSS-IE12 .805
Math Skills
  Any difficulty vs. Adequate progress 240 218 SRSS-E7 .702
  SRSS-I5 .604
  SRSS-IE12 .710
  Significant difficulty vs. Adequate progress 20 218 SRSS-E7 .760
  SRSS-I5 .767
  SRSS-IE12 .817
Motivation to Learn
  Any difficulty vs. Adequate progress 207 250 SRSS-E7 .805
  SRSS-I5 .629
  SRSS-IE12 .801
  Significant difficulty vs. Adequate progress 30 250 SRSS-E7 .902
  SRSS-I5 .630
  SRSS-IE12 .904
Prosocial Behavior
  Any difficulty vs. Adequate progress 204 254 SRSS-E7 .845
  SRSS-I5 .652
  SRSS-IE12 .859
  Significant difficulty vs. Adequate progress 33 254 SRSS-E7 .956
  SRSS-I5 .750
  SRSS-IE12 .972

Note. SRSS-E7 = the original seven items constituting the SRSS; SRSS-I5 = the five items retained to measure internalizing behaviors; SRSS-IE12 =
the original seven items from the SRSS developed by Drummond (1994) combined with the new five items constituting the SRSS-I5; SSiS-PSG = the
Social Skills Improvement System–Performance Screening Guide (Elliott & Gresham, 2007); SRSS-IE = Student Risk Screening Scale–Internalizing and
Externalizing; ROC = receiver operating characteristics; AUC = area under the curve; Any Difficulty = SSiS scores of 1, 2, or 3; Significant Difficulty =
SSiS-PSG Score of 1 and Adequate Progress to scores of 4 or 5.

students were rated as having difficulty and 250 were mak-
ing adequate progress. Findings of ROC curve analyses
with SRSS-E7 scores yielded an AUC of .805 in predicting
Motivation to Learn (0.305 better than chance). The AUC
was comparable for the full scale SRSS-IE12 (AUC of
.801) and lower for the SRSS-I5 (AUC of .629).
In comparing scores students with significant difficulty
in Motivation to Learn with those making adequate prog-
ress according to SSiS-PSG Motivation to Learn scores, 30
students were rated as having significant difficulty. Findings
of ROC curve analyses with SRSS-E7 scores yielded an
AUC of .902 in predicting Motivation to Learn (0.402 bet-
ter than chance). The AUC was slightly higher for the full
scale SRSS-IE12 (AUC of .904) and lower for the SRSS-I5
(AUC of .630).
Convergent Validity With Math Skills as
Measured by the SSiS-PSG
SSiS-PSG Math Skills scores were statistically significantly
(p < .0001) and negatively correlated with the SRSS-E7
(r = −.39), SRSS-I5 (r = −.25), and SRSS-IE12 (r = −.43)
scores, with the full scale correlation coefficients of medium
magnitude.
In comparing scores of students with any difficulty in
Math Skills with those making adequate progress accord-
ing to SSiS-PSG Math Skills scores, 240 students were
rated as having difficulty and 218 were making adequate
progress. Findings of ROC curve analyses with SRSS-E7
scores yielded an AUC of .702 in predicting math (0.202
better than chance). The AUC was slightly higher for the

Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015

Lane et al. 121
full scale SRSS-IE12 (AUC of .710) and lower for the
SRSS-I5 (AUC of .604).
In comparing scores of students with significant diffi-
culty in Math Skills with those making adequate progress
according to SSiS-PSG Math Skills scores, 20 students
were rated as having significant difficulty. Findings of ROC
curve analyses with SRSS-E7 scores yielded an AUC of
.760 in predicting math (0.26 better than chance). The AUC
was substantially higher for the full scale SRSS-IE12 (AUC
of .817) and slightly higher for the SRSS-I5 (AUC of .767).
Convergent Validity With Reading Skills as
Measured by the SSiS-PSG
SSiS-PSG Reading Skills scores were statistically signifi-
cantly (p < .0001) and negatively correlated with the
SRSS-E7 (r = −.39), SRSS-I5 (r = −.19), and SRSS-IE12
(r = −.41) scores, with the full scale correlation coefficients
of medium magnitude.
In comparing scores of students with any difficulty in
reading with those making adequate progress per SSiS-PSG
Reading Skills scores, 264 students were rated as having dif-
ficulty and 193 were making adequate progress. Results of
ROC curve analyses with SRSS-E7 scores yielded an AUC
of .706 in predicting reading (0.206 better than chance). The
AUC was comparable for the full scale SRSS-IE12 (AUC of
.705) and lower for the SRSS-I5 (AUC of .587).
In comparing scores of students with significant diffi-
culty in reading with those making adequate progress
according to SSiS-PSG Reading Skills scores, 51 students
were rated as having significant difficulty. Findings of the
ROC curve analyses with SRSS-E7 scores yielded an AUC
of .778 in predicting reading (0.278 better than chance). The
AUC was higher for the full scale SRSS-IE12 (AUC of
.805) and slightly lower for the SRSS-I5 (AUC of .656).
Summary
The SRSS-IE12, relative to the SRSS-E7 and the SRSS-I5,
had the highest correlation coefficients with each of the
SSiS-PSG scores, just slightly above SRSS-E7 correlation
coefficients. As expected, the correlation coefficients were
of the greatest magnitude with the Prosocial Behavior
scores, which is the inverse of antisocial behavior.
Collectively, these findings offer initial evidence of conver-
gent validity between the three versions of the SRSS
(SRSS-E7, SRSS-I5, and SRSS-IE12) and SSiS-PSG
Reading Skills, Math Skills, Prosocial Behavior, and
Motivation to Learn scores.
Discussion
As schools move toward prevention frameworks with tiered
supports to address the learning and behavioral needs of
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
students, accurate detection of those students who need
more than the primary plan is essential. Universal screening
procedures for behavior offer an efficient process for
school-site leadership teams to ensure all students have
equal access to secondary and tertiary interventions and
supports, if needed (Lane, Menzies, Oakes, & Kalberg,
2012). Furthermore, these data allow leadership teams to
monitor the level of student risk in a school building to
assess and improve their primary prevention program
(Lane, Kalberg, & Menzies, 2009). That is, if risk is increas-
ing, the school team may examine the level of fidelity of the
primary plan, assess the need for professional development,
reevaluate the goals and responsibilities of the plan, and
address identified areas of need with additional instruc-
tional programming, such as implementing a bullying pre-
vention program (e.g., Bully Prevention in Positive
Behavior Support; Ross & Horner, 2014). Because school-
wide screening procedures to detect students with behav-
ioral concerns at the earliest possible juncture are so
important for early intervening, all schools must have
access to high-quality, socially valid screening tools.
There are behavior screening tools available PK to
Grade 12, both commercially (e.g., BASC-2 BESS, SSiS-
PSG) and free access (e.g., SDQ, SRSS) to meet the needs
of all schools regardless of monetary resources available.
Tools have been developed to provide schools a compre-
hensive approach for detecting students with increased
behavioral needs as well as addressing those needs through
instructional programs (e.g., BASC-2 BESS, SSiS); yet,
not all schools have the resources to invest. The purpose of
this article was to compare a newly adapted, user-friendly,
free-access screening tool the SRSS-IE (Lane, Menzies,
Oakes, Lambert, et al., 2012; Lane, Oakes, et al., 2012) to
the existing user-friendly, commercially available mea-
sure, the SSiS-PSG.
The SRSS-IE scales (e.g., SRSS-E7, SRSS-I5, and
SRSS-IE12) were all significantly and negatively correlated
with the SSiS-PSG Scales (Reading Skills, Math Skills,
Motivation to Learn, Prosocial Behavior). The correlation
coefficients between the SRSS-E7 and SSiS-PSG scales
were comparable with those reported by Lane et al. (2014),
also suggesting a particularly strong relation between the
SRSS-E7 and Motivation to Learn and Prosocial Behavior
subscale scores as compared with the Reading Skills and
Math Skills subscales scores. Furthermore, the newly
adapted, combined scale (SRSS-IE12) had the strongest
correlations with the SSiS-PSG scales indicating the com-
bined scale would detect the largest proportion of students
identified at risk on the SSiS-PSG for academic and behav-
ioral concern. These findings add to previous preliminary
studies of the technical qualities of the SRSS-IE12.
Consistent with the findings of Lane, Little, et al. (2009)
in their study comparing the SRSS and SSBD, the utility of
the SRSS-IE depends—at least partially—on how the tool
122 Assessment for Effective Intervention 40(2)
is used. For example, when looking for students who strug-
gle with respect to Prosocial Behavior, it appears the
SRSS-IE is most useful when contrasting the low (adequate
progress) and high risk (significant difficulties) groups. In
this case, the AUC for the SRSS-IE12 is 97.20%, slightly
better than the original 7 items (SRSS-E7, AUC .956).
When contrasting the SRSS with other cutting scores (e.g.,
any difficulty vs. adequate progress), the predictive accu-
racy wanes (e.g., for Prosocial Behavior, SRSS-IE12 is
85.90%). ROC curve analyses provide initial evidence to
suggest the SRSS-IE12 is an effective tool for predicting
challenges associated with Prosocial Behavior and
Motivation to Learn as measured by the SSiS-PSG. We note
the five internalizing items (SRSS-I5) were less effective in
predicting these outcomes than when combined with the
SRSS-E7 items. This supports using the SRSS-IE12 total
score when making these predictions.
A final noteworthy outcome is the accuracy of the
SRSS-IE12 with respect to academic performance.
Specifically, the accuracy for the SRSS-IE12 is 80.50% for
Reading Skills and 81.70% for Math Skills, improving
chance estimates substantially. This suggests the SRSS-IE
is also an effective tool for predicting teacher ratings of aca-
demic performance in reading and math (highlighting the
link between academic and behavioral performance), but to
a lesser extent compared with predicting prosocial skills
and motivation for learning.
Implications
First, a very strong, significant, negative relation exists
between SRSS-IE12 scores and Prosocial Skills on the
SSiS-PSG indicating students with more risk on the
SRSS-IE12 may have fewer prosocial skills. In terms of
predicting academic achievement, social skills are the sec-
ond strongest predictor after motivation (DiPerna & Elliott,
2002). Malecki and Elliott (2002) found prosocial behavior
to be a significant and independent predictor of academic
achievement when examining teacher ratings of academic
competence, achievement in math and reading, and prob-
lem behavior. The negative consequences of limited social
skills cannot be overstated. Prosocial skills are essential for
building and maintaining relationships with peer and teach-
ers. Peer and teacher relationships, in turn, are related to
students’ engagement, interest, motivation, and school
attainment (Pianta, Hamre, & Allen, 2012; Wentzel &
Watkins, 2002). For example, teachers may offer more
learning opportunities to students who display strong social
skills (e.g., by regularly calling on them) and social skills
can impact academic practice and responding (e.g., raising
hand to answer a question; Greenwood, Horton, & Utley,
2002). In terms of prevention efforts, instructional environ-
ments facilitating the development of students’ prosocial
skills are characterized by organizational structures such as
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
clear expectations, predictable routines, active supervision,
and maximizing instructional time (Lane, Menzies, Bruhn,
& Crnobori, 2011; Pianta et al., 2012).
Second, this study provides information on the relation
between students who exhibit behavioral needs as detected
by the SRSS-IE12 and Motivation to Learn. A very large
magnitude, significant, negative relation between higher
scores on the SRSS-IE12 and the SSiS-PSG Motivation to
Learn scale shows that students with higher levels of behav-
ioral risk have lower levels of Motivation to Learn.
Motivation is one of the strongest predictors of academic
achievement (DiPerna & Elliott, 2002). Therefore, particu-
larly for classes with higher numbers of students with
increased risk on the SRSS-IE12, strategies to increase stu-
dent motivation to learn can be implemented by the teacher
through low-intensity strategies such as offering choices in
instructional activities, optimal levels of challenge, control
over instructional programming, and contextualization of
learning utilizing relevant contexts for problem solving and
application of knowledge (Lane et al., 2011). In terms of
Tier 2 supports, motivation can be developed through inter-
ventions that target improvements in self-determination
skills both separately and in conjunction with academic
skills (Oakes et al., 2012).
Third, a moderate, significant, negative relation was found
between SRSS-IE12 scores and SSiS-PSG scores of Math
Skills and Reading Skills indicating students with greater
behavioral risk on the SRSS-IE12 are perceived by their
teachers to have lower reading and math skill attainment.
ROC curves also suggest chance estimates are improved by
approximately 20–30% in these domains. Previous studies at
the elementary level have found SRSS scores are inversely
related to year-end reading (Oakes et al., 2010) and language
arts (Menzies & Lane, 2011) achievement. At the secondary
level, SRSS scores are predictive of course failures and grade
point averages (Lane, Bruhn, Eisner, & Kalberg, 2010; Lane,
Kalberg, Parks, & Carter, 2008; Lane et al., 2013; Lane, Parks,
Kalberg, & Carter, 2007). The relation between academic
underachievement and behavior problems is well documented
(Nelson, Benner, Lane, & Smith, 2004). The importance of
detecting and intervening with students with comorbid aca-
demic and behavioral difficulties is critical in that this group
of students is our most at-risk population (Reinke, Herman,
Petras, & Ialongo, 2008) and, often, the students deemed non-
responsive to secondary or tertiary academic efforts (Kamps
& Greenwood, 2005). These findings offer additional evi-
dence of the need for CI3T models of prevention to address
behavioral and academic needs together (Kalberg, Lane, &
Menzies, 2010) across all levels of prevention.
Future Directions and Limitations
We encourage readers to interpret these modest outcomes
with caution in light of the following limitations, the first of
Lane et al. 123
which pertains to generalization concerns. Given these data
were collected at just one school, it is imperative that addi-
tional inquiry be conducted to explore the convergent valid-
ity between the SRSS-IE and the SSiS-PSG before drawing
any definitive conclusions. Additional studies are needed in
other locales and with larger samples that would allow the
nested nature of these data to be addressed (e.g., potential
dependency issues resulting from students being nested in
teachers’ classes).
Second, as with the Lane, Little, et al. (2009) study, this
study compared only two screeners: the SRSS-IE and SSiS-
PSG based on premise the SSiS-PSG is a very user-friendly
tool, offering a wealth of information to inform primary,
secondary, and tertiary supports using the family of SSiS
tools available for screening, assessment, and intervention
efforts. We elected to compare the relative value—psycho-
metrically and in terms of feasibility—of the SRSS-IE12 as
this expanded version of the SRSS continues to allow each
student to receive a total score and contains only 12 items
for completion, with the addition of five items measuring
characteristic patterns of internalizing behaviors. We rec-
ommend additional studies be conducted to compare the
SRSS-IE with available tools such as the BASC-2 Behavior
and Emotional Screening Scale (Kamphaus & Reynolds,
2007), another commercially available screening tool. We
also encourage additional inquiry into secondary schools, to
examine the convergent validity of screening tools as
applied in middle and high school settings (e.g., Lane et al.,
2008; Lane et al., 2007). Such information would help
school-site and district-level leadership teams in making
decisions about which systematic screening tools to adopt
as part of regular school practices based on screening goals
and resource considerations.
Third, in this study, we did not explore specific cutting
scores to establish low, moderate, and high risk status for
the SRSS-IE12 and the SRSS-E5 nor did this study address
the nested nature of the data (students nested in teacher’s
classrooms) due to the small sample size (number of teach-
ers). Next steps in the development of this measure are (a)
to identify cutting scores at the elementary, middle, and
high school levels (American Educational Research
Association [AERA], American Psychological Association
[APA], & National Council for Measurement in Education
[NCME], 1999); (b) examine the predictive validity of the
SRSS-IE12 with important learning and behavioral out-
comes (e.g., office discipline referrals, attendance patterns,
referrals for mental health supports, state achievement
assessments); and (c) determine the social validity of the
screening procedures (e.g., efficiency; addressing important
goals; and resources for preparing, conducting, scoring, and
interpreting the measure). School-site and district-level
leadership teams will be able to use cutting scores to deter-
mine which students would benefit from secondary and ter-
tiary supports currently available at the school sites and
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
additional interventions and supports needed to fully
address existing student needs. Cutting scores must demon-
strate high sensitivity (the proportion of students detected
that actually do present at-risk behaviors) and specificity
(the proportion of students who fall below the at-risk cut
point who do not present at-risk behaviors; AERA, APA, &
NCME, 1999; Kraemer, 1992; Lane, Kalberg, et al., 2010).
Future studies with larger samples should examine specific
risk categories for the SRSS-IE12 and SRSS-I5 in predict-
ing important behavioral, social, and academic outcomes
for students, while addressing issues of dependency associ-
ated with the nested nature of the data (Elliott & Gresham,
2007; Lane, Oakes, et al., 2012). We currently have a multi-
state validation study nearing completion to establish cut-
ting scores, using the Child Behavior Checklist Teacher
Report form (Achenbach, 1991) to inform the development
of these scores. This is clearly a critical next step for
researchers and practitioners alike as this information will
be central for use in connecting students to Tier 2 and 3 sup-
ports to address internalizing issues. However, given that
many schools have adopted the SRSS-IE, despite the
absence of cutting scores for the SRSS-IE12 and SRSS-I5,
we feel this modest study presented here is still important
for educators in informing the decision-making processes
involved with selecting behavior screening tools.
Fourth, as we mentioned previously, this study focused
on comparing two screening tools and did not examine
actual outcomes for these students. One purpose of screen-
ing is for the early detection of students who may, without
intervention, have deleterious school and post school out-
comes (e.g., low grades, failed courses, school dropout, jus-
tice system involvement; Wagner, Kutash, Duchnowski,
Epstein, & Sumi, 2005). Therefore, it is important screening
tools accurately predict, as early as possible, which students
may experience these difficulties. Findings of the signifi-
cant relation between SRSS-IE12 scores and SSiS-PSG
Reading Skills and Math Skills Scales offer evidence as to
the utility of the SRSS-IE12 in detecting students with
comorbid risk; however, the SSiS-PSG scales are teacher
ratings of student performance and therefore must be con-
firmed with further evidence of actual academic outcomes.
Previous studies of the SRSS have examined the tool’s abil-
ity to predict students’ year-end outcomes. The SRSS-IE
must now be examined in terms of its ability to predict out-
comes for students with externalizing and internalizing
behavioral risk as in other studies of the predictive validity
of the SRSS (e.g., Menzies & Lane, 2011).
Finally, this study did not address issues of social
acceptability. For any new practice to be fully imple-
mented and sustained, teachers must have input into the
decision-making process of selecting a universal behavior
screener. Given teachers’ multiple demands, it is essential
that they have a voice in the choosing of the screening tool
for their school and in the procedures for its use (Harrison,
124 Assessment for Effective Intervention 40(2)
Vannest, & Reynolds, 2013). Therefore, assessing the
social validity of the screening tool selected and proce-
dures is important. Future research should examine social
validity of their process as well as the screening tools. As
school contexts and goals are unique, this practice should
be considered by each school site adopting universal
screening. Nonetheless, research assessing social validity
of screening would provide information to school systems
and researchers developing tools on broader areas of
teacher concern. Assessing and responding to teachers’
perspectives would provide valuable information in plan-
ning for the sustainable adoption of universal screening
for behavior as a primary prevention practice within multi-
tiered systems of support (McIntosh, Filter, Bennett, Ryan,
& Sugai, 2010).
Summary
Despite the noted limitations, findings from this very mod-
est psychometric study provide initial evidence to support
the use of the SRSS-IE as an equally reliable tool as the
SSiS-PSG when attempting to identify students with chal-
lenges in prosocial behaviors and motivation for learning.
While the SRSS improves chance estimates of detecting
which elementary-age students might have deficits in read-
ing or math performance as measured by the SSiS-PSG and
there are statistically significant relations between SRSS-IE
(SRSS-IE12, SRSS-E7, and SRSS-I5) scores and SSiS-
PSG (Reading Skills, Math Skills, Prosocial Behavior, and
Motivation to Learn) scores, the SRSS-IE does not share
the same level of predictive accuracy as the SSiS-PSG for
students’ academic performance.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect
to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support
for the research, authorship, and/or publication of this article: This
study was funded in part by an internal grant from the University
of North Carolina at Chapel Hill: Identifying and Supporting K–12
Students Within the Context of Three-Tiered Models of Prevention
to Meet Students Multiple Needs: A Collaborative Effort, Research
Triangle Schools Partnership Community/Schools Partnership
Grant.
References
Achenbach, T. M. (1991). Integrative guide for the 1991 CBCL/4-
18, YRS, and TRF profiles. Burlington: Department of
Psychiatry, University of Vermont.
American Educational Research Association, American Psycho­
logical Association, & National Council for Measurement in
Education. (1999). Standards for educational and psychological
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
testing. Washington, DC: American Educational Research
Association.
Bradshaw, C. P., Buckley, J. A., & Ialongo, N. S. (2008). School-
based service utilization among urban children with early-
onset educational and mental health problems: The squeaky
wheel phenomenon. School Psychology Quarterly, 23,
169–186. doi:10.1037/1045-3830.23.2.169
Cohen, J. (1992). A power primer. Psychological Bulletin, 112,
155–159.
DiPerna, J. C., & Elliott, S. N. (2002). Promoting academic
enablers to improve student achievement: An introduction to
the mini-series. School Psychology Review, 31, 293–297.
Drummond, T. (1994). The Student Risk Screening Scale (SRSS).
Grants Pass, OR: Josephine County Mental Health Program.
Drummond, T., Eddy, J. M., Reid, J. B., & Bank, L. (1994,
November). The Student Risk Screening Scale: A brief
teacher screening instrument for conduct disorder. Paper
presented at the Fourth Annual Prevention Conference,
Washington, DC.
Elliott, S. N., & Gresham, F. M. (2007). Social Skills Improvement
System: Performance Screening Guides. Bloomington, MN:
Pearson Assessments.
Fuchs, D., & Fuchs, L. (2006). Introduction to response to inter-
vention: What, why, and how valid is it? Reading Research
Quarterly, 41, 93–99. doi:10.1598/RRQ.41.1.4
Goodman, R. (1997). The Strengths and Difficulties Questionnaire:
A research note. Journal of Child Psychology and Psychiatry,
38, 581–586. doi:10.1111/j.1469-7610.1997.tb01545.x
Goodman, R. (2001). Psychometric properties of the Strengths and
Difficulties Questionnaire (SDQ). Journal of the American
Academy of Child & Adolescent Psychiatry, 40, 1337–1345.
doi:10.1097/00004583-200111000-00015
Greenwood, C. R., Horton, B. T., & Utley, C. A. (2002). Academic
engagement: Current perspectives on research and practice.
School Psychology Review, 31, 328–349.
Gresham, F. M., & Elliott, S. N. (2008). Social Skills Improvement
System: Rating Scales. Bloomington, MN: Pearson Assessments.
Harrison, J. R., Vannest, K. J., & Reynolds, C. R. (2013). Social
acceptability of five screening tools for social, emotional, and
behavioral challenges. Behavioral Disorders, 38, 171–189.
Hopkins, W. G. (2002). A scale of magnitudes for effect statis-
tics. Retrieved from http://www.sportsci.org/resource/stats/
effectmag.html
Individuals With Disabilities Education Improvement Act of 2004,
20 U.S.C. 1400 et esq. (2004). (reauthorization of Individuals
With Disabilities Act 1990).
Institute. (2004). SAS/STAT 9.1: User’s guide. Cary, NC: SAS
Publishing.
Kalberg, J. R., Lane, K. L., & Menzies, H. M. (2010). Using sys-
tematic screening procedures to identify students who are
nonresponsive to primary prevention efforts: Integrating aca-
demic and behavioral measures. Education & Treatment of
Children, 33, 561–584.
Kamphaus, R. W., & Reynolds, C. R. (2007). BASC™-2 Behavior
and Emotional Screening System (BASC™-2-BESS). San
Antonio, TX: Pearson.
Kamps, D. M., & Greenwood, C. R. (2005). Formulating secondary-
level reading interventions. Journal of Learning Disabilities,
38, 500–509. doi:10.1177/00222194050380060501
Lane et al. 125
Kettler, R. J., Elliott, S. N., Beddow, P. A., Compton, E., McGrath,
D., Kraase, K. K., . . . Hinton, K. (2010). What do an alternate
academic achievement standards measure? A multi-trait-mul-
timethod analysis. Exceptional Children, 76, 457–474.
Kettler, R. J., Elliott, S. N., Davies, M., & Griffin, P. (2012).
Testing a multi-stage screening system: Predicting perfor-
mance on Australia’s national achievement test using teachers’
ratings of academic and social behaviors. School Psychology
International, 33, 93–111. doi:10.1177/0143034311403036
Kraemer, H. C. (1992). Evaluating medical tests: Objective and
quantitative guidelines. Newbury Park, CA: SAGE.
Lane, K. L., Bruhn, A. L., Eisner, S. L., & Kalberg, J. R. (2010).
Score reliability and validity of the Student Risk Screening
Scale: A psychometrically-sound, feasible tool for use in
urban middle schools. Journal of Emotional and Behavioral
Disorders, 18, 211–224. doi:10.1177/1063426609349733
Lane, K. L., Kalberg, J. R., Lambert, W. E., Crnobori, M., &
Bruhn, A. L. (2010). A comparison of systematic screen-
ing tools for emotional and behavioral disorders: A replica-
tion. Journal of Emotional and Behavioral Disorders, 18,
100–112. doi:10.1177/1063426609341069
Lane, K. L., Kalberg, J. R., & Menzies, H. M. (2009). Developing
schoolwide programs to prevent and manage problem behav-
iors: A step-by-step approach. New York, NY: Guilford
Press.
Lane, K. L., Kalberg, J. R., Parks, R. J., & Carter, E. W. (2008).
Student risk screening scale: Initial evidence for score
reliability and validity at the high school level. Journal
of Emotional and Behavioral Disorders, 16, 178–190.
doi:10.1177/106342660831421
Lane, K. L., Little, M. A., Casey, A. M., Lambert, W., Wehby, J.
H., Weisenbach, J. L., & Phillips, A. (2009). A comparison
of systematic screening tools for emotional and behavioral
disorders: How do they compare? Journal of Emotional and
Behavioral Disorders, 17, 93–105.
Lane, K. L., Menzies, H., Bruhn, A., & Crnobori, M. (2011).
Managing challenging behaviors in schools: Research-based
strategies that work. New York, NY: Guilford Press.
Lane, K. L., Menzies, H., Kalberg, J. R., & Oakes, W. P. (2012). An
comprehensive, integrated three-tier model to meet students’
academic, behavioral, and social needs. In K. Harris, T. Urdan,
& S. Graham (Eds.), American Psychological Association.
Educational psychology handbook (Vol. 3, pp. 551–581).
Washington, DC: American Psychological Association.
Lane, K. L., Menzies, H. M., Oakes, W. P., & Kalberg, J. R. (2012).
Systematic screenings of behavior to support instruction:
From preschool to high school. New York, NY: Guilford.
Lane, K. L., Menzies, H. M., Oakes, W. P., Lambert, W., Cox,
M., & Hankins, K. (2012). A validation of the Student Risk
Screening Scale for internalizing and externalizing behaviors:
Patterns in rural and urban elementary schools. Behavioral
Disorders, 37, 244–270.
Lane, K. L., Oakes, W. P., Ennis, R., Cox, M. L., Schatschneider,
C., & Lambert, W. (2013). Additional evidence for the reli-
ability and validity of the student risk screening scale at
the high school level: A replication and extension. Journal
of Emotional and Behavioral Disorders, 21, 97–115.
doi:10.1177/1063426611407339
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
Lane, K. L., Oakes, W. P., Harris, P. J., Menzies, H. M., Cox, M.
L., & Lambert, W. (2012). Initial evidence for the reliability
and validity of the Student Risk Screening Scale for internal-
izing and externalizing behaviors at the elementary level.
Behavioral Disorders, 37, 99–122.
Lane, K. L., Parks, R. J., Kalberg, J. R., & Carter, E. W. (2007).
Systematic screening at the middle school level: Score reli-
ability and validity of the student risk screening scale. Journal
of Emotional and Behavioral Disorders, 15, 209–222.
Lane, K. L., Richards-Tutor, C., Oakes, W. P., & Connor, K.
(2014). Initial evidence for the reliability and validity of the
student risk screening scale with elementary age English
learners. Assessment and Effective Intervention, 39, 219–232.
doi:10.1177/1534508413496836
Malecki, C. K., & Elliott, S. N. (2002). Children’s social behaviors
as predictors of academic achievement: A longitudinal analy-
sis. School Psychology Quarterly, 17, 1–23. doi:10.1521/
scpq.17.1.1.19902
McIntosh, K., Chard, D. J., Boland, J. B., & Horner, R. H. (2006).
Demonstration of combined efforts in school-wide academic
and behavioral systems and incidence of reading and behav-
ior challenges in early elementary grades. Journal of Positive
Behavior Interventions, 8, 146–154. doi:10.1177/109830070
60080030301
McIntosh, K., Filter, K. J., Bennett, J. L., Ryan, C., & Sugai,
G. (2010). Principles of sustainable prevention: Designing
scale-up of school-wide positive behavior support to pro-
mote durable systems. Psychology in the Schools, 47, 5–21.
doi:10.1002/pits.20448
McIntosh, K., Frank, J. L., & Spaulding, S. A. (2010). Establishing
research-based trajectories of office discipline referrals for
individual students. School Psychology Review, 39, 380–394.
Menzies, H. M., & Lane, K. L. (2011). Using self-regulation
strategies and functional assessment-based interventions to
provide academic and behavioral support to students at risk
within three-tiered models of prevention. Preventing School
Failure, 55, 181–191. doi:10.1080/1045988X.2010.520358
National Governors Association Center for Best Practices & the
Council of Chief State School. (2010). Common Core State
Standards Initiative. Washington, DC: Author.
Nelson, J. R., Benner, G. J., Lane, K., & Smith, B. W. (2004). An
investigation of the academic achievement of K-12 students
with emotional and behavioral disorders in public school set-
tings. Exceptional Children, 71, 59–73.
No Child Left Behind (NCLB) Act of 2001, 20 U.S.C.A. § 6301
et seq. (West 2003).
Oakes, W. P., Lane, K. L., Cox, M., Magrane, A., Jenkins, A., &
Hankins, K. (2012). Tier 2 supports to improve motivation
and performance of elementary students with behavioral chal-
lenges and poor work completion. Education & Treatment of
Children, 35, 547–584. doi:10.1353/etc.2012.0024
Oakes, W. P., Wilder, K. S., Lane, K. L., Powers, L., Yokoyama, L.
T. K., O’Hare, M. E., & Jenkins, A. B. (2010). Psychometric
properties of the Student Risk Screening Scale: An effective
tool for use in diverse urban elementary schools. Assessment
for Effective Intervention, 35, 231–239.
Pearson Education. (2008). AIMSweb® Benchmark and progress
monitoring system for grades K-8. Bloomington, MN: Author.
126 Assessment for Effective Intervention 40(2)
Petras, H., Chilcoat, H. D., Leaf, P. J., Ialongo, N. S., & Kellam,
S. G. (2004). Utility of TOCA-R scores during the elemen-
tary school years in identifying later violence among ado-
lescent males. Journal of the American Academy of Child &
Adolescent Psychiatry, 43, 88–96.
Pianta, R. C., Hamre, B. K., & Allen, J. P. (2012). Teacher-student
relationships and engagement: Conceptualizing, measuring,
and improving the capacity of classroom interactions. In S. L.
Christenson, A. L. Reschley, & C. Wylie (Eds.), Handbook of
research on student engagement (pp. 365–386). New York,
NY: Springer. doi:10.1007/978-1-4614-2018-7_17
Reinke, W. M., Herman, K. C., Petras, H., & Ialongo, N. S. (2008).
Empirically derived subtypes of child academic and behav-
ior problems: Co-occurrence and distal outcomes. Journal
of Abnormal Child Psychology, 36, 759–770. doi:10.1007/
s10802-007-9208-2
Ross, S. W., & Horner, R. H. (2014). Bully prevention in posi-
tive behavior support: Preliminary evaluation of third-,
fourth-, and fifth-grade attitudes toward bullying. Journal
of Emotional and Behavioral Disorders, 22, 225–236.
doi:10.1177/106342661349142
Schatschneider, C. (2013). I am ROC curves (and so can you!).
In Y. Petscher, C. Schatschneider, & D. L. Compton (Eds.),
Applied quantitative analysis in education and the social sci-
ences (pp. 65–92). New York, NY: Routledge.
Sprague, J., Walker, H., Golly, A., White, K., Myers, D. R., &
Shannon, T. (2001). Translating research into effective prac-
tice: The effects of a universal staff and student intervention
on indicators of discipline and school safety. Education &
Treatment of Children, 24, 495–511.
Downloaded from aei.sagepub.com at ARIZONA STATE UNIV on October 16, 2015
View publication stats
Sugai, G., & Horner, R. H. (2009). Responsiveness-to-intervention
and school-wide positive behavior supports: Integration of
multi-tiered system approaches. Exceptionality, 17, 223–237.
doi:10.1080/09362830903235375
Sugai, G., Horner, R. H., Dunlap, G., Hieneman, M., Lewis, T.
J., Nelson, C. M., . . . Wilcox, B. (1999). Applying positive
behavioral support and functional behavioral assessment
in schools (Technical Assistance Guide 1, Version 1.4.3).
Washington, DC: Center on Positive Behavioral Interventions
and Supports, U.S. Department of Education.
U.S. Department of Education. (2011). Common core of data:
School directory information. Washington, DC: National
Center for Education Statistics.
Wagner, M., Kutash, K., Duchnowski, A. J., Epstein, M. H., &
Sumi, W. C. (2005). The children and youth we serve: A
national picture of the characteristics of students with emo-
tional disturbances receiving special education. Journal of
Emotional and Behavioral Disorders, 13, 79–96.
Walker, H. M., Ramsey, E., & Gresham, F. M. (2004). Antisocial
behavior in school: Evidence-based practices (2nd ed.).
Belmont, CA: Wadsworth.
Walker, H. M., & Severson, H. H. (1992). Systematic screening
for behavior disorders: Technical manual. Longmont, CO:
Sopris West.
Walker, H. M., Severson, H. H., & Feil, E. G. (1995). The Early
Screening Project: A proven child find process. Longmont,
CO: Sopris West.
Wentzel, K. R., & Watkins, D. E. (2002). Peer relationships and
collaborative learning as contexts for academic enablers.
School Psychology Review, 31, 366–377.