Using Early Warning Indicators to

Identify Students at Highest Risk

of Dropping Out
Ruth Curran Neild
Center for Social Organization of Schools, Johns Hopkins University
 Relationship to college and career
readiness
• Earning high school diploma is a
critical juncture in the pipeline

• Provides a conceptual model for

tracking likelihood of young adult
success in postsecondary education
and/or work

Copyright © 2010. All rights reserved.

 The Checklist Manifesto

Copyright © 2010. All rights reserved.

 All professions in the 21st
century face increasing
complexity in their work

Copyright © 2010. All rights reserved.

 The Modern Challenge:
Keeping Track of a Lot of “Moving Pieces”

• It’s easy for well-trained professionals

to:
• Forget
• Fail to communicate
• Overlook
• Fail to “connect the dots”

Copyright © 2010. All rights reserved.

 Most errors of are those of
ineptitude
Mistakes that occur because we do not make proper use of
what we know

(As contrasted with errors of ignorance

– mistakes that arise from not knowing
what to do)

Copyright © 2010. All rights reserved.

 Like other professions, teachers operate in an
environment of increasing complexity

•Increased responsibility for outcomes of all

students, including those who are disengaging from
school
•Increased responsibility to “individualize”
education - to find the “right solution” or “right fit” for
each student

•Substantial amounts of data about students

collected and made available to teachers
Copyright © 2010. All rights reserved.
 Early Warning Indicator Systems
enable teachers to:

• Use empirically-developed data indicators

that are most predictive of a given outcome
as a “flag” that a student are in trouble
• Track interventions that have been assigned
to particular students
• Systematically track associations between
interventions and outcomes for students at
their school

Copyright © 2010. All rights reserved.

 Logic of
Early Warning Indicators
Of High School Dropout

Copyright © 2010. All rights reserved.

 There are many underlying reasons for
dropping out of school

• Social science research using secondary data

sets has helped us to understand correlates
of dropping out and…most importantly, that…
• Dropping out is the culmination of a gradual
process of disengagement from school

Copyright © 2010. All rights reserved.

 For educators, this research has real
limitations

How do we know which specific individuals are

most likely to drop out so that we can target
interventions to them?

How early in students’ careers can we reliably

identify those on the path to dropping out?

Can we identify students with readily-available

data or do we need specialized assessments?

Copyright © 2010. All rights reserved.

 Developed in the Context of “Dropout
Factory” Schools

• At these schools, 40% or more of the students

fail to graduate
• Family income, family structure, race/ethnicity,
scores on nationally-normed tests are usually the
same or within a narrow band in these schools

Copyright © 2010. All rights reserved.

 Four questions about EWIs

• What are the characteristics of a good EWI

system?
• What are the signals?
• What technological and organizational
infrastructure is needed to “capture” the
signal?
• What can schools and districts do once the
signals are identified and captured?

Copyright © 2010. All rights reserved.

 Characteristics of a good EWI system

Empirically developed: The “signals” are identified through

analysis of longitudinal data for prior cohorts of students.
High accuracy: A high percentage of students with the “signals”
drop out. Conversely, a low percentage of students without the
“signals” graduate.
High yield: These “signals” capture most of the dropouts
(avoiding the “1% problem”).
Accessible data: Data that provide the “signals” are readily
available and relatively inexpensive to access.
Copyright © 2010. All rights reserved.
 How did we identify the “signals” of eventual
dropout?

• Empirical analysis of cohorts in Philadelphia,

starting with 6th graders (Balfanz, Herzog, &
MacIver), and 8th graders (Neild & Balfanz, 2006)
• Data scan of longitudinal student record
data
– Test scores
– Report card grades
– Attendance
– Special education and ELL status
– Gender
– Age
– Race/ethnic background
Copyright © 2010. All rights reserved.
 Looked for a 75% threshold – why?

• Choosing a “strong signal” – students who

are at highest risk of dropping out
• By not making the net too broad, scarce
resources can be targeted at those
students who are greatest risk

Copyright © 2010. All rights reserved.

 The Big Four in 6th grade

• Failing Math
• Failing English
• Attendance <80%
• At least one poor behavior mark

Copyright © 2010. All rights reserved. (Balfanz, Herzog, & MacIver)

 8th grade warning signals

• Three factors gave students at least

a 75% probability of dropping out:

1. Failing math in 8th grade

2. Failing English in 8th grade
3. Attending less than 80% of
the time

Copyright © 2010. All rights reserved.

 54% of the dropouts
sent one or more
of these signals
in 8th grade

Copyright © 2010. All rights reserved.

 Had an 8th grade
“signal”

Did not have an 8th grade signal:

Passed 8th grade English

Passed 8th grade Math
Attended at least 80% of the time

Copyright © 2010. All rights reserved.

 9th Grade signals

• Three factors gave students at least

a 75% probability of dropping out:

1. Earning fewer than 2 credits

2. Not being promoted to 10th
grade
3. Attending less than 70% of
the time

Copyright © 2010. All rights reserved.

 80% of the dropouts
sent one or more
of these signals
in 8th or 9th grade

Copyright © 2010. All rights reserved.

Copyright © 2010. All rights reserved.
 Technological Infrastructure: Real time data

2006-07 School A 6th g., Teacher A

BEHAVIOR
STUDENTS ATTENDANCE COURSES PSSA READING LEVEL
Where Student Needs Support
FINAL FINAL MP1 MP1
Att Att to Final Behavior Marks EXIT Change,
LAST FIRST 05-06 05-06 06-07 06-07 MP1 Behavior Mar-06 Dec-06
05-06 Date 2005-2006 Jun-06 Jun-Dec
Math Rdg Math Rdg Marks 2006-07
Does NOT: accept
responsibil. for choices;
BELOW BASIC
complete work on time;
Student A 88% 95% D D NA NA NA Rdg 927 Math 4 NA NA
follow school rules; handle
1157
conflict; show positive
attitude

Does NOT: complete work

BASIC Rdg
on time; follow school
Student B 96% 99% B C C C None 1259 PROF 5 5.5 0.5
rules; make appropriate
Math 1335
transitions; organize self

BELOW BASIC
Fails to be attentive Rdg 968
Student C 85% 90% C D C C ALL AREAS 4 4 0
(Math) PROF Math
1335
PROF Rdg
Student D 97% 100% A B B B None None 1280 ADV 5.5 6 0.5
Math 1544

Copyright © 2010. All rights reserved.

 Conceptual frame for intervention

Whole school
interventions

More labor intensive

Targeted More specialized
Interventions
More costly
Intensive Interventions

Copyright © 2010. All rights reserved.

 Organizational Infrastructure

“Near-peers” to
nag and nurture
TEAMS of Teachers,
ideally all teaching Supported by…
the same group of Links to social
services
students

Copyright © 2010. All rights reserved.

 Implications for College and Career
Readiness
Possibility of using indicators across systems to
address readiness

EXAMPLE
Connecting school district and local college data to
identify high school predictors of key postsecondary
outcomes, such as:
Placement out of remedial courses
Overall credit accumulation and in key areas
Return for a second semester or a second year
Copyright © 2010. All rights reserved.
 EXAMPLE

New York City

Copyright © 2010. All rights reserved.

 Implications for College and Career
Readiness

Possibility of using indicators across systems to

address readiness

There is a great deal that is unknown about

whether there are readily accessible, high
accuracy, high-yield high school predictors
of postsecondary outcomes

Copyright © 2010. All rights reserved.

 Implications for College and Career
Readiness
Possibility of using indicators within a higher
education system to identify students at-risk of
course failure

EXAMPLE
Survey data about study habits in high school and
other non-cognitive predictors, combined with data
on class attendance and interim grades

Copyright © 2010. All rights reserved.

 The Checklist Manifesto

The purpose of a good checklist

EWI System
is not to fill out paperwork or to
prove to others that we’ve
“covered our bases,” but to help
teachers
well-trained professionals cope
with the complexity and detail of
their work
keeping students on track
in the modern world.
Copyright © 2010. All rights reserved.
 Ruth Curran Neild
Center for Social Organization of Schools
Johns Hopkins University

rneild@csos.jhu.edu

Copyright © 2010. All rights reserved.