You are on page 1of 5

Practice:

Evaluate outcomes to show your program is


making a difference
Key Action: Design the most rigorous evaluation possible

SAMPLE MATERIAL: Process for Selecting Comparison Schools

Purpose: When you evaluate magnet program outcomes, selection of appropriate


comparison schools is essential to producing credible data analysis.
The screenshots, tables, and narrative in this report outline the process
one district used to select comparison schools for an MSAP rigorous
evaluation. The document illustrates the steps the district took to narrow
the pool of potential comparison schools to select the best candidates,
including matching by demographics and eliminating those with similar
program elements, or “treatment.”

Source: Northwest Suburban Integration School District, MN, is an interdistrict


consortium that received Magnet Schools Assistance Program (MSAP)
funds in 2004 and 2007. The sample material is from the district’s 2007
MSAP rigorous evaluation documentation.

1
Practice: Evaluate outcomes to show your program is
making a difference
Key Action: Design the most rigorous evaluation possible

Northwest Suburban Integration School District - Selection of Comparable Schools for Rigorous Evaluation

The selection of comparable schools was based on the closest match of demographics to the Magnet Schools Assistance Program (MSAP)
magnet schools being evaluated. The demographics that were closest for total enrollment, percentage of black students, percentage of
free or reduced-price lunch, and percentage of limited English proficient (LEP) students, determined the first round of selection for
comparable schools. Once those factors were determined, then each of the proposed comparable schools was contacted to find out if the
school had programs like International Baccalaureate or other treatments that would give it characteristics similar to those that the magnet
schools are implementing. The schools that had similar treatments were eliminated and the next closest school was checked until a final
selection could be made.

Step 1. Gathering the demographic


information
October 1 enrollment data
for each year is available from the
Minnesota Department of Education
(MDE) website. The source of the data is
the Minnesota Automated Reporting
Student System (MARSS).

The data downloads were saved as an


Excel file. The October 1 enrollment file
includes data from every school in the
state by grade, ethnicity, and gender.
Total enrollments by school are also
included in this file.

A second data file is available for


enrollments by special population such
as school, grade, free lunch, reduced
lunch, special education, and limited
English proficient. Free lunch and
reduced lunch were added together.
That file contains all schools in the state
and can be downloaded as an Excel file.

2
Practice: Evaluate outcomes to show your program is
making a difference
Key Action: Design the most rigorous evaluation possible

Step 2. Preparing the data for evaluation


The two data files both contain a detailed list
of enrollments by grade, and a summary list
by school. The October 1 enrollment
summary file contains over 2,000 records,
one for every school in the state. The
detailed enrollment files contain over 11,000
records. These files were reduced to only
the metro schools that had potentially similar
demographics. The data download of AYP
records for each school contains over 2,600
records, and these records also were scaled
down to just the metro schools. Records
from out-of-state schools and all charter
schools were taken out of all files.

Formulas were created in the October 1


enrollment file to calculate minority and
black percentages on the remaining records.
Formulas were created in the special
population enrollment file to determine the
percentage of limited English proficient and
the percentage of free or reduced-price
lunch population.

The two Excel files were then merged to


show one record for each school so that the
key factors such as total enrollment,
. percentage of black student enrollment,
percentage of LEP and the percentage of
free or reduced-price lunch could easily
be sorted and compared for the evaluation.

3
Practice: Evaluate outcomes to show your program is
making a difference
Key Action: Design the most rigorous evaluation possible

Step 3. Using the data to select comparable schools


October 1 Enrollment Data - MDE
School Grade Total Total # % Total % Black LEP SPE % LEP % SpEd Total % Free-
This chart
(* name is Minority Student Minority Black K-12 K12 K-12 K-12 Free- reduced includes schools
protected) reduced that had the
SCHOOL A All Grades 708 2259 31.34% 392 17.35% 233 197 10.31% 8.72% 567 25.10% most similar
SCHOOL B All Grades 276 884 31.22% 166 18.78% 77 115 8.71% 13.01% 321 36.31% demographics to
SCHOOL C All Grades 1714 2030 84.43% 386 19.01% 968 268 47.68% 13.20% 1660 81.77%
the MSAP grant
schools. The
SCHOOL D All Grades 431 1316 32.75% 267 20.29% 61 188 4.64% 14.29% 295 22.42% magnet schools
SCHOOL E All Grades 628 1718 36.55% 398 23.17% 99 174 5.76% 10.13% 459 26.72% are highlighted in
SCHOOL F All Grades 930 1978 47.02% 492 24.87% 135 206 6.83% 10.41% 724 36.60%
green and the
potential
SCHOOL G All Grades 829 1437 57.69% 366 25.47% 314 231 21.85% 16.08% 743 51.70%
comparable
SCHOOL H All Grades 986 1476 66.80% 420 28.46% 419 206 28.39% 13.96% 909 61.59% schools in
SCHOOL I All Grades 764 1420 53.80% 411 28.94% 213 146 15.00% 10.28% 650 45.77% yellow.
SCHOOL J All Grades 1264 1607 78.66% 490 30.49% 489 240 30.43% 14.93% 1229 76.48%

SCHOOL K All Grades 535 962 55.61% 308 32.02% 157 146 16.32% 15.18% 530 55.09%

SCHOOL L All Grades 1430 1495 95.65% 482 32.24% 743 204 49.70% 13.65% 1372 91.77%

SCHOOL M All Grades 1403 2134 65.75% 697 32.66% 443 179 20.76% 8.39% 1168 54.73%

SCHOOL N All Grades 979 1943 50.39% 663 34.12% 72 257 3.71% 13.23% 790 40.66%

SCHOOL O All Grades 1012 1522 66.49% 542 35.61% 195 139 12.81% 9.13% 743 48.82%

SCHOOL P All Grades 732 1111 65.89% 410 36.90% 147 166 13.23% 14.94% 569 51.22%

SCHOOL Q All Grades 533 724 73.62% 301 41.57% 160 83 22.10% 11.46% 504 69.61%

SCHOOL R All Grades 757 868 87.21% 376 43.32% 343 186 39.52% 21.43% 752 86.64%

SCHOOL S All Grades 367 506 72.53% 223 44.07% 79 104 15.61% 20.55% 398 78.66%

SCHOOL U All Grades 1034 1202 86.02% 535 44.51% 192 147 15.97% 12.23% 888 73.88%

SCHOOL V All Grades 957 1126 84.99% 510 45.29% 463 184 41.12% 16.34% 900 79.93%

SCHOOL W All Grades 810 1053 76.92% 542 51.47% 215 159 20.42% 15.10% 627 59.54%

SCHOOL X All Grades 899 1020 88.14% 567 55.59% 397 144 38.92% 14.12% 853 83.63%

SCHOOL Y All Grades 298 447 66.67% 252 56.38% 0 51 0.00% 11.41% 239 53.47%

SCHOOL Z All Grades 610 634 96.21% 417 65.77% 122 161 19.24% 25.39% 510 80.44%

4
Practice: Evaluate outcomes to show your program is
making a difference
Key Action: Design the most rigorous evaluation possible

The selection of comparable schools was narrowed to the closest in demographics to the magnet schools, based first on black
population, secondly by free or reduced-price lunch, and third by LEP population, using the size of the school’s population when
possible.

A sample chart of the Adequate Yearly Progress (AYP) report from the Minnesota Department of Education follows:

2008
AYP Results
Schools
8/1/2008
School Name Title 1 Free AYP Status
in &Reduced
2009 Lunch
Percentage
School XYZ NO 19 Making AYP
(Equivalency School)
School K NO 55 Not Making AYP
School B NO 36 Not Making AYP
School E NO 26 Not Making AYP
School P NO 51 Not Making AYP
School O NO 48 Not Making AYP
School Q NO 69 Not Making AYP

Step 4. Final selection of comparable schools


The final step was to explore the status of the school to determine if it was implementing a treatment similar to those of the MSAP
magnet schools. If the proposed school was using any treatment similar to the magnet schools being evaluated, then it was not
selected, even if the demographics were closer than other potential comparable schools.