Practice:

Key Action:

Evaluate your outcomes to show your program is making a difference

Design the most rigorous evaluation possible

TOOL: Decision Tree: Determining Feasibility for Rigorous Evaluation Design

Purpose:

This decision tree guides you through a series of questions to help you determine

which rigorous evaluation design is most appropriate for your program. Progressing from most to least rigorous designs the flow chart presents the criteria that must be met for each type of evaluation. The accompanying list of evaluation design definitions provides an e!planation of the structure and methods used in each design. "ote: Design # or $E!perimental% is not the only type of rigorous evaluation design& designs ' through ( in the chart which are all $)uasi*e!perimental % are also considered to be rigorous. Instructions: #. Review the decision tree and accompanying definitions.

'. +et answers to any questions you have about whether your schools meet the various criteria. ,. Develop an opinion about which evaluation design will be most appropriate.

Practice:
Key Action:

Evaluate your outcomes to show your program is making a difference

Design the most rigorous evaluation possible

Decision Tree: Determining Feasibility for

-re students randomly assigned to magnet school.s/ by lottery0 1s there an oversubscription or more students applying than are accepted to the magnet school.s/0

igorous Evaluation Design

5E6

5E6

2an you ensure that students who aren3t accepted do not enroll in a school similar to the magnet school.s/0

5E6

E7PER18E"T-9 DE61+"#

"4

"4

"4

)uasi*E!perimental Design'

-re consistent .(* year/ student achievement data available for magnet and non* magnet schools0

5E6

-re consistent .(* year/ demographic data available for magnet and non* magnet schools0

5E6

:ave attendance boundaries for non* magnet schools been consistent for the past ( years0

5E6

-re there a large number of schools .e.g. '; magnet schools and <; non* magnet comparison schools/0

5E6

:ave students used the same test for ( years0

-re consistent .( "4 "4

:ave attendance "4 "4 "4

PRE*P46T TE6T 4R P46T*TE6T 4"95(

"4 "4

Does the district contain pre* conversion schools with similar demographics and prior student achievement0

5E6

5E6

248P-R164" +R4=P PR4PE"61T5 624R1"+ T4 8-T2: 6T=DE"T6<

5E6

-re there multi*year student data in both magnet and non* magnet comparison schools0

248P-R164" +R4=P6 PR4PE"61T5 624R1"+ T4 8-T2: 62:4496<

1"TERR=PTED T18E 6ER1E6,

1"TERR=PTED

-re there multi*year

'

Practice:
Key Action:

Evaluate your outcomes to show your program is making a difference

Design the most rigorous evaluation possible

Decision Tree for !agnet Program igorous Evaluation Design: Definitions

Refer to these >ey definitions of various rigorous evaluation designs to help you navigate the decision tree on page '.

" E#perimental Design

This design is also referred to as a $randomi?ed controlled trial.% E!perimental design is possible only when more students apply for the magnet school than can be accommodated. @hen there is oversubscription students can be randomly assigned to the magnet school through a lottery system. 1t is also critical that nonselected students then enroll in school.s/ that are not similar to the magnet school to be able to attribute differences in outcomes to differences in the magnet program. -ll types of quasi*e!perimental designs share the characteristic of attempting to control for some un>nown quality that may influence an outcome. -n important variable is selection bias: whether the very act of choosing to attend a magnet school indicates a difference between a magnet school student and a non*magnet school student even if both share >ey characteristics such as demographics and prior achievement. -ll quasi*e!perimental designs then are efforts to $equali?e% as much as possible the two groups. This design requires student*level data from repeated intervals before and after magnet school attendance. Trends in each magnet student3s outcomes such as achievement prior to enrollment in a magnet school are compared to trends after magnet school attendance. Thus the evaluation avoids the issue of selection bias by comparing each $chooser% with himAherself and it is possible to attribute changes in outcomes to attending the magnet school. 1nterrupted time series require a fairly long history of scores and sufficient numbers of participants to ensure that the changes are in fact associated with enrollment in magnet schools. 8any quasi*e!perimental designs compare groups of magnet school students to groups of non*magnet school students. The >ey to this type of design is to ensure as much as possible that students in the comparison group are similar to those in the magnet school. 4ne technique used is propensity score matching which selects a comparison group using variables that are most li>ely to have an effect on the outcome such as demographics and prior test sores in the case of magnet school achievement. 4ften the matching occurs in two stagesBfirst schools are matched on demographics prior achievement and other variables of interest& then students within those schools are matched to form the comparison group. Depending on the data available these types of comparison group designs may be the most rigorous design possible for your evaluation. Coth compare outcomes of magnet students with non*magnet students. 1n pre*post*test comparison designs the pre*test indicates differences in outcomes before magnet school attendance and additionally matching is used to ma>e sure the differences between the two groups of students is as small as possible. @ith post*test only design the matching of students needs to be strong in order to control for differences in post*test results between the magnet students and non*magnet students.

$ %uasi& e#perimental design

' (nterrupted Time )eries

* +omparison ,roup

- Pre&Post& Test or Post& Test Only