You are on page 1of 18

Data-Driven Decision-Making in K-12

Education: Using the Learning Delta to
Manage for Results
A High Delta Learning  White Paper

Gregory D. Luther
Founder and Principal
High Delta Learning, LLC

May, 2003

their progress that captures each stu- leverage will help move the system dent's change in mastery for a given Part 1: Value-Added Testing itself. at schools do not adequately account continuously accumulating and the University of Tennessee Value- for their primary product: learning. (TVAAS). and influ. spend waiting for learning to hap. education phi. measurement. and timely tical analysis and experimental information on which to base their designs taken from social science 2. 1884 Introduction sector. . analyzing longitudinal Learning Added Research and Assessment Measuring and analyzing the impact Delta data for all students represents Center. Delta is a measure of learning ence toward the right targets. Added Assessment System school districts. and teaching generally. In addition to and Teacher Effectiveness unique blend of private organizations improving the manageability of with public-minded goals-has the individual student progress. financial resources. importance and utility of collecting systems."  Lord Kelvin. In the words of expense and with substantial delays schools. the Thomas B. and parents into the outside research to assess or validate education by managing the ef- position of education consumers a program's salutary effects. and it has clout that arrive at progress measures for Tennessee that substantiates the classrooms. Perhaps most importantly. The philanthropic sector-America's period of time. energy. But a critical difficulty facing educa. but you have scarcely in your thoughts advanced to the state of Science. when you cannot express it in numbers. used to minimize the time students Together with the Tennessee Value- tion philanthropiesas well as cities. and clinical drug trials. Page 1 "…When you can measure what you are speaking about. classroom workflows must be digitized in Private individuals and organizations… In this paper I propose the Learn- order to enable the instrumen- can spend their dollars exactly where ing Delta as a top-level perform- tation of teaching and learning they seek to do the most good. somebody improving the effectiveness of innovations to back with their else has tousually at considerable education reform programs. and parentsis that pen. but when you cannot measure it. it may be the beginning of knowledge. between the point of data capture tion: 3. If they ance metric for schools. Learning There is already a substantial body freedom of action to push on the right Delta data can be aggregated to of empirical work from the state of pressure points. any other single factor. Learning Deltas can be data on students' learning progress. it. prises outside of the education . entire schools. understandable. and school system that uses most powerful. there is a call for 1. management processes like enter. Fordham Founda. whatever the matter may be. Dr. In short. . to be fectiveness of teaching than by who have a growing need for performed using multivariate statis. William Sanders. direct their money. teacher effectiveness was by far the employ performance measures and school. or school i most parents lack . found that of all the factors of reforms and innovations on an embedded research process that influencing the observed change in students' learning progress is a can inform and help to improve student test scores from year to year. In order to implement Learning and the reporting of results. More can be done to improve school districts. measure learning progress is lanthropies have choices to make because schools do not measure fundamental to managing and with regard to which reforms or their own performance. Similarly. In this paper I make a three-part ploying a "generally accepted argument: The emerging market for K-12 accounting principle" for learning education services is placing cities. problem because schools do not instruction in every classroom. Learning in schools. instead of em. Using the Learning Delta to choices. your knowledge is of a meagre and unsatisfactory kind. you know something about it. and express it in numbers. Typically. accurate. Delta Management.

Rate from the bottom quintile. the term. Learning Delta: knowledge that is advanced be- ranked according to their stu. and the average score academic subject divided by the on or near the first day of class to increases of all the students taught change in time. Teachers performing in the two student. ∆ Time • Pre-test resultstagged to each 2." And the key to that effort she/he will not complete the course a whole.e. then the Step 2 . thought of as a scale score that is • The difference between a stu- 3. failed to produce any achieve. or monthly to be highly significant and larger Learning Delta of 100 (i. then. the Rate of dents' average increase in test Learning • A student's pre-test score is a scores--raised their students’ Progress ∆ Mastery measure of the subject knowl- achievement test scores 39 per. and stu- including class size and "hetero. 4. science. Learning Delta is and personnel decisions. Learning Delta is given as the to his or her scores from the previ- change in a student's mastery of an Step 1 . Every student in grades 2-8 is room. Where Sanders' establish the baseline learning level by each teacher are reported to "value-added" is a measure of of each student. Testing of high school Part 2: Using Learning Delta Data Discipline students began in 1995. tions that would appear on a final ings in Tennessee include the exam.At the time of each assess- 60 (i. The teacher quality effect on student is proceeding at a pace of additional tests are administered on student performance was found 100 points per term and has a a weekly. Page 2 The TVAAS has been in use since progress being made in the class. each test is in size than any other factor-. the student has dents' scores reflect their new level geneity. 50 points basis. The "distance" term can be recorded and/or stored." only mastered curriculum units of subject knowledge attained since worth 30 points by the mid-term.5 terms = 100 points/term). administrators for use in evaluations academic distance.5 terms = 60 ment Learning Delta figures are by improving the effectiveness of points/term). units making up that course in one Delta for that student. The top quintile of teachers-. language arts. yond the scope of the course. is collecting data on the learning . Time is measured in school score required to demonstrate consecutive teachers in the top terms. distance from the same set of test ques- The headlines from Sanders' find. which also three in the bottom one-fifth was and a student masters half of the serves as the Planned Learning 50 percentile points(!!). to the specific subject lowest quintiles in Tennessee units and skills being examined. ÷ 0. Once again. If effectively a final exam. Sanders view. divided by time. A pre-test may also in- Figure 2 followingii: clude questions that represent prerequisite knowledge and 1. average is calculated for the class as factor. material satisfactorily by the end of 1991. "…more that student has a Learning Delta of can be done to improve education Step 3 . the beginning of the term. The performance difference used in the same way as Sanders dent's pre-test score and the between students having three uses it. • Pre-test questions are drawn a measure of speed. and social studies. 30 points ÷ 0.As instruction progresses. In Dr.A "pre-test" is administered ous year. as shown in Figure and to the teacher and classare ment gains with most of their 2. is their ini- one-fifth of the ranking and term is given a value of 100 points. half of the academic term.e. i. Each (and An Illustration) The "data discipline" associated with student’s test results are compared Learning Delta is quite simple. bi-weekly. reading. tested each year in math. students. edge they brought into the class centile points more than teachers Success with them. on the other hand. So if a course lasting one mastery in a subject. tial Learning Gap. and unless she/he calculated for each student and the teachers than by any other single increase their Learning Delta to 140.e.

"return" of an education reform pro- next month. William Sanders and TVAAS funded reading reform program A key difference between the to this sample education philan- (with some of the details disguised) Learning Delta and "value-added" thropy program and posing the in a state where roughly one out of measures is that Learning Delta is following questions: every five students in the target designed to be a frequent measure population fail to achieve "profi- that enables the teacher and school • What if we measured the Learning leaders to make immediate-term cient" status on the state's reading Deltas being produced by teachers course corrections in every class.9% 5.About Two Years Deltas that identify them for students' Behind (assume they score special assistance or motivating. Delta data would look like over a term and how that allows classrooms Having made this calculation. technology innovations. tests. . the The Base Case and teachers to be grouped into question remaining for the donor to performance quintiles. and illustra. The example I will use some of the findings from the work tion into a more advanced class.000 1. and it costs about reading on how a school performed $2 million per year to administer.523 still time. the students. specific classrooms next week or specialized curricular materials. "Shall we continue in any particular school is further Reading Standards Who Would. The "value-added" measure strate only "minimal" reading ability.Reading at Grade can adjust course while there is experiment" Level (at 50th %ile or above) 78. "Do you think $4000 per Measures of learning progress are student is a satisfactory num.4% 658 658 progress • Students who are moving much Total 7.000. as a result. and about one in ten demon- participating in a philanthropy-funded room. they is a "thought Group 1 . or all these gram or other innovation that is un- derway? Appendix A shows what Learning elements. is based on an actual philanthropy- of Dr.000 impact may be greater?" order to meet standards of compe- Success tency can vary greatly from one student to another. Page 3 • As interim Learning Deltas are Thought Figure 3 measured. highly effective teachers donor is a venture-philanthropist move their students along at a much Number of Students in Program 7.000 Students in % of Number of Students Population Students Below Grade progress is being made to the What follows Reform Program Level goal of mastery. learning around 10%ile) 9. ing data from the students on a measure that. the further question Number of Students Meeting 500 may be. is retro.About One Year Behind (assume they score • Students who are progressing too tion of why around 20%ile) 11. Although it can give a across 20 schools. with this program or do complicated because the curricular n't Have Otherwise something else where the "distance" that must be covered in Program Cost per Student. and if not.000 students monthly or quarterly basis? spective. education reform program by captur- as used by TVAAS is a once-a-year For the purposes of this example. Experiment and parents can see if adequate Out of 7. And the situation in results. the program covers 7. how would this less and less help with how to gramwhich may involve expendi- improve learning performance in information enable us to increase the tures on professional development.577 should be faster than the rest of the group used to drive will have high Learning Deltas that identify them for additional decision-making in education The experiment involves applying "enrichment" work or accelera.Program Cost Per Success ber?" In the event that the shown. as any particular Figure 1 shows a simple cost-per- performance in Tennessee were school year progresses it provides success calculation for this pro- duplicated elsewhere. $4. philanthropy. Group 2 .000 with an especially keen interest greater pace than average or below- Total Program Cost for One Year $2.000 average teachers. as Sanders has Figure 1 . important because. answer is. • If Sanders' findings about teacher in the prior year. teachers.7% 819 819 slowly will have low Learning data on Group 3 .

based on Sanders' findings in ($3. come their reading skills. only 40% of ($400.000. or b) what is most interesting.000) ($400. the 7. only 7% of the 7.000. Quintile 4 teachers immediately obvious. So Quintile 5 teachers performance quintile makes certain "BASIC" on the statewide compe. and only 20% of the ($2.000) -500 will be brought up to grade level.000.400 and the return can teachers in the bottom two quintiles we get roughly 1. and their students. as shown in the figure.500 students out of be thought of as 2. Page 4 school year. As a result Write-off Yield = 4. difference in results between quin- tiles 3 through 5.500 students is ceed in bringing students up to ness of lower quintile teachers.000) ($400.000) -1500 Teacher Performance Quintiles Tennessee. Specifically. For example. and the Quintile 4 yield is heterogeneous classroom these 4. we then assume that the 800+ to be distributed evenly across these analyzing results according to Group 2 students that scored quintiles.000) Group 2 students ($1. produce gains of 39%ile points in program improvement actions tency test are one year behind.000 1500 not enough to bring their students up to standard (see table $2.3 of these combina- $1.4 successes.000 who are functioning below $10.000) ($800. only Quintile 4 cesses.000 500 tions of teacher Number of Successes 132 effectiveness and Program Cost 132 328 $0 0 164 164 0 student deficit.4 Yield = 2.1. in at about $4.000 1000 in Appendix B for detail).3 successes-per- do not produce any improvement. thropy. therefore. and the yield achieved by Quintile 5 scored "MINIMAL" are two years Quintile 3 teachers produce gains of teachers is 7. The cost per are one and two years behind in reading tests and about 10% demon.000.000 "invested" by the philan- and there is a 39 percentile point grade-level.000.000) -1000 Group 3 students. strate only "minimal" reading ability.per- behind. The Base grade level by the end of the move more Group 2 and 3 students . Overall. students who met the statewide teacher effectiveness predicted from we are looking at score below reading standard who would not Sanders' findings for students who "proficient" status on the state's have otherwise). Figure 4 Quintile 3 teachers Base Case accomplish some Q 1+2 Q3 Q4 Q5 Total improvement.000. the remaining Group 3 students who produce gains of 26%ile points. rooms that produced no suc- levels of skill deficit.000. But over half of the $2 million student groups are distributed Under these conditions. when uniformly across all the teacher program cost was spent on class- combined with the children's varying quintiles.000 Program Cost Group 2 Students Group 3 Students students in this program are likely to be counted as Case graphic in Figure 4 shows the As shown in Figure 3. but $3. $10. What if actions were taken The implications of the Sanders and 5 teachers are likely to suc- to either a) increase the effective- findings for these 1. student-success will.000.e.1 Yield = 7.000) ($2. because 20% successes at the end of the year effect of the differing levels of of students in the reform program (i. 0 Q 1+2 0 Q3 Q4 Q5 Total then. which is assumed For the purposes of this illustration However. To represent a typical 13%ile points in their students.

line? yield has increased from 2. the program's yield has doubled from to rank classrooms/teachers in order critical first step is to identify 2.000 Cost per Student Success . the pro- out of the bottom quintile classes need to be better understood and gram's success rate would increase that are candidates for scrutiny and and into the top two quintiles. Even by 193%.3 to 3.000. Instead.3 to 4. if scenarios two and three identify and target the areas that ment moves Group 2 and 3 students are both put into effect.00 resource consumption. and identified. This timeassuming much harder because decisions improvement was seen.08 millionthe observations. opinions.000 2500 40.00 efficient causes. Otherwise. the cost per student-success translated into politics rather than and put those kids across the finish drops by 37%. What if additional funds program's success rate increases by back them up. In the third scenario we extend the forming teachers in order to increase Unlike impact studies. "who am I to 13%. course. depend on a chain of personal enough to meet the proficiency ally by 4% to $2. the cost per student-success going to call?" As was illustrated in If. we had in our possession has dropped by 50%. change. and judge- ments.000. the cost per student. Student Successes Students Successes per $10. and then file an Base Case Scenario 2 Scenario 3 Scenarios 2 + 3 accident report later. a jump of 176%! happened?"but rather. In addition. and the program's action.3 to 6.000. though.00 ment is very much like a steering wheel in a car.00 attached to and a part of a 1000 school's everyday processes.6 successes-per-$10. and Successes per $10.) of students to teachers and/or approachas in the examples enhance the skills of the low per.6 student-successes per of average actual Learning Delta per exceptions either based upon $10.3 successes. (See table in Appendix C class. we could modify the allocation specific criteria or using a normative for detail. like the Learning Delta. above.00 into them. there are borderline cases the next grade in order to finish the the absence of such data. Page 5 into quintile 4 and 5 classrooms? In program's reading interventions into the program's impact and yield. So because performance measure- 2000 30. they can only drive 0 0. (See table in Managing with the Learning Delta Appendix D for detail. the purpose of follows the kids and overall spending the program's yield would increase management information is not to remains at $2 million the pro. to be of value.00 immediate feedback on and 3000 control over speed. it needs to be 1500 20.) What-If Scenarios The point of management informa- As shown in Figure 5 (and in Ap. perfor- mance measures and manage- 4000 ment information. proactive management of this type is table in Appendix B) where some line students. the purpose of performance management information is not to produce Figure 5 scientifically precise analyses of Summary of Scenarios 4500 70. tion. is to The second scenario in this experi- pendix E). answer the question "what really gram's success rate almost doubles $10. teachers and school 10.00 must support real-time navigation Number of Successes / $ Cost per Success 3500 in the sense of providing near 50.per. 60. which without hard data to standard. and the Learning Delta data that allowed us the above thought experiment. Once exceptions have been assuming that program money success would drop by 64%. but not that spending increases proportion. (the yellow cells in the Base Case job that was started with the border. more often get were spent to extend the program 64%.00 500 leaders cannot steer around obstacles. though. from 2. then.

and which represent performance actions that improve perform- beyond what we thought we could ance? provement produce? 4. What are the standards that the subject of another paper. repeatable im- • Are they being or have they been met? provements to classroom and school-wide processes that raise . class. performance measure. Are we improving? objectives? sustainable. or a school 2.e. We know. We can express that pect educational proc- etc. parent communication. inspecting) classes that are already meeting 4. school leaders can • Are measures specific enough to enable positive identification of people and/or processes demonstrat- do two things: a) reduce the ing excellence? cost of observing (i. How do we measure actual system's performance are what we Learning Delta. Do our measures lead us to rial Approach to Education Im. by capturing and analyzing identification of the people and/or processes needing Learning Delta data to identify improvement? exceptions. is a first-level expect (under a particular reform dards? response to these questions no matter what other type of reform is being pursued. and we may be able support performance measures? to apply additional interventions tual performance against • Are they fact-based. stu- dent performance planning. In either • Are measures specific enough to enable positive case. Using this fact-base. but success of these other reforms. • Are there data capture mechanisms in place that too slowly. 3. model). as one top-level performance against these stan. What are the standards • Processes include student grouping. proach to education improvement 5. requirement as a Learning Delta esses to perform? • What are the reform-related or other standards that like "150 points per term. What is our track record • Is there a store of historical measurement data performance standards. What is our track record vis a vis A managerial approach to education these standards? improvement asks different ques. But tions: The key elements of a managerial the answers to the managerial model for education improvement is questions can be critical to the 1. classroom's. How do we measure ac. Page 6 Learning Delta Supports a Manage. tutoring. which need to be improved. some of the important considera- example. curriculum to meet the standard for reading that represent how we ex- planning. for Figure 6 example. a managerial ap- tions about schools than a curricu. For represent how we expect educa. schools will be able to make • Are there clearly articulated improvement goals and 5. instruction delivery. grade. that a rising fifth grader who reads at a third grade level Key Performance must progress more quickly Management Questions Selected Specifics than other fifth graders in order 1. lesson planning. a school's. At a minimum. which attributes of a tional processes to perform? tions are listed in Figure 6. Do our measures allow us • Are measures captured and reported frequently in use were never implemented enough and with adequate turn-around to enable cor- to improve performance? rective action? in certain classrooms. competence by the end of fifth work/homework assignments. Or we may find teaching? that the original interventions prescribed by the reform model 3. etc." At correspond to each of the educational processes above? Are they widely publicized and/or formally an interim point in the school adopted? year we can identify students or classrooms who are progressing 2. Are we improving? lum-based reform or one based requires answers to five key ques- upon community outreach. testing. and timely? to accelerate them to a success- these standards? • Are they efficient to use and non-disruptive to ful conclusion. objective. and b) vis a vis these standards? allowing for trend analysis? build their fact-base regarding • What are the performance trends versus key what works and what doesn't in educational processes and standards? practice.

if a student scored 10 and learning progress improved numbers! They are critical. Page 7 performance for the future. Equity (ROE) and Price-Earnings learning progress. like that she/he is not in the right Learning Delta. The case for Learning Delta is the to achieve learning progress of same: don’t mistake Learning Delta 90 points in one term. both of which require that studentand that student's substantially above or below the an earnings number. we'd be in trouble achieved. like graduation change it. it is un. classroom that was grouped by age. had to compare the performance of ment or curriculum. that should be attended to. Such a school would be different from it is almost useless if used in isola- conventional schools in several ways: These examples show how the tion. or adequate Simply by using pre-tests and these disparate companies were raw learning progress will not be Learning Delta measures to proac. during a school year rather than are also just the beginning. Some. likely that student would be able one or two years later. (…more trouble). or waiting for someone to clothing retailer which are them- points per term of learning prog- ress. And it is up to Delta as a measure behaves a lot like And there are also other important the teacher and the school to an earnings number for a commer. but . Because the student's help them understand what is going selves performance measuresare maximum gain in that class is on. As shown in Figure 7. earnings numbers. But the key teacher would not be able to class average in attainment are either drivers of ROE for a nationwide meet the school's standard of 60 waiting for the class to catch up to them. and there are underlying thing needs to change if she/he success of a classroom. Learning should be measured and managed. In fact. or drivers of Learning Delta that is to meet the school's learning education reform program. is not and companies. progress standard.iii Imagine a school that has as its standard that every student will achieve a Learning Delta of 60 points per term. we can improve the state of education if we move beyond the "random acts of progress" too often seen in K-12 schools to data-driven decision-making that focuses on actual learning progress and its underlying drivers. top-level measures. cial company. student very different from those for finan- only 20 points. tively manage learning progress. In other words. cannot be the single measure of than that. it is up to the wait-states are the mirror image of cial services companies or car teacher and the school to make a the distribution of attainments in a manufacturers. However. but they out of 100 on a pre-test. indicator of a company's health. Maximizing Learning Progress by Reducing Student Wait-States Another important reason to adopt the Learning Delta as an interim performance measure is that it provides a better means than annual standardized tests for ensuring that all students will make adequate learning progress. though. Publicly traded companies get problem of student "wait-states" in • If an advanced student were to compared on ratios like Return On conventional classrooms retard score 80 out of 100 on a pre-test. If as investors all we change to that student's place. Limitations for the only measure of a school's that student's low score shows instructional success. Students who are ratio (P/E). Earnings is a key rates. we've student wait-states will be reduced still got to have accurate earnings • Similarly. school. Schools. are more complicated classroom to begin with.

" i. up if schools would only do what searched in Samuel Casey Carter's the rest of the U. the "Web-enhanced campus" preparatory schools. we as teamwork.e. high perfor. students. and acknowledged stan- yearsin spite of the tripling of per dards. As a result. rooms deliver instruction. students. even very effective manual processes actually sitting in a lecture hall. classwork. scale at work and classic "locus of major distributed-learning platforms control" problems that stand in the like Blackboard. Manag- learning progress is the very thing that feedback on key performance ing the effectiveness of large num- schools are supposed to produce. On average. collect and correct • They create new information teachers and schools because homework. campus locations. Conventional schools and class- need Learning Delta as a measure of havior. Accordingly. presented with: high-performance pupil expendituresiv. Four or five years ago the excite- Learning Deltas. and administer tests in flows that provide same-day an entirely manual fashion. post the learning progress produced by assignments. as well break down under high volumes. WebCT. For example: leaders "managing-by-walking. became an object of enthusiasm in KIPP's students make 6 year's There are distinct diseconomies of higher education. culture. pub- unimproved over the last forty critics miss the opportunity we are lished. and classes. schools because of the special learning. and so ordinary schools. tools and techniques that duplicate "No Excuses" results using lectern.S. In the ing for the management of tens of educational institutions began to space of four years KIPP turns out thousands of students rather than discover the benefits to on-campus ninth graders operating at a ninth hundreds?" And they conclude that students of on-line courses. homework. For example. growth: get digitized. And it is as a result that We can identify high performance high performance schooling is Digitize what? schools because they are the ones negatively correlated with school that produce greater than average size. Without attributes. it is hard to imagine support teachers. practices. They ask. Page 8 But. the surrounding managed by leaders who are em. function at a third grade level. and so grade level--as defined by exclusive there is none. economy did Part 3: Putting the Learning the book No Excusesvis based starting 20 years ago as a response to Delta into Practice largely on "management-by-walking. the elimination of time niques and systems. the academic progress in 4 year's time. perhaps tions and bringing responses to it is because we have been muddling By failing to separate the negative bear so that the exceptions are through without such a measure that effects of size from the positive brought into compliance with schools have remained essentially effects of good management. around". These feedback loops bers of such classrooms eventually such a measure. attempting to line analogues to the blackboard. and be. more principals and instructional debate about "e-Learning" has powered to do so. small size and space as barriers to taking KIPP Academy in Houston enrolls among them. conditions involved. And they are well school settings would require many forth. and testing. the the school's predefined. the classroom discussion. focused on distance learning and of good leaders providing sound vant to the problems of large how it enabled "anywhere. Simply put. as with company earnings. or whether roles and standards for teachers. and Lotus KIPPand other high performance way of scaling up "No Excuses" LearningSpace tried to provide on- schools like it employ methods. are different from those found in "No Excuses" techniques in large the student's raised hand. and parents to play But a well understood lesson from listening to a lecture is as rich an within well-defined processes outside the education sector is that experience on-line as it would be covering attendance. face-to-face variety. school management can be scaled- But high performanceas re. And as with other Critics of the "No Excuses" research ment about "cyber-schools" and the human enterprises. proved. and becomes infeasible as a simple result how teachers and schools can be parents in their roles by quickly of increasing numbers of students systematically managed and im- identifying exception condi. . mostly focused on whether an on- around" than school systems are line class discussion is as good as the • They explicitly define common able or willing to pay for. On the other hand. the practices. Subsequently. often claim that the schools studied "virtual classroom" was mostly mance schooling is largely a matter are statistical outliers that are irrele. anytime management using effective tech. "What is the courses or earning degrees from off- fifth graders who for the most part lesson of high performance school.

techniques. correlate learning results to teaching components are required: room to accomplish instrumenta. Eventually. performance. Should not simply impose a new clerical process. these questions may get answered . instructional leaders. anywhere. and then scrutinize • A testing process producing performance improvement through classroom methods. but without the disrup- each student. Most • Should automate the testing process from start-to-finish in order to relieve teachers of manual tasks and free up new time Learning Gap. though. Capturing data like 3. and school systems 1. Figure 7 ent from the "virtual classroom" Key Attributes of a Learning Delta Management System discussion described above. schools. Digitize How? being tested. age classroom performance and improve results. and digitiz. and who trated above. parents. classroom workflowswhich connect the teacher to the students. elements--new or old--work very • A reporting and analysis facility ence can be shared. practices. Digitizing the class. Least • Curriculum-Neutral  Accommodates whatever curriculum is already in use by a school. teachers and schools. Data capture and feedback mechanisms are user- digitized in K-12 education are the friendly. and which schools. As a result. tools that track individual stu- teachers needing help to im. The current state in our to the curriculum. actual efficient for teaching. is that very little produces measurements that a) identify the Master Teachers information comes out of class. analysis-driven facilitation possible. processes. mapped) relationship facilities and "drill-down" analysis to: actions. which are not. and classrooms are on schedule and to classes. schools could begin to the Learning Delta alone. Class- rooms. school and district leaders improve outcomes for the students can exercise anytime. or monthly basis which dent scores and maintain links prove their performance. planned vs. or which stu. model. tered. ing the classroom is the critical have fallen behind. teaching be replaced with computer-based instruction. which–like others– intrusive • Pedagogy-Neutral  Does not require that classroom-based are comprised of methods. Does not impose a new one. tools and scales (scores) that have a de- the regular use of on-line reporting techniques to identify improvement fined (i. well or very poorly. are enterprises. useful performance necessary and desirable because it analyses on demand in order to increase teacher and class- enables instrumentation of the room effectiveness. Page 9 The point I wish to make here about "digitizing" the classroom is differ. What needs to be routines. and the like is both • Should provide teachers with detailed. 2. who have pulled ahead. that supports the type of data- dents are functioning at grade-level. administrators. tools. driven decision-making illus- That is a fitting role for school leaders and managers. and the schoolso that costly • Minimizes requirement for network and/or centralized applica- basic data on what is getting done tion support though use of application-service-provider (ASP) and what isn't can be used to man. weekly. level. "line-of-sight" management of bricks-and-mortar classrooms in With digital mechanisms for meas- much the same way as with virtual To put into practice data-driven uring the actual learning progress of performance management based on classrooms. Learning Delta. knowl. but by then it is too late to mation. rank classrooms tion would enable broad-based accordingly. extend above and below grade who are the exemplars of high rooms once the teachers and stu. Least • Allows use of hardware with lowest total cost of ownership: handheld computers like Palm OS or Pocket PC devices. three key tion of trying to deliver instruction in a new way. which curriculum and c) bring the two groups together so that expertise and experi. dents have gone in. • Ease-of-Use  Fits well with and supports classroom edge and people. classroom rather than virtualiza- tion. • A database and data capture b) identify the classrooms and and parents cannot tell on a daily. performance that should be emulated. first step to making that type of when high-stakes tests are adminis- Using digitized performance infor.e.

but they are made available. the minimum infrastructure for data. For example. a common goals. "people factors" are unlike smaller class size and in- chasedalong with substantial often more decisive than a system's creased professional develop- amounts of "course-ware" and design or features in determining the success or failure of a project. both among a school's teaching they go about their jobs in order staff and the groups in charge of the The broad-based application of to use the system successfully? environment. though. and ment system because they don't have parents with the usual and cus. if obtain performance tracking and tempting to implement such a it's not in the interest of these management functionality in the system will likely fail. Learning Delta measures in schools • Environment . the training or the release time to get tomary benefits of automation the training. staff ing well. groups that control the environ. and curriculum manage.Do the intended users the clear potential to create winners attributes of vendor offerings of a system have the training. Three in some ways and certainly benefit be appropriate or useable by class- room teachers. are tional methods and materials. can all attest. In addition. control of the many schools having a system with System can only enable perform. parents and communities believe that doing what it takes have long wanted tools to measure easier-to-implement Learning to make the system successful Delta Management System be our children's absolute and relative will be recognized and rewarded educational progress. Taken together. or anyone at. sions. If starting point for "getting digital" and serious demand for meaningful either one of these groups is op- and starting to practice data-driven decision-making. If a school's teaching staff are opposed to a learning manage- ment system because they don't trust administrators/management to use . system willing to change how ment. ability and willingness. K-12 mentwhich probably benefit kids embedded curriculum that may not schools are no exception. then they will not. to develop and implement. will most probably fail. Page 10 These components comprise a Change Management and People the data fairly or constructively. these • Ability . To be broadly as a positive achievement by the applicable in divers types of schools not the ones being asked to change. and so it is harder to win • Willingness . and losers. high-impact tion is already in place: a concerted unions and school administration. And for Learning Delta Management In the case of schools. It increases the riskiness represent substantial barriers to time. Similarly.Do system users requires that a more focused and Of course. But for a school to performance data from schools. between two groups: the teachers' represent a logical. if a cost of ownership. such dimensions always need to be the adults performance measure- systems require amounts of com. it has to come first. • Accommodate rather than It turns out that the first two dimen- disrupt the established instruc. That posed to adopting a new system. heavily conditioned by the third: • Be designed for very low total environment. groups for learning performance educational software market often management practices to take root requires that an entire instructional As experienced system implementers and spread. the environment needs rather than creating additional to change to accommodate those work (see Figure 7). So management system be pur. students. sion. addressed: ment as a means of managing puter hardware that many schools schools is problematic because it has can ill afford. needs.Are the users of a votes on the "willingness" dimen- training. environment is shared mostly just these components would ance improvement if the real solu. and school's teaching staff are unable to implement a new learning manage- • Provide teachers. environment needs to change to Factors driven performance improvement in address those fears and establish schools that would relatively simple Most importantly. and resources required to of being a teacher or an administra- adoption by schools that compound use it successfully? tor in a school that is not perform- the usual issues of cost. And they do not control the envi- and communities it should: ment? ronment.

designing tools that give those barriers to the success of data-driven people the information they need to make good instructional decisions.including the recruitment. And so paying Because improving the effectiveness profits to shareholders is in some of teachers can improve education sense justified if it makes possible a more than by any other single factor. (emphasis added) change may be the New Schools Venture Funda leading venture philanthropywho in their recent systemic improvement in K-12 press release point to a shift away One particularly good chance to schools is to transform the class- from schools being compliance. Box 398 ! Litchfield. demonstrably better education for our greatest opportunity to achieve our kids. and supporting practices learning management within K-12 that promote and reinforce success.The ventures in the Performance Accelerator Fund will focus on developing people who know how to lead and teach in a performance-based environment.and performance tools -.567.harvard. . ures like the Learning Delta to program or EMO while the market tion reforms are in an excellent manage and improve school per- for Learning Delta management position to demonstrate the power formance as a regular part of a software and services develops. . May 12 /PRNewswire/ -. preparation and support of high-quality teachers and leaders -. Page 11 PRESS RELEASE (Excerpt) Conclusion PALO ALTO." they schools cannot do). candidate solution for the systemic because they are for-profit and so improvement of education in this get criticized for diverting money country (something that No Excuses "away from the children. ance-based cultures focused on "managerial model" of school The first step is to digitize the achievement. targeted at $20 million and designed to invest in ven- There are signs of positive change. . LLC ! P. . schools. has ever been. tures that enhance the capacity of school systems to produce high levels of student achieve- however.including information systems and assessment tools that enable teachers and leaders to make data-based decisions about their students' in- One bellwether of this positive struction. For more information contact: Gregory D. It needs to Schools and Chancellor Beacon have shown its effectiveness on a Academies are also in a position to substantial scale in order to be a make a positive impact. fund and build ventures that address education. human capital -. Left Behind Act and other factors. In fact. . we must ensure that they have the tools they the secular climate for learning need to be successful. per- formance-based cultures focused on high student achievement." said Kim Smith. a demonstration of measures like Learning Delta repre.8242 ! e-mail: greg_luther_ab82@post. co-founder and CEO of NewSchools Venture Fund. But ideology and labor relations remain very high . CT 06759 tel: 860. Founder and Principal ! High Delta Learning. The Fund will identify. Delta management is. performance measurement in State and federal education policies are increasingly pushing for school systems to shift from classrooms is probably better than it compliance-driven organizations -.O. Calif. be convincing. the managerial model needs to span sent. "As districts work to increase student .governed by rules. should be strongly motivated to use Learning Delta measures to en- sure—and then later prove—that * * * they are doing a better job than traditional schools. Because of the No Child ment. To for positive change that performance school's weekly routine. and organizations (EMOs) like Edison even school systems. regulations and court orders -. .NewSchools Venture Fund announced the launch of its new Performance Accelerator Fund. Luther. improvement to be incubated inside classroom and to start using meas- a substantial education reform Venture philanthropy-based educa. demonstrate the value of Learning room into the atomic unit of data- driven organizations to perform. . Education management multiple classrooms. is for the driven performance management.. .

Smith Mastery Score: 80 Learning Delta Standard: 40 points per term Plan vs. Page 11 Appendix A EXAMPLE OF LEARNING DELTA DATA .Classroom A Teacher Name: Mr.Dec Student 100 39 41 47 24 61 34 82 43 2 Student 101 21 59 28 21 33 19 55 34 -25 Student 102 24 56 36 37 55 47 80 56 0 Student 103 55 25 57 6 72 25 95 40 15 Student 104 35 45 53 56 58 35 67 33 -13 Class Average Test Score 35 44 56 76 Class Average Learning Delta 45 29 64 41 Class needs to Going too Speeding Finished a achieve this slowly! to catch little under average "speed" up plan Teacher's Learning Delta Result for Term 41 % of Students Meeting Learning Delta Standard 40% Teacher Inter-ranking Average Class Name Rank Learning Delta Quintile Hume 1 72 Q1 Hobbes 2 68 James 3 66 Locke 4 60 Q2 Descarte 5 58 Smith 6 41 Weber 7 40 Q3 Berkeley 8 39 Russell 9 38 Machievelli 10 32 Q4 Mosca 11 22 Michels 12 18 Rousseau 13 16 Q5 Nozick 14 15 Rawls 15 15 . Actual Timeframe: Aug Sep Oct Dec Learning Delta Learning Gap / Interim Interim Final Planned Learning Learning Learning Learning Student Pre-Test Delta Test 2 Delta Test 3 Delta Final Test Delta Aug .

523 GROUP 2 .400 1.4 2.3 .Reading at Grade Level (at 50th %ile or above) 78.4% NONE (@10%ile) NONE (@23%ile) NONE (@36%ile) 132 132 20% Total 0 0 164 295 459 7% Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Program ROI Teachers Teachers Teachers Teachers Total All-In Program Cost/Students Participating in Program $286 Program Cost per Teacher Quintile $ 800.400 7. Page 12 Appendix B Figure 4 Base Case Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Teachers Teachers Teachers Teachers Total Students % Impacted % of 7000 Third Grade Children Population Number of Students in BRI Program GROUP 1 .400 1.Two Years Behind (between 10th %ile and below) 9. .1 Teacher Quintile write-off write-off $ 2.000.000 $ 400.One Year Behind (between 10th and 25th %ile) 11.000 $2.000 Predicted Student Improvement Based on Sanders' Findings (%ile Points) 0 13 26 39 % of Program Impact Population Predicted Number of Additional Students Meeting Reading Standard GROUP 1 .000 4.800 1.4% 263 132 132 132 658 Total 2.Two Years Behind (assume at 10th %ile) 9.One Year Behind (assume at 20th %ile) 11.000 Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade . 164 295 459 All-In Program Cost/Student Meeting Standard -.355 Students Successes per $10.Reading at Grade Level (at 50th %ile or above) 78.7% 328 164 164 164 819 GROUP 3 .000 $ 400.354 $4.000 $ 400.442 $ 1.9% already @ std already @ std already @ std already @ std 0 0% GROUP 2 .9% 2209 1105 1105 1105 5.7% NONE (@20%ile) NONE (@33%ile) 164 164 328 40% GROUP 3 .

000 Predicted Student Improvement Based on Sanders' Findings (%ile Points) 0 13 26 39 % of Program Impact Population Predicted Number of Additional Students Meeting Reading Standard GROUP 1 .000.178 -50% Students Successes per $10. .6 .7% --. NONE (@23%ile) NONE (@36%ile) 263 263 40% Total 0 0 328 591 918 13% 100.000 $ Teacher Quintile --. $ 400.Two Years Behind (between 10th %ile and below) 9.000 $ 800.One Year Behind (assume at 20th %ile) 11.Reading at Grade Level (at 50th %ile or above) 78.000 $2. 328 591 918 100% All-In Program Cost/Student Meeting Standard -.523 GROUP 2 . Program ROI Teachers Teachers Teachers Teachers Total Base Case All-In Program Cost/Students Participating in Program $286 Program Cost per Teacher Quintile $ .209 1.000 0% Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade .Two Years Behind (assume at 10th %ile) 9.5 Case 2 .442 $ 1.000 4. Page 13 Appendix C Figure 4.354 $2.695 7.695 1. Teachers Teachers Teachers Teachers Total Students % Impacted Base Case % of 7000 Third Grade Children Population Number of Students in BRI Program GROUP 1 .4% --. 164 328 328 819 GROUP 3 .One Year Behind (between 10th and 25th %ile) 11. write-off $ 2.9% 2209 Shift Quintile 1105 1 & 2 students 1105 to Quintile 41105 & 5 teachers. NONE (@33%ile) 328 328 655 80% GROUP 3 .7% --.9% already @ std already @ std already @ std already @ std 0 0% GROUP 2 .1 7.Reading at Grade Level (at 50th %ile or above) 78.4% --. 132 263 263 658 Total 2.0% Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Change vs. 5.400 1.4 4.Shift Quintile 1 & 2 Upward Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Increase vs.

3% Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Change vs.9% 2209 Extend the 1105 program into 4th1105 Grade for borderline 1105 students.4% 263 132 132 132 658 Total 2.481 $1. Teachers Teachers Teachers Teachers Total Students % Impacted Base Case % of 7000 Third Grade Children Population Number of Students in BRI Program GROUP 1 .Extend Program into 4th Grade Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Change vs.Reading at Grade Level (at 50th %ile or above) 78.7% NONE (@20%ile) 164 164 164 491 60% GROUP 3 .800 1.762 -37% Students Successes per $10.000 $2.39 3.75 7.800 $ Teacher Quintile write-off $2.600 $ 400.62 .523 GROUP 2 .Two Years Behind (between 10th %ile and below) 9.6 Case 3 .One Year Behind (assume at 20th %ile) 11.One Year Behind (between 10th and 25th %ile) 11.728 $1.Two Years Behind (assume at 10th %ile) 9.400 1.000 $ 446.400 7.600 Program Cost per Teacher Quintile $ 800.9% already @ std already @ std already @ std already @ std 0 0% GROUP 2 . Page 14 Appendix D Figure 4.4% NONE (@10%ile) NONE (@23%ile) 132 132 263 40% Total 0 164 295 295 755 11% 64.7% 328 164 164 164 819 GROUP 3 .000 Predicted Student Improvement Based on Sanders' Findings (%ile Points) 0 13 26 39 % of Program Impact Population Predicted Number of Additional Students Meeting Reading Standard GROUP 1 .67 6.Reading at Grade Level (at 50th %ile or above) 78. Program ROI Teachers Teachers Teachers Teachers Total Base Case All-In Program Cost/Students Participating in Program $298 Incremental Program Cost in 4th Grade (Students * Cost per Student) $46.354 $2.400 4% Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade .5.000 3.084. 164 295 295 755 64% All-In Program Cost/Student Meeting Standard -.400 1.800 $ 437.

Two Years Behind (assume at 10th %ile) 9.0% Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Change vs.9% 2209 1105 1105 1105 5.000 6% Add'l Students Reading @ 3rd Grade Level by End of 3rd Grade --.695 1.7% NONE (@20%ile) 164 328 328 819 100% GROUP 3 .67 6.75 7.4% NONE (@10%ile) NONE (@23%ile) 263 263 526 80% Total 0 164 591 591 1.7% --. Program ROI Teachers Teachers Teachers Teachers Total Base Case All-In Program Cost/Students Participating in Program $303 Incremental Program Cost in 4th Grade (Students * Cost per Student) $46.One Year Behind (assume at 20th %ile) 11. $ 446. $2. 164 328 328 819 GROUP 3 .Reading at Grade Level Shift Quintile 1 & 2 students to Quintile 4 & 5 teachers.000 3.577 -64% Students Successes per $10.800 $75.200 $ 122.209 1. Teachers Teachers Teachers Teachers Total Students % Impacted Base Case % of 7000 Third Grade Children Population Number of Students in BRI Program GROUP 1 .800 $ 875.523 GROUP 2 .Reading at Grade Level (at 50th %ile or above) 78.122.Two Years Behind (between 10th %ile and below) 9.345 19% 193.000 Program Cost per Teacher Quintile $ .728 $1. 132 263 263 658 Total 2.39 6.695 Teacher Quintile --.481 $1.Scenario 2 -PLUS.400 1. Predicted Student Improvement Based on Sanders' Findings (%ile Points) 0 13 26 39 % of Program Impact Population Predicted Number of Additional Students Meeting Reading Standard GROUP 1 .4% --.One Year Behind (between 10th and 25th %ile) 11.34 176% . (at 50th %ile or above) 78.Scenario 3 Quintiles 1 & 2 Quintile 3 Quintile 4 Quintile 5 Change vs.000 Extend the program into 4th Grade for borderline students.200 $ 800. 164 591 591 1345 193% All-In Program Cost/Student Meeting Standard -.9% already @ std already @ std already @ std already @ std 0 0% GROUP 2 .354 $1.000 $2. Page 15 Appendix E Figure 5 Case 4 .

S. p. “Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation. 16 ii See the following: Wright. Page 16 End Notes: i Finn. the student-to-teacher ratio dropped from 26:1 to 17:1. a key component of public education and the current "accountability" discussion.” Journal of Personnel Evaluation in Education. During the same period. Vol. 1997.896.638 to $7. Fordham Foundation.” Research Progress Report. Horn. p. Paul. Samuel Casey. Paul. University of Tennessee Value-Added Research and Assessment Center. per-pupil expenditures in U.S." (Source: "MAKING IT COUNT: A Guide to High-Impact Education Philanthropy. S. Fordham Foundation) v Carter. April 1. High-Poverty Schools. test scores by themselves are not a sufficient management tool because they measure attainment levels rather than the extent to which a school produced a change in attainment levels. Chester E.” Journal of Personnel Evaluation in Education. Sanders. A consequence of managing schools according to test scores appears to be that students who already meet or exceed test standards for their age group ten to get less attention than other students.11. 1996. 1997 Sanders. Sanders. Heritage Foundation.'” See: Wright. Sanders reported that to a disproportionate degree "high-scoring students were found to make somewhat lower gains than average and lower-scoring students… This finding indicates that it cannot be assumed that higher-achieving students will 'make it on their own. Rivers.11. September 2001. Vol. does not by itself provide the information needed to manage school performance. As Sanders and the state of Tennessee have shown. Sandra P. and June C. Horn. “Teacher and Classroom Context Effects on Student Achievement: Implications for Teacher Evaluation." The Thomas B. and Kelly Amis. and William L. public schools grew (in constant 1998-99 dollars) from $2. iii Standardized testing. . 14 iv "From 1960 to 1999." Thomas B. and William L. 2000. William L. Sandra P. "Making it Count: A Guide to High-Impact Education Philanthropy. No Excuses: Lessons from 21 High-Performing. “Cumulative and Residual Effects of Teachers on Future Student Academic Achievement.