ROTTEN APPLES: AN INVESTIGATION OF THEPREVALENCE AND PREDICTORS OF TEACHERCHEATING*B
RIAN
A. J
ACOB AND
S
TEVEN
D. L
EVITT
We develop an algorithm for detecting teacher cheating that combines infor-mation on unexpected test score uctuations and suspicious patterns of answersfor students in a classroom. Using data from the Chicago public schools, weestimate that serious cases of teacher or administrator cheating on standardizedtests occur in a minimum of 4–5 percent of elementary school classrooms annu-ally. The observed frequency of cheating appears to respond strongly to relativelyminor changes in incentives. Our results highlight the fact that high-poweredincentive systems, especiallythose with bright line rules, may induce unexpectedbehavioral distortions such as cheating. Statistical analysis, however, may pro- vide a means of detecting illicit acts, despite the best attempts of perpetrators tokeep them clandestine.
I. I
NTRODUCTION
High-powered incentive schemes are designed to align thebehavior of agents with the interests of the principal implement-ing the system. A shortcoming of such schemes, however, is thatthey are likely to induce behavior distortions along other dimen-sions as agents seek to game the rules (see, for instance, Holm-strom and Milgrom [1991] and Baker [1992]). The distortionsmay be particularly pronounced in systems with bright line rules[Glaeser and Shleifer 2001]. It may be impossible to anticipatethe many ways in which a particular incentive scheme may begamed.Test-based accountability systems in education provide anexcellent example of the costs and benets of high-powered in-centive schemes. In an effort to improve student achievement, anumber of states and districts have recently implemented pro-grams that use student test scores to punish or reward schools.
* We would like to thank Suzanne Cooper, Mark Duggan, Susan Dynarski, Arne Duncan, Michael Greenstone, James Heckman, Lars Lefgren, two anony-mous referees, the editor, Edward Glaeser, and seminar participants too numer-ous to mention for helpful comments and discussions. We also thank Arne Dun-can, Philip Hansen, Carol Perlman, and Jessie Qualles of the Chicago publicschools for their help and cooperation on the project. Financial support wasprovided by the National Science Foundation and the Sloan Foundation. Allremaining errors are our own. Addresses: Brian Jacob, Kennedy School of Gov-ernment, Harvard University, 79 JFK Street, Cambridge, MA 02138; StevenLevitt, Department of Economics, University of Chicago, 1126 E. 59th Street,Chicago, IL 60637.
©
2003 by the President and Fellows of Harvard College and the Massachusetts Institute of Technology.
The Quarterly Journal of Economics,
August 2003
843
Recent federal legislation institutionalizes this practice, requir-ing states to test elementary students each year, rate schools onthe basis of student performance,and intervenein schoolsthat donot make sufcient improvement.
1
Several prior studies suggestthat such accountability policies may be effective at raising stu-dent achievement [Richards and Sheu 1992; Grissmer, Flanagan,et al. 2000; Deere and Strayer 2001; Jacob 2002; Carnoy and Loeb2002; Hanushek and Raymond 2002]. At the same time, however,researchers have documented instances of manipulation, includ-ing documented shifts away from nontested areas or “teaching tothe test”[Kleinet al. 2002; Jacob 2002], and increasing placementin special education [Jacob 2002; Figlio and Getzler 2002; Cullenand Reback 2002].In this paper we explore a very different mechanism forinating test scores:outright cheating on the part of teachersandadministrators.
2
As incentives for high test scores increase, un-scrupulous teachers may be more likely to engage in a range of illicit activities, including changing student responses on answersheets, providing correct answers to students, or obtaining copiesof an exam illegitimately prior to the test date and teachingstudents using knowledge of the precise exam questions.
3
Whilesuch allegations may seem far-fetched, documented cases of cheating have recently been uncovered in California [May 2000],Massachusetts [Marcus 2000], New York [Loughran and Comis-key 1999], Texas [Kolker 1999], and Great Britain [Hofkins 1995;Tysome 1994].There has been very little previous empirical analysis of teacher cheating.
4
The few studies that do exist involve investi-
1. The federal legislation,No Child Left Behind,was passed in 2001. Prior tothis legislation, virtually every state had linked test-score outcomes to schoolfunding or required students to pass an exit examinationto graduate high school.In the state of California, a policy providing for merit pay bonuses of as much as$25,000 per teacher in schools with large test score gains was recently put intoplace.2. Hereinafter,we uses the phrase “teacher cheating” to encompass cheatingdone by either teachers or administrators.3. We haveno way ofknowing whetherthe patterns we observearise becausea teacher explicitly alters students’ answer sheets, directly provides answers tostudents during a test, or perhaps makes test materials available to students inadvance of the exam (for instance, by teaching a reading passage that is on thetest). If we had access to the actual exams, it might be possible to distinguishbetween these scenarios through an analysis of erasure patterns.4. In contrast, there is a well-developed statistics literature for identifyingwhether one student has copied answers from another student [Wollack 1997;Holland 1996; Frary 1993; Bellezza and Bellezza 1989; Frary, Tideman, andWatts 1977; Angoff 1974]. These methods involve the identication of unusual
844
QUARTERLY JOURNAL OF ECONOMICS
gations of specic instances of cheating and generally rely on theanalysis of erasure patterns and the controlled retesting of stu-dents.
5
While this earlier research provides convincing evidenceof isolated cheating incidents, our paper represents the rst sys-tematic attempt to (1) identify the overall prevalence of teachercheating empirically and (2) analyze the factors that predictcheating. To address these questions, we use detailed adminis-trative data from the Chicago public schools (CPS) that includesthe question-by-question answers given by every student ingrades 3 to 8 who took the Iowa Test of Basic Skills (ITBS) from1993 to 2000.
6
In addition to the test responses, we also haveaccess to each student’s full academic record, including past testscores, the school and room to which a student was assigned, andextensive demographic and socioeconomic characteristics.Our approach to detecting classroom cheating combinestwo types of indicators: unexpected test score uctuations andunusual patterns of answers for students within a classroom.Teacher cheating increases the likelihood that students in aclassroom will experience large, unexpected increases in testscores one year, followed by very small test score gains (or evendeclines) the following year. Teacher cheating, especially if done in an unsophisticated manner, is also likely to leavetell-tale signs in the form of blocks of identical answers, un-usual patterns of correlations across student answers withinthe classroom, or unusual response patterns within a student’sexam (e.g., a student who answers a number of very difcult
patterns of agreement in student responses and, for the most part, are onlyeffective in identifying the most egregious cases of copying.5. In the mid-eighties, Perlman [1985] investigated suspected cheating in anumber of Chicago public schools (CPS). The study included 23 suspect schools—identied on the basis of a high percentage of erasures, unusual patterns of scoreincreases,unnecessarilylarge orders of blankanswer sheets for the ITBS and tipsto the CPS Ofce of Research—along with 17 comparison schools. When a secondform of the test was administered to the 40 schools under more controlled condi-tions, the suspect schools did much worse than the comparison schools. Ananalysis of several dozen Los Angeles schools where the percentage of erasuresand changed answers was unusually high revealed evidence of teacher cheating[Aiken 1991]. One of the most highly publicized cheating scandals involved Strat-eld elementary, an award-winning school in Connecticut. In 1996 the rm thatdevelopedand scoredthe examfoundthat the rate of erasuresatStrateldwas upto ve times greater than other schools in the same district and that 89 percent of erasures at Strateld were from an incorrect to a correct response. Subsequentretesting resulted in signicantly lower scores [Lindsay 1996].6. We do not, however, have access to the actual test forms that studentslled out so we are unable to analyze these tests for evidence of suspiciouspatterns of erasures.
845
ROTTEN APPLES
Search History:
Result 00 of 00
00 results for result for
p.
(2003) Rotten Apples - An Investigation of the Prevalence and Predictors of Teacher Cheating
113
Reads
0
Readcasts
0
Embed Views
More From This User
Notes
Load more
