ARTICLE IN PRESS

Economics of Education Review 25 (2006) 327–333 www.elsevier.com/locate/econedurev

Measurement error, education production and data envelopment analysis
John RuggieroÃ
Department of Economics and Finance, University of Dayton, Dayton, OH 45469-2251, USA Received 30 September 2004; accepted 18 March 2005

Abstract Data Envelopment Analysis has become a popular tool for evaluating the efficiency of decision making units. The nonparametric approach has been widely applied to educational production. The approach is, however, deterministic and leads to biased estimates of performance in the presence of measurement error. Numerous simulation studies confirm the effect that measurement error has on cross-sectional deterministic models of efficiency. It is also known that panel data models have the ability to smooth out measurement error, leading to more reliable efficiency estimates. In this paper, we exploit known properties of educational production to show that aggregation can also have a smoothing effect on production with measurement error, suggesting that efficiency analyses are more reliable than previously believed. r 2005 Published by Elsevier Ltd.
JEL Classification: I21 Keywords: Educational production; Data envelopment analysis

1. Introduction Data Envelopment Analysis (DEA) is a nonparametric mathematical programming approach to the measurement of efficiency that was introduced in the operations research literature by Charnes, Cooper, and Rhodes (1978) and Banker, Charnes, and Cooper (1984). Using linear programming, an observed decision making unit (DMU) is evaluated relative to the production frontier, which consists of combinations of observed production possibilities using minimal assumptions. The primary advantage of the approach is the ability to handle multiple inputs and multiple outputs, particularly in the case when input prices are unavailÃTel.: +1 937 229 2258; fax: +1 937 229 2477.

E-mail address: ruggiero@notes.udayton.edu. 0272-7757/$ - see front matter r 2005 Published by Elsevier Ltd. doi:10.1016/j.econedurev.2005.03.003

able. The linear programming approach has withstood the test of time and has become an acceptable approach for efficiency evaluation. One important application of DEA is to the analysis of educational production. Many states have undergone legal challenges because school districts are not providing educational services efficiently and outcomes are not adequate. Reform has moved away from traditional issues like equity to adequacy and efficiency. The important policy implication is that school districts need to spend their money more wisely and increase their outcomes to acceptable levels. One popular technique that has been used for measuring efficiency in education is DEA. Research using DEA to measure performance of educational production in the United States include Bessent, Bessent, Kennington, and Reagan (1982), Fare, Grosskopf, and Weber (1989), ¨

ARTICLE IN PRESS
328 J. Ruggiero / Economics of Education Review 25 (2006) 327–333

Ray (1991), McCarty and Yaisawarng (1993), Ruggiero (1996, 2001), and Duncombe, Miner, and Ruggiero (1997). DEA studies analyzing performance in other countries include Silva Portela and Thanassoulis (2001), Farren (2002) and Muniz (2002). ˜ The ability of DEA to measure efficiency is dependent on the nature of observed production. Numerous studies have shown that the performance of DEA deteriorates in the presence of measurement error and other statistical noise. Simulations in Banker, Gadh, and Gorr (1993), Ruggiero (1999), and Bifulco and Bretschneider (2001), for example, clearly show that deterministic models including DEA are sensitive to measurement error in cross-sectional analyses. Given an econometrician’s view of the world, the use of DEA could be problematic and potentially harmful given the important policy implications. Gong and Sickles (1992) and Ruggiero (2004) use simulation to show that the problem of measurement error is alleviated when efficiency is estimated using panel data. These models assume that the production technology is the same across time and use averaging to smooth production with respect to measurement error. In a separate but related literature, Hanushek (1979) discusses other important issues with empirical educational production analyses. One of the important issues involves the choice of aggregation. Conceptually, education occurs at the student level, but most analyses rely on aggregated data (primarily due to availability) to analyze educational production. One potential advantage, which is exploited in this paper, is the smoothing of production with respect to measurement error. Hence, aggregation, like panel data, helps alleviate the problem of measurement error. In particular, we assume that educational production with measurement error occurs at a disaggregated level and consider efficiency measurement using both disaggregate and aggregate data. Simulation analysis is used to show that the use of DEA on disaggregated data in the presence of measurement error leads to biased efficiency estimation, confirming the results of previous analyses. We also apply DEA to aggregated data and show that aggregation can lead to unbiased efficiency estimates. These results represent an important contribution to the DEA literature: aggregation effectively controls for statistical noise. This suggests that DEA can be used effectively to identify inefficiency of school districts (schools), where data are aggregated from the school (classroom) level. We note that such aggregation can be applied to other areas such as analyzing bank performance where production occurs at the branch level. For purposes of this paper, we consider district data as the aggregate of school data. The rest of the paper is organized as follows. The next section provides a model of educational production and efficiency using DEA. The third

section analyzes the performance of DEA using simulation analysis and the last section concludes.

2. Educational production We describe the educational production process as follows. Assume that each of Ns schools uses a vector X ¼ ðx1 ; :::; xM Þ of M discretionary school inputs to produce a vector Y ¼ ðy1 ; :::; yT Þ of T outcomes. For school j, inputs are given by X j ¼ ðx1j ; :::; xMj Þ and outputs by Y j ¼ ðy1j ; :::; yTj Þ. For purposes of this paper, we ignore the role of non-discretionary socioeconomic variables to focus on measurement error and aggregation. This further allows comparisons to the approach used by Bifulco and Bretshneider (2001). Ruggiero (1998) extended DEA to the case of multiple non-discretionary inputs; allowing measurement error with non-discretionary inputs is a trivial extension of Ruggiero’s model using the approach of this paper. Assuming variable returns to scale, technical efficiency can be estimated at the school level using the Banker, Charnes, and Cooper (1984) input-oriented model: Min yo
l1 ;:::;lN S

s:t.
NS X i¼1 NS X i¼1 NS X i¼1

li yti Xyto

8 t ¼ 1; :::; T 8 m ¼ 1; :::; M

li xmi pyo xmo li ¼ 1

li ! 0 8 i ¼ 1; :::; N S .

ð1Þ

In the absence of measurement error and with a sufficiently large sample size, this model works well in measuring relative performance with multiple inputs and outputs. The returns to scale assumption can be tightened to allow only constant returns by excluding the convexity constraint; leading to the Charnes, Cooper, and Rhodes (1978) model. The analysis of a given school seeks a maximum equi-proportional contraction of all inputs consistent with frontier production with all outputs at least as high as the levels of the school under analysis. Alternatively, we could have focused on the output oriented models. Model (1) leads to biased efficiency estimates in the presence of measurement error. Numerous simulation analyses confirm that even moderate levels of measurement error can lead to severe bias. For this reason, the usefulness of DEA to analyze educational production must be questioned. Consider measurement error in one

ARTICLE IN PRESS
J. Ruggiero / Economics of Education Review 25 (2006) 327–333 329

~ ~ output j, with yj ¼ yj þ j , where yji ¼ yji þ ji for school i. We note that this extends Ruggiero (2004) by considering additive error in the output. The BCC model for measuring efficiency of a given school is then given by Min yo
l1 ;:::;lN S

s:t.
NS X i¼1 NS X i¼1 NS X i¼1 NS X i¼1

li yti Xyto ~ ~ li yji Xyjo

8 t ¼ 1; :::; T; taj

li xmi pyo xmo li ¼ 1

8 m ¼ 1; :::; M

With the additional assumption that the EðÞ ¼ 0, the error term is averaged out with a sufficiently large number of schools in the district. In the simulation analysis performed in the next section, we vary the number of schools within a district to be 5, 10 or 20 and show that averaging data even with few schools can effectively eliminate measurement error problems. One concern raised by an anonymous reviewer was the assumption of aggregating from the school level to the district level. We note that state policy typically funds at the district level and compares district performance on state tests. And, in cases where this is not true, aggregation could happen from the student or classroom level to the school level. For space consideration, we show only the aggregation from the school to the district level; the results from a lower level to the school level follow.

li X0 8 i ¼ 1; :::; N S .

3. Simulation analysis ð2Þ We assume that production can be characterized by the conversion of discretionary inputs into output. As mentioned above, non-discretionary inputs can be effectively incorporated into the DEA model (see Ruggiero, 1998). In this simulation, we focus exclusively on the impact of measurement error. In particular, we assume that three inputs x1, x2 and x3 are used by schools to produce one output y according to the following stochastic production function: y ¼ uÀ1 vx0:4 x0:4 x0:2 , 1 2 3 where u and v represent inefficiency and statistical noise, respectively. Input data were generated uniformly on the interval (5,10) for each input. Measurement error was generated log-normal with standard deviation sv where sv took on values 0.1, 0.2 and 0.3. Notably, sv ¼ 0:2 represented high measurement error in the Bifulco and Bretschneider (2001) and hence, we consider an even higher measurement error variance. The inefficiency component ln u was generated half-normal with standard deviation 0.2. By varying the standard deviation of the measurement error, we effectively allow varying ratios of measurement error to inefficiency variances. Finally, we consider various combinations of schools and school districts; we vary the number of schools within each district to take on values of 5, 10 and 20 and we allow 50, 100 and 200 districts. Initially, each school within a given district had the same level of inefficiency. In the last simulation, we allow school inefficiency to vary within a district. Three measures of performance are considered in this study: mean absolute deviation (MAD) between true and measured efficiency, the correlation coefficient and the rank correlation coefficient. Because the assumed

As discussed in Ruggiero (2004), measurement error biases efficiency estimation for two reasons. First, the right-hand side of the contaminated output constraint distorts the measurement by comparing frontier output to a level of output not consistent with that school. This results from measurement error for the school under analysis. The second effect involves the location of the ‘‘frontier output’’; measurement error in other schools will lead to a biased location of the true frontier. For a further discussion as it relates to panel data, see Ruggiero (2004). We consider the aggregation of the data to the school district level. We assume that each of the NS schools can be assigned to one and only one of the ND school districts. Defining N Sd as the number of schools in district d for d ¼ 1; :::; N D , we construct an index that identifies the school according to the district to which it belongs. Then, for each district d, we can define a vector for each input m and output t given by X md ¼ ðxmd;1 ; :::; xmd;N Sd Þ and Y td ¼ ðytd;1 ; :::; ytd;N S Þ; respectively. d PD We note that N D ¼ d¼1 N Sd prior to estimating efficiency, we first average school level data to the district level for each district d xmd ¼ ¯
N Sd 1 X xmd;i 8 m, N Sd i¼1

ytd ¼ ¯ yjd ¼ ¯

N Sd 1 X y 8 taj; and N Sd i¼1 td;i N Sd 1 X ~ y : N Sd i¼1 jd;i

ARTICLE IN PRESS
330 J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Table 1 Performance results using school level simulated data sv Number of schools Number of districts Absolute deviation Mean 0.1 5 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 0.111 0.115 0.128 0.128 0.157 0.163 0.134 0.140 0.123 0.260 0.287 0.314 0.302 0.299 0.336 0.272 0.336 0.326 0.383 0.388 0.404 0.381 0.419 0.607 0.418 0.452 0.472 Standard deviation 0.075 0.074 0.071 0.072 0.074 0.073 0.072 0.073 0.071 0.122 0.123 0.114 0.116 0.119 0.118 0.120 0.119 0.115 0.152 0.146 0.148 0.147 0.146 0.114 0.131 0.125 0.132 Correlation coefficients Pearson 0.648 0.716 0.726 0.683 0.545 0.709 0.680 0.740 0.714 0.463 0.500 0.462 0.447 0.510 0.443 0.423 0.481 0.481 0.318 0.389 0.332 0.344 0.319 0.288 0.348 0.406 0.332 Rank 0.622 0.680 0.694 0.666 0.532 0.680 0.665 0.723 0.682 0.454 0.473 0.450 0.432 0.498 0.435 0.406 0.461 0.460 0.296 0.395 0.339 0.325 0.319 0.301 0.337 0.409 0.329

10

20

0.2

5

10

20

0.3

5

10

20

All calculations by author. School efficiency was held constant within each district.

production function is characterized by constant returns to scale, we use the CCR model that excludes the convexity constraint. This will allow us to focus exclusively on measurement error in technical efficiency estimation. The performance results using school level data are reported in Table 1. Not surprisingly, these results confirm the results known in the literature: measurement error has a devastating effect on DEA performance. Consider the absolute deviations reported in Table 1. In general, we note that as the measurement error variance increases, the MAD gets larger. Further, the standard deviation increases as well. Even when sv was low (0.1), the MAD was still unacceptably high in the 0.10–0.16 range. Holding constant the measurement error variance, we also note the general result that the MAD gets worse even as the number of schools and the number of districts increase.

Turning to the correlation and rank correlation results, we see a similar pattern. DEA performs better when the measurement error variance is relatively low while the correlation and rank correlation decrease as the variance is increased. Even with a low measurement error variance, DEA does not provide accurate estimates of efficiency. The correlation and rank correlations are all below 0.75 with sv ¼ 0:1. With sv ¼ 0:3, the highest correlation is about 0.41. No general pattern for improvement appears as the number of schools or districts increases. Clearly, the use of DEA on disaggregated data in the presence of measurement error is suspect. Next, we estimate efficiency using the constant returns to scale model using data averaged to the district level. The results are reported in Table 2. The MAD results in Table 2 reveal that DEA performs extremely well relative to the performance using school level data.

ARTICLE IN PRESS
J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Table 2 Performance results using simulated data aggregated to the district level sv Number of schools Number of districts Absolute deviation Mean 0.1 5 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 0.045 0.035 0.054 0.025 0.025 0.029 0.029 0.040 0.028 0.096 0.100 0.112 0.054 0.059 0.061 0.027 0.042 0.053 0.198 0.134 0.147 0.084 0.104 0.128 0.088 0.081 0.083 Standard deviation 0.031 0.028 0.034 0.019 0.018 0.023 0.021 0.025 0.019 0.064 0.059 0.062 0.047 0.042 0.039 0.023 0.030 0.032 0.096 0.082 0.087 0.065 0.073 0.067 0.053 0.043 0.050 Correlation coefficients Pearson 0.869 0.924 0.915 0.949 0.960 0.952 0.974 0.934 0.974 0.744 0.822 0.776 0.830 0.895 0.834 0.931 0.914 0.932 0.553 0.701 0.642 0.755 0.753 0.712 0.806 0.876 0.837 Rank 0.866 0.897 0.856 0.937 0.941 0.917 0.948 0.904 0.942 0.714 0.802 0.695 0.775 0.874 0.840 0.908 0.875 0.889 0.635 0.692 0.617 0.718 0.679 0.689 0.723 0.852 0.762 331

10

20

0.2

5

10

20

0.3

5

10

20

See comments in Table 1.

With low measurement error, the MAD is under 0.06 when the variance of the measurement error is low. As the measurement error variance increases, holding the number of districts and schools constant, the performance declines but at levels far better than reported for the school level data. All MADs are lower than 0.2 when using the aggregated data. We also note that the performance of DEA improves as the number of schools increases and remains about the same as the number of districts increases. Similar results are observed for the correlation and rank correlations reported in Table 3. In general, all correlations show marked improvement using the aggregate data compared to the school level analysis. In the case of low measurement error variance, the correlation and rank correlation coefficients are above 0.85 and usually above 0.90. Similar to the MAD results, the performance of DEA declines as the measurement

error variance increases. This is offset by improved performance as the number of schools increases. This is an important finding; as the number of units at the disaggregated level increases, the effect of measurement error on DEA performance at the aggregate level becomes less pronounced. It is also important to remember that the units chosen for the disaggregated level were schools. Extending the units to the classroom or even the student level suggests that DEA at the district level provides meaningful results even with high measurement error. Of course, the standard assumption that the expected value of the error term is zero was employed. One questionable assumption that was used in the simulation was constant efficiency at the school level within each district. For completeness, we now consider random efficiency within a given school district. Holding the number of schools at 20 and the number of districts

ARTICLE IN PRESS
332 J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Table 3 Simulation results assuming varying school efficiency within districtsa (number of districts ¼ 200, number of schools ¼ 20, sv ¼ 0:2) District descriptive statistics s 0b Range usc Mean Standard deviation Minimum Maximum Efficiency results Correlation coefficient Rank correlation coefficient Absolute deviation Mean Standard deviation
a

0.02

0.04

0.06

0 0 0 0 0.932 0.889 0.053 0.032

0.061 0.014 0.024 0.095 0.924 0.890 0.053 0.032

0.118 0.028 0.044 0.200 0.926 0.887 0.050 0.031

0.168 0.041 0.077 0.305 0.926 0.890 0.043 0.027

Average district efficiency was randomly generated as: ln ud $jNð0; 0:2Þj. School efficiency was calculated as ln us ¼ ln ud þ ln , where ln $Nð0; sÞ and s is given in the table. If us was calculated to be greater than unity, it was set equal to one. b The range is the maximum us–minimum us, calculated for each district. c This column is taken from previous tables and serves only as a reference.

at 200, school efficiency was regenerated as ln us ¼ ln ud+ln e, where ln ud was generated as above and ln e generated as N(0,s) with s taking on values of 0.02, 0.04 and 0.06. The value of us was set to one if it was calculated to be greater than unity. As shown in Table 3, this generating process led to large differences in school efficiency within districts. The results of this simulation are reported in Table 3. We note that the first column of results repeats the results from the earlier simulation where there was no variation in efficiency among schools in the same district. As s increases, we note that the variation increases within a district. In the last column, the average difference between maximum and minimum efficiency is approximately 0.17. The performance of DEA is measured by all three criteria: MAD, correlation and rank correlation. District efficiency was measured using the aggregated data and compared to the true value, determined by the average true efficiency. We note that the results are similar across columns for all measures. This suggests that the assumption of constant efficiency within a district does not effect performance evaluation at the aggregate level. Of course, a trade-off exists; with the aggregate data, it is not possible to identify efficiency of the individual units. However, the approach does work in identifying average efficiency of the aggregate units.

exists when measurement error exists. Previous simulation studies have shown that performance drops significantly as the variance of measurement error increases. In this paper, the problem of measurement error was highlighted by two effects. First, measurement error on a particular unit affects the placement of the unit relative to the true frontier. Also, measurement error on other units effects the evaluation of a given unit because these other units might serve as reference units for the unit under analysis. Unlike stochastic frontier models that use panel data, DEA is deterministic. However, as shown in this paper, aggregation of data to higher units (for example, from school level to district data or from classroom to school level) can smooth measurement error. And, as a result, performance evaluation using aggregate data can produce reliable results, eve when measurement error is substantial. As recommended by an anonymous referee, an interesting extension of this paper would be a real-world application. In particular, a useful application would be to show how aggregation of individual student or classroom data could help identify school and district inefficiency. Data requirements prevented such an application; however, I expect such data will be available in the near future.

References 4. Conclusions The use of DEA to evaluate performance has been called into question because of the potential bias that
Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078–1092.

ARTICLE IN PRESS
J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Banker, R., Gadh, V., & Gorr, W. (1993). A monte carlo comparison of two production frontier estimation methods: Corrected ordinary least squares and data envelopment analysis. European Journal of Operational Research, 67(3), 332–343. Bessent, A., Bessent, E. W., Kennington, J., & Reagan, B. (1982). An application of mathematical programming to assess productivity in the Houston independent school district. Management Science, 28(12), 1335–1366. Bifulco, R., & Bretschneider, S. (2001). Estimating school efficiency: A comparison of methods using simulated data. Economics of Education Review, 20(5), 417–429. Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444. Duncombe, W., Miner, J., & Ruggiero, J. (1997). Empirical evaluation of bureaucratic models of inefficiency. Public Choice, 93(1), 1–18. Fare, R., Grosskopf, S., & Weber, W. (1989). Measuring school ¨ district performance. Public Finance Quarterly, 17(4), 409–428. Farren, D. (2002). The technical efficiency of schools in Chile. Applied Economics, 34(12), 1533–1542. Gong, B., & Sickles, R. (1992). Finite sample evidence on the performance of stochastic frontiers and data envelopment analysis using panel data. Journal of Econometrics, 51(1), 259–284. Hanushek, E. (1979). Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources, 14(3), 351–388. 333 McCarty, T., & Yaisawarng, S. (1993). Technical efficiency in New Jersey school districts. In H. O. Fried, C. A. K. Lovell, & S. S. Schmidt (Eds.), The measurement of productive efficiency (pp. 271–287). New York: Oxford University Press. Muniz, M. (2002). Separating managerial inefficiency and ˜ external conditions in data envelopment analysis. European Journal of Operational Research, 143(3), 625–643. Ray, S. (1991). Resource-use efficiency in public schools: A study of connecticut data. Management Science, 37(12), 1620–1628. Ruggiero, J. (1996). Measuring technical efficiency in the public sector: An analysis of educational production. Review of Economics and Statistics, 78(3), 499–509. Ruggiero, J. (1998). Non-discretionary inputs in data envelopment analysis. European Journal of Operational Research, 111(3), 461–468. Ruggiero, J. (1999). Efficiency estimation and error decomposition in the stochastic frontier model: A monte carlo analysis. European Journal of Operational Research, 115(3), 555–563. Ruggiero, J. (2001). Determining the base cost of education: An analysis of Ohio school Districts. Contemporary Economic Policy, 19(3), 268–279. Ruggiero, J. (2004). Data envelopment with stochastic data. Journal of the OR Society, 55(9), 1008–1012. Silva Portela, M. C., & Thanassoulis, E. (2001). Decomposing school and school-type Efficiency. European Journal of Operational Research, 132(2), 357–373.