## Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

Economics of Education Review 25 (2006) 327–333 www.elsevier.com/locate/econedurev

**Measurement error, education production and data envelopment analysis
**

John RuggieroÃ

Department of Economics and Finance, University of Dayton, Dayton, OH 45469-2251, USA Received 30 September 2004; accepted 18 March 2005

Abstract Data Envelopment Analysis has become a popular tool for evaluating the efﬁciency of decision making units. The nonparametric approach has been widely applied to educational production. The approach is, however, deterministic and leads to biased estimates of performance in the presence of measurement error. Numerous simulation studies conﬁrm the effect that measurement error has on cross-sectional deterministic models of efﬁciency. It is also known that panel data models have the ability to smooth out measurement error, leading to more reliable efﬁciency estimates. In this paper, we exploit known properties of educational production to show that aggregation can also have a smoothing effect on production with measurement error, suggesting that efﬁciency analyses are more reliable than previously believed. r 2005 Published by Elsevier Ltd.

JEL Classiﬁcation: I21 Keywords: Educational production; Data envelopment analysis

1. Introduction Data Envelopment Analysis (DEA) is a nonparametric mathematical programming approach to the measurement of efﬁciency that was introduced in the operations research literature by Charnes, Cooper, and Rhodes (1978) and Banker, Charnes, and Cooper (1984). Using linear programming, an observed decision making unit (DMU) is evaluated relative to the production frontier, which consists of combinations of observed production possibilities using minimal assumptions. The primary advantage of the approach is the ability to handle multiple inputs and multiple outputs, particularly in the case when input prices are unavailÃTel.: +1 937 229 2258; fax: +1 937 229 2477.

E-mail address: ruggiero@notes.udayton.edu. 0272-7757/$ - see front matter r 2005 Published by Elsevier Ltd. doi:10.1016/j.econedurev.2005.03.003

able. The linear programming approach has withstood the test of time and has become an acceptable approach for efﬁciency evaluation. One important application of DEA is to the analysis of educational production. Many states have undergone legal challenges because school districts are not providing educational services efﬁciently and outcomes are not adequate. Reform has moved away from traditional issues like equity to adequacy and efﬁciency. The important policy implication is that school districts need to spend their money more wisely and increase their outcomes to acceptable levels. One popular technique that has been used for measuring efﬁciency in education is DEA. Research using DEA to measure performance of educational production in the United States include Bessent, Bessent, Kennington, and Reagan (1982), Fare, Grosskopf, and Weber (1989), ¨

ARTICLE IN PRESS

328 J. Ruggiero / Economics of Education Review 25 (2006) 327–333

Ray (1991), McCarty and Yaisawarng (1993), Ruggiero (1996, 2001), and Duncombe, Miner, and Ruggiero (1997). DEA studies analyzing performance in other countries include Silva Portela and Thanassoulis (2001), Farren (2002) and Muniz (2002). ˜ The ability of DEA to measure efﬁciency is dependent on the nature of observed production. Numerous studies have shown that the performance of DEA deteriorates in the presence of measurement error and other statistical noise. Simulations in Banker, Gadh, and Gorr (1993), Ruggiero (1999), and Bifulco and Bretschneider (2001), for example, clearly show that deterministic models including DEA are sensitive to measurement error in cross-sectional analyses. Given an econometrician’s view of the world, the use of DEA could be problematic and potentially harmful given the important policy implications. Gong and Sickles (1992) and Ruggiero (2004) use simulation to show that the problem of measurement error is alleviated when efﬁciency is estimated using panel data. These models assume that the production technology is the same across time and use averaging to smooth production with respect to measurement error. In a separate but related literature, Hanushek (1979) discusses other important issues with empirical educational production analyses. One of the important issues involves the choice of aggregation. Conceptually, education occurs at the student level, but most analyses rely on aggregated data (primarily due to availability) to analyze educational production. One potential advantage, which is exploited in this paper, is the smoothing of production with respect to measurement error. Hence, aggregation, like panel data, helps alleviate the problem of measurement error. In particular, we assume that educational production with measurement error occurs at a disaggregated level and consider efﬁciency measurement using both disaggregate and aggregate data. Simulation analysis is used to show that the use of DEA on disaggregated data in the presence of measurement error leads to biased efﬁciency estimation, conﬁrming the results of previous analyses. We also apply DEA to aggregated data and show that aggregation can lead to unbiased efﬁciency estimates. These results represent an important contribution to the DEA literature: aggregation effectively controls for statistical noise. This suggests that DEA can be used effectively to identify inefﬁciency of school districts (schools), where data are aggregated from the school (classroom) level. We note that such aggregation can be applied to other areas such as analyzing bank performance where production occurs at the branch level. For purposes of this paper, we consider district data as the aggregate of school data. The rest of the paper is organized as follows. The next section provides a model of educational production and efﬁciency using DEA. The third

section analyzes the performance of DEA using simulation analysis and the last section concludes.

2. Educational production We describe the educational production process as follows. Assume that each of Ns schools uses a vector X ¼ ðx1 ; :::; xM Þ of M discretionary school inputs to produce a vector Y ¼ ðy1 ; :::; yT Þ of T outcomes. For school j, inputs are given by X j ¼ ðx1j ; :::; xMj Þ and outputs by Y j ¼ ðy1j ; :::; yTj Þ. For purposes of this paper, we ignore the role of non-discretionary socioeconomic variables to focus on measurement error and aggregation. This further allows comparisons to the approach used by Bifulco and Bretshneider (2001). Ruggiero (1998) extended DEA to the case of multiple non-discretionary inputs; allowing measurement error with non-discretionary inputs is a trivial extension of Ruggiero’s model using the approach of this paper. Assuming variable returns to scale, technical efﬁciency can be estimated at the school level using the Banker, Charnes, and Cooper (1984) input-oriented model: Min yo

l1 ;:::;lN S

s:t.

NS X i¼1 NS X i¼1 NS X i¼1

li yti Xyto

8 t ¼ 1; :::; T 8 m ¼ 1; :::; M

li xmi pyo xmo li ¼ 1

li ! 0 8 i ¼ 1; :::; N S .

ð1Þ

In the absence of measurement error and with a sufﬁciently large sample size, this model works well in measuring relative performance with multiple inputs and outputs. The returns to scale assumption can be tightened to allow only constant returns by excluding the convexity constraint; leading to the Charnes, Cooper, and Rhodes (1978) model. The analysis of a given school seeks a maximum equi-proportional contraction of all inputs consistent with frontier production with all outputs at least as high as the levels of the school under analysis. Alternatively, we could have focused on the output oriented models. Model (1) leads to biased efﬁciency estimates in the presence of measurement error. Numerous simulation analyses conﬁrm that even moderate levels of measurement error can lead to severe bias. For this reason, the usefulness of DEA to analyze educational production must be questioned. Consider measurement error in one

ARTICLE IN PRESS

J. Ruggiero / Economics of Education Review 25 (2006) 327–333 329

~ ~ output j, with yj ¼ yj þ j , where yji ¼ yji þ ji for school i. We note that this extends Ruggiero (2004) by considering additive error in the output. The BCC model for measuring efﬁciency of a given school is then given by Min yo

l1 ;:::;lN S

s:t.

NS X i¼1 NS X i¼1 NS X i¼1 NS X i¼1

li yti Xyto ~ ~ li yji Xyjo

8 t ¼ 1; :::; T; taj

li xmi pyo xmo li ¼ 1

8 m ¼ 1; :::; M

With the additional assumption that the EðÞ ¼ 0, the error term is averaged out with a sufﬁciently large number of schools in the district. In the simulation analysis performed in the next section, we vary the number of schools within a district to be 5, 10 or 20 and show that averaging data even with few schools can effectively eliminate measurement error problems. One concern raised by an anonymous reviewer was the assumption of aggregating from the school level to the district level. We note that state policy typically funds at the district level and compares district performance on state tests. And, in cases where this is not true, aggregation could happen from the student or classroom level to the school level. For space consideration, we show only the aggregation from the school to the district level; the results from a lower level to the school level follow.

li X0 8 i ¼ 1; :::; N S .

3. Simulation analysis ð2Þ We assume that production can be characterized by the conversion of discretionary inputs into output. As mentioned above, non-discretionary inputs can be effectively incorporated into the DEA model (see Ruggiero, 1998). In this simulation, we focus exclusively on the impact of measurement error. In particular, we assume that three inputs x1, x2 and x3 are used by schools to produce one output y according to the following stochastic production function: y ¼ uÀ1 vx0:4 x0:4 x0:2 , 1 2 3 where u and v represent inefﬁciency and statistical noise, respectively. Input data were generated uniformly on the interval (5,10) for each input. Measurement error was generated log-normal with standard deviation sv where sv took on values 0.1, 0.2 and 0.3. Notably, sv ¼ 0:2 represented high measurement error in the Bifulco and Bretschneider (2001) and hence, we consider an even higher measurement error variance. The inefﬁciency component ln u was generated half-normal with standard deviation 0.2. By varying the standard deviation of the measurement error, we effectively allow varying ratios of measurement error to inefﬁciency variances. Finally, we consider various combinations of schools and school districts; we vary the number of schools within each district to take on values of 5, 10 and 20 and we allow 50, 100 and 200 districts. Initially, each school within a given district had the same level of inefﬁciency. In the last simulation, we allow school inefﬁciency to vary within a district. Three measures of performance are considered in this study: mean absolute deviation (MAD) between true and measured efﬁciency, the correlation coefﬁcient and the rank correlation coefﬁcient. Because the assumed

As discussed in Ruggiero (2004), measurement error biases efﬁciency estimation for two reasons. First, the right-hand side of the contaminated output constraint distorts the measurement by comparing frontier output to a level of output not consistent with that school. This results from measurement error for the school under analysis. The second effect involves the location of the ‘‘frontier output’’; measurement error in other schools will lead to a biased location of the true frontier. For a further discussion as it relates to panel data, see Ruggiero (2004). We consider the aggregation of the data to the school district level. We assume that each of the NS schools can be assigned to one and only one of the ND school districts. Deﬁning N Sd as the number of schools in district d for d ¼ 1; :::; N D , we construct an index that identiﬁes the school according to the district to which it belongs. Then, for each district d, we can deﬁne a vector for each input m and output t given by X md ¼ ðxmd;1 ; :::; xmd;N Sd Þ and Y td ¼ ðytd;1 ; :::; ytd;N S Þ; respectively. d PD We note that N D ¼ d¼1 N Sd prior to estimating efﬁciency, we ﬁrst average school level data to the district level for each district d xmd ¼ ¯

N Sd 1 X xmd;i 8 m, N Sd i¼1

ytd ¼ ¯ yjd ¼ ¯

N Sd 1 X y 8 taj; and N Sd i¼1 td;i N Sd 1 X ~ y : N Sd i¼1 jd;i

ARTICLE IN PRESS

330 J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Table 1 Performance results using school level simulated data sv Number of schools Number of districts Absolute deviation Mean 0.1 5 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 0.111 0.115 0.128 0.128 0.157 0.163 0.134 0.140 0.123 0.260 0.287 0.314 0.302 0.299 0.336 0.272 0.336 0.326 0.383 0.388 0.404 0.381 0.419 0.607 0.418 0.452 0.472 Standard deviation 0.075 0.074 0.071 0.072 0.074 0.073 0.072 0.073 0.071 0.122 0.123 0.114 0.116 0.119 0.118 0.120 0.119 0.115 0.152 0.146 0.148 0.147 0.146 0.114 0.131 0.125 0.132 Correlation coefﬁcients Pearson 0.648 0.716 0.726 0.683 0.545 0.709 0.680 0.740 0.714 0.463 0.500 0.462 0.447 0.510 0.443 0.423 0.481 0.481 0.318 0.389 0.332 0.344 0.319 0.288 0.348 0.406 0.332 Rank 0.622 0.680 0.694 0.666 0.532 0.680 0.665 0.723 0.682 0.454 0.473 0.450 0.432 0.498 0.435 0.406 0.461 0.460 0.296 0.395 0.339 0.325 0.319 0.301 0.337 0.409 0.329

10

20

0.2

5

10

20

0.3

5

10

20

All calculations by author. School efﬁciency was held constant within each district.

production function is characterized by constant returns to scale, we use the CCR model that excludes the convexity constraint. This will allow us to focus exclusively on measurement error in technical efﬁciency estimation. The performance results using school level data are reported in Table 1. Not surprisingly, these results conﬁrm the results known in the literature: measurement error has a devastating effect on DEA performance. Consider the absolute deviations reported in Table 1. In general, we note that as the measurement error variance increases, the MAD gets larger. Further, the standard deviation increases as well. Even when sv was low (0.1), the MAD was still unacceptably high in the 0.10–0.16 range. Holding constant the measurement error variance, we also note the general result that the MAD gets worse even as the number of schools and the number of districts increase.

Turning to the correlation and rank correlation results, we see a similar pattern. DEA performs better when the measurement error variance is relatively low while the correlation and rank correlation decrease as the variance is increased. Even with a low measurement error variance, DEA does not provide accurate estimates of efﬁciency. The correlation and rank correlations are all below 0.75 with sv ¼ 0:1. With sv ¼ 0:3, the highest correlation is about 0.41. No general pattern for improvement appears as the number of schools or districts increases. Clearly, the use of DEA on disaggregated data in the presence of measurement error is suspect. Next, we estimate efﬁciency using the constant returns to scale model using data averaged to the district level. The results are reported in Table 2. The MAD results in Table 2 reveal that DEA performs extremely well relative to the performance using school level data.

ARTICLE IN PRESS

J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Table 2 Performance results using simulated data aggregated to the district level sv Number of schools Number of districts Absolute deviation Mean 0.1 5 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 50 100 200 0.045 0.035 0.054 0.025 0.025 0.029 0.029 0.040 0.028 0.096 0.100 0.112 0.054 0.059 0.061 0.027 0.042 0.053 0.198 0.134 0.147 0.084 0.104 0.128 0.088 0.081 0.083 Standard deviation 0.031 0.028 0.034 0.019 0.018 0.023 0.021 0.025 0.019 0.064 0.059 0.062 0.047 0.042 0.039 0.023 0.030 0.032 0.096 0.082 0.087 0.065 0.073 0.067 0.053 0.043 0.050 Correlation coefﬁcients Pearson 0.869 0.924 0.915 0.949 0.960 0.952 0.974 0.934 0.974 0.744 0.822 0.776 0.830 0.895 0.834 0.931 0.914 0.932 0.553 0.701 0.642 0.755 0.753 0.712 0.806 0.876 0.837 Rank 0.866 0.897 0.856 0.937 0.941 0.917 0.948 0.904 0.942 0.714 0.802 0.695 0.775 0.874 0.840 0.908 0.875 0.889 0.635 0.692 0.617 0.718 0.679 0.689 0.723 0.852 0.762 331

10

20

0.2

5

10

20

0.3

5

10

20

See comments in Table 1.

With low measurement error, the MAD is under 0.06 when the variance of the measurement error is low. As the measurement error variance increases, holding the number of districts and schools constant, the performance declines but at levels far better than reported for the school level data. All MADs are lower than 0.2 when using the aggregated data. We also note that the performance of DEA improves as the number of schools increases and remains about the same as the number of districts increases. Similar results are observed for the correlation and rank correlations reported in Table 3. In general, all correlations show marked improvement using the aggregate data compared to the school level analysis. In the case of low measurement error variance, the correlation and rank correlation coefﬁcients are above 0.85 and usually above 0.90. Similar to the MAD results, the performance of DEA declines as the measurement

error variance increases. This is offset by improved performance as the number of schools increases. This is an important ﬁnding; as the number of units at the disaggregated level increases, the effect of measurement error on DEA performance at the aggregate level becomes less pronounced. It is also important to remember that the units chosen for the disaggregated level were schools. Extending the units to the classroom or even the student level suggests that DEA at the district level provides meaningful results even with high measurement error. Of course, the standard assumption that the expected value of the error term is zero was employed. One questionable assumption that was used in the simulation was constant efﬁciency at the school level within each district. For completeness, we now consider random efﬁciency within a given school district. Holding the number of schools at 20 and the number of districts

ARTICLE IN PRESS

332 J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Table 3 Simulation results assuming varying school efﬁciency within districtsa (number of districts ¼ 200, number of schools ¼ 20, sv ¼ 0:2) District descriptive statistics s 0b Range usc Mean Standard deviation Minimum Maximum Efﬁciency results Correlation coefﬁcient Rank correlation coefﬁcient Absolute deviation Mean Standard deviation

a

0.02

0.04

0.06

0 0 0 0 0.932 0.889 0.053 0.032

0.061 0.014 0.024 0.095 0.924 0.890 0.053 0.032

0.118 0.028 0.044 0.200 0.926 0.887 0.050 0.031

0.168 0.041 0.077 0.305 0.926 0.890 0.043 0.027

Average district efﬁciency was randomly generated as: ln ud $jNð0; 0:2Þj. School efﬁciency was calculated as ln us ¼ ln ud þ ln , where ln $Nð0; sÞ and s is given in the table. If us was calculated to be greater than unity, it was set equal to one. b The range is the maximum us–minimum us, calculated for each district. c This column is taken from previous tables and serves only as a reference.

at 200, school efﬁciency was regenerated as ln us ¼ ln ud+ln e, where ln ud was generated as above and ln e generated as N(0,s) with s taking on values of 0.02, 0.04 and 0.06. The value of us was set to one if it was calculated to be greater than unity. As shown in Table 3, this generating process led to large differences in school efﬁciency within districts. The results of this simulation are reported in Table 3. We note that the ﬁrst column of results repeats the results from the earlier simulation where there was no variation in efﬁciency among schools in the same district. As s increases, we note that the variation increases within a district. In the last column, the average difference between maximum and minimum efﬁciency is approximately 0.17. The performance of DEA is measured by all three criteria: MAD, correlation and rank correlation. District efﬁciency was measured using the aggregated data and compared to the true value, determined by the average true efﬁciency. We note that the results are similar across columns for all measures. This suggests that the assumption of constant efﬁciency within a district does not effect performance evaluation at the aggregate level. Of course, a trade-off exists; with the aggregate data, it is not possible to identify efﬁciency of the individual units. However, the approach does work in identifying average efﬁciency of the aggregate units.

exists when measurement error exists. Previous simulation studies have shown that performance drops signiﬁcantly as the variance of measurement error increases. In this paper, the problem of measurement error was highlighted by two effects. First, measurement error on a particular unit affects the placement of the unit relative to the true frontier. Also, measurement error on other units effects the evaluation of a given unit because these other units might serve as reference units for the unit under analysis. Unlike stochastic frontier models that use panel data, DEA is deterministic. However, as shown in this paper, aggregation of data to higher units (for example, from school level to district data or from classroom to school level) can smooth measurement error. And, as a result, performance evaluation using aggregate data can produce reliable results, eve when measurement error is substantial. As recommended by an anonymous referee, an interesting extension of this paper would be a real-world application. In particular, a useful application would be to show how aggregation of individual student or classroom data could help identify school and district inefﬁciency. Data requirements prevented such an application; however, I expect such data will be available in the near future.

References 4. Conclusions The use of DEA to evaluate performance has been called into question because of the potential bias that

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefﬁciencies in data envelopment analysis. Management Science, 30(9), 1078–1092.

ARTICLE IN PRESS

J. Ruggiero / Economics of Education Review 25 (2006) 327–333 Banker, R., Gadh, V., & Gorr, W. (1993). A monte carlo comparison of two production frontier estimation methods: Corrected ordinary least squares and data envelopment analysis. European Journal of Operational Research, 67(3), 332–343. Bessent, A., Bessent, E. W., Kennington, J., & Reagan, B. (1982). An application of mathematical programming to assess productivity in the Houston independent school district. Management Science, 28(12), 1335–1366. Bifulco, R., & Bretschneider, S. (2001). Estimating school efﬁciency: A comparison of methods using simulated data. Economics of Education Review, 20(5), 417–429. Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efﬁciency of decision making units. European Journal of Operational Research, 2(6), 429–444. Duncombe, W., Miner, J., & Ruggiero, J. (1997). Empirical evaluation of bureaucratic models of inefﬁciency. Public Choice, 93(1), 1–18. Fare, R., Grosskopf, S., & Weber, W. (1989). Measuring school ¨ district performance. Public Finance Quarterly, 17(4), 409–428. Farren, D. (2002). The technical efﬁciency of schools in Chile. Applied Economics, 34(12), 1533–1542. Gong, B., & Sickles, R. (1992). Finite sample evidence on the performance of stochastic frontiers and data envelopment analysis using panel data. Journal of Econometrics, 51(1), 259–284. Hanushek, E. (1979). Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources, 14(3), 351–388. 333 McCarty, T., & Yaisawarng, S. (1993). Technical efﬁciency in New Jersey school districts. In H. O. Fried, C. A. K. Lovell, & S. S. Schmidt (Eds.), The measurement of productive efﬁciency (pp. 271–287). New York: Oxford University Press. Muniz, M. (2002). Separating managerial inefﬁciency and ˜ external conditions in data envelopment analysis. European Journal of Operational Research, 143(3), 625–643. Ray, S. (1991). Resource-use efﬁciency in public schools: A study of connecticut data. Management Science, 37(12), 1620–1628. Ruggiero, J. (1996). Measuring technical efﬁciency in the public sector: An analysis of educational production. Review of Economics and Statistics, 78(3), 499–509. Ruggiero, J. (1998). Non-discretionary inputs in data envelopment analysis. European Journal of Operational Research, 111(3), 461–468. Ruggiero, J. (1999). Efﬁciency estimation and error decomposition in the stochastic frontier model: A monte carlo analysis. European Journal of Operational Research, 115(3), 555–563. Ruggiero, J. (2001). Determining the base cost of education: An analysis of Ohio school Districts. Contemporary Economic Policy, 19(3), 268–279. Ruggiero, J. (2004). Data envelopment with stochastic data. Journal of the OR Society, 55(9), 1008–1012. Silva Portela, M. C., & Thanassoulis, E. (2001). Decomposing school and school-type Efﬁciency. European Journal of Operational Research, 132(2), 357–373.

- data envelopment analysis
- A New Approach for Ranking Efficient Units in Data Envelopment Analysis and Application to a Sample of Vietnamese Agricultural Bank Branches
- Chapter2.doc
- 201502271602295 Data Envelopment Analysis
- data envelopment analysis
- Data Envelopment Analysis
- 2008 WP-BI ASCARYA ETAL Measuring the Efficiency of Conv N Islamic Banks in MAL n IND Para-Nonpara
- 12
- Asset Management Efficiency of Selected Cement Companies in Tamil Nadu
- IMECS2009_pp2087-2092
- Banking 1
- Data Envelopment Analysis
- Efficiency Evaluation of Banking Sector in India - Prasad
- 14. a Significant - Full
- A Review on Data Envelopment Analysis
- Back Caluclating Regression Values Through Pizza Example
- Manual Of Modern Physics Experiment
- Descriptive Statistics
- Statistics Assignment of B.A Psychology IGNOU
- pertemuan 2
- Introduction to Tropical Fish Stock Assessment Part 2
- 2010 OCT QMT500
- Comparison of TV Viewer Final
- Newsworthy Survey
- Helms 2016
- Eco No Metric Analysis of Efficiency in Indian Cement Industry
- Weighting
- So Far Everyone Has Written About Thi...
- Some Solved Questions
- Statistics Notes in the British Medical Journal (Bland JM, Altman DG. - NEJ)

- Social Intelligence the New Science of Human Relationships
- Galicia+-+A+Mariña+lucense
- Science 77
- Hybrid force/position control scheme for flexible joint robot with friction between and the end-effector and the environment
- USAL salamanca
- Extrusion analysis and workability prediction of three-layer composite hexagonal clad rods
- Quantum theoretical approach to a near-®eld optical system
- 1101_ftp
- Science 59
- Surface Functionalized Electrospun Biodegradable Nanofibers for Immobilization of Bioactive Molecules
- Science

Close Dialog## Are you sure?

This action might not be possible to undo. Are you sure you want to continue?

Loading