Test Score Analysis Shows State Schools Doing Well

Jonathan Marshall, Chronicle Economics Editor

The San Francisco Chronicle, 9 September 1997
The widespread belief that public schools in California rank among the nation's worst is
unfounded, a Chronicle computer analysis of recent 12th-grade test scores shows.
Comparing students of similar family backgrounds across the country, California seniors
performed right at the average in reading and almost 2 percent above average in basic math
skills in 1992 -- the most recent year for which data are available.
In addition, the analysis shows that California seniors have improved their relative standing
since the early 1970s, a time some education critics consider a "golden age" in the state's
system of K-12 schooling. . . .
The Chronicle study, done in collaboration with University of Oregon economist David Figlio, is
the first to make a statistically defensible comparison of state educational quality by accounting
for the extraordinary challenges California faces in trying to educate huge numbers of students
from poorly educated or non-English-speaking families. . . .
For the full story:
http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/1997/09/09/MN69579.DTL

California Student Achievement, 1972 to 1992: Technical notes

By Jonathan Marshall
Chronicle Economics Editor 1997
The Chronicles analysis of Calfornia educational quality relative to the rest of the country was motivated
by dissatisfaction with misleading comparisons and poor methodology that have dominated the education
debate.
The analysis draws upon standard statistical methods long adopted by economists, sociologists and other
academic experts to isolate the effects of education by controlling for individual and family characteristics
that affect measured achievement.
The first requirement was for a consistent test. I rejected the Scholastic Aptitude Test because states vary
widely in the fraction of students taking the test: In 1993, for instance, only 4 percent of Utah students took
the test compared to 47 percent in California. One study showed that 85 percent of variation in state SAT
scores is simply due to the rate at which students in each state take the test. (Brian Powell and Lala Carr
Steelman, Bewitched, Bothered and Bewildered: The Use and Misuse of State SAT and ACT Scores,
Harvard Educational Review, Spring 1996, 27-59.) The College Board itself, which administers the SAT,
warns against misusing the test for invalid comparisons: To make useful comparisons of students
performance between states, a common test given to all students would be required. Because the percentage
of SAT takers varies widely among the states, and because the test takers are self-selected, the SAT is
inappropriate for this purpose. In addition, it notes, non-academic factors such as family background and
education of parents could very well have a significant influence on average scores. (August 26, 1997
press release)
The National Assessment of Educational Progress, administered by the Educational Testing Service for the
Department of Education, uses statistically valid state samples to test math and reading proficiency at 4 th
and 8th grades. But Congress barred any reporting of state scores before 1990, making trend studies

impossible over long periods of time. In addition, NAEP collects only rudimentary demographic
information about students. Some of its reported information, such as parental education, is also suspect
because it comes from students themselves instead of parents.
The surveys used in The Chronicle analysis, the National Longitudinal Study of the High School Class of
1972 (NLS-72) and National Educational Longitudinal Study of 1992, are staples of social science
researchers. They have nationally representative samples of high school seniors, detailed demographic
information and tests of basic verbal and math skills.
Both studies are problematic for state comparisons, however. Neither survey stratified the sample along
state lines, so student samples within states are not guaranteed to be random. In short, a detailed ranking of
states would be statistically questionable.
But nearly all scholars contacted by The Chronicle agreed that the California samples are large enough to
be reasonably representative of the true population of high school seniors. The NELS sample used here,
the smaller of the two, has nearly 600 students from more than 40 high schools in California. Comparing
California against the rest of the nation is a defensible statistical exercise, even if comparing California to
Rhode Island is not. (A parallel analysis of NELS performed for The Chronicle by another scholar, who
asked to remain anonymous, used a more complete California sample of 1,081 seniors from 127 high
schools, and came to conclusions qualitatively identical to those reported here. We report David Figlios
numbers from the smaller sample because of the consistency of his controls between NLS-72 and NELS.)
The analysis is based on a simple regression of 12th grade math and reading test scores against a group
(vector) of demographic characteristics, a state dummy variable for California to indicate any special
influence of the state on test scores, and an error term, or residual. The specification reads:
Ai=bFi+aCi+ei
where A is the test score for the ith individual, F is the vector of demographic characteristics, C is the
California dummy (C=1 for California students, 0 for others) and e is the error term for the ith individual.
An alternative, and statistically more efficient specification (with smaller standard errors), drops the
dummy variable and just compares the mean error terms for California students against the mean error
terms for non-California students. Figures for this differences of residuals approach are also reported
below.
Note that this specification is similar to ones used in the large literature assessing the impact of school
quality on educational outcomes. But instead of including variables to proxy for school quality, such as
class size and teacher qualifications, we use the state dummy to summarize the myriad of factors that
influence educational environment within a state.
Achievement scores are based on a recalibration of NLS-72 and NELS scores by University of Oregon
economist David Figlio and Tufts University economist Thomas Downes (see Thomas A. Downes and
David N. Figlio, School Finance Reforms, Tax Limits and Student Performance: Do Reforms Level-Up or
Dumb Down? University of Oregon Working Paper Series 209, May 1997). Their recalibration in theory
permits an absolute comparison of scores over time, from 1972 to 1992 and could suggest a substantial
growth in real scores of about 27 percent over the period, based on the means reported below. Such a trend
is consistent with growth in national NAEP math and reading scores among 17-year-olds since the early
1970s, and the fact that California seniors apparently improved their relative performance vis--vis the
national average. (On national NAEP trends, see David Grissmer, et. al, Student Achievement and the
Changing American Family (RAND, 1994), especially p. 13.) But the strong absolute growth in California
scores suggested by our mean results is not reported in our news story because of uncertainty about the
impact of different sampling methods in NLS-72 (which picked a sample of 12th graders) and NELS (which
picked a sample of 8th graders in 1988, and freshened it with additional students to arrive at the 1992
sample of seniors).
The charts below give summary statistics on students surveyed by NLS and NELS, as well as regression
results showing the impact of each demographic variable on mean test outcomes. The Calif variable is
the state dummy indicating the estimated impact of educational quality in the state relative to the rest of the
country. Results from the alternative specification, based on the difference in mean residuals, are reported
at the end.

The test score variables shown in the regression tables below are nlseq_ma (math) and nlseq_re (reading).
The demographic control variables are female, black, Hispanic, Protestant, Catholic, Jewish, atheist, father
with bachelors degree, mother with bachelors degree, no date on fathers education, no data on mothers
education, family income over $10,000 (1988), family income over $25,000, family income over $35,000,
family income over $50,000, no family income data, number of siblings, 6 or more siblings, family
subscribes to a newspaper, and students self-reported command of English.
Many more demographic variables are available in NELS, including parental information on country of
origin. A richer set of controls could and should be used in follow-up studies. This set was selected because
they are consistent between the 1972 and 1992 surveys.
The University of Oregons Figlio generously agreed to perform the actual computer runs for The
Chronicle. The data sets are extremely difficult to obtain from the Department of Education because of
official concerns over privacy. Public data available to The Chronicle had no information on students
states, for example. Figlio, an academic researcher who has pledged to follow strict security procedures,
has special access to individual-level records needed in a regression analysis such as this.
A note on interpreting the results: The coefficient on the Calif dummy variable sums up state-specific
effects on test scores. These could include overall differences in school quality, formal education policy
and general educational climatefor example, whether parents in a given state have a higher commitment
to education. Unmeasured demographic differences due to limited controls may also bias the result,
however. Thus the results should be seen as suggestive but not definitive.
To get the percentage difference between average California scores and those of students from other states,
divide the Calif coefficient by the mean scores, reported below, for out-of-state students.
Achievement test scores can be related to California educational quality only to the extent that children
were educated in Californias schools. If 12th graders who took the test in California spent the previous 16
years of their life in Ohio, this analysis will give a biased result. An alternative specification favored by
some researchers looks at test score changes from 10th to 12th grade for students who remain within the
same state. (See, for example, Eric Hanushek and Lori Taylor, Alternative Assessments of the
Performance of Schools: Measurement of State Variations in Achievement, Journal of Human Resources,
XXV (Spring 1990), 179-201.) The Chronicle obtained the services of another generous economist, who
asked to remain anonymous, to run a value-added regression based on NELS. After controlling for a
similar (though not identical) group of demographic characteristics, as well as 10th grade scores, California
seniors ranked again at the middle of the pack of states. However, there was one oddity: Californias raw
value-added scores were slightly higher before controls than after. With or without controls, however,
California seniors ranked at least average.

Means of all variables used in Chronicle analysis:

NLS-72, out-of-state students
-> nels= 0.00 calif= 0.00
Variable | Obs
Mean Std. Dev.
Min
Max
---------+----------------------------------------------------nlseq_ma | 11436
12.79
7.22
-5.67 25.00
nlseq_re | 11436
9.67
4.93
-5.00 20.00
female |
11436
0.51
0.50
0.00
1.00
black |
11436
0.11
0.31
0.00
1.00
hispanic | 11436
0.03
0.18
0.00
1.00
protesta | 11436
0.41
0.49
0.00
1.00
catholic | 11436
0.26
0.44
0.00
1.00
jewish | 11436
0.03
0.16
0.00
1.00
atheist | 11436
0.05
0.22
0.00
1.00
fbach | 11436
0.16
0.37
0.00
1.00
mbach | 11436
0.10
0.30
0.00
1.00
nofathed | 11436
0.12
0.33
0.00
1.00

nomothed | 11436
over10k | 11436
over25k | 11436
over35k | 11436
over50k | 11436
noincdat | 11436
siblings | 11436
sibs_6up | 11436
newsppr | 11436
speakeng | 11436

0.11
0.75
0.56
0.38
0.18
0.19
2.04
0.06
0.88
0.92

0.31
0.43
0.50
0.49
0.39
0.39
1.72
0.24
0.33
0.27

0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

1.00
1.00
1.00
1.00
1.00
1.00
6.00
1.00
1.00
1.00

NLS-72, California students

-> nels= 0.00 calif= 1.00
Variable | Obs
Mean Std. Dev.
Min
Max
---------+----------------------------------------------------nlseq_ma | 1340
11.74
7.31
-3.00 25.00
nlseq_re | 1340
9.27
4.86
-4.00
20.00
female | 1340
0.47
0.50
0.00
1.00
black | 1340
0.09
0.28
0.00
1.00
hispanic | 1340
0.13
0.34
0.00
1.00
protesta | 1340
0.34
0.47
0.00
1.00
catholic | 1340
0.32
0.47
0.00
1.00
jewish | 1340
0.02
0.14
0.00
1.00
atheist |
1340
0.11
0.31
0.00
1.00
fbach | 1340
0.19
0.39
0.00
1.00
mbach | 1340
0.11
0.31
0.00
1.00
nofathed | 1340
0.16
0.37
0.00
1.00
nomothed | 1340
0.15
0.36
0.00
1.00
over10k | 1340
0.78
0.41
0.00
1.00
over25k | 1340
0.61
0.49
0.00
1.00
over35k | 1340
0.43
0.50
0.00
1.00
over50k | 1340
0.22
0.41
0.00
1.00
noincdat | 1340
0.18
0.38
0.00
1.00
siblings |
1340
2.10
1.69
0.00
6.00
sibs_6up | 1340
0.06
0.23
0.00
1.00
newsppr | 1340
0.85
0.36
0.00
1.00
speakeng | 1340
0.89
0.32
0.00
1.00
NELS, out-of-state students
NELS=0, Calif=0
Variable | Obs
Mean Std. Dev.
Min
Max
---------+----------------------------------------------------nlseq_ma | 7119
14.86
4.44
4.42
23.81
nlseq_re | 7106
11.92
4.85
0.23
20.43
female | 7119
0.52
0.50
0.00
1.00
black |
7119
0.10
0.30
0.00
1.00
hispanic | 7119
0.08
0.27
0.00
1.00
protesta | 7119
0.49
0.50
0.00
1.00
catholic | 7119
0.25
0.43
0.00
1.00
jewish | 7119
0.02
0.13
0.00
1.00
atheist | 7119
0.08
0.27
0.00
1.00
fbach | 7119
0.28
0.45
0.00
1.00
mbach | 7119
0.23
0.42
0.00
1.00
nofathed | 7119
0.13
0.34
0.00
1.00
nomothed | 7119
0.10
0.30
0.00
1.00
over10k | 7119
0.93
0.26
0.00
1.00

over25k |
over35k |
over50k |
noincdat |
siblings |
sibs_6up |
newsppr |
speakeng |

7119
7119
7119
7119
7119
7119
7119
7119

0.72
0.57
0.36
0.04
2.14
0.05
0.76
0.99

0.45
0.49
0.48
0.19
1.48
0.23
0.42
0.11

0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

1.00
1.00
1.00
1.00
6.00
1.00
1.00
1.00

NELS, California students

-> nels= 1.00 calif= 1.00
Variable | Obs
Mean Std. Dev.
Min
Max
---------+----------------------------------------------------nlseq_ma | 581
14.87
4.56
5.02
23.66
nlseq_re | 577
11.75
5.05
0.31
20.30
female | 581
0.52
0.50
0.00
1.00
black |
581
0.05
0.22
0.00
1.00
hispanic | 581
0.31
0.46
0.00
1.00
protesta | 581
0.22
0.41
0.00
1.00
catholic | 581
0.32
0.47
0.00
1.00
jewish | 581
0.02
0.13
0.00
1.00
atheist |
581
0.13
0.33
0.00
1.00
fbach |
581
0.30
0.46
0.00
1.00
mbach | 581
0.23
0.42
0.00
1.00
nofathed | 581
0.18
0.38
0.00
1.00
nomothed | 581
0.16
0.36
0.00
1.00
over10k | 581
0.91
0.29
0.00
1.00
over25k | 581
0.68
0.47
0.00
1.00
over35k | 581
0.54
0.50
0.00
1.00
over50k | 581
0.36
0.48
0.00
1.00
noincdat | 581
0.03
0.17
0.00
1.00
siblings |
581
2.24
1.55
0.00
6.00
sibs_6up | 581
0.07
0.26
0.00
1.00
newsppr | 581
0.66
0.47
0.00
1.00
speakeng | 581
0.95
0.21
0.00
1.00

Regression results:
Coefficients indicate impact of each factor on test scores. The P statistic (fifth column) indicates the
probability that the observed impact of each factor was caused by chance.
NLS-72 math
Number of obs = 12776
R-squared = 0.2573
Adj R-squared = 0.2561
-----------------------------------------------------------------------------nlseq_ma |
Coef. Std. Err.
t P>|t|
[95% Conf. Interval]
---------+-------------------------------------------------------------------female | -1.895515 .1124653 -16.854 0.000
-2.115964 -1.675066
black | -4.111116 .198446 -20.717 0.000
-4.5001 -3.722133
hispanic | -3.812555 .3006134 -12.683 0.000
-4.401802 -3.223307
protesta | 2.648259 .1470329 18.011 0.000
2.360053 2.936466
catholic | 1.463332 .1639702
8.924 0.000
1.141925 1.784738
jewish | 4.431192 .3726178 11.892 0.000
3.700805 5.161578
atheist | .6443837 .2604705
2.474 0.013
.1338224 1.154945

fbach | 2.37318 .1742981 13.616 0.000

2.031529 2.71483
mbach | 1.279017 .2029863
6.301 0.000
.8811339 1.676901
nofathed | -2.072725 .3535043 -5.863 0.000
-2.765646 -1.379803
nomothed | -2.188753 .3645196 -6.004 0.000
-2.903266 -1.47424
over10k | .4713802 .2755212
1.711 0.087
-.0686826 1.011443
over25k | .8425078 .1844261
4.568 0.000
.4810049 1.204011
over35k | .4496444 .1811041
2.483 0.013
.0946532 .8046355
over50k | .3715734 .1830592
2.030 0.042
.0127499 .7303969
noincdat | .3203551 .2828804
1.132 0.257
-.2341329 .8748431
siblings | .1464534 .0406402
3.604 0.000
.0667924 .2261143
sibs_6up | -.6371684 .2852631 -2.234 0.026
-1.196327 -.07801
newsppr | .6688888 .1741692
3.840 0.000
.327491 1.010287
speakeng | .0563944 .2065272
0.273 0.785
-.3484299 .4612188
calif | -.7324334 .1835991 -3.989 0.000
-1.092315 -.3725516
_cons | 10.69361 .3567829 29.972 0.000
9.994264 11.39296
------------------------------------------------------------------------------

NELS-92 math
.
Number of obs = 7700
R-squared = 0.1945
Adj R-squared = 0.1923
-----------------------------------------------------------------------------nlseq_ma |
Coef. Std. Err.
t P>|t|
[95% Conf. Interval]
---------+-------------------------------------------------------------------female | -.3181873 .0917426 -3.468 0.001
-.4980278 -.1383468
black | -2.530242 .1644141 -15.389 0.000
-2.852539 -2.207946
hispanic | -1.402276 .1691595 -8.290 0.000
-1.733875 -1.070677
protesta | .2931639 .1305534
2.246 0.025
.0372436 .5490842
catholic | .7167906 .1436612
4.989 0.000
.4351753 .9984058
jewish | 1.421358 .3745107
3.795 0.000
.6872146 2.155501
atheist | -.0397016 .1931732 -0.206 0.837
-.4183738 .3389706
fbach | 1.720557 .1279631 13.446 0.000
1.469714
1.9714
mbach | 1.122406 .1324369
8.475 0.000
.8627934 1.382019
nofathed | -.0113348 .1648308 -0.069 0.945
-.3344483 .3117786
nomothed | .0501948 .1809047
0.277 0.781
-.3044277 .4048173
over10k | .6888703 .2003106
3.439 0.001
.2962068 1.081534
over25k | .8417342 .1557847
5.403 0.000
.5363537 1.147115
over35k | .1972385 .1552826
1.270 0.204
-.1071579 .5016348
over50k | .6303073 .1321488
4.770 0.000
.3712597 .8893549
noincdat | -1.034 .2555051 -4.047 0.000
-1.53486 -.5331402
siblings | -.1206136 .0401552 -3.004 0.003
-.1993287 -.0418985
sibs_6up | .2759278 .2564317
1.076 0.282
-.2267484 .778604
newsppr | -.2671773 .1096224 -2.437 0.015
-.4820671 -.0522875
speakeng | -.2518075 .3923113 -0.642 0.521
-1.020845 .5172296
calif | .2403698 .1788039
1.344 0.179
-.1101347 .5908744
_cons | 13.44165 .4466044 30.097 0.000
12.56618 14.31711
------------------------------------------------------------------------------

NLS-72 reading
Number of obs = 12776

R-squared = 0.2223
Adj R-squared = 0.2210
-----------------------------------------------------------------------------nlseq_re | Coef. Std. Err.
t P>|t|
[95% Conf. Interval]
---------+-------------------------------------------------------------------female | .1672243 .0783478
2.134 0.033
.0136508 .3207978
black | -2.381285 .1382454 -17.225 0.000
-2.652266 -2.110303
hispanic | -2.53755 .2094193 -12.117 0.000
-2.948043 -2.127056
protesta | 1.812733 .1024289 17.697 0.000
1.611957 2.013509
catholic | .9280258 .1142282
8.124 0.000
.7041214 1.15193
jewish | 2.121936 .2595804
8.174 0.000
1.613119 2.630752
atheist | .6861661 .1814541
3.781 0.000
.3304888 1.041843
fbach | 1.625984 .121423 13.391 0.000
1.387976 1.863991
mbach | .9670505 .1414083
6.839 0.000
.689869 1.244232
nofathed | -1.532428 .2462652 -6.223 0.000
-2.015145 -1.049711
nomothed | -1.267486 .2539388 -4.991 0.000
-1.765244 -.7697276
over10k | .7085746 .191939
3.692 0.000
.3323453 1.084804
over25k | .5171014 .1284786
4.025 0.000
.2652641 .7689386
over35k | .259213 .1261643
2.055 0.040
.011912 .506514
over50k | .2944597 .1275263
2.309 0.021
.0444889 .5444304
noincdat | .5090541 .1970658
2.583 0.010
.1227757 .8953326
siblings | .0049067 .0283116
0.173 0.862
-.0505883 .0604017
sibs_6up | -.6068512 .1987256 -3.054 0.002
-.9963832 -.2173191
newsppr | .2777782 .1213332
2.289 0.022
.0399469 .5156095
speakeng | .3562325 .1438751
2.476 0.013
.0742157 .6382492
calif | -.1324731 .1279024 -1.036 0.300
-.3831811 .1182348
_cons | 7.208975 .2485492 29.004 0.000
6.721781 7.696168
------------------------------------------------------------------------------

NELS-92 reading
Number of obs = 7693
R-squared = 0.1506
Adj R-squared = 0.1483
-----------------------------------------------------------------------------nlseq_re | Coef. Std. Err.
t P>|t|
[95% Conf. Interval]
---------+-------------------------------------------------------------------female | 1.131385 .1031656 10.967 0.000
.9291519 1.333617
black | -2.361354 .1846977 -12.785 0.000
-2.723412 -1.999296
hispanic | -.8529017 .190095 -4.487 0.000
-1.22554 -.4802635
protesta | .1165895 .1467804
0.794 0.427
-.1711401 .4043191
catholic | .387323 .1616099
2.397 0.017
.0705236 .7041225
jewish | .9071151 .4209
2.155 0.031
.0820361 1.732194
atheist | -.0159674 .2173539 -0.073 0.941
-.4420404 .4101056
fbach | 1.61334 .143921 11.210 0.000
1.331216 1.895465
mbach | .9637923 .1489582
6.470 0.000
.6717935 1.255791
nofathed | .1835033 .1853887
0.990 0.322
-.1799092 .5469158
nomothed | -.2953427 .2035978 -1.451 0.147
-.69445 .1037646
over10k | .7646136 .2253685
3.393 0.001
.3228298 1.206397
over25k | .7173256 .1751682
4.095 0.000
.3739481 1.060703
over35k | .3856531 .1745585
2.209 0.027
.0434707 .7278355
over50k | .4205737 .1485452
2.831 0.005
.1293846 .7117629

noincdat | -1.480225 .2867232 -5.163 0.000

siblings | -.2379768 .0451313 -5.273 0.000
sibs_6up | .4723024 .28841
1.638 0.102
newsppr | -.3865551 .1231851 -3.138 0.002
speakeng | 1.149963 .4409071
2.608 0.009
calif | -.0206665 .2015605 -0.103 0.918
_cons | 8.885379 .501885 17.704 0.000
.
Differences in residuals:
Calif - rest of US (std error in parens)

-2.042281 -.9181691
-.3264465 -.149507
-.09306 1.037665
-.6280315 -.1450787
.2856643 2.014261
-.4157802 .3744471
7.901547 9.869211

NLS-Math: -0.706 (0.064)

NELS- Math: 0.190 (0.051)
NLS-Read: -0.128 (0.044)
NELS-Read: -0.016 (0.056)

Acknowledgments: David Figlio, Meredith Phillips, Eric Hanushek, Lori Taylor, David Card, Alan
Krueger, Jeff Grogger, Julian Betts, Cecilia Rouse, Henry Levin, Ruben Rumbaut, Hans Johnson, Pat
McCabe, David Grissmer, Gary Phillips, Dominic Brewer, Nabeel Al-Salam, Anonymous.