See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/328792927

Divergences between Effects on Test Scores and Effects on Non-Cognitive Skills

Article  in  SSRN Electronic Journal · January 2018

DOI: 10.2139/ssrn.3273422

CITATIONS READS
4 445

1 author:

Corey A. DeAngelis
Reason Foundation
80 PUBLICATIONS   543 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Corey A. DeAngelis on 02 March 2019.

The user has requested enhancement of the downloaded file.

 Divergences between Effects on Test Scores and Effects on Non-Cognitive Skills

Corey A. DeAngelis, Ph.D.

Center for Educational Freedom
Cato Institute
CDeAngelis@cato.org

October 23, 2018

*Corresponding author is Corey A. DeAngelis, CDeAngelis@cato.org, (210) 818-6713.

Address: 1000 Massachusetts Ave NW, Washington, DC 20001. Declarations of interest: none.
Funding: none.

Electronic copy available at: https://ssrn.com/abstract=3273422

 Abstract

Students’ abilities are multidimensional; teachers and schools shape students’ cognitive skills,

most often captured by test scores, but they also shape students’ non-cognitive skills such as

character, tolerance, effort, and conscientiousness. This study reviews the evidence indicating

disconnects between test scores and non-cognitive skills from the literature on value-added

modeling and private school choice programs. Because several studies reveal disconnects,

designers of public and private school policies should consider the potential unintended

consequences of shaping teacher and school incentives based on standardized test scores.

Keywords: accountability; economics of education; educational policy; educational reform

JEL Classifications: I28, I20

Electronic copy available at: https://ssrn.com/abstract=3273422

 Introduction

Standardized test scores – primarily for math and reading – have been the state’s preferred metric

for gauging the success of school systems all across the U.S. and the rest of the world. After all,

standardized tests are convenient measures that are captured for every student every year. Test

scores are also objective measures that capture students’ cognitive abilities on math and reading

exams. However, while standardized tests may be valuable accountability metrics for schools

and teachers – in theory – they cannot capture everything that society wants from education.

The original arguments for a common system of public schools were not to maximize

math and reading test scores; the traditional public schooling model in the United States was

established to create proper citizens by teaching children how to get along with others and to

obey the established laws of society (Dewey, 1916; Mann, 1855; Rush, 1786). Obviously, a basic

level of cognitive skills is necessary for individuals to become good citizens. Citizens cannot

make an informed vote on Election Day if they cannot research the candidates or even read the

ballots. However, a system focusing on maximizing math and reading test scores may miss a lot

of what is necessary for people to become well-functioning members of society. That is,

allocating scarce educational resources to math and reading test score maximization could reduce

the level of time and resources available for improving students’ non-cognitive skills such as

behavior, tolerance-of-others, effort, and political engagement.

Education scholars such as Jay P. Greene (2016) have noticed that some studies find that

teachers and schools that have positive effects on test scores in the short-run do not have effects

on long-term outcomes such as graduation rates (Angrist et al., 2016; Tuttle et al., 2015) and

earnings (Dobbie & Fryer, 2016). On the other hand, some studies find that teachers and schools

that have little to no effects on test scores in the short-run have moderate to large positive effects

Electronic copy available at: https://ssrn.com/abstract=3273422

on long-term outcomes such as attainment (Cowen et al., 2013; Wolf et al., 2013) and crime

reduction (DeAngelis & Wolf, 2016).

This is important because many educational accountability systems for public and private

schools rely heavily on standardized tests scores. If accountability systems incentivize teachers

and schools to maximize objective measures such as test scores in the short-run, children may

receive less difficult-to-measure non-cognitive skill development. If children receive less

character development, they might miss out on honing skills that lead to better outcomes in the

long-run. If several causal studies suggest that math and reading test scores are not strong proxies

for long-term outcomes, policymakers ought to consider accountability systems that rely less on

standardized test scores.

One formal literature review exists revealing the divergences between the effects of

schools of choice on test scores and their effects on attainment (Hitt, McShane, & Wolf, 2018). I

build on this literature by expanding the divergences beyond high school graduation and college

enrollment. To my knowledge, this is the first literature review to formally present the private

school choice and value-added modeling studies that reveal divergences between effects on test

scores and effects on non-cognitive skills. I find five studies using value-added methodology

revealing discrepancies, one study using regression discontinutity design, and eleven studies of

private schooling indicating divergences.

Literature

I examine literature from quasi-experimental and experimental studies that use teacher-value

added methodology and studies that evaluate the effects of private school choice programs – and

private schooling in general – on student-level outcomes. I compile recent (published since 2000)

studies by contacting education experts who are familiar with the literature on value-added

Electronic copy available at: https://ssrn.com/abstract=3273422

methodology and private school choice programs. Studies that find discrepancies between

teachers’ or schools’ effects on students’ test scores and either non-cognitive skills or long-term

outcomes are included in the following three sections.

Value Added Modeling

Value-added models predict student outcomes using observable characteristics and prior

academic performance. Teacher value-added models determine the effect of individual teachers

on student outcomes by assessing how well their students did relative to their predicted

performance. Recent literature using value-added modeling calculates teachers’ effects on

cognitive and non-cognitive outcomes. Some of these studies have revealed disconnects in

teachers’ effects on students’ test scores and their non-cognitive skills.

This literature review reveals five recent studies using value-added methodology that find

evidence of disconnects between teachers’ effects on test scores and non-cognitive skills. As

shown in Table 1 below, I classify the disconnect found in three different ways: (1) the effects on

test scores are weakly correlated or uncorrelated with effects on non-cognitive skills, (2) the

effects on test scores are much different in size than the effects on non-cognitive skills, and (3)

the effects on test scores faded out at different rates than the effects on non-cognitive skills.

Three of the studies found a weak correlation between the different types of effects

(Cheng & Zamarro, 2018; Gerhsenson, 2016; Kraft, forthcoming), two found different effect

sizes (Jackson, 2018; Jennings & DiPrete, 2010), while one study found evidence of differential

fadeout (Gershenson, 2016). Only one of the evaluations found more than one type of disconnect

within the same study sample (Gershenson, 2016).

Gershenson (2016) found that teachers that improve test scores do not necessarily

improve attendance and their effects on test scores persist longer than their effects on student

Electronic copy available at: https://ssrn.com/abstract=3273422

absences. The reported Spearman rank correlations between absence and student achievement

rankings are near zero and negative, indicating that teachers who are great at shaping test scores

are not as good at shaping students’ noncognitive skills. Of course, teachers that are great at

teaching to the test may not be all that interesting to students. The difference in persistence found

is in accord with previous literature indicating that non-cognitive skills are more malleable than

cognitive skills (Cunha & Heckman, 2008). After all, an interesting teacher may entice students

to come their classroom, but that doesn’t mean that the same students will be interested in a

different teacher the following year. On the other hand, if a student learns real math skills one

year, they will be able to use those skills to their advantage in math class the following year.

Cheng and Zamarro (2018) and Kraft (forthcoming) also found that teachers’ effects on

non-cognitive skills were uncorrelated with their effects on test scores. Cheng and Zamarro

(2018) found that effects on non-cognitive skills, captured by survey response patterns, were

correlated with classroom observations, student and principal ratings, but not effects on student

test scores. Similarly, Kraft (forthcoming) found that survey measures of non-cognitive skills

such as growth mindset, grit, and effort were weakly correlated with test scores, at best. In fact,

most of the relationships between test scores and non-cognitive skills were not statistically

significant.

Jennings and DiPrete (2010) and Jackson (2018) found that teacher’s effects on non-

cognitive skills are larger than their effects on test scores. Jennings and DiPrete (2010) found that

moving a student from the 25th to the 75th percentile in the teacher effects distribution resulted in

around a 16 percent of a standard deviation increase in math scores, around 20 percent of a

standard deviation increase in reading scores, and about a 28 percent of a standard deviation

increase in social and behavioral skills. Jackson (2018) found that a one standard deviation

Electronic copy available at: https://ssrn.com/abstract=3273422

increase in the behavior index (i.e. value-added on GPA, on-time grade progression, absences,

and suspension) translates to much larger gains in high school graduation, crime reduction,

employment, and income than a one standard deviation increase in the test score index (i.e. the

average of 9th grade math and English scores). For example, Jackson (2018) found that a one

standard deviation increase in the test score index resulted in a 1.86 percentage point increase in

high-school graduation, while a one standard deviation increase in the behavior index resulted in

a 15.8 percentage point increase. The effect of the behavior index on high school graduation was

about 8.5 times the size of the effect of the test score index.

Regression Discontinuity Design

One study using regression discontinuity design found a divergence between effects of schools

on test scores and their effects on long-term outcomes. Beuermann and Jackson (2018) used data

from over 40,000 students in Barbados. Families in the Caribbean island are able to rank-order

their nine preferred schools for their children after 6th grade. In addition, because the Barbados

education system uses test score cutoffs to determine which students ultimately get access to

their most preferred schools, the researchers were able to use a regression discontinuity design to

arrive at causal estimates. They found that attending a preferred school did not improve

secondary school exit exam scores overall and reduced the number of tests passed for male

students. However, getting a preferred school led to long-term benefits for students, especially

for females. Indeed, by the time respondents were around 25 to 40 years old, females that were

able to attend their preferred secondary school were 17.4 percentage points more likely to have a

university degree, 59 percent less likely to have a teenage birth, and had 42 percent higher

monthly wages. Attending a preferred school increased the likelihood of attending a gym at least

once a week by 12.5 percentage points, increased the likelihood of being within a normal BMI

Electronic copy available at: https://ssrn.com/abstract=3273422

by 16.7 percentage points, and increased the chance that males were in the labor market by 11.4

percentage points.

[Table 1 about here]

Private School Choice

The literature on the effects of private schooling is abundant. Literally hundreds of studies have

evaluated the effects on private schooling on students’ academic outcomes. Fewer evaluations

have been able to determine the effects of private school choice programs – and private

schooling in general – on students’ non-cognitive skills. This review of the evidence finds eleven

disconnects (from twelve studies) between the effects of private schooling on students’ math and

reading test scores and non-cognitive outcomes such as attainment, tolerance, effort, happiness,

and civic engagement.

As shown in Table 2 below, seven rigorous evaluations of the effects of private school

choice voucher programs in the U.S. have found null to mixed effects on students test scores

while finding large positive effects on outcomes that rely on non-cognitive skills. For example,

Wolf et al. (2013) found that the D.C. Opportunity Scholarship Program (OSP) had statistically

insignificant impacts on math and reading test scores by the third and final year of the

evaluation; however, Wolf et al. (2013) also found that attending a private school through the

OSP had a substantially large 21-percentage point (30 percent) positive effect on the likelihood

of high school graduation. Similarly, Cowen et al. (2013) found statistically insignificant effects

of the Milwaukee Parental Choice Program (MPCP) on students’ reading test scores after four

years, but found that the MPCP increased their likelihood of graduation by 3 percentage points (4

percent).

Electronic copy available at: https://ssrn.com/abstract=3273422

 Of course, these disconnects are not limited to graduation. The experimental evaluation

of the D.C. Washington Scholarship Fund (WSF) by Wolf, Peterson, & West (2001) found

mixed effects on student achievement in year one, but also found evidence to suggest that the

WSF improved student tolerance by over 50 percent according to multiple survey measures.

Other evaluations using student-level data from the state-mandated evaluation of the MPCP

found that although their reading scores did not improve, they were less likely to commit crimes

(DeAngelis & Wolf, 2016), more likely to enroll in a 4-year college (Wolf, Witte, & Kisida,

2018), and exhibited higher levels of tolerance and volunteerism (Fleming, Mitchell, & McNally,

2014). In addition, an experimental evaluation of the Children’s Scholarship Fund (CSF) in Ohio

by Bettinger and Slonim (2006) found no effects on student test scores but positive effects on

student charitable activity. The experimental evaluation of the Louisiana Scholarship Program

(LSP) by Mills and Wolf (2017) found large negative effects on students’ math and reading test

scores in the first two years; however, at the same time, Mills et al. (2016) found no statistically

significant effects on students’ self-reported measures of non-cognitive skills.

Two quasi-experimental studies found effects that were in completely opposite

directions. Heller-Sahlgren (2018) used the Catholic share of countries’ populations in 1900 to

exogenously predict their private share of schooling in 2012. Using student-level data from the

2012 round of PISA, he found that private schools around the world increased student math and

reading achievement but decreased self-reported happiness in the classroom. Similarly,

DeAngelis (2018a) used student-level data from the 2009 round of PISA – and the same

instrumental variable – and found that private schools increased students’ test scores and the

effort they exerted on tests, but decreased students’ effort put forth on long PISA surveys.

[Table 2 about here]

Electronic copy available at: https://ssrn.com/abstract=3273422

 Discussion

This review of the literature on value-added and private schooling suggests that significant

disconnects between teachers’ and schools’ effects on test scores and their effects on non-

cognitive skills and long-term outcomes. Of course, this does not mean that effects on test scores

and effects on non-cognitive skills are completely unrelated. Indeed, some studies find that

teachers that influence test scores may also have an impact on their later life outcomes (Chetty,

Friedman, & Rockoff, 2014), while others find that non-cognitive skills can have indirect

positive effects on student achievement (DeAngelis, 2018a; Jennings & DiPrete, 2010). Instead,

this review suggests that teacher quality and student ability are multidimensional.

All skills exist on a sliding scale with some mix of non-cognitive and cognitive attributes

that cannot be empirically captured with absolute accuracy. Nonetheless, the evidence presented

in this review suggested that programs incentivizing teachers and schools largely based on test

scores could have unintended consequences. If a teacher or school is financially rewarded by the

state for maximizing standardized test scores through a merit pay program, for example, they

will likely shift some their resources away from activities that improve character education.

But what non-cognitive skills are most important in the long-run? Should teachers focus

more on increasing civic engagement or on improving student obedience? A limitation of this

review is that it cannot answer these questions; it only tells us that the disconnects exist in the

literature. However, it appears that private schools improve some mix of non-cognitive skills that

lead to better civic (DeAngelis, 2017) and attainment outcomes (Foreman, 2017). Furthermore,

families match their individual children to the schools and teachers that best fit their needs by

shaping the mix of cognitive and non-cognitive skills that benefit them (DeAngelis & Holmes

Erickson, 2018).

Electronic copy available at: https://ssrn.com/abstract=3273422

 Since it is impossible for experts to know what cognitive or non-cognitive skills are

needed to improve the lives of each individual student (Hayek, 1945), and because the existing

empirical evidence suggests that school choice improves long-term outcomes (DeAngelis, 2017;

Foreman, 2017; Wolf, 2007), policymakers should consider expanding access to private school

choice programs. In addition, mandating that private schools in these programs be subject to

high-stakes evaluations based primarily on standardized test scores could have unintended

consequences. After all, recent studies suggest that more highly regulated private school choice

programs may reduce the quantity (Sude, DeAngelis, & Wolf, 2018), quality (DeAngelis, Burke,

& Wolf, 2018; DeAngelis & Hoarty, 2018; Sánchez, 2018; Sude, DeAngelis, & Wolf, 2018), and

specialization (DeAngelis & Burke, 2017; DeAngelis & Burke, forthcoming; DeAngelis, 2018b)

of participating private schools.

Of course, deregulation should not only occur in private schools. Since divergences were

found in public schools in the value-added literature, policymakers should also consider

decreasing the test-based accountability in public schools of choice. Residentially assigned

public schools could benefit from deregulating testing requirements; however, some other

mechanism for quality control should be in place before deregulation occurs. One way to control

quality levels is to implement top-down accountability systems that rely less on standardized test

scores and more on non-cognitive skills; another way is to implement bottom-up accountability

by allowing families to choose the private or public schools that work best for them.

10

Electronic copy available at: https://ssrn.com/abstract=3273422

 References

Angrist, J. D., Cohodes, S. R., Dynarski, S. M., Pathak, P. A., & Walters, C. R. (2016). Stand

and deliver: Effects of Boston’s charter high schools on college preparation, entry, and

choice. Journal of Labor Economics, 34(2), 275-318.

Bettinger, E., & Slonim, R. (2006). Using experimental economics to measure the effects of a

natural educational experiment on altruism. Journal of Public Economics, 90(8-9), 1625-

1648.

Beuermann, D. W., & Jackson, C. K. (2018). Do parents know best? The short and long-run

effects of attending the schools that parents prefer. NBER Working Paper No. 24920.

Retrieved from http://www.nber.org/papers/w24920.

Cheng, A. & Zamarro, G. (2018). Measuring teacher conscientiousness and its impact on

students: Insight from the Measures of Effective Teaching Longitudinal Database. Economics

of Education Review, 64, 251-260.

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). Measuring the impacts of teachers II:

Teacher value-added and student outcomes in adulthood. American Economic Review,

104(9), 2633-79.

Chingos, M. M., & Peterson, P. E. (2015). Experimentally estimated impacts of school vouchers

on college enrollment and degree attainment. Journal of Public Economics, 122, 1-12.

Cowen, J. M., Fleming, D. J., Witte, J. F., Wolf, P. J., & Kisida, B. (2013). School vouchers and

student attainment: Evidence from a state‐mandated study of Milwaukee's parental choice

program. Policy Studies Journal, 41(1), 147-168.

Cunha, F., & Heckman, J. J. (2008). Formulating, identifying and estimating the technology of

cognitive and noncognitive skill formation. Journal of Human Resources, 43(4), 738-782.

11

Electronic copy available at: https://ssrn.com/abstract=3273422

DeAngelis, C. A., & Burke, L. (2017). Does regulation induce homogenisation? An analysis of

three voucher programs in the United States. Educational Research and Evaluation,

23(7-8), 311-327.

DeAngelis, C. A., & Burke, L. (forthcoming). Does regulation reduce specialization? Examining

the impact of regulations on private schools of choice in four locations. EdChoice.

DeAngelis, C. A., Burke, L., & Wolf, P. J. (2018). The effects of regulations on private school

choice program participation: Experimental evidence from Florida. EDRE Working

Paper No. 2018-08.

DeAngelis, C. A., & Hoarty, B. (2018). Who participates? An analysis of school participation

decisions in two voucher programs in the United States. Cato Institute Policy Analysis

No. 848. Retrieved from https://www.cato.org/publications/policy-analysis/who-

participates-analysis-school-participation-decisions-two-voucher.

DeAngelis, C. A., & Holmes Erickson, H. (2018). What leads to successful school choice

programs? A review of the theories and evidence. Cato Journal, 38(1), 247-263.

DeAngelis, C., & Wolf, P. (2016). The School Choice Voucher: A 'Get Out of Jail' Card The

School Choice Voucher: A “Get Out of Jail” Card? Retrieved from

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2743541.

DeAngelis, C. A. (2017). Do self-interested schooling selections improve society? A review of

the evidence. Journal of School Choice, 11(4), 546-558.

DeAngelis, C. A. (2018a). Does private schooling affect non-cognitive skills? International

evidence based on test and survey effort on PISA. SSRN Working Paper No. 3186771.

Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3186771.

12

Electronic copy available at: https://ssrn.com/abstract=3273422

DeAngelis, C. A. (2018b). Which schools participate? An analysis of private school voucher

program participation decisions across seven locations. Unpublished Manuscript.

Dewey J. (1916). Democracy and education. New York, NY: The Macmillan Company.

Dobbie, W. S., & Fryer Jr, R. G. (2016). Charter schools and labor market outcomes. National

Bureau of Economic Research No. w22502. Retrieved from

http://www.nber.org/papers/w22502.

Fleming, D. J., Mitchell, W., & McNally, M. (2014). Can markets make citizens? School

vouchers, political tolerance, and civic engagement. Journal of School Choice, 8(2), 213-

236.

Foreman, L. M. (2017). Educational attainment effects of public and private school choice.

Journal of School Choice, 11(4), 642-654.

Gershenson, S. (2016). Linking teacher quality, student attendance, and student achievement.

Education Finance and Policy, 11(2), 125-149.

Greene, J. P. (2016). Evidence for the disconnect between changing test scores and changing

later life outcomes. Education Next. Retrieved from http://educationnext.org/evidence-

for-the-disconnect-between-changing-test-scores-and-changing-later-life-outcomes/.

Hayek, F. A. (1945). The use of knowledge in society. American Economic Review, 35(4), 519-

530.

Heller-Sahlgren, G. (2018). Smart but unhappy: Independent-school competition and the

wellbeing-efficiency trade-off in education. Economics of Education Review, 62, 66-81.

Hitt, C., McShane, M. Q., & Wolf, P. J. (2018). Do impacts on test scores even matter? Lessons

from long-run outcomes in school choice research. American Enterprise Institute.

13

Electronic copy available at: https://ssrn.com/abstract=3273422

 Retrieved from https://www.aei.org/publication/do-impacts-on-test-scores-even-matter-

lessons-from-long-run-outcomes-in-school-choice-research/.

Jackson, C. K. (2018). What do test scores miss? The importance of teacher effects on non–test

score outcomes. Journal of Political Economy, 126(5), 2072-2107.

Jennings, J. L., & DiPrete, T. A. (2010). Teacher effects on social and behavioral skills in early

elementary school. Sociology of Education, 83(2), 135-159.

Kraft, M. A. (2017). Teacher effects on complex cognitive skills and social-emotional

competencies. Journal of Human Resources. Online version retrieved from

http://jhr.uwpress.org/content/early/2017/09/01/jhr.54.1.0916.8265R3.abstract.

Mann, H. (1855). Lectures on education. Boston, MA: WB Fowle and N. Capen.

Mills, J. N., & Wolf, P. J. (2017). Vouchers in the bayou: The effects of the Louisiana

Scholarship Program on student achievement after 2 years. Educational Evaluation and

Policy Analysis, 39(3), 464-484.

Mills, J., Cheng, A., Hitt, C., Wolf, P., & Greene, J. (2016). Measures of student non-cognitive

skills and political tolerance after two years of the Louisiana Scholarship Program.

Louisiana Scholarship Program Evaluation Report #2. Retrieved from

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2738782.

Rush, B. (1786). Thoughts upon the mode of education proper in a republic. In Rudolph, F.

(1968) (Ed.) Essays on education in the early republic. Cambridge, MA.

Sánchez, C. (2018). Understanding school competition under voucher regimes. University of

Maryland. Retrieved from http://econweb.umd.edu/~sanchez/files/csanchez_jmp.pdf.

14

Electronic copy available at: https://ssrn.com/abstract=3273422

Sude, Y., DeAngelis, C. A., & Wolf, P. J. (2018). Supplying choice: An analysis of school

participation decisions in voucher programs in Washington, DC, Indiana, and Louisiana.

Journal of School Choice, 12(1), 8-33.

Tuttle, C. C., Gleason, P., Knechtel, V., Nichols-Barrer, I., Booker, K., Chojnacki, G., ... &

Goble, L. (2015). Understanding the effect of KIPP as it scales: Volume I, impacts on

achievement and other outcomes. Mathematica Policy Research. Final report of KIPP’s

Investing in Innovation grant evaluation. Retrieved from

https://ideas.repec.org/p/mpr/mprres/7d8e94c5e77a4a9c8bf09000d6cc4456.html.

Wolf, P. J. (2007). Civics Exam Schools of choice boost civic values. Education Next, 7(3), 66.

Wolf, P. J., Kisida, B., Gutmann, B., Puma, M., Eissa, N., & Rizzo, L. (2013). School vouchers

and student outcomes: Experimental evidence from Washington, DC. Journal of Policy

Analysis and Management, 32(2), 246-270.

Wolf, P., Peterson, P. E., & West, M. (2001). Results of a school voucher experiment: The case

of Washington, DC after two years. KSG Working Paper No. RWP02-022. Retrieved

from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=313822.

Wolf, P., Witte, J., & Kisida, B. (2018). Do Voucher Students Attain Higher Levels of

Education? Extended Evidence from the Milwaukee Parental Choice Program. Urban

Institute Research Report. Retrieved from

https://www.urban.org/research/publication/do-voucher-students-attain-higher-levels-

education/view/full_report.

15

Electronic copy available at: https://ssrn.com/abstract=3273422

Table 1: Studies with Disconnects Between Effects on Test Scores and Non-Cognitive Skills

Study Method Disconnects Found Weak Effect Persistence

Correlation Size
Jennings & Value-Added Teachers’ effects on non-
DiPrete (2010) cognitive skills are larger X
than their effects on test
scores.
Gershenson Value-Added Teachers that improve
(2016) test scores do not X X
necessarily improve
attendance. Teachers’
effects on attendance
persist less than test score
effects.
Cheng & Value-Added Survey measures are
Zamarro correlated with classroom X
(2018) observations and
student/principal ratings,
but not test scores.
Kraft Value-Added Open-ended math and
(forthcoming) reading tasks and
measures of growth
mindset, grit, and effort X
are weakly related to test
score value-added. The
majority of correlations
between cognitive and
non-cognitive skill
effects are not
statistically significant.
Jackson (2018) Value-Added Non-cognitive skill
(behavior index) effects X
were much larger than
test score effects on long-
term outcomes
Beuermann & Regression Attending preferred X
Jackson (2018) Discontinuity school does not increase
Design test scores but largely
improves long-term
outcomes such as
attainment, earnings, and
health.

16

Electronic copy available at: https://ssrn.com/abstract=3273422

 Table 2: Studies with Disconnects Between Effects on Test Scores and Non-Cognitive Skills

Studies Program Disconnects Found Test Score Non-Cognitive

Effect Effect
Wolf, Peterson, & D.C. WSF Mixed effects on test scores in Mixed +
West (2001) year one, large positive effects
on student tolerance
Wolf et al. (2013) D.C. OSP No effect on test scores after Null +
three years, 21 percentage point
increase in high school
graduation.
Cowen et al. (2013) MPCP No effect on reading test scores Null +
after four years, positive effects (Reading)
on high school graduation and
enrollment in a 4 year college.
Fleming, Mitchell, & MPCP No effect on reading test scores Null +
McNally (2014) after four years, positive effects (Reading)
on political tolerance, civic
skills, future political
participation, and volunteering.
Cowen et al. (2013); MPCP No effect on reading test scores Null +
DeAngelis & Wolf after four years, large positive (Reading)
(2016) effects on crime reduction.
Wolf, Witte, & MPCP No effect on reading test scores Null +
Kisida (2018) after four years, positive effects (Reading)
on enrollment in a 4 year
college.
Bettinger & Slonim CSF – No effects on test scores, Null +
(2006) Ohio positive effects on altruism.
Chingos & Peterson NYC Modest positive effects on test + +
(2015) SCSF scores, very large positive (Modest) (Large)
effects on college enrollment
(for African American students)
Mills et al. (2016); LSP Large negative effects on test - Null
Mills & Wolf (2017) scores after two years, no effects
on self-reported measures of
non-cognitive skills at the same
time.
Heller-Sahlgren World Positive effects on test scores, + -
(2018) negative effects on happiness at
school.
DeAngelis (2018a) World Positive effects on test scores + Mixed
and test effort, negative effects
on survey effort
Notes: “+” means positive effects, “-“ means negative effects, “null” means no statistically
significant effects were found, and “mixed” means the overall results were not clear.

17

View publication stats Electronic copy available at: https://ssrn.com/abstract=3273422