Jel 20191507

Journal of Economic Literature 2020, 58(4), 897–996
https://doi.org/10.1257/jel.20191507
Upgrading Education with Technology:

Insights from Experimental Research†
Maya Escueta, Andre Joshua Nickow, Philip Oreopoulos,
and Vincent Quan*
In recent years, there has been widespread interest around the potential for technology
to transform learning. As investment in education technology continues to grow, stu-
dents, parents, and teachers face a seemingly endless array of education technologies
from which to choose—from digital personalized learning platforms to online courses
to text message reminders to submit financial aid forms. Amid the excitement, it is
important to step back and understand how technology can help—or in some cases
hinder—learning. This review article synthesizes and discusses rigorous evidence on
the effectiveness of technology-based approaches to education in developed countries
and outlines areas for future inquiry. In particular, we examine randomized controlled
trials and regression discontinuity studies across the following categories of educa-
tion technology: (i) access to technology, (ii) computer-assisted learning, (iii) technol-
ogy-enabled behavioral interventions in education, and (iv) online learning. We hope
this synthesis will advance academic understanding of how technology can improve
education, outline key areas for new experimental research, and help drive improve-
ments to the policies, programs, and structures that contribute to successful teaching
and learning. ( JEL H52, H75, I20, O33)
1. Introduction e ducation landscape. Revolutionary advances

in information and communications tech-
T echnological innovation over the past two

decades has indelibly altered today’s
nology (ICT)—particularly associated with
computers, mobile phones, and the internet
* Escueta: Columbia University. Nickow: Northwestern Oscar Sweeten-Lopez for providing helpful and detailed
University. Oreopoulos: University of Toronto and NBER. comments as we put together this review. We also thank
Quan: Abdul Latif Jameel Poverty Action Lab (J-PAL). We Rachel Glennerster for detailed support throughout the
are extremely grateful to Caitlin Anzelone, Rekha Balu, project, Jessica Mardo and Sophie Shank for edits, and to
Peter Bergman, Brad Bernatek, Ben Castleman, Angela the Spencer Foundation for financial support. Any errors
Duckworth, Jonathan Guryan, Alex Haslam, Andrew Ho, or omissions are our own. The views expressed herein are
Ben Jones, Matthew Kraft, Kory Kroft, David L aibson, those of the authors and do not necessarily reflect the views
Susanna Loeb, Andrew Magliozzi, Ignacio Martinez, of the National Bureau of Economic Research.
†
Susan Mayer, Steve Mintz, Piotr Mitros, Lindsay Page, Go to https://doi.org/10.1257/jel.20191507 to visit the
John Pane, Justin Reich, Jonah Rockoff, Kirby Smith, and article page and view author disclosure statement(s).
897
898 Journal of Economic Literature, Vol. LVIII (December 2020)
—have precipitated a renaissance in edu- roduction function, as technology has done

p
cation technology (ed-tech), a term we use in other sectors.
here to refer to any ICT application that aims But the rapid proliferation of new tech-
to improve education. In the United States, nologies within education has proved to be
the market for PreK–12 software alone has a double-edged sword. The speed at which
exceeded $8 billion (SIIA Communications new technologies and intervention models
2015), and a recent industry report projects are reaching the market has far outpaced
an estimated value of $252 billion for the the ability of policy researchers to keep up
global ed-tech industry by 2020 (Morrison with evaluating them. The situation is well
2017). Governments, schools, and families summarized by a recent headline: “Ed-Tech
increasingly value technology as a central part Surges Internationally—and Choices for
of the education process, and invest accord- Schools Become More Confusing” (Molnar
ingly (Bulman and Fairlie 2016). In the 2017). While most agree that e d-tech can be
coming years, emerging fields like machine helpful under some circumstances, research-
learning, big data, and artificial intelligence ers and educators are far from a consensus on
will likely compound the influence of these what types of ed-tech are most worth invest-
technologies even further, expanding the ing in, in which contexts, and for which pop-
already dizzying range of available education ulations. At the same time, there is reason to
products and speeding up cycles of learning believe that some uses of technology could
and adjustment. potentially be harmful, and many parents are
Collectively, these technologies offer the growing increasingly concerned that screen
potential to open doors and build bridges time may distract from student learning and
by expanding access to quality education, development or that technology could dis-
facilitating communication between edu- place the role of the teacher (Bowles 2019).
cators, students, and families, and alle- Furthermore, the transformations associ-
viating frictions across a wide variety of ated with ed-tech are occurring in a context
educational contexts from early childhood of deep and persistent inequality. Despite
through adulthood. For example, educa- expanding access to some technologies, the
tional software developers work to enable digital divide remains very real and very wide.
educators to deliver the latest learning sci- While 98 percent of children in US house-
ence advances to schools in inner cities and holds with incomes exceeding $100,000 per
remote rural areas alike. The proliferation of year have a computer at home, only 67 per-
cell phones and growing ease in connecting cent of children in households with incomes
them to internet-based information systems lower than $25,000 have them (Bulman and
has enabled the scaling of automated text Fairlie 2016). Even when disadvantaged stu-
messaging systems that aim to inform, sim- dents can physically access technology, they
plify, and encourage students and their par- may lack the guidance needed for productive
ents as they traverse difficult sticking points utilization—a “digital-use divide” (Brotman
in education, like the transition from high 2016). Depending on design and implemen-
school to college. And online educational tation, education technologies could alleviate
institutions may bring opportunities to earn or aggravate existing inequalities. For at-risk
degrees to students who would otherwise be students in particular, there are real con-
constrained by work, families, disabilities, cerns around the opportunity costs of receiv-
or other barriers to traditional higher edu- ing instruction through technology rather
cation. Taken together, technology may play than through an in-person teacher or tutor.
a significant role in shaping the education Equity considerations thus add another layer
Escueta et al.: Upgrading Education with Technology 899
to the need for caution when implementing To position these findings within the
technology-based education programs. broader economics literatures on education
Of course, not every intervention model and human capital development, we orga-
can be evaluated, and the extent of suc- nize them according to a framework based
cess inevitably varies across educational on Cunha and Heckman’s “technology of
approaches and contexts even within skill formation” model (Cunha et al. 2006,
well-established fields. But the speed and Cunha and Heckman 2007, see also figure
scale with which many ed-tech interventions 28 in Almlund et al. 2011). This model posits
are being adopted, along with the enormous that the “skill formation process is governed
impact they could have over the next genera- by a multistage technology” spanning the
tion, demand a closer look at what we know. educational life cycle, and that, at each stage,
To confront this issue, the present review individuals “possess a vector of abilities…
takes stock of rigorous quantitative studies [that] are multiple in nature and range from
on technology-based education interven- pure cognitive…to noncognitive” (Cunha
tions that have been conducted so far, with and Heckman 2007). An individual’s abili-
the goals of identifying insights for the eco- ties and skills at a given stage are, in turn,
nomics of education and highlighting key a function of cognitive and noncognitive
areas for future inquiry. In particular, for rea- skills from the preceding stage in combina-
sons explained in the following section, we tion with environmental factors, and past
assembled what we believe to be a compre- investment in education or other forms of
hensive list of all publicly available studies on human capital. We define cognitive skills as
technology-based education interventions
the core skills that are used to think, read,
that report findings from studies following and learn. On the other hand, n oncognitive
either of two research designs, randomized skills are traits such as personality, social, and
controlled trials or regression discontinuity emotional traits that are now increasingly
designs, and based our analyses primarily acknowledged as being important contribu-
on these studies.1 The studies are clustered tors to learning.
into four ed-tech domains, which generally Figure 1 illustrates how we adapt this
focus on distinct families of tools, logics, and framework to the ed-tech context. We
approaches: (i) access to technology, that is, narrow the focus from human capital in
opportunities to use computers, software, general to educational advancement in par-
and other forms of ICT with the potential ticular and focus on the role of ed-tech in
for use with education; (ii) computer-assisted improving environmental conditions or the
learning (CAL), that is, computer programs efficiency of direct investments in cognitive
and other software applications designed to or noncognitive skills at different life-cycle
improve academic skills; (iii) online courses, stages (early childhood, elementary, second-
that is, classes offered by educational institu- ary, and postsecondary). E d-tech consists of
tions partially or entirely through the inter- those technologies that rely on ICT. In this
net; and (iv) behavioral interventions, that framework, ed-tech products are success-
is, nudges, reminders, and other ICT-based ful to the extent that they increase the effi-
programs that aim to overcome psychologi- ciency of investment in skills relative to the
cal or sociological barriers. next best technology (whether I CT-based or
traditional) since “expenditures devoted to
1 For simplicity, and as explained further in the follow-
technology necessarily offset inputs that may
ing section, we refer to studies employing these designs as be more or less efficient, and time allocated
“experimental research.” to using technology may displace traditional
Figure 1. Technology of Skill Formation in the Education Technology Context
Source: Authors, inspired by Almlund et al. (2011), figure 28.
classroom instruction and educational activ- improve or fill in gaps caused by noncognitive
ities at home” (Bulman and Fairlie 2016). skill deficits. By exploring the results of the
We hone in on the role technology may play experimental research on e d-tech in the con-
in overcoming longstanding constraints in text of this framework, we hope to shed light
education. on how technology can resolve important
Ed-tech products may address cognitive, frictions and binding constraints that impact
noncognitive, or environmental factors at one the education production function and, ulti-
or more educational life-cycle stages. The mately, the accumulation of human capital.
domains we examine tend to differ, however, In the next section, we discuss our liter-
with regard to the inputs they focus on in ature review methodology in greater depth.
each stage. In particular, access-to-technol- Sections 3–6 constitute the core of the review,
ogy interventions tend to focus on improving respectively synthesizing the evidence on
environmental factors, while CAL tends to access to technology, C AL, online courses,
focus on cognitive skills. Online courses, on and behavioral interventions. In each of these
the other hand, may focus on both improv- sections, we discuss background and context
ing environmental factors—by expanding for the e d-tech domain in question; consider
the environment where instruction is deliv- how ed-tech programs within the domain
ered—and supporting cognitive skill for- seek to mitigate constraints and improve the
mation—by varying how content is taught. efficiency of investments in cognitive skills,
Behavioral interventions typically attempt to noncognitive skills, or environmental factors
at each educational stage; review the exper- RDDs are quasi-experiments that identify a
imental evidence on programs fitting into well-defined cutoff threshold which defines
the domain; and reflect on implications for a change in eligibility or program status for
future research, theory, and policy. Section 7 those above it—for instance, the minimum
offers concluding observations and consid- test score required for a student to be eligi-
ers several of the priority areas for future ble for financial aid. While very h igh-scoring
research that we consider vital to ongoing and very low-scoring students likely differ
efforts at more effectively and equitably from one another in ways other than their
leveraging technology for learning. eligibility for financial aid, “it may be plausi-
ble to think that treatment status is ‘as good
as randomly assigned’ among the subsample
2. Literature Review Methodology
of observations that fall just above and just
While a few existing reviews have cov- below the threshold” (Lee and Card 2008).
ered subsets of ed-tech, no recent review So, when some basic assumptions are met,
has attempted to cover the full range of the jump in an outcome between those just
ed-tech interventions (Bulman and Fairlie above and those just below the threshold
2016; Lavecchia, Liu, and Oreopoulos 2016; can be interpreted as the causal effect of
Means et al. 2009). In particular, no previous the intervention in question for those near
review, to our knowledge, brings together the threshold (Imbens and Lemieux 2008,
computer- and internet-based learning on Thistlewaite and Campbell 1960). One
one hand with technology-based behavioral caveat is that, while RCTs estimate the aver-
interventions on the other. We also focus on age treatment effect for the entire sample,
studies presenting evidence from random- RDDs measure the “local average treatment
ized controlled trials (RCTs) and regression effect” (Imbens and Angrist 1994), that
discontinuity designs (RDDs). Our core is, the treatment effect for cases near the
focus on RCT- and RDD-based studies con- threshold.
stitutes a second unique contribution of this We chose to include RDDs, but not other
review—we argue that, in addition to helping quasi-experimental designs, because of their
us define sufficiently clear and narrow inclu- minimal sensitivity to underlying theoretical
sion conditions, a focus on RCTs and RDDs assumptions and the fact that RDDs with
adds a productive voice to broader and more large samples and well-defined thresholds
methodologically diverse policy research
produce estimated program effects iden-
dialogues in an environment characterized tical to RCTs for participants at the cutoff
by complex tangles of cause and effect. (Berk et al. 2010, Cook and Wong 2008,
Why focus on RCTs and RDDs? In the Shadish et al. 2011). Although RDDs are
field of applied microeconomics, RCTs— quasi-experiments, in the remainder of this
when properly implemented—are generally review we refer to the RCTs and RDDs
considered the strongest research design included in this review as experimental
for quantitatively estimating average causal research for simplicity. We chose to focus on
effects (Angrist and Pischke 2008). Because it experimental research not only because we
is not always possible to conduct RCTs, meth- believe it can often be more convincing than
odologists have, over the past several decades, other types of research for causal inference,
developed a toolkit of research designs such but because we felt that the economic litera-
as instrumental variables, d ifference in dif- ture on e d-tech is flooded with observational
ferences, and propensity score designs, research and could benefit from a synthe-
known broadly as q uasi-experiments. sis of evidence from the designs most likely
to produce unbiased estimates of causal like technology-based behavioral interven-

effects. Furthermore, we introduce, frame, tions, is less developed there so far.
and interpret the experimental results in the Our first task in constructing this review
context of broader observational literatures. was thus to collect all publicly available
Experimental evaluation studies estimate studies using RCT or RDD designs within
the impact of a program or policy on out- developed countries that estimate the
comes of interest. But the estimates they effects of an ed-tech intervention on any
come up with are sometimes difficult to education-related outcome. To locate the

compare with one another, given that studies studies, we assembled a list of search terms,
test for impact on different outcomes using and used these to search a range of academic
different measurement tools, in populations search engines, leading economics and edu-
that differ in their internal diversity. While cation journals, and evaluation databases.
these differences cannot be completely elim- To ensure that no relevant studies had been
inated and effect sizes must always be con- omitted, we followed backward and forward
sidered in the contexts within which they citations for all included articles.
were identified, standard deviations offer The “file drawer” problem—the notion
a roughly comparable unit that can give us that studies with significant results are rel-
a broad sense of the general magnitude of atively more likely to be published, while
impact across program contexts. We thus studies showing null results tend to be filed
report effect sizes in standard deviations away—presents a perennial challenge for
whenever the relevant data are available literature reviews oriented toward impact
below in order to facilitate comparison, evaluations. While no review can fully cir-
while cautioning that these effect sizes must cumvent this challenge, we took steps to
be considered in context to be meaningful. minimize its presence within this article. In
We also limited our core focus to stud- particular, we chose not to exclude any stud-
ies conducted within developed countries, ies based on publication status. Our final list
although we touch on research conducted thus consists of published academic articles,
in developing countries where relevant to working papers, evaluation reports, and
the discussion. After considering both litera- unpublished manuscripts. Furthermore, we
tures, we determined that the circumstances conducted extensive consultations with lead-
surrounding the ed-tech interventions that ing researchers, evaluators, and practitioners
have so far been experimentally studied in the field, asking each about every study
differed too greatly across developed and that s/he was aware of in his or her area of
developing country education systems to specialization, whether or not the study was
allow for integrating findings from both in published or unpublished, and whether its
a way that would yield meaningful policy findings were significant or null. The file
implications. Our decision to focus on the drawer problem may extend beyond publi-
developed rather than developing world in cation bias, in that papers may not even be
particular was driven by this review’s goal written up if null results are detected. While
of analyzing experimental research on the we cannot entirely rule out this possibility,
full range of ed-tech interventions. While we believe we have taken all feasible steps
experimental policy and evaluation litera- to avoid it, and that our approach is certainly
ture on certain classes of ed-tech literature, more effective in doing so than the modal lit-
like computer distribution and C AL, have erature review approach of solely performing
already begun to flourish in the developing keyword searches within databases that con-
world, experimental research on other areas, sist entirely of published articles. A related
problem is that researchers may emphasize with the best of our interpretation and leave
the most significant findings in their studies it to readers to make a final determination
such that interventions seem more effec- or to await future research. While quantita-
tive than they actually are. We addressed tive techniques of m eta-analysis have grown
this problem by closely reading each article in popularity and sophistication in recent
and considering the broader constellations years, we felt that these techniques would
of estimates, rather than solely the headline not be appropriate for this paper because
findings, and integrating these consider- of our goals and the nature of the literature.
ations into our review as relevant. Quantitative meta-analytic techniques work
Once the articles had been assembled, well when studies fall into relatively dense
we divided them into the four categories clusters with comparable outcomes and sam-
into which we felt that they most natu- ples. However, with a few exceptions, the
rally clustered: access to technology, CAL, subliteratures reviewed here are relatively
technology-based behavioral interventions
thin and internally diverse. And, as explained
in education, and online courses. Although above, our goal was to bring together, within
not all studies fit neatly into these categories a single framework, the full range of ed-tech
and there is some overlap, we felt that these interventions that have been experimentally
four best encapsulated the differences in the evaluated. Thus, we instead put forward a
studies’ underlying themes, motivations, and qualitative discussion, with the goals of inte-
theories of change. The full list of studies is grating the studies into a broader economic
contained—separated by category—in tables framework, charting important findings and
1 through 4D. gaps, and distilling overarching lessons. In
Within each category, we closely read all the remainder of the review, we present the
studies and organized them further accord- results of this analysis.
ing to the approach of the intervention
evaluated. We then considered each study’s
3. Access to Technology
findings in light of the others’, taking into
account to the greatest extent possible vari-
3.1 Background, Context, and Mechanisms
ations in both the nature of the programs
of Impact
evaluated, the contexts in which they are
implemented, and the research design spe- A natural starting point when exploring
cifics. Although we provide counts of the the effects of ed-tech is to consider what
numbers of studies falling into different cat- happens when students are provided with
egories, this is done to provide readers with increased access to computers or the inter-
a view of the research landscape—we do not net. Since the acceleration in ed-tech first
make inferences based on “vote counting,” began in earnest during the 1990s, govern-
that is, deciding on whether a particular ments and other stakeholders have invested
intervention model is effective when there substantial resources in an array of computer
are more studies with positive than null or and internet distribution and subsidy initia-
negative findings, or vice versa. Instead, we tives. While the handful of studies that have
consider the big picture of research design, been conducted so far on such initiatives are
the nature of the intervention itself, and far from sufficient to judge the effective-
sample characteristics when putting for- ness of these efforts, they do suggest some
ward substantive arguments. Where findings early lessons. Access programs evaluated
from the literature are ambiguous or incon- thus far have been effective at increasing
clusive, we lay out central parameters along use of computers and improving computer
skills. These outcomes alone are notewor- underrepresented minority students dispro-
thy, given the logistical challenges of tech- portionately lack access: only 78 percent of
nology d istribution—particularly within African American and Hispanic schoolchil-
lower-capacity and otherwise disadvantaged dren have computers at home, in contrast to
delivery contexts—and given the poten- 92 percent of white schoolchildren (Bulman
tial reluctance of students and educators to and Fairlie 2016, p. 263). There is also a stark
change their routines by incorporating the technology access divide between rural and
technologies. Effects on academic achieve- urban areas (West and Karsten 2016).
ment and other learning outcomes, however, Several program models have emerged to
have been mixed at best. But the research address these gaps in access to technology.
suggests some potential areas of promise One model that has recently risen to promi-
here as well, particularly computer distribu- nence has been “one-to-one” technology, “in
tion at the postsecondary level and distribu- which all the students in a class, grade level,
tion at the K –12 level when combined with school, or district are provided computers
additional learning software. for use throughout the school day and, in
A large and growing share of students in some cases, at home” (Zheng et al. 2016).
developed countries can now access com- Several one-to-one initiatives have been
puters with h igh-speed internet at home implemented at large scales. For instance,
and at school. Today, nearly three-quarters the state of Maine provides all of its middle
of American adults have broadband access and high school students with laptops for
at home—a remarkable increase from only use during the school year.2 More recently,
one percent of adults in 2000 (Pew Research some school districts around the country
Center 2017). Among adults with children, have been pairing students up with tablets
the rate of at-home broadband access is even (McLester 2012). One-to-one distribution
higher. A 2015 Pew Research Center study has also caught on within developing coun-
found that 82.5 percent of American house- tries, and governments as diverse as those of
holds with school-age children have broad- Peru, Kenya, Turkey, and India have invested
band access (Horrigan 2015). in variations of such programs (Trucano
But damaging holes in coverage remain. 2013, BBC 2013, Simhan 2011). One par-
Approximately 5 million school-age children ticularly prominent c ivic-led one-to-one ini-
do not have a broadband internet connec- tiative has been the One Laptop per Child
tion at home (Anderson and Kumar 2017), (OLPC) program, which aims to “empower
potentially leading to a “homework gap” the children of developing countries to learn
(Kang 2016) and other compounding layers by providing one connected laptop to every
of disadvantage. Students without comput- school-age child.”3 OLPC has distributed
ers or internet are likely to be the students laptops to disadvantaged students in roughly
who could most benefit from a boost in a dozen developing countries, along with two
human capital, as they are much more likely US cities.4
to come from lower-income households: “In Other initiatives have provided schools
the United States, 98 percent of the 12 mil- with subsidies to buy computers or s oftware,
lion schoolchildren living in households with
$100,000 or more in income have access to
a computer at home, but only 67 percent 2 Maine Learning Technology Initiative (MLTI). http://
of the 12 million schoolchildren living in maine.gov/doe/mlti/about/history/index.html.
3 One Laptop per Child: http://laptop.org/en/vision/
households with less than $25,000 in income mission/.
have access” (Bulman and Fairlie 2016). And 4See http://one.laptop.org/stories.
or to acquire or improve internet connec- for other learning activities. Much as access
tions. In 1997, the US federal government to books represented an important feature
launched its largest-ever program to con- of the educational environment throughout
nect US schools and classrooms to the the twentieth century, access to computers
internet. Known as E-Rate, the program and internet will likely represent an import-
has connected 97 percent of US class- ant feature of educational environments in
rooms to the internet. In 2013, President the twenty-first century. On the other hand,
Barack Obama announced a new initiative access to computers and the internet could
known as ConnectED, which sought to harm learning outcomes if they lead students
bring high-speed broadband to 99 percent to spend time on n oneducational games or
of K–12 students by 2018 (Benton Institute other leisure activities that take time away
for Broadband & Society 2013). The initia- from studying.
tive helped provide an additional 20 million
3.2 Experimental Evidence
students5 with in-classroom access to broad-
band. Both the private and public sectors While access to technology is clearly a nec-
have invested heavily to increase broadband essary condition for the operation of e d-tech,
access around the country. Since 2009, more the extent to which technology access pro-
than 115,000 miles of network infrastructure grams result in measurably improved out-
have been built at a cost of more than $260 comes remains an empirical question. Given
billion (Council of Economic Advisers 2016). the wave of investment and policy interest
Unlike the other ed-tech domains dis- in access to technology, what have been
cussed in this article, interventions focus- the effects of access programs? With only
ing on access to technology do not directly a handful of RCT and RDD studies on the
seek to alter the skill formation process subject, the experimental literature on its
itself. Instead, as illustrated by their place- own cannot say much definitively. However,
ment within our framework, these inter- these studies provide valuable suggestive
ventions aim to help create an environment insights, particularly when viewed within the
that is more favorable to skill formation. As context of the broader quasi-experimental
an increasing share of education in and out and observational literatures.
of the classroom occurs through ICT, avail- In particular, we identified 13 experimen-
ability of the hardware and software through tal papers that we coded as studying access
which ICT operates becomes increasingly to technology interventions—nine report-
vital to children’s and students’ educational ing on RCTs, and four reporting on RDDs.
environments. In particular, availability of These papers are laid out in table 1. Of the
ICT may overcome traditional constraints nine papers reporting on RCTs, eight are
associated with how we access information authored or coauthored by economist Robert
and how quickly we can access information. W. Fairlie and are based on two impact
For example, rather than going to the library evaluations. Three of the papers (Fairlie
to check out a book, many students today can and Robinson 2013, Fairlie and Kalil 2017)
simply download a book from the internet, report findings from a relatively large-scale
reducing time costs associated with doing impact evaluation of a laptop distribution
homework and freeing up time and resources program for sixth to tenth graders, with a
sample of 1,123 students in 15 schools cov-
5 See the Obama White House Archives, https://
ering five school districts across California.
obamawhitehouse.archives.gov/issues/education/k-12/ The remaining five (Fairlie 2012a, b; Fairlie
connected. and Bahr 2018; Fairlie and London 2012;
906
TABLE 1
Access to Technology
Standardized effect Sample Unit of

Author(s) Publication Intervention Direction of effect size Sample size randomization
Carter, Economics Prohibiting use Positive effect on 0.18 standard 50 postsecondary 726 students in Classroom
Greenberg, and of Education of computers final exam scores deviations classrooms and 726 50 classrooms
Walker (2017) Review during a college students in West
economics class Point, New York
Faber, Sanchis- NBER Working Differences Null effects on Null Primary and 4.5 million Not randomized—
Guarner, and Paper in broadband student time spent secondary school students spatial regression
Weinhardt connection studying or learning students living in in 580,000 discontinuity using
(2015) speeds productivity residential postcodes postcodes in postcodes within
in England England 1 km of telephone
exchange system
boundaries
Fairlie (2012a) Economics One-to- 1. Null effect on Main reported Community college 286 students Student
of Education one laptop GPA effect sizes (not students receiving
Review distribution 2. Positive impact on standardized): financial aid in
course completion 1. Null on GPA California
rates and course 2. 6.5 percentage
success rate point difference in
3. Null effect on course completion
graduation rate rates
3. 8.6 percentage
point difference for
course success rate
Fairlie (2012b) Information One-to- Positive effect on Main reported Community college 286 students Student
Economics and one laptop high-level computer effect sizes (not students receiving
Policy distribution skills, especially standardized): financial aid in
Journal of Economic Literature, Vol. LVIII (December 2020)
among young, 17 percentage point California

minority, low- difference
income, and female
students
(Continued)
TABLE 1
Access to Technology (Continued)
Fairlie (2015) B.E. Journal One-to- Null results on Null Children enrolled 1,123 students Student
of Economic one laptop academic outcomes in grades 6–10 in
Analysis & distribution 15 different middle
Policy and high schools in
5 school districts in
California
Fairlie and Bahr Economics One-to- Null results on Null Community college 286 students Student
(2018) of Education one laptop earnings students receiving
Review distribution financial aid in
California
Fairlie and Economic One-to- Positive effect on Main reported Community college 286 students Student
Grunberg (2014) Inquiry one laptop transferable courses effect sizes (not students receiving
distribution enrollment; null standardized): financial aid in
results on actual 1. 4.5 percentage California
transfer to 4-year point difference
colleges on enrollment in
courses that are
transferable to 4-year
colleges
2. Null
Fairlie and Kalil Economics One-to- 1. Null effect on 1. Null Children enrolled 1,123 students Student
(2017) of Education one laptop social development 2. 9 percentage in grades 6–10 in
Review distribution 2. Positive effect point increase in 15 different middle
on online social having an online and high schools in
Escueta et al.: Upgrading Education with Technology
networking social networking site 5 school districts in

3. Positive effect (not standardized) California.
on in-person friend 3. 0.10 standard
interaction deviation increase
in friends outcome
index
(Continued)
907
908
TABLE 1

Fairlie and Economic One-to- Positive effect on 0.14 standard Community college 286 students Student
London (2012) Journal one laptop a “summary index deviations students receiving
distribution of educational financial aid in
outcomes” that California
includes variables
like grades and
degree completion,
especially for
students who live
farther from campus
or have a job
Fairlie and American One-to- Null results Null Sixth–tenth graders 1,123 students Student
Robinson (2013) Economic one laptop in 15 middle and
Journal: distribution high schools in 5
Applied districts in California;
Economics vast majority of
sample is middle
school students
Goolsbee and Review of E-Rate, subsidy Null results on Main reported California public 31,240 Not randomized—
Guryan (2006) Economics and for internet in academic outcomes; effect sizes (not schools (primary, classrooms with regression
Statistics schools Positive effect on standardized): middle, and high internet access discontinuity-based
internet connectivity 1. Null on academic schools) design
outcomes
2. 68 percent
increase in
internet-connected
classrooms per
teacher
(Continued)
TABLE 1

Leuven et al. Review of Subsidies for Null results Main reported Primary schools in 267 schools in Not randomized—
(2007) Economics and computers effect sizes (not the Netherlands 1998, and 551 regression
Statistics and software standardized): that had at least 70 schools in 1999 discontinuity design
in under- 1. Null (seems percent of pupils
resourced especially belonging to the
schools detrimental for girls) disadvantaged
minority
Malamud and Quarterly Euro 200 1. Negative effect 1. 0.25–0.33 Households from 3,354 Not randomized—
Pop-Eleches Journal of program, on math/English/ standard deviation several regions of households regression
(2011) Economics subsidy for Romanian decrease in math/ Romania (primary discontinuity design
low-income 2. Positive effect English/Romanian and secondary
families with on computer skills 2. 0.27 standard students)
schoolchildren 3. Positive effect deviation increase
to buy on cognitive skills in computer skills
computers (not robust in all 3. 0.33 standard
specifications) deviation increase in
cognitive skills
909
Fairlie and Grunberg 2014) report on a study on laptop distribution in California

smaller evaluation of a laptop distribution found p recisely estimated null effects
program in a single California community on learning outcomes. In particular, no
college with a sample of 286 students on significant impact—positive or negative—

financial aid. Another field experiment esti- was found on homework time, grades, stan-
mates the effect of prohibiting laptop use dardized test scores, attendance, or several
during class time in a West Point econom- other outcomes, leading the authors to con-
ics course (Carter, Greenberg, and Walker clude that “increasing access to home com-
2017), and the remaining four are RDD puters among students who do not already
studies estimating the impact of access to have access is unlikely to greatly improve
computers (Leuven et al. 2007, Malamud educational outcomes, but is also unlikely
and Pop-Eleches 2011) or internet (Faber, to negatively affect outcomes” (Fairlie and
Sanchis-Guarner, and Weinhardt 2015; Robinson 2013). At the same time, the
Goolsbee and Guryan 2006) on a variety of estimated average effects may be masking
education-related outcomes. heterogeneous outcomes, with some stu-
Despite the differences in interventions dents benefiting from receiving access to
and settings explored within the studies, the computers and others doing worse. Perhaps
papers consistently report success in pro- even more concerning are the findings from
grams’ intended proximate outcomes—dis- an RDD study of subsidized computers
tributing computers, increasing time spent for households in Romania (Malamud and
using computers, or decreasing time spent Pop-Eleches 2011) and another that subsi-
accessing computers (e.g., less time wait- dized schools in purchasing computers and
ing for computers in labs to become avail- software in the Netherlands (Leuven et
able). For example, in the California field al. 2007). Both found negative impacts on
experiment—the only large-scale random- achievement outcomes, likely in part a result
ized evaluation to date of laptop distribution of the students spending more time play-
programs—computer ownership reportedly ing games. However, the negative effects
increased by 55 percentage points among in the Netherlands study are minor, and
treatment students, computer usage report- in the Romania study negative impacts on
edly increased by 2.5 hours per week, and academic achievement are accompanied by
the likelihood of at-home internet connection positive impacts on computer skills and cog-
increased by 25 percentage points relative to nitive test scores. RDD studies that respec-
those who were not assigned to receive free tively looked at effects of the E
-Rate school
laptops (Fairlie and Robinson 2013). These internet subsidy program in the United
findings are noteworthy considering that States (Goolsbee and Guryan 2006) and
the significant resources required to expand coincidental differences in connection speed
computer and internet access may be wasted in England (Faber, Sanchis-Guarner, and
because of the logistical difficulties of distri- Weinhardt 2015) similarly found no evidence
bution. And students and teachers facing con- of substantial positive or negative impact on
straints on time and cognitive capacity may be academic achievement.
reluctant to adopt technologies in the ways On the other hand, one randomized eval-
intended by providers. uation of laptop distribution for students
On the other hand, findings of effects on on financial aid in a community college
learning outcomes have been more mixed, finds positive results on academic out-
with little evidence to suggest that technol- comes. As reported in five papers, this study
ogy access alone will improve education. The observed modest but positive effects, with
an overall impact on an academic perfor- to use computers may suggest that, to the
mance index of 0.14 standard deviations extent that there are resources for expanding
(Fairlie and London 2012). The academic access to technology, it may be most effective
performance index is a measure the authors to invest them in contexts where the technol-
constructed to aggregate four separate out- ogy is clearly integrated into an educational
comes: course success rate, the likelihood of curriculum, but where accessing the tech-
taking a course for a grade, the likelihood of nology is somehow costly to students.
taking a transfer course for a four-year college, Where do these findings from the exper-
and graduation rate. Analyses suggested that imental literature in developed countries
the benefits occurred not by increasing the stand within the broader literature on
time that students spend using computers, interventions related to technology access?
but by reducing costs associated with time Relative to the other sections, technology
using computers in the college’s computer access interventions have been well studied
labs. Two separate papers reporting on the within the developing world. This research
same study also find that positive academic has, for the most part, come up with sim-
effects are significantly stronger for minority ilar results as for the developed world.
than for n onminority students (Fairlie Interventions giving computers to schools in
2012a), and that the program increased Colombia,6 One Laptop per Child efforts in
computer skills most strongly for minorities, Peru (Beuermann et al. 2015, Cristia et al.
women, lower-income, and younger students 2017), and tablets distributed to students
(Fairlie 2012b). However, a follow-up study in Kenya (Piper et al. 2016) showed no
showed no impact on earnings seven years impact on learning outcomes in the exper-
after the program was implemented (Fairlie imental studies. One study based in rural
and Bahr 2018). Peru reported positive effects on cogni-
With only two randomized evaluations tive outcomes and another study based in
(albeit reported on in nine papers) and four China (Mo et al. 2015) found that increased
disparate RDD studies, the experimental access to computers improved math scores.
literature in the developed world on its own Perhaps instructively, the intervention in
cannot speak definitively to the effective- China was the only one of the computer dis-
ness of programs seeking to improve access tribution initiatives in which computers were
to computers and the internet. However, reliably equipped with educational software
the consistency of findings of null impact that was actually used by the students, which
on learning outcomes for elementary and suggests that access to technology may be
secondary students—and especially the rel- effective when there is clear integration
atively large-scale and w ell-powered study with a broader educational program. When
of the laptop distribution program among viewed within our framework, this study sug-
sixth–tenth graders in California—indicates gests that initiatives may be effective when
that efforts to design and test technology they attempt to improve the environment
interventions may do best to focus else- and cognitive skill formation simultaneously.
where. On the other hand, the positive find- Observational and quasi-experimental
ings in the community college study by no studies in both developed and developing
means prove the potential of laptop distribu-
tion at the community college level, particu-
6 Barrera-Osorio and Linden (2009), Rodriguez et al.
larly given the evaluation’s small sample size.
(2011) find a positive impact from the same program after
But given that effects seem to have worked more time had elapsed, but the latter study is primarily
through reduction in time spent waiting nonexperimental.
countries have tended to find more pos- on earnings, the one r andomized evaluation
itive results. One recent review of both on this topic has shown promising results in
observational and experimental studies on this area, but findings from a single study at
one-to-one programs implemented between a single college may be difficult to general-
2001 and 2015 finds that an expansive range ize. More research is needed to understand
of positive impacts have been documented, the long-term impacts of increased access
including “…increased academic achieve- to technology especially given technology’s
ment in science, writing, math, and English; outsize role in tomorrow’s labor market. If
increased technology use for varied learn- we believe that increased computer profi-
ing purposes; more student-centered, indi- ciency is related to one’s ability to succeed
vidualized, and project-based instruction; in an increasingly automated workforce,
enhanced engagement and enthusiasm increased access to technology during child-
among students; and improved teacher–stu- hood may support human capital develop-
dent and home–school relationships” (Zheng ment in ways not captured by the existing
et al. 2016). However, many of the stud- research.
ies reviewed are not equipped for rigorous Second, while the few technology access
causal inference. These conclusions should programs that have been experimentally
thus be tempered in light of the experimen- evaluated at the primary and secondary
tal findings discussed in this review, which levels show few positive effects on aca-
suggests that increased access to technology demic achievement, improving access in
may create a better environment by reducing combination with other activities may yield
constraints associated with acquiring infor- better results. For instance, the survey con-
mation or completing homework, but by ducted for the Romania study discussed
itself may not necessarily facilitate more effi- above found some suggestive evidence
cient formation of cognitive or noncognitive that the negative effects of home comput-
skills. ers on grades was attenuated with certain
parental rules—approaches to regulating
3.3 Looking Forward
children’s computer use or providing more
What insights does the experimental structure and guidance for how the com-
literature bring to ongoing debates on puter should be used may be worth study-
ed-tech? First, more research is needed
ing. More research is needed to understand
on efforts to improve access to technology the interaction between increased access to
at the postsecondary level. Postsecondary technology coupled with learning-oriented
education demands a variety of more com- activities. And, although increasing access
plex tasks that, in many cases, truly neces- to computers and internet may not on
sitate the need for a computer. Although their own measurably improve academic
students enrolled in colleges are more achievement, they have been successful in
likely to already have computer access increasing the time and/or ease of use. This
(Anderson 2015, School Guides 2014, observation, in combination with the posi-
Advanced Micro Devices 2014), computer tive results found for educational software
ownership and internet access are far from discussed in the following section, suggests
universal among lower-income and other- that the most promising policy models may
wise disadvantaged students (Cohn 2014) be those that integrate hardware distribu-
and seeking access to computers at labs tion with particular learning programs. We
may waste scarce time or other resources. turn to discussing learning software pro-
Notwithstanding the lack of impact found grams in the following section.
4.
Computer-Assisted Learning improves formation of cognitive skills more
so than the next best technology—or, for
4.1 Background, Context, and Mechanisms practical purposes, business as usual. One
of Impact way CAL products may supplement the skill
formation process is by overcoming binding
Computer and learning scientists have constraints faced by teachers related to man-
been working for decades to develop soft- aging heterogeneous learning levels within a
ware to deliver educational content, and the single classroom.
popularity of these programs has exploded in Perhaps the most central mechanism
the wake of the 1990s’ ICT revolution. For through which CAL is expected to person-
the purposes of this review, we refer to ini- alize learning and overcome traditional
tiatives relating to educational software as classroom constraints is adaptivity—the
CAL programs.7 CAL programs differ from increasingly sophisticated ability of software
the technology distribution programs of the to harness emerging artificial intelligence
previous section in that they do not involve and machine-learning techniques to model
the provision of hardware for general use, the cognitive processes of students and to
but instead center on “well-defined” (Rouse offer content accordingly.
and Krueger 2004) uses of specific software An exciting body of experimental studies
packages. Moreover, they differ from the has reported large and significant effects
online courses discussed in the following from instructional models that enable stu-
section in that they are software packages dents to dedicate time working through
designed to develop particular skills, for content that matches their level of academic
example, improving math computation or preparedness (Banerjee et al. 2007, Banerjee
improving reading comprehension, rather et al. 2016). Such efforts can better allow
than platforms through which to adminis- students to master relatively basic concepts
ter courses. A multitude of companies have before moving on to more advanced con-
entered the market to meet growing demand cepts, and to practice more in areas where
from educators and policy makers for CAL, they are struggling and less in areas in which
resulting in the advent of a plethora of prod- they are already strong.
ucts being used daily by millions of students Yet when teaching a classroom with stu-
worldwide. dents of many different levels, even the most
Under what circumstances might we skilled teacher can only adapt so much—a
expect CAL to deliver significant impacts major constraint that educators have long
on student learning? In terms of our frame- attempted to overcome. When confronted
work, CAL programs generally function as with a wide range of student ability, teach-
inputs that aim to improve the formation ers often end up teaching the core curricu-
of cognitive skills. With this in mind, a CAL lum and tailoring instruction to the middle
program is successful to the extent that it of the class. By leveraging technology, CAL
programs have the potential to overcome
these long-standing instructional constraints
7 The programs discussed in this section are also fre-
and amplify cognitive skill formation by effi-
quently discussed under the rubric of “personalized learn-
ing.” While this latter term is sometimes used as a synonym ciently targeting content to every student.
for CAL, we chose to use the term CAL in this paper since Aside from directly tailoring content
definitions of personalized learning sometimes lack a tech- toward students, CAL programs can over-
nology component and because, while personalization is
often a key goal of CAL, CAL programs may vary in the come significant time constraints that teach-
extent to which they focus on it. ers face and enhance the efficiency of an
educator’s efforts to facilitate cognitive skill suggest that CAL programs of the types
formation in his or her students. For exam- evaluated in these studies show enormous
ple, CAL programs can offer students indi- promise in improving learning outcomes,
vidualized feedback and rapidly gather data particularly when it comes to mathematics.
on student performance, which would be Of the 31 studies included, 21 report statisti-
challenging for any educator to do given cally significant effects, with many precisely
time constraints. The program theories guid- estimated and substantial in magnitude. The
ing the interventions evaluated in the stud- majority of the studies finding positive effects
ies that we review typically include multiple (16 of 21) were focused on improving math
approaches to improving language, math, outcomes.8 Math content may be particu-
and other cognitive skills. While many CAL larly suited to personalized learning software
programs attempt to improve education by that matches students’ learning levels since
facilitating the increased personalization of answers to math problems are typically more
learning, these programs vary widely in how objective relative to language. Hence, soft-
they do so. They can range from light-touch ware designers can base algorithms on for-
interventions that provide practice oppor- mal rubrics that judge correct and incorrect
tunities outside of class, to more intensive answers to reliably place students at specific
interventions that provide courses with learning levels. Information on these studies
entirely new curricula, to (in a few cases) is presented in table 2.
initiatives in which schools are organized Of those evaluated, several interventions
entirely around CAL. show especially strong promise, for example,
an evaluation of a math homework program
4.2 Experimental Evidence
in Maine showed an effect size of 0.18 stan-
To what extent and under what circum- dard deviations despite involving less than
stances are these CAL programs effective? 30–40 minutes per week (Roschelle et al.

In this subsection, we review the experimen- 2016), while a more intensive software-based
tal literature on this question. We identified math curriculum intervention in Texas
31 experimental studies of CAL programs improved seventh and eighth grade math
in developed countries, all based on RCTs. scores by 0.63 and 0.56 standard devia-
While CAL can conceivably include a wide tions, respectively (Roschelle et al. 2010).
range of program types from games to Many of the CAL interventions compare
research and networking tools, the CAL favorably with interventions commonly dis-
programs that have been evaluated experi- cussed in the economics of education liter-
mentally generally fall within the broad cat- ature like reduced class sizes, longer school
egory of “intelligent tutoring systems,” that days, and intensive face-to-face tutoring. We
is, software systems that aim to help students first review findings from studies on CAL
practice particular skills (Kulik and Fletcher
2015). As a caveat, the types of CAL programs 8 Barrow, Markman, and Rouse (2009); Beal et al.
that have been experimentally evaluated (2013); Hegedus, Dalton, and Tapper (2015); Karam et
constitute only a small portion of the many al. (2017); Kelly et al. (2013); Morgan and Ritter (2002);
CAL products on the market and currently Pane et al. (2014); Ragosta, Holland, and Jamison (1982);
Ritter et al. (2007); Roschelle et al. (2010); Roschelle et al.
used by schools, and the results of these (2016); Schenke, Rutherford, and Farkas (2014); Singh et
studies cannot be generalized to other CAL al. (2011); Snipes et al. (2015); Tatar et al. (2008); Wang
products that might not function within sim- and Woodworth (2011). Pane (2014) only finds positive
impacts on math outcomes in the second year. Campuzano
ilar theories of change. Nevertheless, taken et al. (2009) did not focus exclusively on math outcomes
together, the findings from these studies and is therefore not included in this count.
TABLE 2
Computer-Assisted Learning
Unit of
Author Publication Intervention Direction of effect Effect size Sample Sample size randomization
Barrow, Markman, American I Can Learn© Positive effect on 0.17 standard 8 high schools and 1,605 students in Classroom
and Rouse (2009) Economic also known as test scores deviations increase on a 2 middle schools 142 classes
Journal: “Interactive pre-algebra and algebra in 3 large urban
Economic Computer Aided test; the strongest school districts
Policy Natural Learning” effects were for larger in the Northeast,
program for classes (especially with Midwest, and
pre-algebra and more heterogeneity South with a high
algebra in student levels) and proportion of
classes with more minority students
absences
Beal et al. (2013) Presented AnimalWatch Positive effects on 0.3 standard deviations 58 teachers’ 1,291 students Teacher
at annual web-based math student scores (approximate) classes for sixth in 58 teachers’
meeting of tutoring program grade classes
the American
Educational
Research
Association
Borman, Benson, Educational Fast ForWord Null results for 1. Second graders: Second and 415 students Student
and Overman Evaluation and computer-based second graders null in reading seventh grade
(2009) Policy Analysis language and in reading and comprehension or students in
reading training language and for language Baltimore in 8
program seventh graders in 2. Seventh graders: elementary and
language; positive 0.21 standard deviation middle schools
effects on reading increase in reading
comprehension comprehension and

for seventh null in language
graders
Cabalo, Ma, and Empirical Cognitive Tutor’s Null results Null 32 pre-algebra 32 classrooms Classroom
Jaciw (2007) Education Bridge to Algebra classes in 5
program schools in the
Maui, Hawaii,
school district
915
(Continued)
916
TABLE 2
Computer-Assisted Learning (Continued)
Unit of
Campuzano et al. Institute of 16 types of 1. Null results for 1. Null for reading 3 3 US school 9,424 students Teacher
(2009) Education software products reading 2. Lower sixth math districts, 132 in 439 teachers’
Sciences for math and 2. Negative results test scores in the schools, 428 classes in first
reading for sixth grade second year than in the teachers. It cohort, 3,280
math scores first year focused on school students in 176
3. Positive results 3. 0.15 standard districts that teachers’ classes
for Algebra I test deviation increase in had low student in second cohort
scores Algebra I student test achievement and
scores in the second large proportions
year over the first year. of students in
poverty.
Cavalluzzo et al. Institute of Kentucky Virtual Null results Null 47 Kentucky 6,908 students in School
(2012)* Education Schools hybrid schools (30 of 25 schools
Sciences program for which were in
Algebra 1 rural areas) with
grade 9 algebra
classes
Dynarksi et al. Institute of 16 types of Null results Null 3 3 US districts, 9,424 students Teacher
(2007) Education software products 132 schools, and in 439 teachers’
Sciences for math and 439 teachers classes
reading participated
in the study; it
focused on school
districts that
had low student
achievement and
large proportions
of students in
poverty
*This could also be a blended online learning and face-to-face intervention. In Kentucky Virtual Schools, instruction time is 60 percent face-to-face instruction and 40
percent is using online resources. The findings from this paper are consistent with the outcomes we observe in other blended classroom interventions.
(Continued)
TABLE 2
Unit of
Deault, Svage, and Journal of ABRACADABRA Positive and 1. Synthetic group: Grade 1 students 144 students Students
Abrami (2009) Research on web-based null effects in 0.41 standard from schools
Educational literacy program reducing negative deviations on listening in Montreal,
Effectiveness correlations comprehension, Canada, from
between attention null on vocabulary, 13 different
and learning and 0.35 standard classrooms.
outcomes deviations on reading
comprehension; null
and positive impacts
across more than a
dozen other reading
and related measures
2. Analytic group:
null on listening
comprehension,
vocabulary, and reading
comprehension; mainly
null across more than
a dozen other reading
and relating measures
Faber and Visscher Computers & Snappet digital Null impacts Null 69 Dutch primary 1,605 students in School
(2018) Education formative on spelling schools 69 schools
assessment achievement
tool focused on
spelling
Hegedus, Dalton, Educational SimCalc Positive effect on 1. 0.29 standard 7 high schools 280 students in 30 Classroom
and Tapper (2015) Technology interactive math student learning deviations in Southeast classrooms
Research & software of core algebra 2. 0.25 standard Massachusetts
Development concepts deviations in replication of varying
study achievement
levels
917
(Continued)
918
TABLE 2
Unit of
Karam et al. (2017) Educational Cognitive Tutor Positive effect 0.14 increase from Middle and high 469 teachers in 73 School
Technology Algebra I on Algebra 1 nontraditional student schools in 51 high schools and
Research & outcomes activities (use of school districts 74 middle schools
Development traditional student representing 7
activities was negatively states in the US
associated with student that varied in
outcomes on Algebra contexts.
I for middle schools in
years 1 and 2).
Kelly et al. (2013) Artificial ASSISTments Positive effects 0.56 standard Seventh grade 63 students Student
Intelligence in online math on learning for deviations students who
Education homework students and were currently
support teacher’s review of enrolled in an
homework eighth grade
math class in
a suburban
middle school in
Massachusetts
Mitchell and Fox Reading DaisyQuest and 1. Positive effects 1. 0.69 standard US kindergarten 72 students Student
(2001) Research and Daisy’s Castle on phonological deviation increase and first grade
Instruction reading game processing in phonological students
2. The t eacher- awareness from
facilitated computer-administered
computer phonological
instruction group instruction
outperformed 2. 1.03 standard

the computer- deviation increase in
administered phonological awareness
group on several from teacher-facilitated
literacy measures. computer instruction
(Continued)
TABLE 2
Unit of
Morgan and Ritter Unpublished Cognitive Tutor Positive effects in 0.29 standard Ninth graders 444 students Students
(2002) Algebra I math outcomes on deviations in 5 junior high
ETS test schools in Moore
Independent
School District,
Oklahoma
Pane et al. (2010) Journal of Cognitive Tutor 1. Negative 1. –0.19 standard 8 high schools 699 students in 19 Teacher and
Research on Geometry effects on math deviations in Baltimore teachers’ classes student
Educational outcomes 2. No 2. Null Country Public
Effectiveness effect on student School District
attitudes toward (BCPS)
mathematics and
technology.
Pane et al. (2014) Educational Cognitive Tutor 1. No effect in the 1. Null Public middle 18,700 high School
Evaluation and Algebra I first year 2. 0.20 standard and high schools school students in
Policy Analysis 2. Positive effect deviations in second across 7 US states 73 high schools,
in second year for year for high schoolers, and 51 school 6,800 middle
high schoolers but null for middle districts in urban, school students in
schools. suburban, and 74 middle schools
rural areas
Ragosta, Holland, Unpublished Cognitive Tutor Positive effects 1. Math: 0.80 standard Four elementary Unknown Students
and Jamison (1982) (report for math on student test deviations in year schools in Los
available scores in math 1; 0.91 standard Angeles
through and reading deviations in year 2;
Educational curriculum and 1.23 standard

Testing specific tests deviations in year 3
Service) 2. Reading: 0.38
standard deviations in
year 1; 0.52 standard
deviations in year 2;
and null in year 3
919
(Continued)
TABLE 2
920
Unit of
Ritter et al. (2007) Supporting Cognitive Tutor Positive effect 0.416 standard Ninth graders 343 students in 19 Class sections
Learning for math on student math deviations for first in 5 junior high class sections
Flow through grades; null on semester grades; 0.356 schools in Moore
Integrative Algebra ETS standard deviations for Independent
Technologies score final grades; null for School District,
Algebra ETS score Oklahoma
Rockoff (2015) Unpublished School of One Null results Null New York City 5,070 students in School
middle school public schools 8 schools
mathematics
program
Roschelle et al. American SimCalc Positive effects on 0.63 standard Seventh and 140 seventh grade School
(2010) Educational interactive math math outcomes deviations for seventh eighth grade classrooms in 77
Research software grade classrooms classrooms in schools, 80 eighth
Journal and 0.50 standard Texas public grade classrooms
deviations for eighth schools in 56 schools
grade classrooms
Roschelle et al. AERA Open ASSISTments Positive effects on 0.18 standard Seventh graders 2,850 students in Schools
(2016) online math math outcomes deviations in Maine 43 schools
homework
support
Rouse and Krueger Economics Fast ForWord Null results Null 4 schools in an 89 students Students
(2004) of Education computer-based urban school
Review language and district in the
reading training Northeast; around
program 40 percent
African American
and 50 percent
Hispanic
Rutherford et al. Journal of Spatial-Temporal Null results Null 52 elementary 13,803 students in School
(2014) Research on (ST) Math schools in 34 schools
Educational Southern
Effectiveness California
(Continued)
TABLE 2
Unit of
Schenke, Computers & Spatial–Temporal Positive effects 0.14 standard 50 elementary 10,860 students School
Rutherford, and Education (ST) Math on basic number deviations on the schools in from 50 schools
Farkas (2014) sense skills Number Sense I (NSI) Southern
skills test California
Singh et al. (2011) Unpublished ASSISTments Positive effects on 0.40 standard 8 classes of eighth 172 students Student
online math math outcomes deviations grade students in
homework Maine
support
Snipes et al (2015) Institution Elevate summer Positive effects on 0.70 standard Eighth grade 496 students Student
of Education math program math achievement deviations on a test of students from
Sciences and algebra algebra readiness 8 schools in
readiness, yet still 6 districts in
did not prepare California’s Silicon
students for Valley
Algebra I content
Tatar et al. (2008) Journal of SimCalc Positive effects Main reported effect 21 seventh grade 827 students in 21 Teacher
the Learning interactive math on student size (not standardized): mathematics teachers’ classes
Sciences software and teacher positive effects on teachers in Texas
mathematics student and teacher
knowledge mathematics knowledge
(exact magnitude not
reported)
Van Klaveren, Economics Adaptive Null results Null Dutch secondary 1,021 students Student
Vonk, and of Education CAL program schools

Cornelisz (2017) Review compared against
a static one across
multiple subjects
(Continued)
921
922
TABLE 2
Unit of
Wang and SRI 1. DreamBox 1. Positive effects 1. DreamBox: 0.14 Kindergarten 583 students Student
Woodworth (2011) International math program from DreamBox standard deviations through fifth
Center for 2. Reasoning on test scores on NWEA math test grade students
Education Mind math 2. Null results and 0.16 standard in 3 schools in
Policy program from Reasoning deviations on geometry an elementary
Mind on test subtest charter school
scores 2. Reasoning Mind: null network in San
Francisco
Wijekumar, Meyer, Education ITSS (Intelligent Positive effects on 0.10 standard Rural and 2,643 students Classroom
and Lei (2012) Technology Tutoring for language deviations on language suburban fourth in 60 rural and
Research and Structure and 0.49 standard grade classrooms 71 suburban
Development Strategy) program deviations on main idea classrooms
for reading and quality
language
Wijekumar et al. Journal of ITSS program Positive effects 0.20 standard Fifth-grade 2,645 students Classroom
(2014) Research on for reading and on literacy and deviations on literacy classrooms in 45 in 58 rural and
Educational language signaling tests and 0.42 standard schools within 12 70 suburban
Effectivness deviations on signaling school districts classrooms
tests in rural and
suburban settings
in Pennsylvania
rograms in math, considering models from

p constraints a teacher typically faces when
light-touch homework supplements to class providing feedback and gathering timely
curriculum changes to school-wide person- information on student homework perfor-
alized learning models. After reviewing the mance. Two small-scale proof-of-concept
experimental evidence on math software, we studies (Kelly et al. 2013, Singh et al. 2011)
then turn to the few experimental studies on found promising effects, but these studies
CAL reading programs. Finally, we consider had samples numbering only in the dozens
findings from the studies we included within of students and implementation time num-
the broader research context and highlight bering only in the days.
potentially promising directions moving for- More recently, however, a full-scale impact
ward. For our discussion, we draw our con- evaluation of an ASSISTments intervention
clusions from w ell-powered evaluations that was conducted with a sample of 2,850 s eventh
estimate the impacts of CAL programs with graders across 43 schools in Maine (Roschelle
a high degree of precision. et al. 2016). The authors found that the pro-
Beginning with light-touch approaches, gram improved math scores for treatment
ASSISTments represents an especially students by 0.18 standard deviations. This
promising example of a CAL program. impact is particularly noteworthy given that
ASSISTments is a math homework plat- treatment students used the program, on
form released by the Worcester Polytechnic average, for less than ten minutes per night,
Institute that does not require that schools three to four nights per week (Roschelle et al.
adjust their curriculum or textbooks, and 2016, p. 6). Students at or below median ben-
is available free of charge (ASSISTments efited even more from using the program,
2016). The program is designed to carry with an effect size of 0.29 standard deviations
out “formative assessments,” that is, to use (Roschelle et al. 2016, p. 8). It is worth not-
“data from students’ independent work ing that the program depends on students’
to give them helpful feedback and guid- ability to access a laptop or tablet—part
ance while enabling the teacher to use the of the reason that this evaluation was con-
data to adjust instruction to meet students’ ducted in Maine, given its policy of lending
learning needs” (Roschelle et al. 2016). As laptops to all public middle and high school
students work through individual problems, students. This further reinforces the promise
the computer informs them about whether of initiatives that aim to bolster the learning
their answer is correct and offers guidance environment through increased access to
if necessary. Students are expected to ben- technology while simultaneously supporting
efit from the customized practice, as well cognitive skill formation through educational
as from rapid feedback and data supplied software. While this hurdle may raise some
to teachers (in addition to, in some cases, external validity concerns with regard to this
supplementary professional development particular study, a variety of possibilities exist
to train the teachers on optimizing use for enabling access in other states, especially
of ASSISTments). Referring back to our given that software and licensing are free, so
model, the ASSISTments program aims to costs are otherwise low.
support cognitive skill formation by pro- Other programs move beyond homework
viding students with immediate feedback supplements and instead offer full curricula.
on homework performance, a critical step A prime example—perhaps the most prom-
in advancing one’s understanding of mate- inent of all of the CAL products discussed
rial and acquisition of content knowledge. in this review—is the set of Cognitive Tutor
Moreover, the program attempts to mitigate programs published by Carnegie Learning.
Unlike ASSISTments, the Cognitive Tutor 2010). Rather than reducing constraints on
programs generally provide curricula for teacher time, the introduction of the soft-
entire mathematics courses, including les- ware may have been an additional constraint.
son plans, textbooks, training for teachers, A 2014 experiment across eight states in
and detailed guidelines. When viewed in the 73 high schools and 74 middle schools has
context of our framework, Cognitive Tutor sought to increase the external validity of the
may support teachers in promoting cognitive Cognitive Tutor literature by seeking to rep-
skills formation by reducing time to develop licate realistic scale-up conditions in a wide
lesson plans while simultaneously delivering variety of locations (Pane et al. 2014). They
more targeted content. The company rec- found no effect the first year, but a 0.20 stan-
ommends 40 percent computer time and dard deviation impact with the second-year
60 percent class time (Cabalo, Ma, and Jaciw cohort. Interestingly, the improvement in the
2007; Pane et al. 2010). Through the “tutor,” second-year cohort was not associated with
students receive individualized instruction increased fidelity of implementation, but
in the form of challenging problems that instead with teachers reducing (although not
reflect real-world situations, enabling stu- completely eliminating) their use of the activ-
dents to move from concrete to abstract ities called for by Cognitive Tutor guidelines
thinking (Pane et al. 2014). We identified for noncomputer class time (Pane et al. 2014).
nine papers reporting on experimental stud- While many studies have posited that
ies on Cognitive Tutor programs in a variety quality of implementation is an important
of locations, including California, Hawaii, factor in determining outcomes (Pane et al.
Maryland, and Oklahoma (Cabalo, Ma, and 2010, Pane et al. 2014, Ritter et al. 2007),
Jaciw 2007; Campuzano et al. 2009; Dynarski one study examining implementation fidel-
et al. 2007; Karam et al. 2017; Morgan and ity of the Cognitive Tutor is more specific
Ritter 2002; Pane et al. 2010; Pane et al. about how quality of implementation relates
2014; Ragosta, Holland, and Jamison 1982; to time use in the classroom. Adaptive soft-
Ritter et al. 2007). Earlier papers were nar- ware may ensure that the amount of time
row in scope and had mixed results ranging students spend engaged on the platform
from negative effects of 0.19 standard devi- is time spent learning academic concepts,
ations (SDs) to positive effects of 0.23 SDs rather than playing games or partaking in
on math outcomes. One 2010 experiment other n onacademic tasks. Their findings
of Cognitive Tutor’s geometry program suggest that learner efficiency is associated
found that students who used the software with whether teachers take an active role
performed worse than students who were in turning the time students are engaged in
taught the standard geometry curriculum. activities required by the software into aca-
Qualitative analysis from the study suggests demic learning (Fancsali et al. 2016). Such
that teachers had difficulty implementing variations in implementation may explain
the program in a way that engaged students, mixed results across the Cognitive Tutor
potentially because the program represented studies, and at least warrant further review
too large of a shift from the b
usiness-as-usual for understanding the extent to which this
approach. In particular, the study noted that theory about time use describes a true
teachers struggled to effectively facilitate mechanism of impact. Moreover, if one goal
group work required by the curriculum and of CAL is to help teachers overcome con-
also often struggled to connect students’ indi- straints in the teaching process, it raises
vidualized work in the self-paced modules to important open questions for how teachers
what was being taught in class (Pane et al. should be deploying software to best do so.
Another intervention that has recently risen been, in some cases, marketed and used
to prominence is SimCalc. Although SimCalc to cope with broader reading challenges.
has not been used or tested as extensively The program works by providing students
as Cognitive Tutor programs, those stud- with individualized exercises in a g ame-like
ies that have been conducted demonstrate computerized environment, where stu-
strong potential. SimCalc aims to improve dents receive on-screen rewards for correct
the instruction of mathematics relating to answers and attentiveness to instruction.
algebra and leading to calculus (Roschelle Lower-performing students were pulled out
et al. 2010) by supporting cognitive skill for- of the classroom to devote 90–100 minutes
mation through computer-assisted modeling of time to the software five days a week for
techniques. Using methods of “representa- six to eight weeks, so the software acted as
tional infrastructure,” the program enables a substitute for other classroom curriculum.
students to control the motions of animated These studies—the only ones, to our knowl-
characters by building or editing mathemat- edge, that have evaluated Fast ForWord
ical functions. After editing the functions, within a broader education setting—found
students can press a “play” button to see mostly weak and insignificant results,
the corresponding animation (Kaput and although one study only had a sample of 89
Roschelle 2013). A study on a SimCalc inter- students. While the specific mechanisms
vention in 140 seventh grade and 80 eighth were not explicitly tested, the study did note
grade Texas classrooms turned up one of that the supervisor overseeing the computer
the largest effect sizes of any large-sample sessions needed to know how to proactively
study covered in this review, with 0.63 and identify and help students who were strug-
0.56 standard deviation improvements in gling (Rouse and Krueger 2004). Hence, it
math scores for seventh and eighth graders, seems that the software did not utilize adap-
respectively (Roschelle et al. 2010). Although tivity or the fast feedback of data features
it is unknown whether these effect sizes that other effective CAL programs retain. In
would translate to other contexts outside of other words, it may not have helped instruc-
the Texas public school system, the study did tors overcome time and resource constraints
attempt to recruit a diverse sample of teach- to the degree that effective CAL programs
ers and students, suggesting its relevance to did. While small effect sizes of the Fast
varying school populations. Any single effect ForWord program cannot be entirely ruled
size, and particularly one of this magnitude, out, experimental evidence suggests that
must be taken with caution—but an effect of further adjustments or at least more test-
even half this size would be noteworthy. ing may be needed before scale-up can be
We identified only seven studies (Borman, recommended.
Benson, and Overman 2009; Deault, Savage, In contrast, two recent w ell-powered
and Abrami 2009; Faber and Visscher 2018; studies (Wijekumar, Meyer, and Lei 2012;
Mitchell and Fox 2001; Rouse and Krueger Wijekumar et al. 2014) that evaluated a
2004; Wijekumar, Meyer, and Lei 2012; reading comprehension program called
Wijekumar et al. 2014) within the developed Intelligent Tutoring for the Structure
world exclusively examining reading and Strategy (ITSS), which teaches students a
spelling programs. Of these, two evaluated particular technique for breaking down texts,
Fast ForWord, a program initially designed show significant positive results. It differs
for students with particular learning disabil- from Fast ForWord in that it is geared toward
ities (Rouse and Krueger 2004; Borman, middle school students and aims to improve
Benson, and Overman 2009), but that has reading comprehension rather than basic
l iteracy. ITSS is a web-based intelligent tutor and other challenges could impede CAL
that utilizes a “structure strategy” to teach- implementation. Findings so far have been
ing literacy that begins a lesson by describing largely positive.
what the student is going to learn, models One recent study conducted in Delhi
the strategy, and asks the student to practice. (Muralidharan, Singh, and Ganimian 2016)
The tutor then provides feedback to the stu- finds especially large effects that seem to
dent based on his/ her answers and gives the occur through mechanisms of personaliza-
student the chance to correct the answer if tion and adaptivity akin to those described
needed. In this way the program seeks to the above. The program, called Mindspark,
mitigate the aforementioned constraints on administers its self-developed educational
teacher time by replicating how an i n-person software at study centers for a small fee.
tutor might help a student systematically The Mindspark program conducts an initial
break down a passage of text. Effect sizes on assessment to determine a student’s learning
a series of reading comprehension measures level and then continually adapts content to
ranged from 0.2 to 0.53 standard deviations. match a student’s learning level based on
Though further research is needed to better how the student progresses through earlier
tease out the underlying mechanisms and content. After a treatment period of under
ascertain external validity beyond the schools five months, the authors find an effect of
recruited for this study, the use of the “struc- 0.36 standard deviations on math scores and
ture strategy,” which simulates for students 0.22 standard deviations on Hindi language
how effective readers process text, may be scores, the two subject areas for which the
a key driver of impact, as opposed to a pro- program was tested. Although there is no
gram like Fast ForWord, which works more treatment arm that offers the same content
through reward and practice. Moreover, without the adaptivity component, they pres-
ITSS offered students immediate feedback ent strong suggestive evidence that adaptiv-
on answers and the opportunity to correct ity played a key role in accounting for the
answers if needed, which again suggests that impact. There is an expansive range of levels
personalization may be an important under- between students within each grade, and the
lying mechanism to the successful formation Mindspark program records that report the
of cognitive skills. questions generated show that they matched
CAL is becoming increasingly popular this wide range. Given that no teacher could
within the developing world as well, and an possibly have covered such a huge spread of
experimental literature on these interven- levels, the authors argue that the adaptation
tions is growing rapidly in China (Bai et al. element of the program must have played a
2016; Lai et al. 2013; Lai et al. 2015; Lai et central role in enabling its positive impact
al. 2016; Mo, Zhang, Luo, et al. 2014; Mo, and could therefore be an integral part of a
Zhang, Wang, et al. 2014; Mo et al. 2015) and solution to the unevenness of levels that chal-
India (Banerjee et al. 2007; He, Linden, and lenge many schools in India and elsewhere.
MacLeod 2007; Linden 2008; Muralidharan, From an implementation standpoint, CAL
Singh, and Ganimian 2016; Naik et al. 2016). may be useful in solving gaps in staffing or
On one hand, CAL programs may prove to infrastructure problems, which can be partic-
be more effective in developing countries ularly useful in developing country contexts
given the often tight capacity constraints where there is low supply of qualified teach-
faced, such as low supply of qualified teachers, or teachers are less likely to be computer
ers or large pupil-to-teacher ratios. On literate. However, it could also be useful in
the other hand, infrastructure limitations US contexts where the binding constraints
to learning are similar, for e xample in rural are adaptivity of instruction, rapid feedback
areas to supplement low quality or low sup- to teachers, or more engaging delivery of
ply of teachers. content—that most contribute to the success
Taken together, CAL may be able to sig- of effective programs.
nificantly improve learning outcomes, with Another crucial question is to test the
the evidence particularly strong for math. extent to which learning from CAL lasts in
By using technology to provide students the longer term. To what extent do effects
with rapid feedback and teachers with data compound or diminish in subsequent years?
on student performance, supplementary Another important task will be to further
programs can overcome long-standing class- explore whether and when CAL can work
room constraints and yield significant bene- effectively for subjects other than math. Do
fits to cognitive skill formation (Roschelle et the cognitive processes that underpin math-
al. 2016). Promising studies of more inten- ematical reasoning inherently lend them-
sive interventions also suggest that technol- selves better to software algorithms? More
ogy has the potential to replicate aspects of broadly, which areas of education could CAL
in-person tutoring for both math and read- add most value to, and which pedagogical
ing (Roschelle et al. 2010, Wijekumar et al. strategies are best enhanced by CAL? And
2014). when are light- versus heavy-touch interven-
Nevertheless, there are some caveats to tions most appropriate and cost effective?
note when drawing inferences from the An important crosscurrent that undercuts
results of these studies. First, it may very many of these other concerns is the issue
well be that the promising CAL interven- of implementation. One way to gain greater
tions are the ones that are more likely rig- leverage on this issue could be to test a partic-
orously evaluated. Hence, it is important to ular CAL program in a particular population
consider the potential mechanisms of impact while varying elements of the implementa-
for each intervention and not overgener- tion plan. Finally, we still know little about
alize to infer that the results from these 31 how CAL programs interact with teachers’
studies imply that all CAL interventions are efforts. Unpacking interconnections could
promising. Rather, we should consider which highlight opportunities for how CAL can
specific features of CAL interventions have best complement and increase the efficiency
been shown to be efficient in improving of teacher’s efforts. As technology continues
returns to investment in cognitive skill for- to evolve, testing new emerging CAL models
mation, such as personalized learning, adap- will be vital.
tivity, and rapid feedback. Assuming teacher
quality is a critical input into the education
5. Online Courses
production function, identifying CAL pro-
grams that successfully enhance teachers’ Since their emergence during the 1990s,
ability to deliver instruction at scale could online courses have come to constitute a siz-
yield profound benefits. able presence within the education field. By
2013, over a third of US college students had
4.3 Looking Forward
taken an online course at some point during
Numerous important questions remain, their college career (Bettinger et al. 2017, cit-
however, for future researchers to answer if ing Allen and Seaman 2013), and more than
CAL’s potential is to be efficiently leveraged. 11 percent were enrolled in entirely online
One vital area is to more explicitly test for programs (Deming et al. 2016). The rise of
the underlying mechanisms—whether they online learning bears heavily on policy issues
relating to educational equity, since two key as a discussion forum” (Hodges, Lowenthal,
justifications for the proliferation of online and Grant 2016). Between 2012 and 2015,
education have been its promise of improv- MOOCs saw enrollment rates exceeding 25
ing access and reducing marginal costs associ- million (Kizilcec et al. 2017). While conven-
ated with teaching more students. Moreover, tional online courses and MOOCs developed
at least at the p
ostsecondary level, students to serve largely separate purposes, the lines
in online programs tend to have faced sig- between them are becoming blurred. For
nificant disadvantages. For instance, data instance, MOOC companies have increas-
from the National Postsecondary Student ingly offered certification programs for a fee,
Aid Study’s 2010/2011 representative sur- such as MicroMasters programs,10 and MIT
vey indicate that “online students are older, has launched a MOOC-based program that
have lower levels of parental education, are will lead to a traditional master’s degree.11
more likely to be single parents themselves, In this section, we first discuss the back-
and are more likely to be working full-time ground, context, potential mechanisms of
while enrolled in school than are other col- impact, and the experimental evidence on
lege students” (Deming et al. 2016). So how conventional online courses, and then turn
does online education perform in terms of to a discussion of the same with regard to
access, learning, and other important out- studies on MOOCs.
comes? How does online education impact
5.1 Conventional Online Courses
the learning environment and the formation
of cognitive skills?
5.1.1 Background, Context, and
Online courses have, over the past several
Mechanisms of Impact
years, coalesced into two broad categories.
First, what we refer to as conventional online Online courses build on a tradition of
courses represent an online extension of correspondence courses that has existed for
the “distance learning” or “correspondence over a century within the higher education
course” format, an approach that has a long field (Means et al. 2009). As early as the
history in higher education (Means et al. late 1800s, institutions like the University
2009). These courses are typically offered as of Chicago and the University of Wisconsin
part of a degree program that consists entirely were teaching faraway students via the postal
of online courses, or that includes online, service (Deming, Goldin, and Katz 2012).
face-to-face, or blended9 courses. Second
Educators and entrepreneurs brought online
are massive open online courses (MOOCs). college courses and degree programs to mar-
Unlike conventional online courses, MOOCs ket beginning in the 1990s, but proliferation
are typically not part of official degree pro- expanded rapidly after a 2006 decision to
grams and are open to anyone to enroll via end a regulation that had limited federal aid
the internet. They broadly consist of “struc- money for institutions conducting more than
tured and sequenced teacher-led activities half of their coursework via correspondence
(e.g., videos, readings, p roblem sets) cou- (Deming et al. 2016). Some institutions offer
pled with online assessments and usually both online and face-to-face instruction,
some venue for student interactions such
10 MicroMasters, https://www.edx.org/micromasters.
9 The term blended takes on different meanings in dif- 11 MIT announces MITx MicroMasters program in
ferent contexts within the ed-tech literature—in this case, development economics, with a path to full master’s
we use the term to refer to a single course that has both degree, http://news.mit.edu/2016/mitx-micromasters-
online and f ace-to-face components. program-development-economics-masters-degree-1205.
while others offer online courses exclusively. because of geographic location, work or
While a growing mass of selective universi- family obligations during class hours, or
ties offers online programs, large, for-profit disabilities (Goodman, Melkers, and Pallais
colleges (Deming, Goldin, and Katz 2012) 2019; Means et al. 2009; Poirier and Feldman
like the University of Phoenix (McKenzie 2004). And online courses may allow stu-
2018) continue to control a large share of dents more flexibility in accessing course
the market, although nonprofit online uni- materials at the most convenient times, and
versities have experienced a recent surge in in spending more time on content that they
enrollment. are struggling with and less on content that
How, then, might conventional online they have mastered (Figlio, Rush, and Yin
courses add value to education? Thus far, 2013). This potential for online courses to
experimental research on conventional increase access and lower cost may, in some
online courses has compared online against cases, constitute outcomes in their own right,
face-to-face courses to judge the extent to and in some cases these may be necessary
which the former improves access and can conditions for improving academic perfor-
act as a viable substitute for face-to-face mance and the formation of cognitive skills.
education. In the framework presented in Educators and researchers have also
this review, their aim is to identify whether pointed out potential drawbacks of online
online learning cultivates an improved learn- courses. The flip side of online courses’ flex-
ing environment—by expanding access to ibility is that students who do better with
education—and improves test scores or externally induced structure may be more
academic performance—signals of improve- likely to face time management issues than
ment in the formation of cognitive skills—at they would for a face-to-face class, and may
a given level of cost per student. thus fall behind (Donovan, Figlio, and Rush
However, online courses may work 2006). It is also possible that too large a
through different channels to achieve shift toward online courses could take away
impacts on learning. For example, a key jus- opportunities for networking and interaction
tification for online courses is that, in many that arise more naturally in f ace-to-face envi-
contexts, they may be much less expensive ronments (Sleeter 2014). More generally,
to implement than face-to-face courses.12 some educators and researchers believe that
“If Internet-based classes are at least rea- a valuable element of the teaching process
sonable substitutes for live lecture classes, is lost when the face-to-face dimension is
then the use of I nternet-based classes could reduced or eliminated (Sleeter 2014).
be a cost-effective method of combating
5.1.2 Experimental Evidence
increased fiscal constraints” (Figlio, Rush,
and Yin 2013; see also Cowen and Tabarrok We identified 17 experimental studies
2014, Means et al. 2009). Second, online examining the effects of conventional online
courses can expand access by allowing peo- courses. Of these, 15 RCTs13 compared
ple to take courses or entire degree pro-
grams that would not otherwise be possible
13 Alpert, Couch, and Harmon (2016); Bowen et al.
or worthwhile for them to take, for instance
(2014); Esperanza, Fabian, and Toto (2016); Figlio, Rush,
and Yin (2013); Foldnes (2016); Harringon et al. (2015);
Heppen et al. (2011); Heppen et al. (2012); Jackson and
12 There are at times high fixed costs associated with the Makarin (2018); Joyce et al. (2015); Keefe (2003); Poirier
development of some online courses. Although there is no and Feldman (2004); Wozny, Balser, and Ives (2018);
assumption that online education is always cheaper this is, Zhang (2005); Zhang et al. (2006). Another experiment,
on average, a general trend. reported by Snipes et al. (2015) and included in table 2
online and face-to-face delivery (or various to blended could decrease costs without neg-
gradations in between) of particular courses. atively affecting quality.
Four of the 15 RCTs (Esperanza, Fabian, The first full-scale field experiment to
and Toto 2016; Foldnes 2016; Harrington et compare face-to-face with online courses
al. 2015; Wozny, Balser, and Ives 2018) exam- took place in an introductory economics
ine the impact of flipped classroom models, course in 2013 at a major research univer-
a type of blended learning that reverses the sity with a sample of over 300 students.14
traditional learning environment by deliv- The course was identical for all students,
ering instructional content, which is tra- but some students were provided access to
ditionally taught inside the classroom, at online video lectures, while others attended
home via the internet (Tucker 2012). One these lectures in person. The study finds
RDD (Goodman, Melkers, and Pallais 2019) that students in the in-person group show
tested the extent to which offering an online higher outcomes, and that the differences
degree option increased enrollment, and one are relatively small but significant—around
audit RCT tested whether employers distin- 3 percentage points on the midterm and
guished between online and face-to-face about 2.5 percentage points on the final.
degree when selecting resumes to follow up In actual university settings however, the
on (Deming et al. 2016). choice will not necessarily be between
To what extent does the evidence sug- courses that are entirely face-to-face or
gest that online classes can match or exceed entirely online. Instead, the two are often
learning outcomes from face-to-face classes? mixed into blended courses. Two subse-
While a great deal more exploration and quent experiments studied blending learn-
replication would be needed to draw robust ing environments of this sort. One compared
conclusions, the studies reviewed here are outcomes for a statistics course in which
consistent with the hypothesis that, without one group received three hours per week of
some degree of face-to-face teaching, learn- face-to-face instruction time, while another
ing outcomes may suffer, leading to (albeit group received only one hour of instruction
small) sacrifices in test scores for fully online time but additional internet-based exercises.
courses relative to face-to-face courses. In The second experiment tested the effects of
contrast, blended learning environments— reducing face-to-face time in an economics
meaning, in this case, courses that have both course where all students also had access to
a face-to-face component and an online online resources. Neither experiment found
component—have not been found to sig- significantly better outcomes to be associated
nificantly underperform purely face-to-face
courses in studies meeting our method-
ological criteria. So, while the evidence at 14 Keefe (2003) conducted a related study in an under-
this point would not back substantial shifts graduate business course and comes up with results that are
toward fully online courses, it does indicate in the same direction as Figlio, Rush, and Yin (2013), but
this study had a sample of only 35 students (with students
that switching courses from fully in-person in face-to-face classes performing better). Another study
conducted in a university psychology class with a sample
of only 23 students found opposite results, with students in
(under the Computer Assisted Learning section), experi- the online version performing marginally better than those
mentally evaluates a middle school summer math program in a face-to-face group. Zhang (2005) and Zhang et al.
that includes an hour daily use of Khan Academy, but since (2006) run experiments on 155 and 138 undergraduates
the study compares the program as a complete package respectively and find that interactive online modules out-
against a control group that does not attend any program, perform noninteractive online modules and face-to-face
the study cannot identify independent effects of the online sessions, but the context is single-session lab experiments
component. rather than a field experiment with actual classes.
with more in-person class time in a blended thus were better able to accommodate and
learning context. engage the students (Heppen et al. 2012).
Finally, the most comprehensive study A complementary study conducted in
in this strand of the literature—the only schools in Maine and Vermont evaluated the
one to test fully online, blended, and fully effect of offering online Algebra I content
face-to-face courses within the same exper- to grade 8 students who were ready to take
iment—found results consistent with each algebra, but who attended schools that did
of the above (Alpert, Couch, and Harmon not typically offer a full-section of Algebra I
2016). Here, the authors test the impact in grade 8 (Heppen et al. 2011). In the
in an economics course of two treatment study, treatment schools offered a full online
arms—one purely online and one blended— Algebra I course to eligible students, while
along with a fully face-to-face c ontrol group control schools offered the regular math
in a single experimental context. This study curriculum, which included some Algebra I,
finds that students in the purely online ver- but not the full course. The study found
sion of the course do not perform as well as that taking the online algebra course signifi-
those in the purely face-to-face group, while cantly improved students’ algebra achieve-
outcomes for the blended treatment group ment at the end of grade 8 and increased
are not statistically different from the control the likelihood of participating in advanced
(Alpert, Couch, and Harmon 2016). course-taking sequences in high school.

The majority of research on online courses Here, the mechanism is similar to other
has been conducted in p ostsecondary set- studies of online courses in which the online
tings, but educators have increasingly component facilitates access to content stu-
attempted to leverage online learning in dents would otherwise not be able to access,
middle and high school environments as which in turn potentially improves the for-
well. We identified two such experimental mation of cognitive skills. Nevertheless,
studies, both of which tested the effect of while this study yielded positive results, the
offering an online version of algebra con- majority of the schools that agreed to partic-
tent. One study tested the effectiveness of ipate in the study were small rural schools.
online summer credit recovery courses rela- Hence, the generalizability of the findings to
tive to face-to-face courses for students who schools outside this sample is unknown and
had failed freshman algebra (Heppen et al. further research to understand longer-term
2012). This study was conducted in 15 high effects of similar online course offerings is
schools in the Chicago Public Schools sys- needed.
tem with the lowest rates of students passing While these studies tell us something
freshman algebra, with a sample of nearly about how online learning may increase
1,400 students across two cohorts. The hope access to discrete curricula or content, the
was that the online course would provide “a next logical question is: to what extent can
more individualized, interactive experience” online courses increase access to further
prompting students to “be more engaged education for those who face barriers to
and more likely to persist in the course.” pursuing a face-to-face degree at all? One
However, students in the face-to-face course of the main justifications for the potential
outperformed those in the online course. usefulness of online courses is that they can
Suggestive evidence from the study indicates improve access to degree programs for pop-
that one significant reason was that teachers ulations that otherwise might have trouble
in the face-to-face course were better able accessing them. We identified only a single
to flexibly incorporate a range of topics, and study fitting our criteria that addressed this
question. Specifically, the researchers relied there are some cautions for moving toward a
on an RDD design to reveal that p rospective purely online curriculum given the evidence
students applying to Georgia Institute of of null or negative effects on test scores.
Technology’s online master’s program in com- While the specific mechanisms are unclear,
puter science who were just above an admis- it may be that less structure for time man-
sions cutoff (which was not known to the agement or less opportunity to learn from
applicants) for the online version of the pro- peers or instructors in a purely online setting
gram were 20 percentage points more likely is detrimental to learning. Nevertheless, we
to eventually enroll in the online program do see that online instruction can, in some
than those just below the cutoff (Goodman, cases, improve the learning environment
Melkers, and Pallais 2019). Given that the by increasing access when it is otherwise
estimates also showed a 20 percentage point not possible, which can in turn facilitate the
difference in the probability of admission formation of cognitive skills. However, the
to the online program between those just evidence remains thin and mixed, suggest-
above and below the c utoff, the results indi- ing that further research is still needed to
cate that almost all of the additional admits disentangle the potential mechanisms. For
actually attended the college. The strongest example, are the null or negative effects of
effects were observed among mid-career pure online learning due to a loss of per-
prospective students, who otherwise may sonal interaction with teachers or peers, less
have chosen not to complete a degree at all structured processes to help students with
had the online program not been offered to time management, an inability for instruc-
them (Goodman, Melkers, and Pallais 2019). tors to adjust lessons to the students’ par-
Another recent experiment, however, finds ticular needs, or some combination of the
that “a business bachelor’s degree from a above? If this is the case, online courses may
for-profit online institution is 22 percent less have difficulty supporting the formation of
likely to receive a callback than one from a certain noncognitive skills, such as persever-
non-selective public institution” (Deming
ance, for which in-person instruction may
et al. 2016). But the design does not allow be better designed. Additionally, the nature
for disentangling the effect of the education of study recruitment and participation, and
medium (online versus face-to-face) from the fact that most studies have been con-
the institution’s for-profit/not-for-profit sta- ducted with introductory economics, sta-
tus. And even if employers do place a penalty tistics, or math courses may pose concerns
on online degrees, this may change in the with external validity. It may be that online
coming years given the ongoing expansion of learning may be relatively more or less ben-
the online education sector. eficial to these particular academic subjects,
or the study populations involved. Further
5.1.3 Looking Forward
studies to test online learning in other sub-
Overall, the experimental research on con- jects with other populations and specifically
ventional online courses has explored their designed to disentangle these mechanisms
potential to reduce costs, strengthen the are needed to fully understand the poten-
learning environment, and facilitate the for- tial of conventional online learning courses,
mation of cognitive skills through increasing and to better understand potential design
access to discrete curricula or entire degree solutions.
programs. The evidence so far indicates a With these lessons from conventional
potential for blended learning to reduce online learning studies in mind, we now turn
costs without lowering quality; however, to a discussion of the experimental evidence
TABLE 3A
Traditional Online Courses
Unit of
Alpert, Couch, American Face-to-face 1. Negative effect Main reported College students 519 students Student
and Harmon Economic versus blended on learning for effect sizes (not of a principles of
(2016) Review versus purely purely online versus standardized): microeconomics
online course face-to-face 1. 4.9 points lower in course taught at a large
content 2. Null effect on purely online course public university in the
learning for blended relative to face-to-face Northeast.
versus face-to-face course
2. Null
Bowen et al. Journal of Policy Blended Null effect on Null 6 public university 605 students Student
(2014) Analysis and instruction versus learning for blended campuses in the
Management face-to-face only versus face-to-face United States
Deming et al. American Audit of fictitious Negative effect on Main reported Employers posting job 10,492 Resume
(2016) Economic resumes varied by callback rates for effect sizes (not vacancies in business resumes
Review for-profit versus business bachelor’s standardized): and health identified
public, online degrees from a 22 percent decrease by a nationally
versus brick-and- for-profit online in likelihood of recognized online
mortar, and more institution. callback relative to a job search website in
selective versus nonselective public Chicago, Los Angeles,
nonselective institution. Miami, New York City,
postsecondary and San Francisco
institutions,
based on degrees
and programs
in business and
health
(Continued)
933
934
TABLE 3A
Traditional Online Courses (Continued)
Unit of
Esperanza, Adaptive and Flipped classroom Positive effect on 1. 0.31 standard High school algebra 91 students Student
Fabian, and Adaptable model enjoyment in math; deviation increase in students
Toto (2016) Learning positive effect reported enjoyment
on math scores in math
(under analysis 2. 0.59 standard
of covariance deviations (p = .09)
(ANCOVA) increase on math
specification) scores; significant
under ANCOVA
specification
Figlio, Rush, Journal of Labor Online lectures Negative effect on Main reported University students 327 students Student
and Yin (2013) Economics learning outcomes effect sizes (not enrolled in a
standardized): 2.508 large principles of
(out of 100) point microeconomics course
decrease relative to
live courses.
Foldnes (2016) Active Learning Flipped classroom Positive effect on Main reported First-year 235 students Student
in Higher model math scores effect sizes (not undergraduate
Education standardized): business students
12 percentage
point increase in
performance relative
to the lecture group
(Continued)
TABLE 3A
Unit of
Goodman, Journal of Labor Online Master Positive effect on Main reported Online and in-person 17,632 Not randomized—
Melkers, and Economics of Science in overall enrollment effect sizes (not applicant pools for students regression
Pallais (2019) Computer in formal education standardized): Georgia Institute discontinuity
Science 20 percentage point of Technology’s design
increase in overall online and in-person
enrollment in format computer science
education; at least master’s program
7 percent increase in
annual production of
American computer
science master’s
degrees
Harrington Nursing Flipped classroom Null effect on Null Nursing students 82 students Student
et al. (2015) Education model grades and exam
Perspectives scores.
Heppen et al. Institute of Online Algebra Positive effects 1. 0.40 standard Schools in Maine and 445 School
(2011) Education I course (for on eighth deviation increase in Vermont that serve students in
Sciences students who grade Algebra eighth grade algebra students in grade 8 and 68 schools
would otherwise I achievement achievement did not offer a stand-
have no access to and likelihood of 2. 1.96 times as likely alone algebra class
algebra) participation in to do advanced math
advanced math course sequence in
course sequence in high school
high school
Heppen et al. Society for Online algebra 1. Negative effects 1. 0.18 standard 2 cohorts of students at 592 students Student
(2012) Research on courses for credit on grades for online deviation decrease Chicago Public Schools in first
Educational recovery students versus in end-of-course who failed Algebra cohort, 792
Effectiveness students in the face- assessment score I in ninth grade and students
(SREE) to-face course 2. 7 percentage point enrolled in summer in second
Conference 2. Negative effects decrease in credit recovery program. cohort
Paper on credit recovery recovery
935
(Continued)
TABLE 3A
936
Unit of
Jackson and American Teacher access Positive effect on 1. 0.06 standard All middle school 27,613 Classroom
Makarin Economic to online math achievement deviations from access teachers in 3 school students
(2018) Journal: off-the-shelf alone districts in Virginia in 363
Economic Policy quality lessons 2. 0.09 standard classrooms
and support to deviations from access
promote their use and support
Joyce et al. Economics 1 class/week Positive effect on 0.21 standard Students at 725 students Student
(2015) of Education (blended) versus midterm score for deviation increase on Baruch College in
Review 2 classes/week students in face-to- midterm scores; null microeconomics course
(face-to-face) face format; on the final
null effect on final
score.
Keefe (2003) Educause 2 studies: 1. Negative effect Main reported 6 sections of students 6 sections of Student
Quarterly 1. Lecture and on course and effect sizes (not in an organizational 118 students
interaction online professor ratings standardized): behavior course in
versus traditional versus students 1. 0.7 decrease on Indiana University
face-to-face taking the course instructor satisfaction Southeast
2. Interaction face-to-face; and 0.6 decrease
versus regular 2. Negative effect (on scale out of 6.0)
lecture experience on exam scores for relative to face-to-face
student taking the course
online course 2. 7.6 percent
decrease in exam
scores relative to the
face-to-face course
Poirier and Teaching of Traditional face- Positive effect on 1. 0.57 standard Students from a large 23 students Student
Feldman Psychology to-face versus grades for students deviation increase in state university who
(2004) online course in the online course. overall course exam indicate that either
2. Null on paper a face-to-face or an
assignments compared online course was
to students in the acceptable
traditional course
(Continued)
TABLE 3A
Unit of
Wozny, Balser, Journal of Flipped classroom Positive effect 0.16 standard Introductory 137 students Student
and Ives Economic model on high-stakes deviation increase undergraduate
(2018) Education assessments relative to a lecture econometrics course at
class the United States Air
Force Academy
Zhang (2005) American The interactive Positive effect on 1.18 standard Undergraduate 155 students Student
Journal of e-classroom performance and deviations from fully students from a large
Distance component of satisfaction for interactive LBA public university in the
Education the Learning By students in the system; 0.45 standard United States
Asking (LBA) fully interactive deviations from less
system versus multimedia interactive LBA
traditional face-to- based e-learning system.
face classrooms environment.
Zhang et al. Information & Non-interactive Null effects for Positive impacts (see Undergraduate 138 students Student
(2006) Management video learning students who used entry above); students from a large
environments the e-learning null for e-learning university in Southwest
environment environment United States
that provided that provided
noninteractive noninteractive video
video.
937
on MOOCs before discussing future direc- than reduce disparities in educational out-
tions for both types of online learning. comes related to socioeconomic status”
(Hansen and Reich 2015). But their over-
5.2. Massive Open Online Courses
all impact on learning outcomes is difficult
(MOOCs)
to evaluate. While researchers are inter-
ested in the effects of MOOCs on educa-
5.2.1 Background, Context and Mechanisms
tion, it is less clear what to compare them
of Impact
to since they generally do not substitute for
The term MOOC was first used in 2008 face-to-face courses that students would

by media theorists George Siemens and otherwise take. People may take MOOCs
Stephen Downes for a course they taught for a wide variety of reasons, from practic-
at the University of Manitoba entitled ing skills for school or work to fun and per-
“Connectivism and Connected Knowledge,” sonal interest. Because MOOCs, broadly
with 25 students participating in face-to-face speaking, lack a clear counterfactual in that
sessions at the university and content broad- there is no single function they seek to ful-
casted to 2,300 additional students via fill or institution they attempt to substitute
the internet (Greene et al. 2015, see also for, no clear experimental evidence has yet
Cormier and Siemens 2010). In the subse- emerged on their overall impact, although
quent decade, MOOCs have proliferated this is likely to change over the next sev-
rapidly, with thousands of courses offered eral years given the outpouring of interest.
and hundreds of thousands of students Nonetheless, MOOCs are being given to
enrolled worldwide (Greene et al. 2015). millions of students each year (Shah 2018),
Like online courses, educators and educa- and researchers have begun to delve exper-
tion policy makers saw in MOOCs the poten- imentally into questions of how MOOC
tial to decrease costs and increase access usage can be improved for interested stu-
(Greene et al. 2015). Because MOOCs are dents. In fact, MOOCs lend themselves
generally “open,” they have the potential to well to low-cost RCTs, among other types
reach exponentially more students in a much of data generation and analysis (Lamb et al.
more diverse range of contexts than can con- 2015).
ventional online courses granted for credit. Hence, in the framework presented in this
However, because MOOCs usually do not study, the potential paths toward increased
build toward a degree and may or may not cognitive skills tested in the experimental
be valued on the labor market, it is less clear research on MOOCs are slightly different
what, if any benefits, MOOCs may bring from those expected in conventional online
beyond the value of the educational content courses. While, as for online courses, the
they impart. goal of MOOCs is ultimately to improve
What has been the effect of MOOC pro- academic and other cognitive skills, exper-
liferation? Observational research has found iments associated with MOOCs have
that expectations that MOOCs will “democ- focused on evaluating strategies to build or
ratize education” have been overblown and overcome gaps in noncognitive skills that
that, although MOOCs have offered the prevent students from fully engaging with
opportunity for many disadvantaged individ- MOOCs.15 In particular, MOOCs face very
uals to access h igh-quality educational con-
tent, enrollment and success rates are highly 15 We forego an in-depth discussion of ed-tech and
skewed toward advantaged populations. oncognitive skills in this section since it constitutes a cen-
n
Thus, MOOCs may even “exacerbate rather tral element of the next section.
low completion rates: “few users actually s tudies found positive effects from at least
complete the class” (Banerjee and Duflo one treatment arm.16 Through what chan-
2014). These low rates in themselves do not nels did these effects occur? One approach
necessarily signal a problem: many students adopted from the behavioral economics
enroll with no intention of completing the literature has been the model of “social
course, and students may generally be get- comparison” interventions—programs that
ting what they want from the MOOCs even inform students of their performance rel-
if they are only accessing bits and pieces. ative to other students. The behavioral
But low rates may, at least in part, reflect economics literature suggests that social
missed learning opportunities that could be comparisons may drive individuals to try
avoided with modifications to the MOOC harder to excel. Two recent RCTs (Davis et
platform (Banerjee and Duflo 2014). Thus, al. 2017, Martinez 2014a) found that social
experimental research on MOOCs up to comparison interventions can improve
this point has focused primarily on whether MOOC performance and completion,
and how a range of behavioral interventions although one of these (Martinez 2014a)
can improve MOOC completion rates and found significant effects only when framed
extend coverage to disadvantaged groups, for “negatively” (i.e., when target students
example, by increasing interest, effort, and were informed of how many students had
persistence. outperformed them rather than how many
Interventions aiming to improve stu- students they had outperformed).
dent MOOC effort have generally fol- Even if fully motivated to succeed in
lowed the approaches of the behavioral and a course, MOOC students may struggle
psychosocial interventions that will be dis- with time management issues and, in par-
cussed in greater detail in the following sec- ticular, the temptation to procrastinate.
tion. These interventions use strategies such Procrastination may be a particularly acute
as social comparisons, nudges to overcome temptation for MOOC students since
procrastination or to increase participation they are not being directly observed by an
in the online platform, or writing exercises instructor. One study that attempted to
to reduce social identity threat. In con- address problems of procrastination found
trast to evaluations of conventional online that a commitment device that encour-
courses, which primarily test the extent to aged students to commit to limitations on
which online courses improve cognitive skill time spent on distracting internet sites
formation, the experimental evidence on increased the likelihood of completion
MOOCs primarily tests interventions that by 40 percent and grades by 0.29 stan-
aim to change behavioral outcomes through dard deviations, while treatment arms that
psychological nudges. In this sense, these reminded students how much time they
nudges are successful to the extent that were spending on these websites or blocked
they can help overcome a constraint to stu- them while on the course page showed
dent engagement and persistence with the no significant effect for both completion
MOOC platforms which, in turn, is expected and academic performance (Patterson
to lead to a low-cost boost in cognitive or aca-
demic skills.
16 Banerjee and Duflo (2014), Davis et al. (2017),
5.2.2 Experimental Evidence Kizilcec et al. (2014), Kizilcec et al. (2017), Lamb et al.
(2015), Martinez (2014a, b), Patterson (2018), Yeomans
We identified 11 studies evaluating and Reich (2017). Banerjee and Duflo (2014) and Kizilcec
MOOC interventions. Seven of the 11 et al. (2014) do not find positive impacts.
2018). Relatedly, sending MOOC students 5.2.3 Looking Forward

a “planning prompt” improved course
completion by 29 percent (Yeomans and While these studies show some hope for
Reich 2017). behavioral nudges that encourage partic-
Many educators firmly believe that dis- ipation with online platforms and course
cussion and interaction is a central compo- completion of MOOCs, which may in turn
nent of education. But because MOOCs help MOOCs better accomplish their goal
have thousands of students who generally of democratizing education, it should be
access content at different times, regular noted that the literature is still nascent and
discussions of the types that occur in class- there is a lot we still do not know. The online
room are rarely feasible. MOOC designers learning field is changing quickly, and new
have attempted to at least partially address models that do not easily fit into the cate-
this problem by building discussion forums gories discussed here are springing up. For
into MOOCs, but participation is often rel- one, websites that offer more independent
atively low. Two experimental studies have stand-alone modules—which allow for easier
evaluated efforts to increase participation in picking and choosing of content and use in
discussion forums. One study found insig- supplementing other classes—are becoming
nificant or negative impacts from an email increasingly important. The iconic website
prompt (depending on the content of the in this category is Khan Academy, which is
email) (Kizilcec et al. 2014), while another currently undergoing several experimen-
found positive impacts on forum partici- tal evaluations, with studies either not yet
pation from asking participants to fill out a completed or results not yet publicly avail-
self-evaluation about forum participation
able. Also popular in this space has been
(Lamb et al. 2015). BrainPOP, which provides instructors with
Another friction preventing efficient and an expansive library of educational videos
equitable use of MOOCs may be “social intended to be fun and engaging.
identity threat,” the tendency of individ- Another new development has been the
uals—typically from marginalized social rise of quasi-formal certification schemes,
backgrounds—to “suffer from the cognitive like Nanodegrees and MicroMasters, as
burden of wrestling with feeling unwelcome alluded to above. These are certifications
while trying to learn and, therefore, under- granted for completing sets of courses that
perform” (Kizilcec et al. 2017). Social iden- are not formal degrees in the sense of college
tity threat has been shown to impair learning degrees, but that programs’ designers hope
in a variety of ways. One recent set of RCTs will increase their legitimacy and acceptance
evaluations tested the effects of writing exer- as real skill creators. Whether or not these
cises aimed at reducing social identity threat quasi-formal certifications will be accepted
and found them to be effective in increasing as useful by employers and will come to take
persistence and completion among MOOC on some kind of labor market premium may
students from developing countries (Kizilcec become clear over the next few years. If
et al. 2017). While this study focused on employers had better ways of assessing skills
closing the gap between students from during the hiring process, these programs
developed and developing countries, related could significantly expand education options.
interventions could also plausibly reduce With regard to MOOCs, an important task
social identity threat-driven gaps between for the research agenda will be to better
advantaged and marginalized populations articulate how exactly we judge success of
within the developed world. MOOCs. For example, is access and course
TABLE 3B
Massive Open Online Courses (MOOCs)
Unit of
Banerjee and Duflo American “Deadline effect” Students who Main reported Students 42,314 students Not
(2014) Economic Review in the 14.73x: enrolled 1 day late effect sizes (not registering randomized—
Challenges of were less likely to standardized): within 15 days regression
Global Poverty get a certificate students who of deadline discontinuity
MOOC—are and had lower enrolled 1 day late for 14.73x: design
students who grades. were less likely to Challenges of
register late get a certificate Global Poverty
less likely to do (a reduction of MOOC
well or receive a 16.6 percentage
certificate in the points), and their
course? grades were 10.7
percentage points
lower.
Banerjee and Duflo Unpublished 1. Option to Null effects Null Online course 19,694 students Student
(2016) commit to on grades or participants
structured study engagement with
time courses
2. Self-efficacy
messages
3. Tutoring
services in groups
of 20
Davis et al. (2017) Proceedings A personalized Positive effects on Main reported Learners across 4 33,726 students Students
of the Seventh feedback course completion effect sizes (not MOOCs provided
International system that rates standardized): by the Delft
Learning facilitates social 3.4 percentage University of

Analytics & comparison of point increase in Technology on the
Knowledge current students course completion edX platform
Conference with previously rates
successful
learners.
(Continued)
941
942
TABLE 3B
Massive Open Online Courses (MOOCs) (Continued)
Unit of
Davis et al. (2018) Journal of MOOC-based Null effect on Null Learners enrolled 1,047 students Students
Learning Adaptive Retrieval grades and in edX geoscience
Analytics Practice System, obtaining course course
which delivers certificate
quiz questions
from prior course
units
Kizilcec et al. Open Education “Collectivist,” 1. Null effect on Main reported A subset of 3,907 students in Student
(2014) Europa eLearning “individualist,” or learners’ decision effect sizes (not learners who first study, 7,522
Papers “neutral” emails to contribute to standardized): enrolled in a students in second
sent to MOOC the forum 1. Null MOOC on an study
participants to 2. Negative effect 2. Significantly undergraduate-
encourage forum on contributions lower contributions level computer
participation made by learners from those science topic
receiving the receiving individual at a major US
individualist encouragement university
encouragement and collectivist
and the messages (exact
collectivist magnitude not
message versus reported).
those receiving
the neutral
message
(Continued)
TABLE 3B
Unit of
Kizilcec et al. Science Mindset 1. Positive effects Main reported 2 samples: 2,286 students in Student
(2017) interventions on persistence effect sizes (not 1. Students from a first experiment,
addressing for learners in standardized): computer science 1,165 students
social identity less-developed 1. First experiment MOOC offered at in second
threat using a countries (LDCs) doubled Stanford experiment
“value relevance and null effects persistence for 2. Students in a
affirmation” on persistence learners in LDCs; 6-week Harvard
exercise and a for learners in null for learners in MOOC
“social-belonging moe-developed MDCs;
intervention” countries 2. Second
(MDCs); experiment: the
2. In the second social belonging
experiment, the intervention
social belonging increases
intervention persistence for
increased LDC learners
persistence for without affecting
LDC learners persistence for
without affecting MDC learners,
persistence and the affirmation
for MDC experiment
learners, and reduced
the affirmation persistence for
experiment MDC learners,
reduced but increased

persistence for persistence for
MDC learners, LDC learners
but increased (exact magnitude
persistence for not reported)
LDC learners
(Continued)
943
944
TABLE 3B
Unit of
Lamb et al. (2015) ACM (Association Self-assessment Positive effect on Main reported MOOC students 4,777 students Student
for Computing questions aimed engagement and effect sizes (not in JusticeX, a
Machinery) to improve forum participation standardized): HarvardX course
Learning @Scale participation for 4.2 more forum
Conference Paper MOOC students: actions than users
1. A self- in the control
participation group.
check,
2. Discussion
priming and
3. Discussion
preview emails
Martinez (2014a) EdPolicyWorks Emails informing Positive effect Main reported Students 7,924 students Student
Working Paper students of their on performance effect sizes (not registered for a
relative position on subsequent standardized): Coursera MOOC,
in the course: quizzes from both 2 percentage point Foundations of
1. A “positive” “positive” and increase from Business Strategy
one telling how “negative” emails “positive” emails at the Univeristy
many students and 3 percentage of Virginia
recipients did point increase from
better than, and “negative” emails
2. A “negative”
one stating how
other students
outperformed the
recipient
Martinez (2014b) EdPolicyWorks E-mails on Positive effect Main reported Students from the 24,122 students Student
Working Paper the negative on course effect sizes (not third and fourth from third
correlation completion. standardized): Foundations of MOOC, 5,675
between 16.85 percent Business Strategy from fourth
procrastination increase in course MOOC at the MOOC
and achievement completion. University of
Virginia
(Continued)
TABLE 3B
Unit of
Patterson (2018) Journal of 1. A commitment 1. Commitment 1. 0.29 standard MOOC 657 students Student
Economic device where device: positive deviations participants in a
Behavior and students effect on time 2. Null Stanford OpenX
Organization pre-commit spent working on course
to time limits course, course
on distracting grade, and course
Internet activities; completion
2. A reminder 2. Reminder
tool by time spent and focusing
on districting treatments: null
websites; effects
3. A focusing
tool that allows
students to block
distracting sites
whole on the
course website
Yeomans and Reich Proceedings Open-ended Positive effects on Main reported Students in 1,792 students Student
(2017) of the Seventh planning prompts course completion effect sizes (not 3 HarvardX
International asking students standardized): MOOCs
Learning to describe any 29 percent
Analytics & specific plans they increase in course
Knowledge made to engage completion.
Conference course content

and complete
assignments on
time.
945
completion of all participants the best metric With knowledge accumulating about how
of success, given that many participants may behavioral barriers get in the way of realiz-
not even aim to complete courses? More ing better long-run outcomes, technology
work is needed to hammer out what out- may increasingly be used to develop simple
comes should be measured, beyond comple- and inexpensive solutions to give individuals
tion rates, to judge the success through closer more support for making better choices. The
investigations of where specifically they may predominant model of policy intervention
add value to the education process. This will that has arisen from the behavioral econom-
in turn require more nuanced study of stu- ics literature is the “nudge,” an “aspect of
dents’ reasons for accessing MOOCs, and, the choice architecture that alters people’s
more broadly, the role of MOOCs within the behaviors in a predictable way without for-
broader education field. bidding any options or significantly chang-
ing their economic incentives” (Thaler and
Sunstein 2008). Nudges can often be carried
6.
Technology-Enabled Behavioral
out inexpensively through technologies like
Interventions
text messages.
Next, we shift focus to education technol- We identified 47 experimental papers
ogies that draw on the theory and practice of studying technology-enabled behavioral
behavioral economics to guide students (and, interventions. These studies evaluated pro-
in some cases, their family members and grams aimed at solving a wide variety of
teachers) toward behaviors that are expected problems and drawing on a variety of tech-
to facilitate greater academic achievement. niques implemented at different points
The idea behind this approach is that people across the education life cycle, from giving
are subject to systematic biases in d ecision parents ideas of how to practice reading
making and other psychological factors that skills in early childhood to reminding college
lead to suboptimal outcomes (Kahneman students to submit for financial aid. In par-
2011), like parents neglecting to read to chil- ticular, we identified studies of interventions
dren despite the best of intentions, or high across four clusters: seven on encouraging
school graduates not enrolling in college as a parental engagement in learning activities,
result of missing financial aid or registration 13 on attempting to improve information
deadlines. The behavioral insights literature flows in postprimary and secondary school,
was relatively slow to come to the education 19 on encouraging success in transitioning to
sector, but behavioral economics research in and through college, and 15 on psychosocial
education has taken off over the past several interventions. Information on the studies
years (Koch, Nafziger, and Nielsen 2015; is presented in tables 4A–4D. The studies
Lavecchia, Liu, and Oreopoulos 2016; Levitt show strong promise in each of these areas.
et al. 2016). Behavioral issues are especially While technology access interventions aim
important to think about in the context of to create an enabling environment for learn-
education, since important long-run deci- ing, and CAL and online courses attempt to
sions are being made during a time when directly improve the efficiency of academic
the brain’s ability to think of the future is and cognitive skills investments, behavioral
not fully developed. While we all face chal- interventions typically seek to overcome
lenges in making decisions involving uncer- hurdles in the education process brought
tain long-run benefits and immediate costs, on by gaps in certain types of noncognitive
children and youth particularly struggle skills. The term “noncognitive” skills refers
(Lavecchia, Liu, and Oreopoulos 2016). to a wide range of social and emotional skills,
TABLE 4A
Technology-Enabled Behavioral Interventions (Early Childhood)
Unit of
Cortes et al. (2018) NBER Working Text messag- Negative effect 1. 0.25 standard Preschool stu- 648 parents Parent
Paper ing program to from having too deviation decrease dents and families
nudge parents of few messages (1/ from 5/week texts in Dallas, Texas
kindergarteners week) and too compared to 3/
to engage in many messages week
literacy activities (5/week) com- 2. 0.19 standard
with children pared to baseline deviation decrease
(continuation of (3/week) from 1/week text.
Doss et al. 2018).
Varies content
and number of
text messages per
week.
Doss et al. (2018) NBER Working Text messag- 1. Positive effects 1. 50 percent Kindergarten stu- 794 students Student
Paper ing program to on reading level more likely to dents and families
nudge parents of 2. Positive effect read at a higher in California
kindergarteners to on parental level compared
engage in literacy engagement in to the general
activities with literacy activities group (effect size
children (con- by compared to not standardized)
tinuation of York the control group 2. 0.31 standard
and Loeb 2014). 3. Null effects for deviation increase
Includes addi- other treatment in literacy activities
tional differenti- arm 3. Null
ated/personalized
treatment arm
Kraft and ANNALS of the Parents texted Positive effect on 0.21 to 0.29 183 US families 232 students in Student
Monti-Nussbaum American Acad- to encourage to reading compre- standard deviations (elementary 183 families
(2017) emy of Political engage in activi- hension, more increase for third school kids)
and Social Science ties to counteract than compensat- and fourth graders
summer learning ing for summer on reading compre-
loss learning loss hension
947
(Continued)
TABLE 4A
Technology-Enabled Behavioral Interventions (Early Childhood) (Continued)
948
Unit of
Mayer et al. (2019) Journal of Human Texting program Positive effect on 1. 1.01 standard Parents of chil- 169 parents Parents
Resources to promote learn- reading applica- deviation increase dren in subsidized
ing engagement tion usage in minutes read preschool pro-
of Head Start 2. 0.60 standard grams in Chicago
parents deviation increase
in number of books
read and
3. 0.31 standard
deviation increase
in any reading
Meuwissen et al. Minnesota Chil- Text2Learn, a 1. Positive effect Main reported Parents of pre- 110 parents Parents
(2017) dren’s Museum mobile phone on parental effect sizes (not school children in
and the Center texting program engagement in standardized): Minnesota
for Early Educa- for low-income more literacy 1. Parents reported
tion and Develop- parents of pre- activities with engaging in more
ment, University schoolers. their children literacy activities
of Minnesota 2. Null effects with their children
on community after receiving the
resources usage, texts (exact magni-
such as libraries, tude not reported)
or changes in 2. Null
attitudes about
literacy
York and Loeb Journal of Human Text messaging Positive effect on 1. 0.22–0.34 Parents of 4-year- 440 parents Parent
(2019) Resources program to nudge engagement in standard deviation olds in California
preschool parents literacy activities, increase in literacy
to engage in liter- parental involve- activities
acy activities with ment at school, 2. 0.13–0.19

children and learning gains standard deviation
increase in parental
involvement at
school, and
3. 0.21–0.34
standard deviation
increase in learning
gains
most of which are well outside the scope of might contribute toward reducing dispari-
this review. We refer instead to the much ties in family engagement.
narrower set of n oncognitive skills—such Why might nudges be expected to
as time management, motivation, and resil- increase parental learning engagement
ience—that can conceivably be affected by within disadvantaged households? After all,
nudges, like reminders, or messages contain- behavioral interventions are unlikely to sub-
ing small bits of information and encourage- stantially address resource constraints like
ment. While these nudge interventions may the time scarcity faced by low-income par-
or may not actually directly help to develop ents. However, the behavioral economics
the noncognitive skills in question, they at literature suggests that cognitive constraints
least attempt to compensate for gaps to lower as well as resource limitations lead to under-
their potential for inhibiting the education investment. Even when cognitive burdens
process. In the remainder of this section, we themselves are aggravated by resource con-
review the rationale and evidence underlying straints, small adjustments in the decision
each of the four clusters enumerated above structures that people face can help to cor-
in turn. rect these biases and move them toward
more optimal behavior (Thaler and Sunstein
6.1 Encouraging Parental Learning
2008).
Engagement during Early Childhood
So, in the present context, a behavioral
economics perspective would indicate poten-
6.1.1 Background, Context and Mechanisms
tial benefits from reminders and instructions
of Impact
inspiring and guiding parents toward more
Research suggests that an effective strat- productive engagement. Behavioral inter-
egy to improve educational outcomes is ventions are likely to be helpful to the extent
for parents to engage in learning activities that the current lack of parental involvement
with their children (Price 2010, Sénéchal in learning activities reflects frictions like lim-
and Lefevre 2002). But parents report ited mental bandwidth (Schilbach, Schofield,
spending less time on these activities than and Mullainathan 2016) and procrastination
might be expected in light of the possible (Thaler and Benartzi 2004) rather than the
benefits. The problem of low engagement reflecting w ell-reasoned cost–benefit deci-
is particularly acute among disadvantaged sions or insurmountable resource barriers.
households, a pattern that may reinforce In the context of our framework, interven-
broader disparities in educational outcomes tions within this category could help to com-
(Guryan, Hurst, and Kearney 2008; Kalil pensate for deficits in n oncognitive skills
2015; Lee and Bowen 2006). Policy makers associated with procrastination.
have found cost-effective responses elusive,
with even expensive and resource-intensive
programs turning up modest results (York, We identified six experimental evaluations
Loeb, and Doss 2019). Yet because young of technology-based interventions aiming
children spend a great deal of time at home, to increase the quantity and quality of time
school-based programs cannot substantially spent by parents practicing skills with their
substitute for engagement “unless they preschoolers (Cortes et al. 2018, Hurwitz et
are very intensive, extensive and expen- al. 2015, Mayer et al. 2019, Meuwissen et al.
sive” (Mayer et al. 2015). This dilemma has 2017, York and Loeb 2014), kindergarten-
inspired a growing literature that explores ers (Doss et al. 2018), or fi
rst–fourth grad-
whether and how behavioral interventions ers (Kraft and Monti-Nussbaum 2017). All
of the programs studied relied centrally on impressive given the exceedingly low costs
sending text message reminders to parents, of the intervention, at less than a dollar per
and all found positive results. family (York and Loeb 2014). To build on the
Ready4K—a preschool literacy pro- promising evidence, Ready4K is now explor-
gram implemented in San Francisco— ing whether the program can improve math
was the earliest experimentally evaluated, performance and executive functioning.
technology-based intervention to leverage
Ensuing research has attempted to work
this rationale to improve parental learn- toward untangling the specific mechanisms
ing engagement. The program sent parents underpinning the effectiveness of this type
three text messages per week with tips and of intervention, as well as better under-
encouragement to engage in literacy activ- standing potentially differing effects across
ities (York and Loeb 2014). The behavioral subgroups. The only remaining intervention
logic that guides Ready4K suggests that “the of this type for preschoolers to have been
complexity of parenting may overwhelm experimentally evaluated took place within
some parents, leading them to underinvest Midwestern Head Start and Early Head Start
in their children” (York and Loeb 2014). centers. The program provided all sample
Furthermore, literacy activities constitute a households with tablets containing numer-
case of “delayed gratification,” necessitate ous children’s books (Mayer et al. 2019).
“interrupting the status quo,” and are often The treatment group additionally received
overcome by “limited attention” (York and three nudges—daily text message reminders
Loeb 2014). So the program sends sugges- to read to the kids, a g oal-setting tool that
tions of small, easy tasks that parents can asked the parents to set reading goals and
carry out without feeling overwhelmed; reported back on whether these goals were
provides encouragement to sustain parents’ met, and social rewards, specifically congrat-
investment in longer term gratification; pro- ulatory texts or cartoons when goals were
vides tips for integrating the activities into reached (Mayer et al. 2019). Following the
daily life so that the status quo barrier can six-week study period, the group receiving
be overcome; and addresses attention con- the behavioral interventions used the tablet
straints by regularly reminding parents. a full standard deviation more than parents
The study found an impact of 0.29 stan- who did not. They read more than twice as
dard deviations of the program on a compos- many books to their children, with control
ite score for “global early literacy parenting” group families reading an average 14.8 books
measuring activities like reading to a child, during the six-week intervention period
pointing out words that rhyme, and taking while treatment group families read an aver-
the child to a library or museum (York and age of 31.4 books. The fact that effects were
Loeb 2014). The study also found effect sizes stronger for more p resent-oriented than for
ranging from 0.21 to 0.34 standard deviations more future-oriented parents suggests that
on the Phonological Awareness Literacy the mechanism was behavioral—in this case,
Screening (York and Loeb 2014). The fact helping to correct for temporal bias—rather
that the program led to an increase in specific than simply arising from the introduction of
literacy tasks but not general ones suggests new information (Mayer et al. 2019).
that the impact was likely generated by the As children progress from preschool to
program’s provision of particular, manage- kindergarten and then first grade, they tend
able tasks, rather than reminding parents to to spend larger shares of their time at school.
engage in activities they might have engaged To what extent might programs like the ones
in anyway. The effect sizes detected are described above prove effective beyond
preschool? Two interventions were recently that the texting intervention improved read-
experimentally evaluated that adopt a sim- ing comprehension scores for students in the
ilar model, but for kindergarteners (Doss treatment group by 0.21–0.29 standard devi-
et al. 2018) and fi rst–fourth graders (Kraft ations (Kraft and Monti-Nussbaum 2017).
and Monti-Nussbaum 2017) instead of pre-
schoolers. The kindergarten intervention
was an extension of Ready4K with the eval- Given the enormous importance of early
uation including the same preschool sample childhood skill development for the entire
as the children entered kindergarten, along educational life course, coupled with the
with additional San Francisco kindergar- central role of parents and caretakers in this
teners (Doss et al. 2018). Beyond trying skill development process, behavioral nudges
to replicate the same intervention within a to encourage educational interactions with
kindergarten context, a second treatment children represent a vital area for continued
arm was also added that sent parents “per- experimentation and s cale-up. New work in
sonalized” and “differentiated” texts. Texts this area should explore several directions.
to parents in this second treatment arm con- One important question to address is when
tained child-specific information and sent and how customization should be factored
recommendations for tasks matching the in. Of the studies discussed here, only one
child’s level. Interestingly, the researchers study (Doss et al. 2018) involves a customized
found the original treatment that had been treatment arm, and this arm shows strong
effective in preschool showed no significant effects. While non-customized interven-
effects in kindergarten. However, the per- tions show positive effects at the preschool
sonalized and differentiated text messages (Cortes et al. 2018; Mayer et al. 2019; York,
did show substantial benefits, with children Loeb, and Doss 2019) and third and fourth
whose parents received the treatment “50 grade levels (Kraft and Monti-Nussbaum
percent more likely to read at a higher level” 2017), it is noteworthy—although of course
(Doss et al. 2018). far from conclusive—that non-customized
Finally, the most recent intervention, interventions show no effects for kindergar-
falling into this category, to undergo exper- teners (Doss et al. 2018) and first– second
imental evaluation extended the idea of tex- graders (Kraft and Monti-Nussbaum 2017).
ting parents to encourage engagement in Customization can add costs, but may lead to
literacy activities to the fi
rst–fourth grades. greater cost effectiveness for particular ages
Recognizing that elementary students spend and particular subjects or learning tasks—
more time engaged in school throughout sorting this out is an empirical question
the year, this intervention targeted a specific that will require m ulti-armed experiments.
point of friction within the elementary edu- Moreover, with the growing sophistication
cation process—“summer reading loss”— of artificial intelligence, there is potential for
the tendency of elementary students to fall customized text messaging platforms to be
behind in their reading skills because of the developed at a lower cost.
gap in practice they experience during the Second—as suggested by the large effect
summer. Parents in the treatment group sizes found in the Head Start study (Mayer
were sent 18 text messages across July and et al. 2019)—a potentially promising route
August with information on the importance is to overlay messaging interventions onto
of summer reading, as well as affordable and other interventions providing resources.
accessible resources to practice reading and Program designers have often struggled
tips on parental engagement. The study finds with the challenge of distributing potentially
v aluable resources, only to come up against that constitutes parenting. This issue may
low usage rates. Similarly, it is easy to imag- be especially important for low-performing
ine that, while tips and reminders might schools, which already exhibit lower rates
be helpful, parents would be more likely to of communication satisfaction from parents
heavily engage with a robust set of resources and where parents may be relatively more
at their disposal, like books to read to their constrained in their ability to absorb moni-
children. It is thus worth exploring how toring costs (Bergman 2015).
messaging interventions could complement Within our framework, improving p arent–
other resource distribution programs. child information flows can help to build or
Finally, it will be important for researchers compensate for noncognitive skill deficits.
to explore the effects of delivery mechanisms From the perspective of parents, many of
other than text messages. While text messages the same dynamics mentioned in the pre-
are still relatively novel for many parents ceding section may hold here. Parents may
and are likely more noticeable than email, be aware of the importance of obtaining
this may gradually change as organizations information on their children’s performance
increasingly leverage text messages for pro- and providing guidance accordingly, but they
motions and parents generally become more may procrastinate given the many important
inundated with text messages. Messages topics competing for attention, or they may
can also be sent through WhatsApp, social feel overwhelmed and have trouble getting
media, and other platforms. Experiments started. Children also face many compet-
could compare the effects of different plat- itors for their attention and are notoriously
forms—or combinations of platforms—in wont to prioritize social identity over aca-
multiple treatment arms. demic performance (Akerlof and Kranton
2002), leading them to procrastinate or
6.2 Improving School–Parent Information
forego schoolwork altogether if not suffi-
Flows
ciently prompted by parents. To the extent
that a child’s academic standing is clear and
salient to parents, this could spark the action
needed to cultivate the child’s conscien-
As children get older, the role of par- tiousness, which is one of the traits that has
ents shifts away from practicing skills with been identified as closely associated with
their kids directly and towards encouraging academic and professional success (Almlund
the kids to put more effort into school. So, et al. 2011, Almund 2011).
behavioral interventions for middle and high
schoolers tend to focus on sending parents
information on their kids’ performance— We identified 13 R
CT-based studies eval-
for example updates on grades, attendance, uating technology that seeks to improve the
and behavior—to nudge the parents toward flow of information in p
ostprimary and sec-
providing this encouragement. If parents are ondary school, the majority of which focus on
constrained by a gap in information on how improving school–parent information flows
hard their children are working or how well (Balu, Porter, and Gunton 2016; Bergman
they are performing, and if children are not 2015, 2016; Bergman, Denning, and Manoli
already expending maximum effort, then 2019; Bergman and Chan 2017; Bergman
closing these gaps may provide parents with and Rogers 2016; Fryer 2016; Kraft and
the opportunity to apply that alchemical com- Dougherty 2013; Kraft and Rogers 2015;
bination of guidance, pressure, and support Kraft and Monti-Nussbaum 2017; Rogers
and Feller 2018; McGuigan, McNally, and 22 middle and high schools in a district of
Wyness 2012). These programs followed two West Virginia (Bergman and Chan 2017).
main approaches: first, sending information to This program automatically pulled infor-
parents that was generated as part of regular mation from the schools’ student informa-
school records (like grades and attendance) tion system and texted it directly to parents.
and, second, having teachers send personal- Parents received weekly texts stating the
ized messages to parents. Overall, these stud- number of classes and/or assignments that
ies have found positive results, indicating a students had missed, as well as monthly texts
potentially fruitful set of opportunities. if their child was averaging below 70 per-
The majority of the s chool–parent informa- cent on any class (Bergman and Chan 2017).
tion flow interventions that have been exper- Because of the automation, the intervention
imentally evaluated fall into the first of the was extremely cheap, with 32,000 text mes-
two categories listed above. The first inter- sages totaling to only $63 and training com-
vention in this category to be experimentally ing down to $7 per student (Bergman and
evaluated was a program aimed at middle Chan 2017). The study showed impacts that
and high school students at a single public were very impressive, given the low costs of
school in a low-income neighborhood of Los the intervention: the treatment group saw a
Angeles (Bergman 2015). Parents whose 39 percent reduction in failed courses and
children were in the treatment groups were an 18 percent increase in class attendance,
notified when their children missed attend- meaning that the treatment group attended
ing class or missed an assignment through 50 more classes on average and had a 0.10
text messages, phone calls, and e-mails. standard deviation improvement in GPA
Following the semester-long intervention, (Bergman and Chan 2017). Interestingly,
students in the treatment group had earned the data suggest that parents already had a
GPAs and standardized math test scores good idea of their children’s final grades, but
that were about 0.20 standard deviations the program reduced parents’ underestima-
over those of the control group (Bergman tion of the number of assignments their kids
2015). An evaluation of a similar interven- were missing, which likely helped to better
tion—Papás al Día (“Parents up to Date”), target the pressure they placed on their kids
carried out in two low-income municipali- to increase effort (Bergman and Chan 2017).
ties of Santiago, Chile—also finds positive The strongest benefits went to those with
results, including a 0.09 standard deviation below-average GPAs, who saw a reduction
improvement in math grades, a reduction in in class failures of 0.9 classes, an increase in
bad behavior, and positive spillover effects attendance of 64 classes, and a GPA increase
within classes (Berlinski et al. 2016). of 0.26 points out of a 4.0 scale (Bergman
While these two interventions sought to and Chan 2017).
channel existing information on students’ In contrast, another fully automated inter-
performance to parents rather than gener- vention that focused exclusively on atten-
ating new information, both were somewhat dance showed no evidence of improving
labor intensive, requiring substantial man- attendance rates (Balu, Porter, and Gunton
ual data entry. More recent interventions 2016). Here, parents of New York City
have tended to automate the process to Public School students received automated
the greatest extent possible to cut down on text messages on each day their student did
costs. One recent experimental study eval- not show up for school, in addition to weekly
uated the effects of an automated school– attendance reports. This lack of impact may
parent information program on a sample of have arisen from the intervention’s e xclusive
focus on attendance, its location in New This massive difference in adoption shaped
York City (which may be more saturated the effectiveness of the texting program
with automated information flows than most in generating academic performance out-
other environments), or something more comes: while no significant effects on perfor-
contingent and specific to the interven- mance outcomes emerged from the first two
tion in question. However, it is noteworthy treatment groups, the automatic enrollment
that this study—the only one to focus on an group saw improvements in GPA by roughly
attendance-only intervention—was also the a quarter to a third of a letter grade, and
only one to show no effects. Further research reduced class failure by roughly a fifth to a
will be needed to explore the possibility that quarter (Bergman and Rogers 2016). These
it is effort within school, rather than atten- lessons on the importance of encourage-
dance, that constitutes a binding constraint ment and especially o pt-in systems to pro-
on p arent–child information flows in most mote technology adoption are relevant to a
high school settings. broad range of e d-tech applications, but are
Two recent studies have highlighted an mentioned here since they were evaluated in
important qualification to the line of research reference to school–parent communication
just described (Bergman 2016). While tech- intervention.
nologies that improve school–parent infor- The interventions discussed so far
mation flows may be effective in improving in this subsection attempt to transfer
education, these effects will be heavily medi- already-existing information to parents.

ated by the extent to which the technologies Another approach that has been experimen-
are actually used. For instance, one recent tally evaluated in the context of two separate
study showed a letter and phone call prompt- interventions has teachers communicate
ing students to access an online system con- personalized messages to parents. The first
taining attendance and grades significantly experimentally evaluated intervention fall-
increased rates of access and ultimately ing into this category took place during
resulted in a GPA increase of 0.10 points a required summer program in a Boston
(Bergman 2016). Another program—this charter school (Kraft and Dougherty 2013).
one conducted in a dozen Washington, DC Parents received two communications per
middle and high schools—offered text mes- day for five consecutive school days—a
sage updates of the kind mentioned above, phone call from an English teacher and
but varied in how the program was imple- a text message from a math teacher. The
mented (Bergman and Rogers 2016). Three intervention improved engagement as mea-
treatment groups—one that received a text sured by three variables: homework comple-
instructing them on how to sign up online for tion, participation, and number of instances
the service, one that received a text inviting in which teachers had to direct students’
sign-up through a text message response,
attention back to the topic at hand (Kraft
and one that automatically enrolled parents and Dougherty 2013). Qualitative evidence
in the texting program but gave them the suggests that this effect occurred through
opportunity to opt out—were contrasted three mechanisms: improving relationships
with a control group that did not receive any between students and teachers, expanding
prompt to sign up for the texting service. parental involvement, and increasing stu-
Only 1 percent of participants in the first dents’ motivation (Kraft and Dougherty
group and 8 percent in the second group 2013).
signed up, while only 4 percent in the auto- The second intervention in this category
matic enrollment group chose to opt out. took place “during a traditional summer
school program offered by a large urban within this latter area may increase as schools
school district in the Northeastern United increasingly turn to computerized grading
States” (Kraft and Dougherty 2013). Here, and feedback systems.
teachers themselves wrote out o ne-sentence It would be productive to extend research
messages that were then sent to parents on parent–child information flows in several
weekly by research assistants through directions. The interventions studied so far
text message, phone, or email (Kraft and have been focused primarily on providing
Dougherty 2013). Two separate program parents with information about their chil-
variations were given: one consisting of “pos- dren’s performance, but without any guid-
itive” messages about what the student was ance on what to do with this information.
already doing well, and the other consisting Messages could be coupled with actionable
of “improvement” messages about areas that tips about how to encourage struggling stu-
the student could use work on. Averaging dents, or resources like informative websites
across the two treatment arms, inclusion in or contact information for school counselors.
the program led to an increase in the success And they can be imbued with tiers, or impor-
rate of students passing the class and obtain- tance rankings, in order to catch a parent’s
ing the credit, up 6.5 percentage points from attention if something especially worrying is
an 84.2 percent passing rate in the control going on (e.g., if the student is on the verge
group. Interestingly, the impact estimate is of failing a class). As in the preceding section,
substantially higher for the improvement experimentation with more mechanisms
treatment arm, although the experiment of communication beyond text messages
lacks the power to detect significance in this would be worthwhile. And sending chil-
difference (Kraft and Dougherty 2013). The dren information or alerts relating to their
program seems to work not by increasing performance—with the understanding that
the amount of time parents spend talking their parents will receive a message if things
with their kids about school, but rather by do not change—could also increase student
directing the content of these conversations. effort without even necessarily involving the
The program also seems to have led to the parents.
unintended consequences of lower stu-
6.3 Transitioning to and Succeeding in
dent perceptions of their own performance,
College
and weaker student–teacher relationships
as reported by teachers also (Kraft and
Dougherty 2013). Perhaps the best of both
variations could be captured by sending mes-
sages that include actionable steps as in the Another area of focus for technology-based
“improvement” version, but are more posi- nudge interventions in the education sector
tive in tone. has been the challenge of transitioning to and
making it through college. The behavioral
economics literature suggests that people—
Overall, other than the lack of impact and especially children, adolescents, and
generated by the New York City attendance young adults—tend to rely heavily on rou-
program, interventions that seek to improve tines, and the transition to college requires
school–parent information flows seem highly students to break from routine (Lavecchia,
promising. They may be especially cost effec- Liu, and Oreopoulos 2016). The behavioral
tive when automated and drawing on existing literature has also documented the paralyz-
school administrative data. Opportunities ing effect of too much information and too
956
TABLE 4B
Technology-Enabled Behavioral Interventions (Primary and Secondary)
Unit of
Balu, Porter, and MDRC Automated text Null effect Null New York City 3,957 students Student
Gunton (2016) messages to high school
parents of high students
school students
informing about
absence
Bergman (2015) CESifo Working Automated texts 1. Positive effects 1. 0.19 standard Students in grades 462 students Student
Paper to parents about on high school deviation increase in 6–11 in Los
performance GPA high school GPA 2. Angeles
2. Decrease in 7.5 percentage point
missing final exam decrease of missing
project final exam project
3. Positive effect (not standardized) 3.
on math standard- 0.21 standard devi-
ized exam scores ation increase for
4. Null effects on math standardized
English exam scores
4. Null for English
Bergman (2016) CESifo Working Learning Manage- 1. A quarter Main reported effect 15 US school 59 schools School
Paper ment System (par- of parents use sizes (not standard- districts operating
ents have access online portal 2. ized): 24 percent of learning manage-
to an online Adoption follows parents log into the ment company;
portal with child’s an S-shape 3. system 2-stage experi-
classes, grades, Significant spill- ment providing
assignments, etc.) overs occur along families their
intensive but not account informa-
extensive margins tion in schools

4. There is across 3 districts.
evidence student
grades improve as
a result
(Continued)
TABLE 4B
Technology-Enabled Behavioral Interventions (Primary and Secondary) (Continued)
Unit of
Bergman and Chan CESifo Working Automated texts 1. Positive effect 1. 40 percent 22 middle and 1,137 students School-by-grade
(2017) Paper to parents about on reducing reduction in course high schools in in 22 middle and level
performance course failures, failures (not stan- Kanawha County high schools
GPA, and atten- dardized) Schools in West
dance 2. 0.10 of a point Virginia
2. Null effects on GPA increase for
state math and middle school
reading scores students and .25
of a point GPA
3. Positive effect
increase for high
on in-class exam
school students (not
scores
standardized) 3. 17
percent increase in
class attendance (not
standardized)
4. Null in state math
and reading scores
5. 0.10 standard
deviation increase on
in-class exam scores
Bergman, Economics of Providing regular 1. Positive effect 1. Information only: Students’ families 1,121 students’ Students’ family
Edmond-Verley, Education Review information to on GPA and math 0.13 standard devi- from 3 participat- families
and Notario-Risk families about and reading test ations in GPA; null ing schools in an
(2018) their child’s scores for the on math and reading urban, Midwest-
academic progress treatment arm test scores ern school district
in one arm and with home visits 2. Information +
supplementing 2. Positive effects skills: 08 standard
with home visits on retention deviations increase

in GPA; 0.13 and
on skills-based for both groups
0.12 standard
information in a during the year
deviation increase
separate arm
in math and reading
test scores
3. 4 percentage
points less likely to
leave the district
957
(pooled)
(Continued)
958
TABLE 4B
Unit of
Bergman and Hill Economics of Publishing 1. High-perform- A 1 standard devi- hird–fifth grade
T 3,089 teachers Not random-
(2018) Education Review teacher ratings ing students sort ation increase in a teachers in Los ized—regression
online into classrooms teacher’s value added Angeles discontinuity
with highly causes the math test design
rated teachers scores of incoming
2. Low-rated students to be three
teachers teach tenths of a standard
lower-performing deviation higher on
students and are average than those
more likely to of a similar teacher
leave the district whose rating is not
in subsequent published
years relative
to higher-rated
teachers
Bergman and Society for Text message to 1. Positive effect 1. 0.06 standard Middle and high 6,976 students Student
Rogers (2016) Research on Edu- parents regard- on grades for deviation increase school students in
cational Effective- ing their child’s those assigned to from being assigned 12 US schools
ness Spring 2016 academic perfor- opt out group to opt out for term
Conference Paper mance, including 2. Overall, 3 and 0.04 standard
grades, upcoming positive effect on deviation increase
tests, and missing grades for term 4
assignments 2. 0.05 standard
deviation increase
in grades overall for
terms 3 and 4
(Continued)
TABLE 4B
Unit of
Bursztyn and Quarterly Journal Two interventions: 1. Negative Main reported effect Study 1: students 5,000 students in Student
Jensen (2015) of Economics 1. Performance effect on student sizes (not standard- across more than first study, 825
leaderboard into performance; the ized): 100 schools in Los students in second
computer-based decline appears 1. 24 percent per- Angeles study
high school to be driven by formance decline Study 2: 26
courses a desire to avoid 2. 11 percentage classrooms across
2. Complimentary the leaderboard point decrease 4 schools in Los
access to an 2. In non-honors in sign-ups in Angeles
online SAT pre- classes, sign-up non-honors classes
paratory course; was 11 percentage when decisions were
sign-up forms points lower when public rather than
differed randomly decisions were private; null in hon-
across students public rather than ors class sign-up.
only in whether private; honors
they said the class sign-up was
decision would be unaffected
kept private from
classmates
Fryer (2016) Journal of Public Provided free 1. Positive impact Main reported effect Students in sixth 1,907 students Student
Economics cellular phones on students’ sizes (not standard- and seventh
and daily infor- reported beliefs ized): grades in Okla-
mation about about the rela- 1. 23 percent more homa
the link between tionship between questions answered
human capital and education and correctly on rela-
future outcomes outcomes tionship between
via text message 2. Null effects on human capital and

in one treatment student effort, life outcomes
and minutes to attendance, sus- 2. Null
talk and text as pensions, or state
an incentive in a test scores; evi-
second treatment dence of positive
impact on college
entrance exams 4
959
years later
(Continued)
TABLE 4B
960
Unit of
Kraft and Dough- Journal of Parents texted on 1. Positive effect Main reported effect Rising sixth and 140 students Student
erty (2013) Research on student behavior/ on homework sizes (not standard- ninth grade stu-
Educational performance completion ized): dents in Boston,
Effectiveness 2. Decreased 1. 40 percent Massachusetts
instances in which increase in home-
teachers had to work completion
redirect students’ 2. 25 percent
attention to the decrease in instances
task at hand in which teachers
3. Positive effect had to redirect stu-
on class participa- dents’ attention
tion rates 3. 15 percent
increase in class
participation rates
Kraft and Rogers Economics of Parents texted on Decreased the Main reported effect High school 435 students Students
(2015) Education Review student behavior/ percentage of stu- sizes (not standard- students in North-
performance dents who failed ized): 41 percent eastern United
to earn course reduction in failure States
credit to earn course credit
(Continued)
TABLE 4B
Unit of
McGuigan, Journal of Human Information Positive effect on Main reported effect Year 10 students 6,614 students in School
McNally, and Wyn- Capital campaign about understanding of sizes (not standard- in England 54 schools
ess (2012) the costs and ben- university educa- ized):
efits of pursuing tion financing and 1. 41 percent
post-compulsory secondary school increase in knowing
education retention; null when university fees
impact on plans are paid
to apply to and 2. 25 percent
enroll in college increase in knowing
the student loans are
a cheaper/better way
to borrow
3. 16 percent
increase in staying in
full-time education
after age 16
4. Null impacts
on the perceived
importance of
family constraints on
college enrollment
5. Null impacts on
plans to apply to and
enroll in college
Rogers and Feller Nature Human 1 of 3 person- Positive effect Main reported effect Low-income 28,080 households Households
(2018) Behavior alized message on reduction of sizes (not standard- households across
information treat- chronic absentee- ized): 10 percent 203 US schools
ments throughout ism comparably reduction in chronic
the school year across all grade absenteeism
levels.
961
many choices, and the transition to college and Manoli 2019). Another intervention,
is fraught with these as well (Lavecchia, conducted in a single public university,
Liu, and Oreopoulos 2016). In the context emailed letters to students explaining their
of our framework, behavioral interventions current financial aid package and associated
can help to overcome or compensate for plans, but this information too had negligible
noncognitive skill deficits that could oth-
effects (Darolia and Harper 2018).
erwise hamper a student’s entire academic On the other hand, two information inter-
career. For instance, procrastinating—or, ventions implemented in Canada and Chile,
for that matter submitting to information respectively, found positive effects. The
overload paralysis—can harm a student aca- first of these interventions showed videos
demically if it causes him or her to turn in on the benefits of higher education to stu-
a late or low-quality assignment. But miss- dents in disadvantaged Toronto high schools
ing an important admissions or financial and allowed the students an opportunity to
aid deadline can mean consequences that try out a financial aid calculator. Students
are orders of magnitude more significant, who participated in the program reported
for example delaying college, ending up in more favorable views of higher education
a low-quality institution, or even foregoing (Oreopoulos and Dunn 2013). The other
college all together. The interventions dis- program sent eighth graders in metropolitan
cussed in this subsection aim to provide stu- Santiago, Chile, DVDs containing practical
dents with tools of varying intensity to avoid information on higher education financing.
such problems. In particular, we review evi- Participants not only showed greater knowl-
dence on four main types of college-related edge of financial aid, but also were more
behavioral interventions: information cam- likely to enroll in college preparatory high
paigns, nudges to complete important tasks, schools, and also exhibited p ostsecondary
intensive application assistance, and college attendance rates that were 8.8 percent
advising. higher (Dinkelman and Martínez 2014). This
latter intervention is also unique among pro-
grams that have been experimentally evalu-
Our paper reviews 19 studies (included ated in that it targets higher education at the
in table 4C) that evaluate the use of eighth-grade level, which could allow more
technology-enabled behavioral interven-
time for participants to plan for college.
tions to support the transition to and success More research would be needed to defini-
in college. First, several interventions have tively explain why the first two interventions
sought to leverage information technology showed no results and the next two did.
to inexpensively provide students with more However, one possibility is that the first two
college-related information. On one hand, relied on letters, which the intended benefi-
two relatively minimalistic interventions in ciaries may or may not have found compelling
the United States generated no impact. One or even read. Perhaps video—the primary
of these—tested in a field experiment with medium used in the second two interven-
a sample of over a million prospective and tions—was more effective at capturing stu-
enrolled college students in Texas—sent one dents’ attention. For the Texas tax incentive
e-mail and one letter containing information experiment, a simpler interpretation—and
about higher education tax credits, but those the one offered by the authors—is that the
who received these showed no more likeli- tax incentives themselves did not figure into
hood of applying to or enrolling in college students’ decision making, and so more infor-
than those who did not (Bergman, Denning, mation on the incentives did not help.
Another approach to supporting the tran- Most recently, the largest experimen-
sition to college has been through nudge tally evaluated FAFSA n udge campaign
campaigns. Although the term “nudge” to date sent three versions of a message to
is applicable to interventions discussed low-income and fi rst-generation students fill-
throughout this section, here we use the ing out the Common Application encourag-
term “nudge campaigns” to refer to inter- ing them to apply early for the FAFSA. One
ventions providing sustained efforts to guide, version provided specific planning structure,
encourage, and/or remind program partici- one gave information on the human capital
pants about one or more aspects of college returns to college, and one attempted to
success. Five recent studies all suggest that advocate productive identities. No effects
nudge campaigns can be effective in improv- were found for the latter two frames, but
ing decisions and task fulfillment surround- the planning message led to a 1.1 percentage
ing financial aid and college matriculation point increase in college enrollment among
and enrollment. all recipients and 1.7 percentage points for
Of these, three interventions attempted first-generation college students (Bird et al.
to encourage better-informed financial aid 2017). In addition to supporting task com-
decisions. One program sent students at pletion related to financial aid, one nudge
a large community college in Baltimore campaign has been experimentally shown
County eight text messages over a period of to reduce “summer melt,” the phenomenon
several weeks prompting them to make more whereby students who are admitted to and
“active” financial aid decisions. The inter- indicate a decision to attend a particular col-
vention resulted in a 3.1 percentage point lege do not actually complete the matricula-
reduction among students who received tion process or do not actually show up for
the text messages in accepting unsubsi- classes (Castleman and Page 2015).
dized Stafford loans, and those who still A nudge campaign may be sufficient to
did accept the loans borrowed less. Results induce students to think through financial
were strongest among students showing less aid decisions and remind them to do the
financial literacy and with more debt. The right paperwork on time to enroll in and
study also produced some evidence that the get through school. However, it is perhaps
text messages led students who had attained less likely that nudges would be effective
marginal academic success to leave school at getting a student to fill out an admis-
earlier (Barr, Bird, and Castleman 2016). sions or financial aid application in the first
Another program sent text messages to col- place—this is a much more daunting task.
lege freshmen who, as high school students, We identified evaluations of two programs
had worked with a Massachusetts-based that leveraged technology for more intensive
education nonprofit called uAspire. The application assistance and support (Bettinger
messages encouraged students to refile et al. 2012, Oreopoulos and Ford 2019). In
the Free Application for Federal Student the first instance of these programs, fami-
Aid (FAFSA) for their sophomore year and lies with a college-age child who were filing
found an increase of nearly 14 percentage their taxes at H&R Block were given the
points on continuous enrollment through opportunity to quickly file their FAFSA at
sophomore year among students attending the same time. This was possible as a result
community colleges (those attending four- of a software program designed to automati-
year universities already had high rates of cally feed data from the tax entry system into
continuous enrollment) (Castleman and the FAFSA, collecting additional FAFSA
Page 2016). questions not covered during the course of
the regular tax filing in ten or so minutes upon which the program was based made
following the filing. College enrollment of it possible for the computer to respond to a
high school seniors with parents receiving large majority of incoming questions, saving
the treatment increased by 8 percentage scarce time for college advisers and admin-
points (Bettinger et al. 2012). The program istrators. For the sample of students that
LifeAfterHighSchool, on the other hand, had committed to attending Georgia State,
focused on providing support for the admis- the texting program increased enrollment
sions process directly to students by incorpo- there by 3.3 percentage points (Page and
rating relevant activities into the high school Gehlbach 2017).
curriculum (Oreopoulos and Ford 2019).
The program aimed to ensure that every
senior in participating high schools graduate The studies reviewed in this subsection
with a college program offer of acceptance suggest that behavioral interventions aiding
and a financial aid package. The program in the transition can exert substantial bene-
consisted of workshops involving interactive fits. It would perhaps not have been intuitive
activities, for instance having students enter to education economists a decade ago that
their grades into a computer program, which a few text messages could make the differ-
would then generate a list of local programs ence between attending and not attending
in their area for which they would likely be college, but recent evidence suggests pre-
accepted if they applied. In addition to large cisely this possibility for a significant num-
gains in application rates, college enroll- ber of students. The studies on information
ment increased by about 9 percentage points treatments reviewed above, while far from
among the seniors who had not been taking definitive, suggest that videos might be a
any u niversity-track courses (Oreopoulos more effective communication medium for
and Ford 2019). students (potentially) transitioning from high
Finally, two recent studies have examined school to college than letters. Those studies
the extent to which technology can be lev- relied on DVDs, but in the present day and
eraged to increase access to college advis- age, texting or emailing videos to students is
ing. One experiment conducted at a large a more likely approach that could be tested
Canadian university tested three treat- further. Text messaging campaigns to remind
ment arms: one-on-one coaching, an online students about financial aid and enrollment
exercise, and a text-messaging support deadlines have shown consistent success;
program. Only the one-on-one coaching though over time, saturation of receiving
arm showed significant results, potentially text messages might dampen effects. Future
indicating limits in using electronic com- research can further explore innovative
munication in helping foster longer-term ways to leverage university information sys-
academic performance (Oreopoulos and tems to facilitate such campaigns, as well as
Petronijevic 2017). The other study evalu- varying the content, timing, and approach
ated a program at Georgia State University of messaging. More extensive application
that leveraged artificial intelligence (AI) assistance is also an important strategy to fol-
technology in developing a texting pro- low, and it would be worth developing and
gram with AdmitHub that sent custom- testing computer programs—perhaps to be
ized messages to students guiding them made available for free on websites—that
through many aspects of the college enroll- help households confidently fill out FAFSAs.
ment process (Page and Gehlbach 2017). Last, while AI advising clearly has limits,
The “augmented intelligence technology” future research can further explore which
TABLE 4C
Technology-Enabled Behavioral Interventions (Post-secondary Access and Success)
Unit of
Barr et al. EdPolicyWorks Text messaging 1. Positive effect Main reported effect sizes Community 2,807 Student
(2016) Working Paper campaign prompting on reduction of (not standardized): college loan students
loan applicants at unsubsidized loan 1. Null on receipt of applicants
a large community borrowing subsidized loan in Baltimore
college to 2. Null impacts on 2. 3.1 percentage point County
make informed and enrollment, credit decrease in share of
active borrowing accumulation, and students who received
decisions GPA an unsubsidized loan and
5.3 percent decrease in
amount of unsubsidized
loans disbursed
3. Null on enrollment,
credit accumulation, and
GPA
Bergman, Journal of Policy -mails and letters to
E Null effects Null Students who 1,042,303 Student
Denning, and Analysis and potential/ prospective/ had applied students
Manoli (2019) Management current college to any public
students on financial Texas college or
aid/incentives university using
the ApplyTexas.
org portal.
(Continued)
965
TABLE 4C
966
Technology-Enabled Behavioral Interventions (Post-secondary Access and Success) (Continued)
Unit of
Bettinger et al. Quarterly FAFSA assistance 1. Positive effect on Main reported effect Individuals in 4,187 Individuals
(2012) Journal of during tax filing FAFSA submissions sizes (not standardized): families with individuals
Economics and ultimately 1. 15.7 percentage point incomes less from the
the likelihood of increase in likelihood of than $45,000 dependent
college attendance, filing FAFSA from being and with a sample;
persistence, and offered help to complete family member 15,874
aid receipt for the form; null impact between the individuals
students in the from information-only ages of 15 from the
combined assistance treatment and 30 who independent
and information 2. 8.1 percentage point did not have sample
treatment increase in college a bachelor’s
2. Positive effect on enrollment among degree in Ohio
likelihood to have dependent students and North
completed 2 years of 3. 10.6 percentage point Carolina
college during the increase in Pell Grant
first 3 years receipt
following the 4. 8 percentage point
experiment increase in persistence
among dependent
students
Bird et al. EdPolicyWorks Nudges for early Positive effect for Main reported effect sizes US high school 454,243 Student
(2017) Working Paper FAFSA filing through treatment arm that (not standardized): seniors who had students
the Common involves concrete 1.1 percentage point registered with
Application planning prompts increase in college the Common
enrollment Application
Castleman, Journal of Providing college Positive effects on Main reported effect sizes Senior students 162 students Student
Arnold, and Research on counseling to low- both the rate and (not standardized): across 7 high
Wartman Educational income students quality of college 14 percentage point schools in
(2012) Effectiveness during the summer enrollment increase in likelihood Providence,

through email, text to enroll immediately Rhode Island
message, and in-person in college and 19
consultation percentage point increase
in likelihood to keep the
postsecondary plans they
developed during senior
year
(Continued)
TABLE 4C
Unit of
Castleman and EdPolicyWorks A text messaging Positive effect on Main reported effect sizes Rising college 1,198 Student
Meyer (2016) Working Paper campaign to provide completion of more (not standardized): 2–2.23 students in West students
lower-income freshman year additional credits over Virginia
college students with credits the year
simplified information,
encouragement, and
access to one-on-one
advising
Castleman and Journal of Text messages to Positive effect on Main reported effect sizes Recent 12,676 Student
Page (2015) Economic reduce summer melt enrollment among (not standardized): high school students
Behavior & students with less 3.0 percentage point graduates in
Organization access to c ollege- increase in 2-year college Dallas, Boston,
planning supports enrollment; Null on Lawrence,
and who were not as enrollment in 4-year Springfield, and
far along with their colleges Philadelphia
college planning at
the completion of
high school; null full
sample impacts
Castleman and Journal of Text message to Positive effects Main reported effect sizes First-time 808 students Student
Page (2016) Human improve FAFSA re- (not standardized): college
Resources filing for sophomore 14 percentage point freshmen in
year increase in likelihood Massachusetts
to remain continuously
enrolled through spring
of sophomore year
among community
college freshman; null
impacts on sophomore
year persistence among
freshmen at 4-year
institutions
(Continued)
967
968
TABLE 4C
Unit of
Castleman and EdPolicyWorks Text messages to Positive effects Main reported effect sizes High school 4,754 Student
Page (2017) Working Paper improve enrollment (although no (not standardized): 3.1 graduates students
tasks additional benefit percentage point increase in Boston,
from including in on-time college Fall River,
parents on nudges) enrollment (although no Lawrence, and
additional benefit from Springfield,
including parents on Massachusetts;
nudges) and Miami,
Florida
Chande et al. Harvard Texting motivational 1. Positive effects Main reported effect sizes Adult learners 1,179 Class
(2015) Business School messages and on reduction of the (not standardized): 36 in England students in
Working Paper organizational proportion of percent decrease in the 152 classes
reminders to students, students that stop proportion of students
with messages drawing attending that stop attending and
on insights from 2. Positive effects on a 7 percent increase in
behavioral economics increasing average average attendance
attendance relative
to the control group.
Darolia and Educational Letter e -mailed to Null effects Null College students 9,802 Student
Harper (2018) Evaluation and students regarding at the University students
Policy Analysis financial aid of Missouri
Hyman (2018) University of Mailing letters with Null impact on Main reported effect sizes Eleventh grade 49,156 Student
Michigan web address to college college enrollment (not standardized): public school students
Education information website for full sample; 1. Null students in
Policy Initiative positive impacts 2. 1.4 percentage point Michigan
Working Paper among poor students increase in the probability
that they will enroll in

college
(Continued)
TABLE 4C
Unit of
Ksoll et al. Center Innovative mobile 1. Positive effect on Main reported effect sizes Adult learners 70 students Student
(2014) for Global phone-based adult students’ basic and (not standardized): in Los Angeles
Development education program broad reading scores 1. 2–4 year increase in
Working Paper (Cell-Ed) 2. Positive effect on reading levels over a f our-
participants’ self- month period
esteem 2. 7 percent increase in
participants’ self-esteem
O’Connell and Journal of Personalized email Positive effects on 0.2 standard deviation First-year 281 students Student
Lang (2018) Research on reminders encouraging exam performance increase on exam students at
Technology in out-of-class study performance a mid-sized
Education university in
the northeast
United States
Oreopoulos Scandinavian 3 -minute video and Positive effects Main reported effect sizes Disadvantaged 1,616 Student
and Dunn Journal of opportunity to use on understanding (not standardized): high school students
(2013) Economics financial aid calculator college-related 40 percent increase in students in
benefit–cost ratio between expected Canada
earnings after college and
after high school
Oreopoulos Journal of Policy Application assistance Positive effect on Main reported effect High schools 11,356 School
and Ford Analysis and incorporated into application rates and sizes (not standardized): in Canada with students at 86
(2019) Management the high school college enrollment 14 percentage point low college schools
curriculum. Program increase in application transition rates
included a website rates and 5.2 percentage
that was designed to point increase in college
provide students a enrollment with virtually
“one-stop-shop” with all of this increase in

directed access to 2-year community college
application websites, programs
informational videos,
tools for identifying
suitable programs
for each student, and
a financial aid and
budget calculator
969
(Continued)
970
TABLE 4C
Unit of
Oreopoulos NBER Working Text-based advising Null effects found Null First-year 4,900 Student
and Paper college students students
Petronijevic in Canada
(2017)
Page, SSRN FAFSA texting Positive effect on Main reported effect sizes High schools 17,000+ Texas: school
Castleman, program enrollment among (not standardized): 5–6 in Texas and Texas Delaware: not
and Meyer students with less percentage point increase Delaware students in 66 randomized—
(2016) access to college- in FAFSA submission and high schools quasi-
planning supports completion; 4 percentage experimental
point increase in timely 4,095
college enrollment Delaware
students in 47
high schools
Smith et al. Journal of Software that sends Positive effect 3.3 percentage point First year 122 students Student
(2018) Economic a “grade nudge,” a on homework increase in grades (with microeconomics
Education personalized message performance evidence suggesting a students at
to each homework larger impact from nudges Washington
assignment regarding early in the semester) State University
the student’s current
grade. The message
explains precisely how
the assignment will
impact the student’s
final grade given their
current standing in the
class.
tasks are adequately handled by AI systems school and learning that are associated with
and which are not. positive academic outcomes and school suc-
cess,” and some education theorists expect
6.4 Social-Psychology Interventions that they will shape educational outcomes
by affecting “the quality, duration, and inten-
6.4.1 Background, Context, and sity with which students engage in critical
Mechanisms of Impact academic behaviors … and deploy learning
strategies” (Snipes et al. 2015) Typically,
Finally, several recent programs have been mindset interventions involve short reading
evaluated that take advantage of ed-tech to and writing exercises that attempt to con-
implement interventions aimed at provid- vince students that an individual’s intelli-
ing students with social psychological sup- gence develops over time and that new skills
port. It is widely agreed among education can be learned with practice (i.e., a “growth
scholars that students’ educational perfor- mindset”), rather than academic perfor-
mance is heavily affected by emotions and mance being an unchanging attribute (“fixed
mentalities, which are in turn tied up in mindset”). Examples of other social-psy-
students’ perceptions of themselves in rela- chology interventions include programs that
tion to those around them (Duckworth et attempt to improve self- or values affirma-
al. 2007, Harackiewicz and Priniski 2018). tion (Brady et al. 2016; Borman, Grigg, and
The interventions discussed in this subsec- Hanselman 2016), reduce stereotype threat
tion center on short exercises administered (Cohen et al. 2006; Good, Aronson, and
to students that are intended to cultivate Inzlicht 2003; Hanselman et al. 2014), high-
mentalities conducive to learning and light relevance of school or academic work to
thereby overcoming such barriers. Like the one’s life goals (Hulleman and Harackiewicz
other behavioral interventions discussed 2009), and reinforce sense of purpose.
in this section, social-psychology interven- Social-psychology interventions can con-
tions provide subtle promotion of ideas that ceivably be delivered at any point during
designers hope will alleviate psychological the educational life course, at least once an
constraints on learning, at least among a sub- individual is old enough to have acquired
set of students susceptible to them. Within the abilities for self-reflection and abstract
our theory-of-change framework, social-psy- thinking needed to make a mindset interven-
chology interventions may help to improve tion’s material meaningful. The interventions
the efficiency of education by cultivating tested so far have tended to cluster around
noncognitive skills like s elf-efficacy and per- the transition to high school and the tran-
sistence. But whereas the other behavioral sition to college. This is perhaps because it
interventions discussed in this subsection is easier to experimentally test samples of
aim to overcome cognitive limitations, such older youth in larger classrooms, or because
as procrastination or inattention, the present it is typically understood that persistence in
subsection’s interventions focus on emotions the face of difficulty becomes increasingly
and beliefs, such as feeling out of place or important in high school and college, when
“not smart.” even students with strong academic and cog-
The class of social-psychology inter- nitive skills almost invariably run into mate-
ventions that has been most prominent in rial that challenges them and must decide
scholarly discussions is that of “mindset how to interpret these difficulties. During
interventions.” Academic mindsets consist of transitional periods, as when entering high
the “attitudes, beliefs, and dispositions about school or college, students may be more
open to outside influences, like the materials are generally small or insignificant, but that
presented in a social-psychology interven- there may be some signs of promise for
tion. By the time an individual has reached certain subgroups, including students with
adulthood, these interventions may be less low socioeconomic status (Sisk et al. 2018).
likely to deliver effects that are as strong, Another m eta-analysis notes that experimen-
since the individuals’ thinking habits are tal evaluations of social-psychology inter-
likely to have crystallized more, and most of ventions in education typically find smaller
the academic work that could have benefited effect sizes than do quasi-experimental eval-
from improved mindset is likely behind the uations (Lazowski and Hulleman 2016).
individual. It is, as of yet, unclear the extent to which
Several small-sample studies suggest that the mixed findings relating to social-psychol-
social-psychology interventions can yield ogy interventions reflect their general ineffi-
significant learning effects, especially rela- cacy at scale versus c ontext-bound potential.
tive to the interventions’ typically low costs Ed-tech offers a range of possibilities for

in time and money (Yeager and Walton testing and potentially improving these
2011). However, other studies show mini- interventions. Up until the last few years, the
mal or null effects, particularly when larger typical model has been for students to com-
sample sizes are used. For instance, while plete program activities in classrooms using
one of the pioneering mindset intervention paper and pencils. Delivering interventions
studies showed an impact of 0.3 GPA points via computers may:
in a study involving 158 seventh graders
allow materials to be delivered to recipi-
from a Northeastern school, an attempt to ents exactly as designed without extensive
replicate the study using the same mindset researcher involvement or facilitator training
materials among 374 seventh graders from … eliminate geographic constraints, opening
11 schools in a single Midwestern school access to students at multiple school sites and
district found an impact of only 0.065 GPA sites far from research centers; and …drasti-
cally reduce logistical burdens, the marginal
points, and then an entirely null result when cost of additional participants, and the costs
they replicated the experiment with 449 stu- of data collection and large-scale evaluation….
dents (Cohen et al. 2006; Borman, Grigg, (Paunesku et al. 2015)
and Hanselman 2016; Yong 2016). An inter-
vention targeting stereotype threat—when By offering to drastically reduce the costs of
individuals are or feel themselves to be at scale-up, delivering social-psychology inter-
risk of conforming to stereotypes about their ventions through computers can enable the
social group—among minority and female wide-ranging and systematic testing of these
students showed no significant effects in a interventions necessary to fully understand
sample of over 1,300 students in three urban their workings, potential benefits, and limita-
high schools (Bancroft, Bratter, and Rowley tions—in addition to reducing the cost side
2017). A self-affirmation exercise, which of the cost–benefit ratio for the treatment
encourages students to identify and reflect itself.
upon core personal values, showed mostly In particular, experimental research on
insignificant effects—and even a reduction ed-tech—although still in its infancy—has

in performance among girls—on a sample already begun to contribute toward schol-
of 2,500 seventh–eighth graders from six arly understanding of education-oriented
middle schools in and around Philadelphia social-psychology interventions in three
(Dee 2015). A recent meta-analysis found broad areas. First, evaluating the deliv-
that average effects of these interventions ery through computers of traditional
social-psychology interventions previously an advertisement—the students who might

tested using paper-and-pencil format allows be most likely to enthusiastically make use of
researchers to evaluate these interventions such an intervention—making it difficult to
on larger scales than would otherwise be assess the external validity of these findings.
possible. The endeavor also holds potential Indeed, an attempt to replicate this online
for generating evidence about the extent to goal-setting exercise by one of this article’s
which these interventions work differently coauthors (Oreopoulos) found precise null

when delivered through computers, as find- effects for a sample of nearly 2,000 fi rst-year
ings accumulate. Second, ed-tech research university students, as well as for subgroups of
offers the potential for studying delivery students more at risk of poor academic perfor-
of social-psychology interventions through mance (Oreopoulos, Patterson, et al. 2018).
media that is not possible in traditional Similarly, a second study (Yeager, Henderson,
paper-and-pencil approaches, like the use et al. 2014) tests a web-based “prosocial,”
of audiovisual content and computer games. “purpose-for-learning” intervention—which
Third, ed-tech can facilitate iterative exper- teaches students to learn with the goal of
imentation and rapid feedback for learn- making a broader impact—on 338 ninth
ing what works best between alternative graders in a relatively affluent and h igh-
intervention variations. Many colleges and performing San Francisco Bay Area high
high schools could test social-psychology school, and finds significant impact on
interventions regularly and converge on STEM GPA (d = 0.21)—but only for stu-
what helps most, for whom, and when. The dents with baseline GPAs below 3.0 out of
next subsection reviews the evidence that 4.0. P ooled-sample impacts are not statis-
has accumulated so far on each of these tically significant. While in some ways, sig-
topics. nificant effects in a well-off school like the
one studied here may represent a strong
test, since more privileged students are
We identified 15 recent experimental more likely to have other forms of social
studies that have tested the extent to which psychological support, guidance from teach-
traditional social-psychology interventions ers and use of other resources in a decent
are effective when computers are used for school environment may be a necessary link
delivery. Of these, two were carried out on enabling students to channel improved moti-
small to medium scales comparable to those vations into higher GPA, making generaliz-
of other studies from this literature that do ability of these findings to struggling schools
not use ed-tech. Findings are similar, and difficult to judge. In fact, poorly performing
the authors of these studies conclude that students in h igh-achieving schools—one way
the interventions appear promising. In one to describe the s ubsample of this study that
of these studies (Morisano et al. 2010), the benefited from the program—may well be
authors test the effects of an online written an especially likely group to obtain positive
2.5-hour goal-setting intervention on a sam- impacts, since they face some challenges
ple of 85 fi rst-year McGill University stu- and have scope for improvement, but also
dents, finding a positive impact of 0.5 SDs have the supports and resources to succeed
on end-of-semester GPA, as well as positive should they choose to increase their efforts.
and significant impacts on full-time enroll- Furthermore, the outcome examined is very
ment and self-reported affect. However, short term: after only one
quarter of the
the study’s sample size is very small and was school year. Few studies, including this one,
selected from volunteers who responded to test for longer-lasting effects.
While educational psychologists have been materials highlighted economic self-interest.

intrigued by social-psychology interventions The study found positive and statistically
since the early 1990s, researchers have only significant—although substantively small—
recently begun testing efforts to scale up in impacts on average grades for core courses,
ways that explicitly account for the changes for the bottom third of students, of 6.4 per-
that are likely to come with the transition centage points for both of the interventions,
from small, controlled settings to real-life although there was no added benefit to
policy situations. Two recent studies pres- receiving both treatments over receiving just
ent stronger evidence on social-psychology one. No significant effects are reported for
interventions than the two just discussed, the full sample.
given their larger and more representative The second large-scale study (Yeager,
samples, rigorous methods, and scaled-up Walton, et al. 2016) tested several variations
implementation structures. Both of these of a mindset intervention given to incoming
studies reinforce the proposition—per- college students in advance of matriculation.
haps more strongly than before, given their The researchers ran three experiments: one
large and diverse sample sizes and rigorous on 584 graduating seniors from four urban
methods—that social-psychology interven- charter schools going on to a variety of two-
tions can, under some circumstances, lead and four-year colleges; one on 7,335 incom-
to meaningful effects, but that these effects ing students at a h
igh-quality public f our-year
tend to be concentrated within subsamples university; and one on 1,592 incoming stu-
and, even then, tend to be quite small (not dents at a private university with competitive
to mention that whether these effects can admissions. Using pooled effects from the
last beyond a single semester or school year different mindset variations, experiment 1
remains an open question). saw an improvement of f ull-time continuous
The first l arge-scale technology-based enrollment in the first year of 9 percentage
mindset intervention study was conducted points (from 32 percent to 41 percent); exper-
in 2015 (Paunesku et al. 2015), testing iment 2 showed significant effects on disad-
the effects of two interventions—one vantaged students’ enrollment (as defined
“growth mindset” intervention, and one by racial minority and/or first-generation
“
sense-of-purpose” intervention—on aca- college attendance status—which accounted
demic performance. The sample included for all participants in experiment 1), but not
1,594 students from 13 different high advantaged ones, leading to a drop in the
schools. The authors purposely included a gap between disadvantaged and advantaged
diverse sample of schools in order to max- students in fi rst-year full-time continuous
imize external validity, despite the accom- enrollment by 40 percent (the rate increased
panying loss in statistical power, choosing from 69 percent to 73 percent in the disad-
schools that varied across geography, socio- vantaged group and stayed at 79 percent in
economic characteristics, and public versus the advantaged group). Experiment 3 found
charter versus private status. The interven- a positive impact on GPA of 0.25 standard
tion consisted of two 45-minute computer deviations for disadvantaged students. While
sessions—the first a growth mindset module the interventions overall seem to have left an
or control, and the second a s ense-of-purpose impression on the students, only the third
module or control—separated by roughly experiment reported effects on learning
two weeks. The sense-of-purpose interven- outcomes, and here significance was again
tion sought to cultivate a sense of prosocial restricted to a subsample. Furthermore,
purpose for studying hard, while the control the study used a factorial design to test the
separate and joint impacts of two different ed-tech may also add to social-psychology
social-psychology interventions, mindset and interventions by facilitating intervention
social belonging, and only the latter leads models that use technology-enabled multime-
to any significant impact when separated dia going beyond simple text. Although litera-
out. Given the large-scale and diverse sam- ture testing such studies is still in its infancy,
ple of the study, the reported impacts raise three recent studies provide some initial
serious questions about the generalizability insight into this area. One study (Walton et
of ostensibly positive effects from mindset al. 2015) tested an intervention on a sample
interventions found in small, highly con- of 228 first-year students at the University of
trolled samples, while lending new evidence Waterloo that was delivered in a face-to-face
to the potential promise of social-belonging classroom setting, but involved participants
interventions. listening to audio testimonies from fellow
Two other experimental studies show that students to reinforce mindset messages. The
social-psychology interventions can exert experiment found large positive effects of
psychological effects, but without necessarily more than a standard deviation on semes-
improving final outcomes. One of these stud- ter GPA for women (no effects for men)—
ies (Unkovic, Sen, and Quinn 2016) found that although only within male-dominated majors
encouraging e-mails can raise the probability (i.e., >80 percent male). This large effect size
that graduate students will apply to a com- suggests the potential value of future tests of
petitive conference, with stronger effects for related intervention models that use mul-
women than for men in a gender-imbalanced timedia rather than traditional reading and
field. However, these effects did not trans- writing, although the present study provides
late into higher acceptance rates, and women limited evidence on its own given the narrow
in the control group ended up with higher sample size, short time frame, and restriction
acceptance rates than those in the treatment to male-dominated majors.
groups. In this case, the intervention proba- Two other education-focused social-psy-
bly did not cause harm to the female gradu- chology studies (Yeager, Trzesniewski, and
ate students with unsuccessful applications, Dweck 2013; Yeager, Johnson, et al. 2014)
and taking academic risks is generally con- involve a computer game known as Cyberball
sidered a positive step. But the intervention that engages the participant in an electronic
would likely have needed to blend in addi- game of catch with two other individuals that
tional resources in order to be effective in are allegedly other students from the same
increasing final outcomes, a lesson that may school. The game can create a sense of exclu-
prove to be more general. A small-sample sion if the other “players” begin by aiming
study (Forsyth et al. 2007) conducted within the ball at the participant but then begin
a single undergraduate psychology class even ignoring the participant—this then allows
found that an intervention aimed at bolster- researchers to test variation in students’
ing self-esteem significantly decreased final reactions to exclusion. The game functions
exam scores for students already receiving as a measurement device within these two
Ds and Fs, suggesting that overconfidence studies, rather than part of the intervention
could be an unintended consequence of itself, but the studies nonetheless help to
self-esteem interventions that lack additional highlight the potential role that computer
components. games can play in testing social-psychology
In addition to more efficiently delivering interventions, and one can imagine potential
interventions that are similar or identical in applications through which games become
content to paper-and-pencil interventions, part of the intervention.
Finally, one study (Yeager, Romero, et al. of social-psychology interventions. This

2016) shows the potential utility of ed-tech does not mean that ed-tech can, on its own,
as a way to iteratively improve the design of make these interventions effective or worth-
mindset interventions. Yeager and c olleagues while. The studies by Paunesku et al. (2015)
take the growth mindset intervention used and Yeager, Romero, et al. (2016) both con-
by Paunesku et al. (2015) and use ICT in an tained large and diverse samples and used
attempt to improve the intervention using rigorous methods—if there were large and
techniques of “design thinking.” In particu- wide-ranging effects of the social-psychol-

lar, they modified the intervention content ogy interventions on academic outcomes,
using qualitative p retesting in combination these studies would likely have caught them.
with quantitative A/B testing using Amazon Instead, the studies reinforce a picture in
Mechanical Turk with about 3,000 partici- which social-psychology interventions can
pants. The improvement process led to sev- have significant and meaningful effects rela-
eral changes, including the incorporation of tive to their low costs, but that these effects
quotes from celebrities and the use of bullet tend to be small, and are statistically signifi-
points instead of paragraphs. The research- cant only for subsamples.
ers then tested the revised intervention in Future ed-tech research can advance
two studies, one with a sample of 7,501 ninth scholarly understanding of social-psychol-
graders across 69 high schools in the United ogy interventions in several directions. First,
States and Canada (looking at proximate out- future studies should attempt to refine
comes, i.e., the impact of the intervention on understanding of the specific student types
growth mindset), and one with a sample of that are most likely to significantly gain from
3,676 ninth graders representing 95 percent these interventions. Enough studies have
of the students at 10 schools. Both studies been done to demonstrate that there is no
found positive and significant effects when meaningful answer as to whether social-psy-
comparing the revised version of the inter- chology interventions “work” or not—they
vention to the original (there was no pure work differently and to different extents
control group entirely without a mindset among different populations (and, presum-
intervention, and thus the study says nothing ably, depending on specific design features).
about the impact of the mindset intervention Whether researchers and practitioners can
itself). Effect sizes on the main outcome of efficiently identify the conditions under
interest—an index for challenge-seeking which these interventions are likely to work
behavior—were small, typically below 0.10 can make all the difference as to whether
standard deviations, although larger for these interventions are cost-effective relative
lower-performing students than for aver-
to the next best option.
age or h igher-performing ones. But these Findings from studies so far have generated
impacts are still significant considering that hints in this direction. For instance, effect
the control group also received a very similar sizes tend to be larger for those who start out
mindset intervention. further behind in terms of academic perfor-
mance and/or s ocial-psychological attitudes.
However, the evidence they provide is far
Given its potential to reduce costs, incor- from sufficient in this regard because these
porate multimedia, and identify treatment studies were typically designed to measure
and population characteristics that yield full-sample, rather than subgroup, effects.
the largest effects, ed-tech represents a Even where studies are adequately powered
promising route for the study and honing to test heterogeneity, contemporary best
practices in experimental research demand urposes. Psychologists have emphasized the

p
detailed preliminary plans for heterogeneity importance of participants being engaged,
tests and, wherever possible, documented which is why they are often asked to write
pre-analysis plans. A team of researchers is essays or letters to hypothetical future stu-
currently leading exactly such an endeav- dents rather than simply absorbing the infor-
or—a study aiming specifically to identify mation. But games may offer more active
heterogeneity with a detailed analysis plan on forms of engagement than writing prompts,
a nationally representative sample of 20,000 particularly for younger children.
ninth graders.17 These researchers are con- Fourth, while several recent collaborators
sidering variation not only in student char- have already spearheaded the use of feedback
acteristics, but also in school and classroom techniques in mindset interventions, more
conditions and resources. Future research work in this direction is needed, given that
should pursue a similar agenda in a college this is the only study that has incorporated
setting and should involve new research ICT-based feedback and that the A/B testing
teams (since researchers from Stanford used adults on Mechanical Turk rather than
University and collaborators have conducted the population of interest itself. This means
nearly all of the h
igh-quality mindset experi- that the quality of customization may have
ments in the e d-tech space). been limited, and thus that the effects may
Second, more studies are needed that represent a lower range. Furthermore, qual-
directly compare different social-psychology itative piloting—which should be a require-
intervention models—for instance, mindset ment for almost any program evaluation
versus sense-of-purpose versus stereotype regardless—was used to alter the treatment
threat reduction—within the same experi- relative to control in addition to ICT-based
mental context, so that their significance and feedback, making it difficult to disentan-
effect sizes may be compared to one another. gle the impacts. Beyond developing inter-
Different social-psychology interventions ventions that are more effective for larger
may target different problems and hold dif- groups of students, feedback could be used
ferent theories of change—there is no rea- to customize interventions for students with
son that the same findings should hold for different personality types, or socioeconomic
different intervention types. Furthermore, or identity backgrounds, leading to improved
alternative models may work differently on targeting.
different populations. Finally, while the focus so far on the
Third, the literature on ed tech-enabled transitions to high school and college is
multimedia social-psychology interventions understandable, future research should test
has, perhaps surprisingly, barely begun. The social-psychology interventions delivered
Walton et al. (2015) study described above through e d-tech within different age ranges.
uses only audio technology; perhaps vary- While it may be unlikely that young children
ing blends of audio, video, and text could would be affected by these interventions,
produce emotional responses capable of middle school children might benefit from
engendering lasting change in students. them, and such early interventions could
Furthermore, games have thus far been yield especially large effects to the extent
used for testing rather than intervention that these effects are recursive, as discussed
above. Furthermore, younger children might
17 National Study of Learning Mindsets. http://
be more likely to respond well to engaging
mindsetscholarsnetwork.org/about-the-network/current- multimedia content on a computer versus
initatives/national-mindset-study/. detailed articles and writing prompts. And,
TABLE 4D
978
Technology-Enabled Behavioral Interventions (Social Psychology)
Unit of
Forsyth et al. Journal of Social Self-esteem bolstering 1. Negative results Main reported effect sizes US college 90 students Student
(2007) and Clinical intervention on self-esteem (not standardized): the D students
Psychology for the D and F and F students got worse
students as a result of self-esteem
2. Null impacts for bolstering and students
students in the other in the other conditions
conditions. did not change (exact
magnitude not reported)
Good, Applied -mail mentorship by
E 1. Positive effect on 1. Math scores for female Seventh grade 138 students Student
Aronson, Developmental college students who math standardized students: 1.13 standard students in
and Inzlicht Psychology encouraged middle test scores for deviations (incremental Texas
(2003) school students to females condition); 1.50 standard
view intelligence 2. Positive effect on deviations (attribution
as malleable or to reading standardized condition) 1.30 standard
attribute academic test scores deviations (combined
difficulties in the condition) 2. Reading
seventh grade to scores: 0.52 standard
the novelty of the deviations (incremental
educational setting condition) and 0.71
standard deviations
(attribution condition)
Harackiewicz Psychological 3 -part intervention (2 Positive effect Main reported effect sizes High school 188 students Students and
et al. (2012) Science brochures mailed to on taking nearly (not standardized): nearly students in and their parents
parents and a website) 1 semester more an extra semester of math Wisconsin parents
highlighting the of science and and science courses in the
usefulness of STEM mathematics in the last 2 years of high school
courses last 2 years of high

school
Morisano et al. Journal of Goal-setting program Positive effects 1. 0.50 standard deviation College students 85 students Student
(2010) Applied on academic increase in GPA in Canada
Psychology performance 2. 0.46 decrease in
compared with the negative affect scores
control group after 3. Null for enthusiasm
4-month period scores
(Continued)
TABLE 4D
Technology-Enabled Behavioral Interventions (Social Psychology) (Continued)
Unit of
Oreopoulos, NBER Working Online planning Null impacts on Null 1. Selective 9,000+ Student
Patterson, et Paper exercise with course grades, credit urban college students
al. (2018) information and accumulation, or 2. Non-selective
guidance to create retention urban college
a weekly schedule 3. Online
containing sufficient college
study time and other
obligations
Oreopoulos, NBER Working Choose-Your-Own- Null impacts on Null First-year 3,395 Student
Petronijevic, et Paper Challenge online course grades, university students
al. (2018) modules designed GPA, or credit students at
to teach students accumulation a Canadian
effective learning university
behaviors and adaptive
perspectives
Paunesku et al. Psychological Growth mindset and Positive effect Main reported effect US students 1,594 Student
(2015) Science sense-of-purpose on GPA in core sizes (not standardized): in 13 students
interventions academic courses 6.4 percentage point geographically
and increased increase in satisfactory diverse high
the rate at which performance in core schools
students performed courses
satisfactorily in
core courses among
students at risk of
dropping out of high

school
Rege et al. American A program teaching Positive program 0.23 decrease in number US ninth grade 14,866 Student
(forthcoming) Psychologist a growth mindset of effects on challenge- of “easy” problems taken. students students
intelligence seeking behavior
with modest
heterogeneity
979
(Continued)
980
TABLE 4D
Unit of
Unkovic, Sen, PLOS One Personalized emails Positive effect Main reported effect sizes US graduate 3,945 Student
and Quinn encouraging graduate among application (not standardized): 1. 2.7 students students
(2016) students to apply for a rates; null results on percentage point increase
conference acceptance rates to in applications (larger
conference effect for women) 2. Null
for conference acceptance
Walton et al. Journal of 1. Social-belonging Positive effects 1.04 standard deviation First-year 228 students Student
(2015) Educational intervention to protect on women’s increase in women’s GPA; students at the
Psychology students’ sense of s elf- school-reported null impacts for men University of
belonging engineering GPA Waterloo
2. Affirmation-training
intervention to help
students manages
stress related with
social marginalization
Yeager et al. Child 6-session intervention Positive effects on Main reported effect sizes Ninth and tenth 230 students Student
(2013) Development that taught an reducing aggressive (not standardized): Nearly grade students
incremental theory (a behavior and 40 percent reduction in in California
belief in the potential increasing prosocial aggressive retaliation after
for personal change) behavior among the a controlled provocation
through Cyberball incremental theory (an experience of
electronic game. group exclusion), 300 percent
increase in prosocial
behavior
(Continued)
TABLE 4D
Unit of
Yeager, Journal of Promoting a prosocial, Positive effects Main reported effect sizes US high school Study 1: Student
Henderson, et Personality self-transcendent on persistence (not standardized): double and college 1,364
al. (2014) and Social purpose on a boring task the amount of time students students
Psychology and persistence in students spent on tedious Study 2: 338
college. exam review questions; students
35 percent increase in the Study 3: 89
number of boring math students
problems Study 4:
429 college
students
Yeager, Journal of A malleable Positive effects Main reported effect Ninth grade 158 students Student
Johnson, et al. Personality (incremental) theory on reactions to sizes (not standardized): students in
(2014) and Social of personality—the an immediate the incremental theory California
Psychology belief that people can experience of group showed less
change. social adversity and negative reactions to an
better academic immediate experience
performance; lower of social adversity and,
overall stress and 8 months later, reported
physical illness. lower overall stress and
physical illness. They also
achieved better academic
performance over the
year.
(Continued)
981
982
TABLE 4D
Unit of
Yeager, Journal of Growth mindset Positive effect on Main reported effect US ninth grade Study 1: Student
Romero, et al. Educational interventions during ninth grade core- sizes (not standardized): students 7,501
(2016) Psychology the transition to high course GPA and the intervention was students
school: qualitative reduced D/F GPAs an improvement over Study 2:
inquiry and rapid, for lower achieving previous versions in 3,676
iterative, randomized students when terms of short-term proxy students
“A/B” experiments delivered via the outcomes and it improved
were conducted to Internet ninth grade core-course
inform intervention GPA and reduced D/F
revisions for this GPAs for lower achieving
population. students when delivered
via the internet
Yeager, Proceedings of “Lay theory” Positive effect on Main reported effect US high school Study 1: 584 Student
Walton, et al. the National intervention that full-time enrollment sizes (not standardized): and college students
(2016) Academy of explains the meaning rates and grade 9 percentage point students Study
Sciences of commonplace point averages increase in full-time 2: 7,335
difficulties before and reduced the enrollment (lay theory students
college matriculation overrepresentation intervention—experiment Study
of socially 1); null (growth mindset 3: 1,592
disadvantaged intervention—experiment students
students among the 1); 4 percentage point
bottom 20 percent increase in full-time
of class rank enrollment among
disadvantaged students
(lay theory intervention—
experiment 2); null for

advantaged students (lay
theory intervention—
experiment 2); 0.09
increase in GPA for
disadvantaged students for
disadvantaged students
(experiment 3)
although mindsets may crystallize later in for ed-tech and potential reasons for why
life, interventions still could have effects for ed-tech in some circumstances successfully
nontraditional students considering pursuing contributes to the development of cognitive
education or training later in life. and noncognitive skills.
We found that simply providing students
with access to technology yields largely
7. Conclusion mixed results. At the K–12 level, much of
the experimental evidence suggests that
Technology has transformed large seg- giving a child a computer may have limited
ments of society in ways that were once impacts on learning outcomes, but generally
considered unimaginable. Education is no improves computer proficiency and other
exception. Around the world, there is tre- cognitive outcomes. While access to technol-
mendous interest in leveraging technology to ogy likely improves the learning environment
transform how students learn. In the coming by expanding opportunities for learning, by
years, new uses of e d-tech will continue to and large, increased access alone does not
flood the market, providing students, par- seem to advance cognitive skill formation.
ents, and educators with a seemingly limitless One bright spot that warrants further study
array of options. And experimental literatures is the provision of technology to students at
are beginning to emerge in new domains, the postsecondary level, an area where the
including in-class technology like iClickers limited RCT evidence has reported positive
(Lantz and Stawiski 2014) and adult educa- impacts. This may be due to the increased
tion offered through text messages and other necessity of technology—and perhaps tech-
new platforms (Ksoll et al. 2014). nology’s more critical role in skill forma-
Amid the buzz and sizeable investment in tion—during later stages of the education
ed-tech, we aim to step back and take stock life cycle.
of what we currently know from the exper- From our review, C AL and technology-
imental evidence in this nascent field and enabled behavioral interventions emerge as
shed light into how technology may impact two areas that show considerable promise.
the education production function. This CAL has the potential to advance cognitive
review hopes to advance the knowledge base skill formation by mitigating standard teacher
by identifying and discussing the most prom- and classroom constraints, thereby increas-
ising uses of e d-tech to date that have been ing the efficiency of investments. Especially
explored with experimental research and when equipped with a feature of personaliza-
highlighting areas that merit further exploration, CAL has been shown to be quite effec-
tion. We contextualize our discussion within tive in helping students, particularly with
a framework for how education technology math. Two interventions in the United States
may or may not address traditional educa- stand out as being particularly promising—a
tion constraints and increase the efficiency fairly low-intensity online program that pro-
of investments in cognitive and n oncognitive vides students with immediate feedback on
skill formation. Our review categorizes the math homework was found to have an effect
existing literature into four categories: (i) size of 0.18 standard deviations, and a more
access to technology, (ii) CAL, (iii) online intensive software-based math curriculum
courses, and (iv) technology-enabled behav- intervention improved seventh and eighth
ioral interventions. Taken together, these grade math scores by a remarkable 0.63
studies and their potential mechanisms and 0.56 standard deviations. These results
of impact suggest a few areas of promise mirror those from promising interventions
examined in the developing country liter- technology-enabled models that can further
ature, such as the adaptive Indian learning facilitate the development of noncognitive
software Mindspark, which was found to skills in a cost-effective, scalable manner.
have large, positive impacts on math and Though online learning courses have
Hindi. The consistent results of personal- exploded in popularity over the last decade,
ized software across different contexts sug- there continues to be limited rigorous
gests that this feature may be an effective research on its effectiveness. From our
mechanism that helps to overcome common review, we found that online courses, in
educational challenges in classrooms with some cases, do increase access to education,
heterogeneous learning levels. However, it particularly when easy alternates are not
should be noted that quality of implemen- available. Yet relative to courses with some
tation is an important feature of CAL, and degree of face-to-face teaching, students
the degree to which a software successfully taking online-only courses may experience
helps teachers to engage students in dedi- negative learning outcomes. This may be
cated time spent on academic learning that due to the difficulty of online courses to rep-
appropriately matches student’s actual learn- licate the pedagogical strengths of in-person
ing levels may be an important determi- instruction, including noncognitive char-
nant of CAL software’s ability to drastically acteristics such as the rapport between a
improve student learning. Far more research teacher and student. On the other hand, the
is needed to help us isolate the mechanisms effects of blended learning are generally o n
for when and why certain CAL programs par with those of fully in-person courses.
improve cognitive skill formation and where This suggests that the appropriate combina-
these findings may be generalizable. tion of online and in-person learning may be
Like with CAL, evaluations of behavioral cost effective, but further research on cost
interventions generally find positive effects effectiveness of online learning are needed
across all stages of the education life cycle, to better understand its potential.
although they are generally smaller than Nevertheless, we also note the poten-
those found with the most effective CAL tial unintended consequences of ed-tech,
models. Given that t echnology-enabled and encourage more work to identify these
behavioral interventions generally target mechanisms for both failure and success
noncognitive skill formation, it is perhaps
with more clarity through further research.
unsurprising that the positive impacts found The evidence on access to technology and
in this category were, for the most part, online learning seems to caution against
smaller in magnitude than CAL models merely providing technological devices or
that successfully augmented the develop- transferring content to online platforms
ment of cognitive skills that may be crucial for technology’s sake, which suggests that
to understanding specific concepts. At the merely improving the learning environ-
same time, technology-enabled behavioral ment with technology is insufficient if the
interventions, such as large-scale text mes- aim is to bolster learning more broadly. On
sage campaigns, are often incredibly cheap the other hand, a body of positive evidence
to carry out and hold great promise as a exists for using technology to overcom-
cost-effective solution to many challenges
ing binding constraints to skill formation,
associated with behavioral barriers in educa- whether that be through n oncognitive or
tion. Given the promising impacts reported cognitive processes. However, it should be
across all stages of the education life cycle, noted that technology without a mechanism
it would be valuable to build on and test for facilitating skill formation may serve as
a distraction with negative consequences. i mplementation models. Open questions

There is empirical evidence that concerns also remain regarding the underlying mecha-
about “brain drain”—the hypothesis that nisms of effective CAL programs, specifically
the mere presence of a phone may undercut how the software interacts with teachers and
cognitive performance—may be substanti- current curriculum. Additionally, it is worth
ated. For example, one experimental study noting that many CAL interventions that are
found that even when people are success- currently being used by schools and families
ful at not checking their phones, the mere have not been rigorously tested, and there-
presence of a smartphone reduces cognitive fore the impacts of such CAL products that
capacity, and that these costs are highest for do not retain the characteristics we identify
those with the greatest smartphone depen- as promising mechanisms of impact—per-
dence (Ward et al. 2017). sonalization, adaptivity, or fast feedback of
Given that our review specifically focuses data—remain unknown. We caution policy
on those ed-tech products and strategies makers and educational researchers to not
that have been evaluated using experimental take labels of CAL software at face value,
methodologies, it is important to note that but to discern the potential mechanisms of
there are many popular ed-tech products impact before judging the promise of a par-
that have not been evaluated using experi- ticular software. For example, to what extent
mental methods to date, and that the ones does a software that claims personalization
that have been tested may be limited in their truly adapt to match lessons to a student’s
generalizability given recruitment and study learning level without requiring the teacher
sample characteristics. This, along with the to recognize and redirect struggling stu-
limited scope of experimental research, even dents? Additionally, as mentioned previously,
on products and strategies that show prom- the personalized and adaptive component of
ise, suggests that there is still a substantial CAL software seems to be more consistently
need for more rigorous research on e d-tech. effective for math, yet a handful of studies
We recommend the following areas for show that CAL can be effective for language
future experimental research in ed-tech. as well. More research will be needed to test
First, replications of interventions mechanisms and impacts of newly emerging
showing promise, such as CAL and CAL products of varying subjects.
technology-enabled behavioral nudges, are Given the promising evidence on
needed to better understand both the gen- technology-enabled behavioral interventions,
eralizability and the credibility of the current researchers should prioritize understanding
findings. We recognize that the large effect when these nudges most effectively support
sizes in some of these studies may be cause noncognitive skill formation by varying the
for skepticism, but the fact that such effects timing and content of messages and testing
emerge in more than one study by different how these models interact with other edu-
authors suggests that these models may truly cational supports. New experimental stud-
be effective and worth exploring in greater ies should test natural extensions of existing
depth. Such results warrant replication to behavioral models, for example messaging
see if the same effect is found in different campaigns with a high degree of personaliza-
contexts and with different populations of tion or parent-information flow systems that
students, and over longer periods of time. draw on predictive analytics to trigger timely
For C AL, additional research is alerts to parents based on students’ perfor-
needed to understand to what extent the mance in school. As artificial intelligence
observed impacts are related to specific enters the realm of education technology,
more research is needed to understand the that measure longer term outcomes, are also
extent to which AI systems can substitute for needed. We need better theories to model
human guidance in navigating complex pro- how and when technology is likely to help
cesses, such as helping students fill out the processes of skill formation so that we can
FAFSA or providing parents with actionable design studies that specifically test these
guidance on how to help a student who is at hypotheses. We hope that the discussion in
risk of dropping out of school. Meanwhile, this paper will provide some roadmaps to
subsequent studies may need to evaluate testing potential mechanisms of impact so
different methods of delivery of messages that findings can be more generalizable, and
such as social media. With many parents, that designing rigorous studies that measure
teachers, and students being inundated with long-term outcomes is more feasible. We rec-
text messages on a daily basis, engagement ognize that there is a shortage of RCTs and
with these messages may decrease with time. encourage more research that follows sam-
Taken together, the experimental research ples over time to determine whether effects
on technology-enabled behavioral inter- fade or sustain in the long term. Given the
ventions is quite promising, and additional framework described in this paper, there is
research should focus on building upon this also a need to better understand how e d-tech
already exciting body of evidence. impacts children of younger ages. The role
More work to understand effects of online of technology to facilitate skill formation at
learning is also needed. We recognize that these younger ages is likely to be different,
the rigorous evidence is sparse, but given its so as more and more technologies are devel-
growing popularity, this is all the more reason oped for children of all ages, these potential
to recognize what studies do exist and pursue mechanisms also need to be tested.
further work. Future research can help us Finally, more rigorous evaluations of pop-
better understand costs and c ost effectiveness ular ed-tech products that have not yet been
of online learning, and to test different types evaluated experimentally. Evaluations on
of online and blended courses relative to products such as flipped classrooms, smart
face-to-face. In particular, there is a need for boards, or even virtual reality technologies to
studies disentangling why online learning on explore other parts of the world or to foster
its own does worse than f ace-to-face in some empathy are also needed. We recognize that
circumstances. Is it the loss of interaction some types of products are less amenable to
with instructors and peers, or is it the lack of an experimental design than others, but this
a structured environment for time manage- challenge in itself should be a part of the con-
ment and study skills, or both? Better under- versation about which e d-tech products and
standing these issues will provide information strategies are working and the extent to which
on whether it is possible to make online we have certainty about their impacts. Other
learning work better if a blended component popular products, such as Khan Academy,
is not possible, and if so, how. Additionally, as have begun engaging in rigorous evaluations
the online learning field is constantly evolv- over the last few years, but studies are not
ing, new research is needed to understand yet complete or results are not yet published.
how new models—such as MicroMasters Researchers should keep their eye on this
programs and nanocredentials—may impact emerging work and continue to build on this.
or democratize learning. The ed-tech field is rapidly changing,
Fourth, more experiments that identify and innovative tools and programs are
the key mechanisms of impact at different frequently considered out-of-date after only
stages in the process of skill formation, and several years. When faced with purchasing
decisions, education administrators often as-lower-income-americans-make-gains-in-tech-

demand research that is timely, relevant, adoption/.
and actionable. The direction and form of Angrist, Joshua D., and Jörn-Steffen Pischke. 2008.
Mostly Harmless Econometrics: An Empiricist’s
the research may need to change to inte- Companion. Princeton: Princeton University Press.
grate more seamlessly into decision making. ASSISTments. 2016. Worcester Polytechnic Institute.
New tools have emerged to address some of https://www.assistments.org/.
these challenges, including Mathematica’s Bai, Yu, Di Mo, Linxiu Zhang, Matthew Boswell, and
Ed-Tech Rapid-Cycle Evaluation Coach and Scott Rozelle. 2016. “The Impact of Integrating ICT
with Teaching: Evidence from a Randomized Con-
the EduStar RCT platform. While r apid-cycle trolled Trial in Rural Schools in China.” Computers
product testing is of course valuable, more & Education 96: 1–14.
research is needed to evaluate how under- Balu, Rekha, Kristin Porter, and Brad Gunton. 2016.
lying mechanisms—rather than a specific Can Informing Parents Help High School Students
product—can advance learning. In the end, it Show up for School? Results from a Partnership
between New Visions for Public Schools and MDRC.
should not be about the most popular product Oakland: MDRC.
or even necessarily the technology itself, but Bancroft Amanda, Jenifer Bratter, and Kristie Row-
about the best way to help students of all ages ley. 2017. “Affirmation Effects on Math Scores: The
and levels learn. Importance of High School Track.” Social Science
Research 64: 319–33.
References Banerjee, Abhijit, et al. 2016. “Mainstreaming an
Effective Intervention: Evidence from Random-
Advanced Micro Devices (AMD). 2014. “Laptops ized Evaluations of ‘Teaching at the Right Level’ In
Move to the Head of the Class among College Stu- India.” NBER Working Paper 22746.
dents, According to AMD Back-to-School Survey,” Banerjee, Abhijit V., Shawn Cole, Esther Duflo, and
July 10. https://www.amd.com/en/press-releases/ Leigh Linden. 2007. “Remedying Education: Evi-
laptops-move-2014jul10. dence from Two Randomized Experiments in India.”
Akerlof, George A., and Rachel E. Kranton. 2002. Quarterly Journal of Economics 122 (3): 1235–64.
“Identity and Schooling: Some Lessons for the Eco- Banerjee, Abhijit V., and Esther Duflo. 2014. “(Dis)
nomics of Education.” Journal of Economic Litera- Organization and Success in an Economics MOOC.”
ture 40 (4): 1167–1201. American Economic Review 104 (5): 514–18.
Allen, I. Elaine, and Jeff Seaman. 2013. “Changing Banerjee, Abhijit, and Esther Duflo. 2016. “Struc-
Course: Ten Years of Tracking Online Education in tured Study Time, Self-Efficacy, and Tutoring.”
the United States.” https://www.onlinelearningsur- Unpublished.
vey.com/reports/changingcourse.pdf. Barr, Andrew, Kelli Bird, and Benjamin L. Castleman.
Almlund, Mathilde, Angela Lee Duckworth, James 2016. “Prompting Active Choice among High-Risk
J. Heckman, and Tim D. Kautz. 2011. “Personality Borrowers: Evidence from a Student Loan Counsel-
Psychology and Economics.” In Handbook of the ing Experiment.” EdPolicyWorks Working Paper 41.
Economics of Education. Vol. 4, edited by Eric A. Barrera-Osorio, Felipe, and Leigh L. Linden. 2009.
Hanushek, Stephen Machin, and Ludger Woessman, “The Use and Misuse of Computers in Education:
1–181. Amsterdam: North-Holland. Evidence from a Randomized Experiment in Colom-
Alpert, William T., Kenneth A. Couch, and Oskar bia.” World Bank Policy Research Working Paper
R. Harmon. 2016. “A Randomized Assessment of 4836.
Online Learning.” American Economic Review 106 Barrow, Lisa, Lisa Markman, and Cecilia Elena Rouse.
(5): 378–82. 2009. “Technology’s Edge: The Educational Benefits
Anderson, Monica. 2015. “The Demographics of of Computer-Aided Instruction.” American Eco-
Device Ownership.” Pew Research Center, Octo- nomic Journal: Economic Policy 1 (1): 52–74.
ber 29. http://www.pewinternet.org/2015/10/29/ BBC. 2013. “India Uttar Pradesh State Gives Away
the-demographics-of-device-ownership/. Free Laptops to Students.” March 11. http://www.
Anderson, Monica, and Madhumitha Kumar. 2017. bbc.com/news/world-asia-india-21738237.
“Digital Divide Persists Even as Lower-Income Beal, Carole, Shandy Hauk, Steven A. S chneider,
Americans Make Gains in Tech Adoption.” Fact Weiling Li, and Christopher Harrison. 2013.
Tank, March 22. http://www.pewresearch.org/ “Randomized Controlled Trial (RCT) Eval-
fact-tank/2017/03/22/digital-divide-persists-even- uation of a Tutoring System for Algebra
R eadiness.”  https://pdfs.semanticscholar.org/0 Application Assistance and Information in College

742/747cbc378e2ac1e949c8beb0fd1be4adb9f8. Decisions: Results from the H&R Block FAFSA
pdf?_ga=2.37305632.1017916876.1565637357- Experiment.” Quarterly Journal of Economics 127
1422984928.1537456066. (3): 1205–42.
Benton Institute for Broadband & Society. 2013. Beuermann, Diether W., Julian Cristia, Santiago
“EducationSuperHighway 2018 State of the States Cueto, Ofer Malamud, and Yyannu Cruz-Aguayo.
Report.” Unpublished. 2015. “One Laptop per Child at Home: Short-Term
Bergman, Peter. 2015. “Parent-Child Information Fric- Impacts from a Randomized Experiment in Peru.”
tions and Human Capital Investment: Evidence from American Economic Journal: Applied Economics 7
a Field Experiment.” CESifo Working Paper 5391. (2): 53–80.
Bergman, Peter. 2016. “Technology Adoption in Edu- Bird, Kelli A., Benjamin L. Castleman, Joshua Good-
cation: Usage, Spillovers and Student Achievement.” man, and Cait Lamberton. 2017. “Nudging at a
CESifo Working Paper 6101. National Scale: Experimental Evidence from a
Bergman, Peter, and Eric Chan. 2017. “Leveraging FAFSA Completion Campaign.” EdPolicyWorks
Technology to Engage Parents at Scale: Evidence Working Paper 54.
from a Randomized Controlled Trial.” CESifo Work- Borman, Geoffrey D., James G. Benson, and Laura
ing Paper 6493. Overman. 2009. “A Randomized Field Trial of the
Bergman, Peter, Jeffrey T. Denning, and Dayanand Fast ForWord Language Computer-Based Training
Manoli. 2019. “Is Information Enough? The Effect Program.” Educational Evaluation and Policy Analy-
of Information about Education Tax Benefits on sis 31 (1): 82–106.
Student Outcomes.” Journal of Policy Analysis and Borman, Geoffrey D., Jeffrey Grigg, and Paul Hansel-
Management 38 (3): 706–31. man. 2016. “An Effort to Close Achievement Gaps at
Bergman, Peter, Chana Edmond-Verley, and Nicole Scale through Self-Affirmation.” Educational Evalu-
Notario-Risk. 2018. “Parent Skills and Informa- ation and Policy Analysis 38 (1): 21–42.
tion Asymmetries: Experimental Evidence from Bowen, William G., Matthew M. Chingos, Kelly A.
Home Visits and Text Messages in Middle and Lack, and Thomas I. Nygren. 2014. “Interactive
High Schools.” Economics of Education Review 66: Learning Online at Public Universities: Evidence
92–103. from a Six-Campus Randomized Trial.” Journal of
Bergman, Peter, and Matthew J. Hill. 2018. “The Policy Analysis and Management 33 (1): 94–111.
Effects of Making Performance Information Public: Bowles, Nellie. 2019. “Silicon Valley Came to Kansas
Regression Discontinuity Evidence from Los Ange- Schools. That Started a Rebellion.” New York Times,
les Teachers.” Economics of Education Review 66: April  21.  https://www.nytimes.com/2019/04/21/
104–13. technology/silicon-valley-kansas-schools.html.
Bergman, Peter, and Todd Rogers. 2016. “Parent Brady, Shannon T., Stephanie L. Reeves, Julio Gar-
Adoption of School Communications Technology: A cia, Valerie Purdie-Vaughns, Jonathan E. Cook, and
12-School Experiment of Default Enrollment Poli- Suzanne Taborsky-Barba. 2016. “The Psychology of
cies.” Paper presented at the Society for Research on the Affirmed Learner: Spontaneous Self-Affirmation
Educational Effectiveness Spring 2016 Conference, in the Face of Stress.” Journal of Educational Psy-
Washington, DC, March 4. chology 108 (3): 353–73.
Berk, Richard, Geoffrey Barnes, Lindsay Alhman, and Brotman, Stuart N. 2016. “The Real Digital
Ellen Kurtz. 2010. “When Second Best Is Good Divide in Educational Technology.” Brook-
Enough: A Comparison between a True Experiment ings TechTank, January 28. https://www.
and a Regression Discontinuity Quasi-Experiment.” brookings.edu/blog/techtank/2016/01/28/
Journal of Experimental Criminology 6 (2): 191–208. the-real-digital-divide-in-educational-technology/.
Berlinski, Samuel, Matias Busso, Taryn Dinkelman, and Bulman, G., and R. W. Fairlie. 2016. “Technology and
Claudia Martinez. 2016. “Reducing Parent-School Education: Computers, Software, and the Internet.”
Information Gaps and Improving Education Out- In Handbook of the Economics of Education. Vol.
comes: Evidence from High Frequency Text Mes- 5, edited by Eric A. Hanushek, Stephen Machin,
saging in Chile.” Unpublished. and Ludger Woessmann, 239–80. Amsterdam:
Bettinger, Eric P., Lindsay Fox, Susanna Loeb, and Eric North-Holland.
S. Taylor. 2017. “Virtual Classrooms: How Online Bursztyn, Leonardo, and Robert Jensen. 2015.
College Courses Affect Student Success.” American “How Does Peer Pressure Affect Educational
Economic Review 107 (9): 2855–75. Investments?” Quarterly Journal of Economics 130
Bettinger, Eric P., Bridget Terry Long, Philip Oreo- (3): 1329–67.
poulos, and Lisa Sanbonmatsu. 2012. “The Role of Cabalo, Jessica Villaruz, Boya Ma, and Andrew Jaciw.
2007. Comparative Effectiveness of Carnegie Learn- ConnectED Initiative. 2013. Obama White House
ing’s Cognitive Tutor Bridge to Algebra Curriculum: Archives. https://obamawhitehouse.archives.gov/
A Report of a Randomized Experiment in the Maui issues/education/k-12/connected.
School District. Palo Alto: Empirical Education Inc. Cook, Thomas D., and Vivian C. Wong. 2008. “Empir-
Campuzano, Larissa, Mark Dynarski, Roberto Agodini, ical Tests of the Validity of the Regression Disconti-
and Kristina Rall. 2009. Effectiveness of Reading and nuity Design.” Annales d’Économie et de Statistique
Mathematics Software Products: Findings from Two 91: 127–50.
Student Cohorts. Washington, DC: National Center Cohn, D’vera. 2014. “Census: Computer Ownership,
for Education Evaluation and Regional Assistance at Internet Connection Varies Widely across U.S.” Pew
the Institute of Education Sciences. Research Center FactTank, September 19. https://
Carter, Susan Payne, Kyle Greenberg, and Michael www.pewresearch.org/fact-tank/2014/09/19/census-
S. Walker. 2017. “The Impact of Computer Usage computer-ownership-internet-connection-varies-
on Academic Performance: Evidence from a widely-across-u-s/.
Randomized Trial at the United States Military Acad- Cormier, Dave, and George Siemens. 2010. “Through
emy.” Economics of Education Review 56: 118–32. the Open Door: Open Courses as Research, Leaning,
Castleman, Benjamin L., Karen Arnold, and Katherine and Engagement.” Educause Review 45 (4), 30–39.
Lynk Wartman. 2012. “Stemming the Tide of Sum- Cortes, Kalena E., Hans Fricke, Susanna Loeb, and
mer Melt: An Experimental Study of the Effects of David S. Song. 2018. “Too Little or Too Much?
Post-High School Summer Intervention on Low-In- Actionable Advice in an Early-Childhood Text Mes-
come Students’ College Enrollment.” Journal of saging Experiment.” NBER Working Paper 24827.
Research on Educational Effectiveness 5 (1): 1–17. Council of Economic Advisers. 2016. The Digital
Castleman, Benjamin L., and Katharine Meyer. 2016. Divide and Economic Benefits of Broadband Access.
“Can Text Message Nudges Improve Academic Out- Washington, DC: White House.
comes in College? Evidence from a West Virginia Cowen, Tyler, and Alex Tabarrok. 2014. “The Industrial
Initiative.” EdPolicyWorks Working Paper 43. Organization of Online Education.” American Eco-
Castleman, Benjamin L., and Lindsay C. Page. 2015. nomic Review 104 (5): 519–22.
“Summer Nudging: Can Personalized Text Mes- Cristia, Julian, Pablo Ibarrarán, Santiago Cueto, Ana
sages and Peer Mentor Outreach Increase College Santiago, and Eugenio Severín. 2017. “Technology
Going among Low-Income High School Graduates?” and Child Development: Evidence from the One
Journal of Economic Behavior & Organization 115: Laptop per Child Program.” American Economic
144–60. Journal: Applied Economics 9 (3): 295–320.
Castleman, Benjamin L., and Lindsay C. Page. 2016. Cunha, Flavio, and James Heckman. 2007. “The Tech-
“Freshman Year Financial Aid Nudges: An Exper- nology of Skill Formation.” American Economic
iment to Increase FAFSA Renewal and College Review 97 (2): 31–47.
Persistence.” Journal of Human Resources 51 (2): Cunha, Flavio, James J. Heckman, Lance Lochner,
389–415. and Dimitriy V. Masterov. 2006. “Interpreting the
Castleman, Benjamin L., and Lindsay C. Page. 2017. Evidence on Life Cycle Skill Formation.” In Hand-
“Parental Influences on Postsecondary Decision book of the Economics of Education, Vol. 1, edited
Making: Evidence from a Text Messaging Experi- by E. Hanushek and F. Welch, 697–812. Amsterdam:
ment.” Educational Evaluation and Policy Analysis North-Holland.
39 (2): 361–77 Darolia, Rajeev, and Casandra Harper. 2018. “Informa-
Cavalluzzo, Linda, Deborah L. Lowther, Christine tion Use and Attention Deferment in College Stu-
Mokher, and Xitao Fan. 2012. Effects of the Kentucky dent Loan Decisions: Evidence from a Debt Letter
Virtual Schools’ Hybrid Program for Algebra I on Experiment.” Educational Evaluation and Policy
Grade 9 Student Math Achievement: Final Report. Analysis 40 (1): 129–50.
Washington, DC: National Center for Education Davis, Dan, Ioana Jivet, René F. Kizilcec, Guanliang
Evaluation and Regional Assistance at the Institute Chen, Claudia Hauff, and Geert-Jan Houben. 2017.
of Education Sciences. “Follow the Successful Crowd: Raising MOOC Com-
Chande, Raj, et al. 2015. “Curbing Adult Student Attri- pletion Rates through Social Comparison at Scale.”
tion: Evidence from a Field Experiment.” Harvard In Proceedings of the Seventh International Learning
Business School Working Paper 15-065. Analytics and Knowledge Conference, 454–63. New
Cohen, Geoffrey L., Julio Garcia, Nancy Apfel, and York: Association for Computing Machinery.
Allison Master. 2006. “Reducing the Racial Achieve- Davis, Dan, René F. Kizilcec, Claudia Hauff, and
ment Gap: A Social-Psychological Intervention.” Sci- Geert-Jan Houben. 2018. “Scaling Effective Learn-
ence 313 (5791): 1307–10. ing Strategies: Retrieval Practice and Long-Term
Knowledge Retention in MOOCs.” Journal of Learn- Faber, Janke M., and Adrie J. Visscher. 2018. “The
ing Analytics 5 (3): 21–41. Effects of a Digital Formative Assessment Tool on
Deault, Louise, Robert Savage, and Philip Abrami. Spelling Achievement: Results of a Randomized
2009. “Inattention and Response to the ABRACA- Experiment.” Computers and Education 122: 1–8.
DABRA Web-Based Literacy Intervention.” Jour- Fairlie, Robert W. 2012a. “Academic Achievement,
nal of Research on Educational Effectiveness 2 (3): Technology and Race: Experimental Evidence.” Eco-
250–86. nomics of Education Review 31 (5): 663–79.
Dee, Thomas S. 2015. “Social Identity and Achievement Fairlie, Robert W. 2012b. “The Effects of Home Access
Gaps: Evidence from an Affirmation Intervention.” to Technology on Computer Skills: Evidence from a
Journal of Research on Educational Effectiveness 8 Field Experiment.” Information Economics and Pol-
(2): 149–68. icy 24 (3–4): 243–53.
Deming, David J., Claudia Goldin, and Lawrence F. Fairlie, Robert W. 2015. “Do Boys and Girls Use Com-
Katz. 2012. “The For-Profit Postsecondary School puters Differently, and Does It Contribute to Why
Sector: Nimble Critters or Agile Predators?” Journal Boys Do Worse in School Than Girls?” CESifo
of Economic Perspectives 26 (1): 139–64. Working Paper 5496.
Deming, David J., Noam Yuchtman, Amira Abu- Fairlie, Robert W., and Peter Riley Bahr. 2018. “The
lafi, Claudia Goldin, and Lawrence F. Katz. 2016. Effects of Computers and Acquired Skills on Earn-
“The Value of Postsecondary Credentials in the ings, Employment and College Enrollment: Evi-
Labor Market: An Experimental Study.” American dence from a Field Experiment and California UI
Economic Review 106 (3): 778–806. Earnings Records.” Economics of Education Review
Dinkelman, Taryn, and Claudia Martínez A. 2014. 63: 51–63.
“Investing in Schooling in Chile: The Role of Infor- Fairlie, Robert W., and Samantha H. Grunberg. 2014.
mation about Financial Aid for Higher Education.” “Access to Technology and the Transfer Function of
Review of Economics and Statistics 96 (2): 244–57. Community Colleges: Evidence from a Field Experi-
Donovan, Colleen, David N. Figlio, and Mark Rush. ment.” Economic Inquiry 52 (3): 1040–59.
2006. “Cramming: The Effects of School Account- Fairlie, Robert W., and Ariel Kalil. 2017. “The Effects
ability on College-Bound Students.” NBER Working of Computers on Children’s Social Development
Paper 12628. and School Participation: Evidence from a Random-
Doss, Christopher J., Erin M. Fahle, Susanna Loeb, ized Control Experiment.” Economics of Education
and Benjamin N. York. 2018. “More Than Just a Review 57: 10–19.
Nudge: Supporting Kindergarten Parents with Dif- Fairlie, Robert W., and Rebecca A. London. 2012.
ferentiated and Personalized Text-Messages.” NBER “The Effects of Home Computers on Educational
Working Paper 24450. Outcomes: Evidence from a Field Experiment with
Duckworth, Angela L., Christopher Peterson, Michael Community College Students.” Economic Journal
D. Matthews, and Dennis R. Kelly. 2007. “Grit: 122 (561): 727–53.
Perseverance and Passion for Long-Term Goals.” Fairlie, Robert W., and Jonathan Robinson. 2013.
Journal of Personality and Social Psychology 92 (6): “Experimental Evidence on the Effects of Home
1087–1101. Computers on Academic Achievement among
Dynarski, Mark, et al. 2007. “Effectiveness of Reading Schoolchildren.” American Economic Journal:
and Mathematics Software Products: Findings from Applied Economics 5 (3): 211–40.
the First Student Cohort.” Washington, DC: National Fancsali, Stephen E., Steven Ritter, Michael Yudelson,
Center for Education Evaluation and Regional Assis- Michael Sandbothe, and Susan R. Berman. 2016.
tance at the Institute of Education Sciences. “Implementation Factors and Outcomes for Intelli-
Esperanza, Peter, Khristin Fabian, and Criselda Toto. gent Tutoring Systems: A Case Study of Time and
2016. “Flipped Classroom Model: Effects on Per- Efficiency with Cognitive Tutor Algebra.” In Pro-
formance, Attitudes and Perceptions in High School ceedings of the Twenty-Ninth International Florida
Algebra.” In Adaptive and Adaptable Learning: 11th Artificial Intelligence Research Society Conference,
European Conference on Technology Enhanced edited by Zdravko Markov and Ingrid Russell, 473–
Learning, edited by Katrien Verbert, Mike Shar- 78. Palo Alto: AAAI Press.
ples, and Tomaž Klobučar, 85–97. Amsterdam: Figlio, David, Mark Rush, and Lu Yin. 2013. “Is It Live
North-Holland. or Is It Internet? Experimental Estimates of the
Faber, Benjamin, Rosa Sanchis-Guarner, and Felix Effects of Online Instruction on Student Learning.”
Weinhardt. 2015. “ICT and Education: Evidence Journal of Labor Economics 31 (4): 763–84.
from Student Home Addresses.” NBER Working Foldnes, Njål. 2016. “The Flipped Classroom and
Paper 21306. Cooperative Learning: Evidence from a Randomised
Experiment.” Active Learning in Higher Education Perspectives 36 (3): 179–81.

17 (1): 39–49. He, Fang, Leigh L. Linden, and Margaret MacLeod.
Forsyth, Donelson R., Natalie K. Lawrence, Jeni L. 2007. “Helping Teach What Teachers Don’t Know:
Burnette, and Roy F. Baumeister. 2007. “Attempting An Assessment of the Pratham English Language
to Improve the Academic Performance of Struggling Program.” https://pdfs.semanticscholar.org/9192/418
College Students by Bolstering Their Self-Esteem: b948e7447d0013e44c9a06f345b801d42.pdf.
An Intervention That Backfired.” Journal of Social Hegedus, Stephen J., Sara Dalton, and John R. Tap-
and Clinical Psychology 26 (4): 447–59. per. 2015. “The Impact of Technology-Enhanced
Fryer, Roland G. 2016. “Information, Non-financial Curriculum on Learning Advanced Algebra in US
Incentives, and Student Achievement: Evidence High School Classrooms.” Educational Technology
from a Text Messaging Experiment.” Journal of Pub- Research and Development 63 (2): 203–28.
lic Economics 144: 109–21. Heppen, Jessica B., et al. 2011. Access to Algebra I: The
Good, Catherine, Joshua Aronson, and Michael Effects of Online Mathematics for Grade 8 Students.
Inzlicht. 2003. “Improving Adolescents’ Standard- Washington, DC: National Center for Education
ized Test Performance: An Intervention to Reduce Evaluation and Regional Assistance at the Institute
the Effects of Stereotype Threat.” Applied Develop- of Education Sciences.
mental Psychology 24: 645–62. Heppen, Jessica, Nicholas Sorensen, Elaine
Goodman, Joshua, Julia Melkers, and Amanda Pal- Allensworth, Kirk Walters, Suzanne Stachel, and Val-
lais. 2019. “Can Online Delivery Increase Access erie Michelman. 2012. “Efficacy of Online Algebra
to Education?” Journal of Labor Economics 37 (1): I for Credit Recovery for At-Risk Ninth Graders:
1–34. Consistency of Results from Two Cohorts.” Paper
Goolsbee, Austan, and Jonathan Guryan. 2006. “The presented at the Society for Research on Educational
Impact of Internet Subsidies in Public Schools.” Effectiveness (SREE) Spring 2012 Conference,
Review of Economics and Statistics 88 (2): 336–47. Washington, DC.
Greene, Jeffrey A., Christopher A. Oswald, and Jef- Hodges, Charles, Patrick Lowenthal, and Michael
frey Pomerantz. 2015. “Predictors of Retention and Grant. 2016. “Teacher Professional Development in
Achievement in a Massive Online Open Course.” the Digital Age: Design considerations for MOOCs
American Educational Research Journal 52 (5): for Teachers.” Paper presented at the Society for
925–55. Information Technology and Teacher Education
Guryan, Jonathan, Erik Hurst, and Melissa Kearney. International Conference, Savannah.
2008. “Parental Education and Parental Time with Horrigan, John B. 2015. “The Numbers behind the
Children.” Journal of Economic Perspectives 22 (3): Broadband ‘Homework Gap.’” Fact Tank, April 20.
23–46. http://www.pewresearch.org/fact-tank/2015/04/20/
Hansen, John D., and Justin Reich. 2015. “Democra- the-numbers-behind-the-broadband-homework-
tizing Education? Examining Access and Usage Pat- gap/.
terns in Massive Open Online Courses.” Science 350 Hulleman, Chris S., and Judith M. Harackiewicz. 2009.
(6265): 1245–48. “Promoting Interest and Performance in High School
Hanselman, Paul, Sarah K. Bruch, Adam Gamoran, Science Classes.” Science 326 (5958): 1410–12.
and Geoffrey D. Borman. 2014. “Threat in Context: Hurwitz, Lisa B., Alexis R. Lauricella, Ann Hanson,
School Moderation of the Impact of Social Identity Anthony Raden, and Ellen Wartella. 2015. “Support-
Threat on Racial/Ethnic Achievement Gaps.” Sociol- ing Head Start Parents: Impact of a Text Message
ogy of Education 87 (2): 106–24. Intervention on Parent–Child Activity Engagement.”
Harackiewicz, Judith M., and Stacy J. Priniski. 2018. Early Child Development and Care 185 (9): 1373–89.
“Improving Student Outcomes in Higher Educa- Hyman, Joshua. 2018. “Nudges, College Enrollment,
tion: The Science of Targeted Intervention.” Annual and College Persistence: Evidence from a Statewide
Review of Psychology 69: 409–35. Experiment in Michigan.” University of Michigan
Harackiewicz, Judith M., Christopher S. Rozek, Chris Education Policy Initiative. Working Paper 02-2018.
S. Hulleman, and Janet S. Hyde. 2012. “Helping Par- Imbens, Guido W., and Joshua D. Angrist. 1994. “Iden-
ents to Motivate Adolescents in Mathematics and tification and Estimation of Local Average Treatment
Science: An Experimental Test of a Utility-Value Effects.” Econometrica 62 (2): 467–75.
Intervention.” Psychological Science 23 (8): 899–906. Imbens, Guido W., and Thomas Lemieux. 2008.
Harrington, Susan Ann, Melodee Vanden Bosch, “Regression Discontinuity Designs: A Guide to
Nancy Schoofs, Cynthia Beel-Bates, and Kirk Ander- Practice.” Journal of Econometrics 142 (2): 615–35.
son. 2015. “Quantitative Outcomes for Nursing Stu- Jackson, Kirabo, and Alexey Makarin. 2018. “Can
dents in a Flipped Classroom.” Nursing Education Online Off-the-Shelf Lessons Improve Student
Outcomes? Evidence from a Field Experiment.” “The Effect of Teacher–Family Communication on

American Economic Journal: Economic Policy 10 (3): Student Engagement: Evidence from a Randomized
226–54. Field Experiment.” Journal of Research on Educa-
Joyce, Ted, Sean Crockett, David A. Jaeger, Onur Altin- tional Effectiveness 6 (3): 199–222.
dag, and Stephen D. O’Connell. 2015. “Does Class- Kraft, Matthew A., and Manuel Monti-Nussbaum.
room Time Matter?” Economics of Education Review 2017. “Can Schools Enable Parents to Prevent Sum-
46: 64–77. mer Learning Loss? A Text-Messaging Field Exper-
Kahneman, Daniel. 2011. Thinking, Fast and Slow. iment to Promote Literacy Skills.” ANNALS of the
New York: Farrar, Straus and Giroux. American Academy of Political and Social Science
Kalil, Ariel. 2015. “Inequality Begins at Home: The 674 (1): 85–112.
Role of Parenting in the Diverging Destinies of Rich Kraft, Matthew A., and Todd Rogers. 2015. “The
and Poor Children.” In Families in an Era of Increas- Underutilized Potential of Teacher-to-Parent Com-
ing Equality. Vol. 5, National Symposium on Family munication: Evidence from a Field Experiment.”
Issues, 63–82. Berlin: Springer. Economics of Education Review 47: 49–63.
Kang, Cecilia. 2016. “Bridging a Digital Divide That Ksoll, Christopher, Jenny Aker, Danielle Miller, Karla
Leaves Schoolchildren Behind.” New York Times, C. Perez-Mendoza, and Susan L. Smalley. 2014.
February 22. https://www.nytimes.com/2016/02/23/ “Learning without Teachers? A Randomized Exper-
technology/fcc-internet-access-school.html. iment of a Mobile Phone-Based Adult Education
Kaput, James J., and Jeremy Roschelle. 2013. “The Program in Los Angeles.” Center for Global Devel-
Mathematics of Change and Variation from a Mil- opment Working Paper 368.
lennial Perspective: New Content, New Context.” Kulik, James A., and J. D. Fletcher. 2015. “Effective-
In The SimCalc Vision and Contributions, edited by ness of Intelligent Tutoring Systems: A Meta-analytic
Stephen J. Hegedus and Jeremy Roschelle, 13–26. Review.” Review of Educational Research 86 (1):
Dordrecht, Netherlands: Springer. 42–78.
Karam, Rita, John F. Pane, Beth Ann Griffin, Abby Lai, Fang, et al. 2013. “Computer Assisted Learning as
Robyn, Andrea Phillips, and Lindsay Daugherty. Extracurricular Tutor? Evidence from a Randomised
2017. “Examining the Implementation of Technol- Experiment in Rural Boarding Schools in Shaanxi.”
ogy-Based Blended Algebra I Curriculum at Scale.” Journal of Development Effectiveness 5 (2): 208–31.
Educational Technology Research and Development Lai, Fang, et al. 2016. “More Is Not Always Better:
65 (2): 399–425. Evidence from a Randomised Experiment of Com-
Keefe, Thomas J. 2003. “Using Technology to Enhance puter-Assisted Learning in Rural Minority Schools in
a Course: The Importance of Interaction.” Educause Qinghai.” Journal of Development Effectiveness 8 (4):
Quarterly 1: 24–34. 449–72.
Kelly, Kim, Neil Heffernan, Cristina Heffernan, Susan Lai, Fang, Renfu Luo, Linxiu Zhang, Xinzhe Huang,
Goldman, James Pellegrino, and Deena Soffer Gold- and Scott Rozelle. 2015. “Does Computer-Assisted
stein. 2013. “Estimating the Effect of Web-Based Learning Improve Learning Outcomes? Evidence
Homework.” In 16th International Conference on from a Randomized Experiment in Migrant Schools
Artificial Intelligence in Education, edited by H. in Beijing.” Economics of Education Review 47:
Chad Lane, Kalina Yacef, Jack Mostow, and Philip 34–48.
Pavlik, 824–27. Berlin: Springer. Lamb, Anne, Jascha Smilack, Andrew Ho, and Jus-
Kizilcec, René F., Andrew J. Saltarelli, Justin tin Reich. 2015. “Addressing Common Analytic
Reich, Geoffrey L. Cohen. 2017. “Closing Global Challenges to Randomized Experiments in MOOCs:
Achievement Gaps in MOOCs.” Science 355 (6322): Attrition and Zero-Inflation.” In L@S ‘15: Pro-
251–52. ceedings of the Second (2015) ACM Conference on
Kizilcec, René F., Emily Schneider, Geoffrey L. Cohen, Learning @ Scale, 21–30. New York: Association for
and Daniel A. McFarland. 2014. “Encouraging Computing Machinery.
Forum Participation in Online Courses with Collec- Lantz, Michael E., and Angela Stawiski. 2014. “Effec-
tivist, Individualist and Neutral Motivational Fram- tiveness of Clickers: Effect of Feedback and the Tim-
ings.” Open Education Europa eLearning Papers 37: ing of Questions on Learning.” Computers in Human
13–22. Behavior 31: 280–86.
Koch, Alexander, Julia Nafziger, and Helena Skyt Niel- Lavecchia, A. M., H. Liu, and P. Oreopoulos. 2016.
sen. 2015. “Behavioral Economics of Education.” “Behavioral Economics of Education: Progress and
Journal of Economic Behavioral and Organization Possibilities.” In Handbook of the Economics of Edu-
115 (C): 3–17. cation, Vol. 5, edited by Eric A. Hanushek, Stephen
Kraft, Matthew A., and Shaun M. Dougherty. 2013. Machin, and Ludger Woessmann, 1–74. Amsterdam:
North-Holland. Evidence-Based Practices in Online Learning: A

Lazowski, Rory A., and Chris S. Hulleman. 2016. Meta-analysis and Review of Online Learning Stud-
“Motivation Interventions in Education: A Meta-an- ies. Washington, DC: US Department of Education.
alytic Review.” Review of Educational Research 86 Meuwissen, Alyssa, Alison Giovanelli, Madelyn Labella,
(2): 602–40. and Amy Susman-Stillman. 2017. Text2Learn: An
Lee, Jung-Sook, and Natasha K. Bowen. 2006. “Parent Early Literacy Texting Intervention by Commu-
Involvement, Cultural Capital, and the Achievement nity Organizations. St. Paul: Minnesota Children’s
Gap among Elementary School Children.” American Museum and University of Minnesota.
Educational Research Journal 43 (2): 193–218. Mitchell, Mary Jane, and Barbara J. Fox. 2001. “The
Leuven, Edwin, Mikael Lindahl, Hessel Oosterbeek, Effects of Computer Software for Developing Pho-
and Dinand Webbink. 2007. “The Effect of Extra nological Awareness in Low-Progress Readers.”
Funding for Disadvantaged Pupils on Achievement.” Reading Research and Instruction 40 (4): 325–32.
Review of Economics and Statistics 89 (4): 721–36. Mo, Di, Weiming Huang, Yaojiang Shi, Linxiu Zhang,
Levitt, Steven D., John A. List, Susanne Neckermann, Matthew Boswell, and Scott Rozelle. 2015. “Com-
and Sally Sadoff. 2016. “The Behavioralist Goes puter Technology in Education: Evidence from a
to School: Leveraging Behavioral Economics to Pooled Study of Computer Assisted Learning Pro-
Improve Educational Performance.” American Eco- grams among Rural Students in China.” China Eco-
nomic Journal: Economic Policy 8 (4): 183–219. nomic Review 36: 131–45.
Linden, Leigh L. 2008. “Complement or Substitute? Mo, Di, Linxiu Zhang, Renfu Luo, Qinghe Qu, Weim-
The Effect of Technology on Student Achievement ing Huang, Jiafu Wang, et al. 2014. “Integrating
in India.” http://www.leighlinden.com/Gyan_Shala_ Computer-Assisted Learning into a Regular Curric-
CAL_2008-06-03.pdf. ulum: Evidence from a Randomised Experiment in
Malamud, Ofer, and Cristian Pop-Eleches. 2011. Rural Schools in Shaanxi.” Journal of Development
“Home Computer Use and the Development of Effectiveness 6 (3): 300–323.
Human Capital.” Quarterly Journal of Economics Mo, Di, Linxiu Zhang, Jiafu Wang, Weiming Huang,
126 (2): 987–1027. Yaojiang Shi, Matthew Boxwell, et al. 2014. “The
Martinez, Ignacio. 2014a. “The Effects of Informa- Persistence of Gains in Learning from Computer
tional Nudges on Students’ Effort and Performance: Assisted Learning (CAL): Evidence from a Random-
Lessons from a MOOC.” EdPolicyWorks Working ized Experiment in Rural Schools in Shaanxi Prov-
Paper 19. ince in China.” Rural Education Action Program
Martinez, Ignacio. 2014b. “Never Put Off ‘till Tomor- (REAP) Working Paper 268.
row?” EdPolicyWorks Working Paper 28. Molnar, Michele. 2017. “Ed-Tech Surges Interna-
Mayer, Susan E., Ariel Kalil, Philip Oreopoulos, and tionally—And Choices for Schools Become More
Sebastian Gallegos. 2015. “Using Behavioral Insights Confusing.” EdWeek Market Brief, January 20.
to Increase Parental Engagement: The Parents and https://marketbrief.edweek.org/marketplace-k-12/
Children Together (PACT) Intervention.” NBER ed-tech-surges-internationally-choices-schools-be-
Working Paper 21602. come-confusing/.
Mayer, Susan E., Ariel Kalil, Philip Oreopoulos, and Morgan, Pat, and Steven Ritter. 2002. An Experimental
Sebastian Gallegos. 2019. “Using Behavioral Insights Study of the Effects of Cognitive Tutor Algebra I on
to Increase Parental Engagement: The Parents and Student Knowledge and Attitude. Pittsburgh: Carne-
Children Together Intervention.” Journal of Human gie Learning, Inc.
Resources 54 (4): 900–925. Morisano, Dominique, Jacob B. Hirsh, Jordan B.
McGuigan, Martin, Sandra McNally, and Gill Wyn- Peterson, Robert O. Pihl, and Bruce M. Shore. 2010.
ess. 2012. “Student Awareness of Costs and Benefits “Setting, Elaborating, and Reflecting on Personal
of Educational Decisions: Effects of an Informa- Goals Improves Academic Performance.” Journal of
tion Campaign.” Journal of Human Capital 10 (4): Applied Psychology 95 (2): 255–64.
482–519. Morrison, Nick. 2017. “Google Leapfrogs Rivals
McKenzie, Lindsay. 2018. “The 100KClub.” Inside to Be Classroom King.” Forbes, May 9. https://
Higher Ed, April 23. https://www.insidehighered. www.forbes.com/sites/nickmorrison/2017/05/09/
com/news/2018/04/23/nonprofits-poised-unseat-u- google-leapfrogs-rivals-to-be-classroom-
phoenix-largest-online-university. king/#32966ae927a6.
McLester, Susan. 2012. “One Tablet Per Child?” Muralidharan, Karthik, Abhijeet Singh, and Alejandro
Unpublished. J. Ganimian. 2016. “Disrupting Education? Experi-
Means, Barbara, Yukie Toyama, Robert Murphy, Mar- mental Evidence on Technology-Aided Instruction
ianne Bakia, and Karla Jones. 2009. Evaluation of in India.” NBER Working Paper 22923.
Naik, Gopal, Chetan Chitre, Manaswini Bhalla, and Journal of Economic Behavior and Organization 153:
Jothsna Rajan. 2016. “Can Technology Overcome 293–321.
Social Disadvantage of School Children’s Learning Paunesku, David, Gregory M. Walton, Carissa Romero,
Outcomes? Evidence from a Large-Scale Experi- Eric N. Smith, David S. Yeager, and Carol S. Dweck.
ment in India.” Available on SSRN at 2775558. 2015. “Mind-Set Interventions Are a Scalable Treat-
O’Connell, Stephen D., and Guido Lang. 2018. “Can ment for Academic Underachievement.” Psychologi-
Personalized Nudges Improve Learning in Hybrid cal Science 26 (6): 784–93.
Classes? Experimental Evidence from an Introduc- Pew Research Center. 2017. “Internet/Broadband Fact
tory Undergraduate Course.” Journal of Research on Sheet.” Accessed on January 12, 2017. http://www.
Technology in Education 50 (2): 105–19. pewinternet.org/fact-sheet/internet-broadband/.
One Laptop per Child (OLPC). 2015. “One Laptop per Piper, Benjamin, Stephanie Simmons Zuilkowski, Dun-
Child (OLCP) Mission.” http://laptop.org/en/vision/ ston Kwayumba, and Carmen Strigel. 2016. “Does
mission/. Technology Improve Reading Outcomes? Compar-
Oreopoulos, Philip, and Ryan Dunn. 2013. “Informa- ing the Effectiveness and Cost-Effectiveness of ICT
tion and College Access: Evidence from a Random- Interventions for Early Grade Reading in Kenya.”
ized Field Experiment.” Scandinavian Journal of International Journal of Educational Development
Economics 115 (1): 3–26. 49: 204–14.
Oreopoulos, Philip, and Reuben Ford. 2019. “Keeping Poirier, Christopher R., and Robert S. Feldman. 2004.
College Options Open: A Field Experiment to Help “Teaching in Cyberspace: Online versus Traditional
All High School Seniors through the College Appli- Instruction Using a Waiting-List Experimental
cation Process.” Journal of Policy Analysis and Man- Design.” Teaching of Psychology 31 (1): 59–62.
agement 38 (2): 426–54. Price, Joseph. 2010. “The Effect of Parental Time
Oreopoulos, Philip, and Uros Petronijevic. 2017. “Stu- Investments: Evidence from Natural Within-Family
dent Coaching: How Far Can Technology Go?” Jour- Variation.” http://citeseerx.ist.psu.edu/viewdoc/down
nal of Human Resources 53 (2): 299–329. load?doi=10.1.1.716.7451&rep=rep1&type=pdf.
Oreopoulos, Philip, Richard W. Patterson, Uros Ragosta, Marjorie, Paul W. Holland, and Dean T. Jami-
Petronijevic, and Nolan G. Pope. 2018. “When son. 1982. Computer-Assisted Instruction and Com-
Studying and Nudging Don’t Go as Planned: Unsuc- pensatory Education: The ETS/LAUSD Study: The
cessful Attempts to Help Traditional and Online Col- Final Report. Washington, DC: National Institute of
lege Students.” NBER Working Paper 25036. Education.
Oreopoulos, Philip, Uros Petronijevic, Christine Rege, Mari, et al. Forthcoming. “How can we inspire
Logel, and Graham Beattie. 2018. “Improving nations of learners? Investigating growth mindset
Non-academic Student Outcomes Using Online and and challenge-seeking in two countries.” American
Text-Message Coaching.” NBER Working Paper Psychologist.
24992. Ritter, Steven, Jonna Kulikowich, Pui-Wa Lei, Christy
Page, Lindsay C., Benjamin Castleman, and Katha- L. McGuire, and Pat Morgan. 2007. “What Evi-
rine Meyer. 2016. “Customized Nudging to Improve dence Matters? A Randomized Field Trial of Cog-
FAFSA Completion and Income Verification.” Avail- nitive Tutor Algebra I.” In Supporting Learning
able on SSRN at 2854345. Flow through Integrative Technologies. Vol. 162 of
Page, Lindsay C., and Hunter Gehlbach. 2017. “How Frontiers in Artificial Intelligence and Applications,
an Artificially Intelligent Virtual Assistant Helps Stu- edited by Tsukasa Hirashima, Ulrich Hoppe, and
dents Navigate the Road to College.” AERA Open 3 Shelley Shwu-Ching Young, 13–20. Amsterdam: IOS
(4): https://doi.org/10.1177/2332858417749220. Press.
Pane, John F., Daniel F. McCaffrey, Mary Ellen Rockoff, Jonah E. 2015. “Evaluation Report on the
Slaughter, Jennifer L. Steele, and Gina S. Ikemoto. School of One i3 Expansion.” edweek.org/media/
2010. “An Experiment to Evaluate the Efficacy of evaluation%20of%20the%20school%20of%20
Cognitive Tutor Geometry.” Journal of Research on one%20i3%20expansion%20--%20final%20copy.
Educational Effectiveness 3 (3): 254–81. pdf.
Pane, John F., Beth Ann Griffin, Daniel F. McCaffrey, Rodriguez, C., Fabio Torres, and Juliana Zúñiga. 2011.
and Rita Karam. 2014. “Effectiveness of Cognitive “Impact of the ‘Computers to Educate’ Program on
Tutor Algebra I at Scale.” Educational Evaluation student dropout, school achievement and income
and Policy Analysis 36 (2): 127–44. to higher education.” CEDE Documents ISSN
Patterson, Richard W. 2018. “Can Behavioral Tools 1657-7191.
Improve Online Student Outcomes? Experimental Rogers, Todd, and Avi Feller. 2018. “Reducing Student
Evidence from a Massive Open Online Course.” Absences at Scale by Targeting Parents’ Misbeliefs.”
Nature Human Behavior 2: 335–42. economy/policy/distribution-of-free-laptops-to-tn-

Roschelle, Jeremy, et al. 2010. “Integration of Tech- students-from-sept-15/article2123738.ece.
nology, Curriculum, and Professional Develop- Singh, Ravi, et al. 2011. “Feedback during Web-Based
ment for Advancing Middle School Mathematics: Homework: The Role of Hints.” In Artificial Intelli-
Three Large-Scale Studies.” American Educational gence in Education: 15th International Conference,
Research Journal 47 (4): 833–78. AIED 2011, edited by G. Biswas, S. Bull, J. Kay, and
Roschelle, Jeremy, Mingyu Feng, Robert F. Murphy, A. Mitrovic, 328–36. Berlin: Springer.
and Craig A. Mason. 2016. “Online Mathematics Sisk, Victoria F., Alexander P. Burgoyne, Jingze Sun,
Homework Increases Student Achievement.” AERA Jennifer L. Butler, and Brooke N. Macnamara. 2018.
Open 2 (4): Article 2332858416673968. “To What Extent and under Which Circumstances
Rouse, Cecilia Elena, and Alan B. Krueger. 2004. Are Growth Mind-Sets Important to Academic
“Putting Computerized Instruction to the Test: A Achievement? Two Meta-analyses.” Psychological
Randomized Evaluation of a ‘Scientifically Based’ Science 29 (4): 549–71.
Reading Program.” Economics of Education Review Sleeter, Nate. 2014. “Meaningful Interaction in
23 (4): 323–38. Online Courses.” Inside Higher Ed, July 30. http://
Rutherford, Teomara, et al. 2014. “A Randomized Trial www.insidehighered.com/blogs/higher-ed-gamma/
of an Elementary School Mathematics Software meaningful-interaction-online-courses.
Intervention: Spatial-Temporal Math.” Journal of Smith, Ben O., Dustin R. White, Patricia C. Kuzyk, and
Research on Educational Effectiveness 7 (4): 358–83. James E. Tierney. 2018. “Improved Grade Outcomes
Schenke, Katerina, Teomara Rutherford, and George with an Emailed ‘Grade Nudge.’” Journal of Eco-
Farkas. 2014. “Alignment of Game Design Fea- nomic Education 49 (1): 1–7.
tures and State Mathematics Standards: Do Results Snipes, Jason, Chun-Wei Huang, Karina Jaquet, and
Reflect Intentions?” Computers and Education 76: Neal Finkelstein. 2015. The Effects of the Elevate
215–24. Math Summer Program on Math Achievement and
Schilbach, Frank, Heather Schofield, and Sendhil Algebra Readiness. Washington, DC: National Cen-
Mullainathan. 2016. “The Psychological Lives of the ter for Education Evaluation and Regional Assis-
Poor.” American Economic Review 106 (5): 435–40. tance at the Institution of Education Sciences.
School Guides. 2014. “Survey Reveals How Much Col- Tatar, Deborah, Jeremy Roschelle, Jennifer Knudsen,
lege Students Rely on Technology,” July 13. http:// Nicole Shechtman, Jim Kaput, and Bill Hopkins.
www.schoolguides.com/College_News/Survey_ 2008. “Scaling up Innovative Technology-Based
reveals_how_much_college_students_rely_on_ Mathematics.” Journal of the Learning Sciences 17
technology_643742.html. (2): 248–86.
Sénéchal, Monique, and Jo-Anne Lefevre. 2002. Thaler, Richard H., and Shlomo Benartzi. 2004. “Save
“Parental Involvement in the Development of More Tomorrow™: Using Behavioral Economics
Children’s Reading Skill: A Five-Year Longitudinal to Increase Employee Saving.” Journal of Political
Study.” Child Development 73 (2): 445–60. Economy 112 (S1): S164–87.
Shadish, William R., Rodolfo Galindo, Vivian C. Wong, Thaler, Richard H., and Cass R. Sunstein. 2008. Nudge:
Peter M. Steiner, and Thomas D. Cook. 2011. “A Improving Decisions about Health, Wealth, and
Randomized Experiment Comparing Random and Happiness. New York: Penguin Group.
Cutoff-Based Assignment.” Psychological Methods Thistlewaite, Donald L., and Donald T. Campbell.
16 (2):179–91. 1960. “Regression-Discontinuity Analysis: An Alter-
Shah, Dhawal. 2018. “A Product at Every Price: native to the Ex Post Facto Experiment.” Journal of
A Review of MOOC Stats and Trends in 2017.” Educational Psychology 51 (6): 309–17.
EdSurge, January 22. https://www.edsurge.com/ Trucano, Michael. 2013. “Big Educational Laptop and
news/2018-01-22-a-product-at-every-price-a-review- Tablet Projects—Ten Countries to Learn from.”
of-mooc-stats-and-trends-in-2017. EduTech,  July  31.  http://blogs.worldbank.org/
SIIA Communications. 2015. “SIIA Estimates $8.38 edutech/big-educational-laptop-and-tablet-projects-
Billion US Market for PreK-12 Educational Software ten-countries.
and Digital Content.” Accessed on February 24, Tucker, Bill. 2012. “The Flipped Classroom.” Educa-
2017. http://www.siia.net/Press/SIIA-Estimates-838- tionNext 12 (1): 82–83.
Billion-Dollars-US-Market-for-PreK-12-Education- Unkovic, Cait, Maya Sen, and Kevin M. Quinn. 2016.
al-Software-and-Digital-Content. “Does Encouragement Matter in Improving Gender
Simhan, T. E. Raja. 2011. “Distribution of Free Lap- Imbalances in Technical Fields? Evidence from a
tops to TN Students from Sept 15.” Business Line, Randomized Controlled Trial.” PLOS ONE 11 (4):
June  21.  http://www.thehindubusinessline.com/ e0151714.
van Klaveren, Chris, Sebastiaan Vonk, and Ilja Cor- Believing People Can Change: Implicit Theories of
nelisz. 2017. “The Effect of Adaptive versus Static Personality Shape Stress, Health, and Achievement
Practicing on Student Learning—Evidence from a during Adolescence.” Journal of Personality and
Randomized Field Experiment.” Economics of Edu- Social Psychology 106 (6): 867–84.
cation Review 58: 175–87. Yeager, David Scott, Kali H. Trzesniewski, and Carol
Walton, Gregory M., Christine Logel, Jennifer M. S. Dweck. 2013. “An Implicit Theories of Person-
Peach, Steven J. Spencer, and Mark P. Zanna. 2015. ality Intervention Reduces Adolescent Aggression
“Two Brief Interventions to Mitigate a ‘Chilly Cli- in Response to Victimization and Exclusion.” Child
mate’ Transform Women’s Experience, Relation- Development 84 (3): 970–88.
ships, and Achievement in Engineering.” Journal of Yeager, David S., Carissa Romero, Dave Paunesku,
Educational Psychology 107 (2): 468–85. Christopher S. Hulleman, Barbara Schneider, Cin-
Wang, Haiwen, and Katrina Woodworth. 2011. Eval- tia Hinojosa, et al. 2016. “Using Design Thinking to
uation of Rocketship Education’s Use of DreamBox Improve Psychological Interventions: The Case of
Learning’s Online Mathematics Program. Menlo the Growth Mindset during the Transition to High
Park: SRI International Center for Education Policy. School.” Journal of Educational Psychology 108 (3):
Ward, Adrian F., Kristen Duke, Ayelet Gneezy, and 374–91.
Maarten W. Bos. 2017. “Brain Drain: The Mere Pres- Yeager, David S., and Gregory M. Walton. 2011.
ence of One’s Own Smartphone Reduces Available “Social-Psychological Interventions in Education:
Cognitive Capacity.” Journal of the Association for They’re Not Magic.” Review of Educational Research
Consumer Research 2 (2): 140–54. 81 (2): 267–301.
West, Darrell M., and Jack Karsten. 2016. “Rural and Yeager, David S., Gregory M. Walton, Shannon T.
Urban America Divided by Broadband Access.” Brady, Ezgi N. Akcinar, David Paunesku, Laura
TechTank, July 18. https://www.brookings.edu/blog/ Keane, et al. 2016. “Teaching a Lay Theory before
techtank/2016/07/18/rural-and-urban-america- College Narrows Achievement Gaps at Scale.” Pro-
divided-by-broadband-access/. ceedings of the National Academy of Sciences of the
Wijekumar, Kausalai, et al. 2014. “Multisite Random- United States of America 113 (24): E3324–48.
ized Controlled Trial Examining Intelligent Tutor- Yeomans, Michael, and Justin Reich. 2017. “Planning
ing of Structure Strategy for Fifth-Grade Readers.” Prompts Increase and Forecast Course Completion
Journal of Research on Educational Effectiveness 7 in Massive Open Online Courses.” In Proceedings
(4): 331–57. of the Seventh International Learning Analytics and
Wijekumar, Kausalai Kay, Bonnie J. F. Meyer, and Knowledge Conference, 464–73. New York: Associa-
Puiwa Lei. 2012. “Large-Scale Randomized Con- tion for Computing Machinery.
trolled Trial with 4th Graders Using Intelligent Yong, Ed. 2016. “A Worrying Trend for Psychology’s
Tutoring of the Structure Strategy to Improve Non- ‘Simple Little Tricks.’” Atlantic, September 9. https://
fiction Reading Comprehension.” Educational Tech- www.theatlantic.com/science/archive/2016/09/
nology Research and Development 60 (6): 987–1013. can-simpletricks-mobilise-voters-and-help-
Wozny, Nathan, Cary Balser, and Drew Ives. 2018. students/499109/.
“Evaluating the Flipped Classroom: A Randomized York, Benjamin N., Susanna Loeb, and Christopher
Controlled Trial.” Journal of Economic Education 49 Doss. 2019. “One Step at a Time: The Effects of an
(2): 115–29. Early Literacy Text Messaging Program for Parents
Yeager, David S. 2017. “Social and Emotional Learning of Preschoolers.” Journal of Human Resources 54 (3):
Programs for Adolescents.” Future of Children 27 537–66.
(1): 73–94. Zhang, Dongsong. 2005. “Interactive Multime-
Yeager, David S., et al. 2017. “How Can We Foster dia-Based E-Learning: A Study of Effectiveness.”
Nations of Learners? An Experiment in a National American Journal of Distance Education 19 (3):
Probability Sample.” Unpublished. 149–62.
Yeager, David S., Marlone D. Henderson, David Zhang, Dongsong, Lina Zhou, Robert O. Briggs, and
Paunesku, Gregory M. Walton, Sidney D’Mello, Jay F. Nunamaker, Jr. 2006. “Instructional Video
Brian J. Spitzer, et al. 2014. “Boring but Important: in E-Learning: Assessing the Impact of Interactive
A Self-Transcendent Purpose for Learning Fosters Video on Learning Effectiveness.” Information &
Academic Self-Regulation.” Journal of Personality Management 43 (1): 15–27.
and Social Psychology 107 (4): 559–80. Zheng, Bibin, Mark Warschauer, Chin-Hsi Lin, and Chi
Yeager, David Scott, Rebecca Johnson, Brian James Chang. 2016. “Learning in One-to-One Laptop Envi-
Spitzer, Kali H. Trzesniewski, Joseph Powers, and ronments: A Meta-analysis and Research Synthesis.”
Carol S. Dweck. 2014. “The Far-Reaching Effects of Review of Educational Research 86 (4): 1052–84.

Jel 20191507

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Jel 20191507

Uploaded by

Copyright:

Available Formats

Journal of Economic Literature 2020, 58(4), 897–996

Upgrading Education with Technology:

1. Introduction e­ ducation landscape. Revolutionary advances

T echnological innovation over the past two

—have precipitated a renaissance in edu- ­ roduction function, as technology has done

Figure 1. Technology of Skill Formation in the Education Technology Context

Source: Authors, inspired by Almlund et al. (2011), figure 28.

to produce unbiased estimates of causal like ­technology-based behavioral interven-

Standardized effect Sample Unit of

among young, 17 percentage point California

networking social networking site 5 school districts in

Standardized effect Sample Unit of

Standardized effect Sample Unit of

Fairlie and Grunberg 2014) report on a study on laptop distribution in California

comprehension comprehension and

outperformed 2. 1.03 standard

Educational curriculum and 1.23 standard

­Computer-Assisted Learning (Continued)

Vonk, and of Education CAL program schools

­ rograms in math, considering models from

Traditional Online Courses (Continued)

2018). Relatedly, sending MOOC students 5.2.3 Looking Forward

Learning facilitates social 3.4 percentage University of

reduced but increased

Conference course content

acy activities with ment at school, 2. 0.13–0.19

extensive margins tion in schools

with home visits on retention deviations increase

via text message 2. Null effects on human capital and

­Technology-Enabled Behavioral Interventions (Primary and Secondary) (Continued)

­Technology-Enabled Behavioral Interventions (­Post-secondary Access and Success) (Continued)

(2012) Effectiveness during the summer enrollment increase in likelihood Providence,

that they will enroll in

“one-stop-shop” with all of this increase in

s­ocial-psychology interventions previously an advertisement—the students who might

While educational psychologists have been materials highlighted economic ­self-interest.

Finally, one study (Yeager, Romero, et al. of social-psychology interventions. This

practices in experimental research demand ­ urposes. Psychologists have emphasized the

­Technology-Enabled Behavioral Interventions (­Social Psychology)

courses last 2 years of high

dropping out of high

experiment 2); null for

a ­ distraction with negative consequences. i­ mplementation models. Open questions

decisions, e­ducation administrators often as-lower-income-americans-make-gains-in-tech-

­R eadiness.” https://pdfs.semanticscholar.org/0 Application Assistance and Information in College

Experiment.” Active Learning in Higher Education Perspectives 36 (3): 179–81.

Outcomes? Evidence from a Field Experiment.” “The Effect of Teacher–Family Communication on

North-Holland. Evidence-Based Practices in Online Learning: A

Nature Human Behavior 2: 335–42. economy/policy/distribution-of-free-laptops-to-tn-

You might also like

1. Introduction e ducation landscape. Revolutionary advances

—have precipitated a renaissance in edu- roduction function, as technology has done

to produce unbiased estimates of causal like technology-based behavioral interven-

Computer-Assisted Learning (Continued)

rograms in math, considering models from

Technology-Enabled Behavioral Interventions (Primary and Secondary) (Continued)

Technology-Enabled Behavioral Interventions (Post-secondary Access and Success) (Continued)

social-psychology interventions previously an advertisement—the students who might

While educational psychologists have been materials highlighted economic self-interest.

practices in experimental research demand urposes. Psychologists have emphasized the

Technology-Enabled Behavioral Interventions (Social Psychology)

a distraction with negative consequences. i mplementation models. Open questions

decisions, education administrators often as-lower-income-americans-make-gains-in-tech-

R eadiness.”  https://pdfs.semanticscholar.org/0 Application Assistance and Information in College