Technologys Edge: The Educational

Benefits of Computer-Aided Instruction

Lisa Barrow
Federal Reserve Bank of Chicago
Lisa Markman
Princeton University
Cecilia Rouse
Princeton University and NBER

Student achievement is critical for

individuals and society
U.S. math skills have been
increasing (NAEP), but
proficiency levels remain low.
Math skills may explain a large
portion of wage inequality
(Grogger, 1996; Murnane,
Willet, & Levy, 1995)

School districts are turning to advances in

computers to
Reduce administrative
burden;
Compensate for poor
teacher content knowledge
(especially in districts that
report difficulty recruiting
and retaining teachers,
particularly in math and
science);
Allow more individualized
student attention; students
can progress at own pace.

Evidence on effectiveness of CAI is

surprisingly weak
Poorly defined computer use.
Little use of randomized
controlled study design.

In a 2001 review of the literature, Larry

Cuban (2001, p. 179) concludes,
When it comes to higher teacher and
student productivity and a transformation
of teaching and learning there is little
ambiguity. Both must be tagged as
failures. Computers have been oversold
and underused, at least for now.

Computer-Aided Instruction (CAI)

Treatment:
Typically used in large urban districts;
Software and hardware package designed
to deliver one-on-one instruction;
Software described as meeting National
Council of Teachers of Mathematics
standards;
Software may be configured to state
standards;
Includes classroom management tools.

The Program
5 components per lesson:
Pretest
Review
Lesson
Cumulative review
Comprehensive test
Required to achieve certain
degree of mastery before
advancing;
Teachers can monitor student
progress.

The design of this experiment

Randomly assign 8th and 9th grade
classes in three large urban districts to
be taught using computer-aided algebra
and pre-algebra instruction.
Assess the impact on statewide tests
and tests designed to target algebra and
pre-algebra skills.
Note that the computer use was welldefined.

Research Design:
Within-school random assignment to
CAI or traditional instruction at the
classroom level in three districts
Research design addresses two forms of
selection bias:
Non-random assignment of students to CAI;
Non-random assignment of teachers to CAI.

Table 1: Characteristics of the Districts in

Our Study
Top 100
Districts

3 Districts
Combined

District 1

District 2

District 3

# Students

112,807

62,507

~68,000

~22,000

~97,000

% Female

48.8

49.4

49.7

48.8

49.3

% Native American

0.6

0.5

0.1

0.1

1.0

% Asian

7.1

3.1

1.9

0.8

4.4

% African American

28.1

69.5

93.6

40.3

59.4

% Hispanic

34.1

16.2

1.1

54.3

18.0

Implementation of Random
Assignment
Schools provided us with class
schedules of pre-algebra and
algebra classes;
Given option of eliminating
particular teachers or classes
from the experiment;
Randomization information
provided to schools after
students classes were
scheduled.

Randomized Evaluation of a Computerized Math Curriculum

Current School Schedule

Teacher

Period 1

Period 2

Period 3

Alg 1

Alg 1A

Alg 1

Alg 1

Alg 1A

C
D

Alg 1A
Alg I

Period 4

Alg 1A
Alg I
Alg 1

Alg 1A

Randomized Evaluation of a Computerized Math Curriculum

Current School Schedule

Teacher

Period 1

Period 2

Period 3

Alg 1

Alg 1A

Alg 1

Alg 1

Alg 1A

C
D

Alg 1A
Alg I

Period 4

Alg 1A
Alg I
Alg 1

Alg 1A

Below is a sample schedule that would be returned to each school after random
assignment
Teacher

Period 1

Period 2

Period 3

Regular

Regular

LAB

LAB

LAB

C
D

Regular
Regular

Period 4

Regular
Regular
Regular

LAB

Table 2a: Schools and Students in Study District 1

Relevant
Schools

Schools in
Study

Students
in Study

29,603

8,148

973

% Grade 8

19.3

16.8

40.4

% Grade 9

18.0

18.3

47.2

% Grade 10

15.1

17.8

4.4

% Female

50.5

49.0

52.0

% African American

94.2

97.2

87.8

% Hispanic

1.0

0.8

0.8

% White

2.6

0.4

0.1

% Native American

<0.1

<0.1

0.0

% Asian

2.2

1.6

1.8

Number of Students

% Missing
demographic data

9.6

Table 2b: Schools and Students in Study District 2

Relevant
Schools

Schools in
Study

Students
in Study

5,270

4,476

412

% Grade 8

2.3

0.0

0.0

% Grade 9

38.0

40.0

52.7

% Grade 10

22.0

23.2

31.8

% Female

48.4

48.2

46.7

% African American

43.6

42.0

47.1

% Hispanic

50.1

51.2

44.7

% White

5.5

5.9

6.6

% Native American

0.2

0.1

0.2

% Asian

0.7

0.8

0.5

Number of Students

% Missing
demographic data

0.2

Table 2c: Schools and Students in Study District 3

Relevant
Schools

Schools in
Study

Students
in Study

27,572

3,540

200

% Grade 8

1.4

0.0

3.5

% Grade 9

35.6

40.0

91.5

% Grade 10

23.3

25.1

3.0

% Female

49.9

47.6

47.7

% African American

61.1

92.5

94.5

% Hispanic

15.2

1.2

0.5

% White

18.3

4.0

1.5

% Native American

1.1

0.4

0.0

% Asian

4.5

1.9

3.0

Number of Students

% Missing
demographic data

0.5

Numbers of Schools Classes, Teachers, and

Randomization Pools
Analysis Sample
Combined District 1 District 2 District 3
Number of schools

17

10

Number of randomization
pools

60

31

19

10

Number of classes

141

74

44

23

Number of teachers

57

36

14

Number of students

1,585

973

412

200

Outcomes
Algebra and pre-algebra tests by
Northwest Evaluation Association
(NWEA) to be consistent with state
and district standards;
State-wide administered math tests;
District benchmark tests in prealgebra.

Table 3: Randomization of Treatment and Control Using Full

Sample
Random Assignment
Traditional
Instruction

CAI

p-value of
diff.

Full Sample
Baseline algebra test
score

24.7

24.7

0.494

Female

47.2

47.1

0.637

African American

80.0

83.2

0.561

Hispanic

15.9

13.5

0.195

Class Size

25.8

25.7

0.860

# of Observations

1,133

1,145

Table 3 (cont.): Randomization of Treatment and Control

Using Analysis Sample
Random Assignment
Traditional
Instruction

CAI

p-value of
diff.

Analysis Sample
Baseline algebra test
score

24.7

24.8

0.304

Female

51.1

48.9

0.148

African American

81.9

84.0

0.060

Hispanic

13.8

12.1

0.061

Class Size

25.8

26.2

0.549

# of Observations

785

800

Intent-to-Treat Estimates of the Effect of CAI on

Algebra Achievement
(with and without Teacher Fixed Effects)

Advantages/Disadvantages of
the Intent-to-Treat Effect
Represents the gains a policy
maker might reasonably
expect to observe.
Does not necessarily
represent the effectiveness of
the program.

Amount of Time in the Computer Lab by the

Random Assignment of the Students Class

Intent-to-Treat and Treatment on the Treated

Estimates of the Effect of CAI
(with and without teacher fixed effects)

Intent-to-Treat Estimates in District 1 Using

Different Tests

Intent-to-Treat Estimates in Districts 2 and 3 Using

Different Tests

We might expect to see an

advantage of CAI in
Classes where curriculum
best suited to students;
Larger classes;
Classes with more
disruptive students;
Classes with
heterogeneous students.

Differential Intent to Treat Effects of the

Computerized Instruction on Pre-Algebra and
Algebra Achievement by Class Type

Differential Intent to Treat Effects of CAI on PreAlgebra and Algebra Achievement by Baseline
Test Score Quartile

Differential Intent to Treat Effects of CAI on PreAlgebra and Algebra Achievement by Individual
Attendance Rates

Differential Intent-to-Treat Effects by Class

Characteristic: Attendance, Class Size, and Class
Baseline Test Score S.D.

Overall we find:
On average, students in classes
taught using CAI scored higher on
algebra tests than students in
traditionally-taught classes.
The effects appear larger for
students in larger classes
(especially large, heterogenous
classes), those with worse
attendance rates, and those in
classes with lower average
attendance rates.

Cost-Benefit Analysis
Cost of CAI
# Classes

Total
students

Class
size

# periods

CAI labs
needed

Cost/
student

School A

22

730

33.2

3.0

$218

School B

12

321

26.8

1.5

$245

District 1

74

1736

23.5

9.3

$279
Cost/
student

Cost of reducing class size to 13

# Classes

Total
students

Class
size

# periods

# new
teachers

School A

22

730

33.2

5.7

$329

School B

12

321

26.8

2.1

$278

District 1

74

1736

23.5

9.9

$241