Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT

PROGRAMS 1

Program Evaluation:

An Evaluation of Computer-Assisted Reading Management Programs

Elise Harp
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 2

Table of Contents

Introduction

Evaluation Methodology

Participants

Data Sources and Instruments

Procedures

References

Appendices
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 3

Introduction

The organization that will be involved in this program evaluation will be Public Safety

Academy of San Bernardino. Public Safety Academy is a charter school for students that are

interested in a career in police, fire, and military service. There are approximately four-hundred

students in grades six through twelve. Before the implementation of Common Core, Public

Safety Academy had been in Program Improvement (PI) status as a result of multiple years with

low Academic Performance Index (API) scores. In April of 2015, the board of directors

approved a budget to purchase enough Chromebook computers for each student to use while at

school in core subjects. This concept is commonly referred to as being 1:1. This decision was

a direct result of the Smarter Balanced Assessment Consortium (SBAC) using computers for

high-stakes testing. The board of directors expressed that the computers also needed to be used

to increase students reading and mathematics scores from when the school was in PI. In 2016,

Public Safety Academy went through the WASC (Western Association of Schools and Colleges)

Accreditation process. One of the outcomes of that process was for the school to implement a

standardized benchmark assessment to measure student growth.

The intended purpose of this program evaluation is to determine the effectiveness of

using computer-assisted reading management programs. Program evaluation will determine if

the outcomes are beneficial to student reading levels (Dunsworth & Billings, 2012, p. 1).

Program evaluation will be using results of Accelerated Reader, STAR Assessments, and

California Assessment of Student Performance and Progress (CAASPP) results. These

computer-assisted reading programs cost a significant amount of money (Renaissance Learners

Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 4

STAR and Accelerated Reader cost $15,115.15 per year), and it is important to the organization

to determine its return on investment.

Program Background

The participating school had not used a reading management system, computer-based or

otherwise, until the 2015 school year once Chromebooks were purchased. Teachers originally

began using Front Row Education to supplement math curriculum so that students could

reinforce concepts that were being learned in class. In 2016, Front Row Education came out

with an English Language Arts (ELA) curriculum. Front Row Education is a free online

platform that uses a unique algorithm to provide students with material that is based on their skill

level. Students must take a diagnostic test in order to determine this algorithm. As a result of

the high useage of the free version of this resource, Public Safety Academy purchased licenses

(Front Row Education costs $3,487.50 per year) that allow teachers to access all content, assign

an unlimited amount of articles, and view reports on student progress. Teachers do not have the

ability to change goals, and the material that students read comes from articles provided by the

website.

To fulfill the need for a standardized benchmarks, Public Safety Academy also purchased

licences for Renaissance Learner STAR Assessment benchmark system. Reading levels and

levels in mathematics are assessed using the STAR Assessment program. The reading

assessment uses a combination of reading comprehension passages and the cloze method which

is a sentence that omits a key word and the student has to choose the best word. These types of

questions assess students current levels of understanding (Meador, 2014). The test contains

about thirty questions that are adaptive to each students responses. Teachers are given a choice
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 5

of several reports that show the students assessed grade levels, reading levels, and percentile

ranks which are compared to the overall performance of other students in the school. Students

may take the assessment as many times as they want. Multiple attempts are encouraged because

the program will be able to hone in on the students correct levels (Meador, 2014). The STAR

Math Assessment uses a similar adaptive format that tests students understanding of the eleven

domains of mathematics. Scores are provided in a variety of reports that teachers can access.

These reports provide the same information that the STAR Reading Assessment provides.

STAR Reading Assessment and STAR Math Assessment are both supplemented by

Accelerated Reader and Accelerated Math. Both programs have been shown to be very popular

in elementary schools because they provide incentives that motivate students to increase their

levels of understanding throughout the school year (Meador, 2014). Teachers can set attainable

goals for students to help motivate them. In regards to AR, students will read their appropriate

level book and then they can take a quiz online. Passing the quiz essentially says that the student

comprehends the text, and they are given points based on the complexity of the book (Meador,

2014). For the purpose of this program evaluation, the organization will only be evaluating

STAR Reading Assessment and Accelerated Reader. Reading skills are the focus of the

participating organization for the 2016-2017 school year. This was decided as a result of a staff

Professional Learning Community (PLC) meeting that math, reading, and writing CAASPP

scores are low as a result of student reading levels. These low reading levels are what prompted

the investigation of an intervention strategy for students, hence the need to evaluate the

effectiveness of computer-assisted reading management programs.

Evaluation Questions
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 6

The following questions were developed in order to evaluate the effectiveness of using

computer assisted reading program (Dunsworth & Billings, 2012, p. 14).

A. To what extent do teachers implement computer-assisted reading programs with fidelity,

as measured by usage of time, materials, expectations, and pacing?

B. Are there differences in achievement based on demographically-defined subpopulations

of students, such as English Language Learners (ELLs), grade levels, and/or gender?

C. Do students reading levels increase as a result of using a computer assisted reading

program?

D. Have students attitudes changed towards reading as a result of using computer-assisted

reading programs?

Literature Review

Program Implementation. In order for children to be successful in school, in college,

and in their careers, they need to be able to read. Reading is an important skill to have when

reading informational and fictional texts, but it also serves an important function when learning

mathematics and science. Common core states that when students graduate from high school,

they must be career and college ready. Part of that readiness is being fluent in 21st century

skills. In todays society we are in constant contact with computers and smartphones, with a

virtual universe and all of its information at the palm of our hands. Research indicates that

computer technology can help support learning, and that it is especially useful in developing the

higher-order skills of critical thinking, analysis, and scientific inquiry. In several studies

conducted by Roschelle, Pea, Hoadley, Gordin & Means (2002), computer-based technologies
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 7

proved to increase student understanding in reading, vocabulary, and increased student interest in

the concepts being evaluated.

However, just having computers in the classroom does not mean that they will be used

properly. Some computer applications have been shown to be more successful than others, and

many factors influence how well even the most promising applications are implemented

(Roschelle, Pea, Hoadley, Gordin & Means, 2002). One of those factors is how the classroom

teacher implements the computer-based program. In order for computer-assisted reading

management programs to be successful, teachers must be trained appropriately on the programs

that are being implemented. Teachers and schools who succeed in using technology and

computer-assisted reading management programs often make major changes in their pedagogy

and in the curriculum they use. High quality implementation of these programs prove to result in

greater gains in independent reading (Hansen, Collins, & Warschauer, 2009, p. 64). This means

that teachers take an active role in promoting independent reading by assisting the book selecting

process and celebrating when students show growth. However, making such changes is difficult

without appropriate support and commitment from school administration (Roschelle, Pea,

Hoadley, Gordin & Means, 2002, p. 91-93).

In addition to teacher training and administrative support, computer-assisted reading

management programs can be costly in terms of time and startup costs. Books, leveling

materials, and software licences need to be purchased in order to properly implement these

programs (Hansen, Collins, & Warschauer, 2009, p. 64). It is important for sites that decide to

use computer-assisted reading management programs to understand the investment that they are,

and create a sustainable plan for a successful implementation.

Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 8

Reading Management Programs. Reading management programs have been around

since the early 1980s. They have always maintained a simple goal: read a book and take a quiz

to see if the reader has comprehended what they have read. When students receive direct reading

instruction, often they are taught constrained skills, which are alphabetic and phonological skills.

These skills are necessary, but do not promote unconstrained skills like vocabulary development,

reading motivation, and reading strategies (Hansen, Collins, & Warschauer, 2009, p. 59). To

promote these skills, schools have promoted computer-assisted reading management programs

that encourage sustained reading time. The important part of these programs is that students are

selecting books that are related to their own reading level and using them during independent

reading. Studies have shown that when students use computer-assisted reading management

programs, they are more likely to read outside of school and are more likely to hold themselves

accountable in their own reading comprehension (Hansen, Collins, & Warschauer, 2009, p. 62).

The Impact. There is certainly a positive impact when schools use computer-assisted

reading management programs. Even if the reason is because sites that do use these programs

tend to allot more time for independent reading, the fact is that students increase the quantity and

the quality in which they are reading. In a study done in 2002, students in subgroups that were

at-risk for low academic achievement showed reading improvement using the computer-assisted

reading management program Accelerated Reader (AR). In the same study, English Language

Learners were able to maintain or increase their reading levels by increasing the amount of books

that they were reading (Hansen, Collins, & Warschauer, 2009, p. 66). These studies, however,

did show that computer-assisted reading management programs show more results with students

in grades three through five. Students in grades six and above did not produce as promising of
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 9

results (Hansen, Collins, & Warschauer, 2009, p. 67). This is possibly because the influence of

reading has not been ingrained into the student's attitude towards reading and comprehension.

Attitudes of Readers. One of the major benefits of computer-assisted reading

management programs is that they can change the way students perceive reading. A lot of the

way that teachers, parents, and administrators promote these programs also helps because they

see them as tools to assist learners. However, these benefits do not affect all subgroups. In

studies completed in 2005, it was found that girls tend to like computer-assisted reading

management programs more than boys. It was also evident that students that meet their expected

reading goal excel at reading and comprehension, but students that fall below their goal tend to

feel defeated and make little progress in their reading levels . Attitude has as much to do with

reading as understanding the text does (Hansen, Collins, & Warschauer, 2009, p. 68).

Accelerated Reader. The computer-based reading management system Accelerated

Reader (AR) will be examined closely in this program evaluation and literature review because

this is the program that the participants are using to track student reading levels. Accelerated

Reader (AR) was created by Judith Paul in 1984 so that she could motivate her own children and

keep track of their reading. Her husband then wrote a computer program for her reading system

and Advantage Learning Systems purchased the idea in 1985 (Stefl-Mabry, 2005, p. 1).

Accelerated Reader (AR) became popular by word of mouth because it claimed that it provides

an easier way to use technology tools to assess students reading levels, keep track of student

progress, and determine whether or not the student has read the AR books based on short

comprehension quizzes (Stefl-Mabry, 2005, p. 11). Teachers that use computer-based reading

management systems like Accelerated Reader (AR), use it because they believe that the more
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 10

these programs are used, the more they encourage students to read. AR claims that the more

students read, the more likely they are to fluently read complex texts and become better equipped

to have the 21st century skills that Common Core demands.

In 1999, a study was done on middle school campus for a one year period. Students did

show improvement in vocabulary and comprehension, but did not make any significant gains

during this time period. Studies have shown that Accelerated Reader (AR) is more successful

when it has been on campus for a longer period of time (Topping & Paul, 1999, p. 224). This

study also showed gains in attitude in regards to reading while using Accelerated Reader (AR)

(Stefl-Mabry, 2005, p. 3). Regardless of the implementation process of each site, studies have

proven that the more reading practice students receive, shows a correlation to higher

performance on standardized testing (Topping & Paul, 1999, p. 223).

One of the selling points of Accelerated Reader is to build lifelong readers. Studies have

been conducted to provide evidence as to whether students exposed to Accelerated Reader (AR)

in elementary school will be more likely to continue recreational reading in middle school. In a

study conducted in 2002, there were no significant differences between those students who had

used AR and those who had not (Pavonetti, Brimmer, & Cipielewski, 2002, p. 303). A more

recent study was conducted in 2014 that was commissioned by Renaissance Learner, the parent

company of Accelerated Reader (AR). Over 29,422 children and young people aged eight to

sixteen participated in the survey, which was conducted in November through December of

2013. This survey asked the participants if they enjoy reading, how often they read outside class

and for how long, what type of materials they read outside class, how many books they read in a

month and what they think about reading (Clark, 2014, p. 9). In this study, students who use AR
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 11

tend to have slightly more positive attitudes towards reading than their peers who do not use AR.

The students that were participants in this study indicated that they believed that they will

become better readers if they put in effort to do so. Out of these participants, more AR users

believe that they will get better jobs as a result of being better readers (Clark, 2014, p. 13). Just

the act of reading seems to be the most powerful motivator for encouraging additional reading.

Available studies show clear increases of students reading abilities when they have used reading

management programs. These studies also show an increase in the amount of free reading they

do outside of school and the effect appears to last years after the program ends (Stefl-Mabry,

2005, p. 9).

The Counter-Argument. In all of the research reviewed for this evaluation, each one

notes that there is a lack of research of computer-based reading management programs. This is

surprising since these programs have been around for at least thirty-five years. In the time that

the computer-assisted reading program Accelerated Reader (AR) has been available, very little

has been written about the programs ability to increase a students reading levels. Much of the

literature that is available regarding Accelerated Reader (AR) was written in the 1980s and

1990s and is considered to be outdated. Of the literature and studies that are available, most

suggest that the use of computer-based reading management programs do not show evidence of

reading levels or comprehension increasing, which include gender subgroups (Nichols, 2013).

There is also no significant evidence that supports that these programs support lifelong reading

habits displayed by students (Pavonetti, Brimmer, & Cipielewski, 2002, p. 307). All of the

literature that has been reviewed for this program evaluation calls for additional research into the

impact of measurable achievement correlating with reading management programs.

Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 12

What these studies do show is that students that have more resources have a greater

chance of being successful readers. Out of the four aspects of AR, which are access to books,

time devoted to reading, tests, and rewards; only the first two are supported by research.

Educators first priority should be to make sure that high-interest reading material is easily

available to students, and that students have time to read and a place to read (Krashen, 2003, p.

22). There are many books that are not supported within computer-assisted reading management

programs, and many times this discourages students from reading these kinds of books. Another

discouraging factor is that some of the programs will not allow students to take quizzes that are

too high above their reading level (Hansen, Collins, & Warschauer, 2009, p. 69-70).

With such limited evidence, many schools and districts are hesitant to front the huge

start-up cost of purchasing reading management programs like Accelerated Reader (AR) and

supplemental materials such as hardware, books, and time needed to create a leveled library

(Pavonetti, Brimmer, & Cipielewski, 2002, p. 301). Inconsistency with program

implementation also causes problems with these types of programs, which usually leads to the

program not being used for long enough for educators to make a difference. (Stefl-Mabry, 2005,

p. 9).

Evaluation Methodology

Participants

Students. This study focuses on students that meet the criteria of being enrolled in 6th,

7th, or 8th grade Language Arts classes. Currently there are forty-six 6th graders, seventy 7th

graders and seventy-one 8th graders enrolled in the organization. Thirty-six point six percent

(36.6%) of these three grades are female, and sixty-three point three percent (63.3%) are male.
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 13

Eighty-six percent (86%) of these students are considered Hispanic/Latino. Sixty-eight (68%) of

these students qualify for free and reduced lunch. As of December 2016, the average grade point

average (GPA) of 6th grade was 2.29, 7th grade was 2.31, and 8th grade was 2.77.

Teachers. Four middle school Language Arts teachers will be considered participants.

One of these teachers also teaches the English Language Development (ELD) course for students

that have not exited out of the California English Language Development Test (CELDT). The

criteria for selection is teaching Language Arts at the middle school level. All participants are

new or fairly inexperienced with both programs that are being evaluated. All teachers involved

in program evaluation have been educators for over five years, and have worked with grades

K-8. Two of the participants have partial masters degrees, one has a PPS credential and masters

degree, and one is the site Instructional Coach, who has a single subject credential in English.

Data Sources and Instruments

The following questions shall be answered through various data collection strategies

including classroom observations, demographic information collection, and surveys given to

participating students and teachers.

School Demographic Data. All data related to students is entered into PowerSchool.

This includes attendance, grades, CAASPP information, health information, contact information,

and behavior. To supplement PowerSchool, the organization has a contract with Charter School

Management Corporation (CSMC) which provides back office services. They host a website

called CharterVision, which displays all student-related data in a user friendly format. (See

Appendix A.) CharterVision also provides the organization with easy-to-use snapshots of

financial information. For the purpose of this program evaluation, the data that will be acquired
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 14

from CharterVision will be related to ethnicity, gender, age, and grade levels. The organization

also uses a shared Google Drive folder that provides information related to Grade Point Averages

(GPAs) in which all teachers have access to and create goals based on this data. This

information will be used as a resource to compare with computer-assisted reading management

programs.

Computer-Assisted Reading Management Program Data. For the purpose of this

program evaluation, Accelerated Reader and STAR data will be used. Renaissance Learning

provides a variety of reports for stakeholders to access in order to identify the students current

levels of understanding. Diagnostic reports generated by STAR Reading Assessments provide

the current grade level, reading level, and percentile rank compared to students at the school.

The growth report shows if and how the student grew or regressed based on a specific time frame

in which the student took the assessment. These reports will be used in conjunction with

demographic data to determine the rate of growth or regression. Accelerated Reader generates

similar reports to show the students reading choices and whether or not they are truly

understanding the text based on their quiz scores.

Surveys. The purpose of giving the participants a survey is to understand when and how

AR will be used. For participating teachers, an anonymous Google Forms survey of about 20

questions will be given. Questions will be close-ended and vary from multiple choice and Likert

scale. A Likert scale is a rating system that measures responses of the participants. There will

also be several open-ended questions. Topics will include the respondents teaching experience

and their familiarity of Renaissance Learning programs. There will also be questions regarding

how the participant plans to implement computer-assisted reading management programs into
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 15

their classroom instruction and whether or not they plan on making it part of their students

grade. The evaluation team will also be looking for fidelity of implementation from the survey

results.

A survey will also be given to the students that will be using Accelerated Reader (AR).

The survey will include questions related to students experiences using AR at their previous

schools, their reading interests, and how they plan to use AR as a tool for success. The survey

will be submitted to students using Google Forms so that responses can be recorded to ensure

that the methodology is valid (Dunsworth & Billings, 2012, p. 73). Survey questions will be

open-ended and close-ended and will also include multiple choice and Likert scale questions.

Classroom Observations. Observations will need to be conducted in order to

understand the level at which teachers will be implementing computer-assisted reading

management programs. Each teacher has a unique pedagogy that affects their classroom

management and their students achievement. It is important to understand the capacity at which

computer-assisted reading management programs will be implemented in each classroom. An

observation rubric will be used while observing computer-assisted reading management

programs in action in each participating classroom. Some observations will be scheduled and

others will be impromptu in order to make data more valid. Students will also be observed to

gain clarity for their understanding of AR and how and when they use it to help increase their

levels of understanding in all core subjects. Observations will be conducted by the academic

coach and teachers involved in the evaluation team at the beginning of computer-assisted reading

management programs implementation and as well as at the end of the school year during the

time that computer-assisted reading management programs is being used.

Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 16

Procedures

In August of 2016, students were assessed using Renaissance Learner STAR reading and

math assessments. This program evaluation will be using data collected from STAR reading

assessments only, as the purpose of evaluation is to evaluate computer-assisted reading

management programs. This initial assessment was used as the basis of student reading levels.

STAR reading assessments will continue quarterly throughout the 2016-2017 academic year. In

October of 2016, Accelerated Reader and Frontrowed.com were implemented in all middle

school Language Arts and ELD classes. The first method of collection will come from the

schools back office student information system (SIS), CharterVision. This SIS has a

compilation of student data that pertains to sub-group information such as grades, CAASPP

scores, and demographics.

Classroom observations will be conducted starting in March 2017. These observations

have been approved by all participating teachers and the principal. There will be at least one

formally schedule observation of each of the participating classrooms. Students will understand

why the research is present in the classroom by verbal communications. At least one observation

in each of the participating classrooms will be unannounced in order to determine how the

computer-assisted reading management programs are being implemented.

The final method of collection will be in the form of surveys. Student surveys will be

given in March 2017 via Google Forms. Students will be notified that participating is not

required and will in no way affect their grade. Anonymous teacher surveys will also be

distributed in March 2017 and will also be taken using Google Forms. Teachers will have been
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 17

given background information pertaining to program evaluation. The parent survey will be

verbally administered in April 2017. This survey will take place during the quarterly coffee

with the principal session in which the principal discusses school matters with parents that wish

to attend. This verbal parent survey will be anonymous and parents will be able to opt out if they

do not wish to participate.

References

Biggers, D. (2001). The Argument Against Accelerated Reader. Journal of Adolescent & Adult

Literacy, 45(1), 72-75.

Clark, C. (2014). Accelerated Reader and Young People's Reading: Findings from the National

Literacy Trust's 2012 Annual Literacy Survey on Reading Enjoyment, Reading Behaviour

outside Class and Reading Attitudes. National Literacy Trust.

Dunsworth, M., & Billings, D. L. (2012). Effective Program Evaluation. Bloomington, IN:

Solution Tree Press.

Hansen, L., Collins, P., & Warschauer, M. (2009). Reading Management Programs: A Review of

the Research. Journal of Literacy and Technology,10(3), 55-80. Retrieved March 7, 2017.

Krashen, S. (2003). The (lack of) experimental evidence supporting the use of Accelerated

Reader. Journal of Childrens Literature, 29(2), 9-16.

Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 18

Meador, D. (2014). Is STAR Reading Right for You? Retrieved October 12, 2016, from

http://teaching.about.com/od/tech/fr/Star-Reading.htm

Nichols, J. S. (2013). ACCELERATED READER AND ITS EFFECT ON FIFTH-GRADE

STUDENTS READING COMPREHENSION (Doctoral dissertation, Liberty University).

Pavonetti, L. M., Brimmer, K. M., & Cipielewski, J. F. (2002). Accelerated Reader: What are the

lasting effects on the reading habits of middle school students exposed to Accelerated Reader in

elementary grades?. Journal of Adolescent & Adult Literacy, 46(4), 300-311.

Renaissance Learning. (2016). Parents Guide to Accelerated Reader. Retrieved October 12,

2016, from https://doc.renlearn.com/KMNet/R003985016GG79F2.pdf

Roschelle, J. M., Pea, R. D., Hoadley, C. M., Gordin, D. N., & Means, B. M. (2000). Changing

how and what children learn in school with computer-based technologies. The future of children,

76-101.

Stefl-Mabry, J. (2005). Accelerated Reading: Silent Sustained Reading Camouflaged in a

Computer Program? Research Journal of the American Association of School Librarians,8, 1-15.

Retrieved February 22, 2017, from

http://www.ala.org/aasl/sites/ala.org.aasl/files/content/aaslpubsandjournals/slr/vol8/SLMR_Acce

leratedReading_V8.pdf
Running head: AN EVALUATION OF COMPUTER ASSISTED READING MANAGEMENT
PROGRAMS 19

Topping J., & Paul, T. (1999). Computer-assisted assessment of practice at reading: A large scale

survey using Accelerated Reader data. Reading & Writing Quarterly, 15(3), 213-231.

Appendices
Appendix A- CharterVision