Rock Chapel Elementary School STAR Assessments: Reading and Math

Program Evaluation

Group 7 Scarlett Walker Lacey Phillips Theresa Abernathy

FRIT 8435 Fall 2012 Georgia Southern University

December 3, 2012

Table of Contents

Executive Summary...3

Introduction4

Focus of the Evaluation.8

Brief Overview of Evaluation Plan and Procedures...10

Presentation of Evaluation Results..11

Conclusions and Recommendations13

Appendices15

Executive Summary
The following report provides a detailed analysis and evaluation of two products marketed by Renaissance Place Learning, STAR Reading and Math. The evaluators sought to determine the effectiveness and reliability of these programs to determine the academic success of students at a moderately sized Title I elementary school in a large urban school district. Methods for collecting data include cut scores on the assessments determine into which category students are placed, as well as surveys administered to stakeholders and conducted by the evaluators. Any tables, charts, and or graphs can be found within the appendices of this report. The results of this evaluation found that if implemented, the program would be utilized by general education classroom teachers, exceptional education teachers, and EIP teachers. Furthermore, it is recommended that the contract between Renaissance Place Learning and the school be renewed for the following school year.

Introduction
Defining Response to Intervention is cumbersome and challenging simply because of its multi-tiered structure and support components. What these tiers and components are can leave educators scratching their heads in search for an approach or direction. Simply put, RTI is the practice of providing high-quality instruction and interventions which are matched to student need, frequent progress monitoring to make decisions about changes in instruction or individual goals, and applying child response data to educational decisions. Most importantly RTI involves quality differentiated instruction to meet the needs of all students, frequent monitoring of student progress, and making adjustments to instruction based upon data from that monitoring.

Evaluation Goal
The purpose of this evaluation is to assess the effectiveness of Renaissance Place Learnings STAR assessments, in both reading and math at a Title I elementary school in a large urban school district in the Atlanta, Georgia metropolitan area. Specifically, this evaluation strives to achieve how well these assessments determine the academic growth of students over the course of a school year. Additionally, this evaluation attempts to determine the reliability of the progress monitoring portion of the program. STAR Math and Reading are two of the many products marketed by Renaissance Place Learning. It is a computer-assisted assessment that has been designed to provide accurate, reliable, and valid data on student performance quickly so that educators can make informed decisions about instruction and intervention. STAR Math and Reading are the perfect tools for response to intervention because it provides a wealth of information about students math ability. Teachers are able to identify students who need help. With STAR Math and Reading, you can analyze problems with individual students, grades, or schools. It allows educators to set learning goals and make instructional plans for meeting those goals.

Evaluation Purpose
Renaissance Learning, with the use of their STAR Reading and Math products, claims to be able to quickly and efficiently identify at-risk students. The purpose of this evaluation is to determine the capability of these programs to inform educators about which students are on track to succeed on the year-end assessment and those students that may need additional assistance to reach proficiency on their assessments. Once the evaluation is completed, the information will be used by the local school administrative team to determine the effectiveness of the program on increasing student performance on state administered standardized tests. If the evaluation determines that the program does indeed provide accurate, reliable, and valid data on student performance, then the contract between Renaissance Learning and the school will be renewed for an additional school year. For the purposes of this evaluation, the following stakeholders and their collective interests in the program have been identified. The stakeholders had a critical role in the development of this evaluation. Although the use of the evaluation was determined prior to administration of the program, the stakeholders continue to have considerable influence as to the necessity of the program. Once approved by Title I stakeholders, administrators would then decide exactly how the program would be administered at their school. This would ensure that everyone at the school level would be fully aware of the implementation of the program.

Table 2. Stakeholder Assessment and Engagement Plan

Stakeholder categories

Interests/perspectives

Role in the evaluation

How to engage

Persons involved in program operations

Classroom teachers, exceptional education teachers, EIP teachers, paraprofessional staff

Fear program may be too time consuming See program evaluation as a waste of time and effort

Implementation of program Identifying data sources Collecting RTI data Interpreting findings Disseminating and implementing findings

Monthly RTI meetings Direct roles in conducting evaluation

Persons served or affected by the program

General education students, Exceptional education students, EIP students

May fear not being successful when taking assessments May find assessments too difficult

Providing data

Survey

Intended users of evaluation findings

Principal, assistant principal, academic data coach

To show effectiveness Use findings to enhance the program

Defining information needed from the evaluation Developing and implementing recommendatio ns

Direct role in conducting evaluation

Focus of the Evaluation

The focus of our evaluation pertains to Renaissance Learnings STAR Math and STAR Reading programs. This program was designed to assess student skills in reading, math, and literacy in K-12 education. STAR stands for Standardized Test for the Assessment of Reading. However, it has been dropped to include other content areas, especially math. The goal is to provide teachers with instant feedback to determine academic growth for the individual student and provide a foundation for RTI (Response to Intervention). Students can take multiple standardized tests since the program allows for individualized tests with its immense question bank. Teachers and administrators can then score the tests to receive instant feedback on the progress of the student. Reports can also be generated to ascertain how demographics and other factors may affect a students progressive rate of improvement. Teachers can then incorporate this data in their lesson planning and allow them to focus on the areas that need improvement. STAR assessments run on the web-based Renaissance Place Read Time platform, which provides a single management system, hence allowing districts to centralize all student data. This varies from daily practice to interim assessments (benchmarks, screening, and progressmonitoring), and finally summative assessments (annual state tests). It allows for numerous practice allowances for students to identify strengths and weaknesses on standardized testing which has become the norm in todays classrooms. The STAR assessments are broken into a three-tiered Information Pyramid. Level 1 is the most prominent because of its focus on the daily practice monitoring of student progress. Level 2 revolves around the benchmark testing to assess student progress without the consequences of the state standardized tests. It is a more efficient way to compile data at certain critical points in the curriculum. The pyramid concludes with Level 3 which culminates all the testing practice in the form of the summative state assessment which carries such critical progress data (Appendix B). The STAR assessment aims to provide feedback on where students can expect to be at that point and give detailed analysis on needs-improvement areas. Long-term goals are to aid teachers and administrators with valuable data to create lessons and curriculum to, not only monitor student growth, but to analyze a schools improvement in correlation with national norms. Feedback from individual students, classes, and grade levels are intended to be used to tailor needs assessments in attempt to meet the new,
8

high stakes testing challenges that have been implemented in recent years. The assessments can also be used to predict a students learning curve. These programs have been used at Rock Chapel Middle School for four years. All teachers in grades 1-5 are required to attend training seminars that address software implementation and analysis training. They are also required to keep abreast of any software modifications and attend meetings regarding individual student progress. Each teacher must set progress goals based on the programs assessment and customize lessons to meet the progress needs of his/her students. Our groups evaluation questions centered on two decisive evaluation questions. The first question we focused on was if there have been significant improvements in test gains on standardized test scores since the implementation of this program. The second question revolves around perceptions of the STAR assessment program from the stakeholders (parents, students, and staff) point of view of the implementation of this program.

Brief Overview of Evaluation Plan and Procedures

This evaluation plan centers on the task of collecting qualitative data using surveys and questionnaires (Appendix A). These were delivered to the various stakeholders who include parents, students, teachers, and school administrators. Other factors include comparing test scores in classes that use STAR assessments to a control group, which, in this instance, is a class who did not use the STAR assessments. The desired objective is to monitor test score increases to determine if the tests can be credited with increasing scores and comprehension of the curriculum. The survey administered to affected parties addressed the following issues:

How do the assessments determine student growth over the course of a school year? How do teachers use the assessments to plan for long-term instruction? How reliable is the progress monitoring portion of the program? How reliable is the initial screening of students? Is there a relationship between this program and student success on standardized testing? What are the perceptions of teachers regarding the accuracy of this program?

10

Presentation of Evaluation Results

Upon completion of survey distribution and data analysis, the results of the survey were overwhelming positive. 86.4% of the stakeholders believe strongly that the program is a relevant addition to the curriculum at Rock Chapel Elementary School. Only 5.3% felt the program is not necessary or worthy of financial resources directed at it. The remaining 8.3% were ambivalent on the program. An overwhelming majority of the stakeholders feel that the program has a positive impact on curriculum and instruction.

Test scores have continued to increase since the implantation of the STAR assessment program and the majority of the stakeholders credit this program as being successful.
rd

The test

scores below highlight the most recent CRCT scores for 3 graders at Rock Chapel Elementary School for 2012. Comparisons are made for the students using the STAR assessment program versus the control group who did not use the program.

2012: Students in classes using STAR assessments in reading Students Tested Reading Mean Scale Score % Not meeting standards % Meeting standards % Exceeding standards Total % meeting or exceeding standards 79.7

32

823.14

20.3

54.4

25.3

2012: Students in control group (not participating in STAR assessment program) Students Tested Reading Mean Scale Score % Not meeting standards % Meeting standards % Exceeding standards Total % meeting or exceeding standards 63.3

27

810.43

36.7

46.8

16.5

11

The mean score for the program users in the reading section of the CRCT was over 13 test points higher than the control group while the percentage of students who did not meet state standards was significantly lower for the program users (20.3 % compared to 36.7 %). The startling contrast centers on the total % of students meeting or exceeding standards was 79.7% of the STAR reading assessment users compared to 63.3% of the non-users. The test scores below reflect the testing result for the CRCT results in 3rd grade math. They are even more telling than the reading results. The mean score is 16 points higher in the students using the STAR math assessment and the percentage not meeting standards is notably lower, as well (27.6% compared to 55.8%). Once again the total number meeting or exceeding standards jump from 44.2% for the control group to 72.4% of the STAR assessment participants. These results are drastic but enlightening in the respect that this program gives students the testing practice and experience needed to be successful on standardized testing. The additional data analysis through this testing gives teachers a valuable resource to identify weaknesses to address it for future testing.

2012: Students in classes using STAR assessments in math Students Tested Math Mean Scale Score % Not meeting standards % Meeting standards % Exceeding standards Total % meeting or exceeding standards 72.4

32

816.39

27.6

53.9.4

18.4

2012: Students in control group (not participating in STAR assessment program) Students Tested Math Mean Scale Score % Not meeting standards % Meeting standards % Exceeding standards Total % meeting or exceeding standards 44.2

27

800.75

55.8

32.5

11.7

12

Conclusions and Recommendations

Our first evaluation question was, Has there been a significant improvement in test gains on standardized test scores since the implementation of this program? After our research and interviews with school personnel, we found that there had indeed been improvements in student test scores on the Criterion Referenced Criterion Tests. Renaissance Learning offers STAR Reading and STAR Math student assessments. The companys suite of STAR assessments is used by more than 43,000 schools, and provides screening, progress monitoring, and diagnostic assessment results in as little as 15 minutes per student. Each is completely computeradministered and requires little administration time to achieve reliable data. Assessments may be repeated frequently throughout the year for screening and progress monitoring at no additional cost. According to the National Center on Response to Intervention (NCRTI) the STAR Reading and STAR Math assessments have received the highest-possible scores in all the screening categories. The research proves that not only do teachers and administrators speak highly of Renaissance Learning; it has proven that the implementation of the program does indeed lead to an improvement in student test scores. In regards to the STAR Reading (or Accelerated Reader) program, students should see an increase in their reading level throughout the year (helping to close achievement gaps.) Also, the STAR Math (Accelerated Math) assigns students work based on areas they are weak on when they take the initial STAR Math test at the beginning of the year. The teacher also has the option to go in and make changes to the work assigned, in order to assign work that is only on grade-level, since students will be tested on grade-level Math on standardized tests. Our second evaluation question for our program was What are the perceptions of the STAR assessment program from the stakeholders (parents, students, and staff) point of view of the implementation of this program? After conducting surveys to get feedback from stakeholders, we found that almost all stakeholders had a positive view of the implementation of the Renaissance program. We found that the majority of administrators encourage their teachers to implement these programs into their lessons, and teachers like how the assignments are tailored to fit each individual students needs. Students like that they can work at their own pace,
13

and once they submit an assignment for Accelerated Math, they are free to move on to a new assignment. Regarding Accelerated Reader, students like that their points add up, and many school systems offer rewards for students with a certain number of points. Overall, there are positive perceptions of Renaissance Learnings STAR assessment programs. Our group would definitely recommend teachers implement this program if it is available to them. We would also recommend that schools purchase this program, if they do not already have it. As our research shows, there are mostly positive reviews and attitudes involving the program. Schools are always looking for teachers to differentiate instruction to tailor work to the needs of individual students. This is a very difficult task for teachers (especially those who have 30+ children in a class.) However, this program, through STAR assessments, does a wonderful job of finding the weaknesses of students, and bringing them up to grade level in Reading and Math, two very important subjects on standardized tests.

14

Appendices

Appendix A
Unacceptable Needs Improvement Satisfactory Exemplary

The STAR assessment program for reading and math is beneficial for students. Teachers use the STAR assessment program to construct beneficial lesson plans and curriculum to facilitate maximum understanding for students. Do you feel that there been a significant improvement in test gains on standardized test scores since the implementation of this program?

15

Do you feel like the financial costs involved with this program are reasonable? Is the program capable of prescribing instructional strategies to improve reading and math performance?

16

Appendix B

17

References http://www.redorbit.com/news/technology/1112403784/renaissance_learning_ assessments_receive_highestpossible_ratings_from_the_national_center/

http://public.doe.k12.ga.us/Curriculum-Instruction-andAssessment/Assessment/Pages/CRCT-Statewide-Scores.aspx

http://www.renlearn.com/se/overview.aspx

http://doc.renlearn.com/KMNet/R00535357C40712A.pdf

http://public.doe.k12.ga.us/Curriculum-Instruction-andAssessment/Assessment/Documents/CRCT_2012_school_summaries.zip

18