This action might not be possible to undo. Are you sure you want to continue?
Program Evaluation FRIT 8435 Spring 2012
Ayoko Dogbe Richard Morris Connie Ryals
Study Island Science Page |2
Table of Contents
Executive Summary Introduction to the Report Focus of the Evaluation Overview of Evaluation Plan and Procedures Presentation of Evaluation Results Conclusions and Recommendations Appendices A. Details of Plan, Data Analysis , Interpretation B. Details of Data
3 5 7 9 10 16
Study Island Science Page |3 Executive Summary: This document is an evaluation report for evaluation of a particular program implemented at Bleckley County Middle School in Cochran, GA. The program being evaluated is Study Island Science for 6th grade students. Included in this report are the introduction, the focus, a brief overview of the evaluation plan and procedures, presentation of the evaluation results, and conclusions and recommendations based on the evaluation results. The introduction states the purpose of the evaluation which is to evaluate the effectiveness of Study Island Science on 6th grade students’ post-test scores and to evaluate whether the use of Study Island Science improves motivation and confidence in 6th grade students at Bleckley County Middle School. In this section, the authors make it clear that there are some limitations to the evaluation including the fact that some students with learning disabilities did not complete all of the Study Island Science assignments and it is also noted that students were receiving instruction from their teachers during this time. These two items might affect the post-test scores. In addition to these limitations, it is clear that there were not two groups being evaluated in this project; therefore the evaluators were not able to make a comparison between a group who received tutoring and a group that did not receive tutoring. The focus of the evaluation is the computer program Study Island Science. It is described in more detail in the proceeding sections. There are two questions that will be answered by this evaluation: 1) Does the use of the Study Island Science program generate higher student scores on standardized tests? and 2) Does the use of Study Island Science improve students’ motivation and confidence for the post-test? These two questions were answered through the results of a pre-test/post-test comparison and a student survey. Each of these evaluation tools has been
Study Island Science Page |4 completed by each 6th grade student at Bleckley Middle School. The presentation of the results is laid out in written form and by the use of charts to show the comparison between the pre-test and post-test. Also, the complete survey performed on the students is presented along with the results from that survey. In conclusion, it is made clear the results of the evaluation were satisfactory. The evaluators found that 66% of the students passed the post-test with a 70 or higher which met the criteria for the first question. Also, they found that 75% of the students were showing more confidence and motivation after using Study Island Science. These standards and criteria were used to judge the program and found that the program is successful. However, due to the limitations presented in the report, the evaluators have offered some recommendations for the use of Study Island at Bleckley County Middle School.
Study Island Science Page |5 Introduction: This report was written to show the results of an evaluation of Study Island Science for 6th grade students at Bleckley County Middle School in Cochran, GA. The purpose of the evaluation has two parts. The first part is evaluating the effectiveness of Study Island Science on 6th grade students’ post-test scores. The second part is evaluating whether confidence and motivation of students in 6th grade science improves as a result of using Study Island Science. The audience of this evaluation report is the stakeholders that include the teachers, students and parents, and administration of Bleckley County Middle School. It is important to note that there were some limitations with this evaluation. One limitation is that low-level readers and special education students struggled with completing the assignments in Study Island Science on time; therefore they did not receive as much tutoring and their scores might be lower than expected. Also, the students were receiving instruction in class on a daily basis during the months of Study Island Science usage, so all improvement cannot be credited to Study Island Science. One other limitation was the fact that there was only one group participating in Study Island Science. Since there were not two differing groups of similar competency, we were not able to compare a group that received tutoring and a group that did not receive tutoring. A limitation such as this can make the analysis and recommendation less convincing. However, since we did evaluate the entire sixth grade at Bleckley Middle School, the pre-test and post-test comparison will be enough evidence to make a conclusion about the effectiveness of Study Island Science on sixth grade students at Bleckley County Middle School. This report contains four main sections followed by the appendix. The first section describes the focus of the evaluation including the description of the evaluation followed by the questions used to focus the study and the information needed to complete the evaluation. The
Study Island Science Page |6 next section is a brief overview of the evaluation plan and procedures. The third section is the summary of the evaluation findings and the interpretation of those findings. In the final section, conclusions and recommendations will be made by the evaluation team based on the criteria and standards used to judge the evaluation object. The team will list the strengths and limitations of the evaluation object followed by recommendations.
Study Island Science Page |7 Focus of the Evaluation: The program being evaluated is Study Island Science. Study Island is a computer-based program that is implemented as extra tutoring for students. The program was created by an independent company marketed to k–12 school systems across the United States. Each program of tutoring is geared towards each state’s specific standards for each grade level and course offerings. For example, the standards for the state of Georgia differ per grade level from the standards for the state of New Jersey; therefore Study Island has a different program set up for the state of Georgia. Each computer program starts with a specific log-in name for each student and a pre-test to evaluate what standards need to be addressed for tutoring. After the pre-test, students begin their work in Study Island gaining ribbons to show mastery of a specific standard. The program is designed to help students master all standards of a particular subject area, in this case 6th grade science. The focus of this evaluation will be on the effectiveness of Study Island in this particular subject area at Bleckley County Middle School. At Bleckley County Middle School, the students began using the program Study Island Science in November. Two evaluation questions were used to focus the study of this program. 1. Does the use of Study Island Science generate higher student scores on a Science standardized test? 2. Does the use of Study Island Science improve motivation and confidence for the posttest? For the evaluation to be complete, the evaluators need the following: 1. Study Island pre-test results for each student in 6th grade at Bleckley County Middle School.
Study Island Science Page |8 2. Post-test results for each student in 6th grade at Bleckley County Middle School given on March 26, 2012. 3. Final results of the Motivation/Confidence survey administered to each student in 6th grade at Bleckley County Middle School. This survey was administered through Survey Monkey.
Study Island Science Page |9 Overview of Evaluation Plan and Procedures: This evaluation called for the use of three sets of data. The first set of data was obtained from Study Island and included results on pre-tests taken by 164 6th graders at Bleckley County Middle School near the beginning of the school year and results of their work through March 26, 2012. The second set of data was taken from post-tests given on March 26 during science class. The third set of data was obtained using a survey through the web site www.surveymonkey.com that had to be completed by the students near the end of the school year.
Study Island Science P a g e | 10 Presentation of Evaluation Results All 164 6th grade science students at Bleckley County Middle School participated in the Study Island Science program and were included in our evaluation. These students completed the Study Island pretest around November 11, 2011 and began completing science topics by answering questions to earn blue ribbons. Students were required to earn 16 blue ribbons by April 3, 2012 to complete Study Island. Our data collection ended on March 26, 2012 when a post-test was given to all students. Not all students earned all 16 blue ribbons by March 26. The evaluation was conducted to answer two questions. Question 1 Does the use of the Study Island Science program generate higher student scores on standardized tests? Forty-seven students completed the Study Island assignment by March 26. Of those students, 39 increased their post-test score over their pre-test score. This means 83% increased their test scores. Eight students, or 17%, showed a decrease in post-test scores compared to pretest scores after completing the Study Island assignment and earning all 16 blue ribbons.
Study Island Science P a g e | 11
Students Completing Study Island Assignment On Time and That Effect on Post-Test Scores
50 Number of Students 40 30 20 10 0 Total Increased Gain Score Decreased Gain Score 8 47 39
Out of 164 students, 123 increased their post-test score after using Study Island Science, even though most did not complete it by our data collection date. This means 75% had an increase after using Study Island. 41 students, or 25%, showed a decrease in their post-test scores. Furthermore, 66% of the students passed the post-test with a 70 or above.
Gain Score of Post-Test Over Pre-Test
180 160 140 120 100 80 60 40 20 0 164 123 Number of Students
Total Students Participating
Students Increasing Post-Test Score Gain In Post-Test Scores
Students Decreasing Post-Test Score
Study Island Science P a g e | 12 Data was collected on the amount of time students spent working in Study Island Science. The average gain in test scores for students spending less than 1 hour 30 minutes was 6.57 points while students who spent more than 1 ½ hours increased their scores by 10.27 points. Data was also analyzed to compare post-test success based upon the amount of ribbons earned. Students who earned 9 ribbons or less increased their test scores by 5.74 points, while students with 10 or more ribbons increased scores by 12.09 points.
Gain Score Increase By Time Spent
Points Increased On Test 10 5 0 Less than 1 hour More than 1 hour 30 minutes 30 minutes Time Spent on Study Island 6.57 Points Increased On Test 15 10.27
Gain Score Increase Compared to Number of Ribbons Earned
15 10 5 0 9 or less 10 or more Ribbons Earned 5.74 12.09
Question 2 Does the use of Study Island Science improve students’ motivation and confidence for the post test? To determine the answer for this question, we surveyed all students using an online survey at www.surveymonkey.com. The survey consisted of seven questions that focused on increased motivation and confidence after using Study Island. Students completed the survey on school laptops during science class. The majority of students reported that working on Study Island had helped prepare them for a final test, that it had helped them remember material learned in class, and that earning ribbons or playing games was at least somewhat motivating.
Study Island Science P a g e | 13
Study Island Survey
Question I feel that Study Island Science helped prepare me for the CRCT or final test. Answer Options and Results Very Much: 50% Some: 25% A Little: 23.7% Not At All: 1.3% Very Much: 23.7% Some: 47.4% A Little: 19.7% Not At All: 9.2% Very Much: 34.2% Some: 35.5% A Little: 19.7% Not At All: 10.5% Very Much: 51.3% Some: 26.3% A Little: 19.7% Not At All: 2.6% Very Much: 18.4% Some: 21.1% A Little: 32.9% Not At All: 27.6% Very Much: 36.8% Some: 23.7% A Little: 23.7% Not At All: 15.8% Yes, I earned 16 ribbons by April 3: 55.3% I missed the deadline, but finished before CRCT: 1.3% I missed the deadline, but am finished now: 13.2% No, I have not finished Study Island yet: 30.3%
I became better at science the more I practiced on Study Island.
I increased my science vocabulary the more I practiced on Study Island Science.
Working on Study Island Science helped me remember what I had learned in science class.
Earning blue ribbons made me want to work on Study Island again.
Playing games while working on Study Island made me want to work more.
I completed Study Island Science by the deadline of April 3?
Study Island Science P a g e | 14 Student Opinions About Study Island
60 50 50 Percentages of Student Answers 47 51 55
36 34 2524 26 20 20 20
37 33 28 2424 21 18 15 13 30
20 9 1
Practice Helped Me Very Much Some A Little Not At All Increased Vocab Very Much Some A Little Not At All Helped Prepare Very Much Some A Little Not At All
Remember from Class Very Much Some A Little Not At All Blue Ribbons encouraged Very Much Some A Little Not At All Games encouraged Very Much Some A Little Not At All
Completed On Time Yes No, but before CRCT No, finshed after CRCT Not yet finished
Survey Questions and Results
This evaluation focused on whether or not Study Island Science is beneficial for students in 6th grade at Bleckley County Middle School. This will help teachers determine whether to continue using Study Island and how best to use it with their students. The results of the evaluation are as follows: Question 1 Does the use of the Study Island Science program generate higher student scores on standardized tests? Based on our evaluation of the data, using Study Island Science does generate higher student scores on standardized tests. Seventy-five percent of students demonstrated a gain in
Study Island Science P a g e | 15 test scores after using Study Island. Students who earned ten or more ribbons increased their post-test scores by double students that earned nine or less ribbons. Students who spent more time working on Study Island saw a larger increase gain scores. In conclusion, 66% of the students passed the post-test with a 70 or above. Question 2 Does the use of Study Island Science improve students’ motivation and confidence for the post-test? Study Island Science improves motivation and student confidence for final tests in science. Seventy-five percent of students indicated they felt Study Island helped them on the final test. Seventy-seven percent indicated using Study Island helped them remember content learned in class. Only one percent of students surveyed felt Study Island did not help prepare them for final tests.
Study Island Science P a g e | 16 Conclusions and recommendations: Criteria and standards must be established to determine if a program is meeting goals. For this evaluation, there are two questions, each with one criteria and one standard. To determine if using Study Island Science generates higher student scores on standardized tests, a post-test will be given near the end of instruction time and a score of 70 or above by 65% of the students will be considered successful. To determine if using Study Island Science improves student motivation and confidence for the post-test, a student survey will be given and a result of 70% of students reporting feeling motivated and confident in taking the post test will be considered successful. This evaluation discovered strengths and weaknesses with the use of Study Island Science at Bleckley County Middle School for 6th grade science students. Study Island is an internet-based program that allows students to review material from class and practice with test questions that are structured like standardized test questions. Students benefit from the opportunity to see and practice answering questions that are similar to the post-test questions. Study Island motivates students by allowing them to play games and earn blue ribbons. Sixty-six percent of students (109 of 164) scored 70 or higher on the post-test. While Study Island Science is a useful program, some limitations were discovered. All students participated in the Study Island Science assignment but low-level readers and special education students struggled with completing the assignments on time. Because reading the lessons and questions was harder for these students and most were not provided with reading assistance throughout the program, they did not receive an equal amount of tutoring to other students and their scores were lower. These lower scores caused them to work longer on each
Study Island Science P a g e | 17 topic to earn a blue ribbon, which also may have resulted in them becoming frustrated with the assignment. Also, all students were also receiving instruction in science class each day, so all gains cannot be attributed to work on Study Island. For this evaluation to have been more effective, all students should have completed Study Island by the March 26 data cutoff date. Finally, there was not a group of 6th grade science students that did not participate in the Study Island Science assignment. Teachers at Bleckley County Middle School felt the tutoring and practice available from Study Island was too significant to withhold from any group of students. Therefore, there is not a control group to compare improvements against in the post-test score gains. After careful review of the data obtained in this evaluation, we recommend Study Island continue to be utilized by all science teachers at Bleckley County Middle School. Teachers should work to find ways to provide reading assistance to struggling students. Also, teachers should plan for more time in class to work on Study Island Science to ensure all students complete Study Island by the deadline. By implementing these changes, all 6th grade science students at Bleckley County Middle School will receive the maximum benefits of the Study Island Science program.
Study Island Science P a g e | 18 Appendices: A. Description of Evaluation Plan/Design, Instruments, and Data Analysis & Interpretation Teachers and Administrators of Bleckley County Middle School in Cochran, GA needed to know if the use of Study Island in their 6th grade science classes was helping their students obtain higher scores on standardized tests. They have requested an evaluation be conducted and results shared with all teachers, administrators, parents, and students of the school. The evaluation of Study Island Science focused on two questions. (1) Does the use of the Study Island Science program generate higher student scores on standardized tests? (2) Does the use of Study Island Science improve motivation and confidence for the post-test? Our team designed an evaluation to occur in an ongoing school year where Study Island had already been implemented with the entire 6th grade student body. An evaluation begun at a different time could utilize two groups of students, one receiving Study Island work and one that did not. However, due to the timing of this evaluation, that would not work in this situation. This evaluation would require a pre-test, a post-test, and a student survey. The pre-test would indicate where students were in their scientific knowledge at the beginning of the school year. The pre-test questions were randomly selected for each student by Study Island from its 6th Grade Georgia Science question bank. Any level of performance would be acceptable at this point. The post-test was obtained by giving a pencil and paper test to students on March 26. This date was selected because all necessary instruction would be complete and teachers would begin reviewing all material for CRCT. The test was taken from other sources, but included several questions representing each standard. The student
Study Island Science P a g e | 19 survey was designed to measure how students felt about having used Study Island. The intent of the questions was to learn if students felt using Study Island had helped them to learn or remember science content and if students felt motivated to review and study the science topics. This survey was administered to all students during their science class just days after taking the CRCT for science. To determine if using Study Island is effective, our team set two standards to be met. Success would be indicated by students scoring 70 or higher on the post-test and by a minimum of seventy percent of students indicating they felt motivated and more confident in taking the post-test after using Study Island. Our team would have liked to also include actual test scores from the 2012 CRCT administered just after the post-test, but that data is not yet available. If that data were available, a minimum score of 800 for each student would also have indicated success. Sixth grade teachers from Bleckley County Middle School were responsible for administering all tests and surveys, collecting all test and survey information, and providing all data to the evaluation team. The teachers converted all test data into Excel spreadsheets for evaluators to analyze. The evaluation team studied the data, focusing on the gain in test scores from the pre-test administered early in the school year to the post-test administered on March 26. Several different paths were considered for success, including the amount of time students worked on Study Island, whether or not they earned all 16 ribbons, and if the amount of ribbons earned affected their gain score. Evaluators feel confident the results are informative and useful. If the opportunity arises to conduct another evaluation, timing should be extended to allow for the inclusion of the
Study Island Science P a g e | 20 Study Island post-test and CRCT science scores. This additional information would provide more data to support future decisions.
Study Island Science P a g e | 21 B. Detailed Analyses of Data
Summary of Analysis of Study Island Science Data Bleckley County Middle School 2011-2012 Avg Gain Score Pre to Benchmark 3 for 9 or less ribbons Avg Gain Score Pre to Benchmark 3 for 10 or more ribbons Avg Gain Score Pre to Coach Post for 9 or less ribbons Avg Gain Score Pre to Coach Post for 10 or more ribbons Avg Gain Score Pre to Coach Post 11 or less ribbons Avg Gain Score Pre to Coach Post 12 or more ribbons Avg Gain from Pre to Coach Post for less than 1 hr 30 min spent Avg Gain from Pre to Coach Post for more than 1 hr 30 min spent Number of Students Increased Test Scores After Using Study Island Number of Students Decreased Test Scores After Using Study Island Total Students Increase of Test Scores After Using Study Island Decrease of Test Scores After Using Study Island Number of Students Increased Test Scores After Completing Study Island Number of Students Decreases Test Scores After Completing Study Island Total Students Completing Study Island Percentage Increased Test Scores After Completing Study Island Percentage Did Not Increase Test Scores After Completing Study Island Number of Students Passing Pre-test Percentage of Students Passing Pre-test Number of Students Passing Coach Post Test Percentage of Students Passing Post Test 14.47 18.91 5.74 12.09 7.04 12.26 6.57 10.27 123 41 164 75% 25% 39 8 47 83% 17% 0 0 109 66%
For more detailed information on the individual test scores and analysis performed, view the following excel workbook.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.