CPS Pulse Response System Evaluation

Janice Bezanson
EdTech 505-4172
December 5, 2010

Table of Contents
Part 1. Learning Reflection ............................................................................................3 Part 2. Executive Summary ...........................................................................................4 Part 3. Purpose of the Evaluation .................................................................................5
Part 3a. What was the purpose of doing this particular evaluation? ............................................................. 5 Part 3b. What were the central questions to be answered through the evaluation? ...................................... 5 Part 3c. Who would be most impacted by the results of the evaluation? ..................................................... 5

Part 4. Background Information....................................................................................6

Part 4a. What was the origin of the product? ................................................................................................ 6 Part 4b. What are the goals of the product? .................................................................................................. 6 Part 4c. What did previous products look like? ............................................................................................ 7 Part 4d. Who are the people involved with the product? .............................................................................. 7 Part 4e. What are the characteristics of the product? .................................................................................... 7

Part 5. Description of Evaluation Design .....................................................................9 Part 6. Results ..............................................................................................................11

Part 6a. Teacher Results.............................................................................................................................. 11 Part 6b. Student Results .............................................................................................................................. 13

Part 7. Discussion of the Results................................................................................18

Part 7a. How good were the results of the product? ................................................................................... 18 Part 7b. What are the costs associated with the product? ........................................................................... 19 Part 7c. What are the benefits associated with the product? ...................................................................... 19

Part 8. Conclusion & Recommendations ...................................................................19

Part 8a. Immediate Conclusions.................................................................................................................. 19 Part 8b. Long-Range Conclusions .............................................................................................................. 20 Part 8c. Evaluation Insights ........................................................................................................................ 20

Part 9. References ........................................................................................................20 Part 10. Appendices .....................................................................................................21

Part 1. Learning Reflection

I learned early in this semester that evaluation is a systematic process filled with intricacies that cannot be overlooked. There are many different types of evaluations, each one requiring different steps, processes and types of data that need to be collected, reviewed and analyzed. Through this process, an evaluator will determine whether or not the pre-determined objectives have been met. When I think of an evaluation, a picture of a juggler comes to mind. An evaluation is similar to a balancing act. A juggler calculates each move carefully; he moves fluidly and with purpose to a successful conclusion. This is similar to what occurs during an evaluation. An evaluator must create thoughtful timelines to ensure that each necessary step is not missed or forgotten. It is through this careful planning and attention to detail, that allows an evaluation to be successful. An evaluation can be small, like the one prepared for this assignment, or a large-scale undertaking, requiring a cohesive working team or unit. All evaluations have one thing in common; they all require carefully collected data. Data can look differently in each evaluation. There are both qualitative and quantitative data sources and there are four levels of data; nominal, ordinal, interval; and ratio. Data plays a significant part of an evaluation. What is collected, how it is collected and ultimately how it is analyzed, defines an evaluation. It is through this carefully calculated process that an evaluations effectiveness will be determined. Evaluations can assist in the decision making process. A formative evaluation examines a program in progress while a summative evaluation assesses the overall effectiveness of a program. The use of a structured evaluation process, allows for decisions to be made based on data instead of instinct. Before taking EdTech 505, I had not considered what was involved in conducting an evaluation. When I think of all the different programs and products purchased by school districts, some successful and some not successful, I now understand the importance of conducting evaluations. I wonder how many decisions or purchases have been made by instinct rather than by data. What I have learned this semester, I will take with me throughout my career in educational technology. The field of technology is constantly changing. There are a plethora of programs and products that claim to be the best and new ones are constantly being added. I now understand the importance of evaluating what is being offered, rather than deciding by intuition.

Part 2. Executive Summary

In 2008, the City of Upland passed Measure K that provided for public school classrooms to be upgrade to what has been termed 21st Century Classrooms. Measure K included the following improvements: Classroom Technology: Renovate classrooms to include wiring and controls for new and future technology. Add computers and printers connected through a wireless system. Install instructional computerized audio and visual systems. Install teacher microphone and speaker systems. Install sky lighting to improve student performance (where possible). Install teaching walls with dry-erase boards and additional storage. Install new flooring, window coverings, low noise fiberboard wall coverings and paint interiors. Install new furniture (from Exhibit B, Full Ballot Text, Magnolia School). In year one of the renovations, a proto-type classroom was completed in one of the three six grade classrooms at Magnolia Elementary School. This renovation included all of the above improvements plus a classroom set of CPS Pulse Response clickers by eInstruction. The next year when all of the classrooms underwent renovations, the response system was deleted from the upgrades. One of the six grade teachers was able to secure two additional classroom sets of CPS Pulse Response clickers on loan from a different school district. The three teachers currently using the response system have stated that they feel this system has facilitated learning and that the system helps engage the learner. The cost per classroom is approximately $2,500.00. The president, Mrs. Andrews, of Magnolias Parent Teacher Association (PTA) has asked for an evaluation to present to PTA to substantiate PTA funding the purchase of CPS Pulse Response clickers for the three fifth-grade classrooms. Mrs. Andrews indicated that an evaluation would have to clearly indicate that the response system is beneficial to classroom learning for PTA to consider purchasing additional classroom sets. The results of this evaluation show that the CPS Pulse Response System both facilitates learning and engages the learner. The data collected in this evaluation includes information collected from teacher and learner surveys, classroom observations, and from teacher and student interviews. The data overwhelmingly concludes that this response system is beneficial to classroom learning. The stakeholders for this evaluation are the three fifth-grade teachers, staff, fifthgrade students, and parents of Magnolia Elementary School.

Part 3. Purpose of the Evaluation

Part 3a. What was the purpose of doing this particular evaluation?
The purpose of choosing this particular evaluation was to determine if the CPS Pulse Response System is effective in facilitating learning and engaging the learner. This evaluation is a necessary instrument for Magnolias PTA to determine if it will allocate approximately $7,500 to purchase three classroom sets of response systems for the fifth-grade classrooms. The three teachers currently using this system seem to be satisfied in the way this response system engages their learners and facilitates learning. The systems immediate feedback provides important information both to the teacher and to the learner. The teacher is able to adjust instruction based on class answers to problems. The learner knows immediately if their answer is right or wrong and whether further questions need to be asked for clarification. The primary aim of this evaluation will be to draw on the experience of the three six-grade teachers and 89 students currently using the CPS Pulse Response System. Their knowledge and experience will prove valuable in determining if both teachers and students find the response system an advantageous tool for both teaching and learning. The information obtained through this evaluation is critical for the PTA to determine whether or not they will fund this project.

Part 3b. What were the central questions to be answered through the evaluation?
Many questions were asked on this evaluation, the results of all the questions can be found in the Results section of this evaluation. Following are the three central questions that are the focus of this evaluation: Does the CPS Pulse Response System facilitate learning? Does the CPS Pulse Response System engage the learner? Can the cost of the CPS Pulse Response System be justified by the value they add to learning?

The answers to these questions are critical to obtaining PTA funding. The CPS Pulse Response System is an expensive addition for a classroom. In order for PTA to vote in favor of this allocation, it is necessary that an evaluation prove the merit of such a system.

Part 3c. Who would be most impacted by the results of the evaluation?
Magnolia fifth-grade students will be most impacted by the results of this evaluation. It is their education that is first and foremost taken into consideration. The fifth-grade teachers of Magnolia will also be impacted as the response system will have an affect on their lesson plans and impact the way they teach. 5

The PTA will also be impacted. It is only through the generous efforts and volunteer hours of the parents associated with Magnolia PTA that thousands of dollars are raised each year. These monies allow students of Magnolia to have items purchased for the classroom above and beyond what is allocated through the district budget.

Part 4. Background Information

Part 4a. What was the origin of the product?
The initial proto-type Century 21 Classroom included the CPS Pulse Response System. The upgrades to the classrooms are a result of Measure K bond money. The goal was to upgrade the public school classrooms in the city of Upland with classrooms that would prepare students to compete in the 21st century. Initially the response systems were to be included in all classrooms for grades five through twelve. Because the bond money did not go quite as far as anticipated, the response systems were cut from the list of technology items to be included for each classroom. The only classrooms equipped with them are the five proto-type classrooms throughout the district that were retrofitted during the first year of renovations. The district felt that out of all of the upgrades that were being provided, the response system would be the least missed by the teachers. Magnolia Elementary School was one of the sites picked for a proto-type classroom. As a result Mrs. A____, a sixth grade teacher, had the benefit of receiving a response system. She immediately saw the advantage of such a system and shared her successes using this system with her teammates, Mrs. B____ and Mr. C____. When the renovations to all of the classrooms were complete and it became apparent that the response systems would not be provided, her teammates were disappointed. Mrs. B____ was able to secure a loan of two additional classroom sets of response systems, through her husband who is head of technology in a different school district. The six-grade team has been successfully using the CPS Pulse Response System and at this point the fifth-grade team feels that the fifth-grade students would benefit from having the same system in the fifth-grade classrooms.

Part 4b. What are the goals of the product?

The eInstruction CPS Pulse Response system was designed to transform a classroom into a high-performing interactive learning environment. eInstruction makes the following claims: Capture real-time assessment data to gauge student comprehension as you teach. Identify individual learning needs and differentiate instruction as needed.

Automate and streamline time-consuming administrative tasks like taking attendance, grading quizzes or tests, and recording the results. Generate personalized reports in seconds.

The response systems are designed to keep students engaged and involved in class. They facilitate greater student-teacher interaction. This system creates a dynamic learning environment that encourages class discussions and participation.

Part 4c. What did previous products look like?

Prior to using the CPS Pulse Response system, sixth grade teachers taught using traditional methods of teaching. They used paper and pencil quizzes to gain feedback that helped guide their instruction. All three teachers would use the thumbsup if you get it or thumbs-down if you dont method of inquiry. This method is not effective, as often children look around to see if more thumbs are up or down and they go along with the majority. Each sixth grade teacher has voiced that the biggest benefit to this system is engaging the students and gaining immediate feedback. Prior to using this system it was difficult to engage the students and keep them tuned-in to learning. The CPS Pulse Response system has been a tremendous help in keeping students engaged. The immediate feedback it provides allows the teacher to move-on to the next concept or keep teaching if a concept is not understood.

Part 4d. Who are the people involved with the product?
The people currently involved with the CPS Pulse Response System are the sixth-grade team at Magnolia Elementary School. The team consists of Mrs. A____, Mrs. B____ and Mr. C____. Classroom observations were conducted in each of the sixth-grade classrooms. The findings were consistent in each of the classrooms. The students appeared to be engaged and the results provided by the response systems were driving the instruction.

Part 4e. What are the characteristics of the product?

The CPS Pulse Response System is a handheld device that wirelessly transmits information. A picture of the device can be seen on the cover of this evaluation report. The device enables each student to respond anonymously to a question asked by the teacher. Once the results are in, the teacher knows immediately how many students got the answer correct and how many missed it. The teacher also knows exactly which students got the right answer and which ones did not. This information shows up only on the teachers hand-held device. The students see only the correct answer displayed and how many students got the correct answer and how many students got the wrong answer. Their individual answer is publicly anonymous.

The following pictures were taken during a classroom observation in Mrs. B____s sixth-grade classroom. The pictures show the engagement of the students as well as how the information is displayed on the SmartBoard. Additional photos can be seen in the Appendices section at the end of this report.

Part 5. Description of Evaluation Design

A goal-based model was used for this summative evaluation. The aim of this evaluation was to establish whether or not the CPS Pulse Response System helps to facilitate learning and engage the learner when used in the classroom. The goal was to ascertain the benefits of the response systems so that Magnolia PTA could justify an approximately $7,500.00 allocation of funds for the purchase of three additional response systems for the three fifth-grade classrooms. Two surveys, a teacher version and a student version, three classroom observations and six interviews were used to collect necessary data. The study took place over a two-week period of time from November 8, 2010 through November 19, 2010. Participants in the study included three sixth-grade teachers and 88 six-grade students. There are a total of 42 girls and 46 boys. Of these, seven are designated as resource students (RSP), nine are identified as gifted (GATE) and four are identified as second language learners (ESL). The six-grade classrooms are broken down as follows: Mr. C____ has 29 students; 14 girls, 15 boys, 3 RSP, 4 GATE and 4 ESL; Mrs. B____ has 30 students; 13 girls, 17 boys, 2 RSP, 3 GATE; Mrs. A____ has 29 students; 15 girls, 14 boys, 2 RSP, 2 GATE. Initially only a sample group from each class was going to be used to conduct the survey, but ultimately each sixth-grade student completed the survey. This was 9

necessary as the sixth-grade students are ability grouped for core classes and these groups change every few days based on four-point assessments. It was difficult to determine which student was with which teacher and ultimately, logistically, although more time-consuming, it was easier to have each student complete the survey. Teacher Survey. The teacher survey consisted of ten questions. The form was created using Survey Monkey. I made the survey anonymous, as I wanted honest answers. I feel that anonymity provides a safe shield for people to answer questions openly and honestly. The questions used for this survey were direct. They were designed to ascertain if the response system was being used on a daily basis and if they felt the system facilitated learning and engaged the learner. The exact questions can be read in the Results section below. The survey link was emailed to each of the three teachers involved in the study. Their answers were tabulated in Survey Monkey. Student Survey. The student survey consisted of ten questions. The form was created using Survey Monkey and was anonymous. Seventy-seven of the students took the survey as they rotated through their technology class with Mr. C____. Mr. C____ had the link displayed on the SmartBoard and students typed in the link and took the survey. Mr. C____ informed the students that the survey was anonymous and to provide open and honest answers. He also instructed the students not to discuss the questions with each other prior to answering the questions. He explained the purpose of the survey and the importance of honest answers. Eleven of the students were either absent or outside of the classroom for another activity while the survey was being administered. These students came to my class during recess and took the survey using my classroom laptops. The student questions are similar to the teacher questions. The exact questions can be found in the Results section of this evaluation. Classroom Observations. Observations took place for thirty minutes in each of the sixth-grade classrooms over the course of two weeks. The observations were scheduled rather than random to ensure that the CPS Pulse Response System would be in use during the observation. During each observation I sat quietly in the back of the room in an effort to record how the classroom actually functions while using the response system. I took pictures in only one of the classrooms during a second visit to this classroom, as students tend to take on a performance nature when they see a camera. Teacher Interviews. I interviewed each of the sixth-grade teachers separately. They appeared to be open and honest in their responses. The interviews took place in each of their own classrooms. The interviews were spread out over the two-week research window. Student Interviews. I interviewed three students, one from each of the three sixth-grade classrooms. I chose three students that I felt would be open and honest when responding face-to-face. The students were interviewed separately in my classroom during lunchtime. The interviews were spread out over the two-week research window.

10

This evaluation was designed to determine if the CPS Pulse Response System was useful in facilitating learning and engaging the learner. The surveys asked specific questions of both the students and the teachers to rule out if the devices themselves were a distraction or a useful educational tool. The surveys and the interviews were used to determine in the CPS Pulse Response System lived up to the claims made by eInstruction. The goal-based model was used for this evaluation as this model is designed to gauge the effectiveness or the impact of a program or product. A goalbased model is concerned with stated objectives rather than ancillary items (Boulmetis & Dutwin, 205). The goal-based model uses quantitative or qualitative data and an evaluators tasks for this model include data collection, analysis and interpretation (Boulmetis & Dutwin, 2005).

Part 6. Results
Teacher Survey Link Student Survey Link Above are links to both surveys used in this evaluation. An additional link to the surveys can be found in the Appendices section of this document.

Part 6a. Teacher Results.

Questions 1, 3, and 9 were yes or no questions and each had the same response. 1. Do you find the response clickers are a useful teaching tool? 3. Do you feel the response clickers engage the learners and facilitate learning? 9. Do you believe the cost of the response clickers can be justified by the value they add to student learning?

Questions 1, 3 and 9
Yes No

2. How many times a day are you using the response clickers? 11

Teacher 3 Teacher 2 Teacher 1 1 2 3 4 5

# of Times

4. Which subject to you find the response clickers most useful?

Subject
Other Social Studies Science Writing English Math 0 1 2 3 Subject

5. On a scale of 1 to 4, with 4 being the best, how effective do you find the instant feedback in guiding what and how you are teaching?

Effectiveness
Teacher 3 Teacher 2 Teacher 1 0 1 2 3 4 Effectiveness

Questions 6, 7, and 8 were yes or no questions and each had the same response. 12

6. Do you find the response clickers a distraction when not being used? 7. Do you find the response clickers a distraction when they are being used? 8. Do you feel that you are using the response clickers to their full potential?

Questions 6, 7 and 8
Yes No

10. What do you like the most about the response clickers and what do you like the least about them? Most: Teacher 1 Novel, engaging, provides another method of quick feedback Teacher 2 The instant feedback on lesson objectives; student interest and engagement Teacher 3 I like the immediate feedback they provide Least: Teacher 1 I havent found anything I dont like yet. Teacher 2 No comment listed. Teacher 3 There are no negatives. The interviews conducted with each of the sixth grade teachers produced little new information other than what is presented above. Each of the teachers expressed that the CPS Pulse Response System was extremely useful in guiding their instruction. When questioned specifically about whether or not the handheld device was distracting or how they felt about students inputting an erroneous answer it was clear that they had addressed these issues as a team. The six-grade team has clearly stated rules regarding the handling of the devices. Initially a few of the students were playing with the device but those students now follow the rules or they forfeit use for a period of time. Each teacher also had students input what looked like silly answers because they did not realize that the teachers device knows individual students answer. This quickly stopped when the teacher held the student accountable.

Part 6b. Student Results

1. How many times during the school day are you asked to use a response clicker?

13

Times Used During a Day

Five or More Four Three Two One 0 10 20 30 40 50 60

Times Used

2. Which subject do you find that the response clicker is most frequently used?

Subject
Other Social Studies Science Writing English Math 0 10 20 30 40 50 60 70 80 90 Subject

3. Do you feel the use of a response clicker keeps you more engaged in class?

14

Engaged

Yes No

4. On a scale of 1 to 5, with 5 being the most effective, how effective do you find this system in helping you stay focused and on task?

Stay Focused
40 30 20 10 0 One Two Three Four Five Stay Focused

5. Do you feel the response clickers increase learning?

Increased Learning
Yes No

6. Do you find yourself distracted by the use of a response clicker?

15

Distracted
Yes No

7. Do you find yourself playing with the response clicker rather than paying attention to what is being taught?

Playing
Yes No

8. Do you wish your teacher used the response clickers more often?

More Often
Yes No

9. Do you wish your teacher used the response clicker less frequently?

16

Less Frequent
Yes No

10. Please feel free to add any comments about your use of the response clicker. 18 students out of 88 took time to write a comment. Below is a list of the unedited comments: 1. I really like the clickers, but they distract many people and are kind of like toys. Other than that they are great learning tools!!! Sometimes toys are the only things that can help you learn! 2. it is great 3. I think these are a great way to lean more if you need help. 4. I like the responed clickers but sometimes they get out of hand with other students. Thats why I sometimes don't like it. But other then that the response clicker are O.K. 5. I love using the responce clickers. It helps me alot and helps me pay attention in class. I also like typing in my answer to the teacher it makes it more fun and exciting to see what other people think. 6. I think that the clickers are a great way too learn though and you can even see what everyone is thinking! GET THEM :) 7. many of my class mates l like to play around with them and put silly answers. And they might think it is funny to put in wrong answers. 8. clickers are awesome 9. I think that there great beacause the teacher can see how many people get it right or wrong. Its like a little qwiz for the whole class! 10. i think other grades will like the response pad. They are fun to use!!!

17

11. I love the clickers 12. do wish that my teacher would use it more often. 13. i like them they keep the room more quiet. 14. it is fun to use the clickers because you dont have to write anything down. you just press a button or two, and press send. 15. I think it helps some of the kids, but some of them use it as an ecuse to play. 16. I think the response clickers have changed evreything! It is so much easier than to rather raise your hand and get called on. 17. It's fun and helpful 18. i really like them. they r really easy to use and they make learning fun Three student interviews were conducted with one student picked from each of the three six-grade classrooms. One girl and two boys were selected. Of the three one of the boys is identified as GATE. The other boy and girl are average students and not identified in any special group. I asked each student about students playing with the response clickers or writing a silly answer. Each of the students indicated that both of these issues occurred at the start of the school year and that this seldom happens anymore. The students offered little new information to add to the data collected from the survey.

Part 7. Discussion of the Results

Part 7a. How good were the results of the product?
The results of the product were overwhelmingly positive. The purpose of this evaluation was to answer the following questions: Does the CPS Pulse Response System facilitate learning? Does the CPS Pulse Response System engage the learner? Can the cost of the CPS Pulse Response System be justified by the value they add to learning?

When teachers were asked if the CPS Pulse Response System facilitated learning the unanimous answer was yes. This is a claim made by eInstruction and was substantiated through surveys, interviews and observations. When teachers were asked if the CPS Pulse Response System engaged the learner the answer was again a unanimous yes. Although each teacher did state that initially

18

some students were playing or distracted by the unit, after modeling the rules and enforcing the rules the incidents of this happening has greatly diminished. The final question is a difficult one to answer although again the vote was another unanimous yes, Can the cost of the CPS Pulse Response System be justified by the value they add to learning? It is difficult to put a dollar value on the benefits a product adds to learning. It is the belief of this evaluator that the benefits this system has added to the classroom must be very high for each of the teachers to respond affirmatively to this question.

Part 7b. What are the costs associated with the product?
The costs associated with this product are approximately $7,500.00. A classroom set of response clickers, 32, is about $2,500.00. There are three fifth-grade classrooms which brings the total to $7,500.00 plus tax and shipping. There are no additional fees for training as this product is an extension of other products already in use in the classrooms. Training is already being provided so this cost is not an issue.

Part 7c. What are the benefits associated with the product?
There are many benefits associated with this product. The main benefit experienced by both teachers and students is keeping students engaged and on task. In addition, teachers are given immediate feedback that allows them to adjust their instruction based on the needs of their students. Students are able to immediately realize whether or not they understand what is being taught and can immediately ask clarifying questions. Each of the teachers involved in this evaluation stated that they did not feel they were using this system to its full potential. I estimate that the value of this system will increase as the teachers incorporate its use more frequently throughout the day.

Part 8. Conclusion & Recommendations

Part 8a. Immediate Conclusions
The CPS Pulse Response System is an effective learning tool based on the following reasons: Keeps students more engaged in the class; keeps students focused and on task; facilitates learning; gives immediate feedback; and guides instruction. 19

Part 8b. Long-Range Conclusions

The long-range conclusions for the CPS Pulse Response System are as follows: Continued daily use by sixth-grade team of response system during math lessons to ensure students understand concepts that are being taught. Sixth-grade teachers will increase their use of response system during other subjects to help facilitate learning. Sixth-grade teachers will continue to monitor the proper use of the response system and hold students accountable for inappropriate use. Fifth-grade teachers and students would benefit from classroom sets of CPS Pulse Response Systems.

Part 8c. Evaluation Insights

This evaluation went fairly smoothly. The only part that did not go as planned was including the entire six-grade class in the population being tested, rather than just testing a sample population. It did take more time making sure everyone was tested, but ultimately I feel it made the evaluation more valid. The purpose of this evaluation was to create a tool that PTA can use to help make a decision whether or not to appropriate funds for the purpose of purchasing three sets of CPS Pulse Response Systems. To that purpose, I feel including all of the population currently using this system helped to validate the evaluation. This is the reason I included the 18 comments that students took the time to write in the surveys. I felt it validated the study and offering the comments in an unedited form made the statements authentic. I understand that it is not always logistically feasible to include an entire population in an evaluation, but sometimes, as in this study, it lends more credibility.

Part 9. References
Boulmetis, J., & Dutwin, P. (2005). The ABCs of Evaluation (2nd ed.). San Francisco, CA: Jossey-Bass.

20

Part 10. Appendices

Student Survey Teacher Survey Additional Photos

21