The Effectiveness of LEGO Robotics at Simmons Elementary School

An evaluation submitted to Karen Kanter, Principal

Lead Evaluator: Kevin Deissler

EDTECH 505: Evaluating Programs for Educational Technologists

Summer 2015

P a g e | 2

Summary
The LEGO robotics program at Simmons Elementary School is about to enter its third year of a
program overhaul. The program encourages students to fail because failure breeds learning
opportunities. The robotics program started with 18 students engaging in robotics and
programming over a span of 10 after-school sessions and has evolved into this years taking on of
78 boys and girls over a span from January to June. Based on the need to adjust learning and
teaching for a fluctuating number of students, the program instructors, or advisors, felt the need
to evaluate the program to assure that they are meeting student needs. The advisors feel that all
students (whether working individually or in teams) should learn how to program and debug
their robot of errant code. This should ensure that students are able to complete specific
challenges as part of a team. By utilizing collaboration and failure, students will learn how to
effectively communicate and solve problems through learning opportunities. These learning
opportunities will transfer into building a foundation for improved self-motivation and resiliency
and into an appreciation for robotics.
The purpose of this evaluation is to determine if the instructors are hitting the mark in regards
to improving communication, programming, and resiliency skills among all students
participating in the program. Prior to this evaluation, the instructors only verbally assessed the
program with little meaningful feedback from the students and zero feedback from parents or
teachers. This evaluation attempts to identify in greater detail whether needs are being met as
well as overall opinion on its effectiveness by parents and teachers regarding student success.
Immediate results of the program based on student and parent feedback have been
overwhelmingly positive. A majority of the students enjoyed the combined experience of
working with LEGOs and completing challenges as a team. They learned programming and
problem solving skills while inherently learning strategies to aid in working with others. The
participants did not let failure stop them and were able to transfer learned skills into their general
classroom environment. Some parents noticed an improvement in their childs study skills and
grades and most of them noticed that their child did not give up as easily. Those were core skills
that this program aimed to transfer into learning opportunities, and parents perspective of a
positive difference in their childs learning habits solidify the positive results of this program.

The Program Description

The LEGO robotics program at Simmons Elementary School has evolved over the course of two
years. It started as a 10-session program where 18 fourth and fifth grade students competed in a
Battlebots type scenario to multiple, leveled sessions with 78 fourth and fifth grade
participants. The most recent incarnation of a program ran from January through the beginning
of June. In January, an advanced group of fourth and fifth graders met over the course of 10
sessions (after school days) to learn how to complete challenges surrounding a FIRST Lego
League (FLL) challenge from a previous year. The advanced students were chosen based on the
prerequisite that theyve programmed using the LEGO programming software previously and
thus didnt need much introduction. Students were paired, teamed, and challenged in a way that

P a g e | 3

simulates participation on an FLL team. At the end of the 10 sessions, students competed in an
exhibition in front of teachers and students to demonstrate what theyve learned and completed
over the course of their sessions. Each individual session ran from 3:15-4:45 pm.
Following the advanced session, the program offers an opportunity for any fourth or fifth grader
who does not have any experience with LEGO robots or programming to learn and compete in a
similar type of challenge. Students are paired up and challenged to complete more basic
movements using simplistic commands for their robot. They are encouraged to collaborate and
seek answers to problems on their own. These students also meet over the course of 10 sessions
and also exhibit what they learned in front of teachers and staff at the end of the program.

Program Objectives
Its important that both of these portions of the program share similar objectives, as ideally
students are meant to move from one program to the next.

Students will demonstrate an understanding of LEGO programming language through

manipulation and debugging, to successfully complete tasks using their LEGO robot
Students will improve collaborative social skills by working in teams, while also building
a stronger foundation for self-motivation
Students will improve resiliency skills through failure and will transfer those skills to
their classroom
Students will align their programming skill set with that of our middle school program to
ensure success at an upper level

Each program, advanced and beginner, are structured a bit differently. Because the advanced
group has pre-existing knowledge of basic programming skills, they are unleashed on their
challenge almost right away. There are moments where robot sensors are re-introduced to the
group, as they are needed for their challenge. Each day that students arrived, they were given a
challenge coin. This coin would be their pass to test their robot on the challenge mat. This
helped ensure students practiced and gave more thought to their program instead of consistently
moving back and forth between their program and the challenge mat. The beginner group was
not given these coins and instead was able to move freely from computer to challenge mat.
There was more emphasis placed on the experience and fun of robotics within the beginner
group and less on the challenge. When students of either group were successful at completing a
challenge, they had to be able to replicate it in front of one of the advisors. Replication of
challenges coupled with an emphasis on taking notes throughout the process helped to foster
problem-solving skills and perfection. Almost every student group was successful in completing
challenges within both of these groups and the ones that were not successful, never stopped
trying.

Evaluation Method

P a g e | 4

Students, parents, and teachers were surveyed via Google Forms and data was collected and
analyzed alongside observational notes from the mentors. The most important information was
gathered from students first and parents second, since students were directly affected.
Between the advanced group and beginner group, every student who was interested in
participating in robotics and who was eligible (advanced group) were allowed to participate.
Throughout each session, three advisors work one on one with student groups at times to help
guide them through the programming process. When students would test their robot on the
challenge mat, one or more advisors would often be present to see how the students did. The
advisors would ask the students what they thought happened and how they might fix it should
they not be successful, as well as offer suggestions. Students were encouraged to take notes
throughout their programming and testing process so they could go back and view their progress.
These notes also served as a self-assessment for the students, allowing them to go back and learn
from their mistakes. Advisor observations of student groups helped ensure continued
communication throughout the program. At the conclusion of each session the advisors would
talk about what worked and what didnt work, who succeeded and who didnt. They would
attempt to make changes to the next days session based on their discussion.

Procedures
At the end of each session, students were given a survey for them to complete asking them
simple questions, such as whether or not they had fun, what their most proud moment was, what
was most frustrating, etc. These questions really did not focus on whether or not objectives were
met. This survey was restructured to better reflect whether or not objectives were made and was
emailed to students (Appendix A) and parents (Appendix B) following the school year for
feedback. A survey was also sent to teachers of robotics students (Appendix C) asking them to
reflect on their perception of their students during and following their involvement in the
program. These surveys included questions utilizing the Likert Scale, open-ended questions, and
nominal data. Every parent, student, and teacher received this survey, however participation was
not 100% due to summer vacation. 24.36% of students responded to the survey, 43.9% of
parents responded, and 27.3% of teachers responded. To rectify the response rate, surveys will
be done immediately following the program in the future when all parties are present and more
easily accessible.

Participants
Only fourth and fifth grade students were able to participate in the program, which was spread
out over three different sessions. An advanced session comprised 23 fifth graders and 1 fourth
grader, and two beginner level sessions comprised 36 fourth graders and 22 fifth graders. Of
these 78 students, 55 were boys and 23 were girls, all making up roughly 26% of total fourth and
fifth grade students.

P a g e | 5

The mentors for this program that assisted in the day-to-day evaluation of the program were
Kevin Deissler, Dan Beck, and Malcolm Stanley. Kevin is a technology integration teacher and
specialist and the lead contact for the program. He is in charge of contacting parents, organizing
events, and providing other point contact access. He, along with Dan Beck, a fourth grade
teacher, provides training on LEGO EV3 software while encouraging students to find any hidden
freedoms of robot challenges. Malcolm Stanley is a marketing consultant for a technology firm
and a parent in the school district. He helps guide students problem solving skills by asking
relative questions. All of the mentors have a stance that failure is ok and with failure comes
learning opportunities. They work in sync to ensure that every student learns and participates in
the program.
The program currently has one official sponsor, in Noetic Software, Inc., a software design
company interested in providing STEM resources to students. Howard Knorr, the companys
president, has donated funds to help ensure the success of the program based on its involvement
of a large number of students, including girls. He has attended at least one showcase function
and likes what he sees from the final product. This evaluation will also help him realize the
benefits of his gracious generosity.

Results
By re-tooling the surveys to better address the goals and objectives of the group, the evaluators
are able to approximately determine the overall success of the program.
In response to whether or not students felt they had learned how to program robots using the
EV3 software:

Did you feel you learned how

to program the LEGO robot?
0

Do you feel that your student

learned how to program their
robot?
11

9
14

25

Strongly Disagree

Disagree

Strongly Disagree

Disagree

No Opinion

Agree

No Opinion

Agree

Strongly Agree

Strongly Agree

Overall, students and parents saw an immediate impact of the program, as the students were
either learning something new or were building upon previous programming experience.
Learning how to program a robot, upon observation, allowed students to build confidence while

P a g e | 6

displaying failure because very few students were successful at programming the robots on their
first try.

Have you noticed an increase in

resiliency with your student?

Students: Did you become

frustrated at any point but manage
to work through problems with the
help of others?

8 2 3

1 0
2
9

23

Strongly Disagree

Disagree

Strongly Disagree

Disagree

No Opinion

Agree

No Opinion

Agree

Strongly Agree

Strongly Agree

Resiliency is the ability to overcome failure and both parents and students are recognizing a
change in this. This, more than anything, has the ability to transfer to student performance
outside of robotics.

Student Collaboration
9

10

8
6

4
2

2
0

Strongly Disagree

Disagree

1
No Opinion

Agree

Strongly Agree

Did you and your partner work well together?

Did you learn anything about working with a partner?

Students could choose with whom they worked for the duration of the sessions, however,
depending on the numbers, they may not have had the option of working with a partner of their

P a g e | 7

choosing. Its most likely in these instances where you find students who did not work well
together. More students, however, were able to work together and at the very least, learn
something about working with someone during a project that requires partners to be in sync with
each other. The learned experiences that students were a part of will help students moving
forward, as they move to the middle school and will be working with students from other
elementary schools. They might not know those students but may have a better idea of how to
work with them.

Did you become frustrated at any point and give

up on something?
0
2

Strongly Disagree

Disagree

10

No Opinion

Agree

Strongly Agree

The students never gave up on anything; their mentors told them that it was ok to fail; its
possible that the message got through to them. When the mentors asked questions, the students
were able to think about the problems and talk out possible solutions. While there is no data on
whether or not this program will help the students at the middle school level, based on this
survey and one on one experiences, the evaluation team can speculate that students will not give
up and there is potential for them to succeed at the middle school level because they will not quit
if given the opportunity.

P a g e | 8

Survey Sample
15

16
14

13
11

12
10

4
2
0
Boys

Girls

4th Grade 5th Grade

Advanced Beginner

Even though a small percentage of students responded to the survey, there was a good mix in
responses across all 78 students.

Data Sources
Teachers of robotics students were not able to help with this study at this time due to summer
break. Only three out of eleven possible teachers (two fourth grade teachers and one fifth grade
teacher) responded to the survey and provided the following results:

Teacher Responses #1
5
0
Strongly Disagree

Disagree

2 1 2 2 2

1 2 1 1 1

I Don't Know

Agree

Strongly Agree

Did you notice an academic improvement in your students who participated in

robotics?
Did you notice an improvement in communication or collaboration skills with
students who participated?
Did you notice an improvement in student resiliency?
Did you notice an improvement in student leadership or willingness to take a
leadership role?

P a g e | 9

Approx. how many students

Teacher Responses #2
2.5
2
1.5
1
0.5
0

2
1

2
1

2
1

2
1

Did you see an Did you see an Did you see an Did you see an Did you see a
academic improvement in improvement in improvement in more positive
improvement? communication
student
communication outlook on
or collaboration resiliency? or collaboration learning?
skills?
skills?

What change in students did you see?

None

Some

All

I couldn't Tell

When students were asked to offer up their own advice as to whether or not students should join
robotics, some answered:

Don't give up and keep trying.

It is fun because you learn new things and get your first experience of programming.
To be creative with your robot and keep trying no matter how much you fail.
Don't get mad when you can't complete a challenge right when you want to finish it.
Try hard and don't give up! Remember that you will not fail you will just find a lot of
ways that don't work.
The hardest part is working together.

Discussion
The purpose of this survey was to determine if the LEGO robotics program at Simmons
Elementary School is an effective program. The evaluators determined that in order to be
effective, students needed to learn a programming language, increase resiliency, and work
collaboratively. Success in those fields should transfer to success in the students classrooms
and at the middle school level.
The students appear to be succeeding in the field of programming and debugging their robot.
They arent giving up on something that is new and challenging to them. Based on their advice
to future students, they understand the idea that failure helps the learning process. They are still
learning how to work with a partner effectively, however. A few outlier results show that there
were some students that did not work well with others, but probability states that is to be
expected with a group of 78 individual students. There is very little information stating that
students are benefitting in the classroom from this program, unfortunately.
The parents seem happy with the program and would benefit from more communication with
their students. Perhaps the 10 sessions were not enough for an impact on students for parents to

P a g e | 10

notice, however they were given the opportunity to witness their students perform at the end of
each session. Their knowledge is based on communication and that performance.
It would serve the program well for the mentors to involve more teachers. Its important for
teachers to connect with their students and know they are participating in robotics. When the
showcase nights were held, the only teachers in attendance were those with children who
participated. If more teachers saw how the program ran, they may be able to use some of the
same philosophies about failure in their own classroom, since this robotics program appears
successful using that moniker.
The mentors are doing an effective job at communicating challenges and working with students
who are struggling. They need to find a way to help those who are having difficulty working
with others to ensure that all students are learning and not just standing by while their partners do
all of the work. As the daily operations continue to be revised for better time management and
hands-on opportunities, this should allow for more mentor engagement with these struggling
groups.
Student voice speaks loudest. Students are enjoying the time their time in robotics and their
comments in the survey speak to student success. With hope, they will remember what theyve
learned that they can do if given the opportunity to fail.
The evaluators conclude that this robotics program is indeed successful and should continue.
The evaluators recommend involving more teachers so they can see first hand how student
failure breeds learning opportunities. They also recommend including some team building
exercises to make sure all partnerships can work and allowing more demonstration time for each
group of students so that the mentors can better evaluate their progress. With these minor
improvements, this program should continue to be successful.

P a g e | 11

Appendix A
Student Evaluation

Strongly
Disagree
Did you have fun?
Did you feel welcome in class?
Did you make new friends?
Did you feel you learned how to program
the LEGO robot?
Did you find the tasks challenging?
Were friends, teammates, and interns
available and willing to help with
problems?
Did you feel successful after completing a
task?
Did you and your partner work well
together?
Did you learn anything about working
with a partner?
Did you become frustrated at any point
but manage to work through problems by
yourself?
Did you become frustrated at any point
but manage to work through problems
with the help of others?
Did you become frustrated at any point
and give up on something?
Were you able to apply any skills learned
in robotics to other classes (math, science,
etc.)?

Disagree Not
Sure

Agree Strongly
Agree

P a g e | 12

1. Why did you decide to join LEGO robotics?

a. Friends
b. Like to play with LEGOs
c. Wanted to see what robotics was about
d. Parents made me
e. other
2. What was your favorite part about using the robots?
a. Building your robot
b. Designing attachments for your robot
c. Programming your robot
d. Fixing what was wrong with your robot
e. other
3. Which subject could you see using robots in outside of robotics?
a. Math
b. Science
c. Reading
d. Writing
e. other
4. Did you feel that by participating in robotics you improved in any other subject such as:
a. Math
b. Science
c. Reading
d. Writing
e. I dont feel like I improved elsewhere
5. What would you have liked to learn more of?
a. Using sensors
b. Tracking your progress using your dossier
c. Building measuring skills to help with programming
d. Working better with others
e. other
6. What was your most proud moment while participating in robotics?
7. What was your most frustrating moment while participating in robotics?
8. What was your favorite thing about participating in robotics?
9. What was your least enjoyable thing about robotics?
10. What advice would you give to future students who will participate in robotics?
11. Are you a: boy/girl
12. What grade are you in: 4th/5th

P a g e | 13

Appendix B
Parent Evaluation
Strongly
disagree
Do you feel your student felt welcome in
robotics?
Do you feel that your student made new
friends in robotics?
Do you feel that your student learned how
to program their robot?
Does your student take an interest in and/or
excel in math?
Does your student take an interest in and/or
excel in science?
Have you noticed an improvement in your
students study skills since participating in
robotics?
Have you noticed any improvement in
grades in any subject since your student has
participated in robotics?
Have you noticed an increase in resiliency
(not giving up as easily) in your student
since their participation in robotics?
Have you noticed an improvement in
social/collaborative skills in your student
since their participation in robotics?
Does your student build with LEGOs in
their free time?
Do you believe that participating in a
robotics program can help students succeed
in the classroom?

Disagree No
opinion

Agree Strongly
Agree

P a g e | 14

Would you be interested in seeing more

robotics integrate into math and science
classrooms?
Do you feel your student will want to
participate in robotics in the future?
How important do you feel that your
student receives lessons in programming,
coding, and debugging (figuring out whats
wrong)?
How important do you feel that all students
should receive lessons in programming,
coding, and debugging?
Based on your overall knowledge of our
robotics program and discussions with your
student, do you feel that our program is
hitting the mark, i.e., is it beneficial or
necessary to continue?
1. How did your student become interested in robotics?
a. Loves LEGOs and was naturally interested
b. Friends wanted to join
c. Expresses an interest in math and/or science
d. other
2. When did you notice your student first become interested in LEGOs or robotics?Just this
year
a. Within the last 2 years
b. 3 or more years ago
c. other
3. What grade is your student in?
a. 4th
b. 5th
4. Is your student a:
a. boy
b. girl

P a g e | 15

Appendix C
Teacher Survey
1. Did you notice an academic improvement in your students who participated in robotics?
a. strongly disagree
b. disagree
c. no opinino
d. agree
e. strongly agree
2. Approximately how many did you see an academic improvement in?
a. None
b. Some
c. All
d. I couldnt tell
3. Did you notice an improvement in communication or collaboration skills with students
who participated?
a. yes
b. no
c. i dont know
4. Did you notice an improvement in student resiliency?
5. Did you notice an improvement in student leadership or willingness to take a leadership
role?
6. Did any of the involved students have a more positive outlook on learning following their
time with robotics?