Running head: INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 1

Influence of Khan Academy on Algebra 1 Success

Laura A. Hickman

Coastal Carolina University

INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 2

Influence of Khan Academy on Algebra 1 Success

Introduction:
As a middle and high school instructional coach, my job is to provide support to teachers in

the areas of math and science. With more and more schools moving to 1-to-1 devices, classrooms

activities, instruction, and assessments are moving to more digital applications, therefore the

integration of effective digital resources is a priority. This year, one of our school’s focus areas is

Algebra 1. Historically, our students have struggled to perform at a high level on the End of Course

Algebra 1 assessment. One of the resources I suggest for teachers to integrate is Khan Academy for

practice, remediation, and differentiation. Therefore, my research question is, ​Does the

integration of Khan Academy for remediation and differentiation influence student academic

success in Algebra 1 as measured by benchmark scores?

Through this action research project, I would like to see if implementing Khan Academy

consistently will help increase our students’ performance in Algebra 1. I worked closely with the

Algebra 1 teacher to plan, monitor, observe, and reflect the use of Khan Academy throughout the

process of this technology integration. We met for digital lesson planning and I observed on a

bi-weekly basis. Data collected and analyzed was from classroom observations, district made USA

Test Prep Benchmarks, and student and teacher interviews and surveys.

For my action research project, I choose a teacher at my school that teachers Algebra 1 and

asked if she would be willing for me to use her class for my research. After giving a short synopsis

of the requirements and confirming her willingness to participate, I collected class information

from the teacher. The teacher is serving her 7th year at our school and in education. The classes are

at Green Sea Floyds High School and contain both 7th and 8th grade students. There are 48
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 3

students total in the two classes with 19 seventh grades and 29 eighth graders and 20 male and 28

female students.

Review of Literature:

There are many studies that address the effectiveness of Khan Academy on the academic

success of students. The studies range from effects of using Khan Academy in elementary grades

through college aged students. In the study, “Can Khan Move the Bell Curve to the Right?”, J.

Kronholz explains a pilot of Khan Academy at Los Altos and Oakland’s Envision Academy of Arts

and Technology with 5th and 7th grade classes. Test results suggested that Khan Academy was

working. Both districts in the study stated that it was too early to claim success, but enough to say

this is promising. Most observers of the Khan experiment agreed that the measure of success must

be student achievement. All agreed that learning how to blend Khan into the curriculum and not

substitute for it (Kronholz, 2012). This study supported my original research topic, but prompted

me to narrow down my data points significantly. I will decided to focus my research efforts on

benchmark score growth before and after the integration of Khan Academy.

Although not in the United States, I gained new insight from reading, “Can Khan Academy

e-learning video tutorials improve mathematics achievement in Sri Lanka?” The study evaluated

the impact of Khan Academy in a blended learning environment for ninth grade students in Sri

Lanka. On average, results increased students’ raw and scaled test scores by 3.77 and 3.15

percentage points, respectively, and standardized test scores by 0.20 standard deviations above the

mean test score. The evaluation showed a positive impact of Khan Academy on students’

achievement in mathematics. This evaluation provided initial evidence that using Khan Academy
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 4

would help Sri Lankan students to overcome some of the challenges related to the high failure rate

in mathematics (Weeraratne and Chin 2018). Although my research project is that of South

Carolina students, I learned a lot from the research of Sri Lankan students. I do not have two

different classes to compare like in this study, so I will have to focus on growth and comparison to

benchmark growth from the beginning of the school year before the implementation of Khan

Academy with the post benchmark scores after the year long integration of Khan Academy.

The article, “Khan Academy as Supplemental Instruction: A Controlled Study of a

Computer-Based Mathematics Intervention” studied the impact of the use of Khan Academy as a

mathematics intervention among seventh grade students over a 4-week period versus a control

group. They found no statistically significant differences in student test scores; however, only one

parameter of student success was measured. They determined that continued research into Khan

Academy and other open educational resources is needed to determine the true impact of these

resources and determine best practices for the use of these programs both in and out of the

classroom (Kelly and Rutherford 2017). Although the students in this study were not successful, I

learned from this study is that a 4-week period may not be long enough to show the true growth or

effect of the program on students. I decided to use data over a much larger time frame to hopefully

have time to see student progress.

Dickerson’s doctoral dissertation, “One-to-One Mobile Devices in Rural School Districts:

A Mixed Methods Study Investigating the Impact Impact of Khan Academy on Mathematics

Achievement and Teacher Pedagogy” used Khan Academy in a one-to-one setting and measured

results with MAP. Scores in the study revealed that 9th grade and 10th grade high school students

utilizing Khan Academy in a one-to-one program demonstrated significantly more growth when
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 5

compared to national norms. In addition, interviews of teachers and administrators were conducted

to reveal themes related to teacher pedagogy and student perseverance with mathematical

problems. Further analysis of the interviews revealed as school districts explore mastery learning,

tools such as Khan Academy become common due to their support of this style of learning

Teachers and administrators in this study were all concerned with student achievement growth and

showed how Khan Academy can be used to achieve this goal (Dickerson, 2016). After reading this

study, I realized the importance of adding qualitative data including surveys, interviews, and

observations in conjunction with the quantitative data of benchmark scores to ensure a mixed

method framework for my action research project.

The article, “Using Khan Academy in Community College Developmental Math Courses:

Results and Lessons Learned from Developmental Math Demonstration Project” is a study that

investigated whether the use of Khan Academy could increase community college student success

in developmental math coursework. There were multiple pilot community colleges that used Khan

Academy in different ways and several types of data were collected during the project. The

conclusion was that Khan Academy is not a “silver bullet” and may or may not adequately replace

a textbook. It cannot substitute for a good instructor and it might not be for every faculty member

or student, but it does address multiple learning styles and reinforces good teaching and learning

habits. When used effectively, Khan Academy can help developmental math students meet success

(Chan, O'Connor, and Peat, 2016). Despite that this study was analyzing data from community

college students and not middle or high school students, the information supports that Khan

Academy must be systematically implemented with the guidance and expertise of a trained teacher.
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 6

I decided to continue to support, as instructional coach, the teacher whose class data I am using to

help reinforce Khan Academy with good teaching and learning habits.

My review of literature was expansive and provided a variety of results. The results

revealed both positive and neutral effects of the implementation of Khan Academy. I believe I can

conclude that my research will not show negative effects of the implementation of Khan Academy

in the middle school Algebra 1 classroom. Reading the above research and pairing my previous

observations and experiences lead me to believe that it is likely that I may see an improvement of

the progress of such a large period of time, but there are many factors that may also contribute to

that success.

Methodology:

My methodology for this action research project answering the question, ​Does the

integration of Khan Academy for remediation and differentiation influence student academic

success in Algebra 1 as measured by benchmark scores?​, is gain scores based. The teacher’s

Algebra 1 classes used for the study did not teacher the same content in previous years. In order to

eliminate a variable in the research project of having data from two different teachers, I decided to

select a gain scores based methodology. The baseline data used is the Initial Benchmark

Assessment as their pretest as it is given in late August during the second week of school.

As instructional coach, I met with the teacher to discuss the implementation of Khan

Academy in her class. We decided that she would use Khan Academy as part of a blended learning

class at least two to three times a week. Students would be assigned the content and asked to watch

the video and answer the corresponding practice questions. Students were asked to hand in hand

written “worksheets” with the assignment to ensure they were doing their own work. Students were
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 7

also required to complete the assignments at 75% or better in order to count as complete. In

addition, students were assigned corresponding Khan Academy assignments with any topic in

which they did not master on a cummaivea ssessemnt to complete as unit recovery. The same

guidelines applied to those assignments as they were to hand in hand written “worksheets” and

complete activities as many times as necessary to earn a 75% mastery or higher.

Students in Horry County School are required to take three district created benchmark tests

through the year in Algebra 1. All three Benchmark Assessments were created with 50 questions in

USA Test Prep by lead teachers and vetted by the county mathematics learning specialist for

mathematics.To obtain my quantitative data, the teacher administered an Initial Benchmark at the

end of August, a Midpoint Benchmark in January, and a Final Benchmark at the end of April. Data

collection started with the Initial Benchmark for the 2018-19 school year which included 50

questions covering all Algebra 1 South Carolina State Standards with 120 minutes provided to take

each assessment.

To obtain my qualitative data, I used multiple methods including observations in the

classroom, interviews with students, and surveys via Google Forms. I observed multiple classes

with the teacher implementing Khan Academy in the blended learning environment and took

anecdotal notes each time. I interviewed multiple students both in person and through the Google

Form survey to gain their perspective on the implementation of Khan Academy. In addition, I had

many conversations with the teacher about the integration this new digital tool and what impact it

has made on her instruction and class as a whole.

INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 8

Analysis

I used the review of previous studies and the collection of quantitative and qualitative data

to start analyzing the information and data to determine if​ the integration of Khan Academy for

remediation and differentiation influences student academic success in Algebra 1 as

measured by benchmark scores​. During the process of this action project, I found myself already

starting to analyze data even before students completed their benchmarks. I first analyzed the

quantitative data for each assessment.

Figure 1: Initial Benchmark Assessment- USA Test Prep 2018-19 GSFH

In Figure 1 above, you will see the results from the Pretest or Initial Benchmark Assessment

which was used as the baseline data for this research project. The average student score on the

Initial Benchmark was 21% mastery with the majority of students scoring between 10% and 29%

mastery on the Pretest.

INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 9

Figure 2: Midpoint Benchmark Assessment- USA Test Prep 2018-19 GSFH

After approximately three months on instruction and implementation of Khan Academy in

the Algebra 1 class, students took the Midpoint Benchmark Assessment for the district and already

proved to have increased their mastery by 1oo percent. As indicated in Figure 2 above, the average

benchmark score for the Midpoint Benchmark Assessment was 48% as compared to the Initial

Benchmark average of 21%. This data truly got the teacher excited that the integration of Khan

Academy was already having a positive influence on student success in her Algebra 1 class. She

admitted that she had never seen such a drastic improvement in mastery with only half of the

content taught for the year in any year prior of mathematics course taught.
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 10

Figure 3: Final Benchmark Assessment- USA Test Prep 2018-19 GSFH

The last assessment given at the end of April gives a good indication to the teacher how

students will perform on the End of Course Examination with less than one month to review the

content. As indicated by Figure 3 above, On the Final Benchmark, students increased their average

mastery to 62%. Although students did not make as much progress toward mastery as they did

previously, but still gained and average of 14 more percentage points from the Midpoint

Benchmark. The total amount of growth shown from the Initial Benchmark to the Final Benchmark

was and average gain of 41% which is almost a 200 percent gain in average mastery as indicated in

Figure 4 below.
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 11

Figure 4: Initial vs. Final Benchmark Assessment- USA Test Prep 2018-19 GSFH

After analyzing the quantitative data, it was important that I also analyze the qualitative

data of observation anecdotal notes, students interviews, students surveys, and teacher interviews.

When I asked the students to rate Khan Academy in the areas of Instructional Videos, Practice

Problems, and Point Systems the results were overwhelming clear as shown in Figure 5. Students

found the practice problems to be very good and the scoring system to be very good and even

excellent, but the instructional videos to be only satisfactory. In a following up question as shown

in Figure 6, students were asked to rate the Khan Academy Instructor. Most students said that the

information was clear, organized, and effective, but did not stimulate student interest. Additionally,

in student interviews about the videos specifically, students shared that it was a great way to get

information, but the instructional videos were not engaging enough and some said that the videos

were simply boring.

In addition to the students appreciating the scoring and points system which allows you to

gain energy points that can be used to upgrade your avatar, the area that seemed to be most

appealing and effective for students was immediate feedback also show in Figure 6 below. The
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 12

majority of students found that feature to be very good and even excellent feature of Khan

Academy.

Figure 5: Student rating of Khan Academy 2018-19 GS

Figure 6: Student rating of Khan Academy Instructor 2018-19 GSFH

In Figure 7 students were asked to indicate the rating for which they feel Khan Academy

has contributed to their mastery of the Algebra 1 content. The majority of students were neutral to

the question but with a greater number agreeing than disagreeing that Khan Academy played a role

in their mastery. As a follow up, I asked the students what could make Khan Academy more
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 13

effective besides making the videos more engaging and the overwhelming majority said that they

should put forth more effort and use Khan Academy more regularly, particularly at home.

Figure 7: Contribution Khan Academy has played in mastery of Algebra 1 course- Student Survey

2018-19 GSFH

When asked what role do they believe Khan Academy has played in their journey toward

mastery of the Algebra 1 content, most students answered, “Khan Academy has helped me with

things I do not understand in class. It has been good with practicing on topics and the videos are

some of the best sources for learning any math topic. It helped me practice and learn some things I

didn't know. It gave a different explanation and view on the same problems taught in class.” There

were also a few students that said Khan Academy did not play a role in their mastery of Algebra 1

this year, but looking over the data, the teachers pointed out that those students tended to be the

ones that did not take advantage of all opportunities to use Khan Academy to its maximum

potential.
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 14

Findings:

After reading and reflecting on the studies of others and conducting my own action research

project on the implementation of Khan Academy in Algebra 1 class, I have determined that Khan

Academy is an effective digital tool. The student’s growth this year with the implementation of

Khan Academy paired with interviews, observations, and previous research lead me to believe that

Khan academy had a positive influence on student academic success in Algebra 1. Just as the

researchers in four of the five studies listed found, there was a positive influence from Khan

Academy with students. Kronholz, Dickerson, and Chan, O’Connor, and Peat’s research all parallel

what I found in my research. The research indicates that students show improvement in

understanding and mastery when using Khan Academy when paired with good instruction and

pedagogy.

In reflecting on this action research project, I will continue to advocate for the integration of

Khan Academy in Algebra 1 classrooms. I plan to add a Khan Academy session to our school

based summer Professional Development where I can share the research from this study. I also plan

to send suggestions and praises to Khan Academy based off of the quantitative and qualitative data

gathered during my action research project. In addition, I am already working on future action

research to support the use of Khan Academy over other grades and disciplines. I am curious to see

if we would see the same success in growth in classes with other levels, younger grades, high

school students, and other content areas. I look forward to those continued projects.
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 15

References

Chan, M., O'Connor, T., & Peat, S. (2016). Using Khan Academy in Community College

Developmental Math Courses: Results and Lessons Learned from Developmental Math

Demonstration Project. New England Board of Higher Education,1-8. Retrieved February

17, 2019.

Dickerson, B. D. (2016). One-to-One Mobile Devices in Rural School Districts: A Mixed Methods

Study Investigating the Impact Impact of Khan Academy on Mathematics Achievement and

Teacher Pedagogy (Doctoral dissertation, Northwest Nazarene University, 2016) (pp.

1-175). Ann Arbor, MI: ProQuest. Retrieved February 17, 2019.

Kelly, D. and Rutherford, T. (2017). Khan Academy as Supplemental Instruction: A Controlled

Study of a Computer-Based Mathematics Intervention. International Review of Research in

Open and Distributed Learning (NC State University, 2017), 18(4), 1-8. Retrieved February

17, 2019, from ​https://files.eric.ed.gov/fulltext/EJ1146230.pdf​.

Kronholz, J. (2012). Can Khan Move the Bell Curve to the Right? Education Next,12(2), 16-22.

Retrieved February 16, 2019, from

http://educationnext.org/can-khan-move-the-bell-curve-to-the-right/

Weeraratne, B., & Chin, B. (2018). Can Khan Academy e-Learning Video Tutorials Improve
INFLUENCE OF KHAN ACADEMY ON ALGEBRA 1 SUCCESS 16

Mathematics Achievement in Sri Lanka? International Journal of Education and

Development Using Information and Communication Technology,14(3), 93-112. Retrieved

February 16, 2019, from ​https://files.eric.ed.gov/fulltext/EJ1201489.pdf​.