EDIT 677: Digital Assessment Project Jeff Pilcher

USATestprep Standards Based Practice as Review

Introduction:

​Standardized assessments are increasingly rigorous for middle school science students.

The South Carolina Pass Science Test for 6 grade blueprint includes 55 questions that assess five

standards. These assessment items include multiple choice questions and technology enhanced

items written at DOK levels 1,2, and 3. As a 6th grade science teacher, it is critical that I provide

standards based instruction that is rigorous and assesses my students in a way that mirrors the

Pass test. I am fortunate to have access to USATestprep which is aligned to South Carolina

Science Standards. Students who use this digital practice with fidelity typically score higher on

unit test, benchmarks, and Pass testing. This led me to the research question: Does using the

USATestprep 6th Grade Science Standards Based Playlist as test preparation increase Science

Assessment Scores?

In order to get more of my students to use this program with fidelity, I have shifted my

requirements to include using the Standards Based Practice playlist as required review for unit

assessments in my 6th grade science class. These playlist include videos, vocabulary, questions,

and performance tasks for students. I also included a tracking system aligned with the PASS

Mastery cut scores as a way to incentivise the practice. The students mastery is measured using a

dot rank for each standard they practice. I look forward to seeing if the “Playlist” type feature

and immediate feedback will increase students’ science standards mastery level.

Review of the Literature:

Before beginning my research, I reviewed relevant literature to help me clarify my

research question and provide me with additional information relevant to digital testprep review.
EDIT 677: Digital Assessment Project Jeff Pilcher

When selecting articles I tried to find studies and literature related to the specific digital content

provider USATestprep, importance of learner feedback, and implications for student motivation.

The article “ Evaluating the effectiveness of the USATestprep intervention to increase

high school test scores” was a research study conducted to see if using USATestprep affected

English II test scores in two Mississippi High Schools. This study included data collection of

assessment scores and teacher opinion surveys. My study used a similar approach but I choose to

include student opinion surveys as my qualitative data. The results of the study concluded that

teachers believe USATestprep is beneficial when it is implemented correctly. Because of

accountability measures, “Administrators and teachers have to be prepared to use and view data

to understand where their students are in academics and create ways to improve (Christian,

2012).” USATestprep proved to be a good solution. This study proved beneficial to my research

because it made me confident in my decision to use USATestprep. When completing my initial

planning of the research, I did not consider getting students’ opinions of the test review. This

study helped inform me on the importance of including qualitative data in my action research.

Another article I reviewed that specifically studied USATestprep was “Increasing

Inclusive Students' Achievement Through Use of USATestPrep’s Integrated Learning Systems.”

This study focuses on determining what barriers are keeping teachers from implementing

USATestprep with students in rural South Carolina. The study concluded that time, leadership,

and resources were the most significant barrier preventing students from implementing UTP. The

research suggests that both non-disabled and disabled students benefited using USATestprep.

Most of the barriers identified in this setting do not apply to my school’s implementation of

USATestprep. However, the study made me realize that time could become a factor in my action
EDIT 677: Digital Assessment Project Jeff Pilcher

research. “Time was believed to be an area of concern in terms of teachers having not having

enough time to learn the program or to implement it successfully” (Darnell, 2017). This

information made me realize that if students were successfully using USATestprep, it was most

likely because it was not implemented correctly. I could not just let students work on

USATestprep at their leisure with no requirements or accountability. This led me to create a

visual pacing calendar for students in addition to grading the work. The students would be given

a week to complete each indicator.

I also wanted to explore the importance of providing learner feedback to students.

“Defining the Learner Feedback Experience” discusses the 6 methods of feedback. These

include “namely, timeliness, frequency, distribution, source, individualization, and content”

(Crisp, 2019). Different methods of feedback have different effects on people. This article

raised awareness to me that USATestprep is limited in the type of feedback it provides. This

could affect performance levels for students. Because of this information, I made sure to include

questions about the feedback USATestprep provides on my student opinion surveys.

The article “Formative Use of Summative Tests: Using Test Preparation to Promote

Performance and Self-Regulation” focuses on how testprep impacts student performance on

summative assessment. According to this study test preparation is defined as “strategies which

facilitate the formative use of summative tests” (Lam, 2013). The purpose of the study was to

determine if “self-regulation” student practices on formative assessment helps students be better

prepared to be successful on summative assessments. The results suggests that effective testprep

does have positive results on summative assessments. It was also determined that it is

“important to utilize test preparation strategies to get students actively involved in the assessment
EDIT 677: Digital Assessment Project Jeff Pilcher

process, and motivate them to become agents in constructing knowledge for their learning”

(Lam, 2013). These finding were the most critical when planning my research. The implications

on my process included developing a plan to motivate students to be self regulated when

working on USATestprep. USATestprep Standards Playlist is independent practice. As a result, I

created a USATestprep Tracking Incentive System to help get similar results. I also decided to

create a student feedback survey after the unit.

Methodology:

After reviewing the literature, I determined a course of action for my research. I decided

to have my first academic block (20 students) serve as my control group. Students in the control

group would not use the USATestprep standards practice playlist at all as part of review for this

unit. They would complete the review provided in the curriculum map. The experimental group

(20 students) was my second academic class. The experimental group would use the USA

Testprep Standards Playlist for the 6th Grade Animal Unit. This playlist provides students with

practice using videos, questions, performance tasks, and games for each of the indicators for the

standards. The students mastery levels are tracked as they progress through each activity. The

students would use this content as there review throughout the course.

The first step in the process was to create a standards based pretest and posttest for the

Animal Unit. In order to create a valid assessment, I generated an Pre-Test & Post- Test in

USATestprep using the 6th Grade South Carolina Standards for Science. This would serve as

quantitative data. This unit has 7 indicators. My assessment includes 5 items for each indicator,

so all of the indicators are weighted the same on each of the assessment. These indicators mirror

the the indicators of the USATestprep standards practice playlist. The students in both classes
EDIT 677: Digital Assessment Project Jeff Pilcher

took the pretest on the same day using USATestprep assessment feature. I recorded the results

for each students to compare the pretest and posttest assessments. I also recorded the students

current proficiently levels based on the SC Pass Correlations.

Next, I determined how I would track student progress in USATestprep and keep students

motivated. In order to do this I created a​ USATestprep tracker​ for students to track their mastery

levels for each indicator. They received a sticker for the tracking chart and a prize for each

indicator they scored equivalent to met or exemplary. This helped me get students motivated to

do their best work when working on the Standards Playlist. It also provided me with additional

quantitative data for my study. I allocated the same amount of class time for review in my first

academic class, but the review did not consist of any activities in USATestprep.

Upon completion of unit, each student completed the post assessment on USATestprep to

gather summative data. In order to get qualitative data, I had the students who completed the

USATestprep Standards Playlist complete a survey on Google Forms to gather feedback. The

questions focused on effectiveness of the review, student effort, use of program feedback, and

feedback on incentives. The survey was anonymous and provided them the opportunity to

provide open ended feedback as well as answer multiple choice items.

Analysis:

In order to determine the effectiveness of USATestprep Standards Based Practice I

analyzed both my quantitative and qualitative data. I started by analyzing the pretest data from

both groups. I was satisfied that pretest assessment were similar between each Then I used the

data provided in USATestprep to look at mastery in the Standards practice for group 2

(experimental group). I used this data to find correlations to mastery in USATestprep Standards
EDIT 677: Digital Assessment Project Jeff Pilcher

Practice Proficiency levels and the Unit Assessment Proficiency levels. After analyzing this data,

I looked at the qualitative USATestprep Feedback Data using the responses Google Forms. This

data provided me insight on the students opinions of the effectiveness of using USATestprep as

testprep review. Finally, I analyzed the posttest data for each of the groups. All assessment

scores are color coded to match the SC Pass performance levels. I used this data to draw

conclusions on my research question: ​Does using the USATestprep 6th Grade Science

Standards based playlist as test preparation increase Science Assessment Scores?

Proficiency Levels:

PASS Proficiency Levels 

0 - 39 40 - 49 50 - 64 65 - 100
Not Met Approaches Met Exemplary

Pretest Results:

Group 1: Pretest Individual Scores- ​The table below outlines the Group 1 students Pretest

scores on the USATestprep Pretest. The class average was a 40% , which would correlate with

an approaches proficiency level.

Group 1 Students: Control Group

Student Pretest Score

1 20

2 43

3 71

4 29

5 63

6 34
EDIT 677: Digital Assessment Project Jeff Pilcher

7 43

8 23

9 63

10 60

11 31

12 54

13 54

14 34

15 31

16 69

17 14

18 40

19 77

20 46

Class Average: 40

Group 1: Pretest Item analysis- ​The graph below shows the percentage of students who

answered each question correctly.

EDIT 677: Digital Assessment Project Jeff Pilcher

Group 2: Pretest Scores Group 2-​The table below outlines the Group 2 students Pretest scores

on the USATestprep Pretest. The students average on the pretest was a 35%, which would be a

proficiency level of not met.

Group 2 Students: Control Group

Student Pretest Score

1 14

2 26

3 43

4 34

5 20

6 49

7 49

8 31

9 34

10 23

11 43

12 63

13 34

14 40

15 31

16 6

17 37

18 40

19 37

20 49
EDIT 677: Digital Assessment Project Jeff Pilcher

Class Average: 35

Group 2:Pretest Item analysis- ​The graph below shows the percentage of students who

answered each question correctly.

Group 2: Standards Practice Mastery - ​This data shows the mastery level for each standard in

USATestprep using the Standards Playlist. (Note: USATestprep uses different colors for their

proficiency levels. The average proficiency level is Exemplary.

Group 2: USATestprep Student Feedback- ​The results of Group 2 student surveys are listed

below.
EDIT 677: Digital Assessment Project Jeff Pilcher
EDIT 677: Digital Assessment Project Jeff Pilcher
EDIT 677: Digital Assessment Project Jeff Pilcher

Posttest Results:

Posttest: Group 1 Results- ​The table below includes the students pretest score, posttest score,

proficiency level, and growth.

EDIT 677: Digital Assessment Project Jeff Pilcher

Group 1 Students: Control Group 

Student  Pretest Score  Posttest Score  Growth 

1  20  46  +26 

2  43  47  +4 

3  71  86  +15 

4  29  29  +0 

5  63  66  +3 

6  34  49  +13 

7  43  37  -6 

8  23  80  +57 

9  63  60  -3 

10  60  46  -14 

11  31  Moved   

12  54  71  +17 

13  54  57  +3 

14  34  60  +24 

15  31  43  +12 

16  69  80  +11 

17  14  26  +12 

18  40  60  +20 

19  77  83  +6 

20  46  77  +31 

Class Average:  40  59  +12 

EDIT 677: Digital Assessment Project Jeff Pilcher

Posttest: Group 2 Posttest results- ​The table below includes the students pretest score, posttest

score, proficiency level, and growth.

Group 2 Students: Experimental Group 

Student  Pretest Score  Posttest Score  Growth 

1  14  89  +75 

2  26  83  +57 

3  43  49  +6 

4  34  97  +63 

5  20  69  +49 

6  49  54  +5 

7  49  86  +37 

8  31  83  +52 

9  34  91  +57 

10  23  60  +37 

11  43  77  33 

12  63  Absent   

13  34  40  +6 

14  40  31  -9 

15  31  51  +20 

16  6  80  +74 

17  37  34  -4 

18  40  69  +29 

19  37  57  +20 

20  49  77  +28 

Class Average:  35  67  +31.75 

EDIT 677: Digital Assessment Project Jeff Pilcher

Group 1 & Group 2 Comparison: ​This table compares the growth and proficiency levels on

the Unit Post Assessment.

Group 1 & Group 2 Comparison: 

Group 1   Class Average:  40  59  +12 

Group 2  Class Average:  35  67  +31.75 

Findings:

After reviewing the literature, designing and carrying out my action research, and

analyzing my data, I concluded that using the USATestprep Standards Playlist Review increases

assessment scores. The students who used USATestprep as review gained an average of +31.75

points and would be predicted to score Exemplary on the animal unit of SC Pass. The students in

Group 1 made average gains of +12 points and would be predicted to score Met on Pass.

The results of the student feedback survey revealed that 90% of students worked harder

or as hard when working in USATestprep and 70% of students felt it positively impacted their

assessment grades. Sixty five percent of students participating in the USATestprep Standards

review felt that the incentive motivates them to work harder. Research showed that feedback

was critical for student success and motivation. My data also supported that students used the

feedback and felt it was beneficial. The survey results indicate 85% of students read and used the

feedback provided when they missed a question. Of these students 90% of students felt it helped

them. Sixty Five percent of students read the feedback for questions that were answered correctly

and 60% felt that it was helpful. The open ended feedback provided by students was not as

specific as I would have preferred. This item should have been more specific to get better
EDIT 677: Digital Assessment Project Jeff Pilcher

results. My research suggest that all of my middle school science students could benefit from

using USATestprep Standards Playlist as review for each unit. For my next unit, all of my

students will be using USATestprep to review for our unit assessments.

Overall, my results were reflective of the literature I reviewed. Prior to conducting this

research I did not realize how significant the role of feedback is on students, especially middle

schoolers. According to Crisp, the feedback that USATestprep would be considered “ context

feedback.” This feedback provides students with feedback if the question was answered correctly

or incorrectly with elaborative feedback. This is one of the more effective way to provide learner

feedback. This suggests that any review or formative assessment is more effective when teachers

or programs provide students with this type of feedback. As a middle school teachers this will

impact my future instruction. I realize that my students that were not using USATestprep were

only provided with “ knowledge” or “verification” feedback. It is essential that I give my

students more elaborative feedback when working on activities, formative assessment, and

projects throughout the learning cycle.

In completing this project, I was surprised that the incentive system did not have as much

of impact on the students. This leads me to believe that I need to revisit by design of the

incentive program. I may need to individual the incentive program and get student input on

designing my incentives. I also realized that my incentive is currently only measuring

proficiency. This could be negatively affecting students who are showing growth but not

proficiently. This could be impacting the students opinions on the incentive program. This leads

me to want to do additional research on using incentives to motivate students by tracking using

growth and proficiency levels.

EDIT 677: Digital Assessment Project Jeff Pilcher

Works Cited

​Crisp, E. A., & Bonk, C. J. (2018). Defining the Learner Feedback Experience. TechTrends:

Linking Research & Practice to Improve Learning, 62(6), 585–593.

https://doi-org.login.library.coastal.edu:8443/10.1007/s11528-018-0264-y

Christian, V. F. (2012). Evaluating the effectiveness of the USA testprep intervention to

increase high school test scores. Retrieved from

http://login.library.coastal.edu:2048/login?url=https://search-proquest-com.login.library.coastal.edu:8

443/docview/1237877870?accountid=26722

Darnell Elom, R. (2017). Increasing Inclusive Students' Achievement Through Use of

USATestPrep 's Integrated Learning Systems. WALDEN DISSERTATIONS AND DOCTORAL

STUDIES. Retrieved from

https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=4981&context=dissertations​.

Lam, R. (2013). Formative Use of Summative Tests: Using Test Preparation to Promote

Performance and Self-Regulation. Asia-Pacific Education Researcher, 22(1), 69–78. Retrieved from

http://login.library.coastal.edu:2048/login?url=http://search.ebscohost.com/login.aspx?direct=true&db

=eric&AN=EJ1180457&site=eds-live