Introduction

Teaching 21st century learners requires teachers to regularly evaluate their teaching

methods and testing modalities to ensure that we are nurturing the skills kids will need to be

successful in this quickly advancing world. In my own teaching practice, I have contemplated the

best mode of administering assessments in order to meet the needs of all my learners. The

specific question that I have aimed to address in my study is: Is student achievement greater

when assessments are administered digitally or paper copies? Historically, assessments have

been administered to students on paper and pencil. More often lately you see assessments, both

high-stakes and low stakes, administered digitally. I wanted to find out whether students were

more successful using a digital testing platform versus traditional paper and pencil tests. To

experiment with this question, I tested 12 of my fifth-grade students over the course of a grade-

level reading unit. I assigned assessments to a treatment group and a control group in two

opposite formats- digital and paper copies. The assessments ranged from low stakes to high

stakes in order to collect a range of assessment type data. I used this data in addition to

observations, interviews and numerous scholarly articles relating to this topic to formulate my

own assumption of which testing modality is most effective.

Review of Literature

Before conducting my own research, I spent some time reviewing literature that explored

the findings of several other studies surrounding the topic of testing modalities. These studies

helped me to make more informed decisions about the way I would execute my own experiment

and collect data, both quantitative and qualitative.

It was initially revealed to me in reading Bridget Anakwe’s Comparison of Student

Performance in Paper-Based versus Computer-Based Testing that among the many studies in
which researchers compare student performance outcomes in online courses versus traditional

courses, there was largely found to be no significant difference in student performance. This

article investigates how student performance differs on an accounting exam between paper-based

testing and computer-based testing. Anakwe points out that her study differs from previous

studies in that her focus is solely addressing the testing methods of traditionally taught courses

rather than the course as a whole. The study revealed that across three cohorts of accounting

students, there was no notable difference in test scores between online and traditional testing

methods (Anakwe, 2008). When I first began thinking and planning my own research, I was

confidently anticipating finding more student success in paper testing. Upon reviewing this first

piece of literature, my thinking began to shift towards a more open-minded expectation for what

type of results may be uncovered within my classroom.

The next article I read was from a study that was presented at the Annual International

NARST Conference in which researchers compared student performance on paper-based tests to

three different types of computer-based tests. The computer-based tests varied in their navigation

and answer selection features. The study was conducted using a sample of 9,698 fourth through

twelfth grade students, all of whom were enrolled in a science course. Cari F. Herrmann-Abell,

Joseph Hardcastle, and George E. DeBoer concluded in Comparability of Computer-Based and

Paper-Based Science Assessments that paper-based testing and computer-based testing produce

equivalent results. In addition to these findings, the researchers also found that students are more

successful when the computer-based test allows test-takers to directly click on an answer choice

and that there is no impact on free versus restricted navigation. When surveyed, elementary and

middle school age students largely prefer computer-based tests over paper-based tests

(Herrmann-Abell, et al., 2018). I found it interesting that again, there seemed to be so little
disparity between paper-based testing and computer-based testing results. I also never considered

the different factors of an online test that could impact student success. This article inspired me

to incorporate a more quantitative piece into my research in which I surveyed kids on their

preferences and perceptions of online tests versus traditional paper tests.

Another study that inspired me to consider additional factors and implement more

qualitative data into my research was one conducted at the University of Florence. This study

consisted of 606 students who were analyzed while using their preferred method of testing (paper

or digital). Those who selected computer-based testing used their own personal devices and were

asked to participate in a survey where they rated their perceived self-efficacy and satisfaction

when testing digitally. In their study, Andrea Nardi and Maria Ranieri explored the potential for

paper-based testing to successfully be replaced by computer based-testing as well as the

relationship between computer-based testing and student self-efficacy and satisfaction. The

results of this study found that students performed better on computer-based tests and that a

positive relationship exists between students and their recognition of self-efficacy and

satisfaction when taking digital tests (Nardi and Ranieri, 2019). Before reading this article, I’d

never considered the idea of self-efficacy or positive and negative relationships among students

and testing modes. This theory led me to survey my students beyond just their preference of one

mode over the other and to consider the factors that led them to their selection. In order to gather

this qualitative data, I added a section to my student survey in which students had the

opportunity to elaborate on their preferred testing method and explain their decision-making

process. These raw thoughts and opinions from students proved to be the most insightful

resource upon my own reflection and planning next steps in my classroom.

 As I moved through the literature surrounding this topic, I found that in studies which

paper testing versus digital testing was being researched, there were always additional factors or

conditions being considered. The research is no longer simply “apples versus oranges”, but

rather an analysis of which conditions contribute to the best development of said “apples” or

“oranges”. Roy Clariana and Patricia Wallace conducted a study comparing different factors

associated with paper-based testing and computer-based testing. The factors considered were

content familiarity, computer familiarity, competitiveness, and gender. This study was conducted

using a sample of 105 college freshman enrolled in a Computer Fundamentals course. 54

students were post-tested digitally, and 51 students were selected as the paper-tested group. This

study found that computer-based tests positively impacted student scores. When identifying

correlations between computer familiarity, competitiveness and gender, no relationship was

found between testing modes. Alternatively, there was a relationship found among content

familiarity and testing mode (Clariana and Wallace, 2002). It was concluded that higher attaining

students performed better on computer-based tests. At this point in my review of the literature

surrounding this topic, I realized that it may be more beneficial for my research purposes that I

rely more on the qualitative data than the number crunching and score analyzing.

Muhammet Yasar Özden, Ismail Ertürk, and Refik Sanli conducted a study on students’

perceptions of online assessments. The purpose of this study was to gather data on what students

think of computer-based testing as well as to gather feedback on desired features of computer-

based testing. Data was collected using a paper-based survey and conducting of student

interviews. The study found that a majority of the students who were surveyed noted key features

such as immediate feedback, randomized question order, item analysis of the questions, and

obtaining the scores immediately after the exam (Ozden, et al., 2004). The researchers agreed
that these features contributed to positive achievement on exams. After reading this article, I was

curious to hear perceptions from my elementary age students of online assessments and the

factors that contribute to positive achievement and success on exams for their age group. I chose

to leave my survey question open ended rather offering them selections in order to allow for

authentic responses of the factors that impact their experience with online testing.

After reading each of these articles and reviewing the findings of many researchers, I was

surprised to see how little difference was recorded when comparing paper-based testing to

computer based testing. It seems that the comparison is no longer one mode versus the other, but

rather the conditions of each respective mode that contribute to more positive achievement. After

some reflection, my perception of my own research question has changed drastically. I was

almost certain that I was going to find more clear success in one mode versus the other. I now

understand the importance of the qualitative data that will come from my study as well as the

different factors I should consider when grouping and surveying students such as preference, test

features, familiarity with devices and test taking platforms, etc.

Methodology

Once I reviewed the literature surrounding my topic, I felt more prepared to formulate my

own assessment instrument in order to collect both quantitative and qualitative data. I planned to

test and survey 12 students in my English Language Arts class.

In order to collect quantitative data, I planned to focus my study on the district-wide unit

collection assessments that are administered at the beginning and end of each English Language

Arts unit. The test is typically 10-15 multiple choice questions based on lengthy reading

passages. The digital version is implemented as a google form where the passages are viewed,

and answers are selected on their screen. The paper version is a traditional document where
students circle their multiple-choice answer. On the day of the pre-test I randomly selected six

students for a heterogeneous treatment group and six students for a heterogeneous control group.

For the treatment group, I administered the fully digital assessment and for the control group I

administered the traditional paper and pencil version. On the day of the post-test, at the end of

the unit, I again administered the digital version to the treatment group and the paper version to

the control group. I chose these assessment instruments and stuck with the same mode of

administering each time in order to see which group would perform better, and also to compare

the results in both low-stakes (pre-test) and high-stakes (post-test) testing environments.

In order to collect qualitative data, I administered a student survey in which they shared

their preferences and perceptions of online tests versus traditional paper tests in both high-stakes

and low-stakes environments. The survey was a simple Google form made up of four questions

where students select either “paper” or “digital” preference, one question gauging student effort

with a “pre-test”, “post-test” or “equal” selection, and one open ended question that allowed for

students to explain their digital or paper testing preferences and perceptions. The questions were

as follows:

1. Do you prefer to take Collection Pre/Posttests digitally or on paper?

2. Is it easier to go back and check your answers when you test digitally or on paper?

3. Which type of test do you feel is more important? Digital or Paper?

4. Do you find it easier to read long texts when they are given digitally or on paper?

5. Do you put more effort into Collection Pretests or Posttests?

6. Explain why you prefer digital or paper Collection Pre/Posttests better. Give 3

reasons supporting your answer.

 Before administering this survey I explained to my students that there were no right or

wrong answers, there is no grade associated with their responses, but rather that I just wanted to

know their thoughts and feelings about the way they display their learning on tests. I made sure

they understood that nobody would see their answers except me, so they had the freedom to be

honest and reflect and share their own feelings.

Analysis

Once the course of my research was complete, I began to analyze the results. I began first

with my qualitative data. I reviewed the results of the student survey to gain an understanding of

my students’ perceptions of online versus virtual testing as well as their narratives of why they

prefer one mode over the other. Summaries of the results are below:
 I organized my quantitative data into two charts. The first chart represents the highest,

lowest and average scores from the Pre-Test for both digital and paper responses. The second

chart represents the highest, lowest and average scores from the Post-Test for paper and digital

responses as well. I thought that these three data points would give the best overview of the

results from each respective testing modality. The charts are presented below:

Pre-Test (Low Stakes)

Highest Score Lowest Score Class Average

Digital 81% 27% 54%

Paper 63% 18% 44%

Post-Test (High Stakes)

Highest Score Lowest Score Class Average

Digital 81% 54% 71%

Paper 72% 27% 60%

Once my data was compiled, I took a moment to reflect on the validity of my data. I

contemplated whether the effects of the nature of this school year had any impact on the

outcomes of my research. Due to Covid-19 protocols, this school year has been unlike any other.

Did a year of digital learning positively or negatively impact my students’ views of digital

testing? Would my results have been different had this school year been more traditional? I

concluded that there was no true way for me to weigh the impact of the pandemic on my

students’ performance without conducting further research in the years to come.

Findings
 After much reflection and analysis of the data produced from my research, I was

surprised to see such little disparity among the quantitative data. Though my own expectations

were not met, the quantitative results did align well with the literature I reviewed previously. As

seen in the charts above, class averages between digital and paper testing only revealed a 10-11-

point difference. As a whole, students only achieved slightly more success when testing digitally

compared to on paper- not vastly different. When comparing the Pre-Test averages to the Post-

Test averages, there is a much clearer disparity between high-stakes testing results and low-

stakes testing results.

In addition to the quantitative data revealing little disparity, the surveys of student

perceptions of digital testing versus paper testing revealed a fairly moderate and balanced

response as well. When directly asked their preference, approximately half of the students chose

digital and half chose paper. It was interesting though, that almost 70% of students selected that

they feel paper tests are more important than digital tests. I attribute this response to the fact that

end-of-year, high-stakes testing is traditionally administered on paper. Adversely though, 70% of

students reported that they put equal effort into collection pre-tests (low stakes) as they do post-

tests (high stakes).

The most interesting aspect of this research study was the responses I received when

asking students to explain and elaborate on why they prefer one testing mode over the other.

Many students cited the ability to focus, internet connectivity issues, ease of reading and ability

to make notes as reasons which would support paper-testing. In support of digital testing, kids

explained that they preferred typing rather than writing, digital navigation rather than flipping

pages and immediate feedback at the completion of a digital test. I found the qualitative data to

be the most insightful to me as a teacher. Moving forward, I plan to allow my students to choose
their preferred method of testing when taking Collection Pre and Posttests. I am curious to find

out whether this will impact student success in the future. I look forward to continuing this

research as the school year concludes.

 References

Anakwe, B. (2008). Comparison of Student Performance in Paper-Based versus Computer-Based

Testing. Journal of Education for Business, 84(1), 13–17.

Clariana, R., & Wallace, P. (2002). Paper-Based versus Computer-Based Assessment: Key

Factors Associated with the Test Mode Effect. British Journal of Educational

Technology, 33(5), 593–602.

Herrmann-Abell, C. F., Hardcastle, J., & DeBoer, G. E. (2018). Comparability of Computer-

Based and Paper-Based Science Assessments. Grantee Submission.

Nardi, A., & Ranieri, M. (2019). Comparing Paper-Based and Electronic Multiple-Choice

Examinations with Personal Devices: Impact on Students’ Performance, Self-Efficacy

and Satisfaction. British Journal of Educational Technology, 50(3), 1495–1506.

Ozden, M. Y., Erturk, I., & Sanli, R. (2004). Students’ Perceptions of Online Assessment: A

Case Study. Journal of Distance Education, 19(2), 77–92.