1

An Analysis of a Virtual Reality Experiment by Barsom et al. (2020)

Jennifer Guth

Faculty of Education, University of British Columbia

ETEC 500: Research Methodology in Education

Dr. Oksana Bartosh

April 11, 2021

 2

An Analysis of a Virtual Reality Experiment by Barsom et al. (2020)

A popular topic in educational technology is the use of emerging technologies, such as

virtual reality (VR), to enhance learning effectiveness and engage learners. Barsom et al. (2020)

present a study on the use of VR in the teaching of cardio pulmonary resuscitation (CPR) to

high school students. The study, though not without its own weaknesses and gaps, does

manage to determine that the VR model is effective at increasing CPR knowledge and

perceived self confidence in high school students.

The purpose of this study is to design and test a VR simulation to teach high school

students the steps of CPR with measures of test scores and a questionnaire measuring

confidence in students CPR knowledge (Barsom et al., 2020). The authors wanted to build on

earlier studies by Creutzfeldt et al. (2012) and Aksoy (2019) that showed benefits of serious

gaming and VR for teaching CPR in health care students. Barsom et al. (2020) were able to

show that serious gaming and VR is also beneficial in teaching CPR to high school students.

Barsom et al. (2020) also wanted to see if the findings of Judd (2018) held in their study that the

VR group would also have greater self confidence than alternative methods. The authors did

observe greater perceived self confidence in their study but the lack of statistical analysis on this

point to show the self confidence was greater than that likely by chance alone was a weakness

of the study.

The study is an experimental design with pretest and post test, randomization into

control and experimental groups and a post test survey or questionnaire. The pre and post test

measures theoretical CPR knowledge and CPR sequencing while the survey measures how

students felt after the training in various measures including self confidence (Barsom et al.,

2020). The pretest provides additional internal validity that the groups were similar in knowledge

prior to the experimental condition. The pre and post test were the same but the questions in the

post test were in a different order to control for memorization effects (Barsom et al., 2020).

Barsom et al. (2020) describe the pre and post test as a standard online pretest but it is unclear
 3

whether they received it from a CPR training agency such as the Red Cross. The questions are

available in an appendix and appear appropriate though using a test from a reputable agency

may increase external validity. The 18 question survey uses a five‐point Likert scale to measure

self confidence, perceived competence, and other course factors. The survey was not validated

by a pilot representative sample and expert review which would have increased internal and

external validity (Artino et al., 2014). Barsom et al. (2020) acknowledge this in their limitations

section and indicate that this may have introduced subjectivity but the questions do not appear

to be obviously biased or flawed. Overall the test and survey appear non biased and are

included for review by the authors for greater transparency.

The control and experimental groups both complete an elearning module then the

control group watches CPR videos while the experimental group completes an interactive VR

CPR module. The VR intervention being interactive while the video was not introduced a

potential confounding of interactivity on technology used in instruction. A study by Ulrich et al.

(2019) showed that use of 2D video compared to 3D video (non interactive VR) showed no

statistical difference in outcomes measured but that in class learning which may have been

more interactive beat both technology interventions. This in fact may have introduced an

alternative hypothesis that higher interactivity of CPR training increased CPR knowledge and

perceived self confidence among high school students. The use of 2D version of the interactive

VR on a tablet would have been a more appropriate control to test against in this instance to

prevent this confounding. However due to how the hypothesis was phrased by the authors the

study remains valid despite the confounding.

The independent variables are the control and experimental groups which vary by use of

video or VR in addition to an elearning module. The dependent variables measured are scores

on the pre and post test which approximate CPR knowledge and the survey scores measuring

perceived competence and self confidence. Attribute variables are not used for comparison only

to check for roughly equal groups according to sex and age.

 4

The sample size was 40 high school students in the Netherlands who were randomly

assigned to the equal in size control and experimental groups (Barsom et al., 2020). The gender

and age characteristics were not matched but when compared did appear to create 2 roughly

equal groups and p value was not statistically significant (Barsom et al., 2020). Barsom et al.

(2020) indicate the sample size was based on a formal power analysis but does not provide the

calculations and assumptions used. Having sufficient power is important to prevent type 1 or

type 2 errors. Suter (2012) indicates for a large effect size a sample size of at least 25 in each

group is required which is not met by Barsom et al. (2020) but they would meet the chi-squared

rule of thumb that requires a minimum of 20 participants (Suter, 2020). Barsom et al. (2020) also

observe that a larger sample size could have provided greater accuracy. From the

acknowledgments it appears like a single high school and perhaps a single class within was

targeted for student voluntary participants (Barsom et al., 2020). This may result in the findings

not being as broadly applicable to all high school students in the Netherlands or in the world.

The authors could have considered getting students from a range of high school ages, most

participants were 16 years of age, and multiple schools to make results more broadly applicable

(Barsom et al., 2020). Though Barsom et al. (2020) could have obtained a larger or more

diverse sample size there is no data indicating the the sample chosen would not be

representative of high school students as a whole.

The study is controlled by excluding participants who had taken CPR training in the last

year to prevent an extraneous variable on previous CPR training (Barsom et al., 2020). Barsom

et al. (2020) indicated past studies that CPR skills depreciate quickly over one year. It would

have been good for authors to check this assertion by checking this variable for internal validity

by asking all students the amount of time since last CPR training. Depending on the number

who had past training the pretest scores between those who had or hadn’t had previous training

could have been compared. Sixteen year olds could be swim instructors or first aid instructors

themselves which could create another extraneous variable, but those students would also be
 5

more likely to have received CPR training in the previous year. Alternatively Barsom et al.

(2020), since they indicate only 17% of Dutch high school students know how to perform CPR,

could have excluded all students with previous training in CPR to eliminate error if students’

CPR knowledge decayed less rapidly than other studies found. Recent media/public service

announcements on CPR would likely impact both groups equally and not be an extraneous

variable. Given that the authors did cite literature supporting their position for using the one year

exclusion would appear to be an adequate, though not ideal, control for this variable.

Barsom et al. (2020) analyzed the test scores using a Mann‐Whitney U test. The Mann-

Whitney U test as opposed to the t test does not assume a set of data is normally distributed

and is typically used for nominal or ordinal data (Suter, 2012). Barsom et al. do not provide a

rationale for the choice of test but Suter (2012) indicates some educational researchers feel that

achievement scores are ordinal rather than interval data which could explain this choice. The

test scores were described using the median and interquartile range. The survey was described

using graphs showing the percentage of each group who agree, disagreed or was neutral about

each statement but was not statistically analyzed (Barsom et al., 2020). Barsom et al. (2020)

found statistically significant differences in the two groups in post test scores between the

groups.

Graphs showed that students in the VR group agreed with the survey questions on self

confidence to a higher degree than the video group but that was not true for perceived

competency where both groups agreed roughly the same amount (Barsom et al., 2020). Barsom

et al. (2020) concluded that the VR model they developed was effective for increasing CPR

knowledge and self confidence in high school students. The results section supports the

conclusion that the CPR knowledge was significantly increased based on test scores. Though

descriptively we can see that perceived self confidence was higher in the experimental group

the lack of statistical analysis of this conclusion when statistical analysis was possible and the
 6

lack of survey validation leads to some doubt on this part of the conclusion. Also it is important

to note that Barsom et al. (2020) conclude that their particular interactive VR model is effective

rather than VR in general is effective. Given the confounding of interactivity a claim that VR is

superior to video would not be as valid. If the confounding was absent we could more generally

discuss the benefits of VR over video or the importance of interactivity in learning more

generally.

The implications of the study are that this interactive VR module could be deployed in

high schools throughout the Netherlands to increase the number of high school students trained

in CPR, a current petition by the Red Cross there (Barsom et al., 2020). This VR module could

also be deployed for refresher training or as a supplement to instructor-led courses though

Barson et al. (2020) acknowledge the limitations of the device in areas like learning how to get

the proper depth of chest compressions. Though the study has some weaknesses they still are

able to prove their hypothesis that “VR Resuscitation Training is an effective learning method

regarding knowledge and increasing self‐confidence for high school student trainees” (Barsom

et al., 2020, p. 2059). The model could gain traction to improve CPR training access to high

school students in the Netherlands and beyond.

 7

References

Aksoy, E. (2019). Comparing the effects on learning outcomes of tablet-based and virtual reality-

based serious gaming modules for basic life support training: Randomized trial. JMIR

Serious Games, 7(2), e13442. https://doi.org/10.2196/13442

Artino, A. R., La Rochelle, J. S., Dezee, K. J., & Gehlbach, H. (2014). Developing

questionnaires for educational research: AMEE guide no. 87. Medical Teacher, 36(6),

463-474. https://doi.org/10.3109/0142159X.2014.889814

Barsom, E. Z., Duijm, R. D., Dusseljee-Peute, L. W. P., Landman-van der Boom, E. B., van

Lieshout, E. J., Jaspers, M. W., & Schijven, M. P. (2020). Cardiopulmonary

resuscitation training for high school students using an immersive 360-degree

virtual reality environment. British Journal of Educational Technology,

51(6), 2050-2062. https://doi.org/10.1111/bjet.13025

Creutzfeldt, J., Hedman, L., & Fellander-Tsai, L. (2012). Effects of pre-training using serious

game technology on CPR performance - an exploratory quasi-experimental

transfer study. Scandinavian Journal of Trauma, Resuscitation and

Emergency Medicine, 20(1), 79. https://doi.org/10.1186/1757-7241-20-79

Suter, W. N., & SAGE Research Methods Core. (2011). Introduction to educational research: A

critical thinking approach (2nd ed.). SAGE.

Ulrich, F., Helms, N. H., Frandsen, U. P., & Rafn, A. V. (2019). Learning effectiveness of 360°

video: Experiences from a controlled experiment in healthcare education.

Interactive Learning Environments, 29(1), 98-111.

https://doi.org/10.1080/10494820.2019.1579234