International Journal of Medical Informatics 141 (2020) 104226

Contents lists available at ScienceDirect

International Journal of Medical Informatics

journal homepage: www.elsevier.com/locate/ijmedinf

A comparative case study of 2D, 3D and immersive-virtual-reality T

applications for healthcare education
Omar López Cháveza, Luis-Felipe Rodrígueza, J. Octavio Gutierrez-Garciab,*
a
Instituto Tecnológico de Sonora, Cd. Obregón, Sonora 85000, Mexico
b
ITAM, Río Hondo 1, Ciudad de México 01080, Mexico

ARTICLE INFO ABSTRACT

Keywords: Background and objective: The workings of medical educational tools are implemented using a myriad of ap-
Healthcare education proaches ranging from presenting static content to immersing students in gamified virtual-reality environments.
Virtual reality The objective of this paper is to explore whether and how different approaches for designing medical educational
Comparative case study tools affect students’ learning performance.
Materials and methods: Four versions of an educational tool for the study of clinical cases were implemented: a
2D version, a gamified 2D version, a gamified 3D version, and a gamified immersive-virtual-reality version. All
complying with the same functional requirements. Each version was used and evaluated by an independent
group of students. The participants (n = 78) evaluated the applications regarding usefulness, usability, and ga-
mification. Afterward, the students took an exam to assess the retention of information on the clinical cases
presented.
Results: One-sample Wilcoxon signed-rank tests confirmed that the participants perceived that it was at least
quite likely that gamification helped improved their learning. In addition, based on the participants’ perception,
the gamification of the immersive-virtual-reality version helped the most to improve their learning performance
in comparison with the gamified 2D and 3D versions.
Conclusions: Regardless of whether different versions of a medical educational tool (complying with the same
functional requirements) are perceived as equally useful and usable, the design approach (either 2D, 3D, or
immersive-virtual-reality with or without gamification) affects students’ retention of information on clinical
cases.

1. Introduction innovative methods based on immersive virtual reality to provide si-

Effective and efficient training of medical students is critical to
ensure a reliable future healthcare workforce capable of providing high
quality healthcare services, see [1–3]. In this context, a variety of
medical educational tools exist to complement traditional learning
methods [4–6], which are designed to help students enhance acquired
theoretical knowledge and support practical training by providing
technology-based learning resources [7,8]. Designers of computer-as-
sisted medical educational tools take advantage of diverse technological
approaches, ranging from presenting 2D static content (e.g., [9]), of-
fering fully immersive-virtual-reality environments (e.g., [10]), using
social networks (e.g., [11]), to incorporating gamification elements
such as badges, rankings, challenges, and rewards in serious games
(e.g., [12,13]). More recently, medical educational tools have adopted
mulated realistic environments, see [14–16].
Medical educational tools, regardless of their underlying design
approach, are developed to engage medical students and improve their
learning performance. In this regard, most medical educational tools
reported in the literature have been positively evaluated as tools that
promote and improve learning, see [4,10,14]. As an illustration, in [17]
a game-based prototype was positively evaluated by nursing students as
a useful, usable, and satisfying educational tool. In [18–20] medical
educational tools based on simulated virtual environments and 3D
models were evaluated by medical students, resulting as a realistic,
easy-to-navigate, and useful approach that enhances learning, which is
more effective than traditional methods. Nevertheless, although most
medical educational tools reported in the literature present evaluations,
there is still a need for comparative studies analyzing and identifying

Corresponding author.
⁎

E-mail addresses: omar.lopez@potros.itson.edu.mx (O. López Chávez), luis.rodriguez@itson.edu.mx (L.-F. Rodríguez),

octavio.gutierrez@itam.mx (J.O. Gutierrez-Garcia).

https://doi.org/10.1016/j.ijmedinf.2020.104226
Received 7 April 2020; Received in revised form 25 May 2020; Accepted 16 June 2020
Available online 17 June 2020
1386-5056/ © 2020 Elsevier B.V. All rights reserved.

Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
O. López Chávez, et al.
design approaches and technological elements (e.g., gamification and
virtual reality) that are perceived by medical students as more usable,
useful, and effective toward a given set of learning goals. Moreover, the
benefits of identifying such design approaches and technological ele-
ments go beyond the medical domain, since computer-based educa-
tional tools are commonly implemented and used across multiple dis-
ciplines to improve learning performance, see [29–31].
This paper presents a comparative case study implementing, eval-
uating, and analyzing four versions of a medical educational tool for
training medical students (hereafter referred to as MediTool). The pri-
mary objective is to explore whether and how different approaches for
designing medical educational tools affect students’ learning perfor-
mance. To attain this objective, a group of 78 participants evaluated the
usefulness, usability, and gamification elements of different versions of
MediTool. Afterward, the participants took a paper-based exam to assess
their learning performance. It is important to note that learning perfor-
mance in the present comparative case study was measured as the
ability of the individual to retain information, which is linked to
learning [21,22]. In this sense, the paper-based exam assessed the re-
tention of information by the participants on clinical cases presented in
MediTool. The four versions of MediTool (a non-commercial software)
implemented by the authors were (i) a 2D desktop version presenting
static content; (ii) a gamified 2D version including badges, challenges,
and other gamification elements; (iii) a gamified 3D version using 3D
avatars and non-immersive virtual reality; and (iv) a gamified im-
mersive-virtual-reality version.
2. Related work
In [17] a serious game software prototype was designed to help
nursing students develop clinical reasoning and decision-making skills
associated with the care of patients with chronic obstructive pulmonary
disease in home healthcare settings. The prototype applied gamification
components such as challenges, interaction, and objectives. The pro-
totype was a single-player online game that provides nursing students
with a realistic video-based simulated scenario of a home healthcare
clinical practice, information about the patient, and quiz-based tasks.
The perceived usability of the tool was evaluated by six participants
using questionnaires, interviews, and cognitive walkthroughs. In gen-
eral, the results showed that participants perceived the serious game as
useful, usable, and satisfying.
In [18] it was designed and evaluated a virtual coach in a mixed
reality simulator to guide users during a functional endoscopic sinus
surgery. The virtual coach provides assistance to users by using visual
and auditory cues in an immersive-virtual-reality operating room. This
study involved 17 participants grouped into novice and expert sur-
geons. Participants evaluated the virtual coach using a Likert-type
questionnaire. Both groups of participants regarded the virtual coach as
a useful and beneficial educational tool that enhances learning in a
surgical mixed-reality environment.
In [19] an anatomy-augmented virtual simulation training module
was evaluated to determine whether it contributes to the development
of students’ abilities to place a nasogastric tube. This tool is supported
by video and 3D computer graphics to simulate organ systems and vi-
sualize anatomical structures. The study involved 69 nursing students
divided into control and experimental groups. The former group was
trained on how to place a nasogastric tube using traditional methods,
whereas the latter was trained using the anatomy-augmented virtual
simulation module. After training, participants of both groups practiced
the placement of a nasogastric tube on a manikin and were evaluated by
a group of experts using a 17-item competency checklist. Results
showed that, in general, competency scores were better for participants
trained with the anatomy-augmented virtual simulation module. In
addition, the participants regarded the simulation training module as
realistic, easy to use, and useful.
In [20] a web-based interactive application was developed to train
Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
International Journal of Medical Informatics 141 (2020) 104226
healthcare students in the use of medical instruments. The application
presents 3D virtual models of medical devices for manipulation as well
as self-assessment activities for promoting learning. The perceived ef-
fectiveness of the application was evaluated by 50 participants using a
10-item questionnaire. Participants considered that the application was
simple to use, interactive, and more effective than traditional methods.
In addition, the results of [20] showed that the effectiveness of the
designed learning tool held regardless of the participant age and level of
computer skills.
As seen in this section, most medical educational tools reported in
the literature present evaluations of the proposed tools in terms of us-
ability, effectiveness, usefulness, and satisfaction. However, there is still
a need for comparative studies of medical educational tools reported in
the literature as well as for analyses of the extent to which a particular
technological element such as gamification and/or virtual reality con-
tributes to end users’ perceived usability and perceived effectiveness.
3. Materials and method
3.1. Sample of participants
A sample of 78 undergraduate medical students were recruited from
a local university: 8 first-year students, 6 third-year students, 31 fourth-
year students, and 33 fifth-year students. The only restriction for se-
lection was that students were familiar with the use of computers. The
median age was 22 years (range 18–26) with a standard deviation of
1.65 years. With respect to gender, there were 36 female participants
and 42 male participants. See Table 1 for a detailed description of the
participants for each evaluation group of MediTool.
3.2. Instruments
Three types of instruments were designed to conduct the present
study: (i) four versions of MediTool for the study of clinical cases; (ii) a
traditional paper-based, multiple-choice exam to evaluate participants’
learning performance after studying clinical cases using one version of
MediTool; and (iii) a formal questionnaire to evaluate perceived useful-
ness, perceived ease of use, and the impact of gamification elements on
learning performance.
3.2.1. A medical educational tool for the study of clinical cases: MediTool
Four versions of a medical educational tool for the study of clinical
cases were designed and implemented by the authors (see Fig. 1): (i) a
2D version; (ii) a gamified 2D version; (iii) a gamified 3D version; and
(iv) a gamified immersive-virtual-reality version. All the versions
comply with the same functional requirements (see Table 2) and pre-
sented the same five clinical cases to participants organized into three
difficulty levels: three clinical cases for beginners, one case for inter-
mediates, and one for advanced students.
The five clinical cases are based on the Mexican Clinical Practice
Guides [23], which make use of the best national and international
clinical practices adapted for the health profile of Mexicans. The in-
formation presented for each clinical case was organized into three
parts: (i) patient's antecedents; (ii) patient's vital signs (e.g., body
temperature and blood pressure) and appearance; and (iii) patient's
personal information (e.g., gender and age). There was a series of
multiple-choice questions associated with each clinical case, which
were the same for the four versions of MediTool. It is important to note
that some of the questions associated with the intermediate and ad-
vanced clinical cases included lab examinations to present additional
information about patient's findings. Once a participant answered a
question, MediTool provided feedback according to the selected answer
and a link to the clinical practice guide so that the participant could
access additional information.
The graphical user interfaces, interactive elements, and navigation
options were different for each version of MediTool (see Table 3). The
O. López Chávez, et al. International Journal of Medical Informatics 141 (2020) 104226

Table 1
Description of the participants for each evaluation group of MediTool.
2D version Gamified 2D version Gamified 3D version Immersive-virtual-reality version

Id Gender Age Year Id Gender Age Year Id Gender Age Year Id Gender Age Year

1 Male 22 4 18 Female 22 4 38 Female 22 5 58 Female 22 4

2 Male 22 4 19 Male 22 5 39 Female 24 5 59 Female 22 4
3 Female 22 4 20 Male 24 5 40 Male 23 4 60 Male 23 4
4 Female 21 4 21 Female 22 5 41 Male 22 5 61 Male 21 4
5 Male 21 4 22 Female 23 5 42 Female 24 5 62 Male 22 4
6 Female 22 4 23 Male 21 4 43 Male 21 4 63 Male 21 4
7 Male 25 4 24 Male 24 5 44 Male 23 5 64 Male 22 4
8 Female 22 4 25 Male 21 4 45 Female 22 5 65 Male 21 3
9 Male 24 5 26 Female 26 4 46 Male 21 4 66 Male 21 3
10 Female 22 5 27 Female 22 4 47 Male 22 5 67 Female 21 3
11 Male 23 5 28 Male 24 5 48 Female 22 5 68 Male 21 3
12 Male 22 5 29 Male 23 5 49 Male 23 5 69 Male 20 3
13 Female 23 5 30 Female 21 4 50 Female 24 5 70 Male 22 3
14 Male 22 5 31 Male 22 5 51 Female 21 4 71 Male 18 1
15 Male 22 5 32 Male 22 4 52 Male 23 4 72 Female 19 1
16 Female 22 5 33 Male 26 5 53 Male 21 4 73 Female 19 1
17 Male 23 5 34 Female 23 5 54 Female 21 4 74 Female 18 1
35 Female 23 4 55 Female 22 5 75 Female 18 1
36 Female 21 4 56 Female 23 5 76 Female 18 1
37 Male 22 5 57 Female 22 5 77 Female 18 1
78 Male 18 1

Median 22 5 Median 22 5 Median 22 5 Median 21 3

3D and immersive-virtual-reality versions offered virtual scenarios that
simulate a consulting room and interactive 3D objects that were used by
participants to (de)activate the description of clinical cases and the
associated examinations. These versions also included 3D avatars
playing the role of a patient and allowed participants to navigate the
virtual scenario. In particular, in the immersive-virtual-reality version,
participants navigated the virtual scenario by moving in a physical real-
world space using a virtual reality headset and hand controllers to in-
teract with virtual objects. To promote gamification, the gamified
versions of MediTool included avatars, rewards, comments, rankings,
badges, and challenges (e.g., resolving a given number of clinical
cases).
3.2.2. Exam to evaluate learning performance
A traditional paper-based exam was designed to evaluate the par-
ticipant's learning performance after studying the clinical cases using a
particular version of MediTool. The exam consisted of the same five
clinical cases incorporated into MediTool and their associated multiple-
choice questions.
3.2.3. Questionnaire to evaluate usefulness, ease of use, and gamification
The four versions of MediTool were evaluated in terms of perceived
usefulness and perceived ease of use by participants using a 12-item
Likert-type questionnaire based on the Technology Acceptance Model
(TAM) [25], see Table 4, questions Q1−12. The TAM is a universal,
standardized questionnaire [26] that can be adapted to the domain of
the application and is widely accepted [24,27] as a robust tool for
measuring usefulness and ease of use. This questionnaire was com-
plemented with six questions to evaluate the perception of participants
on whether incorporating gamification elements helped improve their
overall learning performance, see Table 4, questions Q13−18. These
questions were asked only to participants that utilized the gamified
versions of MediTool.
3.3. Procedure
Each version of MediTool was evaluated by an independent group of
participants. Particularly, the 2D version, the gamified 2D version, the
gamified 3D version, and the immersive-virtual-reality version were
evaluated by 17, 20, 20, and 21 participants, respectively. The group
size varied because each group corresponded to either one or two
groups of a medical school, which usually have different sizes.
The evaluation sessions consisted of four phases. In the first phase,
the objective of the experiment was explained and participants were
asked to read and sign a consent form. Afterward, the functionality of
the corresponding version of MediTool was explained by directly ex-
ploring how it works and all participants’ questions were answered. The
instruments used to evaluate MediTool were provided to participants
and explained. In particular, all the questions of the evaluation in-
struments (see Table 4) and related concepts (such as learning perfor-
mance) were explained. This phase lasted approximately 20 min. In the
second phase, participants were requested to register on MediTool and
were allowed to freely utilize the corresponding version of MediTool for
20 min. Assistance was provided to participants as requested. In the
third phase, the participants evaluated the corresponding version of
MediTool in regard to usefulness, ease of use, and gamification by using
the questionnaire reported in Table 4. In the fourth phase, the partici-
pants were handed over a (traditional) multiple-choice exam to de-
termine whether and to what extent they retained information about
the clinical cases. There was no time limit for this last phase. On
average, the participants took 15 min to complete the exam.
4. Results
For this analysis, statistical tests were performed using R and a p-
value less than 0.05 was considered significant. Raw results are re-
ported in Tables 5 and 6, and shown in Figs. 2–4 .
4.1. Results on the participants’ scores on the exams
Shapiro-Wilk's tests confirmed the normality of the scores obtained
by the participants on the traditional exam (Table 5) after using the 2D
version (W = 0.91564, p-value = 0.1246), the gamified 2D version
(W = 0.93928, p-value = 0.2324), the gamified 3D version
(W = 0.96008, p-value = 0.5454), and the immersive-virtual-reality
version (W = 0.9243, p-value = 0.1058). In addition, two-sample F-
tests for equal variances confirmed the equality of the variances of the
scores obtained by the participants on the traditional exam after using

Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
O. López Chávez, et al. International Journal of Medical Informatics 141 (2020) 104226

Fig. 1. Different versions of MediTool.

Table 2
A representative selection of functional requirements of the four versions of MediTool.
Functional requirements

Req 1 MediTool should show a user registration screen (requires user name, password, participant's name, and email).
Req 2 MediTool should show a login screen to enter the system (requires user name and password).
Req 3 MediTool should show a user tutorial.
Req 4 MediTool should include five clinical cases organized into three difficulty levels: beginner, intermediate, advanced.
Req 5 Level 1 (beginner) should include three clinical cases at random.
Req 6 Level 2 (intermediate) should include one clinical case.
Req 7 Level 3 (advanced) should include one clinical case.
Req 8 The description of clinical cases should be composed of patient's antecedents, vital signs, and personal information.
Req 9 MediTool should include quizzes with multiple-choice questions corresponding to each clinical case.
Req 10 MediTool should present feedback to the user once a response is provided.
Req 11 MediTool should present the user's score.

Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
O. López Chávez, et al. International Journal of Medical Informatics 141 (2020) 104226

Table 4

3D interactive objects, 3D virtual agents, and avatar

Questionnaire based on the TAM [24] used to evaluate the four versions of

User input and output based on virtual reality hand

MediTool.
Perceived usefulness
Gamified immersive-virtual-reality version

Badges, challenges, rankings, score table

Q1 Using MediTool in my studies would enable me to accomplish learning
goals more quickly.
Q2 Using MediTool in my studies would improve my learning performance.
Q3 Using MediTool in my studies could increase my learning efficiency.
Q4 Using MediTool in my studies would enhance my learning effectiveness.
Q5 Using MediTool would make it easier to study.
Q6 I would find MediTool useful in my studies.
controller and headset
3D consulting room

Perceived ease of use

Q7 Learning to operate MediTool was easy for me.
Q8 I found it easy to get MediTool to do what I want it to do.
Q9 My interaction with MediTool was clear and understandable.
Q10 I found MediTool to be flexible to interact with.
Q11 It would be easy for me to become skillful at using MediTool.
Q12 I found MediTool easy to use.
User input and output based on keyboard, mouse,

Questions about gamification elements

3D interactive objects and 3D virtual agents

Q13 Including badges helped improve my learning performance.

Badges, challenges, rankings, score table

Q14 Including ranking helped improve my learning performance.

Q15 Including challenges helped improve my learning performance.
Q16 Including comments helped improve my learning performance.
Q17 Including 2D/3D avatars helped improve my learning performance.
Q18 Including rewards helped improve my learning performance.

Values and interpretation for possible answers are as follows: 7: extremely

Gamified 3D version

3D consulting room

and screen devices

likely, 6: quite likely, 5: slightly likely, 4: neither, 3: slightly unlikely, 2: quite

unlikely, and 1: extremely unlikely

Table 5
Descriptive statistics of the participants’ scores on a traditional multiple-choice
exam.
User input and output based on keyboard, mouse,

Median Std dev Min Max 25th 50th 75th

percentile percentile percentile
Badges, challenges, rankings, score table

Scores obtained by the participants using the 2D version of MediTool

395 43.13 330 470 350 395 430

Scores obtained by the participants using the gamified 2D version of MediTool

440 45.16 320 500 413.75 440 463.75
2D icons and 2D images

Scores obtained by the participants using the gamified 3D version of MediTool

Gamified 2D version

and screen devices

415 40.03 340 500 390 415 442.50

A representative selection of particular characteristics of each version of MediTool.

2D interfaces

Scores obtained by the participants using the immersive-virtual-reality version of MediTool

400 51.79 265 480 390 400 425

Possible scores for the traditional exam ranged from 0 to 500.

User input and output based on keyboard, mouse,

the 2D version and (i) the gamified 2D version (p-value = 0.876), (ii)
the gamified 3D version (p-value = 0.7339), and (iii) the immersive-
virtual-reality version (p-value = 0.4767). Based on these tests’ results,
t tests were used to assess the statistical significance of the results on
learning performance.
From the data reported in Table 5, a two-tailed two-sample t test
(with equal variances) showed that the scores on the traditional exams
2D icons and 2D images

for the 2D version and the scores for the gamified 2D version are sig-
nificantly different (t = −3.0518, p-value = 0.004322). In addition, a
and screen devices

two-tailed two-sample t test (with equal variances) showed that the

2D interfaces

scores on the traditional exams for the 2D version and the scores for the
2D version

gamified 3D version are significantly different (t = −2.153, p-

value = 0.03829). However, a two-tailed two-sample t test also showed
N/A

that the scores for the 2D version and the immersive-virtual-reality

version are not significantly different (t = −0.84511, p-
Gamification elements
Interactive elements

value = 0.4036). Also, as shown in Table 5, the highest average scores

on the traditional exams were obtained for the gamified 2D version and
the gamified 3D version, whereas the lowest average scores were ob-
Navigation

tained for the 2D version and the immersive-virtual-reality version.

Table 3

GUI

Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
O. López Chávez, et al. International Journal of Medical Informatics 141 (2020) 104226

Table 6
Descriptive statistics of the evaluation results of MediTool.
Construct Median Mode Min Max 25th percentile 50th percentile 75th percentile

Evaluation of the 2D version

Perceived usefulness 6 5 2 7 5 6 6
Perceived ease of use 7 7 3 7 6 7 7

Evaluation of the gamified 2D version

Perceived usefulness 6 6 1 7 6 6 7
Perceived ease of use 7 7 1 7 6 7 7
Gamification 6 7 2 7 5 6 7

Evaluation of the gamified 3D version

Perceived usefulness 6 6 5 7 6 6 7
Perceived ease of use 6 6 3 7 6 6 7
Gamification 6 7 1 7 5 6 7

Evaluation of the immersive-virtual-reality version

Perceived usefulness 6 6 5 7 6 6 7
Perceived ease of use 7 7 3 7 6 7 7
Gamification 7 7 3 7 6 7 7

Values and interpretation for possible answers are as follows: 7: extremely likely, 6: quite likely, 5: slightly likely, 4: neither, 3: slightly unlikely, 2: quite unlikely, and
1: extremely unlikely.

4.2. Results on usefulness and ease of use gamified 3D version was 6 (i.e., quite likely that gamification helped
improved learning), whereas the median for the immersive-virtual-
For all the results on the evaluation constructs (namely, usefulness, reality version was 7 (i.e., extremely likely that gamification helped
ease of use, and gamification) reported in Table 6, one-sample Wilcoxon improved learning).
signed-rank tests showed that the constructs’ medians µ1/2 i
are sig-
nificantly greater than specified medians µ1/2 1 (p-values <0.00001),
i
5. Discussion
which confirm the significance of the medians. The one-sample Wil-
coxon signed-rank test was used because Likert scales are ordinal and
5.1. Discussion about the participants’ scores on the exams
evaluation results do not follow a normal distribution according to re-
sults from Shapiro–Wilk's tests (p-values <0.00000001).
The highest average scores on the traditional exams were obtained
The median of the perceived ease of use construct for the 2D version,
by the participants using either the gamified 2D version or the gamified
the gamified 2D version, and the immersive-virtual-reality version was
3D version (Fig. 2), whereas the lowest average scores were obtained by
7 (i.e., extremely likely to be usable), whereas, for the same construct,
the participants using the immersive-virtual-reality version and the 2D
the median for the gamified 3D version was 6 (i.e., quite likely to be
version. In fact, the lowest average score was obtained by the partici-
usable). With respect to the perceived usefulness construct, the median
pants using the 2D version. This may suggest that gamification elements
for all the versions was 6 (i.e., quite likely to be useful). As for the
play a positive role in student's learning performance as discussed in
gamification construct, the median for the gamified 2D version and the
[28] and as indicated by the participants of this study (see Fig. 3).

Fig. 2. Participants’ scores on a traditional multiple-choice exam after using MediTool.

Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
O. López Chávez, et al.
Fig. 3. Evaluation results of MediTool.
In general, the participants (that used all the gamified versions)
perceived that it was (at least) quite likely that the gamification helped
improve their learning performance. However, the average score on the
traditional exams of the participants using the immersive-virtual-reality
version (that also included gamification elements) was not as high as
the average score of the participants using the other gamified versions.
This contrasts even more with the fact that the immersive-virtual-rea-
lity version was perceived as quite likely to be useful (the same as the
other versions) and extremely likely to be usable (among the most
usable versions). Moreover, in terms of gamification, the immersive-
virtual-reality version received the highest rating, namely the partici-
pants indicated that it was extremely likely that the gamification of the
immersive-virtual-reality version helped improve their learning per-
formance. A potential explanation for this result is that the participants
that evaluated the immersive virtual reality version had fewer years of
medical school preparation (with a median of 3) compared to the other
evaluation groups (with a median of 5), see Table 1. Another factor that
may have affected the average exam score of the participants that used
the immersive-virtual-reality version is that, while using MediTool,
some of them moved hesitantly because they were not seeing the
physical real-world environment and may have not been aware of its
boundaries. It is hypothesized that this uncertainty may have distracted
students from learning. Another potential explanation for this result
(based on the authors’ observations) is that the participants that used
the immersive-virtual-reality version spent some time exploring the
environment instead of spending time studying the clinical cases. This
may have been because for the majority of these participants, it was the
first time that they used an immersive-virtual-reality application.
Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
International Journal of Medical Informatics 141 (2020) 104226
It is acknowledged that, in this study, the assessment of learning
performance was limited to administering an exam on the clinical cases
taught in MediTool and grading the participants’ responses. This as-
sessment method is mainly focused on evaluating the participants’
short-term retention (or working memory), which plays a role in
learning [22] and according to [21] even underpin learning. None-
theless, to completely determine participants’ learning performance,
additional metrics should be involved and more elaborate assessment
methods are required.
5.2. Discussion about usefulness and ease of use
In general, all the versions of MediTool were perceived as extremely
usable, except the gamified 3D version, which was perceived as quite
usable (Fig. 3). In this regard, according to a couple of comments from
two participants, the gamified 3D version may have not been perceived
as extremely likely to be usable because it was difficult to interact with
the environment using avatars. In spite of this difference in usability, all
the versions of MediTool were perceived as quite useful (Fig. 3).
Nonetheless, it should be noted that for the gamified 3D version and the
immersive-virtual-reality version, the most rigorous participants con-
sidered them as slightly likely to be useful. However, in the case of the
2D version and the gamified 2D version, there were two participants
(one in each evaluation) that indicated that the corresponding version
of MediTool was quite unlikely to be useful and extremely unlikely to be
useful, respectively. This may suggest that including 3D components
either using non-immersive virtual reality or immersive virtual reality
may have helped improve the perception of potential usefulness of
O. López Chávez, et al. International Journal of Medical Informatics 141 (2020) 104226

18
16
14
12
10
8
6
4
2
0
(1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7)
Question 1 Question 2 Question 3 Question 4 Question 5 Question 6
2D version Gamified 2D version Gamified 3D version Immersive-virtual-reality version

Responses regarding ease of use

18
16
14
12
10
8
6
4
2
0
(1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7)
Question 7 Question 8 Question 9 Question 10 Question 11 Question 12
2D version Gamified 2D version Gamified 3D version Immersive-virtual-reality version

Responses regarding gamification

18
16
14
12
10
8
6
4
2
0
(1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7) (1) (2) (3) (4) (5) (6) (7)
Question 13 Question 14 Question 15 Question 16 Question 17 Question 18
2D version Gamified 2D version Gamified 3D version Immersive-virtual-reality version

Values and interpretation for possible answers are as follows: (7) extremely likely, (6) quite likely,
(5) slightly likely, (4) neither, (3) slightly unlikely, (2) quite unlikely, and (1) extremely unlikely.
Fig. 4. Detailed evaluation results of Meditool.

MediTool regardless of whether it may be difficult to interact with 3D
environments using avatars (as indicated by a couple of participants in
relation to the gamified 3D version).
6. Conclusions and future work
The novelty of this paper resides in the comparative nature of the
analysis enabled by implementing and evaluating four versions of an
educational tool for the study of clinical cases using different ap-
proaches (combining gamification with 2D and 3D virtual environ-
ments and fully immersive virtual reality).
The results of this study show that using different design approaches
(either 2D, 3D, or immersive-virtual-reality with or without gamifica-
tion) to implement a medical educational tool may affect students’
retention of information on clinical cases. In addition, regardless of
whether the medical educational tool is perceived as highly useful and
highly usable and the tool complies with the expected functional re-
quirements, there were significant differences on the average scores on
the exams obtained by each group of participants. Finally, it should be
noted that, in terms of gamification and based on the perception of the
participants, the immersive-virtual-reality version received the highest
rating, i.e., it was extremely likely that its gamification helped improve
their learning performance.
An immediate future research direction is to explore how learning
efficiency and effectiveness evolve with respect to usage time when
medical students make use of either 2D, 3D or immersive-virtual-reality
medical educational tools.

Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
O. López Chávez, et al.
Summary table
What was already known on the topic:
• The workings of medical educational tools are implemented using a
myriad of approaches ranging from presenting static content to
immersing students in gamified virtual-reality environments.
• Current medical educational tools have been positively evaluated as
tools that engage students and improve learning, however, there is a
lack of studies comparing and discussing their design approaches
and technological elements in relation to learning performance.
What this study added to our knowledge:
• Regardless of whether different versions of a medical educational
tool (complying with the same functional requirements) are per-
ceived as equally useful and usable, the design approach (either 2D,
3D, or immersive-virtual-reality with or without gamification) af-
fects students’ retention of information on clinical cases.
•
Based on the perception of the participants, the gamification of an
immersive-virtual-reality version of a medical education tool helped
the most to improve learning performance in comparison with ga-
mified 2D and 3D versions.
Authors’ contributions
OL developed the four versions of the medical educational tool. OL,
LR and JG designed the experiments. OL performed the experiments.
OL, LR, and JG conducted the data analysis and interpretation. LR and
JG conceived and designed the research project. All authors have
contributed to the manuscript and have read and approved the final
version.
Conflicts of interest
None declared.
Acknowledgments
This work was supported by PROFEXCE 2020 and PROFAPI 2020. J.
O. Gutierrez-Garcia gratefully acknowledges the financial support from
the Asociación Mexicana de Cultura, A.C.
References
[1] A. Herrmann-Werner, H. Weber, T. Loda, K. Keifenheim, R. Erschens, S. Mölbert,
C. Nikendei, S. Zipfel, K. Masters, “but dr google said..“-training medical students
how to communicate with e-patients, Med. Teacher 41 (12) (2019) 1434–1440.
[2] A. Rajaram, Z. Hickey, N. Patel, J. Newbigging, B. Wolfrom, Training medical
students and residents in the use of electronic health records: a systematic review of
the literature, J. Am. Med. Inform. Assoc. 27 (1) (2020) 175–180.
[3] R. Tsopra, M. Courtine, K. Sedki, D. Eap, M. Cabal, S. Cohen, O. Bouchaud,
F. Mechaï, J.-B. Lamy, Antibiogame®: a serious game for teaching medical students
about antibiotic use, Int. J. Med. Inform. 136 (2020) 104074.
[4] S. Ammanuel, I. Brown, J. Uribe, B. Rehani, Creating 3d models from radiologic
images for virtual reality medical education modules, J. Med. Syst. 43 (6) (2019)
166.
[5] M.-S. Bracq, E. Michinov, P. Jannin, Virtual reality simulation in nontechnical skills
training for healthcare professionals: a systematic review, Simul. Healthc. 14 (3)
Downloaded for Anonymous User (n/a) at University of Sharjah from ClinicalKey.com by Elsevier on September 22,
2023. For personal use only. No other uses without permission. Copyright ©2023. Elsevier Inc. All rights reserved.
International Journal of Medical Informatics 141 (2020) 104226
(2019) 188–194.
[6] D. Qi, A. Ryason, N. Milef, S. Alfred, M.R. Abu-Nuwar, M. Kappus, S. De, D.B. Jones,
Virtual reality operating room with ai guidance: design and validation of a fire
scenario, Surg. Endosc. (2020) 1–8.
[7] J. Herron, Augmented reality in medical education and training, J. Electron.
Resour. Med. Libr. 13 (2) (2016) 51–55.
[8] F. Lateef, Simulation-based learning: just like the real thing, J. Emerg. Trauma
Shock 3 (4) (2010) 348–352.
[9] A. Katz, R. Tepper, A. Shtub, Simulation training: evaluating the instructor's con-
tribution to a wizard of oz simulator in obstetrics and gynecology ultrasound
training, JMIR Med. Educ. 3 (1) (2017) e8.
[10] R. Lohre, A.J. Bois, G.S. Athwal, D.P. Goel, Improved complex skill acquisition by
immersive virtual reality training: a randomized controlled trial, J. Bone Joint Surg.
102 (6) (2020) e26.
[11] A. Dehghani, J. Kojuri, M.R. Dehghani, A. Keshavarzi, S. Najafipour, Experiences of
students and faculty members about using virtual social networks in education: a
qualitative content analysis, J. Adv. Med. Educ. Prof. 7 (2) (2019) 86–94.
[12] D.P. de Sena, D.D. Fabrício, V.D. da Silva, L.C. Bodanese, A.R. Franco, Comparative
evaluation of video-based on-line course versus serious game for training medical
students in cardiopulmonary resuscitation: a randomised trial, PLOS ONE 14 (4)
(2019) e0214722.
[13] C.R. Nevin, A.O. Westfall, J.M. Rodriguez, D.M. Dempsey, A. Cherrington, B. Roy,
M. Patel, J.H. Willig, Gamification as a tool for enhancing graduate medical edu-
cation, Postgrad. Med. J. 90 (1070) (2014) 685–693.
[14] E. Parkhomenko, M. O’Leary, S. Safiullah, S. Walia, M. Owyong, C. Lin, R. James,
Z. Okhunov, R. Patel, K. Kaler, J. Landman, R. Clayman, Pilot assessment of im-
mersive virtual reality renal models as an educational and preoperative planning
tool for percutaneous nephrolithotomy, J. Eendourol. 33 (4) (2019) 283–288.
[15] K. Stepan, J. Zeiger, S. Hanchuk, A. Del Signore, R. Shrivastava, S. Govindaraj,
A. Iloreta, Immersive virtual reality as a teaching tool for neuroanatomy,
International Forum of Allergy & Rhinology vol. 7, Wiley Online Library, 2017, pp.
1006–1013.
[16] D. Takagi, M. Hayashi, T. Iida, Y. Tanaka, S. Sugiyama, H. Nishizaki, Y. Morimoto,
Effects of dental students’ training using immersive virtual reality technology for
home dental practice, Educ. Gerontol. 45 (11) (2019) 670–680.
[17] H.M. Johnsen, M. Fossum, P. Vivekananda-Schmidt, A. Fruhling, Å. Slettebø,
Teaching clinical reasoning and decision-making skills to nursing students: design,
development, and usability evaluation of a serious game, Int. J. Med. Inform. 94
(2016) 39–48.
[18] J.P. Richards, A.J. Done, S.R. Barber, S. Jain, Y.-J. Son, E.H. Chang, Virtual coach:
the next tool in functional endoscopic sinus surgery education, Int. Forum Allergy
Rhinol. 10 (1) (2020) 97–102, https://doi.org/10.1002/alr.22452.
[19] M. Aebersold, T. Voepel-Lewis, L. Cherara, M. Weber, C. Khouri, R. Levine,
A.R. Tait, Interactive anatomy-augmented virtual simulation training, Clin. Simul.
Nurs. 15 (2018) 34–41.
[20] M.G. Violante, E. Vezzetti, Design and implementation of 3d web-based interactive
medical devices for educational purposes, Int. J. Interact. Des. Manuf. (IJIDeM) 11
(1) (2017) 31–44.
[21] N. Cowan, Working memory underpins cognitive development, learning, and edu-
cation, Educ. Psychol. Rev. 26 (2) (2014) 197–223.
[22] S. Gathercole, T.P. Alloway, Working Memory and Learning: A Practical Guide for
Teachers, Sage, 2008.
[23] Guías de práctica clínica, (2020) (accessed 25.03.20), http://www.imss.gob.mx/
guias_practicaclinica?field_categoria_gs_value=All.
[24] Y. Lee, K.A. Kozar, K.R. Larsen, The technology acceptance model: past, present,
and future, Commun. Assoc. Inf. Syst. 12 (1) (2003) 50.
[25] F.D. Davis, Perceived usefulness, perceived ease of use, and user acceptance of
information technology, MIS Q. 13 (3) (1989) 319–340.
[26] A. Assila, K. Oliveira, H. Ezzedine, Standardized usability questionnaires: features
and quality focus, Electron. J. Comput. Sci. Inf. Technol.: eJCIST 6 (1) (2016)
15–31.
[27] L.G. Wallace, S.D. Sheetz, The adoption of software measures: a technology ac-
ceptance model (tam) perspective, Inf. Manag. 51 (2) (2014) 249–259.
[28] M. Sailer, L. Homner, The gamification of learning: a meta-analysis, Educ. Psychol.
Rev. 32 (2020) 77–112.
[29] J. Marty, J. Heraud, T. Carron, L. France, Matching the performed activity on an
educational platform with a recommended pedagogical scenario: a multi-source
approach, J. Interact. Learn. Res. 18 (2) (2007) 267–283.
[30] L. De-Marcos, E. Garcia-Lopez, A. Garcia-Cabot, On the effectiveness of game-like
and social approaches in learning: comparing educational gaming, gamification &
social networking, Comput. Educ. 95 (2016) 99–113.
[31] N. Zhao, M. Wu, J. Chen, Android-based mobile educational platform for speech
signal processing, Int. J. Electr. Eng. 54 (1) (2017) 3–16.