Acta Ophthalmologica 2017
modules A to D (Fig. 1A–C). Based on
Letter to the Editor
Virtual reality-based
proficiency test in direct
ophthalmoscopy
Nanna Jo Borgersen,1,2,3
Ann Sofia Skou Thomsen,1,3,4
Lars Konge,1,3 Torben Lykke Sørensen2,3
and Yousif Subhi2,3
1
Copenhagen Academy for Medical
Education and Simulation, Copenhagen,
Denmark; 2Department of Ophthalmology,
Zealand University Hospital, Roskilde,
Denmark; 3Faculty of Health and Medical
Sciences, University of Copenhagen,
Copenhagen, Denmark; 4Department of
Ophthalmology, Copenhagen University
Hospital Rigshospital-Glostrup,
Copenhagen, Denmark
doi: 10.1111/aos.13546
Editor,
D irect ophthalmoscopy is poorly
mastered among young physi-
cians and avoided due to lack of
proficiency and confidence in the skill
(Gupta & Lam 2006). It is tradition-
ally taught by an instructor guiding
students through the procedure with-
out actually seeing what the student
sees through the ophthalmoscope.
This limits the ability to deliver con-
textual feedback (Borgersen et al.
2016). Virtual reality simulators have
the potential to deliver important
contextual feedback not otherwise
possible with traditional teaching.
However, ensuring an evidence-based
approach is important when imple-
menting costly simulators as lack of
validity evidence may lead to expen-
sive irrelevant training at best or
misdiagnosis at worst.
We conducted a prospective validity
study of the automated assessments
provided by the EyeSi Direct Ophthal-
moscope Simulator (v1.4, VRmagic,
Mannheim, Germany) on 13 novices
(medical students with no previous
experience in direct ophthalmoscopy)
and eight experienced (consultants in
ophthalmology). The simulator provided
scores for different aspects of the fundo-
scopic examination through prespecified
failed and all experienced passed
expert recommendations, two modifi-
cations were made to improve the
alignment of the assessment content
with the construct: we removed ques-
tions covering general knowledge in
ophthalmology in module D because
they were unrelated to the skill of
direct ophthalmoscopy; we turned off
scores adjustment for percentage of
total retinal area examined because
experienced individuals examine more
focused. All participants signed informed
consent. Principles of the Declaration
of Helsinki were followed. Ethics
committee deemed that approval was
unnecessary (waiver no. 16025789).
Validity evidence was evaluated using
Messick’s framework (Thomsen et al.
2015):
Content
Experienced participants considered
the simulator realistic and found that
the training programme met the train-
ing needs on how to perform direct
ophthalmoscopy (Fig. 1D).
Response process
A pilot study was conducted to
address potential sources of bias in
data collection. The simulator auto-
matically collected all scores. The
same instructor interacted with all
participants using predefined informa-
tion script. To account for familiar-
ization effect, participants underwent
20 minutes warmup before data col-
lection.
Internal structure
Internal consistency between the simu-
lator modules was high (Cronbach’s
a=0.79).
Relations to other variables
Each of the four module scores, the
total score and the total training time
all significantly differed between no-
vices and experienced (Fig. 1E).
Consequences
Based on the sum score of all modules,
a pass/fail score of 2615 was calculated
using the contrasting groups method
(Fig. 1F). Consequences of applying
this standard were that all novices
(Fig. 1G).
To our knowledge, this is the first
study to explore and present validity
evidence of a direct ophthalmoscope
virtual reality simulator. Overall, we
found strong evidence of validity:
content was aligned with direct oph-
thalmoscopy skills as evaluated by
experienced consultants. Response
process was ensured through a pilot
study and standardizing all given
instructions. Internal structure showed
high level of reliability with intermod-
ule reliability approaching a level con-
sidered eligible for certification (0.8)
(Downing 2004). Relations to other
variables showed that the experienced
significantly outperformed the novices.
Consequences of applying a pass/fail
standard showed excellent discrimina-
tory ability with no false positives/
negatives.
The pass/fail standard at 2615 points
represents the point where a trainee has
received sufficient training, whether
trained on the simulator or otherwise.
Hence, our results not only establish a
proficiency test for direct ophthal-
moscopy training on the simulator –
the test can also be used for other
purposes (e.g. exams for medical stu-
dents).
Future studies should explore details
of weighting and composition of mod-
ules. Whether shorter programmes
provide better balance between time
consumption and reliability remain
unexplored for future studies. Cheaper
eye models exist and should be consid-
ered when appropriate and relevant
validity evidence exists (Wu et al.
2014).
References
Borgersen NJ, Henriksen MJ, Konge L,
Sørensen TL, Thomsen AS & Subhi Y
(2016): Direct ophthalmoscopy on You-
Tube: analysis of instructional YouTube
videos’ content and approach to visualiza-
tion. Clin Ophthalmol 10: 1535–1541.
Downing SM (2004): Reliability: on the repro-
ducibility of assessment data. Med Educ 38:
1006–1012.
Gupta RR & Lam WC (2006): Medical
students’ self-confidence in performing
direct ophthalmoscopy in clinical training.
Can J Ophthalmol 41: 169–174.
Thomsen AS, Subhi Y, Kiilgaard JF, la Cour
M & Konge L (2015): Update on simula-
tion-based surgical training and assessment
1
Acta Ophthalmologica 2017

Fig. 1. (A) The EyeSi Direct Ophthalmoscope Simulator is a virtual reality-based interactive simulator. (B) Module A and B consist of cases where
the trainee practice handling of the ophthalmoscope, examining the fundus and identifying objects. (C) Module C and D consist of cases where the
trainee practice the interpretation and documentation of healthy and pathological retinas. (D) Content questionnaire including only the experienced,
who found the content to align with the construct of interest. (E) Scores and training time in relation to experience showed a positive discriminatory
ability for all modules. (F) The contrasting groups’ methods were applied to determine a pass/fail level of ophthalmoscopy skills: the proficiency level
was defined by the intersection of the mean sum scores for the novices (blue) and experienced (green). The curves show normal distribution using
mean and standard deviations for both groups. (G) An investigation of the consequences of the determined pass/fail level showed that all novices
failed (no false positives) and that all experts passed (no false negatives).

in ophthalmology: a systematic review. DK-4000 Roskilde

Ophthalmology 122: 1111–1130. Correspondence: Denmark
Wu GT, Kang JM, Bidwell AE, Gray JP & Yousif Subhi, MD Tel: +45 4732 3900
Mirza RG (2014): The use and evaluation Department of Ophthalmology Fax: +45 4636 2645
of an inexpensive eye model in direct Zealand University Hospital Email: ysubhi@gmail.com
ophthalmoscopy training. JAO 7: e21– Vestermarksvej 23
e25.