Assessment of Durability of Online and

Multisensory Learning Using an

Ophthalmology Model
Linda Mottow Lippa, MD,1 Craig L. Anderson, PhD2

Purpose: To explore the impact of online learning and multisensory small-group teaching on acquisition and
retention of specialty knowledge and diagnostic skills during a third-year family medicine rotation.
Design: Exploratory, observational, longitudinal, and multiple-skill measures.
Participants: Two medical school classes (n ¼ 199) at a public medical school in California.
Methods: Students engaged in online self-study, small-group interactive diagnostic sessions, picture
identification of critical pathologic features, and funduscopic simulator examinations. The authors compared
performance on testing immediately after online learning with testing at end-rotation, as well as picture identifi-
cation versus simulator diagnostic ability in students with (n ¼ 94) and without (n ¼ 105) practice tracing contours
on whiteboard projections of those same slides depicting fundus pathologic features of common systemic
diseases.
Main Outcomes Measures: Picture identification, accuracy of funduscopic descriptions, online module
post-tests, and end-rotation tests.
Results: Proprioceptive reinforcement of fundus pattern recognition significantly reduced the need for
remediation for misdiagnosing optic disc edema during end-rotation funduscopic simulator testing, but it had no
effect on fundus pattern recognition or diagnostic ability overall. Near-perfect immediate online post-test scores
contrasted sharply with poor end-rotation scores on an in-house test (average, 59.4%). Rotation timing was not a
factor because the patterns remained consistent throughout the academic school year.
Conclusions: Neither multisensory teaching nor online self-study significantly improved retention of
ophthalmic knowledge and diagnostic skills by the end of a month-long third-year rotation. Timing such training
closer to internship when application is imminent may enhance students’ appreciation for its value and perhaps
may improve retention. Pulsed quizzes over time also may be necessary to motivate students to retain the
knowledge gained. Ophthalmology 2015;-:1e7 ª 2015 by the American Academy of Ophthalmology.

Medical education is evolving to adapt to the electronic
media revolution and to the resultant unique learning styles
of millennial learners. The Liaison Committee on Medical
Education mandates fewer lecture hours and more self-
directed online learning, applying acquired knowledge in
interactive small group sessions. The inherent scheduling
flexibility of online self-study and the personal nature of
small-group interactive learning understandably have been
popular with students. Medical schools have furnished
e-tablets loaded with textbooks, schedules, and apps to
make just-in-time learning more accessible than ever. The
explosion of knowledge and instant availability of relevant
online information and the trend toward 3-year programs
have all been used as justifications to eliminate specialty
rotations from the required medical school curriculum. The
untoward effects of curricular marginalization of specialty
skills on patient care must not be underestimated. In one
study of history-taking and physical examination skills
among a New York City teaching hospital’s house-staff, no
interns or residents examined hypertensive patients’ fundi,
and only 4% tested extraocular muscle excursions in any
patients. Furthermore, their notes documented evaluations
that had not been performed in 60% of charts regarding eye
movements, 80% for pupils, and 69% for accommodation.1
Elements of the specialty physical examination, relevant
knowledge, and clinical decision-making skills are crucial to
best practices of general medicine and should be taught by those
most proficient in those skills. With limited curricular time,
specialist educators have less opportunity than ever to have an
enduring impact on knowledge and skills. Although the use of
online learning can serve as so-called faculty extenders, the
accompanying small-group sessions of this flipped classroom
model severely strains limited time resources of clinical faculty.
Questions arising from this paradigm shift in teaching
include the effectiveness and longevity of knowledge self-
taught online and the benefits of small-group interactive
sessions. Another is whether recognition of visualized pat-
terns via an alternate proprioceptive pathway would rein-
force students’ pattern recognition skills of physical
findings. There is precedent in the education literature for
multisensory, alternate pathway input addressing the needs
of visual, auditory, and tactile learners.2

 2015 by the American Academy of Ophthalmology http://dx.doi.org/10.1016/j.ophtha.2015.06.027 1

Published by Elsevier Inc. ISSN 0161-6420/15
 Ophthalmology Volume -, Number -, Month 2015
Figure 1. Fundus photography images showing (A) vascular contours marked to illustrated elevation of a swollen disc and (B) the tracing of disc margin and
cup on a slide depicting nonproliferative diabetic retinopathy.
We conducted this exploratory study to address these
questions in one marginalized subspecialty, ophthalmology, by
examining retrospectively the scores of 2 entire classes of
medical students (2010e2012) in the course of the normal
educational process before and after implementing teaching
method innovations. The curriculum of the University of Cal-
ifornia, Irvine, has featured competency-based, longitudinally
integrated ophthalmology education spanning the first 3 years
of medical school since 2002. As part of the third-year family
medicine 4-week ambulatory clerkship, students participate in a
required 3-hour small-group session taught year round by a
single faculty member (L.M.L.). The sessions begin with reit-
eration of material taught in the first 2 curricular years: a review
of the screening eye examination and the disc mantra pattern
description algorithm.3 Diagnostic reasoning is explored using
3 common medical case scenarios for which examination of the
eye and fundus hold the clue to the correct diagnosis. All
students use a direct ophthalmoscope to examine 35-mm
wide-angle fundus Kodachrome slides (Eastman Kodak,
Rochester, NY) inserted into funduscopic simulators (Clinical
Learning Experience in Ophthalmoscopy [CLEO]).4e6 Stu-
dents rotate between examining different slides in 3 CLEO
simulators and supervised examination of each other’s undi-
lated fundi, followed by case discussions.
Students are given an additional half-day of independent
study, examining in CLEO an 18-slide teaching set covering
fundus manifestations of common systemic diseases. Their
analysis is assisted by instructional posters and tables
illustrating the disc mantra algorithm elements for multiple
common conditions. The students are instructed to examine
each of the study set slides first through CLEO’s 8-mm right
pupil, then through the 4-mm left pupil, to record their de-
scriptions and then to check their findings against a table
provided with correct descriptions for each slide for im-
mediate feedback and correction. They then reexamine the
slide to reinforce correct identification of the findings.
Orientation documents outline the ophthalmology
components during the month-long rotation: flexibly
timed, self-directed learning of core ophthalmic material
using an extensive set of online learning modules devel-
oped at the University of Michigan7 and approved by the
American Academy of Ophthalmology. Immediately
after each subsection, students complete the embedded
2
electronic assessment on that subsection; upon
completion of the program, a final score is generated,
which then is incorporated in the calculation of their
ophthalmology grade.
At the rotation’s end, students are tested on their ability
to identify correctly the disc mantra diagnostic elements for
1 of 3 test slides viewed in the simulator’s dilated eye
through a direct ophthalmoscope. The test slide represents
one of the disease processes exemplified in the teaching
slide set but is from a different patient’s fundus. Thus,
application of the disc mantra elements is critical to sub-
stantiate their diagnosis and to provide a differential diag-
nosis. Identification of characteristic mantra elementsdthe
cup, disc color, disc contour, disc margins and vessels, and
retinal patternsdmore accurately reflects a student’s ability
to diagnose a patient than mere recognition of a familiar
slide from short-term memory. Funduscopic descriptions are
graded on a 25-point scale, with a maximum of 4 points for
the retinal findings, and 3 points for each of the other mantra
elements, as well as for the diagnosis and differential di-
agnoses; partial credit is granted. All answers are scored by
a single grader (L.M.L.).
The ophthalmology portion of family medicine counts
toward 5% of the final grade in the family medicine rotation.
Ophthalmology grades are based on a weighted algorithm of
knowledge and skills, based on the Michigan module scores,
CLEO score, and an in-house end-rotation quiz applying
factual information and identifying pictures of pathologic
features covered in class and in the Michigan modules.
Methods
This observational study followed students in their usual experience of
ophthalmology integrated into the family medicine rotation as
described above. Therefore, the University of California, Irvine,
Institutional Review Board ruled that approval was not required for this
study. For the sake of parity, the small number of students (n ¼ 4e5)
who deferred the rotation to their fourth year were excluded so that
increased experience from electives, including perhaps even in
ophthalmology, would not compromise the validity of the comparison.
In the first cohort, group 1, consisting of 1 academic year plus
the first rotation of the next (n ¼ 105), 3 clinical scenariosdcentral
retinal artery occlusion (CRAO) in temporal arteritis, various
 Lippa and Anderson 
Online and Multisensory Learning

Table 1. Summary of Data

Group 1 Group 2
Central
Retinal Artery Diabetes Central Retinal
Overall Occlusion Mellitus Papilledema Overall Artery Occlusion Diabetes Mellitus Papilledema
(n ¼ 105) (n ¼ 37) (n ¼ 38) (n ¼ 30) (n ¼ 94) (n ¼ 32) (n ¼ 30) (n ¼ 32)
Raw CLEO score* 19.8 (5.3) 20.1 (3.5) 22.4 (3.9) 16.2 (6.4) 20.1 (5.0) 19.9 (4.3) 23.1 (1.6) 17.5 (6.1)
CLEO % score 79.2 (21.1) 80.4 (13.9) 89.7 (15.4) 64.9 (25.8) 80.4 (19.8) 79.5 (17.0) 92.4 (6.3) 70.1 (24.2)
Students <40% CLEO score (%) 12.4 5.3 5.4 30.0 8.5 6.2 0 18.8
Students with correct 71.4 86.5 76.3 46.7 72.3 81.2 76.7 59.4
Dx (CLEO) (%)
MI test score (%) 97.7 (2.9) 97.0 (3.0) 97.8 (2.3) 98.0 (3.4) 96.4 (3.4) 97.1 (3.1) 96.0 (3.6) 96.3 (3.3)
(n ¼ 34)
MI fundus ID subscore (%) d d d d 95.5 (6.3) 97.1 (3.1) 94.9 (6.3) 95.6 (5.2)
MI systemic disease subscore (%) d d d d 97.5 (4.4) 98.0 (4.2) 94.3 (16.6) 97.7 (4.4)
ERQ score (%) d d d d 59.4 (10.7) 56.8 (17.4) 60.4 (9.4) 60.9 (9.5)
ERQ fundus ID subscore (%) d d d d 63.6 (13.4) 61.9 (14.9) 66.3 (10.8) 62.8 (13.5)

d ¼ no data; CLEO ¼ Clinical Learning Experience in Ophthalmoscopy; Dx ¼ diagnosis; ID ¼ identification; MI ¼ Michigan module total; ERQ ¼ end-
rotation in-house quiz score.
Data are mean (standard deviation) unless otherwise indicated. Group 1 comprised the cohort taught without multisensory teaching reinforcement, and
group 2 comprised the cohort taught with multisensory teaching reinforcement.
*CLEO scores refer to the mantra elements identified during the end-rotation fundus simulator test for elements of fundus pathologic features in systemic
disease.

manifestations of diabetes mellitus (DM), and papilledema (bilat-
eral disc edema of intracranial origin)dwere discussed, and
students examined relevant funduscopic slides for each scenario in
the simulator. Starting with students in the 2nd through 12th ro-
tations of the second academic class, group 2 (n ¼ 94), kinesthetic
proprioceptive reinforcement was added: The same fundus images
from the simulator were projected on a whiteboard so that students
could identify, trace, and label with a dry erase marker all struc-
tures and pathologic features present on the slide (Fig 1). They
reviewed the projected image with labels erased, then
reexamined the slide viewed through the ophthalmoscope in
CLEO. Because of the additional time required for this
reinforcement exercise, only the DM scenario was accompanied
by supervised funduscopic simulator examination of 3 fundus
slides with classic diabetic presentations. All 3 clinical
scenariosdDM, CRAO, and papilledemadwere discussed in
class with projections of the fundi and tracing of features.
We compared end-rotation CLEO scores of students in group 1,
the first cohort with the original curriculum, with those of group 2,
in the second cohort academic year exposed to proprioceptive
reinforcement. We assessed whether there was any difference in
performance in group 1 versus group 2 with regard to CLEO
scores, the number of scores less than 40% triggering remediation,
and correct diagnoses between the groups, as well as whether there
was any difference in Michigan module test scores.
We also compared the effect of test slide assigned on total CLEO
scores, correct diagnoses and need for remediation, and whether
performance on each slide differed between groups 1 and 2. Within
each group, we compared overall Michigan test scores between the 3
slide subgroups to assess whether there was any difference in overall
ability between student subgroups assigned to each slide. Within
group 2, we additionally compared end-rotation quiz (ERQ) score and
Michigan subscores between the 3 slide subgroups.
The announced in-house ERQ also was initiated with group 2,
based on material in the Michigan modules and classroom didactics.
The ERQ consisted of 14 written questions and 10 picture identi-
fications. We compared Michigan online learning end-module test
scores and subscores against scores on the same material on the
ERQ. We looked specifically for correlations between performance
on the picture recognition and systemic disease manifestation sub-
sections in the ERQ and Michigan tests versus performance
describing fundus elements and pathologic features in CLEO at
rotation’s end. We also examined the effect of learning stage on
performance, comparing early family medicine rotators with rota-
tors later in the third year.
Data were entered in an Excel spreadsheet (Microsoft Corp,
Redmond, WA) and analyzed using the BioStats Calculator iPad
app version 1.3.1 (Sam McCall), Excel, and Stata software version
12.1 (StataCorp, College Station, TX). Contingency tables were
analyzed with the chi-square test for independence for tables with
expected cells sizes of 5 or more and with an exact test for smaller
cells. Continuous variables for the 2 groups were compared using t
tests and analyses of variance. A 1-way analysis of variance was
used to compare the 3 slide subgroups.
Results
Effects of Proprioceptive Reinforcement on
Funduscopic Performance
Raw CLEO scores were not significantly different between group
1, which did not undergo proprioceptive reinforcement, and group
2, which did (P ¼ 0.72; Table 1). Chi-square analysis showed no
significant difference in the percentage of students needing reme-
diation (P ¼ 0.64), correct diagnoses (P ¼ 0.98), or perfect scores
(P ¼ 0.46) between the groups.
Approximately one-third of each class were assigned 1 of 3 test
slides to analyze in CLEO, and the scores did vary among subgroups
by the fundus test slide examined (Fig 2). In group 1, a significantly
higher percentage of the subgroup tested with a papilledema slide
needed remediation than the groups tested with a CRAO or a DM
fundus slide (P ¼ 0.005). The percentage of students correctly
diagnosing papilledema was significantly lower than that for DM
and CRAO (P ¼ 0.001). The mean raw CLEO score for the
papilledema slide was lower than that for CRAO or DM (P < 0.001).

3
 Ophthalmology Volume -, Number -, Month 2015

Figure 2. Bar graph showing a comparison of ability to diagnose a central retinal artery occlusion (CRAO), diabetic retinopathy (DM), and papilledema
(PAP) in groups 1 and 2. CLEO ¼ Clinical Learning Experience in Ophthalmoscopy (mantra elements identified during the end-rotation fundus simulator
test for elements of fundus pathology in systemic disease); Dx ¼ diagnosis; GRP ¼ group (group 1 ¼ cohort taught without multisensory teaching rein-
forcement; group 2 ¼ cohort taught with multisensory teaching reinforcement); % students requiring remediation ¼ percentage of students scoring <40% of
mantra elements on the CLEO examination. Avg CLEO % mantra score; % students with correct CLEO Dx; % students requiring remed.

In group 2, there was no statistically significant difference in the
number of remediations by test slide between those tested on
CRAO versus DM or between papilledema and CRAO, but a
significant difference was found in remediation between DM and
papilledema (P ¼ 0.03). There was no difference in the number
correctly diagnosing the disease process between test slide sub-
groups except for between CRAO and papilledema (P ¼ 0.05).
However, the mean raw CLEO score for DM was significantly
better than the mean for the CRAO test slide and for papilledema
(P < 0.001). On the ERQ, there was no difference in scores be-
tween students in the 3 slide subgroups (P ¼ 0.24). The overall
mean for the ERQ was 59.4% (standard deviation, 10.6%).
Comparing groups 1 and 2, there was no statistically significant
difference on the total CLEO scores by the test slide assigned (P ¼
0.39) or in the percentage of students correctly diagnosing DM
(P ¼ 0.98) or CRAO (P ¼ 0.74). However, the percentage of
students able to diagnose papilledema correctly in CLEO was
significantly lower than for the percentage able to diagnose DM or
CRAO for both groups (P < 0.001). Although 93 students (98.9%)
in group 2 correctly identified papilledema pictured in the ERQ,
only 46.7% of the students in group 1 and 59.4% of students in
group 2 recognized papilledema in the simulator (P ¼ 0.45). The
percentage of students correctly establishing the diagnoses in
CLEO overall did not differ with or without proprioceptive rein-
forcement (P ¼ 0.98). In particular, the percentage of students in
both groups missing enough papilledema mantra elements to
trigger remediation (scoring <40% on the CLEO test) was similar
in both groups (P ¼ 0.30).
Durability of Learning with Online Self-Instruction
The Michigan modules were completed on a flexible schedule
during the clerkship. The average overall scores on the intrinsic
total Michigan end-module test in both cohorts were very tightly
clustered, with very small but statistically significant better per-
formance in group 1 versus group 2 (P < 0.001).
Students performed significantly better on the immediate post-
module Michigan testing than on the ERQ on the same material. The
results are illustrated in Figure 3. Although the group 2 Michigan mean
total percent score was 96.4% (standard deviation, 3.4%), their mean
ERQ score was 59.4% (P < 0.001). Groups 1 and 2 performed
comparably on the Michigan post-test and funduscopic examination
simulator. Comparison of group 2’s performance on the Michigan
immediate post-test and the ERQ, which applied material from class
discussions and exercises, demonstrated a dramatic drop in scores;
the high test scores on the Michigan, fundus identification, and
systemic disease module subscores all contrasted with the markedly
poorer performance on comparable ERQ picture identification tests
and on funduscopic examination in the CLEO simulator.
On the fundus identification Michigan module, the mean of
95.5% was significantly better than on the fundus picture identi-
fication section of the ERQ (63.6%; P < 0.001). Both the mean
Michigan fundus identification and systemic disease scores
(97.5%) were significantly better than the end-rotation CLEO
simulator score (80.4%; P < 0.001). Tested at the same end-
rotation time point as the CLEO score (80.4%), the ERQ fundus
picture identification score of 63.6% was still significantly weaker

4
 Lippa and Anderson 
Online and Multisensory Learning

Figure 3. Comparison of Michigan Scores and End Rotation Quiz Results. Avg ¼ average; AVG CLEO ¼ average percentage score on the end-rotation
fundus simulator test on 1 of 3 common ocular manifestations of systemic disease; CRAO ¼ subgroup examining central retinal artery occlusion slide; DM ¼
subgroup examining diabetes mellitus slide; GRP ¼ group (group 1 ¼ cohort taught without multisensory teaching reinforcement; group 2 ¼ cohort taught
with multisensory teaching reinforcement); MI ¼ Michigan module total test score; MI fundus ID ¼ fundus picture identification subsection score of the
Michigan test; PAP ¼ subgroup examining papilledema slide; quiz Pic ID ¼ picture identification subsection score of the end-rotation in-house quiz score;
syst dz manif ¼ the manifestations of systemic disease Michigan subsection score. N¼105 GRP1 overall; n¼37 Grp1 CRAO; N¼38 GRP 1 DM;
N¼30 GRP 1 PAP; N¼94 GRP2 overall; N¼32 GRP2 CRAO; N¼30 GRP2 DM; N¼32 GRP2 PAP.

(P < 0.001). In group 2, where Michigan subscores were available
for analysis, there was no difference between the systemic disease
(P ¼ 0.87) or picture identification module scores (P ¼ 0.26)
among the 3 slide groups. There was no difference between the
systemic disease (P ¼ 0.87) or picture identification module scores
(P ¼ 0.26) among the 3 slide subgroups.
On the ERQ picture portion, all but 1 of the 94 students could
identify papilledema, 89 students (95%) could identify a cotton wool
spot, 86 students (91%) could identify disc neovascularization, 77
students (82%) could identify a Roth spot, 81 students (86%) could
identify hard exudates, 63 students (67%) could identify a flame
hemorrhage, and 39 students (41%) could identify a blot hemorrhage.
Only 39 students (41.4%) recognized optic atrophy, 16 students (17%)
recognized a microaneurysm, and 12 students (13.8%) recognized a
CRAO. In addition to being discusseddand tested fordin the
Michigan module, all were components of slides examined and dis-
cussed at length in the small-group interactive sessions. Educational
experience was not a factor in performance; students in the first 2
rotations did not do statistically significantly better than those in the last
2 rotations (P ¼ 0.26) on the ERQ.
Discussion
According to medical educator and informaticist Charles P.
Friedman, the ubiquity of electronic media obviates the need
for medical students to memorize information, because ac-
curacy is increasingly evanescent; partial medical knowl-
edge is sufficient, supplemented by that available “in the
cloud” (Merrill Flair Lecture, Association of American
Medical Colleges Annual Meeting, November 2011; Den-
ver, Colorado). Students increasingly rely on their pocket
electronic devices as reference tools on rotations and are
adept at retrieving information. However, their capacity to
recall that information is diminished,8 functionally
resembling a form of memory-based factual processing
pathway suppression equivalent to visual pathway sup-
pression, namely, amblyopia. Our data add further support
to that argument. When the Internet goes down or electricity
faltersdfor whatever reasondwill clinical care be
jeopardized?
This study suggests a disturbingly short duration of
retention. Although immediate online post-test scores on the
Michigan online module were nearly perfect, the average on
an in-house test on that same covered material at rotation’s
end averaged 59.4%. Although groups 1 and 2 differed
significantly in their performance on the Michigan test, the
scores were so high and the standard deviation so small that
the practical significance was minimal compared with the
drastically lowered mean at the rotation’s end. That the
knowledge portion of the ERQ was perceived as more
difficult simply reflects that applying information to clinical
scenarios is more difficult than simply regurgitating facts.
However, the striking difference in scores on identification
of similar pictures of fully developed disc edema or CRAO
on the Michigan versus the ERQ reflects not a matter of
difficulty, but rather the students’ inability to apply and
synthesize the same knowledge at a later point in time.
Although facts and theories may change as research
proliferates, physical findings do not. Pattern recognition,
dependent on astute observation skills, forms the basis of
diagnostic abilitydin the fundus, on the skin, or with
physical findings that are detected visually, aurally, or
tactilelydas a whole. Prior studies indicate that the duration

5
 Ophthalmology Volume -, Number -, Month 2015
of skills retention correlates inversely with time elapsed
from initial instruction to the time of testing rather than the
level of initial skills mastery, with an approximate 10%
decrement in funduscopic skill by 6 months.4,9
The relative ease of diagnosing DM fundus changes in
both groups may stem from the time spent with the
instructor (L.M.L.) on the simulator for that case in both
cohorts. The fact that mean raw CLEO scores were similar,
as were remediations and perfect scores, but with a wide
spread range of scores between the 2 cohorts suggests that
either there was recognition of the disc mantra elements but
an inability to synthesize the correct diagnosis, or that di-
agnoses were made but not substantiated; evidence of both
occurred in each group.
All but one student in group 2 were able to identify pap-
illedema on a picture, and yet only approximately 60% in the
group with added proprioceptive training were able to
formulate the diagnosis in the simulatordnot statistically
significantly better than the scores for the nonproprioceptively
trained group, suggesting minimal improvement with this type
of multisensory reinforcement. Limitation in ophthalmoscopic
technique may well have been the predominant factor. Only
with practice will that improve. One cannot expect to be able
to master auscultation of the heart or funduscopy instanta-
neously, but the ability to examine and interpret findings is
what differentiates a physician from a technician.
A disturbing trend favors a streamlined physical, the “core
þ clusters” paradigm, reducing practice of what some edu-
cators have decided are ancillary skills to a few select clinical
scenarios.10,11 Relevant specialty examination skills cannot
be marginalized and then be expected to be available on
demand. Only routine practice during complete head-to-toe
physical examinations in medical school will result in
improved skill and confidence and assurance that the skill will
be available on demand later. Incidental findingsdfor
example, retinal embolidon routine screening physicals can
trigger diagnosis and treatment before irreversible and
devastating morbidity ensues. Although ophthalmoscopy ul-
timately will be replaced by mobile fundus photography, the
fundus pattern recognition issues are the same, whether
viewed through an eyepiece or on a screen. This study sug-
gests that the time to streamline the physical examination is
later in practice, guided by clinical judgment that comes with
maturity, and not in early training during medical school,
when diagnostic skills are just being learned.
Limitations of the study include the availability of
comparative Michigan subset question data for less than half
of the 2 cohorts combined, but still significant at 94 students.
There were a total of 162 Michigan examination questions
versus 24 on the in-house quiz, although the latter were tar-
geted to specific items discussed in class. In the comparison of
picture identification in the ERQ with that in the Michigan
examination, the picture identification question sample size
was small (10 for the ERQ and 17 for the Michigan exami-
nation). When comparing results for early and late academic
year results, the cohorts were reduced to roughly 20 students
for each time point. Ideally, retesting with the exact Michigan
online test at rotation’s end in addition to the ERQ would have
been preferable for exact comparison; however, the material
tested was the same, and the end point was the ability to
6
recognize and apply the knowledge rather than recognize a
previously tested question.
Nearly 15 years ago, the elimination of a dedicated required
ophthalmology clerkship at the University of California,
Irvine, prompted the institution of longitudinally integrated
content embedded in mainstream classes and rotations
throughout the first 3 years. Studies of acquisition and reten-
tion of ophthalmic knowledge and skills through that curric-
ulum demonstrated that reiteration was key to retention.4,9 The
Medical Student Educators Group of the Association of Uni-
versity Professors of Ophthalmology was founded to allow for
collaboration on novel strategies to introduce and reinforce
core ophthalmic content in the mainstream curriculum. The
group focuses on the development and assessment of efficacy
of innovative open-access teaching tools with which key
ophthalmic content can be introduced in a manner adaptable to
the needs of different learning environments. The Michigan
modules, being a self-contained unit, are readily applicable
online in any learning environment, and therefore results are
independent of any particular individual curriculum.
This study, while focused on ophthalmic findings, has
broader implications for the development of diagnostic ability
and clinical reasoning in general. Pattern recognition skills are
central to the interpretation of histologic or pathologic slides,
patterns of dermatologic eruptions, or a constellation of general
physical findings that constitute a diagnosis.
Spaced education has been used to identify and remediate
struggling students for the United States Medical Licensing
Examination score (USMLE) examination.12 Pulsed online
modules and quizzes over time, superimposed on the regular
required curriculum, may be helpful to increase the
motivation of students to retain specialty knowledge and
skills gained. Departments of surgery have found that
competency-based preparation targeted to fourth-year stu-
dents matched into surgery programs improved basic surgical
skill competence and confidence when exposed to a month-
long rotation with interactive didactics, simulated surgical
ward emergencies, and procedural instruction and skills
practice in April of the fourth year.13 A targeted advanced
clinical skills capstone course reiterating relevant ophthalmic
content and other specialty skills following the match and
immediately preceding postgraduate year 1 may enhance
retention in the face of imminent application during
internship and reinforcement during residency. Further
studies are indicated to determine the degree to which
pulsed online modules over time, pulsed requizzing on the
material covered, capstone course reviews, and other
interventions by the Medical Student Educators Group of
the Association of University Professors of Ophthalmology
may improve retention of ophthalmic knowledge and skills.
References
1. Sharma S. A single-blinded, direct observational study of PGY-
1 interns and PGY-2 residents in evaluating their history taking
and physical examination skills. Permanente J 2011;15:23–9.
2. Kharb P, Samanta PP, Jindal M, Singh V. The learning styles
and the preferred teaching-learning strategies of first year
medical students. J Clin Diagn Res 2013;7:1089–92.
 Lippa and Anderson 
Online and Multisensory Learning
3. Lippa L. The disc mantra: an algorithm for fundus description.
AAMC MedEdPORTAL; 2008. Available at: www.meded-
portal.org/publication/1086. Accessed August 23, 2014.
4. Mottow Lippa L, Boker J, Duke A, Amin A. A novel three
year longitudinal pilot study of medical students’ acquisition
and retention of screening eye exam skills. Ophthalmology
2006;113:133–9.
5. Mottow-Lippa L, Boker J, Duke A, Amin A. Correction to
figure for Ophthalmology 2006;113:133e9. Ophthalmol
2006;113:984.
6. Mottow-Lippa L, Boker J. Simulator assessment of fundu-
scopic skills in three consecutive medical school classes.
J Acad Ophthalmol 2009;2:1–5.
7. Trobe J. The Eyes Have It. Available at: http://www.kellogg.
umich.edu/theeyeshaveit/. Accessed June 13, 2014.
8. Sparrow B, Liu J, Wegner DM. Google effects on memory:
cognitive consequences of having information at our finger-
tips. Science 2011;333:776–8.
Footnotes and Financial Disclosures
Originally received: May 12, 2015.
Final revision: June 16, 2015.
Accepted: June 17, 2015.
Available online: ---. Manuscript no. 2015-756.
1
Department of Ophthalmology, Gavin Herbert Eye Institute, University of
California, Irvine, Irvine, California.
2
Department of Emergency Medicine, University of California, Irvine,
Irvine, California.
Presented in part at: American Academy of Ophthalmology Annual
Meeting, October 2014, Chicago, Illinois; Gavin Herbert Eye Institute
Donors Event, November 2014; Association of University Professors of
Ophthalmology Annual Meeting, January 2105, Tucson, Arizona (as AAO/
AUPO Excellence in Medical Student Education Award Lectures).
Financial Disclosure(s):
The author(s) have no proprietary or commercial interest in any materials
discussed in this article.
9. Mottow Lippa L, Boker J, Stephens F. A prospective study of
the longitudinal effects of an embedded specialty curriculum
on physical examination skills using an ophthalmol model.
Acad Med 2009;84:1622–30.
10. Gowda D, Blatt B, Fink MJ, et al. A core physical exam for
medical students: results of a national survey. Acad Med
2014;89:436–42.
11. Yudkowsky R, Otaki J, Lowenstein T, et al. A hypothesis-
driven physical examination learning and assessment proce-
dure for medical students: initial validity evidence. Med Educ
2009;43:729–40.
12. Kerfoot BP, Baker H, Pangaro L, et al. An online spaced ed-
ucation game to teach and assess medical students: a multi-
institutional prospective trial. Acad Med 2012;87:1443–9.
13. Anotonoff MB, Swanson JA, Green CA, et al. The significant
impact of a competency-based preparatory course for senior
medical students entering surgical residency. Acad Med
2012;87:308–19.
Supported in part by an unrestricted departmental grant from Research to
Prevent Blindness, Inc., New York, New York.
Author Contributions:
Conception and design: Lippa
Analysis and interpretation: Lippa, Anderson
Data collection: Lippa
Obtained funding: none
Overall responsibility: Lippa, Anderson
Abbreviations and Acronyms:
CLEO ¼ Clinical Learning Experience in Ophthalmoscopy;
CRAO ¼ central retinal artery occlusion; DM ¼ diabetes mellitus;
ERQ ¼ end-rotation quiz.
Correspondence:
Linda Mottow Lippa, MD, Gavin Herbert Eye Institute, Department of
Ophthalmology, University of California, Irvine, 850 Health Sciences
Road, Irvine, CA 92697. E-mail: llippa@uci.edu.