PE R S PE C T IV E
sessments could evaluate whether
Preparing Physicians for the Clinical Algorithm Era
Preparing Physicians for the Clinical Algorithm Era
Katherine E. Goodman, J.D., Ph.D., Adam M. Rodman, M.D., M.P.H., and Daniel J. Morgan, M.D.​​
Downloaded from nejm.org at UNIVERSITA STUDI DI TORINO on August 15, 2023. For personal use only. No other uses without permission.
Copyright © 2023 Massachusetts Medical Society. All rights reserved.
able for some people, particularly Disclosure forms provided by the authors
potential PrEP users, particularly
those from underserved popula-
tions, would be interested in this
option. Modeling studies could
An audio interview project the public
with Douglas Krakower health effects of
is available at NEJM.org
OTC availability so
that the potential benefits asso-
ciated with increased PrEP access
could be weighed against the po-
tential risks associated with re-
duced monitoring. Strategies for
minimizing potential risks, such
as development and FDA approv-
al of more sensitive HIV self-tests
and hepatitis B self-tests, could
be pursued. Finally, self-selection
and label-comprehension stud-
ies would be needed to evaluate
whether people would use PrEP
safely and effectively without a
clinician’s supervision, including
whether they would seek medical
care if they have contraindica-
tions to OTC use or need testing
for STIs.
OTC availability of PrEP
wouldn’t be a panacea. Even with-
out the costs of provider visits and
laboratory monitoring, generic
TDF–FTC may remain unafford-
C linical decision support (CDS)
systems provide information
or data, usually at the point of
care, to guide clinical decision
making and help improve health
care delivery. As the use of these
systems has grown since their
introduction in the 1970s, the
scope of their clinical applications
n engl j med 389;6  nejm.org  August 10, 2023
The New England Journal of Medicine
Over-the-Counter HIV Preexposure Prophylaxis
with the additional cost of an
HIV self-test. Legislation was re-
cently introduced to require insur-
ers to fully cover the cost of OTC
oral contraceptive pills when they
become available. Similarly, poli-
cies requiring insurance coverage
for OTC PrEP and HIV self-tests
would be critical to keeping PrEP
affordable. A national PrEP pro-
gram has been proposed to sup-
port PrEP care for people who are
uninsured,5 and inclusion of OTC
medications in such a program
could help ensure broad and eq-
uitable access.
It has taken 63 years since ini-
tial approval to free the Pill from
prescription-only status. We can’t
wait that long for PrEP: the fed-
eral Ending the HIV Epidemic in
the U.S. initiative aims to reduce
national HIV incidence by 90% by
2030. Building on lessons learned
from contraception, we believe it’s
incumbent on the HIV-prevention
community — including health
professionals, advocates, and man-
ufacturers — to lay the ground-
work for a collaborative movement
to “free the PrEP.”
has changed. Whereas initially,
CDS primarily automated the
provision of “facts” (e.g., drug-
interaction checkers), many cur-
rent CDS systems algorithmically
make predictions under conditions
of clinical uncertainty. Algorithmic
CDS assumes many forms, from
a simple regression-derived risk
are available at NEJM.org.
From the Division of Infectious Diseases,
Beth Israel Deaconess Medical Center
(D.K.), and the Department of Population
Medicine, Harvard Medical School and
Harvard Pilgrim Health Care Institute (D.K.,
J.L.M.) — both in Boston.
This article was published on August 5,
2023, at NEJM.org.
1. Zhao A, Dangerfield DT II, Nunn A, et
al. Pharmacy-based interventions to in-
crease use of HIV pre-exposure prophylaxis
in the United States: a scoping review. AIDS
Behav 2022;​26:​1377-92.
2. Potter JE, McKinnon S, Hopkins K, et al.
Continuation of prescribed compared with
over-the-counter oral contraceptives. Obstet
Gynecol 2011;​117:​551-7.
3. Arnold T, Whiteley L, Elwy RA, et al. Map-
ping Implementation Science with Expert
Recommendations for Implementing Change
(MIS-ERIC): strategies to improve PrEP use
among black cisgender women living in Mis-
sissippi. J Racial Ethn Health Disparities 2022
November 17 (Epub ahead of print).
4. Draft evidence review. Pre-exposure pro-
phylaxis for the prevention of HIV infection:​
a systematic review for the U.S. Preventive
Services Task Force. December 13, 2022
(https://www​.­uspreventiveservicestaskforce​
.­org/​­uspstf/​­document/​­draft​-­evidence​-­review/​
­prevention​-­human​-­immunodeficiency​-­virus​
-­h iv​-­infection​-­prep).
5. Johnson J, Killelea A, Farrow K. Investing
in national HIV PrEP preparedness. N Engl J
Med 2023;​388:​769-71.
DOI: 10.1056/NEJMp2305644
Copyright © 2023 Massachusetts Medical Society.
Over-the-Counter HIV Preexposure Prophylaxis
calculator to a more complex sys-
tem based on machine learning
or artificial intelligence that op-
erates in the electronic health
record.
The way in which physicians
interpret and act on algorithmic
CDS predictions can substantial-
ly affect patient care. For exam-
483
PERS PE C T IV E
ple, sepsis warning systems use
real-time data to identify patients
in early-stage sepsis before most
physicians detect clinical deterio-
ration. When physicians act on
such CDS alerts quickly, the likeli-
hood of survival may increase
significantly.1 However, inaccurate,
biased, or poorly implemented
CDS algorithms can cause patient
harm. Sepsis algorithms that fire
frequent false positive alerts may
prompt unnecessary antibiotic use
or, conversely, result in “alert fa-
tigue,” causing physicians to fail
to intervene when necessary. CDS
false negatives or “misses” may
cause physicians to overlook pa-
tients who require early treatment.
Given these stakes, the U.S.
government recently took sub-
stantial steps to ensure that CDS
algorithms are safe and effective
for clinical use. The Food and
Drug Administration (FDA) began
regulating many CDS algorithms
as medical devices — a move
that we hope will improve their
quality, interpretability, and trans-
parency for real-time clinical de-
cision making.2 Separately, the
Department of Health and Human
Services has proposed to regulate
bias in clinical algorithms under
health care antidiscrimination
laws.2 We believe this focus on
CDS algorithms and the machines
that deploy them could represent
an important first step toward en-
suring that such algorithms mean-
ingfully improve patient care. The
next and larger step, however, re-
quires focusing on human users:
teaching physicians how to use
CDS effectively.
Fundamentally, CDS algorithms
are prediction models. They cal-
culate the likelihood of a patient
outcome or stratify patients by
risk on the basis of some set of
484
Downloaded from nejm.org at UNIVERSITA STUDI DI TORINO on August 15, 2023. For personal use only. No other uses without permission.
Copyright © 2023 Massachusetts Medical Society. All rights reserved.
Preparing Physicians for the Clinical Algorithm Era
input variables (e.g., age, coexist-
ing conditions, or laboratory val-
ues). These predictions take the
form of probabilities, not black-
or-white “answers,” though many
physicians erroneously treat them
as the latter. When the goal is
prognostic — for example, “Is
this patient with heart failure
likely to have an adverse event af-
ter discharge?” — it’s obvious
that the CDS output will be an
uncertain prediction. But when
CDS algorithms are used for di-
agnosis, their probabilistic out-
puts are inconsistent with most
physicians’ conceptions of diag-
nosis.3 In medical education and
practice, diagnosis retains its
long-standing focus on patho-
physiology. Most instruction in
diagnosis takes place in patho-
physiology courses or case confer-
ences and involves using heuris-
tics such as schemas and illness
scripts.4 Clinical discussions of
probability are often limited to
simplified aphorisms, such as
Occam’s razor (“When you hear
hoofbeats, think horses, not ze-
bras”) and Hickam’s dictum (“Pa-
tients may have as many diseases
as they damn well please”).
Although pathophysiological
constructs and heuristics can help
physicians reason through diag-
nostic uncertainty, they are not
how CDS algorithms operate. Al-
gorithms are probabilistic and re-
quire conceptualizing a given di-
agnosis as more or less likely,
while retaining some explicit de-
gree of uncertainty. Although such
a probabilistic perspective is the
basis of evidence-based diagnosis
and Bayesian diagnostic reason-
ing, instruction in these concepts
is limited during medical educa-
tion and has not led to improve-
ments in physicians’ numeracy.3
n engl j med 389;6  nejm.org  August 10, 2023
The New England Journal of Medicine
As CDS algorithms play a
growing role in clinical care,
probabilistic information will be-
come increasingly critical to every-
day practice, as will the need to
interpret probabilities rapidly for
clinical decision making. To pre-
pare physicians effectively, medi-
cal education and clinical train-
ing will therefore have to more
explicitly cover probabilistic rea-
soning tailored specifically to CDS
(see table).
First, students, trainees, and
other physicians can learn that
probability and uncertainty are
fundamental aspects of diagnosis
and use visualization techniques
to make thinking in terms of
probability more intuitive. Cur-
rently, even when physicians use
probabilities, their estimates are
often inaccurate, and their un-
derstanding of test and algo-
rithm performance characteris-
tics is poor.3 To support real-time
use of probabilistic information,
early in medical school students
can practice working with proba-
bilities and interpreting perfor-
mance measures such as sensitiv-
ity and specificity, with evaluation
of these skills on board exams.
Because probability is founda-
tional to much of evidence-based
medicine, improving physicians’
probabilistic skills offers benefits
beyond CDS.
Second, with this probabilistic
foundation in place, physicians
can practice incorporating algo-
rithmic output into their clinical
decision making. CDS predictions
require critical evaluation and
clinical judgment. Some evidence-
based diagnosis instruction al-
ready provides relevant founda-
tional concepts, such as calculation
of pretest and post-test probabil-
ities and positive and negative
 PE R S PE C T IV E Preparing Physicians for the Clinical Algorithm Era

Recommended Changes to Medical Education and Training to Improve Probabilistic Reasoning

and Support Effective Use of CDS Algorithms.*

Recommendation Approaches to Implementation

Preclinical medical education

Teach probability in medical school using intuitive,
modern approaches
Teach probabilistic clinical reasoning
Assess probability and probabilistic reasoning skills
Teach core, foundational working knowledge of CDS
and EHR implementation, relevant to clinical use
Practice interpreting CDS output in applied learning
Clinical training
Reinforce probabilistic training and application
Build CDS interpretation into curricula
Reinforce working knowledge of CDS and EHR imple-
mentation, relevant to clinical use
Include working knowledge of CDS in ACGME core
competencies
• Create new curricula or access online curricula that use natural frequency trees
and icon arrays to visualize probability of disease and to convey the concept
that probability is fundamental to clinical medicine.
• Integrate instruction in probability and probabilistic reasoning throughout med-
ical school curricula, beyond diagnosis courses.
• Emphasize practical examples of probabilistic reasoning in both CDS use and
traditional evidence-based diagnosis.
• Encourage the use of gamified training for honing probabilistic reasoning skills.
• Include clinically relevant questions of probabilistic interpretation of CDS on
USMLE board and shelf exams (instead of questions on definitions).
• Integrate the basics of machine learning into the curriculum, including discus-
sions about biases and equity.
• Make explicit the human–technology interaction that often determines whether
a CDS is adopted.
• Discuss principles of user-centered design of CDS that affect whether an algo-
rithm is accepted and how it is used.
• Provide a simplified overview of how CDS works in the clinical EHR.
• Develop and use CDS-specific problem-based learning scenarios that empha-
size core concepts:
— Applying CDS algorithms to individual patients.
— Examining how different inputs affect prediction.
— Discussing potential sources of bias in algorithms.
— Interpreting basic model performance concepts.
— Communicating with patients about CDS-guided decision making.
• Provide resources for incorporating probability into case discussions.
• Develop longitudinal curricula on the variety and use of CDS.
• Incorporate basic algorithmic CDS principles (e.g., accuracy and bias) into clini-
cal discussions and real-world case studies to elucidate how CDS accuracy or
bias may affect clinical decision making.
• Incorporate technical CDS knowledge and assessments (e.g., how CDS oper-
ates in the EHR and where to find more information regarding CDS alerts) into
clinical orientations for new physicians.
• The ACGME requires evidence-based medicine–related skills as part of its core
competency in practice-based learning and improvement; these should be up-
dated to explicitly include CDS interpretation.

* ACGME denotes Accreditation Council for Graduate Medical Education, CDS clinical decision support, EHR electronic health record, and
USMLE United States Medical Licensing Examination.

predictive values. In practice, how-
ever, these concepts are rarely
taught as clinical skills that
could be used in fast-paced deci-
sion making.3 The probabilistic
reasoning skills required for CDS
interpretation are also sufficiently
distinct from diagnostic test in-
terpretation to require dedicated
instruction (see flowchart). Case
studies and practice-based learn-
ing can build on current training
and be tailored to the use of algo-
rithms in actual practice settings.
Consider, for instance, a CDS
algorithm for predicting whether
a bloodstream infection is anti-
biotic-resistant while awaiting

n engl j med 389;6  nejm.org  August 10, 2023 485

The New England Journal of Medicine
Downloaded from nejm.org at UNIVERSITA STUDI DI TORINO on August 15, 2023. For personal use only. No other uses without permission.
Copyright © 2023 Massachusetts Medical Society. All rights reserved.
PERS PE C T IV E Preparing Physicians for the Clinical Algorithm Era

they do not include or may have

Diagnostic 1. Physician estimates 2. Physician 3. Post-test probability of disease is missed (e.g., previous resistant in-
Testing pretest probability orders estimated from pretest probability,
Pathway of disease tests test characteristics, and test results fection at an out-of-network hos-
pital). This assessment, also re-
Diagnostic A. CDS estimates quiring clinical judgment, is the
CDS pretest probability dial that tunes an algorithm’s pos-
Algorithms of disease
itive or negative predictive value to
B. CDS combines pretest-probability estimation, test results, and other inputs
settle on a given patient’s final
to generate patient’s probability of disease probability of an outcome. This
process is similar but not identi-
cal to interpreting diagnostic test
CDS output becomes the pretest probability for sequential updating
results, and physicians can engage
in practice-based learning, with
Diagnostic Testing Pathway and CDS Algorithms. real examples, to master it.
Diagnosis requires Bayesian reasoning, which involves estimating the patient’s pretest probability Physicians do not need to be-
of disease, ordering diagnostic tests, and estimating the post-test probability of disease on the come experts in math or comput-
basis of the pretest probability, test performance characteristics, and test results. Example A er science to use CDS algorithms
describes diagnostic clinical decision support (CDS) algorithms that operate only at the first step effectively: CDS systems will au-
of this process to estimate the pretest probability of disease and guide diagnostic testing decisions
tomate most calculations and,
(e.g., Wells’ criteria for pulmonary embolism), whereas example B describes diagnostic CDS algo-
rithms that operate at all three steps, combining pretest-probability estimation and some test when implemented well, provide
results to estimate a patient’s probability of disease (e.g., sepsis warning systems). The latter user-friendly interfaces. Rather,
algorithms generally “fire” in the electronic medical record without intentional engagement from clinicians need to understand
the physician and guide treatment decisions or subsequent testing. where in the decision-making
pathway individual CDS algo-
rithms are operating and how
microbiologic test results.5 In a pa- CDS appropriately is recognizing various clinical and institutional
tient with bacteremia, the algo- that algorithmic positive and neg- factors will change the interpre-
rithm might have 74% sensitivity ative predictive values — and tation of the resulting predic-
and 87% specificity for the de- therefore the meaning of a given tions. Reporting of algorithms’
tection of antibiotic resistance. CDS output — change with the performance characteristics and
With an institutional resistance context. Moreover, only clinicians, methods, for which recent FDA
prevalence of 15%, a high-risk not algorithms, can decide wheth- regulatory actions provide incen-
flag would convey that the prob- er an 11% risk of ineffective ther- tives, will make this type of rea-
ability (positive predictive value) apy is acceptable; algorithms can- soned interpretation possible.
of resistant infection was 50%, not replace clinical judgment. Given technological advances
and a low-risk flag that the proba- No individual patient is an and quality and transparency im-
bility of susceptible infection (neg- average patient. Algorithms take provements prompted by recent
ative predictive value) was 95%. many patient factors into account, U.S. federal regulation, CDS algo-
At these probabilities, physicians which is why this algorithm’s pos- rithms will increasingly be inte-
would probably accept the CDS itive and negative predictive values grated into routine clinical care.
predictions and initiate broad- or provide more nuanced probabili- We expect that, willingly or not,
narrow-spectrum therapy accord- ties than simply using institu- the current generation of trainees
ingly. But with an institutional tional prevalence to estimate re- will use CDS algorithms regular-
resistance prevalence of 30%, the sistance risk. But algorithms can ly in their practice. This shift will
probabilities would be 71% and never incorporate all relevant pa- bring powerful opportunities to
89%, respectively — and a low- tient factors and clinical knowl- improve care — but also draw-
risk flag would indicate an 11% edge. Another component of using backs, if algorithms are relied on
chance that the patient actually algorithms effectively is under- inappropriately. For CDS to do its
has a resistant infection. A fun- standing their limitations, by job effectively, we need to train
damental component of using considering what patient factors medical students in its use. Some

486 n engl j med 389;6  nejm.org  August 10, 2023

The New England Journal of Medicine

Downloaded from nejm.org at UNIVERSITA STUDI DI TORINO on August 15, 2023. For personal use only. No other uses without permission.
Copyright © 2023 Massachusetts Medical Society. All rights reserved.
 PE R S PE C T IV E
of the necessary changes will be
small and easily implemented, but
others will require conceptual
shifts in how we teach probabi-
listic reasoning through all stages
of medical education.
Disclosure forms provided by the au-
thors are available at NEJM.org.
From the Department of Epidemiology and
Public Health, University of Maryland
School of Medicine (K.E.G., D.J.M.), and the
VA Maryland Healthcare System (D.J.M.) —
both in Baltimore; the University of Mary-
Reading Kafka in the Hospital Cafeteria
Reading Kafka in the Hospital Cafeteria
Suzanne Koven, M.D.​​
T he last patient I was sched-
uled to see on the last day
of my 32 years as a primary care
physician was an elderly woman
who had ended each visit with
me during the decades I’d cared
for her by saying, “I love you,
Doctor!” — to which I would re-
ply, “I love you, too!” This affec-
tionate exchange seemed to me
the perfect finale to my clinical
career, and I imagined it often in
the weeks leading up to my re-
tirement. It never happened. Four
days before I was to see this pa-
tient and then hang up my stetho-
scope forever, I fell at home and
shattered my right wrist. With one
misstep, I was transformed from
a doctor to a patient, my carefully
planned departure hijacked by an
abrupt and painful exit.
I had surgery to repair my ra-
dius and ulna and, due to a neu-
rologic complication of the injury,
spent the next several weeks in a
world as quiet as my practice had
been bustling. My chief preoccu-
n engl j med 389;6  nejm.org  August 10, 2023
The New England Journal of Medicine
Downloaded from nejm.org at UNIVERSITA STUDI DI TORINO on August 15, 2023. For personal use only. No other uses without permission.
Copyright © 2023 Massachusetts Medical Society. All rights reserved.
Preparing Physicians for the Clinical Algorithm Era
land Institute for Health Computing, Bethes-
da (K.E.G.); and the Department of
Medicine, Beth Israel Deaconess Medical
Center, Boston (A.M.R.).
This article was published on August 5, 2023,
at NEJM.org.
1. Adams R, Henry KE, Sridharan A, et al.
Prospective, multi-site study of patient out-
comes after implementation of the TREWS
machine learning-based early warning sys-
tem for sepsis. Nat Med 2022;​28:​1455-60.
2. Goodman KE, Morgan DJ, Hoffmann
DE. Clinical algorithms, antidiscrimination
laws, and medical device regulation. JAMA
2023;​329:​285-6.
pations became pain, pain medi-
cation, and side effects of pain
medication. I lost sleep and ap-
petite and, requiring my husband
to squeeze toothpaste onto my
toothbrush and wrap my arm in
a plastic garbage bag before
putting me in the shower, some
dignity.
One thing I had plenty of was
time. When I was seeing patients,
time had moved too quickly; I al-
ways ran behind. I had recurrent
nightmares about charts piling up.
Then, when records went paper-
less, my dreams did too, featur-
ing screens of unpopulated tem-
plates. Now that I was a patient,
time moved too slowly. Minutes
seemed like hours — or became
hours — as I waited for messag-
es and phone calls to my doctors
to be returned, waited for my
next dose of pain medication,
waited to be out of pain. To pass
the time, I scrolled through social
media and binge-watched televi-
sion shows. When I tired of these,
3. Morgan DJ, Pineles L, Owczarzak J, et al.
Accuracy of practitioner estimates of proba-
bility of diagnosis before and after testing.
JAMA Intern Med 2021;​181:​747-55.
4. Bowen JL. Educational strategies to pro-
mote clinical diagnostic reasoning. N Engl J
Med 2006;​355:​2217-25.
5. Goodman KE, Lessler J, Harris AD, Mil-
stone AM, Tamma PD. A methodological
comparison of risk scores versus decision
trees for predicting drug-resistant infections:
a case study using extended-spectrum beta-
lactamase (ESBL) bacteremia. Infect Control
Hosp Epidemiol 2019;​40:​400-7.
DOI: 10.1056/NEJMp2304839
Copyright © 2023 Massachusetts Medical Society.
Preparing Physicians for the Clinical Algorithm Era
I turned to books. Not being an
e-reader, I favored slim volumes I
could hold easily in my nondom-
inant, uninjured left hand.
One of these was Franz Kaf-
ka’s 1915 novella The Metamorpho-
sis. I’d read it many times, in-
cluding once a few years earlier
with the monthly reading group I
facilitate at my hospital. The story
concerns Gregor Samsa, a 27-year-
old man, the sole support of his
parents and teenage sister with
whom he lives, who wakes one
morning to find that he has turned
into a giant insect. This tale has
been read as an allegory of the
psychological struggle of sons
with fathers (Kafka’s relationship
with his own father was notori-
ously difficult), of the plight of
the worker, and of antisemitism
(Kafka was Jewish, and Jews have
historically been depicted in an-
tisemitic propaganda as vermin).
It also has much to tell us about
being a patient, as my colleagues
and I found when we discussed
487