PERSPECTIVE - HYPOTHESIS
race proxy in its assessment and clinicians
Implications of predicting race
variables from medical images
AI-predicted race variables pose risks and opportunities
for studying health disparities
By James Zou1, Judy Wawira Gichoya2,
Daniel E. Ho3, Ziad Obermeyer4
T
here are now more than 500 US Food
and Drug Administration (FDA)–
approved medical artificial intel-
ligence (AI) devices, and AI is used
in diverse medical tasks such as as-
sessing the risk of heart failure and
diagnosing cancer from images (1). Beyond
predicting standard diagnoses, AI models
can infer a substantial number of patient
features from medical images in ways that
humans cannot. For example, several stud-
ies have demonstrated that some AI models
can infer race variables (crude simplistic
categories) directly from medical im-
ages such as chest x-rays and cardiac
ultrasounds, even though there are
“…the ability of AI to predict race variables
no known human-readable race cor-
relates in these images (2–4). This has
raised concerns about the possibility
of AI systems to discriminate. At the
same time, AI predictions of demographic
attributes, including race variables, also
have the potential to help assess and moni-
tor health care disparities and generate new
insights into risk factors (5).
Race is a complex and problematic so-
cial construct that does not have a bio-
logical basis (6). The concept of race varies
across time and geography and according
to social factors. Demographic-prediction
AI models can impute (or infer) race vari-
ables that are captured in electronic health
records. These are simplistic and dated
reporting categories, such as white, Black,
and Asian. Despite such limitations, mea-
sures of health care equity and algorithmic
fairness often depend on stratifying out-
comes by race variables.
How AI predicts race variables is puz-
zling. It is likely that other demographic
attributes, comorbidities, or imaging ar-
tifacts are at play. For example, AI can
predict patients’ age and sex from cardiac
1
Department of Biomedical Data Science, Stanford
University, Stanford, CA, USA. 2Department of Radiology,
Emory University, Atlanta, GA, USA. 3Stanford Law School,
Stanford University, Stanford, CA, USA. 4School of Public
Health, University of California, Berkeley, Berkeley, CA,
USA. Email: jamesz@stanford.edu
SCIENCE science.org
ultrasound videos because heartbeat pat-
terns vary with age and sex (3). Therefore,
if patients of one race skew older or
younger in the training dataset, then the
model can use this correlation to predict
race. Measured confounders (variables
that may cause spurious association) that
could lead to correlations can be adjusted
for directly. But in a study of chest x-rays,
measured potential confounders were bal-
anced across the dataset, and a deep learn-
ing model was still able to predict race
variables from the images, achieving an
area under the receiver operating charac-
teristic curve (AUROC) above 0.9 (AUROC
is a common metric of prediction accuracy
from medical images could be useful
for monitoring health care disparity…”
where 1 is perfect) (2). This suggests that
unmeasured confounders also play a role.
Moreover, illnesses can be unequally dis-
tributed across racial groups; if those ill-
nesses have imaging manifestations, algo-
rithms could also use them to predict race.
Could the ability of AI to predict race
variables from medical images, whatever
the mechanism, exacerbate health care
disparities? Although diagnostic AI is
not trained to directly read out race vari-
ables, it is possible that race or its corre-
lates could be a shortcut that is used by
AI algorithms to make disease diagnoses.
There are some settings where this is pos-
sible and potentially harmful. For example,
suppose the patients of one race are more
likely to have their images taken by type
A machines and, because of differences in
where the machines are deployed, most
of the positive cases in the training data
are collected from type B machines. AI can
learn to use machine type to predict race
variables and disease status (it is less likely
to predict disease for images taken from
machine A). In deployment, the AI would
then underdiagnose patients of a particu-
lar race, leading to care disparity. This
raises serious questions of algorithmic dis-
crimination, especially if the AI uses the
are not aware of this.
Therefore, it is critical to evaluate the
reliability of AI in truly independent and
representative datasets in which race vari-
ables are available—for example, from pa-
tient self-reporting or linked administra-
tive data. Ideally, this dataset would come
from multiple settings or health systems to
avoid overfitting. Then, equity can be mea-
sured by disaggregating the model’s per-
formance by race variables to ensure that it
does not have excess errors or differential
performance between groups.
The ability of AI to pick up hidden phe-
notypes, such as race variables, from medi-
cal data could also be a resource for equity.
AI can generate valuable information for
auditing datasets to detect biases (7). For
example, if an AI model can accurately pre-
dict attributes that may not be medically
Downloaded from https://www.science.org on July 14, 2023
relevant (such as race, or the machine type
that took the image), this suggests possible
confounders in the training dataset. Maybe
there are spurious correlations involving
race variables in the data, or parts of the
hospital have different imaging setups. In
such cases, these confounders should
be explicitly accounted for in subse-
quent analyses. Unexpected predic-
tions by AI force these data warts
into the open and could improve
transparency.
In many settings, race variables are
not available, but laws in numerous coun-
tries require assessment of disparities.
How can it be ensured that a model, treat-
ment, or policy benefits a diverse popula-
tion if the relevant demographic attributes
are missing? Predicted race variables from
an AI algorithm could be a useful, albeit
imprecise, proxy. For example, if research-
ers develop a method to diagnose Long
Covid from chest x-ray images, but they do
not have race metadata for many images
in their dataset, a demographic-prediction
AI model could impute race variables from
their images. The model’s performance
when stratified by race variables can then
be assessed to ensure that it works well
across diverse populations. There is prec-
edent for this imputation approach: The
US Centers for Medicare and Medicaid
Services use race inferred from surname
and location to monitor health care dis-
parity (8), and the Consumer Financial
Protection Bureau uses a similar approach
to monitor equity in lending (9). The
Internal Revenue Service is statutorily pro-
hibited from using Census demographic
data to assess disparities in tax administra-
tion, and the US Treasury Department has
in turn relied on imputed race variables to
14 JULY 2023 • VOL 381 ISSUE 6654 149
 SPECIAL SEC TION A M AC H I N E - I N T E L L I G E N T WO R L D
uncover important disparities in tax ex-
penditures and audits (10).
The usage of imputed race variables in
health care should focus on disparity as-
sessment and algorithmic auditing instead
of medical decision-making. The accuracy
of imputations should be carefully vali-
dated in each new application. When the
imputed race variables are broad catego-
ries, downstream analysis may miss dispar-
ities within subgroups of the population.
Furthermore, in applications involving
interpretations of DNA sequencing, other
population labels such as genetic ancestry
are more meaningful categories for assess-
ment than race variables (11).
AI could help to provide more nuanced
representations of race than the standard
reporting categories, which are often very
coarse. AI produces continuous scores or
probabilities, not categories. This mirrors
work in human genetics showing that it
is more accurate to model populations as
continuous spectrums of variation in an-
cestry rather than modeling according to
a few discrete race labels (12). AI models,
such as neural networks, also learn to rep-
resent individuals as points in a continu-
ous space (often called embedding) (13).
Instead of treating, for example, “Asian” as
a single discrete category, an AI model that
assesses chest x-rays maps images from
Asian patients to a continuous geometric
space before making its diagnosis predic-
tion. Variation in this space could reflect
ancestry subgroups within Asians as well
as differences in health care access, comor-
bidities, and other unmeasured factors.
More research is needed to characterize
the implications for health care from dif-
ferences between patient embedding.
AI represents patients on a continuous
spectrum rather than as discrete racial
categories. This makes it easier to evaluate
and ensure that the model makes accurate
predictions across fine-grained subpopula-
tions. Instead of measuring how well the
AI model performs on Asians as one group,
which can mask limitations on less popu-
lous subgroups, the model’s performance
can be assessed on different clusters or
spectra of patients in the embedding space.
Suppose the model makes more mistakes
than expected in a particular data cluster;
further investigation of what subpopula-
tion is represented in this error cluster can
be undertaken, and the model could be
retrained to improve on this subgroup. In
response to advocacy by civil rights groups,
the White House recently proposed updat-
ing the 1997 coarse race reporting stan-
dard to foster data disaggregation accord-
ing to reporting outcomes by subgroups
(for example, Vietnamese American versus
150 14 JULY 2023 • VOL 381 ISSUE 6654
Asian American). It may take years for that
process to materialize in health data, and
AI imputations can potentially help to un-
cover health care disparities between gran-
ular subgroups in the meantime (14).
Although race variables are not a gener-
ally meaningful category in medicine, the
ability of AI to predict race variables from
medical images could be useful for monitor-
ing health care disparity and ensuring that
algorithms work well across diverse popu-
lations. Where feasible, researchers can
try to predict race and other demographic
variables from their data as a mechanism to
thoroughly audit their dataset and model.
If race variables and other attributes can
be accurately predicted, this could reveal
potential data or model biases that should
be addressed. Understanding what features
AI mechanistically uses to predict race
variables is important, and AI interpreta-
tion methods such as concept activation,
saliency map, and counterfactual explana-
tions could help. Overall, being transparent
and deliberate about race variable predic-
tions should reduce the risk of algorithmic
discrimination and could generate valuable
information about the dataset, the model,
and health care disparities. j
REF ERENCES AND NOTES
1. E. Wu et al., Nat. Med. 27, 582 (2021).
2. J. W. Gichoya et al., Lancet Digit Health. 4, e406 (2022).
3. G. Duffy et al., NPJ Digit. Med. 5, 188 (2022).
4. J. Adleberg et al., J. Am. Coll. Radiol. 19, 1151 (2022).
5. R. Poplin et al., Nat. Biomed. Eng. 2, 158 (2018).
6. A. Deyrup, J. L. Graves Jr., N. Engl. J. Med. 386, 501
(2022).
7. W. Liang et al., Nat. Mach. Intell. 4, 669 (2022).
8. A. Fremont, J. S. Weissman, E. Hoch, M. N. Elliott, Rand
Health Q. 6, 16 (2016).
9. Consumer Financial Protection Bureau, Using publicly
available information to proxy for unidentified race
and ethnicity (2014); https://www.consumerfinance.
gov/data-research/research-reports/using-publicly-
available-information-to-proxy-for-unidentified-race-
and-ethnicity.
10. J.-A. Cronin, P. DeFilippes, R. Fisher, Tax Expenditures
by Race and Hispanic Ethnicity: An application of the
U.S. Treasury Department’s race and hispanic ethnic-
ity imputation, Department of Treasury Office of Tax
Analysis working paper (2023).
11. Committee on the Use of Race, Ethnicity, and
Ancestry as Population Descriptors in Genomics
Research, Board on Health Sciences Policy,
Committee on Population, Health and Medicine
Division, Division of Behavioral and Social Sciences
and Education, National Academies of Sciences,
Engineering, and Medicine, Using Population
Descriptors in Genetics and Genomics Research:
A New Framework for an Evolving Field (National
Academies, 2023).
12. M. Yudell, D. Roberts, R. DeSalle, S. Tishkoff, Science 351,
564 (2016).
13. Y. LeCun, Y. Bengio, G. Hinton, Nature. 521, 436
(2015).
14. L. Cheng, I. Gallegos, D. Ouyang, J. Goldin,
D. E. Ho, Proceeding of the ACM Conference
on Fairness, Accountability, and Transparency
10.1145/3593013.3594034 (2023).
10.1126/science.adh4260
PERSPECTIVE
Mitigating bias
in AI at the
point of care
Promoting equity in AI
in health care requires
addressing biases at
clinical implementation
By Matthew DeCamp1 and
Charlotta Lindvall2,3,4
A
rtificial intelligence (AI) shows
Downloaded from https://www.science.org on July 14, 2023
promise for improving basic and
translational science, medicine, and
public health, but its success is not
guaranteed. Numerous examples
have arisen of racial, ethnic, gen-
der, disability, and other biases in AI ap-
plications to health care. In ethics, bias
generally refers to any systematic, unfair
favoring of people in terms of how they
are treated or the outcomes they experi-
ence. Consensus has emerged among sci-
entists, ethicists, and policy-makers that
minimizing bias is a shared responsibility
among all involved in AI development. For
example, ensuring equity by eliminating
bias in AI is a core principle of the World
Health Organization for governing AI (1).
But ensuring equity will require more than
unbiased data and algorithms. It will also
require reducing biases in how clinicians
and patients use AI-based algorithms—a
potentially more challenging task than re-
ducing biases in algorithms themselves.
Examination of bias in AI has tended to-
ward removing bias from datasets, analy-
ses, or AI development teams. For example,
because of unequal recruitment and en-
rollment, oncology datasets demonstrate
racial, ethnic, and global geographic biases
(2). In another example, developers as-
sumed that health care costs were a proxy
for health care needs, but then learned
that Black Americans often receive less
medical care even when they have greater
1
Center for Bioethics and Humanities, Department of
Medicine, University of Colorado Anschutz Medical
Campus, Aurora, CO, USA. 2Department of Psychosocial
Oncology and Palliative Care, Dana-Farber Cancer
Institute, Boston, MA, USA. 3Department of Informatics and
Analytics, Dana-Farber Cancer Institute, Boston, MA, USA.
4
Department of Medicine, Harvard Medical School, Boston,
MA, USA. Email: matthew.decamp@cuanschutz.edu
science.org SCIENCE
 Implications of predicting race variables from medical images
James Zou, Judy Wawira Gichoya, Daniel E. Ho, and Ziad Obermeyer

Science, 381 (6654), .

DOI: 10.1126/science.adh4260

View the article online

https://www.science.org/doi/10.1126/science.adh4260

Downloaded from https://www.science.org on July 14, 2023

Permissions
https://www.science.org/help/reprints-and-permissions

Use of this article is subject to the Terms of service

Science (ISSN 1095-9203) is published by the American Association for the Advancement of Science. 1200 New York Avenue NW,
Washington, DC 20005. The title Science is a registered trademark of AAAS.
Copyright © 2023 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim
to original U.S. Government Works