The American Journal of Bioethics

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/uajb20

Informed Consent for Clinician-AI Collaboration

and Patient Data Sharing: Substantive, Illusory, or
Both

Charles E. Binkley & Bryan C. Pilkington

To cite this article: Charles E. Binkley & Bryan C. Pilkington (2023) Informed Consent for
Clinician-AI Collaboration and Patient Data Sharing: Substantive, Illusory, or Both, The
American Journal of Bioethics, 23:10, 83-85, DOI: 10.1080/15265161.2023.2250289

To link to this article: https://doi.org/10.1080/15265161.2023.2250289

Published online: 09 Oct 2023.

Submit your article to this journal

Article views: 55

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=uajb20

THE AMERICAN JOURNAL OF BIOETHICS
2023, VOL. 23, NO. 10, 83–85
https://doi.org/10.1080/15265161.2023.2250289
OPEN PEER COMMENTARIES
Informed Consent for Clinician-AI Collaboration and Patient Data Sharing:
Substantive, Illusory, or Both
Charles E. Binkleya,b and Bryan C. Pilkingtonb,c
a
Hackensack Meridian Health; bHackensack Meridian School of Medicine; cSeton Hall University
In the piece, “What Should ChatGPT Mean for
Bioethics?” Professor Cohen proposes that the intro-
duction of AI generally, and generative AI specifically,
requires that patients be informed of, and consent to,
both their clinician collaborating with AI systems to
make clinical decisions, and their data being used to
train these systems (Cohen 2023). While, per se, the
claims are valid, when applied to AI systems for clin-
ical decision making, the informational and the vol-
itional components of informed consent deserve
separate consideration. First, this piece will consider
why information about clinician-AI collaborations and
data sharing should be communicated to patients.
Next, it will consider both the eventuality that consent
for data sharing and physician-AI collaboration will
become illusory, and also some arguments that might
justify the omission of consent. Finally, it will propose
potential means of communicating information about
AI in a way that is understandable across diverse
patient populations.
INFORMATIONAL CONSIDERATIONS FOR AI
Clinicians who collaborate with AI systems share with
the systems some of their own decisional agency and
thus introduce a third entity into the clinician-patient
relationship (Binkley and Pilkington 2023). Several
reasons have been proposed for why patients should
be informed when their clinician introduces an AI
system into their relationship. Some arguments are
rooted in the idea that patients have the expectation
of exclusivity in their relationships with clinicians,
and if a third party is introduced, they should be
informed (Cohen 2019). In other arguments, it is sug-
gested that if most patients would refuse to allow AI
CONTACT Charles E. Binkley charles.binkley@hmhn.org
ß 2023 Taylor & Francis Group, LLC
THE AMERICAN JOURNAL OF BIOETHICS 83
systems to participate in their care, then patients
should be informed, lest patient autonomy be violated.
(Richardson et al. 2021)
One argument in favor of informing patients if
their health data is used to train AI systems is that
their medical record is their digital phenotype, the
account of how their genetic code is expressed over
time. As such, it can be thought of as part of their
person, a complement to their own DNA. Thus,
patients’ sharing of their health information is a kind
of donation, and it is generally understood that
donors should be informed (Beskow 2016). As with
any donation, patients should be aware of the benefits
as well as the risks of donating their health data, even
if the data are de-identified (Mittelstadt and Floridi
2016). The ease with which personal information can
be reidentified, or how individuals can remain deiden-
tified but nonetheless profiled across platforms, high-
lights the potential for violating patient privacy and
confidentiality. Not only would this be a dereliction of
the clinician’s ethical duty, but it could also result in
harm to patients in terms of loss of insurance or
employment benefits should their personal health
information be readily available.
More important than the aforementioned consider-
ations is that informing patients about their clinician
collaborating with an AI system, and the patient’s per-
sonal health information being shared, engenders
trust. Clinician transparency and truthfulness are
essential ways through which patients come to trust
their clinicians (Kaldjian and Pilkington 2021). The
risk of nondisclosure about the involvement of AI or
the sharing of health information may lead patients to
wonder what other information was withheld from
them. This is not simply a consideration for clinician
Hackensack Meridian Health, Edison, NJ, USA.
84 OPEN PEER COMMENTARIES
practice, but for institutions, as well. In some cases,
clinicians may not have control over their collabora-
tions with AI systems. For instance, health systems
may choose to push out AI generated predictions to
clinicians as “Best Practice Alerts” within the elec-
tronic medical record. Clinicians must factor the
information into their decision making, whether or
not the information was requested, or even desired. It
is unlikely that individual clinicians will play a role in
deciding how patient data is used to train AI systems.
Nevertheless, it is with their clinician that patients
have a relationship, and it is the clinician who is vul-
nerable to mistrust if patients are not informed.
SEPARATING INFORMATION FROM VOLITION
IN INFORMED CONSENT
What would justify informing patients about clinician
AI-collaborations and the use of their personal health
information to train AI systems, without allowing for
consent or refusal? In the very near future, for many
clinical tasks, collaborations between clinicians and
AI systems will be proven to be more beneficial and
less harmful than clinicians acting alone. When this
happens, clinician-AI collaborations will likely be
considered the standard of care for those tasks.
Offering patients health care alternatives which do
not include clinician-AI collaborations may not only
be below the standard of care, but also potentially
harmful, especially if clinical skills have deteriorated
as a result of AI collaborations. It may not be oper-
ationally feasible to create workflows which could
provide equivalent care for patients who refuse to
allow their clinicians to collaborate with AI systems.
Additionally, clinicians may be concerned about pro-
fessional liability if patients are able to refuse to allow
them to collaborate with AI systems. Consent and
refusal may thus become illusory if reasonable and
equivalent alternatives to clinical AI collaborations
are not available.
Although informing patients about clinician-AI col-
laborations and the use of health information for
training AI systems can be considered separately,
patient data is necessary in order to train AI systems
to perform the tasks for which they are programmed.
Thus, there is no AI system without patient data to
train it. This creates a necessary quid pro quo. While
it is possible to segregate AI systems which use patient
data exclusively to make predictions from those sys-
tems that use data for both for predicting and train-
ing, it is far more difficult, if not impossible, to sort
patients within a health system into those who
consent to have their data used for training AI sys-
tems from those patients who refuse.
Even if it were technically or operationally possible
to allow patients to refuse to share their personal
health information in order to train AI clinical sys-
tems, a public goods argument could be made, similar
to that which has been made for the participation in
medical research (Schaefer, Emanuel, and Wertheimer
2009). Further, allowing patients to refuse to share
their health data runs the risk that if discrete groups
of patients refuse, training data would be unrepresen-
tative of the population at large. This could lead to AI
models making more accurate predictions for some
groups compared to others, thus perpetuating rather
than correcting the risk of AI bias.
The proposed justification for informing patients
about AI-clinician collaborations and patient data
sharing without consent is not intended to be with-
out qualification. AI systems collaborating with clini-
cians may be demonstrably superior to clinicians
acting alone before the system is deployed or early in
its clinical use. However, since almost all of these
systems are continuously learning and will undergo
regular updates, it is essential that the increased
benefit and/or decreased harm of the collaboration
be reestablished in the local clinical ecosystem and
when the system is updated. In addition, the risk of
patient harm that could result from the use of their
personal health data to train AI systems must be
mitigated in ways other than depersonalizing the
data as prescribed by current HIPAA standards.
These regulations were intended for a different era of
personal health information and seem woefully
unsuited to the current digital use of patient infor-
mation. Regulatory bodies must assume that patient
data can be reidentified and that patients can be pro-
filed across digital platforms. Other safeguards of
privacy and confidentiality will be required in order
to protect patients from harm.
THE WELL INFORMED INFORMER
Given that patients should be informed about both
their clinician’s collaboration with an AI system and
the donation of their personal health information
without the option of consenting or refusing, the issue
thus arises who, or what, should provide the informa-
tion, and how should it be provided? Generally, the
clinician responsible for performing an intervention is
also responsible for informing the patient about the
intervention since that clinician would best know the
patient and the details of the intervention being

offered. However, when it comes to AI systems and
data donations, the clinician may not possess suffi-
cient knowledge about the AI system, or its function
and validation, or the benefits and risks of data shar-
ing, to be the optimal informer. In fact, there is con-
cern that informing patients of the involvement of AI
systems in their care by clinicians without sufficient
foundational knowledge might overwhelm patients
(Blumenthal-Barby 2023).
Perhaps the ideal informer is actually an AI pow-
ered chatbot. Already these systems have been pro-
posed as ideal for patient education with early data
supporting this notion (Lee et al. 2023). Prior to their
deployment, systems would need to be trained and
validated across a spectrum of knowledge as well as
patient demographics such as spoken language, educa-
tional level, and extent of informational preferences.
Some patients will want more information, will have a
greater foundation of knowledge, and will know more
about the risks and benefits than others. It would be
impossible to have a group of humans with the acces-
sibility, availability, expertise, and affordability that
will be required to adequately inform patients about
clinical AI systems. These systems are uniquely quali-
fied to provide patients with substantive information
without the illusion of consent.
DISCLOSURE STATEMENT
No potential conflict of interest was reported by the
author(s).
FUNDING
The author(s) reported there is no funding associated with
the work featured in this article.
THE AMERICAN JOURNAL OF BIOETHICS 85
ORCID
Charles E. Binkley http://orcid.org/0000-0001-9290-9876
Bryan C. Pilkington http://orcid.org/0000-0001-9373-
8300
REFERENCES
Beskow, L. M. 2016. Lessons from HeLa cells: The ethics
and policy of biospecimens. Annual Review of Genomics
and Human Genetics 17:395–417. doi:10.1146/annurev-
genom-083115-022536.
Binkley, C. E., and B. Pilkington. 2023. The actionless agent:
An account of human-CAI relationships. The American
Journal of Bioethics 23 (5):25–7. doi:10.1080/15265161.
2023.2191035.
Blumenthal-Barby, J. 2023. An AI bill of rights:
Implications for health care AI and machine learning—A
bioethics lens. The American Journal of Bioethics 23 (1):
4–6. doi:10.1080/15265161.2022.2135875.
Cohen, I. G. 2019. Informed consent and medical artificial
intelligence: What to tell the patient? Geo. LJ 108:1425.
Cohen, I. G. 2023. What should ChatGPT mean for bioeth-
ics? The American Journal of Bioethics 23 (10):8–16. doi:
10.1080/15265161.2023.2233357.
Kaldjian, L. C., and B. C. Pilkington. 2021. Why truthful-
ness is the first of the virtues. The American Journal of
Bioethics 21 (5):36–8. doi:10.1080/15265161.2021.1906991.
Lee, T.-C., K. Staller, V. Botoman, M. P. Pathipati, S.
Varma, and B. Kuo. 2023. ChatGPT answers common
patient questions about colonoscopy. Gastroenterology
165 (2):509–11.e7. doi:10.1053/j.gastro.2023.04.033.
Mittelstadt, B. D., and L. Floridi. 2016. The ethics of big
data: Current and foreseeable issues in biomedical con-
texts. In The ethics of biomedical big data, eds. B. D.
Mittelstadt and L. Floridi, 445–80. Cham, Switzerland:
Springer International Publishing.
Richardson, J. P., C. Smith, S. Curtis, S. Watson, X. Zhu, B.
Barry, and R. R. Sharp. 2021. Patient apprehensions about
the use of artificial intelligence in healthcare. NPJ Digital
Medicine 4 (1):140. doi:10.1038/s41746-021-00509-1.
Schaefer, G. O., E. J. Emanuel, and A. Wertheimer. 2009.
The obligation to participate in biomedical research.
Jama 302 (1):67–72. doi:10.1001/jama.2009.931.