ORIGINAL RESEARCH

Artificial Intelligence Tool for Detection and Worklist

Prioritization Reduces Time to Diagnosis of Incidental
Pulmonary Embolism at CT
Laurens Topff, MD • Erik R. Ranschaert, MD, PhD • Annemarieke Bartels-Rutten, MD, PhD • Adina Negoita, MD •
Renee Menezes, PhD • Regina G. H. Beets-Tan, MD, PhD • Jacob J. Visser, MD, PhD
From the Department of Radiology (L.T., A.B.R., R.G.H.B.T.) and Department of Psychosocial Research and Epidemiology (R.M.), The Netherlands Cancer Institute, Ples-
manlaan 121, 1066 CX Amsterdam, the Netherlands; GROW School for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (L.T., R.G.H.B.T.);
Department of Radiology, St Nikolaus Hospital, Eupen, Belgium (E.R.R.); Faculty of Medicine and Health Sciences, Ghent University, Ghent, Belgium (E.R.R.); Department
of Radiology, La Moraleja Hospital, Madrid, Spain (A.N.); and Department of Radiology and Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam, Rot-
terdam, the Netherlands (J.J.V.). Received July 26, 2022; revision requested September 13; revision received January 13, 2023; accepted February 20. Address correspondence
to L.T. (email: l.topff@nki.nl).

Authors declared no funding for this work.

Conflicts of interest are listed at the end of this article.
See also commentary by Elicker in this issue.

Radiology: Cardiothoracic Imaging 2023; 5(2):e220163 • https://doi.org/10.1148/ryct.220163 • Content codes:

Purpose: To evaluate the diagnostic efficacy of artificial intelligence (AI) software in detecting incidental pulmonary embolism (IPE) at
CT and shorten the time to diagnosis with use of radiologist reading worklist prioritization.

Materials and Methods: In this study with historical controls and prospective evaluation, regulatory-cleared AI software was evaluated to
prioritize IPE on routine chest CT scans with intravenous contrast agent in adult oncology patients. Diagnostic accuracy metrics were
calculated, and temporal end points, including detection and notification times (DNTs), were assessed during three time periods (April
2019 to September 2020): routine workflow without AI, human triage without AI, and worklist prioritization with AI.

Results: In total, 11 736 CT scans in 6447 oncology patients (mean age, 63 years ± 12 [SD]; 3367 men) were included. Prevalence
of IPE was 1.3% (51 of 3837 scans), 1.4% (54 of 3920 scans), and 1.0% (38 of 3979 scans) for the respective time periods. The AI
software detected 131 true-positive, 12 false-negative, 31 false-positive, and 11 559 true-negative results, achieving 91.6% sensitiv-
ity, 99.7% specificity, 99.9% negative predictive value, and 80.9% positive predictive value. During prospective evaluation, AI-based
worklist prioritization reduced the median DNT for IPE-positive examinations to 87 minutes (vs routine workflow of 7714 minutes
and human triage of 4973 minutes). Radiologists’ missed rate of IPE was significantly reduced from 44.8% (47 of 105 scans) without
AI to 2.6% (one of 38 scans) when assisted by the AI tool (P < .001).

Conclusion: AI-assisted workflow prioritization of IPE on routine CT scans in oncology patients showed high diagnostic accuracy and
significantly shortened the time to diagnosis in a setting with a backlog of examinations.

Supplemental material is available for this article.

© RSNA, 2023

I ncidental pulmonary embolism (IPE) is a common co-

morbidity in oncology patients and is related to a high
risk of recurrent venous thromboembolism (1). IPEs can be
detected at routine contrast-enhanced CT of the chest that
is performed for indications other than thromboembolic
disease. The pooled frequency of IPE in oncology patients
has recently been reported as 3.4% (2). IPEs are associated
with poor outcomes, including disease progression and
reduced overall survival (3,4). While an IPE is clinically
unsuspected, it can be an urgent and life-threatening find-
ing, depending on the patient’s clinical condition and the
extent of the embolism (4). Timely diagnosis and proper
management of IPEs are therefore essential.
The workload of radiologists has increased substantially
during the past decades, mainly the result of higher vol-
umes and complexity of imaging examinations (5,6). It
is therefore not unusual for radiology departments to be
confronted with a backlog of unreported examinations,
especially when unexpected surges in imaging requests
This copy is for personal use only. To order printed copies, contact reprints@rsna.org
occur and in regions where a shortage of radiologists ex-
ists. Consequently, long report turnaround times (TATs)
pose a risk for delayed diagnosis of unsuspected critical
findings. Artificial intelligence (AI) applications that can
automatically detect critical findings at imaging can be
used to prioritize scans in the reading worklist of the radi-
ologist, with the aim of shortening time to diagnosis and
communication with the treating physician (7). AI-based
prioritization tools have been studied for use cases such as
intracranial hemorrhage at CT, acute pathologic abnor-
malities on chest radiographs, and pulmonary embolism
(PE) on dedicated CT pulmonary angiograms (CTPAs),
with varying results (8–11).
To the best of our knowledge, the clinical benefit of
a worklist triage tool for IPE has not been investigated.
Detection of IPEs can be challenging because filling de-
fects caused by thrombus need to be identified on CT
scans acquired in the venous phase with suboptimal con-
trast enhancement of the pulmonary arteries. The relative
AI Tool for Detection and Worklist Prioritization in Incidental Pulmonary Embolism at CT
Abbreviations
AI = artificial intelligence, CTPA = CT pulmonary angiogram,
DNT = detection and notification time, IPE = incidental PE, PE =
pulmonary embolism, TAT = turnaround time, TTR = time to read
Summary
Artificial intelligence–based prioritization of radiologist reading
worklists significantly reduced the time to diagnosis of incidental
pulmonary embolism at CT in patients with cancer.
Key Points
■ Artificial intelligence (AI) software for detecting incidental pul-
monary embolism (IPE) at chest CT in patients with cancer
showed high diagnostic accuracy in a large sample of 11 736 scans
(sensitivity, 91.6%; specificity, 99.7%; negative predictive value,
99.9%).
■ In a practice with a backlog of unreported examinations, AI-based
worklist prioritization reduced the median detection and notifica-
tion time of IPE in flagged scans from several days to 1.0 hour.
■ The missed rate of IPE was significantly reduced from 44.8% to
2.6% when radiologists were assisted by the AI tool (P < .001).
Keywords
CT, Computer Applications, Detection, Diagnosis, Embolism,
Thorax, Thrombosis
frequency of CT scans that are positive for IPE is also consider-
ably lower when compared with the relative frequency of posi-
tive CTPAs. Therefore, a clinically useful AI tool must perform
with high sensitivity and an acceptable false-positive rate.
The purpose of this study was to evaluate the diagnostic ef-
ficacy of AI software for the detection and prioritization of IPE
at chest CT in terms of diagnostic accuracy and its effect on time
to diagnosis in a real-world clinical environment.
Materials and Methods
Study Design and Patients
We performed a study to evaluate the diagnostic efficacy of
AI software to analyze IPE on chest CT scans in clinical prac-
tice by using data from historical controls and prospectively
analyzed data. The institutional review board approved this
study and waived the requirement for study-specific informed
consent. The AI software was provided by Aidoc Medical. The
authors are not affiliated with the industry and had control of
the data and information submitted.
The study was conducted in the radiology department of a
comprehensive cancer center during three periods of 15 weeks
each (Fig 1). All adult (≥18 years) patients with cancer who un-
derwent CT of the chest with intravenous contrast agent were
included. In total, 11 736 chest CT scans were assessed across
the three periods. During the first period (April to July 2019),
radiologists received no AI assistance and no special instructions
while reporting. During the second period (November 2019 to
February 2020), radiologists received no AI assistance but were
instructed to screen all newly acquired CT scans for the presence
of IPE. The task of human triage was performed by staff radiolo-
gists during working hours. Examinations with IPE were imme-
diately reported, and examinations without acute findings were
2
marked and returned to the reading worklist. During the third
period (June to September 2020), radiologists were assisted with
an AI-based worklist prioritization tool. The 15-week intervals
between periods allowed radiologists to get accustomed to the
new instructions and implementation of the AI software.
Imaging Data
Eligible chest CT scans acquired with intravenous contrast
agent were selected using the study description. CT scans were
acquired in the venous phase, 60–70 seconds after contrast me-
dium administration. Dedicated CTPAs were not selected and
were excluded (n = 3) after data collection. Sixty-four scans
were excluded because of incomplete or missing image files.
AI Software
Aidoc Medical provided commercially available AI-based im-
age analysis software that has been cleared by Conformité Eu-
ropéenne and the U.S. Food and Drug Administration. The
intended use of the software was specifically the prioritization
of IPE. The tool was connected with the picture archiving and
communication system to immediately analyze eligible imag-
ing studies after acquisition. After processing, a positive or neg-
ative result was transferred to the radiologist’s reading worklist
in the radiology information system. Positive examinations
were marked with a bright color in the worklist. Additionally,
a heatmap highlighting the suspected abnormality was sent to
the picture archiving and communication system.
Examinations that were not analyzed by the AI software due
to failed data orchestration or technical validation were excluded
(n = 378). Failure of AI analysis was possibly related to image
quality (eg, insufficient contrast enhancement, incompatible ac-
quisition parameters).
Reference Standard
The radiology report was used to categorize all CT scans for
the presence of IPE. Subsequently, two board-certified radi-
ologists (A.N., general radiologist with 3 years of experience;
and L.T., subspecialized in oncologic imaging with 5 years of
experience) independently reviewed CT scans that were cat-
egorized as positive by the radiology report or AI software. The
readers reviewed only imaging data, including heatmap results,
without any clinical information to label CT scans as positive,
negative, or inconclusive. In cases of disagreement between
readers, consensus was reached by a third dedicated chest ra-
diologist (A.B.R., with 8 years of experience). CT scans that
were categorized as negative by both the radiology report and
AI software were not additionally reviewed for the presence or
absence of IPE.
The location of the most proximal embolism was identified
for each positive CT scan. False-positive detections were classi-
fied as technical artifact, flow artifact, or abnormalities adjacent
to or within a pulmonary artery.
Diagnostic Accuracy
The performance of the AI software in the detection of IPE
was evaluated on both prospectively analyzed and historically
rcti.rsna.org ■ Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023

Figure 1: Flowchart shows study design and data selection per time period. AI = artificial intelligence, CTPA = CT pulmonary
angiogram, TP1 = time period 1, TP2 = time period 2, TP3 = time period 3.
collected data to increase the sample size. Sensitivity, specific-
ity, negative predictive value, positive predictive value, and ac-
curacy were calculated.
Time-related End Points
Timestamps were registered electronically to calculate the fol-
lowing time intervals: (a) time to process, defined as the time
interval between forwarding of the study to the AI software
and availability of the AI analysis result; (b) detection and no-
tification time (DNT), defined as the time interval between
availability of the study in the radiologist’s reading worklist and
the opening of the study by the radiologist; (c) TAT, defined
as the time interval between availability of the study in the
worklist and finalization of the report; and (d) time to read
(TTR), defined as the time interval between opening of the
study and finalization of the report. To avoid changes in report-
ing behavior, radiologists were not informed about the registra-
tion of time-related end points.
Statistical Analysis
The prevalence of IPE in different time periods was compared
using a χ2 test. CIs for diagnostic accuracy metrics, such as
sensitivity and specificity, were computed using a binomial
distribution. A Fisher exact test was used to investigate the
reduction in missed IPEs by radiologists with AI versus with-
out AI assistance.
Per time variable (DNT, TAT, and TTR), we were inter-
ested in comparing values between IPE-positive and -negative
CT scans for every time period. A Student t test was used
to compare each given log-transformed time variable be-
tween groups of positive and negative CT scans. CIs were
determined for these differences between positive and nega-
tive CT scans, per time period. Subsequently, we compared
CIs among the three time periods. Given that different time
periods involved different and independent examinations,
nonoverlapping CIs led to the conclusion that differences
were statistically distinct. In all cases, the null hypothesis was
Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023 ■ rcti.rsna.org
Topff et al
rejected for P < .05. Statistical analyses were performed by
a statistician (R.M.) using the R environment for statistical
computing (version 3.6.3; https://www.r-project.org) (12).
Results
Patient Demographics
A total of 11 736 chest CT scans from 6447 unique patients
(mean age, 63 years ± 12 [SD]; 3367 men [52.2%]) were in-
cluded. Patient demographics per time period are shown in Ta-
ble 1. Most imaging studies (11 333 of 11 736 [96.6%]) were
performed in outpatients.
Imaging Data
Chest CT scans were acquired with seven models from three
manufacturers: Toshiba (5836 scans of 11 736 [49.7%]),
Siemens (5140 of 11 736 [43.8%]), and Philips (760 of
11 736 [6.5%]) (Table S1). CT series reconstructed with a
soft-tissue kernel were obtained with a section thickness of
1.0–2.0 mm. In the majority of CT scans (9923 of 11 736
[84.6%]), the chest was imaged in combination with the
abdomen and/or neck.
Diagnostic Accuracy
The prevalence of IPE in the sample was 1.2% (143 of 11 736
scans). The prevalence of IPE was similar across the three
time periods (χ2 test, P = .17). Overall, the AI software re-
turned 131 true-positive, 12 false-negative, and 31 false-pos-
itive results (Table 2). Three were inconclusive, and the other
11 559 examinations were categorized as negative by both the
radiology report and AI software. The AI software showed
a sensitivity of 91.6% (131 of 143 scans; 95% CI: 86.7,
95.8), specificity of 99.7% (11 559 of 11 590 scans; 95% CI:
99.6, 99.8), negative predictive value of 99.9% (11 559 of
11 571 scans; 95% CI: 99.8, 99.9), positive predictive value
of 80.9% (131 of 162 scans; 95% CI: 74.7, 86.4), and accu-
racy of 99.6% (11 690 of 11 733 scans; 95% CI: 99.5, 99.7).
3
AI Tool for Detection and Worklist Prioritization in Incidental Pulmonary Embolism at CT

Three examinations were marked as

Table 1: Patient Demographics per Time Period
inconclusive by the review panel. The
diagnostic accuracy per time period is Routine Workflow Human Triage Worklist Prioritization
available in Tables S2–S5. Variable without AI (TP1) without AI (TP2) with AI (TP3)
Table 3 lists the location of the most No. of CT scans 3837 3920 3979
proximal filling defect in pulmonary No. of unique patients 3309 3387 3441
arteries of IPE-positive examinations. Age (y)* 62 ± 12 62 ± 12 63 ± 12
In total, 54 of 143 clots (37.8%) were
Sex
located in the main or lobar pulmonary
M 1725 (52.1) 1773 (52.3) 1787 (51.9)
arteries. Fifty-two of these (96.3%) were
detected by the AI software, including F 1584 (47.9) 1614 (47.7) 1654 (48.1)
all main pulmonary clots (n = 9). Two No. of outpatient scans 3712 (96.7) 3786 (96.6) 3835 (96.4)
examples of true-positive detections are No. of inpatient scans 125 (3.3) 134 (3.4) 144 (3.6)
shown in Figure 2. Note.—Unless otherwise noted, data are numbers, with percentages in parentheses. AI =
False-negative findings were lo- artificial intelligence, TP1 = time period 1, TP2 = time period 2, TP3 = time period 3.
cated in the segmental or subsegmen- * Data are means ± SDs.
tal arteries in 10 of 12 examinations
(83.3%). The other two false-negative
findings (16.7%) were located in lobar
Table 2: Diagnostic Accuracy in Detection of IPE by the AI
arteries; however, the clots were small and most likely Software Alone
chronic (Fig 3).
False-positive findings (n = 31) were consistent with Variable IPE Present IPE Absent Inconclusive Total
known mimickers of PE (Table 4). Most false-positive AI positive 131 31 3 165
findings were categorized as flow artifacts (13 of 31 AI negative 12 11 559 0 11 571
[41.9%]) (Fig 4A, 4B). Technical artifacts (seven of 31 Total 143 11 590 3 11 736
findings [22.6%]) were predominantly caused by respira-
Note.—Data are numbers of scans. AI = artificial intelligence, IPE =
tory motion. In 11 of 31 examinations (35.5%), extravas-
incidental pulmonary embolism.
cular abnormalities (eg, lymphadenopathy [Fig 4C, 4D])
or intravascular abnormalities (eg, stump thrombus) led
to false-positive detections. The number of false-negative
and false-positive findings per time period is available in Table 3: Location of the Most Proximal Filling Defect in Pul-
Tables S2–S4. monary Arteries on Positive Incidental Pulmonary Embolism
In 44.8% (47 of 105) of IPE-positive CT scans from the Scans
first two time periods (historically collected data), the find- No. of CT Scans
ing was missed in the radiology report but correctly identi-
fied by the AI software. In comparison, in the third time pe- True-Positive False-Negative Total
riod when radiologists were assisted by the AI software, only Artery Location Scans (n = 131) Scans (n = 12) (n = 143)
one of 38 studies positive for IPE (2.6%) was missed by the Main 9 (6.9) 0 (0) 9 (6.3)
radiologists. This resulted in a 94% reduction (44.8% vs Lobar 43 (32.8) 2 (16.7) 45 (31.5)
2.6%) in missed IPE with use of AI assistance (Fisher exact Segmental 62 (47.3) 6 (50) 68 (47.6)
test, P < .001). Categorization of IPEs missed by the radi- Subsegmental 17 (13.0) 4 (33.3) 21 (14.7)
ologists according to the location of the most proximal clot
Note.—Data in parentheses are percentages.
was as follows: lobar in eight of 48 examinations (16.7%),
segmental in 28 of 48 examinations (58.3%), and subseg-
mental in 12 of 48 examinations (25%). No main IPE was
missed by the radiologists. Time-related End Points
The study design excluded CT scans that were not success- The AI analysis of CT scans was performed without human
fully analyzed by the AI software due to failed data orchestration intervention. The median processing time of CT scans by the
or technical validation. The scans from the first two time periods AI software was 3 minutes, with a maximum of 20 minutes. In
in which the AI analysis failed (63 of 7820 [0.8%]) were all re- 99.5% (3958 of 3979) of studies, the AI result was available to
ported as negative for IPE. However, during the third time pe- the radiologist at the time of opening the study.
riod, the AI software was deployed in the clinical environment, The median DNTs for IPE-positive examinations in all
and scans were analyzed in real time. As a result, a larger propor- patients were 7714, 4973, and 87 minutes for the respective
tion of CT scans (315 of 4294 [7.3%]) was not processed and time periods of routine workflow without AI, human triage
thus excluded. Nevertheless, two of the 315 excluded CT scans without AI, and worklist prioritization with AI. The median
were reported as positive for IPE, with a segmental and subseg- DNTs for IPE-positive examinations in outpatients only were
mental location, respectively. 8950, 5454, and 80 minutes for the respective time periods.

4 rcti.rsna.org ■ Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023

 Topff et al

Figure 2: True-positive detection of incidental pulmonary embolism (PE) by the artificial intelligence (AI) software. (A, B) Im-
ages in a 68-year-old woman who underwent routine CT with intravenous contrast agent for outpatient follow-up of melanoma.
(A) Axial CT image shows a large filling defect straddling the bifurcation of the pulmonary trunk (arrow) and extending into both
pulmonary arteries, compatible with an incidental saddle PE. (B) Corresponding AI heatmap highlights the detected abnormality
(red), thereby prioritizing the case in the radiologists’ worklist. (C, D) Images in a 58-year-old woman with a history of rectal cancer
undergoing outpatient follow-up. (C) Axial restaging CT image with intravenous contrast agent shows a small incidental subsegmen-
tal PE in the right lower lung lobe (arrow). (D) Corresponding AI heatmap enables the radiologist to localize the finding (red).

Figure 3: False-negative findings of two chronic lobar incidental pulmonary embolisms (PEs) that were not detected by the
artificial intelligence (AI) software. (A) Axial CT image with intravenous contrast agent in a 70-year-old man (an outpatient) with
urothelial carcinoma shows a small incidental PE (IPE) located against the vessel wall in the right pulmonary artery bifurcation (ar-
row), compatible with a small chronic IPE. (B) Contrast-enhanced coronal CT image in a 62-year-old man (an inpatient) with lung
cancer shows a small eccentric filling defect in the pulmonary artery of the left lower lobe (arrow). These findings were not detected
by the AI software.

When only considering true-positive examinations flagged by
the AI software in the third time period (n = 34), the median
DNT was 62 minutes. The DNT of 29 of the 34 true-positive
examinations (85.3%) was less than 6 hours. In comparison,
the DNT of the four nonprioritized false-negative examina-
tions in the third time period ranged from 1280 minutes to
12 684 minutes. Figure 5 shows the DNTs of all IPE-negative
versus -positive studies for each time period. CIs were calcu-
lated for the time differences between negative and positive
studies in each time period (Fig 6). Given that the CIs did not
overlap between the time period with AI assistance and those
without AI assistance, differences were statistically significant
between the time periods. In contrast, we found no evidence
of a difference between routine workflow without AI and hu-
man triage.
Report TAT showed similar results as DNT. The median
TATs for IPE-positive examinations were 7772, 4983, and 148
minutes for the three respective time periods. When only consid-
ering true-positive examinations flagged by the AI software, the
median TAT was 91 minutes.

Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023 ■ rcti.rsna.org 5

AI Tool for Detection and Worklist Prioritization in Incidental Pulmonary Embolism at CT

The median TTR for all examinations was 16 minutes. The
median TTR for positive examinations was 21 minutes. We
found no evidence of a difference in TTR between time periods.
Discussion
We evaluated the clinical value of AI software for the analysis of
IPE on a large sample of chest CT scans (n = 11 736) in oncol-
ogy patients. The AI tool accurately detected IPE on chest CT
scans with intravenous contrast agent, with a high sensitivity
of 91.6% (131 of 143 scans), specificity of 99.7% (11 559 of
11 590 scans), and negative predictive value of 99.9% (11 559
of 11 571 scans). False-negative classification occurred in 12 of
143 examinations (8.4%) but was limited to segmental, sub-
segmental, and small chronic lobar clots. No IPE in the main
pulmonary arteries was missed by the software. The number
of false-positive detections by the software, 31 of 165 flagged
examinations (18.8%), can be considered acceptable because
radiologists could easily identify false-positive findings by us-
ing the heatmap. In total, only 0.3% of all analyzed examina-
tions (31 of 11 736) were falsely positive. The impact of false-
positive alerts on radiologist workflow was therefore limited.
During prospective evaluation, the AI software was de-
ployed in a clinical environment with a backlog of unreported
examinations to prioritize IPE on routinely acquired chest CT
scans, mostly obtained in outpatients with known primary ma-
lignancy for posttreatment follow-up. The AI-based worklist
prioritization resulted in a significantly reduced median DNT
and TAT for flagged scans with IPE, from several days to 1.0 and
1.5 hours, respectively. In contrast, unassisted triage of CT scans
by radiologists did not have a significant effect on the reduction
of DNT or TAT when compared with the routine workflow.
This is likely a result of the time-consuming nature of this task,
contributing to low yield.
To the best of our knowledge, no other published study has
investigated the diagnostic performance of AI software for the
detection and prioritization of IPE. Previous studies have as-
sessed the diagnostic accuracy of deep learning algorithms in the
detection of PE on dedicated CTPAs (13–18). For this task, sen-
sitivities and specificities ranged from 73% to 96% and 77% to
96%, respectively. Although our study focused on the detection
Table 4: Causes of False-Positive Detections by the Arti-
ficial Intelligence Software
No. of CT
Cause Scans (n = 31)
Technical artifact 7 (22.6)
Flow artifact 13 (41.9)
Abnormality adjacent to a pulmonary artery 9 (29)
Abnormality within a pulmonary artery 2 (6.5)
Note.—Data in parentheses are percentages.

Figure 4: False-positive detections by the artificial intelligence (AI) software. (A, B) Images in a 59-year-old woman (an out-
patient) with melanoma who underwent CT with intravenous contrast agent. (A) Axial CT image shows slightly decreased contrast
opacification in a segmental pulmonary artery in the right lower lobe (arrow). This finding was compatible with a flow artifact without
any clinical significance. (B) Corresponding AI software heatmap misclassified the finding as a possible incidental pulmonary em-
bolism (IPE), as highlighted in red. (C, D) Images in a 36-year-old woman (an outpatient) with cervical cancer who underwent CT
with intravenous contrast agent. (C) Axial CT image shows hilar and mediastinal lymphadenopathy. An enlarged right hilar lymph
node shows impression on the right pulmonary arteries (arrow). The finding was misclassified by the AI software as IPE, as shown on
(D) the corresponding axial heatmap in red.

6 rcti.rsna.org ■ Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023

 Topff et al

Figure 5: Box plot shows detection and notification times (DNTs) of incidental pulmonary embolism (IPE)–negative versus IPE-positive CT scans
per time period: routine workflow without artificial intelligence (AI), human triage without AI, and worklist prioritization with AI. DNT was markedly
reduced for positive CT scans during the third time period with AI assistance (median DNT, 87 minutes vs routine workflow DNT of 7714 minutes [5
days]). The horizontal line in each box plot indicates the median, and the box corresponds to the IQR. The whiskers indicate minimum and maximum
values in the data. Circles represent outliers. TP1 = time period 1, TP2 = time period 2, TP3 = time period 3.

Figure 6: Graph shows 95% CIs of detection and notification time (DNT) differences between incidental pulmonary
embolism–positive and –negative CT scans per time period: routine workflow without artificial intelligence (AI) (TP1), human
triage without AI (TP2), and worklist prioritization with AI (TP3). The DNT difference was largest for the third time period.
Given that the CIs of the third time period versus the first and second periods did not overlap, differences were significant
between these periods.

of IPE on venous CT scans, which can be considered more chal-
lenging, our sensitivity and specificity were comparable with or
higher than studies identifying PE on CTPAs. Furthermore, ret-
rospective studies on diagnostic accuracy might not determine
the real clinical impact on patient care (18). Schmuelling et al
(19) evaluated the clinical implementation of AI software for
prioritization of positive CTPAs in the emergency setting; the
authors showed good diagnostic accuracy, with 79.6% sensitivity
and 95.0% specificity. However, there was no significant reduc-
tion in report communication times. This is likely related to the
overall short TATs of examinations in an emergency department.
Tools to prioritize the reading worklist would provide the most
benefit in clinical settings with a high workload and a backlog of
unreported examinations, as in our situation. The shorter time
to detection of IPE in our study has limited generalizability to
practices with short report TATs.

Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023 ■ rcti.rsna.org 7

AI Tool for Detection and Worklist Prioritization in Incidental Pulmonary Embolism at CT
PE is one of the diagnoses that is most commonly missed or
delayed by physicians (20). Detection of IPE on routine contrast-
enhanced chest CT scans can be especially challenging when the
IPE is small and isolated. When analyzing historical CT data, we
found that 44.8% of IPEs were missed by radiologists. Low detec-
tion rates of IPE by radiologists have also been reported in other
studies (21,22). Wildman-Tobriner et al (23) applied a different
AI algorithm to retrospectively analyze 11 913 CT examinations
for undiagnosed IPE and found 49 missed IPEs (0.41%), lead-
ing to a missed rate of 38% (49 of 128 IPEs). In our study, we
prospectively evaluated the effect of AI assistance on missed IPEs.
The number of missed IPEs was reduced to one scan of 38 (2.6%),
thereby demonstrating that AI software can assist radiologists to
significantly improve the detection rate of IPE.
The clinical relevance and proper management of IPE remain
a subject of debate. It is well known that venous thromboembo-
lism in oncology patients is associated with high morbidity and
mortality (24). Observational studies suggest that the prognosis
of IPE is similar to that of symptomatic PE with regard to the
risk of recurrence and mortality (25). Consequently, treatment
guidelines for IPE are similar to those in symptomatic PE (26).
Radiologic findings, such as thrombus load and central location,
have been associated with adverse clinical outcomes in acute PE
(27). These findings can also help determine IPE severity (4).
In our study, 37.8% (54 of 143) of IPE-positive scans showed
emboli in the main or lobar pulmonary arteries. Therefore, the
benefit of AI-based worklist prioritization for timely assessment
and treatment is most evident in these patients. The majority of
physicians also treat smaller, more distal incidental emboli in pa-
tients with cancer (28). In our study, most IPEs that were missed
by radiologists but detected by the AI software were segmental
(28 of 48 [58.3%]) or subsegmental (12 of 48 [25%]). We must
therefore consider the risk of overdiagnosis, specifically of iso-
lated subsegmental IPE, which if left untreated would cause no
more harm than treatment complications (29). To our knowl-
edge, no randomized controlled trials have assessed the effective-
ness of anticoagulation therapy in patients with subsegmental
PE (30). However, recent studies support the use of anticoagula-
tion therapy for subsegmental PE in oncology patients (1,31).
Further studies are needed to assess the relevance of diagnosing
and treating small incidental emboli.
The intended use of the investigated AI software is limited to
workflow triage and not diagnostics (32). Automated worklist
prioritization can assist radiologists, who remain responsible for
verifying flagged examinations and must be aware of possible
false-negative findings. Our study showed that a considerable
number of scans (315 of 4294 [7.3%]) were not analyzed by
the deployed AI software due to issues with data retrieval and
technical validation, resulting in delayed diagnosis of IPE in
these patients. This demonstrates the importance of monitoring
and improving the yield of scans analyzed by AI software after
deployment.
The study had limitations. First, the number of false-neg-
ative IPEs may be underestimated because examinations that
were categorized as negative by both the radiology report and
AI software were not reviewed—this was not feasible due to
the high number of included scans. Second, the study sample
8
consisted solely of oncology patients. The frequency of IPE is
likely lower in the general patient population, which might im-
pact the clinical relevance of the AI software. Third, the study
design was not randomized. Although patient volumes in all
time periods were similar and unaffected by the COVID-19
pandemic, TATs in the radiology department can be affected
by many factors, such as staffing levels. To account for varia-
tions between periods, we calculated the time differences be-
tween positive and negative IPE examinations within each
period separately and compared CIs of the difference among
the periods. Fourth, statistical analysis of the time variables
assumed independence of examinations; however, the study
included multiple examinations per patient. Fifth, the study
focused on diagnostic efficacy; we did not evaluate the value on
patient outcomes and cost-effectiveness. Future studies should
investigate the effect of early diagnosis of IPE on morbidity
and mortality.
In conclusion, we demonstrated that commercially available
AI software had high diagnostic accuracy in the detection of IPE
on chest CT scans in patients with cancer and was effective in
significantly reducing the time to diagnosis of positive examina-
tions compared with the routine workflow in a setting with a
backlog of unreported scans.
Author contributions: Guarantor of integrity of entire study, L.T.; study concepts/
study design or data acquisition or data analysis/interpretation, all authors; manu-
script drafting or manuscript revision for important intellectual content, all authors;
approval of final version of submitted manuscript, all authors; agrees to ensure any
questions related to the work are appropriately resolved, all authors; literature re-
search, L.T., E.R.R., J.J.V.; clinical studies, L.T., A.B.R., A.N., J.J.V.; statistical
analysis, L.T., R.M., J.J.V.; and manuscript editing, L.T., E.R.R., A.B.R., R.M.,
R.G.H.B.T., J.J.V.
Data sharing: Data generated or analyzed during the study are available from the
corresponding author by request.
Disclosures of conflicts of interest: L.T. No relevant relationships. E.R.R. No
relevant relationships. A.B.R. No relevant relationships. A.N. No relevant relation-
ships. R.M. No relevant relationships. R.G.H.B.T. No relevant relationships. J.J.V.
Grant to institution from Qure.ai; consulting fees from Tegus; payment to institu-
tion for lectures from Roche; travel grant from Qure.ai; participation on a data
safety monitoring board or advisory board from Contextflow, Noaber Foundation,
and NLC Ventures; leadership or fiduciary role on the steering committee of the
PINPOINT Project (payment to institution from AstraZeneca) and RSNA Com-
mon Data Elements Steering Committee (unpaid); phantom shares in Contextflow
and Quibim.
References
1. Kraaijpoel N, Bleker SM, Meyer G, et al. Treatment and long-term clinical
outcomes of incidental pulmonary embolism in patients with cancer: an
international prospective cohort study. J Clin Oncol 2019;37(20):1713–1720.
2. Meyer HJ, Wienke A, Surov A. Incidental pulmonary embolism in onco-
logic patients—a systematic review and meta-analysis. Support Care Cancer
2021;29(3):1293–1302.
3. Lawson P, Raskin S, Soffer S, et al. Incidental pulmonary embolism in CT
scans of oncological patients with metastatic disease undergoing clinical trials:
frequency and linkage with onset of disease progression (PE-PD association).
Br J Radiol 2020;93(1115):20200591.
4. Qdaisat A, Kamal M, Al-Breiki A, et al. Clinical characteristics, management,
and outcome of incidental pulmonary embolism in cancer patients. Blood
Adv 2020;4(8):1606–1614.
5. Kwee TC, Kwee RM. Workload of diagnostic radiologists in the foreseeable
future based on recent scientific advances: growth expectations and role of
artificial intelligence. Insights Imaging 2021;12(1):88.
6. McDonald RJ, Schwartz KM, Eckel LJ, et al. The effects of changes in
utilization and technological advancements of cross-sectional imaging on
radiologist workload. Acad Radiol 2015;22(9):1191–1198.
rcti.rsna.org ■ Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023

7. Ranschaert E, Topff L, Pianykh O. Optimization of radiology workflow
with artificial intelligence. Radiol Clin North Am 2021;59(6):955–966.
8. O’Neill TJ, Xi Y, Stehel E, et al. Active reprioritization of the reading worklist
using artificial intelligence has a beneficial effect on the turnaround time for
interpretation of head CT with intracranial hemorrhage. Radiol Artif Intell
2020;3(2):e200024.
9. Baltruschat I, Steinmeister L, Nickisch H, et al. Smart chest x-ray worklist
prioritization using artificial intelligence: a clinical workflow simulation. Eur
Radiol 2021;31(6):3837–3845.
10. Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G.
Automated triaging of adult chest radiographs with deep artificial neural
networks. Radiology 2019;291(1):196–202.
11. Arbabshirani MR, Fornwalt BK, Mongelluzzo GJ, et al. Advanced machine
learning in action: identification of intracranial hemorrhage on computed
tomography scans of the head with clinical workflow integration. NPJ Digit
Med 2018;1(1):9.
12. R Core Team. R: a language and environment for statistical computing.
Vienna, Austria: R Foundation for Statistical Computing; 2020. https://
www.R-project.org.
13. Huang SC, Kothari T, Banerjee I, et al. PENet—a scalable deep-learning
model for automated diagnosis of pulmonary embolism using volumetric
CT imaging. NPJ Digit Med 2020;3(1):61. [Published correction appears
in NPJ Digit Med 2020;3:102.]
14. Weikert T, Winkel DJ, Bremerich J, et al. Automated detection of pulmonary
embolism in CT pulmonary angiograms using an AI-powered algorithm.
Eur Radiol 2020;30(12):6545–6553.
15. Buls N, Watté N, Nieboer K, Ilsen B, de Mey J. Performance of an artificial
intelligence tool with real-time clinical workflow integration—detection of
intracranial hemorrhage and pulmonary embolism. Phys Med 2021;83:154–
160.
16. Liu W, Liu M, Guo X, et al. Evaluation of acute pulmonary embolism and
clot burden on CTPA with deep learning. Eur Radiol 2020;30(6):3567–3575.
17. Cheikh AB, Gorincour G, Nivet H, et al. How artificial intelligence improves
radiological interpretation in suspected pulmonary embolism. Eur Radiol
2022;32(9):5831–5842.
18. Soffer S, Klang E, Shimon O, et al. Deep learning for pulmonary embolism
detection on computed tomography pulmonary angiogram: a systematic
review and meta-analysis. Sci Rep 2021;11(1):15814.
19. Schmuelling L, Franzeck FC, Nickel CH, et al. Deep learning-based auto-
mated detection of pulmonary embolism on CT pulmonary angiograms: no
Radiology: Cardiothoracic Imaging Volume 5: Number 2—2023 ■ rcti.rsna.org
Topff et al
significant effects on report communication times and patient turnaround
in the emergency department nine months after technical implementation.
Eur J Radiol 2021;141:109816.
20. Schiff GD, Hasan O, Kim S, et al. Diagnostic error in medicine: analysis of
583 physician-reported errors. Arch Intern Med 2009;169(20):1881–1887.
21. Gladish GW, Choe DH, Marom EM, Sabloff BS, Broemeling LD, Munden
RF. Incidental pulmonary emboli in oncology patients: prevalence, CT
evaluation, and natural history. Radiology 2006;240(1):246–255.
22. Deniz MA, Deniz ZT, Adin ME, et al. Detection of incidental pulmonary
embolism with multi-slice computed tomography in cancer patients. Clin
Imaging 2017;41:106–111.
23. Wildman-Tobriner B, Ngo L, Mammarappallil JG, Konkel B, Johnson JM,
Bashir MR. Missed incidental pulmonary embolism: harnessing artificial intel-
ligence to assess prevalence and improve quality improvement opportunities.
J Am Coll Radiol 2021;18(7):992–999.
24. Schmaier AA, Ambesh P, Campia U. Venous thromboembolism and cancer.
Curr Cardiol Rep 2018;20(10):89.
25. Klok FA, Huisman MV. Management of incidental pulmonary embolism.
Eur Respir J 2017;49(6):1700275.
26. Mulder FI, Di Nisio M, Ay C, et al. Clinical implications of incidental venous
thromboembolism in cancer patients. Eur Respir J 2020;55(2):1901697.
27. Meinel FG, Nance JW Jr, Schoepf UJ, et al. Predictive value of computed
tomography in acute pulmonary embolism: systematic review and meta-
analysis. Am J Med 2015;128(7):747–59.e2.
28. den Exter PL, van Roosmalen MJG, van den Hoven P, et al. Physicians’
management approach to an incidental pulmonary embolism: an international
survey. J Thromb Haemost 2013;11(1):208–213.
29. Dobler CC. Overdiagnosis of pulmonary embolism: definition, causes and
implications. Breathe (Sheff) 2019;15(1):46–53.
30. Yoo HH, Nunes-Nogueira VS, Fortes Villas Boas PJ. Anticoagulant treat-
ment for subsegmental pulmonary embolism. Cochrane Database Syst Rev
2020;2(2):CD010222.
31. Yan M, Kieser R, Wu CC, Qiao W, Rojas-Hernandez CM. Clinical factors
and outcomes of subsegmental pulmonary embolism in cancer patients.
Blood Adv 2021;5(4):1050–1058.
32. U.S. Food & Drug Administration. K201020. https://www.accessdata.fda.
gov/cdrh_docs/pdf20/K201020.pdf. Published August 26, 2020. Accessed
May 6, 2022.
9