Professional Documents
Culture Documents
a r t i c l e i n f o a b s t r a c t
Article history: Background: Lung ultrasound can evaluate for pulmonary edema, but data suggest moderate inter-rater reliability
Received 13 April 2023 among users. Artificial intelligence (AI) has been proposed as a model to increase the accuracy of B line interpre-
Received in revised form 13 May 2023 tation. Early data suggest a benefit among more novice users, but data are limited among average residency-
Accepted 20 May 2023 trained physicians. The objective of this study was to compare the accuracy of AI versus real-time physician
assessment for B lines.
Keywords:
Methods: This was a prospective, observational study of adult Emergency Department patients presenting with
Artificial intelligence
Ultrasound
suspected pulmonary edema. We excluded patients with active COVID-19 or interstitial lung disease. A physician
Lung performed thoracic ultrasound using the 12-zone technique. The physician recorded a video clip in each zone and
Pulmonary edema provided an interpretation of positive (≥3 B lines or a wide, dense B line) or negative (<3 B lines and the absence
of a wide, dense B line) for pulmonary edema based upon the real-time assessment. A research assistant then uti-
lized the AI program to analyze the same saved clip to determine if it was positive versus negative for pulmonary
edema. The physician sonographer was blinded to this assessment. The video clips were then reviewed indepen-
dently by two expert physician sonographers (ultrasound leaders with >10,000 prior ultrasound image reviews)
who were blinded to the AI and initial determinations. The experts reviewed all discordant values and reached
consensus on whether the field (i.e., the area of lung between two adjacent ribs) was positive or negative
using the same criteria as defined above, which served as the gold standard.
Results: 71 patients were included in the study (56.3% female; mean BMI: 33.4 [95% CI 30.6–36.2]), with 88.3%
(752/852) of lung fields being of adequate quality for assessment. Overall, 36.1% of lung fields were positive
for pulmonary edema. The physician was 96.7% (95% CI 93.8%–98.5%) sensitive and 79.1% (95% CI
75.1%–82.6%) specific. The AI software was 95.6% (95% CI 92.4%–97.7%) sensitive and 64.1% (95% CI
59.8%–68.5%) specific.
Conclusion: Both the physician and AI software were highly sensitive, though the physician was more specific.
Future research should identify which factors are associated with increased diagnostic accuracy.
© 2023 Elsevier Inc. All rights reserved.
https://doi.org/10.1016/j.ajem.2023.05.029
0735-6757/© 2023 Elsevier Inc. All rights reserved.
Descargado para Anonymous User (n/a) en Cayetano Heredia Pervuvian University de ClinicalKey.es por Elsevier en septiembre 07, 2023. Para
uso personal exclusivamente. No se permiten otros usos sin autorización. Copyright ©2023. Elsevier Inc. Todos los derechos reservados.
M. Gottlieb, D. Patel, M. Viars et al. American Journal of Emergency Medicine 70 (2023) 109–112
hyperechoic reverberation artifacts arising from the pleural line that if they had symptomatic COVID-19 infection or interstitial lung disease,
extend to the bottom of the ultrasound screen and move synchronously were clinically unstable, declined to participate, had been previously
with lung sliding [7]. The presence of three or more B lines in the space enrolled in the study, or if the treating clinician was not concerned for
between two contiguous ribs are suggestive of pulmonary edema [7]. pulmonary edema. Prior to beginning the study, research assistants col-
Identification of these findings can help improve diagnostic accuracy lected the patient's age, gender, height and weight (to calculate BMI),
overall and shorten the time to diagnosis by assessing for these while and relevant past medical history (defined as hypertension, heart failure,
the clinician is at the patient's bedside [6]. This could also be useful in end-stage renal disease, cirrhosis, and asthma/COPD).
resource-limited settings and to assess responses to therapeutic All ultrasound examinations were performed by a single physician
interventions [8]. sonographer, who was a new ultrasound fellow that had previously
Importantly, POCUS is a user-dependent skill that requires completed an Emergency Medicine residency. The physician received
structured training and assessment [9,10]. Artificial intelligence (AI) a one-hour study-specific training session followed by 25 proctored
has been increasingly utilized to automate assessments and improve lung ultrasound examinations by an expert sonographer (fellowship-
diagnostic accuracy. Prior studies focused on AI for pulmonary edema trained physician with Advanced Emergency Medicine Ultrasound
have primarily been performed by non-physicians or early learners Focused Practice Designation certification). The physician did not
[11,12], while others have been limited by small sample sizes [13]. receive any additional training and the study was initiated in early fel-
There are limited data directly comparing the diagnostic accuracy of lowship, so as to best reflect the average residency-trained physician
AI with real-time assessment by trained physicians. This aspect is criti- who would use thoracic ultrasound in practice [15].
cal to better understand the potential role in comparison with typical The ultrasound examinations were performed using a C1–5 trans-
use of B line assessment in practice and how AI would compare. ducer (Venue; GE Healthcare) in the lung preset at a depth of 18 cm
The primary aim of this study was to directly compare the sensitivity [16]. Images were obtained in the sagittal plane between two adjacent
and specificity of lung ultrasound with AI versus a trained physician rib spaces. A 12-zone technique (six views per side) was utilized. This
sonographer performing this in real-time. As a secondary objective, included the following views: Left-Anterior-Inferior, Left-Anterior-
we sought to compare the sensitivity and specificity among patients Superior, Left-Lateral-Inferior, Left-Lateral-Superior, Left-Posterior-
with a low versus high body mass index (BMI). Inferior, Left-Posterior-Superior, Right-Anterior-Inferior, Right-
Anterior-Superior, Right-Lateral-Inferior, Right-Lateral-Superior,
2. Methods Right-Posterior-Inferior, and Right-Posterior-Superior. The physician
sonographer recorded a single six-second video clip from each region.
This was a prospective, observational study comparing the diagnostic The physician sonographer reported a real-time assessment of whether
accuracy of AI with real-time physician assessment of B lines among pa- the lung field was positive (≥3 B lines or a wide, dense B line) or negative
tients with suspected pulmonary edema. We followed the Strengthening (<3 B lines and the absence of a wide, dense B line) for pulmonary
the Reporting of Observational Studies in Epidemiology guidelines [14]. edema in each region. The research assistant then utilized the AI soft-
The study was conducted at Rush University Medical Center, a 70,000 ware (Auto B-Lines; GE Healthcare) to retrospectively analyze the
visit per year ED located in Chicago, Illinois. Rush University Medical same saved clip to determine if it was positive versus negative for pul-
Center is a quaternary care hospital with a three-year Emergency Medi- monary edema (Fig. 1). We used a similar protocol for the AI software,
cine residency program and a Clinical Ultrasound fellowship. This study wherein we defined positive as ≥3 B lines or a large, dense B line and neg-
was approved by the Rush University Medical Center institutional ative as <3 B lines and the absence of a large, dense B line. The physician
review board and all patients signed informed consent. sonographer was blinded to the AI assessment. Lung regions were
Adult (age ≥ 18 years) ED patients with suspected cardiogenic excluded if they were unable to be adequately visualized (e.g., cardiac
pulmonary edema were identified by research assistants via a conve- interference, obscuring structure).
nience sample of when the physician sonographer was present. The phy- The clips were then reviewed by two expert sonographers (ultra-
sician sonographer was present three days per week from 8:00–16:00. sound leaders with >10,000 prior ultrasound image reviews) who
Patients were eligible for inclusion if they were willing to participate in were blinded to both the initial assessment and AI findings, as well as
the study, spoke English as their primary language, and the treating clini- any clinical, laboratory, or alternate imaging findings from the patient.
cian had a clinical concern for pulmonary edema. Patients were excluded The experts began by reviewing images together to reach consensus
Fig. 1. Artificial intelligence for the sonographic assessment of Pulmonary Edema. A, Positive Lung field; B, Negative Lung field.
110
Descargado para Anonymous User (n/a) en Cayetano Heredia Pervuvian University de ClinicalKey.es por Elsevier en septiembre 07, 2023. Para
uso personal exclusivamente. No se permiten otros usos sin autorización. Copyright ©2023. Elsevier Inc. Todos los derechos reservados.
M. Gottlieb, D. Patel, M. Viars et al. American Journal of Emergency Medicine 70 (2023) 109–112
Table 2
Diagnostic accuracy of real-time physician assessment and artificial intelligence for pulmonary edema using ultrasound.
Group Accuracy (95% CI) Sensitivity (95% CI) Specificity (95% CI) LR + (95% CI) LR-(95% CI)
Physician 85.4% (82.6%–87.8%) 96.7% (93.8%–98.5%) 79.1% (75.1%–82.6%) 4.61 (3.87–5.49) 0.04 (0.02–0.08)
Artificial Intelligence 75.4% (72.2%–78.4%) 95.6% (92.4%–97.7%) 64.1% (59.8%–68.5%) 2.66 (2.36–3.01) 0.07 (0.04–0.12)
CI, confidence interval; LR+, positive likelihood ratio; LR-, negative likelihood ratio.
111
Descargado para Anonymous User (n/a) en Cayetano Heredia Pervuvian University de ClinicalKey.es por Elsevier en septiembre 07, 2023. Para
uso personal exclusivamente. No se permiten otros usos sin autorización. Copyright ©2023. Elsevier Inc. Todos los derechos reservados.
M. Gottlieb, D. Patel, M. Viars et al. American Journal of Emergency Medicine 70 (2023) 109–112
user, they did receive additional training in other POCUS applications [5] Martindale JL, Wakai A, Collins SP, et al. Diagnosing acute heart failure in the emer-
gency department: a systematic review and Meta-analysis. Acad Emerg Med. 2016;
during the portion of their fellowship dedicated to enrollment, 23(3):223–42. https://doi.org/10.1111/acem.12878.
which may have led to greater overall ultrasound skill than the [6] Chiu L, Jairam MP, Chow R, et al. Meta-analysis of point-of-care lung ultrasonogra-
average user. Moreover, we utilized a single AI software, and these phy versus chest radiography in adults with symptoms of acute decompensated
heart failure. Am J Cardiol. 2022;174:89–95. https://doi.org/10.1016/j.amjcard.
findings may not apply to other software or future AI updates. The
2022.03.022.
AI was conducted using saved loops. While this was important to en- [7] Volpicelli G, Elbarbary M, Blaivas M, et al. International evidence-based recommen-
sure the physician and AI software used the exact same images, it did dations for point-of-care lung ultrasound. Intensive Care Med. 2012;38(4):577–91.
not allow use of the AI quality indicator, which may have influenced https://doi.org/10.1007/s00134-012-2513-4.
[8] Pang PS, Russell FM, Ehrman R, et al. Lung ultrasound-guided emergency De-
the accuracy. Finally, we did not compare accuracy of AI using differ- partment Management of Acute Heart Failure (BLUSHED-AHF): a randomized
ent probes or depths and future work may be needed to determine controlled pilot trial. JACC Heart Fail. 2021;9(9):638–48. https://doi.org/10.
how these affect AI accuracy. 1016/j.jchf.2021.05.008.
[9] Gullett J, Donnelly JP, Sinert R, et al. Interobserver agreement in the evaluation of
B-lines using bedside ultrasound. J Crit Care. 2015;30(6):1395–9. https://doi.org/
5. Conclusion 10.1016/j.jcrc.2015.08.021.
[10] Gottlieb M, Duanmu Y. Beyond the numbers: assessing competency in point-of-care
Both physician and AI software were highly sensitive for pulmonary ultrasound. Ann Emerg Med. 2023;81(4):427–8. https://doi.org/10.1016/j.
annemergmed.2023.01.020.
edema, though the physician was more specific. This suggests that the [11] Russell FM, Ehrman RR, Barton A, Sarmiento E, Ottenhoff JE, Nti BK. B-line quantifi-
greatest utility of AI for this application may be for excluding the pres- cation: comparing learners novice to lung ultrasound assisted by machine artificial
ence of pulmonary edema, while caution should be exercised when in- intelligence technology to expert review. Ultrasound J. 2021;13(1):33. https://doi.
org/10.1186/s13089-021-00234-6.
terpreting positive findings. Future research should identify which
[12] Moore CL, Wang J, Battisti AJ, et al. Interobserver agreement and correlation of an
factors are associated with increased diagnostic accuracy. automated algorithm for B-line identification and quantification with expert
Sonologist review in a handheld ultrasound device. J Ultrasound Med. 2022;41
Prior presentations (10):2487–95. https://doi.org/10.1002/jum.15935.
[13] Short J, Acebes C, Rodriguez-de-Lema G, et al. Visual versus automatic ultrasound
scoring of lung B-lines: reliability and consistency between systems. Med Ultrason.
None. 2019;21(1):45–9. https://doi.org/10.11152/mu-1885.
[14] von Elm E, Altman DG, Egger M, et al. The strengthening the reporting of observa-
tional studies in epidemiology (STROBE) statement: guidelines for reporting obser-
Funding
vational studies. Ann Intern Med. 2007;147(8):573–7. https://doi.org/10.7326/
0003-4819-147-8-200710160-00010.
This study was supported by the Society for Academic Emergency [15] Ultrasound Guidelines. Emergency, point-of-care and clinical ultrasound guidelines
Medicine Foundation/Academy of Emergency Ultrasound Research in medicine. Ann Emerg Med. 2017;69(5):e27–54. https://doi.org/10.1016/j.
annemergmed.2016.08.457.
Grant. The ultrasound machines were temporarily donated by GE [16] Duggan NM, Goldsmith AJ, Saud AAA, Ma IWY, Shokoohi H, Liteplo AS. Optimizing
Healthcare only for the length of the study. GE Healthcare did not lung ultrasound: the effect of depth, gain and focal position on sonographic B-
have influence over the study design, data acquisition, data analysis, lines. Ultrasound Med Biol. 2022;48(8):1509–17. https://doi.org/10.1016/j.
ultrasmedbio.2022.03.015.
manuscript, or decision to publish.
[17] Damodaran S, Kulkarni AV, Gunaseelan V, Raj V, Kanchi M. Automated versus man-
ual B-lines counting, left ventricular outflow tract velocity time integral and inferior
Declaration of Competing Interest vena cava collapsibility index in COVID-19 patients. Indian J Anaesth. 2022;66(5):
368–74. https://doi.org/10.4103/ija.ija_1008_21.
[18] Tsaban G, Galante O, Almog Y, Ullman Y, Fuchs L. Feasibility of machine integrated
We have no conflicts of interest to declare and this manuscript has point of care lung ultrasound automatic B-lines tool in the Corona-virus 2019 critical
not been submitted elsewhere. care unit. Crit Care. 2021;25(1):345. https://doi.org/10.1186/s13054-021-03770-8.
[19] Schuh S, Man C, Cheng A, et al. Predictors of non-diagnostic ultrasound scanning in
children with suspected appendicitis. J Pediatr. 2011;158(1):112–8. https://doi.org/
Acknowledgements
10.1016/j.jpeds.2010.07.035.
[20] Trout AT, Sanchez R, Ladino-Torres MF, Pai DR, Strouse PJ. A critical evaluation of US
The authors wish to thank all of the patients who were a part of this for the diagnosis of pediatric acute appendicitis in a real-life setting: how can we im-
study and GE Healthcare for donating the ultrasound machine. The prove the diagnostic value of sonography? Pediatr Radiol. 2012;42(7):813–23.
https://doi.org/10.1007/s00247-012-2358-6.
authors also wish to thank Faith Geevarghese, Fae Kayarian, Leslie Mar- [21] Macaione I, Galvano A, Graceffa G, et al. Impact of BMI on preoperative axillary ultra-
tinez, Jonas Neichin, and Simone Ymson for their help with the study. sound assessment in patients with early breast Cancer. Anticancer Res. 2020;40(12):
7083–8. https://doi.org/10.21873/anticanres.14736.
References [22] Jeeji AK, Ekstein SF, Ifelayo OI, et al. Increased body mass index is associated with de-
creased imaging quality of point-of-care abdominal aortic ultrasonography. J Clin Ul-
[1] Centers for Disease Control and Prevention. National Center for Health Statistics. Na- trasound. 2021;49(4):328–33. https://doi.org/10.1002/jcu.22929.
tional Hospital Ambulatory Medical Care Survey. Emergency Department Summary [23] Abu-Zidan FM, Hefny AF, Corr P. Clinical ultrasound physics. J Emerg Trauma Shock.
Tables. Accessed April 13, 2023. https://www.cdc.gov/nchs/data/nhamcs/web_ 2011;4(4):501–3. https://doi.org/10.4103/0974-2700.86646.
tables/2020-nhamcs-ed-web-tables-508.pdf; 2020. [24] Uppot RN. Technical challenges of imaging & image-guided interventions in obese
[2] Rutten FH, Cramer MJM, Lammers JWJ, Grobbee DE, Hoes AW. Heart failure and patients. Br J Radiol. 2018;91(1089):20170931. https://doi.org/10.1259/bjr.
chronic obstructive pulmonary disease: an ignored combination? Eur J Heart Fail. 20170931.
2006;8(7):706–11. https://doi.org/10.1016/j.ejheart.2006.01.010. [25] Haaksma ME, Smit JM, Heldeweg MLA, Pisani L, Elbers P, Tuinman PR. Lung ultra-
[3] Jabbour A, Macdonald PS, Keogh AM, et al. Differences between beta-blockers in pa- sound and B-lines: B careful! Intensive Care Med. 2020;46(3):544–5. https://doi.
tients with chronic heart failure and chronic obstructive pulmonary disease: a ran- org/10.1007/s00134-019-05911-8.
domized crossover trial. J Am Coll Cardiol. 2010;55(17):1780–7. https://doi.org/10. [26] Arntfield R, VanBerlo B, Alaifan T, et al. Development of a convolutional neural net-
1016/j.jacc.2010.01.024. work to differentiate among the etiology of similar appearing pathological B lines on
[4] Hawkins NM, Jhund PS, Simpson CR, et al. Primary care burden and treatment of pa- lung ultrasound: a deep learning study. BMJ Open. 2021;11(3):e045120. https://doi.
tients with heart failure and chronic obstructive pulmonary disease in Scotland. Eur J org/10.1136/bmjopen-2020-045120.
Heart Fail. 2010;12(1):17–24. https://doi.org/10.1093/eurjhf/hfp160.
112
Descargado para Anonymous User (n/a) en Cayetano Heredia Pervuvian University de ClinicalKey.es por Elsevier en septiembre 07, 2023. Para
uso personal exclusivamente. No se permiten otros usos sin autorización. Copyright ©2023. Elsevier Inc. Todos los derechos reservados.