14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023].

See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Ultrasound Obstet Gynecol 2020; 56: 588–596
Published online in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/uog.21878

Development and validation of a machine-learning model for

prediction of shoulder dystocia
A. TSUR1,2 , L. BATSRY2 , S. TOUSSIA-COHEN2 , M. G. ROSENSTEIN3 , O. BARAK4 ,
Y. BREZINOV4 , R. YOELI-ULLMAN2 , E. SIVAN2 , M. SIROTA6 , M. L. DRUZIN1 ,
D. K. STEVENSON5 , Y. J. BLUMENFELD1 and D. ARAN6
1
Department of Obstetrics and Gynecology, Division of Maternal Fetal Medicine, Stanford University School of Medicine, Stanford, CA,
USA; 2 Department of Obstetrics and Gynecology, The Sheba Medical Center, Tel Hashomer, Israel; 3 Department of Obstetrics and
Gynecology, Division of Maternal Fetal Medicine, University of California, San Francisco, CA, USA; 4 Department of Obstetrics and
Gynecology, The Kaplan Medical Center, Rehovot, Israel; 5 Department of Pediatrics, Division of Neonatal and Developmental Medicine,
Stanford University School of Medicine, Stanford, CA, USA; 6 Bakar Computational Health Sciences Institute, University of California,
San Francisco, CA, USA

K E Y W O R D S: anthropometry; artificial intelligence; biometry; EHR; macrosomia

CONTRIBUTION vaginal deliveries, of which 131 were complicated by

What does this work add to what is already known?
We developed and externally validated a machine-learning
model integrating maternal risk modifiers with fetal
biometry for prediction of shoulder dystocia (ShD). The
model was significantly more accurate than was prediction
based on estimated fetal weight either alone or combined
with maternal diabetes and was able to stratify the risk
of ShD and neonatal injury in the context of suspected
macrosomia.
What are the clinical implications of this work?
We demonstrated the potential clinical efficacy of applying
this model to stratify the risk of ShD and related newborn
injury among women carrying a fetus with estimated fetal
weight ≥ 4000 g.
ABSTRACT
Objectives To develop a machine-learning (ML) model
for prediction of shoulder dystocia (ShD) and to externally
validate the model’s predictive accuracy and potential
clinical efficacy in optimizing the use of Cesarean delivery
in the context of suspected macrosomia.
Methods We used electronic health records (EHR) from
the Sheba Medical Center in Israel to develop the model
(derivation cohort) and EHR from the University of
California San Francisco Medical Center to validate the
model’s accuracy and clinical efficacy (validation cohort).
Subsequent to application of inclusion and exclusion
criteria, the derivation cohort included 686 singleton
ShD, and the validation cohort included 2584 deliveries,
of which 31 were complicated by ShD. For each of these
deliveries, we collected maternal and neonatal delivery
outcomes coupled with maternal demographics, obstetric
clinical data and sonographic fetal biometry. Biometric
measurements and their derived estimated fetal weight
were adjusted (aEFW) according to gestational age at
delivery. A ML pipeline was utilized to develop the model.
Results In the derivation cohort, the ML model pro-
vided significantly better prediction than did the current
clinical paradigm based on fetal weight and maternal
diabetes: using nested cross-validation, the area under
the receiver-operating-characteristics curve (AUC) of the
model was 0.793 ± 0.041, outperforming aEFW com-
bined with diabetes (AUC = 0.745 ± 0.044, P = 1e−16 ).
The following risk modifiers had a positive beta that
was > 0.02, i.e. they increased the risk of ShD: aEFW
(beta = 0.164), pregestational diabetes (beta = 0.047),
prior ShD (beta = 0.04), female fetal sex (beta = 0.04)
and adjusted abdominal circumference (beta = 0.03). The
following risk modifiers had a negative beta that was
< −0.02, i.e. they were protective of ShD: adjusted
biparietal diameter (beta = −0.08) and maternal height
(beta = −0.03). In the validation cohort, the model out-
performed aEFW combined with diabetes (AUC = 0.866
vs 0.784, P = 0.00007). Additionally, in the validation
cohort, among the subgroup of 273 women carrying a
fetus with aEFW ≥ 4000 g, the aEFW had no predictive
power (AUC = 0.548), and the model performed signifi-
cantly better (0.775, P = 0.0002). A risk-score threshold
of 0.5 stratified 42.9% of deliveries to the high-risk group,

Correspondence to: Dr A. Tsur, 531 Maybell Avenue, Palo Alto, CA, 94306, USA (e-mail: avitsur@stanford.edu); and Dr D. Aran, Faculty
of Biology, Technion-Israel Institute of Technology, Haifa, Israel (e-mail: dviraran@technion.ac.il)
Accepted: 16 September 2019

Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd. ORIGINAL PAPER

Utilizing AI to predict shoulder dystocia
which included 90.9% of ShD cases and all cases accom-
panied by maternal or newborn complications. A more
specific threshold of 0.7 stratified only 27.5% of the
deliveries to the high-risk group, which included 63.6%
of ShD cases and all those accompanied by newborn
complications.
Conclusion We developed a ML model for prediction
of ShD and, in a different cohort, externally validated
its performance. The model predicted ShD better than
did estimated fetal weight either alone or combined with
maternal diabetes, and was able to stratify the risk of
ShD and neonatal injury in the context of suspected
macrosomia. Copyright © 2019 ISUOG. Published by
John Wiley & Sons Ltd.
INTRODUCTION
Shoulder dystocia (ShD) is an obstetric emergency, defined
as difficulty in delivering the newborn’s shoulders after
successful delivery of the head, thus requiring additional
maneuvers1,2 . Once ShD occurs, even if all appropriate
action is taken, there is an increased risk for permanent
neonatal and maternal injury3–5 .
Given the association between fetal macrosomia
(excess fetal size) and ShD6 , the American College of
Obstetricians and Gynecologists (ACOG) recommends
considering Cesarean delivery (CD) to prevent ShD for
women with diabetes carrying a fetus with an estimated
fetal weight (EFW) of at least 4500 g1 and for non-diabetic
women carrying a fetus with an EFW of at least 5000 g1 .
Since more than 50% of ShD cases occur at an EFW
below these cut-offs7 , many healthcare providers discuss
with their patient the options of CD at lower cut-offs.
In a large study investigating physician-documented
indications for CD at Yale New Haven Hospital, CD
was generally offered at weight thresholds 500 g below
the ACOG cut-offs for both diabetic and non-diabetic
women. In some circumstances, even lower thresholds
were used8 . Unfortunately, any arbitrary lowering of the
ACOG cut-off leads to a growing number of potentially
avoidable CDs, which in turn can lead to obstetric
morbidity in subsequent pregnancies9 .
In order to optimize the prediction of ShD, several stud-
ies have utilized regression methods10–14 in an attempt
to integrate EFW and maternal diabetes with additional
established maternal, fetal and intrapartum risk modifiers.
These include prior ShD15 , the need for operative vaginal
delivery14 , maternal height and weight13 and fetal anthro-
pometric variations10 . Unfortunately, these studies have
not resulted in a validated and established clinical model,
partly because of the limitations of previously employed
statistical methods. Machine-learning (ML) algorithms
complement and extend classic statistical methods and
can detect patterns from large, heterogeneous and
complex data16,17 .
In the current study, therefore, we developed a ML
model for prediction of ShD, and, in a different popula-
tion, validated its predictive accuracy and clinical efficacy.
Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.
14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
589
METHODS
Derivation cohort
To develop the model (derivation cohort), we used
electronic health records (EHR) from the Sheba Medical
Center, a tertiary-care medical center affiliated with
Tel-Aviv University in Israel. EHR data of all deliveries
at Sheba are collected and stored by the obstetrics team
as part of routine clinical care. The study was approved
by the Sheba Medical Center Committee for Human
Subjects Research (Institutional Review Board (IRB)
number 5221-18-SMC). Because ShD is a rare event, we
enriched the cohort with ShD cases, to beyond the natu-
rally occurring incidence. For this purpose, we retrieved
EHR from 53 754 singleton vaginal deliveries between
January 2011 and August 2018. During this period,
238 cases of ShD were recorded (0.44% of singleton
vaginal deliveries). These cases were compared with 1008
consecutive deliveries not complicated by ShD from 2
consecutive months in 2016. We reviewed manually the
charts of all cases and controls to identify and verify the
following 18 parameters that may modify the risk of ShD:
maternal age, gestational age (GA) at delivery, maternal
height, maternal weight, smoking status, gravidity, parity,
intrauterine fetal death, ShD in a previous pregnancy,
pregestational diabetes, gestational diabetes, insulin treat-
ment, fetal sex, four sonographic biometric measurements
(fetal biparietal diameter (BPD), head circumference
(HC), abdominal circumference (AC) and femur length
(FL)) and the EFW derived from these measurements.
Additionally, for all ShD cases, we identified and
verified potentially related neonatal and maternal com-
plications. The neonatal complications identified were:
brachial plexus injuries (BPI), fractures of the clavicle or
humerus and hypoxic ischemic encephalopathy (HIE)3 .
The maternal complications were: severe perineal lac-
eration (defined as third- or fourth-degree tear)4 and
postpartum hemorrhage (PPH)5 .
Inclusion criteria were: (1) sonographic EFW assess-
ment within 5 weeks prior to delivery; (2) complete data
on all 18 parameters that may modify the risk of ShD;
and (3) GA at delivery > 34 + 0 weeks. We excluded any
case with one or more outlier parameters, defined as 5
SD over the average value for any of the variables. This
filtering process, summarized in Table 1, reduced our
derivation cohort to 686 vaginal deliveries, of which 131
(19.1%) had ShD.
Validation cohort
To validate the model’s accuracy and clinical efficacy
(validation cohort), we used EHR from the University
of California, San Francisco (UCSF), a tertiary-care, aca-
demic medical center. Clinical obstetric data from all
deliveries at UCSF are entered into a clinical research
database as part of routine clinical care by the obstetric
and maternal-fetal medicine team. Data are validated by
research coordinators shortly thereafter. The UCSF Com-
mittee for Human Subjects Research approved this study
Ultrasound Obstet Gynecol 2020; 56: 588–596.
 14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
590 Tsur et al.

Table 1 Overview of data in derivation and validation cohorts of singleton vaginal deliveries used in development and validation of
machine-learning model for prediction of fetal shoulder dystocia (ShD)

Derivation cohort (n = 53 754) Validation cohort (n = 23 794)

ShD Controls ShD Without ShD

Potentially eligible cases 238 1008 302 23 492

Did not meet inclusion criteria
Biometry > 5 weeks prior to delivery 25 167 268 20 532
Not all parameters available 82 273 2 339
GA at delivery ≤ 34 + 0 weeks 0 9 0 46
Excluded 0 4 1 22
Final cohort 131 555 31 2553

Data are given as n. Derivation cohort included deliveries in period 2011–2018; validation cohort included deliveries in period 2003–2018.
GA, gestational age.

(IRB number #17-22929). We analyzed all UCSF deliver- (using the function pnorm in the R statistics package
ies between March 2003 and September 2018, identifying (https://www.R-project.org/)):
23 492 singleton vaginal deliveries that were not and 302  

cases that were complicated by ShD. We applied the same EFWi − EFW (USGA)
Pi =  .
inclusion and exclusion criteria as had been applied to the 0.12 · EFW (USGA)
derivation cohort, which resulted in 2584 vaginal deliv-
eries, of which 31 (1.2%) had ShD (Table 1). Details on Next, we converted the calculated percentiles back to
ShD in a previous pregnancy were not available in the val- values, according to GA at time of delivery, as follows.
idation cohort and thus could not be used. We reviewed We calculated the median expected EFW for the GA on
manually all charts of ShD cases to verify the diagnosis the day of delivery (DGA):
and collected data on the same neonatal and maternal  2
EFW (DGA) = e0.578+0.332·DGA −0.00354·DGA .
complications potentially related to ShD as had been col-
lected for the derivation cohort (BPI, clavicular or humeral We then calculated aEFW based on the values calculated
fracture, HIE, PPH and severe perineal laceration). in the previous steps:

aEFWi = EFW (DGA)
Adjusting sonographic biometry according to 
gestational age + 0.12 · EFW (DGA)· qnorm(Pi ).

We confirmed that, in both cohorts, EFW had been
calculated from biometric parameters using the same
Hadlock equation18 :
Log10 EFW = 1.3596 − 0.00386 AC × FL
+ 0.0064 HC + 0.00061 BPD × AC
+ 0.0424 AC + 0.174 FL.
EFW may change between the time of the growth ultra-
sound examination and delivery (mean ± SD time inter-
val, 5.2 ± 6.7 days in derivation cohort and 18.1 ± 9.3
in validation cohort). Thus, we adjusted the biometric
parameters according to GA at delivery. To do so, first,
we calculated the percentile of each sonographic measure-
ment at the GA at which it had been performed, based
on the Hadlock equations19,20 , as follows. We calculated
the median expected EFW for GA on the day on which
the measurement was performed (USGA) based on the
Hadlock equation:
 2
EFW (USGA) = e0.578+0.332·USGA−0.00354·USGA .
For each observation, i, we calculated the percentile
of the EFW at the time of delivery based on the normal
distribution and the difference between the measured
EFW and the median EFW on the day of measurement
These steps were for adjusting EFW to aEFW. The
same steps were performed, using the relevant equations,
to adjust BPD, HC, AC and FL to aBPD, aHC, aAC and
aFL, respectively.
As described in detail in Figure S1, the correlation
coefficient of aEFW with birth weight (BW) was 0.84,
significantly higher than the correlations of BW with
unadjusted EFW (Fisher’s r-to-Z transformation test
P = 0.0248). In addition, the correlation of aEFW
with BW was superior to the correlation between the
weight percentile at the time of the most recent growth
ultrasound examination and BW percentile (P = 0.006).
Moreover, the correlation of aEFW was superior even
to the correlation between clinical EFW obtained within
1 week of delivery and BW (P = 0.0349). Therefore, for
developing and validating the predictive model, we used
the adjusted sonographic measurements.
Machine-learning pipeline
To develop the ML model we used the generalized
linear model algorithm with lasso penalty21 (using the R
statistics package, glmnet). Lasso regression is a powerful
technique for creating parsimonious models and reduces
issues related to overfitting. To test the ability of ML to
predict ShD, we first split the derivation cohort into a

Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd. Ultrasound Obstet Gynecol 2020; 56: 588–596.

Utilizing AI to predict shoulder dystocia
80% training set and a 20% test set. Using the training
set, a 10-fold cross-validation analysis was performed,
with ShD as the response vector. Misclassification error
was used as loss function for the cross-validation. To
cope with the imbalance in classes, the observations
were weighted with values of the fraction of the opposite
class: ShD observations were weighted with 0.8090,
while control observations were weighted with 0.1906.
This algorithm generates models for a range of λ,
which is a tuning parameter that controls the overall
strength of the penalty. We chose the model that is
based on the minimal value of λ that gives minimum
mean cross-validated error. Finally, the area under the
receiver-operating-characteristics curve (AUC) was calcu-
lated for the test set. Since the initial split for training and
test sets was random, we performed nested analysis: the
procedure described was repeated 500 times, each time
with random split for training and test sets. Subsequently,
we used the entire derivation cohort as a training set to
fit a robust model. This model was then applied to the
validation cohort. Analyses based on aEFW and maternal
diabetes (as defined in the ACOG practice bulletin1 :
either pre- or gestational diabetes) alone were performed
using logistic regression. In all analyses, the features were
centered and scaled to mean zero with a SD of 1 (z-score)
using parameters learned from the training cohort.
Statistical analysis
Fisher’s exact test was used to calculate P-values for
association of binary variables with ShD (Table 2 and
Figure 1). Logistic regression was used for continuous
variables, in which the odds ratios are the exponent of the
predictor coefficient, and the P-value is based on the nor-
mal distribution of the dispersion of the coefficient. The
AUC was used as the primary metric of the model’s perfor-
mance. A paired-samples t-test was used to test statistical
significance of the repeated AUCs in the derivation cohort.
The standard non-parametric test for comparing two cor-
related ROC curves (DeLong’s test)22 was used to test the
significance of this comparison in the validation cohort.
Pearson’s coefficients were used for correlation analyses.
The significance of differences in correlation coefficients
was tested using Fisher’s r-to-Z transformation.
RESULTS
Demographics
Between January 2011 and August 2018, there were
53 754 singleton vaginal deliveries at the Sheba Medical
Center in Israel. Of these, 238 (0.4%) deliveries were
complicated by ShD. Among these cases with ShD,
44 (18.5%) were complicated by one or more of the
following: PPH (n = 11; 4.6%), severe perineal laceration
(n = 8; 3.4%), BPI (n = 10; 4.2%), clavicular or humeral
fracture (n = 19; 8.0%) and HIE (n = 1; 0.4%). Applying
the inclusion and exclusion criteria (Table 1) reduced our
derivation cohort to 686 vaginal deliveries, of which 131
(19.1%) were complicated by ShD.
Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.
14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
591
Between March 2003 and September 2018, there were
23 794 singleton vaginal deliveries at the UCSF Medical
Center. Of these, 302 (1.27%) were complicated by ShD.
Applying the inclusion and exclusion criteria (Table 1)
reduced the validation cohort to 2584 singleton vaginal
deliveries, of which 31 (1.2%) were complicated by ShD
(Table 1). Among these cases with ShD, 12 (38.7%)
were complicated by one or more of the following: PPH
(n = 4; 12.9%), severe perineal laceration (n = 4; 12.9%),
BPI (n = 8; 25.8%; two of these cases resolved before
neonatal discharge) and clavicular or humeral fracture
(n = 1; 3.2%).
Analysis in the derivation and validation cohorts of
the 18 maternal and prenatal sonographic variables that
may modify the risk of ShD showed that both datasets
contained similar associations with ShD (Table 2 and
Figure 1). BW, aEFW, aAC, pregestational diabetes and
insulin treatment were significantly associated with ShD
in both cohorts.
Shoulder dystocia prediction model
Figure 2b depicts the ML model. The following risk
modifiers had a positive beta > 0.02, i.e. they increased
the risk of ShD: aEFW (beta = 0.164), pregestational
diabetes (beta = 0.047), prior ShD (beta = 0.04), female
fetal sex (beta = 0.04) and adjusted AC (beta = 0.03). The
following risk modifiers had a negative beta < −0.02, i.e.
they were protective of ShD: adjusted BPD (beta = −0.08)
and maternal height (beta = −0.03). In contrast to
pregestational diabetes, gestational diabetes had a beta
of 0 and therefore its presence or absence did not modify
the risk of ShD.
Prediction of shoulder dystocia: derivation cohort
The AUC of aEFW to predict ShD was only
0.745 ± 0.044. Considering diabetes (as defined in
the ACOG practice bulletin1 : either pre- or gestational
diabetes) in addition to aEFW did not improve its
prediction accuracy (AUC = 0.745 ± 0.044). We trained
a ML model using the 18 potential ShD risk modifiers.
On 500 random splits of training and test sets, the AUC
of the ML model was 0.793 ± 0.041, outperforming
aEFW alone as well as aEFW and diabetes in 95.2% of
the repeats (P < 1e−16 , Figure 2a). We tested an ensemble
of ML algorithms but did not observe any improvement
over our regularized generalized linear model (Figure S2).
We next utilized the entire derivation cohort to generate
a robust predictive model (Figure 2b). The variable with
the highest contribution to the model was aEFW, but
other sonographic measurements were also important,
most notably aBPD, which had a negative association
with ShD risk in the model. Independently, increasing
aBPD was significantly associated with increased risk of
ShD (P = 0.028). However, after adjusting the aBPD to
aEFW, we observed that increasing aBPD was associated
with a significantly reduced risk for ShD (P = 3e−7 )
(Figure 2c).
Ultrasound Obstet Gynecol 2020; 56: 588–596.
 14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
592 Tsur et al.

Table 2 Demographics and odds ratios (OR) of risk modifiers in derivation and validation cohorts of singleton vaginal deliveries used in
development and validation of machine-learning model for prediction of shoulder dystocia (ShD)

Derivation cohort Validation cohort

Controls Cases Log2 OR Controls Cases Log2 OR
(n = 555) (n = 131) (95% CI) P (n = 2553) (n = 31) (95% CI) P

Maternal factors
Age (years) 32.5 ± 4.9 31.9 ± 5.1 –0.19 0.186 33.0 ± 5.3 32.4 ± 6.2 –0.18 0.486
(–0.46 to 0.09) (–0.67 to 0.33)
Weight (kg)* 74.9 ± 12.6 79.1 ± 15.0 0.43 0.001 79.0 ± 17.0 84.1 ± 17.5 0.35 0.101
(0.17 to 0.69) (–0.10 to 0.75)
Height (cm) 164.0 ± 6.1 163.2 ± 5.6 –0.18 0.215 163.3 ± 6.9 160.9 ± 6.6 –0.48 0.048
(–0.45 to 0.10) (–0.94 to 0.01)
Gravidity 2.7 ± 1.8 2.6 ± 1.7 –0.03 0.821 2.4 ± 1.6 2.6 ± 2.5 0.18 0.433
(–0.32 to 0.24) (–0.32 to 0.6)
Nulliparous 357 (64.3) 87 (66.4) 0.13 0.685 1178 (46.1) 12 (38.7) –0.44 0.471
(–0.32 to 0.24) (–1.49 to 0.61)
Previous ShD 0 (0) 4 (3.1) Inf < 1e−16 N/A N/A N/A N/A
Smoker 29 (5.2) 3 (2.3) –1.23 0.174 74 (2.9) 3 (9.7) 1.86 0.061
(–2.97 to 0.50) (0.11 to 3.61)
Current pregnancy
GA (days) 277.9 ± 9.6 280.3 ± 8.1 0.40 0.009 272.3 ± 10.4 274.0 ± 7.7 0.25 0.364
(0.11 to 0.70) (–0.27 to 0.82)
IUFD 2 (0.4) 2 (1.5) 2.10 0.167 3 (0.1) 0 (0) 0 1
(–0.74 to 4.94)
Diabetes
Pregestational 0 (0) 4 (3.1) Inf < 1e−16 74 (2.9) 5 (16.1) 2.67 0.002
diabetes (1.25 to 4.09)
Gestational 58 (10.5) 15 (11.5) 0.15 0.753 812 (31.8) 14 (45.2) 0.82 0.122
diabetes (–0.72 to 1.02) (–0.20 to 1.85)
Insulin treatment 8 (1.4) 7 (5.3) 1.95 0.013 406 (15.9) 10 (32.3) 1.33 0.024
(0.46 to 3.44) (0.23 to 2.43)
US examination
Female sex 273 (49.2) 80 (61.1) 0.70 0.015 1254 (49.1) 19 (61.3) 0.72 0.207
(0.14 to 1.26) (–0.33 to 1.76)
HC (adjusted, mm) 331.9 ± 11.3 335.3 ± 9.9 0.44 0.002 332.9 ± 13.6 336.2 ± 13.0 0.37 0.178
(0.16 to 0.72) (–0.16 to 0.91)
BPD (adjusted, mm) 93.2 ± 3.5 93.9 ± 3.2 0.32 0.028 92.7 ± 4.4 93.8 ± 3.5 0.39 0.150
(0.04 to 0.60) (–0.13 to 0.93)
AC (adjusted, mm) 335.1 ± 19.4 353.2 ± 15.6 1.75 < 1e−16 320.9 ± 21.3 344.7 ± 17.6 1.56 1e−9
(1.37 to 2.15) (1.06 to 2.07)
FL (adjusted, mm) 74.7 ± 3.2 76.4 ± 2.7 0.86 6e−8 73.6 ± 4.1 74.8 ± 3.7 0.42 0.13
(0.55 to 1.17) (–0.11 to 0.97)
EFW (adjusted, g) 3372.2 ± 401.7 3725.2 ± 340.8 1.50 1e−16 3349.2 ± 517.7 3838.8 ± 411.4 1.56 2e−7
(1.16 to 1.87) (1.06 to 2.07)
HC (percentile) 48.5 ± 1.2 48.8 ± 1.1 0.29 0.041 49.0 ± 1.4 49.2 ± 1.3 0.25 0.233
(0.01 to 0.56) (–0.26 to 0.77)
BPD (percentile) 49.3 ± 4.5 49.7 ± 4.4 0.14 0.324 50.1 ± 5.5 51.2 ± 4.4 0.28 0.287
(–0.14 to 0.41) (–0.23 to 0.81)
AC (percentile) 49.4 ± 1.3 50.6 ± 1.3 1.44 < 1e−16 50.1 ± 1.7 52.3 ± 1.3 1.68 4e−12
(1.12 to 1.79) (1.21 to 2.17)
FL (percentile) 46.4 ± 4.7 48.4 ± 4.4 0.63 1e−5 46.8 ± 5.5 48.0 ± 4.9 0.32 0.233
(0.35 to 0.92) (–0.20 to 0.83)
EFW (percentile) 36.5 ± 23.5 58.1 ± 24.0 1.26 < 1e−16 45.7 ± 28.6 75.4 ± 20.8 1.68 3e−7
(0.97 to 1.56) (1.07 to 2.38)
Birth weight (g) 3264.1 ± 427.5 3878.9 ± 387.9 2.66 < 1e−16 3157.8 ± 515.7 3880.3 ± 423.2 1.91 2e−13
(2.20 to 3.16) (1.41 to 2.44)

Data are given as mean ± SD or n (%), unless stated otherwise. Fisher’s exact test used to calculate P-values for binary variables; logistic
regression used for continuous variables: ORs are exponents of predictor coefficient and P-value is based on normal distribution of
dispersion of coefficient. *At admission to labor. AC, abdominal circumference; BPD, biparietal diameter; EFW, estimated fetal weight; FL,
femur length; GA, gestational age at delivery; HC, head circumference; Inf, infinity; IUFD, intrauterine fetal death; N/A, not available; US,
ultrasound.

Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd. Ultrasound Obstet Gynecol 2020; 56: 588–596.
 14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Utilizing AI to predict shoulder dystocia 593

Maternal age
Maternal height
Maternal weight
Gravidity
Nulliparous
Smoker
Gestational age
IUFD
Pregestational diabetes
Gestational diabetes
Insulin treatment
Sex (female)
Head circumference*
Femur length*
Biparietal diameter*
Abdominal circumference*
Sonographic EFW*
Previous ShD 0.0 2.5
–2 0 2 4 Odds ratio (log2)
Odds ratio (log2)

Figure 1 Odds ratios (log2 ) of association of features with shoulder dystocia (ShD) in derivation (left) and validation (right) cohorts.
Previous ShD data were not available for validation cohort. Maximum values were set as 4. *Gestational age-adjusted measurements. EFW,
estimated fetal weight; IUFD, intrauterine fetal death.

Prediction of shoulder dystocia: validation cohort
The ML model, which was learned exclusively from the
derivation cohort, achieved an AUC of 0.866, significantly
better than that of aEFW alone (0.772, P = 0.0002) or that
of aEFW and diabetes (0.784, P = 0.00007, Figure 3a),
validating in an independent cohort its ability to improve
the prediction of ShD.
We then evaluated the potential clinical efficacy
of the model in stratifying the risk among women
carrying a fetus with a sonographic aEFW ≥ 4000 g who
delivered vaginally. We identified 273 such deliveries,
of which 11 (4.0%) were complicated by ShD. In these
deliveries, aEFW had minimal predictive power for ShD
(AUC = 0.548) and risk could not be stratified by increas-
ing weight. In contrast, the ML performed significantly
better (AUC = 0.775, P = 0.0002). In Table 3, we present
two thresholds for stratifying the risk of women carrying
a fetus with EFW ≥ 4000 g. A risk-score threshold of 0.5
stratified 42.9% of deliveries to the high-risk group, which
included 90.9% of ShD cases and all those accompanied
by maternal or newborn complications (Figure 3b). Using
this threshold would require 11.7 CD to prevent one
ShD case (number needed-to-treat). Alternatively, a more
specific threshold of 0.7 stratified only 27.5% of the
deliveries to the high-risk group, which included 63.6%
of ShD cases, but all those accompanied by newborn
complications (Figure 3b). Using this threshold would
require 10.7 CD to prevent one ShD case (Table 3).
DISCUSSION
In this study, we developed and validated a new
risk-prediction model to estimate the risk of ShD based on
maternal risk factors and fetal sonographic parameters.
Prior seminal studies attempting to create a model
for predicting ShD based on different risk-factor
combinations, with and without sonographic EFW,
were unable to outperform EFW10–12,14 . In contrast,
in both cohorts, our ML model performed significantly
better than did EFW, either alone or combined with
diabetes (as defined in the ACOG practice bulletin1 :
either pre- or gestational diabetes). By integrating
18 different parameters, we were able to untangle
and add precision to previously reported associations.
For example, we observed that pregestational diabetes
significantly increased the risk of ShD, while gestational
diabetes did not affect the risk at all. Additionally, the
model was able to integrate the additive risk associated
with insulin treatment and obesity.
After establishing the model accuracy in the validation
cohort, we advanced to evaluating the potential clinical
efficacy of the model in optimizing the use of CD in
the context of suspected macrosomia. In the validation
cohort, we observed that increasing EFW beyond 4000 g
did not further affect the risk for ShD. Therefore, in
this group, risk could not be stratified by EFW alone.
Conversely, the ML model was able to stratify women
in this group into two further distinct groups in terms
of their risk for ShD. A risk-score threshold of 0.5
stratified 42.9% of deliveries to the high-risk group, which
included 90.9% of ShD cases and all those accompanied
by maternal or newborn complications. Furthermore, as
the main concern with ShD is associated newborn injury,
we present an additional, more specific, threshold of 0.7
that stratified only 27.5% of the deliveries to the high-risk
group, which included 63.6% of ShD cases, but all
those accompanied by newborn injury. Given that many
healthcare providers discuss CD with patients carrying

Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd. Ultrasound Obstet Gynecol 2020; 56: 588–596.
 14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
594 Tsur et al.

P = 0.028
(c) 105
(a) 100
P < 1e–16
95

aBPD
0.9
90
85
0.8
AUC

80
Controls ShD
0.7
P = 3e–7
105

aBPD adjusted for aEFW

100
0.6
95
FW

es

d)
et

se
aE

ab

a
90

-b
di

W
+

EF
FW

85
(a
aE

el
od

80
m
L

Controls ShD
M

(b) Sonographic EFW*

Pregestational diabetes
Previous ShD
Sex (female)
Abdominal circumference*
Maternal weight
IUFD
Insulin treatment
Variable

Gestational diabetes
Femur length*
Nulliparous
Smoker
Gravidity
Head circumference*
Maternal age
Gestational age
Maternal height
BPD*
0.0 0.1
Beta

Figure 2 Prediction of fetal shoulder dystocia (ShD) in derivation cohort. (a) Box-and-whiskers plots of predictive performance of estimated
fetal weight (EFW) adjusted for gestational age at delivery (aEFW) alone, logistic model trained with aEFW and diabetes, and machine-
learning (ML) model, measured by area under receiver-operating-characteristics curve (AUC). Dataset was split 500 times into training and
test sets, and AUCs presented are for test sets. Boxes represent median and upper and lower quartiles, and whiskers denote largest and
smallest values no more than 1.5 × interquartile range from limits. (b) Beta values for each variable in final model that was generated by ML
algorithm. *Gestational age-adjusted measurements. IUFD, intrauterine fetal death. (c) Box-and-whiskers plots showing association with
ShD of biparietal diameter (BPD) adjusted for gestational age at delivery (aBPD) (above) and aBPD adjusted for aEFW (below). aBPD had
significant positive association with ShD, while aBPD adjusted for aEFW had significant negative association with ShD, as suggested by the
ML model. Adjustment to aEFW was performed by fitting a linear model between aBPD and aEFW, extracting residuals and adding mean
aBPD. P-values are based on logistic regression with ShD as response variable.

a fetus with EFW ≥ 4000 g8 , we speculate that universal
application of our model in such patients could not only
reduce the number of ShD cases, but also optimize the use
of CD in the context of suspected macrosomia.
To our knowledge, and according to a review of
the literature (via a MedLine search using different
combinations of the terms ‘shoulder dystocia’, ‘machine
learning’, ‘artificial intelligence’ and ‘prediction’), this is
the first attempt to utilize a complete ML pipeline for
predicting ShD. Previous studies used regression analyses
for prediction, and were therefore required to rely on
a large number of assumptions16 related to interactions
between the features. Examples for such assumptions
are specific interactions, such as the difference between
the abdominal diameter and the BPD10 , the difference
between the AC and the HC23 , the AC/HC ratio and,
in patients with gestational diabetes mellitus, also the
FL/AC ratio24 . The use of ML obviated this dependence
and allowed us to generate an assumption-free model
quantifying the association with ShD of each of the 18
parameters while considering the complex interactions
between them.

Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd. Ultrasound Obstet Gynecol 2020; 56: 588–596.
 14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Utilizing AI to predict shoulder dystocia 595

(a) 1.00 (b) 1.00

0.75 0.75

Sensitivity

Risk score
0.50 0.50

0.25 0.25

0.00 0.00
0.00 0.25 0.50 0.75 1.00 4000 4250 4500 4750 5000
1 – Specificity aEFW (g)

Figure 3 External validation and clinical efficacy of the machine-learning (ML) model in prediction of fetal shoulder dystocia (ShD).
(a) Receiver-operating-characteristics (ROC) curves for prediction, in validation cohort, of ShD by estimated fetal weight adjusted for
gestational age at delivery (aEFW) alone ( ; area under ROC curve (AUC) = 0.772), logistic model trained with aEFW and diabetes ( ;
AUC = 0.784) and ML model ( ; AUC = 0.866). (b) Scatterplot of aEFW vs risk score for 273 deliveries with aEFW ≥ 4000 g. Risk-score
threshold of 0.5 stratified 42.9% of deliveries to high-risk group, which included 90.9% of ShD cases and all those accompanied by
maternal or newborn complications; more specific threshold of 0.7 stratified only 27.5% of deliveries to high-risk group, which included
63.6% of ShD cases, but all those accompanied by newborn complications. Plots indicate non-ShD cases ( ) and ShD cases with: newborn
injury ( ), maternal complications only ( ) and no additional related complications ( ).

Table 3 Predictive accuracy for shoulder dystocia (ShD) in validation cohort of singleton vaginal deliveries

Model risk-score Deliveries ShD

threshold (n) (n) Specificity Sensitivity NPV PPV NNT AUC

aEFW ≥ 4000 g 273 11 24.8 0.775

0.7 75 7 0.74 0.64 0.98 0.09 10.7
(0.68–0.79) (0.31–0.89) (0.95–0.99) (0.04–0.18)
0.5 117 10 0.59 0.91 0.99 0.09 11.7
(0.53–0.65) (0.59–1.00) (0.96–1.00) (0.04–0.15)
All 2584 31 83.4 0.866
0.7 144 11 0.95 0.35 0.99 0.08 13.1
(0.94–0.96) (0.19–0.55) (0.99–0.99) (0.04–0.13)
0.5 229 16 0.92 0.52 0.99 0.07 14.3
(0.91–0.93) (0.33–0.70) (0.99–1.00) (0.04–0.11)
Values in parentheses are 95% CI. aEFW, adjusted estimated fetal weight; AUC, area under receiver-operating-characteristics curve; NNT,
number needed-to-treat (number of Cesarean deliveries required to prevent one case of ShD); NPV, negative predictive value; PPV, positive
predictive value.

In many obstetric centers, women with suspected
macrosomia and/or diabetes are referred for fetal growth
ultrasound examination at ∼37 weeks’ gestation, while
delivery may take place several weeks later. In order to
increase the clinical generalizability of our findings, we
allowed up to 35 days between sonographic biometry
assessment and delivery in both cohorts. For this reason,
we adjusted all biometric parameters (BPD, HC, AC and
FL), as well as the EFW, according to GA at delivery,
by means of commonly used Hadlock equations18–20 . We
validated the accuracy of this practice in our data and,
furthermore, we observed that the AUC of aEFW alone
in the derivation and validation cohorts was 0.745 and
0.771, respectively, both being higher than the AUC of
0.70 observed in a previous study using biometry obtained
within 7 days of delivery10 . The prospective use of aEFW
can not only provide the specific risk for ShD at the exact
day of delivery, but also provide information regarding
the optimal GA for labor induction in the context of ShD
risk. This may be of particular importance in women at
intermediate risk for ShD who desire vaginal delivery,
because their risk keeps increasing as their fetus keeps
growing.
Another strength of our study is the use of two different
cohorts (from two different continents), one serving for
development and the other for validation of the model.
This practice was chosen to prevent a prevalent overfitting
error and to establish generalizability of the model.
Our study is not, however, without limitations. One
limitation is its retrospective design. We could assess
the model’s performance only when vaginal delivery was
achieved and not when CD was performed either before
or during the course of labor. Additionally, although the
derivation cohort included 131 ShD cases, the validation
cohort had only 31 ShD cases for which complete data for
all 18 parameters were available. However, even in this
relatively small cohort, the improvement achieved with
the model was highly statistically significant.

Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd. Ultrasound Obstet Gynecol 2020; 56: 588–596.

596
The model is available without cost at www
.mfmcalculator.org. Currently, until further validation,
we advocate using the model only as part of an
IRB-approved clinical investigation. We propose a
prospective randomized study focusing on women carry-
ing a fetus with EFW ≥ 4000 g but below current ACOG
cut-offs for CD. These women would be randomized to
either management based on their healthcare provider’s
clinical considerations or management based on ML risk
stratification. The two main outcomes would be CD rate
and ShD rate. Another important future step is to develop
a complementary ML model for reassessing the risk of
ShD before performing operative vaginal delivery.
In conclusion, we developed a ML model for prediction
of ShD. The model was more accurate in the prediction
of ShD than was EFW either alone or combined with
diabetes and provides a robust tool for optimizing the use
of CD in the context of suspected macrosomia.
ACKNOWLEDGMENT
We would like to acknowledge the contribution of Cinthia
Blat in extracting data from the UCSF deliveries database.
REFERENCES
1. ACOG. Practice Bulletin No 178. Shoulder Dystocia. Obstet Gynecol 2017; 129:
e.123–133.
2. RCOG. Green-top Guidedline, No 42. Shoulder Dystocia; 2012 (updated 2017).
https://www.rcog.org.uk/en/guidelines-research-services/guidelines/gtg42
3. Gherman RB, Ouzounian JG, Goodwin TM. Obstetric maneuvers for shoulder
dystocia and associated fetal morbidity. Am J Obstet Gynecol 1998; 178:
1126–1130.
4. Gauthaman N, Walters S, Tribe IA, Goldsmith L, Doumouchtsis SK. Shoulder
dystocia and associated manoeuvres as risk factors for perineal trauma. Int
Urogynecol J 2016; 27: 571–577.
5. Gherman RB, Goodwin TM, Souter I, Neumann K, Ouzounian JG, Paul RH. The
McRoberts’ maneuver for the alleviation of shoulder dystocia: how successful is it?
Am J Obstet Gynecol 1997; 176: 656–661.
SUPPORTING INFORMATION ON THE INTERNET
The following supporting information may be found in the online version of this article:
Figure S1 Adjustment of estimated fetal weight to gestational age at delivery.
Figure S2 Performance of machine-learning algorithms to predict shoulder dystocia.
Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.
14690705, 2020, 4, Downloaded from https://obgyn.onlinelibrary.wiley.com/doi/10.1002/uog.21878 by Nat Prov Indonesia, Wiley Online Library on [27/01/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Tsur et al.
6. Beta J, Khan N, Khalil A, Fiolna M, Ramadan G, Akolekar R. Maternal and neonatal
complications of fetal macrosomia: systematic review and meta-analysis. Ultrasound
Obstet Gynecol 2019; 54: 308–318.
7. Ouzounian JG, Korst LM, Miller DA, Lee RH. Brachial plexus palsy and shoulder
dystocia: obstetric risk factors remain elusive. Am J Perinatol 2013; 30: 303–307.
8. Barber EL, Lundsberg LS, Belanger K, Pettker CM, Funai EF, Illuzzi JL. Indications
contributing to the increasing cesarean delivery rate. Obstet Gynecol 2011; 118:
29–38.
9. Spong CY, Berghella V, Wenstrom KD, Mercer BM, Saade GR. Preventing the first
cesarean delivery: summary of a joint Eunice Kennedy Shriver National Institute of
Child Health and Human Development, Society for Maternal-Fetal Medicine, and
American College of Obstetricians and Gynecologists Workshop. Obstet Gynecol
2012; 120: 1181–1193.
10. Burkhardt T, Schmidt M, Kurmanavicius J, Zimmermann R, Schäffer L. Evaluation
of fetal anthropometric measures to predict the risk for shoulder dystocia. Ultrasound
Obstet Gynecol 2014; 43: 77–82.
11. Gupta M, Hockley C, Quigley MA, Yeh P, Impey L. Antenatal and intrapartum
prediction of shoulder dystocia. Eur J Obstet Gynecol Reprod Biol 2010; 151:
134–139.
12. Belfort MA, Dildy GA, Saade GR, Suarez V, Clark SL. Prediction of shoulder dystocia
using multivariate analysis. Am J Perinatol 2007; 24: 5–10.
13. Dyachenko A, Ciampi A, Fahey J, Mighty H, Oppenheimer L, Hamilton EF.
Prediction of risk for shoulder dystocia with neonatal injury. Am J Obstet Gynecol
2006; 195: 1544–1549.
14. Palatnik A, Grobman WA, Hellendag MG, Janetos TM, Gossett DR, Miller ES.
Predictors of shoulder dystocia at the time of operative vaginal delivery. Am J Obstet
Gynecol 2016; 215: 624.e1–5.
15. Ouzounian JG, Gherman RB, Chauhan S, Battista LR, Lee RH. Recurrent shoulder
dystocia: analysis of incidence and risk factors. Am J Perinatol 2012; 29: 515–518.
16. Bzdok D, Altman N, Krzywinski M. Statistics versus machine learning. Nat Methods
2018; 15: 233–234.
17. Beam AL, Kohane IS. Big Data and Machine Learning in Health Care. JAMA 2018;
319: 1317–1318.
18. Hadlock FP, Harrist RB, Sharman RS, Deter RL, Park SK. Estimation of fetal weight
with the use of head, body, and femur measurements–a prospective study. Am
J Obstet Gynecol 1985; 151: 333–337.
19. Hadlock FP, Deter RL, Harrist RB, Park SK. Estimating fetal age: computer-assisted
analysis of multiple fetal growth parameters. Radiology 1984; 152: 497–501.
20. Hadlock FP, Harrist RB, Martinez-Poyer J. In utero analysis of fetal growth: a
sonographic weight standard. Radiology 1991; 181: 129–133.
21. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Generalized Linear
Models via Coordinate Descent. J Stat Softw 2010; 33: 1–22.
22. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or
more correlated receiver operating characteristic curves: a nonparametric approach.
Biometrics 1988; 44: 837–845.
23. Endres L, DeFranco E, Conyac T, Adams M, Zhou Y, Magner K, O’Rourke L,
Bernhard KA, Siddiqui D, McCormick A, Abramowicz J, Merkel R, Jawish R,
Habli M, Floman A, Magann EF, Chauhan SP, Network CFR. Association of
Fetal Abdominal-Head Circumference Size Difference With Shoulder Dystocia: A
Multicenter Study. AJP Rep 2015; 5: e099–104.
24. Duryea EL, Casey BM, McIntire DD, Twickler DM. The FL/AC ratio for prediction
of shoulder dystocia in women with gestational diabetes. J Matern Fetal Neonatal
Med 2017; 30: 2378–2381.
Ultrasound Obstet Gynecol 2020; 56: 588–596.