Translational Stroke Research

https://doi.org/10.1007/s12975-020-00811-2

ORIGINAL ARTICLE

Stability Assessment of Intracranial Aneurysms Using Machine

Learning Based on Clinical and Morphological Features
Wei Zhu 1 & Wenqiang Li 1 & Zhongbin Tian 1 & Yisen Zhang 1 & Kun Wang 1 & Ying Zhang 1 & Jian Liu 1 & Xinjian Yang 1

Received: 19 February 2020 / Revised: 19 March 2020 / Accepted: 19 March 2020

# Springer Science+Business Media, LLC, part of Springer Nature 2020

Abstract
Machine learning (ML) as a novel approach could help clinicians address the challenge of accurate stability assessment of
unruptured intracranial aneurysms (IAs). We developed multiple ML models for IA stability assessment and compare their
performances. We enrolled 1897 consecutive patients with unstable (n = 528) and stable (n = 1539) IAs. Thirteen patient-specific
clinical features and eighteen aneurysm morphological features were extracted to generate support vector machine (SVM),
random forest (RF), and feed-forward artificial neural network (ANN) models. The discriminatory performances of the models
were compared with statistical logistic regression (LR) model and the PHASES score in IA stability assessment. Based on the
receiver operating characteristic (ROC) curve and area under the curve (AUC) values for each model in the test set, the AUC
values for RF, SVM, and ANN were 0.850 (95% CI 0.806–0.893), 0.858 (95 %CI 0.816–0.900), and 0.867 (95% CI 0.828–
0.906), demonstrating good discriminatory ability. All ML models exhibited superior performance compared with the statistical
LR and the PHASES score (the AUC values were 0.830 and 0.589, respectively; RF versus PHASES, P < 0.001; RF versus LR,
P = 0.038). Important features contributing to the stability discrimination included three clinical features (location, sidewall/
bifurcation type, and presence of symptoms) and three morphological features (undulation index, height-width ratio, and irreg-
ularity). These findings demonstrate the potential of ML to augment the clinical decision-making process for IA stability
assessment, which may enable more optimal management for patients with IAs in the future.

Keywords Intracranial aneurysms . Risk evaluation . Artificial intelligence . Machine learning . Unstable aneurysm

Introduction decades, a growing number of incidental unruptured aneu-

Intracranial aneurysms (IAs) affect about 3–5% of the adult
population [1]. As the application of non-invasive imaging
modalities as diagnostic tools has increased over the past
This work was originated from Beijing Neurosurgical Institute and
Beijing Tian Tan Hospital, No.119 South Fourth Ring West Road,
Fengtai District, Beijing
Electronic supplementary material The online version of this article
(https://doi.org/10.1007/s12975-020-00811-2) contains supplementary
material, which is available to authorized users.
* Jian Liu
jianliu_ns@163.com
* Xinjian Yang
yangxinjian@voiceoftiantan.org
1
Department of Interventional Neuroradiology, Beijing Neurosurgical
Institute and Beijing Tiantan Hospital, Capital Medical University,
No.119 South Fourth Ring West Road, Fengtai District,
Beijing 100050, China
rysms are being detected [2]. As a consequence, clinicians
are faced with uncertainty regarding the choice of optimal
management for such IAs, including preventive treatment,
which is associated with risk of procedure-related complica-
tions, and conservative management, which is associated with
risk of IA rupture. The ability to accurately and promptly
distinguish unstable IAs from stable ones could facilitate treat-
ment optimization and, thus lead to better outcomes [3].
In recent years, a large number of factors have been report-
ed to be related to unstable IAs with high rupture risk. These
factors include patient-specific clinical features, such as gen-
der, age, smoking, and hypertension [4, 5], and morphological
features of IAs, such as maximum size, aspect ratio (AR), and
size ratio (SR) [6, 7]. However, as the correlation among those
risk factors and aneurysm status is complex and multifactorial,
no consensus has been reached and stability assessment re-
mains difficult. Furthermore, the difficulty has also been ex-
acerbated by the discrepancy between two-dimensional (2D)
projection and three-dimensional (3D) projection [8, 9]. The

PHASES score is one major attempt for assessing the 5-year
rupture risk of unruptured IAs from a set of easily accessible
patient and aneurysm characteristics [10]. The score is based
on the six most rupture-related features, including population,
age, hypertension, size, earlier subarachnoid hemorrhage
(SAH) history, and the IA location. However, the score does
not account for any morphological features, so it needs further
improvement before it can be widely used in clinical setting
[11].
Machine learning (ML) has been widely used in medical
fields for image detection, diagnosis, and outcome prediction.
Since it belongs to the domain of artificial intelligence (AI),
ML outweighs traditional statistical methods in the process of
non-linear relationships and complex patterns problem [12,
13]. Previous studies have proven the efficacy and potential
of ML approaches in several medical fields [14–16]. For the
purpose of IA stability assessment, Liu et al. developed a two-
layer feed-forward artificial neural network model to predict
the rupture risk of anterior communicating artery aneurysms,
and the overall prediction accuracy could reach 94.8% for all
samples [17]. Kim et al. developed a convolutional neural
network based on 3D digital subtraction angiography (DSA)
for predicting the rupture risk in patients harboring small-size
aneurysms, and the overall accuracy was 76.84% [18]. Liu
et al. used ML to predict aneurysm stability and found that
flatness was the most important morphological determinant in
aneurysm stability [19]. However, to the best of our knowl-
edge, no previous study has ever compared ML models with
statistical method or the PHASES score in IA stability assess-
ment. Furthermore, the correlations between IA stability and
various rupture-related risk features have not been studied in
detail or examined in a large sample size.
In this study, we utilize a large data set to build multiple
ML models to discriminate IA stability (i.e., stable versus
unstable) based on patient clinical features and aneurysm mor-
phological features measured from 3D DSA. We compare the
discriminatory ability of each ML model with those of the
other models, classic statistical logistic regression (LR), and
the PHASES score. We further identify and compare features
with significant contributions to IA stability selected in the
aforementioned models.
Method
Patient Population
This study included clinical and radiological data in patients
diagnosed with IAs at Beijing Tiantan hospital between
January 2014 and December 2018. This study was approved
by the Institutional Review Board of Beijing Tiantan hospital,
and written informed consent was obtained from each patient
before operation. The patients’ information was de-identified
Transl. Stroke Res.
prior to analysis. We retrospectively review the medical re-
cords and image data in our database. The inclusion criteria
were (a) age over 18 years, (b) diagnosis of saccular aneu-
rysm, and (c) accessibility of clinical and radiological data.
The exclusion criteria were (a) diagnosis of traumatic, infec-
tious, fusiform, or dissecting aneurysms; (b) patients with
moyamoya disease or vascular malformation; and (c) absence
of clinical data or high-quality radiological data. The criteria
for unstable aneurysm include ruptured aneurysm within
1 month and growing aneurysm in a sequential imaging fol-
low-up. The process of patient selection is summarized in
Fig. 1. A total of 2067 IAs in 1897 patients with unstable
(n = 528) and stable (n = 1539) IAs were included in this
study.
Clinical and Morphological Features
The following clinical features were collected from medical
records: age, sex, smoking, drinking, hypertension, hyperlipe-
mia, diabetes mellitus, coronary heart disease, family history,
presence of symptoms, and multiplicity, as well as the IA
location. The IA locations were categorized as (a) internal
carotid artery (ICA), (b) middle cerebral artery (MCA), (c)
anterior cerebral artery (ACA), (d) anterior communicating
artery (AComA), (e) posterior communicating artery
(PComA), and (f) posterior circulation (PC) IAs.
To acquire precise and objective measurements, quantita-
tive morphological features in our study were extracted and
Fig. 1 The flowchart shows the patients selection, process of
discriminatory model generation, and final evaluation. SAH indicates
subarachnoid hemorrhage, CV indicates cross-validation. ROC means
receiver operating characteristic curve
Transl. Stroke Res.
measured from the reconstructed 3D DSA. Three-dimensional
models of IAs were reconstructed from DSA into a standard
tessellation language (STL) format; the method of construct-
ing and refining the 3D STL models was described in detail
previously [20, 21]. Morphological features were defined as in
previous studies and calculated using the GEOMAGIC 12.0
software (Geomagic, Morrisville, North Carolina) and Matlab
(The Math Works, Inc., Natick, Ma, USA). Blind to patient
information and stability status, morphological feature evalu-
ation was independently done by two interventionalists
(Zhongbin Tian and Kun Wang) before group allocation; dis-
cordance was resolved by a third evaluator (Jian Liu with
more than 10 years of experience in neuroradiology). A total
of eighteen morphological features that comprehensively de-
scribe the IAs and parent vessel geometries were measured,
including the maximum height (the maximum distance of the
dome from the neck center), perpendicular height (the maxi-
mum perpendicular distance of the dome from the neck
plane), neck diameter, width (the maximum distance of dome
perpendicular to maximum height), transverse diameter (the
maximum distance of dome perpendicular to perpendicular
height), maximal diameter (the largest distance within aneu-
rysm sac, used as the size), and volume. In addition, the AR,
SR, undulation index (UI), nonsphericity index (NSI),
volume-to-neck ratio (VNR), height-width ratio, and bottle-
neck factor were defined and calculated as described in previ-
ous studies [22–25]. Features related to the parent vessel, such
as the flow angle (angles between the vector of the IA size and
vector of the centerline of the feeding parent vessel) and an-
eurysm angle (angles between the neck and maximum height),
were also calculated [24]. Irregular shape was defined based
on the presence of small bleb(s), bi- or multi-lobular, or pro-
truding bulge(s) from the IA fundus. The relative location of
aneurysm to parent vessel was dichotomized as the sidewall
type or bifurcation type. Detailed descriptions of all features
are provided in the Supplementary Materials.
Machine Learning Models
Three binary classification ML models, RF, SVM, and feed-
forward ANN models, were built and applied to assess IA
stability. All of these models are widely used and have shown
good performance in clinical classification and outcome pre-
diction studies. The models are described in detail in the
Supplementary Materials.
The whole dataset was randomly divided into two subsets:
a training set (1656 IAs, 80%) and a test set (411 IAs, 20%).
The ratio of stable IAs to unstable IAs was maintained across
both the two subsets. The training set was used to process the
input features and generate the prediction models with a 10-
fold cross-validation and select the optimal model and hyper-
parameters. Additional holdout as test set was set aside and
not used during the model training phase. The test set was later
to estimate the performance of each model with new data in a
generalizable way. The overall accuracy, sensitivity, and spec-
ificity of each model were calculated accordingly. This pro-
cess is outlined in Fig. 1.
Statistical Logistic Regression Model
LR is a classic statistical regression model that is used to select
significantly important variables and predict outcome proba-
bility. The variables used for the binary LR model were ini-
tially selected and evaluated by the univariate logistic regres-
sion. Then, a classic LR model was constructed with these
variables and used to assess IA stability.
PHASES Score System
We aim to compare the performances of ML models with the
previously reported PHASES score, which is based on six
readily accessible risk factors, to predict the risk of rupture
for unruptured IAs. Here, PHASES scores less or equal to 5
were considered as stable IAs with low rupture risk, which
correlates with a 5-year risk of IA rupture of 1.3% [10].
Statistical Analysis
Statistical analysis was performed on the clinical and morpho-
logical variables to identify those that were significantly dif-
ferent between the stable IA and unstable IA groups. For con-
tinuous variables, the differences between the groups were
tested using the Student t test or Mann-Whitney U test. For
categorical variables, a chi-square test was used to evaluate the
differences between groups. Receiver operating characteristic
(ROC) curves were generated, and the area under the curve
(AUC) values were calculated to compare the efficacies of the
different models. Since the aim of our study was to validate
the superiority of ML models relative to previous discrimina-
tory models, the worst ML model was selected for comparison
with the statistical LR model and the PHASES score. The
ROC curves between different methods were compared by
the asymptotically exact method described by Delong et al.
[26]. Statistical significance was defined as P < 0.05. The R
software was used for all statistical analyses (version 3.6.1, R
Foundation for Statistical Computing, Vienna, Austria).
Results
Demographic and Clinical Features
As shown in Figs. 1, 282 patients were excluded from this
analysis because of age < 18 (n = 13); traumatic, infectious,
fusiform, or dissecting IAs (n = 139); uncertain SAH history
(n = 24); missing data (n = 17); and poor radiological quality
 Transl. Stroke Res.

(n = 99). A total of 1897 patients with 2067 IAs were includ-
ed: the mean age was 55.30 ± 10.32 years. A total of 1262
(66.5%) were female. Among the IAs, 1539 were stable (mean
age 55.46 ± 9.81) and 528 were unstable (mean age 54.83 ±
11.67), of which 449 ruptured within 1 month and 79 grew in
imaging follow-up (median 15.6 months; range 5–
39 months).
The demographic and clinical features are summarized in
Table 1: female, smoking, drinking, hypertension, coronary
heart disease, presence of symptoms, and positive family his-
tory were more common among those with unstable IAs
(P < 0.05). IA stability was significantly associated with loca-
tion: the portion of unstable IAs located in the AComA and
PComA was significantly higher than that in the groups of
stable IAs (29.7% vs. 10.0%, respectively, in the AComA;
13.8% vs. 9.9%, respectively, in the PComA, P < 0.001). In
contrast, stable IAs were more prevalent in the ICA than un-
stable IAs (54.8% vs 33.9%, respectively). Multiple aneu-
rysms were more common in the stable group than in the
unstable group (P < 0.001). Considering the relative location
of IAs, 71.2% (374 of 528) of unstable IAs were bifurcation
Table 1 Demographic and clinical features between stable and unstable
IAs
Stable (n = 1539) Unstable P value
(n = 528)
type, implying that the bifurcation-type IAs are less stable
than sidewall-type IAs (P < 0.001).
Aneurysm Morphological Features
The morphological features of the stable and unstable IAs are
summarized in Table 2. Aneurysms were categorized based
on maximum size: 56 were categorized as tiny (< 3 mm), 1299
as small (< 7 mm), 462 as medium (7–10 mm), and 250 as
large (> 10 mm). The mean size of unstable IAs was 6.80 ±
2.76 mm, which was not significantly different from that of
the stable IAs (6.71 ± 3.26 mm, P = 0.548), implying the sim-
ilar size of those unstable and stable IAs. Among other size
indices, significant differences were found in the aneurysm
width, neck diameter, transverse diameter, and volume be-
tween the stable and unstable groups (P < 0.05); the discrep-
ancy in size indices demonstrated that the differences between
two groups were not mainly focused on aneurysm size dimen-
sion but on the aneurysm shape variances.
The shape indices, AR, SR, UI, NSI, and height-width ratio
were significantly higher in unstable IAs than in stable IAs
(P < 0.001), indicating that unstable IAs were often in abnor-
mal shape than their stable counterpart. Regard to regularity,
30.1% (159 of 528) of all unstable IAs exhibited irregular
shapes, which is higher than the portion of 11.9% (183 of

Age (mean) year 55.46 ± 9.81 54.83 ± 11.67 0.229 Table 2 Aneurysm morphological features between stable and unstable
Gender (female) 1055 (68.6%) 321 (60.8%) 0.001* IAs
Hypertension 758 (49.3%) 324 (61.4%) < 0.001* Stable (n = 1539) Unstable (n = 528) P value
Hyperlipidemia 144 (9.4%) 61 (11.6%) 0.152
Diabetes mellitus 159 (10.3%) 55 (10.4%) 0.934 Max diameter 6.71 ± 3.26 6.80 ± 2.76 0.548
Coronary heart disease 93 (6%) 53 (10.4%) 0.003* Maximum height 4.78 ± 2.56 5.02 ± 2.22 0.055
Smoker 307 (19.9%) 175 (33.1%) < 0.001* Perpendicular height 4.09 ± 2.33 4.14 ± 1.85 0.658
Drinker 292 (19.0%) 163 (30.9%) < 0.001* Neck diameter 5.52 ± 2.29 5.02 ± 1.99 < 0.001*
Presence of symptom 784 (50.9%) 434 (82.2%) < 0.001* Aneurysm width 5.41 ± 2.67 5.05 ± 2.16 0.001*
Family history 110 (7.1%) 63 (11.9%) 0.001* Transverse diameter 5.65 ± 2.77 5.34 ± 2.34 0.005*
Multiplicity 592 (38.5%) 137 (25.9%) < 0.001* Aneurysm volume 0.11 ± 0.28 0.08 ± 0.17 0.017*
Location < 0.001* Aneurysm angle 52.72 ± 20.87 49.70 ± 20.37 0.004*
ICA 843 (54.8%) 173 (33.9%) Flow angle 99.39 ± 32.76 113.10 ± 29.36 < 0.001*
MCA 189 (12.3%) 62 (11.7%) AR 0.78 ± 0.41 0.89 ± 0.41 < 0.001*
ACA 64 (4.2%) 21 (4.0%) SR 1.68 ± 1.07 2.02 ± 1.04 < 0.001*
AComA 154 (10.0%) 157 (29.7%) UI 0.15 ± 0.06 0.18 ± 0.74 < 0.001*
Posterior circulation 136 (8.8%) 36 (6.8%) NSI 0.28 ± 0.11 0.31 ± 0.10 < 0.001*
PComA 153 (9.9%) 73 (13.8%) VNR 3.11 ± 4.12 3.11 ± 3.71 0.995
Sidewall/bifurcation < 0.001* Height-width ratio 0.89 ± 0.21 1.01 ± 0.26 < 0.001*
Sidewall 970 (63%) 154 (28.8%) Bottleneck factor 0.88 ± 0.38 0.95 ± 0.33 0.964
Bifurcation 569 (37%) 374 (71.2%) Vessel diameter 3.21 ± 0.85 2.72 ± 0.78 < 0.001*
Irregularity 183 (11.9%) 159 (30.1%) < 0.001*
*P < 0.05
ICA, internal carotid artery; MCA, middle cerebral artery; ACA, anterior *P < 0.05
cerebral artery; AComA, anterior communicating artery; PComA, poste- AR, aspect ratio; SR, size ratio; UI, undulation index; NSI, nonspherical
rior communicating artery index; VNR, volume-to-neck ratio
Transl. Stroke Res.
1539) in the stable IAs, and this difference was statistically
significant (P < 0.001).
Features related to the parent vessel, including the aneu-
rysm angle and flow angle, also differed between the stable
and unstable IAs (P = 0.004 and P < 0.001, respectively).
Furthermore, compared with stable IAs, unstable IAs tended
to have smaller parent vessel size (P < 0.05), which is consis-
tent with the observed differences in the IA location between
the stable and unstable IA groups.
Comparison of Discriminatory Models in the Training
Set
The ROC curve and AUC value of each model in the
training set are shown in Fig. 2. The ANN model
showed the best performance with an AUC value of
0.851 (95% CI 0.828–0.873). The AUC values of the
RF model and the SVM model were 0.832 (95% CI
0.809–0.855) and 0.840 (95% CI 0.817–0.863). The
AUC value of the LR was 0.811 (95% CI 0.788–
Fig. 2 The receiver operating characteristic curve of the machine learning
models, classic logistic regression model, and PHASES score system in
the training set. The area under curve value was provided at the bottom.
The area under curve (AUC) is displayed as bar graph in the training set.
0.833). PHASES score had the worst performance with
an AUC value of 0.615 (95% CI 0.586–0.644). Among
the three ML models, the RF model had the worst per-
formance; however, the AUC value of the RF model
was still significantly higher than those of the LR mod-
el and the PHASES score (P = 0.045 and P < 0.001, re-
spectively), implying that the ML models provided bet-
ter performance than LR model and the PHASES score
in the training set.
The accuracy metrics: sensitivity, specificity, and accuracy
of each model are listed in Table 3. Although the specificities
of ML models and classic statistical model show no evident
difference (in the range of 88.9–95.4%), the sensitivity (in the
range of 50.8–56.7% for the ML models compared with
37.2% for LR) and accuracy (in the range of 82.6–85.5% for
the ML models compared with 75.7% for LR) were much
higher in ML models (up to about 10–20%), These findings
in accuracy metrics further demonstrate that the ML models
displayed better performance than the statistical method in IA
stability assessment.
There were significant differences between machine learning and
PHASES score, statistical model and PHASES score, and machine
learning and statistical model.
 Transl. Stroke Res.

Table 3 Performance of each models in the training set The sensitivity, specificity, and accuracy of each models in
Sensitivity Specificity Accuracy AUC the test set are listed in Table 4. Similar to the finding in the
training set, the specificity was similar between the ML
Machine learning models and the classic statistical model (ranging from 88.3–
RF 56.7% 95.4% 85.5% 0.832 (0.809–0.859) 92.9%). However, the sensitivity (ranging from 51.5–61.2%
SVM 50.8% 93.8% 82.6% 0.840 (0.817–0.863) for the ML models compared with 33.9% for the LR model)
ANN 55.0% 95.3% 85.0% 0.851 (0.829–0.873) and accuracy (ranging from 81.5–82.4% for the ML models
Classic statistical compared with 74.5% for the LR model) were much higher
Classic LR 37.2% 88.9% 75.7% 0.811 (0.788–0.833) with the ML models.
Clinical score
PHASES 13.8% 88.7% 69.6% 0.615 (0.586–0.644) Important Features to Aneurysm Stability
RF, random forest; SVM, linear support vector machine; ANN, artificial
neural network; LR, logistic regression The importance of each variable is calculated for the RF and
SVM models. In addition, the classical LR model is common-
Comparison of Discriminatory Models in the Test Set ly used as a statistical method for variable selection. Thus, in
accordance with the composition of the PHASES score, the
The models were further tested in a separate test set to evaluate six most important IA stability-related features were selected
their generalizability and broad applicability. The ROC curve and shown in Table 5. The UI, height-width ratio, and pres-
and AUC values of each model are shown in Fig. 3. The AUC ence of symptoms were among the most important features in
values of each ML models are as follows: RF 0.850 (95% CI both the classic statistical model and the ML models. The
0.8061–0.893), SVM 0.858 (95% CI 0.816–0.900), ANN location, irregularity, and sidewall/bifurcation type were also
0.867 (95% CI 0.828–0.906). As in the training set, the RF the most important in two of these three discriminatory
model had the worst performance among the three ML models. Notably, among the risk factors that are included in
models. However, similar to the finding in the training set, the PHASES score, only the location was identified as the
the AUC of the RF model was still better than those of the most important features in the other discriminatory models.
LR and the PHASES score: 0.818 (95% CI 0.776–0.860) and
0.589 (95% CI 0.527–0.651), respectively; these differences
were statistically significant (P < 0.001 and P = 0.038, respec- Discussion
tively). These findings further confirm the superior perfor-
mance of the ML models and demonstrate that this finding In our study, we developed multiple ML models for IA sta-
is generalizable beyond the training set. bility assessment based on clinical features and morphological

Fig. 3 The receiver operating characteristic curve of the machine learning logistic regression, NN feed-forward neural network, SVM support
models, classic logistic regression model, and PHASES score system in vector machine, and RF random forest. There were significant
the test set. The area under curve value were provided as the bar graph. differences between machine learning and PHASES score, statistical
PHASES indicates PHASES score system, LR indicates statistical model and PHASES score, and machine learning and statistical model
Transl. Stroke Res.

Table 4 Performance of each models in the test set Although the specificity and accuracy of the PHASES appear
Sensitivity Specificity Accuracy AUC to be moderate (in the range of approximately 70–90%), the
sensitivities were only 13.8% and 9.7% in the training set and
Machine learning the test set, respectively. This gap indicates that the PHASES
RF 54.4% 90.9% 81.8% 0.850 (0.806–0.893) score tends to conservatively misclassify IAs as having low
SVM 61.2% 88.3% 81.5% 0.858 (0.816–0.900) rupture risk and cannot reliably recognize differences between
ANN 51.5% 92.9% 82.4% 0.867 (0.838–0.906) stable and unstable IAs. Considering the potential catastrophic
Classic statistical outcome of misdiagnosing unstable IAs with high rupture risk
Classic LR 33.9% 88.3% 74.5% 0.818 (0.776–0.860) as stable IAs with low rupture risk, it is difficult to consider the
Clinical score PHASES score as a useful tool in daily clinical practice.
PHASES 9.7% 95.4% 73.0% 0.589 (0.527–0.651) Previously, Foreman et al. and Pagiola et al. both reported that
the majority of a SAH patients present with low PHASES
RF, random forest; SVM, linear support vector machine; ANN, artificial scores (≤ 5), which represent stable IAs with relatively low
neural network; LR, logistic regression
rupture risk [28, 29]. Thus, the poor performance of the
PHASES score in our IA cohort is consistent with the reports
features. Their performances were compared with those of a
of inconsistencies between the PHASES score and the actual
statistical LR model and the PHASES score. Result showed
rupture outcomes in numerous reports. In fact, to the best of
that all three ML models displayed good discriminatory abil-
our knowledge, this is the first study to evaluate the efficacy of
ity, outperforming both the statistical LR model and the
the PHASES score based on a large sample of IAs (2067) and,
PHASES score in discrimination between unstable and stable
at the same time, compare its discriminatory ability with those
IAs. This result suggests that ML models have great potential
of ML models.
in clinical IA stability assessment.
The classic statistical model had an AUC value of 0.818
ML represents a set of novel and efficient approaches en-
with a sensitivity of 33.9%, a specificity of 88.3%, and an
abling classification and outcome prediction to support clini-
accuracy of 74.5% in the final test set. As one of the most
cal decision-making and individualized management. The
classic and practical models in statistics, LR exhibited good
main advantage of ML over traditional statistical method is
performance in IA stability assessment. This result was con-
the ability to simultaneously process massive numbers of var-
sistent with previous reports: Prestigiacomo et al. developed a
iables and the capability to model non-linear relationships
binary LR model based on morphological features, which
[27]. Conventional statistical methods, like LR, however, are
yield a sensitivity, specificity, and accuracy of 83%, 78%,
limited in that they can only capture linear relationships.
and 80%, respectively [30]. Xiang et al. used multivariate
Similarly, most clinical scoring systems, including the
LR model that included three important variables (size ratio,
PHASES scoring system, were derived using the same statis-
average wall shear stress, and oscillatory shear index) and
tical method to select important factors; thus, they are also
found that its AUC value was 0.89 for discriminating IA status
restricted to linear relationships. Thus, theoretically, the ML
[31]. In this study, although the classic LR model achieved a
models should outperform conventional statistical methods
pretty good performance, it was still secondary to the ML
and the PHASES score system. Indeed, the results showed
models, especially in terms of the accuracy metrics.
that the ML models had good discriminatory ability and better
We compared three different types of ML models and val-
performance compared with the classic LR model and the
idated their performance in the test set. They had comparable
PHASES score in both the training set and the final test set.
discriminatory ability with AUC values between 0.850 and
0.867, with that of the ANN model being the highest. The
Table 5 Important features in models accuracy values were similar, with that of the ANN model
Classic statistical model Machine learning models being the highest. For all ML models, the specificity values
in the test set were higher than their sensitivity values (88.3–
LR SVM RF 92.9% and 51.5–61.2%, respectively); this may be explained
by the differences in proportion between unstable and stable
1 UI Symptom Symptom
IAs (unstable:stable = 1:3); less unstable cases caused less
2 Height-width ratio Height-width ratio Height-width ratio
efficient discriminatory ability in all models. Furthermore,
3 Symptom Location UI
the sensitivities of the three ML models were notably higher
4 Sidewall/bifurcation Vessel size Sidewall/bifurcation
than that of the LR model in both the training set and the test
5 Irregularity Flow angle Width
set, demonstrating another advantage of ML models over the
6 Location UI Irregularity
classic LR model. It is particularly important for a discrimina-
RF, random forest; SVM, linear support vector machine; LR, logistic tory model to have higher sensitivity given the devastating
regression; UI, undulation index

and potentially fatal consequences of IA rupture if unstable
IAs with high rupture risk are falsely classified as a low risk.
Though the ANN model had the best performance among
the three ML models, the other two models, the RF and SVM
models, offered an advantage in the ability to calculate the
relative importance of each feature with respect to the model
outcome, making the algorithm easy to understand and ex-
plain in clinical settings. The most important features contrib-
uting to IA stability selected by using the ML models were
largely consistent with those selected by using the classic LR
model. Both the ML and LR models recognized UI, height-
width ratio, presence of symptoms, sidewall/bifurcation type,
irregularity, and location as the most stability-related features.
This finding is consistent with the findings in previous studies
[20–22, 24]. Interestingly, among the top six most important
risk features comprised equal numbers of clinical features and
morphological features, suggesting both types of features are
critical in IA stability assessment. Thus, disregarding either
aspect would inhibit the ability to conduct a comprehensive
and informed assessment. Of these top six most important
features, only the location was included in the PHASES score,
which did not include any morphological features; this may
explain the mediocre discriminatory ability of the PHASES
score in our study. Notably, three shape indices (the UI,
height-width ratio, and irregularity) were identified as impor-
tant contributors to IA stability in both the ML and classic LR
model, indicating that abnormal shape is a key distinction
between stable and unstable IAs, rather than size indices.
AI technology offer a variety of strengths: First, AI algo-
rithms can be used to learn patterns from massive, complex
datasets and generate helpful predictive outputs. Second, AI
algorithms demonstrate striking performance, including high
accuracy and speed. AI technologies are well adapted to rapid,
high-volume data processing, enabling tailored and specific
management based on each patient’s characteristics. Third,
algorithms like RF and SVM can allow for outcome interpre-
tation by providing the relative importance of each feature,
allowing clinicians to more directly understand the algo-
rithm’s internal mechanisms. The application of AI may sub-
stantially improve clinical safety and effectiveness. In partic-
ular, AI has been initially applied to IA diagnosis, risk strati-
fication, and outcome prediction [19, 32, 33]. As the technol-
ogy matures, AI algorithms with different functions can be
integrated into decision-making systems to aid clinicians. In
the future, such systems may be introduced into clinical
workflows as supportive and complementary tools for making
day-to-day clinical judgements. In this capacity, they are ex-
pected to improve clinicians’ effectiveness and help devise
accurate and individualized treatment plans for patients.
Based on the results from our study, we suggest that the
application of ML has excellent potential in IA stability as-
sessment and could facilitate better clinical management of
IAs in the future.
Transl. Stroke Res.
Limitations
The present study has several limitations. First, our major
limitation is the retrospective nature of this study; stable IAs
at diagnosis did not warrant no rupture risk in the future; a
further prospective cohort would be ideal; however, because
evidence about rupture risk factors already existed, it would
have been unethical to leave UIAs with seemingly high risk
rupture without treatment. Second, although previous study
reported that IA morphology does not change dramatically
[34], the event of rupture itself may have affected the mor-
phology of IAs, this may generate a possible bias on our result
[35]. Third, all the patients with IAs came from one single
center; further validation needs to be done in other multiple
centers. Also, we did not include any aneurysm wall informa-
tion in this study. Further work should be done to strengthen
and verify our model by involving more risk-related parame-
ters. Finally, the stable and unstable IA cohort number is not
perfectly balanced; this may affect the generalization of these
ML models. In the future, the validation of these model per-
formance will be done in a prospective, multicenter, longitu-
dinal data set. Also, the development of a clinically used
decision-making assistant software in real clinical setting is
our aim.
Conclusions
ML models outperformed the statistical LR method and the
PHASES score in IA stability assessment. The ML models
demonstrated great potential in aiding the clinical decision-
making process, and their future application in clinical prac-
tice may provide individualized and optimal management for
patients with IAs.
Funding Information This work was supported by National Key
Research and Development Plan of China (grant number:
2016YFC1300800), the National Natural Science Foundation of China
(grant numbers: 81801156, 81801158, and 81671139), the Special
Research Project for Capital Health Development (grant number: 2018-
4-1077), and Beijing Hospitals Authority Youth Programme (code:
QML20190503).
Compliance with Ethical Standards
Conflict of Interest The authors declare that they have no conflict of
interest.
Ethical Approval All procedures performed in this study were in accor-
dance with the ethical standards of the Institutional Review Board of
Beijing Tiantan Hospital and with the 1964 Helsinki Declaration and its
later amendments or comparable ethical standards.
Informed Consent Informed consent was obtained from all individual
participants included in the study.
Transl. Stroke Res.
References
1. Vlak MH, Algra A, Brandenburg R, Rinkel GJ. Prevalence of
unruptured intracranial aneurysms, with emphasis on sex, age, co-
morbidity, country, and time period: a systematic review and meta-
analysis. Lancet Neurol. 2011;10(7):626–36.
2. Brown RD Jr, Broderick JP. Unruptured intracranial aneurysms:
epidemiology, natural history, management options, and familial
screening. Lancet Neurol. 2014;13(4):393–404.
3. Thompson BG, Brown RD Jr, Amin-Hanjani S, et al. Guidelines
for the management of patients with unruptured intracranial aneu-
rysms: a guideline for healthcare professionals from the American
Heart Association/American Stroke Association. Stroke.
2015;46(8):2368–400.
4. Korja M, Lehto H, Juvela S, Kaprio J. Incidence of subarachnoid
hemorrhage is decreasing together with decreasing smoking rates.
Neurology. 2016;87(11):1118–23.
5. Vlak MH, Rinkel GJ, Greebe P, Algra A. Risk of rupture of an
intracranial aneurysm based on patient characteristics: a case-
control study. Stroke. 2013;44(5):1256–9.
6. Kashiwazaki D, Kuroda S. Size ratio can highly predict rupture risk
in intracranial small (<5 mm) aneurysms. Stroke. 2013;44(8):2169–
73.
7. Ujiie H, Tamano Y, Sasaki K, et al. Is the aspect ratio a reliable
index for predicting the rupture of a saccular aneurysm?
Neurosurgery. 2001;48(3):495–502 discussion 502-3.
8. Wong SC, Nawawi O, Ramli N, Abd Kadir KA. Benefits of 3D
rotational DSA compared with 2D DSA in the evaluation of intra-
cranial aneurysm. Acad Radiol. 2012;19(6):701–7.
9. Brinjikji W, Cloft H, Lanzino G, Kallmes DF. Comparison of 2D
digital subtraction angiography and 3D rotational angiography in
the evaluation of dome-to-neck ratio. AJNR Am J Neuroradiol.
2009;30(4):831–4.
10. Greving JP, Wermer MJ, Brown RD Jr, et al. Development of the
PHASES score for prediction of risk of rupture of intracranial an-
eurysms: a pooled analysis of six prospective cohort studies. Lancet
Neurol. 2014;13(1):59–66.
11. Neyazi B, Sandalcioglu IE, Maslehaty H. Evaluation of the risk of
rupture of intracranial aneurysms in patients with aneurysmal sub-
arachnoid hemorrhage according to the PHASES score. Neurosurg
Rev. 2019;42(2):489–92.
12. Brusko GD, Kolcun JPG, Wang MY. Machine-learning models:
the future of predictive analytics in neurosurgery. Neurosurgery.
2018;83(1):E3–e4.
13. Senders JT, Arnaout O, Karhade AV, et al. Natural and artificial
intelligence in neurosurgery: a systematic review. Neurosurgery.
2018;83(2):181–92.
14. Hostettler IC, Muroi C, Richter JK, Schmid J, Neidert MC, Seule
M, et al. Decision tree analysis in subarachnoid hemorrhage: pre-
diction of outcome parameters during the course of aneurysmal
subarachnoid hemorrhage using decision tree analysis. J
Neurosurg. 2018;129(6):1499–510.
15. Rubbert C, Patil KR, Beseoglu K, Mathys C, May R, Kaschner
MG, et al. Prediction of outcome after aneurysmal subarachnoid
haemorrhage using data from patient admission. Eur Radiol.
2018;28(12):4949–58.
16. Heo, J., J.G. Yoon, H. Park, et al., Machine learning-based model
for prediction of outcomes in acute stroke. Stroke, 2019. 50(5): p.
1263–1265.
17. Liu J, Chen Y, Lan L, Lin B, Chen W, Wang M, et al. Prediction of
rupture risk in anterior communicating artery aneurysms with a
feed-forward artificial neural network. Eur Radiol. 2018;28(8):
3268–75.
18. Kim HC, Rhim JK, Ahn JH, et al. Machine learning application for
rupture risk assessment in small-sized intracranial aneurysm. J Clin
Med. 2019;8(5).
19. Liu Q, Jiang P, Jiang Y, Ge H, Li S, Jin H, et al. Prediction of
aneurysm stability using a machine learning model based on
PyRadiomics-derived morphological features. Stroke. 2019;50(9):
2314–21.
20. Zhang Y, Jing L, Liu J, Li C, Fan J, Wang S, et al. Clinical, mor-
phological, and hemodynamic independent characteristic factors
for rupture of posterior communicating artery aneurysms. J
Neurointerv Surg. 2016;8(8):808–12.
21. Liu J, Xiang J, Zhang Y, Wang Y, Li H, Meng H, et al.
Morphologic and hemodynamic analysis of paraclinoid aneurysms:
ruptured versus unruptured. J Neurointerv Surg. 2014;6(9):658–63.
22. Hoh BL, Sistrom CL, Firment CS, et al. Bottleneck factor and
height-width ratio: association with ruptured aneurysms in patients
with multiple cerebral aneurysms. Neurosurgery. 2007;61(4):716–
22 discussion 722-3.
23. Ryu C-W, Kwon OK, Koh JS, Kim EJ. Analysis of aneurysm
rupture in relation to the geometric indices: aspect ratio, volume,
and volume-to-neck ratio. Neuroradiology. 2010;53(11):883–9.
24. Dhar S, Tremmel M, Mocco J, et al. Morphology parameters for
intracranial aneurysm rupture risk assessment. Neurosurgery.
2008;63(2):185–96 discussion 196-7.
25. Raghavan ML, Ma B, Harbaugh RE. Quantified aneurysm shape
and rupture risk. J Neurosurg. 2005;102(2):355–62.
26. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the
areas under two or more correlated receiver operating characteristic
curves: a nonparametric approach. Biometrics. 1988;44(3):837–45.
27. Obermeyer Z, Emanuel EJ. Predicting the future - big data, machine
learning, and clinical medicine. N Engl J Med. 2016;375(13):1216–
9.
28. Foreman PM, Hendrix P, Harrigan MR, Fisher WS 3rd, Vyas NA,
Lipsky RH, et al. PHASES score applied to a prospective cohort of
aneurysmal subarachnoid hemorrhage patients. J Clin Neurosci.
2018;53:69–73.
29. Pagiola I, Mihalea C, Caroff J, et al. The PHASES score: to treat or
not to treat? Retrospective evaluation of the risk of rupture of intra-
cranial aneurysms in patients with aneurysmal subarachnoid hem-
orrhage. J Neuroradiol. 2019.
30. Prestigiacomo CJ, He W, Catrambone J, Chung S, Kasper L,
Pasupuleti L, et al. Predicting aneurysm rupture probabilities
through the application of a computed tomography angiography-
derived binary logistic regression model. J Neurosurg.
2009;110(1):1–6.
31. Xiang J, Natarajan SK, Tremmel M, Ma D, Mocco J, Hopkins LN,
et al. Hemodynamic-morphologic discriminants for intracranial an-
eurysm rupture. Stroke. 2011;42(1):144–52.
32. Park A, Chute C, Rajpurkar P, Lou J, Ball RL, Shpanskaya K, et al.
Deep learning–assisted diagnosis of cerebral aneurysms using the
HeadXNet model. JAMA Netw Open. 2019;2(6):e195600.
33. Paliwal N, Jaiswal P, Tutino VM, Shallwani H, Davies JM,
Siddiqui AH, et al. Outcome prediction of intracranial aneurysm
treatment by flow diverters using machine learning. Neurosurg
Focus. 2018;45(5):E7.
34. Rahman M, Ogilvy CS, Zipfel GJ, et al. Unruptured cerebral aneu-
rysms do not shrink when they rupture: multicenter collaborative
aneurysm study group. Neurosurgery. 2011;68(1):155–60 discus-
sion 160-1.
35. Skodvin TO, Johnsen LH, Gjertsen O, Isaksen JG, Sorteberg A.
Cerebral aneurysm morphology before and after rupture: nation-
wide case series of 29 aneurysms. Stroke. 2017;48(4):880–6.
Publisher’s Note Springer Nature remains neutral with regard to jurisdic-
tional claims in published maps and institutional affiliations.