International Journal of Medical Informatics 145 (2021) 104326

Contents lists available at ScienceDirect

International Journal of Medical Informatics

journal homepage: www.elsevier.com/locate/ijmedinf

Machine learning-based models to support decision-making in emergency

department triage for patients with suspected cardiovascular disease
Huilin Jiang a, 1, Haifeng Mao a, 1, Huimin Lu a, Peiyi Lin a, Wei Garry b, Huijing Lu a,
Guangqian Yang a, Timothy H. Rainer c, Xiaohui Chen a, *
a
Emergency Department, The Second Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
b
Goodwill Hessian Health Technology Co., Ltd, Beijing, China
c
Accident and Emergency Medicine Academic Unit, Chinese University of Hong Kong, Prince of Wales Hospital, Hong Kong, China

A R T I C L E I N F O A B S T R A C T

Keywords: Background: Accurate differentiation and prioritization in emergency department (ED) triage is important to
Triage identify high-risk patients and to efficiently allocate of finite resources. Using data available from patients with
Emergency department suspected cardiovascular disease presenting at ED triage, this study aimed to train and compare the performance
Decision-making
of four common machine learning models to assist in decision making of triage levels.
Machine learning
Methods: This cross-sectional study in the second Affiliated Hospital of Guangzhou Medical University was
Cardiovascular disease
High-risk conducted from August 2015 to December 2018 inclusive. Demographic information, vital signs, blood glucose,
and other available triage scores were collected. Four machine learning models – multinomial logistic regression
(multinomial LR), eXtreme gradient boosting (XGBoost), random forest (RF) and gradient-boosted decision tree
(GBDT) – were compared. For each model, 80 % of the data set was used for training and 20 % was used to test
the models. The area under the receiver operating characteristic curve (AUC), accuracy and macro- F1 were
calculated for each model.
Results: In 17,661 patients presenting with suspected cardiovascular disease, the distribution of triage of level 1,
level 2, level 3 and level 4 were 1.3 %, 18.6 %, 76.5 %, and 3.6 % respectively. The AUCs were: XGBoost (0.937),
GBDT (0.921), RF (0.919) and multinomial LR (0.908). Based on feature importance generated by XGBoost,
blood pressure, pulse rate, oxygen saturation, and age were the most significant variables for making decisions at
triage.
Conclusion: Four machine learning models had good discriminative ability of triage. XGBoost demonstrated a
slight advantage over other models. These models could be used for differential triage of low-risk patients and
high-risk patients as a strategy to improve efficiency and allocation of finite resources.

1. Introduction
Cardiovascular disease (CVD) is the leading cause of mortality and
morbidity worldwide accounting for approximately one third of all
deaths [1–3]. It is an acute time-sensitive condition, so it is important to
identify high-risk patients at an early stage. The number of patients with
CVD visiting emergency departments (ED) in China is enormous [4]. For
most of these patients, the first emergency care they encounter is at
triage. Triage, a specialized role typically performed by registered
nurses, is sorting patients by acuity to prioritize them for full evaluation
[5].
The Chinese Emergency Triage Scale (CETS) was originally devel­
oped in 2011 and further improved in 2018 [6,7]. The scale is now used
in many hospitals in Mainland China based on the Chinese policy. It has

Abbreviations: AHGZMU, Affiliated Hospital of Guangzhou Medical University; AUC, area under the ROC curve; CETS, Chinese Emergency Triage Scale; CTAS,
Canadian Triage and Acuity Scale; CVD, cardiovascular disease; ED, emergency department; ESI, Emergency Severity Index; GBDT, gradient-boosted decision tree;
GCS, Glasgow Coma Scale; IQR, interquartile range; LR, logistic regression; MTS, Manchester Triage System; PHI, Prehospital Index; RF, random forest; ROC, receiver
operating characteristic; VAS, Visual Analogue Scale; XGBoost, eXtreme gradient boosting.
* Corresponding author.
E-mail addresses: lifisher@126.com (H. Jiang), maomao2010x@163.com (H. Mao), 416905920@qq.com (H. Lu), linpeiyi@163.com (P. Lin), ganwei@
hessianhealth.com (W. Garry), 3025515@qq.com (H. Lu), 413116384@qq.com (G. Yang), thrainer@cuhk.edu.hk (T.H. Rainer), cxhgz168paper@163.com (X. Chen).
1
These authors contributed equally to this work.

https://doi.org/10.1016/j.ijmedinf.2020.104326
Received 23 August 2020; Received in revised form 16 October 2020; Accepted 30 October 2020
Available online 3 November 2020
1386-5056/© 2020 Elsevier B.V. All rights reserved.
H. Jiang et al.
been proved that the CETS is a reliable system for ED triage and can
promote rapid and effective triage in Mainland China [8].
The CETS classifies patients into four levels based on the assessment
of vital complaints and vital parameters [7,8]. Although most criteria
are objective indicators, the final triage level assigned to the patient is
influenced by the subjective judgment of the triage nurse, even if there
are strict requirements for the triage personnel’s ability and maintaining
regular education and training for them. Consequently, drawbacks such
as human dependency and ambiguity of judgment have been high­
lighted. These potential problems could worsen when the volume of
patients increases and information accumulates.
An algorithmic electronic triage (e-triage) model has been shown to
support decision-making and to improve patient risk management in the
ED [10–13]. Little e-triage research has been reported in Mainland
China. Machine learning models, which were developed based on the
electronic algorithm, have been reported to have superior ability for
prediction in different disease conditions [14,15]. E-triage essentially is
a rule-based expert system, which is difficult to apply to the clinical
practice because of the inherent complexity and uncertainty of clinical
information [16]. However, machine learning models can learn patterns
from large, complex and heterogeneous data and perform practical ac­
curate predictions [17]. Additionally, machine learning used for
decision-support systems can potentially alleviate the increased cogni­
tive load of medical professionals, allowing them to focus more on
clinical care [18].
In this context, we used ED visit data to develop four machine
learning models by using routinely available triage data to accurately
predict triage levels for patients with suspected CVD, and to compare the
predictive performance of each model.
2. Methods
2.1. Study design
A cross-sectional study of ED patients with suspected CVD was
conducted from August 2015 to December 2018. Ethical approval was
obtained from the Clinical Research Ethics Committee of Guangzhou
Medical University.
2.2. Study setting
This study was conducted in the ED of the second Affiliated Hospital
of Guangzhou Medical University (AHGZMU), which is a teaching hos­
pital with 1500 beds. The ED receives more than 150,000 new patients
per year and serves a population of approximately 1.56 million people in
the Hai Zhu district, Guangzhou.
2.3. Inclusion and exclusion criteria
ED admissions of patients aged ≥14 years old were identified from
the ED information system of the second Affiliated Hospital of
Guangzhou Medical University and were included if they had any one of
these chief complaints or diagnosis: palpitations, chest distress, chest
pain, hypertension, coronary heart disease, coronary insufficiency,
angina, myocarditis, acute heart failure, cardiac insufficiency, cardiac
failure, myocardial infarction, arrhythmia, cardiac arrest, sudden death
and shortness of breath. The chief complaints and diagnosis were also
obtained from the ED information system. Visits were excluded if pa­
tients were <14 years old or where more than a half variables were
missing.
2.4. Predictors
Predictor data obtained from the ED information system of the sec­
ond Affiliated Hospital of Guangzhou Medical University from August
2015 to December 2018 included age, gender, vital signs (temperature,
2
International Journal of Medical Informatics 145 (2021) 104326
pulse rate, systolic and diastolic blood pressure, respiratory rate and
oxygen saturation), blood glucose, Visual Analogue Scale (VAS) pain
score, Glasgow Coma Scale (GCS) score, and Prehospital Index (PHI),
which were routinely available at ED triage setting.
2.5. Outcome
The outcome was different triage levels of CETS from 1 to 4 recorded
by nurses who have more than 5 years working experience in the ED.
Level 1 is immediate, level 2 is emergent, level 3 is urgent, and level 4 is
semi-urgent or non-urgent.
2.6. Definitions
Triage was defined as a sorting process where patients were first
sorted by acuity in order to prioritize individuals requiring urgent
medical intervention when arriving at the ED [5,19]. In Mainland China,
the CETS is most widely used. In contrast to the most triage instruments,
e.g., Emergency Severity Index (ESI), Canadian Triage and Acuity Scale
(CTAS), and Manchester Triage System (MTS), which are focused on
complicated complaint-specific triage criteria, the CETS relies on mea­
surements of vital complaints and vital parameters [8,20]. Vital com­
plaints are the most critical symptoms derived from the Airway,
Breathing, Circulation, Disability, Exposure (ABCDE) assessment model
[7,8]. Vital parameters, including individual and comprehensive
vital-sign indicators, can provide an objective method of facilitating
rapid triage decision making [7–9]. The CETS is a four-level category
triage system, level 1 is immediate, level 2 is emergent, level 3 is urgent,
and level 4 is semi-urgent or non-urgent [7]. PHI is a triage-oriented
trauma severity scoring system comprising four components: systolic
blood pressure, pulse, respiratory status, and level of conscious, each
scored 0–5. A PHI of 0–3 indicated minor trauma, and a PHI of 4–20
signified major trauma [21]. PHI is a useful tool to screen preliminarily
for potential critical illness [22].
2.7. Statistical analyses
Descriptive analysis was constructed for both predictors and
outcome. Continuous variables were discretized and transformed into
categorical variables, then the missing data in each categorical variable
were binned into a separate category, and finally we performed one-hot
encoding for all categorical variables (e.g., temperature was divided into
4 categories: normal, low fever, high fever, and missing data, of which 4
types of feature values are [1,2,3,4], so one-hot encoding can be
expressed as [1000,0100,0010,0001]…)
In the training set (80 % random sample), we developed multinomial
logistic regression (multinomial LR) and other three machine learning
models to predict the probability of the 4-level outcome: (1) random
forest (RF), (2) eXtreme gradient boosting (XGBoost), and (3) gradient-
boosted decision tree (GBDT). Multinomial LR, which generalizes lo­
gistic regression to multiclass problems, is a model used to predict the
probability of different possible outcomes of dependent variables with
class distribution. XGBoost used a more regularized model formalization
to control over-fitting, which eliminate interference with outliers and
makes the model more stable and accurate. RF is an ensemble of decision
trees to reduce training variance and improve integration and general­
ization. GBDT is another ensemble approach estimated by gradient
descent.
In the test set (20 % random sample), we measured the prediction
performance of each model by computing: (1) accuracy (i.e., calculation
based on sensitivity, specificity, positive predictive value, and negative
predictive value), (2) AUC (i.e., the area under the receiver operating
characteristic [ROC] curve), (3) macro- F1 (i.e., the macro-averaged F1
-score used to assess the quality of problems with multiple binary labels
or multiple classes). To gain a deeper understanding of the contribution
of each predictor to machine learning models, we also calculated the
H. Jiang et al. International Journal of Medical Informatics 145 (2021) 104326

Fig. 1. Flowchart of ED visits with suspected CVD.

Abbreviations: ED, emergency department; CVD, cardiovascular disease.

analytic cohorts were generally similar (Appendix A). In the analytic

Table 1
cohort, the median age of the patients was 65 years old (IQR 51-78 years
Predictor variables and triage levels in 17,661 patient visits with suspected
old) and 52.56 % were women. Overall, the distribution of triage of level
cardiovascular disease presenting to the emergency department.
1, level 2, level 3 and level 4 was 1.3 %, 18.6 %, 76.5 %, and 3.6 %,
Variables Value respectively (Table 1).
Age, median (IQR), y 65 (51− 78)
Female, No. (%) 9282 (52.6)
3.2. Model performance
Vital signs, median (IQR)
Temperature, ℃ 36.9 (36.5− 37.4) The predictive performance of all models was shown in Fig. 2 and
Pulse rate, beats/min 88 (75− 102) Table 2. All four machine-learning models had a high discriminative
Systolic blood pressure, mmHg 145 (126− 168) ability, as AUC of XGBoost is the highest (0.937), following GBDT
Diastolic blood pressure, mmHg 83 (73− 94)
(0.921), RF (0.919) and multinomial LR (0.908). The accuracy metric
Respiratory rate, breaths/min 20 (18− 20)
Oxygen saturation, % 98 (97− 99) checks the proportion of correctly classified samples, and the macro- F1
Blood glucose, median (IQR), mmol/L 9.6 (7.4− 13.7) metric is the harmonic mean of precision and recall. In our models,
VAS pain score, median (IQR) 0 (0− 1.5) accuracy and macro- F1 of XGBoost (0.785 and 0.478 respectively) were
Glasgow Coma Scale score, median (IQR) 15 (15− 15) also moderately good, and of other models are multinomial LR (0.743
Prehospital Index, median (IQR) 0 (0− 0)
and 0.430 respectively) RF (0.745 and 0.513 respectively) and GBDT
Triage levels, No. (%)
(0.767 and 0.419 respectively). The performance discriminated by
level 1 221 (1.3) triage level of the four machine learning models is shown in Appendix B.
level 2 3288 (18.6)
level 3 13515 (76.5)
3.3. Variable importance
level 4 637 (3.6)

Abbreviations: VAS, Visual Analogue Scale; IQR, interquartile range.
variable importance in the XGBoost for each outcome. The variable
importance is a scaled measure to have a maximum value of 1.0. All
analyses were performed with R 3.5.3 and Python 3.6.7.
3. Results
3.1. Characteristics of study samples
There were 28,242 visits with suspected CVD recorded in the ED
information system during August 2015 and December 2018. Of these,
429 visits younger than 14 years old and 10,152 with more than a half
missing variables were excluded, leaving 17,661 visits in the analytic
cohort (Fig. 1). The characteristics of patients in the analytic and non-
The variables importance in outcome prediction generated by
XGBoost is presented in Fig. 3. Blood pressure, pulse rate, oxygen
saturation, and age were the most significant variables for making de­
cision of the triage.
4. Discussion
Using data available at triage, we developed four machine-learning
models (i.e., multinomial LR, RF, XGBoost, and GBDT) to make deci­
sion of the triage levels of patients with suspected CVD. Machine
learning models performed well achieving AUCs greater than 0.90 and
had moderately good capability to accurately classify patients with
suspected CVD into different triage levels. Among the four models,
XGBoost performed slightly well. The major goals of ED triage are to
accurately differentiate high-risk patients from less-urgent patients and

3
H. Jiang et al. International Journal of Medical Informatics 145 (2021) 104326

Fig. 2. Receiver operating characteristic (ROC) curves of multinomial LR, RF, XGBoost and GBDT in predicting triage of patients with suspected cardiovascular
disease on test data.
Abbreviations: LR, logistic regression; RF, random forest; XGBoost, eXtreme gradient boosting; GBDT, gradient-boosted decision tree.

variable importance was systolic blood pressure, pulse rate, and oxygen
Table 2
saturation, followed by age and diastolic blood pressure which were
Prediction ability of 4 machine learning models in patients with suspected
sensitive in CVD [24]. Being consistent with prior studies, systolic blood
cardiovascular disease in the test data.
pressure was one of the most significant risk factors in CVD, while
Model Accuracy AUC Macro-F1
gender had minimum contribution to the model [25–28]. It is generally
Multinomial logistic regression 0.743 0.908 0.430 assumed in medicine that the effects of the major risk factors on CVD
Random forest 0.745 0.919 0.513 outcome are similar in women and men [28–30]. Other variables (e.g.,
eXtreme gradient boosting 0.785 0.937 0.478
Gradient-boosted decision tree 0.767 0.921 0.419
temperature, GCS score) also did not have apparent contribution to the
model illustrating that they had less specific correlation with CVD.
Therefore, medical professionals should pay more attention on these
to efficiently allocate of finite ED resources. Other studies illustrated significant variables when treating patients with suspected CVD.
that machine-learning model (AUC = 0.91) had better ability to identify Since the models performed moderately well and were clinically
patients assigned MTS/ESI 3 who are at risk for ICU admission [23]. The practical, they may be of assistance to triage personnel as adjunct tools.
model presented in this study also had outstanding ability to discrimi­ A survey was clearly shown the demand of triage nurses, where the most
nate the current CETS. highly requested feature for a new tool being built was an automatic
Not only the performance of the models was good in this study, but severity calculator [31]. Due to the subjective judgement of triage and
also the clinically sensible of the models was practical in differentiating increasing information of patients, misclassification, over-triage, and
the triage levels of patients with suspected CVD. The descending order of under-triage are possible, even using ESI and other 5-level triage

Fig. 3. Variable importance of predictors in the eXtreme gradient boosting model. The variable importance is a scaled measure to have a maximum value of 1.0.
Abbreviations: VAS, Visual Analogue Scale; GCS, Glasgow Coma Scale; PHI, Prehospital Index.

4
H. Jiang et al. International Journal of Medical Informatics 145 (2021) 104326

Summary points
What was already known on the topic:

• Cardiovascular disease is the leading cause of mortality and morbidity, and since it is an acute time-sensitive disease, it is important to identify
high-risk patients at an early stage.
• The Chinese Emergency Triage Scale (CETS) is widely used in Mainland China, but drawbacks such as human dependency and ambiguity of
judgment have been highlighted.
• Machine learning algorithm can learn patterns from large, complex and heterogeneous data and has superior ability for prediction.

What this study added to our knowledge:

• Machine learning models can accurately identify the high-risk patients from the large proportion of non-urgent visits.
• The machine learning models using a small set of variables also perform well, and it is practical in ED triage setting owing to the limited
information and time pressure.

instruments [11,32]. A support tool based on machine learning may for in-home use by patients for self-assessment in deciding whether to go
reduce triage errors and be more robust, but this would be worth further to the ED or not. It also may be valuable to developed for use in disaster
exploring. zones and warzones. These can be explored in future studies.
In the present study, there were some differences from others, which
were the novelty of this study. First, the outcome was classified into four 5. Limitations
categories that may lead to higher error rate than binary classification,
which is widely used in other machine learning models [11,14,15,18, This study has several limitations. First, visits with more than a half
19]. However, machine learning can easily perform iterative recalibra­ missing variables were excluded; however, the analytic and non-analytic
tion of models over time as new data are provided [33]. Second, we used were still generally comparable, which can argue against substantial
only 12 features that were deemed to be available at the point of triage, selection bias. Second, this study was performed at a single institution
while the use of a complete set of information, such as physical exami­ and without external validation dataset; it might not be generalized to
nations, patient history, and socioeconomic status, may improve other hospitals. Individualized site-specific machine learning models
discrimination abilities [17,19,33]. However, the goal of the present need to be developed to improve accuracy in the future. Third, as visits
study is not to develop models using a broad set of features but rather to with suspected CVD were included according to their chief complaints or
derive machine learning models using a limited set of variables that are diagnosis, we cannot be certain that all patients were included, espe­
routinely available at current ED triage settings. Moreover, too many cially for some patients with infrequent diagnosis. Finally, since some
features included in the model might be impractical at ED triage settings subjective judgment is included into the determination of the triage,
because of the limited information and time. some triage levels might be inaccurate in the practices of different triage
Developing a machine learning model to differentiate the triage is nurses. Therefore, we are modeling nurses’ behavior. Given the large
necessary, since in the analytic cohort of 17,661 ED visits with suspected data set including the practice of many providers may be likely to reduce
CVD, the majority of them were low-risk patients (level 3 and 4). In individual bias.
western countries, it is also common that a large number of ED visits are
for non-urgent conditions [34,35]. The large proportion of non-urgent
6. Conclusion
visits would be the cause of ED overcrowding and the finite medical
resources waste, potentially leading to poor outcomes for persons who
Machine learning models were successfully developed to support
truly require emergency care [34,36,37]. Therefore, it is particularly
decision-making of triage levels for patients with suspected CVD using
compelling to identify the critical patients from the volumes of those
only data available at the time of triage. These models had good
with less urgent needs, and machine learning model showed good ability
discriminative ability of triage, and XGBoost demonstrated a slight
to assist to this process accurately and efficiently.
advantage over other models. Moreover, these models could be used for
Machine learning models could improve the decision-making ability
differential triage of low-risk patients and high-risk patients as a strategy
because of their advantages. Firstly, machine learning models are able to
to improve efficiency and allocation of finite resources.
incorporate the high-order nonlinear interactions between predictors
and the outcome [38], so our models can accommodate large data set
Contributions
and also identify the feature interactions. Additionally, advanced ma­
chine learning models own scalable benefits regarding they are able to
HJ and HM conceived and designed the study. HJ, PL and XC
update the models by automatically extracting the electronic health
monitored the entire planning, execution and analysis of the study. HJ
record data and integrating the digital images, natural language pro­
and XC obtained the ethical and grant for the study. HJ, PL, XC, HjL and
cessing, and continuous monitoring of physiological data [38–41].
GY participated in the data collection for the study. HM, HmL and WG
Therefore, machine learning, as an indispensable assistive technology,
analyzed and explained the data. HJ, HM, XC and THR provided advice
can further enhance the medical professional’s decision making in the
on the study methods and manuscript writing. HJ, HM, HmL, WG and
future.
THR wrote the first draft of the manuscript and prepared the manuscript.
Furthermore, this study cannot be denied that further exploration
All authors contributed to the final version. XC takes responsibility for
would be desirable. In the future, new models will be developed to
the paper as a whole.
predict disease prognosis, support decision-making of diagnosis, and all
of these models constitute a comprehensive data-based machine
learning decision support system for being used by medical professionals Funding
treating patients. Moreover, this supportive triage tool can be developed
This work was supported by the Major Project of Guangzhou Health

5
H. Jiang et al. International Journal of Medical Informatics 145 (2021) 104326

Science and Technology [grant number 2020A031005]; and The Key
Medical Disciplines and Specialities Program of Guangzhou.
Declaration of Competing Interest
The authors report no declarations of interest.
Acknowledgments
This research study was supported by the Major Project of Guangz­
hou Health Science and Technology (Grant No. 2020A031005) and The
Key Medical Disciplines and Specialities Program of Guangzhou.

Appendix A. Comparison of predictors variables and triage levels between the analytic and non-analytic cohort

Analytic cohort Non-analytic cohort

n = 17661 Missing (%) n = 10152 Missing (%)

Age, median (IQR), y 65 (51− 78) 0 64 (50− 77) 0

Female, No. (%) 9282 (52.56) 0 5586 (55.02) 0
Vital signs, median (IQR)
Temperature, ℃ 36.9 (36.5− 37.4) 11790 (66.7) 37 (36.7− 37.7) 9243 (91.0)
Pulse rate, beats/min 88 (75− 102) 106 (0.6) 86 (75− 100) 1190 (11.7)
Systolic blood pressure, mmHg 145 (126− 168) 219 (1.2) 148 (130− 171) 1200 (11.8)
Diastolic blood pressure, mmHg 83 (73− 94) 244 (1.3) 84 (74− 95) 1222 (12.0)
Respiratory rate, breaths/min 20 (18− 20) 8476 (47.9) 20 (20− 20) 9955 (98.0)
Oxygen saturation, % 98 (97− 99) 5381 (30.4) 98 (97− 99) 7315 (72.0)
Blood glucose, median (IQR), mmol/L 9.6 (7.4− 13.7) 16323 (92.4) 9.4 (6.7− 12.6) 10037 (98.8)
VAS pain score, median (IQR) 0 (0− 1.5) 3774 (21.3) 1.5 (0− 3.5) 9668 (95.2)
Glasgow Coma Scale score, median (IQR) 15 (15− 15) 2632 (14.9) 15 (15− 15) 9679 (95.3)
Prehospital index, median (IQR) 0 (0− 0) 5798 (32.8) 0 (0− 0) 9914 (97.6)

Triage levels, No. (%)

level 1 221 (1.3) 0 98 (0.9) 0
level 2 3288 (18.6) 0 1288 (12.6) 0
level 3 13515 (76.5) 0 7750 (76.3) 0
level 4 637 (3.6) 0 1016 (10.0) 0

Abbreviations: VAS, Visual Analogue Scale; IQR, interquartile range.

Appendix B. Performance by triage level of four machine learning models

Level 1 Level 2 Level 3 Level 4

Multinomial LR
Sensitivity 0.295 0.506 0.842 0.008
Specificity 0.998 0.840 0.467 1.000
PPV 0.650 0.420 0.838 1.000
NPV 0.991 0.881 0.476 0.964

RF
Sensitivity 0.227 0.350 0.882 0.055
Specificity 0.997 0.895 0.350 0.476
PPV 0.455 0.433 0.816 0.125
NPV 0.990 0.858 0.476 0.965

XGBoost
Sensitivity 0.250 0.161 0.982 0.000
Specificity 0.995 0.978 0.180 1.000
PPV 0.367 0.631 0.796 0.000
NPV 0.991 0.836 0.753 0.964

GBDT
Sensitivity 0.341 0.310 0.919 0.047
Specificity 0.993 0.930 0.318 0.990
PPV 0.375 0.505 0.815 0.154
NPV 0.992 0.855 0.547 0.965

Abbreviations: LR, logistic regression; RF, random forest; XGBoost, eXtreme gradient boosting; GBDT, gradient-boosted decision tree; PPV, positive
predictive value; NPV, negative predictive value.

References
[1] P. Joseph, D. Leong, M. McKee, S.S. Anand, J. Schwalm, K. Teo, et al., Reducing the
global burden of cardiovascular disease, part 1, Circ. Res. 121 (6) (2017) 677–694.
[2] E.J. Benjamin, P. Muntner, A. Alonso, M.S. Bittencourt, C.W. Callaway, A.
P. Carson, et al., Heart disease and stroke statistics-2019 update: a report from the
American Heart Association, Circulation 139 (10) (2019) e56–528.
[3] R. Lozano, M. Naghavi, K. Foreman, S. Lim, K. Shibuya, V. Aboyans, et al., Global
and regional mortality from 235 causes of death for 20 age groups in 1990 and
2010: a systematic analysis for the Global Burden of Disease Study 2010, Lancet
380 (9859) (2012) 2095–2128.
[4] W. Wang, C. Wang, J. Wang, Epidemiological analysis and disease spectrum
characteristics of emergency patients with critical illness: 3176 emergency cases in
2017 in a hospital in Beijing were analyzed, Zhonghua Wei Zhong Bing Ji Jiu Yi
Xue 30 (10) (2018) 987–990.

6
H. Jiang et al.
[5] M. Ebrahimi, A. Mirhaghi, R. Mazlom, A. Heydari, A. Nassehi, M. Jafari, The role
descriptions of triage nurse in emergency department: a Delphi study, Scientifica
2016 (2016) 1–6.
[6] Ministry of Health of the People’s Republic of China, Guidelines for the
classification of emergency patients (draft for solicitation), Chin. J. Crit. Care Med.
(Electron. Ed.). 4 (04) (2011) 241–243.
[7] D. Shi, X. Liu, Y. Zhou, Consensus of emergency triage experts, Chin. J. Emerg.
Med. 27 (06) (2018) 599–604.
[8] G. Zhiting, J. Jingfen, C. Shuihong, Y. Minfei, W. Yuwei, W. Sa, Reliability and
validity of the four-level Chinese emergency triage scale in mainland China: a
multicenter assessment, Int. J. Nurs. Stud. 101 (2020), 103447.
[9] X. Xie, W. Huang, Q. Liu, W. Tan, L. Pan, L. Wang, et al., Prognostic value of
modified early warning score generated in a Chinese emergency department: a
prospective cohort study, BMJ Open 8 (12) (2018), e24120.
[10] Af Dugas, Td Kirsch, M. Toerper, F. Korley, G. Yenokyan, D. France, et al., An
electronic emergency triage system to improve patient distribution by critical
outcomes, J. Emerg. Med. 50 (6) (2016) 910–918.
[11] S. Levin, M. Toerper, E. Hamrock, J.S. Hinson, S. Barnes, H. Gardner, et al.,
Machine-learning-based electronic triage more accurately differentiates patients
with respect to clinical outcomes compared with the Emergency Severity Index,
Ann. Emerg. Med. 71 (5) (2018) 565–574.
[12] S.W. Choi, T. Ko, K.J. Hong, K.H. Kim, Machine learning-based prediction of
Korean Triage and Acuity Scale level in emergency department patients, Healthc.
Inform. Res. 25 (4) (2019) 305–312.
[13] E.A. Cicolo, H.H.C. Peres, Electronic and manual registration of Manchester
System: reliability, accuracy, and time evaluation, Rev. Lat.-Am. Enferm. 27 (2019)
e3241.
[14] R.A. Taylor, J.R. Pare, A.K. Venkatesh, H. Mowafi, E.R. Melnick, W. Fleischman, et
al., Prediction of in-hospital mortality in emergency department patients with
sepsis: a local big data-driven, machine learning approach, Acad. Emerg. Med. 23
(3) (2016) 269–278.
[15] C. Eken, U. Bilge, M. Kartal, O. Eray, Artificial neural network, genetic algorithm,
and logistic regression applications for predicting renal colic in emergency settings,
Int. J. Emerg. Med. 2 (2) (2009) 99–105.
[16] B. Greg, IEEE/IBM Watson student showcase, Computer 49 (1) (2016) 102–104.
[17] K. Shameer, K.W. Johnson, B.S. Glicksberg, J.T. Dudley, P.P. Sengupta, Machine
learning in cardiovascular medicine: are we there yet? Heart 104 (14) (2018)
1156–1164.
[18] K. Rendell, I. Koprinska, A. Kyme, A.A. Ebker White, M.M. Dinh, The Sydney Triage
to Admission Risk Tool (START2) using machine learning techniques to support
disposition decision-making, Emerg. Med. Australas. 31 (3) (2019) 429–435.
[19] W.S. Hong, A.D. Haimovich, R.A. Taylor, Predicting hospital admission at
emergency department triage using machine learning, PLoS One 13 (7) (2018)
e201016.
[20] J. Karjala, S. Eriksson, Inter-rater reliability between nurses for a new paediatric
triage system based primarily on vital parameters: the Paediatric Triage Instrument
(PETI), BMJ Open 7 (2) (2017), e12748.
[21] H. Ruan, W. Ge, J. Chen, Y. Zhu, W. Huang, Prehospital Index provides prognosis
for hospitalized patients with acute trauma, Patient Prefer. Adherence 12 (2018)
561–565.
[22] J.J. Koehler, L.J. Baer, S.A. Malafa, M.S. Meindertsma, N.R. Navitskas, J.
E. Huizenga, Prehospital Index: a scoring system for field triage of trauma victims,
Ann. Emerg. Med. 15 (2) (1986) 178–182.
[23] M. Fernandes, R. Mendes, S.M. Vieira, F. Leite, C. Palos, A. Johnson, et al.,
Predicting Intensive Care Unit admission among patients presenting to the
International Journal of Medical Informatics 145 (2021) 104326
emergency department using machine learning and natural language processing,
PLoS One 15 (3) (2020), e0229331.
[24] L. Goldman, A.I. Schafer, Goldman’s Cecil Medicine-24th Edition, Elsevier
Saunders, America, 2011.
[25] J.T. Wright, J.D. Williamson, P.K. Whelton, J.K. Snyder, K.M. Sink, A randomized
trial of intensive versus standard blood-pressure control, N. Engl. J. Med. 373 (22)
(2015) 2103–2116.
[26] S.L. Stevens, S. Wood, C. Koshiaris, K. Law, P. Glasziou, R.J. Stevens, et al., Blood
pressure variability and cardiovascular disease: systematic review and meta-
analysis, BMJ 354 (2016) i4098.
[27] S. Lewington, R. Clarke, N. Qizilbash, R. Peto, R. Collins, Age-specific relevance of
usual blood pressure to vascular mortality: a meta-analysis of individual data for
one million adults in 61 prospective studies, Lancet 360 (9349) (2002) 1903–1913.
[28] S.A.E. Peters, R.R. Huxley, M. Woodward, Comparison of the sex-specific
associations between systolic blood pressure and the risk of cardiovascular disease,
Stroke 44 (9) (2013) 2394–2401.
[29] S.A.E. Peters, P. Muntner, M. Woodward, Sex differences in the prevalence of, and
trends in, cardiovascular risk factors, treatment, and control in the United States,
2001 to 2016, Circulation 139 (8) (2019) 1025–1035.
[30] K.K. Hyun, E.R.C. Millett, J. Redfern, D. Brieger, S.A.E. Peters, M. Woodward, Sex
differences in the assessment of cardiovascular risk in primary health care: a
systematic review, Heart Lung Circ. 28 (10) (2019) 1535–1548.
[31] T. Levis, D. Schwartz, Y. Bitan, Triage nurses decision support application design,
Proc. Int. Symp. Hum. Factors Ergon. Healthc. 7 (1) (2018) 52–55.
[32] Z. Elsayed, A. El-Zeny, H. Ellouly, Comparison between Australasian triage scale
and emergency severity index, Egypt. J. Surg. 39 (2) (2020) 455–460.
[33] S.J. Patel, D.B. Chamberlain, J.M. Chamberlain, A machine learning approach to
predicting need for hospitalization for pediatric asthma exacerbation at the time of
emergency department triage, Acad. Emerg. Med. 25 (12) (2018) 1463–1470.
[34] A. Durand, S. Gentile, B. Devictor, S. Palazzolo, P. Vignally, P. Gerbeaux, et al., ED
patients: how nonurgent are they? Systematic review of the emergency medicine
literature, Am. J. Emerg. Med. 29 (3) (2011) 333–345.
[35] L. UscherPines, J. Pines, A. Kellermann, E. Gillen, A. Mehrotra, Deciding to visit the
emergency department for non-urgent conditions: a systematic review of the
literature, Am. J. Manag. Care 19 (1) (2013) 47.
[36] M.L.V. Carret, A.G. Fassa, M.R. Domingues, Inappropriate use of emergency
services: a systematic review of prevalence and associated factors, Cad. Saúde Públ.
Rio de Janeiro 25 (1) (2009) 7–28.
[37] M.J. Schull, A. Kiss, J. Szalai, The effect of low-complexity patients on emergency
department waiting times, Ann. Emerg. Med. 49 (3) (2007) 257–264.
[38] Y. Raita, T. Goto, M.K. Faridi, D.F.M. Brown, C.A. Camargo, K. Hasegawa,
Emergency department triage prediction of clinical outcomes using machine
learning models, Crit. Care 23 (1) (2019).
[39] K. Lekadir, A. Galimzianova, A. Betriu, M. Del Mar Vila, L. Igual, D.L. Rubin, et al.,
A convolutional neural network for automatic characterization of plaque
composition in carotid ultrasound, IEEE J. Biomed. Health 21 (1) (2017) 48–55.
[40] Z. Zeng, S. Espino, A. Roy, X. Li, S.A. Khan, S.E. Clare, et al., Using natural
language processing and machine learning to identify breast cancer local
recurrence, BMC Bioinform. 19 (S17) (2018) 498.
[41] M. Motwani, D. Dey, D.S. Berman, G. Germano, S. Achenbach, M.H. Al-Mallah, et
al., Machine learning for prediction of all-cause mortality in patients with
suspected coronary artery disease: a 5-year multicentre prospective registry
analysis, Eur. Heart J. 38 (2017) 500–507.