Journal of Biomedical Informatics 119 (2021) 103826

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier.com/locate/yjbin

Original Research

Improving hospital readmission prediction using individualized

utility analysis
Michael Ko a, Emma Chen a, 1, Ashwin Agrawal a, 1, Pranav Rajpurkar a, Anand Avati a,
Andrew Ng a, Sanjay Basu b, Nigam H. Shah c, *
a
Department of Computer Science, Stanford University, CA, USA
b
Center for Primary Care, Harvard Medical School, MA, USA
c
Center for Biomedical Informatics Research, Stanford University, Stanford, CA, USA

A R T I C L E I N F O A B S T R A C T

Keywords: Objective: Machine learning (ML) models for allocating readmission-mitigating interventions are typically
Health informatics selected according to their discriminative ability, which may not necessarily translate into utility in allocation of
Machine learning resources. Our objective was to determine whether ML models for allocating readmission-mitigating in­
terventions have different usefulness based on their overall utility and discriminative ability.
Materials and methods: We conducted a retrospective utility analysis of ML models using claims data acquired
from the Optum Clinformatics Data Mart, including 513,495 commercially-insured inpatients (mean [SD] age 69
[19] years; 294,895 [57%] Female) over the period January 2016 through January 2017 from all 50 states with
mean 90 day cost of $11,552. Utility analysis estimates the cost, in dollars, of allocating interventions for
lowering readmission risk based on the reduction in the 90-day cost.
Results: Allocating readmission-mitigating interventions based on a GBDT model trained to predict readmissions
achieved an estimated utility gain of $104 per patient, and an AUC of 0.76 (95% CI 0.76, 0.77); allocating in­
terventions based on a model trained to predict cost as a proxy achieved a higher utility of $175.94 per patient,
and an AUC of 0.62 (95% CI 0.61, 0.62). A hybrid model combining both intervention strategies is comparable
with the best models on either metric. Estimated utility varies by intervention cost and efficacy, with each model
performing the best under different intervention settings.
Conclusion: We demonstrate that machine learning models may be ranked differently based on overall utility and
discriminative ability. Machine learning models for allocation of limited health resources should consider
directly optimizing for utility.

1. Introduction disconnect. Unplanned readmissions have a high financial burden for

Machine learning–based models for predicting a future health state
are now common, however, evidence that the use of these models to
guide interventions has improved patient outcomes is lacking [1]. A key
reason for this gap is that typical measures of predictive performance,
such as discrimination (the ability to differentiate people at higher risk
of having an event from those at lower risk) and calibration (the extend
to which the predicted values agree with the observed risk in a popu­
lation) [2], do not necessarily reflect a useful model [1,3]. Useful models
[4] are those that lead to a favorable change in clinical decision-making
[5], as measured by improved patient outcomes [6], increased utility, or
lower costs [6]. Hospital readmissions are a well known example of this
hospitals [7], are associated with adverse patient outcomes [8], and are
reflections of low quality of care [9]. The Medicare Payment Advisory
Commission (MedPAC) has estimated that 12% of readmissions are
potentially avoidable, and has estimated the potential cost-savings at $1
billion [10].
In this study, we investigated how models for allocating readmission-
mitigating interventions are ranked based on utility. Utility is measured
by taking into account the dollar value costs of allocating an interven­
tion or treatment, along with estimates of future individual patient ex­
penses. We examined whether higher discriminative ability of a model
translated into better utility by developing three models: one to directly
predict 30-day readmission using claims data, another to predict 90-day

* Corresponding author at: 1265 Welch Road X-235 MC: 5479, Stanford, CA, USA.
1
These authors contributed equally to this work.

https://doi.org/10.1016/j.jbi.2021.103826
Received 20 January 2021; Received in revised form 23 May 2021; Accepted 28 May 2021
Available online 1 June 2021
1532-0464/© 2021 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
M. Ko et al.
cost as a proxy for the task of readmission prediction, and a hybrid
model that incorporated the predictions of the 2 models. We compared
all three models on discriminative ability measured by AUC and on
utility measured by estimated cost savings.
2. Methods
2.1. Ethics
The Stanford University administrative panel for the Protection of
Human Subjects approved this study.
2.2. Data
Claims data were acquired from the Optum Clinformatics Data Mart,
which collects administrative health claims for commercially insured
members nationally. Claims data were verified, adjudicated, adjusted
with a standard pricing methodology to account for differences in
pricing across health plans and provider contracts, and de-identified
prior to inclusion in the Data Mart dataset. For this analysis, data are
health plan claims from all 50 states for commercially-insured in­
dividuals who were admitted as inpatients over the period January 2016
through January 2017. The data include demographics (age, sex from
enrollment applications) and all medical claims data, including inpa­
tient visits, International Classification of Disease series 10 diagnostic
codes, and payments. The data were split into a training set (80%) and a
test set (20%). The training set was further split into two non-
overlapping subsets, with 12.5% used to train the propensity score
model as part of our method to estimate utility, and the other 87.5%
used for training the readmission and cost models.
2.3. Outcome
The primary outcome in this study was all-cause unplanned read­
mission within 30 days of discharge. A readmission was defined as a
subsequent hospitalization following an eligible index admission. An
index admission was eligible if the following criteria were met: (1) pa­
tient was enrolled for 12 consecutive months prior to discharge and 90
continuous days post-discharge, (2) patient was discharged alive, (3)
patient did not leave against medical advice, (4) patient was not trans­
ferred to another acute care hospital setting, (5) patient’s primary
diagnosis was not one among psychiatric disorders, cancer treatments or
rehabilitation care, (6) patient’s previous index admission was not
within 30 days prior to discharge. Eligible index admissions were
considered to have a subsequent readmission if the patient had an un­
planned admission within 30 days of discharge. An admission was
defined as “planned” and not part of the readmission measure if it
contained procedures from a set of pre-specified planned procedure
codes, i.e. organ transplants or chemotherapy, and did not include acute
diagnosis codes for potentially planned procedures as described by
Planned Readmission Algorithm (Version 4.0, 2019) [11,12]. All other
readmissions were considered as unplanned, regardless of cause. The
outcome used in this study was consistent with the methodology used in
the U.S. Centers for Medicare & Medicaid Services (CMS) definitions,
with the exception that we required 90 continuous days of enrollment
post-discharge, rather than 30 days, for the purposes of our cost analysis.
The details of the labelling and cohort exclusions procedure adapted can
be viewed in Appendix D & E of CMS 2019 Measures Update: HWR [12].
The secondary outcome in this study was the cost accrued in the 90
days post-discharge of an index admission, herein referred to as post-
discharge cost. The accrued cost was defined as total standardized
gross payments (not charges) to all providers and facilities. The cost was
computed for each patient index admission by summing standardized
costs (in US Dollars) over 90 days, including post year claims corrections
and zero spending among enrolled individuals without medical claims.
2
Journal of Biomedical Informatics 119 (2021) 103826
2.4. Model development and validation
We developed machine learning models using gradient boosted de­
cision trees (GBDTs) to predict readmission risk (readmission model),
and post-discharge cost (cost model). We chose GBDTs because they
employ decision trees to capture nonlinear relationships in data that
traditional linear models are unable to capture, can handle mixes of
categorical and continuous covariates and scales well with large
amounts of data [13]. Moreover, it is straightforward to obtain variable
importance ranking from the model, which makes the approach more
interpretable than many other machine learning models [14]. GBDTs
have also been shown to achieve state-of-the-art performance in read­
mission risk prediction [15]. The training procedure for GBDTs involves
the construction of a sequence of decision trees such that each tree learns
from the errors of the prior tree to iteratively improve predictions [16].
We used the LightGBM framework to develop the models, which im­
plements several algorithmic optimizations on standard gradient
boosting to allow for additional training efficiency [17].
We trained the readmission and cost models on an identical set of
features extracted for every index admission. We included demographics
(age, sex) associated with the patient, diagnostic and procedure codes,
and location of index admission. Sex and index admission location were
one hot encoded while diagnostic and procedure codes were encoded as
aggregated counts of Clinical Classification Software categories [18]
from 12 months prior to discharge. In addition, the following features
included in the HOSPITAL Score [19] were computed: (1) number of
procedures during hospital stay, (2) number of admissions in the pre­
vious year, (3) number of hospital stays for 5 or more days, and (4) type
of index admission (urgent or elective). All features included in the LACE
Index [20] were also computed: (1) length of stay, (2) whether the
admission was acute, (3) Charlson Comorbidity Index [21], and (4)
number of ED visits prior to the index admission. A total of 539 features
were used. No missing data was present in our final dataset. Feature
selection was not used prior to model training since previous research
has shown that using the full set of features rather than subsets of fea­
tures was better for GBDTs in predicting readmission risk [15].
We used 3-fold cross-validation on the training data subset to select
the hyper-parameters for the models, including the number of trees, the
maximum depth of each tree, and the required minimum loss reduction
to partition leaf nodes based on the cross-validation area under the
receiver operating characteristic curve (AUC) and R2 for the read­
mission model and cost model respectively. Each model was refitted to
the full training set using the best parameters determined from 3-fold
cross validation.
The readmission model, cost models and an additional hybrid model,
were all built for the task of unplanned readmission prediction. The
readmission model employed standard GBDT binary classification for
predicting readmissions whereas the cost model employed standard
GBDT regression for predicting 90 day cost. The hybrid model inde­
pendently multiplied the output of the readmission model with the
output of the cost model to obtain an expected value of cost. We chose to
develop a hybrid model that combined predictions of the readmission
and cost model with the hopes of netting predictive and utility benefits
from both models. The outputs of all 3 models (readmission risk, 90 day
cost, and expected cost) were used to rank patients in our test set (see
Fig. 1). We evaluated the discriminative performance of these models
using AUC on the test set along with 95% confidence intervals computed
using the nonparametric bootstrap method with 1,000 bootstrap repli­
cates. Additionally, for the cost model only, we evaluated discriminative
performance on the task of predicting cost using R2 and mean absolute
error using the same bootstrapping method for 95% confidence
intervals.
For supplemental analysis, we constructed an additional cost-
sensitive readmission model for comparison. The cost-sensitive read­
mission model was trained with the same protocol of that of the read­
mission model, but employed a loss weighted by the 90 day cost of
M. Ko et al. Journal of Biomedical Informatics 119 (2021) 103826

Fig. 1. The readmission model, cost models and an additional hybrid model, were all built for the task of unplanned readmission prediction. The readmission model
employed standard GBDT binary classification for predicting readmissions whereas the cost model employed standard GBDT regression for predicting 90 day cost.
The hybrid model independently multiplied the output of the readmission model with the output of the cost model to obtain an expected value of cost. The outputs of
all 3 models (readmission risk, 90 day cost, and expected cost) were used to rank patients.

patients. 2.6. Outcome cost matrix

2.5. Individualized utility analysis
We used an approach, which we refer to as Individualized Utility
Analysis (IUA), for comparing predictive models based on estimated
utility. IUA estimates the utility, in dollars, of allocating interventions
for reducing readmission risk based on the reduction in the subsequent
90-day cost of care, incorporating heterogeneity in outcomes and costs
across patients [23]. Although other utility analysis are similar in esti­
mating model utility, other approaches treat all patients with the same
global estimates of outcomes and costs [22] which may not be gener­
alizable to real-world settings. We used IUA which estimates the utility
at an individual level. Although we present the example using dollar
cost, other units such as estimates of disability adjusted life years can
also be used for such utility estimates.
IUA constructs an outcome-cost matrix for each patient to outline the
utility for every possible outcome scenario. Each matrix uses the
observed patient cost, estimated counterfactual patient cost, known
intervention cost and intervention efficacy. This method makes the
following assumptions: (1) intervention cost and efficacy is fixed across
patients (2) the estimated counterfactual cost is a good estimate of the
cost incurred for a patient, had they received an effective intervention.
Upon assigning interventions based on a predictive model, the 90-
day cost of the patient’s care may follow these 4 scenarios (see Fig. 2):
• Scenario 1: Intervention is not assigned to a patient who was not
readmitted
• Scenario 2: Intervention is not assigned for a patient who was
readmitted
• Scenario 3: Intervention is assigned to a patient who was not
readmitted
• Scenario 4: Intervention is assigned to a patient who was readmitted
In both Scenario 1 and 2, the 90-day cost is the observed 90-day cost
in the dataset since no intervention has occurred. In Scenario 3, the 90-
day cost is the observed 90-day cost plus the cost of intervention. It is
important to note that the intervention in Scenario 3 is assumed to be
both harmless and effective. In Scenario 4, intervention is assumed to be
effective in reducing readmission risk and the 90-day cost is defined to
be the weighted average of their counterfactual cost (with weight p,
which is the efficacy of the intervention), and their observed cost (with
weight 1-p).
2.7. Counterfactual patient cost

Counterfactual patient cost is needed in estimating the cost

Fig. 2. Upon assigning an intervention based

on one of the machine learning derived
models, the 90-day cost of the patient’s care
is calculated as follows. If intervention is not
assigned, the future cost will be the observed
cost in the dataset for both readmitted (C1)
and not readmitted patients (C0). If inter­
vention is assigned, there are two scenarios. If
intervention was assigned to the not read­
mitted subgroup, the future cost is the actual
cost (C0) plus the cost of the intervention (t).
The effect of the intervention on them is assumed to be harmless, meaning a readmission will not occur due to the intervention. If intervention was assigned to the
readmitted subgroup, the future cost is defined to be the weighted average of their counterfactual cost (μ0) and their observed cost (C1) with the efficacy of the
intervention (p) serving as the weight. μ0 is estimated via propensity score matching as described in the methods section.

3
M. Ko et al. Journal of Biomedical Informatics 119 (2021) 103826

associated with Scenario 4. However, since we do not have access to the
counterfactual cost in observational data, we must estimate it using
causal inference. For this study we used propensity score matching
(PSM) [24], a commonly used statistical method, to estimate counter­
factual patient costs. We note that our method IUA is not limited to PSM,
but can be used with any reasonable causal inference method (nearest
neighbor matching, optimal matching) suitable for the dataset at hand.
We find that for our high dimensional dataset, PSM can match patients
with similar probability of readmission in readmitted to non-readmitted
groups with the advantage of being a well-established approach [24].
Propensity scores were obtained from an L1 regularized logistic
regression model trained to predict 90-day cost using an identical
feature set and hyperparameter selection procedure as the readmission
and cost models. The matching procedure assigned patients to one of 20
equal width propensity score bins from 0.0 to 0.6. Within each bin, the
median cost of the non-readmitted patient group was used to impute the
counterfactual costs for the readmitted patients (i.e. their cost if they
would not have a readmission) (see Supplemental Table 2). We evalu­
ated the goodness of fit of the logistic regression model on the test set
using AUC, with its 95% confidence intervals computed using the non-
parametric bootstrap with 1000 bootstrap replicates. The propensity
score model had an AUC of 0.691 (95% CI 0.688, 0.695), a calibration
slope of 0.945 (95% 0.931, 0.957) and a calibration intercept of 0.0124
(95% 0.0085, 0.016).
2.8. Feature importances
We quantified the impact of each input feature on the readmission
and cost models. The SHAP (SHapley Additive exPlanations) method
was used. The method explains prediction by allocating credit among
the input features; feature credit is calculated using Shapley Values [25],
as the change in the expected value of the model’s prediction of
improvement for a symptom when a feature is observed versus un­
known. To uncover clinically important features that were globally
predictive of the readmissions outcome and the cost outcome, the
Shapley values for features on individual predictions were aggregated
and reported along with their averaged absolute Shapley contributions
as a percent of the contributions of all the features.
3. Results
The final dataset consisted of 513,495 patients, divided into a
training set of 410,796 patients, a test set of 102,699 patients. There was
no patient overlap between the two sets. Overall, the average number of
index admissions per patient was 1.18. The prevalence of readmission
was 25.41%, and the mean 90-day cost was $11,552. Table 1 details the
patient, admission, and outcome characteristics in the dataset.
3.1. Model validation
On the task of predicting unplanned readmissions, the readmission
model achieved an AUC of 0.76 (95% CI 0.76, 0.77) on the test set; the
cost model achieved an AUC of 0.62 (95% CI 0.61, 0.62); the hybrid
model achieved an AUC of 0.72 (95% CI 0.71, 0.72). On the task of
predicting cost, the cost model achieved an R2 of 0.22 (95% CI 0.20,
0.23) and a mean absolute error of $13,574 (95% CI $13,439, $13,714).
The receiver operating characteristic (ROC) curves for each of the
models are shown in Fig. 3.
3.2. Individualized utility analysis
Utility at 0% intervention was $-12,719 which we defined as baseline
utility. We express all computed utility relative to baseline utility. A
fixed cost of $500 and fixed efficacy of 10% was used for the IUA
analysis based on prior literature [26–28], which estimates efficacy of
readmission reduction interventions to be between 10 and 50% and the
cost to be between $100–$2000. As computed by the IUA, the read­
mission model had a maximum utility gain of $104.23 per patient when
21.4% of the patient population was treated, the cost model had a
maximum utility gain of $175.94 per patient when 34.0% of the patient
population was treated, and the hybrid model had a maximum utility
gain of $177.21 when 34.3% of the patient population was treated. Both
the cost and hybrid models’ maximum utility gain per patient were
significantly greater than that of the readmission model using boot­
strapping and Bonferroni correction (p < .001). No significant difference
in utility was found between the cost and the hybrid model. Fig. 3 shows
the utility gain at each % patient population treated of each model.
Rerunning the analysis with a cost sensitive readmission model instead
of the readmission model yielded similar results (see Supplemental
Fig. 3).
3.3. Sensitivity analysis
A sensitivity analysis of intervention cost and efficacy was performed
to measure the utility of each model under various settings of cost and
efficacy of the intervention. Under low intervention cost and high
intervention efficacy settings, the readmission model is able attain the
best utility. But when intervention is expensive and less effective, the
cost and hybrid model outperform the readmission model based on
overall utility. Fig. 4 shows the best model based on highest utility gain
under various intervention cost and efficacy values. Supplemental
Figs. 1 and 2 shows individual model utility and model utility
differences.
3.4. Feature importances

The most important features for the readmission prediction model

were the number of hospital admissions within the past year (greater
counts increased risk), number of readmissions within the past year
Table 1
(greater counts increased risk) and Osteoarthritis (less visits with the
Summary statistics of the training set, test set, and overall dataset.
diagnosis in the past year increased risk) with mean contributions of
Characteristics Training set Test set Overall
10.1%, 6.6% and 5.5%.
Total members 410,796 102,699 513,495 For the cost model, the most important features were Hemodialysis
Total female 235,836 59,059 294,895 (more visits with the diagnosis in the past year increased cost), Age
(57.41%) (57.51%) (57.43%)
Age, mean (std) 69.11 (18.79) 69.15 (18.76) 69.12 (18.79)
(higher value increased cost) and number of ED visits in the past year
Total index admission 484,609 121,265 605,874 (greater counts increased cost) with mean contributions of 10.2%, 7.2%
Readmission rate 123,405 30,574 153,979 and 6.1%. Fig. 5 shows additional important features for the readmission
(25.46%) (25.21%) (25.41%) and cost model.
Mean 90-day cost $ 11,556 $ 11,538 $ 11,552
25th Quantile 90-day $ 415 $ 418 $ 415
cost 4. Discussion
50th Quantile 90-day $ 2,521 $ 2,521 $ 2,521
cost Useful models are those that lead to a favorable change in clinical
75th Quantile 90-day $ 11,556 $ 11,538 $ 11,552 decision-making in the form of improved patient outcomes, increased
cost
utility, or lower costs. In this study, we assessed the utility of machine

4
M. Ko et al. Journal of Biomedical Informatics 119 (2021) 103826

Fig. 3. The left panel shows the ROC curve for three models for assigning interventions to prevent a readmission. A readmission model (AUC = 0.76), which assigns
intervention according to readmission probability, a cost model (AUC = 0.62), which assigns intervention according to the cost of patient, and a hybrid cost +
readmission model (AUC = 0.72), which assigns intervention according to the product of cost and readmission probability. The right panel shows the utility realized
in terms of the average savings per patient given a certain number of interventions provided. With a fixed intervention cost of $500 and efficacy of 10%, the
readmission model had a maximum utility gain of $104.23 per patient when 21.4% of the patient population was treated, the cost model had a maximum utility of
$175.94 per patient when 34.0% of the patient population was treated, and the hybrid model had a maximum utility of $177.21 when 34.3% of the patient
population was treated.

interventions according to its rankings. Second, under the same inter­

Fig. 4. Different models have the highest utility gain under variable inter­
vention cost and efficacy. In low intervention cost and high intervention effi­
cacy settings, the readmission model is able attain the highest utility. However,
with higher intervention cost and lower intervention efficacy, the cost and
hybrid model outperform the readmission model in terms of overall utility.
learning models for the task of allocating interventions to avoid un­
planned readmissions. We took into account the dollar value costs of an
intervention allocation, along with estimates of future individual patient
expenses. Our study led to three main findings. First, although a model
trained to predict readmissions achieved a higher AUC than a model
trained to predict cost, at 10% intervention efficacy and $500 inter­
vention cost, the cost model provided a higher utility in allocation of the
vention settings, a hybrid model that combined the predictions of the
readmission model with the cost model achieved comparable utility and
higher AUC than the cost model. Third, individualized utility analysis
identifies different models that perform the best in terms of utility gain
under different intervention cost and efficacy settings.
Our results suggest that classification or prediction models with
higher discriminative performance may not necessarily give higher
utility. In healthcare, there is always a capacity constraint (either
monetary or staffing) which induces a hard limit on the number of ac­
tions that can be taken in response to a model’s ranking of patients at
risk of an outcome such as readmission [30,31]. The intent underlying
most of the risk-scoring [32] (from readmissions [33], to cost [34], to
mortality [15]) is to allocate interventions to those in the most need.
Utility per unit cost (or the budget) is one objective measure for eval­
uating clinical risk scoring. We acknowledge that “cost” is an imperfect
proxy for need across patient groups as outlined by Obermeyer et al.
[34] and that caution needs to be exercised in defining the outcome for
which a prediction model is built [34]. However, within a homogenous
patient group where label-bias can be avoided, incorporating cost in­
formation directly into model building provides a much tighter align­
ment with the end goal–which is allocation of limited resources to derive
the highest value from interventions.
Our proposed approach for analyzing individualized utility poses
several advantages over prior literature studying 30-day readmission
[20]. Our approach uses an evaluation that focuses on utility obtained
by taking actions based on predictions whilst most 30-day readmission
work has focused on metrics such as AUC that do not account for the cost
implications of alternative actions [33]. Although a case study of
reducing readmissions for heart failure done by Bayati et al. has
demonstrated that improvements in prediction quality can improve
utility [22], a key difference is that the prior work used the average cost
of readmission in calculating utility, which may not translate into an
individual-level net benefit [23,35]. We found that 90 day costs have a
high variability per patient and our individualized utility analysis

5
M. Ko et al.
Fig. 5. Important features for the readmission model (left) and cost model (right). The impact of each feature on the discrimination ability of the readmission and
cost models was quantified using the SHAP (SHapley Additive exPlanations) method, which explains prediction by allocating credit among the input features. Feature
importances on individual predictions were aggregated and reported along with their averaged absolute Shapley contributions as a percent of the contributions of all
the features.
addresses this heterogeneity in costs across patients [23].
Furthermore, analyzing individualized utility can reveal different
areas in the intervention cost and efficacy space where different models
perform the best. Previous work done by Bayati et al., highlights inter­
vention settings that lead to failure modes in which a readmission model
would not be useful. Our work extends these findings by proposing al­
ternatives, such as the cost and hybrid models, which attain a higher
utility and may be used in place of a readmission prediction model in
such situations. When allocating interventions to improve patient out­
comes, examining individualized utility may help identify the suitable
model given the cost and efficacy of the intervention.
While we have demonstrated the usefulness of our proposed
approach using the well understood readmissions prediction setting, the
idea of IUA is applicable to many settings where the output of a pre­
dictive model is used to guide allocation of interventions. Since
healthcare interventions have a wide range of costs and efficacies, IUA
may be a generalized framework that identifies the best modeling ap­
proaches for the intervention setting at hand. We believe that the pro­
posed method is another concrete step towards making machine
learning models more useful in practice [1,36].
5. Limitations
This study has several important limitations. First, utility was
defined as the dollar value with a linear U-Curve, and may be redefined
to incorporate measures such as quality of life, number of actions a care
team takes, essential resources consumed with a nonlinear U-curve.
Second, our analysis is based on retrospective data. Rigorous validation
of our findings using IUA for estimating the utility of 30-day readmission
models would require evaluation in a prospective setting. Third, we
assume a fixed efficacy of readmission mitigating interventions. While
there is no doubt that treatment effect heterogeneity exists, estimating
heterogeneous treatment effects remains unsolved. Fourth, we do not
investigate the scenario of explicitly performing model selection based
on our IUA method; we expect that training procedures that directly
optimize and select models based on the utility may be a fruitful avenue
for further exploration.
6
Journal of Biomedical Informatics 119 (2021) 103826
6. Conclusion
Machine learning models are used to predict the probability of
occurrence of an outcome, in an effort to allocate limited health re­
sources to get higher value for the care delivered. We show that models
for intervention allocation can be made more useful by jointly ranking
on individual utility and discriminative ability. Machine learning
models for allocation of limited health resources should consider
directly optimizing for utility.
CRediT authorship contribution statement
Michael Ko: Conceptualization, Methodology, Software, Writing -
original draft. Emma Chen: Methodology, Software, Writing - original
draft. Ashwin Agrawal: Methodology, Software, Writing - original
draft. Pranav Rajpurkar: Conceptualization, Writing - review & edit­
ing. Anand Avati: Conceptualization. Andrew Ng: Supervision. Sanjay
Basu: Supervision. Nigam H. Shah: Conceptualization, Supervision,
Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Appendix A. Supplementary material
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.jbi.2021.103826.
References
[1] N.H. Shah, A. Milstein, Ph.D.S.C. Bagley, Making machine learning models
clinically useful, JAMA 322 (14) (2019) 1351, https://doi.org/10.1001/
jama.2019.10306.
[2] A.C. Alba, T. Agoritsas, M. Walsh, et al., Discrimination and calibration of clinical
prediction models: users’ guides to the medical literature, JAMA 318 (14) (2017)
1377–1384, https://doi.org/10.1001/jama.2017.12126.
[3] V.X. Liu, D.W. Bates, J. Wiens, N.H. Shah, The number needed to benefit:
estimating the value of predictive analytics in healthcare, J. Am. Med. Inform.
Assoc. JAMIA 26 (12) (2019) 1655–1659, https://doi.org/10.1093/jamia/ocz088.
M. Ko et al.
[4] E.J. Emanuel, R.M. Wachter, Artificial intelligence in health care: will the value
match the hype? JAMA 321 (23) (2019) 2281, https://doi.org/10.1001/
jama.2019.4914.
[5] S.W. Grant, G.S. Collins, S.A.M. Nashef, Statistical primer: developing and
validating a risk prediction model†, Eur. J. Cardiothorac. Surg. 54 (2) (2018)
203–208, https://doi.org/10.1093/ejcts/ezy180.
[6] A.K. Triantafyllidis, A. Tsanas, Applications of machine learning in real-life digital
health interventions: review of the literature, J. Med. Internet Res. 21 (4) (2019),
e12286, https://doi.org/10.2196/12286.
[7] S.F. Jencks, M.V. Williams, E.A. Coleman, Rehospitalizations among patients in the
medicare fee-for-service program, N. Engl. J. Med. 360 (14) (2009) 1418–1428,
https://doi.org/10.1056/NEJMsa0803563.
[8] Daniel R. Levinson, Inspector General, Adverse events in hospitals: national
incidence among medicare beneficiaries. Department of Health and Human
Services Office of the Inspector General, 2010.
[9] H.M. Krumholz, K. Wang, Z. Lin, et al., Hospital-readmission risk — isolating
hospital effects from patient effects, N. Engl. J. Med. 377 (11) (2017) 1055–1064,
https://doi.org/10.1056/NEJMsa1702321.
[10] C.K. McIlvennan, Z.J. Eapen, L.A. Allen, Hospital readmissions reduction program,
Circulation 131 (20) (2015) 1796–1803, https://doi.org/10.1161/
CIRCULATIONAHA.114.010270.
[11] L.I. Horwitz, J.N. Grady, D.B. Cohen, et al., Development and validation of an
algorithm to identify planned readmissions from claims data, J. Hosp. Med. 10 (10)
(2015) 670–677, https://doi.org/10.1002/jhm.2416.
[12] CMS. U.S. Centers for Medicare & Medicaid Services (CMS) Measure Methodology.
Published 2019. https://www.qualitynet.org/inpatient/measures/readmission/
methodology.
[13] H. Zhang, S. Si, C.-J. Hsieh, GPU-acceleration for Large-scale Tree Boosting.
ArXiv170608359 Cs Stat. Published online June 26, 2017. Accessed March 24,
2020. http://arxiv.org/abs/1706.08359.
[14] F. Cabitza, R. Rasoini, G.F. Gensini, Unintended consequences of machine learning
in medicine, JAMA 318 (6) (2017) 517, https://doi.org/10.1001/jama.2017.7797.
[15] L. Einav, A. Finkelstein, S. Mullainathan, Z. Obermeyer, Predictive modeling of U.
S. health care spending in late life, Science 360 (6396) (2018) 1462–1465, https://
doi.org/10.1126/science.aar5045.
[16] J. Friedman, T. Hastie, R. Tibshirani, Additive logistic regression: a statistical view
of boosting (With discussion and a rejoinder by the authors), Ann. Stat. 28 (2)
(2000) 337–407, https://doi.org/10.1214/aos/1016218223.
[17] G. Ke, Q. Meng, T. Finley, et al., LightGBM: a highly efficient gradient boosting
decision tree, NIPS17 Proc 31st Int Conf Neural Inf Process Syst, Published online
December 2017:3149-3157.
[18] CCS_10 (Clinical Classifications Software 10) - Synopsis. https://www.nlm.nih.
gov/research/umls/sourcereleasedocs/current/CCS_10/index.html.
[19] J.D. Donzé, M.V. Williams, E.J. Robinson, et al., International validity of the
HOSPITAL score to predict 30-day potentially avoidable hospital readmissions,
JAMA Intern. Med. 176 (4) (2016) 496, https://doi.org/10.1001/
jamainternmed.2015.8462.
[20] C. van Walraven, I.A. Dhalla, C. Bell, et al., Derivation and validation of an index to
predict early death or unplanned readmission after discharge from hospital to the
community, Can. Med. Assoc. J. 182 (6) (2010) 551–557, https://doi.org/
10.1503/cmaj.091117.
Journal of Biomedical Informatics 119 (2021) 103826
[21] H. Quan, V. Sundararajan, P. Halfon, et al., Coding algorithms for defining
comorbidities in ICD-9-CM and ICD-10 administrative data, Med. Care 43 (11)
(2005) 1130–1139, https://doi.org/10.1097/01.mlr.0000182534.19832.83.
[22] M. Bayati, M. Braverman, M. Gillam, et al., Data-driven decisions for reducing
readmissions for heart failure: general methodology and case study, Zhang H, ed.
PLoS ONE. 2014;9(10):e109264. doi:10.1371/journal.pone.0109264.
[23] J.P.A. Ioannidis, A.M. Garber, Individualized cost-effectiveness analysis, PLoS Med.
8 (7) (2011), e1001058, https://doi.org/10.1371/journal.pmed.1001058.
[24] A. Abadie, G.W. Imbens, Matching on the estimated propensity score,
Econometrica. 84 (2) (2016) 781–807, https://doi.org/10.3982/ECTA11293.
[25] L.S. Shapley, A value for n-person games, Contrib. Theory Games 2 (28) (1953)
307–317.
[26] R. Nair, H. Lak, S. Hasan, D. Gunasekaran, A. Babar, K.V. Gopalakrishna, Reducing
all-cause 30-day hospital readmissions for patients presenting with acute heart
failure exacerbations: a quality improvement initiative, Cureus 12 (3) (2020),
e7420, https://doi.org/10.7759/cureus.7420.
[27] S. Kripalani, C.N. Theobald, B. Anctil, E.E. Vasilevskis, Reducing hospital
readmission rates: current strategies and future directions, Annu. Rev. Med. 65
(2014) 471–485, https://doi.org/10.1146/annurev-med-022613-090415.
[28] V. Baky, D. Moran, T. Warwick, et al., Obtaining a follow-up appointment before
discharge protects against readmission for patients with acute coronary syndrome
and heart failure: A quality improvement project, Int. J. Cardiol. 257 (2018)
12–15, https://doi.org/10.1016/j.ijcard.2017.10.036.
[30] W. Crown, N. Buyukkaramikli, P. Thokala, et al., Constrained optimization
methods in health services research-an introduction: Report 1 of the ISPOR
optimization methods emerging good practices task force, Value Health J. Int. Soc.
Pharmacoeconomics Outcomes Res. 20 (3) (2017) 310–319, https://doi.org/
10.1016/j.jval.2017.01.013.
[31] W. Crown, N. Buyukkaramikli, M.Y. Sir, et al., Application of constrained
optimization methods in health services research: report 2 of the ISPOR
optimization methods emerging good practices task force, Value Health J. Int. Soc.
Pharmacoeconomics Outcomes Res. 21 (9) (2018) 1019–1028, https://doi.org/
10.1016/j.jval.2018.05.003.
[32] D.W. Challener, L.J. Prokop, O. Abu-Saleh, The proliferation of reports on clinical
scoring systems: issues about uptake and clinical utility, JAMA 321 (24) (2019)
2405, https://doi.org/10.1001/jama.2019.5284.
[33] A. Artetxe, A. Beristain, M. Graña, Predictive models for hospital readmission risk:
A systematic review of methods, Comput. Methods Programs Biomed. 164 (2018)
49–64, https://doi.org/10.1016/j.cmpb.2018.06.006.
[34] Z. Obermeyer, B. Powers, C. Vogeli, S. Mullainathan, Dissecting racial bias in an
algorithm used to manage the health of populations, Science 366 (6464) (2019)
447–453, https://doi.org/10.1126/science.aax2342.
[35] T.K. Nuckols, E. Keeler, S. Morton, et al., Economic evaluation of quality
improvement interventions designed to prevent hospital readmission: a systematic
review and meta-analysis, JAMA Intern. Med. 177 (7) (2017) 975, https://doi.org/
10.1001/jamainternmed.2017.1136.
[36] K. Jung, S. Kashyap, A. Avati, et al., A framework for making predictive models
useful in practice, J. Am. Med. Inform. Assoc. Published online December 22, 2020:
ocaa318. doi:10.1093/jamia/ocaa318.