Research

JAMA Dermatology | Brief Report

Utility of a Model for Predicting the Risk of Sentinel Lymph Node

Metastasis in Patients With Cutaneous Melanoma
Michael A. Marchetti, MD; Stephen W. Dusza, DrPH, MPH; Edmund K. Bartlett, MD

Editor's Note page 684

IMPORTANCE A neural network-based model (i31-GEP-SLNB) that uses clinicopathologic Supplemental content
factors (thickness, mitoses, ulceration, patient age) plus molecular analysis (31-gene
expression profiling) has become commercially available to guide selection for sentinel lymph
node (SLN) biopsy in cutaneous melanoma, but its clinical utility is not well characterized.

OBJECTIVE To determine if use of the i31-GEP-SLNB model is associated with clinical benefit
when used to select patients for SLN biopsy.

DESIGN, SETTING, AND PARTICIPANTS This decision-analytic study used data derived from a
published external validation study of the i31-GEP-SLNB prediction model. Participants
included patients with primary cutaneous melanoma.

MAIN OUTCOMES AND MEASURES The primary outcome was the net benefit associated with
using the i31-GEP-SLNB model for SLN biopsy selection compared with other selection
strategies (SLN biopsy for all patients and SLN biopsy for no patients) at a 5% risk threshold.
Analyses were stratified by American Joint Committee on Cancer (AJCC) T category. The
reduction in the number of avoidable SLN biopsies and relative utility were also calculated.

RESULTS Compared with other SLN biopsy selection strategies, use of the i31-GEP-SLNB
model had greater net benefit for patients with T1b (+0.012), T2a (+0.002), and T2b
melanoma (+0.002) but not for those with high-risk T1a (−0.003) disease. The improvement
in relative utility was +22% in patients with T1b, +1% in T2a, and +2% in T2b melanoma.
Compared with SLN biopsy for all patients, use of the model would equate to a 23% decrease
in SLN biopsies among patients with T1b disease without an SLN metastasis with no increase
in the number of patients with an SLN metastasis left untreated; among patients with T2a
and T2b melanoma, the net decrease in avoidable biopsies compared with SLN biopsy for all Author Affiliations: Dermatology
was 3% and 4%, respectively. Service, Department of Medicine,
Memorial Sloan Kettering Cancer
CONCLUSIONS AND RELEVANCE The findings of this decision-analytic study suggest that Center, New York, New York
i31-GEP SLNB has significant potential for risk-stratifying patients with T1b melanoma if using (Marchetti, Dusza); Gastric and Mixed
Tumor Service, Department of
a 5% risk threshold; its role among patients with T1a and T2 melanoma or using other risk Surgery, Memorial Sloan Kettering
thresholds requires further study. A prospective validation study confirming the added Cancer Center, New York, New York
clinical benefit and cost-effectiveness of i31-GEP-SLNB compared with free (Bartlett).
clinicopathologic-based prediction models is needed in patients with T1b melanoma. Corresponding Author: Michael A.
Marchetti, MD, Dermatology Service,
Department of Medicine, Memorial
JAMA Dermatol. 2022;158(6):680-683. doi:10.1001/jamadermatol.2022.0970 Sloan Kettering Cancer Center, 530 E
Published online April 27, 2022. 74th St, New York, NY 10021
(marchetm@mskcc.org).

N
ational Comprehensive Care Network guidelines rec-
ommend that a sentinel lymph node (SLN) biopsy be
considered in patients diagnosed with cutaneous mela-
noma with a 5% or higher risk of positivity and offered to those
with a greater than 10% risk of positivity.1 To improve selec-
tion for this procedure, a neural network-based prediction
model using clinicopathologic factors (thickness, mitoses, ul-
ceration, age) plus molecular analysis (31-gene expression pro-
filing) has become commercially available and received Medi-
care approval (i31-GEP-SLNB).2 Although this model provides
absolute risk estimates, its clinical utility is not well charac-
terized. Our objective was to identify whether use of the
i31-GEP-SLNB model would lead to clinical benefit in select-
ing patients for SLN biopsy.3-7
Methods
The study was exempt from institutional review board re-
view because data were publicly available. Raw data and clas-
sification measures of the i31-GEP-SLNB model by American
Joint Committee on Cancer (AJCC) T category were extracted
from Whitman et al2 for patients with T1a-HR, T1b, and T2 dis-
ease; data were not reported for patients with T3/T4 disease

680 JAMA Dermatology June 2022 Volume 158, Number 6 (Reprinted) jamadermatology.com

© 2022 American Medical Association. All rights reserved.

Utility of a Model for Predicting the Risk of Sentinel Lymph Node Metastasis in Patients With Cutaneous Melanoma Brief Report Research

and could not be analyzed. Patients with T1a melanoma and

mitotic index of 2 mm2 or larger, lymphovascular invasion, ab-
sence of tumor infiltrating lymphocytes, age 40 years or
younger, microsatellites, regression, or transected base were
considered high-risk T1a (T1a-HR). Classification measures
were reported by Whitman et al2 using a 5% risk threshold (ie,
≥5% risk was a positive test and a <5% risk was a negative test).
The net benefit4,8,9 of the model and competing strategies (SLN
biopsy for all patients [100% sensitivity, 0% specificity] and
SLN biopsy for no patients [0% sensitivity, 100% specificity])
were computed. Net benefit is a decision analytic measure that
puts the benefits of identifying an SLN metastasis and the
harms of unnecessary SLN biopsies on the same scale through
the use of an exchange rate (net benefit = [true posi-
tives ÷ total sample size] – [(false positives ÷ total sample
size) × (exchange rate)]). The unit of net benefit is true
positives.4 A 5% risk threshold implies that the harm of delay-
ing the detection of an SLN metastasis is 19 times greater than
an unnecessary SLN biopsy. Thus, the exchange rate is 1 ÷ 19
(exchange rate = [pt ÷ (1-pt)]), where pt is the threshold prob-
Key Points
Question What is the clinical benefit of the i31-GEP-SLNB model
for sentinel lymph node (SLN) biopsy selection using a 5% risk
threshold?
Findings This decision-analytic study found that, compared with
SLN biopsy for all or none, use of i31-GEP-SLNB had greater net
benefit for patients with T1b, T2a, and T2b melanoma, but not for
those with high-risk T1a melanoma. Compared with SLN biopsy all,
use of i31-GEP-SLNB would be equivalent to a strategy that
reduced unnecessary biopsies by 23% in patients with T1b and 3%
to 4% in those with T2 melanoma without missing any patients
with SLN metastasis.
Meaning The potential for benefit of i31-GEP-SLNB appears
greatest among patients with T1b melanoma.
ability that defines a positive result). The reduction in the num-
ber of avoidable SLN biopsies (avoidable biopsies = [true nega-
tives ÷ total sample size] − [(false negatives ÷ total sample

Table 1. Classification Measures of the i31-GEP-SLNB Prediction Model

SLN cases, No. i31-GEP-SLNB, % (95% CI)

Overall SLN Predictive value
SLN positivity biopsy reduction
T Category Positive Negative rate, % Sensitivity Specificity rate Positive Negative
T1a-HR 7 228a 3 43 (10-82) 69 (62-75) 69 (62-75) 4 (1-11) 98 (94-99)
T1b 18 310b 5 83 (59-96) 42 (37-48) 41 (36-46) 8 (4-12) 98 (94-100)
T2a 48 368c 12 96 (86-99) 14 (11-18) 13 (9-16) 13 (9-17) 96 (87-100)
T2b 15 103d 13 100 (78-100) 5 (2-11) 4 (1-8) 13 (7-21) 100 (48-100)
Abbreviations: SLN, sentinel lymph node; T1a-HR, T1a high-risk patients (mitotic examination in 49 cases.
index ⱖ2 mm2; lymphovascular invasion, absence of tumor infiltrating c
Negative SLN status determined via SLN biopsy in 330 cases and clinical
lymphocytes, age <40 years, microsatellites, regression, or transected base). examination in 38 cases.
a
Negative SLN status determined via SLN biopsy in 86 cases and clinical d
Negative SLN status determined via SLN biopsy in 91 cases and clinical
examination in 142 cases. examination in 12 cases.
b
Negative SLN status determined via SLN biopsy in 261 cases and clinical

Table 2. Net Benefit and Relative Utility of the i31-GEP-SLNB Prediction Model Using a 5% Risk Threshold

SLN biopsy
T Category Strategy 1: nonea Strategy 2: allb Strategy 3: using i31-GEP SLNB model
Net benefit (95% CI)
T1a-HR 0 −0.021 (−0.044 to 0.001) −0.003 (−0.018 to 0.119)
T1b 0 0.005 (−0.019 to 0.030) 0.017 (−0.006 to 0.040)
T2a 0 0.069 (0.037 to 0.100) 0.070 (0.039 to 0.101)
T2b 0 0.081 (0.026 to 0.136) 0.083 (0.025 to 0.142)
Relative utility, % (95% CI)c
T1a-HR 0 NA NA
T1b 0 9 (0-64) 31 (0-69)
T2a 0 60 (46-73) 61 (48-74)
T2b 0 64 (38-89) 66 (46-85)
c
Abbreviations: SLN, sentinel lymph node; T1a-HR, T1a high-risk patients (mitotic Relative utility is calculated by dividing the net benefit by the maximum
index ⱖ2 mm2; lymphovascular invasion, absence of tumor infiltrating achievable utility (prevalence) and ranges from 0% to 100%. In other words,
lymphocytes, age <40 years, microsatellites, regression, or transected base). relative utility is the maximum fraction of expected utility achieved by risk
a
The SLN biopsy for none is equivalent to a strategy with 0% sensitivity and prediction compared with perfect prediction. Relative utility allows an
100% specificity. assessment of the potential for improved performance with better prediction
b
models.
The SLN biopsy for all is equivalent to a strategy with 100% sensitivity and 0%
specificity.

jamadermatology.com (Reprinted) JAMA Dermatology June 2022 Volume 158, Number 6 681

© 2022 American Medical Association. All rights reserved.

 Research Brief Report Utility of a Model for Predicting the Risk of Sentinel Lymph Node Metastasis in Patients With Cutaneous Melanoma
Figure. Association of Net Benefit Using a 5% Risk Threshold
for i31-GEP-SLNB Model, SLN Biopsy for All, and SLN Biopsy for None,
Stratified by T Category
0.15
SLN biopsy for all
i31-GEP SLNB model
0.10 SLN biopsy for none
Net benefit
0.05
0 set, use of the model would result in failing to biopsy 3 of 18
-0.05
T1a-HR T1b T2a T2b
Melanoma T stage
Net benefit is a decision-analytic measure that informs clinical choices involving
trade-offs. In classical decision theory, the strategy with the highest expected
benefit to patients should generally be chosen, irrespective of size or statistical
significance. That said, if 1 strategy required data from an invasive, harmful, or
expensive procedure, the added net benefit may not be justifiable. The unit of
net benefit is true-positive results. It signifies the strategy that leads to the
greatest net true-positive patients treated, after appropriately subtracting the
harms of unnecessary treatment (false-positive results) from the benefits of
appropriate treatment (true-positive results) using an exchange rate. A risk
threshold of 5% for SLN biopsy decisions implies that the harms of not
biopsying an SLN metastasis are 19 times greater than the harms of an
unnecessary SLN biopsy. The maximum achievable net benefit (perfect
prediction) is the disease prevalence (ie, proportions of 0.03 T1a-HR, 0.05 T1b,
0.12 T2a, and 0.13 T2b) in this data set. In clinical terms, use of i31-GEP-SLNB for
SLN biopsy decisions vs SLN biopsy for all would lead to 1 more net true-positive
result identified for every 83 patients with T1b, 500 with T2a, and 500 with T2b
disease, assuming a risk threshold of 5%. SLN Indicates sentinel lymph node;
T1a-HR, T1a high-risk patients (mitotic index ⱖ2 mm2; lymphovascular invasion,
absence of tumor infiltrating lymphocytes, age <40 years, microsatellites,
regression, or transected base).
size) ÷ (exchange rate)]) and relative utility (relative util-
ity = net benefit ÷ maximum achievable utility) and relative
utility (net benefit divided by maximum achievable utility)
were also calculated.3
Results
With higher T categories, sensitivity of i31-GEP-SLNB in-
creased and specificity decreased (Table 1). The positive pre-
dictive value (PPV) using 5% risk as the cutoff for a positive
result ranged from 4% among patients with T1a-HR mela-
noma to 13% among those with T2b disease. The negative pre-
dictive value was greater than 95% for patients with T1a-HR,
T1b, T2a, and T2b disease.
Compared with SLN biopsy all, use of i31-GEP-SLNB was
associated with greater net benefit for patients with T1b, T2a,
and T2b melanoma (Table 2 and the Figure). Among patients
with T1a-HR disease, use of i31-GEP-SLNB was associated with
lower net benefit than SLN biopsy none (−0.003) but higher
net benefit than SLN biopsy all (+0.018).
682 JAMA Dermatology June 2022 Volume 158, Number 6 (Reprinted)
© 2022 American Medical Association. All rights reserved.
The greatest improvement in net benefit (+0.012) and rela-
tive utility (+22%) occurred in patients with T1b melanoma.
This improvement remained for most conditions in a sensi-
tivity analysis (eTable in the Supplement). In clinical terms,
use of i31-GEP-SLNB for SLN biopsy decisions vs biopsy all
would lead to 1 more net true positive identified for every 83
patients with T1b disease; this is the equivalent of a 23% de-
crease in unnecessary SLN biopsies with no increase in the
number of patients with an SLN metastasis left untreated. Per
100 patients, this could avoid $81 052 to $350 129 in direct SLN
biopsy-related charges10,11 (potentially higher if adjusted to
2022 dollars) but at a cost of $719 300 for i31-GEP-SLNB test-
ing at the CMS Medicare rate.11,12 In absolute terms, for this data
patients with T1b melanoma with an SLN metastasis but would
correctly avoid biopsy in 131 of 310 node-negative patients. It
should be emphasized that a selection technique can miss SLN
metastases but achieve benefit via avoidance of more than 19
negative SLN biopsies for each missed SLN metastasis (at a 5%
risk threshold).
Among patients with T2a and T2b melanoma, the model
also was associated with improvements in net benefit (+0.002
for both) and relative utility (+1% and +2%, respectively) com-
pared with biopsy all. In clinical terms, use of the model could
lead to 1 more net true positive result identified for approxi-
mately every 500 patients with each of T2a and T2b disease,
respectively. Compared with biopsy all, this is equivalent to a
3% to 4% decrease in unnecessary SLN biopsies with no in-
crease in the number of patients with an SLN metastasis left
untreated.
Discussion
These analyses suggest that use of i31-GEP-SLNB to risk stratify
patients for SLN biopsy appears most promising for patients
diagnosed with T1b melanoma. Prior to adoption, however, ad-
ditional prospective studies confirming the accuracy of the
model are needed. Of note, Whitman et al2 determined SLN
status via clinical examination alone (ie, no SLNB) for 25% of
the validation cohort (approximately 15% T1b cases), poten-
tially affecting the reported classification estimates. Second,
few patients with T1b melanoma with SLN metastasis were en-
rolled (n = 18), resulting in uncertainty in the sensitivity esti-
mate and a lower bound of the net benefit 95% CI that crossed
zero. Lastly, it remains untested whether i31-GEP-SLNB has
greater utility compared with free clinicopathologic-based
prediction models; such a comparison should be conducted
on an independent external validation data set.13-15
The i31-GEP-SLNB model was found to be associated with
a negative net benefit for patients with T1a-HR melanoma, sug-
gesting potential for harm. These data should be interpreted
cautiously because they are affected by the low PPV of the
model (ie, among those in which the risk of SLN metastasis was
predicted to be ≥5%, the observed risk was 4%). The SLN sta-
tus was determined via clinical examination alone for 60% of
patients with T1a-HR disease, so it is possible that the true PPV
is greater than 5% if only 1 or more patients had been incor-
jamadermatology.com
Utility of a Model for Predicting the Risk of Sentinel Lymph Node Metastasis in Patients With Cutaneous Melanoma Brief Report Research

rectly labeled as false positive results. In addition, use of i31-
GEP-SLNB could lead to clinical benefit if a clinician were to
perform SLN biopsy on all high-risk patients with T1a mela-
noma (ie, a 34% net decrease in avoidable biopsies).
Although i31-GEP-SLNB was associated with a net ben-
efit among patients with T2a and T2b melanoma, the abso-
lute magnitude of the difference compared with SLN biopsy
for all patients was small. For every 100 patients with T2 dis-
ease in which the i31-GEP-SLNB is performed, a net of 3 to 4
patients would be spared an unnecessary SLN biopsy (after ad-
justing for missed positive SLN biopsies). Further analyses
across T categories are needed to determine if i31-GEP-SLNB
is cost-effective.
Limitations
A significant limitation was that we could not compute net ben-
efit at other risk thresholds. Although a 5% or greater thresh-
old is commonly used for SLN biopsy decisions, some physi-
cians and patients may choose a different threshold. Future
studies should compare net benefit across a range of relevant
thresholds (5%-10%), which may show a greater difference in
net benefit between i31-GEP-SLNB and SLN biopsy for all.
Second, Whitman et al2 ascertained SLN status via clinical ex-
amination alone for many T1 cases, potentially affecting the
accuracy of our estimates. Finally, net benefit analyses as-
sume that a risk threshold has been chosen rationally, and that
the expected benefits and harms are the same for all patients,
independent of predicted risk.
Conclusions
At a risk threshold of 5%, the clinical benefit associated with
i31-GEP-SLNB appeared most promising in risk-stratifying pa-
tients with T1b melanoma. Further prospective study of these
patients is needed to define if i31-GEP-SLNB adds benefit be-
yond free clinicopathologic-based prediction models, its cost-
effectiveness, and its utility at other risk thresholds.

ARTICLE INFORMATION 15, 2021. https://www.nccn.org/professionals/ Med Decis Making. 2006;26(6):565-574.

Accepted for Publication: March 4, 2022. physician_gls/pdf/cutaneous_melanoma.pdf doi:10.1177/0272989X06295361

Published Online: April 27, 2022.
doi:10.1001/jamadermatol.2022.0970
Author Contributions: Michael Marchetti, MD, had
full access to all the data in the study and takes
responsibility for the integrity of the data and the
accuracy of the data analysis. Drs Marchetti and
Bartlett contributed equally.
Concept and design: Marchetti, Bartlett.
Acquisition, analysis, or interpretation of data:
All authors.
Drafting of the manuscript: Marchetti, Bartlett.
Critical revision of the manuscript for important
intellectual content: All authors.
Statistical analysis: Marchetti, Dusza.
Conflict of Interest Disclosures: Dr Bartlett
reported institutional research support from
SkylineDx and personal fees from Excite
International, LLC, outside the submitted work.
No other disclosures were reported.
Funding/Support: This research was funded in part
through the Memorial Sloan Kettering Cancer
Center institutional National Institutes of Health
(NIH)/National Cancer Institute Cancer Center
Support Grant P30 CA008748.
Role of the Funder/Sponsor: The NIH had no role
in the design and conduct of the study; collection,
management, analysis, and interpretation of the
data; preparation, review, or approval of the
manuscript; and decision to submit the manuscript
for publication.
REFERENCES
1. NCCN Clinical Practice Guidelines in Oncology -
Cutaneous Melanoma. 2021; Accessed December
2. Whitman ED, Koshenkov VP, Gastman BR, et al.
Integrating 31-gene expression profiling with
clinicopathologic features to optimize cutaneous
melanoma sentinel lymph node metastasis
prediction. JCO Precis Oncol. 2021;5:5. doi:10.1200/
PO.21.00162
3. Kerr KF, Brown MD, Zhu K, Janes H. Assessing
the clinical impact of risk prediction models with
decision curves: guidance for correct interpretation
and appropriate use. J Clin Oncol. 2016;34(21):
2534-2540. doi:10.1200/JCO.2015.65.5654
4. Vickers AJ, Van Calster B, Steyerberg EW. Net
benefit approaches to the evaluation of prediction
models, molecular markers, and diagnostic tests. BMJ.
2016;352:i6. doi:10.1136/bmj.i6
5. Fitzgerald M, Saville BR, Lewis RJ. Decision curve
analysis. JAMA. 2015;313(4):409-410. doi:10.1001/
jama.2015.37
6. Localio AR, Goodman S. Beyond the usual
prediction accuracy metrics: reporting results for
clinical decision making. Ann Intern Med. 2012;157
(4):294-295. doi:10.7326/0003-4819-157-4-
201208210-00014
7. Holmberg L, Vickers A. Evaluation of prediction
models for decision-making: beyond calibration and
discrimination. PLoS Med. 2013;10(7):e1001491.
doi:10.1371/journal.pmed.1001491
8. Vickers AJ, van Calster B, Steyerberg EW.
A simple, step-by-step guide to interpreting
decision curve analysis. Diagn Progn Res. 2019;3:18.
doi:10.1186/s41512-019-0064-7
9. Vickers AJ, Elkin EB. Decision curve analysis:
a novel method for evaluating prediction models.
10. Agnese DM, Abdessalam SF, Burak WE Jr,
Magro CM, Pozderac RV, Walker MJ.
Cost-effectiveness of sentinel lymph node biopsy in
thin melanomas. Surgery. 2003;134(4):542-547.
doi:10.1016/S0039-6060(03)00275-7
11. Hu Y, Briggs A, Gennarelli RL, et al. Sentinel
lymph node biopsy for T1b melanoma: balancing
prognostic value and cost. Ann Surg Oncol. 2020;27
(13):5248-5256. doi:10.1245/s10434-020-08558-8
12. CMS. Accessed December 17, 2021. https://www.
cms.gov/medicaremedicare-fee-service-
paymentclinicallabfeeschedclinical-laboratory-fee-
schedule-files/21clabq2
13. Lo SN, Ma J, Scolyer RA, et al. Improved risk
prediction calculator for sentinel node positivity in
patients with melanoma: the Melanoma Institute
Australia Nomogram. J Clin Oncol. 2020;38(24):
2719-2727. doi:10.1200/JCO.19.02362
14. Wong SL, Kattan MW, McMasters KM, Coit DG.
A nomogram that predicts the presence of sentinel
node metastasis in melanoma with better
discrimination than the American Joint Committee
on Cancer staging system. Ann Surg Oncol. 2005;12
(4):282-288. doi:10.1245/ASO.2005.05.016
15. Maurichi A, Miceli R, Eriksson H, et al. Factors
Affecting Sentinel node metastasis in thin (T1)
cutaneous melanomas: development and external
validation of a predictive nomogram. J Clin Oncol.
2020;38(14):1591-1601. doi:10.1200/JCO.19.01902

jamadermatology.com (Reprinted) JAMA Dermatology June 2022 Volume 158, Number 6 683

© 2022 American Medical Association. All rights reserved.

Copyright of JAMA Dermatology is the property of American Medical Association and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission. However, users may print, download, or email
articles for individual use.