European Journal of Surgical Oncology 48 (2022) 492e499

Contents lists available at ScienceDirect

European Journal of Surgical Oncology

journal homepage: www.ejso.com

Systematic review and meta-analysis of validated prognostic models

for resected hepatocellular carcinoma patients
Berend R. Beumer a, Stefan Buettner a, Boris Galjart a, Jeroen L.A. van Vugt a,
Robert A. de Man b, Jan N.M. IJzermans a, Bas Groot Koerkamp a, *
a
Erasmus MC Transplant Institute, Department of Surgery Division of HPB & Transplant Surgery, Erasmus MC, University Medical Centre Rotterdam,
Rotterdam, the Netherlands
b
Erasmus MC Transplant Institute, Department of Gastroenterology and Hepatology, Erasmus MC, University Medical Centre Rotterdam, Rotterdam, the
Netherlands

a r t i c l e i n f o a b s t r a c t

Article history: Background: Many prognostic models for Hepatocellular Carcinoma (HCC) have been developed to
Received 21 June 2021 inform patients and doctors about individual prognosis. Previous reviews of these models were quali-
Received in revised form tative and did not assess performance at external validation. We assessed the performance of prognostic
13 September 2021
models for HCC and set a benchmark for biomarker studies.
Accepted 15 September 2021
Methods: All externally validated models predicting survival for patients with resected HCC were sys-
Available online 21 September 2021
tematically reviewed. After selection, we extracted descriptive statistics and aggregated c-indices using
meta-analysis.
Keywords:
External validation
Results: Thirty-eight validated prognostic models were included. Models used on average 7 (IQR:4e9)
Prognostic model prognostic factors. Tumor size, tumor number, and vascular invasion were almost always included.
Liver resection Alpha-fetoprotein (AFP) was commonly incorporated since 2007. Recently, the more subjective items
Hepatocellular carcinoma ascites and encephalopathy have been dropped. Eight established models performed poor to moderate at
Survival external validation, with a pooled C-index below 0.7; including the Barcelona Clinic Liver Cancer (BCLC)
Recurrence system, the American Joint Committee on Cancer (AJCC) 7th edition, the Cancer of the Liver Italian (CLIP)
C-index Program, and the Japan Integrated Staging (JIS) score. Out of 24 prognostic models predicting OS, only 6
(25%) had good performance at external validation with pooled C-indices above 0.7; the Li-post (0.77), Li-
OS (0.74), Yang-pre (0.74), Yang-post (0.76), Shanghai-score (0.70), and Wang-nomogram (0.71). Models
improved over time, but overall performance and study quality remained low.
Conclusions: Six validated prognostic models demonstrated good performance for predicting survival
after resection of HCC. These models can guide patients and doctors and are a benchmark for future
models incorporating novel biomarkers.
© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).

1. Introduction models may describe mere artefacts caused by local practices in

Prognosis is important in the care for patients with hepatocel-
lular carcinoma (HCC). To aid in the difficult task of predicting a
patient's prognosis, previous experiences can be used to develop a
prognostic model.
Immediately after prognostic models are created, concerns are
raised about their generalizability. Rather than a general pattern,
* Corresponding author. 40 3015 GD Rotterdam Department of surgery Room,
RG-219, the Netherlands.
E-mail address: b.grootkoerkamp@erasmusmc.nl (B.G. Koerkamp).
patient selection, care, and data collection. Therefore, external
validation, preferably by an independent research group, is
required prior to implementation into clinical practice.
Numerous prognostic models for HCC have been developed. The
most established ones are the Barcelona Clinic Liver Cancer (BCLC)
system, the American Joint Committee on Cancer (AJCC) 7th edi-
tion, the Cancer of the Liver Italian Program (CLIP), and the Japan
Integrated Staging (JIS) score. Nonetheless, new proposals for
staging systems are being published on a monthly basis [1e3].
Several literature overviews of prognostic models for HCC have
been published [4e8]. These reviews broadly discuss the historical
context and classification schemes of the most common staging

https://doi.org/10.1016/j.ejso.2021.09.012
0748-7983/© 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
B.R. Beumer, S. Buettner, B. Galjart et al.
systems. These reviews, however, lack a systematic approach to the
inclusion of models. Moreover, conclusions are biased, because they
are primarily informed by the performance in development studies,
rather than external validations studies.
Therefore, the primary aim of this study is to assess and
compare model performance of all externally validated prognostic
models for survival outcomes of patients after resection of HCC.
Secondly, we aim to determine trends in the selection of prognostic
factors and model performance.
2. Method
This study was embedded in the ‘Validated surgical models of
hepatobiliary malignancies’ (VALIDATE) initiative, in which we
appraised all externally validated prognostic models developed for
hepato-pancreato-biliary malignancies. In the present study,
prognostic models used to stage patients with HCC were investi-
gated. The scope of this research was further narrowed to external
validations in patients who had undergone first time liver resec-
tion. The study was not prospectively registered. We performed a
systematic review and meta-analysis in accordance with the
PRISMA and CHARMS guidelines (Supplemental Digital Content 1)
[9,10].
2.1. Selection criteria
An experienced librarian performed a search of Embase, Med-
line, Web of Science, the Cochrane database and Google Scholar
using the search terms provided in Supplemental Digital Content 2.
The last search was conducted on the 18th of July 2019.
Eligible studies were those in which an external validation was
performed of a prognostic model in resected HCC patients with
either disease-free survival (DFS) or overall survival (OS) as
endpoint. Studies written in English and published after 1990 were
considered. Reviews, expert opinions, abstracts, and posters were
excluded. Studies with prognostic models containing prognostic
factors not readily available in clinical practice (e.g., RNA/DNA
sequencing data or liquid biopsies) were also excluded. Further-
more, studies on recurrence of malignancies were excluded. Studies
that in their analysis pooled data of resected patients with data of
patients treated without resection were excluded. Lastly, studies
were excluded that did not report model performance in terms of a
Harrell's concordance index (c-index) or area under the curve
(AUC) of the receiver operating characteristic (ROC) with specifi-
cation of the censor time.
Three reviewers (SB, BGa, and BB) independently screened the
abstracts of all studies identified by the search. Eligibility was
determined by reviewing the full manuscript of potentially relevant
studies. Disagreement between the reviewers was resolved by
discussion. Descriptive, methodological, and outcome data for each
of the included studies was extracted by BB and SB.
2.2. Validation studies
Descriptive statistics included year of publication, country,
medical center, inclusion period, selection criteria, sample size, and
type of outcome. If a study externally validated more than one
prognostic model, or used multiple independent cohorts, data was
extracted for each instance.
Discriminatory performance was assessed using the c-index or
the AUC-ROC at 1, 3, and 5 years after surgery. The c-index served as
the principal measure of external validity, because it is a commonly
reported discrimination measure, remains unbiased in the pres-
ence of censoring, and does not require an arbitrary cut-off point.
The c-index reflects the probability that predictions and
493
European Journal of Surgical Oncology 48 (2022) 492e499
observations of two random patients are concordant. A c-index of
0.5 indicates no predictive discrimination; a value of 1.0 indicates a
perfect risk ordering of all patients. A c-index of below 0.60 was
considered poor; a value between 0.60-0.69 was considered mod-
erate; and a value above 0.70 was considered good [11].
A meta-analysis was performed using meta package for R 3.6.1 if
there were multiple external validations available that reported the
confidence interval or a standard error alongside their c-index
[12,13]. Pooling was performed on the probability scale using in-
verse variance weighting [14]. We applied a random effects model
with the DerSimonian-Laird method to estimate between study
variance. For studies with overlapping populations, only the largest
cohort was considered for pooling. Heterogeneity was investigated
by means of a forest plot and I2 statistic [15]. To summarize these
results and inspect the across model performance, the pooled c-
indices and year of publication are displayed in a weighted scatter
plot.
2.3. Development studies
The development study is the publication in which the model is
first presented. The reference lists of included validation studies
were screened to identify the development studies of the prog-
nostic models. From these studies, we extracted the following data:
year of publication, country, medical center, inclusion period,
number of patients, patient selection criteria, estimation technique,
end point definition, number of events, median follow-up
(months), number of coefficients (incl. candidate prognostic fac-
tors, transformations, indicator variables of categorical prognostic
factors, and interaction terms), type of information incorporated in
the final model, number of patients in internal validation cohort,
and model performance at internal validation. Effective sample size
was calculated as the number of events divided by the total number
of coefficients. For interpretation the one in ten rule of thumb was
used [16,17]. If the number of events was poorly reported, the
fraction of the cohort size and total number of parameter estimates
was used as a proxy.
2.4. Methodological and reporting quality
The quality of included studies was assessed independently by
two authors (BB and SB). As widely adopted consensus guidelines
are lacking, we used the items of the CHARMS checklist and TRIPOD
statement to score the development studies [9,18]. Items were
grouped in categories (e.g. patient selection, handling of missing
data, and candidate predictors). The aggregate score per category
was low (1) if all items were listed in the paper, medium (2) if at
least one item was not reported, and high (3) if more than one item
was missing. An aggregate score per study was constructed by
summation of the category scores. Study was considered at low risk
of bias if the sum score was lower than 12, medium between 13-14
and high for scores above 15. We only scored the studies in which
the model was derived using statistical methods (rather than
expert opinion), as most items concern collection, preparation, and
analysis of the data. The consensus-derived models were consid-
ered to be at high risk of bias by default.
3. Results
The search identified 10,282 potentially relevant citations for all
HPB malignancies. After screening of the titles and abstracts, we
selected 451 studies for full-text assessment. Ultimately, we
included 36 external validation studies in this review (Fig. 1). These
36 studies performed 150 validations for DFS and 190 validations
for OS of 38 different prognostic models.
B.R. Beumer, S. Buettner, B. Galjart et al.
Fig. 1. Flowchart studies' screening and selection.
3.1. Development studies
Table 1 displays the key characteristics of the 38 prognostic
models. Additional information about the development studies is
presented in Supplemental Digital Content 3. The prognostic
models were developed between 1984 and 2019. They were pri-
marily based on Asian cohorts with the exception of the CLIP, BCLC,
(modified) Glasgow prognostic score (GPS), and the Memorial
Sloan Kettering Cancer Center (MSKCC) nomogram. The sample size
for model development was on average 451 patients (interquartile
range (IQR): 310e850). Inclusions spanned on average 9 years (IQR:
6e12). The median number of patients per estimate was 13
(IQR:7e23), and the median number of events per coefficient was
12 (IQR:7e24).
Models developed before 2002 typically included the total HCC
population, regardless of stage or treatment. In comparison, models
developed after 2015 increasingly target subgroups such as patients
undergoing resection of multiple tumors or patients with portal
vein tumor thrombosis (PVTT). Before 2015, the MSKCC nomogram
was the only model for DFS, rather than OS. After 2015, DFS was the
endpoint in the majority (10/18) of models (Supplemental Digital
Content 3).
The median number of prognostic factors in the risk score was 7
(IQR: 4e9). Traditional tumor characteristics such as tumor size,
tumor number, and vascular invasion were almost always included.
Alpha-fetoprotein (AFP) was commonly incorporated since 2007;
15 out of 25 models (60%). Recently, the more subjective prognostic
factors ascites and encephalopathy have been dropped. Prognostic
factors were selected based on the literature and expert opinion
(i.e. consensus based) in thirteen models (34%). All other models
used statistical methods to determine the independent prognostic
494
European Journal of Surgical Oncology 48 (2022) 492e499
value of each factor; 20 of the 38 models (53%) used Cox propor-
tional hazards, 3 logistic regression, and only one (HKCL model)
used a decision-tree algorithm. Sixteen (42%) prognostic models
created risk groups using arbitrary cut-off points and displayed
Kaplan-Meier curves for each risk group.
Models marked with an * have a pooled C-index > 0.7. For
consensus models only the pooled c-index for OS validations is
shown. For models with NA pooling was not performed as no
confidence intervals were reported in the validation studies. Ab-
breviations: C-reactive protein (CRP), Eastern cooperative oncology
group (ECOG), Alkaline Phospatase (Aphos), Gamma-glutamyl
transferase (gGT), Aspartate aminotransferase (AST), Sodium (Na),
Lymphocytes (Lympho), Neutrocytes (Neutro), Hepatitis B (HBV).
Most models (45%) divided the study population in a separate
training and test set. In these studies, on average 70% (IQR 67e75)
of the data was assigned to the training cohort. Two studies divided
the population multiple times and averaged model performance
(i.e. cross-validation) [34,39] and one model used bootstrap
resampling [37]. Three studies used only the development cohort to
describe model performance [26,40,47].
3.2. Validation studies
A total of 32,825 patients from 36 validation studies provided
data for 340 validations [39,40,42,43,45e76]. Apart from two
studies [49,57] all validation cohorts were Asian, of which the
majority originated from China. A full listing of all validation
studies, their design, and results per model are presented for DFS in
Supplemental Digital Content 4 and for OS in Supplemental Digital
Content 5.
The median sample size in which prediction models were vali-
dated was 349 (IQR: 212e768). The BCLC system and AJCC 7th
edition were validated most frequently, with 30 and 29 validations
for OS, and 24 and 22 validations for DFS. For DFS 7 out of 27 (26%)
models were externally validated only once. For OS this was 6 out of
24 model (25%).
Eligibility criteria for patients included in the validation studies
were typically: good performance status, no previous treatment,
resection, and pathologically proven HCC. Additional criteria in one
or more studies required patients with an R0 resection [45,46,51],
the presence or absence of HBV [43,46,50], PVTT [51], large HCC
[42], and multiple-lesion HCC [44].
3.3. Meta-analysis of model performance (c-indices)
Meta-analysis of c-indices was performed for 33 models. Data
from 17 of the 36 validation studies could be used. Fig. 2 shows the
forest plots of 8 established models for OS validations. A complete
overview of all forest plots for DFS and OS of all prognostic models
is available in Supplemental Digital Content 6 and 7 and in table
format in Supplemental Digital Content 8 and 9.
The pooled c-indices at validation of the 22 prognostic models
for OS ranged from 0.52 to 0.77. Several established models per-
formed poor to moderate at external validation, with a pooled C-
index below 0.7; BCLC (0.61), AJCC 7th (0.62), CLIP (0.63), and JIS
(0.61). Six models had good performance with a pooled c-index
above 0.7: the Li-post (0.77), Li-OS (0.74), Yang-pre (0.74), Yang-
post (0.76), Shanghai-score (0.70), and Wang-nomogram (0.71)
[42e46]. These models were, however, only validated on a limited
number of external validation cohorts (<3) and is there no confir-
mation from an independent research group.
The number of prognostic factors for the 6 best models ranged
from 7 (Li models) to 15 (Shanghai model). All 6 models require
tumor size and number of tumors, 5 require vascular invasion and
AFP, and 3 require encapsulation, HBV, bilirubin, and creatinine
B.R. Beumer, S. Buettner, B. Galjart et al.
Table 1
All validated prognostic models [19e51].
(Table 1).
Fig. 3 displays the (pooled) C-indices from external OS valida-
tions versus year of publication of the development study. The
figure for the DFS setting is displayed in Supplemental Digital
content 10. For both outcome measures model performance
increased over time.
3.4. Risk of bias
The risk of bias for the development studies is summarized in
Supplemental Digital Content 11, and a detailed overview is pre-
sented in Supplemental Digital Content 12. Out of the 38 models 13
models were expert based leaving 25 statistically derived models to
be investigated for risk of bias. The risk of bias was scored low for 7
models (28%), intermediate for 10 models (40%), and high for 8
models (32%). None of the studies reported the data source (e.g.,
cohort study, randomized control trial or registry) and whether the
data was collected retrospectively or prospectively. The inclusion
criteria, baseline characteristics, and inclusion period were always
stated. However, the number of patients with missing values was
only reported for 4 of the 25 models (16%).
In 10 of the 25 models, the definitions of OS and DFS did not
specify how and when they were assessed. Only 8 of the studies
(27%) reported the time of measurement of the prognostic factors
(e.g., whether tumor size was measured preoperatively on imaging
or postoperatively at pathological examination). None of the
studies investigated non-linearity of continuous prognostic factors.
495
European Journal of Surgical Oncology 48 (2022) 492e499
Model discriminatory performance at internal validation was
assessed in 11 of the 30 studies (36%) using the c-index and
accompanied by a confidence interval in 8 studies (72%). Only 9
development studies (30%) inspected the calibration of the model.
4. Discussion
In this systematic review and meta-analysis, we compared all 38
validated prognostic models for survival after resection of HCC.
Established models, such as the BCLC, CLIP, and AJCC performed
moderate to poor at external validation. Six models had good per-
formance for OS at external validation; the Li-post, Li-OS, Yang-pre,
Yang-post, Shanghai score, and Wang nomogram [42e46]. More-
over, we found that prognostic models improved over time with
better performance and fewer ambiguous prognostic factors such
as ascites and encephalopathy.
Several other reviews of prognostic models for HCC have been
published [4e8]. These reviews, however, were qualitative in na-
ture, did not require external validation, and did not perform meta-
analysis for model performance (c-indices). They conclude that
there is no ideal staging system accounting for the geographical and
etiological differences [4,5]. Most reviews conclude that the BCLC is
the most comprehensive and commonly used staging system. The
main advantage of the BCLC system is that each stage has specific
treatment recommendations [24]. The prognostic performance of
the BCLC system, however, is limited because some known prog-
nostic factors (e.g., AFP) are unaccounted for. We found that the
B.R. Beumer, S. Buettner, B. Galjart et al.
Fig. 2. Pooled c-indices of validations of 8 established models. The forest plots show the individual validations in relation to each other for eight established prognostic models. A
study can appear multiple times if in the same study multiple independent cohorts were used.
BCLC staging system had poor to moderate prognostic performance
in 17 external validation studies with a pooled c-index of 0.61.
The six models that achieved a good prognostic performance all
required tumor size and tumor number as prognostic factor. Most
models also included vascular invasion and AFP. They mainly
differed regarding tumor differentiation, capsulation, HBV status,
and laboratory values (i.e. bilirubin, INR, creatinine, and sodium).
The two Li models have a high c-index of about 0.75 and require
only 7 prognostic factors. The Li-post model was developed for
patients with large HCC; the Li-OS model for patients with HBV-
related early-stage HCC. The Li models are attractive models to
predict OS after resection of HCC in the postoperative setting,
496
European Journal of Surgical Oncology 48 (2022) 492e499
because they are concise and perform well. The Li models cannot be
used in the preoperative setting, because they require factors that
are only available after resection (e.g., microvascular invasion).
The two Yang models also had a good performance with a c-
index of about 0.75. The main difference with the Li models was
that the Yang models required additional laboratory values. The
preoperative Yang model was based on only preoperatively avail-
able factors; the postoperative Yang model added 4 postoperative
factors. Performance of the preoperative and postoperative Yang
models were similar at external validation. The preoperative model
of Yang is the best validated model to predict OS after resection of
HCC in the preoperative setting.
B.R. Beumer, S. Buettner, B. Galjart et al.
Fig. 3. Pooled c-indices for OS versus year of publication of the development study. Each point represents a model evaluated for OS with its size corresponding to the total
number of patients on which the performance estimate is based. The horizontal dashed lines represent the performance thresholds.
Prognostic models such as the preoperative model of Yang can
be used to inform patients and doctors in the preoperative setting
about life expectancy after resection of HCC. The potential benefit of
surgery should be balanced against surgical mortality that can be
high in patients with advanced age, cirrhosis, and requiring major
hepatectomy. Models that require postoperative factors such as the
models of Li can be used to inform patients and doctors in the
postoperative setting. These prognostic models can also be used to
stratify patients in RCT's. Moreover, individual prediction of prog-
nosis could guide personalization of follow-up schemes. The
theoretical ground work that links prognostic models to the fre-
quency of scans has been published in 1983 [77].
The risk of bias was high in most prognostic models. Moreover,
the quality of most models was poor. Quality could be improved if
non-linear prognostic factors and missing values are better
addressed. Moreover, explicitly creating separate pre- and post-
operative models would increase applicability. We strongly
recommend that adherence to reporting guidelines (TRIPOD
statement and CHARMS checklist) becomes a prerequisite for
publication of prognostic models [9,18]. Moreover, researchers
should focus on independent external validation studies rather
than only developing more models. The field is too focused on
model creation, whereas model validation, and implementation
receive little attention. This trend is in line with systematic reviews
from other specialties and needs to be addressed in order to in-
crease applicability of prognostic models [78,79].
Several limitations of our study are important. First, the out-
comes of pooled analyses cannot be regarded as the true model
performance, because of considerable heterogeneity across cohorts.
Furthermore, although good model performance at external vali-
dation is necessary, it offers no guarantee that a model will
generalize to other populations. Although, East-West differences in
performance of resection models are in our view likely, due to the
focus on transplantation in western validation studies, they could
not be examined. We hypothesize that the focus of validation
studies is a reflection of the preferred HCC treatment in the two
regions. Eastern cohorts are more often offered resection, in
contrast, in western cohorts increasingly more HCC patients receive
liver transplantation [80,81]. Nevertheless, the proportion of pa-
tients receiving liver resection in the West is still about 50% of those
surgically treated [82,83]. These patients would benefit from
dedicated external validation studies.
497
European Journal of Surgical Oncology 48 (2022) 492e499
Also, we recognize that the estimated performance of more
recent models might decrease when more validations become
available. Therefore, electing a single best model is complicated as
it requires a tradeoff between performance and the uncertainty in
the performance estimate. Nonetheless, our pooled analyses pro-
vide the best possible summary and a clear overview of the current
landscape. Second, we only included models with published
external validations. We recognize that due to this restriction
several recent models (that have not yet been validated) were not
included [74,84,85]. Also, models containing prognostic factors not
readily available in clinical practice are missing in our analysis.
Furthermore, we would like to emphasize that prognostic models
are a means to an end rather than an end in itself. Most prognostic
models do not make the final step towards clinical decision making.
The BCLC system is the main exception. The link to treatment might
be the reason why the European association for the study of the
liver (EASL) and the European society for medical oncology (ESMO)
still recommend the BCLC despite its moderate performance for
prognosis [86,87]. Further research should investigate how the best
prognostic models can guide treatment decisions [88,89].
The American Association for the Study of Liver Diseases
(AASLD) guidelines also stated that with the typical set of clinical
prognostic factors only marginal improvements can be made in
prognostic models [90]. Serum biomarkers and gene signatures are,
however, a dynamic and promising field that aims to improve
prognostic and predictive models. The best prognostic models of
this meta-analysis serve as benchmarks for novel biomarkers. Such
biomarkers are typically costly to measure and are only useful if it is
demonstrated that they can improve performance of the best
models that only require readily available patient and tumor
characteristics.
In conclusion, we identified 6 prognostic models for OS after
resection of HCC that achieved good performance at external vali-
dation [42e46]. These models are a benchmark for future models
incorporating novel biomarkers.
CRediT authorship contribution statement
Berend R. Beumer: screened the articles, performed the data
extraction, Formal analysis, wrote the paper which was edited by all
co-authors. Stefan Buettner: screened the articles, performed the
data extraction. Boris Galjart: screened the articles. Jeroen L.A. van
B.R. Beumer, S. Buettner, B. Galjart et al.
Vugt: Conceptualization. Robert A. de Man: Methodology. Jan N.M.
IJzermans: Methodology. Bas Groot Koerkamp: Methodology, All
authors have approved the final version of the manuscript
including the authorship list.
Declaration of competing interest
None.
Acknowledgements
Wichor M. Bramer's help with constructing the search term is
gratefully acknowledged.
Appendix A. Supplementary data
Supplementary data to this article can be found online at
https://doi.org/10.1016/j.ejso.2021.09.012.
Financial support statement
None.
References
[1] Lin C-Y, et al. The ALBI grade is a good predictive model for very late recur-
rence in patients with hepatocellular carcinoma undergoing primary resec-
tion. World J Surg 2019:1e11.
[2] Tsilimigras DI, et al. Utilizing machine learning for pre-and postoperative
assessment of patients undergoing resection for BCLC-0, A and B hepatocel-
lular carcinoma: implications for resection beyond the BCLC guidelines. Ann
Surg Oncol 2019:1e9.
[3] Xue Y, et al. Establishment and validation of a prognostic model for hepato-
cellular carcinoma after radical liver resection. Journal of BU ON.: official
journal of the Balkan Union of Oncology 2019;24(4):1420e8.
[4] Tellapuri S, et al. Staging systems of hepatocellular carcinoma: a review. In-
dian J Gastroenterol 2018;37(6):481e91.
[5] Karademir S. Staging of hepatocellular carcinoma. 2018.
[6] Adhoute X, et al. Usefulness of staging systems and prognostic scores for
hepatocellular carcinoma treatments. World J Hepatol 2016;8(17):703.
[7] Maida M, et al. Staging systems of hepatocellular carcinoma: a review of
literature. World J Gastroenterol: WJG 2014;20(15):4141.
[8] Marrero JA, Kudo M, Bronowicki J-P. The challenge of prognosis and staging
for hepatocellular carcinoma. Oncol 2010;15(Supplement 4):23e33.
[9] Moons KG, et al. Critical appraisal and data extraction for systematic reviews
of prediction modelling studies: the CHARMS checklist. PLoS Med
2014;11(10):e1001744.
[10] Moher D, et al. Preferred reporting items for systematic reviews and meta-
analyses: the PRISMA statement. Ann Intern Med 2009;151(4):264e9.
[11] Borges LSR. Diagnostic accuracy measures in cardiovascular research. Inter-
national Journal of Cardiovascular Sciences 2016;29(3):218e22.
[12] Schwarzer G. meta: an R package for meta-analysis. R News 2007;7(3):40e5.
[13] R Development Core Team R. A language and Environment for statistical
Computing. Vienna, Austria: R Foundation for Statistical computing; 2019.
[14] van Klaveren D, et al. Assessing discriminative ability of risk models in clus-
tered data. BMC Med Res Methodol 2014;14(1):5.
[15] Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat
Med 2002;21(11):1539e58.
[16] Harrell Jr FE, Lee KL, Mark DB. Multivariable prognostic models: issues in
developing models, evaluating assumptions and adequacy, and measuring
and reducing errors. Stat Med 1996;15(4):361e87.
[17] Harrell Jr FE, et al. Regression modelling strategies for improved prognostic
prediction. Stat Med 1984;3(2):143e52.
[18] Collins GS, et al. Transparent reporting of a multivariable prediction model for
individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med
2015;13(1):1.
[19] Onodera T, Goseki N, Kosaki G. Prognostic nutritional index in gastrointestinal
surgery of malnourished cancer patients. Nihon Geka Gakkai Zasshi
1984;85(9):1001e5.
[20] Okuda K, et al. Natural history of hepatocellular carcinoma and prognosis in
relation to treatment study of 850 patients. Cancer 1985;56(4):918e28.
[21] Sheng J, et al. The Chinese classification system compared with TNM staging
in prognosis of patients with primary hepatic carcinoma after resection.
Hepatobiliary Pancreat Dis Int: HBPD INT 2005;4(4):561e4.
[22] Yan P, Yan L-N. Staging of hepatocellular carcinoma. Hepatobiliary Pancreat
Dis Int: HBPD INT 2003;2(4):491e5.
[23] Yang Bh RZ, Tang ZY. The consideration and proponent of the liver cancer
498
European Journal of Surgical Oncology 48 (2022) 492e499
staging classification. Chin J Hepatobiliary Surg 1999;5:67e8.
[24] Llovet JM, Brú C, Bruix J. Prognosis of hepatocellular carcinoma: the BCLC
staging classification. In: Seminars in liver disease. © 1999 by Thieme Medical
Publishers, Inc; 1999.
[25] Greene FL, et al. AJCC cancer staging handbook: TNM classification of malig-
nant tumors. Springer Science & Business Media; 2002.
[26] Kudo M, Chung H, Osaki Y. Prognostic staging system for hepatocellular car-
cinoma (CLIP score): its value and limitations, and a proposal for a new
staging system, the Japan Integrated Staging Score (JIS score). J Gastroenterol
2003;38(3):207e15.
[27] Nanashima A, et al. The Japanese integrated staging score using liver damage
grade for hepatocellular carcinoma in patients after hepatectomy. Eur J Surg
Oncol 2004;30(7):765e70.
[28] Minagawa M, et al. Staging of hepatocellular carcinoma: assessment of the
Japanese TNM and AJCC/UICC TNM systems in a cohort of 13,772 patients in
Japan. Annals of surgery 2007;245(6):909.
[29] McMillan DC, et al. Evaluation of an inflammation-based prognostic score
(GPS) in patients undergoing resection for colon and rectal cancer. Int J
Colorectal Dis 2007;22(8):881.
[30] Edge SB, Compton CC. The American Joint Committee on Cancer: the 7th
edition of the AJCC cancer staging manual and the future of TNM. Ann Surg
Oncol 2010;17(6):1471e4.
[31] Kasymjanova G, et al. The predictive value of pre-treatment inflammatory
markers in advanced non-small-cell lung cancer. Curr Oncol 2010;17(4):52.
[32] Kamarajah SK, et al. Critical evaluation of the American joint commission on
cancer (AJCC) staging system for patients with hepatocellular carcinoma
(HCC): a surveillance, epidemiology, end results (SEER) analysis. J Surg Oncol
2018;117(4):644e50.
[33] Tokumitsu Y, et al. A new prognostic model for hepatocellular carcinoma
recurrence after curative hepatectomy. Oncol Lett 2018;15(4):4411e22.
[34] Investigators C.o.t.L.I.P.. A new prognostic system for hepatocellular carci-
noma: a retrospective study of 435 patients. Hepatology 1998;28(3):751e5.
[35] Leung TW, et al. Construction of the Chinese University Prognostic Index for
hepatocellular carcinoma and comparison with the TNM staging system, the
Okuda staging system, and the Cancer of the Liver Italian Program staging
system: a study based on 926 patients. Cancer 2002;94(6):1760e9.
[36] Forrest L, et al. Evaluation of cumulative prognostic scores based on the
systemic inflammatory response in patients with inoperable non-small-cell
lung cancer. British journal of cancer 2003;89(6):1028.
[37] Tateishi R, et al. Proposal of a new prognostic model for hepatocellular car-
cinoma: an analysis of 403 patients. Gut 2005;54(3):419e25.
[38] Zheng S-S, et al. Liver transplantation for hepatocellular carcinoma: hangzhou
experiences. Transplantation 2008;85(12):1726e32.
[39] Cho CS, et al. A novel prognostic nomogram is more accurate than conven-
tional staging systems for predicting survival after resection of hepatocellular
carcinoma. J Am Coll Surg 2008;206(2):281e91.
[40] Yang T, et al. A new staging system for resectable hepatocellular carcinoma:
comparison with six existing staging systems in a large Chinese cohort.
J Cancer Res Clin Oncol 2011;137(5):739e50.
[41] Yau T, et al. Development of Hong Kong Liver Cancer staging system with
treatment stratification for patients with hepatocellular carcinoma. Gastro-
enterology 2014;146(7):1691e1700. e3.
[42] Li Y, et al. Prognostic nomograms for pre-and postoperative predictions of
long-term survival for patients who underwent liver resection for huge he-
patocellular carcinoma. J Am Coll Surg 2015;221(5):962e74. e4.
[43] Li J, et al. Nomograms for survival prediction in patients undergoing liver
resection for hepatitis B virus related early stage hepatocellular carcinoma.
European Journal of Cancer 2016;62:86e95.
[44] Yang P, et al. Nomograms for pre-and postoperative prediction of long-term
survival for patients who underwent hepatectomy for multiple hepatocellu-
lar carcinomas. Annals of surgery 2016;263(4):778e86.
[45] Sun HC, et al. Shanghai score: a prognostic and adjuvant treatment-evaluating
system constructed for Chinese patients with hepatocellular carcinoma after
curative resection. Chin Med J 2017;130(22):2650e60.
[46] Wang Y, et al. Novel prognostic nomograms based on Inflammation-related
markers for patients with hepatocellular carcinoma underwent Hepatec-
tomy. 2019.
[47] Ang SF, et al. The Singapore Liver Cancer Recurrence (SLICER) Score for relapse
prediction in patients with surgically resected hepatocellular carcinoma. PloS
One 2015;10(4):e0118658.
[48] Shim JH, et al. Prognostic nomograms for prediction of recurrence and sur-
vival after curative liver resection for hepatocellular carcinoma. Annals of
surgery 2015;261(5):939e46.
[49] Chan AWH, et al. Development of pre and post-operative models to predict
early recurrence of hepatocellular carcinoma after surgical resection. J Hepatol
2018;69(6):1284e93.
[50] Qin W, et al. A novel score predicts HBV-related hepatocellular carcinoma
recurrence after Hepatectomy: a retrospective multicenter study. 2018.
[51] Zhang XP, et al. A nomogram to predict early postoperative recurrence of
hepatocellular carcinoma with portal vein tumour thrombus after R0 liver
resection: a large-scale, multicenter study. Eur J Surg Oncol 2019;45(9):
1644e51.
[52] Chen TW, et al. Comparison of clinical staging systems in predicting survival
of hepatocellular carcinoma patients receiving major or minor hepatectomy.
Eur J Surg Oncol 2007;33(4):480e7.
B.R. Beumer, S. Buettner, B. Galjart et al.
[53] Noda T, et al. Usefulness of the CLIP scoring system for prediction of post-
operative prognosis of patients with large hepatocellular carcinoma.
J Hepatobiliary Pancreat Surg 2009;16(4):538e45.
[54] Xu L-B, et al. Staging systems for predicting survival of patients with hepa-
tocellular carcinoma after surgery. World J Gastroenterol: WJG 2010;16(41):
5257.
[55] Huang JT, et al. A modified TNM-7 staging system to better predict the sur-
vival in patients with hepatocellular carcinoma after hepatectomy. J. Cancer
Res. Clin. Oncol. 2013;139(10):1709e19.
[56] Kinoshita A, et al. The Glasgow Prognostic Score, an inflammation based
prognostic score, predicts survival in patients with hepatocellular carcinoma.
BMC Canc 2013;13.
[57] Santambrogio R, et al. External validation of a simplified BCLC staging system
for early hepatocellular carcinoma. Eur J Surg Oncol 2013;39(8):850e7.
[58] Huang J, et al. The inflammation-based scores to predict prognosis of patients
with hepatocellular carcinoma after hepatectomy. Med Oncol 2014;31(4).
[59] Liu C, et al. Prognosis evaluation in patients with hepatocellular carcinoma
after hepatectomy: comparison of BCLC, TNM and Hangzhou criteria staging
systems. PloS One 2014;9(8).
[60] Ruan DY, et al. Nomogram for preoperative estimation of long-term survival
of patients who underwent curative resection with hepatocellular carcinoma
beyond Barcelona clinic liver cancer stage A1. Oncotarget 2016;7(38):
61378e89.
[61] Shen J, et al. Prognostic nomograms for patients with resectable hepatocelluar
carcinoma incorporating systemic inflammation and tumor characteristics.
Oncotarget 2016;7(49):80783e93.
[62] Sposito C, et al. Development of a prognostic scoring system for resectable
hepatocellular carcinoma. World J Gastroenterol 2016;22(36):8194e202.
[63] Wang JQ, et al. Comparison of the prognostic value of C-reactive protein-
based prognostic scores in patients with hepatitis B virus-related hepatocel-
lular carcinoma. Int J Clin Exp Med 2016;9(5):8272e9.
[64] Yang P, et al. Nomograms for pre- and postoperative prediction of long-term
survival for patients who underwent hepatectomy for multiple hepatocellular
carcinomas. Ann Surg 2016;263(4):778e86.
[65] Feng LH, et al. Novel microvascular invasion-based prognostic nomograms to
predict survival outcomes in patients after R0 resection for hepatocellular
carcinoma. J Cancer Res Clin Oncol 2017;143(2):293e303.
[66] Fu YP, et al. Prognostic nomograms stratify survival of patients with hepa-
tocellular carcinomawithout portal vein tumor thrombosis after curative
resection. Oncol 2017;22(5):561e9.
[67] Cao Y, et al. Prediction of long-term survival rates in patients undergoing
curative resection for solitary hepatocellular carcinoma. Oncol Lett
2018;15(2):2574e82.
[68] Gan W, et al. Prognostic impact of lactic dehydrogenase to albumin ratio in
hepatocellular carcinoma patients with childepugh I who underwent curative
resection: a prognostic nomogram study. Canc Manag Res 2018;10:5383e94.
[69] Hu Y, et al. Nomogram predicting survival of hepatocellular carcinoma with
portal vein tumour thrombus after curative resection. 2018.
[70] Huang JL, et al. A novel and validated prognostic nomogram based on liver
fibrosis and tumor burden for patients with hepatocellular carcinoma after
499
European Journal of Surgical Oncology 48 (2022) 492e499
curative resection. J Surg Oncol 2018;117(4):625e33.
[71] Liao R, et al. Combined preoperative ALBI and FIB-4 is associated with
recurrence of hepatocellular carcinoma after curative hepatectomy.
J Gastrointest Surg 2018;22(10):1679e87.
[72] Tokumitsu Y, et al. A new prognostic model for hepatocellular carcinoma
recurrence after curative hepatectomy. Oncology letters 2018;15(4):4411e22.
[73] Zheng BH, et al. Radiomics score: a potential prognostic imaging feature for
postoperative survival of solitary HCC patients. BMC Cancser 2018;18(1).
[74] Ng KK, et al. A simplified prediction model for early intrahepatic recurrence
after hepatectomy for patients with unilobar hepatocellular carcinoma
without macroscopic vascular invasion: an implication for adjuvant therapy
and postoperative surveillance. Surgical Oncology 2019;30:6e12.
[75] Wang YY, et al. A modified staging of early and intermediate hepatocellular
carcinoma based on single tumour >7 cm and multiple tumours beyond up-
to-seven criteria. Aliment Pharmacol Ther 2019;49(2):202e10.
[76] Xu L, et al. Novel prognostic nomograms for hepatocellular carcinoma patients
with microvascular invasion: experience from a single center. Gut Liver
2019;13(6):669e82.
[77] Dwyer AJ, et al. Use of the hazard rate to schedule follow-up exams efficiently:
an optimization approach to patient management. Med Decis Making
1983;3(2):229e44.
[78] Damen JA, et al. Prediction models for cardiovascular disease risk in the
general population: systematic review. bmj 2016;353:i2416.
[79] Moons KG, et al. Prognosis and prognostic research: application and impact of
prognostic models in clinical practice. Bmj 2009;338:b606.
[80] Akamatsu N, et al. Surgery and hepatocellular carcinoma. Liver cancer
2017;6(1):44e50.
[81] Belghiti J, Fuks D. Liver resection and transplantation in hepatocellular car-
cinoma. Liver cancer 2012;1(2):71e82.
[82] Chapman WC, et al. Surgical treatment of hepatocellular carcinoma in North
America: can hepatic resection still be justified? J Am Coll Surg 2015;220(4):
628e37.
[83] Liver E.A.F.T.S.O.T.. EASL clinical practice guidelines: management of hepato-
cellular carcinoma. J. Hepatol. 2018;69(1):182e236.
[84] Cai B-B, et al. Use of a novel index, the A-index, and its associated nomogram
to predict overall survival rates after resection of primary hepatocellular
carcinoma. Clin Chim Acta 2020;500:34e41.
[85] Lin, Z.-X., et al., Controlling nutritional status (CONUT) score-based nomogram
to predict overall survival of patients with HBV-associated hepatocellular
carcinoma after curative hepatectomy. Clin Transl Oncol: p. 1-11.
[86] Galle PR, et al. EASL clinical practice guidelines: management of hepatocel-
lular carcinoma. Journal of hepatology 2018;69(1):182e236.
[87] Vogel A, et al. Hepatocellular carcinoma: ESMO Clinical Practice Guidelines for
diagnosis, treatment and follow-up. Ann Oncol 2018;29(Supplement_4):
iv238e55.
[88] Steyerberg EW. Clinical prediction models, 381. Springer; 2009.
[89] Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating
prediction models. Med Decis Making 2006;26(6):565e74.
[90] Heimbach JK, et al. AASLD guidelines for the treatment of hepatocellular
carcinoma. Hepatology 2018;67(1):358e80.