European Journal of Preventive Cardiology (2023) 30, 1293–1303 FULL RESEARCH PAPER

https://doi.org/10.1093/eurjpc/zwad198 Risk prediction/assessment & stratification

Validation and comparison of cardiovascular

Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
risk prediction equations in Chinese patients
with Type 2 diabetes
Jingyuan Liang 1, Qianqian Li 1, Zhangping Fu 1, Xiaofei Liu 1, Peng Shen2,
Yexiang Sun2, Jingyi Zhang3, Ping Lu3, Hongbo Lin2, Xun Tang 1,4*†, and
Pei Gao 1,4,5*†
1
Department of Epidemiology and Biostatistics, Peking University, 38 Xueyuan Road, Haidian District, Beijing 100191, China; 2Department of Chronic Diseases and Health Promotion,
Yinzhou District Centre for Disease Control and Prevention, Ningbo, China; 3Department of Medical Big Data, Wonders Information Co. Ltd, Shanghai, China; 4Key Laboratory of
Epidemiology of Major Diseases (Peking University), Ministry of Education, Beijing, China; and 5Peking University Clinical Research Institute, Peking University, Beijing, China

Received 23 February 2023; revised 2 June 2023; accepted 8 June 2023; online publish-ahead-of-print 14 June 2023

See the editorial comment for this article ‘The need for population-specific cardiovascular risk prediction models for guiding risk stratifi­
cation and treatment approaches in type 2 diabetes’, by Q.H. Lim et al., https://doi.org/10.1093/eurjpc/zwad230.

Aims For patients with diabetes, the European guidelines updated the cardiovascular disease (CVD) risk prediction recommenda­
tions using diabetes-specific models with age-specific cut-offs, whereas American guidelines still advise models derived from
the general population. We aimed to compare the performance of four cardiovascular risk models in diabetes populations.
............................................................................................................................................................................................
Methods Patients with diabetes from the CHERRY study, an electronic health records-based cohort study in China, were identified.
and results Five-year CVD risk was calculated using original and recalibrated diabetes-specific models [Action in Diabetes and Vascular
disease: PreterAx and diamicroN-MR Controlled Evaluation (ADVANCE) and the Hong Kong cardiovascular risk model
(HK)] and general population-based models [Pooled Cohort Equations (PCE) and Prediction for Atherosclerotic cardiovas­
cular disease Risk in China (China-PAR)]. During a median 5.8-year follow-up, 46 558 patients had 2605 CVD events. C-sta­
tistics were 0.711 [95% confidence interval: 0.693–0.729] for ADVANCE and 0.701 (0.683–0.719) for HK in men, and 0.742
(0.725–0.759) and 0.732 (0.718–0.747) in women. C-statistics were worse in two general population-based models.
Recalibrated ADVANCE underestimated risk by 1.2% and 16.8% in men and women, whereas PCE underestimated risk by
41.9% and 24.2% in men and women. With the age-specific cut-offs, the overlap of the high-risk patients selected by every
model pair ranged from only 22.6% to 51.2%. When utilizing the fixed cut-off at 5%, the recalibrated ADVANCE selected simi­
lar high-risk patients in men (7400) as compared to the age-specific cut-offs (7102), whereas age-specific cut-offs exhibited a
reduction in the selection of high-risk patients in women (2646 under age-specific cut-offs vs. 3647 under fixed cut-off).
............................................................................................................................................................................................
Conclusion Diabetes-specific CVD risk prediction models showed better discrimination for patients with diabetes. High-risk patients
selected by different models varied significantly. Age-specific cut-offs selected fewer patients at high CVD risk especially
in women.
............................................................................................................................................................................................
Lay summary This large electronic health records-based real-world study indicated that the diabetes-specific cardiovascular risk models had
better discriminative abilities than the models derived from the general population in Chinese patients with Type 2 diabetes.
• Current guidelines-recommended models, i.e. ADVANCE, PCE, and China-PAR, selected significantly different high-risk
groups with various observed cardiovascular risks, indicating the potential considerable misclassification of risk stratifica­
tion in clinical decision-making for preventive interventions.
• Compared with the fixed cut-off, the influence of the age-specific cut-offs for high risk of cardiovascular disease was dif­
ferent in men and women: age-specific cut-offs selected ∼27% fewer high-risk patients in women but similar in men.

* Corresponding authors. Tel: +86 10 82805642, Fax: +86 10 82805642, Emails: peigao@bjmu.edu.cn (P.G.); tangxun@bjmu.edu.cn (X.T.)
†
The last two authors contributed equally to the study.
© The Author(s) 2023. Published by Oxford University Press on behalf of the European Society of Cardiology.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits
non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com
1294 J. Liang et al.

Graphical Abstract

Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
Left panel: summary of the design and the four cardiovascular disease risk prediction models assessed in the study. Middle panel: C-index for the
PCE, China-PAR, ADVANCE, and HK models in the study population. Right panel: overlap of high-cardiovascular disease-risk patients in the study
population in the PCE-ADVANCE model pair in men and the China-PAR-ADVANCE model pair in women. ADA, American Diabetes Association;
ADVANCE, Action in Diabetes and Vascular disease: PreterAx and diamicroN-MR Controlled Evaluation; China-PAR, Prediction for
Atherosclerotic cardiovascular disease Risk in China; CVD, cardiovascular disease; ESC, European Society of Cardiology; HK, the Hong Kong car­
diovascular risk model; PCE, Pooled Cohort Equations.
............................................................................................................................................................................................
Keywords Cardiovascular disease • Chinese • Cohort analysis • Diabetes • Risk prediction model

Introduction
Type 2 diabetes is on the rise worldwide with a prevalence of currently
nearly 9% of the adult population.1 Cardiovascular disease (CVD) is one
of the most common complications and causes of death in patients with
Type 2 diabetes.2 Accurate risk assessment of CVD could guide appro­
priate prevention and management, especially with an increasing num­
ber of new drugs with proven cardiovascular benefit for this group.3
There are noteworthy differences across international guidelines for
cardiovascular risk prediction in patients with diabetes.4 The
European Society of Cardiology (ESC) recently updated its guidelines
to advocate using the diabetes-specific cardiovascular risk prediction
equation [i.e. the ADVANCE (Action in Diabetes and Vascular disease:
PreterAx and diamicroN-MR Controlled Evaluation) risk score] for
patients with diabetes.5 The ADVANCE risk score was explicitly
established for patients with diabetes, where diabetes and its
complications-related measures, e.g. glycated haemoglobin (HbA1c)
and renal function biomarkers, were included as predictors for CVD.
A similar diabetes-specific model, for example, the Hong Kong
cardiovascular risk (HK) model,6 has been developed for the Chinese
people. In contrast, the American College of Cardiology/American
Heart Association and American Diabetes Association (ADA) have re­
commended the use of the Pooled Cohort Equation (PCE) for an indi­
vidual’s 10-year CVD risk calculation, including patients with diabetes.7,8
The PCE was developed from the general population with a ‘history
of diabetes’ as a predictor, but not including any diabetes-specific
variables. Similarly in the Chinese guidelines, the Prediction for
Atherosclerotic cardiovascular disease Risk in China (China-PAR) mod­
el, a model also derived from the general population, has been recom­
mended for both the general population and patients with Type 2
diabetes.9
Risk stratification is closely related to treatment allocation, calling for
external validation studies before implementation.10 However, these
validation studies were limited among patients with diabetes. The
PCE and China-PAR models have been widely externally validated in
the general population11,12 but were rarely validated in patients with
diabetes. The diabetes-specific models (e.g. the ADVANCE and the
HK models) were not externally validated, especially in Chinese
Validation of CVD prediction equations in diabetes
populations.13 Moreover, few studies have directly compared these
two types of models. Most of these studies conducted in western po­
pulations had inconclusive results.14,15
Therefore, in this study, we aimed to assess the performance of the
four models (i.e. PCE, China-PAR, ADVANCE, and HK models) for pa­
tients with Type 2 diabetes in real-world clinical practice to answer: (i)
whether the diabetes-specific models perform better than the general
models and (ii) to what extent do these models differ when practically
applied in selecting a high-risk population.
Methods
Source of data
The source data were from the CHinese Electronic health Records
Research in Yinzhou (CHERRY) study, a longitudinal and population-based
cohort study in China. The detailed study protocol and some findings on
CVD have been previously published.12,16 In short, the CHERRY study
was established based on the integrated Health Information System in
Yinzhou, a developed area in Eastern China. Approximately, 98% of all
Yinzhou District residents were registered in the system before 1 January
2009. The system consists of various databases, e.g. Population Census
and Registered Health Insurance Database, Health Check Database,
Disease Management Database, Death and Disease Surveillance
Database, and Electronic Medical Records (EMRs). Detailed information re­
garding data sources was shown in Supplementary material online, Table S1.
Individual information about cardiovascular risk factors, clinical measure­
ments, and outcomes from different databases were linked via a unique
and encoded identifier. This study was approved by the Peking University
Institutional Review Board (IRB00001052-20086).
Participants
The 1 January 2010 was originally chosen as the CHERRY cohort incep­
tion to bypass the system’s integration and preliminary test period. In
this study, we included patients with Type 2 diabetes aged 30–74 years
from the CHERRY study, including (i) patients with diabetes who already
entered the study on 1 January 2010, and their index date was 1 January
2010; (ii) patients with diabetes who entered the study after 1 January
2010, and their baseline was the date of registration for health service
in Yinzhou; and (iii) patients in CHERRY study who were newly diagnosed
with diabetes between 1 January 2010 and 31 December 2018, where
their index date was set as the date of diabetes diagnosis. Diagnoses by
International Classification of Diseases, tenth revision (ICD-10) (E11,
E13, and E14) or Chinese text in the Disease Management Database,
Disease Surveillance Database and hospitals’ EMRs were used to identify
patients with diabetes.
Patients were excluded if they had no valid healthcare identifier, had con­
troversial records (30 patients who had the date of death recorded earlier
than the date of diabetes diagnosis), or had a history of CVD or kidney dys­
function [defined by estimated glomerular filtration rate (eGFR) < 30 mL/
min/1.73 m2 or history of renal dialysis/renal transplant]. Patients of the
study were followed up from baseline until the date of death, date of first
CVD event, withdrawal from study for various reasons or study end date
(31 December 2018)—whichever came first.
Cardiovascular disease risk prediction models
Cardiovascular disease risk prediction models in this study were chosen
based on the guidelines recommendations, study populations, and
prediction outcomes. Four models were used in our study: PCE17 and
China-PAR18 were derived from the general population, and
ADVANCE19 and HK6 were diabetes-specific models from patients with
Type 2 diabetes. The summary of models was shown in Supplementary
material online, Tables S2 and S3. Both original and recalibrated PCE,
ADVANCE, and HK models and the original China-PAR model were eval­
uated. We recalibrated PCE20 and ADVANCE models19 by replacing the
5-year baseline hazard and the mean score of algorithms from the
CHERRY patients with diabetes. This recalibration method has been widely
used in other validation studies.21,22 The HK model was also recalibrated
using the same method, considering the difference in outcome definition.6
1295
No recalibration was conducted for China-PAR as it was initially derived for
Chinese populations. In this study, except for the original ADVANCE mod­
el, we assessed the predictive performance of the models at 5 years. The
original ADVANCE was derived as a 4-year CVD risk prediction model
whereas the recalibrated ADVANCE model was set to 5-year for direct
comparison. For each participant, the observed CVD risk was calculated
using the Kaplan–Meier method with 95% confidence interval (CI) and
Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
the predicted CVD risk was calculated using the corresponding models.
Definition of outcomes
The definitions of CVD in the four original models were slightly different
(see Supplementary material online, Table S2). For directly comparing the
performance of different models, a unified CVD outcome was used, which
was defined as the first fatal or non-fatal myocardial infarction (MI), fatal
coronary heart disease (CHD), or fatal or non-fatal stroke using ICD-10
(I21, I22 for MI, I20-I25 for CHD, and I60, I61, I63, and I64 for stroke),
which was also used by the original PCE and China-PAR models. In the
sensitivity analysis, we also validated the HK model using HK-model spe­
cific outcomes: MI (I21, I22), fatal and non-fatal CHD (I20–I25), heart fail­
ure (I50), fatal and non-fatal stroke (I60–I69), transient ischaemic attack
(G45), and sudden death (R99). In this study, CVD outcomes were ob­
tained by linking the participants to relevant databases (Death and
Disease Surveillance Database, inpatient EMRs and Disease Management
Database). For patients with multiple CVD diagnosis dates, the earliest
one was chosen as the date of CVD.
Definition of predictors
To be consistent with the original models, only baseline measurement of
risk predictors was used to evaluate each model. Detailed methods of
measurement and definitions of risk factors were described in
Supplementary material online, Table S4. When evaluating the HK model,
eGFR was calculated using the revised MDRD equation23 as it was originally
used in the HK model. Alternatively, we used CKD-EPI equation24 to calcu­
late eGFR in the sensitivity analysis for the HK model as this equation was
the most commonly recommended one.25 In terms of the ‘regions’ used in
the China-PAR model, all participants were considered southerners ac­
cording to the North–South boundary of China. Consistent with previous
electronic health records (EHR)-based studies, for all predictors, the closest
recorded values before or at the time of index assessment were used. For
missing values of the predictors, the multiple imputation by chained equa­
tions method was used to avoid excluding observations with missing data.
Detailed method for imputation and the percentage of missingness of
each predictor were summarized in the Supplementary material online,
Method.
Statistical analyses
Discrimination, calibration, and reclassification measures were assessed for
the models’ predictive performance.26,27 Discriminative ability was exam­
ined using Harrell’s C-statistics. Calibration was evaluated in calibration
plots and Hosmer–Lemeshow χ2 (HL-χ2) tests. Expected–observed ratio
was also used to assess models’ performance.11 To compare the models
in the context of clinical relevance, we assessed the categorization of parti­
cipants across different models with net reclassification improvement (NRI)
and integrated discrimination improvement (IDI),27 using the recalibrated
ADVANCE model as the reference model. In addition, we compared the
concordance of high-risk patients selected by these models and calculated
the Kaplan–Meier adjusted 5-year CVD risks for high-risk individuals.
Cut-offs of ‘high risk’ of 5-year CVD were defined according to the 2021
ESC Guidelines on cardiovascular disease prevention in clinical practice:
3.75% for people <50 years, 5.0% for the 50–69 years, and 7.5% for those
≥70 years.5 Alternatively, 5.0% of 5-year CVD risk was used as the single
cut-off as a comparison. A two-sided P-value of <0.05 was defined as stat­
istically significant. Statistical analyses were performed using Stata version
16.0 (Stata Corporation, College Station, TX, USA). The report of this val­
idation study followed the Transparent Reporting of a multivariable predic­
tion model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines28
(see Supplementary material online, Appendix).
1296 J. Liang et al.

Results
Participants
There were 46 558 patients with diabetes (23 220 men and 23 338 wo­
men) in the final analysis (see Supplementary material online, Figure S1).
Baseline characteristics and cardiovascular outcomes were shown in
Table 1. The mean age of patients at baseline was 57.0 (SD: 9.6) years.
About two-thirds (28 938) of patients were newly diagnosed, resulting
in an average duration of diabetes of 1.7 years. Overall, there were 2605
incident CVD events with the median follow-up of 5.8 years. A total of
27 877 (59.9%) patients were followed up for at least 5 years. CVD
incidence was slightly higher using the HK-model outcomes (see
Supplementary material online, Table S5).

Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
Table 1 Baseline characteristics and cardiovascular outcomes of the study population, by sex

Overall (N = 46 558) Men (N = 23 220) Women (N = 23 338)

.........................................................................................................................................................
Age at baseline (years) 57.0 (9.6) 56.2 (9.9) 57.8 (9.2)
Newly diagnosed 28 938 (62.2) 15 104 (65.0) 13 834 (59.3)
Diabetes duration (years) 1.7 (3.5) 1.5 (3.3) 1.9 (3.7)
BMI (kg/m2) 24.1 (3.2) 24.1 (3.0) 24.1 (3.3)
Waist circumference (cm) 83.9 (12.5) 85.7 (12.0) 82.2 (13.2)
Urban 23 875 (51.3) 11 835 (51.0) 12 040 (51.6)
SBP (mmHg) 130.4 (15.5) 130.2 (16.9) 130.6 (13.2)
DBP (mmHg) 80.2 (14.8) 80.4 (12.2) 79.9 (11.5)
PP (mmHg) 50.3 (10.7) 49.8 (12.5) 50.8 (12.0)
Smoking status
Yes (current) 7691 (16.5) 7522 (32.4) 169 (0.7)
No (former or never) 38 867 (83.5) 15 698 (67.6) 23 169 (99.3)
HbA1c
mmol/mol 59.7 (54.5) 62.4 (42.8) 57.0 (50.1)
% 7.6 (5.0) 7.9 (3.9) 7.4 (4.6)
Lipid profiles
TC (mmol/L) 5.0 (1.4) 4.8 (1.4) 5.1 (1.3)
HDL (mmol/L) 1.3 (0.5) 1.2 (0.4) 1.3 (0.5)
TC/HDL 4.1 (1.7) 4.2 (1.7) 4.0 (1.7)
Non-HDL (mmol/L) 3.7 (1.4) 3.6 (1.4) 3.8 (1.3)
Renal function
eGFR (mL/min/1.73 m2) 96.1 (18.9) 97.4 (22.2) 94.9 (19.0)
ACR (mg/gCr)a 14.6 (6.5–39.4) 13.6 (5.9–36.5) 15.6 (6.9–42.1)
Medical history
History of atrial fibrillation 22 (0.05) 14 (0.06) 8 (0.03)
History of retinopathy 1052 (2.3) 496 (2.1) 556 (2.4)
Family history of atherosclerotic CVD 1414 (3.0) 686 (3.0) 728 (3.1)
Medications
Insulin 2314 (5.0) 1318 (5.7) 996 (4.3)
Oral hypoglycaemic agents 25 967 (55.8) 13 195 (56.8) 12 772 (54.7)
Blood pressure-lowering medications 24 798 (53.3) 11 971 (51.6) 12 827 (55.0)
Lipid-lowering medications 8806 (18.9) 4213 (18.1) 4593 (19.7)
CVD outcomes
Median follow-up time, years 5.8 5.6 6.2
Follow-up time exceeds 5 years 27 877 (59.9) 13 292 (57.2) 14 585 (62.5)
Incident CVD events 2605 (5.6) 1333 (5.7) 1272 (5.5)
CHD events 201 (0.4) 120 (0.5) 81 (0.3)
Stroke events 2404 (5.2) 1213 (5.2) 1191 (5.1)
Incidence of CVD, per 100 000 person-year 988.5 1049.0 932.2
5-year Kaplan–Meier CVD rate 4.1 4.5 3.8

Values are mean (SD) or n (%) unless otherwise noted. To convert cholesterol to mg/dL, multiply values by 38.67.
a
Presented as median (IQR).
ACR, albumin-to-creatinine ratio; BMI, body mass index; CHD, coronary heart disease; CVD, cardiovascular disease; DBP, diastolic blood pressure; eGFR, estimated glomerular filtration
rate; HbA1c, glycated haemoglobin; HDL, high-density lipoprotein cholesterol; PP, pulse pressure; SBP, systolic blood pressure; TC, total cholesterol; TC/HDL, total/HDL cholesterol ratio.
Validation of CVD prediction equations in diabetes 1297

Model performance
The performance of four models in predicting CVD risk in patients
with diabetes was shown in Table 2. Overall, the discriminative abilities
of diabetes-specific models (ADVANCE and HK) were better
than those derived from the general population (PCE and
China-PAR). In terms of the C-statistics, the ADVANCE model de­
the 2021 ESC Guidelines. Results of NRI and IDI were shown in
Table 3. When compared with ADVANCE, we found that PCE,
China-PAR, and HK models had significantly worse predictive ability
in men with NRI and IDI. In women, PCE and HK indicated an overall
worse performance in NRI and IDI compared with ADVANCE. IDI of
−0.6% (95% CI, −0.8%, −0.4%) was obtained in comparing
China-PAR and ADVANCE. Similar results were observed when

monstrated the best results, followed by the HK model. The PCE
and China-PAR models had sex-specific C-statistics of <0.7 in men,
and the results were better in women. Similar results for HK model
were obtained when evaluated using the HK-model outcomes or
using CKD-EPI calculated eGFR (see Supplementary material online,
Tables S6 and S7).
The original PCE model overestimated the 5-year CVD risk by 16.9%
in men before recalibration. After recalibration, it underestimated risk
by 41.9%. In women, it underestimated risk by 37.5% and 24.2% before
and after recalibration (Table 2 and Figure 1). The original China-PAR
calibrated well in men (underestimated risk only by 5.0%) but overes­
timated risk by 33.0% in women. Both original and recalibrated
ADVANCE models underestimated CVD risk in men (by 15.2% and
1.2% before and after recalibration) and women (by 28.5% and
16.8% before and after recalibration) with poor calibration (P <
0.001). The original HK model overestimated 5-year CVD risk by
58.1% in men and 21.5% in women, and the recalibrated HK model still
underestimated the 5-year CVD risk by 13.7% and 15.6% in men and
women. The calibration results for HK model were similar in the sen­
sitivity analysis (see Supplementary material online, Figures S2 and S3).
To further assess the clinically categorical improvements of four
models, we compared the recalibrated ADVANCE model vs. the re­
calibrated PCE, original China-PAR, and recalibrated HK models using
the thresholds of predicted 5-year CVD risk recommended by
Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
models were compared using 5% as the high-risk threshold (see
Supplementary material online, Table S8).
High-risk patients selected by risk
prediction models
Using the recalibrated ADVANCE as the reference, we compared the
pairwise overlap of patients classified as high risk by recalibrated PCE,
original China-PAR, and recalibrated HK models. We found significant
heterogeneity in each pairwise comparison (Figure 2). The largest dis­
parity was seen in the China-PAR-ADVANCE pairwise in women:
only a small percentage (2.5%) of patients were solely identified as
high risk by ADVANCE, but much bigger (74.9%) by China-PAR, which
appeared to be in line with the models’ calibration performance. In con­
trast, although China-PAR and ADVANCE selected a similar number of
high-risk patients in men, these two groups varied greatly: out of 10 053
patients identified by either China-PAR or ADVANCE, only 48.4%
were selected by both models. The high-risk patients exclusively deter­
mined by the PCE, China-PAR, or HK models had significantly lower
observed 5-year CVD risk compared to those by ADVANCE in both
sexes (Figure 2). Similar results were obtained using 5% as the high-risk
cut-off for 5-year CVD risks (see Supplementary material online,
Figure S4).

Table 2 Performance of the PCE, China-PAR, ADVANCE, and HK cardiovascular risk models, by sex

Model Discrimination C-statistics (95% CI) Expected–observed ratio Calibration χ2 (P-value)

.........................................................................................................................................................
Original version
Men
PCE 0.689 (0.673–0.704) 1.169 375.7 (<0.001)
China-PAR 0.671 (0.656–0.686) 0.950 26.5 (<0.001)
ADVANCEa 0.711 (0.693–0.729) 0.848 59.5 (<0.001)
HK 0.701 (0.683–0.719) 1.581 262.4 (<0.001)
Women
PCE 0.722 (0.708–0.737) 0.625 239.8 (<0.001)
China-PAR 0.702 (0.687–0.716) 1.330 95.2 (<0.001)
ADVANCEa 0.742 (0.725–0.759) 0.715 161.6 (<0.001)
HK 0.732 (0.718–0.747) 1.215 54.9 (<0.001)
Recalibrated versionb
Men
PCE 0.689 (0.673–0.704) 0.581 1074.5 (<0.001)
ADVANCE 0.711 (0.693–0.729) 0.988 45.2 (<0.001)
HK 0.701 (0.683–0.719) 0.863 127.4 (<0.001)
Women
PCE 0.722 (0.708–0.737) 0.758 93.5 (<0.001)
ADVANCE 0.742 (0.725–0.759) 0.832 134.5 (<0.001)
HK 0.732 (0.718–0.747) 0.844 78.6 (<0.001)
a
Original ADVANCE model was assessed as a 4-year CVD risk prediction model.
b
In the recalibrated version, the 5-year baseline CVD hazard and the mean score of algorithms from each of the PCE, ADVANCE, and HK models were replaced with those generated
from the study population. ADVANCE, Action in Diabetes and Vascular disease: PreterAx and diamicroN-MR Controlled Evaluation; China-PAR, Prediction for Atherosclerotic
cardiovascular disease Risk in China; HK, the Hong Kong cardiovascular risk model; PCE, Pooled Cohort Equations.
1298 J. Liang et al.

Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024

Figure 1 Calibration plots using different risk prediction models. The original ADVANCE model was assessed as a 4-year cardiovascular risk pre­
diction model. Recalibrated ADVANCE, original and recalibrated PCE and HK, and original China-PAR models estimated a risk of over 5 years. (A) PCE
in men, (B) PCE in women, (C) China-PAR in men, (D) China-PAR in women, (E) ADVANCE in men, (F) ADVANCE in women, (G) HK in men, (H ) HK
in women. ADVANCE, Action in Diabetes and Vascular disease: PreterAx and diamicroN-MR Controlled Evaluation; China-PAR, Prediction for
Atherosclerotic cardiovascular disease Risk in China; HK, the Hong Kong cardiovascular risk model; PCE, Pooled Cohort Equations.
Validation of CVD prediction equations in diabetes 1299

Table 3 Reclassification improvement calculations by recalibrated ADVANCE vs. recalibrated PCE, original China-PAR
and recalibrated HK models

PCE vs. ADVANCE China-PAR vs. ADVANCE HK vs. ADVANCE

.........................................................................................................................................................
Men

Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
NRI overall (95% CI) −11.8% (−15.5 to −8.0) −13.3% (−16.9 to −9.7) −8.1% (−11.5 to −4.7)
NRI cases (95% CI) −25.7% (−29.4 to −22.1) −10.0% (−13.5 to −6.5) −19.3% (−22.6 to −15.9)
NRI non-cases (95% CI) 14.0% (13.3 to 14.7) −3.3% (−4.1 to −2.5) 11.1% (10.5 to 11.8)
IDI (95% CI) −1.0% (−1.2 to −0.7) −1.6% (−1.9 to −1.4) −1.3% (−1.5 to −1.1)
Women
NRI overall (95% CI) −5.3% (−8.9 to −1.7) 3.7% (−1.1 to 8.6) −3.4% (−6.4 to −0.4)
NRI cases (95% CI) −3.7% (−7.2 to −0.1) 35.7% (31.0 to 40.5) −3.1% (−6.1 to −0.1)
NRI non-cases (95% CI) −1.6% (−2.1 to −1.1) −32.0% (−32.9 to −31.0) −0.3% (−0.7 to 0.1)
IDI (95% CI) −0.4% (−0.6 to −0.2) −0.6% (−0.8 to −0.4) −0.4% (−0.5 to −0.3)

The ‘high risk’ was defined according to the recommendation by the 2021 ESC Guidelines: 3.75% for people aged <50 years, 5.0% for people aged 50–69 years, and 7.5% for people aged
≥70 years.
ESC, European Society of Cardiology; IDI, integrated discrimination improvement; NRI, net reclassification improvement.

Figure 2 Venn diagram for high-risk groups using different risk prediction models. Overlap in those classified as high risk when applying for recali­
brated ADVANCE vs. recalibrated PCE, original China-PAR, and recalibrated HK models in participants under the age-specific high-risk cut-offs recom­
mended by the 2021 European Society of Cardiology guidelines (3.75% for people aged <50 years, 5.0% for people aged 50–69 years, and 7.5% for
people aged ≥70 years) in men (A) and women (B). n represents the number of individuals classified as high risk with either model. The observed risks
presented here were Kaplan–Meier adjusted 5-year CVD risks.

Lastly, the overlap of high-risk patients in ADVANCE under different

cut-offs (5.0% and age-specific cut-offs) was shown in Figure 3. With
these two cut-offs, the high-risk population selected by ADVANCE
overlapped greatly in men, whereas in women, 1011 (27.6%) patients
were additionally identified under the 5.0% cut-off.
Discussion
To the best of our knowledge, this is the first study to externally valid­
ate four CVD risk prediction models (PCE, China-PAR, ADVANCE,
and HK models) in Chinese patients with Type 2 diabetes. We found
1300
Figure 3 Venn diagram for high-risk group using different high-risk cut-offs with the ADVANCE model. Overlap in those classified as high risk by
recalibrated ADVANCE in men (A) and women (B) when applying different high-risk cut-offs. Age-specific cut-offs: 3.75% for people aged <50 years,
5.0% for people aged 50–69 years, and 7.5% for people aged ≥70 years.
that the discriminative ability was better in two diabetes-specific mod­
els (ADVANCE and HK) than in the general models (PCE and
China-PAR). The original models did not accurately predict absolute
CVD risk, and recalibration improved marginally. The ADVANCE mod­
el appeared to be the best in terms of the model’s reclassification. With
the age-specific cut-offs, ADVANCE model selected fewer high-risk pa­
tients with diabetes than the fixed cut-off, especially in women.
General population-based vs.
diabetes-specific cardiovascular disease
risk prediction models
First of all, the inclusion of ‘diabetes’ in the risk prediction model as a
single categorical variable seems insufficient on cardiovascular assess­
ment, though many current guidelines still recommend. We found
that diabetes-specific models (ADVANCE and HK) had an overall bet­
ter discriminative ability, and ADVANCE had substantially better re­
classification of patients at high risk compared with PCE in both
sexes and with China-PAR in men. This could be largely explained by
the difference in variables: diabetes-specific variables were included in
the ADVANCE and HK models, for example, the duration of diabetes,
HbA1c, and diabetes medication use,29 which were not included in ei­
ther PCE or China-PAR. Specifically for PCE, recalibration did not im­
prove its performance, indicating that models from general population
may not be appropriate in diabetes patients, even after recalibration.
J. Liang et al.
Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
Several previous literatures have also suggested a possible discrimin­
atory advantage of diabetes-specific over general population-based
models in patients with diabetes, though they were not directly com­
pared in a common patient population. Echouffo-Tcheugui and
Kengne30 reported that diabetes-specific models were superior to
the general population models in 14 out of 22 comparisons, and a
meta-analysis of C-statistics showed an overall pooled C-statistic of
0.67 and 0.64 for the models developed in diabetes and in general popu­
lation, respectively.31 In contrast, Dziopa et al.15 found conflict results in
the performance of the two types of models. However, the included
models were not appropriately recalibrated and heterogeneous CVD
outcome definitions were also not carefully considered. More import­
antly, the ADVANCE model, the one recommended by the 2021 up­
dated ESC guidelines, was not assessed in their study. Therefore, we
believe that our conclusion to favour diabetes-specific models is
reasonable.
We found great disparities in recruitment periods, distributions of
risk factors, and incidence of CVD between the study population and
the participants in the derivation cohorts of the evaluated models
(see Supplementary material online, Tables S2 and S9). In the general
population, recalibration could often improve the agreement between
predicted and observed CVD risks when a foreign model is used locally.
However, recalibration for diabetes patients in our study gained little
on models’ performance. For example, recalibration even worsened
the performance of the PCE model in men. This may be due to the po­
tential differences in the strength of the association between predictors
Validation of CVD prediction equations in diabetes
and CVD outcomes in different populations indicated by the fact that
the original PCE overestimated risks in low-risk deciles and underesti­
mated risks in high-risk deciles. Consistently, Read et al.14 also reported
that simple recalibration could not improve the predictive abilities of
QRISK2, CHS, and DCS models in Scottish Type 2 diabetes populations.
However, a similar recalibration method was used in the PREDICT study,
which assessed the DCS model in 46 652 patients with Type 2 diabetes
from New Zealand and found that after recalibration, the model was well
calibrated, albeit slightly underestimated risk in women.32 We noted that
DCS model was also established from New Zealand patients with dia­
betes,33 whereas the PCE and ADVANCE models were derived from
heterogeneous populations. The results indicated that more advanced
recalibration methods, for example adjusting regression coefficients,
should be considered to improve model’s performance in people with
diabetes. Moreover, to date, most guidelines have used original predic­
tion models, or models simply recalibrated using national cardiovascular
mortality data, when recommending CVD risk assessment tools.34,35
Our study suggested that risk prediction models, before recalibration
and validation, should be cautiously recommended for patients in exter­
nal clinical settings, especially those differing in ethnic background.
From the perspective of clinical relevance, we found that generally,
PCE, China-PAR, and HK models had meaningfully worse reclassifica­
tion than ADVANCE, and there existed significant discordance in iden­
tifying high-risk patients in each model pair. Although the two
diabetes-specific equations had similar discriminative ability, only half
of the total selected high-risk patients overlapped in both sexes, and
those who were chosen exclusively as high-risk by ADVANCE had sig­
nificantly higher observed CVD risk than patients identified by HK mod­
el. Interestingly, although China-PAR and ADVANCE had similar
calibration results and they selected similar number of patients as
high risk in men, the overlap of these high-risk patients was only
48.4%, and the patients solely identified by ADVANCE had significantly
higher observed 5-year CVD risk than those solely selected by
China-PAR (8.9% vs. 3.1%). This again suggested that two models
with similar discriminative abilities or calibrations could have varied per­
formance in identifying high-risk subgroups and therefore influence the
prescription of cardiovascular protective medications.
Age-specific high-risk cut-offs
For the first time, the 2021 ESC Guidelines on CVD prevention used
the age-specific CVD risk to avoid undertreatment in the young and
overtreatment in the old. In contrast, most guidelines have been using
age-independent cut-offs, for example, the 15% as the high-risk thresh­
old for 10-year CVD risk in the 2023 ADA guideline8 and the 5% as the
point for preventive medication initiation for 5-year CVD risk in the
New Zealand Guideline.36 In this study, we explored how differently
the ADVANCE model selected high-risk patients under various cut-offs
(5.0% vs. age-specific cut-offs). We found that high-risk patients in men
overlapped mostly, but in women it varied a lot: a total of 1011 patients
were identified by the 5.0% cut-off but not by the age-specific cut-offs.
Similarly, Mortensen et al.37 reported that the age-specific cut-offs in
the 2021 ESC guidelines dramatically reduced eligibility, especially
among women, for primary prevention with statins in the general popu­
lation. Further studies are still needed, especially in diverse ethnic
groups, to evaluate the cost-effectiveness of preventive treatment of
CVD under different cut-offs in patients with diabetes.
Extensive EHR data have been more and more widely used by re­
searchers and regulatory authorities.38,39 This study was intended to
evaluate the performance of risk prediction models using these EHR
data in real-world scenarios. It is known that EHR data are generally
less systematic and involve missing values of predictors. However, indi­
viduals with partial information in real-world scenario also require risk
stratification. Moreover, missing patterns of required variables some­
times also demonstrate the disparity of commonly measured
1301
biomarkers between developing and developed countries. For example
in the current study, we had the missing rate of albumin-to-creatinine
ratio (ACR) exceeding 70%, which is markedly higher compared to
the ∼40% missing rate observed in the DIAL model developed in
Sweden.29 This is mainly due to different routine recommendations
and utilization of ACR measurement in clinical practice in China.
Although missing values may have a relatively minor impact in validation
Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
studies, it warrants additional attention in model derivation studies, es­
pecially in developing countries.
Strengths and limitations
The main strength of our study was its population-based nature, which
enabled a low risk of selection bias and allowed for reliable estimation
with large sample size. Several limitations of this study should be ac­
knowledged. As mentioned above, missing values are common in
EHR-based studies where variables are taken from administrative data­
bases, although this also reflects the clinical practice under a real-world
scenario where individuals with partial information also require risk as­
sessment. In our study, we addressed this issue using multiple imputa­
tions, which are widely used in other EHR-based studies,40 to reduce
the selection bias and increase the power and precision of our results.
Secondly, the median follow-up time of this study was only 5.8 years,
thus limiting the ability to conduct subgroup analyses to address the
influence of recruitment time on CVD risk predictions. In addition, al­
though the CHERRY study had a relatively large number of participants,
it was a regional cohort located in a developed area of China and, as
such, would not be nationally representative. Fourthly, as outcome
data are unavailable beyond 2018, we are not able to assess the impact
of new drugs on CVD risk predictions. Moreover, all patients were re­
cruited before the global pandemic of coronavirus disease 2019
(COVID-19), and implementing these models to predict CVD risk in
people with Type 2 diabetes after infection of COVID-19 still needs fur­
ther investigation.
Conclusions
In summary, this study compared the PCE, China-PAR, ADVANCE,
and HK models in Chinese patients with diabetes. Although many
guidelines still recommended general population-based CVD risk pre­
diction models, we found that the diabetes-specific models performed
better. The original models did not accurately predict absolute CVD
risk. Recalibration improved marginally. The ADVANCE model ap­
peared to be the best in terms of the model’s reclassification. With
the age-specific cut-offs, the ADVANCE model selected fewer high-risk
patients with diabetes, especially in women. We support the use of the
diabetes-specific CVD risk prediction model to guide personalized
treatments for patients with diabetes.
Supplementary material
Supplementary material is available at European Journal of Preventive
Cardiology.
Acknowledgements
The authors thank Yinzhou District Health Bureau for providing access to
the administrative databases used in the study and all the CHERRY study
investigators for their contributions.
Authors’ contributions
J.L., X.T., and P.G. had full access to all the data in the study and take respon­
sibility for the integrity of the data as a whole. J.L., X.T., and P.G. conceived
1302
the idea of the study. J.Z. and P.L. contributed to data quality control. P.S.,
Y.S., and H.L. contributed to administrative, technical, or material support.
J.L. and X.L. carried out data preparation. J.L. conducted statistical analyses
and wrote the first draft of the article. J.L., Q.L., Z.F., X.T., and P.G. contrib­
uted to the interpretation of the findings and the manuscript’s critical revi­
sion. All authors agreed that the accuracy and integrity of the work has been
appropriately investigated and resolved and all approved the submitted ver­
sion of the manuscript. X.T. and P.G. attests that all listed authors meet
authorship criteria and that no others meeting the criteria have been omit­
ted. X.T. and P.G. had the final responsibility for the decision to submit for
publication.
Funding
Funding for this project was provided by the National Key Research and
Development Program of China (2020YFC2003503) and the National
Natural Science Foundation of China (NSFC) (81961128006 and
81973132). The study funders were not involved in the design of the study;
the collection, analysis, and interpretation of data; writing the report; and
did not impose any restrictions regarding the publication of the report.
Conflict of interest: P.G. reported receiving research funds from Bayer
and Merck. These funding sources had no relation of this study. All other
authors have reported that they have no relationships relevant to the con­
tents of this article to disclose.
Data availability
The data underlying this article were provided by Yinzhou District Centre
for Disease Control and Prevention. Data will be shared on request to the
corresponding authors with permission of the Yinzhou District Health
Bureau.
References
1. Standl E, Khunti K, Hansen TB, Schnell O. The global epidemics of diabetes in the 21st
century: current situation and perspectives. Eur J Prev Cardiol 2019;26:7–14.
2. Tsao CW, Aday AW, Almarzooq ZI, Alonso A, Beaton AZ, Bittencourt MS, et al. Heart
Disease and Stroke Statistics-2022 update: a report from the American Heart
Association. Circulation 2022;145:e153–e639.
3. Palmer SC, Tendal B, Mustafa RA, Vandvik PO, Li S, Hao Q, et al. Sodium-glucose co­
transporter protein-2 (SGLT-2) inhibitors and glucagon-like peptide-1 (GLP-1) recep­
tor agonists for type 2 diabetes: systematic review and network meta-analysis of
randomised controlled trials. BMJ 2021;372:m4573.
4. Kelsey MD, Nelson AJ, Green JB, Granger CB, Peterson ED, McGuire DK, et al.
Guidelines for cardiovascular risk reduction in patients with type 2 diabetes: JACC
guideline comparison. J Am Coll Cardiol 2022;79:1849–1857.
5. Visseren FLJ, Mach F, Smulders YM, Carballo D, Koskinas KC, Bäck M, et al. 2021 ESC guide­
lines on cardiovascular disease prevention in clinical practice. Eur Heart J 2021;42:3227–3337.
6. Wan EYF, Fong DYT, Fung CSC, Yu EYT, Chin WY, Chan AKC, et al. Development of a
cardiovascular diseases risk prediction model and tools for Chinese patients with type 2
diabetes mellitus: a population-based retrospective cohort study. Diabetes Obes Metab
2018;20:309–318.
7. Arnett DK, Blumenthal RS, Albert MA, Buroker AB, Goldberger ZD, Hahn EJ, et al. 2019
ACC/AHA guideline on the primary prevention of cardiovascular disease: a report of
the American College of Cardiology/American Heart Association Task Force on
Clinical Practice Guidelines. Circulation 2019;140:e596–e646.
8. ElSayed NA, Aleppo G, Aroda VR, Bannuru RR, Brown FM, Bruemmer D, et al. 10.
Cardiovascular disease and risk management: standards of care in diabetes-2023.
Diabetes Care 2023;46:S158–S190.
9. The Joint Task Force for Guideline on the Assessment and Management of
Cardiovascular Risk in China. Guideline on the assessment and management of cardio­
vascular risk in China. Chin J Prev Med 2019;53:13–35.
10. Adibi A, Sadatsafavi M, Ioannidis JPA. Validation and utility testing of clinical prediction
models: time to change the approach. JAMA 2020;324:235–236.
11. Cook NR, Ridker PM. Calibration of the pooled cohort equations for atherosclerotic
cardiovascular disease: an update. Ann Intern Med 2016;165:786–794.
12. Liu X, Shen P, Zhang D, Sun Y, Chen Y, Liang J, et al. Evaluation of atherosclerotic
cardiovascular risk prediction models in China. JACC: Asia 2022;2:33–43.
J. Liang et al.
13. Kostopoulos G, Antza C, Doundoulakis I, Toulis KA. Risk models and scores of
cardiovascular disease in patients with diabetes mellitus. Curr Pharm Des 2021;27:
1245–1253.
14. Read SH, van Diepen M, Colhoun HM, Halbesma N, Lindsay RS, McKnight JA, et al.
Performance of cardiovascular disease risk scores in people diagnosed with type 2 dia­
betes: external validation using data from the National Scottish Diabetes Register.
Diabetes Care 2018;41:2010–2018.
Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
15. Dziopa K, Asselbergs FW, Gratton J, Chaturvedi N, Schmidt AF. Cardiovascular risk
prediction in type 2 diabetes: a comparison of 22 risk scores in primary care settings.
Diabetologia 2022;65:644–656.
16. Lin H, Tang X, Shen P, Zhang D, Wu J, Zhang J, et al. Using big data to improve
cardiovascular care and outcomes in China: a protocol for the CHinese Elec­
tronic health Records Research in Yinzhou (CHERRY) Study. BMJ Open 2018;8:
e019698.
17. Goff DC Jr, Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al. 2013
ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American
College of Cardiology/American Heart Association Task Force on Practice Guidelines.
Circulation 2014;129:S49–S73.
18. Yang X, Li J, Hu D, Chen J, Li Y, Huang J, et al. Predicting the 10-year risks of atheroscler­
otic cardiovascular disease in Chinese population: the China-PAR project (prediction
for ASCVD risk in China). Circulation 2016;134:1430–1440.
19. Kengne AP, Patel A, Marre M, Travert F, Lievre M, Zoungas S, et al. Contemporary mod­
el for cardiovascular risk prediction in people with type 2 diabetes. Eur J Cardiovasc Prev
Rehabil 2011;18:393–398.
20. Muntner P, Colantonio LD, Cushman M, Goff DC Jr, Howard G, Howard VJ, et al.
Validation of the atherosclerotic cardiovascular disease Pooled Cohort risk equations.
JAMA 2014;311:1406–1415.
21. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores.
Stat Methods Med Res 2016;25:1692–1706.
22. Sud M, Sivaswamy A, Chu A, Austin PC, Anderson TJ, Naimark DMJ, et al.
Population-based recalibration of the Framingham Risk Score and Pooled Cohort
Equations. J Am Coll Cardiol 2022;80:1330–1342.
23. Ma YC, Zuo L, Chen JH, Luo Q, Yu XQ, Li Y, et al. Modified glomerular filtration rate
estimating equation for Chinese patients with chronic kidney disease. J Am Soc Nephrol
2006;17:2937–2944.
24. Levey AS, Stevens LA, Schmid CH, Zhang YL, Castro AF III, Feldman HI, et al.
A new equation to estimate glomerular filtration rate. Ann Intern Med 2009;
150:604–612.
25. Kidney Disease: Improving Global Outcomes (KDIGO) Glomerular Diseases Work
Group. KDIGO 2021 Clinical practice guideline for the management of glomerular dis­
eases. Kidney Int 2021;100:S1–S276.
26. McLernon DJ, Giardiello D, Van Calster B, Wynants L, van Geloven N, van Smeden M,
et al. Assessing performance and clinical usefulness in prediction models with survival
outcomes: practical guidance for cox proportional hazards models. Ann Intern Med
2023;176:105–114.
27. Pencina MJ, D’Agostino RB Sr, Demler OV. Novel metrics for evaluating improvement
in discrimination: net reclassification and integrated discrimination improvement for
normal variables and nested models. Stat Med 2012;31:101–113.
28. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable
prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD state­
ment. BMJ 2015;350:g7594.
29. Berkelmans GFN, Gudbjörnsdottir S, Visseren FLJ, Wild SH, Franzen S, Chalmers J, et al.
Prediction of individual life-years gained without cardiovascular events from lipid, blood
pressure, glucose, and aspirin treatment based on data of more than 500 000 patients
with type 2 diabetes mellitus. Eur Heart J 2019;40:2899–2906.
30. Echouffo-Tcheugui JB, Kengne AP. Comparative performance of diabetes-specific and
general population-based cardiovascular risk assessment models in people with diabetes
mellitus. Diabetes Metab 2013;39:389–936.
31. Chowdhury MZI, Yeasmin F, Rabi DM, Ronksley PE, Turin TC. Prognostic tools for car­
diovascular disease in patients with type 2 diabetes: a systematic review and
meta-analysis of C-statistics. J Diabetes Complications 2019;33:98–111.
32. Pylypchuk R, Wells S, Kerr A, Poppe K, Harwood M, Mehta S, et al. Cardiovascular risk
prediction in type 2 diabetes before and after widespread screening: a derivation and
validation study. Lancet 2021;397:2264–2274.
33. Elley CR, Robinson E, Kenealy T, Bramley D, Drury PL. Derivation and validation of a
new cardiovascular risk score for people with type 2 diabetes: the New Zealand dia­
betes cohort study. Diabetes Care 2010;33:1347–1352.
34. Fegers-Wustrow I, Gianos E, Halle M, Yang E. Comparison of American and European
Guidelines for Primary Prevention of Cardiovascular Disease: JACC Guideline
Comparison. J Am Coll Cardiol 2022;79:1304–1313.
Validation of CVD prediction equations in diabetes
35. Khanji MY, Bicalho VV, van Waardhuizen CN, Ferket BS, Petersen SE, Hunink MG.
Cardiovascular risk assessment: a systematic review of guidelines. Ann Intern Med
2016;165:713–722.
36. Ministry of Health. Cardiovascular Disease Risk Assessment and Management for Primary
Care. Wellington: Ministry of Health; 2018.
37. Mortensen MB, Tybjærg-Hansen A, Nordestgaard BG. Statin eligibility for primary
prevention of cardiovascular disease according to 2021 European Prevention
Guidelines compared with other international guidelines. JAMA Cardiol 2022;7:
836–843.
1303
38. Beesley LJ, Salvatore M, Fritsche LG, Pandit A, Rao A, Brummett C, et al. The emerging
landscape of health research based on biobanks linked to electronic health records: existing
resources, statistical challenges, and potential opportunities. Stat Med 2020;39:773–800.
39. Kotecha D, Asselbergs FW, Achenbach S, Anker SD, Atar D, Baigent C, et al.
CODE-EHR best practice framework for the use of structured electronic healthcare
records in clinical research. BMJ 2022;378:e069048.
40. Hippisley-Cox J, Coupland C, Brindle P. Development and validation of QRISK3 risk
Downloaded from https://academic.oup.com/eurjpc/article/30/12/1293/7197865 by University of Science and Technology of China user on 16 January 2024
prediction algorithms to estimate future risk of cardiovascular disease: prospective co­
hort study. BMJ 2017;357:j2099.