Professional Documents
Culture Documents
Research Article
Comparison of Semiparametric, Parametric, and Nonparametric
ROC Analysis for Continuous Diagnostic Tests Using a Simulation
Study and Acute Coronary Syndrome Data
Ertugrul Colak,1 Fezan Mutlu,1 Cengiz Bal,1 Setenay Oner,1 Kazim Ozdamar,1
Bulent Gok,2 and Yuksel Cavusoglu3
1 Department
of Biostatistics and Medical Informatics, School of Medicine, Eskisehir Osmangazi University, 26480 Eskisehir, Turkey
of Cardiology, Private Bursa Anadolu Hospital, Izmir Yolu 105, 16120 Nilufer/Bursa, Turkey
3 Department of Cardiology, School of Medicine, Eskisehir Osmangazi University, 26480 Eskisehir, Turkey
2 Department
1. Introduction
Receiver operating characteristic (ROC) curve analysis is
used in many scientific fields to determine the accuracy of
a diagnostic test, for example, in signal detection theory and
medicine [17]. An ROC curve is a plot of the false positive
rates against the true positive rates for various cuto values
of the diagnostic test result. The most commonly used value
to summarize the accuracy is the area under the ROC curve
(AUC). The AUC can take values between 0 and 1, and
greater AUC values denote better accuracy [6].
The result of a diagnostic test may be binary, ordinal,
or continuous. Most medical diagnostic tests, such as
biomarkers for myocardial injury or cancer, have continuous
test results [5, 8]. The most common ROC analyses are
nonparametric. The nonparametric ROC methods do not
require any assumptions about the diagnostic test result
(1)
FP(c) = P(Y c | D0 ).
The ROC curve is denoted by
ROC(t) = 1 F1 F01 (1 t) ,
(2)
Y1 N 1 , 12 ,
Y0 N 0 , 02 .
(3)
ROC(t) = a + b1 (t) ,
(4)
a=
,
b = 0,
(5)
1
1
see [1113].
The AUC equals the probability that a randomly selected
diseased subject has diagnostic higher than a randomly
selected nondiseased subject:
a
,
AUC =
(6)
1 + b2
where is the normal probability density function [5, 12,
13]. Thus, the estimates of a, b, and AUC (denoted by a, b,
resp.) are computed using 1 , 0 , 1 , and 0 . The
and AUC,
respective variances of a and b are
V (a) =
(n + n )b
2
0
,
V b = 1
(7)
2n0 n1
s1 (c)
,
n1
s (c)
FP(c) = 0 ,
n0
(8)
n n
1 1 0
Yi1 , Y j0 ,
n1 n0 i=1 j =1
(9)
where
if Yi1 > Y j0
if Yi1 = Y j0
Yi1 , Y j0 = 2
if Yi1 < Y j0
V AUC
1 AUC
AUC
+ (n1 1) Q1
n1 n0
2
(n0 1) Q2 AUC
2
AUC
n1 n0
=y 2
=y 2
n1
1 = y >y 2
>y
=y
Q1 =
n0 n 1
+ n1 n 1 +
2
3
n0 n1 Y
n0
1 = y <y 2
<y
=y
n1 n 0
+ n0 n 0 +
Q2 = 2
3
n0 n1 Y
(12)
=y
Ui j = I Yi1 Y j0 ,
i = 1, . . . , n1 , j = 1, . . . , n0
(13)
t j = FP Y j0 t j T =
1
n
,..., 0 .
n0
n0
(14)
(15)
g E Ui j
k hk t j .
(16)
k=1
k hk (t),
E Ui j = 1 + 2 1 t j
(11)
K
k=1
(10)
and Yi1 and Y j0 are the diagnostic test results for the diseased
and nondiseased individuals, respectively. The variance
of the estimated AUC is computed using Mann-Whitney
Statistic [10]:
g ROC (t) =
(17)
=
AUC
1
1 + 2
(18)
NSTE-ACS subjects
0
2(a/b) (1/b) 12
,
Y1 Uniform l =
2
2(a/b) + (1/b) 12
r=
,
2
Y0 Uniform l = 3, r = 3 ,
Non-ACS subjects
(19)
120
240
360
480
Myoglobin levels
600
720
Table 1: Means of the parameter estimates and AUCs with their standard errors, RMSE, and 95% confidence intervals (CI) obtained from
the parametric, semiparametric, and nonparametric ROC methods using various sample sizes from 1000 simulated data sets generated from
the normal distribution.
n1 : n0
Methods
P
15 : 15
S
N
P
25 : 25
S
N
P
50 : 50
S
N
P
100 : 100
S
N
a
1.465
1.578
1.432
1.446
1.429
1.430
1.411
1.409
SE(a)
0.451
0.507
0.341
0.356
0.240
0.245
0.168
0.170
b
0.939
1.170
0.911
1.007
0.914
0.966
0.905
0.932
SE(b)
0.242
0.302
0.182
0.202
0.129
0.137
0.091
0.093
Parameters
a = 1.400, b = 0.900 and AUC = 0.850
BIAS for AUC
SE AUC
RMSE for AUC
AUC
0.844
0.006
0.067
0.069
0.822
0.028
0.072
0.075
0.849
0.001
0.070
0.071
0.847
0.003
0.052
0.053
0.834
0.016
0.052
0.058
0.850
0.000
0.054
0.054
0.850
0.000
0.037
0.037
0.843
0.007
0.037
0.039
0.852
0.002
0.038
0.038
0.850
0.000
0.026
0.027
0.846
0.004
0.026
0.027
0.851
0.001
0.027
0.027
Table 2: Means of the parameter estimates and AUCs with their standard errors, RMSE, and 95% confidence intervals (CIs) from the
parametric, semiparametric, and nonparametric ROC methods using various sample sizes from 1000 simulated datasets generated from the
lognormal distribution.
n1 : n0
Methods
P
15 : 15
S
N
P
25 : 25
S
N
P
50 : 50
S
N
P
100 : 100
S
N
a
0.729
1.600
0.693
1.461
0.646
1.420
0.601
1.406
SE(a)
0.303
0.511
0.230
0.358
0.159
0.243
0.111
0.169
b
0.236
1.176
0.216
1.020
0.196
0.955
0.183
0.930
SE(b)
0.061
0.304
0.043
0.204
0.028
0.135
0.018
0.093
Parameters
a = 1.400, b = 0.900 and AUC = 0.850
AUC
BIAS for AUC
SE AUC
RMSE for AUC
0.754
0.096
0.088
0.113
0.822
0.028
0.224
0.079
0.848
0.002
0.070
0.073
0.746
0.104
0.067
0.116
0.835
0.015
0.109
0.059
0.851
0.001
0.054
0.056
0.734
0.116
0.050
0.127
0.843
0.007
0.058
0.038
0.851
0.001
0.038
0.038
0.721
0.129
0.036
0.133
0.846
0.004
0.037
0.026
0.851
0.001
0.027
0.026
Table 3: Means of the parameter estimates and AUCs with their standard errors, RMSE, and 95% confidence intervals (CI) obtained from
the parametric, semiparametric and nonparametric ROC methods using various sample sizes from 1000 simulated data sets generated from
the uniform distribution.
n1 : n0
Methods
P
15 : 15
S
N
P
25 : 25
S
N
P
50 : 50
S
N
P
100 : 100
S
N
a
1.434
1.402
1.418
1.373
1.413
1.357
1.402
1.350
SE(a)
0.440
0.474
0.338
0.353
0.238
0.242
0.167
0.169
b
0.906
1.130
0.907
1.060
0.902
0.992
0.908
0.967
SE(b)
0.234
0.292
0.181
0.212
0.128
0.140
0.090
0.097
Parameters
a = 1.400, b = 0.900 and AUC = 0.850
BIAS for AUC
SE AUC
RMSE for AUC
AUC
0.847
0.003
0.067
0.067
0.810
0.040
0.068
0.084
0.837
0.013
0.073
0.077
0.847
0.003
0.052
0.050
0.820
0.030
0.053
0.062
0.835
0.015
0.057
0.056
0.850
0.000
0.037
0.036
0.829
0.021
0.038
0.045
0.835
0.015
0.040
0.044
0.850
0.000
0.026
0.024
0.832
0.018
0.027
0.033
0.834
0.016
0.029
0.032
Table 4: The descriptive statistics and normality test results for the myoglobin levels in the ACS data set.
Groups
Mean
SD
Median
Minimum
Maximum
NSTE-ACS
Non-ACS
62
20
178.03
54.75
194.27
64.29
104
33.95
20
11.60
800
304
Shapiro-Wilk test
of normality
P < 0.001
P < 0.001
Table 5: The results of applying the parametric, semiparametric, and nonparametric ROC methods to the ACS data set.
n1 : n0
62 : 20
Methods
Parametric
Semiparametric
Nonparametric
a
0.635
1.310
SE(a)
0.249
0.331
b
0.331
1.018
SE(b)
0.060
0.185
AUC
0.727
0.821
0.845
SE(AUC)
0.049
0.030
0.044
4. Conclusions
We propose using the semiparametric GLM ROC method
to assess the accuracy of a continuous diagnostic test if
the test results have a skewed distribution. The robust
estimators of this method provide a smooth ROC curve,
which is important when determining the optimal cuto
value. The model also has the advantage of being easy to
implement in certain statistical packages. If the results of
TP rate
0.8
0.6
0.4
0.2
0
0
0.2
0.6
0.4
0.8
FP rate
References
[1] D. M. Green and J. A. Swets, Signal Detection Theory and
Psychophysics, Wiley, New York, NY, USA, 1966.
[2] J. P. Egan, Signal Detection Theory and ROC Analysis, Academic Press, New York, NY, USA, 1975.
[3] L. B. Lusted, Logical analysis in roentgen diagnosis, Radiology, vol. 74, pp. 178193, 1960.
[4] J. A. Swets and R. M. Pickett, Evaluation of Diagnostic Systems:
Methods from Signal Detection Theory, Academic Press, New
York, NY, USA, 1982.
[5] M. S. Pepe, The Statistical Evaluation of Medical Tests for
Classification and Prediction, Oxford University Press, New
York, NY, USA, 2003.
MEDIATORS
of
INFLAMMATION
The Scientific
World Journal
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Gastroenterology
Research and Practice
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Journal of
Diabetes Research
Volume 2014
Volume 2014
Volume 2014
International Journal of
Journal of
Endocrinology
Immunology Research
Hindawi Publishing Corporation
http://www.hindawi.com
Disease Markers
Volume 2014
Volume 2014
PPAR Research
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Volume 2014
Journal of
Obesity
Journal of
Ophthalmology
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Evidence-Based
Complementary and
Alternative Medicine
Stem Cells
International
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Volume 2014
Journal of
Oncology
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Volume 2014
Parkinsons
Disease
Computational and
Mathematical Methods
in Medicine
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
AIDS
Behavioural
Neurology
Hindawi Publishing Corporation
http://www.hindawi.com
Volume 2014
Volume 2014
Volume 2014