James A. Hanley, Ph.D.

Barbara J. McNeil, M.D., Ph.D.

A Method of Comparing the Areas
under Receiver Operating
Characteristic Curves Derived from
the Same Cases’

Receiver operating characteristic (ROC) S EVERAL questions dealing with comparative benefits for altemna-
curves are used to describe and compare tive diagnostic algorithms, diagnostic tests, or therapeutic regi-
the performance of diagnostic technology mens have recently emerged in medicine. For example, how do we
and diagnostic algorithms. This paper re- know whether one diagnostic algorithm is better than another in
fines the statistical comparison of the sorting patients into diseased and nondiseased groups? Whether the
areas under two ROC curves derived addition of a new test or procedure to an established algorithm im-
from the same set of patients by taking proves its performance? Whether it matters who of several available
into account the correlation between the readers interprets a mammogram? Whether one type of hard-copy
areas that is induced by the paired nature unit in radiology is better than another? Whether reading a CT scan
of the data. The correspondence between in conjunction with the patient’s history allows a more accurate di-
the area under an ROC curve and the Wil- agnosis than reading it without the history? The analyses of such
coxon statistic is used and underlying problems have started with construction of receiver operating char-
Gaussian distributions (binormal) are as- acteristic (ROC) curves (1-3). Generally these analyses have used as
sumed to provide a table that converts the cutoff points either different posterior probabilities on a continuous
observed correlations in paired ratings of scale or different thresholds on a discrete rating scale. The latter ap-
images into a correlation between the two proach has been particularly popular in radiology.
ROC areas. This between-area correlation Major gaps in the understanding of statistical properties of ROC
can be used to reduce the standard error curves have limited their usefulness, especially for questions in-
(uncertainty) about the observed differ- volving comparisons of curves based on the same sample of subjects
ence in areas. This correction for pairing, or objects. These comparative situations contrast with those involving
analogous to that used in the paired t- a single data set and a single ROC curve. In such cases, the investigator
test, can produce a considerable iflcrease generally only needs to know that a single modality or diagnostic
in the statistical sensitivity (power) of the approach has “poor”, “moderate”, or “good” accuracy, and the loca-
comparison. For studies involving multi- tion of the ROC curve gives a rough assessment. However, when a
pie readers, this method provides a mea- comparison of two algorithms or modalities is relevant, more formal
sure of a component of the sampling van- statistical criteria are needed in order to judge whether observed
ation that is otherwise difficult to obtain. differences in accuracy are more likely to be random than real. Thus
far these criteria have not been fully developed for ROC curves.
Index term: Receiver operating characteristic curve In a recent paper (4) we dealt with one popular accuracy index that
(ROC) can be derived from and used as a summary of the ROC curve. We
showed that the relationship of the area under the ROC curve to the
Radiology 148: 839-843, September 1983
Wilcoxon statistic could be used to derive its statistical properties, such
as its standard error (SE) and the sample sizes required to measure the
area with a prespecified degree of precision (reliability) and to provide
a desired level of statistical power (low type II error) in comparative
experiments. This paper extends our statistical analysis to another
large class of situations, where the two or more ROC curves are gen-
erated using the same set of patients. In these situations, it is map-
propriate to calculate the standard error of the difference between
two areas (Area1 and Area2) as

SE (Area1 - Area2) s/E(Ar#{234}ai)-l-SE(Ar#{234}a2) (1)

1 From the Department of Epidemiology and
Health, McGill University, Montreal, Canada (J.A.H.)
since Area1 and Area2 are likely to be correlated. This correlation is
and the Department of Radiology, Harvard Medical likely to be positive; if the vagaries of random sampling of cases
School and Brigham and Women’s Hospital, Boston, produce a higher/lower than expected accuracy index for one mo-
MA, USA (B.J.M.). Received June 3, 1981; revision re- dality (e.g., if the sample consisted of a larger than usual number of
quested July 21, 1981; final revision received and ac-
easy/difficult cases), then the accuracy of the second modality will
cepted Feb. 15, 1983.
Supported in part by the Hartford Foundation and probably also be correspondingly higher/lower than one would ex-
the National Center for Health Care Technology. ht pect. In other words, while the two indices may fluctuate indepen-

839
dently by amounts SE1 and SE2 in sep- tamed in three ways: (i) by the trape- ratings (rN among the normals, rA
arate samples, they will tend to fluc- zoidal rule; (ii) as output from the among the abnormals) are obtained, it
tuate in tandem when derived from a Dorfman and Alf maximum likelihood is necessary to calculate the correlation
single sample. estimation program (5); or (iu) from the that they induce between the two areas
In this paper we have developed an slope and intercept of the original data 1 and A,. for ease of notation we have
approach to take account of this corre- when plotted on binormal graph paper called this r (without any subscript).
lation. In brief, we indicate that the (3). As indicated in our companion This is the coefficient present in
relevant standard error for such com- paper (4) the trapezoidal approach Equations 2 and 3. Tabulation of r
parisons is not that shown in Equation systematically underestimates areas. (TABLE I) is the fundamental contribu-
1 but rather Because the Dorfman and AIf approach tion of this papers; therefore, in our
is becoming readily accessible to those subsequent example we will illustrate
SE(Ar#{234}a1- Area2)
interested in this area, we will calculate its use.
= s/E(Ar#{234}ai) + SE2(Ar#{234}a2)
areas using this approach. (For those
-2rSE(Ar#{234}a1) SE(Ar#{234}a2) (2) limited to graphical methods, the area
Experimental Data for
can be derived from the slope and in-
where r is a quantity representing the Illustrative Examples
tercept according to the rule Area
correlation introduced between the
Percentage of Gaussian distribution to We studied 1 12 phantoms that were
two areas by studying the same sample
left of ZA ‘ where ZA Inter- specially constructed to evaluate the
of patients. This paper reviews the
cept/1 + slope2). accuracy of two different computer al-
calculations for comparing the ROC
gorithms used in image reconstruction
curves of two modalities and illustrates
for CT. Fifty-eight of these phantoms
this new approach using data from a Calculating Standard Errors
were of uniform density and were
series of experiments involving phan-
The standard errors associated with designated “normal”; the remaining 54
toms.
areas can be obtained in three ways: (i) contained an area of reduced density to
as output directly from the Dorfman simulate a lesion and were designated
METHODS and Alf maximum likelihood estima- “abnormal”. Two images of each
tion program; (ii) from the variance of phantom were reconstructed using the
The general approach to assessing
the Wilcoxon statistic as illustrated in two different algorithms, which we
whether the difference in the areas
detail in Reference 4; or (iii) from an will refer to as modality 1 and modality
under two ROC curves derived from
approximation to the Wilcoxon statistic 2. A single reader read each image and
the same set of patients is random or
by making an assumption, shown to be rated it on a 6-point scale: 1 = Defi-
real is to calculate a critical ratio z, de-
conservative (compared with assuming nitely Normal; 2 Probably Normal;
fined as
a Gaussian-based ROC curve), that the 3 = Possibly Normal; 4 Possibly Ab-
- A1-A2 3 underlying signal (diseased) and noise normal; 5 = Probably Abnormal; 6
z - VSE + SE 2rSE1SE2 (nondiseased) distributions are expo- Definitely Abnormal. From the re-
nential in type (4). We will use the sulting data, we constructed two ROC
where and SE1 refer to the observed
standard errors estimated from the curves. The data were submitted to the
area and estimated standard error of
Dorfman and Alf program. Dorfman and Alf maximum likelihood
the ROC area associated with modality
program to produce areas under the
1; where A2 and SE2 refer to corre-
Calculating the Correlation ROC curves and standard errors.
sponding quantities for modality 2; and
where r represents the estimated cor- Coefficient, r, Between Areas
relation between and A2.2 This Two intermediate correlation coef- RESULTS
quantity z is then referred to tables of ficients are required, which are then Our results will be divided into two
the normal distribution and values of converted into a correlation between parts. First, the analysis of the example
z above some cutoff, e.g. z ,
1 .96, are 1 and A2 via a table that we supply involving CT phantoms will be illus-
taken as evidence that the “true” ROC below. The first is rN, the correlation trated. Then, in order to verify that the
areas are different. The importance of coefficient for the ratings given to im- z statistic performs correctly, results of
introducing the 2rSE1SE2 term in the ages from nondiseased patients by the several simulations will be summa-
above equation is obvious: failure to
two modalities. The second is rA the rized.
subtract out from the sampling van- correlation coefficient for the ratings
ability those fluctuations that the
of diseased patients imaged by the two
paired design has already eliminated modalities. Each of these can be calcu- CT Phantom Example
will leave the denominator of Equation lated in traditional ways using either The basic data are presented in the
3 too large and z too small, thereby re-
the Pearson product-moment correla- Appendix, along with the calculations
ducing the chance of detecting a dif-
tion method or the Kendall tau. The produced from them. The areas under
ference between two modalities.
former approach is usually used for the ROC curves were 89.45% (SE 3.0%)
results derived from an interval scale and 93.82% (SE 2.6%). The (Kendall tau)
Calculating Areas whereas the latter is more appropriate correlations between the paired ratings
for results obtained from an ordinal were rN 0.39 (nondiseased patients)
Areas under ROC curves can be ob-
scale. ROC curves in radiology are de- and rA 0.60 (diseased patients), giv-
rived from ordinal scale data and ing an “average” correlation between
therefore we have used the Kendall tau the ratings of 0.50. With this average
2 we will see later, the SE of an estimated for calculating rN and rA. Standard correlation of 0.50 and with an average
area depends on the magnitude
of the underlying statistical packages (e.g., SPSS, SAS) area of (89.45 + 93.82)/2 91.64, TABLE
or “true” area. When calculating
z to test the null provide tau; when the number of rat-
hypothesis that this underlying area is the same
ing categories is small, however, say
for both modalities, one should equate SE5 and
SE,., calculating them both from a common esti-
four or less, the calculation can also be
mate of the area. In this case the denominator performed manually. 3 Mathematical derivation available upon re-
becomes 2SE(1 - r) or SE 2(1 - r). Once the correlations between the quest.

840 #{149}
Radiology September 1983
 I indicates that the correlation r be- TABLE I: Correl ation Coefficients*
tween areas is approximately 0.40.
Average
Equation 3 was then used to calculate
Correlation
the critical ratio z in order to test the between Average Area
null hypothesis that the observed dif- Ratingst .700 .725 .750 .775 .800 .825 .850 .875 .900 .925 .950 .975
ference between observed areas was
0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.02 0.01 0.01 0.01 0.01 0.01
merely a result of random sampling.
0.04 0.04 0.04 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.02 0.02 0.02
Using the above data 0.06 0.05 0.05 0.05 0.05 0.05 0.05 0.05 0.04 0.04 0.04 0.03 0.02
0.08 0.07 0.07 0.07 0.07 0.07 0.06 0.06 0.06 0.06 0.05 0.04 0.03
z = (0.9382 - 0.8945)/ 0.10 0.09 0.09 0.09 0.09 0.08 0.08 0.08 0.07 0.07 0.06 0.06 0.04
0.12 0.11 0.11 0.11 0.10 0.10 0.10 0.09 0.09 0.08 0.08 0.07 0.05
/OOz+ 0.0262 2(0.4)(0.03)(0.026)
0.14 0.13 0.12 0.12 0.12 0.12 0.11 0.11 0.11 0.10 0.09 0.08 0.06
= 0.0437/0.0309 = 1.41 0.16 0.14 0.14 0.14 0.14 0.13 0.13 0.13 0.12 0.11 0.11 0.09 0.07
0.18 0.16 0.16 0.16 0.16 0.15 0.15 0.14 0.14 0.13 0.12 0.11 0.09
As mentioned earlier, one might av- 0.20 0.18 0.18 0.18 0.17 0.17 0.17 0.16 0.15 0.15 0.14 0.12 0.10
erage the two areas to obtain a common 0.22 0.20 0.20 0.19 0.19 0.19 0.18 0.18 0.17 0.16 0.15 0.14 0.11
0.24 0.22 0.22 0.21 0.21 0.21 0.20 0.19 0.19 0.18 0.17 0.15 0.12
area of 0.9164, and use the formula in 0.26 0.24 0.23 0.23 0.23 0.22 0.22 0.21 0.20 0.19 0.18 0.16 0.13
Reference 4 to predict each of the 0.28 0.26 0.25 0.25 0.25 0.24 0.24 0.23 0.22 0.21 0.20 0.18 0.15
standard errors as 0.0281; using 0.0281 0.30 0.27 0.27 0.27 0.26 0.26 0.25 0.25 0.24 0.23 0.21 0.19 0.16
0.32 0.29 0.29 0.29 0.28 0.28 0.27 0.26 0.26 0.24 0.23 0.21 0.18
/2(1 - 0.4) = 0.0308 in the denomi- 0.34 0.31 0.31 0.31 0.30 0.30 0.29 0.28 0.27 0.26 0.25 0.23 0.19
nator of Equation 3 yields an almost 0.36 0.33 0.33 0.32 0.32 0.31 0.31 0.30 0.29 0.28 0.26 0.24 0.21
identical z value of 1.42. 0.38 0.35 0.35 0.34 0.34 0.33 0.33 0.32 0.31 0.30 0.28 0.26 0.22
If we have reason to believe a priori 0.40 0.37 0.37 0.36 0.36 0.35 0.35 0.34 0.33 0.32 0.30 0.28 0.24
0.42 0.39 0.39 0.38 0.38 0.37 0.36 0.36 0.35 0.33 0.32 0.29 0.25
that modality 2 is likely to be better 0.44 0.41 0.40 0.40 0.40 0.39 0.38 0.38 0.37 0.35 0.34 0.31 0.27
than modality 1 and are only interested 0.46 0.43 0.42 0.42 0.42 0.41 0.40 0.39 0.38 0.37 0.35 0.33 0.29
in improvements, then a one-tailed test 0.48 0.45 0.44 0.44 0.43 0.43 0.42 0.41 0.40 0.39 0.37 0.35 0.30
0.50 0.47 0.46 0.46 0.45 0.45 0.44 0.43 0.42 0.41 0.39 0.37 0.32
is appropriate. The Gaussian distribu- 0.52 0.49 0.48 0.48 0.47 0.47 0.46 0.45 0.44 0.43 0.41 0.39 0.34
tion indicates that a value of 1.41 or 0.54 0.51 0.50 0.50 0.49 0.49 0.48 0.47 0.46 0.45 0.43 0.41 0.36
higher should occur roughly once in 0.56 0.53 0.52 0.52 0.51 0.51 0.50 0.49 0.48 0.47 0.45 0.43 0.38
0.58 0.55 0.54 0.54 0.53 0.53 0.52 0.51 0.50 0.49 0.47 0.45 0.40
every 13 samples (p = 0.079); this evi- 0.60 0.57 0.56 0.56 0.55 0.55 0.54 0.53 0.52 0.51 0.49 0.47 0.42
dence suggests that the observed dif- 0.62 0.59 0.58 0.58 0.57 0.57 0.56 0.55 0.54 0.53 0.51 0.49 0.45
ference may not be random. This con- 0.64 0.61 0.60 0.60 0.59 0.59 0.58 0.58 0.57 0.55 0.54 0.51 0.47
0.66 0.63 0.62 0.62 0.62 0.61 0.60 0.60 0.59 0.57 0.56 0.53 0.49
trasts with the weaker inference that 0.68 0.65 0.64 0.64 0.64 0.63 0.62 0.62 0.61 0.60 0.58 0.56 0.51
would be drawn from a critical ratio of 0.70 0.67 0.66 0.66 0.66 0.65 0.65 0.64 0.63 0.62 0.60 0.58 0.54
1.10 (or a p value of 0.136 or 1 in 7) that 0.72 0.69 0.69 0.68 0.68 0.67 0.67 0.66 0.65 0.64 0.63 0.60 0.56
0.74 0.71 0.71 0.70 0.70 0.69 0.69 0.68 0.67 0.66 0.65 0.63 0.59
would have been calculated had the 0.76 0.73 0.73 0.72 0.72 0.72 0.71 0.71 0.70 0.69 0.67 0.65 0.61
correlation between areas been as- 0.78 0.75 0.75 0.75 0.74 0.74 0.73 0.73 0.72 0.71 0.70 0.68 0.64
sumed to be equal to zero (in other 0.80 0.77 0.77 0.77 0.76 0.76 0.76 0.75 0.74 0.73 0.72 0.70 0.67
0.82 0.79 0.79 0.79 0.79 0.78 0.78 0.77 0.77 0.76 0.75 0.73 0.70
words, had we failed to take into ac-
0.84 0.82 0.81 0.81 0.81 0.81 0.80 0.80 0.79 0.78 0.77 0.76 0.73
count the increased sensitivity induced 0.86 0.84 0.84 0.83 0.83 0.83 0.82 0.82 0.81 0.81 0.80 0.78 0.75
by studying the same set of patients 0.88 0.86 0.86 0.86 0.85 0.85 0.85 0.84 0.84 0.83 0.82 0.81 0.79
0.90 0.88 0.88 0.88 0.88 0.87 0.87 0.87 0.86 0.86 0.85 0.84 0.82
with both modalities). If we had no a
priori interest in one particular direc- S Correlation coefficient r between two ROC areas and 2 as a function of average correlation

tion, then a two-tailed test would have between ratings (rows) and average area (columns).
t (rN + TA)!2.
been appropriate. t (A1 + A2)/2.

General Performance of the

vations acceptably close to 1 . The a 50% sensitivity (expected from an
Paired Test
false-positive rates were low, and close unpaired analysis) to 60-75%.
A good statistical test should mdi- to what one should expect. Specifically,
cate a difference when one is really among the 4,800 trials (12 combina-
DISCUSSION
present, but it should minimize in a tions each run 400 times) the average
predictable way the number of in- proportion of z values above 2.0 or In this investigation we have de-
stances in which a difference is said to below -2.0 (values often taken as in- scnibed a method comparing of the
exist when, in fact, none does exist dicating a statistically significant dif- areas under two ROC curves derived
(high sensitivity and specificity). To ference) was 5.1%, i.e., a specificity of from the same sample of patients. Two
determine these characteristics for this 94.9%. In a perfect Gaussian distnibu- immediate results are apparent. We
new statistical test, we
examined its tion, this would have been 4.6%, i.e., a have shown that the comparison can be
diagnostic performance over a range of specificity of 95.4%. made more sensitive if the investigator
simulated situations, using methods Evaluation of the sensitivity (power) takes into account the smaller sampling
analogous to those used by Pollack and for this statistic required comparison of variability of the difference in areas
Hsieh (6) and Metz and Kronman (7). two modalities with different ROC induced by studying each patient
In order to calculate the specificity, areas. For this purpose, sets of 200 ex- twice. Second, our data can be extrap-
400 simulated analyses were performed peniments were simulated from vary- olated to indicate the statistical econo-
for each of several combinations of ing correlation coefficients. The per- my that emerges from this kind of ex-
underlying ROC areas and correla- formance was evaluated by tabulating perimental design and analysis. We
tions. The tabulated distributions of the the percentage of paired and unpaired discuss these points in turn.
test statistic z obtained from these tests indicating a significant difference. The larger the correlation between
various simulations were for all prac- Over four combinations of correlations the areas, the more sensitive the paired
tical purposes indistinguishable from and baseline accuracies, one could z test will be. This observation may
Gaussian ones and had standard den- project that the paired test would raise explain why a-number of studies using

Volume 148 Number 3 Radiology #{149}

841
an unpaired z test that assumed the two equal; in this case, Equation 2 differs taming r now allows the methods
areas were statistically independent from Equation 1 only in the presence of therein to be used with greater sensi-
failed to find a significant difference the factor (1 - r). When the sample tivity. This is best appreciated by re-
between the modalities. The degree of sizes associated with the two tech- producing the formula that the authors
correlation expected between ROC niques are arranged so that the paired give (Equation 2, Chapter 3) for the
areas obtained with different modali- and unpaired tests produce the same z standard error of a difference between
ties varies considerably depending value, then a simple algebraic identity the value of an accuracy index (such as
upon the types of modalities involved. emerges: the area under an ROC curve) for one
For example, if the two images are ob- modality (averaged over 1 readers, each
tamed from the same machine with flu = fl/(1 - r) reading each image m times) and the
two different settings or if a radiologist or value of the same accuracy index (again
reads a CT scan with and without ex- averaged over readers and readings)
tensive clinical history, high correla- flp = (1 - r)fl for a second modality. The expression
tion can be expected. In this study in- where n and fl are the numbers of involves three sources of variation: S,
volving different reconstruction algo- patients per modality in the respective the variation in the index due to dif-
nithms with CT, the correlation be- unpaired and paired designs4. For ex- ferences in mean difficulty of cases
tween the paired ratings of abnormal ample, if r is anticipated to be roughly from case sample to case sample; S,
phantoms was 0.60 and between paired 0.3 and an unpaired design called for between-reader variance due to dif-
ratings of normal phantoms was 0.39. 100 patients per modality, then a paired ferences in diagnostic capability from
We have observed similar results in a design should require only 70 per reader to reader; and within-reader
study of ours (8) involving the inter- modality. Thus the total number of variance due to differences in an mdi-
pretation of CT studies of the head images read would be 140 rather than vidual reader’s diagnoses of the same
with and without extensive clinical 200. This efficiency is even more im- case in repeated occasions. It also in-
history. On the other hand, when the portant if the limiting factor is the volves two correlation coefficients: r
only common denominator in the number of available patients with a to denote the correlations introduced
comparison is the patient, the correla- proved outcome (rather than the by using similar (or even the same)
tions are likely to be weaker. For ex- number of images a reader can be ex- cases with both modalities and rb to
ample, a study by Alderson et al.(9) pected to read), since the total of 140 denote correlations between the accu-
comparing CT, ultrasound, and nuclear paired images is obtained from just 70 racy index obtained by using matched
medicine imaging in the diagnosis of patients, rather than from 200 patients (or possibly the same) readers. With
liver metastases found considerably in the unpaired design. The investi- this notation, the formula becomes
lower rating-pair correlations (0.36 in gator must weigh very carefully the SE (differefice)
abnormal patients and 0.28 in normal practical and statistical issues, keeping
patients). Obviously, in the latter sit- J2IS(1 - r) + S(1 - rj,)/l + S/lm
in mind that if one uses an unpaired
uation the gains from using a paired design, one must establish (through
rather than an unpaired analysis are case matching and/or random alloca- The authors describe fully via several
smaller. tion) that the method of constructing worked examples how to evaluate each
Two other points must be made two independent samples of subjects of these terms. They point out, how-
about correlation coefficients. First, in does not give one modality an inbuilt ever, that the estimation of the two
general we have noted that whatever advantage. components r and S creates prob-
the modalities under study, the matings The discussion thus far has centered lems. First, if m 1, i.e., if each image
tend to be less correlated in the non- on a rather restricted design where just is read just once, then S. and S are
diseased patients than in the diseased one reader read the images generated not separable, and one is forced to
patients. This suggests that in diag- by the two modalities being compared. overestimate the SE. The second, and
nostic imaging agreement tends to be The statistical test simply asked the more serious, problem is that if m 1
greater if there is in fact underlying question: if this one reader read an in- and if one does not have a large num-
disease, and less if there is not. Second, finite rather than a finite number of ben of cases, enough (for example) to
if an investigator knew a priori that the images, would his/her accuracy be split them into a number of subsamples
correlations between the modalities comparable in both modalities?5 and fit an ROC curve to each, one is
under study were small, then an ex- Clearly, a more general question is unable to estimate r. In such cases, the
penimental design that did not involve relevant: how do the modalities com- authors explain that one has no alter-
pairing could be used, provided that it pare over many readers? native but to assume r 0, thereby
was no more difficult to separate For the sake of completeness, we giving up any benefits attainable from
(diagnose) the patients studied by one refer briefly to this problem of multiple case matching.
modality than it was to diagnose those readers and readings in each modality. The method we have presented here
studied by the other modality. This situation has been discussed ex- means that if one uses the area under
The statistical economy resulting tensively by Swets and Pickett (10); our the ROC curve as an index of accuracy,
from this new statistical test is large. main reasons for mentioning it here are one is not forced to assume r = 0. The
Statistical economy relates to the to draw readers’ attention to a very quantity we have called r, which is
question of how many more patients extensive treatment of the design and obtainable via TABLE I from the area
are required in an unpaired design analysis of imaging experiments, and and from the correlations between
then in a paired design to achieve the to point out that our method of ob- ratings, is the same quantity
same sensitivity or statistical power. A mentioned in Equation 5, Chapter 4 of
comparison of Equations 1 and 2 pro- Swets and Pickett (8)6. The interested
vides an answer to this question. Each
of the standard errors is inversely 4 This simple relation allows the user to mul-
proportional to the square root of the tiply the sample sizes in TABLE III of our first 6 If m > 1, one can correct the quantity
publication (4) by the appropriate (1 - r)and use (obtained from TABLE I) for the “attenuation”
sample size n . Also, the equations can them for paired designs. produced by S, and estimate the “true” corre-
be simplified by assuming that the 5 One could also use the z test to compare two lation r introduced by using similar(or the same)
standard errors of the two areas are specific readers on one modality. cases.

842 #{149}
Radiology September 1983
reader can consult that reference for APPENDIX
full details on how it is used. Ratings given to two images of each of
of z test to test whether one modality sub-
In summary, then, we have provided
1 12 phantoms, together with calculations tended a greater ROC area than the other:
a method for estimating the correlation
between the areas under two ROC (a) Basic data:
curves derived from the same sample
of patients and have shown how to use Rating* Rating* with Modality 2
this correlation to perform a more with Normal Phantoms Abnormal Phantoms
sensitive comparison of the areas. Modalityl 1 2 3 4 5 6 Total 1 2 3 4 5 6 Total
Moreover, this provides an item that
1 9 3 - - - - 12 - - 1 - - - 1
was previously only guessed at, or un-
2 17 9 2 - - - 28 1 - 2 - - - 3
denestimated, in studies of multiple 3 3 4 1 - - - 8 1 1 1 3 - - 6
readers. 4 1 2 2 1 - - 6 1 1 1 9 1 - 13
5 1 1 - 2 - - 4 - - - 7 10 5 22
6 4 5 9
Acknowledgments: We are indebted to Total 31 19 5 3 - - 58 3 2 5 19 15 10 54
Richard Swensson and Philip Judy for * Ratings: from 1 Definitely Normal to 6 Definitely Abnormal.
providing the CT phantoms and rereading
results. We are also indebted to Charles
Metz for many helpful discussions in the
course of this investigation. Irene McCam- (b) Correlation between ratings (Kendall
mon typed the manuscript. tau): modality 1 vs. modality 2 TA = 0.60

Department of Epidemiology and Health

modality 1 vs. modality 2 TN 0.39 average correlation 0.50
McGill University
3775 University Street
Montreal, Quebec
Canada H3A 2B4
(c) ROC analysis:

Rating Standard
References Modality 1 2 3 4 5 6 Slope Intercept ZA Area Error (Area)
1. Green D, SwetsJA. Signaldetection theory
Modality 1
and psychophysics. New York: John Wiley
Normal 12 28 8 6 4 - 0.945 1.72 1.25 0.8945 0.030
and Sons, 1966.
Abnormal 1 3 6 13 22 9
2. Metz CE Basic principles of ROC analysis. Modality 2
Semin Nucl Med 1978; 8:283-298. Normal 31 19 5 3 - - 0.467 1.70 1.54 0.9382 0.026
3. Swets JA. ROC analysis applied to the Abnormal 3 2 5 19 15 10
evaluation of medical imaging techniques.
Invest Radiol 1979; 14:109-121. Difference in areas 0.0437.
4. Hanley JA, McNeil BJ. The meaning and Average area 0.9146.
use of the area under a receiver operating Correlation between areas 0.40.
characteristic (ROC) curve. Radiology 1982;
143:29-36.
5. Dorfman DD, AIf E. Maximum likelihood
estimation of parameters of signal detection (d) Test statistic: z 0.0437/0.0302 + 0.0262 2(0.040)(0.030)(0.026) 1.41
theory and determination of confidence
intervals-rating-method data. J Math
Psych 1969; 6:487-496.
6. Pollack I, Hsieh R. Sampling variability of
the area under the ROC-curve and of d.
Psych Bull 1969; 71:161-173.
7. Metz CE, Kronman HB. Statistical signifi-
cance tests for binormal ROC curves. J Math
Psych 1980; 22:218-243.
8. McNeil BJ, Hanley JA, Funkenstein HH,
Wallman J. Paired receiver operating
characteristic curves and the effect of history
on radiographic interpretation: CT of the
head as a case study. Radiology 1983; in
press.
9. Alderson P0, Adams DF, McNeil BJ, et al.
Computed tomography, ultrasound, and
scintigraphy of the liver in patients with
colon or breast carcinoma: A prospective
study. Radiology 1983; in press.
10. Swets JA, Pickett RM. Evaluation of diag-
nostic systems. New York: Academic Press,
1982.

Volume 148 Number 3 Radiology #{149}

843