䡲 NEURORADIOLOGY

Detection of Simulated Multiple

Sclerosis Lesions on T2-
weighted and FLAIR Images of
ORIGINAL RESEARCH

the Brain: Observer Performance1

John H. Woo, MD
Purpose: To determine observer performance in the detection of
Lana P. Henry, MD
multiple sclerosis (MS) lesions on magnetic resonance
Jaroslaw Krejza, MD, PhD2
(MR) images of the brain and to assess the dependence of
Elias R. Melhem, MD observer performance on lesion size, parenchymal location,
pulse sequence, and supratentorial versus infratentorial
level.

Materials and This HIPAA-compliant protocol was approved by the insti-

Methods: tutional review board, and previously acquired MR data
from a healthy volunteer and a patient with MS were used
to derive parameter maps, with waiver of informed con-
sent. Parameter maps and image simulator software were
used to generate 320 phantom brain images with simu-
lated supratentorial and infratentorial MS lesions. Images
were displayed with T2-weighting or fluid-attenuated in-
version recovery (FLAIR) contrast. Four readers indepen-
dently evaluated the images, rating lesions on a five-point
certainty scale. Observer performance was measured by
using the area under the alternative free-response receiver
operating characteristic curve (A1), and significance was
determined with the z test.

Results: Pooled A1 scores were significantly better for FLAIR imag-

ing (0.96 ⫾ 0.01 [standard error]) than for T2-weighted
MR imaging (0.89 ⫾ 0.04) supratentorially (P ⫽ .05) but
were similar for FLAIR imaging (0.90 ⫾ 0.06) and T2-
weighted MR imaging (0.88 ⫾ 0.05) infratentorially. A1
scores for cortical, deep white matter, and periventricular
lesions were 0.93 ⫾ 0.05, 0.97 ⫾ 0.02, and 0.89 ⫾ 0.04,
respectively, for FLAIR imaging and 0.77 ⫾ 0.06, 0.99 ⫾
0.01, and 0.89 ⫾ 0.05, respectively, for T2-weighted MR
imaging. FLAIR scores were significantly higher than T2-
weighted scores for cortical lesions. Linear correlation was
found between A1 and lesion size (r ⫽ 0.5).

Conclusion: Supratentorially, performance was better with FLAIR im-

aging than with T2-weighted MR imaging. Infratentorially,
performance was moderate with both modalities. Observ-
ers did better with FLAIR imaging in the detection of
1
From the Department of Radiology, Division of Neurora- cortical lesions, and performance improved with increas-
diology, Hospital of the University of Pennsylvania, 3400
ing lesion size.
Spruce St, Dulles 2, Philadelphia, PA 19104. From the
2004 RSNA Annual Meeting. Received May 9, 2005; revi-
sion requested July 7; revision received October 9; ac- 娀 RSNA, 2006
cepted November 14; final version accepted December
22. Address correspondence to J.H.W. (e-mail:
woojohn@uphs.upenn.edu).
2
Current address: Bialystok Medical Academy, Bialystok,
Poland.

姝 RSNA, 2006

206 Radiology: Volume 241: Number 1—October 2006

NEURORADIOLOGY: Detection of Simulated Multiple Sclerosis Lesions
M
agnetic resonance (MR) imag-
ing of the brain plays a crucial
role in the imaging of multiple
sclerosis (MS). However, it is difficult to
obtain objective measures of observer
performance in the task of MS lesion
detection on MR images. Lesions are (in
general) easily seen, particularly on T2-
weighted and fluid-attenuated inversion
recovery (FLAIR) MR images, but
pathologic correlation with tissue sam-
pling is rarely, if ever, performed. As a
result, any attempt to design a study by
using real MR images to measure ob-
server performance in the detection of
MS lesions would be impeded by this
lack of correlative truth data.
As a partial solution to this problem,
previous investigators used an MR im-
age simulator to create phantom images
with simulated lesions (1,2). Herskovits
et al (1) tested observers to assess how
performance depended on lesion loca-
tion and MR sequence. Their testing
paradigm was limited, however, by the
forced binary nature of the observer re-
sponse—a lesion was either present or
absent. As a result, their analysis
yielded only sensitivity and specificity
values as measures of performance.
Moreover, they did not test observers
with images of the posterior fossa,
where many researchers believe that
MS lesions are less detectable on FLAIR
images (3,4). Thus, the purpose of our
study was to determine observer per-
formance in the detection of MS lesions
on MR images of the brain and to assess
Advances in Knowledge
䡲 Observer performance on fluid-
attenuated inversion recovery
(FLAIR) images is superior to that
on T2-weighted MR images su-
pratentorially and is similar to
that on T2-weighted MR images
infratentorially.
䡲 Evidence indicates that perfor-
mance on FLAIR images is supe-
rior to that on T2-weighted MR
images in the detection of cortical
lesions.
䡲 Evidence indicates that perfor-
mance improves with increasing
lesion size.
Radiology: Volume 241: Number 1—October 2006
the dependence of observer perfor-
mance on lesion size, parenchymal loca-
tion, pulse sequence, and supratentorial
versus infratentorial level.
Materials and Methods
The details of the MR simulator have
been previously published (1,2). A spe-
cialized acquisition sequence was used
to derive MR parameter maps of the
brain in a healthy 40-year-old male vol-
unteer who had no known neurologic
disease. These images were judged to
be normal by all four authors (0 –12
years of experience in diagnostic neuro-
radiology). Parameter maps were simi-
larly acquired in a 32-year-old woman
with MS. Both participants had previ-
ously given written informed consent to
a prior study that was approved by the
institutional review board. For the cur-
rent study, which was also approved by
the institutional review board, informed
consent was waived. The study was
compliant with the Health Insurance
Portability and Accountability Act be-
cause only the MR data were used, with
all the identifying information deleted.
Template Data
A 1.5-T MR imager (ACS-NT; Philips
Medical Systems, Best, the Nether-
lands) was used to perform mixed mul-
tiecho spin-echo and inversion-recovery
MR imaging in the healthy volunteer.
This sequence yielded eight spin-echo
MR images (repetition time, 1500
msec) with different echo times (20, 40,
60, 80, 100, 120, 140, and 160 msec)
and eight inversion-recovery MR im-
ages (repetition time, 2000 msec; inver-
sion time, 400 msec) with the same
eight echo times. Section thickness was
5 mm, in-plane resolution was 0.80 ⫻
0.86 mm (rectangular field of view,
165 ⫻ 220 mm; matrix, 205 ⫻ 256), and
acquisition time was 9 minutes 30 sec-
onds. Pixel maps (256 ⫻ 256) of T1
relaxation rate, T2 relaxation rate, and
proton density were generated online
(software release 6.2; Philips Medical
Systems) by using the multiecho data.
The multiecho sequence was performed
twice, with images obtained in the brain
at two transverse levels (supratentorial
Woo et al
and infratentorial). The supratentorial
level included the lateral ventricles at
the septum pellucidum, and the infra-
tentorial level intersected the fourth
ventricle and orbital floors.
Lesion Data
By performing the same multiecho MR
sequence on the patient with MS, we
obtained similar parameter maps for a
lesion in the left centrum semiovale. All
four authors agreed on the presence of
this lesion. The lesion was digitally ex-
tracted by including only those pixels
that had T1, T2, and proton density val-
ues that were more than 2 standard de-
viations above the mean value in the
contralateral normal-appearing white
matter, which resulted in a lesion diam-
eter of 7 pixels. This threshold of 2 stan-
dard deviations above the mean was
previously shown to separate the lesion
from the surrounding white matter (2).
Resampling with bicubic interpolation
generated smaller lesions that were 2,
3, 4, 5, or 6 pixels in diameter.
MR Image (Test Case) Generation
The MR image simulator software used
a steady-state solution to the Bloch
equation and calculated signal intensity
(S) as a function of three tissue parame-
ters (ie, proton density [␳], T1, and T2)
and three machine parameters (repeti-
tion time [TR], echo time [TE], and in-
Published online
10.1148/radiol.2411050792
Radiology 2006; 241:206 –212
Abbreviations:
A1 ⫽ area under the AFROC curve
AFROC ⫽ alternative free-response receiver operating
characteristic
FLAIR ⫽ fluid-attenuated inversion recovery
MS ⫽ multiple sclerosis
Author contributions:
Guarantors of integrity of entire study, J.H.W., L.P.H.;
study concepts/study design or data acquisition or data
analysis/interpretation, all authors; manuscript drafting or
manuscript revision for important intellectual content, all
authors; manuscript final version approval, all authors;
literature research, J.H.W., L.P.H.; experimental studies,
J.H.W., L.P.H.; statistical analysis, J.H.W., J.K.; and
manuscript editing, all authors
Authors stated no financial relationship to disclose.
207
NEURORADIOLOGY: Detection of Simulated Multiple Sclerosis Lesions Woo et al

version time [TI]) at the x and y coordi- four readers were authors. Each reader any number of lesions, including zero,
nates: was instructed to find and designate all with no maximum specified. Each test-
suspected lesions on the images by us- ing session occurred at the same work-
S共x,y兲 ⫽ ␳共x,y兲兩1 ⫺ 2e⫺TI/T1共x,y兲兩 ing a point-and-click mouse interface station with a 21-inch monitor, and win-
(described below). Before actual testing dow width and center levels were cho-
关1 ⫺ e ⫺TR/T1共x,y兲兴e⫺TE/T2共x,y兲/
began, the reader was allowed to prac- sen to match those of standard MR
关1 ⫹ e ⫺TR/T1共x,y兲e⫺TR/T2共x,y兲兴. tice with this interface during a demon- images of the brain.
stration session. The reader was in- The 320 images, each linearly inter-
This is the same formula used in pre- formed that each image could contain polated to a 512 ⫻ 512 matrix, were
vious implementations of the MR simu-
lator (1,2), with an additional term in Figure 1
the denominator that better approxi-
mates steady-state signal intensity (5).
By applying this formula on a pixel-by-
pixel basis, we generated images with
either T2-weighting (4500/100 [repeti-
tion time msec/echo time msec]) or
FLAIR contrast (11 000/140/2600 [rep-
etition time msec/echo time msec/in-
version time msec]). A lesion was em-
bedded in a template by choosing the
greater of the signal intensities com-
puted for the lesion and normal brain
tissue.
A total of 80 supratentorial images
were first generated, with 20 images
each containing zero, one, two, or three
lesions. Thus, a total of 120 lesions were
placed. These lesions were deliberately
chosen in order to represent equally
each of the five lesion sizes (2, 3, 4, 5, or
6 pixels), with 24 lesions of each size.
An approximately equal number of le-
sions was also distributed among corti-
cal, deep white matter, and periven-
tricular locations, with about 40 lesions
in each location. This procedure was
repeated at the infratentorial level, with
similar attention given to the distribu-
tion of sizes and locations, thereby cre-
ating 80 more images. Each of the re-
sulting 160 images was used twice (once
with T2-weighting and once with FLAIR
contrast), yielding 320 total test cases
(Fig 1). All 320 images were shown to
each reader.

Testing
Our experiment followed a multiple-
reader multiple-case design. Four
board-certified neuroradiologists, each Figure 1: Representative simulated transverse FLAIR (11000/140/2600) and T2-weighted (4500/100)
in their 1st year of diagnostic neuroradi- MR images show lesions in same location. (a) At the supratentorial level, the callosal lesion (arrow) is seen
ology fellowship training, evaluated the equally well on T2-weighted (right) and FLAIR (left) images, but the cortical lesion (arrowhead) is much more
visible on the FLAIR image. (b) At the infratentorial level, the two lesions are seen equally well on T2-weighted
images by using testing software written
(right) and FLAIR (left) images, with one lesion (arrowhead) in the pontine white matter and the other (arrow) in
in Interactive Data Language (Research
the periventricular white matter near the fourth ventricle.
Systems, Boulder, Colo). None of the

208 Radiology: Volume 241: Number 1—October 2006

NEURORADIOLOGY: Detection of Simulated Multiple Sclerosis Lesions
displayed one at a time in a random
order, alternating between supratento-
rial and infratentorial images. After de-
tecting a possible lesion, the reader
used the mouse to locate the lesion with
a button click. A drop-down menu then
allowed the reader to assign one of the
following certainty ratings: 4, definite
lesion; 3, probable lesion; 2, possible
lesion; and 1, cannot exclude lesion.
These ratings reflected a decreasing
scale of reader certainty regarding the
presence of a lesion so that, for exam-
ple, a rating of 1 (cannot exclude lesion)
was less certain than a rating of 2 (pos-
sible lesion). The location and rating
score were recorded for each response.
If desired, the reader could delete a re-
sponse with a different button click.
This functionality was allowed, for ex-
ample, in case the reader mistakenly
clicked at an unintended location or if
the reader simply reconsidered a prior
response.
The reader was asked to locate and
score all lesions that were found on the
image within a 30-second time limit. A
visual signal indicated when 25 seconds
had elapsed, and at 30 seconds the im-
age would be erased from the screen. If
satisfied that all the lesions were found,
the reader could proceed to the next
image before the time limit expired.
Breaks were allowed between images,
and each reader was allowed to take the
test in two separate 160-image sessions
to reduce fatigue.
Scoring
The tests were scored by using the al-
ternative free-response receiver operat-
ing characteristic (AFROC) scoring
method (6). A response was deemed
true-positive if located within 5 pixels
(in both x and y coordinates) of a true
lesion. Each true-positive response was
assigned a “hit” score of 1– 4, depending
on the reader’s certainty (1 for least
certain to 4 for most certain). An unde-
tected lesion (ie, a false-negative find-
ing) was assigned a hit score of 0. In this
way, each lesion (detected or not) was
assigned a hit score of 0 – 4.
Any response not within 5 pixels of
a true lesion was deemed a false-posi-
tive response. For each image, the false-
Radiology: Volume 241: Number 1—October 2006
positive image score recorded the max-
imum rating of all false-positive re-
sponses for that image. For example, if
two false-positive responses were given,
with certainty scores of 1 and 2, then
the resulting false-positive image score
for that image would be 2. An image
with no false-positive responses was as-
signed a false-positive image score of 0
(true-negative result). In this way, each
image was assigned a false-positive im-
age score of 0 – 4.
For each reader, this scoring
method enabled the construction of an
AFROC curve (6), which is analogous to
the receiver operating characteristic
curve that is used in traditional receiver
operating characteristic studies. This
curve plotted the true-positive fraction
of lesions versus the false-positive frac-
tion of images. Observer performance
was quantified as the area under the
AFROC curve (A1), which was esti-
mated by using trapezoidal integration.
The four A1 values were averaged to-
gether to form a pooled metric of ob-
server performance.
Statistical Analysis
We applied the z test to assess for sta-
tistical significance at a P level of .05 for
comparing performance by using a sta-
tistical software program (Excel 2000;
Microsoft, Redmond, Wash). To apply
the z test for comparing pooled metrics
of observer performance, we needed to
estimate the appropriate means and
standard errors. Mean performance
was estimated by using the average of
the four A1 values of the four individual
readers. Then, as outlined by Hanley
(7), the overall standard error could be
estimated by using a function that effec-
tively sums the error components
caused by interreader and intercase
variability. Interreader variability was
estimated by using the sample variance
of the four individual A1 values. Inter-
case variability was estimated with a
jackknife approach as the sample vari-
ance of the resulting pseudovalues cal-
culated by the JAFROC software (version
1.00; www.devchakraborty.com) (8).
We tested for significant differences
in performance with respect to modality
(FLAIR vs T2-weighted MR imaging) at
Woo et al
the supratentorial and infratentorial
levels by using the z test. We also ap-
plied the z test to evaluate differences in
performance (pooled across supraten-
torial and infratentorial levels) in the
detection of lesions located at cortical,
deep white matter, and periventricular
locations. Finally, we evaluated the cor-
relation coefficient in order to see how
overall performance (pooled across
both modalities and both levels) varied
with lesion size.
Results
At the supratentorial level, observer
performance was excellent on FLAIR
images (pooled A1 score, 0.96 ⫾ 0.01
[standard error]) and only moderate on
T2-weighted MR images (pooled A1
score, 0.89 ⫾ 0.04). AFROC curves for
the four readers and pooled performance
values are plotted in Figure 2a (for T2-
weighted MR images) and 2b (for FLAIR
images). The difference in performance
between modalities was statistically sig-
nificant (z ⫽ 1.97, P ⫽ .05).
At the infratentorial level, pooled A1
scores showed moderate performance
on FLAIR images (0.90 ⫾ 0.06) and T2-
weighted MR images (0.88 ⫾ 0.05).
Corresponding AFROC curves are plot-
ted in Figure 3a (for T2-weighted MR
images) and 3b (for FLAIR images).
This difference was not statistically sig-
nificant (P ⫽ .76).
Performance also varied according
to lesion location (Fig 4), with scores for
the supratentorial and infratentorial
levels aggregated together. For cortical
lesions, performance was good on
FLAIR images (0.93 ⫾ 0.05) but only
moderate on T2-weighted MR images
(0.77 ⫾ 0.06). For deep white matter
lesions, performance was excellent on
FLAIR images (0.97 ⫾ 0.02) and T2-
weighted MR images (0.99 ⫾ 0.01). For
periventricular lesions, performance
was moderate on FLAIR images (0.89 ⫾
0.04) and T2-weighted MR images
(0.89 ⫾ 0.05). FLAIR scores were sig-
nificantly higher than T2-weighted MR
imaging scores for cortical lesions (z ⫽
2.06, P ⫽ .04) but not for deep white
matter (P ⫽ .51) or periventricular (P ⫽
.93) lesions. We also found a moderate
209
NEURORADIOLOGY: Detection of Simulated Multiple Sclerosis Lesions Woo et al

linear correlation (r ⫽ 0.5, P ⬍ .05) Perhaps the explanation lies in the ventricles are larger and the sulci are
between A1 and lesion size (Fig 5). anatomic differences between the pos- more abundant. This hypothesis, while
terior fossa and the cerebrum. Because reasonable, remains unproved.
the main advantage of FLAIR imaging is We found a monotonic decrease in
Discussion that it nulls the high signal intensity performance with decreasing lesion
In this study, we applied free-response within the cerebrospinal fluid spaces, size, which is an expected and reason-
methods to obtain prospective mea- one might postulate that the “perfor- able finding because smaller lesions
sures of observer performance in the mance gain” of FLAIR imaging is more should be more difficult to detect. We
detection of simulated MS lesions. We accentuated supratentorially, where the deliberately chose this size range,
found that, supratentorially, perfor-
mance was significantly better on FLAIR
Figure 2
images than on T2-weighted MR im-
ages. This finding is in agreement with
the findings of Herskovits et al (1) and is
also consistent with the results of older
studies (3,4,9–13). In many of these
older studies, researchers compared mo-
dalities by calculating the number or vol-
ume of lesions, as determined by the con-
sensus opinion of radiologists (3,4,9–11)
or by using semiautomated lesion-detec-
tion software (12,13). While such meth-
ods can provide some basis for compari-
son, they do not approximate the usual
conditions of image interpretation, and
they cannot account for interreader vari-
ability. By using a multiple-reader multi-
ple-case design, we provide better objec- Figure 2: AFROC curves for supratentorial (a) T2-weighted and (b) FLAIR images. Individual AFROC
tive evidence that FLAIR images allow a curves for the four observers (R1, R2, R3, and R4) are plotted as dashed lines; the pooled AFROC curve is
meaningful improvement in performance plotted as a solid line. In a, overall performance in lesion detection is good. Compared with performance in
compared with T2-weighted MR images a, performance in b is improved. The y-axis shows the true-positive fraction of lesions, and the x-axis shows
at the supratentorial level. false-positive fraction of images.
Performance was equivalent and
moderate for the two modalities at the
infratentorial level, corroborating the
belief that FLAIR imaging performs Figure 3
worse in the posterior fossa (3,4). Vari-
ous explanations for this phenomenon
include the additional artifacts intro-
duced by the FLAIR sequence—for ex-
ample, cerebrospinal fluid flow artifacts
(14) and/or possible differences in the
intrinsic MR characteristics of lesions.
Indeed, Stevenson et al (15) reported
differences in the T1 and T2 parameters
of infratentorial MS lesions that may at
least partially explain their decreased
visibility on FLAIR images. Interest-
ingly, our results demonstrated that
FLAIR imaging performed worse infra-
tentorially, even though we did not
model these additional artifacts or con-
sider any differences in underlying le- Figure 3: AFROC curves for infratentorial (a) T2-weighted and (b) FLAIR images. Individual AFROC curves
for the four observers (R1, R2, R3, and R4) are plotted as dashed lines; the pooled AFROC curve is plotted as a
sion characteristics. Therefore, an addi-
solid line. In a, overall performance in lesion detection is good. Performance is equivalent in a and b. The
tional factor must be responsible for
y-axis shows the true-positive fraction of lesions, and the x-axis shows false-positive fraction of images.
this performance decrease.

210 Radiology: Volume 241: Number 1—October 2006

NEURORADIOLOGY: Detection of Simulated Multiple Sclerosis Lesions
knowing full well that MS lesions are
usually larger, so that our measures
likely underestimate the true perfor-
mance of radiologists in clinical prac-
tice. However, by using small sizes, we
assessed performance with the most
difficult of lesions to detect, thereby
heightening any differences between
the variables that we tested, such as
modality, location, or transverse level.
Had we included lesions of larger size,
the differences would have been harder
to detect because all performance mea-
sures would have been high.
We found that observer perfor-
mance varied with lesion location, with
improved performance on FLAIR im-
ages in the detection of cortical lesions,
which corroborates the results of Her-
skovits et al (1). Conceptually, this find-
ing makes sense because FLAIR imaging
nulls the cerebrospinal fluid signal in-
tensity that might obscure these lesions
on T2-weighted MR images. The corti-
cal and subcortical lesion burden has
been found to correlate with regional at-
rophy (10) and with cognitive impairment
(9,16), thereby highlighting the impor-
tance of lesion detection in this location.
The number of cases used in this
study represented a compromise be-
tween too many images, which would
lengthen the study time and perhaps in-
troduce the confounding element of fa-
tigue, and too few images, which might
reduce the power of the study. In a pre-
Figure 4
Figure 4: Observer performance in the detec-
tion of MS lesions is significantly better on FLAIR
images (white bars) than T2-weighted MR images
(black bars) for gray matter (GM) lesions. Perfor-
mance is similar for deep white matter (DWM) and
periventricular (PVWM) lesions. Error bars ⫽
standard error.
Radiology: Volume 241: Number 1—October 2006
vious study, Obuchowski (17) estimated
the number of cases that would be
needed for a receiver operating charac-
teristic study to have adequate power in
demonstrating a significant difference
between diagnostic modalities. For stud-
ies with four observers, a high (approxi-
mately 0.90) accuracy of techniques, a
small (approximately 0.05) difference be-
tween modalities, a 1:1 ratio of normal to
abnormal cases, and a small interreader
variability, Obuchowski estimated that a
total of 287 cases would be needed to
detect a difference with 80% power. Our
study had a smaller ratio (1:3) of normal
to abnormal cases and was a free-re-
sponse design rather than a receiver op-
erating characteristic design, but we be-
lieve that 320 cases should supply ade-
quate power to detect any relevant
differences.
Our study has several limitations.
We used only a single template at either
transverse level to generate the test
cases. Therefore, we did not account
for interindividual variability in the
brain or for section to section variabil-
ity. Moreover, having only a single tem-
plate may have falsely elevated our per-
formance measures because of recall ef-
fect. Although the images were shown
in an alternating sequence to reduce
this effect, almost certainly the readers,
as trained neuroradiologists, would be
able to remember previous images as
they interpreted subsequent images.
False-positive responses would be re-
Figure 5
Figure 5: Observer performance in the detec-
tion of MS lesions on FLAIR and T2-weighted MR
images is shown as a function of lesion size. Per-
formance increases with increasing maximum
lesion size from 2 to 6 pixels (r ⫽ 0.5; P ⫽ .04).
Error bars ⫽ standard error.
Woo et al
duced, because readers may not score a
lesion that they might otherwise have
questioned simply because they remem-
ber it as an artifact from a previous
image. This bias, however, would have
affected all measurements equally so
that comparisons of performance may
still be valid.
Our study is also limited by our use
of a single MS lesion. Similar to Melhem
et al (2), we used the MR characteristics
of an actual MS lesion as a model for the
simulated lesions. This method con-
trasts with the work of Herskovits et al
(1), wherein the model lesion was artifi-
cially constructed by using an octagonal
shape and MR parameters that are typ-
ically reported in MS plaques. However,
MS lesions in general have a widely vari-
able appearance on MR images, which
likely reflects the heterogeneous pro-
cesses, such as demyelination, gliosis,
axonal loss, remyelination, inflamma-
tion, edema, or some combination
thereof, that are involved at the his-
topathologic level. Because we did not
account for interlesion variability, it
may be difficult to extrapolate our re-
sults to estimate the true observer per-
formance in the detection of all MS le-
sions. Moreover, the validity of our
method to generate gray matter lesions
may be questioned because our model
lesion was extracted from white matter.
However, removing interlesion variabil-
ity should reduce our resulting vari-
ances, perhaps increasing the power of
our results in demonstrating significant
differences in performance.
Besides the limitations related to
image generation, others arise from our
observer testing methods. We believe
that 30 seconds allows readers ade-
quate time to evaluate each image, but
this allotment may not accurately reflect
actual reading conditions in which sin-
gle images are usually evaluated only for
a few seconds, perhaps longer if abnor-
malities are detected. We did allow
readers to spend less time if they were
satisfied that all lesions were found.
Still, it is unclear what effect this would
have had on observer performance. On
the one hand, by looking at an image
longer, readers might find more true
lesions than they might have otherwise
211
NEURORADIOLOGY: Detection of Simulated Multiple Sclerosis Lesions
found, thereby improving performance.
On the other hand, they might also find
more false-positive lesions, thereby reduc-
ing performance. Again, we expect that
such biases would affect performance mea-
sures equally so that differences in perfor-
mance may still be applicable.
Finally, the accuracy of our testing
methods is limited because of other fun-
damental differences with standard im-
age interpretation. Our tests used single
images, whereas normally readers can
assess contiguous images to help decide
whether the findings represent actual
lesions. Moreover, modern picture ar-
chiving and communication systems al-
low readers to adjust the window set-
tings to improve lesion visibility; our
software did not have this feature.
Again, because these limitations can be
expected to affect performance mea-
sures equally, differences may still be
detected and valid.
In summary, performance on FLAIR
images is excellent and is better than
that on T2-weighted MR images su-
pratentorially. Infratentorially, perfor-
mance is moderate and is similar be-
tween the two modalities. Observers
performed better on FLAIR images in
the detection of cortical lesions, and
performance improved with increasing
lesion size. We believe that, as a first
approximation, our methods provide a
useful approach to measure observer
performance in the task of MS lesion
detection on MR images of the brain. In
the future, one might imagine improving
the method, perhaps with multiple brain
templates, multiple digitized lesions, and
testing software that more closely approxi-
mated actual viewing conditions.
Acknowledgments: We thank Edward H. Her-
skovits, MD, PhD, for his original work in perfor-
212
mance measurements by using an MR simulator
and Abbas Jawad, PhD, for his help in statistical
analysis.
References
1. Herskovits EH, Itoh R, Melhem ER. Accu-
racy for detection of simulated lesions: com-
parison of fluid-attenuated inversion-re-
covery, proton density–weighted, and T2-
weighted synthetic brain MR imaging. AJR
Am J Roentgenol 2001;176:1313–1318.
2. Melhem ER, Herskovits EH, Karli-Oguz K, et
al. Defining thresholds for changes in size of
simulated T2-hyperintense brain lesions on
the basis of qualitative comparisons. AJR
Am J Roentgenol 2003;180:65– 69.
3. Gawne-Cain ML, O’Riordan JI, Thompson
AJ, Moseley IF, Miller DH. Multiple sclerosis
lesion detection in the brain: a comparison of
fast fluid-attenuated inversion recovery and
conventional T2-weighted dual spin echo.
Neurology 1997;49:364 –370.
4. Filippi M, Yousry T, Baratti C, et al. Quanti-
tative assessment of MRI lesion load in mul-
tiple sclerosis. IV. A comparison of conven-
tional spin-echo with fast fluid-attenuated in-
version recovery. Brain 1996;119:1349 –
1355.
5. Droege RT, Wiener SN, Rzeszotarski MS,
Holland GN, Young IR. Nuclear magnetic
resonance: a gray scale model for head im-
ages. Radiology 1983;148:763–771.
6. Chakraborty DP, Winter LH. Free-response
methodology: alternate analysis and a new
observer-performance experiment. Radiol-
ogy 1990;174:873– 881.
7. Hanley JA. Receiver operating characteristic
(ROC) methodology: the state of the art. Crit
Rev Diagn Imaging 1989;29:307–335.
8. Chakraborty DP, Berbaum KS. Observer
studies involving detection and localization:
modeling, analysis, and validation. Med Phys
2004;31:2313–2330.
9. Moriarty DM, Blackshaw AJ, Talbot PR, et
al. Memory dysfunction in multiple sclerosis
corresponds to juxtacortical lesion load on
fast fluid-attenuated inversion-recovery MR
Woo et al
images. AJNR Am J Neuroradiol 1999;20:
1956 –1962.
10. Bakshi R, Ariyaratana S, Benedict RH, Ja-
cobs L. Fluid-attenuated inversion recovery
magnetic resonance imaging detects cortical
and juxtacortical multiple sclerosis lesions.
Arch Neurol 2001;58:742–748.
11. Yousry TA, Filippi M, Becker C, Horsfield
MA, Voltz R. Comparison of MR pulse se-
quences in the detection of multiple sclerosis
lesions. AJNR Am J Neuroradiol 1997;18:
959 –963.
12. Tubridy N, Barker GJ, MacManus DG,
Moseley IF, Miller DH. Three-dimensional
fast fluid attenuated inversion recovery (3D
fast FLAIR): a new MRI sequence which in-
creases the detectable cerebral lesion load in
multiple sclerosis. Br J Radiol 1998;71:840 –
845.
13. Bastianello S, Bozzao A, Paolillo A, et al. Fast
spin-echo and fast fluid-attenuated inver-
sion-recovery versus conventional spin-echo
sequences for MR quantification of multiple
sclerosis lesions. AJNR Am J Neuroradiol
1997;18:699 –704.
14. Arakia Y, Ashikaga R, Fujii K, Nishimura Y,
Ueda J, Fujita N. MR fluid-attenuated inver-
sion recovery imaging as routine brain T2-
weighted imaging. Eur J Radiol 1999;32:
136 –143.
15. Stevenson VL, Parker GJ, Barker GJ, et al.
Variations in T1 and T2 relaxation times of
normal appearing white matter and lesions
in multiple sclerosis. J Neurol Sci 2000;178:
81– 87.
16. Rovaris M, Filippi M, Minicucci L, et al. Cor-
tical/subcortical disease burden and cogni-
tive impairment in patients with multiple
sclerosis. AJNR Am J Neuroradiol 2000;21:
402– 408.
17. Obuchowski NA. Sample size tables for re-
ceiver operating characteristic studies. AJR
Am J Roentgenol 2000;175:603– 608.
Radiology: Volume 241: Number 1—October 2006