Radiation Protection Dosimetry (2005), Vol. 114, Nos 1-3, pp.

201–207
doi:10.1093/rpd/nch554

EVALUATION AND TESTING OF COMPUTED RADIOGRAPHY

SYSTEMS
P. Charnock
, P. A. Connolly, D. Hughes and B. M. Moores
IRS Ltd, Unit 188 Century Building, Brunswick Business Park, Liverpool, UK

The implementation of film replacement digital radiographic imaging systems throughout Europe is now gathering
momentum. Such systems create the foundations for totally digital departments of radiology, since radiographic examinations
constitute the most prevalent modality. Although this type of development will lead to improvements in the delivery and
management of radiological service, such widespread implementation of new technology must be carefully monitored. The
implementation of effective QA tests on installation, at periodic intervals and as part of a routine programme will aid this
process. This paper presents the results of commissioning tests undertaken on a number of computed radiography imaging
systems provided by different manufacturers. The aim of these tests was not only to provide baseline performance measure-
ments against which subsequent measurements can be compared but also to explore any differences in performance,
which might exist between different units. Results of measurements will be presented for (1) monitor and laser printer set-
up; (2) imaging plates, including sensitivity, consistency and uniformity; (3) resolution and contrast detectability; and (4)
signal and noise performance. Results from the latter are analysed in relationship with both system and quantum noise
components.

INTRODUCTION there is still a need to be able to evaluate their per-

formance in the real world of the X-ray department.
In recent years conventional screen-film radiography
Such tests should be suitable for both acceptance
is increasingly being replaced by digital radiographic
testing and routine quality control.
systems. The possible impact of this development on
Methods for the routine evaluation of CR sys-
radiological practice and the different types of radio-
tems, which involve contrast-detail measurements,
graphic transducers has been discussed(1), and the
have been described(11). These measurements
key physical principles are described previously(2).
indicate that the CR systems is well below the
Ideally, the aim of the new digital imaging systems
ideal performance, perhaps owing to a variety of
is to improve the quality of diagnostic information
physical and psychological signal-to-noise ratio
while at least maintaining existing levels of patient
(SNR) degradation processes. However, threshold
doses. The introduction of digital radiographic sys-
contrast detectability measurements did correlate
tems will certainly enable the radiological working
with the measurements of detective quantum effici-
environment to be modernised. Film-based techno-
ency (DQE). This work did highlight the need
logy and its associated processing and viewing
to ensure that the image display conditions are
requirements are now 100 years old. Consequently
fully considered when performing this type of
improved image handling and image processing are
measurement.
the immediate benefits, but a number of exciting
Methods for measuring the SNR of CR systems
possibilities have been created. For example, the
have also been described where the ratio of the mean
development and use of computer-aided diagnostic
pixel value (MPV), in a defined region of interest
(CAD) technique are already in progress. However,
(ROI), to the standard deviation is employed(12,13).
new forms of imaging data have now become a
By logarithmically transforming the numbers for
practical reality(3,4).
mean and standard deviation calculated from the
Given the potential impact that digital detectors
raw data into the mean air kerma and standard
will have on the future of diagnostic radiology it is
deviation of air kerma to the imaging plate, the
important that effective methods for the evaluation
performance of different systems can be compared
of these systems are developed. Such methods may
directly. This can be undertaken from a knowledge
be undertaken as either laboratory or field-based
of the relationship between the sensitivity value and
methods. Much effort has been made over the past
the logarithm of air kerma.
10–15 y in developing physical methods to evaluate
More recently, a protocol for the QA of CR
digital radiographic systems(5–10). However, while
systems—commissioning and annual QA tests
detailed physical measurements provide extremely
has been produced by the King’s Centre for the
useful information, as well as understanding the
Assessment of Radiological Equipment (KCARE)
physical basis of new digital radiographic systems,
in the United Kingdom(14). This protocol has been
employed in the evaluation of three manufacturers’


Corresponding author: PaulCharnock@irs-limited.com systems at different hospital sites in order to evaluate

ª The Author 2005. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oupjournals.org
 P. CHARNOCK ET AL.
both the protocol and the CR performance. Meas-
urements of SNR on these systems are also included
in the assessment.
Test procedures—general considerations
The test protocol involves the assessment of the
following when commissioning a system:
 monitor and laser printer set-up,
 dark noise,
 erasure cycle efficiency,
 sensitivity index consistency,
 uniformity,
 scaling errors,
 blurring,
 limiting spatial resolution,
 threshold contrast detectability,
 laser beam function,
 Moiré patterns.
The details of the test methods are described in
the KCARE protocol. Once commissioning meas-
urements have been undertaken it is recommended
that the following annual QA tests are performed:
 monitor and laser printer set-up,
 sensitivity index consistency,
 uniformity,
 blurring and stitching artefacts,
 limiting spatial resolution (45 only),
 threshold contrast detectability.
The remedial levels are recommended in Table 1.
The commissioning tests have been performed on
a number of CR units representing three manu-
facturers, Agfa Gevaert, Kodak and Philips (Fuji).
A short paragraph on the different plate systems
is included in the Appendix section. Details of
only those tests relating to sensitivity index and
threshold contrast detectability are presented here.
Also included are some measurements undertaken
with respect to signal-to-noise. Before presenting
these results it is worth noting that it is not always
possible to access particular information at the
local level. It is also noted that measurements
were only performed on the CR systems. At present,
only limited tests have been performed on direct
Table 1. The recommended remedial levels for the annual
QA test.
Test Remedial level
Sensitivity index consistency Baseline 20%
Limiting resolution Baseline 20%
Threshold contrast detectability Baseline 30%
202
digital radiography (DDR) and so no analysis is
presented here.
Monitor and laser printer set-up
This involves the use of the SMPTE test pattern. On
one unit the pattern generator was not included in
the system. On another unit normal viewing condi-
tions on screen were not full size centred. It was
80% size and displaced 2 cm to the left.
Sensitivity index calibration
The protocol requires that tube factors are selected
to provide an air kerma of 10 mGy to the plate. On
one particular system the output was very variable
for a slight adjustment in the set-up. Eventually an
air kerma of 12 mGy was produced. Although the
KCARE protocol recommends that no filtration is
employed using a Fuji system, a protocol obtained
from Philips, which appears to come from the Insti-
tute of Physics and Engineering in Medicine (IPEM)
protocol that recommends 1.5 mm Al.
Sensitivity index consistency
For high-resolution plate the sensitivity was
2.5 times higher for one system at least, when com-
pared with normal-resolution plates. Also on another
system one plate recorded a sensitivity equivalent to
an unexposed plate even though it had been exposed.
Apparently this may occur if there is a small artefact
on the plate or the phosphor is not quite lined up.
Limiting spatial resolution
Problems were encountered in determining the pixel
dimensions for a CR system. Local staff did not
understand what this meant in relationship with the
CR system. When performing tests on a DDR sys-
tem, the pixel size was given on an image printout.
Pixel dimensions for other CR systems were not
clearly understood and a proposal was made by the
support staff to divide the image matrix by 3.2 mega
pixels. Clarification is required on a number of issues
if field testing of the CR / DDR systems is to be
consistent.
RESULTS AND DISCUSSION
Sensitivity index
The sensitivity index was measured for different val-
ues of entrance air kerma to the imaging plate for all
three manufacturers’ systems. The results are presen-
ted in Figure 1 where the calculated and measured
values are shown. Note that all the manufacturers
give the sensitivity a different label. Agfa calls it
‘lgM’, Kodak calls it ‘EI’ or exposure index and
 EVALUATION AND TESTING OF CR SYSTEMS
4.00

3.50

3.00
Log [Sensitivity]

2.50

2.00

1.50

1.00
0.0 0.5 1.0 1.5 2.0 2.5
Log [Air Kerma (µGy)]
Kodak - Measured Kodak - Calculated
Philips - Measured Philips - Calculated
Agfa - Measured Agfa - Calculated

Figure 1. Log sensitivity vs. log air kerma.

Fuji simply calls it ‘S’. The functional forms relating
exposure to sensitivity are:
Agfa exposure ¼ 9.33  10lgM  3.2766
Kodak exposure ¼ 8.7  10n,
where n ¼ (EI  2000)/1000
Fuji (Philips) exposure ¼ 1740/S
The curves shown in Figure 1 demonstrate the wide
variations in sensitivity as a function of exposure
among the different manufacturers. Not only are
the gradients different but also the functional form.
The curve for the Kodak system roles off markedly
for very low exposures.
Previous measurements performed on Agfa sys-
tems have indicated a square root dependence
between log (sensitivity) and log (air kerma)(15),
whereas Figure 1 shows an almost direct proportion-
ality. The slope of the Fuji (Philips) system shown in
Figure 1 was the negative reflection of the Agfa
system. It must be noted that the sensitivity index
‘S’ given for the Fuji (Philips) system probably does
not represent the actual pixel value employed. Fur-
ther work required to show what it does represent is
in progress.
The results presented in Figure 1 were nominally
obtained for unprocessed data. However, once the
CR plate has been read, the system assigns an MPV
to the histogram of pixel values obtained in an
image. This ensures that all relevant image data are
optimally displayed. Figure 2 shows the log(MPV)
plotted against log(incident air kerma).
203
For the Kodak system the MPV and sensitivity
values are identical. However, for the Agfa and
Fuji (Philips) system they are somewhat different.
The MPV data for the Agfa system now does show
a square root dependence on air kerma. The Fuji
(Philips) system shows an almost constant MPV
for air kerma values >10 mGy.
Signal-to-noise
In order to explore the SNR performance of the
CR systems the average pixel value in a given ROI
as well as the standard deviation was meas-
ured(12,13), where SNR equals the MPV divided
by the standard deviation. These measurements
could not be performed on the Agfa system since
the image display facility would not provide a
measure of standard deviation for an ROI. This
again highlights the variations in system perform-
ance between manufacturers. The results for the
Kodak and Fuji (Philips) systems are shown in
Figure 3. Obviously, it is not possible to compare
the two systems directly when SNR has been
expressed in this manner since both systems have
completely different exposure characteristics, owing
to the fact that there is no common relationship
between the pixel value and the exposure. How-
ever, it should be possible to compare systems if
signal and noise values are expressed in terms of
plate exposure(12).
 P. CHARNOCK ET AL.
3.6
Kodak
Philips
Agfa
3.4

3.2
Log [MPV]

3.0

2.8

2.6

2.4
0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9 2.1
Log [Air Kerma (µGy)]

Figure 2. Log (MPV for 0.5 cm2 ROI) vs. log (air kerma).

650
Kodak
Philips
550

450
SnR

350

250

150

50
0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0
Log [Air Kerma (µGy)]

Figure 3. SNR of 0.5 cm2 ROI vs. log (air kerma).

Figure 4 shows the SNR, expressed in exposure
terms, for the Kodak system. Unfortunately,
given the exposure characteristics expressed in
terms of sensitivity it was not possible to make a
sensible appraisal of SNR for the Fuji (Philips)
system. In fact values calculated for this system
using the information shown in Figure 1 gave
a decreasing SNR value with exposure. As indic-
ated previously the sensitivity value ‘S’ given
for the Fuji (Philips) system probably does not
correspond to the actual pixel value. The varia-
tion in MPV with exposure shown in Figure 2

204
 EVALUATION AND TESTING OF CR SYSTEMS
for the Fuji (Philips) system does not clarify the Threshold contrast detectability
situation.
Threshold contrast detectability measurements
The value of SNR vs. exposure presented in
were undertaken on the three CR systems, Agfa,
Figure 4 for the Kodak system is in reasonable
Kodak and Fuji (Philips), using the Leeds FAXIL
agreement with the values calculated by this method
test phantom TO20(16). This consists of 144 circu-
previously(12). The value of SNR expressed in this
lar details with 12 sizes and 12 contrasts. Detail
form corresponds to the number of independent
sizes range from 11 to 0.25 mm diameter and
noise levels in the dynamic (average signal) range of
contrast range from 0.97 to 49.7% for an 80 kV
the system at the appropriate exposure level. Such a
beam filtered by 1 mm Cu. Measurements were
measure should be related with the threshold
made when employing this phantom viewed under
contrast for the appropriate detail size and exposure
the normal local viewing conditions. No effort
conditions.

90

80

70
Air Kerma / Delta Air Kerma

60

50

40

30

20

10

0
0.0 0.5 1.0 1.5 2.0 2.5
Log [Air Kerma (µGy)]

Figure 4. Exposure corrected SNR for a Kodak system.

1
3.9 µGy
12.7 µGy
Contrast (%)

0.1

0.01
0.1 1 10 100
Diameter (mm)

Figure 5. Threshold contrast detectability for a Kodak system.

205
 P. CHARNOCK ET AL.
was made to optimise the conditions since we were visual detection the threshold signal is give by:
interested in performing true field type measure-
ments.
S ¼ k  noise, ð1Þ
Viewing conditions varied from a small darkened where k is the threshold SNR.
room to a larger room with multiple VDUs (Agfa). Threshold contrast for a given detail size is given
No attempt was made to process or contrast enhance by:
the image.

S noise
Contrast-detail diagrams for each of the systems Cmin ¼ ¼k , ð2Þ
are shown in Figures 5–7. The Agfa and Fuji s average signal
(Philips) systems showed similar contrast-detail where s is the average signal. The value of noise/
behaviour across all detail sizes and exposure condi- average signal is the reciprocal of the SNR values
tions in Figures 6 and 7, respectively. The Kodak shown in Figure 4. For the Kodak system, the
system showed most exposure dependence and lower 0.5 cm2 ROI gave an SNR value at 3.9 mGy of 44.
threshold detectability for smaller detail sizes. For Assuming a threshold SNR value of 2 for circular

4.2 µGy
10.9 µGy
18.4 µGy
Contrast (%)

0.1

0.01
0.1 1 10 100
Diameter (mm)

Figure 6. Threshold contrast detectability for an Agfa system.

2.9 µGy
4.9 µGy
12.14 µGy
Contrast (%)

0.1

0.01
0.1 1 10 100
Diameter (mm)

Figure 7. Threshold contrast detectability for a Philips system.

206
 EVALUATION AND TESTING OF CR SYSTEMS
detail, the calculated threshold contrast value for a
0.5 cm2 ROI on the Kodak system is 0.05 (5%). The
calculated value is roughly twice that observed in
Figure 5 (2.3%) and perhaps indicates that with
more careful field measurements simple SNR meas-
urements might usefully underpin threshold
contrast-detail measurements.
SUMMARY AND CONCLUSIONS
Work has been undertaken in order to evaluate the
KCARE test protocol for the CR and DDR imaging
systems. Results indicate that the protocol is
extremely useful for those wishing to gain a more
detailed knowledge of the performance of these sys-
tems. However, our experience has shown that sup-
port and knowledge on these new imaging systems is
extremely varied. Information which is important
for the assessment of performance is not available.
Equally, information that is available is not easily
understood in terms of its relevance to the assess-
ment of imaging performance. Preliminary measure-
ments of SNR indicate that, given the right technical
information, the simple SNR assessment methods
could provide useful information on imaging per-
formance and help to underpin threshold contrast-
detail measurements.
ACKNOWLEDGEMENT
This work was undertaken with support provided by
the Community of European Communities under
grant no. FIGM-CT-2000-0036.
REFERENCES
1. Kamm, K. F. The future of digital imaging. Br. J.
Radiol. 70(Special issue), S145–S152 (1997).
2. Rowlands, J. A. The physics of computed radiology.
Phys. Med. Biol. 47(23), R123–R166 (2002).
3. Doi, K. Present status and future horizons for computer
aided diagnostic in radiology. In: Physics and Engineer-
ing in Medicine in the New Millennium. Sharp, P. F.
and Perkins, A. C., Eds. (York: IPEM) pp. 84–87 (2000).
4. Dobbins, J. T. and Godfrey, D. J. Digital X-ray tomo-
synthesis: current state of the art and clinical potential.
Phys. Med. Biol. 48(19), R65–R106 (2003).
5. Dobbins, J. T. Image quality metric for digital systems.
In: Handbook of Medical Imaging. Vol. 1. Physics
and Psychophysics. Beutel, J., Kundel, H. L. and Van
Meter, R. L., Eds. (Bellingham: SPIE) pp. 161–222
(2000).
6. Samei, E., Flynn, M. J. and Reimann, D. A. A method
for measuring the pre-sampled MTF of digital radio-
graphic systems using an edge test device. Med. Phys.
25, 102–113 (1998).
7. Workman, A., Cowen, A. R. and Brettle, D. S.
Physical evaluation of computed radiography as a
mammographic X-ray imaging systems. Br. J. Radiol.
67, 988–996 (1994).
207
8. Buhr, E., Gunther-Kohfahl, S. and Neitzel, U. Accur-
acy of a simple method for deriving the pre-sampled
modulation transfer function of a digital radiographic
system from an edge image. Med. Phys. 30(9), 2323–
2330 (2003).
9. Illers, H., Buhr, E., Benjamin, D. and Hoeschen, C.
Measurement of the detective quantum efficiency (DQE )
of digital x-ray imaging device. According to the Forth-
coming Standard IEC 6220-1. In: Proceedings of the
SPIE Conference on Medical Imaging, San Diego
(Bellingham: SPIE) (2004).
10. Illers, H., Buhr, E. and Vandenbracke, D. Measure-
ment of correlated noise in images of computed radio-
graphy systems and its influence on the detective
quantum efficiency. In: Proceedings of the SPIE Con-
ference on Medical Imaging, San Diego (Bellingham:
SPIE) (2004).
11. Cowen, A. R., Workman, A. and Price, J. S. Physical
aspects of photostimulable phosphor computed radio-
graphy. Br. J. Radiol. 66, 332–345 (1993).
12. Meier, N. and Fiebich, M. Standard tools for analysis
of image quality in digital imaging. Radiat. Prot.
Dosim. 57(1–4), 141–143 (1995).
13. Wagner, R. F. and Brown, D. G. Unified SNR analysis
of medical imaging systems. Phys. Med. Biol. 30, 489–
578 (1985).
14. KCARE. Protocol for the QA of computed radiography
systems—commissioning and annual QA tests. Draft
3.0, King’s College Hospital, London (2003).
15. Ter-Pogossian, M. M. The Physical Aspects of Dia-
gnostic Radiology (London: Harper and Row) (1967).
16. Cowen, A. R., Haywood, J. M., Workman, A. and
Clarke, O. F. A set of x-ray test objects for image
quality control in digital subtraction fluorography. I.
Design considerations. II. Application and interpretation
of results. Br. J. Radiol. 60, 1001–1009, 1010–1018
(1987).
17. KCARE, MDA. Computed Radiography (CR)
System. Agfa HealthCare ADC Solo/Compact. Evalu-
ation Report No. MDA02062 (2002).
18. KCARE, MDA. Computed Radiography (CR)
System. Fuji Medical Imaging FCR 5000. Evaluation
Report No. MDA02062 (MHRA) (2003).
APPENDIX
All three systems encountered are based on the fol-
lowing design; plates, cassettes, ID system, processor
and acquisition workstation(17–18). They all have
various plate/cassette sizes but this work only
encountered high resolution plates with the Kodak
system. Agfa plates can be read using a different
speed class, which refers to the acceptable exposure
range. Speed class ranges from 50 to 400, which
corresponds to a saturation exposure of 35 to
290 mGy respectively(17). The Philips (Fuji) system
encountered here appears to consist of a Fuji / Philips
collaboration. This may not be the case with all Fuji
sites. The plates themselves have a wide dynamic
range, which may account for a lack of high resolu-
tion plates(18). Kodak have both normal and high
resolution plates.