47

ORIGINAL PAPER

A likelihood ratio approach to meta-analysis of

diagnostic studies
D Stengel, K Bauwens, J Sehouli, A Ekkernkamp, F Porzsolt
...................................................................................................

J Med Screen 2003;10 :47–51

Objective: To develop a clinically and methodologically sound approach to diagnostic meta-analysis.

Methods: Two-step model was used involving four Žctitious sets of 10 studies each with varying
sensitivity and speciŽcity; this was followed by the application of the method to data from a published
systematic review of emergency ultrasound. Multidimensional test characteristics (relating to the
detection or exclusion of the condition of interest) were described by likelihood ratio scatterplots and
pooled likelihood ratios. Likelihood ratios summarise the ability of a test to revise the prior probability
of disease. They can be summarised by established Žxed-effects and random-effects methods.
See end of article for Results: Likelihood ratios precisely describe both directions of test performance. By plotting positive
authors’ afŽliations
................ against negative likelihood ratios, together with their 95% conŽdence intervals, a multidimensional
forest plot is obtained that can be interpreted in analogy to therapeutic meta-analyses. There are
Correspondence to:
accepted threshold values of positive and negative likelihood ratios (i.e. 10.0 and 0.1) to recommend
Dr D Stengel, Department
of Trauma Surgery, a test for clinical use. In the matrix space, distinct test characteristics can even be assessed by
Unfallkrankenhaus Berlin eyeballing. With regard to data from the real meta-analysis, the suggested high discriminatory power
Trauma Centre, Warener of ultrasound was only partially qualiŽed by likelihood ratios. The positive value conŽrms the reliability
Str 7, 12683 Berlin,
Germany; of a positive scan, whereas the negative value questions a normal sonogram.
dirk.stengel@ukb.de Conclusions: A full characterisation of test performance requires multidimensional effect measures.
Likelihood ratios are recommended descriptors of the two dimensions of diagnostic research evidence
Accepted for publication
11 December 2002
and provide a convenient means to visualise and to communicate results as weighted summary
................ estimates of a diagnostic meta-analysis.

S
ystematic reviews and meta-analyses have emerged as an
important branch of biomedical research. They are con-
sidered to provide the best scientiŽc evidence to support
or to reject the use of certain interventions and to acknowl-
edge prognostic, aetiological and risk factors of a particular
condition or disease. Whereas the terms “systematic review”
and “meta-analysis” are often used synonymously, the latter
should mainly be regarded as a statistical technique to quan-
tify and to summarise the information provided by a thorough
systematic review of published and unpublished studies.
The major goals of meta-analysis are to display the range
of effect sizes of individual investigations and to compile
these data into some measure of overall efŽ cacy that can be
uniformly interpreted and communicated (which means
that audiences with varying background knowledge are able
to gain similar information on the topic of interest). These
basic requirements have been met by the established
methods of meta-analysis of therapeutic studies.
Results are usually presented graphically by means of a forest
plot, which gives an excellent impression of the dispersion of
effect sizes. The common point estimates, such as odds ratios
(OR), relative risks (RR) and risk differences (RD), in combi-
nation with their related 95% or 99% conŽ dence intervals,
allow clear inferences to be drawn from the available data: the
experimental arm fares better, fares equivocally (there is no
evidence of a difference) or fares worse than the control arm.
The problem that occurs in a meta-analysis of diagnostic
studies is the multidirectional performance of the diagnostic
instrument regarding its ability to detect (speciŽcity) or to
exclude (sensitivity) the characteristic of interest. Multi-
dimensional outcomes cannot be summarised well by a
single estimate. The results gained from a diagnostic test will
increase or decrease the probability of there being disease in
the presence of certain signs, symptoms and other test
results; this is often referred to as Bayes’ theorem.1
In diagnostics, the ability of a test to revise the prior
probability of disease is summarised by the likelihood ratio.
The likelihood ratio is the likelihood that a given test result
will be observed in a patient with the target disorder
compared with the likelihood that the same result will be
observed in a patient without the target disorder.2 The positive
likelihood ratio is the ratio between the chance of a positive
test result in the presence of the characteristic under investi-
gation and the chance of a positive result in the absence of
this attribute. For example, a positive likelihood ratio of 4.0
means that a positive test result is four times more likely in a
diseased subject than in a healthy person. Likewise, the
negative likelihood ratio represents the ratio between the
chance of a negative test result in the absence of the charac-
teristic under investigation and the chance of a negative
result in the presence of this attribute. Given the positive and
the negative likelihood ratio of a test procedure and a certain
prevalence, or prior disease probability, the related prob-
ability of disease can easily be obtained from nomograms.2,3
By convention, marked changes in prior disease probability
can be assumed in positive likelihood ratios exceeding 10.0
and negative likelihood ratios below 0.1.4,5 The scientiŽc back-
ground for these somewhat arbitrary thresholds can be found
in Bayesian decision theory: “classical” signiŽcance levels
often correspond to still substantial posterior probabilities of
the null hypothesis of “no change in prior belief”. For
instance, a p value of 0.05 (which would normally support
rejection of the assumption of a null effect) can still corre-
spond to a 52% posterior probability of the null hypothesis.6

www.jmedscreen.com Journal of Medical Screening 2003 Volume 10 Number 1

48 Stengel, Bauwens, Sehouli, et al.

For the purpose of this study we stress only that an investi-
gation of a certain medical intervention must provide a large
amount of additional information to change prior knowl-
edge. Since no threshold values of sensitivity or speciŽcity are
available that would allow either the adoption or the
rejection of the routine application of a diagnostic procedure,
likelihood ratios appear as preferable indices of test perform-
ance, at least in the setting of clinical decision-making.
In this investigation we were interested in how likelihood
ratios can be used to conduct a diagnostic meta-analysis. Our
objectives were to develop a clearly arranged graphical
presentation of the results from individual diagnostic
studies, and to obtain a summary measure of diagnostic test
efŽ cacy by reasonable computational efforts.
METHODS
Basic considerations
The results of a diagnostic test with a dichotomous outcome
can be easily summarised in a 2 ´ 2 table. Sensitivity (SN, the
number of true positives divided by all test positives) and
speciŽcity (SP, the number of true negatives divided by all
test negatives) are well known characteristics of a diagnostic
intervention. Knowledge of both indices is required to
appraise test precision fully. However, one might think of
clinical situations in which only one of these characteristics
is of real interest. For example, the Ottawa ankle rules
provide 99% SN to exclude ankle fractures, whereas a 99%
SP of transcranial Doppler sonography allows detection of
vasospasm in the middle cerebral artery.7,8
Many attempts have been made to calculate a single
summary statistic from a set of 2 ´ 2 tables; this statistic is
often referred to as the “diagnostic odds ratio”. In fact, the
application of a diagnostic test always yields two odds ratios:
the prior odds will be increased with positive test Žndings
(leading to an odds ratio ranging from 1 to inŽ nity) and will
be decreased if the test turns out negative (with an odds ratio
between 0 and 1).
Multidimensional forest plots
The efŽ cacy of a diagnostic test is precisely characterised by
its positive and its negative likelihood ratio in both quali-
tative (the vector of test performance) and quantitative (the
size of this vector) terms.
A test set of four different meta-analytic scenarios was
constructed, comprising 10 Ž ctitious studies of identical size
(n = 2485). The studies included in these meta-analyses
showed the possible combinations of test characteristics: a
test of high SN and SP; a test of low SN and high SP; a test of
high SN and low SP; and a test of low SN and SP.
The individual positive likelihood ratio (LR+) can be
expressed as:
LR+ = SN/(1 - SP)
The negative likelihood ratio is calculated as:
LR- = (1 - SN)/SP.
Positive and negative likelihood ratios were then arranged
in a “multidimensional forest plot” together with their 95%
conŽ dence intervals. We hypothesised that this method of
graphical presentation could be easily interpreted, especially
by readers already used to the “classic” forest plots of
therapeutic meta-analyses. The discriminating features of a
test will be reected by the localisation of the data cloud
within the quadrants of the matrix space, and this in turn
will allow a rapid assessment of the potential usefulness of a
diagnostic procedure, even by eyeballing.
The basic steps to obtain a summary measure in a meta-
analysis are to weight individual effect measures according
to their variance and then to divide the sum of the weighted
quantities by the sum of weights.
We adopted general inverse-variance weighted Ž xed-
effects models to summarise likelihood ratios.9–13 In case of
statistical heterogeneity, the DerSimonian–Laird random-
effects method was applied to account for both within-study
and between-study variance.
In analogy to therapeutic meta-analyses, summary esti-
mates within the matrix space were given prominence over
individual results by the use of Ž lled symbols.
Worked example
Focused abdominal sonography for trauma (FAST) is an
emergency ultrasound protocol to screen for free intra-
abdominal  uid after blunt injuries. In general, the exami-
nation is performed in three or four standard planes to detect
 uid collections in the perihepatic space (Morrison’s pouch),
surrounding the spleen, and in the Douglas’ spatium.
Approximately 20% of all intra-abdominal traumatic lacera-
tions are not accompanied by signiŽcant haemoperitoneum.
In comparisons with other reference standards, such as diag-
nostic peritoneal lavage (DPL) or helical computed tomogra-
phy (HCT), the ability of FAST examination to detect or to
exclude the underlying visceral lesion remains unclear.
In the meta-analysis, 11 trials were eligible for analysis,
with a total of 2819 subjects – they employed ultrasono-
graphy to detect organ injury and made comparisons with
either DPL or HCT. 14 It was obvious that ultrasound provided
extremely high speciŽcity (range 0.84–1.00) but sensitivity
was below 90% in nine of the 11 trials. Table 1 summarises

Table 1 Major characteristics and descriptive statistics of studies included in the meta-analysis of emergency ultrasound
(FAST) 14
Prior
Reference n TP FN TN FP probability Sensitivity (95% CI) SpeciŽcity (95% CI)

Akgür F 208 42 8 157 1 0.24 0.84 (0.79, 0.89) 0.99 (0.98, 1.00)
Froelich JW 26 13 2 10 1 0.58 0.87 (0.74, 1.00) 0.91 (0.80, 1.00)
Förster R 140 28 1 107 4 0.21 0.97 (0.94, 1.00) 0.96 (0.93, 0.99)
Goletti O 73 26 4 42 1 0.41 0.87 (0.79, 0.94) 0.98 (0.94, 1.00)
Healey MA 796 45 6 728 17 0.06 0.88 (0.86, 0.90) 0.98 (0.97, 0.99)
Katz S 121 10 1 92 18 0.09 0.91 (0.86, 0.96) 0.84 (0.77, 0.90)
Krupnick AS 64 20 12 32 0* 0.52 0.62 (0.50, 0.74) 0.98 (0.95, 1.00)
McGahan JP 121 24 14 79 4 0.31 0.63 (0.55, 0.72) 0.95 (0.91, 0.99)
McKenney KL 884 95 15 768 6 0.12 0.86 (0.84, 0.89) 0.99 (0.99, 1.00)
Röthlin MA 313 24 31 257 1 0.18 0.44 (0.38, 0.49) 1.00 (0.99, 1.00)
Singh G 73 26 9 33 5 0.48 0.74 (0.64, 0.84) 0.87 (0.79, 0.95)

* In case of zero values, all cells were corrected for continuity by adding 0.5.
The number of subjects enrolled into the trials is denoted by n. TP, FN, TN and FP are the number of true-positive, false-negative, true-negative, and false-positive
Žndings, respectively. Values in parentheses are 95% conŽdence intervals.

Journal of Medical Screening 2003 Volume 10 Number 1 www.jmedscreen.com

Meta-analysis of diagnostic studies
all the relevant descriptive statistics of the studies included in
the Ž nal model.
RESULTS
Fictitious studies
Figure 1 shows the distribution of the Ž ctitious data on a
scatterplot diagram. The matrix presentation enables a quick
visual impression of the strengths and the weaknesses of a
diagnostic test in either direction. Tests with excellent
discriminatory properties will be plotted to the upper right
corner of the matrix space (i.e. exceeding the 10.0 and 0.1
threshold levels), whereas almost useless tests can be found
in the lower left quadrant (in other words, in the south-
western edge of the diagram).
With regard to the Ž ctitious sample of trials with high SN
and poor SP, their expression as likelihood ratios indicates
that the majority of studies support the validity of negative
test results, whereas uncertainty reigns for positive Ž ndings.
In this example, high SN and poor SP translates to a
positive summary likelihood ratio of 1.44 (which means
virtually no change in the prior disease probability in the
presence of positive results) and a negative likelihood ratio
of 0.07 (indicating sufŽ cient power to exclude the presence
of disease in negative Ž ndings).
Worked example
Figure 2 illustrates the dispersion of effect sizes observed in
the FAST meta-analysis in the scatterplot matrix. The
discriminatory power of FAST can be divided into its two
components by plotting the positive against the negative
likelihood ratios. By including the 95% conŽ dence intervals,
we are able to localise the weight of the data cloud, as well as
its outliers. Studies with large variance and wide conŽ dence
intervals will contribute less to the overall weight than
100
LR pos (log scale)
10
1
1.00 0.10 0.01
LR neg (log scale)
Figure 1 Likelihood ratio (LR) scatterplot of the four Žctitious meta-
analyses of 10 studies. UnŽlled symbols represent individual studies;
Žlled symbols are weighted summary likelihood ratios (Žxed-effects
model) with related 95% conŽdence intervals. Circles = studies with
high SN and high SP; diamonds = studies with low SN and low SP;
triangles = studies with low SN and high SP; squares = studies with
high SN and low SP. In analogy to the “classic” forest plot, test
characteristics can be assessed by eyeballing. The threshold values
of 10.0 and 0.1 are a useful procedure by which to divide the
matrix into four quadrants.
www.jmedscreen.com
49
1000
100
LR pos (log scale)
10
1
1.00 0.10 0.01
LR neg (log scale)
Figure 2 Likelihood ratio (LR) scatterplot matrix of the FAST meta-
analysis. UnŽlled circles represent individual studies. The Žlled circle
shows the weighted summary likelihood ratios (random-effects
model). Error bars represent 95% conŽdence intervals. FAST
provides a reasonable shift in prior probability for positive Žndings,
but has only weak power to exclude the presence of organ injury.
studies with more precise estimates. Obviously, FAST has
complex characteristics. As already indicated by the scope of
the data cloud, the relaxed combined estimate of the positive
likelihood ratio (as derived from the random-effects model)
of this meta-analysis is 25.03, with its associated 95%
conŽ dence interval ranging from 11.74 to 53.35. This con-
siderably exceeds the above-mentioned threshold level, and
positive ultrasound Žndings should substantially inuence
the prior probability of the presence of organ lacerations.
The corresponding negative summary likelihood ratio is
estimated at 0.22 (95% conŽ dence interval 0.15–0.33). This
translates to a fourfold increased chance of normal ultra-
sound Žndings in the absence of intra-abdominal lesions
(compared with the chance of a normal scan in the presence
of organ injuries) or a moderate to minor expectation of a
negative FAST examination.
DISCUSSION
The usefulness of a diagnostic test can be examined in three
ways:15
(1) its efŽ ciency in discriminating between a diseased and a
healthy population (which is expressed in terms of
sensitivity and speciŽcity);
(2) its consequences for clinical decision-making;
(3) the outcome related to the medical actions taken on the
basis of test Ž ndings.
The proven discriminatory power of a diagnostic test is a
prerequisite to investigate patient outcome, and a systematic
work-up of all the scientiŽc information is needed to assure
health-care professionals of the properties of tests used in
daily practice. Interestingly, there are far more systematic
reviews of therapeutic interventions than of diagnostic
tests.16 This discrepancy might relate to difŽ culties in inter-
preting diagnostic meta-analyses.
Journal of Medical Screening 2003 Volume 10 Number 1
50
One might argue against combining data from studies of
diagnostic tests. Obviously, the Ž ndings from such studies
will be in uenced by the characteristics and the size of the
patient sample, the prevalence of the condition under
investigation and the test threshold used; this provides an
argument for a description only of the range of data.
The same limitations also apply to meta-analyses of
therapeutic studies. By its nature, meta-analysis provides the
most probable approximation of a common effect size based
on the available data. However, with proper weighting
methods and sensitivity analyses, there is no reason why the
central estimate gained from individual diagnostic studies
should provide different or even less information than the
summary measure calculated from a set of therapeutic trials.
The problem of combining different diagnostic studies in a
meta-analysis was Ž rst addressed by summary receiver
operating characteristics (SROC) curves, proposed by Moses,
Shapiro and Littenberg.17,18 Regressing the true-positive
rates (sensitivity) to their false-positive rates (1 – speciŽcity)
visualises the trade-off between sensitivity and speciŽcity,
that is, the price in terms of false-positive Ž ndings that must
be paid for a reasonable true-positive rate. The SROC
approach is the recommended reference standard for
diagnostic meta-analysis.19 As a univariate summary
measure, the Q* value was proposed as the point of inter-
section where sensitivity equals speciŽcity. Q* shows the
desirable characteristics of a univariate measure of the over-
all discriminatory features of a diagnostic test. However,
there are some situations in which Q* leads to some mis-
interpretation.
Using the test set of Ž ctitious studies, it can be shown that
Q* hardly distinguishes highly sensitive but unspeciŽ c tests
(Q* = 0.70) from worthless procedures (Q* = 0.58).
Moreover, tests with poor sensitivity might yield virtually
similar Q* values, regardless of their speciŽcity (0.52 and
0.58, respectively) (see Figure 3).
If one is concerned about true-positive rates, the SROC
sufŽ ciently depicts the range of effect sizes. The associated
1.0
0.9
0.8
True Positive Rate (Sensitivity)
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
0.0 0.1 0.2 0.3 0.4 0.5
False Positive Rate (1 – Specificity)
Figure 3 Summary receiver operating characteristics (ROC) as the
standard approach to a meta-analysis of diagnostic tests, performed
on data of the four Žctitious meta-analyses displayed in Figure 1.
Q* values (where sensitivity equals speciŽcity) have been proposed
as the one-point estimate of test efŽcacy. Q* hardly differs between
tests with low SN and high SP and tests with both low SN and SP.
Calculations were made by generalised estimating equations (GEE)
models.
Journal of Medical Screening 2003
Stengel, Bauwens, Sehouli, et al.
one-point estimate, Q*, provides some global evidence of
test validity but it fails to discriminate tests with singular
strengths (or weaknesses) on one aspect of test performance.
If one generally doubts the clinical value of a diagnostic test
when SN and SP decrease, Q* perfectly meets the minimal
requirements of a univariate measure of effectiveness. If not,
relying on a single index of test efŽ cacy risks missing
procedures that perform well in one direction but poorly in
the other.
Likelihood ratios are helpful in showing both the bipolar
and the unipolar weaknesses of a test (as well as its unipolar
strengths). In analogy to forest plots, the likelihood ratio
scatterplot matrix illustrates the distribution and the centre
of effect sizes from a pool of individual studies and allows
identiŽ cation of outliers, as well as of studies relevant for
sensitivity analyses.
By using established Ž xed-effects and random-effects
techniques, likelihood ratios from individual trials can be
condensed to a point estimate from diagnostic meta-analysis.
Summary likelihood ratios are convenient numerical
descriptors that cover both qualitative and quantitative
characteristics of test performance.
ACKNOWLEDGEMENTS
The authors are indebted to Dr Barbara Herzberger for editorial
assistance. We also wish to thank the referee team of the Journal of
Medical Screening for helpful remarks.
.................
Authors’ affiliations
Dirk Stengel, Senior Surgeon, Department of Trauma Surgery,
Unfallkrankenhaus Berlin, and Lecturer in Evidence-Based Medicine,
Clinical Epidemiology and Biostatistics, Ernst-Moritz-Arndt-University of
Greifswald, Germany
Kai Bauwens, Senior Surgeon, Department of Trauma Surgery,
Unfallkrankenhaus Berlin, and Lecturer in Evidence-Based Medicine,
Clinical Epidemiology and Biostatistics, Ernst-Moritz-Arndt-University of
Greifswald, Germany
Jalid Sehouli, Senior Oncological Surgeon, Department of
Gynaecology and Gynaecological Oncology, and Lecturer in Evidence-
Based Medicine, Charité Virchow University Hospital, Berlin, Germany
Axel Ekkernkamp, Professor of Surgery and Head, Department of
Trauma Surgery, Ernst-Moritz-Arndt-University of Greifswald, and
Department of Trauma Surgery, Unfallkrankenhaus Berlin Trauma Centre,
Germany
Franz Porzsolt, Professor of Haematology, Oncology and Evidence-
Based Health Care, Human Studies Centre, Ludwig-Maximilians
University, Munich, Germany
REFERENCES
1 Berger JO. Statistial decision theory and bayesian analysis. New York:
Springer, 1985.
2 Fagan TJ. Nomogram for Bayes’ theorem. N Engl J Med 1975;293:257.
High SN/High SP
3 Centre for Evidence-Based Medicine. Toolbox: Likelihood ratios. Available
Q*=0.96 (95% CI0.73, 1.18)
at: http://minerva.minervation.com/cebm (last accessed November
Low SN/High SP 2002).
Q*=0.52 (95% CI0.36, 0.68) 4 Jaeschke R, Guyatt GH, Sackett DL, for the Evidence-Based Medicine
High SN/Low SP Working Group. Users’ guides to the medical literature. III. How to use an
Q*=0.70 (95% CI0.37, 1.04) article about a diagnostic test. A. Are the results of the study valid? JAMA
Low SN/Low SP 1994;271:389–91.
Q*=0.58 (95% CI0.49, 0.67) 5 Jaeschke R, Guyatt GH, Sackett DL, for the Evidence-Based Medicine
Working Group. Users’ guides to the medical literature. III. How to use an
0.6 0.7 0.8 0.9 1.0 article about a diagnostic test. B. What are the results and will they help
me in caring for my patients? JAMA 1994;271:703–7.
6 Goodman SN. Toward evidence-based medical statistics. 2. The Bayes
factor. Ann Intern Med 1999;30:1005–13.
7 Markert RJ, Walley ME, Guttman TG, Mehta R. A pooled analysis of the
Ottawa ankle rules used on adults in the ED. Am J Emerg Med
1998;16:564–7.
8 Lysakowski C, Walder B, Costanza MC, Tramer MR. Transcranial
Doppler versus angiography in patients with vasospasm due to a ruptured
cerebral aneurysm: a systematic review. Stroke 2001;32:2292–8.
9 Pettiti DB. Meta-analysis, decision analysis, and cost-effectiveness analysis.
Methods for quantitative synthesis in medicine. Oxford: Oxford University
Press, 2000.
Volume 10 Number 1 www.jmedscreen.com
Meta-analysis of diagnostic studies
10 Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Methods for meta-
analysis in medical research. Chichester: Wiley, 2000.
11 Deeks J, on behalf of the Statistical Methods Working Group of the
Cochrane Collaboration. Statistical Methods Programmed in Meta View,
Version 4. 1999. Available at http://www.cochrane.org (last accessed
January 2002).
12 Mantel N, Haenszel W. Statistical aspects of the analysis of data
from retrospective studies of disease. J Natl Cancer Inst
1959;22:719–48.
13 DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials
1986;7:177–88.
14 Stengel D, Bauwens K, Sehouli J, et al. Systematic review and meta-
analysis of emergency ultrasonography for blunt abdominal trauma. Br J
Surg 2001;88:901–12.
www.jmedscreen.com
51
15 Fineberg HV, Bauman R, Sosman M. Computerized cranial tomography.
Effect on diagnostic and therapeutic plans. JAMA 1977;238:224–7.
16 Knottnerus JA, van Weel C, Muris JWM. Evidence base of clinical
diagnosis: evaluation of diagnostic procedures. BMJ 2002;324:477–80.
17 Littenberg B, Moses LE. Estimating diagnostic accuracy from multiple
conicting reports: a new meta-analytic method. Med Decis Making
1993;13:313–21.
18 Moses LE, Shapiro D, Littenberg B. Combining independent studies of a
diagnostic test into a summary ROC curve: data analytic approaches and
some additional considerations. Stat Med 1993;12:1293–316.
19 Cochrane Methods Group on Systematic Review of Screening and
Diagnostic Tests. Recommended Methods. 1996. Available at
http://www.cochrane.org/cochrane/sadtdoc1.htm (last accessed April
2002).
Journal of Medical Screening 2003 Volume 10 Number 1