World J. Surg.

29, 582–587 (2005)

DOI: 10.1007/s00268-005-7917-7

Role of Systematic Reviews and Meta-analysis in Evidence-based Medicine

Stefan Sauerland, M.D., M.P.H.,1 Christoph M. Seiler, M.D., M.Sc.2
1
Biochemical and Experimental Division, Medical Faculty, University of Cologne, Ostmerheimer Strasse 200, D-51109, Köln, Germany
2
Clinical Study Center of the German Surgical Society (SDGC), Department of Surgery, University of Heidelberg, Im Neuenheimer
Feld 110, D-69120 Heidelberg, Germany

Published Online: April 14, 2005

Abstract. The overwhelming increase in the quantity of clinical evidence
has led to detachment of the evidence and practice because new evidence
can be integrated into clinical practice only after it has been critically
appraised and synthesized on the basis of the existing evidence. Because
many clinicians lack the skills and the time for such information pro-
cessing, systematic reviews and meta-analyses, their quantitative coun-
terparts, play an important role in health care. Well performed system-
atic reviews provide clinically relevant information for surgeons, abro-
gating the need to identify, read, and evaluate many individual studies.
This article reviews the basic principles of meta-analysis, discusses its
potential weaknesses such as heterogeneity and publication bias, and
highlights special situations when dealing with surgical trials.
The practice of evidence-based surgery consists of three essential
parts: the preferences, concerns, and expectations of each patient;
the clinical expertise including skills, past experience, and
knowledge of the surgeon; and the best research evidence that is
relevant for clinical practice [1]. The challenge for a surgeon to-
day is the last part of this definition. The traditional acquisition of
knowledge from inter- and intraspecialty consultation during
work or at a congress is limited in many ways (e.g., incomplete or
biased knowledge of colleagues, uncontrolled statements without
quantification, conflicts of interest). The most important source
for external evidence is therefore the medical literature. In the
medical literature, randomized controlled trials (RCTs) are con-
sidered a reference standard of clinical research methods owing
to a number of unique characteristics that try to avoid possible
biases.
Surgeons, however, who try to upgrade their knowledge by
reading journal articles face an enormous task. There are about
200 major surgical journals, and each journal publishes about 250
articles per year. Thus, any surgeon who wants to keep abreast of
new knowledge must browse through 50,000 articles per year,
which means 137 articles per day. This figure may be smaller for a
highly specialized surgeon, who can carefully select his or her
Correspondence to: Stefan Sauerland, M.D., M.P.H., e-mail: S.Sauer-
land@uni-koeln.de
reading, but it still requires much more time than a busy surgeon
can spare [2].
In addition to the huge volume of literature, its scattered nature
poses further problems. Every time a new article appears, readers
must compare this new piece of evidence with the existing
external evidence to come to an overall clinical conclusion. This
process of collecting and summarizing scientific information over
many years requires extraordinary memory capacities that most
human brains simply lack.
A third problem of the surgical literature lies in the small
proportion of high-quality evidence in most journals [3]. The
number of surgical randomized controlled trials is still small, and
case reports and series seem to be the predominant publication
type. Reading surgical articles can also be trying owing to dif-
ferences in study quality, such as insufficient sample size, unclear
methodology, and nonclinical outcome parameters [4].
It is therefore not surprising that many surgeons subscribe to
only a few selected journals. In consequence, clinicians are
often unaware of important surgical innovations. On the other
hand, researchers are sometimes disappointed by the small
overall impact of the publication of a major trial. Here, sys-
tematic reviews are an important tool for finding important and
valid studies while filtering out the large number of seriously
flawed and irrelevant articles. By condensing the results of
many trials, systematic reviews allow readers to obtain a valid
overview on a topic with substantially less reading time in-
volved.
Traditional and Systematic Review Articles
Medical review articles have been popular for many decades
because someone else has spent the time to find and read the
relevant primary articles, enabling clinical readers to gain an
overview quickly on the current state of science. In the past,
clinical experts were usually nomintated to write review articles.
Because there were only a handful of surgical journals at that
time, no special skills were required to retrieve the relevant
articles from these sources. With the increasing amount of
medical literature, however, traditional review articles occa-
sionally failed to identify the key studies on a given topic, thus
Sauerland and Seiler: Role of Systematic Reviews and Meta-analysis
drawing questionable conclusions for clinical practice. In gen-
eral, many reviews are invited commentaries and not properly
conducted pieces of research. At worst, the conclusions of
nonsystematic reviews depend on the financial affiliations of the
authors [5].
Today, our reliance on review articles depends on clinical
expertise combined with a systematic approach to literature. A
review earns the adjective ‘‘systematic’’ if it used systematic ways
to identify relevant studies, appraising their quality and formu-
lating evidence-based conclusions. An elegant approach by Ant-
man et al. revealed that systematic reviews reflect the current
state of science better than traditional narrative review articles,
which tended to ignore innovations despite the innovationÕs
proven effectiveness [6].
Systematic reviews can be distinguished from traditional re-
views by the reviewÕs methods section. Most narrative reviews
simply lack a methods section, whereas systematic reviews de-
scribe their methodology in detail, including questions under
investigation, search strategy, study selection criteria, assessment
of study quality, and data synthesis. The introduction of system-
atic reviews does not necessarily mean that narrative reviews are
an outdated scientific concept. The strength of narrative reviews
lies in their broader scope, which allows one to discuss the context
of a treatment strategy. In addition, educational review articles
(such as the present article) can be a valuable instrument for
continuing medical education.
The statistical pooling of results from independent studies is
called meta-analysis [7]. Meta-analysis is a possible but not nec-
essary statistical extension of a systematic review. The two terms
are not synonymous. Only if several studies with similar patients
and study designs have been identified does pooling these studies
seem reasonable so a meta-analysis can be included as a part of a
systematic review. It should be noted that despite the obvious
differences between the two terms they are still often used
interchangeably. Even worse, many systematic reviews are in-
dexed in MEDLINE as traditional or educational review articles,
which can make it a difficult task to find systematic reviews for
answering a clinical question.
Although meta-analysis had been a common research method
in agricultural research, psychology, and physics [8], it was not
until the 1980s that medical researchers used meta-analytic
methods [7, 9]. Medical meta-analysis increased significantly
thereafter, but there are still plenty of primary studies from var-
ious fields that have not yet been examined by meta-analysis. One
reason for this shortage of high-quality meta-analyses is the lack
of cooperation between surgeons and meta-analysts. In addition,
production of a systematic review is an extremely demanding task,
which is comparable to a clinical trial.
583
Fig. 1. Seven steps when
performing a meta-analysis.
(Modified from Neugebauer et al.
[10], with permission.)
Basic Principles of a Systematic Review
A systematic review (with or without meta-analysis) can be de-
scribed as a seven-step process (Fig. 1). Prior to the beginning of
this process, selected sources of systematic reviews and guidelines
(e.g., Cochrane, MEDLINE, Embase) should be checked for re-
views already existing in the respective field. Every research
project then starts with a clear clinical question and the setup of a
protocol. The review protocol is necessary because meta-analysis
itself is a retrospective study, where data-driven analyses can be as
misleading as in any clinical study. Inclusion or exclusion of cer-
tain studies can heavily distort the results of a meta-analysis.
Therefore, criteria for study selection must be specified in ad-
vance to the literature research. Most meta-analyses are done
based on randomized controlled trials, but a meta-analysis can
aggregate the results from other epidemiologic designs such as
nonrandomized or case-control studies in exactly the same man-
ner. Only data from noncomparative studies cannot be pooled by
this method. Different study types should rarely be merged in a
meta-analysis.
Searching the literature for potentially relevant articles [11] is
the most time-consuming part of a systematic review. Conse-
quently, systematic reviews still appear in the literature for which
careless and superficial literature searches were performed. It has
been demonstrated that restricting a literature search to only
English-language or MEDLINE-indexed articles carries the po-
tential risk of an incorrectly increased treatment estimate [12, 13].
Any valid search strategy should include non-MEDLINE data-
bases such as Embase and Cochrane. Language restrictions
should be avoided.
Two independent persons should screen and select potentially
relevant abstracts. All retrieved studies are assessed again inde-
pendently for inclusion and exclusion criteria. The exclusion of
studies should be documented so readers can understand the
essential selection process [14]. Once a set of studies has been
selected, these studies must be critically appraised in detail [15].
The use of checklists may be helpful for this assessment. How-
ever, the summary score determined by the checklists should not
be used as a measure of overall study quality [16]. Important
clinical and methodologic study characteristics should then be
tabulated. At this stage, it might be necessary to contact the au-
thors of primary studies to obtain important information missing
in the publication. Contact with the authors is also essential when
a study has been published as an abstract only.
The fourth step in a systematic review is the extraction of data
from primary studies. In systematic reviews the original data
summarized in publications is analyzed, but again it can be nec-
essary to contact the authors if an article does not contain the
584
Fig. 2. Example of a meta-analysis without major apparent problems. All
studies vary around the summary estimate of 0.81. Although none of the
10 primary studies is significant, the pooled estimate becomes significant
because of the increased power (p = 0.04). Larger studies can be
recognized by having smaller confidence intervals and larger boxes.
(Hypothetical data for this forest plot were simulated assuming an effect
size of 0.80 and an average study size of 50 patients.)
relevant data in the right format. For a meta-analysis, original
data should be given as rates (for binary outcomes) or as mean
values with standard deviations (or any other measure of spread)
for both treatment and control groups. If results are reported only
as being ‘‘significant,’’ it is nearly impossible to use this infor-
mation. Another problem arises when data are skewed in a way
that normal distribution is missing. Hospital stay, for example, is
often reported as median and interquartile range. According to
the distribution of data, this approach is correct and even desir-
able; but the meta-analysis cannot work with nonnormally dis-
tributed data. Similar but less severe problems occur when
survival data must be analyzed.
Statistical Principles
Step five of a systematic review consists of the statistical meta-
analysis. Although it is frequently assumed that pooling data from
various trials is equivalent to a simple addition of trial results, this
assumption is not correct. In fact, each study must be treated as a
singly entity. Differences in baseline risk, concomitant therapy,
and outcome definition of studies should not be combined di-
rectly. Consequently, treatment estimates, such as relative risks,
must be calculated for each trial result. Multiplication of relative
risks with a weighting factor that is dependent on the sample size
of each trial ensures comparability of qualitatively and quantita-
tively diverse studies. This pooling method is valid, as it does not
use data from different studies as if they came from one large
single study. Nowadays, freely available statistical software pack-
ages allow a quick meta-analytic calculation of different effect
measures with 95% confidence intervals (CIs) (Fig. 2). The choice
of the effect measure (e.g., relative risk, odds ratio, or risk dif-
ference) depends on the effect sizes and their relation to baseline
risk [17].
The main argument against meta-analysis claims that highly
unequal studies are forced into a common treatment estimate.
This mixture of ‘‘apples and oranges’’ may represent an unjus-
tified simplification, whereas in truth a broad diversity of results
exists. Therefore, it is generally agreed upon that any meta-
analysis should include a formal examination of heterogeneity.
World J. Surg. Vol. 29, No. 5, May 2005
It can (and should) be tested statistically whether the results of
the studies in a meta-analysis differ among each other to a
greater extent than can be expected from pure chance alone.
For this purpose, a new quantity, 12, was recently proposed
[18]. It allows measurement of heterogeneity as a percentage,
with values over 75% indicating high heterogeneity. Alterna-
tively, heterogeneity can also be examined graphically, as shown
in Figure 3.
Dealing with Heterogeneity among Trials
Any systematic review inevitably includes studies that are to some
extent heterogeneous. Heterogeneity primarily denotes that the
range of results varies among included trials. Although hetero-
geneity may be due to chance alone, it can also be caused by
clinical or methodologic differences among trials and might
thereby result in systematic errors. Clinical considerations are
also necessary when deciding whether a set of trials should be
integrated into a common treatment estimate. Heterogeneity in
surgical trials is likely to arise through diversity in technical
expertise of trial surgeons. From this viewpoint, heterogeneity
could represent an important opportunity to understand treat-
ment effects better [19]. Ideally, meta-analysis can resolve the
discrepancies between individual trial results by identifying con-
founding variables that modify trial results. Investigation of pos-
sible effect modifiers is called sensitivity analysis—step six in
Figure 1.
In most cases, sensitivity analyses classify primary studies into
two (or three) groups (Fig. 3). Groups are formed according to
the methodologic or clinical characteristics likely to influence trial
results. From a clinical point of view, patient characteristics (e.g.,
age, gender, co-morbidity), treatment characteristics (e.g., com-
pliance, doses, or duration of therapy), and outcome measure-
ment (e.g., length of follow-up, definition of diagnostic
procedures) are important variables that might interfere with trial
results [20]. Methodologically, it has been shown that blinding,
proper randomization (with concealed allocation), and correct
analysis (according to intention-to-treat) have an impact on trial
results [15, 21]. Depending on the topic, some of these variables
should be selected early in the review protocol to define clearly
the prospective sensitivity analyses.
In predefined sensitivity analyses, different trial subgroups are
then compared with each other. It is not important whether the
results of compared groups remain significant; rather, differences
between the groups deserve attention. Significant differences
indicate that trial results might vary with the presence (or ab-
sence) of this covariable. Such a finding can help to resolve dis-
crepancies among primary trials and for generating new
hypotheses. If the covariable is continuous, trial results may also
be modeled along this variable without creating subgroups [22].
Such meta-regression may, for example, find that treatment ef-
fects continually decrease with increasing age of each studyÕs
patient sample.
However, if at the end the trial results are still highly hetero-
geneous and no explanation can be found for this heterogeneity,
the pooling of trial results is better omitted. In this case, the
review would stop at the state of a systematic review without
extension to a meta-analysis. A systematic review that abstains
from pooling studies should not be considered a ‘‘failed’’ meta-
analysis.
Sauerland and Seiler: Role of Systematic Reviews and Meta-analysis 585

Fig. 3. Left. Example of a meta-analysis with evidence of strong heterogeneity. Note that the confidence intervals of many primary studies are not
overlapping each other. The pooled estimate gives a misleading impression of the primary trials. Right. Heterogeneity can be resolved by subdividing the
studies into two groups according to clinical or methodologic characteristics. Two different effects of 0.52 and 1.18 appear in the two subgroups.

Fig. 4. Left. Example of a meta-anaiysis with evidence of publication bias. Note that only two small studies (E and H) reach borderline signficance; the
larger studies are closer to the line of equivalence (RR = 1). Right. Publication bias can be detected by arranging studies according to sample size (or
statistical weight), as shown in the funnel plot. Apparently, some smaller, nonsignificant studies went unpublished. The gray fields indicate the 95%
confidence limits for study results.

Publication Bias Once the funnel plot shows clear asymmetry, the interpretation
of the meta-analysis becomes complicated, as there is no oppor-
Studies reporting significant treatment effects are more likely to
tunity to locate the apparently missing unpublished ‘‘phantom’’
be published [23], published in English [13], and published more
studies. In consequence, for evidence-based decision-making, one
quickly [24] than nonsignificant studies. This problem, known as
rigorously conducted study of 1000 patients is a better informa-
publication bias, affects the validity of the medical literature as a
tion source than 10 studies of a 100 patients each [26].
whole because the obtained results may be misleading. Although
In the seventh and final step of a systematic review, the results
exhaustive literature searches can partly compensate for the
are summarized and published. Publication of systematic reviews
problem, unpublished studies cannot be traced even through the
should comply with the QUORUM standards [14], which rec-
best literature search. Therefore, any withholding of study results
ommend a clear description of all critical steps. The conclusions
should be banned as scientific and ethical misconduct. In addi-
of a systematic review should envisage clinical and scientific rec-
tion, prospective registration of planned or ongoing trials may
ommendations. Especially in cases where not a single study was
reduce the proportion of unpublished data.
found addressing an important question, systematic reviews are
Until these efforts are realized, other measures are necessary to
important for guiding future research funding. For further details
examine how seriously the results of a meta-analysis are influ-
on systematic review methodology, readers are referred to recent
enced by publication bias. In this regard, the funnel plot is a
comprehensive textbooks [27, 28].
useful graphic tool, which Egger et al. have complemented with a
statistical test [25]. The principle of the funnel plot is quite simple
(Fig. 4). It goes by the assumption that larger studies are more
Special Types of Systematic Reviews
likely of being published, whereas smaller trials may get published
only if they report a significant result. Thus, any difference be- If only a small number of large studies have been published on a
tween the results of large versus small studies should cast suspi- topic, it may be possible to obtain the original data sets for all
cion on the overall validity of the published evidence because this primary studies, so meta-analysis can be performed with these
finding indicates that some small studies have not been published data instead of published means and percentages. Sometimes
owing to their nonsignificance. Today, the funnel plot has become such an individual patient data (IPD) meta-analysis generates
a standard procedure in meta-analysis, although sometimes con- more precise results [29], but usually this increase in precision
structing a funnel plot is impossible because of the low number of does not justify the increased efforts of data collection. One well
available studies. Also noteworthy is the fact that publication bias recognized example of a surgical IPD meta-analysis is the Euro-
does not necessarily lead to heterogeneity. pean Hernia TrialistsÕ Collaboration [30].
586
Recently, network meta-analysis was proposed for situations
for which only indirect comparisons have been published. If, for
example, a set of studies has compared treatments A and C,
whereas other studies have compared treatments B and C, it is a
valid approach to take C as a basis for comparing A and B [31].
This technique is valuable when many placebo-controlled trials
are available on a topic but head-to-head comparisons are lack-
ing. In the surgical field, such an approach might be chosen for a
future meta-analysis of inguinal hernia trials.
Finally, it should be noted that meta-analyses of diagnostic
studies differ from therapeutic meta-analyses in many ways. First,
the literature search looks for cohort studies instead of random-
ized trials. Various study characteristics, such as independent and
blinded performance of diagnostic tests, are important [32]. Also
different are the statistical procedures, which most often use
weighted estimates of sensitivity and specificity for pooling.
Role of Evidence Based Guidelines
The methodology of the systematic review has strongly influenced
surgeonsÕ opinion of clinical guidelines. If review articles should
be systematic, guidelines and consensus conference also should be
evidence-based [33]. Today, most guideline projects start with a
systematic review of the literature. If guideline authors find a
systematic review, there is often no need for another look through
all of the primary studies. The major difference between a
guideline and a systematic review is that a guideline usually in-
cludes more than one question so more than one systematic re-
view is necessary. In addition, a guideline has to reach a clinical
conclusion, whereas a systematic review often concludes only that
insufficient evidence is available.
In essence, practice guidelines and meta-analyses are not
competitors; rather, they are complementary approaches to
closing the gap between research and practice. One example of a
fruitful coexistence of systematic reviews and guidelines (or
health technology assessment) is the guideline development
program of the European Association for Endoscopic Surgery
(EAES). Another example is the Australian Safety and Efficacy
Register of New Interventional Procedures (ASERNIP), which is
partly led by the Royal Australasian College of Surgeons.
Surgical Meta-analyses
Most of the differences between meta-analyses in surgery and
those in other fields originate from the differences between, for
example, a surgical procedure and a pharmaceutical drug. While a
tablet acts more or less uniformly, the success of an operation
depends on the expertise of the surgeon. Furthermore, most
surgeons do not perform their operations in a fully standardized
and reproducible manner. Slight modifications occur over time
that may lead to variable treatment effects. Although a few sur-
gical RCTs have prescribed the commitment of all trial surgeons
to a standard technique, this is still not the case in most surgical
studies.
Although there is a large body of evidence showing superior
treatment results from experienced versus inexperienced sur-
geons, this association has not yet been shown in surgical meta-
analyses. A comparison of operating times and wound infection
rates after laparoscopic and open appendectomy failed to find a
difference between experienced and inexperienced trial surgeons
World J. Surg. Vol. 29, No. 5, May 2005
[34, 35]. In primary studies, however, reporting on surgical
expertise is often vague or completely missing. These and other
problems may lead to a more heterogeneity among surgical trials
than medical trials, thus threatening the validity of a meta-analysis
based on these data. Those who use systematic reviews for clinical
decision-making and evidence-based medicine must take into
account the possible shortcomings of surgical meta-analyses.
Cochrane Collaboration
Given the complex and difficult methodology of systematic re-
views, it is understandable that some reviews are still published
despite major flaws. Meta-analysts therefore have decided to
collaborate on an international level to raise the quality standards
of systematic reviews. Based on preliminary experience in the
united kingdom, the Cochrane Collaboration was founded a
decade ago. Its aim is to prepare, maintain, and disseminate
systematic reviews of the effects of health care [36]. The organi-
zational structure of the Collaboration is based on the scientific
enthusiasm of its members, and financial issues are kept in the
background. The revenues from selling the Cochrane Library are
used mostly for burning and distributing CDs and for software
development.
Cochrane reviews, now numbering over 1700, have been found
to be of higher quality than paper-based reviews [37]. This finding
can be explained by the close collaboration of researchers and
clinicians. Furthermore, manual searching of non-MEDLINE
journals and abstract volumes for RCT reports is a cornerstone of
the Cochrane Collaboration. Probably the biggest advantage of
Cochrane reviews is their periodic updating. Each time a major
new study appears (or at least every second year), a review must
be updated or it is excluded.
For surgeons, the Cochrane Library also contains valuable
information, although clearly more reviews have been published
on nonsurgical topics [38]. The interested reader should note that
many surgical topics are as yet unreviewed, although conducting a
review would earn the reviewer a free copy of the Cochrane Li-
brary.
Conclusions
The art of writing an overview has developed from the classic and
unsystematic into a systematic, often quantitative review. For
surgeons such articles provide the opportunity to obtain evidence-
based summaries more quickly than from primary studies. Be-
cause heterogeneity, publication bias, and learning curve effects
represent serious problems to systematic reviews, surgeons should
have a basic understanding of methodology. Not every meta-
analysis represents level I evidence.
References
1. Sackett DL, Rosenberg WM. The need for evidence-based medicine.
J. R. Soc. Med. 1995;88:620–624
2. Millat B, Fingerhut A, Flamant Y, et al. Survey of the impact of
randomised clinical trials on surgical practice in France. Eur. J. Surg.
1999;165:87–94
3. Sauerland S, Bouillon B, Neugebauer E. Praktische Anwendung der
EBM in der Chirurgie: Evidenz und Erfahrung. Z. Arztl. Fortbild.
Qualitatssich. 2003;97:271–276
4. Uhl W, Wente MN, Büchler MW. Chirurgisch-klinische Studien in
der praktischen Durchführung. Chirurg 2000;71:615–625
Sauerland and Seiler: Role of Systematic Reviews and Meta-analysis
5. Barnes DE, Bero LA. Why review articles on the health effects of
passive smoking reach different conclusions. J.A.M.A. 1998;279:1566–
1570
6. Antman EM, Lau J, Kupelnick B, et al. A comparison of results of
meta-analyses of randomized control trials and recommendations of
clinical experts: treatments for myocardial infarction. J.A.M.A.
1992;268:240–248
7. LÕAbbe KA, Detsky AS, OÕRourke K. Meta-analysis in clinical re-
search. Ann. Intern. Med. 1987;107:224–233
8. Sauerland S, Sauerland T, Antes G, et al. Meta-Analysen von Quarks,
Baryonen und Mesonen: eine ‘‘Cochrane Collaboration’’ in der
Physik. Z. Arztl. Fortbild. Qualitatssich. 2002;96:95–98
9. Neugebauer E, Lorenz W. Meta-analysis: from classical review to a
new refined methodology. Theor. Surg. 1989;4:79–85
10. Neugebauer, EAM, Lefering, R, McPeek, B et al. (1998) "Systemat-
ically reviewing previous work" In: Troidi, H, McKneally, MF, Mulder,
DS (editors), Surgical Research: Basic Principles and Clinical Practice,
3rd ed., Springer, Berlin, pp 341-355
11. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from
number of citations retrieved. J.A.M.A. 1999;282:634–635
12. McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey
literature influence estimates of intervention effectiveness reported in
meta-analyses? Lancet 2000;356:1228–1231
13. Egger M, Zellweger-Zahner T, Schneider M, et al. Language bias in
randomised controlled trials published in English and German. Lancet
1997;350:326–329
14. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of
reports of meta-analyses of randomised controlled trials: the QUO-
ROM statement. Lancet 1999;354:1896–1900
15. Jüni P, Altman DG, Egger M. Assessing the quality of controlled
clinical trials. B.M.J. 2001;323:42–46
16. Jüni P, Witschi A, Bloch R, et al. The hazards of scoring the quality of
clinical trials for meta-analysis. J.A.M.A. 1999;282:1054–1060
17. Deeks JJ. Issues in the selection of a summary statistic for meta-
analysis of clinical trials with binary outcomes. Stat. Med.
2002;21:1575–1600
18. Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency
in meta-analyses. B.M.J. 2003;327:557–560
19. Thompson SG. Why sources of heterogeneity in meta-analysis should
be investigated. B.M.J. 1994;309:1351–1355
20. Glasziou PP, Sanders SL. Investigating causes of heterogeneity in
systematic reviews. Stat. Med. 2002;21:1503–1511
21. Moher D, Pham B, Jones A, et al. Does quality of reports of
randomised trials affect estimates of intervention efficacy reported in
meta-analyses?. Lancet 1998;352:609–613
22. Lau J, Ioannidis JP, Schmid CH. Summing up evidence: one answer is
not always enough. Lancet 1998;351:123–127
587
23. Dickersin K. The existence of publication bias and risk factors for its
occurrence. J.A.M.A. 1990;263:1385–1389
24. Krzyzanowska MK, Pintilie M, Tannock IF. Factors associated with
failure to publish large randomized trials presented at an oncology
meeting. J.A.M.A. 2003;290:495–501
25. Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis
detected by a simple, graphical test. B.M.J. 1997;315:629–634
26. LeLorier J, Grégoire G, Benhaddad A, et al. Discrepancies between
meta-analyses and subseuqent large randomized, controlled trials. N.
Engl. J. Med. 1997;337:536–542
27. Egger M, Davey Smith G, Altman DG. Systematic reviews in health
care. London: BMJ Publishing, 2001
28. Khan K, Kunz R, Kleijnen J, et al. Systematic reviews to support
evidence-based medicine. London: Royal Society of Medicine Press,
2003
29. Stewart LA, Parmar MKB. Meta-analysis of the literature or of
individual patient data: is there a difference?. Lancet 1993;341:418–
422
30. McCormack K, Scott NW, Go PM. et al. Laparoscopic techniques
versus open techniques for inguinal hernia repair. Cochrane Database
Syst. Rev. 2003;CD001785
31. Song F, Altman DG, Glenny AM, et al. Validity of indirect com-
parisons for estimating efficacy of competing interventions: empirical
evidence from published meta-analyses. B.M.J. 2003;326:472–475
32. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of
design-related bias in studies of diagnostic tests. J.A.M.A.
1999;282:1061–1066
33. Sauerland S, Neugebauer E. Consensus conferences must include a
systematic search and categorization of the evidence. Surg. Endosc.
2000;14:908–910
34. Sauerland S, Lefering R, Neugebauer EAM. Laparoscopic versus
open surgery for suspected appendicitis. Cochrane Database Syst.
Rev. 2002;CD001546
35. Sauerland S, Lefering R, Neugebauer E. Explaining heterogeneity in
surgical trials by assessing learning-curves [abstract] In 2nd Sympo-
sium on Systematic Reviews. Oxford, Centre for Statistics in Medi-
cine, 1999;65
36. Bero L, Rennie D. The Cochrane Collaboration: preparing, main-
taining, and disseminating systematic reviews of the effects of health
care. J.A.M.A. 1995;274:1935–1938
37. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of
systematic reviews and meta-analyses: a comparison of Cochrane re-
views with articles published in paper-based journals. J.A.M.A.
1998;280:278–280
38. Emond SD, Wyer PC, Brown MD, et al. How relevant are the sys-
tematic reviews in the Cochrane library to emergency medical prac-
tice? Ann Emerg. Med. 2002;39:153–158