STATISTICS AND RESEARCH DESIGN
Publication bias: Graphical and
statistical methods
Loukia M. Spinelia and Nikolaos Pandisb
Hannover, Germany, and Bern, Switzerland
V
arious statistical approaches and visual tools
have been developed to detect, estimate, and
evaluate the impact of publication bias in
meta-analysis results. In this article, we present the
most popular statistical methods and graphic tools to
address publication bias using an example.
COMPARING FIXED-EFFECT AND RANDOM-
EFFECTS RESULTS
The fixed-effect model is known to favor larger
studies; hence, it assigns greater weights to these
studies. By contrast, the random-effects model aims to
balance the weights more evenly across small and large
studies. With substantial small-study effects, where an
intervention seems to be more beneficial in smaller
studies, the random-effects summary effect size will pre-
sent the intervention as being more beneficial than the
fixed-effect summary effect size.1
The small-study effects issue is one of many factors
responsible for heterogeneity in the meta-analysis re-
sults. This means that if an intervention seems to be
more beneficial under the random-effects model than
under the fixed-effect model, the researchers should
investigate further whether they should attribute this
difference to the small-study effects alone (the interven-
tion was more effective in smaller studies) or to other
study characteristics.2
The forest plot in Figure 1 displays the meta-analysis
results on the effectiveness of fluoride gel against pla-
cebo for preventing dental caries in children and adoles-
cents under the random-effects and fixed-effect
models.3 The point estimates differ very slightly, and
the confidence intervals overlap perfectly. Note how
a
Midwifery Research and Education Unit, Hanover Medical School, Hanover,
Germany.
b
Department of Orthodontics and Dentofacial Orthopedics, University of Bern,
Bern, Switzerland.
Address correspondence to: Nikolaos Pandis, Department of Orthodontics and
Dentofacial Orthopedics, University of Bern, Freiburgstrasse 7, CH-3010 Bern,
Switzerland; e-mail, npandis@yahoo.com.
Am J Orthod Dentofacial Orthop 2021;159:248-51
0889-5406/$36.00
Ó 2020.
https://doi.org/10.1016/j.ajodo.2020.11.005
248
much wider the confidence interval under the random-
effects model is compared with the confidence interval
under the fixed-effect model. The similarity of the results
is an indication of the possible low impact of small-study
effects. Forest plots provide only a visual exploration;
hence, further investigation is required (eg, funnel plot
and proper statistical methods) to determine any
small-study effects and possible publication bias.
FUNNEL PLOT FOR SMALL-STUDY EFFECTS AND
PUBLICATION BIAS
Another graphical tool to investigate the relationship
between study size and effect size is the funnel plot.4
The funnel plot is a scatter plot in which the effect sizes
are plotted on the x-axis and the standard errors of the
effect sizes on the y-axis. The spread of the points cre-
ates a pattern like a funnel. In the funnel plot, the points
corresponding to studies with smaller sample size are
scattered on the bottom of the funnel (because they
yield effects with larger standard errors), and points cor-
responding to studies with larger sample size are scat-
tered in a narrow range of values at the top of the
funnel (because they yield effects with smaller standard
errors). Instead of standard errors, we could have used
the sample size of the studies or the variance of the effect
sizes.4 However, only the standard errors can spread out
the points on the bottom of the funnel where the smaller
studies are found and create a funnel-like pattern.4
To determine whether there is publication bias or
small-study effects, we need to understand how the
points are distributed. The symmetrical distribution of
the points about the summary effect size is an indication
of the absence of possible small-study effect or publica-
tion bias. However, any asymmetrical distribution of the
points may support the presence of possible small-study
effect or publication bias.2,4 The typical pattern in the
presence of small-study effects is a prominent asymme-
try at the bottom that progressively disappears as we
move up to larger studies.4
Figure 2 illustrates the funnel plot of our example. The
effect sizes have been estimated using the fixed-effect
model. The black line displays the summary effect size,
Statistics and research design
Fig 1. A forest plot that illustrates the meta-analysis results under the random-effects model (overall)
and the fixed-effect model (overall fixed-effect estimate).
Fig 2. Funnel plot on the effectiveness of fluoride gel
against placebo for preventing dental caries in children
and adolescents.
American Journal of Orthodontics and Dentofacial Orthopedics
249
and the red dotted line refers to no effect. The diagonal
lines represent the pseudo 95% confidence limits around
the summary effect size for each standard error on the ver-
tical axis. In the absence of heterogeneity, 95% of the
studies should be scattered within the funnel as defined
by these diagonal lines. The asymmetry of the funnel
plot is evident; toward the bottom of the plot, there is
only 1 small study, whereas the majority of the studies
are scattered above the middle of the funnel plot. Four
studies are outside the pseudo 95% confidence limits.
The absence of smaller studies (equivalently, only 1 small
study) at the bottom of the plot and studies with small ef-
fects (on the right side of the plot) are strong indications of
possible publication bias in the meta-analysis results. In
this case, we suspect that the summary effect may be
biased. However, the absence of smaller studies might
also indicate that the effectiveness of fluoride gel was
February 2021  Vol 159  Issue 2
250
likely to be investigated mainly in moderate and large
studies. Therefore, publication bias cannot be perceived
as the only cause of funnel asymmetry.2
Although funnel plot asymmetry has been associated
mostly with publication bias and small-study effects,
they are not the sole reasons for an asymmetric funnel
plot.2 Was the literature search intensive and compre-
hensive? Is there any difference in the quality of the
studies? For instance, it has been shown that small
studies with poor design and conduct tend to overesti-
mate the effect sizes; hence, this lower methodological
quality of smaller studies might contribute to the asym-
metry at the bottom of the funnel plot.2 Is there a plau-
sible reason to explain the larger effect sizes in smaller
trials? For instance, the choice of the measure of effect
can also result in funnel plot asymmetry,2 It has been
shown that risk difference provides more heterogeneous
effect sizes than risk ratio or odds ratio because they
tend to underestimate the effect sizes in studies with
low event rates5; hence, they can result in funnel plot
asymmetry. Therefore, various reasons related to study
characteristics and/or the analysis of the study results
should be considered before we attribute the asymmetry
entirely to publication bias.2,6 Meta-regression and sub-
group analysis are the tools to investigate possible asso-
ciations between study characteristics and effect sizes.
Publication bias has been equated mainly with the sup-
pression of studies with statistically nonsignificant results.
The funnel plot in Figure 2 fails to display the level of sta-
tistical significance of the effect sizes to assess whether the
level of significance is likely to explain the asymmetry.6 A
contour-enhanced funnel plot is an extension of the con-
ventional funnel plot, which incorporates contours of the
level of significance of the effect sizes.6 If more studies are
Fig 3. Contour-enhanced funnel plot on the effectiveness
of fluoride gel against placebo for preventing dental caries
in children and adolescents. Contours indicate the level of
significance of the effect sizes.
February 2021  Vol 159  Issue 2 American Journal of Orthodontics and Dentofacial Orthopedics
Statistics and research design
present in the contours of statistical significance than in
areas of statistical nonsignificance, the funnel plot will
be asymmetric.6 This asymmetry is indicative of the
inherent association between effect size and level of sig-
nificance.6 Studies that tend to favor the experimental
than the control intervention are more likely to provide
statistically significant results, and hence, to be present
in the contours of statistical significance. By contrast, if
more studies are absent in the contours of statistical sig-
nificance than in areas of statistical nonsignificance,
then the risk of publication bias might be low. Figure 3 de-
picts the contour-enhanced funnel plot of our example.
The black line displays the summary effect size, and the
red dotted line refers to no effect. The gray-scaled con-
tours correspond to different levels of significance, as indi-
cated in the accompanying box.
Studies are missing in all levels of significance both
on the right side and at the bottom of the plot, creating
pronounced funnel plot asymmetry. The fact that mostly
moderate and large studies with statistically significant
larger effect sizes are present indicates that publication
bias may be the cause of funnel asymmetry; hence,
further investigation is needed to detect possible associ-
ations between effect size and study characteristics.
EGGER TEST FOR FUNNEL PLOT ASYMMETRY
Because the interpretation of the funnel plot is sub-
jective, statistical tests that assess the relationship be-
tween sample size and effect size have been
developed. The test for small-study effects proposed
by Egger et al2 is the most widely used approach
(Fig 4). A regression line of the effect sizes against their
standard errors (using the inverse variance weighting
scheme) is drawn on the funnel plot.2 If there is no asso-
ciation between effect size and standard error, the
regression line will be parallel to the x-axis.2 To examine
whether there is evidence for statistically significant
Fig 4. Egger test for asymmetry.
Statistics and research design
asymmetry, we can perform a statistical test, known as
Egger's test. Specifically, we test whether the intercept
is equal to 0 which implies that the regression line
runs through the origin of the plot, and hence, the fun-
nel plot is symmetrical. If the P value is below 0.10, we
may conclude that there is evidence asymmetry because
of small-study effects.2
However, this test suffers the caveat of any test of
significance: it yields results of low power.1 Therefore,
the rejection of the null hypothesis should not be consid-
ered as definite evidence of asymmetry. In addition, tests
for funnel plot asymmetry should be performed when
there are at least 10 studies in the meta-analysis; other-
wise, the power will be too low to detect any true rela-
tionship.1 These tests should not be implemented
when the studies have similar sample sizes.1
Please note that small-study effects are not the only
source of asymmetry in funnel plots; language bias, cita-
tion bias, true heterogeneity, poor methodological
design of small studies, poor choice of effect measures,
and chance are also potential sources of asymmetry.2
JUDGMENT ON THE ROBUSTNESS OF THE META-
ANALYSIS EFFECT SIZE
In the presence of publication bias, the meta-analysis
results will not reflect reality. If we could retrieve all the
missing studies, then the direction or the statistical sig-
nificance of the results might be different.1 Approaches
to investigate the robustness of the summary effect size
have been proposed by Rosenthal and Orwin.7,8 These
approaches are presented very briefly below.
According to Rosenthal, we could estimate the num-
ber of (additional ‘negative’) studies needed to be
retrieved and included in the meta-analysis to increase
the P value for the meta-analysis above 0.05.7 If we
need only a small number of studies, we should be con-
cerned about the robustness of the summary effect size.
This approach is known as Fail-safe N.
Orwin's approach is an extension of Rosenthal's strat-
egy: researchers can determine the number of missing
studies needed to bring the mean effect size below a spe-
cific value other than 0 (for continuous measures) or 1
(for ratio measures).8 This specific value would be an ef-
fect size that indicates a minimum clinically important
difference between the compared interventions. The re-
searchers could also specify the mean effect size in the
missing studies to be any value other than 0 or 1.
JUDGMENT ON THE IMPACT OF PUBLICATION BIAS
Two popular approaches to judge the impact of pub-
lication bias on the conclusions drawn from the meta-
analysis results are the trim-and-fill approach and the
American Journal of Orthodontics and Dentofacial Orthopedics
251
Copas probabilistic model.9,10 A brief description of
these approaches is provided below.
A method known as “trim and fill” is implemented to
understand the impact of publication bias.9 This method
is an iterative procedure that identifies and corrects the
asymmetry in the funnel plot. It removes the small studies
with the most extreme results (trim) and recalculates the
summary effect size at each iteration until the funnel
plot becomes symmetric.9 Then, the removed studies are
added back into the analysis, and a “mirror value” is
computed for each one (fill). The output of this method
is an “adjusted” funnel plot that depicts both the observed
and the imputed studies so that the researcher can see how
much the summary effect size changes when the imputed
studies are included (trivially, modestly, or substantially).9
Another approach to assess the impact of publication
bias is to use probabilistic models. Copas10 examined the
relationship between the standard error of a study and the
probability that this study will be included in a meta-
analysis. A sensitivity analysis is performed to estimate
the summary effect sizes under a range of assumptions
about the probability of publication. Then the researcher
can explore how the estimated summary effect sizes and
their confidence intervals vary across the different sce-
narios assumed in the sensitivity analysis and determine
the impact of publication bias in the results.
REFERENCES
1. Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A,
Hrobjartsson A. Chapter 7. Considering bias and conflicts of inter-
est among the included studies. In: Higgins JPT, Thomas J,
Chandler J, Cumpston M, Li T, Page MJ, et al, editors. Cochrane
Handbook for Systematic Reviews of Interventions. Chichester,
United Kingdom: Cochrane; 2019.
2. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-
analysis detected by a simple, graphical test. BMJ 1997;315:629-34.
3. Marinho VCC, Higgins JP, Logan S, Sheiham A. Fluoride gels for
preventing dental caries in children and adolescents. Cochrane
Database Syst Rev 2002;(2):CD002280.
4. Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Chapter 30.
Publication bias. In: Introduction to Meta-Analysis. Chichester,
United Kingdom: John Wiley & Sons; 2009.
5. Deeks JJ. Issues in the selection of a summary statistic for meta-
analysis of clinical trials with binary outcomes. Stat Med 2002;
21:1575-600.
6. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-
enhanced meta-analysis funnel plots help distinguish publication
bias from other causes of asymmetry. J Clin Epidemiol 2008;61:991-6.
7. Rosenthal R. The file drawer problem and tolerance for null results.
Psychol Bull 1979;86:638-41.
8. Orwin RG. A fail-safe N for effect size in meta-analysis. J Educ Be-
hav Stat 1983;8:157-9.
9. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based
method of testing and adjusting for publication bias in meta-anal-
ysis. Biometrics 2000;56:455-63.
10. Copas J. What works?: selectivity models and meta-analysis. J R
Stat Soc A 1999;162:95-109.
February 2021  Vol 159  Issue 2