Publication Bias in Meta-

analysis Prevention,
Assessment and Adjustments
Dr.Noha Saleh
Associate professor of public health
Publication Bias

Publication bias arises when trials with statistically significant results

are more likely to be published and cited, and are preferentially
published in English language journals and those indexed in Medline.

In practice, the studies which are less likely to appear in the

published literature tend to be the less conclusive one (because of
smaller sample sizes or less statistical precision).
Preventing Publication Bias
prevention of publication bias through
• Search in prospective registration of trials, prior to their inception,
or at least prior to the availability of their results, is one strategy
that has been proposed to avoid publication bias. This registration
creates an unbiased sampling frame for subsequent meta-analyses.
However, registration alone does not guarantee access to data.

• Conduct a comprehensive search as much possible of both

published and unpublished ( hand searching) papers.
prevention of publication bias through

• Searching the grey literature.

• Contact with researchers

• Searching research registers. There are many national registers such

as the National Research Register in the UK, which contains ongoing
health research of interest to, and funded by, the UK National
Health Service (http://www.doh.gov.uk/research/nrr.htm).
 Statistical Methods for
Assessing Publication Bias
• The Funnel Plot
• Statistical tests
The Funnel Plot

• Funnel plots are a primary visual tool for the investigation of

publication and other bias in meta-analysis.

• Funnel plots are simple scatter plots of the treatment effects (effect
size estimate) estimated from individual studies against a measure
of study size. The name ‘funnel plot’ is based on the fact that the
precision in the estimation of the underlying treatment effect
increases as the sample size of the studies in the review increases.
The Funnel Plot
• So, when a measure of study size is plotted on the vertical
axis, results from small studies will scatter widely at the
bottom of the graph, with the spread narrowing among
larger studies. In the absence of bias (and when studies
estimate the same underlying effect) the plot will resemble a
symmetrical inverted funnel
 95% confidence limit

Individual study

Overall effect

Funnel plot
How many studies are required to examine
funnel plot asymmetry?
• Because the assessment of funnel plots is subjective, it is not possible
to provide formal guidance as to the number of studies in a meta-
analysis that are required before funnel plots may be used to assess
evidence for small-study effects in a meta-analysis.

• Informally, it seems unlikely that funnel plots will be useful in meta-

analyses containing a small number of studies (fewer than five, say).
Funnel Plot: what and how to read
Plots the effect size against
the size of the study
To study a funnel plot, look
at its LOWER LEFT corner,
that’s where negative or null
studies are located
If EMPTY, this indicates
“PUBLICATION BIAS”
Note that here, the plot fits in
a funnel, and that the left
corner is not all that empty,
but we cannot rule out
publication bias
Funnel plots for studies with binary
outcomes
• For binary outcomes, the power of a study, and the precision of the estimate
of a intervention’s effect, are determined both by the total sample size and
by the number of participants developing the event of interest.

• In the absence of bias the shape of plots using sample size on the vertical
axis will therefore not necessarily correspond to a funnel.
• Standard error, precision, variance and inverse variance are four
valid choices for the vertical axis. However, there are important
differences in the shapes of plots using these measures.

• the use of standard error is likely to be preferable in many

situations. Interpretation of funnel plots is facilitated by
inclusion of lines representing the 95 % confidence limits
around the summary intervention effect, which show the
expected distribution of studies in the absence of heterogeneity
or selection biases.
• Such lines can be drawn for all four variables; however, the lines will
be straight and thus represent the shape of an inverted funnel only
for standard error.

• For all other measures lines will be curved and, in the case of
precision and inverse variance, smaller studies will often be
compressed at the bottom of the graph. This makes the visual
assessment of asymmetry more difficult and gives emphasis to the
larger studies, which on average are less prone to bias.
• Because use of precision or the inverse variance for the
vertical axis increases the distance between the largest study
or studies and the rest, this choice might be preferable when
investigators wish to focus on a comparison of meta-analyses
of small studies with subsequent large studies
• funnel plot asymmetry were similar whether log odds ratio or log risk
ratio was chosen.

• n summary, standard error is likely to be the best choice of the

vertical axis in funnel plots, unless the aim is to emphasize the
difference between the largest studies and all others, when precision
is more appropriate. A log ratio measure, preferably the log odds
ratio, should generally be used for the horizontal axis.
Saudi Board of Preventive Medicine, Riyadh

Funnel plot
Give comment
Symmetrical plot in the absence of bias (open
circles indicate smaller studies showing no
beneficial effects)

Asymmetrical plot in the presence of

publication bias (smaller studies showing
no beneficial effects are missing)
 Give comment

data in which the standard error of the log risk ratio is plotted against the log risk ratio.
The horizontal axis displays the risk ratio on a log scale; this gives a plot that is identical in
shape to a plot of standard error of log RR against log RR. The plot shows asymmetry
because smaller studies tend to show greater adverse effects of smoking (risk ratios
greater than 1). This could, however, be due to the play of chance. Visual inspection alone
does not allow us to say definitively whether the funnel plot is asymmetric
Funnel plots for studies with numerical
outcomes
• there are many reasons for considering treatment effects
measured as mean differences to be more readily
interpretable than standardized mean differences or
correlation coefficients, these choices do not have obvious
implications for the shape of funnel plots.
• Given the typically low power of funnel plot asymmetry we think it
unlikely that it will be possible to distinguish statistically between
different forms of bias unless the meta-analysis contains a large
number of studies (say, more than 50
• The many possible causes of funnel plot asymmetry can be
seen as both a weakness and a strength of this means of
probing for bias in meta-analysis, compared to methods that
look for specific types of bias, in particular publication bias.
While it is natural to wish to reach definitive conclusions
when funnel plot asymmetry
Exercise 1
• meta-analysis of 37 studies of the effect of environmental tobacco
smoke on the risk of lung cancer in lifetime non-smokers (Hackshaw
et al., 1997). Whether this meta-analysis was affected by publication
bias has been a matter of ongoing controversy –Odds ratios
comparing spouses of smokers with spouses of non-smokers were
derived from five cohort studies and 34 case– control studies. Each
study provided an estimate of the risk ratio comparing the risk of lung
cancer for non-smoking spouses of smokers relative to the risk of lung
cancer in non-smoking spouses of nonsmokers
Comment

• The plot shows asymmetry because smaller studies tend to show

greater adverse effects of passive smoking (risk ratios greater than 1).
This could, however, be due to the play of chance.
Exercise 2
• Raudenbush (1984) reviewed 18 studies of teacher expectancy effects.
In these studies students had been randomly assigned to either a
control or ‘expectancy induction’ group where teachers were told to
expect superior performance by their students. The work grew out of
Robert Rosenthal’s interest (see Rosenthal and Jacobson, 1968) in the
effects of teachers’ expectations of students’ ability on students’
performance – the so-called ‘expectancy effect’. The effect size reflects
the difference between these two groups on scores on an intelligence
test.
• The results of each study were expressed as a standardized mean
difference.
Comment
• funnel plot of these data, which appears asymmetric.
 Saudi Board of Preventive Medicine, Riyadh

Ministry of Health, KSA

Detection of Publication bias by

Significance tests
• ‘Begg’s test’ (Begg and Mazumdar 1994)
• ‘Eggar’s test’ (Egger et al., 1997)
• Both ask: ‘Is the trial estimate related to the size of the trial?’
•   Begg and Mazumdar rank correlation method:
1.
 It uses Kendall's correlation between the standardized treatment
effect ti * and the variance of the treatment effect (vi* ) as:

t* = (t - tpoooled) /

 vi* is the standardized variance of the treatment effect as:

vi* = vi – {∑ (1/ vi)}2

Begg and Mazumdar’s test is equivalent to deriving
the rank correlation between the standardized effect
sizes and their standard errors.
According to Begg test, the asymmetry due to
publication bias may be explained by the association
between the large standard errors (small studies) and
large effect sizes.

The non significant test results refer to absence of

association between effect size and precision and so
absence of publication bias.
2. Egger regression method:
 A regression model is fitted using
the standardized estimate of the
treatment effect (zi = ti/√vi) as the
dependent variable and its
precision (1= √vi) as the
independent variable.

 This approach assumes that when

there is no publication bias, the
intercept will have an expected
value of zero and the slope will be
an unbiased estimate of the true
(underlying) effect (Figure a).
Smaller studies show effects that differ on
average from larger studies, the fitted line
will not pass through the origin (0 point)
(Figure b).
38
Funnel plot Rosenthal’s Fail-safe N Orwin’s Fail-safe N
Duval and Tweedie’s Trim & Fill rank correlation (P>0.05)
Regression

39
1-Failsafe N or File-Drawer Number
a.Rosenthal’s Failsafe N
• It was one of the earliest approaches for dealing with the problem of
publication bias and is still one of the most popular.

• Suppose a meta-analysis reports a significant p-value based on k studies.

We are concerned that studies with smaller effects are missing, and if we
were to retrieve all the missing studies and include them in the analysis,
the p-value for the summary effect would no longer be significant.
1-Failsafe N or File-Drawer Number
a. Rosenthal’s Failsafe N
• Rosenthal’s Fail-safe N (1979) suggested that we actually
compute how many missing studies we would need to
retrieve and incorporate in the analysis before the p-value
became nonsignificant. If it should emerge that we needed
only a few studies (say, five or ten) to ‘nullify’ the effect, then
we would be concerned that the true effect was indeed zero.
• Rosenthal referred to this as a File drawer analysis (file
drawers being the presumed location of the missing studies),
Failsafe N or File-Drawer Number
b.Orwin’s Fail-safe N
• Orwin noted, a problem with Rosenthal’s approach that it assumes
that the mean effect size in the missing studies is zero.

• Orwin (1983) proposed a variant on the Rosenthal formula that

addresses this issue. First, Orwin’s method allows the researcher to
determine how many missing studies would bring the overall effect to
a specified level other than zero.
 2. Duval and Tweedie’s Trim and Fill
• Trim’ (remove) the smaller studies causing funnel plot
asymmetry.
• Use the trimmed funnel plot to estimate the true ‘centre’ of the
funnel.
• Replace the omitted studies and their missing ‘counterparts’
around the centre (filling).
• The trim and fill method does not take into account reasons for
funnel plot asymmetry other than publication bias.

43
Thanks