Interpretation of forest plots:

Definition of systematic review:

A review of the evidence on a clearly formulated question
that uses systematic and explicit methods to identify, select
and critically appraise relevant primary research, and to
extract and analyse data from the studies that are included
in the review.
Importance of systematic Reviews:
 To dependably answer questions using standard
approach to reviewing previous research. Those
answers are needed in everyday health practice.
 To reduce large quantities of information that are
overwhelming for any practitioner. When you need to
find an answer to a specific question and you search-
hundreds or thousands of studies are available. Its
impractical for any practitioner to read all the
information available. So they reduce the large
quantities of information into manageable portions,
proving an objective up to-date summary of the
evidence.
 To improve the standards of research conduction, both
in the conduction and the reporting process.
Advantages of systematic reviews:
They can systematically aggregate or combine mathematical
data into meta analysis.
Definition of meta-analysis:
A statistical analysis that combines or integrates the results
of several independent clinical trials considered by the
analyst to be combinable.
The only factor is that the author must believe that these
studies should be joined together.
Not every systematic review has meta-analysis because
sometimes the author believes that systematic review they
have cannot be combined mathematically and can only
provide a qualitative description or review of the studies it
included.
 Statistical method to combine the results from two or
more studies
 Estimates an average or common effect
 Optional part of a systematic review
Rationale for meta-analysis:
Depends mainly on the researcher conducting the review as
it is sometimes feasible and sometimes not.
By combining the samples of individual studies, the overall
sample size is increased, thereby improving the statistical
power of the analysis as well as of estimates of treatment
effects.
Explanation: e.g. effectiveness of a new ttt X. The RCTs
conducted might be 15. Each study for that is conducted on a
limited sample size. If a team of researchers go for systematic
reviews to combine all these studies into one study. If they
combine those into a meta-analysis.
Transformed the small sample sized RCTs into a large mega
sample size study. This increases statistical power.
Statistical power; is the ability of a study to detect a
difference if it is there or not.
Why perform meta-analysis:
 Quantify treatment effects and their uncertainty
 Increase power
 Increase precision
 Explore differences between studies
 Generate new hypotheses
When to do a meta-analysis:
1. More than one study: when more than one study has
estimated a ttt effect or association.
2. When the difference in the study characteristics are
unlikely to affect the treatment effect, this effect has
been measured and reported in similar ways (or when
the data are available to allow this)
Which means: combining independent studies
conducted by different authors so accordingly there are
going to be inherent differences. The differences in the
studies and the results are sometimes huge and other
tine can be easily overcome.
These differences (heterogeneity are always present in a
systematic review)
When NOT to do meta- analysis:
• “Garbage in- garbage out” (when they are combining
poor studies they will result in poor results)
• A meta- analysis is only as good as the studies in it
• Narrower confidence interval of combination of studies
that is worse than the biased studies of their own
• Beware of reporting biases
Steps of meta-analysis:
 Formulate the question
 Collect data
 Record data
 Analyse data
 Report the results (Forest plot)
The typical graph for displaying results of a meta-analysis is
called a “forest plot”.
If there is a forest plot--- meta-analysis is conducted
Interpretation of forest plots:
1. Headings: explain the comparisons - indicate the
question in the meta analysis
In this example: comparing the caffeinated and
decaffeinated coffee in production of headache at 24hrs
 2. List of studies included:
Eg. This meta analysis was based on 7 studies

3. Raw data for each study: can be used to calculate rates,

rates ratio etc.

Eg. Amore-coffea 2000:

Caffeine group: number of events: 2/32
Decaffeinate group: number of events: 10/34
4. The horizontal lines (with square on it) in the forest
plot represent the confidence interval: the extension of
the line indicates the CI while the box itself indicated a
 point estimate (result of tha) of the effect. Size od the
square represents the number of participants in that
study.

5. Total data for all studies:

e.g. 277 is the summation (total of all the studies)

6. Weight given to each study: based on contribution in

that forest plot based on the weight. The weight is going
to be based on the strength. The strength is based on
the sample size. So highest weight-largest sample size
7. Risk Ratio: effect estimate for each study with CI

8. Scale and direction of benefit: mathematical scale

e.g. favors caffeine and favors decaffeinated. It will show
results and groups of comparison.
 9. Pooled effect estimate for all studies with CI:

Diamond- represents the pooled effect and a total estimate

(numerical) and 95% CI- pooled (combination of all point
estimate)

Interpreting CI:
Imp- gives an idea of precision of results.
Range that is not v wide.
• Precision
-point estimate is the best guess of the effect
-CI express uncertainty- range of values we can be reasonably
sure includes the true effect
• Significance
-if the CI includes the null (absolute diff)/one (relative diff)
value
-effect cannot be confirmed or refuted by the available
evidence
-consider what level of change is clinically important

Null value can be 0 or 1: if using risk ratio null value is 1

If absolute risk reduction: 0
10. Vertical line or line of no difference:
This vertical line is at 1- it is called so as it stems at the null
value. (Risk ratio-null value at 1)
Any CI that passes through or cross the vertical line which
means that that study results is insignificant.
The diamond- it crosses the vertical line- pooled effect of that
forest plot showed statistically non-significant results.

Heterogeneity:
In language of meta-analysis:
-Homogeneity means results of each individual trial are
compatible with the results of any of the others
-Heterogeneity is variation between the studies’ results
It stems from the differences within the studies in terms of
participants, intervention, outcomes, follow up time
True variation:
(A) Clinical Diversity:
Participants
• e.g. condition, age, gender, location, study eligibility
criteria
Interventions
• intensity/dose, duration, delivery, additional
components, experience of practitioners, control
(placebo, none, standard care)
Outcomes
• follow-up duration, ways of measuring, definition of an
event, cut-off points
 (B) Methodological Studies:
design
• e.g. randomised vs non-randomised, crossover vs
parallel vs cluster randomised, length
conduct
• e.g. risk of bias (allocation concealment, blinding, etc.),
approach to analysis
(C) Statistical Heterogeneity:
there will always be (sampling) variation between the
results of different studies random
The author of the review should decide if the differences are
too large to be able to go for meta-analysis or not that much
and could go for it.
How to judge?
1. Identifying true heterogeneity:
Each separate tool is not strong enough to determine the
heterogeneity so they should be used combined.
Observed variation = “random variation” + “true variation”
Tools used:
• visual inspection of the forest plots
• chi-squared (c2) test (Q test)
• I2 statistic to “quantify” true heterogeneity
 (a) Visually:
Forest plot: do confidence intervals of studies overlap with
each other and summary effect?
If the CI overlap each other- gives an idea if results are
similar to each other or not.
(b) Statistically: 2 methods: Chi-sqaure/ I2 Test
Chi- square test for heterogeneity (Mantle Hanszel test or
Cochran Q test)
 Tests whether the individual effects are further away
from the common effect, beyond what is expected by
chance
 Has poor power
 tests the null hypothesis of homogeneity
 may detect clinically unimportant differences with
many studies
 P-value < 0.10 indicates significant heterogeneity/
Reject the null hypothesis if p< 0.10- if u reject the
null hypothesis it indicates that the test has
heterogeneity in the studies included in that forest
plot.
I2 statistics: quantifies the heterogeneity
I2 statistic describes the percentage of variability due to true
heterogeneity rather than chance (0% to 100%)
– low values indicate that (almost) all observed
variation can be explained by “random error”(low
heterogeneity)
 – high values indicate that a substantial proportion of
observed variation is due to true differences
• be cautious in interpreting (cannot be relied on alone)

Forest plot A: most studies CI are overlapping- no serious

heterogeneity
Forest plot B: not overlapping- shows heterogeneity
Shows heterogeneity

Example:
Visually: the CI overlap
X2 = 7.82 df=6 (close to df- so no heterogeneity), p= 0.25-
non-significant.
I2= 23.2%- No or little heterogeneity

Diamond shows that pooled effect is insignificant- no

statistical significant difference between ttt and control
Pooled effect: favours caffeine but results shows insignificant
difference