Nature Meta-Analysis Workshop - Print

About me…
University of Pennsylvania
Author
Peer reviewer
Academic editor
Editorial Development
Manager
1 This content is not to be shared or distributed without the expressed consent of Springer Nature © Springer Nature Limited 2019
Who is Springer Nature?
In May 2015, Nature Publishing Group, Palgrave Macmillan, and Macmillan
Education and Springer Science+Business Media merged to form Springer Nature.
Home of the world’s leading Springer publishes One of the world’s leading
international science academic journals and open access publishers,
journal, Nature (founded in books in all areas of founded in 2000, BMC
1869), Nature Genetics, academic research now publishes around
Nature Cell Biology, Nature including fundamental 300 scientific journals in
Materials, Nature Physics, science, engineering, biology, medicine, and
Nature Communications, mathematics, humanities, beyond, including
Scientific Reports, Scientific social science, economics Genome Biology, BMC
Data and more. and more. Biology & BMC Medicine.
We publish ~3000 journals

What are editors looking for?
• Suitability for their journal
• Reliability of the results
• Reproducibility of the study
• Clinical relevance for the field
Today’s agenda
Definitions Presenting data
Searching for
Analyzing data
studies
Extracting data Rating the quality
Increase your chance of publication and

maximize your impact
1.0
Introduction
Review types
Meta-analyses
Systematic reviews
Literature reviews
Systematic review Literature review
Goals Answer a specific question Summary of a topic
Authors >3 >1
Timeline Months to years Weeks to months
Many databases One database
Search
White + grey literatures White literature
Value Evidence-based medicine Overview of the literature
Modified from Kysh L. (2013): Difference between a systematic review and a literature review.
[figshare]. Available at: http://dx.doi.org/10.6084/m9.figshare.766364 [CC BY 4.0]
Review types
Abelman D.D. (2017) Mitigating risks of students use of study drugs through understanding
motivations for use and applying harm reduction theory: a literature review. Harm Reduction
Journal 14(1):68
Revesz D. et al. (2017) Decision support systems for incurable non-small cell lung cancer: a
systematic review BMC Medical Informatics and Decision Making 17:144
De Rosa S. et al. (2017) Long term outcomes of coronary artery bypass grafting versus
stent-PCI for unprotected left main disease: a meta-analysis BMC Cardiovascular Disorders
17:240
History
1955
1st “Meta-analysis” evaluating the placebo effect
(effective in 35% of patients)
1976
1st time the term “meta-analysis” is used by the
psychologist Dr. G. Glass
1992
Cochrane Collaboration is born,
promoting the use of meta-analysis in clinical practice
Cochrane Collaboration
Dr. Archie Cochrane

(1909–1988)
British epidemiologist who

promoted the use of RCTs
for better informed
healthcare
Father of
evidence-based medicine
By Cardiff University Library, Cochrane Archive,
University Hospital Llandough [CC BY 3.0], via
Wikimedia Commons
Cochrane Collaboration
Excellent source for systematic reviews
http://handbook.cochrane.org/
Cochrane Interactive Learning
Excellent source for systematic reviews
https://training.cochrane.org/interactivelearning
Levels of clinical relevance
Secondary Meta-analyses
data Systematic reviews
RCTs
Cohort studies
Primary
data Case-control studies
Cross-sectional studies
Case series / Case reports
Epidemiology of results
Unit of analysis?
Patients
Studies
Epidemiology of results
Studies have different weights
Studies
Systematic reviews
Synthesis of empirical evidence from pre-specified eligibility

criteria to address a specific research question
Usually about the efficacy of a treatment
Systematic review Meta-analysis

Narrative synthesis Statistical synthesis
Systematic reviews
Why would you want to conduct them?
Because they are easy!

Other people did all the work…
Systematic reviews
Good systematic reviews are a lot of work!
Often take 9–24 months to conduct
When should they be done?

• Important research question
• Several empirical studies have been done
• No conclusive results
• To avoid further unnecessary studies
Usefulness of meta-analyses
Meta-analysis demonstrated the clinical efficacy of ACE
inhibitors for nondiabetic renal failure after 10 unclear RCTs
Giatras et al. Ann Intern Med. 1997; 127: 337-345.
Avoid research waste
Lancet 2009; 374: 86–89
~85% of biomedical research is waste

• Not addressing relevant questions
• Incomplete review of the literature
• Inappropriate methodology
• Incomplete reporting
• Unpublished
Systematic review of 28 clinical trials

(7521 patients): are calcium antagonists
effective for acute ischemic stroke?
“no difference between people

given calcium antagonists and
those who were not, in terms
of death or disability”
Horn & Limburg. Cochrane Database Syst Rev. 2000; 1: CD001928.
Did animal studies show calcium antagonists were
effective in treating acute ischemic stroke?
“We conclude that the results of the animal

experiments reviewed in the present investigation did
not show convincing empirical evidence to
substantiate the decision for trials with
nimodipine in stroke patients.”
(20 studies)
Horn et al. Stroke. 2001; 32: 2433–2438.
Systematic reviews
Useful for clinicians to conduct
• Flexible time (can go in and out of the review when

you have time)
• Affordable (funding can always be an issue)
• Learn how to evaluate the quality of study and

evidence (improve your skills as a clinical researcher)
Systematic reviews
Advantages Disadvantages
▪ Appropriate for highly ▪ Higher risk of bias

heterogeneous studies ▪ Subjective approach
▪ Conclusion often in line ▪ Does not consider
with the author’s field sample size, effect size
▪ Affordable and study design
Meta-analysis
Advantages Disadvantages
▪ Long
▪ Objective approach ▪ Data gathering can be
▪ Increase statistical power by challenging!
pooling samples together ▪ Risk of publication bias
▪ Increase confidence about the ▪ Not appropriate for
conclusion heterogeneous studies
▪ High clinical relevance ▪ Shows general
▪ Affordable recommendation, not ideal for
a specific patient
Meta-analysis
Writing the research question

PICOS, FINER
Identifying previous studies

Screen the grey and white literature
Assessing the quality of evidences

Both at study-level and outcome-level
Summarizing the evidences

Forest plot, Summary of finding (SoF) tables
Interpreting the results

Answering the question, addressing limitations
Formulating your research question
P Population: who does it relate to?
Intervention: which treatment? What dosage?

I Use the TIDieR checklist
C Comparison: what is the control?
O Outcome: what are you measuring?
S Study design: what type of study are you selecting?
Good example
In adult patients undergoing orthopedic surgery,

does treatment with oral analgesic tramadol
compared with subcutaneous morphine lead to
lower pain scores and fewer side effects (vomiting,
constipation, drowsiness) as assessed by
randomized controlled trials?
Bad example
Do mentally stimulating activities (e.g. cross-word

puzzles) reduce the relative risk of developing
dementia?
older adults (>50 y)
compared with no activity
cohort studies
Bad example - corrected
In older adults (> 50 y), do mentally stimulating

activities (e.g. cross-word puzzles) compared with
no activity, reduce the relative risk of developing
dementia as assessed in cohort studies?
How to define a research question? FINER
F Feasible: Do you have the expertise, budget & resources?
Interesting: Does the study address an important problem

I in the field now?
N Novel: Are the findings new or simply derivative?
Ethical: Are you respecting international guidelines?

E Did you obtain ethics committee approval?
R Relevant: Is the study clinically relevant?

Hulley SB et al., Designing clinical research. 3rd ed.
Philadelphia, PA: Lippincott Williams & Wilkins; 2007.
Register your study before you begin
PROSPERO
http://www.crd.york.ac.uk/PROSPERO/
Register your study before you begin
http://www.crd.york.ac.uk/PROSPERO/
Avoid duplication
Remember to search PROSPERO before

you begin to ensure someone else is not
currently conducting a meta-analysis or
a systematic review on your topic
34
Questions?
35
Activity 1
Evidence-based medicine
Please turn to page 10 of your workbook
Activity 1
Two orthopedists discuss the recent advances in the treatment of osteoarthritis
of the ankle. Among the well-known studies in the field, which one should guide
healthcare practitioners in their decision making?
A. The conclusions of a recent randomized clinical trial (RCT) published in

2016 and performed on 70 Egyptian patients.
B. The latest systematic review from a famous orthopedist surgeon published

in 2011.
C. The conclusions of a Cochrane meta-analysis published in 2015

summarizing 6 randomized controlled trials on 240 patients.
Better option
Activity 1
A. The conclusions of a recent randomized clinical trial (RCT) published in

2016 and performed on 70 Egyptian patients.
Population of interest Most recent study
Lower
Only one study!
clinical relevance
Activity 1
B. The latest systematic review from a famous orthopedist surgeon published

in 2011.
Higher clinical
Oldest study
relevance than RCT
Risk of subjective bias Narrative synthesis
Activity 1
C. The conclusions of a Cochrane meta-analysis published in 2015

summarizing 6 randomized controlled trials on 240 patients.
Highest
clinical relevance
Cochrane standards Statistical synthesis
2.0
Searching for studies
How to select studies?
Define inclusion / exclusion criteria
• Published or unpublished?
• Which languages?
• Which databases?
Define rationale for each criterion
Designing a search strategy
P Population Ask help from a librarian!
I Intervention Text words

+
C Comparison Controlled
vocabulary
(e.g. MeSH terms)
O Outcome
S Study design
Use synonyms, alternative or international
Synonyms spellings and related terms
E.g.: flavor, taste
Boolean Use AND, OR, NOT between terms to

narrow down your results
operators E.g.: fracture AND child, flavor OR taste
AND OR NOT
Use synonyms, alternative or international
Synonyms spellings and related terms
E.g.: flavor, taste
Boolean Use AND, OR, NOT between terms to

narrow down your results
operators E.g.: fracture AND child, flavor OR taste
Use symbols (*,$ or ?) to include all

Truncation declinations of a root word
E.g.: child* = child + children + childhood
Use NEAR, NEXT or ADJ to search for

Proximity
terms that need to appear together
operators E.g.: fracture ADJ foot = fracture of the foot
Designing a search strategy: example
Efficacy of Q10 in preventing heart failure in hypertensive adults
Hypertensive adults
P MESH: adult AND hypertension
Text words: hypertens* OR (High ADJ blood ADJ pressure)
Coenzyme Q10
I MESH: ubiquinone
Text words: ubiquinol-10 OR Q10 OR co-enzyme Q10 OR bio-quinone
Q10 OR CoQ10 OR 2,3-dimethoxy-5-methyl-6-decaprenylbenzoquinone
Randomized Controlled Trial

S MESH: therapeutic human experimentation AND randomized controlled trial
OR drug therapy NOT animal experimentation
Filter: Controlled trial OR controlled clinical trial
Text words: random* OR trial
Designing a search strategy: example
https://www.ncbi.nlm.nih.gov/pubmed/advanced
Published or unpublished?
White literature Grey literature
Clinical trial registries
Cochrane Library
https://www.cochranelibrary.com/central/about-central
Cochrane Library
Cochrane Library
AdisInsight
Focused on pharmacological interventions
https://adisinsight.springer.com/
AdisInsight
Focused on pharmacological interventions
Contacting authors
Contacting authors
How many researchers named

Mohammed Ali exist?
Selecting studies using the flow chart
PRISMA
flowchart
http://www.prisma-statement.org/
Systematic reviews/meta-analyses
629 identified records Should clearly
• Medline: 356 explain how they
• Embase: 225
• Cinahl: 14
identified studies
• CENTRAL: 14
• CDR database: 7
• Clinical trial registries: 11
• Other: 2
20 articles excluded as follows:
• 7 were not RCTs
• 5 not meet inclusion criteria
44 full-text articles analyzed • 4 were reviews
for eligibility • 1 no answer from authors
• 3 RCTs still ongoing
25 articles included 5 RCTs included in quantitative

6 primary RCTs analysis (meta-analysis)
La Mantia et al. Cochrane Database of Systematic Reviews. 2016; 11: CD009333.
Flow chart: example
Question: What are the clinical characteristics and geographical
variations in prevalence of severe vivax malaria since 1900?
Search strategy:
Medline Scopus
Databases: Medline + Scopus
274 studies 412 studies
Language: English
Period: 1900-2014
Medline includes work 686 studies

from >1946!
Modified from Rahimi et al., (2014) Severe vivax malaria: a
systematic review and meta-analysis of clinical studies since
1900 Malaria Journal 13:481 https://doi.org/10.1186/1475-
Before 1946: Reference list, books, etc. 2875-13-481
Flow chart: example
Medline Scopus
274 studies 412 studies
289
686 studies
duplicated
Modified from Rahimi et al., (2014) Severe

vivax malaria: a systematic review and meta-
analysis of clinical studies since 1900 Malaria
397 studies
Journal 13:481
https://doi.org/10.1186/1475-2875-13-481
Flow chart: example
397 studies 278 ineligible studies

• 89 not related topic
• 71 case reports/series
• 41 non-English
• 31 review
63 studies with no • 23 non-vivax malaria
severe vivax • 8 with comorbidities
• 5 mixed infections
• 5 non-human
• 3 guidelines
Modified from Rahimi et al., (2014) Severe • 1 comment
57 studies • 1 report
Journal 13:481
https://doi.org/10.1186/1475-2875-13-481
Flow chart: example
57 studies
21 studies < 1946
Modified from Rahimi et al., (2014) Severe

77 studies Journal 13:481
https://doi.org/10.1186/1475-2875-13-481
Flow chart: example
Rahimi et al., (2014) Severe vivax malaria: a

systematic review and meta-analysis of clinical
studies since 1900
Malaria Journal 13:481
https://doi.org/10.1186/1475-2875-13-481
62
Questions?
63
Activity 2
Selecting studies
Activity 2
During your literature search, you identified 324 RCTs in Medline and Web of
Science databases together with 4 unpublished studies from conference
proceedings. Although all published RCTs possess an English abstract, 34 of
them are written in either French (21), Spanish (8) or Japanese (5). How will you
narrow down the final studies to include in your meta-analysis?
A. Since no researcher in my team speaks French, Spanish or Japanese,

remove the 34 foreign-language studies from the 324 and analyze the
resulting 290.
B. Remove all duplicated studies from the 324 RCTs. Consider adding 4
unpublished studies if all necessary data and methodology are available.
Keep the 21 French studies that can be understood by my 2 French-
speaking colleagues. Remove the 8 Spanish and 5 Japanese studies.
C. Remove all duplicated studies from the 324 RCTs. Add 4 unpublished
studies if all necessary data is available. Keep 34 foreign-language studies
since the abstracts are written in English.
Activity 2
During your literature search, you identified 324 RCT in Medline and Web of
proceedings. Although all published RCT possess an English abstract, 34 of
A. Since no researcher in my team speaks French, Spanish or Japanese,

remove the 34 foreign-language studies from the 324 and analyze the
resulting 290.
What about duplicates?
Activity 2
B. Remove all duplicated studies from the 324 RCT. Consider adding 4
unpublished studies if all necessary data and methodology are available.
Keep the 21 French studies that can be understood by my French-speaking
colleagues. Remove the 8 Spanish and 5 Japanese studies.
Minimum of 2 French-
Best approach
speaking reviewers
Activity 2
C. Remove all duplicated studies from the 324 RCT. Add 4 unpublished
studies if all necessary data is available. Keep 34 foreign-language studies
since the abstracts are written in English.
Unpublished Critical appraisal

detailed methodology? of non-English studies?
3.0
Extracting data
Free software
• Screen abstracts & full texts

• Data collection
• Risk of bias assessment
• Reach consensus
www.covidence.org
• Export in RevMan or Excel
Plot Digitizer
• Extract data directly from
graphs
• Read .gif,.jpeg and .png
formats
http://plotdigitizer.sourceforge.net/
Data extraction form
Data extraction form
2 independent observers
Blind to authors &

institutions
http://www.biomedcentral.com/content/supp
lementary/1471-2288-2-10-S3.pdf
Common measurements
Risk Ratio (RR) Odds Ratio (OR)
Absolute risk Number Needed

reduction (ARR) to Treat (NNT)
Standardized format
To compare studies, express outcomes in
a standardized format
Single study Meta-analysis
Continuous Standardized mean

outcomes Mean difference
difference
Mean difference Standardized mean

= difference
Standard deviation
Or
Cohen’s d
Standardized format
To compare studies, express outcomes in
a standardized format
Single study Meta-analysis
Continuous Standardized mean

outcomes Mean difference
difference
Binary
outcomes Risk Ratio Risk Ratio
Weight
To compare effect sizes between studies,
attribute a weight to each study
Effect size Weight
Mean Standardized Sample

+
difference mean difference size
Inverse
Risk Ratio Risk Ratio +
variance
Confidence Intervals
CI is related to the precision of measured mean,

(standard error of the mean, SEM)
• SD = variability of the data within the sample

• SEM = precision of the mean for the population
SEM = SD/√n
95% CI = 2 SEM
Precision & accuracy
High precision High precision

High accuracy Low accuracy
Low precision Low precision

High accuracy Low accuracy
80
Questions?
4.0
Presenting your data
Free software
• Official software for

Cochrane reviewers
• Import Covidence data
• Perform meta-analysis
• Creates plots
http://tech.cochrane.org/revman
• Generates summary of
finding (Sof) tables
Forest plot
Studies Weight RR [95% CI]
Gagnon et al., 2015 20% 4.50 [1.15-7.85]
Robens et al., 2014 30% 5.40 [3.29-7.51]
Heterogeneity
Newton et al., 2012 15% 1.29 [0.04-9.54]
Giuliani et al., 2013 20% 3.04 [0.98-7.63]

15% 1.54 [0.73-7.71]
Pavri et al., 2010
100% 3.27 [1.42-5.62]
Total
Protective effect Harmful effect
0.1 1 10
Visual summary of
Risk Ratio [log scale] evidences
Effect measure
Forest plot
Studies
Gagnon et al., 1998 Cumulative
summary of
+ Robens et al., 1999
evidences
+ Newton et al., 2000
+ Giuliani et al., 2001
+ Pavri et al., 2001
+ Reyes et al., 2002
+ Ali et al., 2004
+ Kotani et al., 2005 Clear effect
+ Ballout et al., 2006
0.1 1 10
86 Risk Ratio [log scale]© Springer Nature Limited 2019
This content is not to be shared or distributed without the expressed consent of Springer Nature
Forest plot
Studies
Gagnon et al., 1998
Cumulative
+ Robens et al., 1999 summary of
+ Newton et al., 2000 evidences
+ Giuliani et al., 2001
+ Pavri et al., 2001
+ Reyes et al., 2002 Clear effect
+ Ali et al., 2004
+ Kotani et al., 2005 Research waste
+ Ballout et al., 2006
0.1 1 10
87 Risk Ratio [log scale]
This content is not to be shared or distributed without the expressed consent of Springer Nature
© Springer Nature Limited 2019
Funnel plot
95% limit lines

Summary
estimate How is the symmetry
Precision of effect of the distribution?
Standard
Error
Visual estimate
Egger’s test
p > 0.05
Risk Ratio [log scale] Low risk of bias
Effect estimate
Funnel plot
95% limit lines
Precision How is the symmetry

of the distribution?
Standard
Error
Not useful if
< 10 studies!
Risk Ratio [log scale]

Effect estimate
Funnel plot
95% limit lines
Precision How is the symmetry

of the distribution?
Standard
Error

Effect estimate
Funnel plot
95% limit lines
Asymmetrical
Precision
Standard Risk of
Error
publication bias?
Unpublished
data?

Effect estimate
Trim-and-fill method
Reduced risk of
publication bias
Imputed studies
Phan et al. (2015). Systematic review and meta-analysis: techniques and a guide for the academic surgeon Ann
Cartdiothorac Surg 4(2):112-122
Contour-enhanced funnel plot
Help to identify publication bias by adding contours of

significance (e.g., p = 0.001, 0.01, 0.05, etc.
Studies missing in areas Asymmetry likely due to

of non-significance publication bias
Studies missing in areas Asymmetry likely due to

of significance other factors
p < 0.01
0.01 < p < 0.05
Precision
0.05 < p < 0.10
Standard p > 0.10
Error
Asymmetrical
Non-significant
Risk Ratio [log scale] Bias due to

Effect estimate other causes
p < 0,01
0,01 < p < 0,05
Precision
0,05 < p < 0,10
Standard p > 0,10
Error
Asymmetrical
Non-significant
Risk Ratio [log scale] Publication

Effect estimate bias!
p < 0,01
0,01 < p < 0,05
Precision
0,05 < p < 0,10
Standard p > 0,10
Error
Tunnel effect
Non-significant
Risk Ratio [log scale] Publication

Effect estimate bias!
101
Activity 3
Funnel plot
Please turn to pages 10-11 of your workbook
Activity 3
When reading a published article, the authors state, “The shape of the
funnel plot showed symmetry, suggesting no evidence of publication
bias.” Based on the presented funnel plot, do you agree with the
authors? If no, why not?
Funnel plot with 95% confidence limits

Standard Error
Risk Ratio
Activity 3
When reading a published article, the authors state, “The shape of the
funnel plot showed symmetry, suggesting no evidence of publication
bias.” Based on the presented funnel plot, do you agree with the
authors? If no, why not?
Funnel plot with 95% confidence limits

Standard Error
4 studies!
2
4
Not appropriate for
1 funnel plot or evaluating
3 publication bias
Risk Ratio
104
Questions?
Break – 30 min
5.0
Analyzing data
Systematic review or meta-analysis?
Systematic reviews are easier, but meta-analyses are stronger
Homogenous Meta-analysis preferred for more

population precise estimate of the effect
Heterogeneous Meta-analysis may not be

population appropriate
Does it make sense to pool the results?
If not, a narrative review is better
Heterogeneity
Different patients, interventions,

Clinical outcomes, regions
• Systematic review or meta-analysis?

• Fixed-effect or random-effects model
Do the results among trials vary more

Statistical than would be expected by chance
• Quantifying heterogeneity (I2 statistic)

• Investigating heterogeneity (meta-regression)
Homogenous sample
What is the prognosis of pancreatic cancer patients?

Egyptian patients are classified in 5 groups according to their name:
A-E F-J K-O P-T U-Z

3.8 y 3.6 y
3.5 y 3.3 y 3.2 y
Fixed effects model
Meta-analysis models
Fixed effects Not common for
model most meta-analyses
• Common effect across studies

• Variability is due to within-study variance only
• More powerful
• Narrow confidence interval
Conditional inference
The conclusion is limited to the samples of studies
Heterogeneous sample
What is the prognosis of pancreatic cancer patients?

Patients are classified in 5 groups according to their origin:
USA France China Egypt India
4.2 y 0.8 y
3.5 y 2.1 y 2.4 y
Random effects model

Random effects More common for
model most meta-analyses
• Different effects across studies

• Variability is due to within-study & between-study
variances
• Less powerful
• Larger confidence interval
Unconditional inference
The conclusion can be generalized to the
population of similar studies
Is choosing the right model problematic in the literature?
Reviewing 60 published
meta-analyses with highly
heterogeneous samples
(I2>50%)
It’s getting better…
Schroll et al. BMC Medical Research Methodology 2011, 11:22
Visually assessing statistical heterogeneity
Is the dotted summary line crossing all

point estimates’ confidence intervals?
Yes No
0.1 1 10 0.1 1 10
Homogenous Heterogeneous
Identifying statistical heterogeneity
Cochran’s Q-test
Is heterogeneity present?
Within-study
variability
0.1 1 10 0.1 1 10
P > 0.05 P < 0.05
Homogenous Heterogeneous
Quantifying statistical heterogeneity
I2 statistic
How much heterogeneity?
Within-study I2 (%)
variability
Between-
study variability
MILD MODERATE MARKED
0–30% 31-50% > 50%
Investigating statistical heterogeneity
May be useful to identify why there is heterogeneity
Examine the impact of variables on

Meta-regression study effect size using regression-
based techniques
Understanding why there are differences can

generate new hypotheses to investigate
“…the type of tumor is the reason for

the formation of heterogeneity.”
“…the type of tumor is the reason for the formation of heterogeneity.”

“We conducted subgroup analyses for the expression of circRNA in
different types of tumors.”
Sub-group analysis
Evaluates the consistency of the results between

multiple groups
Useful for generating hypotheses

What factors to consider?
Sociodemographics
Comorbidity Gender
Sub-group analysis: example
Phan et al. (2015). Systematic review and meta-analysis: techniques and a guide for the academic surgeon
Ann Cartdiothorac Surg 4(2):112-122
126
Questions?
127
Activity 4
Data analysis
Activity 4
You decide to evaluate the efficacy and safety of sorafenib combined with
transarterial chemoembolization (TACE) in treating hepatocellular carcinoma
based on previous studies. After selecting 11 potential studies, you notice
differences in how TACE was performed — 6 studies used conventional TACE
with lipiodol, while 5 studies used DEB-TACE with drug-eluting beads. Further,
you also notice differences in sorafenib treatment across the studies as well.
What would be the best option to evaluate these studies in your analysis?
A. Decide to only conduct a systematic review (qualitative synthesis).

B. As they are all comparing similar treatments with the same outcomes, it is
fine to pool the data and conduct a meta-analysis with a fixed-effect model.
C. Evaluate heterogeneity using the Q-statistic and I2 index. If heterogeneity is
high (I2 > 50%), then use a random-effects model to conduct the meta-
analysis.
Activity 4
A. Decide to only conduct a systematic review (qualitative
synthesis).
Systematic review can always be done,

regardless of heterogeneity
Lacks the usefulness of quantitative

(statistical) synthesis
Better for results that cannot be pooled for a

quantitative synthesis
Activity 4
B. As they are all comparing similar treatments with the same

outcomes, it is fine to pool the data and conduct a meta-
analysis with a fixed-effect model.
Fixed-effect model only for homogenous studies

(e.g., same treatments with similar patients)
Always calculate heterogeneity to justify using

fixed-effect model (never assume!)
Activity 4
C. Evaluate heterogeneity using the Q-statistic and I2 index. If
heterogeneity is high (I2 > 50%), then use a random-
effects model to conduct the meta-analysis.
Common mistake
• Random-effects model (correct!) is based on
clinical heterogeneity, not statistical
• Consider sub-group analyses (TACE vs. DEB-TACE)

• Discuss heterogeneity in Discussion section
6.0
Rating the quality of evidences
Assessing data quality
Outcome A
Study 1 Outcome B
Outcome C
Study Outcome A Quality of

limitations Study 2 evidences
Outcome B
Surrogate
Study 3 Outcome A
Outcome C
Assessing study limitations
Cochrane risk of bias

evaluation tool
Risk of bias
Study 1 + - + + + + +
+ ? + + + + ? + Low
Study 2
Study 3 + ? + + - + + ? Unclear
Study 4 + - + + - + - - High
Study 5 + - + + - + +
Performance
Attrition Other
Selection Detection
Reporting
Assessing study limitations
Cochrane risk of bias graph
Random sequence allocation
Allocation concealment
Blinding of participants & personnel
Blinding of outcome assessor
Incomplete outcome data
Selective outcome reporting
Other bias
0% 20% 40% 60% 80% 100%
Risk of bias Low Unclear High
Assessing risk of selection bias
Random sequence allocation Risk of bias?
1. Participants were randomized into 2 groups - High

based on their medical record number.
2. Participants were randomized into 2 groups + Low

by tossing a coin.
3. Participants were randomized into 2 ? Unclear

groups.
Allocation concealment
blinding vs concealment ?
Sequence
generation
Assignment
Study
Concealment Blinding
Always Not always

possible! possible!
CC-BY 0
Allocation concealment Risk of bias?
1. Treatment allocation was performed

+ Low
using a centralized telephone system.
2. Participants were allocated to one of the

two treatments using envelopes. ? Unclear
3. Participants were allocated to treatment

A if their birth date was an odd number, - High
and to treatment B if their birth date was
an even number.
Assessing risk of performance bias
Blinding of participants and personnel Risk of bias?
1. Participants and investigators were blind

to the treatment allocation. […] The drug
- High
produced important side effects
compared with the placebo.
2. We used a double blinding model for this

RCT where the two drugs were matched + Low
in size, color, weight and flavor.
3. Patients were randomized to each ? Unclear

intervention.
Assessing risk of detection bias
Blinding of outcome assessment Risk of bias?

1. Based on their medical chart number, patients
were randomized to the treatment (even number) - High
or placebo (odd number) groups. […] Two
independent clinicians rated each outcome.
2. In this RCT, we used a double-blind procedure. ? Unclear
3. Both Alzheimer patients and clinicians were blind

to treatment (drug vs placebo) allocation and the
tablets shared similar sensory features. Two + Low
experienced neurologists assessed the brain
scans. The patient’s treatment was revealed at
the end of the study by opening the sealed
envelopes detailing the randomization sequence.
Assessing risk of attrition bias
Incomplete outcome data Risk of bias?
1. A total of 1375 participants completed the

? Unclear
study.
2. As illustrated in the flow chart, among the

463 enrolled participants, 461 were
randomized. Ten participants (2%) did not + Low
complete the study. Therefore, the data of
451 participants were analyzed.
3. The attrition rate was 23%. - High
Assessing risk of reporting bias
Selective outcome reporting Risk of bias?

1. In this RCT testing the efficacy of an hypertensive
drug, our primary and secondary outcomes are the
differences in systolic (SBP) and diastolic (DBP) + Low
blood pressures from baseline. […] The reduction
in SBP and DBP were 7 and 8 mmHg for the
hypertensive drug group (p < 0.05) as well as 2
and 3 mmHg for the placebo group (p > 0.05).
2. In this dietary study, our outcomes of interest were

weight loss, BMI, % body fat and fat mass. […]
Participants significantly lost weight (from 106 to - High
101 kg; p < 0.05) and reduced their BMI (from 32.8
to 31.9, p < 0.05). However, there were no
significant decreases in % body fat or fat mass.
Assessing risk of reporting bias
Selective outcome reporting Risk of bias?
3. In this RCT, we measured the efficacy of

a new analgesic compared to morphine.
[…] The pain scores were similar
? Unclear
between groups. The main side effects
were comparable and included dizziness
and nauseas for both analgesics.
Assessing data quality
Grading of Recommendations,
Assessment, Development and Evaluation
1. Risk of bias: study limitations

2. Heterogeneity: inconsistency of results
3. Indirectness of evidence (PICO)
4. Imprecision (confidence interval)
5. Publication bias
6. Other factors
145
Questions?
146
Activity 5
Assessing risk of bias
Please turn to pages 11-15 of your workbook
Due to time constraints, please do at home. The answers are

available in the slides
156
8.2
Writing the manuscript
Guidelines
QUOROM statement
(1999)
(2009)
Title
Specify type of review in the title
Umer A. et al. (2017Childhood obesity and adult cardiovascular disease risk factors: a systematic review with meta-
analysis BMC Public Health 17:683
Abstract
Umer A. et al.
(2017Childhood obesity and
adult cardiovascular
disease risk factors: a
systematic review with
meta-analysis BMC Public
Health 17:683
Authorship
Definition from ICMJE

“Substantial contributions to the conception or design of the work;
OR the acquisition, analysis, or interpretation of data for the work”
+
“Drafting the work or revising it critically for important intellectual
content”
+
“Final approval of the version to be published”
+
“Agreement to be accountable for all aspects of the work”
http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-
and-contributors.html
Introduction
Introduce the topic
What is known
What is not known

Aims
Keep it brief
PICOS
Introduction
Why does the study need to be done?
Research problem
Whether the incidence of avascular necrosis (AVN) is less after operative fixation versus
conservative treatment for proximal humeral fractures is controversial. In a multicenter
analysis reported by Schai et al., open reduction and internal fixation was associated
with significantly less AVN... By contrast, Fjalestad and coworkers reported that
surgical treatment of displaced proximal humeral fractures was associated with
markedly more AVN...
Study objectives
There is a need to gather the current best evidence of AVN incidence in patients with
proximal humeral fractures in regard to operative fixation versus conservative treatment.
Therefore, we conducted a systematic review of published and unpublished research
and applied a meta-analysis to integrate the results quantitatively.
Modified from: Xu et al. J Orthopaedic Surg Res. 2014; 9: 31.
Methods
Clearly explain exactly how your studies were

chosen and analyzed
Inclusion & • Patients, interventions, outcomes

exclusion criteria • Language/year restrictions
• Databases, keywords, grey literature

Study identification • Funnel plot, Egger’s test
• Patient characteristics & outcomes

Data extraction ✓ Two independent reviewers
Methods
Clearly explain exactly how your studies were
chosen and analyzed
Quality • Use EQUATOR checklist

assessment • Quality/risk assessment tools
Methods
Cochrane Collaboration tool for assessing risk of bias
http://handbook.cochrane.org/chapter_8/table_8_5_a_the_cochrane_collaborations_tool_for_assessing.htm
Methods
Clearly explain exactly how your studies

were chosen and analyzed
Quality • Use EQUATOR checklists

assessment • Quality/risk assessment tools
• E.g. calculate odds ratio for each

outcome, 95% CI, and P-value
Data analysis
• Calculate heterogeneity (I2 value)
• Subgroup & sensitivity analyses
Results
Logically present your findings
Results
• Identified studies & how many

Search results included/excluded from analysis
✓ Use PRISMA flowchart
Results
PRISMA
flowchart
http://www.prisma-statement.org/
Results
• Identified studies & how many

Search results included/excluded from analysis
✓ Use PRISMA flowchart
Study • Patient characteristics

• Intervention/outcome characteristics
characteristics ✓ Summarize in tables
Results
• Outcomes from individual

Outcome analysis studies and meta-analysis
✓ Present in forest plot
For meta-analyses, state if you used a

fixed effects or random effects model
Results
Forest plots
• Low heterogeneity (I2 = 0%)

• Authors used both fixed and
random effects models
Ide et al. BMC Public Health. 2016; 16:396
Results
• Present the study quality

Quality/risk
• Median quality score
assessment • Cochrane quality table
Example of Cochrane quality table

examining therapeutic efficacy of
antiviral drugs in HIV patients
Higgins et al. BMJ 2011;343:d5928.
Discussion
Summary of findings
Reliability of findings
Applicability of findings
Implications Implications of findings
Discussion
Summary of your findings
Main conclusion
Our pooled analysis of 20 RCTs addressing use of GLP-1 agonists for type 2 diabetes
found moderate quality evidence suggesting no increase in heart failure.
Relationship to other studies
A recent meta-analysis of RCTs found that GLP-1 agonists were associated with a
modest reduction in blood pressure and a slight increase in heart rate. These studies
suggest that GLP-1 agonists might even reduce the incidence of heart failure. Though
results of RCTs fail to show this decrease, confidence intervals do not exclude the
possibility of a modest reduction.
Modified from: Li et al. BMC Cardiovasc Disord. 2016; 16: 91.
Discussion
Reliability of your findings – Limitations
Quality of included studies

• Size of included studies
• Size and significance of observed effects
• Consistency of effects across studies
• Potential biases
Strengths/weaknesses of review method

• Find published & unpublished studies
• Not include non-English/Arabic studies
Discussion
Strengths and limitations
Strengths
The present analysis has several important strengths. First, it is based on a large
sample of patients… Second, each included study was of high quality according to the
Cochrane Collaboration tool… Third, there was no evidence of publication bias; the
funnel plot was symmetrical and the Egger’s test was not statistically significant.
Limitations
Nevertheless, our meta-analysis does have certain limitations. Some of the studies
were at risk for reporting bias, but excluding these studies did not affect the pooled
estimates. Further, the results are dominated by three studies that account for more
than 80% of the pooled results. However, sensitivity analyses without these studies did
not change the results of the meta-analysis (data not shown).
Implications
Modified from: Salvo et al. BMJ. 2016; 353: i2231.
Discussion
Applicability of your findings
Main conclusion
Our meta-analysis demonstrates that anaesthesia contributes disproportionately to

maternal mortality in low-income and middle-income countries.
Regional limitations/variations of applicability
In high-income countries such as the United States, no measured differences were

recorded in anaesthetic complications between physician and non-physician
anaesthetists. However, compared with the rigorous additional training provided to non-
physician anaesthetists in the United States, their counterparts in low-income and
middle-income countries have very little training, which varies between countries.
Modified from: Sobhy et al. Lancet Glob Health. 2016; 4: e320–327.
Discussion
Implications of your findings –
What you want your readers to remember
The present study suggests that GDF-15 is a strong predictor of

Conclusion
mortality and recurrent myocardial infarction in acute coronary
syndrome prognosis. Thus, it has the potential to become a
clinically useful novel biomarker that can provide independent Implications
prognostic information and also help to direct optimal treatment
strategies. Current ongoing clinical trials are needed to further
clarify the benefits of GDF-15 in predicting the prognoses of Future
patients with acute coronary syndrome. directions
Zhang et al. BMC Cardiovasc Disord. 2016; 16: 82.
Logically linking your ideas
Answer the four key questions for your reader
Introduce topic
Currently published studies

Why this study
needs to be done Problem in the field
Objectives
What you did Methodology
What you found Results and figures
Summary of findings
How your study
will advance Strengths & limitations
the field
Implications for the field
Logically link your ideas throughout your manuscript

Today we discussed…
Definitions Presenting data
Searching for
Analyzing data
studies
Extracting data Rating the quality
Increase your chance of publication and

maximize your impact
Thank you!
Any questions?
These slides can be downloaded at:
http://bit.ly/ClinicalMethodology_Jan2019
Dr. Jeffrey Robens

Editorial Development Manager
jeffrey.robens@nature.com

Nature Meta-Analysis Workshop - Print

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Nature Meta-Analysis Workshop - Print

Uploaded by

Copyright:

Available Formats

About me…

We publish ~3000 journals

• Suitability for their journal

• Reliability of the results

• Reproducibility of the study

• Clinical relevance for the field

Definitions Presenting data

Extracting data Rating the quality

Increase your chance of publication and

Dr. Archie Cochrane

British epidemiologist who

Excellent source for systematic reviews

Excellent source for systematic reviews

Case series / Case reports

Synthesis of empirical evidence from pre-specified eligibility

Usually about the efficacy of a treatment

Systematic review Meta-analysis

Why would you want to conduct them?

Because they are easy!

Good systematic reviews are a lot of work!

Often take 9–24 months to conduct

When should they be done?

Giatras et al. Ann Intern Med. 1997; 127: 337-345.

Lancet 2009; 374: 86–89

~85% of biomedical research is waste

Systematic review of 28 clinical trials

“no difference between people

“We conclude that the results of the animal

Horn et al. Stroke. 2001; 32: 2433–2438.

Useful for clinicians to conduct

• Flexible time (can go in and out of the review when

• Affordable (funding can always be an issue)

• Learn how to evaluate the quality of study and

▪ Appropriate for highly ▪ Higher risk of bias

Writing the research question

Identifying previous studies

Assessing the quality of evidences

Summarizing the evidences

Interpreting the results

P Population: who does it relate to?

Intervention: which treatment? What dosage?

C Comparison: what is the control?

O Outcome: what are you measuring?

S Study design: what type of study are you selecting?

In adult patients undergoing orthopedic surgery,

Do mentally stimulating activities (e.g. cross-word

older adults (>50 y)

compared with no activity

In older adults (> 50 y), do mentally stimulating

F Feasible: Do you have the expertise, budget & resources?

Interesting: Does the study address an important problem

N Novel: Are the findings new or simply derivative?

Ethical: Are you respecting international guidelines?

R Relevant: Is the study clinically relevant?

Remember to search PROSPERO before

A. The conclusions of a recent randomized clinical trial (RCT) published in

B. The latest systematic review from a famous orthopedist surgeon published

C. The conclusions of a Cochrane meta-analysis published in 2015

A. The conclusions of a recent randomized clinical trial (RCT) published in

Population of interest Most recent study

B. The latest systematic review from a famous orthopedist surgeon published

Risk of subjective bias Narrative synthesis

C. The conclusions of a Cochrane meta-analysis published in 2015