How to Conduct Systematic

Reviews and Meta-analysis

Nenacia Ranali Nirena P. Mendoza, MD, FPAFP
PAFP Research Committee
 Definition of narrative Preparation before
review, systematic review conducting meta-
and meta-analysis analysis

Steps in conducting meta-analysis

Outline
Demonstration on how to use RevMan

How to write a systematic review/meta-

analysis
 Narrative Review
• Subjective selection of
publications, summarizing the
findings and drawing a
conclusion from these findings
• No specific search strategy, no
critical appraisal, and no
quantitative synthesis of the data
(Lee YH, 2019)
Scoping Review
• Purpose is to provide an
overview of the available
research evidence without
producing a summary answer to
a discrete research question
• determine the scope or coverage
of a body of literature on a given
topic and give clear indication of
the volume of literature and
studies available (Munn, 2018)
 Systematic
Review
• Use of systematic method to
search and summarize evidence
on a clinical problem

• Starts with question, then

identifying studies with
systematic search, appraise the
quality of included studies,
summarize the evidence, and
interpret the results.
Tawfik, 2019
Indications for a Systematic Review

Uncover the Confirm current practice/ address

international any variation/ identify new
evidence practices

Identify and
Identify and investigate
inform areas for
conflicting results
future research

Produce statements to
guide decision-making
Munn, 2018
Sample Results Results are tabulated and discussed
in Systematic in a narrative manner in systematic
Review reviews
 Meta-analysis
• Process of integrating the
results of many studies to
arrive at a synthesis or
summary

• It is essentially a systematic
review, but there is a
statistical combination and
analysis of results
Basu, 2017
Systematic • Systematic review and meta-analysis follow the same steps
Review and • In meta-analysis, assumption that the studies are fairly uniform
• Data can be pooled to come up with a summary statistical estimate
Meta-analysis Basu, 2017
 Why Conduct Meta-analysis
• Synthesize results of studies with small sample sizes

• Decisions are transparent and the statistical analysis

results in an objective measure of the integrated
quantitative evidence

• Different findings from primary studies can be

compared against each other and yield conclusions
that are more significant

(Lee YH, 2019)

 Caution in Meta-analysis
• Critics point out the mixing of • Another criticism is the issue of
“apples and oranges,” publication bias
• Protocols, data collection • Positive studies are more
methods, definitions of likely to be published
disease, exposures etc. will • Bias towards finding a
vary from study to study significant result
 Decisions to Make Prior to Conducting
Meta-analysis
• How many authors? • PAFP Research Committee Registry
• Cochrane guidance, two reviewers • Provide technical support
• PAFP recommends at least three • Align with research agenda
reviewers

• Protocol registration should be done for

transparency
• Cochrane and Campbell collaborations
• PROSPERO
(Tawfik, 2019)
 Steps in Conducting Meta-analysis
Frame a question (based on a theory)

Literature search (PubMed/Medline, Google Scholar, other sources)

Selecting studies for inclusion

• Assessment for bias

Extract data from the selected set of final articles

Analysis using RevMan

Write the report using PRISMA guide

Step 1 - Frame a Question
P - population
PICO guide is usually used for I – intervention
systematic review and meta-analysis
C – comparator
of clinical trial study (Tawfik, 2019)
O – outcome

Population and intervention need to be well-defined during the initial

search

Comparator and outcome can be refined based on the search yield

(Basu, 2017)
 Frame a Question:
Validate the
Question
• Simple search in PubMed or Google Scholar
with search terms in PICO
• Preliminary search to determine if articles
are available
• Gain familiarity with topic
Tawfik, 2019

• Refine the question to make sure it is now

correctly focused
• Find if there is a recent meta-analysis on
the topic
• Differentiate your focus
Rodseth and Marais, 2016
Validate the
Question:
Example
Validate the
Question:
Example
Validate the
Question:
Example
 Step 2 – Literature Search
Search must be structured and logical, and each step must be recorded
(Rodseth and Marais, 2016)

Based on the research question PICO (Tawfik, 2019)

• free-text terms and MeSH

• Asterisk (*) marks for example, if you use something like "diabet*”

Boolean Logic connectors "AND", "OR", and "NOT" (Basu, 2017)

• Trial search by stepwise combination start with “P AND I”, then “P AND I AND O”

Some databases do not support Boolean or quotation

 Step 3 – Selecting Studies for Inclusion
Title and abstract review (Rodseth and
Marais, 2016)
• Screened by two or more investigators
• Abstracts that do not address the research
question are excluded
• Duplications and irrelevant articles will be
removed manually
• When in doubt, be inclusive rather than
exclusive (Tawfik, 2019)
Basis of decision to include/exclude
• Irrelevant for the study question
• Not the relevant population
• Not the relevant intervention
• Not the outcome of interest
• Foreign language and cannot be
translated
• Duplicate of another article
 Retrieving Full Text Articles
Many search engines provide links for free to access full text
articles. (Tawfik, 2019)
• ResearchGate, Google Scholar, Google may have full text or purchase

Searching reference lists of included articles, citation tracking,

follow all “related to” or “similar” articles (Tawfik, 2019)

Second screening (Rodseth and Marais, 2016)

• careful record must be kept of the number of excluded abstracts and trials as well
as the reasons for their exclusion
• Selection process, must be included in the report (PRISMA Diagram)
 Assess the Quality of Articles
• Sources of bias
Each trial must be • Randomization process
critically evaluated • Deviations from intended
interventions
• Missing outcome data
Asses the risk of bias • Measurement of the outcome
• Selection of the reported result
• Any other source of bias
• Low
• Unclear Rodseth and Marais, 2016

• High risk
Step 4 – Extract Data

Extraction should be executed by 2–3 independent reviewers

At the minimum, you must obtain the following information

• The name of the first author
• The year the article was published
• The population on whom the study was conducted
• The type of research
• Brief description of the intervention
• The comparison (what was it compared with?)
• What was the outcome and how was it measured?
Step 4 – Extract Data
• Outcome data for extraction
• Continuous scale
mean and standard deviation of the outcome in the intervention arm?
mean and standard deviation of the outcome in the comparison arm?

• Binary scale
number of people with the outcome and the number of people in the
intervention arm
number of people with the outcome and the number of people in the
comparison arm
Sample Data Extraction Form
Questions or Clarifications
 Step 5 - Analysis
• Qualitative and quantitative analysis
• Qualitative analysis mostly
describes data
• Quantitative analysis
• Meta-analysis
• Network meta-analysis
• Subgroup, sensitivity,
cumulative analyses and meta-
regression

Tawfik, 2019
Review
Manager 5
(RevMan 5)
RevMan 5
Creating
New
Review
Creating
New
Review
Creating
New
Review
Creating
New
Review
Creating
New
Review
Adding a Study
Adding a Study
Adding a
Study
Adding a Study

Collinsworth, 2018
Assessment of Bias
Assessment of Bias
Entering Data
Sample Data Extraction Form
Sample Data Extraction Form
Entering Data: Add Comparison
Entering Data: Add Outcome
Entering Data: Add Outcome
Entering Data: Add Outcome
Entering Data: Add Outcome
Entering Data: Add Outcome
Entering Data: Add Outcome
Entering Data: Add Outcome
Entering Data: Add Study
Entering Data: Add Study
Entering Data: Add Study
Sample Data Extraction Form
Entering Data
Generate Forrest Plot
 Determine Heterogeneity

• Tests of heterogeneity include the chi2 and the I2 tests

• Low heterogeneity - not much difference between studies i.e.
homogenous
• High heterogeneity - there is difference i.e. heterogenous and this is
cause for concern
Basu, 2017
 Determine Heterogeneity

A rough guide to interpretation of I2 is as follows:

0% to 40%: might not be important;

30% to 60%: may represent moderate heterogeneity
50% to 90%: may represent substantial heterogeneity
75% to 100%: considerable heterogeneity
Cochrane handbook
 Fixed Effects and Random Effects
Fixed Effects
• Assumption: the true effect of
intervention (in both magnitude
and direction) is the same value in
every study (i.e. fixed across
studies)
• implies that the observed
differences among study results
are due solely to the play of
chance (i.e. that there is no
statistical heterogeneity)
Random Effects
• provides a result that may be
viewed as an ‘average
intervention effect’
• assumes that intervention effects
follow (usually) a normal
distribution
• implies that the observed
differences among study results
are due to a combination of the
play of chance and some genuine
variation in the intervention
effects.
Cochrane Handbook
Funnel Plot for Publication Bias
 Funnel Plot for Publication Bias
Reasons for publication bias
• Preference of journal editors to
select those with positive study
findings
• Funders support studies that may
have positive findings
• Investigators are less likely to publish
smaller studies with ambiguous
findings
• Smaller studies are delayed in their
publishing and are not therefore
captured

Basu, 2017
Risk of Bias Graph
Risk of Bias Graph
Risk of Bias Graph
Questions or Clarifications
PRISMA Guide
Step 6 – Writing the Report (PRISMA)
Title
• Specify that it is a systematic review, meta-analysis, or both

Abstract
• background, objectives, data sources, study eligibility criteria, participants, interventions, study
appraisal and synthesis methods, results, limitations, conclusions and implications of key findings
Introduction
• Describe the condition
• Describe the intervention being evaluated
• Describe how the intervention might provide benefit
• Why should the meta-analysis be conducted

Objectives
 Step 6 – Writing the Report (PRISMA)
• Methods
• Indicate if a review protocol was registered, provide registration information
• Eligibility criteria
• Specify study characteristics (such as PICOS, length of follow-up) and report
characteristics (such as years considered, language, publication status) used as
criteria for eligibility, giving rationale
• Information sources
• Describe all information sources (such as databases with dates of coverage,
contact with study authors to identify additional studies) in the search and date
last searched
 Step 6 – Writing the Report (PRISMA)
• Methods
• Search
• Present full electronic search strategy for at least one database, including
any limits used, such that it could be repeated
• Study selection
• State the process for selecting studies (that is, screening, eligibility,
included in the review)
• Data collection process
• Describe method of data extraction from reports and any processes for
obtaining and confirming data from investigators
• Data items
• List and define all variables for which data were sought
 Step 6 – Writing the Report (PRISMA)
• Methods
• Risk of bias in individual studies
• Describe methods used for assessing risk of bias of individual studies
• Summary measures
• State the principal summary measures (such as risk ratio, difference in means).
• Including measures of heterogeneity (such as I2 statistic)
• Synthesis of results
• Describe how
• Risk of bias across studies
• Describe how
• Additional analyses
 Step 6 – Writing the Report (PRISMA)
• Results
• Flow diagram of numbers of studies screened, assessed for eligibility, and included, with
reasons for exclusions at each stage
• Characteristics of included studies
• Method, participants, interventions, outcomes
• Results studies (Forrest plot for meta analysis, narrative for systematic review)
• Synthesis of results for all outcomes considered
• Present results of each meta-analysis done, including confidence intervals and measures
of consistency
• Risk of bias across studies
• Present results of any assessment of risk of bias across studies
• Additional analysis
PRISMA
Flow
Diagram
 Step 6 – Writing the Report (PRISMA)
• Discussion
• Summarize the main findings including the strength of evidence for each main
outcome; consider their relevance
• Summary of evidence
• Discuss limitations at study and outcome level (such as risk of bias), and at
review level (such as incomplete retrieval of identified research, reporting
bias)
• Provide a general interpretation of the results in the context of other
evidence, and implications for future research
• Limitations
• Conclusions
• References
 Acknowledgement

Dr. Noel Espallardo for the original lecture slides

Dr. Anna Guia Limpoco for consent to utilize as examples the RevMan
outputs for the study Effectiveness of Health Education Interventions
Directed to Patient, Family, Community on Chronic Obstructive
Pulmonary Disease in Family and Community Practice: A Meta-Analysis,
2021
Questions or Clarifications
Thank you