Meta-Analysis - Overview

Competence and Credible, Excel in Organization, Excel in Operation, Beyond Expectations, Management by Objective
Critical Appraisals of
Systematic Review and Meta-analysis
Siti Rizny F. Saldi
12 April 2019
Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

RSUPN Dr. Cipto Mangunkusumo – Fakultas Kedokteran Universitas Indonesia

Type of Article and Best study design

Type of Question Type of Study/Methodology
Therapy Information needed about Systematic review/Meta-
treatments (effectiveness, cost, analysis of RCTs
etc.) Double-Blind RCT
Diagnosis Information needed about a Systematic review/Meta-
diagnostic test (sensitivity, analysis of diagnostic studies
accuracy, etc.) Cross-sectional study with
random or consecutive sample
Prognosis Information needed about the Systematic review/Meta-
course of the disease over time, analysis of prognostic studies
expected complications, etc. Cohort/survival study
Etiology / Information needed about RCTs
Harm causes of disease or Cohort study
contributing factors of disease
Clinical Epidemiology and Case-control Study
Evidence-Based Medicine (CEEBM) Unit
What is a systematic review?

 “The application of strategies that limit bias in the
assembly, critical appraisal, and synthesis of all
relevant studies on a specific topic.”
Oxford Centre of Evidence Based Medicine (OCEBM) Levels Table
 Ensures that all available evidence is taken into

account and minimises ‘cherry-picking’
 Not performing SRs can be dangerous and/or

unethical!

Streptokinase for Myocardial Infarction
Why Systematic Reviews?

 Information overload → Systematically search
 Not all studies in journals → Systematically assessed

are good in quality the quality of included
(valid/unbiased) studies
 Result from many studies → Systematically combined

not conclusive to arrive at conclusion
(controversies exist)

• State the reasons why the review

Components of SR/MA is needed
• Based on problems in clinical
setting (high volume, high risk,
 Introduction: high cost)
• Any controversies in the
 Background
literatures? → In treatment,
 Research question(s) choice of diagnostic modalities,
 Objectives determination of prognosis
• Ended with objective of SR based
 Hypothesis on clinical question (PICO)
 Methods
• How the individual studies searched
 Results
and selected
 Discussion • How to appraise/assess the
individual studies
 Conclusions • How to combine (synthesis)

Steps of a Systematic Review following Cochrane Methods
1. define the question
2. plan eligibility criteria
3. plan methods
publish protocol
4. search for studies
5. apply eligibility criteria
6. collect data
7. assess studies for risk of bias
8. analyse and present results
9. interpret results and draw conclusions
publish review
10. improve and update review publish update
“Hang on. Systematic reviews collect,
appraise and combine evidence.”
“So why do we need to appraise them?”
Not all systematic reviews are high quality!

Evidence-Based Practice
Patient with
problems
Apply the Formulate

answerable
evidence clinical questions
Critical Search for

appraisal the evidence

Critical appraisals of
Systematic Review/Meta-analysis

How to read a Systematic Review:

The FAITH tools
Question
Find
Appraise
Include
Total up
Paul Glasziou
Heterogeneity
Centre for Research in Evidence Based Practice
Bond University, Gold Coast, Australia

Other tools:
etc.
Q-
F-
A-
I-
T-
H-
Question
Introduction section in SR/MA

Paragraphs containing:
• State the reasons why the review is needed
• Based on problems in clinical setting (high volume, high
risk, high cost)
• Any controversies in the literatures? → In treatment, choice
of diagnostic modalities, determination of prognosis
• Ended with objective of SR based on clinical question
(PICO)

I
O’s
Final paragraph of the introduction
C
Article selection, when conducting SR:
 Be methodical: plan first → protocol
 Study design: randomization (for clinical trial), cohort, etc.
 Patients’ characteristics: diagnosis (including severity), gender,
age, group, race, etc.
 Similarity of exposure or treatment (e.g. drug class, dosage)
 Similarity of outcomes (case definitions)
 Setting (emergency department, outpatient, inpatient)
Find
How to search?
▪ Be methodical: plan first
▪ A comprehensive and reproducible literature search is the
foundation of a systematic review
▪ Sources:
▪ Find published relevant articles:
▪ Major bibliographic databases: PubMed/MEDLINE, EMBASE, Cochrane.
▪ Hand-searching: textbooks, printed journals, reference lists
▪ Unpublished materials:
▪ Theses, dissertations, trial registries, contact with experts/personal
communication (peer group), etc.
▪ Search strategies: MeSH terms and text words
▪ No limitation on years and languages
Fishing for information

✓
Fishing the River Nairn by Duncan Brown www.flickr.com/photos/cradlehall/3789625428/

Publication bias
Is finding all published studies enough?
 Negative studies less likely to be published than ‘Positive’ ones
 Editors tend to accept studies with positive than negative results
 Positive results tend to be submitted to international journals, whilst
negative results are submitted to local journals
 Many negative results are product of studies with small sample size
 Inference by manufacturers
Some solutions:
 All trials registered at inception in meta-registry of trial registries:
www.clinicaltrials.gov, www.controlled-trials.com.
 Search in other sources: Conference proceedings, Technical reports
(Research, governmental agencies), Dissertations, theses, Contact with
primary researchers

https://www.bmj.com/content/331/7512/313?panels_ajax_tab_tab=jnl_bmj_tab_r
elated_art&panels_ajax_tab_trigger=related#datasupp
✓  ✓
✓
✓ ✓
✓ ✓
✓
✓ ✓
✓ ✓
clinical trial registry (?)

Appraise
Include
How to appraise/assess the individual

studies
• The validity of a systematic review ultimately
depends on the scientific method of the retrieved
studies and the reporting of data
• In systematic review to assess treatment effect,

RCTs are considered to be more rigorous than
observational studies
• A review based on well-designed RCT will likely be more
valid and accurate than a review based on observational
studies or case reports

Quality Assessment (cont.)

• Quality assessment should be performed by at
least two assessors
• The most common way to assess and report study

quality has been using COCHRANE RISK OF BIAS
assessment. Other example: JADAD score, CEBM
Oxford critical appraisal tool

Total up
Summarized the effect

Combine data to arrive at a summary, 3 measures:
 Effect Size (OR/RR/MD)
 Variance with 95% Confidence Interval
 Test of heterogeneity and examine why the studies
are heterogeneous (if they are)

Forest Plot
Heterogeneity
What is heterogeneity
Variation or Differences
between the true intervention effects underlying
the different studies
Differences across studies in meta-analysis.

Studies should have similar participants,
interventions, comparisons, and outcomes
Clinical Methodological Statistical

Sources of heterogeneity
 Clinical Diversity
 Participants, e.g. condition, age, gender, location, study
eligibility criteria
 Interventions, e.g. intensity/dose, duration, delivery,
additional components, experience of practitioners, control
(placebo, none, standard care)
 Outcomes, e.g. follow-up duration, ways of measuring,
definition of an event, cut-off points
 Methodological Diversity
 Design, e.g. randomised vs non-randomised
 Conduct, e.g. risk of bias (allocation concealment, blinding,
etc.), approach to analysis

Identifying heterogeneity
 Visual inspection (eyeball test) of forest plots
 Chi-squared (χ2) test (Q test)
 I2 statistic to quantify heterogeneity

The I2 statistic (1)

 I2 statistic describes the percentage of variability due to
heterogeneity rather than chance (0%-100%)
 Low values indicate no, or little heterogeneity
 0%-40%: might not be important
 High values indicate a lot of heterogeneity
 30%-60%: may represent moderate heterogeneity
 50%-90%: may represent substantial heterogeneity
 75%-100%: considerable heterogeneity
 Be cautious in interpreting: can be misleading since the

importance of inconsistency depends on several
factors
From: Cochrane Handbook for Systematic Reviews of Interventions

Methods
How well was the research done? (Internal validity) Yes No Unclear
✓
Question – Does the systematic review address a focused
question (PICO)?
……and use it to direct the search and select articles

for inclusion? ✓
Find – Did the search find all the relevant evidence? 
Appraise – Have the studies been critically appraised?
✓
Include – Did they only include high quality studies?  Explaining
heterogeneity
and excluding
✓
Total up – Have the results been totaled up with low quality
appropriate summary tables and plots? studies only for
MPCE outcome,

Heterogeneity – ……and heterogeneity between studies how about other
assessed and explained? outcomes?
What were the results?

Meta-analysis
 = calculated “best guess” of the true effect size
 The statistical combination of the results gives a
pooled, weighted average of the primary results
 It weights the effect size (result) of each study in
relation to sample size of the study
 Optional part of SR

MPCE++
Event MPCE--
Event
-blocker
Experiment 25 402 427
Control 31 399 430

MPCE++
Event MPCE--
Event
-blocker
Control 31 399 430
Control event rate (CER) = 31/430 = 7,2%

Experimental event rate (EER) = 25/427 = 5,9%
Absolute risk reduction (ARR) = CER – EER = 7,2% - 5,9% = 1,3 %
-blocker treatment reduced the risk of major perioperative cardiovascular events in patients
having non-cardiac surgery, of 1,3% from those who received placebo or standard treatment
Number needed to treat (NNT) = 1/ARR = 1/0,013 = 76,9  77
We need to treat 77 patients having non-cardiac surgery with perioperative -blocker treatment
to have one additional person who avoid major perioperative cardiovascular events
BNT++
Event BNT--
Event
-blocker
Control 30 492 522

BNT++
Event BNT--
Event
-blocker
Control 30 492 522
Control event rate (CER) = 30/522 = 5,7%

Experimental event rate (EER) = 77/516 = 14,9%
Absolute risk reduction (ARR) = CER – EER = 5,7% - 14,9% = -9,2 %
Absolute risk increased (ARI)
-blocker treatment increased the risk of bradycardia needing treatment in patients having
non-cardiac surgery, of 9,2% from those who received placebo or standard treatment
Number needed to harm (NNH) = 1/ARI = 1/0,092 = 10,8  11
We would have to treat 11 patients having non-cardiac surgery with perioperative -blocker,
to cause 1 extra bradycardia needing treatment
Applicability
Will they help me look after my patients?
 Mengacu pada pertanyaan terkait ‘applicability’ dalam
masing-masing worksheet critical appraisal yang relevan
(etiology/diagnosis/therapy/ prognosis)
 Diagnosis:
 Can I do the test in my setting?
 Do results apply to the mix of patients I see?
 Will the result change my management?
 Costs to patient/health service?
 Therapy:
 Is my patient so different to those in the study that the results
cannot apply?
 Is the treatment feasible in my setting?
 Will the potential benefits of treatment outweigh the potential
harms of treatment for my patient?

Conclusions
EBM and Systematic Review
EBM Systematic Review
 Steps  Steps
 Answerable Question  Answerable Question
 Search  Search ++++
 Appraise  Appraise x 2
 Synthesize
 Apply  Apply
 Time: 90 seconds  Time: 6 months

 < 20 articles  < 2,000 articles
 This patient survives!  This patient is dead
Find a systematic review!!
Terima Kasih
Further question(s):
email
rizny.saldi@ceebm.org
siti.rizny@ui.ac.id


Meta-Analysis - Overview

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Meta-Analysis - Overview

Uploaded by

Copyright:

Available Formats

Competence and Credible, Excel in Organization, Excel in Operation, Beyond Expectations, Management by Objective

Siti Rizny F. Saldi

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Type of Article and Best study design

What is a systematic review?

 Ensures that all available evidence is taken into

 Not performing SRs can be dangerous and/or

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Why Systematic Reviews?

 Not all studies in journals → Systematically assessed

 Result from many studies → Systematically combined

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

• State the reasons why the review

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

“So why do we need to appraise them?”

Not all systematic reviews are high quality!

Apply the Formulate

Critical Search for

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

How to read a Systematic Review:

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Introduction section in SR/MA

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Fishing the River Nairn by Duncan Brown www.flickr.com/photos/cradlehall/3789625428/

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

clinical trial registry (?)

How to appraise/assess the individual

• In systematic review to assess treatment effect,

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Quality Assessment (cont.)

• The most common way to assess and report study

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Summarized the effect

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Differences across studies in meta-analysis.

Clinical Methodological Statistical

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

The I2 statistic (1)

 Be cautious in interpreting: can be misleading since the

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

……and use it to direct the search and select articles

What were the results?

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

Control 31 399 430

Control 31 399 430

Control event rate (CER) = 31/430 = 7,2%

Number needed to treat (NNT) = 1/ARR = 1/0,013 = 76,9  77

Control 30 492 522

Control 30 492 522

Control event rate (CER) = 30/522 = 5,7%

Number needed to harm (NNH) = 1/ARI = 1/0,092 = 10,8  11

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

 Time: 90 seconds  Time: 6 months

Clinical Epidemiology and Evidence-Based Medicine (CEEBM) Unit

You might also like