(Systematic Review and Meta-analysis)

. . 1, .. 2, .. 3
1
, 2

Systematic Review Meta-analysis

1. Concept of systematic review
a. Narrative vs. systematic review
b. Why is it important?
2. Question and inclusion criteria formulation
a. Patients
b. Intervention
c. Outcome
d. Methodology
3. Publication bias
4. Study identification
Comprehensive literature search
Why is it important ? -Concept
Database
Keyword from context and from methodology filter
Short literature search in some database (pubmed)
Comprehensive literature search
5. Study selection
6. Validity assessment
7. Data extraction
8. Analysis
a. Choices of effect measures
b. Heterogeneity, : concept
c. Statistical model for meta-analysis: Random
vs. Fixed effect model, concept, when ?
d. Example
9. Interpretation
Overview from clinical question regarding evidence


(Concept of Systematic Review)
-

Systematic Review

Systematic review

(systematic)


(quantitative)
Meta-analysis Meta-analysis systematic
review

-

Systematic Review vs. Narrative Review

Systematic review review
1. Systematic review

2. Systematic review
(quantitative)

3. Systematic review

Systematic review

(sample size calculation)

type I error ( ) type II
error ( )
10%, 20% 50%

(
)
1

 1. (Sample Size Calculation)

20%
20%
20%

Relative risk
reduction (RRR)

18%
10%
16%
20%
10%
50%
type I error = 0.05 power 80%

()
12,280
2,994
438

1 (RRR =10% ,
20% 50%)


1
RRR 10 RRR 20

50 4 30


Systematic review Meta-analysis





80-90

10 10
10

Systematic review Meta-analysis

Meta-analysis The Lancet .
2547 observational Antioxidant beta
carotene, vitamins A, C, E
Systematic review Meta-analysis Controlled trial
Antioxidant
Antioxidant


(Question and inclusion criteria formulation for
selecting the primary studies)

(Population),
(Intervention/Exposure),
(Comparison)
(Outcome) PICO1
1. (therapy)
1. :

2. :
Transarterial therapy

3. :
4. :
1
Transarterial therapy
1

2. (risk factor/etiology)
1. :

2. :
(passive smoking)

3. :
4. :

Richardson et al. The well-built clinical question: a key to evidence-based decisions.

Counsell C. Formulating questions and locating primary studies for inclusion in systematic
reviews. Ann Intern Med 1997;127:380-7


(Inclusion criteria)

1. :
2
1.1.

1.1.1.
1.1.2.
(Imaging study
Ultrasonography, CT MRI scan) alpha
fetoprotein
1.2.

1.2.1.
2. :

beta blocker

2.1. Transarterial therapy

2.1.1. Transarterial chemoembolization
2.1.2. Transarterial embolization
2.1.3. Transarterial chemotherapy
2.2.
(co-intervention)

2.2.1.
( )
3. :


3.1.
4. :



4.1. 1 (one-year survival)
4.2. (tumor response)
4.3. (adverse effect)
5. : Randomized Controlled Trial (RCT)
systematic review
RCT

 systematic review
Observational studies
case-control, cohort cross-sectional
systematic review


5.1. Randomized Controlled Trial

keywords

Publication Bias
Systematic review Meta-analysis

systematic review meta-analysis

Publication bias

Publication bias

(positive trials)
(negative trials)
negative trial
negative trial

Systematic review Meta-analysis

positive trial


(overestimate)
Systematic review
Meta-analysis publication bias Funnel plot
(plot graph) (X)
(Y)


Meta-analysis

 Sample Size

Pooled Estimate
of treatment effect

Favor Intervention

Favor Control

Sample Size

1: Funnel Plot Meta-Analysis Publication Bias

Biased estimate
(publication biased)

Favor Intervention

Unbiased estimate

Favor Control

2: Funnel Plot Meta-Analysis Publication Bias

Meta-analysis publication bias
(funnel) meta-analysis
publication bias
publication bias
language biases negative trial

 negative trials,



Publication bias
(unpublished trial)

 (Study Identification)
(Population, Intervention,
Comparison, Outcome PICO)

systematic review narrative traditional review

narrative review

systematic review

2


-

(Existing reviews)

PUBMED
keyword methodology filters
PUBMED clinical queries Systematic reviews

o PUBMED

- Evidence Based Medicine Reviews
.

Clinical Evidence UP-TO-DATE

Cancer Care Ontario : Clinical Practice Guidelines

American Society of Clinical Oncology Practice Guideline

methodology filter Search Strategies

Shojania KG Haynes B

Shojania KG, Bero LA. Taking advantage of the explosion of systematic reviews: an
efficient MEDLINE search strategy. Eff Clin Pract. 2001 Jul-Aug;4(4):157-62.

1. (review or review, tutorial or review, academic).pt.

2. (medline or medlars or Embase).ti,ab,sh.
3. (scisearch or psychinfo or psycinfo).ti,ab,sh.
4. (psychlit or psyclit).ti,ab,sh.
5. cinahl.ti,ab,sh.
6. (hand search$ or manual search$).tw.
7. (electronic database$ or bibliographic database$).tw.
8. (pooling or pooled analys$).tw.
9. (peto or der simonian or dersimonian or fixed effect or mantel haenszel).tw.
10. or/2-9
11. 1 and 10
12. meta-analysis.pt.
13. meta-analysis.sh.
14. (meta-analy$ or metaanaly$ or meta analy$).tw,sh.
15. (systematic$ adj25 review$).tw,sh.
16. (systematic$ adj25 overview$).tw,sh.
17. (quantitative$ adj25 review$).tw,sh.
18. (quantitative$ adj25 overview$).tw,sh.
19. (methodologic$ adj25 review$).tw,sh.
20. (methodologic$ adj25 overview$).tw,sh.
21. (integrative research review$ or research integration).tw,sh.
22. (quantitative$ adj25 synthesi$).tw,sh.
23. or/12-22
24. 11 or 23
25. (random$ or placebo$).tw,sh,pt.
26. (clinical trial or controlled clinical trial).pt.
27. double blind.tw,sh,pt.
28. 25 or 26 or 27
29. 24 and 28
30. 1 or 23
31. exp asthma/
32. 24 and 31
33. 29 and 31
34. 30 and 31
Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. J Am Med
Inform Assoc. 1994 Nov-Dec;1(6):447-58. Developing optimal search strategies
for detecting clinically sound studies in MEDLINE.

 methodology filter MEDLINE

EMBASE, CINAHL, HealthSTAR

1. Keyword methodology
filter ( systematic review, metaanalysis)
a. Keyword Hepatocellular Carcinoma,

Hepatoma, Transarterial Chemoembolization Keyword

- (PICO),
- index keyword
subject heading
-
index

b. Keyword methodology filter

2. PUBMED
clinical queries systematic reviews methodology filter
National Library of Medicine Dr.Haynes B et al,

McMaster University, Canada

Keyword
3. Evidence
Based Medicine Reviews Cochrane Library Database of
Abstracts of Reviews of Effects Clinical Evidence UPto date systematic review / meta-analysis
Keyword
4.
search engine web site
http://www.shef.ac.uk/scharr/ir/netting/ Cancer Care
Ontario Clinical Practice Guidelines American Society of Clinical
Oncology

Systematic review 3

1 2

From our review of review, currently there are three meta-analyses addressing this
method of treatment Simonetti (5) in 1997, Mathurin (6) in 1998 and Comma (7) in 2002. The
first and the second meta-analysis included 3 and 6 RCTs which compared TAE or TACE with no
active treatment or co-intervention in both treatment and control groups. Simonetti et al. reported
border-line significant therapeutic benefit (Odd ratio of 1-year survival rate = 2.0, 95% CI 1.1-3.6)
in the set of 3 RCTs where embolization was compared to no treatment or to intravenous 5-FU in
1997. In 1998, Mathurin et al reported the result from a meta-analysis of 6 RCTs, comparing
TACE or TAE with no active treatment or tamoxifen or intravenous 5-FU, which failed to show a
benefit of TACE on 1-year survival (Risk difference=0.3%, 95%CI = -11 to 12 %). In this year,
2002, instead of reporting the result of 1-year survival, Comma reported the benefit of 2-years
survival of 5 RCTs meta-analysis (OR = 0.54, 95% CI = 0.33-0.89).
Since hepatocellular carcinoma is the fatal disease with only 20% survival rate at
one-year (8-10). For this reason, one year-survival rate would be the better information in the
decision making for both HCC patients and the treating physicians. Therefore, we conducted this
meta-analysis on the updated and comprehensive searched for the RCTs by using the overall oneyear survival as the end point.

Keyword methodology filter
( randomized controlled trial systematic review metaanalysis)

o PUBMED (national library of medicine)

clinical queries therapy , specific search

 Dr.Haynes Keyword

o
PUBMED Cochrane Central Register
of Controlled Trials

o methodology filter

methodology filter Cochrane maximally sensitive search

strategy for RCTS Medline
1.

randomized controlled trial.pt.

2.

controlled clinical trial.pt.

3.

randomized controlled trials.sh.

4.

random allocation.sh.

5.

double blind method.sh.

6.

single blind method.sh.

7.

or/1-6

8.

(animal not human).sh.

9.

7 not 8

10. clinical trial.pt.

11. exp clinical trials/
12. (clin$ adj25 trial$).ti,ab.
13. ((singl$ or doubl$ or trebl$ or tripl$) adj25 (blind$ or mask$)).ti,ab.
14. placebos.sh.

15. placebo$.ti,ab.
16. random$.ti,ab.
17. research design.sh.
18. or/10-17
19. 18 not 8
20. 19 not 9
21. comparative study.sh.
22. exp evaluation studies/
23. follow up studies.sh.
24. prospective studies.sh.
25. (control$ or prospectiv$ or volunteer$).ti,ab.
26. or/21-25
27. 26 not 8
28. 27 not (9 or 20)
29. 9 or 20 or 28
30. exp asthma/
31. 29 and 30
Lines 30 and 31 for illustration only.

Search Strategies



1.

nasopharynx

2.

3. (hand searching)

(Reference lists)

 search strategies

 (Study selection)


systematic review
(review) (selection criteria) review

4
1. (study population)

2. (study intervention)

(risk factor) (prognostic factor)
(diagnostic test)
3. (outcomes)




4. (research methodology)
review review randomized controlled trials



review review
/
(selection criteria) 1

review
review

 review

review

(
systematic review )
(bias)
review
systematic review (reliability
reproducibility) review
2 (
)

chance corrected agreement kappa ()
100
2 1
review 20 2 25 2
18 2
2 review 2
1

18
7
25

a b
a+b
2
2
73
75

c d

20

c+d

80
a+c b+d

100
n

2 2 ( ) 18
+ 73 = 91 100 91/100
(observed agreement raw agreement)
2 2
(chance)
1 20 100 20 % 2
25 100 25 % 2
( agreement ) 25 2
1 5 ( 20 % 25 ) 75

 2 1 60 ( 80 %
75 ) 2
5 + 60 = 65 100 ( 3) 65/100
(chance agreement expected agreement)
3 review 2
2
1

5
20
25

2
15
60
75

20
80
100

2

2
kappa

observed agreement chance
agreement 1
chance agreement
kappa =

observed agreement chance agreement

1 chance agreement

( 2)

a+d
n
(
a + c )(a + b ) + (b + d )(c + d )
chance agreement =
n2
(91 / 100) (65/100) = 0.74

kappa =
1 (65/100)
observed agreement =

kappa 2

kappa 1 kappa 1 2
review 2
kappa 0 2
2 kappa
2 kappa 1
review

 2 (
)
2
(consensus)
2
review

(consensus)
review

systematic review

(bias)





review


systematic review 1 (duplicate
publication)
3 5
3
(
)
review review

review duplicate publication
(
)

duplicate publication review

 (validity assessment quality

assessment)
review systematic review
(validity)

(error)
(systematic error) (bias) ( 4)
4 2
(treatment or prevention)
1. (subject)
(randomization) (randomized
1

controlled trial)
2. concealed randomization (

subject )
3.
( blinding)
4. outcome (completeness
of follow-up)
5. subject subject ( )
subject (

intention-to-treat principle)
(diagnostic test)
1. diagnostic test

2. diagnostic test (gold standard)

3. diagnostic test gold standard gold standard
diagnostic test (blind comparison)
(risk factor etiology)
(prognostic factor)
1. cohort study (bias)
case-control study

Guyatt H, Rennie D, editors. The Evidence-Based Midicine working Group. Users guides to the medical
literature: A manual for evidence-based medicine clinical practice. AMA Press, 2002.
2

2.

2

3. (exposure assessment)
case-control study
4. (outcome assessment)

5. (completeness of follow-up)
systematic review 2
scale



Jadad scale 3 Chalmers scale 4
4

systematic
review

review
heterogeneity ( )

(reliability
reproducibility)
2
kappa
(data extraction)
systematic review
systematic review
1.

2.
Jadad AR, Moore RA, Carroll D, Jenklinson C, Reynolds DJM, Gavaghan DJ, McQuay HJ. Assessing the quality of
reports of randomized clinical trials: Is blinding necessary?. Controlled Clinical Trials 1996; 17: 1 12.
4
Chalmers TC, Smith H Jr, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A. A method for assessing
the quality of a randomized control trial. Controlled Clinical Trials 1981; 2: 31 49.
3

3.

4.

5.

150 30 20 %
30 150 20 %
systematic review
intention-to-treat principle
subject



(
)

2


reliability reproducibility






(corresponding author)

(e-mail)


systematic review sensitivity analysis (
)


( Plan of Analysis of Systematic Reviews)
4 ( Cochrane
Reviewers Handbook 4.2.4 March 2004)
1. 2

transarterial chemotherapy
(supportive treatment)
- (direction of effect )
transarterial chemotherapy supportive treatment
- ( size of effect )
transarterial chemotherapy supportive care 10%, 20%
50%
-
(heterogeneity)
2.

transarterial chemotherapy transvenous chemotherapy

(supportive treatment)




3.

meta-analysis


4.
transarterial chemotherapy chemotherapy


1

1. (Type of outcome data)

2. (Effect measures)

3. (Summarising effects across studies / Meta-analysis)

4. (Heterogeneity)

5. (Sensitivity
Analysis)

(Type of outcome data)

5
o Dichotomous binary data

o Continuous data

o Ordinal data
measurement scales
o Counts Rates

o Time- to- event data ( survival data)
( censored
data)

survival data (
-disease free survival overall survival) Meta-analysis survival data
individual patient data
individual patient data assumption constant hazard ratio follow
up period literature-based data fixed data 1 year
survival Dichotomous binary data (
)

o Dichotomous or binary data

Risk ratio (RR) relative risk
Odds ratio (OR)
Risk diferrence (RD) absolute risk reduction
(ARR)
Number needed to treat (NNT)

* Cochrane reviewers handbook

4 Odds ratio
RR RR
initial risk 20%
o Continuous data
Mean difference
Standardised mean difference
o Ordinal data
Proportional odds ratio ( Cochrane review
manager)
Continuous data Binary data
o Counts and rates
Rate ratio

Odds ratio Risk ratio

(Summarising effects across studies /

Meta-analysis)
o ( Meta-analysis )

Meta-analysis statistical power

(precision)

 Meta-analysis



publication bias

Meta-analysis
o Meta-analysis

Meta-analysis 2 treatment effect

treatment effect
(pooled results)

model Random
effect model Fixed effect model Random effect model
(treatment effect)
(distribution)
heterogeneity clinical heterogeneity
statistical heterogeneity Fixed effect model
review manager Cochrane Library
model

 heterogeneity intervention
Random effect model

review manager

software Cochrane Library Review Manager Review Manager
software Cochrane Library
, Protocol
(Version 4.2.2)
click add
next stage ( )

 tree reference included

studies click

 tables comparisons and data

click one year overall survival

click one year overall survival

dichotomous data ( death/alive at one year)
Data extraction plot survival curve

 outcome description TACE

TAE TAC vs palliative care , statistical Relative risk, 95%
confidence interval Random effect model graph

click title TACE TAE TAC vs palliative care

included studies save close

 click toolbar Analysis

double click TACE TAE TAC vs palliative care

 Forest Plot treatment effect

treatment effect

Heterogeneity
o Heterogeneity

Heterogeneity

clinical diversity
clinical heterogeneity

methodological diversity methodological heterogeneity statistical
heterogeneity
power p value 0.1 0.05 p
heterogeneity
clinical heterogeneity methodological heterogeneity

o Statistical heterogeneity
Meta-analysis
heterogeneity

(Priori hypothesis) (Post
hoc analysis) Statistical heterogeneity

clinical heterogeneity
methodological heterogeneity meta-analysis

heterogeneity subgroup analysis
meta-regression
Ignore heterogeneity Fixed effect model

Random effect model
heterogeneity
heterogeneity

effect measures
continous outcome

test for heterogeneity , p =

0.05 ( < 0.1 ) statistical heterogeneity
explore heterogeneity
Subgroup Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus
Symptomatic treatment: One-Year Overall Survival (Random Effect Model) by inclusion of poor prognostic
patients

Subgroup Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: One-Year Overall Survival (Random Effect Model) by type of intervention

Subgroup Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: One-Year Overall Survival (Random Effect Model) by Type of Chemotherapy

Subgroup Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: One-Year Overall Survival (Random Effect Model) by Application of Cointervention

test of heterogeneity poor prognostic

factors type of intervention
heterogeneity
o (Sensitivity Analysis)

meta-analysis

(robust)







Fixed effect
model Random effect model
sensitivity analysis
poor prognostic factors, type of intervention- embolization or chemotherapy or both,
type of chemotherapy, methodological issues, best case and worst case scenario of
some studies

Sensitivity Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: Removing the Trial with Ambigous Allocation Concealment, One-year Overall
Survival (Random Effect Model)

Sensitivity Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: Best Case Scenario of Madden, 1993 (61)

Sensitivity Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: Worst Case Scenario of Madden, 1993 (61)

Sensitivity Analysis of Transarterial Chemoembolization, Embolization, Chemotherapy Versus

Symptomatic treatment: Madden, 1993 (61) All patient in both treatment and controlled group were death

allocation concealment
sensitivity analysis Madden, 1993

worst case scenario
1.
2.
3.

While we are waiting for the complete one-year over all survival data of Madden (61), the sensitivity analyses
assuming three scenarios were also carried out. The scenarios are as follows:
Best case scenario; All of the surviving patients at 150 days in the TAC group were still alive, but all of the surviving
patients at 150 days in controlled group were dead at one year
Worst case scenario; All of the surviving patients at 150 days in the TAC group were dead, but all of the surviving
patients at 150 days in controlled group were alive at one year
The study did not contribute any weight for the pooled estimate i.e. all patients in TAC and controlled group were
dead at one year.
The significant difference in one-year overall survival in favor of patients who received the transarterial interventions
was not changed in the best case and unweighted scenario but disappeared in the worst case scenario.

Alderson P, Green S, Higgins JPT, editors. Cochrane Reviewers Handbook 4.2.2
[updated March 2004]. In: The Cochrane Library, Issue 1, 2004. Chichester, UK:
John Wiley & Sons, Ltd.


(Presenting Results of Systematic Reviews)

(QUOROM)
checklist 5 trial flow
5 Checklist


Moher D. Cook DJ, Eastwood S et al. Improving the quality of reports
of meta-analyses of randomized controlled trials: the QUOROM
statement. Lancet 1999;354:1896-900.


( Interpreting Results )

(Cochrane Reviewers handbook)
1. (What is the direction of effect ?)
(survival )
transarterial chemotherapy supportive care hepatocellular carcinoma

(RR=0.85; 95% CI = 0.74 to 0.99; p=0.04) :
transarterial
chemotherapy marginal
(heterogeneity)
2. (What is the size of the effect ?)
10% ,
20% 50%
transarterial chemotherapy
0.85 supportive treatment
(clinical importance) 0.99
3. (Is the effect
consistent across the studies ? )

test for heterogeneity, p =0.05
<0.1 (heterogeneity)

4. (What is strength of the

evidence of the effect ? )

3 (study design)
(quality of the studies)
:

(Randomized Controlled trials)

publication bias

o heterogeneity

o ( 1 )

update
o effect measures survival data
survival outcome Individual patient data Meta-analysis
Literature-based Meta-analysis
1
6 1
5.
transarterial chemotherapy
(validity)



(Prolongation of overall survival, disease free survival or
progression free survival ) (more quality of life score)
(tolerable toxicity of chemotherapy)



patient, intervention and
disease variation biologic and cultural variation, variation in compliance,
variation in baseline risk

implication Cochrane
reviewers handbook 4.2.2


overall survival 1 marginal



disease free progression free
survival, quality of life score


transarterial chemotherapy

inclusion criteria
subgroup analysis TACE
TAE TAC
( systematic review
meta-analysis )
marginal significance
heterogeneity