Writing A Systematic Review: An Introduction: Hening Pujasari, PHD

WRITING A SYSTEMATIC REVIEW:
AN INTRODUCTION
Hening Pujasari, PhD

pujasari@ui.ac.id
Presented at
One Day Training Scientific Writing: Systematic Review
FoNUI Post Graduate Student Association
November 6, 2021
11/09/2021 pujasari@ui.ac.id
Outline what a systematic review is
To discuss scope and the formulation

of a review question
Aims of this
The purpose and format of a
section…. protocol
Systematic searching/screening of
studies
Data extraction/quality appraisal and
intro to evidence synthesis
WHAT IS A
SYSTEMATIC:
Done or acting
according to a fixed
SYSTEMATIC plan or system:

methodical
REVIEW? REVIEW: A
critical appraisal of a
book, play or other
work
What is a systematic review?
“A systematic review is a review in which there is a comprehensive search for relevant
studies on a specific topic, and those identified are then appraised and synthesized
according to a predetermined and explicit method.”* (*Klassen et al. Guides for reading
and interpreting systematic reviews. Arch Pediatr Adolesc Med 1998;152:700-704.)
A systematic review attempts to collate all empirical evidence that fits pre-specified
eligibility criteria in order to answer a specific research question. It uses explicit,
systematic methods that are selected with a view to minimizing bias, thus providing
more reliable findings from which conclusions can be drawn and decisions made
(Antman 1992, Oxman 1993)
 Minimise the impact of
bias/errors
 Can help to end confusion
 Highlight where there is not
Why we need sufficient evidence
systematic  Combining findings from
reviews different studies can highlight
new findings
 Can mitigate the need for
further trials
Why we need systematic reviews
• Facilitate rational decision making
• Health care providers, researchers and policy makers are
inundated with unmanageable amounts of information
• Over 20 million citations in PubMed
• Approx. 75 to 100 RCTs published daily
• Usually impossible to consider all relevant individual
primary research studies in a decision-making context
• Enable practitioners to keep up to date with evidence
accumulating in field and to practice evidence-based
medicine
 ‘Unscientific’ rarely pre-specify or make
methods explicit
 Rarely transparent or reproducible
Why not  Usually qualitative, subjective, opinions of

individual
traditional
reviews  Often incomplete, filing cabinet or MEDLINE
review
 Difficult to make sense across groups of

studies, especially when conflicting based on
qualitative reading alone
Hierarchy
of
evidence
Who undertakes systematic
reviews?
• Cochrane/Campbell Collaboration
• NICE/Regulatory bodies
• Health Technology Assessment
• Academics/researchers/Clinicians
• Students
Multidisciplinary teams
Clinicians
Who Health services researchers
undertakes Information scientists

systematic
reviews? Statisticians
Health Economists
Patient and public involvement

– particularly for guidelines
Key Stage Systematic Review- the process
Data extraction /checking
Define research/review question
Develop data extraction from into which study
In consultation/collaboration with the information and outcome data can be extracted,
clinical community, commissioners and checked & verified
patient/public representatives
Study assessment/appraisal
Develop review protocol
Assess the quality and validity of the included
Pre-specify the type of studies to be studies using the pre-defined method.
included, the methods of collating,
appraising and analysing data
Synthesis
Identify relevant studies
Narratively and/or statistically
Develop a comprehensive search summarise/describe the data, exploring
strategy and undertake systematic similarities and differences between studies.
searches of the literature
Assess eligibility
Knowledge translation
Select those studies which meet the pre-
defined inclusion criteria Review details and results are disseminated to
relevant target audiences using appropriate
formats
 Questions may be broad or
narrow
 Well-formulated questions will
guide many aspects of the
Define review process
research/revie  Searching strategy
w question  Inclusion/exclusion criteria
 Data extraction
 Choice of synthesis method
 Presentation/dissemination of
findings
•Discuss a very broad question and how you might
narrow it? (10 mins)
•Discuss the potential limitations of your review

questions
•If time and resource were not a limitation – consider

how useful would the answer to your review question
be?
Quick Activity
 a clear and concise statement of a
review's objectives (or questions) is
critical and should begin with a
precise statement of the primary
objective, including the
interventions reviewed and the
targeted problem; ideally, this would
be presented in a single sentence
Current
Cochrane & Prisma Statement guidance
“To assess the effects of [intervention
or comparison] for [health problem] in
[types of people, disease or problem, and
setting if specified].” Current
guidance
 Several criteria/frameworks proposed
to help guide question development
Question formulation
• Determining the scope is a decision dependent upon multiple
factors:
• Perspectives regarding a question’s relevance and potential
impact;
• Supporting theoretical, biologic and epidemiological
information;
• The potential generalizability and validity of answers to the
questions;
• Available resources;
• The wider literature base – has a recent high-quality SR
been conducted?
• Advantages and disadvantages to both broad and narrow
questions
• The validity of very broad question may be criticized for ‘mixing

apples and pears’; but advantages might include
• Comprehensive summary of the evidence
• Generalizability of findings
• Most obvious advantage of narrow focus is clarity of objectives
and ease of reading; but disadvantages might include
• Sparse evidence may limit findings/usefulness
• Generalizability of findings
 Often dealing with complex interventions
 Might be a need to develop working
definitions of the intervention of interest
 Several options on how to do this (pragmatic
real world v theoretical, logic models, etc.)
 Use content experts outside the review team
to ensure that the resulting definitions are
likely to be robust and meaningful
• A protocol is an essential component
of the systematic review process
• Helps to ensure careful a priori
planning
Protocol • Consistency
Development • Transparency
• Integrity
• Integral part of the process for
leading organizations/publication
process
 One of the features that
distinguish a systematic review
from a narrative review is the
pre-specification of criteria
Protocol  Inclusion
Development  Exclusion
 Methods
 Outcomes to be synthesized
 Etc.
PROSPERO – CRD initiative
• Search for existing current
reviews
• Register their planned review
online
• Publish protocol online
• Update record on Prospero
website as the review
progresses
• Avoids duplication of reviews
 Types of
 Studies (RCTs, non-RCTs, cohort/case-controlled)
 Population and setting
 Interventions
 Outcome measures
Searching for  Cochrane Handbook and CRD Guidelines

Information  Both provide explanations re the difference study
designs, likely biases and issues to consider when
including them
www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf
http://handbook.cochrane.org/
Searching for Information
 MESH terms and key words/synonyms

 Medical Subject Heading – controlled vocabulary thesaurus used for indexing articles
 young; adoles*; teen*; child*...................
*end of the ‘stem’ of the word it will automatically search for all the endings for that
word stem
 Child* will also return children, childbearing, childbirth and so on…
Word variants
• AIDS
• acquired immunodeficiency syndrome
• acquired immuno-deficiency syndrome
Searching • acquired immune deficiency syndrome

• acquired immune-deficiency syndrome
for Synonyms e.g. Newborn: infant, toddler, baby,

etc.
Information
Plurals e.g. child : children OR teenager : teenagers
Spelling variants (UK vs US) e.g.

randomise/randomize
Searching for Information
Where to search
 Electronic databases: Medline, Embase, Cochrane, PsycInfo, etc.
 Grey literature, dissertations, theses, conference proceedings,
national bodies (NICE, HTA), clinical trial database (
www.clincialtrails.gov/)
 Look at the databases own guidance for searching they vary!
Boolean
operators
 Reference manager software package
 Endnote – RefMan – ProCite – Mendeley
 Import results and screen

 Assess titles/abstracts against your predetermined criteria
Selection of  If in doubt include
Studies  Retrieve full text articles of initial selections
 Assess full text for inclusion

 Requires judgement (>1 reviewer)
 Check reviewer agreement (3rd review to resolve)
 Use a selection form to ensure consistency and record
decisions
Data Extraction/Quality
Appraisal
Be clear what information you want from the studies:
• Study details
• Data for your analysis
Information will need to be collected relating to:

Data
• Methodology
Extraction
• Population
• Interventions being compared
• Outcomes evaluated
What effect How are you
measures you planning to
are you going to group studies for
calculate the analysis?
• What data do you • By intervention?
need to do this? • By study design?
Give
consideration
What
information do
you need to
REMEMBER
YOUR to….
PROTOCOL – IT
extract to enable
IS YOUR
you to organise
ROADMAP,
and analyse the
FOLLOW IT!
way you want?
Level of judgement
However
is required
• Sufficient to • You need to limit
describe studies unnecessary
• Sufficient to allow detail
How much to you to undertake
the planned
extract?? analysis
• Sufficient so you
do not need to
return to the full
text papers
There is a wide selection of software to choose
from
Data Selection depends on a number of factors

extraction
software? Main considerations are probably
• What are you are familiar with?
• What package best suits your data?
• How many included studies do you have?
 Word
 Excel
 Access
 EPPI reviewer
 COEVIDENCE
 REVMAN
 ????
Which
software?
Consistency/ • We all have to be doing the same thing
• Essential >one reviewer is extracting data
Standardization
• Data must be interpreted in the same way
by all reviewers
• Independent piloting of data extraction

forms – always one standardised form
• Regular discussion of
progress/disagreements
• Regular comparison of data extraction –
don’t wait till the end
Once data extraction is complete you may need to:
• Sort/search your data

• Filter data
• Calculate frequencies
Efficient data • Transform data (e.g. SE to SD)
extraction Categorising/coding data will make these tasks easier
• Needs to be implemented with consistency by the whole team
A database can be designed to have this functionality
• You may need separate forms for each study
Are you including design
more than one • However, you are still answering the same
question, so make sure the core information
study design? extracted is the same
Have one or a
• Will the data still be useful?
Things to few studies
• Should you include it?
reported data • Make sure the core information extracted is the
consider differently from same
the others?
You may need to

update the form, • Any changes need to be agreed and made
or have more consistently
than one form
 Be careful about collecting ‘extra’ data
 It is very tempting to collect data that are not
directly relevant to the review question
 The data needed to answer the review

question should have already been decided
(REMEMBER YOUR PROTOCOL)
 Collect data for good reasons – stay focused
and don’t get side-tracked
 Time and effort to collect, only to find it is not Stay on

useful track……
Quality Assessment & Critical Appraisal
Why bother????
What are we trying to achieve?
Not all published and unpublished literature is rigorous!

• being in a journal doesn’t mean it is good
Quality may be used as an explanation for differences in study

results or to guide interpretation of findings, strength of inferences
 Quantitative studies
 Internal validity
Quality  Bias: selection; performance; detection; attrition;
reporting
Assessment &  External validity
 Move away from checklists/numerical scores to domain
Critical based assessment
 Cochrane Risk of Bias - RCTs
Appraisal  QUADAS 2 – diagnostic accuracy
 ROBIS for systematic reviews
 Qualitative studies
 Three broad categories
 Rigour: has a thorough and
Quality appropriate approach been
Assessment & applied to key research methods
in the study?
Critical  Credibility: are the findings well
Appraisal presented and meaningful?
 Relevance: how useful are the
findings to you and your
organisation?
 Clear aims of research (goals, why it is
important, relevance)
 Appropriate methodology
 Sampling strategy
 Data collection
CASP appraisal  Relationship between researcher and
checklist participants
 Ethical issues
 Data analysis
8. Findings
9. Value of research (context dependent)
Data Synthesis
 Building up; putting together;

making a whole out of the
parts; the combination of
separate elements of thought
into a whole; reasoning from
principles to a conclusion
 Results from different studies need to be synthesised
 Are studies and results similar enough to be
combined into a single numerical result?
 NO – qualitative descriptive/narrative summary
 YES – quantitative meta-analysis
Data Synthesis  Heterogeneity
 Difference in results can arise due to differences in
study design, population, selection, intervention
delivery
 How similar is similar? Results from heterogeneous
studies should not be pooled
 Instead of/alongside meta-
analysis
 Potential bias in presentation
 Lack of a take home message
Narrative synthesis
Tools for  Partly informed by methodological
work in qualitative synthesis
narrative 

Tabulation
Groupings and clusters
synthesis 

Vote counting as a descriptive tool
Examination of moderator variables
(elements of e.g. setting, population)
 Rodgers et al Evaluation
2009 15 49-72
Meta-analysis/forest Plot
Most important thing:
Be organised!!!
Timeline
Use Use a reference manger to sift and store
Keep all citations retrieved

Keep • Add in those you can’t download
Reference
management
Use Use to de-duplicate results
Sift citations for inclusion/exclusion

Sift • Can use codes/notes
Dates – YYYYMMDD
Version numbering
• v0.1 = first draft
• v1.0 = final version
• v1.1 = minor amendments to final version
Version control • v2.0 = major revision
Avoid using draft, draft 1, final final, etc.
Clear naming convention

• E.g. Date_project_title
• 20/11_Autism HTA_resultsv1.2.doc
DATA
EXTRACTION,
PRESENTING
FINDINGS, &
WRITING REPORT
DATA
EXTRACTION
Systematic Review Process Overview
To describe why data extraction is
important
To identify challenges in data

Learning extraction
Objectives To describe the general layout of a
data extraction form
To suggest methods for collecting

data accurately and efficiently
Why Is Data Extraction Important?
To summarize studies in a
To obtain information to
common format to
To identify numerical data assess more objectively
facilitate synthesis and
for meta-analyses the risk of bias in and
coherent presentation of
applicability of studies
data
To identify systematically
missing or incorrectly
assessed data, outcomes
that are never studied, and
underrepresented
populations
On Data Extraction
Extracted data should:
• Accurately reflect information reported in the publication

• Remain in a form close to the original reporting, so that disputes can be easily resolved
• Provide sufficient information to understand the studies and to perform analyses
Extract only the data needed, because the extraction process:
• Is labor intensive
• Can be costly and error prone
Different research questions may have different data needs
Data Elements:
Population, Intervention, and Comparator
Population-generic elements may include patient characteristics, such

as age, gender distribution, and disease stage.
• More specific items may be needed, depending upon the topic.
Intervention or exposure and comparator items depend upon the

extracted study.
• Study types include randomized trial, observational study, diagnostic test study,
prognostic factor study, family-based or population-based genetic study, et cetera.
Data Elements: Timing and Study Design
 The data elements to be extracted vary by type of study.

 Consider collecting this information when recording study characteristics for
randomized trials:
 Number of centers (multicenter studies)
 Method of randomization (adequacy of allocation concealment)
 Blinding
 Funding source
 Whether or not an intention-to-treat analysis was used
Provide “operational definitions” (instructions)
Provide indicating exactly what should be extracted in
each field of the form.
Always
Provide
Instructions Make sure that all data extractors understand the
operational definitions the same way.
Make • Pilot-test the forms on several published papers.
• Encourage communication to clarify even apparently
mundane questions.
Single Versus Double Extraction
■ Independent extraction of data by at least two experienced reviewers is ideal but
is also resource intensive.
■ There is a tradeoff between cost and the quality of data extraction.
– Data extraction often takes longer than 2 hours per paper.
– A reduction in the scope of the work may be necessary if independent data

extraction is desired.
■ Careful single extraction by experienced reviewers, with or without

crosschecking
11/09/2021
of selected items by a second reviewer, is a good compromise.
pujasari@ui.ac.id
To address all needs, a generic data extraction form will have to be
very comprehensive.
Developing Although there are common generic elements, forms need to be

adapted to each topic or study design to be most efficient.
Data Organization of information in the PICOTS (population, intervention,
Extraction comparator, outcome, timing, and setting) format is highly desirable.
Forms Balance the structure of the form with the flexibility of its use.
(Evidence
Tables) Anticipate the need to capture unanticipated data.
Use an iterative process and have several individuals test the form on
multiple studies.
Common Problems Encountered When
Creating
Data Extraction Forms (Evidence Tables) (I)
 Forms have to be constructed before any serious data extraction is underway.

 Original fields may turn out to be inefficient or unusable when coding begins.
 Reviewers must:
 be as thorough as possible in the initial set-up,
 reconfigure the tables as needed, and
 use a dual review process to fill in gaps.
Evidence Tables: Example (I)
Second Draft
First Draft
Evidence Tables: Example (II)
Final Draft
Common Problems Encountered When Creating
Data Extraction Forms (Evidence Tables) (II)
 Lack of uniformity among outside reviewers:

 No matter how clear and detailed are the instructions, data will not be entered
identically by one reviewer to the next.
 Solutions:
 Develop an evidence table guidance document—instructions on how to input
data.
 Limit the number of core members handling the evidence tables to avoid
discrepancies in presentation.
Sample Fields From a Table Guidance Document:
Vanderbilt University Evidence-based Practice Center
In the “country, setting” field, data

In the “study design” field, data
extractors should list possible
extractors should list one of the
settings that could be encountered
following:
in the literature:
• Academic medical center(s), • Randomized controlled trial,
community, database, tertiary cross-sectional study,
care hospital(s), specialty care longitudinal study, case-control
treatment center(s), substance study, et cetera.
abuse center(s), level I trauma
center(s), et cetera.
Example:
Two Reviewers Extract Different Data
Reviewer A Reviewer B
Samples of Final Data Extraction Forms
(Evidence Tables)
For evidence reports or

technology assessments that Remember, there is more than
have many key questions, data one way to structure a data
extraction forms may be extraction form.
several pages long.
Trikalinos TA, et al. AHRQ Technology Assessment. Available at:

http://www.cms.gov/determinationprocess/downloads/id48TA.pdf.
Pencil and paper
Tools Word processing software (e.g., Microsoft

Word)
Available for Spreadsheet (e.g., Microsoft Excel)
Data
Extraction Database software (e.g., Microsoft Access,
Epi Info™)
and Dedicated off-the-shelf commercial software
Collection
Homegrown software
Extracting the Data
 Who should extract the data?

 Domain experts versus methodologists
 What extraction method should be used?
 Single or double independent extraction followed by reconciliation
versus single extraction and independent verification
 Should data extraction be blinded (to authors, journal, results)?
Berlin J, for the University of Pennsylvania Meta-analysis Blinding Study Group. Lancet 1997;350:185-6.
Problems in data
reporting
Challenges in
Inconsistencies in
Data
Extraction published papers
Data reported in graphs
Data extraction is laborious and tedious.
To err is human: data extractors will identify

errors and will err themselves.
Conclusions
Interpretation and subjectivity are unavoidable.
Data are often not reported in a uniform manner

(e.g., quality, location in paper, metrics, outcomes,
numerical value vs. graphs).
Key Messages
Extracting data requires Be persistent:

• Often, one can extract more
Key questions will guide familiarity with the content
There is no single correct information than the paper initially
reviewers in choosing which and knowledge of appears to contain (e.g., by
way to record extracted data.
information to extract. epidemiological principles digitizing graphs).
and statistical concepts.
Try to verify the same piece Inconsistencies indicate

Sometimes there are
Be comprehensive: of information from different suboptimal reporting quality
surprising inconsistencies.
places in the same article. at least.
PRESENTATION OF
FINDINGS
Learning Objectives
To become familiar with

To understand the goals
approaches to graphical
of presenting systematic
presentations of findings
review data in tables that
not synthesized through
summarize data across
statistical or meta-
studies
analytic techniques
Organizing and Reporting Findings
■ Once the evidence tables are completed, there is an enormous (sometimes

overwhelming) amount of data yet to synthesize.
■ Various types of heterogeneity may preclude a quantitative synthesis (e.g., meta-
analysis).
■ Synthesizing heterogeneous studies by using tables (nonquantitative synthesis) can
capture underlying similarities to support conclusions.
■ Making sense of the data requires good tabular presentation, in addition to clear
organization and writing of the text.
Why Not Meta-analysis?
 Even studies meeting the same inclusion criteria can vary.

 Clinical heterogeneity — variation in the study population, interventions, and outcomes
 Methodological heterogeneity — variation in study design
 Statistical heterogeneity — variation in observed treatment effect (for trials)
 Many factors can contribute to variation in seemingly similar studies.
 Some examples are evolving diagnostic criteria, evolving diseases, differences in baseline
characteristics, and differences in care.
 Not all studies can (or should) be combined statistically.
 Various statistical models are useful for identifying the level of heterogeneity between
studies.
AHRQ . Methods reference guide for effectiveness and comparative effectiveness reviews. Version 1.0.
Available at: http://effectivehealthcare.ahrq.gov/repFiles/2007_10DraftMethodsGuide.pdf.
Solution
Nonquantitative
synthesis using Use of evidence
tables that maps to provide an
summarize data overview of the data
across studies
PICOTS
Intervention: hierarchy of Comparator: where multiple

Population: primary
interventions may reflect comparisons are made, one
populations of interest are
most common to least might present the more
followed by subpopulations
common “usual” decision first
Outcome: a hierarchy of
outcomes may reflect higher Setting: an intervention may
to lower acuity or target Timing: short term generally be available in various
outcomes followed by precedes long term settings (e.g., inpatient or
collateral ones; harms are outpatient)
generally presented last
Evidence tables are the first step to summarization, but
each evidence table represents the data in only one
study.
Incorporating multiple studies into a single table allows

entire subsets of the literature to be summarized and
compared (e.g., by key question or study design).
Combining
Studies Summary tables and evidence maps are two approaches
with which information about individual studies and
results are combined.
Authors should use caution when deciding which

evidence to summarize so as not to introduce a biased
perspective.
Combine data from multiple studies to
illustrate trends in the data
May be focused on describing study

characteristics, results, or both
Summary Can be designed to include characteristics of

all included studies
Tables (I) • Examples: funding sources, assessment method, country
of study
Can be designed for subsets of included
studies
• Examples: summary tables for randomized controlled
trials, prevalence studies, harms/side effects, outcomes
for specific treatments
Summary Tables (II)
■ Simplified entry (one row) for each study •PICOTS = population, intervention,
■ Table columns may include, for example: comparator, outcomes, timing, and setting
– PICOTS (may be listed in table title or headers)
– Methodological quality
– Applicability
– Study size (weight)
– Magnitude of effect
■ A single study may be represented in multiple
summary tables (e.g., different outcomes)
Example:
Summary Table of Study Characteristics
■ A basic summary table is the “study

characteristics” table.
■ The overall summary provides an
overview of the state of the
available studies in the literature.
Hartmann KE, et al. AHRQ Evidence Report/Technology Assessment

No. 187. Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/
bladder/bladder.pdf.
Example:
Summary Table of Prevalence Findings
Summary tables
can be specialized
for different types
of questions.

No. 187. Available at: http://www.ahrq.gov/ downloads/pub/evidence/
pdf/ bladder/bladder.pdf.
Example:
Summary Table for Randomized Controlled Trials
Summary tables can

be specialized for
different subgroups
of the literature.
Hartmann KE, et al. AHRQ Evidence Report/Technology Assessment No. 187.

Available at: http://www.ahrq.gov/ downloads/pub/evidence/pdf/bladder/bladder.pdf.
Example:
Presentation of Patient-Reported Outcomes
Summary tables
can be
specialized for
different types of
outcomes.

No. 187. Available at: http://www.ahrq.gov/ downloads/pub/evidence/
pdf/ bladder/bladder.pdf.
Forest Plot Without Summary
Favors Wide-Area Favors Pulmonary

Circumferential Ablation Vein Isolation
Ip S, et al. AHRQ Comparative Effectiveness Review No. 15. Available at:

http://www.effectivehealthcare.ahrq.gov/ehc/products/51/114/2009_0623RadiofrequencyFinal.pdf.
 Summary tables and evidence maps provide
key information on study characteristics and
study findings. Through table and graphical
Key
formats, respectively
 Properly constructed summary tables:
Messages
 Effectively convey results
 Provide an overview of the literature in a
given field
 Enable the reader to grasp results for
subsets of the literature
REPORTING THE REVIEW
Learning
Objectives
 To describe the various elements that need to be reported upon completion

of a systematic review
 To distinguish examples of reporting that are adequate from those that are
inadequate
Writing the Report
Abstract and Executive Summary

Follow a standard Chapter 1. Introduction
template for the overall Chapter 2. Methods
Chapter 3. Results
report: Chapter 4. Discussion
Ordering of subsections Should adhere to principles of clarity

Should be consistent with key questions
may vary but: May be guided by PICOTS
PICOT(S) = population, intervention, comparator, outcome, time frame, and study design or setting
Abstract and Executive Summary
Chapter 1: Introduction
• The purpose of this chapter is to define the project, the
Systematic purpose and scope of the review, the key research questions,
the analytic framework, et cetera.
Review Chapter 2: Methods

• The purpose of this chapter is to explain the methods used in
Report the review, including the experts involved, the literature
search strategy used, the inclusion and exclusion criteria
Structure (I) applied, how the evidence tables were developed, the
approach used to assess the quality of studies, and data
abstraction and data synthesis methods.
• This chapter should not present any “results” (e.g., tables),
but rather should serve as a guide for how the study
information was collected and the evidence tables were
created so that the research can be replicated.
Systematic Review Report Structure (II)
Chapter 3: • The purpose of this chapter is to report the results of the

data analyses, which should be broken down according to
the key research questions.
Results • Subsections should be used when applicable.
Chapter 4: • The purpose of this chapter is to discuss the strength of the

literature and evidence, the principal findings (broken
Discussion
down by key questions) of the review, areas of future
research, and any conclusions that can be drawn.
Using Formal Guidelines To Improve
the Reporting of Systematic Reviews
 The PRISMA (Preferred Reporting Items for Systematic
Reviews and Meta-Analyses) Statement is a guideline that
was developed to help improve the quality of review reports.
 The guideline consists of a 27-item checklist and a flow
diagram.
 Investigators can access the guideline online
(http://www.prisma-statement.org/index.htm).
Title of the Review Report
■ Identify the report as a systematic review, meta-analysis, or
comparative effectiveness review.
■ Use the PICOTS framework to guide construction of the title.
■ Make the title as succinct as possible yet keep it informative.
– Example of a short title:
Comparative Effectiveness of Lipid-Modifying Agents
– Example of a longer but more informative title:
Mortality in Randomized Trials of Antioxidant Supplements for
Primary And Secondary Prevention: Systematic Review and
Meta-analysis
PICOT(S) = population, intervention, comparator, outcome, time frame, and study design or setting
Executive Summary (I)
■ Should be structured as follows:
 Background
 Objectives
 Key questions
 Methods
 Data sources
 Eligibility criteria
 Study appraisal
 Data synthesis
 Results
 Limitations
 Conclusions
 Implications of key findings
Executive Summary (II)
 Should be a distillation of the entire report
 Should exclude study-by-study results
 Should describe the evidence that supports all summary statements
 Should describe the strength of the evidence as categorized in the evidence
review
Introduction
Should state the purpose and scope of the review:
• Identify the clinical decisional dilemma.

• Identify the current literature and state of practice.
• Give readers the context in which the review was conducted.
Should include these important components:
• Objective(s) and key question(s)

• Provide an explicit statement of the research questions being
addressed with reference to the PICOTS framework.
• Analytic framework
• Use such a framework to model existing evidence (refer to the
Analyticintervention,
PICOTS = population, Framework module
comparator, for time
outcome, details).
frame, and study design or setting
Methods Overview
 Should provide the following information in a clear and transparent
manner:
 Literature Search Strategy and Data Sources
 Eligibility Criteria
 Specific to study characteristics (e.g., PICOTS, length of followup)
 Specific to report characteristics (e.g., years considered, language,
publication status)
 Data Extraction and Data Items (e.g., variables for which data were
sought, assumptions and simplifications )
 Quality Assessment
 Synthesis of Results
 Grading Strength of Evidence
 Additional Analyses
Methods: Literature Search Strategy
 Present the complete electronic search strategy — including any limits
used — in the Appendix of the report.
 Brief example:
We used the following search terms to search all trials registers and
databases: immunoglobulin; IVIG [intravenous immunoglobulin];
sepsis; septic shock; septicaemia; and septicemia.
 The purpose of including the entire search strategy is to ensure
transparency and to permit replication of the review.
Alejandria MM, et al. Cochrane Database Syst Rev 2002;(1):CD001090.
Methods: Data Sources
 Describe all information sources used in the literature search:
 Databases with dates of coverage
 Contacts with study authors to identify additional studies
 Date of the last search
 Example:
We searched the following databases for primary studies for the
periods in parentheses: MEDLINE® (1966 to January 2006),
EMBASE® (1974 to January 2006), and the Cochrane Central
Register of Controlled Trials (1966 to January 2006). We also
searched for systematic reviews until November 2005.
Bolen S, et al. AHRQ Comparative Effectiveness Review No. 5. Available at:

http://www.effectivehealthcare.ahrq.gov/ehc/products/6/39/OralFullReport.pdf.
Methods: Eligibility Criteria (l)
 State the processes used to select studies for review:
 Screening
 Eligibility assessment
 Inclusion/exclusion criteria applied for the systematic review and, if applicable,
the meta-analysis
 Example:
We included trials if the randomization scheme included groups that assigned
patients to treatment guided by the PAC [pulmonary artery catheter] or treatment
without the PAC. We only included trials if they reported death and number of
days hospitalized or the number of days in the ICU as outcome measures. Studies
were excluded if the randomization scheme did not specify groups as PAC or no
PAC, if patients were not randomized to a conventional PAC, if investigators
combined randomized and nonrandomized groups when reporting outcomes, or if
there were no outcome data on death or hospitalizations.
Shah MR, et al. JAMA 2005;294:1664-70.
Methods: Eligibility Criteria (II)
 Type of studies:
 Example:
Randomised clinical trials studying the administration of hepatitis B
vaccine to CRF [chronic renal failure] patients, with or without
dialysis. No language, publication date, or publication status
restrictions were imposed.
 Types of participants:
 Example:
Participants of any age with CRF or receiving dialysis
(haemodialysis or peritoneal dialysis) were considered.…Renal
transplant patients were excluded from this review.
Schroth RJ, et al. Cochrane Database Syst Rev 2004;(3):CD003775.
Methods: Eligibility Criteria (III)
 Types of interventions:
 Example:
Trials comparing the beneficial and harmful effects of hepatitis B vaccines with
adjuvant or cytokine co-interventions [and] trials comparing the beneficial and
harmful effects of immunoglobulin prophylaxis.…Hepatitis B vaccines (plasma or
recombinant [yeast] derived) of all types, dose, and regimens versus placebo,
control vaccine, or no vaccine.
 Types of outcomes:
 Example:
Primary outcome measures: Seroconversion, [that is], proportion of patients with
adequate anti-HBs response (10 IU/L or Sample Ratio Units).…Secondary
outcome measures: Adverse events of hepatitis B vaccinations…[and] mortality.
Schroth RJ, et al. Cochrane Database Syst Rev 2004;(3):CD003775.
Methods: Data Extraction
 Describe the method of data extraction.
 Example:
We developed a data extraction sheet[,]…pilot-tested it on ten
randomly-selected included studies, and refined it accordingly. One
review author extracted the…data…and the second author checked
the extracted data. …Disagreements were resolved by discussion
between the two review authors.
 Describe any processes used to obtain and confirm data from other
investigators.
Mistiaen P, Poot E. Cochrane Database Syst Rev 2006;(4):CD004510.
Methods: Data Items
 List and define all variables for which data were sought, using
the PICOTS framework as a guide.
 List any assumptions and simplifications that were made in
defining the variables.
 Example:
Information was extracted from each included trial on: (1)
characteristics of trial participants…and the trial’s inclusion and
exclusion criteria; (2) type of intervention…(versus placebo or
versus the type, dose, duration and frequency of another NSAID
[nonsteroidal antiinflammatory drug]; or versus another pain
management drug; or versus no treatment); (3) type of outcome
measure.
PICOT(S) = population, intervention, comparator, outcome, timing, and study design or setting
Allen C, et al. Cochrane Database Syst Rev 2005;(4):CD004753.
Methods: Quality Assessment
 Describe the methods and criteria used to assess the quality (risk of bias)
of individual studies.
 Specify whether or not the assessment was carried out at the study or
outcome level, or both.
 Describe how this information is to be used in any data synthesis.
 Examples from two separate studies:
Pairs of reviewers…determined the adequacy of randomization and
concealment of allocation, blinding of patients, health care providers,
data collectors, and outcome assessors; and extent of loss to follow-
up.
To explore variability in study results (heterogeneity) we
Tracz MJ, et al. J Clin Endocrinol Metab 2006;91:2011-6; Bucher HC, et al. BMJ 2000;321:73-7.
Methods: Synthesis of Results
 For each meta-analysis:

 Describe the methods used to handle the data and to combine the results of studies.
 Describe measures of consistency (e.g., I-squared).
 Examples from two separate studies:
In very few instances, estimates of baseline mean or mean QOL [quality of life] responses were obtained without
corresponding estimates of variance (standard deviation [SD] or standard error). In these instances, an SD was
imputed from the mean of the known SDs. In a number of cases, the response data available were the mean and
variance in a prestudy condition and after therapy.…
We tested for heterogeneity with the Breslow-Day test, and used the method proposed by Higgins et al. to measure
inconsistency.
Methods: Grading Strength of Evidence
 Detail any assessment of risk of bias that may affect the cumulative
evidence.
 Publication bias
 Selective reporting within studies
 Example:
For each trial we plotted the effect by the inverse of its standard
error. The symmetry of such “funnel plots” was assessed both
visually, and formally with Egger’s test, to see if the effect decreased
with increasing sample size.
Hróbjartsson A, Gøtzsche PC. Cochrane Database Syst Rev 2004;(1):CD003974.
Methods: Additional Analyses
Describe the methods of additional analyses (e.g.,

sensitivity or subgroup analyses, meta-regression).
Example:
• Sensitivity analyses were pre-specified. The
treatment effects were examined according to
Indicate which of these quality components (concealed treatment
allocation, blinding of patients and caregivers,
blinded outcome assessment), time to initiation
analyses were prespecified. of statins, and the type of statin. One post-hoc
sensitivity analysis was conducted including
unpublished data from a trial using
cerivastatin.
Briel M, et al. JAMA 2006;295:2046-56.
Results Overview
 The results section of the review should contain the following subsections:
 Study Selection
 Study Characteristics
 Quality Assessment
 Individual Studies
 Synthesis of Results
 Grading Strength of Evidence
 Additional Analyses
 Sensitivity
 Subgroup
Place at the beginning of the Results section, not in
Place the Methods section.
Results:
Study Give Give numbers of studies screened, assessed for
eligibility, and included in the review.
Selection
Give the reasons for exclusions at each stage of the
Give assessment, ideally illustrated with a flow diagram.
• Example:
Examples of Flow Diagrams
Sharma M, et al. Ann Intern Med 2009:151:622-30. Reprinted Fuccio L, et al. Ann Intern Med 2007;147:53-62. Reprinted
with permission from the American College of Physicians. with permission from the American College of Physicians.
Results: Study Characteristics
 Study characteristics can be presented:
 Within text
 In summary tables and graphs
 Example:
All four studies finally selected for the review were randomised
controlled trials published in English. The duration of the
intervention was 24 months for the RIO-North America and 12
months for the RIO-Diabetes RIO-Lipids and RIO-Europe study.
Although the last two described a period of 24 months during which
they were conducted, only the first 12-months results are provided.
All trials had a run-in, as a single blind period before the
randomisation.
Curioni C, André C. Cochrane Database Syst Rev 2006;(4):CD006162.
Results: Quality Assessment
 Present data on risk of bias for each study analyzed.
 Present the results of outcome-level assessments, if available.
 Example:
Devereaux PJ, et al. BMJ 2005;331:313-21, as adapted in Liberati A, et al. Ann Intern Med 2009;151:W65-
94.
Results: Individual Studies*
 For all outcomes considered (benefits and harms), present the following
for each study:
 simple summary data for each intervention group, and
 effect estimates and confidence intervals (ideally with a forest plot).
 Present the results of individual studies in evidence tables and summary
tables and not in the text.
 Refer to the Presentation of Findings module to see examples of the tables
and graphs used to present summaries of individual studies.
* This may appear in the appendix for Evidence-based Practice Center reports.
Results: Synthesis of Results
 Present synthesized results in text, summary tables, or evidence maps.
 Text example:
Mortality data were available for all six trials, randomizing 311
patients and reporting data for 305 patients. There were no deaths
reported in the three respiratory syncytial virus/severe bronchiolitis
trials; thus our estimate is based on three trials randomizing 232
patients, 64 of whom died. In the pooled analysis, surfactant was
associated with significantly lower mortality (relative risk = 0.7, 95%
confidence interval = 0.4–0.97, P = 0.04). There was no evidence of
heterogeneity (I2 = 0%).
 Summary tables: refer to the Presentation of Findings module
 Evidence maps: refer to the Presentation of Findings
Results: Grading Strength of Evidence
 Include risk of bias, directness, consistency, and precision in
reporting how evidence was graded.
 Example:
There is a low level of evidence…that RFA [radiofrequency
catheter ablation] improves quality of life more than medical
treatment. Three RCTs [randomized controlled trials] and one
observational study reported more improvement in the general or
physical functioning…in patients who underwent RFA.…However,
these studies assessed the results at nonuniform time points and
therefore the findings may be difficult to interpret.
Ip S, et al. AHRQ Comparative Effectiveness Review No. 15. Available at:

http://www.effectivehealthcare.ahrq.gov/ehc/products/51/114/2009_0623RadiofrequencyFinal.pdf.
Results: Additional Sensitivity Analyses
 Give the results of additional analyses, such as sensitivity or
subgroup analyses and meta-regressions.
 Include the results of additional analyses to facilitate a better
understanding of heterogeneity.
 Example 1:
[B]enefits of chondroitin were smaller in trials with adequate
concealment of allocation compared with trials with unclear
concealment (P for interaction = 0.050), in trials with an intention-
to-treat analysis compared with those that had excluded patients from
the analysis (P for interaction = 0.017), and in large compared with
small trials (P for interaction = 0.022).
Reichenbach S, et al. Ann Intern Med 2007;146:580-90.
Results: Additional Subgroup Analyses
 Example 2:
Subgroup analyses according to antibody status, antiviral
medications, organ transplanted, treatment duration, use of
antilymphocyte therapy, time to outcome assessment, study
quality and other aspects of study design did not demonstrate
any differences in treatment effects. Multivariate meta-
regression showed no significant difference in CMV
[cytomegalovirus] disease after allowing for potential
confounding or effect-modification by prophylactic drug used,
organ transplanted or recipient serostatus in CMV positive
recipients and CMV negative recipients of CMV positive
donors.
Hodson EM, et al. Cochrane Database Syst Rev 2008;(2):CD003774.
Summarize the main findings, including the strength of
Summarize evidence for each main outcome.
Discussion:
Summary of Consider
Consider the applicability of findings to key groups (e.g.,
health care providers, users, and policymakers).
Evidence
Refer to the Assessing Applicability and Grading Strength of
Refer Evidence modules for additional guidance.
Discussion: Summary of Evidence
Example:
Compared with men who used watchful waiting , men with
clinically localized prostate cancer detected by methods other
than PSA [prostate-specific antigen] testing and treated with
radical prostatectomy experienced fewer deaths from prostate
cancer, marginally fewer deaths from any cause, and fewer
distant metastases. The greater benefit of RP on cancer-specific
and overall mortality appears to be limited to men under 65
years of age but is not dependent on baseline PSA level or
histologic grade.
Discussion: Limitations
 Discuss the limitations of the review at different levels:
 Study level (e.g., risk of bias)
 Outcome level (e.g., benefits or harms)
 Review level (e.g., incomplete retrieval of identified research;
reporting bias)
 Example:
The meta-analysis reported here combines data across studies in
order to estimate treatment effects with more precision than is
possible in a single study. The main limitation of this meta-
analysis, as with any overview, is that the patient population, the
antibiotic regimen and the outcome definitions are not the same
across studies.
Liberati A, et al. Cochrane Database Syst Rev 2004;(1):CD000022.
Discussion: Conclusions
 Provide a general interpretation of the results in the context of other
evidence and the implications for future research.
 Example:
[T]he available clinical trial evidence supporting the use of
combination therapies over higher dose statin therapy is insufficient
to guide clinical decisions. The long term clinical benefits and risks
of combination therapies have yet to be demonstrated. There are
some instances, such as failure to reach targets in spite of maximal
statin therapy, and populations with elevated triglycerides who need
to achieve secondary goals, in which clinicians may choose
combinations pending definitive evidence.
Sharma M, et al. AHRQ Comparative Effectiveness Review No. 16. Available at:
http://www.effectivehealthcare.ahrq.gov/ehc/products/11/171/reptbodyfin-typofixed4-12-2010.pdf.
Key Messages
 Reporting a systematic review is the final step of the review process.
 The report should convey in a transparent manner the methods and results
to readers, including consumers, clinicians, and policymakers.
 Inadequate reporting makes it more difficult to judge the validity of the
methods and results.
THANK YOU

Writing A Systematic Review: An Introduction: Hening Pujasari, PHD

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Writing A Systematic Review: An Introduction: Hening Pujasari, PHD

Uploaded by

Copyright:

Available Formats

WRITING A SYSTEMATIC REVIEW:

Hening Pujasari, PhD

To discuss scope and the formulation

SYSTEMATIC plan or system:

 Rarely transparent or reproducible

Why not  Usually qualitative, subjective, opinions of

 Difficult to make sense across groups of

Who Health services researchers

undertakes Information scientists

Patient and public involvement

•Discuss the potential limitations of your review

•If time and resource were not a limitation – consider

• The validity of very broad question may be criticized for ‘mixing

Searching for  Cochrane Handbook and CRD Guidelines

 MESH terms and key words/synonyms

 Child* will also return children, childbearing, childbirth and so on…

Searching • acquired immune deficiency syndrome

for Synonyms e.g. Newborn: infant, toddler, baby,

Spelling variants (UK vs US) e.g.

 Look at the databases own guidance for searching they vary!

 Import results and screen

Studies  Retrieve full text articles of initial selections

 Assess full text for inclusion

Information will need to be collected relating to:

Data Selection depends on a number of factors

• Independent piloting of data extraction

• Sort/search your data

extraction Categorising/coding data will make these tasks easier

• Needs to be implemented with consistency by the whole team

A database can be designed to have this functionality

You may need to

 The data needed to answer the review

 Time and effort to collect, only to find it is not Stay on

What are we trying to achieve?

Not all published and unpublished literature is rigorous!

Quality may be used as an explanation for differences in study

 Building up; putting together;

Keep all citations retrieved

Sift citations for inclusion/exclusion

Avoid using draft, draft 1, final final, etc.

Clear naming convention

To identify challenges in data

To suggest methods for collecting

Extracted data should:

• Accurately reflect information reported in the publication

Extract only the data needed, because the extraction process:

Different research questions may have different data needs

Population-generic elements may include patient characteristics, such

Intervention or exposure and comparator items depend upon the

 The data elements to be extracted vary by type of study.

■ There is a tradeoff between cost and the quality of data extraction.

– Data extraction often takes longer than 2 hours per paper.

– A reduction in the scope of the work may be necessary if independent data

■ Careful single extraction by experienced reviewers, with or without

Developing Although there are common generic elements, forms need to be

Data Organization of information in the PICOTS (population, intervention,

Extraction comparator, outcome, timing, and setting) format is highly desirable.

 Forms have to be constructed before any serious data extraction is underway.

 Lack of uniformity among outside reviewers:

In the “country, setting” field, data

For evidence reports or

Trikalinos TA, et al. AHRQ Technology Assessment. Available at:

Tools Word processing software (e.g., Microsoft