Systematic Reviews: What, Why, and How

Judy Welsh
NIH Library
welshju@mail.nih.gov
August 2017
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
Cochrane graphic illustration of the systematic review process
What is a systematic review?

• Uses transparent procedures to find, evaluate and

synthesize the results of independent studies.
• Procedures are explicitly defined in advance, to
assure that the process is transparent and can be
replicated.
• This process is also designed to minimize bias.
Unique characteristics of a systematic review

• A systematic review must have:

– Clear inclusion and exclusion criteria
– Explicit search strategy
– Systematic coding and analysis of included studies
– Meta-analysis (where possible)
What is a meta-analysis?

• Optional component of a systematic review

– A statistical analysis of results from individual studies
• Increase power
• Improve estimates of the size of the effect
• Resolve uncertainty when reports disagree
Traditional vs Systematic Reviews
What is a traditional review?

• Uses informal, unsystematic and subjective methods

to collect, interpret, and summarize information.
– Searching, quality appraisal and data synthesis are often
not documented.
– Since scientific methods are not routinely used to identify,
assess and synthesize information, assumptions cannot be
easily verified.
– Greater risk of author and selection bias
Comparison of traditional and systematic reviews

Traditional review Systematic review

• Subjective • Objective
• Broad question • Narrow question
• Methods unclear • Methods clearly defined
• Studies appraised
Evidence based pyramid

Reviews
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
Information overload
EBP and clinical practice guidelines

• Healthcare decisions should be informed by the best

available research evidence.
– Many guidelines rely on previously published systematic
reviews
– Practice guideline quality is dependent on rigorous
systematic review methods and high quality primary
studies
• Primary studies
• Randomized controlled trials
• Systematic reviews
• Evidence based practice and clinical practice guidelines
Known and unknown areas of study

• SRs can show which treatments and prevention

methods have been proven to work - and what
remains unknown.
• SRs are important for pointing to areas where more
research is needed.
• Systematic reviews are the basis for what is often
called evidence-based medicine or health care.
Learning Objectives

1. What is a systematic review?

2. What is the difference between a systematic review
and a traditional review?
3. Why are systematic reviews important?
4. What are the steps in performing a systematic
review?
 Stages of a systematic review

Develop a focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as necessary
 Stages of a systematic review

Develop a focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as needed
 Research Question

• A clearly defined, focused systematic review begins

with a well formulated research question.
Research Question

• The research question guides the author in working

through many stages of the systematic review
process
– Defining inclusion and exclusion criteria
– Searching the literature
– Selecting studies
– Extracting data
– Analyzing and presenting results
FINER Criteria

• A research question should be:

– Feasible
– Interesting
– Novel
– Ethical
– Relevant
PICO Framework

• A well-established format for structuring research

questions is known by the acronym PICO.
– Patient or Population
– Intervention or Indicator
– Comparator or Control
– Outcome
PICO
• Patients • Patients undergoing upper abdominal surgery
• Intervention • Prophylactic physical therapy
• Comparison • No prophylactic physical therapy
• Outcome • Prevent post operative pulmonary complications
Types of clinical research questions

• Many types of research questions can be expressed

using PICO components.
– Therapy
– Diagnosis
– Prognosis
– Etiology / Harm
– Clinical Prediction Guides
Stages of a systematic review

Develop A focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as needed
Inclusion and exclusion criteria

• One of the features that distinguish a systematic

review from a traditional review is the pre-
specification of inclusion and exclusion criteria.
Inclusion criteria

• Inclusion criteria are a combination of

• Aspects of the research question
• Population
• Intervention
• Comparison
• Outcome
• Study type
• Randomized controlled trials
• Observational studies
Stages of a systematic review

Develop A focused research question

Define inclusion and exclusion criteria
Search the literature
Select studies
Assess study quality
Extract data
Analyze and present results
Interpret results and draw conclusions
Update as needed
Search the literature

• The goal of the literature search is to discover all

studies that meet the inclusion criteria
– Search comprehensively
• Terminology
• Databases
– Search for grey literature
• Not commercially published
– Search for unpublished studies
• Reduce risk of publication bias
Recommended databases

• The Cochrane Collaboration recommends searching

the following databases (at minimum):
– PubMed
– EMBASE
– Cochrane Central Register of Controlled Trials
Supplementary databases

• Interdisciplinary databases
– Scopus
– Web of Science
• Specialized databases
– CINAHL Plus
– PEDro: Physiotherapy Evidence Database
– PsycINFO
Search strategy

• The search strategy should be designed to identify

the maximum number of studies relevant to the
research question.
– The search strategy should be systematic, transparent and
reproducible.
– Database specific controlled vocabulary terms and all
relevant text words should be included in the search
strategy.
Supplementary searching techniques

• Search cited and citing references

– Scopus
– Web of Science
Supplementary searching techniques

• Hand search selected journals and conference

proceedings
• Conduct author searches for recent articles written
by topic experts
Unpublished and grey literature

• The inclusion of unpublished and grey literature may

minimize the potential effects of publication bias.
– Publication bias
• Occurs when the outcome of an experiment or research study
influences the decision about whether—or how quickly—the
manuscript may be published
What is grey literature?

• Grey literature refers to academic, business,

government or industry print or electronic literature
that is not controlled by commercial publishers.
– Conference proceedings
– Research reports
– Government reports
– Dissertations, theses
– Research monographs
– Organization websites
Sources of grey literature

• Conference proceedings
– EMBASE
– Scopus
– Web of Science
– Google
Sources of unpublished literature

• Clinical trials
– ClinicalTrials.gov
– Centerwatch.com
– EU Clinical Trials Register
– ISRCTN Registry
– OpenTrials
– WHO International Clinical Trials Registry Platform
Compile search results

• Compile search results using reference management

software (EndNote or Mendeley)
• Remove duplicate records
Document search process

• Document the search process

– Databases
– Dates searched
– Search strategies
– Limits (date ranges, publication types, language
restrictions)
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search The Literature
Select studies
Assess Study Quality
Extract Data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update As Needed
Select studies

• Identification of studies meeting inclusion criteria

should be done independently by two review
authors.
– Review titles and abstracts of retrieved citations.
– Review full text of studies which are found to meet the
inclusion criteria.
– Keep a record of reasons for inclusion or exclusion.
Tools for sorting

• Tools for sorting

– Reference management software
• EndNote
– New web application
• Rayyan
– Excel
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess study quality
Extract Data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update As Needed
Assess study quality

• Following the full text review, assess the selected

studies for risk of bias and study quality.
Tools for assessing risk of bias

• Tools for assessing risk of bias

– ROBIS: Risk of Bias in Systematic Reviews
– ROBINS-I tool: Risk of Bias in Non-randomized studies of Int
erventions
– RoB 2.0 tool (revised tool for Risk of Bias in randomized tria
ls
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update As Needed
Extract data

• Extract reported findings from selected studies using

a data extraction form.
– Extraction forms and approaches should be determined by
the needs of the specific review.
– At least two review authors should independently extract
data from study reports.
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract Data
Analyze and present results
Interpret Results And Draw Conclusions
Update As Needed
Analyze and present results

• The findings from individual studies are aggregated

to produce a type of evidence synthesis appropriate
to the type of data within the review.
– Narrative synthesis – findings are summarized and
explained in words
– Quantitative/statistical synthesis – data from individual
studies are combined statistically and then summarized
(meta‐analysis)
Tables and Figures

• Tables and figures are used to present included

studies and their findings in a systematic and clear
format.
– Flow diagram
– Summary of findings table
– Forest plot
PRISMA Flow diagram

• PRISMA Flow diagram

– Depicts the flow of information through the different phases
of a systematic review
– Documents the number of studies that remain after each
stage of the selection process
– Maps the number of studies identified, included and
excluded, and the reasons for being excluded
– PRISMA Flow Diagram Generator
PRISMA Flow Diagram
Summary of findings table

• Summary of findings table

– Provides key information concerning the quality of evidence
– Depicts the magnitude of effect of the interventions
– Illustrates the sum of available data on all important
outcomes for a given comparison

Cochrane Handbook
Summary of findings table using GRADE methodology
Forest Plot

• Forest Plot
– A graphical display designed to illustrate the relative
strength of treatment effects in multiple quantitative
scientific studies addressing the same question (meta-
analysis)
 Forest Plot

Names of fictional studies on left; odds ratios and confidence intervals on right; odds ratios
(squares proportional to weights used in meta-analysis); summary measure (center line of
diamond); associated confidence intervals (lateral tips of diamond); solid vertical line of no effect
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract Data
Analyze And Present Results
Interpret results and draw conclusions
Update As Needed
Interpret results and draw conclusions

• Statement of findings, discussion and conclusions

– Information on all important outcomes, including adverse
outcomes
– Quality of evidence for each outcome
– How values and preferences may bear on balance of
benefits, harms, and costs of interventions
Stages of a systematic review

Develop A Focused Research Question

Define Inclusion And Exclusion Criteria
Search the Literature
Select Studies
Assess Study Quality
Extract Data
Analyze And Present Results
Interpret Results And Draw Conclusions
Update as necessary
Whether and when to update

• Decisions about whether and when to update a

systematic review
– The currency of the question asked
– The need for updating to maintain credibility
– The availability of new evidence
– Whether new research or new methods will affect the
findings

When and how to update systematic reviews, BMJ (2016)

Cochrane graphic illustration of the systematic review process
Registries, tools and archives

• Cochrane Collaboration
– Cochrane Library
• Cochrane protocols, systematic reviews, other reviews and trials
– Guides and handbooks
• Cochrane Handbook for Systematic Reviews of Interventions
• Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
• Cochrane Standards for conduct and reporting of new reviews of interventions
• GRADE Handbook
– Cochrane tools for assessing risk of bias
• ROBIS: Risk of Bias in Systematic Reviews
• ROBINS-I tool: Risk of Bias in Non-randomized studies of Interventions
• RoB 2.0 tool (revised tool for Risk of Bias in randomized trials
– Methodological Expectations of Cochrane Intervention Reviews
Registries, tools and archives

• GRADE Working Group

– GRADE (Grading of Recommendations, Assessment,
Development and Evaluation)
• Method of grading quality of evidence and strength of
recommendations in guidelines
• GRADEpro software can be used to create Summary of Findings
tables
Registries, tools and archives

• PRISMA (Preferred Reporting Items for Systematic

Reviews and Meta-Analyses)
– PRISMA checklist
• 27 items for inclusion in systemic reviews and meta-analyses
– PRISMA flow diagram
• Diagram depicting flow of information and number of records
through phases of a systematic review
• Prospero
– International prospective register of health-related
systematic reviews
Software for systematic reviews

• Software
– Software for Systematic Reviewing
– Software Programs for Preparing and Maintaining Systematic Reviews
– Systematic Review Toolbox
Special Services offered at the NIH
Library:  A quick overview

Judith Welsh, BSN, MLS

NIH Library
August 2017
Learning Objectives

1. What special services are offered at the NIH Library?

Special Services

• 3D Printing—Verma Walker
• Bibliometrics—Chris Belter; Ya-Ling Lu
• Bioinformatics—Lynn Young
• Data Services—Lisa Federer
• Editing—Cindy Clark
– Plagiarism Checking software
• EndNote site license
• Technology Hub—Doug Joubert
• Translations—Monica Valencia; Doug Doty
3D Printing and Modeling

• NIH Library offers free 3D printing to NIH staff

• Printers
– Models must be used for NIH research
– Self service
– First come, first served
– 60 minute orientation required
– Located in Technology Hub
– https://nihlibrary.nih.gov/services/3d-printing-service
3D Printing and Modeling

• Modeling software
– Several open source and commercial 3D modeling software
packages are available on Technology Hub computers
• Reservations are required
• NIH 3D Print Exchange
– Virtual collection of bioscientific 3D models and tutorials
for 3D printing
• http://3Dprint.nih.gov
• Prosthetics Collection

Ribbon
Structure
Bibliometrics Service Program

• Provides publication analysis services to NIH staff

– Measure productivity
– Detect collaboration
– Identify research topics
– Assess citation impact
– https://nihlibrary.nih.gov/services/bibliometrics
Bibliometrics Service Program

• Measure productivity
– Identify publications by scientists, laboratories, and grant
programs
– How much research? Change over time?
Bibliometrics Service Program

• Detect collaboration
– Generate individual and institutional co-authorship
networks
– Which scientists and institutions are working together?
Bibliometrics Service Program

• Identify research topics

– Analyze topical and structural links between papers
– How are papers by a lab, program, or discipline related?
Change over time?
Bibliometrics Service Program

• Assess citation impact

– Calculate citation scores
– How useful has research published by your lab, division, or
program been to other scientists?
Topic analysis for RMD : Chris Belter

Dots in the map

represent papers
by the Rehab
Medicine
Department, dot
size indicates the
paper’s times
cited count, and
dot color
indicates the
paper’s topic as
described on the
right of the topic
page.

Lines indicate
shared
references
between the
connected
papers.
Bioinformatics Support Program

• Supports NIH researchers by assisting in the analysis

and interpretation of genomic data to reveal the
molecular mechanisms of disease
– Consultations
– Training
– Tutorials
– Powerful tools and bioinformatics resources are available
via the NIH Library website and Bioinformatics
workstations located in the library reading room
– https://nihlibrary.nih.gov/services/bioinformatics-support
Data Services Program

• The Data Services program provides assistance to

NIH investigators throughout the research process
from project planning to post project preservation
– Locating existing data and processing datasets for reuse
– Organizing and describing data to facilitate analysis,
collaboration, and sharing
– Data mining and visualization
– Retention, long term preservation, and sharing data
– https://nihlibrary.nih.gov/services/data-services
Data Services Training and Classes

• Training and classes

– Classes, tutorials and consultations are offered on a variety
of data-related topics from basic data management to
more in depth classes on specialized topics.
• General data management courses
• R statistical programming courses
• R data visualization programming courses
Data Sciences Workstation

• Data Sciences Workstation

– The Technology Hub includes a Data Sciences Workstation
and collaboration pods
• Tools for data analysis, processing and visualization of different
types of data
Data Visualization Service

• The Data Visualization Service provides

– Access to licensed and open source software
– Use of a large touch display for visualizing data
• 65" interactive digital display used to visualize and demonstration
data
– Data visualization community of practice
EndNote X8

• The NIH Library has acquired a one-year enterprise

wide license for EndNote through December 31,
2017.
• This license allows anyone directly associated with
NIH to download and/or upgrade to the latest edition
of EndNote.
• The software can be installed up to three computers
per user—work and home.
– https://nihlibrary.nih.gov/about-us/news/endnote-x8-now
-available
Technology Hub

• The Technology Hub

– Experience ways to use cutting edge technology
• 3D Printing and Modeling
• Data Visualization Touchscreen
• Collaborative Workspaces
• Digital Production Studio
• Smartpens
• Virtual Reality
• https://nihlibrary.nih.gov/services/technology-hub
Translations

• NIH Library Translations Office

– Work related documents
• Personal documents
• Medical and Scientific documents
• Articles
• Service Costs
– Free
• From English to Russian
• From German to English
• From Spanish to English
• From Italian to English
• From French to English
– All other translation rates
• 15 to 25 cents per word
Writing Center and Editing Service

• NIH Library Writing Center

– Virtual access
• Grammar and punctuation
• Journal impact factor
• H-index
• Copyright
• Publishing process
– Physical Writing Center
– Laptops
– Editing service
– Plagiarism checking service
– https://nihlibrary.nih.gov/services/editing