Systematic Reviews: What? Why,?

and
How?
Dr.C.Susila,Msc (N),PhD, MBA,
Principal
Billroth College of Nursing
Chennai
 introduction
Mr Smith is 64 years old and recently diagnosed with atrial
fibrillation (AF), a condition associated with a high risk of
stroke. You wish to know if prescribing warfarin will reduce
his risk of stroke?

How will you answer this?

Conduct a trial by own?
Search and appraise a relevant RCT?
Conduct a literature review?
 WE NEED SOME EVIDENCE TO PRACTICE
 EBP and clinical practice guidelines
 Healthcare decisions should be informed by the best
available research evidence.
◦ Many guidelines rely on previously published research reviews
◦ Practice guideline quality is dependent on rigorous review methods
and high quality primary studies
 Primary studies Randomized controlled trials
 Systematic reviews Evidence based practice and \
clinical practice guidelines
EBM and Systematic Review
 EBM (quick & dirty)  Systematic Review
Steps Steps
1. Question (PICO)? 1. Question (PICO)
2. Find the best evidence? 2. Find the best evidence x 2+
3. Appraise? 3. Appraise x 2+
4. Synthesized? 4. Synthesize
5. Apply? 5. ---

Time: 6 months+, team

Time: 120 seconds
< 2,000 articles
1 - 20 articles
This patient is dead
This patient survives!

Find a systematic review (and appraise it quickly)!

Information overload
Evidence based pyramid

Reviews
 What is a systematic review?

 Uses transparent procedures to find, evaluate and

synthesize the results of independent studies.
 Procedures are explicitly defined in advance, to assure

that the process is transparent and can be replicated.

 This process is also designed to minimize bias.
What is a systematic review?
“A systematic review is a review in which there is a comprehensive
search for relevant studies on a specific topic, and those identified are
then appraised and synthesized according to a predetermined and
explicit method.”* (*Klassen et al. Guides for reading and interpreting
systematic reviews. Arch Pediatr Adolesc Med 1998;152:700-704.)

A systematic review attempts to collate all empirical evidence that fits

pre-specified eligibility criteria in order to answer a specific research
question. It uses explicit, systematic methods that are selected with a
view to minimizing bias, thus providing more reliable findings from
which conclusions can be drawn and decisions made (Antman 1992,
Oxman 1993)
 Why we need systematic reviews?

Known and unknown areas of study

• Minimise the impact of bias/errors
• Can help to end confusion
• Highlight where there is not sufficient evidence

• Combining findings from different studies can highlight new

findings
• Can mitigate the need for further trials
 Why we need systematic reviews?

• Health care providers, researchers and policy makers are

inundated with unmanageable amounts of information
– Over 20 million citations in PubMed
– Approx. 75 to 100 RCTs published daily
– Usually impossible to consider all relevant individual
primary research studies in a decision making context
• Enable practitioners to keep up to date with evidence
accumulating in field and to practice evidence-based medicine
 Example

• Protocols in neurology units - 80% still

recommend bed rest after LP
• Systematic review of 10 trials of bed rest after
spinal puncture
– no change in headache with bed rest
– Increase in back pain

Serpell M, BMJ 1998;316:1709–10

Unique characteristics of a systematic review

 A systematic review must have:

◦ Clear inclusion and exclusion criteria
◦ Explicit search strategy
◦ Systematic coding and analysis of included studies
◦ Meta-analysis (where possible)
 Why not traditional reviews?
• ‘Unscientific’ rarely pre-specify or make methods to explicit

• Rarely transparent or reproducible

• Usually qualitative, subjective, opinions of individual

• Often incomplete, filing cabinet or MEDLINE review

• Difficult to make sense across groups of studies, especially when

conflicting based on qualitative reading alone
 Comparison of traditional and systematic reviews

Traditional review Systematic review

 Subjective
 Objective
 Narrow question
 Broad question  Methods clearly defined
 Methods unclear  Studies appraised
 Who undertakes systematic reviews?
• Multidisciplinary teams
– Clinicians
– Health services researchers
– Information scientists
– Statisticians
– Health Economists
– Patient and public involvement
 What are the steps in performing a systematic review?
Data extraction /checking
Define research/review question Develop data extraction from into which study information and outcome
In consultation/collaboration with the clinical community, data can be extracted, checked & verified
commissioners and patient/public representatives

Study assessment/appraisal
Develop review protocol Assess the quality and validity of the included studies using the pre-defined
Pre-specify the type of studies to be included, the methods of method.
collating, appraising and analysing data

Synthesis
Identify relevant studies Narratively and/or statistically summarise/describe the data, exploring
Develop a comprehensive search strategy and undertake similarities and differences between studies.
systematic searches of the literature

Assess eligibility
Select those studies which meet the pre-defined inclusion Knowledge translation
criteria Review details and results are disseminated to relevant target audiences
using appropriate formats
Cochrane graphic illustration of the systematic review process
?
Delay or not delay?
step 1 Develop a focused research question

• A clearly defined, focused systematic review begins

with a well formulated research question.
• Delayed surgeries are better than immediate surgeries?
• Recurrence of injury and return to former activity is
related to time of surgery?
 Step 1: Problem formulation
• W question should be focused.
• How are these two questions different?
– Does homework improve student achievement?
– In high school students who are below average performers,
does daily homework lead to better performance on unit
tests, compared to no homework?
FINER Criteria

 A research question should be:

◦ Feasible
◦ Interesting
◦ Novel
◦ Ethical
◦ Relevant
 PICO Framework

 A well-established format for structuring research

questions is known by the acronym PICO.
◦ Patient or Population
◦ Intervention or Indicator
◦ Comparator or Control
◦ Outcome
Our clinical question
Population

Amongst adults with acute ACL injuries, does

Intervention

early reconstructive surgery compared with

Control

delayed reconstructive surgery lead to

Outcome 1

favourable
Outcome 2 return to former activity and/or

risk of recurrent knee injury?

Research Question-purpose

 The research question guides the author in working

through many stages of the systematic review process
◦ Defining inclusion and exclusion criteria
◦ Searching the literature
◦ Selecting studies
◦ Extracting data
◦ Analyzing and presenting results
Step 2 Develop review protocol

Develop review protocol

Pre-specify the type of studies to be included, the
methods of collating, appraising and analysing
data
 How to Write a protocol?
• Forces you to read and understand the background
• Makes you to formulate a focused question
• Makes you to plan your information retrieval strategy
• Makes you to think through and describe
inclusion/exclusion criteria clearly
• Makes you to think about the data you want to collect
and the methods you will use to analyze them
Define inclusion and exclusion criteria

 One of the features that distinguish a

systematic review from a traditional review is
the pre-specification of inclusion and
exclusion criteria.
Inclusion criteria

 Inclusion criteria are a combination of

• Aspects of the research question
 Population
 Intervention
 Comparison
 Outcome
• Study type
 Randomized controlled trials
 Observational studies
 Step 3 Identify relevant studies
Search the literature

Develop a comprehensive search strategy and undertake systematic

searches of the literature

 The goal of the literature search is to discover all studies that meet the
inclusion criteria
◦ Search comprehensively
 Terminology
 Databases
◦ Search for grey literature
 Not commercially published
◦ Search for unpublished studies
 Reduce risk of publication bias
 Search strategy

 The search strategy should be designed to identify the

maximum number of studies relevant to the research
question.
◦ The search strategy should be systematic, transparent and
reproducible.
◦ Database specific controlled vocabulary terms and all relevant
text words should be included in the search strategy.
 Recommended databases

 The Cochrane Collaboration recommends searching the

following databases (at minimum):
◦ PubMed
◦ EMBASE
◦ Cochrane Central Register of Controlled Trials
 Supplementary databases

 Interdisciplinary databases
◦ Scopus
◦ Web of Science
 Specialized databases
◦ CINAHL Plus
◦ PsycINFO
 Supplementary searching techniques

 Search cited and citing references

◦ Scopus
◦ Web of Science
Supplementary searching techniques

 Hand search selected journals and conference

proceedings
 Conduct author searches for recent articles

written by topic experts

Unpublished and grey literature

 The inclusion of unpublished and grey literature may

minimize the potential effects of publication bias.
◦ Publication bias
 Occurs when the outcome of an experiment or research study
influences the decision about whether—or how quickly—the
manuscript may be published
 What is grey literature?

 Grey literature refers to academic, business, government

or industry print or electronic literature that is not
controlled by commercial publishers.
◦ Conference proceedings
◦ Research reports
◦ Government reports
◦ Dissertations, theses
◦ Research monographs
 Sources of grey literature

 Conference proceedings
◦ EMBASE
◦ Scopus
◦ Web of Science
◦ Google
Sources of unpublished literature

 Clinical trials
◦ ClinicalTrials.gov
◦ Centerwatch.com
◦ EU Clinical Trials Register
◦ ISRCTN Registry
◦ OpenTrials
◦ WHO International Clinical Trials Registry Platform
Compile search results

 Compile search results using reference

management software (EndNote or
Mendeley)
 Remove duplicate records
 Document search process

 Document the search process

◦ Databases
◦ Dates searched
◦ Search strategies
◦ Limits (date ranges, publication types, language restrictions)

 Step 4 Select studies

 Identification of studies meeting inclusion

criteria should be done independently by two
review authors.
 Review titles and abstracts of retrieved citations.
 Review full text of studies which are found to meet the
inclusion criteria.
 Keep a record of reasons for inclusion or exclusion.
Tools for sorting

 Tools for sorting

◦ Reference management software
 EndNote
◦ New web application
 Rayyan
◦ Excel
 Step 5 Extract data

 Extract reported findings from selected studies using a

data extraction form.
◦ Extraction forms and approaches should be determined by the
needs of the specific review.
◦ At least two review authors should independently extract data
from study reports.
 Data Extraction

• Be clear what information you want from the studies:

– Study details
– Data for your analysis

• Information need to be collected relating to:

– Methodology
– Population
– Interventions being compared
– Outcomes evaluated
 Give consideration to….

• What effect or measures you are going to calculate

– What data do you need to do this?
• How are you planning to group studies for the analysis?
– By intervention?
– By study design?
• What information do you need to extract to enable you to
organise and analyse the way you want?
• REMEMBER YOUR PROTOCOL – IT IS YOUR ROADMAP, FOLLOW
IT!
 How much to extract??

• Level of judgement is required

– Sufficient to describe studies
– Sufficient to allow you to undertake the planned analysis
– Sufficient so you do not need to return to the full text
papers
• However
– You need to limit unnecessary detail
 Data extraction software?
• Main considerations are probably
• What are you familiar with?
• What package best suits your data?
• How many included studies do you have?
• Word
• Excel
• Access
• EPPI reviewer
• COEVIDENCE
• Consistency/Standardisation is essential
 Things to consider
• Are you including more than one study design?
• You may need separate forms for each study design
• However, you are still answering the same question, so make sure the core
information extracted is the same

• Have one or a few studies reported data differently from the others?
• Will the data still be useful?
• Should you include it?
• Make sure the core information extracted is the same

• You may need to update the form, or have more than one form
• Any changes need to be agreed and made consistently
 Stay on track……
• Be careful about collecting ‘extra’ data
• It is very tempting to collect data that are not directly relevant to the
review question
• The data needed to answer the review question should have already been
decided (REMEMBER YOUR PROTOCOL)
• Collect data for good reasons – stay focused and don’t get side-tracked

• Time and effort to collect, only to find it is not useful

 Step 6 Assess study quality

 Following the full text review, assess the selected

studies for risk of bias and study quality.
Assess eligibility
Select those studies which meet the pre-defined
inclusion criteria
Quality Assessment & Critical Appraisal
• Why bother????
• What are we trying to achieve?
• Not all published and unpublished literature is rigorous!
– being in a journal doesn’t mean it is good
• Quality may be used as an explanation for differences in
study results or to guide interpretation of findings,
strength of inferences
Quality Assessment & Critical Appraisal
• Quantitative studies
– Internal validity
– Bias: selection; performance; detection; attrition; reporting
– External validity
• Move away from checklists/numerical scores to domain based
assessment
– Cochrane Risk of Bias - RCTs
– QUADAS 2 – diagnostic accuracy
– ROBIS for systematic reviews
 Quality Assessment & Critical Appraisal
• Qualitative studies
• Three broad categories
– Rigour: has a thorough and appropriate approach been applied
to key research methods in the study?
– Credibility: are the findings well presented and meaningful?
– Relevance: how useful are the findings to you and your
organisation?
Tools for critical appraisal

 CASP: Critical Appraisal Skills

Programme Checklists
 Critically Appraised Topics:
generic systematic reviews
(ACP Journal club)
 SIGN: Scottish Intercollegiate
Guidelines Network
 CASP appraisal checklist

1. Clear aims of research (goals, why it is

important, relevance)
2. Appropriate methodology
3. Sampling strategy
4. Data collection
5. Relationship between researcher and
participants
6. Ethical issues
7. Data analysis
8. Findings
9. Value of research (context dependent)
Tools for assessing risk of bias

 Tools for assessing risk of bias

◦ ROBIS: Risk of Bias in Systematic Reviews
◦ ROBINS-I tool: Risk of Bias in Non-randomized studies o
f Interventions
◦ RoB 2.0 tool (revised tool for Risk of Bias in randomized
trials
Step-7 synthesis - Analyze and present results

 The findings from individual studies are aggregated to

produce a type of evidence synthesis appropriate to the
type of data within the review.
◦ Narrative synthesis – findings are summarized and explained
in words
◦ Quantitative/statistical synthesis – data from individual studies
are combined statistically and then summarized (meta‐
analysis)
Data Synthesis
 Building up; putting together; making a whole
out of the parts; the combination of separate
elements of thought into a whole; reasoning
from principles to a conclusion
Narrative synthesis

 Instead
of/alongside
meta-analysis
 Potential bias in

presentation
 Lack of a take

home message
Tools for narrative synthesis
 Partly informed by methodological work in qualitative
synthesis
◦ Tabulation
◦ Groupings and clusters
◦ Vote counting as a descriptive tool
◦ Examination of moderator variables (elements of e.g. setting,
population)
 Rodgers et al Evaluation 2009 15 49-72
Tables and Figures

 Tables and figures are used to present included studies

and their findings in a systematic and clear format.
◦ Flow diagram
◦ Summary of findings table
◦ Forest plot
 PRISMA Flow diagram

 PRISMA Flow diagram

◦ Depicts the flow of information through the different phases of a
systematic review
◦ Documents the number of studies that remain after each stage of
the selection process
◦ Maps the number of studies identified, included and excluded,
and the reasons for being excluded
◦ PRISMA Flow Diagram Generator
PRISMA Flow Diagram
Summary of findings table

 Summary of findings table

◦ Provides key information concerning the quality of evidence
◦ Depicts the magnitude of effect of the interventions
◦ Illustrates the sum of available data on all important outcomes for
a given comparison

Cochrane
Handbook
Summary of findings table using GRADE methodology
 Forest Plot

 Forest Plot
 A graphical display designed to illustrate the relative strength
of treatment effects in multiple quantitative scientific studies
addressing the same question (meta-analysis)

 Forest Plot

Names of fictional studies on left; odds ratios and confidence intervals on right; odds ratios
(squares proportional to weights used in meta-analysis); summary measure (center line of
diamond); associated confidence intervals (lateral tips of diamond); solid vertical line of no
effect
step-8 translation of knowledge Interpret results and draw
conclusions

 Statement of findings, discussion and conclusions

◦ Information on all important outcomes, including adverse
outcomes
◦ Quality of evidence for each outcome
◦ How values and preferences may bear on balance of
benefits, harms, and costs of interventions
Step-9 Whether and when to update?

 Decisions about whether and when to update a

systematic review
◦ The currency of the question asked
◦ The need for updating to maintain credibility
◦ The availability of new evidence
◦ Whether new research or new methods will affect the
findings

When and how to update systematic reviews, BMJ (201

6)
 Registries, tools and archives

 Cochrane Collaboration
◦ Cochrane Library
 Cochrane protocols, systematic reviews, other reviews and trials
◦ Guides and handbooks
 Cochrane Handbook for Systematic Reviews of Interventions
 Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy
 Cochrane Standards for conduct and reporting of new reviews of interventions
 GRADE Handbook
◦ Cochrane tools for assessing risk of bias
 ROBIS: Risk of Bias in Systematic Reviews
 ROBINS-I tool: Risk of Bias in Non-randomized studies of Interventions
 RoB 2.0 tool (revised tool for Risk of Bias in randomized trials
◦ Methodological Expectations of Cochrane Intervention Reviews
Registries, tools and archives

 GRADE Working Group

◦ GRADE (Grading of Recommendations, Assessment,
Development and Evaluation)
 Method of grading quality of evidence and strength of
recommendations in guidelines
 GRADEpro software can be used to create Summary of
Findings tables
Registries, tools and archives

 PRISMA (Preferred Reporting Items for Systematic

Reviews and Meta-Analyses)
◦ PRISMA checklist
 27 items for inclusion in systemic reviews and meta-analyses
◦ PRISMA flow diagram
 Diagram depicting flow of information and number of records
through phases of a systematic review
 Prospero
◦ International prospective register of health-related systematic
reviews
Software for systematic reviews

 Software
◦ Software for Systematic Reviewing
◦ Software
Programs for Preparing and Maintaining Systema
tic Reviews
◦ Systematic Review Toolbox
Most important thing:
Be organised!!!
Gantt charts
Reference management

 Use a reference manger to sift and store

 Keep all citations retrieved

◦ Add in those you can’t download

 Use to de-duplicate results
 Sift citations for inclusion/exclusion

◦ Can use codes/notes

Version control
 Dates – YYYYMMDD
 Version numbering
◦ v0.1 = first draft
◦ v1.0 = final version
◦ v1.1 = minor amendments to final version
◦ v2.0 = major revision
 Avoid using draft, draft 1, final final, etc.
 Clear naming convention
◦ E.g. Date_project_title
 20/11_Autism HTA_resultsv1.2.doc
Other help available
 FMS Systematic review group
◦ Informal monthly session where methods are
discussed and issues can be raised
◦ fiona.beyer@ncl.ac.uk or jenni.hislop@ncl.ac.uk
◦ alternatively you have my contact details

 MSc in Public Health and Health Services

Research (~October 2016)
◦ 10 credit module ‘Introduction to systematic reviewing and
critical appraisal’
◦ pghealth@newcastle.ac.uk
 Limitations of Systematic
Reviews
• How do you capture qualitative distinctions between studies?
• Primary studies are reported poorly
• “Apples and oranges”
• Nearly all reviews include “flawed” studies to some extent
(e.g., a randomized design with attrition)
• Publication bias is always an issue
• Analysis of between- study differences is correlational or
“interocular”
Conclusion
 Selected References
Abrami, P.C., Wade, A. & Bernard, R.M. (2005). Evidenced-based Practice: Bridging the gap between educational
Cooper, H., Nye, B., Charlton, K., Lindsay, J.,& Greathouse, S. (1996). The effects of summer
vacation on achievement test scores: A narrative and meta-analytic review. Review of
Educational Research, 66, 227-268
Cooper, H., Robinson, J.C. & Patall, E.A. (2006). Does Homework Improve Academic
Achievement?: A Synthesis of Research, 1987-2003. Review of Educational Research.
Ennett, ST, Tobler, NS, Ringwalt, CL & Flewelling, RL(1994).How effective is drug abuse
resistance education? A meta-analysis of Project DARE outcome evaluations. American
Journal of Public Health, 84,1394-401.
Hanushek, E.A. 1989. "The Impact of Differential Expenditures on School Performance."
Educational Researcher 18(4): 45-65.
Hedges,L.V., R.D. Laine, and R. Greenwald. 1994. "Does Money Matter? A Meta-Analysis of
Studies of the Effects of Differential School Inputs on Student Outcomes." Educational
Researcher 23(3): 5-14
Glass, Gene V., and Mary Lee Smith. (1979) "Meta-Analysis of Research on Class Size and
Achievement." Educational Evaluation and Policy 1, 2-16.
Paterson, B. L., Thorne, S. E., Canam, C., Jillings, C. (2001). Meta-Study of Qualitative Health
Research. Thousand Oakes, CA: Sage
Shadish, W.R., Matt, G.E., Navarro, A.M., & Phillips, G. (2000). The effects of psychological
therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin,
126, 512-529.

Smith, Mary Lee, and Gene V. Glass. (1980) "Meta-Analysis of Research on Class Size and
Its Relationship to Attitudes and Instruction.“American Education Research Journal,
17,419-433
Whitehead, MF & Becker, BJ (2000). The Young Child’s Post-Divorce Adjustment: A Meta-
Analysis with implications for parenting arrangements. Journal of Family Therapy, 14, 5-26.
Thank you