12‐February‐2020

How to do a 
systematic review: a 
guide for HDR 
students and EMCR’s

Associate Professor 
Susan Torres
storres@deakin.edu.au

Deakin Lab for Meta‐analysis 
Research (DeLMAR)
Deakin University CRICOS Provider Code: 00113B

Deakin Lab for Meta Analysis 
Research (DeLMAR)

Vision: To become the leader in evidence 
based meta‐analysis approaches within 
Australia
https://www.deakin.edu.au/business/research/delmar

The effect of diet combined with exercise on health‐
related quality of life: a systematic review and meta‐
analysis of randomized controlled trials

Systematic review Finding/critically 
appraising the evidence

Meta‐analysis
Analysing the evidence

1
 12‐February‐2020

Outline
1. Different types of reviews
2. Why would you do a systematic review
3. Registering your review
4. Developing your review question
5. Developing the search terms and search 
strategy
6. Selecting your databases
7. Conducting your searches
8. Screening your studies

Outline

9. Data extraction
10. Assessing risk of bias
11. Synthesising the data
12. Guides for conducting systematic 
reviews
13. Systematic review exemplars
14. The outcome of your systematic review
15. Meta‐analysis courses
16. References

1. Different types of reviews

Factor Evidence  Mapping  Scoping  Rapid  Rapid  Systematic  Meta‐

Briefing Review Review Evidence  Realist  Review analysis
Assessment Review

Question Narrow Broad Narrow Narrow Narrow Narrow Narrow

Timescale 2 weeks 4‐16 weeks 24 weeks 8‐24 weeks 8‐24  36‐52 weeks 52‐78 weeks

weeks

Resources Single  Single Single  Double  Double  Double Double

reviewer, reviewer,  review,  reviewer,  reviewer,  reviewer,  reviewer, 
limited  comprehensive  limited  limited  limited  comprehensive  comprehensiv
databases databases databases databases databases databases e databases

Adapted from: Booth, A. (2016) EVIDENT Guidance for Reviewing the Evidence: a compendium of 
methodological literature and websites DOI: 10.13140/RG.2.1.1562.9842 

2
 12‐February‐2020

Scoping review

To identify types of evidence in a given field
To clarify key concepts/definitions in the literature
Examine how research is conducted on a certain topic 
or field
Identify and analyse knowledge gaps
As a precursor to a systematic review

Munn et al. Systematic review or scoping review? Guidance for authors when choosing between a 
systematic or scoping review approach.  BMC Medical Research Methodology (2018) 18:143

Scoping review

Review protocol
Transparent, peer reviewed search strategy
Include steps to reduce error and increase reliability 
(multiple reviewers) 
Standardised data extraction forms

Defining a systematic review

“A review of a clearly formulated question that 
uses systematic and explicit methods to identify, 
select and critically appraise relevant research, 
and to collect and analyse data from studies that 
are included in the review” (Cochrane 
Collaboration, 2003)

3
 12‐February‐2020

2. Why would you do a 
systematic review?
You have a research question that is best answered by 
synthesising existing evidence rather than conducting a 
new study
There is no recent review that answers that same 
question
To develop high level skills in searching and 
synthesising the literature

“High quality literature reviews bring 
together, synthesise and critique one or 
more literatures to provide an overall 
impression of the extent, nature and 
quality of evidence in relation to a 
particular research question, highlighting 
gaps” (Siddaway, 2019)

Time required to complete a
systematic review

Category Mean ± SD Median Range

Authors/team members 5±3 5 1–27

Time (in weeks; registered project start to 
67.3±31.0 65.8 6–186
publication date)

Quantitative analysis yield rate (%) 2.6±4.7 1.0 0.03–32.43

Qualitative analysis yield rate (%) 2.7±4.6 1.0 0.05–26.19

Borah R, Brown AW, Capers PL, et al Analysis of the time and workers needed to conduct systematic 

reviews of medical interventions using data from the PROSPERO registry BMJ 
Open 2017;7:e012545. doi: 10.1136/bmjopen‐2016‐012545

4
 12‐February‐2020

3. Registering your review
Prospero: https://www.crd.york.ac.uk/prospero/
Search for any existing similar systematic reviews
The protocol should specify the following:
Background
Review question
Inclusion and exclusion criteria
Databases and search terms
Participants/population
Intervention/control
Study design (eg RCT, cohort)
Main outcome
Data extraction
Risk of bias (quality) assessment
Strategy for data synthesis

Other organisations to register 
your review
Cochrane Library 
https://www.cochranelibrary.com/

Joanna Briggs Institute https://joannabriggs.org

Campbell Collaboration 
https://www.campbellcollaboration.org/

4. Developing your review question

PICOS
Participants
Intervention
Comparison
Outcome
Study design

5
 12‐February‐2020

Participants Intervention

What is the effect of weight loss on psychological 
measures of stress in randomized controlled trials 
that induced weight loss by dietary restriction 
compared to usual diet in overweight and obese 
adults?

Comparison Outcome Study design

16
Deakin University CRICOS Provider Code: 00113B

5. Developing your search terms
and search strategy
Spend time developing your search terms and 
strategy
Talk to your Faculty librarian about developing 
your search strategy
Systematic and systematic‐like review toolkit
http://deakin.libguides.com/c.php?g=638978&p=4473
298
Developing the research question
Developing the search
Selection and screening of articles

Advanced Search Guide‐ Health
http://deakin.libguides.com/ld.php?content_id=29454607

Example search strategy

What is the effect of diet combined with exercise compared to 
diet or exercise alone on health‐related quality of life in 
community dwelling adults?
quality of life
resistance training
diet
randomised controlled trial

6
 12‐February‐2020

19
Deakin University CRICOS Provider Code: 00113B

20
Deakin University CRICOS Provider Code: 00113B

6. Selecting your databases

Consult with your Faculty Librarian
3‐4 databases is usually sufficient

Note that Google Scholar will give different 
results to different people with identical 
search terms!  
This is a big problem for a systematic 
review.

7
 12‐February‐2020

Grey literature

Theses, government reports, white papers, 
unpublished trials
PsycEXTRA, Informit

Hand searching

Review contents page of relevant journals
Review reference list of all articles included 
in your final search
Search for articles which have cited articles 
included in your final search (Scopus)

8
 12‐February‐2020

25
Deakin University CRICOS Provider Code: 00113B

7.  Conducting your search

Register and save your searches in the 
databases
Export to separate Endnote 
libraries/separate folders in one library
Combine all the articles in one Endnote 
library
Remove all duplicates
Document your process

Register and save your searches

27
Deakin University CRICOS Provider Code: 00113B

9
 12‐February‐2020

Documenting your searches

Diet, exercise and QOL: October 2019
Medline Complete: 1696
PsycINFO: 195
Embase: 1709
Total= 3600
Total (duplicates removed)= 3250

28

A Note on Documentation

Everyone has had a computer disk crash at some point

Keep a running record of every step of your search process 
in at least four places

1. On your hard drive

2. On your web based email

3. On a backup drive that is NOT in your office

4. On paper

29

Deleting duplicates in Endnote

Diet, exercise and QOL: October 2019
Medline Complete: 1696
PsycINFO: 195
Embase: 1709
Total= 3600
Total (duplicates removed)= 3110

30

10
 12‐February‐2020

References/ Find duplicates/
Cancel then send references to Trash

31
Deakin University CRICOS Provider Code: 00113B

References/ Find duplicates/

32
Deakin University CRICOS Provider Code: 00113B

Deleting duplicates in Endnote

Manually screen for duplicates 2 times

You will still miss duplicates! 
Ballin Marcel
Marcel Ballin

33

11
 12‐February‐2020

8. Screening your studies

This is the most time consuming part of the 
systematic review
Different tools to screen articles
Rayyan https://rayyan.qcri.org
Covidence https://www.covidence.org/reviews/active
remove duplicates
title/abstract screening 2 People
full text screening 2 People‐ with reason for 
prisma flow chart excluding paper
data extraction
risk of bias

Demonstration of Covidence

https://www.youtube.com/watch?v=G4JEQ_curT4

9. Data extraction

Use a standard data extraction process for 
all studies
Duplicate extraction of key study data 
where possible with a second researcher
Record source location for all extracted 
data
May need to contact authors for data

12
 12‐February‐2020

37
Deakin University CRICOS Provider Code: 00113B

38
Deakin University CRICOS Provider Code: 00113B

10. Assessing risk of bias

Risk of bias assessment determines overall 
quality of studies in your review
There are widely used tools for randomised 
and quasi‐experimental designs (Cochrane 
Risk of Bias Tool and RoB 2)
https://sites.google.com/site/riskofbiastool/welcome/rob‐2‐0‐
tool/current‐version‐of‐rob‐2
There is less consensus about appropriate 
tools for observational studies

13
 12‐February‐2020

https://methods.cochrane.org/bias/sites/methods.cochrane.org.bias/files/public/uploads/6.%20Assessing%20risk%20of%20
bias%20in%20included%20studies%20v1.0%20Standard%20author%20slides.pdf

Intervention studies

Random sequence generation
Allocation concealment
Blinding of participants and personnel
Blinding of outcome assessment
Incomplete outcome data
Selective reporting
Overall Risk of Bias

Random sequence generation
• occurs at the start of trial before    
allocation of participants
• avoids differences between groups
• Low risk: computer random number 
generator

14
 12‐February‐2020

Allocation concealment
• occurs at the start of trial during    
allocation of participants
• avoids selection bias
• Low risk: sequentially numbered, 
sealed envelopes

Blinding of participants and personnel 
• avoids different treatment of groups
• Low risk: no/incomplete blinding but 
outcome unlikely to be influenced

Blinding of outcome assessment
• avoids measurement of outcome 
affected by knowledge of the 
intervention received
• Low risk: no blinding, but 
measurement unlikely to be 
influenced

15
 12‐February‐2020

Incomplete outcome data
• complete outcome data for all 
participants is not available
• Low risk: no missing data, missing 
data balanced across groups and 
reasons similar

Selective reporting
• can lead to reporting bias
• statistically significant results more 
likely to be reported
• Low risk: protocol is available and all 
outcomes of interests to the review 
are reported in the pre‐specified way

Cochrane handbook: Chapter 8

https://training.cochrane.org/handbook/current

Cochrane training: Assessing risk of bias 
in included studies
https://bit.ly/317I2zu

https://methods.cochrane.org/bias/sites/methods.cochrane.
org.bias/files/public/uploads/6.%20Assessing%20risk%20of%
20bias%20in%20included%20studies%20v1.0%20Standard%2
0author%20slides.pdf
48
Deakin University CRICOS Provider Code: 00113B

16
 12‐February‐2020

Observational studies

Quality assessment tool for observational cohort 
and cross‐sectional studies
https://www.nhlbi.nih.gov/health‐topics/study‐quality‐
assessment‐tools

11. Synthesising the data

There are different approaches
Results
Study selection flow chart
General characteristics of the included 
studies
eg sample size, age of participants, country of origin
Tools to measure exposure eg diet
Tools to measure outcome eg stress

51 http://www.prisma‐statement.org/  Template for flow chart
Deakin University CRICOS Provider Code: 00113B

17
 12‐February‐2020

12. Guides for conducting a 
systematic review
PRISMA
http://www.prisma‐statement.org/
Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. 
The PRISMA statement for reporting systematic reviews and meta‐analyses of 
studies that evaluate healthcare interventions: explanation and elaboration. 
BMJ. 2009;339:b2700.
Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items 
for systematic reviews and meta‐analyses: the PRISMA statement. BMJ. 
2009;339:b2535.

Cochrane collaboration
https://training.cochrane.org/handbook/current
Version 6 October 2019

PRISMA Checklist: How to write
your systematic review 
http://www.prisma‐statement.org/

Title: Psychological stress reactivity and future 
health and disease outcomes: a systematic review 
of prospective evidence 

Protocol and registration: This review protocol was 
registered with PROSPERO on 19 Dec 2017 (Registration
ID CRD42017084161)
Eligibility criteria: Included studies were conducted in 
apparently healthy adult populations.  They were 
observational, prospective, population‐based cohort studies.

18
 12‐February‐2020

Study selection: Our primary search yielded 1719 
studies and we identified one additional study via our 
included studies.  
Risk of bias: For this body of work as a whole, study quality is 
generally high and risk of bias is generally low (Figure 3)

Evidence Synthesis Academy
www.evsynthacademy.org

Boland A, Cherry GM, Dickson R. 
Doing a systematic review. A 
students guide. 1st ed. London: 
Sage; 2014. 

19
 12‐February‐2020

Deakin library

https://www.deakin.edu.au/library/research

13. Systematic review 
exemplars
British Medical Journal
https://www.bmj.com/content/bmj/348/bmj.g3253.full.pdf
https://www.bmj.com/content/bmj/364/bmj.k4718.full.pdf
Qualitative systematic review
http://web.a.ebscohost.com/ehost/pdfviewer/pdfviewer?vid=1
&sid=37fd0fd6‐e2de‐4a68‐8fdb‐6692fb98f990%40sdc‐v‐
sessmgr03
New England Journal of Medicine

14. The outcome of your 
systematic review
HDR students: Confirmation document
HDR and EMCR: Publication

20
 12‐February‐2020

HDR students: confirmation
document

Literature 
review Systematic review
background
search strategy
results (Table)
discussion
Aim
Research proposal
Study 1 (Systematic review‐ one page overview)
Study 2 (Original study 1)
Study 3 (Original study 2)
Study 4 (original study 3)
61
Deakin University CRICOS Provider Code: 00113B

HDR students: final thesis

Chapter 1 Literature review
Chapter 2 Systematic review
Chapter 3 Original study 1
Chapter 4 Original study 2
Chapter 5 Original study 3
Chapter 6 General discussion

Outcome of the systematic 
review
Decide not to publish
Publish as a systematic review
Develop into a meta‐analysis

21
 12‐February‐2020

When does a systematic 
review become a meta‐analysis?
Statistical combination of results from two or 
more separate studies
When you have similar data across studies
eg mean changes, odds ratios
What is the effect of weight loss on stress?

15.  Meta‐analysis courses

Faculty of Health Biostatistics unit: Stata (free)
https://www.deakin.edu.au/students/faculties/faculty‐of‐
health/research/deakin‐biostatistics‐unit
DeLMAR (free) 
Cochrane: RevMan (free)
https://training.cochrane.org/online‐learning/core‐software‐cochrane‐
reviews/review‐manager‐revman
Comprehensive Meta‐Analysis: 3 day course ($3500) 
and software ($410 2 year licence) 
https://www.meta‐analysis‐workshops.com/
R: training? (free)
Available on Deakin Software Centre

Any Questions?

22
 12‐February‐2020

16. References

Booth, A. (2016) EVIDENT Guidance for Reviewing the Evidence: a compendium of 
methodological literature and websites DOI: 10.13140/RG.2.1.1562.9842
Borah R, Brown AW, Capers PL, et al Analysis of the time and workers needed to 
conduct systematic reviews of medical interventions using data from the PROSPERO 
registry BMJ Open 2017;7:e012545. doi: 10.1136/bmjopen‐2016‐012545
Siddaway AP et al. How to do a systematic review: a best practice guide for conducting 
and reporting narrative reviews, meta‐analyses, and meta‐syntheses. Annu Rev 
Psychol 2019 70:747‐70
Cochrane Collab 2003 http://community.Cochrane.org/glossary

23