Research Methodology:

Systematic Review
S.S.S. SHASUN JAIN COLLEGE
FOR WOMEN
IQAC – RESEARCH
METHODOLOGY
WORKSHOP
RESOURCE PERSON:
Prof. (Dr.) Paresh Shah
FCMA., Ph.D., D.Ed., D.D.M., D.Ed. Psy.
Alumnus of Indian Institute of Management, Ahmedabad,
(India); Institute of Cost Accountants of India; University of
London, UK; University of Illinois, US; Darden School of
Business, University of Virginia, US; IESE Business School,
University of Navarra, Spain; University of New South
Wales, Sydney, Australia; and Alison Education, US.
Cell no. 9824358505
EMAIL ID: profpareshshah@yahoo.co.in; paresh@profparesh.in
Isn’t a systematic review another name for
a comprehensive literature review?
• Literature reviews tend to be narrative- they
give an overview, and can be one-sided
• Systematic reviews follow an explicit protocol
to identify ALL the available evidence on a
given theme (positive and negative studies)-
they give practice guidelines
• Rejected evidence is catalogued with the
reasons for its rejection made explicit
On being “systematic” in literature
reviews
• “SLRs by their very definition differ from traditional
literature reviews: their scope is to address a highly
specific research question (e.g., 'what works' or
'what works best') for which evidence from the
literature is sought. SLRs thus do not aim to provide
what traditional literature reviews do: an
assessment of a state of knowledge in a problem
domain and identification of weaknesses and needs
for further research.” (Boell and Cecez-kecmanovic
2015)
Boell, S.K. and Cecez-kecmanovic, D. (2015) On being 'systematic' in literature reviews in IS. Journal of Information
Technology 30(2), 161-173.
Am I doing a systematic review?
• Publishable systematic reviews are typically done
by a team of researchers and search specialists
• They take several months to a year to complete
• They involve looking through thousands of
articles
• Maybe you want to do a literature review in a
systematic manner instead?

4 22 November, 2021
 Stages of a Systematic Review

1. Literature search (several!!)

2. Studies are quality appraised and data is
extracted
3. Data is synthesized in narrative form, and
sometimes as a Meta-analysis of numerical
data
4. Report produced
Sensitivity Vs Precision

• Systematic reviews need to be as extensive as

possible
• Need to strike balance between comprehensive
searching (sensitivity) and maintaining relevance
(precision) in your search
• But err on the side of sensitivity (and then reject
by hand based on your criteria)
Tools for writing your systematic review
report

•PRISMA: Preferred Reporting Items for

Systematic Reviews and Meta-Analyses
•Refer guidelines given in PDF
• http://prisma-statement.org/
What does a systematic review look
like
• Good examples in The Cochrane
Library. Mainly biomedical. Includes
existing Systematic Reviews and also
includes CENTRAL – Cochrane register
of controlled trials
• The Campbell Library does SRs in non-
biomedical subjects
• Social policy SRs: EPPI Centre
Searching for evidence: first things first
Electronic databases:
• Medline
• Proquest
• Scopus
• Etc, depending on your subject- PsycInfo for
Psychology, Business Source Premier for
Management…..
• https://www.bradford.ac.uk/library/subject-support/
Searching for evidence: onwards

• Key journals
• Bibliographies of systematic reviews
• Bibliographies of key journal articles
• Who has cited key journal articles
• Internet searches
• Grey literature
• Monographs- find through British Library, COPAC
• Book reviews – find through Summon, Proquest
Publication Bias

• Produces misleading results – need to

acknowledge it may be present (suspect it if
all the papers you find have positive
results)
• Journals choosing not to publish trials that
do not produce “interesting” findings
• Unpublished trials
• Negative or null effects
• “hidden” trials
Getting ready to search
• Focus your question
• Break it down into concepts
oPICOS {Population, Intervention, Comparison, Outcome, Study
type} or
oSPIDER {Sample, Phenomenon of Interest, Design, Evaluation,
Research Type}
• Use keywords / subject headings
• Structure your search
• Choose databases to search
• Run your search
• Refine your search
A useful way to start

• Have some articles already! (A “test set”.)

• From a Summon search (an On-line library), or some related
research you’ve done, or from colleagues…
• Two reasons this is helpful:
oThey will help you focus your search- you can look
through them for keywords and subject headings

oThey will help you check if your search is working- does

your search find your test articles? If not, how can you
change your search?
Generate the question

An example: Does stress lead to overeating,

and what support systems mitigate this?
Narrow down! E.g.:
• Short-term or chronic stress?
• Eating disorders, or changes in eating
behaviour?
• Adults, children, adolescents?
• Kinds of support- family, community,
professional?
Focus the Question: PICOS
P Population or Problem Describe the group of problem or problem

I Intervention (or E for What are you considering doing/using

Exposure)

C Comparison What is the alternative?

O Outcome What do you hope to achieve? How will it be

measured?

S Study type What would be the best kind of study to find

this?
Alternatives to PICOS
• If you are trying to understand rather than treat, some of
the elements of PICOS are less relevant
• Researchers have devised alternatives- one useful one is
SPIDER
• Sample
• Phenomenon of Interest
• Design
• Evaluation
• Research type
(Cooke A, Smith D and Booth A (2012) ‘Beyond PICO: The SPIDER Tool for Qualitative Evidence Synthesis’ Qualitative Health
Research 22 (10): 1435-43)
Alternatives to PICOS- ECLIPS(e)
Used for management
• Expectation
• Client group
• Location
• Impact
• Professionals
• Service

(Wildridge, v. and Bell, L. (2002) How CLIP became ECLIPSE: a mnemonic to assist in searching for health policy/management
information. Health information and libraries journal 19(2), 113-115.)
 Alternatives to PICOS- SPICE

• Setting
• Perspective
• Intervention
• Comparison
• Evaluation
(Booth, A. (2004). Formulating answerable questions. In Booth, A. and Brice, A (editors) Evidence based practice: an information
professional’s handbook. Facet, 61-70.)

• Look for guidelines from your own field

• Eg for computing: Kitchenham, B (2004) Procedures for performing systematic reviews, Keele
University Technical Report TR/SE-0401, Keele: Software Engineering Group, Department of
Computer Science, Keele University.
Examples
Patient/population Exposure Comparison Outcomes Study type

Adults Chronic stress Non-stressed group Change in eating

Or Stress with Or stress without pattern, weight gain
support system support system

(A more “medical-style” question with an intervention comparison)

Patient/population Intervention Comparison Outcomes Study type

Adults with chronic stress Teaching about Teaching about diet Weight loss
exercise

A more “investigatory” question using SPIDER

Sample Phenomenon of Interest Design Evaluation Research type

Adults with chronic Change in eating habits: Questionnaire, survey, Qualitative, mixed
stress variability among those focus groups, population method
with differing social study
support systems
Brainstorm alternative keywords
• Acronyms and synonyms
o Long-term stress, stress-induced, stress-related
Coping behaviours?
o Eating patterns, eating behaviours, appetite,
food habits, food intake, food choice, emotional
eating, soothing. Binge?
• Risk factors?
• Support systems/structures.
o Family, friends, community, society
Attitudes? Happiness, self-esteem?
Some aspects of database searching

• Subject headings
• And / Or search combinations
• Phrase and adjacency searches
• Truncation
• Field codes
Subject headings

• Medline: When an issue of one of the 5,000 journals

arrives with one of the Medline indexers, they read each
article and assign it headings (MeSH or Medical Subject
Headings) which describe the content
• Other databases may have subject headings (not called
MeSH!)
 An example- an article from Medline

Subject
headings
Combining keywords

• And: using and will search for articles containing both

keywords
• E.g. vision AND perception
• Or: using or will search for articles containing either one
keyword or the other keyword
• vision OR sight

Mnemonic (Patterns of letters, ideas or associations in remembering): Or is

More

• Not: using not will search for articles containing one

keyword and not the other
• inpatient NOT outpatients
Truncation

• Behav* will pick up behaviour, behaviours, behaving,

behaved etc.
• Also useful for authors’ names. Lodge T* will pick Lodge
T and Lodge TD
Phrase and adjacency searching

• Quote marks turn the words you are searching for into a
phrase- give me these words together in exactly this order.
➢E.g: “Generic prescribing” finds articles with that phrase,
but won’t find ones with generic in the title and prescribing
in the abstract.
• Sometimes that’s not what you want. Adjacency searching finds
the words near each other- you specify how far apart.
➢E.g: Generic* adj4 prescri* finds all articles with generic*
within four words of prescri*: so it will find prescribing
generics, prescribe a generic, prescriptions for lower cost
generics, generic versions of commonly prescribed drugs…
etc
Field Codes

• Databases such as EBSCO’s will allow you to search in a

particular field
• E.g.
✓AU=hardy m* (searches only in the “author” field)
✓JN=Journal of comparative psychology (searches only in the
“Journal title” field)

• Each database does field codes and adjacency searches

differently- look at their Help pages
After your first search

• MODIFY!!!
• Developing a search is an iterative process –terms used
are modified and added to based on what has been
retrieved
• Experts say results can be scanned at the rate of 120 per
hour
• SR’s are usually high yield and low precision – so have to
spend time scanning results
Getting your search right

• Don’t use too many different search concepts BUT use a

wide variety of synonyms and related terms
• Aim for high sensitivity and be prepared to accept low
precision
• Get your search right on one database before searching
others
• Revisit and modify your search
• Keep track of ALL your results- you will need to report how
many you got for each search
 Bibliographic and Citations

• Use bibliographic software (EndNote is the University’s

tool) to manage your references, identify duplicates (a
very big job if you find thousand of results) and aid
referencing
• You can do your screening of articles in EndNote if you
wish
• You can also do through MS WORD REFERENCES
FUNCTION – CITATION, BIBLIOGRAPHIC

30 22 November, 2021
Reference lists and hand searching

• Use reference lists of key articles and systematic reviews

• Use “cited by” in many databases and Google Scholar to find
newer articles
❖ This step WILL pick up articles you haven’t found in the
databases!
• Identify main journals
❖Hand search their contents – no matter how much of an
expert searcher you are, you could have missed
something
Grey Literature

 Research reports
 Policy documents
 Dissertations and theses
 Conference abstracts
 Unpublished research
 Company reports
Grey Literature
 Use Google’s advanced search features
https://www.google.com/advanced_search
Limiting by file type is useful- most reports will
be PDFs

 Identify key institutions in the field and do a quick look

through their publications lists
PRISM CHECKLIST
Locati
on
where
Section Item
Checklist item item
and Topic #
is
report
ed
TITLE
Title 1 Identify the report as a systematic review.

ABSTRACT
Abstract 2 See the PRISMA 2020 for Abstracts checklist.

INTRODUCTION
Rationale 3 Describe the rationale for the review in the context of existing knowledge.

Objectives 4 Provide an explicit statement of the objective(s) or question(s) the review

addresses.

34
 PRISM CHECKLIST
METHODS

Eligibility 5 Specify the inclusion and exclusion criteria for the review and how studies were
criteria grouped for the syntheses.

Information 6 Specify all databases, registers, websites, organisations, reference lists and
sources other sources searched or consulted to identify studies. Specify the date when
each source was last searched or consulted.

Search 7 Present the full search strategies for all databases, registers and websites,
strategy including any filters and limits used.

Selection 8 Specify the methods used to decide whether a study met the inclusion criteria
process of the review, including how many reviewers screened each record and each
report retrieved, whether they worked independently, and if applicable, details
of automation tools used in the process.

35
 PRISM CHECKLIST
Data 9 Specify the methods used to collect data from reports, including how many
collection reviewers collected data from each report, whether they worked
process independently, any processes for obtaining or confirming data from study
investigators, and if applicable, details of automation tools used in the
process.
Data items 10a List and define all outcomes for which data were sought. Specify whether all
results that were compatible with each outcome domain in each study were
sought (e.g. for all measures, time points, analyses), and if not, the methods
used to decide which results to collect.

10b List and define all other variables for which data were sought (e.g. participant
and intervention characteristics, funding sources). Describe any assumptions
made about any missing or unclear information.
Study risk of 11 Specify the methods used to assess risk of bias in the included studies,
bias including details of the tool(s) used, how many reviewers assessed each
assessment study and whether they worked independently, and if applicable, details of
automation tools used in the process.

Effect 12 Specify for each outcome the effect measure(s) (e.g. risk ratio, mean
measures difference) used in the synthesis or presentation of results.
 PRISM CHECKLIST
Synthesis methods 13a Describe the processes used to decide which studies were eligible for
each synthesis (e.g. tabulating the study intervention characteristics
and comparing against the planned groups for each synthesis (item
#5)).
13b Describe any methods required to prepare the data for presentation or
synthesis, such as handling of missing summary statistics, or data
conversions.
13c Describe any methods used to tabulate or visually display results of
individual studies and syntheses.
13d Describe any methods used to synthesize results and provide a
rationale for the choice(s). If meta-analysis was performed, describe
the model(s), method(s) to identify the presence and extent of
statistical heterogeneity, and software package(s) used.
13e Describe any methods used to explore possible causes of
heterogeneity among study results (e.g. subgroup analysis, meta-
regression).
13f Describe any sensitivity analyses conducted to assess robustness of
the synthesized results.
Reporting bias assessment 14 Describe any methods used to assess risk of bias due to missing
results in a synthesis (arising from reporting biases).

Certainty assessment 15 Describe any methods used to assess certainty (or confidence) in the
body of evidence for an outcome.
37
PRISM CHECKLIST
RESULTS
Study selection 16a Describe the results of the search and selection process, from the
number of records identified in the search to the number of studies
included in the review, ideally using a flow diagram.
16b Cite studies that might appear to meet the inclusion criteria, but which
were excluded, and explain why they were excluded.
Study characteristics 17 Cite each included study and present its characteristics.

Risk of bias in studies 18 Present assessments of risk of bias for each included study.

Results of individual 19 For all outcomes, present, for each study: (a) summary statistics for each
studies group (where appropriate) and (b) an effect estimate and its precision
(e.g. confidence/credible interval), ideally using structured tables or
plots.
Results of syntheses 20a For each synthesis, briefly summarise the characteristics and risk of bias
among contributing studies.
20b Present results of all statistical syntheses conducted. If meta-analysis
was done, present for each the summary estimate and its precision (e.g.
confidence/credible interval) and measures of statistical heterogeneity. If
comparing groups, describe the direction of the effect.
20c Present results of all investigations of possible causes of heterogeneity
among study results.
20d Present results of all sensitivity analyses conducted to assess the
robustness of the synthesized results.
11/22/2021 38
PRISM CHECKLIST
Reporting 21 Present assessments of risk of bias due to missing results (arising from reporting
biases biases) for each synthesis assessed.

Certainty of 22 Present assessments of certainty (or confidence) in the body of evidence for each
evidence outcome assessed.

DISCUSSION
Discussion 23a Provide a general interpretation of the results in the context of other evidence.

23b Discuss any limitations of the evidence included in the review.

23c Discuss any limitations of the review processes used.

23d Discuss implications of the results for practice, policy, and future research.

39
PRISM CHECKLIST
OTHER INFORMATION
Registration 24a Provide registration information for the review, including register name and registration
and protocol number, or state that the review was not registered.
24b Indicate where the review protocol can be accessed, or state that a protocol was not
prepared.
24c Describe and explain any amendments to information provided at registration or in the
protocol.
Support 25 Describe sources of financial or non-financial support for the review, and the role of the
funders or sponsors in the review.
Competing 26 Declare any competing interests of review authors.
interests

Availability of 27 Report which of the following are publicly available and where they can be found:
data, code template data collection forms; data extracted from included studies; data used for all
and other analyses; analytic code; any other materials used in the review.
materials

40
 Grey Literature: some sources
European Grey Literature OpenGrey
http://www.opengrey.eu/
EThOS for British theses http://ethos.bl.uk
Trial registers: eg
http://www.clinicaltrials.gov/ ;
http://www.who.int/ictrp/en/;
https://www.clinicaltrialsregister.eu/ctr-search/search
Most databases will find conference abstracts: eg
https://brad.idm.oclc.org/login?url=http://www.scopus.co
m
APA’s PsycExtra
http://www.apa.org/pubs/databases/psycextra/index.asp
x
So… Systematic Literature Searches
Include:
• Electronic database searches
• Reference lists and hand searching
• Consider grey literature

• Don’t cherry pick

• Keep revising your search
• Get it right before moving onto the next source

http://libguides.library.usyd.edu.au/content.php?pid=27940&sid=3
993050