Medicine and Internet

SESSION 7. SYSTEMATIC INFORMATION

SEARCH
Ta m a r i ( Te k l a ) D a k h u n d a r i d z e
tekladakhundaridze@gmail.com
tekla.dakhundaridze@atsu.edu.ge
The literature searching process

“is not an exact science but an art.”

Samuel Butler

Try to think of the process as a

‘journey not a destination’ Hearst 1999
What is a systematic review?
SYSTEMATIC: Done or acting according to a fixed plan or system:
methodical

REVIEW: A critical appraisal of a book, play or other work

What is a systematic review?
“A systematic review is a review in which there is a comprehensive search for relevant studies on
a specific topic, and those identified are then appraised and synthesized according to a
predetermined and explicit method.”* (*Klassen et al. Guides for reading and interpreting systematic reviews. Arch
Pediatr Adolesc Med 1998;152:700-704.)

A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility
criteria in order to answer a specific research question. It uses explicit, systematic methods that
are selected with a view to minimizing bias, thus providing more reliable findings from which
conclusions can be drawn and decisions made (Antman 1992, Oxman 1993)
Why we need systematic reviews

• Minimise the impact of bias/errors

• Can help to end confusion

• Highlight where there is not sufficient evidence

• Combining findings from different studies can highlight new

findings
• Can mitigate the need for further trials
Why we need systematic reviews?
• Facilitate rational decision making
• Health care providers, researchers and policy makers are inundated with
unmanageable amounts of information
– Over 20 million citations in PubMed
– Approx. 75 to 100 RCTs published daily
– Usually impossible to consider all relevant individual primary research studies in a decision
making context

• Enable practitioners to keep up to date with evidence accumulating in field

and to practice evidence-based medicine
Why not traditional reviews?
• ‘Unscientific’ rarely pre-specify or make methods explicit

• Rarely transparent or reproducible

• Usually qualitative, subjective, opinions of individual

• Often incomplete

• Difficult to make sense across groups of studies, especially when

conflicting based on qualitative reading alone
Hierarchy of evidence
Who undertakes systematic reviews?

• Academics/researchers/Clinicians
• MSc/PhD students
• Regulatory bodies
• Health Technology Assessment
• Cochrane/Campbell Collaboration (NGOs)
Who undertakes systematic reviews?
• Multidisciplinary teams
– Clinicians
– Health services researchers
– Information scientists
– Statisticians
– Health Economists
– Patient and public involvement – particularly for
guidelines
 Key Stages in a Systematic Review- the process
Data extraction /checking
Define research/review question Develop data extraction from into which study
In consultation/collaboration with the information and outcome data can be extracted,
clinical community, commissioners and checked & verified
patient/public representatives

Study assessment/appraisal
Develop review protocol
Assess the quality and validity of the included
Pre-specify the type of studies to be
studies using the pre-defined method.
included, the methods of collating,
appraising and analysing data

Identify relevant studies
Develop a comprehensive search strategy
and undertake systematic searches of the
literature
Synthesis
Narratively and/or statistically summarise/describe
the data, exploring similarities and differences
between studies.

Assess eligibility
Knowledge translation
Select those studies which meet the pre-
Review details and results are disseminated to
defined inclusion criteria
relevant target audiences using appropriate formats
Define research/review question

• Questions may be broad or narrow

• Well-formulated questions will guide many aspects of
the review process
– Searching strategy
– Inclusion/exclusion criteria
– Data extraction
– Choice of synthesis method
– Presentation/dissemination of findings
 Current guidance

• a clear and concise statement of a review's

objectives (or questions) is critical and should begin
with a precise statement of the primary objective,
including the interventions reviewed and the targeted
problem; ideally, this would be presented in a single
sentence

Cochrane & Prisma Statment

Current guidance

“To assess the effects of [intervention or comparison]

for [health problem] in [types of people, disease or
problem, and setting if specified].”

• Several criteria/frameworks proposed to help guide

question development
Question formulation

• Determining the scope is a decision dependent upon multiple

factors:
– Perspectives regarding a question’s relevance and potential impact;
– Supporting theoretical, biologic and epidemiological information;
– The potential generalizability and validity of answers to the questions;
– Available resources;
– The wider literature base – has a recent high-quality SR been conducted?
Question formulation

• Advantages and disadvantages to both broad and narrow questions

• The validity of very broad question may be criticized for ‘mixing apples and
pears’; but advantages might include
– Comprehensive summary of the evidence
– Generalizability of findings
• Most obvious advantage of narrow focus is clarity of objectives and ease of
reading; but disadvantages might include
– Sparse evidence may limit findings/usefulness
– Generalizability of findings
Question formulation

• Often dealing with complex interventions

• Might be a need to develop working definitions of the intervention of interest
– Several options on how to do this (pragmatic real world v theoretical, logic
models, etc.)
– Use content experts outside the review team to ensure that the resulting
definitions are likely to be robust and meaningful
Protocol Development

• A protocol is an essential component of the

systematic review process
• Helps to ensure careful a priori planning
– Consistency
– Transparency
– Integrity
• Integral part of the process for leading
organisations/publication process
 Protocol Development

• One of the features that distinguish a systematic review

from a narrative review is the pre-specification of criteria
– Inclusion
– Exclusion
– Methods
– Outcomes to be synthesised
– Etc.
Searching for Information
MESH terms and key words/synonyms
◦ Medical Subject Heading – controlled vocabulary thesaurus used for indexing
articles
◦ young; adoles*; teen*; child*...................
*end of the ‘stem’ of the word it will automatically search for all the endings
for that word stem

Child* will also return children, childbearing, childbirth and so on…

Searching for Information
Word variants
◦ AIDS
◦ acquired immunodeficiency syndrome
◦ acquired immuno-deficiency syndrome
◦ acquired immune deficiency syndrome
◦ acquired immune-deficiency syndrome

Synonyms e.g. Newborn: infant, toddler, baby, etc.

Plurals e.g. child : children OR teenager : teenagers

Spelling variants (UK vs US) e.g. randomise/randomize

Searching for Information
Where to search
◦ Electronic databases: Pubmed, Medline, etc.
◦ Grey literature, dissertations, theses, conference proceedings, national bodies
(NICE, HTA), clinical trial database (www.clincialtrails.gov/)

◦ Look at the databases own guidance for searching they vary!

Boolean operators
Selection of Studies
Reference manager software package
◦ Endnote – RefMan – ProCite – Mendeley

Import results and screen

◦ Assess titles/abstracts against your predetermined criteria
◦ If in doubt include
◦ Retrieve full text articles of initial selections

Assess full text for inclusion

◦ Requires judgement (>1 reviewer)
◦ Check reviewer agreement (3rd review to resolve)
◦ Use a selection form to ensure consistency and record decisions
Example
Data Extraction/Quality Appraisal
Data Extraction
• Be clear what information you want from the studies:
– Study details
– Data for your analysis

• Information will need to be collected relating to:

– Methodology
– Population
– Interventions being compared
– Outcomes evaluated
 Give consideration to….

• What effect measures you are you going to calculate

– What data do you need to do this?
• How are you planning to group studies for the analysis?
– By intervention?
– By study design?
• What information do you need to extract to enable you to organise and
analyse the way you want?
• REMEMBER YOUR PROTOCOL – IT IS YOUR ROADMAP, FOLLOW IT!
How much to extract??

• Level of judgement is required

– Sufficient to describe studies
– Sufficient to allow you to undertake the planned analysis
– Sufficient so you do not need to return to the full text papers

• However
– You need to limit unnecessary detail
 Data extraction software?

• There is a wide selection of software to choose from

• Selection depends on a number of factors

• Main considerations are probably

• What are you are familiar with?
• What package best suits your data?
• How many included studies do you have?
Which software?

• Word
• Excel
• Access
• EPPI reviewer
• COEVIDENCE
• REVMAN
• ????
Consistency/Standardisation
• We all have to be doing the same thing
• Essential >one reviewer is extracting data

• Data must be interpreted in the same way by all reviewers

• Independent piloting of data extraction forms – always one standardised

form

• Regular discussion of progress/disagreements

• Regular comparison of data extraction – don’t wait till the end
Efficient data extraction

• Once data extraction is complete you may need to:

• Sort/search your data
• Filter data
• Calculate frequencies
• Transform data (e.g. SE to SD)

• Categorising/coding data will make these tasks easier

• Needs to be implemented with consistency by the whole team

• A database can be designed to have this functionality

Things to consider
• Are you including more than one study design?
• You may need separate forms for each study design
• However, you are still answering the same question, so make sure the core information
extracted is the same

• Have one or a few studies reported data differently from the others?
• Will the data still be useful?
• Should you include it?
• Make sure the core information extracted is the same

• You may need to update the form, or have more than one form
• Any changes need to be agreed and made consistently
Stay on track……
• Be careful about collecting ‘extra’ data
• It is very tempting to collect data that are not directly
relevant to the review question

• The data needed to answer the review question should

have already been decided (REMEMBER YOUR PROTOCOL)
• Collect data for good reasons – stay focused and don’t get
side-tracked

• Time and effort to collect, only to find it is not useful

Quality Assessment & Critical
Appraisal
• Why bother????
• What are we trying to achieve?
• Not all published and unpublished literature is rigorous!
– being in a journal doesn’t mean it is good
• Quality may be used as an explanation for differences in
study results or to guide interpretation of findings,
strength of inferences
Quality Assessment & Critical Appraisal

• Quantitative studies
– Internal validity
– Bias: selection; performance; detection; attrition; reporting
– External validity
• Move away from checklists/numerical scores to domain based
assessment
– Cochrane Risk of Bias - RCTs
– QUADAS 2 – diagnostic accuracy
– ROBIS for systematic reviews
Quality Assessment & Critical Appraisal

• Qualitative studies
• Three broad categories
– Rigour: has a thorough and appropriate approach been applied
to key research methods in the study?
– Credibility: are the findings well presented and meaningful?
– Relevance: how useful are the findings to you and your
organisation?
Checklist

1. Clear aims of research (goals, why it is important, relevance)

2. Appropriate methodology
3. Sampling strategy
4. Data collection
5. Relationship between researcher and participants
6. Ethical issues
7. Data analysis
8. Findings
9. Value of research (context dependent)
Data Synthesis
Building up; putting together; making a whole out of the parts; the combination of separate
elements of thought into a whole; reasoning from principles to a conclusion
Data Synthesis
Results from different studies need to be synthesised
Are studies and results similar enough to be combined into a
single numerical result?
◦ NO – qualitative descriptive/narrative summary
◦ YES – quantitative meta-analysis

Heterogeneity
◦ Difference in results can arise due to differences in study design, population,
selection, intervention delivery
◦ How similar is similar? Results from heterogeneous studies should not be pooled
Meta analysis
A meta-analysis is a statistical analysis that combines the results of multiple scientific
studies. Meta-analysis can be performed when there are multiple scientific studies addressing
the same question, with each individual study reporting measurements that are expected to
have some degree of error
“Meta-analysis is a quantitative, formal, epidemiological study design used to systematically
assess previous research studies to derive conclusions about that body of research. Outcomes
from a meta-analysis may include a more precise estimate of the effect of treatment or risk
factor for disease, or other outcomes, than any individual study contributing to the pooled
analysis:
Haidich A. B. (2010). Meta-analysis in medical research. Hippokratia, 14(Suppl 1), 29–37.
Meta-analysis/forest Plot
Most important thing:
Be organised!!!
Gantt charts
Reference management
Use a reference manger to sift and store
Keep all citations retrieved
◦ Add in those you can’t download

Use to de-duplicate results

Sift citations for inclusion/exclusion
◦ Can use codes/notes
Top Tips
Keep notes of searches and results
Scope and Re-Scope
Test and Re-Test- gold standard
Searching is never fully transparent, nor reproducible but make it Rigorous.
Version control
Dates – YYYYMMDD
Version numbering
◦ v0.1 = first draft
◦ v1.0 = final version
◦ v1.1 = minor amendments to final version
◦ v2.0 = major revision

Avoid using draft, draft 1, final final, etc.

Clear naming convention
◦ E.g. Date_project_title
◦ 20/11_Autism HTA_resultsv1.2.doc