Professional Documents
Culture Documents
Supported by sanofi-aventis
What is a
systematic
review?
● Systematic reviews have increasingly replaced traditional
Pippa Hemingway narrative reviews and expert commentaries as a way of summarising
PhD BSc (Hons) RGN research evidence.
RSCN Research Fellow
● Systematic reviews attempt to bring the same level of rigour to
in Systematic reviewing research evidence as should be used in producing that
Reviewing, School of research evidence in the first place.
Health and Related
Research (ScHARR), ● Systematic reviews should be based on a peer-reviewed protocol so that
they can be replicated if necessary.
University of Sheffield
Nic Brereton PhD BSc ● High quality systematic reviews seek to:
(Hons) Health
● Identify all relevant published and unpublished evidence
Economist, NB Select studies or reports for inclusion
●
Consulting Services,
● Assess the quality of each study or report
Sheffield
● Synthesise the findings from individual studies or reports
in an unbiased way
● Interpret the findings and present a balanced and
impartial summary of the findings with due consideration of
any flaws in the evidence.
● Many high quality peer-reviewed systematic reviews are available in journals
as well as from databases and other electronic sources.
● Systematic reviews may examine quantitative or qualitative evidence;
put simply, when the two or more types of evidence are examined within
one review it is called a mixed-method systematic review.
● Systematic reviewing techniques are in a period of rapid development.
Many systematic reviews still look at clinical effectiveness, but methods
now exist to enable reviewers to examine issues of appropriateness,
feasibility and meaningfulness.
● Not all published systematic reviews have been produced with meticulous
care; therefore, the findings may sometimes mislead. Interrogating
For further titles in the series, visit: published reports by asking a series of questions can uncover
www.whatisseries.co.uk deficiencies.
unbiased. The reader will want to interrogate the impact of missing information assessed
any review that purports to be systematic to for its possible impact on the findings?
assess its limitations and to help decide if the ● Do the included studies seem to
recommendations should be applied to indicate similar effects? If not, in the
practice. Further guidance on appraising the case of clinical effectiveness, was the
quality of a systematic review can be found in heterogeneity of effect investigated,
several useful publications.16,30,31 Guidance assessed and discussed?
focuses on the critical appraisal for reviews of ● Were the overall findings assessed for
clinical effectiveness. To reflect this, the their robustness in terms of the
following questions provide a framework. selective inclusion or exclusion of
● Is the topic well defined in terms of the doubtful studies and the possibility of
intervention under scrutiny, the patients publication bias?
receiving the intervention (plus the ● Was the play of chance assessed? In
settings in which it was received) and the particular, was the range of likely effect
outcomes that were assessed? sizes presented and were null findings
● Was the search for papers thorough? interpreted carefully? For example, a
Was the search strategy described? Was review that finds no evidence of effect may
manual searching used as well as electronic simply be an expression of our lack of
databases? Were non-English sources knowledge rather than an assertion that
searched? Was the ‘grey literature’ covered the intervention is worthless.
– for example, non-refereed journals, ● Are the recommendations based
conference proceedings or unpublished firmly on the quality of the evidence
company reports? What conclusions were presented? In their enthusiasm, reviewers
drawn about the possible impact of can sometimes go beyond the evidence in
publication bias? drawing conclusions and making their
● Were the criteria for inclusion of recommendations.
studies clearly described and fairly All studies have flaws. It is not the mere
applied? For example, were blinded or presence of flaws that vitiates the findings.
independent reviewers used? Even flawed studies may carry important
● Was study quality assessed by blinded information. The reader must exercise
or independent reviewers? Were the judgement in assessing whether individual
findings related to study quality? flaws undermine the findings to such an
● Was missing information sought from extent that the conclusions are no longer
the original study investigators? Was adequately supported.
What is a
References 18. Higgins J, Green S (eds). Cochrane Handbook for
1. Pearson A, Wiechula R, Court A, Lockwood C. The JBI
model of evidence-based healthcare. Int J Evid Based
Systematic Reviews of Interventions, Version 5.0.1 [updated
September 2008]. www.cochrane-handbook.org/ (last
systematic review?
Healthc 2005; 3: 207–215. accessed 19 November 2008)
2. Mulrow CD. The medical review article: state of the 19. NHS Centre for Reviews and Dissemination.
science. Ann Intern Med 1987; 106: 485–488. Undertaking systematic reviews of research on effectiveness:
3. Teagarden JR. Meta-analysis: whither narrative review? CRD’s guidance for those carrying out or commissioning
Pharmacotherapy 1989; 9: 274–281. reviews. CRD Report 4. York: University of York, 2001.
4. Spector TD, Thompson SG. The potential and limita- 20. Khan KS, Kunz R, Kleijnen J (eds). Systematic reviews
tions of meta-analysis. J Epidemiol Community Health to support evidence-based medicine: how to review and apply
1991; 45: 89–92. findings of healthcare research. London: Royal Society of
5. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers Medicine Press, 2003.
TC. A comparison of results of meta-analyses of random- 21. Government Social Research. Rapid Evidence
ized control trials and recommendations of clinical Assessment Toolkit. www.gsr.gov.uk/professional_guid-
experts. Treatments for myocardial infarction. JAMA ance/rea_toolkit/index.asp (last accessed 23 January
1992; 268: 240–248. 2009)
6. Lau J, Antman EM, Jimenez-Silva J et al. Cumulative 22. Underwood L, Thomas J, Williams T, Thieba A. The
meta-analysis of therapeutic trials for myocardial infarc- effectiveness of interventions for people with common mental
tion. N Engl J Med 1992; 327: 248–254. health problems on employment outcomes: a systematic
7. Torgerson C. Systematic Reviews. London: Continuum, rapid evidence assessment.
2003. http://eppi.ioe.ac.uk/cms/LinkClick.aspx?fileticket=Wgr
8. The Cochrane Library. The Cochrane Library publishes %2bPGyPMD0%3d&tabid=2315&mid=4279&language=e
the most thorough survey of MMR vaccination data which n-US (last accessed 23 January 2009)
strongly supports its use. 23. Campbell Collaboration Users Group. User
www.cochrane.org/press/MMR_final.pdf (last accessed 19 Involvement in the systematic review process. Campbell
November 2008) Collaboration Policy Brief. http://camp.ostfold.net/art-
9. Wakefield AJ, Murch SH, Anthony A et al. Ileal-lym- man2/uploads/1/Minutes_March_2008_Oslo.pdf (last
phoid-nodular hyperplasia, non-specific colitis, and per- accessed 19 November 2008)
vasive developmental disorder in children. Lancet 1998; 24. Evans D, Pearson A. Systematic reviews: gatekeepers
351: 637–641. of nursing knowledge. J Clin Nurs 2001; 10: 593–599.
10. Murch SH, Anthony A, Casson DH et al. Retraction of 25. Harden A, Thomas J. Methodological Issues in
an interpretation. Lancet 2004; 363: 750. Combining Diverse Study Types in Systematic Reviews.
11. Demicheli V, Jefferson T, Rivetti A, Price D. Vaccines International Journal of Social Research Methodology 2005;
for measles, mumps and rubella in children. Cochrane 8: 257–271.
Database Syst Rev 2005: CD004407. 26. Phillips B, Ball C, Sackett D et al; Centre for Evidence-
12. Dubben HH, Beck-Bornholdt HP. Systematic review Based Medicine. Levels of Evidence.
of publication bias in studies on publication bias. BMJ www.cebm.net/index.aspx?o=1025 (last accessed 19
2005; 331: 433–434. November 2008)
13. Egger M, Zellweger-Zahner T, Schneider M et al. 27. LeLorier J, Gregoire G, Benhaddad A, Lapierre J,
Language bias in randomised controlled trials published Derderian F. Discrepancies between meta-analyses and
in English and German. Lancet 1997; 350: 326–329. subsequent large randomized, controlled trials. N Engl J
14. Sandelowski M, Barroso J. Toward a metasynthesis of Med 1997; 337: 536–542. First edition published 2001
qualitative findings on motherhood in HIV-positive 28. Egger M, Smith GD. Misleading meta-analysis. BMJ Authors: Huw TO Davies and
women. Res Nurs Health 2003; 26: 153–170. 1995; 310: 752–754.
15. Crombie IK, Davies HTO. What is meta-analysis? 29. Evans D. Hierarchy of evidence: a framework for Iain K Crombie
London: Hayward Medical Communications, 2009. ranking evidence evaluating healthcare interventions. J
16. Public Health Resource Unit. Critical Appraisal Skills Clin Nurs 2003; 12: 77–84.
Programme (CASP). Making sense of evidence: 10 questions 30. Crombie IK. Pocket Guide to Critical Appraisal. This publication, along with
to help you make sense of reviews. London: BMJ Publishing Group, 1996.
www.phru.nhs.uk/Doc_Links/S.Reviews%20Appraisal%2 31. Moher D, Cook DJ, Eastwood S et al. Improving the the others in the series, is
0Tool.pdf (last accessed 19 November 2008) quality of reports of meta-analyses of randomised con- available on the internet at
17. Centre for Evidence-Based Medicine. Critical trolled trials: the QUOROM statement. Quality of
Appraisal. www.cebm.net/index.aspx?o=1157 (last Reporting of Meta-analyses. Lancet 1999; 354: www.whatisseries.co.uk
accessed 23 January 2009) 1896–1900.
The data, opinions and statements
appearing in the article(s) herein
are those of the contributor(s)
concerned. Accordingly, the
sponsor and publisher, and their
respective employees, officers
and agents, accept no liability
for the consequences of any such
inaccurate or misleading data,
opinion or statement.