You are on page 1of 17

Research Methods & Reporting

PRISMA harms checklist: improving harms reporting in

systematic reviews
Liliane Zorzela,1 Yoon K Loke,2 John P Ioannidis,3 Su Golder,4 Pasqualina Santaguida,5
Douglas G Altman,6 David Moher,7 Sunita Vohra,1 PRISMA harms group

1Department of Pediatrics, ABSTRACT

Faculty of Medicine and Introduction and potentially biased assessment of an intervention
Dentistry, and Integrative Health For any health intervention, accurate knowledge of when efficacy results are emphasised and harms are
Institute, University of Alberta,
Edmonton, T6G 2C8, AB, both benefits and harms is needed. Systematic inadequately reported.1-10  A misconception may be per-
Canada reviews often compound poor reporting of harms in petuated that a given intervention is safe, when its
2Norwich Medical School, primary studies by failing to report harms or doing so safety (that is, absence of harm) is actually uncertain.4-10
Norwich, UK inadequately. While the PRISMA statement (Preferred Systematic reviews often compound poor reporting of
3Stanford University School of
Reporting Items for Systematic reviews and Meta- harms in primary studies by failing to report harms or
Medicine, Stanford University
School of Humanities and Analyses) helps systematic review authors ensure doing so inadequately.11-20  There are guidelines for
Sciences, Stanford, CA, USA complete and transparent reporting, it is focused reporting systematic reviews1 21 22  but they focus on how
4Department of Health mainly on efficacy. Thus, a PRISMA harms checklist has best to report treatment benefits. Systematic reviews can
Sciences, University of York, been developed to improve harms reporting in be misleading if they do not represent properly the true
Heslington, UK
systematic reviews, promoting a more balanced risk-to-benefit assessment of a given treatment.4 8 14-17
5Department of Clinical
assessment of benefits and harms. Because many adverse events are rare and are not typi-
Epidemiology and Biostatistics,
McMaster University, Hamilton, Methods cally the primary outcome of included studies, the
ON, Canada A development strategy, endorsed by the EQUATOR
6Centre for Statistics in
search strategy, eligibility of study designs, and statisti-
Network and existing reporting guidelines (including cal methods might need to differ from those of system-
Medicine, Nuffield Department
of Orthopaedics, Rheumatology, the PRISMA statement, PRISMA for abstracts, and atic reviews that only address efficacy.
and Musculoskeletal Sciences, PRISMA for protocols), was used. After the Adverse events are the primary outcome assessed in
Botnar Research Centre, development of a draft checklist of items, a modified
University of Oxford, Oxford, UK less than 10% of systematic reviews.11-14  In 1994, only
7Ottawa Methods Centre,
Delphi process was initiated. The Delphi consisted of five reviews retrieved from the Database for Abstracts of
Ottawa Hospital Research three rounds of electronic feedback followed by an Reviews of Effects (DARE) and the Cochrane Database
Institute, Ottawa Hospital, in-person meeting. of Systematic Reviews (CDSR) were specifically
Ottawa, ON, Canada
Results designed to analyse an unintended effect of an inter-
Correspondence to: S Vohra
The PRISMA harms checklist contains four essential vention. While the absolute number has increased over
Cite this as: BMJ 2016;352:i157
reporting elements to be added to the original PRISMA time, the proportion has remained stable. In 2010, 104 statement to improve harms reporting in reviews. reviews retrieved from CDSR and DARE evaluated
These are reported in the title (“Specifically mention adverse events exclusively, but the proportion of
Accepted: 11 December 2015
‘harms’ or other related terms, or the harm of interest reviews of harms in comparison to efficacy reviews
in the review”), synthesis of results (“Specify how zero remained stable at 5% between 1994 and 2010.13
events were handled, if relevant”), study A systematic review12  of systematic reviews published
characteristics (“Define each harm addressed, how it between 2008 and 2011 retrieved from CDSR and DARE
was ascertained (eg, patient report, active search), identified 296 DARE reviews and only 13 Cochrane
and over what time period”), and synthesis of results reviews with a singular primary intent to measure
(“Describe any assessment of possible causality”). adverse events of interventions. Even though systematic
Additional guidance regarding existing PRISMA items
reviews increasingly try to consider all outcomes (both
was developed to demonstrate relevance when
beneficial and harmful), data on adverse events may be
synthesising information about harms.
more fragmented and incomplete, and given more cur-
Conclusion sory treatment than efficacy data. Even when reviews
The PRISMA harms checklist identifies a minimal set of are exclusively designed to measure adverse events,
items to be reported when reviewing adverse events. identified reporting deficiencies have included: lack of a
This guideline extension is intended to improve harms clear definition of the adverse event reviewed, lack of
reporting in systematic reviews, whether harms are a specification regarding study designs selected for inclu-
primary or secondary outcome. sion, and no report on length of participants’ follow-up
or measurement of any associated patient risk factors
Introduction that could lead to the adverse event.12  We set out to
Evidence based healthcare practice seeks the best, extend the PRISMA guideline (Preferred Reporting Items
unbiased evidence to make appropriate decisions to for Systematic reviews and M ­ eta-Analyses)1 to facilitate
improve patient outcomes. For optimal healthcare deci- balanced reporting of benefits and harms.
sion making, accurate knowledge of both benefits and
harms is needed. Development of PRISMA harms
Even well designed, conducted, and reported ran- We followed the strategy developed by the EQUATOR
domised controlled trials can provide an incomplete Network22  and used for previous reporting guidelines

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 1

Research Methods & Reporting

including PRISMA statement,1  PRISMA for abstracts,23  of a health regulatory agency (Health Canada). Meeting
and PRISMA for protocols.24  To document the need for participants had the results of the review identifying
a reporting guideline regarding adverse event reporting the current state of reporting in systematic reviews of
in systematic reviews, our team evaluated the reporting harms12 and the results of the Delphi process to inform
characteristics of reviews with adverse events as a pri- their discussion of relevant items to be included in this
mary outcome and found several areas (title, abstract, guideline extension.
methods, results, and conclusion) that could benefit
from more transparent reporting.12 Scope of PRISMA harms
A list of 37 potential new items for PRISMA harms A goal of the in-person consensus meeting was to define
was developed on the basis of preliminary findings of the PRISMA harms guideline’s applicability. After dis-
previous systematic reviews.12 25 These potential items cussion, it was agreed that the modifications to the
were then compared against the original PRISMA state- existing PRISMA statement should aim to improve the
ment to assess overlap and refine wording. The wording reporting of adverse events (defined in table 1 27-29) in
and content were further refined by the PRISMA harms systematic reviews, whether adverse events are a pri-
steering committee. mary or secondary outcome.
After the development of a draft list of potential Systematic reviews assessing harms may include
items, a modified Delphi26 process was initiated to observational studies exclusively or in addition to inter-
obtain feedback from a broad spectrum of stakeholders. ventional studies.30 PRISMA harms items and recom-
We surveyed 324 people through an online survey con- mendations for reporting harms in systematic reviews
sisting of three rounds of participant feedback. Delphi are applicable to observational studies (with and with-
participants were selected on the basis of their exper- out a comparison group) and to prospective interven-
tise in systematic reviews in general, and in particular tional studies if deemed to be included in the review.
reviews of adverse events. This selection was comple- This report is an extension of the PRISMA statement1
mented by other content experts including methodolo- and, as such, should be used in addition to the original
gists, statisticians, epidemiologists, clinicians, journal statement. The main goal of the four PRISMA harms
editors, a consumer, and a member of a federal health items is to bring attention to a minimal set of items to be
regulatory agency (Health Canada). The list of potential reported in any review assessing harms. The recom-
items was sent to the participants in four weeks inter- mendation for reporting harms in systematic reviews
vals. A total of 112 participants contributed to at least provides elaboration and examples on how existing
one of the three Delphi rounds; 56 participants com- PRISMA items should be applied to improve reporting
pleted more than one round. Delphi results led to one of harms in systematic reviews.
potential item excluded, eight items received scattered
votes, and all remaining 28 items were voted relevant by How to use the PRISMA harms checklist
the Delphi participants. The PRISMA harms checklist (table 2) contains at least
An in-person two day consensus meeting was held in four extension items that must be used in any system-
Banff, Canada, in May 2012. It included 25 experts from atic review addressing harms, irrespective of whether
seven countries with extensive experience in systematic harms are analysed alone or in association with bene-
reviews, adverse events research, and guideline devel- fits. These include:
opment, and also included a consumer and a member
• Item 1—title: specifically mention “harms” or other
related terms, or the harm of interest in the system-
atic review.
Table 1 | Glossary of terms • Item 14—synthesis of results: specify how zero events
Term Definition were handled, if relevant.
Adverse drug reaction* An adverse effect specific to a drug • Item 18—study characteristics: define each harm
Adverse effect*† An unfavourable outcome that occurs during or after the use of a drug or addressed, how it was ascertained (eg, patient report,
other intervention but is not necessarily caused by it
active search), and over what time period.
Adverse event*† An unfavourable outcome that occurs during or after the use of a drug or
other intervention and the causal relation between the intervention and the • Item 21—synthesis of results: describe any assess-
event is at least a reasonable possibility ment of possible causality.
Complication* An adverse event or effect following surgical and other invasive intervention
Harm‡ The totality of possible adverse consequences (if single or multiple) of an
These items are added to the original PRISMA state-
intervention or therapy; harms are the direct opposite of benefits ment, such that a systematic review addressing adverse
Safety‡ Substantive evidence of an absence of harm. The term is often misused events should report the PRISMA statement items and
when there is simply absence of evidence of harm the PRISMA harms.
Side effect* Any unintended effect, adverse or beneficial, of a drug that occurs at doses
normally used for treatment
In this article, we provide explanation and guidance
Toxicity‡ Drug related harm. The term may be most appropriate for laboratory for the four PRISMA harms items, a minimum set of
determined measurements, although it is also used in relation to clinical items to be reported in any review assessing harms. We
events. The disadvantage of the term “toxicity” is that it implies causality. also discuss recommendations for reporting harms in
If authors cannot prove causality, the terms “abnormal laboratory
measurements” or “laboratory abnormalities” are more appropriate systematic reviews for those items in the original
*Adapted from reference 27. PRISMA checklist that require special consideration
†Adapted from reference 28. when reporting on harms. The recommendations are
‡Adapted from reference 29.
considered a desirable set, and relevant to harms

2 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Table 2 | PRISMA harms checklist
Section/topic PRISMA harms Check
(page no) Item PRISMA checklist item (minimum) Recommendations for reporting harms in systematic reviews (desirable) if done
Title (3) 1 Identify the report as a systematic review, meta-analysis, or both. Specifically mention —
“harms” or other
related terms, or the
harm of interest in
the review.
Structured 2 Provide a structured summary including, as applicable: background; objectives; — Abstracts should report any analysis of harms undertaken in the review, if harms
summary (4) data sources; study eligibility criteria, participants, and interventions; study are a primary or secondary outcome.
appraisal and synthesis methods; results; limitations; conclusions and

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157

implications of key findings; systematic review registration number.
Rationale (5) 3 Describe the rationale for the review in the context of what is already known. — It should clearly describe in introduction or in methods section which events are
considered harms and provide a clear rationale for the specific harm(s),
condition(s), and patient group(s) included in the review.
Objectives (5) 4 Provide an explicit statement of questions being addressed with reference to — PICOS format should be specified, although in systematic reviews of harms the
participants, interventions, comparisons, outcomes, and study design (PICOS). selection criteria for P, C, and O may be very broad (same intervention may have
been used for heterogeneous indications in a diverse range of patients)
Protocol and 5 Indicate if a review protocol exists, if and where it can be accessed (eg, web — No specific additional information is required for systematic reviews of harms.
registration (6) address), and, if available, provide registration information including registration
Eligibility 6 Specify study characteristics (eg, PICOS, length of follow-up) and report — Report how handled relevant studies (based on population and intervention)
criteria (6) characteristics (eg, years considered, language, publication status) used as when the outcomes of interest were not reported.Report choices for specific study
criteria for eligibility, giving rationale. designs and length of follow-up.
Information 7 Describe all information sources (eg, databases with dates of coverage, contact — Report if only searched for published data, or also sought data from unpublished
sources (7) with study authors to identify additional studies) in the search and date last sources, from authors, drug manufacturers and regulatory agencies. If includes
searched. unpublished data, provide the source and the process of obtaining it.
Search (7) 8 Present full electronic search strategy for at least one database, including any — If additional searches were used specifically to identify adverse events, authors
limits used, such that it could be repeated. should present the full search process so it can be replicated.
Study selection (8) 9 State the process for selecting studies (ie, screening, eligibility, included in — If only included studies reporting on adverse events of interest, defined if
systematic review, and, if applicable, included in the meta-analysis). screening was based on adverse event reporting in title/abstract or full text. If no
harms reported in the text, report if any attempt was made to retrieve relevant
data from authors.
Data collection 10 Describe method of data extraction from reports (eg, piloted forms, — No specific additional information is required for systematic reviews of harms.
process (9) independently, in duplicate) and any processes for obtaining and confirming data
from investigators.
Data items (9) 11 List and define all variables for which data were sought (eg, PICOS, funding — Report the definition of the harm and seriousness used by each included study
sources) and any assumptions and simplifications made. (if applicable). Report if multiple events occurred in the same individuals, if this
information is available. Consider if the harm may be related to factors associated
with participants (eg, age, sex, use of medications) or provider (eg, years of
practice, level of training). Specify if information was extracted and how it was
used in subsequent results. Specify if extracted details regarding the specific
methods used to capture harms (active/passive and timing of adverse event).
Risk of bias in 12 Describe methods used for assessing risk of bias of individual studies (including — The risk of bias assessment should be considered separately for outcomes of
individual specification of whether this was done at the study or outcome level), and how benefit and harms.
studies (10) this information is to be used in any data synthesis.
Summary 13 State the principal summary measures (eg, risk ratio, difference in means). — No specific additional information is required for systematic reviews of harms.
measures (11)
Research Methods & Reporting

Table 2 | PRISMA harms checklist
Section/topic PRISMA harms Check
(page no) Item PRISMA checklist item (minimum) Recommendations for reporting harms in systematic reviews (desirable) if done
Synthesis of 14 Describe the methods of handling data and combining results of studies, if done, Specify how zero
results (11) including measures of consistency (eg, I2) for each meta-analysis. events were
handled, if relevant.
Risk of bias across 15 Specify any assessment of risk of bias that may affect the cumulative evidence — Present the extent of missing information (studies without harms outcomes), any
studies (11) (eg, publication bias, selective reporting within studies). factors that may account for their absence, and whether these reasons may be
related to the results.
Additional 16 Describe methods of additional analyses (eg, sensitivity or subgroup analyses, — Sensitivity analyses may be affected by different definitions, grading, and
analyses (12) meta-regression), if done, indicating which were prespecified. attribution of adverse events, as adverse events are typically infrequent or
reported using heterogeneous classifications. Report the number of participants
and studies included in each subgroup.
Study selection (13) 17 Give numbers of studies screened, assessed for eligibility, and included in the — If a review addresses both efficacy and harms, display a flow diagram specific for
review, with reasons for exclusions at each stage, ideally with a flow diagram. each (efficacy and harm).
Study 18 For each study, present characteristics for which data were extracted (eg, study Define each harm Add additional characteristics to: “P” (population) patient risk factors that were
characteristics (14) size, PICOS, follow-up period) and provide the citations. addressed, how it considered as possibly affecting the risk of the harm outcome. “I” (intervention)
Research Methods & Reporting

was ascertained professional expertise/skills if relevant (for example if the intervention is a

(eg, patient report, procedure). “T” (time) timing of all harms assessments and the length of
active search), and follow-up.
over what time
Risk of bias within 19 Present data on risk of bias of each study and, if available, any outcome level — Consider the possible sources of biases that could affect the specific harm under
studies (15) assessment (see item 12). consideration within the review. Sample selection, dropouts and measurement of
adverse events should be evaluated separately from the outcomes of benefit as
described in item 12, above.
Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: (a) — Report the actual numbers of adverse events in each study, separately for each
studies (16) simple summary data for each intervention group (b) effect estimates and intervention.
confidence intervals, ideally with a forest plot.
Synthesis of 21 Present results of each meta-analysis done, including confidence intervals and Describe any If included data from unpublished sources, report clearly the data source and the
results (17) measures of consistency. assessment of impact of these studies to the final systematic review.
possible causality.
Risk of bias across 22 Present results of any assessment of risk of bias across studies (see item 15). — No specific additional information is required for systematic reviews of harms.
studies (18) See item 15 above.
Additional 23 Give results of additional analyses, if done (eg, sensitivity or subgroup analyses, — No specific additional information is required for systematic reviews of harms.
analysis (18) meta-regression (see item 16)).
Summary of 24 Summarise the main findings including the strength of evidence for each main — No specific additional information is required for systematic reviews of harms.
evidence (18) outcome; consider their relevance to key groups (eg, healthcare providers,
users, and policy makers).
Limitations (18) 25 Discuss limitations at study and outcome level (eg, risk of bias), and at review — Recognise possible limitations of meta-analysis for rare adverse events (ie,
level (eg, incomplete retrieval of identified research, reporting bias). quality and quantity of data), issues noted previously related to collection and
Conclusions (19) 26 Provide a general interpretation of the results in the context of other evidence, — State conclusions in coherence with the review findings. When adverse events
and implications for future research. were not identified we caution against the conclusion that the intervention is
“safe,” when, in reality, its safety remains unknown.
Funding (19) 27 Describe sources of funding for the systematic review and other support (eg, — No specific additional information is required for systematic reviews of harms.
supply of data); role of funders for the systematic review.

doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

­ ublications in general. We include relevant examples

p bisphosphonates compared with non-exposure.
of good reporting for all PRISMA harms items and rec- Data on atrial fibrillation as the primary out-
ommendations for reporting harms in systematic come, and stroke and cardiovascular mortality
reviews. as secondary outcomes, were extracted. Data
Synthesis/Results: We calculated pooled odds
Item 1—title ratio (OR) using random effects meta-analysis,
PRISMA statement: “Identify the report as a and estimated statistical heterogeneity with
systematic review, meta-analysis, or both.” the I2 statistic. Bisphosphonate exposure was
significantly associated with risk of atrial fibril-
PRISMA harms extension: “Specifically mention
lation serious adverse events in a meta-analy-
‘harms’ or other related terms, or the adverse
sis of four trial datasets (OR 1.47; 95% CI 1.01,
event(s) of interest in the review.”
2.14; p=0.04; I2 =46%). However, meta-analy-
sis of all atrial fibrillation events (serious and
Example non-serious) from the same datasets yielded
“Perinatal mortality and other severe adverse a pooled OR of 1.14 (95% CI 0.96, 1.36; p=0.15;
pregnancy outcomes associated with treat- I2 =0%). We identified two case-control studies,
ment of cervical intraepithelial neoplasia: one of which found an association between
meta-­analysis.” 31 bisphosphonate exposure (ever users) and
atrial fibrillation (adjusted OR 1.86; 95% CI
PRISMA harms item explanation 1.09, 3.15) while the other showed no associa-
The title should clearly reflect the objectives of the sys- tion (adjusted OR 0.99; 95% CI 0.90, 1.10). Both
tematic review, be accessible to all readers, and provide studies failed to demonstrate a significant asso-
a one line summary of the authors’ intention. If adverse ciation in ‘current’ users. We did not find a sig-
events are part of the review’s primary objective, as a nificant increase in the risk of stroke (three trial
primary or secondary outcome, the title should state datasets; OR 1.00; 95% CI 0.82, 1.22; p=0.99;
this clearly. It can name the specific adverse event I2 =0%) or cardiovascular mortality (three trial
under review or any generic harms related terms (for datasets; OR 0.86; 95% CI 0.66, 1.13; p=0.28;
example, risk, complication, adverse effects, or adverse I2 =31%). Conclusion: While there are some data
reaction). If the harm is a coprimary outcome (for exam- linking bisphosphonates to serious atrial fibril-
ple, the measurement of efficacy and harms), the title lation, heterogeneity of the existing evidence,
should indicate this. as well as paucity of information on some of the
agents, precludes any definitive conclusions on
Item 2—abstract the exact nature of the risk.”32
PRISMA statement: “Provide a structured sum-
mary including, as applicable: background; Recommendations for reporting harms in systematic
objectives; data sources; study eligibility cri- reviews
teria, participants, and interventions; study As abstracts reach a broad audience, the abstract of a
appraisal and synthesis methods; results; systematic review should be a clear summary of the
limitations; conclusions and implications of review.33  Abstracts should report any analysis of harms
key findings; systematic review registration undertaken in the review, if harms are a primary or sec-
number.” ondary outcome, as recommended by PRISMA for
“Background: Bisphosphonates are widely used Item 3—introduction
in osteoporosis, but there have been concerns PRISMA statement: “Describe the rationale
about a potential link between bisphospho- for the review in the context of what is already
nate therapy and atrial fibrillation. Objective: known.”
We aimed to systematically evaluate the risk
of atrial fibrillation associated with bisphos- Example
phonate use. Methods: We searched MEDLINE, “Epilepsy has a prevalence of 5-10 persons/1000.
regulatory authority websites, pharmaceutical During pregnancy, women with epilepsy can-
company trial registers and product informa- not generally safely discontinue their antiepi-
tion sheets for randomized controlled trials leptic therapy, and the risks to the unborn child
(RCTs) and controlled observational studies from maternal antiepileptic medication need
published in English through to May 2008. We to be balanced against the risk of uncontrolled
selected RCTs of bisphosphonates versus pla- epilepsy both to the mother and the baby. In the
cebo for osteoporosis or fractures, with at least 3 last decade, an increasing number of studies
months of follow-up, and data on atrial fibrilla- have addressed the long-term safety of these
tion. For the observational studies, we included drugs on child development, with conflicting
case control or cohort studies that evaluated the results. Synthesizing these data into an overall
risk of atrial fibrillation in patients exposed to risk assessment is critical for clinical counselling

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 5

Research Methods & Reporting

of women with epilepsy and their families. No specific additional information is required for sys-
The objective of this study was to perform a tematic reviews of harms.
systematic review of the literature pertaining
to long-term neurodevelopment after in utero Item 6—eligibility criteria
exposure to antiepileptic drugs (AEDs) and to PRISMA statement: “Specify study characteris-
conduct a meta-analysis to allow overall risk tics (eg, PICOS, length of follow-up) and report
estimation.”34 characteristics (eg, years considered, language,
publication status) used as criteria for eligibil-
Recommendations for reporting harms in systematic ity, giving rationale.”
The introduction should inform the reader of the Example
review’s overall goal and provide the rationale for the “We included studies with data on severe
approach taken.1  Systematic reviews of harms can be obstetric or neonatal outcomes in women
designed with a narrow focus, evaluating a specific type treated for cervical intraepithelial neoplasia
of adverse event, or with a broad focus to evaluate all and in a control group of untreated women. Two
adverse events associated with a given intervention.28 types of treatment were considered: excisional
The systematic review should clearly describe in the procedures (cold knife conisation, large loop
introduction or methods section which events are con- excision of the transformation zone, and laser
sidered harms and provide a clear rationale for the spe- conisation) and ablative procedures (laser abla-
cific harm(s), condition(s), and patient group(s) tion, cryotherapy, and diathermy).
included in the review.
Outcome measures: The severe adverse obstet-
Item 4—objective ric or neonatal events were perinatal mortality,
PRISMA statement: “Provide an explicit severe (at less than 32/34 weeks’ gestation) and
statement of questions being addressed with extreme (<28/30 weeks) preterm delivery, and
reference to participants, interventions, com- severe low birth weight (<2000 g, <1500 g, and
parisons, outcomes, and study design (PICOS).” <1000 g).
. . . There was no language restriction. Three
authors . . . verified inclusion and exclusion cri-
“Our objective was to systematically determine
teria independently and reached consensus in
the comparative effects of rosiglitazone and
case of discordance.
pioglitazone on cardiovascular outcomes (myo-
cardial infarction and congestive heart failure) . . . We contacted authors to obtain data on out-
and mortality from observational studies in comes by particular treatment procedure if they
patients with type 2 diabetes.”35 were not provided in the original reports. In
addition, we collected data on the study design
Recommendations for reporting harms in systematic and matching criteria applied for the selection
reviews of a control group of non-treated women.”31
The objective of a systematic review should be clearly
stated, preferably at the end of the introduction.1  The
PRISMA statement1 suggests the PICOS format (popula- Recommendations for reporting harms in systematic
tion, intervention, comparison, outcomes, and study reviews
design). Overall, the PICOS format should be specified, Population and patient characteristics are important
although in systematic reviews of harms the selection when considering harms and should be clearly
criteria for population, comparison, and outcomes can reported. The review authors should report how they
be broad. For example, the same intervention might have handled relevant studies when the outcomes of interest
been used for heterogeneous indications in a diverse were not reported (that is, the primary study did not
range of patients, such that the systematic review allows mention adverse events, but it was a relevant study
a broad range of comparisons to be included. Similarly, if based on the population, intervention and comparator).
a review is attempting to evaluate any or all possible Explicit systematic review methods will provide readers
harms (including new or unexpected events) associated with important information on whether the review
with a given intervention, the ­potential outcomes (that might be affected by missing outcome data or missing
is, adverse events) cannot be completely defined a priori studies.
(in the protocol phase) in detail. Also relevant is transparent reporting regarding the
review authors’ choices for specific study designs (for
Item 5—protocol and registration example, limiting the review to randomised controlled
PRISMA statement: “Indicate if a review pro- trials v including other study designs), if specific study
tocol exists, if and where it can be accessed designs were chosen specifically to address harms.
(eg, web address), and, if available, provide Depending on the characteristics of the adverse event
registration information including registration under investigation, different study designs have
number.” ­different strengths and weaknesses, and accordingly,

6 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

appropriate study designs should be reported for the and regulatory agencies. Published data can differ sub-
particular outcome of interest.16 stantially from unpublished data for various reasons,
Whatever methods are chosen by review authors to especially in relation to harms.37-41 If a systematic
determine which studies are included should be review includes unpublished data, a clear description
explicit, allowing readers to better understand how should be provided of the source and the process of
adverse events were identified and extracted from the obtaining it.
included studies.
Item 8—search
Item 7—information sources PRISMA statement: “Present full electronic
PRISMA statement: “Describe all information search strategy for at least one database,
sources (eg, databases with dates of coverage, including any limits used, such that it could be
contact with study authors to identify additional repeated.”
studies) in the search and date last searched.”
Example “We searched PubMed, Embase, and CENTRAL
“Systematic literature searches were conducted using the terms: ‘angiotensin receptor block-
in the following electronic databases, all from ers’, ‘angiotensin receptor antagonists’, ‘ARBs’,
their respective inception until February 2008 and the names of individual angiotensin recep-
and without any language restrictions: PubMed tor blockers in humans until August 2010.
via Medline, AMED, EMBASE, CINAHL and the Appendix 1 on gives details of the
Cochrane Library - the Cochrane Central Regis- search and the MeSH terminologies used. We
ter of Controlled Trials (CENTRAL). The search checked the reference lists of review articles,
terms were the common names(s), scientific meta-analyses, and original studies identified
names(s) and synonyms for S. repens . . . No lim- by the electronic searches to find other eligible
its were placed on the search function. Further trials. There was no language restriction for the
relevant data were retrieved by hand search- search. Authors of trials were contacted when
ing the reference lists of identified papers and results were unclear or when relevant data
searching our files at the Complementary Med- were not reported. In addition, we searched
icine department, Peninsula Medical School, Food and Drug Administration (FDA) dockets
Universities of Exeter and Plymouth, Exeter, UK. by hand searching all documents submitted for
drug approval/labelling change as well as the
Additional data were requested from the fol-
minutes from FDA meetings available on the
lowing reporting schemes: Adverse Drug Reac-
FDA website.”42
tions Advisory Committee (ADRAC), Australia;
Bundesinstitut fur Arzneimittel und Mediz-
Recommendations for reporting harms in systematic
inprodukte (BfArM), Germany; US Food and
Drug Administration (FDA); and the Medicine
If additional searches were used specifically to identify
and Healthcare products Regulatory Agency
adverse events, authors should present the full search
(MHRA), UK. The WHO Collaborating Centre
process so it can be replicated. Reviews of harms might
for International Drug Monitoring, Uppsala,
have different search methods and study selection crite-
Sweden (WHO-UMC) was also requested to pro-
ria from reviews of efficacy. Reviews of efficacy have
vide the total numbers of adverse events reports
their search strategy and screening based on the effi-
received up until September 2007 involving the
cacy question (does the intervention provide the
use of S. repens. Twenty-four manufacturers/
intended effect, for example, cure the disease?) and
distributors of S. repens preparations were
often cannot deal with harms adequately (for example,
identified from a review, standard text and from
what are the unintended effects (harms) associated
Internet searches. They were contacted and
with the intervention?). Reviews of harms might need a
asked for adverse event reports and any other
distinct database filter and data source searches.16 30 43-46
safety information held on file. Four herbalist
organizations (British Herbal Medicine Associ-
Item 9—study selection
ation, UK; European Herbal & Traditional Med-
PRISMA statement: “State the process for
icine Practitioners Association, UK; European
selecting studies (ie, screening, eligibility,
Scientific Cooperative on Phytotherapy, UK;
included in systematic review, and, if applica-
National Institute of Medical Herbalists, UK)
ble, included in the meta-analysis).”
were also contacted for relevant information.”36
Recommendations for reporting harms in systematic “Inspection of citations: After duplicate cita-
reviews tions were removed, all titles and abstracts
Review authors should be explicit if they only searched were independently reviewed by two reviewers
for published data, or also sought data from . . . with reference to the inclusion/exclusion cri-
­unpublished sources, authors, drug manufacturers, teria (Supplementary Table S3), and a decision

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 7

Research Methods & Reporting

was made whether to retrieve the full report of especially in the reporting of carcinoma in situ
the study. The number of titles/abstracts identi- and non melanoma skin cancers. As manu-
fied, accepted, and rejected was recorded. scripts may aggregate malignancies differently,
malignancies were allocated to three classes
Inspection of retrieved reports: Once the full
allowing for comparisons of similar outcomes:
reports were retrieved, they were inspected for
lymphomas; non-melanoma skin cancers and
relevance to the review and the inclusion and
the composite endpoint of non-cutaneous can-
exclusion criteria applied. Studies not meet-
cers and melanomas. If a subject presented
ing the predetermined criteria were excluded.
with two types of cancer, the cancers were allo-
If there was any disagreement about whether
cated as a single event in the following order
to include any of the studies, a third reviewer
of priority: lymphoma, non-cutaneous cancer/
(  .  .  . assessed them and, together with the
melanoma, non-melanoma skin cancer. When
other  reviewers, made a consensus decision
the number of events instead of the number of
about whether to include or exclude. A record
subjects experiencing an event was reported,
was made of the number of papers retrieved
an assumption of one event per subject was
and the number of papers excluded.”47
made. All data were abstracted as reported in
the publications. If an event described in a pub-
Recommendations for reporting harms in systematic lication could not be allocated to a particular
reviews time or treatment group other sources of infor-
Review authors should specify whether only studies mation were used. All data were compiled by
reporting on adverse events were included in the two authors (TRE and JPL) and disagreements
review, as noted previously in item 6. If the systematic were resolved by consensus.”48
review only includes studies reporting on adverse
events of interest, it should be clearly defined whether Recommendations for reporting harms in systematic
screening was based on adverse event reporting in the reviews
title or abstract or in the full text. Harms are especially Categorisation—Harms may be categorised in a hetero-
poorly reported in titles and abstracts, leading to the geneous fashion by primary study authors. For exam-
potential exclusion of relevant studies.27 Moreover, ple, when investigating haemorrhagic stroke, some
studies might not report harms in the full text, even primary studies might report a combination of events
though such data were collected, thus additional rele- under “neurological events,” others might report them
vant data may only be obtained on request. under “cardiovascular events,” and few might report
them as “stroke” but not subdivide further (that is, hae-
Item 10—data collection process morrhagic or ischaemic). Also, many harms can have
PRISMA statement: “Describe method of data different severities, and the definitions of seriousness
extraction from reports (eg, piloted forms, used can vary between studies. These issues should
independently, in duplicate) and any pro- preferably be considered at the protocol phase. All
cesses for obtaining and confirming data from operational definitions used to classify adverse events
investigators.” under review should be explicitly identified by review
No specific additional information is required for sys- Events/participants—Studies usually report on the
tematic reviews of harms. number of events but these might not accurately reflect
the number of participants experiencing the event. For
Item 11—data items example, the same patient might have angina, followed
PRISMA statement: “List and define all vari- by myocardial infarction, and finally cardiovascular
ables for which data were sought (eg, PICOS, related death. Studies might report these three events
funding sources) and any assumptions and sim- as isolated findings (angina, myocardial infarction,
plifications made.” death), but they all occurred in one participant during
the study. Participants might also experience the same
event multiple times. Review authors should report if
Example multiple events occurred in the same individuals, if this
“Anti-TNF groups were divided into drug and information is available.
dose categories. Dose was defined according Factors associated with the event—When reporting
to the recommended maintenance dose from on adverse events, consider whether the incidence of
the product labelling. Only recommended and adverse events is related to factors associated with par-
high doses were considered in the analysis. The ticipants (such as age, sex, use of drug treatments) or
number of subjects experiencing death or at provider (such as years of practice, level of training).
least one serious adverse event or serious infec- Review authors should specify whether such informa-
tion was extracted for each treatment group. tion was extracted and how it was used in subsequent
Extracting malignancy data from published results.
clinical trial manuscripts requires caution as Measurement—Different methods used to measure
there is considerable variation in reporting, harms could lead to different results. Active methods

8 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

(actively seeking information about harms) are associ- Item 14—synthesis of results
ated with more reported events than passive methods PRISMA statement: “Describe the methods of
(waiting for patients to report them).29  The timing and handling data and combining results of studies,
frequency of adverse events measurement is also if done, including measures of consistency (eg,
important. For example, measurement might be done I2) for each meta-analysis.”
only at the end of the study intervention (when partici-
PRISMA harms extension: “Specify how zero
pants might not accurately recall how they felt during
events were handled, if relevant.”
the entire course of the study) or done at regular inter-
vals throughout the treatment period.4 9 Review authors
should specify if they extracted details regarding the Example
specific methods used to capture harms. “In cases when only one study reported an
Reporting—Poor/unclear reporting in primary stud- adverse event, relative risks (RR) were calcu-
ies should be anticipated and the approach used to lated with the corresponding 95% confidence
overcome them included in the systematic review intervals (95% CI) using the data extracted.
­protocol. When in one cell there were zero cases, 0.5 was
added to all four cells of a 2×2 table.”49
Item 12—risk of bias in individual studies
PRISMA statement: “Describe methods used PRISMA harms item explanation
for assessing risk of bias of individual stud- Because harms are often rare events, it is common to
ies (including specification of whether this find studies reporting no instances of the specific harm.
was done at the study or outcome level), and This situation would require systematic reviewers to
how this information is to be used in any data consider relevant statistical issues, ideally a priori and
synthesis.” documented in the systematic review protocol. The
review authors should clearly report the steps taken to
Example overcome problems associated with studies including
“Quality assessment: For RCTs that compared zero events in one or more groups, in meta-analysis.50-53
dexamethasone with another intervention Harms are typically not the primary outcome of studies
and reported haemorrhage rate or for which evaluating efficacy. Review authors should plan and
haemorrhage rate data were obtained from specify how they will deal with studies not reporting on
the author(s), the methodological quality was harms of interest, studies reporting a general statement
assessed using the Cochrane Collaboration’s indicating the absence of the event (for example, “no
domain-based evaluation tool for assessing serious harms were identified in any group,” but with no
risk of bias. The methodological quality of definitions of seriousness), or studies not reporting on
haemorrhage rate recording and reporting was any adverse events. The review authors should clarify if
assessed for both randomized and NRS using there was an absence of events or an absence of report-
selected elements of the McMaster Quality ing. They should specify if the situation of “no events
Assessment Scale of Harms for primary stud- reported” was treated as “zero events” or handled in
ies (the McHarm Scale), http://hiru.mcmaster. some other manner. Additionally, the review authors
ca/epc/mcharm.pdf. The elements used were should report whether any effort was made to clarify
selected based on an evaluation of their rele- ambiguity with the authors of the primary studies.
vance to our research question and they aimed
to evaluate: the quality and appropriateness of Item 15—risk of bias across studies
study design and reporting, the applicability of PRISMA statement: “Specify any assessment of
the study findings to the population, and mea- risk of bias that may affect the cumulative evi-
sures taken to reduce bias.”47 dence (eg, publication bias, selective reporting
within studies).”
Recommendations for reporting harms in systematic
reviews Example
Studies that are well designed to assess efficacy of an “Munshi et  al. did not report the non-statis-
intervention may not necessarily preserve the same tically significant adjusted odds ratio for the
qualities when assessing harms.26 29 The risk of bias risk of hypoglycaemia in those with cognitive
assessment should be considered separately for out- impairment. The incomplete reporting of null
comes of benefit and harms. results means that our meta-analyses could
potentially have overestimated the strength of
Item 13—summary measures the association between hypoglycaemia and
PRISMA statement: “State the principal sum- cognitive impairment.”54
mary measures (eg, risk ratio, difference in
means).” Recommendations for reporting harms in systematic
No specific additional information is required for sys- There is mounting evidence that the risk of reporting
tematic reviews of harms. bias in relation to harms is more frequent than bias in

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 9

Research Methods & Reporting

relation to efficacy outcomes.55-57  An analysis of system- Item 16—additional analyses

atic reviews from the Cochrane Collaboration found PRISMA statement: “Describe methods of addi-
that the single primary harm outcome was inadequately tional analyses (eg, sensitivity or subgroup
reported in 76% of the studies included and not analyses, meta-regression), if done, indicating
reported in 47% of reviews outside Cochrane that have which were prespecified.”
been designed specifically to measure harms.56  A simi-
lar study of reviews of efficacy found that 31% of trials Example
did not report on the primary benefit outcome.57  There “Additional predefined sensitivity analyses
is also evidence that those reporting on primary studies were done to explore the influence on effect size
might choose to play down the estimates of harms and of different doses of tiotropium (10 µg v 5 µg),
emphasise the efficacy of the intervention instead.56-58  the effect of trial duration, and the influence of
Selective outcome reporting and publication bias individual trials. We estimated the annualized
(where entire studies are unpublished due to the unex- number needed to treat for mortality associated
pected findings of harms) could therefore work in differ- with tiotropium mist inhaler by applying the
ent directions from that observed with efficacy trials pooled relative risk from the meta-analysis to
where benefits are emphasised and harms are neglected the average control event rate in the long term
or distorted.4 56-63 trials . . . The number needed to treat for mor-
When statistical approaches are used to probe for the tality is the number of patients with chronic
possibility of biased reporting, they should be explicitly obstructive pulmonary disease treated with
described and used with caution. It is a common mis- tiotropium mist inhaler for one year, rather
conception that tests (such as funnel plots) can confirm than placebo, associated with one additional
that publication bias does not exist.64 65  Assessments of death.”67
risk of bias across studies should focus more on pre-
senting the extent of missing information (studies Recommendations for reporting harms in systematic
­without harms outcomes), any factors that might reviews
account for their absence, and whether these reasons Sensitivity analyses might be affected by different
may be related to the results of the harms outcomes.66 definitions, grading, and attribution of adverse
events, because adverse events are typically infre-
quent or reported using heterogeneous classifications
Records identified through database Additional screening (screening reference lists, (see item 11). If factors associated with harms are
searching (n=5062): Conference Proceedings Citation Index - Science, investigated, review authors should report the num-
Medline (n=1690) Central (n=504) OpenSIGLE, trial registries, Google)
ber of participants and studies included in each sub-
Embase (n=2477) Toxline (n=391)
Records screened (n=5062) Studies identified for inclusion (n=5)
Item 17—study selection
Did not meet selection criteria PRISMA statement: “Give numbers of studies
or duplicate record (n=4028)
screened, assessed for eligibility, and included
Full text articles assessed for eligibility in the review, with reasons for exclusions at
(n=1034): each stage, ideally with a flow diagram.”
Medline (n=483) Central (n=66)
Embase (n=441) Toxline (n=44)
Full text articles excluded (n=896): “The initial database searching identified
Review article (systematic or narrative) 5,062 articles. Review of the abstracts/titles
No adverse maternal effects reported and exclusion of irrelevant/duplicate articles
(n=182) yielded 1,034 articles. Of these articles, we
Magnesium compared to another drug
or given in combination (n=44) excluded 896 for the documented reasons. We
Not a relevant patient group or therefore included 138 studies, along with an
indication (n=19)
No abstract and/or no full text available
additional five studies identified through other
(including where published in another searching; a total of 143 studies (see Additional
language and no translation available) file 3 for References to all included reports). In
Other (trial protocol, editorial, guideline, the case of multiple publications from the same
commentary, opinion piece, letter to study, we included the report with the most rel-
editor (no adverse effects)) (n=142)
Another publication from an included evant data relating to adverse effects.”69
study (n=32)
Recommendations for reporting harms in systematic
Studies for inclusion in review (n=143):
Randomised controlled trials (II*) (n=21) Case series (IV*) (n=32) If a review addresses both efficacy and harms, it can be
Non-randomised controlled trials (II-2*) (n=15) Reports of individual cases (IV*) (n=15) useful to display a flow diagram specific for each, so the
reader can have a clear idea of how many studies were
Fig 1 | Flow diagram, relating to example in item 17 (study selection) of the PRISMA harms included and excluded for the efficacy outcome and the
checklist. Figure adapted from reference 69. *Numbers indicate level of evidence harms outcome (example shown in fig 1 69).

10 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

Item 18—study characteristics Recommendations for reporting harms in systematic

PRISMA statement: “For each study, present reviews
characteristics for which data were extracted The PRISMA statement suggests following the PICOS
(eg, study size, PICOS, follow-up period) and format for this item. For reviews of adverse events, we
provide the citations.” would add additional characteristics as follows:
PRISMA harms extension: “Define each harm • Population: patient risk factors that were considered
addressed, how it was ascertained (eg, patient as possibly affecting the risk of the harm outcome
report, active search), and over what time period.” • Intervention: any relevant professional expertise or
skills (for example, if the intervention is a procedure)
• Time: timing of all harms assessments and the length
of follow-up. The timing of assessment of harms will
Figure 2  presents an example on item 18 (study charac-
vary across studies and these differences are import-
ant to document.
PRISMA harms item explanation Review authors should also clarify whether studies
Reporting the characteristics of included studies is were selected based on the length of follow-up. Some
important to allow readers to assess the validity and adverse outcomes may take longer to occur than the
generalisability of the results. Because harms are typi- usual time required to measure efficacy of an interven-
cally not reported or measured in a standardised for- tion (for example, hospital readmission for total hip
mat, we suggest reporting the following for every replacement). An appropriate interval for follow-up
included study, which could be combined with the should be specified a priori for each type of adverse
study characteristics or presented separately: event evaluated by the review author, preferably
during the protocol development. If the timing of the
• Definitions for specific adverse event collection of outcomes in the primary study is insuffi-
• The method of adverse events ascertainment—that is, cient (relative to the time interval necessary for the out-
if passive methods (patients reported a harms as come to occur), then the number of events may be
emerged), or active methods (harms actively sought) underestimated.1 12
were used
• The method of measurement—that is, if any validated Item 19—risk of bias within studies
tool was used to measure them, along with appropri- PRISMA statement: “Present data on risk of bias
ate reference to its validation of each study and, if available, any outcome
• How severity or seriousness was measured. level assessment (see item 12).”

First author, Funding Study type Comparison Methods of AE assessment Dose (mg); Sample size Treatment; Treatment duration
Year [reference] IV or oral comparison (washout period)

Amsden, 1999 [32] Industry Cross-over RCT Alatrofloxacin 200 mg Non-systematic active+passive 500; IV 12 7 (14)

Chien, 1998 [33] NR Parallel RCT Placebo Non-systematic active+passive 750; oral 10 ; 6 7

Chien, 1998 [33] NR Parallel RCT Placebo Non-systematic active+passive 1000; oral 10 ; 6 13

Chien, 1997 [34] Industry Parallel RCT Placebo Non-systematic active+passive 500; oral 10 ; 10 7

Chien, 1997 [34] Industry Parallel RCT Placebo Non-systematic active+passive 500; IV 10 ; 10 7

Chow, 2001 [35] NR Parallel RCT Placebo Systematic objective+non-systematic

active 750; IV 12 ; 6 7

Tsikouris, 2006 [36] NR Cross-over RCT Ciprofloxacin 1000 mg Not defined 400; oral 13 7 (7)

Zhang, 2002 [47] NR Single arm No comparison Not defined 200; IV 10 7

Ayalasomayajula, 2008 [37] Industry Parallel RCT Placebo Systematic objective+non-systematic
active+passive 400; oral 76 ; 77 7

Burkhardt, 2002 [38] NR Cross-over RCT Clarithromycin 500 mg Systematic objective+non-systematic

active 400; oral 12 7 (42)

Peeters, 2008 [39] Industry Cross-over RCT Placebo Systematic objective+non-systematic

active 400; oral 41 8

Sullivan, 1999 [40] NR Parallel RCT Placebo Systematic subjective+systematic

objective+non-systematic active+passive 400; oral 10 ; 5 10

Tsikouris, 2006 [36] NR Cross-over RCT Ciprofloxacin 1000 mg Not defined 400; oral 13 7

Fig 2 | Characteristics of included studies, relating to example in item 18 (study characteristics) of the PRISMA harms checklist. Figure adapted from
reference 70

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 11

Research Methods & Reporting

Example group (b) effect estimates and confidence inter-

Figure 3  presents an example for item 19 (risk of bias vals, ideally with a forest plot.”
within studies).71
Recommendations for reporting harms in systematic Figure 4  presents an example on item 20 (results of
reviews individual studies).72
Review authors should consider the possible sources of
bias that could affect the specific harm under consider- Recommendations for reporting harms in systematic
ation within the review. The study designs may be con- reviews
sidered ideal for efficacy measurement (for example, in It is especially important that review authors report the
terms of standard indicators for risk of bias such as con- actual numbers of adverse events in each study, sepa-
cealment of allocation, sequence generation, and dou- rately for each intervention. This can pose a challenge if
ble blinding). However, consideration of the sample studies report adverse events in heterogeneous formats
selection, dropouts, and measurement of adverse (that is, proportions, or frequency beyond a specific
events should be evaluated separately from the out- threshold (not reporting infrequent harm outcomes)).
comes of benefit as described in item 12. Study authors could be contacted for clarification.
Reporting of this item (item 20) can be combined with
Item 20—results of individual studies information for each study on how adverse events were
PRISMA statement: “For all outcomes consid- assessed (item 18). Graphical display in a forest plot is
ered (benefits or harms), present, for each study: often useful, even when the data are not combined in a
(a) simple summary data for each intervention meta-analysis (such as significant heterogeneity).68

Evaluated item [ 10 ] [ 11 ] [ 12 ] [ 13 ] [ 14 ] [ 15 ] [ 16 ] [ 17 ] [ 18 ] [ 19 ] [ 20 ]

Prospective study design? 1 1 1 0 1 1 1 1 1 1 1

Description of the type of studied ICU? 1 1 1 1 1 1 1 1 1 0 1

Complementary information on the setting environment? 0 0 0 1 0 0 0 1 1 0 0

Study size rationale? 0 0 0 0 0 0 0 0 0 0 0

Definition of ADE according to IOM? 0 0 0 0 0 0 0 0 0 0 1

Evaluation of inter-rater reliability for inclusion decision? 0 0 0 0 0 0 0 0 0 0 0

Description of evaluators’ training? 1 1 1 1 1 1 1 1 1 1 1

Description of patients’ screening? 1 1 1 1 1 1 1 1 1 1 1

Description of inclusion/exclusion criteria? 1 0 0 1 1 1 1 1 1 1 1

Description of collected data? 1 1 0 1 0 1 1 0 1 1 1

Description of the drug history collecting method? 0 0 0 0 0 0 1 0 0 0 0

Description of causality assessment method? 1 0 1 1 1 0 1 1 1 1 0

Description of preventability method/criteria? 0 0 1 1 1 1 1 1 0 1 1

Definition of ADE severity? 1 1 0 1 0 0 1 1 1 0 0

Description of study duration? 1 0 1 1 1 1 1 1 1 1 1

Results for incidence of ADE requiring ICU admission? 1 1 1 1 1 1 1 1 1 1 1

Results for inter-rater reliability for inclusion decision ? 0 0 0 0 0 0 0 0 0 0 0

Description of the characteristics of patients with ADE (age, gender,

severity score at admission, reason of admission, origin of patients)? 0 1 0 0 0 0 1 0 0 0 0

Description of number and classes of drugs suspected to be involved

in the ADE responsible for ICU admission? 1 0 0 1 1 0 1 0 1 1 1

Results for causality assessment? 1 0 0 1 0 0 1 1 0 0 0

Results for preventability rate? 0 0 0 1 1 0 1 1 0 1 1

Results for ADE severity? 1 1 0 1 0 0 1 1 1 0 1

Results for ICU mortality rate of patients with ADE? 1 1 0 1 0 0 1 1 1 0 1

Results for the length of stay for patients with and without ADE (separately)? 0 0 0 1 0 0 1 1 1 0 1

Proportion of items completely reported 0.58 0.42 0.33 0.71 0.46 0.38 0.79 0.67 0.63 0.46 0.63

ADE, adverse drug events; IOM, Institute of Medicine

Fig 3 | Risk of bias in individual studies, relating to example in item 19 (risk of bias within studies) of the PRISMA harms checklist. Figure adapted from
reference 71

12 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

Outcome No of Exp Con Risk ratio Risk ratio P τ2
trials (95% CI) (95% CI)
General safety
Any AE 19 213/858 295/1086 0.91 (0.79 to 1.05) 0.20 0.00
Any SAE 4 9/211 24/427 0.85 (0.41 to 1.74) 0.65 0.00
Dropout overall 13 64/761 75/797 1.22 (0.89 to 1.67) 0.21 0.00
Dropout due to AE 7 8/550 10/563 1.19 (0.48 to 2.97) 0.70 0.00
Dropout due to SAE 1 0/46 1/46 0.33 (0.01 to 7.98) 0.50 -
Organ specific safety
Pain at site of injection 6 29/367 23/373 1.27 (0.31 to 5.16) 0.74 1.80
Nausea 29 100/1180 112/1268 1.04 (0.83 to 1.30) 0.74 0.02
Vomiting 18 38/699 47/729 1.07 (0.73 to 1.57) 0.72 0.00
Other digestive 20 37/895 44/823 1.15 (0.80 to 1.65) 0.46 0.00
Blood (excluding dyscrasias) 1 2/20 2/20 1.00 (0.16 to 6.42) 1.00 -
Eye 1 0/108 1/113 0.66 (0.03 to 15.95) 0.80 -
Ear 1 0/48 1/49 0.33 (0.01 to 7.98) 0.50 -
Cardiovascular 8 8/396 8/396 1.04 (0.42 to 2.59) 0.93 0.00
Musculoskeletal 3 0/177 3/183 0.42 (0.07 to 2.61) 0.35 0.00
Neurological 33 49/1656 87/1475 0.75 (0.57 to 0.99 0.04 0.00
Psychological 5 6/281 14/296 0.75 (0.30 to 1.86) 0.53 0.00
Respiratory 2 1/137 2/134 0.74 (0.05 to 10.46) 0.82 1.18
Skin 17 9/1050 11/892 0.73 (0.37 to 1.45) 0.37 0.00
Urological 5 8/371 3/385 2.78 (0.93 to 8.27) 0.07 0.00
Obstetrical - - - - - -
Female genital - - - - - -
Male genital - - - - - -
General, unspecified 20 24/1199 23/990 0.71 (0.43 to 1.18) 0.19 0.00

0.05 0.1 0.2 0.5 1 2 5 10 20

Favours Favours
metamizole NSAIDs

Fig 4 | Forest plot of adverse events in comparison of metamizole versus non-steroidal anti-inflammatory drugs, relating
to example in item 20 (results of individual studies) of the PRISMA harms checklist. Figure adapted from reference 72.
Adverse events categorised according to the International Classification of Primary Care. Results from single studies are
of limited interpretability but are displayed for the sake of completeness. RR=risk ratio; AE=adverse events; SAE=serious
adverse events

Item 21—synthesis of results disability, congenital anomaly, required inter-

PRISMA statement: “Present results of each vention to prevent permanent damage, other)
meta-analysis done, including confidence inter- were recorded. Only those reports in which IV
vals and measures of consistency.” haloperidol was considered by the reporter to
be the primary causative agent for the adverse
PRISMA harms extension: “Describe any
event were reviewed. Available information
assessment of possible causality.”
included diagnosis, laboratory parameters, QTc
measurement, cardiac symptoms, outcomes
Example and a description of recovery. No peer review
“We reviewed all adverse drug events reported was applied to the MedWatch reports and the
through MedWatch or those submitted by the data reported in this p ­ublication reflect the
manufacturer from November 1997 to April 2008 original information from the FDA MedWatch
through the Freedom of Information Act (FOIA) database.”73
request. The FDA provided a full-text summary
of 5944 reports involving oral, intramuscular PRISMA harms item explanation
and IV use of haloperidol. The FDA data were The assessment of causality is important when review-
transferred to a Microsoft Access database and ing harms, it should be done in the light of the dataset
screened for the key terms torsade, QT, prolon- obtained, and if it was limited to instances where the
gation, wave. Incident report number, date of relation between intervention and adverse reaction was
report, age, gender, origin of report, medication judged as “related,” “probable,” or “possible,” and how
name, role of drug as categorized by the FDA these categories are defined. Review authors should
(suspect, concomitant, primary suspect, sec- report whether causality was assessed, how it was
ondary suspect), route, dose, units, duration, determined, and the definitions used to establish cau-
symptoms and FDA outcome category (death, life sality, such as Bradford Hill’s principles or the World
threatening, hospitalization initial or prolonged, Health Organization causality assessment tool.74 75

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 13

Research Methods & Reporting

Recommendations for reporting harms in systematic imprecision (small number of events), heteroge-
reviews neity (for the outcomes of cardio metabolic risk
When a rare outcome is being reviewed, the synthesis of factors, hemoglobin and hematocrit), and meth-
results could make a major difference to the results and odological limitations of the included trials. In
inferences drawn.76-78 The author should clearly report particular, the brief duration of most testos-
the number of comparisons made and the reasons for terone trials limited inferences about the long
their choices. If data from unpublished sources are term safety of this treatment. In addition, pub-
included, report clearly the data source and the effect of lication and reporting biases likely affected the
these studies on the results of the systematic review. inferences in this review because not all studies
reported the outcomes of interest.”79
Item 22—risk of bias across studies
PRISMA statement: “Present results of any Recommendations for reporting harms in systematic
assessment of risk of bias across studies (see reviews
item 15).” Issues of missing data and heterogeneity in data collec-
tion or definitions of harms are common limitations of
No specific additional information is required for sys- reviews addressing harms. It is important to recognise
tematic reviews of harms. See item 15. possible limitations of meta-analysis for rare adverse
events (that is, quality and quantity of data), issues
Item 23—additional analysis noted previously related to collection and reporting.
PRISMA statement: “Give results of additional
analyses, if done (eg, sensitivity or subgroup Item 26—conclusions
analyses, meta-regression (see item 16)).” PRISMA statement:“Provide a general inter-
pretation of the results in the context of other
No specific additional information is required for sys- evidence, and implications for future research.”
tematic reviews of harms.
Item 24—summary of evidence “Further complicating the interpretation of our
PRISMA statement: “Summarise the main find- findings is that few high-quality studies have
ings including the strength of evidence for each compared the safety profiles of other induction
main outcome; consider their relevance to key agents. None of the induction agents used in
groups (eg, healthcare providers, users, and the ED are devoid of potential adverse effects.
policy makers).” Our study did not examine immediate adverse
events such as hemodynamic and cardio
No specific additional information is required for sys- depressant adverse effects and factors such as
tematic reviews of harms. ease of tracheal intubation and simplicity of use
that also need to be taken into consideration
Item 25—limitations when choosing the most appropriate induction
PRISMA statement: “Discuss limitations at agent for a given patient.
study and outcome level (eg, risk of bias), and
The available evidence suggests that etomi-
at review level (eg, incomplete retrieval of iden-
date suppresses adrenal function transiently,
tified research, reporting bias).”
without demonstrating a significant effect
on mortality. However, to our knowledge no
studies to date have been powered to detect
“This systematic review and meta-analyses
a difference in mortality or time in the hospi-
demonstrated that testosterone therapy in men
tal, the ICU, or receiving ventilator support.
was associated with significant increases in
According to robust evidence that etomidate
hemoglobin and hematocrit . . . However, cau-
transiently decreases adrenal function and
tion should be exercised in interpreting these
that a significant effect on mortality cannot
analyses because they are considered observa-
be excluded, alternate induction agents may
tional in nature (despite the fact that the original
be considered for use in rapid sequence intu-
studies were randomized) and the associations
bation, particularly for septic patients. More
found can be attributable to chance due to the
data are needed to determine etomidate’s
multiple simultaneous comparisons. Subgroup
effect on mortality.”80
interactions generated by study-level meta-anal-
yses are considered hypothesis- generating and
should be confirmed at a patient-level (in a large Recommendations for reporting harms in systematic
trial or individual patient meta-analysis) before reviews
clinical implications are inferred. It is important to state conclusions in accordance with
the review findings. Not infrequently, the review author
. . . Nevertheless, the quality of the evidence states the limitations and weaknesses of the included
varied from low to medium considering the studies regarding adverse event reporting in the

14 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

­ iscussion (for example, high risk of bias, poor report-

d (­methods) and what was found (results), and encourages
ing of adverse events), but the conclusions fail to repre- reporting of reasons for methodological choices made.
sent these weaknesses. Because of poor quality and Peer reviewers and journal editors can also benefit from
quantity of data on harms, it is often not possible to implementing PRISMA harms, because it clarifies partic-
draw strong conclusions. In particular, when adverse ular issues that affect reviews addressing adverse events.
events were not identified, we caution against the con- Improved reporting of benefits and harms can facilitate
clusion that the intervention is “safe,” when, in reality, evidence informed decision making by healthcare pro-
its safety remains unknown. fessionals, policy makers, and patients.
As an extension of the PRISMA statement, PRISMA
Item 27—funding harms should be used in every systematic review
PRISMA statement: “Describe sources of fund- assessing adverse events as a primary or secondary out-
ing for the systematic review and other support come. The four essential PRISMA harms items are a
(eg, supply of data); role of funders for the sys- minimum set to be reported to provide transparency.
tematic review.” We strongly suggest that the “recommendation for
reporting harms in systematic reviews” relating to the
No specific additional information is required for sys- main PRISMA checklist should also be used to report
tematic reviews of harms. every systematic review assessing adverse events, to
promote complete reporting. Like all reporting guide-
Discussion lines, PRISMA harms will be improved through critical
The PRISMA harms add to PRISMA because systematic review and feedback. Its goal is not only to improve
reviews of harms differ from reviews of efficacy. Import- reporting, but also to stimulate an increase in the num-
ant differences in search strategy, screening, assess- ber of reviews addressing harms, promoting a balanced
ment of bias, and analysis must be considered when assessment of health interventions.
reviewing unintended effects of interventions.33 43-46 56 57  We encourage journals that already endorse the
As harms are often rare, “zero” is an important value. PRISMA statement to extend endorsement to include
PRISMA harms clarifies the importance of distinguish- PRISMA harms, and other journals to endorse both doc-
ing between the absence of an event, an event that was uments. Better reporting of systematic reviews depends
not measured, or an event that was not reported.12  Even not on endorsement alone but on adherence to the rec-
when harms are reported, different formats can make it ommendations, ultimately leading to transparency of
difficult to pool and summarise data across studies, information and improvement of patient care.
especially if primary reports did not have a comparison PRISMA harms steering committee: Sunita Vohra (Convenor)
group. PRISMA harms encourage transparent reporting (University of Alberta), David Moher (Ottawa Hospital Research
Institute; University of Ottawa), Doug Altman (University of Oxford),
of how harms were evaluated, including any causality Yoon Loke (University of East Anglia), John Ioannidis (Stanford
assessment. Although a systematic review cannot rem- University), Pasqualina Santaguida (McMaster University), Su Golder
edy deficiencies in primary studies, the review authors (University of York), Liliane Zorzela (University of Alberta). All authors
participated in the development of this guideline, collaborating in the
have a unique opportunity to access and evaluate the draft of the manuscript and approved the submitted version.
entire evidence base, and both strengths and deficits in PRISMA harms group (collaborators): Heather Boon (University of
primary studies should be highlighted in the systematic Toronto), Jocalyn Clark (University of Toronto, at time of work on
PRISMA harms senior editor, PLOS Medicine), Sheena Derry (University
review.12 56
of Oxford), Jim Gallivan (Health Canada), Paula Gardiner (Boston
Systematic reviews can present a misleading picture University School of Medicine), Peter Gøtzsche (Nordic Cochrane
to readers if the lack of evidence of harm is presented as Center), Elizabeth Loder (acting head of research, The BMJ), Maryann
Napoli (Center for Medical Consumers), Karen Pilkington (University of
evidence of safety. Inclusion of studies that did not
Westminster), Paul Shekelle (Veterans Affairs Greater Los Angeles
report on harms can be as frequent as 75% in Cochrane Health Care System), Sonal Singh (Johns Hopkins University School of
reviews and 47% in non-Cochrane reviews.56  Special Medicine), Claudia Witt (University Zurich), Toby Lasserson (Cochrane
Editorial Unit), Taixiang Wu (Chinese Clinical Trial Registry, Chinese
attention should be given to systematic reviews that
Cochrane Center), Larissa Shamseer (Ottawa Hospital Research
underpin the development of treatment guidelines, Institute; University of Ottawa), Cynthia Mulrow (senior deputy editor,
because harms can be misrepresented and under-re- Annals of Internal Medicine)
ported in those reviews. Treatment guidelines are ulti- Competing interests: None declared.
mately the final pathway to translate research findings Funding: PRISMA harms was partly funded by Alberta Innovates,
Health Solutions. The researchers were fully independent from funders.
and should represent efficacy and harms equally. Poor
Ethical approval: Approval granted by the University of Alberta (no
reporting in research has serious implications for waste Pro00021294).
in healthcare research investments because it requires Provenance and peer review: Not commissioned; externally peer
duplication of experiments.81-84  There is strong evi- reviewed.
dence that poor reporting of efficacy—and potentially of 1 Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred
reporting items for systematic reviews and meta-analyses: the
harms, in animal studies, clinical studies, and system- PRISMA statement. PLoS Med 2009;6:e1000097. doi:10.1371/
atic reviews—lead to misinterpretation of research journal.pmed.1000097.
results and ultimately contributes to poor patient 2 Gray JAM. Evidence-based healthcare. How to make health policy and
management decisions.Churchill Livingstone, 2001.
care.12 56-58 81-85 3 Sackett DL, Wennberg JE. Choosing the best research design for each
PRISMA harms have been designed to help promote question. BMJ 1997;315:1636. doi:10.1136/bmj.315.7123.1636.
4 Ioannidis JP. Adverse events in randomized trials: neglected,
transparency in reporting. The guidance extension asks restricted, distorted, and silenced. Arch Intern Med 2009;169:1737-9.
authors to describe in particular what was done doi:10.1001/archinternmed.2009.313.

the bmj | BMJ 2016;352:i157 | doi: 10.1136/bmj.i157 15

Research Methods & Reporting

5 Singh S, Loke YK. Drug safety assessment in clinical trials: 30 Loke YK, Price D, Herxheimer A. Cochrane Adverse Effects Methods
methodological challenges and opportunities. Trials 2012;13:138. Group. Systematic reviews of adverse effects: framework for a
doi:10.1186/1745-6215-13-138. structured approach. BMC Med Res Methodol 2007;7:32.
6 Papanikolaou PN, Ioannidis JP. Availability of large-scale evidence on doi:10.1186/1471-2288-7-32.
specific harms from systematic reviews of randomized trials. Am J Med 31 Arbyn M, Kyrgiou M, Simoens C, et al Perinatal mortality and other
2004;117:582-9. doi:10.1016/j.amjmed.2004.04.026. severe adverse pregnancy outcomes associated with treatment of
7 Pitrou I, Boutron I, Ahmad N, Ravaud P. Reporting of safety results in cervical intraepithelial neoplasia: meta-analysis. BMJ 2008;337:a1284.
published reports of randomized controlled trials. Arch Intern Med 32 Loke YK, Jeevanantham V, Singh S. Bisphosphonates and atrial
2009;169:1756-61. doi:10.1001/archinternmed.2009.306. fibrillation: systematic review and meta-analysis. Drug Saf 2009;32:219-
8 Loke YK, Derry S. Reporting of adverse drug reactions in randomised 28. doi:10.2165/00002018-200932030-00004.
controlled trials - a systematic survey. BMC Clin Pharmacol 2001;1:3. 33 Hartley J. Clarifying the abstracts of systematic literature reviews. Bull
doi:10.1186/1472-6904-1-3. Med Libr Assoc 2000;88:332-7.
9 Ioannidis JP, Lau J. Completeness of safety reporting in randomized 34 Banach R, Boskovic R, Einarson T, Koren G. Long-term developmental
trials: an evaluation of 7 medical areas. JAMA 2001;285:437-43. outcome of children of women with epilepsy, unexposed or exposed
doi:10.1001/jama.285.4.437. prenatally to antiepileptic drugs: a meta-analysis of cohort studies.
10 Yazici Y. Some concerns about adverse event reporting in randomized Drug Saf 2010;33:73-9. doi:10.2165/11317640-000000000-00000.
clinical trials. Bull NYU Hosp Jt Dis 2008;66:143-5. 35 Loke YK, Kwok CS, Singh S. Comparative cardiovascular effects of
11 Ernst E, Pittler MH. Assessment of therapeutic safety in systematic thiazolidinediones: systematic review and meta-analysis of
reviews: literature review. BMJ 2001;323:546. doi:10.1136/ observational studies. BMJ 2011;342:d1309. doi:10.1136/bmj.d1309.
bmj.323.7312.546. 36 Agbabiaka TB, Pittler MH, Wider B, Ernst E. Serenoa repens (saw
12 Zorzela L, Golder S, Liu Y, et al. Quality of reporting in systematic palmetto): a systematic review of adverse events. Drug Saf
reviews of adverse events: systematic review. BMJ 2014;348:f7668. 2009;32:637-47. doi:10.2165/00002018-200932080-00003.
doi:10.1136/bmj.f7668. 37 Hammad TA, Neyarapally GA, Pinheiro SP, Iyasu S, Rochester G, Dal
13 Golder S, Loke YK, Zorzela L. Some improvements are apparent in Pan G. Reporting of meta-analyses of randomized controlled trials
identifying adverse effects in systematic reviews from 1994 to 2011. with a focus on drug safety: an empirical assessment. Clin Trials
J Clin Epidemiol 2013;66:253-60. doi:10.1016/j.jclinepi.2012.09.013. 2013;10:389-97. doi:10.1177/1740774513479467.
14 McIntosh HM, Woolacott NF, Bagnall AM. Assessing harmful effects in 38 Michele TM, Pinheiro S, Iyasu S. The safety of tiotropium--the FDA’s
systematic reviews. BMC Med Res Methodol 2004;4:19. conclusions. N Engl J Med 2010;363:1097-9. doi:10.1056/
doi:10.1186/1471-2288-4-19. NEJMp1008502.
15 Cornelius VR, Perrio MJ, Shakir SA, Smith LA. Systematic reviews of 39 Anello C, O’Neill RT. Does research synthesis have a place in drug
adverse effects of drug interventions: a survey of their conduct and regulatory policy?Clin Res Regul Aff 1996;13:13-
reporting quality. Pharmacoepidemiol Drug Saf 2009;18:1223-31. 21doi:10.3109/10601339609019625. .
doi:10.1002/pds.1844. 40 Golder S, Loke Y, McIntosh HM. Room for improvement? A survey of
16 Loke YK, Golder SP, Vandenbroucke JP. Comprehensive evaluations of the methods used in systematic reviews of adverse effects. BMC Med
the adverse effects of drugs: importance of appropriate study Res Methodol 2006;6:3. doi:10.1186/1471-2288-6-3.
selection and data sources. Ther Adv Drug Saf 2011;2:59-68. 41 Golder S, Loke YK, Bland M. Unpublished data can be of value in
doi:10.1177/2042098611401129. systematic reviews of adverse effects: methodological overview. J Clin
17 Chou R, Fu R, Carson S, Saha S, Helfand M. Methodological Epidemiol 2010;63:1071-81. doi:10.1016/j.jclinepi.2010.02.009.
shortcomings predicted lower harm estimates in one of two sets of 42 Bangalore S, Kumar S, Wetterslev J, Messerli FH. Angiotensin receptor
studies of clinical interventions. J Clin Epidemiol 2007;60:18-28. blockers and risk of myocardial infarction: meta-analyses and trial
doi:10.1016/j.jclinepi.2006.02.021. sequential analyses of 147 020 patients from randomised trials. BMJ
18 Chou R, Helfand M. Challenges in systematic reviews that assess 2011;342:d2234. doi:10.1136/bmj.d2234.
treatment harms. Ann Intern Med 2005;142:1090-9. 43 Golder S, Loke Y, McIntosh HM. Poor reporting and inadequate
doi:10.7326/0003-4819-142-12_Part_2-200506211-00009. searches were apparent in systematic reviews of adverse effects. J
19 Hopewell S, Wolfenden L, Clarke M. Reporting of adverse events in Clin Epidemiol 2008;61:440-8. doi:10.1016/j.jclinepi.2007.06.005.
systematic reviews can be improved: survey results. J Clin Epidemiol 44 Derry S, Loke YK, Aronson JK. Incomplete evidence: the inadequacy of
2008;61:597-602. doi:10.1016/j.jclinepi.2007.10.005. databases in tracing published adverse drug reactions in clinical
20 Hernandez AV, Walker E, Ioannidis JP, Kattan MW. Challenges in trials. BMC Med Res Methodol 2001;1:7. doi:10.1186/1471-2288-1-7.
meta-analysis of randomized clinical trials for rare harmful 45 Golder S, Loke YK. The contribution of different information sources
cardiovascular events: the case of rosiglitazone. Am Heart J for adverse effects data. Int J Technol Assess Health Care 2012;28:133-
2008;156:23-30. doi:10.1016/j.ahj.2008.03.002. 7. doi:10.1017/S0266462312000128.
21 Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational 46 Golder S, Loke YK. Failure or success of electronic search strategies to
studies in epidemiology: a proposal for reporting. Meta-analysis Of identify adverse effects data. J Med Libr Assoc 2012;100:130-4.
Observational Studies in Epidemiology (MOOSE) group. JAMA doi:10.3163/1536-5050.100.2.012.
2000;283:2008-12. doi:10.1001/jama.283.15.2008. 47 Bellis JR, Pirmohamed M, Nunn AJ, et al. Dexamethasone and
22 Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of haemorrhage risk in paediatric tonsillectomy: a systematic review and
health research reporting guidelines. PLoS Med 2010;7:e1000217. meta-analysis. Br J Anaesth 2014;113:23-42. doi:10.1093/bja/aeu152.
doi:10.1371/journal.pmed.1000217. 48 Leombruno JP, Einarson TR, Keystone EC. The safety of anti-tumour
23 Beller EM, Glasziou PP, Altman DG, et al. PRISMA for Abstracts Group. necrosis factor treatments in rheumatoid arthritis: meta and
PRISMA for Abstracts: reporting systematic reviews in journal and exposure-adjusted pooled analyses of serious adverse events. Ann
conference abstracts. PLoS Med 2013;10:e1001419. doi:10.1371/ Rheum Dis 2009;68:1136-45. doi:10.1136/ard.2008.091025.
journal.pmed.1001419. 49 Albavera-Hernández C, Rodríguez JM, Idrovo AJ. Safety of
24 Shamseer L, Moher D, Clarke M, et al. PRISMA-P Group. Preferred botulinum toxin type A among children with spasticity secondary
reporting items for systematic review and meta-analysis protocols to cerebral palsy: a systematic review of randomized clinical trials.
(PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647. Clin Rehabil 2009;23:394-407. doi:10.1177/0269215508099860.
doi:10.1136/bmj.g7647. 50 Rücker G, Schwarzer G, Carpenter J, Olkin I. Why add anything to
25 Pilkington K, Boshnakova A. Complementary medicine and safety: a nothing? The arcsine difference as a measure of treatment effect in
systematic investigation of design and reporting of systematic meta-analysis with zero cells. Stat Med 2009;28:721-38. doi:10.1002/
reviews. Complement Ther Med 2012;20:73-82. doi:10.1016/j. sim.3511.
ctim.2011.10.002. 51 Bradburn MJ, Deeks JJ, Berlin JA, Russell Localio A. Much ado about
26 Murphy MK, Black NA, Lamping DL, et al. Consensus development nothing: a comparison of the performance of meta-analytical
methods, and their use in clinical guideline development. Health methods with rare events. Stat Med 2007;26:53-77. doi:10.1002/
Technol Assess 1998;2:i-iv, 1-88. sim.2528.
27 Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews 52 Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? Use and
of interventions. Version 5.1.0 [updated March 2011]. Cochrane avoidance of continuity corrections in meta-analysis of sparse
Collaboration, 2011. data[correction in: Stat Med 2006;25:2700]. Stat Med 2004;23:1351-
28 Chou R, Aronson N, Atkins D, et al. AHRQ series paper 4: assessing 75. doi:10.1002/sim.1761.
harms when comparing medical interventions: AHRQ and the 53 Stijnen T, Hamza TH, Ozdemir P. Random effects meta-analysis of
effective health-care program. J Clin Epidemiol 2010;63:502-12. event outcome in the framework of the generalized linear mixed
doi:10.1016/j.jclinepi.2008.06.007. model with applications in sparse data. Stat Med 2010;29:3046-67.
29 Ioannidis JP, Evans SJ, Gøtzsche PC, et al. CONSORT Group. Better doi:10.1002/sim.4040.
reporting of harms in randomized trials: an extension of the 54 Mattishent K, Loke YK. Bi-directional Interaction Between
CONSORT statement. Ann Intern Med 2004;141:781-8. Hypoglycaemia And Cognitive Impairment In Elderly Patients Treated
doi:10.7326/0003-4819-141-10-200411160-00009. with Glucose Lowering Agents: Systematic Review and Meta-analysis.
Diabetes Obes Metab 2015.

16 doi: 10.1136/bmj.i157 | BMJ 2016;352:i157 | the bmj

Research Methods & Reporting

55 Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. 70 Langendam MW, Tiemersma EW, van der Werf MJ, Sandgren A.
Selective reporting bias of harm outcomes within studies: findings Adverse events in healthy individuals and MDR-TB contacts treated
from a cohort of systematic reviews. BMJ 2014;349:g6501. with anti-tuberculosis drugs potentially effective for preventing
doi:10.1136/bmj.g6501. development of MDR-TB: a systematic review. PLoS One
56 Loke YK, Mattishent K. If nothing happens, is everything all right? 2013;8:e53599. doi:10.1371/journal.pone.0053599.
Distinguishing genuine reassurance from a false sense of security. 71 Jolivot PA, Hindlet P, Pichereau C, et al. A systematic review of adult
CMAJ 2015;187:15-6. doi:10.1503/cmaj.141344. admissions to ICUs related to adverse drug events. Crit Care
57 Kirkham JJ, Dwan KM, Altman DG, et al. The impact of outcome 2014;18:643. doi:10.1186/s13054-014-0643-5.
reporting bias in randomised controlled trials on a cohort of 72 Kötter T, da Costa BR, Fässler M, et al. Metamizole-associated adverse
systematic reviews. BMJ 2010;340:c365. doi:10.1136/bmj.c365. events: a systematic review and meta-analysis. PLoS One
58 Page MJ, McKenzie JE, Kirkham J, et al. Bias due to selective inclusion 2015;10:e0122918. doi:10.1371/journal.pone.0122918.
and reporting of outcomes and analyses in systematic reviews of 73 Meyer-Massetti C, Cheng CM, Sharpe BA, Meier CR, Guglielmo BJ. The
randomised trials of healthcare interventions. Cochrane Database FDA extended warning for intravenous haloperidol and torsades de
Syst Rev 2014;10:MR000035. doi:10.1002/14651858.MR000035. pointes: how should institutions respond?J Hosp Med 2010;5:E8-16.
pub2. doi:10.1002/jhm.691.
59 Keus F, Wetterslev J, Gluud C, Gooszen HG, van Laarhoven CJ. Robustness 74 Hill AB. The Environment and Disease: Association or Causation?Proc
assessments are needed to reduce bias in meta-analyses that include R Soc Med 1965;58:295-300.
zero-event randomized trials. Am J Gastroenterol 2009;104:546-51. 75 Uppsala Monitoring Centre. World Health Organization causality
doi:10.1038/ajg.2008.22. assessment criteria. 2015.
60 Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. 76 Shuster JJ, Jones LS, Salmon DA. Fixed vs random effects meta-
Empirical evidence for selective reporting of outcomes in randomized analysis in rare event studies: the rosiglitazone link with myocardial
trials: comparison of protocols to published articles. JAMA infarction and cardiac death. Stat Med 2007;26:4375-85.
2004;291:2457-65. doi:10.1001/jama.291.20.2457. doi:10.1002/sim.3060.
61 Scharf O, Colevas AD. Adverse event reporting in publications 77 Shuster JJ. Empirical vs natural weighting in random effects
compared with sponsor database for cancer clinical trials. J Clin Oncol meta-analysis. Stat Med 2010;29:1259-65. doi:10.1002/sim.3607.
2006;24:3933-8. doi:10.1200/JCO.2005.05.3959. 78 Ioannidis JP, Hozo I, Djulbegovic B. Optimal type I and type II error
62 Mahoney MR, Sargent DJ. Adverse-event rates: journals versus pairs when the available sample size is fixed. J Clin Epidemiol
databases. Lancet 2007;369:171-2. doi:10.1016/ 2013;66:903-910.e2. doi:10.1016/j.jclinepi.2013.03.002.
S0140-6736(07)60086-5. 79 Fernández-Balsells MM, Murad MH, Lane M, et al. Clinical review 1:
63 Chan AW, Krleza-Jerić K, Schmid I, Altman DG. Outcome reporting bias Adverse effects of testosterone therapy in adult men: a systematic
in randomized trials funded by the Canadian Institutes of Health review and meta-analysis. J Clin Endocrinol Metab 2010;95:2560-75.
Research. CMAJ 2004;171:735-40. doi:10.1503/cmaj.1041086. doi:10.1210/jc.2009-2575.
64 Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for 80 Hohl CM, Kelly-Smith CH, Yeung TC, Sweet DD, Doyle-Waters MM,
examining and interpreting funnel plot asymmetry in meta-analyses Schulzer M. The effect of a bolus dose of etomidate on cortisol levels,
of randomised controlled trials. BMJ 2011;343:d4002. doi:10.1136/ mortality, and health services utilization: a systematic review. Ann
bmj.d4002. Emerg Med 2010;56:105-13.e5. doi:10.1016/j.
65 Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the annemergmed.2010.01.030.
misleading funnel plot. BMJ 2006;333:597-600. doi:10.1136/ 81 Moher D, Glasziou P, Chalmers I, et al. Increasing value and reducing
bmj.333.7568.597. waste in biomedical research: who’s listening?Lancet 2015,.
66 Lathyris D, Panagiotou OA, Baltogianni M, Ioannidis JP, Contopoulos- doi:10.1016/S0140-6736(15)00307-4.
Ioannidis DG. Safety of medical interventions in children versus 82 Glasziou P, Altman DG, Bossuyt P, et al. Reducing waste from
adults. Pediatrics 2014;133:e666-73. doi:10.1542/peds.2013-3128. incomplete or unusable reports of biomedical research. Lancet
67 Singh S, Loke YK, Enright PL, Furberg CD. Mortality associated with 2014;383:267-76. doi:10.1016/S0140-6736(13)62228-X. 24411647.
tiotropium mist inhaler in patients with chronic obstructive pulmonary 83 Shamseer L, Galipeau J, Turner L, Moher D. Improving the reporting
disease: systematic review and meta-analysis of randomised controlled and usability of research studies. Can J Anaesth 2013;60:337-44.
trials. BMJ 2011;342:d3215. doi:10.1136/bmj.d3215. doi:10.1007/s12630-013-9895-9.
68 Ioannidis JP, Patsopoulos NA, Rothstein HR. Reasons or excuses for 84 Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the
avoiding meta-analysis in forest plots. BMJ 2008;336:1413-5. CONSORT Statement impact the completeness of reporting of
doi:10.1136/bmj.a117. randomised controlled trials published in medical journals? A Cochrane
69 Bain ES, Middleton PF, Crowther CA. Maternal adverse effects of review. Syst Rev 2012;1:60. doi:10.1186/2046-4053-1-60.
different antenatal magnesium sulphate regimens for improving 85 Macleod MR, Michie S, Roberts I, et al. Biomedical research:
maternal and infant outcomes: a systematic review. BMC Pregnancy increasing value, reducing waste. Lancet 2014;383:101-4.
Childbirth 2013;13:195. doi:10.1186/1471-2393-13-195. doi:10.1016/S0140-6736(13)62329-6.

© BMJ Publishing Group Ltd 2016

No commercial reuse: See rights and reprints Subscribe: