Chapter 19: Adverse effects

Guy Peryer, Su Golder, Daniela R Junqueira, Sunita Vohra, Yoon Kong Loke; on behalf of the
Cochrane Adverse Effects Methods Group

Key Points:

• To achieve a balanced perspective, all reviews should try to consider adverse aspects of
interventions.

• A detailed analysis of adverse effects is particularly relevant when evidence on the

potential for harm has a major influence on treatment or policy decisions.

• There are major challenges in specifying relevant outcomes and study designs for
systematic reviews evaluating adverse effects. This is due to high diversity in the number
and type of possible adverse effects, as well as variation in their definition, methods of
ascertainment, incidence and time-course.

• Review authors should pre-specify their approach to reviewing studies of adverse effects
within the review protocol. The approach may be confirmatory (focused on particular
adverse effects of interest), exploratory (opportunistic capture of any adverse effects
that happen to be reported), or a hybrid (combination of both).

• Depending on the approach used and outcomes of interest to the review, identification
of relevant adverse effects data may require a bespoke search process that includes a
wider selection of sources than that required to identify data on beneficial outcomes.

• Because adverse effects data are often handled with less rigour than the primary
beneficial outcomes of a study, review authors must recognize the possibility of poor
case definition, inadequate monitoring and incomplete reporting when synthesizing
data.

Cite this chapter as: Peryer G, Golder S, Junqueira D, Vohra S, Loke YK. Chapter 19: Adverse
effects. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA
(editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated
February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

19.1 Introduction to issues in addressing adverse effects

Every healthcare intervention comes with the risk, great or small, of harmful or adverse
effects. A Cochrane Review that considers only the favourable outcomes of the interventions
that it examines, without also assessing the adverse effects, will lack balance and may make
the intervention look more favourable than it should. All reviews should try to consider the
adverse aspects of interventions.

Copyright © 2022 The Cochrane Collaboration

 Copyright © 2022 The Cochrane Collaboration

This chapter addresses special issues about adverse effects in Cochrane Reviews. It focuses
on methodological differences when assessing adverse effects compared with other
outcomes.

19.1.1 Terminology and definitions

Poor standardization and usage of adverse effects terminology in study reports can produce
challenges for review authors. Common, and closely related, terms include adverse event,
adverse effect, serious adverse event, serious adverse effects, adverse drug reaction, side
effect, complications and harms (Zorzela et al 2016). In this chapter we use the term adverse
event for an unfavourable or harmful outcome that occurs during, or after, the use of a drug
or other intervention, but is not necessarily caused by it, and an adverse effect (or harm)
as an adverse event for which the causal relation between the intervention and the event is
at least a reasonable possibility.

19.1.2 Special issues for addressing adverse effects

In this section we discuss some of the particular challenges when addressing adverse
effects. First, there can be wide diversity across studies in how adverse events are defined,
ascertained, analysed and reported. Second, adverse effects may not be known when
studies were planned, so data collection processes and analytic strategies may not be in
place. Third, many adverse events are too uncommon or too long-term to be observed
within randomized trials.

19.1.2.1 Diversity in defining and monitoring of adverse events

A huge range of adverse events can occur in a research study, and there are multiple ways
in which adverse effects can be ascertained and categorized by study investigators (Smith
et al 2015). There are two broad strategies for collecting information on adverse events.
Study investigators may use active monitoring or surveillance, which directs enquiry
towards pre-defined adverse events of interest, usually following protocol-defined
procedures for data collection, case definitions and adjudication. For example, if the event
of interest is myocardial infarction, the study protocol might require collection of laboratory
and electrocardiogram data for suspected events. These results might then be referred to
an independent panel which adjudicates or ascertains the occurrence of an event. Such
active monitoring usually relates to sets of potential adverse events that are either known
or suspected to be associated with an intervention.

Although prospective collection of adverse event data is desirable, many adverse effects
cannot be pre-specified because they are not yet known or suspected to be associated with
an intervention. Thus, spontaneous report monitoring may occur, which involves
recording all adverse events (pre-defined or not) throughout the duration of the study. Both
participants and researchers recognizing any adverse event can file a report at any time.
This may uncover new or unexpected adverse effects not previously associated with the
intervention. For regulated products (e.g. drugs, biologics, vaccines), spontaneously
reported adverse events are usually coded, grouped and categorized following established
dictionaries for analysis and presentation.

Whichever monitoring method is used to collect information about adverse events, study
investigators may combine adverse events into global or composite measures, which are
 Copyright © 2022 The Cochrane Collaboration

often reported as total number of serious adverse events, or number of withdrawals due to
adverse events, or total number of adverse events in an anatomic or organ system (e.g.
gastrointestinal, cardiovascular). However, these composite measures do not give
information on what exactly the events were, and so it is usually necessary to drill down for
details of distinct or individual adverse events, such as nausea or rash.

Ideally, the definition and ascertainment of adverse events should be as uniform as possible
across the included studies in the review. The lack of systematic monitoring or follow-up,
coupled with divergent methods of seeking, verifying and classifying adverse events, can
introduce heterogeneity in effect estimates among studies. Review authors will therefore
need to pay close attention to outcome definition and method of monitoring when
interpreting or comparing frequencies, rates and risk estimates for adverse effects.

19.1.2.2 Inconsistent and poor reporting of adverse effects

Inconsistent outcome definition and poor ascertainment are problematic for reviews that
rely exclusively on published data. Information taken from published reports may be
incomplete or lack specificity. Across multiple investigations of published versus
unpublished studies, Golder and colleagues found a median of 43% of published studies
reported adverse events data, compared with a median of 83% of unpublished studies
(Golder et al 2016). A wider range of specific adverse events was found in sources other than
published journal articles. In addition, when published and unpublished reports of the same
study were compared, it was shown that the unpublished version was more likely to contain
adverse effects data (median 95%) compared with the published version (median 46%).
Similarly, a study of an obesity drug (orlistat or Xenical) by Schroll and colleagues compared
study documents (protocol, clinical study report (CSR), and published report), and
identified important inconsistencies (Schroll et al 2016). For example, adverse events in
published studies were coded to appear less severe, with reduced incidence, compared with
events reported in the unpublished CSRs. Of the total number of adverse events reported by
trial investigators in CSRs, between 3% and 33% were subsequently reported in the
corresponding published journal articles.

19.1.2.3 Different study designs to measure adverse events

Some adverse effects occur rarely or may only become apparent long after the start of
intervention. This contrasts with adverse effects that have a higher incidence and occur
soon after the intervention is delivered. A small randomized trial with only short-term
follow-up may be able to capture common, immediately apparent adverse effects (e.g. skin
reaction after injection) adequately. However, rare or long-term adverse effects may only
be observed in non-randomized studies such as large cohort studies or case-control studies.
Therefore, depending on the type of adverse outcome of interest, review authors may need
to consider evidence extending beyond the time frame of randomized trials.

19.2 Formulation of the review

A starting point for assessing adverse effects of an intervention is to consider whether a
review will evaluate both beneficial and adverse effects of an intervention, or just the
adverse effects. Although most Cochrane Reviews look at both beneficial and adverse
 Copyright © 2022 The Cochrane Collaboration

effects, review authors may decide to conduct a separate review of only the adverse effects
of an intervention (see Box 19.2.a). Whichever strategy is taken, review authors will need to
decide whether to focus only on a pre-specified set of adverse events (a ‘confirmatory’
approach), or analyse data on adverse events identified during the conduct of the review
(an ‘exploratory’ approach). In practice, some review authors will use a hybrid of these two
approaches. Consideration will also be needed of whether the same sources of evidence will
be used to look at beneficial and adverse effects, or whether additional types of evidence
will be sought to examine the adverse effects. Finally, the specific selection and definition
of adverse effects will need to be considered. In this section we tackle these key
considerations for formulating a review to look at adverse effects.

Box 19.2.a Reviews of adverse effects alone

For an intervention that is given for a variety of diseases or conditions, yet whose adverse
effect profile might be expected to be similar in different populations and settings, it may
be reasonable to examine adverse effects regardless of the condition for which the
intervention was delivered. This can be achieved in a stand-alone Cochrane Review
focusing only on adverse effects.

For example, aspirin is used for many conditions, such as in patients after a stroke, with
peripheral vascular disease, and with coronary artery disease. The main effects of aspirin
on outcomes relevant to these different conditions would typically be addressed in
separate Cochrane Reviews. However, the mechanism of harm and susceptibility to
adverse effects (such as bleeding into the brain or gut) are sufficiently similar across the
different disease groups that an independent review might address them together.
Indeed, if trials exist on combined populations, such a question would be difficult to
address in any other way.

Similarly, there may be limited adverse effects data for an intervention in a

subpopulation. Analysing all available data for this subpopulation – such as adverse
effects of selective serotonin reuptake inhibitors in children – may be worthwhile, even if
the trials were aimed at different disease conditions.

Reviews of adverse effects alone should provide adequate cross-referencing to related

reviews of intended effects of the intervention. If new safety concerns are identified when
an efficacy review is updated, then the adverse effects review should be updated as soon
as possible.

19.2.1 Which adverse events to look at

19.2.1.1 Confirmatory approach
In a confirmatory approach, review authors list one or more adverse effects as outcomes of
interest in their review protocol. Golder and colleagues found that approximately 80% of
systematic reviews of adverse effects published between 1994 and 2011 used this approach,
selecting particular events, or categories of events, as their main interest (Golder et al 2013).
 Copyright © 2022 The Cochrane Collaboration

When adopting the confirmatory approach, review authors should aim to pre-specify
adverse effects that are anticipated or already recognized to be associated with the
intervention, and assumed to be measured regularly and consistently in studies. Selection
of adverse effects of interest can be based on biological, physiological or psychological
plausibility. For example, in a review of a surgical intervention it is plausible to pre-specify
‘wound infection’ as an adverse outcome of interest. Similarly, a systematic review of drug
therapy that affects platelets or clotting would be justified in pre-specifying bleeding as an
adverse outcome of interest. In some cases, it may be reasonable to select adverse effects
for review based on previously established observation or association, although the
plausible mechanism of effect has not yet been established.

A key limitation of the confirmatory approach is the inability to handle unanticipated

adverse effects that are reported in the included studies.

19.2.1.2 Exploratory approach

An exploratory approach to reviewing adverse effects does not include pre-specification of
any particular adverse outcomes of interest. Rather, it typically involves extracting any, or
all, of the adverse event data found within the included studies. Only about 20% of reviews
of adverse effects specify this as their main approach (Golder et al 2013).

The exploratory approach can identify unanticipated and rare adverse effects of an
intervention. This may inform which outcomes are investigated in future reviews of pre-
specified adverse events that use the confirmatory approach. In addition, the exploratory
approach may provide data on possible associations between an intervention and a list of
observed adverse events, which can be used to generate new signals to add to existing
safety profiles.

A limitation of the exploratory approach is that the specific adverse effects reported may
have been selectively analysed and reported because of the nature of the findings (e.g.
based on statistical significance rather than clinical importance). Also, post-hoc or arbitrary
analytic decisions regarding data extraction and analysis are often required when review
authors encounter long lists of adverse events. Processes for selection and synthesis of such
data need consideration in the review protocol, even if the outcomes of interest are not fully
specified.

19.2.1.3 Hybrid approach

The hybrid approach combines elements of both confirmatory and exploratory approaches
to capture anticipated and previously unrecognized adverse effects of an intervention.
Reviews based on this approach might list a small number of adverse outcomes of interest
in the protocol, whilst allowing post-hoc exploratory analyses to capture adverse events
data available from the studies identified. An example is provided in Box 19.2.b.

Regardless of the approach adopted, review authors should be mindful of the potential for
problems related to definition and ascertainment of adverse events when reviews are based
solely on published data.
 Copyright © 2022 The Cochrane Collaboration

Box 19.2.b Illustration of three approaches to reviewing the adverse effects of a particular
intervention: acupuncture

Confirmatory approach: Review authors aim to synthesize data on the pre-specified

adverse events of skin infection and pain on needle insertion.

Exploratory approach: Review authors aim to synthesize data on all or any adverse
effects that are mentioned in the included studies.

Hybrid approach: Review authors aim to synthesize data on pre-specified outcomes of

skin infection and total number of withdrawals due to adverse events, along with any
other adverse effects found in the included studies.

19.2.2 Strategies for assessing beneficial and adverse effects in the same
review
When conducting a review of both beneficial and adverse effects of interventions, review
authors may:

1. use the same eligibility criteria to assess intended (beneficial) and unintended
(adverse) effects, in terms of types of studies, types of participant and types of
interventions; or
2. use different eligibility criteria for selecting studies that address unintended (adverse)
effects compared with studies that address intended (beneficial) effects.
Using the same eligibility criteria to gather data on both types of outcome makes the review
easier to conduct, not least because a single search can usually be undertaken if outcome
terms are not stipulated in the search string. It also may allow for a direct comparison
between beneficial and adverse effects, because the data are derived from the same types
of studies (although it will not necessarily be the case that exactly the same studies report
data on both beneficial and adverse effects). Two disadvantages of using the same eligibility
criteria are (i) that the types of studies that are most appropriate to address the beneficial
effects – typically randomized trials – may not be large enough or long enough to capture
important adverse effects; and (ii) that it may lead to omission of relevant data on adverse
effects if the adverse effects are also observed when the intervention is given for other
conditions (see also Box 19.2.a).

Thus, review authors may apply different eligibility criteria when attempting to identify
adverse effects data. The two main aspects of eligibility that may differ are the types of study
design and the types of participants. It is also possible that studies performed for a different
purpose may be eligible for the adverse effects component of the review.

• Different study designs: To address adverse effects it may be necessary to seek non-
randomized studies, because the effects are unlikely to be seen in randomized trials due
to their size, duration or restricted eligibility for participants: see Section 19.2.3.
 Copyright © 2022 The Cochrane Collaboration

• Different types of participants: Adverse effects data might be obtained from

randomized trials evaluating the same or similar intervention but conducted in different
populations or diseases (see also Box 19.2.a).
• Different purposes: There may be randomized trials with adverse effects data on
participants of interest to the review, but which did not measure the beneficial
outcomes relevant to the review (e.g. a pharmacokinetic study assessing drug
concentrations in patients with the disease).
When different eligibility criteria are used to address beneficial and adverse effects, it will
often be necessary to conduct a separate search for the two (or more) sets of studies (see
Section 19.3), and it may be necessary to plan different methods in other aspects such as
assessing risk of bias (see Section 19.4).

19.2.3 Selecting types of study design

Cochrane Reviews typically include randomized trials because randomization should
distribute both known and unknown confounding variables equally across intervention
groups (see Chapter 3, Section 3.3.1). However, the duration of follow-up in a randomized
trial may not be sufficient to capture long-term adverse effects, and criteria for selecting
participants into randomized trials may exclude participants at increased risk of harm (such
as people with comorbidities or older adults living with frailty). Also, randomized crossover
trials (see Chapter 22, Section 23.2) may not be appropriate for investigating some adverse
effects, particularly if exposure to an intervention in one period results in an adverse event
occurring in a later period. Non-randomized studies of interventions such as cohort studies
(assembled from disease or drug/device registries) and case-control studies may be more
likely than randomized trials to provide data on some types of adverse effects. However,
non-randomized studies tend to be at greater risk of bias (see Chapter 24).

Spontaneous case reports or case series may assist in signalling rare and previously
unknown events. However, for most Cochrane Reviews, these data sources should be used
for scoping purposes only (particularly as they do not have denominator data to allow
estimation of risks or rates). These spontaneous reports may guide drafting of the protocol
when there is a need to choose relevant or important adverse effects as outcomes of
interest.

19.2.4 Selecting adverse effects of interest

Review authors may define outcomes of interest based on severity, timing or the type of
adverse effects that could occur based on the known mode of action of the intervention.
Different sources may be used to inform pre-specification of adverse effects of interest.
These sources include clinicians’ observations in case reports, patients’ reports on internet
forums, scoping reviews, regulatory approved product information leaflets (e.g. from the US
Food and Drug Administration) or other sources (e.g. British National Formulary, Meyler’s
Side Effects of Drugs).

Composite adverse outcomes are often reported by trials. Common examples include ‘total
number of participants with adverse events’, or ‘numbers of withdrawals due to adverse
events’. Review authors should recognize major difficulties in interpreting composite
adverse outcomes that are potentially constructed from hundreds of diverse events,
 Copyright © 2022 The Cochrane Collaboration

because an important signal of rare serious adverse events could be masked by common,
trivial adverse events. Also, review authors should hesitate to interpret data on withdrawals
as surrogate markers for safety or tolerability, for the following reasons.

• The attribution of reason(s) for discontinuation is complex and may be due to mild but
irritating side effects, toxicity, lack of efficacy, non-medical reasons, or a combination of
causes.
• The pressures on patients and investigators under trial conditions to reduce the number
of withdrawals and dropouts can result in rates that do not reflect the experience of
adverse events within the wider population.
• Unblinding of intervention assignment often precedes the decision to withdraw. This
can lead to an over-estimate of the intervention’s effect on patient withdrawal. For
example, symptoms of patients in the placebo arm are less likely to lead to
discontinuation. Conversely, patients in the active intervention group who complained
of symptoms suggesting adverse effects may have been more readily withdrawn.

19.3 Identification of evidence

19.3.1 Search methods for adverse effects data
When considering the search process, review authors may decide to perform a single search
to retrieve studies evaluating both benefits and harms. If so, the search strategy should be
designed to take account of the selected approach, either confirmatory, exploratory or
hybrid, and any differences in eligibility criteria for addressing beneficial and adverse
effects. A single search may be reasonable if it is sufficiently broad (e.g. if it captures all
studies containing a specific drug name or intervention) without being limited to specific
study designs or types of participants.

In general, we recommend consideration of a separate bespoke search for data on adverse

effects, particularly if the study designs that evaluate adverse effects of interest are different
from those that report efficacy. It is unlikely that a single search that is focused on efficacy
or effectiveness studies will be sufficient to identify evidence on all adverse effects in a
comprehensive manner.

19.3.2 Allocating resources for the search

Despite significant improvements in reporting of adverse effects in primary studies, specific
terms relating to adverse effects may not feature in the title, abstract, keywords or
bibliographic database indexing systems. To determine the necessary work and resources
involved, careful scoping when drafting the review question is recommended. This may
need to account for the inclusion of unpublished data (see Section 19.3.4 and Chapter 4)
and non-randomized studies (see Chapter 24).

19.3.3 Sources to search

Due to the variable content and indexing techniques of healthcare databases, it is important
not to restrict adverse effect review searches to a single source, nor to a limited combination
 Copyright © 2022 The Cochrane Collaboration

of the primary clinical research databases. Performing a search in MEDLINE alone is not
recommended.

A case study reviewing adverse effects of thiazolidinedione use in patients diagnosed with
type 2 diabetes mellitus tested over 60 sources (Golder and Loke 2012).The results indicated
that the minimum combination of sources required to identify all relevant references
included 11 sources: the pharmaceutical company website, Science Citation Index, Embase,
BIOSIS Previews, British Library Direct, Medscape DrugInfo, American Hospital Formulary
Service (AHFS First), Thomson Reuters Integrity, Conference Papers Index, hand searching
and reference checking. In this specific example, just searching MEDLINE failed to retrieve
66% of relevant references. A search strategy conducted in MEDLINE, Embase and the
Cochrane Central Register of Controlled Trials (CENTRAL) failed to retrieve 57% of relevant
references. This example illustrates the breadth of sources needed to ensure identification
of relevant data. Authors will need to consider sources most relevant to their clinical
question; the list above is an illustration only.

Identifying adverse effects of pharmacological interventions often requires search methods

that are different from those required for reviews of non-pharmacological interventions, or
medical devices. Further guidance for sourcing adverse effects data is given in the online
Technical Supplement to Chapter 4.

19.3.4 Including unpublished sources

Review authors should search for unpublished sources of data on adverse effects. We
consider unpublished sources to be those outside of a peer-reviewed journal. This includes:
clinical study reports (CSR), trials registers and regulatory agency websites. Tang and
colleagues showed the value of searching ClinicalTrials.gov for data on serious adverse
events (Tang et al 2015). Among 300 trials with serious adverse events mentioned in
ClinicalTrials.gov, 78 (26%) did not have a corresponding publication, and for the remaining
202 trials, 26 (13%) published articles did not mention serious adverse events. Limiting
search strategies to published reports may therefore not produce a balanced review,
leading to underestimates of harm.

Mandatory changes applied to trials regulated by the Food and Drug Administration (FDA)
regarding the submission of adverse events data to ClinicalTrials.gov, and the legislated
publication of clinical data by the European Medicines Agency (EMA), means that previous
accessibility limitations are steadily improving. Although accessibility is likely to continue
to improve, the logistics and feasibility of routinely using such data sources for adverse
effects reviews has yet to be established. Review authors should therefore report on the
number of unpublished studies identified and instances where data on adverse effects were
inaccessible.

19.3.5 Search methods: specific and generic outcome terms

Searching for specific adverse effects outcomes is similar to searching for specific benefit
outcomes, so that search terms for the particular adverse effects outcome(s) are included
in the search string. Examples of specific adverse effects terms are: ‘headache’, ‘blood loss’
or ‘dysphagia’. However, it is likely that this method will lack sensitivity due to variation in
reporting and indexing.
 Copyright © 2022 The Cochrane Collaboration

A possible option for the larger databases is to use a broad search involving two
components at the same time: generic index terms combined with specific free-text
searches using the ‘OR’ Boolean function. Both specific and generic search techniques have
strengths and limitations, but the strengths are increased and limitations reduced when
they are combined. It is therefore advisable to combine index terms and free-text searching
(where possible) to increase search sensitivity and reduce the possibility of missing relevant
material. More details are provided in the online Technical Supplement to Chapter 4.

19.4 Appraisal of evidence

19.4.1 Challenges in assessing risk of bias for adverse effects data
Assessing risk of bias for pre-specified adverse effects that are actively monitored in
included studies is generally the same as for the pre-specified beneficial effects. However,
adverse effects are seldom specified as primary outcomes, and often are not pre-specified
at all, so there is often lack of clarity in the methods used to obtain adverse effects data.
Thus, different susceptibilities to bias can arise for adverse effects due to the way in which
they are measured, recorded and reported. It is important that the outcome measure is
appropriate for detection of the adverse effect, and that the outcomes are measured or
ascertained using a method that is comparable across intervention groups (see Chapter 8,
Section 8.6). Study participants prematurely stopping assigned intervention or withdrawing
from the study (due to adverse events) can result in dissimilar observation times for
ascertaining future adverse events. When assessing the risk of bias for missing outcome
data, it is important to consider the possibility of differential follow-up and informative
censoring. A particular challenge when assessing risk of bias for adverse effects data is that
of selective reporting. Results based on spontaneously reported adverse outcomes may
lead to concerns that these were selected post hoc based on the finding being noteworthy.
Similarly, unusual composite outcomes may be reported to hide or emphasize particular
findings.

19.4.2 Recommended tools for assessing risk of bias in adverse effects data
Review authors should use the currently recommended risk-of-bias tools, the RoB 2 tool for
randomized trials (see Chapter 8), and the ROBINS-I tool for non-randomized studies (see
Chapter 25). Although these tools are most easily directed at outcomes that have been pre-
specified by the review team, they are suitable for any type of quantitative outcome
analysed in a review. Where adverse effects are extracted post hoc from included trials in an
exploratory approach, it may not be possible to list important co-interventions or
confounding variables in the review protocol, as would usually be expected for using the
ROBINS-I tool.

Particular issues in assessing risk of bias for adverse effects data include outcome definition
and methods of monitoring adverse effects. These warrant special attention when there are
significant concerns over bias towards the null stemming from poor definition,
ascertainment or reporting of harms. This is particularly important for new or unexpected
adverse events that have not been pre-specified as outcomes of interest in the trials, and
where monitoring and reporting may be potentially inadequate. Additional resources such
as the McHarm tool (Chou et al 2010) and the Agency for Healthcare Research and Quality
 Copyright © 2022 The Cochrane Collaboration

(AHRQ) assessment tool (Chou et al 2007, Viswanathan and Berkman 2012) provide further
discussion of these issues.

19.4.3 Selective outcome reporting bias of adverse effects data

Selective outcome reporting refers to authors reporting a subset of variables, based on the
results, from among all the outcomes originally analysed (see Chapter 7). Selective outcome
reporting distorts the body of available evidence on which to conduct data synthesis and
can lead to a high risk of bias (Kicinski et al 2015). Missing or partially reported adverse
effects data are common in systematic reviews evaluating adverse effects (Saini et al 2014).

There is evidence that Cochrane Reviews may suffer from reporting bias. Kicinski and
colleagues explored the potential impact of reporting bias on meta-analyses in Cochrane
Reviews published between 1990 and 2005 (Kicinski et al 2015). They applied hypothesized
mechanisms of reporting bias to 802 meta-analyses of efficacy and 304 meta-analyses of
safety that each combined at least 10 individual estimates. The results from their model
indicated that statistically significant results favouring treatment were more likely to be
included in meta-analyses of efficacy than non-significant results. In contrast, results
showing no evidence of adverse effect had greater probability of inclusion in a meta-
analysis of safety than statistically significant results of adverse effects. Reporting bias
therefore, may lead to the erroneous conclusion that an intervention is safe or relatively free
from adverse effects.

19.5 Synthesis and interpretation of evidence

19.5.1 Estimating intervention effects from adverse effects
Review authors can have greater confidence in their interpretation of adverse effects data
when outcomes are defined, monitored and reported as pre-specified outcomes in the
research studies. In contrast, where the adverse effects are unexpected or ascertained ad
hoc through spontaneous reporting, review authors will have to make more cautious
interpretations regarding perceived safety or lack of harm, unless there is evidence that
monitoring and reporting were sufficiently robust to have accurately captured any events
of concern (Loke and Mattishent 2015).

It is important to evaluate the consistency and similarity of case definitions and methods of
ascertainment for harms outcomes from the various included studies before comparing or
synthesizing adverse effects data across studies. An important source of potential
heterogeneity in effect estimates for adverse effects is variation in outcome definition and
measurement. Review authors should ask study authors to resolve any ambiguity by
providing additional data, or disaggregated data, which can be reanalysed more
consistently.

Important analytical challenges relating to imprecision of estimates and rare events are
covered in Chapter 10 (Section 10.4.4); see also Section 19.5.2 for particular challenges of
determining whether there were zero adverse events.

Grouping adverse effects together in a composite measure (e.g. total number of adverse
effects) can only give a broad impression, and may lead to genuine differences between the
 Copyright © 2022 The Cochrane Collaboration

interventions in individual adverse effects being obscured. Owing to differences in coding

and categorization of adverse effects between studies, review authors should avoid trying
to increase numbers of events available for analysis by constructing composite categories
that have not been reported in the primary studies. Conversely, review authors should be
alert to situations in which the coding of adverse effects splits data unnecessarily (e.g. pain
in leg, pain in arm), which may dilute the signal of a more global effect (e.g. all patients
affected by pain).

Review authors should include at least one adverse effect outcome in the ‘Summary of
findings’ table. If the review did not focus on detailed evaluation of any adverse effects, then
the review authors should make an explicit statement that harms were not assessed, rather
than say (or imply) the intervention appears to be safe.

19.5.2 Synthesizing and interpreting ‘zero events’

It can be difficult, or unwise, to determine that there were no adverse events of a specific
type. Although trial reports may provide tables detailing withdrawals (and reasons) or
serious adverse effects, they will not necessarily include all events of interest to the review
authors. New or unexpected adverse events may have been missed if ascertainment relied
solely on spontaneous reporting. Furthermore, trials may report statements such as “no
serious harms were found” without specifying their definition of serious harms, or that
“there was no evidence of significant adverse effects”, without giving the numbers of events
on which such a conclusion is based.

If a serious adverse event of interest, such as heart failure, was not explicitly mentioned in
the text or the serious adverse effects tables, the question then arises as to whether it is
reasonable to interpret this as zero heart failure events. We generally recommend against
extracting data as ‘zero’ unless it is clearly listed as such in the study report. Even where
heart failure is explicitly reported as ‘zero’, we suggest that review authors carefully check
the methods section of the included study for details on the rigour of monitoring for the
adverse outcome (e.g. specific active surveillance for heart failure, versus reliance only on
spontaneous reports that are prone to under-reporting). Ambiguity frequently crops up in
the extraction and interpretation of absence of harms, so review authors should record how
they reached a decision of ‘zero events’.

19.6 Chapter information

Authors: Guy Peryer, Su Golder, Daniela Junqueira, Sunita Vohra, Yoon Kong Loke; on behalf
of the Cochrane Adverse Effects Methods Group

Acknowledgements: We thank Julian Higgins, Jamie Kirkham and Barbara Jennings.

Funding: This work was supported by Cochrane Methods Innovation Fund.

19.7 References
Chou R, Fu R, Carson S, Saha S, Helfand M. Methodological shortcomings predicted lower
harm estimates in one of two sets of studies of clinical interventions. Journal of Clinical
 Copyright © 2022 The Cochrane Collaboration

Epidemiology 2007; 60: 18-28.

Chou R, Aronson N, Atkins D, Ismaila AS, Santaguida P, Smith DH, Whitlock E, Wilt TJ,
Moher D. AHRQ series paper 4: assessing harms when comparing medical interventions:
AHRQ and the effective health-care program. Journal of Clinical Epidemiology 2010; 63:
502-512.

Golder S, Loke YK. The contribution of different information sources for adverse effects
data. International Journal of Technology Assessment in Health Care 2012; 28: 133-137.

Golder S, Loke YK, Zorzela L. Some improvements are apparent in identifying adverse
effects in systematic reviews from 1994 to 2011. Journal of Clinical Epidemiology 2013; 66:
253-260.

Golder S, Loke YK, Wright K, Norman G. Reporting of adverse events in published and
unpublished studies of health care interventions: a systematic review. PLoS Medicine 2016;
13: e1002127.

Kicinski M, Springate DA, Kontopantelis E. Publication bias in meta-analyses from the

Cochrane Database of Systematic Reviews. Statistics in Medicine 2015; 34: 2781-2793.

Loke YK, Mattishent K. If nothing happens, is everything all right? Distinguishing genuine
reassurance from a false sense of security. CMAJ: Canadian Medical Association Journal
2015; 187: 15-16.

Saini P, Loke YK, Gamble C, Altman DG, Williamson PR, Kirkham JJ. Selective reporting bias
of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ 2014;
349: g6501.

Schroll JB, Penninga EI, Gøtzsche PC. Assessment of Adverse Events in Protocols, Clinical
Study Reports, and Published Papers of Trials of Orlistat: A Document Analysis. PLoS
Medicine 2016; 13: e1002101.

Smith PG, Morrow RH, Ross DA. Outcome measures and case definition in: Field Trials of
Health Interventions: A Toolbox. Smith PG, Morrow RH, Ross DA, editors. Oxford (UK):
Oxford University Press; 2015.

Tang E, Ravaud P, Riveros C, Perrodeau E, Dechartres A. Comparison of serious adverse

events posted at ClinicalTrials.gov and published in corresponding journal articles. BMC
Medicine 2015; 13: 189.

Viswanathan M, Berkman ND. Development of the RTI item bank on risk of bias and
precision of observational studies. Journal of Clinical Epidemiology 2012; 65: 163-178.

Zorzela L, Loke YK, Ioannidis JP, Golder S, Santaguida P, Altman DG, Moher D, Vohra S,
PRISMA Harms Group. PRISMA harms checklist: improving harms reporting in systematic
reviews. BMJ 2016; 352: i157.