Environment International 157 (2021) 106868

Contents lists available at ScienceDirect

Environment International
journal homepage: www.elsevier.com/locate/envint

An approach to quantifying the potential importance of residual

confounding in systematic reviews of observational studies: A GRADE
concept paper
Jos H. Verbeek a, *, Paul Whaley b, Rebecca L. Morgan c, Kyla W. Taylor d, Andrew A. Rooney d,
Lukas Schwingshackl e, Jan L. Hoving f, S. Vittal Katikireddi g, Beverley Shea h,
Reem A. Mustafa i, M. Hassan Murad j, Holger J. Schünemann k, GRADE Working Group
a
Department of Public and Occupational Health, Amsterdam Public Health Research Institute, Amsterdam UMC, University of Amsterdam, Meibergdreef 9, 1105 AZ
Amsterdam, the Netherlands
b
Lancaster Environment Centre, Lancaster University, UK
c
McMaster University, Hamilton, Canada
d
National Institute of Environment Health Science, USA
e
Medical Center - University of Freiburg; Faculty of Medicine, University of Freiburg, Freiburg, Germany
f
Department of Public and Occupational Health, Amsterdam Public Health Research Institute, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
g
University of Glasgow, UK
h
University of Ottawa, Canada
i
University of Kansas Medical Center, US
j
Mayo Clinic, Rochester, US
k
McMaster University, Hamilton, Canada

A R T I C L E I N F O A B S T R A C T

Handling Editor: Adrian Covaci Small relative effect sizes are common in observational studies of exposure in environmental and public health.
However, such effects can still have considerable policy importance when the baseline rate of the health outcome
Keywords: is high, and many persons are exposed. Assessing the certainty of the evidence based on these effect sizes is
Body of evidence challenging because they can be prone to residual confounding due to the non-randomized nature of the evi­
Sensitivity analysis
dence. When applying GRADE, a precise relative risk >2.0 increases the certainty in an existing effect because
E-value
residual confounding is unlikely to explain the association. GRADE also suggests rating up when opposing
Certainty of evidence
Observational studies plausible residual confounding exists for other effect sizes. In this concept paper, we propose using the E-value,
defined as the smallest effect size of a confounder that still can reduce an observed RR to the null value, and a
reference confounder to assess the likelihood of residual confounding. We propose a 4-step approach. 1. Assess
the association of interest for relevant exposure levels. 2. Calculate the E-value for this observed association. 3.
Choose a reference confounder with sufficient strength and information and assess its effect on the observed
association using the E-value. 4. Assess how likely it is that residual confounding will still bias the observed RR.
We present three case studies and discuss the feasibility of the approach.

1. Introduction
When studying the effects of environmental exposure on adverse
health outcomes, randomisation of humans is not normally feasible for
ethical reasons (e.g., persons cannot purposely be assigned to receive a
potentially harmful exposure). Thus, observational studies will almost
always constitute the body of evidence to evaluate harm from exposures
in human research. Even after adjusting for all possible known con­
founders, due to non-randomised design, there will still be a risk that
unknown confounders could have biased the results in these studies
(Rosenbaum, 2017). This is called “residual confounding” (Glymour and
Greenland, 2008; Greenland and Lash, 2008; Liang et al., 2014).
Assessment of the impact of residual confounding on the certainty of the
evidence for effects of environmental exposures has been a challenge for

* Corresponding author.
E-mail address: jos@jverbeek.eu (J.H. Verbeek).

https://doi.org/10.1016/j.envint.2021.106868
Received 22 April 2021; Received in revised form 4 August 2021; Accepted 5 September 2021
Available online 13 September 2021
0160-4120/© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
J.H. Verbeek et al.
researchers. In the Grading of Recommendations Assessment, Develop­
ment and Evaluation (GRADE) Framework, concerns about residual
confounding lead to setting the initial rating of the certainty of the ev­
idence from observational studies as “low”. Then, so long as it has not
been rated down on any of the five downgrade domains, evidence from
observational studies can be rated up if one or more of three conditions
obtain: if the effect size is sufficiently large that it cannot be explained by
residual confounding; if there is a dose–response gradient; and when
there is plausible opposing bias in the evidence (Guyatt et al., 2011a;
Schunemann et al., 2019).
In this paper we are proposing a modification to the GRADE
approach for rating up due to large magnitude of effect. It has long been
recognised that bias due to residual confounding is much less likely to
explain the entire observed effect when the observed effect size is large,
provided that the body of evidence is otherwise of good methodological
quality (Cornfield et al., 1959; Hill, 1965). GRADE has incorporated this
Bradford Hill tenet for causation in its framework (Schunemann et al.,
2011). The current GRADE approach to defining such a “large magni­
tude of effect” takes a precise relative risk (RR) of > 2 as a cut-off value.
Based on modelling studies conducted over 50 years ago, this magnitude
of effect is considered to be large enough that it is unlikely to be
explained by residual confounding (Bross, 1967). When the results of a
systematic review based on observational evidence exceeds this
threshold, and when the effect is precise and the observational studies
do not exhibit other important limitations in study design or execution,
the certainty of evidence may be rated up by one level (Guyatt et al.,
2011b). A RR of > 5 represents a very large magnitude of effect which
may lead to rating up the certainty of the evidence by two levels.
Even though there are arguments as to why RRs of 2 and 5 are
reasonable cut-offs, these cut-offs are based on generalisations across a
wide range of exposures and observational studies and it has been sug­
gested that more tailored approaches are warranted (Shimonovich et al.,
2021). Also Bradford Hill suggested not to dismiss smaller effect sizes
too easily (Hill, 1965). The extent to which these cut-offs apply to the
specific set of studies included in any given systematic review can
therefore be questioned and while their general use is helpful it remains
arbitrary.
In exposure studies in public, environmental and occupational
health, effect sizes are often relatively small such as the effects of air
pollution with a few exceptions such as the effects of smoking. Assessing
certainty in small relative effects can be of high relevance when a large
proportion of the population is exposed to environmental compounds
with adverse health effects such as air pollution. In these cases, small
relative effect sizes can still lead to an absolute effect with a considerable
number of persons being affected (Bellinger, 2007; Kunzli et al., 2000;
Rose, 1985; Verbeek et al., 2021). In this context, judgements of the
certainty of the evidence will be made against a very low threshold of,
for example, 0.1% risk difference for mortality (Hultcrantz et al., 2017).
Given the societal importance of these small effect sizes from the pop­
ulation perspective, it is even more important to judge the certainty of
the evidence in the best possible way. While societal importance is
obviously not an argument to increase the certainty of the evidence, the
implications of even a small effect size at the population level is an
important argument to further scrutinize if our methods assess the cer­
tainty correctly.
GRADE’s approach to large effect sizes has its origin in a modelling
study from the 1960′ s (Bross, 1966; 1967). Although the findings were
not rebutted in the literature, recently, a similar new method for
quantifying unmeasured confounding has been proposed based on a
derivation of the observed effect size (VanderWeele and Ding, 2017).
The authors quantify how large the effect of a confounder would have to
be to reduce an observed RR to the null value, thereby explaining away
the apparent effect of the exposure of concern. They coined this number
the “E-value”.
The concept of the E-value is someway similar to the large magnitude
of effect: the larger the E-value is, the less likely it is that the results of a
2
Environment International 157 (2021) 106868
study will be explained by unmeasured or residual confounding. (Fig. 1)
The benefit of the E-value is that it is not an arbitrary cut-off but that it is
determined relative to the observed effect size in the study of interest. As
an alternative, the authors propose determining the ‘bias factor’ which is
the amount by which a confounder would reduce the observed effect size
given known relations between confounder and outcome and exposure
of interest. This will also show if the confounder can reduce the observed
effect size to the null value. The same authors have also proposed using
the E-value to assess unmeasured or residual confounding in meta-
analyses in systematic reviews (Mathur and VanderWeele, 2019).
In this GRADE concept paper, we explore the application of the E-
value concept in place of RR cut-offs, when judging whether to upgrade
certainty of the evidence due to a large magnitude of effect. We further
describe how the E-value approach can be applied to systematic reviews
and use three case-studies as a proof-of-concept. The focus of this
exploration is small effect sizes in studies of exposures in environmental
and public health. The approach is applied using the relative effect
measures, although the impetus for evaluating small relative effect sizes
is driven by the importance of their large absolute impact on a popu­
lation level. Application of the proposed approach in clinical topics or
when effect sizes are not small are beyond the scope of this manuscript
and requires further evaluation.
2. Definition of a confounder
A confounder is an exposure or variable that is associated with both
the exposure of interest and the outcome of interest or one of their
antecedent variables (see Fig. 2) which accounts for at least part of the
observed effect size (Hernan et al., 2002). Randomisation is the best way
to prevent bias due to confounding because it reduces the chance of
baseline differences in unknown confounders that could cause spurious
associations. When randomization may not be possible, statistical ad­
justments can be incorporated within observational studies to account
for known confounders. We define observational studies as studies in
which the investigator observes individuals without manipulation or
intentional intervention. Observational studies are necessarily non-
randomised. Intervention studies may be randomised or non-
randomised.
3. Unmeasured and residual confounding
There are three situations in a study or a systematic review that
confounding can play a role in: a known confounder has not been
RR for exposure, outcome and two confounders
and E-value
RR obs 1.7
E-value 2.8
RRs Confounder A-Outcome/Exp 3.5
RR obs Confounder A adjusted 1
RRs Confounder B-Outcome/Exp 1.4
RR obs Confounder B adjusted 1.6
0 0.5 1 1.5 2 2.5 3 3.5 4
Fig. 1. The E-value ( ) for a RRobs ( ) of 1.7 is 2.8. Confounder A associations
with both exposure and outcome ( ) of RR = 3.5 are larger than the E-value
and reduce observed RR to the null value RR adj for A = 1 ( ) but Confounder B
associations with both exposure and outcome ( ) of RR = 1.4 are not larger
than the E-value and don’t reduce the observed RR to the null value but to RRadj
for B = 1.6 ( ).
J.H. Verbeek et al. Environment International 157 (2021) 106868

one should first determine what would be a true causal effect of a

scientifically meaningful size and how large the part of the distribution
of effect sizes underpinning this meaningful effect size should be among
the included studies. The meaningful effect size is used in a similar way
to the threshold against which the GRADE approach judges the certainty
of the evidence. Then it should be assessed how large unmeasured
confounding would need to be to explain away the observed results. For
studies of the effects of public health, environmental and occupational
exposure, the threshold or the ‘meaningful relative effect size’ would be
very small because of the possible population impact in absolute terms
and may often be close to the null value.
Fig. 2. Relationship of a confounder C to exposure E and outcome (O). Because The E-value approach does not require making a priori assumptions
the confounder is associated with both the exposure and the outcome, its and the calculation is relatively straightforward, as compared to other
presence in the system will distort the effect estimate of the exposure on the methods used for sensitivity analyses. The drawback of the E-value
outcome. The presence of undiagnosed confounders therefore introduces bias approach is that even though no a priori assumptions are needed, the
into studies. Residual confounding is of concern in observational studies interpretation is not straight forward. Although the E-value informs
because, while many major confounders may be identified, there is potential for about how big an unmeasured confounder would need to be to explain
unidentified confounders to be influencing the study results.
the observed effect size away, one must still make assumptions of how
likely it is that residual confounding of that magnitude is occurring. It
measured in the included studies; a known confounder has been has been shown that this aspect of interpretation of the E-value is highly
measured but the control for this confounder is not perfect; and there are variable (Blum et al., 2020).
potential unknown confounders that cannot be controlled for, i.e. re­ We propose a four-step approach for use of the E-value in a sensi­
sidual confounding. The first two situations are dealt with in the risk of tivity analysis of residual confounding in a systematic review of obser­
bias assessment step of a systematic review. The third situation is the vational studies (Fig. 3). We propose using the E-value in the GRADE
focus of this article. assessment of the certainty of the pooled results to assess the sensitivity
In developing our proposed approach, we first reviewed the relevant to unmeasured residual plausible confounding such as currently done
literature on unmeasured and residual confounding. The basic ideas for the domains of large magnitude of effect and residual plausible
around how to assess the effect of unmeasured confounding in a study opposing confounding. The assumption is that this procedure is only of
have remained more-or-less unchanged since Cornfield first stated in interest if there are no limitations in the design and execution of
1959 that “the excess lung cancer risk among smokers is so great that the observational studies that are included in the systematic review at hand.
results cannot be interpreted as arising from an indirect association of The included studies should have adjusted for the effects of all known
cigarette smoking with some other agent or characteristic” (Cornfield critical confounders. If the studies included in the reviews have concerns
et al., 1959; Greenland, 1996). Axelson and Steenland termed this “in­ about risk of bias or have not adjusted for all known critical confounders
direct adjustment” for the effects of smoking (Axelson and Steenland,
1988; Lubin et al., 2018). Greenland and Lash et al., provided extensive
methods for what they call ‘quantitative bias analysis’ including that
resulting from residual confounding (Greenland and Lash, 2008; Lash
et al., 2014). Greenland especially advocates this type of analysis for
systematic reviews. Schneeweis et al., and Udin et al., describe similar
methods which they call ‘sensitivity analyses’ (Schneeweiss, 2006;
Uddin et al., 2016). They use this to assess how the results of a study or a
systematic review change in response to the effects of assumed residual
or unmeasured confounding.
The approach for conducting the sensitivity analysis is to make as­
sumptions about the prevalence of the confounder among the exposed
and unexposed and the strength of the association between the
confounder and the outcome of interest (Greenland and Lash, 2008).
Simple calculations then enable to assess how the observed effect size
changes after adjustment for the unmeasured confounder. This is called
‘external adjustment’ because the data are not derived from the study as
in conventional adjustment for confounding. Several mathematical ap­
proaches to implementing the assumptions have been developed (Cabral
and Luiz, 2007; Groenwold et al., 2010; Gustafson et al., 2010). Many
researchers have expressed concerns that even though it is valuable to
conduct a sensitivity analysis and quantify the effect of unmeasured
confounding in observational epidemiologic research, this is seldom
done (Arah et al., 2008; Greenland, 1996; Groenwold et al., 2010;
Steenland and Greenland, 2004).

4. The use of the E-value in systematic reviews of observational

studies

When conducting systematic reviews of observational studies,

sensitivity analysis is the only feasible approach to handling residual
confounding. Mathur et al., propose using the E-value in a random ef­
fects meta-analysis (Mathur and VanderWeele, 2019). In their approach,
Fig. 3. Flow diagram of analysis to assess sensitivity to residual confounding.
The numbers refer to the four steps of the process in the text. RRobs = observed
Relative Risk; RRCO = RR of confounder outcome association; RRCE = RR of
confounder exposure association.

3
J.H. Verbeek et al.
this approach should not be applied.
Step 1: Identify the association measure of interest based on relevant
exposure levels.
In many fields, the risk of adverse effects of exposure is assessed
relative to an arbitrary unit of exposure that is determined by conven­
tion rather than biological relevance. For example, in studies of health
risks posed by exposure to particulate air pollution, it is the habit to
assess increase in risk of an adverse health outcome per 10 µg/m3 in­
crease in exposure to particulate matter. We propose considering first if
it is helpful to transform such an incremental RR per unit of exposure to
a relevant exposure contrast by taking a known level of high exposure
versus a known level of low exposure (Morgan et al., 2018). In Europe or
the US, a realistic contrast between areas with high levels of particulate
pollution against low levels is approximately 40 µg/m3 (European
Environment Agency, 2019).
If the observed RR is an incremental RR per unit of exposure, we
suggest calculating the RR of the adverse health outcome for this rele­
vant exposure contrast based on the observed pooled RR per exposure
unit from the systematic review. This can be done by exponentiating the
incremental RR with the number of exposure levels that span the
exposure contrast. In the air pollution case, the exponent would be 4.
In a similar way, VanderWeele et al., advised taking the RR for a
larger contrast because it will be more likely that there will also be a
difference in confounder prevalence between these contrasts (Vander­
Weele et al., 2019b). In other epidemiologic studies, the comparison is
often between a high level of exposure versus a low level of exposure.
Then, one should find out what these high levels and low levels of
exposure refer to and what kind of exposure contrast can be inferred.
Step 2: Calculate the E-value for the association of interest.
The E-value is an expression of the observed RR and can be calcu­
lated with the following formula: E-value = (RRobs + sqrt(RRobs*(RRobs-
1)) (VanderWeele and Ding, 2017). For other effect-sizes such as odds
ratios (OR) or mean differences (MD) the authors also provide the
appropriate formulas in a supplement.
For an observed RR of 2, the E-value is 3.4 and for an observed RR of
5, the E-value is 9.5.
Step 3: Identify a reference confounder and assess its effect on the
observed RR.
A reference confounder is a known confounder for the comparison of
interest that has a known estimate of association with exposure and
outcome of interest, and preferably has the strongest association of all
known confounders. The reference confounder should have a strong
association with exposure and outcome of interest because we want to
use it as a threshold that we are certain exceeds that of any plausible
residual confounder. Instead of the cut-off of RR > 2 or RR > 5, we will
now use a known confounder as the reference for our judgement if re­
sidual confounding is likely to influence the observed effect size. If this
reference confounder can reduce the effect size to null or below a rele­
vant threshold, a potential unknown residual strong confounder could
too. And vice versa, if the reference confounder will not be able to
reduce the observed effect size to null, then it is unlikely that an un­
known residual confounder will be able to this.
We suggest assessing the effects of the reference confounder as
follows.
a. Determine the magnitude of the association of the reference
confounder with the exposure and with the outcome of interest,
preferably based on a systematic review of the literature (Lash et al.,
2014) or if this is not available on the best available primary studies.
For the association with the exposure, use the same relevant expo­
sure levels as determined under Step 1.
b. If the association of the confounder with exposure and outcome are
judged to be of similar magnitude, then the magnitude of this RR can
be directly compared to the E-value. If the RR for the associations is
smaller than the E-value, it is unlikely that the reference confounder
can reduce the observed RR to null and vice versa if the RR is larger
4
Environment International 157 (2021) 106868
than it is likely that a residual confounder could reduce the observed
RR to null as well.
c. If the association of the confounder with outcome and exposure
differs in magnitude, calculate the bias-factor. The bias factor is the
amount of the observed RR that is due to the effect of the confounder.
It can be calculated with the known associations of the reference
confounder with exposure and outcome of interest with the formula
BF = (RRCE*RRCO)/(RRCE + RRCO-1). It is easy to make this calcu­
lation in a spreadsheet or otherwise the website www.evalue-
calculator.com can be used (Mathur et al., 2018). Then divide the
observed RR by the bias factor to see if the reference confounder
reduces the observed RR to the null value or below a relevant
threshold.
Step 4: Make a judgement about rating up.
With the information about the effect of the reference confounder on
the observed effect size, we can make a judgement how likely it is that
residual confounding can reduce the observed effect to null or below a
threshold of interest. This judgement can then be used to decide about
upgrading the certainty of the evidence.
Determining whether certainty in the evidence should be upgraded
based on an E-value calculation not only depends on the numerical
values under Steps 2 and 3, but also on other aspects of residual con­
founding and the systematic review. If a long list of confounders has
been measured and adjusted for in the studies included in the review,
there will be residual confounding only. If, in this situation, the calcu­
lation of a strong reference confounder yields a value smaller than the E-
value, it would be highly unlikely that unmeasured residual confound­
ing can bias the observed RR. This would then warrant upgrading of the
certainty of the evidence because it is unlikely that residual confounding
has biased the results.
If on the other hand, the studies in the review have not been able to
adjust for more independent confounders that are not judged to be
critical, then the conclusion might still be that unmeasured residual
confounding is likely to have biased the observed RR. Simulation studies
show that multiple independent unmeasured confounders can lead to
larger bias (Fewell et al., 2007; Groenwold et al., 2016). In this situation,
it would not be warranted to upgrade the certainty of the evidence.
However, in many situations, confounders are related such as is the case
with lifestyle factors such as exercise and BMI. Then the effects of these
confounders would not be larger.
5. Case studies
5.1. Case study 1: Air pollution and overall mortality
Step 1. A systematic review of the effects of air pollution on mor­
tality reported a relative risk of mortality of RR = 1.06 per 10 µg per
cubic meter increase of fine particulate matter in air (PM2.5) (Hoek et al.,
2013). However, in Europe or the US, a realistic contrast between areas
with heavy and little PM2.5 pollution would be 40 µg/m3 (European
Environment Agency, 2019). If we take > 40 µg/m3 PM2.5 as high
exposure and < 10 µg/m3 PM2.5 as low exposure there is a contrast of
four levels of exposure. The RRobs across four levels of exposure = 1.06 4
= 1.26.
Step 2. For RRobs = 1.26 the E-value is 1.83 based on the formula
given above ((1.26 + sqrt(1.26*(1.26–1)) = 1.83)
Step 3. Smoking is one of the confounders that is correlated with
both mortality and air-pollution and for which information is available
Therefore, we used smoking as a reference confounder. Unfortunately,
there were no systematic reviews identified; therefore, the information
of a large cohort study was used. In this recent big Australian cohort
study, the RRCO of mortality resulting from smoking versus never
smoking adjusted for age, other life-style factors and area of residence,
was reported as 2.96 (95% CI 2.69 to 3.25) (Banks et al., 2015). Even
though in most studies there seems to be a relation between smoking and
J.H. Verbeek et al.
air-pollution, the direction of the relation is less clear. Some studies
report a positive association with more persons smoking in areas with
high pollution (Jerrett et al., 2004) and others report less persons
smoking in areas with high pollution (Villeneuve et al., 2011). This
could be due to the scale of the measurement, where the first studies
looked at city level data and others at country level data. We took the
study of Jerret et al., which reported the smoking prevalence in areas
with high and with low air pollution of 42.8% and 28.1%. This results in
a RRCE = 1.52. Because the values of the RRCE and RRCO diverge, we
calculated the bias factor as (1.52*2.96)/(1.52 + 2.96–1) = 1.29. We
divided the RR obs by the bias factor and the unconfounded true effect
size could therefore be RR unconfounded = 1.26/1.29 = 0.97.
Step 4. This means that a reference confounder as smoking could
reduce the observed RR of 1.26 to the null value. Unmeasured residual
confounding could have a similar effect as smoking. This led us to the
conclusion that residual confounding could still play a role in the
observed pooled effect size and we would not upgrade the certainty of
the evidence for the effect size found in the systematic review. Given
that smoking is considered a critical confounder in air pollution studies
and there is uncertainty about the associations with exposure and
outcome, a systematic review of the relations of smoking with air
pollution and mortality is warranted. If such a systematic review would
clearly show that smoking is not a critical confounder, our judgement of
the effect of residual confounding might also change.
5.2. Case study 2: Red meat and overall mortality
Step 1. Zeraatkar et al conducted a systematic review of red and
processed meat consumption and overall mortality (Zeraatkar et al.,
2019). The review identified 31 prospective cohort studies which were
included in their analyses, but the results were based on 8 studies that
were at low risk of bias. The authors conducted a meta-analysis of the
risk of overall mortality for a contrast of 3 servings red and processed
meat consumption per week. They found a pooled RR of 0.88 (95% CI
0.84 to 0.93; low certainty evidence) for persons consuming 3 servings
red and processed meat per week less than those in the comparison
group.
The average difference in combined red and processed meat con­
sumption between the highest and the lowest categories was 3 servings
per week. The authors estimated the weight of a serving of combined
processed and red meat at 100 g. Thus, the exposure contrast is 300 g/
week or 43 g per day. The average American diet contained 471 g per
week or 67 g per day of red processed and unprocessed meat in
2015–2016. The contrast measured in the systematic review would
mean that the average American consumption would be lowered by 43 g
from 67 g to 24 g/day. This decrease in consumption of 64% seems a
relevant contrast that may have an important effect on mortality. For
this contrast, we will look at the effect a reference confounder has.
The authors of the systematic review assessed that all studies were
definitely or probably sufficiently adjusted for confounders. In subgroup
analyses, they found a difference between studies with high risk of bias
and studies with low risk of bias, and therefore, the results are based on
the studies at low risk of bias only. They started the GRADE assessment
at low certainty because of the observational design and did not see a
reason for downgrading the evidence because the results were based on
studies at low risk of bias only.
Step 2. The E-value for a RR of 0.88 is 1.54 based on the formula for
the E-value where the RR smaller than one has been substituted by its
reciprocal: ((1.14 + sqrt(1.14*(1.14–1)) = 1.54).
Step 3. We took BMI as an informative reference confounder and
searched for associations of a BMI > 30 and red meat exposure and
mortality. Grosso et all evaluated the relation between BMI and red meat
in their systematic review of health risk factors associated with meat,
fruit and vegetable consumption in cohort studies (Grosso et al., 2017).
Based on the regression equation for BMI and red meat, the likelihood of
a BMI > 30 at the average red meat consumption of 67 g/day is 20.1%
5
Environment International 157 (2021) 106868
and at the low level of red meat consumption it is 15.7%. This leads to an
RRCE of BMI and red meat of 1.28 which is considerably smaller than the
E-value of 1.54.
Aune et al., conducted a systematic review of cohort studies of BMI
and mortality (Aune et al., 2016). They conducted subgroup analyses for
healthy never smokers because this subgroup will provide the least
biased results. They provided RRs per category of 2.5 BMI points
compared to the lowest category of BMI = 23. For BMI categories over
30 the risks increase quite fast with RRCO = 4 for a BMI = 45. Since these
very high categories are not prevalent, we considered the RRCO for a BMI
of 32.5 as most relevant. The RRCO of mortality for a BMI of 32.5
compared to a BMI of 23 is 1.39. This confounder-outcome relation is
also smaller than the E-value.
Step 4. Associations of an informative reference confounder with
both exposure and outcome are smaller than the E-value of 1.54. The
studies already adjusted for a large number of confounders that are
likely to be correlated. Only an unknown unmeasured independent
confounder stronger than BMI (RR = 1.39) could result in a reduction of
the observed RR to the null value. This leads to the conclusion that it is
unlikely that unmeasured residual confounding can easily reduce the
observed RR to a null effect and may justify rating up the certainty of the
evidence for this reason.
A further three case studies on the effects of changes in diets can be
found in Annex 1.
5.3. Case study 3: Organochlorine chemicals and endometriosis
Step 1. Cano-Sancho et al conducted a systematic review of studies
evaluating the relation between exposure to organochlorine chemicals
and endometriosis. They categorized exposures as dioxins, poly­
chlorinated biphenyls (PCBs) and organochlorine pesticides (OCPs). We
further evaluated the relationship between dioxins and endometriosis.
The authors found an OR of 1.65 (95% CI 1.14 to 2.39). Given that the
prevalence of endometriosis is between 5% and 15%, the OR is probably
an overestimation. Therefore, we transformed the OR to a RR assuming a
prevalence of 15% based on the formula RR = OR/(1 − p0+(p0 × OR))
in which p0 is the prevalence at baseline (Grant, 2014). This leads to a
RR of 1.50. The observed RR is for the ‘high versus low percentiles’ but
the authors do not specify what this means in terms of dioxin exposure or
measurements. Because there are no numerical values attached to the
high and low percentiles, it is impossible to know what the contrast is in
absolute terms. It is therefore not possible to assess what the levels of
exposure are that are measured in this contrast.
The review authors considered a long list of potential confounders
and assessed age and body mass index as critical confounders which
absence would lead to a high risk of bias. We categorized them as soci­
odemographic: employment status, income, health insurance status,
ethnicity; life-style related: smoking, caffeine consumption; adipose tissue-
related: adipose tissue, body mass index, weight loss, weight modifica­
tion; reproduction-related: familial history, age at menarche, parity,
gravidity, breastfeeding conditional on parity; menstruation-related:
tampon use, menstrual cycle; menstrual cycle dysregulation; menarche
status; menstrual regularity; and other exposures: oral contraceptives;
dioxin-like compounds; polybrominated biphenyls; polybrominated
diphenyl ethers; phthalates esters; brominated flame retardants.
Step 2. The E-value for a RR of 1.50 is 2.37 based on the formula
1.50 + sqrt(1.50*(1.50–1) = 2.37.
Step 3. To judge the likelihood of unmeasured residual confounding
based on an E-value of 2.37, we considered BMI as an informative
reference confounder. Kim reported that measures of organochlorine
chemicals are two to three times higher in obese as compared to lean
persons (Kim et al., 2011). A RRCE of 2.5 for the relation between BMI
and dioxin could therefore be a good estimate of the relation. However,
Backonja et al report that there may be a small preventive effect of being
obese on the occurrence of endometriosis (Backonja et al., 2016). This
would mean that BMI does not have a confounding effect or at best an
J.H. Verbeek et al.
unclear effect on the observed RR.
Step 4. Given the uncertainty about the exposure contrast for which
the RR was observed, the uncertainty about the effects of the reference
confounder and the large number of different independent confounders
we could not reach a conclusion about the likelihood of residual con­
founding. Consequently, we did not see a reason to upgrade the certainty
of evidence.
6. Discussion
We used the concept of the E-value instead of proposed thresholds of
large or very large magnitude of effect to make a judgement about the
potential impact of residual confounding in the conclusions of system­
atic reviews of public health and environmental and occupational health
studies. In all three case studies the effect sizes were smaller than 2 and
therefore none would be rated up for certainty of the evidence due to
large effect size under current GRADE guidance. However, we have
demonstrated that, based on the E-value, one of these bodies of evidence
may be rated up for certainty. This is because residual confounding is
unlikely to explain the observed association, even though the effect size
does not cross the current GRADE thresholds for large or very large ef­
fects. The use of the E-value was relatively easy and feasible in all three
cases. It is important to emphasize that rating up the certainty in evi­
dence requires that there be no other serious concerns that would lower
certainty, such as risk of bias in important domains such as exposure or
outcome assessment, inconsistency between study results, imprecision
in the final pooled result, or publication bias.
Even though the use of the E-value seems helpful in making judge­
ments about the likelihood of residual confounding, there are still issues
that require further testing. In the GRADE approach, a large effect size is
a reason to upgrade where we argue that it is not the large effect size in
itself but the reason should be that residual confounding is not likely to
affect the observed estimate of effect. Confounding and other bias issues
are currently dealt with in three different domains in GRADE. First, one
should assess if there are reasons to downgrade based on limitations in
studies such as risk of bias, then one should assess if unlikely residual
confounding is a reason to upgrade and then one should again assess if
the direction of all plausible confounding opposes the observed associ­
ation as a reason to upgrade. It could be argued that these three domains
should be addressed in the ‘risk of bias’ domain alone, but this goes
beyond the scope of this paper and should be further explored.
6.1. Strengths and limitations
The strength of this proposal is that it shows that the concept of the E-
value is feasible to use in systematic reviews of observational studies of
exposure in environmental and public health with small effect sizes. This
approach provides more transparency in assessing the effects of con­
founding. First, it takes the RR for a relevant contrast as the point of
departure and not an arbitrary incremental RR. It also increases trans­
parency by assessing the numerical values for the relation of the con­
founders with both exposure and outcomes. This is an important step
because in the GRADE large effect size approach the cut-off of RR = 2
assumes that a confounder would hardly ever have an association with
exposure or outcome>3.4. However, age or sex could easily reach these
magnitudes. Also, for the E-value approach, the judgement of a large E-
value assumes that these confounder associations will seldom be >2 for
all-cause mortality (VanderWeele et al., 2019a). We believe that the
comparison with a reference confounder makes the procedure more
transparent which is also advocated by VanderWeele and Mathur
(VanderWeele and Mathur, 2020). The approach does contain some
mathematics, but these are all simple and can be easily solved without
the help of sophisticated software. Even though a cut-off value is simpler
to use, the proposed E-value approach is based on more insight into the
effect of confounders.
There are also some limitations of the proposed approach. For
6
Environment International 157 (2021) 106868
example, we have not taken into account the confidence intervals sur­
rounding the observed effect size. However, as part of step 4, the E-value
can also be applied to the lower limit of the confidence interval and one
can assess if this value also excludes residual confounding. The judge­
ment could then be made with more certainty. We have not considered
the certainty of the evidence of the relations of the confounder with the
exposure and the outcome. It would also be possible to use several
reference confounders to see if there are differences. This should be
considered in step 4 before deciding to upgrade or not.
We have restricted the application of the E-value to the assessment of
residual confounding, but it can also be applied to missing known con­
founders as a part of the risk of bias assessment. If specific confounders
are missing in a study or the body of evidence, and information on the
relation with outcome and exposure is available, a similar sensitivity
analysis or external adjustment as for residual confounding can be made.
This should then be part of the risk of bias assessment and the decision
whether to rate down for this missing confounder or not.
The choice of an appropriate known reference confounder that is
informative remains arbitrary. Nevertheless, quantifying the relations of
the confounder with outcome and exposure of interest is very informa­
tive (Lash et al., 2014). As Fig. 1 shows, the impact of a confounder can
lead to a reduction of the observed RR to the null value or just a slight
decrease in the magnitude of the observed RR. These two situations
would lead to very different conclusions and can only be assessed
quantitatively. A limited number of confounders plays an important role
in many studies of environmental and occupational health for example
lifestyle-factors such as smoking, exercise or BMI. It would be helpful for
making a judgement about the magnitude of their confounding if the
relation with important outcomes such as mortality, diabetes, myocar­
dial infarction would be readily available and accessible in a database.
6.2. Implications for practice and research
Other authors assessed E-values in air pollution and nutrition studies
but did so at the aggregated level of 100 studies in a specific field of
science (Trinquart et al., 2019). For the air pollution studies the average
RR was 1.16 which has a corresponding E-value of 1.59. They concluded
that with this E-value air pollution could still be a field in which studies
are vulnerable to bias due to unmeasured residual confounding. We
contend that in the absence of analysis that specifies the confounders
and assesses the relation of the confounders to the exposure and
outcome of interest, the E-value adds little-value to the finding of an RR
of 1.16. The E-value only becomes informative in the light of the specific
context of a known measured confounder. Blum et al explored how
authors of studies used the E-value and found that in only 14 of 87 pa­
pers that had reported the E-value, authors concluded that their results
could be biased by unmeasured residual confounding and only 19 out of
87 related the E-value to specific known confounders (Blum et al.,
2020). Not surprisingly, they found that E-values of the same magnitude
could lead to divergent conclusions. Therefore, it is important that the E-
value be interpreted cautiously and related to known confounders. Even
then, the likelihood of residual confounding will be a judgement and not
a calculation. Lastly, there are inherent known limitations to the
different association measures that should be considered, such as
asymmetry and statistical properties of the relative risk and interpreta­
tion challenges of the odds ratio (Murad et al., 2009).
This approach can be useful also for other fields and other types of
studies especially where small effect sizes can have important implica­
tions from the population perspective. This should be further explored.
In conclusion, in this GRADE concept paper, we have explored the
application of the E-value to the results of systematic reviews in envi­
ronmental and public health topics with small effect sizes. We propose
that this approach can facilitate the judgement of the likelihood of un­
measured residual confounding. This approach appears to have value as
a transparent basis for upgrading the certainty of the evidence that relies
on available data. With current guidance based on cut-off values there is
J.H. Verbeek et al.
a chance that we are missing an opportunity to upgrade for evidence that
has a RR < 2. Still, it is not likely that we will upgrade inappropriately
based on the E-value approach. We recommend further testing and
exploration to evaluate if it will qualify for GRADE guidance.
CRediT authorship contribution statement
Jos H. Verbeek: Conceptualization, Methodology, Data curation,
Visualization, Writing – original draft. Paul Whaley: Conceptualization,
Methodology, Writing – review & editing, Visualization. Rebecca L.
Morgan: Conceptualization, Writing – review & editing. Kyla W. Tay­
lor: Conceptualization, Writing – review & editing. Andrew A. Rooney:
Conceptualization, Writing – review & editing. Lukas Schwingshackl:
Conceptualization, Data curation, Writing – review & editing. Jan L.
Hoving: Conceptualization, Writing – review & editing. S. Vittal
Katikireddi: Conceptualization, Writing – review & editing. Beverley
Shea: Conceptualization, Writing – review & editing. Reem A. Mustafa:
Conceptualization, Writing – review & editing. M. Hassan Murad:
Conceptualization, Methodology, Writing – review & editing. Holger J.
Schünemann: Conceptualization, Methodology, Writing – review &
editing, Supervision.
Declaration of Competing Interest
The authors declare the following financial interests/personal re­
lationships which may be considered as potential competing interests:
Jos H. Verbeek: paid by WHO for methodological advice on guideline
and systematic review development according to the WHO handbook for
guideline development.
Paul Whaley: declares personal fees in the last five years from
Elsevier Ltd, the Cancer Prevention and Education Society, the Evidence
Based Toxicology Collaboration, Yordas Group, and grants from Lan­
caster University, which are outside the submitted work but relate to the
development and promotion of systematic review and other evidence-
based methods in environmental health research, for delivering
training around these methods, and for providing editorial services. PW
is Systematic Reviews Editor for Environment International and is a
Guest Editor of the Special Issue to which this manuscript was submit­
ted. In accordance with conflict-of-interest policy at Environment In­
ternational, the Editors-in-Chief selected an alternative editor to handle
the submission, and PW did not communicate with the handling editor
about this submission during the editorial decision-making process.
Rebecca L. Morgan: none
Kyla W. Taylor: none
Andrew A. Rooney: none
Lukas Schwingshackl: none
Jan L. Hoving: none
S. Vittal Katikireddi: Acknowledges funding from the Medical
Research Council (MC_UU_00022/2) and Scottish Government Chief
Scientist Office (SCAF/15/02 and SPHSU17) unrelated to this article.
Beverley Shea: none
Reem A. Mustafa: none
M. Hassan Murad: none
Holger J. Schünemann: co-chair of the GRADE working group, no
direct financial conflicts of interest.
Appendix A. Supplementary material
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.envint.2021.106868.
References
Arah, O.A., Chiba, Y., Greenland, S., 2008. Bias formulas for external adjustment and
sensitivity analysis of unmeasured confounders. Ann. Epidemiol. 18 (8), 637–646.
Aune, D., Sen, A., Prasad, M., Norat, T., Janszky, I., Tonstad, S., Romundstad, P., Vatten,
L.J., 2016. BMI and all cause mortality: systematic review and non-linear dose-
7
Environment International 157 (2021) 106868
response meta-analysis of 230 cohort studies with 3.74 million deaths among 30.3
million participants. Bmj 353:i2156.
Axelson, O., Steenland, K., 1988. Indirect methods of assessing the effects of tobacco use
in occupational studies. Am. J. Ind. Med. 13 (1), 105–118.
Backonja, U., Buck Louis, G.M., Lauver, D.R., 2016. Overall adiposity, adipose tissue
distribution, and endometriosis: a systematic review. Nurs. Res. 65, 151–166.
Banks, E., Joshy, G., Weber, M.F., Liu, B., Grenfell, R., Egger, S., Paige, E., Lopez, A.D.,
Sitas, F., Beral, V., 2015. Tobacco smoking and all-cause mortality in a large
Australian cohort study: findings from a mature epidemic with current low smoking
prevalence. BMC Med. 13, 38.
Bellinger, D.C., 2007. Interpretation of small effect sizes in occupational and
environmental neurotoxicology: individual versus population risk. Neurotoxicology
28 (2), 245–251.
Blum, M.R., Tan, Y.J., Ioannidis, J.P.A., 2020. Use of E-values for addressing
confounding in observational studies-an empirical assessment of the literature. Int. J.
Epidemiol. 49:1482-1494.
Bross, I.D., Spurious effects from an extraneous variable. J. Chronic. Dis. 19:637-647.
Bross, I.D.J., 1967. Pertinency of an extraneous variable. J. Chronic. Dis. 20 (7),
487–495.
Cabral, M.D.B., Luiz, R.R., 2007. Sensitivity analysis for unmeasured confounders using
an electronic spreadsheet. Rev. Saude Publica 41 (3), 446–452.
Cornfield, J., Haenszel, W., Hammond, E.C., Lilienfeld, A.M., Shimkin, M.B., Wynder, E.
L., 1959. Smoking and lung cancer: recent evidence and a discussion of some
questions. J. Natl. Cancer Inst. 22, 173–203.
European Environment Agency. Country comparison of the PM2.5 concentrations in
2017. 2019; Accessed 22 January 2021;https://www.eea.europa.eu/data-and-maps/
figures/country-comparison-of-the-pm2.
Fewell, Z., Davey Smith, G., Sterne, J.A.C., 2007. The impact of residual and unmeasured
confounding in epidemiologic studies: a simulation study. Am. J. Epidemiol. 166 (6),
646–655.
Glymour, M.M., Greenland, S., 2008. Causal Diagrams. in: Rothman K., Greenland S.,
Lash T.l., eds. Modern Epridemiology. Philadelphia, US: Wolters Kluwer.
Grant, R.L., 2014. Converting an odds ratio to a range of plausible relative risks for better
communication of research findings. Bmj 348:f7450.
Greenland, S., 1996. Basic methods for sensitivity analysis of biases. Int. J. Epidemiol. 25
(6), 1107–1116.
Greenland, S., Lash, T.L., 2008. Bias Analysis. In: Rothman K., Greenland S., Lash T.l.,
eds. Modern Epridemiology. Philadelphia, US: Wolters Kluwer.
Groenwold, R.H., Nelson, D.B., Nichol, K.L., Hoes, A.W., Hak, E., 2010. Sensitivity
analyses to estimate the potential impact of unmeasured confounding in causal
research. Int. J. Epidemiol. 39:107-117.
Groenwold, R.H., Sterne, J.A., Lawlor, D.A., Moons, K.G., Hoes, A.W., Tilling, K., 2016.
Sensitivity analysis for the effects of multiple unmeasured confounders. Ann.
Epidemiol. 26:605-611.
Grosso, G., Micek, A., Godos, J., Pajak, A., Sciacca, S., Galvano, F., Boffetta, P., 2017.
Health risk factors associated with meat, fruit and vegetable consumption in cohort
studies: A comprehensive meta-analysis. PLoS One 12:e0183787.
Gustafson, P., McCandless, L.C., Levy, A.R., Richardson, S., 2010. Simplified Bayesian
sensitivity analysis for mismeasured and unobserved confounders. Biometrics 66,
1129–1137.
Guyatt, G.H., Oxman, A.D., Kunz, R., Atkins, D., Brozek, J., Vist, G., Alderson, P.,
Glasziou, P., Falck-Ytter, Y., Schünemann, H.J., 2011a. GRADE guidelines: 2.
Framing the question and deciding on important outcomes. J. Clin. Epidemiol. 64
(4), 395–400.
Guyatt, G.H., Oxman, A.D., Sultan, S., Glasziou, P., Akl, E.A., Alonso-Coello, P.,
Atkins, D., Kunz, R., Brozek, J., Montori, V., Jaeschke, R., Rind, D., Dahm, P.,
Meerpohl, J., Vist, G., Berliner, E., Norris, S., Falck-Ytter, Y., Murad, M.H.,
Schünemann, H.J., 2011b. 9. Rating up the quality of evidence. J. Clin. Epidemiol.
64 (12), 1311–1316.
Hernan, M.A., Hernandez-Diaz, S., Werler, M.M., Mitchell, A.A., 2002. Causal knowledge
as a prerequisite for confounding evaluation: an application to birth defects
epidemiology. Am. J. Epidemiol. 155, 176–184.
Hill, A.B., 1965. The environment and disease: association or causation? Proc. R. Soc.
Med. 58, 295–300.
Hoek, G., Krishnan, R.M., Beelen, R., Peters, A., Ostro, B., Brunekreef, B., Kaufman, J.D.,
2013. Long-term air pollution exposure and cardio- respiratory mortality: a review.
Environ. Health 12, 43.
Hultcrantz, M., Rind, D., Akl, E.A., Treweek, S., Mustafa, R.A., Iorio, A., Alper, B.S.,
Meerpohl, J.J., Murad, M.H., Ansari, M.T., Katikireddi, S.V., Östlund, P.,
Tranæus, S., Christensen, R., Gartlehner, G., Brozek, J., Izcovich, A.,
Schünemann, H., Guyatt, G., 2017. The GRADE Working Group clarifies the
construct of certainty of evidence. J. Clin. Epidemiol. 87, 4–13.
Jerrett, M., Burnett, R.T., Brook, J., Kanaroglou, P., Giovis, C., Finkelstein, N.,
Hutchison, B., 2004. Do socioeconomic characteristics modify the short term
association between air pollution and mortality? Evidence from a zonal time series in
Hamilton, Canada. J. Epidemiol. Community Health 58, 31–40.
Kim, M.-J., Marchand, P., Henegar, C., Antignac, J.-P., Alili, R., Poitou, C., Bouillot, J.-L.,
Basdevant, A., Le Bizec, B., Barouki, R., Clément, K., 2011. Fate and complex
pathogenic effects of dioxins and polychlorinated biphenyls in obese subjects before
and after drastic weight loss. Environ. Health Perspect. 119 (3), 377–383.
Kunzli, N., Ackermann-Liebrich, U., Brandli, O., Tschopp, J.M., Schindler, C.,
Clinically, L.P., 2000. “small” effects of air pollution on FVC have a large public
health impact. Swiss Study on Air Pollution and Lung Disease in Adults (SAPALDIA) -
team. Eur. Respir. J. 15, 131–136.
J.H. Verbeek et al.
Lash, T.L., Fox, M.P., MacLehose, R.F., Maldonado, G., McCandless, L.C., Greenland, S.,
2014. Good practices for quantitative bias analysis. Int. J. Epidemiol. 43 (6),
1969–1985.
Liang, W., Zhao, Y., Lee, A.H., 2014. An investigation of the significance of residual
confounding effect. Biomed. Res. Int. 2014, 1–6.
Lubin, J.H., Hauptmann, M., Blair, A., 2018. Indirect adjustment of relative risks of an
exposure with multiple categories for an unmeasured confounder. Ann. Epidemiol.
28 (11), 801–807.
Mathur, M.B., Ding, P., Riddell, C.A., VanderWeele, T.J., 2018. Web site and R package
for computing E-values. Epidemiology 29 (5), e45–e47.
Mathur, M.B., VanderWeele, T.J., 2019. Sensitivity analysis for unmeasured confounding
in meta-analyses. J. Am. Stat. Assoc. 1–20.
Morgan, R.L., Whaley, P., Thayer, K.A., Schünemann, H.J., 2018. Identifying the PECO:
A framework for formulating good questions to explore the association of
environmental and other exposures with health outcomes. Environ. Int. 121,
1027–1031.
Murad, M.H., Montori, V.M., Walter, S.D., Guyatt, G.H., 2009. Estimating risk difference
from relative association measures in meta-analysis can infrequently pose
interpretational challenges. J. Clin. Epidemiol. 62 (8), 865–867.
ROSE, GEOFFREY, 1985. Sick individuals and sick populations. Int. J. Epidemiol. 14 (1),
32–38.
Rosenbaum, P.R., 2017. Observation and experiment ed^eds. Harvard University Press,
Cambridge (Ma) USA.
Schneeweiss, S., 2006. Sensitivity analysis and external adjustment for unmeasured
confounders in epidemiologic database studies of therapeutics. Pharmacoepidemiol.
Drug Saf. 15 (5), 291–303.
Schunemann, H., Hill, S., Guyatt, G., Akl, E.A., Ahmed, F., 2011. The GRADE approach
and Bradford Hill’s criteria for causation. J. Epidemiol. Community Health 65 (5),
392–395.
Schünemann, H.J., Cuello, C., Akl, E.A., Mustafa, R.A., Meerpohl, J.J., Thayer, K.,
Morgan, R.L., Gartlehner, G., Kunz, R., Katikireddi, S.V., Sterne, J., Higgins, J.PT.,
Guyatt, G., 2019. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk
of bias in nonrandomized studies should be used to rate the certainty of a body of
evidence. J. Clin. Epidemiol. 111, 105–114.
Environment International 157 (2021) 106868
Shimonovich, M., Pearce, A., Thomson, A., Katikireddi, S.V., 2021. Assessing causality in
epidemiology: Revisiting Bradford Hill to incorporate developments in causal
thinking. Eur. J. Epidemiol. in press.
Steenland, K., Greenland, S., 2004. Monte Carlo sensitivity analysis and Bayesian
analysis of smoking as an unmeasured confounder in a study of silica and lung
cancer. Am. J. Epidemiol. 160:384-392.
Trinquart, L., Erlinger, A.L., Petersen, J.M., Fox, M., Galea, S., 2019. Applying the E value
to assess the robustness of epidemiologic fields of inquiry to unmeasured
confounding. Am. J. Epidemiol. 188:1174-1180.
Uddin, M.J., Groenwold, R.H., Ali, M.S., de Boer, A., Roes, K.C., Chowdhury, M.A.,
Klungel, O.H., 2016. Methods to control for unmeasured confounding in
pharmacoepidemiology: an overview. Int. J. Clin. Pharm. 38:714-723.
VanderWeele, T.J., Ding, P., 2017. Sensitivity analysis in observational research:
introducing the E-value. Ann. Intern. Med. 167 (4), 268. https://doi.org/10.7326/
M16-2607.
VanderWeele, T.J., Ding, P., Mathur, M.B., 2019a. Technical considerations in the use of
the E-value. J. Causal Inference, 20180007.
VanderWeele, T.J., Mathur, M.B., 2020. Commentary: Developing best-practice
guidelines for the reporting of E-values. Int. J. Epidemiol. 49:1495-1497.
VanderWeele, T.J., Mathur, M.B., Ding, P., 2019b. Correcting misinterpretations of the E-
value. Ann. Intern. Med. 170 (2), 131. https://doi.org/10.7326/M18-3112.
Verbeek, J., Hoving, J.L., Boschman, J., Chong, L.Y., Livingstone, J., Bero, L., 2021.
Systematic reviews should consider effects both from the population and the
individual perspective. Am. J. Public Health; in press.
Villeneuve, P.J., Goldberg, M.S., Burnett, R.T., van Donkelaar, A., Chen, H., Martin, R.V.,
2011. Associations between cigarette smoking, obesity, sociodemographic
characteristics and remote-sensing-derived estimates of ambient PM2.5: results from
a Canadian population-based survey. Occup. Environ. Med. 68 (12), 920–927.
Zeraatkar, Dena, Ah Han, Mi, Guyatt, Gordon, Vernooij, Robin, El Dib, Regina, et al.,
2019. Red and processed meat consumption and risk for all-cause mortality and
cardiometabolic outcomes: a systematic review and meta-analysis of cohort studies.
Ann. Intern. Med. 171, 703–710. https://doi.org/10.7326/M19-0655.