The American Statistician

ISSN: 0003-1305 (Print) 1537-2731 (Online) Journal homepage: https://www.tandfonline.com/loi/utas20

On Causal Inferences for Personalized Medicine:

How Hidden Causal Assumptions Led to Erroneous
Causal Claims About the D-Value

Sander Greenland, Michael P. Fay, Erica H. Brittain, Joanna H. Shih, Dean A.

Follmann, Erin E. Gabriel & James M. Robins

To cite this article: Sander Greenland, Michael P. Fay, Erica H. Brittain, Joanna H. Shih, Dean
A. Follmann, Erin E. Gabriel & James M. Robins (2020) On Causal Inferences for Personalized
Medicine: How Hidden Causal Assumptions Led to Erroneous Causal Claims About the D-Value,
The American Statistician, 74:3, 243-248, DOI: 10.1080/00031305.2019.1575771

To link to this article: https://doi.org/10.1080/00031305.2019.1575771

Published online: 20 May 2019.

Submit your article to this journal

Article views: 1263

View related articles

View Crossmark data

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=utas20
THE AMERICAN STATISTICIAN
2020, VOL. 74, NO. 3, 243–248: General
https://doi.org/10.1080/00031305.2019.1575771

On Causal Inferences for Personalized Medicine: How Hidden Causal Assumptions Led
to Erroneous Causal Claims About the D-Value
Sander Greenlanda , Michael P. Fayb , Erica H. Brittainb , Joanna H. Shihc , Dean A. Follmannb , Erin E. Gabrield , and
James M. Robinse
a
Department of Epidemiology and Department of Statistics, University of California, Los Angeles, Los Angeles, CA; b Biostatistics Research Branch,
National Institute of Allergy and Infectious Diseases, Bethesda, MD; c Biometric Research Program, National Cancer Institute, Rockville, MD; d Department
of Medical Epidemiology and Biostatistics, Karolinska Institute, Stockholm, Sweden; e Department of Epidemiology and Department of Biostatistics,
Harvard T. Chan School of Public Health, Boston, MA

ABSTRACT ARTICLE HISTORY

Personalized medicine asks if a new treatment will help a particular patient, rather than if it improves the Received June 2017
average response in a population. Without a causal model to distinguish these questions, interpretational Accepted January 2019
mistakes arise. These mistakes are seen in an article by Demidenko that recommends the “D-value,” which
is the probability that a randomly chosen person from the new-treatment group has a higher value KEYWORDS
for the outcome than a randomly chosen person from the control-treatment group. The abstract states Causality; D-value; Effect size;
Individualized treatment;
“The D-value has a clear interpretation as the proportion of patients who get worse after the treatment” Patient-centered
with similar assertions appearing later. We show these statements are incorrect because they require
assumptions about the potential outcomes which are neither testable in randomized experiments nor
plausible in general. The D-value will not equal the proportion of patients who get worse after treatment
if (as expected) those outcomes are correlated. Independence of potential outcomes is unrealistic and
eliminates any personalized treatment effects; with dependence, the D-value can even imply treatment is
better than control even though most patients are harmed by the treatment. Thus, D-values are misleading for
personalized medicine. To prevent misunderstandings, we advise incorporating causal models into basic
statistics education.

1. Causal Models for Individual Effects
1.1. Background: The D-Value Is Not the Proportion of
Patients Harmed by Treatment
Despite great advances in causal modeling over recent decades,
it still seems largely unrecognized that intuitive causal interpre-
tations of statistical parameters can be rendered fallacious via
their dependence on hidden assumptions about causal struc-
ture. This problem is well illustrated with a recent article (Demi-
denko 2016) which made a number of remarkable causal claims
about a measure of association called the D-value.1 Over the
ensuing two years it became one of the most downloaded articles
in The American Statistician—which is alarming in light of
the fact that all the causal claims in the article are incorrect.
Foremost, the article states in the abstract that “The D-value
has a clear interpretation as the proportion of patients who
get worse after the treatment.” This statement is false under
even the simplest plausible causal model, yet is repeated on
page 37. Related incorrect assertions appear throughout the
article, for example, on page 36 “The D-value is for personalized
medicine when the treatment is sought, not on a group, but on
an individual level.” Worse, under the quoted misinterpretations
the D-value can indicate that the new treatment is better than the
control (D < 0.5) even though the treatment harms a majority of
its recipients (Hand 1992; Fay et al. 2018).
How were such profoundly erroneous claims justified? We
will show that the claims can be derived by introducing a statis-
tically nonidentified causal assumption, one which we regard as
extremely implausible in every setting we can imagine. Because
similar hidden assumptions appear to be behind other common
misinterpretations of effect measures, and given the attention
received by Demidenko (2016), we provide a detailed review of
the core problem: failure to recognize when interpretations are
based on strong and often implausible assumptions about the
effect of treatment on outcome.
1.2. Experimental Statistics and Modern Causal Modeling
Theory
Scientists have been making causal inferences from experiments
since the dawn of modern science, and formal causal models
for the design and analysis of randomized experiments go back
nearly a century (Neyman 1923; Welch 1937). Since then, the
topic of causal inference has become a major sub-branch of

CONTACT Sander Greenland lesdomes@ucla.edu Department of Epidemiology and Department of Statistics, University of California, Los Angeles, Los Angeles,
CA.
1 The D-value (possibly with tie adjustments) is also known as the Mann–Whitney parameter (Fay et al. 2018), the probabilistic index (Thas et al. 2012), the relative
treatment effect (Brunner and Munzel 2000), and for discrimination problems the concordance index or c-index (Harrell, Lee, and Mark 1996), where it equals the area
under the ROC curve (Demidenko 2016).
© 2019 American Statistical Association
244 S. GREENLAND ET AL.
statistical theory with the growing recognition that causal claims
require additional formal structure to link them deductively
to statistical analyses. Starting with the earliest randomization-
test literature, by far the most popular structures for this pur-
pose have been in the form of potential outcomes, which have
gradually begun to appear in textbooks, for example, see recent
examples by Morgan and Winship (2015), Imbens and Rubin
(2015), VanderWeele (2015), Pearl, Glymour, and Jewell (2016),
Rosenbaum (2017), and Hernán and Robins (2019). Using this
model to study effects of treatment, the traditional outcome
or response variable is expanded into a list (vector) of poten-
tial outcomes that shows what the outcome would be for the
same unit under different treatments; conditional independence
(“ignorability”) assumptions about treatment assignment are
then used to deduce tests and estimates of causal effects, with the
latter defined as contrasts comparing the marginal distributions
of the different potential outcomes in the same study group (e.g.,
all trial participants).
The resulting causal-inference theory is more complex than
classical regression or ANOVA, for there is now a vector variable
inserted where only one outcome variable appeared before.
Beyond that however it is distinct from traditional mathematical
statistics to the extent that the latter focuses strictly on deduc-
tions derived from assumptions about probability distributions
for the data. It has long been known that even complete knowl-
edge of those probability distributions cannot pinpoint the exact
causal mechanism (the explanation or “story”) generating the
data; for example, two mutually inconsistent potential-outcome
models arising from distinct mechanisms can lead to identical
data distributions, even when treatment is randomized (Robins
1986, sec. 2A; Robins and Greenland 1989a, 1989b; Dawid 2000;
Pearl 2009, sec. 11.1.1).
Thus, although experimental designs can identify and con-
trast various marginal distributions for the component poten-
tial outcomes, without further assumptions they cannot iden-
tify their joint (vector) distribution. That limitation may be
unsurprising: once we apply a treatment to a group, we can
only observe its outcome distribution under that treatment;
the potential outcomes under the unreceived (counterfactual)
treatments are now unobservable. Some critics have labeled this
nonidentification a problem with potential-outcome models;
others have responded that it is instead a valuable representation
of an innate limitation of purely statistical studies of causation,
for example, contrast Dawid (2000) with its discussants.2 As we
will explain in detail, it is precisely these inherent limitations
that were overlooked in Demidenko (2016).
To show how the quoted claims fail, we will use a basic
potential-outcome model for treatment effects in which Xi is
what the outcome of a randomly selected trial subject indexed
by i would be if given the control treatment (e.g., placebo or
standard of care) and Yi is what the outcome of the same subject
2
Indeed, one alternative to potential outcomes (Dawid 2015) deals with the
problem by using a reduced structure in which only marginal responses to
treatments are defined. This causal model reproduces the same observable
distributions as those derived from potential-outcome models, and so
yields identical inferential implications and limitations of statistical exper-
iments; it thus lacks sufficient structure to correctly represent concepts
like “proportion harmed” and “probability of causation” which contrast
potential outcomes within single individuals.
would be if given the new treatment (e.g., an experimental
drug). In more common causal notation Xi is Yi (0) and Yi is
Yi (1), but we here use Xi and Yi to better match Demidenko’s
notation. Let (Xi , Yi ) be the potential outcome vector for a
random subject, that is, the outcomes under control and new
treatment for subject i. Randomization splits the subjects into
two random samples from the total: a control sample in which
only the control response X is observed and Y is missing;
and a new-treatment sample in which only the new-treatment
response Y is observed and X is missing. The randomization
allows estimation (identification) of the marginal X and Y dis-
tributions by classical techniques (Rubin 1978).
Nonetheless, as has been long known (see, e.g., Dawid 2000
and discussants), treatment randomization3 does not identify
the distribution of the pairwise (joint) potential-outcome (Xi ,
Yi ) distribution. Specifically, the randomized-trial design allows
identification of only the marginal distributions of Xi and Yi .
In particular, if higher outcome values are considered undesir-
able, the proportion of patients who would be harmed by the
treatment is π = Pr(Yi > Xi ), which is a statement about
the pairs (Xi , Yi ). It thus cannot be identified from the trial
data alone because it depends on the joint (Xi , Yi ) distribution.
Because no complete (Xi , Yi ) pair is observed, we cannot verify
harm (Yi > Xi ), benefit (Yi < Xi ), or no effect (Yi = Xi ) in
any individual from the data alone; we only observe marginal
averages over these individuals (Senn 2009).
2. Why the D-Value Is Not the Proportion Harmed
2.1. D-Value Versus Proportion Harmed Under a Basic
Model
Demidenko (2016) introduced the D-value first under normal-
ity assumptions for X and Y, and then (in his sec. 3.1) non-
parametrically, referring to the D-value as both the sample
estimate and the population parameter. We focus on the D-value
parameter, first defining it generally, then examine assumptions
and their consequences.
Suppose we have N subjects in the study; then the complete
unobserved potential outcomes are the N pairs, (X1 , Y1 ), …,
(XN , YN ). The treatment-assignment process picks n of the N
subjects to get the control treatment (say subjects 2, 3, 5, 8, …)
and N − n subjects to get new treatment (say 1, 4, 6, 7, 9, …).
Then we only observe (., Y1 ), (X2 , .), (X3 , .), (., Y4 ), (X5 , .),
(., Y6 ), (., Y7 ), (X8 , .), (., Y9 ), and so on, where the “.” represents
the missing part of the pair (Rubin 1978). The D-value popula-
tion parameter is δ = Pr(Yi > Xj ), where Yi is the outcome
of a patient randomly drawn from the new-treatment group,
and Xj is the outcome of a distinct patient independently and
randomly drawn from the control group (cf. Demidenko 2016,
sec. 3.1). Note well, δ is defined using different indices i and j to
indicate that the two random variables in this sampling cannot
be from the same individual, since δ is derived from sampling
two distinct groups.
3
This means any weaker condition such as ignorability (independence of
treatment and potential outcomes across patients) will also not identify the
(Xi , Yi ) distribution.

When larger response denotes more harm, one might be
tempted to shorten the description of δ to “the proportion
harmed,” leading to the incorrect causal interpretations in
Demidenko (2016). To see why this is wrong, we must contrast
the D-value, δ = Pr(Yi > Xj ), to the proportion harmed,
which is π = Pr(Yi > Xi ). The key difference overlooked in
Demidenko (2016) is that (in sharp contrast to δ), the indices of
X and Y in the proportion harmed π are by definition identical:
use of the same index i in π denotes that the two outcomes must
come from the same individual. This follows common sense: to
claim patient i was harmed by the new treatment is to claim that
patient i would have done better on the control treatment, i.e.,
that Yi > Xi .
To highlight the difference between δ and π , we revisit the
example in Demidenko (2016) which compares placebo to a
weight-loss drug in a randomized trial. The average ending
weight is one pound less in the drug group than the placebo
group; the corresponding D-value is 0.486, which the article
states is “the proportion of patients who get worse after the
treatment.” Suppose however that the 1-pound average loss
arose because the diet drug produced some weight loss in every
patient and on average that loss was one pound (e.g., as would
happen if the treatment causes a one-pound weight loss relative
to placebo in everyone: Yi − Xi = −1 for all i); then no one was
or would be harmed by treatment (π = 0). Thus the D-value of
0.486 cannot be the proportion harmed.
If the outcome were transient with no carry-over or time
trend, and the trial could be repeated in the same patient
(an N-of-1 crossover trial), we could see both Yi and Xi , and
thus see directly whether a patient was harmed (Yi > Xi ),
unaffected (Yi = Xi ), or helped (Yi < Xi ) by the treatment.
Otherwise, one must turn to a causal model to impute the
missing potential outcome (Xi missing in the treatment group,
Yi missing in the control group) (Greenland, Rothman, and
Lash 2008; Morgan and Winship 2015; Imbens and Rubin
2015; Hernán and Robins 2019). The error in Demidenko
(2016) can thus be attributed to not holding the individual
subscript i constant when claiming to estimate the proportion
harmed π or other “personalized” effects of treatment, and thus
failing to recognize that the marginal distributions identified
by randomization do not determine the joint distribution on
which the proportion harmed π is defined. This oversight will
invalidate any statistical claim about personalized effects based
solely on the treatment-assignment mechanism (as opposed to
assumptions about the mechanism connecting treatment to the
outcome).
2.2. D-Value With Normality Assumptions
Demidenko (2016, sec. 3) assumed the marginal distributions
were normal, but made no explicit assumptions about the
bivariate distribution Fxy . Gadbury and Iyer (2000) assumed
that the (Xi , Yi ) pairs were draws from a bivariate normal
distribution; they noted that π is not identifiable from a
two-sample randomized experiment because the correlation
between Xi and Yi is not identifiable (since as noted above
for each individual only one of the potential outcomes is
observed). Under the bivariate normality assumption, we get
THE AMERICAN STATISTICIAN 245
δ = π if and only if the correlation is 0 so that Xi and Yi are
independent. Although π is not identifiable, Gadbury and Iyer
(2000) explored bounds on π under the bivariate normality
assumption. Even without bivariate normality, if the outcomes
of the same subject under different treatments are independent,
Pr(Xi , Yi ) = Pr(Xi )Pr(Yi ); under randomization the latter
product equals Pr(Xi )Pr(Yj ), leading to δ = π ; however, this
independence would mean a subject’s control outcome has
no predictive value at all for their treatment outcome. As we
hope will become obvious to the reader, this is an untenable
assumption.
It follows that any statement derived from confusing the D-
value with the proportion harmed can be highly misleading,
because strong positive correlations of potential outcomes
should be expected: regardless of treatment, an individual will
retain the same baseline outcome predictors, such as age, sex,
baseline weight, their entire genome, their entire biome, and
countless unmeasured predictors. Thus, a patient who would
have ended with a higher outcome than other patients under
control is also likely to end with a higher outcome under the
new treatment, simply by virtue of having identical baseline
predictors.
Not only can δ and π differ to a large extent, they can even
differ in direction in the sense of δ < 1/2 < π , and thus convey
a very different impression of typical direction of effect, so that
the new treatment could look better in terms of the D-value
(i.e., δ < 1/2) but could actually be worse for most patients (i.e.,
π > 1/2). Hand (1992) showed how this could happen even if X
and Y are marginally normal; specifically, with means μx and μy
and the same variance, σ 2 , for X and Y, then it is possible to have
opposite directional
  effects, and these opposite effects can occur
 μy μx 
whenever  σ  < 1.35, a result sometimes called “Hand’s
paradox. ” Under
 these normality assumptions, the D-value is
μ μ
δ = Φ √y x , where Φ is the standard normal distribution; it
2σ
follows that the paradox (i.e., δ and π indicate opposite direction
of effects) can occur whenever δ ∈ (0.170, 0.830) (Fay et al.
2018). So even if δ indicates that treatment is worse than control,
we cannot rule out the possibility that treatment benefits more
than half the population. Basically, with δ, large harms for the
minority can outweigh small benefits for the majority (and vice
versa).
2.3. D-Value Without Normality Assumptions
To illustrate how the D-value behaves without normality
assumptions, we explore three simple discrete-outcome exam-
ples. Let Fxy be the bivariate distribution of (Xi , Yi ), and let Fx
and Fy be the marginal distribution of Xi and Yi , respectively.
Figure 1 shows three bivariate distributions Fxy , all with
the same marginal distributions, Fx and Fy . The marginal
distribution for X has probability 1/3 on each of the points: 2,
4, and 6, while the marginal distribution on Y has probability
1/3 on each of the points: 1, 3, and 5. As before, assume larger
values are worse, so parameter values less than 1/2 imply that
the new treatment is better than the control. Since the marginal
distributions are the same in all three panels in Figure 1, the δ
values are the same: all have δ = Pr(Yi > Xj ) = 1/3. Figure 1(a)
246 S. GREENLAND ET AL.

Figure 1. Three bivariate distributions, all with the same marginal distributions: Fx has equal probabilities on 2, 4, and 6, and Fy has equal probabilities on 1, 3, and 5. Circle
areas are proportional to the probabilities.

shows the bivariate distribution when X and Y are independent; 3. Personalized Medicine and Statistical Inferences
this is the case when π = Pr(Yi > Xi ) = δ = 1/3. Figure 1(b)
In order for any measure to be used for personalized medicine,
shows the bivariate distribution with a constant effect Yi − Xi =
we need baseline covariates to describe different subgroups
−1 for all patients, for which case π = 0 since everyone benefits
with different effects. Without subgroup analysis we can only
from treatment (the effect is strictly downward monotonic); yet,
approach individual effects via their group averages. Examples
under Demidenko’s misinterpretation of δ, we would falsely
in simple two-sample randomized trials include the average
infer that δ = 1/3 of patients would do worse under the
causal effect E(Yi ) − E(Xi ) = E(Yi − Xi ), which shows how the
new treatment than under the control. Finally, in Figure 1(c)
usual mean difference is a simple average of individual effects, a
illustrates Hand’s paradox where π = 2/3 > 1/2, so that the
property not shared by other measures; notably, odds ratios are
new treatment harms most patients, yet under Demidenko’s
not weighted averages of individual causal effects (Greenland
misinterpretation of δ = 1/3 < 1/2, we would falsely infer that
Robins, and Pearl 1999). In the next section, we show how
most patients are not harmed by the new treatment. Of course,
δ − 1/2 can be reconstructed as an average individual effect,
Hand’s paradox can occur with continuous data as well. To see
albeit not a simple one. Because these effects are averaged over
this, imagine mixtures of bivariate normal distributions, with
individuals for each group, it is misleading to state “the D-value
their centers at the points in Figure 1(c).
is for personalized medicine when the treatment is sought, not
As in the case where Fxy is bivariate normal, in the gen-
on a group, but on an individual level” (Demidenko 2016, p. 36).
eral case we can identify δ but not π using only the marginal
If we want to come closer to individual effects we must
distributions Fx and Fy (which are all that are identified by
measure baseline covariates to allow us to “personalize” the
the randomization indicator alone, as in classical randomiza-
effects. Given data sufficient in quantity, quality, and detail (as
tion tests and their approximations).4 Nevertheless, we can get
from large trials and pooling projects), personalized treatment
bounds on π . Fay et al. (2018) reviewed bounds for π given the
decisions can be aided by estimating potential outcomes as
marginal distributions Fx and Fy under both nonparametric and
functions of baseline covariates Z available in the trial data
semiparametric assumptions.
(Robins 1986, sec. 3C; 2004; Murphy 2003). Even with such
For continuously distributed responses, Fx = Fy implies
resources however we think it probable that Y and X will remain
δ = 1/2, but for discrete responses we can have δ < 1/2
highly correlated within covariate levels, and thus may render
when, unlike the examples of Figure 1, there is the possibility
the covariate-specific D-values far from the actual proportion
of ties (i.e., Pr(Yi = Xj ) > 0). We could adjust the D-value
harmed.
for ties using Pr(Yi > Xj ) + Pr(Yi = Xj )/2, so that when
Fx = Fy then the adjusted D-value parameter equals 1/2 even
with discrete responses with ties. We could similarly adjust π
4. A Causal Interpretation of the D-Value
to Pr(Yi > Xi ) + Pr(Yi = Xi )/2 but this adjustment would
destroy its interpretation as the proportion harmed; for example, To summarize to this point: we can interpret δ causally by
π would then equal 1/2 for a treatment with no harm or benefit, viewing its departure from its null value of 1/2 as one measure
i.e., Pr(Yi = Xi ) = 1, and would equal 1/4 for a treatment of the change in the marginal outcome distribution produced
that harmed no one and benefitted half the patients, that is, by the new treatment relative to the control. Nonetheless, as we
Pr(Yi > Xi ) = 0, Pr(Yi = Xi ) = Pr(Yi < Xi ) = 1/2. have seen this interpretation is not a patient-specific (“individ-
ualized”) causal effect because it averages over within-patient
effects within each group, so the same average effect can come
from very different distributions of individual effects. For exam-
4
To see this note that δ is a functional gδ (Fx , Fy ) = Pr(Yi > Xj ) of the
randomization-identified marginal distributions, whereas π is a functional ple, δ = 1/2 if there is no individual effect (Yi = Xi for everyone,
gπ (Fxy ) = Pr(Yi > Xi ) of the nonidentified bivariate distribution. in which case π = 0), but δ = 1/2 could also occur if everyone

was affected (e.g., if half the patients had Yi − Xi = 1 and half
had Yi − Xi = −1, in which case π = δ = 1/2).
To understand the limitations of δ within a causal model,
we can imagine that the patients enrolled in the trial define a
population of potential outcomes under control treatment (the
X marginal distribution) and a population of potential outcomes
under the new treatment (the Y marginal distribution). Then
δ is the probability that a randomly selected potential outcome
under the new treatment is larger than an independently selected
potential outcome under the control treatment. In contrast, π
refers to pairs (Xi , Yi ) of these potential outcomes matched by
patient: π is the probability that a randomly selected patient has
a potential outcome under new treatment that is larger than that
same patient’s potential outcome under the control treatment.
While in general δ = π we may ask if there is at least
some sort of precise causal interpretation for δ in terms of the
change in outcome distribution produced by treatment. It turns
out there is one for the null-centered version δ − 1/2, albeit
one much less straightforward than the causal interpretation
for π because it depends on averaging explicit quantiles of the
marginal distributions of X and Y under the different treat-
ments. Specifically, Fay et al. (2018) derive δ − 1/2 in terms of
what they call the expected quantile difference (EQD) causal
effect, which should not be confused with the quantile causal
effect Fy−1 (Yi ) − Fx−1 (Xi ); see Xu, Daniels, and Winterstein
(2018). For simplicity, we will only describe the EQD under the
continuity assumption; see Fay et al. (2018) for the general tie-
adjusted version.
We first define the randomized outcome quantile level for the
total patient population in a 1:1 randomized experiment. Let the
observed outcome be Wi = Ri Yi + (1 − Ri )Xi , where Ri is the
independent random (Bernoulli) treatment-assignment indica-
tor with P(R = 1) = 1/2. Wi represents randomized observation
of one of the two potential outcomes Y and X for the ith indi-
vidual, which yields the ith individual’s observed response. The
distribution of Wi is G(w) = (Fx (w)+Fy (w))/2 = Pr(Wi ≤ w).
With G(wi ) = qi , the observed outcome wi for patient i is
then at the qi th quantile of G, and qi is the quantile level of the
observed outcome in the total patient population. The difference
in quantile levels for the ith individual due to treatment is
Di = G(Yi ) − G(Xi ) which Fay et al. (2018) called the quantile
difference causal effect for the ith individual. Di represents an
individual causal treatment effect insofar as it measures the
change in the individual’s ordinal outcome location in the total
patient population produced by the new treatment. For example,
if Di is 0.10, then the quantile level for the ith individual would
increase by 0.10 on treatment (e.g., go from 0.16 on control to
0.26 on treatment). Fay et al. (2018) showed that the expectation
of those individual causal treatment effects is δ − 1/2, that is,
δ − 1/2 = E(Di ) = E(G(Yi )) − E(G(Xi )).
Since G is a function of the marginal distributions only, δ is
identifiable. So a proper causal interpretation of δ − 1/2 from
a randomized experiment is the expectation of individual dif-
ference in quantile level causal effects. Although the causal
interpretation of δ is less straightforward than that of π , the lack
of identifiability of π limits its practical use, and so it may be
unsurprising that most of the modern causal modeling litera-
THE AMERICAN STATISTICIAN 247
ture has focused on models for treatment effects on marginal
distributions.
5. Discussion
The difference between the D-value parameter δ and the pro-
portion harmed π has long been known (see, e.g., Hand 1992)
yet has been overlooked repeatedly. One reason for the oversight
may be that in a randomized trial the observed outcomes on
which δ is defined (Xj and Yi from different patients j and i)
are independent, while the potential-outcome pairs on which
π is defined (Xi and Yi from the same patient i) are likely to
be highly correlated. Confusion thus seems inevitable when (as
in Demidenko 2016) there is no underlying causal model that
distinguishes the two cases. Once such a model is given, the mis-
take of confusing δ with π can be seen as a mistake of equating
independent marginal sampling to joint-distribution sampling
(sampling of patient-specific potential-outcome pairs).
This issue is closely related to other confusions of marginal
and individual causal measures. Consider a binary outcome or
failure-time outcome. Most textbooks of the past century as well
as many legal and policy documents mistakenly equated the
attributable fraction ϕ = (Pr(Yi = 1) − Pr(Xi = 1))/Pr(Yi =
1) (“attributable risk” or excess fraction) to the probability of
causation Pr(Yi > Xi )/Pr(Yi = 1) = π /Pr(Yi = 1). But as
above for δ versus π , the fraction ϕ is defined from the marginal
distributions and is thus identified by treatment randomization,
whereas π is not (Greenland and Robins 2000). Without strong
assumptions about the bivariate distribution of potential out-
comes (Xi , Yi ) we can only say ϕ ≤ π /Pr(Yi = 1) ≤ 1 (Robins
and Greenland 1989a, 1989b).
Demidenko (2016) repeats the well-known fact that a p-value
does not represent an effect-size estimate, which certainly bears
emphasis. Unfortunately, his article concludes that replacing the
p-value with the D-value offers a partial solution to “the poor
reproducibility of scientific experiments.” We strongly disagree
with that statement regardless of the chosen measure of effect.
Any measure is vulnerable to data-driven or otherwise moti-
vated selection effects by the analyst, including artefacts of sub-
group analysis. Even with perfectly honest and valid reporting,
in fields like psychology, social sciences, and medicine, effect
sizes can and should be expected to vary dramatically across
groups due to differences in the distribution of innumerable
modifiers of the effect. It is for example often lamented that the
effects of drugs seen in randomized trials do not predict well
the effects seen in clinical practice—a fact unsurprising when
one realizes that trials have numerous admission criteria that
could alter effects (e.g., excluding reproductive-age women), but
clinical prescribers can and do prescribe far beyond those limits
and practice in a much less controlled environment.
Those estimation problems aside, there seems to be poor
understanding that a valid p-value should vary dramatically
across studies (Senn 2001, 2002; Gelman and Stern 2006; Boos
and Stefanski 2011; Greenland 2019). Consequently, “replica-
tion failures” are often simply an artefact of using an arbi-
trary and easily crossed threshold such as 0.05 to dichotomize
results into “positive” and “negative” and from focusing on
only one hypothesis and its p-value (Poole 1987; Rothman,
248 S. GREENLAND ET AL.
Johnson, and Sugano 1999; Senn 2001, 2002; Greenland 2017a;
Amrhein, Trafimow, and Greenland 2019). Such dichotomania
and nullism has been condemned in scores if not hundreds
of sources, and is addressed simply by reporting precisely and
discussing the different p-values that one gets from testing dif-
ferent parameter values, rather than focusing on where those
p-values fall relative to some arbitrary if conventional cutoff
[in this journal see recent articles by Wasserstein and Lazar
(2016), Greenland et al. (2016), and Greenland (2019)]. Simply
adopting a different measure of effect does nothing to address
this core source of spurious “nonreproducibility.”
The main goal of this note has been to highlight the misinter-
pretation of statistical parameters that arise when the parameter
is not precisely defined by an explicit causal model. This has
been illustrated by claims that the D-value (δ) is the propor-
tion harmed (π ), when in fact those are two different param-
eters which may not even be close under realistic assump-
tions. It is difficult to see that the two parameters are different
without using potential outcomes or similar causal formalisms.
Although there remains disagreement about the relative merits
of formal and informal approaches to causal inference (Green-
land 2017b), we strongly advise that basic causal models become
part of elementary statistics education so that mistaken intuitive
interpretations of effect estimates can be avoided and valid
methods for effect prediction can be more widely deployed.
References
Amrhein, V., Trafimow, D., and Greenland, S. (2019), “Inferential Statistics
are Descriptive Statistics,” The American Statistician, 73(S1), 262–270.
[248]
Boos, D. D., and Stefanski, L. A. (2011), “P-Value Precision and Repro-
ducibility,” The American Statistician, 65, 213–221. [247]
Brunner, E., and Munzel, U. (2000), “The Nonparametric Behrens–Fisher
Problem: Asymptotic Theory and a Small-Sample Approximation,” Bio-
metrical Journal, 42, 17–25. [243]
Dawid, A. P. (2000), “Causal Inference Without Counterfactuals” (with
discussion), Journal of the American Statistical Association, 95, 407–448.
[244]
Dawid, A. P. (2015), “Statistical Causality From a Decision-Theoretic Per-
spective,” Annual Review of Statistics and Its Application, 2, 273–303.
[244]
Demidenko, E. (2016), “The p-Value You Can’t Buy,” The American Statis-
tician, 70, 33–38. [243,244,245,246,247]
Fay, M. P., Brittain, E. H., Shih, J. A., Follmann, D. A., and Gabriel, E. E.
(2018), “Causal Estimands and Confidence Intervals Associated With
Wilcoxon–Mann–Whitney Tests in Randomized Experiments,” Statis-
tics in Medicine, 37, 2923–2937. [243,245,246,247]
Gadbury, G. L., and Iyer, H. K. (2000), “Unit-Treatment Interaction and its
Practical Consequences,” Biometrics, 56, 882–885. [245]
Gelman, A., and Stern, H. (2006), “The Difference Between ‘Significant’
and ‘Not Significant’ Is Not Itself Statistically Significant,” The American
Statistician, 60, 4, 328–331. [247]
Greenland, S. (2017a), “The Need for Cognitive Science in Methodology,”
American Journal of Epidemiology, 186, 639–645. [248]
Greenland, S. (2017b), “For and Against Methodology: Some Perspectives
on Recent Causal and Statistical Inference Debates,” European Journal of
Epidemiology, 32, 3–20. [248]
Greenland, S. (2019), “Valid P-Values Behave Exactly as They Should: Some
Misleading Criticisms of P-Values and Their Resolution With S-Values,”
The American Statistician, 73, 106–114. [247,248]
Greenland, S., and Robins, J. M. (2000), “Epidemiology, Justice, and the
Probability of Causation,” Jurimetrics, 40, 321–340. [247]
Greenland, S., Robins, J. M., and Pearl, J. (1999), “Confounding and Col-
lapsibility in Causal Inference,” Statistical Science, 14, 29–46. [246]
Greenland, S., Rothman, K. J., and Lash, T. L. (2008), “Measures of Effect
and Measures of Association,” in Modern Epidemiology (3rd ed.), eds.
K. J. Rothman, S. Greenland, and T. L. Lash, Philadelphia: Lippincott-
Wolters-Kluwer (chapter 4). [245]
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman,
S. N., and Altman, D. G. (2016), “Statistical Tests, P-Values, Confidence
Intervals, and Power: A Guide to Misinterpretations,” The American
Statistician, 70, (S1), 1–12. [248]
Hand, D. J. (1992), “On Comparing Two Treatments,” The American Statis-
tician, 46, 190–192. [243,245,247]
Harrell, F. E., Lee, K. L., and Mark, D. B. (1996), “Multivariable Prognos-
tic Models: Issues in Developing Models, Evaluating Assumptions and
Adequacy, and Measuring and Reducing Errors,” Statistics in Medicine,
15, 361–387. [243]
Hernán, M. A., and Robins, J. M. (2019), Causal Inference, Boca Raton:
Chapman & Hall/CRC, forthcoming. [244,245]
Imbens, G., and Rubin, D. B. (2015), Causal Inference for Statistics, Social,
and Biomedical Sciences: An Introduction, New York, Cambridge Univer-
sity Press. [244,245]
Morgan, S., and Winship, C. (2015), Counterfactuals and Causal Inference:
Methods and Principles for Social Research (2nd ed.), New York: Cam-
bridge University Press. [244,245]
Murphy, S. A. (2003), “Optimal Dynamic Treatment Regimes,” Journal of
the Royal Statistical Society, Series B, 65, 331–355. [246]
Neyman, J. (1923), On the “Application of Probability Theory to Agricul-
tural Experiments. Essay on Principles.” Sec. 9, translation in Statistical
Science 1990; 5:465–472. [243]
Pearl, J. (2009), Causality: Models, Reasoning, and Inference (2nd ed.), New
York: Cambridge University Press. [244]
Pearl, J., Glymour, M., and Jewell, N. P. (2016), Causal Inference in Statistics:
A Primer, New York: Wiley. [244]
Poole, C. (1987), “Beyond the Confidence Interval,” American Journal of
Public Health, 77, 195–199. [247]
Robins, J. M. (1986), “A New Approach to Causal Inference in Mortality
Studies with Sustained Exposure Periods–Application to Control of the
Healthy Worker Survivor Effect.” Mathematical Modelling, 7, 1393–1512.
[244]
(2004), “Optimal Structural Nested Models for Optimal Sequential
Decisions,” in Proceedings of the Second Seattle Symposium in Biostatis-
tics, New York: Springer, pp. 189–326. [246]
Robins, J. M., and Greenland, S. (1989a), “Estimability and Estimation
of Excess and Etiologic Fractions,” Statistics in Medicine, 8, 845–859.
[244,247]
(1989b), “The Probability of Causation Under a Stochastic Model
for Individual Risks,” Biometrics, 45, 1125–1138. [244,247]
Rosenbaum, P. R. (2017), Observation and Experiment: An Introduction to
Causal Inference, Cambridge, MA: Harvard University Press. [244]
Rothman, K. J., Johnson, E. S., and Sugano, D. S. (1999), “Is Flutamide
Effective in Patients With Bilateral Orchiectomy?,” The Lancet, 353, 1184.
[248]
Rubin, D. B. (1978), “Bayesian Inference for Causal Effects: The Role of
Randomization,” Annals of Statistics, 6, 34–58. [244]
Senn, S. J. (2001), “Two Cheers for P-Values,” Journal of Epidemiology and
Biostatistics, 6, 193–204. [247,248]
(2002), “Letter to the Editor re: Goodman 1992,” Statistics in
Medicine, 21, 2437–2444. [247,248]
(2009), “Three Things That Every Medical Writer Should Know
About Statistics,” The Journal of the European Medical Writers Associa-
tion, 18, 159–162. [244]
Thas, O., Neve, J. D., Clement, L., and Ottoy, J. P. (2012), “Probabilistic Index
Models,” Journal of the Royal Statistical Society, Series B, 74, 623–671.
[243]
VanderWeele, T. (2015), Explanation in Causal Inference: Methods for Medi-
ation and Interaction. New York, NY: Oxford University Press. [244]
Wasserstein, R. L., and Lazar, N. A. (2016), “The ASA’s Statement on p-
Values: Context, Process, and Purpose,” The American Statistician, 70,
129–133. [248]
Welch, B.L. (1937), “On the Z-Test in Randomized Blocks and Latin
Squares,” Biometrika, 29, 21–52. [243]
Xu, D., Daniels, M. J., and Winterstein, A. G. (2018), “A Bayesian Nonpara-
metric Approach to Causal Inference on Quantiles” Biometrics, 74, 986–
996. [247]