Effect Size
asking for meaningful effect sizes and their
Helena Chmura Kraemer
Stanford University, U.S.A and University of Pittsburgh,
U.S.A.
An effect size is a population parameter (esti-
mated in a sample) designed to indicate the
practical or clinical importance of a phe-
nomenon under study. As the limitations of
reporting and understanding p values from
statistical hypothesis testing become ever more
apparent, it is clearer that for any hypothesis
one might want to test (i.e., any phenomenon of
interest), it would be good practice to define an
appropriate population parameter that would
indicate how far the actual situation deviates
from the null hypothesis: an effect size.
First, an effect size is important in designing
any hypothesis-testing study. To be a valid 5%
significance test, for example, the probability of
rejecting the null hypothesis must be less than
5% for any effect size consistent with the null
hypothesis. To have, say, 80% power, one sets a
critical value of the effect size that indicates the
lower limit of practical/clinical significance,
and then demonstrates that the probability of
rejecting the null hypothesis is greater than
80% for any effect size greater than that critical
value.
Second, an effect size is also important in
reporting the results of a hypothesis-testing
study. Many a statistically significant result
has no practical/clinical importance—that
is, the estimated effect size is less than the
critical value of that effect size. A statistically
insignificant result means either that the study
was poorly designed (inadequate sample size,
unreliable measures, etc.) or the effect size is
well below the critical value. If the former is
true, a better design might be considered; if the
latter, the hypothesis might not be worth fur-
ther pursuit. It is the effect size that guides how
one interprets results. Hence, in recent years,
publication guidelines, as well as reviewers and
The Encyclopedia of Clinical Psychology, First Edition. Edited by Robin L. Cautin and Scott O. Lilienfeld.
© 2015 John Wiley & Sons, Inc. Published 2015 by John Wiley & Sons, Inc.
DOI: 10.1002/9781118625392.wbecp048
editors, have begun to protest naked p values,
confidence intervals.
Third, meta-analysis is based on effect sizes.
From each study of a phenomenon, an effect
size and a measure of its precision are extracted.
These are compared and, if appropriate, com-
bined over the studies to achieve a consensus
estimation of the effect size.
Finally, exploration (hypothesis-generation)
studies must necessarily place primary impor-
tance on the estimation of effect sizes because
there are no a priori hypotheses to be tested.
There are no universally accepted criteria for
what constitutes a well-chosen effect size.
Certainly, the effect size must be interpretable
in the context of its use. Many propose that the
effect size should be scale free; that it should
have null value zero; that it should be invariant
under at least linear rescaling of the measures,
perhaps preferably under monotonic rescaling.
As an illustration, let us consider probably the
most common phenomenon: the two-group
comparison.
Two-Group Comparisons
The researchers sample n1 subjects from one
population and n2 subjects from another. Here
Y is a random variable measured on all subjects
with possibly different distributions for the two
populations.
Cohen’s d
Here the most familiar measure of effect size
is Cohen’s d (Cohen, 1988). This is a mea-
sure designed to compare two populations,
where Y has a normal distribution in both
with equal standard deviations. Cohen’s d is
defined as the difference between the means in
the two populations divided by their common
standard deviation. It indicates the amount
of overlap between the two standard normal
distributions. Cohen’s d is invariant under
linear transformation of Y. Whenever the

2 EFFECT SIZE
two-sample t test is valid, so is Cohen’s d as an
effect size.
There are variations on this theme. Instead of
dividing by the estimate of the common stan-
dard deviation, one might divide by the stan-
dard deviation of one of the groups (e.g., the
control group in a randomized clinical trial) or
by the square root of the average variance. The
magnitude of Cohen’s d can be visualized as the
overlap between two standard normal distribu-
tions but it is a challenge to visualize what such
variations in Cohen’s d represent.
The Mean Difference
Cohen’s d is the standardized mean difference
between two groups. Why standardize, partic-
ularly if the scale used is a familiar one? The
problem is that if the two group means were
to differ by 10 units, and the within-group
standard deviation were 1, there would be
no overlap between the distributions, and no
doubt of the clinical significance of such a
finding; however, if the within-group stan-
dard deviation were 100, the overlap between
the distributions would be almost complete,
and the clinical significance very doubtful.
Thus ignoring the standard deviation leaves
the question of clinical/practical significance
completely in doubt.
Area under the ROC Curve (AUC)
There are many situations in which the variable
Y does not have a normal distribution in one of
the groups, or where the variances are unequal.
Cohen’s d is quite robust to minor deviations
from its underlying assumptions, but using
it with three-, four- or five-point scales, or
with continua with outliers, or unequal vari-
ances, can be quite misleading. The AUC,
representing the area under the receiver oper-
ating characteristic curve (ROC curve), is one
alternative. For present purposes:
AUC = probability (G1 > G2)
+.5 probability (G1 = G2),
where, for example, “G1 > G2” means that
if one compared a randomly selected sub-
ject in the first group (G1) with a randomly
10.1002/9781118625392.wbecp048, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/9781118625392.wbecp048 by Cochrane Philippines, Wiley Online Library on [13/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
selected subject in the second group (G2),
the subject in the first group has a bigger
(or better) response on Y and “G1 = G2”
indicates equality of responses. If the normal
distribution underlying Cohen’s d holds, then
√
AUC = Φ(d/ 2), where Φ() is the cumulative
standard normal distribution function and d
is either Cohen’s d (using the pooled variance
in the denominator) or d using the average of
the two groups’ variances in the denominator.
The AUC is invariant under all monotonic
transformations and can be used with any Y
(including binary, three-, four- and five-point
scales, and continua with highly skewed or
long- tailed distributions). Perhaps its only
weakness is that its null value is 0.5, rather than
0, but that is easily corrected by using instead
SRD (success rate difference) = 2AUC − 1. The
SRD shares all the qualities of AUC, but now
the null value is zero, and the two extremes
where the two populations do not overlap at all
are +1 and −1.
Number Needed to Treat/Take (NNT)
Clinicians, patients and policy makers often
have problems interpreting probability points
(as in AUC or SRD). The NNT instead reports
on number of subjects, a scale sometimes eas-
ier to interpret. Suppose one defines a subject
as a “success” if that subject had a response
(Y) bigger (better) than a randomly chosen
subject in the other group. How many subjects
would one need to sample from G1 to have
one more “success” than if one sampled the
same number from G2? Answer: NNT, which
is equal to 1/SRD. The NNT is most familiar
when Y is success/failure, in which case SRD =
p1 − p2, the difference in the probabilities of
success in the two groups (hence “success rate
difference”). However, ease of interpretation
here translates to confusing mathematical
properties. For example, NNT is undefined
when SRD = 0 (i.e., when there is no difference
between G1 and G2). Generally, therefore, for
research purposes, SRD is preferred to NNT
but one can always translate a SRD to NNT for
clear communication purposes.

Interpreting Effect Size
When Cohen introduced d (Cohen, 1988), he
suggested as a general guideline, that “small”,
“medium” and “large” effects corresponded to
d = 0.2, 0.5, 0.8 respectively (hence, SRD = 0.1,
0.3, 0.4 or NNT = 9, 4, 2). However, Cohen
also clearly warned against reification of this
guideline, and indicated that the appropriate
level should always be dictated by the specific
context. Thus, for example, NNT = 2,500
might be large in comparing subjects with vac-
cine versus saline injections to prevent polio
in a randomized clinical trial, but would be
ridiculously small in comparing a drug versus
psychotherapy to treat major depression, where
NNT = 3 might be a more logical choice. Thus
the critical effect size and the interpretation of
the magnitudes of the effect size are context
bound, guided by what is already known and
accepted in the particular context of use.
Beyond Two-Group Comparisons
A discussion like the above for the two-group
comparison problem is possible for each type
of phenomenon of interest. What is the best
choice of effect size to describe the correla-
tion between two random variables, or the
reliability of an ordinal measure or a binary
measure, or the predictive value of a set of
variables to an outcome, and so forth? The
principles underlying effect sizes have been
known as long as statistical hypothesis-testing
has been known but it is only in recent years
that these discussions have taken on urgency
and importance (Cumming, 2012 ; Grissom &
Kim, 2012).
10.1002/9781118625392.wbecp048, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/9781118625392.wbecp048 by Cochrane Philippines, Wiley Online Library on [13/08/2023]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
EFFECT SIZE 3
SEE ALSO: Clinical Significance; Cohen, Jacob
(1922–98); Randomized Clinical Trials
References
Cohen, J. (1988). Statistical power analysis for the
behavioral sciences. Hillsdale, NJ: Lawrence
Erlbaum Associates.
Cumming, G. (2012). Understanding the new
statistics: Effect sizes, confidence intervals, and
meta-analysis. New York, NY: Routledge.
Grissom, R. J., & Kim, J. J. (2012). Effect sizes for
research: Univariate and multivariate
applications. New York, NY: Routledge.
Further Reading
Acion, L., Peterson, J. J., Temple, S., & Arndt, S.
(2006). Probabilistic index: An intuitive
non-parametric approach to measuring the size
of treatment effects. Statistics in Medicine, 25(4),
591–602.
Kraemer, H. C. (2007). Correlation coefficients in
medical research: From product moment
correlation to the odds ratio. Statistical Methods
in Medical Research, 15(6), 525–545.
Kraemer, H. C., & Kupfer, D. J. (2006). Size of
treatment effects and their importance to clinical
research and practice. Biological Psychiatry,
59(11), 990–996.
McGough, J. J., & Faraone, S. V. (2009). Estimating
the size of treatment effects: Moving beyond p
values. Psychiatry (Edgmont), 6(10), 21–29.
Preacher, K. J., & Kelley, K. (2011). Effect size
measure for mediation models: Quantitative
strategies for communicating indirect effects
Psychological Methods, 16(2), 93–115.
Shearer-Underhill, C., & Marker, C. (2010). The use
of number needed to treat (NNT) in randomized
clinical trials in psychological treatment. Clinical
Psychology: Science and Practice, 17, 41–48.