PHARMACEUTICAL STATISTICS

Pharmaceut. Statist. 2007; 6: 155–160

Published online in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/pst.303

The difficult and ubiquitous problems of

multiplicities
Donald A. Berry*,y
Department of Biostatistics, M.D. Anderson Cancer Center, Houston, TX, USA

Multiplicities are ubiquitous. They threaten every inference in every aspect of life. Despite the focus in
statistics on multiplicities, statisticians underestimate their importance. One reason is that the focus is
on methodology for known multiplicities. Silent multiplicities are much more important and they are
insidious. Both frequentists and Bayesians have important contributions to make regarding problems
of multiplicities. But neither group has an inside track. Frequentists and Bayesians working together
is a promising way of making inroads into this knotty set of problems. Two experiments with
identical results may well lead to very different statistical conclusions. So we will never be able to
use a software package with default settings to resolve all problems of multiplicities. Every
problem has unique aspects. And all problems require understanding the substantive area of
application. Copyright # 2007 John Wiley & Sons, Ltd.

Keywords: multiplicities; Bayesian statistics; multiple comparisons; subset analyses

Most scientists are oblivious to the problems of
multiplicities. Yet they are everywhere. In one or
more of its forms, multiplicities are present in
every statistical application. They may be out in
the open or hidden. And even if they are out in the
open, recognizing them is but the first step in a
difficult process of inference. Problems of multi-
plicities are the most difficult that we statisticians
face. They threaten the validity of every statistical
conclusion.
Some statisticians do not understand that the
possibility of multiplicities may be a problem.
*Correspondence to: Donald A. Berry, Department of
Biostatistics, M.D. Anderson Cancer Center, Houston, TX,
USA.
y
E-mail: dberry@mdanderson.org
Others recognize the problem and overreact. I will
suggest my approach below, but I do not claim
that it is an ideal compromise. And indeed, it may
not be a compromise at all.
Multiplicities humble us all. The way we deal
with them separates good statisticians from the
less good. I have no qualms about my own abilities
in this regard and unhesitatingly place myself in
the less-good category. Too frequently I carry out
statistical tests or make posterior probability
calculations based on the ‘data’ sitting in front of
me. But the fact that these particular data are
sitting in front of me at all is part of the data! I
may choose to ignore this critical information
because I do not know what probability model to
use for the process that brought the data to me.

Copyright # 2007 John Wiley & Sons, Ltd.

156 D. A. Berry
But it nags at me. I sometimes try to bring this
information into the process at the very end, when
I report my conclusions. Multiplicities caveat all of
my analyses. The only favorable light I can shine
on my ability to handle them is that I recognize
their importance and their ubiquity.
My first exposure to multiplicities was in my
first-grade class, which was my first time away
from home. There were two redheads in the class.
They were from different families and they were
both very bright. I came to associate red hair with
intelligence. It took several years of meeting
redheads of normal intelligence and smart brun-
ettes and blondes for me to come to understand
that I had read too much into the early data.
The two brightest kids in any class are
necessarily similar in some other way – perhaps
in several other ways. Perhaps they are both girls,
both boys, both tall, both short, extreme in height
(one may be very tall and the other very short),
both of the same nationality or religion or ethnic
group, both overweight, both underweight, have
similar hair color, have buck teeth, have freckles,
speak with a lisp, can run fast, cannot run fast, are
handsome, are not handsome, etc. So I was
doomed. I was bound to learn something that
was wrong!
Part of the reason I drew the wrong conclusions
and was slow to unlearn was that I did not know
Bayes’ theorem. However, knowing Bayes theorem
is not enough: I might have applied it badly. Like
everyone else, Bayesians have great difficulty
properly interpreting observational data – whether
they know it or not! I dare say that most
applications of Bayes theorem are ‘bad’! For
example, the third redhead I met might have been
of normal or subnormal intelligence. But by this
time I had a pretty clear notion of the implications
of redheadedness and I was entrapped by my early
observations. So it would be easy to explain away
my latest observation. This apparent dolt was
probably bright after all, but just shy. Or maybe
his hair was not quite as red as the others. Or
maybe it is only redheaded girls who are smart.
Or maybe the third redhead was just weird and
not representative of his brethren. A mathematical
application of Bayes’ theorem for all three
Copyright # 2007 John Wiley & Sons, Ltd.
observations at once gives the same answer as
three sequential applications of Bayes’ theorem,
one after each observation. However, in practice
people will end of with different conclusions.
There are countless examples of this phenom-
enon in drug development. And it has the same
two traps: discounting valid observations, and
subsetting. So important are these phenomena
that they may account for as much as 30%
wastage of research expenditures in the pharma-
ceutical industry. (This subjective assessment is
subject to the vagaries of observational inter-
pretation, and you may reasonably wonder
whether my inferential abilities have improved
since the first grade!) Drug developers are en-
trapped by early results and can usually find
characteristics of more recent results that ‘explain’
why they are different.
Sometimes there are no known covariates to
explain the differences. In such a case one would
think that there is no recourse for the entrappee.
However, here is a real counterexample, with
names withheld to protect me from libel suits. A
company runs a randomized phase II study. At the
halfway point they observe a statistically signifi-
cant advantage for their drug over placebo. The
trial continues. In the second half the data
completely flip-flop, with a statistically significant
advantage for placebo. Overall, no difference. The
patient characteristics in the two halves were the
same. The company was sure that the first half was
the truth!
A related phenomenon is a subset analysis,
which is a difficult but reasonably well-understood
multiplicity [1,2]. There is always a subset of
patients that does better on the experimental drug
than does the complementary subset. A Bayesian
might observe a subset difference, assume a
noninformative prior distribution within the
subset, and find the corresponding posterior
distribution. This calculation is flawed. It
fails to model what is sometimes the most
important data of all: the process of determining
the subset. Of course, Bayesians are not unique in
this regard. Unfortunately, many researchers
and many drug developers fall prey to exactly
the same problem.
Pharmaceut. Statist. 2007; 6: 155–160
DOI: 10.1002/pst
 The difficult and ubiquitous problems of multiplicities
DOOMED BY SILENT
MULTIPLICITIES
All data analysis is subject to the curse of multi-
plicity. Some multiplicities are reasonably well
understood. Examples include subgroup analyses
and multiple comparisons of k>2 treatment groups.
[3,4] I do not mean to suggest that these ‘visible
multiplicities’ are easy to handle. However, they pale
in difficulty compared with ‘silent multiplicities.’ [5]
The latter are aspects of the ‘data’ that are hidden
from the analyst. And there are always some that are
hidden! You may think this statement is hyperbole.
After all, the analyst can design and run every aspect
of the experiment. But even in this case the analyst
must rely on his or her objectivity and memory to
ensure that nothing is hidden or discounted. None of
us can do this. And did the analyst consider when he
took a lunch break, why he took a lunch break then
and what he had for lunch?!
There is a fundamental distinction between
missing data or incomplete information and silent
multiplicities. The latter are not known to be
missing. They can reflect the data structure, or
aspects that are not part of the experiment under
consideration and therefore seem to be ancillary.
I distinguish two categories of silent multi-
plicities, depending on whether or not it is possible
to bring them into the open.
I collaborate with physicians and scientists in
the substantive fields. They frequently ask for an
analysis of a data set that they provide in a
spreadsheet. The data have usually been ‘cleaned.’
Perhaps the investigators have removed duplicates,
which seems innocuous enough. Or they have
averaged some observations, which starts to get
dicey. First, this process inflates the precision of
the measurements. More importantly, it may bias
the results in a number of ways because some extra
measurements may have been made ‘because the
original measurements were unusual.’ Or they may
have removed outliers, which is an easier problem
with which to deal because it is so obvious what
they have done and why. Or they may have
restricted to experimental units of most interest –
perhaps based on a perusal of the data! Sometimes
the most important statistical analyses have been
Copyright # 2007 John Wiley & Sons, Ltd.
157
carried out before I get to see the data. I have
learned to handle silent multiplicities by asking
lots of questions. I enquire about aspects of the
experiment all the way back to the time it was
conceived. Especially important are methods of
data collection and data processing. The answers
are usually revealing. In the worst case, the
answers make clear that the data as presented
are useless for inference.
Some types of silent multiplicities are avoidable,
depending on the circumstances. An extreme
approach is for the statistician to run the experi-
ment and coordinate all data collection and
handling. However, this does not make good use
of statisticians’ abilities. (Moreover, as I have
already indicated, it does not completely resolve
the problem.) We would get little else done.
However, it is possible and wise for us to be
involved in the data collection process and we can
supervise some of the critical aspects of this process.
More difficult to handle – and perhaps even
impossible to handle! – are aspects that have nothing
to do with the experiment per se. An investigator
conducts an experiment and for some reason does
not like the results. You never get to see the data.
Perhaps the investigator throws them out and so no
one else gets to see them. The fact that you did not
see the results of a particular experiment is informa-
tive and should be included in future inferences that
you might make. How on Earth to do that? Good
luck in building a model based on things you have
not seen, and that might not even exist!
Statisticians react to this dilemma by becoming
pessimists. It is an occupational hazard! Just as
with the publication bias, study results we are
given to analyze tend to be positive. Because there
are likely to be negative studies addressing the
same question but that never saw the light of day,
the next study is likely to be negative – or at least
less positive. (This effect is in addition to but
confounded with regression to the mean.) Our
pessimism leads us to give greater credibility to the
null hypothesis than would usually be appropriate.
However, some statisticians overreact.
As regards silent multiplicities, pharmaceutical
statisticians are better off than academic statisti-
cians. Firstly, they are more likely to be involved
Pharmaceut. Statist. 2007; 6: 155–160
DOI: 10.1002/pst
158 D. A. Berry
from the start. Secondly, the pharmaceutical
industry is much more closely regulated. So studies
have protocols. However, pharmaceutical statisti-
cians are not immune to the problem. For
example, in the preclinical setting an investigator
might restart an experiment because the first try
‘was not going very well.’
Even in a heavily regulated clinical setting there
are insidious silent multiplicities to plague phar-
maceutical statisticians. Suppose a clinical trial
shows a positive result. There are likely to be trials
sponsored by other companies in similar diseases
and with similar drugs. They matter! But informa-
tion about them, and even about their existence, is
lacking. It is hard to tell the size of the wave from
the perspective of the wave’s peak. The regression
effect (dampening the estimated positivity of the
result) is stronger if this were one of many studies.
Not all information about ‘other studies’ is
silent, of course. For example, the disease may
be one with the history of failed therapies (the
common cold, stroke, sepsis, and pancreatic
cancer come to mind). In such circumstances,
investigators should be more cautious. Unfortu-
nately, there are no frequentist statistical methods
for dealing with ‘other studies that may be related.’
The Bayesian approach can address this problem,
but the art of doing so is not well understood by
anyone, including by Bayesians. We have to rely
on the subjectivity of regulators in such matters.
As an example, despite many attempts, as of this
writing there is no approved vaccine for the
treatment of cancer. If there were to be a positive
vaccine trial, it should have a higher hurdle for
approval than if the results were identical but the
experimental drug is one in a known class of
cytotoxic agents.
BAYESIAN VS FREQUENTIST
APPROACHES
Based on my observations of statisticians in
action, frequentists are more likely to overreact
in adjusting what they see based on other things
that seem ancillary to less sensitive eyes. They tend
Copyright # 2007 John Wiley & Sons, Ltd.
to give more credence – sometimes too much – to
null hypotheses than do Bayesians. For example, a
standard frequentist approach is to adjust infer-
ences (such as Type I error rate) assuming that the
null hypothesis is true. In the extreme when there
is a large number of multiplicities, adjustments to
Type I error rates in this ‘familywise’ approach
make it nigh impossible to reject any of the various
null hypotheses.
On the other hand, Bayesians tend to be too
casual in handling multiplicities. In effect, their
approach gives too much prior credibility to
alternative hypotheses. They calculate posterior
probabilities based on the superficial data. They
assume some probability model for calculating the
likelihood function. However, this model is usually
inadequate in that it ignores multiplicities –
sometimes because it is difficult or impossible to
do otherwise, as I mentioned above in the context
of silent multiplicities. Too frequently Bayesians
misunderstand the ‘data.’ In addition to the
superficial data they should consider what they
have not observed. (Former U.S. Secretary of
Defense Donald Rumsfeld was ridiculed for saying
something that is actually quite insightful:
‘Reports that say that something hasn’t happened
are always interesting to me, because as we
know, there are known knowns; there are things
we know we know. We also know there are
known unknowns; that is to say we know there are
some things we do not know. But there are also
unknown unknowns – the ones we don’t know we
don’t know.’)
This is more than a theoretical problem.
Bayesians make major errors in analyzing existing
data, paying no heed to how they came to analyze
it in the first place. For example, Bayesian analyses
of ‘classical data sets’ are problematic. The fact
that a classical data set became classical is related
to the various conclusions of the data. Perhaps
Bayesians can be forgiven because frequentists are
similarly duped. However, Bayesian posterior
probabilities refer to the truth of the original
scientific question while frequentist measures
are much more limited in inferential scope,
referring only to the degree of concordance of
the superficial observations with hypothetical
Pharmaceut. Statist. 2007; 6: 155–160
DOI: 10.1002/pst
 The difficult and ubiquitous problems of multiplicities
distributions. So they are immune from the most
serious Bayesian mistakes.
Bayesians are wont to claim that they address
questions of direct scientific and clinical impor-
tance, and they do. However, this ability has a
huge price tag. Contrary to conventional wisdom
it is not because they must assume a prior
probability. It is that they have to work very hard
to assess their prior probabilities and they have to
work even harder to ensure that the likelihood
function they are using reflects all aspects of the
‘data’ at hand – including why they have it and
why they are analyzing it.
If I had to choose between the conservatism and
pessimism of the frequentist and the naiveté of the
Bayesian, it could be a difficult choice. I would
favor one in some instances and the other in
others. On balance I would choose the frequentist.
The naiveté that characterizes many Bayesians and
their approaches to inference too often leads to
serious errors.
HIERARCHICAL BAYESIAN
APPROACH
There are compromises that dominate both the
naı̈ve Bayesian approach and the conservative
frequentist approach. None of them is a panacea,
but such compromises elucidate the issues and may
enable us to begin understanding how to appro-
priately deal with multiplicities. One such com-
promise is hierarchical Bayesian modeling [6–11].
The idea is to borrow across groups within each
level of the hierarchy based on prior information
and on the extent to which the data in the groups
are concordant. A hierarchical Bayesian approach
is superior to both the frequentist and naı̈ve
Bayesian approaches. But it is not perfect.
Nothing is perfect.
The hierarchical Bayesian approach is especially
helpful for handling inferences when the dimen-
sion of the visible multiplicities is huge. For
example, genes on a cDNA microarray may
number in the tens of thousands. And there are
an estimated 1018 drug-like molecules and so the
number of possibilities is effectively limitless in
Copyright # 2007 John Wiley & Sons, Ltd.
159
protein/drug interaction experiments. (Exploiting
the elegance of hierarchical modeling is also
important in designing such experiments.) Borrow-
ing via modeling is critical. Understanding the
biology or the chemistry or the other relevant
science is essential for good modeling, of both the
likelihood and the prior distribution.
Although the hierarchical Bayesian approach
is an effective method of handling visible multi-
plicities, there is no panacea for silent multi-
plicities. Understanding and effectively addressing
these require both experience and knowledge.
Experience is necessary almost by definition
for it teaches the range of possibilities that
may be unseen in any particular experiment.
The issue is what might have happened that is
not stated and that is not obvious from the
numerical results. Knowledge of the subject matter
is necessary to place the experimental results in
context.
In addition, knowledge of the subject matter is
necessary for questioning the investigator in the
process of assessing whether there are unseen
multiplicities. For example, I know the culture in
some laboratory sciences is to redo parts of
experiments if the results do not seem to fit. The
investigators tend to drop the anomalous observa-
tions and present me with those that ‘seem right.’
We statisticians understand that this can bias the
results, and it always leads to underestimates of
variability. However, few scientists have our kind
of training. Since I know that this kind of thing
can happen, in any particular experiment I know
to ask if it did happen – and to worry even if the
answer is ‘no.’
THE BOTTOM LINE
Both frequentists and Bayesians have important
contributions to make regarding problems of
multiplicities. Neither group has an inside track
to the answer. Frequentists and Bayesians working
together is a promising way to make inroads into
this knotty set of problems. Two experiments with
identical results may well lead to very different
statistical conclusions. So we will never be able to
Pharmaceut. Statist. 2007; 6: 155–160
DOI: 10.1002/pst
160 D. A. Berry
use a software package with default settings to
resolve all problems of multiplicities. Every
problem has unique aspects. And all problems 5.
require understanding the substantive area of 6.
application.
7.
REFERENCES 8.
1. Berry DA. Multiple comparisons, multiple tests,
and data dredging: a Bayesian perspective (with
discussion). In Bayesian statistics, Bernardo JM 9.
et al. (eds.), vol. 3. Oxford University: Oxford, 1988.
2. Berry DA. Subgroup analyses. Biometrics 1990; 47:
1227–1230. 10.
3. Berry DA, Hochberg Y. Bayesian perspectives on
multiple comparisons. Journal of Statistical Plan-
ning and Inference 1999; 82:215–227. 11.
4. Gopalan R, Berry DA. Bayesian multiple
comparisons using Dirichlet process priors. Journal
Copyright # 2007 John Wiley & Sons, Ltd.
of the American Statistical Association 1998; 93:
1130–1139.
Berry DA. Statistics: a Bayesian perspective. Dux-
bury: Belmont, CA, 1996.
Berry DA. Bayesian clinical trials. Nature Reviews
Drug Discovery 2006; 5:27–36.
Berry DA, Stangl DK. Bayesian biostatistics.
Marcel-Dekker: New York, 1996.
Berry SM, Berry DA. Accounting for multi-
plicities in assessing drug safety: a three-level
hierarchical mixed model. Biometrics 2004; 60:
418–426.
Carlin BP, Louis TA. Bayes and empirical Bayes
methods for data analysis. Chapman & Hall: New
York, 1996.
Gelman A, Carlin JB, Stern HS, Rubin DB.
Bayesian data analysis. Chapman & Hall: New
York, 1995.
Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian
approaches to clinical trials and health-care evalua-
tion. Wiley: Chichester; 2004.
Pharmaceut. Statist. 2007; 6: 155–160
DOI: 10.1002/pst