Current Medicine Research and Practice 8 (2018) 222e229

Contents lists available at ScienceDirect

Current Medicine Research and Practice

journal homepage: www.elsevier.com/locate/cmrp

Review Article

Why the p-value is under fire?

R.L. Sapra a, *, Samiran Nundy b
a
Biostatistian, Sir Ganga Ram Hospital, New Delhi, India
b
Department of Surgical Gastroenterology & Liver Transplantation, Sir Ganga Ram Hospital, New Delhi, India

a r t i c l e i n f o a b s t r a c t

Article history: For almost a century after its introduction the p-value remains the most frequently used inferential tool
Received 4 September 2018 of statistical science for steering research in various scientific domains. This ubiquitous powerful statistic
Accepted 15 October 2018 itself is now under fire being surrounded by numerous controversies. We here review some of the
Available online 19 October 2018
important papers which highlight the prevailing myths, misunderstandings and controversies about this
statistic. We also discuss recent developments made by the American Statistical Association (ASA) in
interpreting p-value and guiding researchers to avoid confusion. Our paper is based on a search of se-
lective databases and we do not claim it to be an exhaustive review. It specifically aims to help medical
researchers/professionals who have little background of this contentious statistic and have been chasing
it indiscriminately in publishing significant findings.
© 2018 Sir Ganga Ram Hospital. Published by Elsevier, a division of RELX India, Pvt. Ltd. All rights
reserved.

1. Introduction The genesis of the p-value is credited to Sir Ronald Fisher,5 a

The formulation and philosophy of hypothesis testing known
today was largely developed between 1915 and 1933 by three sci-
entists - R. A. Fisher, J. Neyman and E. S. Pearson. Two schools of
thought for testing of hypotheses emerged from their philosophy
popularly known as ‘Fisherian’ and ‘Neyman-Pearsonian’. The
distinction between the two approaches is that the former sets the
null hypothesis and tests for its rejection using a significance level,
whereas the latter sets the alternative hypothesis along with the a
priori effect size and two types of errors (Type I - rejecting the null
hypothesis when there is no effect and Type II enot rejecting when
there is effect); and tests for acceptance of the alternative hy-
pothesis.1,2 Besides, there were many other basic philosophical
differences between the two schools. Lehman and his coworker
argued that the two theories were complementary rather than
contradictory. However, Perezgonzalez argued that Fisher's is the
closest approach to null hypothesis significance testing (NHST); it is
also the philosophy underlying common statistical packages, such
as SPSS. There are other approaches to hypothesis testing, such as
the Bayes' hypotheses testing (Lindley,1965)3 and Wald's (1950)4
decision theory which are not popular among medical re-
searchers owing to their computational complexity.
* Corresponding author.
E-mail address: saprarl@gmail.com (R.L. Sapra).
statistician and a biologist who devised this statistic to test the null
hypothesis. The Null Hypothesis is a statement of no difference
between an intervention(s) (and non-intervention) or no rela-
tionship between variables. For example, a researcher might want
to compare the effects of two drugs. One can set a hypothesis prior
to beginning the study that ‘Drug A and Drug B have the same ef-
ficacy’ or ‘There is no difference between the efficacy of Drug A and
B’. Researcher then conducts an experiment by taking care of all
mandatory requirements such as randomization and blinding etc.,
records data; and under certain statistical assumptions computes
the probability (p) of finding a result at least or more extreme than
the observed result assuming the null hypothesis is true.6 Fisher
advocated 5% to be the standard level for this probability (with 1%
as a more stringent alternative).1 Thus, if this probability is found to
be  0.05, researcher rejects the test (null) hypothesis. Fisher
perhaps might not have thought that his seemingly simple statistic
would be misunderstood and misused extensively later by re-
searchers for NHST, and would be surrounded by a number of
controversies. The major controversy is that it is largely misun-
derstood and hence misinterpreted. Some researchers argue that
there is something wrong with the p-value itself, whereas others
argue for its weak statistical standards (0.05) leading to a repro-
ducibility crisis of findings. Some critics have questioned its arbi-
trary cut-off (usually 0.05) and its dichotomous nature resulting in
widespread publication bias in the scientific literature. A few have
even challenged its validity and argued for alternatives. There are

https://doi.org/10.1016/j.cmrp.2018.10.003
2352-0817/© 2018 Sir Ganga Ram Hospital. Published by Elsevier, a division of RELX India, Pvt. Ltd. All rights reserved.
 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229
critics who complain of its ‘dancing attitude’. To share the flavor of
the debate, we give below quotes or titles of some of the important
papers out of a large number published in high impact journals:
1. “Mindless Statistics”,7
2. “The Difference Between ‘Significant’ and ‘Not Significant’ is
not Itself Statistically Significant”,8
3. “A Dirty Dozen: Twelve P-Value Misconceptions”,9
4. “Replication and p Intervals: p Values Predict the Future Only
Vaguely, but Confidence Intervals Do Much Better”.10 It's
science's dirtiest secret: The ‘scientific method’ of testing
hypotheses by statistical analysis stands on a flimsy
foundation.”,11
5. “Null Misinterpretation in Statistical Testing and its Impact
on Health Risk Assessment”,12
6. “False-Positive Psychology: Undisclosed Flexibility in Data
Collection and Analysis Allows Presenting Anything as
Significant”,13
7. “Bad Statistical Practice in Pharmacology and Other Basic
Biomedical Disciplines): “You Probably Don't Know P”,14
8. “Non significance Plus High Power Does Not Imply Support
for the Null Over the Alternative”,15
9. “The primary product of a research inquiry is one or more
measures of effect size, not P values”,16
10. “Why most published research findings are false?”,17
11. “The Statistical Crisis in Science [online]”,18
12. “Statistical techniques for testing hypotheses …have more
flaws than Facebook's privacy policies.”,19
13. “P values are not doing their job, because they can't”20 says
Stephen Ziliak, an economist at Roosevelt University in Chi-
cago, Illinois,
14. “P values, the 'gold standard' of statistical validity, are not as
reliable as many scientists assume”,20
15. “The fickle P value generates irreproducible results”,21
16. “Do Not Over (P) Value Your Research Article”22
17. “P-value shake-up proposed”.23
The widespread criticism of the inferential statistic used in
NHST among the scientific community is thus a matter of serious
concern. Some of these observations led David Trafimow, Editor of
the Basic and Applied Social Psychology (BASP) journal to take a
radical step and announce the ban of the p-value in future publi-
cations submitted to BASP. He says, “Null hypothesis significance
testing has been shown to be logically invalid and to provide little
information about the actual likelihood of either the null or
experimental hypothesis”.24 The journal now prefers strong
descriptive statistics, including effect sizes etc. instead of p-values.
Its editorial board also made two other major policy decisions in
favor of publishing papers with negative results or “null effects”
and those that contradicted already published work.25
These developments concerning the null hypothesis signifi-
cance testing and the rising criticism, and concerns surrounding
NHST built up pressure on the American Statistical Association
(ASA) to clarify its position on the p-value to avoid its further
misinterpretation, misuse and large-scale confusion among the
scientific fraternity. The ASA took up this challenge and finally in
January, 2016, released a formal statement clarifying several widely
agreed principles underlying the proper use and interpretation of
the p-value. Below we discuss “what is wrong” and “what is right”
with this inferential, yet, highly controversial statistic.
2. Misunderstandings or misinterpretations of p-value
A major concern expressed by critics is that NHST is misunder-
stood by many of those who use it.6 The p-value's inferential
223
meaning is widely and often wildly misconstrued during the last
eight decades and this has been pointed out in large number of
papers and books.9 A paper by Goodman,9 “A Dirty Dozen: Twelve
P-Value Misconceptions” reviews a dozen of common mis-
interpretations and their possible consequences. His observations
regarding the interpretation of the p-value are relevant when he
says, “It is not the fault of researchers that the p-value is difficult
to interpret correctly. The man who introduced it as a formal
research tool, the statistician and geneticist, R.A. Fisher, could not
explain exactly its inferential meaning”. A recent study by
Greenland et al26 provides a detailed list of 25 misinterpretations of
p-value, confidence intervals, and power; and also guidelines for
improving statistical interpretation and reporting. We advise
readers to refer to these papers for clarifications of their doubts
regarding null hypothesis significance testing and interpreting
their results. The most common misinterpretations of the p-value
are:
i) The p-value is the probability that the null hypothesis is true
when p > 0.05
ii) 1- minus the p-value is the probability that the alternative
hypothesis is true
iii) When p  0.05, the null hypothesis is false or should be
rejected
iv) When the p-value >0.05 there is no effect
v) A statistically significant result is also medically significant
The basic reason for misinterpretation of the p-value is the lack
of a researcher's understanding of its inferential capability. The p-
value is the probability of an observed or more extreme result
assuming that the null hypothesis is true. For example, if a
researcher wants to test whether a new drug is better than an old
one, he applies the t-statistic and gets a ‘t-value’ of say 2.0. Then he
looks for the probability of getting a result of 2.0 or more extreme
(t  2.0). This value which is shown under the green area (Figure) is
the probability and is called the p-value. This is automatically
calculated by the software. If the p-value is found to be less than or
equal to the cut off (usually p  0.05), he straightaway rejects the
null hypothesis. However, rejecting the null hypothesis does not
imply that it is false. Similarly, when p > 0.05 it is incorrect to say
there is ‘no effect’ or that the ‘null hypothesis is true’. Because we
did not know a priori whether the test hypothesis was true or false.
That is why we had to conduct the experimentation to find out. The
p-value was calculated on the assumption that it was true. When
p  0.05, the safer inferential statement we can make is that there is
at the most a 5% probability that our results are compatible with the
null hypothesis of ‘no effect’. The ASA's recent interpretation of the
p-value clearly says, “P-values can indicate how incompatible the
data are with a specified statistical model”.27 Here the statistical
model also includes the null hypothesis. The lower the p-value, the
greater the statistical incompatibility of the data with the null hy-
pothesis, if the underlying assumptions used to calculate the p-
value hold (Fig. 1).
Fig. 1. P-value.
224 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229
The definition of p-value does not in any way permit us to
calculate the reverse conditional probability i.e. the probability of a
hypothesis for given data. Thus, for example with a p-value of 0.02,
it is wrong to say that there is 98% probability that the alternative
hypothesis is true. p-value does not measure the probability that
the studied hypothesis is true, or the probability that the data were
produced by random chance alone.27
If we critically examine word by word the definition of the p-
value, it never makes any mention of the strength or effect of a
relationship. Thus, if we draw inferences about the strength of re-
lationships or effects on the basis of the magnitude of p-value, it is
simply over expectation from this poor statistic and is certainly
beyond its capability. A lower p-value does not ensure that the
results will also be medically significant or reflect a higher strength
of a relationship or effect. It measures a probability that judges
evidence against the null hypothesis.
P-value is a data dependent statistic and depends upon the
sample size as well, and decreases as the sample size increases.
Thus, normally a huge sample size leads to a low p-value until and
unless there is absolutely no effect. One of the most cited examples
in the literature in this context pertains to the study recommending
aspirin in preventing myocardial infarction (MI) based on more
than 22000 subjects.28 The results indicated statistically highly
significant reduction in MI (p < 0.00001), but surprisingly revealed
very small effect sizes (a risk difference of 0.77% with a r2 ¼ 0.001).
However, based on significant ‘p-values’ aspirin was recommended
for MI reduction29 without paying attention to the extremely low
effect sizes and later on these recommendations were modified.
Therefore, one should be cautious that extremely large studies may
be more likely to find a formally statistically significant difference
even for a trivial effect that may not be medically or biologically
meaningful; or different from the null.30e32
The p-value has been used and misused extensively in the sci-
entific literature without understanding its meaning. It was
developed to test inferential evidence against the hypothesis which
we do not know to be true or false. Many researchers have an
obsession about this, thinking that ‘p-values’ will add value to their
publications and therefore include more ‘p-values’ than necessary
in their studies. Statistical software have simplified and automated
this task. But each p-value requires a test hypothesis followed by
interpretation. It is better to clearly define the hypothesis at the
beginning of the study with a few specific questions whose answers
one seeks from the study by applying the ‘p-values’. There is
absolutely no need to estimate ‘p-values’ for each and every com-
parison or association until and unless they are directly or indi-
rectly associated with the evidence of the hypothesis(s) whose
answer one is seeking. NHST essentially requires setting up with
great caution, a test hypothesis prior to the conduct of study. The
test hypothesis should be such that the researcher has doubts about
it and does not know its answer. For example, a researcher may be
interested in knowing ‘Is there any difference in the taste when
milk is poured into tea decoction or when tea decoction is added to
milk’? The null hypothesis here is that the two methods of pre-
paring tea do not have any effect of its taste. Here, we do not know
whether the null hypothesis is true or false.
Kuffner and Walker33 say, “The p-value is innocent, but
woefully misunderstood”. But at the same time they question the
validity of the p-value and argue how a p-value which is data
dependent can form a valid formal ‘decision rule’ unlike a type-I
error which is fixed in advance.
3. Reproducibility crisis and the p-value
A steadily rising concern about reproducibility of results has
been witnessed among the scientific community especially during
the last decade. Recently there has been a doubt among
researchers34e37 that it is difficult to replicate the published re-
sults. However, its nomenclature (reproducibility, replicability,
reliability, robustness, and generalizability) as well as criteria for
checking vary among researchers. “The lexicon of reproducibility
to date has been multifarious and ill-defined”.38 There is no clear-
cut definition of reproducibility as on today. Perhaps the simplest
definition is that if some other researcher uses the same methods
then he or she should get similar results and can draw the same
conclusions.39 This definition was used by the journal while
conducting the online survey of their readers on reproducibility.
Nature's survey of 1,576 researchers revealed that more than 70%
of them had tried and failed to reproduce another scientist's ex-
periments, and more than half had failed to reproduce their own
experiments. To generate awareness about the irreproducibility
the journal Nature has recently started publishing a series entitled
“Challenges in Irreproducible Research” and the “Reproducibility
Initiative”, a project intended to identify and reward reproducible
research.21
The lack of reproducibility of the results questions the validity of
ongoing as well as earlier research programs and shakes the con-
fidence of their targeted audience including the project sponsoring
authorities. Generally it is a myth amongst the researchers that a
small p-value has the higher probability of producing significant
results if the experiment is replicated. It has been shown long ago
by Goodman40 that if the effect is a real one, the probability of
repeating a statistically significant result is substantially lower than
expected. Goodman40 and others41e46 have shown that ‘p-values’
overstate against the null hypothesis and that is why we sometimes
fail to get significant results. The high rate of non-replication is a
consequence of the convenient, ill-founded strategy based on a
single study and assessed on the p-value less than 0.05.47e49 One
may ask this question, “Is there something wrong with the p-value
which is generally misunderstood or there are some other factors
such as poor study designs, faulty statistical analysis, and scientific
misconduct contributing to non-reproducibility”? Johnson50 shows
that there is something wrong with the p-value itself and proves
mathematically that the use of the 0.05 standard accounts for most
of the problems of non-reproducibility in science - even more than
other factors, such as bias and scientific misconduct. He advocates
for lowering down the level of significance from 0.05 to the 0.005 or
0.001 as very few studies fail to replicate results that are based on
p-values of 0.005 or smaller. It is hard to believe that nearly one-
quarter of studies concluded on the basis of commonly used sta-
tistical threshold (0.05) may be false.51
Let us understand what is wrong with the p-value and why it
does not ensure reproducibility? The p-value is estimated from the
data comprising of random variables, hence it is a random variable
itself.52e54 It is surprising that the p-value being the most popular
statistic among researchers for testing evidence against the null
hypothesis is never estimated with a standard error or confidence
interval unlike other statistics. Indeed, it is not possible to do so for
p, because it estimates unobservable population parameters; and
because the p-value is a property of the sample, there is no unob-
servable ‘true p-value’ in the larger population.55 The random
behavior of the p-value is generally neglected. This huge sampling
variability in the p-value was called the ‘dance of the p-values’.56
The major cause of the lack of repeatability is the wide variability
from sample to sample in p-value which is not considered.21,57 A
recent study in genomics suggests that over interpretation of very
significant, but highly variable, p-values is an important factor
contributing to the incidence of non-reproducible studies.58 It is
only as reliable as the sample from which it is calculated21 and
inherits its vagueness from the uncertainty of the estimates from
which it is calculated.59
 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229
A recent story published by Nosek60 and co-researchers
including Motyl, a psychology PhD student is an eye opener for
all of us who are somehow stuck to drawing inferences solely based
on the unpredictable nature of the p-value. They conducted a study
to investigate the embodiment of political extremism on 1979
participants and completed the study with a statement that polit-
ical moderates saw shades of grey more accurately than did either
left-wing or right-wing extremists (p ¼ 0.01). However, before
writing and submitting the manuscript they had a chance to look at
the two recent articles highlighting the possibility that research
practices spuriously inflate the presence of positive results in the
published literature.13,61 Triggered by these findings, investigators
replicated the experiment with 1300 participants ensuring a sta-
tistical power of 0.995 to detect an effect of the original effect size at
a significance level of 5%. To their great surprise, the effect dis-
appeared (p ¼ 0.59). These controversial results discouraged the
authors from submitting their manuscript for publication. This true
story clearly depicts the slippery nature of p-values and raises
serious concern about irreproducibility among the scientific com-
munity. If every researcher did what Nosek et al.60 did, i.e. repeated
their experiments before sending manuscripts for publication,
perhaps some of them might have met the same fate. It won't be
wrong to say that some of the researchers might be genuinely
interested in replicating their experiments and verifying results,
but in the mad race for publishing sooner because of placement in
jobs or profile raising etc. they opted not to do so. However, it is also
true that sometimes constraints of resources or unnecessarily
placing subjects at a risk from intervention do not permit re-
searchers to do this. Authors of the story truly admit e “When in-
centives favour novelty over replication, false results persist in the
literature unchallenged”. A detailed quantitative assessment of
non-reproducibility among psychological studies based on repli-
cations of 100 experimental and correlational studies published in
three psychology journals using high-powered designs and original
materials, indicated a decline of sixty one percent in significant
findings in replications (36%) over the originals (97%).62
Ioannidis has a pessimistic view about the reliability of the
publications and says, “Why most published research findings are
false?“17 This thought-provoking statement made by the author is
not based on whimsical grounds but on simulation studies. The
author says “for most study designs and settings, it is more likely
for a research claim to be false than true”. These observations
certainly compel us to think ‘should we believe in our past research
claims concluded merely on the basis of a single study without
verifying and replicating the experiment?’ We cannot undo what
has occurred in the past. Certainly, some precautions can be taken
in ongoing and future research programs. A recent study reported a
large scale non-reproducibility (89%) in preclinical cancer research
studies based on some 53 selected key papers and suggested some
useful guidelines including ‘tags’ that mention whether the key
findings of a seminal paper were confirmed.63 The scientific com-
munity is shaken by the recent reports of the huge proportion of
non-reproducibility of peer-reviewed preclinical studies and
certain guidelines have been formulated for publication of these in
the magazine Science.64
A recent study59 reviews that degrading p-values into signifi-
cant and non significant contributes to making studies irrepro-
ducible, or to making them seem irreproducible. Authors of the
study further emphasize interpreting p-values cautiously as a
graded evidence against the null hypothesis by making use of effect
sizes and interval estimates.
A serious concern about reproducibility has been raised in the
ASA's statement on p-values while introducing the statement,
however, it does not say what strategy should be adopted to check
the prevailing high level of non-reproducibility in the scientific
225
literature. But, it can be checked to a greater extent if researchers
understand the meaning of p-values in the new perspective and
adhere to its 4th principle that requires full reporting and trans-
parency for drawing proper inferences.
4. P-hacking, publication bias and false positives
“P-hacking” or “data dredging” or “data manipulation” is a
malpractice followed by researchers knowingly or unknowingly
where they keep on fiddling with the data i.e. selecting or dis-
carding data or trying various statistical tools until non-significant
results become significant.13,65 The background behind this practice
is the publication bias which is popularly known as the “file drawer
effect” where studies with non-significant or negative results are
sent to the file drawer and consequently have lower publication
rates compared to significant ones. There is strong evidence that
journals, especially those with higher impact factors, dispropor-
tionately publish significant results.66e68 A recent study conducted
on three top orthopedic journals clearly indicated an evidence of p-
hacking in the orthopedic literature.69 Authors used frequency
plots of p-curves for these journals and revealed a statistical evi-
dence of p-hacking.
The dichotomous nature of NHST using p-value (if p  0.05,
reject the null hypothesis or not) is also a driving force for p-
hacking contributing to publication of false positives and conse-
quently most of the manipulations in results, if any, are done by the
researchers when they get p-values slightly above the threshold
value (0.05). This threshold value of 0.05 is also under criticism. It is
not feasible to assess the extent of p-hacking as it depends upon the
integrity of the researcher. An indirect way of detecting this is to
look for an increase in the relative frequency of p-values just below
0.05, where we expect the signal of p-hacking to be the strongest.69
P-hacking can also be tested during meta-analysis and its effect
seems to be weak relative to the real effect sizes; and probably does
not drastically alter scientific consensuses drawn from meta-
analyses.70
P-hacking promotes publication of false positives and probably
amounts to scientific misconduct. Its infection spreads fast and
remains undetected. It may adversely affect the other related
ongoing or future research programs in interpreting or reviewing
their results and finally drawing conclusions and making policy
decisions etc. The main culprit that promotes p-hacking is the de-
grees of freedom available with the researcher. The issue of this
freedom has been discussed by a number of authors.13,61,71e80 The
undisclosed flexibility or degrees of freedom13 enjoyed by the re-
searchers in the course of collecting and analyzing data, excluding
and including certain observations, adding or removing variables,
applying alternative statistical tools ultimately takes him to the
path of significant results by getting p  0.05. A recent review
article critically presents p-hacking in relation to degrees of
freedom and develops an extensive list of 34 degrees of freedom
that researchers have in formulating hypotheses, and in designing,
running, analyzing, and reporting of psychological research.81 P-
hacking is not a problem with p-value per se. Researchers get
trapped into p-hacking to report strong or significant results lead-
ing to publication bias. Simmons et al13 offers a practical solution to
the flexibility-ambiguity problem by offering six requirements for
authors and four guidelines for reviewers.
The recent ASA statement on the p-value discussed in the last
section of the statement, unequivocally discourages the wide-
spread and ongoing practice of p-hacking by making it mandatory
full reporting and transparency by the author for drawing a proper
inference. It further adds in the explanation of its 4th principle
e“Cherry picking promising findings, also known by such terms
as ‘data dredging’, ‘significance chasing’, ‘significance questing’,
226 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229
‘selective inference’, and ‘p-hacking’, leads to a spurious excess of
statistically significant results in the published literature and
should be vigorously avoided”.27 Amrhein et al.59 strongly argue
for removing fixed significance threshold to overcome the problem
of p-hacking and emphasize interpreting ‘p-values’ as graded
measures of the strength of evidence against the null hypothesis.
5. Weak and dichotomous threshold of p-values
Most of the research findings are concluded on the threshold
value of p i.e. if p is below 0.05 results are considered “statistically
significant” otherwise not. No doubt, this arbitrary cutoff has made
the task easier for the researchers in concluding their findings but
at the same time has led to scientifically dubious practice of
regarding significant findings as more valuable, reliable and
reproducible.6 This arbitrary cutoff (usually 0.05) has been ques-
tioned primarily on two grounds e one that many researchers feel
that this cutoff is too high and the other that it promotes p-hacking
leading to generating false positives. The weak evidence provided
by p-values between 0.01 and 0.05 has been recently explored by
Colquhoun82 in terms of false positive risks and he proved that
when p ¼ 0.05, the odds in favour of there being a real effect (given
by the likelihood ratio) are about 3: 1 and these odds are very low
compared to 19 to 1 inferred from the p-value. Further to achieve a
false positive risk of 5%, one would need to observe p ¼ 0.00045 for
a prior probability of a real effect of 0.1.82
In an another recent communication researchers also argued that
there is weak evidence when the p-value is 0.05 and this threshold
should be lowered to 0.005 for the social and biomedical sciences.83
In other scientific fields such as particle physics and DNA studies
scientists have asked for a p-value as low as 0.0000003 and for a
genome-wide association study a p-value of 5  10-8.23 No doubt,
lowering the p-value raises the evidence against the null hypothesis
and reduces false positives. But at the same time this requires a
higher sample size for rejecting the null hypothesis which may be a
limitation in many medical experiments and may even involve a
higher risk besides increasing the project cost. The other major
problem in setting the lower thresholds p-value is that it enhances
the chances of a false negative which is popularly known as a type II
error i.e. we may conclude that effects do not exist where in reality
they do. Consequences of getting such negative results have a higher
likelihood of the “file drawer effect” and have a lower publication
rate. In other words keeping lower thresholds may deprive users the
benefits of new drugs or technologies.
There is another think-tank which questions even the validity of
the p-value for NHST and favours the use of Bayesian methods over
the p-value. “The family of Bayesian methods has been well
developed over many decades now, but somehow we are stuck to
using frequentist approaches”.84 The basic reason being that they
are computationally complex compared to traditional ‘frequentist
methods’ and require technical expertise.
The problem arising as a result of threshold values of ‘p’ can be
overcome to a greater extent if we adhere to recent statement is-
sued by the American Statistical Association with respect to’ p-
values’ given here under the section ‘ASA's statement on p-values’.
The third principle clearly says “Scientific conclusions and business
or policy decisions should not be based only on whether a p-value
passes a specific threshold”. Furthermore, principles 4 and 5 of the
said statement are straight forward in defining the limitations of
the p-value that “it does not measure the size of an effect or the
importance of a result “ and “By itself, a p-value does not provide a
good measure of evidence regarding a model or hypothesis”. It is
worth mentioning here that the ASA's statement does not say
anything about what should be the value of threshold. Perhaps it
has left it to the requirement of the researchers and hypothesis to
be tested. However, the ASA's interpretation of the third principle
cautions the users while making decisions purely based on the
threshold value and emphatically say that the p-value alone cannot
ensure that a decision is correct or incorrect and researchers should
consider contextual factors into play to derive scientific inferences.
It is hardly two years since the ASA statement was released. But a
huge concern is being witnessed among the methodologists about
the threshold of the p-value. Recently a group of 72 methodologists
argued that the prevalent threshold of 0.05 was too high and pro-
posed its lowering to 0.005 for claiming statistical significance for
new discoveries.85 However, their proposal met a mixed response
with strong endorsement in some circles and concerns in others.86
6. The ASA's statement on p-values: context, process, and
purpose
Null Hypothesis Significance Testing (NHST) has been in practice
for almost nine decades since 1925 and a subject of debate with
numerous controversies. However, unfailing rising criticism and
concerns surrounding the NHST during the past few years certainly
built up pressure on the American Statistical Association (ASA) to
clarify its position on the p-value to avoid its further misinterpre-
tation, misuse and large scale confusion among the scientific fra-
ternity. ASA is the world's largest body of statisticians and the
oldest continuously operating professional science society in USA.
Stimulated by a number of observations and several intriguing
thought provoking articles published in recent past such as “It's
science's dirtiest secret: The ‘scientific method’ of testing hypoth-
eses by statistical analysis stands on a flimsy foundation.“,11
“numerous deep flaws in null hypothesis significance testing”,87
“statistical techniques for testing hypotheses …have more flaws
than Facebook's privacy policies.“,19 “Scientific Method: Statistical
Errors”,88 “Why do so many colleges and grad schools teach
p ¼ 0.05?” & “Why do so many people still use p ¼ 0.05?“, questions
posed to the American Statistical Association (ASA) discussion
forum by George Cobb, Professor Emeritus of Mathematics and
Statistics at Mount Holyoke College, “Psychology journal bans P
values”89 and the issues related reproducibility crisis, the ASA
Board decided to take up the challenge of developing a policy
statement on p-values and statistical significance. Based on the
widespread consensus in the statistical community, ASA developed
a formal statement clarifying several widely agreed upon principles
underlying the proper use and interpretation of the p-value. The
Executive Committee of the ASA took almost three months, held
discussions at length and finally approved the statement on January
29, 2016 by defining the p-value along with six guiding principles. It
is worth mentioning here that the ASA has not previously taken
positions on specific matters of statistical practice.27 ASA's state-
ment has been written in very simple language avoiding the tech-
nical terms such as alternative hypothesis, statistical power and
errors associated with hypothesis testing etc.; and can be easily
understood even by who don't have statistical background.
7. What is a p-value?
“Informally, a p-value is the probability under a specified sta-
tistical model that a statistical summary of the data (e.g., the
sample mean difference between two compared groups) would be
equal to or more extreme than its observed value”.
7.1. Principles
The statement issued by the ASA includes following six guiding
principles along with its explanation citing examples wherever
required:
 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229
1. “P-values can indicate how incompatible the data are with a
specified statistical model”
2. “P-values do not measure the probability that the studied hy-
pothesis is true, or the probability that the data were produced
by random chance alone”
3. “Scientific conclusions and business or policy decisions should
not be based only on whether a p-value passes a specific
threshold”
4. “Proper inference requires full reporting and transparency”
5. “A p-value, or statistical significance, does not measure the size
of an effect or the importance of a result”
6. “By itself, a p-value does not provide a good measure of evidence
regarding a model or hypothesis”.
We are not here giving the interpretation part included in the
ASA statement with each of the principle. Readers are advised to
look for the same available on the net. As far as contents of the
statement are concerned, Wasserstein & Lazar who happen to be
associated with the development of the statement clearly admit by
saying “Nothing in the ASA statement is new”. Statisticians and
others have been sounding the alarm about these matters for de-
cades, to little avail”. Wellek90 while critically evaluating the cur-
rent p-value controversy feels that principles included in the ASA
statement provide much in terms of guidance for good practice of
handling and interpreting p-values, and hypothesis tests. However,
the author strongly criticizes the 6th principle of the statement and
questions the consensus on the precise meaning of concept of ev-
idence i.e. what is a good measure of evidence and what is a bad
measure. While discussing the problem of reproducibility of results
and misinterpretation of p-value, Colquhoun feels that the ASA
statement includes “what you should not do, but failed to say what
you should do”.82 We are of this opinion that the statement leaves
most of the job up to the discretion of researchers while inter-
preting and making decisions. But at the same time it has come out
with a clear definition and interpretation of p-value which had
been misunderstood over the past several decades by many of re-
searchers having over expectations from this value and causing
damage to science. The statement will definitely help in reducing
researchers’ over dependency on p-values in concluding their
findings.
With the ASA guidelines, the dichotomy of the p-value has been
made porous so as to enable the researcher to see the truth on
either side of the threshold. This threshold has been a major cause
of augmenting false positives in the scientific literature. Ron Was-
serstein says “In the post p < 0.05 era, scientific argumentation is
not based on whether a p-value is small enough or not. Attention is
paid to effect sizes and confidence intervals. Evidence is thought of
as being continuous rather than some sort of dichotomy. (As a start
to that thinking, if p-values are reported, we would see their
numeric value rather than an inequality (p ¼ 0.0168 rather than
p < 0.05)). All of the assumptions made that contribute information
to inference should be examined, including the choices made
regarding which data is analyzed and how. In the post p < 0.05 era,
sound statistical analysis will still be important, but no single
numerical value, and certainly not the p-value, will substitute for
thoughtful statistical and scientific reasoning”.91 Amrhein and his
co-researchers59 share the same opinion and suggest “ i) do not use
the word ‘significant’ and do not deny observed effect if the p-value
is relatively large and ii) discuss how strong we judge the evidence,
and how practically important the effect is”.
It is over two years when the ASA's statement was released and
duly highlighted by the media as well. Our personal observations
are that the majority of medical researchers in India and other
countries may not be aware, or if aware might not have read these
newly formulated guidelines. After one year of its release Matthews
227
et al92 commented, “Yet a year on, it is not clear that the ASA's
statement has had any substantive effect at all”. They further add
that the business is going as usual even in the leading journals like
The Lancet or Proceedings of the National Academy of Sciences. It is
now time to sensitize this issue which happens to be the inferential
base of most our research findings on a massive scale, through
workshops, conferences and holding teaching classes etc. A pro-
active role is expected from statisticians all over world in this sci-
entific endeavor. Such ventures will help in changing the
conventional thought of researchers, reviewers and editors in un-
derstanding the meaning of the p-value, its limitations and its
worth. In this context the scientific community must keep in mind
the concluding remark made in ASA statement that “No single in-
dex should substitute for scientific reasoning”. Therefore, we
should also be open to other alternatives such as Bayesian inference
tools etc. No doubt, these tools appear to be computationally
complex. But so far none of the alternative tools have been shown
to be superior to p-statistic. Abandoning the p-value is not a solu-
tion to the problem. Much more needed is the understanding of the
pulse of the data while using this statistics for testing the evidence
of hypothesis.
8. Conclusions
Much of the criticism that has been prevailing around the p-
value is because of researchers' own lack of understanding of the
statistic, its limitations & worth and over expectations from this
value. Most of the researchers today are ignorant of the behavior of
p-values also known as the ‘dance of the p-values’ which is also
responsible for irreproducibility crisis. It is now time to sensitize
this issue which happens to be the inferential base of most our
research findings on a massive scale, through workshops, confer-
ences and holding teaching classes etc. We believe the recent policy
statement of the ASA will have a far reaching effect in checking the
publication of false positives and curtailing down the “file drawer
effect”. We hope this brief essay will be helpful to those who may
be interested in knowing ‘Why the p-value is under fire?’
Potential conflicts of interest
The authors have nothing to disclose.
Acknowledgements
We sincerely acknowledge Dr. S.R. Bhat, Former Principal Sci-
entist, NRCPB, IARI campus, New Delhi (India) for going through the
draft manuscript critically and valuable suggestions.
References
1. Fisher Lehman EL. Neyman-Pearson theories of hypotheses testing: one theory
or two? J Am Stat Assoc. 1993;88:1242e1249.
2. Fisher Perezgonzalez JD. Neyman-Pearson or NHST? A tutorial for teaching data
testing. Front Psychol. 2015;6:223. https://doi.org/10.3389/fpsyg.2015.00223.
3. Lindley DV Introduction to Probability and Statistics from a Bayesian Viewpoint,
Part 1: Probability; Part 2: Inference. Cambridge: Cambridge University Press;
1965.
4. Wald A. Statistical Decision Functions. New York, NY: Wiley; 1950.
5. Fisher RA. Statistical Methods for Research Workers London. Oliver & Boyd; 1925.
6. Nickerson RS. Null hypothesis significance testing: a review of an old and
continuing controversy. Psychol Methods. 2000;5:241e301.
7. Gigerenzer G. Mindless statistics. J Soc Econ. 2004;33:587e606.
8. Gelman A, Stern HS. The difference between ‘significant’ and ‘not significant’ is
not itself statistically significant. Am Statistician. 2006;60:328e331.
9. Goodman SN. A dirty dozen: Twelve P-value Misconceptions. Semin Hematol.
2008;45:135e140.
10. Cumming G. Replication and p intervals: p values Predict the future only
vaguely, but confidence intervals do much better. Perspect Psychol Sci. 2008;3:
286e300.
228 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229
11. Siegfried T. Odds Are, It's Wrong e science fails to face the shortcomings of
statistics. Sci News. 2010;177:26. https://wattsupwiththat.com/2010/03/20/.
12. Greenland S. Null misinterpretation in statistical testing and its impact on
Health risk assessment. Prev Med. 2011;53:225e228.
13. Simmons JP, Nelson LD, Simonsohn U. False-Positive Psychology: undisclosed
flexibility in data collection and analysis allows presenting anything as sig-
nificant. Psychol Sci. 2011;22:1359e1366.
14. Lew MJ. Bad statistical practice in Pharmacology (and other basic biomedical
disciplines): you probably don't know P. Br J Pharmacol. 2012;166:1559e1567.
15. Greenland S. Non significance plus high power does not imply support for the
null over the alternative. Ann Epidemiol. 2012;22:364e368.
16. Cohen J. Things I have learned (so far). Am Psychol. 1990;45;1304e12.
17. Loannidis JPA. Why most published research findings are false. PLoS Med.
2005;2, e124.
18. Gelman A, Loken E. The statistical crisis in science [online]. Am Sci. 2014;102:
567e606.
19. Siegfried T. To make Science Better, Watch out for Statistical Flaws; 2014. Science
News Context Blog https://www.sciencenews.org/blog/context/make-science-
better-watch-out-statistical-flaws (Accessed on 18th April 2018).
20. Nuzzo R. P values, the 'gold standard' of statistical validity, are not as reliable as
many scientists assume. Nature. 2014;506:150e152.
21. Halsey LG, Curran-Everett D, Vowler SL, Drummond GB. The fickle p value
generates irreproducible results. Nat Methods. 2015;12:179e185.
22. Thomas LE, Pencina MJ. Do not over (P) value Your research article. JAMA
Cardiol. 2016;1:1055. https://doi.org/10.1001/jamacardio.2016.3827.
23. Chawla DS. P-value shake-up proposed. Nature. 2017;548:16e17.
24. Trafimow D. Editorial. Basic Appl Soc Psychol. 2014;36:1e2.
25. Trafimow D, Marks M. Editorial. Basic Appl Soc Psychol. 2015;37:1e2. https://
doi.org/10.1080/01973533.2015.1012991.
26. Greenland S, Senn SJ, Rothman KJ, et al. Eur J Epidemiol. 2016;31:337e350.
27. Wasserstein RL, Lazar NA. The ASA's statement on p-values: context, process,
and purpose. Am Statistician. 2016;70:129e133. https://doi.org/10.1080/
00031305.2016.1154108.
28. Bartolucci AA, Tendera M, Howard G. Meta-analysis of multiple primary pre-
vention trials of cardiovascular events using aspirin. Am J Cardiol. 2011;107:
1796e1801.
29. Sullivan GM, Feinn R. Using effect size- or why the P value is not enough. J Gard
Med Educ. 2012;4:279e282.
30. Lindley DV. A statistical paradox. Biometrika. 1957;44:187e192.
31. Bartlett MS. A comment on D.V. Lindley's statistical paradox. Biometrika.
1957;44:533e534.
32. Senn SJ. Two cheers for P-values. J Epidemiol Biostat. 2001;6:193e204.
33. Kuffner TA, Walker SG. Why are p-values controversial? Am Statistician. 2018.
https://doi.org/10.1080/00031305.2016.1277161.
34. Woolston C. A blueprint to boost reproducibility of results. Nature. 2014;513.
https://doi.org/10.1038/nature.2014.16222.
35. Mobley A, Linder SK, Braeuer R, Ellis LM, Zwelling LA. Survey on data repro-
ducibility in cancer research provides insights into our limited ability to
translate findings from the laboratory to the clinic. PLoS One. 2013;8. e63221.
36. Russell JF. If a job is worth doing, it is worth doing twice: researchers and
funding agencies need to put a premium on ensuring that results are repro-
ducible. Nature. 2013;496:7.
37. Bohannon J. Replication effort provokes praisedand 'bullying' charges. Science.
2014;344:788e789.
38. Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility
mean? Sci Transl Med. 2016;8:341ps12.
39. Editorial. Reality check on reproducibility. Nature. 2016;533:437. https://
doi.org/10.1038/533437a.
40. Goodman SN. A comment of replication, P-values and evidence. Stat Med.
1992;11:875e879.
41. Berger J. Are P-values reasonable measures of accuracy? In: Francis Manly BFJI,
Lam FC, eds. Pacific Statistics Congress. North-Holland: Elsevier; 1986.
42. Berer J, Sellke T. Testing a point null hypothesis: the irreconcilability of P-
values and evidence. J Am Stat Assoc. 1987;82:112e139.
43. Good I. Good Thinking: The Foundations of Probability and its Applications. Uni-
versity of Minnesota Press; 1983:129.
44. Pratt J. Bayesian interpretation of standard inference statements. J Roy Stat Soc
B. 1965;27:169e203.
45. Edwards W, Lindman H, Savage L. Bayesian statistical inference for psycho-
logical research. Psychol Rev. 1963;70:193e242.
46. Diamond G, Forrester J. Clinical trials and statistical verdict: probable grounds
for appeal. Ann Intern Med. 1983;98:385e394.
47. Sterne JA, Davey Smith G. Sifting the evidence- What's wrong with significance
tests. BMJ. 2001;322:226e231.
48. Wacholder S, Chanock S, Garcia-Closas M, Elghormli L, Rothman N. Assessing
the probability that a positive report is false: an approach for molecular
epidemiology studies. J Natl Cancer Inst. 2004;96:434e442.
49. Risch NJ. Searching for genetic determinants in the new millennium. Nature.
2000;405:847e856.
50. Johnson VE. Revised standards for statistical evidence. Proc Natl Acad Sci Unit
States Am. 2013;110:19313e19317.
51. Hayden EC. Weak statistical standards implicated in scientific irreproducibility.
Nature. 2013. https://doi.org/10.1038/nature.2013.14131.
52. Hung HMJ, ONeill Bauer, Kohne P. The behavior of the P-value when the
alternative hypothesis is true. Biometerics. 1997;53:11e12.
53. Sackrowitz H, Samuel-Cahn EP. Values as random variables-expected P values.
Am Statistician. 1999;53:326e331.
54. Murdoch DJ, Yu-Ling Tsai Y, Adcock J. P-values are random variables. Am
Statistician. 2008;62:242e243.
55. Cumming G. Understanding the New Statistics: Effect Sizes, Confidence Intervals,
and Meta-analysis. New York: Routledge; 2012.
56. Cumming G. New statistics: why and how. Psychol Sci. 2014;25:7e29.
57. Boos DD, Stefanski LA. P-value precision and reproducibility. Am Statistician.
2011;65:213e221.
58. Lazzeroni LC, Lu Y, Beitskaya-Levy. P-values in genomics: apparent precision
masks high uncertainity. Mol Psychiatr. 2014;19:1336e1340.
59. Amrhein V, Korner-Nievergelt F, Roth T. The earth is flat (p > 0.05): significance
thresholds and the crisis of unreplicable research. Peer J. 2017;5. e3544 https://
doi.org/10.7717/peerj.3544.
60. Nosek BA, Spies JR, Motyl M. Scientific utopia II. Restructuring incentives and
practices to promote truth over publishability. Perspect Psychol Sci. 2012;7:
615e631.
61. John L, Loewenstein G, Prelec D. Measuring the prevalence of questionable
research practices with incentives for truth-telling. Psychol Sci. 2012;23:
524e532.
62. Open Science Collaboration. PSCHOLOGY. Estimating the reproducibility of
psychological science. Science. 2015;349. https://doi.org/10.1126/
science.aac4716.
63. Begley CG, Ellis LM. Drug development: raise standards for preclinical cancer
research. 483. 2012:531e533.
64. Marcia M. Reproducibility. Science. 2014;343:229.
65. Gadbury GL, Allison DB. Inappropriate fiddling with statistical analysis to
obtain a desirable p-value: tests to detect its presence in published literature.
PLoS One. 2012;7, e46363.
66. Begg CB, Berlin JA. Publication bias: a problem in interpreting medical data. J R
Stat Soc Ser A. 1988;151:419e463.
67. Fanelli D. Negative results are disappearing from most disciplines and coun-
tries. Scientometrics. 2012;90:891e904.
68. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. Publication and related
biases. Health Technol Assess. 2000;4:1e115.
69. Rajak HRBA, Ang JGE, Attal H, Allen JC. P-hacking in orthopaedic literature: a
twist to the tail bone joint. Surg Am. 2016;98. e91 https://doi.org/10.2106/JBJS.
16.00479.
70. Head ML, Holman L, Lanfear R, Kahn AT, Jennions MD. The extent and conse-
quences of P-hacking in science. PLoS Biol. 2015;13. e1002106.
71. Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. Circular analysis in
systems neuroscience: the dangers of double dipping. Nat Neurosci. 2009;12:
535e540.
72. Nieuwenhuis S, Forstmann BU, Wagenmakers EJ. Erroneous analyses of in-
teractions in neuroscience: a problem of significance. Nat Neurosci. 2011;14:
1105e1107.
73. Wagenmakers EJ, Wetzels R, Borsboom D, van der Maas HLJ. Why psycholo-
gists must change the way they analyze their data: the case of psi: comment on
Bem. J Pers Soc Psychol. 2011;100:426e432.
74. Bakker M, van Dijk A, Wicherts JM. The rules of the game called psychological
science. Perspect Psychol Sci. 2012;7:543e554.
75. Chambers CD. Registered reports: a new publishing initiative at Cortex. Cortex.
2013;49:609e610.
76. Francis G. Replication, statistical consistency, and publication bias. J Math
Psychol. 2013;57:153e169.
77. Bakker M, Wicherts JM. Outlier removal, sum scores, and the inflation of the
Type I error rate in independent samples t tests. The power of alternatives and
recommendations. Psychol Methods. 2014;19:409e427.
78. Simonsohn U, Nelson LD, Simmons JP. p-curve and effect size. Correcting for
publication bias using only significant results. Perspect Psychol Sci. 2014;9:
666e681.
79. Steegen S, Tuerlinckx F, Gelman A, Vanpaemel W. Increasing transparency
through a multiverse analysis. Perspect Psychol Sci. 2016;11:702e712.
80. van Aert RCM, Wicherts JM, van Assen MALM. Conducting meta-analyses based
on p-values: reservations and recommendations for applying p- uniform and
p-curve. Perspect Psychol Sci. 2016;11:713e729.
81. Wicherts JM, Veldkamp CL, Augusteijn HE, Bakker M, van Aert RC, van
Assen MA. Degrees of freedom in planning, running, analyzing, and reporting
psychological studies: a checklist to avoid p-hacking. Front Psychol. 2016;7:
1832.
82. Colquhoun D. The reproducibility of research and the misinterpretation of p-
values. R.Soc.Open Sci. 2017;4:171085. https://doi.org/10.1098/rsos.171085.
83. Benjamin D, Berger J, Johannesson M, et al. Redefine statistical significance.
Preprint on PsyArXiv http://osf.io/preprints/psyarxiv/mky9j. 2017; 2017.
84. Ioannidis JP, Tarone R, McLaughlin. The false-positive to false-negative ratio in
epidemiologic studies. J Epidemiol. 2011;22:450e456.
85. Benjamin DJ, Berger JO, Johnson VE, et al. Redefine statistical significance. Nat
Hum Behav. 2018;2:6e10.
86. Ioannidis JP. Viewpoint: the proposal to lower p-value thresholds to .005. J Am
Med Assoc. 2018;319:1429e1430.
87. Physorg Science News Wire. The problem with p values: how significant are
 R.L. Sapra, S. Nundy / Current Medicine Research and Practice 8 (2018) 222e229 229

they, really?. Available at http://phys.org/wire-news/145707973/the-problem-
with-p-values-how-significant-are-they-really.html; 2013.
88. Nuzzo R. Scientific method: statistical errors. Nature. 2014;506:150e152.
Available at http://www.nature.com/news/scientific-methodstatistical-errors-
1.14700.
89. Woolston C. Psychology journal bans P values. Nature. 2015;519. https://
doi.org/10.1038/519009f.
90. Wellek S. A critical evaluation of the current “p-value controversy”. Biom J.
2017;00:1e19.
91. Wasserstein R. American Statistical Association Seek “Post p < 0.05" Era.
Debunking Denialism; 2016. Available at https://debunkingdenialism.com/
2016/03/08/american-statistical-association-seek-post-p-0-05-era/.
92. Matthews R, Wasserstein R, Spiegelhalter D. The ASA's p-value statement, one
year on. Significance. 2017;14:38e41.