Hypothesis Testing & Statistical

Significance, Including Common

Mistakes to Avoid

“Statistically significant” is a widely used yet
commonly misunderstood phrase. Before
analyzing data and presenting statistical
results, make sure you understand what
statistical “significance” means and, equally
important, what it doesn’t mean.
How to use statistics for
hypothesis testing
Statistics is a powerful tool that can help scientists
decide whether or not a hypothesis is valid. But with
great power comes great responsibility, and it’s
important to use statistics wisely. Here we provide a
brief overview on how to use statistics for hypothesis
testing as well as guidance on the much-used and
often misunderstood term, “significant.”
ADVICE: Avoid the word “significant” so you
can precisely communicate what you mean
The word “significant” has two meanings in research:
• A P value is less than a preset threshold (often 0.05)
so you can reject the null hypothesis and state that
the results are deemed statistically significant.
• A difference (or ratio or correlation...) is large
enough that you think the result has biological (or
scientific or practical) relevance.
To avoid this ambiguity, don’t use the “S” word. If the
P value is less than a threshold, say so. If an observed
effect is large enough to be relevant, say so. The
word “significant” is never needed.
Accept or reject?
Much of the statistical reasoning we use today
was originally developed in the context of quality
control where you need a definite yes or no answer
from every analysis. Should you accept or reject the
batch? The logic used to obtain the answer is called
hypothesis testing.
ADVICE: Avoid the concept of “statistical
significance” (it’s only relevant in limited
situations)
The term “significant” is seductive and easy to
misinterpret because the use of the word in statistics
has a meaning entirely distinct from its use in everyday
conversation. Just because a difference is statistically
significant does not mean that it is biologically
or clinically important or interesting. Moreover, a
result that is not statistically significant (in the first
experiment) may turn out to be very important.
The use of “statistically significant” in hypothesis
testing makes sense in situations where you must
make a firm decision based on the results of one P
value. While this situation occurs in quality control,
where you need to decide whether to accept or reject
a specific batch, it doesn’t really happen in other
situations. For example, in clinical trials, decisions
are made based on several kinds of evidence, and
in basic research, decisions and conclusions are
typically made based on multiple experiments.
Thus, if you do not need to make a decision based
on one P value, then there is no need to declare a
result “statistically significant” or not. Simply report
the P value as a number, without using the term
“statistically significant”. Better yet, simply report
the confidence interval, without a P value.
Can you reject the null hypothesis and
declare a “statistically significant” difference
between sample and control?
Probably the easiest part of your statistical analysis
will be the following three steps to determine
whether or not you can reject the null hypothesis and,
if the context of your analysis merits it (see ADVICE:
Avoid the concept of “statistical significance” (it’s
only relevant in limited situations)) declare that
you’ve found a statistically significant difference
between the sample and the control.
1. Before you even begin your experiment, first
define a threshold P value.
Ideally, you should set the P value based on the
relative consequences of missing a true difference or
falsely finding a difference. In practice, the threshold
value (called alpha) is almost always set to 0.05, an
arbitrary value that has been widely adopted.
2. Next, define the null hypothesis.
When analyzing an experiment, the null hypothesis
is usually the opposite of the experimental
hypothesis. Your experimental hypothesis —
the reason you did the experiment — is that the
treatment changes the mean. The null hypothesis
is that the two populations have the same mean,
i.e. that the treatment has no effect.
3. Now, perform the appropriate statistical test to
compute the P value.
If the P value is less than the threshold, state
that you “reject the null hypothesis” and that the
difference is “statistically significant.”
If the P value is greater than the threshold, state
that you “do not reject the null hypothesis” and that
the difference is “not statistically significant”. You
cannot conclude that the null hypothesis is true.
All you can do is conclude that you don’t have
sufficient evidence to reject the null hypothesis.
But even if you find a “statistically significant” result,
what is the likelihood that this difference is actually
true? We discuss this topic in the next section.
Defining P value and alpha
The P value is a probability, with a value ranging from
zero to one, that answers the following question
(which you probably never thought to ask):
For an experiment of this size, if the populations
really have the same mean, what is the probability
of observing at least as large a difference between
sample means as was, in fact, observed?
Alpha is a threshold P value below which a difference
in means is judged “significant.” For an alpha set
to its typical value of 0.05, a result is said to be
statistically significant when a difference that large
(or larger) would occur less than 5% of the time if the
populations were, in fact, identical.
Population A and Population B have the same mean, but if a
scientist only sees the data indicated in purple (Population A) or
blue (Population B), the observed means will be very different from
the true mean of each population. The P value provides a value for
the probability that you will observe the purple and blue data points,
given a total population indicated by the gray data points.
Significant vs. not significant: a legal
analogy to guilty or not guilty
The statistical concept of “significant” vs. “not
significant”can be understood by comparing it to the
legal concept of “guilty” vs. “not guilty.”
In the American legal system (and much of the
world) a criminal defendant is presumed innocent
until proven guilty. If the evidence proves that the
defendant is guilty beyond a reasonable doubt,
the verdict is “guilty,” otherwise the verdict is “not
guilty”. In some countries, this verdict is “not proven”,
which is a better description. A “not guilty” verdict
does not mean that the judge or jury concluded that
the defendant is innocent -- it just means that the
evidence was not strong enough to persuade the
judge or jury that the defendant was guilty.
The concepts of “significant” and “not significant” in
statistical hypothesis testing is similar. In statistical
hypothesis testing you start with the null hypothesis,
which is usually that there is no difference between
groups. If the evidence produces a small enough P
value, you reject that null hypothesis and conclude
that the difference is real. If the P value is higher
than your threshold (usually 0.05), you don’t reject
the null hypothesis. This doesn’t mean the evidence
convinced you that the treatment had no effect,
only that the evidence was not persuasive enough to
convince you that there is an effect.
Interpreting low P values, the false discovery
rate, and how much faith to put into a
“statistically significant” result
The three steps discussed in the previous section
sound straightforward but it’s important to note
that you can’t interpret statistical significance or a
P value in a vacuum—your interpretation depends
on the context of the experiment, which affects the
false discovery rate (also called the false positive
rate). Depending on the context of your study, the
false discovery rate can be much higher than the
value of alpha, which means that even if your result
is statistically significant, it might still not be true.
Thus, interpreting results requires common sense,
intuition, and judgment. Here’s an example.
Imagine that you are screening a drug to see if it
lowers blood pressure. You use the usual threshold
of P<0.05 as defining statistical significance. Based
on the amount of scatter you expect to see and
the minimum change you would care about, you’ve
chosen the sample size for each experiment to have
80% power to detect the difference you are looking
for with a P value less than 0.05.
If you do get a P value less than 0.05, what is the
chance that the drug truly works?
The answer: It depends on the context of your
experiment.
Let’s start with the scenario where, based on the
context of the work, you estimate there is a 10% chance
that the drug actually has an effect. What happens when
you perform 1000 drugs that have a 10% chance of
having an effect (Table 1)?
Given your 10% estimate, you would expect 100 drugs
that work and 900 drugs that don't work. But these are
what you expect. Given these expectations, what
would you observe?
To understand what you would observe, let’s focus on
the column that covers the 10% probability that a drug
actually works. Since the power of your study is 80%,
you expect only 80% of the 100 drugs that actually work
to yield a P value less than 0.05, so the upper left cell is
80 drugs (i.e. 100 * 80%).
Now let’s look at the column that covers the 90%
probability that a drug does not actually work. Since
you set the definition of statistical significance to 0.05,
you expect 5% of the 900 drugs don't actually work to
yield a P value less than 0.05, so the upper right cell is
45 drugs (i.e. 900 * 5%).
Table 1. Reality vs observation – how often does the P value deliver false positive and false negative results?*

EXPECTED

REALITY: Drugs work REALITY: Drugs don't TOTAL NUMBER OF

work DRUGS

P<0.05
A “significant” 80 45 125
OBSERVED

effect is observed

P>0.05
“no significant” 20 855 875
effect is observed

TOTAL NUMBER OF DRUGS 100 900 1,000

*In this scenario, a drug is estimated to have a 10% chance of actually having an effect, our sample size is
chosen to have an 80% power of observing the expected difference with a P value less than 0.05, and we
evaluate 1,000 drugs with a 10% probability of having an effect.

Adding up the first row, you find that 125 drugs yield a
"statistically significant" result, however only 80 of
these drugs actually work. The other 45 drugs yield a
"statistically significant" result when the reality is that
they don’t actually work – these results are false
positives or false
discoveries. The false discovery rate (abbreviated
FDR) is 45/125 or 36%. Not 5%, but 36%. This is also
called the False Positive Rate (FPR).
Table 2, from chapter 12 of Essential Biostatistics1,
shows the FDR for this and three other scenarios.

Table 2. FDR for three different scenarios.

FDR FOR
PRIOR PROBABILITY FDR FOR P<0.05 0.045 < P < 0.050

Comparing randomly assigned groups

0% 100% 100%
in a clinical trial prior to treatment

Testing a drug that might possibly work 10% 36% 78%

Testing a drug with 50:50 chance

50% 6% 27%
of working

Positive controls 100% 0% 0%

Each row in the table above is for a different
scenario defined (before collecting data) by a
different prior probability of there being a real
effect. The middle column shows the expected
FDR (also called FPR) as calculated above. This
column answers the question: “If the P value is less
than 0.05, what is the chance that there really is
no effect and the result is just a matter of random
sampling?” Note this answer is not 5%. The FDR is
quite different than alpha, the threshold P value
used to define statistical significance.
The right-most column, determined by simulations,
asks a slightly different question based on work by
Colquhoun2,3: “If the P value is just a little bit less
than 0.05 (between 0.045 and 0.050), what is the
chance that there really is no effect and the result is
just a matter of random sampling?” These numbers
are much higher. Focus on the third row where the
prior probability is 50%. In this case, if the P value
is just barely under 0.05 there is a 27% chance that
the effect is due to chance. Note: 27%, not 5%! And
in a more exploratory situation where you think the
prior probability is 10%, the false discovery rate
for P values just barely below 0.05 is 78%. In this
situation, a statistically significant result (defined
conventionally) means almost nothing if you're
trying to minimize false positives.
Which brings us back to the question of how to
interpret a low P value and what a statistically
significant finding means.
The bottom line is:
Even if you have a low P value and, thus, a statistically
significant difference between your control and your
sample, there are many situations where a high false
discovery rate/false positive rate can render that
result close to meaningless. You have to use
judgment, intuition, common sense, and your
scientific acumen to interpret your results.
More Mistakes to Avoid
ADVICE: Don’t P-Hack, it can introduce bias
into your results leading to incorrect
conclusions
Statistical results can only be interpreted at face
value when every choice in data analysis was
performed exactly as planned and documented as
part of the experimental design. If you adjust and re-
analyze your data after your initial analysis you can
introduce bias, leading to incorrect conclusions.
Here’s an example of a problematic data analysis
workflow: you collect and analyze data, are unhappy
with the result and then add additional data and
re-analyze; remove a few outliers; transform to
logarithms; try a nonparametric test; redefine
the outcome by normalizing (say, dividing by each
animal’s weight); use a method to compare one
variable while adjusting for differences in another;
the list of possibilities is endless. Keep trying until
you obtain a statistically significant result or until you
run out of money, time, or curiosity.
The problem with the approach above is that if there
really is no difference or no effect, the chance of
finding a “statistically significant” result will still exceed
5%, sometimes by a large amount. Therefore, if you
only collect more data or analyze the data differently
when the P value is greater than 0.05, your results
will be biased. If the P value was less than 0.05 in
the first analysis, it might be larger than 0.05 after
collecting more data or using an alternative analysis.
However, you’d never see this outcome because you
only collected more data or tried different data analysis
strategies when the first P value was greater than 0.05
(see the next section for a more detailed discussion).
The term P-hacking was coined by Simmons et al4 who
also use the phrase, “too many investigator degrees
of freedom.” This is a general term that encompasses
dynamic sample size collection, HARKing, and more.
There are three kinds of P-hacking:
• The first kind of P-hacking involves changing the
actual values analyzed. Examples include ad hoc
sample size selection, switching to an alternate
control group (if you don’t like the first results and
your experiment involved two or more control
groups), trying various combinations of independent
variables to include in a multiple regression (whether
the selection is manual or automatic), trying
analyses with and without outliers, and analyzing
various subgroups of the data.
• The second kind of P-hacking is reanalyzing a
single data set with different statistical tests.
Examples: Trying parametric and nonparametric
tests; analyzing the raw data, then analyzing the
logarithms of the data.
• The third kind of P-hacking is the garden of
forking paths5. This happens when researchers
perform a reasonable analysis given their
assumptions and their data, but would have done
other analyses that were just as reasonable had the
data turned out differently.
Exploring your data can be a very useful way
to generate hypotheses and make preliminary
conclusions. But all such analyses need to be clearly
labeled, and then retested with new data.
ADVICE: Don’t HARK
(Hypothesizing after the result is known)
Hypothesizing After the Result is Known (HARKing,
Kerr 1998) is when you analyze the data many
different ways (say different subgroups), discover an
intriguing relationship, and then publish the data so
it appears that the hypothesis was stated before the
data were collected.
ADVICE: Don’t keep adding subjects until you
hit significance, you’ll get misleading results
The problem with continuing to add subjects until
you hit significance is the same as the previous point
about p-hacking—you’ll introduce bias into your
results, leading you to come to incorrect conclusions.
The best approach is to calculate what your sample
size should be to see the difference between means
that you expect at the alpha that you define as
significant, and then stick with that sample size.
Here’s a simulation that demonstrates the problem.
Three different simulations that demonstrate the hazards of adding
subjects until data reach statistical significance (indicated by the
green area adjacent to the x-axis).
Here we simulated data by drawing values from a
Gaussian distribution (mean=40, SD=15, but these
values are arbitrary). Both groups were simulated
using exactly the same distribution. We picked N=5
in each group and computed an unpaired t test and
recorded the P value. Then we added one subject
to each group (so N=6) and recomputed the t test
and P value. We repeated this until N=100 in each
group. Then we repeated the entire simulation three
times. These simulations were done by comparing
two groups with identical population means, so any
“statistically significant” result we obtain must be a
coincidence — a Type I error.
Experiment 1 (green) reached a P value of less than
0.05 when N=7, but the P value is higher than 0.05 for
all other sample sizes. Experiment 2 (red) reached a
P value of less than 0.05 when N=61 and also when
N=88 or 89. In Experiment 3 (blue) the curve hit a P
value of less than 0.05 when N=92 to N=100.
If we followed the sequential approach, we would
have declared the results in all three experiments to
be “statistically significant”. We would have stopped
when N=7 in the first (green) experiment, so would
never have seen the dotted parts of its curve. We
would have stopped the second (red) experiment
when N=6, and the third (blue) experiment when
N=92. In all three cases, we would have declared the
results to be “statistically significant”.
Since these simulations were created for values
where the true mean in both groups was identical,
any declaration of “statistical significance” is a Type
I error (rejection of a true null hypothesis). If the null
hypothesis is true, i.e. the two population means
are identical, we would expect to see this kind of
Type I error in 5% of experiments (if we use the
traditional definition of alpha=0.05 so P values less
than 0.05 are declared to be significant). But with
the sequential approach used in the simulations,
all three of our experiments resulted in a Type I
error. If you extended the experiment long enough
(infinite N) all experiments would eventually reach
statistical significance. Of course, in some cases
you would eventually give up even without “statistical
significance”. But this sequential approach will
produce “significant” results in far more than 5% of
experiments, even if the null hypothesis were true,
rendering this approach invalid.
Bottom line: It is important that you choose a sample
size and stick with it. You’ll fool yourself if you stop
when you like the results but keep going when you
don’t. The alternative is using specialized sequential
or adaptive methods that take into account the fact
that you analyze the data as you go. To learn more
about these techniques, look up “sequential” or
“adaptive” methods in advanced statistics books.
REFERENCES
1
Motulsky HJ. Essential Biostatistics: A nonmathematical
approach. Oxford University Press; 1st edition (June 30,
2015). ISBN: 978-0199365067. www.essentialbiostatistics.
com.
2
Colquhoun, D. An investigation of the false discovery rate
and the misinterpretation of p-values. Royal Society Open
Science. 2014. 1(3):140216–140216. http://doi.org/10.1098/
rsos.140216.
3
Colquhoun, D. The False Positive Risk: A Proposal
Concerning What to Do About p-Values. The American
Statistician. 2019. 73:supplement 1.
4
Simmons, J. P., Nelson, L. D., & Simonsohn, U. False-
positive psychology: undisclosed flexibility in data
collection and analysis allows presenting anything as
significant. Psychological Science 2011. 22(11):1359–1366.
5
Gelman, A., & Loken, E. The garden of forking paths: Why
multiple comparisons can be a problem, even when there
is no “fishing expedition” or ‘p-hacking’ and the research
hypothesis was posited ahead of time. (2013). Unpublished
as of Jan. 2016.