Acta Pædiatrica ISSN 0803-5253

REVIEW ARTICLE

Understanding type I and type II errors, statistical power and sample size
Anthony K. Akobeng (aakobeng@sidra.org)1,2
1.Sidra Medical and Research Centre, Doha, Qatar
2.Royal Manchester Children’s Hospital, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK

Keywords ABSTRACT
Power, Sample size, Type I error, Type II error The results of a clinical trial may be subject to random error because of the variability in the
Correspondence measured data, which arises purely by chance. There are two types of random error – type I
Anthony K Akobeng, Sidra Medical and Research error and type II error. In this study, type I and type II errors are explained, and the important
Center, PO Box 26999, Doha, Qatar.
Tel: +974 4012 5850 |
concepts of statistical power and sample size estimation are discussed.
Email: aakobeng@sidra.org Conclusion: The most important way of minimising random errors is to ensure adequate
Received sample size; that is, a sufficient large number of patients should be recruited for the study.
11 October 2015; revised 6 December 2015;
accepted 29 February 2016.

DOI:10.1111/apa.13384

INTRODUCTION Statistical hypothesis testing allows us to make inferences

The aim of a clinical trial was to gather information on the
benefits and risks of a treatment that would be applicable
to a target population of people with a certain disease (e.g.
babies with necrotising enterocolitis, children with
asthma). Practically, it is usually impossible to perform a
study on all the people in the target population, and
studies are performed on a sample of individuals drawn
from the population. It is then the hope that the results
obtained for the sample could be extrapolated to all
individuals in the population (1). For instance, when a
new treatment is found to be more effective than placebo
for treating acute exacerbations of asthma in individuals in
a research study, we assume that the results from the study
will be applicable to the whole population of people with
asthma.
However, the accuracy of the study results and, therefore,
the validity of our assumption are subjected to random
error, even if bias (systematic error) is excluded (1,2).
Random error is due to the variability in the measured data,
which arises purely by chance. Thus, even in a well-
designed, well-executed clinical trial, we cannot be certain
whether the results obtained accurately reflects the true
population results, that is whether the results in the study
are real or arose by chance as it is possible for random error
(the play of chance) alone to lead to an inaccurate estimate
of the treatment effect.
about the whole of the target population based on infor-
mation obtained from a sample of individuals from the
population by providing some information on how much
random error might be associated with a result. In other
words, hypothesis testing quantifies the evidence that the
collected data support a specified hypothesis about the
population (2).
HYPOTHESIS TESTING
Hypothesis testing is used to determine whether there is a
difference between two comparison groups in a study (3).
To do this, we initially make an assumption that there is no
difference between the groups. This assumption of ‘no
Key notes
 A type I error occurs when we wrongly conclude that
one treatment is more effective than another when, in
fact, it is not.
 A type II error occurs when we wrongly conclude that
there is no difference in treatment effects when, in fact,
there is a difference.
 An important way of minimising random errors is to
ensure adequate sample size.

©2016 Foundation Acta Pædiatrica. Published by John Wiley & Sons Ltd 2016 105, pp. 605–609 605
Type I and type II errors, power and sample size
difference’ is known as the null hypothesis (4). Following
statistical analysis, we either reject or fail to reject the null
hypothesis (see below). When the null hypothesis is
rejected, the alternate hypothesis (the opposite of the null
hypothesis – that there is a difference between the groups) is
accepted.
There are a number of statistical tests that may be used in
hypothesis testing, and these can be broadly classified as
parametric tests (e.g. t-test, ANOVA) or nonparametric
tests (e.g. Mann–Whitney U-test, Kruskal–Wallis test). The
choice of the correct test depends mainly on the nature of
the data and the study design. Whatever the test used, the
end result is to generate what is called a p value (or
probability value) (1). Being a probability, the p value can
take any value between 0 and 1.
The p value resulting from a statistical hypothesis test is
used to establish whether the sample data support the null
hypothesis or provide evidence of a difference as specified
by the alternative hypothesis (2). The p value therefore
represents the strength of evidence in support of the null
hypothesis. A large p value suggests that the sample data
support the null hypothesis, whereas a small p value
suggests they do not. By convention, the cut-off between a
large and a small p value is set at 0.05 (5%) (5). What this
means is that if the p value is less than 0.05, we say that the
evidence in favour of the null hypothesis is so small that we
will reject the null hypothesis and accept the alternative
hypothesis that there is a statistical difference between the
treatment groups; that is, the results are unlikely to have
occurred by chance. When the p value is below the cut-off,
the result is generally referred to as being ‘statistically
significant’. If the p value is equal or greater than 0.05, then
we say that the evidence in favour of the null hypothesis is
big enough for us to fail to reject the null hypothesis that
there is no difference between the groups; that is, the results
might have occurred by chance. In such a situation, we are
unable to demonstrate a statistical difference between the
groups and the result is referred to as ‘not statistically
significant’.
RANDOM ERRORS
As explained above, the aim of hypothesis testing is to get
some idea on whether any observed differences seen
between comparison groups are due to the treatment under
investigation or due to chance. When p values are used,
results are usually expressed as either ‘statistically signifi-
cant’ (unlikely to have occurred by chance) or ‘not
statistically significant’ (might have occurred by chance).
There are four possible conclusions that may be drawn
from these statistical results (6). These are illustrated in
Figure 1. As shown in this figure, two of the four possibil-
ities lead to correct conclusions whilst the other two
possibilities result in wrong conclusions (errors). In the
absence of bias (systematic error), random error is respon-
sible for these wrong statistical conclusions, which lead to
inaccurate estimates of the treatment effect. Two types of
random error can occur when hypothesis testing is
606 ©2016 Foundation Acta Pædiatrica. Published by John Wiley & Sons Ltd 2016 105, pp. 605–609
Akobeng
TRUTH
Difference No
difference
Type I error
RESULT OF Significant Correct
STATISTICAL
TEST
Not Type II error
significant Correct
Figure 1 Four possible conclusions in a clinical trial.
performed. These are referred to as type I (a) and type II
(b) errors (6–8).
Type I error
A type I error occurs when the null hypothesis of no
difference is rejected when, in fact, it is true. In such a
situation, we wrongly accept that there is a difference
between treatment groups (a ‘false-positive’ result) and
wrongly conclude that one treatment is more effective than
another when, in fact, it is not. The probability of making a
type I error (a) is equivalent to the cut-off value for
statistical significance (9). Thus, the chosen cut-off value
represents the probability of a type I error, that is the risk of
‘false positives’ that is accepted as being tolerable in the
study. What this means is that when a researcher uses the
conventional 0.05 (5%) as the threshold for statistical
significance, the probability of a type I error (a) is 5%
indicating that a positive conclusion based on finding a p
value <0.05 will be wrong 5% of the time. In other words, if
the study was repeated 20 times, the researchers would
accept that the results of 1 of those 20 replicates could be a
false-positive result (10). Although, by convention, 5% is
generally taken to be an acceptable type I error rate,
researchers may choose to use more stringent rates. For
instance, if the consequences of false-positive result are
serious, for example if the risk of harm associated with a
false-positive result is great, the researchers may decide to
reduce the tolerable type I error rate to, say, 1% (p value
<0.01).
There are a number of issues that could increase the
likelihood of a type I error in a study (11,12). Most of these
are related to multiple testing which occurs when research-
ers investigate several endpoints and perform more and
more statistical tests on the same data. The more tests are
performed on a set of data, the more likely that one or more
of them will yield a ‘statistically significant difference’ just
because of the play of chance. The problem of multiple
testing may also be encountered in other situations such as
performing secondary analyses not planned in the original
study design, performing multiple interim analyses of
accumulating data before all required patients have been
recruited and stopping trials earlier than planned (9,12).
There are three ways of minimising the problem of type I
errors (11). One way is to avoid multiple testing as
Akobeng
described above. The other way is to choose a more
stringent p value as the cut-off level for statistical signifi-
cance (say 0.01 instead of 0.05). This approach will,
however, likely to increase the risk of a false-negative result
(a type II error) [see below] unless the sample size of the
study is increased appropriately to take account of the new
alpha level. Another approach would be to refrain from
referring to results as being ‘statistically significant’ or ‘not
statistically significant’ (i.e. not using hypothesis testing)
and simply describing them as estimates with confidence
intervals (1).
Type II error
As shown in Figure 1, when the results of a study show ‘no
statistically significant’ difference between treatments, this
can mean two things: (i) that there is truly no difference
between the treatments (true negative), or (ii) that the study
has been unable to detect a difference because of the play of
chance (false negative) (11). When the second possibility
occurs, we say there has been a type II error. In other
words, a type II error occurs when we accept the null
hypothesis of no difference and conclude that there is no
difference in treatment effects when in fact, there is a
difference. In such a situation, we obtain a ‘false-negative’
result and may wrongly conclude that a treatment is not
effective when, in fact, it is.
The probability of making a type II error is referred to as
beta or b. By convention, b is usually set at 20% or 0.20,
which means that the researchers are willing to accept a
20% chance of wrongly concluding that there is no
difference between groups. If the consequences of a false-
negative conclusion are serious, the researchers may choose
to decrease the acceptable level of b to, say, 10% or 5% (10).
The understanding of type II errors is very important
especially as there are several studies in the medical
literature that firmly make conclusions of ‘no difference’
between interventions without due regard being paid to the
possibility of a type II error. Closely related to type II errors
is the concept of statistical power which is described below.
STATISTICAL POWER
In simple terms, the concept of statistical power (usually
referred to simply as ‘power’) refers to the ability of a
statistical test to detect a true difference between two
groups. In other words, it is a measure of the ability of the
test to identify that there is a difference between treatments
when such a difference truly exists. Power derives from the
type II error rate (b) and is mathematically the complement
of b, that is Power = 1 b (7).
As with the setting of the level of statistical significance (a),
the choice of b and therefore power (1 b) is arbitrary and the
value should be decided by the researchers prior to the trial
(6). By convention, a minimum power of 80% is generally
considered adequate (13), and in studies where power is
reported, researchers commonly choose a power of 80%
(0.8). When in a clinical trial, the statistical test is reported to
have a power of 80%, it means that the test has an 80%
©2016 Foundation Acta Pædiatrica. Published by John Wiley & Sons Ltd 2016 105, pp. 605–609
Type I and type II errors, power and sample size
probability of correctly detecting a difference between groups
when such a difference exists. Put in another way, it means
that the probability of the test not being able to detect such a
true difference (probability of making a type II error) is 20%.
When, as described under ‘Type I error’ (above), researchers
choose to set b as 10%, then the power of the study will be
90% which means that the researchers are willing to accept a
10% chance of wrongly concluding that there is no difference
between groups.
The power of a study is determined by several factors
including the frequency of the outcome being studied, the
magnitude of the effect, the study design and, more
importantly, the sample size of the study. For an RCT to
have a reasonable chance of answering the research
question it sets out to address, the sample size must be
large enough; that is, there must be enough participants in
the study.
SAMPLE SIZE
At the planning stage of a study, a calculation of sample size
(i.e. the number of patients needed for the trial) should be
performed. The calculation allows the determination of a
sample size that is large enough to have a good chance of
detecting a benefit of treatment, if such a benefit exists, but
which is also no larger than it needs to be (14).
Factors that affect sample size calculations
Four components are required to allow the calculation of an
adequate sample size (6,7,15). These are as follows:
 the type I error rate (a),
 the type II error rate (b) or statistical power (1 b),
 the smallest effect size of clinical interest,
 Nature of the data
o the variability of the outcome of interest (for contin-
uous outcomes)
o the expected frequency of the outcome of interest in
the control group (for dichotomous outcomes).
Type I error rate (a)
This is the same as the critical level of statistical significance
set by the researchers which is usually 5% (see above).
Type II error rate (b) or more commonly statistical power
(1-b)
The type II error rate (b) is usually set at 20%, and power
(1 b) at 80% (see above).
The smallest effect size of clinical interest
This refers to a treatment effect that the researchers are
interested in and is usually taken to be the smallest
difference in outcome between the two groups that is
considered to be clinically important. In other words, it is
the standard that the treatment should meet to be consid-
ered efficacious (10).
607
Type I and type II errors, power and sample size
Determining the minimum clinically important difference
is not always straightforward. Some researchers base this on
the results of previous similar studies or pilot work (16,17).
Clinical judgment may also be used to estimate the clinically
important difference, and one approach is to consider
differences that have led to the adoption of new therapies
for the condition of interest in the past. Another approach is
to survey experts and patients to determine what difference
in outcomes would need to be demonstrated for them to
adopt the new therapy in view of its costs and known risks
(17).
The smaller the clinically relevant difference, the more
patients are required for the study, and the bigger the
difference, the fewer patients are required. It must be noted
that if, to reduce the calculated sample size, the chosen
minimum clinically important difference is too big, then the
study will still be underpowered to detect smaller but
clinically important effects (18). If on the other hand, the
chosen minimum clinically important difference is too
small, then the required sample size may be unnecessarily
increased substantially.
Nature of the data
When the outcome of interest is continuous (e.g. blood
pressure, body mass index), sample size is affected by the
variability of the outcome (19,20). In general, the greater
the variability of the outcome variable, the larger the sample
size required to assess whether an observed effect is a true
effect. Variability, in this case, is estimated by means of the
standard deviation (20). When as is often the case, the
variability is unknown, the researchers may estimate it from
a previous similar study, pilot work or an estimate from
clinical observation.
For dichotomous outcomes (e.g. remission versus no
remission), the expected frequency of the outcome of
interest in the control group (event rate in the control
group) influences the number of patients required (6,7).
Again, the researchers may estimate this from a similar
study, pilot work or an estimate from clinical observation. If
the predicted frequency used for the study’s sample size
calculation is very different from the rate actually observed
in the study, the study may have been underpowered to
show a difference between treatments (12).
The importance of sample size
Prestudy calculation of sample size is very important. When
the sample size of a study is too small, it may be impossible
to detect any true differences in outcome between the
groups. Such a study might be a waste of resources and
potentially unethical. Frequently, however, a number of
small sized studies are published claiming no difference in
outcome between groups without the power of the studies
being reported. Right from the planning stage of a trial,
researchers should aim to recruit enough participants in
order to ensure that the study has a high probability of
detecting, as statistically significant, the smallest effect that
would be regarded as clinically important, and sample size
608
Akobeng
estimation and the related power calculation should be
reported (13).
The primary endpoint of the study, that is the clinical
feature considered to be the most relevant to the condition
and the treatment being studied should be used as the basis
for sample size calculations. It is also important to realise
that when the stated power of a study is based on the total
number of participants in the study, then any subgroup
analysis performed within the study may not have adequate
statistical power because the number of people in the
subgroups will be lower than the total sample size of the
study. One way of avoiding this problem is for researchers
(at the planning stage) to base their sample size calculation
on statistical power needed to detect the desired treatment
effect in the smallest anticipated subgroup (6).
CONCLUSIONS
The results of large studies are associated with less random
error whilst those of small studies may be subject to a
greater degree of random error. Therefore, for a study to be
able to properly answer the research question that it sets out
to answer, due consideration should be paid to the number
of patients that needs to be recruited into the study (sample
size). Researchers should report a-priori estimates of sample
size in clinical trial reports. Readers need to understand the
principles discussed in this study and be critical of studies
that do not report appropriate sample size calculations.
FUNDING AND CONFLICT OF INTERESTS
The author has no financial disclosures or conflict of
interests.
References
1. Akobeng AK. Confidence intervals and p-values in clinical
decision making. Acta Paediatr 2008; 97: 1004–7.
2. Sedgwick P. Pitfalls of statistical hypothesis testing: type I and
type II errors. BMJ 2014; 349: g4287.
3. Whitley E, Ball J. Statistics review 3: hypothesis testing and P
values. Crit Care 2002; 6: 222–5.
4. Last JM. A dictionary of epidemiology. New York: Oxford
University Press, 2001.
5. Akobeng AK. Understanding randomised controlled trials.
Arch Dis Child 2005; 90: 840–4.
6. Fletcher RW, Fletcher SW. Clinical epidemiology – the
essentials. Philadelphia: Lippincroft Williams & Wilkins, 2005.
7. Schulz KF, Grimes DA. Sample size calculations in randomised
trials: mandatory and mystical. Lancet 2005; 365: 1348–53.
8. Coggon D. Statistics in clinical practice. London: BMJ
Publishing Group, 1995.
9. Altman DG. Practical statistics for medical research. Boca
Raton: Chapman & Hall/CRC, 1991.
10. Ward MM. Primer: measuring the effects of treatment in
clinical trials. Nat Clin Pract Rheumatol 2007; 3: 291–7.
11. Keirse MJ, Hanssens M. Control of error in randomized clinical
trials. Eur J Obstet Gynecol Reprod Biol 2000; 92: 67–74.
12. Scales DC, Rubenfeld GD. Estimating sample size in critical
care clinical trials. J Crit Care 2005; 20: 6–11.
©2016 Foundation Acta Pædiatrica. Published by John Wiley & Sons Ltd 2016 105, pp. 605–609
Akobeng
13. Sedgwick P. Sample size: how many participants are needed in
a trial? BMJ 2013; 346: f1041.
14. Beck RW. Sample size for a clinical trial: why do some trials
need only 100 patients and others 1000 patients or more?
Ophthalmology 2006; 113: 721–2.
15. Kadam P, Bhalerao S. Sample size calculation. Int J Ayurveda
Res 2010; 1: 55–7.
16. Endacott R, Botti M. Clinical research 3: sample selection.
Intensive Crit Care Nurs 2005; 21: 51–5.
©2016 Foundation Acta Pædiatrica. Published by John Wiley & Sons Ltd 2016 105, pp. 605–609
Type I and type II errors, power and sample size
17. Brasher PMA, Brant RF. Sample size calculations in randomized
trials: common pitfalls. Can J Anaesth 2007; 54: 103–6.
18. Whitley E, Ball J. Statistics review 4: sample size calculations.
Crit Care 2002; 6: 335–41.
19. Noordzij M, Tripepi G, Dekker FW, Zoccali C, Tanck MW,
Jager KJ. Sample size calculations: basic principles and
common pitfalls. Nephrol Dial Transplant 2010; 25: 1388–93.
20. Noordzij M, Dekker FW, Zoccali C, Jager KJ. Sample size
calculations. Nephron Clin Pract 2011; 118: c319–23.
609