Hypothesis Testing and Confidence Intervals
T R Konold and X Fan, University of Virginia, Charlottesville, VA, USA
ã 2010 Elsevier Ltd. All rights reserved.
Hypothesis testing involves estimating the probability (p)
of observing a sample statistic (e.g., sample mean) that is
equal to or more extreme than what is obtained from a
sample, assuming that the sample is drawn from a popu-
lation with a hypothesized population parameter value
(e.g., population mean, m). This article provides an over-
view of hypothesis testing through both point and interval
estimation approaches. Although we use a population
mean in some examples, the general principles discussed
here extend to situations involving other sample statistics
(multiple sample means, variance, proportion, correlation
coefficient, etc.). Sampling distribution, sampling error,
point estimate, interval estimate, statistical errors, and
statistical power are among the topics discussed below.
Research Questions and Hypotheses
Research questions and hypotheses are usually based on a
researcher’s educated guess or intuition grounded in a good
understanding of the phenomenon of interest. For exam-
ple, a teacher might ponder whether a new instructional
approach has an effect on student achievement as measured
on a state-wide assessment of learning (the research ques-
tion). Based on his/her pedagogical content knowledge,
the teacher may generate a research hypothesis that this
new instructional approach does impact student achieve-
ment. Although this research hypothesis does not indicate
the direction of influence, such qualifications can easily be
incorporated to reflect whether the researcher believes
that this instructional approach will increase or decrease
measured achievement. In general, a research hypothesis
reflects a researcher’s predictions and it directly shapes
the statistical hypotheses to be empirically evaluated.
Statistical Hypothesis
Null Hypothesis
In order to test a research hypothesis, it is necessary to
translate it into a statistical hypothesis. A statistical
hypothesis is a numerical statement regarding a hypothe-
sized population parameter. For example, knowing that
the average state-wide assessment score for the general
student population is 72 raw score points, a researcher
might state the null hypothesis (H0) as
H0 : m ¼ 72
216
The H0 is often stated to reflect the general state of affairs,
or the condition of no difference. In hypothesis testing,
assuming that the null hypothesis is true, we evaluate the
probability of observing a sample statistic (e.g., sample
mean) equal to or more extreme than what was obtained
from our sample drawn from the hypothesized population
with the population value (population parameter) stated
in the H0 (i.e., m ¼ 72). In most substantive applications,
researchers hope to demonstrate that this probability is
small, and the observed sample estimate is unlikely to
have arisen from a population as hypothesized in the H0.
Although this example concerns a hypothesis about a
population mean m, hypotheses concerning any popula-
tion parameters (e.g., variance, correlation coefficient, and
proportion) are essentially the same. So we may express
H0 in a generic form:
H0 : y ¼ K
where y is the population parameter under investigation,
and K is the hypothesized value of that parameter.
Directional and Nondirectional Alternative
Hypotheses
Hypothesis testing involves two statistical hypotheses.
The first is the null hypothesis (H0) as described above.
For each H0, there is an alternative hypothesis (Ha) that
will be favored if the null hypothesis is found to be
statistically not viable. The Ha can be either nondirec-
tional or directional, as dictated by the research hypothe-
sis. For example, if a researcher only believes the new
instructional approach will have an impact on student test
scores, but is unsure whether the effect will be positive or
negative, the null and alternative hypotheses would be
H0: m ¼ 72
Ha: m 6¼ 72
Here, Ha reflects the researcher’s uncertainty regarding
the directionality, and it allows for a statistical test that
considers both possibilities that the new instructional
approach could increase test scores or decrease test scores.
This is commonly referred to as a nondirectional alterna-
tive hypothesis, and is also referred to as a two-tailed test
for reasons that are described below.
A directional alternative hypothesis, on the other hand,
is useful to accommodate the researcher’s prediction that,
for example, the new instructional approach will decrease

test scores (Ha: m < 72 bpm) or will increase test scores
(Ha: m > 72). A directional alternative hypothesis is often
referred to as a one-tailed test as described below. It is
important to note, however, that for every specified H0
there will be a single Ha that may assume one of the three
forms
Ha: y 6¼ K
Ha: y < K
Ha: y > K
Sample, Population, and Inferential
Statistics
A sample is a subset of a population of interest that is
defined by the researcher. For example, one could con-
ceivably collect math achievement data on all fifth graders
in the United States. However, given the limited re-
sources, it is more realistic and economical to work with
a subset of this group (i.e., sample), with the goal of
generalizing the findings to the total population. It is
possible, or even very common, that the population of
interest is a statistical population that does not really
exist. For example, our educator interested in the effec-
tiveness of a new instructional approach may statistically
compare two student samples: one sample receiving this
new approach, and the other receiving the business-
as-usual approach. In this situation, there are two hypo-
thetical statistical populations: one population of students
taught under this new approach, and another population
of students taught under the business-as-usual instruc-
tional approach. In reality, we do not really have these two
populations.
A sample statistic y^ is a measure of a sample character-
istic (sample mean, proportion, correlation, etc.), whereas
a measure of the corresponding population characteristic
is a population parameter (y). Researchers are typically
interested in understanding the characteristics of a popu-
lation, but it is often impossible or impractical to obtain
data from the whole population (e.g., lack of resources to
do so; or a hypothesized statistical population). Inferential
statistics are useful for estimating population character-
istics (i.e., parameters), and for generalizing the sample
findings to the population from which the sample was
drawn.
Hypothesis testing is at the center of inferential statis-
tics. Both point estimation and interval estimation are
commonly used first steps in the hypothesis testing pro-
cess. In point estimation, a single population parameter
value is hypothesized and used in hypothesis testing. By
contrast, interval estimation serves as a useful supplement
to point estimation because it provides a likely range of
values for the unknown population parameter. The key to
Hypothesis Testing and Confidence Intervals 217
both point and interval estimation is in estimating the
sampling variability of a given sample statistic.
Sampling Distribution
In hypothesis testing, sampling variability (i.e., sampling
error) of a sample statistic y^ is the foundation for esti-
mating the probability of observing a sample statistic
equal to or more extreme than what is obtained under
the true null hypothesis. A statistic’s sampling distribution
provides the estimation for sampling error. For example,
assume we are interested in the average performance on a
math achievement exam (parameter) of all fifth graders in
the United States (population). Our best estimate of the
unknown population mean parameter (m) would be the
sample average (X ) based on a random sample of n ob-
servations from this population. If this process of drawing
random samples of size n from the same population were
repeated many times, the means from all the samples
would vary, and these means themselves would form a
distribution, which is called a sampling distribution of the
mean. This sampling distribution concept also extends to
other sample statistics (e.g., sample variance, proportion,
and correlation).
This sampling distribution captures the sample-to-
sample variability of a sample statistic. In the case of X ,
theoretically, the average of all possible sample means, or
the expected value of the statistic, E(X ), is equal to the

 mean (i.e., m). Statistics with this property, that
population
is, E y^ ¼ y, are referred to as unbiased estimators.
Standard error
Sample-to-sample variability of a statistic, or sampling
error, can be quantified by standard error. In the case of
sample means as described above, the standard error of
the mean quantifies the variability of the sampling distri-
bution of the mean. When samples are randomly selected
from a population, the resulting distribution will be ap-
proximately normally distributed for reasonable sample
size of n (see the subsection titled ‘Central limit theorem’
below). When the population standard deviation (s) is
known, the standard deviation of the sampling distribu-
tion of means, usually referred to as standard error of the
mean, is:
s
sX ¼ pffiffiffi
n
As is obvious, sX is a function of the sample size (n). As n
increases, sX decreases to its lower limit of 0 (i.e.,
n ! 1; sX ! 0). Likewise, as sample size decreases, sX
increases to its upper limit of the population standard
deviation s (i.e., n ! 1; sX ! s). In general, over
repeated sampling, sample statistic y’s^ based on a larger
218 Statistics
sample size has less variability (i.e., less sampling error)
than those based on a smaller sample size.
Central limit theorem
Several properties of sampling distributions that have
important implications for hypothesis testing are captured
in the central limit theorem (CLT). In a nutshell, CLT states
the following. First, as the size (n) of the samples drawn
from a population increases, the standard error of the means
decreases, leading to less variability of sample means and
smaller sampling error. Figure 1 presents two sampling
distributions of the means, one based on n ¼ 10, and the
other based on n ¼ 30. As shown in Figure 1, the distri-
bution based on a smaller sample size (n ¼ 10) spreads out
more than the one based on a larger sample size (n ¼ 30).
Second, the shape of the sampling distribution of the
mean becomes increasingly normal as the sample size
increases. This is true regardless of the shape of the parent
population from which the samples are drawn. As a result,
 can generally be consid-
the sampling distribution of X
ered as being normally distributed for samples of size
n ¼ 30 or greater, even though the parent population
may not be normally distributed.
Sampling distribution as probability distribution
Hypothesis testing involves estimating the probability of
^ equal to or more extreme
observing a sample statistic (y)
n = 10
n = 30
Figure 1 Two sampling distributions of the mean from the
same population: n ¼ 10 and n ¼ 30.
than what is obtained from a sample, assuming true H0.
For this purpose, the sampling distribution of the statistic
developed under H0 serves as the probability distribution.
Under this probability distribution, the closer the observed
) is to the population parameter
sample statistic (e.g., X
(e.g., m ¼ K) under H0, the more likely that the difference
 and m may have been the result of sampling
between X
variation (i.e., sampling error). Sample statistic values near
the tails of the sampling distribution represent a greater
difference between the sample statistic and the population
parameter under H0. These values occur less frequently in
repeated sampling, and consequently, the probability of
observing the sample statistic values near the tails is smal-
ler. Under H0, if this probability is judged to be smaller
than a predetermined threshold probability level (denoted a),
we will conclude that H0 is likely not true; if it were, it
would be very unlikely to observe the sample statistic.
Conventionally, a ¼ 0.05 is often used as such a threshold,
leading to the following decision rules with regard to H0:


^ 0  a ¼ 0:05 ! reject H0
If p yjH


^ 0 > a ¼ 0:05 ! fail to reject H0
If p yjH
This rule implies that when the observed sample statistic
value becomes so rare under the H0 sampling distribution
(e.g., p < 0.05), we would be willing to conclude that the
value specified in the H0 is not a plausible population
value for the observed sample statistic. It should be
noted that this rule does not suggest that such sample
statistic values cannot occur under H0; it only states that
values so extreme occur less frequently under H0. More-
over, there is nothing magical about the conventional
a ¼ 0.05 decision rule: researchers may choose other values
for evaluating H0, based on their considerations about
type I and type II errors (described below). It is important
to note that, given the observed sample statistic, hypothe-
sis testing is unable to address the question of whether H0
is true. The process only involves the determination of the
probability of obtaining sample statistic values equal to or
more extreme than the sample result under the assump-
tion that H0 is true (Cohen, 1994).
Test Statistics and Probability Estimates
In hypothesis testing, to assess the probability of observ-
ing values more extreme than a given sample statistic
under H0, we translate the observed sample statistic value
into a test statistic. For example, the test statistic for
evaluating a single sample mean (X ) when the population
standard deviation (s) is known is:
 m
X
Zobs ¼
s
X

df = 5
Figure 2 Three t-distributions with df ¼ 5, df ¼ 15, and df ¼ 1.
This zobs test statistic represents the distance between the
sample mean and hypothesized population mean in stan-
dard deviation units, where the standard deviation is the
standard deviation of the sampling distributionpof ffiffiffi the
mean (i.e., standard error of the mean: ðsX ¼ ðs= nÞ.
A larger absolute value of zobs is further away from the
hypothesized population mean. The exact probability of
observing a value of zobs or larger can be obtained from a
normal distribution table.
Application of the z test statistic requires that the
population standard deviation s be known. When s is
unknown and must be estimated with the sample standard
deviation (s), the estimated standard error of the mean is
sX ¼ ps ffiffin, and the test statistic becomes:
 m
X probability-based decision rule for testing H0:
tobs ¼
SX
In contrast to the single z distribution, there exists a
family of t distributions, as defined by degrees of freedom
(df ) that are based on sample size (df ¼ n–1). Figure 2
presents three t distributions with different df (df ¼ 5, 15,
or 1). Because the t distributions have different shapes,
probabilities beyond a given t value depend not only on
the value of t, but also on the df on which that statistic is
based. As sample size n increases, t distributions converge
on the z distribution. Beyond df ¼ 120, the difference
between z and t distributions is negligible, and a t-test
practically becomes a z-test.
Test statistics (e.g., F statistic, w2 statistic) other than z
and t may be needed as required by a statistical analysis.
Regardless of the specific test statistic used in a research
situation, the logic and procedure described here for
hypothesis testing remain the same.
Decision Rules Based on Test Statistic
The discussion above focused on testing H0 by comparing
the probability of obtaining sample statistic values more
extreme than a given sample statistic, assuming the true
Hypothesis Testing and Confidence Intervals 219
df = 15
df = ⬁
H0, against a predetermined threshold probability level
(i.e., a ¼ 0.05). In practice, this is done through a direct
comparison of the observed sample test statistic (e.g., zobs,
tobs, Fobs) against the critical value of the statistic (e.g., zcv,
tcv, or Fcv) as determined by the predetermined threshold
probability level a. For example, the critical value of
t (i.e., tcv) associated with a given value of a and a given
df is readily available from a table of t distributions. By
comparing tobs against tcv, we can establish the following
decision rule for testing H0:
lf jtobs j  jtcv j ! reject H0
lf jtobs j < jtcv j ! fail to reject H0
This decision rule is a direct translation of our previous


^ 0  a ! reject H
lf p yjH 0


^ 0 > a ! fail to reject H
lf p yjH
0
For example, our educator researcher previously hypo-
thesized mean student achievement test score of 72 for
a population (H0: m ¼ 72), and she would like to test this
H0 against the nondirectional alternative hypothesis Ha:
m 6¼ 72 for a ¼ 0.05. From a random sample of 100
 ¼ 68, and s ¼ 8
participants, the researcher obtained X
(i.e., sX ¼ ps ffiffin ¼ 0:8). The test statistic is:
  m 68  72
X
tobs ¼ ¼ ¼ 5
s 0:8
X
As | tobs | ¼ 5 exceeds the tcv [t(.975, df ¼ 99) ¼ 2.28, a ¼ .05],
the H0: m ¼ 72 is rejected, as it is very unlikely ( p < 0.05)
to observe sample mean values equal to or greater than
 ¼ 68 if H0 were true.
X
Either a nondirectional or directional alternative
hypothesis may be evaluated. When a nondirection alternative
hypothesis is evaluated (i.e., Ha: m 6¼ K) , the overall a (e.g.,
a ¼ 0.05) is evenly divided into the two tails of the sampling
distribution of H0, to accommodate the possibility that the
direction of difference between X  and m can be either
220 Statistics
a/2
Figure 3 The a at distribution tails for nondirectional (two-tailed) and directional (upper tail) tests.
positive or negative, as shown in Figure 3 (two solid lines
at the tails).
When a directional alternative hypothesis is used (e.g.,
Ha: m > K ), we only consider the critical value at one tail
that corresponds to the hypothesized direction in Ha, and
the overall a is located at this tail, as shown in Figure 3
(dashed line at the right tail for Ha: m >K ). Nondirectional
and directional alternative hypotheses are often referred
to as two-tailed and one-tailed tests, respectively. For a
directional (one-tailed) test, because all a is located at one
tail, the critical value of the test statistic will be smaller in
absolute value than that in a nondirectional test. For example,
for a z-test with a ¼ 0.05, the critical values are 1.96 for
a two-tailed test (nondirectional test), and the critical value
for a one-tailed (directional) test (Ha: m > K) is only 1.65.
This makes it easier for a one-tailed test to reject H0, as
long as the test statistic is in the hypothesized direction.
Because of this, if the test statistic is in the hypothesized
direction, the one-tailed test has more statistical power (see
description of statistical power below) over the two-tailed test.
When the test statistic is not in the hypothesized direction,
however, the test automatically fails to reject the H0 .
Confidence Intervals in Hypothesis
Testing
The hypothesis testing approach described above is a
point-estimate approach that results in a dichotomous
decision (i.e., reject or fail to reject H0). On the other
hand, interval-estimation approach provides a range of
values within which a population parameter is likely to
reside. The importance of using confidence intervals (CIs)
in educational and psychological research has been
emphasized by the American Psychological Association
(APA) task force (Wilkinson and Task Force on Statistical
Inference, 1999), and such emphasis has been reflected in
a/2
a
the guidelines in the APA publication manual as best
practice in reporting empirical results (APA, 2001).
For a hypothesized population value y, using the sam-
^ and for a given a, the following CI can be
ple estimate y,
constructed:
100ð1  aÞC:I : ¼ y^  Tð1aÞ Sy^
2
(This is the classical definition of CI under true H0, where
the critical values from a central distribution (e.g., central
t distribution) are used to construct a symmetrical CI.
However, if a true H0 is not assumed, critical values from
a noncentral distribution (e.g., noncentral t distribution)
may be used for constructing the CI for the unknown
population parameter y, and this will result in a nonsym-
metrical CI. For more details, readers may consult, e.g.,
Cumming and Finch (2001), Steiger and Fouladi (1997).
where Tð1aÞ is the appropriate test statistic (e.g., z, t)
2
 
value at the 100 1  a2 percentile point of its distribu-
s ¼ps ffi
tion, and sy^ is the estimated standard error (i.e., y^ n )
of the sampling distribution. Adding and subtracting the
quantity of Tð1aÞsy^ from y^ results in the lower and upper

2 
limits of a CI y^  Tð1aÞ Sy^; y^ þ Tð1aÞ Sy^ that has
2 2
probability of 1 – a of containing the unknown popula-
tion parameter value y. If the hypothesized population
parameter value under H0 is outside this CI, it leads to
the rejection of H0. Conversely, if the CI contains the
parameter value under H0, H0 will not be rejected.
Using our previous example about a new instructional
method, the researcher obtained n ¼ 100, X  ¼ 68, and
s ¼ 8 (i.e., sX ¼ pffiffin ¼ 0:8). For a ¼ 0.05, t(.975, df ¼ 99) ¼
s
2.28, the 95% CI would be (66.18, 69.82). (Lower
limit is 68 – (2.28  0.8) ¼ 66.18; upper limit is 68 þ
(2.28  0.8) ¼ 69.82.) As this CI does not contain 72 under
H0, the H0 would be rejected. With a CI, we are not only

Y
m
Figure 4 Fifty 95% confidence intervals for population
parameter m.
able to evaluate the H0 (point estimation), but also able to
estimate a range of probable population parameter values
(interval estimation). The value of a CI is that, it provides
information not only about the sample-based point-estimate,
but also about the estimation error associated with the
statistic. In the social sciences, the resulting estimation
error may be large, and this may explain why such inter-
vals are not often reported (Cohen, 1994).
It is important to note that this 95% CI does not indicate
that we can be 95% confident that this interval contains
the population parameter (see Thompson, 2007: 427). Rather,
this is interpreted to mean that 95% of similarly con-
structed intervals will contain the population parameter,
while 5% of them will not. This is illustrated in Figure 4
for a hypothetical situation where 50 random samples
were drawn from a population with parameter m, and a
95% confidence interval was constructed from each sam-
ple. Out of the 50 CIs, 47 contained the population
parameter, and three did not.
Statistical Errors (Type I and Type II)
and Power
In hypothesis testing, two types of statistical errors may
occur: reject H0 when it is true (type I error), or fail to
reject H0 when it is false (type II error). On the other
hand, a correct decision is made when a true H0 is not
rejected, or when a false H0 is rejected. These possible
outcomes in hypothesis testing are shown in Figure 5.
Hypothesis Testing and Confidence Intervals 221
Decision based on sample data
Reject H0 Fail to reject H0
True state of nature
Type I error Correct decision
H0 true
a 1−a
Correct decision
Type II error
H0 false (power)
b
1−b
Figure 5 Hypothesis testing errors and correct decisions.
H0 Ha
Power
(1−b)
(1−a)
b (type II) a (type I)
Figure 6 Type I, type II errors and statistical power.
In a hypothesis test, once a decision is made with
regard to H0, either a type I error (when H0 is rejected),
or a type II error (when H0 is not rejected), is possible, but
not both. We cannot know with certainty whether the
decision is actually an error. When H0 is rejected, only a
type I error is possible. The probability for a type I error is
known, because it is directly controlled by the researcher
through the choice of the a level (e.g., a ¼ 0.05), which
represents the risk we are willing to take in concluding
that the H0 is false when it may be true. On the other hand,
when H0 is not rejected, only a type II error is possible.
Here, the probability for a type II error is unknown, but
can be estimated, and is generally not directly controlled
by the researcher. In Figure 5, when H0 is true (upper row),
the probability for a type I error is a, and the probability
of making a correct decision (i.e., not to reject the true H0)
is 1 – a. When H0 is false (lower row), the probability for a
type II error is b, and the probability of making a correct
decision (i.e., to reject the false H0) is the power of a
statistical test (1 – b).
Figure 6 graphically illustrates two sampling distribu-
tions under H0 and Ha, respectively, as well as the relation-
ships among the probabilities of type I and type II errors,
and the statistical power (probability to reject a false H0).
The sampling distribution under H0 represents all possible
values of a statistic, including those in the specified region
of a (vertically shaded area under H0 to the left of the
vertical dashed line) where a decision to reject H0 would
be made. This probability is often set at 0.05 to indicate that
there is a 5% chance of rejecting a true H0 (i.e., making
a type I error).
222 Statistics
Now consider the sampling distribution under Ha,
which overlaps with that under H0. Sample-based test
statistics in the type II error or b region (horizontally
shaded area under Ha to the right of the vertical dashed
line) would not lead to the rejection of H0, even though
these sample statistics may in fact belong to the sampling
distribution of Ha. As shown in Figure 6, statistical power
(i.e., the probability to reject a false H0) denoted by 1 – b,
is the area under Ha to the right side of the vertical dashed
line defined by a.
Education researchers may desire to have a reasonable
level of power (e.g., b  0.80) for a statistical test. One way
to increase statistical power is to increase a (e.g., from 0.05
to 0.10). Imagine the rejection line (vertical dashed line) in
Figure 6 sliding to the left to accommodate a higher a
level (i.e., higher probability of making a type I error), and
this would result in reducing type II error probability (b)
and in increasing statistical power (1 – b). Conversely,
decreasing a (e.g., from 0.05 to 0.01; sliding the rejection
line under H0 distribution to the right) will result in an
increase in type II error probability (b) and a decrease in
statistical power. Statistical power can also be increased by
using larger sample sizes. Other things being equal, larger
sample sizes lead to smaller standard errors of the distri-
bution. Consequently, there will be less overlap between
the distribution under H0 and that under Ha, thus a smal-
ler b, and more power (i.e., 1 – b is larger). Finally, an
increase in statistical power can also come by way of
greater treatment effects or greater separation between
the H0 and Ha distributions. Researchers need to under-
stand such interplay among type I error, type II error, and
statistical power, and consider the relative cost and conse-
quence of one type of error versus another in a given
substantive research application.
See also: Point Estimation Methods with Applications to
Item Response Theory Models; Sampling; Statistical
Power Analysis; Statistical Significance Versus Effect
Size.
Bibliography
APA (American Psychological Association) (2001). Publication Manual of
the American Psychological Association, 5th edn. Washington, DC:
Author.
Cohen, J. (1994). The earth is round (p < 05). American Psychologist
49, 997–1003.
Cumming, G. and Finch, S. (2001). A primer on the understanding, use,
and calculation of confidence intervals that are based on central and
non-central distributions. Educational and Psychological
Measurement 61, 532–574.
Steiger, J. H. and Fouladi, R. T. (1997). Noncentrality interval estimation
and the evaluation of statistical models. In Harlow, L. L., Mulaik, S. A.,
and Steiger, J. H. (eds.) What if there were no significance tests?,
pp 221–257. Mahwah, NJ: Erlbaum.
Thompson, B. (2007). Effect sizes, confidence intervals, and confidence
intervals for effect sizes. Psychology in the Schools 44, 423–432.
Wilkinson, L. Task Force on Statistical Inference, American
Psychological Association, Science Directorate (1999). Statistical
methods in psychology journals: Guidelines and explanations.
American Psychologist 54, 594–604.