Errors in Hypothesis Testing

Errors in Hypothesis Testing 𝑯𝟎 is true 𝑯𝟎 is False

Since we decide to reject or fail to the reject the
✔ ❌
null hypothesis on the basis of an extremely rare Fail to Reject
sample mean, there is a chance that we make an Null Hypothesis Type II Error
incorrect decision. That is, just because a sample ❌
✔
Reject Null
mean is extremely rare (or common) does not Hypothesis Type I Error
mean it is necessarily from the alternative (or
null) distribution.

Type I Error Type II Error

Rejected 𝐻 when it should have failed to be Failed to reject 𝐻 when it should have been
rejected. That means, we concluded an effect rejected. That means, we concluded there is no
exists when there wasn't one. effect when there was one.
In short, a false positive. In short, a false negative.

Example Example
A smoke alarm going off when there is no fire. A smoke alarm not going off when there is a fire.

Example Example
A person is not guilty but is convicted. A person is guilty but is acquitted.

Example VCAA 2016 Practice Exam 2 Question 6eii

A certain type of computer, once fully charged, is claimed by the manufacturer to have 𝜇 = 10 hours
lifetime before recharge is needed. When checked, a random sample of 𝑛 = 25 such computers is
found to have an average lifetime of 𝑥̅ = 9.7 hours and a standard deviation of 𝑠 = 1 hour.

To decide whether the information gained from the sample is consistent with the claim 𝜇 = 10, a
statistical test is to be carried out. Assume that the distribution of lifetimes is normal and that 𝑠 is
sufficiently accurate estimate of the population (of lifetimes) standard deiation 𝜎.

𝐻 : 𝜇 = 10, 𝐻 : 𝜇 < 10, The 𝑝-value for this test is 0.067.

At the 5% level of significance, we failed to reject 𝐻 as the 𝑝-value 0.067 > 0.05

If the mean lifetime of all computers is in fact 𝜇 = 9.5, then the null hypothesis was not rejected when
it should have been. That is, a type II error was made.

Level of Significance and the Chance of Making Errors

The level of significance is also the threshold probability of committing a type I error since we reject the
null hypothesis for any probability in that range. That means, we can decide how often we are willing
to make a type I error. The trade off is that we affect how often we will make a type II error instead.

• 𝛼% level of significance means that 𝛼% of the time the 𝐻 is rejected when it is actually true.
• 5% level of significance means that 5% of the time the 𝐻 is rejected when it is actually true.

Decreasing the level of significance decreases the chance of making type I errors but increases chance
of making type II errors since the tails become smaller.

Conversely, increasing the level of significance increases the chance of making type I errors but
decreases the chance of making type II errors since the tails become larger.

Increasing/decreasing the sample size will increase/decrease the level of significance.

Probability of Making a Type II Error
To determine the chance of making a type II error, that is failing to reject the null hypothesis when the
it should have been rejected, we need to determine the set of means that we would reject the null
hypothesis with for that level of significance (using the original mean). Then, using the true mean,
determine the probability for the complementary set that would fail to reject the null hypothesis.
That probability is the chance you make a type II error.

Example VCAA 2019 NHT Exam 2 Question 6

A paint company claims that the mean time taken for its paint to dry when motor vehicles are repaired
is 3.55 hours, with a standard deviation of 0.66 hours. Assume that the drying time for the paint
follows a normal distribution and that the claimed standard deviation value is accurate.

Let the random variable 𝑋 represent the mean time taken for the paint to dry for a random sample of
36 motor vehicles.

At a crash repair centre, it was found that the mean time taken for the paint company’s paint to dry on
36 randomly selected vehicles was 3.85 hours. The management of this crash repair centre was not
happy and believed that the claim regarding the mean time taken for the paint to dry was too low.

To test the paint company’s claim, a statistical test was carried out.
Suitable null and alternative hypotheses 𝐻 and 𝐻 respectively to test whether the mean time taken
for the paint to dry is longer than claimed are 𝐻 : 𝜇 = 3.55, 𝐻 : 𝜇 > 3.85

e. At the 1% level of significance, find the set of sample mean values that would support the conclusion
that the mean time taken for the paint to dry exceeded 3.55 hours.
Give your answer in hours, correct to three decimal places.

At the 1% level of significance, mean time would need to be at least

Pr(𝑋 > 𝑥̅ |𝜇 = 3.55) = 0.01 ⇒ 𝑥̅ ≈ 3.806, ∴ 𝑥̅ ≥ 3.806 h

f. If the true mean time taken for the paint to dry is 3.83 hours, find the probability that the paint
company’s claim is not rejected at the 1% level of significance, assuming the standard deviation for the
paint to dry is still 0.66 hours. Give your answer correct to two decimal places.

If the mean time is actually 3.83 and not 3.55, then

Pr(Type II error) = Pr(fail to reject 𝐻 when 𝐻 is false) = Pr(𝑋 < 3.806|𝜇 = 3.83) ≈ 0.41