Type I & Type II Errors

When you perform a statistical test, you’re only taking ONE SAMPLE from a
population – and there are tons of different samples you could potentially
be collecting. You have to think about your sample in the context of ALL
the potential samples you could have collected… fortunately this is made
easy thanks to the sampling distributions of proportions and means.

Type I Error, also called , is the likelihood

that you incorrectly reject the null hypothesis

Type II Error, also called , is the likelihood

that you fail to detect the effect
that your alternative
hypothesis is trying
to uncover
 Getting Picked Up By the Cops
The scenario: A cop pulls some person over and suspects that you have just committed a
crime. He is trying to figure out whether or not to take that person to jail. The cop
needs to do a little hypothesis test in his head to figure out whether to bring in person.
H0: Person is not guilty.
HA: Person is guilty. Hiding evidence.

There are four things that can happen here:

The cop can be ACCURATE, and say that: Reality
1) Person is guilty when he actually is, OR
H0 True H0 False
2) Person is not guilty – no crime committed (Really NOT Guilty;
(Really ARE Guilty)
you just look sketchy)

Or he can be INACCURATE, and say that: I’m Taking

3) Person is guilty when he IS NOT You to Jail
(a FALSE ARREST), or (Reject H0)
What the
4) Person is not I’m Letting
Cop Decided
guilty when You Go
He actually IS…. (Fail to Reject
H0)
(a FAILURE TO DETECT the crime
and eluding the law)
 Getting Picked Up By the Cops #2
The scenario: A cop pulls you over and suspects that you have just committed a murder. He
is trying to figure out whether or not to take you to jail. The cop needs to do a little
hypothesis test in his head to figure out whether to bring you in.
H0: You are not guilty.
HA: You ARE guilty. You are hiding evidence and a body in your trunk.

There are four things that can happen here:

The cop can be ACCURATE, and say that: Reality
1) you’re guilty when you actually are, OR
H0 True H0 False
2) you’re not guilty – you haven’t just killed someone (Really NOT Guilty;
(Really ARE Guilty)
you just look sketchy)

Or he can be INACCURATE, and say that: I’m Taking = probability of

3) you’re guilty when you’re NOT You to Jail this test leading to
(a FALSE ARREST), or (Reject H0) FALSE ARREST
What the
4) you’re not I’m Letting  = probability of
Cop Decided
guilty when You Go this cop FAILING
TO DETECT the
you actually ARE…. (Fail to Reject
murder!
H0)
(a FAILURE TO DETECT the murder
and eluding the law)
 Consequences of Errors
• Type I Error – The test raised a false alarm and got person
sent to jail and into a HUGE legal mess even though he didn’t
deserve it. It costs him time and will probably cost him tons
of money – bailing out, court fees, lawyers… a trial!!
• Type II Error - The cop let person go when he hide out the
evidence! Probably good for the person, bad for the cop, and
VERY BAD for society.
Type II Error should be kept as low as possible in this case,
and risk some false alarms to AVOID letting criminals go.
Keeping the Type II Error low ALSO keeps the power of the
test pretty high.
 Power of the Test

Power = 1 - 
• It’s the probability of successfully detecting an effect
The power of the test INCREASES as the effect size increases…
 Interrelationships
P0 is the proportion that you’re
assuming is true – often a
standard or “known” value

P* is the TRUE population

proportion

The bigger the difference

between P0 and P*, the
bigger the EFFECT SIZE (the
bottom curve shifts to the
right as effect size gets
bigger)

If the proportion you measured from your sample is

If the proportion you measured
LESS than the real proportion P*, but STILL
from your sample is
BIGGER than the real GREATER than the values in the top curve that
proportion P*, you’ll are way out to the left all on their own, You’ll
incorrectly reject the null fail to reject the null and incur a Type II Error.
and incur a Type I Error.
Defect Prevention
 Projected Software Defects
In general, defect arrivals follow a Rayleigh Distribution Curve…can predict,
based upon project size and past defect densities, the curve, along with the
Upper and Lower Control Bounds
F(t) = 1 – e^((-t/c)^2)
f(t) = 2*((t/c)^2) *e ^((-t/c)^2) \

Defects

Upper Limit

Lower
Limit

Time

Recall that F(t) is the cumulative distribution density, f(t) is the

probability distribution, t is time, and c is a constant.