Lecture 5: Limit Theorem

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 1

0. Outline

1. Introduction

2. Chebyshev’s Theorem

3. Law of Large Numbers

4. Central Limit Theorem

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 2

1. Introduction

Frequency intepretation of probability depends on the long-run

proportion of times the outcome would occur in repeated
experiments.
Some binomial probabilities can be computed using the normal
distribution (using limiting arguments).
Many random variables that we encounter in nature have
distributions close to the normal probability distribution.

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 3

2.Chebyshev’s Theorem

The theorem gives a lower bound for the area under a curve
between two points that:
◮ are on opposite sides of the mean
◮ equidistant from the mean
Why is this result important?
◮ We do not need to know the distribution of the underlying population
◮ We only need to know its mean and variance.
The theorem was developed by a Russian mathematician,
Pafnuty Chebyshev (1821-1894)

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 4

2.Chebyshev’s Theorem

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 5

2.Chebyshev’s Theorem

This definition says that the probability that a random variable x

differs from its mean by at least K standard deviations is less
than or equal to 1/ K 2 .
For example, suppose that we do not know the
population distribution.
We can use Chebyshev’s theorem to have some understanding
about the population distribution:

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 6

2.Chebyshev’s Theorem

(Recall from Lecture 1b p. 39-41)

• For any population with mean μ and standard

deviation σ , and k > 1 , the percentage of
observations that fall within the interval
[μ + kσ]
is at least

100[1 − (1/k 2 )]%

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 7

2.Chebyshev’s Theorem

(Recall from Lecture 1b p. 39-41)

• Regardless of how the data are distributed, at least (1 - 1/k2)

of the values will fall within k standard deviations of the
mean (for k > 1)
– Examples:
At least within

(1 - 1/1.52) = 55.6% ……... k = 1.5 (μ ± 1.5σ)

(1 - 1/22) = 75% …...........k = 2 (μ ± 2σ)
(1 - 1/32) = 89% …….…... k = 3 (μ ± 3σ)

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 8

2.Chebyshev’s Theorem
(Recall from Lecture 1b p. 39-41)
The Empirical Rule
If the data distribution is bell-shaped, then the interval:
• 𝜇 ± 1𝜎 contains about 68% of the values in the population
or the sample.
• 𝜇 ± 2𝜎 contains about 95% of the values in the population
or the sample.
• 𝜇 ± 3𝜎 contains about 99.7% of the values in the population
or the sample.

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 9

2.Chebyshev’s Theorem

Example

Suppose that a random variable x has mean 24 and variance

9. Obtain a bound on the probability that the random
variable x assumes values between 16.5 to 31.5.

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 10

2.Chebyshev’s Theorem

Example

Let x be a random variable that represents the systolic blood

pressure of the population of 18- to 74-year old men. Suppose
that x has mean 129 mm Hg and standard deviation 19.8 mm Hg
► What is the bound on the probability that the systolic blood
pressure of this population will assume values between 89.4 and
168.6 mm Hg?
► In addition, assume that the distribution of x is approximately
normal. Using the normal table, find P (89.4 < x < 168.6).

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 11

2. Law of large numbers
Introduction

• If you flip a coin once, you will either get 100% heads or 0%
heads.

• If you flip a coin 1000 times, you will probably get close to
50% heads.

The Law of Large Numbers states that for many trials, the
proportion of times an event occurs settles down to one number.

• This number is called the empirical probability.

See: https://youtu.be/MntX3zWNWec

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 12

2. Law of large numbers
Requirements : iid
(Independent and identically distributed random variables)

Identical Probabilities
• The probabilities for each event must remain the same for each
trial.

Independence
• The outcome of a trial is not influenced by the outcomes of
the previous trials.

Empirical probability
# times A occurs
• P(A) = (in the long run)
# of trials

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 13

2. Law of large numbers
Example: Red Light, Green Light

– After many days,

the proportion of
green lights
encountered is
approximately 0.35.

– P(green) = 0.35.

– If we recorded more
days, the probability
would still be about 0.35.

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 14

2. Law of large numbers

The Nonexistent Law of Averages

• Wrong
– If you flip a coin 6 times and get 6 heads, then you are due for
a tail on the next flip.
– You put 10 quarters in the slot machine and lose each time.
You are just a bad luck person, so you have a smaller chance
of winning on the 11th try.

– There is no such thing as the Law of Averages for short runs.

See: https://youtu.be/D_pGQBdX55Y

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 15

2. Law of large numbers

We can now use Chebyshev’s theorem to prove the (weak) law of

large numbers.
The law says that if the sample size n is large, the sample mean
rarely deviates from the mean of the distribution of x .

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 16

2. Law of large numbers

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 17

2. Law of large numbers

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 18

3. Central limit theorem (Fundamental Theorem of Statistics)

The Central Limit Theorem is one of the most important results

in probability theory.
The theorem suggests that the z-transform of the sample mean
is asymptotically standard normal.
No matter what the shape of the original distribution, the
sampling distribution of the mean approaches a normal probability
distribution.
Even if the population is not normal, sample means from the
population will be approximately normal as long as the sample size is
large enough.

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 19

3. Central limit theorem

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 20

3. Central limit theorem

• Let X1, X2, . . . , Xn be a set of n independent random

variables having identical distributions with mean µ, variance
σ2, and X as the mean of these random variables.
• As n becomes large, the central limit theorem states that the
distribution of
X − μx
Z=
σX
approaches the standard normal distribution

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 21

3. Central limit theorem

As the sample n↑ the sampling distribution

size gets large becomes almost normal
enough… regardless of shape of
population

x
Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 22
3. Central limit theorem
If the Population is not Normal

Population
Sampling distribution
Distribution
properties:

Central Tendency

μx = μ
μ x
Variation Sampling Distribution
σ
σx = (becomes normal as n
increases) Larger
n sample
size
Smaller sample
size

μx x
Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 23
3. Central limit theorem

How Large is Large Enough?

• For most distributions, n > 25 will give a sampling

distribution that is nearly normal
• For normal population distributions, the sampling
distribution of the mean is always normally distributed

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 24

3. Central limit theorem

Example of Application

• Suppose a large population has mean μ = 8 and standard

deviation σ = 3. Suppose a random sample of size n = 36 is
selected.

• What is the probability that the sample mean is between 7.8

and 8.2?

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 25

3. Central limit theorem
Example

Solution:
• Even if the population is not normally distributed, the central
limit theorem can be used (n > 25)
• … so the sampling distribution of x is approximately normal
• … with mean μx = 8
• …and standard deviation

σ 3
σx = = = 0.5
n 36

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 26

 3. Central limit theorem

Example
Solution (continued):
 
 7.8 - 8 μ X -μ 8.2 - 8 
P(7.8  μ X  8.2) = P   
3 σ 3 
 36 n 36 
= P(-0.4  Z  0.4) = 0.3108

Population Sampling Standard Normal

Distribution Distribution Distribution .1554
??? +.1554
? ??
? ? Sample Standardize
? ? ?
?
7.8 8.2 -0.4 0.4
μ=8 X x Z
μX = 8 μz = 0

Hengki Purwoto (Econ UGM) Statistics 2 : Lecture 5 March 15, 2021 27