Probability Final

Laws of Large Number
Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Ujjivan Mitra
November 29, 2022
Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Ujjivan
of Mitra
Large Number November 29, 2022 1 / 42
Markov Inequality
Theorem 1
If X be a random variable where E(|X|k ) exists for any k ≥ 0 then for any
ε≥0
E(|X|k
P (|X| ≥ ε) ≤
εk
Proof.
Case I. X is a discrete random variable. Then P (|X| ≥ ε) = |xi |≥ε fi
P
where P (X = xi ) = fi , xi being a point of the spectrum of X.

k
Now for each xi , satisfying |xi | ≥ ε,we have |xεK
i|
≥ 1 where k ≥ 0,ε > 0.
k
So, fi |xεik| ≥ fi , sincefi ≥ 0
for all xi satisfying |xi | ≥ ε Therefore
1 P k 1 P∞ k
|xi |≥ε fi ≤ εk |xi |≥ε |xi | fi ≤ εk i=−∞ |xi |fi
P
k
Since |xi | fi ≥ 0 for every point of the spectrum of X.
Now it is given that E(|X|k )exists.
Ujjivan
of Mitra
P∞ k
So i=−∞ |xi | fi = E(|X|k )
k
Then we get P (|X| ≥ ε) ≤ E(|X|εk
)
Case II X is a continuous random variable.

Let f be the probability
R
density function of X.
Then P (|X| ≥ ε) = |x|≥ε f (x)dx
Now |x| ≥ ε ⇐⇒ x ∈ (−∞, −ε] [ε, ∞)
S
Therefore
P (|X| ≥ ε)
Z −ε Z ∞
= f (x)dx + f (x)dx
−∞ ε
|x|k
Now εK
≥ 1 whenever |x| ≥ ε
k
So f (x) ∗ |x| ≥ f (x) for all x ∈ (−∞, −ε] [ε, ∞)
S
εK
Since f (x) ≥ 0 for all x where f is defined.
R −ε R −ε |x|k
Therefore −∞ f (x)dx ≤ −∞ εK
f (x)dx and
Ujjivan
of Mitra
R∞ R ∞ |x|k
ε f (x)dx ≤ ε εK
f (x)dxHenceweget
1 −ε k 1 ∞ k
Z Z
P (|X| ≥ ε) ≤ k |x| f (x)dx + k |x| f (x)dx
ε −∞ ε ε
1 k −ε k
Z Z ∞ Z ε
[ |x| f (x)dx + |x|k f (x)dx + |x|k
ε −∞ ε −ε
Z ε
1
Since |x|k f (x)dx ≥ 0
εk −ε
Therefore we get
Z ∞
1 1
P (|X| ≥ ε) ≤ k |x|k f (x)dx = E(|X|k )
ε −∞ εk
k
So it is proved that P (|X| ≥ ε) ≤ E(|X|
εK
Hence Markov inequality is proved in case(i) and case (ii),i.e, where Xis
discrete or continuous random variable.
Ujjivan
of Mitra
Tchebycheff’s Inequality
Theorem 2
If X be any random variable having finite variance σ 2 (and hence finite
2
mean m), then for any ε > 0,P (|X − m| ≥ ε) ≤ σε2
Proof.
Case 1. Let X be a discrete random variable. Let fi =P (X = xi ), xi
being a point in the spectrum of X. Then
X
P (|X − m| ≥ ε) = fi
|xi −m|≥ε
1 X
≤ (xi − m)2 fi
ε2 |xi −m|≥ε
1 X
≤ 2 (xi − m)2 fi
ε i

Ujjivan
of Mitra
1
= V ar(X)
ε2
σ2
= 2
ε
now, |xi − m| ≥ ε
=⇒ (xi − m)2 ≥ ε2
=⇒ (xi − m)2 fi ≥ ε2 fi , sincefi ≥ 0
1
=⇒ fi ≤ 2 (xi − m)2 fi
ε
Case 2. Let X be a continuous random variable. Let f (x) be the p.d.f of
X. Then
Z
P (|X − m| ≥ ε) = f (x)dx
|X−m|≥ε
1
Z
= (x − m)2 f (x)dx
ε2 |x−m|≥ε
Ujjivan
of Mitra
Z ∞
1
≤ 2 (x − m)2 f (x)dx
ε −∞
1
= V ar(X)
ε2
σ2
=
ε2
|x − m| ≥ ε
=⇒ (x − m)2 ≥ ε2
=⇒ (x − m)2 f (x) ≥ ε2 f (x), sincef (x) ≥ 0
1
=⇒ f (x) ≤ 2 (x − m)2 f (x)
ε
Hence the proof.

Ujjivan
of Mitra
Remark 3
σ2
P (|X − m| ≥ ε) ≤
ε2
σ2
=⇒ 1 − P (|X − m| < ε) ≤
ε2
σ2
=⇒ P (|X − m| < ε) ≥ 1 − 2
ε
which is another form of Tchebycheff’s inequality.
Remark 4
Now from (1), if σε is sufficiently small, then the probability
P (|X − m| < ε) closed to 1. Therefore it is highly probable that X lies in
the interval (m − ε, m + ε), provided σε is very small.

Ujjivan
of Mitra
Markov Inequality
Theorem 5
If X be a random variable where E(|X|k ) exists for any k ≥ 0 then for any
ε≥0
E(|X|k
P (|X| ≥ ε) ≤
εk
Proof.
Case I. X is a discrete random variable. Then P (|X| ≥ ε) = |xi |≥ε fi
P
where P (X = xi ) = fi , xi being a point of the spectrum of X.

k
Now for each xi , satisfying |xi | ≥ ε,we have |xεK
i|
≥ 1 where k ≥ 0,ε > 0.
k
So, fi |xεik| ≥ fi , sincefi ≥ 0
for all xi satisfying |xi | ≥ ε Therefore
1 P k 1 P∞ k
|xi |≥ε fi ≤ εk |xi |≥ε |xi | fi ≤ εk i=−∞ |xi |fi
P
k
Since |xi | fi ≥ 0 for every point of the spectrum of X.
Now it is given that E(|X|k )exists.
Ujjivan
of Mitra
P∞ k
So i=−∞ |xi | fi = E(|X|k )
k
Then we get P (|X| ≥ ε) ≤ E(|X|εk
)
Case II X is a continuous random variable.

Let f be the probability
R
density function of X.
Then P (|X| ≥ ε) = |x|≥ε f (x)dx
Now |x| ≥ ε ⇐⇒ x ∈ (−∞, −ε] [ε, ∞)
S
Therefore
P (|X| ≥ ε)
Z −ε Z ∞
= f (x)dx + f (x)dx
−∞ ε
|x|k
Now εK
≥ 1 whenever |x| ≥ ε
k
So f (x) ∗ |x| ≥ f (x) for all x ∈ (−∞, −ε] [ε, ∞)
S
εK
Since f (x) ≥ 0 for all x where f is defined.
R −ε R −ε |x|k
Therefore −∞ f (x)dx ≤ −∞ εK
f (x)dx and
Ujjivan
of Mitra
R∞ R ∞ |x|k
ε f (x)dx ≤ ε εK
f (x)dxHenceweget
1 −ε k 1 ∞ k
Z Z
P (|X| ≥ ε) ≤ k |x| f (x)dx + k |x| f (x)dx
ε −∞ ε ε
1 k −ε k
Z Z ∞ Z ε
[ |x| f (x)dx + |x|k f (x)dx + |x|k
ε −∞ ε −ε
Z ε
1
Since |x|k f (x)dx ≥ 0
εk −ε
Therefore we get
Z ∞
1 1
P (|X| ≥ ε) ≤ k |x|k f (x)dx = E(|X|k )
ε −∞ εk
k
So it is proved that P (|X| ≥ ε) ≤ E(|X|
εK
Hence Markov inequality is proved in case(i) and case (ii),i.e, where Xis
discrete or continuous random variable.
Ujjivan
of Mitra
Borel-Cantelli Lemma
Theorem 6
P∞
(a) Let {An } be a sequence of events such that n=1 P An < ∞ then
PA=0.
(b) If {An } is an independent sequence of events such that
P∞
n=1 P An = ∞ then P A = 1.
Proof.
P A = P (limn→∞ ∞
S∞
≤
S
k=n Ak ) = limn→∞ P ( k=n Ak )
(a)
limn→∞ ∞
P
k=n P Ak = 0.
S∞ T∞
(b) We have Ac = n=1
c
k=n Ak , so that
∞ ∞
! !
[ \
P Ac = P lim Ack = lim P Ack
n→∞ n→∞
k=n k=n

Ujjivan
of Mitra
F
T∞ c Tn0 c
or n0 > n we see that k=n Ak ⊂ k=n Ak , so that
∞ n0 n0
! !
\ \ Y
P Ack ≤ lim P Ack = lim (1 − P Ak ),
n0 →∞ n0 →∞
k=n k=n k=n
because {An } is an independent sequence of events. Now we use the

elementary inequality
 
n0
X n0
Y n0
X
1−exp − αj  ≤ 1− (1−αj ) ≤ αj , n0 > n, 1 ≥ αj ≥ 0,
j=n j=n j=n
to conclude that
∞ n0
! !
\ X
P Ack ≤ lim exp − P Ak .
n0 →∞
k=n k=n
P∞
Since the series n=1 P An diverges, it follows that P Ac = 0 or P A = 1.
Ujjivan
of Mitra
Weak law of large number
Definition 7
Let {Xn } be a sequence of RVs, and let Sn = nk=1 Xk , i = 1, 2, . . . We
P
say that {Xn } obeys the weak law of large numbers (WLLN) with respect
to the sequence of constants {Bn }, Bn > 0, Bn ↑ ∞, if there exists a
P
sequence of real constants An such that Bn−1 (Sn − An ) −
→ 0 as n → 0.
An are called centering constants.
Bn are called norming constants
Theorem 8
Let Xn be sequence of pairwise uncorrelated RVs with EXi = µi and
var(Xi ) = σi2 , i = 1, 2, . . .. If ni=1 σi2 → ∞ as n → ∞, we can choose
P
Pn Pn
An = k=1 µk and Bn = i=1 σi2 , that is,
n
X Xi − µi P
Pn 2 −
→0 as n → ∞
i=1 i=1 σi
Ujjivan
of Mitra
Proof.
We have, by Chebychev’s inequality
n n
( )
E[ ni=1 (Xi − µi )]2
P
X X
2
P Sn − µk > ε σi ≤

Pn 2

k=1 ε2 σ2
i=1 i=1 i
1
= Pn 2 →0 as n → ∞
ε2 i=1 σi
Corollary 9
If the {Xn }’s are identically distributed and pairwise uncorrelated with
EXi = µ and var(Xi ) = σ 2 < ∞, we can choose An = nµ, and
Bn = nσ 2 .

Ujjivan
of Mitra
Corollary 10
In Theorem 8 we can chose Bn = n, provided that
n−2 nk=1 σi2 → 0 as n → ∞.
P
Corollary 11
In Corollary 9 we can take An = nµ and Bn = n, since
nσ 2 /n2 → 0 as n → ∞. Thus, if Xn are pairwise-uncorrelated identically
P
distributed RVs with finite variance,Sn /n −
→ µ.
Theorem 12
Let {Xn } be any sequence of RVs. Write Yn = n−1 nk=1 Xk . A necessary
P
and sufficient condition for the sequence {Xn } to satisfy the weak law of
large numbers is that
( )
Yn2
E →0 as n → ∞.
1 + Yn2

Ujjivan
of Mitra
Proof.
For any two positive numbers a, b, a > b > 0,we have
a 1+b
≥1 (0.1)
1+a b
LetA = {|Yn | ≥ ε}. Then ω ∈ A =⇒ |Yn |2 ≥ ε2 > 0. Using (0.1), we

see that ω ∈ A implies that
Yn2 1+ε2
1+Yn2 ε2
≥1
It follows that
( )
Yn2 ε2
PA ≤ P ≥
1 + Yn2 1 + ε2
Yn2 /1 + Yn2
≤E by Markov’s inequality
ε2 /1 + ε2
→0 as n → ∞.

Ujjivan
of Mitra
That is
P
Yn −
→0 as n → ∞
Conversely, we will show that for every ε > 0,
( )
Yn2
P {|Yn | ≥ ε} ≥ E − ε2 (0.2)
1 + Yn2
We will prove (0.2) for the case in which Yn is of the continuous type. If
Yn has a PDF fn (y), then
!
y2 y2
Z ∞ Z Z
fn (y) dy = + f (y) dy
2 n
−∞ 1 + y2 |y|>ε |y|≤ε 1 + y
Z ε
1

≤ P {|Yn | > ε} + 1− fn (y) dy
−ε 1 + y2
ε2
≤ P {|Yn | > ε} +
1 + ε2
≤ P {|Yn | > ε} + ε2 , Which is(0.2).
Ujjivan
of Mitra
Lemma 13
n
X
P {|Sn − mn | > ε} ≤ P {|Snc − mn | > ε} + P {|Xk | > c}.
k=1
Proof.
P {|Sn − mn | > ε} =P {|Sn − mn | > ε and|Xk | ≤ c for k = 1, 2, . . . , n}

+ P {|Sn − mn | > ε and|Xk | > c for at least one k, k
≤P {|Snc − mn | > ε} + P {|Xk | > c for at least one k, 1
n
X
≤P {|Snc − mn | > ε} + P {|Xk | > c}.
k=1

Ujjivan
of Mitra
Corollary 14
If X1 , X2 , . . . , Xn are exchangeable, then
P {|Sn − mn | > ε} ≤ P {|Snc − mn | > ε} + nP {|X1 | > c}. (0.3)
If, in addition, the RVs X1 , X2 , . . . , Xn are independent, then
nE(X1n )2
P {|Sn − mn | > ε} ≤ + nP {|X1 | > c}. (0.4)
ε2
Theorem 15
Let {Xn } be a sequence of iid RVs with common finite mean µ = EX1 .
Then
P
n−1 Sn − →µ as n → ∞.
Proof.
Let us take c = n in (0.4) and replace ε by nε; then we have
Ujjivan
of Mitra
1
P {|Sn − mn | > nε} ≤ E(X1n )2 + nP {|X1 | > n},
nε2
Where X1n is X1 truncated at n.
First note that E|X1 | < ∞ =⇒ nP {|X1 | > n} → 0 as n → ∞.
Z n
E(X1n )2 = 2 xP {|X1 | > x} dx
0
Z A Z n!
=2 + xP {|X1 | > x} dx
0 A
Where A is chosen sufficiently large that

δ
xP {|X1 | > x} < for all x ≥ A, δ > 0 arbitrary
2
Then Z n
E(X1n )2 ≤ c + δ dx ≤ c + nδ,
A
Where c is a constant.
Ujjivan
of Mitra
It follows that
1 n 2 ε2 δ
E(X1 ) ≤ +
nε2 nε2 ε2
and since δ is arbitrary, (1/nε2 )E(X1n )2 can be made arbitrarily small for
sufficiently large n. The proof is now completed by the simple observation
that since EXj = µ,
mn
→0 as n → ∞
n

Ujjivan
of Mitra
Truncated Random Variable:
Definition 16
Let X be a continuous random variable with PDF fx (x) and cumulative
distributive function Fx (x). If restricting the domain of X by a ≤ X ≤ b,
then we have a new random variable denoted by X̃. and we call it a
truncated random variable.
PDF of X̃ → fx̃ (x)
CDF Of X̃ → Fx̃ (x)
fx̃ (x) = px × fx (x), wherea ≤ x ≤ b

Integrating the above pdf from a to b we get 1 and therefore:
1
px =
Fx (b) − Fx (a)
where Fx (b), Fx (a) are the CDFs OF X b and a respectively
Fx̃ (x) = px × [Fx (x) − Fx (a)], a ≤ x ≤ b
Ujjivan
of Mitra
g(x)
fx̃ (x|a ≤ x ≤ b) =
Fx (b) − Fx (a)
fx (x).I(a ≤ x ≤ b)
=
Fx (b) − Fx (a)
∝ fx (x).I(a ≤ x ≤ b)
where g(x) = fx (x) ∀a ≤ x ≤ b and g(x) = 0, others. i.eg(x) = fx (x)

I(a ≤ x ≤ b) where I is an indicator function.

Ujjivan
of Mitra
Strong law of large number
Definition 17
We say that the sequence {Xn } obeys the strong law of large numbers
(SLLN) with respect to the norming constants {Bn } if there exists a
sequence of (centering) constants {An } such that
a.s
Bn−1 (Sn − An ) −→ 0 as n → ∞. (0.5)
Here Bn > 0 and Bn → ∞ as n → ∞.
Theorem 18
If X1 , X2 , . . . are iid RVs with common mean µ and finite fourth moment,
then
Sn

P lim = µ = 1.
n→∞ n

Ujjivan
of Mitra
Proof.
We have E ( (Xi − µ))4 = nE(X1 − µ)4 + 6 n 4
≤ Cn2 By Markov’s
P
2 σ
inequality,
( n
E [ ni=1 (Xi − µ)]4
)
Cn2 C′
X P
P (Xi − µ) > nε ≤ ≤ =

i=1
(nε)4 (nε)4 n2
Therefore,
∞
X
P {|Sn − µn| > nε} < ∞,
n=1
and it follows by the Borel-Cantelli lemma that with probability 1 only

finitely many of the events
{ω : |(Sn /n) − µ| > ε}
occur, that is P Aε = 0, where

Ujjivan
of Mitra

Sn

Aε = lim sup
− µ > ε

n→∞ n
The sets Aε increases, as ε → 0, to the ω set on which Sn /n ↛ µ.
Letting ε → 0 through a countable set of values, we have
!
[
P {Sn /n − µ ↛ 0} = P A1/k =0
k
Corollary 19
Let X1 , X2 , . . . are iid RVs such that P {|Xn | < K} = 1 for all n where K
a.s
is a positive constant, then n−1 Sn −−→ µ.
Theorem 20
Let X1 , X2 , . . . be a sequence of independent RVs. Then
a.s
Xn −−→ 0 ⇔ ∞ n=1 P {|Xn | > ε} < ∞
P
for all ε > 0.

Ujjivan
of Mitra
Proof.
Writing An = {|Xn | > ε}, we see that [An] is a sequence of independent
a.s
events. Since Xn −−→ 0, Xn → 0 on a set E c with P E = 0. A point
ω ∈ E c belongs only to a finite number of An . It follows that
lim sup An ⊂ E,
n→∞
hence P (An i.o) = 0. By the Borel-Cantelli lemma we must have

P∞ P∞
n=1 P An < ∞ [Otherwise, n=1 P An = ∞ and then P (An i.o) = 1. ]
In other direction, let
1

A1/k = lim sup |Xn | > ,
n→∞ k
and use the argument in the proof of Theorem 18 .

Ujjivan
of Mitra
Lemma 21 ((Kolmogorov’s Inequality)
Let X1 , X2 , . . . , Xn be independent RVs with common mean 0 and
variances σk2 , k = 1, 2, . . . , n Then for any ε > 0,
n
σ2
X
i
P max |Sk | > ε ≤ . (0.6)
1≤k≤n
i=1
ε2
Proof.
Let A0 = Ω,
Ak = {max1≤j≤k |Sj | ≤ ε} , k = 1, 2, . . . , n and
Bk = Ak−1 ∩ Ack
= {|S1 | ≤ ε, . . . , |Sk−1 | ≤ ε} ∩ {at least one of |S1 |, . . . , |Sk | is > ε}
= {|S1 | ≤ ε, . . . , |Sk−1 | ≤ ε, |Sk | > ε}.

Ujjivan
of Mitra
Pn
It follows that Acn = k=1 Bk and
Bk ⊂ {|Sk−1 | ≤ ε, |Sk | > ε}
As usual, let us write IBk , for the indicator function of the event Bk . Then
E(Sn IBk )2 = E{(Sn − Sk )IBk + Sk IBk }2

= E{(Sn − Sk )2 IBk + Sk2 IBk + 2Sk (Sn − Sk )IBk }.
Since Sn − Sk = Xk+1 + · · · + Xn and Sk IBk are independent, and

EXk = 0 for all k. It follows that
E(Sn IBk )2 = E{(Sn − Sk )IBk }2 + E(Sk IBk )2

≥ E(Sk IBk )2 ≥ ε2 P Bk .
The last inequality follows from the fact that in Bk , |Sk | > ε. Moreover,

Ujjivan
of Mitra
n
X n
X
E(Sn IBk )2 = E(Sn2 IAcn ) ≤ E(Sn2 ) = σk2 ,
k=1 k=1
so that n n
X X
σk2 ≥ε 2
P Bk = ε2 P (Acn ),
k=1 k=1
as asserted.
Corollary 22
Take n = 1; then
σ12
P {|X1 | > ε} ≤ ,
ε2
which is Chebychev’s inequality.

Ujjivan
of Mitra
Lemma 23 (Kronecker Lemma)
P∞
If n=1 xn convergences to s (finite) and bn ↑ ∞, then
n
b−1
X
n bk xk → 0
k=1
Proof.
Pn
Writing b0 = 0, ak = bk − bk−1 , and Sn+1 = k=1 xk , we have
n n
1 X 1 X
bk xk = bk (sk+1 − sk )
bn k=1 bn k=1
n n
!
1 X 1 X
= bn sn+1 + bk−1 sk − bk sk
bn k=1
bn k=1
n
1 X
= sn+1 − (bk − bk−1 )sk
bn k=1

Ujjivan
of Mitra
n
1 X
= sn+1 − ak sk .
bn k=1
Pn
It therefore suffices to show that b−1
n k=1 ak sk → s. Since sn → s, there
exist an n0 = n0 (ε) such that
ε
|sn − s| < f or n > n0 .
2
Since bn ↑ ∞, let n1 be an integer > n0 such that
n
0
X ε
b−1 (bk − bk−1 )(sk − s) < f or n > n1 .

n
k=1
2
Writing
n
rn = b−1
X
n (bk − bk−1 )sk ,
k=1

Ujjivan
of Mitra
We see that n
1 X
|rn − s| = (bk − bk−1 )(sk − s)

bn
k=1

and choosing n > n1 , we have

n0 n

1 X 1
X ε
|rn − s| ≤ (bk − bk−1 )(sk − s) + (bk − bk−1 ) < ε.

bn bn 2
k=1 k=n0 +1
This complete the proof.
Theorem 24
P∞ P∞
If n=1 var(xn ) < ∞, then n=1 (Xn − EXn ) converges almost surely.
Proof.
Without loss of generality, assume that EXn = 0. By Kolmogorv’s
inequality,
Ujjivan
of Mitra
1 Pn
P {max1≤k≤n |Sm+k − Sm | ≥ ε} ≤ ε2 k=1 var(Xm+k )
Letting n → ∞ we have

P max |Sk − Sm | ≥ ε = P max |Sm+k − Sm | ≥ ε
k≥1 k≥m+1
∞
1 X
= var(Xk )
ε2 m+1
It follows that
lim P max |Sk − Sm | < ε = 1
m→∞ k>m
and since ε > 0 is arbitrary, we have

 
 ∞ 
X
P lim Xj = 0 = 1
m→∞ 
j=m
P∞
Consequently, j=1 Xj converges a.s.
Ujjivan
of Mitra
Corollary 25
Let {Xn } be independent RVs. If
∞
X var(Xk )
< ∞, Bn ↑ ∞
k=1
Bk2
Sn −ESn a.s.
Then Bn −−→ 0
The corollary follows from Theorem 24 and the Kronecker lemma
Corollary 26
Every sequence {Xn } of independent RVs with uniformly bounded
variances obeys the SLLN. If var(xk ) ≤ A for all k, and Bk = k, then
P∞ σk2 Pn 1
k=1 k2 ≤A k=1 k2 < ∞ and
it follows that
Sn − ESn a.s.
−−→ 0
n
Ujjivan
of Mitra
Corollary 27 (BorePs Strong Law of Large Numbers)
For a sequence of Bernoulli trials with (constant) probability p of success,
the SLLN holds (with Bn = n and An = np). Since
1
EXk = p, var(Xk ) = p(1 − p) ≤ , 0 < p < 1,
4
the result follows from Corollary 26.
Corollary 28
Let {Xn } be iid RVs with common mean µ and finite variance σ 2 . Then
Sn

P lim = µ = 1.
n→∞ n

Ujjivan
of Mitra
Parametric Space
all possible values of the parameter. Θ
Examples 29
a) X1 , X2 , · · · , Xn ∼iid P (λ) 99K Θ = R
b) X1 , X2 , · · · , Xn ∼iid N (µ, 1) 99K Θ = R
c) X1 , X2 , · · · , Xn ∼iid N (µ, σ 2 ) 99K Θ = R × R+
Statistic:
X1 , X2 , · · · , Xn be n independent random samples on RV X. f : Rn → Rk
be Borel measurable function. the RV f (X1 , X2 , · · · , Xn ) is called statistic
Sample Mean
X1 , X2 , · · · , Xn be random sample from DF F. Then the statistic
X̄ = X1 +X2n+···+Xn is called Sample Mean.

Ujjivan
of Mitra
Sample Variance:
n Pn
2
X (Xi − X̄)2 2
i=1 Xi − n(X̄)2
S = =
i=1
n−1 n−1
exponential family of Distribution

If there exists real-valued function Q(θ) and D(θ) on Θ and
Borel-measurable function T (x1 , x2 , · · · , xn ) and S(x1 , x2 , · · · , xn ) on Rn
then
fθ (x1 , x2 , · · · , xn ) = exp[Q(θ)T (θ) + D(θ) + S(X)]
{fθ : θ ∈ Θ} is one-parameter exponential family.
1 X ∼ N (µ, σ 2 )
2 X ∼ P (λ), λ > 0

Ujjivan
of Mitra
Estimate and Estimator:
let X be a population RV. and α be an unknown parameter of the
population. to determine the experimental value of α based on a sample
drawn from the population, we define a statistic a whose computed value
is approximately equal to α.
let A be RV corresponding to the statistic a.
Then a is called an estimate of the unknown parameter. and A is called
estimator of α
Theorem 30
The Sample Mean from the exponential family of distribution is a strongly
consistent estimator of the population mean.

Ujjivan
of Mitra
Proof.
let X be a population RV. and m= population mean
(x1 , x2 , · · · , xn ) be a sample of size n from the population
n
1 X
Sample mean = x̄ = × xi
n i=1
n
1 X
RV corresponding to x̄ = × Xi
n i=1
E(X̄) = m
σ2
V ar(X̄) =
n
by Tchebycheff’s inequality
σ2
0 ≤ P (| X̄ − m |≥ ε) ≤ f or all ε ≥ 0
nε2

Ujjivan
of Mitra
f or n → ∞
0 ≤ limn→∞ P (| X̄ − m |≥ ε) ≤ 0
limn→∞ P (| X̄ − m |≥ ε) = 0
X̄ →in p m as n → ∞
X̄ is a consistent estimator of the population mean
If we replace convergence in probability with a.s convergence, then the
estimator is said to be strongly consistent

Ujjivan
of Mitra

Probability Final

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Probability Final

Uploaded by

Copyright:

Available Formats

Laws of Large Number

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Ujjivan Mitra

November 29, 2022

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

where P (X = xi ) = fi , xi being a point of the spectrum of X.

Case II X is a continuous random variable.

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

which is another form of Tchebycheff’s inequality.

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

where P (X = xi ) = fi , xi being a point of the spectrum of X.

Case II X is a continuous random variable.

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

because {An } is an independent sequence of events. Now we use the

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

LetA = {|Yn | ≥ ε}. Then ω ∈ A =⇒ |Yn |2 ≥ ε2 > 0. Using (0.1), we

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

P {|Sn − mn | > ε} =P {|Sn − mn | > ε and|Xk | ≤ c for k = 1, 2, . . . , n}

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

P {|Sn − mn | > ε} ≤ P {|Snc − mn | > ε} + nP {|X1 | > c}. (0.3)

If, in addition, the RVs X1 , X2 , . . . , Xn are independent, then

Where A is chosen sufficiently large that

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

fx̃ (x) = px × fx (x), wherea ≤ x ≤ b

where g(x) = fx (x) ∀a ≤ x ≤ b and g(x) = 0, others. i.eg(x) = fx (x)

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Here Bn > 0 and Bn → ∞ as n → ∞.

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

and it follows by the Borel-Cantelli lemma that with probability 1 only

{ω : |(Sn /n) − µ| > ε}

occur, that is P Aε = 0, where

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

hence P (An i.o) = 0. By the Borel-Cantelli lemma we must have

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Bk ⊂ {|Sk−1 | ≤ ε, |Sk | > ε}

E(Sn IBk )2 = E{(Sn − Sk )IBk + Sk IBk }2

Since Sn − Sk = Xk+1 + · · · + Xn and Sk IBk are independent, and

E(Sn IBk )2 = E{(Sn − Sk )IBk }2 + E(Sk IBk )2

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

and choosing n > n1 , we have

This complete the proof.

and since ε > 0 is arbitrary, we have

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

exponential family of Distribution

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

Sagnik Banerjee, Ritankar Bhattacharyya,Surojit Manna, Laws

You might also like