Section 9

Large Deviations and The Law of The

Iterated Logarithm
Po-Ning Chen, Professor
Institute of Communications Engineering
National Chiao Tung University
Hsin Chu, Taiwan 30010, R.O.C.
Low ta-t (X
1
+ X
2
+ + X
n
)/n convcrc- to mcan 9-1
Attcr thc vaidation ot thc -tron aw, thc ncxt quc-tion i- naturay how ta-t
(X
1
+ X
2
+ + X
n
)/n convcrc- to marina mcan
`orc -jccicay, wc conccrn that how ta-t
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

_
convcrc- to zcro
Thi- conccrn rin- uj an intcrc-tin rc-carch -u,cctlarge deviations, which
u-c- thc tcchniquc ot moment generating function and Markovs inequality
(l- it nicc to -cc thc od tricnd ot Markovs inequality aain So u-ctu it i-')
Low ta-t (X
1
+ X
2
+ + X
n
)/n convcrc- to mcan 9-2
Denition (Moment generating function) Thc momcnt cncratin tunc-
tion ot a random variac X i- dcncd a-
M(t) E|cxj(tX)|
_

e
tx
dF(x),
tor a t tor which thi- i- nitc, whcrc F() i- thc cumuativc di-triution tunction
(cdt) ot thc random variac X
A cncra rcjrc-cntation tor intcration wrt a (cumuativc) di-triution tunc-
tion F() i-
_
A
dF(x), which i- namcd thc Stieltjes integral
Such a rcjrc-cntation can c ajjicd tor oth di-crctc -ujjort and continu-
ou- -ujjort \c wi u-c thi- convcntion to trcc thc urdcn ot dicrcntiatin
di-crctc random variac- trom continuou- random variac-
Low momcnt cncratin tunction ct- it- namc 9-3
Thc Tayor cxjan-ion ot cxjtx i- cqua to

k=0
t
k
k'
x
k
tor a rca x
Lcncc, cxjtX

k=0
t
k
k'
X
k
, which ivc- that
E |cxjtX| E
_

k=0
t
k
k'
X
k
_

k=0
t
k
k'
E|X
k
|.
it a momcnt- ot X cxi-t
Quc-tion \hat it -omc momcnt- ot X do not cxi-t Scc thc ncxt cxamjc
Example lt thc jdt ot X i- 1/(x + 1)
2
tor x 0, and zcro, othcrwi-c, thcn
E|X
k
| tor k 1 ln -uch ca-c, E|e
tX
|
_
_
_
1 + te
t
_

t
s
1
e
s
ds, it t < 0.
1, it t 0.
, it t > 0.
Lut

k=0
t
k
k'
E|X
k
| 1 +

k=1
t
k
k'
i- ajjarcnty not a wc-dcncd tunction tor
t < 0 Thcrctorc, wc cannot writc E |cxjtX|

k=0
t
k
k'
E|X
k
|.
Low momcnt cncratin tunction ct- it- namc 9-4
Lemma ([A29]) lt f(x)

k=0
a
k
x
k
convcrc- tor cvcry [x[ < r tor -omc r > 0,
thcn f() i- dicrcntiac tor [x[ < r, and
f
/
(x)
df(x)
dx

k=1
k a
k
x
k1
.
Irom thc cmma, it i- ccar that a
1
f
/
(0)
lt f
/
(x)

k=1
k a
k
x
k1
a-o convcrc- tor [x[ < r tor -omc r > 0, thcn a
2

1
2!
f
//
(0)
. . . . . .
\c can rcjcat thc jrocc-- to otain a thc coccicnt-, it convcrcncc-tor--omc-r
kccj- vaid
Low momcnt cncratin tunction ct- it- namc 9-5
Thu-, it E|[X
k
[| < A
k
tor -omc A (which, tor cxamjc, i- vaid whcn X i- a
oundcd random variac), thcn E|e
tX
|

k=0
t
k
k'
E|X
k
| (and a-o cvcry ordcr
ot it- dcrivativc-) convcrc- tor [t[ < 1/A
Accordiny, tor k 1,
E|X
k
| M
(k)
(0)
d
k
M(t)
dt
k

t=0
.
Thi- concudc- that thc momcnt ot X can c otaincd y -uccc--ivc dicrcn-
tiation, whcncc M(t) ct- it- namc
For convenience, we assume throughout (unless stated other-
wise) that all moments of the concerned random variables exist,
and these moments can be obtained by taking successive deriva-
tive of the moment generating function.
Loc- momcnt- aonc dctcrminc thc di-triution 9-6
`ot nccc--ary, cvcn it a momcnt- cxi-t' \c nccd jo-itivc radiu- ot convcrcncc
tor thc jowcr -cric- cow
Theorem 30.1 Sujjo-c a momcnt- ot X cxi-t Thcn it thc jowcr -cric-

k=0
t
k
k'
E|X
k
| ha- a jo-itivc radiu- ot convcrcncc (ic, convcrcncc tor [t[ < r
tor -omc r), thc di-triution i- uniqucy dctcrmincd y thc momcnt-
Application of the distribution-determined-by-its-moments property
Example 9.2 lt X
1
, . . . , X
n
arc iid with lr|X
n
1| 1 lr|X
n
0| p
Lctcrminc thc di-triution ot S
n
X
1
+ + X
n
y mcan- ot Thcorcm 301
Loc- momcnt- aonc dctcrminc thc di-triution 9-7
Solution: Thc momcnt cncratin tunction ot S
n
i-
E|e
tS
n
| E|e
t(X
1
++X
n
)
|
n

j=1
E|e
tX
j
| (y indcjcndcncc)
(pe
t
+ (1 p))
n

k=0
__
n
k
_
p
k
(1 p)
k
_
e
tk
(y inomia cxjan-ion).
Sincc a momcnt- ot S
n
cxi-t (y E|[S
k
n
[| n
k
< ), and

k=0
t
k
k'
E|S
k
n
|

k=0
[t[
k
k'
E|[S
n
[
k
|

k=0
[t[
k
k'
n
k

k=0
[tn[
k
k'
e
[tn[
<
tor cvcry t 1, thc di-triution ot S
n
-ha cqua lr|S
n
k|
_
n
k
_
p
k
(1 p)
k
2
Cumuant cncratin tunction 9-8
Denition (Cumulant generating function) Thc cumulant generating
function i- dcncd a-
C(t) o M(t),
whcrc M(t) i- thc momcnt cncratin tunction
Polynomial approximate to cumulant generating function
Thc Tayor cxjan-ion ot o(x + 1) i-

v=1
(1)
v+1
v
x
v
.
Lcncc, it M(t)

k=0
t
k
k'
E|X
k
|,
C(t) o M(t)
o|(M(t) 1) + 1|

v=1
(1)
v+1
v
(M(t) 1)
v

v=1
(1)
v+1
v
_

k=1
t
k
k'
E|X
k
|
_
v

i=1
c
i
i'
t
i
,
whcrc c
i

i=1
arc cumulants ot X
Cumuant cncratin tunction 9-9
Lquatin coccicnt- cad- to
C
(1)
(0) c
1
E|X|
C
(2)
(0) c
2
E|X
2
| E
2
|X| \ar|X|
C
(3)
(0) c
3
E|X
3
| 3E|X||X
2
| + 2E
3
|X|
C
(4)
(0) c
4
E|X
4
| !E|X|E|X
3
| 3E
2
|X
2
| + 12E
2
|X|E|X
2
| oE
4
|X|

ln ca-c E|X| 0,
C
(1)
(0) c
1
0
C
(2)
(0) c
2
E|X
2
| \ar|X|
C
(3)
(0) c
3
E|X
3
|
C
(4)
(0) c
4
E|X
4
| 3E
2
|X
2
|

lrojcrtic- ot cumuant cncratin tunction 9-10

Property 1 C
S
n
(t) C
X
1
(t) + C
X
2
(t) + + C
X
n
(t) tor S
n

n
j=1
X
j
with
indcjcndcnt X
j

n
j=1

Proof: Takin oarithm- convcrt- thc jroduct rcation into additivc rcation
So, tor indcjcndcnt random variac- X
1
, X
2
, . . . , X
n
, thc cumuant cncratin
tunction ot S
n
X
1
+ X
2
+ + X
n
cqua
C
S
n
(t) C
X
1
(t) + C
X
2
(t) + + C
X
n
(t).
2
Ly thi-, wc can ca-iy conrm that thc variancc ot -um ot indcjcndcnt -amjc-
cqua thc -um ot individua variancc- y takin thc -ccond dcrivativc-
Property 2 C
Xx
(t) C
X
(t) t x
Proof:
C
Xx
(t) o E
_
e
t(Xx)
_
o E
_
e
tX

tx C
X
(t) tx.
2
Larc dcviation- 9-11
O-crvc that
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

_
lr
_
1
n
(X
1
+ X
2
+ + X
n
) m
_
+ lr
_
1
n
(X
1
+ X
2
+ + X
n
) m
_
.
Ly cttin Y (X
1
+ X
2
+ + X
n
)/n and y + m tor thc r-t tcrm,
and cttin Y (X
1
+ X
2
+ + X
n
)/n and y m tor thc -ccond
tcrm, thc conccrncd jroaiity ccomc- thc c-timatc (or nd a ound tor)
lr|Y y|
Can wc jrovidc a nicc ound to lr|Y y| y momcnt- ot Y , a- momcnt- arc
ca-icr to otain in jracticc Ior cxamjc, tor noi-c in communication-, ony
thc r-t two momcnt- arc a--umcd jriori known u-uay'
(Onc ot thc) Answer(-) `arkov- incquaity it Y i- non-ncativc random
variac
\hat it Y may a--umc ncativc vauc-
Larc dcviation- 9-12
Solution L-timatc lr|Y y| in tcrm- ot lr|e
tY
e
ty
| tor any jo-itivc t ln
othcr word-, -incc e
tY
i- non-ncativc, `arkov incquaity ivc- that
lr|Y y| lr
_
e
t
/
Y
e
t
/
y
_

E
_
(e
t
/
Y
)
k
_
(e
t
/
y
)
k

E
_
e
tY

e
ty

M
Y
(t)
e
ty
e
[tyC
Y
(t)]
.
Thcrctorc, wc can jrovidc an ujjcr ound tor lr|Y y| in tcrm- ot thc
cumuant cncratin tunction a-
lr|Y y| cxj|ty C
Y
(t)| cxjC
Y y
(t) tor t 0.
Lcncc thc c-t cxjoncnt i- int
t0
C
Y y
(t) Ior E|Y | 0, wc wi -cc atcr
that int
t0
C
Y y
(t) int
t1
C
Y y
(t)
Iunction I
Y
(y) -uj
t1
|ty C
Y
(t)| int
t1
C
Y y
(t) i- cacd thc large de-
viation rate function \c wi carn that thc arc dcviation ratc tunction
jrovidc- thc rate of convergence vcry -horty
Larc dcviation- 9-13
lt -ucc- to dcrivc thcorcm- on lr|

Y 0|, -incc wc can trcat |Y y| a- |

Y
Y y 0|, and -ti yicd thc -amc ound lc,
lr|Y y| lr|(Y y) 0|
lr|

Y 0|
cxj|t 0 C
Y
(t)| ( M
Y
(t))
cxj|t 0 C
(Y y)
(t)|
cxj|ty C
Y
(t)| tor t 0.
Athouh thc aovc tormua i- away- vaid, it ccomc- trivia it |ty C
Y
(t)|
0
Quc-tion i- undcr what condition thc ound ccomc- non-trivia
An-wcr lt E|

Y | < 0, thcn lr|

Y 0| cxjC
Y
(t)| M
Y
(t) < 1
lr|Y 0| M
Y
(t) undcr < E|Y | < 0 9-14
Claim: E|Y
2
| > 0 (t 1)M
//
Y
(t) > 0 Proof: Sujjo-c M
//
Y
(t) E|Y
2
e
tY
|
0 tor -omc t s Thcn lr|Y
2
e
sY
0| 1 and E|Y
2
e
sY
| 0 ,ointy imjy that
lr|Y
2
e
sY
0| 1 Thcrctorc, lr|Y 0| 1, which contradict- to E|Y
2
| > 0
Observation 1 M
/
Y
(0) E|Y | < 0 and M
//
Y
(0) E|Y
2
| E
2
|Y | > 0 imjy
that M
Y
(t) i- convcx and ha- minimum at -omc t > 0, whcrc M
Y
() < 1
-
6
1

Observation 2 Thc minimum can jo--iy c innity, it lr|Y > 0| 0

-
6
1
Example for Observation 2 Sujjo-c P|Y 1| P|Y 0| 1/2 Thcn
lr|Y > 0| 0, M
Y
(t) (e
t
+ 1)/2, , and lr|Y 0| 1/2
Lound undcr < E|Y | < 0 and < 9-15
Liin-cy- ook a--umc- that lr|Y > 0| > 0 lndccd, lr|Y > 0| > 0
imjic- M
Y
(t) a- t , which in turn- imjic- < , which i-
actuay what dc-irc- y thc -u-cqucnt dcrivation-
`otay, imjic- lr|Y > 0| 0 So lr|Y 0| lr|Y 0|, a trivia
ca-c ot ittc intcrc-t'
ln addition, undcr < and < E|Y | < 0 (and imjicity, > 0),
M
Y
() > 0 ccau-c it M
Y
() E|e
Y
| 0, thcn lr|e
Y
0| 1, which
imjic- lr|Y | 1, a contradiction to E|Y | >
Liin-cy- ook did not a--umc that E|Y | > Lut thi- ca-c i- actuay
imjicity incudcd (y a--umin simple random variac) in thc jroot
Example lr|Y n| 3/(
2
n
2
) tor n 1, and lr|Y 0| lr|Y 1| 1/!
Thcn E|Y | and M
Y
(t)
3

k=1
e
kt
k
2
+
1 + e
t
!

_
_
_
, tor t < 0
1, tor t 0
M
Y
(t), tor t > 0
`otay, in thi- cxamjc, thc momcnt- cannot c dctcrmincd y takin thc dcriva-
tivc- ot M
Y
(t)
Lound undcr < E|Y | < 0 and < 9-16
So wc havc an ujjcr ound int
t1
M
Y
(t) M
Y
() on lr|Y 0| undcr thc
condition that a momcnt- cxi-t and thc momcnt- can c otaincd y takin
thc -uccc--ivc dcrivativc ot thc momcnt cncratin tunction
l- thc ujjcr ound tiht Low to otain a owcr ound to lr|Y 0|
Answer: Thc ound i- tiht exponentially Thi- can c vaidatcd y jrovidin
a owcr ound to lr|Y 0| in tcrm- ot thc twisting di-triution tcchniquc
Lound undcr < E|Y | < 0 and < 9-17
Denition (Twisted distribution) Thc twisted distribution ot P
Y
(y) with
twi-tcd jaramctcr t i- dcncd a-
dP
Y
(t)
(y)
e
ty
dP
Y
(y)
_
}
e
ty
/
dP
Y
(y
/
)

e
ty
dP
Y
(y)
M
Y
(t)
.
Ior di-crctc random variac, thc aovc cquation can c rcwrittcn a-
lr|Y
(t)
y|
e
ty
lr|Y y|

y
/
}
e
ty
/
lr|Y y
/
|

e
ty
lr|Y y|
M
Y
(t)
.
\hy twi-tin a di-triution 9-18
An-wcr To taciitatc a jroaiity c-timation
ln jrincijc,
lr|Y 0|
_

0
dP
Y
(y)
_

0
M
Y
(t)e
ty
dP
Y
(t)
(y) M
Y
(t)
_

0
e
ty
dP
Y
(t)
(y).
Sujjo-c lr|Y 0| i- vcry, vcry -ma (-o it i- unca-y to makc an accuratc
c-timatc ot it) Thcn choo-c t -uch that M
Y
(t) 1 immcdiatcy ivc- that
__

0
e
ty
dP
Y
(t)
(y)
_
lr|Y 0|.
Conccjtuay, mca-urin a i quantity (c, 0.32113 or
__

0
e
ty
dP
Y
(t)
(y)
_
)
i- ca-icr than mca-urin an innitcy -ma quantity (c, 0.0 . . . 032113 or
lr|Y 0|)
Such a rcvcr-ic jroaiity amjityin tcchniquc y momcnt cncratin
tunction hcj- u- to makc a cttcr c-timatc on thc jcrhaj- vcry -ma lr|Y
0| (`otc Thc tcchniquc ha- to c rcvcr-ic. othcrwi-c, wc may know ittc
aout lr|Y 0|)
Question Low to cnarc
__

0
e
ty
dP
Y
(t)
(y)
_
to thc cxtrcmc Takc t
that ivc- thc -mac-t M
Y
(t)
lrojcrtic- ot twi-tin di-triution 9-19
Ly convcntion, dcnotc y Y
(t)
thc random variac havin P
Y
(t)
() a- it- di--
triution
Thc momcnt cncratin tunction ot Y
(t)
i-
M
Y
(t)
(s) E
_
e
sY
(t)
_

_
}
e
sy
dP
Y
(t)
(y)

_
}
e
sy
e
ty
M
Y
(t)
dP
Y
(y)

M
Y
(s + t)
M
Y
(t)
.
Thc momcnt- ot Y
(t)
ccomc
E
_
_
Y
(t)
_
k
_

d
k
M
Y
(t)
(s)
ds
k

s=0

M
(k)
Y
(s + t)
M
Y
(t)

s=0

M
(k)
Y
(t)
M
Y
(t)
.
So E
_
Y
()

M
/
Y
()
M
Y
()
0. ln othcr word-, thc arc-t amjicrha-
twi-tcd thc mean to thc conccrncd margin, ic, zcro Thi- cnarc- thc dc-irc-
to-c-timatc jroaiity to thc cxtrcmc
Lound undcr < E|Y | < 0 and < 9-20
`ow wc havc an ujjcr ound
lr|Y 0| M
Y
()
_

0
e
y
dP
Y
()
(y)
M
Y
()
_

0
dP
Y
()
(y) (-incc > 0 )
M
Y
() lr
_
Y
()
0
_
M
Y
().
l- thc ujjcr ound co-c to thc truc jroaiity ln othcr word-, i- thc ujjcr
ound an ovcrc-timatc ot thc truc jroaiity
A -traihttorward ajjroach to -u-tantiatc thc tihtnc-- ot an ujjcr ound i- to
nd a owcr ound that i- co-c to thc ujjcr ound
Thc twi-tin di-triution i- c-jcciay u-ctu in ndin a owcr ound
Lound undcr < E|Y | < 0 and < 9-21
Lct dP
W
(y)
dP
Y
()
(y)
lr
_
Y
()
0

c a di-triution with -ujjort y 1 y 0, and

dcnotc it- a--ociatcd random variac y W
Irom -idc 9-20, lr|Y
()
0| lr|Y 0|/M
Y
(). A- 0 < M
Y
() < 1, lr|Y
()

0| > 0 tor it it wcrc not truc, lr|Y 0| 0, which imjic-

ln tact, lr|Y 0| 0 arcady ivc- u- what wc dcmand to dcrivc, ic, thc
jroaiity ot lr|Y 0|' So thc -u-cqucnt dcrivation i- unnccc--ary tor -uch a
trivia ca-c
lr|Y 0| M
Y
()
_

0
e
y
dP
Y
()
(y)
M
Y
() lr
_
Y
()
0
_
_

0
e
y
dP
Y
()
(y)
lr
_
Y
()
0

M
Y
() lr
_
Y
()
0
_
_

0
e
w
dP
W
(w)
M
Y
() lr
_
Y
()
0
_
E|e
W
|.
(Jensens Inequality) Ior a convcx tunction (), (E|X|) E|(X)|
Ly !cn-cn- incquaity with (x) e
x
,
e
E[W]
(E|W|) E|(W)| E
_
e
W

.
Lound undcr < E|Y | < 0 and < 9-22
Thi- ivc- that
lr|Y 0| M
Y
() lr
_
Y
()
0
_
e
E[W]
.
O-crvc that
E|W|
_

0
wdP
W
(w)

_

0
y
dP
Y
()
(y)
lr
_
Y
()
0

1
lr
_
Y
()
0

[y[ dP
Y
()
(y)

1
lr
_
Y
()
0

E
_

Y
()

1
lr
_
Y
()
0

E
1/2
_
_
Y
()
_
2
_
(y Lyajounov- incq)
\c can thcn concudc that it < E|Y | < 0 and < , thcn
M
Y
() lr|Y 0| M
Y
() lr
_
Y
()
0
_
cxj
_

lr
_
Y
()
0

E
1/2
_
_
Y
()
_
2
_
_
.
A owcr ound to lr|U 0| tor zcro mcan U 9-23
To rcjacc lr|Y
()
0| in thc jrcviou- incquaity y momcnt- ot Y
()
, wc nccd a
owcr ound to it
Theorem 9.2 lt E|U| 0 and E|U
2
| > 0, thcn
lr|U 0|
E
2
|U
2
|
!E|U
4
|
.
(Schwarzs inequality) E|[XY [| E
1/2
|X
2
|E
1/2
|Y
2
|
(H olders inequality) E|[XY [| E
1/p
|[X[
p
|E
1/q
|[Y [
q
| tor p > 1, q > 1 and
1/p + 1/q 1
Proof:
Lct U
+
U I
[U0]
and U

U I
[U<0]
, whcrc I
[]
i- an indicator random
variac that cqua- 1 whcn thc cvcnt i- truc, and 0 whcn thc cvcnt i- ta-c
Ly thcir dcnition-, U
+
and U

arc oth non-ncativc

Thcn y Schwarz- incquaity,
E
_
(U
+
)
2

E
_
U
2
I
[U0]

E
1/2
_
U
4

E
1/2
_
I
2
[U0]
_
E
1/2
_
U
4

E
1/2
_
I
[U0]

,
A owcr ound to lr|U 0| tor zcro mcan U 9-24
and y L odcr- incquaity,
E
_
(U

)
2

E
_

2/3

4/3
_
E
1/(3/2)
_
_

2/3
_
3/2
_
E
1/3
_
_

4/3
_
3
_
E
2/3
_

E
1/3
_

4
_
E
2/3
_
U

E
1/3
_
U
4

.
A-o y Lodcr- incquaity, tocthcr with E|U| 0 and U U
+
U

,
E|U

| E|U
+
|
E|U I
[U0]
|
E|[U I
[U0]
[|
E
1/4
|U
4
| E
3/4
|I
4/3
[U0]
|
E
1/4
|U
4
| E
3/4
|I
[U0]
|.
A owcr ound to lr|U 0| tor zcro mcan U 9-25
Lcncc,
E|U
2
| E|(U
+
)
2
| + E|(U

)
2
|
E
1/2
_
U
4

E
1/2
_
I
[U0]

+ E
2/3
_
U

E
1/3
_
U
4

E
1/2
_
U
4

E
1/2
_
I
[U0]

+
_
E
1/4
|U
4
| E
3/4
|I
[U0]
|
_
2/3
E
1/3
_
U
4

E
1/2
_
U
4

E
1/2
_
I
[U0]

+ E
1/2
_
U
4

E
1/2
_
I
[U0]

2E
1/2
_
U
4

E
1/2
_
I
[U0]

2E
1/2
_
U
4

(lr|U 0|)
1/2
,
which immcdiatcy vaidatc- thc thcorcm 2
A owcr ound to lr|U 0| tor zcro mcan U 9-26
E|(Y
()
)
2
| > 0 lr|Y
()
0|
E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
E|(Y
()
)
2
| 0 lr|Y
()
0| 1 lr|Y 0| 1 ccau-c Y
()
ha-
thc -amc -ujjort a- Y E|Y | 0, a contradiction to thc a--umjtion that
< E|Y | < 0
M
Y
() lr|Y 0| M
Y
() lr
_
Y
()
0
_
cxj
_

lr
_
Y
()
0

E
2
|(Y
()
)
2
|
_
M
Y
()
E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
cxj
_

E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
E
1/2
|(Y
()
)
2
|
_

_
M
Y
()
E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
cxj
_

!E|(Y
()
)
4
|
E
3/2
|(Y
()
)
2
|
_
whcrc
_
E|(Y
()
)
2
| C
//
Y
()
(0) C
//
Y
()
E|(Y
()
)
4
| C
(4)
Y
()
(0) + 3E
2
|(Y
()
)
2
| C
(4)
Y
() + 3 (C
//
Y
())
2
Chcrno- Thcorcm 9-27
lcca that what wc conccrn i-
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

_
lr
_
1
n
(X
1
+ X
2
+ + X
n
) m
_
+lr
_
1
n
(X
1
+ X
2
+ + X
n
) m
_
lr
_
(X
1
m) + (X
2
m) + + (X
n
m) 0

+lr
_
(mX
1
) + (mX
2
) + + (mX
n
) 0

lr|Y 0| + lr|

Y 0|,
whcrc
Y (X
1
m) + (X
2
m) + + (X
n
m)
and

Y (mX
1
) + (mX
2
) + + (mX
n
).
A--umc X
i

n
i=1
arc iid
Chcrno- Thcorcm jjcr ound 9-28
M
Y
() lr|Y 0| M
Y
()
E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
cxj
_

!E|(Y
()
)
4
|
E
3/2
|(Y
()
)
2
|
_
.
< E|Y | n(E|X| m) n < 0
M
Y
(t)
n

i=1
M
(X
i
m)
(t) M
n
(Xm)
(t)
M
/
Y
(t) nM
/
(Xm)
(t)M
n1
(Xm)
(t)
-ati-c- M
/
(Xm)
() 0, which i- assumed nite
Or cquivacnty, wc can -ay -ati-c- C
/
(Xm)
() 0
`otay, i- indcjcndcnt ot n
Lct M
(Xm)
() Sincc i- thc minimizcr tor M
(Xm)
(t) that ha-
ncativc mcan and < , 0 < < 1
\c thcn otain

n
lr|Y 0|
Chcrno- Thcorcm Lowcr ound 9-29
M
Y
() lr|Y 0| M
Y
()
E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
cxj
_

!E|(Y
()
)
4
|
E
3/2
|(Y
()
)
2
|
_
.
M
Y
(t) M
n
(Xm)
(t)
M
Y
() M
n
(Xm)
()
n
C
Y
(t) nC
(Xm)
(t) n|C
X
(t) (m + )t|
E|(Y
()
)
2
| C
//
Y
()
(0) C
//
Y
() nC
//
X
() n
2
,
whcrc
2
C
//
X
()
E|(Y
()
)
4
| C
(4)
Y
()
(0) + 3E
2
|(Y
()
)
2
| C
(4)
Y
() + 3n
2

2
2
nC
(4)
X
() + 3n
2

2
2
n
4
+ 3n
2

2
2
,
whcrc
4
C
(4)
X
()

E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|

E
2
|(Y
()
)
2
|/n
2
!E|(Y
()
)
4
|/n
2

1
12 + !
4
/(
2
2
n)
Chcrno- Thcorcm Lowcr ound 9-30

!E|(Y
()
)
4
|
E
3/2
|(Y
()
)
2
|

!|n
4
+ 3n
2

2
2
|
n
3/2

3/2
2
!
1/2
2
_
3 +
4
/(
2
2
n)

n
lr|Y 0| M
Y
()
E
2
|(Y
()
)
2
|
!E|(Y
()
)
4
|
cxj
_

!E|(Y
()
)
4
|
E
3/2
|(Y
()
)
2
|
_

n

1
12 + !
4
/(
2
2
n)
cxj
_
!
1/2
2
_
3 +
4
/(
2
2
n)

n
_
A- jrovcd in -idc 9-2o,
2
E|(Y
()
)
2
|/n > 0
n
4
+ 3n
2

2
2
E|(Y
()
)
4
| E
2
|(Y
()
)
2
| n
2

2
2
imjic-
4
2n
2
2
Low-
cvcr,
4
i- not nccc--ariy jo-itivc
Chcrno- Thcorcm 9-31
Thi- concudc-

n
lr|Y 0|
n

1
12 + !
4
/(
2
2
n)
cxj
_
!
1/2
2
_
3 +
4
/(
2
2
n)

n
_
,
whcrc
_

_
i- thc uniquc, jo-itivc, nitc -oution to C
/
X
() (m + ), indcjcndcnt ot n
o C
X
() (m + )

2
C
//
X
()

4
C
(4)
X
()
Thc owcr ound can c imjrovcd uj to
C

n

n
Lut thc currcnt owcr ound i-
-ucicnt tor our jrc-cnt oa
Low aout lr|

Y 0| Thc -amc tcchniquc can c ajjicd

Chcrno- Thcorcm jjcr ound 9-32
M

Y
( ) lr|

Y 0| M

Y
( )
E
2
|(

Y
( )
)
2
|
!E|(

Y
( )
)
4
|
cxj
_

! E|(

Y
( )
)
4
|
E
3/2
|(

Y
( )
)
2
|
_
.
< E|

Y | n(mE|X| ) n < 0
M

Y
(t)
n

i=1
M
(mX
i
)
(t) M
n
(mX)
(t)
M
/

Y
(t) nM
/
(mX)
(t)M
n1
(mX)
(t)
-ati-c- M
/
(mX)
( ) 0, which i- assumed nite
Or cquivacnty, wc can -ay -ati-c- C
/
(mX)
( ) 0
`otay, i- indcjcndcnt ot n
Lct M
(mX)
( ) Sincc i- thc minimizcr tor M
(mX)
(t) that ha-
ncativc mcan and < , 0 < < 1
\c thcn otain

n
lr|

Y 0|
Chcrno- Thcorcm Lowcr ound 9-33
M

Y
( ) lr|

Y 0| M

Y
( )
E
2
|(

Y
( )
)
2
|
!E|(

Y
( )
)
4
|
cxj
_

! E|(

Y
( )
)
4
|
E
3/2
|(

Y
( )
)
2
|
_
.
M

Y
(t) M
n
(mX)
(t)
M

Y
( ) M
n
(mX)
( )
n
C

Y
(t) nC
(mX)
(t) n|C
X
(t) + (m)t|
E|(

Y
( )
)
2
| C
//

Y
( )
(0) C
//

Y
( ) nC
//
X
( ) n

2
,
whcrc

2
C
//
X
( )
E|(

Y
( )
)
4
| C
(4)

Y
( )
(0) + 3E
2
|(

Y
( )
)
2
| C
(4)

Y
( ) + 3n
2

2
2
nC
(4)
X
( ) + 3n
2

2
2
n

4
+ 3n
2

2
2
,
whcrc

4
C
(4)
X
( )

E
2
|(

Y
( )
)
2
|
!E|(

Y
( )
)
4
|

E
2
|(

Y
( )
)
2
|/n
2
!E|(

Y
( )
)
4
|/n
2

1
12 + !

4
/(

2
2
n)
Chcrno- Thcorcm Lowcr ound 9-34

! E|(

Y
( )
)
4
|
E
3/2
|(

Y
( )
)
2
|

! |n

4
+ 3n
2

2
2
|
n
3/2

3/2
2
!

1/2
2
_
3 +

4
/(

2
2
n)
_

n
lr|

Y 0| M

Y
( )
E
2
|(

Y
( )
)
2
|
!E|(

Y
( )
)
4
|
cxj
_

! E|(

Y
( )
)
4
|
E
3/2
|(

Y
( )
)
2
|
_

n

1
12 + !

4
/(

2
2
n)
cxj
_
!

1/2
2
_
3 +

4
/(

2
2
n)
_

n
_
A- jrovcd in -idc 9-2o,

2
E|(

Y
( )
)
2
|/n > 0
n

4
+ 3n
2

2
2
E|(

Y
( )
)
4
| E
2
|(

Y
( )
)
2
| n
2

2
2
imjic-

4
2n

2
2
Low-
cvcr,

4
i- not nccc--ariy jo-itivc
Chcrno- Thcorcm 9-35
Thi- concudc- to

n
lr|

Y 0|
n

1
12 + !

4
/(

2
2
n)
cxj
_
!

1/2
2
_
3 +

4
/(

2
2
n)
_

n
_
,
whcrc
_

_
i- thc uniquc, jo-itivc, nitc -oution to C
/
X
( ) (m), indcjcndcnt ot n
o C
X
( ) + (m)

2
C
//
X
( )

4
C
(4)
X
( )
Thc owcr ound can c imjrovcd uj to
C

n

n
Lut thc currcnt owcr ound i-
-ucicnt tor our jrc-cnt oa
Chcrno- Thcorcm 9-36
To -ummarizc,

n
+
n
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

_

n

1
12 + !
4
/(
2
2
n)
e
4
|
3+
4
/(
2
2
n)
|

2
n
+
n

1
12 + !

4
/(

2
2
n)
e
4
|
3+

4
/(

2
2
n)
|

2
n
whcrc
_

_
int
t1
M
(Xm)
(t) int
t1
e
t
e
tm
M
X
(t)
cxj
_
-uj
t1
|t(m + ) C
X
(t)|
_
cxj I
X
(m + )
int
t1
M
(mX)
(t) int
t1
e
t
e
tm
M
X
(t)
int
t1
e
t
e
tm
M
X
(t) cxj
_
-uj
t1
|t(m) C
X
(t)|
_
cxj I
X
(m)
whcrc I
X
(x) -uj
t1
|tx C
X
(t)| i- thc arc dcviation ratc tunction
Chcrno- Thcorcm 9-37
Observation Low ta-t
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

>
_
convcrc- to zcro, it X
1
, X
2
, . . ., X
n
arc iid
Answer Lxacty cxjoncntiay ta-t (wc havc oth ujjcr and owcr ound-) And
thc cxjoncntia ratc can c dctcrmincd y thc large deviation rate function ot thc
marina X
Lxamjc 1 9-38
Sujjo-c wc wi-h to c-timatc thc variancc
2
ot a zcro-mcan Gau--ian noi-c y
-uccc--ivc o-crvation- N
1
, N
2
, . . ., N
n
Thcn
lr
_

N
2
1
+ N
2
2
+ + N
2
n
n

2

>
_

n
+
n
,
whcrc cxj
_
I
N
2(
2
+ )
_
and cxj
_
I
N
2(
2
)
_

M
N
2(t) E|e
tN
2
|
_

e
tu
2 1

2
2
e
u
2
/(2
2
)
du
_
_
_
1

1 2
2
t
, it t < 1/(2
2
).
, othcrwi-c.
C
N
2(t)
_

1
2
o(1 2
2
t), it t < 1/(2
2
).
, othcrwi-c.
I
N
2(x) -uj
t1
|tx C
N
2(t)| -uj
t<1/(2
2
)
_
tx +
1
2
o(1 2
2
t)
_

_
_
_
, it x 0.
_
1
2
2

1
2x
_
x +
1
2
o
_
1 2
2
_
1
2
2

1
2x
__
, it x > 0

_
, it x 0.
1
2
_
x

2
1
_

1
2
o
_
x

2
_
, it x > 0
Lxamjc 1 9-39
Lcncc, it 0.01
2
, thcn
_

_
o() I
N
2(
2
+ )
1
2
(0.01)
1
2
o(1 + 0.01) 2.!S3!o 10
5
o( ) I
N
2(
2
)
1
2
(0.01)
1
2
o(1 0.01) 2.`1o9 10
5
So, it wc dc-irc that
lr
_

N
2
1
+ N
2
2
+ + N
2
n
n

2

> 0.01
2
_

n
+
n
10
5
+ 10
5
,
it i- -atcr to havc thc -amjc numcr -ati-tyin
n max
_
o(10
5
)
o()
,
o(10
5
)
o( )
_
max!o3`S3, !`!!! !o3`S3.
2
Observations
\hcn thc rcquircd dcrcc ot accuracy i- hihcr (ic, -macr), morc -amjc-
arc nccdcd'
Thc aovc rcquircd -amjc numcr i- nothin to do with
2

Lxamjc 1 9-40
C
N
2(t)
_

1
2
o(1 2
2
t), it t < 1/(2
2
).
, othcrwi-c


2
2
(
2
+ )
and
1

(
2
+ )
1/2
e
/(2
2
)

2
C
//
N
2
() 2(
2
+ )
2

4
C
(4)
N
2
() !S(
2
+ )
4


2
2
(
2
)
and
1

(
2
)
1/2
e
/(2
2
)

2
C
//
N
2
( ) 2(
2
)
2

4
C
(4)
N
2
( ) !S(
2
)
4

4
/
2
2

4
/

2
2
12 and
1/2
2

1/2
2
/(

2
2
)
lr
_

N
2
1
+ N
2
2
+ + N
2
n
n

2

> 0.01
2
_
(
n
+
n
)
e
0.06(1+4/n)

2n
12(1 + !/n)
o.o!`1! 10
27
(
n
+
n
) tor n !o3S`3 and 0.01
2
.
Lxamjc 2 9-41
Sujjo-c wc wi-h to c-timatc thc hcad-ajjcarin jroaiity ot a coin ij y -uc-
cc--ivc tria- X
1
, X
2
, . . ., X
n
, whcrc X
i
1 rcjrc-cnt- a hcad ajjcarancc Thcn
lr
_

X
1
+ X
2
+ + X
n
n
p

>
_

n
+
n
,
whcrc cxj I
X
(p + ) and cxj I
X
(p )
M
X
(t) E|e
tX
|
_

e
tx
dP
X
(x) e
t
p + 1 p
C
X
(t) o(e
t
p + 1 p)
I
X
(x) -uj
t1
|tx C
X
(t)| -uj
t1
_
tx o(e
t
p + 1 p)

_
x o
_
x
p
_
+ (1 x) o
_
1 x
1 p
_
, it 0 < x < 1.
o(p), it x 1.
o(1 p), it x 0.
, othcrwi-c
Lcncc, it p 0.1 and 0.01p 10
3
, thcn
_

_
o() I
X
(p + ) 0.101 o
0.101
0.1
+ 0.S99 o
0.S99
0.9
`.`391S 10
6
o( ) I
X
(p ) 0.099 o
0.099
0.1
+ 0.901 o
0.901
0.9
`.`21 10
6
Lxamjc 2 9-42
So, it wc dc-irc that
lr
_

X
1
+ X
2
+ + X
n
n
p

> 0.01p
_

n
+
n
10
5
+ 10
5
,
it i- -atcr to havc thc -amjc numcr -ati-tyin
n max
_
o(10
5
)
o()
,
o(10
5
)
o( )
_
max2.0S!`10
6
, 2.0oo110
6
20S!`0.
2
Observation
\hcn p 0.` and 0.00`, o() o( ) `.0000S 10
5
Lcncc,
thc rcquircd numcr ot -amjc- ccomc- 2302`` Accordiny, it i- ca-icr to
cn-urc a tair coin than a ia-cd coin
Irom thc two cxamjc-, you -ha carn that what aw ot arc numcr- conccrn
i- rcay a large numcr
Lxamjc 2 9-43
C
X
(t) o(e
t
p + 1 p)
o
(1 p)(p + )
p(1 p )
and
_
1 p
1 p
_
1p
_
p
p +
_
p+

2
C
//
X
() (p + )(1 p )

4
C
(4)
X
() (p + )(1 p )|1 o(p + ) + o(p + )
2
|
o
p(1 p + )
(1 p)(p )
and
_
1 p
1 p +
_
1p+
_
p
p
_
p

2
C
//
X
( ) (p )(1 p + )

4
C
(4)
X
( ) (p )(1 p + )|1 o(p ) + o(p )
2
|
lr
_

X
1
+ X
2
+ + X
n
n
p

> 0.01p
_
(.`21SS 10
27
)
n
+ (.`219 10
27
)
n
tor n 20S!`0, p 0.1, 0.01p, 0.0110o21,
2
0.09099,
4
0.0!13322
0.0111o0S,

2
0.0S9199,

4
0.0!1!o02
Lxamjc 2 9-44
lr
_

X
1
+ X
2
+ + X
n
n
p

> 0.01p
_
(S.20011 10
27
)
n
+ (S.20011 10
27
)
n
tor n 2302``, p 0.`, 0.01p, 0.020000,
2
0.2!99`,
4
0.12!9`
0.020000,

2
0.2!99`,

4
0.12!9`
Stron aw ot arc numcr rcvi-itcd 9-45
Theorem lt

n=1
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

_
< tor any > 0
aritrariy -ma, thcn thc -tron aw hod-
Theoremlt X
j

j=1
arc iid, and a momcnt- ot X
i
cxi-t and can c dctcrmincd
y takin thc dcrivativc- ot thc momcnt cncratin tunction, thcn

n=1
lr
_

1
n
(X
1
+ X
2
+ + X
n
) m

n=1
(
n
+
n
) <
tor any > 0 aritrariy -ma. hcncc, thc -tron aw hod-
Proof: Choo-c 0 < < -uch that M
(Xm))
() < 1 and choo-c 0 < <
-uch that M
(mX)
( ) < 1 (`otc that and nccd not to c thc minimizcr-)
lr|(X
1
m) + + (X
n
m) 0| lr
_
e
[(X
1
m)++(X
n
m)]
1
_
E
n
|e
(Xm)
| (y `arkov- incquaity)
M
n
(Xm)
(),
Stron aw ot arc numcr rcvi-itcd 9-46
and
lr|(mX
1
) + + (mX
n
) 0| lr
_
e
[(mX
1
)++(mX
n
)]
1
_
E
n
|e
(mX)
| (y `arkov- incquaity)
M
n
(mX)
().
2
`otc that thc vaidity ot thc ujjcr ound ony rcquirc- ncativc mcan a--umj-
tion
Chcrno- Thcorcm 9-47
Theorem 9.3 (Chernos Theorem) Lct X
1
, X
2
, . . . c indcjcndcnt, idcnticay
di-triutcd random variac- -ati-tyin E|X
n
| < 0 (and E|X
n
| > ), P|X
n
> 0| >
0 (or int
t1
M
X
(t) M
X
() tor -omc nitc ) and a momcnt- ot X cxi-t- and can c
dctcrmincd y takin thc dcrivativc ot it- momcnt cncratin tunction Thcn
im
n

1
n
o P
_
X
1
+ X
2
+ + X
n
n
> 0
_
int
t1
o M
X
(t) int
t1
C
X
(t) I
X
(0),
whcrc I
X
(x) -uj
t1
|tx C
X
(x)|
Theorem (Cram ers Theorem) Lct X
1
, X
2
, . . . c indcjcndcnt, idcnticay di--
triutcd random variac- -ati-tyin M
X
(t) < tor a t Thcn
int
xa
I
X
(x) imint
n

1
n
o lr
_
X
1
+ X
2
+ + X
n
n
> a
_
im-uj
n

1
n
o lr
_
X
1
+ X
2
+ + X
n
n
> a
_
int
x>a
I
X
(x)
Law ot ltcratcd Loarithm 9-48
Thc jrcviou- di-cu--ion introducc- thc ratc ot convcrcncc tor lr|X
1
+ X
2
+
+ X
n
> 0| tor iid X
j

n
j=1
and < E|X
j
| < 0
\c arc thu- condcnt that
X
1
+ + X
n
n
convcrc- to it- mcan cxjoncntiay
ta-t
_
in thc -cn-c that thc jroaiity ot
X
1
+ + X
n
n
dcviatin trom it-
mcan dccrca-c- to zcro cxjoncntiay ta-t
_

Thc aovc rc-ut i- otaincd y twisting thc oriina di-triution with a -jc-
ciay cho-cn tactor , rc-utin a arcr lr|Y
()
> 0| with E|Y
()
| 0
`otay, it Y X
1
+ X
2
+ + X
n
, thcn Y
()
X
()
1
+ X
()
2
+ + X
()
n
,
and X
()
i

n
i=1
arc a-o iid
Claim: (i) X
j

2
j=1
iid (ii) Lcnc a random variac Y X
1
+X
2
(iii) Lcnc
ncw iid variac- Z
j

2
j=1
, whcrc Z
j
ha- di-triution P
X
()
(iv) Lcnc a ncw
random variac W Z
1
+ Z
2

Thcn, Y
()
and W havc cxacty thc -amc di-triution lc, to comjutc lr|Y
()

/| tor any mca-urac -ct / can c imjcmcntcd y lr|X

()
1
+ X
()
2
/|
Law ot ltcratcd Loarithm 9-49
Proof:
lr|Y
()
b|
_
b

dP
Y
()
(y)
_
b

e
y
M
Y
()
dP
Y
(y)
1
M
2
X
()
_
b

e
y
dP
Y
(y)
lr|W b|
_

_
bx
1

dP
X
()
(x
2
)dP
X
()
(x
1
)

_
bx
1

e
x
2
M
X
()
dP
X
(x
2
)
e
x
1
M
X
()
dP
X
(x
1
)

1
M
2
X
()
_

_
bx
1

e
(x
1
+x
2
)
dP
X
(x
2
)dP
X
(x
1
)
Thc cquaity ot thc aovc two tcrm- can thcn c jrovcd y notin that tor a
tunction f(),
_
b

f(y)dP
Y
(y)
_

_
bx
1

f(x
1
+ x
2
)dP
X
(x
2
)dP
X
(x
1
).
2
\c thcn jrovidc a owcr ound to lr|Y
()
> 0| lr|X
()
1
+X
()
2
+ +X
()
n
>
0|, whcrc E|X
()
j
| 0 tor 1 j n
Law ot ltcratcd Loarithm 9-50
Question Can wc -ay morc aout X
()
1
+ X
()
2
+ + X
()
n
with X
()
j

n
j=1
iid and E|X
()
j
| 0
Law ot ltcratcd Loarithm 9-51
lt X
j

j=0
arc iid with zcro mcan and nitc variancc
2
(and imjicity a mo-
mcnt- ot X cxi-t and can c dctcrmincd y takin thc dcrivativc- ot thc momcnt
cncratin tunction), thcn
1. lr
_
im
n
X
1
+ X
2
+ + X
n
n
0
_
1 (Stron aw ot arc numcr-)
2. lr
_
im-uj
n
X
1
+ X
2
+ + X
n

_
2no o(n)
1
_
1 (Law ot itcratcd oarithm)
3. im
n
lr
_
X
1
+ X
2
+ + X
n

n
y
_
(y) (Ccntra imit thcorcm)
whcrc () i- thc cdt ot thc zcro-mcan unit-variancc Gau--ian di-triution
Implication Thc aw ot thc itcratcd oarithm tc- u- that -um ot a iid -amjc-,
normaizcd y
_
2no o(n), away- ajjroachc- 1 innitcy ottcn, rcardc-- ot
thc marina di-triution'
Law ot ltcratcd Loarithm 9-52
im-uj
n
a
n
a it, and ony it, ( > 0)( N)( n > N) a
n
a < and
( > 0)( N)( n > N) a
n
a >
Recall im
n
a
n
a it, and ony it, ( > 0)( N)( n > N) a
n
a < and
( > 0)( N)( n > N) a
n
a >
lndccd, im-uj
n
a
n
im
n
-uja
n
, a
n+1
, a
n+2
, . . . im
n
-uj
kn
a
k

Lcncc, to jrovc
lr
_
im-uj
n
X
1
+ X
2
+ + X
n

_
2no o(n)
1
_
1
it -ucc- to jrovc that tor cvcry jo-itivc , (whcrc i- undcr-tood a- countac
in it- ranc),
P
__
x 1

( N)( n > N)
x
1
+ x
2
+ + x
n

_
2no o(n)
1 <
__
1
and
P
__
x 1

( N)( n > N)
x
1
+ x
2
+ + x
n

_
2no o(n)
1 >
__
1.
Law ot ltcratcd Loarithm 9-53
Lquivacnty,
P
_

_
N=1

n=N
_
x 1

x
1
+ x
2
+ + x
n

_
2no o(n)
1 <
__
1
and
P
_

N=1

_
n=N
_
x 1

x
1
+ x
2
+ + x
n

_
2no o(n)
1 >
__
1.
Or cquivacnty, y Lc `oran- aw,
P
_

N=1

_
n=N
_
x 1

x
1
+ x
2
+ + x
n

_
2no o(n)
1
__
0
and
P
_

N=1

_
n=N
_
x 1

x
1
+ x
2
+ + x
n

_
2no o(n)
1 >
__
1.
Law ot ltcratcd Loarithm 9-54
Or cquivacnty (a- wrotc in Liin-cy- ook),
lr
_
X
1
+ X
2
+ + X
n
(1 + )
_
2no o(n) io
_
0
and
lr
_
X
1
+ X
2
+ + X
n
> (1 )
_
2no o(n) io
_
1.
ln ordcr to rcducc thc contu-ion, l jcr-onay jrctcr to di-tinui-h ctwccn thc
cvcnt and thc -ct Thi- i- why l jut lr|cvcnt| and u-c P(-ct), whcrc thc
cvcnt i- dcncd throuh -omc random variac-, and thc jroaiity ot a -ct i-
mca-urcd y jroaiity mca-urc P Liin-cy- ook -omctimc- mixc- thc two
tocthcr, which may c contu-cd to cinncr-
Assumption \ithout o-- ot cncraity, wc a--umc that thc variancc
2
i- unity
lroot ot aw ot itcratcd oarithm 9-55
ln ordcr to jrovc thc jrcviou- two cquaitic-, wc nccd two jrciminary thcorcm-
Theorem 9.4 Lct S
n
X
1
+ X
2
+ + X
n
, whcrc X
j

j=1
arc iid with
zcro mcan and unit variancc (and imjicity a momcnt- ot X
j
cxi-t and can c
dctcrmincd y takin thc dcrivativc ot thc momcnt cncratin tunction) Sujjo-c
that thc jo-itivc con-tant -cqucncc a
1
, a
2
, . . . , a
n
, . . . -ati-c-
a
n
and
a
n

n
0.
Thcn thcrc cxi-t- a -cqucncc
1
,
2
, . . . with
n
0 -uch that
P
_
S
n
a
n

e
a
2
n
(1+
n
)/2
.
Recall S
n
/

n convcrc- in di-triution to zcro-mcan unit-variancc Gau--ian So

P
_
S
n
a

_

a
1

2
e
s
2
/2
ds

2a
e
a
2
/2
_
1
1
a
2
+
1 3
a
4

1 3 `
a
6
+
_
,
whcrc thc a-t cquaity hod- tor a > 0 ln thi- thcorcm, wc dividc S
n
y a ittc
arcr quantity than

n
lroot ot aw ot itcratcd oarithm 9-56
Proof: O-crvc that
lr|S
n
a
n

n| lr|X
1
+ X
2
+ + X
n
a
n

n|
lr|(X
1
a
n
/

n) + (X
2
a
n
/

n) + + (X
n
a
n
/

n) 0|.
Thcn y cttin Y (X
1
a
n
/

n) + (X
2
a
n
/

n) + + (X
n
a
n
/

n) and
notin that X
j
a
n
/

n
n
j=1
arc iid, < E|Y | a
n

n < 0 and tor a

-ucicnty arc n,
lr|Y > 0|
lr|(X
1
a
n
/

n > 0) and (X
2
a
n
/

n) > 0 and and (X

n
a
n
/

n) > 0|

_
lr|X
1
a
n
/

n > 0|
_
n
> 0 (Thc -trict jo-itivity i- jrovcd cow)
\c caim that ( > 0) lr|X
1
> | > 0 ccau-c it it wcrc not truc, ic, it
( > 0) lr|X
1
> | 0, thcn lr|X
1
0| 1, which tocthcr with E|X
1
|
0 imjic- that lr|X
1
0| 1, which vioatc- E|X
2
1
| 1 Accordiny, a-
a
n
/

n 0, thcrc cxi-t- N -uch that tor n > N, a

n
/

n < , and hcncc, tor

n > N, lr|X
1
> a
n
/

n| lr|X
1
> | > 0 2
So wc can ajjy thc jrcviou-y dcrivcd ujjcr and owcr ound- to lr|Y 0|
lroot ot aw ot itcratcd oarithm 9-57
Denition (o-notation) Sujjo-c that g(x) , 0 tor a x , a in -omc ojcn
intcrva containin a Thcn f(x) i- ittc-oh ot g(x) or f(x) i- ot -macr ordcr
than g(x), dcnotcd y
f(x) o(g(x)) a- x a,
it
im
xa
f(x)
g(x)
0.
Denition (O-notation) Sujjo-c that g(x) , 0 tor a x , a in -omc ojcn
intcrva containin a f(x) i- i-oh ot g(x) or f(x) i- ot thc -amc ordcr a- g(x),
dcnotcd y
f(x) O(g(x)) a- x a,
it
im
0
-uj
x1:[xa[<

f(x)
g(x)

< .
lroot ot aw ot itcratcd oarithm 9-58
M
Y
(
n
) lr|Y 0| M
Y
(
n
)
E
2
|(Y
(
n
)
)
2
|
!E|(Y
(
n
)
)
4
|
cxj
_

!
n
E|(Y
(
n
)
)
4
|
E
3/2
|(Y
(
n
)
)
2
|
_
.
Upper bound M
Y
(t)
n

i=1
M
(X
i
a
n
/

n)
(t) M
n
(Xa
n
/

n)
(t)
M
/
Y
(t) nM
/
(Xa
n
/

n)
(t)M
n1
(Xa
n
/

n)
(t)

n
-ati-c- M
/
(Xa
n
/

n)
(
n
) 0
Lquivacnty,
n
-ati-c- C
/
(Xa
n
/

n)
(
n
) C
/
X
(
n
) a
n
/

n 0
Thc zcro-mcan a--umjtion imjic- that C
/
X
(0) 0 A-o, thc -tricty
jo-itivc -ccond momcnt indicatc- thc -trict convcxity ot C
X
(t), which in
turn imjic- thc -trict incrca-innc-- ot C
/
X
(t) in t A- a rc-ut, a
n
/

n
0 a- n ivc- that
n
0 a- n , ccau-c C
/
X
(
n
) a
n
/

n
Ly Tayor cxjan-ion,
C
X
(t) C
X
(0) + C
/
X
(0)t +
C
//
X
(0)
2'
t
2
+ O(t
3
)
1
2
t
2
+ O(t
3
) a- t 0,
trom which wc otain (y takin thc dcrivativc)
C
/
X
(
n
)
n
+ O(
2
n
) a
n
/

n a-
n
0 (or a- n ),
lroot ot aw ot itcratcd oarithm 9-59
which imjic-
n
a
n
/

n +O(a
2
n
/n) 0 a- n . hcncc,
n
i- cvcntuay
nite
im-uj
n
[
n
a
n
/

n[

2
n
< C imjic- ( N)( n > N)[
n
a
n
/

n[ < C
2
n
,
or cquivacnty,

n
C
2
n
< a
n
/

n <
n
+ C
2
n
.
Lct N
1
c thc -mac-t n -ati-tyin 1 C
n
> 1/2 tor a n > N
1
Thcn
tor n > maxN, N
1
, a
2
n
/n >
2
n
(1 C
n
)
2
, which concudc- that
( n > maxN, N
1
)
[
n
a
n
/

n[
a
2
n
/n
<
C
2
n

2
n
(1 C
n
)
2

C
(1 C
n
)
2
< !C.
(Thi- i- onc ot thc rca-on- why thc thcorcm a--umc- a
n
/

n 0. thc thcorcm
doc- rcquirc
n
0)
`otay,
n
i- now dependent on n
lroot ot aw ot itcratcd oarithm 9-60
o
n
o M
(Xa
n
/

n)
(
n
) C
(Xa
n
/

n)
(
n
) C
X
(
n
) a
n

n
/

_
1
2

2
n
+ O(
3
n
)
_
|
n
+ O(
2
n
)|
n

1
2

2
n
+ O(
3
n
) a- n
\c thcn otain

n
n
lr|Y 0|
M
Y
(
n
) lr|Y 0| M
Y
(
n
)
E
2
|(Y
(
n
)
)
2
|
!E|(Y
(
n
)
)
4
|
cxj
_

!
n
E|(Y
(
n
)
)
4
|
E
3/2
|(Y
(
n
)
)
2
|
_
.
Lower bound
M
Y
(t) M
n
(Xa
n
/

n)
(t)
M
Y
(
n
) M
n
(Xa
n
/

n)
(
n
)
n
n
C
Y
(t) nC
(Xa
n
/

n)
(t) n|C
X
(t) a
n
t/

n|
E|(Y
(
n
)
)
2
| C
//
Y
(
n
)
(0) C
//
Y
(
n
)
nC
//
X
(
n
)
n
2,n
whcrc
2,n
C
//
X
(
n
)
lroot ot aw ot itcratcd oarithm 9-61
E|(Y
(
n
)
)
4
| C
(4)
Y
(
n
)
(0) + 3E
2
|(Y
(
n
)
)
2
| C
(4)
Y
(
n
) + 3n
2

2
2,n
nC
(4)
X
(
n
) + 3n
2

2
2,n
n
4,n
+ 3n
2

2
2,n
,
whcrc
4,n
C
(4)
X
(
n
)
Thcn
E
2
|(Y
(
n
)
)
2
|
!E|(Y
(
n
)
)
4
|

E
2
|(Y
(
n
)
)
2
|/n
2
!E|(Y
(
n
)
)
4
|/n
2

1
12 + !
4,n
/(
2
2,n
n)
and
!
n
E|(Y
(
n
)
)
4
|
E
3/2
|(Y
(
n
)
)
2
|

!
n
|n
4,n
+ 3n
2

2
2,n
|
n
3/2

3/2
2,n
!
n
_
3 +
4,n
/(
2
2,n
n)
_

2,n
n
Accordiny,
lr|Y 0|
n
n

1
12 + !
4,n
/(
2
2,n
n)
cxj
_
!
n
_
3 +
4,n
/(
2
2,n
n)
_

2,n
n
_
lroot ot aw ot itcratcd oarithm 9-62
Thi- concudc- to

n
n
lr|Y 0|
n
n

1
12 + !
4,n
/(
2
2,n
n)
cxj
_
!
n
_
3 +
4,n
/(
2
2,n
n)
_

2,n
n
_
,
whcrc (a- n or t 0 tor thc O-notation)
_

_
C
X
(t)
1
2
t
2
+ O(t
3
)
_

1
2
t
2
+
E|X
3
|
3'
t
3
+ O(t
4
) tor thc comjutation ot C
(4)
X
(t)
_

n

a
n

n
+ O
_
a
2
n
n
_ _
or
a
n

n

n
+ O(
2
n
)
_
o
n

1
2

2
n
+ O(
3
n
)
a
2
n
2n
+ O
_
a
3
n
n
3/2
_

2,n
C
//
X
(
n
) 1 + O(
n
) 1 + O
_
a
n

n
_

4,n
C
(4)
X
(
n
) O(1)
Lcncc,

4,n

2
2,n

O(1)
_
1 + O
_
a
n

n
__
2
O(1) a- n
lroot ot aw ot itcratcd oarithm 9-63
A- a rc-ut, y cttin lr|Y 0| e
a
2
n
(1+
n
)/2
, wc otain (a- n tor thc
O-notation)
no
n

1
2
a
2
n
(1 +
n
) no
n
o
_
12 + !

4,n

2
2,n
n
_
!
n
_
3 +

4,n

2
2,n
n
_
_

2,n
n

a
2
n
2
+ O
_
a
3
n
n
1/2
_

1
2
a
2
n
(1 +
n
)
a
2
n
2
+ O
_
a
3
n
n
1/2
_
o
_
12 + O
_
1
n
__
!
_
a
n

n
+ O
_
a
2
n
n
___
3 + O
_
1
n
__

_
1 + O
_
a
n

n
__
n
O
_
a
n

n
_

n
O
_
a
n

n
_
+
2
a
2
n
o
_
12 + O
_
1
n
__
+ S
_
1
a
n
+ O
_
1

n
___
3 + O
_
1
n
__

1 + O
_
a
n

n
_
Thu-, it a
n
and a
n
/

n 0, thcn
n
0 2
lroot ot aw ot itcratcd oarithm 9-64
Theorem 9.6 Lct S
n
X
1
+ +X
n
, whcrc X
i

i=1
arc iid with mcan 0 and
variancc 1 (and a momcnt- ot X
i
cxi-t, which can c dctcrmincd y takin thc
dcrivativc- ot it- momcnt cncratin tunction) A-o, ct lr|S
0
0| 1 Thcn tor

2,
lr
_
maxS
0
, S
1
, . . . , S
n

n

_
2 lr
_
S
n

2
_
.
Proof: Lcnotc M
n
maxS
0
, S
1
, . . . , S
n
Thcn M
n
i- non-ncativc and non-
dccrca-in in n
lr
_
M
n

n

_
lr
__
M
n

n

M
n1

n
<
_

_
M
n

n

M
n1

n

M
n2

n
<
_

_
M
n

n

M
n1

n

M
1

n

M
0

n
<
__
lr
__
M
n

n
>
M
n1

n
_

_
M
n1

n
>
M
n2

n
_

_
M
1

n
>
M
0

n
__

j=1
lr
_
M
j

n
>
M
j1

n
_
lroot ot aw ot itcratcd oarithm 9-65
Lcncc,
lr
_
M
n

n

_
lr
__
M
n

n

_

_
S
n

2
__
+ lr
__
M
n

n

_

_
S
n

n
<

2
__
lr
__
M
n

n

_

_
S
n

2
__
+
n

j=1
lr
__
M
j

n
>
M
j1

n
_

_
S
n

n
<

2
__
lr
_
S
n

2
_
+
n

j=1
lr
__
M
j

n
>
M
j1

n
_

_
S
n

n
<

2
__
.
lroot ot aw ot itcratcd oarithm 9-66
Sincc
_
M
j

n
>
M
j1

n
_
imjic-
S
j

n
, wc havc
lr
__
M
j

n
>
M
j1

n
_

_
S
n

n
<

2
__
lr
__
M
j

n
>
M
j1

n
_

_
S
n

n
<
S
j

2
__
lr
__
M
j

n
>
M
j1

n
_

_
S
j
S
n

n
>

2
__
lr
_
M
j

n
>
M
j1

n
_
lr
_
S
j
S
n

n
>

2
_
(M
j
, M
j1
ony dcjcnd- on X
1
, . . . , X
j
. yct S
j
S
n
ony dcjcnd- on X
j+1
, . . . , X
n
.
So thcy arc indcjcndcnt cvcnt-)
lr
_
M
j

n
>
M
j1

n
_
lr
_
[S
j
S
n
[

n
>

2
_
lr
_
M
j

n
>
M
j1

n
_
\ar|S
n
S
j
|
2n
(y Chcy-hcv- incq)
lr
_
M
j

n
>
M
j1

n
_
(n j)
2n

1
2
lr
_
M
j

n
>
M
j1

n
_
.
lroot ot aw ot itcratcd oarithm 9-67
Con-cqucnty,
lr
_
M
n

n

_
lr
_
S
n

2
_
+
n

j=1
lr
__
M
j

n
>
M
j1

n
_

_
S
n

n
<

2
__
lr
_
S
n

2
_
+
1
2
n

j=1
lr
_
M
j

n
>
M
j1

n
_
lr
_
S
n

2
_
+
1
2
lr
_
M
n

n

_
2
lroot ot aw ot itcratcd oarithm 9-68
Theorem 9.5 (Law of the Iterated Logarithm) Lct S
n
X
1
+ +X
n
,
whcrc X
i

i=1
arc iid with mcan 0 and variancc 1 (and a momcnt- ot X
i
cxi-t, which can c dctcrmincd y takin thc dcrivativc- ot it- momcnt cncratin
tunction) Thcn
P
_
im-uj
n
S
n
_
2no o(n)
1
_
1.
Proof:
1. lr
_
S
n
(1 + )
_
2no o(n) io
_
0
l havc modicd thc jroot in Liin-cy- ook -o that thc jroot i- morc
-traihttorward
Choo-c a jo-itivc -uch that 1 <
3
< 1 +
Lct n
k

k
| >
k
1, and
tor k o

(e
e
+ 1) (ic,
k
1 e
e
, which imjic- o o(n
k
) 1),
ct x
k

_
2 o o(n
k
)

2
lroot ot aw ot itcratcd oarithm 9-69
Thcn Thcorcm- 9! and 9o ivc that tor k o

(e
e
+1) and tor -omc
k
0,
lr
_
M
n
k

n
k
x
k
_
2 lr
_
S
n
k

n
k
x
k

2
_
(Thcorcm 9o)
2 cxj
_

1
2
(x
k

2)
2
(1 +
k
)
_
(Thcorcm 9!)
2 cxj
_

2
o(k)
(x
k

2)
2
(1 +
k
)
2
2
o(k)
_
.
Sincc
im
k
(x
k

2)
2
(1 +
k
)
2
2
o(k)
im
k
(
_
2 o o(
k
)

2)
2
2
2
o(k)
(Lccau-c
k
0)
im
k
(
_
o o(
k
) 1)
2

2
o(k)
im
k
(
_
o(k) + o o() 1)
2

2
o(k)
1,
thcrc cxi-t- K -uch that tor k K,
(x
k

2)
2
(1 +
k
)
2
2
o(k)

1

.
lroot ot aw ot itcratcd oarithm 9-70
Accordiny, tor k K
0
maxK, o

(e
e
+ 1),
lr
_
M
n
k

n
k
x
k
_
2 cxj o(k)
2
k

.
`ow tor n e
e

xcd, thcrc cxi-t- k -uch that n

k1
< n n
k
, and
_
2no o(n)
_
2(n
k1
+ 1) o o(n
k1
+ 1) >
_
2
_
n
k

_
o o
_
n
k

_
,
Thc a-t -trict incquaity toow- trom
n
k1

k1
| >
k1
1

k

1

k
|

1
n
k

1
which imjic- that (Thi- i- thc ony -tcj rcquirin
3
< (1 + ))
(1 + )
_
2no o(n) >
3
_
2
_
n
k

_
o o
_
n
k

_

2
_
2n
k
o o
_
n
k

_

_
2n
k
o o (n
k
) (-incc
2
> )
lroot ot aw ot itcratcd oarithm 9-71
Claim: Givcn > 1, f

(x) o o
_
x

o o(x) 0 tor x e
e

Proof: Thc caim can c vaidatcd y f

/

(x)
( 1) o(x) + o()
x o(x)|o(x) o()|
> 0 tor
x e
e

and f

(e
e

) o|e

o()| 1 > 0 tor any > 1 2

Accordiny,
_
S
n
(1 + )
_
2no o(n)
_
imjic- that
M
n
k
_
S
n
(1 + )
_
2no o(n)
_

_
2n
k
o o (n
k
),
whcrc k i- thc uniquc intccr -ati-tyin
k1
| < n
k
| (namcy, k
,o

(n)|) A- a con-cqucncc,
lr
_
S
n
(1 + )
_
2no o(n) io in n
_
lr
_
M
n
k

_
2n
k
o o (n
k
) io in k
_
.
Theorem 4.3 (First Borel-Cantelli Lemma) lt

n=1
P(A
n
) convcrc-, thcn
P
_
im-uj
n
A
n
_
P(A
n
io) 0
lroot ot aw ot itcratcd oarithm 9-72
Ly thc r-t Lorc-Cantci cmma,

k=1
lr
_
M
n
k

_
2n
k
o o (n
k
)
_

K
0

k=1
lr
_
M
n
k

_
2n
k
o o (n
k
)
_
+

k=K
0
+1
lr
_
M
n
k

_
2n
k
o o (n
k
)
_
K
0
+

k=K
0
+1
2
k

K
0
+
_

K
0
2
x

dx
K
0
+
2
1
1
K
1
0
< ,
wc otain (Thc cow cquaity hod- without thc condition that
3
< (1 +))
lr
_
M
n
k

_
2n
k
o o (n
k
) io in k
_
0.
Thi- comjctc- thc jroot ot thc r-t jart
lroot ot aw ot itcratcd oarithm 9-73
2. lr
_
S
n
> (1 )
_
2no o(n) io
_
1
Choo-c -ati-tyin > max1, 9/
2
Takc n
k

k
|
Ior any k, ct m
k
n
k
n
k1
and a
k

_
1
1

_
_
2n
k
o o(n
k
)

m
k

Thcn a
k
and
a
k

m
k
0 a- k
a
k

_
1
1

_
_
2n
k
o o(n
k
)

n
k

_
1
1

_
_
2 o o(n
k
)

im
k
a
k

m
k
im
k
_
1
1

_
_
2n
k
o o(n
k
)
m
k
im
k
_
1
1

_
_
2
k
o o(
k
)

k1
im
k

2 o o(
k
)

k
0
lroot ot aw ot itcratcd oarithm 9-74
Thcorcm 9! thcn imjic-
lr
_
S
n
k
S
n
k1

_
1
1

_
_
2n
k
o o(n
k
)
_
lr
_
X
n
k1
+1
+ + X
n
k
a
k

m
k

cxj
_

1
2
a
2
k
(1 +
k
)
_
tor -omc
k
0
cxj
_

1
2
_
_
_
_
_
1
1

_
_
2n
k
o o(n
k
)

m
k
_
_
_
_
2
(1 +
k
)
_

_
cxj
_

( 1)
2
|n
k
o o(n
k
)|(1 +
k
)

2
(n
k
n
k1
)
_
cxj
_

( 1)
2
|
k
o o(
k
)|(1 +
k
)

2
(|
k
1|
k1
)
_
(-incc
k
1 < n
k

k
and n
k1

k1
)
cxj
_

( 1)
2
|
k
o o(
k
)|(1 +
k
)

2
(
k

k1
)
_
1
1

k1
_
_

_
cxj
_

( 1) o o(
k
)(1 +
k
)

_
1
1

k1
_
_

_
lroot ot aw ot itcratcd oarithm 9-75
A-
k
0, thcrc cxi-t- K
1
-uch that
k
<
1
2 1
tor a k K
1

A-o, k
o() + o(2 1) o( 1)
o()
it, and ony it,
1

k1

1
2 1

Lcncc, tor k K
2
max
_
K
1
,
o() + o(2 1) o( 1)
o()
_
,
lr
_
S
n
k
S
n
k1

_
1
1

_
_
2n
k
o o(n
k
)
_
cxj
_

( 1) o o(
k
)(1 +
k
)

_
1
1

k1
_
_

_
cxj
_

( 1) o o(
k
)
_
1 +
1
2 1
_

_
1
1
2 1
_
_

_
cxj
_
o o(
k
)
_

1
k o()
.
lroot ot aw ot itcratcd oarithm 9-76
Theorem 4.4 (Second Borel-Cantelli Lemma) lt A
n

n=1
torm- an indc-
jcndcnt -cqucncc ot cvcnt- tor a jroaiity mca-urc P, and

n=1
P(A
n
) divcrc-,
thcn P
_
im-uj
n
A
n
_
1
Sincc
__
S
n
k
S
n
k1

_
1
1

_
_
2n
k
o o(n
k
)
__

k=1
arc indcjcndcnt cvcnt-,
and

k=1
lr
_
S
n
k
S
n
k1

_
1
1

_
_
2n
k
o o(n
k
)
_

k=K
2
lr
_
S
n
k
S
n
k1

_
1
1

_
_
2n
k
o o(n
k
)
_

k=K
2
1
k o()
,
it toow- y thc -ccond Lorc-Cantci cmma that with jroaiity 1, S
n
k

S
n
k1

_
1
1

_
_
2n
k
o o(n
k
) innitcy ottcn in k
lroot ot aw ot itcratcd oarithm 9-77
`ow wc can ct

X
n
X
n
, and ct

M
n
and

S
n
c rc-jcctivcy thc countcrjart-
ot M
n
and S
n
tor

X
n
(and n
k

k
|), and ajjy thc jroot in thc r-t jart
with

2 to -how that (ct -idc 9-2)(Thi- hod- without

3
< (1 + ))
lr
_

M
n
k

_
2n
k
o o (n
k
) io in k
_
0.
Lcncc, it i- with jroaiity 1 that S
n
k1


S
n
k1


M
n
k1
<
_
2n
k1
o o (n
k1
)
tor a ut nitcy many k
O-crvc that
_
2n
k1
o o (n
k1
)
2

_
2n
k
o o (n
k
).
Thc vaidity ot thc aovc incquaity toow-
Takc n
k1

k1
|
k1

n
k
+ 1

2n
k

tor n
k1
out-idc o o()
Takc n
k1
n
k
tor n
k1
in-idc o o()
A- a rc-ut, it i- with jroaiity 1 that S
n
k1

2

_
2n
k
o o (n
k
) tor a
ut nitcy many k
lroot ot aw ot itcratcd oarithm 9-78
To -ummarizc, it i- with jroaiity 1 that tor innitcy many k,
S
n
k

_
1
1

_
_
2n
k
o o(n
k
) + S
n
k1

_
1
1

_
_
2n
k
o o(n
k
)
2

_
2n
k
o o (n
k
)

_
1
1

_
_
2n
k
o o(n
k
)

_
1
1

_
_
2n
k
o o(n
k
)

_
1
3

_
_
2n
k
o o(n
k
)
(1 )
_
2n
k
o o(n
k
).
2
Final note
Thc oriina -tatcmcnt ot thc aw ot thc itcratcd oarithm wa- duc to A Y Ihinchin
in 192!
Anothcr -tatcmcnt ot thc aw ot thc itcratcd oarithm wa- ivcn y A ` Io-
moorov in 1929