Steven R.

Dunbar
Department of Mathematics
203 Avery Hall
University of Nebraska-Lincoln
Lincoln, NE 68588-0130
http://www.math.unl.edu
Voice: 402-472-3731
Fax: 402-472-8466

Topics in
Probability Theory and Stochastic Processes
Steven R. Dunbar

Law of the Iterated Logarithm

Rating
Mathematicians Only: prolonged scenes of intense rigor.

1
Section Starter Question

Key Concepts
1. The Law of the Iterated Logarithm tells very precisely how far the num-
ber of successes in a coin-tossing game will make excursions from the
average value.

2. The Law of the Iterated Logarithm is a high-point among increasingly

precise limit theorems characterizing how far the number of successes in
a coin-tossing game will make excursions from the average value. The
theorems start with the Strong Law of Large Numbers and the Central
Limit Theorem, to Hausdorff’s Estimate, and the Hardy-Littlewood
Estimate leading to the Law of the Iterated Logarithm.

3. Khinchin’s Law of the Iterated Logarithm says: Almost surely, for all
 > 0, there exist infinitely many n such that
√ p
Sn − np > (1 − ) n 2p(1 − p) ln(ln(n))

and furthermore, almost surely, for all  > 0, for every n larger than a
threshold value N
√ p
Sn − np < (1 + ) n 2p(1 − p) ln(ln(n)).

2
Vocabulary
1. The limsup, abbreviation for limit superior is a refined and general-
ized notion of limit, being the largest dependent-variable subsequence
limit. That is, among all subsequences of independent-variable val-
ues tending to some independent-variable value, usually infinity, there
will be a corresponding dependent-variable subsequence. Some of these
dependent-variable sequences will have limits, and among all these, the
largest is the limsup.

2. The liminf, abbreviation for limit inferior is analogous, it is the least

of all dependent-variable subsequence limits.

3. Khinchin’s Law of the Iterated Logarithm says: Almost surely, for

all  > 0 there exist infinitely many n such that
√ p
Sn − np > (1 − ) n 2p(1 − p) ln(ln(n))

and furthermore, almost surely, for all  > 0, for every n larger than a
threshold value N
√ p
Sn − np < (1 + ) n 2p(1 − p) ln(ln(n)).

Mathematical Ideas
Overview
We again consider the number of successes in a coin-tossing game. That
is, we consider the sum Sn where the independent, identically distributed
random variables in the sum Sn = X1 + · · · + Xn are the Bernoulli random
variables Xi = +1 with probability p and Xi = 0 with probability q = 1 − p.
Note that the mean µ = p is and the variance is σ 2 = p(1 − p) for each of
the summands Xi .

3
 The Strong Law of Large Numbers says that
Sn − np
lim =0
n→∞ n
with probability 1 in the sample space of all possible coin flips. This says the
denominator n is “too strong”, it “condenses out” all variation in the sum
Sn .
The Central Limit Theorem applied to this sequence of coin flips says
Sn − np
lim p =Z
n→∞ np(1 − p)

where Z ≡ N (0, 1) is a normal random variable and the limit is interpreted

as convergence in distribution. In fact, this implies that for large n about
68% of the points in the sample space of all possible coin flips satisfy
 
 S − np 
n
≤1
 
p
 np(1 − p) 

and about 95% of the points in the sample space of all possible coin flips
satisfy  
 S − np 
n
 ≤ 2.
 
p
 np(1 − p) 
√
This says the denominator n is “too weak”, it doesn’t condense out enough
information. In fact, using the Kolmogorov zero-one law and the Central
Limit Theorem, almost surely
Sn − np
lim inf p = −∞
n→∞ np(1 − p)

and almost surely

Sn − np
lim sup p = +∞.
n→∞ np(1 − p)
The Strong Law and the Central
√ Limit Theorem together suggest that
“somewhere in between n and n” we might be able to make stronger state-
ments about convergence and the variation in the sequence Sn .

4
 In fact, Hausdorff’s estimate tells us:
 
 Sn − np 
lim  =0
n→∞  n1/2+ 

with probability 1 in the sample space of all possible coin flips for all values
of  > 0. This says the denominator n1/2+ is “still too strong”, it condenses
out too much information.
Even better, Hardy and Littlewood’s estimate tells us:
 
 Sn − np 
lim √
  ≤ constant
n→∞  n ln n 
with probability 1 in the sample
p space of all possible coin flips for all values
of  > 0. In a way, this says n ln n) is “still a little too strong”, it condenses
out most information.
Khinchin’s Law of the Iterated Logarithm has a denominator that is “just
right.” It tells us very precisely how the deviations of the sums from the mean
vary with n. Using a method due to Erdös, it is possible to refine the law
even more, but for these notes a refinement is probably past the point of
diminishing returns.
Like the Central Limit Theorem, the Law of the Iterated Logarithm illus-
trates in an astonishingly precise way that even completely random sequences
obey precise mathematical laws.
Khinchin’s Law of the Iterated Logarithm says that:

Almost surely, for all  > 0, there exist infinitely many n such
that p p
Sn − np > (1 − ) np(1 − p) 2 ln(ln(n))
and furthermore, almost surely, for all  > 0, for every n larger
than a threshold value N depending on 
p p
Sn − np < (1 + ) np(1 − p) 2 ln(ln(n)).

These
√ appear in a slightly non-standard way, with the additional factor
2 ln ln n times the standard deviation from the Central Limit Theorem to
emphasize the similarity to and the difference from the Central Limit Theo-
rem.

5
Theorem 1 (Law of the Iterated Logarithm). With probability 1:

Sn − np
lim sup p = 1.
n→∞ 2np(1 − p) ln(ln(n))

This means that with probability 1 for any  > 0, only finitely many of the
events: √ p
Sn − np > (1 + ) n 2p(1 − p) ln(ln(n))
occur; on the other hand, with probability 1,
√ p
Sn − np > (1 − ) n 2p(1 − p) ln(ln(n))

occurs for infinitely many n.

For reasons of symmetry, for  > 0, the inequality

√ p
Sn − np < −(1 + ) n 2p(1 − p) ln(ln(n))

can only occur for finitely many n; while

√ p
Sn − np < −(1 − ) n 2p(1 − p) ln(ln(n))

must occur for infinitely many n. That is,

Sn − np
lim inf p = −1
n→∞ 2np(1 − p) ln(ln(n))

with probability 1.
Compare the Law of the Iterated Logarithm to the Central
p Limit The-
orem. The Central Limit Theorem, says that (Sn − np)/ np(1 − p) is ap-
proximately distributed as a N (0, 1) random variable for large n. Therefore,
for a large p
but fixed n, there is probability about 1/6 that the values of
(S
pn − np)/ np(1 − p) can exceed the standard deviation 1, or Sn − np >
np(1
p − p). For a fixed but large n, with probability about 0.025, (Sn −
np)/
p np(1 − p) can exceed twice the standard deviation 2, or (Sn − np) >
2 np(1 − p). The Law of the Iterated Logarithm tells us the more precise
information that there are infinitely many n such that
p
Sn − np > (1 − ) 2np(1 − p) ln(ln(n))

6
for any  > 0. The Law of the Iterated Logarithm does not tell us how
long we will have to wait between such repeated crossings however, and the
wait can be very, very long indeed, although it must (with probability 1)
eventually occur again. Moreover, the Law of the Iterated Logarithm tells
us in addition that
p
Sn − np < −(1 − ) 2np(1 − p) ln(ln(n))

must occur for infinitely many n.

Khinchin’s Law of the Iterated Logarithm also applies to the cumulative
fortune in a coin-tossing game, or equivalently, the position in a random
walk. Consider the independent Bernoulli random variables Yi = +1 with
probability p and Yi = −1 with probability q = 1−p. The mean is µ = 2p−1
and the variance is σ 2 = 4p(1−p) for each of the summands Yi . Then consider
the sum Tn = Y1 + · · · + Yn with mean (2p − 1)n and variance 4np(1 − p).
Since Yn = 2Xn − 1, then Tn = 2Sn − n and Sn = 21 Tn + n2 . Then applying
the Law of the Iterated Logarithm says that with probability 1 for any  > 0,
only finitely many of the events:
√ p
|Tn − n(2p − 1)| > (1 + )2 n 2p(1 − p) ln(ln(n))

occur; on the other hand, with probability 1,

√ p
|Tn − n(2p − 1)| > (1 − )2 n 2p(1 − p) ln(ln(n))

occurs for infinitely many n. This means that the fortune must (with prob-
ability 1) oscillate back and forth across the net zero axis infinitely often,
crossing the upper and lower boundaries:
p
±(1 − )2 2p(1 − p)n ln(ln(n))

The statement puts some strength behind an understanding of the long-term

swings backs and forth in value of a random process. It also implies a form
of recurrence, that is, a random walk must visit every integer value.
The Law of the Iterated Logarithm for Bernoulli trials stated here is a
special case of an even more general theorem first formulated by Kolmogorov
in 1929. It is also possible to formulate even stronger and more general
theorems! The proof here uses the Large Deviations and Moderate Deviations
results with the Borel-Cantelli Lemmas. In another direction, the Law of the

7
Iterated Logarithm can be proved using invariance theorems, so it is distantly
related to the Central Limit Theorem.
Figure 1 gives an impression of the growth of the function in the Law of
the Iterated Logarithm compared to the square root function. Figure 2 gives
an impression of the Law of the Iterated Logarithm by showing a piecewise
linearly connected graph of 2000 steps of Sn − np with p = q = 1/2. In
this figure, the random walk must return again to cross the blue curves with
(1 − ) = 0.9 infinitely many times, but may only cross the red curve with
1+ = 1.1 finitely many times. Of course, this is only a schematic impression
since a single random walk (possibly atypical, from the negligible set!) on
the finite interval 0 ≤ n ≤ 2000 can only suggest the almost sure infinitely
many crossings of (1 − )α(x) for any  > 0.
Figure 3 gives a comparison of impressions of four of the limit theorems.
The individual figures deliberately are “spaghetti graphs” to give an impres-
sion of the ensemble of sample paths. Each figure shows a different scaling of
the same 15 sample paths for a sequence of 100,000 fair coin flips, each path
a different color. Note that the steps axis has a logarithmic scale, meaning
that the shape of the paths is distorted although it still gives an impression of
the random sums. The top left figure shows Sn /n − p converging to 0 for all
paths in accord with the Strong
p Law of Large Numbers. The top right figure
plots the scaling (Sn − np)/ 2p(1 − p)n. For large values of steps the values
over all paths is a distribution ranging from about −2 to 2, consistent with
the Central Limit Theorem. The lower left figure plots (Sn − np)/n0.6 as an
illustration of Hausdorff’s Estimate with  = 0.1. It appears that the scaled
paths are very slowly converging to 0, the range for n p= 100,000 is within
[−0.5, 0.5]. The lower right figure shows (Sn − np)/ 2p(1 − p)n ln(ln(x))
along with lines ±1 to suggest the conclusions of the Law of the Iterated
Logarithm. It suggests that all paths are usually in the range [−1, 1] but
with each path making a few excursions outside the range.

Hausdorff ’s Estimate
Theorem 2 (Hausdorff’s Estimate). Almost surely, for any  > 0,
Sn − np = o n+1/2

as n → ∞.
Proof. The proof resembles the proof of the Strong Law of Large Numbers
for independent random variables with mean 0 and uniformly bounded 4th

8
y 50

40

30

20

10

0
0 1000 2000
x
p
Figure 1: The Iterated Logarithm function α(x) = 2p(1 − p)x ln(ln(x))
in green along with functions (1 + )α(x) in red andp(1 − )α(x) in blue, with
 = 0.1. For comparison, the square root function 2p(1 − p)x is in black.

9
y 50

40

30

20

10

0
0 1000 2000
x

Figure 2: An impression of the Law of the Iterated Logarithm using a

piecewise linearly connected graph of 2000 steps of Sn − np with p = q = 1/2
with the blue curve pwith (1 − )α(x) and the red curve with (1 + )α(x) for
 = 0.1 and α(x) = 2p(1 − p)x ln(ln(x)).

10
Figure 3: A comparison of four of the limit theorems. The individual figures
deliberately are “spaghetti graphs” to give an impression of the ensemble of
sample paths. Each figure shows a different scaling of the same 15 sample
paths for a sequence of 100,000 fair coin flips, each path a different color.
Note that the steps axis has a logarithmic scale, meaning that the shape of
the sample paths is distorted.

11
moments. That proof showed that using the independence and identical
distribution assumptions
 2
E Sn4 = nE X14 + 3n(n − 1)E X12 ≤ nC + 3n2 σ 4 ≤ C1 n2 .
   

Then adapting the proof of the Markov and Chebyshev inequalities with
fourth moments
E [|Sn |4 ] C 1 n2
≤ .
n4 n4
∞
P
Use the Corollary that if E [|Xn |] converges, then the sequence (Xn )n≥1
n=1
∞ E |S |4
P [ n ]
converges almost surely to 0. By comparison, n4
converges so that
n=1
Sn /n → 0 a.s. Using the same set of ideas

E [|Sn |4 ] C 1 n2
≤
nα n4α
provided that α > 3/4. Then using the same lemma Sn /nα → 0 for α > 3/4
for a simple version of Hausdorff’s Estimate.
Now adapt this proof to higher moments. Let k be a fixed positive integer.
n n
(Xk0 where Xk0 =
P P
Recall the definition Rn (ω) = Sn (ω)−np = (Xk −p) =
  k=1 k=1
Xk − p and consider E Rn2k . Expanding the product Rn2k results in a sum of
products of the individual random variables Xi0 of the form Xi01 Xi02 · · · Xi02k .
Each product Xi01 Xi02 · · · Xi02k results from a selection or mapping from indices
{1, 2, . . . , 2k} to the set {1, . . . , n}. Note that if an index j ∈ {1, 2, . . . , n} is
selected only once so that Xj0 appears only once in the product Xi01 Xi02 · · · Xi0k ,
then E Xi01 Xi02 · · · Xi0k = 0 by independence. Further notice that for all sets
 

of indices
E Xi01 Xi02 · · · Xi0k ≤ 1.
 

Thus X
E Rn2k = E Xi01 Xi02 · · · Xi02k ≤ N (k, n),
   
1≤i1 ,...,i2k ≤n

where N (k, n) is the number of functions from {1, . . . , 2k} to {1, . . . , n} that
take each value at least twice. Let M (k) be the number of partitions of
{1, . . . , 2k} into subsets each containing at least two elements. If P is such a
partition, then P has at most k elements. The number of functions N (k, n)

12
that are constant on each element of P is at most nk . Thus, N (k, n) ≤
nk M (k).
Now let  > 0 and consider
h
2k i
−−1/2
E n Rn ≤ n−2k−k N (k, n) ≤ n−2k M (k).

1
Choose k > 2
. Then
X h
2k i
E n−−1/2 Rn < ∞.
n≥1

Recall the Corollary 2 appearing in the section on the Borel-Cantelli

Lemma:

Let (Xn )n≥0 be a sequence of random variables. If ∞

P
n=1 E [|Xn |]
converges, then Xn converges to zero, almost surely.

By this corollary, the sequence of random variables

n−−1/2 Rn → 0

almost surely as n → ∞.
This means that for each  > 0, there is a negligible event (depending on
) outside of which n−−1/2 Rn converges to 0. Now consider a countable set
of values of  tending to 0. Since a countable union of negligible events is
negligible, then for each  > 0, n−−1/2 Rn converges to 0 almost surely.

Hardy-Littlewood Estimate
Theorem 3 (Hardy-Littlewood Estimate).
√ 
Sn − np = O n ln n a.s. for n → ∞.
√
Remark. The proof shows that Sn − np ≤ n ln n a.s. for n → ∞.
Proof. The proof uses the Large Deviations Estimate as well as the Borel-
Cantelli Lemma. Recall the Large Deviations Estimate says
 
Sn
P ≥ p +  ≤ e−nh+ () ,
n

13
where
   
p+ 1−p−
h+ () = (p + ) ln + (1 − p − ) ln .
p 1−p
 2
Note that as  → 0, h+ () = 2p(1−p) + O(3 ).
Note that  
Sn
P ≥ p +  = P [Sn − np ≥ n] ,
n
q
and take  = lnnn and note that
√
h √ i
−nh+ ln n

P Sn − np ≥ n ln n ≤ e n
.

Then r !  
ln n ln n 1
h+ = +o ,
n 2p(1 − p)n n


ln n 3/2 1


since O n
= o n
. Thus, the probability is less than or equal to the
following:
r !!  
ln n 1
exp −nh+ = exp − ln n + o (1)
n 2p(1 − p)
 
− ln n
= exp exp (o (1))
2p(1 − p)
−1
= n 2p(1−p) · exp (o (1)) .
 q  −1 −1
ln n
P
Hence exp −nh+ n
∼ n 2p(1−p) , and n≥1 n 2p(1−p) is convergent be-
cause of the following inequalities
1
p(1 − p) ≤
4
1
2p(1 − p) ≤
2
1
≥2
2p(1 − p)
−1
≤ −2
2p(1 − p)
−1
n 2p(1−p) ≤ n−2 .

14
Thus, X h √ i
P Sn − np > n ln n < ∞,
n≥1

and so h √ i
P Sn − np ≤ n ln n i.o. = 1.

Proof of Khinchin’s Law of Iterated Logarithm

Theorem 4 (Khinchin’s Law of Iterated Logarithm). Almost surely,

Sn − np
lim sup p = 1,
n→∞ 2p(1 − p)n ln (ln n)

and
Sn − np
lim inf p = −1.
n→∞ 2p(1 − p)n ln (ln n)
First establish a two lemmas, and for convenience, let
p
α(n) = 2p(1 − p)n ln (ln n).

Lemma 5. For all positive a and δ and large enough n,

2 (1+δ) 2 (1−δ)
(ln n)−a < P [Sn − np > aα(n)] < (ln n)−a .

Proof of Lemma 5. Recall that the Large Deviations Estimate gives

P [Rn ≥ aα(n)] = P [Sn − np ≥ aα(n)]

 
Sn aα(n)
=P −p≥
n n
  
aα(n)a
≤ exp −nh+ .
n
α(n)
Note that n
→ 0 as n → ∞. Then
2 3 !
a2
   
aα(n) α(n) α(n)
h+ = +O ,
n 2p(1 − p) n n

15
and so
α(n)3
   
aα(n) 2
nh+ = a ln (ln n) + O ≥ a2 (1 − δ) ln (ln n)
n n2
for large enough n. This means that
 
Sn 2
− p ≥ aα(n) ≤ exp −a2 (1 − δ) ln (ln n) = (ln n)−a (1−δ) .


P
n
p

Since ln (ln n) = o n1/6 , the results of the Moderate Deviations The-
orem apply to give
  " r #
Sn aα(n) Sn p(1 − p) p
P −p≥ =P −p≥ a 2 ln (ln n)
n n n n
1
exp −a2 ln (ln n)


∼√ p
2πa 2 ln (ln n)
1 2
= p (ln n)−a .
2a π ln (ln n)
 2

Since ln (ln n) = o (ln n)a δ ,
p

 
Sn aα(n) 2
P −p≥ ≥ (ln n)−a (1+δ)
n n
for large enough n.
Lemma 6 (12.5, Kolmogorov Maximal Inequality). Suppose (Yn )n≥1 are in-
dependent random variables. Suppose further that E [Yn ] = 0 and Var(Yn ) =
σ 2 . Define Tn := Y1 + · · · + Yn . Then
√ 
 
4 
P max Tk ≥ b ≤ P Tn ≥ b − 2σ n .
1≤k≤n 3
Remark. Lemma 6 is an example of a class of lemmas called maximal in-
equalities. Here are two more examples of maximal inequalities.
Lemma 7 (Karlin and Taylor, page 280). Let (Yn )n≥1 be identical indepen-
2
Pn random variables with E [Yn ] = 0 and Var(Yn ) = σ < ∞.
dently distributed
Define Tn = k=1 Yk . Then
 
 P max |Tk | >  ≤ nσ 2 .
2
0≤k≤n

16
Lemma 8 (Karlin and Taylor, page 280). Let (Xn )n≥0 be a submartingale for
which Xn ≥ 0. For λ > 0,
 
λP max Xk > λ ≤ E [Xn ] .
0≤k≤n

Proof of Lemma 6. Since the Yk ’s are independent, then Var(Tn − Tk ) =

(n − k)σ 2 for 1 ≤ k ≤ n. Chebyshev’s Inequality tells us that
 √  Var(Tn − Tk ) n−k 3
P |Tn − Tk | ≤ 2σ n ≥ 1 − 2
=1− ≥ .
4σ n 4n 4
Note that
  Xn
P max Tk ≥ b = P [T1 < b, . . . , Tk−1 < b, and Tk ≥ b]
0≤k≤n
k=1
n
X 4
≤ P [T1 < b, . . . , Tk−1 < b, and Tk ≥ b] · P [|Tn − Tk | ≤ 2σn]
k=1
3
n
4X  √ 
= P T1 < b, . . . , Tk−1 < b, and Tk ≥ b and |Tn − Tk | ≤ 2σ n
3 k=1
n
4X  √ 
≤ P T1 < b, . . . , Tk−1 < b, and Tk ≥ b and Tn ≥ b − 2σ n
3 k=1
4  √ 
≤ P Tn ≥ b − 2σ n .
3

Remark. Note that the second part of the Law of the Iterated Logarithm,
n −np
lim inf n→∞ Sα(n) = −1 follows by symmetry from the first part by replacing
p with (1 − p) and Sn with n − Sn .
Remark. The proof of the Law of the Iterated Logarithm proceeds in two
parts. First it suffices to show that
Sn − np
lim sup < 1 + η for η > 0, a.s.. (1)
n→∞ α(n)
The second part of the proof is to establish that
Sn − np
lim sup > 1 − η for η > 0, a.s. (2)
n→∞ α(n)

17
It will only take a subsequence to prove (2). However this will not be easy
because it will use the second Borel-Cantelli Lemma, which requires inde-
pendence.
Remark. The following is a simplified proof giving a partial result for integer
sequences with exponential growth. This simplified proof illustrates the basic
ideas of the proof. Fix γ > 1 and let nk := bγ k c. Then
2
P [Snk − pnk ≥ (1 + η)α(nk )] < (ln nk )−(1+η) (1−δ)
 2

= O k −(1+η) (1−δ) .

Choose δ so that (1 + η)2 (1 − δ) < 1. Then

X
P [Snk − nk p ≥ (1 + η)α(nk )] < ∞.
k≥1

By the first Borel-Cantelli lemma,

P [Snk − nk p ≥ (1 + η)α(nk ) i.o.] = 0,
and so  
Snk − nk p
P lim sup ≤ (1 + η) = 1,
k→∞ α(nk )
or
Snk − nk p
≤ (1 + η) a.s.
α(nk )
The full proof of the Law of the Iterated Logarithm takes more work to
complete.
Proof of (1) in the Law of the Iterated Logarithm. Fix η > 0 and let γ > 1
be a constant chosen later. For k ∈ Z, let nk = bγ k c. The proof consists of
showing that
X  
P max (Sn − np) ≥ (1 + η)α(nk ) < ∞.
n≤nk+1
k≥1

From Lemma 6
X  
P max (Sn − np) ≥ (1 + η)α(nk )
n≤nk+1
k≥1
4 h p i
≤ P Rnk+1 ≥ (1 + η)α(nk ) − 2 nk+1 p(1 − p) ,
3

18
where Rn = Sn − np. We do know that
 
P max (Sn − np) ≥ (1 + η)α(nk ) ≤
n≤nk+1
4 h p i
P Rnk+1 ≥ (1 + η)α(nk ) − 2 nk+1 p(1 − p) . (3)
3
√
Note that nk+1 = o (α(nk )) because this is approximately
p p p
γ k+1 compared to c1 γ k ln(ln γ γ ) = c1 γ k/2 ln(ln γ) + ln(k),
which is the same as
p
γ 1/2 compared to c1 c2 + ln(k).
p
Then 2 nk+1 p(1 − p) < 12 ηα(nk ) for large enough k. Using this inequality
in the right hand side of Equation (3), we get
 
4  
P max Sn − np ≥ (1 + η)α(nk ) ≤ P Snk+1 − nk+1 p ≥ (1 + η/2)α(nk ) .
n≤nk+1 3
√ √
Now, α(nk+1 ) ∼ γα(nk ). Choose γ so that 1 + η/2 > (1 + η/4) γ. Then
for large enough k, we have
(1 + η/2)α(nk ) > (1 + η/4)α(nk+1 ).
Now
 
4  
P max Sn − np ≥ (1 + η)α(nk ) ≤ P Snk+1 − nk+1 p ≥ (1 + η/4)α(nk+1 ) .
n≤nk+1 3
Use Lemma 5 with a = (1 − δ)−1 = (1 + η/4). Then we get
 
4
P max Sn − np ≥ (1 + η)α(nk ) ≤ (ln nk+1 )−(1+η/4)
n≤nk+1 3
for k large. Note that
(ln nk+1 )−(1+η/4) ∼ (ln γ)−(1+η/4) k −(1+η/4) ,
which is the general term of a convergent series so
X  
P max Rn ≥ (1 + η)α(nk ) < ∞.
n≤nk+1
k≥1

19
Then the first Borel-Cantelli Lemma implies that
max Rn ≥ (1 + η)α(nk ) i.o. with probability 0.
n≤nk+1

or equivalently
max Sn − np < (1 + η)α(nk ) a.s. for large enough k.
n≤nk+1

Then in particular
max Sn − np < (1 + η)α(nk ) a.s. for large enough k.
nk ≤n<nk+1

This in turn implies that almost surely

Sn − np < (1 + η)α(n).
for n > nk and large enough k which establishes (1).
Proof of (2) in the Law of the Iterated Logarithm. Continue with the proof
of Equation (2). To prove the second part, it suffices to show that there
exists nk so that Rnk ≥ (1 − η)α(nk ) i.o. almost surely. Let nk = γ k for γ
chosen later with γ ∈ Z sufficiently large. The proof will show
X h  η i
P Rγ n − Rγ n−1 ≥ 1 − α(γ n ) = ∞, (4)
n≥1
2

and also
−η
Rγ n−1 ≥ α(γ n ) a.s. for large enough n. (5)
2
dist.
Note that Rγ n − Rγ n−1 = Rγ n −γ n−1 . It suffices to consider
h  η n
i
P Rγ n −γ n−1 ≥ 1 − α(γ ) .
2
Note that
p
α(γ n − γ n−1 ) c (γ n − γ n−1 ) ln (ln (γ n − γ n−1 ))
= p
α(γ n ) cγ n ln (ln γ n )
v   
u
u  ln n ln γ + ln 1 − 1
t 1 γ
= 1−
γ ln (n ln γ)
r
1
→ 1− .
γ

20
Choose γ ∈ Z so that
η r
1− 2 1
η < 1− .
1− 4
γ
Then note that we can choose n large enough so that

1 − η2


α(γ n − γ n−1 )
<
1 − η4


α(γ n )
or  η  η
1−α(γ n ) < 1 − α(γ n − γ n−1 ).
2 4
Now considering equation (4), and the inequality above
h  η i h  η i
P Rγ n − Rγ n−1 ≥ 1 − α(γ n ) ≥ P Rγ n −γ n−1 ≥ 1 − α(γ n − γ n−1 ) .
2 4
Now using Lemma 5, with a = (1 + δ)−1 = 1 − η4 , we get

h  η i
P Rγ n − Rγ n−1 ≥ 1 − α(γ n ) ≥
2
  −(1− η4 )
n n−1 −(1− 4 )

η 1
ln γ − γ = n ln γ + ln 1 − .
γ

The series with this as its terms diverges. Thus, we see that Equation (4)
has been proven.
Now notice that
p
α(γ n ) = cγ n ln (ln γ n )
p
= cγ n (ln n + ln ln γ)

and so p
α(γ n−1 ) = cγ n−1 (ln(n − 1) + ln ln γ),
which means that
√ p
γα(γ n−1 ) = cγ n (ln(n − 1) + ln ln γ).
√ √
Thus, α(γ n ) ∼ γα(γ n−1 ). Now choose γ so that η γ > 4. Then ηα(γ n ) ∼
√
η γα(γ n−1 ) > 4α(γ n−1 ) for large enough n.

21
 Thus, we have
 
−η
α(γ ) ⊆ −Rγ n−1 ≥ 2α(γ n−1 )
n
 
Rγ n−1 ≤
2
since  
−η
α(γ ) ⊆ Rγ n−1 ≤ −2α(γ n−1 ) .
n
 
Rγ n−1 ≤
2
Now use (4) and we see that −Rγ n−1 < 2α(γ n−1 ) happens almost surely for
large enough n.
Now Rγ n − Rγ n−1 is a sequence of independent random variables, and so
the second Borel-Cantelli Lemma says that almost surely
 η
Rγ n − Rγ n−1 > 1 − α(γ n ) i.o.
2
Adding this with Equation (5), we get that
Rγ n > (1 − η) α(γ n ) i.o.
This is enough to show that
Sn − np
lim inf > 1 − η a.s.,
n→∞ α(n)
which is enough to show the only remaining part of Khinchin’s Law of the
Iterated Logarithm.

Sources
This section is adapted from: W. Feller, in Introduction to Probability Theory
and Volume I, Chapter III, and Chapter VIII, and also E. Lesigne, Heads
or Tails: An Introduction to Limit Theorems in Probability, Chapter 12,
American Mathematical Society, Student Mathematical Library, Volume 28,
2005. Some of the ideas in the proof of Hausdorff’s Estimate are adapted
from J. Lamperti, Probability: A Survey of the Mathematical Theory, Second
Edition, Chapter 8. Figure 3 is a recreation of a figure in the Wikipedia
article on the Law of the Iterated Law.

22
 Algorithms, Scripts, Simulations
Algorithm
Scripts
R script for comparison figures.
1 p <- 0.5
2 k <- 15
3 n <- 100000
4 coinFlips <- array ( runif ( n * k ) <= p , dim = c (n , k ) )
5 S <- apply ( coinFlips , 2 , cumsum )
6 steps <- c (1: n )
7
8 steps2 <- steps [2: n ]
9 S2 <- S [2: n , ]
10 steps3 <- steps [3: n ]
11 S3 <- S [3: n , ]
12

13 ones <- cbind ( matrix (1 ,n -2 ,1) , matrix ( -1 ,n -2 ,1) )

14
15 par ( mfrow = c (2 ,2) )
16
17 matplot (( S - steps * p ) / steps ,
18 log = " x " , type = " l " , lty = 1 , ylab = " " , main = " Strong Law
")
19 matplot (( S - steps * p ) / sqrt (2 * p * (1 - p ) * steps ) ,
20 log = " x " , type = " l " , lty = 1 , ylab = " " , main = " Central
Limit Theorem " )
21 matplot (( S - steps * p ) / ( steps ^(0.6) ) ,
22 log = " x " , type = " l " , lty = 1 , ylab = " " , main = " Hausdorff ’
s Estimate " )
23 # # matplot (( S2 - steps2 * p ) / sqrt ( steps2 * log ( steps2 ) ) , log =" x " ,
xlim = c (1 , n ) , type =" l " , lty = 1)
24 matplot ( steps3 , ( S3 - steps3 * p ) / sqrt (2 * p * (1 - p ) * steps3 * log ( log (
steps3 ) ) ) ,
25 log = " x " , xlim = c (1 , n ) , type = " l " , lty = 1 , ylab = " " ,
main = " Law of Iterated Logarithm " )
26 matlines ( steps3 , ones , type = " l " , col = " black " )

23
Problems to Work for Understanding
p p
1. The “multiplier of the variance 2p(1 − p)n” function ln(ln(n)) grows
very slowly. pTo understand how very slowly, calculate a table with
j
n = 10 and 2 ln(ln(n)) for j = 10, 20, 30, . . . , 100. (Remember, in
mathematical work above calculus, ln(x) is the natural logarithm, base
e, often written ln(x) in calculus and below to distinguish it from the
“common” or base-10 logarithm. Be careful, some software and tech-
nology cannot directly calculate with magnitudes this large.)

2. Consider the sequence

(−1)n
an = (−1)bn/2c +
n
for n = 1, 2, 3 . . .. Here bxc is the “floor function”, the greatest integer
less than or equal to x, so b1c = 1, b3/2c = 1, b8/3c = 2, b−3/2c = −2,
etc. Find
lim sup an
n→∞

and
lim inf an .
n→∞

Does the sequence an have a limit?

3. Show that the second part of the Law of the Iterated Logarithm,
n −np
lim inf n→∞ Sα(n) = −1 follows by symmetry from the first part by
replacing p with (1 − p) and Sn with n − Sn .

Solutions to Problems

24
Reading Suggestion:
References
[1] William Feller. An Introduction to Probability Theory and Its Applica-
tions, Volume I, volume I. John Wiley and Sons, third edition, 1973. QA
273 F3712.

[2] John W. Lamperti. Probability: A Survey of the Mathematical Theory.

Wiley Series in Probability and Statistics. Wiley, second edition edition,
1996.

[3] Emmanuel Lesigne. Heads or Tails: An Introduction to Limit Theorems

in Probability, volume 28 of Student Mathematical Library. American
Mathematical Society, 2005.

Outside Readings and Links:

1.

2.

3.

4.

I check all the information on each page for correctness and typographical
errors. Nevertheless, some errors may occur and I would be grateful if you would
alert me to such errors. I make every reasonable effort to present current and
accurate information for public use, however I do not guarantee the accuracy or
timeliness of information on this website. Your use of the information from this
website is strictly voluntary and at your risk.

25
 I have checked the links to external sites for usefulness. Links to external
websites are provided as a convenience. I do not endorse, control, monitor, or
guarantee the information contained in any external website. I don’t guarantee
that the links are active at all times. Use the links here with the same caution as
you would all information on the Internet. This website reflects the thoughts, in-
terests and opinions of its author. They do not explicitly represent official positions
or policies of my employer.
Information on this website is subject to change without notice.
Steve Dunbar’s Home Page, http://www.math.unl.edu/~sdunbar1
Email to Steve Dunbar, sdunbar1 at unl dot edu
Last modified: Processed from LATEX source on March 22, 2018

26