STAT253/317 Lecture 4: 4.

4 Limiting Distribution I Example 1: 2-state Markov Chain

Stationary Distribution
(n)
Define πi = P(Xn = i), i ∈ X to be the marginal distribution of 0 1
 
Xn , n = 1, 2, . . . , and let π (n) be the row vector 0 1−α α
X = {0, 1}, P=
1 β 1−β
(n) (n) (n)
π (n) = (π0 , π1 , π2 , . . .),
(
π0 = (1 − α)π0 + βπ1
From Chapman-Kolmogrov Equation, we know that πP = π ⇒
X π1 = απ0 + (1 − β)π1
(n) (n−1)
π (n) = π (n−1) P i.e. πj =
(
πi Pij for all j ∈ X, απ0 = βπ1
i∈X
⇒
If π is a distribution on X satisfying βπ1 = απ0
X Need one more constraint: π0 + π1 = 1
πP = π i.e. πj = πi Pij for all j ∈ X,
i∈X  
β α
⇒ π = (π0 , π1 ) = ,
then π (0) = π implies π (n) = π for all n. α+β α+β
We say π is a stationary distribution of the Markov chain.
Lecture 4 - 1 Lecture 4 - 2

Example 2: Ehrenfest Diffusion Model Stationary Distribution May Not Be Unique

Consider a Markov chain with transition matrix P of the form
 i

 if j = i − 1 0 1 2 3 4
 2a
 
0 ∗ ∗ 0 0 0

Pij = 2a − i if j = i + 1
 1
∗ ∗ 0 0 0  
 2a

  Px 0
0 otherwise P = 20 0 ∗ ∗ ∗ =
 0 Py
!2a
3 0 0 ∗ ∗ ∗
1
π0 = π1 P10 = 2a π1 ⇒ π1 = 2aπ0 = 1 π0 4 0 0 ∗ ∗ ∗
2a(2a−1) !

π1 = π0 P01 + π2 P21 = π0 + 2
2a π2 ⇒ π2 = 2 π0 = 2a2 π0
This Markov chain has 2 classes {0,1} and {2, 3, 4}, both closed
2a(2a−1)(2a−2) !
and recurrent. So this Markov chain can be reduced to two
π2 = π1 P12 + π3 P32 = 2a−1 3
2a π1 + 2a π3 ⇒ π3 = 6 π0 = 2a
3 π0 sub-Markov chains, one with state space {0,1} and the other {2, 3,
..
. ... 4}. Their transition matrices are respectively Px and Py .
In general, Say πx = (π0 , π1 ) and πy = (π2 , π3 , π4 ) be respectively the
stationary distributions of the two sub-Markov chains, i.e.,
   2a
2a 1 πx P x = π x , πy Py = πy
πi = for i = 0, 1, 2, . . . , 2a
i 2
Verify that π = (cπ0 , cπ1 , (1 − c)π2 , (1 − c)π3 , (1 − c)π3 ) is a
stationary distribution of {Xn } for any c between 0 and 1.
Lecture 4 - 3 Lecture 4 - 4

Not All Markov Chains Have a Stationary Distribution Limiting Distribution

For one-dimensional symmetric random walk, the transition
probabilities are
Pi,i+1 = Pi,i−1 = 1/2
Try to solve for the stationary distribution A Markov chain is said to have a limiting distribution if for all
i, j ∈ X, πj = limn→∞ n
X 1 1 PPij exists, independent of the initial state
πj = πi Pij = πj−1 + πj+1 X0 , and πj ’s satisfy j∈X πj = 1.
i∈X 2 2
 
πj must be of the form π0 π1 π2 π3 · · ·
π0 π1 π2 π3 · · · 
πj = a + bj, for all integer j and some constants a, b i.e., lim Pn = π π π π · · · 
 
n→∞  0 1 2 3 
To make πj ≥ 0 for all integer j, ⇒ b = 0. Thus ... ... ... ... . . .

πj = a for all integer j

P∞
Impossible to make j=−∞ πj = 1.

Conclusion: 1-dim symmetric random walk does not have a

stationary distribution.
Lecture 4 - 5 Lecture 4 - 6
Limiting Distribution is a Stationary Distribution Limiting Distribution is Unique
The limiting distribution of a Markov chain is a stationary If a Markov chain has a limiting distribution π, then
distribution of the Markov chain. (n)
lim πj = πj for all j ∈ X, whatever π (0) is
n→∞
Proof (not rigorous). By Chapman Kolmogorov Equation,
X Proof (not rigorous). Since
(n+1) (n)
Pij = Pik Pkj X
k∈X (n) (0) (n)
πj = πk Pkj
k∈X
Letting n → ∞, we get
X Letting n → ∞, we get
(n+1) (n)
πj = lim Pij = lim P Pkj (n)
X (0) (n)
n→∞ n→∞ k∈X ik lim π = lim π Pkj
X (n) n→∞ j n→∞ k∈X k
=∗ lim P Pkj (needs justification) X (0) (n)
k∈X n→∞ ik =∗ π lim P (needs justification)
X k∈X k n→∞ kj
= πk Pkj X (0)
k∈X = π πj = π j
k∈X k
| {z }
Thus P
the limiting distribution πj ’s satisfies the equations =1
πj = k∈X πk Pkj for all j ∈ X and is a stationary distribution.
i.e., if a limiting distribution exists, it is the unique stationary
See Karlin & Taylor (1975), Theorem 1.3 on p.85-86 for a rigorous proof.
distribution.
Lecture 4 - 7 Lecture 4 - 8

Example: Two-State Markov Chain Not All Markov Chains Have Limiting Distributions I
Consider simple random walk on {0, 1, 2, 3, 4} with absorbing
boundary.
0 1
  0 1 2 3 4
0 1−α α 0

1 0 0 0 0

X = {0, 1}, P=
1 β 1−β 1 1/2 0 1/2 0 0 
By induction, one can show that P = 2 0 1/2 0 1/2 0 
3 0 0 1/2 0 1/2 
  4 0 0 0 0 1
β α n α α n
+ (1−α−β) − (1−α−β)
α + β α + β α+β α+β  Classes: {0}, {1,2,3}, {4}. One can show that
Pn =  β

β n α β n

+ (1−α−β) − (1−α−β) 0 1 2 3 4
α+β α+β α+β α+β  
  0 1 0 0 0 0
β α
1 3/4 0 0 0 1/4 

α + β α + β 
→  as n → ∞ Pn → 2 
 2/4 0 0 0 2/4  as n → ∞
 β α  
3  1/4 0 0 0 3/4 
α+β α+β 4 0 0 0 0 1
β
The limiting distribution π is ( α+β α
, α+β ). This Markov chain has no limiting distribution since limn→∞ Pijn
depends on the initial state i.
Lecture 4 - 9 Lecture 4 - 10

Not All Markov Chains Have Limiting Distributions II Periodicity

For example, in the Ehrenfest diffusion model,
A state of a Markov chain is said to have period d if
0 1 2 3 ... 2a
  Piin = 0, whenever n is not a multiple of d
0 ∗ 0 ∗ 0 ... ∗
  In fact, as n → ∞,
1 0 ∗ 0 ∗
 ... 0 
 In other words, d is the greatest common divisor of all the n’s such
( !

P2n 2 2a
∗ 0 ∗ 0  1 2a
= 2 ... ∗ that
j (2) if i + j is even
  2n
.. . . . .
. . . . .. .. 
 P ij → Piin > 0
. . . . . . .  0 if i + j is odd
(
2a ∗ 0 ∗ 0 ... ∗
0 if i + j is even We say a state is aperiodic if d = 1, and periodic if d > 1.
Pij2n+1 → !
1 2a
0 1 2 3 ... 2a 2 2a
j (2) if i + j is odd For the Ehrenfest diffusion model, all states are of period d = 2.
 
0 0 ∗ 0 ∗ ... 0
 
1  ∗ 0 ∗ 0 ... ∗  limn→∞ P n doesn’t exist
ij
Fact: Periodicity is a class property.
P2n+1
 0 
= 2 0 ∗ 0 ∗ ...  for all i, j ∈ X That is, all states in the same class have the same period.
... 
. .. .. .. .. .. 

 .. . . . . .  For a proof, see Problem 2&3 on p.77 of Karlin & Taylor (1975).
2a 0 ∗ 0 ∗ ... 0

Lecture 4 - 11 Lecture 4 - 12