Chapter 9 Stationary Distribution of Markov

Chain (Lecture on 02/02/2021)

Previously we have discussed irreducibility, aperiodicity, persistence, non-null persistence, and a

application of stochastic process. Now we tend to discuss the stationary distribution and the
limiting distribution of a stochastic process.

How does a Markov chain behave after a long time n has elapsed?

The sequence {Xn }n≥1 cannot generally converge to some particular state in the state space
since it enjoys the random fluctuation which is specified by the transition matrix. However, we
might hold out some hope that the distribution of Xn settles down. Indeed, subject to certain
conditions, this turns out to be the case.

The distribution settels down means that as n → ∞ , the marginal distribution of Xn

is the same as the marginal distribution of Xn+1 .

We shall see the existence of a limiting distribution for Xn as n → ∞ is closely tied

with the existence of a stationary distribution.

Definition 9.1 (Stationary distribution) The vector π is called a stationary distribution of the
chain if π has entries (πj : j ∈ S) such that:

a. πj ≥ 0, ∀j , and ∑j πj = 1 .

b. π = πP , where P is the trnasition matrix of the chain. Thus, πj = ∑ πi pij , ∀j

i
.

Such a distribution is called stationary as πP2 = (πP)P = πP = π , similarly, πPn = π, for

all n ≥ 0 .
 Note that for a time homogenous Markov chain, if X0 has marginal distribution π,
then Xn has marginal distribution πPn . Now if πPn = π , every Xn has the same
marginal distribution. It suggests that, once the chain hits the stationary distribution,
meaning that the marginal distribution of the chain becomes to the stationary
distribution, then the marginal distribution of the chain will remain the stationary
distribution.

Theorem 9.1 (Renewal Theorem) An irreducible chain has a stationary distribution π if and only if
all states are non-null persistent. In this case, π is the unique stationary distribution and is given by
πi =
1

μi
for each i ∈ S , where μi is the mean recurrence time for state i.

This is a intuitive result in the following sense. The μi is the average number of times
the Markov chain is going to visit a certain state. It should be connecting with the
probability of visiting that state. Emperically, one over the average number of times the
Markov chain visit a certain state is the probability of visiting that state.

Note that we have already proved the following two results:

a. Decomposition theorem (Theorem 7.1): S = T ∪ C1 ∪ C2 ∪ ⋯ where T is the set of

transient state and C1 , C2 , ⋯ are sets of intercommunicating persistent states. Every state
with in each Ci has the same properties.

b. Lemma 7.1: For a Markov chain with finite state space, at least one state is non-null
persistent.

If we are consider a finite Marko chain, we can look at the intercommunication patterns to find a
decomposition of S (we have discussed one example, see Example 7.1). You know that each
persistent group of states is non-null persistent. The reason is that since the state space is finite,
then using result (b) mentioned above, at least one state is non-null persistent and therefore,
since they are intercoummunicating, all of them are non-null persistent. Thus, within every group
of persistent states, there exist a unique stationary distribution.

If the entire state space of a Markov chain is irreducible, we can find a unique
stationary distribution. When the entire state space of a Markov chain is not
irreducible, we have to use the decomposition theorem, and find stationary
 distribution for every persistent group of states.

Example 9.1 Let S = {1, 2, 3, 4, 5, 6} and the transition probability matrix is given by

1 1
⎛ 0 0 0 0 ⎞
2 2

⎜ 1 3 ⎟
⎜ 4 0 0 0 0 ⎟
4
⎜ ⎟
⎜ 1 1 1 1 ⎟
⎜ 0 0 ⎟
⎜ 4 4 4 4
⎟ (9.1)
⎜ 1 1 1 1 ⎟
⎜ 0 0 ⎟
⎜ 4 4 4 4 ⎟
⎜ ⎟
1 1
⎜ 0 0 0 0 ⎟
⎜ 2 2 ⎟

1 1
⎝ 0 0 0 0 ⎠
2 2

We have showed previously (Example 7.1) that the state space of this Markov chain can be
decomposed into the set of transient states T = {3, 4} and two sets of persistent states
C1 = {1, 2} and C2 = {5, 6} .

Now let us find the stationary distribution of C1 . We need to find π = (π1 , π2 ) such that

1 1

2 2
( π1 π1 ) = ( π1 π1 ) ( ) (9.2)
1 3

4 4

Therefore, we have

1 1
⎧
⎪
⎪ π = π + π2 1
⎪
⎪
1 1 ⎧
⎪
⎪ 2 4 ⎪ π1 =
3
⎨ 1 3 ⟺ ⎨ (9.3)
π2 = π1 + π2 2
⎪
⎪ ⎪
⎩π =
⎪
⎪ 2 4
⎪
⎩
⎪
2
3
π1 + π2 = 1

Note that in Example 7.1, we have calculate that the mean recurrence time for state 1
is 3. Using the Theorem 9.1 we can actually get the stationary distribution of this
subchain directly from the result of Example 7.1.

Example 9.2 (Random walk with retaining barriers) A particle performs a random walk on the
non-negative integers with a retaining barrier at 0. The transition porbabilities are p00 = q ,
pi,i+1 = p, ∀i ≥ 0 and pi,i−1 = q for i ≥ 1 p . and q satisfies p + q = 1 .
This chain is obviously irreducible since all states are intercommunicating. Now let us find the
stationary distribution. By definition, the stationary distribution π satisfies π = πP . Therefore,
we have

q p 0 0 0 ⋯
⎛ ⎞

⎜q 0 p 0 0 ⋯⎟
( π0 ⎜ ⎟ = (
π1 ⋯) = π0 π1 ⋯) (9.4)
⎜0 q 0 p 0 ⋯⎟
⎜ ⎟

⎝ ⎠
⋮ ⋮ ⋮ ⋮ ⋮ ⋮

We get the following system of equations:

π0 q + π1 q = π0 (1)

π0 p + π2 q = π1 (2)

⋯⋯⋯

πn−1 p + πn+1 q = πn (n + 1)

⋯⋯⋯

p p
From (1) we have π1 q = π0 (1 − q) or π1 =
q
π0 . From (2), we obtain π0 p + π2 q = π1 =
q
π0 ,
p p p
therefore π2 = (
q
2
) π0 . By induction, we prove that πn = (
q
n
) π0 . Assume πn−1 = (
q
n−1
) π0
p
and πn = (
q
n
) π0 , we try to find πn+1 . From equation (n + 1), we have
p p p
(
q
n−1
) π0 p + πn+1 q = (
q
n
) π0 , from which we can obtain πn+1 = (
q
n+1
) π0 .

∞ ∞ p ∞ p
In addition, ∑n=0 πn = 1 , we have π0 [1 + ∑n=1 ( q )n ] = 1 . The sum 1 + ∑n=1 ( q )n is finite if
p p
and only if q
< 1 . Hence, no chance of having a stationary distribution when q
≥ 1 . This also
p
tells you that the chain is not non-null persistent when q
≥ 1 . This is an intuitively correct result,
because when p > q , there are less force of bringing the chain back to 0, the chain have a
porpensity to go diverge.
p p p p
When q
< 1 , we have π0 = 1 −
q
and it implies that πn = (
q
n
) (1 −
q
. The stationary
)

distribution is obtained in a nice close form.

This random walk is different with the simple random walk we discussed before. It
does not allow the chain to visit negative integers. When it hits state 0, it can only
remains there with probability q, or go up with probability p. All the state in simple
random walk have the period equal to 2, while for this random walk, it is not true. In
fact, every state in this chain is aperiodic. The reason is that every state is
intercommunicating and state 0 is aperiodic, since the chain can retain at 0.
Now we will establish the link between stationary distribution and limiting distribution of a
Markov chain. Thus, we will try to characterize limn→∞ pij (n). While doing this, periodicity of the
chain can pose some issues. For example, suppose the transition matrix of a Markov chain is
given by

0 1
P = ( ) (9.5)
1 0

then we have

0 n is odd
p11 (n) = p22 (n) = { (9.6)
1 n is even

Therefore, {Pii (n)}n≥1 is an alternating sequence and limn→∞ pii (n) does not exist.

To avoid such complications, when we try to characteristic limn→∞ pij (n), we will only deal with
irreducible, aperodic chains.