MARKOV CHAIN

MARKOV CHAIN 1 / 14
1 Markov Chain

MARKOV CHAIN 2 / 14
 Markov Chain

Outlines

1 Markov Chain

MARKOV CHAIN 3 / 14
 Markov Chain

Markov Chain

Let {Xn }, n ≥ 0, be a sequence of random variables which take

values in some finite set S = {s1 , s2 , · · · , sk }, called the state space.
Each Xn is a discrete random variable that takes one of k possible
states and its probability distribution can be represented by the
(probability) vector πn = (πn (1) , πn (2) , · · · , πn (k)) ∈ Rk , in which
πn (j) = P (Xn = sj ), for j = 1, 2, · · · , k.
It is clear that πn (j) ≥ 0, for all j, and kj=1 πn (j) = 1.
P
Definition 1. The sequence {Xn } is called a Markov chain if it
satisfies the Markov condition,

P (Xn = s| X0 = x0 , X1 = x1 , · · · , Xn−1 = xn−1 ) = P (Xn = s| Xn−1 = xn−

(1)
for all n ≥ 1 and all s, x0 , x1 , · · · , xn−1 ∈ S.

MARKOV CHAIN 4 / 14
 Markov Chain

Remarks. (a) The Markov property (1) is equivalent to each of the

stipulations (2) and (3) below: for each s ∈ S and for every sequence
{xi : i ≥ 0} in S, for all n1 < n2 < · · · < nk ≤ n,

P (Xn+1 = s| Xn1 = xn1 , Xn2 = xn2 , · · · , Xnk = xnk ) = P (Xn+1 = s| Xnk =

(2)

P (Xm+n = s| X0 = x0 , X1 = x1 , · · · , Xm = xm ) = P (Xm+n = s| Xm = xm
(3)
(b) Without loss of generality, we can denote the elements of S as 1,
2, . . . , k, although in some examples we may use the original labeling
of the states to avoid confusion.
(c) The distribution vector of Xn , πn = (πn (1) , πn (2) , · · · , πn (k)),
is called the distribution at t = n and π0 the initial distribution of
the Markov chain.
MARKOV CHAIN 5 / 14
 Markov Chain

Definition 2. The Markov chain {Xn } is called homogeneous if

P (Xn+1 = i| Xn = j) = P (X1 = i| X0 = j)
for all n, i, j. The transition matrix P = (pij ) is the k × k matrix of
transition probabilities
pij = P (Xn+1 = i| Xn = j)
State j at time t = n
 ↓ 

P= pij  ← State i at time t = n + 1

From now on, all Markov chains are assumed homogeneous, with
state space S = {1, 2, · · · , k}, transition matrix P ∈ Rk×k , and
probability distributions πn ∈ Rk , n = 0, 1, 2, · · · , where
πn (i) = P (Xn = i), for i = 1, 2, · · · , k.
MARKOV CHAIN 6 / 14
 Markov Chain

Proposition 3. Let {Xn } be a Markov chain. For all m, n ≥ 0 and

i, j ∈ S, we have

P (Xm+n = i| Xm = j) = P (Xn = i| X0 = j) .
 
(n) (n)
The matrix P(n) = pij ∈ Rk×k , where pij = P (Xn = i| X0 = j),
(n)
is called the n-step transition matrix and the pij are called the
n-step transition probabilities. In particular, P is the 1-step
transition matrix of the Markov chain.
State j at time t = m
↓
 
(n)
P(n) =  pij  ← State i at time t = m + n

MARKOV CHAIN 7 / 14
 Markov Chain

Theorem 4 (Chapman-Kolmogorov equations). Let {Xn } be a

Markov chain. For all n ≥ 0 and i, j ∈ S, we have
k
X
P (Xn+1 = i| X0 = j) = P (Xn+1 = i| Xn = l) P (Xn = l| X0 = j) .
l=1
(4)

Proof
Pk
Since P (Xn+1 = i, X0 = j) = l=1 P (Xn+1 = i, Xn = l, X0 = j), and
for each l = 1, 2, · · · , k,

P (Xn+1 = i, Xn = l, X0 = j) = P (Xn+1 = i| Xn = l, X0 = j) P (Xn = l

= P (Xn+1 = i| Xn = l) P (Xn = l| X0 = j) P (X0 = j) ,

MARKOV CHAIN 8 / 14
 Markov Chain

we have
Pk
P (Xn+1 = i| X0 = j) = P(XP(X
n+1 =i,X0 =j)
0 =j)
= P(X10 =j) l=1 P (Xn+1 = i, Xn =
Pk
= l=1 P (Xn+1 = i| Xn = l) P (Xn = l| X0 = j)

as desired.
Since P (Xn+1 = i| Xn = l) = P (X1 = i| X0 = l) = pil , the equation
(4) can be rewritten as
k
X
(n+1) (n)
pij = pil plj ,
l=1

where the right hand side is equal to the inner product of the i-th row
of P with the j-th column of P(n) , and we arrive at

MARKOV CHAIN 9 / 14
 Markov Chain

Corollary 5. For each n ∈ N,

P(n+1) = P × P(n) .
In particular, P(n) = Pn , i.e., the n-step transition matrix is equal to
the n-th power of P, the 1-step transition matrix. Corollary 6. Let
{Xn } be a Markov chain with initial distribution π0 . For all n ∈ N, we
have
πn = P n π 0 .
2, · · · , k,
Proof. For each i = 1,P
πn (i) = P (Xn = i) = kj=1 P (Xn = i; X0 = j), and for each
j = 1, 2, · · · , k,
P (Xn = i; X0 = j) = P (Xn = i| X0 = j) P (X0 = j) = P(n) (i, j) π0 (j) =
i.e.,
k
X
πn (i) = Pn (i, j) π0 (j) .
j=1

MARKOV CHAIN 10 / 14
 Markov Chain

Since the right hand side is the inner product of the i-th row of Pn
with the (column) vector π0 , we conclude that πn = P n π0 as desired.
Definition 7. Let {Xn } be a Markov chain and let
π = (π (1) , π (2) , · · · , π (k)) be a distribution on S, i.e., π (i) ≥ 0,
for all i = 1, 2, · · · , k, and ki=1 π (i) = 1.
P
(i) π is called the limiting distribution of {Xn } if
lim Pn (i, j) = π (i), for all i, j = 1, 2, · · · , k.
n→∞
(ii) π is called a stationary distribution of {Xn } if
π = Pπ.
Remarks. (a) By the uniqueness property of the limit, a Markov
chain has at most one limiting distribution, whereas it may have more
than one stationary distribution.
(b) If π is the limiting distribution, then P (Xn = i| X0 = j) → π (i)
as n → ∞, for all j = 1, 2, · · · , k, i.e., regardless of what the initial
state j is, the probability that the chain is at the state i at time
t = n, P (Xn = i), can be approximated by π (i) for large values of n.
MARKOV CHAIN 11 / 14
 Markov Chain

(c) If the chain is at a stationary distribution π at time t = n, πn = π,

then at all subsequent times, the distribution of the state of the chain
remains exactly the same as π, i.e., πn+m = π, for all m ∈ N.
Definition 8. The transition matrix P of a Markov chain is said to
be regular if there exists n ∈ N such that all elements of Pn are
positive.
Theorem 9. Let {Xn } be a Markov chain with regular transition
matrix P. Then the limiting distribution of the chain exists and is
equal to the unique stationary distribution.
Example. Let {Xn } be a Markov chain with state space
S = {1, 2, 3}, transition matrix

 1 2 3 
1 0.2 0 1
P=
2  0.8 0.4 0 
3 0 0.6 0
and initial distribution π0 = (0.5, 0.5, 0).
MARKOV CHAIN 12 / 14
 Markov Chain

The 3-step transition matrix is given by

 
0.488 0.36 0.04
P3 =  0.224 0.544 0.48 
0.288 0.096 0.48
which gives the distribution of X3 ,
 
0.424
π3 = P3 π0 =  0.384  .
0.192
Therefore, we have
P (X10 = 2| X7 = 1, X6 = 3) = P (X10 = 2| X7 = 1) = P3 (2, 1) = 0.224
P (X4 = 2, X3 = 1) = P (X4 = 2| X3 = 1) P (X3 = 1) = P (2, 1)·π3 (1) =
All elements of P3 are positive implies that P is regular and then the
limiting distribution of the exists and is equal to its unique stationary
distribution π = (a, b, c), with a, b, c ≥ 0, a + b + c = 1, and
MARKOV CHAIN 13 / 14
 Markov Chain

π = Pπ ⇔ (I3 − P) π = 0. (5)
By transforming
− 45 1 0 − 45
     
0.8 0 −1 1 0
I3 −P =  −0.8 0.6 0  →  0 0.6 −1  →  0 1 − 53 
0 −0.6 1 0 −0.6 1 0 0 0

and letting c = α ∈ R, we get the general solution π = 54 α, 53 α, α ,

α ∈ R, of (5). The condition a, b, c ≥ 0 implies that α ≥ 0, and the

47
condition 1 = a + b + c = 12 α gives α = 12
47
, i.e.,
 
15 20 12
π= , , ≈ (0.3192, 0.4255, 0.2553) .
47 47 47
Thus in the long run, regardless of the initial distribution, the chain
will be at the state 1, 2, 3, with probability 0.3192, 0.4255, 0.2553,
respectively.
MARKOV CHAIN 14 / 14