Operations Research

Chapter 4: Discrete Time Markov Chain

Sonia REBAI
Tunis Business School
University of Tunis
Introduction
ü Deterministic and stochastic models are two broad categories of
mathematical models that aim at providing quantitative characterizations of
a real system under study.
ü A deterministic model predicts a single outcome, whereas a stochastic
model predicts a set of possible outcomes along with the likelihood of each
outcome.
ü Stochastic processes are models used to depict the dynamic relationship of
a family of random variables evolving in time or space.
Introduction - continued
ü A stochastic process {Xt}t is a sequence of random variables indexed by a
parameter such as time or space.

ü A discrete-time stochastic process is a description of the relation between

the random variables X0, X1, X2, . . . .

ü A continuous-time stochastic process is a stochastic process in which the

state of the system can be viewed at any time, not just at discrete instants in
time.
Discrete Time Markov Chain
ü A state is a possible value that can be taken by a given random variable.
ü A Markov process is a stochastic process satisfying the Markov property also
called memory-less property stipulating that : “A future state depends only on
the present state and not on the past”.
More specifically, if (Xt)t denotes a sequence of random variables then the
conditional probability of having a future state (j) given the present state (i) and
all the past states (a0, a1, …, at-1) is equal to the probability of having the future
state given the present state.
P{Xt+1 = j/Xt = i , Xt-1 = at-1 ,..., X0= a0 } = P{Xt+1 = j/Xt = i}
Discrete Time Markov Chain - continued
Remark
ü The memory-less property indicates that the knowledge of the present state
at a given moment provides as much information about the future as the
knowledge of the entire past. A process satisfying this phenomenon is
recognized as 1st order Markovian process.

ü It is sometimes possible that knowledge of the future depends not only on

the present state but on a finite number, k, of states in the past. A such
process is recognized as a Markovian process of order k.
Discrete Time Markov Chain - continued
ü Stationarity Assumption: For all states i and j and all t, the transition from i
to j in one period is independent of t. The probability relating the next
period’s state to the current state does not change over time (or remains
stationary). This assumption allows us to write
P{Xt+1 = j/Xt = i}= pij

where pij is the probability that given the system is in state i at period t, it
will be in a state j at period t+1. The pij are often referred to as the
transition probabilities of the stochastic process.
Discrete Time Markov Chain - continued
ü A Markov chain is a discrete-time stochastic process that can be in one of a
finite number of states satisfying the stationarity and the memory-less
properties.
ü A Markov process is completely characterized by its transition matrix P = (pij).

æp 11
p12
... p1j
... ö
çp p ... ... ... ÷ where
ç 21 22
÷
P = ç ... ... ... ... ... ÷ 𝑝!" ≥ 0 and ' 𝑝!" = 1
çp ... ... p ... ÷
ç i1 ij
÷ !"
ç ... ... ... ... ... ÷ø
è
Example 1
Every day, Ali uses one of the paths A or B to go to work. If Ali meets

congestion on the selected day along the chosen path, he will change the route

the next day. It is assumed that the chance of congestion on path A is equal to

1/3, and on path B is equal to 1/2.

Make sure that the problem may be described by a Markov chain and provide

its transition matrix.

Example 1 - continued
ü Let’s denote by Xn the random variable specifying the used path at day n.

ü The number of states is finite: A and B.

ü The stochastic process (Xn) satisfies the Markovian property. In fact, the
probability of using a given road depends only on the last used one.

ü The process is stationary because the transition probability from one state
to another does not depend on the specific day of transition.

The problem may be describe by a Markov Chain

Example 1 - continued A B
Aæç
2 1ö
÷
ü The transition matrix is given by: P= ç3
1
3÷
1÷
ç
ç ÷
B
è2 2ø

ü A Markov chain can be represented through a transition diagram in

which each state is pictured by a node and each possible transition is
shown by an oriented arc from the source node to the destination node.
The transition probability is represented on the arc.
ü The figure below displays the transition diagram for example 1
1/3
A B 1/2
2/3
1/2
Transition Probabilities over n periods
Let (Xn)n≥0 be a Markov Chain with r states .

ü The initial distribution or initial state vector, denoted by P(0), is the

probability distribution of X0 given by P(0) = (P(X0=a1), ..., P(X0=ar))

ü The distribution at time n or the state vector at time n, P(n), is the

probability distribution of Xn given by P(n) = (P(Xn= a1), ..., P(Xn= ar))

with P(Xn= ai) is the probability that the process be at state i at period n.

ü P(0) and P(n) have positive entries, with sum equal 1.

Transition Probabilities over n periods - continued
Let (Xn)n≥0 be a Markov Chain with r states .
ü The m-step transition matrix is denoted by P(m) = (pij(m))

• where pij(m) = P(Xn+m= aj /Xn= ai)

• pij(m) is called the m-step transition probability from state i to state j.
Remark

ü The m-step transition matrix P(m) specifies the origin of the process for each
destination m periods ahead.
ü Note that pij(m) ¹ (pij)m
Transition Probabilities over n periods - continued

Properties

ü P(n) = PP(n-1) = PPP(n-2) = ... = Pn

ü P(n) = P(m)´ Pn-m = P(n-m)´ Pm

($) ) (*) ($+*)

ü 𝑝!" ∑
= &'( 𝑝!& 𝑝&" ∀0<𝑚 <𝑛

This last equation is called Chapman-Kolmogorov equation

Example 1 - continued
Reconsider the previous example and suppose that the 1st day, Ali choses A or
B with equal chances. What is the probability of choosing A in 3 days?

Note that: P(0) = (1/2,1/2) and P(3) = P(0) x P3

æ2 1ö æ2 1 ö æ 11 7ö æ 11 7 ö æ2 1 ö æ 65 43 ö
ç ÷ ç ÷ ç ÷ ç ÷ ç ÷ ç ÷
3 3÷ ç3 3 ÷ ç 18 18 ÷ 3 ç
18 18 ÷ ç 3 3 ÷ ç 108 108 ÷
2 ç P =ç
P =ç ÷´ç ÷=ç ÷ ÷´ç ÷=ç ÷
ç1 1÷ ç1 1÷ ç 7 5÷ ç7 5 ÷ ç1 1 ÷ ç 43 29 ÷
ç ÷ ç ÷ ç ÷ ç ÷ ç ÷ ç ÷
è2 2ø è2 2 ø è 12 12 ø è 12 12 ø è 2 2 ø è 72 72 ø

æ 65 43 ö
ç ÷
ç 108 108 ÷ 259 173
P(3) = (1 / 2, 1 / 2) ´ ç ÷ = (
432
,
432
) It follows that P(X3=A) = 259/432
ç 43 29 ÷
ç ÷
è 72 72 ø
First passage probabilities
Let’s denote by fij(n) the probability of 1st passage from state i to state j in n
periods.

Back to the previous example and suppose that the 1st day, Ali uses road A.
What is the probability of using B for the 1st time in 3 days?
1st Approach : fAB(3) = (pAA)2 x pAB = (2/3)2 x 1/3 = 4/27

2nd Approach : we may obtain the same result by considering the following
recurrence formula:
= å pil ´ f lj
(n) ( n -1)
f ij
l¹ j
First passage probabilities - continued
fAB(1) = pAB = 1/3
fAB(2) = (fAB)(1) x pAA = (1/3) x (2/3) = 2/9
fAB(3) = (fAB)(2) x pAA = (2/9) x (2/3) = 4/27

3rd Approach : we may as well obtain the same result by considering the
following recurrence formula:
$+(
($) ($) (&) ($+&)
𝑓!" = 𝑝!" − ' 𝑓!" ×𝑝""
&'(
First passage probabilities - continued
fAB(3) = pAB(3) - fAB(1) x pBB(2) - fAB(2) x pBB(1)

pAB(3) = 43/108

fAB(1) = pAB(1) = pAB = 1/3

pBB(2) = 5/12

fAB(2) = pAA x pAB = 2/3 x 1/3 = 2/9

fAB(3) = 43/108 - 1/3 x 5/12 - 2/9 x 1/2 = 4/27

First passage probabilities - continued
ü Let’s denote by fij the probability of 1st passage from state i to state j.
¥
f ij = å f ij( n)
n =1
Note that fij ≤ 1

ü fii is called the probability of 1st return to state i.

ü fij may be computed as follows:
'
𝑓!" = 𝑝!" + % 𝑝!# ×𝑓#"
#$%
#&"
First passage time
ü Let’s denote by µij the average time of 1st passage from state i to state j

Time Probability
¥
1 fij(1)
2 fij(2)
µ ij = å ij
nf ( n)

n =1
. .
n fij(n)
. .
. .

ü µii is the mean recurrence time or average return time to state i

First passage time – continued

Two cases are possible:

ü If the passage from i to j after an infinite number of transitions is uncertain

(fij < 1) then the mean time of 1st passage from i to j is infinite (µij = ¥).

ü If the passage from i to j is certain (fij = 1) then the mean time of 1st passage

from i to j is finite (µij < ¥).

First passage time – continued
In case µij < ¥, this time is calculated as follows:
1
i j
1 Time Probability
µ1j
1 1 pij
1 1 µ2j
1+µ1j pi1 f1j= pi1
2 µkj
1
1+µ2j pi2 f2j= pi2
k
µrj . .
1+µkj pik fkj= pik
r . .

r r 1+µrj pir frj= pir

µij = pij + å ( 1 + µ kj ) pik =1 + å pik µ kj
k =1 k =1
k ¹ j k ¹ j
Example 1 - continued
Given that Ali uses road A, how many days in on a row he will continue to

use this road on the average?

µAB = 1 + pAAµAB

µAB = 1/(1-pAA) = 1/(1/3) = 3 days

On the average, he will use road A for 3 consecutive days before moving to

road B.
Classifying states in a Markov chain
ü A path from state i to state j is a sequence of transitions starting from i and
finishing on j.

ü State j is said to be accessible from i if there is a path from i to j. That is,

$ n ³ 1 where pij(n) > 0.

ü It follows that if j is not accessible from i then," n ³ 1, pij(n) = 0.

ü States i and j are said to communicate if both i is accessible from j and j is

accessible from i.
Classifying states in a Markov chain - continued
ü All states that communicate form an equivalence class.
ü State i is said to be absorbing if pii = 1.
ü That is, once a Markov chain is at an absorbing state, it will never leave it.
ü State i is said to be transient if there is some state j accessible from i but i is
not accessible from j. That is, (fii<1); or the return to state i is not
guaranteed.
ü Any non-transient state i is said to be recurrent. That is, once the chain is at
i, then the return to i is guaranteed; or (fii =1).
Classifying states in a Markov chain - continued
ü If i is recurrent, then µii is the recurrence time.

ü If Tii is the random variable time to return to state i, then µii=E(Tii).

ü If a Markov chain has only one equivalence class, then all the states are of
the same type.

ü Any Markov chain consisting of only one equivalence class is called

irreducible.

ü All states of an irreducible finite Markov chain are recurrent.

Classifying states in a Markov chain - continued
ü State i is said to be periodic of period d if for any integer n not multiple of
($)
d, 𝑝!! = 0. That is, the return to state i can be performed only on a number
of periods which is a multiple of d.
($)
ü The period d is the greatest common divisor of all n such that 𝑝!! > 0.
ü If d =1, we say that state i is aperiodic.
ü In the same equivalence class, all states have the same period.
ü An irreducible Markov chain for which all states are aperiodic is called an
ergodic Markov chain.
Example 2
Classify the states of the following Markov chains while specifying their
æ1 1 ö
periods. æ 1 1ö ç 0 0÷
æ0 1 0ö ç0 ÷ ç2 2 ÷
ç ÷ ç 2 2÷ ç1 1
0 0÷
1 1 1÷
P1 = ç 0 0 1 ÷, P2 = ç , P3 = ç
ç
2 2 ÷
ç ÷ ç3 3 3 ÷ 2 1÷
ç1 ç1 3÷ ç0 0 ÷
è 0 0÷ø 0 ç 3 3÷
ç ÷ 1 3÷
è4 4ø ç
ç0 0 ÷
è 4 4ø
æ1 1 ö
æ0 1 0 0ö ç 0 0 0÷
ç ÷ ç2 2 ÷
ç0 0 1 0÷ ç1 3
0 0 0÷
ç ÷ ç4 4 ÷
P4 = ç 1 1 ÷, P5 = ç 0 0 0 1 0÷
0 0 ç ÷
ç 2 2÷
ç1 1 ÷ ç0 1 1÷
ç 0 0
ç
è2
0
2
0÷
ø 2 2÷
ç ÷
è0 0 0 1 0ø
Limiting Probabilities
Theorem

Consider an ergodic Markov chain with a transition matrix P. There exists a

vector π which is independent on the initial state, so that:

p = lim P(n)
n®¥

Moreover, as n®¥ all the rows of the matrix Pn converge to the vector π .

The vector π is called the steady-state distribution of the Markov chain.

Limiting Probabilities - continued
Steady-State Distribution
To determine the steady state distribution, we proceed as follows:
Since 𝑃 𝑛 + 1 = 𝑃 𝑛 ×𝑃 then

lim 𝑃 𝑛 + 1 = lim 𝑃 𝑛 ×𝑃
$→- $→-
π=πxP
This last expression gives rise to a system of r equations with r unknowns.
Given that matrix P has a rank £ r-1, we need one more independent equation.
Remember that we have π1 + π2 + … + πr = 1. This is exactly what is needed.
Example 3
To keep a triangular castle equipped with a guard at every corner, the guard

must flip a coin every 5 minutes to determine which of the next two corners to

occupy. If head, the guardian must go left; else, he would go to the right

corner. He must stay for another 5 minutes, then again take a coin and so on.

Determine the steady state distribution of this Markov chain.

Example 3 - continued
The transition matrix of the Markov chain is given by: P =

This Markov chain is ergodic.

(π1,π2,π3) = (π1,π2,π3) x P
π1 + π2 + π3 = 1
½ π2 + ½ π3 = π1 π1 + π2 + π3 = 1
½ π1 + ½ π3 = π2 π1 = π2 = π3
½ π1 + ½ π2 = π3
π1 = π2 = π3 = 1/3
π1 + π2 + π3 = 1
Example 3 - continued
On the long-term, the guard of the three corners of the castle will be equally
likely. æ 1 1 1 ö
ç ÷
ç 3 3 3 ÷
ç 1 1 1 ÷
The matrix Pn will converge to the matrix: ç 3 3 3 ÷
ç 1 1 1 ÷
ç ÷
è 3 3 3 ø
The stationary distribution can also easily determine the mean recurrence time:
µii = 1/πi

In the previous example, we have:

µ11 = 1/π1 = 3; µ22 = 1/π2 = 3; µ33 = 1/π3 = 3

Limiting Probabilities -continued
Case of Absorbing Markov Chain

ü A Markov chain may have more than one absorbing state.

ü An absorbing state will have zero entry for all other states in the
corresponding row of the transition matrix, but 1 for the entry related to
the row and column of that state.

ü An absorbing Markov chain can only have transient states that can
communicate with each others.
Limiting Probabilities -continued

ü It is possible to rewrite the transition matrix by changing perhaps the order of

the states so that the first rows and columns would correspond to the

transient states and the remaining states will be absorbing states.

ü It follows that a Markov chain having k transient states and n-k absorbing

states can be represented as follows:

Limiting Probabilities -continued

Or equivalently
Limiting Probabilities -continued
Some facts about absorbing chains can be found:
(1) If the chain begins in a given transient state, and before reaching an
absorbing state, what is the expected number of times that each state will
be entered? How many periods do we expect to spend in a given transient
state before absorption takes place?
If we are at present in a transient state i, the expected number of periods that
th
will be spent in transient state j before absorption is the ij element of the
-1
matrix (I-Q) .
Limiting Probabilities -continued
(2) If a chain begins in a given transient state, what is the probability that we
end up in each absorbing state?

If we are at present in transient state i, the probability that we will eventually

th -1
be absorbed by the absorbing state j is the ij element of the matrix (I-Q) R.
-1
The matrix (I-Q) is often referred to as the Markov chain’s fundamental
matrix.
Example 4
Consider the following Markov chain :

1. Find the the expected number of periods that will be spent in each transient
state.
2. Find the probability to be absorbed by each absorbing state starting from
each possible transient state.
Example 4 - continued