Discrete-Time Markov Chains: He Shuangchi

Discrete-Time Markov Chains
He Shuangchi
IE5004 Discrete-Time Markov Chains 1 / 61

Example
Example
The NUS Co-op sells a specific model of laptop computers. Assume that
the weekly demands are iid following distribution
d 0 1 2 3 4
P[Dn = d] 0.3 0.3 0.2 0.1 0.1
where Dn is the demand of week n. By the end of each week, if the
inventory is less than or equal to 1, the store orders more to bring the
inventory back to 4 by the beginning of next week. Assume that all unmet
demand will be lost. Let Xn be the inventory level at the beginning of
week n. If X0 = 4, find E[X10 ].

Stochastic processes
Let X = {X0 , X1 , . . .}, which is a set of random variables indexed by the

week number.
Definition
A stochastic process is a collection of random variables indexed by time.
If the stochastic process consists of countably many random variables,

it is called a discrete-time stochastic process.
If the stochastic process consists of uncountably many random
variables, it is called a continuous-time stochastic process.
Example
Let Yi be the highest IBM stock price on day i. Then, {Y0 , Y1 , . . .} is
a discrete-time stochastic process.
Let Y (t) be the IBM stock price at time t on a specific day. Then,
{Y (t) : t ≥ 0} is a continuous-time stochastic process.
State space of a stochastic process
Definition
The set of all possible values that a random variable in a stochastic
process can take is called the state space of the stochastic process.
Question
What is the state space of X ?
The value of X0 is 4. At the end of week 0, the leftover is 4 − D0 . Then,

(
4 − D0 if D0 < 3
X1 =
4 if D0 ≥ 3
So the possible values that X1 can take are 2, 3, or 4. Similarly, we can
argue that the possible values that X2 , X3 , . . . can take are also 2, 3, or 4.
Therefore, the state space is
S = {2, 3, 4}
How does the stochastic process X evolve?
The stochastic process X follows

(
Xn − Dn if Xn − Dn ≥ 2
Xn+1 =
4 if Xn − Dn ≤ 1
We can see X10 depends on X9 and D9 , X9 depends on X8 and D8 , etc.

In general, the state of a stochastic process at any time depends on
all states at previous times.
Such dependence makes the analysis of a stochastic process difficult.
It is necessary to introduce simplified models.
Observation
Once the value of Xn is known, the distribution of Xn+1 no longer depends
on Xn−1 , Xn−2 , . . ..

Discrete-time Markov chains
Definition
A sequence of random variables {X0 , X1 , . . .} with state space S is called a
discrete-time Markov chain if for all n = 0, 1, . . . and all i0 , . . . , in , j ∈ S,
P[Xn+1 = j|X0 = i0 , . . . , Xn−1 = in−1 , Xn = in ] = P[Xn+1 = j|Xn = in ]
At any time n, the future state of the Markov chain Xn+1 depend on
the history X0 , . . . , Xn only through the present state Xn .
If a discrete-time stochastic process X can be expressed by
Xn+1 = f (Xn , Yn )
where f is a function and Y0 , Y1 , . . . are independent of X . Then, X

must be a Markov chain.

Time-homogeneous Markov chains
Definition
A discrete-time Markov chain {X0 , X1 , . . .} is called time-homogeneous if
for all n = 0, 1, . . . and all i, j ∈ S,
P[Xn+1 = j|Xn = i] = P[X1 = j|X0 = i]
For a time-homogeneous Markov chain,
Pij = P[Xn+1 = j|Xn = i]
is called the transition probability from state i to state j.

Question
In the laptop example, is X a Markov chain? Is it time-homogeneous?
From now on, we discuss time-homogeneous Markov chain only.

Transition probabilities
Question
Find the transition probabilities of X in the laptop example.
P22 = P[Xn+1 = 2|Xn = 2] = P[Dn = 0] = 0.3

P23 = P[Xn+1 = 3|Xn = 2] = 0
P24 = P[Xn+1 = 4|Xn = 2] = 1 − P[Dn = 0] = 0.7
P32 = P[Xn+1 = 2|Xn = 3] = P[Dn = 1] = 0.3
P33 = P[Xn+1 = 3|Xn = 3] = P[Dn = 0] = 0.3
P34 = P[Xn+1 = 4|Xn = 3] = P[Dn = 2] + P[Dn = 3] + P[Dn = 4] = 0.4
P42 = P[Xn+1 = 2|Xn = 4] = P[Dn = 2] = 0.2
P43 = P[Xn+1 = 3|Xn = 4] = P[Dn = 1] = 0.3
P44 = P[Xn+1 = 4|Xn = 4] = P[Dn = 0] + P[Dn = 3] + P[Dn = 4] = 0.5

Example
Question
In the laptop example, find the distribution of X1 .
By the above transition probabilities,
P[X1 = 2|X0 = 4] = P42 = 0.2

P[X1 = 3|X0 = 4] = P43 = 0.3
P[X1 = 4|X0 = 4] = P44 = 0.5

Transition matrix
We may organize the transition probabilities into a matrix,
P = [Pij ]i,j∈S
which is called the (one-step) transition matrix.
Example
The transition matrix of the laptop example is given by
2 3 4
 
2 0.3 0 0.7
P = 3 0.3 0.3 0.4 
4 0.2 0.3 0.5

Transition matrix
With S = {1, . . . , m}, the transition matrix is an m × m matrix given by
1 2 ··· j ··· m
 
1 P11 P12 ··· P1j ··· P1m
2  P21 P22 ··· P2j ··· P2m 

..  .. .. .. .. .. .. 
.  . . . . . . 
P= 
i   Pi1 Pi2 ··· Pij ··· Pim 

..  .. .. .. .. .. .. 
.  . . . . . . 
m Pm1 Pm2 ··· Pnj ··· Pmm
The (i, j)th entry is Pij = P[X1 = j|X0 = i].

The sum of each row is one.

Transition diagram
We may also use a transition diagram to describe the transitions.
0.3
0.3 2 0.2 0.3 3 0.3
0.7 4 0.4
0.5

Example
Example (Gambler’s ruin)

Two gamblers, Tom and Jerry, are flipping a fair coin. If the outcome is
heads, Tom wins one dollar; otherwise, Jerry wins one dollar. At the
beginning both friends have three dollars. They will repeat the game until
one of them lose all three dollars. Let Zn be the money Tom has after the
nth game. Find the transition probabilities of {Z0 , Z1 , . . .}.

Example
The transition matrix of the gambler’s ruin example is
0 1 2 3 4 5 6
 
0 1
1
 0.5 0.5 

2
 0.5 0.5 

P = 3
 0.5 0.5 

4
 0.5 0.5 

5 0.5 0.5 
6 1
The transition diagram is
1.0 0.5 0.5 0.5 0.5 0.5
0 1 2 3 4 5 6
0.5 0.5 0.5 0.5 0.5 1.0

Two-step transition probabilities
Question
Clearly,
P[X2 = 2|X0 = 4] = P[X2 = 2, X1 = 2|X0 = 4]

+ P[X2 = 2, X1 = 3|X0 = 4] + P[X2 = 2, X1 = 2|X0 = 4]
In the above expression,

P[X2 = 2, X1 = 2, X0 = 4]
P[X2 = 2, X1 = 2|X0 = 4] =
P[X0 = 4]
P[X1 = 2, X0 = 4]P[X2 = 2|X1 = 2, X0 = 4]
=
P[X0 = 4]
= P[X1 = 2|X0 = 4]P[X2 = 2|X1 = 2]
= P42 P22

Similarly,
P[X2 = 2, X1 = 3|X0 = 4] = P43 P32
P[X2 = 2, X1 = 4|X0 = 4] = P44 P42
Therefore,
P[X2 = 2|X0 = 4] = P42 P22 + P43 P32 + P44 P42
We may compute this probability using the transition matrix
   
0.3 0 0.7 0.3 0 0.7
0.3 0.3 0.4 × 0.3 0.3 0.4
0.2 0.3 0.5 0.2 0.3 0.5
In other words,
P[X2 = 2|X0 = 4] = (the row of state 4) × (the column of state 2)
= 0.2 × 0.3 + 0.3 × 0.3 + 0.5 × 0.2
= 0.25
Using the transition matrix
   
0.3 0 0.7 0.3 0 0.7
0.3 0.3 0.4 × 0.3 0.3 0.4
0.2 0.3 0.5 0.2 0.3 0.5
we may compute

= 0.2 × 0 + 0.3 × 0.3 + 0.5 × 0.3
= 0.24
and

= 0.2 × 0.7 + 0.3 × 0.4 + 0.5 × 0.5
= 0.51

Two-step transition matrix
Consider the two-step transition matrix

 (2) (2) (2)
  
P22 P23 P24 ? ? ?
 (2) (2) (2) 
P (2) = P32 P33 P34  =  ? ? ? 
(2) (2) (2) 0.25 0.24 0.51
P42 P43 P44
where
(2)
Pij = P[Xn+2 = j|Xn = i] = P[X2 = j|X0 = i]
You may have already seen that
P (2) = P × P = P 2
where P is the one-step transition matrix.

k-step transition matrix
Definition
Let {X0 , X1 , . . .} be a time-homogeneous Markov chain with state space
S = {1, . . . , m}. The k-step transition matrix of X is
1 2 ··· j ··· m
 (k) (k) (k) (k)

1 P P12 ··· P1j ··· P1m
 11
2  P (k) (k)
P22 ···
(k)
P2j ···
(k) 
P2m 
 21 
.. 
 ... .. .. .. .. .. 
. .
P (k) = .  (k)
.
(k)
.
(k)
. 
(k) 

i  Pi1 Pi2 ··· Pij ··· Pim 
..  ..
 .. .. .. .. .. 
.  . . . . . . 

(k) (k) (k) (k)
m Pm1 Pm2 ··· Pnj ··· Pmm
where
(k)
Pij = P[Xn+k = j|Xn = i] = P[Xk = j|X0 = i]

Chapman–Kolmogorov equations
Theorem (Chapman–Kolmogorov equations)

Let {X0 , X1 , . . .} be a Markov chain with k-step transition probabilities
(k)
Pij for all i, j ∈ S, where S is the state space. Then, for any nonnegative
integers k and `
(k+`)
X (k) (`)
Pij = Pis Psj
s∈S
or equivalently,
P (k+`) = P (k) P (`)
The Chapman–Kolmogorov equations are equivalent to
P (k) = P k for k = 1, 2, . . .
where P is the transition matrix of the Markov chain. In other words,

(k)
Pij = P[Xn+k = j|Xn = i] = P[Xk = j|X0 = i] = Pijk

Multi-step transition probabilities
Exercise
In the laptop example, find P[X6 = 2, X3 = 3, X1 = 2|X0 = 4].
P[X6 = 2, X3 = 3, X1 = 2|X0 = 4]
= P[X1 = 2|X0 = 4]P[X6 = 2, X3 = 3|X1 = 2, X0 = 4]
= P[X1 = 2|X0 = 4]P[X6 = 2, X3 = 3|X1 = 2]
= P[X1 = 2|X0 = 4]P[X3 = 3|X1 = 2]P[X6 = 2|X3 = 3, X1 = 2]
= P[X1 = 2|X0 = 4]P[X3 = 3|X1 = 2]P[X6 = 2|X3 = 3]
2 3
= P42 P23 P32

Example
Example
In the laptop example, if X0 = 4, find E[X10 ].
We may use Matlab to obtain

 
0.2473 0.2258 0.5269
P (10) = P 10 = 0.2473 0.2258 0.5269
0.2473 0.2258 0.5269
Hence,
P[X10 = 2|X0 = 4] = 0.2473
P[X10 = 3|X0 = 4] = 0.2258
P[X10 = 4|X0 = 4] = 0.5269
It follows that
E[X10 ] = 2 × 0.2473 + 3 × 0.2258 + 4 × 0.5269 = 3.2796
Example
Example
In the laptop example, suppose that X0 follows the distribution below
j 2 3 4
P[X0 = j] 0.5 0.2 0.3
Find the distribution of X3 .
Let
(0) (0) (0)
µ(0) = (µ2 , µ3 , µ4 ) = (0.5, 0.2, 0.3)
be the initial distribution. First,
P[X3 = 2] = P[X3 = 2, X0 = 2] + P[X3 = 2, X0 = 3] + P[X3 = 2, X0 = 4]

= P[X0 = 2]P[X3 = 2|X0 = 2] + P[X0 = 3]P[X3 = 2|X0 = 3]
+ P[X0 = 4]P[X3 = 2|X0 = 4]

Example
It follows that
(0)
3 3 (0) 3 (0)
P[X3 = 2] = µ2 P22 + µ3 P32 + µ4 P42
Note that
 3 3 3
  
P22 P23 P24 0.244 0.231 0.525
P 3 = P32
3 3
P33 3  = 0.247 0.222 0.531
P34
3 3 3
P42 P43 P44 0.249 0.225 0.526
In fact,
P[X3 = 2] = µ(0) × the column of state 2 in P 3
 
0.244 0.231 0.525
0.5 0.2 0.3 × 0.247 0.222 0.531
| {z } 0.249 0.225 0.526
µ(0) | {z }
P3
Therefore,
P[X3 = 2] = 0.5 × 0.244 + 0.2 × 0.247 + 0.3 × 0.249 = 0.2461
Example
Similarly, we can argue that

3 (0) 3 (0)
3 (0)
P[X3 = 3] = µ2 P23 + µ3 P33 + µ4 P43
= µ(0) × the column of state 3 in P 3
3 (0) 3 (0)
3 (0)
P[X3 = 4] = µ2 P24 + µ3 P34 + µ4 P44
= µ(0) × the column of state 4 in P 3
Write
(3) (3) (3)
µ2 = P[X3 = 2], µ3 = P[X3 = 3], µ4 = P[X3 = 4]
(3) (3) (3)
Let µ(3) = (µ2 , µ3 , µ4 ) be the distribution of X3 . Then,
µ(3) = µ(0) P 3 = (0.2461, 0.2274, 0.5265)

Distribution of a Markov chain at an arbitrary step
Proposition
For a time-homogeneous Markov chain {X0 , X1 , . . .} with state space
S = {1, . . . , m} and transition matrix P. Let
(k) (k)
µ(k) = (µ1 , . . . , µm )
(k)
be the distribution of Xk , i.e., µj = P[Xk = j] for k = 0, 1, . . . and
j = 1, . . . , m. Then,
µ(k) = µ(0) P k

Example
Example
In the laptop example, suppose that the initial state X0 follows the
distribution below
j 2 3 4
P[X0 = j] 0.5 0.2 0.3
Find the distribution of X10 .
 
0.2473 0.2258 0.5269
µ(10) = µ(0) P 10

= 0.5 0.2 0.3 0.2473 0.2258 0.5269
0.2473 0.2258 0.5269
= (0.2473, 0.2258, 0.5269)
Question
Can we obtain the distribution of X10 if we do not know µ(0) ?
Distribution after many steps of transitions
Observation
Each column of the ten-step transition matrix has “identical” entries
 
0.2473 0.2258 0.5269
P 10 = 0.2473 0.2258 0.5269
0.2473 0.2258 0.5269
In other word, the distribution of X10 becomes “independent” of X0 .
The entries in the same column are not exactly the same, but the
difference is very small.
After a number of transitions, the distribution of the state becomes
independent of the initial distribution.

Steady state of a Markov chain
Observation
Using Matlab we can check that
 
0.2473 0.2258 0.5269
P 10 ≈ P 11 ≈ · · · = 0.2473 0.2258 0.5269
0.2473 0.2258 0.5269
As a result,
µ(10) ≈ µ(11) ≈ · · · = (0.2473, 0.2258, 0.5269)
When k is large, the distribution of Xk becomes independent of the

initial distribution.
When k is large, the distribution of Xk does not change with time k.
If the above two conditions hold, the Markov chain reaches a steady state.

Limit distribution of a Markov chain
Definition
Let {X0 , X1 , . . .} be a Markov chain with state space S = {1, . . . , m} and
transition matrix P. A probability distribution µ = (µ1 , . . . , µm ) is called
the limit distribution of the Markov chain if
µj = lim P[Xk = j|X0 = i] = lim Pijk for all i, j = 1, . . . , m

k→∞ k→∞
The limit distribution does not depend on the initial distribution.

µj is the long-run fraction of time the Markov chain is in state j.

Stationary distribution of a Markov chain
In the previous example,

µ(11) = µ(10) P
Because µ(11) ≈ µ(10) , we have
µ(10) P ≈ µ(10)
Definition
Let {X0 , X1 , . . .} be a Markov chain with state space S = {1, . . . , m} and
transition matrix P. A probability distribution π = (π1 , . . . , πm ) is called a
stationary distribution of the Markov chain if
πP = π

Stationary distribution of a Markov chain
Since π is a probability distribution,
π1 + · · · + πm = 1 and πj ≥ 0 for j = 1, . . . , m
Suppose µ(0) = π. Then,
µ(1) = µ(0) P = πP = π, µ(2) = µ(1) P = πP = π, ···
If the initial distribution is a stationary distribution π, the distribution

of the Markov chain at any time is always π, i.e., the distribution of
the Markov chain does not change with time.

Limit distribution and stationary distribution
Proposition
If a Markov chain has a limit distribution π, then π must also be a
stationary distribution of the Markov chain.
By this proposition, we can compute the limit distribution π by solving

(
πP = π
π1 + · · · + πm = 1
The above equations are called balance equations.

We have m + 1 equations with m unknowns π1 , . . . , πm .
There is one redundant equation in πP = π.
We can arbitrarily remove one equation from πP = π and solve the
remaining m equations for π1 , . . . , πm .
Never remove π1 + · · · + πm = 1.
Example
Example
Because µ(10) is close to the stationary distribution π, we can solve for π,

instead of computing P 10 , to obtain µ(10) . Because
 
0.3 0 0.7
P = 0.3 0.3 0.4
0.2 0.3 0.5
Let π = (π2 , π3 , π4 ) be the stationary distribution. Then, the balance
equations are given by

0.3π2 + 0.3π3 + 0.2π4 = π2



0.3π + 0.3π = π
3 4 3


 0.7π 2 + 0.4π 3 + 0.5π 4 = π4

π + π + π = 1
2 3 4

Example
We can arbitrarily remove one of the first three equations, say, the third
one. Then, we have

0.3π2 + 0.3π3 + 0.2π4 = π2

0.3π3 + 0.3π4 = π3

π2 + π3 + π4 = 1

By solving this system of linear equations, we obtain

23 21 49
π2 = = 0.2473, π3 = = 0.2258, π4 = = 0.5269
93 93 93
Then,
µ(10) ≈ π = (0.2473, 0.2258, 0.5269)

Example
Example
In the laptop example, the weekly demand follows the distribution below
d 0 1 2 3 4
P[Dn = d] 0.3 0.3 0.2 0.1 0.1
Suppose that a $50 holding cost is incurred for each unsold laptop at the
end of a week. When placing an order, the store pays $1000 for each
computer and sells it at $1200. All unfulfilled demand will be lost. What is
the long-run average profit per week?
Let fj be the expected weekly profit if there are j laptops at the beginning
of the week. Then,
f2 = 0.3 × 2 × (−50) + 0.3(1200 − 50 − 3 × 1000)
+ (0.2 + 0.1 + 0.1)(2 × 1200 − 4 × 1000)
= −1225
Example
Similarly,
f3 = 0.3 × 3 × (−50) + 0.3(1200 − 2 × 50)

+ 0.2(2 × 1200 − 50 − 3 × 1000) + (0.1 + 0.1)(3 × 1200 − 4 × 1000)
= 75
f4 = 0.3 × 4 × (−50) + 0.3(1200 − 3 × 50) + 0.2(2 × 1200 − 2 × 50)

+ 0.1(3 × 1200 − 50 − 3 × 1000) + 0.1(4 × 1200 − 4 × 1000)
= 850
Therefore, the long-run average profit is given by

23 21 49
f2 π2 + f3 π3 + f4 π4 = −1225 × + 75 × + 850 × = 161.83
93 93 93

Balance equations
By π = πP,
πj = π1 P1j + · · · + πm Pmj
πj is the probability of making a transition from state j.

π1 P1j + · · · + πm Pmj is the probability of making a transition into
state j.
These two probabilities are equal, so that the distribution does not
change with time.
In the steady state, the probability of making transition from a set of
states must be equal to the probability of making transition to this
set of states.

Example
Example
In the Markov chain given below, find the stationary distribution.
1/3 1/3 1/3
2/3 0 1 2 3
2/3 2/3 1

Solution
Using the above “cuts” of sets, we have the following balance equations

1 2

 π 0 = π1
3 3



1π = 2π



1 2
3 3

 1
 3 π2 = π3





π0 + π1 + π2 + π3 = 1

which has solution 12 6 3 1

π= , , ,
22 22 22 22

Limit distribution revisited
Question
For a Markov chain, does limk→∞ P[Xk = j|X0 = i] always exist?
Consider the following Markov chain with X0 = 2,

1.0
1 2
1.0
(
0 for k even
P[Xk = 1|X0 = 2] =
1 for k odd
(
1 if k is even
P[Xk = 2|X0 = 2] =
0 if k is odd
Hence, limk→∞ P[Xk = 1|X0 = 2] does not exist.
Question
For a Markov chain, if limk→∞ P[Xk = j|X0 = i] exists for i, j ∈ S, is this
limit probability always independent of the initial distribution?
Consider the following Markov chain:
1.0 1 2 1.0
0.7 3 0.3
If X0 = 1, then µ(k) = (1, 0, 0) for all k ≥ 1.

If X0 = 2, then µ(k) = (0, 1, 0) for all k ≥ 1.
If X0 = 3, then µ(k) = (0.7, 0.3, 0) for all k ≥ 1.
The limit probability is not independent of the initial distribution.
Question
When will a Markov chain have a limit distribution that does not depend
on the initial distribution?
To answer this question, we need to check the properties of each state.

Accessibility and communication
Consider a Markov chain with state space S and transition matrix P. Let
Pijk = P[Xn+k = j|Xn = i]
be the k-step transition probability from i to j. Assume the convention

(
0 1 if i = j
Pij =
0 if i 6= j
Definition
State j is said to be accessible from state i if Pijk > 0 for some k ≥ 0, and
we write i → j. Two states i and j accessible to each other are said to
communicate, and we write i ↔ j.
Because Pii0 = 1, all states communicate with themselves.

Checking accessibility by a transition diagram
Exercise
The transition diagram of a Markov chain is given below. Suppose that all
indicated transitions have a positive probability.
1 3
Is state 3 accessible from state 1? Do they communicate?
Because 1 → 2 and 2 → 3, we have 1 → 3. Similarly, we can argue that

3 → 1. Hence, 1 ↔ 3.

Checking accessibility by a transition diagram
Exercise
For the following Markov chain, is state 1 accessible from state 3? Do they
communicate? Is state 2 accessible from state 4? Do they communicate?
1 2
4 3
We can see 3 → 1 but they do not communicate; 4 → 2, but they do not

communicate either.

Irreducibility
Definition
Two states that communicate are said to be in the same class, i.e., a class
is a set of states in which any two states communicate.
Each state can belong to only one class.

Any two classes are either disjoint or identical.
Definition
A Markov chain is said to be irreducible if all states communicate, i.e., the
state space contains only one class.

Exercise
Exercise
Is the Markov chain on page 45 irreducible? Is the Markov chain on
page 46 irreducible?
Exercise
Consider a Markov chain that has transition matrix
 
0 1 0 0
1 0 0 0
P= 0 0 0 1

0 0 1 0
Is this Markov chain irreducible?

Stationary distribution and irreducibility
The transition diagram of the previous Markov chain is
1 2
4 3
For all 0 ≤ p ≤ 1,
π = (p/2, p/2, (1 − p)/2, (1 − p)/2)
is a stationary distribution.
The stationary distribution is not unique.
Proposition
If a Markov chain is irreducible and has a finite state space, then it has a
unique stationary distribution.
Period of a state
Definition
For a state j ∈ S, let
d(j) = gcd{k : Pjjk > 0}
where gcd denotes the greatest common divisor. Then, d(j) is called the
period of state j. A state of period 1 is said to be aperiodic.
Example
What is the period of state 1 of the Markov chain below?
1.0
1 2
1.0

Exercise
Exercise
What is the period of state 1 of the Markov chain below?
1 2

Period of an irreducible Markov chain
Proposition
Let d(i) be the period of state i. If i ↔ j, then d(i) = d(j).
Period is a class property, i.e., if one state has this property, then all
states in the same class has this property.
All states of an irreducible Markov chain have the same period.
If the period is 1, we say that the Markov chain is aperiodic; if the
period is greater than 1, we say that the Markov chain is periodic.
Periodic Markov chains cannot have a limit distribution.

Recurrence and transience
For each j ∈ S, let τj be the smallest k ≥ 1 such that Xk = j, i.e., τj is the

first time that the Markov chain makes transition to state j after time 0.
Definition
State j is said to be recurrent if
P[τj < ∞|X0 = j] = 1
i.e., with probability 1, the Markov chain starting from state j will go back
to state j. State j is said to be transient if it is not recurrent.
Starting with a recurrent state j, the Markov chain will visit state j
infinite times.
A transient state can only be visited finite times, i.e., if state j is
transient,
lim Pijk = 0 for all i ∈ S
k→∞

Positive recurrence
Exercise
List recurrent and transient states for the examples on pages 45 and 46.
In the example on page 45, all states are recurrent.

In the example on page 46, state 1 is recurrent and states 2, 3, and 4
are transient.
Definition
A recurrent state j is said to be positive recurrent if
E[τj |X0 = j] < ∞

Exercise
Exercise
In the following Markov chain, is state 1 positive recurrent?
0.5
0.5 1 2
1.0
Because
P[τ1 = 1|X0 = 1] = P[X1 = 1|X0 = 1] = 0.5
P[τ1 = 2|X0 = 1] = P[X2 = 1, X1 6= 1|X0 = 1] = 0.5 × 1 = 0.5
and it is not possible that τ1 > 2. Hence,
E[τ1 |X0 = 1] = 0.5 × 1 + 0.5 × 2 = 1.5
State 1 is positive recurrent.
Class properties
Recurrence, transience, and positive recurrence are all class properties.

Proposition
For two states i, j ∈ S with i ↔ j,
state i is recurrent if and only if state j is recurrent
state i is transient if and only if state j is transient
state i is positive recurrent if and only if state j is positive recurrent
If any state of an irreducible Markov chain is recurrent/transient/positive

recurrent, then all states are recurrent/transient/positive recurrent.

Irreducibility and positive recurrence
Proposition
An irreducible Markov chain that has a finite state space must be positive
recurrent.
Exercise
Is the Markov chain on page 45 positive recurrent? Is the Markov chain on
page 46 positive recurrent?

Stationary distribution and positive recurrence
Theorem
Let {X0 , X1 , . . .} be an irreducible, aperiodic Markov chain with state
space S and transition matrix P. Assume that the Markov chain has a
stationary distribution π, i.e.,
X
πP = π, πj = 1, πj ≥ 0
j∈S
Then,
1 the Markov chain is positive recurrent;
2 π is a limit distribution, i.e.,
πj = lim P[Xk = j|X0 = i] for all i, j ∈ S

k→∞
3 πj > 0 for all j ∈ S;

4 the stationary distribution is unique.
Example
Example
Consider the following Markov chain that has an infinite state space
S = {0, 1, . . .}. Is it positive recurrent?
1/3 1/3 1/3 1/3
2/3 0 1 2 3 ……
2/3 2/3 2/3 2/3

Example
This Markov chain is irreducible and aperiodic. Using the “cut” method,
we obtain the balance equations
1π = 2π

j j+1 for j = 0, 1, . . .
3 3
π0 + π1 + · · · = 1

By the first equation, we have

π0
πj =
2j
Then by the second equation,
1 1
π0 1 + + + · · · = 1
2 4
It follows that
1
πj = for j = 0, 1, . . .
2j+1
The Markov chain is positive recurrent since π is a stationary distribution.
Homework
Reading assignment
Read Chapters 5.1–5.4 in Applied Probability and Stochastic
Processes by Feldman and Valdez-Flores
Exercise problems
To be posted on LumiNUS

Discrete-Time Markov Chains: He Shuangchi

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Discrete-Time Markov Chains: He Shuangchi

Uploaded by

Copyright:

Available Formats

Discrete-Time Markov Chains

IE5004 Discrete-Time Markov Chains 1 / 61

IE5004 Discrete-Time Markov Chains 2 / 61

Let X = {X0 , X1 , . . .}, which is a set of random variables indexed by the

If the stochastic process consists of countably many random variables,

The value of X0 is 4. At the end of week 0, the leftover is 4 − D0 . Then,

The stochastic process X follows

We can see X10 depends on X9 and D9 , X9 depends on X8 and D8 , etc.

IE5004 Discrete-Time Markov Chains 5 / 61

P[Xn+1 = j|X0 = i0 , . . . , Xn−1 = in−1 , Xn = in ] = P[Xn+1 = j|Xn = in ]

where f is a function and Y0 , Y1 , . . . are independent of X . Then, X

IE5004 Discrete-Time Markov Chains 6 / 61

P[Xn+1 = j|Xn = i] = P[X1 = j|X0 = i]

For a time-homogeneous Markov chain,

Pij = P[Xn+1 = j|Xn = i]

is called the transition probability from state i to state j.

From now on, we discuss time-homogeneous Markov chain only.

IE5004 Discrete-Time Markov Chains 7 / 61

P22 = P[Xn+1 = 2|Xn = 2] = P[Dn = 0] = 0.3

IE5004 Discrete-Time Markov Chains 8 / 61

By the above transition probabilities,

P[X1 = 2|X0 = 4] = P42 = 0.2

IE5004 Discrete-Time Markov Chains 9 / 61

We may organize the transition probabilities into a matrix,

which is called the (one-step) transition matrix.

IE5004 Discrete-Time Markov Chains 10 / 61

With S = {1, . . . , m}, the transition matrix is an m × m matrix given by

The (i, j)th entry is Pij = P[X1 = j|X0 = i].

IE5004 Discrete-Time Markov Chains 11 / 61

We may also use a transition diagram to describe the transitions.

0.3 2 0.2 0.3 3 0.3

IE5004 Discrete-Time Markov Chains 12 / 61

Example (Gambler’s ruin)

IE5004 Discrete-Time Markov Chains 13 / 61

0.5 0.5 0.5 0.5 0.5 1.0

IE5004 Discrete-Time Markov Chains 14 / 61

P[X2 = 2|X0 = 4] = P[X2 = 2, X1 = 2|X0 = 4]

In the above expression,

IE5004 Discrete-Time Markov Chains 15 / 61

P[X2 = 3|X0 = 4] = (the row of state 4) × (the column of state 3)

P[X2 = 4|X0 = 4] = (the row of state 4) × (the column of state 4)

IE5004 Discrete-Time Markov Chains 17 / 61

Consider the two-step transition matrix

where P is the one-step transition matrix.

IE5004 Discrete-Time Markov Chains 18 / 61

IE5004 Discrete-Time Markov Chains 19 / 61

Theorem (Chapman–Kolmogorov equations)

where P is the transition matrix of the Markov chain. In other words,

IE5004 Discrete-Time Markov Chains 20 / 61

IE5004 Discrete-Time Markov Chains 21 / 61

We may use Matlab to obtain

P[X3 = 2] = P[X3 = 2, X0 = 2] + P[X3 = 2, X0 = 3] + P[X3 = 2, X0 = 4]

IE5004 Discrete-Time Markov Chains 23 / 61

Similarly, we can argue that

µ(3) = µ(0) P 3 = (0.2461, 0.2274, 0.5265)

IE5004 Discrete-Time Markov Chains 25 / 61

IE5004 Discrete-Time Markov Chains 26 / 61

In other word, the distribution of X10 becomes “independent” of X0 .

IE5004 Discrete-Time Markov Chains 28 / 61

µ(10) ≈ µ(11) ≈ · · · = (0.2473, 0.2258, 0.5269)

When k is large, the distribution of Xk becomes independent of the

IE5004 Discrete-Time Markov Chains 29 / 61

µj = lim P[Xk = j|X0 = i] = lim Pijk for all i, j = 1, . . . , m

which has solution 12 6 3 1