TIME REVERSIBLE MARKOV CHAINS

Reversibility of Markov Chains: Ross, Sec. 4.8 and Grimmett and Stirzaker, Sec. 6.5

Consider an ergodic Markov chain { 𝑋𝑡 } with state space 𝐸, transition probabilities 𝑝𝑖𝑗 ,
and limiting probabilities
𝑚
𝜋𝑗 = lim 𝑝𝑖𝑗 ; 𝑖, 𝑗 ∈ 𝐸
𝑚→∞

The limiting probabilities are also called the stationary probabilities.

Suppose that the Markov chain { 𝑋𝑡 } has been in operation for a long time.

Define a stochastic process { 𝑌𝑛 } where

𝑌𝑛 = 𝑋𝑚−𝑛 ; 𝑛 = 0, 1, 2 …

where 𝑚 is very large, i.e. { 𝑋𝑡 } has been in operation for a long time.

Note that:

𝑌0 = 𝑋𝑚 , 𝑌1 = 𝑋𝑚−1 , 𝑌2 = 𝑋𝑚−2 , 𝑌3 = 𝑋𝑚−3 …

That is, { 𝑌𝑛 } is obtained by reversing the process { 𝑋𝑡 }, after it has been in operation
for a long time.

1
𝑌4 ← 𝑌3 ← 𝑌2 ← 𝑌1 ← 𝑌0

𝑋𝑚−4 → 𝑋𝑚−3 → 𝑋𝑚−2 → 𝑋𝑚−1 → 𝑋𝑚

What are the transition probabilities of { 𝑌𝑛 }?

Let 𝑄𝑖𝑗 = 𝑃(𝑌𝑛+1 = 𝑗 | 𝑌𝑛 = 𝑖)

Assuming stationary transition probabilities which do not depend on the value of 𝑛:

𝑄𝑖𝑗 = 𝑃(𝑌1 = 𝑗 | 𝑌0 = 𝑖) = 𝑃(𝑋𝑚−1 = 𝑗 | 𝑋𝑚 = 𝑖)

𝑃(𝑋𝑚−1 = 𝑗 𝑎𝑛𝑑 𝑋𝑚 = 𝑖)
𝑄𝑖𝑗 =
𝑃(𝑋𝑚 = 𝑖)

𝑃(𝑋𝑚 = 𝑖 | 𝑋𝑚−1 = 𝑗) 𝑃(𝑋𝑚−1 = 𝑗)

𝑄𝑖𝑗 =
𝑃(𝑋𝑚 = 𝑖)

Now, 𝑃(𝑋𝑚 = 𝑖 | 𝑋𝑚−1 = 𝑗) = 𝑝𝑗𝑖

𝑚−1
Since 𝑚 is very large, 𝑃(𝑋𝑚−1 = 𝑗) = lim 𝑝𝑖𝑗 = 𝜋𝑗
𝑚→∞

And 𝑃(𝑋𝑚 = 𝑖) = lim 𝑝𝑖𝑖𝑚 = 𝜋𝑖

𝑚→∞

𝑝𝑗𝑖 𝜋𝑗
𝑄𝑖𝑗 =
𝜋𝑖

2
 𝑝𝑗𝑖 𝜋𝑗
Note that the expression for this probability 𝑄𝑖𝑗 = 𝑃(𝑌𝑛+1 = 𝑗 | 𝑌𝑛 = 𝑖) = shows
𝜋𝑖
that the state of the process at time 𝑛 + 1 depends only on the state at time 𝑛 and not
on the states before time 𝑛. This is the Markovian property.

This argument can also be made as follows:

𝑌4 ← 𝑌3 ← 𝑌2 ← 𝑌1 ← 𝑌0

𝑋𝑚−4 → 𝑋𝑚−3 → 𝑋𝑚−2 → 𝑋𝑚−1 → 𝑋𝑚

Since { 𝑋𝑡 } is a Markov Chain, 𝑋𝑚−1 , 𝑋𝑚 , 𝑋𝑚+1 … are independent of 𝑋𝑚−3 .

⇒ 𝑌1 and 𝑌0 are independent of 𝑌3 .

⇒ 𝑌3 is independent of 𝑌1 and 𝑌0

And since { 𝑋𝑡 } is a Markov Chain, 𝑋𝑚−2 and 𝑋𝑚−3 are dependent.

⇒ 𝑌2 and 𝑌3 are dependent.

⇒ 𝑌3 and 𝑌2 are dependent.

⇒ 𝑌3 is independent of 𝑌1 and 𝑌0 , but 𝑌3 and 𝑌2 are dependent.

i.e. the value of 𝑌3 depends only on the value of 𝑌2 (state of the process at the preceding
time period), and does not depend on the states at any earlier time periods. This is the
Markovian property.

⇒ { 𝑌𝑛 } is a Markov Chain.

3
Time Reversible Markov Chains

If the transition matrices for a Markov Chain { 𝑋𝑡 } and the corresponding reverse
process { 𝑌𝑛 } are the same, then the Markov Chain is said to be time reversible.

That is { 𝑋𝑡 } is time reversible if:

𝑄𝑖𝑗 = 𝑝𝑖𝑗 ; ∀ 𝑖, 𝑗 ∈ 𝐸

Now:
𝑝𝑗𝑖 𝜋𝑗
𝑄𝑖𝑗 = 𝑝𝑖𝑗 ⇔ = 𝑝𝑖𝑗 ⇔ 𝜋𝑖 𝑝𝑖𝑗 = 𝜋𝑗 𝑝𝑗𝑖 ; ∀ 𝑖, 𝑗 ∈ 𝐸
𝜋𝑖

𝜋𝑖 𝑝𝑖𝑗 can be interpreted as the rate at which the process goes from state 𝑖 to state 𝑗.

𝜋𝑗 𝑝𝑗𝑖 can be interpreted as the rate at which the process goes from state 𝑗 to state 𝑖.

Theorem:

Let { 𝑋𝑡 } be an ergodic Markov Chain with state space 𝐸, transition probabilities 𝑝𝑖𝑗 ,
𝑚
and limiting probabilities 𝜋𝑗 = lim 𝑝𝑖𝑗 ; 𝑖, 𝑗 ∈ 𝐸. If there exists a set of non-negative real
𝑚→∞
numbers {𝑎𝑖 ; 𝑖 ∈ 𝐸} such that

𝑎𝑖 𝑝𝑖𝑗 = 𝑎𝑗 𝑝𝑗𝑖 ; ∀ 𝑖, 𝑗 ∈ 𝐸

and

∑ 𝑎𝑖 = 1
𝑖∈𝐸

then { 𝑋𝑡 } is time reversible, and 𝑎𝑖 = 𝜋𝑖 ∀ 𝑖 ∈ 𝐸.

4
Example:

Consider a Markov Chain with state space 𝐸 = {0, 1, 2, … 𝑀}

Transition probabilities:

𝑝00 = 1 − 𝛼0 , 𝑝01 = 𝛼0

𝑝10 = 1 − 𝛼1 , 𝑝12 = 𝛼1

𝑝21 = 1 − 𝛼2 , 𝑝23 = 𝛼2

𝑝𝑖,𝑖−1 = 1 − 𝛼𝑖 , 𝑝𝑖,𝑖+1 = 𝛼𝑖

𝑝𝑀−1,𝑀−2 = 1 − 𝛼𝑀−1 , 𝑝𝑀−1,𝑀 = 𝛼𝑀−1

𝑝𝑀,𝑀−1 = 1 − 𝛼𝑀 , 𝑝𝑀,𝑀 = 𝛼𝑀

Transition diagram:

0 1 2 3 … 𝑀−2 𝑀−1 𝑀

It can be shown that this Markov Chain is time-reversible.

Let us find the limiting probabilities 𝜋𝑖 ; 𝑖 = 0, 1, 2 … 𝑀

5
Now:

𝜋𝑖 𝑝𝑖𝑗 = 𝜋𝑗 𝑝𝑗𝑖 ; ∀ 𝑖, 𝑗 ∈ 𝐸

For 𝑖 = 0, 𝑗 = 1: 𝑝𝑖𝑗 = 𝑝01 = 𝛼0 ; 𝑝𝑗𝑖 = 𝑝10 = 1 − 𝛼1

⇒ 𝜋0 𝛼0 = 𝜋1 (1 − 𝛼1 )

𝛼0
⇒ 𝜋1 = 𝜋
(1 − 𝛼1 ) 0

For 𝑖 = 1, 𝑗 = 2: 𝑝𝑖𝑗 = 𝑝12 = 𝛼1 ; 𝑝𝑗𝑖 = 𝑝21 = 1 − 𝛼2

⇒ 𝜋1 𝛼1 = 𝜋2 (1 − 𝛼2 )

𝛼1 𝛼1 𝛼0
⇒ 𝜋2 = 𝜋1 = 𝜋
(1 − 𝛼2 ) (1 − 𝛼2 ) (1 − 𝛼1 ) 0

For 𝑖 = 2, 𝑗 = 3: 𝑝𝑖𝑗 = 𝑝23 = 𝛼2 ; 𝑝𝑗𝑖 = 𝑝32 = 1 − 𝛼3

⇒ 𝜋2 𝛼2 = 𝜋3 (1 − 𝛼3 )

𝛼2 𝛼2 𝛼1 𝛼0
⇒ 𝜋3 = 𝜋2 = 𝜋
(1 − 𝛼3 ) (1 − 𝛼3 )(1 − 𝛼2 ) (1 − 𝛼1 ) 0

and so on…

In general:
𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
𝜋𝑖 = 𝜋 ; 𝑖 = 1, 2 … 𝑀
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 ) 0

Now, the limiting probabilities are such that:

∑ 𝜋𝑖 = 1
𝑖∈𝐸
6
 𝑀 𝑀

⇒ ∑ 𝜋𝑖 = 1 ⇒ 𝜋0 + ∑ 𝜋𝑖 = 1
𝑖=0 𝑖=1

𝑀
𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
⇒ 𝜋0 + ∑ [ 𝜋 ]=1
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 ) 0
𝑖=1

𝑀
𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
⇒ 𝜋0 ( 1 + ∑ [ ] )=1
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 )
𝑖=1

⇒ The limiting probabilities are:

𝑀 −1
𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
𝜋0 = ( 1 + ∑ [ ] )
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 )
𝑖=1

𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
𝜋𝑖 = 𝜋 ; 𝑖 = 1, 2 … 𝑀
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 ) 0

In the special case where 𝛼0 = 𝛼1 = 𝛼2 = ⋯ = 𝛼𝑀 = 𝛼 :

𝑀 −1
𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
𝜋0 = ( 1 + ∑ [ ] )
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 )
𝑖=1

7
 𝑀 −1 𝑀 −1
𝛼𝑖 𝛼𝑖
𝜋0 = ( 1 + ∑ [ ] ) = ( ∑[ ] )
(1 − 𝛼)𝑖 (1 − 𝛼)𝑖
𝑖=1 𝑖=0

𝛼𝑖
But ∑𝑀
𝑖=0 [(1−𝛼)𝑖 ] is the sum of (𝑀 + 1) terms of a Geometric Progression with first term
𝛼
of 1 and a common ratio of (1−𝛼)

𝛼
Let 𝛽 = (1−𝛼) , the common ratio.

−1
1 − 𝛽 𝑀+1 1−𝛽
𝜋0 = ( ) =
1−𝛽 1 − 𝛽 𝑀+1

And:
𝛼𝑖−1 𝛼𝑖−2 … 𝛼1 𝛼0
𝜋𝑖 = 𝜋 ; 𝑖 = 1, 2 … 𝑀
(1 − 𝛼𝑖 )(1 − 𝛼𝑖−1 ) … (1 − 𝛼2 ) (1 − 𝛼1 ) 0

𝛼𝑖 𝑖
𝛽 𝑖 (1 − 𝛽)
𝜋𝑖 = 𝜋 = 𝛽 𝜋0 = ; 𝑖 = 1, 2 … 𝑀
(1 − 𝛼)𝑖 0 1 − 𝛽 𝑀+1

Note that 𝛽 0 = 1, so a general expression for the limiting probabilities for this special
case is:

𝛽 𝑖 (1 − 𝛽)
𝜋𝑖 = ; 𝑖 = 0, 1, 2 … 𝑀
1 − 𝛽 𝑀+1