MARKOV CHAINS - CLASSIFICATION OF STATES

Review:

{𝑋𝑛 ; 𝑛 = 0, 1 , 2 … } is a Markov Chain (i.e. 𝑋𝑛 is a discrete-time stochastic process with a

discrete state space).

One-step stationary transition probabilities: 𝑝𝑖𝑗 = 𝑃( 𝑋𝑡+1 = 𝑗 | 𝑋𝑡 = 𝑖); 𝑖, 𝑗 ∈ 𝐸; 𝑡 =

0, 1, 2 …

One-step transition matrix, 𝑃: a matrix of one-step transition probabilities, whose

(𝑖, 𝑗)𝑡ℎ element is 𝑝𝑖𝑗 .
𝑝00 𝑝01 𝑝02 ⋯
𝑝 𝑝11 𝑝12 ⋯
𝑃 = [ 10 ]
𝑝20 𝑝21 𝑝22 ⋯
⋮ ⋮ ⋮ ⋮

𝑛
𝑛-step transition probabilities: 𝑝𝑖𝑗 = 𝑃( 𝑋𝑛+𝑡 = 𝑗 | 𝑋𝑡 = 𝑖); 𝑖, 𝑗 ∈ 𝐸; 𝑡 = 0, 1, 2 … ; 𝑛 =
0, 1 ,2

𝑛-step transition matrix 𝑃(𝑛) : a matrix of 𝑛-step transition probabilities, whose (𝑖, 𝑗)𝑡ℎ
𝑛
element is 𝑝𝑖𝑗

𝑃(𝑛) = 𝑃𝑛

1
Classification of States

𝑛
Definition: A state 𝑗 is said to be accessible from state 𝑖 if 𝑝𝑖𝑗 > 0 for some 𝑛 ≥ 0

That is: state 𝑗 is accessible from state 𝑖 if there is a non-zero probability of reaching
state 𝑗 from state 𝑖 after 𝑛 steps for some 𝑛 ≥ 0.

We write: 𝑖 → 𝑗

Note that 𝑖 → 𝑗 does not require that state 𝑗 can be reached from state 𝑖 directly in one
step. As long as a path with non-zero probability exists from state 𝑖 to state 𝑗 (possibly
passing through many other intermediate states and requiring many steps), then 𝑖 → 𝑗.

Note also that the definition allows the possibility of going from state 𝑖 to state 𝑗 in
zero steps (“for some 𝑛 ≥ 0”). But the only state that can be reached from state 𝑖 in zero
steps is state 𝑖 itself since 𝑝𝑖𝑖0 = 𝑃( 𝑋𝑡+0 = 𝑋𝑡 = 𝑖 | 𝑋𝑡 = 𝑖) = 1; 𝑡 = 0, 1, 2, …

Hence 𝑖 → 𝑖 ∀𝑖 ∈ 𝐸. That is, every state is accessible from itself.

Definition: States 𝑖 and 𝑗 are said to communicate if 𝑖 → 𝑗 and 𝑗 → 𝑖.

We write: 𝑖 ↔ 𝑗

Note again that 𝑖 ↔ 𝑗 does not require that state 𝑗 can be reached from state 𝑖 directly
in one step or that 𝑖 can be reached from state 𝑗 in one step.
2
Theorem:

i) 𝑖 ↔ 𝑖 ∀𝑖 ∈ 𝐸 {Communication is reflexive}

ii) If 𝑖 ↔ 𝑗 then 𝑗 ↔ 𝑖 {Communication is symmetric}

iii) If 𝑖 ↔ 𝑗 and 𝑗 ↔ 𝑘 then 𝑖 ↔ 𝑘 {Communication is transitive}

Proof:

i) 𝑝𝑖𝑖0 = 𝑃( 𝑋𝑡+0 = 𝑋𝑡 = 𝑖 | 𝑋𝑡 = 𝑖) = 1; ∀𝑖 ∈ 𝐸; 𝑡 = 0, 1, 2, … ⇒ 𝑖 → 𝑖

𝑖 → 𝑖 and 𝑖 → 𝑖 ⇒ 𝑖 ↔ 𝑖 ∀𝑖 ∈ 𝐸 ∎

ii) This result follows from the definition of communication.

iii) Suppose 𝑖 ↔ 𝑗
𝑛
Then from the definition of accessibility, there exists an integer 𝑛 ≥ 0 such that 𝑝𝑖𝑗 >0

Suppose 𝑗 ↔ 𝑘
𝑚
Then from the definition of accessibility, there exists an integer 𝑚 ≥ 0 such that 𝑝𝑗𝑘 >0

𝑛 𝑚 𝑛 𝑚
𝑝𝑖𝑗 > 0, 𝑝𝑗𝑘 >0 ⇒ 𝑝𝑖𝑗 𝑝𝑗𝑘 >0

By the Chapman-Kolmogorov Equations:

𝑚+𝑛 𝑛 𝑚
𝑝𝑖𝑘 = ∑ 𝑝𝑖𝑟 𝑝𝑟𝑗
𝑟∈𝐸
3
All terms in the summation above are non-negative.
𝑛 𝑚
And the term for 𝑟 = 𝑘 is 𝑝𝑖𝑗 𝑝𝑗𝑘 >0

𝑚+𝑛 𝑛 𝑚 𝑛 𝑚
⇒ 𝑝𝑖𝑘 = ∑ 𝑝𝑖𝑟 𝑝𝑟𝑗 ≥ 𝑝𝑖𝑗 𝑝𝑗𝑘 > 0
𝑟∈𝐸

𝑚+𝑛
⇒ There exists an integer 𝑚 + 𝑛 ≥ 0 such that 𝑝𝑖𝑘 >0

⇒ By the definition of accessibility, 𝑖 → 𝑘

Similarly, we can show that 𝑘 → 𝑖 (left as an exercise).

𝑖 → 𝑘 and 𝑘 → 𝑖 ⇒ 𝑖↔𝑘 ∎

From the theorem above, it can be seen that communication is a binary relation that is
reflexive, symmetric and transitive ⇒ Communication is an equivalence relation.

Definition: Let 𝑅 be an equivalence relation defined on a set 𝑋. The equivalence class

of 𝑎 ∈ 𝑋 is the subset {𝑥 ∈ 𝑋: 𝑥 𝑅 𝑎}.

That is, the equivalence class of 𝑎 is the set of all elements of 𝑋 that are equivalent to 𝑎
(i.e. that are related to 𝑎 through the binary relation 𝑅).

4
It can be shown that the equivalence classes of 𝑋 form a partition of 𝑋, i.e. they are
disjoint, and their union is 𝑋.

Hence, for a Markov Chain with state space 𝐸 and for the equivalence relation of
communication , the equivalence class of state 𝑖 is {𝑗 ∈ 𝐸: 𝑖 ↔ 𝑗}, the set of all states in 𝐸
that communicate with state 𝑖.

The equivalence classes of a Markov Chain produced by the binary relation of

communication are called the communicating classes of the Markov Chain.

The communicating classes therefore form a partition of the state space.

A transition diagram is often useful for determining the communicating classes of a

Markov Chain.

5
Example:

A finite Markov chain {𝑋𝑛 } with state space 𝐸 = {1, 2, 3, 4, 5, 6, 7, 8} has stationary
transition matrix 𝑃, where

1 2 3 4 5 6 7 8
1 1
1 0 0 0 0 0 0
2 2

0 0 0 1 0 0 0 0
2
2 1
3 0 0 0 0 0 0
3 3

4 1 0 0 0 0 0 0 0
𝑃=
5 1 1 1
0 0 0 0 0
2 4 4

6 0 0 0 0 0 0 0 1
1 2
7 0 0 0 0 0 0
3 3

8 [1 0 0 0 0 0 0 0 ]

Find the communicating classes of the Markov chain.

Look at each row in succession.

If the (𝑖, 𝑗)𝑡ℎ element is non-zero, this means that 𝑝𝑖𝑗 > 0, so state 𝑗 is accessible from
state 𝑖 in one step, i.e. state 𝑖 can transition to state 𝑗 in one step. Draw an arrow → from
state 𝑖 to state 𝑗.

Note that if 𝑝𝑖𝑗 > 0 and 𝑝𝑗𝑖 > 0, then an arrow → must be drawn from state 𝑖 to state 𝑗,
and another arrow ← must be drawn from state 𝑗 to state 𝑖.

If the (𝑖, 𝑖)𝑡ℎ element (on the diagonal) is non-zero, this means that 𝑝𝑖𝑖 > 0, so state 𝑖
can transition to itself in one step. Draw an arrow ↻ from state 𝑖 to itself.

6
From the diagram, it is clear that 3 ↔ 7.

The communication and accessibility between the other states may not be as clear.

For example, does state 6 communicate with state 4? That is: is state 4 accessible from
state 6, and is state 6 accessible from state 4?

Recall that it is not required that state 4 is accessible from state 6 directly in one step.

Examination of the diagram shows that state 4 is indeed accessible from state 6,
through the path: 6 → 8 → 1 → 2 → 4.

And state 6 is accessible from state 4 through the path: 4 → 1 → 6

7
So 6 ↔ 4

Another example: does state 5 communicate with state 2?

State 2 is accessible from state 5, 5 → 2, through the path 5 → 1 → 2. But no path exists
from state 2 to state 5, so state 5 is not accessible from state 2. So state 5 does not
communicate with state 2.

From careful examination of the transition diagram, it can be observed that:

States 1, 2, 4, 6, and 8 all communicate with one another.

States 3 and 7 communicate with each other.

State 5 does not communicate with any other state. But every state always
communicates with itself, so state 5 communicates with itself.

The communicating classes of the Markov Chain are therefore:

{1, 2, 4, 6, 8}, {3, 7}, {5}

Note that these communicating classes partition the state space 𝐸 = {1, 2, 3, 4, 5, 6, 7, 8}.
They are disjoint, and their union is the set 𝐸.

8
If every state in a Markov Chain can be reached from every other state, then all states
communicate. There is only one communicating class – which is the entire state space
𝐸. In this case, the Markov Chain is said to be irreducible.

Example:

A finite Markov chain {𝑋𝑛 } with state space 𝐸 = {1, 2, 3} has stationary transition
matrix 𝑃, where

1 2 3
1 1
1 0
2 2

2 1 1 1
𝑃= 2 4 4

1 2
3 [0 3 3 ]

The transition diagram for this Markov Chain is:

1 3

All states communicate ⇒ the Markov Chain is irreducible and the only
communicating class is {1, 2, 3}.

9
Example:

A finite Markov chain {𝑋𝑛 } with state space 𝐸 = {1, 2, 3, 4} has stationary transition
matrix 𝑃, where

1 2 3 4
1 1
1 0 0
2 2

2 1 1
0 0
2 2
𝑃=
1 1 1 1
3 4 4 4 4

4 [0 0 0 1 ]

The transition diagram for this Markov Chain is:

1 2

3 4

The communicating classes of this Markov Chain are:

{1, 2}, {3}, {4}

Note that state 4 is an absorbing state. If the process reaches state 4, then it never
leaves.

10
Definition: Let 𝐶 be a communicating class of a Markov Chain.
𝑛
If 𝑖 ∈ 𝐶 and 𝑝𝑖𝑗 = 0, ∀𝑛 ≥ 0, 𝑖 ∈ 𝐶, 𝑗 ∉ 𝐶 then 𝐶 is said to be closed.

That is: if no state 𝑗 outside of 𝐶 is accessible from any state 𝑖 in 𝐶, then 𝐶 is closed.

This means that if the process enters the communicating class 𝐶, it will never leave. It
may move between states in 𝐶, but it will not transition outside of 𝐶.

In the previous example:

{1, 2}, {4} are closed communicating classes.

But {3} is not closed. (Don’t say “open”). States outside of {3} are accessible from state
3.

Exercise: Classify each communicating class in the previous examples as closed or not
closed.

11