The Long Run Behavior of

Markov Chains
Chapter 4
Dr. Shimelis T.
The Long Run Behavior of Markov Chains
Assignment
Doubly Stochastic Matrices
Doubly
Stochastic
Matrices
Reducible Markov Chains
Reducible Markov Chains
Reducible Markov Chains
Reducible Markov Chains
Classification of states
irreducible Markov Chains
Classification of States
Communication is an equivalence relation. That means that

• Every state communicates with itself

• If , then
• If and , then

Chapter 11 © 2020 UMass Amherst Global. All rights reserved.

20
 Classification of States
Periodicity:
Consider the Markov chain shown in
the following Figure. There is a periodic
pattern in this chain.

Consider the Markov chain shown in Figure. There is a periodic pattern in this
chain. Starting from state 0, we only return to 0 at times n=3,6,⋯.

In other words, , if n is not divisible by 3. Such a state is called a periodic

state with period d(0)=3.

Chapter 11 © 2020 UMass Amherst Global. All rights reserved.

21
Classification of States
The period of a state is the largest integer satisfying the following property:
, where is not divisible by . The period of is shown by . If
, for all , then we let .

• If , we say that state is periodic.

• If , we say that state is aperiodic.

➢ If , then .

Chapter 11 © 2020 UMass Amherst Global. All rights reserved.

22
 Classification of States
Consider the Markov chain indicated on the figure
a.Is Class 1={state 1,state 2} aperiodic?
b.Is Class 2={state 3,state 4} aperiodic?
c.Is Class 4={state 6,state 7,state 8} aperiodic?

greatest
Chapter common
11 divisor (GCD) © 2020 UMass Amherst Global. All rights reserved.
23
Using the Law of Total Probability with Recursion
Absorption Probabilities:
Example.

recurrent
recurrent transient
•There are three classes: Class 1 consists of one state, state 0, which is a recurrent state. Class two consists of two states, states 1 and 2,
both of which are transient. Finally, class three consists of one state, state 3, which is a recurrent state.
Note that states 0 and 3 have the following property:

once you enter those states, you never leave them. For this reason, we call them absorbing states.

For our example here, there are two absorbing states. The process will eventually get absorbed in one of them. The first question that we
would like to address deals with finding absorption probabilities.
Chapter 11 © 2020 UMass Amherst Global. All rights reserved.
24
 Poisson Processes
The Poisson Distribution and the Poisson Process
Chapter 5
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Poisson Distribution and the Poisson Process
The Law of Rare Events
The Law of Rare Events
A typical sample path
of a Poisson process
showing the waiting
times Wn and the

sojourn times Sn.

Nonhomogeneous Processes
Nonhomogeneous Processes
 Chapter 6
Continuous Time Markov Chains
Pure Birth Processes
Pure Birth Processes
Pure Birth Processes
Stationary and Limiting Distributions
Stationary and Limiting Distributions
Stationary and Limiting Distributions
Stationary and Limiting Distributions
Stationary and Limiting Distributions
Stationary and Limiting Distributions
Pure Birth Process
Pure Birth Process
Pure Birth Process
Pure Birth Process
Pure Birth Process
Pure Birth Process
Chapter 7
Renewal Phenomena