CSE710

Advanced Artificial Intelligence

Lecture 3: Markov Chain

Dr. Md. Golam Rabiul Alam

BRAC University
Markov Chain & Stochastic
Process

 Stochastic: having a random

probability distribution or pattern
that may be analyzed statistically
but may not be predicted precisely.
 Stochastic process: stochastic or
random process is a mathematical
object usually defined as a family
of random variables.
Stochastic Process

 Many stochastic processes can be

represented by time series.
 However, a stochastic process is by
nature continuous while a time
series is a set of observations
indexed by integers.
 A stochastic process may involve
several related random variables.
Example Stochastic Processes

 Common examples include the

growth of a bacterial population,
an electrical current fluctuating
due to thermal noise.
Example 1:

 Consider, there are 3 mobile network

operator company in the market i.e.,
A, B, and C respectively.

 A consumer currently using a mobile

SIM of company A
 There is a 70% chance that the
customer will use the same company’s
SIM in next year. There also 20% and
10% chances the s/he will shift to
company B and C respectively in next
year.
 Example 1:

 A consumer currently using a mobile

SIM of Company B
 There is 50% and 10% chances the s/he
will shift to company A and C
respectively in next year.
 A consumer currently using a mobile
SIM of brand C
 There is 60% and 20% chances the s/he
will shift to company A and B
respectively in next year.
What will be the market share of the three MNOs in next year?
Example 2:

 An employee tried not to be late in

his office. If s/he is late one day, he
is 90% sure to be on time in next
day. If s/he is on time, then there
will be 30% chance that he will be
late in next day.

In the long run, how often is the

employee late in his/her office?
Example:3
 Consider, there are 3 cloud service providers (CSPs)
in the market namely A, B, and C respectively with
equal market share at this moment. But during the
year, the following changes took place:
 CSP-A retained 90% of its customers and lost 3% to CSP-B and
7% to CSP-C.
 CSP-B retained 70% of its customers and lost 10% to CSP-A and
20% to CSP-C.
 CSP-C retained 80% of its customers and lost 10% to CSP-A and
10% to CSP-B.
Assuming that no change take place in the usage
habits of the customers. What will be the market
shares of the three CSPs at the end of the 1st year and
also in the 2nd year?
 Markov Model
 A Markov model is a stochastic model used to
model randomly changing systems over time.

X0 be the number of student entered in this classroom at time t=t0

X1 be the number of student entered in this classroom at time t=t0 + 1
.
.
.
Xn be the number of student entered in this classroom at time t=tn

Let, { Xn , n=0, 1, 2, …} be a stochastic process that takes on a

finite number of possible values.
 Markov Model
Notational convention:

Xn= i

Where n is the time-stamp and i is the number of

student (state).
Andrei Markov (1856-1922)
(A Russian mathematician)
http://www-gap.dcs.st-and.ac.uk/~history/Mathematicians/Markov.html

 Markov is particularly
remembered for his
study of Markov chains,
sequences of random
variables in which the
future variable is
determined by the
present variable but is
independent of the way
in which the present
state arose from its
predecessors. This work
launched the theory of
stochastic processes
Terminologies of Markov Model

 States
 Transition
Probabilities
 States in MM
Timestamps Today Tomorrow

Rain No-Rain
No-Rain Rain
Rain Rain
No-Rain No-Rain

States
 States and Transition Probabilities
Matrix (TPM) in MM
Timestamps Today Tomorrow

Rain No-Rain
No-Rain Rain
Rain Rain
States
No-Rain No-Rain

TPM
Rain 0.5 No-
Tomorrow
0.5 Rain 0.5
Rain No-
Rain
0.5
Rain 0.5 0.5
Today

No- 0.5 0.5 Rain No-Rain

Initial
rain 0.2 0.8
Probabilities
Transition Probabilities (1-step)

 Probabilities from state i to state

j after 1 step time period ,
denoted by pij is defined as,
pij=P{Xn+1=j | Xn=i}
Examples:
P{X2=N | X1=R}=pRN
P{X2=N | X1=R}=pRN(1)
P{X2=2 | X1=1}=p12(1)
Transition Probabilities (n-step)

 Probabilities from state i to state

j after n-step time period ,
denoted by pij(n) is defined as,
pij (n) =P{Xn+1=j | X1=i}
Examples:
P{X3=N | X1=R}=pRN(2)
P{X3=2 | X1=1}=p12(2)
Types of MC Problems:
 How to calculate the probability of the
states?
Example: P(X2=1)=?
What is the probability that on the 2nd day it
will be rain?
 How to calculate the probability after n-
steps?
Example: P(X3=2|X1=1)=p13(2)=?

 How to find probability of the chain?

Example: P(X3=2, X2=2, X1=1)=?
 Notations:
q0 Initial probabilities of the states
q1 Probabilities of the states after 1 time period
.
.
qn Probabilities of the states after n-time period

P TPM after 1 time period

P2 TPM after 2 time period
.
.
Pn TPM after n time period
Types of MC Problems:
 How to calculate the probability of the
states?
Example: P(X2=1)=?
Transition probability matrix: Rows indicate the current state and
column indicate the transition . For example, given the current state of
A, the probability of going to the next state A is s. Given the current
state A’, the probability of going from this state to A is r. Notice that the
rows sum to 1. We will call this matrix P.
Initial State distribution matrix:

 This is the initial probabilities of being in state A as well as not A

, A’ . Notice again that the row probabilities sum to one, as they
should.

A A'
S0   t 1 t
First and second state matrices:

 If we multiply the Initial state matrix by the transition

matrix, we obtain the first state matrix.

S1  S o P

 If the first state matrix is multiplied by the transition

matrix we obtain the second state matrix:

S 2  S1 P  S o P  P  S 0 P 2
Kth – State matrix

 If this process is repeated we will obtain the following

expression: The entry in the ith row and j th column indicates
the probability of the system moving from the i th state to the j th
state in k observations or trials.

S k  S k 1 P  S 0 P k
 An example: An insurance company classifies drivers as low-risk if they are accident-
free for one year. Past records indicate that 98% of the drivers in the low-risk category (L)
will remain in that category the next year, and 78% of the drivers who are not in the low-
risk category ( L’) one year will be in the low-risk category the next year.

1. Find the transition matrix, P

L L'
L  0.98 0.02 
P  
L '  0.78 0.22 

1. If 90% of the drivers in the community are in the low-risk category

this year, what is the probability that a driver chosen at random
from the community will be in the low-risk category the next year
P(X2=L)= [i.e. P(X1=L)]? The year after next [i.e. P(X2=L)]? (answer 0.96,
S2(L)=? 0.972Lfrom L 'matrices) S  S P  S P  P  S P2
2 1 o 0

S 0   0.90 0.10 
P(X1=L)=
S1(L)=? S1  S o P L L'
S 2   0.972 0.028 
L L'
S1   0.96 0.04 
Finding the kth State matrix

 Use the formula k S S P

0
k
to find the 4th state
matrix for the previous problem.

S4  S0 P 4

4
 0.98 0.02 

 0.90 0.10 
 =
 0.78 0.22 

 .97488 0.02512 
 after four states, the percentage of low-risk drivers has
increased to .97488
Types of MC Problems:
 How to calculate the probability after n-
steps?
Example: P(X3=2|X1=1)=p13(2)=?
 Notations:
q0 Initial probabilities of the states
q1 Probabilities of the states after 1 time period
.
.
qn Probabilities of the states after n-time period

P TPM after 1 time period

P2 TPM after 2 time period
.
.
Pn TPM after n time period
Example 1:

 Consider, there are 3 mobile network

operator company in the market i.e.,
A, B, and C respectively.

 A consumer currently using a mobile

SIM of company A
 There is a 70% chance that the
customer will use the same company’s
SIM in next year. There also 20% and
10% chances the s/he will shift to
company B and C respectively in next
year.
Example 1:

 A consumer currently using a mobile

SIM of Company B
 There is 50% and 10% chances the s/he
will shift to company A and C
respectively in next year.
 A consumer currently using a mobile
SIM of brand C
 There is 60% and 20% chances the s/he
will shift to company A and B
respectively in next year.
Example 1:
 Consider the present market share of A,B,C MNOs
are 60%, 30% and 10% respectively.

 Find the probability that a customer who is currently

using a SIM of MNO A, will shift to MNO B after 2
years?

P(X2=B|X0=A)=pAB(2)=?
Example 1:

 Find the probability that a customer who is currently

using a SIM of MNO A, will shift to MNO B after 2
years?

P(X2=B|X0=A)=pAB(2)=0.24
Types of MC Problems:

 How to find probability of the chain?

Example: P(X3=2, X2=2, X1=1)=?
Types of MC Problems:
Stationary Probability for a MC

Let, {Xn, n>=0} be a Markov chain

with TPM P. If there exists a
probability vector q such that:
qP=q
Then q is called a stationary
probability or steady-state or limiting
distribution for the Markov chain.
Stationary Probability for a MC
Stationary Probability for a MC
Example 1:

 Consider, there are 3 mobile network

operator company in the market i.e.,
A, B, and C respectively.

 A consumer currently using a mobile

SIM of company A
 There is a 70% chance that the
customer will use the same company’s
SIM in next year. There also 20% and
10% chances the s/he will shift to
company B and C respectively in next
year.
Example 1:

 A consumer currently using a mobile

SIM of Company B
 There is 50% and 10% chances the s/he
will shift to company A and C
respectively in next year.
 A consumer currently using a mobile
SIM of brand C
 There is 60% and 20% chances the s/he
will shift to company A and B
respectively in next year.
Example 1:
 Consider the present market share of A,B,C MNOs
are 60%, 30% and 10% respectively.

 Find the market share of the MNOs in the steady-

state?
Market share of the MNOs in the
steady-state:
Thank You

 We will discuss HMM in the next

class.