S TOCHASTIC PROCESSES AND M ARKOV CHAINS1

Matteo Brunelli

University of Trento, Italy

April 17, 2018

1 Based on the book: Hillier & Lieberman, Introduction to Operations Research

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 1 / 25
Stochastic processes (discrete time – discrete states

Consider a system whose state is observed in the discrete time. The state of
the system is also finite and ranges in {0, 1, 2, . . . , M − 1, M }

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 2 / 25

Stochastic processes (discrete time – discrete states

Consider a system whose state is observed in the discrete time. The state of
the system is also finite and ranges in {0, 1, 2, . . . , M − 1, M }

D EFINITION
A stochastic process is an indexed collection of random variables {Xt }t=1,...,T
where the index t runs through a given set T .

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 2 / 25

Stochastic processes (discrete time – discrete states

Consider a system whose state is observed in the discrete time. The state of
the system is also finite and ranges in {0, 1, 2, . . . , M − 1, M }

D EFINITION
A stochastic process is an indexed collection of random variables {Xt }t=1,...,T
where the index t runs through a given set T .

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 2 / 25

Stochastic processes (discrete time – discrete states

Consider a system whose state is observed in the discrete time. The state of
the system is also finite and ranges in {0, 1, 2, . . . , M − 1, M }

D EFINITION
A stochastic process is an indexed collection of random variables {Xt }t=1,...,T
where the index t runs through a given set T .

E XAMPLE
The value of a given investment
The price of oil
The amount of rain
The quantity of a given good in the inventory
···

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 2 / 25

Inventory problem

E XAMPLE
1 A camera store stocks a particular model camera that can be ordered
weekly. Let {D1 , D2 , . . .} represent the demand for this camera (the
number of units that would be sold if the inventory is not depleted) during
the first week, second week, . . ., respectively. It is assumed that the Di
are independent and identically distributed random variables having a
Poisson distribution with a mean of 1.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 3 / 25

Inventory problem

E XAMPLE
1 A camera store stocks a particular model camera that can be ordered
weekly. Let {D1 , D2 , . . .} represent the demand for this camera (the
number of units that would be sold if the inventory is not depleted) during
the first week, second week, . . ., respectively. It is assumed that the Di
are independent and identically distributed random variables having a
Poisson distribution with a mean of 1.
2 Let X0 represent the number of cameras on hand at the outset, X1 the
number of cameras on hand at the end of week 1, X2 the number of
cameras on hand at the end of week 2, and so on.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 3 / 25

Inventory problem

E XAMPLE
1 A camera store stocks a particular model camera that can be ordered
weekly. Let {D1 , D2 , . . .} represent the demand for this camera (the
number of units that would be sold if the inventory is not depleted) during
the first week, second week, . . ., respectively. It is assumed that the Di
are independent and identically distributed random variables having a
Poisson distribution with a mean of 1.
2 Let X0 represent the number of cameras on hand at the outset, X1 the
number of cameras on hand at the end of week 1, X2 the number of
cameras on hand at the end of week 2, and so on.
3 On Saturday night the store places an order that is delivered in time for
the next opening on Monday. The store uses the following order policy: If
there are no cameras in stock, the store orders 3 cameras. However, if
there are any cameras in stock, no order is placed.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 3 / 25

Inventory problem

E XAMPLE
1 A camera store stocks a particular model camera that can be ordered
weekly. Let {D1 , D2 , . . .} represent the demand for this camera (the
number of units that would be sold if the inventory is not depleted) during
the first week, second week, . . ., respectively. It is assumed that the Di
are independent and identically distributed random variables having a
Poisson distribution with a mean of 1.
2 Let X0 represent the number of cameras on hand at the outset, X1 the
number of cameras on hand at the end of week 1, X2 the number of
cameras on hand at the end of week 2, and so on.
3 On Saturday night the store places an order that is delivered in time for
the next opening on Monday. The store uses the following order policy: If
there are no cameras in stock, the store orders 3 cameras. However, if
there are any cameras in stock, no order is placed.
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1
M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 3 / 25
Markov chain

D EFINITION (M ARKOV CHAIN )

A stochastic process {Xt } is said to be a Markov chain if

P {Xt+1 = j|X0 = k0 , X1 = k1 , . . . , Xt−1 = kt−1 , Xt = i} = P {Xt+1 = j|Xt = i}

for t = 0, 1, . . . and every sequence i, j, k0 , . . . , kt .

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 4 / 25

Markov chain

D EFINITION (M ARKOV CHAIN )

A stochastic process {Xt } is said to be a Markov chain if

P {Xt+1 = j|X0 = k0 , X1 = k1 , . . . , Xt−1 = kt−1 , Xt = i} = P {Xt+1 = j|Xt = i}

for t = 0, 1, . . . and every sequence i, j, k0 , . . . , kt .

That is: the conditional probability of Xt+1 depends only on Xt .

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 4 / 25

Markov chain

D EFINITION (M ARKOV CHAIN )

A stochastic process {Xt } is said to be a Markov chain if

P {Xt+1 = j|X0 = k0 , X1 = k1 , . . . , Xt−1 = kt−1 , Xt = i} = P {Xt+1 = j|Xt = i}

for t = 0, 1, . . . and every sequence i, j, k0 , . . . , kt .

That is: the conditional probability of Xt+1 depends only on Xt .

D EFINITION (T RANSITION PROBABILITES )

We call
P {Xt+1 = j|Xt = i}
the transition probability that we move from state i at time t to the state j at
time t + 1.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 4 / 25

Definitions

D EFINITION (S TATIONARY TRANSITION PROBABILITIES )

If for each i and j

P {Xt+1 = j|Xt = i} = P {X1 = j|X0 = i}

for t = 0, 1, . . . and every sequence i, j, k0 , . . . , kt , then the transition

probabilities are stationary.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 5 / 25

Definitions

D EFINITION (S TATIONARY TRANSITION PROBABILITIES )

If for each i and j

P {Xt+1 = j|Xt = i} = P {X1 = j|X0 = i}

for t = 0, 1, . . . and every sequence i, j, k0 , . . . , kt , then the transition

probabilities are stationary.

D EFINITION (n- STEP TRANSITION PROBABILITES )

We have

pij =P {Xt+1 = j|Xt = i} (1)

(n)
pij =P {Xt+n = j|Xt = i} (2)

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 5 / 25

Properties of transition probabilites

P ROPERTIES

(n)
pij ≥ 0, ∀i, j, n (3)
M
(n)
X
pij = 1, ∀i, n (4)
j=0

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 6 / 25

Properties of transition probabilites

P ROPERTIES

(n)
pij ≥ 0, ∀i, j, n (3)
M
(n)
X
pij = 1, ∀i, n (4)
j=0

Stationary transition probabilities can be collected in a matrix

0 1 ··· M
(n) (n) (n)
0 p00 p01 ··· p0M
 

1  p(n) 0 (n)
p11 ···
(n)
p1M 
 10
P(n) = . 

..  .. .. .. .. 
. . . . 
(n) (n) (n)
M pM 0 pM 1 ··· pM M

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 6 / 25

Gambling

E XAMPLE
Suppose that a player has 1 and with each play of the game wins 1 with
probability p > 0 or loses 1 with probability (1 − p). The game ends when the
player either accumulates 3 or goes broke. This game is a Markov chain with
the states representing the player’s current holding of money, that is, 0, 1, 2, or
3, and with the transition matrix given by

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 7 / 25

Gambling

E XAMPLE
Suppose that a player has 1 and with each play of the game wins 1 with
probability p > 0 or loses 1 with probability (1 − p). The game ends when the
player either accumulates 3 or goes broke. This game is a Markov chain with
the states representing the player’s current holding of money, that is, 0, 1, 2, or
3, and with the transition matrix given by

0 1 2 3
 
0 1 0 0 0
11 − p 0 p 0
P(1) =P=  
2 0 1−p 0 p
3 0 0 0 1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 7 / 25

Cameras

E XAMPLE
0 1 2 3
 
0 p00 p01 p02 p03
1 p10 p11 p12 p13 
P=  
2  p20 p21 p22 p23 
3 p30 p31 p32 p33

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 8 / 25

Cameras

E XAMPLE
0 1 2 3
 
0 p00 p01 p02 p03
1 p10 p11 p12 p13 
P=  
2  p20 p21 p22 p23 
3 p30 p31 p32 p33

The demand Dt+1 follows a Poisson distribution with mean 1,

(1)n e−1
P {Dt+1 = n} =
n!

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 8 / 25

Cameras

E XAMPLE
0 1 2 3
 
0 p00 p01 p02 p03
1 p10 p11 p12 p13 
P=  
2  p20 p21 p22 p23 
3 p30 p31 p32 p33

The demand Dt+1 follows a Poisson distribution with mean 1,

(1)n e−1
P {Dt+1 = n} =
n!
which yields

P {Dt+1 = 0} = 0.368
P {Dt+1 = 1} = 0.368
P {Dt+1 = 2} = 0.184
P {Dt+1 ≥ 3} = 1 − P {Dt+1 ≤ 2} = 1 − (0.368 + 0.368 + 0.184) = 0.080
M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 8 / 25
Cameras (cont.)

E XAMPLE
Remember the reorder policy,
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1

Then, in the first row we have

p03

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 9 / 25

Cameras (cont.)

E XAMPLE
Remember the reorder policy,
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1

Then, in the first row we have

p03 = P {Dt+1 = 0} = 0.368

p02

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 9 / 25

Cameras (cont.)

E XAMPLE
Remember the reorder policy,
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1

Then, in the first row we have

p03 = P {Dt+1 = 0} = 0.368

p02 = P {Dt+1 = 1} = 0.368
p01

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 9 / 25

Cameras (cont.)

E XAMPLE
Remember the reorder policy,
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1

Then, in the first row we have

p03 = P {Dt+1 = 0} = 0.368

p02 = P {Dt+1 = 1} = 0.368
p01 = P {Dt+1 = 2} = 0.184
p00

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 9 / 25

Cameras (cont.)

E XAMPLE
Remember the reorder policy,
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1

Then, in the first row we have

p03 = P {Dt+1 = 0} = 0.368



p = P {D

02 = 1} = 0.368
t+1
p01 = P {Dt+1 = 2} = 0.184

p00 = P {Dt+1 ≥ 3} = 0.080


M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 9 / 25

Cameras (cont.)

E XAMPLE
Remember the reorder policy,
(
max{3 − Dt+1 , 0} if Xt = 0
Xt+1 =
max{Xt − Dt+1 , 0} if Xt ≥ 1

Then, in the first row we have

p03 = P {Dt+1 = 0} = 0.368

  
 0.080 0.184 0.368 0.368
p = P {D

02 = 1} = 0.368
t+1  p10 p11 p12 p13 
=⇒ P = 
 p20

p01 = P {Dt+1 = 2} = 0.184 p21 p22 p23 
p30 p31 p32 p33

p00 = P {Dt+1 ≥ 3} = 0.080


M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 9 / 25

Cameras (cont.)

E XAMPLE
Similarly

p12 = p13 = p23 = 0 


p11 = P {Dt+1 = 0} = 0.368



p10 = P {Dt+1 ≥ 1} = 0.632

p22 = P {Dt+1 = 0} = 0.368




p20 = P {Dt+1 ≥ 2} = 0.264


M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 10 / 25

Cameras (cont.)

E XAMPLE
Similarly

p12 = p13 = p23 = 0 
  
0.080 0.184 0.368 0.368

p11 = P {Dt+1 = 0} = 0.368


 0.632 0.368 0 0 
p10 = P {Dt+1 ≥ 1} = 0.632 =⇒ P =  
  0.264 0.368 0.368 0 
p22 = P {Dt+1 = 0} = 0.368

0.080 0.184 0.368 0.368



p20 = P {Dt+1 ≥ 2} = 0.264


M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 10 / 25

Graphical representation of P

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 11 / 25

Example

Q UESTION
Can we find P(2) ?
Consider a Markov chain with states 0, 1, . . . , M . What is the probability that
from 1 at time t we will end up at 2 at state t + 2?

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 12 / 25

Example

Q UESTION
Can we find P(2) ?
Consider a Markov chain with states 0, 1, . . . , M . What is the probability that
from 1 at time t we will end up at 2 at state t + 2?
t t+1 t+2

0 0 0

1 1 1

2 2 2

.. .. ..
. . .

M M M

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 12 / 25

Example

So if we start from 1 the probability of being in 2 in two steps is

M
(2)
X
p12 = (p10 · p02 ) + (p11 · p12 ) + · · · + (p1M · pM 2 ) = p1k pk2
| {z } | {z } | {z }
via 0 via 1 via M k=0

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 13 / 25

Example

So if we start from 1 the probability of being in 2 in two steps is

M
(2)
X
p12 = (p10 · p02 ) + (p11 · p12 ) + · · · + (p1M · pM 2 ) = p1k pk2
| {z } | {z } | {z }
via 0 via 1 via M k=0

In general
M
(2)
X
pij = (pi0 · p0j ) + (pi1 · p1j ) + · · · + (piM · pM j ) = pik pkj .
k=0
| {z }
matr. multipl.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 13 / 25

Example

So if we start from 1 the probability of being in 2 in two steps is

M
(2)
X
p12 = (p10 · p02 ) + (p11 · p12 ) + · · · + (p1M · pM 2 ) = p1k pk2
| {z } | {z } | {z }
via 0 via 1 via M k=0

In general
M
(2)
X
pij = (pi0 · p0j ) + (pi1 · p1j ) + · · · + (piM · pM j ) = pik pkj .
k=0
| {z }
matr. multipl.

Therefore,
P(2) = PP

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 13 / 25

Example

So if we start from 1 the probability of being in 2 in three steps is

M X
M
(3)
X
p12 = (p10 · p03 · ·p32 ) + · · · + (p1M · pM 0 · p02 ) = p1k1 pk1 k2 pk2 2
| {z } | {z }
via 0 − 3 via M − 0 k1 =0 k2 =0

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 14 / 25

Example

So if we start from 1 the probability of being in 2 in three steps is

M X
M
(3)
X
p12 = (p10 · p03 · ·p32 ) + · · · + (p1M · pM 0 · p02 ) = p1k1 pk1 k2 pk2 2
| {z } | {z }
via 0 − 3 via M − 0 k1 =0 k2 =0

In general
M X
M
(3)
X
pij = pik1 pk1 k2 pk2 j
k1 =0 k2 =0
| {z }
matrix multiplication

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 14 / 25

Example

So if we start from 1 the probability of being in 2 in three steps is

M X
M
(3)
X
p12 = (p10 · p03 · ·p32 ) + · · · + (p1M · pM 0 · p02 ) = p1k1 pk1 k2 pk2 2
| {z } | {z }
via 0 − 3 via M − 0 k1 =0 k2 =0

In general
M X
M
(3)
X
pij = pik1 pk1 k2 pk2 j
k1 =0 k2 =0
| {z }
matrix multiplication

Therefore,
P(3) = PPP = P(2) P

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 14 / 25

Example

So if we start from 1 the probability of being in 2 in three steps is

M X
M
(3)
X
p12 = (p10 · p03 · ·p32 ) + · · · + (p1M · pM 0 · p02 ) = p1k1 pk1 k2 pk2 2
| {z } | {z }
via 0 − 3 via M − 0 k1 =0 k2 =0

In general
M X
M
(3)
X
pij = pik1 pk1 k2 pk2 j
k1 =0 k2 =0
| {z }
matrix multiplication

Therefore,
P(3) = PPP = P(2) P
In the case of the general n

P(n) = |P ·{z
· · P}
n times

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 14 / 25

Kolmogorov-Chapman equations

We know that we can always rewrite

P(n) = P · · P} = |P ·{z
| ·{z · · P} = P(m) P(n−m)
· · P} |P ·{z
n times m times (n−m) times

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 15 / 25

Kolmogorov-Chapman equations

We know that we can always rewrite

P(n) = P · · P} = |P ·{z
| ·{z · · P} = P(m) P(n−m)
· · P} |P ·{z
n times m times (n−m) times

From which we deduce that the Kolmogorov-Chapman equations

M
(n) (m) (n−m)
X
pij = pik pkj
k=0

must hold for all i, j = 0, 1, . . . , M , m = 1, . . . , n − 1, n = m + 1, m + 2, . . .

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 15 / 25

States

D EFINITION (ACCESSIBILITY )
(n)
State j is accessible from i if there is at least one pij > 0 for some n

E XAMPLE (G AMBLING EXAMPLE )

I can 0 from 2, but I cannot access 2 from 0.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 16 / 25

States

D EFINITION (ACCESSIBILITY )
(n)
State j is accessible from i if there is at least one pij > 0 for some n

E XAMPLE (G AMBLING EXAMPLE )

I can 0 from 2, but I cannot access 2 from 0.

D EFINITION (C OMMUNICATING STATES )

States i and j communicate if I can access i from j and j from i.

E XAMPLE (G AMBLING EXAMPLE )

2 and 1 communicate, 2 and 0 do not.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 16 / 25

States

D EFINITION (ACCESSIBILITY )
(n)
State j is accessible from i if there is at least one pij > 0 for some n

E XAMPLE (G AMBLING EXAMPLE )

I can 0 from 2, but I cannot access 2 from 0.

D EFINITION (C OMMUNICATING STATES )

States i and j communicate if I can access i from j and j from i.

E XAMPLE (G AMBLING EXAMPLE )

2 and 1 communicate, 2 and 0 do not.

D EFINITION (I RREDUCIBILITY )
A Markov chain where all the states communicate with each other is called
irreducible.
M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 16 / 25
Recurrent and transient states

D EFINITION (T RANSIENT STATE )

A state is said to be a transient state if, upon entering this state, the process
may never return to this state again. State i is transient if and only if there
exists a state j (j 6= i) that is accessible from state i but not vice versa, that is,
state i is not accessible from state j.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 17 / 25

Recurrent and transient states

D EFINITION (T RANSIENT STATE )

A state is said to be a transient state if, upon entering this state, the process
may never return to this state again. State i is transient if and only if there
exists a state j (j 6= i) that is accessible from state i but not vice versa, that is,
state i is not accessible from state j.

D EFINITION (R ECURRENT STATE )

A state is said to be a recurrent state if, upon entering this state, the process
definitely will return to this state again. Therefore, a state is recurrent if and
only if it is not transient.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 17 / 25

Recurrent and transient states

D EFINITION (T RANSIENT STATE )

A state is said to be a transient state if, upon entering this state, the process
may never return to this state again. State i is transient if and only if there
exists a state j (j 6= i) that is accessible from state i but not vice versa, that is,
state i is not accessible from state j.

D EFINITION (R ECURRENT STATE )

A state is said to be a recurrent state if, upon entering this state, the process
definitely will return to this state again. Therefore, a state is recurrent if and
only if it is not transient.

D EFINITION (A BSORBING STATE )

A state is said to be an absorbing state if, upon entering this state, the
process never will leave this state again. Equivalently, state i is an absorbing
state if and only if pii = 1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 17 / 25

Convergence

D EFINITION (P ERIODICITY )
The period of state i is defined to be the integer t (t > 1) such that pnii = 0 for
all values of n other than t, 2t, 3t, . . . and t is the largest integer with this
property. If a state has a period of 1, then it is said to be aperiodic.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 18 / 25

Convergence

D EFINITION (P ERIODICITY )
The period of state i is defined to be the integer t (t > 1) such that pnii = 0 for
all values of n other than t, 2t, 3t, . . . and t is the largest integer with this
property. If a state has a period of 1, then it is said to be aperiodic.

D EFINITION (E RGODIC M ARKOV CHAIN )

In a finite-state Markov chain, recurrent states that are aperiodic are called
ergodic states. A Markov chain is said to be ergodic if all its states are
ergodic states.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 18 / 25

Periodicity

Consider the matrix P from the problem with cameras. We have

 
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
P(8) ≈ 
0.286 0.285 0.264 0.166


0.286 0.285 0.264 0.166

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 19 / 25

Periodicity

Consider the matrix P from the problem with cameras. We have

 
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
P(8) ≈ 
0.286 0.285 0.264 0.166


0.286 0.285 0.264 0.166

R EFLECTIONS
After a large enough number of steps the probability of being in j does
not depend on the starting point i

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 19 / 25

Periodicity

Consider the matrix P from the problem with cameras. We have

 
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
P(8) ≈ 
0.286 0.285 0.264 0.166


0.286 0.285 0.264 0.166

R EFLECTIONS
After a large enough number of steps the probability of being in j does
not depend on the starting point i
We have some ‘steady state’ probabilities. In the long run the probability
of having 2 cameras in the storage will be

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 19 / 25

Periodicity

Consider the matrix P from the problem with cameras. We have

 
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
P(8) ≈ 
0.286 0.285 0.264 0.166


0.286 0.285 0.264 0.166

R EFLECTIONS
After a large enough number of steps the probability of being in j does
not depend on the starting point i
We have some ‘steady state’ probabilities. In the long run the probability
of having 2 cameras in the storage will be 0.264.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 19 / 25

Periodicity

Consider the matrix P from the problem with cameras. We have

 
0.286 0.285 0.264 0.166
0.286 0.285 0.264 0.166
P(8) ≈ 
0.286 0.285 0.264 0.166


0.286 0.285 0.264 0.166

R EFLECTIONS
After a large enough number of steps the probability of being in j does
not depend on the starting point i
We have some ‘steady state’ probabilities. In the long run the probability
of having 2 cameras in the storage will be 0.264.
When can we play this trick?

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 19 / 25

Existence of steady state probabilities

T HEOREM (S TEADY STATE PROBABILITIES )

(n)
For any irreducible ergodic Markov chain, limn→∞ pij exists and is
independent of i.

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 20 / 25

Existence of steady state probabilities

T HEOREM (S TEADY STATE PROBABILITIES )

(n)
For any irreducible ergodic Markov chain, limn→∞ pij exists and is
independent of i. Furthermore,
(n)
lim p = πj > 0
n→∞ ij

where πj is the unique solution of the steady-state equations

( PM
πj = i=0 πi pij for j = 0, 1, . . . , M.
PM
i=0 πj = 1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 20 / 25

Cameras (cont.)

E XAMPLE
For the example of cameras, with
 
0.080 0.184 0.368 0.368
0.632 0.368 0 0 
 
0.264 0.368 0.368 0 
0.080 0.184 0.368 0.368

we shall solve



π0 = 0.080π0 + 0.632π1 + 0.264π2 + 0.080π3
 1 = 0.184π0 + 0.368π1 + 0.368π2 + 0.184π3
π



π2 = 0.368π0 + 0.632π2 + 0.368π3

π 3 = 0.368π0 + 0.368π3





π + π + π + π = 1
0 1 2 3

and obtain
π0 = 0.286 π1 = 0.285 π2 = 0.264 π3 = 0.166
M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 21 / 25
Cameras (cont.)

Steady state probabilities do not always exist if we relax the assumption of

aperiodicity
E XAMPLE
Consider  
0 1
P=
1 0

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 22 / 25

Cameras (cont.)

Steady state probabilities do not always exist if we relax the assumption of

aperiodicity
E XAMPLE
Consider    
0 1 1 0
P= P(2) =
1 0 0 1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 22 / 25

Cameras (cont.)

Steady state probabilities do not always exist if we relax the assumption of

aperiodicity
E XAMPLE
Consider      
0 1 1 0 0 1
P= P(2) = P(3) =
1 0 0 1 1 0

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 22 / 25

Cameras (cont.)

Steady state probabilities do not always exist if we relax the assumption of

aperiodicity
E XAMPLE
Consider      
0 1 1 0 0 1
P= P(2) = P(3) =
1 0 0 1 1 0

However, the following limit always exists for an irreducible (finite-state)

Markov chain !
n
1 X (k)
lim pij = πj > 0
n→∞ n
k=1

E XAMPLE
n
!
1 X (k)
For P as above, we have that lim pij = 0.5
n→∞ n
k=1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 22 / 25

Cameras (cont.)

Steady state probabilities do not always exist if we relax the assumption of

aperiodicity
E XAMPLE
Consider      
0 1 1 0 0 1
P= P(2) = P(3) =
1 0 0 1 1 0

However, the following limit always exists for an irreducible (finite-state)

Markov chain !
n
1 X (k)
lim pij = πj > 0
n→∞ n
k=1

E XAMPLE
n
!
1 X (k)
For P as above, we have that lim pij = 0.5
n→∞ n
k=1

Alright, but how can we use and interpret this result?

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 22 / 25
Average cost

Consider the costs of being at different states C(0), . . . , C(M ). The expected
average cost for the first n periods is
" n #
1X
E C(Xt )
n t=1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 23 / 25

Average cost

Consider the costs of being at different states C(0), . . . , C(M ). The expected
average cost for the first n periods is
" n #
1X
E C(Xt )
n t=1

What is the expected average cost in the long run?

" n #
1X
C̄ = lim E C(Xt )
n→∞ n t=1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 23 / 25

Average cost

Consider the costs of being at different states C(0), . . . , C(M ). The expected
average cost for the first n periods is
" n #
1X
E C(Xt )
n t=1

What is the expected average cost in the long run?

" n #
1X
C̄ = lim E C(Xt )
n→∞ n t=1
When n → ∞, it tends to become

C̄ = E [π0 C(0) + · · · + πM C(M )]

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 23 / 25

Average cost

Consider the costs of being at different states C(0), . . . , C(M ). The expected
average cost for the first n periods is
" n #
1X
E C(Xt )
n t=1

What is the expected average cost in the long run?

" n #
1X
C̄ = lim E C(Xt )
n→∞ n t=1
When n → ∞, it tends to become

C̄ = E [π0 C(0) + · · · + πM C(M )]

but what is inside E[·] is a constant, therefore

C̄ = π0 C(0) + · · · + πM C(M )

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 23 / 25

Cameras (cont.)

E XAMPLE
Let C(xt ) be the yearly cost of having xt cameras in the inventory


 0 if xt =0

2 if xt =1
C(xt ) =


 8 if xt =2
18 if xt =3


M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 24 / 25

Cameras (cont.)

E XAMPLE
Let C(xt ) be the yearly cost of having xt cameras in the inventory


 0 if xt =0

2 if xt =1
C(xt ) =


 8 if xt =2
18 if xt =3


Then we have,

| {z } ·0) + (0.285
C̄ = (0.286 | {z } ·2) + (0.263
| {z } ·8) + (0.166
| {z } ·18) = 5.662
π0 π1 π2 π3

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 24 / 25

Gambling example

Reprise the example of gambling with p = 0.7.

E XAMPLE
   
1 0 0 0 1 0 0 0
0.3 0 0.7 0.0 0.379 0 0.006 0.6145
P=
0
 P(7) = 
0.3 0 0.7 0.113 0.003 0 0.884 
0 0 0 1 0 0 0 1

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 25 / 25

Gambling example

Reprise the example of gambling with p = 0.7.

E XAMPLE
   
1 0 0 0 1 0 0 0
0.3 0 0.7 0.0 0.379 0 0.006 0.6145
P=
0
 P(7) = 
0.3 0 0.7 0.113 0.003 0 0.884 
0 0 0 1 0 0 0 1

One observation:
Object of our observation is P
Next lecture:
First passage
Probability of absorption
Exercises

M. B RUNELLI M ARKOV CHAINS A PRIL 17, 2018 25 / 25