Lecture 2: Review of Markov

Chain (Part 2)
MH4702: Probabilistic Methods in
Operations Research

Lecturer: Yan Zhenzhen

Office: SPMS-MAS-05-19
Email: yanzz@ntu.edu.sg
Tel: 6513 7466

Chapter 17 of Hillier & Lieberman, 10th Ed

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

 Continuous Time Markov Chains

Continuous time,
Continuous value Process

Discrete time,  Let represent the number of

Continuous value process customers in a waiting line at time .
 In continuous time Markov Chain,
when a state is visited, the process
Continuous time,
stays in that state for an exponentially
Discrete value process
distributed length of time before
moving to a new state.
Discrete Time,
Discrete value process

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

2
 Continuous Time Markov Chains

is a past time
is the current time
is time units into the future

Given the state of the system has been observed at and

and

To seek the probability distribution of the state of the system at time :

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

3
 Continuous Time Markov Chains

A continuous time stochastic process has the Markovian

property if

for all and for all

Note that is a transition probability.

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

4
 Stationary Continuous Time Markov Chains

If the transition probabilities are independent of , that is,

they are called stationary transition probabilities.

To simplify notation, is a function of time .

where is refereed to as the continuous time transition probability function.

Assume that

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

5
 Irreducibility of Continuous Time Markov Chains

Chapman-Kolmogorov equations — for any states and and nonnegative numbers

and ,

A pair of states and are said to communicate if there are times and such that
()and.

If the Markov chain is irreducible, then

for some and all states and .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

6
 Steady-state Probability of Continuous Time Markov Chains

Define the steady-state probabilities (or stationary probabilities) of a Markov chain as

Then, always exists for an irreducible Markov chain and is independent of its initial state.

The satisfy the equations

𝑀 𝑴
𝜋 𝑗=∑ 𝜋 𝑖 𝑝𝑖 𝑗 (𝑡), for
𝑖=0
and every with ∑ 𝝅 𝒋=𝟏
𝒋=𝟎

Or, for , where .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

7
 The Second Way of describing a Continuous Time Markov Chains

A key random variable — Each time the process enters state , the amount of time it spends in that state
before moving to a different state is a random variable where

is exponential distributed.

Transition from state 10 to state 11

Transition from state 11 to state 10

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

8
 The Second Way of describing a Continuous Time Markov Chains (cont.)

Since the amount of time until a transition occurs is exponentially

distributed,
, for

Therefore,
0

It implies that the probability of two or more state transitions to

be made simultaneously is zero.

𝑖𝑗 {
( 𝑡 ) = ¿ 1 if 𝑖 = 𝑗 ¿
0 if 𝑖 ≠ 𝑗
Only one step at a time.

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

9
The Second Way of describing a Continuous Time Markov Chains (cont.)

An equivalent way of describing a continuous time Markov chain:

 The random variable has an exponential distribution with a mean of 1

 When leaving state , the process moves to a state with probability , where the satisfy the
conditions
is not a function of time .
𝑀

∑ 𝑝𝑖𝑗 =1 The matrix is a transition matrix

with zeros at diagonal.
𝑗 =0

 The next state visited after state is independent of the time spent in state .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

10
The Second Way of describing a Continuous Time Markov Chains (cont.)

Each time the process enters state the amount of time it will spend in state before a transition to state
occurs (if a transition to some other state does not occur first) is a random variable

Each has an exponential distribution with parameter

The time spent in state until a transition occurs is the minimum (over ) of the .

What is the distribution of ?

𝑗≠𝑖

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

11
Transition Rate

MH4702: Probabilistic Methods

in Operations Research

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

 Alarm Clocks and Transition Rates

Image at each state , there are independent alarm

clocks ( of them) associated with each of the states
that the process can visit after .
If can visit , then the alarm clock associated with (,)
will ring after an exponentially distributed length of
time with parameter .
When the process first arrives at state , all the clocks
start simultaneously.
The first alarm that rings determine the next state to
visit.
If the process moves to , a new set of clocks will be
started, and so on.

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

13
 Transition Intensity (Rate)

Because the chain is stationary and memoryless, you only need to study the instantaneous transition,
which is the rate of change.

Intensity of transitioning out of state , is the expected number of transition out of state in unit time,
in a small amount of time .

For small time interval , the number of transition is either 1 or 0, hence the expected number is the
same as the probability of transition out of state

Note that , is the probability of remaining in state , in the interval .

Hence, intensity of transitioning out of state is .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

14
 Transition Intensity (Rate) (cont.)

Now,
,

since is the amount of time the chain spends in state, has an exponential distribution
with parameter and

Therefore, intensity of transitioning out of state is,

.

Note that is the derivative of , which is the derivative of .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

15
 Transition Intensity (Rate) (cont.)
From to

Note that for , , is the probability of transitioning to state from , in the interval .

Hence, intensity of transitioning from to is,

Therefore, intensity of transitioning out of state to state is,

.

Note that is the derivative of .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

16
Transition Matrix

MH4702: Probabilistic Methods

in Operations Research

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

 Intensities and Transition Matrix
Remember the matrix is a transition matrix with zeros at diagonal.

Now, for , is the probability that the chain transition to state when leaving state That is, at any time ,

𝒒 𝒊𝒋 =𝒒 𝒊 𝒑 𝒊𝒋

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

18
 Intensities and Transition Matrix (cont.)

 is the parameter of the exponential distribution for

 is the parameter of the exponential distribution for

- When the transition occurs, the probability that it is to

state is

𝑞𝑖 =
1 ,
𝐸 [𝑇𝑖 ]
𝑞𝑖 𝑗=
1 , 𝑞= 𝑞
𝑖 ∑
𝑖𝑗
𝐸 [𝑇 𝑖 𝑗 ] 𝑗≠ 𝑖

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

19
 The Generator  Matrix

For modelling purpose, you usually start with identifying the transition rates, , instead of the
transition function
The purpose is to find the stationary probabilities.

Let

From the earlier discussion, you know that is related to the derivative of .

Recall that is the derivative of and that is the derivative of .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

20
 The Generator  Matrix

The transition matrix is

[ ]
𝑝 00 (𝑡 ) 𝑝 01 (𝑡 ) 𝑝 02 (𝑡 ) … … 𝑝 0 𝑀 (𝑡 )
𝑝1 0 (𝑡 ) 𝑝 11 (𝑡 ) 𝑝 12 (𝑡 ) ⋯ ⋯ … 𝑝1 𝑀 ( 𝑡 ) …
𝑷 (𝑡 )= 𝑝 20 (𝑡 ) 𝑝2 1 (𝑡 ) 𝑝 22 (𝑡 ) ⋯ ⋯ 𝑝 2 𝑀 (𝑡 )
¿
⋮ ⋮ ⋮ ⋱ ¿⋮
⋮ ⋮ ⋮ ⋱ ⋮ 𝑝 𝑀𝑀 (𝑡 )
𝑝 𝑀 0 (𝑡 ) 𝑝 𝑀 1 ( 𝑡 ) 𝑝 𝑀 2(𝑡 ) … ¿

[ ]
−𝑞 0 𝑞0 1 𝑞 02 … … 𝑞0 𝑀
𝑞1 0 − 𝑞1 𝑞1 2 ⋯ ⋯ 𝑞1 𝑀
𝑸= 𝑞20 𝑞2 1 − 𝑞2 ⋯ ⋯ 𝑞2 𝑀
¿
⋮ ⋮ ⋮ ⋱ ¿⋮
⋮ ⋮ ⋮ ⋱ ⋮ −𝑞 𝑀
𝑞𝑀 0 𝑞𝑀 1 𝑞𝑀 2 … ¿

Note that is the derivative of or .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

21
 Stationary Probability of Continuous Time Markov Chains (cont.)

R for every, where .

How to solve ?

Take the derivative of both sides at you will get

𝟎= 𝝅 𝑷 ′ ( 0 )=𝝅 𝑸❑

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

22
 Example: Three-State Continuous-Time Weather Markov Chain

Consider a weather chain with state space .

Assume that
Rainfall lasts, on an average, 3 hours at a time. 𝑞𝑟 =1 /3
When it snows, the duration, on an average is 6 hours. 𝑞 𝑠=1 /6
The weather stays clear, on an average, for 12 hours. 𝑞𝑐 =1/ 12

Furthermore, changes in weather states are described by the (embedded) stochastic

transition matrix

What is the generator matrix, ? What is the stationary distribution?

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

23
 Example: Three-State Continuous-Time Weather Markov Chain (cont.)

The stationary distribution satisfies :

( )
−1/3 1/6 1/6
𝑸= 𝑅𝑎𝑖𝑛 𝑆𝑛𝑜𝑤 𝐶𝑙𝑒𝑎𝑟 3/24 −1/6 1/24 ¿
¿
1/48 3/48 −1/12
with
𝑞𝑖𝑗 =𝑞 𝑖 𝑝𝑖𝑗
Hence, .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

24
 Another Interpretation of Stationary Probability
The steady-state equations are:

𝜋 𝑗 𝑞 𝑗 =∑ 𝜋 𝑗 𝑞 𝑖 𝑗 , for 𝑗=0,1 ,…, 𝑀

𝑖≠ 𝑗
The rate at which the process enters state
from state .
The rate at which the
The rate at which the process enters
process leaves state .
state from any other state.

The rate at which the process leaves

state must equal the rat at which the
process enters state.

Rate In = Rate Out Principle

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

25
 Example: Machine Repair

A shop has two identical machines operating

continuously except when they are broken
down. A maintenance person has to repair
them whenever needed.

 Broken down — The time until the next

breakdown of a given machine has an
exponential distribution with a mean of
one day.

 Repair — The time required to repair a

machine has an exponential distribution
with a mean of one half day.

= Number of machines broken down at time

is a CTMC with state 0,1,2.

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

26
 Example: Machine Repair (cont.)
𝑞0 1 =2 𝑞12 =1 1. Since both breakdowns and repairs occur one at a time:

State: 0 1 2

𝑞10 =2 𝑞2 1=2
The rate diagram 2. Since the expected repair time is day:

( )
? 2 0
𝑸= 0 1 2
2 ? 1¿
¿
0 2 ?

3. Since the expected time until a particular operational

machine breaks down is 1 day:

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

27
 Example: Machine Repair (cont.)
𝑞0 1 =2 𝑞12 =1 1. The total transition rate out each state:

( )
0 1 2
−2 2 0
𝑸=0 1 2 2 −3 1 ¿
State:

𝑞10 =2 𝑞2 1=2 ¿
0 2 −2
The rate diagram
2. The steady-state equations:

3. The steady-state distribution:

In the long run, both machines will be broken down

simultaneously 20 percent of time and one machine will be
broken down another 40 percent of the time.
© 2020 Nanyang Technological University, Singapore. All Rights Reserved.
28
 Transition Function and Generator
Recall: Chapman-Kolmogorov equations — for any states and and time and

Kolmogorov forward and backward equations

A continuous-time Markov chain with transition function and generator satisfies the forward equation: and the
backward equation:

Equivalently, for all states and ,

And

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

29
 Example: Two-State Chain

𝜆 ~
𝑷=
0
1 ( 1
0 )  A two-state continuous-time Markov
chain is specified by two holding time
parameters as depicted in the transition
1 2 graph given here.
𝑸=( −𝜇𝜆 𝜆
−𝜇 )  The process stays in state 1 for an
𝜇 exponential length of time with
parameter λ before moving to 2. It stays
in 2 for an exponential length of time
with parameter before moving to 1 and
so on.

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

30
 Example: Two-State Chain (cont.)

The forward equations give

~
𝑷=
0
1 ( 1
0 )

𝑸=
( −𝜆
𝜇
𝜆
−𝜇 )
The solution to the linear differential equation is

Similarly,

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

31
 Example: Two-State Chain (cont.)

Hence, the transition function is

~
(
𝑷= 0
1
1
0 ) 𝑸=
( −𝜆
𝜇
𝜆
−𝜇 )

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

32
Summary

MH4702: Probabilistic Methods

in Operations Research

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

 Summary
Here are the key takeaways from the lesson.

Description of a continuous time Markov Chain:

{ 𝑋 ( 𝑡′ ) : 𝑡 ′ ≥ 0   }
𝑝 𝑖𝑗 ( 𝑡 ) =𝑃 { 𝑋 ( 𝑡 ) = 𝑗∨ 𝑋 ( 0 )=i   }
where is referred to as the continuous time transition probability
function. Assume:

lim 𝑝𝑖𝑗 (𝑡 )=
𝑡 →0 {1 if 𝑖= 𝑗
0 if 𝑖 ≠ 𝑗

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

34
 Summary (cont.)
Here are the key takeaways from the lesson.

An equivalent way of describing a continuous time Markov chain:

 The random variable has an exponential distribution with a
mean of
 When leaving state , the process moves to a state with
probability where the satisfy the condition:
for all
𝑀

∑ 𝑝𝑖𝑗 =1 for all

𝑗=0
 The next state visited after state is independent of the time
spent in state .

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

35
 Summary (cont.)
Here are the key takeaways from the lesson.

Another way of describing a continuous time Markov chain:

 Transition Intensity (Rate) :
 The intensity of transitioning out of state

- where is the amount of time the chain spends in

state

 The intensity of transitioning out of state to state =

- where is the amount of time the chain spends in

state before transition to state

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

36
 Summary (cont.)
Here are the key takeaways from the lesson.

Transition Intensity (Rate) is:

 It helps to calculate stationary probabilities of a Markov Chain

 It helps to calculate probability transition function

Kolmogorov forward equation
Kolmogorov backward equation

© 2020 Nanyang Technological University, Singapore. All Rights Reserved.

37