Newsvendor model deals with perishable products. You have to get rid of leftover after each decision epoch. In this chapter, we talk about non-perishable products.
Motivation Example
Consider an inventory model: • Dn: the demand of the nth period (e.g. days or weeks). Demand is Random and its the distribution is known. • Inventory that is left at the end of this period can be used to satisfy the demand in the next period (e.g. paper, pencils, tires, ink etc.) Suppose the Demand {Dn} is an i.i.d. sequence and it follows the following distribution: d 0 1 2 3
P[Dn = d] 1/8 1/4 1/2 1/8
Assume Cp = $100,Cv = $50,Cf = $100,h = $10, where h is the holding cost, i.e., h = −Cs. Question: What is the optimal inventory policy to maximize the long-run average proﬁt per week?
Please don’t be confused this with the newsvendor problem. For newsvendor, you have to get rid of items at the end of each period. In that case, time 0: make order decision; time 1: demand is realized and any leftover is removed. Now we consider the case where the leftover can be used for the following period.
How to analyze the performance of an inventory policy for this case? Every Friday 5 pm, let’s say we decide how much to order for the following week so that the ordered items will arrive at 8 am the following Monday. We sell from Monday to Friday. Let’s again consider (s,S) policy.
Let Xn be the inventory level at the end of period n. (i) If Xn < s, order S −Xn items (ii) Otherwise, do not order.
The goal of our class is again to determine the (s,S) policy so that our long-run average proﬁt can be maximized.
Let’s do some analysis ﬁrst on Xn.
Deﬁnition 1. Time series: A sequence of data points over time. Time can be discrete or continuous. Hence, {Xn}n≥0 is a time series. Assume our policy is (s,S) = (2,4). Then (i) If Xn ≤ 1, order 4−Xn items (ii) Otherwise, do not order.
Let’s consider the following probability:
P(Xn+1 = 3|Xn = 1)
• Xn = 1: the nth week ends with 1 item. • Xn+1 = 3: the (n + 1)th week ends with 3 items. • P(Xn+1 = 3|Xn = 1): given the inventory level of nth week is 1, what is the probability that (n+1)th week ends with 3 items?
I want to remind you this is a conditional probability. Please go back to your 2027, ﬁrst week lectures and homework 1 to review basic concepts. Especially the Bayes’ rule.
How to calculate this probability ?
Xn = 1 means the nth week ends with 1 item. Then according to (s,S) policy, we order 4−1 = 3 items. So next Monday, we start with 1 + 3 = 4 items that can sell. Since we want to have 3 items at the end of (n+1)th week, it simply means the demand of (n+1)th week is 1. What is the probability that the demand of (n + 1)th week is 1? We know P(Dn+1 = 1) = 1/4. Hence,
P(Xn+1 = 3|Xn = 1) = P(Next week ends with 3 items|This week ended with 1 item) = P(Next week ends with 3 items|Next week begins with 4 items) = P(Selling 1 itme during the next week) = P(Dn+1 = 1) = 1/4
Can you calculate the following probabilities using the same idea? • P(Xn+1 = 4|Xn = 2) = 0 • P(Xn+1 = 3|Xn = 2) = 0 • P(Xn+1 = 2|Xn = 2) = 1/8 • P(Xn+1 = 1|Xn = 2) = 1/4 • P(Xn+1 = 0|Xn = 2) = 5/8 Do you know the possible value of Xn+1? What do you observe? State space S = {0,

© All Rights Reserved

30 views

Newsvendor model deals with perishable products. You have to get rid of leftover after each decision epoch. In this chapter, we talk about non-perishable products.
Motivation Example
Consider an inventory model: • Dn: the demand of the nth period (e.g. days or weeks). Demand is Random and its the distribution is known. • Inventory that is left at the end of this period can be used to satisfy the demand in the next period (e.g. paper, pencils, tires, ink etc.) Suppose the Demand {Dn} is an i.i.d. sequence and it follows the following distribution: d 0 1 2 3
P[Dn = d] 1/8 1/4 1/2 1/8
Assume Cp = $100,Cv = $50,Cf = $100,h = $10, where h is the holding cost, i.e., h = −Cs. Question: What is the optimal inventory policy to maximize the long-run average proﬁt per week?
Please don’t be confused this with the newsvendor problem. For newsvendor, you have to get rid of items at the end of each period. In that case, time 0: make order decision; time 1: demand is realized and any leftover is removed. Now we consider the case where the leftover can be used for the following period.
How to analyze the performance of an inventory policy for this case? Every Friday 5 pm, let’s say we decide how much to order for the following week so that the ordered items will arrive at 8 am the following Monday. We sell from Monday to Friday. Let’s again consider (s,S) policy.
Let Xn be the inventory level at the end of period n. (i) If Xn < s, order S −Xn items (ii) Otherwise, do not order.
The goal of our class is again to determine the (s,S) policy so that our long-run average proﬁt can be maximized.
Let’s do some analysis ﬁrst on Xn.
Deﬁnition 1. Time series: A sequence of data points over time. Time can be discrete or continuous. Hence, {Xn}n≥0 is a time series. Assume our policy is (s,S) = (2,4). Then (i) If Xn ≤ 1, order 4−Xn items (ii) Otherwise, do not order.
Let’s consider the following probability:
P(Xn+1 = 3|Xn = 1)
• Xn = 1: the nth week ends with 1 item. • Xn+1 = 3: the (n + 1)th week ends with 3 items. • P(Xn+1 = 3|Xn = 1): given the inventory level of nth week is 1, what is the probability that (n+1)th week ends with 3 items?
I want to remind you this is a conditional probability. Please go back to your 2027, ﬁrst week lectures and homework 1 to review basic concepts. Especially the Bayes’ rule.
How to calculate this probability ?
Xn = 1 means the nth week ends with 1 item. Then according to (s,S) policy, we order 4−1 = 3 items. So next Monday, we start with 1 + 3 = 4 items that can sell. Since we want to have 3 items at the end of (n+1)th week, it simply means the demand of (n+1)th week is 1. What is the probability that the demand of (n + 1)th week is 1? We know P(Dn+1 = 1) = 1/4. Hence,
P(Xn+1 = 3|Xn = 1) = P(Next week ends with 3 items|This week ended with 1 item) = P(Next week ends with 3 items|Next week begins with 4 items) = P(Selling 1 itme during the next week) = P(Dn+1 = 1) = 1/4
Can you calculate the following probabilities using the same idea? • P(Xn+1 = 4|Xn = 2) = 0 • P(Xn+1 = 3|Xn = 2) = 0 • P(Xn+1 = 2|Xn = 2) = 1/8 • P(Xn+1 = 1|Xn = 2) = 1/4 • P(Xn+1 = 0|Xn = 2) = 5/8 Do you know the possible value of Xn+1? What do you observe? State space S = {0,

© All Rights Reserved

- Algorithms of Hidden Markov Model and a Prediction Method on Product Preferences
- HW2Solutions.pdf
- HW1Soltution
- HW3Solutions.pdf
- ECE697AA 08-10-31 Queuing Systems I
- The Need to Clarify the Relationship between Physics and Mathematics in Science Curriculum: Cultural Knowledge as Possible Framework
- HW7Solutions.pdf
- MA6451-Probability and Random Processes QB
- Semantic RichMM
- DLP WEEK 10 DAY 4-1.docx
- Assignment 1
- Pattern and Predictability in Music.pdf
- CONVERGENCE OF ESTIMATED OPTION PRICE IN A REGIME SWITCHING MARKET
- Year 5 Parent Plan Summer 2017 2018
- Dbn Models
- SEATWORK-OR2
- International Journal of Engineering Research and Development
- Spring15Midterm2_Sol.pdf
- Time Varying
- Analysis of the Effects of Delays on Load Sharing 1989

You are on page 1of 28

YL. Chang

Fall 2015

Newsvendor model deals with perishable products. You have to get rid of leftover after each decision epoch.

In this chapter, we talk about non-perishable products.

Motivation Example

Consider an inventory model:

Dn : the demand of the nth period (e.g. days or weeks). Demand is Random and its the distribution

is known.

Inventory that is left at the end of this period can be used to satisfy the demand in the next period

(e.g. paper, pencils, tires, ink etc.)

Suppose the Demand {Dn } is an i.i.d. sequence and it follows the following distribution:

d

P[Dn = d]

1/8

1/4

1/2

1/8

Assume Cp = $100, Cv = $50, Cf = $100, h = $10, where h is the holding cost, i.e., h = Cs .

Question: What is the optimal inventory policy to maximize the long-run average profit per week?

Please dont be confused this with the newsvendor problem. For newsvendor, you have to get rid of items

at the end of each period. In that case, time 0: make order decision; time 1: demand is realized and any

leftover is removed. Now we consider the case where the leftover can be used for the following period.

How to analyze the performance of an inventory policy for this case? Every Friday 5 pm, lets say we decide

how much to order for the following week so that the ordered items will arrive at 8 am the following Monday.

We sell from Monday to Friday. Lets again consider (s, S) policy.

Let Xn be the inventory level at the end of period n.

(i) If Xn < s, order S Xn items

(ii) Otherwise, do not order.

The goal of our class is again to determine the (s, S) policy so that our long-run average profit can be

maximized.

Lets do some analysis first on Xn .

Definition 1. Time series: A sequence of data points over time. Time can be discrete or continuous.

Hence, {Xn }n0 is a time series.

Assume our policy is (s, S) = (2, 4). Then

(i) If Xn 1, order 4 Xn items

(ii) Otherwise, do not order.

Lets consider the following probability:

P (Xn+1 = 3|Xn = 1)

Xn+1 = 3: the (n + 1)th week ends with 3 items.

P (Xn+1 = 3|Xn = 1): given the inventory level of nth week is 1, what is the probability that (n + 1)th

week ends with 3 items?

I want to remind you this is a conditional probability. Please go back to your 2027, first week

lectures and homework 1 to review basic concepts. Especially the Bayes rule.

How to calculate this probability ?

Xn = 1 means the nth week ends with 1 item. Then according to (s, S) policy, we order 4 1 = 3 items.

So next Monday, we start with 1 + 3 = 4 items that can sell. Since we want to have 3 items at the end of

(n + 1)th week, it simply means the demand of (n + 1)th week is 1. What is the probability that the demand

of (n + 1)th week is 1? We know P (Dn+1 = 1) = 1/4. Hence,

P (Xn+1 = 3|Xn = 1) = P (Next week ends with 3 items|This week ended with 1 item)

= P (Next week ends with 3 items|Next week begins with 4 items)

= P (Selling 1 itme during the next week)

= P (Dn+1 = 1) = 1/4

Can you calculate the following probabilities using the same idea?

P (Xn+1 = 4|Xn = 2) = 0

P (Xn+1 = 3|Xn = 2) = 0

P (Xn+1 = 2|Xn = 2) = 1/8

P (Xn+1 = 1|Xn = 2) = 1/4

P (Xn+1 = 0|Xn = 2) = 5/8

Do you know the possible value of Xn+1 ? What do you observe? State space S = {0, 1, 2, 3, 4}

What if Xn = 1?

P (Xn+1 = 4|Xn = 1) = 1/8

P (Xn+1 = 3|Xn = 1) = 1/4

P (Xn+1 = 2|Xn = 1) = 1/2

P (Xn+1 = 1|Xn = 1) = 1/8

P (Xn+1 = 0|Xn = 1) = 0

0

1

P = 2

3

4

0

0

0

5/8

1/8

0

1

1/8

1/8

1/4

1/2

1/8

2

1/2

1/2

1/8

1/4

1/2

3

1/4

1/4

0

1/8

1/4

1/8

1/8

0

0

1/8

1.1

Elements in DTMC

(1) State space (S): State space is a set containing all possible values that Xn can take. You will see that

S does not have to be finite. For example,

(a) Inventory level Xn in the motivation section. S = {0, 1, 2, 3, 4}

(b) The status of a machine S = {working, down}

(c) Tomorrows weather S = {Hot, Cold}

(d) Simple random walk. Suppose you toss a coin at each time n and you go up by 1 if you get

a head, or go down by 1 if you get a tail. You are interested in your physical position. Then

S = {..., 2, 1, 0, 1, 2, ...}.

(2) Transition probability matrix P = (Pij ). Transition probability matrix describes the probability of

a future event given the information about our current state.

Pij 0

P

jS Pij = 1, i The sum of each row should be 1, but the sum of each column does not have to

be 1.

P is a square matrix.

The size of P is (# of elements in state space)(# of elements in state space), i.e., |S| |S|.

(3) Initial Distribution a0 : Initial distribution provides information from which state the system stars off

at the very beginning. It determines where the system starts at time 0.

For example, if your current inventory level is 2, then the initial distribution a0 = (0, 0, 1, 0, 0).

If you have no idea about your initial inventory level and you believe it is equally likely to be 0,1,2,3,4,

then a0 = (1/5, 1/5, 1/5, 1/5, 1/5).

Definition 2. (Stochastic process) A stochastic process is a collection of random variables, representing the

evolution of some values (random) of the system over time.

A time series can be seen as a realization from a stochastic process. Or a stochastic process is a time series

model.

Definition 3. (Discrete Time Markov Chain) A discrete time stochastic process X = {Xn : n = 0, 1, 2, ...}

is said to be a DTMC on state space S with transition matrix P if for each n 1 for i0 , i1 , i2 , ..., i, j S

P (Xn+1 = j|X0 = i0 , X1 = i1 , X2 = i2 , ..., Xn1 = in1 , Xn = i) = Pij = P (Xn+1 = j|Xn = i).

(1)

Past states (history): X0 = i0 , X1 = i1 , X2 = i2 , ..., Xn1 = in1 .

Current state: Xn = i.

Future state Xn+1 = j.

Equation (1) simply means once you know todays state, tomorrows state has nothing to do with the past

states. Tomorrows state is irrelevant to how you reach todays state in the past.

Definition 4. (Markov Property) Given the current information (state), future and past are independent.

From the information gathering perspective, it is very appealing because you just need to remember

the current state. For an opposite example, Wikipedia keeps track of all the histories of each article.

It requires tremendous effort.

What if you think your situation depends not only on the current state but also on the state of one

week ago? Then you can properly redefine the state space so that each state describes the status of

your system for the two weeks, instead of one week. At this point, I want to stress that you are the

one who decide how your model should be. You can add more assumptions to fit your situation to a

Markov model.

1.2

DTMC Model

Define a DTMC:

Step 1: Define Xn , n 0. Xn usually has some physical meaning about your system, such as the

inventory level at the end of every Friday, the weather, etc.

Step 2: find state space S: find all possible value of Xn for n is sufficient large.

Step 3: find the transition probability matrix P . If you cannot find it, check whether the Markov

property hold. If it does not hold, you may need to redefine Xn . Or say the system cannot be modeled

as a DTMC.

Step 4: find the initial distribution a0 by initial information about the system.

1. A Two State Model S = {0, 1}

P = 0

1

0

1

1

Consider we are modeling the weather of a day. State 0 means hot and state 1 means cold. Then,

the probability of hot after a hot day is 3/4. The probability of hot after a cold day is 1/6.

P = Hot

Cold

Hot

3/4

1/6

Cold

1/4

5/6

Machine repairing process. State 0: machine is up and running. State 1: machine is down.

2. Inventory model in the motivation example: Let Yn be the

n. Then Y = {Yn : n = 0, 1, 2, ...} is a DTMC.

State space S = {2, 3, 4}.

2

3

2 1/8

0

P = 3 1/2 1/8

4 1/4 1/2

4

7/8!

3/8

1/4

3. A simple random walk: suppose you toss a coin at each time. You go right if you get a head, left if

you get a tail. Then S = {..., 2, 1, 0, 1, 2, ...} (infinite state space).

Let Xn is the position after nth toss of the coin.

p, j = i + 1

Pij = q, j = i 1

0, otherwise.

4. (Weird Frog Jumping Around). Imaging a situation that a frog is jumping around lily pads.

(1) Suppose the frog is jumping around 3 pads. Label each pad as 1, 2, and 3. The frogs jumping has

been observed extensively and behavioral scientists concluded this frogs jumping pattern to be as

follows.

1 2 3 2 1 2 3 2 ...

Let Xn denote the pad label that the frog is sitting on at time n. Is Xn a DTMC? If so, what are

the state space and the transition probability matrix? If not, why?

It is not a DTMC because Xn does not have the Markov property. To show it, lets try to complete

the transition probability matrix given that the state space S = {1, 2, 3}.

1

P = 2

3

1

0

?

0

2

1

?

1

3

!

0

?

0

It is impossible to predict the next step of the frog by just looking at the current location of it

if the frog is sitting on pad 2. If we know additionally where it case from to reach pad 2, it

becomes possible to predict its next step. That is, it requires past information to determine

probability of future. Hence, the Markov property does not hold. Xn cannot be a DTMC.

(2) A friend of the frog is also jumping around lily pads in the near pond. This friend frog jumps

around 4 pads instead of 3. Label each pad as 1, 2A, 2B, and 3. The friend frogs jumping pattern

is characterized as follows.

1 2A 3 2B 1 2A 3 2B 1 2A ...

Let Yn denote the pad on which the friend frog is sitting at time n. Is Yn a DTMC? If so, what

are the state space and the transition probability matrix? If not, why?

It is a DTMC. State space S = {1, 2A, 2B, 3} and the transition probability matrix is

1

1

0

0

P = 2A

2B 1

3

0

2A

1

0

0

0

2B

0

0

0

1

0

1

0

0

I strongly suggest you read the textbook. There are many examples in the textbook and many

exercises at the end of each chapter. You can find the solution at the end of your textbook.

Those examples and exercises will be very helpful for your midterms and final.

1.3

Sample Path

Let X = {Xn : n = 0, 1, 2, ...} be a DTMC with state space S = {0, 1, 2}.

0

P = 1

2

1.4

0

0

1/2

1

1

1/4

0

0

2

3/4!

1/2

0

Transition diagram

P = 0

1

0

P = 1

2

0

3/4

1/2

0

0

1/2

1

1

1/4

1/2

1

1/4

0

0

2

!

3/4

1/2

0

1.5

1.5.1

Calculating Probabilities

Math Review:

(2) P (E) = P (E F ) + P (E F c ) since (E F ) (E F c ) = .

(3) Law of total probability: if F1 F2P

... Fn = (mutually

Pn exclusive events),

n

P (F1 F2 ... Fn ) = 1, P (E) = i=1 P (E Fi ) = i=1 P (E|Fi )P (Fi ).

(4) P (A|B) =

1.5.2

P (AB)

P (B) , P (A

B) = P (A|B)P (B).

P = 0

1

0

0.7

0.5

1

0.3

0.5

and the initial distribution a0 = (0.6, 0.4), i.e., P (X0 = 0) = 0.6, P (X0 = 1) = 0.4. What is P (X1 = 0)?

P (X1 = 0) = P (X1 = 0, X0 = 0) + P (X1 = 0, X0 = 1) (Law of total probability)

= P (X1 = 0|X0 = 0)P (X0 = 0) + P (X1 = 0|X0 = 1)P (X0 = 1) (Conditional Probability)

= 0.7 0.6 + 0.5 0.4 = 0.62.

Exercise: Show that P (X1 = 1) = 0.38.

Let an be the probability distribution for Xn , then we say a1 = (0.62, 0.38), i.e., P (X1 = 0) = 0.62, P (X1 =

1) = 0.38.

We note,

a0 P = (0.6 0.4)

0.7 0.3

0.5 0.5

= (0.6 0.7 + 0.4 0.5 0.6 0.3 + 0.4 0.5) = (0.62 0.38) = a1

1.5.3

What is P (X2 = 0) ?

P (X2 = 0) = P (X2 = 0, X1 = 0) + P (X2 = 0, X1 = 1)

= P (X2 = 0|X1 = 0)P (X1 = 0) + P (X2 = 0|X1 = 1)P (X1 = 1)

= 0.7 0.62 + 0.5 0.38 = 0.624.

0.3

= (P (X2 = 0) P (X2 = 1)) = a2

0.5

0.7 0.3

0.7 0.3

a1 P = (a0 P )P = (0.6 0.4)

= (P (X2 = 0) P (X2 = 1)) = a2

0.5 0.5

0.5 0.5

a1 P = (0.62 0.38)

0.7

0.5

a0 P 2 = a2

Hence,

an = an1 P = (an2 P )P = an2 P 2 = a0 P n

an+1 = an P

an+k = an P k = an+1 P k1

Remark: Pij = P (Xn+1 = j|Xn = i), (P 2 )ij 6= (Pij )2 . (P 2 )ij = P (Xn+2 = j|Xn = i) =

(Pij )2 = [P (Xn+1 = j|Xn = i)]2 . In general, (P k )ij = P (Xn+k = j|Xn = i).

kS

0

P = 1

2

0

0

1/2

1

1

1/4

0

0

2

3/4!

1/2

0

P (X4 = 0|X2 = 0) = (P 2 )0,0 = 0.875 or

P (X4 = 0|X2 = 0) =

2

X

P (X4 = 0, X3 = i|X2 = 0)

i=0

= P (X4 = 0|X3 = 0, X2 = 0)P (X3 = 0|X2 = 0) + P (X4 = 0|X3 = 1, X2 = 0)P (X3 = 1|X2 = 0)

+ P (X4 = 0|X3 = 2, X2 = 0)P (X3 = 2|X2 = 0)

= P (X4 = 0|X3 = 0)P (X3 = 0|X2 = 0) + P (X4 = 0|X3 = 1)P (X3 = 1|X2 = 0)

+ P (X4 = 0|X3 = 2)P (X3 = 2|X2 = 0) (Markov Property)

= 0 0 + 0.5 0.25 + 1 + 0.75 = 7/8

a0 = (1/3, 1/3, 1/3),

P (X0 = 1|X1 = 2) =

P (X1 = 2, X0 = 1)

P (X1 = 2)

P (X1 = 2|X0 = 0)P (X0 = 0) + P (X1 = 2|X0 = 1)P (X0 = 1) + P (X1 = 2|X0 = 2)P (X0 = 2)

0.8 1/3

0.5

=

=

0.75 1/3 + 0.5 1/3

1.25

=

0

P 2 = 0.5

1

0.25

0

0

0.75

0

0.5 0.5

0

1

0.25

0

0

0.75

0.875

0.5 = 0.5

0

0

0

0.125

0.25

0.125

0.375

0.75

P (X4 = 2, X2 = 0|X1 = 1) = P (X4 = 2|X2 = 0)P (X2 = 0|X1 = 1) = (P 2 )0,2 P1,0 = 0.125 0.5

P (X10 = 0, X4 = 2, X2 = 0|X1 = 1) = P (X10 = 0|X4 = 2)P (X4 = 2|X2 = 0)P (X2 = 0|X1 = 1) =

(P 6 )2,0 (P 2 )0,2 P1,0

1.6

Stationary Distribution

Suppose this week, half of my time I drink coke and half of the time i drink Pepsi. i.e. S = {Coke, Pepsi}, a0 =

(0.5, 0.5). The transition probability matrix is:

P = Coke

Pepsi

Coke

0.7

0.5

Pepsi

0.3

0.5

Observe :

a1 = [0.6

0.4] = a0 P

0.38] = a1 P = a0 P 2

a2 = [0.62

a3 = [0.624

0.376] = a2 P = a0 P 3

0.3752] = a3 P = a0 P 4

a4 = [0.6248

a5 = [0.625

0.375] = a4 P = a0 P 5

a6 = [0.625

0.375] = a5 P = a0 P 6

a7 = [0.625

0.375] = a6 P = a0 P 7

...

a100 = [0.625

0.375] = a7 = a8 = ...a

Once you get to this distribution [0.625 0.375], it does not change over time any more. This distribution

describe the steady state of your system: what can happen if we run this system for a long period of time

(dynamic equilibrium).

Does the steady state always exist? Sometimes not. We will learn the conditions that can guarantee the

existence of the steady state later.

Definition 5. (Stationary Distribution). Suppose = (i , i S) satisfies

P

i 0 for all i S and iS i = 1,

= P

Then, is said to be a stationary distribution of the DTMC.

Remark: We could have either unique or infinite number of stationary distributions. The conditions that

can guarantee the uniqueness of stationary distribution will be given later.

What is the stationary distribution for the Coke and Pepsi example?

c 0, p 0

(c

p )

c + p = 1

0.7 0.3

= (c

0.5 0.5

c = 0.7c + 0.5p

p = 0.3c + 0.5p

c + p = 1

c = 0.625, p = 0.375. = [0.625

which is exactly same to what we observed above.

p )

p )

0.375].

In the test, you could be asked to give the stationary distribution for a simple transition probability matrix.

Hence, you should practice to compute the stationary distribution from the given transition matrix.

0

P = 1

2

0

0

1/2

1

1

1/4

0

0

2

!

3/4

1/2

0

0 0 + 1/2 1 + 2 = 0

1/4 + 0 + 0 =

0

1

2

1

3/4

+

1/2

+

0

=

2

0

1

2

0 + 1 + 2 = 1

= (0.4706

1.6.1

0.1176

0.4118).

(1) Stationary distribution represents the long-run fraction of time that the DTMC stays in each state.

(2) Stationary distribution represents the probability that the DTMC will be in a certain state if you run it

for very long time.

1.6.2

Examples

1. = [0.625

0.375] means

(1) In long run, 62.5% of the time, I drink coke. 37.5% of the time, I drink Pepsi.

(2) After a long time, you visit me on a day. Then the chance that you see i drink coke on that day

is 62.5%. The chance that you see i drink Pepsi on that day is 37.5%.

Questions:

(1) Suppose Pepsi is $1, and Coke is $1.5. How much on average I spend on soda in a month?

$1.5 (30 0.625) + $1 30 0.375

(2) If I drink soda for next 10 years (3650 days), how many days I will drink Pepsi?

3650 0.375

10

2. Inventory example. Suppose the Demand {Dn } is an i.i.d. sequence and it follows the following

distribution:

d

P[Dn = d]

0.1

0.4

0.3

0.2

Assume Cp = $2000, Cv = $1000, Cf = $1500, h = $100, where h is the holding cost for each item left

by the end of a Friday. Consider the (s = 2, S = 3) policy. What is the long-run average profit per

week?

Let f (i) be the expected profit of the following week, given that this weeks inventory ends with i

items.

i = 0:

no holding cost

Place order, and order 3 items. Cf + 3Cv

next week we start with S = 3 items.

f (0) = P (D = 1) 1 Cp + P (D = 2) 2 Cp + P (D = 3) 3 Cp (Cf + 3Cv )

= [1 0.4 $2000 + 2 $2000 0.3 + 3 $2000 0.2] [3$1000 + $1500] = $1300.

i = 1:

holding cost. 1 h

Place order, and order 2 items. Cf + 2Cv

next week we start with S = 3 items.

f (1) = P (D = 1) 1 Cp + P (D = 2) 2 Cp + P (D = 3) 3 Cp (Cf + 2Cv + h)

= [1 0.4 $2000 + 2 $2000 0.3 + 3 $2000 0.2] [2$1000 + $1500 + $100] = $400.

i = 2:

holding cost. 2 h

no order.

next week we start with 2 items.

f (2) = P (D = 1) 1 Cp + P (D = 2) 2 Cp + P (D = 3) 2 Cp (2h)

= [1 0.4 $2000 + 2 $2000 0.3 + 2 $2000 0.2] [2$100] = $2600.

Similarly,

f (3) = P (D = 1) 1 Cp + P (D = 2) 2 Cp + P (D = 3) 3 Cp (3h)

= [1 0.4 $2000 + 2 $2000 0.3 + 3 $2000 0.2] [3$100] = $2900.

Please construct the transition probability matrix P and solve the stationary distribution by yourself.

Note, the state space S = {0, 1, 2, 3}. Then,

Long-run average profit: f (0)0 + f (1)1 + f (2)2 + f (3)3 . The percentage of time that the system

is losing money: 0 + 1 . Assume you will get $500 if the system is making money and -$100 if the

system is losing money. Then your average bonus is $500 (2 + 3 ) $100 (0 + 1 ). The manager

may want to talk with you if he/she sees that the system is losing more than $1000 on a day. Then

the probability that you have to talk with your manager on a day is 0 .

11

1.6.3

Flow Balance Equations Under stationary condition, for any state, rate into a state should be equal to

the rate out of the state. (Ignore self loop)

State space S = {1, 2, 3}.

1

P = 2

3

1

0.25

0.5

0

2

0.25

0

1

3

0.5!

0.5

0

State 1:

rate in: 0.52

rate out: (0.5 + 0.25)1

Hence, 0.52 = 0.751

State 2:

rate in: 0.251 + 3

rate out: 2

Hence, 3 + 0.251 = 2

State 3:

rate in: 0.51 + 0.52

rate out: 3

Hence, 0.5(1 + 2 ) = 3

1 + 2 + 3 = 1 and = (0.2667, 0.4, 0.3333).

Cut Method

P0,0 = q, P0,1 = q, Pi,i+1 = p, Pi,i1 = q, i 1.

Consider A = {0}, then p0 = q1 , 1 = pq 0 ;

A = {0, 1}, then p1 = q2 , 2 = pq 1 = ( pq )2 0 ;

A = {0, 1, 2}, then p2 = q3 , 3 = pq 2 = ( pq )3 0 ;

A = {0, 1, 2, ..., n 1}, then pn1 = qn , n = pq n1 = ( pq )n 0 ;

And 0 + 1 + 2 + ... = 1.

Then 0 (1 + pq + ( pq )2 + ( pq )3 + ...) = 1

1

1

and ( 1p/q

) = 1 (Recall 1 + + 2 + 3 + ... = 1

for 0 < 1.)

0 = 1

p

= 1/2, n = (1/2)n 0 = (1/2)n+1 .

q

study CTMC.

12

1.7

Limiting probability

Recall n-step transition probability Pijn = P (Xk+n = j|Xk = i). let n , what can happen?

Examples:

(1) Chain goes well (I will explain when a chain goes well in next section.).

0.7 0.3

P =

0.5 0.5

P

= lim P = lim

n

0.7

0.5

n

0.3

0.5

=

0.625 0.375

0.625 0.375

(a) In the long run, what do you do today has little effect on the future. Limiting probability is

independent of the initial state. Each row is identical, which means your initial state is irrelevant

once you have run the system for a long time.

(b) Each row of the matrix converges to the stationary distribution.

0.625 0.375

n

P = lim P =

=

0.625 0.375

n

Hence, if the chain goes well, we could use P n (for n large) to get the stationary distribution quickly.

(2) Limit may not exist. For example,

P =

0

1

1

0

= P

(1

2 ) = (1

(

P 100 =

P

1000

1

0

0

1

1

0

0

1

2 )

0

1

1

0

1 + 2 = 1

1 = 2

= [1 2 ] = [0.5 0.5].

0 1

101

6=

,P

=

6=

1 0

0 1

1001

6=

,P

=

6

=

1 0

The behavior is similar to (1)n , where the limit of (1)n does not exist. The chain is obviously

oscillating. However,

1 100

101

(P

+P )=

2

1 1000

1001

(P

+P

)=

2

(3)

0.7 0.3

0.5 0.5

P =

0

0

0

0

0

0

0.6

0.3

0

5/8

0

, P = 5/8

0

0.4

0.7

0

13

3/8 0

3/8 0

0 3/7

0 3/7

0

0

4/7

4/7

The first two rows are identical and the last two rows are identical. However,

the second

and the third row

0.7 0.3

are different. In fact, = [5/8 3/8] is the stationary distribution of

and = [3/7 4/7]

0.5 0.5

0.6 0.4

is the stationary distribution of

. If you draw the transition diagram of this chain, you will

0.3 0.7

see it has two disjoint subchains. In long run, each of these subchains converges to its own stationary

distribution.

What is the stationary distribution?

= P

(1

4 ) = (1

0.7

0.5

4 )

0

0

0.71 + 0.52 = 1

0.31 + 0.52 = 2

0.63 + 0.34 = 3

0.43 + 0.74 = 4

+ + + = 1

1

2

3

4

0.3

0.5

0

0

0

0

0.6

0.3

0

0

0.4

0.7

52 = 31

43 = 34

1 + 2 + 3 + 4 = 1

We have 4 variables but we only have 3 equations. Hence, we may have infinite many of stationary

distribution. Indeed, if we set 1 + 2 = , 0 1, then 1 = 58 , 2 = 38 , 3 = 37 (1 ), 4 =

4

7 (1 ), i.e., = [ (1 )], 0 1.

1.8

1.8.1

DTMC Techniques

Accessibility and Irreducibility

Definition 6. (1) State i can reach state j if there exists n such that P (Xn = j|X0 = i) > 0, i.e., Pijn > 0.

This is mathematically noted as i j.

(2) State i and j are said to communicate if i j and j i.

(3) X is said to be irreducible if all states communicate. Otherwise, it is said to be reducible.

Why is irreducibility important? If a chain is reducible, we may have more than one stationary distribution.

See Example (3) in Section 1.7.

Examples:

1.

P =

0.2

0

0.1

0

0.5

0

0.5

0

0.2

0

0.5

0

0.1

0.5

0.5

0

0.5

0

0.3

0

0.3

0

0.8

0

0

2.

0 0.5

0.5 0

P =

0 0.5

0.5 0

This chain is irreducible, since 1 2 3 4.

14

0

0.5

0

0.5

0.5

0

0.5

0

(2) If X has a finite state space, it has at least one stationary distribution.

Corollary 1. For a finite state, irreducible DTMC, there exists a unique stationary distribution.

1.8.2

Periodicity

Lets look at the Example (2) in Section 1.7 again. It is a finite state, irreducible DTMC. Hence, we should

have a unique stationary distribution. However, P 1000 6= P 1001 and each row doesnt converge to the stationary distribution .

In this case, we cannot obtain the limiting probability, because the chain oscillates. Then what is the relationship between P and ?

Definition 7. (Periodicity) For a state i S,

n

d(i) = gcd{n : Pi,i

> 0}

Example:

1.

0

0.5

P =

0

0.5

0.5

0

0.5

0

0

0.5

0

0.5

0.5

0

0.5

0

Then,

d(1) = gcd{2, 4, 6, 8, ...} = 2

d(2) = 2.

We say this DTMC is periodic with period d = 2.

If fact, if i j, d(i) = d(j). It is called the solidarity property. Since all states in a irreducible

DTMC communicate, the periods of all states are the same. We will revisit it after covering recurrence

and transience.

2.

0.1 0.2

P = 0.5 0

0.6 0.4

0.7

0.5

0

Then,

d(1) = gcd{1, 2, 3, ...} = 1

d(2) = gcd{2, 3, 4, ...} = 1

We say this DTMC is aperiodic.

Theorem 2.

(ii) If DTMC is period with period d 2, then limiting probability does not exist. However,

P n + P n+1 + P n+2 + ... + P n+d1

n

d

lim

exists.

15

Examples:

1.

P =

0.2

0.5

0.8

0.5

Yes, 1 2.

(b) Is this chain aperiodic?

Yes, Pi,i > 0, hence,

d(i) = 1.

This is an easy way to check if a DTMC is aperiodic (It contains a self loop!).

(c) Does the DTMC have any stationary distributions? If so, how many? Can you solve them?

It is a finite state, irreducible DTMC. According to Corollary 1, there exists a unique stationary

distribution.

[1

P =

0.8

= [1

0.5

0.2

0.5

2 ]

2 ]

1 + 2 = 1

= [5/13

8/13].

According to Theorem 2, the limiting probability exists and even without the help from Matlab,

we can say

5/13 8/13

lim P n =

5/13 8/13

n

On the other hand, you dont need to calculate the stationary distribution. If you know P , you

can get the stationary distribution directly. In reality, very likely this is the way you obtain : use

any software to calculate P n for a very large n!

2.

0

0.5

P =

0

0.5

(a) Is this DTMC irreducible?

Yes, 1 2 3 4.

(b) Is this chain aperiodic?

No, the period is d = 2.

16

0.5

0

0.5

0

0

0.5

0

0.5

0.5

0

0.5

0

(c) Does the DTMC have any stationary distributions? If so, how many? Can you solve them?

It is a finite state, irreducible DTMC. According to Corollary 1, there exists a unique stationary

distribution.

[1

0

0.5

4 ]

0

0.5

0.5

0

0.5

0

P =

0.5

0

= [1

0.5

0

0

0.5

0

0.5

4 ]

1 + 2 + 3 + 4 = 1

= [1/4

1/4

1/4

1/4].

It is a DTMC with period 2. Hence, according to Theorem 2, the limiting probability does not

exists. However, limn (P n + P n+1 )/2 exists and it is

0.25 0.25 0.25 0.25

0.25 0.25 0.25 0.25

Summary:

MC

irreducible

aperiodic

irreducible

periodic

reducible

aperiodic

1.9

limiting

Exist,

independent of initial state

not exist

Stationary

unique

Remark

nice

unique

think about

P n +...P n+d1

d

exist

depends on initial state

maybe

Let X be a DTMC on state space S with transition matrix P . For each state i S, let i denote the first

time n 1 such that Xn = i. For example,

P (1 = 2|X0 = 2) = P (X1 6= 1, X2 = 1|X0 = 2)

P (3 = 3|X0 = 3) = P (X1 6= 3, X2 6= 3, X3 = 3|X0 = 3)

P (i = k|X0 = i): The first time the system returns to state i.

Definition 8. (1) State i is said to be recurrent if P (i < |X0 = i) = 1

(2) State i is said to be positive recurrent if E(i |X0 = i) < .

(3) State i is said to be transient if it is not recurrent.

(4) State i is said to be an absorbing state if Pi,i = 1.

(1) means there is always a way to get back to state i. (3) means it is possible that you cannot return to

state i.

17

1.9.1

What is the difference between P (i < |X0 = i) = 1 vs E(i |X0 = i) < ? In general, What is the

difference between P (X < ) = 1 and E[X] < ? Arent they same?

Lets look at the meaning of each expression:

(1) P (X < ) = 1 means the random variable X can only take finite values. X could be 1, 2, 100, 10000

billon etc. However, X cannot be infinite, i.e., P (X = ) = 0.

(2) E[X] < means the expected value of X is finite.

Example:

Let X be the number of tosses to get the 1st head, then X is a geometric random variable. It may take

millions of tosses for you to get the 1st head.

P (X = n) = q n1 p

where p is the probability of getting a success and q = 1 p.

Do we have P (X < ) = 1?

P (X < ) =

n=1

1

=1

1q

Yes, we do. It means no matter how small the success probability is, it will take a finite

number of trials to get the first success for sure! I hope you can remember this fact. You will

have many challenges in reality, and sometimes you may be so frustrated. However, Keep going! You

will definitely make it!

How about the expectation?

E[X] = 1p + 2qp + 3pq 2 + ... = p(1 + 2q + 3q 2 + ...) = p(q + q 2 + q 3 + ...)0

q 0

1q+q

p

1

= p(

) = p[

]=

= <

1q

(1 q)2

(1 q)2

p

Hence, the expected number of trails to get a success is also finite.

You may ask: if we have a random variable X such that P (X < ) = 1, should we always have E[X] < ?

P

2

Not necessary. For example, we know n=1 n12 = 6 . Consider the random variable X such that P (X =

n) = n12 62 for n 1. Then

P (X < ) =

P (X = n) =

n=1

However,

E[X] =

nP (X = n) =

n=1

6 X 1

=1

2 n=1 n2

X

6 1

= .

2 n

n=1

Yes, this is true. If P (X = ) > 0, then E[X] = .

Hence, go back to our Definition 8. Now you should understand condition (1) and condition (2) are different.

Condition (2) is stronger than Condition (1). A positive recurrent state is recurrent. However, a recurrent

state may not be positive recurrent. In fact, we say state i is null recurrent if P (i < |X0 = i) = 1 but

E(i |X0 = i) = .

18

Definition 9. (Solidarity property). If i j, then d(i) = d(j) where d(k) is the period of state k. Also, i

is (positive) recurrent or transient if and only if j is (positive) recurrent or transient.

The solidarity property states that if two states communicate with each other, it is not possible that one

state i is recurrent and the other state j is transient. Therefore, if we apply the solidarity property to a

irreducible DTMC, only one of the following states is true:

(1) All states are transient.

(2) All states are (positive) recurrent.

The reason is simple: all states communicate with each other in an irreducible DTMC.

Equivalently,

Assume a Markov chain is irreducible, if state i is (positive) recurrent, then every state is (positive)

recurrent and we say this Markov chian is (positive) recurrent.

If a Markov chain is irreducible, then every state is either (positive) recurrent or transient.

Theorem 3. Assume that X is irreducible in a finite state space. Then,

(1) X is positive recurrent.

(2) X has a unique stationary = (i )i .

n

(3) If the DTMC is aperiodic, limn Pi,j

= j , independent of what i is.

n+d1

n+1

n

+...+Pi,j

Pi,j

+Pi,j

d

= j

Theorem 4. Assume X is irreducible. X is positive recurrent if and only if X has a (unique) stationary

distribution = (i ). Furthermore,

1

E(i |X0 = i) =

i

Example:

P =

0.5

1

0.5

0

P (1 = 1|X0 = 1) = 0.5

P (1 = 2|X0 = 1) = P (X1 6= 1, X2 = 1|X0 = 1) = P (X1 = 2, X2 = 1|X0 = 1) = P (X2 = 1|X1 =

2)P (X1 = 2|X0 = 1) = 1 0.5 = 0.5

P (1 = 3|X0 = 1) = P (X1 6= 1, X2 6= 1, X3 = 1|X0 = 1) = P (X1 = 2, X2 = 2, X3 = 1|X0 = 1) = 0

P (1 = 4|X0 = 1) = 0

For n 4, P (1 = n|X0 = 1) = 0

Hence, E[1 |X0 = 1] = 1 0.5 + 2 0.5 = 1.5 < and state 1 is positive recurrent. From the solidarity

property (or class property), we know state 2 is also positive recurrent. You could verify that the stationary

distribution = (2/3 1/3). From Theorem 4, we know E(1 |X0 = 1) = 11 = 3/2 which is equal to what

we calculated. This chain is aperiodic (because of a self loop), then according to Theorem 3, we know

2/3 1/3

lim P n =

2/3 1/3

n

In addition,

there is only one class {1, 2} which is positive recurrent. Hence, this chain is also irreducible.

since it contains a self loop and it is irreducible, we know this chain is aperiodic, i.e., d(1) = d(2) = 1.

19

1.9.2

Classification of States

Definition 10. (Class) Two states that communicate are said to be in the same class. Within each class,

all states communicate to each other, but no pair of states in different classes communicates.

In other words, the concept of communication divides the state space into a number of separate classes.

Hence,

The Markov chain is said to be irreducible if there is only one class, that is, if all states communicate

with each other.

Solidarity property Class property.

Definition 11. (Closed Subset) A subset S0 S of states is closed if Pij = 0 for each i S0 and j

/ S0 .

In plain English, once entered, a closed set cannot be existed.

Theorem 5. If a closed subset S0 only has finitely many states, then there must be at least one recurrent

state. In particular, any finite Markov chain must contain at least one recurrent state.

Proof. Start from any state from S0 . By definition, the chain stays in S0 forever. If all states in S0 are

transient, then each of them is visited not at all or only finitely many times. This is impossible.

Examples:

1.

1

0

0.7

0.8

1

P = 2

3

2

0.6

0

0.2

3

0.4!

0.3

0

We have only class: {1, 2, 3} positive recurrent according to Theorem 3.

Is the chain irreducible?

Yes. It has only one class.

2.

1

2

P = 3

4

5

0

0

1

0

2

1

0

0

0

0

3

0

1

0

0

0

4

0

0

1

0

1

0

0

0

0

We have two classes: {1, 2, 3, 4} positive recurrent and {5} transient.

Is the chain irreducible?

No. We have more than 1 class.

3. Consider the chain on states 1,2,3,4,5 and

1

2

P = 3

4

5

1

0

1

0

0

0

2

1

0

0

0

0

20

3

0

0

0.25

0.3

0

4

0

0

0.75

0.6

0

0

0

0

0.1

1

class {1, 2} is positive recurrent; class {3, 4} is transient; class {5} is positive recurrent.

Is the chain irreducible?

No. We have more than 1 class.

What is the period of state 1? What is the period of state 5?

d(1) = 2, d(5) = 1, d(1) 6= d(5) since state 1 and state 5 do not communicate with each other.

4. Consider a simple random walk on S = {..., 3, 2, 1, 0, 1, 2, 3, ...}. Pi,i+1 = p and Pi,i1 = q = 1 p.

Let p = 2/3 and q = 1/3. Then this chain is transient.

Proof. Define

(

1

Xi =

1

Then Sn =

Pn

i=1

with probability p

with probability q

Pn

Xi

Sn

= i=1

E[Xi ] = (1)q + 1p = 1/3

n

n

Hence, Sn = 13 n and limn Sn = +. So if you start from state 2, you will never get back to state

2 in long-run. This means every state in the chain is transient.

In fact, if p 6= q, you can show this chain is always transient by the same method. If p = q, this chain

is null recurrent. i.e., it is recurrent, but not positive recurrent. The proof is omitted.

Final Remarks

In a DTMC with finite state space, not all states can be transient. i.e., there exists at least one

recurrent state.

A recurrent state is accessible from all states in its class, but is not accessible from recurrent states in

other classes.

A transient state is not accessible from any recurrent state.

At least one, possibly more, recurrent states are accessible from a given transient state.

21

1.10

Absorption Probability

How to calculate limn P n if the DTMC has possibly more than one class?

Key observation: When n is sufficiently large, any transient state will definitely enter one of the recurrent

classes. i.e., after a very large n, transient states will detach from the DTMC (intuitively). Hence, we

could focus on the recurrent classes first and then come back to analyze the transient classes.

Let

fi,j = P (X = j|X0 = i) = lim (P n )i,j

n

This is P

the probability that starting from state i, the DTMC will be at state j eventually.

Hence, jS fi,j = 1. i.e., the sum of each row of P equals 1.

Question: What is P (X = j|X20 = i)?

P (X = j|X20 = i) = P (X = j|X200 = i) = fi,j from definition.

Examples:

1.

1

P = 1 (1)

What is P ?

P = f1,1 = 1.

2.

1

0.2

0.8

P = 1

2

What is P ? Assume you know that

0.2

(0.5 0.5)

0.8

0.8

0.2

2

0.8

0.2

= (0.5

0.5) = (1

0.5 0.5

P =

.

0.5 0.5

3.

P = 1

2

1

1

1

2

0

0

What is P ?

f1,1 = 1

f1,2 = 0 since

jS

fi,j = 1.

f2,2 = 0 since state 2 is transient.

P =

22

1

1

0

0

.

2 )

4.

1

1

0.5

P = 1

2

2

0

0.5

What is P ?

f1,1

f1,2

f2,1

f2,2

=1

P

= 0 since jS fi,j = 1.

= 1 since state 2 is transient.

= 0 since state 2 is transient.

P

=

1

1

0

0

.

5.

1

0.25

0.5

1

1

P = 2

3

2

0.75

0.5

0

3

!

0

0

0

(0.4

f1,1

f1,2

f1,3

f2,1

f2,2

f2,3

f3,1

f3,2

f3,3

0.6)

0.25

0.5

0.75

0.5

= (0.4

0.6) = (1

2 )

= 0.4

= 0.6

P

= 0 since jS fi,j = 1 or {1, 2} is a closed subset.

= 0.4

= 0.6

P

= 0 since jS fi,j = 1 or {1, 2} is a closed subset.

= 0.4 = 1 Make sure you understand why f3,1 = 1

= 0.6 = 2 Make sure you understand why f3,2 = 2

= 0 since it is transient.

0.4 0.6 0

P = 0.4 0.6 0 .

0.4 0.6 0

6.

1

P = 2

3

1

1

0.25

0

2

0

0.5

0

3

!

0

0.25

1

What is P ?

Class {1} is recurrent. Class {2} is transient. Class {3} is recurrent. And we know P1,1

= P3,3

=1

because they are absorbing states. Furthermore, we know the probability that state 3 goes to class {1}

is equal to the probability that state 3 goes to class {2}. Hence,

1

= 2

3

23

1

1

0.5

0

2

0

0

0

3

!

0

0.5

1

Note, P2,2

= 0 since it is a transient state.

7.

1

1

1/3

0

1

P = 2

3

2

0

1/2

0

3

!

0

1/6

1

What is P ?

Same as before:

Class {1} is recurrent. Class {2} is transient. Class {3} is recurrent.

= P3,3

= 1 because they are absorbing states. P2,2

= 0 since it is a transient state.

1

= 2

3

1

1

?

0

2

0

0

0

3

!

0

?

1

However, we know state 3 can go to class {1} or class {2} with different probabilities. Note,

f2,1 = P (X = 1|X0 = 2)

= P (X = 1, X1 = 1|X0 = 2) + P (X = 1, X1 = 2|X0 = 2) + P (X = 1, X1 = 3|X0 = 2)

= P (X = 1|X1 = 1)P (X1 = 1|X0 = 2) + P (X = 1|X1 = 2)P (X1 = 2|X0 = 2)

+ P (X = 1|X1 = 3)P (X1 = 3|X0 = 2)

= 1 1/3 + f2,1 1/2 + 0 1/6 = 1/3 + 1/2 f2,1

And

f2,1 + f2,3 = 1

Hence, we have

(

f2,1 + f2,3 = 1

f2,1 = 1/3 + 1/2 f2,1

1

1

2/3

0

1

= 2

3

2

0

0

0

3

!

0

1/3

1

8.

1

1 0.25

0.5

P = 2

3 0

4

0

2

0.75

0.5

1/3

0

3

0

0

1/2

0

4

0

0

1/6

1

What is P ?

Class {1, 2} is recurrent. Class {3} is transient. Class {4} is recurrent.

From Problem 2, 5, 7, we know

1

1

1

1

= 2

3 f3,{1,2} 1

4

0

2

2

2

f3,{1,2} 2

0

3

0

0

0

0

0.4

0

0.4

0 =

2/3 0.4

f3,4

0

1

24

0.6

0

0.6

0

2/3 0.6 0

0

0

0

0

1/3

1

= P (X {1, 2}, X1 = 1|X0 = 3) + P (X {1, 2}, X1 = 2|X0 = 3)

+ P (X {1, 2}, X1 = 3|X0 = 3) + P (X {1, 2}, X1 = 4|X0 = 3)

= P (X {1, 2}|X1 = 1)P (X1 = 1|X0 = 3) + P (X {1, 2}|X1 = 2)P (X1 = 2|X0 = 3)

+ P (X {1, 2}|X1 = 3)P (X1 = 3|X0 = 3) + P (X {1, 2}|X1 = 4)P (X1 = 4|X0 = 3)

= 1 0 + 1 1/3 + f3,{1,2} 1/2 + 0 1/6 = 1/3 + 1/2 f3,{1,2}

Make sure you understand the following:

P (X {1, 2}|X1 = 2) = 1 since {1, 2} is a closed subset.

P (X {1, 2}|X1 = 3) = f3,{1,2} from definition.

P (X {1, 2}|X1 = 4) = 0 since {4} is closed. It cannot get out of state 4.

And

f3,{1,2} + f3,4 = 1

Hence, we have

(

f3,{1,2} + f3,4 = 1

f3,{1,2} = 1/3 + 1/2 f3,{1,2}

Please compare this problem with Problem 7. You should be able to see that they are

exactly same! I will use Problem 9 to illustrate the general procedure for calculating

absorption probabilities.

Steps:

Draw the transition diagram.

Identify the classes. Are they recurrent or transient?

Calculate the stationary distribution for each recurrent class. And fill the rows for each recurrent

state.

Fill the rows for each transient state using and fi,j

Set up a system of equations to calculate fi,j and complete P .

9. Calculate limn P n , where

P =

0.2

0.5

0

0

0

0.8

0.5

0.25

0

0

25

0

0

0

0.5

0

0

0

0.75

0

0

0

0

0

0.5

1

{1,2}, recurrent class

{3,4}, transient class

{5}, recurrent class (absorption)

Step 2: Work on rows of P for current classes (fill the blanks, not question marks).

1 2

1 2

?

?

?

?

P = ?

.

?

?

?

?

?

where

(1

2 ) = (1

2 )

0.2

0.5

0.8

0.5

1 + 2 = 1

0.21 + 0.52 = 1

0.81 + 0.52 = 2

1 + 2 = 1

(1

2 ) = (5/13

P =

1

1

?

?

0

2

2

?

?

0

8/13).

0 0 0

0 0 0

? ? ?

.

? ? ?

0 0 1

Step 3: Work on rows of P for transient classes(fill the blanks), using f3,{1,2} , f4,{1,2} , f3,5 , f4,5 ,

5/13

8/13

5/13

8/13

f3,{1,2} 1

f3,{1,2} 2

f3,5

f4,{1,2} 1

f4,{1,2} 2

f4,5

where:

f3,{1,2} = P (X {1, 2}|X0 = 3)

f4,{1,2} = P (X {1, 2}|X0 = 4)

f3,5 = P (X = 5|X0 = 3)

f4,5 = P (X = 5|X0 = 4)

26

Again, why P3,1

= f3,{1,2} 1 ?

is the probability that state 3 will be trapped in recurrent class {1, 2} times the

probability that it visits state 1 once it enters the class {1, 2}.

Step 4: Solve the f3,{1,2} , f4,{1,2} , f3,5 , f4,5 and complete the matrix.

5/13

8/13

5/13

8/13

2/5 5/13

2/5 8/13

3/5

1/5 5/13

1/5 8/13

4/5

5

X

i=1

5

X

5

X

i=1

i=1

+ P (X {1, 2}|X1 = 4)P34 + P (X {1, 2}|X1 = 5)P35

= 1 0 + 1 0.25 + f3,{1,2} 0 + f4,{1,2} 0.75 + 0 0 = 0.25 + 0.75f4,{1,2}

5

X

i=1

5

X

i=1

5

X

i=1

+ P (X {1, 2}|X1 = 4)P44 + P (X {1, 2}|X1 = 5)P45

= 1 0 + 1 0 + f3,{1,2} 0.5 + f4,{1,2} 0 + 0 0.5 = 0.5f3,{1,2}

f3,{1,2} + f3,5 = 1

f4,{1,2} + f4,5 =

Hence, f3,{1,2} = 2/5, f3,5 = 3/5, f4,{1,2} = 1/5, f4,5 = 4/5.

27

1.11

Review

Markov property

Being able to model a problem by a DTMC, including (1) identify the state space; (2) construct

transition probability matrix; (3) initial distribution

Meaning of transition probability P and P n

Sample path and transition diagram

Calculate 1-step transition probability P (X1 ) using Pij

Calculate n-steps transition probability P (Xn |X0 ) using Pijn and notice that (Pi,j )n 6= (P n )ij

Calculate stationary distribution (3 methods: (1) definition (2) flow balance equation and (3) cut

method)

Explanation of stationary distribution

DTMC technique

(1) Accessibility, Irreducibility and definition of a class.

(2) Existence of stationary distribution and number of stationary distributions.

(3) Difference between P (i < |X0 = i) and E[i |X0 = i] < .

(4) Recurrent/transient/absorption state: you need to identify if a state/class is recurrent or transient.

(5) Period of a state.

(6) Solidarity (class) property.

(7) Limiting probability, existence and its relationship with stationary distribution.

(8) Relationship between stationary distribution and E[i ].

(9) Calculate absorption probability matrix P .

28

- Algorithms of Hidden Markov Model and a Prediction Method on Product PreferencesUploaded byJournal of Computing
- HW2Solutions.pdfUploaded byDavid Lee
- HW1SoltutionUploaded bySamuel Diaz
- HW3Solutions.pdfUploaded byDavid Lee
- ECE697AA 08-10-31 Queuing Systems IUploaded byabarai_renji1978
- The Need to Clarify the Relationship between Physics and Mathematics in Science Curriculum: Cultural Knowledge as Possible FrameworkUploaded bydavid.valenzuela.z3870
- HW7Solutions.pdfUploaded byDavid Lee
- MA6451-Probability and Random Processes QBUploaded bySabari
- Semantic RichMMUploaded bySandeep Rai
- Assignment 1Uploaded bysunnybohra
- Pattern and Predictability in Music.pdfUploaded byorestis willemen
- Dbn ModelsUploaded byjuwaldo
- DLP WEEK 10 DAY 4-1.docxUploaded bysusan narvas
- CONVERGENCE OF ESTIMATED OPTION PRICE IN A REGIME SWITCHING MARKETUploaded byAnindya Goswami
- SEATWORK-OR2Uploaded byMaria Sy
- International Journal of Engineering Research and DevelopmentUploaded byIJERD
- Year 5 Parent Plan Summer 2017 2018Uploaded byAnonymous z6TIpq6el
- Spring15Midterm2_Sol.pdfUploaded byDavid Lee
- Time VaryingUploaded bynarendra
- Analysis of the Effects of Delays on Load Sharing 1989Uploaded byAnkita Singhal
- Managing Dinamic InventoryUploaded byAlexander Bardales
- 4. IJAMSS - Multi Server Batch Queueing Model With Variable Service RatesUploaded byiaset123
- Innovation and SelectionUploaded byJorge Rasner
- Castell_OPER/RES SyllabusUploaded byMisty Chan
- FALL 2014 Operation Management OM 0010Uploaded byRituTiwari
- kalmanUploaded byMariella Gonzales
- SSRN-id968338.pdfUploaded bygoogley71
- 0034_Valuation_and_hedging_of_european_contingent_claims_on_power_with_spikes_a_non-Markovian_approach[1]Uploaded byDiffeomorfizmus
- Exercise 1Uploaded byShubham Upadhyay
- Stochastic Dynamical SystemsUploaded byNam

- CTMCUploaded byDavid Lee
- Fall15Midterm1 SolUploaded byDavid Lee
- DTMCUploaded byDavid Lee
- A+Quick+Guide+to+R+SimulationUploaded byDavid Lee
- t1Fall2013A.pdfUploaded byDavid Lee
- TeamRegistrationb.pdfUploaded byDavid Lee
- zTable.pdfUploaded byDavid Lee
- t2fall2014B.pdfUploaded byDavid Lee
- t2Fall2014A.pdfUploaded byDavid Lee
- t2Fall2013A.pdfUploaded byDavid Lee
- t1Spring2013key.pdfUploaded byDavid Lee
- t1Spring2013A.pdfUploaded byDavid Lee
- t1Fall2014Keys.pdfUploaded byDavid Lee
- t1Fall2014Keys (1).pdfUploaded byDavid Lee
- t1Fall2014A.pdfUploaded byDavid Lee
- Spring15Midterm2_Sol.pdfUploaded byDavid Lee
- Spring15Midterm2.pdfUploaded byDavid Lee
- Spring15Midterm1_Sol.pdfUploaded byDavid Lee
- Spring15Midterm1.pdfUploaded byDavid Lee
- Quiz13Solution.pdfUploaded byDavid Lee
- Quiz10Solution.pdfUploaded byDavid Lee
- Quiz11Solution.pdfUploaded byDavid Lee
- Quiz7Solution.pdfUploaded byDavid Lee
- Quiz9Solution.pdfUploaded byDavid Lee
- Quiz5Solution.pdfUploaded byDavid Lee
- Quiz6Solution.pdfUploaded byDavid Lee
- Quiz3Solution.pdfUploaded byDavid Lee
- Quiz4Solution.pdfUploaded byDavid Lee
- Quiz1Solution.pdfUploaded byDavid Lee

- s s-7 relation versus functionUploaded byapi-253679034
- Solutions for 2007 a Level H2 Maths Paper 1Uploaded byyoungyip
- slope intercept form stationsUploaded byapi-366582437
- Successful Prediction of Horse Racing Results using a Neural Network.pdfUploaded byBartoszSowul
- Data structures projectUploaded bysarath
- Final Practice SolutionsUploaded byClaudio Ramón Rodriguez Mondragón
- Cal Culo Diferencia lUploaded byDaniel Cifuentes
- 2005 JanUploaded bylle_91
- Numerical Continuation MethodUploaded byMaroju Prashanth
- chapter 3 sumUploaded byapi-201420434
- Projection From 4D to 3D - Zacharias VelichovaUploaded byNilo de Roock
- Digital Signal Processing LabUploaded byDhiraj Thakur
- E4Qdif_XUploaded byRevelation
- Book TotUploaded byCleverence Kombe
- A guide to vehicle routing heuristics (1).pdfUploaded byasvenesekar
- SPM Graph TrigoUploaded byCgzam Zam
- Special Elements of a Ternary SemiringUploaded byAnonymous 7VPPkWS8O
- Chapter 4_Sorting AlgorithmUploaded byNguyễnHoàngCầm
- Bmath 311 Problem Set.docxUploaded byZace Hayo
- Basic Matrix OperationsUploaded byNeha Karthikeyan
- Elliptic Pde Lecture notes `Uploaded bycrovaxIII
- problemsUploaded byevanlyn
- Reference Book for CsirUploaded bySuhas Phalak
- Signal Flow GraphUploaded bysymbo11
- Mesiar-general Chebyshev Type Inequalities for Universal IntegralUploaded byAnnisa Zakiya
- (Lecture Notes in Mathematics 1772) Burstall F.E. - Chapter 7 Metric and Affine Conformal Geometry-Springer (2002)Uploaded byCarlos Lopez
- Curriculum Specifications Form 4Uploaded byEta Puzi
- Exam - 2011 - (Solutions)Uploaded byRaymond Long
- Dynamic Programming (Coin Change)Uploaded byVineet Kumar
- 2018 III Ing Civil Matematica III( List 10)bUploaded byRichar Ivan Saldaña Bustamante