IE 301 Fall 2019 – Recitation Week #12

Discrete Time Markov Chains

(Closed sets = classes)

Solution:

a. State 4 is transient.
b. States 1,2,3,5 and 6 are recurrent.
c. {1, 3, 5} and {2,6} and {4} are the classes.
d. Since states 4 and 1 do not communicate the chain is not
ergodic.

Solution:

P1 is ergodic. All states communicate, hence there is

only one class. All states are recurrent. And these states
are aperiodic. Hence the states are ergodic and MC is
ergodic.

P2 is not ergodic. State 4 doesn’t communicate with any

other state. {1,2,3} is one transient class. {4} is a
recurrent class. Also notice that state 4 is absorbing. All
states are aperiodic.

Solution:

P1 is ergodic. P2 is not ergodic (note that states 1 and 2 do

not communicate).

Solution:

a. π1 = 2π1/3 + .5π2 and π1 + π2 = 1. Thus π1 = .6 and π2 = .4.

b. π1 = .8π1 + .8π3, π3 = .8π2, and π1 + π2 + π3 = 1. Solving these equations, we find that π1 = 16/25, π2 = 1/5, and
π3 = 4/25.

This study source was downloaded by 100000835991620 from CourseHero.com on 04-08-2023 08:56:31 GMT -05:00

https://www.coursehero.com/file/53152347/IE-301-Fall-2019-Recitation12-Solutionspdf/
 5. Consider the Markov chain that has the following (one-step) transition matrix.

0  0 4 / 5 0 1/ 5 0 

1 1 / 4 0 1 / 2 1 / 4 0 
P = 2  0 1 / 2 0 1 / 10 2 / 5
 
3 0 0 0 1 0 
4 1 / 3 0 1 / 3 1 / 3 0 

(a) Determine the classes of this Markov chain and, for each class, determine whether it is recurrent or
transient.
(b) For each of the classes identified in part (a), determine the period of the states in that class.

Solution:

0 1 2 3 4
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1

a) There are two classes {0, 1, 2, 4} transient, {3} recurrent and absorbing.
b) Period of 3 is 1.
Time 0 1 2 3 4 5 6
States 0 1 0 1 0 1 0
3 2 3 2 3 2
3 4 3 4 3

Period of {0, 1, 2, 4} is 2.

6. A soap company specializes in a luxury type of bath soap. The sales of this soap fluctuate between two levels
- "Low" and "High" - depending upon two factors: (1) whether they advertise, and (2) the advertising and
marketing of new products being done by competitors. The second factor is out of the company's control, but
it is trying to determine what its own advertising policy should be. For example, the marketing manager's
proposal is to advertise when sales are low but not to advertise when sales are high. Advertising in any quarter
of a year has its primary impact on sales in the following quarter. Therefore, at the beginning of each quarter,
the needed information is available to forecast accurately whether sales will be low or high that quarter and to
decide whether to advertise that quarter.

This study source was downloaded by 100000835991620 from CourseHero.com on 04-08-2023 08:56:31 GMT -05:00

https://www.coursehero.com/file/53152347/IE-301-Fall-2019-Recitation12-Solutionspdf/
 The cost of advertising is $1 million for each quarter of a year in which it is done. When advertising is done
during a quarter, the probability of having high sales the next quarter is 0.5 or 0.75, depending upon whether
the current quarter’s sales are low or high. These probabilities go down to 0.25 or 0.5 when advertising is not
done during the current quarter. The company's quarterly profits (excluding advertising costs) are $4 million
when sales are high but only $2 million when sales are low. (Hereafter, use units of millions of dollars.)

(a) Construct the (one-step) transition matrix for each of the following advertising strategies: (i) never
advertise, (ii) always advertise, (iii) follow the marketing manager's proposal.
(b) Determine the steady-state probabilities for each of the three cases in part (a).
(c) Find the long-run expected average profit (including a deduction for advertising costs) per quarter for each
of the three advertising strategies in part (a). Which of these strategies is best according to this measure of
performance?
Solution:

a)
No advertising Always advertising Advertising when sales are low

Low High Low High Low High

P= Low 0.75 0.25 P= Low 0.5 0.5 P= Low 0.5 0.5
High 0.5 0.5 High 0.25 0.75 High 0.5 0.5

b) No advertising: Always advertising: Advertising when sales are low:

𝜋𝑙𝑜𝑤 = 𝜋𝑙𝑜𝑤 0.75 + 𝜋ℎ𝑖𝑔ℎ 0.5 𝜋𝑙𝑜𝑤 = 𝜋𝑙𝑜𝑤 0.5 + 𝜋ℎ𝑖𝑔ℎ 0.25 𝜋𝑙𝑜𝑤 = 𝜋𝑙𝑜𝑤 0.5 + 𝜋ℎ𝑖𝑔ℎ 0.5
𝜋ℎ𝑖𝑔ℎ = 𝜋𝑙𝑜𝑤 0.25 + 𝜋ℎ𝑖𝑔ℎ 0.5 𝜋ℎ𝑖𝑔ℎ = 𝜋𝑙𝑜𝑤 0.5 + 𝜋ℎ𝑖𝑔ℎ 0.75 𝜋ℎ𝑖𝑔ℎ = 𝜋𝑙𝑜𝑤 0.5 + 𝜋ℎ𝑖𝑔ℎ 0.5
𝜋𝑙𝑜𝑤 + 𝜋ℎ𝑖𝑔ℎ = 1 𝜋𝑙𝑜𝑤 + 𝜋ℎ𝑖𝑔ℎ = 1 𝜋𝑙𝑜𝑤 + 𝜋ℎ𝑖𝑔ℎ = 1
==> 𝜋𝑙𝑜𝑤 0.25 = 𝜋ℎ𝑖𝑔ℎ 0.5 ==> 𝜋𝑙𝑜𝑤 0.5 = 𝜋ℎ𝑖𝑔ℎ 0.25 ==> 𝜋𝑙𝑜𝑤 0.5 = 𝜋ℎ𝑖𝑔ℎ 0.5
==> 𝜋𝑙𝑜𝑤 = 2𝜋ℎ𝑖𝑔ℎ ==> 𝜋𝑙𝑜𝑤 = 0.5𝜋ℎ𝑖𝑔ℎ ==> 𝜋𝑙𝑜𝑤 = 𝜋ℎ𝑖𝑔ℎ
2 1 1 2 1 1
==> 𝜋𝑙𝑜𝑤 = , 𝜋ℎ𝑖𝑔ℎ = ==> 𝜋𝑙𝑜𝑤 = , 𝜋ℎ𝑖𝑔ℎ = ==> 𝜋𝑙𝑜𝑤 = , 𝜋ℎ𝑖𝑔ℎ =
3 3 3 3 2 2

c) No advertising:
2 1 8
𝐿𝑜𝑛𝑔 − 𝑟𝑢𝑛 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑃𝑟𝑜𝑓𝑖𝑡 = 𝜋𝑙𝑜𝑤 ∗ 2 + 𝜋ℎ𝑖𝑔ℎ ∗ 4 = ∗2+ ∗4=
3 3 3
Always advertising:
1 2 7
𝐿𝑜𝑛𝑔 − 𝑟𝑢𝑛 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑃𝑟𝑜𝑓𝑖𝑡 = 𝜋𝑙𝑜𝑤 ∗ 2 + 𝜋ℎ𝑖𝑔ℎ ∗ 4 − 1 = ∗2+ ∗4−1=
3 3 3
Advertising when sales are low:
1 1 5
𝐿𝑜𝑛𝑔 − 𝑟𝑢𝑛 𝐴𝑣𝑒𝑟𝑎𝑔𝑒 𝑃𝑟𝑜𝑓𝑖𝑡 = 𝜋𝑙𝑜𝑤 ∗ (2 − 1) + 𝜋ℎ𝑖𝑔ℎ ∗ 4 = ∗1+ ∗4=
2 2 2

The highest is the “no advertising” policy.

This study source was downloaded by 100000835991620 from CourseHero.com on 04-08-2023 08:56:31 GMT -05:00

https://www.coursehero.com/file/53152347/IE-301-Fall-2019-Recitation12-Solutionspdf/
 7. A production process contains a machine that deteriorates rapidly in both quality and output
under heavy usage, so that it is inspected at the end of each day. Immediately after inspection,
the condition of the machine is noted and classified into one of four possible states:

State Condition
0 Good as new
1 Operable - minimum deterioration
2 Operable - major deterioration
3 Inoperable and replaced by a good-as-new machine

The process can be modeled as a Markov chain with its (one-step) transition matrix P given by

State 0 1 2 3
0 0 7/8 1/16 1/16
1 0 3/4 1/8 1/8
2 0 0 1/2 1/2
3 1 0 0 0

(a) Find the steady-state probabilities.

(b) If the costs of being in states 0, 1, 2, 3, are 0, $1,000, $3,000, and $6,000, respectively, what is the
long-run expected average cost per day?

Solution:

a)
𝜋0 = 𝜋3
7 3
𝜋1 = 𝜋0 + 𝜋1
8 4
1 1 1
𝜋2 = 𝜋0 + 𝜋1 + 𝜋2
16 8 2
1 1 1
𝜋3 = 𝜋0 + 𝜋1 + 𝜋2
16 8 2
𝜋0 + 𝜋1 + 𝜋2 + 𝜋3 = 1
7 1 7
==> 𝜋0 = 𝜋1 ==> 𝜋0 = 𝜋1
8 4 2
==> 𝜋2 = 𝜋3
7 2 7 2 2
==> 𝜋0 (1 + 1 + 1 + ) = 1 ==> 𝜋0 = , 𝜋1 = , 𝜋2 = , 𝜋3 =
2 13 13 13 13
b) Long-run average cost
𝐶 = 𝜋0 ∗ 0 + 𝜋1 ∗ 1000 + 𝜋1 ∗ 3000 + 𝜋1 ∗ 6000
2 7 2 2 2500
=0∗ + 1000 ∗ + 3000 ∗ + 6000 ∗ =
13 13 13 13 13

This study source was downloaded by 100000835991620 from CourseHero.com on 04-08-2023 08:56:31 GMT -05:00

https://www.coursehero.com/file/53152347/IE-301-Fall-2019-Recitation12-Solutionspdf/
 8) A camera store stocks a particular model camera that can be ordered weekly. Let D1, D2, … represent
the demand for this camera (the number of units that would be sold if the inventory is not depleted)
during the first week, second week, …, respectively. It is assumed that the Di’s are independent and
identically distributed random variables having a Poisson distribution with a mean of 1. Let X0 represent
the number of cameras on hand at the outset, X1 the number of cameras on hand at the end of week 1,
X2 the number of cameras on hand at the end of week 2, and so on.

– Assume that X0 = 3.

– On Saturday night the store places an order that is delivered in time for the next
opening of the store on Monday.

– The store uses the following order policy: If there are no cameras in stock, 3 cameras are
ordered. Otherwise, no order is placed.

– Sales are lost when demand exceeds the inventory on hand

Cost of ordering z units is 10+25z and there is a penalty of $50 for each unit of demand not satisfied.
Find the expected average cost per week.

Solution:

Define Markov Chain as Xt: # of units at the end of each week (end of week=Saturday night)
max{3 − 𝐷𝑡+1 , 0} 𝑖𝑓 𝑋𝑡 = 0 (𝑏𝑒𝑐𝑎𝑢𝑠𝑒 𝑜𝑓 𝑜𝑟𝑑𝑒𝑟𝑖𝑛𝑔
𝑋𝑡+1 = {
max{3 − 𝐷𝑡+1 , 0} 𝑖𝑓 𝑋𝑡 > 0
State Space = {0,1,2,3}

Now let’s write the events that needs to happen for each transaction. For instance, if at the end of the
week, there are no items in the inventory, we will place an order for 3 units. We will start Monday with
3 units. For us to have 0 units at the end of next week, the demand has to be at least 3.

0 1 2 3
P= 0 P(Dt+1≥3) P(Dt+1=2) P(Dt+1=1) P(Dt+1=0)
1 P(Dt+1≥1) P(Dt+1=0) 0 0
2 P(Dt+1≥2) P(Dt+1=1) P(Dt+1=0) 0
3 P(Dt+1≥3) P(Dt+1=2) P(Dt+1=1) P(Dt+1=0)

Since the demand has Poisson distribution with mean 1, we can compute these probabilities as follows:

(1)𝑛 𝑒 −1
𝑃(𝐷𝑡+1 = 𝑛) =
𝑛!
𝑃(𝐷𝑡+1 = 0) = 𝑒 −1 = 0.368
𝑃(𝐷𝑡+1 = 1) = 𝑒 −1 = 0.368
(1)𝑒 −1
𝑃(𝐷𝑡+1 = 2) = = 0.184
2!
𝑃(𝐷𝑡+1 ≥ 3) = 1 − 𝑃(𝐷𝑡+1 ≤ 2) = 1 − 0.368 − 0.368 − 0.184 = 0.080
𝑃(𝐷𝑡+1 ≥ 2) = 1 − 𝑃(𝐷𝑡+1 ≤ 1) = 1 − 0.368 − 0.368 = 0.264
𝑃(𝐷𝑡+1 ≥ 1) = 1 − (𝐷𝑡+1 = 0) = 1 − 0.368 = 0.632

This study source was downloaded by 100000835991620 from CourseHero.com on 04-08-2023 08:56:31 GMT -05:00

https://www.coursehero.com/file/53152347/IE-301-Fall-2019-Recitation12-Solutionspdf/
 0 1 2 3
P= 0 0.080 0.184 0.368 0.368
1 0.632 0.368 0 0
2 0.264 0.368 0.368 0
3 0.080 0.184 0.368 0.368

Then we can compute the steady state probabilities as follows:

 0 = 0.080 0 + 0.632 1 + 0.264 2 + 0.080 3

 1 = 0.184 0 + 0.368 1 + 0.368 2 + 0.184 3
 2 = 0.368 0 + 0.368 2 + 0.368 3
 3 = 0.368 0 + 0.368 3
1 =  0 + 1 +  2 +  3

 0 = 0.286  1 = 0.285  2 = 0.263  3 = 0.166

Now we need to calculate the expected cost of each state.

Expected Cost(state 0) = E[ordering cost + lost sales penalty] = 10+3*25 + 50*E[lost sales]

= 85+ 50*max{Dt-3,0}] = 85 + 50*(1*P(Dt=4) + 2*P(Dt=5) + 3*P(Dt=6) + …)

= 85 + 50(0.015 + 2(0.003) + 3(0.001)…) = 86.2

Expected Cost(state 1) = E[lost sales penalty] = 50*E[lost sales] = 50*E[max{Dt-1,0}]

= 50*(1*P(Dt=2) + 2*P(Dt=3) + 3*P(Dt=4) + …) = 18.4

Expected Cost(state 2) = E[lost sales penalty] = 50*E[lost sales] = E[C(2,Dt)] = 50*E[max{Dt-2,0}]

= 50*(1*P(Dt=3) + 2*P(Dt=4) + 3*P(Dt=5) + …) = 5.2

Expected Cost(state 3) = E[lost sales penalty] = 50*E[lost sales]= E[C(3,Dt)] = 50*E[max{Dt-3,0}]

= 50*(1*P(Dt=4) + 2*P(Dt=5) + 3*P(Dt=6) + …) = 1.2

Expected average cost per week: 86.2(0.286) + 18.4(0.285) + 5.2(0.263) + 1.2(0.166) = $31.46

This study source was downloaded by 100000835991620 from CourseHero.com on 04-08-2023 08:56:31 GMT -05:00

https://www.coursehero.com/file/53152347/IE-301-Fall-2019-Recitation12-Solutionspdf/
Powered by TCPDF (www.tcpdf.org)