DDA/STA 4001 Stochastic Processes Fall 2019

J. G. Dai

Homework 5 Solution

1. (a) Analytic answer required here, Matlab not ok. Even though the state space, and
therefore the transition matrix, is infinite, solving for the stationary distribution
is no different. Namely, need to solve

P T π = π.

This results in the system of equations

P0,0 π0 + P1,0 π1 = π0 ,
P0,1 π0 + P1,1 π1 + P2,1 π2 = π1
...,
Pk−1,k πk−1 + Pk,k πk + Pk+1,k πk+1 = πk
...

To solve this system, we will write all of the πk ’s in terms of π0 . I.e. π1 =

π0 (1 − P0,0 )/P1,0 , π2 = . . ..

π0
π1 = 2/3 ∗ π0
π2 = 5/9 ∗ π0
π3 = 5/27 ∗ π0
...
k
πk = 5/3 ∗ π0
...
P∞
Then to find π0 , use the condition that k=0 πk = 1 and the formula for geo-
metric series.
∞
X
πk = π0 + π1 + π2 + π3 + . . . ,
k=0
= π0 ∗ (1 + 2/3 + 5/9 + 5/27 + ...)
= π0 ∗ 5 ∗ (1/5 + 2/15 + 1/9 + 1/27 + ...)
= π0 ∗ 5 ∗ [(1 + 1/3 + 1/9 + 1/27 + ...) − 1]
X∞
= π0 ∗ 5 ∗ (1/3)k − π0 ∗ 5
k=0
= π0 ∗ 5 ∗ /(1 − 1/3) − 5π0
= 5/2 ∗ π0

1
 Hence,π0 =2/5, and π= 2/5 4/15 2/9 2/27 2/81 .... 2/3k ...

***Note: Cutting method works well for this case: for each ’cut’ (partition)
(A, Ac ), the flow across A to Ac is equal to the flow across Ac to A. Therefore,
by setting up equations such as
X
π0 ∗ P (leaving 0) = πj ∗ P (entering 0)
(j entering 0)

π0 ∗ (P0,1 + P0,2 ) = π1 ∗ P1,0

....

We can get the same result as shown above.

(b) Yes, because DTMC is irreducible; any state is accessible from any other state.
(c) Waiting “at least one time slot” does not count the timeslot of the packet’s
arrival. Let π = (π0 , π1 , . . .), be the stationary distribution, where πk is the
long-run fraction of time there are k packets in the system. Answer is
∞
1 X 2 4 1 1 1
(π0 + π1 ) P(A0 = 2|A0 > 0) + πk = ( + ) + =
2 5 15 4 3 2
k=2

Interpretation: if packets arrive during a timeslot with ≥ 2 packets already in the

system, then 100% of them will have to wait. If packets arrive during a timeslot
with 0 or 1 packets already in the system, then the first of the new packets won’t
have to wait, but the rest will.
(d) Treat the stationary distribution π = (π0 , π1 , . . .) just like a regular probability
distribution on the non-negative integers. The long-run expected number of
packages in the system equals to
∞
X
πk ∗ Xn
k=0
= π0 ∗ 0 + π1 ∗ 1 + π2 ∗ 2 + . . .
= 0 + (2/3 − 6/15) + 2/9 ∗ 2 + 2/27 ∗ 3 + ...
∞
X
= (−6/15) + 2 ∗ (1/3)k ∗ k
k=0
∞
X
= (−6/15) + 3 ∗ (1 − 2/3)k ∗ 2/3 ∗ k
k=0

We can treat
P the sum part as a geometric random variable
P∞with p=2/3.Using
formula, ∞k=0 (1 − 2/3)k ∗ 2/3 ∗ k=(1-p)/p=1/2. Hence,
k=0 πk ∗ Xn = 1.1
(e) T2,2 = 1/π2 =9/2. More generally, Ti,i = 1/πi for any i = 0, 1, 2, . . ..

2
2. (a) Let Dn be the demand experienced during the week between the (n − 1)st and
nth Saturdays. The DTMC describing Xn , the number of rings in inventory on
the morning of the nth Saturday, can be written as follows:
Xn+1 =(3 − Xn )+ +(5 − Dn+1 ) for Xn = 0,1,2
Since Xn is less than three, you will place a standard shipping order to arrive
next Friday, before you review inventory for the (n + 1)th week. You will also
place an express order to have five units on hand Monday morning before your
inventory levels are decreased by demand.
Xn+1 =(5 − Dn+1 ) for Xn = 3,4
You will not place a standard shipping order because Xn is three or greater.
However, you will place an express order to have 5 units on hand on Monday,
from which inventory levels will decrease.
Xn+1 =(Xn − Dn+1 ) for Xn ≥ 5
Here, you will not place any standard or express orders, and your inventory level
in the following week will be the nth week inventory decreased by the number of
units of demand seen.
Hence, we can write
Xn+1 = f (Xn , Dn+1 ), n≥0
where
(
(3 − x)+ + (5 − d), x ≤ 5,
f (x, d) =
5 − d, x > 5,
{Dn } is an identically distributed sequence of random variables and Dn+1 is
independent of Xn . This proves that {Xn } is a DTMC.
(b) If you have zero units of inventory on the nth Saturday morning, you will order
five units for Monday via express shipping and three units for Friday night via
standard shipping. In the case where you have zero demand over the five days,
you will have eight units in inventory on the (n + 1)th Saturday. Since no orders
will be placed that Saturday, this is the maximum number of units that can be
held in inventory. Thus, the state space is {0, 1, 2, 3, 4, 5, 6, 7, 8}. The transition
matrix is given by:
0 0 0 1/6 1/6 1/6 1/6 1/6 1/6
 
 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 
 
 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0 
 
1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 
 
P =1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 

1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 
 
 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0 
 
 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 
0 0 0 1/6 1/6 1/6 1/6 1/6 1/6

3
(c) If you were calculating the stationary distribution by hand, the pi method would
be used. When calculating the stationary distribution, however, it is sufficient
to calculate Pn for a very large n. In the following Matlab code, several large n
values were used to ensure that the stationary distribution had been reached:
1 P = [ 0 0 0 1/6 1/6 1/6 1/6 1/6 1/6 ;
2 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 ;
3 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0 ;
4 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 ;
5 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 ;
6 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 ;
7 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0 ;
8 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 ;
9 0 0 0 1/6 1/6 1/6 1/6 1/6 1 / 6 ] ;
10 P200 = Pˆ ( 2 0 0 ) ;
11 p i v e c = P200 ( 1 , : ) ;
From this, the stationary distribution is found to be [0.0833, 0.1222, 0.1500,
0.1667, 0.1667, 0.1667, 0.0833, 0.0444, 0.0167].
(d) Suppose Xn = x. Then the expected cashflow breakdown for the following week
is
1. −15 × (3 − x)+ for an order using standard shipping
2. −35 × (5 − x)+ for an express order
3. −10 × E(max(5, x) − Dn+1 )+ for cleaning at the end of the sales day on the
(n + 1)st Friday. Note that new inventory arriving on that Friday doesn’t
require cleaning.
4. 100 × EDn+1 = 250 for sales made (since we always start the week with ≥ 5
rings, there are no lost sales).
Since demand is at most 5, it follows that

E(max(5, x) − Dn+1 )+ = E(max(5, x) − Dn+1 ) = max(5, x) − 2.5

Let f (x) be the expected profit for the n + 1st week, given that Xn = x. Then

f (x) = −15(3 − x)+ − 35(5 − x)+ − 10(max(5, x) − 2.5) + 250

or

 −220 + 50x − 25 + 250, x ≤ 3,

f (x) = −175 + 35x − 25 + 250, x = 4, 5,

−10(x − 2.5) + 250, x = 6, 7, 8.


P8
Then find the long run average profit using the stationary distribution by k=0 πk f (k).
Use Matlab to find this value as follows:

4
 1 P = [ 0 0 0 1/6 1/6 1/6 1/6 1/6 1/6 ;
2 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 ;
3 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0 ;
4 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 ;
5 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 ;
6 1/6 1/6 1/6 1/6 1/6 1/6 0 0 0 ;
7 0 1/6 1/6 1/6 1/6 1/6 1/6 0 0 ;
8 0 0 1/6 1/6 1/6 1/6 1/6 1/6 0 ;
9 0 0 0 1/6 1/6 1/6 1/6 1/6 1 / 6 ] ;
10 P200 = Pˆ ( 2 0 0 ) ;
11 p i v e c = P200 ( 1 , : ) ;
12 f = [ 5 55 105 155 190 225 215 205 1 9 5 ] ;
13 sum ( p i v e c . ∗ f )
We get that the long run average profit for the week is $148.17.

3. (a) Let Dn be the demand experienced during the week between the (n−1)st and nth
(1) (2)
Saturdays. To have a DTMC, we need to keep track of the triple (Xn , Rn , Rn ).
The one-step transition obeys the following dynamics. Given that we observe Xn
units on the nth Saturday, we start out the following Monday with Xn units of
inventory, and a week’s worth of sales, we end up with (Xn − Dn+1 )+ units.
On Friday evening, we get to shipments. The rustic shipment from 3 weeks ago
(1)
arrives, adding Rn units, and the standard shipment from last week arrives,
(1)
adding (5 − Xn − Rn )+ units. Therefore,
(1) (1)
Xn+1 =(Xn − Dn+1 )+ + (5 − Xn − Rn )+ + Rn .
Furthermore,
(1) (2)
Rn+1 =Rn
This is because one week has elapsed, so that the order that had been two weeks
out from shipping is now only one week out. Lastly,
(2)
Rn+1 =(5 − Xn )+ ,
because a rustic order placed in the nth week is two weeks out from delivery in
the (n + 1)st week.
For this DTMC, the state space is

S = {(x, r1 , r2 ) : x = {0, 1, 2 . . .}, r1 ∈ {0, 1, 2, 3, 4, 5}, r2 ∈ {0, 1, 2, 3, 4, 5}},

although observe that the states corresponding to x > 10 are transient, because
once we have 10 units of inventory (or less), there is no way to increase the
inventory level beyond 10.

5
 (1) (2)
To describe the transition probability matrix, suppose (Xn , Rn , Rn ) = (x, r1 , r2 ).
Based on the above description, the only possible states at time n + 1 are
 
(x, r1 , r2 ) → (x − 0)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+
 
(x, r1 , r2 ) → (x − 1)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+
 
(x, r1 , r2 ) → (x − 2)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+
 
(x, r1 , r2 ) → (x − 3)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+
 
(x, r1 , r2 ) → (x − 4)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+
 
(x, r1 , r2 ) → (x − 5)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+ ,

and based on the distribution of Dn+1 , each of the above transitions is equally
likely. Hence,
  
1 + + (5 − x − r )+ + r , r , (5 − x)+ ,


 6 , (x̃, r̃1 , r̃2 ) = (x − 0) 1 1 2
  
1 + + (5 − x − r )+ + r , r , (5 − x)+ ,
−




 6 , (x̃, r̃1 , r̃2 ) = (x 1) 1 1 2
  
 1 , (x̃, r̃ , r̃ ) = (x − 2)+ + (5 − x − r )+ + r , r , (5 − x)+ ,

6

 1 2 1 1 2
 
P(x,r1 ,r2 ),(x̃,r̃1 ,r̃2 ) = 61 , (x̃, r̃1 , r̃2 ) = (x − 3)+ + (5 − x − r1 )+ + r1 , r2 , (5 − x)+ ,

  
1 + + +
, (x̃, r̃1 , r̃2 ) = (x − 4) + (5 − x − r1 ) + r1 , r2 , (5 − x) ,


6


  
1 + + (5 − x − r )+ + r , r , (5 − x)+ ,
−




 6 , (x̃, r̃ ,
1 2 r̃ ) = (x 5) 1 1 2

0, otherwise.


(1) (2)
(b) Suppose (Xn , Rn , Rn ) = (x, r1 , r2 ). Then the weekly cashflow associated with
that state is
1. −15 × (5 − x − r1 )+ for an order using standard shipping
2. −5 × (5 − x)+ for a rustic order
3. −10 × E(x − Dn+1 )+ for cleaning at the end of the sales day on the (n + 1)st
Friday. Note that new inventory arriving on that Friday doesn’t require
cleaning.
4. 100 × E(min(x, Dn+1 )) for sales made.
The expected profit is then

f (x, r1 , r2 ) = −15 × (5 − x − r1 )+ − 5 × (5 − x)+ − 10E(x − Dn+1 )+ + 100E(min(x, Dn+1 ))

The following Matlab simulation can be used to find the long-run cost:
1 x = 0;
2 r1 = 0 ;
3 r2 = 0 ;

6
 4 n = 10ˆ6;
5 costs = zeros (1 ,n) ;
6 for i = 1:n
7 d = unidrnd ( 6 ) − 1 ;%demand
8 r 2 = max(5−x , 0 ) ;
9 r1 = r2 ;
10 x = max( x − d , 0 ) + max(5−x−r1 , 0 ) + r 1 ;
11 c o s t s ( i ) = −15∗max(5−x−r1 , 0 ) − 5∗max(5−x , 0 ) −10∗max( x−
d , 0 ) + 100∗ min ( x , d ) ;
12 end
13 mean ( c o s t s )
and we arrive at the long-run profit of approximately $115.02.
4. Note that

{Ti < ∞} = ∪∞
n=1 {Xn = i}.

Since i is assumed to be recurrent, Pi (Ti < ∞) = 1. Since i → j, there exists k > 0

such that Pijk > 0. Since i is recurrent, starting from state i, with probability one, the
DTMC will visit state i infinitely often. Therefore

Pi (∪`=k+1 {X` = i}) = 1. (0.1)

Now, using the fact that

X X
Pi (A) = Pi ({Xk = m} ∩ A) = Pi ({Xk = m})P{∩A|Xk = m)
m∈S m∈S
X
k
= Pim P{∩A|Xk = m)
m∈S

for any event A, one has

X
k
Pi (∪`=k+1 {X` = i}) = Pim Pi (∪`=k+1 {X` = i}|Xk = m)
m∈S
X
k
= Pim Pi (∪`=1 {X` = i}|X0 = m)
m∈S
X
k
= Pim Pm (Ti < ∞)
m∈S
X
= Pijk Pj (Ti < ∞) + k
Pim Pm (Ti < ∞)
m∈S\{j}

From (0.1), one has

X
Pijk Pj (Ti < ∞) + k
Pim Pm (Ti < ∞) = 1. (0.2)
m∈S\{j}

7
 k = 1, and each term P (T < ∞) ≤ 1. Thus,
P
Since m∈S Pim m i
X
Pijk Pj (Ti < ∞) + k
Pim Pm (Ti < ∞) ≤ 1. (0.3)
m∈S\{j}

Since Pijk > 0, if Pj (Ti < ∞) < 1, the inequality in (0.3) will be strict, contradicting
(0.2). Therefore, Pj (Ti < ∞) = 1, proving all the statements in the problem.