4

CHAPTER 1. MARKOV CHAINS

Example 1.6. Inventory chain. We will consider the consequences of using

an s, S inventory control policy. That is, when the stock on hand at the end
of the day falls to s or below, we order enough to bring it back up to S. For
simplicity, we suppose happens at the beginning of the next day. Let Xn be
the amount of stock on hand at the end of day n and Dn+1 be the demand on
day n + 1. Introducing notation for the positive part of a real number,
(
x if x > 0
+
x = max{x, 0} =
0 if x 0
then we can write the chain in general as
(
(Xn Dn+1 )+
Xn+1 =
(S Dn+1 )+

if Xn > s
if Xn s

In words, if Xn > s we order nothing and begin the day with Xn units. If the
demand Dn+1 Xn we end the day with Xn+1 = Xn Dn+1 . If the demand
Dn+1 > Xn we end the day with Xn+1 = 0. If Xn s then we begin the day
with S units, and the reasoning is the same as in the previous case.
Suppose now that an electronics store sells a video game system and uses
an inventory policy with s = 1, S = 5. That is, if at the end of the day, the
number of units they have on hand is 1 or 0, they order enough new units so
their total on hand at the beginning of the next day is 5. If we assume that
for k =
P (Dn+1 = k)

0
.3

1
.4

2
.2

3
.1

then we have the following transition matrix:

0
1
2
3
4
5

0
0
0
.3
.1
0
0

1 2 3 4 5
0 .1 .2 .4 .3
0 .1 .2 .4 .3
.4 .3 0 0 0
.2 .4 .3 0 0
.1 .2 .4 .3 0
0 .1 .2 .4 .3

To explain the entries, we note that when Xn 3 then Xn Dn+1 0. When

Xn+1 = 2 this is almost true but p(2, 0) = P (Dn+1 = 2 or 3). When Xn = 1 or
0 we start the day with 5 units so the end result is the same as when Xn = 5.
In this context we might be interested in:
Q. Suppose we make $12 profit on each unit sold but it costs $2 a day to store
items. What is the long-run profit per day of this inventory policy? How do we
choose s and S to maximize profit?
Example 1.7. Repair chain. A machine has three critical parts that are
subject to failure, but can function as long as two of these parts are working.
When two are broken, they are replaced and the machine is back to working
order the next day. To formulate a Markov chain model we declare its state
space to be the parts that are broken {0, 1, 2, 3, 12, 13, 23}. If we assume that