CHAPTER 16: MARKOV CHAINS

16.2-1.
In this study, Markov chains are used to model the changes in credit ratings of
corporations that work with Merrill Lynch Bank USA. The bank manages a portfolio of
revolving credit-line commitments worth billions of dollars. A corporation that has a
credit line can withdraw a significant amount of money from the bank on short notice.
The risk associated with the bank's ability to meet these cash requests is referred to as the
liquidity risk. Merrill Lynch developed a model to assess this risk and to evaluate various
scenarios like financial stress. The model consists of a mix of multiple OR techniques.
The core of the model is a Monte Carlo simulation of revolving credit lines. In doing this,
the monthly changes in credit ratings for each company are modeled as a discrete-time
Markov chain. A company's rating in a month is assumed to depend only on its rating in
the previous month and the transition probabilities used in forming the credit-migration
matrix are assumed to be stationary.
The model provided Merrill Lynch Bank a systematic way to measure and to manage the
liquidity risk. After the implementation of the model, required liquidity reserves have
been decreased by 30% and $4 billion that is freed up consequently can now be used in
more profitable investments. During the first 21 months, the bank's portfolio has
increased from $8 billion to $13 billion and from 80 companies to 100. The evaluation of
different scenarios enabled the bank to ensure liquidity even during financial crises. The
basic model is now run once every month and is also used in long-term planning.
16.2-2.
(a) Since the probability of rain tomorrow is only dependent on the weather today,
Markovian property holds for the evolution of the weather.
(b) Let the two states be ! œ Rain and " œ No Rain. Then the transition matrix is
P œ PÐ"Ñ œ ˆ !Þ&
!Þ"
!Þ& ‰
!Þ* .

16.2-3.
(a) Let " œ increased today and yesterday,
# œ increased today and decreased yesterday,
$ œ decreased today and increased yesterday,
% œ decreased today and yesterday.

Î !" ! Ñ
Ð ! ! Ó
! "  !"
œÐ # Ó
! "  !#
P œ PÐ"Ñ
Ï ! "  !% Ò
! !$ ! "  !$
!% !
(b) The state space is properly defined to include information about changes yesterday
and today. This is the only information needed to determine the next state, namely
changes today and tomorrow.

16-1
16.2-4.
Yes, it can be formulated as a Markov chain with the following ) Ð œ #$ Ñ states.
State Today 1 Day Ago 2 Days Ago
1 inc inc inc
2 inc inc dec
3 inc dec inc
4 inc dec dec
5 dec inc inc
6 dec inc dec
7 dec dec inc
8 dec dec dec
These states include all the information needed to predict the change in the stock
tomorrow whereas the states in Prob. 16.2-2 do not consider the day before yesterday, so
they do not contain all necessary information to predict the change tomorrow.
16.3-1.
(a)

PÐ#Ñ œ Œ
!Þ)' 
PÐ&Ñ œ Œ
!Þ"'& !Þ)$& 
!Þ$ !Þ( !Þ"(& !Þ)#&
!Þ"%

PÐ"!Ñ œ Œ
!Þ)$$ 
PÐ#!Ñ œ Œ
!Þ)$$ 
!Þ"'( !Þ)$$ !Þ"'( !Þ)$$
!Þ"'( !Þ"'(
(b)
Ð8Ñ
PÐRain 8 days from now | Rain todayÑ œ P""
Ð8Ñ
PÐRain 8 days from now | No rain todayÑ œ P#" .
If the probability it will rain today is !Þ&,
Ð8Ñ Ð8Ñ
PÐRain 8 days from nowÑ œ :8 œ !Þ&P""  !Þ&P#" .
Hence, :# œ !Þ##, :& œ !Þ"(, :"! œ !Þ"'(, :#! œ !Þ"'(.
(c) We find 1" œ !Þ"'( and 1# œ !Þ)$$. As 8 grows large, PÐ8Ñ approaches

Œ1 1# 
1" 1#
,
"

the stationary probabilities. Indeed,

PÐ"!Ñ œ PÐ#!Ñ œ Œ "

1# 
1 1#
.
1"

16-2
16.3-2.
(a) Let states ! and " denote that a ! and a " have been recorded respectively. Then the
transition matrix is

PœŒ
";
"; ;
ß
;
where ; œ !Þ!!&.
(b)

PÐ"!Ñ œ Œ
!Þ*&# 
!Þ*&# !Þ!%)
!Þ!%)
The probability that a digit will be recorded accurately after the last transmission is
!Þ*&#.
(c)

PÐ"!Ñ œ Œ
!Þ*) 
!Þ*) !Þ!#
!Þ!#
The probability that a digit will be recorded accurately after the last transmission is !Þ*).
16.3-3.
(a)

Î ! !Þ& ! ! !Þ& Ñ
Ð !Þ& ! !Þ& ! ! Ó
Ð Ó
P œ Ð ! !Þ& ! !Þ& ! Ó
Ð Ó
.

Ï !Þ& ! ! !Þ& ! Ò
! ! !Þ& ! !Þ&

(b)

16-3
(c) 1" œ 1# œ 1$ œ 1% œ 1& œ !Þ#.
16.4-1.
(a) P has one recurrent communicating class: Ö!ß "ß #ß $×.
(b) P has 3 communicating classes: Ö!× absorbing, so recurrent; Ö"ß #× recurrent and Ö$×
transient.
16.4-2.
(a) P has one recurrent communicating class: Ö!ß "ß #ß $×.
(b) P has one recurrent communicating class: Ö!ß "ß #×.
16.4-3.
P has 3 communicating classes: Ö!ß "× recurrent, Ö#× transient and Ö$ß %× recurrent.
16.4-4.
P has one communicating class, so each state has the same period %.
16.4-5.
(a) P has two classes: Ö!ß "ß #ß %× transient and Ö$× recurrent.
(b) The period of Ö!ß "ß #ß %× is # and the period of Ö$× is ".
16.5-1.

PœŒ
" 
! "!
""
1P œ 1 Ê !1"  Ð"  " Ñ1# œ 1" and 1"  1# œ "

Ê 1 œ Š #"!" " ß #"!

!
" ‹.

16-4
16.5-2.
We need to show that 14 œ Q"" for 4 œ !ß "ß á ß Q satisfies the steady-state equations:
14 œ !Q 3œ! 13 T34 and
!Q
3œ! 13 œ ". These are easily verified, using
!Q
3œ! T34 œ " for every
4. The chain is irreducible, aperiodic and positive recurrent , so this is the unique
solution.
16.5-3.
Q œ & Ê 1" œ 1# œ 1$ œ 1% œ 1& œ "Î& œ !Þ#
The steady-state probabilities do not change if the probabilities for moving steps change.
16.5-4.
1 œ Ð!Þ&""ß !Þ#)*ß !Þ#Ñ
The steady-state market share for A and B are !Þ&"" and !Þ#)* respectively.
16.5-5.
(a) Assuming demand occurs after delivery of orders:

Î !Þ' !Þ% ! ! Ñ
Ð !Þ$ !Þ$ !Þ% ! ! Ó
! ! !
Ð Ó
Ð !Þ" !Þ# !Þ$ !Þ% ! ! Ó
! !
Ð Ó
P œ Ð ! !Þ" !Þ# !Þ$ !Þ% ! ! Ó
!
Ð Ó
Ð ! ! !Þ" !Þ# !Þ$ !Þ% ! Ó
Ð Ó
Ï ! ! !Þ" !Þ# !Þ( Ò
! ! ! !Þ" !Þ# !Þ$ !Þ%
! !

(b) 1P œ 1 and !4 14 œ " Ê 1 œ Ð!Þ"$* !Þ"$* !Þ"$* !Þ"$) !Þ"%" !Þ"$! !Þ"(%Ñ.
(c) The steady-state probability that a pint of blood is to be discarded is
PÐH œ !Ñ † PÐstate œ (Ñ œ !Þ% x !Þ"(% œ !Þ!'*'.
(d) PÐneed for emergency deliveryÑ œ !#3œ" PÐstate œ 3Ñ † PÐH  3Ñ
œ !Þ"$* x Ð!Þ#  !Þ"Ñ  !Þ"$* x !Þ"
œ !Þ!&&'
16.5-6.
For an Ð=ß WÑ policy with = œ # and W œ $:

-ÐB>" ß H> Ñ œ œ
"!  #&Ð$  B>" Ñ  &!maxÐH>  $ß !Ñ for B>"  #
&!maxÐH>  B>" ß !Ñ for B>"   #.

OÐ!Ñ œ IÒ-Ð!ß H> ÑÓ œ )&  &!Ò!_

4œ% Ð4  $Ñ † T ÐH> œ 4ÑÓ ¶ )'Þ#,

OÐ"Ñ œ IÒ-Ð"ß H> ÑÓ œ '!  &!Ò!_

4œ% Ð4  $Ñ † T ÐH> œ 4ÑÓ ¶ '"Þ#,

16-5
OÐ#Ñ œ IÒ-Ð#ß H> ÑÓ œ !  &!Ò!_
4œ% Ð4  #Ñ † T ÐH> œ 4ÑÓ ¶ &Þ#,

OÐ$Ñ œ IÒ-Ð$ß H> ÑÓ œ !  &!Ò!_

4œ% Ð4  #Ñ † T ÐH> œ 4ÑÓ ¶ "Þ#.

B>" œ œ
maxÐ$  H>" ß !Ñ for B>  #
maxÐB>  H>" ß !Ñ for B>   #

Î !Þ!)! !Þ$') Ñ
Ð !Þ!)! !Þ$') Ó
!Þ")% !Þ$')
PœÐ Ó
!Þ")% !Þ$')

Ï !Þ!)! !Þ$') Ò
!Þ#'% !Þ$') !Þ$') !
!Þ")% !Þ$')

Then the long-run average cost per week is !$4œ! OÐ4Ñ † 14 œ $!Þ$(.
Solving the steady-state equations gives Ð1! ß 1" ß 1# ß 1$ Ñ œ Ð!Þ"%)ß !Þ#&#ß !Þ$')ß !Þ#$#Ñ.

16.5-7.
(a)

B>" œ œ
maxÐB>  #  H>" ß !Ñ for B> Ÿ "
maxÐB>  H>" ß !Ñ for B>   #

Î !Þ#'% ! Ñ
Ð !Þ!)! !Þ$') Ó
!Þ$') !Þ$')
PœÐ Ó
!Þ")% !Þ$')

Ï !Þ!)! !Þ$') Ò
!Þ#'% !Þ$') !Þ$') !
!Þ")% !Þ$')

Solving the steady-state equations gives Ð1! ß 1" ß 1# ß 1$ Ñ œ Ð!Þ")#ß !Þ#)&ß !Þ$')ß !Þ"'&Ñ.

(b) lim8Ä_ I ˆ 8" !8>œ" -ÐB> Ñ‹ œ ! † 1!  # † 1"  ) † 1#  ") † 1$ œ 'Þ%).

16.5-8.
(a) P"" œ PÐH8" œ !Ñ  PÐH8" œ #Ñ  PÐH8" œ %Ñ œ $Î&
P"# œ PÐH8" œ "Ñ  PÐH8" œ $Ñ œ #Î&
P#" œ PÐH8" œ "Ñ  PÐH8" œ $Ñ œ #Î&
P## œ PÐH8" œ !Ñ  PÐH8" œ #Ñ  PÐH8" œ %Ñ œ $Î&

PœŒ
$Î& 
$Î& #Î&
#Î&
(b) 1 œ 1P and 1"  1# œ " Ê 1" œ 1# œ "Î#.
(c) P is doubly stochastic and there are two states, so 1" œ 1# œ "Î#.
(d) OÐ"Ñ œ IÒ-Ð"ß H8 ÑÓ
œ Ð#Î&ÑÒ$  #Ð"ÑÓ  Ð#Î&ÑÒ$  #Ð#ÑÓ  Ð"Î&ÑÐ"Ñ  Ð%Î&ÑÒ"  #  $Ó
œ *Þ),

16-6
 OÐ#Ñ œ IÒ-Ð#ß H8 ÑÓ
œ Ð#Î&ÑÒ$  #Ð"ÑÓ  Ð"Î&ÑÒ$  #Ð#ÑÓ  Ð"Î&ÑÐ#  "Ñ  Ð%Î&ÑÒ"  #Ó
œ 'Þ%.
So the long-run average cost per unit time is *Þ)Ð"Î#Ñ  'Þ%Ð"Î#Ñ œ )Þ".
16.5-9.
Ð8Ñ
(a) PÐthe unit will be inoperable after 8 periodsÑ œ P!#

Ð8Ñ Ð8Ñ
8 œ #: P!# œ !Þ!%; 8 œ &: P!# œ !Þ!$(;
Ð8Ñ Ð8Ñ
8 œ "!: P!# œ !Þ!$*; 8 œ #!: P!# œ !Þ!$).
(b) 1! œ !Þ'"&, 1" œ !Þ"*#, 1# œ !Þ!$), and 1$ œ !Þ"&%.
(c) Long-run average cost per period is $!ß !!!1$ œ %ß '#!.
16.6-1.
(a)

PœŒ
!Þ&! 
!Þ*& !Þ!&
!Þ&!
(b) .!! œ "  !Þ!&."!
.!" œ "  !Þ*&.!"
."! œ "  !Þ&!."!
."" œ "  !Þ&!.!"
Ê .!! œ "Þ"ß .!" œ #!ß ."! œ #ß ."" œ ""
16.6-2.
(a) States: ! œ Operational, " œ Down, # œ Repaired.

Î !Þ* !Þ" ! Ñ

Ï !Þ* !Þ" ! Ò
Pœ ! ! "

16-7
(b) We need to solve .34 œ "  !5Á4 P35 .54 for every 3 and 4.
.!! œ "  !Þ"."!
."! œ "  .#!
.#! œ "  !Þ"."!
Ê .!! œ ""Î*ß ."! œ #!Î*ß .#! œ ""Î*
.!" œ "  !Þ*.!"
."" œ "  .#"
.#" œ "  !Þ*.!"
Ê .!" œ "!ß ."" œ ""ß .#" œ "!
.!# œ "  !Þ*.!#  !Þ"."#
."# œ "  !
.## œ "  !Þ*.!#  !Þ"."#
Ê .!# œ ""ß ."# œ "ß .## œ ""
The expected number of full days that the machine will remain operational before the
next breakdown after a repair is completed is .!" œ "!.
(c) It remains the same because of the Markovian property. The expected number of days
the machine will remain operational starting operational does not depend on how long the
machine remained operational in the past.
16.6-3.
(a) We order the states as Ð"ß "Ñ, Ð!ß "Ñ and Ð"ß !Ñ and write the transition matrix:

Î !Þ* !Þ" ! Ñ

Ï !Þ* !Þ" ! Ò
P œ !Þ* ! !Þ" .

(b) .$$ œ "Î1$ . From 1 œ 1P and 1 † " œ ", we get 1$ œ "Î""!, so the expected
recurrence time for the state Ð"ß !Ñ is .$$ œ ""!.
16.6-4.

(a)

Î !Þ#& !Þ& !Þ#& Ñ

Ï !Þ#& !Þ& !Þ#& Ò

P œ !Þ(& !Þ#& !

(b)

Î !Þ& !Þ$(& !Þ"#& Ñ

Ï !Þ& !Þ$(& !Þ"#& Ò

PÐ#Ñ œ !Þ$(& !Þ%$) !Þ"))

16-8
 Î !Þ%%* !Þ% !Þ"& Ñ

Ï !Þ%%* !Þ"& Ò
Ð&Ñ
P œ !Þ%&" !Þ$** !Þ"%*
!Þ%

Î !Þ%& !Þ% !Þ"& Ñ

Ï !Þ%& !Þ"& Ò
Ð"!Ñ
P œ !Þ%& !Þ% !Þ"&
!Þ%
(c) .!! œ "  !Þ&."!  !Þ#&.#!
."! œ "  !Þ#&."!
.#! œ "  !Þ&."!  !Þ#&.#!
Ê .!! œ #!Î*ß ."! œ %Î$ß .#! œ #!Î*
.!" œ "  !Þ#&.!"  !Þ#&.#"
."" œ "  !Þ(&.!"
.#" œ "  !Þ#&.!"  !Þ#&.#"
Ê .!" œ #ß ."" œ # "# ß .#" œ #
.!# œ "  !Þ#&.!#  !Þ&."#
."# œ "  !Þ(&.!#  !Þ#&."#
.## œ "  !Þ#&.!#  !Þ&."#
Ê .!# œ #!Î$ß ."# œ )ß .## œ #!Î$
(d) The steady-state probability vector is Ð!Þ%& !Þ% !Þ"&Ñ.
(e) 1 † G œ !Ð!Þ%&Ñ  #Ð!Þ%Ñ  )Ð!Þ"&Ñ œ $ # Î week
16.6-5.
(a)

Î! !Þ)(& !Þ!'# !Þ!'# Ñ

Ð ! !Þ(& !Þ"#& !Þ"#& Ó
PœÐ Ó
Ï" ! Ò
! ! !Þ& !Þ&
! !
1! œ !Þ"&%, 1" œ !Þ&$), 1# œ !Þ"&%, and 1$ œ !Þ"&%
(b) 1 † G œ "Ð!Þ&$)Ñ  $Ð!Þ"&%Ñ  'Ð!Þ"&%Ñ œ $ "*#$Þ!)
(c) .!! œ "  !Þ)(&."!  !Þ!'#&.#!  !Þ!'#&.$!
."! œ "  !Þ(&."!  !Þ"#&.#!  !Þ"#&.$!
.#! œ "  !Þ&.#!  !Þ&.$!
.$! œ"!
So the expected recurrence time for state ! is .!! œ 'Þ&.

16-9
16.7-1.
(a) P!! œ PX X œ "; P3ß3" œ ; ; P3ß3" œ :; P3ß5 œ ! else.

Î" ! ! Ñ
Ð ; ! : Ó
! â
Ð Ó
Ð Ó
! â
PœÐ Ó
Ð !Ó
ã ä
Ð Ó
; ! :

Ï "Ò
! ; ! :
! ! !

(b) Class 1: Ö!× absorbing

Class 2: ÖX × absorbing
Class 3: Ö"ß #ß á ß X  "× transient
(c) Let 03O œ PÐabsorption at O starting at 3Ñ. Then 0!! œ 0$$ œ ", 0$! œ 0!$ œ !.
Since P34 œ ! for l3  4l Á " and P3ß3" œ :, P3ß3" œ ; , we get:
0"! œ ;  :0#!
0"$ œ "  0"!
0#! œ ;0"!
0#$ œ "  0#!
Solving this system gives
;
0"! œ ":; œ !Þ))', 0"$ œ !Þ""%, 0#! œ !Þ'#, 0#$ œ !Þ$).
(d) Plugging in : œ !Þ( in the formulas in part (c), we obtain
0"! œ !Þ$), 0"$ œ !Þ'#, 0#! œ !Þ""%, 0#$ œ !Þ))'.
Observe that when :  "Î#, the drift is towards X and when :  "Î#, it is towards !.
16.7-2.
(a) ! œ Have to honor warranty
" œ Reorder in 1st year
# œ Reorder in 2nd year
$ œ Reorder in 3rd year

Î " ! Ñ
Ð !Þ!" ! ! Ó
! !
PœÐ Ó
!Þ**

Ï ! " Ò
!Þ!& ! ! !Þ*&
! !
(b) The probability that the manufacturer has to honor the warranty is 0"!.
0"! œ !Þ!"0!!  !0"!  !Þ**0#!  !0$!
0#! œ !Þ!&0!!  !0"!  !0#!  !Þ*&0$!
0!! œ " and 0$! œ !
Ê 0"! œ !Þ!"  !Þ**0#! and 0#! œ !Þ!&
Ê 0"! œ !Þ!&*& œ &Þ*&%.

16-10
16.8-1.
In 1998, the new management of PSA Peugeot Citroën set new goals regarding the
amount of production, the introduction of new models and the profit in the following
years. To achieve these goals, the car-body shops, which were the bottlenecks of
production, needed to be redesigned. A new architecture was needed to process diverse
models on the same platform and to introduce new models quickly. To evaluate the
performance of various designs, PSA adopted a combination of simulation and analytic
models. The states of a machine or of a worker in a production line are modeled as a
continuous time Markov chain. A machine can be either up or down. When it is up, it can
go down with the average failure rate -. When it is down, it can be repaired with the
average repair rate .. If the machine can fail only when it is processing a car part, an
additional state is included to represent an idle machine. The operators are also treated as
machines. An operator's state is "up" when he is working regular time and "down" when
he is working overtime.
As a result of this study, a software called DispO is developed. A conservative estimate of
the additional profit generated using this software is $130 million, which is around 6.5%
of the total profit. PSA acquired a 4% increase in productivity in 2001. Additionally, the
new model enabled PSA to understand its assembly lines better and to correct "some
incorrect but deeply ingrained beliefs and practices" [p. 46]. The ability to compare the
estimates obtained from analysis with the actual values convinced the personnel about the
reliability of the methods. As a consequence, OR gained importance throughout the
company. The efficiency in improving production-line designs enhanced throughput
without overloading workers. The social climate and the quality of production are both
improved. The software developed is used also by the suppliers of PSA. This, in turn,
reduces the time to negotiate schedules.
16.8-2.
(a)

(b) Steady-state equations:

$1! œ #1"
%1" œ $1!  #1#
$1# œ #1"  #1$
#1$ œ 1#
1!  1 "  1 #  1 $ œ "

(c) Solving the steady-state equations gives 1 œ Š "* ß "* ß "* ß "* ‹.
% ' ' $

16-11
16.8-3.
(a) Let the state be the number of jobs at the work center.

(b) Steady-state equations:

"
# 1! œ 1"
$
# 1" œ "# 1!  1#
1# œ "# 1"
1!  1 "  1 # œ "
(c) Solving the steady-state equations gives 1 œ Š %( ß #( ß "( ‹.

16-12