Homework Solution 7 for APPM4/5560 Markov Processes

8.4 A salesman flies around between Atlanta, Boston, and Chicago as follows. She stays in each city for
an exponential amount of time with mean 1/4 month if the city is A or B, but with mean 1/5 month if the
city is C. From A she goes to B or C with probability 1/2 each; from B she goes to A with probability 3/4
and to C with probability 1/4; from C she always goes back to A. (a) Find the limiting fraction of time she
spends in each city. (b) What is her average number of trips each year from Boston to Atlanta?
(a) The transition rate matrix is

4 2
2
Q = 3 4 1 .
5
0 5
Therefore,

=

1 1 1
, ,
2 4 4


.

(b) On average, she spends 1/4 of the year in Boston (3 months) and leaves for Atlanta at a rate of 3 times
per month. So on average, she will make 9 trips per year from Boston to Atlanta. 8.6 Consider two machines
that are maintained by a single repairman. Machine i functions for an exponentially with rate i . They are
repaired in the order in which they fail. (a) Let Xt be the number of working machines at time t. Is Xt a
Markov chain. (b) Formulate a Markov chain model for this situation with state space {0, 1, 2, 12, 21}. (c)
Suppose that 1 = 1, 1 = 2, 2 = 3, 2 = 4. Find the stationary distribution.
(a) In general, it is not. For example: If Xt = 1, then exactly one of the machines is not working and the
time to repair or failure is either 1 + 2 or 2 + 1 which are not necessarily the same.
(b) Let
0 = Neither machine is broken.
1 = Machine 1 is being repaired.
2 = Machine 2 is being repaired.
12 = Both machines are broken and machine 1 is being repaired.
21 = Both machines are broken and machine 2 is being repaired.
Then the transition rate matrix is

1 2

2
Q=

0
0

1
1 2
0
0
2

2
0
2 1
1
0

0
2
0
1
0

0
0
1
0
2

(c)

0 =

4
2
4
0
0

1
5
0
0
4

3
0
5
2
0

0
3
0
2
0

0
0
1
0
4

= (44/129, 16/129, 12/43, 8/43, 3/43).

8.8 Two people are working in a small office selling shares in a mutual fund. Each is either on the phone
or not. Suppose that salesman i is on the phone for an exponential amount of time with rate i and then
off the phone for an exponential amount of time with rate i . (a) Formulate a Markov chain model for this
system with state space {0, 1, 2, 12} where the state indicates who is on the phone. (b) Find the stationary
distribution. (a) The transition rate matrix would be

1 2
1
2
0

1
1 2
0
2
.
Q=

2
0
2 1
1
0
2
1
2 1
(b) We solve it directly,
= C(1 2 , 1 2 , 2 1 , 1 2 )
C = (1 2 + 1 2 + 2 1 + 1 2 )1
8.3 Consider a pure death process in which i i 1 at rate when i 1. Find the transition probability
pt (i, j).
The number of states which cover (i j) in time t will follow a Poisson(t) distribution up to the point
where we hit the absorbing state 0.

ij
t (t)

0<ji
e
(ij)!
Pi
(t)ij
t
pt (i, j) =
1e
j=0
k=1 (ij)!

0
otherwise
8.9(a) Consider the special case of the previous problem in which 1 = 2 = 1, and 1 = 2 = 3, and find
the stationary probabilities. (b) Suppose they upgrade their telephone system so that a call to one line that
is busy is forwarded to the other phone and lost if that phone is busy. Find the new stationary probabilities.
(a)

2 1
1
0
3 4 0
1
.
Q=
3
0 4 1
0
3
3 6
= (9/16, 3/16, 3/16, 1/16).
(b)

2 1
3 5
Q=
3
0
0
3

1
0
5
3

0
2
.
2
6

= (9/17, 3/17, 3/17, 2/17).

2

8.27Customers arrive at a carnival ride at rate . The ride takes an exponential amount of time with rate
, but when it is in use, the ride is subject to breakdowns at rate . When a breakdown occurs all of
the people leave since they know that the time to fix a breakdown is exponentially distributed with rate .
(a) Formulate a Markov chain model with state space {1, 0, 1, 2, , } where 1 is broken and the states
0, 1, 2, , indicate the number of people waiting for or in service. (b) Show that the chain has a stationary
distribution of the form (1) = a, (n) = bn for n 0.
(a) The transition rate matrix is given by
Q(i, 1)
=
i 0.
Q(1, 0)
= .
Q(i, i + 1) =
i 0.
Q(i, i 1) =
i 1.
Q(1, 1) = .
Q(0, 0)
= .
Q(i, i)
= i 1.
(b)The vector must satisfy

(i) = 1

i=1

(1) =

(i).

j=0

(i) + (i + 2) = ( + + )(i + 1)

i 0.

Assuming (1) = a, and (n) = bn for n 0, it follows from each of these three equations respectively
that
(1 )b = 1 a
(1 )a = b.
+ 2 = ( + + ).

The solution to these equations is unique, given by

=

( + + )

( + + )2 4
.
2

.
(1 )2 +
1a
b=
.
1

a=