Stochastic Processes Solutions Manual

Stochastic Processes: An Introduction
Solutions Manual
Third Edition
c
Peter W Jones and Peter Smith
School of Computing and Mathematics, Keele University, UK

Preface
The website includes answers and solutions of all the end-of-chapter problems in the textbook
Stochastic Processes: An Introduction, third edition. We hope that they will prove helpful to
lecturers in designing courses, and to students as a source of model examples. The original problems
as numbered in the text are also included.
There are obviously references to results and examples from the textbook, and the manual
should be viewed as a supplement to the book. To help identify the sections and chapters, the full
contents of Stochastic Processes follow this preface.
Every effort has been made to eliminate misprints or errors (or worse), and the authors, who
were responsible for the LaTeX code, apologise in advance for any which occur.
Peter W. Jones
Peter Smith Keele, 2017
1
Contents of Stochastic Processes
Chapter 1: Some Background in Probability

1.1 Introduction
1.2 Probability
1.3 Conditional probability and independence
1.4 Discrete random variables
1.5 Continuous random variables
1.6 Mean and variance
1.7 Some standard discrete probability distributions
1.8 Some standard continuous probability distributions
1.9 Generating functions
1.10 Conditional expectation
Problems
Chapter 2: Some Gambling Problems

2.1 Gambler’s ruin
2.2 Probability of ruin
2.3 Some numerical simulations
2.4 Expected duration of the game
2.5 Some variations of gambler’s ruin
2.5.1 The infinitely rich opponent
2.5.2 The generous gambler
2.5.3 Changing the stakes
Problems
Chapter 3: Random Walks

3.1 Introduction
3.2 Unrestricted random walks
3.3 Probability distribution after n steps
3.4 First returns of the symmetric random walk
Problems
Chapter 4: Markov Chains

4.1 States and transitions
4.2 Transition probabilities
4.3 General two-state Markov chain
4.4 Powers of the transition matrix for the m-state chain
4.5 Gambler’s ruin as a Markov chain
4.6 Classification of states
4.7 Classification of chains
4.8 A wildlife Markov chain model
Problems
Chapter 5: Poisson Processes

5.1 Introduction
5.2 The Poisson process
5.3 Partition theorem approach
5.4 Iterative method
5.5 The generating function
5.6 Variance for the Poisson process
2
5.7 Arrival times
5.8 Summary of the Poisson process
Problems
Chapter 6: Birth and Death Processes

6.1 Introduction
6.2 The birth process
6.3 Birth process: generating function equation
6.4 The death process
6.5 The combined birth and death process
6.6 General population processes
Problems
Chapter 7: Queues
7.1 Introduction
7.2 The single server queue
7.3 The stationary process
7.4 Queues with multiple servers
7.5 Queues with fixed service times
7.6 Classification of queues
Problems
Chapter 8: Reliability and Renewal

8.1 Introduction
8.2 The reliability function
8.3 The exponential distribution and reliability
8.4 Mean time to failure
8.5 Reliability of series and parallel systems
8.6 Renewal processes
8.7 Expected number of renewals
Problems
Chapter 9: Branching and Other Random Processes

9.1 Introduction
9.2 Generational growth
9.3 Mean and variance
9.4 Probability of extinction
9.5 Branching processes and martingales
9.6 Stopping rules
9.7 A continuous time epidemic
9.8 A discrete time epidemic model
9.9 Deterministic epidemic models
9.10 An iterative scheme for the simple epidemic
Problems
Chapter 10: Brownian Motion: Wiener Process

10.1 Introduction
10.2 Brownian motion
10.3 Wiener process as a limit of a random walk
10.4 Brownian motion with drift
10.5 Scaling
10.6 First visit times
3
10.7 Other Brownian motions in one dimension
10.8 Brownian motion in more than one dimension
Problems
Chapter 11: Computer Simulations and Projects
4
Contents of the Solutions Manual
Chapter 1: Some Background in Probability 6
Chapter 2: Some Gambling Problems 20
Chapter 3: Random Walks 34
Chapter 4: Markov Chains 49
Chapter 5: Poisson Processes 74
Chapter 6: Birth and Death Processes 80
Chapter 7: Queues 105
Chapter 8: Reliability and Renewal 121
Chapter 9: Branching and Other Random Processes 130
Chapter 10: Brownian Motion: Wiener Process 150
5
Chapter 1
Some background in probability
1.1. The Venn diagram of three events is shown in Figure 1.5(in the text). Indicate on the diagram the
following events:
(a) A ∪ B; (b) A ∪ (B ∪ C); (c) A ∩ (B ∪ C); (d) (A ∩ C)c ; (e) (A ∩ B) ∪ C c .
S S
A B A B
C C
(a) (b)
S S
A B A B
C C
(c) (d)
S
A B
(e)
Figure 1.1:
The events are shaded in Figure 1.1.
1.2. In a random experiment, A, B, C are three events. In set notation write down expressions for the
events:
(a) only A occurs;
(b) all three events A, B, C occur;
(c) A and B occur but C does not;
(d) at least one of the events A, B, C occurs;
(e) exactly one of the events A, B, C occurs;
(f ) not more than two of the events occur.
(a) A ∩ (B ∪ C)c ; (b) A ∩ (B ∩ C) = A ∩ B ∩ C; (c) (A ∩ B) ∩ C c ; (d) A ∪ B ∪ C;
6
(e) A ∩ (B ∪ C)c represents an event in A but not in either B nor C: therefore the answer is
(A ∩ (B ∪ C)c ) ∪ (B ∩ (A ∪ C)c ) ∪ (C ∩ (A ∪ B)c ).
(f) A ∩ B ∩ C
1.3. For two events A and B, P(A) = 0.4, P(B) = 0.5 and P(A ∩ B) = 0.3. Calculate
(a) P(A ∪ B); (b) P(A ∩ B c ); (c) P(Ac ∪ B c ).
(a) From (1.1) P(A ∪ B) = P(A) + P(B) − P(A ∩ B), it follows that
P(A ∪ B) = 0.4 + 0.5 − 0.3 = 0.6.
(b) Since A = (A ∩ B c ) ∪ (A ∩ B) and A ∩ B c , and A ∩ B are mutually exclusive, then,
P(A) = P[(A ∩ B c ) ∪ (A ∩ B)] = P(A ∩ B c ) + P(A ∩ B),
so that
P(A ∩ B c ) = P(A) − P(A ∩ B) = 0.4 − 0.3 = 0.1.
(c) Since Ac ∪ B c = (A ∩ B)c , then
P(Ac ∪ B c ) = P[(A ∩ B)c ] = 1 − P(A ∩ B) = 1 − 0.3 = 0.7.
1.4. Two distinguishable fair dice a and b are rolled. What are the elements of the sample space? What is
the probability that the sum of the face values of the two dice is 9? What is the probability that at least one
5 or at least one 3 appears?
The 36 elements of the sample space are listed in Example 1.1. The event A1 , that the sum is 9, is
given by
A1 = {(3, 6), (4, 5), (5, 4), (6, 3)}.
4
Hence P = 36 = 19 .
Let A2 be the event that at least one 5 or at least one 3 appears. Then by counting the elements in
20
the sample space in Example 1.1, P(A2 ) = 36 = 59 .
1.5. Two distinguishable fair dice a and b are rolled. What is the probability that the sum of the faces is
not more than 6?
Let the random variable X be the sum of the faces. By counting events in the sample space in Example
15 5
1.1, P(X) = 36 = 12 .
1.6. For the probability generating function

1
G(s) = (2 − s)− 2
find {pn } and its mean.
Note that G(1) = 1. Using the binomial theorem (see the Appendix)
∞
1 1 1 1 X − 21 n
s − 21
G(s) = √ (1 − s)− 2 = √ where = 1.
2 2 2 n=0 n 2 0
Hence
1 1 1 − 21
p0 = √ , pn = √ n , (n = 1, 2, . . .).
2 22 n
The mean is
3
µ = G′ (1) =

1
2
(2 − s)− 2 = 12 .
s=1
7
1.7. Find the probability generating function G(s) of the Poisson distribution (see Section 1.7) with param-
eter α given by
e−α αn
pn = , n = 0, 1, 2, . . . .
n!
Determine the mean and variance of {pn } from the generating function.
Given pn = e−α αn /n!, the generating function is given by

∞ ∞ ∞
X X e−α αn sn X (αs)n
G(s) = p n sn = = e−α = eα(s−1) .
n! n!
n=0 n=0 n=0
The mean and variance are given by

d α(s−1)
µ = G′ (1) = e = α,
ds s=1
σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = [α2 eα(s−1) + αeα(s−1) − α2 e2α(s−1) ]s=1 = α.
1.8. A panel contains n warning lights. The times to failure of the lights are the independent random
variables T1 , T2 , . . . , Tn which have exponential distributions with parameters α1 , α2 , . . . , αn respectively.
Let T be the random variable of the time to first failure, that is
T = min{T1 , T2 , . . . , Tn }.
Pn
Show that T has an exponential distribution with parameter j=1
αj .
The probability that no warning light has failed by time t is
P(T ≥ t) = P(T1 ≥ t ∩ T2 ≥ t ∩ · · · ∩ Tn ≥ t)
= P(T1 ≥ t)P(T2 ≥ t) · · · P(Tn ≥ t)
= e−α1 t e−α2 t · · · e−αn t = e−(α1 +α2 +···+αn )t .
1.9. The geometric distribution with parameter p is given by
p(x) = q x−1 p, x = 1, 2, . . .
where q = 1−p (see Section 1.7). Find its probability generating function. Calculate the mean and variance
of the geometric distribution from its pgf.
The generating function is given by

∞ ∞
X pX p qs ps
G(s) = q x−1 psx = (qs)x = = ,
q q 1 − qs 1 − qs
x=1 x=1
using the formula for the sum of a geometric series.

The mean is given by

′ d ps p pqs 1
µ = G (1) = = + = .
ds 1 − qs s=1
1 − qs (1 − qs)2 s=1
p
For the variance,

′′ d p 2pq
G (s) = = .
ds (1 − qs)2 (1 − qs)3
is required. Hence
2q 1 1 q
σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = + − 2 = 2.
p2 p p p
8
1.10. Two distinguishable fair dice a and b are rolled. What are the probabilities that:
(a) at least one 4 appears;
(b) only one 4 appears;
(c) the sum of the face values is 6;
(d) the sum of the face values is 5 and one 3 is shown;
(e) the sum of the face values is 5 or only one 3 is shown?
From the Table in Example 1.1:

(a) If A1 is the event that at least one 4 appears, then P(A1 ) = 11
36
.
10 5
(b) If A2 is the event that only one 4 appears, then P(A2 ) = 36 = 18 .
5
(c) If A3 is the event that the sum of the faces is 6, then P(A3 ) = 36 .
2 1
(d) If A4 is the event that the face values is 5 and one 3 is shown, then P(A4 ) = 36 = 18 .
10 5
(e) If A5 is the event that the sum of the faces is 5 or only one 3 is shown, then P(A5 ) = 36 = 18
.
1.11. Two distinguishable fair dice a and b are rolled. What is the expected sum of the face values? What
is the variance of the sum of the face values?
Let N be the random variable representing the sum x + y, where x and y are face values of the two
dice. Then " 6 #
6 6 6
1 XX 1 X X
E(N ) = (x + y) = 6 x+6 y = 7.
36 36
x=1 y=1 x=1 y=1
and
6 6
2 1 XX
2
V(N ) = E(N ) − E(N ) = (x + y)2 − 72
36
x=1 y=1
" 6 6
!2 #
1 X 2 X
= 12 x +2 x − 49
36
x=1 x=1
329 35
= − 49 = = 5.833 . . . .
6 6
1.12. Three distinguishable fair dice a, b and c are rolled. How many possible outcomes are there for the
faces shown? When the dice are rolled, what is the probability that just two dice show the same face values
and the third one is different?
The sample space contains 63 = 216 elements of the form, (in the order a, b, c),
S = {(i, j, k)}, (i = 1, . . . , 6; j = 1, . . . , 6; k = 1, . . . , 6).
Let A be the required event. Suppose that a and b have the same face values, which can occur in 6 ways,
and that c has a different face value which can occurs in 5 ways. Hence the total number of ways in which
a and b are the same but c is different is 6 × 5 = 30 ways. The faces b and c, and c and a could also be
the same so that the total number of ways for the possible outcome is 3 × 30 = 90 ways. Therefore the
required probability is
90 5
P(A) = = .
216 12
1.13. In a sample space S, the events B and C are mutually exclusive, but A and B, are not. Show that
P(A ∪ (B ∪ C)) = P(A) + P(B) + P(C) − P(A ∩ (B ∪ C)).
From a well-shuffled pack of 52 playing cards a single card is randomly drawn. Find the probability that
it is a club or an ace or the king of hearts.
From (1.1) (in the book)
P(A ∪ (B ∪ C)) = P(A) + P(B ∪ C) − P(A ∩ (B ∪ C)) (i).
9
Since B and C are mutually exclusive,
P(B ∪ C) = P(B) + P(C). (ii)
From (i) and (ii), it follows that
P(A ∪ (B ∪ C)) = P(A) + P(B) + P(C) − P(A ∩ (B ∪ C)).
Let A be the event that the card is a club, B the event that it is an ace, and C the event that it is the
king of hearts. We require P(A ∩ (B ∪ C)). Since B and C are mutually exclusive, we can use the result
above. The individual probabilities are
13 1 4 1 1
P(A) = = ; P(B) = = ; P(C) = ,
52 4 52 13 52
and since A ∩ (B ∪ C) = A ∩ B (A and C are mutually exclusive) is the ace of clubs,
1
P(A ∩ (B ∪ C)) = .
52
Finally
1 1 1 1 17
P(A ∪ (B ∪ C)) = + + − = .
4 13 52 52 52
1.14. Show that 

 0 x<0
1
f (x) = 2a
0≤x≤a
 1 −(x−a)/a
2a
e x>a
is a possible probability density function. Find the corresponding probability function.
Check the density function as follows:

Z ∞ Z a Z ∞
1 1
f (x)dx = dx + e−(x−a)/a dx
−∞
2a 0
2a a
= 1
2
− 21 [e−(x−a)/a ]∞
a = 1.
The probability function is given by, for 0 ≤ x ≤ a,

Z x Z x
1 x
F (x) = f (u)du = du = ,
−∞ 0
2a 2a
and, for x > a, by

Z x Z a Z a
1 1 −(u−a)/a
F (x) = f (u)du = du + e du
0 0
2a 0
2a
1 1
= − [ae−(u−a)/a ]xa
2 2a
1
= 1 − e−(x−a)/a .
2
1.15. A biased coin is tossed. The probability of a head is p. The coin is tossed until the first head appears.
Let the random variable N be the total number of tosses including the first head. Find P(N = n), and its
pgf G(s). Find the expected value of the number of tosses.
The probability that the total number of throws is n (including the head) until the first head appears
is (assuming independent events)
(n−1) times
z }| {
P(N = n) = (1 − p)(1 − p) · · · (1 − p) p = (1 − p)n−1 p, (n ≥ 1)
10
The probability generating function is given by
∞
X ∞
p X
G(s) = (1 − p)n−1 psn = [(1 − p)s]n
1−p
n=1 n=1
p s(1 − p) ps
= · = ,
1 − p [1 − s(1 − p)] 1 − s(1 − p)
after summing the geometric series.
For the mean, we require G′ (s) given by,
p sp(1 − p) p
G′ (s) = + = .
[1 − s(1 − p)] [1 − s(1 − p)]2 [1 − s(1 − p)]2
The mean is given by µ = G′ (1) = 1/p.
1.16. The m random variables X1 , X2 , . . . , Xm are independent and identically distributed each with a
gamma distribution with parameters n and α. The random variable Y is defined by
Y = X1 + X2 + · · · + Xm .
Using the moment generating function, find the mean and variance of Y .
The probability density function for the gamma distribution with parameters n and α is
αn n−1 −αx
f (x) = x e .
Γ(n)
It was shown in Section 1.9 that the moment generating function for Y is given, in general, by
MY (s) = [MX (s)]m .
For any gamma distribution X with parameters α and n, its mgf is
α n
MX = .
α−s
Hence

nm
α s −nm
MY (s) = = 1−
α−s α
nm nm(nm + 1) 2
= 1+ s+ s + ···
α 2α2
Hence
nm nm
E(Y ) = , V(Y ) = E(Y 2 ) − [E(Y )]2 = .
α α2
1.17. A probability generating function with parameter 0 < α < 1 is given by

1 − α(1 − s)
G(s) = .
1 + α(1 − s)
Find pn = P(N = n) by expanding the series in powers of s. What is the mean of the probability function
{pn }?
Applying the binomial theorem

1 − α(1 − s) (1 − α)[1 + (α/(1 − α))s]
G(s) = =
1 + α(1 − s) (1 + α)[1 − (α/(1 + α))s]
∞
X
1−α αs αs n
= 1+
1+α 1−α 1+α
n=0
∞ n ∞ n
1−α X α α X α
= sn + sn+1 .
1+α 1+α 1+α 1+α
n=0 n=0
11
The summation of the two series leads to
∞ n ∞ n
1−α X α n
X α
G(s) = s + sn
1+α 1+α 1+α
n=0 n=1
∞
X n
1−α 2 α
= + sn
1+α 1+α 1+α
n=1
Hence
1−α 2αn
p0 = , pn = , (n = 1, 2, . . .).
1+α (1 + α)n+1

′ d 1 − α(1 − s) 2α
G (1) = = = 2α
ds 1 + α(1 − s) s=1
[1 + a(1 − s)]2 s=1
1.18. Find the moment generating function of the random variables X which has the uniform distribution

1/(b − a) a≤x≤b
f (x) =
0 for all other values of x
Deduce E(X n ).
The moment generating function of the uniform distribution is

Z b
exs 1 1 bs
MX (s) = dx = [e − eas ]
a
b−a b−as
∞
1 X bn − a n
= sn−1
b−a n!
n=1
Hence
1 bn+1 − an+1
E(X) = (b + a), E(X n ) = , (n = 2, 3, . . .).
2 (n + 1)(b − a)
1.19. A random variable X has the normal distribution with mean µ and variance σ 2 . Find its moment
generating function.
By definition
Z ∞

1 (x − µ)2
MX (s) = E(eXs ) = √ esx exp − dx
σ 2π −∞
2σ 2
Z ∞

1 2σ 2 xs − (x − µ)2
= √ exp dx
σ 2π −∞
2σ 2
Apply the substitution x = µ + σ(v − σs): then

Z ∞
1 1 2
MX (s) = exp(sµ + 1 2 2
2
σ s ) √ e− 2 v dv
−∞ 2π
= exp(sµ + 12 σ 2 s2 ) × 1 = exp(sµ + 21 σ 2 s2 )
(see the Appendix for the integral).

Expansion of the exponential function in powers of s gives
1 2
MX (s) = 1 + µs + (µ + σ 2 )s2 + · · · .
2
So, for example, E(X 2 ) = µ2 + σ 2 .
12
1.20. Find the probability generating functions of the following distributions, in which 0 < p < 1:
(a) Bernoulli distribution: pn = pn (1 − p)1−n , (n = 0, 1);
(b) geometric distribution: pn = p(1 − p)n−1 , (n = 1, 2, . . .);
(c) negative binomial distribution with parameter r expressed in the form:

r+n−1 r
pn = p (1 − p)n , (n = 0, 1, 2, . . .)
r−1
where r is a positive integer. In each case find also the mean and variance of the distribution using the
probability generating function.
(a) For the Bernoulli distribution
G(s) = p0 + p1 s = (1 − p) + ps.

µ = G′ (1) = p,
and the variance by
σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = p − p2 = p(1 − p).
(b) For the geometric distribution (with q = 1 − p)
∞ ∞
X X ps
G(s) = pq n−1 sn = ps (qs)n =
1 − qs
n=1 n=0
summing the geometric series. The mean and variance are given by

′ p 1
µ = G (1) = = ,
(1 − qs)2 s=1
p

2pq 1 1 1−p
σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = + − 2 = .
(1 − qs)3 s=1
p p p2
(c) For the negative binomial distribution (with q = 1 − p)
∞
X r+n−1 r n n r r(r + 1)
G(s) = p q s =p 1 + r(qs) + (qs)2 + · · ·
r−1 2!
n=0
pr
=
(1 − qs)r
The derivatives of G(s) are given by
rqpr r(r + 1)q 2 pr

G′ (s) = , G′′ (s) = .
(1 − qs)r+1 (1 − qs)r+2
Hence the mean and variance are given by
rq
µ = G′ (1) = ,
p
r(r + 1)q 2 rq r2 q2 rq
σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = 2
+ − 2 = 2
p p p p
1.21. A word of five letters is transmitted by code to a receiver. The transmission signal is weak, and there
is a 5% probability that any letter is in error independently of the others. What is the probability that the
word is received correctly? The same result is transmitted a second time with the same errors in the signal.
If the same word is received, what is the probability now that the word is correct?
13
Let A1 , A2 , A3 , A4 , A5 be the events that the letters in the word are correct. Since the events are
independent, the probability that the word is correctly transmitted is
P(A1 ∩ A2 ∩ A3 ∩ A4 ∩ A5 ) = P(A1 )P(A2 )P(A3 )P(A4 )P(A5 ) = 0.955 ≈ 0.774.
If a letter is sent a second time the probability that one error occurs twice is 0.052 = 0.0025. Hence
the probability that the letter is correct is 0.9975. For 5 letters the probability that the word is correct is
0.99755 ≈ 0.988.
1.22. A binary code of 500 bits is transmitted across a weak link. The probability that any bit has a
transmission error is 0.0004 independently of the others.
(a) What is the probability that only the first bit fails?
(b) What is the probability that the code is transmitted successfully?
(c) What is the probability that at least two bits fail?
(a) Probability = 0.0004 × 0.9996499 = 0.000328...

(b) Probability = 0.1638...
(c) Probability is
500
X
0.0004j × 0.9996500−j = 0.01750.
j=2
For any practical purposes the upper limit in the series can be replaced by 7 for sufficient accuracy.
1.23. The source of a beam of light is a perpendicular distance d from a wall of length 2a, with the
perpendicular from the source meeting the wall at its midpoint. The source emits a pulse of light randomly
in a direction θ, the angle between the direction of the pulse and the perpendicular is chosen uniformly in
the range − tan−1 (a/d) ≤ θ ≤ tan−1 (a/d). Find the probability distribution of x (−a ≤ x ≤ a) where the
pulses hit the wall. Show that its density function is given by
d
f (x) = ,
2(x2 + d2 ) tan−1 (a/d)
(this the density function of a Cauchy distribution). If a → ∞, what can you say about the mean of
this distribution?
Figure 1.2 shows the beam and wall. Let X be the random variable representing any displacement
wall x
-a a
d beam
0
Figure 1.2: Source and beam for Problem 1.23
between −a and x. Then
P(−a ≤ X ≤ x) = P(−a ≤ d tan θ ≤ x)

= P(tan−1 (−a/d) + tan−1 (x/d))
tan−1 (x/d) + tan−1 (a/d)
=
2 tan−1 (a/d)
14
by uniformity. The density is given by

d tan−1 (x/d) + tan−1 (a/d)
f (x) =
dx 2 tan−1 (a/d)
d
=
2(x2 + d2 ) tan−1 (a/d)
The mean is given by Z a
xd
µ= dx = 0,
−a
2(x2 + d2 ) tan−1 (a/d)
since the integrand is an odd function and the limits are ±a.
For the infinite wall the integral defining the mean becomes divergent.
1.24. Suppose that the random variable X can take the integer values 0, 1, 2, . . .. Let pj and qj be the
probabilities
pj = P(X = j), qj = P(X > j), (j = 0, 1, 2, . . .).
Show that, if
∞ ∞
X X
G(s) = p j sj , H(s) = qj sj ,
j=0 j=0
then (1 − s)H(s) = 1 − G(s).

Show also that E(X) = H(1).
Using the series for H(s),

∞ ∞ ∞
X X X
(1 − s)H(s) = (1 − s) qj sj = qj sj − qj sj+1
j=0 j=0 j=0
∞
X
= q0 + (qj − qj−1 )sj
j=1
∞
X
= q0 − P(X = j)sj
j=1
∞
X
= 1 − p0 − pj sj = 1 − G(s)
j=1
Note that generally H(s) is not a probability generating function.

The mean of the random variable X is given by
∞
X
E(X) = jpj = G′ (1) = H(1),
j=1
differentiating the formula above.
1.25 In a lottery players can choose q numbers from the consecutive integers 1, 2, . . . , n (q < n). The player
wins if r numbers (3 ≤ r ≤ q) from q agree with those numbers of the q numbers randomly chosen from
the n integers. Show that the probability of r numbers being correct is

q n−q n
.
r q−r q
Compute the probabilities if n = 49, q = 6, r = 3, 4, 5, 6 (the UK lottery).
This is a combinatorial problem. There are

n
q
15
possible combinations of choosing q numbers from n. From the n − q numbers which are not randomly
chosen this must include the q − r losing numbers: they can be chosen in

n−q
q−r
ways. Also there are
q
r
ways of choosing the r numbers from q. Hence the probability of winning in the lottery is

q n−q n
.
r q−r q
If n = 49, q = 6, and r = 3, 4, 5, 6, then the probabilities are given in the table:
r formula exact probability approximation

6
43
49
8815
3 3 3 6 499422
0.0177

6
43 49
645
4 4 2 6 665896
0.000969

6
43 49
43
5 5 1 6 2330636
0.0000184

6
43 49
43
6 6 0 6 6
0.0000000715
The chance of correctly choosing 6 numbers is 1 in 13,983,816.
1.26. A count of the second edition of this book showed that it contains 181,142 Roman letters (not case
sensitive: Greek not included). The table of the frequency of the letters.
Frequency table
a 13011 j 561 s 12327

b 4687 k 1424 t 15074
c 6499 l 6916 u 5708
d 5943 m 5487 v 2742
e 15273 n 14265 w 3370
f 5441 o 11082 x 2361
g 3751 p 6891 y 3212
h 9868 q 1103 z 563
i 12827 r 10756
The most frequent letter is e closely followed by t, n and a. However, since this a mathematical textbook
there is considerable distortion compared with a piece of prose caused by the extensive use of symbols
particularly in equations. What is the probability that a letter is i ? What is the probability that a letter is
a vowel a, e, i , o, u?
A word count shows that the word probability occurs 821 times.
From the table

P(i) = 12827/181142 = 0.0708...
P(vowel) = (13011 + 15273 + 12827 + 11082 + 5708)/181142 = 0.3196....
1.27 Using the table of probabilities in Example 1.9, calculate the conditional probabilities and random
variable of the conditional expectation E(Y |X).
From Example 1.9 the table of mass functions is
16
p(xi , yj ) y1 y2 y3
x1 0.25 0 0.05
x2 0.05 0.10 0.15
x3 0.05 0.25 0.10
The required conditional probabilities are:

3
X 5
pY (y1 |x1 ) = p(x1 , y1 )/ p(x1 , yj ) = 0.25/0.3 = ,
6
j=1
3
X
pY (y2 |x1 ) = p(x1 , y2 )/ p(x1 , yj ) = 0,
j=1
3
X 1
pY (y3 |x1 ) = p(x1 , y3 )/ p(x1 , yj ) = 0.5/0.3 = ,
6
j=1
1 1 1
pY (y1 |x2 ) = , pY (y2 |x2 ) = , pY (y3 |x2 ) = ,
6 3 2
1 5 1
pY (y1 |x3 ) = , pY (y2 |x3 ) = , pY (y3 |x5 ) = .
8 8 4
It follows that the conditional probabilities are:
3
X 5 1 4
E(Y |X = x1 ) = yi p(yj |x1 ) = 1 × +2×0+3× = ,
6 6 3
j=1
3
X 1 1 1 7
E(Y |X = x2 ) = yj p(yj |x2 ) = 1 × +2× +3× = ,
6 3 2 3
j=1
3
X 1 5 1 17
E(Y |X = x3 ) = p(yj |x3 ) = 1 × +2× +3× = .
8 8 4 8
j=1
Hence the random variable U = E(Y |X)) takes the values { 34 , 73 , 17

8
}, and
4 7 17 39
E(U ) = E[E(Y |X)] = × 0.3 + × 0.3 + × 0.4 = .
3 3 8 20
For comparison
39
E(Y ) = 1 × 0.35 + 2 × 0.35 + 3 × 0.3 =
20
which agrees with E(U ).
1.28. (a) Show that the moment generating function (mgf ) of a N (µ, σ 2 ) random variable is exp(µs +
1 2 2
2
σ s ).
(b) Use (a) to identify the distribution of Z = (X − µ)/σ where X is N (µ, σ 2 ).
(c) Use a mgf to identify the distribution of Y = aX + b where a, b are constants and X is N (µ, σ 2 ).
(d) A sequence Xi (i = 1, 2, . . . n) of n independent
Pn normally distributed random variables has mean µi
and variance σ 2i . Derive the
Pn distribution of i=1
Xi . If Zi are n iid standard normal random variables,
what is the distribution of i=1
Z i .
Pn
(e) Consider the sequence of random variables in (c): find the distribution of a Xi , where the
i=1 i P
n
ai ’s (i = 1, 2, . . . , n) are constants. What is the distribution of the mean of the Xi ’s, namely i−1
Xi /n?
(a) The mgf of N (µ, σ 2 ) is (if X is the random variable)

Z ∞
1 (x − µ)2
MX (s) = E(esX ) = √ exp(sx) exp − dx.
2πσ −∞
2σ 2
17
Using the substitution y = (x − µ)/σ,
Z ∞
1 y2
MX (s) = √ exp[(yσ + µ)s] exp − dy
2π −∞
2
Z ∞
1 1 1
= √ exp[µs + s2 σ 2 ] exp[− (y − sσ 2 )2 ]dy
2π 2 −∞
2
= exp[µs + 21 s2 σ 2 ],
(see Appendix for the value of the integral)

(b) As in (a) use the substitution z = (x − µ)/σ so that the mgf
Z ∞

1 (z − µ)2
MZ (s) = E(esZ ) = √ exp(zs) exp − dz = exp( 21 σ 2 ),
2πσ −∞
2σ 2
which is the mgf of a standardised normal distribution N (0, 1).

(c) For the random variable Y = aX + b, the mgf is
MY (s) = E(eaX+b ) = eb E(eaX ).
From (a) it can be deduced that Y has the normal distribution N (aµ + b, a2 σ 2 ).
Pn
(d) As in Section 1.9 the mgf of W = i=1
Xi is
n n
X Y
MW (s) = E[exp( Xi )] = MXi (s)
i=1 i=1
" n
#
X Pn
1 2
= exp s µi + 2
s σ2
i=1 i
i=1
. Hence W is normal " n #

n
X X
N µi , σ 2i
i=1 i=1
.
The distribution of the iid standard normal random variables Zi is therefore, from above N (0, n).
Pn
(e) Let U = i=1
ai Xi . Then the mgf of U is given by
n
X n
Y
MU (s) = E[exp( ai Xi )] = Mai Xi (s)
i=1 i=1
From (c) ai Xi has the normal distribution
N [ai µi , a2i σ 2 ].
Hence U has the normal distribution

" n n
#
X X
N ai µi , a2i σ 2 .
i=1 i=1
Pn
The distribution of the mean i=1
Xi /n is therefore normal given by
" n n
#
X X 2 2
N µi /n, σ /n .
i=1 i=1
1
1.29. (a) Let Z have a standard normal distribution, show that the mgf of Z 2 is (1 − 2s)− 2 . This is the
mgf of a χ2 distribution on 1 degree of freedom (χ21 ). Hence Z 2 has a χ21 distribution.
18
(b) Let Z1 , Z2 . . . , Zn be a sequence of n iid standard normal random variables . Show that the mgf of
n
X
Y = Zi2
i=1
1
is (1 − 2s)− 2 n , which is the mgf of a χ2n distribution.
(c) Find the mean and variance of Y in (b).
(a) If U = Z 2 , then the mgf of Z 2 is given by

Z ∞
2 1
MU (s) = E[exp(sZ )] = √ exp(sz 2 ) exp(− 21 z 2 )dz
2 π −∞
Z ∞
1 1
= √ exp[( 12 − s)z 2 ]dz = (1 − 2s)− 2
2 π −∞
(see Appendix for value of the integral).

(b) Result
1
MY (s) = (1 − 2s)− 2 n
follows from Problem 28(d).
(c) We require the derivatives of MY (s) in (b):
1 1
MY′ (s) = n(1 − 2s)− 2 n−1 , MY′′ (s) = n(n + 2)(1 − 2s)− 2 n−2 ,
so that
MY′ (0) = n, MY′′ (0) = n(n + 2).
Therefore
E(Z 2 ) = MY′ (0) = n, σ 2 = MY′′ (0) − [MY′ (0)]2 = 2n.
19
Chapter 2
Some gambling problems
2.1. In the standard gambler’s ruin problem with total stake a and gambler’s stake k and the gambler’s
probability of winning at each play is p, calculate the probability of ruin in the following cases;
(a) a = 100, k = 5, p = 0.6;
(b) a = 80, k = 70, p = 0.45;
(c) a = 50, k = 40, p = 0.5.
Also find the expected duration in each case.
For p 6= 12 , the probability of ruin uk and the expected duration of the game dk are given by

sk − sa 1 a(1 − sk )
uk = , dk = k− .
1 − sa 1 − 2p (1 − sa )
(a) uk ≈ 0.132, dk ≈ 409.

(b) uk ≈ 0.866, dk ≈ 592.
(c) For p = 21 ,
a−k
uk = , dk = k(a − k).
a
so that uk = 0.2, dk = 400.
2.2. In a casino game based on the standard gambler’s ruin, the gambler and the dealer each start with 20
tokens and one token is bet on at each play. The game continues until one player has no further tokens. It
is decreed that the probability that any gambler is ruined is 0.52 to protect the casino’s profit. What should
the probability that the gambler wins at each play be?
The probability of ruin is

sk − sa
u= ,
1 − sa
where k = 20, a = 40, p is the probability that the gambler wins at each play, and s = (1 − p)/p. Let
r = s20 . Then u = r/(1 + r), so that r = u/(1 − u) and
u 1/20
s= .
1−u
Finally, with u = 0.52,
1 (1 − u)1/20
p= = ≈ 0.498999.
1+s (1 − u)1/20 + u1/20
2.3. Find general solutions of the following difference equations:

(a) uk+1 − 4uk + 3uk−1 = 0;
(b) 7uk+2 − 8uk+1 + uk = 0;
(c) uk+1 − 3uk + uk−1 + uk−2 = 0.
20
(d) puk+2 − uk + (1 − p)uk−1 = 0, (0 < p < 1).
(a) The characteristic equation is

m2 − 4m + 3 = 0
which has the solutions m1 = 1 and m2 = 3. The general solution is
uk = Amk1 + Bmk2 = A + 3k B,
where A and B are any constants.

(b) The characteristic equation is
7m2 − 8m + 1 = 0,
which has the solutions m1 = 1 and m2 = 71 . The general solution is
1
uk = A + B.
7k
(c) The characteristic equation is the cubic equation
m3 − 3m2 + m + 1 = (m − 1)(m2 − 2m − 1) = 0,
√ √
which has the solutions m1 = 1, m2 = 1 + 2, and m3 = 1 − 2. The general solution is
√ √
uk = A + B(1 + 2)k + C(1 − 2)k .
(d) The characteristic equation is the cubic equation
pm3 − m + (1 − p) = (m − 1)(pm2 + pm − (1 − p)) = 0,

√ √
which has the solutions m1 = 1, m2 = − 21 + 12 [(4 − 3p)/p] and m3 = − 21 − 21 [(4 − 3p)/p]. The general
solution is
uk = A + Bmk2 + Cmk3 .
2.4 Solve the following difference equations subject to the given boundary conditions:
(a) uk+1 − 6uk + 5uk−1 = 0, u0 = 1, u4 = 0;
(b) uk+1 − 2uk + uk−1 = 0, u0 = 1, u20 = 0;
(c) dk+1 − 2dk + dk−1 = −2, d0 = 0, d10 = 0;
(d) uk+2 − 3uk + 2uk−1 = 0, u0 = 1, u10 = 0, 3u9 = 2u8 .
(a) The charactedristic equation is

m2 − 6m + 5 = 0,
which has the solutions m1 = 1 and m2 = 5. Therefore the general solution is given by
uk = A + 5k B.
The boundary conditions u0 = 1, u4 = 0 imply
A + B = 1, A + 54 B = 0,
which have the solutions A = 625/624 and B = −1/624. The required solution is
625 5k
uk = − .
624 624
(b) The characteristic equation is
m2 − 2m + 1 = (m − 1)2 = 0,
which has the repeated solution m = 1. Using the rule for repeated roots,
uk = A + Bk.
21
The boundary conditions u0 = 1 and u20 = 0 imply A = 1 and B = −1/20. The required solution is
uk = (20 − k)/20.
(c) This is an inhomogeneous equation. The characteristic equation is
m2 − 2m + 1 = (m − 1)2 = 0,
which has the repeated solution m = 1. Hence the complementary function is A + Bk. For a particular
solution, we must try dk = Ck2 . Then
dk+1 − 2dk + dk−1 = C(k + 1)2 − 2Ck2 + C(k − 1)2 = 2C = −2
if C = −1. Hence the general solution is
dk = A + Bk − k2 .
The boundary conditions d0 = d10 = 0 imply A = 0 and B = 10. Therefore the required solution is
dk = k(10 − k).
(d) The characteristic equation is
m3 − 3m + 2 = (m − 1)2 (m + 2) = 0,
which has two solutions m1 = 1 (repeated) and m2 = −2. The general solution is given by
uk = A + Bk + C(−2)k .
The boundary conditions imply
A + C = 1, A + 10B + C(−2)10 = 0, 3A + 27B + 3C(−2)9 = 2[A + 8B + C(−2)8 ].
The solutions of these linear equations are

31744 3072 1
A= , B= , C=−
31743 31743 31743
so that the required solution is
1024(31 − 2k) − (−2)k
uk = .
31743
2.5. Show that a difference equation of the form
auk+2 + buk+1 − uk + cuk−1 = 0,
where a, b, c ≥ 0 are probabilities with a + b + c = 1, can never have a characteristic equation with complex
roots.
The characteristic equation can be expressed in the form
am3 + bm2 − m + c = (m − 1)[am2 + (a + b)m − (1 − a − b)] = 0,
since a + b + c = 1. One solution is m1 = 1, and the others satisfy the quadratic equation
am2 + (a + b)m − (1 − a − b) = 0.
The discriminant is given by
(a + b)2 + 4(1 − a − b) = (a − b)2 + 4a(1 − a) ≥ 0,
since 0 ≤ a ≤ 1.
2.6. In the standard gambler’s ruin problem with equal probabilities p = q = 21 , find the expected duration
of the game given the usual initial stakes of k units for the gambler and a − k units for the opponent.
22
The expected duration dk satisfies
dk+1 − 2dk + dk−1 = −2.
The complementary function is A + Bk, and for a particular solution try dk = Ck2 . Then
dk+1 − 2dk + dk−1 + 2 = C(k + 1)2 − 2Ck2 + C(k − 1)2 + 2 = 2C + 2 = 0
if C = −1. Hence
dk = A + Bk − k2 .
The boundary conditions d0 = da = 0 imply A = 0 and B = a. The required solution is therefore
dk = k(a − k).
2.7. In a gambler’s ruin problem the possibility of a draw is included. Let the probability that the gambler
wins, loses or draws against an opponent be respectively, p, p, 1 − 2p, (0 < p < 12 ). Find the probability that
the gambler loses the game given the usual initial stakes of k units for the gambler and a − k units for the
opponent. Show that dk , the expected duration of the game, satisfies
pdk+1 − 2pdk + pdk−1 = −1.
Solve the difference equation and find the expected duration of the game.
The difference equation for the probability of ruin uk is
uk = puk+1 + (1 − 2p)uk + puk−1 or uk+1 − 2uk + uk−1 = 0.
The general solution is uk = A + Bk. The boundary conditions u0 = 1 and ua = 0 imply A = 1 and
B = −1/a, so that the required probability is given by uk = (a − k)/a.
dk+1 − 2dk + dk+1 = −1/p.
The complementary function is A + Bk. For the particular solution try dk = Ck2 . Then
C(k + 1)2 − 2Ck2 + C(k − 1)2 = 2C = −1/p.
Hence C = −1/(2p). The boundary conditions d0 = da = 0 imply A = 0 and B = a/(2p), so that the
required solution is
dk = k(a − 2p)/(2p).
2.8. In the changing stakes game in which a game is replayed with each player having twice as many units,
2k and 2(a − k) respectively, suppose that the probability of a win for the gambler at each play is 12 . Whilst
the probability of ruin is unaffected by how much is the expected duration of the game extended compared
with the original game?
With initial stakes of k and a − k, the expected duration is dk = k(a − k). If the initial stakes are
doubled to 2k and 2a − 2k, then the expected duration becomes, using the same formula,
d2k = 2k(2a − 2k) = 4k(a − k) = 4dk .
2.9. A roulette wheel has 37 radial slots of which 18 are red, 18 are black and 1 is green. The gambler bets
one unit on either red or black. If the ball falls into a slot of the same colour, then the gambler wins one
unit, and if the ball falls into the other colour (red or black), then the casino wins. If the ball lands in the
green slot, then the bet remains for the next spin of the wheel or more if necessary until the ball lands on
a red or black. The original bet is either returned or lost depending on whether the outcome matches the
23
original bet or not (this is the Monte Carlo system). Show that the probability uk of ruin for a gambler
who starts with k chips with the casino holdiing a − k chips satisfies the difference equation
36uk+1 − 73uk + 37uk−1 = 0.
Solve the difference equation for uk . If the house starts with ∈1,000,000 at the roulette wheel and the
gambler starts with ∈10,000, what is the probability that the gambler breaks the bank if ∈5,000 are bet at
each play.
In the US system the rules are less generous to the players. If the ball lands on green then the player
simply loses. What is the probability now that the player wins given the same initial stakes? (see Luenberger
(1979))
There is the possibility of a draw (see Example 2.1). At each play the probability that the gambler
18
wins is p = 37 . The stake is returned with probability

1 18 1 18 1 18 1
+ + ··· = = ,
37 37 372 37 36 37 74
or the gambler loses after one or more greens also with probability 1/74 by the same argument. Hence uk ,
the probability that the gambler loses satisfies
18 1 18
uk = uk+1 + (uk + uk−1 ) + uk+1 ,
37 74 37
or
36uk+1 − 73uk + 37uk−1 = 0.
The charactersitic equation is
36m2 − 73m + 37 = (m − 1)(36m − 37) = 0,
which has the solutions m1 = 1 and m2 = 37/36. With u0 = 1 and ua = 0, the required solution is
sk − sa 37
uk = , s= .
1 − sa 36
The bets are equivalent to k = 10000/5000 = 2, a = 1010000/5000 = 202. The probability that the
gambler wins is
1 − sk 1 − s2
1 − uk = = = 2.23 × 10−4 .
1 − sa 1 − s202
In the US system, uk satisfies
18 19
uk = uk+1 + uk−1 , or 18uk+1 − 37uk + 19uk−1 = 0.
37 37
in this case the ratio is s′ = 19/18. Hence the probability the the gambler wins is
1 − s′2
1 − uk = = 2.06 × 10−6 ,
1 − s′202
which is less than the previous value.
2.10. In a single trial the possible scores 1 and 2 can occur each with probability 21 . If pn is the probability
of scoring exactly n points at some stage, that is score after several trials is the sum of individual scores
in each trial. Show that
pn = 12 pn−1 + 21 pn−2 ,
Calculate p1 and p2 , and find a formula for pm . How does pm behave as m becomes large? How do you
interpret the result?
Let An be the event that the score is n at some stage. Let B1 be the event score 1, and B2 score 2.
Then
P(An ) = P(An |B1 )P(B1 ) + P(An |B2 )P(B2 ) = P(An−1 ) 21 + P(An−2 ) 21 ,
24
or
pn = 12 pn−1 + 21 pn−2 .
Hence
2pn − pn−1 − pn−2 = 0.
The characteristic equation is
2m2 − m − 1 = (m − 1)(2m + 1) = 0,
which has the solutions m1 = 1 and m2 = − 12 . Hence
pn = A + B(− 21 )n .
1 1 1 1
The initial conditions are p1 = 2
and p2 = 2
+ 2 2
= 34 . Hence
A − 21 B = 21 , A + 14 B = 34 ,
so that A = 32 , B = 31 . Hence
pn = 2
3
+ 13 (− 12 )n , (n = 1, 2, . . .).
2
As n → ∞, pn → 3
.
2.11. In a single trial the possible scores 1 and 2 can occur with probabilities q and 1 − q, where 0 < q < 1.
Find the probability of scoring exactly n points at some stage in an indefinite succession of trials. Show
that
1
pn → ,
2−q
as n → ∞.
Let pn be the probability. Then
pn = qpn−1 + (1 − q)pn−2 , or pn − qpn−1 − (1 − q)pn−2 = 0.
m2 − qm − (1 − q) = (m − 1)[m + (1 − q)] = 0,
which has the solutions m1 = 1 and m2 = −(1 − q). Hence
pn = A + B(q − 1)n .
The initial conditions are p1 = q, p2 = 1 − q + q 2 , which imply
q = A + B(q − 1), 1 − q + q 2 = A + B(q − 1)2 .
The solution of these equations leads to A = 1/(2 − q) and B = (q − 1)/(q − 2), so that
1
pn = [1 − (q − 1)n+1 ].
2−q
2.12. The probability of success in a single trial is 13 . If un is the probability that there are no two consecutive
successes in n trials, show that un satisfies
un+1 = 32 un + 92 un−1 .
What are the values of u1 and u2 ? Hence show that

√ n √ n
1 √ 1+ 3 √ 1− 3
un = (3 + 2 3) + (3 − 2 3) .
6 3 3
25
Let An be the event that there have not been two consecutive successes in the first n trials. Let B1 be
the event of success and B2 the event of failure in a single trial. Then
P(An ) = P(An |B1 )P(B1 ) + P(An |B2 )P(B2 ).
Now P(An |B2 ) = P(An−1 ): failure will not change the probability. Also
P(An |B1 ) = P(An−1 |B2 )P(B2 ) = P(An−2 )P(B2 ).

1
Since P(B1 ) = 3
, P(B2 ) = 32 ,
un = 29 un−2 + 32 un−1 or 9un − 6un−1 − 2un = 0,
where un = P(An ).
9m2 − 6m − 2 = 0,
√ √
which has the solutions m1 = 31 (1 + 3) and m2 = 13 (1 − 3). Hence
1 √ 1 √
un = A n
(1 + 3)n + B n (1 − 3)n .
3 3
11
The initial conditions are u1 = 1 and u2 = 1 − 33
= 98 . Therefore A and B are defined by
A √ B √
(1 + 3) + (1 − 3),
1=
3 3
8 A √ 2 B √ 2 A √ B √
= (1 + 3) + (1 − 3) = (4 + 2 3) + (4 − 2 3).
9 9 9 9 9
√ √
The solutions are A = 61 (2 3 + 3) and B = 16 (−2 3 + 3). Finally
1 √ √ √ √
un = [(2 3 + 3)(1 + 3)n + (−2 3 + 3)(1 − 3)n ].
6 · 3n
2.13. A gambler with initial capital k units plays against an opponent with initial capital a − k units. At
each play of the game the gambler either wins one unit or loses one unit with probability 21 . Whenever
the opponent loses the game, the gambler returns one unit so that the game may continue. Show that the
expected duration of the game is k(2a − 1 − k) plays.
dk+1 − 2dk + dk−1 = −2, (k = 1, 2, . . . , a − 1).
The boundary conditions are d0 = 0 and da = da−1 , indicating the return of one unit when the gambler
loses. The general solution for the duration is
dk = A + Bk − k2 .
The boundary conditions imply
A = 0, A + Ba − a2 = A + B(a − 1) − (a − 1)2 ,
so that B = 2a − 1. Hence dk = k(2a − 1 − k).
2.14. In the usual gambler’s ruin problem, the probability that the gambler is eventually ruined is
sk − sa q
uk = , s= , (p 6= 12 ).
1 − sa p
In a new game the stakes are halved, whilst the players start with the same initial sums. How does this
affect the probability of losing by the gambler? Should the gambler agree to this change of rule if p < 12 ?
By how many plays is the expected duration of the game extended?
26
The new probability of ruin vk (with the stakes halved) is, adapting the formula for uk ,

s2k − s2a (sk + sa )(sk − sa ) sk + sa
vk = u2k = 2a
= = uk .
1−s (1 − sa )(1 + sa ) 1 + sa
Given p < 21 , then s = (1 − p)/p > 1 and sk > 1. It follows that

1 + sa
vk > uk = uk .
1 + sa
With this change the gambler is more likely to lose.
From (2.9), the expected duration of the standard game is given by

1 a(1 − sk )
dk = k− .
1 − 2p (1 − sa )
With the stakes halved the expected duration hk is

1 2a(1 − s2k )
hk = d2k = 2k − .
1 − 2p (1 − s2a )
The expected duration is extended by

1 2a(1 − s2k ) a(1 − sk )
hk − dk = k− 2a
+
1 − 2p (1 − s ) (1 − sa )

1 a(1 − sk )(sa − 1 − 2sk )
= k+ .
1 − 2p (1 − s2a )
1
2.15. In a gambler’s ruin game, suppose that the gambler can win £2 with probability 3
or lose £1 with
probability 32 . Show that
(3k − 1 − 3a)(−2)a + (−2)k
uk = .
1 − (3a + 1)(−2)a
Compute uk if a = 9 for k = 1, 2, . . . , 8.
The probability of ruin uk satisfies

uk = 31 uk+2 + 32 uk−1 or uk+2 − 3uk + 2uk = 0.
m3 − 3m + 2 = (m − 1)2 (m + 2) = 0,
which has the solutions m1 = 1 (repeated) and m2 = −2. Hence
uk = A + Bk + C(−2)k .
The boundary conditions are u0 = 1, ua = 0, ua−1 = 32 ua−2 . The constants A, B and C satisfy
A + C = 1, A + Ba + C(−2)a = 0,
3[A + B(a + 1) + C(−2)a−1 ] = 2[A + B(a − 2) + C(−2)a−2 ],
or
A + B(a + 1) − 8C(−2)a−2 = 0.
The solution of these equations is
−(−2)a (3a + 1) 3(−2)a 1
A= , B= , C= .
1 − (−2)a (3a + 1) 1 − (−2)a (3a + 1) 1 − (−2)a (3a + 1)
Finally
(3k − 1 − 3a)(−2)a + (−2)k
uk = .
1 − (−2)a (3a + 1)
The values of the probabilities uk for a = 9 are shown in the table below.
27
k 1 2 3 4 5 6 7 8
uk 0.893 0.786 0.678 0.575 0.462 0.362 0.241 0.161
2.16. Find the general solution of the difference equation
uk+2 − 3uk + 2uk−1 = 0.
A reservoir with total capacity of a volume units of water has, during each day, either a net inflow
of two units with probability 13 or a net outflow of one unit with probability 23 . If the reservoir is full or
nearly full any excess inflow is lost in an overflow. Derive a difference equation for this model for uk , the
probability that the reservoir will eventually become empty given that it initially contains k units. Explain
why the upper boundary conditions can be written ua = ua−1 and ua = ua−2 . Show that the reservoir is
certain to be empty at some time in the future.
m3 − 3m + 2 = (m − 1)2 (m + 2) = 0.
The general solution is (see Problem 2.15)
uk = A + Bk + C(−1)k .
The boundary conditions for the reservoir are
u0 = 1, ua = 13 ua + 23 ua−1 , ua−1 = 13 ua + 32 ua−2 .
The latter two conditions are equivalent to ua = ua−1 = ua−2 . Hence
A + C = 1, A + Ba + C(−2)a = A + B(a − 1) + C(−2)a−1 = A + B(a − 2) + C(−2)a−2 .
which have the solutions A = 1, B = C = 0. The solution is uk = 1, which means that that the reservoir
is certain to empty at some future date.
2.17. Consider the standard gambler’s ruin problem in which the total stake is a and gambler’s stake is k,
and the gambler’s probability of winning at each play is p and losing is q = 1 − p. Find uk , the probability
of the gambler losing the game, by the following alternative method. List the difference equation (2.2) as
u2 − u1 = s(u1 − u0 ) = s(u1 − 1)
u3 − u2 = s(u2 − u1 ) = s2 (u1 − 1)
...
uk − uk−1 = s(uk−1 − uk−2 ) = sk−1 (u1 − 1),
where s = q/p 6= 12 and k = 2, 3, . . . a. The boundary condition u0 = 1 has been used in the first equation.
By adding the equations show that
s − sk
uk = u1 + (u1 − 1) .
1−s
Determine u1 from the other boundary condition ua = 0, and hence find uk . Adapt the same method for
the special case p = q = 21 .
Addition of the equations gives
s − sk
uk − u1 = (u1 − 1)(s + s2 + · · · + sk−1 ) = (u1 − 1)
1−s
summing the geometric series. The condition ua = 0 implies
s − sa
−u1 = (u1 − 1) .
1−s
28
Hence
s − sa
u1 = ,
1 − sa
so that
sk − sa
uk = .
1−s
2.18. A car park has 100 parking spaces. Cars arrive and leave randomly. Arrivals or departures of cars
are equally likely, and it is assumed that simultaneous events have negligible probability. The ‘state’ of the
car park changes whenever a car arrives or departs. Given that at some instant there are k cars in the car
park, let uk be the probability that the car park first becomes full before it becomes empty. What are the
boundary conditions for u0 and u100 ? How many car movements can be expected before this occurs?
The probability uk satisfies the difference equation

1 1
uk = uk+1 + uk−1 or uk+1 − 2uk + uk−1 .
2 2
The general solution is uk = A + Bk. The boundary conditions are u0 = 0 and u100 = 1. Hence A = 0
and B = 1/100, and uk = k/100.
The expected duration of car movements until the car park becomes full is dk = k(100 − k).
2.19. In a standard gambler’s ruin problem with the usual parameters, the probability that the gambler loses
is given by
sk − sa 1−p
uk = , s= .
1 − sa p
1 1
If p is close to 2
, given say by p = 2
+ ε where |ε| is small, show, by using binomial expansions, that
h i
a−k 4
uk = 1 − 2kε − (a − 2k)ε2 + O(ε3 )
a 3
as ε → 0. (The order O terminology is defined as follows: we say that a function g(ε) = O(εb ) as ε → 0 if
g(ε)/εb is bounded in a neighbourhood which contains ε = 0. See also the Appendix in the book.)
1
Let p = 2
+ ε. Then s = (1 − 2ε)/(1 + 2ε), and
(1 − 2ε)k (1 + 2ε)−k − (1 − 2ε)a (1 + 2ε)−a

uk = .
1 − (1 − 2ε)a (1 + 2ε)−a
Apply the binomial theorem to each term. The result is
h i
a−k 4
uk = 1 − 2kε − (a − 2k)ε2 + O(ε3 ) .
a 3
[Symbolic computation of the series is a useful check.]
2.20. A gambler plays a game against a casino according to the following rules. The gambler and casino
each start with 10 chips. From a deck of 53 playing cards which includes a joker, cards are randomly and
successively drawn with replacement. If the card is red or the joker the casino wins 1 chip from the gambler,
and if the card is black the gambler wins 1 chip from the casino. The game continues until either player
has no chips. What is the probability that the gambler wins? What will be the expected duration of the
game?
From (2.4) the probability uk that the gambler loses is
sk − sa
uk = ,
1 − sa
with k = 10, a = 20, p = 26/53, and s = 27/26. Hence
(27/26)10 − (27/26)20
u10 = ≈ 0.593.
1 − (27/26)20
29
Therefore the probability that the gambler wins is approximately 0.407.
From (2.10)
1 a(1 − sk
dk = k− = 98.84,
1 − 2p 1 − sa
for the given data.
2.21. In the standard gambler’s ruin problem with total stake a and gambler’s stake k, the probability that
the gambler loses is
sk − sa
uk = ,
1 − sa
where s = (1 − p)/p. Suppose that uk = 21 , that is fair odds. Express k as a function of a. Show that,
ln[ 12 (1 + sa )]
k= .
ln s
Given
sk − sa
uk = and uk = 12 ,
1 − sa
then 1 − sa = 2(sk − sa ) or sk = 21 (1 + sa ). Hence
ln[ 21 (1 + sa )]
k= ,
ln s
but generally k will not be an integer.
2.22. In a gambler’s ruin game the probability that the gambler wins at each play is αk and loses is 1 − αk ,
(0 < αk < 1, 0 ≤ k ≤ a − 1), that is, the probability varies with the current stake. The probability uk that
the gambler eventually loses satisfies
uk = αk uk+1 + (1 − αk )uk−1 , uo = 1, ua = 0.
Suppose that uk is a specified function such that 0 < uk < 1, (1 ≤ k ≤ a − 1), u0 = 1 and ua = 0. Express
αk in terms of uk−1 , uk and uk+1 .
Find αk in the following cases:
(a) uk = (a − k)/a;
(b) uk = (a2 − k2 )/a2 ;
(c) uk = 21 [1 + cos(kπ/a)].
From the difference equation

uk − uk−1
αk = .
uk+1 − uk−1
(a) uk = (a − k)/a. Then
(a − k) − (a − k + 1) 1
αk = = ,
(a − k − 1) − (a − k + 1) 2
which is to be anticipated from eqn (2.5).
(b) uk = (a2 − k2 )/a2 . Then
(a2 − k2 ) − [a2 − (k − 1)2 ] 2k − 1

αk = = .
[a2 − (k + 1)2 ] − [a2 − (k − 1)2 ] 4k
(c) uk = 1/(a + k). Then
[1/(a + k)] − [1/(a + k − 1)] a+k+1

αk = = .
[1/(a + k + 1)] − [1/(a + k − 1)] 2(a + k)
30
2.23. In a gambler’s ruin game the probability that the gambler wins at each play is αk and loses is 1 − αk ,
(0 < αk < 1, 1 ≤ k ≤ a − 1), that is, the probability varies with the current stake. The probability uk that
the gambler eventually loses satisfies
uk = αk uk+1 + (1 − αk )uk−1 , uo = 1, ua = 0.
Reformulate the difference equation as
uk+1 − uk = βk (uk − uk−1 ),
where βk = (1 − αk )/αk . Hence show that
uk = u1 + γk−1 (u1 − 1), (k = 2, 3, . . . , a)
where
γk = β1 + β1 β2 + · · · + β1 β2 . . . βk .
Using the boundary condition at k = a, confirm that
γa−1 − γk−1
uk = .
1 + γa−1
Check that this formula gives the usual answer if αk = p 6= 21 , a constant.
The difference equation can be expressed in the equivalent form
uk+1 − uk = βk (uk − uk−1 ),
where βk = (1 − αk )/αk . Now list the equations as follows, noting that u0 = 0,:
u2 − u1 = β1 (u1 − 1)
u3 − u2 = β1 β2 (u1 − 1)
··· = ···
uk − uk−1 = β1 β2 · · · βk−1 (u1 − 1)
Adding these equations, we obtain

uk − u1 = γk−1 (u1 − 1),
where
γk−1 = β1 + β1 β2 + · · · + β1 β2 · · · βk−1 .
The condition ua = 0 implies
−u1 = γa−1 (u1 − 1),
so that
γa−1
u1 + .
1 + γa−1
Finally
γa−1 − γk−1
uk = .
1 + γa−1
If αk = p 6= 21 , then βk = (1 − p)/p = s, say, and
s − sk+1
γ k = s + s2 + · · · + sk = .
1−s
Hence
(s − sa )/(1 − s) − (s − sk )/(1 − s) sk − sa
uk = a
=
1 + (s − s )/(1 − s) 1 − sa
as required.
2.24. Suppose that a fair n-sided die is rolled n independent times. A match is said to occur if side i is
observed on the ith trial, where i = 1, 2, . . . , n.
31
(a) Show that the probability of at least one match is
n
1
1− 1− .
n
(b) What is the limit of this probability as n → ∞?
(c) What is the probability that just one match occurs in n trials?
(d) What value does this probability approach as n → ∞?
(e) What is the probability that two or more matches occur in n trials?
(a) The probability of no matches is n

n−1
.
n
The probability of at least one match is
n n
n−1 1
1− =1− 1− .
n n
(b) As n → ∞,
1 n
→ e−1 .
1−
n
Hence for large n, the probability of at least one match approaches 1 − e−1 = (e − 1)/e.
(c) There is only one match with probability
n−1
n−1
.
n
(d) As n → ∞
n
n − 1 n−1 1 1
= 1− 1− = e−1 .
n n n
(e) Probability of two or more matches is
n−1 n n−1
n−1 n−1 1 1
− = 1− .
n n n n
2.25. (Kelly’s strategy) A gambler plays a repeated favourable game in which the gambler wins with prob-
ability p > 12 and loses with probability q = 1 − p. The gambler starts with an initial outlay K0 (in some
currency). For the first game the player bets a proportion rK0 , (0 < r < 1). Hence after this play the stake
is K0 (1 + r) after a win or K0 (1 − r) after losing. Subsequently, the gambler bets the same proportion of
the current stake at each play. Hence, after n plays of which wn are wins the stake Sr will be
Kn (r) = K0 (1 + r)wn (1 − r)n−wn .
Construct the function

1 Kn (r)
Gn (r) = ln .
n K0
What is the expected value of Gn (r) for large n? For what values of r is this expected value a maximum?
This value of r indicates a safe betting level to maximise the gain, although at a slow rate. You might
consider why the logarithm is chosen: this is known as a utility function in gambling and economics. It
is a matter of choice and is a balance between having a reasonable gain against having a high risk gain.
Calculate r if p = 0.55. [At the extremes r = 0 corresponds to no bet whilst r = 1 to betting K0 the whole
stake in one go which could be catastrophic.]
From
Kn (r) = K0 (1 + r)wn (1 − r)n−wn ,
it follows that
1 wn (n − wn )
Gn (r) = ln[(1 + r)wn (1 − r)n−wn = ln(1 + r) + ln(1 − r).
n n n
32
G + n(r) is our chosen utility function. For large n, the ratio wn /n is approximately the probability p, and
(n − wn )/n the probability q. Hence Gn (r) approaches
H(r) = p ln(1 + r) + q ln(1 − r).
This function is zero where r = 0 and approaches −∞ as r → 1−. Also

p q
H ′ (r) = − .
1+r 1−r
Hence H ′ (0) = p + q = 1 > 0, so that the slope is positive which implies that H(r) has a maximum for in
0 < r < 1. This occurs at H ′ (r) = 0 where where rm = 2p − 1 so that
H(rm ) = p ln(2p) + (1 − p) ln(2 − 2p).
The function Gn (r) is an example of a utility function which attempts to avoid possible catastrophic
losses, and keeps gains modest.
As an example let p = 0.55. Then rm = 2p − 1 = 0.1. Suppose that n = 100 and wn = 55 (reflecting
the odds). Then for this value of rm it can be calculated that K100 (rm ) ≈ 1.65K0 .
33
Chapter 3
Random Walks
3.1. In a random walk the probability that the walk advances by one step is p and retreats by one step is
q = 1 − p. At step n let the position of the walker be the random variable Xn . If the walk starts at x = 0,
enumerate all possible sample paths which lead to the value X4 = −2. Verify that

4
P{X4 = −2} = pq 3 .
1
If the walks which start at x = 0 and finish at x = −2, then each walk must advance one step with
probability p and retreat 3 steps with probability q 3 . the possible walks are:
0 → −1 → −2 → −3 → −2
0 → −1 → −2 → −1 → −2
0 → 1 → 0 → −1 → −2
0 → −1 → 0 → −1 → −2
By (3.4),
4
P{X4 = −2} = pq 3 .
1
3.2. A symmetric random walk starts from the origin. Find the probability that the walker is at the origin
at step 8. What is the probability also at step 8 that the walker is at the origin but that it is not the first
visit there?
By (3.4), the probability that the walker is at the origin at step 8 is given by

8 1 8! 1
P(X8 ) = = = 0.273.
4 28 4!4! 28
The generating function of the first returns fn is given by (see Section 3.4)
∞
X 1
Q(s) = fn sn = 1 − (1 − s2 ) 2 .
n=0
We require f8 in the expansion of Q(s). Thus, using the binomial theorem, the series expansion for Q(s)
is
1 1 1 6 5 8
Q(s) = s2 + s4 + s + s + O(s10 ).
2 8 16 128
Therefore the probability of a first return at step 8 is 5/128 = 0.039. Hence the probability that the walk
is at the origin but not a first return is 0.273 − 0.039 = 0.234.
34
3.3. An asymmetric walk starts at the origin. From eqn (3.5), the probability that the walk reaches x in n
steps is given by
n 1 1
vn,x = 1 p 2 (n+x) q 2 (n−x) ,
2
(n + x)
where n and x are both even or both odd. If n = 4, show that the mean value position x is 4(p − q),
confirming the result in Section 3.2.
The furthest positions that the walk can reach from the origin in 4 steps are x = −4 and x = 4, and
since n is even, the only other positions reachable are x = −2, 0, 2. Hence the required mean value is
µ = −4v4,−4 − 2v4,−2 + 2v4,2 + 4v4,4

4 4 4 4 3 4 4
= −4 q −2 pq 3 + 2 p q+4 p
0 1 3 4
= −4q 4 − 8pq 3 + 8p3 q + 4p4
= 4(p − q)(p + q)3 = 4(p − q).
3.4. The pgf for the first return distribution {fn }, (n = 1, 2, . . .), to the origin in a symmetric random walk
is given by
1
F (s) = 1 − (1 − s2 ) 2 ,
(see Section 3.4).
(a) Using the binomial theorem find a formula for fn , the probability that the first return occurs at the n-th
step.
(b) What is the variance of fn ?
Using the binomial theorem

∞ 1
1
X
F (s) = 1 − (1 − s2 ) 2 = 1 − 2 (−1)n s2n .
n
n=0
(a) From the series, the probability of a first return at step n is

 1

 1
(−1) 2 n+1 1
2
n even
fn = 2
n

0 n odd
(b) The variance of fn is defined by
V = F ′′ (1) − F ′ (1) − [F ′ (1)]2 .

We can anticipate that the variance will be infinite (as is the case with the mean) since the limit of the
derivatives of F (s) are unbounded as s → 1.
3.5. A coin is spun 2n times and the sequence of heads and tails is recorded. What is the probability that
the number of heads equals the number of tails after 2n spins?
This problem can be viewed as a symmetric random walk starting at the origin in which a head is
represented as a step to the right, say, and a tail a step to the left. We require the probability that the
walk returns to the origin after 2n steps, which is (see Section 3.3)

1 2n (2n)!
v2n,0 = = .
22n n 22n n!n!
3.6. For an asymmetric walk with parameters p and q = 1 − p, the probability that the walk is at the origin
after n steps is
2n n n
q2n = v2n,0 = p q , (n = 1, 2, 3, . . .),
n
35
from Eqn (3.5). Note that {q2n } is not a probability distribution. Find the mean number of steps of a
return to the origin conditional on a return occurring.
The generating function H(s) is defined by

∞ ∞
X 2n n n 2n
X 2n − 21 n n 2n
H(s) = p q s = 2 p q s
n n
n=0 n=0
1
= (1 − 4pqs2 )− 2
using the binomial identity from Section 3.3 or the Appendix.

The mean number of steps to the return is

4pqs 4pq
µ = H ′ (1) =
3 = 3 .
(1 − 4pqs2 ) 2 s=1 (1 − 4pq) 2
3.7. Using eqn (3.13) relating to the generating functions of the returns and first returns to the origin,
namely
H(s) = [H(s) + 1]F (s),
which is still valid for the asymmetric walk, show that
1
Q(s) = 1 − (1 − 4pqs2 ) 2 ,
where p 6= q. Show that a first return to the origin is not certain unlike the situation in the symmetric
walk. Find the mean number of steps to the first return.
From Problem 3.6,

1
H(s) = (1 − 4pqs2 )− 2 ,
so that, by (3.12),
1 1
Q(s) = 1 − = 1 − (1 − 4pqs2 ) 2 .
H(s)
It follows that
∞
X 1 1
Q(1) = fn = 1 − (1 − 4pq) 2 = 1 − [(p + q)2 − 4pq] 2
n=1
1
= 1 − [(p − q)2 ] 2 = 1 − |p − q| < 1,
if p 6= q. Hence a first return to the origin is not certain.

The mean number of steps is
4pq 4pq
µ = Q′ (1) = 1 = .
(1 − 4pq) 2 |p − q|
3.8. A symmetric random walk starts from the origin. Show that the walk does not revisit the origin in the
first 2n steps with probability
hn = 1 − f2 − f4 − · · · − f2n ,
where f2n is the probability that a first return occurs at the 2n-th step.
The generating function for the sequence {f2n } is
1
F (s) = 1 − (1 − s2 ) 2
(see Section 3.4). Show that
1 (2n − 3)!
f2 = f2n = , (n = 2, 3, . . .).
2 n!(n − 2)!22n−2
36
Show that hn satisfies the first-order difference equation
(2n + 1)!
hn+1 − hn = f2n+2 = .
(n + 1)!(n − 1)!22n
Verify that this equation has the general solution

2n 1
hn = C + ,
n 22n
where C is a constant. By calculating h1 , confirm that the probability of no return to the origin in the first
2n steps is
1 2n
.
22n n
The probability that a first return occurs at step 2j is f2j : a first return cannot occur after an odd
number of steps. Therefore the probability hn that a first return has not occurred in the first 2n steps is
given by the difference
hn = 1 − f2 − f4 − · · · − f2n .
The probability f2n , that the first return to the origin occurs at step 2n is the coefficient of sm in the
expansion
∞ 1
1
X
F (s) = 1 − (1 − s2 ) 2 = − 2 (−s2 )n .
n
n=1
2n
The coefficient of s is f2n . It follows that
1
1 n+1 2
(2n − 3)!
f2 = , f2n = (−1) = , (n = 2, 3, . . .),
2 n n!(n − 2)!22n−2
where 0! = 1.
The difference
hn = 1 − f2 + f4 − · · · − f2n
1 1 1
2 2 n+1 2
= 1− + − · · · − (−1)
1 2 n
Hence hn satisfies the difference equation

1

(2n − 1)!
hn+1 − hn = (−1)n 2 =− .
n+1 (n + 1)!(n − 1)!22n
The homogeneous part of the difference equation has a constant solution C, say. For the particular solution
try the choice suggested in the question, namely

2n 1
hn = .
n 22n
Then

2n + 2 1 2n 1
hn+1 − hn = −
n + 1 22n+2 n 22n

2n 1 1
= −
n 22n+1 (n + 1)

(−1)n+1 − 21
= (using the binomial identity before (3.7))
2(n + 1) n
1

= (−1)n+1 2
n+1
37
Hence
1 2n
hn = C + .
22n n
The initial condition is h1 = 21 , from which it follows that C = 0. Therefore the probability that no return
to the origin has occurred in the first 2n steps is

1 2n
hn = 2n .
2 n
3.9. A walk can be represented as a connected graph between coordinates (n, y) where the ordinate y is the
position on the walk, and the abscissa n represents the number of steps. A walk of 7 steps which joins
(0, 1) and (7, 2) is shown in Fig. 3.1. Suppose that a walk starts at (0, −y1 ) and finishes at (n, y2 ), where
Figure 3.1: Representation of a random walk
y1 > 0, y2 > 0 and n + y2 − y1 is an even number. Suppose also that the walk first visits the origin at
n = n1 . Reflect that part of the path for which n ≤ n1 in the n-axis (see Fig. 3.1), and use a reflection
argument to show that the number of paths from (0, y1 ) to (n, y2 ) which touch or cross the n-axis equals
the number of all paths from (0, −y1 ) to (n, y2 ). This is known as the reflection principle.
All paths from (0, −y1 ) to (n, y2 ) must cut the n axis at least once. Let (n1 , 0) be the first such contact
with n axis. Reflect the path for n ≤ n1 and y ≤ 0 in the n axis. The result is a path from (0, y1 ) to
(n, y2 ) which touches or cuts the n axis at least once. All such paths must be included.
3.10. A walk starts at (0, 1) and returns to (2n, 1) after 2n steps. Using the reflection principle (see Problem
3.9) show that there are
(2n)!
n!(n + 1)!
different paths between the two points which do not ever revisit the origin. What is the probability that
the walk ends at (2n, 1) after 2n steps without ever visiting the origin, assuming that the random walk is
symmetric?
Show that the probability that the first visit to the origin after 2n+1 steps is
1 (2n)!
pn = .
22n+1 n!(n + 1)!
38
Let M (m, d) represent the total number of different paths in the (n, y) plane (as in Fig. 3.1) which are
of length m joining positions denoted by y = y1 and y = y2 : here d is the absolute difference d = |y2 − y1 |.
The total number of paths from (0, 1) to (0, 2n) is

2n
M (2n, 0) = .
n
By the reflection principle (Problem 3.9) the number of paths which cross the n axis (that is, visit the
origin) is
2n
M (2n, 2) = .
n−1
Hence the number of paths from (0, 1) to (0, 2n) which do not visit the origin is

2n 2n (2n)! (2n)!
M (2n, 0) − M (2n, 2) = − = −
n n−1 n!n! (n − 1)!(n + 1)!
(2n)!
=
n!(n + 1)!
The total number of paths is 22n . Also to visit the origin for the first time at step 2n + 1, the walk
must be at y = 1 at step 2n, from where there is a probability of 12 that the walk moves to the origin.
Hence the probability is
1 (2n)! 1 1 (2n)!
pn = 2n = 2n+1 .
2 n!(n + 1)! 2 2 n!(n + 1)!
3.11. A symmetric random walks starts at the origin. Let fn,1 be the probability that the first visit to
position x = 1 occurs at the n-th step. Obviously, f2n,1 = 0. The result from Problem 3.10 can be adapted
to give
1 (2n)!
f2n+1,1 = 2n+1 , (n = 0, 1, 2, . . .).
2 n!(n + 1)!
Suppose that its pgf is
∞
X
G1 (s) = f2n+1,1 s2n+1 .
n=0
Show that 1
G1 (s) = [1 − (1 − s2 ) 2 ]/s.
[Hint: the identity
1
1 (2n)!
= (−1)n 2 , (n = 0, 1, 2, . . .)
22n+1 n!(n + 1)! n+1
is useful in the derivation of G1 (s).]
Show that any walk starting at the origin is certain to visit x > 0 at some future step, but that the
mean number of steps in achieving this is infinite.
The result
1 (2n)!
f2n+1,1 =
22n+1 n!(n + 1)!
is simply the result in the last part of Problem 3.10.
Its pgf is
∞ ∞
X X 1 (2n)!s2n+1
G1 (s) = f2n+1,1 s2n+1 =
22n+1 n!(n + 1)!
n=0 n=0
The identity before eqn (3.7) (in the book) states that

2n − 21 2n
= (−1)n 2 .
n n
39
Therefore, using this result

(2n)! 2n 1 − 21 1 1
= = (−1)n = (−1)n 2 .
22n+1 n!(n + 1)! n 22n+1 (n + 1) n 2(n + 1) n+1
Hence
∞
X 1

G1 (s) = (−1)n 2 s2n+1
n+1
s=0
1 1 1
3
= 2 s− 2 s + 2 s5 − · · ·
1 2 3
1 1
= [1 − (1 − s2 ) 2 ]
s
using the binomial theorem.
That G1 (1) = 1 implies that the random walk is certain to visit x > 0 at some future step. However,
G′ (1) = ∞ which means that expected number of steps to that event is infinite.
3.12. A symmetric random walk starts at the origin. Let fn,x be the probability that the first visit to position
x occurs at the n-th step (as usual, fn,x = 0 if n + x is an odd number). Explain why
n−1
X
fn,x = fn−k,x−1 fk,1 , (n ≥ x > 1).
k=1
If Gx (s) is its pgf, deduce that

Gx (s) = {G1 (s)}x ,
where G1 (s) is given explicitly in Problem 3.11. What are the probabilities that the walk first visits x = 3
at the steps n = 3, n = 5 and n = 7?
Consider k = 1. The first visit to x − 1 has probability fn−1,x−1 in n − 1 steps. Having reached there
the walk must first visit x in one further step with probability f1,1 . Hence the probability is fn−1,x−1 f1,1 .
If k = 2, the first visit to x − 1 in n − 2 steps occurs with probability fn−2,x−1 : its first visit to x must
occur after two steps. Hence the probability is fn−2,x−1 f2,1 . And so on. The sum of these probabilities
gives
n−1
X
fn,x = fn−k,x−1 fk,1 , (n ≥ x > 1).
k=1
Multiply both sides of this equation by sn and sum over n from n = x:
∞ n−1
X X
Gx (s) = fn−k,x−1 fk,1 sn = Gx−1 (s)G1 (s).
n=x k=1
By repeated application of this difference equation, it follows that

h n oix
1 1
Gx (s) = 1 − (1 − s2 ) 2 .
s
For x = 3,
h1 n 1
oi3
G1 (s) = 1 − (1 − s2 ) 2
.
s
Expansion of this function as a Taylor series in s gives the coefficients and probabilities:
1 3 1 5 1 7
G2 (s) = s + s + s + O(s9 ).
4 8 64
3.13. Problem 3.12 looks at the probability of a first visit to position x ≥ 1 at the n-th step in a symmetric
random walk which starts at the origin. Why is the pgf for the first visit to position x where |x| ≥ 1 given
by
Gx (s) = {G1 (s)}|x| ,
40
where F1 (s) is defined in Problem 3.11?
First visits to x > 0 and to −x at step n must be equally likely. Hence fn,x = fn,−x . Therefore
1
Gx(s) = [1 − (1 − s2 ) 2 ]|x| .
3.14. An asymmetric walk has parameters p and q = 1 − p 6= p. Let gn,1 be the probability that the first visit
to x = 1 occurs at the n-th step. As in Problem 3.11, g2n,1 = 0. It was effectively shown in Problem 3.10
that the number of paths from the origin, which return to the origin after 2n steps is
(2n)!
.
n!(n + 1)!
Explain why
(2n)!
g2n+1,1 = pn+1 q n .
n!(n + 1)!
Suppose that its pgf is
∞
X
G1 (s) = g2n+1,1 s2n+1 .
n=0
Show that 1
G1 (s) = [1 − (1 − 4pqs2 ) 2 ]/(2qs).
(The identity in Problem 3.11 is required again.)
What is the probability that the walk ever visits x > 0? How does this result compare with that for the
symmetric random walk?
What is the pgf for the distribution of first visits of the walk to x = −1 at step 2n + 1?
The probability that the first return to x = 1 at the (2n + 1)th step is g2n+1,1 The number of paths of
length 2n which never visit x = 1 is (adapt the answer in Problem 3.10),
(2n)!
.
n!(n + 1)!
The consequent probability of this occurrence is, since there are n steps to the right with probability p
and to the left with probability q,
(2n)!
pn q n .
n!(n + 1)!
the probability that the next step visits x = 1 is
(2n)!
pn+1 q n ,
n!(n + 1)!
which is the previous probability multiplied by p. Using the identity in Problem 3.11,
1

g2n+1,1 = 2 (−1)n (2p)n (2q)n 2p.
n+1
Its pgf G1 (s) is given by

∞ 1

X 1
G1 (s) = 2 (−1)n (4pq)n 2ps2n+1 = [1 − (1 − 4pqs2 ) 2 ]/(2qs)
n+1
n=0
using a binomial expansion.

Use an argument that any walk which enters x > 0 must first visit x = 1 as follows. The probability
that the walks first visit to x = 1 occurs at all is
∞
X 1 1 1
g2n+1,1 = G1 (1) = [1 − {(p + q)2 − 4pq} 2 ] = [1 − |p − q|].
2q 2q
n=0
41
A symmetry argument in which p and q are interchanged gives the pgf for the distribution of first visits
to x = −1, namely
1
H1 (s) = [1 − (1 − 4pqs2 ) 2 ]/(2ps).
3.15. It was shown in Section 3.3 that, in a random walk with parameters p and q = 1 − p, the probability
that a walk is at position x at step n is given by

n 1 1
vn,x = 1 p 2 (n+x) q 2 (n−x) , |x| ≤ n,
2
(n + x)
1
where 2
(n + x) must be an integer. Verify that vn,x satisfies the difference equation
vn+1,x = pvn,x−1 + qvn,x+1 ,
subject to the initial conditions

v0,0 = 1, vn,x = 0, (x 6= 0).
Note that this difference equation has differences of two arguments.
Can you develop a direct argument which justifies the difference equation for the random walk?
Given
n 1 1
vn,x = 1 p 2 (n+x) q 2 (n−x) , |x| ≤ n,
2
(n + x)
then

n 1 1
pvn,x−1 + qvn,x+1 = p 1 p 2 (n+x−1) q 2 (n−x+1)
2
(n + x − 1)

n 1 1
+q 1 p 2 (n+x+1) q 2 (n−x−1)
2
(n + x + 1)

1 1 n n
= p 2 (n+x+1) q 2 (n−x+1) 1 + 1
2
(n + x − 1) 2
(n + x + 1)
1 1
p 2 (n+x+1) q 2 (n−x+1) n!
= [ 1 (n + x + 1) + 21 (n − x + 1)]
[ 12 (n + x + 1)]![ 12 (n − x + 1)]! 2

1 1 n+1
= p 2 (n+x+1) q 2 (n−x+1) 1 = vn+1,x .
2
(n + x)
3.16. In the usual notation, v2n,0 is the probability that, in a symmetric random walk, the walk visits the
origin after 2n steps. Using the difference equation from Problem 3.15, v2n,0 satisfies
v2n,0 = 21 v2n−1,−1 + 12 v2n−1,1 = v2n−1,1 .
How can the last step be justified? Let

∞
X
G1 (s) = v2n−1,1 s2n−1
n=1
be the pgf of the distribution {v2n−1,1 }. Show that

1
G1 (s) = [(1 − s2 )− 2 − 1]/s.
By expanding G1 (s) as a power series in s show that

2n − 1 1
v2n−1,1 = .
n 22n−1
42
By a repetition of the argument show that
∞
X 1
G2 (s) = v2n,2 s2n = [(2 − s2 )(1 − s2 )− 2 − 2]/s2 .
n=0
Use a symmetry argument. Multiply both sides of the difference equation by s2n and sum from n = 1
to infinity. Then
∞
X ∞
X
v2n,0 s2n = s v2n−1,1 s2n−1 .
n=1 n=1
Therefore in the notation in the problem
H(s) − 1 = sG1 (s),

1
where H(s) = (1 − s2 ) . Therefore
2
∞
X
G1 (s) = v2n−1,1 s2n−1
n=1
as required.
From the series for G1 (s) expanded as a binomial series, the general coefficient is

− 12 2n 1 (2n)! 2n − 1 1
v2n−1,1 = (−1)n = = 2n = .
n n 22n 2 n!n! n 22n−1
From the difference equation

v2n+1,1 = 21 v2n,0 + 21 v2n,2 .
Multiplying by s2n+1 and summing over n
∞ ∞
1X 1X 1 1
G1 (s) = v2n,0 s2n+1 + v2n,2 s2n+1 = sH(s) + sG2 (s).
2 2 2 2
n=0 n=0
Therefore 1
G2 (s) = [(2 − s2 )(1 − s2 )− 2 − 2]/s2 .
Figure 3.2: The cyclic random walk of period n for Problem 3.17.
3.17. A random walk takes place on a circle which is marked out with n positions. Thus, as shown in
Fig. 3.2, position n is the same as position O. This is known as a cyclic random walk of period n. A
symmetric random walk starts at O. What is the probability that the walk is at O after j steps in the cases:
(a) j < n;
43
(b) n ≤ j < 2n?
Distinguish carefully the cases in which j and n are even or odd.
(a) j < n. The walk cannot circumscribe the circle. This case is the same as the walk on a line. Let
pj be the probability that the walk is at O at step j. Then by (3.6)

 n 1
1 n even
pn = vn,0 = 2
n 2n

0 n odd
(b) n ≤ j < 2n. Since n can be reached in both clockwise and counterclockwise directions,
pj = vj,0 + vj,n + vj,−n


j 1 j 1 j 1

 + + (j, n both even)
 1 j 2j

2
1
(j + n) 2j
2
1
(j − n) 2j 2
= 0
(j odd, n even, or j even, n odd)

 j 1 j 1

 1 + (j, n both odd)
1
2
(j + n) 2j 2
(j − n) 2j
3.18. An unrestricted random walk with parameters p and q starts from the origin, and lasts for 50 paces.
Estimate the probability that the walk ends at 12 or more paces from the origin in the cases:
(a) p = q = 12 ;
(b) p = 0.6, q = 0.4.
Consult Section 3.2. From (3.2)
Xn − n(p − q)
Zn = √ ≈ N (0, 1),
4npq
where Xn is the random variable of the position of the random walk at step n. Since n = 50 is discrete we
use the approximation
P(−11 ≤ X50 ≤ 11) ≈ P(−11.5 < X50 < 11.5).
(a) Symmetric random walk: p = q = 12 . Then
11.5 X50 11.5

−1.626 = − √ < Z50 = √ < √ = 1.626.
50 50 50
Hence
P(−1.626 < Z50 < 1.626) = −Φ(−1.626) + Φ(1.626) = 2Φ(1.626) − 1 = 0.896.
Therefore the probability that the final position is 12 or more paces from the origin is approximately
1 − 0.896 = 0.104 approximately.
(b) p = 0.6, q = 0.4. The bounds on Z50 are given by
−11.5 − 10 X50 − 10 11.5 − 10
−3.103 = √ < Z50 = √ < √ = 0.217.
48 48 48
Hence
P(−3.103 < Z50 < 0.217) = Φ(0.217) − Φ(−3.103) = 0.585.
The probability that the final position is 12 or more paces from the origin is approximately 1−0.585 = 0.415.
3.19. In an unrestricted random walk with parameters p and q, for what value of p are the mean and
variance of the probability distribution of the position of the walk at stage n the same?
From Section 3.2 the mean and variance of Xn , the random variable of the position of the walk at step
n, are given by
E(Xn ) = n(p − q), V(Xn ) = 4npq,
44
where q = 1 − p. The mean and variance are equal if 2p − 1 = 4p(1 − p), that is, if
4p2 − 2p − 1 = 0.
√
The required probability is p = 41 (1 + 5).
3.20. Two walkers each perform symmetric random walks with synchronized steps both starting from the
origin at the same time. What is the probability that they are both at the origin at step n?
If A and B are the walkers, then the probability an that A is at the origin is given by

 n 1
1 (n even)
an = 2
n 2n .

0 (n odd)
The probability bn for B is given by the same formula. They can only visit the origin if n is even, in which
case the probability that they are both there is
2
n 1
a n bn = 1 .
2
n 22n
3.21. A random walk takes place on a two-dimensional lattice as shown in Fig. 3.3. In the example
shown the walk starts at (0, 0) and ends at (2, −1) after 13 steps. In this walk direct diagonal steps are not
Figure 3.3: A two-dimensional random walk.
permitted. We are interested the probability that, in the symmetric random walk, which starts at the origin,
has returned there after 2n steps. Symmetry in the two-dimensional walk means that there is a probability of
1
4
that, at any position, the walk goes right, left, up, or down at the next step. The total number of different
walks of length 2n which start at the origin is 42n . For the walk considered, the number of right steps
(positive x direction) must equal the number of left steps, and the number of steps up (positive y direction)
must equal those down. Also the number of right steps must range from 0 to n, and the corresponding steps
up from n to 0. Explain why the probability that the walk returns to the origin after 2n steps is
n
(2n)! X 1
p2n = .
42n [r!(n − r)!]2
r=0
Prove the two identities

2 n 2
(2n)! 2n n 2n X n
= , = .
[r!(n − r)!]2 n r n r
r=0
45
[Hint: compare the coefficients of xn in (1 + x)2n and [(1 + x)n ]2 .] Hence show that
2
1 2n
p2n = 2n .
4 n
Calculate p2 , p4 , 2/(πp20 ), 2/(πp40 ). How do you would you guess that p2n behaves for large n?
At each intersection there are 4 possible paths. Hence there are 42n different paths which start the
origin.
For a walk which returns to the origin there must be r, say, left and right steps, and n − r up and down
steps (r = 0, 1, 2, . . . , n) to ensure the return. For fixed r, the number of ways in which r left, r right,
(n − r) up and (n − r) down steps can be chosen from 2n is the multinomial formula
(2n)!
.
r!r!(n − r)!(n − r)!
For all r, the total number of ways is
n
X (2n)!
.
r!r!(n − r)!(n − r)!
r=0
Therefore the probability that a return to the origin occurs is

n
(2n)! X 1
p2n = .
42n [r!(n − r)!]2
r=0
For example if n = 2, then

2
4! X 1 4! 1 1 3
p4 = = + 1 + = .
44 r!(2 − r)! 44 2 2 16
r=0
For the first identity

2 2
(2n)! (2n)! n! 2n n
= = .
[r!(n − r)!]2 n!n! r!(n − r)! n r
For the second identity

2n
= the coefficient of xn in the expansion of (1 + x)2n
n
= the coefficient of xn in the expansion of [(1 + x)n ]2
2
n n 2 n n
= the coefficient of xn in 1+ x+ x + ··· + x
1 2 n

n n n n n n
= + + ··· +
0 n 1 n−1 n 0
2 2 2
n n n n n
= + + ··· since =
0 1 n r n−r
n 2
X n
=
r
r=0
Hence 2
1 2n
p2n = 2n .
4 n
The computed values are p20 = 0.0310 . . . and p40 = 0.01582 . . .. Then
2 2
= 10.12, = 20.12,
πp20 πp40
46
approximately, which imply that possibly p2n ∼ 2/(nπ) as n → ∞. However, this conjecture would require
further investigation.
3.22. A random walk takes place on the positions {. . . , −2, −1, 0, 1, 2, . . .}. The walk starts at 0. At step
n, the walker has a probability qn of advancing one position, or a probability 1 − qn of retreating one step
(note that the probability depends on the step not the position of the walker). FindP∞ the expected position of
the walker at step n. Show that if qn = 21 + rn , (− 12 < rn < 12 ), and the series r is convergent, then
j=1 j
the expected position of the walk will remain finite as n → ∞.
If Xn is the random variable representing the position of the walker at step n, then
P(Xn+1 = j + 1|Xn = j) = qn , P(Xn+1 = j − 1|Xn = j) = 1 − qn .
If Wi is the modified Bernoulli random variable (Section 3.2), then

n n n
X X X
E(Xn ) = E Wi = E(Wi ) = [1.qi + (−1)(1 − qi )]
i=1 i=1 i=1
n
X
= 2 qi − n.
i=1
1
Let qn = 2
+ rn , (− 21 < rn < 21 ). Then
n n
X X
E(Xn ) = 2 ( 21 + ri ) − n = ri .
i=1 i=1
Hence E(Xn ) is finite as n → ∞ if the series on the right is convergent.
3.23. A symmetric random walk starts at k on a position chosen from 0, 1, 2, . . . , a, where 0 < k < a. As in
the gambler’s ruin problem, the walk stops whenever 0 or a is first reached. Show that the expected number
of visits to position j where 0 < j < k is 2j(a − k)/a before the walk stops.
One approach is to this problem by repeated application of result (2.5) for gambler’s ruin.
A walk which starts at k first reached j before a with probability (by (2.5))
(a − j) − (k − j) a−k
p= = .
a−j a−j
A walk which starts at j reaches a (and stops) before reaching j (again by (2.5)) with probability
1 1 1
= ,
2a−j 2(a − j)
and reaches 0 before returning to j with probability
j − (j − 1) 1
= .
2j 2j
Hence the probability that the walk from j stops without returning to j with probability
1 1 a
q= + = .
2(a − j) 2j 2j(a − j)
Given that the walk is at j, it next visits j with probability
a 2j(a − j) − a
r =1−q =1− = .
2j(a − j) 2j(a − j)
Therefore the probability that the walk starts from k visits j m times before stopping is
hm = pr m−1 q.
47
The expected number of visits to j is
∞ ∞
X X pq
µ= mhm = pq mr m−1 = ,
(1 − r)2
m=1 m=1
summing the quasi-geometric series. Substituting for p, q, and r,

−2
a−k a 2j(a − j) − a
µ = · 1−
a − j 2j(a − j) 2j(a − j)
a(a − k) 4j 2 (a − j)2 2(a − k)j
= · =
2j(a − j)2 a2 a
48
Chapter 4
Markov chain
4.1. If T = [pij ], (i, j = 1, 2, 3) and

i+j
pij =
,
6 + 3i
show that T is a row-stochastic matrix. What is the probability that a transition between states E2 and E3
occurs at any step?
If the initial probability distribution in a Markov chain is

p(0) = 1
2
1
4
1
4
,
what are the probabilities that states E1 , E2 and E3 are occupied after one step. Explain why the probability
that the chain finishes in state E2 is 13 irrespective of the number of steps.
Since
i+j
pij = ,
6 + 3i
then
3 3
X X i+j 3i 6
pij = = + = 1,
6 + 3i 6 + 3i 6 + 3i
j=1 j=1
for all i. Also 0 < pi,j < 1. Therefore T is a stochastic matrix. The probability that a transition from E2
5
to E3 occurs is p23 = 12 .
The probabilities that the states E1 , E2 and E3 are occupied after one step given are given by
 2 1 4 
9 3 9

p(1) = p(0) T = 1
2
1
4
1
4
 1
4
1
3
5
12
= 173
720
1
3
307
720
4 1 2
15 3 5
Each term in the second column of T is a 31 . By row-on-column matrix multiplication, each element
in the second column of T n is a 31 . Hence the second term in p(0) T n is 31 independently of p(0) .
4.2. If
 1 1 1 
2 4 4
1 1 1
T = 3 3 3
,
1 1 1
4 2 4
(2) (2) (2)
calculate p22 , p31 and p13 .
(2)
pij are the elements of T 2 . The matrix multiplication gives
 19 1 13 
48 3 48
T2 =  13
36
13
36
5
18
.
17 17 7
48 48 24
49
Therefore
(2) 13 (2) 17 (2) 13
p22 = , p31 = , p13 = .
36 48 48
4.3. For the transition matrix

1 2
3 3
T = 1 3
4 4
(3) (2)

calculate p12 , p2 and p(3) given that p(0) = 1
2
1
2
. Also find the eigenvalues of T , construct a
formula for T n and obtain limn→∞ T n .
We require
1 2
2 5 13
1 2
3 59 157

T2 = 3
1
3
3
= 18
13
18
35
, T3 = 3
1
3
3
= 216
157
216
419
.
4 4 48 48 4 4 576 576
(3)
Directly from T 3 , p12 = 216
157
.
The element can be read off from
5 13

p(2) = p(0) T 2 = 1
2
1
2
18
13
18
35
= 79 209
,
288 288
,
48 48
(2)
namely p2 = 209
288
.
The vector
59 157
(3) (0) 3 1 1 216 216 943 2513
p =p T = 2 2 157 419
= 3456 3456
576 576
The eigenvalues of T are given by

1 2
−λ
3
1 3
3 = 0, or, 12λ2 − 13λ + 1 = 0.
4 4
−λ
1
The eigenvalues are λ1 = 1, λ2 = 12
. Corresponding eigenvectors are given by the transposes
T T
r1 = 1 1 , r2 = − 83 1
The matrix C is defined as

1 − 38
C= r1 r2 = .
1 1
By (4.18),
T n = CDn C −1 ,
where D is the diagonal matrix of eigenvalues. Therefore
3 8
8 8

n 1 − 38 1 0 11 11 1 3+ 12n
8− 12n
T = 1 3 3
= 3 3
.
1 1 0 12n
− 11 11
11 3− 12n
8+ 12n
It follows that
n 1 3 8
lim T = .
n→∞ 11 3 8
4.4. Sketch transition diagrams for each of the following three-state Markov chains.
" 1 1 1 # " 1 1 1 # " 1 1 #
3 3 3 2 4 4
0 2 2
(a) A = 0 0 1 ; (b) B = 0 1 0 ; (c) C = 1 0 0 .
1 1 1 1 1
1 0 0 2 2
0 3 3 3
50
(a) (b) (c)
1
3 1
E1 2 E1 E1
1
3 1
1
1 4
1 2
1
3 4 1
2 1
1 3
1 2
1 1
E2 1 E3 E2 2 E3 E2 3 E3
1
1 3
Figure 4.1: Transition diagrams for Problem 4.4
The transition diagrams are shown in Figure 4.1.
4.5. Find the eigenvalues of

" #
a b c
T = c a b , (a > 0, b > 0, c > 0).
b c a
Show that the eigenvalues are complex if b 6= c. (If a+b+c = 1, then T is a doubly- stochastic matrix.)
Find the eigenvalues and eigenvectors in the following cases:
(a) a = 21 , b = 14 , c = 41 ;
(b) a = 12 , b = 18 , c = 83 .
The eigenvalues are given by

a−λ b c

c a−λ b = 0,
b c a−λ
or
(a + b + c − λ)(λ2 + (b + c − 2a)λ + a2 + b2 + c2 − bc − ca − ab) = 0.
The eigenvalues are
1 √
λ1 = a + b + c, λ2,3 =
(2a − b − c ± i 3|b − c|).
2
(a) a = 21 , b = 14 , c = 41 . The eigenvalues are λ1 = 1, λ2 = λ3 = 14 . The eigenvectors are
" # " # " #
1 −1 −1
r1 = 1 , r2 = 1 , r3 = 0 .
1 0 1
(b) a = 41 , b = 18 , c = 83 . The eigenvalues are

√ √
1 3 1 3
λ1 = 1, λ2 = +i , λ3 = −i ,
4 8 4 8
The eigenvectors are
 √   √ 
" #
1 − 21 − i √23 − 21 + i √23
r1 = 1 , r2 =  − 1 + i 3  , r3 =  − 1 − i 3  .
2 2 2 2
1 1 1
51
4.6. Find the eigenvalues, eigenvectors, the matrix of eigenvectors C, its inverse C −1 , a formula for T n
and limn→∞ T n for each of the following transition matrices;
(a) 1 7
T = 8 8 ;
1 1
2 2
(b)
" 1 1 3 #
2 8 8
1 3 3
T = 4 8 8
.
1 5 1
4 8 8
(a) The eigenvalues are λ1 = 1, λ2 = − 38 . The corresponding eigenvectors are

1 − 47
r1 = , r2 = .
1 1
The matrix C and its inverse are given by

1 − 47 4
11
7
11
C= r1 r2 = , C −1 = 4 4
.
1 1 − 11 11
The matrix T n is given by

n
n n −1
1 − 74 1 0
4
11
7
11
T = CD C =
1 1 0 − 38 4
− 11 4
11

1 4 + 7(− 38 )n 7 − 7(− 38 )n 1 4 7
= →
11 4 − (− 83 )n 7 + (− 83 )n 11 4 7
as n → ∞.
(b) The eigenvalues are λ1 = − 41 , λ2 = 41 , λ3 = 1, and the corresponding eigenvectors are
 
− 73 "
−2
# "
1
#
r1 =  − 73 , r2 = 1 , r3 = 1
1 1 1
The matrix C is given by

 
− 73 −2 1

C= r1 r2 r3 =  − 37 1 1 
1 1 1
Then
 " # 0 
− 73 −2 1 (− 41 )n 0 0
7
− 10 7
10
Tn = CDn C −1 =  − 37 1 1  0 ( 41 )n 0  − 13 1
3
1 
1 1 1 0 0 1 1 11 3
3 30 10
 
1
3
+ 13 21−2n 11
30
+ 35 (−1)n 2−1−2n − 13 21−2n 3
10
− 35 (−1) 2 n −1−2n
 n 
1
− 43 11
+ 35 (−1)n 2−1−2n + 4 3 3
− 35 (−1)n 2−1−2n  ,
−n
=  3 30 10
1
− 43 11
− 75 (−1)n 2−1−2n + 4 3 3
+ 75 (−1)n 2−1−2n
−n −n
3 30 10
so that  
1 11 3
3 30 10
lim T n =  1
3
11
30
3
10
.
t→∞
1 11 3
3 30 10
52
4.7. The weather in a certain region can be characterized as being sunny(S), cloudy(C) or rainy(R) on any
particular day. The probability of any type of weather on one day depends only on the state of the weather
on the previous day. For example, if it is sunny one day then sun or clouds are equally likely on the next
day with no possibility of rain. Explain what other day-to-day possibilities are if the weather is represented
by the transition matrix.
S C R
1 1
S 2 2
0
T = 1 1 1
C 2 4 4
1 1
R 0 2 2
Find the eigenvalues of T and a formula for T n . In the long run what percentage of the days are sunny,
cloudy and rainy?
The eigenvalues of T are given by

1 1
−λ 0
2
1 1
2
1 1
2 4
−λ 4 = − (4λ + 1)(2λ − 1)(λ − 1) = 0.
1 1 8
0 2 2
−λ
Let λ1 = − 14 , λ2 = 1
2
and λ3 = 1. The coresponding eigenvectors are
  " # " #
1 − 21 1
r1 =  − 23  , r2 = 0 , r3 = 1
1 1 1
The matrix of eigenvectors C is given by

 
1 − 12 1

C= r1 r2 r3 =  − 23 0 1 
1 1 1
If D is the diagonal matrix of eigenvalues, then by (4.18)

 " # 4 
1 − 21 1 (− 41 )n 0 0 15
− 25 2
15
n n −1
T = CD C =  − 32 0 1  0 ( 12 )n 0  − 32 0 2
3

1 1 1 0 0 1 2 2 1
5 5 5
As n → ∞
 " # 
1 − 21 1 0 0 0
4
15
− 52 2
15
"
2 2 1
#
T →n
− 32 0 1  0 0 0  − 32 0 3
2 = 1 2 2 1 .
5
1 1 1 0 0 1 2 2 1 2 2 1
5 5 5
In the long run 40% of the days are of the days are sunny, 40% are cloudy and 20% are rainy.
4.8. The eigenvalue method of Section 4.4 for finding general powers of stochastic matrices is only guaran-
teed to work if the eigenvalues are distinct. Several possibilities occur if the stochastic matrix of a Markov
chain has a repeated eigenvalue. The following three examples illustrate these possibilities.
(a) Let
" 1 1 1 #
4 4 2
T = 1 0 0
1 1 1
2 4 4
be the transition matrix of a three-state Markov chain. Show that T has the repeated eigenvalue λ1 = λ2 =
− 41 and λ3 = 1, and two distinct eigenvectors
" # " #
1 1
r1 = −4 r3 = 1 .
1 1
53
In this case diagonalization of T is not possible. However it is possible to find a non-singular matrix C
such that
T = CJC −1 ,
where J is the Jordan decomposition matrix given by
" # " #
λ1 1 0 − 41 1 0
J= 0 λ1 0 = 0 − 41 0 ,
0 0 1 0 0 1

C= r1 r2 r3 ,
and r2 satisfies
(T − λ1 I3 )r2 = r1 .
Show that we can choose " #
−10
r2 = 24 .
0
Find a formula for J n and confirm that, as n → ∞,
 12 1 8 
25 5 25
Tn →  12
25
1
5
8
25
.
12 1 8
25 5 25
(b) A four-state Markov chain has the transition matrix

 
1 0 0 0
3 1
 4
0 4
0 
S= 1 3 .
0 4
0 4
0 0 0 1
Sketch the transition diagram for the chain, and note that the chain has two absorbing states and is therefore
not a regular chain. Show that the eigenvalues of S are − 41 , 14 and 1 repeated. Show that there are four
distinct eigenvectors. Choose the diagonalizing matrix C as
 
0 0 −4 −5
 −1 1 −3 4 
C= .
1 1 0 1 
0 0 1 0
Find its inverse, and show that, as n → ∞,

 
1 0 0 0
4 1
0 0
Sn → 
 5 5 
1
0 0 4 .
5 5
0 0 0 1
Note that since the rows are not the same this chain does not have an invariant distribution: this is caused
by the presence of two absorbing states.
(c) Show that the transition matrix
" 1 #
2
0 21
1 1 1
U= 6 3 2
1 5
6
0 6
has a repeated eigenvalue, but that, in this case, three independent eigenvectors can be associated with U .
Find a diagonalizing matrix C, and find a formula for U n using U n = CDn C −1 , where
" 1 #
3
0 0
1
D= 0 3
0 .
0 0 1
54
Confirm also that this chain has an invariant distribution.
(a) The eigenvalues are given by

1 1 1
−λ
4 4 2 1 2
1 −λ 0 = − (λ − 1)(1 + 4λ)
1 1 1 16
2 4 4
−λ
Hence they are λ1 = − 41 (repeated) and λ2 = 1. This leads to the two eigenvectors
" # " #
1 1
r1 = −4 , r3 = 1
1 1
The Jordan decomposition matrix is given by
" #
− 41 1 0
J= 0 − 14 0 .
0 0 1
Let r2 satisfy    
1 1 1
2 4 2
1
1
[T − λ1 I3 ]r2 = r1 or  1 4
0  r2 =  −4  .
1 1 1
2 4 2
1
The solution for the linear equations for the components of r2 gives
T
r2 = −10 24 0 .
The matrix C is defined in the usual way as

" #
1 −10 1
C= r1 r2 r3 = −4 24 1 .
1 0 1
Its computed inverse is
 12 
− 25 − 51 17
25
C −1 =  − 10
1
0 1
10
.
12 1 8
25 5 25
If T = CJC −1 , then T n = CJ n C −1 , where

 n  
− 41 1 0 (− 14 )n n(− 14 )n−1 0
Jn =  0 − 14 0  = 0 (− 41 )n 0 .
0 0 1 0 0 1
As n → ∞,
   12 
1 −10 1 0 0 0 − 25 − 51 17
25
Tn →  −4 24 1  0 0 1
0   − 10 0 1
10

12 1 8
1 0 1 0 0 1 25 5 25
 12 1 8 
25 5 25
12 1 8
=  25 5 25
.
12 1 8
25 5 25
(b) The transition diagram is shown in Figure 4.2. The eigenvectors are given by

1−λ 0 0 0

3 1
4 −λ 4
0
1 2
0 1
−λ 3 = (λ − 1) (4λ − 1)(4λ + 1) = 0.
4 4 16
0 0 0 1−λ
55
3
E1 4 E2
1
4
1
4
E3
3 E4 1
4
Figure 4.2: Transition diagram for Problem 4.8(b).
The eigenvalues are λ1 = − 41 , λ2 = 41 , λ3 = 1 (repeated). The eigenvectors for λ1 and λ2 are

T T
r1 = 0 −1 1 0 , r2 = 0 1 1 0 .
Let the eigenvector for the repeated λ3 be

T
r3 = a b c d ,
where the constants a, b, c, d satisfy

3
4
a − b + 14 c = 0, 1
4
b − c + 43 d = 0.
We can express the solution in the form
c = −3a + 4b, d = 12a − 15b.
Hence the eigenvector is  

a
 b 
r= ,
−3a + 4b 
−4a + 5b
which contains two arbitrary constants a and b. In this case of a repeated eigenvalue, two eigenvectors can
be defined using different pairs of values for
T T
r3 = −4 −3 0 1 , r4 = 5 4 1 0 .
The matrix C and its inverse are

   
0 0 −4 5 3 −5 5 −3
 −1 1 −3 4  1  −5 5 5 −5 
C= , C −1 = .
1 1 0 1  10
 0 0 0 1 
0 0 1 0 2 0 0 8
The matrix power of T is given by
Tn = CDn C −1
   
0 0 −4 5 (− 41 )n 0 0 0 3 −5 5 −3
1  −1 1 −3 4  0 ( 14 )n 0 0   −5 5 5 −5 
=  1
10 1 0 1  0 0 1 0  0 0 0 1 
0 0 1 0 0 0 0 1 2 0 0 8
 
1 0 0 0
4 1
 5
0 0 5

→ 
1 4


5
0 0 5

0 0 0 1
56
as n → ∞
(c) The eigenvalues are given by
1 1
−λ 0
2
1 1
2
1 1 2
6 3
−λ 2 = − (λ − 1)(3λ − 1) .
1 5 9
6
0 6
−λ
1
Hence the eigenvalues are λ1 = 3
(repeated) and λ3 = 1. Corresponding to the eigenvalue λ1 , we can
obtain the eigenvector
" # " # " #
−3b −3 0
r1 = a =b 0 +a 1 ,
b 1 0
where a and b are arbitrary. Two distinct eigenvectors can be obtained by putting a = 0, b = 1 and by
putting a = 1, b = 0. The three eigenvectors are
" # " # " #
−3 0 1
r1 = 0 , r2 = 1 , r3 = 1 .
1 0 1
The matrix C and its inverse become

" # " #
−3 0 1 −1 0 1
1
C= 0 1 1 , C −1 = −1 4 −3 .
4
1 0 1 1 0 3
With " #
1
3
0 0
1
D= 0 3
0 ,
0 0 1
then
" #" # " #
−3 0 1 ( 31 )n 0 0 −1 0 1
n 1
U = 0 1 1 0 ( 13 )n 0 −1 4 −3
4
1 0 1 0 0 1 1 0 3
" #" # " #
−3 0 1 0 0 0 −1 0 1
1
→ 0 1 1 0 0 0 −1 4 −3
4
1 0 1 0 0 1 1 0 3
 1 3 
4
0 4
1 3
=  4
0 4
,
1 3
4
0 4
as n → ∞.
4.9. Miscellaneous problems on transition matrices. In each case find the eigenvalues of T , a formula for
T n and the limit of T n as n → ∞. The special cases discussed in Problem 4.8 can occur.
(a)
" 1 7 9 #
2 32 32
T = 1 0 0 ;
1 1 1
2 4 4
(b)
" 1 1 5 #
3 4 12
T = 1 0 0 ;
1 1 1
4 4 2
(c)
" 1 3 9 #
4 16 16
3 1
T = 4
0 4
;
1 1 1
4 4 2
57
(d)
" 1 1 1 #
4 4 2
5 1 1
T = 12 3 4
;
1 1 1
2 4 4
(e)
 
1 0 0 0
1 1
 2
0 0 2 
T = .
0 0 1 0 
1 1
0 2 2
0
(a) The eigenvalues of T are λ1 = − 81 (repeated) and λ3 = 1, with the corresponding eigenvectors
" 1 # " #
4
1
r1 = −2 , r2 = 1 .
1 1
The Jordan decomposition matrix J is required, where
" #
− 81 1 0
J= 0 − 18 0 ,
0 0 1
and r2 is given by    
5 7 9 1
8 32 32 4
1
[T − λ1 I3 ]r2 = r1 or  1 8
0  r2 =  −2 
1 1 3
2 4 8
1
The remaining eigenvector is
" #
−3
r2 = 8 .
4
3
The matrix C and its inverse are given by
 1   10 13 11 
4
−3 1 − 27 − 54 18
C =  −2 8 1 , C −1 =  − 61 1
24 8
1 
4 16 5 2
1 3
1 27 27 9
Then    
1
4
−3 1 (− 18 )n 1 0 − 10
27
13
− 54 11
18
T n =  −2 8 1  0 (− 81 )n 0   − 16 1
24 8
1 
4 16 5 2
1 3
1 0 0 1 27 27 9
Matrix multiplcation gives  
16 5 2
27 27 9
lim T n =  16
27
5
27
2
9
.
n→∞
16 5 2
27 27 9
(b) The eigenvalues of T are λ1 = − 41 , λ2 = − 61 , and λ3 = 1, which are all different, so that the
calculations are straightforward. The eigenvectors are given by
   5   
1 12
1
r1 =  −4  , r2 =  − 25 , r3 =  1 
1 1 1
The matrix C and its inverse are
 5   6 
1 12
1 5
− 51 −1
C =  −4 − 25 1 , C −1 =  − 12
7
0 12
7

18 1 2
1 1 1 35 5 7
58
Finally
 5   6   
1 12
1 0 0 0 5
− 51 −1 18
35
1
5
2
7
lim T n =  −4 − 52 1  0 0 0  − 12
7
0 12
7
= 18
35
1
5
2
7
.
n→∞
18 1 2 18 1 2
1 1 1 0 0 1 35 5 7 35 5 7
(c) The eigenvalues are given by

1 3 9

4
3
16 16
1 1 2
4
0 4 = − (λ − 1)(32λ + 8λ + 1).
1 1 1 32
4 4 2
Hence the eigenvalues are λ1 = − 18 (1 + i), λ2 = − 81 (1 − i), and λ3 = 1. This stochastic matrix has complex
eigenvalues. The corresponding eigenvectors are
 3 15
  3 15
 " #
− 26 + 26
i − 26 − 26
i 1
31
r1 =  − 13 14 , 31
r2 =  − 13 14 r3 = 1 .
− 13
i + 13
i 
1 1 1
The diagonal matrix of eigenvalues is

" #
− 81 (1 + i) 0 0
D= 0 − 18 (1 − i) 0 .
0 0 1
After some algebra (easily computed using software)

 14 15 39 
41 82 82
lim T n =  14
41
15
82
39
82
.
n→∞
14 15 39
41 82 82
(d) The eigenvalues are given by λ1 = − 14 , λ2 = 1

12
, and λ3 = 1. The corresponding eigenvectors are
   
− 11
9
1 "
1
#
r1 =  4 , r2 =  − 38  r3 = 1 .
9
1 1 1
The matrix C and the diagonal matrix D are given by

" # " #
− 11
9
1 1 − 41 0 0
4
C= 9
− 38 1 , D= 0 1
12
0 .
1 1 1 0 0 1
It follows that  
21 3 19
55 11 55
lim T n =  21
55
3
11
19
55
.
n→∞
21 3 19
55 11 55
(e) The eigenvalues of T are λ1 = − 12 , λ1 = 21 , λ3 = 1 (repeated). There is a repeated eigenvalue but

we can still find four eigenvectors given by
       
0 0 3 −2
 −1   1   2   −1 
r1 =  , r2 =  , r3 =  , r4 = 
0  0  0  1 
1 1 1 0
The matrix C and its inverse can be compiled from these eigenvectors:
   1 
0 0 3 −2 6
− 12 − 61 1
2
 −1 1 2 −1  − 12 1
− 21 1
C −1 = 
 2 2 
C= , 1 2 .
0 0 0 1  3
0 3
0 
1 1 1 0 0 0 1 0
59
The diagonal matrix D in this case is given by
 
− 12 0 0 0
1
 0 2
0 0 
D= .
0 0 1 0 
0 0 0 1
Hence
Tn = CDn C −1
   
0 0 3 −2 (− 12 )n 0 0 0 1
6
− 21 − 61 1
2
 −1 1 2 −1   0 ( 12 )n 0 0   − 21 1
2
− 21 1
2 
=  0 0 1 2
0 1  0 0 1 0  
3
0 3
0 
1 1 1 0 0 0 0 1 0 0 1 0
   1 
0 0 3 −2 0 0 0 0 6
− 21 − 61 1
2
 −1 1 2 −1   0 0 0 0  − 12 1
2
− 21 1
2 
→  0 1 2
0 0 1  0 0 1 0  3
0 3
0 
1 1 1 0 0 0 0 1 0 0 1 0
 
1 0 0 0
 32 0 1
3
0 
=  0 .
0 1 0 
1 2
3
0 3
0
4.10. A four-state Markov chain has the transition matrix

 1 1 
2 2
0 0
 1 0 0 0 
T = 1 1 1 .
4 2
0 4
3 1
4
0 4
0
Find fi , the probability that the chain returns at some step to state Ei , for each state. Determine which
states are transient and which are persistent. Which states form a closed subset? Find the eigenvalues of
T , and the limiting behaviour of T n as n → ∞.
The transition diagram for the chain is shown in Figure 4.3. For each state, the probability that a first
return occurs is as follows, using the diagram:
(1) (2) (n)
State E1 : f1 = 12 , f1 = 12 , f1 = 0, (n ≥ 3);
1
1
2
E2
E1
1
2
3 1 1
4 4 2
1
4
E4
E3 1
1
4
Figure 4.3: Transition diagram for Problem 4.10.
(1) (2) (n)

State E2 : f2 = 0, f2 = 21 , f2 = 1/2n−1 (n ≥ 3);
60
(1) (2) 1 (n)
State E3 : f3 = 0, f3 = 42
, f3 = 0, (n ≥ 3):
(n) (n)
State E4 : f4 = f3 for all n.
The probability of a return at any stage is
∞
X
fn = fr(n) ,
r=1
for each n. Therefore

1 1 11 1
f1 = + = 1,
+ 2 + 3 + · · · = 1, f2 =
2 22 2 2
1
f3 = f4 = ,
16
summing geometric series for f2 and f3 . Hence E1 and E2 are persistent states, but E3 and E4 are
transient.
The eigenvalues of T are λ1 = − 21 , λ2 = − 41 , λ3 = 14 , and λ4 = 1. The corresponding eigenvectors are
       
− 31 0 0 1
2
 3   0   0   1 
r1 =  , r2 =  , r3 =  , r4 =  .
−1  −1  1  1 
1 1 1 1
Therefore D, C and its inverse are given by

     
− 12 0 0 0 − 31 0 0 1 −1 1 0 0
 0 − 41 0 0   2
0 0 1   1 −1 − 21 1 
D= 1
, C= 3 , C −1 = 
 −2
2 
 0 0 4
0   −1 −1 1 1  3
− 31 1
2
1
2

2 1
0 0 0 1 1 1 1 1 3 3
0 0
Hence
   
− 13 0 0 1 0 0 0 0 −1 1 0 0
 23 0 0 1  0
0 0 0  − 21 1
 1 −1 
lim T n =   2 
n→∞  −1 −1 1 1  0 0 0 0   − 23 − 31 1
2
1
2

2 1
1 1 1 1 0 0 0 1 3 3
0 0
 2 1 
3 3
0 0
2 1
 3 3
0 0 
= 
2 1


3 3
0 0 
2 1
3 3
0 0
4.11. A six-state Markov chain has the transition matrix

 1 1 1

4 2
0 0 0 4
 0 0 0 0 0 1 
 1 1 1 
 0 4
0 4 2
0 
T = .
 0 0 0 0 1 0 
1 1
 0 0 0 2 2
0 
0 0 1 0 0 0
Sketch its transition diagram. From the diagram which states do you think are transient and which do
you think are persistent? Which states form a closed subset? Determine the invariant distribution in the
subset.
Intuitively E1 , E2 , E3 and E6 are transient since paths can always escape through E3 and not return.
61
1
E6 2
E5
1
2
1
4
1
1
1 2 1
4 E1 E4
1
1
1
2 4
E2 1 E3
4
For state E4 , the probabilities of first returns are
(1) (n) 1
f4 = 0, f4 = , (n = 2, 3, 4, . . .).
2n−1
It follows that a return to E4 occurs at some step is
∞
X ∞
X
(n) 1
f4 = f4 = = 1.
2n−1
n=1 n=2
Hence E4 is persistent. For E5 ,
(1) 1 (2) 1 (n)

f5 = , f5 = , f5 = 0, (n ≥ 3),
2 2
so that f5 = 12 + 12 = 1. Hence E5 is also persistent.
The states E4 , E5 form a closed subset since no escape paths occur. The subset has the transition
matrix
0 1
S= 1 1 .
2 2
1 2

In the notation of Section 4.3, α = 1 and β = 21 . Hence the invariant distribution is 3 3
.
4.12. Draw the transition diagram for the seven-state Markov chain with transition matrix
 0 1 0 0 0 0 0

 0 0 1 0 0 0 0 
 1 0 0 1
0 0 0 
 2 2 
T =
 0 0 0 0 1 0 0 .

 0 0 0 0 0 1 0 
 1 1 
2
0 0 0 0 0 2
0 0 0 0 0 0 1
(n) (n)
Hence discuss the periodicity of the states of the chain. From the transition diagram calculate p11 and p44
(3) (3) (3n)
for n = 2, 3, 4, 5, 6. (In this example you should confirm that p11 = 12 but that p44 = 0: however, p44 = 6 0
for n = 2, 3, . . . confirming that state E4 is periodic with period 3.)
Consider state E1 . Returns to E1 can occur in the sequence E1 E2 E3 E4 which takes 3 steps, or as
E1 E2 E3 E4 E5 E6 E1 which takes 6 steps which is a multiple of 3. Hence returns to E1 can only occur at
steps 3, 6, 9, . . .: hence E1 has periodicity 3. Similarly E2 and E3 also have periodicity 3. On the other
hand for E4 returns are possible at steps 6, 9, 12, . . . but it still has periodicity. The same is true of states
E5 and E6 .
62
E1 1 E2 1 E3 1 E4 1 E5 1 E6 1 E7
2 2
1
2
1 1
2
E7 is an absorbing state.
4.13. The transition matrix of a 3-state Markov chain is given by

 3 1 
0 4 4
1 1
T = 2
0 2
.
3 1
4 4
0
Show that S = T 2 is the transition matrix of a regular

chain. Find its eigenvectors and confirm that S has
14 13 10
an invariant distribution given by 37 37 37
for even steps in the chain.
The matrix S is given by  

9 1 3
16 16 8
S = T2 =  3
8 2
1
8
1 ,
1 9 5
8 16 16
3 1
which is regular since all elements are non-zero. The eigenvalues of S are given by λ1 = 16
− 4
i,
E1
3 1
4 4
1 3
2 4
1
2
E2 E3
1
4

3
λ2 = 16
+ 14 i, λ3 = 1, with corresponding eigenvectors
     
− 16
25
+ 13
25
i 2
− 25 − 14
25
i 1
r1 =  − 16
25
− 13
25
i , 2
r2 =  − 25 + 14
25
i , r3 =  1  .
1 1 1
The matrix C is given by the matrix of eigenvectors, namely

 16 13 2 14 
− 25 + 25
i − 25 − 25
i 1
16 13 2 14
C =  − 25 − 25
i − 25 + 25
i 1 .
1 1 1
Let " #
3
16
− 14 i 0 0
3
D= 0 16
+ 41 i 0 .
0 0 1
63
Finally (computation simplifies the algebra)
 14 13 10 
37 37 37
CDC −1 =  14
37
13
37
10
37
,
14 13 10
37 37 37
which gives the limiting distribution.
4.14. An insect is placed in the maze of cells shown in Figure 4.11. The state Ej is the state in which the
insect is in cell j. A transition occurs when the insect moves from one cell to another. Assuming that exits
are equally likely to be chosen where there is a choice, construct the transition matrix T for the Markov
chain representing the movements of the insect. Show that all states are periodic with period 2. Show that
T 2 has two subchains which are both regular. Find the invariant distributions of both subchains. Interpret
the results.
If the insect the insect starts in any compartment (state), then it can only return to that compartment
after an even number of steps. Hence all states are periodic with period 2.
1
E4 E2
1
2
1
2
E4 E2
1
2
E1
E1
1
2
1
1 2
2
E5 E3
1
E5 E3
1
2
Figure 4.7: Transition diagram for Problem 4.13, and the maze.
The matrix  
1 1 1
2 4
0 0 4
1 1

 2 2
0 0 0 

 2 3 1 
S=T = 0 0 4 4
0 
 1 3

 0 0 4 4
0 
1 1
2
0 0 0 2
has two subchains corresponding to E1 , E2 , E5 and E3 , E4 : this follows since the zeros in columns 3 and
4, and the zeros in rows 3 and 4, remain for all powers of S. The subchains have the transition matrices
 1 1 1 
2 4 4
3 1

1 1 4 4
T1 =  2 2
0 , T2 = 1 3
.
1 1 4 4
2
0 2
The eigenvalues of T1 are λ1 = 0, λ2 = 21 , λ3 = 1, with the corresponding eigenvectors

" # " # " #
−1 0 1
r1 = 1 , r2 = −1 , r3 = 1 .
1 1 1
The matrix C1 and its inverse, and the diagonal D1 are given by
" # " # " #
−1 0 1 − 21 1
4
1
4
0 0 0
C1 = 1 −1 1 , C1−1 = 0 − 12 1
2
, D= 0 1
2
0
1 1 1
1 1 1 2 4 4
0 0 1
64
Therefore
T1n = C1 D1n C1−1

   
−1 0 1 0 0 0 − 12 1
4
1
4
=  1 −1 1  0 ( 21 )n 0  0 − 21 1
2

1 1 1
1 1 1 0 0 1 2 4 4
" 1 1 1 #
2 4 4
1 1 1
→ 2 4 4
,
1 1 1
2 4 4
as n → ∞.
The eigenvalues of T2 are λ1 = 21 , λ2 = 1, and the corresponding eigenvectors are

−1 1
r1 = , r2 = .
1 1
The matrix C2 and its inverse, and the diagonal matric D2 are given by

−1 1 − 21 1
− 21 0
C2 = , C2−1 = 1
2
1
, D2 = .
1 1 2 2
0 1
Hence
−1 1 (− 21 )n 0 − 21 1 1 1
T2n = C2 D2n C2−1 = 1
2
1
→ 2
1
2
1
,
1 1 0 1 2 2 2 2
as n → ∞.
Combination of the two limiting matrices leads to
 1 1 1

2 4
0 0 4
1 1 1
 2 4
0 0 4

lim T n =  0
 1 1 
0 2 2
0 .
n→∞  0 1 1
0 2 2
0 
1 1 1
2 4
0 0 4
4.15. The transition matrix of a four-state Markov chain is given by

 
1−a a 0 0
 1−b 0 b 0 
T = , (0 < a, b, c < 1).
1−c 0 0 c 
1 0 0 0
(n)
Draw a transition diagram, and, from the diagram, calculate f1 , (n = 1, 2, . . .), the probability that a first
return to state E1 occurs at the n-th step. Calculate also the mean recurrence time µ1 . What type of state
is E1 ?
The first return probabilities for state E1 are

(1) (2) (3) (4) (n)
f1 = 1 − a, f1 = a(1 − b), f1 = ab(1 − c), f1 = abc, f1 = 0, (n ≥ 5).
Hence
∞
X (n)
f1 = f1 = 1 − a + a(1 − b) + ab(1 − c) + abc = 1,
n=1
which implies that E1 is persistent. Also, the mean

∞
X (n)
µ1 = nf1 = 1 − a + 2a(1 − b) + 3ab(1 − c) + 4abc = 1 + a + ab + abc.
n=1
Hence µ1 is finite so that E1 is non-null. It is also aperiodic so that E1 is an ergodic state.
65
1-a
E1 E2 b E3 c E4
a
1-b
1-c 1
1
4.16. Show that the transition matrix

 
1−a a 0 0
 1−a 0 a 0 
T =
1−a 0 0 a 
1 0 0 0
1
where 0 < a < 1, has two imaginary (conjugate) eigenvalues. If a = 2
, confirm that T has the invariant
8 4 2 1
distribution p = 15 15 15 15
.
The eigenvalues of T are given by λ1 = −a, λ2 = −ai, λ3 = ai, λ4 = 1, of which two are imaginary
conjugates.
If a = 21 , then λ1 = − 21 , λ2 = − 21 i, λ3 = 12 i, λ4 = 1, with corresponding eigenvectors
       
− 21 − 21 i 1
2
i 1
 1  1
 − + 12 i   − 21 − 21 i   1 
r1 =  1  , r2 =  12 , r3 =  1 , r4 =  .
−2 2
+i  −i  2
1 
1 1 1 1
The matrix C of eigenvalues and its inverse, and the diagonal matrix D are given by
   
− 21 − 12 i 1
2
i 1 − 31 1
3
− 31 1
3
 1 − 12 + 21 i − 21 − 12 i 1  1
 − 10 + 3
i 3
− 10 − 1
i 1
− 3
i 3
+ 1
i 
C=
 −1 1 1
, C −1 = 
 −1 −
10
3 3
10
1
10
1
10
3
10
3
10
1
,
2 2
+i 2
−i 1  10 10
i − 10 + 10
i 10
+ 10
i 10
− 10
i 
8 4 2 1
1 1 1 1 15 15 15 15
 
− 21 0 0 0
 0 − 12 i 0 0 
D= 1
. 
 0 0 2
i 0 
0 0 0 1
n −1
In the limit n → ∞, it follows that all the rows of CD C are given by
8 4 2 1

15 15 15 15
,
which is the same as the last row of C −1 .
4.17. A production line consists of two manufacturing stages. At the end of each manufacturing stage each
item in the line is inspected, where there is a probability p that it will be scrapped, q that it will be sent
back to that stage for reworking, and (1 − p − q) that it will be passed to the next stage or completed. The
production line can be modelled by a Markov chain with four states: E1 , item scrapped; E2 , item completed;
E3 , item in first manufacturing stage; E4 , item in second manufacturing stage. We define states E1 and
E2 to be absorbing states so that the transition matrix of the chain is
 
1 0 0 0
 0 1 0 0 
T = .
p 0 q 1−p−q 
p 1−p−q 0 q
66
An item starts along the production line. What is the probability that it is completed in two stages?
(n) (n)
Calculate f3 and f4 . Assuming that 0 < p + q < 1, what kind of states are E3 and E4 ? What is the
probability that an item starting along the production line is ultimately completed?
The transition diagram is shown in Figure 4.9. E1 and E2 areabsorbing states. The initial position of
the item can be represented by the vector p(0) = 0 0 1 0 . We require
q q
1
1-p-q 1-p-q
E3 E4 E2
p p
E1
 2
1 0 0 0
 0 1 0 0
p(2) p(0) T 2 =

= 0 0 1 0 
p 0 q 1−p−q 
p 1−p−q 0 q

= 2p − p2 (1 − p − q)2 q2 q(1 − p − q)
(2)
Therefore the probability that an item is completed in two stages is p2 = (1 − p − q)2 .
The first return probabilities are
(1) (n)
f3 = q, f3 = 0, (n ≥ 3),
(1) (n)
f4 = q, f4 = 0, (n ≥ 3).
Hence E3 and E4 are transient states.
The probability of an item that is completed without reworking is (1 − p − q)2 , with one reworking is
2q(1 − p − q)2 , with two reworkings 3q 2 (1 − p − q)2 , and n reworkings (n + 1)q n (1 − p − q)2 . Hence the
probability that an item starting is ultimately completed is
(1 − p − q)2
(1 − p − q)2 [1 + 2q + 3q 2 + · · ·] = ,
(1 − q)2
4.18. The step-dependent transition matrix of Example 4.9 is

" 1 1 #
2 2
0
Tn = 0 0 1 , (n = 1, 2, 3, . . .).
1/(n + 1) 0 n/(n + 1)
Find the mean recurrence time for state E3 , and confirm that E3 is a persistent, non-null state.
The transition matrix is shown in Figure 4.10. Assuming that a walk starts at E3 , the probabilities of
first returns to state E3 are (using the diagram)
(1) 1 (2) (3) 1 1 1 (n) 1 1 1 1

f3 = , f3 = 0, f3 = × × 1 = , . . . , f3 = × n−3 × × 1 = n−1 , . . . .
2 1+1 2 4 1+1 2 2 2
Hence
67
1
2
E1
1 1/(n+1)
2
E3
E2 1
n/(n+1)
∞ ∞
X (n)
X 1
f3 = f3 = = 1,
2n
n=1 n=1
using the formula for the sum of a geometric series. This means that E3 is persistent. The mean recurrence
time is given by
∞ ∞
X (n) 1 X n 1 5
µ3 = nf3 = + = +2= ,
2 2n−1 2 2
n=1 n=3
using the formula for the sum of the geometric series. Hence E3 is persistent and non-null.
4.19. In Example 4.9, a persistent, null state occurred in a chain with step-dependent transitions: such a
state cannot occur in a finite chain with a constant transition matrix. However, chains over an infinite
number of states can have persistent, null states. Consider the following chain which has an infinite number
of states E1 , E2 , . . . with the transition probabilities
1 1 1 j
p11 = , p12 = , pj1 = , pj,j+1 = , (j ≥ 2).
2 2 j+1 j+1
Find the mean recurrence time for E1 , and confirm that E1 is a persistent, null state.
From the transition diagram, the probabilities of first returns to E1 are given by
1 E1 1 E2 2 E3 3 E4 4 E5
2 3 4 5
2
1
3
1
4 1
5 1
6
(1) 1 (2) 1 (3) 1 (n) 1

f1 = , f1 = , f1 = , ... , f1 = , ....
2 2.3 3.4 n(n + 1)
Therefore
∞ ∞ N h i h i
X (n)
X 1 X 1 1 1
f1 = f1 = = lim − = lim 1 − = 1,
n(n + 1) N→∞ n n+1 N→∞ N
n=1 n=1 n=1
which implies that E1 is persistent. However the mean recurrence time is

∞ ∞
X (n)
X 1
µ1 = nf1 = = ∞,
n+1
n=1 n=1
68
the series being divergent. According to the definition, E1 is a null state.
4.20. A random walk takes place on 1, 2, . . . subject to the following rules. A jump from position i to
position 1 occurs with probability qi , and from position i to i + 1 with probability 1 − qi for i = 1, 2, . . .,
where 0 < qi < 1. Sketch the transition diagram for the chain. Explain why to investigate the persistence
of every state, only one state, say state 1, need be considered. Show that the probability that a first return
to state 1 occurs at some step is
∞ j−1
X Y
f1 = (1 − qk ) qj .
j=1 k=1
If qj = q (j = 1, 2, . . .), show that every state is persistent.
The transition diagram is shown in Figure 4.12 with the states labelled E1 , E2 , . . .. The chain is
irreducible since every state can be reached from every other state. The diagram indicates that every state
E1 E2 E3 E4 E5
1- q1 1- q2 1- q3 1- q4
q1
q2
q3
q4
q5
is aperiodic since a return to any state can be achieved in any number of steps. Need only consider one
state since the others will have the same properties.
Consider state E1 . Then, from the diagram, the probabilities of first returns are
j−1
(1) (2) (3) (j)
Y
f1 = q1 , f1 = (1 − q1 )q2 , f1 = (1 − q1 )(1 − q2 )q3 , . . . , f1 = (1 − qk )qj , . . . .
k=1
Therefore
∞ ∞ j−1
X (j)
X Y
f1 = f1 = (1 − qk )qj .
j=1 j=1 k=1
If qj = q, (j = 1, 2, . . .), then
∞ j−1 ∞
X Y X
f1 = (1 − q)q = q (1 − q)j−1 = q/q = 1,
j=1 k=1 j=1
using the formula for the sum of geometric series. Hence E1 and every state is recurrent.
4.21. A Markov chain maze. Figure 4.12 shows a maze with entrance E1 , further gates E2 , E3 , E4 , E5 , E6
with target E7 These gates can be represented by states in a Markov chain. At each E1 , . . . , E6 there are
two possible new paths which are assumed equally likely to be chosen. The target E7 can be considered to
be an absorbing state. Construct a 7 × 7 transition matrix for the maze assuming that the walker does not
return to a previous state and does not learn from previous choices: for example, at E1 can still make the
mistake of walking the dead-end again. Find the probabilities that the walker reaches the target in 6, 7, 8, . . .
steps.
Suppose now that the walker learns from wrong choices. To accommodate this, let E11 be the entrance
and E12 the return to the entrance after a wrong choice (the dead-end); let E21 and E22 have the same
meaning for the second entrance, and so on. Hence the probabilities are:
P(E12 → E12 ) = 21 ; P(E11 → E21 ) = 12 ; P(E12 → E21 ) = 1;
P(E21 → E22 ) = 12 ; P(E31 → E21 ) = 12 ; P(E22 → E31 ) = 1;
69
E4
E5
E3 E7 E6 E2
E1
Figure 4.13:
and similarly for the remaining probabilities. The transition matrix is now 13 × 13. with states
E11 , E12 , E21 , E22 , . . . , E61 , E62 , E7 .
Find the probabilities that the walker reaches the centre in 6,7,8 steps. (Really need a computer program
to compute the matrix products.)
With the maze rules as described the transition matrix for walks from the entrance to the target is
given by the table:
 
E1 E2 E3 E4 E5 E6 E7
1 1
 E1 2 2
0 0 0 0 0 
 E2 0 1 1 
 2 2
0 0 0 0 
 E3 0 1 1
0 2 2
0 0 0 
T = E4 0 1 1

 0 0 2 2
0 0 

1 1
 E5 0 0 0 0 2 2
0 
 1 1 
E6 0 0 0 0 0 2 2
E7 0 0 0 0 0 0 1
The initial position of the walker can be represented by

p0 = 1 0 0 0 0 0 0 .
After n steps the probabilities are given by
pn = p0 T n , (n ≥ 6).
The probability of choosing randomly the shortest path is 1/26 = 1/64. Computation makes this problem
easy. The first row in T 6 which is
[1/64, 3/32, 15/64, 5/16, 15/64, 3/32, 1/64]
lists the probabilities of the maze-walker being at E1 , E2 , etc reaches after 6 gates. It follows that

p7 = p0 T 2 = 1
128
7
128
21
128
35
128
35
128
21
128
1
16
.
Hence the probability that the walker reaches the centre in exactly 7 steps is
1 1 3
− = .
16 64 64
1
[The answer is not 16 since we must exclude repeats at the absorbing state E7 .] A similar arguments leads
to the probability that the walker reaches the centre in exactly 8 steps is
37 1 1 21
− − = .
256 64 16 256
70
In the second case the table is:
 
E11 E12 E21 E22 E31 E33 E41 E44 E51 E55 E61 E66 E7
1 1
 E11 0 2 2
0 0 0 0 0 0 0 0 0 0 
 E12 0 0 1 0 0 0 0 0 0 0 0 0 0 
 1 1


 E21 0 0 0 2 2
0 0 0 0 0 0 0 0 


 E22 0 0 0 0 1 0 0 0 0 0 0 0 0 

1 1
 E31 0 0 0 0 0 2 2
0 0 0 0 0 0 
 
 E33 0 0 0 0 0 0 1 0 0 0 0 0 0 
S= 1 1 
 E41 0 0 0 0 0 0 0 2 2
0 0 0 0 
 E44 0 0 0 0 0 0 0 0 1 0 0 0 0 
 1 1


 E51 0 0 0 0 0 0 0 0 0 2 2
0 0 


 E55 0 0 0 0 0 0 0 0 0 0 1 0 0 

1 1
 E61 0 0 0 0 0 0 0 0 0 0 0 2 2

 
E66 0 0 0 0 0 0 0 0 0 0 0 0 1
E7 0 0 0 0 0 0 0 0 0 0 0 0 1
Let the walk start from E1 so that the initial vector is

v0 = 1 0 0 0 0 0 0 0 0 0 0 0 0
Then after one step

1 1

v1 = v0 T = 0 2 2
0 0 0 0 0 0 0 0 0 0 :
1
in other words, the walker is back at E1 with probability of 2
or moved on to E2 with the same probability.
For the next step

v2 = v0 T 2 = 0 0 1
2
1
4
1
4
0 0 0 0 0 0 0 0 ,
and so on. By step 12 the centre of the maze must have been reached which is confirmed by

v12 = v0 T 12 = 0 0 0 0 0 0 0 0 0 0 0 0 1 .
The probability that the walker reaches the centre in 6 steps is 1/64 (as before), in 7 steps is 3/64
(both as in the previous case) and in 8 steps is 15/256.
4.22. In a finite Markov chain a subset C of states is said to be closed if, for states i and j,
i ∈ C, transition i → j ⇒ j ∈ C.
Find the closed subset in a chain with transition matrix

 1 1

0 0 2 2
0 0
1 1 1
 0 0 2
0 4 4

 1 1 1 
 0 0 3 3
0 3 
T = .
 0 0 0 0 0 1 
1 1 

2
0 0 0 0 2
0 0 1 0 0 0
The closed subset C the consists of the the three states E3 , E4 , E6 . A transition diagram is helpful:
also compute T 4 which reveals the closed subset.
4.23. Chess knight moves. A knight moves on a reduced chess board with 4 × 4 = 16 squares. The knight
starts at the bottom left-hand corner and moves according to the usual chess rules. Treating moves as a
Markov chain, construct a 16 × 16 transition matrix for the knight moving from any square (easier if you
design a computer program for the matrix). Show that the knight returns to it starting corner after 2,4,6
moves (must be even) with probabilities 41 , 61 , 54
7
respectively. Find the corresponding first returns. [ Just a
71
(n)
reminder: if f11 is the probability that the first return to corner (1, 1) (say) after n moves, and p11 (m) is
the probability that the knight is at (1, 1) after m moves] ,then
n−1
(1) (n) (n)
X (m) (n−m)
f11 = p11 , f11 = p11 − f11 p11 , (n ≥ 2).]
m=1
The chessboard is shown in Figure 4.14 with the squares labelled E11 ,E12 ,. . .,E44 as shown. The knight
moves start in E11 . It is helpful to itemize possible return paths to E11 . For example a path with four
E41 E42 E43 E44
E31 E32 E33 E34
E21 E22 E23 E24
E11 E12 E13 E14
Figure 4.14:
transitions is
E11 → E23 → E44 → E32 → E11 .
Assuming that all transitions are equally likely the transition E11 → E23 occurs with probability 21 ,E23 →
E44 with probability 41 since there are 4 possible moves from E23 . The probability that a return on this
path occurs is
1 1 1 1 1
× × × = .
2 4 2 4 64
There are 4 paths which pass through E44 which occur wth probability
1 1
4× =
64 16
(remember we are restricting return paths to 4 transitions with no intermediate return to E11 . A further
path is
E11 → E23 → E42 → E23 → E11 ,
which occurs with probability
1 1 1 1 1
× × × = .
2 4 3 4 96
There are a further 3 similar paths with the same probabilities. Hence the probability that a first return
to E11 after 4 moves is
(4) 1 1 5
f11 = 4 × +4× = .
64 96 48
The transition matrix for all the moves of the knight on the 4 × 4 chessboard is given by the matrix T .
72
The order is given by the list of squares as shown in the figure
 1 1 
0 0 0 0 0 0 2
0 0 2
0 0 0 0 0 0
1 1 1
 0 0 0 0 0 0 0 3 3
0 3
0 0 0 0 0 
 1 1 1 
 0 0 0 0 3
0 0 0 0 3
0 3
0 0 0 0 
 1 1 
 0 0 0 0 0 2
0 0 0 0 2
0 0 0 0 0 
1 1 1

 0 0 3
0 0 0 0 0 0 0 3
0 0 3
0 0 

1 1 1 1

 0 0 0 4
0 0 0 0 0 0 0 4 4
0 4
0 

1 1 1 1
 4
0 0 0 0 0 0 0 4
0 0 0 0 4
0 4

 1 1 1 
 0 3
0 0 0 0 0 0 0 3
0 0 0 0 3
0 
T = 1 1 1 
 0 3
0 0 0 0 3
0 0 0 0 0 0 0 3
0 
 1 1 1 1 
 4
0 4
0 0 0 0 4
0 0 0 0 0 0 0 4 
1 1 1 1

 0 4
0 4 4
0 0 0 0 0 0 0 4
0 0 0 

1 1 1

 0 0 3
0 0 3
0 0 0 0 0 0 0 3
0 0 

1 1
 0 0 0 0 0 2
0 0 0 0 2
0 0 0 0 0 
 1 1 1 
 0 0 0 0 3
0 3
0 0 0 0 3
0 0 0 0 
 1 1 1 
0 0 0 0 0 3
0 3 3
0 0 0 0 0 0 0
1 1
0 0 0 0 0 0 2
0 0 2
0 0 0 0 0 0
Returns to E11 can only occur after an even number of moves. We can obtain the returns to E11 by
computing T 2n , and noting the number in the top left. Hence, in the usual notation
(2) 1 (4) 1 (6) 7 (8) 107

p11 = , p11 = , p11 = , p11 = ,
4 6 54 972
and, the first returns are given by
1 (2)
f2 = p11 =
4
(4) (2) 1 1 1 5
f4 = p11 − f2 p11 = − × = ,
6 4 4 48
(6) (4) 107
f6 = p11 − f2 p11 − f4 p11 ∗ (2) = = 0.0619 . . . .
1728
73
Chapter 5
Poisson processes
5.1. The number of cars which pass a roadside speed camera within a specified hour is assumed to be a
Poisson process with intensity λ = 92: on average 92 cars pass in the hour. It is also found that 1% of
cars exceed the designated speed limit. What are the probabilities that (a) at least one car exceeds the speed
limit, (b) at least two cars exceed the speed limit in the hour?
Let time be measured in hours. With λ = 92 in the Poisson process, the mean number of cars in the
hour is 92 × 1 = 92. Of these, on average, 0.92 cars exceed the speed limit. Assume that the cars which
exceed the limit form a Poisson process with parameter λ1 = 0.92. Let N (t) be a random variable of the
number of cars exceeding the speed limit by time t measured from the beginning of the hour.
(a) The probability that at least one car has exceeded the limit within the hour is
1 − P(N (t) < 1) = 1 − e−λ1 = 1 − 0.398 = 0.602.
(b) The probability that at least two cars have exceeded the limit within the hour is
1 − P(N (t) < 2) = 1 − e−λ1 − λ1 e−λ1 = 1 − 0.398 − 0.367 = 0.235.
5.2. If the between-event time in a Poisson process has an exponential distribution with parameter λ with
density λe−λt , then the probability that the time τ for the next event to occur is at least t is
P{τt > t} = e−λt .
Show that, if t1 , t2 ≥ 0, then

P{τt > t1 + t2 |τt > t1 } = P{τt > t2 }.
What does this result imply about the Poisson process and its memory of past events?
By formula (1.2) on conditional probability
P(τ > t1 + t2 ∩ τ > t1 ) P(τ > t1 + t2 )

P(τ > t1 + t2 |τ > t1 ) = =
P(τ > t1 ) P(τ > t1 )
e−λ(t1 +t2 )
= = e−λt2
e−λt1
= P(τ > t2 )
The result shows the no memory property of the Poisson process.
5.3. The number of cars which pass a roadside speed camera are assumed to behave as a Poisson process
with intensity λ. It is found that the probability that a car exceeds the designated speed limit is p.
(a) Show that the number of cars which break the speed limit also form a Poisson process.
(b) If n cars pass the camera in time t, find the probability function for the number of cars which exceed
the speed limit.
74
(a) Let N (t) be the random variable representing m the number of speeding cars which have occurred in
time t. The probability qn (t) = P(N (t) = m) satisfies
qn (t + δt) ≈ qn−1 (t)λpδt + qn (t)(1 − λpδt),
where λpδt is the probability that a speeding car appears in the time δt. This is the equation for a Poisson
process with intensity λσ.
(b) Of the n cars the number of ways in which m speeding cars can be arranged is

n! n
= .
m!(n − m)! m
The probability that any individual event occurs is

n m
p (1 − p)n−m , (m = 0, 1, 2, . . . , n),
m
which is the binomial distribution.
5.4. The variance of a random variable Xt is given by
V(Xt ) = E(Xt2 ) − E(Xt )2 .
In terms of the generating function G(s, t), show that

" 2 #
∂ ∂G(s, t) ∂G(s, t)
V(Xt ) = s − .
∂s ∂s ∂s
s=1
(an alternative formula to (5.20)). Obtain the variance for the Poisson process using its generating function
G(s, t) = eλ(s−1)t
given by eqn (5.17), and check your answer with that given in Problem 5.3.
The assumption is that the random variable Xt is a function of the time t. Let pn (t) = P(Xt = n).
The probability generating function G(s, t) becomes a function of two variables s and t. It is defined by
∞
X
G(s, t) = pn (t)sn .
n=0
The mean of Xt is given by

∞
X ∂G(s, t)
E(Xt ) = npn (t) =
∂s s=1
n=1
The expected value of Xt2 is given by

∞
X ∂ ∂G(s, t)
E(Xt2 ) = 2
n pn (t) = s .
∂s ∂s s=1
n=1
Hence " 2 #
∂ ∂G(s, t) ∂G(s, t)
V(Xt ) = s − .
∂s ∂s ∂s
s=1
For the Poisson process G(s, t) = eλ(s−1)t . Then

∂ λ(s−1)t
E(Xt ) = [e ] = λt,
∂s s=1
and
∂
E(Xt2 ) = [sλteλ(s−1)t ] = λt − (λt)2 .

∂s s=1
75
Hence V(Xt ) = λt.
5.5. A telephone answering service receives calls whose frequency varies with time but independently of
other calls perhaps with a daily pattern—more during the day than the night. The rate λ(t) ≥ 0 becomes a
function of the time t. The probability that a call arrives in the small time interval (t, t + δt) when n calls
have been received at time t satisfies
pn (t + δt) = pn−1 (t)(λ(t)δt + o(δt)) + pn (t)(1 − λ(t)δt + o(δt)), (n ≥ 1),
with
p0 (t + δt) = (1 − λ(t)δt + o(δt))p0 (t).
It is assumed that the probability of two or more calls arriving in the interval (t, t + δt) is negligible. Find
the set of differential-difference equations for pn (t). Obtain the probability
R t generating function G(s, t) for
the process and confirm that it is a stochastic process with intensity 0 λ(x)dx. Find pn (t) by expanding
G(s, t) in powers of s. What is the mean number of calls received at time t?
From the difference equation it follows that

pn (t + δt) − pn (t)
= λ(t)pn−1 (t) − λ(t)pn (t) + o(1),
δt
p0 (t + δt) − p0 (t)
= −λ(t)p0 (t) + o(1).
δt
Let δt → ∞. Then
p′n (t) = λ(t)pn−1 (t) − λ(t)pn (t), (i)
p′0 (t) = −λ(t)p0 (t). (ii)
Let
∞
X
n=0
Multiply (i) by sn , sum from n = 1, and add (ii) so that
∂G(s, t)
= λ(t)(s − 1)G(s, t). (iii)
∂t
The initial value is G(s, 0) = 1. The solution of (iii) (which is essentially an ordinary differential equation)
subject to the initial condition is
Z t
G(s, t) = exp (s − 1) λ(u)du .
0
Expansion of the generating function gives the series

Z t
Z t
∞
X
G(s, t) = exp −1 λ(u)du exp s λ(u)du = pn (t)sn ,
0 0 n=0
where the probability Z Z n

t t
1
pn (t) = exp −1 λ(u)du λ(u)du .
n! 0 0
The mean of the process is Z t
∂G(s, t)
µ(t) = = λ(u)du.
∂s s=1 0
5.6. For the telephone answering service in Problem 5.5, suppose that the rate is periodic given by λ(t) =
a + b cos(ωt) where a > 0 and |b| < a. Using the probability generating function from Problem 5.6 find the
probability that n calls have been received at time t. Find also the mean number of calls received at time t.
Sketch graphs of p0 (t), p1 (t) and p2 (t) where a = 0.5, b = 0.2 and ω = 1.
76
Using the results from Problem 5.5,
Z t
G(s, t) = exp (s − 1) (a + b cos ωu)du = exp[(s − 1)(at + (b/ω) sin ωt)].
0
Hence
1 −[at+(b/ω) sin ωt]
pn (t) = e [at + (b/ω) sin ωt]n ,
n!
and the mean is
µ(t) = at + (b/ω) sin ωt.
The first three probabilities are shown in Figure 5.1.
p0(t)
p1(t)
p (t)
2
t
Figure 5.1: Graphs of the probabilities p0 (t), p1 (t) and p2 (t) versus t in Problem 5.6.
5.7. A Geiger counter is pre-set so that its initial reading is n0 at time t = 0. What are the initial conditions
on pn (t), the probability that the reading is n(≥ n0 ) at time t, and its generating function G(s, t)? Find
pn (t), and the mean reading of the counter at time t.
The probability generating function for this Poisson process is (see eqn (5.16))
G(s, t) = A(s)eλ(s−1)t .
The initial condition is G(s, 0) = xn0 . Hence A(s) = sn0 and
G(s, t) = sn0 eλ(s−1)t .
The power series expansion of G(s, t) is

∞
X sn λn−n0 e−λt
G(s, t) = .
(n − n0 )!
n=n0
Hence
(λt)n−n0 e−λt
pn (t) = 0, (n < n0 ), pn (t) = , (n ≥ n0 ).
(n − n0 )!
The mean reading at time t is given by

∂G(s, t)
µ(t) = = n0 + λt.
∂s s=1
5.8. A Poisson process with random variable N (T ) has probabilities
(λt)n e−λt
pn (t) = P[N (t) = n] = .
n!
77
If λ = 0.5, calculate the following probabilities associated in the process:
(a) P[N (3) = 6];
(b) P[N (2.6) = 3];
(c) P[N (3.7) = 4|N (2.1) = 2];
(d) P[N (7) − N (3) = 3].
(0.5 × 3)6 e−0.5×3

(a) P(N (3) = 6) = p6 (3) = = 0.00353.
6!
(0.5 × 2.6)3 e−0.5×2.6
(b) P(N (2.6) = 3) = p3 (2.6) = = 0.100.
3!
(0.5 × 1.6)2 e−0.5×1.6
(c) P[N (3.7) = 4|N (2.1) = 2] = P[N (1.6) = 2] = p2 (1.6) = = 0.144.
2!
(d) P[N (7) − N (3) = 3] = P[N (4) = 3] = 0.180.
5.9. A telephone banking service receives an average of 1000 call per hour. On average a customer transac-
tion takes one minute. If the calls arrive as a Poisson process, how many operators should the bank employ
to avoid an expected accumulation of incoming calls?
Let time t be measured in minutes . The intensity of the Poisson process λ = 1000/60 = 50/3. The
expected inter-arrival time is
1 3
= = 0.06 minutes.
λ 50
This must be covered
1
= 16.7 operators.
0.06
Hence 17 operators would be required to cover expected incoming calls.
5.10. A Geiger counter automatically switches off when the nth particle has been recorded, where n is fixed.
The arrival of recorded particles is assumed to be a Poisson process with parameter λt. What is the expected
value of the switch-off times?
The probability distribution for the switch-off times is
F (t) = 1 − {probabilities that 0, 1, 2, . . . , n − 1 particles recorded by time t)}

(λt)n−1
= 1 − e−λt 1 + λt + · · · +
(n − 1)!
n−1
X (λt)r
= 1 − e−λt
r!
r=0
Its density is, for t > 0,

( n−1
)
dF (t) d −λt
X (λt)r
f (t) = = 1−e
dt dt r!
r=0
n−1 n−1
X (λt)n X λn tn−1
= λe−λt − e−λt
n! (n − 1)!
r=0 r=1
(λt)n−1
= λe−λt .
(n − 1)!
which is gamma. Its expected value is

Z ∞ Z ∞ Z ∞
λn−1 n λn
µ = tf (t)dt = λe−λt t dt = e−λt tn dt
0 0
(n − 1)! (n − 1)! 0
Z ∞
1 −s n! n
= e ds = = .
λ(n − 1)! 0
λ(n − 1)! λ
78
5.11. Particles are emitted from a radioactive source, and, N (t), the random variable of the number of
particles emitted up to time t form t = 0, is a Poisson process with intensity λ. The probability that any
particle hits a certain target is p, independently of any other particle. If M (t) is the random variable of
the number of particles that hit the target up to time t, show, using the law of total probability, that M (t)
forms a Poisson process with intensity λp.
For any two times t1 , t2 , (t2 > t1 ≥ 0), using the law of total probability, with t2 − t1 = t,
∞
X n k
P[M (t2 ) − M (t1 ) = k] = P[N (t2 ) − N (t1 ) = n] p (1 − p)n−k
k
n=k
∞
X (λt)n n k
= e−λt p (1 − p)n−k
n! k
n=k
∞
pk X (λt)n n
= e−λt k
(1 − p)n
(1 − p) n! k
n=k
∞
k
−λt (pλt)
X [(1 − p)λt]n−k
= e
k! (n − k)!
n=k
k −λpt
(λpt) e
= ,
k!
which is a Poisson process of intensity λp.
79
Chapter 6
Birth and death processes
6.1. A colony of cells grows from a single cell. The probability that a cell divides in a time interval δt is
λδt + o(δt).
There are no deaths. Show that the probability generating function for this birth process is
se−λt
G(s, t) = .
1 − (1 − e−λt )s
Find the probability that the original cell has not divided at time t, and the mean and variance of population
size at time t (see Problem 5.4, for the variance formula using the probability generating function).
This is a birth process with parameter λ and initial population size 1. Hence the probability generating
function is (see eqn (6.12))
G(s, t) = se−λt [1 − (1 − e−λt )s]−1 ,
which satisfies the initial condition G(s, 0) = s. It follows that the probability that the population size is
1 at time t is p1 (t) = e−λt .
The mean population size is
∂G(s, t)
µ(t) = = eλt .
∂s s=1
Using the generating function, the variance of the population size is
" 2 #
2 ∂ ∂G(s, t) ∂G(s, t)
σ = s − = [2e2λt − eλt ] − e2λt = e2λt − eλt .
∂s ∂s ∂s
s=1
6.2. A simple birth process has a constant birth-rate λ. Show that its mean population size µ(t) satisfies
the differential equation
dµ(t)
= λµ(t).
dt
How can this result be interpreted in terms of a deterministic model for a birth process?
From Section 6.3, the mean population size of the birth process with initial population n0 is given by
µ(t) = n0 eλt . It can be verified that
dµ
− λµ = n0 λeλt − n0 λeλt = 0,
dt
This differential equation is a simple deterministic model for the population in a birth process, so that
the mean population size in the stochastic process satisfies a deterministic equation. [However, this is not
always the case for the relation between stochastic and deterministic models.]
80
6.3. The probability generating function for a simple death process with death-rate µ and initial population
size n0 is given by n0
se−µt
G(s, t) = (1 − e−µt )n0 1 +
1 − e−µt
(see Equation (6.17)). Using the binomial theorem find the probability pn (t) for n ≤ n0 . If n0 is an even
number, find the probability that the population size has halved by time t. A large number of experiments
were undertaken with live samples with a variety of initial population sizes drawn from a common source
and the times of the halving of deaths were recorded for each sample. What would be the expected time for
the population size to halve?
The binomial expansion of G(s, t) is given by

n0
−µt n0 se−µt
G(s, t) = (1 − e ) 1+
1 − e−µt
n0 −nµt n
X n0 e s
= (1 − e−µt )n0
n (1 − e−µt )n
n=0
From this series, the coefficient of sn is

−µt n0 −n −nµt n0
pn (t) = (1 − e ) e , (n = 0, 1, 2, . . . , n0 )
n
which is the probability that the population size is n at time t.

Let n0 = 2m0 , where m0 is an integer, which ensures that n0 is even. We require

2m0
pm0 = (1 − e−µt )m0 e−m0 µt .
m0
The mean population size at time t is given by
µ = Gs (1, t) = n0 e−µt .
This mean is half the initial population if n0 eµt = 21 n0 , which occurs, on average, at time t = µ−1 ln 2.
6.4. A birth process has a probability generating function G(s, t) given by

s
G(s, t) = .
eλt + s(1 − e−λt )
(a) What is the initial population size?

(b) Find the probability that the population size is n at time t.
(c) Find the mean and variance of the population size at time t.
(a) Since G(s, 0) = s, the initial population size is n0 = 1.

(b) Expand the generating function using the binomial theorem:
∞
s X n
G(s, t) = λt −λt
= se−λt s (1 − e−λt )n .
e + s(1 − e )
n=0
The coefficients give the required probabilities:
p0 (t) = 0, pn (t) = e−λt (1 − e−λt )n−1 .
(c) Since
∂G(s, t) eλt
= λt λt 2
= eλt . (i)
∂s
s=1
[e + s(1 − e )]
s=1
Hence the mean population size is µ = eλt .
81
From (i),
∂ 2 G(s, t) −2eλt (1 − eλt )
= = −2eλt (1 − eλt ).
∂s2 s=1 [eλt + s(1 − eλt )]3 s=1
Hence the variance is given by
" 2 #
∂G(s, t) ∂ 2 G(s, t) ∂G(s, t)
V(t) = +s −
∂s ∂s2 ∂s
s=1
λt λt λt 2λt
= e − 2e (1 − e ) − e
= e2λt − eλt .
6.5. A random process has the probability generating function

2 + st r
G(s, t) = ,
2+t
where r is a positive integer. What is the initial state of the process? Find the probability pn (t) associated
with the generating function. What is pr (t)? Show that the mean associated with G(s, t) is
rt
µ(t) = .
2+t
Since G(s, 0) = 1, this implies that the initial size is 1.

Expansion of G(s, t) using the binomial theorem leads to
r r r
2 + st 2 1
G(s, t) = = 1+ st
2+t 2+t 2
∞
2 r X
r st n
=
2+t n 2
n=0
Hence the probability that the size is n at time t is

r n
2 r t
pn (t) = , (n = 0, 1, 2, . . .),
2+t n 2
With n = r,
r r r
2 r t rt
pr (t) = = .
2+t r 2 2+t
The mean size is given by

∂G(s, t) rt(2 + st)r−1 rt
µ(t) = = r
= .
∂s
s=1
(2 + t)
s=1
2 +t
6.6. In a simple birth and death process with unequal birth and death-rates λ and µ, the probability gener-
ating function is given by n0
µ(1 − s) − (µ − λs)e−(λ−µ)t
G(s, t) = ,
λ(1 − s) − (µ − λs)e−(λ−µ)t
for an initial population size n0 (see Equation (6.23)).
(a) Find the mean population size at time t.
(b) Find the probability of extinction at time t.
(c) Show that, if λ < µ, then the probability of ultimate extinction is 1. What is the probability if λ > µ?
(d) Find the variance of the population size.
82
(a) Let G(s, t) = [A(s, t)/B(s, t)]n0 with obvious definitions for A(s, t) and B(s, t). Then
∂G(s, t) [A(s, t)]n0 −1

= [(−µ + λe−(λ−µ)t )B(s, t) − (−λ + λe−(λ−µ)t ))A(s, t)].
∂s [B(s, t)]n0 +1
If s = 1, then A(1, t) = B(1, t) = −(µ − λ)e−(λ−µ)t . Therefore the mean population size is given by
µ = n0 e(λ−µ)t .
(b) The probability of extinction at time t is

n0
µ − µe−(λ−µ)t
p0 (t) = G(0, t) = .
λ − µe−(λ−µ)t
(c) If λ < µ, then e(λ−µ)t → 0 as t → ∞. Hence p0 (t) → 1.

If λ > µ, then p0 (t) → (µ/λ)n0 as t → ∞.
(d) This requires a lengthy differentiation to obtain the second derivative of G(s, t): symbolic compu-
tation is very helpful. The variance is given by
(λ + µ) (λ−µ)
V(t) = Gss (1, t) + Gs (1, t) − [Gs (1, t)]2 = n0 [e − 1], (λ 6= µ).
(λ − µ)
6.7. In a population model, the immigration rate λn = λ, a constant, and the death rate µn = nµ. For an
initial population size n0 , the probability generating function is (Example 6.3)
G(s, t) = eλs/µ exp[−λ(1 − (1 − s)e−µt )/µ][1 − (1 − s)e−µt ]n0 .
Find the probability that extinction occurs at time t. What is the probability of ultimate extinction?
The probability of extinction is
p0 (t) = G(0, t) = (1 − e−µt )n0 e−λ(1−e )/µ

−µt
.
The probability of ultimate extinction is
lim p0 (t) = e−λ/µ .

t→∞
6.8. In a general birth and death process a population is maintained by immigration at a constant rate λ,
and the death rate is nµ. Using the differential-difference equations (6.25) directly, obtain the differential
equation
dµ(t)
+ µµ(t) = λ,
dt
for the mean population size µ(t). Solve this equation assuming an initial population n0 and compare the
answer with that given in Example 6.3.
In terms of a generating function the mean of a process is given by

X
µ(t) = npn (t).
n=1
From (6.26) (in the book) the differential-difference equation for this immigration-death model is
dpn (t)
= λpn−1 (t) − (λ + nµ)pn (t) + µ(n + 1)pn+1 (t), (n = 1, 2, . . .).
dt
83
Multiply this equation by n and sum over over n:
∞ ∞ ∞ ∞ ∞
X dpn (t) X X X X
n = λ npn−1 (t) − λ npn (t) − µ npn (t) + µ n(n + 1)pn+1 (t)
dt
n=1 n=1 n=1 n=1 n=1
∞ ∞ ∞
X X X
= λ (n + 1)pn (t) − λµ(t) − µ n2 pn (t) + µ n(n − 1)pn (t)
n=0 n=1 n=2
= λµ(t) + λ − λµ(t) − µµ(t)
= λ − µµ(t).
Hence
dµ(t)
+ µµ(t) = λ.
dt
This is a first-order linear equation with general solution
λ
µ(t) = Ae−µt + .
µ
The initial condition implies A = n0 − (λ/µ). Hence

λ λ
µ(t) = n0 − e−µt + ,
µ µ
which is the same as the result in Example 6.3.
6.9. In a death process the probability of a death when the population size is n 6= 0 is a constant µ but
obviously zero if the population size is zero. Verify that, if the initial population is n0 , then pn (t), the
probability that the population size is n at time t is given by
Z t
µn0 (µt)n0 −n −µt
p0 (t) = sn0 −1 e−µs ds, pn (t) = e , (1 ≤ n ≤ n0 ),
(n0 − 1)! 0
(n0 − n)!
Show that the mean time to extinction is n0 /µ.
The probability that a death occurs in time δt is a constant µ independently of the population size.
Hence
pn0 (t + δt) = (1 − µ)pn0 (t), (i)
pn (t + δt) = µpn+1 (t) + (1 − µ)pn (t), (n = 1, . . . , n0 − 1), (ii)
p0 (t + δt) = µp1 (t) + p0 (t). (iii)
subject to the initial conditions pn0 (0) = 1, pn (0) = 0, (n = 0, 1, 2, . . . n0 − 1). Divide through each of the
eqns (i) and (ii) by δt, and let t → ∞ to obtain the differential-difference equations
p′n0 (t) = −µpn0 (t), (iv)
p′n (t) = −µpn (t) + µpn+1 (t), (n = 1, 2, . . . , n0 − 1). (v)

p′0 (t) = µp1 (t). (vi)
From (iv) and the initial conditions,
pn0 (t) = Ae−µt = e−µt .
From (v) with n = n0 − 1,
p′n0 −1 (t) = −µpn0 −1 (t) + µpn0 (t) = −µpn0 −1 (t) + e−µt .
Subject to the initial condition pn0 −1 (0) = 0, this first-order linear equation has the solution
pn0 −1 (t) = te−µt .
84
Repeat this process which leads to the conjecture that
(µt)n0 −n −µt
pn (t) = e , (n = 1, 2, . . . , n0 − 1),
(n0 − n)!
which can be proved by induction. The final probability satisfies
µn0 tn0 −1 −µt

p′0 (t) = µp1 (t) = e .
(n0 − 1)!
Direct integration gives Z t
µn0
p0 (t) = sn0 −1 e−µs ds. (vii)
(n0 − 1)! 0
It can be checked that p0 (t) → 1 as t → ∞ which confirms that extinction is certain using an integral
formula for the factorial again.
The probability distribution of the random variable T of the time to extinction is p0 (t) = P[T ≤ t]
given by (viii). Its density is
dp0 (t) µn0 tn0 −1 −µt
f (t) = = e .
dt (n0 − 1)!
Hence the expected value of T is
Z ∞ Z ∞
µn0 tn0 −µt n0
E(T ) = tf (t)dt = e = .
0 0
(n0 − 1)! µ
6.10. In a birth and death process the birth and death rates are given by
λn = nλ + α, µn = nµ,
where α represents a constant immigration rate. Show that the probability generating function G(s, t) of
the process satisfies
∂G(s, t) ∂G(s, t)
= (λs − µ)(s − 1) + α(s − 1)G(s, t).
∂t ∂s
Show also that, if
G(s, t) = (µ − λs)−α/λ H(s, t),
then H(s, t) satisfies
∂H(s, t) ∂H(s, t)
= (λs − µ)(s − 1) .
∂t ∂s
Let the initial population size be n0 . Solve the partial differential equation for H(s, t) using the method of
Section 6.5 and confirm that
(µ − λ)α/λ [(µ − λs) − µ(1 − s)e(λ−µ)t ]n0

G(s, t) = .
[(µ − λs) − λ(1 − s)e(λ−µ)t ]n0 +(α/λ)
(Remember the modified initial condition for H(s, t).)

Find p0 (t), the probability that the population is zero at time t (since immigration takes place even
when the population is zero there is no question of extinction in this process). Hence show that
α/λ
µ−λ
lim p0 (t) =
t→∞ µ
if λ < µ. What is the limit if λ > µ?

The long term behaviour of the process for λ < µ can be investigated by looking at the limit of the
probability generating function as t → ∞. Show that
α/λ
µ−λ
lim G(s, t) = .
t→∞ µ − λs
85
This is the probability generating function of a stationary distribution and it indicates that a balance has
been achieved the birth and immigration rates, and the death rate. What is the long term mean population
?
If you want a further lengthy exercise investigate the probability generating function size in the special
case λ = µ.
The differential-difference equations are
p′0 (t) = −αp0 (t) + µp1 (t), (i)
p′n (t) = [(n − 1)λ + α]pn−1 (t) − (nλ + α + nµ)pn (t) + (n + 1)µpn+1 (t), (n = 1, 2, . . .) (ii)
n
Multiply (ii) by s , sum over n ≥ 1 and add (i) leads to
∞ ∞
X X
p′n (t)sn = {[(n − 1)λ + α]pn−1 (t) − (nλ + α + nµ)pn (t)sn }
n=0 n=1
∞
X
+ (n + 1)µpn+1 (t)sn
n=0
P∞
Let the probability generating function be G(s, t) = n=0
pn (t)sn . Then the summations above lead to
∂G(s, t) ∂G(s, t)
= (λs − µ)(s − 1) + α(α − 1)G(s, t).
∂t ∂s
Let G(s, t) = (µ − λs)−α/λ H(s, t). Then
∂G(s, t) ∂H(s, t)
= (µ − λs)−α/λ + α(µ − λs)−(α/λ)−1 H(s, t),
∂s ∂s
and
∂G(s, t) ∂H(s, t)
= (µ − λs)−α/λ .
∂t ∂t
This transformation removes the non-derivative term to leave
∂H(s, t) ∂H(s, t)
= (λs − µ)(s − 1) .
∂t ∂s
Now apply the change of variable defined by
ds
= (λs − µ)(s − 1)
dz
as in Section 6.5(a). Integration gives (see eqn (6.21))
λ − µe(λ−µ)z
s= = h(z) (say). (iii)
λ − λe(λ−µ)z
The initial condition is equivalent to G(s, 0) = sn0 or H(s, 0) = (µ − λs)α/λ sn0 . It follows that
α/λ n0
λ(µ − λ) λ − µe(λ−µ)z
H(h(z), 0) = = w(z) (say).
λ − λe(λ−µ)z λ − λe(λ−µ)z
Since H(h(z), t) = w(z + t) for any smooth function of z + t, it follows that
G(s, t) = (µ − λs)−α/λ H(f (z), t) = (µ − λs)−α/λ w(z + t)

α/λ n0
λ(µ − λ) λ − µe(λ−µ)(z+t)
= (µ − λs)−α/λ ,
λ − λe(λ−µ)(z+t) λ − λe(λ−µ)(z+t)
where z is defined by s = h(z) given by (iii). Finally
(µ − λ)α/λ [(µ − λs) − µ(1 − s)e(λ−µ)t ]n0

G(s, t) = , (iv)
[(µ − λs) − λ(1 − s)e(λ−µ)t ]n0 +(α/λ)
86
as displayed in the problem.
The probability that the population is zero at time t is
(µ − λ)α/λ (µ − µe(λ−µ)t )n0

p0 (t) = G(0, t) = .
(µ − λe(λ−µ)t )n0 +(α/λ)
If λ < µ, then
α/λ
µ−λ
p0 (t) → .
µ
If λ > µ, then
(µ − λ)α/λ (µe−(λ−µ)t − µ)n0
α/λ n0
λ−µ µ
p0 (t) = → ,
(µe−(λ−µ)t − λ)n0 +(α/λ) λ λ
as t → ∞.
The long term behaviour for λ < µ is determined by letting t → ∞ in (iv) resulting in
α/λ
µ−λ
lim G(s, t) = .
t→∞ µ − λs
Express G(s, t) in the form

A(s, t)
G(s, t) = (µ − λ)α/λ ,
B(s, t)
where
A(s, t) = [(µ − λs) − µ(1 − s)e(λ−µ)t ]n0 , B(s, t) = [(µ − λs) − λ(1 − s)e(λ−µ)t ]n0 +(α/λ) .
Then
As (s, t)B(s, t) − A(s, t)Bs (s, t) ∂A ∂b
Gs (s, t) = (µ − λ)α/λ As = , Bs = .
[B(s, t)]2 ∂s ∂s
For the mean we require s = 1, for which value
A(1, t) = (µ − λ)n0 , B(1, t) = (µ − λ)n0 +(α/λ) ,
As (1, t) = n0 (−λ + µe(λ−µ)t )(µ − λ)n0 −1 ,

Bs (1, t) = (n0 + (α/λ))(−λ + λe(λ−µ)t )(µ − λ)n0 +(α/λ)−1 .
Hence
As (1, t)B(1, t) − A(1, t)Bs (1, t)
µ(t) = Gs (1, t) = (µ − λ)α/λ
[B(1, t)]2
n0
= [α + {n0 (µ − λ) − α}e(λ−µ)t ].
µ−λ
If λ < µ, then
n0 α
,
µ(t) →
µ−λ
as t → ∞. If λ > µ, then the mean becomes unbounded as would be expected.
6.11. In a birth and death process with immigration, the birth and death rates are respectively
λn = nλ + α, µn = nµ.
Show directly from the differential-difference equations for pn (t), that the mean population size µ(t) satisfies
the differential equation
dµ(t)
= (λ − µ)µ(t) + α.
dt
Deduce the result
α
µ(t) →
µ−λ
as t → ∞ if λ < µ. Discuss the design of a deterministic immigration model based on this equation.
87
The difference equations for the probability pn (t) are given by
p′0 (t) = −αp0 (t) + µp1 (t), (i)
p′n (t) = [(n − 1)λ + α]pn−1 (t) − [(nλ + α + nµ)pn (t) + (n + 1)µpn+1 (t). (ii)
P∞
The mean µ(t) is given by µ(t) = n=1
npn (t). Multiply (ii) by n and sum from n = 1. Then, re-ordering
the sums
∞ ∞ ∞ ∞
dµ(t) X ′ X X X
= npn (t) = λ n(n − 1)pn−1 (t) + α pn−1 (t) − α npn (t)
dt
n=1 n=2 n=1 n=1
∞ ∞
X X
−(λ + µ) n2 pn (t) + µ n(n + 1)pn+1 (t)
n=1 n=1
∞ ∞ ∞
X X X
= λ n(n + 1)pn (t) + α pn (t) − α npn (t)
n=1 n=0 n=1
∞ ∞
X X
−(λ + µ) n2 pn (t) + µ n(n − 1)pn (t)
n=1 n=2
= α + (λ − µ)µ(t).
The mean of the stochastic process satisfies a simple deterministic model for a birth and death process
with immigration.
6.12. In a simple birth and death process with equal birth and death rates λ, the initial population size has
a Poisson distribution with probabilities
αn
pn (0) = e−α (n = 0, 1, 2, . . .),
n!
with intensity α. It could be thought of as a process in which the initial distribution has arisen as the result
of some previous process. Find the probability generating function for this process, and confirm that the
probability of extinction at time t is exp[−α/(1 + λt)], and that the mean population size is α for all t.
In Section 6.5(b), the probability generating function G(s, t) for the case in which the birth and death
rates are equal satisfies
∂G(s, t) ∂G(s, t)
= λ(1 − s)2 .
∂t ∂s
To solve the partial differential equation, the transformation
1 λz − 1
z= , or s= (i)
λ(1 − s) λz
is used. The result is that G(s, t) = w(z + t) for any smooth function w. The initial condition at t = 0 is
∞ ∞ h i
X X αn e−α λz − 1
w(z) = pn (0)sn = sn = eα(s−1) = exp α −1 = e−α/(λz) ,
n! λz
n=0 n=0
usng the transformation (i). Hence

1 −α(1 − s)
G(s, t) = w(z + t) = e−α/[λ(z+t)] = exp −α + λt = exp
1−s 1 + λt(1 − s)
The probability of extinction at time t is
p0 (t) = G(0, t) = exp[−α/(1 + λt)].
The mean
µ(t) = Gs (1, t) = α.
88
6.13. A birth and death process takes place as follows. A single bacterium is allowed to grow and assumed
to behave as a simple birth process with birth rate λ for a time t1 without any deaths. No further growth
then takes place. The colony of bacteria is then allowed to die with the assumption that it is a simple death
process with death rate µ for a time t2 . Show that the probability of extinction after the total time t1 + t2 is
∞
X
eλt1 (1 − e−λt1 )n−1 (1 − e−µt2 )n .
n=1
Using the formula for the sum of a geometric series, show that this probability can be simplified to
eµt2 − 1
.
eλt1 + eµt2 − 1
Suppose that at time t = t1 the population size is n. From Section 6.3, the probability that the
population is of size n at time t1 entirely through births is
pn (t1 ) = e−λt1 (1 − e−λt1 )n−1 .
From Section 6.4 on the death process, given that the population is of size n, the probability that the
population becomes extinct after a further time t2 is
q0 (t2 ) = (1 − e−µt2 )n .
The probability that the population increases to n and then declines to zero is
pn (t1 )q0 (t2 ) = e−λt1 (1 − e−λt1 )n−1 (1 − e−µt2 )n
Now n can take any value equal to or greater than 1. Hence the probability of extinction through every
possible n is
∞
X
s(t1 , t2 ) = e−λt1 (1 − e−λt1 )n−1 (1 − e−µt2 )n .
n=1
The probability s(t1 , t2 ) can be expressed as a geometric series in the form

∞
e−λt1 X
s(t1 , t2 ) = [(1 − e−λt1 )(1 − e−µt2 )]n
1 − e−λt1
n=1
e−λt1 (1 − e−λt1 )(1 − e−µt2 )

= ·
(1 − e−λt1 ) [1 − (1 − e−λt1 )(1 − e−µt2 )]
eµt2 − 1
= .
eλt1 + eµt2 − 1
6.14. As in the previous problem a single bacterium grows as a simple birth process with rate λ and no
deaths for a time τ . The colony numbers then decline as a simple death process with rate µ. Show that the
probability generating function for the death process is
(1 − e−µt (1 − s))e−λτ
,
1 − (1 − e−λτ )(1 − e−µt (1 − s))
where t is measured from the time τ . Show that the mean population size during the death process is eλτ −µt .
During the birth process the generating function is (see eqn (6.12))
se−λt
G(s, t) = ,
1 − (1 − e−λt )s
89
assuming an initial population of 1. For the death process suppose that time restarts from t = 0, and that
the new probability generating function is H(s, t). At t = 0,
se−λτ
H(s, 0) = G(s, τ ) = .
1 − (1 − e−λτ )s
For the death process the transformation is s = 1 − e−µz , so that
(1 − e−µz )e−λτ
H(s, 0) = w(z) = .
1 − (1 − e−λτ )(1 − e−µz )
Then, in terms of s,
[1 − e−µt (1 − s)]e−λt
H(s, t) = w(z + t) = .
1 − (1 − e−λτ )[1 − e−µt (1 − s)]
The mean population size in the death process is
[1 − (1 − e−λτ )]e−µt−λτ + e−λτ −µt (1 − e−λτ )

Hs (1, t) = = eλτ −µt .
[1 − (1 − e−λτ ]2
6.15. For a simple birth and death process the probability generating function ( equation (6.23)) is given by
n0
µ(1 − s) − (µ − λs)e−(λ−µ)t
G(s, t) =
λ(1 − s) − (µ − λs)e−(λ−µ)t
for an initial population of n0 . What is the probability that the population is (a) zero, (b) 1 at time t?
n0
µ(1 − s) − (µ − λs)e−(λ−µ)t
G(s, t) =
λ(1 − s) − (µ − λs)e−(λ−µ)t
[{µ − µe−(λ−µ)t } − s{µ − λe−(λ−µ)t }]n0
=
[{λ − µe−(λ−µ)t } − s{λ − λe−(λ−µ)t }]n0
n0
µ − µe−(λ−µ)t λ − λe−(λ−µ)t µ − λe−(λ−µ)t
= 1 + n0 s − + ···
λ − µe−(λ−µ)t λ − µe−(λ−µ)t µ − µe−(λ−µ)t
The probabilities p0 (t) and p1 (t) are given by the first two coefficients of s in this series.
6.16. (An alternative method of solution for the probability generating function) The general solution of
the first-order partial differential equation
∂z ∂z
A(x, y, z) + B(x, y, z) = C(x, y, z)
∂x ∂y
is f (u, v) = 0, where f is an arbitrary function, and u(x, y, z) = c1 and v(x, y, z) = c2 are two independent
solutions of
dx dy dz
= = .
A(x, y, z) B(x, y, z) C(x, y, z)
This is known as Cauchy’s method.
Apply the method to the partial differential equation for the probability generating function for the
simple birth and death process, namely (equation (6.19))
∂G(s, t) ∂G(s, t)
= (λs − µ)(s − 1) ,
∂t ∂s
by solving
ds dt dG
= = .
(λs − µ)(1 − s) −1 0
90
Show that
1−s
u(s, t, G) = G = c1 , and v(s, t, G) = e(λ−µ)t µ = c2 .
λ
−s
are two independent solutions. The general solution can be written in the form

(λ−µ)t 1−s
G(s, t) = H e µ .
λ
−s
Here H is a function determined by the initial condition G(s, 0) = sn0 . Find H and recover formula (6.22)
for the probability generating function.
Note that in the birth and death equation the function C is zero. Comparing the two partial differential
equations, we have to solve
ds dt dG
= = .
(λs − µ)(1 − s) −1 0
The second equality is simply dG = 0. This equation has a general solution which can be expressed as
u(s, t, G) ≡ G = c1 .
The first equality requires the solution of the differential equation

ds
= −(λs − µ)(1 − s).
dt
The integration is given essentially in eqn (6.20) in the text which in terms of v can be expressed as

(λ−µ)t 1−s
v(s, t, G) ≡ e = c2 .
(µ/λ) − s
Hence the general solution is

1−s
f (u, v) = 0, or f G, e(λ−µ)t = 0.
(µ/λ) − s
Alternatively, this can be written in the form

1−s
G(s, t) = H e(λ−µ)t ,
(µ/λ) − s
where he function H is determined by initial conditions.

Assuming that the initial population size is n0 , then G(s, 0) = sn0 , which means that H is determined
by
1−s
H = sn0 .
(µ/λ) − s
Let u = (1 − s)/((µ/λ) − s). Then
h λ − µu in0
H(u) = ,
λ − λu
which determines the functional form of H. The result follows by replacing u by

1−s
e(λ−µ)t ,
(µ/λ) − s
as the argument of H.
6.17. Apply the Cauchy’s method outlined in Problem 6.16 to the immigration model in Example 6.3. In
this application the probability generating function satisfies
∂G(s, t) ∂G(s, t)
= λ(s − 1)G(s, t) + µ(1 − s) .
∂t ∂s
91
Solve the equation assuming an initial population of n0 .
Reading the coefficients of the partial differential equation in Problem 6.16,

ds dt dG
= = .
µ(1 − s) −1 λ(s − 1)
Integration of the second equality gives
λ
u(s, t, G) ≡ G + s = c1 ,
µ
whilst the first gives
v(s, t, G) = eµt (1 − s) = c2 .
The general solution can be expressed in the functional form
λ
G(s, t) = − s + H(eµt (1 − s)).
µ
From the initial condition G(s, 0) = sn0 . Therefore
λ
− s + H(1 − s) = sn0 ,
µ
so that
λ
H(1 − s) = s + sn0 .
µ
Let u = 1 − s: then
λ
H(u) = (1 − u) + (1 − u)n0 .
µ
The result follows by replacing u by eµt (1 − s) in this formula.
6.18. In a population sustained by immigration at rate λ with a simple death process with rate µ, the
probability pn (t) satisfies
dp0 (t)
= −λp0 (t) + µp1 (t),
dt
dpn (t)
= λpn−1 (t) − (λ + nµ)pn (t) + (n + 1)µpn+1 (t).
dt
Investigate the steady-state behaviour of the system by assuming that
pn (t) → pn , dpn (t)/dt → 0
for all n, as t → ∞. Show that the resulting difference equations for what is known as the corresponding
stationary process
−λp0 + µp1 = 0,
λpn−1 − (λ + nµ)pn + (n + 1)µpn+1 = 0, (n = 1, 2, . . .)
can be solved iteratively to give
λ λ2 λn
p1 = p0 , p2 = p0 , ··· pn = p0 , ···.
µ 2!µ2 n!µn
P∞
Using the condition p = 1, and assuming that λ < µ, determine p0 . Find the mean steady-state
n=0 n
population size, and compare the result with that obtained in Example 6.3.
From the steady-state difference equations

λ
p1 = p0 ,
µ
1 λ2
p2 = [−λp0 + (λ + 2µ)p1 ] = ,
2µ 2!µ2
92
1 λ3
p3 = [−λp1 + (λ + 2µ)p2 ] = p0 ,
3µ 3!µ3
P∞
and so on: the result can be confirmed by an induction proof. The requirement n=0
pn = 1 implies
∞ n
X λ 1
p0 = eλ/µ p0 = 1
µ n!
n=0
if p0 = e−λ/µ .
The mean steady state population is given by
∞
X ∞
X n ∞
X n
np0 λ p0 λ
µ = npn = =
n! µ (n − 1)! µ
n=1 n=1 n=1

λ λ
= p0 eλ/µ = .
µ µ
6.19. In a simple birth process the probability that the population is of size n at time t given that it was n0
at time t = 0 is given by

n−1
pn (t) = e−λn0 t (1 − e−λt )n−n0 , (n ≥ n0 ).
n0 − 1
(see Section 6.3 and Figure 6.1). Show that the probability achieves its maximum value for given n and n0
when t = (1/λ) ln(n/n0 ). Find also the maximum value of pn (t) at this time.
Differentiating pn (t), we obtain

dpn (t) n−1
=λ e−λn0 t (1 − e−λt )n−n0 −1 [−n0 (1 − e−λt ) + λ(n − n0 )e−λt ].
dt n0 − 1
The derivative is zero if

n0 1 n
e−λt =, or t= ln .
n λ n0
Substituting this time back into pn (t), it follows

n − 1 nn 0
max[pn (t)] = 0
(n − n0 )n−n0 .
t n0 − 1 nn
6.20. In a birth and death process with equal birth and death parameters, λ, the probability generating
function is (see eqn(6.24)) n0
1 + (λt − 1)(1 − s)
G(s, t) = .
1 + λt(1 − s)
Find the mean population size at time t. Show also that the variance of the population size is 2n0 λt.
The derivative of G(s, t) with respect to s is given by
n0 [λt − (λt − 1)s]n0 −1

Gs (s, t) = .
[(1 + λt) − λts]n0 +1
The mean population size at time t is
µ(t) = Gs (1, t) = n0 .
We require the second derivative given by
n0 [λt − (λt − 1)s]n0 −2

Gss (s, t) = [n0 − 1 + 2λts + 2λ2 t2 (1 − s)]
[(1 + λt) − λts]n0 +2
93
The variance of the population size is
V(t) = Gss (1, t) + Gs (1, t) − [Gs (1, t)]2 = [2λn0 t + n20 − n0 ] + n0 − n20 = 2n0 λt.
6.21. In a death process the probability that a death occurs in time δt is a time-dependent parameter µ(t)n
when the population size is n. The pgf G(s, t) satisfies
∂G ∂G
= µ(t)(1 − s) .
∂t ∂s
as in Section 6.4. Show that
G(s, t) = [1 − e−τ (1 − s)]n0 ,
where Z t
τ = µ(u)du.
0
Find the mean population size at time t.
In a death process it is found that the expected value of the population size at time t is given by
n0
µ(t) = , (t ≥ 0),
1 + αt
where α is a positive constant. Estimate the corresponding death-rate µ(t).
Let Z
ds
z= , so that s = 1 − e−z ,
1−s
and Z t
τ= µ(u)du.
0
The equation for the probability generating function becomes
∂G ∂G
= .
∂τ ∂z
The general solution can be expressed as G(s, t) = w(z + τ ) for any arbitrary differentiable function w.
Initially τ = 0. Hence
G(s, 0) = sn0 = w(z) = (1 − e−z )n0 ,
so that
G(s, t) = w(z + τ ) = [1 − e−(z+τ )]n0 = [1 − e−τ (1 − s)]n0 .
The mean population size at time t is given by
Z t

µ(t) = Gs (1, t) = n0 e−τ [1 − e−tau (1 − s)]n0 −1 s=1 = n0 e−τ = n0 exp − µ(u)du .
0
Given the mean Z

t
n0
µ(t) = = n0 exp − µ(u)du , (i)
1 + αt 0
it follows that the death-rate is µ(t) = α/(1 + αt), which can be obtained by differentiating both sides of
(i) with respect to t
6.22. A population process has a probability generating function G(s, t) which satisfies the equation
∂G ∂G
e−t = λ(s − 1)2 .
∂t ∂s
If, at time t = 0, the population size is n0 , show that
n0
1 + (1 − s)(λet − λ − 1)
G(s, t) = .
1 + λ(1 − s)(et − 1)
94
Find the mean population size at time t, and the probability of ultimate extinction.
The generating function satisfies

∂G ∂G
e−t = λ(x − 1)2 .
∂t ∂s
Let Z Z
t
u ds
τ= e du, z= .
0
λ(s − 1)2
The transformations are τ = et − 1 and s = (λz − 1)/(λz). The transformed partial differential equation
has the general solution G(s, t) = w(z + τ ). The initial condition is
n0
1
G(s, 0) = sn0 = 1 − = w(z).
λz
Hence
n0 n0
1 s−1
G(s, t) = 1− = 1−
λ(z + τ ) −1 + λ(s − 1)(et − 1)
n0
(s − 1)(λet − λ − 1) − 1
=
λ(s − 1)(et − 1) − 1
as required.
The mean population size is
µ(t) = Gs (1, t) = n0 .
The probability of extinction at time t is
n0
λ(et − 1)
p0 (t) = G(0, t) = →1
λ(et − 1) + 1
as t → ∞.
6.23. A population process has a probability generating function given by
1 − µe−t (1 − s)
G(s, t) = ,
1 + µe−t (1 − s)
where µ is a parameter. Find the mean of the population size at time t, and its limit as t → ∞. Expand
G(s, t) in powers of s, determine the probability that the population size is n at time t.
We require the derivative

µe−t [1 + µe−t (1 − s)] + µe−t (1 − s)[1 − µe−t (1 − s)]
Gs (s, t) =
[1 + µe−t (1 − s)]2
2µe−t
=
[1 + µe−t (1 − s)]2
Then the mean population size is
µ(t) = Gs (1, t) = 2µe−t .
To find the individual probabilities we require the power series expansion of G(s, t). Using a binomial
expansion
−1
1 − µe−t µe−t s µe−t s
G(s, t) = 1+ 1−
1 + µe−t 1 − µe−t 1 + µe−t
∞
X n
1 − µe−t µe−t s µe−t
= 1+ sn
1 + µe−t 1 − µe−t 1 + µe−t
n=0
−t
"X
∞ −t
n X
∞ n #
1 − µe µe n µe−t µe−t n+1
= s + s
1 + µe−t 1 + µe−t 1 − µe−t 1 + µe−t
n=0 n=1
95
the coefficients of the powers sn give the following probabilities:
1 − µe−t
p0 (t) = ,
1 + µe−t

1 − µe−t µn e−nt µn e−nt 1 2µn e−nt
pn (t) = + = , (n ≥ 1).
1 + µe−t −t
(1 + µe ) n (1 + µe−t )n−1 (1 − µe−t) (1 + µe−t )n+1
6.24. In a birth and death process with equal rates λ, the probability generating function is given by (see
eqn (6.24)) n0 n0
λ(z + t) − 1 1 + (λt − 1)(1 − s)
G(s, t) = = ,
λ(z + t) 1 + λt(1 − s)
where n0 is the initial population size. Show that pi , the probability that the population size is i at time t,
is given by
i
X n0 n0 + i − m − 1
pi (t) = α(t)m β(t)n0 +i−m
m i−m
m=0
if i ≤ n0 , and by
n0
X n0 n0 + i − m − 1
pi (t) = α(t)m β(t)n0 +i−m
m i−m
m=0
if i > n0 , where
1 − λt λt
α(t) = , β(t) = .
λt 1 + λt
Expand G(s, t) as a power series in terms of s using the binomial expansion:

λt n0 h 1 − λt
in0 h λt
i−n0
G(s, t) = 1+ s 1− s
1 + λt λt 1 + λt
n0
n0 X ∞
λt n0 X n0 + j − 1 j j
= αk sk β s ,
1 + λt k j
k=0 j=0
where
1 − λt λt
α= , β= .
λt 1 + λt
Two cases have to be considered.
(a) i ≤ n0 .
λt n0 n
n0 + i − 1 i

n0 n0 + i − 2 i−1

0 0
pi (t) = α β + α β + ···
1 + λt 0 i 1 i−1

n0 n0 − 1 0
+ αi β
i 0
i
n0 +i X m
λt n0 n0 + i − 1 − m α
=
1 + λt m i−m β
m=0
i
n0 +i X m
λt n0 n0 + i − 1 − m 1 − (λt)2
=
1 + λt m i−m (λt)2
m=0
(b) i > n0 .
λt n0 n
n0 + i − 1 i

n0

i−1

0
pi (t) = α0 β +··· + αn0 β i−n0
1 + λt 0 i n0 i − n0
n0 m
1 − (λt)2
λt n0 +i X n0 n0 + i − 1 − m
=
1 + λt m i−m (λt)2
m=0
96
6.25. We can view the birth and death process by an alternative differencing method. Let pij (t) be the
conditional probability
pij (t) = P(N (t) = j|N (0) = i),
where N (t) is the random variable representing the population size at time t. Assume that the process is in
the (fixed) state N (t) = j at times t and t + δt and decide how this can arise from an incremental change
δt in the time. If the birth and death rates are λj and µj , explain why
pij (t + δt) = pij (t)(1 − λi δt − µi δt) + λi δtpi+1,j (t) + µi δtpi−1,j (t) + o(δt)
for i = 1, 2, 3, . . ., j = 0, 1, 2, . . .. Take the limit as δt → 0, and confirm that pij (t) satisfies the differential
equation
dpij (t)
= −(λi + µi )pij (t) + λi pi+1,j (t) + µi pi−1,j (t).
dt
How should p0,j (t) be interpreted?
In this approach the final state in the process remains fixed, that is, the j in pij . We now view pij (t+δt)
as pij (δt + t) — in other words see what happens in an initial δt. There will be a birth with probability
λi in a time δt or a death with probability µi . Then
pij (t + δt) = pij (t)[1 − λi δt − µi δt] + λi δt + µi δtpi−1,j (t) + o(δt)
for i = 1, 2, 3, . . .; j = 0, 1, 2, 3, . . .. Hence
pij (t + δt) − pij (t)
= −(λi + µi )pij (t) + λi pi+1,j (t) + µi pi−1,j (t) + o(1).
δt
In the limit δt → 0,
dpij (t)
= −(λi + µi )pij (t) + λi pi+1,j (t) + µi pi−1,j (t),
dt
where we require
0, j>0
p0,j (t) = P(N (t) = j|N (0) = 0) =
1, j=0
6.26. Consider a birth and death process in which the rates are λi = λi and µi = µi, and the initial
population size is n0 = 1. If
p1,j = P(N (t) = j|N (0) = 1),
it was shown in Problem 6.25 that p1,j satisfies
dp1,j (t)
= −(λ + µ)p1,j (t) + λp2,j (t) + µp0,j (t), (j = 0, 1, 2, . . .)
dt
where
0, j>0
p0,j (t) =
1, j=0
If
∞
X
G(i, s, t) = pij (t)sj ,
j=0
show that
∂G(1, s, t)
= −(λ + µ)G(1, s, t) + λG(2, s, t) + µ.
∂t
2
Explain why G(2, s, t) = [G(1, s, t)] (see Section 6.5). Hence solve what is effectively an ordinary differ-
ential equation for G(1, s, t), and confirm that
µ(1 − s) − (µ − λs)e−(λ−µ)t
G(1, s, t) = ,
λ(1 − s) − (µ − λs)e−(λ−µ)t
as in eqn (6.23) with n0 = 1.
97
Given
dp1,j (t)
= −(λ + µ)p1,j (t) + λp2,j (t) + µp0,j (t),
dt
j
multiply the equation by s and sum over j = 1, 2, . . .. Then
∂G(1, s, t)
= −(λ + µ)G(1, s, t) + λG(2, s, t) + µ(t).
∂t
Also
G(2, s, t) = E[sN1 (t) ]E[sN2 (t) ] = E[sN1 (t) ]2 = G(1, s, t)2 .
Therefore
∂G(1, s, t)
= −(λ + µ)G(1, s, t) + λG(1, s, t)2 + µ.
∂t
This is a separable first-order differential equation with general solution
Z Z
dG(1, s, t)
= dt + A(x) = t + A(x),
(λG(1, s, t) − µ)(G(1, s, t) − 1)
where the ‘constant’ is a function of x. Assume that |G(1, s, t)| < min(1, µ/λ). Then
Z Z
λ dG(1, s, t) 1 dG(1, s, t)
t + A(x) = − +
λG(1, s, t) − µ λG(1, s, t) − µ λ−µ G(1, s, t) − 1

1 1 − G(1, s, t)
= ln .
λ−µ µ − λG(1, s, t)
Hence
µ + B(x)e−(λ−µ)t
G(1, s, t) = ,
λ + B(x)e−(λ−µ)t
where B(x) (more convenient than A(x)) is a function to be determined by the initial conditions.
Initially, G(1, s, 0) = s, so that B(x) = (µ − λs)/(1 − s). Finally G(1, s, t) agrees with G(1, s, t) in
eqn (6.23) with n0 = 1.
6.27. In a birth and death process with parameters λ and µ, (µ > λ), and initial population size n0 , show
that the mean time to extinction of the random variable Tn0 is given by
Z ∞
te−(µ−λ)t [µ − µe−(µ−λ)t ]n0 −1
E(Tn0 ) = n0 µ(µ − λ)2 dt.
0
[µ − λe−(µ−λ)t ]n0 +1
If n0 = 1, using integration by parts, evaluate the integral over the interval (0, τ ), and then let τ → ∞ to
show that
1 µ−λ
E(T1 ) = − ln .
λ µ
The distribution function for Tn0 is given by

n0
µ − µe−(µ−λ)t
F (t) = p0 (t) = G(0, t) = ,
µ − λe−(µ−λ)t
(put s = 0 in (6.23)). Its density is
dF (t) [µ − µe−(µ−λ)t ]n0 −1

f (t) = = n0 µ(µ − λ)2 e−(µ−λ)t , (t > 0).
dt [µ − λe−(µ−λ)t ]n0 +1
The mean time to extinction is
Z ∞ Z ∞
2 [µ − µe−(µ−λ)t ]n0 −1
E(Tn0 ) = tf (t)dt = n0 µ(µ − λ) te−(µ−λ)t dt,
0 0
[µ − λe−(µ−λ)t ]n0 +1
as required.
98
If n0 = 1, then
Z ∞
te−(µ−λ)t
E(T1 ) = µ(µ − λ)2 dt
0
[µ − λe−(µ−λ)t ]2
Z ∞
(µ − λ)2 te−(µ−λ)t
= dt
µ 0
[1 − (λ/µ)e−(µ−λ)t ]2
Integrate the following finite integral between t = 0 and t = τ :
Z τ
Z τ
te−(µ−λ)t dt µ t dt
= − +
0
[1 − (λ/µ)e−(µ−λ)t ]2 λ(µ − λ) 1 − (λ/µ)e−(µ−λ)t 1 − (λ/µ)e−(µ−λ)t 0
τ
µ t 1
= − + ln{e(µ−λ)t − (λ/µ)}
λ(µ − λ) 1 − (λ/µ)e−(µ−λ)t µ−λ 0

µ τ 1
= − +τ + ln{1 − (λ/µ)e−(µ−λ)τ }
λ(µ − λ) 1 − (λ/µ)τ µ−λ

1
− ln{1 − (λ/µ)}
µ−λ
µ
→ ln{1 − (λ/µ)},
λ(µ − λ)2
as τ → ∞. Finally
1 µ−λ
E(T1 ) = − ln .
λ µ
6.28. A death process (see Section 6.4) has a parameter µ and the initial population size is n0 . Its probability
generating function is
G(s, t) = [1 − e−µt (1 − s)]n0 .
Show that the mean time to extinction is
n0 −1
n0 X (−1)k n0 − 1
.
µ (k + 1)2 k
k=0
Let Tn0 be a random variable representing the time to extinction. The probability distribution of Tn0
is given by
F (t) = p0 (t) = G(0, t) = (1 − e−µt )n0 .
The mean time to extinction is
Z ∞ Z ∞
dp0 (t)
E(Tn0 ) = t dt = n0 µ te−µt (1 − e−µt )n0 −1 dt.
0
dt 0
Replace (1 − e−µt )n0 −1 by its binomial expansion, namely

∞
X n0 − 1 −kµt
(1 − e−µt )n0 −1 = (−1)k e ,
k
k=0
and integrate the series term-by-term:

∞ Z ∞ ∞
X n0 − 1 n0 X (−1)k n0 − 1
E(Tn0 ) = n0 µ (−1)k te−(k+1)µt dt = .
k 0
µ (k + 1)2 k
k=0 k=0
6.29. A colony of cells grows from a single cell without deaths. The probability that a single cell divides
into two cells in a time interval δt is λδt + o(δt). As in Problem 6.1, the probability generating function
for the process is
se−λt
G(s, t) = .
1 − (1 − e−λt )s
99
By considering the probability
n−1
X
P(Tn ≤ t) = F (t) = 1 − pk (t),
k=1
where Tn is the random variable representing the time that the population is of size n(≥ 2) for the first
time, show that
n−1
1X1
E(Tn ) = .
λ k
k=1
The expansion of the generating function is

∞
X
G(s, t) = e−λt (1 − e−λt )n−1 sn .
n=1
Consider the probability function

∞ n−1 n−1
X X X
F (t) = pk (t) = 1 − pk (t) = 1 − e−λt (1 − e−λt )k−1 .
k=n k=1 k=1
Its density is defined by, for n ≥ 3, (although it is not required)

n−1
dF (t) X
f (t) = = λe−λt + λe−λt (1 − ke−λt )(1 − e−λt )k−2 , (t > 0)
dt
k=2
and for n = 2,
f (t) = λe−λt , (t > 0).
Then,
Z τ
Z τ

dF (t)
E(Tn ) = lim t dt = lim [tF (t)τ0 − F (t)dt
τ →∞ dt τ →∞
"0 n−1 Z
0
( n−1
) #
X τ X
−λτ −λτ k−1 −λt −λt k−1
= lim τ− e (1 − e ) − 1− e (1 − e ) dt
τ →∞
k=1 0 k=1
" n−1 n−1
#
X −λτ −λτ k−1 1X1
= lim τ− e (1 − e ) −τ + (1 − e−λτ )k
τ →∞ λ k
k=1 k=1
n−1
1X1
= .
λ k
k=1
As n → ∞ the series diverges, so that E(Tn ) → ∞ with n.
6.30. In a birth and death process, the population size represented by the random variable N (t) grows as
a simple birth process with parameter λ. No deaths occur until time T when the whole population dies.
Suppose that the random variable T has an exponential distribution with parameter µ. The process starts
with one individual at time t = 0. What is the probability that the population exists at time t, namely that
P[N (t) > 0]?
What is the conditional probability P[N (t) = n|N (t) > 0] for n = 1, 2, . . .? Hence show that
P[N (t) = n] = e−(λ+µ)t (1 − e−λt )n−1 .
Construct the probability generating function of this distribution, and find the mean population size at time
t.
Since the decline is a Poisson process with intensity µ
P[N (t) > 0] = e−µt .
100
This must be a simple birth process conditional on no deaths, namely
P[N (t) = n|N (t) > 0] = e−λt (1 − e−λt )n−1 , (n = 1, 2, . . .).
Hence
P[N (t) = 0] = 1 − P[N (t) > 0] = 1 − e−µt ,
P[N (t) = n] = P[N (t) = n|N (t) > 0]P[N (t) > 0] = e−(λ+µ)t (1 − e−λt )n−1 , (n = 1, 2, . . .).
The probability generating function is G(s, t), where
∞ ∞
X X
G(s, t) = P[N (t) = n]sn = 1 − e−µt + e−(λ+µ)t (1 − e−λt )n−1 sn
n=0 n=1
s
= 1 − e−µt + e−(λ+µ)t
1 − s(1 − e−λt )
using the formula for the sum of a geometric series.
For the mean, we require

∂G(s, t) 1 s(1 − e−λt )
= e−(λ+µ)t −λt
+ .
∂s 1 − s(1 − e ) [1 − s(1 − e−λt )]2
Then the mean

∂G(s, t)
µ(t) = = e−(λ+µ)t [eλt + e2λt − eλt ] = e(λ−µ)t .
∂s s=1
6.31. In a birth and death process, the variable birth and death rates are, for t > 0, respectively given by
λn (t) = λ(t)n > 0, (n = 0, 1, 2, . . .) µn (t) = µ(t)n > 0, (n = 1, 2, . . .).
If pn (t) is the probability that the population size at time t is n, show that its probability generating function
is
∞
X
G(s, t) = pn (t)sn ,
n=0
satisfies
∂G ∂G
= (s − 1)[λ(t)s − µ(t)] .
∂t ∂s
Suppose that µ(t) = αλ(t) (α > 0, α 6= 1), and that the initial population size is n0 . Show that
n0 1−s Z t
1 − αq(s, t)
G(s, t) = where q(s, t) = exp (1 − α) λ(u)du .
1 − q(s, t) α−s 0
Find the probability of extinction at time t.
Using eqns (6.25), the differential-difference equations are
p′0 (t) = µ(t)p1 (t),
p′n (t) = (n − 1)λ(t)pn−1 (t) − n[λ(t) + µ(t)]pn (t) + (n + 1)µ(t)pn+1 (t), (n = 1, 2, . . .).
In the usual way multiply the equations by sn and sum over n:
∞ ∞ ∞ ∞
X X X X
p′n (t)sn = (n − 1)λ(t)pn−1 (t)sn − [λ(t) + µ(t)] npn (t)sn + (n + 1)µ(t)pn+1 (t)sn
n=0 n=2 n=1 n=0
P∞ n
Let G(s, t) = n=0
pn (t)s . Then the series can be expressed in terms of G(s, t) as
∂G ∂G ∂G ∂G
= λ(t)s2 − [λ(t) + µ(t)]s + µ(t)
∂t ∂s ∂s ∂s
∂G
= (s − 1)[λ(t)s − µ(t)]
∂s
101
Let µ(t) = αλ(t), (α 6= 1). Then
∂G ∂G
= λ(t)(s − 1)(s − α) .
∂t ∂s
Let dτ = λ(t)dt so that τ can be defined by
Z t
τ = λ(u)du.
0
Let ds/dz = (s − 1)(s − α) and define z by

Z Z h i
ds 1 1 1
z = = − ds
(s − 1)(s − α) 1−α α−s 1−s
1
h1−si
= ln ,
1−α α−s
where s < min(1, α). Inversion of this equation gives
1 − αe(1−α)z
s= = q(z),
1 − e(1−α)z
say. Let G(s, t) = Q(z, τ ) after the change of variable. Q(z, τ ) satisfies
∂Q ∂Q
= .
∂τ ∂z
Since the initial population size is n0 , then
n0
1 − αe(1−α)z
Q(z, 0) = sn0 = .
1 − e(1−α)z
Hence n0
1 − αe(1−α)(z+τ )
Q(z, τ ) = .
1 − e(1−α)(z+τ )
Finally n0 Z
1−s t
1 − αq(s, t)
G(s, t) = where q(s, t) = exp (1 − α) λ(u)du
1 − q(s, t) α−s 0
as required.
The probability of extinction is
n0
1 − αq(0, t)
G(0, t) = ,
1 − q(0, t)
where Z
t
1
q(0, t) = exp (1 − α) λ(u)du .
α 0
6.32. A continuous time process has three states E1 , E2 , and E3 . In time δt the probability of a change
from E1 to E2 is λδt, from E2 to E3 is also λδt, and from E2 to E1 is µδt. E3 can be viewed as an
absorbing state. If pi (t) is the probability that the process is in state Ei (i = 1, 2, 3) at time t, show that
p′1 (t) = −λp1 (t) + µp2 (t), p′2 (t) = λp1 (t) − (λ + µ)p2 (t), p′3 (t) = λp2 (t).
Find the probabilities p1 (t), p2 (t), p3 (t), if the process starts in E1 at t = 0.

The process survives as long as it is in states E1 or E2 . What is the survival probability, that is
P(T > t) of the process?
By the usual birth and death method
p1 (t + δt) = µδtp2 (t) + (1 − λδt)p1 (t) + O((δt)2 ),
102
p2 (t + δt) = λδtp1 (t) + (1 − λδt − µδt)p2 (t) + O((δt)2 ),
p3 (t + δt) = λδtp2 (t) + O((δt)2 ).
Let δt → 0 so that the probabilities satisfy
p′1 (t) = µp2 (t) − λp1 (t), (i)
p′2 (t) = λp1 (t) − (λ + µ)p2 (t), (ii)

p′3 (t) = λp2 (t). (iii)
Eliminate p2 (t) between (i) and (ii) so that p1 (t)
p′′1 (t) + (2λ + µ)p′1 (t) + λ2 p1 (t) = 0.
This second-order differential equation has the characteristic equation
m2 + (2λ + µ)m + λ2 = 0,
which has the solutions

m1
= α ± β,
m2
1√
where α = − 21 (2λ + µ) and β = 2
(4λ + µ). Therefore, since p1 (0) = 1, then
p1 (t) = Aem1 t + (1 − A)em2 t .
From (i),
1 ′
p2 (t) = [p1 (t) + λp1 (t)]
µ
1
= [A(m1 + λ)em1 t + (1 − A)(m2 + λ)em2 t ]
µ
Since p2 (0) = 0, then A = (m2 + λ)/(m2 − m1 ). Finally

1
p1 (t) = [−(m2 + λ)em1 t + (m1 + λ)em2 t ].
m1 − m2
The survival probability at time t is p1 (t) + p2 (t).
6.33. In a birth and death process, the birth and death rates are given respectively by λ(t)n and µ(t)n in
eqn (6.25). Find the equation for the probability generating function G(s, t). If µ(t) is the mean population
size at time t, show, by differentiating the equation for G(s, t) with respect to s, that
µ′ (t) = [λ(t) − µ(t)]µ(t),
(assume that (s − 1)∂ 2 G(s, t)/∂s2 = 0 when s = 1). Hence show that
Z t

µ(t) = n0 exp [λ(u) − µ(u)]du ,
0
where n0 is the initial population size.
The differential difference equations for the probability pn (t) are (see eqn (6.26))
p′0 (t) = µ(t)p1 (t),
p′n (t) = λ(t)(n − 1)pn−1 (t) − [λ(t)n + µ(t)n]pn (t) + µ(t)(n + 1)pn+1 (t).
Hence the probability generating function G(s, t) satisfies
∂G(s, t) ∂G(s, t)
= [λ(t)s − µ(t)](s − 1) , (i)
∂t ∂s
103
(the method parallels that in Section 6.5). Differentiating (i) with respect to s:
∂ 2 G(s, t) ∂G(s, t) ∂G(s, t) ∂ 2 G(s, t)

= λ(t)(s − 1) + [λ(t)s − µ(t)] + [λ(t)s − µ(t)](s − 1) .
∂s∂t ∂s ∂s ∂s2
Put s = 1 and remember that µ(t) = Gs (s, t). Then
µ′ (t) = [λ(t) − µ(t)]µ(t).
Hence integration of this differential equation gives

Z t

µ(t) = n0 exp [λ(u) − µ(u)]du .
0
104
Chapter 7
Queues
7.1. In a single-server queue a Poisson process for arrivals of intensity 12 λ and for service and departures
of intensity λ are assumed. For the corresponding limiting process find
(a) pn , the probability that there are n persons in the queue,
(b) the expected length of the queue,
(c) the probability that there are not more than two persons in the queue, including the person being
served in each case.
1
(a) As in Section 7.3, with ρ = 2
1
pn = (1 − ρ)ρn = .
2n+1
(b) If N is the random variable of the number n of persons in the queue (including the person being
served), then its expected value is (see Section 7.3(b))
1−ρ
E(N ) = = 1.
ρ
(c) The probability that there are not more than two persons in the queue is
1 1 1 7
p0 + p1 + p2 = + + = .
2 4 8 8
7.2. Consider a telephone exchange with a very large number of lines available. If n lines are busy the
probability that one of them will become free in small time δt is nµδt. The probability of a new call is λδt
(that is, Poisson), with the assumption that the probability of multiple calls is negligible. Show that pn (t),
the probability that n lines are busy at time t satisfies
p′0 (t) = −λp0 (t) + µp1 (t),
p′n (t) = −(λ + nµ)pn (t) + λpn−1 (t) + (n + 1)µpn+1 (t), (n ≥ 1).
In the limiting process show by induction that
n
e−λ/µ λ
pn = lim pn (t) = .
t→∞ n! µ
Identify the distribution.
If pn = limt→∞ pn (t) (assumed to exist), then the stationary process is defined by the difference
equations
−λp0 + µp1 = 0,
105
(n + 1)µpn+1 − (λ + nµ)pn + λpn−1 = 0.
Assume that n
e−λ/µ λ
pn = .
n! µ
1
pn+1 = [(λ + nµ)pn − λpn−1 ]
µ(n + 1)
" n n−1 #
1 λ + µn −λ/mu λ λ λ
= e −
µ(n + 1) n! µ (n − 1)! µ
n
e−λ/µ λ
= [λ + nµ − nµ]
µ(n + 1)! µ
n+1
e−λ/µ λ
=
(n + 1)! µ
Hence if the formula is true for pn and pn−1 , then it is true for pn+1 . It can be verified that that p1 and
p2 are correct: therefore by induction on the positive integers the formula is proved. The distribution is
Poisson with intensity λ/µ.
7.3. For a particular queue, when there are n customers in the system, the probability of an arrival in the
small time interval δt is λn δt + o(δt). The service time parameter µn is also a function of n. If pn denotes
the probability that there are n customers in the queue in the steady state queue, show by induction that
λ0 λ1 . . . λn−1
pn = p0 , (n = 1, 2, . . .)
µ1 µ2 . . . µn
and find an expression for p0 .
If λn = 1/(n + 1) and µn = µ, a constant, find the expected length of the queue.
Let pn (t) be the probability that there are n persons in the queue. Then, by the usual arguments,
p0 (t + δt) = µ1 p1 (t) + (1 − µ1 δt)p0 (t),
pn (t + δt) = λn−1 pn−1 (t)δt + µn+1 pn+1 (t)δt + (1 − λn δt − µn δt)pn (t), (n = 1, 2, . . .).
Divide through by δt, and let δt → 0:
p′0 (t) = µ1 p1 (t),
p′n (t) = µn+1 pn+1 (t) − (λn + µn )pn (t) + λn−1 pn−1 (t).
Assume that a limiting stationary process exists, such the pn = limt→∞ pn (t). Then pn satisfies
µ1 p1 − λ0 p0 = 0,
µn+1 pn+1 − (λn + µn )pn + λn−1 pn−1 = 0.

Assume that the given formula is true for pn and pn−1 . Then, using the difference equation above,

1 λ0 λ1 . . . λn−1 λ0 λ1 . . . λn−2
pn+1 = (λn + µn ) − λn−1
µn+1 µ1 µ2 . . . µn µ1 µ2 . . . µn−1
λ0 λ1 . . . λn
= p0 ,
µ1 µ2 . . . µn+1
Showing that the formula is true for pn+1 . It can be verified directly that
λ0 λ0 λ1
p1 = p0 , p2 = p0 .
µ1 µ1 µ2
Induction proves the result for all n.
106
P∞
The probabilities satisfy n=0
pn = 1. Therefore
∞
X λ0 λ1 . . . λn−1
1 = p0 + p0 ,
µ1 µ2 . . . µn
n=1
provided that the series converges. If that is the case, then

" ∞
#
X λ0 λ1 . . . λn−1
p0 = 1 1+ .
µ1 µ2 . . . µn
n=1
If λn = 1/(n + 1) and µn = µ, then

p0 1 1 1 p0
pn = · · ··· = n ,
µn 1 2 n µ n!
where " #
∞
X 1
p0 = 1 1+ = e−1/µ .
µn n!
n=1
Hence
e−1/µ
pn = .
µn n!
which is Poisson with intensity 1/µ.
The expected length of the queue is the usual result for the mean:
∞ ∞
X X 1 1 1
µ= npn = e−1/µ = e−1/µ e1/µ = .
µn (n − 1)! µ µ
n=1 n=1
7.4. In a baulked queue (see Example 7.1) not more than m ≥ 2 people are allowed to form a queue. If
there are m individuals, where m is fixed, in the queue, then any further arrivals are turned away. If the
arrivals form a Poisson process with intensity λ and the service distribution is exponential with parameter
µ, show that the expected length of the queue is
ρ − (m + 1)ρm+1 + mρm+2
,
(1 − ρ)(1 − ρm+1 )
where ρ = λ/µ. Deduce the expected length if ρ = 1. What is the expected length of the queue if m = 3 and
ρ = 1?
From Example 7.1, the probability of a baulked queue having length n is

ρn (1 − ρ)
pn = , (n = 0, 1, 2, . . . , m), (ρ 6= 1). (i)
(1 − ρm+1 )
The expected length is
m ∞ X
m
X X nρn (1 − ρ) 1−ρ
E(N ) = npn = = nρn .
(1 − ρm+1 ) 1 − ρm+1
n=1 n=1 n=1
Let
m
X
S= nρn .
n=1
Then
m
X
(1 − ρ)S = ρn − mρm+2 .
n=1
107
Further summation of the geometric series gives
ρ − (m + 1)ρm+1 + mρm+2
S= ,
(1 − ρ)2
so that the expected length of the queue is
ρ − (m + 1)ρm+1 + mρm+2
E(N ) = , (ρ 6= 1). (ii)
(1 − ρ)(1 − ρm+1 )
1
If ρ = 1, then, applying l’Hôpital’s rule in calculus to (i)

d n d 1
pn = [ρ (1 − ρ)] [1 − ρm+1 ] = .
dρ dρ ρ=1
m+1
In this case the expected length is

m m
X 1 X 1 1 1
E(N ) = npn = n= · m(m + 1) = m,
m+1 (m + 1) 2 2
n=1 n=1
using an elementary formula for the sum of the first m integers.

If ρ = 1 and m = 3, then E(N ) = 3/2.
The expected length in (ii) can be re-arranged into
ρ−m−1 − (m + 1)ρ−1 + m
E(N ) = → m,
(ρ−1 − 1)(ρ−1 − 1)
as ρ → ∞. For the baulked queue there is no restriction on ρ.
7.5. Consider the single-server queue with Poisson arrivals occurring with intensity λ, and exponential
service times with parameter µ. In the stationary process, the probability pn that there are n individuals in
the queue is given by n
λ λ
pn = 1 − , (n = 0, 1, 2, . . .).
µ µ
Find its probability generating function
∞
X
G(s) = p n sn .
n=0
If λ < µ, use this function to determine the mean and variance of the queue length.
The probability generating function G(s, t) is defined by

∞
" n+1 #
X n
λ n λ n
G(s, t) = s − s .
µ µ
n=0
Summation of the two geometric series

µ λ µ µ−λ
G(s, t) = − · = .
µ − λs µ µ − λs µ − λs
The first two derivatives of G(s, t) are
λ(µ − λ) λ2 (µ − λ)
Gs (s, t) = , Gss (s, t) = 2
(µ − λs)2 (µ − λs)3
1 In (i) both the numerator and denominator are zero if ρ = 1: the rule states, under suitable conditions, if
pn = f (ρ)/g(ρ), f (a) = g(a) = 0, and g ′ (a) 6= 0, then

f (ρ) f ′ (a)
lim = ′
ρ→a g(ρ) g (a)
108
Hence the mean and variance are given by
λ
E(N ) = Gs (1, t) = ,
µ−λ
2λ2 λ λ2 λµ
V(N ) = Gss (1, t) + Gs (1, t) − [Gs (1, t)]2 = + − =
(µ − λ)2 µ−λ (µ − λ)2 (µ − λ)2
7.6. A queue is observed to have an average length of 2.8 individuals including the person being served.
Assuming the usual exponential distributions for both service times and times between arrivals, what is the
traffic density, and the variance of the queue length?
With the usual parameters λ and µ and the random variable N of the length of the queue, then its
expected length is, with ρ = λ/µ,
ρ
E(N ) = ,
1−ρ
which is 2.8 from the data. Hence the traffic density is ρ = 2.8/3.8 ≈ 0.74.
The probability that the queue length is n in the stationary process is (see eqn (7.5)) pn = (1 − ρ)ρn .
The variance of the queue length is given by
∞
X ρ2
V(N ) = E(N 2 ) − [E(N )]2 = (1 − ρ) n2 ρn −
(1 − ρ)2
n=1
ρ(1 + ρ) ρ2
= 2
−
(1 − ρ) (1 − ρ)2
ρ
=
(1 − ρ)2
If ρ = 0.74, then V(N ) ≈ 10.9.
7.7. The differential-difference equations for a queue with parameters λ and µ are (see equation (7.1))
dp0 (t)
= µp1 (t) − λp0 (t),
dt
dpn (t)
= λpn−1 (t) + µpn+1 (t) − (λ + µ)pn (t),
dt
where pn (t) is the probability that the queue has length n at time t. Let the probability generating function
of the distribution {pn (t)} be
∞
X
n=0
Show that G(s, t) satisfies the equation
∂G(s, t)
s = (s − 1)(λs − µ)G(s, t) + µ(s − 1)p0 (t).
∂t
Unlike the birth and death processes in Chapter 6, this equation contains the unknown probability p0 (t)
which complicates its solution. Show that it can be eliminated to leave the following second-order partial
differential equation for G(s, t):
∂ 2 G(s, t) ∂G(s, t) ∂G(s, t)

s(s − 1) − (s − 1)2 (λs − µ) − − λ(s − 1)2 G(s, t) = 0.
∂t∂s ∂s ∂t
This equation can be solved by Laplace transform methods.
109
Multiply the second equation in the question by sn , sum over all n from n = 1 and add the first
equation to the sum resulting in
∞ ∞ ∞ ∞
X X X X
pn′ (t) = µp1 (t) − λp0 (t) + λ pn−1 (t)sn + µ pn+1 (t)sn − (λ − µ) pn (t)sn
n=0 n=1 n=1 n=1
µ
= µp1 (t) − λp0 (t) + λsG(s, t) + [G(s, t) − sp1 (t) − p0 (t)] − (λ + µ)[G(s, t) − p0 (s, t)]
s
1 µ
= (s − 1)(λs − µ)G(s, t) + (s − 1)p0 (t).
s s
Hence the differential equation for G(s, t) is
∂G(s, t)
s = (s − 1)[(λs − µ)G(s, t) + µp0 (t)].
∂t
Write the differential equation in the form
s ∂G(s, t)
= (λs − µ)G(s, t) + µp0 (t).
s−1 ∂t
Differentiate the equation with respect to s to eliminate the term p0 (t), so that

∂ s ∂G(s, t) ∂
= [(λs − µ)G(s, t)],
∂s s−1 ∂t ∂s
or
1 ∂G(s, t) s ∂ 2 G(s, t) ∂G(s, t)
− 2
+ = λG(s, t) + (λs − µ) .
(s − 1) ∂t s − 1 ∂s∂t ∂s
The required result follows.
7.8. A call centre has r telephones manned at any time, and the traffic density is λ/(rµ) = 0.86. Compute
how many telephones should be manned in order that the expected number of callers waiting at any time
should not exceed 4? Assume a limiting process with inter-arrival times of calls and service times for all
operators both exponential with parameters λ and µ respectively (see Section 7.4).
From (7.11) and (7.12), the expected length of the queue of callers, excluding those being served, is
EHNL
8
r
2 4 6 8 10
Figure 7.1: Expected queue length E(N ) versus number r of manned telephones.
p0 ρr+1
E(N ) = , (i)
(r − 1)!(r − ρ)2
110
where "X #
r−1
ρn ρr λ
p0 = 1 + , ρ= . (ii)
n! (r − ρ)(r − 1)! µ
n=0
Substitute for p0 from (ii) into (i) and compute E(N ) as a function of r with ρ = 0.86r. A graph of E(N )
against r is shown in Figure 7.1 for r = 1, 2, . . . , 10. From the graph the point at r = 6 is (just) below the
line E(N ) = 4. The answer is that 6 telephones should be manned.
7.9. Compare the expected lengths of the two queues M (λ)/M (µ)/1 and M (λ)/D(1/µ)/1 with ρ = λ/µ < 1.
The queues have parameters such that the mean service time for the former equals the fixed service time
in the latter. For which queue would you expect the mean queue length to be the shorter?
From Section 7.3(b), the expected length of the M (λ)/M (µ)/1 queue is, with ρ = λ/µ < 1,
ρ
E1 (N ) = .
1−ρ
Since λτ = λ/µ = ρ (τ is the fixed service time), the expected length of the M (λ)/D(1/µ)/1 is (see end of
Section 7.5)
ρ(1 − 12 ρ)
E2 (N ) = .
1−ρ
It follows that
1 2
ρ ρ ρ
E2 (N ) = − 2 ≤ = E1 (N ).
1−ρ 1−ρ 1−ρ
In this case the queue with fixed service time has the shorter expected length.
7.10. A queue is serviced by r servers, with the distribution of the inter-arrival times for the queue being
exponential with parameter λ and each server has a common exponential service time distribution with
parameter µ. If N is the random variable for the length of the queue including those being served, show
that its expected value is " #
r−1
X ρn ρr [r 2 + ρ(1 − r)]
E(N ) = p0 + ,
(n − 1)! (r − 1)!(r − ρ)2
n=1
where ρ = λ/µ < r, and

"X
r−1
#
ρn ρr
p0 = 1 + .
n! (r − ρ)(r − 1)!
n=0
(see equation (7.11)).

If r = 2, show that
4ρ
E(N ) = .
4 − ρ2
For what interval of values of ρ is the expected length of the queue less than the number of servers?
For the M (λ)/M (µ)/r queue, the probability that there n persons in the queue is

ρn p0 /n! n<r
pn = ,
ρn p0 /(r n−r r!) n≥r
where p0 is given in the question. The expected length of the queue including those being served is
∞
" r−1 ∞
# " r−1 ∞
#
X nρn n
X X nρn X ρn rr X ρ
E(N ) = npn = p0 + = p0 + n .
n! r n−r r! (n − 1)! r! r
n=1 n=1 n=r n=1 n=r
Consider the series

∞ ρ n
X
R= n .
r
n=r
111
Using the method for summing geometric series twice to obtain
r r(r2 + ρ − ρr)
ρ
R= .
r (r − ρ)2
Hence " r−1 #
X ρn ρr [r 2 + ρ(1 − r)]
E(N ) = p0 + ,
(n − 1)! (r − 1)!(r − ρ)2
n=1
as required.
If r = 2 (two servers) then

ρ2 2−ρ
p0 = 1 1+ρ+ = ,
(2 − ρ) 2+ρ
and
ρ2 (4 − ρ) 4ρ
E(N ) = p0 ρ + = .
(2 − ρ)2 4 − ρ2
The expected length of the queue is less than the number of servers if
4ρ
< 2, or ρ2 + 2ρ − 4 < 0.
4 − ρ2
√ √
The roots of ρ2 + 2ρ − 4 = 0 are ρ = −1 ± 5. The required range for ρ is 0 < ρ < 5 − 1.
7.11. For a queue with two servers, the probability pn that there are n servers in the queue, including those
being served, is given by
2−ρ
ρ n
p0 = , p1 = ρp0 , pn = 2 p0 , (n ≥ 2),
2+ρ 2
where ρ = λ/µ (see Section 7.4). If the random variable X is the number of people in the queue, find its
probability generating function. Hence find the mean length of the queue including those being served.
The probability generating function for this queue with two servers is, summing the geometric series,
∞
X ρ n 2−ρ ρ 2 s2
G(s) = p0 + p0 ρs + 2p0 sn = 1 + ρs +
2 2+ρ 2 − ρs
n=2

2−ρ 2 + ρs
= ,
2+ρ 2 − ρs
assuming that 0 ≤ ρs < 2.

For the mean we require
2−ρ 4ρ
G′ (s) = .
2+ρ (2 − ρs)2
Then the mean length of the queue
4ρ
µ = G′ (1) = ,
(4 − ρ2 )
which agrees with the result from Problem 7.10.
7.12. The queue M (λ)/D(τ )/1, which has a fixed service time of duration τ for every customer, has the
probability generating function
(1 − ρ)(1 − s)
G(s) = ,
1 − seρ(1−s)
where ρ = λτ (0 < ρ < 1) (see Section 7.5).
(a) Find the probabilities p0 , p1 , p2 .
(b) Find the expected value and variance of the length of the queue.
112
(c) Customers are allowed a service time τ which is such that the expected length of the queue is two
individuals. Find the value of the traffic density ρ.
(a) The Taylor series for G(s) starts
G(s) = (1 − ρ) + (1 − ρ)(eρ − 1)s + (1 − ρ)(eρ − ρ − 1)eρ s2 + O(s3 ).
Hence
p0 = 1 − ρ, p1 = (1 − ρ)(eρ − 1), p2 = (1 − ρ)(eρ − ρ − 1)eρ .
(b) The first two derivatives of G(s) are

′
esρ (−1 + ρ) esρ + eρ −1 + sρ − s2 ρ
G (s) = ,
(esρ − eρ s)2

eρ+sρ (−1 + ρ) esρ 2 + (2 − 4s)ρ + (−1 + s)sρ2 + eρ −2 + 2sρ − s2 ρ2 + s3 ρ2
G′′ (s) = .
(esρ − eρ s)3
It follows that the mean and variance are
ρ(2 − ρ)
E(N ) = G′ (1) = ,
2(1 − ρ)
ρ
V(N ) = G′′ (1) + G′ (1) − [G′ (1)]2 = (12 − 18ρ + 10ρ2 − ρ3 ).
12(1 − ρ)2
(c) We require G′ (1) = 2, or 2 − 2ρ = ρ − 12 ρ2 . Hence ρ2 − 6ρ + 4 = 0. The roots of this quadratic
√ √
equation are ρ = λτ = 3 ± 5: the traffic density is ρ = 3 − 5 since ρ must be less than 1.
7.13. For the baulked queue which has a maximum length of m beyond which customers are turned away,
the probabilities that there are n individuals in the queue are given by
ρn (1 − ρ) 1
pn = , (0 < ρ < 1), pn = (ρ = 1),
1 − ρm+1 m+1
for n = 0, 1, 2, . . . , m. Show that the probability generating functions are
(1 − ρ)[1 − (ρs)m+1 ]
G(s) = , (0 < ρ < 1),
(1 − ρm+1 )(1 − ρs)
and
1 − sm+1
G(s) = , (ρ = 1).
(m + 1)(1 − s)
Find the expected value of the queue length including the person being served.
The probability generating function G(s) of the baulked queue is defined by, for ρ 6= 1,
m X
m
X n 1−ρ (1 − ρ)(1 − ρm+1 sm+1 )
G(s) = pn s = ρ n sn = ,
1 − ρm+1 (1 − ρm+1 )(1 − ρs)
n=0 n=0
using the formula for the sum of the geometric series. The first derivative of G(s) is
(1 − ρ)ρ (1 − (sρ)m − m(sρ)m (1 − sρ))
G′ (s) = .
(1 − sρ)2 (1 − ρ1+m )
The expected length of the queue is
ρ (1 + m)ρ1+m
E(N ) = G′ (1) = −
1−ρ 1 − ρ1+m
If ρ = 1, then
∞ m
X 1 X n 1 − sm+1
G(s) = p n sn = s = .
m+1 (m + 1)(1 − s)
n=0 n=0
113
Its first derivative is
1 − (1 + m)sm + ms1+m
G′ (s) = .
(1 + m)(1 − s)2
The expected length of the queue is E(N ) = G′ (1) = 21 m.
7.14. In Section 7.3(ii), the expected length of the queue with parameters λ and µ, including the person
being served was shown to be ρ/(1 − ρ). What is the expected length of the queue excluding the person being
served?
The expected length of the queue excluding the person being served is given by
∞
X ∞
X
E(N ) = (n − 1)pn = (n − 1)(ρn − ρn+1 )
n=2 n=2
∞ ∞ ∞ ∞
X X X X
= nρn − ρn − nρn + 2 ρn
n=2 n=2 n=3 n=3
2ρ2 2ρ3
= 2ρ − +
1−ρ 1−ρ
ρ2
=
1−ρ
7.15. An M (λ)/M (µ)/1 queue is observed over a long period of time. Regular sampling indicates that the
mean length of the queue including the person being served is 3, whilst the mean waiting time to completion
of service by any customer arriving is 10 minutes. What is the mean service time?
From Section 7.3, the mean length of the queue is 3, and the mean time to service is 10 minutes: hence
ρ ρ
= 3, = 10.
1−ρ µ(1 − ρ)
Hence
3 ρ 3
ρ= and µ= = .
4 10(1 − ρ) 10
Hence the mean service time is 1/µ = 10/3.
7.16. A person arrives at an M (λ)/M (µ)/1 queue. If there are two people in the queue (including customers
being served) the customer goes away and does not return. If there are there are fewer than two queueing
then the customer joins the queue. Find the expected waiting time for the customer to the start of service.
This is a baulked queue with baulking after 2 customers in the queue but not being served. The
method is similar to that in Section 7.3(c). Let Ti , (i = 1, 2) be the time for customer i to reach the
service position. Let S1 and S2 be the random variables defined by
S1 = T 1 , S2 = T 1 + T 2 .
As explained in Section 7.3(c), the densities of S1 and S2 are gamma with parameters respectively µ, 1
and µ, 2, are
f1 (t) = µe−µt (t > 0), f2 (t) = µ2 te−µt , (t > 0). (i)
From Example 7.1, the probability that there are n persons in the queue is
ρn (1 − ρ)
pn = , (n = 0, 1, 2). (ii)
1 − ρ3
114
Then, using (i) and (ii)
P(S1 + S2 > T ) = P(S1 > t)p1 + P(S2 > t)p2

Z ∞ Z ∞
ρ(1 − ρ) ρ2 (1 − ρ)
= f1 (s)ds + f2 (s)ds
1 − ρ3 t
1 − ρ3 t
Z ∞ Z ∞

ρ(1 − ρ)
= µe−µs ds + ρ µ2 se−µs ds
1 − ρ3 t t
ρ(1 − ρ) −µt
= e [1 + ρµt + ρ].
1 − ρ3
The density of T is therefore

d ρ(1 − ρ) −µt ρ(1 − ρ)
g(t) = 1− e [1 + ρµt + ρ] = µ(1 + ρµt)e−µt .
dt 1 − ρ3 1 − ρ3
Finally, the expected waiting time for the customer to start service is
Z ∞ Z ∞
ρ(1 − ρ)µ ρ(1 − ρ)(1 + 2ρ) ρ(1 + 2ρ)
E(S1 + S2 ) = tg(t)dt = (t + ρµt2 )e−µt dt = =
0
1 − ρ3 0
1 − ρ3 1 + ρ + ρ2
7.17. A customer waits for service in a bank in which there are four counters with customers at each
counter but otherwise no one is queueing. If the service time distribution is exponential with parameter µ
for each counter, for how long should the queueing customer have to wait on average?
For each till, the density is f (t) = µe−µt . The corresponding density function is
Z t Z t
F (t) = f (s)ds = µe−µs ds = 1 − e−µt .
0 0
Let T be random variable of the time to complete service at any till. Then the probability that T is greater
than time t is
P(T > t) = P(T > t for till 1)P(T > t for till 2)P(T > t for till 3)P(T > t for till 4)
= e−µt e−µt e−µt e−µt = e−4µt .
Hence the density of T is g(t) = 4µe−4µt (t > 0), which is exponential with parameter 4µ. The mean time
for a till becoming available is Z ∞
1
E(T ) = 4µte−4µt dt = ,
0
4µ
after integrating by parts.
7.18. A hospital has a waiting list for two operating theatres dedicated to a particular group of operations.
Assuming that the queue is in a steady state, and that the waiting list and operating time can be viewed as
an M (λ)/M (µ)/2 queue, show that the expected value of the random variable N representing the length of
the queue is given by
ρ3 λ
E(N ) = , ρ = < 2.
4 − ρ2 µ
The waiting list is very long at 100 individuals. Why will ρ be very close to 2? Put ρ = 2 − ε where
ε > 0 is small. Show that ε ≈ 0.02. A third operating theatre is brought into use with the same operating
parameter µ. What effect will this new theatre have on the waiting list eventually?
In Section 7.4, apply eqns (7.11) and (7.12) (in the book) with r = 2 giving for the expected length of
the queue of patients as
p0 ρ3
E(N ) = ,
(2 − ρ)2
115
where
ρ2 2−ρ
p0 = 1 1+ρ+ = .
2−ρ 2+ρ
Elimination of p0 between these equations leads to
ρ3
E(N ) = , λ < 2.
4 − ρ2
For E(N ) to be large, 4 − ρ2 is small. Since 0 < ρ < 2, let ρ = 2 − ε with ε small. Then
2(1 − 21 ε) 1
E(N ) = = + O(1).
ε(1 − 14 ε) ε
as ε → 0. If E(N ) = 50, then ε ≈ 2/100 = 0.02.

If r = 3 (which will require λ/µ < 3), then
p0 ρ4
E(N ) = ,
2(3 − ρ)2
where
1 2 ρ3
p0 = 1 1+ρ+ ρ + .
2 2(3 − ρ)
Therefore
ρ4
E(N ) = .
(3 − ρ)(6 + 4ρ + ρ2 )
If ρ = 2 − ε, then the expected length is
8
E(N ) ≈ + O(ε),
9
which is a significant reduction.
7.19. Consider the M (λ)/M (µ)/r queue which has r servers such that ρ = λ/µ < r. Adapting the method
for the single-server queue (Section 7.3 (iii)), explain why the average service time for (n−r +1) customers
to be served is (n − r + 1)/(µr) if n ≥ r. What is it if n < r? If n ≥ r, show that the average value of the
waiting time random variable T until service is
∞
X n−r+1
E(T ) = pn .
rµ
n=r
What is the average waiting time if service is included?
If n ≤ r − 1, then the arriving customer has immediate service. If n ≥ r, the customer must wait until
n − r + 1 preceding customers have been served, and they take time (n − r + 1)/(rµ) since the mean service
time is 1/(rµ) for r servers. The expected value of the service time T until service is
∞
X n−r+1
E(T ) = pn ,
rµ
n=r
where (see Section 7.4)

"X
r−1
#
ρn p0 ρn ρr
pn = n−r , p0 = 1 + .
r r! n! (r − ρ)(r − 1)!
n=0
Then
∞ n−r
2
ρr p0 X ρ ρr p0 ρ ρ
E(T ) = (n − r + 1) = 1+2 +3 + ···
µrr! r µrr! r r
n=r
r
ρ p0
=
µ(r − 1)!(r − ρ)2
116
The mean time represented by random variable T ′ until service is completed is
1
E(T ′ ) = E(T ) + .
µ
7.20. Consider the queue M (λ)/M (µ)/r queue. Assuming that λ < rµ, what is the probability in the long
term that at any instant there is no one queueing excluding those being served?
The probability that no one is queueing is

∞
X
qr = 1 − pn ,
n=r
where (see Section 7.4) using the formula for the sum of a geometric series,
∞ ∞
X p0 r r X ρ n p0 ρr
pn = = ,
r! r (r − 1)!(r − ρ)
n=r n=r
and "X #
r−1
ρn ρr
p0 = 1 + .
n! (r − ρ)(r − 1)!
n=0
7.21. Access to a toll road is controlled by a plaza of r toll booths. Vehicles approaching the toll booths
choose one at random: any toll booth is equally likely to be the one chosen irrespective of the number of cars
queueing (perhaps an unrealistic situation). The payment time is assumed to be exponential with parameter
µ, and vehicles are assumed to approach as Poisson with parameter λ. Show that, viewed as a steady-state
process, the queue of vehicles at any toll booth is an M (λ/r)/M (µ)/1 queue assuming λ/(rµ) < 1. Find
the expected number of vehicles queueing at any toll booth. How many cars would you expect to be queueing
over all booths?
One booth is out of action, and vehicles distribute themselves randomly over the remaining booths.
Assuming that λ/[(r − 1)µ] < 1, how many extra vehicles can be expected to be queueing overall?
It is assumed to be an M (λ)/r/M (µ)/1) queue. For a random number Nn of cars at toll booth n
(n = 1, 2, . . . r) is
λ
E(Nn ) = .
µr − λ
For all booths, the expected number of cars is
rλ
E(N1 + N2 + · · · Nr ) = ,
µr − λ
including cars at any booth.
If one booth is out of action (say booth r) then the expected number of cars queueing is
(r − 1)λ
E(N1 + N2 + · · · Nr−1 ) = ,
µ(r − 1) − λ
Hence the expected extra length of the queue is
(r − 1)λ rλ λ2
− = .
µ(r − 1) − λ µr − λ [µ(r − 1) − λ](µr − λ)
7.22. In an M (λ)/M (µ)/1 queue, it is decided that the service parameter µ should be adjusted to make the
mean length of the busy period 10 times the slack period to allow the server some respite. What should µ
be in terms of λ?
117
From Section 7.3(d), in the stationary process, the expected length of the slack period is 1/λ and that
of the busy period 1/(µ − λ). Therefore the busy period is 10 times of the slack period if
1 10 11
= , or µ= λ.
µ−λ λ 10
7.23. In the baulked queue (see Example 7.1) not more than m ≥ 2 people (including the person being
served) are allowed to form a queue, the arrivals having a Poisson distribution with parameter λ. If there
are m individuals in the queue, then any further arrivals are turned away. It is assumed that the service
distribution is exponential with rate µ. If ρ = λ/µ 6= 1, show that the expected length of the busy periods is
given by (1 − ρm )/(µ − λ).
For the baulked queue with a limit of m customers (including the person being served) the probability
that the queue is of length n is given by
ρn (1 − ρ)
pn = , (n = 0, 1, 2, . . . , m).
1 − ρn+1
In the notation of Section 7.3(iv) for slack and busy periods,
" j
#
1X 1
lim si = ,
j→∞ j λ
i=1
and " # " #

j j
1X 1 − p0 1X 1 − p0 1 − ρm
lim bi = lim si = = .
j→∞ j p0 j→∞ j λp0 µ−λ
i=1 i=1
7.24. The M (λ)/D(τ )/1 queue has a fixed service time τ , and from Section 7.5, its probability generating
function is
(1 − λτ )(1 − s)
G(s) = .
1 − seλτ (1−s)
Show that the expected length of its busy periods is τ /(1 − λτ ).
The average length of the slack periods is, in the notation of Section 7.3(iv),
" j
#
1X 1
lim si = ,
j→∞ j λ
i=1
since it depends on the arrival distribution. Also the average length of the busy periods is
" j
# j
" #
1X 1 − p0 1X 1 − p0
lim bi = lim si = .
j→∞ j p0 j→∞ j p0 λ
i=1 i=1
From the given generating function p0 = G(0) = 1 − λτ , so that the average length of busy periods is
" j
#
1X τ
lim bi = .
j→∞ j 1 − λτ
i=1
7.25. A certain process has the (r+1) states En , (n = 0, 1, 2, . . . r). The transition rates between state n and
state n + 1 is λn = (r − n)λ, (n = 0, 1, 2, . . . , r − 1), and between n and n − 1 is µn = nµ, (n = 1, 2, . . . , r).
These are the only possible transitions at each step in the process. [This could be interpreted as a ‘capped’
birth and death process in which the population size cannot exceed r.]
118
Find the differential-difference equation for the probabilities pn (t), (n = 0, 1, 2, . . . , r), that the process is
in state n at time t. Consider the corresponding steady-state process in which dpn /dt → 0 and pn (t) → pn
as t → ∞. Show that n
λ r µr
pn = , (n = 0, 1, 2, . . . , r).
µ n (λ + µ)r
In (6.25), let
λn = (r − n)λ, (n = 0, 1, 2, . . . , r − 1), µn = nµ, (n = 1, 2, . . . , r),
so that the finite system of differential-difference equations are
p′0 (t) = λrp0 (t) + µp1 (t),
p′n (t) = (r − n + 1)λpn−1 (t) − [(r − n) + nµ]pn (t) + (n + 1)µpn+1 (t), (n = 1, 2, . . . , r − 1),
p′r (t) = λpr−1 (t) − rµpr (t).
The time-independent stationary process satisfies the difference equations
λrp0 + µp1 = 0,
(r − n + 1)λpn−1 − [(r − n) + nµ]pn + (n + 1)µpn+1 = 0, (n = 1, 2, . . . , r − 1),

λpr−1 − rµpr = 0.
Let un = λ(r − n)pn − (n + 1)µpn+1 . The the difference equations become
u0 = 0, un−1 − un = 0, (n = 1, 2, . . . , r − 1), ur−1 = 0.
We conclude that un = 0 for (n = 0, 1, 2, . . . , r − 1). Hence

λ r−n
pn+1 = pn .
µ n+1
Repeated application of this formula gives
n n
λ r(r + 1) · · · (r − n + 1) λ r
pn = p0 = p0 , (n = 0, 1, 2, . . . , r).
µ n! µ n
The probability p0 is defined by

r r n r
X X λ r λ
1= pn = p0 = p0 1+ .
µ n µ
n=0 n=0
Hence
µr
p0 = ,
(λ + µ)r
so that, finally n
λ r µr
pn = .
µ n (λ + µ)r
7.26. In a baulked single-server queue not more than 3 persons are allowed.
t Write down the full time-
dependent equations for p0 ,p1 ,p2 ,p3 . Show that p = p0 p1 p2 p3 satisfies
 
−λ µ 0 0
dp  λ −(λ + µ) µ 0 
= Ap where A= .
dt 0 λ −(λ + µ) µ 
0 0 λ −µ
Whilst the general problem can be solved, to make the algebra manageable let µ = 2λ. Find the eigenvalues
and eigenvectors of A, and write down the general solution for p. If the queue starts with no customers at
time t = 0, find the solution for p.
119
t
For the baulked queue the components of the probability vector p(t) = p0 (t) p1 (t) p2 (t) p3 (t)
satisfy
p′0 (t) = −λp0 (t) + µp1 (t),
p′1 (t) = λp0 (t) − (λ + µ)p1 (t) + µp2 (t),
p′2 (t) = λp1 (t) − (λ + µ)p2 (t) + µp2 (3),
p′3 (t) = λp2 (t) − µp3 (t),
or
dp
= Ap.
dt
If µ = 2λ, then  
−λ 2λ 0 0
 λ −3λ 2λ 0 
A= .
0 λ −3λ 2λ 
0 0 λ −2λ
which has the eigenvalues
ν1 = 0, ν2 = −5λ ν3 = −3λ ν4 = −λ.
The corresponding eigenvectors are
       
8 −2 2 −2
 4   4   −2   0 
r1 =  , r2 =  , r3 =  , r4 =  .
2  −3  −1  1 
1 1 1 1
Hence the solution of the differential equation is
p(t) = r1 A + r2 Be−5λt + r3 Ce−3λt + r4 De−λt ,
where A, B, C, D are constants. The initial condition is

t
p(0) = 1 0 0 0 .
The constants A, B, C, D satisfy the equations
8A − 2B + 2C − 2D = 1,
4A + 4B − 2C = 0,
2A − 3B − C + D = 0,
A + B + C + D = 0.
The solutions are
1 1 1 1
A= , B=− , C= , D=− .
15 40 12 6
The long-term limiting behaviour is
 
8
 4  1
p(t) →  as t → ∞.
2  15
1
120
Chapter 8
Reliability and renewal
8.1. The lifetime of a component has a uniform density function given by

1/(t1 − t0 ) 0 < t0 < t < t1
f (t) =
0 otherwise
For all t > 0, obtain the reliability function R(t) and the failure rate function r(t) for the component.
Obtain the expected life of the component.
For the given density, the distribution function is

(
0 0 ≤ t ≤ t0
F (t) = (t − t0 )/(t1 − t0 ) t0 < t < t1
1 t ≥ t1
Therefore the reliability function R(t) is

(
1 0 ≤ t ≤ t0
R(t) = 1 − F (t) = (t1 − t)/(t1 − t0 ) t0 < t < t1 ,
0 t ≥ t1
and the failure rate function r(t) is

(
0 0 ≤ t ≤ t0
f (t)
r(t) = = 1/(t1 − t) t0 < t < t1 .
R(t)
does not exist t ≥ t1
Failure of the component will not occur for 0 ≤ t ≤ t0 since R(t) = 1 in this interval. The component will
not survive beyond t = t1 .
If T is a random variable of the lifetime of the component, then the expected lifetime is given by
Z ∞ Z t1
tdt t2 − t20 1
E(T ) = tf (t)dt = = 1 = (t0 + t1 ).
0 t0
t1 − t0 2(t1 − t0 ) 2
8.2. Find the reliability function R(t) and the failure rate function r(t) for the gamma density
f (t) = λ2 te−λt , t > 0.
How does r(t) behave for large t? Find the mean and variance of the time to failure.
For the given gamma density,

Z t
2
F (t) = λ se−λs ds = 1 − e−λt (1 + λt).
0
121
Hence the reliability function is
R(t) = 1 − F (t) = (1 + λt)e−λt .
The failure rate function is
f (t) λ2 t
r(t) = = .
R(t) 1 + λt
For fixed λ and large t,
h i−1
1
r(t) = λ 1 + = λ + O(t−1 ),
λt
as t → ∞.
If T is a random variable of the time to failure, then its expected value is
Z ∞ Z ∞
2
E(T ) = sf (s)ds = λ2 s2 e−λs ds = .
0 0
λ
Also the variance

Z ∞ 2 Z ∞
2 4 6 4 2
V(T ) = s2 f (s)ds − = λ2 s3 e−λs ds − = 2 − 2 = 2.
0
λ 0
λ2 λ λ λ
These are the mean and variance of a gamma distribution with parameters λ and 2.
8.3. A failure rate function is given by

t
r(t) = , t ≥ 0,
1 + t2
The rate of failures peaks at t = 1 and then declines towards zero as t → ∞: failure becomes less likely
with time (see Figure 8.1 ). Find the reliability function, and the corresponding probability density.
Figure 8.1: Failure rate distribution r(t) with a = 1 and c = 1
In terms of r(t) (see eqn (8.5), the reliability function is given by

Z t
Z t

sds 1
R(t) = exp − r(s)ds = exp − = exp[− 12 ln(1 + t2 )] = √ ,
0 0
1 + s2 (1 + t2 )
for t ≥ 0. Hence the distribution function

1
F (t) = 1 − R(t) = 1 − √ , (t ≥ 0).
(1 + t2 )
Finally, the density is given by
t
f (t) = F ′ (t) = 3 .
(1 + t2 ) 2
122
8.4. A piece of office equipment has a piecewise failure rate function given by

2λ1 t, 0 < t ≤ t0 ,
r(t) = λ1 , λ2 > 0.
2(λ1 − λ2 )t0 + 2λ2 t, t > t0
Find its reliability function.
The reliability function is given by

Z t
R(t) = exp − r(s)ds ,
0
where, for 0 < t ≤ t0 , Z Z

t t
r(s)ds = 2λ1 sds = λ1 t2 ,
0 0
and, for t > t0 ,
Z t Z t0 Z t
r(s)ds = r(s)ds + r(s)ds
0 0 t0
Z t
= λ1 t20 + [2(λ1 − λ2 )t0 + 2λ2 s]ds
t0
= λ1 t20 + 2(λ1 − λ2 )t0 (t − t0 )] + λ2 (t2 − t20 )

= t0 (λ1 − λ2 )(2t − t0 ) + λ2 t2
Hence the reliability function is

2
e−λ1 t 0 < t ≤ t0
R(t) = 2
e−[t0 (λ1 −λ2 )(2t−t0 )+λ2 t ] t > t0 .
8.5. A laser printer is observed to have a failure rate function r(t) = 2λt (t > 0) per hour whilst in use,
where λ = 0.00021(hours) −2 : r(t) is a measure of the probability of the printer failing in any hour given
that it was operational at the beginning of the hour. What is the probability that the printer is working
after 40 hours of use? Find the probability density function for the time to failure. What is the expected
time before the printer will need maintenance?
Since r(t) = 2λt, the reliability function is

Z t
2
R(t) = exp −2λ sds = e−λt .
0
−0.00021×40×40
Hence R(40) = e = 0.715. The probability of that the printer is working after 40 hours is
2
0.715. The probability of failure is F (t) = 1 − R(t), so that F (t) = 1 − e−λt .
By (8.8), the expectation of T , the time to failure, is
Z ∞ Z ∞ q
2 1 π
E(T ) = R(t)dt = e−λt dt = = 61.2 hours,
0 0
2 λ
(see the Appendix for the value of the integral).
8.6. The time to failure is assumed to be gamma with parameters α and n, with
α(αt)n−1 e−αt
f (t) = , t > 0.
(n − 1)!
Show that the reliability function is given by
n−1 r r
X α t
R(t) = e−αt .
r!
r=0
123
Find the failure rate function and show that limt→∞ r(t) = α. What is the expected time to failure?
The gamma distribution function is

Z t Z t
α(αs)n−1 e−αs αn
F (t, α, n) = ds = sn−1 e−αs ds
0
(n − 1)! (n − 1)! 0
αn−1 n−1 −αt αn−1
= − t e + F (t, α, n − 1)
(n − 1)! (n − 2)!
Z t
αn−1 tn−1 αn−2 tn−2 αt
= e−αt − − − ··· − + e−αs ds
(n − 1)! (n − 2)! 1! 0
n−1 r r
−αt
X α t
= 1−e .
r!
r=0
after repeated integration by parts. The reliability function is therefore

n−1 r r
X α t
R(t) = 1 − F (t, α, n) = e−αt .
r!
r=0
The failure rate function r(t) is defined by

" n−1 r r
#
f (t) α(αt)n−1 e−αt −αt
X α t
r(t) = = e
R(t) (n − 1)! r!
r=0
" n−1 r r
X
#
α t
= αn tn−1 (n − 1)!
r!
r=0
For the limit, express r(t) in the form

" n−1 r r−n+1
#
n
X α t n αn−1
r(t) = α (n − 1)! →α (n − 1)! =α
r! (n − 1)!
r=0
as t → ∞.
The expected time to failure is
Z ∞
αn tn e−αt dt αn n! n
E(T ) = = · = .
0
(n − 1)! (n − 1)! αn+1 α
8.7. A electrical generator has an exponentially distributed failure time with parameter λf and the subse-
quent repair time is exponentially distributed with parameter λr . The generator is started up at time t = 0.
What is the mean time for generator to fail, and the mean time from t = 0 for it to be operational again?
As in Section 8.4, the mean time to failure is 1/λf . The mean repair time is 1/λr , so that the mean
time to the restart is
1 1
+ .
λf λr
8.8. A hospital takes a grid supply of electricity which has a constant failure rate λ. This supply is backed
up by a stand-by generator which has a gamma distributed failure time with parameters (2, µ). Find the
reliability function R(t) for the whole electricity supply. Assuming that time is measured in hours, what
should the relation between the parameters λ and µ be in order that R(1000)=0.999?
For the grid supply, the reliability function is Rg (t) = e−λt . For the stand-by supply, the reliability
function is (see Problem 8.6)
1
X µr tr
Rs (t) = e−µt = e−µt (1 + µt).
r!
r=0
124
The reliability function for the system is
R(t) = 1 − [1 − Rg (t)][1 − Rs (t)] = 1 − [1 − e−λt ][1 − (1 + µt)e−µt ]

= e−λt − (1 + µt)e−(λ+µ)t + (1 + µt)e−µt .
Let T = 1000 hours. Then solving the equation above for λ at time t = T , we have

1 R(T ) − (1 + µT )e−µT
λ = − ln ,
T 1 − (1 + µT )e−µT
where R(T ) = R(1000) = 0.999.
8.9. The components in a renewal process with instant renewal are identical with constant failure rate
λ = (1/50)(hours)−1 . If the system has one spare component which can take over when the first fails, find
the probability that the system is operational for at least 24 hours. How many spares should be carried to
ensure that continuous operation for 24 hours occurs with probability 0.98?
Let T1 and T2 be the times to failure of the components. Let S2 be the time to failure of the spares.
If τ = 24 hours is the operational time to be considered, then, as in Example 8.5,
P(S2 < τ ) = 1 − (1 + λτ )e−λτ.
Hence
24 −24/50
P(S2 < 24) = 1 − 1 + e = 1 − 0.916.
50
The required probability is 0.916.
This is the reverse problem: given the probability, we have to compute n. If Sn is the time to failure,
then
Z τ Z τ
λn
P(SN < τ ) = Fn (τ ) = fn (s)ds = sn−1 e−λs ds
0
(n − 1)! 0

−λτ (λτ )2 (λτ )n−1
= 1−e 1 + λτ + + ··· +
2! (n − 1)!
The smallest value of n is required which makes 1 − Fn (24) > 0.98. Conputation gives
1 − F2 (24) = 0.916, F3 (24) = 0.987 > 0.98.
Three components are required.
8.10. A device contains two components c1 and c2 with independent failure times T1 and T2 from time
t = 0. If the densities of the times to failure are f1 and f2 with probability distribution functions F1 and
F2 , show that the probability that c1 fails before c2 is given by
Z ∞ Z y Z ∞
P {T1 < T2 } = f1 (x)f2 (y)dxdy, = F1 (y)f2 (y)dy.
y=0 x=0 y=0
Find the probability P {T1 < T2 } in the cases:

(a) both failure times are exponentially distributed with parameters λ1 and λ2 ;
(b) both failure times have gamma distributions with parameters (2, λ1 ) and (2, λ2 ).
The probability that c1 fails before c2 is

Z Z
P(T1 < T2 ) = f1 (x)f2 (x)dx,
A
where the region A is shown in Figure 8.2. As a repeated integral the double integral can be expressed as
Z ∞ Z y Z ∞ Z ∞
P(T1 < T2 ) = f1 (x)f2 (y)dxdy = [F1 (y) − F1 (0)]f2 (y)dy = F1 (y)f2 (y)dy,
0 0 0 0
125
y
x
O
Figure 8.2: Region A in Problem 8.10.
since F1 (0) = 0.
(a) For exponentially distributed failure times
f2 (y) = λ2 e−λ2 y , F1 (y) = 1 − e−λ1 y .
Therefore
Z ∞
P(T1 < T2 ) = (1 − e−λ1 y )λ2 e−λ2 y dy
0
h i∞
1 −λ2 y 1
= λ2 − e + e−(λ1 +λ2 )y
λ2 λ1 + λ2 0
h i
1 1 λ1
= λ2 − =
λ2 λ1 + λ2 λ1 + λ2
(b) For gamma distributions with parameters (2, λ1 ) and (2, λ2 ),
f2 (y) = λ22 ye−λ2 y , F1 (y) = 1 − (1 + λ1 y)e−λ1 y .
Hence Z ∞
λ21 (λ1 + 3λ2 )
P(T1 < T2 ) = (1 − e−λ1 y − λ1 ye−λ1 y )(λ22 y)e−λ2 y dy = .
0
(λ1 + λ2 )3
8.11. Let T be the failure time of a component. Suppose that the distribution function of T is
F (t) = P(T ≤ t), t ≥ 0,
with density
f (t) = α1 e−λ1 t + α2 e−λ2 t , α1 , α2 > 0, λ1 , λ2 > 0,
where the parameters satisfy
α1 α2
+ = 1.
λ1 λ2
Find the reliability function R(t) and the failure rate function r(t) for this ‘double’ exponential distribution.
How does r(t) behave as t → ∞?
The probability distribution

Z t Z t
F (t) = f (s)ds = (α1 e−λ1 s + α2 e−λ2 s )ds
0 0
h α α2 −λ2 s t
i
1
= − ee−λ1 s −
λ1 λ2 0
α1 −λ1 t α2 −λ2 t α1 α2
= − e − e + +
λ1 λ2 λ1 λ2
α1 −λ1 t α2 −λ2 t
= 1− e − e
λ1 λ2
126
The reliability function is therefore
α1 −λ1 t α2 −λ2 t
R(t) = 1 − F (t) = e + e .
λ1 λ2
The failure rate function is

f (t) α1 −λ1 t α2 −λ2 t
r(t) = = α1 e−λ1 t + α2 e−λ2 t e + e .
R(t) λ1 λ2
As t → ∞,
(
λ1 if λ2 > λ1
r(t) → λ2 if λ2 < λ1
λ if λ1 = λ2 = λ
8.12. The lifetimes of components in a renewal process with instant renewal are identically distributed with
constant failure rate λ. Find the probability that at least three components have been replaced by time t.
In the notation of Section 8.7

Z t
P(S3 < t) = F3 (t) = F2 (t − y)f (y)dy,
0
where
F2 (t) = 1 − (1 + λt)e−λt , f (t) = λe−λt .
Then
Z t
P(S3 ≤ t) = [1 − (1 + λ(t − y)e−λ(t−y)]λe−λy dy
0
= 1 − (1 + tλ + 21 t2 λ2 )e−λt .
8.13. The lifetimes of components in a renewal process with instant renewal are identically distributed with
a failure rate which has a uniform distribution with density

1/k 0<t<k
f (t) =
0 elsewhere
Find the probability that at least two components have been replaced at time t.
For the uniform density

(
0 t<0
F1 (t) = t/k 0<t<k
1 t>k
As in Example 8.5, Z t
F2 (t) = F1 (t − y)F (y)dy.
0
For 0 < t < k. Z Z
t t
1 1 t2
F2 (t) = F1 (t − y)dy = (t − y)dy = .
k 0
k2 0
2k2
For t > k
Z k Z k
1 1
F2 (t) = F2 (t − y)dy = 2 (t − y)dy
k 0
k t−k
1
= [− 1 (t − k)2 + 12 k2 ]
k2 2
t
= (2k − t)
2k2
127
To summarize (
t2 /(2k2 ) 0<t<k
F2 (t) = 2
,
t(2k − t)/(2k ) t>k
which is the probability that at least two components have failed.
8.14. The lifetimes of components in a renewal process with instant renewal are independent and identically
distributed each with reliability function
1 −λt
R(t) = (e + e−2λt ), t ≥ 0, λ > 0.
2
Find the probability that at least two components have been replaced by time t.
Given the reliability function, it follows that the distribution function and its density are given by
F (t) = 1 − R(t) = 1 − 12 e−λt − 12 e−2λt ,
f (t) = F ′ (t) = 12 [λe−λt + 2λe−2λt ].

In the notation of Example 8.5,
Z t
P(S2 < t) = F2 (t) = F (t − s)f (s)ds
0
Z t
1
= [1 − 12 e−λ(t−s) − 21 e−2λ(t−s) ][λe−λs + 2λe−2λt ]ds
2 0
1
= [4 + (1 − 2λt)e−2λt − (5 + λt)e−λt ],
4
which is the probability that at least two components have failed by time t.
8.15. The random variable T is the time to failure from t = 0 of a system. The distribution function for
T is F (t), (t > 0). Suppose that the system is still functioning at time t = t0 . Let Tt0 be the conditional
time to failure from this time, and let Ft0 (t) be its distribution function. Show that
F (t + t0 ) − F (t0 )
Ft0 (t) = , (t ≥ 0, t0 ≥ 0),
1 − F (t0 )
and that the mean of Tt0 is Z ∞
1
E(Tt0 ) = [1 − F (u)]du.
1 − F (t0 ) t0
The distribution function for the conditional time to failure is

Ft0 (t) = P(Tt0 ≤ t) = P(T − t0 ≤ t|T > t0 )
P(T − t0 ≤ t ∩ T > t0 )
= (by eqn (1.2))
P(T > t0 )
P(t0 < T ≤ t + t0 )
=
∂(T > t0 )
F (t + t0 ) − F (t0 )
= ,
1 − F (t0 )
as required.
For the mean
Z ∞

F (t + t0 ) − F (t0 )
E(Tt0 ) = 1− dt
0
1 − F (t0 )
Z ∞
1
= [1 − F (t + t0 )]dt
1 − F (t0 )
Z0 ∞
1
= [1 − F (u)]du, (where u = t + t0 ).
1 − F (t0 ) t0
128
8.16. Suppose that the time to failure, T , of a system is uniformly distributed over (0, t1 ) given by

t/t1 , 0 ≤ t ≤ t1
U (t) = .
1 t > t1
Using the result from Problem 8.15, find the conditional probability function assuming that the system is
still working at time t = t0 .
There are two cases to consider: t0 ≤ t1 and t0 > t1 .

• t0 ≤ t1 . In the formula in Problem 8.15

(t + t0 )/t1 0 ≤ t + t0 ≤ t1 ,
U (t + t0 ) = .
1 t + t0 > t1
Hence 
 [(t + t0 )/t1 ] − [t0 /t1 ] t
 = 0 ≤ t ≤ t1 − t0
Ut0 (t) = 1 − [t0 /t1 ] t1 − t0 .
 1 − [t0 /t1 ]
 =1 t > t1 − t0
1 − [t0 /t1 ]
• t0 > t1 . Ut0 = P(T − t0 ≤ t|T > t0 ) = 1.
8.17. In the bridge system represented by Figure 8.1(d) suppose that all components have the same reliability
function Rc (t). Show that the reliability function R(t) is given by
R(t) = 2Rc (t)2 + 2Rc (t)3 − 5Rc (t)4 + 2Rc (t)5 .
Suppose that the bridge c3 is removed, What is the reliability function Rx (t) for this system? Show that
R(t) > Rx (t).
What does this inequality imply?
The result follows from (8.16) with R3 (t) = Rc (t).

With the bridge absent the reliability function is
Rx (t) = 1 − [1 − Rc (t)2 ][1 − Rc (t)2 ] = 2Rc (t)2 − Rc (t)4 ,
.
Finally
R(t) = 2Rc (t) − Rc (t)4 + 2Rc (t)3 − 4Rc (t)4 + 2Rc (t)5
= Rx (t) + 2Rc (t)3 [1 − Rc (t)]2
≥ Rx t
Perhaps not surprisingly, the bridge improves reliability
129
Chapter 9
Branching and other random

processes
9.1. In a branching process the probability that any individual has j descendants is given by
1
p0 = 0, pj = , (j ≥ 1).
2j
Show that the probability generating function of the first generation is
s
G(s) = .
2−s
Find the further generating functions G2 (s), G3 (s) and G4 (s). Show by induction that
s
Gn (s) = .
2n − (2n − 1)s
Find pn,j , the probability that the population size of the n-th generation is j given that the process starts
with one individual. What is the mean population size of the n-th generation?
The generating function is given by

∞ ∞ 2
X X s j s s s
G(s) = p j sj = = + +··· = ,
2 2 2 2−s
j=0 j=1
using the geometric series formula for the sum.

For the second generation G2 (s) = G(G(s)), so that
s/(2 − s) s s
G2 (s) = = = .
2 − [2/(2 − s)] 2(2 − s) − s 4 − 3s
Repeating this procedure,

s s
G3 (s) = G(G(G(s))) = G2 (G(s)) = = ,
4(2 − s) − 3s 8 − 7s
s s
G4 (s) = G3 (G(s)) = = .
8(2 − s) − 7s 16 − 15s
Consider the formula
s
Gn (s) = .
2n − (2n − 1)s
Then
s s
Gn+1 (s) = G(Gn (s)) = = n+1 .
2n (2 − s) − (2n − 1)s 2 − (2n+1 − 1)s
130
Hence if the formula is correct for Gn (s) then it is true for Gn+1 (s). The result has been verified for G2 (s)
and G3 (s) so it is true for all n by induction on the integers.
Using the binomial expansion
−1 ∞ n
(2n − 1) 2 − 1 j−1
X
s s
Gn (s) = n n
= n 1− s = sj .
2 − (2 − 1)s 2 2n 2n
j=1
Hence the probability that the population size of the n-th generation is j is given by the coefficient of sj
in this series, namely
2 − 1 j−1
n
pn,j = .
2n
Since G(s) = s/(2−s), then G′ (s) = 2/(2−s)2 , so that the mean of the first generation is µ = G′ (1) = 2.
Using result (9.7) in the text, the mean size of the n-th generation is
µn = G′n (1) = µn = 2n .
9.2. Suppose in a branching process that any individual has a probability given by the modified geometric
distribution
pj = (1 − p)pj , (j = 0, 1, 2, . . .),
of producing j descendants in the next generation, where p (0 < p < 1) is a constant. Find the probability
generating function of the second and third generations. What is the mean size of any generation?
The probability generating function is

∞ ∞
X X 1−p
G(s) = p j sj = (1 − p)pj sj = ,
1 − ps
j=0 j=0
using the formula for the sum of the geometric series.

In the second generation
1−p 1−p
G2 (s) = G(G(s)) = =
1 − pG(s) 1 − p[(1 − p)/(1 − ps)]
(1 − p)(1 − ps)
= ,
(1 − p + p2 ) − ps
and for the third generation
(1 − p)[1 − {p(1 − p)/(1 − ps)}]
G3 (s) = G2 (G(s)) =
(1 − p + p2 ) − {p(1 − p)/(1 − ps)}
(1 − p)(1 − p + p2 − ps)
=
(1 − 2p + 2p2 ) − p(1 − p + p2 )s
The mean size of the first generation is

(1 − p)p p
µ = G′ (1) = = .
(1 − ps)2 s=1 1−p
From (9.7) in the book, it follows that µ2 = µ2 , µ3 = µ3 , and, in general, that µn = µn for the n-th
generation.
9.3. A branching process has the probability generating function
G(s) = a + bs + (1 − a − b)s2
for the descendants of any individual, where a and b satisfy the inequalities
0 < a < 1, b > 0, a + b < 1.
131
Given that the process starts with one individual, discuss the nature of the descendant generations. What
is the maximum possible size of the n-th generation? Show that extinction in the population is certain if
2a + b ≥ 1.
Each descendant produces 0,1,2 individuals with probabilities a, b, 1 − a − b respectively. If Xn

represents the population size in the n-th generation, then the possible values of X1 , X2 , . . . are
{X1 } {0, 1, 2}
{X2 } {0, 1, 2, 3, 4}
{X3 } {0, 1, 2, 3, 4, 5, 6, 7, 8}
··· ··· ···
{Xn } {0, 1, 2, . . . , 2n }
The maximum possible population of the n-th generation is 2n .

The probability of extinction is the smallest solution of G(g) = g, that is,
a + bg + (1 − a − b)g 2 = g, or (g − 1)[(1 − a − b)g − a] = 0.
The equation always has the solution g = 1. The other possible solution is g = a/(1 − a − b). Extinction
is certain if a ≥ 1 − a − b, that is if 2a + b ≥ 1. The region in the a, b where extinction is certain is shown
in Figure 9.1. If a < 1 − a − b, then extinction occurs with probability a/(1 − a − b).
b
extinction certain
a
O 0.5 1
Figure 9.1: Extinction probability region in the a, b plane for Problem 9.3.
9.4. A branching process starts with one individual. Subsequently any individual has a probability (Poisson
with intensity λ)
λj e−λ
pj = , (j = 0, 1, 2, . . .)
j!
of producing j descendants. Find the probability generating function of this distribution. Obtain the mean
and variance of the size of the n-th generation. Show that the probability of ultimate extinction is certain
if λ ≤ 1.

∞ ∞
X X λj e−λ
G(s) = p j sj = sj = eλs−λ .
j!
j=0 j=0
As expected for this distribution, the mean and variance of the population of the first generation are
µ = G′ (1) = λeλs−λ |s=1 = λ,
σ 2 = G′′ (1) + µ − µ2 = λ2 + λ − λ2 = λ.
By Section 9.3, the mean and variance of the population of the n-th generation are
µn = µn = λn ,
132
µn (σ 2 − µ + µ2 )(µn − 1)
σ 2n = (λ 6= 1)
µ(µ − 1)
λn+1 (λn − 1)
= .
λ−1
If λ = 1, then µn = 1 and σ n = n.
9.5. A branching process starts with one individual. Any individual has a probability
λ2j sech λ
pj = , (j = 0, 1, 2, . . .)
(2j)!
of producing j descendants. Find the probability generating function of this distribution. Obtain the mean
size of the n-th generation. Show that ultimate extinction is certain if λ is less than the computed value
2.065.
The probability generating function of this distribution is given by

∞
X λ2j √
G(s, λ) = sechλ sj = sechλ cosh(λ s), (s ≥ 0).
(2j)!
j=0
Its derivative is
λ √
Gs (s, λ) = √ sechλ sinh(λ s).
2 s
Hence the mean size of the population of the first generation is
1
λ tanh λ,
µ = Gs (1, λ) =
2
which implies that the mean population of the n-th generation is
λn
µn = tanhn λ.
2n
λ
6
(1, 2.065)
2
O 0.5 1 1.5
Figure 9.2: Graph of g = G(g) for Problem 9.5.
Ultimate extinction occurs with probability g where g is the smallest solution of

√
g = G(g, λ) = sechλ cosh(λ g).
This equation aways has the solution g = 1, which is the only solution if λ < 2.065, approximately. This
is a numerically computed value. The graph of the equation above is shown in Figure 9.2.
9.6. A branching process starts with two individuals. Either individual and any of their descendants has
probability pj , (j = 0, 1, 2, . . .) of producing j descendants independently of any other. Explain why the
probabilities of 0, 1, 2, . . . descendants in the first generation are
n
X
p20 , p0 p1 + p1 p0 , p0 p2 + p1 p1 + p2 p0 , ... pi pn−i , . . . ,
i=0
133
respectively. Hence show that the probability generating function of the first generation is G(s)2 , where
∞
X
G(s) = p j sj .
j=0
The second generation from each original individual has generating function G2 (s) = G(G(s)) (see Sec-
tion 9.2). Explain why the probability generating function of the second generation is G2 (s)2 , and of the
n-th generation is Gn (s)2 .
If the branching process starts with r individuals, what would you think is the formula for the probability
generating function of the n-th generation?
For each individual, the probability generating function is

∞
X
G(s) = p j sj ,
j=0
and each produces descendants with populations 0, 1, 2, . . . with with probabilities p0 , p1 , p2 , . . .. The
combined probabilities that populations of the generations are
p20 , p0 p1 + p1 p0 , p0 p2 + p21 + p2 p0 , ....
These expressions are the coefficients of the powers of s in

∞ ∞ ∞ k
X X X X
G(s)2 = p j sj p k sk = pj pk−j sk
j=0 k=0 k=0 j=0
(this is known as the Cauchy product of the power series). Hence the probability that the population
of the first generation is of size k is
∞ X
X k
pj pk−j .
k=0 j=0
Repeating the argument, each original individual generates descendants whose probabilities are the co-
efficients of G2 (s) = G(G(s)). Hence the probabilities of populations 0, 1, 2, . . . descandants are coefficients
of G22 . This process is repeated for succeeding generations which have the generating functions Gn (s)2 .
9.7. A branching process starts with two individuals as in the previous problem. The probabilities are
1
pj = , (j = 0, 1, 2, . . .).
2j+1
Using the results from Example 9.1, find Hn (s), the probability generating function of the n-th generation.
Find also
(a) the probability that the size of the size of the n-th generation is m ≥ 2;
(b) the probability of extinction by the n-th generation;
(c) the probability of ultimate extinction.
For either individual the probability generating function is

∞
X sj 1
G(s) = = .
2j+1 2−s
j=0
Then
2−s
G2 (s) = G(G(s)) = ,
3 − 2s
and, in general,
n − (n − 1)s
Gn (s) = .
(n + 1) − ns
134
According to Problem 9.6, the generating function for the combined descendants is
2
2 n − (n − 1)s
Hn (s) = Gn (s) =
(n + 1) − ns

(n − 1)2 s2
−2
n2 2(n − 1)s ns
= 1 − + 1−
(n + 1)2 n n2 n+1
∞ r
n2 2(n − 1)s (n − 1)2 s2 X n
= 2
1 − + 2
(r + 1) sr
(n + 1) n n n+1
r=0
∞
n2
2n X [(r − 1)nr−2 + 2nr ]
= + s+ sr
(n + 1)2 (n + 1)3 (n + 1)r+2
r=2
after some algebra: series expansion by computer is helpful to confirm the formula.
(a) From the series above, the probability pn,m that the population of the n generation is m is the
coefficient of sm in the series, namely
[(m − 1)nm−2 + 2nm ]

pn,m = , (m ≥ 2)
(n + 1)m+2
(b) From the series above, the probability of extinction by the n-th generation is
n2
pn,0 = .
(n + 1)2
(c) The probability of ultimate extinction is

n2
lim pn,0 = lim = 1,
n→∞ n→∞ (n + 1)2
which means that it is certain.
9.8. A branching process starts with r individuals, and each individual produces descendants with probability
distribution {pj }, (j = 0, 1, 2, . . .), which has the probability generating function G(s). Given that the
probability of the n-th generation is [Gn (s)]r , where Gn (s) = G(G(. . . (G(s)) . . .)), find the mean population
size of the n-th generation in terms of µ = G′ (1).
Let Q(s) = [Gn (s)]r . Its derivative is
Q′ (s) = rG′n (s)[Gn (s)]r−1 ,
where
d
G′n (s) =Gn−1 (G(s)) = G′n−1 (G(s))G′ (s).
ds
Hence, the mean population size of the n-th generation is
µn = Q′ (1) = r[Gn (1)]r−1 G′n−1 (1)G′ (1) = rµn .
9.9. Let Xn the population size of a branching process starting with one individual. Suppose that all
individuals survive, and that
Zn = 1 + X1 + X2 + · · · + Xn
is the random variable representing the accumulated population size.
(a) If Hn (s) is the probability generating function of the total accumulated population, Zn , up to and
including the n-th generation, show that
H1 (s) = sG(s), H2 (s) = sG(H1 (s)) = sG(sG(s)),
(which perhaps gives a clue to the form of Hn (s)),

(b) What is the mean accumulated population size E(Zn ) (you do not require Hn (s) for this formula)?
135
(c) If µ < 1, what is limn→∞ E(Zn ), the ultimate expected population?
(d) What is the variance of Zn ?
(a) Let pj be the probability that any individual in any generation has j descendants, and let the
probability generation function of {pj } be
∞
X
G(s) = p j sj .
j=0
The probabilities of the accumulated population sizes are as follows. Since the process starts with one
individual
P(Z0 = 1) = 1 P(Z1 = 0) = 0, P(Z1 = n) = pn−1 (n = 1, 2, 3, . . .).
Hence the generating function of P(Z1 = n) is given by H1 (s), where
∞
X ∞
X ∞
X
H1 (s) = P(Z1 = r)sr = P(X1 = r − 1)sr = pr−1 sr = sG(s).
r=1 r=1 r=1
For the probability of Z2 , use the identity

∞
X
P(Z2 = n) = P(Z2 = n|Z1 = r − 1)P(Z1 = r − 1).
r=1
Then the probability generating function H2 (s) has the series

∞ ∞ ∞
X X X
H2 (s) = P(Z2 = n)sn = P(Z2 = n|Z1 = r)P(Z1 = r)sn
n=1 n=1 r=1
∞ ∞
X X
= pr−1 P(Z2 = n|Z1 = r)sn
n=1 r=1
∞ ∞
X X
= pr−1 E(sZ2 ) = pr−1 E(s(Z1 +Y1 )+(Z1 +Y2 )+···+(Z1 +Yr ) )
r=1 r=1
∞
X
= pr−1 E(sZ1 +Y1 )E(sZ1 +Y2 ) · · · E(sZ1 +Yr )
r=1
∞
X
= pr−1 [sG(s)]r = sG[sG(s)],
r=1
using a method similar to that of Section 9.2. In this analysis, in the second generation, it is assumed that
X2 = Y1 + Y2 + · · · + Yr ,
where {Yj } are iid.

(b) The mean of the accumulated population is (see eqn (9.7))
E(Zn ) = E(1 + X1 + X2 + · · · + Xn ) = E(1) + E(X1 ) + E(X2 ) + · · · + E(Xn )

= 1 + µ + µ2 + · · · + µn
1 − µn+1
= (µ 6= 1).
1−µ
If µ = 1, then E(Zn ) = n + 1.
(c) If µ < 1, then from (b) E(Zn ) → 1/(1 − µ).
136
(d) The variance of Zn is, from Section 9.3(i),
V(Zn ) = V(1 + X1 + X2 + · · · + Xn ) = V(1) + V(X1 ) + V(X2 ) + · · · + V(Xn )

n n
X σ 2 µr−1 (µr − 1) σ 2 X 2r−1
= 0+ = (µ − µr−1 )
µ−1 µ−1
r=1 r=1

σ2 µ(1 − µ2n ) 1 − µn
= 2
− (µ 6= 1)
µ−1 1−µ 1−µ
σ 2 (1 − µn )(1 − µn+1 )
=
(1 − µ)(1 − µ2 )
9.10. A branching process starts with one individual and each individual has probability pj of producing j
descendants independently of every other individual. Find the mean and variance of {pj } in each of the
following cases, and hence find the mean and variance of the population of the n-th generation:
e−µ µj
(a) pj = , (j = 0, 1, 2, . . .) (Poisson);
j!
(b) pj = (1 − p)j−1 p (j = 1, 2, . . . ; 0 < p < 1) (geometric);

r+j−1 j
(c) pj = p (1 − p)r , (j = 0, 1, 2, . . . ; 0 < p < 1) (negative binomial).
r−1
where r is a positive integer, the process having started with one individual (a negative binomial distribu-
tion).
(a) For the Poisson distribution with intensity µ,
e−µ µj
pj = .
j!
Its probability generating function is G(s) = e−µ(1−s) . Therefore
G′ (s) = µe−µ(1−s) , G′′ (s) = µ2 e−µ(1−s) ,
and the mean and variance of the first generation are
µ = µ, σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = µ2 + µ + µ2 = µ.
The mean and variance of the n-th generation are (see Section 9.3), for µ 6= 1,
σ 2 µn−1 (µn − 1) µn (µn − 1)

µn = µn = µn , σ 2n = = .
µ−1 µ−1
(b) The geometric distribution is pj = q j−1 p, where q = 1 − p, which has the probability generating
function G(s) = q/(1 − ps). Then
pq 2p2 q
G′ (s) = , G′′ (s) = .
(1 − ps)2 (1 − ps)3
The mean and variance of the first generation are
p p
µ = G′ (1) = , σ 2 = G′′ (1) + G′ (1) − [G′ (1)]2 = .
q q2
The mean and variance of the size of n-th generation are
n n n
p σ 2 µn−1 (µn − 1) 1 p p
µn = , σ 2n = = −1 (p 6= 21 ).
q µ−1 2p − 1 q q
137
(c) The negative binomial distribution is

r+j −1 j r
pj = p q (q = 1 − p).
r−1
Its probability generating function is r

q
G(s) = .
1 − ps
The derivatives are
rpq r r(r + 1)p2 q r
G′ (s) = r+1
, G′′ (s) = .
(1 − ps) (1 − ps)r+2
Hence, the mean and variance of the first generation are
rp rp
µ= , σ2 = , (p 6= 1).
1−p (1 − p)2
The mean and variance of the size of the populations of the n-th generation are
n
rp
µn = ,
1−p
n n
σ 2 µn−1 (µn − 1) 1 rp rp
σn = = −1 .
µ−1 rp − 1 + p 1−p 1−p
9.11. A branching process has a probability generating function

r
1−p
G(s) = , (0 < p < 1),
1 − ps
where r is a positive integer (a negative binomial distribution), the process having started with one individ-
ual. Show that extinction is not certain if p > 1/(1 + r).
We need to investigate solutions of g = G(g) (see Section 9.4). This equation always has the solution
g = 1, but does it have a solution less than 1? For this distribution the equation for g becomes
g(1 − gp)r = (1 − p)r .
Consider where the line y = (1 − p)r and the curve y = g(1 − gp)r intersect in terms of g for fixed p and
r. The curve has a stationary value where
dy
= (1 − gp)r − rp(1 − gp)r−1 = 0,
dg
which occurs at g = 1/[p(1 + r)], which is a maximum. The line and the curve intersect for a value of g
between g = 0 and g = 1 if p > 1/(1 + r), which is the condition that extinction is not certain. Graphs of
the line and curve are shown in Figure 9.3 for p = 21 and r = 2.
9.12. Let Gn (s) be the probability generating function of the population size of the n-th generation of a
branching process. The probability that the population size is zero at the n-th generation is Gn (0). What
is the probability that the population actually becomes extinct at the n-th generation?
In Example 9.1, where pj = 1/2j+1 (j = 0, 1, 2, . . .), it was shown that
∞
n X nr−1
Gn (s) = + sr .
n+1 (n + 1)r+1
r=1
Find the probability of extinction,

(a) at the n-th generation,
(b) at the n-th generation or later.
What is the mean number of generations until extinction occurs?
138
y
g1 p = 0.5
r=2
1
Figure 9.3: Graphs of the line y = (1 − p)r and the curve y = g(1 − gp)r with p = 2 and r = 2 for
Problem 9.11.
The probability that the population is extinct at the n-th generation is Gn (0), but this includes
extinction of previous generations at r = 1, 2, . . . , n − 1. The probability is therefore Gn (0) given that
individuals have survived at the (n − 1)-th generation, namely Gn (0) − Gn−1 (0).
(a) In this example Gn (s) = n/(n + 1). Hence probability of extinction at the n-th generation is
n n−1 1
Gn (0) − Gn−1 (0) = − = .
n+1 n n(n + 1)
(b) Since ultimate extinction is certain, the probability that extinction occurs at or after the n-th
generation is
n−1 1
1 − Gn−1 (0) = 1 − = .
n n
The mean number of generations until extinction occurs is
∞ ∞ ∞
X X n X 1
n[Gn (0) − Gn−1 (0)] = = .
n(n + 1) n+1
n=1 n=1 n=1
This series diverges so that the number of generations is infinite.
9.13. An annual plant produces N seeds in a season which is assumed to have a Poisson distribution with
parameter λ. Each seed has a probability p of germinating to create a new plant which propagates in the
following year. Let M be the number of new plants. Show that pm , the probability that there are m growing
plants in the first year is given by
pm = (pλ)m e−pλ /m! (m = 0, 1, 2, . . .),
that is Poisson with parameter pλ. Show that its probability generating function is
G(s) = epλ(s−1) .
Assuming that all the germinated plants survive and that each propagates in the same manner in succeeding
years, find the mean number of plants in year k. Show that extinction is certain if pλ ≤ 1.
Given that plant produces seeds as a Poisson process of intensity λ, then

λn e−λ
fn = .
n!
Then
∞ ∞
X r λn e−λ X (1 − p)i λi
pm = pm (1 − p)r−m = (λp)m e−λ
m n! i!
r=m i=0
(pλ)m e−pλ
= .
m!
139
Its probability generating function is
∞
X (pλ)m
G(s) = e−pλ sm = e−pλ+pλs .
m!
m=0
The mean of the first generation is

µ = G′ (1) = pλepλ(s−1) |s=1 = pλ.
The mean of the n-th generation is therefore
µn = µn = (pλ)n .
Extinction occurs with probability g, where g is the smaller solution of g = G(g), that is, the smaller
solution of
g = e−pλ epλg .
Consider the line y = g and the exponential curve y = e−pλ epλg . On the curve, its slope is
y
1.0
0.8
0.6
0.4
0.2
g
0.2 0.4 0.6 0.8 1.0
Figure 9.4: Graphs of the line y = g and the curve y = e−pλ epλg with λ = 2 and p = 1 for Problem
9.11.3
dy
= pλe−pλ epλg ,
dg
and its slope at g = 1 is pλ. Since e−pλ epλg → 0 as g → −∞, and e−pλ epλg and its slope decrease as
g decreases, then the only solution of g = G(g) is g = 1 if pλ ≤ 1. Extinction is certain in this case. If
pλ > 1 then there is a solution for 0 < g < 1. Figure 9.4 shows such a solution for λ = 2 and p = 1.
9.14. The version of Example 9.1 with a general geometric distribution is the branching process with
pj = (1 − p)pj , (0 < p < 1; j = 0, 1, 2, . . .). Show that
1−p
G(s) = .
1 − ps
Using an induction method, prove that
(1 − p)[pn − (1 − p)n − ps{pn−1 − (1 − p)n−1 }]
Gn (s) = , (p 6= 12 ).
[pn+1 − (1 − p)n+1 − ps{pn − (1 − p)n }]
Find the mean and variance of the population size of the n-th generation.
What is the probability of extinction by the n-th generation? Show that ultimate extinction is certain
if p < 21 , but has probability (1 − p)/p if p > 21 .
As in Problem 9.2, the generating function for the first generation is

1−p
G(s) = .
1 − ps
140
Consider
(1 − p)[pn − (1 − p)n − p[(1 − p)/(1 − ps)]{pn−1 − (1 − p)n−1 }]
Gn (G(s)) =
[pn+1 − (1 − p)n+1 − p[(1 − p)/(1 − ps)]{pn − (1 − p)n }]
(1 − p)[{pn − (1 − p)n }(1 − ps) − p(1 − p){pn−1 − (1 − p)n−1 }]
=
{pn+1 − (1 − p)n+1 }(1 − ps) − p(1 − p){pn−1 − (1 − p)n−1 }
(1 − p)[pn+1 − (1 − p)n+1 − ps{pn − (1 − p)n }]
=
pn+2 − (1 − p)n+2 − ps{pn+1 − (1 − p)n+1 }
= Gn+1 (s).
Hence if the formula is true for Gn (s), then it is true for Gn+1 (s). It can be verified for G2 (s), so that by
induction on the integers, it is true for all n.
The probability of extinction by the n-th generation is
(1 − p)[pn − (1 − p)n ]
Gn (0) = .
pn+1 − (1 − p)n+1
If p > 21 , express in the following form
(1 − p)[1 − ((1 − p)/p)n ] 1−p

Gn (0) = →
p[1 − ((1 − p)/p)n+1 ] p
as n → ∞, which is the probability of ultimate extinction. If p < 21 , then
[(p/(1 − p))n − 1]
Gn (0) = → 1,
[(p/(1 − p))n+1 − 1]
as n → ∞: extinction is certain.
9.15. A branching process starts with one individual, and the probability of producing j descendants has
the distribution {pj }, (j = 0, 1, 2, . . .). The same probability distribution applies independently to all
descendants and their descendants. If Xn is the size of the n-th generation, show that
E(Xn ) ≥ 1 − P(Xn = 0).
In Section 9.3 it was shown that E(Xn ) = µn , where µ = E(X1 ). Deduce that the probability of extinction
eventually is certain if µ < 1.
By definition
∞ ∞
X X
E(Xn ) = jP(Xn = j) ≥ P(Xn = j) = 1 − P(Xn = 0).
j=1 j=1
Hence
P(Xn = 0) = 1 − µn .
Therefore, if µ < 1, then P(Xn = 0) → 1 as n → ∞. This conclusion is true irrespective of the distribution.
9.16. In a branching process starting with one individual, the probability that any individual has j descen-
dants is pj = α/2j , (j = 0, 1, 2, . . . , r), where α is a constant and r is fixed. This means that any individual
can have a maximum of r descendants. Find α and the probability generating function G(s) of the first
generation. Show that the mean size of the n-th generation is
n
2r+1 − 2 − r
µn = .
2r+1 − 1
What is the probability of ultimate extinction?
Given pj = α/2j , then for it to be a probability distribution

r
X h i r+1
α 1 1 1 1
= α 1 + + 2 + ··· + r = 2α 1 − = 1.
2j 2 2 2 2
j=0
141
Therefore the constant α is defined by
1
α= . (i)
2[1 − ( 21 )r+1 ]
r
X sj s s2 sr [1 − (s/2)r+1 ]
G(s) = α = α 1 + + 2 + ··· + r =α , (ii)
2j 2 2 2 (1 − 21 s)
j=0
using the formula for the sum of the geometric series: α is given by (i).
The derivative of G(s) is

′
2−r α 21+r − 2(1 + r)sr + rs1+r
G (s) = .
(s − 2)2
Hence the mean value of the first generation is
2r+1 − 2 − r
µ = G′ (1) = 2−r α 21+r − 2 − r) = .
2r+1 − 1
By (9.7), the mean of the n-th generation is
n
n 2r+1 − 2 − r
µn = µ = .
2r+1 − 1
Since
2r+1 − 2 − r
µ= < 1,
2r+1 − 1
then, by Problem 9.15, ultimate extinction is certain.
9.17. Extend the tree in Figure 9.4 for the gambling martingale in Section 9.5 to Z4 , and confirm that
E(Z4 |Z0 , Z1 , Z2 , Z3 ) = Z3 .
confirm also that E(Z4 ) = 1.
Extension of the gambling martingale to Z4 is shown in Figure 9.5. The values for the random variables
Z0 Z1 Z2 Z3 Z4
16
8
0
4
0 8
-8
2
12
4 -4
0
4
-4
1 -12
6 14
2 -2
-2 6
0 -10
10
2 -6
-2
2
-6
-14
Figure 9.5: Martingale for Problem 9.17.
Z4 are:
Z4 = {even numbers between -14 and 16 inclusive}
The mean value of Z4 is given by
15
X 1
E(Z4 ) = (−24 + 2m + 2) = 1,
24
m=0
142
or the mean can be calculated from the mean of the final column of numbers in Figure 9.5.
9.18. A gambling game similar to the gambling martingale of Section 9.5 is played according to the following
rules:
(a) the gambler starts with £1, but has unlimited resources;
(b) against the casino, which also has unlimited resources, the gambler plays a series of games in which
the probability that the gambler wins is 1/p and loses is (p − 1)/p, where p > 1;
(c) at the n-th game, the gambler either wins £(pn − pn−1 ) or loses £pn−1 .
If Zn the gambler’s asset/debt at the n-th game, draw a tree diagram similar to that of Figure 9.3 as
far as Z3 . Show that Zn has the possible outcomes
{−p − p2 , −p2 , −p, 0, p3 − p2 − p, p3 − p2 , p3 − p, p3 }
and confirm that

E(Z2 |Z0 , Z1 ) = Z1 , E(Z3 |Z0 , Z1 , Z2 ) = Z2 ,
which indicates that this game is a martingale. Show also that
E(Z1 ) = E(Z2 ) = E(Z3 ) = 1.
Assuming that it is a martingale, show that, if the gambler first wins at the n-th game, then the gambler
will have an asset gain or debt of £(pn+1 − 2pn + 1)/(p − 1). Explain why a win for the gambler can only
be guaranteed for all n if p ≥ 2.
The tree diagram for this martingale is shown in Figure 9.6. From the last column in the Figure 9.6,
Z0 Z1 Z Z3
2
3
p
2
p
0
p
3 2
p -p
0
2
-p
1 3
p -p
2
p -p
-p
0
3 2
p -p -p
-p
2
-p -p
Figure 9.6: Tree diagram for Problem 9.18.
it can be seen that the elements of Z3 are given by
{−p − p2 , −p2 , −p, 0, p3 − p2 − p, p3 − p2 , p3 − p, p3 }.
For the other conditional means,
E(Z2 |Z0 , Z1 ) has outcomes {p, 0},
E(Z3 |Z0 , Z1 , Z2 , Z3 ) outcomes {p2 , 0, p2 − p, −p}.

For the other means,
1 p−1
E(Z1 ) = p +0 = 1,
p p
2
1 1p−1 p−11 p−1
E(Z2 ) = p2 +0 + (p2 − p) −p = 1,
p2 p p p p p
etc.
143
Suppose the gambler first wins at the nth game: on the tree the path will be lowest track until the last
game. Extending the result for Z3 , the gambler then has an asset of

pn − 1 pn+1 − 2pn + 1
£(pn − 1 − p − p2 − · · · − pn−1 ) = £ pn − =£ .
p−1 p−1
To guarantee winning requires

pn+1 − 2pn + 1
= pn − pn−1 − pn−2 − · · · − p − 1 > 0
p−1
for all n. This will certainly be true if p > 2. A smaller value of p will guarantee winnings but it will
depend on n.
9.19. Let X1 , X2 , . . . be independent random variables with means µ1 , µ2 , . . . respectively. Let
Zn = X1 + X2 + · · · + Xn ,
and let Z0 = X0 = 0. Show that the random variable

n
X
Yn = Zn − µi , (n = 1, , 2, . . .)
i=1
is a martingale with respect to {Xn }. [Note that E(Zn |X1 , X2 , . . . , Xn ) = Zn .]
The result follows since

n+1
X
E(Yn+1 |X1 , X2 , . . . , Xn ) = E(Zn+1 − µi |X1 , X2 , . . . , Xn )
i=1
n+1
X
= E(Zn + Xn+1 − µi |X1 , X2 , . . . , Xn )
i=1
n+1
X
= Zn + µn+1 − µi
i=1
n
X
= Zn − µi = Yn
i=1
Hence the random variable Yn is a martingale.
9.20. Consider an unsymmetric random walk which starts at the origin. The walk advances one position
with probability p and retreats one position with probability 1 − p. Let Xn be the random variable giving
the position of the walk at step n. Let Zn be given by
Zn = Xn + (1 − 2p)n.
Show that E(Z2 |X0 , X1 )

{−2p, 2 − 2p}.
denoted by the random variable Z1 . Generally show that {Zn } is a martingale with respect to {Xn }.
The conditional mean
E(Z2 |X0 , X1 ) = E(X2 + (1 − 2p)2|X0 , X1 )

= E(X2 |X0 , X1 ) + 2(1 − 2p)
which has the possible outcomes
{1 + 1 − 2p, −1 + 1 − 2p} = {2 − 2p, −2p}
144
denoted by the random variable Z1 .
By the Markov property of the random walk,
E(Zn+1 |X0 , X1 , . . . , Xn ) = E(Zn+1 |Xn ) = E(Xn+1 + (1 − 2p)(n + 1)|Xn )
Suppose that Xn = k. Then the walk either advances one step with probability p or retreats one step with
probability 1 − p. Therefore
E(Xn+1 + (1 − 2p)(n + 1)|Xn ) = p(k + 1) + (1 − p)(k − 1) + (1 − 2p)(n + 1)

= k + (1 − 2p)n = Xn .
9.21. In the gambling martingale of Section 9.5, the random variable Zn , the gambler’s asset, in a game
against a casino in which the gambler starts with £1 and doubles the bid at each play is given by Zn which
can take the values
{−2n + 2m + 2}, (m = 0, 1, 2, . . . , 2n − 1).
Find the variance of Zn . What is the variance of
E(Zn |Z0 , Z1 , . . . , Zn−1 )?
Then the sum of the elements in random variable Zn which can take the values
{−2n + 2m + 2}, m = 0, 1, 2, . . . , 2n − 1).
Therefore
2n −1 2n −1 2n −1
X X n n n
X
Zn = (−2 + 2m + 2) = (−2 + 2)2 + 2 m = 2n
m=0 m=0 m=1
Since all the elements in are equally likely to occur after n steps, then
n
2 −1
1 X 1
E(Zn ) = n Zn = n 2n = 1.
2 2
m=0
The variance of Zn is given by

n
2 −1
1 X 1
V(Zn ) = E(Zn2 ) − [E(Zn )]2 = (−2n + 2m + 2)2 − 1 = (22n − 1),
2n 3
m=0
since
2n −1
X 2n
(−2n + 2m + 2)2 = (2 + 22n ).
3
m=0
Since
E(Zn |Z0 , Z1 , . . . , Zn−1 ) = Zn−1 ,
then
1 2(n−1)
V[E(Zn |Z0 , Z1 , . . . , Zn−1 )] = V(Zn−1 ) = [2 − 1]
3
by the previous result.
9.22. A random walk starts at the origin, and, with probability p1 advances one position and with probability
q1 = 1 − p1 retreats one position at every step. After 10 steps the probabilities change to p2 and q2 = 1 − p2
respectively. What is the expected position of the walk after a total of 20 steps?
145
After 10 steps the walk could be at any position in the list of even positions
{−10, −8, −6 . . . 6, 8, 10},
which are the random variable Xr . Let the random variable Yn be the position of the walk after 20 steps
so that
Yn = {−20, −18, −16, . . . , 16, 18, 20}.
Then the position after a further 10 steps is
E(Yn |Xr ) = Xr + 10(p2 − q2 ).
Its expected position is
E[E(Yn |Xr )] = E[Xr + 10(p2 − q2 )] = 10(p1 + p2 − q1 − q2 ).
9.23. A symmetric random walk starts at the origin x = 0. The stopping rule that the walk ends when the
position x = 1 is first reached is applied, that is the stopping time T is given by
T = min{n : Xn = 1},
where Xn is the position of the walk at step n. What is the expected value of T ? If this walk was interpreted
as a gambling problem in which the gambler starts with nothing with equal odds of winning or losing £1
at each play, what is the flaw in this stopping rule as a strategy of guaranteeing a win for the gambler in
every game? [Hint: the generating function for the probability of the first passage is
1
G(s) = [1 − (1 − s2 ) 2 ]/s :
see Problem 3.11.]
The probability generating function for the first passage to x = 1 for the walk starting at the origin is
1 1 s s3 s5
G(s) = [1 − (1 − s2 ) 2 ] = + + + O(s7 ),
s 2 8 16
which imples that the probability that the first visit to x = 1 occurs at the 5-th step is 1/16.
The mean of the first visits is
1
1 − (1 − s2 ) 2
µ = G′ (s)|s=1 = 1 = ∞.
s2 (1 − s2 ) 2 s=1
It seems a good ploy but would take, on average, an infinite number of plays to win £1.
9.24. In a finite-state branching process, the descendant probabilities are, for every individual,
2m−j
pj = , (j = 0, 1, 2, . . . , m),
2m+1 − 1
and the process starts with one individual. Find the mean size of the first generation. If Xn is the size of
the n-th generation, explain why n
2m+1 − 1
Zn = Xn
2m+1 − m − 2
defines a martingale over {Xn }.
In this model of a branching process each descendant can produce not more than m individuals. It can
be checked that
m m
X X 2m−j
pj = m+1
= 1,
2 −1
j=0 j=0
using the formula for the sum of the geometric series.
146
The probability generating function for the first generation is
m m
X 2m X s j 2m+1 − sm+1
G(s) = p j sj = m+1
= m+1 .
2 −1 2 (2 − 1)(2 − s)
j=0 j=0
Its first derivative is

2m+1 − 2(m + 1)sm + msm+1
G′ (s) = .
(2m+1 − 1) (2 − s)2
Therefore the mean of the first generation is
2m+1 − m − 2
µ = G′ (1) = .
2m+1 − 1
The random variable Zn is simply Zn = Xn /µn (see Section 9.5).
9.25. A random walk starts at the origin, and at each step the walk advances one position with probability
p or retreats with probability 1 − p. Show that the random variable
Yn = Xn2 + 2(1 − 2p)nXn + [(2p − 1)2 − 1]n + (2p − 1)2 n2 ,
where Xn is the random variable of the position of the walk at time n, defines a martingale with respect to
{Xn }.
Let α(p, n) = 2(1 − 2p)n and β(p, n) = [(2p − 1)2 − 1]n + (2p − 1)2 n2 in the expression for Yn . Then
E(Yn+1 |Xn ) = p[(Xn + 1)2 + α(p, n + 1)(Xn + 1) + β(p, n + 1)]

+(1 − p)[(Xn − 1)2 + α(p, n + 1)(Xn − 1) + β(p, n + 1)]
= Xn2 + Xn [4p − 2 + α(p, n + 1)] + [1 + (2p − 1)α(p, n + 1) + β(p, n + 1)]
The coefficients in the last expression are
4p − 2 + α(p, n + 1) = 4p − 2 + 2(1 − 2p)(n + 1) = 2(1 − 2p)n = α(p, n),
and
1 + (2p − 1)α(p, n + 1) + β(p, n + 1) =

1 + (2p − 1)2(1 − 2p)(n + 1) + [(2p − 1)2 − 1](n + 1) + (2p − 1)2 (n + 1)2
= [(2p − 1)2 − 1]n + (2p − 1)2 n2 = β(p, n)
Hence
E(Yn+1 |Xn ) = Xn2 + 2(1 − 2p)nXn + [(2p − 1)2 − 1]n + (2p − 1)2 n2 = Yn ,
so that, by definition, Yn is a martingale.
9.26. A simple epidemic has n0 susceptibles and one infective at time t = 0. If pn (t) is the probability that
there are n susceptibles at time t, it was shown in Section 9.7 that pn (t) satisfies the differential-difference
equations (see eqns (9.15 and (9.16))
dpn (t)
= β(n + 1)(n0 − n)pn+1 (t) − βn(n0 + 1 − n)pn (t),
dt
for n = 0, 1, 2, . . . n0 . Show that the probability generating function
n0
X
G(s, t) = pn (t)sn
n=0
satisfies the partial differential equation

∂G(s, t) ∂G(s, t) ∂ 2 G(s, t)
= β(1 − s) n0 −s .
∂t ∂s ∂s2
147
Nondimensionalize the equation by putting τ = βt. For small τ let
G(s, τ /β) = G0 (s) + G1 (s)τ + G2 (s)τ 2 + · · · .
Show that
∂Gn−1 (s) ∂ 2 Gn−1 (s)
nGn (s) = n0 (1 − s) − s(1 − s) ,
∂s ∂s2
for n = 1, 2, 3, . . . n0 . What is G0 (s)? Find the coefficients G1 (s) and G2 (s). Hence show that the mean
number of infectives for small τ is given by
1
n0 − n0 τ − n0 (n0 − 2)τ 2 + O(τ 3 ).
2
In Example 9.9, the number of susceptibles initially is given by n0 = 4. Expand p0 (t), p1 (t) and p2 (t)
in powers of τ and confirm that the expansions agree with G1 (s) and G2 (s) above.
Multiply the difference equation by s and sum from s = 0 to s = n0 giving

n0 n0 −1 n0
X X X
p′ (t)sn = β (n + 1)(n0 − n)pn+1 (t) − β n(n0 + 1 − n)pn (t)sn ,
n=0 n=0 n=1
or,
n0 n0 n0
X X X
Gt (s, t) = βn0 mpm (t)sm−1 − β m(m − 1)pm (t)sm−1 − βn0 npn (t)sn
m=1 m=2 n=1
n0
X
+β n(n − 1)pn (t)sn
n=1
= βn0 Gs (s, t) − βsGss (s, t) − βn0 sGs (s, t) + βs2 Gss (s, t)
= βn0 (1 − s)Gs (s, t) + βs(s − 1)Gss (s, t),
as required.
Let τ = βt. Then the equation for H(s, τ ) = G(s, τ /β) is

∂H(s, τ ) ∂H(s, τ ) ∂ 2 H(s, τ )
= (1 − s) n0 −s .
∂τ ∂s ∂s2
For small τ , let
H(s, τ ) = G(s, τ /β) = H0 (s) + H1 (s)τ + H2 (s)τ 2 + · · · ,
and substitute this series into the partial differential equation for H(s, τ ), so that
H1 (s) + 2H2 (s)τ + · · · = (1 − s)n0 [H0′ (s) + H1′ (s)τ + · · ·] − s(1 − s)n0 [H0′′ (s) + H1′′ (s)τ + · · ·].
Equating powers of τ , we obtain
′ ′
nHn (s) = (1 − s)n0 Hn−1 (s) − s(1 − s)n0 Hn−1 (s), (n = 1, 2, 3, . . .). (i)
For τ = 0,
n0
X
H0 (s) = G(s, 0) = pn (0)sn .
n=0
Since the number of susceptibles is n0 at time t = 0. Therefore
pn0 (0) = n0 , pn (0) = 0, (n 6= n0 ).
n0
Hence H0 (s) = s .
From (i),
∂H0 (s) ∂ 2 H0 (s)
H1 (s) = n0 (1 − s) − s(1 − s)
∂s ∂s2
= n20 (1 − s)sn0 −1 − s(1 − s)n0 (n0 − 1)sn0 −2
= n0 (1 − s)sn0 −1 ,
148

1 ∂H1 (s) ∂ 2 H1 (s)
H2 (s) = n0 (1 − s) − s(1 − s)
2 ∂s ∂s2
1
= n0 (1 − s)[n0 {(n0 − 1)sn0 −2 − n0 sn0 −1 }
2
1
− s(1 − s)[n0 {(n0 − 1)(n0 − 2)sn0 −3 − n0 (n0 − 1)sn0 −2 }]
2
1
= n0 sn0 −2 (1 − s)(2n0 − 2 − n0 s)
2
The mean number of infectives is given by
µ = Hs (1, τ ) = H0′ (1) + H1′ (1)τ + H2′ (1)τ 2 + O(τ 3 )

= n0 sn0 −1 + [n0 (n) − 1)sn0 −2 − n20 sn0 −1 ]τ

1
+ [n30 sn0 −1 + (n0 − 1)(2n0 − 3n20 )sn0 −2 + (n0 − 2)(2n20 − 2n0 )sn0 −3 ]τ 2 + O(τ 3 )

2 s=1
1 2 3
= n0 − n0 τ − n0 (n0 − 2)τ + O(τ ),
2
where τ = βt.
149
Chapter 10
Brownian motion: Wiener process
10.1. Let X(t) be a standard Brownian motion. (a) Find P[X(2) > 3]. (b) Find P[X(3) > X(2)].
(a) The standard Brownian motion has the normal distribution with variance t, and density
1
√ exp[−x2 /(2t)].
2πt
Hence the density for X(2) (that is, for t = 2)is
1
√ exp[−x2 /(4)].
4π
Hence
Z 3
1 2
P[X(2) > 3] = 1− √ e−x /4
dx
2 π −∞
Z √
3/ 2
1 2 √
= 1− √ e−z /2
dz, (x = z 2)
2π −∞
√
= 1 − Φ[3/ 2].
(b) From the definition of the Wiener process X(3)−X(2) has a normal distribution N [0, 3−2] = N [0, 1].
Hence
P[X3 > X2 ] = P[X(3) − X(2)] > 0] = 1 − Φ(0) = 0.5.
10.2 Let X(t) be a Brownian motion with mean 0 and variance σ 2 t starting at the origin.
(a) Find the distribution of |X(t)|, the absolute distance of X(t) from the origin.
(b) If σ = 1, evaluate P(|X(5)|) > 1).
(a) The random variable X(t) is normal N (0, σ 2 t] with density function

1 x2
fX (x) = √ exp − 2 , −∞ < x < ∞.
σ 2πt 2σ t
Let U (t) = |X(t)|. Then

0 u<0
P(U (t) ≤ u) = P(|X(t)| ≤ u) =
P(−u ≤ X(t) ≤ u) u≥0
Differentiating the density function of U (t) is given by
fU (u) = fX (u) + fX (−u).
150
Finally the density function |X(t)| is
r
1 2 x2
fX (x) = 0, x < 0, fX (x) = exp − 2 , x ≥ 0,
σ πt 2σ t
which is known as the half-normal distribution. The random variable |X(t)| defines a reflected
Brownian motion.
(b) With σ = 1, X(t) is normal N (0, t). Hence
√
P(|X(5)| > 5)) = 2[1 − Φ(1/ 5)] = 2[1 − Φ(0.447)] = 0.635
from a table of cumulative normal distribution.

10.3. X(t) = ln[Z(t)] is a Brownian motion with variance σ 2 t.
(a) Find the distribution of Z(t).
(b) Evaluate P[Z(t) > 2] when σ 2 = 0.5.
(a) Denote the density of X(t) by

1 x2
fX (x) = √ exp − .
σ 2 2πt 2σ 2 t
Thus
P(Z ≤ z) = P(eX ≤ z) = P(X ≤ z).
Let fZ (z) be the density of Z. Differentiating the previous equation with respect to z:

d 1 (ln z)2
fZ (z) = fX (ln z) (ln z) = √ exp − , (z > 0).
dz σz 2πt 2σ 2 t
The distribution function of Z is given by

Z z

1 1 (ln u)2
FZ (z) = √ exp − du
σ 2πt 0
u 2σ 2 t
Z ln z
1 v2
= √ exp − 2 dv [where v = ln u]
σ 2πt −∞
2σ t
Z (ln z)/(σt)
1 1 2 √
= √ e− 2 w dw [where w = v/(σ t)]
2π −∞
√
= Φ[(ln z)/(σ t)] [z > 0]
Hence Z is normal and could represent a Brownian motion.

(b) √
2 ln 2
P[Z(t) > 2] = 1 − Φ √ .
t
10.4. Let X(t), (t ≥ 0) be a standard Brownian motion. Let τa be stopping time for X(t) (that is, the first
time that the process reaches state a). Explain why

X(t) 0 < t < τa ,
Y (t) =
2X(τa ) − X(t) t ≥ τa ,
represents the reflected process. Show that Y (t) is also a standard Brownian motion.
The Brownian motion first hits x = a at time t = τa , that is X(τa ) = a. For t ≥ τa the graph of Y (t)
is the reflection of X(t) in the line x = a: subsequently this process never exceeds X(τa ).
For any differences ti+1 − ti where τa > ti+1 > ti , Y (t) will be normal N (0, t). For t > τa choose
intervals with τ < ti < ti+1 . Differences will eliminate 2X(τa ). Then use the observation that if X(0) is
normal then so is −X(t). Therefore Y (t) is a standard Brownian motion.
151
10.5. For a standard Brownian motion X(t), (t ≥ 0) show that E[X 2 (t)] = t using the mgf for X(t)
The mgf of the standard Brownian motion defined by X(t) is
Z ∞
1
E[esX ] = √ exp[sx] exp[−x2 /(2t)]dx
2πt −∞
1
= exp[ s2 t].
2
Expanding as a power series the mgf is
∞ n
X 1 s2 t
.
n! 2
s=0
From the coefficient of s2 , it follows that E[X 2 ] = t.

10.6. In a standard Brownian motion let ta be the first time that the process reaches a ≥ 0 – often known
as the hitting time. Let Y (t) be the maximum value of X(s) in 0 ≤ s ≤ t. Both ta and Y (t) are random
variables with the property that ta ≤ t if and only if Y (t) ≥ a. Using the reflection principle (Section 10.7),
we know that
P[Y (t) ≥ a|ta ≤ t] = 21 .
Using this result show that Z ∞
√ 2
P(ta ≤ t) = exp[−x2 /(2t)]dx.
πt a
What is the mean hitting time?
Since
P[Y (t) ≥ a] = P[Y (t) ≥ a, ta ≤ t] = P[Y (t) ≥ a|ta ≤ t]P[ta ≤ t],
and the reflection principle (Section 10.7 and Figure 10.4), using the result in the question,
√
Z ∞
2
P[ta ≤ t] = 2P[Y (t) ≥ a)] = exp[−x2 /(2t)]dx,
πt a
√
from the normal distribution of the standard Brownian motion. Use the substitution z = x/ t. Then
Z ∞
√2
P[ta ≤ t] = exp(−z 2 /2)dz.
π √
a/ t
Finally differentiation with respect to t, gives the probability density:

a
fa (t) = √ exp[−a2 /(2t)], (0 ≤ t, Inf ty).
(2πt3 )
The mean hitting time should be given by
Z ∞
tfa (t)dt,
0
but this integral diverges implying that the mean hitting time is infinite.
10.7. X(t) is a standard Brownian motion. Show that Y (t) = X(a2 t)/a where a > 0 is a constant is a
standard Brownian motion.
X(t) has the density
1 x2
fX (t) = √ exp − ,
2πt 2t
and X(a2 t) has the density
2 1 x2
fX (a t) = √ exp − 2 .
a 2πt 2a t
Then Z ay
1 x2
P[Y (t) > y] = P[(1/a)X(a2 t)] = P[X(a2 t > ay] = √ exp − dx.
a 2πt −∞
2a2 t
152
Differentiating this expression with respect to y, the density of Y (t) is given by:

1 a2 y 2 1 y2
fY (t) = √ exp − 2 × a = √ exp − ,
a 2πt 2a t 2πt 2t
which is the density of a standard Brownian motion.

10.8. X(t) and Y (t) are independent standard Brownian motions.
(a) Find the probability densities of U (t) = X 2 (t)/t and V (t) = Y 2 (t)/t
(b) Using mgf ’s (see Problem 1.29) show that the probability distribution of W 2 (t) = U 2 (t) + V 2 (t) is
exponential.
√
(c) Find P[R(t) ≥ r] where R(t) = [X 2 (t)+Y 2 (t)] is the Euclidean distance of the two-dimensional
Brownian motion [X(t), Y (t)] form the origin.
(a) Both X(t) and Y (t) have normal N (0, t) distributions. Let U (t) = X 2 (t)/t. Then
√ √
P[U (t) ≤ u] = P[X 2 (t)/t ≤ u] = P[− ut ≤ X(t) ≤ ut].
Differentiating, we obtain the density

" Z √ √#
ut
d 1 2 2 −1 u t 1 1
fU (u) = √ exp[−x /(2t)]dx = √ e 2 · √ = √ e− 2 u ,
du 2πt √
− ut 2πt 2 u 2πu
which that of is a χ21 distribution. Similarly V (t) = Y 2 (t)/t has the same distribution for fV (v). Notice
that these densities are independent of t.
(b) The mgf of U (t) is
Z ∞ Z ∞
1 1 1 √h2i 1 2 1
MU (s) = E[esU (t) ] = √ u− 2 e−(s− 2 )u du = e−(s− 2 )z dz = (1 − 2s)− 2 ,
2π 0
π 0
2
using the substitution u = z , and an integral from the Appendix. A similar result holds for MV (s) =
E[esV (t) ]. Since the random variables U and V are independent
MW 2 (s) = MU (s)MV (s) = 1/(1 − 2s),

1
which is the mgf of the exponential distribution with parameter 2
.
2 2 2
(c) For R (t) = X (t) + Y (t), we have
P[R2 (t) ≥ r] = P[W 2 (t) ≥ r/t] = exp[−r/(2t)]
by (b). Finally, for the Euclidean distance R(t),
P[R(t) ≥ r] = P[R2 (t) ≥ r 2 ] = exp[−r 2 /(2t)],
which gives the probability distribution function of the Euclidean distance.
10.9. X(t) is a standard Brownian motion. Find the moment generating function of

0 t = 0,
Y (t) =
tX(1/t) t > 0,
Show that
X(t)
lim = 0.
t→∞ t
The mgf is given by
MY (s) = E[esY (t) ] = E[estX(1/t) ]

Z ∞
√ t 2
= estx e−x t/2 dx
2π −∞
1 2
= e 2 s t.
153
This is the same mgf as that for X(t). It can be shown that Y (t) is also a standard Brownian motion.
Using the previous result
X(t)
lim = lim Y (1/t) = Y (0) = 0.
t→∞ t t→∞
[As remarked in Section 10.6, this limit is in the sense almost surely.] The limit does indicate a bound on
the possible growth of standard Brownian motion — it is less than t as t → ∞.
10.10. The probability density function for geometric Brownian motion is given by
1
fx (t) = √ exp[−(ln x − (µ − 12 σ 2 )t)2 /(2σ 2 t)], (x ≥ 0).
σx 2πt
for the random variable
1 2
Z(t) = exp[(µ −
σ )t + σX(t)], (x > 0)
2
Show that the mean and variance of this lognormal distribution are given by
2
E[Z(t)] = eµt , V[Z(t)] = e2µt [eσ t
− 1].

Z ∞
1
E[Z(t)] = √ exp[−(ln x − (µ − 21 σ 2 )t)2 /(2σ 2 t)]dx
σ 2πt 0
Z ∞
1
= √ exp[−(z − (µ − 21 σ 2 )t)2 /(2σ 2 t)] exp[z]dz
σ 2πt −∞
= eµt
using the substitution z = ln x and a translated version of an integral in the Appendix.
For the variance we require
Z ∞
1
E[Z 2 (t)] = √ x exp[−(ln x − (µ − 12 σ 2 )t)2 /(2σ 2 t)]dx
σ 2πt 0
Z ∞
1
= √ exp[−(z − (µ − 12 σ 2 )t)2 /(2σ 2 t)] exp[2z]dz
σ 2πt −∞
2
= e2µ+σ )t
,
using the same method. Finally
2
V[Z(t)] = E[Z 2 ] − E[Z]2 = e2µ+σ )t
− e2µt .
10.11. If X(t), (t ≥ 0) is a standard Brownian motion, use the conditional expectation

E[X(t)|X(u), 0 ≤ u < y], (0 ≤ y < t)
to show that X(t) is a martingale (see Section 9.5 for definition of martingales).
The conditional expectation is a random variable derived from X(t) given any X(u) in the interval
0 ≤< y which we have to show equals X(y). Then
E[X(t)|X(u), 0 ≤ u < y] = E[X(y) + X(t) − X(y)|X(u), 0 ≤ u < y],
= E[X(y)|X(u), 0 ≤ u < y] + E[X(t) − X(y)|X(u), 0 ≤ u < y],
= X(y) + E[X(t) − X(y)|X(u), 0 ≤ u < y],
since the first term is the value X(y) and not conditioned. For the other term
E[X(t) − X(s)|X(u), 0 ≤ u < y] = E[X(t) − X(s)] = 0,
since the conditioning is not effective in the interval (t, y), and the Brownian motions have zero mean.
Hence the result
E[X(t)|X(u), 0 ≤ u < y] = X(y)
follows
154

Stochastic Processes Solutions Manual

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Stochastic Processes Solutions Manual

Uploaded by

Copyright:

Available Formats

Stochastic Processes: An Introduction

School of Computing and Mathematics, Keele University, UK

Chapter 1: Some Background in Probability

Chapter 2: Some Gambling Problems

Chapter 3: Random Walks

Chapter 4: Markov Chains

Chapter 5: Poisson Processes

Chapter 6: Birth and Death Processes

Chapter 8: Reliability and Renewal

Chapter 9: Branching and Other Random Processes

Chapter 10: Brownian Motion: Wiener Process

Chapter 11: Computer Simulations and Projects

Chapter 1: Some Background in Probability 6

Chapter 2: Some Gambling Problems 20

Chapter 3: Random Walks 34

Chapter 4: Markov Chains 49

Chapter 5: Poisson Processes 74

Chapter 6: Birth and Death Processes 80

Chapter 7: Queues 105

Chapter 8: Reliability and Renewal 121

Chapter 9: Branching and Other Random Processes 130

Chapter 10: Brownian Motion: Wiener Process 150

Some background in probability

The events are shaded in Figure 1.1.

(a) A ∩ (B ∪ C)c ; (b) A ∩ (B ∩ C) = A ∩ B ∩ C; (c) (A ∩ B) ∩ C c ; (d) A ∪ B ∪ C;

(A ∩ (B ∪ C)c ) ∪ (B ∩ (A ∪ C)c ) ∪ (C ∩ (A ∪ B)c ).

P(A ∪ B) = 0.4 + 0.5 − 0.3 = 0.6.

(b) Since A = (A ∩ B c ) ∪ (A ∩ B) and A ∩ B c , and A ∩ B are mutually exclusive, then,

P(A) = P[(A ∩ B c ) ∪ (A ∩ B)] = P(A ∩ B c ) + P(A ∩ B),

P(Ac ∪ B c ) = P[(A ∩ B)c ] = 1 − P(A ∩ B) = 1 − 0.3 = 0.7.

1.6. For the probability generating function

find {pn } and its mean.

Given pn = e−α αn /n!, the generating function is given by

The mean and variance are given by

The probability that no warning light has failed by time t is

1.9. The geometric distribution with parameter p is given by

The generating function is given by

using the formula for the sum of a geometric series.

For the variance,  

From the Table in Example 1.1:

S = {(i, j, k)}, (i = 1, . . . , 6; j = 1, . . . , 6; k = 1, . . . , 6).

P(A ∪ (B ∪ C)) = P(A) + P(B) + P(C) − P(A ∩ (B ∪ C)).

From (1.1) (in the book)

P(A ∪ (B ∪ C)) = P(A) + P(B ∪ C) − P(A ∩ (B ∪ C)) (i).

P(B ∪ C) = P(B) + P(C). (ii)

From (i) and (ii), it follows that

P(A ∪ (B ∪ C)) = P(A) + P(B) + P(C) − P(A ∩ (B ∪ C)).

1.14. Show that 

Check the density function as follows:

The probability function is given by, for 0 ≤ x ≤ a,

and, for x > a, by

1.17. A probability generating function with parameter 0 < α < 1 is given by

Applying the binomial theorem

The moment generating function of the uniform distribution is

Apply the substitution x = µ + σ(v − σs): then

(see the Appendix for the integral).

(a) For the Bernoulli distribution

The mean is given by

The derivatives of G(s) are given by

rqpr r(r + 1)q 2 pr

P(A1 ∩ A2 ∩ A3 ∩ A4 ∩ A5 ) = P(A1 )P(A2 )P(A3 )P(A4 )P(A5 ) = 0.955 ≈ 0.774.

(a) Probability = 0.0004 × 0.9996499 = 0.000328...

For the variance,

r formula exact probability approximation