UNSW School of Mathematics and Statistics

MATH3801-MATH3901
(Higher) Probability and Stochastic Processes
T UTORIAL EXERCISES

Contents
1 Introduction to Probability theory 2

2 Random variables 4

3 Conditional Probability and Conditional Expectation 9

4 Markov Chains 13

5 The Exponential Distribution and the Poisson Process 19

6 Continuous-Time Markov Chains 26

8 Queueing theory 33

10 Brownian motion and stationary processes 38

11 Introduction to Martingales 41

Note: Most un-starred exercises are fairly straightforward tests of the mastery of the material of the
chapter, and are suitable for all students. Starred exercises require more thought, and are directed
primarily at the MATH3901 students.
1 Introduction to Probability theory
1.1. (a) If an event E1 is independent of itself, show that P(E1 ) is either 0 or 1.
(b) If P(E1 ) is 0 or 1, show that E1 is independent of all other events.
(Hint: Start from the basic definition of independence of events.)

1.2. Let E1 and E2 be indepedent events. Show that E1c and E2 are independent, and deduce that
E1c and E2c are also independent.
(Hint: Use Bayes’ first rule.)

1.3. Let E1 and E2 be two events such that P(E1 )×P(E2 ) 6= 0. Show that, if P(E1 |E2 ) > P(E1 ),
then
P(E2 |E1 ) > P(E2 ).
(Hint: Use Bayes’ first rule.)

1.4. Among the students doing a given course, there are four male students enrolled in the ordi-
nary version of the course, six female students enrolled in the ordinary version of the course,
and six male students enrolled in the higher version of the course. How many female stu-
dents must be enrolled in the higher version of the course if gender and version of the course
are to be independent when a student is selected at random?
(Answer: 9)

1.5. The probability of winning on a single toss of the dice is p (0 < p < 1). A starts, and if she
fails, she passes the dice to B, who then attempts to win on her toss. They continue tossing
the dice back and forth until one of them wins. What are their respective probabilities of
winning?
(Answer: P(A wins) = 1/(2 − p), P(B wins) = (1 − p)/(2 − p))

1.6. Suppose we have ten coins which are such that if the ith coin is flipped then heads will
appear with probability i/10 (i = 1, 2, . . . , 10). When one of the coins is randomly selected
and flipped, it shows heads. What is the conditional probability that it was the fifth coin?
(Answer: 1/11)

1.7. You are imprisoned in a dungeon together with two fellow prisoners. You are informed by
the jailer that one of you has been chosen at random to be hanged, and the other two are to
be freed. You ask the jailer to tell you privately which of your fellow prisoners will be set
free, claiming that there would be no harm in divulging this information, since you already
know that at least one of them will go free.

(a) The jailer refuses to answer the question, pointing out that if you knew which of your
fellows were to be set free, then your own probability of being hanged would rise from
1/3 to 1/2, since you would then be one of two prisoners. Show that, under reasonable
assumptions, the jailer is wrong.
(Hint: Show that P(you hanged) = P(you hanged | whatever the jailer says))
(b) You convince the jailer, and he tells you which of the other two will be set free. You
say this information to your fellow prisoners, and while the spared guy is jumping for
joy, the other one asks you to switch your identities. Would you accept?
(Hint: What is his probability of being hanged?)

2
 1.8. You have to select at random one alternative between two with a probability exactly equal to
1/2. Unfortunately, all you have at your disposal is a coin that you suspect to be biased. De-
note p the (unknown) probability of getting heads when it is flipped. Consider the following
procedure:

1) Flip the coin, and let R1 , either Heads or Tails, be the result.
2) Flip the coin again, and let R2 be the result.
3) If R1 and R2 are the same, return to step 1; otherwise proceed to step 4.
4) If R2 is heads, choose alternative 1, otherwise choose alternative 2.

Show that, using this strategy, either alternative will be selected with exact probability 1/2,
whatever the value of p ∈ (0, 1).
(Answer: the probability is p(1 − p)/[p(1 − p) + p(1 − p)] ≡ 1/2)

1.9∗. Prove Boole’s inequalities: !

n
[ n
X
P Ei ≤ P(Ei )
i=1 i=1

and !
n
\ n
X
P Ei ≥1− P(Eic ).
i=1 i=1

(Hint: For the first inequality, proceed by induction. Show that this is true for n = 1 and
then that this is true for n + 1 if true for n. For the second inequality, use the first one and
De Morgan’s laws.)

1.10∗. A plays tennis against B. During a given game, the score reaches deuce. Each player then
needs to score two more points than the other to win the game. Assuming that each point is
independently won by A with probability p, what is the probability they will have to play a
total of 2n points to end the game? What is the probability that A will win the game?
p2
(Answer: P(2n points are needed) = (2p(1 − p))n−1 (p2 + (1 − p)2 ), P(A wins) = 1−2p(1−p) )

1.11∗. Show that it is not possible to weight two dice in such a way that the sum of the two numbers
shown by these loaded dice is equally likely to take any value between 2 and 12 (inclusive).
(Hint: Write fi = P(X = i) and gi = P(Y = i), i = 1, . . . , 6, where X is the number
shown on the first die and Y the number shown on the second die. Then, write the events
P(X + Y = 2), P(X + Y = 7) and P(X + Y = 12) in terms of the fi ’s and gi ’s and show
that P(X + Y = 2) = P(X + Y = 12) = P(X + Y = 7) = 1/11 leads to a contradiction.)

1.12∗. In a sequence of independent identical trials with two possible outcomes on each trial, Suc-
cess or Failure, and with P(S) = p, what is the probability π that exactly x trials will occur
before the rth success?
(Answer: π = x−1

r
r−1
p (1 − p)x−r )

3
2 Random variables
2.1. We toss n coins, and each one shows heads with probability p, independently of each of the
others. Each coin which shows heads is tossed again. What is the probability mass function
of the number of heads

resulting2from the second round of tosses?
n 2x n−x
(Answer: pX (x) = x p (1 − p ) for x = 0, 1, . . . , n)

2.2. An individual claims to have extrasensory perception (ESP). As a test, a fair coin is tossed
ten times, and he is asked to predict in advance the outcome. Our individual gets seven out
of ten correct. What is the probability he would have done at least this well if he had no
ESP? Would you believe in his powers?
(Answer: P(X ≥ 7) = 0.17, we are not really convinced by his powers)

2.3. A university lecturer never finishes his lecture before the end of the hour and always finishes
his lectures within 2 minutes after the hour. Let X be the time that elapses between the end
of the hour and the end of the lecture and suppose that the probability density of X is

kx2 0 < x < 2
fX (x) =
0 otherwise

(a) What is the value of k?

(Answer: k = 3/8)
(b) What is the cumulative distribution function of X?
(Answer: FX (x) = x3 /8 between 0 and 2, 0 if x < 0 and 1 if x > 2)
(c) What is the probability that the lecture continues beyond the hour for between 60 and 90
seconds? (Answer: 0.2969)

2.4. Let X be a r.v. with cumulative distribution function F (x) and density f (x) = F 0 (x). Find
the probability density function of

(a) the maximum, say X(n) , of n independent random variables, each of which with distri-
bution function F (x).
(Answer: fX(n) (x) = n (F (x))n−1 f (x))
(b) the minimum, say X(1) , of n independent random variables, each of which with distri-
bution function F (x).
(Answer: fX(1) (x) = n (1 − F (x))n−1 f (x))

2.5∗. Suppose the r.v. X is continuous and has the distribution F (x). Consider another r.v. Y
defined by Y = F (X). Find the distribution of Y .
(Answer: Y ∼ U[0,1] )

2.6. If X is a nonnegative integer valued random variable, show that

+∞
X +∞
X
E(X) = P(X ≥ i) = P(X > i)
i=1 i=0
P∞
(Hint: start with X = i=1 1I{X≥i} and take expectation.)

2.7. Let X and Y each take on either the value 1 or −1, with E(X) = E(Y ) = 0. Show that

4
 a) pXY (1, 1) = pXY (−1, −1);
(Hint: what is E(X + Y )?)
b) pXY (1, −1) = pXY (−1, 1)
(Hint: what is E(X − Y )?)

Let p = 2pXY (1, 1). Find

c) Var(X);
(Answer: 1 )
d) Var(Y );
(Answer: 1)
e) Cov(X, Y )
(Answer: 2p − 1 )

2.8. Ascertain in the following cases whether or not F is the joint distribution function of some
pair (X, Y ) of random variables. If your conclusion is affirmative, find the distribution of X
and Y separately.

a)
1 − e−x−y if x, y ≥ 0

F (x, y) =
0 otherwise
b) 
 1 − e−x − xe−y if 0 ≤ x ≤ y
F (x, y) = 1 − e−y − ye−y if 0 ≤ y ≤ x
0 otherwise


∂2F
(Hint: Show that the second-order mixed partial derivative ∂x∂y
satisfies the conditions to be
a joint probability density function.)

2.9. Let f1 (x) and f2 (x) be density functions, and let a be a constant such that 0 ≤ a ≤ 1.
Consider the function f (x) = af1 (x) + (1 − a)f2 (x).

a) Show that f (x) is a density function. Such a density is often referred to as a mixture of
two densities. R
(Hint: Show that f (x) ≥ 0 for all x and f (x) dx = 1)
b) Suppose that Xi (i = 1, 2) is a random variable with density fi (x), and that E(Xi ) = µi
and Var(Xi ) = σi2 . Assume that X is random variable whose density is a mixture of the
densities corresponding to X1 and X2 .
i) Show that E(X) = aµ1 + (1 − a)µ2 ;
ii) Show that Var(X) = aσ12 + (1 − a)σ22 + a(1 − a)(µ1 − µ2 )2 .
R R
(Hint: write explicitly xf (x) dx and x2 f (x) dx)

2.10. You roll a conventional die repeatedly. If it shows 1, you must stop, but you may choose to
stop at any prior time. Your score is the number shown by the die on the final roll. Consider
the strategy: stop the first time that the die shows r or greater.

5
 a) What stopping strategy (i.e. what value of r) yields the greatest expected score?
b) What strategy would you use if your score were the square of the final roll?

(Answer: a) stop at the first throw showing 4 or more; b) stop at the first throw showing 5 or
more.)

2.11. A total of r keys are to be put, one

Pat a time, in k boxes, with each key independently being
k
put in box i with probability pi ( i=1 pi = 1). Each time a key is put in a nonempty box,
we say that a collision occurs.
PkFind the expected number of collisions.
r
(Answer: E(X) = r − k + i=1 (1 − pi ) )

2.12. Let X and Y be independent random variables, each taking the values −1 or 1 with proba-
bility 1/2, and let Z = X × Y . Show that X, Y and Z are pairwise independent. Are they
independent?
(Answer: - )

2.13. A merchant stocks a certain perishable item. She knows that on any given day she will have
a demand for either two, three, or four of these items with probabilities 0.1, 0.4 and 0.5,
respectively. She buys the items for $1 each and sells them for $1.20 each. If any are left
at the end of the day, they are lost. How many items should the merchant stock in order to
maximize her expected daily profit?
(Answer: 3 items should be stocked, E(profit) = 0.48 $)

2.14. Suppose that X takes on each of the values 1, 2, 3 with probability 1/3. What is the moment
generating function? Derive E(X), E(X 2 ) and E(X 3 ) by differentiating, and then compare
with a direct derivation of these moments.
(Answer: ϕX (t) = 13 (et + e2t + e3t ))

2.15. Calculate the moment generating function of the geometric distribution with parameter p.
Obtain E(X) and Var(X) by differentiating.
pet 1−p
(Answer: ϕX (t) = 1−(1−p)et for t < − log(1 − p), E(X) = 1/p, Var(X) = p2 )

2.16. With KX (t) = log ϕX (t) = log(E etX ), show that
0 00
KX (0) = E(X), KX (0) = Var(X).

The function KX is called the cumulant generating function of the random variable X, and
sometimes used instead of ϕX (t) to derive the interesting features of the distribution of X.
(Hint: Differentiate KX (t) and use the properties of ϕX (t))

2.17. a) Calculate the moment generating function of the Gamma distribution with parameters
(α, λ).
λ α


(Answer: ϕX (t) = λ−t , for t < λ)
b) Find the distribution of the sum of n i.i.d. random variables with distribution Exp(λ).
(Answer: Γ(n, λ))

6
2.18∗. A coin is tossed repeatedly and heads turns up on each toss with probability p. Let Hn and
Tn be the numbers of heads and tails in n tosses. Show that, for any  > 0,
 
1
P 2p − 1 −  ≤ (Hn − Tn ) ≤ 2p − 1 +  → 1 as n → ∞
n

(Hint: Use Chebyshev’s inequality and/or the Law of Large Numbers)

2.19∗. Generalise Exercise 2.6 and show that for an arbitrary continuous r.v. X,
Z 0 Z +∞
E(X) = − FX (x)dx + (1 − FX (x))dx
−∞ 0

if at least one of the two terms is finite. (If both terms are infinite, the expectation is unde-
fined, as for the Cauchy distribution).

2.20∗. Let X1 , X2 , . . . be a sequence of independent identically distributed continuous random vari-

ables. For n ≥ 2, define Xn as a record of the sequence if Xn > max(X1 , X2 , . . . , Xn−1 ),
that is, if Xn is larger than each of the values observed so far.

a) Find the probability that X2 is a record. Use symmetry to obtain the answer without
computation;
(Answer: 1/2)
b) Find the probability that Xn is a record, as a function of n. Again use symmetry.
(Answer: 1/n)
c) Find a simple expression for the expected number of records that occur over the first m
trials for any given integer m (X1 is not considered as a record). Show that this number
in the limit m → +∞.
is infinite P
(Answer: m k=2 1/k → ∞ as m → ∞)

Let N1 be the index of the first record in the sequence.

d) Find P(N1 > n) for each n ≥ 2.

(Answer: 1/n)
e) Find E(N1 ) (use Exercise 2.6). Contrast this result to part c). Comment and explain.
(Answer: +∞)

2.21∗. Calculate the moment generating function of the uniform distribution on [0, 1]. Obtain E(X)
and Var(X) by differentiating.
 et −1
if t 6= 0
(Answer: ϕX (t) = t , E(X) = ϕ0X (0) = 12 , Var(X) = ϕ00X (0)−(ϕ0X (0))2 =
1 if t = 0
1
12
)

2.22∗. A type of bacteria cell divides at a constant rate λ over time (that is, the probability that a
cell divides in a small interval of time t is approximately λt). Given that a population starts

7
 out at time zero with k cells of this bacteria and that cell divisions are independent of one
another, the size of the population at time t, say X(t), has the probability distribution
 
n − 1 −λkt
P(X(t) = n) = e (1 − e−λt )n−k , n = k, k + 1, . . .
k−1

(We will show this in Chapter 6)

P+∞ i+k

i 1
a) Find the expected value of X(t) in terms of λ, k and t. Hint: i=0 i
x = (1−x)k+1
for |x| < 1.
(Answer: E(X(t)) = keλt )
b) If, for a type of bacteria cell, λ = 0.1 per second and the population starts out with two
cells at time zero, find the expected value of the population after one minute.
(Answer: 806.9 bacterias)

2.23∗. The velocities V of gas particles can be modeled by the Maxwell distribution, whose proba-
bility density function is given by
r 
2 m 3/2 2 −v2 m
fV (v) = v e 2KT (v > 0),
π KT
where m is the mass of the particle, K is Boltzmann’s constant and T is the absolute tem-
perature. Find the mean
q velocity of these particles.
8KT
(Answer: E(V ) = πm
)

8
3 Conditional Probability and Conditional Expectation
3.1. Let X be Exponential with mean 1/λ. Find E(X|X > 1).
(Answer: 1 + 1/λ)

3.2. Let X be a random variable and suppose that p0 = P(X = 0) with 0 < p0 < 1. Let
µ = E(X) and σ 2 = Var(X). Find

a) E(X|X 6= 0);
(Answer: µ/(1 − p0 ))
b) Var(X|X 6= 0).
(Answer: σ 2 /(1 − p0 ) − p0 µ2 /(1 − p0 )2 )

3.3. An individual whose level of exposure to a certain pathogen is x will contract a disease with
probability p(x). If the exposure level of a randomly chosen member of the population has
density f , find the conditional density of the exposure level given that he or she

a) has the disease; R

(Answer: p(x)f (x)/ p(x)f (x)dx)
b) does not have the disease. R
(Answer: (1 − p(x))f (x)/ (1 − p(x))f (x)dx)

Also, show that c) when p(x) increases with x, then the ratio of the density of part a) to that
of part b) also increases in x.
(Answer: ratio ∼ p(x)/(1 − p(x)), increasing in x)

3.4. (Related to Example 3.13) A miner is trapped in a mine containing three doors. The first
door leads to a tunnel that takes him to safety after two hours of travel, the second door leads
to a tunnel that returns him to the mine after three hours of travel, and the third door leads to
a tunnel that returns him to his mine after five hours. Assume that the miner is at all times
equally likely to choose any one of the tunnels. Let N the number of doors selected by the
miner before he reaches safety. Also, let Ti denote the travel time corresponding to his ith
choice, i ≥ 1, and let X denote the time until the miner reaches safety.

that relates X to N and the Ti ’s;

a) Give an identityP
(Answer: X = N i=1 Ti )
b) Find E(N );
(Answer: 3)
c) Find E(TN );
(Answer: 2)
P  
N
d) Find E i=1 i N ;
T 
(Answer: 4N − 2)
e) Deduce E(X).
(Answer: 10)

3.5. A prisoner is trapped in a cell containing three doors. The first door leads to a tunnel that
returns him to his cell after two days of travel. The second leads to a tunnel that returns him

9
 to his cell after three days of travel. The third door leads immediately to freedom. Assuming
that the prisoner is always equally likely to choose among those doors that he has not yet
used, what is the expected number of days until he reaches freedom? Find the variance of
this number of days.
(Answer: E(N ) = 2.5 days, Var(N ) = 4.25 days2 )
3.6. A gambler wins each game with probability p. In each of the following cases, determine the
expected number of wins.
a) The gambler will play n games; if he wins X of these games, then he will play an addi-
tional X games before stopping;
(Answer: (1 + p)np)
b) The gambler will play until he wins, if it takes him Y games to get this win, then he will
play an additional Y games before stopping.
(Answer: 2)
3.7. Each element in a sequence of binary data is either 1 with probability p or 0 with probability
1 − p. A maximal subsequence of consecutive values having identical outcomes is called
a run. For instance, if the outcome sequence is 1,1,0,1,1,1, the first run is of length 2, the
second run is of length 1, and the third run is of length 3.
a) Find the expected length of the first run.
(Answer: p/(1 − p) + (1 − p)/p)
b) Find the expected length of the second run.
(Answer: 2)
3.8. Suppose that three constestants on a quiz show are each given the same question and that
each answers it correctly with some common probability, independently of the others. But
the difficulty of the question being itself random, let us suppose that the probability of cor-
rect answer is a random variable uniformly distributed over the interval [0, 1]. What is the
probability that exactly two of the contestants answer the question correctly? (Answer: 1/4)
3.9. The number of customers entering a store on a given day is Poisson distributed with mean
λ = 10. The amount of money (in $) spent by a customer is uniformly distributed over
[0, 100]. Find the mean and the standard deviation of the amount of money S that the store
takes in on a given day.
(Answer: E(S) = 500 $, st.dev.(S) = 182.6 $)
3.10. The number N of storms in the upcoming rainy season is Poisson distributed but with a mean
Λ that is uniformly distributed over [0, 5], that is, N |(Λ = λ) ∼ P(λ). Find the probability
there is at least one storm this season.
(Answer: 0.993)

3.11∗. Suppose that independent replications of an experiment, each of which is equally likely to
have any of m possible outcomes, are performed until the same outcome occurs i times in a
row. If Ni denote the required number of replications, show that
mi − 1
E(Ni ) =
m−1

10
 (Hint: Find an expression for E(Ni |Ni−1 ), and then use E(Ni ) = E(E(Ni |Ni−1 )).)

3.12∗. You have two opponents with whom you alternate play. Whenever you play A, you win with
probability pA ; whenever you play B, you win with probability pB , where pB > pA (A is
thus stronger than B). If your objective is to minimise the number of games you need to play
to win two times in a row (that is, to beat both of them in two successive games), should you
start playing against A or B?
(Answer: start with the strongest opponent, that is A)

3.13∗. There are n unstable molecules in a row, m1 , m2 , . . ., mn . One of the n − 1 pairs of

neighbours, chosen at random, combines to form a stable dimer; this process continues until
there remain Un isolated molecules no two of which are adjacent.

a) Show that the probability that m1 remains isolated is

n−1
X (−1)k
→ e−1 as n → ∞.
k=0
k!

(Hint: first condition on the first pair to combine and identify a recurrence.)
b) Show that limn→∞ n−1 E(Un ) = e−2 .
(Answer: - )

3.14∗. An individual travelling on the real line is trying to reach the origin. However, the larger the
desired step, the greater is the variance in the result of that step. Specifically, whenever the
person is at location x, he next moves to a location having mean 0 and variance βx2 . Let Xn
denote the position of the individual after having taken n steps. Supposing that X0 = x0 ,
find

a) E(Xn );
(Answer: 0)
b) Var(Xn ).
(Answer: β n x20 )

3.15∗. Aside from yourself, the number of people who are invited to a party is a Poisson random
variable with mean 10. The times at which people (including you) arrive are independent
Uniform over [0, 1] random variables.

a) Find the expected number of people who arrive before you.

(Answer: 5)
b) Find the probability that you are the first person to arrive.
(Answer: 0.099995)

3.16∗. Let Xi , i ≥ 1 be independent Uniform over [0, 1] random variables, and define N by

N = min{n : Xn < Xn−1 }

with some fixed value X0 = x, 0 < x ≤ 1. Let g(x) = E(N ).

11
 R 1 for g(x) by conditioning on X1 .
a) Derive an integral equation
(Answer: g(x) = 1 + x g(y) dy)
b) Differentiate both sides of the equation derived in part a).
(Answer: g 0 (x) = −g(x))
c) Solve the resulting equation to obtain g(x).
(Answer: g(x) = e1−x )

3.17∗. Let X and Y be independent non-negative random variables with continuous density func-
tions on [0, ∞). If, given X + Y = u, X is uniformly distributed on [0, u] whatever the value
of u, show that X and Y have an exponential distribution. (You may need the fact that the
only non-negative continuous solutions of the functional equation g(s + t) = g(s)g(t) for
all s, t ≥ 0, with g(0) = 1, are exponential functions of the form g(s) = ecs .)
(Hint: Start with writing fX|X+Y (v|u) and force this to be the U[0,u] distribution.)

12
4 Markov Chains
4.1. A Markov Chain {Xn , n ≥ 0} has the transition probability matrix
 
0.6 0.2 0.2
P =  0 0.4 0.6 .
0.5 0 0.5
Determine the conditional probabilities
a) P(X1 = 1, X2 = 1|X0 = 0);
(Answer: 0.08)
b) P(X2 = 1, X3 = 1|X0 = 0);
(Answer: 0.08)
c) P(X3 = 1|X0 = 0);
(Answer: 0.172)
d) P(X4 = 1, X3 = 1|X0 = 0).
(Answer: 0.0688)
If the initial value X0 is selected according to the distribution P(X0 = 0) = P(X0 = 1) =
0.5, determine
e) P(X2 = 1, X1 = 1, X0 = 0);
(Answer: 0.04)
f) P(X2 = 0);
(Answer: 0.38)
g) P(X3 = 0).
(Answer: 0.448)
4.2. A Markov Chain {Xn , n ≥ 0} with states 0, 1, 2 has transition probability matrix
 
1/2 1/3 1/6
P =  0 1/3 2/3 .
1/2 0 1/2
If P(X0 = 0) = P(X0 = 1) = 1/4, find E(X3 ).
(Answer: 0.98)
4.3. Specify the classes of the following Markov Chains, and determine whether they are transient
or recurrent:  
  0 0 0 1
0 1/2 1/2  0 0 0 1
P1 = 1/2 0 1/2
 P2 = 1/2 1/2 0 0

1/2 1/2 0
0 0 1 0
   
1/2 0 1/2 0 0 1/4 3/4 0 0 0
1/4 1/2 1/4 0 0  1/2 1/2 0 0 0
   
P3 = 1/2 0 1/2 0
 0 
 P4 =  0
 0 1 0 0
 0 0 0 1/2 1/2  0 0 1/3 2/3 0
0 0 0 1/2 1/2 1 0 0 0 0

13
 (Answer: - )
4.4. Show that if state i is recurrent and state i does not communicate with state j, then Pij = 0.
(Note: this implies that once a process enters a recurrent class of states it can never leave
that class. For this reason, a recurrent class is often referred to as a closed class.)
4.5. Consider a finite-state Markov Chain in which some given state, say State 1, is accessible
from every other state. Show that the chain has at most one recurrent class of states.
4.6. Suppose that X1 , X2 , . . . are i.i.d. integer-valued random variables with cumulative dis-
tribution function FX and probability mass function pX . Define Mn = max1≤i≤n Xi . Is
{Mn , n ≥ 1} necessarily a Markov Chain? If yes, give its transition probabilities. If no, give
a counterexample.
(Answer: - )
4.7. By definition, we know that a transition probability matrix P is such that the sum over each
row equals one, that is, X
Pij = 1, for all i.
j

Such a matrix is called a stochastic matrix. A transition probability matrix P is said to be

doubly stochastic if the sum over each column also equals one, that is,
X
Pij = 1
i

for all j. If a chain whose transition probability matrix is doubly stochastic, is irreducible
and aperiodic and consists of M + 1 states 0, 1, . . . , M , show that the stationary distribution
is given by
1
πj = , j = 0, 1, . . . , M.
M +1
(Hint: show that the suggested solution satisfies the required equations.)
4.8. Suppose that coin A has probability 0.7 of coming up heads, and coin B has probability 0.6
of coming up heads. If the coin flipped today comes up heads, then we select coin A to flip
tomorrow, and if it comes up tails, then we select coin B to flip tomorrow.
a) If the coin which is initially flipped is equally likely to be coin A or coin B, then what is
the probability that the coin flipped on the third day after the initial flip is coin A?
(Answer: 0.6665)
b) In the long run, what proportion of flips use coin A?
(Answer: 2/3))
4.9. Consider three urns, one coloured red, one white, and one blue. The red urn contains 1 red
and 4 blue balls, the white urn contains 3 white balls, 2 red balls and 2 blue balls, the blue
urn contains 4 white balls, 3 red balls and 2 blue balls. At the initial stage, a ball is randomly
chosen from the red urn and then returned to that urn. At every subsequent stage, a ball
is randomly selected from the urn whose colour is the same as that of the ball previously
selected. In the long run, what proportion of the selected balls are red? What proportion are
white? Blue?
(Answer: 0.281, 0.315, 0.404)

14
4.10. Three out of every four trucks on the road are followed by a car, while only one out of every
five cars is followed by a truck. What fraction of vehicles on the road are trucks?
(Answer: 4/19)

4.11. A professor continually gives exams to her students. She can give three possible types of
exams, and her class is graded as either having done well or badly. Let pi denote the proba-
bility that the class does well on a type i exam, and suppose p1 = 0.3, p2 = 0.6 and p3 = 0.9.
If the class does well on an exam, then the next exam is equally likely to be any of the three
types. If the class does badly, then the next exam is always of type 1. What proportion of
exams are type i (i = 1, 2, 3)?
(Answer: 5/7, 1/7, 1/7)

4.12. An individual possesses r umbrellas which he employs in going from his home to office and
vice-versa. If he is at home (the office) at the beginning (end) of a day and it is raining, then
he will take an umbrella with him to the office (home), provided there is one to be taken. If
it is not raining, then he does not take an umbrella. Assume that, independent of the past, it
rains at the beginning (end) of the day with probability p.

a) Define a Markov Chain with r + 1 states which will help to determine the proportion of
time that our man gets wet (that is, when it is raining and all umbrellas are at the other
location).
b) Show that the stationary distribution is given by
(
1−p
r+1−p
if i = 0
πi = 1
r+1−p
if i = 1, 2, . . . , r

c) What fraction of time does our man get wet?

(Answer: p(1−p)
r+1−p
)
d) When r = 3, what value of p maximises the fraction of time he gets wet?
(Answer: 0.536)

4.13. When a bus arrives at the main entrance of the UNSW campus, the next bus arrives in 2,
3, . . ., 21 minutes with equal probability. You arrive at the bus stop without checking the
schedule. How long, on average, should you wait till the next bus arrives?
(Answer: 6.69 min)

4.14. A particle moves among n + 1 vertices that are situated on a circle in the following manner.
At each step, it moves one step either in the clockwise direction with probability p or the
counter-clockwise direction with probability q = 1 − p. Starting at a specified state, call it
state 0, let T be the time of the first return to state 0. Find the probability that all states have
been visited by time T .
(Answer: (p − q)/(1 − (q/p)n ) + (q − p)/(1 − (p/q)n ))

4.15. Two players, A and B, play the game of matching pennies. At each time n, each player
has a penny and must secretly turn the penny to heads or tails. The players then reveal their
choices simultaneously. If the pennies match (both heads or both tails), Player A wins the
penny of B. If the pennies do not match (one heads and one tails), Player B wins the penny
of A. Suppose the player have between them a total of 5 pennies. If at any time one player

15
 has all of the 5 pennies, to keep the game going, he gives one back to the other player and
the game will continue.
a) Show that this can be formulated as a Markov Chain.
b) Is the chain ergodic?
(Answer: no)
c) If Player A starts with 3 pennies and Player B with 2, what is the probability that A will
lose his pennies first?
(Answer: 0.4)
4.16. Consider a branching process having E(Y ) = µ < 1. Show that if X0 = 1, then the expected
number of individuals that ever exist in this population is given by 1/(1−µ). What if X0 = k
(k > 1)?
(Answer: k/(1 − µ))
4.17. For a branching process, calculate the probability of extinction π0 when the number Y of
offspring of each individual follows the following distributions:
a) P(Y = 0) = 1/4, P (Y = 1) = 1/2, P (Y = 2) = 1/4;
(Answer: 1)
b) P(Y = 0) = 1/4, P (Y = 2) = 3/4;
(Answer: 1/3)
c) P(Y = 0) = 1/6, P (Y = 1) = 1/2, P (Y = 2) = 1/3.
(Answer: 1/2)
4.18. Let {X0 , X1 , X2 , . . .} be a branching process, where Xn denotes the number of individuals
born at time n, and X0 = 1. Let Y be the number of offspring of a given individual, and
suppose that Y ∼ Bin(3, 1/2).
a) Find the probability generating function of Y .
(Answer: G(s) = (s3 + 3s2 + 3s + 1)/8)
b) Find the probability that the process is extinct at the second generation.
(Answer: 0.178)
c) Find the probability of eventual extinction.
(Answer: 0.236)
d) Suppose X2 = 3. Find the probability of eventual extinction.
(Answer: 0.013)

4.19∗. An airline reservation system has two computers, only one of which is in operation at any
given time. A computer may break down on any given day with probability p. There is a
single repair facility that takes two days to restore a computer to normal. The facilities are
such that only one computer at a time can be dealt with.
a) Form a Markov Chain by taking as states the pairs (x, y), where x is the number of
machines in operating condition at the end of a day and y is 1 if a day’s labour has been
expended on a machine not yet repaired and 0 otherwise.
(Answer: the transition probabilities are P(2,0),(2,0) = 1 − p, P(2,0),(1,0) = p, P(1,0),(1,1) =
1 − p, P(1,0),(0,1) = p, P(1,1),(2,0) = 1 − p, P(1,1),(1,0) = p, P(0,1),(1,0) = 1.)

16
 b) What is the long-run proportion of time that both machines are inoperative?
(Answer: p2 /(1 + p2 ).)

4.20∗. A simplified model for the spread of a rumour goes this way. There are N = 5 people in
a group of friends, of which some have heard about the rumour and the others have not.
During any single period of time, two people are selected at random from the group and
assumed to interact. The selection is such that an encounter between any pair of friends is
just as likely as between any other pair. If one of these persons has heard the rumour and the
other has not, then with probability α = 0.7 the rumour is transmitted. Let Xn denote the
number of friends who have heard the rumour at the end of the nth period. Assuming that
the process begins at time 0 with a single person knowing the rumour, what is the mean time
that it takes for everyone to hear it?
(Answer: 11.9 periods.)

4.21∗. Each morning an individual leaves his house and goes for a run. He is equally likely to leave
either from his front or back door. Upon leaving the house, he chooses a pair of running
shoes (or goes running barefoot if there are no shoes at the door from which he departed).
On his return he is equally likely to enter, and leave his running shoes, either by the front or
back door. If he owns a total of k pairs of running shoes, what proportion of the time does
he run barefooted?
1
(Answer: k+1 )

4.22∗. A classical mathematical description of diffusion of molecules through a membrane is the

famous Ehrenfest model, originally introduced in 1907 by Paul and Tatiana Ehrenfest as the
‘dog-flea model’. Suppose that M molecules are distributed among two containers A and
B; and at each time point one of the M molecules is chosen at random, removed from its
container, and placed in the other one. Let the state of the system after the nth transition be
Xn the number of molecules in container A. Clearly, {Xn , n ≥ 0} is a Markov-Chain on the
state space {0, 1, 2, . . . , M }, with transition probabilities
i i
Pi,i+1 = 1 − , Pi,i−1 = , for i = 0, 1, . . . , M.
M M
a) Is this chain ergodic? Explain.
(Answer: No.)
b) Show that the stationary distribution of this chain is the Binomial distribution Bin(M, 1/2).
M


(Hint: show that the probabilities πj = j (1/2)M , for j = 0, 1, . . . , M , satisfy the re-
quired stationary equations.)
c) Deduce the long term average of molecules in container A.
(Answer: limn→∞ E(Xn ) = M/2.)

Note: The Ehrenfest model has been used to explain the second law of thermodynamics,
namely that the entropy of the system is monotonically increasing. This is actually a con-
sequence of the convergence of the distribution of balls in both urns towards its stationary
limit.

4.23∗. In the gambler’s ruin problem, suppose that the gambler’s fortune is presently i, and suppose
that we know that the gambler’s fortune will eventually reach N (before it goes to 0). Given

17
 that information, show that the probability he wins the next gamble is
(
p(1−(q/p)i+1 )
P = 1−(q/p)i
if p 6= 1/2
i+1
2
if p = 1/2

(Hint: Write P = P(Xn+1 = i + 1|Xn = i, eventual win) and use Baye’s rules.)
4.24∗. A coin is tossed repeatedly, heads turning up with probability p on each toss. A gambler
starts with initial fortune i (where 0 < i < N ). He wins $1 for each head and loses $1 for
each tail. If his fortune is ever 0, he is bankrupted and stops playing. If it ever reaches $N
he stops gambling to buy a Jaguar. Suppose that p < 1/2.
a) Show that the gambler can increase his chance of winning by doubling the stakes. For
simplicity, you may assume that i and N are even.
b) What is the corresponding strategy if p ≥ 1/2? If p = 1/2?
(Answer: - )
4.25∗. Suppose {Xn , n ≥ 0} is a branching process, with X0 = 1 and Y the number of offspring
of a given individual following the distribution

0 with probability 1 − p
Y = ,
1 with probability p
for some p ∈ (0, 1). As E(Y ) < 1, the extinction is definite. Find the expected time to
extinction.
(Answer: 1/(1 − p))
4.26∗. Let Xn be the size of the nth generation in an ordinary branching process, with X0 = 1. The
distribution of the number of offspring Y is such that E(Y ) = µ and Var(Y ) = σ 2 > 0.
a) Show that, for m ≤ n,
E(Xn Xm ) = µn−m E(Xm
2
).
(Hint: start with E(Xn |Xm ))
b) Deduce the covariance
 mbetween
 Xm and Xn in terms of µ and σ 2 .
(Answer: µn−1 σ 2 1−µ
1−µ
if µ 6= 1 and mσ 2 if µ = 1.)

4.27∗. Each generation of a branching process (with X0 = 1) is augmented by a random number of

immigrants who are indistinguishable from the other members of the population. Suppose
that the number of immigrants in different generations are independent of each other and
of the past history of the process, each such number having probability generating function
H(s). Show that the probability generating function Gn of the size of the nth generation
satisfies
Gn+1 (s) = Gn (G(s)) × H(s),
where G is the probability generating function of the number of offspring.
(Hint: Write Xn+1 = Cn+1 + In+1 , where Cn+1 is the number of natural offspring of the
previous generation, and In+1 is the number of immigrants, and find the pgf of this random
variable.)

18
5 The Exponential Distribution and the Poisson Process
5.1. The lengths, in inches, of cotton fibres used in a certain mill are exponentially distributed
random variables with parameter λ. It is decided to convert all measurements in this mill
to the metric system. Describe the probability distribution of the length, in centimetres, of
cotton fibres in this mill. Recall that 1 inch = 2.54 cm.
(Answer: Y ∼ Exp(λ/2.54).)

5.2. Suppose that you arrive at a single-teller bank to find five other customers in the bank, one
being served and the other four waiting in line. You join the end of the line. If the service
times are all exponential with rate µ, what is the expected amount of time you will spend in
the bank?
(Answer: 6/µ )

5.3. Machine 1 is currently working. Machine 2 will be put in use at a time t from now. If
the lifetime of machine i is exponential with rate λi (i = 1, 2), what is the probability that
machine 1 is the first to fail?
(Answer: 1 − λ1λ+λ
2
2
e−λ1 t )

5.4. One hundred items are simultaneously put on a life test. Suppose that the lifetime of the
individual items are independent exponential random variables with mean 200 hours. The
test will end when there have been a total of 5 failures. If T is the time at which the test ends,
find E(T ) and Var(T ).
(Answer: E(T ) = 10.21 (hours), Var(T ) = 20.84 (hours2 ))

5.5. A flashlight needs two batteries to be operational. Consider such a flashlight along with a set
of n functional batteries – battery 1, battery 2, . . ., battery n. Initially, battery 1 and battery
2 are installed. Whenever a battery fails, it is immediately replaced by the lowest numbered
functional battery that has not yet been put in use. Suppose that the lifetimes of the batteries
are i.i.d. exponentially distributed with parameter µ. At a random time, call it T , a battery
will fail and our stockpile will be empty. At that moment exactly one of the batteries – say,
battery X – will not yet have failed.

a) Find P(X = n);

(Answer: 1/2)
b) Find P(X = 1);
(Answer: 1/2n−1 )
c) Find P(X = i);
(Answer: 1/2n−i+1 )
d) Find E(T );
(Answer: n−1
2µ
)
e) What is the distribution of T ?
(Answer: Γ(n − 1, 2µ))

5.6. A doctor has scheduled two appointments, one at 1.00 pm and the other at 1.30 pm. The
amounts of time that appointments last are independent exponential random variables with
mean 30 minutes. Assuming that both patients are on time, find the expected amount of time

19
 that the 1.30 appointment spends at the doctor’s office.
(Answer: 41.04 minutes)

5.7. Let {N (t), t ≥ 0} be a Poisson process with rate λ. Let Sn denote the time of the nth arrival.
Find

a) E(S4 );
(Answer: 4/λ)
b) E(S4 |N (1) = 2);
(Answer: 1 + 2/λ)
c) E(N (4) − N (2)|N (1) = 3).
(Answer: 2λ)

5.8. Let {N (t); t ≥ 0} be a Poisson process of rate λ.

a) Find the joint probability mass function of N (t) and N (t + s) for s > 0;
m tn sm−n
(Answer: P(N (t) = n, N (t + s) = m) = e−λ(t+s) λn!(m−n)! for m ≥ n.)
b) Find Cov (N (t), N (t + s)) for s > 0;
(Answer: λt)
c) Find Cov (N (t3 ) − N (t1 ), N (t4 ) − N (t2 )), where 0 < t1 < t2 < t3 < t4 .
(Answer: λ(t3 − t2 ))

5.9. The water level of a certain reservoir is depleted at a constant rate of 1000 units daily. The
reservoir is refilled by randomly occurring rainfalls. Rainfalls occur according to a Poisson
process with rate 0.2 per day. The amount of water added to the reservoir by a rainfall is
5000 units with probability 0.8 and 8000 units with probability 0.2. The present water level
is just slightly below 5000 units. What is the probability that the reservoir will be empty
after five days?
(Answer: 0.37 )

5.10. The index parameter t in a Poisson process {N (t), t ≥ 0} is often thought of as the time. It is
not necessary the case, though. Here is an example of a ‘spatial’ Poisson process. Obviously,
the statistical properties of the process are not affected by its physical nature.

Defects occur along an undersea cable according to a Poisson process of rate 0.1 per kilo-
metre.

a) What is the probability that no defects appear in the first two kilometres of cable?
(Answer: 0.8187)
b) Given that there are no defects in the first two kilometres of cable, what is the probability
of no defects between km 2 and 3?
(Answer: 0.9048)

5.11. Cars pass a certain street location according to a Poisson process with rate λ. A woman who
wants to cross the street at that location waits until she can see that no cars will come by in
the next τ time units.

20
 a) Find the probability that her waiting time is 0;
(Answer: e−λτ )
b) Find her expected waiting time.
λ 0τ xe−λx dx
R
(Answer: 1−λ R τ e−λx dx )
0

5.12. Teams 1 and 2 are playing a match. The teams score points according to independent Poisson
processes with respective rates λ1 and λ2 . If the match ends when one of the teams have
scored k more points than the other, find the probability that team 1 wins.
(Answer: 1+(λ21/λ1 )k )

5.13. Events occur according to a Poisson process with rate λ. Each time an event occurs, we must
decide whether or not to stop, with our objective being to stop at the last event to occur prior
to some specified time τ , where τ > 1/λ. That is, if an event occurs at time t, 0 ≤ t ≤ τ ,
and we decide to stop, then we win if there are no additional events by time τ , and we lose
otherwise. If we do not stop when an event occurs and no additional events occur by time
τ , then we lose. Also, if no events occur by time τ , then we lose. Consider the strategy that
stops at the first event to occur after some fixed time s, 0 ≤ s ≤ τ .

a) Using this strategy, what is the probability of winning?

(Answer: λ(τ − s)e−λ(τ −s) )
b) What value of s maximises the probability of winning?
(Answer: s = τ − 1/λ)
c) Show that one’s probability of winning when using the preceding strategy with the opti-
mal value of s is 1/e.
(Answer: - )

5.14. Customers enter a store according to a Poisson process of rate λ = 6 per hour. Suppose it is
known that only a single customer entered during the first hour. What is the probability that
this person entered during the first 15 min?
(Answer: 0.25)

5.15. Satellites are launched into space at times distributed according to a Poisson process with
rate λ. Each satellite independently spends a random time (having distribution G) in space
before falling to the ground. Find the probability that none of the satellites in the air at time
t was launchedR before time s, where s < t.
s
(Answer: e−λ 0 (1−G(t−y)) dy )

5.16. A security consultant has observed that the attempts to breach the security of the company’s
computer system occurs according to a Poisson process with a mean rate of 2.2 attempts per
day. The system is on 24 hours a day.

a) What is the probability that there will be 4 breach attempts tomorrow?

(Answer: 0.1082)
b) What is the probability that there will be no breach attempts during the evening (8-hour)
shift?
(Answer: 0.4803)

21
 c) It is now midnight and the most recent attempt occurred at 10.30pm. What is the proba-
bility that the next attempt will occur before 5.30am?
(Answer: 0.396)

It has been observed that 10% of the attempts to illegally access the computer make it past
the first security level, independently of the other attempts.

d) What is the probability that there will be exactly three attempted illegal entries that suc-
cessfully make it past the first security level during the next week?
(Answer: 0.1305)
e) It is now 6am on Monday morning. The most recent security attempt that successfully
made it past the first security level occurred at midnight last night. What is the probability
that no illegal attempts make it past the first security level during the time period from
now through 6pm Friday?
(Answer: 0.3716)

5.17. In good years, storms occur according to a Poisson process with rate 3 per unit time, while
in other years they occur according to a Poisson process with rate 5 per unit time. Suppose
next year will be a good year with probability 0.3. Let N (t) denote the number of storms
during the first t time units of next year.

a) Find P(N (t) = n);

n n
(Answer: 0.3 × e−3t (3t)
n!
+ 0.7 × e−5t (5t)
n!
)
b) Is {N (t), t ≥ 0} a Poisson process?
(Answer: No.)
c) Does {N (t), t ≥ 0} have stationary increments? Why or why not?
(Answer: Yes.)
d) Does it have independent increments? Why or why not?
(Answer: No.)
e) If next year starts off with three storms by time t = 1, what is the conditional probability
it is a good year?
(Answer: 0.1857)

5.18. Customers arrive at the automatic teller machine in accordance with a Poisson process with
rate 12 per hour. The amount of money withdrawn on each transaction is a random variable
with mean $30 and standard deviation $50. (A negative withdrawal means that money was
deposited.) The machine is in use for 15 hours daily. Approximate the probability that the
total daily withdrawal is less than $6,000.
(Answer: 0.78, using the Central Limit Theorem)

5.19. Policyholders of a certain insurance company have accidents at times distributed according
to a Poisson process with rate λ. The amount of time from when an accident occurs until a
claim is made has distribution G.

a) Find the probability nthere are exactly nRincurred but as yet unreported claims at time t.
t
(Answer: e−a(t) (a(t))
n!
, where a(t) = λ 0 (1 − G(s)) ds)

22
 b) Suppose that each claim amount has distribution F (with mean µF ), and that the claim
amount is independent of the time that it takes to report the claim. Find the expected
value of the sum of all incurred but as yet unreported claims at time t.
(Answer: a(t) × µF )

5.20. Let S(t) denote the price of a security at time t. A popular model for the process {S(t), t ≥
0} supposes that the price remains unchanged until a “shock” occurs, at which time the price
is multiplied by a random factor. If we let N (t) denote the number of shocks by time t, and
let Xi denote the ith multiplicative factor, then this model supposes that
N (t)
Y
S(t) = S(0) Xi
i=1

(where the product is equal to 1 when N (t) = 0). Suppose that the Xi are independent
exponential random variables with rate µ; that {N (t), t ≥ 0} is a Poisson process with rate
λ; that {N (t), t ≥ 0} is independent of all Xi ; and that S(0) = s. Find E(S(t)).
(Answer: se−λt+λt/µ )

5.21∗. In a certain system, a customer must first be served by server 1 and then by server 2. The
service times at server i are exponential with rate µi (i = 1, 2). An arrival finding server 1
busy waits in line for that server. Upon completion of service at server 1, a customer either
enters service with server 2 if that server is free or else remains with server 1 (blocking any
other customer from entering the service) until server 2 is free. Customers depart the system
after being served by server 2.

a) Suppose that when you arrive there is one customer in the system and that customer is
being served by server 1. What is the expected total time you spend in the system?
(Answer: µ21 + µ22µ(µ11+µ 2
+µ2 )
)
b) Suppose now that you arrive to find two others in the system, one being served by server
1 and the other by server 2. What is the expected time you spend in the system?
(Answer: µ21 + µ22 µ1µ+µ
1
2
+ µ12 )

5.22∗. Let {N (t), t ≥ 0} be a Poisson process of rate λ. Verify that

P(N (t) is odd) = e−λt sinh(λt)

and
P(N (t) is even) = e−λt cosh(λt).
(Hint: What is the probability that N (t) = 2n + 1? Compare this to the nth term of the
Taylor expansion of sinh(λt)).)

5.23∗. Consider a two-server parallel queueing system where customers arrive according to a Pois-
son process with rate λ, and where the service times are exponential with rate µ. Moreover,
suppose that arrivals finding both servers busy immediately depart without receiving any
service (such a customer is said to be lost), whereas those finding at least one free server
immediately enter service and then depart when their service is completed.

23
 a) If both servers are presently busy, find the expected time until the next customer enters
the system;
1
(Answer: 2µ + λ1 )
b) Starting empty, find the expected time until both servers are busy;
(Answer: 2λ+µ
λ2
)
c) Find the expected time between two successive lost customers.
(Answer: λ1 + µ(λ+µ)
λ3
)

5.24∗. Let {N (t), t ≥ 0} be a Poisson process of rate λ, and consider XN (t)+1 the interarrival time
between the last arrival before time t and the first arrival after time t. Show that the density
of XN (t)+1 is  2 −λx
λ xe if 0 ≤ x < t
fXN (t)+1 (x) = .
λ(1 + λt)e−λx if x ≥ t
(Hint: from time t, think also of what happens moving backward in time.)

5.25∗. In Exercise 5.10, it has been noted that a Poisson process may also model a spatial phe-
nomenon. Here is a generalisation of the idea to higher dimensions. A two-dimensional
Poisson process is a process of randomly occurring events in the plane such that
(i) for any region of area A the number of events in that region, say N (A), has a Poisson
distribution with mean λA, and
(ii) the number of events in non-overlapping regions are independent.
For such a process, consider an arbitrary point in the plane and let X denote its distance from
its nearest event (where distance is measured in the usual Euclidean manner). Show that
2
a) P(X > t) = e−λπt ;
(Hint: write the event (X > t) in terms of N (C(0, t)), where C(0, t) is the disk of radius
t centred at 0)
b) E(X) = 2√1 λ
(Hint: use Exercise 2.18.)

This kind of process can model the number of plants of a certain species in a given field, for
instance, or the number of flaws in a newly produced aluminium sheet.

5.26∗. Suppose that people arrive at a bus stop in accordance with a Poisson process with rate λ.
The bus departs at time t. Let X denote the total amount of waiting time of all those who get
on the bus at time t. We want to determine Var(X). Let N (t) denote the number of arrivals
by time t.

a) What is E(X|N (t))?

(Answer: tN2(t) )
N (t)t2
b) Show that Var(X|N (t)) = 12
;
(Answer: -)
c) Find Var(X).
3
(Answer: λt3 )

24
5.27∗. Suppose that electrical shocks having random amplitudes occur at times distributed accord-
ing to a Poisson process {N (t), t ≥ 0} with rate λ. Suppose that the amplitudes of the suc-
cessive shocks are independent both of other amplitudes and of the arrival times of shocks,
and also that the amplitudes have distribution F with mean µ. Suppose also that the am-
plitude of a shock decreases with time at an exponential rate α, meaning that an initial
amplitude A will have value Ae−αx after an additional time x has elapsed. Let A(t) denote
the sum of all amplitudes at time t. That is,
N (t)
X
A(t) = Ai e−α(t−Si )
i=1

where Ai and Si are the initial amplitude and the arrival time of shock i. Find E(A(t)) by
first conditioning on N (t).
(Answer: αλ µ(1 − e−αt ))

5.28∗. Let Sn denote the time of the nth event of the Poisson process {N (t), t ≥ 0} having rate
λ. Show, for an arbitrary function g, that the random variable N
P (t)
PN i=1 g(Si ) has the same
distribution as the compound Poisson random variable i=1 g(Ui ), where U1 , U2 , . . . is a
sequence of independent and identically distributed uniform (0, t) random variables that is
independent of N , a Poisson random variable with mean λt. Consequently, conclude that
   
N (t) Z t N (t) Z t
X X
E g(Si ) = λ g(x) dx Var  g(Si ) = λ g 2 (x) dx.
i=1 0 i=1 0

(Answer: - )

25
6 Continuous-Time Markov Chains
6.1. Consider two machines that are maintained by a single repairman. Machine i functions for
an exponential time with rate λi before breaking down, i = 1, 2. The repair times (for either
machine) are exponential with rate µ. Can we analyse this as a birth and death process? If
so, what are the parameters? If not, how can we analyse it?
(Answer: it is not a birth and death process, but a more general continuous-time Markov
Chain for which you can define the relevant parameters (transition probabilities Pij , rates vi ,
instantaneous rates qij ))
6.2. Potential customers arrive at a single-server station in accordance with a Poisson process
with rate λ. However, if the arrival finds n customers already in the station, then he will
enter the system with probability αn . Assuming an exponential service rate µ, set this up as
a birth and death process and determine the birth and death rates.
(Answer: λn = λαn , µn = µ)
6.3. There are N individuals in a population, some of whom have a certain infection that spreads
as follows. Contacts between two members of this population occur in accordance with a
Poisson
process having rate λ. When a contact occurs, it is equally likely to involve any of
the N2 pairs of individuals in the population. If a contact involves an infected and a non-
infected individual, then with probability p the non-infected individual becomes infected.
Once infected, an individual remains infected throughout. Let X(t) denote the number of
infected members of the population at time t.
a) Is {X(t), t ≥ 0} a continuous-time Markov Chain? Specify its type and its parameters.
−i)
(Answer: pure birth process with λi = λp N2i(N
(N −1)
for i = 1, . . . , N )
b) Starting with a single infected individual, what is the expected time until all members are
infected?
−1) PN −1
(Answer: N (N
2λp
1
i=1 i(N −i) )

6.4. Suppose that a one-celled organism can be in one of two states – either A or B. An indi-
vidual in state A will change to state B at an exponential rate α; an individual in state B
divides into two new individuals of type A at an exponential rate β. Define an appropriate
continuous-time Markov chain for a population of such organisms and determine the appro-
priate parameters for this model. (Hint: define the state of the chain as the couple (number
of A-individuals, number of B-individuals).
(Answer: q(n,m),(n−1,m+1) = nα, q(n,m),(n+2,m−1) = mβ)
6.5. A population of organisms consists of both male and female members. In a small colony
any particular male is likely to mate with any particular female in any time interval of length
h, with probability λh + o(h). Each mating immediately produces one offspring, equally
likely to be male or female. Let N1 (t) and N2 (t) denote the number of males and females in
population at time t. Derive the parameters vi and Pij of the continuous-time Markov Chain
{(N1 (t), N2 (t)), t ≥ 0}.
(Answer: q(n,m),(n+1,m) = q(n,m),(n,m+1) = 21 λnm)
6.6. The birth and death process with parameters λn ≡ 0 and µn = µ, n > 0 is called a pure
death process. Find Pij (t) for any i, j.
i−j
(Answer: Pij (t) = e−µt (µt)
(i−j)!
for 0 < j ≤ i)

26
 6.7. Consider a Yule process {X(t), t ≥ 0} with birth parameter λ starting with a single individ-
ual – that is, suppose X(0) = 1. Suppose that you observe the process at a random time T ,
where T ∼ U[0,1] . Show that

(1 − e−λ )j
P(X(T ) = j) = ,
λj
for j = 1, 2, . . ..
(Hint: start from the expression of P1j (t)) in Example 6.8 and find and expression for
P(X(T ) = j) by first conditioning on T . Set u = 1 − e−λt to solve the integral.)

6.8. A new product (a “Home Helicopter” to solve the commuting problem, say) is being intro-
duced. The sales are expected to be determined by both media (newspaper and television)
advertising and word-of-mouth advertising, wherein satisfied customers tell others about the
product. Assume that media advertising creates new customers according to a Poisson pro-
cess of rate α = 1 customer per month. For the word-of-mouth advertising, assume that each
purchaser of a Home Helicopter will generate sales to new customers at an exponential rate
of θ = 2 customers per month. Let X(t) be the total number of Home Helicopter customers
up to time t.

a) Model {X(t), t ≥ 0} as a pure birth process by specifying the birth parameters λk , for
k = 0, 1, 2, . . ..
(Answer: λk = 1 + 2k)
b) What is the probability that exactly two Home Helicopters are sold during the very first
month of the advertising campaign?
(Answer: 0.1031)

6.9. A chemical solution contains N molecules of type A and an equal number of molecules
of type B. A reversible reaction occurs between type A and B molecules in which they
bond to form a new compound AB. Suppose that in any small time interval of length h,
any particular unbound A molecule will react with any particular unbound B molecule with
probability αh + o(h), where α is a reaction rate of formation. Suppose also that in any
small time interval of length h, any particular AB molecules disassociates into its A and
B constituents with probability βh + o(h), where β is a reaction rate of dissolution. Let
X(t) be the number of AB molecules at time t. Model X(t) as a birth and death process by
specifying the parameters.
(Answer: λi = α(N − i)2 , µi = βi)

6.10. Let {X(t), t ≥ 0} be an asymmetric simple random walk in continuous-time on the integers,
so that, for any i ≥ 1, 
λ if j = i + 1
qij =
µ if j = i − 1
Suppose that λ > µ and X(0) = 0. Show that the total time spent in any state r > 0 is
Exponentially distributed with parameter λ − µ.
(Hint: it can be shown that, for a discrete-time random walk starting from 0 with p > 1/2,
the number of times that any state i > 0 is visited is Geometrically distributed with parameter
2p − 1. Then, recall Property 5, Slide 324.)

27
6.11. Let {X(t), t ≥ 0} be a continuous-time Markov Chain on the states 0, 1, 2, 3 with an em-
bedded Markov Chain having a transition matrix
 
0 0.1 0.2 0.7
0 0 0.4 0.6
P= 0.8 0.1 0 0.1


1 0 0 0

The mean times spent in states 0, 1, 2 and 3 for each visit are respectively 0.5, 0.2, 2 and 1.
Give the generator matrix G for this continuous-time Markov Chain.
(Answer: - )

6.12. An airline reservation system has 2 computers, one on-line and one backup. The operating
computer fails after an exponentially distributed time with parameter µ and is then replaced
by the backup computer. There is one repair facility and the repair time is exponentially
distributed with parameter λ. Let X(t) be the number of computers in operating condition
at time t. Write down the generator G of {X(t), t ≥ 0} and the Kolmogorov’s backward
equations. Answer the same question in the case where the 2 machines are simultaneously
online if they are
 in operating conditions.
  
−λ λ 0 −λ λ 0
(Answer: G =  µ −(λ + µ) λ , G =  µ −(λ + µ) λ )
0 µ −µ 0 2µ −2µ
6.13. A lion is always in one of the three states sleeping, hunting and eating. This lion always
goes from sleeping to hunting to eating and back to sleeping. On average, the lion sleeps for
3 hours, hunts for 2 hours, and eats for 30 minutes. Assuming the lion remains in each state
for an exponential time, model the lion’s life as a continuous time Markov chain. What are
the respective proportions of time that this lion is sleeping, hunting and eating?
(Answer: π0 = 0.55, π1 = 0.36, π2 = 0.09)

6.14. A soccer player’s career is marred by the fact that he is injury prone. He fluctuates be-
tween three states: 0 (fit), 1 (minor injury not preventing competition) and 2 (major injury
requiring complete abstinence from the field). The team doctor, upon observing the transi-
tions between states, concluded these transitions could be modelled by a Markov Chain with
transition matrix  
0 1/3 2/3
P = 1/3 0 2/3 .
1 0 0
The time spent in states 0, 1, 2 are Exponentially distributed with parameters 1/3, 1/3 and
1/6, respectively. Let {X(t), t ≥ 0} be the continuous-time Markov Chain describing the
state of health of the player over time. Assume the time units are days.

a) What is the generator

 matrix of this chain?

−1/3 1/9 2/9
(Answer: G =  1/9 −1/3 2/9 )
1/6 0 −1/6
b) What is the long run proportion of time that the player has to abstain from playing?
(Answer: 57.1%)

28
 c) Suppose he is paid $40/day when fit, $30/day when able to play injured and $20/day
when injured and unable to play. What is his long run average earn rate?
(Answer: 27.5 $/day)

6.15. A small barbershop, operated by a single barber, has room for at most two customers. Po-
tential customers arrive at a Poisson rate of three per hour, and the successive service times
are independent exponential random variables with mean 1/4 hour. What is

a) the average number of customers in the shop?

(Answer: 0.81)
b) the proportion of potential customers that enter the shop?
(Answer: 0.76)

If the barber could work twice as fast, how much more business would he do?
(Answer: 0.45 customer more per hour)

6.16. A service center consists of two servers, each working at an exponential rate of two services
per hour. If customers arrive at a Poisson rate of three per hour, then, assuming a system of
capacity of at most three customers,

a) what fraction of potential customers enter the system?

(Answer: 0.81)
b) what would the value of a) be if there was only a single server, and his rate was twice as
fast (that is, µ = 4)?
(Answer: 0.85)

6.17. At a small company there is a telephone exchange with 3 lines. Calls arrive according to a
Poisson process with intensity 0.5 calls per minute and last an exponentially distributed time
with mean 4 minutes. The lengths of different calls are independent of each other and of the
arrival process. When all lines are busy the exchange is blocked and new calls are rejected.
Consider the exchange has been running for an infinitely long time.

a) Determine the proportion of time that the exchange is blocked.

(Answer: 0.21)
b) The staff at the company feel that blocking occurs too often, and consider increasing the
number of lines so that the blocking proportion of time gets lower than 5%. How many
lines do they need?
(Answer: at least 5)
c) Assuming that infinitely many telephone lines are available, find the long-term distribu-
tion of the number of lines in use at a given time.
(Answer: P(2))

6.18. Each time a machine is repaired it remains up for an exponentially distributed time with rate
λ. It then fails, and its failure is either of two types. If it is a type 1 failure, then the time to
repair the machine is exponential with rate µ1 . If it is a type 2 failure, then the repair time
is exponential with rate µ2 . Each failure is, independently of the time it took the machine
to fail, a type 1 failure with probability p and a type 2 failure with probability 1 − p. What
proportion of time is the machine down due to a type 1 failure? What proportion of time is

29
 it down due to a type 2 failure? What proportion of time is it up?
 −1
(Answer: π0 = 1 + µpλ1 + (1−p)λ
µ2
, π1 = µpλ1 π0 , π2 = λ(1−p)
µ2
π0 )

6.19. Customers arrive at a single-server queue in accordance with a Poisson process having rate
λ. However, an arrival that finds n customers already in the system will join the queue with
probability 1/(n + 1). Show that the limiting distribution of the number of customers in the
system is Poisson with mean λ/µ.
(Answer: - )
6.20. A job shop consists of three machines and two repairmen. The amount of time a machine
works before breaking down is exponentially distributed with mean 10. If the amount of
time it takes a single repairman to fix a machine is exponentially distributed with mean 8,
then
(a) what is the average number of machines not in use?
(Answer: 1.40 )
(b) what proportion of time are both repairmen busy?
(Answer: 0.44 )
6.21. Consider a taxi station where taxis and customers arrive in accordance with Poisson pro-
cesses with respective rates of one and two per minute. A taxi will wait no matter how many
taxis are present. However, an arriving customer that does not find a taxi waiting leaves.
Find
a) what is the average number of taxis waiting
(Answer: 1 )
b) the proportion of arriving customers that get a taxi.
(Answer: 1/2 )
6.22. Consider a population for which new members are born according to a Poisson process with
constant rate λ (the birth rate does not depend on the population size). After a random
amount of time, which is independent of the population size, a disaster occurs and the whole
population immediately dies out. Then the population starts to grow all over again, that is,
after an exponential time with rate λ the first member is born, and so on until the next disaster
occurs. Disasters occur according to a Poisson process with rate µ.
a) Find the long-run n of time the population consists of n members.
 fraction
λ µ
(Answer: πn = λ+µ × λ+µ )
b) What is the long-run average population size?
(Answer: µλ )
c) Determine the mean population size just before a disaster.
(Answer: 1 + µλ )

6.23∗. Show that not every discrete-time Markov Chain can be embedded in a continuous-time
chain. More precisely, let  
α 1−α
P= ,
1−α α

30
 for some α ∈ (0, 1), be a transition matrix.
a) Show that there exists a continuous-time Markov Chain with transition probability func-
tion matrix P(t), with embedded Markov Chain with transition matrix P, if and only if
α ∈ (1/2, 1). Note that saying that the embedded Markov Chain has transition matrix P
amounts to setting P(1) = P.
(Hint: you may use the results from Example 6.11)
b) In this case, show that this continuous-time Markov Chain is unique and find its transition
probability function matrix P(t) in terms of α.
6.24∗. Consider a Yule process with birth parameter λ starting with a single individual – that is,
suppose X(0) = 1. Let Ti denote the time it takes the process to go from a population of
size i to one of size i + 1.
a) Argue that Ti , i ≥ 1, are independent exponentials with respective rates iλ.
b) Let X1 ,. . ., Xj denote independent exponential random variables each having rate λ,
and interpret Xi as the lifetime of the ith component in a given system. Argue that
max(X1 , . . . , Xj ) can be expressed as
max(X1 , . . . , Xj ) = ε1 + ε2 + . . . + εj
where ε1 , . . ., εj are independent exponentials with respective rates jλ, (j − 1)λ,. . . , λ.
Hint: interpret εi as the time between the (i − 1)th and the ith failure.
c) Using a) and b) argue that
P(T1 + T2 + . . . + Tj ≤ t) = (1 − e−λt )j

d) Use c) to obtain
P1j (t) = (1 − e−λt )j−1 − (1 − e−λt )j = e−λt (1 − e−λt )j−1
and hence, given that X(0) = 1, X(t) has a geometric distribution with parameter p =
e−λt .
e) Now conclude that  
j − 1 −λit
Pij (t) = e (1 − e−λt )j−i
i−1
Compare with Slide 425. This is also what was anticipated in Exercise 2.21.
(Answer: - )
6.25∗. Consider a Yule process with birth parameter λ starting with a single individual, and denote
m(t) = E(X(t)) the expected number of individuals at time t. By first conditioning on the
time of the first birth, find an expression for m(t).
(Answer: m(t) = eλt . )
6.26∗. Consider two machines. Machine i operates for an exponential time with rate λi and then
fails; its repair time is exponential with rate µi , i = 1, 2. The machines act independently
of each other. Define a four-state continuous-time Markov Chain which jointly describes
the condition of the two machines. Use the assumed independence to compute the transition
probability functions for this chain. Hint: you may use results from Example 6.11.

31
6.27∗. There are two machines, one of which is used as a spare. A working machine will function
for an exponential time with rate λ and will then fail. Upon failure, it is immediately replaced
by the other machine if that one is in working order, and it goes to the repair facility. The
repair facility consists of a single person who takes an exponential time with rate µ to repair
a failed machine. At the repair facility, the newly failed machine enters service if the repair
person is free. If the repair person is busy, it waits until the other machine is fixed; at that
time, the newly repaired machine is put in service and repair begins on the other one. Starting
with both machines in working condition, find the expected value of the time until both are
in repair facility. In the long-run, what proportion of time is there at least one machine
working?
1+λ/µ
(Answer: E(T ) = λ2 + λµ2 , π = 1+λ/µ+(λ/µ) 2)

6.28∗. Suppose that when both machines are down in the previous Exercise a second repair person
is called in to work on the newly failed one. Suppose all repair times remain exponential
with rate µ. Now find the proportion of time at least one machine is working, and compare
with your previous answer.
1+λ/µ
(Answer: π = 1+λ/µ+(λ/µ) 2
/2
)

6.29∗. Let {X(t), t ≥ 0} be a Markov Chain having stationary distribution π. We may sample X
at the times of a Poisson process: let {N (t), t ≥ 0} be a Poisson process with intensity λ,
independent of {X(t), t ≥ 0}, and define Yn = X(Sn ) the value taken by X at the time of
the nth arrival in {N (t), t ≥ 0}. Show that {Yn , n ≥ 0} is a discrete-time Markov Chain
that has the same stationary distribution as {X(t), t ≥ 0}.
(Hint: find an expression for Qij , the transition probabilities for the discrete-time Markov
Chain {Yn , n ≥ 0}, in terms of Pij (t), the transition probability functions of the continuous
Markov Chain. Show that any stationary distribution for the continuous-time Markov Chain
also satisfies the stationary equations for the discrete-time Markov Chain (π = πQ). )

32
8 Queueing theory
8.1. Customers arrive at a checkout station in a market according to a Poisson process of rate
λ = 1 per minute. The checkout can be operated with or without a bagger. The checkout
times for customers are exponentially distributed, and with a bagger the mean check out
time is 30 sec, while without a bagger this mean time increases to 50 sec. Compare the mean
queue lengths with and without the bagger.
(Answer: 0.5 customer queueing on average with the bagger, 4.17 customers queueing on
average without the bagger.)

8.2. Consider a M/M/1 queueing system with arrival and service parameters λ and µ, with the
constraint that if an arriving customer sees n customers in the line ahead of him, he only
1
joins it with probability pn = n+1 and otherwise leaves in disgust. Denote ρ = µλ .

a) Find the stationary distribution of queue length.

(Answer: P(ρ) )
b) What is the probability that an arriving customer joins the queue?
−ρ
(Answer: 1−eρ )

8.3. For the M/M/1 queue with rates λ and µ, compute

a) the expected number of arrivals during a service period and

(Answer: ρ )
b) the probability that no customers arrive during a service period.
1
(Answer: 1+ρ )

8.4. Cars arrive at a car wash according to a Poisson process with a mean rate of 8 cars per hour.
The policy at the car wash is that the next car cannot pass through the wash procedure until
the car in front of it is completely finished. The car wash has a capacity to hold 10 cars,
including the car being washed, and the time it takes a car to go through the wash process is
exponentially distributed with a mean of 6 minutes. What is the average number of cars lost
to the car wash company every 10-hour day as a result of the capacity limitation?
(Answer: 1.88 cars )

8.5. Determine the mean waiting time E(W ) for an M/M/2 system when λ = 2 and µ = 1.2.
Compare this with the mean waiting time in an M/M/1 system whose arrival rate is λ = 1
and service rate is µ = 1.2.
(Answer: M/M/2 ; E(W ) = 1.894, M/M/1 ; E(W ) = 4.167)

8.6. People arrive at a library to borrow books according to a Poisson process with a mean rate of
15 people per hour. There are two attendants at the library, and the time to serve each person
by either attendant is exponentially distributed with a mean of 3 minutes.

a) What is the probability that an arriving person will have to wait?

(Answer: 0.205 )
b) What is the probability that one or both attendants are idle?
(Answer: 0.795 )

33
 8.7. A clerk provides exponentially distributed service to customers who arrive according to a
Poisson process with an average rate of 15 per hour. If the service facility has an infinite
capacity, what is the mean service time that the clerk must provide in order that the mean
waiting time shall be no more than 10 minutes?
(Answer: 3.06 minutes )

8.8. The manager of a market can hire either Mary or Alice. Mary, who gives service at an
exponential rate of 20 customers per hour, can be hired at a rate of $3 per hour. Alice, who
gives service at an exponential rate of 30 customers per hour, can be hired at a rate of $C per
hour. The manager estimates that, on the average, each customer’s time is worth $1 per hour
(e.g. in terms of satisfaction) and this should be accounted for in the model. If customers
arrive at a Poisson rate of 10 per hour, then

a) what is the average cost per hour if Mary is hired? If Alice is hired?
(Answer: $4 and $C+1/2 )
b) find C if the average cost per hour is the same for Mary and Alice.
(Answer: $3.5 )

8.9. A facility produces items according to a Poisson process with rate 18 items a day. However,
it has shelf space for only 20 items and so it shuts down production whenever 20 items
are present. Customers arrive at the facility according to a Poisson process with rate 20
customers/day. Each customer wants one item and will immediately depart either with the
item or empty handed if there is no item available.

a) Find the proportion of customers that go away empty handed.

(Answer: 0.112 )
b) Find the average number of items on the shelf.
(Answer: 6.42 )
c) Find the average time that an item is on the shelf.
(Answer: 0.36 day )

Suppose now that when a customer does not find any available items it joins the “customers’
queue” as long as there are no more than n − 1 other customers waiting at that time. If there
are n waiting customers then the new arrival departs without an item.

d) Define an appropriate state space and set up balance equations.

e) In terms of the solution of the balance equations, what is the long-term average number
of waiting customers in the system?

(Hint: define negative states when customers are waiting and no items are available.)

8.10. A supermarket has two exponential checkout counters, each operating at rate µ. Arrivals are
Poisson at rate λ. The counters operate in the following way: one queue feeds both counters,
but one counter is operated by a permanent checker and the other by a stock clerk who
instantaneously begins checking whenever there are two or more customers in the system.
The clerk returns to stocking whenever he completes a service, and there are fewer than two
customers in the system.

34
 a) Let πn = proportion of time there are n customers in the system. Set up equations for πn
and solve.
λj
(Answer: π0 = 2µ−λ
2µ+λ
, πj = 2j−1 π )
µj 0
b) At what rate does the number in the system go from 0 to 1? from 2 to 1?
λ2 (2µ−λ)
(Answer: λ(2µ−λ)
2µ+λ
and µ(2µ+λ)
)
c) What proportion of time is the stock clerk checking?
λ2 (2µ−λ) 2λ2
(Answer: 2µ(λ+µ)(2µ+λ) + µ(2µ+λ) )

8.11. Consider a sequential-service system consisting of two servers, A and B. Arriving customers
will enter this system only if server A is free. If a customer does enter, then he is immediately
served by server A. When his service by A is completed, he then goes to B if B is free, or
if B is busy, he leaves the system. Upon completion of service at server B, the customer
departs. Assuming that the (Poisson) arrival rate is two customers an hour, and that A and B
serve at respective (exponential) rates of four and two customers an hour,

a) what proportion of customers enter the system?

(Answer: 2/3)
b) what proportion of entering customers receive service from B?
(Answer: 2/3)
c) what is the average number of customers in the system?
(Answer: 7/9 (customer))
d) what is the average amount of time that an entering customer spends in the system?
(Answer: 7/12 (hour))

8.12. Customers arrive at a two-server system according to a Poisson process having rate λ = 5.
An arrival finding server 1 free will begin service with that server. An arrival finding server
1 busy and server 2 free will enter service with server 2. An arrival finding both servers busy
goes away. Once a customer is served by either server, he departs the system. The service
times at server i are exponential with rates µi , where µ1 = 4 and µ2 = 2.

a) What is the average time an entering customer spends in the system?

(Answer: 0.33)
b) What proportion of time is server 2 busy?
(Answer: 0.54)

8.13. In a restaurant, there are three separate counters. One is for take out food, and the other
two are buffet lines for customers who eat in the restaurant. One of the buffet lines is for
vegetarian food, and the other is for meat eaters. Each counter has a server, and each counter
has its own input. The service times at all three counters have an exponential density with a
mean of 15 seconds. The arrivals to the take out, vegetarian and meat-eating counters form
three Poisson processes, with mean interarrival times of 20, 30 and 18 seconds, respectively.

a) What are the average waiting time for each of the three counters?
(Answer: 45 seconds, 15 seconds, 75 seconds. )
b) What is the average waiting time for a customer entering the restaurant?
(Answer: 49.8 seconds.)

35
 c) The service is redesigned so that each counter is capable of handling all three types of
customers. The three input streams are merged into one input Poisson process. What is
the average waiting time for this system?
(Answer: 7.9 seconds. )
d) Compare the average obtained in c) to that obtained in b). What is the best strategy?
(Answer: M/M/3 )

8.14∗. Suppose that a customer of the M/M/1 system spends the amount of time x > 0 waiting in
queue before entering service.

a) Show that, conditional on the preceding, the number of other customers that were in the
system when the customer arrived is distributed as 1 + X , where X is a Poisson random
variable with mean λx. (Hint: This should be similar to the developments on Slides
492-493).
b) Let W denote the amount of time that a customer spends in queue. As a byproduct of
your analysis in part a), show that
(
1 − µλ if x = 0
P(W ≤ x) = λ λ −(µ−λ)x
1 − µ + µ (1 − e ) if x > 0

(Answer: -)

8.15∗. Consider a single-server exponential system in which ordinary customers arrive at a rate λ
and have service rate µ. In addition, there is a special customer who has a service rate µ1 .
Whenever this special customer arrives, she goes directly into service (if anyone else is in
service, then this person is bumped back into queue). When the special customer is not being
serviced, she spends an exponential amount of time (with mean 1/θ) out of the system.

a) What is the average arrival rate of the special customer?

(Answer: (1/θ + 1/µ1 )−1 )
b) Define an appropriate state space and set up balance equations.
(Answer: -)
c) Find the probability that an ordinary
 customer
n  is bumped n times.
θ µ
(Answer: P(bumped n times) = µ+θ µ+θ
)

8.16∗. a) Consider an M/M/s queueing system, in which customer departs immediately if, on ar-
rival, he sees all the servers occupied ahead of him. Show that, in the long term, the
probability that all servers are occupied is

(λ/µ)s
πs = Ps
s! i=0 (λ/µ)i /i!

(Hint: write the local balance equations and solve.)

36
b) Consider an M/M/∞ queue with servers numbered 1,2,. . .. On arrival, a customer is
served by the lowest numbered server that is free. Show that the fraction pc of time that
server c is busy is
λ(πc−1 − πc )
pc = for c ≥ 2
µ
and p1 = π1 . Note that πc here represents the fraction of time that all servers from 1 to
c are busy (notation from part a)), not the stationary distribution (= fraction of time that
server c is busy = pc ).
(Hint: write the balance equation for server c.)

37
10 Brownian motion and stationary processes
10.1. Let {B(t), t ≥ 0} be a standard Brownian motion.

a) Evaluate P(B(5) ≤ 3|B(1) = 1).

(Answer: 0.8413 )
b) Find the number c fow which P(B(10) > c|B(1) = 1).
(Answer: 4.8447 )

10.2. Consider {B(t), t ≥ 0} a standard Brownian motion. What is the distribution of B(s)+B(t),
0 ≤ s ≤ t?
(Answer: N (0, 3s + t) )

10.3. Consider {B(t), t ≥ 0} a standard Brownian motion. Compute the conditional distribution
of B(s) given that B(t1 ) = a and B(t2 ) = b,where 0 < t1 < s < t2 .
(Answer: N a + ts−t 1
2 −t1
(b − a), ts−t1
2 −t1
(t2 − s) )

10.4. Define Ta as the time it takes a standard Brownian motion to hit the value a. What is P(T1 <
T−1 < T2 )?
(Answer: 1/6 )

10.5. a) For a fixed t > 0, show that M (t) = max0≤s≤t B(s) and |B(t)| have the same probability
distribution.
(Hint: the density of M (t) is given on Slide 537.)
p
b) DeduceR that E(M (t)) = E(|B(t)|) = 2t/π.
∞
(Hint: 0 z φ(z) dz = √12π )
c) For 0 < s < t, do (M (s), M (t)) have the same joint distribution as (|B(s)|, |B(t)|)?
(Hint: you need not compute the joint distributions of (M (s), M (t)) and (|B(s)|, |B(t)|))
Note: the process {|B(t)|, t ≥ 0} is often called the reflected Brownian Motion, for obvious
reasons.

10.6. Suppose you own one share of a stock whose price changes according to a standard Brownian
motion process. Suppose that you purchased the stock at a price b + c, c > 0, and the present
price is b. You have decided to sell the stock either when it reaches back the price b + c or
when additional time t goes by (whichever occurs first). What is the probability that you do
not recover your√purchase price?
(Answer: 2Φ(c/ t) − 1)

10.7. Find the distribution of the following random variables:

R1
a) 0 t dB(t)
R1
b) 0 t2 dB(t)

where B(t) is the standard Brownian motion.

R 
1 R1 2
c) What is Cov 0 t dB(t), 0 t dB(t) ?

(Answer: a) N (0, 1/3), b) N (0, 1/5), c) 1/4.)

38
 10.8. Let Y (t) = tB(1/t), t > 0 and B(0) = 0, where B(t) is the standard Brownian motion.

a) What is the distribution of Y (t)?

b) Compute Cov(Y (s), Y (t)).
c) Argue that {Y (t), t ≥ 0} is a standard Brownian motion process.

(Answer: - )

10.9. Consider {B(t), t ≥ 0} a standard Brownian motion. For 0 < s < t argue that B(s) − st B(t)
and B(t) are independent.
(Answer: - )

10.10. Let {Z(t), 0 ≤ t ≤ 1} denote the Brownian Bridge process. Show that if

Y (t) = (t + 1)Z(t/(t + 1))

then {Y (t), t ≥ 0} is a standard Brownian motion process.

(Answer: - )

10.11. An Ornstein-Uhlenbeck process {V (t), t ≥ 0} has α = 1 and β = 0.2. What is the proba-
bility that V (t) ≤ 1 for t = 1, 10, 100? Assume that V (0) = 0.
(Answer: 0.86, 0.74, 0.74 )

10.12. Let X(t) = N (t + 1) − N (t) where {N (t), t ≥ 0} is a Poisson process with rate λ. Compute
Cov(X(t), X(t + s)).
(Answer: λ(1 − s) for 0 < s < 1, 0 otherwise )

Consider the random walk which in each ∆t time unit either goes up√or down the amount
10.13∗. √
∆t with respective probabilities p and 1 − p, where p = 12 (1 + µ ∆t). Argue that as
∆t → 0, the resulting limiting process is a Brownian motion with drift rate µ.
(Answer: - )

10.14∗. Suppose that the fluctuations in the price of a share of stock in a certain company are well
described by a Brownian motion with drift µ = 1/10 and variance σ 2 = 1. A speculator
buys a share of this stock at a price $100 and will sell if ever the price rises to $110 (profit)
or drops to $95 (loss). What is the probability that the speculator sells at a profit?
(Answer: 0.665)

10.15∗. For a positive constant  and {B(t), t ≥ 0} the standard Brownian motion, show that
√
 
|B(t)|
P >  = 2 × (1 − Φ( t))
t

How does this behave when t is large (t → ∞)? How does this behave when t is small
(t → 0)?
(Answer: - )

39
10.16∗. Let {B(t), t ≥ 0} be a standard Brownian motion and M (t) = max0≤s≤t B(s). Use the
reflection principle to show that, for 0 < x < z,
 
2z − x
P(M (t) ≥ z, B(t) ≤ x) = P(B(t) ≥ 2z − x) = 1 − Φ √
t
(Answer: - )

10.17∗. We call {B(t), t ≥ 0}, with B(t) = (B1 (t), B2 (t)), a two-dimensional (standard) Brown-
ian Motion if {B1 (t)}, {B2 (t)} are two independent R-valued standard Brownian Motion
processes. See below for four realisations of the two-dimensional Brownian Motion.

1.5
1.2
0.8

1.0
B2(t)

B2(t)
0.4

0.5
0.0

0.0
−0.5 0.0 0.5 1.0 −0.2 0.0 0.2 0.4 0.6 0.8
B1(t) B1(t)

0.5
1.0

0.0
0.5
B2(t)

B2(t)
0.0

−0.5
−0.5

−1.0

−0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5

B1(t) B1(t)

Find the probability that |B(t)| <p R, where R > 0 and |x| is the Eucliedean norm of a vector
2
of R , i.e., if x = (x1 , x2 ), |x| = x21 + x22 .
(Hint: Express the probability in terms of the joint probability density of (B1 (t), B2 (t)). It is
convenient to use polar coordinates to use polar coordinates to compute the resulting integral
over a disk.)  2
(Answer: 1 − exp −R 2t
)

40
11 Introduction to Martingales
11.1. Let U1 , U2 , . . . be independent random variables each uniformly distributed over the interval
(0, 1). Show that X0 = 1 and Xn = 2n U1 U2 · · · Un for n = 1, 2 . . . defines a martingale
{Xn , n ≥ 0}.

11.2. Suppose that X1 , X2 , . . . are i.i.d. random variables with expected value µ = E(Xi ) and
E(|Xi |) < ∞.

a) Show that {Mn , n ≥ 0} with M0 = 1 and

X1 X2 . . . Xn
Mn =
µn
is a martingale with respect to Fn the filtration associated to X1 , X2 , . . .
b) Assuming that Xi > 0 for all i, is {log Mn , n ≥ 0} a martingale in general?
c) Modify log Mn to make it into a martingale.

11.3. Suppose in the above problem that


1/3 with probability 3/4
Xi =
3 with probability 1/4

Then show that T = “first time that Mn = 3” is a stopping time. Does the Optional Stopping
Theorem hold?

11.4. Show that {Y (t), t ≥ 0} is a martingale if

Y (t) = B 2 (t) − t,

where {B(t), t ≥ 0} is the Brownian motion. What is E(Y (t))?

11.5. Let
T = min{t : B(t) = 2 − 4t},
that is, T is the first time that standard Brownian motion hits the line 2 − 4t. Use the optional
stopping theorem to find E(T ).

11.6. Let {X(t), t ≥ 0} be Brownian motion with drift coefficient µ and variance parameter σ 2 ,
i.e.,
X(t) = σB(t) + µt.
Let µ > 0 and for a positive constant x let

T = min{t : X(t) = x}

that is, the first time the process {X(t), t ≥ 0} hits x. Use the optional stopping theorem to
show that
x
E(T ) = .
µ

41
11.7∗. Show that {Y (t), t ≥ 0} is a martingale for any constant c if

Y (t) = exp(cB(t) − c2 t/2)

where {B(t), t ≥ 0} is the Brownian motion and c is an arbitrary constant. What is E(Y (t))?

11.8∗. Let X(t) = σB(t)+µt, and for given positive constants A and B, let p denote the probability
that {X(t), t ≥ 0} hits A before −B.

a) Define the stopping time T to be the first time the process hits either A or −B. Use the
optional stopping theorem and the martingale defined in Exercise 11 to show that

E exp(c(X(T ) − µT )/σ − c2 T /2) = 1.

b) Let c = −2µ/σ, and show that

E exp(−2µX(T )/σ 2 ) = 1.

c) Use part b) and the definition of T to find p. Hint: recall de Moivre’s martingale, Slides
627-628.
d) Use part c) and the optional stopping theorem to find E(T ).

42