CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

Problem 1 (10 pts): Consider a finite state Markov chain with transition probabilities P (x, y), x, y ∈ S
and let Fn (x, y) be the probability that y is reached for the first time at n, starting from x

Fn (x, y) = P {Xn = y|X0 = x, X1 6= y, . . . , Xn−1 6= y}, n = 1, 2, ...

Note that Fn (x, y) = Px {Ty = n} where Ty is the first time that the state y is reached. Show that
n
X
P n (x, y) = Fm (x, y)P n−m (y, y), n = 1, 2, ...
m=1
P∞ P∞
In terms of generating matrix functions, P (x, y; s) = n=0 sn P n (x, y) and F (x, y; s) = n=1 sn F n (x, y),
show (i) that
P (x, y; s) = δx,y + F (x, y; s)P (y, y; s)
for 0 ≤ s ≤ P 1 and where δx,y is zero when x 6= y and one otherwise. A state y is said to be persistent if
∞
F (y, y; 1) = n=1 Fn (y, y) = 1, which means that Ty < ∞ with probability one. Show (ii) that a state y is
persistent if and only if P (y, y; 1) = ∞. Since
∞
X ∞
X
P (y, y; 1) = P n (y, y) = Ey { I{Xn =y} }
n=0 n=1

we conclude that a state is persistent if and only if the mean number of returns to it is infinite.
Solution Problem 1 Note that the event {Ty = n} = {X0 = x, X6 = y, . . . , Xn−1 6= y, Xn = y} and so it
depends on the path only up to time n. We thus have
n
X
P {Xn = y|X0 = x} = P {Xn = y, Ty = m|X0 = x}
m=1
Xn
= P {Xn = y|Ty = m, X0 = x}P {Ty = m|X0 = x}
m=1
Xn
= P {Xn = y|Xm = y}P {Ty = m|X0 = x}
m=1
Xn
= Fm (x, y)P n−m (y, y)
m=1

Then, write,
X
P (x, y; s) = sn P n (x, y)
n=0
∞
X
= δx,y + sn P n (x, y)
n=1
∞ n
!
X X
n m n−m
= δx,y + s F (x, y)P (y, y)
n=1 m=1

by the previous result. Interchanging the summations we get the result

P (x, y; s) = δx,y + F (x, y; s)P (y, y; s)

We note that everything is non-negative and so monotone increasing in s → 1. From

1
P (y, y; s) =
1 − F (y, y; s)P (y, y; s)
1
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

we see that as F increases to one, we have that P increases to infinity. Similarly, from
P (y, y; 1) − 1
F (y, y; s) =
P (y, y; 1)
we see that as P increases to infinity, F increases to one.

Problem 2 (10 pts): (Lawler, Problem 1.8) Consider a simple random walk on the graph shown in Figure
1. (Recall that a simple random walks on a graph is the Markov chain which at each time moves to an
adjacent vertex, each adjacent vertex having the same probability.)

1. In the long run, about how much time is spent in vertex A?

2. Suppose a walker starts in vertex A. What is the expected number of steps until the walker returns to
A?
3. Derive a general equation for the expected number of visits to state Y before reaching state Z, starting
from state X.
4. Suppose a walker starts in vertex C. What is the expected number of visits to B before the walker
reaches A?
5. Suppose a walkers starts in vertex B. What is the probability that the walker reaches A before the
walker reaches C?
6. Again, assume the walker starts in C. What is the expected number of steps until the walker reaches
A?

Solution Problem 2
The transition probabilities are given by
1 1 1
 
0 3 3 3 0
1 0 1
0 1
 31 1
3 3
P =
 21 2 0 0 0
1

2 0 0 0 2
1 1
0 2 0 2 0

1. Since this is a random walk on a graph, the stationary distribution is given by,
deg(x) deg(x)
πx = = , x ∈ {A, B, C, D, E}.
2|E| 12

A B

D E

Figure 1: Graph for Problem 1

2
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

For vertex A, we have πA = 14 . This is the long run average time spent at A.
1
2. The average time to return is πA = 4.
3. Let Tz = inf{n ≥ 1 : Xn = z}. Note that z 6= x and z 6= y here. Let f (x) = 1{x=y} . The quantity of
interest is, "T −1 # "T −1 #
z z

1{Xi =y}
X X
u(x) = u(x, z; y) = Ex f (Xi ) = Ex
i=0 i=0

A complete analysis of this is carried out in the notes, for u as a function of z. Equations can also be
obtained for u as a function of x by first step analysis as follows.
"T −1 # " "T −1 ##
z
X Xz

u(x) = Ex f (Xi ) = Ex Ex f (Xi )|X1

i=0 i=0
" "T −1 ##
Xz

= Ex f (x) + Ex f (Xi )|X1

i=1
" "T −1 ##
Xz

= f (x) + Ex EX1 f (Xi )

i=1
"T −1 #
X Xz

= f (x) + P (x, w)Ew f (Xi )

w6=z i=1

= f (x) + (Qu)(x)
= 1{x=y} + (Qu)(x),

where Q is the restriction of P to the continuation region of the Markov chain (z is the stopping region)
and Qu is the matrix-vector product. This system could also be solved via,

u = f + P u, subject to u(z) = 0.

Either way, we have a linear system. Going from the second to the third line in the derivation above
involves time homogeneity, adjusting T → T + 1 and shifting down the summation index (more details
in the notes).

4. Solving part 3 gives u(C) = .8182.

5. The quantity of interest is slightly different from part 3. Here, we wish to calculate,

u(x) = Px (TA < TC ) = Ex 1{TA <TC } .

 

First step analysis gives

1{TA <TC }
 
u(x) = Ex
= Ex Ex 1{TA <TC } |X1
  

= Ex Px (X1 = A) + EX1 1{TA <TC } ; X1 ∈

  
/ {A, C}
= P (x, A) + (Qu)(x),

where Q is, as before, the restriction of P to the continuation region of the Markov chain (A and
C comprise the stopping region) and Qu is the matrix-vector product, and P (x, A) is the transition
probability from x to A. In this case, this is equivalent to,

u = P u, subject to u(A) = 1, u(C) = 0.

Solving the linear system gives u(B) = .5714.

3
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

6. The quantity of interest is "T −1 #

A

1{Xi 6=A} .
X
u(x) = Ex [TA ] = Ex
i=0

From first step analysis,

"T −1 #
A

1{Xi 6=A}
X
u(x) = Ex
i=0
" "T −1 ##
A

1{Xi 6=A} |X1

X
= Ex Ex
i=0
" "T −1 ##
A

1{X0 6=A} + Ex 1{Xi 6=A} |X1

X
= Ex
i=1
" "T −1 ##
A

= 1{x6=A} + Ex EX1 1{Xi 6=A}

X

i=1
= P (x, A) + (Qu)(x).

Solving the linear system gives, u(C) = 2.6364.

Problem 3 (10 pts): Random Walk on a Graph

1. Let G = (V, E) be a finite undirected and connected graph. Let X = (Xn : n ≥ 0) be a Markov
chain that moves from vertex to vertex by choosing uniformly among the available edges. Compute
the stationary distribution of this Markov chain (known as “random walk on the graph G”)

2. A knight moves randomly on a chessboard, making each admissible move with equal probability, and
starting from a corner. What is the average time he takes to return to the corner he started from?

Solution Problem 3

1. The transition probability from a node is uniform between its neighbors so the transition matrix has
entries
1
Pij = 1{{vi ,vj }∈E} .
deg(vi )

From the fact that the graph is connected we know that there is a path from any vertex to any other
vertex. Thus there is a non-zero probability from going from any state (in the corresponding Markov
chain) to any other (with probability given by the probability of that path). Thus the Markov chain is
irreducible. Furthermore, the state space V is finite so that the Markov chain is also positive recurrent
so it will have a unique stationary distribution which is the solution π to

π = πP.

Intuitively, it makes sense that the long run time at a node should be proportional to how many
neighbors it has. Namely, we hypothesize that

πi = α deg(vi ).

We use the fact that

|V |
X
deg(vi ) = 2|E|
i=1

4
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

so that in our hypothesis becomes,

deg(vj )
πj = .
2|E|
We now verify that this solves π = πP :
|E| |E| |E|
1 X deg(vi ) 1 X deg(vj )
1{{vi ,vi }∈E} = 1{{vi ,vj }∈E} =
X
πi Pij = = πj ,
i=1
2|E| i=1 deg(vi ) 2|E| i=1 2|E|

since the last summation counts the neighbors of vj .

2. A chessboard has 64 squares. Each square can be seen as a vertex of a finite graph with each edge
connecting a pair of squares the Knight can move between. The following table shows the degree of
the vertices corresponding to squares of the Chessboard.

2 3 4 4 4 4 3 2
3 4 6 6 6 6 4 3
4 6 8 8 8 8 6 4
4 6 8 8 8 8 6 4
4 6 8 8 8 8 6 4
4 6 8 8 8 8 6 4
3 4 6 6 6 6 4 3
2 3 4 4 4 4 3 2

Applying the result of part 1, we can see that the expected time the knight comes back to the corner
is
1 2|E| 336
E[Time for Knight to return to corner] = = = = 168.
πcorner deg(corner) 2

Problem 4 (10 pts): Choice Theory: Markov Chain Models of Brand Switching
Suppose for a given product, a consumer has a choice between m competing products, labeled 1, 2, . . . , m.
Suppose successive purchases by a consumer are modeled as an ergodic Markov chain {Xn } with transition
matrix P = {pij , 1 ≤ i, j, ≤ m}.
The diagonal entries of P are measures of consumer loyalty to a brand since pjj measures the tendency of a
consumer to repurchase brand j when his last purchase was brand j.
Companies are interested in market share. The market share of a brand j is the percentage of purchases
made which are of brand j. Assuming a homogeneous population, the market share of brand j can be
evaluated as
N
1{Xn =j} ,
X
lim N −1
N →∞
n=0
which we know is just π(j), where π is the stationary distribution for the chain.

1. Consider a simple case. Suppose ∀j, pjj = 1/2 and ∀i 6= j, pij = (2(m − 1))−1 . Check that this matrix
is doubly stochastic and find its stationary distribution.
2. Now assume that for i 6= j,
1 − pii
pij = ,
m−1
so that a consumer not choosing his old brand chooses uniformly among the others. With this simple
specification of choice probabilities, can we see the dependence of market share on consumer loyalty?
Verify
(1 − pjj )−1
π(j) = Pm −1
,
k=1 (1 − pkk )
so that π(j) ↑ 1 as pjj ↑ 1. (Wolf, 1989)
5
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

Solution Problem 4

1. A transition matrix P is stochastic (and also doubly stochastic) if

Pm
j=1 pij = 1 ∀i

From the definition of pij we have

m
X 1 X 1 1 m−1
pij = + = + =1
j=1
2 2 (m − 1) 2 2 (m − 1)
j6=i

Thus the matrix is stochastic and by symmetry doubley stochastic. If a transition matrix is doubly
stochastic each πi is in the form πi = 1/N where N is the number of states. Thus the stationary
distribution is
1
πi = m ∀i
P
2. The stationary distribution is the solution to π = πP and π = 1. Expanding the first relation we
have,
m
X X 1 − pii
πj = πi Pij = πi + πj pjj
1
m−1
i6=j
X 1 − pii
⇒ πj (1 − pjj ) = πi
m−1
i6=j
P 1−pii
i6=j πi m−1
⇒ πj =
(1 − pjj )
From here, two strategies are possible:
P
• We can apply the second equation i πi = 1 and solve for π.
• Or we can verify that the given π satisfy those equations.
P
Thus we have an expression for the denominator i6=j and the distribution πj is given by
−1
(1 − pjj )
πj = Pm −1
k=1 πi (1 − pkk )
From this expression we can see that as pjj → 1 that the denominator of the above expression looks like
(1 − pjj ) / (1 − pjj ) → 1 and thus πj → 1.

Problem 5 (10 pts): Non-Parametric Maximum Likelihood Estimation of Transition Proba-

bilities
Imagine we have a Markov chain {Xn } on the finite state space S with the number of states m known and
the transition P = (pi,j , 1 ≤ i, j ≤ m) matrix unknown. We observe the chain during the times 0, 1, . . . , n
and wish to estimate P .
If x0 , . . . , xn is the realization of the Markov chain to time n, then the likelihood function Ln (P ) is given by
Ln (P ) = P(X0 = x0 , . . . , Xn = xn )
= P(X0 = x0 )px0 ,x1 · · · pxn−1 ,xn .

Show that the maximum likelihood estimator of P is given by

Ni,j
p̂i,j = ,
Ni
6
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

where
n−1
1{Xk =i,Xk+1 =j} ,
X
Ni,j =
k=0

and X
Ni = Nik .
k∈S

Solution Problem 5
We are now solving a constrained optimization problem:
P
Ln (P ) = log P (X0 = x0 ) + (i,j)∈S×S Ni,j log pxi ,xj
Pm
gk (P ) := j=1 pxk ,xj = 1, k = 1, . . . , m
Pm
We introduce one Lagrange multiplier λi for each constraint require that j=1 pxk ,xj = 1 for all k. Equating
derivatives we find that:
∂Ln (P ) Ni,j X  ∂gk (P ) 
= = λi = λk
∂pxi ,xj pxi ,xj ∂pxi ,xj
k
or
Ni,j
pi,j =
λi
Applying the constraints that the rows of P must sum to one, we find:
X X Ni,j Ni
1= pi,j = = ⇒ λi = Ni
j j
λi λi

We find:
Ni,j
pi,j = .
Ni

Problem 6 (10 pts): A Markov chain (Xn , n ≥ 0) is irreducible if for any x, y ∈ S, there exists a finite
path x = x0 , x1 , x2 , . . . , xn = y such that p(xi−1 , xi ) > 0 for any 1 ≤ i ≤ n. This means that it is possible to
go from any state to any other state with positive probability in a finite number of steps.
Suppose that X = (Xn : n ≥ 0) is an irreducible finite state Markov chain.

1. Argue that for each x ∈ S, Px (τ (z) ≤ m) > 0, where m = |S| (the number of points in S) and
τ (z) = inf{n ≥ 1 : Xn = z}.
2. Put δ = min{Px (τ (z) ≤ m) : x ∈ S}. Prove that Px (τ (z) > nm) ≤ (1 − δ)n .
hP i2
τ (z)−1
3. Prove that Ez j=0 |f (Xj )| < ∞ for any bounded function f .

1
4. Recall that Ez [τ (z)] = π(z) . Using first transition analysis (the renewal method), derive a linear
system of equations to solve for  2 
τ (z)−1
X
Ez  f (Xj )  (1)
 
j=0

 k 
τ (z)−1
(and hence σ 2 ) (Hint: Use “first transition analysis” to express the above in terms of Ex
P
j=0 f (Xj )
for k = 1, 2)

Solution Problem 6

7
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

1. We note that by the definition of recurrence, there exists an n > 0 such for every x ∈ S and y ∈ S that

P n (x, y) > 0.

Note that the longest possible path which does not visit the same state twice is of length m. Combining
this with the definition of irreducibility we we arrive at the desired result.
2. Let Yi be the probability of not reaching state z in at most i|S| steps. The base case Y1 = (1 − δ)
follows from the definition above. Now suppose that after those |S| steps you are not in state z. By
the Markov property, the probability of not reaching z in another |S| steps is, at most, (1 − δ). Thus
Y2 ≤ (1 − δ)2 . Following this logic we arrive at the conclusion that Yn ≤ (1 − δ)n which is the desired
result.
3. Consider the stopping time
τ = inf{n ≥ 1 : Xn = z}.
Note that for each integer n we have from the previous part that

P {τ > n|S|} ≤ (1 − δ)n ⇒ Fτ (n|S|) ≥ 1 − (1 − δ)n ,

where Fτ (·) is the CDF of τ . Since the CDF is non-decreasing we have that

Fτ (j) ≥ 1 − (1 − δ)n , nm ≤ j ≤ (n + 1)m

from which it follows that

P {τ = j} ≤ (1 − δ)n − (1 − δ)n+1 = (1 − δ)n (1 − (1 − δ)) = δ(1 − δ)n .

Now note that

∞ ∞
  X X
E τ2 = j 2 P {τ = j} ≤ j 2 δ(1 − δ)bj/mc
j=1 j=1

where the division in the exponent is integer division. Changing to a sum over n we have the bound
∞
X
E τ 2 ≤ const (n + 1)2 (1 − δ)n δ.
 
n=1

Note that by using the ratio test we can show that this sum converges to some finite limit, call it a.
For bounded f we have that
sup f (x) ≤ fmax < ∞.
x∈S

Thus
 2   2 
τ (z)−1 τ (z)−1
X X
2 2
Ex τ (z)2 ≤ fmax
2
 
Ez  f (Xj )  ≤ fmax Ez  1  = fmax a < ∞.
   
j=0 j=0

4. Define  k 
τx (z)−1
X
uk (x) = Ex  f (Xi ) 
 
i=0

where (as before)

τx (z) = inf{n > 0 : Xn = z|X0 = z}.
From lectures or from the class notes we have that u1 (x) satisfies the vector equation

u1 = P u1 + f, x 6= z
8
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

and u1 (z) = 0.
Now consider that
 2  2
τx (z)−1 τx (z)−1 τx (z)−1
X X X
 f (Xi ) = f (X0 )2 + 2f (X0 ) f (Xi ) +  f (Xi ) .
i=0 i=1 i=1

So
 2     2 
τx (z)−1 τx (z)−1 τx (z)−1
X X X
u2 (x) = Ex  f (Xi )2   = Ex EX1 f 2 (X0 ) + 2f (X0 ) f (Xi ) +  f Xi   =
    
i=0 i=1 i=1

     2 
τX1 (z)−1 τX1 (z)−1
X X
= f 2 (x) + f (x)Ex EX1  f (Xi ) + Ex EX1  f (Xi )  =
  
i=0 i=0

= f 2 (x) + f (x) + Ex [u1 (X1 )] + Ex [u2 (X1 )] = f 2 (x) + f (x)(P u1 )(x) + (P u2 )(x).
This gives the matrix system
u2 = f 2 + diag(f )P u1 + P u2 , x 6= z
and u2 (z) = 0.

Problem 7 (10 pts): The page-rank algorithm computes the equilibrium distribution of a finite state
Markov chain having a transition matrix of the form
P = αΛ + (1 − α)Q, 0 < α < 1,
where Λ is a rank-one stochastic matrix having identical nonnegative rows and Q is an arbitrary stochastic
(Markov transition) matrix.

1. Compute the equilibrium distribution π(α) of P (in terms of something involving the inverse of a
matrix).
2. Argue that if an arbitrary stochastic matrix P has a strictly positive column, it is of the above form.
3. Prove that QΛ = Λ2 = Λ.
4. Using the equilibrium calculated in part 1, compute
Pn − Π
in terms of α, Λ, and Q, where Π is the rank one matrix having all rows equal to the equilibrium
distribution π.
5. Use matrix norms to show that there exists a > 0 such that exp (an) kP n − Πk → 0 as n → ∞.

Solution Problem 7

1. Assume Λ has identical rows λ. Observe that if π (α) exists, then π (α) Λ = λ. By the definition of
invariant probabilities,
π (α) = π (α) P = π (α) (αΛ + (1 − α) Q) = αλ + (1 − α) π (α) Q
Thus we have
π (α) (I − (1 − α) Q) = αλ
Note that by the Gershgorin circle theorem the spectral radius of Q is less than or equal to 1. Therefore
all eigenvalues of I − (1 − α) Q are nonzero, which also means that I − (1 − α) Q is nonsingular. Then
−1
π (α) = αλ (I − (1 − α) Q) .

9
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

λT
   T 
p1
 ..   .. 
2. Assume Λ =  .  and P =  . . If P has identical rows, then P = Λ with α = 1. Otherwise
λT pTn
0 ≤ α < 1, and we have the following expression:
 T   1 T α T 
λ 1−α p1 − 1−α λ
P = α  ...  + (1 − α)  ..
   
. 
1 α
λT T
1−α pn − 1−α λ
T

So it remains to find suitable α and λ. We require that (i)λ ≥ 0 (ii)the sumof all elements of λ is one
1 α 1 α
(iii) 1−α pi − 1−α λ ≥ 0, and (iv) the sum of all elements of 1−α pi − 1−α λ is one. Note that (iv) is
automatically true if (ii) is true. Furthermore, (iii) is equivalent to that pi ≥ αλ. From this condition
a good guess of λ is ek , the vector with k -th entry 1 and 0 otherwise. Now let k be the location of the
strictly positive column in P and α be the minimum value in this column. Then conditions (i)-(iv) are
satisfied by this choise.
3. Just note that for any row vector v T whose sum of all elements is 1, v T Λ = Λ. Then QΛ = Λ2 = Λ.
4. By direct computation we have
−1
αλT (I − (1 − α) Q)
 

Pn − Π =
n
[αΛ + (1 − α)Q] − 
 .. 
. 
−1
αλT (I − (1 − α) Q)
n −1
= [αΛ + (1 − α)Q] − αΛ (I − (1 − α) Q)
n P∞ i i
= [αΛ + (1 − α)Q] − αΛ i=0 (1 − α) Q
Use part 2 and induction to argue that
n−1
X
n
[αΛ + (1 − α)Q] = αΛ (1 − α)i Qi + (1 − α)n Qn
i=0

This formula holds for n = 1 and suppose it also holds for some n. Then
h i
n+1 Pn−1
[αΛ + (1 − α)Q] = [αΛ + (1 − α)Q] αΛ i=0 (1 − α)i Qi + (1 − α)n Qn
Pn−1
= [αΛ + (1 − α)Q] αΛ i=0 (1 − α)i Qi
+α(1 − α)n ΛQn + (1 − α)n+1 Qn+1
Pn−1
= αΛ Pi=0 (1 − α)i Qi + α(1 − α)n ΛQn + (1 − α)n+1 Qn+1
n
= αΛ i=0 (1 − α)i Qi + (1 − α)n+1 Qn+1
Finally,
∞
X i
P n − Π = (1 − α)n+1 Qn+1 − αΛ (1 − α) Qi .
i=n

5. From the computation of part 3,

P∞ i
kP n − Πk = k(1 − α)n+1 Qn+1 − αΛ i=n (1 − α) Qi k
∞ i
≤ k(1 − α)n+1 Qn+1 k + kαΛ i=n (1 − α) Qi k
P
∞ i
≤ (1 − α)n+1 kQkn+1 + αkΛk i=n (1 − α) kQki
P

Take the norm to be the infinity norm. Then kΛk∞ and kQk∞ are equal to 1. That implies
P∞ i
kP n − Πk∞ ≤ (1 − α)n+1 + α i=n (1 − α)
n
(1−α)
≤ (1 − α)n+1 + α 1−(1−α)
≤ 2(1 − α)n
= 2en log(1−α)
10
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

Let 0 < a < − log(1 − α), ean kP n − Πk∞ → 0 as n → ∞. Since all norms are equivalent in the finite
dimensional space, this result holds for any matrix norms.

Problem 8 (10 pts): Branching Process.

Consider the following model of the evolving of a population. Let Xn denote the number of individuals in
the n-th generation. We let X0 = 1.
At generation n, each individual independently produces its offsprings according to a fixed probability
distribution. More rigorously, we have
Xn
(n)
X
Xn+1 = Yi
i=1
(n)
where Yi ’s are independently and identically distributed for all i and n, denoting the number of offsprings
produced by the i-th individual in the n-th generation. Obviously this is a Markov chain. We define
(1)
gk = P(Y1 = k).
P∞
We assume that g0 > 0 and k=0 gk = 1.

1. For any positive real variable z, let

∞
X
G(z) = gk z k , and Gn (z) = Ez Xn .
k=0

Of course we have G(z) = G1 (z). Now prove that for all n ≥ 1, Gn+1 (z) = Gn (G(z)).
2. We are interested in the probability that extinction finally happens. That is,

pex = P(∃n : Xn = 0).

Prove that pex = limn→∞ Gn (0).

3. Assume that G(z) is a continuous function in z, prove that pex is the smallest positive root of the
equation G(z) = z.
(1)
4. Let m = G0 (1) = EY1 (you should convince yourself here). Prove that pex = 1 if and only if m ≤ 1.

5. Special case I: If g0 = q, g2 = 1 − q for some 0 < q < 1, and all the rest of the gk ’s are zero, compute
pex .
6. Special case II: If gk = q(1 − q)k for some 0 < q < 1, compute pex .

Solution:

1. We have

Gn+1 (z) = Ez Xn+1

PXn (n)
Yi
= E[E[z i=1 |Xn ]]
Xn
= E[G(z) ]
= Gn (G(z)).

11
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

2. Let An = {Xn = 0} to be the event that at the n-th generation, there are no offsprings at all. Then
we have [∞ 
P(An ) = Gn (0), pex = P An .
n=1

On the other hand, An ⊆ An+1 . Thus {An } is an increasing sequence of sets. Thus
∞
[ 
pex =P An = lim P(An ) = lim Gn (0).
n→∞ n→∞
n=1

3. From Gn+1 (z) = Gn (G(z)) we can obtain Gn+1 (z) = G(Gn (z)). Taking n → ∞ and setting z = 0
gives pex = G(pex ). Hence pex is one root of the equation z = G(z). Now let α to be any positive root
of the equation z = G(z), we must have

α = G(α) ≥ G(0).

Repeating this n times we get α ≥ Gn (0). Taking n → ∞, we obtain pex ≤ α.

4. Let f (z) = G(z) − z. Of course we have f (1) = 0, f (0) > 0. Taking the derivative, we obtain that
f 0 (z) is an increasing function for z ∈ [0, 1], and f 0 (1) = m − 1. If f 0 (1) ≤ 0, then f 0 (z) ≥ 0 for all
z ∈ [0, 1] and hence f (z) > f (1) = 0 for all z ∈ [0, 1). Thus z = 1 is the smallest positive zero of f (z)
and hence pex = 1.
On the other hand, if f 0 (z) = m − 1 > 0, then we must have minz∈[0,1] f (z) < 0. Hence by continuity
of f (z) there exists another zero of f (z) between (0, 1). Hence pex < 1.
5. Now we have G(z) = q + (1 − q)z 2 . Solving the equation

(1 − q)z 2 − z + qz = 0

gives
q
z = 1 or z = .
1−q
Hence if q < 1/2, pex = q/(1 − q). Otherwise pex = 1.

6. Now we have
∞
X q
G(z) = q (1 − q)k z k = .
1 − (1 − q)z
k=0

Solving G(z) = z gives

q
z = 1 or z = .
1−q
Hence if q < 1/2, pex = q/(1 − q). Otherwise pex = 1.

12
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

Problem 9 (10 pts): Birth and Death Chain.

Let Xn be a Markov chain defined on {0, 1, 2, . . .}. For any i ≥ 1, the transition probability p is defined by


 pi if j = i + 1
qi if j = i − 1

p(i, j) =

 ri = 1 − pi − qi if j = i
0 otherwise


Here pi , qi > 0, ri ≥ 0 for all i ≥ 1. Also we require q0 = 0 and p0 > 0. Define

m−1 k
XY qj
h(m) =
p
j=1 j
k=0

for any m ≥ 1 and h(0) = 0.

1. Fix 0 ≤ a < b. Define

τa = inf{k ≥ 0 : Xk = a}
τb = inf{k ≥ 0 : Xk = b}.

For any a ≤ x ≤ b, use first step analysis to verify that

h(b) − h(x)
P(τa < τb ) = .
h(b) − h(a)

2. Considering a = 0 and b → ∞ show that the chain is transient if and only if h(·) is bounded above.
3. Suppose i(pi /qi − 1) → c as i → ∞. Show that the chain is recurrent if c < 1 and transient if c > 1.
In particular, under the special case where pi = p = 1 − qi for all i > 0, the chain is recurrent if and
only if p ≤ 1/2.

Solution:

1. Let γx = Px (τa < τb ). Then we must have γa = 1, γb = 0 and for all a < x < b,
qx px
γx = γx−1 + γx+1 .
1 − rx 1 − rx
Re-arrange the terms gives
qx
γx+1 − γx = (γx − γx−1 )
px
Hence
x x
Y qi Y qj
γx+1 − γx = (γa+1 − γa ) = B .
i=a+1
pi p
j=0 j

for some constant C independent of x. Hence

x−1 k
XY qj
γx = B + A = Bh(x) + A
p
j=0 j
k=0

for some constants B and A independent of x. Since γa = 1, γb = 0 we can calculate A and B to obtain
h(b) − h(x)
γx = .
h(b) − h(a)

13
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

2. If h(·) is bounded above, then limb→∞ = C for some constant C (recall that h(b) is an increasing
function in b). Thus

Px (τ0 < ∞) = lim Px (τ0 < τb )

b→∞
h(b) − h(x)
= lim
b→∞ h(b)
C − h(x)
= < 1.
C
Hence the chain is transient. On the other hand, if h(b) ↑ ∞ as b → ∞, then by the same argument
we obtain
Px (τ0 < ∞) = 1.
Hence the chain is recurrent.

3. If c < 1, then for sufficiently large j we must have j(pj /qj − 1) < c0 < 1. Hence
k k  −1
Y qj Y c0
> 1+
p
j=1 j j=1
j
 Xk 
0
= exp − log(1 + c /j)
j=1
 k 
0
X 1
≥ exp −c
j=1
j
 
≥ C exp −c0 log k

C
=
k c0
Px−1 Qk
for some constant C. Hence h(x) = k=0 j=1 qj /pj is unbounded. This implies that the chain is
recurrent.
Qk 0
If c > 1 then using the same argument we can prove that j=1 qj /pj ≤ Ck −c for some constant C
and c0 > 1. This gives that h(x) is bounded. Hence the chain is transient.

14
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

Problem 10 (10 pts): M/G/1 chain Let ξi be i.i.d. defined on {−1, 0, 1, 2, . . .}. Their distribution is
P(ξi = k − 1) = ak .
P∞
Here ak > 0 and k=0 ak = 1. Define the Markov chain Xn by X0 = x > 0 and
Xn+1 = max(Xn + ξn+1 , 0).
Such chain Xn is called a M/G/1 process. Xn can be regarded as the number of customers in the queue at
time n.

1. Find its transition probability kernel.

2. Define
N0 = inf{i ≥ 0 : Xi = 0}.
Base on the fact that Xn is homogeneous, argue that Ex N0 = cx for some constant c and for any
x ≥ 0.
P∞
3. Define µ = k=0 kak . If µ < 1, use first step analysis on E1 N0 to prove that
1
c= .
1−µ

4. Again, use first step analysis to prove that

a0
E0 T0 = .
1−µ
Here T0 := inf{i ≥ 1 : Xi = 0}.
5. Calculate the stationary distribution {π(i)}∞ i=0 . Here π(i) is an infinite series there is no closed form
solution. However you are asked to get a recursive form (that is, calculate π(0) and provide a expression
for π(k) based on π(0), π(1), . . . , π(k − 1)).

Solution:

1. We have

a0 + a1 if j = 0
if i = 0, p(0, j) =
aj+1 if j ≥ 1

aj−i+1 if j − i + 1 ≥ 0
if i ≥ 1, p(i, j) =
0 otherwise

2. Define
Tk = inf{i ≥ 1 : Xi = k}.
Since the chain is homogeneous, we must have E1 T0 = E2 T1 = E3 T2 = . . . = c for some constant c.
Hence
Ex N0 = Ex Tx−1 + Ex−1 Tx−2 + . . . + E1 T0 = cx.
3. By the first step analysis we have
c = E1 N0 + 1
∞
X
= ak Ek N0 + 1
k=0
X∞
= c kak = cµ + 1.
k=0

Hence c = 1/(1 − µ).

15
CME308-2014: Assignment 4 Due: Friday May 30, 5:00pm

4. Still using the first step analysis

∞
X
E0 T0 = ak Ek−1 N0 + 1
k=2
X∞
= c (k − 1)ak + 1
k=2
µ − a1 − 1 + a0 + a1
= +1
1−µ
a0
= .
1−µ

5. We have
1 1−µ
π0 = = .
E0 T0 a0
The recursive formula is
k+1
X
πk = πi ak+1−i
i=0
or
 k 
1 X
πk+1 = πk − πi ak+1−i
a0 i=0

for any k ≥ 0.

16