Professional Documents
Culture Documents
Problem 1. Let
1/4 1/2 1/4
P = 1/3 0 2/3
1/2 0 1/2
be the transition matrix of a Markov chain {Xn }n≥0 . If P(X0 = 1) = 1/3, determine
Solution. blank
Problem 2. Consider a system that can be in one of two possible states, S = {0, 1} . The transition
matrix is given by
1/2 1/2
P =
1/3 2/3
Suppose that X0 = 0, i.e. the system is in state 0 at time n = 0.
Solution. blank
1
(a) The state transition diagram is given below.
HW/hw7.png
Problem 3. A Markov chain {Xn }n≥0 with states 0, 1, 2 has the transition matrix
1/2 1/3 1/6
P = 0 1/3 2/3
1/2 0 1/2
yielding
2
X 2
X
3
P(X3 = 0) = P(X0 = i) · P(X3 = 0|X0 = i) = P(X0 = i) · Pi0
i=0 i=0
1 13 1 4 1 5 59
= · + · + · = ,
4 36 4 9 2 12 144
2
X 2
X
3
P(X3 = 1) = P(X0 = i) · P(X3 = 1|X0 = i) = P(X0 = i) · Pi1
i=0 i=0
1 11 1 4 1 2 43
= · + · + · = ,
4 54 4 27 2 9 216
59 43 169
P(X3 = 2) = 1 − P(X3 = 0) − P(X3 = 1) = 1 − − = ,
144 216 432
thus the distribution of X3 is given by
x 0 1 2
P(X3 = x) 59/144 43/216 169/432
implying
59 43 169 53
E(X3 ) = 0 · +1· +2· = ≈ 0.982.
144 216 432 54
Problem 4. For gambler’s ruin, assume that the gambler’s initial state is $3 and the gambler plays
until either gaining $8 or going bust. At each play the gambler wins $1, with probability 0.6, or
loses $1, with probability 0.4. Find the gambler’s expected fortune after four plays.
2
Solution. Denote Xn the fortune after the ith play, then {Xn }n≥0 is a Markov chain of states
0, 1, ..., 8 and transition matrix
1 0 0 0 0 0 0 0 0
0.4 0 0.6 0 0 0 0 0 0
0 0.4 0 0.6 0 0 0 0 0
0 0 0.4 0 0.6 0 0 0 0
P =
0 0 0 0.4 0 0.6 0 0 0
0 0 0 0 0.4 0 0.6 0 0
0 0 0 0 0 0.4 0 0.6 0
0 0 0 0 0 0 0.4 0 0.6
0 0 0 0 0 0 0 0 1
Note that
1 0 0 0 0 0 0 0 0
62/125 72/625 0 162/625 0 81/625 0 0 0
148/625 0 36/125 0 216/625 0 81/625 0 0
8/125 72/625 0 216/625 0 216/625 0 81/625 0
4
P = 16/625
0 96/625 0 216/625 0 216/625 0 81/625
,
0 16/625 0 96/625 0 216/625 0 162/625 27/125
0 0 16/625 0 96/625 0 36/125 0 333/625
0 0 0 16/625 0 72/625 0 72/625 93/125
0 0 0 0 0 0 0 0 1
Problem 5. Simple random walk provides a model of the wealth process of a person who makes
a living by flipping a coin and making bets. Consider a gambler who at each play of the game has
probability p of winning one unit and probability 1 − p of losing one unit. Let Xi be the amount
that he gains at the ith play, then
Assuming that successive plays of the game are independent. Let S0 denote an arbitrary integer
that we view as our gambler’s initial wealth and
n
X
Sn = S0 + Xi , ∀n ∈ N
i=1
(a) Find the transition probability of the Markov chain {Sn }n≥0 .
(c) What is the probability that his wealth reaches $8 before going bust given that S0 = 3?
3
(a) Assume that the gambler plays until going bust, the transition probability is given by
0.4 0 < i = j − 1
0.6 0 < i = j + 1
Pi,j =
1 i=j=0
0 otherwise
(b) Note that S0 = 3 is odd, then S1 = S0 ± 1 is even and S2 = S1 ± 1 is odd, thus S2 ̸= 4 since
4 is even. In other words,
P(S2 = 4|S0 = 3) = 0.
(c) Let A be the event that the gambler’s wealth reaches $8 before going bust, that is,
A = {∃n ∈ N : Sn = 8} .
For each i = 0, 8, denote Pi the probability that the gambler’s wealth reaches $8 before going
bust, given that his initial wealth is $i, i.e.
Pi = P(A|S0 = i).
Clearly P0 = 0 and P8 = 1. Now pick an i ∈ N ∩ [0, 8] and consider a scenario in which the
gambler’s wealth is $i at some time t > 0, but his wealth never exceeded 7 during the time
horizon [0, t − 1]. Since all successive plays are independent, what happens after time t does
not depend on the result of previous plays. Hence, we can drop all these assumptions and
consider a new timeline T in which all parameters stay the same except two things:
• The initial time (of the new timeline T ) is at time t of the original timeline;
• The initial wealth (of the new timeline T ) is $i;
Then the probability that A occurs in the new timeline T is the same as the probability that
A occurs in the current timeline after time t, under the given constraints. In short,
P A St = i, max Sn < 8 = P(A|S0 = i) = Pi , ∀i = 1, 8. (1)
0≤n≤t−1
Equation (1) and the Total Probability formula implies that for any i = 1, 7,
Pi = P(A|S0 = i)
= P(S1 = i + 1|S0 = i) · P(A|S0 = i = S1 − 1) + P(S1 = i − 1|S0 = i) · P(A|S0 = i = S1 + 1)
= Pi,i+1 · P(A|S0 = i + 1) + Pi,i−1 · P(A|S0 = i − 1) = Pi,i+1 · Pi+1 + Pi,i−1 · Pi−1
which yields the following system of equations
P0 = 0, P8 = 1,
P 1 = P 1,2 · P2 + P1,0 · P0 = 0.4 · P2
P
2
= P 2,3 · P 3 + P2,1 · P1 = 0.4 · P3 + 0.6 · P1
P3 = P3,4 · P4 + P3,2 · P2 = 0.4 · P4 + 0.6 · P2
P 4 = P4,5 · P5 + P4,3 · P3 = 0.4 · P5 + 0.6 · P3
P5 = P5,6 · P6 + P5,4 · P4 = 0.4 · P6 + 0.6 · P4
P6 = P6,7 · P7 + P6,5 · P5 = 0.4 · P7 + 0.6 · P5
P7 = P7,8 · P8 + P7,6 · P6 = 0.4 + 0.6 · P6