Do all problems and give the process of your solution.

1. A Markov chain {X
0
, X
1
, X
2
, , } has the transition probability
_
_
_
0 1 2
0 0.1 0.1 0.8
1 0.2 0.2 0.6
2 0.3 0.3 0.4
_
_
_
Determine the conditional probabilities
(1). P[X
1
= 1, X
2
= 1|X
0
= 0]
Solution.
P[X
1
= 1, X
2
= 1|X
0
= 0] = P
0,1
P
1,1
= 0.1 0.2 = 0.02
(2). P[X
2
= 1|X
0
= 0]
Solution.
P[X
2
= 1|X
0
= 0] = P[X
1
= 0, X
2
= 1|X
0
= 0]
+P[X
1
= 1, X
2
= 1|X
0
= 0]
+P[X
1
= 2, X
2
= 1|X
0
= 0]
= P
0,0
P
0,1
+P
0,1
P
1,1
+P
0,2
P
2,1
= 0.1 0.1 + 0.1 0.2 + 0.8 0.3 = 0.27
2. Consider the Markov chain {X
n
} with transition probability matrix
_
_
_
_
_
_
0 1 2 3
0 1 0 0 0
1 q 0 p 0
2 0 q 0 p
3 0 0 0 1
_
_
_
_
_
_
where 0 < p < 1 and q = 1p. Let T = min
_
n 0; X
n
{0, 3}
_
be the rst time the process is absorbed.
Find (in terms of p and q)
(1). P[X
T
= 0|X
0
= 1] and P[X
T
= 0|X
0
= 2]
Solution. Write u
1
= P[X
T
= 0|X
0
= 1] and u
1
= P[X
T
= 0|X
0
= 2]. We have that
_
u
1
pu
2
= q
qu
1
+u
2
= 0
So
_

_
u
1
=
q
1 pq
u
2
=
q
2
1 pq
(2). E[T|X
0
= 1] and E[T|X
0
= 2]
Solution. Write v
1
= E[T|X
0
= 1] and v
1
= E[T|X
0
= 2]. We have that
_
v
1
pu
2
= 1
qv
1
+v
2
= 1
So
_

_
v
1
=
1 +p
1 pq
v
2
=
1 +q
1 pq
1
3. A coin is toseed is tossed repeatdly untill four heads in a row appear. Let X
n
record the current
number of successive heads that have appeared. That is, X
n
= 0 if the nth toss resulted in tail; X
n
= 1 if
the nth toss is head and the (n 1)th toss was tail; and so on. Write out the transition probability matrix
with minimal dimension.
Solution. The transition probability matrix is
_
_
_
_
_
_
_
_
0 1 2 3 4
0
1
2
1
2
0 0 0
1
1
2
0
1
2
0 0
2
1
2
0 0
1
2
0
3
1
2
0 0 0
1
2
4 0 0 0 0 1
_
_
_
_
_
_
_
_
2