Chapter 4 > Po=l, py=t, P, Past Peak Mand, Pant ss Pu=5 Pa=5 4 ot Pan’ Pant 23 sen C8 8 BD OR ON OD OB] os where D = dry and R = rain. For instance, (DDR) means that it is raining today, was dry yesterday, and was dry the day before yesterday. 4. Let the state space be § = {0,1,2,5,7,3}, where state i?) signifies that the present value is i, and the present day is even (odd). Cubing the transition probability matrix, we obtain, BP 13/36 11/S4 47/108" 4/9 4p27 yz7 5/12 2/9 13/36 Thus, E[Xs] = P(X = 1) + 2P(% = 2) 13 1 =FPh + GPA + Ph 6, Itis immediate form = 1, s0 assume forn. Now use induction. 7. Ply + Ph =PsiPio + PssPu + PsP. = (.2)(-5) + (8)(0) + (2)(0) + (8.2) = 26 36 Let the state on any day be the number of the coin, thatis flipped on that day. _[73 [64 and so, a _ [67 33 Ba [e a) and e= (2% S| Hence, 1 3, 3 [Ph = Phi] = 6605 If we let the state be 0 when the most recent flip lands heads and let it equal 1 when it lands tails, then the sequence of states is a Markov chain with, transition probability matrix 73 64 The desired probability is P§ Itis not a Markov chain because information about previous color selections would affect probabili- ties about the current makeup of the urn, which would affect the probability that the next selection, is red, The answer is 1 ~ P32 for the Markov chain with transition probability matrix js4a 343 loo1 Pha The answer is —#3-~ forthe Markov chain with transition probability matrix 100 343 23512, B 4 16. 17. 18, ‘Answers and Solutions 37 The result is not true, For instance, suppose that Pp,1 = Po,2 = 1/2,P),0 = 1,P2,3 = 1. Given Xp =0 and that state 3 has not been entered by time 2, the equality implies that X1 is equally likely to be 1 or 2, which isnot true because, given the information, 2X; is equal to 1 with certainty. PE=SPEPL > 0 ® () {01,2} recurrent (ii) {0,1,2,3} recurrent. (iii) {0,2} recurrent, {1} transient, {3,4} recurrent. (iv) {0,1} recurrent, {2} recurrent, {3} transient, {4} transient. Consider any path of states ip = iyinyiay--in = j such that Pi, , > 0. Call this a path from i to j. If j can be reached from i, then there must be a path from i to j. Let jo,...riy be such a path. If all of the values jg,...rin are not distinct, then there is a subpath from i to j having fewer elements (for instance, ifi,1,2,4, 1,3, jis a path, then soisi, 1,3, Hence, if a path exists, there must be one with all distinct states. If Py, were (strictly) positive, then P| would be 0 for all x (otherwise, i and j would communicate) But then the process, starting in i, has a positive probability of at least Pi; of never returning to i This contradicts the recurrence of i. Henee Py =O. Syy/n + EDV] by the strong law of large num: bers. Now E[Y] = 2p—1. Hence, if p > 1/2, then ELY] > 0, and so the average of the Y;s converges in this case to a positive number, which implies that $Y; -> 00 as n+ 00. Hence, sate 0 can be Visited only a finite number of times and so must be transient Similarly, fp < 1/2, then E[Y| <0, and so lim $Y) and the argument is similar. ‘ Ifthe state at time m is the n coin tobe flipped then 1a sequence of consecutive states constitutes a two- state Markov chain with transition probabilities Pir =6=1- Pir Pra (a) The stationary probabilities satisty Pra 5 m= ry + Sm m+m=1 19, 2 Solving yields that m, = 5/9, = = 4/9. So the pro portion of flips that use coin 1 is 5/9. () Pty = 44440 ‘The limiting probabilities are obtained from ro= Tro + Sn =r + 2r5 3r + Sr, ntntntn= and the solution is 1 3 3 (M+ 1) Hence, by uniqueness these are the limiting prob- abilities. ‘The transition probabilities are , -{i ifj=i 1 ew itp fi By symmetry, Pi. jet So, let us prove by induction that 3 . = gt pd 4oy", ifj=i 11 . papa toy, tit ‘As the preceding is true for n = 1, assume it for m. To complete the induction proof, we need to show that 1,3 ye 1 fet gar, aie 14 iti qd aoy, ihe38 2 2, “Answers and Solutions Now, PES PE Pua t DPR Phi a (= 40)" = 3a) 1 " 4a ~4ay Je 4a)"(1 —3a a) doy! By symmetry for j 4 i wet 1, prt) 11g _gqyet Py gt Pi ) qo gt Ay and the induction is complete By letting n > ooin the preceding, or by using that the transition probability matrix is doubly stochas- tic, or by just using a symmetry argument, we obtain that mj = 1/4. Let Xy denote the value of Y, modulo 13, That is, Xristhe remainder when Yq isdivided by 13. Now Xp is a Markov chain with states 0,1,...,12. It is easy to verify that 5, Py = 1 for allj For instance, for) =3: DPj= Pas + Pst Pos + Pina + Pans + Pos 14141 states 1 Hence, from Problem 20,1; = 7 (a) Letting 0 stand for a good year and 1 for a bad year, the successive states follow a Markov chain, ‘with transition probability matrix P: 1a 4/2 1/8 2/8 Squaring this matrix gives P® 5/2 7/12 7/18/18 Hence, if 5; is the number of storms in year i then E[Sy] = E1Sy|X; = OPop + ELS4|%1 = UP =1/2+3/2=2 E{Sa] = ElS2|X2 = O}Pho + ELS2|X2 = 11P 5; = 5/12 + 21/12 = 26/12 Hence, E[Sy + $2] = 25/6. (©) Multiplying the first row of Py the firstcoluma, of P? gives Py = 5/24 + 7/86 = 29/72 Hence, conditioning on the state at time 3 yields 53-0) = M4 -0%4-0)2 + PS=0%4=1) 6p. Bs nn *7' (c) The stationary probabilities are the solution of 1 wnt + 05 + mg giving m=2/5, m= Hence, the long-run average number of storms is 2/8 + 36/8) = 1/5. Let the state be the color of the last ball selected, call it 0 if that color was red, 1 if white, and 2 if blue. The transition probability matrix of this Markov chain is 1s 0 4/5 =|2/7 3/7 2/7 3/9 4/9 2/9 Solve for the stationary probabilities to obtain the solution. Letting X, denote the number of pairs of shoes at the door the runner departs from at the begin- ning of day 1, then {X,} is a Markov chain with transition probabilities = 14, o 0. Because m;P%, is the long-run proportion, of time the chain is currently in statej and had been, in state i exactly n time periods ago, the inequality follows, Because recurrence is a class property it follows that state j, which communicates with the recur- rent state is recurrent. But ifj were positive recur- rent, then by the previous exercise i would be as well. Because iis not, we can conclude that jis mull, recurrent, (a) Follows by symmetry. (b) If.) = a > 0 then, for any n, the proportion of time the chain is in any of the states 1,...,71 is na. But this is impossible when n > L/a. (a) ‘The number of transitions into state i by time n, the number of transitions originating from