> “Stochastic Processes 1 ‘ 1, Stochastic process; theoretical background 1 Stochastic processes; theoretical background 1.1. General about stochastic processes t € T} of random variables X(t), all defined on the same Z. (0, +00 or A stochastic process is a family {X(t} mple space 9, where the domain T of the parameter is a subset of R (nsnally N, No R itself), and where the parameter t € T is interpreted as the time. define a so-called saznple function We note that ase for every fixed w in the sample space © in this way T(..w):T > R on the domain T of the parameter. In the description of such a stochastic process we must know the distribution function of the stochastic process. icc, IA tn} ER, for every n EN. 1 AX (to) <2 A+++ AX (tn) for every t) tn € T, and every %, «++, En This is of course not always possible, so one trics instead to find less complicated expressions connected to some extent can be used to characterize the with the stochastic process, like e.g. means, which t distribution. all discrete of values in No oceurs when the random variables X(f) are ¢ al time L is at state Ex. This can now be further ‘A very important special case Teh X(t) =k, then we say that the pro ; Ys ase } cd.A Markov process is a discrete stochastic process of values in No, for which also P(X (tat) = Brogs |X (tu) = by Ao AX (Lr) = hy} = PEN (tet) = Bie LX (tn) = kin} for any Kio... Ayer in the range, for any ty) < la << beet from Ty and for any 1 € 6. We say that when a Markov process is going to be described at time f,,4 1. then we have just as much information, if we know the process at time tq, as if we even know the process at the times t). tn: provided that these times are all smaller than Gay1. One may coin this in the following ways If the present is given. then the future is independent of the past. 1.2. Random walk Consider a sequence (X,) of mutually independent identically distributed random variables, whe the distribution is given by PIXes1p=p and P{XR=-1h=q pq > Oand p+q= landk EN. We define another sequence of random variables (Sx) by So=0 and forn EN. In Ubis special construction the new sequence (S_)42%5 is called a random walk. In the special ease of 7 1 b= 4 = 5. we call it a symmetric random walk. An outcome of Ny, Xa, ..., Xq is a sequence 21, 02, ..65 sta, Where cach ap is either 1 or = 1 A random walk may be interpreted in several ways, of which we give the following two: 1) A person walks on a road, where he per time unit with probability p takes one step to the right and with probability q takes onc step to the left. At time 0 the person is at state Ey. His position at time n is given by the random variable S.,. [fin particular, p = q = $, this process is also called the “drunkard’s walk”. z Two persons, Peter and Paul. are playing a series of games. In one particular game, Peter wins with probability p, and Paul wins with probability q. After each game the winner receives | $ from the loser. We assume at time 0 that they both have won 0 §. Then the random variable Sy describes Peter's gain (positive or negative) after n games, ie. at time n. We inention Theorem 1.1 (The ballot theorem). At an election a candidate A obtains in total a votes, while another candidate B obtains b votes, where b 0, is given by as inf. (2) } 4 ‘The corresponding probubility that Paul at some tine has the sum of L$ is roams aud the probability that he at some later time has a positive sum of NV $ is or wn. (3)"} Based on this analysis we introduce pa i= P{return to the initial position at time np, EN, Ju e= P{the first return to the initial position at time w}, me N. ne later time} = > fu Notice that Py = fo = 0. if nis an odd number. f P{return to the initial position at »% We shall now demonstrate how the corresponding generating functions profitably can he applied in such situation, Thus we put and F(s)= fas" 7 where we have put py = 1 and fy = 0. It is easily seen that the relationship between these two erating functions isFurthermore, f a 2p, for P= 5 1 f= lim F(y=1- VIA = 1-1-2) = 41, for p= 5 9 fi : 24. for p> 5 we define a random variable T by he symmetric case, where p = In the sy P= 5 nm, if the first return occurs at time n. ‘Then it follows from the above that 7 bas the distribution P{T = 2k} =f and = P{T = 2k- 1} =0, for ke N. ‘The generating function is P(s)=1-¥ hence B{T) = lim F(s) = +90, which we formulate as the expected time of return to the initial position is +70 1.3. The ruin problem The initial position is almost the same as earlier. ‘The two gamblers, Peter and Paul, play a series of games, where Peter has the probability p of winning 1 § from Paul, while the probability is q that he loses 1 $ to Paul. At the beginning Peter owns & 8, and Pan] owns NV —k 8, where 0 < k < N. The games continue, until one of them is ruined. ‘The task hore is to find the probability that Peter is ruined. Let. ag be the probability that Peter is ruined, if he at the beginning has k $, where we allow Uhat k=0.1,.... N. If k = 0, then ag = 1, and if & . then ay = 0. Then consider 0 < & < N. in which case Qk = Pager + qaK~1 We rewrite this as the homogeneous, linear difference equation of second order, k=1,2, SACRE ning the solution of such difference equations, the reader is referred to e.g. the Ventus: Culculus series. We have two possibilities:he starts with k $ then the probability for Peter being r @)-@ = — an = (%) p 2) If instead p= 4 = : then n fork =0, 1,2,- fork=1,2,-.N. , which must. be played before ed number of games /4 problem to finding the expect . with the sum of # $. In this case. We now change the lors is ruined, when Peter starts one of the two gambl fe = Dike + ie tA for k= 12 equation of second order, Given the We rewrite this equation as an inhomogencous lincar difference equation of boundary conditions above, its solution is 1 1) For p# we get - 2, N. me for K=O, 1, 2,0 1 2) For p= = 5 we get instead me = RNA) for k=O, 120. N 1 ie. p= 4. we sum up the results: y In the special ease, where we consider a syrumetric nindome walk, i. p= 5. 8e sum up the 1 Let (Xx) be a sequence of mutually independent identically distributed random variables of distribu- tion given by 1 . P(X =1}=P{Xe=-1}= 5, forn en. In this case, the random variables Sq = 0 and Sou 2 Par Pisin =r) = (2, ) In particular, Xx have the distribution given by for r= n-ne 1... nen. for large ") n, if the first return to Ep uccurs to Lime 7. ‘This random variable has the values 2. 4. 6, ..., with the probabilities Jan = P{T = 2n} )2 on for n EN, where EXT} = 408. For every N € Z there is the probability 1 for the process reaching state Ey to some later time Finally, if the process is at oo Ex, 0 are given numbers, where 0-< J}. <7 < 1, thea P {last visit of By is between 23n and 2yn} = Swap tan—2e neon de = 2 Avesin YF 2 Avesin 3, where we recognize the sum as a mean sum of the integral of Aresin. This implies that P {last visit of Ey before 2nr} ~ = Aresin V7, for r E10. If. One interpretation of this result is that if Peter and Paw! play inany games. then there is a large probability that one of them is almost all the time on the winning side, and the other one is almost all the time on the losing side. P/ 1.4 Markov chains A Markow chain is a (discrete) stochastic process {X(#) | € No}, which has a finite number of states. eg. denoted by £), £2. .... Em, and such that for any 1 < ko, hy... ky S mand every n € N, P{X(n) = ky | X(n = 1) = hyp Ae A X(0) = ho} = P{X(n) Ku a} If furthermore the Markov chain satisfies the condition that the conditional probabilities i Xin- py = PAX(n) =f 1 X(n-1) = 5} do not depend on n, we call the process a stationary Markow chain y tacitly assuming that. they are stationary, shall in the following only consider stationary Markow chains, and we just write Markor chains, A Markov chain models the situation, where a particle moves between the m states Bi, B3, ...5 B where each move happens at discrete times ¢ € N. ‘Then p, represents the probability that the particle in oue step moves from state £ to state B). In particular. py is the probability that the particle stays at state £y We call the pr, the trausétion prububilitics. They are usually lined up in a stochastic matrer Pu Pe Pon Pa Pe Paw Pmt Pat 1 PruIn this matrix the clement p,; in the i-th row and the j-th column represents the probability for the transition from state E, to state E, For every stochastic matrix: we obviously hav Py 20 for every éand 5, Sg =) for every i, thus all swans of the rows ave 1. 7 ‘The probabitities of stale pl” are defined by ph” := P(X(n) =) fori mog ne No ‘The corresponding vector of state is pcs (wr wee vo?) for n © Ny. In particular, the initial distribution is given hy B= (1018 one) Then by ordinary matrix computation (note the order of Uhe matrices). pM =p", hence by iteration —p!”) = pl P", proving that the probabilities of state at time t = n is only determined by the initial condition p\ and the stochastic matrix P, iterated n timers. The elements p{? in P" are called the transition probabilities at step n, and they are given by ( my PAX (em) = 5 | XC) i} We define a probability vector a = (@1.42.....yn) a8 a vector, for whiel 20 fori=h oem, and Yast A probability vector a is called invariant with respect to the stochastic matrix P, or a stationary distribution of the Markov ehiain, if oP he latter mame is due to the Nin+h)ne® wet that if X(A) has its distribution given by a, then every later his also its distribution given by a. hn onder to ese the computation in practice we introduce the following new eoucepts1) We say that a Markoy chain is irreducible, if we to any pair indices (1. j) can find ann =n, © N, such that that py; > 0. This means that the state By can be reached from choice of (i,j). (However, all the n,j € N do not have to be identical). ate Ey, no matter the 2) If we even can choose n € N independently of the pair of indic chain is regular. (i,j). we say that the Markov Remark 1.1 Notice that stochastic regularity has nothing to do with the concept of a regular matrix known from Linear Algebra. We must not confuse the two definitions. oTheorem 1.3 Let P be anim x m regular stochastic matric. +30 1) The sequence (P") converges towards a stochashe limit matrix G for # 2) Foery row in G has the same probability vector B= (Mi. dae Gd where g, > 0 for every i= 1.2 am. 3) If p is any probability vector, then pP"—g — forn—» +2. 4) The regular matria P has precisely one invariant probability vector. g: ‘The theorem shows that for a regular stochastic matrix P the limit distribution is wriquely determined by the invariant probability vector. It may occur for an irreducible Markov chain that P" diverges for n> +7. We have instead y Theorem 1.4 Let P be ani xm irreducible stochastic matrix: 1) The sequence G > iP) converges towards a stochastic limit. matrix G for n — +9< n 2) Every row in G is the same probability vector B= (G1 Gaeev1 Gm) where g, > 0 for ethvert i= 1, 2.....m 3) Given any probability vector p, then for n— +00. 4) The irreducible matrix P has precisely one invariant probability vcctor. namely g Given a Markov chain of the m states Ey, E2,..., Fi. with the corresponding stochastic matrix P ubset C of the states By. Bz, ..., Lm is called closed, if no state outside C can be reached from any state in C. This can also be expressed in the following way: A subset C of the m states is closed, i for every By €C and every E, ¢ C we have p,; = 0. Ifa closed set. only contains oue state, C = {F,}, we call Fy an absorbing state. This is equivalent to Pa = 1. so we can immediately find the absorbing states from the numbers 1 in the diagonal of the stochastic matrix P. , The importance of a closed set is described by the following theorem, which is fair practice, y easy to apply in stim 1.5 A Markow chain is irreducible, if and only if it does not contain any proper closed subset of states A necessary condition of i reducibility of a Markov chain is given in the following theorem:Theorem 1.6 Assume that a Markov chain of the m states Ey. + En ws wreducible, Then to every pair of indices (i,j). 1S i. j Sm there exists an n= nj, such thal L Sn Sm and pl > 0. Concerning the proof of regularity we may use the following method. if the matrix P is not too t (hopefully) reaches a number complicated: Compute successively the matrices P*, until one at | i =n, where all cloments of P" are different from zero. Since we already know that all elements arc 2 0. we can case the computations by just writing * for the elements of the matrices, which are 2 0. We do not have (o compute their exact values. But we must be very careful with the zeros. ‘This inethod is of course somewhat laborious, and one inay often apply the following theorem instead. heorem 1.7 [fa stochastic matric P ts irreducible. ani there exists a positive element in the diay- oual py, > 0, then P is regular. It is usually easy to prove that P is irreducible. The difficult part is to prove that it is also regular We give here another result We introduce for every mx am. irreducible stochastic matrix P the following numbers d, := largest common divisor of all n € N. for which pl? > 0. It can be proved that dy = dz = dy = d, $0 we only have to find one single of the d,-crne. (For one’s own convenience. choose always that d;, which gives the easiest computations) f Theorem 1.8 An irreducible Marken chain is regular, if aud only if d Ifd > 1, then the Markov chain is periodic of period . One may cousider a random walk on the set {1 2. 3..... N} as a Markov chain of the transition ’ probabilities Pant =q and pyign =p. for # = 2. 3. 1, where pg > O og pq 1) Won = pw = 1. then we have a random walk of two absorbing bari 2) If pz = pw.x—1 = 1, then we have a random walk of two reflecting barriers. In this case the corresponding Markov chain is irreducible,2 Random walk Example 2.1 Consider a ruin problem of total capital N 8, where p In every quine the luss/qain is only 50 cents, Is Uris gume more udeantageous for Peter than if the stake was 1 § (i.e. @ smaller probability that Peter ts ruined)? We have 2N +1 states En, Et Egy, where state E, means that A has > 8. 16 A initially has & §. then apy = Oand ay = 1. We got for the values in between ae = PA Fda OS KS BN, which we rewrite as Pldnar ~ ae) = 9 (ae ~ can) Hence by recursion 5 -=(4) (a) =A). ? ay = (ay — pat) + (Oke = Meg) +2 + (1 = a9) +9 = (G0 =G) eG) aforomem k . (= (4) = ap) + ag = SA Z (a1 = 40) +00 To (=a) +40 e vp Now, ao = 1, so we get for k= 2 that ay = ane = 4 (ae) = one fi and we get Then by insertion, G) =] 4 D. D T - aN » (2) “1 aa,These expressions should be compared with cos Gy and azn, 2-50 cents. It clearly follows from y > 4 > p. that dak > ay. Since the a indicate the probability that Peter is ruined, if follows that there is larger probability that he is ruined if the stake is 50 cents than if the stake is 1 3.ch yame Peter has the probability 4 for 1 for tasing (in which case he Example 2.2 Peter and Paul play in total N games. tn ea Find the probability that winning (in which case he reeeives 1 $ from Paul) and probabdlitiy delivers 1 8 to Paul). The games are mutually miependent of each other Peter's total gain never after the start of the games 1s 08. The probability that Peter's gain is never 0 8, is ~SoIn 1 = P{return to the initial position at some tine where Jn = P{lirst return is to time n}. The parity assures that fox) = 0. because we can only return to tu initial position after an even nninber of steps. It follows from p = q = 3 that 1 (2% et ( 2k mai (f )eot=5 k By insertion we get. the probability HAYS aC ALTERNATIVELY. we may use the following considerations which somewhat simplify the task. is 1) If N = 2n is even, then the wanted probability P{S, AOA Sz, A0 A+ A Soy OF This expression is equal to tay, which again is equal to m(2)" aa) 2) IN = 2n +1 is odd, then Sony) is always # 0. Hence, the probability is PS, £0 A Sp ZO A+ A Sop FOAM Songi #0} =P{S, £0AS ZA “AS #0)= (2 Ja on (~ ..) according to the first question.3 Markov chains Example 3.1 Let P be a stochastic matric for a Markov chain of the states Ey, Ea ea 1) Prove that Gor tia tna) gy (anh sca) tan) 1 2 Pie Pa) Pky for IS GRE mM, mm ny E No. Prove that if the Markov chain is irreducible and py > 0 for some i, ten the Marko chain ts regular. 1) Since P™ "24 = Pe Pr P™, and since all matrix clements are > 0, it follows that (atnatona) (madpttiady tna) ofa), sta) fs) pm -rym Yee? pe) > pL) pe" wie 2) Assume that the Markov chain is irreducible and that there is an i, such Uhat py > 0. Since pi? > (pu)” <0, we must have p\"? > 0 for all 1 € No Now, P is irreducible. so to every j there exists an ny, such that PO > 0. index as above) and to every k there exists an nz, such that also Then follow this procedure on all pairs of indices (j. 4). Tf we choose Nj, as the largest of the possible n), and N2 as the largest of the possible m2, then it follows from 1, that pidveXer > plmadptual(Mng Nenad) so, where ny = nj(j) and mg = n2(k) depend on j and k. respectively. Hence all elements of PN'+N? are > 0, so the stochastic matrix P is regular.Example 3.2 Let P be an irreducible stochastic matrir. We tntroduce for every the member dy by 4) = largest common divisor of all n. for which p! > 0. 1) Prove that d, does not depend on i, We denote the common value by d. 2) Prove that if P is regular, then d= 1 1) If we use Example 3.1 with j =, we got utmectinal sy fm)ytna) (ns me oe a and analogously (trata) (ri) (nad faa) Pas 2 Pi Pa” Pes Using that P is irreducible, we can find my and 1g, such that ply? > 0 and py" > 0. Let 11 and rg be as small as possible. Then rs smad 5 yim 5 0 pir) sa) 50 and py in Hence, dj|my + ng and dyin + my, where “a |b” means that a isa divisor in b. By choosing nz = mdg. such that pf?) > 0, we also get (nytnatos) (70) g(a) a) sg ple) gira) on) thus dina + na + m3 We conclude that dilnz =m - dy If ny = nd; is chosen, such that pj?" > 0. then analogously gly tm tng. thus dylntd, It follows that d) is divisor in all numbers np = m-dy, for which p{{2)A0. Since dy is the largest common divisor, we must have did, Analogously. de|d;. hieuce di = dy. Since i and F are chosen arbitrarily. we have proved 1.. 2) If P is regular. there exists ann € N, such that all pf) > 0. Then also pl!" > 0, m € No. and the largest common divisor is clearly 1, The proof that conversely d = 1 implies that P is regular is given in Example-3.3.Example 3.3 +3. Let P be a stochas: ri assume that d= 1 (of. Eaample ay matriz. of an. irreducible Markov chain t Prove that the N ti exists an N €N, such that we for i means that the Markov chain is regular) at we for alin = N ond all é and j have py,’ > 0 (which t 7 : Lin: One may in the proof use without separate proof the following and esult from Number Theory: Let Ay. AQ, ; € ‘ Me ©, ag EN have the largest common divisor 1, Then there exists an NEN, such that for all n> N there are integers e(n), c2(n), ---, ce(n) EN, such tat k n= ej(n) aj. j=l (nay) Since P is irreducible, we can to every par of indices (i, 7) find ni € N- such that pj)" > 0 Since d = 1, we have for every index “7” a finite sequence Gir, @i2) +++ Gin €N, such that the largest (4) > 0. Then hy the result from Number Theory mentioned ir N, can find ci (™), ++ Gin, (n) € NY such that 1 the hint there exists an N,. such that one to every 2 m n= YE eas (t2) a5. =1Then lot 2 max {nj} + max (Ny). f'n > Ny then Pip > pidgin md Since n ~ 1, = N,. it follows that Mm my = Deel) aye m and we conclude that tm) sf) n=) sof) fun pi a pirerpie-re) > le) TT n, (ples) > 0 i ‘This is true for every pair of indices (7. J). thus we conclude that P is regular Example 3.4 Let P be an m xm stochastic matrix. 1) Prove that if P is regular, then P® is also regular. cnn one conclude that P? is also irreducible? 2) Assuming instead that P is irreducibl and alla, j. In particular . thus P? is also 1) If P is regular. then there is an NV € N. such that pl? > 0 for all m > cay) ay regular ) for all (3). Now. pi?) are the matrix elements of P? = (P?) 2) The answer is "no"! In fact, ( ‘ b ) is irreducible, while (23)0)-G2) is not. Example 3.5 Let P be an m xm stochastic matrir. Assume that P is irreducible, and that there 1s ani, such that Po >o and ph >, Prove that P is regular, ise the largest common divisor for 3 and 5 The result follows iinucdiately from Example 3, d= Notice that 0,19) ) 2 my? mene Pe ZP Pe >, A= (Di) > 0, pl = (iP) >0. pl =f WP)" > 0. ance the sueceedine p!" 5 ; hence the succeeding p,, > 0, n 2 12, because one just multiply this sequence successively by BS.Example 3.6 Let P be an m xm irreducible matrix. Prove for encry pair (inj) (where LS 1. jm that there exists ann omic nites Prove f yee aaa depending on i und j. such that 0. When P is irreducible, we can get from every state E, to any other state, When we sketch the graph we see that it must contain a eycle. EB, > EB, = By. where (i), fg. .... i) is a permutation of (1. 2,....1). It follows that we can get from every Fy to any other Fy inn steps. where 1 0. Example 3.7 Let P be « stochastic matrix for an irreducible Markow chain of the states Ey. Ea En. Given that P has the invariant probability vector ( 14 1 ) = 4 mm m Prove that P is double stochastic. The condition gP = g is written ZY mak. ts Dass ™ ‘This proves that the sum of every column is 1, thus the matris is double stochastic. Bxample 3.8 Given a regular Markov chain of the states Ey. Fay... Fm and with the stochastic matric P. Then lim pf? where & = (9. go ---1 9m) is the uniquely determined invariant probability vector of P. Prove that there exist a positive constant K and a constant a 0. 1|, suck thal fp? = | 0 for all n & np and all i,j = 1, 2.-..,m- Let 2) be the column vector which bas | in its row number j and 0 otherwise, Then ny on (PI. ni ite mindy? [mayand m, = min vf and MZ = max pi, then Mi, _ for all j. and Ty mt M2 form > 1 wy Thus |p? ay] 0, hence Q is reg) Then g = gQ implies that n= 3 no 492 go = aga + = 2 gs = 3m + 3S y= (l-a)g3 + G94 3 We ¢ ,=4 eg = 3m By adding the first and the third equation we get gi + ys = 91 + ¥ ge = we Col f th eq B yy = (1 a)as, _4 3B, 93 = al —a)g3 = 3 aan = 21a) and 3.304 : Nes l=gtgetyst yay {a4 at 5720 — a} = g{4+21- a)} = 1 (6 ~ 2a),henee, a= and the invariant probability vector is 1 3 8 (AE Daa" Example 3.12 Given a Markov chain of four states Ey. Ea, Ex and By and with the stochastic matrix bags 6 i147 =[% 244 P=to a if 0001 1. Find for P its invariant probability vector(s). 2. Prove for a randomly chosen initial distribution PO = (a9, 30-70-80) for the distribution p!" = (ay.8,.%u.5u) that Oy + Ba be = G) (ou + By +0) 3. Let p = (1,0,0.0). Find p™. p@ and p™. Given a sequence of random variables (Y'4)% g by the following: The possible values of ¥y are 1. 2, 3. 4, and the corresponding probabilities are ay, Sy. J and dq. resp. (as introduced above) Prove for any initial distribution p) = (ap. fo. 70.50) thal the sequence (Yn) converges in probability towards a random variable ¥. Find the distribution of Y. 1) The last coordinate of the matrix equation g = g P with g — (0.3.7.6) is given hy 1 qleta+a)+5, ths a+ g+9=0 Now a, 4,9 2 0. soa =f (0,0,0, 1). 9 = 0. and hence 6 = 1. ‘The only invariant probability vector is 2) Consider again the last. coordinate, ; _! Bu = 7 (Qn + Bnet tar) + bea We have in general 6= 1 ~ (a +344), so T= (0+ Bn +90) = 5 (On-1 + Bot + nat) +L = (Qn + Bans + a1)