STOCHASTIC PROCESSES Second Edition Sheldon M. Ross University of California, Berkeley | JOHN WILEY & SONS, INC. | ‘New York * Chichester * Brisbane * Toronto * SingaporeACQUISITIONS EDITOR Bad Wiley 11 MARKETING MANAGER Debra Rieger SEMIDR PRODUCTION EDITOR Tony VonOrsis MANUFACTURING MANAGER Dorothy Scar ‘TEXT AND COVER DESIGN A Good Thing, In. PRODUCTION COORDINATION Elm Siret Publishing Series, Ie “This book was et in Times Roman by Bi-Comp, Ie. end printed ad bound by CourenStoughton, The over Wes printed By Phoenix Cole Recognizing the importance of preserving what has boen wet, it sa poe of John ‘Wily & Sons, Ie to hve Books of enduring value published in the United Sats printed on aci-tree paper and we exert ou Bet efor 0 that end, ‘The paper ie this book was manticure by il whose forest management progrtns ingle ssained yes harvesting of its inbeviands. Sutsined yield harvesting principles ‘tear thatthe auiber of tees cu each year does Got exceed the amount of new BrOwt, Copyright © 1996, by John Wiley & Sons, Ine. Al ig reserved. Published simultaneous io Canada Reproduction of tanlation of 2 pat of thir woek beyond that pert by Sections 107 and 108 of the 1976 United States Copyright ‘Ac without the permission ofthe copyright ‘owaer i unlawful. Requests for permission ‘or further information should be addressed to the Permissions Departmen, John Wily & Sons, Ine Library of Congress Cataloging-tn-Publcation Deva: Ros, Sheldon M. ‘Stochastic proctssesSheldon M. Ross-—2nd ed. Ioclaesbibigrapical references and index ISBN 0471-12062. otal paper) 1 Stochastic processes. L Tile OADERES 1996 siga—-a20 osgoi2 cr ot in the United states of Ameria 1098765432 On March 30, 1980, a beautifl six-year-old girl died. This book is dedicated to the memory of Nichole PornarasPreface to the First Edition ‘This text is a nonmeasure theoretic introduction to stochastic processes, and as such assumes a knowledge of calculus and elementary probability. In it we ‘attempt to present some of the theory of stochastic processes, 10 indicate its diverse range of applications, and also to give the student some probabilistic {intuition and insight in thinking about problems. We have attempted, wherever possible, to view processes from a probabilistic instead of an analytic point of view. This attempt, for instance, has led us to study most processes from a sample path point of view. would like to thank Mark Brown, Cyrus Derman, Shun-Chen Niu, Michael Pinedo, and Zvi Schechner for their helpful comments. SHELDON M. RossLa nn CC AR A A Preface to the Second Edition ‘The second edition of Stochastic Processes includes the following changes: (@) Additional material in Chapter 2.on compound Poisson random vari ables, including an identity that can be used to efficiently compute moments, ‘and which leads to an elegant recursive equation for the probability mass function of a nonnegative integer valued compound Polsson random variable; (i) A separate chapter (Chapter 6) on martingales, including sections on the Azuma inequality; and (iil) Anew chapter (Chapter 10) on Poisson approximations, includi both the Stein-Chen method for bounding the error of these approximations and a method for improving the approximation itself. Tn addition, we have added numerous exercises and problems throughout the text. Additions to individual chapters follow: Tn Chapter 1, we have new examples on the probabilistic method, the ‘multivariate normal distribution, random walks on graphs, and the complete ‘match problem. Also, we have new sections on probability inequalities (includ- ing Chernoff bounds) and on Bayes estimators (showing that they are almost never unbiased). A proof of the strong law of large numbers is given in the Appendix to this chapter. 'New examples on patterns and on memoryless optimal coin tossing strate- gies are given in Chapter 3, ‘There is new material in Chapter 4 covering the mean time spentin transient states, as well as examples relating to the Gibb’s sampler, the Metropolis algorithm, and the mean cover time in star graphs. ‘Chapter 5 includes an example on a two-sex population growth model. Chapter 6 has additional examples illustrating the use of the martingale stopping theorem. ‘Chapter’ includes new material on Spitzer's identity and using it to compute ‘mean delays in single-server queues with gamma-distributed interarrival and service times. Chapter 8 on Brownian motion has been moved to follow the chapter on ‘martingales co allow us to utilize martingales to analyze Brownian motion,PREFACE TO THE SECOND EDITION ater on stochsticorder relation now includes section on associated art on ee ellos new examples ilzingcoulingin coupon calle in packing probs. Ing an TS nan al tose who were kind enough fo writ and send cone re the ha edition, wit particular thanks to He Sheng, Same ehorn, Reet Ker, nes Mats, Erol Pekoz, Maria Rieder, sere Me Rol fr thes many hepfl comments : ssto0n M. Ross Contents CHAPTER 1. PRELIMINARIES 1 LL. Probability 1 1.2, Random Variables 7 13, Expected Value 9 14, Moment Generating, Characteristic Functions, and Laplace Transforms 15 15. Conditional Expectation 20 15.1. Conditional Expectations and Bayes Estimators 33 1.6, ‘The Exponential Distribution, Lack of Memoi Hazard Rate Functions 35, 1,7. Some Probability Inequalities 39 18. Limit Theorems 41 1.9. Stochastic Processes 41 Problems 46 References 55 Appendix 56 and CHAPTER 2. THE POISSON PROCESS 59 2.1, The Poisson Process 59 22. Interarrival and Waiting Time Distributions 64 23. Conditional Distribution of the Arrival Times 66 23.1. The MIGII Busy Period 73 2.4, Nonhomogeneous Poisson Process 78 25. Compound Poisson Random Variables and Processes 82 25.1. A Compound Poisson Identity 84 252. Compound Poisson Processes 8726. CHAPTER 3. 3a. 33. 34. 35. 36. BR. 38. CHAPTER 4. 4a, 42. 43. 4a. 45. 46. ‘CONTENTS Conditional Poisson Processes 88 Problems 89 References 97 RENEWAL THEORY 98 Introduction and Preliminaries 98 Distribution of N() 99) Some Limit Theorems 02 33.1. Wald’s Equation 104 332. Back to Renewal Theory 106 ‘The Key Renewal Theorem and Applications 109 34.1. Alternating Renewal Processes 114 3.42. Limiting Mean Excest and Expansion of ma) 119 3.43. Age-Dependent Branching Processes 121 Delayed Renewal Processes 123 Renewal Reward Processes 132 36.1. A Queueing Application 138 Regenerative Processes 140 37.4. The Symmetric Random Walk and the Are Sine Laws 142 Stationary Point Processes 149 Problems 153 References 161 MARKOV CHAINS 163 Introduction and Examples 163 Chapman-Kolmogorov Equations and Classification of States 167 Limit Theorems 173 ‘Transitions among Classes, the Gambler's Ruin Problem, ‘and Mean Times in Transient States 185 Branching Processes 191 ‘Applications of Markov Chains 193 46.1, A Markov Chain Model of Algorithmic Efficiency 193 4.62. An Application to Runs—A Markov Chain with @ Continuous State Space 195 463. List Ordering Rules—Optimality of the ‘Transposition Rule 198 4.7. Time-Reversible Markov Chains 203 48. Semi-Markov Processes 213 Problems 219 References 230 CHAPTER 5. CONTINUOUS-TIME MARKOV CHAINS 231 Sl, Introduction 231 52. Continuows-Time Markov Chains 231 53.-Birth and Death Processes 233 54. The Kolmogorov Differential Equations 239 5.4.1. Computing the Transition Probabilities 249 55. Limiting Probabilities 251 56. Time Reversibility 257 56.1. Tandem Queues 262 562. A Stochastic Population Model 263 5.7, ‘Applications of the Reversed Chain to Queueing Theory 270 S74. Network of Queues 271 57.2. The Eriang Loss Formula 275 5.73. The MIGII Shared Processor System 278 58 Uniformization 282 Problems 286 References 294 CHAPTER 6. MARTINGALES 295 Introduction 295 61. Mariingales 295 62. Stopping Times 298 63. Azuma’s Inequality for Martingales 305 64. Submartingales, Supermartingales, and the Martingale Convergence Theorem 313 65. A Generalized Azuma Inequality 319 Problems 322 References 327 CHAPTER 7. RANDOM WALKS 328 Introduction 328 7A. Duality in Random Walks 329xiv 12. 13. 14 15. 84. 8s. 86. 87. 88 CHAPTER 9. 94 Some Remarks Concerning Exchangeable Random Variables 338 Using Martingales to Analyze Random Walks 341 ‘Applications to G/G/1 Queues and Ruin Problems 348 7A. The GIGI Queue 344 742. A Ruin Problem 347 Blackwell's Theorem on the Line 349 Problems 352 References 355 |. BROWNIAN MOTION AND OTHER MARKOV PROCESSES 356 Introduction and Preliminaries 356 Hitting Times, Maximum Variable, and Are Sine Lays 363 Variations on Brownian Motion 366 83.1. Brownian Motion Absorbed at @ Value 366 832. Brownian Motion Reflected at the Origin 368 833. Geometric Brownian Motion 368 834. Integrated Brownian Motion 369 Brownian Motion with Drift 372 84.1. Using Martingales to Analyze Brownian Motion 381 Backward end Forward Diffusion Equations 383 Applications of the Kolmogorov Equations to Obtaining Limiting Distributions 385 86.1. Semi-Markov Processes 385 862. The MIGII Queue 388 863. A Ruin Problem in Risk Theory 392 ‘A Markov Shot Noise Process 393 Stationary Processes 396 Problems 399 References 403, STOCHASTIC ORDER RELATIONS 404 Introduction 404 Stochastically Larger 404 contents 92. 93, 94. 95, 96. 91. CHAPTER 10. 10.4, 102, 103, Coupling 409 921, Stochastic Monotoniiy Properics of Birth and Death Process 416 922, Exponential Convergence in Markov Chaine 418 Hazard Rate Ordering and Applications to Covnti Processes 420 * . Likettood Ratio Ordering 428 Stochascally More Variable” 433 Applications of Variablly Orderings 37 964. Comparison of GIGIT Queves 439 942. ARetewal Process Appliation 40 963. A Branching Process Application 443 Asocated Random Variables 6 Problems 449 References 456 POISSON APPROXIMATIONS 457 Introduction 457 Brun’s Sieve 457 ‘The Stein-Chen Method for Bounding the Error of the Poisson Approximation 462 Improving the Poisson Approximation 467 Problems 470 References 472 ANSWERS AND SOLUTIONS TO SELECTED PROBLEMS 473 INDEX 505CHAPTER 1 Preliminaries 1.1 Propasmity A basic notion in probability theory is random experiment: an experiment Whose outcome cannot be determined in advance. The set of all possible ‘outcomes of an experiment is called the sample space of that experiment, and we denote it by S. ‘An event is a subset of a sample space, and is said to occur if the outcome of the experiment is an element of that subset. We shall suppose that for each event E of the sample spate $a number P(E) is defined and satisfies the following three axioms*: Axiom (1) 0= P(E) S11 Axiom (2) P(S) = 1. Axiom (3) For any sequence of events E,, E:, ... that are mutually exclusive, that is, events for which E,E) = @ when j (where is the nal set), (ge) We refer to P(E) a8 the probability of the event E. Some simple consequences of axioms (1), (2), and (3) are: ALA IEE C F, then P(E) = PUP). 12. PEE P(E) where E* is the complement of E. LL3. PCS &)) = 31 PCE;) when the E; are mutually exclusive. 1A, P(U; E) = 3) PCE). ‘The inequality (1.1.4) is known as Boole's inequality. * Aciualy P(E) will only be defined forthe so-called measurable events ofS. Bu this estition aed no concern 12 PRELIMINARIES ‘An important property ofthe probability function Ps that it is continuous. ‘To make this more precise, we need the concept of a limiting event, which wedefine as follows: A sequence of events {E,,n = 1}ssaid tobe an increasing Sequence if Ey C Basi, m= 1 and is said to be decreasing if E, 2 Bnei = 1 It (E,, n = 1} is an increasing sequence of events, then we define a new event, denoted by lim... E, by B=) B when B,C Bua m1. Similarly if {E,, n = 1} 8 a decreasing sequence, then define lim... En by fee Ae, wee ne. ‘We may now state the following: {OPOSITION 4.3.1 IS {E,, » = 1) is either an increasing or decreasing sequence of events, then lin 16) = (in) Proof Suppose, first, that {E,, m 2 1} isan increasing sequence, and define events Fane iby R- “That is, F, consists of those points in E, shat art notin any of the earlier E, 1< 1 Ins easy to verily that the Fare mutually exclusive events such that Ur-Ue ma Ur=Ue toratinet PROBABILITY : Thus tim SPC) = hn PEED) which proves the result when (E,, 2 18 =m 2 1) increasing Ut (Em n= 1) 4 deresing sequene, hen ‘quence: hence, oe De), »(We)=tereo But, as UF Es = (7 E,)', we see that 1-(Ae) U-PED or, equivalent, hich proves the result Exnne4 s(n) Consider a population consisting of individual able to prodice ofgpring of the same kind. The numberof individualsinitially present, denoted by Xo is called the size of the zeroth generation. All offspring of the zeroth generation constitute the first generation and their number is denoted by X;. In general, let 1X, denote the sizeof the nth generation. Since X, = Oimplies that Xe = 0, it follows that P{X, = 0} is increasing and thuslim,« P(X. ~ 0} exists. What does it represent? ‘To answer this use Proposition 11.1 as follows: {1imx, = 09} {y%-9} {the population ever dies out}. Jim PEX, = 0} ‘That is, the limiting probability that the th generation is void of individuals is equal to the probability of eventual extinction of the population. Proposition 1.1.1 ean also be used to prove the Borel-Cantelli lemma. PROPOSITION 1.2.2 “The Bore-Cantelli Lemma Let Ey, Es,--- denote a sequence of events. If Sree, then Plan infinite number of the F; occur } Proof The event that an infinite aumber ofthe F, occur, called the lim sup bbe expressed as, finsp = A UE ‘Tis follows since if an infinite number of the F, occur, then Uj. occurs for each thus Mr Uz, , occurs. On the other hand, if 3, Ure &, occurs, then [Uns E,oceurs for each n, and thus foreach n at least one ofthe occurs where i= ‘ns and, hence, an infinite number ofthe E, oceu. PROBABILITY, ‘As Ul Bim & 1, isa decreasing sequence of event, it follows from Proposition 11 that and the result is proven Exams 1.40) Let Xi, Xo... be such that PIX, Unt 1- PIX, th one. It we let E, = {X, = 0), then, as 2; P(E,) < eit follows from the Borel-Cantelli lemma that the probability that X, equals 0 for an infinite number of 7 is equal to 0. Hence, for ail n sufficiently lng, Xq must equal, andso we may conelude that, with proba iy 1, lim X, For a converse to the Borel-Cantelli lemma, independence is required, PROPOSITION 4.1.3, Converse to the Bore!-Cantell Lemma IE), Ey... are independent events such that Srey-«, then lan infinite number of the E, occur} = 1.6 PRELIMINARIES Proof tte rite) =i) 8} -ser(C) [:-(Q)] Now, (Aa) flee erinegentney = fre - PED) afer cyteingutyt x0") ~en(-$ 706) =0 since $6) = fora Hen he et ese Exam 1:20) Let Xi, X;,..-be independent and such that PIX, = 0} = Un =1-P(X.= 1, neh Af we let Ey = = @ it follows from Proposition 1.13 that £, occurs. infinitely often. Also, as Zyei P(ES) = 2 it also follows that E;, also occurs infinitely often. Hence, with probability 1, X, will equal 0 infinitely often and will also equal | infinitely often, Hence, with probability 1, X, will not approach a limiting value as n> ©. RANDOM VARIABLES. 7 1.2 Ranpom VARIABLES Consider a random experiment having sample space S. A random variable X is a function that assigns a real value to each outcome in S. For any set of real numbers A, the probability that X will assume a value that is contained in the set A is equal to the probability thatthe outcome of the experiment is contained in X°¥(A). That i, PIXE Ab = PUX(A)), ‘where X-(A) isthe event consisting ofall pints s € S such that X(3) € A. ‘The distribution function F of the random variable ¥ is defined for aay real number x by Fa) = PIX Sa} = PIX € (—™, 2). We shall denote 1 ~ F(a) by F(x), and so Flay = PX >a, A random variable X is said to be discrete if its set of possible values is countable, For diserete random variables, FQ) = 5 PX= yh ‘A random variable is called continuous if there exists a function f(x), called the probability density function, such that P(Xisin B}= [, /() dx for every set B. Since F(x) fla) di Flows that soy - Zr. The joint distribution function F of two random variables X and Y is de- fined by F(x, y) = PIX =x, ¥ Sy) The distribution functions of X and ¥, Fy) = P(X Sx) and Fry) = PLY = yh,8 PRELIMINARIES can be obtained from F(s,y) by making use of the continuity property ofthe probability operator. Specifically, let ys," = 1, denote an increasing sequence converging to =. Then as the events (X= x, Y's y,},n = 1, are increasing and lim(X=x,Y=y)= UX <2, yb = Osa, it follows from the continuity property that lim POs x, sy.) = PKS), or, equivalently, Fr) lim Foxy). Similarly, Fels) = lin Fes 9) ‘The random variables X and Y are said to be independene if F(x, y) = Fe) Fr(y) for allx and y. ‘The random variables X and Y are said to be jointly continuous if there exists a function f(x, y, called the joint probability density function, such that P(xisin A, YisinB}= J [, fle») dyde fof all sets A and B. i efon dtrbon of ny caesion XX... Ne orandom abies FUR ha) = PUM Stay Xe Sah Furthermore, the n random variables are said to be independent if FOR) = Fil) + Fe) where F(x) = lim Fons. -4%): EXPECTED VALUE 1.3 Expectep VALUE ‘The expectation or mean of the random variable defined by denoted by ELX). is xdF(x) aay FIX) [rapeyde — itXsoniaoe “|Sxper=x} itis discrete provided the above intgral exists ‘Equation (13.1) also defines the expectation of any function of 4% say n(X), Since A(X) i itself @ random variable, it follows from (1.3.1) that EtH(x)] = f° 24h), where Fis the distribution function of A(X). However, it can be shown that this is identical vo f-, M(x) dF(x). Tha i, 132) E(W(x)] = JAG) AFC), “The variance of the random variable X is defined by Var X= BUX - ELX)) = E[X)] - FAX, ‘Two jointly distributed random variables X and Y are said to be uncorre- lated if their covariance, defined by Cov(X, ¥) = BK = EX)(Y ~ EY] = E[XY] ~ FIXIELY] is zero. I follows that independent random variables are uncorrelated. How. ver, the converse need not be true, (The reader should think of an example.) “An important property of expectations is that the expectation of a sum of random variables is equal to the sum of the expectations. (33) e[Sx]-% FIX)» PRELIMINARIES “The corresponding property for variances is that 134 var[ $x] = Svancxy +25 concn x). Exammue 1.3(4) The Matching Problem, At a party n people put their hatsin the center of aroom where the hats are mixed together. Each person then randomly selects one, We are interested in the ‘mean and variance of X—the number that seleet their own hat To solve, we use the representation NeXt Xetot Key where 1. ifthe ith person selects his or her own hat {0 otherwise. Now. as the ith person is equally likely to select any of the m hats, it follows that P(X, = 1} = Lin, and so EX] = Un, Also Cov(X:, Xi) = EEX] ~ ELXJELX) Now, | { ih tangy bo own and thus E[XX) = PIX = 1,.)= 1) = PIX, = PIX) = 1)%i= 1} ie EXPECTED VALUE Hence, covert) sey (9) “ED ‘Therefore, from (1.33) and (1.3.4), EUX}=1 and ‘Thus both the mean and variance of the number of matches are equal to 1. (See Example 1.5(f) for an explanation as 10 why these results are not surprising.) Exurie1.3(0) Some Probability Identities. Let Ay, Az... Ax denote events and define the indicator variables Jj 1,..-.n by 1 if Ajoceurs (0 otherwise. Letting. then N denotes the qumber of the A,, 1 = jn, that occur. A useful identity can be obtained by noting that itn=0 But by the binomial theorem, 236 a= or 3(") cry Co x sine (M) = enim “iaPRELIMINARIES Hence, if we let p-[l tyro “lo itn=o, ‘then (1.3.5) and (1.3.6) yield 118 (eo _ 037) 3 (em ‘Taking expectations of both sides of (1.37) yields 038) eui=wi-e[(%)] + +crre[(")] an = Pfat least one of the A, occurs} -1(04) FIM= [81] = SP, «((%)| = Efaumber of pars ofthe A, that occur] ~#[34] . EXPECTED VALUE and, in general, by the same reasoning, SO ee Hence, (1.3.8) isa statement of the well-known identity (a SAA) - DE MAA) + VET PAAAY ret (=I !P(AAz: + + Ap): ‘Other useful identities can also be derived by this approach, For instance, suppose we want a formula for the probability that exactly rrof the events A,,...., Ay Occur. Then define Ce "(0 otherwise and use the identity \O ayes Taking expectations of both sides of the above yields en) S-("*') (J)ee a PRECeMNaRiEs (1.3.9) Plexactly of the events Ay, : Sey (T yd Pai A.) occur} ‘As an application of (1.39) suppose that m balls are randomly put inn boxes in Such a way that, independent of the locations of the other balls, each ball is equally likely to go into any of the ' boxes. Let us compute the probability that exactly 7 of the boxes fare empty. By letting A, denote the event that the ith box is empty, wwe see from (1.39) that ‘exactly r ofthe bores are empty) SoC where the above follows since Z)<-<,, consists or ( ® ) terms ‘and each term in the sum is equal to the probability that a given set of r + i boxes is empty. ‘Our next example illustrates what has been called the probabilistic method. ‘This method, much employed and popularized by the mathematician Paul Erdos, attempts to solve deterministic problems by first introducing a probabil ity structure and then employing probabilistic reasoning, Exawrut 4.3{c) A graph is a set of clements, called nodes, and a set of (unordered) paits of nodes, called edges. For instance, Figure 1.31 illustrates a graph with the set of nodes NV = {1, 2,3, 4 5) and the set of edges E = {(1,2) (1,3) (1,5), 2,3), 2,4), @.4), G_5)}, Show that for any graph there is asubset of nodes such thag af least one-half of the edges have one of their nodes in A and the other in A*, (For instance, in the graph illustrated in Figure 1.31 we could take A = (1,2, 4).) Solution. Suppose that the graph contains m edges, and arbitrarily ‘number them a8 1,2,.-. ,m. For any set of nodes B, if we let C(B) denote the number of edges that have exactly one of their nodes in B, then the problem isto show that max C(B) = m/2. To verity this, let us introduce probability by randomly choosing a set of ‘nodes $ so that each node of the graph is independently in $ with probability 1/2. If we now let X denote the number of edges in MOMENT GENERATING, CHARACTERISTIC FUNCTIONS, AND LAPLACE as igure 1.3.1. A graph, the graph that have exactly one of thie nodes in S then Xi 0 andom variable whose set of possible values is all of the possible values of C(B). Now, letting X; equal 1 if edge i has exactly one of its nodes in $ and Tetting be 0 otherwise, then / mid. aixi=e[Sx]=Sw Since atleast one of the possible values of a random variable must be at least as large as its mean, we can thus conclude that C(B) = ‘mi2 for some set of nodes B. (In fact, provided that the graph is such that C(B) is not constant, we can conclude that C(B) > ‘mI? for some set of nodes B.) Problems 1.9 and 1.10 give further applications of che probabilistic method, 1.4 Moment Generatinc, CHARACTERISTIC Functions, anp LaPLace TRANSFORMS ‘The moment generating function of X is defined by Wo Ele] e dF)16 PRELIMINARIES All the moments of X can be successively obtained by differentiating y and then evaluating at ¢ = 0. That is, WO = Elxe) vO = Exe] we = ELxre, Evaluating at ¢ = 0 yields vO) = EI, neh It should be noted that we have assumed that itis justifiable to interchange the differentiation and integration operations. This is usually the case. ‘When a moment generating function exist, it uniquely determines the distribution. This is quite important because it enables us to characterize the probability distribution of a random variable by its generating function. Exane d.4(a) Let X and ¥ be independent normal random vari- ables with respective means yx, and jz and respective variances able 14 Diserete Moment Probabity Probability Mess _Generting Bistbation Function: pe) __Fanaio,g) Mean _ Variance Binomial with par (ws (es = py me etl A) ‘ameter man (")e~ pom Gapst : dessa Poles wit pee Crem A mame ASO Oat a0 Geometric wth —_p(l- p)* lp ee te 7 yar Negnive binomial f= 1 m=) ‘ith parame t)ea-e. F tenn renrth, MOMENT GENERATING, CHARACTERISTIC FUNCTIONS, AND LAPLACE ” ojand 0}. The moment generating function of their sumis given by Wror() = Efe) (eEle"] = Md = exp{(u: + wa)t + (o} + of) P12}, (byindependence) where the last equality comes from Table 1.4.2. Thus the moment generating function of X + Y is that of a normal random variable with mean yy + ys and variance of + oj. By uniqueness, this is the distribution of X + ¥. {As the moment generating function of a random variable X need not exist, it is theoretically convenient to define the characteristic function of X by = Ele, 2 0 PQs) = fle arta), ‘This integral exists for complex variables s = a + bi, where a = 0. Asin the ‘case of characteristic functions, the Laplace transform uniquely determines the distribution. ‘Beta with parameters a,b, a > 0, >0 Uniform over (a, 6) Exponential wth parameter A > 0 [Normal wth parameters (4, 2°) ‘Table 142 Gamma wii i 20 PRELIMINARIES ‘We may also define Laplace transforms for arbitrary functions inthe follow= ing manner: The Laplace transform of the function g, denoted g, is defined by 86) = [lewdscn) provided the integral exists. It can be shown that g determines g up to an additive constant, 1.5 Conpitionat Expectation If X and ¥ are discrete random variables, the conditional probability mass function of X, given Y = y, is defined, forall y such that PY = y} > 0, by ax|¥e ypu PEERY) Pum a|¥~ y= ‘The conditional distribution function of X given V = y is defined by Forly) = PX = x1¥ = yh and the conditional expectation of X given Y = y, by E[X|Y =») = J xdFlrly) = S xP =21¥ = yh If X and ¥ have a joint probability density function f(x, y), the conditional probability density function of X, given Y = y, is defined for all y such that fy) > Oby cxtyy = Lees) FEIT) and the conditional probability distribution function of X, given ¥ = y, by Foly)=Pirsx|¥=yb= fi flelyae The contitional expectation of X, given ¥ = ys defined, inthis case, by ELX|Y = y}= f° xflely)ade ‘Thus all definitions are exactly as in the unconditional case except that all probabilities are now conditional on the event that ¥ = y. ee CONDITIONAL EXPECTATION ‘a Let us denote by £[X|¥] that function of the random variable ¥ whose value at ¥ = y is E[X|Y = y]. An exttemely useful property of conditional expectation is that forall random variables X and ¥ asa) EX] = E[ELX|¥I]= | e1x1¥= ylaFy(o) when the expectations exis If Y isa discrete random variable, then Equation (1.5.1) states EIX]= 5 EIXLY= ylPty= yh, While if ¥ is continuous with density f(y), then Equation (1.5.1) says FLX] = [°_ELX1Y = yf ay. We now give a proof of Equation (1.5.1) in the case where X and Y are both discrete random variables. Proof of (1.5.1) when X and ¥ Are Discrete To show E[X] =D EIX1Y = y]PY = y), We write the right-hand side ofthe above as SEIKIY=y1PY= 9) = BY 2p = Y= hPL = yh ~DEwren yay) = Esdras yey = Sarr=9 =EIX and the result is obtained. ‘Thus from Equation (1.51) we see that E[X] is a weighted average of the conditional expected value of X given that Y = y, each of the lerms E(XIY = y| being weighted by the probability of the event on which it is conditioned,PRELIMINARIES Exnnie 1.51) The Sum of @ Random Number of Random Variables. Let X,, X;,--» denote a sequence of independent and identically distributed random variables: and fet W denote nonnegative integer valued random variable that is independent of the sequence ay... We hall compute the moment generating function of ¥ = 3Y X; by frst conditioning on N: Now elo $o-a] =£ [en {rSx}]w-n] ‘m Assuming that all orderings are equally likely, show that the probability that A is always ahead in the count of votes is (n — myn + m). Solution, Let P, denote the desired probability. By condition- ing on which candidate receives the last vote counted we have Pag = Pld abvays ahead A receives fs vor} + P(A always ahead B receives last vote} P(A always ahead| 8 vote) [Now itis easy to see that, given that A receives the last vote, the probability that A is always ahead isthe same as if A had received 2 total of n ~ 1 and Ba total of m votes. As a similar result is true when we are given that B receives the last vote, we see from the above that Poe em Pe eR We can now prove that Pan by induction on.» + m, As it is obviously true when n + m = I—that is, Pi = L—assume it whenever m + m = k. Then whenPRELIMINARIES n+ m= k +1 we have by (1.5.3) and the induction hypothesis mth n+mn—lem* m+antm=1 nem ‘The result is thus proven. ‘The ballot problem has some interesting applications. For exam- ple, consider successive flips ofa coin that always lands on “heads” With probability p, and let us determine the probability distribution of the first time, after beginning, that the total number of heads is equal to the total number of tails. The probability that the frst time this occurs is at time 2n can be obtained by frst conditioning, fon the total number of heads in the first 2n trials. This yields 2n} iirst time equal Pi ime equal = 2a\n headset 201 (””) td =) Now given a total of m heads in the first 2n ips it is easy to see that all possible orderings of the m heads and m tails are equally likely and thus the above conditional probability is equivalent to the probability that in an election in which each candidate receives ‘votes, one of the candidates is always ahead jn the counting until the last vote (which ties them). But by conditioning on whoever receives the last vote, we see that this is just the probability in the ballot problem when m =n ~ 1. Hence, ~ 2a Tt Exe 4.546) The Matching Problem Revisited. Let us recon: sider Example 1.3(a) in which individuals mix their hats up and then randomly make a selection. We shall compute the probability of exactly k matches. First let E denote the event that no matches occur, and to make explicit the dependence on n write P, = P(E). Upon conditioning. ‘on whether or not the first individual selects his orher own hat—call these events M and M'—we obtain P= PE) P(E|M)P(M) + P(E|M9P(M9. CONDITIONAL EXPECTATION Fa Clearly, P(E|M) = 0, and so asa) P.= PEM) Now, P(E| Mis the probability f no matches when n ~ 1 people select from a set of m-~ 1 hats that does not contain the hat of ‘ne of them. This can happen in either of two mutually exclusive ways. Either there are no matches and the extra person does not select the extra hat (this being the hat of the person that chose frst), or there are no matches and the extra person does select the extra hat. The probability ofthe fist ofthese events is Py, which js seen by regarding the exira hat as “belonging” to the extra person. Since the second event has probability [1/(u — 1)|Pu-2 Wwe have PEEIM) = Pe ‘and thus, from Equation (1.54), or, equivalently, ass) Pah To obtain the probability of exactly k matches, we consider any fixed group of k individuals. The probability that they, and onlyPRELIMINARIES they, select their own hats is 1p (a-k noe ae Py where P,-s is the conditional probability that the other. — k individuals, selecting among their own hats, have no matches. As which, for n large, is approximately equal to “YA ‘Thus for n large the number of matches has approximately the Poisson distribution with mean 1. To understand this result better recall that the Poisson distribution with mean A is the limiting distribution of the number of successes in m independent trials, ‘each resulting in a success with probability p,, when np, — A aS n= &, Now if we let 1 ifthe ith person selects his or her own hat x 0 otherwise, then the number of matches, 2 Xi, canbe regardedas the number ‘of successes in m trials when each is'a success with probability Un. Now, whereas the above result is not immediately applicable because these trials are not independent, itis tue that iia rather weak dependence since, for example, PIX, = 1} = Un and PIX, = AX == n= 1), Ak Hence we would certainly hope that the Poisson limit would still remain valid under this type of weak dependence, The results of this example show that it docs, Eumms 15(0) A Packing Problem. Suppose that n points are arranged in linear order, and suppose that a pair of adjacent points is chosen at random. That is, the pair (i, + 1) is chosen with probability 1 ~ I), i= 1, 2,..4n ~ 1. We then continue {o randomly choose pairs, disregarding any pair having a point — CONDITIONAL EXPECTATION jy ab 8 67 8 Figure 1.8.1 previously chosen, until only isolated points remain, We are inter- ested in the mean number of isolated points. For instance, if n = 8 and the random pairs are, in order of appearance, (2, 3), (7, 8), (3, 4), and (4, 5), then there will be two isolated points (the pair (3, 4) is disregarded) as shown in Figure 15.1 If we let 1 ifpoint isisolated ~ [0 otherwise, 11 ip represents the number of isolated points. Hence E{oumber of isolated points] = 5) EU] dr where P,, is defined to be the probability that point iis isolated when there are m points. Let Ph Pap = Pro ‘That is, P, is the probability that the extreme point m (or 1) will be isolated, To derive an expression for P,, note that we can consider the n points as consisting of two contiguous segments, namely, LQeeei and hi+ 1s Since point i will be vacant if and only if the right-hand point of the first segment and the left-hand point of the second segment are both vacant, we see that 156) Pig = PPain Hence the P, will be determined if we ean calculate the correspond ing probabilities that extreme points will be vacant. To derive an ‘expression for the P, condition on the initial pair—say (i, { + 1}—and note that this choice breaks the line into two independent segments—1, 2,...,1— Landi + 2,...,. Thatis, if the initialPRELIMINARIES pair is (i, + 1), then the extreme point n will be isolated if the extreme point of aset of n — i ~ 1 pointsis isolated. Hence we have (n= DP, = Pit + Pr Substituting n — 1 for n gives (n-2) Poteet Pas and subtracting these two equations gives (nm ~A)Py~ (8 ~ 2)Pat Py~ Past ey Since P, = 1 and P; = 0, this yields 1 or Pa or Pe Sow yz aR iskn i=Qn 2 k} let Ey, = 1, .., n, denote the event that component é has survived the frst & shocks. Then pw>n= P(e) a) fee CDP PEE +B To compute the above probability let p denote the probability that 2a given set of j components will all survive some arbitrary shock. Conditioning on the shock's value gives, y= | Plset of jsurvive value is x} d(x) =fa-x/aae), Sinee PCE) = pts PUBED ‘we see that p> y= mot ~ (2) ot + (5) oboe au2 PRELIMINARIES ‘The mean of N can be computed from the above as follows: BIN)=S Pww> ky - BC a sty ahi) =p, The reader, should note that we have made use of the identity EIN] = Soy P(N > 4}, valid for all nonnegative integer-valued random variables NV (see Problem 1.1) Exe 1.8(i) Classifying a Poisson Number of Events. Suppose that we are observing events, and that N the total number that occur, is a Poisson random variable with mean A. Suppose also that each event that occurs is, independent of other events, Glasiied as a type j event with probability py.j = Ty == k 3 p,= 1 Lot denote the numberof type j events that occu, J = 1,..., kad lot us determine thei joint probability mass function For any nonnegative integers ”, j = Then, since N = N,, we have that sky tetn = En, PIN, =m, 7) = PIN, = m),)= 1,2. KIN = n}PIN = 1} +PIN=n, KIN A n}PIN nf = PIN, =n, csk|N =n} PWN = 7} Now, given that there are a total of N = n events it follows, since each event is independently a type j event with probability p,, 1 = i= that Ny, Nyy... Ny has a multinomial distribution ‘with Parameters and py, p ---, Ps. Therefore, PIN, Beek) = ott PRC vy (CONDITIONAL EXPECTATION 38 ‘Thus we can conclude that the Nj are independent Poisson random variables with respective means Ap), = 1, ...,k. Conditional expectations given that Y = y satisfy all of the properties of ordinary expectations, except that now all probabilities are conditioned on the event that (¥ = y}. Hence, we have that BIXIY=y] implying that e[Sxi]-Sewin Also, from the equality E[X] = E[E[X|¥]] we can conclude that E(X|W = w] = ELEX|W = w, YW or, equivalently, EIX|W] = EELXIW, YI ‘Also, we should note thatthe fundamental result FIX] = ELELXIY I] remains valid even when Y is a random vector. 1.5.1 Conditional Expectations and Bayes Estimators Conditional expectations have important uses in the Bayesian theory of statis- tics, A classical problem in this area arises when one is to observe data X = (X, «+. %)) whose distribution is determined by the value of a random variable '@, which has a specified probability distribution (called the prior distribution). Based on the value of the data X a problem of interest is to estimate the unseen value of @. An estimator of @ can be any function d(X) of the data, and in Bayesian statistics one often wants to choose d(X) 10 minimize E[(d(X) — 0|X], the conditional expected squared distance be- ‘oween the estimator and the patameter. Using the facts that conditional on X, d(X) is a constant; and for any random variable W, E{(W ~ ¢)']is minimized when e = E[W]“ PRELIMINARIES it follows that the estimator that minimizes £[(d(X) ~ 6)°|X], called the Bayes estimator, is given by a(x) = E{A1X) ‘An estimator d(X) is said to be an unbiased estimator of 8 if Ela(x)|6] = 6. ‘An important result in Bayesian statistics is that the only time that a Bayes estimator is unbiased is in the trivial case where it sequal to @ with probability 1. To prove this, we start with the following lemma, Lemma 4.5.4 For any random variable Y and random vector Z Ey ~ ELYzpetY|2]] = 0. Proof E(vE(Y|Z]] = ELELYELY 2312), = E(E(Y|Z]E("12I) ‘where the final equality follows because, given Z, E[¥|Z] is a constant and so ELVE[Y12Z]|2] ~ E[¥|ZIEL¥1Z]. Since the inal equality is exactly what we wanted to prove, the lemma follows. Ee ET PROPOSITION 4.5.2 MC P(E[@1X] = 6) #1 then the Bayes estimator £[6)X] isnot unbiased, Proof Letting ¥ = @ and Z = X in Lemma 15.1 yields that as E((@~ EL0|X)E(@AXY = 0. Now let ¥ = E[G|X] and suppose that ¥ i an unbiased estimator of @ s0 that E|Y [4] = 6 Letting Z = awe obtain from Lemma 1.5.1 that ass) E((E(@|X] ~ 8] = 0. Upon adding Equations (15:7) and (1.58) we obtain that E((@ ~ E(@lXDELOIXT + EWE(A1X] ~ 096) = 0. i EXPONENTIAL DISTRIBUTION, LACK OF MEMORY. HAZARD RATE FUNCTIONS 35, B((@ — E[a1X)E[a1X] + (E[eAX] - 06) ~e((@~ E1e|x)/1 =0 implying that, with probably 1, 8 ~ E61] = 1.6 Tue Exponentiat Distrisution, Lack or Memory, anp Hazarp Rate Functions ‘A continuous random variable X is said to have an exponential distribution with parameter A, A > 0, i its probability density function is given by re x20 ro={ 0 x<0, or, equivalently, i its distribution is 1 reo=frno={) ‘The moment generating function of the exponential distribution is given by a 61) Ele] = fo ernet de Al the moments ofXcan aw be obsined by diferent (1.61), and we leave it to the reader to verify that — UX] =A, Varo Ux, ‘The usefulness of exponential random variables derives from the fact that they possess the memoryless property, where a random variable X is said to be without memory, or memoryless, if (162) PUIX>sti|X>=PIX>3} fors,rZ0.36 PRELIMINARIES If we think of X as being the lifetime of some instrument, then (1.62) states that the probability that the instrument lives for at least s + ¢ hours, given that it has survived ¢ hours, is the same as the initial probability that it lives for at least s hours. In other words, ifthe instrument is alive at time r, then the distribution of its remaining life isthe original lifetime distribution. The condition (1.6.2) is equivalent to Ft) = FORO, and since this is satisfied when F is the exponential, we see that such random variables are memoryless. Exam L.6(4) Consider a post office having two clerks, and sup- pose that when A enters the system he discovers that B is being served by one of the clerks and C by the other. Suppose also that A is told that his service will begin as soon as either B or C leaves. If the amount of time a clerk spends with a customer is exponentially distributed with mean 1/A, what is the probability that, of the three customers, A is the last to leave the post office? The answer is obtained by reasoning as follows: Consider the time at which A first finds 2 free clerk. At this point either B or ‘C would have just left and the other one would still be in service. However, by the lack of memory of the exponential, it follows that the amount of additional time that this other person has to spend in the post office is exponentially distributed with mean 1/A. That is, itis the same as if he was just starting his service at this point. Hence, by symmetry, the probability that he'finishes before A must equal i Bune 1.6(8) Let X;, X2,... be independent and identically dis- tributed continuous random Variables with distribution F. We s that a record occurs at time n,n > 0, and has value X, if X, > maxX, X. |), where X, = oo. That is, a record occurs ‘each time a new high is reached. Let 7 denote the time between the ith and the (i + I)th record. What is its distribution? ‘Asa preliminary to computing the distribution of 7, let us note that the record times of the sequence X,, X2, ... will be the same as for the sequence F(X;), F(X)... ,and since F(X) hasa uniform (0,1) distribution (see Problem 1.2), it follows that the distribution of 7 does not depend on the actual distribution F (as long as itis continuous). So let us suppose that Fis the exponential distribution with parameter A= 1 To compute the distribution of 7, we will condition on R; the ith record Value. Now R, =X; is exponential with rate 1. R; has the distribution of an exponeatial with rate 1 given that itis greater than R,. But by the lack of memory property ofthe exponential this a EXPONENTIAL DISTRIBUTION, LACK OF MEMORY, HAZARD RATE FUNCTIONS 37 ‘means that R: has the same distribution as R, plus an independent exponential with rate 1. Hence R; has the same distribution as the sum of two independent exponential random variables with rate 1, The same argument shows that R, has the same distribution as the sum of { independent exponentials with rate 1. But itis well known (see Problem 1.29) that such a random variable has the gamma distribution with parameters (i, 1). That is, the density of Ris given by hO= FF Hence, condoning on els Pin ah [Pte aR EC =fa-ve gy where the last equation follows since if the ith record value equals f, then none of the next k values will be records if they are all less than f Ne di, i=1, turns out that not only is the exponential distribution “memoryless,” but itis the unique distribution possessing this property. To see this, suppose that X is memoryless and let F(x) = P(X > x}. Then Fis += FF. ‘That is, Fsatisties the functional equation als +) = a()8(0. However, the only solutions of the above equation that satisfy any sort of reasonable condition (such as monotonicity, right or left continuity, or even ‘measurability) are of the form ate) for some suitable value ofA. [A simple proof when gis assumed right continu ous is as follows: Since g(s + 1) = g(s)), it follows that g(2/n) = g(I/n + Un) = (Lin). Repeating ths yields g(msin) = g*(Uin). Also g(1) = g(l/n + + Un) = gt(tin). Hence, g(rm/n) = (g(1))%, which implies, since g is Fight continuous, that g(x) = (g(1)). Since g(1) = °(/2) = 0, we obtain 8(2) =e, where A = —log(g(1))}. Since a distribution function is always| 38 PRELIMINARIES right continuous, we soust have Fo) ‘The memoryless property of the exponential is further illustrated by the failure rate function (also called the hazard rate function) of the exponen: tial distribution, Consider a continuous random variable X having distribution function F and density f. The failure (or hazard) rate function A(i) is defined by y= £0. (1.63) 4o-F5 To interpret A(), think of X as being the lifetime of some item, and suppose that X has survived for # hours and we desire the probability that it will not survive for an additional time dt. Thats, consider PLX € (t+ at)|X'> t- Now PIXE (14 di). X>0 PIX> 0) PIXE (14 do} OPS Lode FO = No dt. PIXE (41+ a) |X> 0 ‘That is, A() represents the probability intensity that a -year-old item will fai Suppose now that the lifetime distribution is exponential. Then, by the memoryless property, it follows that the distribution of remaining life for a ‘year-old item is the same as for a new item. Hence A(}) should be constant, This checks out since ‘Thus, the fifure rate function for the exponential distribution is constant. ‘The parameter A is often referred to as the rate ofthe distribution. (Note that the rate is the reciprocal of the mean, and vice versa.) It turns out that the failure rate function A(?) uniquely determines the distribution F. To prove this, we note that 470) FO MO= SOME PROBABILITY INEQUALITIES, Ey Integration yields ~ flank Fiy~ cox{-f.aoat} Letting 1 = 0 shows that ¢ = 1 and so Fo=en{-[aoa} 1.7 Some Pronasiiry Ineguauiries We start with an inequality known as Markov’s inequality. Lemme 2.7.2 Markov’s inoquality 12s a nonnegative random variable, then for any a > 0 PIX= a) = ELXVa Proof Let I{X = a} be 1 if X = a and 0 otherwise, Then, itis easy to see since X= O that allX = o} = X. ‘Taking expectations yields the result, PROPOSITION 4.7.2 Cheroff Bounds Let X be a random variable with moment generating function M(t) = Ele"). Then fora >0 PIX=al = PIXSa}serM() forall <0. Mi) foralle>0“0 PRELIMINARIES Proof Fort>0 PIX = a} = Plo = om) = Elee™ where the inequality follows from Markov’s inequality, The proof for ¢< Os similar Since the Chernoff bounds hold for all rin either the positive or negative quadrant, we obtain the best bound on P[X = a} by using that ¢ that mini- izes e*M(0), Exunne 17(1) Chernoff Bounds for Poisson Random Variables. If Xis Poisson with mean A, then M() = eM", Hence, the Chesnolf bound for P(X = j}is PUK = j} = el, ‘The value of that minimizes the preceding is that value for which = jJA. Provided that A > 1, this minimizing value will be positive and so we obtain in this case that PELE i} SMAI = eH(Ae)i, >A. (Our next inequality relates to expectations rather than probabilities. PROPOSITION 4.7.3 Jensen's Inequality If Fis @ convex function, then ELY/OO) = ERD provided the expectations exit Proof Wewill give a proof under the supposition that fhas a‘Taylor series expansion, Expanding about ~ E{X]and using the Taylor series with a temainder formula yields Fle) =f) + FUN + PO~ 2 fla) + Pade ~ w) since /*(é) = 0 by convexity. Hence, $00) = fla) + PUNK ~ w). 7 STOCHASTIC PROCESSES a “Taking expectations gives that ELF) = fa) + PUDELX ~ wl = fH). ee ——————— 1.8 Limit Tuzorems Some of the most important results in probability theory are in the form of limit theorems. The two most important are: Strong Law of Large Numbers* 10%, X;,-.. are independent and identically distributed with mean y, then fincas + +xyn ea} = Central Limit Theorem ICX;, Xs, ... are independent and identically distributed with mean and variance 0, then Ket X me pfie penne de, fa “Vie ThusitweletS, = Eis X;, where X, Xp... areindependent andidenticaly dissbuted, then the Strong Law of Large Numbers states hat, with probablity 1, S,/n will converge to E[X,]; whereas the central limit theorem states that S, will have an asymptotic normal distribution asm -» =, 1,9 Srocnastic Processes A stochastic process X = {X(), 1 € T} is a collection of random variables. ‘That is, for each rin the index ser T, X(0) is @ random variable. We often interpret ¢as time and call X(0 the state of the process at time f. Ifthe index set Tis a countable set, we call X a discrete-time stochastic process, and if T is a continuum, we call it a coniinuous-time process. ‘Any realization of X is called a sample path, For instance, if events are ‘curring randomly in time and X(¢) represents the number of events that ‘occur in [0,1], then Figure 1.9.1 gives a sample path of X which corresponds “A proot ofthe Song Law of Large Numbers i give i the Appendix to this chaptera PRELIMINARIES Figure 1.9.1. A sample path of X(0) = number of events in (0, te to the initial event occurring at time 1, the next event at time 3 and the third at time 4, and no events anywhere else. ‘A continuous-time stochastic process {X(0), © T} is said to have indepen dent increments if forall ) < fy q, Pi < Ps. Hence, we have Pi — Lbefore i+ 1} + PAI ~ Pi L before i+ 1)) = (Py— P)P(~ 1 before i+ 1} + Py Piislast state} Now, since the event that /~ Lis visited before 7 + 1 implies ‘event that / ~ 2is visited before i, it follows that Pi ~ A before i + 1} < li ~ 2 before i} and thus we can conclude that Qi ~ Lis last state} < Pf is last state} 7 Exwures 1.9(¢) A graph consisting of a central vertex, labeled 0, and rays emanating from that vertex is called a star graph (see Figure 19.3). Let r denote the number of rays of a star graph and let ray i consist of n, vertices, for i = 1... r. Suppose that a particle moves along the vertices of the graph so that itis equally likely to move from whichever vertex it is presently at to any of the neighbors of that vertex, where two vertices are said 10 be neighbors if they are joined by an edge, Thus, for instance, when at vertex 0 the particle is equally likely to move to any of its 7 neighbors. The vertices at the far ends of the rays are called leafs. ‘What is the probability that, starting at node O, the fist leaf visited is the one on ray i, = ly... 7? Soluion. Let L denote the first leaf visited. Conditioning on R, the fist ray visited, yields $1 SLAC = itis ray visited isi) G91) AL Now, ifjis the fist ray visited (that is, the first move of the particle is from vertex 0 to its neighboring vertex on ray j) then it follows, from the remark following Example 1.9(A), that with probability nthe particle wil visit the leaf atthe end of ray jbefore returning 10 0 (for this is the complement of the event that the gambler will be up 1 before being down n — 1). Also, if it does return to 0 before reaching the end of ray j, then the problem in essence begins anew. Hence, we obtain upon conditioning whether the particle reaches the end of ray j before returning to 0 that PL AL i[first ray visited isi} = Un, + (1 ~ Un) PUL =i} i[first ray vsitedis/} = (I= 1n)P{L = i, forj i Léat Figure 1.9.3. A star graph,7 Substituting the preceding into Equation (1.9.1) yields that PPIL= i} = Vin, + (-3 1m) PiL=i} PRopiems LAL. Let N denote a nonnegative integer-valued random variable. Show that EIN] Sriv>a {In general show that if X is nonnegative with distribution F, then ELX| = [Fade and Ex" [[nerFe ae 1.2. If X is a continuous random variable having distribution F show that: (a) F(X) is uniformly distributed over (0, 1); (b) if Uis a uniform (0, 1) random variable, then F-"(U) has distribution F, where F(x) is that value of y such that F(y) = x. 13. Lot X, denote a binomial random variable wth parameters (np). = 1. If mp, + Aas n> ®, show that e Ge PAX, = ei! asn 1.4, Compute the mean and variance of a binomial random variable with parameters n and p. 15, Suppose that m ndependent ial each of which sulin ther ou come 1,2,..., r with respestive probabilities py, ps, .... pare per formed, 3 p= I. Let N, dence the number OF tra esting n utcome i. PROBLEMS a (2) Compute the joint distribution of Nj, ..., N,. This is called the ‘multinomial distribution. (b) Compute Cov(N;, Ni). (©) Compute the mean and variance of the number of outcomes that o not occur. 16. Let X;, Xs... be independent and identically distributed continuous random variables. We say that a record occurs at time n,n > O and has value X, if X, > max(Xy,..., Xe), where Xo = ==, (a) Let Ny, denote the total number of records that have occurred up to (and including) time #. Compute E{N,] and Var(N). (b) Let T= min(n: n > 1 and a record occurs at n}. Compute P{T > rn} and show that P(T < @} = 1 and E[T] = =. (©) Let, denote the time of the first record value greater than y. That T= min(ns X.> 9} Show that T, independent of X;,..That is, the time of the fist value greater than y is independent of that value. (It spay seem ‘more intuitive if you turn this last statement around.) 117. Let X denote the number of white balls elected when k balls ate chosen at random from an urn containing m white and m black balls. Compute E[X] and Var(X). 1.8. Let X; and X; be independent Poisson random variables with means A, and Ay (a) Find the distribution of Xi + X> (b) Compute the conditional distribution of X, given that X + X= of any play being that one of the contestants wins and the other loses. Suppose the players are initially numbered 1,2,....,n. The permutation iy, iis called a Hamiltonian permutation if j, beats ip, f beats é, and i, beats i,. Show that there is an outcome of the round-robin for which the number of Hamiltonians is atleast n!/2"~', (Hine, Use the probabilistic method.) L410, Consider a round-robin tournament having n contestants, and let &,Lu. 142, If P(0= X= a} itis possible for the tournament outcome to be such that for every set of k contestants there isa contestant who beat every member of this set If X isa nonnegative integer-valued random variable then the function P(@), defined for |z| = 1 by Sax PG) = Ele" is called the probability generating function of X. (@) Show that KIPUX= hh. a ge Pe (b) With 0 being considered even, show that P{Xis even} = PCy + Pay (©) IE X is binomial with parameters n and p, show that (xis even) = 1+ 0= 20" (@) If X is Poisson with mean A, show that PUXiseven} = (©) If X is geometric with parameter p, show that PUXiseven} 2p (8) If X isa negative binomial random variable with parameters r and eae fecg Var(X) = ai P\Xiseve) |, show that Ty PROBLEMS o 3. ata, Las. 116. a7. Consider the following method of shuffling a deck of nm playing cards, ‘numbered I through n. Take the top card from the deck and then replace it so that itis equally likely to be put under exactly k cards, for k = 0, 1,...,m = 1, Continue doing this operation until the card that was initially on the bottom of the deck is now on top. Then do it one more time and stop. (a) Suppose that at some point there are k cards beneath the one that ‘was originally on the bottom of the deck. Given this se of & cards explain why each of the possible k! orderings is equally likely to be the ordering of the last k cards. (b) Conclude that the final ordering of the deck is equally likely to be any of the N! possible orderings. (©) Find the expected number of times the shuffling operation is per- formed, AA fair die is continually rolled until an even number has appeared on 10 distinct rolls. Let X, denote the number of rolls that land on side i. Determine: (@) ELX @) E[Xz] {© the probability mass function of X;, (@) the probability mass function of X Let F be a continuous distribution function and let U be a uniform (0, 1) random variable (@) IX = F(W), show that X has distribution function F. (b) Show that ~log(U) is an exponential random variable with mean 1 Let f(x) and g(x) be probability density functions, and suppose that for some constant c, f(s) = eg(x) fr allx. Suppose we can generate random variables having density function g, and consider the following algorithm. Step 1: Generate ¥, a random variable having density function Step 2: Generate U, a uniform (0,1) random variable. step 2: FU = LUD set x = ¥. Otherwise, go back to Step Assuming that successively generated random variables are indepen- dent, show that (@) X has density function f (©) the number of iterations of the algorithm needed to generate X is a geometric random variable with mean Let X;, ..., Xp be independent and identically distributed continuous random variables having distribution F. Let X,, denote the ith smallest of Xi,..-, X, and let F., be its distribution function, Show that:* PRELIMINARIES (a) Fal) = FO) Finnie) + FO) Fila). (0) Fise(2) = 5 Fisalt) + 9" FC. (Hints: For part (a) condition on whether X, = x, and for part (b) start by conditioning on whether X, is among the i smallest of X;, 2X) L 48. A coin, which lands on heads with probability p, is continually lipped. ‘Compute the expected number of flips that are made until a string of 7| heads in a row is obtained. Zi 119. An urn contains a white and b black balls. After a ball is drawn, itis: returned to the urn if itis white; but if itis black, itis replaced by a white ball from another urn. Let M, denote the expected number of white balls in the urn after the foregoing operation has been repeated times. (a) Derive the recursive equation Mo=(1-zh5) Mort () Use pat (a) rove tat Mn a+6-6(1- 45 (©) What is the probability that the (n + 1)st ball drawn is white? 120. A Continuous Random Packing Problem. Consider the interval (0, x) ‘and suppose that we pack in this interval random unit intervals—whose left-hand points are all uniformly distributed over (0,x ~ 1)—as follows. Let the first such random interval be /). If 1, ..., J; have already been packed in the interval, then the next random unit interval will be packed if it does not intersect any of the intervals /, ..., f,, and the interval will be denoted by I. If it does intersect any of the intervals I, ... + 1h, we disregard it and look at the next random interval, ‘The procedure is continued until there is no more room for additional unit intervals. (that is, all the gaps between packed intervals are smaller than 1). Let 'N(x) denote the number of unit intervals packed in {0, x] by this method, For instance, if x = 5 and the successive random intervals are (5, 15), (31, 4.1), (4,5), (17, 2.7), then N(5) = 3 with packing as follows: ee PROBLEMS st Let M(x) = E[N(2)|. Show that M satisfies May=0, x<1, May = 25 [o'MO)dy +1, xP L2L. Let U;, Us... be independent uniform (0, 1) random variables, and let N denote the smallest value of n,n = 0, such that fuse sito, smelt Show that N is a Poisson random variable with mean A. (Hint Show by induction on n, conditioning on U, that P(N =n} = eatin.) 122, The conditional variance of X, given Y, is defined by Var X|¥) = EX ~ ELX1Y IY Prove the conditional variance formula; namely, Var(X) = ElVar(X|¥)] + Var(ELX|¥). ‘Use this to obtain Var(X) in Example 1.5(B) and check your result by differentiating the generating function. septs that moves along the st ofiateges in he following aaa cm itis presenta then text moves 01 +1 With probaly ee ais probeiiy tp. Starting 0, fet Gena the pect that i ver reas (a) Ara that a= p+ (1— pa (b) Show that 1 itp=12 pip) ifp <2. (6) Find the probably thatthe pate ever reaches nn > 0 ventuly reaches {@) Suppose that p< 1/2 and als that the parte eventually aarti the particle s presently at r1-= 1, and ” has no yetee been reached, show that the particle will next move to ¢ + 1 with probability 1 ~ p and toi ~ 1 with probability p. That is, show that Ploext at {+ 1|at éand will each n} = (Note that the roles of p and 1 ~ p are interchanged when it is given that m is eventually reached.) o 1.24, In Problem 1.23, let E[T] denote the expected time until the particle As 121 1.28 reaches 1 (a) Show that LQp-1) ifp>12 ° itp =. Er) (b) Show that, for p > 1/2, Var(T) =P (©) Find the expected time until the particle reaches n, n > 0. (@) Find the variance of the time at which the particle reaches n,n > 0. Consider a gambler who on each gamble is equally likely to either win ‘or lose 1 unit. Starting with i show that the expected time until the gambler’s fortune is either 0 or kis ilk ~ i), 7= 0... k (Hint: Let M, denote this expected time and condition on the result of the first gamble.) . In the ballot problem compute the probability that A is never behind 129, in the count of the votes. ‘Consider a gambler who wins or loses 1 unit on each play with respective possiblities p and 1 ~ p. What is the probability that, starting with 7 units, the gambler will play exactly n + 2i games before going broke? (Hint: Make use of ballot theorem.) Verify the formulas given for the mean and variance of an exponential random variable. IfX,,X2,...Xeare independent and identically distributed exponential random variables with parameter A, show that 27 X;has a gamma distri bution with parameters (n, A). That is, show that the density function ‘of 37 X,is given by FO) = AeMANTMn =D, 0. PROBLEMS 33, 1.30, In Example 1,6(A) if server i serves at an exponential rate Ay, = 1,2, compute the probability that Mr. A is the last one out. 131. IfX and Y are independent exponential random variables with respective ‘means 1/Ay and I/A:, compute the distribution of Z = min(X, ¥). What is the conditional distribution of Z given that Z =X” 1.32, Show that the only. continuous solution of the functional equation als +) = 869) + 8) is g(5) = es. 1.33. Derive the distribution of the ith record value for an arbitrary continuous distribution F (see Example 1.6(B)). 1.34, If X; and X; are independent nonnegative continuous random variables, show that AC), PAX, < Xl min A) = 9 = Pag! where A(0) isthe failure rate function of X;. 1.35, Let X be a random variable with probability density function f(s), and Ele] be its moment generating function. The tilted density n fis defined by ete) 1) = M0) Let X; have density function f, (8) Show that for any function h(x) E (AX) = MDE Tex(—iXIA()] () Show that, for > 0, PIX > a). = M(Qe™PIX, > a} (©) Show that if P(X? > a) = 1/2 then min M(je* = M(M)e84 PRELIMINARIES 1.36. Use Jensen's inequality to prove that the arithmetic mean is at least as large as the geometric mean. That is, for nonnegative 2, show that 137. Let Xj, X2,... be a sequence of independent and identically distributed continuous random variables. Say that a peak occurs at time m if Xeni < Xq > Xo. Argue that the proportion of time that a peak occurs is, with probability 1, equal to 1/3. 1.38, In Example 1.9(A), determine the expected number of steps until all the states 1, 2,...,m are visited. (Hine Let X, denote the number of additional steps after i of these states have been visited until a total of i+ 1 of them have been visited, | 1, -...m — 1, and make use of Problem 1.25.) 139. A particle moves along the following graph so that at each step itis equally likely to move to any of its neighbors. Starting at 0 show that the expected number of steps it takes to reach (Hint Let T; denote the nuraber of steps it takes to go from vertex i= 1 to vertex i, i= 1, ..... Determine E[T)] recursively, first for i= Ly then i = 2, and s0 00.) 5 ‘Suppose that r = 3 in Example 1.9(C) and find the probability that the leaf on the ray of size nis the lat leaf to be visited. 141. Consider a star graph consisting ofa central vertex and r rays, with one ray consisting of m vertices and the other r ~ 1 all consisting of m vertices. Let P, denote the probability that the leaf on the ray of m vertices is the last leaf visited by a particle that starts at 0 and at each step is equally likely to move to any ofits neighbors (@) Find P, () Express P, in terms of P, 1.42, Let ¥), Ys... be independent and identically distributed with PUY, PUY,> 9) Ima)e%, y>0. vy. Define the random variables X,,n = 0 by Prove that ~ 143, For a nonnegative random variable X, show that for a > 0, PIX = a} = E(x ‘Then use this result to show that n! = (nle)* REFERENCES Retna? ate ements opty a nara igorous approaches reba andsocasic process, based on measure theory, Ee ven h References 248, and 8 Example 13(C) i taken fom Relrene Fropostion 152s due to Blackwell and Gishick (Reference 3) 1. N.Alon Spencer an . Erdos, The Probabiltic Method, Wiley, New York, 192 P.Bilingaly, Probably and Measre, id, Wiley, New Yor, 198, D. Blackell and M_A.Cisick, Theory of Games and Sasa Decision, Wiley New York 984 Breiman, Probably, Addon Wesley, Reading, MA, 196. BR Durrte, Probably: Teor and Example, Brooks/Col,Calforsia, 199 6 W. Pele, nInroductonto Probably Theory andi Applications, VoL 1. Wey New York 19 SM: Res, A Fist Couse in Probab, Sth ed, Macmillan, New York, 1996 D. Wiliams, Probably with MaringlexCambigge Unversity Press, Cambie, England, 1991.APPENDIX The Strong Law of Large Numbers If X,, %, isa sequence of independent and identically distributed random variables with mean p, then ofan’ =.) ‘Although the theorem can be proven without this assumption, ovr proof of the strong law of large numbers will assume that the random variables X, have a finite fourth moment. That is, we will suppose that E[X4] = K < =. Proof of the Strong Law of Large Numbers ‘To begin, assume that y, the mean fhe Xs alo. Lat = 3%, and consider EIS! EY AQ EE) EA HEAD Expanding the right side of the above will result in terms of the form Xt, XIX, XIX} XIMK, and XXX where i,j, k,l are all different. As all the X; have mean 0, it follows by independence that ELXGX| = E[XI LX] =0 E [XXX] = ELXIJELYIEL%] = 0 FIXXXAX] = 0. Now, fra given pur and j here will be ($) = 6 terms in the expansion that wil equal X243. Hence, it folows upon expanding the above product % ‘THE STRONG LAW OF LARGE NUMBERS: s7 and taking expectations térms by term that fst] = nex} + 6(") (XIX = nk + 3n(n ~ 1)E[XIEL) where we have once again made use of the independence assumption, Now, since 0 Var X}) = E1Xt] = (ELXI), wwe see that (ELD s BUX ‘Therefore rom the preceding we have that E[S$] =nK + 3n(n— 1)K which implies that ElSin!] = Ki + 3K. ‘petefore it follows that E [§ sim] =Selsuj 0 follows from the Markov inequality that P(Stin' > e} S E[Stin'Ple nd ths tom (*) Spisynt> eo hich implies by the Borel-Cantl lemma that with probability 1, Sat > 6 for only finitely many n. As this is true for any ¢ > 0, we can thus conclude that with probability 1, lim Sint = 0.58 THE STRONG LAW OF LARGE NUMBERS But if Stin' = (S,in)* goes to 0 then so must S/n; and so we have proven that, with probability 1, Sin-+0 as nae, ‘When ja, the mean of the X,, is not equal to 0 we can apply the preceding. argument to the random variables X; — 2 to obtain that, with probability 1, Sa-win or equivalently, lim 3 Xjn= Which proves the result. CHAPTER 2 The Poisson Process 2.1 Tur Porsson Process A stochastic process (N(®), t = 0) is said to be a counting process if N() represents the total number of ‘events that have occurred up to ime f. Hence, a counting process N(v} must satisfy @ N@=0. Gi) N@ is integer valued, i) If 0. ‘One of the most important types of counting processes is the Poisson process, which is defined as follows. Definition 2.2.4 ‘The counting process (N(0) t= O}is sad to bea Poisson process having rate &, A > if @ NO =0. (iy The process has independent increments.o "THE POISSON PROCESS “THE POISSON PROCESS a “The number of events in any interval of length is Poisson distributed with maciitbneas re Trconew aa : Deft 21 and 21.20 eqn rue —neyema er, naa, ‘of We is show that ton 212 ips Deion 214.o do is Pad) = PIN() = mh ‘Note that it follows from condition (ii) that a Poisson process has stationary increments and also that ‘We derive a differential equation for P,() in the following manner: EIN] = At Pat +h) = PINE A) =O} 1.N(C+ A) — NC WPIN(+A)— NO) which explains why A i called the rate of the process. In order to determine if an arbitrary counting process is actually a Poisson process, we must show that conditions (i, (i), and (i) are satisfied. Condition = Pll —ah+ oH), (i), which simply states that the counting of events begins at time = 0, and condition (ji) can usually be directly verified from our knowledge of the where the final two equations follow from Assumption (i) and the fact that (i) and process. However, its not at all clear how we would determine that condition {v) imply that P(N(A) = 0} = 1 = AK + o(h). Hence, (Gi) is satisfied, and for this reason an equivalent definition ofa Poisson process would be useful PAL N= PO «py + 09 ‘AS a prelude to giving a second definition of a Poisson process, we shall 2 ‘ define the concept ofa function fbeing o(?). Letting 0 yields P= aa) Defition . “The function fs sai tobe oh) i to tin Oo, which imple, by integration, We are now in a position to give an alterative definition of a Poisson or process Pai) = Ke — Since (0) = PIN() = 0} = 1, we anve at Definition 2.4.2 ey Paynes “The counting process {N(0), = 0) is said to be a Poisson process with rate A, A> 0, if oe @ NO =0. (i) The process has stationary end independent increments Gil) PIN(A) = 1) = ah + of). iv) PINCH) = 2) = of. Piet h) = PING A) =n (+h) — NO =O) =1,.N(+H)-MO=1D —_ + PIN(+) =n, NCE) —N(Q22}a ‘THE POISSON PROCESS However, by (iv) the last term in the above is o(f); hence, by using ji), we obtain Pi(ct h) = PAPA) + Prs(PAh) + 0(8) = AAYPalt) + AAP (0) + 0(h). Ts PHFD. ap) (9 +20 Leng 4-0 Put) = -AP.(t) + APO) er. cquvalenty, e*[P(t) + AP,(O)] = Ae"Pena(t)- Hence, a1 deormarere Now by 2:11) we have when n ens Glenna PQ) = Ort Oe, which, since P.(0) = PAO = Me™ ‘To show that P,(®) = e°*(i)"in, we use mathematical induction and hence first assume it for» ~ 1. Then by (212), nage Seen Leer.) implying that ant oP.) {HE POISSON PROCESS 8 or sinee P.(0) = PIN(O) =m) = 0, ane Pn, ‘Thus Definition 2.1.2 implies Definition 2.1.1, We wil leave it forthe reader to prove the reverse ‘Remark The result that N(t) has a Poisson distribution is a consequence of| the Poisson approximation to the binomial distribution, To see this subdivide the interval [0,1] into & equal parts where k is very large (Figure 2.1.1. Fitst ‘we note that the probability of Kaving 2 or more events in any subinterval foes to. as k > 2, This follows from P(Q or more events in any subinterval) © Pl2ormore events in the ith subintervall, Hence, N(1) will (with a probability going to 1) just equal the number of subintervals in which an event occurs. However, by stationary and independent increments this number will have a binomial distribution with parameters k and p = Ailk + o(t/k). Hence by the Poisson approximation to the binomial wwe see by letting k approach © that N(j) will have a Poisson distribution with ‘mean equal 10 Figure 21.1“ "THE POISSON PROCESS 2.2 INTERARRIVAL AND WAITING Time DistRIBuTIONs Consider a Poisson process, and let X; denote the time of the first event. Further, forn = 4, let X, denote the time between the (n ~ 1)st and the th event. The sequence {X,, = 1) is called the sequence of interarrival times. ‘We shall now determine the distribution of the X.. To do so we Brst note that the event (X; > 1) takes place if, and only if, no events of the Poisson process occur in the interval [0, ¢], and thus PIX > 8) = PING) = 0) =e. Hence, X; has an exponential distribution with mean 1/A. To obtain the distribution of X; condition on X. This gives PU > IX =9) (events in (s,5+ Xi =) "Oeventsin (s,s+ 4) (byindependent increments) =e" (bystationary increments). ‘Therefore, from the above we conclude that X; is also an exponential random variable with mean 1/A, and furthermore, that X: is independent of Xy. Re- peating the same argument yields the following, PROPOSITION 2.2.4 are independent identically distributed exponential random variables hnaving mean 1. coer Remark The proposition should not surprise ws. The assumption of station- ary and independent increments is equivalent to asserting that, at any point in time, the process probabilistically restarts itself. That is, the process from ‘any pointon is independent of all that has previously occurred (by independent jnerements), and also has the same distribution as the original process (by stationary increments). In other words, the process has no memory, and hence ‘exponential interarrival times are to be expected, Another quantity of interest is S,, the arrival time of the nth event, also called the waiting time until the nth event, Since Set 5. INTERARRIVAL AND WAITING TIME DISTRIBUTIONS 6s it is easy to show, using moment generating functions, that Proposition 2.2.1 implies that S, has a gamma distribution with parameters m and A. That is, its probability density is J) = rer OO ‘The above could also have been derived by noting that the nth event occurs prior or at time if, and only if, the number of events occurring by time fis at least n. That is, oad N() =n 5,51 Hence, PIS. 0) = PIN =n} which upon differentiation yields that the density function of 5, is 20 Spe a 2 a G— Remark Another way of obtaining the density of Ss to use the independent increment assumption as follows: 1, Leventiin (¢,1 + dd} + o(d) 1) P(l event in (+ di)} + ofa) any! ey Adt + o(dt) which yields, upon dividing by d() and then letting it approach 0, that Proposition 2.2.1 also gives us another way of defining a Poisson process. For suppose that we start out with a sequence {X,, = 1) of independent identically distributed exponential random variables each having mean 1/A Now let us define a counting process by saying that the nth event of this6 ‘THE POISSON PROCESS process occurs at time 5,, where SyEX FX to + Xe ‘The resultant counting process (V(t), ¢ = O} will be Poisson with rate A 2.3 Conprrionat DisTRIBUTION OF THE Arrivat Times Suppose we are told that exactly one event of a Poisson process has taken place by time f, and we are asked to determine the distribution of the time at which the event occurred. Since a Poisson process possesses stationary and independent increments, it seems reasonable that each interval in [0,t] of equal length should have the same probability of containing the event. In other words, the time of the event should be uniformly distributed over [0] This is easily checked since, for 5 = ¢, ua m independent trialt when p the probability of success on each PIN) = 0, NAO = mIN(0) Consequently, PUA) = MNs(t) =m) = FAA pect — pyres oe DY gy AHO = DD" m = Which completes the proof. ‘The importance of the above proposition is strated by the following example."THE POISSON PROCESS Examnuc2.3(0) The Infinite Server Poisson Queue. Suppose that customers arrive at a Service station in accordance with a Poisson process with rate A. Upon arrival the customer is immediately served by one of an infinite number of possible servers, and the service times are assumed to be independent with a common distri- bution G. ‘To compute the joint distribution of the number of customers that have completed thei service and the number that are in service at ¢, call an entering customer a type-I customer ifit completes its service by time # and a type-II customer if it does not complete its ‘service by time 1. Now, if the customer enters at time s,s 5, then it will bea type-{ customer if its service time is less than ¢ ~ s, and since the service time distribution is G, the probability ofthis will be Glt ~ 5). Hence, P@)=GU-8), ssh ‘and thus from Propesition 2.3.2 we obtain that the distribution of ‘N\(0—the number of customers that have completed service by time ‘is Poisson with mean BIG) = Af, G0-3) ds = f.Gorgy. Similarly N,(2), the number of customers being served at time tis Poisson distributed with mean FINO) = 4 f, GO) dy. Further N(0) and N(1) are independent. “The following example further ilustrates the use of Theorem 2.3.1. ‘Exawrte2.3(¢) Suppose that a device is subject to shocks that occur in accordance with a Poisson process having rate A. The ith shock gives rise to a damage D,. The D,, i = 1, are assumed to be independent and identically distributed and also to be independent of {(N(0, 1 = 0}, where N(¢) denotes the number of shocks in (0, ‘The damage due to a shock is assumed to decrease exponentially in time, That is, if a shock has an initial damage D, then a time ¢ Tater its damage is De. Tf we suppose that the damages are additive, then D(‘), the damage at 1, can be expressed as po=S de~, CONDITIONAL DISTRIBUTION OF THE ARRIVAL TIMES where 5, represents the arrival time of the ith shock. We can determine E[D(0)] as follows: EDOIN() =n) = E[¥ De vin =n] =E [s De“ INO = "| Sate in SADING = nzle~AiN( =m = HID) Ete 21N =] = aoe [Serine =n] = BipIeE |S eximo =] Now, letting Us,..., Usbe independent and identically distributed uniform [0,f] random Variables, then, by Theorem 2.3.1, e[Secino=n]=2 [Se] flew (ep. ‘at Hence, Eoin =*2a - e410) and, taking expectations, 2100) = “71a ~~), nn “THE POISSON PROCESS Remark Another approach to obtaining E[D(#)] is to break up the interval (0.1) into nonoverlapping intervals of length hand then add the contribution at time ¢ of shocks originating in these intervals. More specifically, let h be given and define X, as the sum of the damages at time ¢ ofall shocks arriving in the interval J, = (ih, (i + 1)h),i = 0, 1, -., [Uh], where [a] denotes the largest integer less than or equal to a. Then we have the representation Do and so w EID] = FLX ‘To compute E[X)} condition on whether or not a shock arrives in the interval 4, This yields wa E[D@)= y (AnE|De™*"*)] + oH), ean etowy=atore[Stae0] [Joa But, since L, € J, it follows upon letting h + 0 that Shem o+f' tn dy ‘and thus from (2.3.1) upon letting A+ 0 B19) = “22 N a ~ e~), tis worth noting that the above is a more rigorous version of the folowing argument: Since a shock occurs in the interval (y, y + dy) with probability dy and since its damage at time f will equal e°*” times its initial damage, it follows that the expected damage at ¢ from shocks originating in (y, ¥ + a) is AdyE[DJe*"”, ‘CONDITIONAL DISTRIBUTION OF THE ARRIVAL TIMES B ‘and so EIDE] = AED] [,e-#™ dy 2.3.1 The M/G/1 Busy Period Consider the queueing system; known as M/GIL, in which cuscomers arrive in accordance with a Poisson process with rate A. Upon arrival they either ‘enter service ifthe server is free or else they join the queue. The successive service times are independent and identically distributed according to G, and are also independent of the arrival process. When an arrival finds the server free, we say that a busy period begins. It ends when there are no longer any ‘customers in the system. We would like to compute the distribution of the length of a busy period. ‘Suppose that @ busy period has just begun at some time, which we shall designate as time 0, Let 5, denote the time until k additional customers have arrived. (Thus, for instance, S, has a gamma distribution with parameters k, A) Also let ¥), Y2,-.- denote the sequence of service times, Now the busy period will last a'time ¢ and will consist of n services if, and only if, SS YEN RE Len. YrertY, i) There are n ~ 1 arrivals in (0, 1) Equation (i is necessary for, if S, > Yj +--+ ¥,, then the kth arrival after the initial customer will find the system empty of customers and thus the busy period would have ended prior to k + 1 (and thus prior to n) services. The ‘easoning behind (ii) and (i) is straightforward and left to the reader. Hence, reasoning heuristically (by treating densities as if they were proba- bilities) we see from the above that, (232) P{busy periodis of length rand consists of» Services}, = PY +2++ ¥,=tn=Larrivalsin (0,0), $,5¥ +--+ + Ya, k an} SPI SY toot Yak +m —1|n— Larrivalsin (0, 1), Yet soo# Yonah Pla 1 arvalsin 0,0, 4-04 Yoo dh" ‘THE POISSON PROCESS Now the arrival process is independent of the service times and thus 233) Pin ~1 arrivals in (0,0), ¥; 4 + ¥,= 0) where G,is the n-fold convolution of G with itself. In addition, we have from ‘Theorem 2.3.1 that, given m -- | arrivals in (0, t), the ordezed arrival times are distributed as the ordered values of aset ofn ~ I independent uniform (0,1) random variables, Hence using this fact along with (2.3.3) and (2.3.2) yields (23.4) P{busy period is of length rand consists of services} x Pins ¥ite--+ Fak HUY te eX =A where 7),..-, %1 ate independent of {¥),..-, Ya) and represent the ordered values of a set of m ~ 1 uniform (0,1) random variables. ‘To compute the remaining probability in (2.34) we need some lemmas. Lemma 2.33 is elementary and its proof is left as an exercise. Lemma 2.3.3 Let ¥), Yoo, Ys be independent and identically distributed nonnegative random Variables. Then BY HHH ty Lemma 2.3.4 Let n.....% denote the ordered values from a set of m independent uniform (0,0) random variables. Let ¥;, ¥:,--.be independent and identically distributed nonnega tive random variables that ate also independent of {s,,..., %} Then 235) PI eo heme fe o. Bl, »), is thus given by 20) = 3 fer avo 2.4 Nonnomoceneous Porsson Process In this section we generalize the Poisson process by allowing the arrival rate” at time 1 to be a function oft Definition 2.4.1 ‘The counting process {M()}, = O} is sai to be a nonstationary or nonhomogeneous Poisson process with intensity function 0), ¢= 0 i @ MO) = 0. iy {M(0, = 0} has independent increments NONHOMOGENEOUS POISSON PROCESS p- iy PIMC + 8) ~ N(O= 2} = oh), (is) PIM + A) ~ NO) Ck + of. itwe kt imp = [aoa then it can be shown that @4l) PING +8) — NO =n} = expl=(m(t +s) — m()}lm(e+ 5) — m(O]nt, 0, ‘That is, N(¢ + 5) ~ N( is Poisson distributed with mean m(t +s) ~ m(i). ‘The proof of (2.4.1) follows along the lines of the proof of Theorem 2.1.1 with a slight modification: Fix r and define Pals) = PIN + 8) ~ MQ = nh ‘Then, Pst hy = PING + s-+.A)— M0) =O} (Oeventsin (,¢+ 5), events in + 5,045 + A)) {0 events in (0,1 + 9)}P{Oevents in (¢+ 5,045 +A) = PAs) Ae+ S)h+ off), where the next-to-last equality follows from Axiom (ii) and the last from Axioms (iil) and (iv). Hence, FON PO vr yng) Letting h = 0 yields Pils) = ~ACE+ S)Pas) log Pals) = ~f1 ale + 4) de80 THE POISSON PROCESS Pals) = ehh ‘The remainder of the verification of (2.41) follows similarly and is left as The importance ofthe nonhomogeneous Poisson process resides in the fact that we no longer require stationary increments, and so we allow for the possibility that events may be more likely to oceur at certain times than at other times, ‘When the intensity function A() s bounded, we can think of the nonhomo. geneous process as being a random sample from a homogeneous Poisson process. Specifically, let A be such that MQ=A forall =O ‘and consider a Poisson process with rate A. Now if we suppose that an event of the Poisson process that occurs at time ris counted with probability A(0)/ A, then the process of counted events is a nonhomogeneous Poisson process with intensity function A(0). This last statement easily follows from Definition 2.4.1. For instance (i), (ii), and (ji) follow since they are also true for the homogeneous Poisson process, Axiom (iv) follows since a0, Plone counted event in (1 + h)) = Plone eventin (,¢ + 4)} “> +o(h) = nA + ony = Noh + oft) ‘The interpretation of a nonhomogeneous Poisson process as a sampling {rom- a homogeneous one also gives us another way of understanding Proposition 23.2 (or, equivalently, it gives us another way of proving that N(j) of Proposi tion 23.2 is Poisson distributed). Buwme2.4(a) Record Values. Let X,,X:,.-.denote a sequence ‘of independent and identically distributed nonnegative continuous random variables whose hazard rate function is given by A(). (That is, AW) = f(O/F(O, where f and F are respectively the density and distribution function of X:) We say that a record occurs at time 1 EX, > max(%,, ..., Xs), where X; = 0. If a record occurs at time n, then X, is calieda record value. Let N(0) denote the number of record values less than or equal to t, That is, N() is a counting process of events where an event is said to oocur at time x if xis 2 record value. [NONHOMOGENEOUS POISSON PROCESS ‘We claim that {N(Q), £ = 0} will be a nonhomogeneous Poisson process with intensity function A(). To verify this claim note that there will be a record value between f and 1 + h if, and only if, the first X; whose value is greater than f lies between ¢ and ¢ + h. But we have (conditioning on which X, this is, say, i = n), by the definition of a hazard rate function, PIX, E (1+ W)IX.> 1 AOA + oft), which proves the claim. Exams 2.43) The Output Process of an Infinite Server Poisson Queue (M/GI=). It turns out that the output process of the Ml Gree queve—that is, of the infinite server queue having Poisson arrivals and general service distribution G—is a nonhomogeneous 3 process having intensity function A(1) = AG(®). To prove this we shall first argue that () the number of departures in (5,5 + 1) i Poisson distributed with mean 4 [)" G(y) dy, and (2) the aumbers of departures in disjoint time intervals are inde- pendent ‘To prove statement (1), call an arrival type Tif it departs in the interval (5,5 +0). Then an arrival at y willbe type I with probability Gls+t-y)-Gls-y) _itys+e ifs Aas ¢—» ©, iis interesting to note that the imi ‘output process after time t (as ¢ > %) is a Poisson process with rate A 2.5 Compounp Poisson Ranpom VARIABLES AND PROCESSES Let X;, Xa, ... be a sequence of independent and identically distributed random variables baving distribution function F, and suppose that this se- quence is independent of N, a Poisson random variable with mean A. The random variable is said to be a compound Poisson random variable with Poisson parameter A ‘and component distribution F. "The moment generating function of W is obtained by conditioning on N. This gives, 3 Ele" | N= njPIN =n} Ele") Seles M[W= me Maryn! 51) Sete leapt (252) = 3 Efetperanrn! where (2.5.1) follows from the independence of [X),X;,.. and N, and (25.2) follows from the independence of the X;. Hence, letting x(0) = Ele" denote the moment generating function ofthe X,, we have from (2.52) that E(e"] = 3 loyiore “aor 253) = explad(dx(®) ~ V) ‘COMPOUND FOISSON RANDOM VARIABLES AND PROCESSES: a Its easily shown, either by differentiating (2.53) oF by directly using @ conditioning argument, that EW VarW) EEX] AEX] where X has distribution F, Bums 2.5(4) Aside from the way in which they are defined, ‘compound Poisson random variables often arise in the following manner. Suppose that events are occurring in accordance with a Poisson process having rate (say) a, and that whenever an event ‘occurs a certain contribution results. Specifically, suppose that an event occurring at time s will, independent of the past, result in a contribution whose value is a random variable with distribution F,, Let W denote the sum of the contributions up to time ‘that is, Ww=dX ‘where N() is the number of events occuring by time t, and X; is the contribution made when event occurs. Then, even though the X, are neither independent nor identically distributed it follows ‘hat Wis @ compound Poisson random variable with parameters 1 heat and Fla) =1f B(x) ds. ‘This ean be shown by calculating the distribution of W by first conditioning on N(), and then using the result that, given NCO, the unordered set of N( event times are independent uniform (0,0) random variables (see Section 23). When Fis a discrete probability distribution function there is an interesting representation of W as a linear combination of independent Poisson random variables. Suppose that the X, are discrete random variables such that PIX = j= Pp If we let N; denote the number of the X/s that are equal to j,j = 1,-... then we can express W as (254) w= DIN,cy THE POISSON PROCESS where, using the results of Example 1.5(I), the N) are independent Poisson random variables with respective means Ap, j = 1,...,k. AS a check, let us use the representation (254) to compute the mean and variance of W. DEIN) = Diary = AELX] EIX? EIw) Fp, Var(W) = 5 'Var(N)) Which check with our previous results. 2.6.1 A Compound Poisson Identity As before, let W= 3X; be a compound Poisson random variable with N being Poisson with mean A and the X; having distribution F. We now preseat 1 useful identity concerning W. ee PROPOSITION 2.5.2 Let X'be a random variable having distribution F that is independent of W. Then, for any function A(x) ELWAQW)] = AELXAOW + X)] Proof Eqwagwy}= 5 E1wacwy|W = mer 3*)| Where the preceding equation follows because all of the random variables X) (3X) have the same distribution, Henee fom the above we obtain, upon condi- COMPOUND POISSON RANDOM VARIABLES AND PROCESSES, 85° tioning on X, that «onal #mol§s) Se atnlel(Een)]arw se[a(Sxr3)]ary x1] are af 3 engv-snin=mipte= mare) =a xEt0 + 914rG) = AJ EbxnOW + xyLx=2]4R0) EIXH(W + X)) Proposition 2.5.2 gives an easy way of computing the moments of W. Corollary 2.5.3 1X has distribution F, then f See E1W"] = ABLXOW + XY")86 "THE POISSON PROCESS Thus, by starting with n = 1 and suoesively increasing the value of, we sce that ELW|= ABIX] EW] = MEL] + EDWIELXD, ABUC]+ ELD? EW] = CEL] + EW IELA] + ELWAIELXD) = AE[X] + SPELXTELX"] + CELI) and son. ‘We will now show that Proposition 2.5.2 yields an elegant recursive formula for the probability mass function of W when the X, are positive integer valued random variables. Suppose this isthe situation, and let a= PK =i 71 and B= PW=j, [=O The successive values of P, can be obtained by making use ofthe following Corollary 2.5.4 ree PAS iaPen neh Proof That Py = &*is immediate, so take n > 0, Let 0 teen "Vim ite=n Since Wh(W) = I{W = n), whichis defined to equal | if W =n and 0 otherwise, we ‘oblain upon applying Proposition 2.52 that POW =n} = AE[XMOW + 29) = AD ELXW + X)IX=j]a, = AL JEWW + Ds =aS LP + =a COMPOUND POISSON RANDOM VARIABLES AND PROCESSES. a7 Remark When the X, are identically equal to 1, the preceding recursion reduces to the well-known identity for Poisson probabilities PIN=0)= 0 a PwN= Apiw=n-1), ne. Exannue 2.5(8) Let W be a compound Poisson random variable with Poisson parameter A = 4 and with PU, 123.4, To determine P{W = 5}, we use the recursion of Corollary 2.5.4 a8 follows: IayPh + Joy) = et {ays 2aiP, + 3asP))= 2 A fasP + 2ayPs + 30yP, + dauPs) 2.5.2 Compound Poisson Processes A stochastic process {X(0), ¢ = 0) is said to be a compound Poisson process if it can be represented, for 1 = 0, by xO 3x, where {(N(), «= 0) isa Poisson process, and (X,,#= 1,2, isa family of independent and identically distributed random variables that is independent Of the process {(9, ¢ = 0}. Thus, if {X(0), « = 0} is a compound Poisson Process then () isa compound Poisson random variable. ‘AS an example of a compound Poisson process, suppose that customers arrive ata store ata Poisson rate A. Suppose, also, thatthe amounts of money spent by each customer form a set of independent and identically distributed random variables that is independent of the arrival process, If X() denotesTy : the total amount spent in the store by’ all customers arriving by time s, then {X(), 1 0} is a compound Poisson process. 2.6 Conprrrowat Poisson Processes Let A be a positive random variable having distribution G and let {N(), = (0) be a counting process such that, given that A= A,{N(0),1 = 0} is @ Poisson process having rate A. Thus, for instance, PIN +s) — N= m= {° er 2 ac) “The process {N(j),t = 0} is called a conditional Poisson process since, condi- tional on the event that A = A, it is a Poisson process with rate A. It should bbe noted, however, that {N(0), ¢ = 0} is not a Poisson process. For instance, ‘whereas it does have stationary increments, it does not have independent ‘ones. (Why not?) Let us compute the conditional distribution of A given that N() = n. For da small, PIA (A+ aa)|NO =m = PIN =nlA€ + daPIAE PIN® = nh A+ dy) er 2 acy and so the conditional distribution of A, given that N(e) = nis given by feraor ace flewanracay P(A = x|N(@) =n) = Eure 2.6(4) Suppose that, depending on factors not at present understood, the average rate at which seismic shocks occur in a certain region over a given season is either A, or A,. Suppose also that itis 4 for 100 p percent of the seasons and A; the remaining / time, A simple model for such a situation would be to suppose that {N(), 0-5 ¢< is a conditional Poisson process such that Aiseither A, or A; with respective probabilities p and | ~ p. Given 'n shocks in the first time units of a season, then the probability PROBLEMS 89 it is a ay season is pet)" Pema + eH ~B) PIA = AO =nh= | ‘Also, by conditioning on whether A = 4; or time from t until the next shock, given N(1) Ay we see that the ias the distribution, Plime from until next shock is = x|N() =n} erhi(auy + (1 = e%'Je™(Ast)( ~ p) eM CAV =P) p= Prosiems 2.1, Show that Definition 2.1.1 of a Poisson process implies Definition 2.1.2. : 2.2 For another approach to proving that Definition 2.1.2 implies Defini- i tion 2.1. (a) Prove, using Definition 2.1.2, that Palt + 8) = Pod dPale) (b) Use (a) to infer thatthe interarrival times X;, X,,...areindependent ‘exponential random variables with rate A, (©) Use (b) to show that N(j) is Poisson distributed with mean At, 23. For a Poisson process show, for s < f, that (OY teen Lj 7A Let {NQ), # = 0} be @ Poisson process with rate A. Calculate E[N() NOt) PIN(s) = KIN = ZA Suppote ta (1 = 0} nd (N()£# Oa independent Poisson proseses wih rates and hy: Show that (NO) UND. =O he Poston procets ith teh Also, shaw tha The babi that the fat event ofthe combined poses comes fom (Dy =O) ACh Aas inependenty ofthe ine of he eve | 2.6. A machine needs two types of components in order to function. We have a stockpile of m type-1 components and m type-2 components.Ars. 2.10, ‘Typei components last for an exponential time with rate py before failing. Compute the mean length of time the machine is operative if a failed component is replaced by one ofthe same type from the stockpile; that is, compute E[min(S; X,,2" ¥,)], where the X,(¥) are exponential with rate (4). 1. Compute the joint distribution of S), S:, Ss Generating a Poisson Random Variable. Let U,, Us... ‘uniform (0, 1) random variables. (@) If X, = (-log UA, show that X; is exponentially distributed with rate A (®) Use part (a) to show that Vis Poisson distributed with mean A when ‘Nis defined to equal that value of n such that be independent flusersiju. where IT, U; 1, Compare with Problem 1.21 of Chapter 1 Suppose that events occur according to a Poisson process with rate A. Each time an event occurs we must decide whether or not to stop, with our objective being to stop at the last event to occur prior to some specified time T. That is, if an event occurs at time 1,0 = ¢ = T and we decide to stop, then we lose if there are any events in the interval (t, T), and win otherwise. If we do not stop when an event occurs, and no additional events occur by time T, then we also lose. Consider the strategy that stops atthe frst event that occurs after some specified time s0sssT, (a) If the preceding strategy is employed, what is the probability of winning? (b) What value of s maximizes the probability of winning? (©) Show that the probability of winning under the optimal strategy is Ve, Buses arrive at a certain stop according to a Poisson process with rate A, Ifyou take the bus from that stop then it takes a time R, measured from the time at which you enter the bus, to arrive home. If you walk {rom the bus stop then it takes a time W to arrive home. Suppose that ‘your policy when arriving at the bus stop is to wait up to a time s, and if a bus has not yet arrived by that time then you walk home. (a) Compute the expected time from when you arrive at the bus stop until you reach home. PROBLEMS 1 2aL. 2 23. Au 28. (b) Show that if W <1/A + R then the expected time of part (a) is ‘minimized by letting s = 0; if W > 1/A + R then itis minimized by letting s = % (that is, you continue to wait for the bus); and when W-= 1/A + Rall values of s give the same expected time. (©) Give an intuitive explanation of why we need only consider the cases s = O and s = % when minimizing the expected time. Cars pass a certain street location according to a Poisson process with rate A. A person wanting to cr0ss the street at that location waits until she can see that no cars will come by in the next T time units. Find the expected time that the person waits before starting to cross. (Note, for instance, that if no cars will be passing in the first T'time units then the waiting time is 0.) Events, occurring according to a Poisson process with rate A, are regi tered by a counter. However, each time an event is registered the counter becomes inoperative for the next b units of time and does not register any new events that aright occur during that interval, Let R(®) denote the number of events that occur by time ¢ and are registered. (a) Find the probability that the first k events are all registered, (b) Fore = (n ~ 1)b, find P(R() = 2) ‘Suppose that shocks occur according to a Poisson process with rate A, ‘and suppose that each shock, independently, causes the system to fail with probability p. Let N denote the number of shocks that it takes for the system to fail and let T denote the time of fajlgre. Find Pt ne . oe ne at j with probability P,. 2, P; = 1. Let O, = number of people getting to conn Fol Consider an rsided coin and suppose that on each fip ofthe coin exactly cone of the sides appears side é with probability P,, 2 P= 1. For given fnumbers i... let N, denote the number of fips required uni side ‘has appeated forthe n time, + Feand let N= min N,216. 247. “THE POISSON PROCESS “Thus Nis the number of flips required until side “has appeared m times for some i= 1-57 (a) What is the distribution of N,? (b) Are the N, independent? [Now suppose that the flips are performed at random times generated by a Poisson process with rate A= 1. Let T; denote the time until side ‘has appeared for the m time, i rand let T= min T ©) What isthe distibution of 7,7 “(@) Are the T; independent? (€) Derive an expression for E17] () Use (@) 10 derive an expression for E(M ‘The number of trials to be performed is a Poisson random variable with mean A. Each trial has n possible outcomes and, independent of everything else, results in outcome number i with probability P,, DI P.=1. Let X; denote the number of outcomes that occur exactly j times, j= 0, 1,..., Compute E[X)}, Var(X)) Let X), X:, +) Xy be independent continuous random variables with common density function f Let X,, denote the ith smallest of Xi,..-, Xe. (a) Note that in order for Xi to equal x, exactly /~ 1 of the X’s must be less than x, one must equal x, and the other n ~ / must all be ‘greater than x. Using this fact argue that the density function of X,) is given by fa) = Tapia FO FON. (by Xp will be less than x if, and only if, how many of the X's are less than x? (©) Use (b) to obtain an expression for PIX) = x) (@) Using (2) and (c) establish the identity pend yar PROBLEMS. 93° (@) Let S, denote the time of the ith event of the Poisson process (M(0), ea 218, Let Uiy,..., Usp denote the order statistics of a set of n uniform (0, 1) random variables. Show that given Uy = y, Uy... Ur-nare distributed as the order statistics of a set of n ~1 uniform (0, ) random variables 219. Busloads of customers arrive at an infinite server queue at a Poisson rate A, Let G denote the service distribution. A bus contains j customers with probability a,j = 1,.... Let X(0 denote the number of customers that have been served by time (@) E[X} (b) Is X(0) Poisson distributed? 2220, Suppose that each event of a Poisson process with rate Ais classified as being either of type 1, 2,..., A. Ifthe event occurs ats, then, indepen- dently ofall else, itis classitied as type i with probability Pi), i = 1, 1k, By P(s) = 1. Let N(0) denote the number of type é arrivals in (0. 1]. Show that the N,(Q, i = 1,..., kare independent and Ni(0) is Poisson distributed with mean A J, PAs) ds. 2.24. Individuals enter the system in accordance with a Poisson process having rate A. Each arrival independently makes its way through the states of the system, Let a(s) denote the probability that an individual isin state ‘atime s after it arrived. Let (9 denote the number of individuals in state fat time f Show that the N(0, = 1, are independent and N,(0) is Poisson with mean equal to AElamount of time an individual is in state / during its frst ¢ units in the system], 2.2, Suppose cars enter a one-way infinite highway at a Poisson rate A. The ith ear to enter chooses a velocity V, and travels a this velocity. Assume that the V/s are independent positive random variables havinga common distribution F. Derive the distribution of the number of cars that are located in the interval (a, b) at time &. Assume that no time is lost when cone car overtakes another car. 223, For the model of Example 2.3(C), find () Vari) () Cov[D0), D(C +}gy] |_|; —_—_— Zz _e-_———-C—E—F-_E—EE xKE[[={_=[=[£ | 94 224, 228, 226. 227. 2.28, 228. 230, 231. “THE POISSON PROCESS Suppose that cars enter a one-way highway of length L in accordance With a Poisson process with rate A. Each car travels at a constant speed that is randomly determined, independently from car to car, from the distribution F. When a faster car encounters a slower one, it passes it with no loss of time. Suppose that a car enters the highway at time r Show that as > © the speed of the car that minimizes the expected ‘number of encounters with other cars, where we say an encounter occurs when a ear is either passed by or passes another car, is the median of the distribution G. Suppose that events occur in accordance with a Poisson process with rate A, and that an event occurring at time s, independent of the past, contributes a random amount having distribution F,, s = 0. Show that W, the sum of all contributions by time r,s a compound Poisson random variable. That is, show that W has the same distribution as 3 X,, where the X; are independent and identically distributed random variables land are independent of N, a Poisson random variable. Identify the distribution of the X, and the mean of N. ‘Compute the conditional distribution of S,,S;,..-, , given that S, = ‘Compute the moment generating function of D(i) in Example 23(C). Prove Lemma 233. ‘Complete the proof that for a nonhomogeneous Poisson process N(t + 8) ~ N(d)is Poisson with mean mit +s) ~ m(0). Let 7), T;,.... denote the interarrival times of events of a nonhomoge- neous Poisson process having intensity function A(). (a) Are the T, independent? (b) Are the T; identically distributed? () Find the distribution of 7), (@) Find the distribution of T;, Consider a nonhomogeneous Poisson process {N(t), 1 = 0}, where 4G) > 0 for allt. Let NO = Now"). Show that {N*(0), ¢ = 0} is a Poisson process with rate A PROBLEMS 95 232, 234, 236, () Let {N(9, = 0} be @ nonhomogeneous Poisson process with mean value function m(i), Given N(@) =n, show that the unordered set of arrival times has the same distribution as n independent and identically distributed random variables having distribution function m(x) Fey=4m 1. Cc ‘Suppose that workers incur accidents in accordance with a nonho- ‘mogeneous Poisson process with mean value function m(). Suppose further that each injured person is out of work for a random amount of time having distribution F. Let X(e) be the number of workers who are out of work at time «Compute E[X()] and Var(X()). ‘A two-dimensional Poisson process is process of events inthe plane such that i) for any region of area A, the umber of events in A is Poisson distributed witb mean 4A, and (i) the numbers of events in nonoverlapping regions are independent. Consider a fixed point, and let X denote the distance from that point to its nearest event, where distance is measured in the usual Euclidean manner. Show that (@) PIX> =e (b) E[X] = W(2Va). Let R,, = 1 denote the distance from an arbitrary point tothe ith closest event to it. Show that, with Ry = 0, (©) sR} ~ sR, = 1 are independent exponential random variables, each with rate A. Repeat Problem 2.25 when the events occur according to a nonhomoge- neous Poisson process with intensity function a(0), t = 0. . Let {N(), ¢ = 0} be a nonhomogeneous Poisson process with intensity function A(), ¢ = 0. However, suppose one starts observing the process at a random time 7 having distribution function F. Let N*(0) = N(x + 1) ~ NG) denote the number of events that occur in the fist time units of observation. (@) Does the process {N*(i), ¢ = 0} possess independent increments? (b) Repeat (a) when (N(d), = 0} is a Poisson process, Let C denote the number of customers served in an M/GI1 busy pe- riod. Find (a) E[Cl. (b) Var(C).96 ‘THE POISSON PROCESS 237. Let {X(),1 = 0} be a compound Poisson process with X(0) = 3 X;, and suppose that the X;can only assume a finite set of possible values. Argue that, for ¢ large, the distribution of X(2) is approximately normal 2.38. Let (X(),1 = 0} be a compound Poisson process with X(0) = 2 X;, and suppose that A= 1 arid P(X; = j} = j/10, j = 1, 2, 3, 4. Calculate P(X(4) = 20). 2.39. Compute Cov(X(s), X(9) for a compound Poisson process. 2.40, Give an example of a counting process {N((), = 0} that is not a Poisson process but which has the property that conditional on N()) = m the first n event times are distributed as the order statistics from 2 set of n independent uniform (0, 1) random variables. 241. For a conditional Poisson process: (a) Explain why a conditional Poisson process has stationary but not independent increments. (b) Compute the conditional distribution of A given (N(9), 0 = s <1, the history of the process up to time 1, and show that it depends on, the history only through (0). Explain why this is true, (©) Compute the conditional distribution of the time of the first event after r given that N() (@) Compute vin PENCOD = 1} i (©) Let x), X;,-.. denote the interartival times. Are they independent? ‘Ate they identically distributed? 242, Consider a conditional Poisson process where the distribution of A is the gamma distribution with parameters m and a: that is, the density is sven by 8A) = ae*(Aayi(m =, OS AS, (a) Show that ronan (Oca Toe REFERENCES ” (b) Show that the conditional distribution of A given N() = n is again gamma with parameters m + n,a +t (©) What is lim POMC + A) — N(Q = 1|N( = nbih? REFERENCES Reference 3 provides an alternate and mathematically easier treatment of the Poisson process. Corollary 254 was originally obtained in Reference | by using a generating function argument. Another approach to the Poisson pracess i ven in Reference 2 1, RM. Adelson, “Compound Poisson Distributions,” Operations Research Quar- terly, 17, 3-75, (196). 2. E, Cinlat, Inroduction to Stochastic Processes, Prentice-Hall, Englewood Clits, NJ, 1975 3. S.M. Ross, Invoduetion o Probability Models, Sth ed., Academic Press, Orlando, FL, 198CHAPTER 3 Renewal Theory 3.1 INTRODUCTION AND PRELIMINARIES In the previous chapter we saw that the interartival times for the Poisson process are independent and identically distributed exponential random vari- ables. A natural generalization is to consider a counting process for which the interarrival times are independent and identically distributed with an arbitrary distribution, Such a counting process is called a renewal process Formally, let (X,, = 1, 2,...) be a Sequence of nonnegative independent random variables with a common distribution F, and to avoid trvialities sup- pose that F(0) = P {X. = 0} <1, We shall interpret X, as the time between the (n~ 1)st and nth event. Let #= E[X) = fj xa) denote the mean time between successive events and note that from the assumptions that X, = 0 and F(0) < 1, it follows that 0 < x = @. Letting S=a3K, nz, it follows that S, is the time of the mth event. As the number of events by time twill equal the largest value of n for which the nth event occurs before or at time 1, we have that N(), the number of events by time fis given by G11) N(@ =supln: 5, <0. Definition 3.4.4 “The counting process {N(), (= 0} is called a renewal process. 98 pisrRiauTion oF N(0) a We will use the terms events and renewals interchangeably, and so we say that the nth renewal occurs at time S,. Since the interarrival times are independent and identically distributed, it follows that at each renewal the process probabilisticaly starts over. ‘The first question we shall atempt co answer is whether an infinite number of renewals can occur in @ finite time. To show that it cannot, we note, by the strong law of large numbers, that with probability 1, 5 Say ane. But since > 0, this means that S, must be going to infinity as m goes to infinity. Thus, S, can be less than or equal to ¢ for at most a finite number of values of n. Hence, by (3.1.1), N(A) must be finite, and we can write N@) = maxtn: 5, = 0h 3.2 Distripution or N(t) ‘The distribution of M() can be obtained, at least in theory, by first noting the important relationship that the number of renewals by time tis greater than ‘or equal to nif, and only if, the nth renewal occurs before or at time t. That is, @2a) NQ=nSS, From (32.1) we obtain (622) PIN() =n} = PIN =n} ~ PIN =n +1) PIS. 51) — PS ya = Now since the random variables X,,i= 1, are independent and have acommon distribution F, it follows that 5, = 2") X, is distributed as F,, the fold convolution of F with itself. Therefore, from (3.2.2) we obtain PIN() = nh = FQ) ~ Fra m() = EIN) ‘m0 is called the renewal function, and much of renewal theory is concerned with determining its properties. The relationship between m(‘) and Fis given by the Following proposition.100 RENEWAL THEORY PROPOSITION 3.24 2a me= SF Proof wo-Si where { ifthe mh renewal occured [0,4] ° otherwise Hence, zwo=2[S] Seu Sew=0 3-0, Where the interchange of expectation and summation is justified by the nonnegativity of the f ‘The next proposition shows that N(j) has finite expectation, PROPOSITION 3.2.2 mQ< 2 foralos1 O such that P(X, = a} > 0, Now define a related renewal process SOME LIMIT THEOREMS | (n= tby [0 ith was N(?)—+ ©. But since N() > © when ¢—+ ©, we obtain Furthermore, writing | “ar - wo” LN+T SOME LIMIT THEOREMS 103 wwe have, bythe same reasoning, S NO “The result now follows by (3.3.1) since 1IN( is between two numbers, each of which converges 10 pa 1» =, oe Exawus 3.3(4) A container contains an infinite collection of coins. Each coin has its own probability of landing heads, and these probabilities are the values of independent random variables that are uniformly distributed over (0, 1). Suppose we are to fip coins sequentially, at any time either flipping a new coin or one that had previously been used. If our objective is o maximize the long-run proportion of lips that lands on heads, how should we proceed? Solution. We will exhibit a strategy that results in the long-run proportion of heads being equal to 1. To begin, let M(n) denote ‘the number of tails in the first m fips, andso the long-run proportion of heads, call it Py, is given by P,= tim NO = tim NO), Consider the strategy that intially chooses a coin and continues to flp it until it comes up tails. At this point that coin is discarded (never to be used again) and a new one is chosen. The process is then repeated. To determine P, for this strategy, note that the times at which a Ripped coin lands on tails constitute renewals. Hence, by Proposition 3.3.1 tim 8) —1/E{number of ips between sucoesive a But, given its probability p of Jaading heads, the number of fips fof a coin until it lands rails is geometric with mean 1/(1 ~ p), Hence, conditioning gives E {number of flips between successive tails] = No) implying that, with probability 1 lim104 RENEWAL THEORY. ‘Thus Proposition 33.1 states that with probability 1, the long-run rate at which renewals occur will equal 1/. For shis reason 2/4 & called the rate of the renewal process. ‘We show that the expected average rate of renewals m(i)/r also converges to 1/q, However, before presenting the proof, we will find it useful to digress to stopping times and Wald’s equation. 3.3.1 Wald’s Equation Let Xi, X:,-.. denote a sequence of independent random variables. We have the following definition, Definition ‘An integer-valued random variable Nis said to be a stopping «ime for the sequence Xj, Xoo ifthe event (N = n} is independent of Novis Nas forall m= 1,2, Intuitively, we observe the X,’s in sequential order and NV denotes the ‘number observed before stopping. If N = n, then we have stopped after observing Xi,-.-,X, and before observing Xyai, Massy +++ Eunnue3.3(0) Let X,, |, 2, be independent and such that PU, = 0) = FAX, = neh. Hewe tt Ne minles X) +--+ X= 10, then Nisa stopping time. We may regard N as being the stopping time ofan experiment that successively fips far cin and then stops when the number of heads reaches 10. Exannur3.3{6) Let X,,70 ,2,.- be independent and such that PIX, = ~Ih= PIX, = Ib ‘Then Ne mines Xt + X= tf SOME LIMIT THEOREMS 105 isa stopping time. It can be regarded as the stopping time for » Gutter who on each play is equaly ely to either win or Tose 1 {init and who decides to stop the first time he is ahead. (It will be Shown in the next chapter that Nis finite with probability 1.) FiEOREM S32 Wald Eason). 19% Aare independet and decay rind random vale bing he Fe acoypin te for Kes uch ha EN < en Proof Lexting ten itN ‘Taking expectations and using Corollary 333 gives bt) + 1)>% implying that 34) int To go the other way, we fix a constant M, and define a new renewal process (%, n= 1,2...) by letting a Ke EXSM, 21,2, iM X> Let, = 21, and (0) = sup: 5, = 9) Since the interarsival times fo this truncated renewal proces are Bounded By M, we obtain Soon 5 + ML Hence by Corolary 33.3, HO + Dawe s+ M, where pu = ER} Thus tinge L Now, since 5, = 8, it follows that N() = N(0) and MH() = m(), thus a5) Letting M— yields 636) and the result follows from (3.34) and (336),108 RENEWAL THEORY When = s, we again consider the truncated process; since wy —+ = as M+ =, ‘the result follows from (3.35). $s Remark At frst glance it might seem that the elementary renewal theorem should be asimple consequence of Proposition 3.3.1. Thatis, since the average renewal rate will, with probability 1, converge to I/., should this not imply that the expected average renewal rate also converges to l/u? We must, however, be careful; consider the following example. Let U be a random variable that is uniformly distributed on (0, 1). Define the random variables ¥,,.n = 1, by 0 itU>1m mn itUsim. Now, since, with probability 1, U will be greater than 0, it follows that ¥, will equal 0 for all sufficiently large n. That is ¥, will equal 0 for all» large enough so that I/n < U, Hence, with probability 1, ¥.>0 asn+o, However, ety =no{u ‘Therefore, even though the sequence of random variables ¥, converges to 0, the expected values of the Y, are all identically 1 We will end this section by showing that N(Q) is asymptotically, as t+ =, normally distributed, To prove this resuk; we make use both of the central limit theorem (to show that S, is asymptotically normal) and the relationship 37) NO e THEOREM 3.3.5 Let wand 0° assumed finite, represent the mean and variance ofan interarival time. Then 0 {JHE KEY RENEWAL THEOREM AND APPLICATIONS m for some constant c. To show that ¢ = 1/p define = (1) ~ m(0) = m(2)— m() = m(n) ~ min = 1) ‘Then implying that xtictsy lim © Hence, by the elementary renewal theorem, ‘When F is latice with period d, then the limit in (34.1) cannot exist. For ‘now renewals can only occur at integral multiples of d and thus the expected ‘number of renewals in an interval far from the origin would clearly depend ‘ot on the intervals’ length per se but rather on how many points of the form rnd, n = 0, i contains. Thus in the lattice case the relevant limit is that of the expected number of renewals at nd and, again, if lim, E [number of renewals, at nd} exists, then by the elementary renewal theorem it must equal dis. If interarrivals are always positive, then part (ji) of Blackwell's theorem states that, in the lattice case, lim P{renewal at nd} # Let h be a function defined on (0, «©, For any a > 0 let m4(a) be the supremum, and (a) the infinum of h(2) over the interval (n= )as 1s na, We say that his directly Riemann integrable if ¥:.,™,(a) andm2 RENEWAL THEORY Zia; my(a) are finite for all a > 0 and lim A sufficient condition for h to be directly Riemann integrable is that © HO = Ofor alle 0, (li) (is noninereasing, iy J; hod < ©. ‘The following theorem, known asthe key renewal theorem, will be stated without proof. ‘THEOREM 3.4.2 (The Key Renewal Theorem). IF Fis not atic, and if he) is directly Riemann integrable, then where To obtain a fee! for the key renewal theorem start with Blackwell's theorem and reason as follows: By Blackwell’s theorem, we have that (ca) — m(e) 1. a @ # and, hence, sm(e-+ a) ~ m(t) 1 @ # ‘Now, assuming that we can justify interchanging the limits, we obtain lim = » ‘The key renewal theorem is a formalization of the above. -THE KEY RENEWAL THEOREM AND APPLICATIONS ua Blackwell's theorem and the key renewal theorem can be shown to be ‘equivalent. Problem 3.12 asks the reader to deduce Blackwell from the key renewal theorem; and the reverse can be proven by approximating a directly Riemann integrable function with step functions. In Section 9.3 a probabilistic proof of Blackwell’s theorem is presented when F is continuous and has a failure rate function bounded away from 0 and « ‘The key renewal theorem is a very important and useful result. It is used when one wants to compute the limiting value of g(@), some probability or ‘expectation at time «. The technique we shall employ for its use is to derive ‘an equation for g() by first conditioning on the time of the last renewal prior to (or at) 1. Ths, as we will see, will yield an equation of the form a(t) = h(t) + fhe 2) dma). ‘We start with a lemma that gives the distribution of Sy, the time of the last renewal prior to (or at) time f. Lemma 3.4.3 Pian Ss) Fos [Fu-namy, case Proof Plsap 8h = 3 PIS. S585) Ft SPSS 2.528 Fo + 3 [Pls 25.5.0 A5, = Fo + Sfp Fe- nary ‘+ [pFu-na(S ro») Foo [)Fe- yd, ‘where the interchange of integral and summation is justified since all terms are nonnegua RENEWAL THEORY Remarks (J) It follows from Lemma 3.433 that PES) = F540) Fo, F—y)dm(y), o 0, amis) = 3 fo) ay = 5 Pith renewal occursin (y,y + dy)} = Plvenewal occurs in (y,y + 4). So, the probability density of Syy is {renewal in (y,y + dy), next interarrival > 1 y} dm(y) FC y). Fig (Vay ‘We now present some examples of the utility of the key renewal theorem, ‘Once again the technique we employ will be to condition on Sy. 3.4.1 Alternating Renewal Processes Consider a system that can be in one of two states: on or off Initially it is on and it remains on for a time Z); it then goes off and remains off for a time Y¥j:it then goes on for a time Z,; then off for a time Ys; then on, and so forth. ‘We suppose that the random vectors (Z,, Y,),m = 1, are independent and identically distributed. Hence, both the sequence of random variables (Z,} and the sequence {¥,} are independent and identically distributed; but we allow Z, and ¥. to be dependent. In other words, each time the process goes ‘on everything starts over again, but when it goes off we allow the length of the off time to depend on the previous on time. Let H be the distribution of Z,, G the distribution of ¥, ,and F the distribu tion of Z, + ¥,,n = 1, Furthermore, let PO ‘system is on at time 9. THE KEY RENEWAL THEOREM AND APPLICATIONS us ———— THEOREM 3.4.4 IY ElZa + Yo) < © and F is nonatie, then iro aa 7 ‘Proof Say that a reewal takes place each time the system goes on, Conditioning on the time of the ast renewal prior to (or at) yields PQ) = Plomat Sun = OPIS + Jf PlonatilSwi = 9} dF, (9) Now Plon attlSyy = 0}= PAZ > AZ, + ¥,> 0) ~ HOF, and, for y < Pon ati|Syq= y) = PIZ> 4 ylz+¥>1-yb He- yiFu-y. Hence, sing Lemma 3.43, Pe = Fo + {Hey Amy), where m(y) = Ein) FQ). Now Fi) is clearly nonnegative, noaineressing, and FH dt = E[Z)"< =. Since this ast statement also implies that H() + 0 a8» ‘=,we have, upon application ofthe key renewal theorem, : fFoa gz Oe ERI ETP] we et O(9 = Plott a) = 1 ~ 2(, then ey 20+ pA ‘We note that the fact thatthe system was initially on makes no difference in the limitus RENEWAL THEORY ‘Theorem 3.44 is quite important because many systems can be modelled by an alternating renewal process. For instance, consider a renewal process and let ¥(0) denote the time from ¢ until the next renewal and let A(@) be the time from rsince the last renewal. That is, YO) = Swine “A= t= Sn ¥(Q) is called the excess or residual life att, and A(j) is called the age at t If we imagine that the renewal process is generated by putting an item in use and then replacing it upon failure, then A(®) would represent the age of the item in use at time ¢ and ¥(e) its remaining lif, Suppose we want to derive P{A(®) = x). To do so let an on-off cycle correspond to 9 renewal and say chat the system is “on” at time ¢if the age at is less than or equal to x. In other words, the system is “on” the first x units of a renewal interval and “off” the remaining time. Then, if the renewal distribution is not latice, we have by Theorem 3.4.4 that lim PLA® $0 Elonin(X, )VELX] ff Poni.) > 9) aretx [,FOv ay. Similarly to obtain the limiting value of P(Y(®) 2}, say thatthe system is “olf” the last x units of a renewal cycle and “on” otherwise. Thus the off time in a cycle is min(x, X), and so Jim P(Y() =x} = lim Plott at Elmin(x, XWVEIX] = [Pod ayin ‘Thus summing up we have proven the following, PROPOSITION 3.4.5 It the intrartval distribution is nonlatice and y < =, then tm PIV) = [Fo) ay {THE KEY RENEWAL THEOREM AND APPLICATIONS 47 Remark To understand why the limiting distribution of excess and age are identical, consider the process after it has been in operation for a long time; for instance, suppose it started at t= —=. Then if we look backwards in time, the time between successive events will sill be independent and have distribution F. Hence, looking backwards we see an identically distributed renewal process. But looking backwards the excess life at is exactly the age at ¢ of the original process. We will find this technique of looking backward in time to be quite valuable in our studies of Markov chains in Chapters 4 and 5. (See Problem 3.14 for another way of relating the distributions of excess and age.) Another random Variable of interest is Xijst = Swiss ~ Syo, OF, equiva dently, Xin = AQ + YO. Thus Xs fepresents the length ofthe renewal interval that contains the point. In Problem 3.3 we prove that PUXngn > 2} = Fe). ‘That is, for any x itis more likely that the length of the interval containing the point ris greater than x than it is that an ordinary renewal interval is, sreater than x. This result, which at first glance may seem surprising, is known 2 the inspection paradox. We will now use alternating renewal process theory to obtain the limiting distribution of Xnjs1- Again let an on-off cycle correspond to a renewal interval, and say that the on time in the cycle is the total cycle time if that time is greater than x and is zero otherwise. That is, the system is either totally fon during a cycle (if the renewal interval is greater than x) or totally off otherwise, Then PAXyn1> 4) = Pllength of renewal interval containing > x} = Plon attime t ‘Thus by Theorem 3.4.4, provided F is not lattice, Elon time in cycle] rn = ELXX> Fan AF, beat lim PX > 2U8 RENEWAL THEORY or, equivalently, (342) Him POs 52 = fi dF Remark To better understand the inspection paradox, reason as follows: ‘Since the line is covered by renewal intervals, is it not more likely that a larger interval—as opposed to a shorter one—covers the point ? Infact, inthe limit (as > ©) its exactly true that an interval of length y isy times more likely to cover f than one of length 1. For if this were the case, then the density of ‘he interval containing the point ¢, call it g, would be g(y) = y dF(yVe (since 4F(y) is the probability that an arbitrary interval is of length y and yie the ty that it contains the point). But by (3.4.3) we see that this is indeed the limiting density. For another illustration of the varied uses of alternating renewal processes ‘consider the following example. Exams 3.4(4) An Inventory Example. Suppose that customers arrive at a store, which sells a single type of commodity, in accor- dance with a renewal process having nonlattice interarrival distribu- tion F. The amounts desired by the customers are assumed to be independent with a common distribution G. The store uses the following (¢, $) ordering policy: Ifthe inventory level after serving 2 customer is below s, then an order is placed to bring it up to S. Otherwise no order is placed, Thus if the inventory level after serving a customer is x, then the amount ordered is Sax ifrS- “THE KEY RENEWAL THEOREM AND APPLICATIONS 119 then itis the N, customer in the cycle that causes the inventory level to fall below x, and itis the Ny customer that ends the cycle, Hence if X,,i = 1, denote the interarrival times of customers, then amount of “on” time in eyel Sx. time of eycle Assuming that the interarrival times are independent of the succes sive demands, we thus have upon taking expectations G44) However, asthe ¥;,i= 1,aroindependent and identically distrib- tuted, it follows from (3.4.3) that we can interpret N, ~ 1 as the number of renewals by time $ — x of a renewal process with interarrival time ¥;, # = 1. Hence, EIN, = me(S ~ x) +1, EIN] = me(S~ 5) +1, where Gis the customer demand distribution and mold) = 3 Galo Hence, from (3.44), we arrive at 1+ me($ =») + meS lim PEK) = ssxsS, 3.4.2 Limiting Mean Excess and the Expansion of m{t) Let us start by computing the mean excess of a nonlattice renewal process. Conditioning on Sy yields (by Lemma 3.4.3) JF) + f ELYCISa9 = FC ») dm). AY 4]120 RENEWAL THEORY Flqure 34.1. Sy = 95 = renewal, Now, ELH(O|Syy = 0] = E[X— AX >A), ELHO1Sy= y] = ELK (€— y)IK> 9h where the above follows since Sjy) = y means that there is a renewal at y and the next interarrival time—call it X—is greater than ¢ ~ y (see Figure 3.4.1). Hence, fobs ELY(O) = EIX dX > FO + [EEX 6 MIX >«— Fey) dim). Now it can be shown that the function h(t) = EL ~ 1X > F() is directiy Riemann integrable provided E[X7] < , and so by the key renewal theorem ELV) [5 1X dX > FO doe [iffe-oare) aie Ji [i -o acarvin (byinterchange of order of integration) = [eden = E[X)2p. “Thus we have proven the following. PROPOSITION 3.4.6 If the intrarrval distribution is nonlatice and E[X"] < =, then lim EY) = ERY2n. Now Syyjss the time ofthe first renewal afters, can be expressed as Suys = YO. pie KEY RENEWAL THEOREM AND APPLICATIONS wa “taking expectations and using Corollary 33.3, we have mln +1) = 0+ ELV] Corollary 3.4.7 IC E[X°] < © and F is nonlattice, then mM) 3.4.3 Age-Dependent Branching Processes Suppose that an organism atthe end ofits lifetime produces a random number af offspring in accordance withthe probability distribution {P, j = 0,1, 2 |} Assume further that all offspring at independently of each other and produce ther own offspein in accordance withthe same probability distribu fion {?.). Finally et us assume thatthe lifetimes of the organisms are indepen- dent random variables With some common distribution F. Tet X() denote the number of organisms alive at The stochastic process (X(), 1 2 0) is ealled an age-dependent branching process. We shall concern ourselves with determining the asymptotic form of M(‘) = ELX()}, when m= poi? > 1 THEOREM 3.4.8 IP Xe=Aom> Ieand F isnot lac, then eee Wafer) emmy ate, where is the unique positive number such that flevarey edm RENEWAL THEORY Proof By dining 07, te Metine oe nora, we ot m= [Fetwotn= nari However, as) ewoin-ai-{ gg et ‘To see why (345) is true, suppose that T; = s,s = 1, and suppose further thatthe ‘organism has j offspring. Then the number of organisms alive t ¢ may he writin as ¥, +--+ ¥, where ¥, i che numberof descendants (including himself) of the fh offspring that are alive at. Clearly, Y;,... ¥;are independent with the same distribu. tion as X(¢= 3). Thus, E(Y, + =° + ¥;) = jM(e~ 3): and (3.4.5) follows by taking the expectation (with respect to) of iM ~ 3) ‘Ths, from the above we obtain 46 Mey = Fig +m M9) aPC, Now, let a denote the unique positive number such that iy and define the distribution G by GG)=mfrerary), Oss = i the above yields | faheheSG, htheme HO= Wd) * J Me~ 3) dels) ‘Now itcan be shown that h(0 s directly Riemann integrable, and thus by the key renewal theorem roa ferFoa 643) Ko Now (49) [lerFod = [re [ares ar = ff flevacare + ehfla-earuy 1 -2(1-4) (by the definition of a), Ao, 410) [fxaoc) =m [ise art ‘Thus fom @.48), (349), and (3410) we obtain “Mo We often consider a counting process for which the first interarrival time has 3.5 Detavep RENEWAL Processes a diferent distibuion from the remaining nes, Por instance, we might tart | |4 RENEWAL THEORY observing a renewal process at some time 1 > 0. Ifa renewal does not occur it then the dtrbuton ofthe tine we mast wat unl he fst ebxeed renewal will not be the same as the remaining interarrival distributions Formally, let (X,, = 1,2,...}be a sequence of independent nonnegative random variables with X; having distribution G, and X. having distribution Fin > 1, Let Sy = 0, §, = 3} Xin = 1, and define No(t) = supln: S, <0) Definition ‘The stochastic process {No(), 1 = 0} is called a general or @ delayed renewal proces, — When G = F, we have, of course, an ordinary renewal process. AS in the ordinary case, we have P{No(t) PAS, = 1} — PAS... S1} = G*F,1(t) — G+ F,(1). La molt) = E[No(®)]. Them it is easy € show that 631) mo) = 3 G+ Fell) and by taking transforms of (3.5.1), we obtain 652) mol) = 1%; By using the corresponding result for the ordinary renewal process, it is easy t0 prove similar limit theorems for the delayed process. We leave the proof of the following proposition for the reader. DELAYED RENEWAL PROCESSES ws PROPOSITION 3.5.1. ( With probability 1 e Mt asim © i I) If Fis not lattice, then molt a) —mo()+2 ast (Gv) If Fand G are lattice with period d, then Elnumberot renewals tnd] asn—+, (9) If Fis not lattice, w < ©, and h directly Riemann integrable, then [5c = amet) [709 dn Ee Exuwrus.5(4) Suppose that a sequence of independent and identi- cally disteibuted discrete random variables Xi, Xs; is observed, ‘and suppose that we keep track of the number of times that a given subsequence of outcomes, or patiern, occurs, That is, suppose the pattern is xy, x:,...,%0 and say that it occurs at time m if Xy = x,, Xyos = Kei. easy = 4). FOr instance, ifthe pattern is 0, 1, 0,1 and the sequence is (X), Xa...) = (1, 0,1,0,1,0, 1,1, 1,0, 1,0, 1, ...), then the pattern occurred at times 5, 7, 13. Ifwe let Non) denote the number of times the pattern occurs by time n, then {Mr}, m= (J is. a delayed renewal process, The distribution until the first renewal is the distribution of the time until the pattern first occurs; whereas the subsequent interarrival distribution is the distribution of time between replications of the pattern. Suppose we want to determine the rate at which the pattern ‘occurs. By the strong law for delayed renewal processes (part (i) ‘of Theorem 35.1) this will equal the reciprocal of the mean time between patterns. But by Blackwell’s theorem (part (iv) of ‘Theorem 35.1) ths is just the limiting probability of a renewal atRENEWAL THEORY time 1, That is, (Eltime between patterns)) {pattern at time n} TL Pix =x}. Hence the rate at which the patern occurs is It, PIX = x) and the mean time between patters is (IY PAX = 23) For instance, if each random variable is | with probability p and 0 with probability q then the mean time between patterns of 0, 1 0,138 p"q"™ Suppose now that we ae interested in the expected time that the pattern 0,10, 1 frst oecurs. Since the expected time to go from 0, 1, 0, 1 to 0, 1, 0, 1 is pq™, it follows that starting with 0,1 the expected number of additional outcomes to obtain 0, 110, Lis p-%q = But since in order forthe pattern 0, 1,0, 1 to.occur we musi fist obtain 0, 1 it follows that Eltime t0 0, 1,0, 1] = Eltime to 0, 1] + p24 By using the same logic on the pattern 0, I we see that the expected time between occurrences of this pattern is 1/(pq); and as this is equal to the expected time of its first occurrence we obtain that Eftime 10 0,1, 0, 1] = pt”? + pg". ‘The above argument can be used to compute the expected num: ber of outcomes needed for aay specified pattern to appear, For instance, if a eain with probability p of coming up heads is succes- sively flipped then Eltime until HTHATHE] = E(time until HTHH] + p%q [time until H] + pq"! + pq? ‘Also, by the same reasoning, Eftime uni kconsecusive heads appear] : (Alp)! + Eltime unit ~ 1 consecutiv heads appear] Suppose now that we want to compute the probability that a siven pattern, say pattern A, occurs before a second pattern, say pattern B. For instance, consider independent flips of a coin that lands on heads with probability p, and suppose we are interested DELAYED RENEWAL PROCESSES a in the probability that A = HTHT occurs before B = THTT. To obtain this probability we will ind it useful to first consider the ‘expected additional time after a given one of these patterns occur until the other one does. Let N gy denote respectively the number ‘of additional flips needed for B to appear starting with A, and similarly for Nu. Also let N, denote the number of flips until A ‘occurs. Then E{Nyu] = Eladditional number to THT starting with HTHT] = Eladditional number to THT starting with THT], But since (Never) = ElNoe] + (Norn we see that Nol = ElNrvee] ~ ENnvc But, Np] = EINs]+ 97" +g ElNoa] = EWN] +479 = 4" 4 47" and so, ElNea] = ap — 9 Also, ENA) =P To compute Py = PIA before B} let M = Min(Ns, Ns). Then EINae) tpg". E[Na] = El] + EIN, ~ M] [iM] + EIN, ~ M\B before A](1 ~ P,) [M] + E{Nas](t ~ Pa). Similarly, E[Ny] = E(M] + E[NoylPo Solving these equations yields + E[Nu [Ninel + E1Nae [Ne] - ElNos] Pa