STOCHASTIC PROCESSES Second Edition Sheldon M. Ross University of California, Berkeley JOHN WILEY & SONS, INC. New York ¢ Chichester ¢ Brisbane * Toronto * SingaporeACQUISITIONS EDITOR Brad Wiley II MARKETING MANAGER Debra Riegert SENIOR PRODUCTION EDITOR Tony VenGraitis MANUFACTURING MANAGER Dorothy Sinclair TEXT AND COVER DESIGN A Good Thing, Inc PRODUCTION COORDINATION Elm Street Publishing Services, Inc ‘This book was set in Times Roman by Bi-Comp, Inc and printed and bound by Courier/Stoughton The cover was printed by Phoenix Color Recognizing the importance of preserving what has been written, it is a policy of John Wiley & Sons, Inc to have books of enduring value published in the United States printed on acid-free paper, and we exert our best efforts to that end ‘The paper in this book was manufactured by a mill whose forest management programs include sustained yield harvesting of its timberlands Sustained yield harvesting principles ensure that the number of trees cut each year does not exceed the amount of new growth Copyright © 1996, by John Wiley & Sons, Inc Alll rights reserved. Published simultaneously in Canada Reproduction or translation of any part of this work beyond that permitted by Sections 107 and 108 of the 1976 United States Copyright ‘Act without the permission of the copyright owner is unlawful Requests for permission or further information should be addressed to the Permissions Department, John Wiley & Sons, Inc Library of Congress Cataloging-in-Publication Data: Ross, Sheldon M Stochastic processes/Sheldon M Ross —2nd ed pcm Includes bibliographical references and index ISBN 0-471-12062-6 (cloth alk paper) 1 Stochastic processes I Title QA274 RES 1996 $19 2—de20 95-38012 CIP Printed in the United States of America 1098765432On March 30, 1980, a beautiful six-year-old girl died. This book is dedicated to the memory of Nichole PornarasPreface to the First Edition This text is a nonmeasure theoretic introduction to stochastic processes, and as such assumes a knowledge of calculus and elementary probability. In it we attempt to present some of the theory of stochastic processes, to indicate its diverse range of applications, and also to give the student some probabilistic intuition and insight in thinking about problems We have attempted, wherever possible, to view processes from a probabilistic instead of an analytic point of view. This attempt, for instance, has led us to study most processes from a sample path point of view. I would like to thank Mark Brown, Cyrus Derman, Shun-Chen Niu, Michael Pinedo, and Zvi Schechner for their helpful comments SHELDON M. RossPreface to the Second Edition The second edition of Stochastic Processes includes the following changes (i) Additional material in Chapter 2 on compound Poisson random vari- ables, including an identity that can be used to efficiently compute moments, and which leads to an elegant recursive equation for the probability mass function of a nonnegative integer valued compound Poisson random variable; (ii) A separate chapter (Chapter 6) on martingales, including sections on the Azuma inequality; and (iii) A new chapter (Chapter 10) on Poisson approximations, including both the Stein-Chen method for bounding the error of these approximations and a method for improving the approximation itself. In addition, we have added numerous exercises and problems throughout the text. Additions to individual chapters follow: In Chapter 1, we have new examples on the probabilistic method, the multivariate normal distribution, random walks on graphs, and the complete match problem Also, we have new sections on probability inequalities (includ- ing Chernoff bounds) and on Bayes estimators (showing that they are almost never unbiased). A proof of the strong law of large numbers is given in the Appendix to this chapter. ‘New examples on patterns and on memoryless optimal coin tossing strate- gies are given in Chapter 3. There is new material in Chapter 4 covering the mean time spent in transient states, as well as examples relating to the Gibb’s sampler, the Metropolis algonthm, and the mean cover time in star graphs. Chapter 5 includes an example on a two-sex population growth model. Chapter 6 has additional examples illustrating the use of the martingale stopping theorem. Chapter 7 includes new material on Spitzer’s identity and using it to compute mean delays in single-server queues with gamma-distributed interarrival and service times. Chapter 8 on Brownian motion has been moved to follow the chapter on Martingales to allow us to utilize martingales to analyze Brownian motion. ixx PREFACE TO THE SECOND EDITION Chapter 9 on stochastic order relations now includes a section on associated random variables, as well as new examples utilizing coupling in coupon collect- ing and bin packing problems. We would like to thank all those who were kind enough to write and send comments about the first edition, with particular thanks to He Sheng-wu, Stephen Herschkorn, Robert Kertz, James Matis, Erol Pekoz, Maria Rieders, and Tomasz Rolski for their many helpful comments. SHELDON M. RossContents CHAPTER CHAPTER 7 LL. 1.2, 1.3, 14 15. 1.6. L7. 1.8. 19. 21. 22. 23 2.4. 25 PRELIMINARIES ul Probability 1 Random Variables a Expected Value 9 Moment Generating, Characteristic Functions, and Laplace Transforms 15 Conditional Expectation 20 1.5.1 Conditional Expectations and Bayes Estimators 33 The Exponential Distribution, Lack of Memory, and Hazard Rate Functions = 35 Some Probability Inequalities 39 Limit Theorems 41 Stochastic Processes 41 Problems 46 References 55 Appendix 56 THE POISSON PROCESS 59 The Poisson Process 59 Interarrival and Waiting Time Distributions 64 Conditional Distribution of the Arrival Times 66 231. The M/G/I Busy Period 73 Nonhomogeneous Poisson Process 78 Compound Poisson Random Variables and Processes 82 251. A Compound Poisson Identity 84 25.2. Compound Poisson Processes 87 xi2.6 CHAPTER 3. may 32 33 34 che 36 3.7. 3.8 CHAPTER 4. 41 42, ba 44. 45 46. CONTENTS Conditional Poisson Processes 88 Problems 89 References 97 RENEWAL THEORY 98 Introduction and Preliminaries 98 Distribution of Nt) 99 Some Limit Theorems 101 331 Wald’s Equation 104 332 Back to Renewal Theory 106 The Key Renewal Theorem and Applications 109 341 Alternating Renewal Processes 114 342 Limiting Mean Excess and Expansion of m() 119 343 Age-Dependent Branching Processes 121 Delayed Renewal Processes 123 Renewal Reward Processes 132 361 A Queueing Application 138 Regenerative Processes 140 3.7.1 The Symmetric Random Walk and the Arc Sine Laws 142 Stationary Point Processes 149 Problems 153 References 161 MARKOV CHAINS: 163 Introduction and Examples 163 Chapman-Kolmogorov Equations and Classification of States 167 Limit Theorems 173 ‘Transitions among Classes, the Gambler's Ruin Problem, and Mean Times in Transient States 185 Branching Processes. 191 Applications of Markov Chains 193 461 A Markov Chain Model of Algorithmic Efficiency 193 462 An Application to Runs—A Markov Chain with a Continuous State Space 195 463. List Ordering Rules—Optimality of the Transposition Rule 198CONTENTS 4,7 Time-Reversible Markov Chains 203 48 Semi-Markov Processes 213 Problems 219 References 230 CHAPTER 5. CONTINUOUS-TIME MARKOV CHAINS 231 51 Introduction 231 52. Continuous-Time Markov Chains 231 53.-Birth and Death Processes. 233 5.4 The Kolmogorov Differential Equations 239 5.4.1 Computing the Transition Probabilities 249 5.5. Limiting Probabilities 251 56. Time Reversibility 257 5.6.1 Tandem Queues 262 5.62 A Stochastic Population Model 263 5.7 Applications of the Reversed Chain to Queueing Theory 270 57.1. Network of Queues 271 572. The Erlang Loss Formula 275 573 The M/G/I Shared Processor System 278 58. Uniformization 282 Problems 286 References 294 CHAPTER 6. MARTINGALES 295 Introduction 295 61 Martingales 295 62 Stopping Times 298 63. Azuma’s Inequality for Martingales 305 64, Submartingales, Supermartingales, and the Martingale Convergence Theorem 313 65. A Generalized Azuma Inequality 319 Problems 322 References 327 CHAPTER 7. RANDOM WALKS 328 Introduction 328 71. Duality in Random Walks 329