McGraw-Hill Series in Electrical Engineering Consulting Editor Stephen W. Director, Carnegic-Mellon University Circuits and Systems Communications and Signal Processing Control Theory Electronics and Electronic Circuits Power and Energy Electromagnetics Computer Engineering Introductory Radar and Antennas VLSI Previous Consulting Editors Ronald N. Bracewell, Colin Cherry, James F. Gibbons, Willis W. Harman, Hubert Heffner, Edward W. Herold, John G. Linvill, Simon Ramo, Ronald A. Rohrer, Anthony E. Siegman, Charles Susskind, Frederick E. Terman, John G. Truxal, Ernst Weber, and John R. WhinneryCommunications and Signal Processing Consulting Editor Stephen W. Director, Carnegié-Mellon University Antoniou: Digital Filters: Analysis and Design Candy: Signal Processing: The Model-Based Approach Candy: Signal Processing: The Modern Approach Carlson: Communications Systems: An Introduction 10 Signals and Noise in Electrical Communication Cherin: An Introduction to Optical Fibers Collin: Antennas and Radiowave Propagation Collin: Foundations for Microwave Engineering Cooper and McGillem: Modern Communications and Spread Spectrum and Random Processes: An Introduction for Applied Scientists and Davenport: Probabili Engineers Drake: Fundamentals of Applied Probability Theory Huelsman and Allen: Introduction to the Theory and Design of Active Filters Jong: Method of Discrete Signal and System Analysis Keiser: Local Area Networks Keiser: Optical Fiber Communications Kraus: Antennas Kue: Introduction to Digital Signal Processing Papoulis: Probability, Random Variables, and Stochastic Processes Papoulis: Signal Analysis Papoulis: The Fourier Integral and Its Applications Peebles: Probability, Random Variables, and Random Signal Principles Proakis: Digital Communications Schwartz: Information Transmission, Modulation, and Noise Schwartz and Shaw: Signal Processing Smith: Modern Communication Circuits Taub and Schilling: Principles of Communication SystemsPROBABILITY, RANDOM VARIABLES, AND STOCHASTIC PROCESSES Third Edition Athanasios Papoulis Polytechnic Institute of New York McGraw-Hill, Inc. New York St.Louis San Francisco Aueklund Bogoté ‘Caracas Hamburg Lisbon London Madrid. Mexico. Milan Montreal New Delhi Paris SanJuan Sio Paulo Singapore Sydney Tokyo Toronto‘This book was set in Times Roman by Science Typographers, Inc ‘The editors were Roger L. Howell and John M. Mon the production supervisor was Richard A, Ausburn, The cover was designed by Joseph Gillians. Project supervision was done by Science Typographers, Inc. R. R. Donnelley & Sons Company was printer and binder. PROBABILITY, RANDOM VARIABLES, AND STOCHASTIC PROCESSES Copyright © 1991, 1984, 1965 by McGraw-Hill, Inc. All rights reserved. Printed in the United States of America. Except as permitted under the United States Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a data base or retrieval system, without the prior written permission of the publisher. 1234567890 DOC DOC 90987654321 ISBN O-07-048477-5 Library of Congress Cataloging-in-Publication Data Papoulis, Athanasios, (date). Probability, random variables, and stochastic processes / Athanasios Papoulis—3rd ed_ Pp m.—(McGraw-Hill series in electrical engincering. Communications and signal processing) Includes bibliographical references and index. ISBN 0-07-048477-5 1. Probabilities. 2. Random variables. 3. Stochastic processes. I, Title. IL. Series, QA23.P2 1991 S$19.2—de20 90-23127CONTENTS Preface to the Third Edition Preface to the Second Edition Preface to the First Edition Part I Probability and Random Variables 1 The Meaning of Probability 1-1 _ Introduction 1-2 The Definitions 1-3. Probability and Induction 1-4 Causality versus Randomness Concluding Remarks 2 The Axioms of Probability 21 Set Theory 2-2 Probability Space 23 Conditional Probability Problems 3 Repeated Trials 3-1 Combined Experiments 32 Bernoulli Trials 3-3 Asymptotic Theorems 3-4 Poisson Theorem and Random Points Problems 4 The Concept of a Random Variable 41 Introduction 42 Distribution and Density Functions xi xiii 12 13 14 15 20. 27 36 38. 38 43. 47 55 60 63 63 viiviii contents 4-3. Special Cases 73 4-4 Conditional Distributions and Total Probability 1” Problems 84 5 Functions of One Random Variable 86 5-1 The Random Variable a(x) 86 5-2 The Distribution of g(x) &7 5-3. Mean and Variance 102 5-4 Moments 109 5-5 Characteristic Functions Ms Problems 120 6 Two Random Variables 124 6-1 _ Bivariate Distributions 124 6-2 One Function of Two Random Variables 13s 6-3 Two Functions of Two Random Variables 142 Problems 148 7 Moments and Conditional Statistics 151 71 Joint Moments 151 7-2 Joint Characteristic Functions 157 73 Conditional Distributions 162 7-4 Conditional Expected Values 169 7-5 Mean Square Estimation 173 Problems 179 8 Sequences of Random Variables 182 8-1 General Concepts 182 82 Conditional Penalties, Characteristic Functions, and Normality 192 83 Mean Square Estimation 201 8-4 Stochastic Convergence and Limit Theorems 208 8-5 Random Numbers: Meaning and Generation 221 Problems 237 9 Statistics 241 9-1 Introduction 241 9-2 Parameter Estimation 244 9-3 Hypothesis Testing 265 Problems 279 Part II Stochastic Processes 10 General Concepts 285 10-1 Definitions 285 10-2 Systems with Stochastic Inputs 30310-3 10-4 il M1 11-2 13 1 11-5 11-6 u-7 12 1241 122 123 124 13 13-1 13-2 13-3 14 14-1 142 14-3 14-4 15 15-1 15-2 15-3 15-4 15-5 15-6 The Power Spectrum Digital Processes Appendix 10A Continuity, Differentiation, Integration Appendix 10B Shift Operators and Stationary Processes Problems Basic Applications Random Walk, Brownian Motion, and Thermal Noise Poisson Points and Shot Noise Modulation Cyclostationary Processes Bandlimited Processes and Sampling Theory Deterministic Signals in Nois¢ Bispectra and System Identification Appendix 11A The Poisson Sum Formula Appendix 11B Schwarz's Inequality Problems: Spectral Representation Factorization and Innovations Finite-Order Systems and State Variables Fourier Series and Karhunen-Lotve Expansions Spectral Representation of Random Processes Problems Spectral Estimation Ergodicity Spectral Estimation Extrapolation and System Identification Appendix 13A Minimum-Phase Functions Appendix 13B _All-Pass Functions Problems Mean Square Estimation Introduction Prediction Filtering and Prediction Kalman Filters Problems Entropy Introduction Basic Concepts Random Variables and Stochastic Processes The Maximum Entropy Method Coding Channel Capacity Problems 427 427 443 455 474 475 477 480 480 487 508 S55 529 533 533 542 558 569 579 59oL 600Selected Topics ‘The Level-Crossing Problem Queueing Theory Shot Noise Markoff Processes Prd Bibliography Index 603 612 629 654 658 661PREFACE TO THE THIRD EDITION In this edition, about a third of the text is either new or substantially revised. The new topics include the following: A chapter on statistics. With this addition, the first nine chapters of the book could form the basis for a senior-graduate course in probability and statistics. A chapter on spectral estimation. This chapter starts with an expanded treatment of ergodicity and it covers the fundamentals of parametric and nonparametric estimation in the context of system identification. A section on the meaning and generation of random numbers. This material is essential for the understanding of computer simulation of random phenomena and the use of statistics in the solution of deterministic problems (Monte Carlo techniques). Other topics include bispectra, state variables and vector processes, factoriza- tion, and spectral representation. T wrote the first edition of this book long ago. My objective was to develop the subject of probability and stochastic processes as a deductive discipline and to illustrate the theory with basic applications of general interest, I tried to stress clarity and economy, avoiding sophisticated mathematics or, at the other extreme, detailed discussion of practical applications. It appears that this approach met with’some success. For over a quarter of a century, the book has been used as a basic text and standard reference not only in this country but throughout the world. I am deeply grateful. McGraw-Hill and I would like to thank the following reviewers for their many helpful comments and suggestions: John Adams, Lehigh University; David Anderson, University of Michigan; V. Krishnan, University of Lowell; Robert J. Mulholland, University of Oklahomia; Stephen Sebo, Ohio State University; and Samir S. Soliman, Southern Methodist University. Athanasios Papoulis xi