This action might not be possible to undo. Are you sure you want to continue?

Print ISBN 0-471-06259-6 Online ISBN 0-471-20061-1

Elements of Information Theory

Elements of Information Theory Thomas M. Cover, Joy A. Thomas Copyright 1991 John Wiley & Sons, Inc. Print ISBN 0-471-06259-6 Online ISBN 0-471-20061-1

**WILEY SERIES IN TELECOMMUNICATIONS
**

Donald L. Schilling, Editor City College of New York

Digital Telephony, 2nd Edition

John Bellamy

Elements of Information Theory

**Thomas M. Cover and Joy A. Thomas
**

Telecommunication System Engineering, 2nd Edition

Roger L. Freeman

Telecommunication Transmission Handbook, 3rd Edition

Roger L. Freeman

Introduction to Communications Engineering, 2nd Edition

Robert M. Gagliardi

Expert System Applications to Telecommunications

Jay Liebowitz

Synchronization in Digital Communications, Volume 1

**Heinrich Meyr and Gerd Ascheid
**

Synchronization in Digital Communications, Volume 2

**Heinrich Meyr and Gerd Ascheid (in preparation)
**

Computational Methods of Signal Recovery and Recognition

**Richard J. Mammone (in preparation)
**

Business Earth Stations for Telecommunications

**Walter L. Morgan and Denis Rouffet
**

Satellite Communications: The First Quarter Century of Service

David W. E. Rees

Worldwide Telecommunications Guide for the Business Manager

Walter L. Vignault

Elements of Information Theory

THOMAS

M. COVER

Stanford University Stanford, California

JOY A. THOMAS

IBM T. 1. Watson Research Center Yorktown Heights, New York

**A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York / Chichester / Brisbane I Toronto
**

I

Singapore

III. 1938 — Elements of Information theory / Thomas M.COM. All rights reserved. John Wiley & Sons. — (Wiley series in telecommunications) “A Wiley-Interscience publication. fax (212) 850-6008. cm. downloading. II. Thomas. Inc. Information theory.Copyright 1991 by John Wiley & Sons. decompiling. printing.54 — dc20 CIP Printed in the United States of America 20 19 18 17 16 15 14 13 . Library of Congress Cataloging in Publication Data: Cover. ISBN 0-471-06259-6 1. 605 Third Avenue. Title. If professional advice or other expert assistance is required. No part of this publication may be reproduced. Inc. except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. recording or otherwise.com. ISBN 0-471-20061-1.” Includes bibliographical references and index. (212) 850-6011. T..C68 1991 90-45484 0030 . E-Mail: PERMREQ@WILEY. NY 10158-0012.Wiley. without the prior written permission of the Publisher. Thomas.. I. Q360. This title is also available in print as ISBN 0-471-06259-6 For more information about Wiley products. Joy A. p. It is sold with the understanding that the publisher is not engaged in rendering professional services. electronic or mechanical. Series. Requests to the Publisher for permission should be addressed to the Permissions Department. the services of a competent professional person should be sought. including uploading. M. New York. stored in a retrieval system or transmitted in any form or by any means. Cover. Joy A. visit our web site at www.

To my father Tom Cover To my parents Joy Thomas .

not by the analysis. this point of view drives our development throughout the book. computer science and statistics. We all know the feeling that follows when one investigates a problem. For example. Also. Perhaps the outstanding examples of this in physics are Newton’s laws and vii .” Although we have not verified the quote (first found in a fortune cookie). there is a chain rule for entropies. but by the inspection of the answer.Preface This is intended to be a simple and accessible book on information theory. As Einstein said. but no simpler. First. goes through a large amount of algebra and finally investigates the answer to find that the entire problem is illuminated. as we shall point out. Thus the answers to problems in data compression and communication admit extensive interpretation. mutual information corresponds to the increase in the doubling rate of wealth given side information. entropy is the minimum descriptive complexity of a random variable. and is intended as an introduction to information theory for students of communication theory. certain quantities like entropy and mutual information arise as the answers to fundamental questions. Second. make the subject appear simple and provide great intuition on new questions. There are two points to be made about the simplicities inherent in information theory. the answers to information theoretic questions have a natural algebraic structure. and entropy and mutual information are related. “Everything should be made as simple as possible. when mastered. For example. There are a few key ideas and techniques that. and mutual information is the communication rate in the presence of noise. This book has arisen from over ten years of lectures in a two-quarter sequence of a senior and first-year graduate level course in information theory.

the response is “sure. Along the way we speculate on the meaning of the second law of thermodynamics.. because of the apparent impossibility of capturing the intangible concept of information. that brings up a point that often occurs in teaching. many of the classical results on determinant inequalities are most easily proved using information theory. Since most of the results in the book are given as theorems and proofs. sure. Does entropy always increase? The answer is yes and no. Vlll PREFACE Schrodinger’s wave equation. in Chapter 2. with no intervening text. AIso the chapter on Kolmogorov complexity has no counterpart in other information theory texts. Again.. We have also taken delight in relating Fisher information. sure. mutual information. While it is clear that Shannon was motivated by problems in communication theory when he developed information theory. probability theory and statistics. To our surprise. This is the sort of result that should please experts in the area but might be overlooked as standard by the novice. the competitive optimality of Huffman codes and the proof of Burg’s theorem on maximum entropy spectral density estimation. In fact. Examples of some of the new material in this text include the chapter on the relationship of information theory to gambling.” But the excitement of teaching the material is greatly enhanced. We were drawn to the field of information theory from backgrounds in communication theory. In many cases we actually describe the properties of the solutions before introducing the problems. we define entropy. Who could have foreseen the awesome philosophical interpretations of Schrodinger’s wave equation? In the text we often investigate properties of the answer before we look at the question.. Even though the field of information theory has grown considerably since Shannon’s original paper. relative entropy and mutual information and study the relationships and a few interpretations of them. the joint typicality proofs of the channel capacity theorem. and the Brunn-Minkowski and entropy power inequalities. followed immediately by the . One innovation in the presentation is our use of long chains of inequalities. we have strived to emphasize its coherence. When one presents these ideas along with the established material in class. showing how the answers fit together in various ways. Thus we have derived great pleasure from investigating a number of new ideas in this text book. It is fun to find new proofs or slightly new results that no one else knows. the work on the universality of the second law of thermodynamics in the context of Markov chains. we expect the elegance of the results to speak for themselves. For example. we treat information theory as a field of its own with applications to communication theory and statistics. the properties are interesting in themselves and provide a natural rhythm for the proofs that follow.

reveals many of the secrets of mathematics. This number. The natural flow of these proofs is so compelling that it prompted us to flout one of the cardinal rules of technical writing. The asymptotic equipartition property (AKP) is given central prominence in Chapter 3. which is the study of the simultaneously achievable flows of information in the presence of . which is the true first chapter of the subject. There is indeed a universal notion of the descriptive complexity of an object. Channel capacity.” Chapter 2. Here we replace the goal of finding a description that is good on the average with the goal of finding the universally shortest description. which accounts for its placement as late as Chapter 13 in the text. is established in Chapter 8. The necessary material on differential entropy is developed in Chapter 9.PREFACE ix explanations. The fundamental idea of Kolmogorov complexity as an intellectual foundation for information theory is explored in Chapter 7. and relatively neglected since. first studied by Kullback in the early 195Os. relative entropy and mutual information as well as a discussion of the second law of thermodynamics and sufficient statistics. which can be roughly paraphrased as “Almost everything is almost equally probable. simplicity and naturalness of information theory. The key idea here is the so-called asymptotic equipartition property. Here also the wonderful number ti is investigated. where the duality of data compression and the growth rate of wealth is developed. A gambling sojourn is taken in Chapter 6. We hope that by the end of the book the reader will share our appreciation of the elegance. which is the binary expansion of the probability that a Turing machine will halt. Rate distortion theory requires a little more background than its noiseless data compression counterpart. The relationship between information theory and statistics. The capacity of the fundamental Gaussian channel is investigated in Chapter 10. we expect that he or she will be able to follow most of these steps without any explanation and will be able to pick out the needed explanations. is developed in Chapter 12. which is the fundamental theorem in information theory. laying the groundwork for the extension of previous capacity theorems to continuous noise channels. By the time the reader comes to many of these proofs. These chains of inequalities serve as pop quizzes in which the reader can be reassured of having the knowledge needed to prove some important theorems. This leads us to discuss the entropy rates of stochastic processes and data compression in Chapters 4 and 5. includes the basic algebraic relationships of entropy. Throughout the book we use the method of weakly typical sequences. which has its origins in Shannon’s original 1948 work but was formally developed in the early 1970s. And the absence of verbiage makes the logical necessity of the ideas evident and the key ideas perspicuous. The huge subject of network information theory.

10. been able to avoid the use of measure theory. 13 and 14. 5. in which a sneak preview and overview of ideas is presented. The bibliography at the end of the book includes many of the key papers in the area and pointers to other books and survey papers on the subject. June 1991 .x PREFACE noise and interference. gives us a chance to recapitulate the interesting inequalities strewn throughout the book. 8. is developed in Chapter 14. Chapter 7 on Kolmogorov complexity is also essential for a deep understanding of information theory. is part of the terrain illuminated by this coherent and beautiful subject. 3. Chapter 1 plays this role. TOM COVER JOY THOMAS Palo Alto. in equation form. do not include the qualifying conditions. with a background of at least one good semester course in probability and a solid background in mathematics. probably senior year or first-year graduate level. put them in a new framework and then add some interesting new inequalities on the entropy rates of randomly drawn subsets. 9. Each chapter ends with a brief telegraphic summary of the key results. which is the generalization of the gambling processes considered in Chapter 6. and shows again the close correspondence of information theory and gambling. We have made an attempt to keep the theory at a consistent level. In our opinion. Every course has its first lecture. At the end of each we have included a variety of problems followed by brief historical notes describing the origins of the main results. however. The primary new ingredients are interference and feedback. This fits in with our belief that the fundamentals of information theory are orthogonal to the techniques required to bring them to their full generalization. the entropy power inequality for the effective variance of the sum of independent random variables and the Fisher information inequalities are made explicit here. The rest. Chapter 15 considers the stock market. The essential vitamins are contained in Chapters 2. This subset of chapters can be read without reference to the others and makes a good core of understanding. These summaries. The beautiful relationship of the Brunn-Minkowski inequality for volumes of set sums. The mathematical level is a reasonably high one. 12. Measure theory comes up only briefly in the proof of the AEP for ergodic processes in Chapter 15. Chapter 16. 4. We have. Many new ideas come into play in network information theory. ranging from gambling to inequalities. on inequalities in information theory.

Hirosuke Yamamoto. Mitchell Trott. Jim Roche. Erik Ordentlich. Aaron Wyner donated his new proof with Ziv on the convergence of the Lempel-Ziv algorithm. Toby Berger. Masoud Salehi. Jack Salz and Raymond Yeung for their suggestions. Will Equitz. Elza Erkip and Vittorio Castelli. We would also like to thank Norman Abramson. Certain key visitors and research associates contributed as well. Ed van der Meulen. students in information theory as the book was being written: Laura Ekroot.Acknowledgments We wish to thank everyone who helped make this book what it is. We would also like to thank the Ph. Don Kimber. Also Mitchell Oslick. We benefited from the advice of John Gill when he used this text in his class.D. Chien-Wen Tseng and Michael Morrell were among the most active students in contributing questions and suggestions to the text. Ben Kawabata. Marc Goldberg and Anil Kaul helped us produce some of the figures. Finally we would like to thank Kirsten Goode11and Kathy Adams for their support and help in some of the aspects of the preparation of the manuscript. Makoto Shimizu and Yoichiro Watanabe. Alon Orlitsky. In particular. Andrew Nobel. including Amir Dembo. Abbas El Gamal made invaluable contributions and helped begin this book years ago when we planned to write a research monograph on multiple user information theory. We would like to thank Bob Gallager for an initial reading of the manuscript and his encouragement to publish it. Jim Mazo and Andrew Barron have made detailed comments on various drafts of the book which guided us in our final choice of content. We were pleased to use twelve of his problems in the text. xi . Paul Algoet.

Fred Jelinek. TOM COVER JOY THOMAS . David Nahamoo and Lalit Bahl for their encouragement and support during the final stages of production of this book. Steve Lavenberg.xii ACKNOWLEDGMENTS Joy Thomas would also like to thank Peter Franaszek.

10 Sufficient statistics / 36 2. relative entropy and mutual information / 21 2.Contents List of Figures 1 Introduction and Preview xix 1 1.1 The AEP / 51 .9 The second law of thermodynamics / 33 2.. 2.1 2.11 Fano’s inequality / 38 Summary of Chapter 2 / 40 Problems for Chapter 2 / 42 Historical notes / 49 3 The Asymptotic Equipartition Property 50 .2 2.7 The log sum inequality and its applications / 29 2.6 Jensen’s inequality and its consequences / 23 2. .4 2.1 Preview of the book / 5 2 Entropy.5 Relative Entropy and Mutual Information 12 Entropy / 12 Joint entropy and conditional entropy / 15 Relative entropy and mutual information / 18 Relationship between entropy and mutual information / 19 Chain rules for entropy.3 2. mrs 3.8 Data processing inequality / 32 2.

5 The horse race / 125 Gambling and side information / 130 Dependent horse races and entropy rate / 131 The entropy of English / 133 Data compression and gambling / 136 .1 6.1 Markov chains / 60 4.3 5.4 6.2 5.4 Hidden Markov models / Summary of Chapter 4 / Problems for Chapter 4 / Historical notes / 77 5 Data Compression 5.11 5.2 6.XiV CONTENTS 3.6 5.3 High probability sets and the typical set / 55 Summary of Chapter 3 / 56 Problems for Chapter 3 / 57 Historical notes / 59 4 Entropy Rates of a Stochastic Process 60 4.8 5.12 a random walk on a weighted 69 71 72 78 Examples of codes / 79 Kraft inequality I 82 Optimal codes / 84 Bounds on the optimal codelength / 87 Kraft inequality for uniquely decodable codes / 90 Huffman codes / 92 Some comments on Huffman codes / 94 Optimality of Huffman codes / 97 Shannon-Fano-Elias coding / 101 Arithmetic coding / 104 Competitive optimality of the Shannon code / 107 Generation of discrete distributions from fair coins / 110 Summary of Chapter 5 / 117 Problems for Chapter 5 / 118 Historical notes / 124 and Data Compression 125 6 Gambling 6.4 5.9 5.7 5.2 Entropy rate / 63 4.3 6.10 5.1 5.2 Consequences of the AEP: data compression / 53 3.3 Example: Entropy rate of graph / 66 4.5 5.

1 7.9 Examples of channel capacity / 184 Symmetric channels / 189 Properties of channel capacity / 190 Preview of the channel coding theorem / 191 Definitions / 192 Jointly typical sequences / 194 The channel coding theorem / 198 Zero-error codes / 203 Fano’s inequality and the converse to the coding theorem / 204 8.8 n / 164 7.5 8.3 8.11 Kolmogorov complexity and universal probability / 169 7.2 8.12 The Kolmogorov sufficient statistic / 175 Summary of Chapter 7 / 178 Problems for Chapter 7 / 180 Historical notes / 182 Capacity 183 8 Channel 8.10 Occam’s razor / 168 7.8 8.1 8.4 7.4 8.5 estimate of the entropy of English / 138 of Chapter 6 / 140 for Chapter 6 / 141 notes / 143 Complexity 144 Models of computation / 146 Kolmogorov complexity: definitions and examples / 147 Kolmogorov complexity and entropy / 153 Kolmogorov complexity of integers / 155 Algorithmically random and incompressible sequences / 156 7.2 7.7 8.9 Universal gambling / 166 7.7 The halting problem and the non-computability of Kolmogorov complexity / 162 7.6 8.12 Feedback capacity / 212 .10 Equality in the converse to the channel coding theorem / 207 8.6 Gambling Summary Problems Historical 7 Kolmogorov 7.3 7.CONTENTS xv 6.6 Universal probability / 160 7.11 Hamming codes I 209 8.

relative entropy and mutual information / 232 9.5 Channels with colored Gaussian noise / 253 10.5 9.1 The Gaussian channel: definitions / 241 10.1 9.3 11.4 11.6 source channel coding theorem / 215 of Chapter 8 / 218 for Chapter 8 / 220 notes / 222 224 Entropy Definitions / 224 The AEP for continuous random variables / 225 Relation of differential entropy to discrete entropy / 228 Joint and conditional differential entropy / 229 Relative entropy and mutual information / 231 Properties of differential entropy.xvi CONTENTS 8.4 9.13 The joint Summary Problems Historical 9 Differential 9.5 11.2 Converse to the coding theorem for Gaussian channels / 245 10.7 Differential entropy bound on discrete entropy / 234 Summary of Chapter 9 / 236 Problems for Chapter 9 / 237 Historical notes / 238 10 The Gaussian Channel 10.3 Band-limited channels / 247 10.3 9.4 Parallel Gaussian channels / 250 10.2 9.6 Entropy and Spectral Estimation 266 239 Maximum entropy distributions / 266 Examples / 268 An anomalous maximum entropy problem / 270 Spectrum estimation / 272 Entropy rates of a Gaussian process / 273 Burg’s maximum entropy theorem / 274 Summary of Chapter 11 / 277 Problems for Chapter 11 / 277 Historical notes / 278 .2 11.1 11.6 Gaussian channels with feedback / 256 Summary of Chapter 10 / 262 Problems for Chapter 10 / 263 Historical notes / 264 11 Maximum 11.

7 12.5 13.CONTENTS mii 12 Information 12.4 13.8 12.11 The method of types / 279 The law of large numbers / 286 Universal source coding / 288 Large deviation theory / 291 Examples of Sanov’s theorem / 294 The conditional limit theorem / 297 Hypothesis testing / 304 Stein’s lemma / 309 Chernoff bound / 312 Lempel-Ziv coding / 319 Fisher information and the Cramer-Rao inequality / 326 Summary of Chapter 12 / 331 Problems for Chapter 12 / 333 Historical notes / 335 Theory 336 13 Rate Distortion 13.3 12.5 14.10 12.9 12.7 Characterization of the rate distortion function / 362 13.4 12.7 Information Theory 374 Gaussian multiple user channels / 377 Jointly typical sequences / 384 The multiple access channel / 388 Encoding of correlated sources / 407 Duality between Slepian-Wolf encoding and multiple access channels / 416 The broadcast channel / 418 The relay channel / 428 .8 Computation of channel capacity and the rate distortion function / 364 Summary of Chapter 13 / 367 Problems for Chapter 13 / 368 Historical notes / 372 14 Network 14.6 Quantization / 337 Definitions / 338 Calculation of the rate distortion function / 342 Converse to the rate distortion theorem / 349 Achievability of the rate distortion function / 351 Strongly typical sequences and rate distortion / 358 13.1 14.6 Theory and Statistics 279 12.2 14.1 13.4 14.3 13.5 12.1 12.2 12.6 14.3 14.2 13.

2 15.5 16.6 15.4 16.*.1 15.8 16.8 14.9 14.10 CONTENTS Source coding with side information / 432 Rate distortion with side information / 438 General multiterminal networks / 444 Summary of Chapter 14 / 450 Problems for Chapter 14 / 452 Historical notes / 457 Theory and the Stock Market 469 15 Information 15.9 Basic inequalities of information theory / 482 Differential entropy / 485 Bounds on entropy and relative entropy / 488 Inequalities for types / 490 Entropy rates of subsets / 490 Entropy and Fisher information / 494 The entropy power inequality and the BrunnMinkowski inequality / 497 Inequalities for determinants / 501 Inequalities for ratios of determinants / 505 Overall Summary / 508 Problems for Chapter 16 / 509 Historical notes / 509 510 526 529 Bibliography List of Symbols Index .2 16.1 16.3 16.5 15.XV111 . 14.3 15.6 16.7 16.4 15.7 The stock market: some definitions / 459 Kuhn-Tucker characterization of the log-optimal portfolio / 462 Asymptotic optimality of the log-optimal portfolio / 465 Side information and the doubling rate / 467 Investment in stationary markets / 469 Competitive optimality of the log-optimal portfolio / 471 The Shannon-McMillan-Breiman theorem / 474 Summary of Chapter 15 / 479 Problems for Chapter 15 / 480 Historical notes / 481 in Information Theory 482 16 Inequalities 16.

2 4.4 1.1 The relationship of information theory with other fields Information theoretic extreme points of communication theory Noiseless binary channel.2 5.2 5.3 1. A noisy channel Binary symmetric channel H(p) versus p Relationship between entropy and mutual information Examples of convex and concave functions Typical sets and source coding Source code using the typical set Two-state Markov chain Random walk on a graph Classes of codes Code tree for the Kraft inequality Properties of optimal codes Induction step for Huffman coding Cumulative distribution function and Shannon-FanoElias coding Tree of strings for arithmetic coding The sgn function and a bound Tree for generation of the distribution ( $.6 5.2 1.3 5.1 4. $ ) Tree to generate a ( i.3 3.1 5.2 2.4 5.List of Figures 1. a.9 7.1 2.1 1.5 2.7 5.8 5.5 5. i ) distribution A Turing machine 2 2 7 7 8 15 20 24 53 54 62 66 81 83 98 100 101 105 109 111 114 147 xix .1 3.

4 12.11 8.4 8.10 8. Pz of A 12.XT LIST OF FIGURES 7.5 7.9 12. Noisy typewriter.2 8.6 8. Noisy channel with nonoverlapping outputs.2 7.6 8.2 10. Binary erasure channel Channels after n uses A communication channel Jointly typical sequences Lower bound on the probability of error Discrete memoryless channel with feedback Joint source and channel coding Quantization of a continuous random variable Distribution of 2 The Gaussian channel Sphere packing for the Gaussian channel Parallel Gaussian channels Water-filling for parallel channels Water-filling in the spectral domain Gaussian channel with feedback Universal code and the probability simplex Error exponent for the universal code The probability simplex and Sanov’s theorem Pythagorean theorem for relative entropy Triangle inequality for distance squared The conditional limit theorem Testing between two Gaussian distributions The likelihood ratio test on the probability simplex The probability simplex and Chernoffs bound Relative entropy D(P.1 One bit auantization of a Gaussian random variable 158 173 177 177 178 184 185 185 186 187 188 192 192 197 207 213 216 228 235 240 243 251 253 256 257 290 291 293 297 299 302 307 308 313 314 317 318 337 .3 10. 1 ) and D(P.12 9.12 Probability simplex for a football game 13.1 12.7 8.10 H.8 8.8 12.2 12.6 12.3 8.9 8.3 12. 11 ) as a function IP.4 7.5 12.1 10.5 8.2 10.6 12.5 10.11 Distribution of yards gained in a run or a pass play 12.4 10.1 9.(p) versus p Assignment of nodes Kolmogorov sufficient statistic Kolmogorov sufficient statistic for a Bernoulli sequence Mona Lisa A communication system Noiseless binary channel.1 8.3 7.7 12. Binary symmetric channel.

3 14.9 14.4 13.5 13.9 13.LIST OF FIGURES xxi 339 343 344 345 346 349 361 365 370 375 375 376 376 382 383 388 389 390 391 391 392 392 395 403 403 406 408 412 414 416 417 417 418 419 13.7 13.8 13.2 14.1 14.3 13.20 14.21 14.5 14.7 14.8 14.14 14.25 Rate distortion encoder and decoder Joint distribution for binary source Rate distortion function for a binary source Joint distribution for Gaussian source Rate distortion function for a Gaussian source Reverse water-filling for independent Gaussian random variables Classes of source sequences in rate distortion theorem Distance between convex sets Joint distribution for upper bound on rate distortion function A multiple access channel A broadcast channel A communication network Network of water pipes The Gaussian interference channel The two-way channel The multiple access channel Capacity region for a multiple access channel Independent binary symmetric channels Capacity region for independent BSC’s Capacity region for binary multiplier channel Equivalent single user channel for user 2 of a binary erasure multiple access channel Capacity region for binary erasure multiple access channel Achievable region of multiple access channel for a fixed input distribution m-user multiple access channel Gaussian multiple access channel Gaussian multiple access channel capacity Slepian-Wolf coding Slepian-Wolf encoding: the jointly typical pairs are isolated by the product bins Rate region for Slepian-Wolf encoding Jointly typical fans Multiple access channels Correlated source encoding Broadcast channel Capacity region for two orthogonal broadcast channels .18 14.15 14.16 14.10 14.22 14.4 14.2 13.6 13.24 14.11 14.23 14.10 14.6 14.13 14.19 14.17 14.12 14.

1 The function fit> = .26 14.1 Sharpe-Markowitz theory: Set of achievable meanvariance pairs 16.38 Capacity region of a broadcast channel 14.34 14.35 14.33 14.27 14.32 14.28 14.36 426 426 427 428 428 433 438 443 444 448 449 452 456 456 460 489 .37 Multiple access channel with cooperating senders 14.29 14.t log t 14.39 Broadcast channel-BSC and erasure channel 15.31 14.30 14.xxii LZST OF FIGURES Binary symmetric broadcast channel Physically degraded binary symmetric broadcast channel Capacity region of binary symmetric broadcast channel Gaussian broadcast channel The relay channel Encoding with side information Rate distortion with side information Rate distortion for two correlated sources A general multiterminal network The relay channel Transmission of correlated sources over a multiple access channel 14.

Elements of Information Theory .

511 Adams.Elements of Information Theory Thomas M.510 AEP (asymptotic equipartition property).3.511.367. 191. 492 ASCII.408. 147.227 discrete random variables. 59.373. 269. 10. 136-138 Arithmetic mean geometric mean inequality.306. 61. Levinson. P. 13 effective size. 221 Additive white Gaussian noise (AWGN) channel.458. 319-326 references.50-59.65. 11. 319-326 Algorithmically random.223. S.216218 joint.421 A&l. 195. 305.. 466 Atmosphere.481. 270 529 .366. 156. S. Cover. R. 181-182 Algorithmic complexity..124. 181. 276 Frank-Wolfe. 124. 162. xi Adaptive source coding. 224. xi.. R. Stirling’s.367. 92.. 147. see Gaussian channel Adler. 284 Approximations to English. 144.201-204. 511 Asymmetric distortion..288-291. 195. 47 1 Ahlawede. 107. R. J.454 Achievable rate distortion pair.223. 275 universal data compression. 146. see AEP Asymptotic optimal@ of log-optimal portfolio.406. 1.510 Algorithm: arithmetic coding. 389.510 Approximation. 104-107.404.. B. 237 input. 220. ix.510 Algoet. 184 Alphabetic code. 6.179. 368 Asymptotic equipartition property (AEP). 191. 239 discrete. L.366. Thomas Copyright 1991 John Wiley & Sons. 226. 184 output.326 Ash. 341 Achievable rate region. N.384 stationary ergodic processes. 166. See also Shannon-McMillan-Breiman theorem continuous random variables. 510 Acceptance region. K.457. Joy A.309-311 Achievable rate.. x. Print ISBN 0-471-06259-6 Online ISBN 0-471-20061-1 Index Page numbers set in boldface indicate the Abramson. 124. 110-l 17 Huffman coding. 191 generation of random variables.373 Durbin. 46. 104-107. 182 Alphabet: continuous. 151. xi. 474-480 stock market. 133-135 Arimoto. 96 Amari.110 Lempel-Ziv.Sl. 167. M. See also Blahut-Arimoto algorithm Arithmetic coding. 49. Inc. 282. 107 Additive channel. 133..136-138 Blahut-Arimoto.510..

380. D. 59.. 209. 154.367..496 Bounded distortion. 525 Bergmans. 272. 458.389. 377-458. 422-424 capacity region.356. 13.43 Bahl..218. 187-189.343.384. 278 Burst error correcting code.44. 305. 248.497. 14.. 177. 457 INDEX Birkhoff’s ergodic theorem. xi.474 Bias. ix.458. viii. 175..425-427. R. 212. 240.. 410. R. J.332 BCH (Bose-Chaudhuri-Hocquenghem) codes.209. 208. C. R.418-428.500. 78. 2.358.. 220 Burg.406 Barron. 512 Breiman. see Channel. J. 276. T. 379-380. 389. 143 Bernoulli random variable. 8. A. W.454-458 common information..451.335..366.382.517 Bellman. 442. 166.374.512 Blahut-Arimoto algorithm. 14 rate distortion function. 326.377. 354 Brain. 422.43. A.. 367. 106. 189. 421 definitions.445 Boltzmann.334. 421.379.334 Bierbaum.407 Band-limited channel. 287. C.373.530 Atom.246 Average probability of error. 391 Binary multiplying channel.42. T. 460 . 104.. 358. 212 BUZO.373.399.238 Autocorrelation.512 Bottleneck.509 BSC (binary symmetric channel).. 422 converse. 247 Bandwidth. 212 Beckner.426 Average codeword length.. 212. 186. 150 graph of. 143.. H.420. 8.288 Blocklength. L.407 Bandpass filter. 166. E. 474 Bit. P. 13. 422 stochastically degraded.477. 396. 367-369 Berry’s paradox. 511.511 Benzel.221 source coding.478 Bose.456 Binning.449.511 Baseball. 239. 512 Broadcast channel. 329.509.512 Berlekamp. 157. L. 457. 114-116.496. 184-223.454 entropy.467. R. 10. viii.335 Biased. R. 396 Capital assets pricing model. 14 Blachman. 14. 249.523 Bernoulli. 356. R.223. 342.512 Binary entropy function. 191. 457 Binary random variable. 14. 512 Blahut..379..422 multiple access channel. 10. 273 Auxiliary random variable. 519 Calculus. 371. 523 Band. 497..379. H.374. 276. L.278. 512. 420-422 degraded: achievability. 340. capacity Capacity region.222.. 418-420. 314-316. 422 examples.392. 511 Bell. 512 Burg’s theorem. 291.137-138. 85 Average distortion.. See also Shannon-McMillan-Breiman theorem Brillouin.511 Berger. 13 Bayesian hypothesis testing. 316 Base of logarithm. xi. 342 Binary symmetric channel (BSC). E. 411. J. 59. P. C. xi. 47 Bounded convergence theorem. 212.. 49. 455 physically degraded.425-427 Gaussian.349. 481. L. 498.223. 454.262. R.361 Average power.262. 85. 163 Betting. 267 Capacity.373 Block code: channel coding. 211. 247-250. 390-458 broadcast channel. 193.501. 221. 53-55.214 multiple access. 511 Bennett. 126133.7-10.220.211.427-428 Brunn-Minkowski inequality. 15 Binary erasure channel. x.. 277 Autoregressive process. 218 with feedback.56.186. 146 Brascamp. 128 Borel-Cantelli lemma.159.86. 194 AWGN (additive white Gaussian noise). 291. 191. See also Maxwell-Boltzmann distribution Bookie. N.512 Blackwell. 273. 457. see Gaussian channel Axiomatic definition of entropy. 342.239-265. M.250.236. 482. 512. 274. 511 Bell.. 191. 49.508 channel.164. 49. 320. see Bernoulli random variable Binary rate distortion function..

90 Caratheodory. 32. G.53-55.422.524 Closed system.78-124. Multiple access channel. 483. 4.318. 333.18.232. random.513 Chernoff bound.423 Cocktail party.107. see Error correcting code Hamming. 194-222. 315.237. 366-367 Computers. vii.466 Central limit theorem. 10. 107.493 differential entropy. 379 Code.378-456 Coding. 147.449 Channel transition matrix.367 examples. A. 220. 184 operational definition. 10 Cloud of codewords. 8.179 Computable probability distribution.400. 189.239. 312. 490. 184. 3.402. 239-256. 222. Broadcast channel. 232 entropy.312-316. Gaussian channel.374-458 alphabetic.21-24.505 Chain rule.470. 182. viii. 1..110 Compression. 226. see Distributed source code error correcting.. 212 Shannon.256. 21 growth rate.511. 193 rate distortion.193-222.397. 341 Reed-Solomon. 398 Competitive optimal& log-optimal portfolio. see Shannon code source. 513 Channel. see Random coding Coin tosses.7-10.184.374 Chebyshev’s inequality. 309.491. 136-137 block. 124. 221. See also Binary symmetric channel.75 2 (Chi-squared) distribution. 28.136137. 184.513.374-458. 78. 240. 337 Codeword.332 Chessboard. 34. 183.377. 13. 441. 191. 3.133. 119 Codepoints.54. 164.377. 8.380 portfolio strategy. J. xi Cauchy distribution.161.508. J. 39. 64. 458. 465. 1.485.447. 147 models of.414.242-258. 422.488 Communication system..351. 223 cascade.401.144-181.204-206.425 Caste& V.215 Communication theory. 170. 23 Chaitin. 186. 70. vii.6.355-362. 346 Cesiro mean. see Data compression Computable. 341 Codelength. G. 57 Chernoff.8.253. Relay channel. 1. 47. ix. B. 141 Carleial.367 halting.258. 96 arithmetic. see Hamming code Huffman.219. 193. 327. 480. 239265.49.486 Choi. 36.. 469.215-217 Channel coding theorem.513 .398..78-124. 312. 146 rate distortion function. 159 Computation: channel capacity. S. 191. 189 Channel code. 398 Cardinality. 486 Cauchy-Schwarz inequality. 194 zero-error. 338.425 discrete memoryless: capacity theorem. 3. 6. 65. 16. 146 Centroid. 215. 4-6.212 Compact set. B. 470. 7. 59. see Channel code convolutional.96. 161 Computable statistical tests. 145.514 Chung. 146 Cipher..337-358. see Source code Codebook: channel coding. 212 distributed source.57. 104-107. see Huffman code Morse. 162 Concatenation. 136 Cleary. K. 469 mutual information.374 Computer science. L. 221. 314. 80 rate distortion. 132.4. 8689. 185-223.335.320. Two-way channel capacity: computation. H.426 Cards. 22 relative entropy. 8. 192 feedback. 257. 435. 3. 7.275.94. 183.163. 110 Coin weighing.318 Chernoff information.INDEX 532 Church’s thesis. 421 Communication channel. 45 Common information.. 206-212 definitions. 212-214 symmetric. 68. 184-190 information capacity. 198-206 converse. 512 Cascade of channels. 145 Compact discs.291 Central processing unit (CPU). 471-474 Shannon code. 198 Channels with memory. see Block code channel. 80. Interference channel.3.329 Causal. 10. 194. 278. 512.

518 Cumulative distribution function.364. see Slepian-Wolf coding Correlation. 46.. 396 conditional rate distortion function.260 inequalities.238. 250.462. 49. 449.210 .. 485-497 table of. 192. 461. H.396-398. 48. 225-238.. 288. D.40.458. 515 de Bruijn’s identity.488. 224. See also Convexity Concavity of entropy.479. M. 121 Decoding function. 396 Conditional relative entropy.454 entropy.155. 432. 136. 511 Data compression. 14. 317. 137. 304. D. 288.505. 137.454.396. 509. 124.273. 49. 146 Cramer. 215 Dimension. xi. 138.237.517 Costello. 161.237. 432. 212-214 Gaussian channel. degraded Dembo. 319 Davisson.24-27. 457 Deterministic function.483 Consistent estimation.370. 355 discrete memoryless channel. 433-436 Convex hull. 509. D. 519 Covariance matrix. 327 Constrained sequences. xii. 230.440-442. see Broadcast channel.459. 450 Convexification. 486-487 Digital.510 Correlated random variables. 45. 59. 450.23. 104. 353. 3 Dempster.458. 226 Converse: broadcast channel. 76. 3.482-484. 514 Density. 375. 182. x. 416.330.297.226. 364-367. 224. 282.. 269.354. 440-442 rate distortion theorem. 124. 305 Differential entropy. 430 Degraded. 191. 222. T. 325-329.299. P. 24-26. 399-402 rate distortion with side information.454 distance between. 87 Daroczy. 22. 245-247 general multiterminal network.508 universal. 268. 332 Conditionally typical set.203-220.454. Relay channel. 510. 193. 136 Degradation.501-505 Cover. degraded.371.254-256. 101. 102. 9.393.362. 464 Convolution.513.319. 143.309. ix.339..336. 38. Rate distortion theory AEP. 129.224 D-adic.323. 238.448. 104.44. 295. 279. 138. 370. 358.481.267.77 Continuous random variable. 212 Coppersmith.335. 359. Z. 167. I. 193 Decryption. 514 Cramer-Rao bound. 224.407. 483.488 Decoding delay.31.215 Digitized. A. see Hypothesis testing Decoder. 454 Convexity. See also Differential entropy. 287. 106.. 297-304. ix 1.454. 3-5.255. 16 Conditional limit theorem. 335. 494 Decision theory.370.INDEX Concavity. 449.215-217.403.377.523 CPU (central processing unit).501. 136 Csiszir.511. L.229. 29-31. 23.486-507 Determinant.191.267-271. 349-351 Slepian-Wolf coding. 42.513-515..291. 53. 137.362. 457.369. J. 501-508 Deterministic. 256. 349 Convex sets. 235.337. 498 Convolutional code. M. 498.494 Crosstalk.514. 219. 265. 413-415 source coding with side information. 117. Quantization.336. 60.375 Cryptography. 23 Conditional type. H. 454 multiple access channel. 28. See also Concavity capacity region: broadcast channel. 43 Dice.461-463.233.31 Conditional differential entropy.225-231. M.395.. 30-32 rate distortion function.479. 14.. 78.247.. A.331.29. 445-447 multiple access channel.506.405-451. 439 entropy and relative entropy.237. 206-212 with feedback. 278.371 Conditional mutual information. 514 Demodulation. vii. 230.267.332. 32. 371 Conditioning reduces entropy. 184. 165. 38. 202. 264 encoding of. 146. 449 Costa. 41. 316. 230 Conditional entropy. 389. 374. 72.

407.110.104. 374.65-67. 133-136 Entropy.449. 125. xi. 296. P. xi Erasure. 515 Dog. 9. 19. 77.. 138. 265.513.249. 273 Gaussian process. xi El Carnal.471.. vii-x.. 49 Equitz.458.482.458. 208 Dobrushin. 357 source coding and generation of random variables. 133-136. L.9-13.332. 319-326. 193 Encrypt. 03.214. 35.. L.352-356.3-6. 300 Distortion. 439444. 139. 4 Effectively computable. 452 Ergodic. 151. See also Slepian-Wolf coding Divergence. 49 DMC (discrete memoryless channel). 340 squared error. See also Rate distortion theory Distortion function. 408 Distributed source coding. 499 Entropy of English. 4.138. 239. G. x.108. 296-298.351. 136 Energy.337. 341 Distortion typical. 80. M.339.135.208. 139 models of. See also Conditional entropy.. discrete memoryless Discrete random variable.. 490-493 Epimenides liar paradox. 327. 137 growth rate and entropy rate. 49 Distance: Euclidean.515 Economics.349. 69 Markov chain. ix. 9. 10. 273. 49. 342.391. 249. 273-276 hidden Markov models.508. 13 Discrete time.473. 103. 460 Eggleston.515 Durbin algorithm. 515. 1. R. 138. 515 Elias. 349.184. see Entropy Discrete memoryless channel. 339 Itakura-Saito. viii. 133.470 multiple access channel and Slepian-Wolf coding. 49.457. 187-189.378 Discrimination. 494.14.336-372. 352. 262.449. 124.258.192 Encoding. Joint entropy.460. 66 subsets. 110 Dueck. 339 Distortion rate function.373 bounded.458.402.377. 126.352.364. E. 299 variational.243. 146 Efficient estimator. 215.64-78.450. 133. 13. 75 Doubling rate. 184 gambling and data compression. 79.518.330 Efficient frontier. 162 Epitaph.496-501. 139.367-369.266. x.20 and relative entropy. W. 27.88-89..485 Empirical entropy. x. 339.452. 494-496 and mutual information.139. 4 16-4 18 rate distortion and channel capacity. vii Ekroot. 266.131-139. 123 Ebert. 128.340-342. 432.255. 106. 339 Distortion measure. 113-116. 459.368. 336.INDEX 533 Dirichlet region. 215-217. Differential entropy..369 Lq.505. 462-474 Doubly stochastic matrix. 129 Dyadic. 398. 193 Encoding function. P.376. 215-218 differential. 133..509 Entropy rate.72 Duality. 263.270 England. 457. See also Shannon-Fano-Elias code Empirical.475-478 E&p.291 entropy rate. 137.279. 419 Dutch book. xi . 155 Encoder. G. 338 Discrete channel.279 Empirical distribution. 195 Empirical frequency. 151.457.361 Distributed source code.379 Hamming. 279. 276 Dutch. 139.59. Relative entropy and Fisher information.5 data compression and data transmission.370.354 Hamming.474.377. 126-131.392. A. A..5. 34 English. 184 Discrete entropy.515 Eigenvalue. 193.367 Einstein.496. 499 Entropy power inequality. 125. 10. 143.443.30 Renyi.143 Entropy power. see Channel. H. 104.

232.. 250-253 Gaussian distribution. Interference channel. 141.131. 165 Finite alphabet. 270. R.87. 13.. 31. 194. 166. 486 Error correcting codes.406 baseball. 222.. 494. 250 colored noise.516 Feller.516 Fisher information. 472.450. 110-l 17 Geodesic. 108.42. 445 Generation of random variables.223 Gaussian channels. 374. 166.447. 143. 515. 72 Flow of water. 273 . 376.515.167. 326.516.. 189. 239-247.364.457. Petersburg.268.516 Functional. 486 Fair odds.219.143. 288 Flag. 334. 142.285 Fisher. 49.308 Geophysical applications.. 309 Geometric distribution. 256-262 parallel Gaussian channels.497 Fixed rate block code. 167 Game: F distribution. 318 Ford.400-402. R. 97. 486 Gas.266. 222.48. 253-256 feedback.534 Erlang distribution. 124. A.319 Finitely often. 322 Geometry: Euclidean. 406 Fulkerson. 80 Face vase illusion. 326-329. 486 Extension of channel. 332.246.213. 337.282 Frequency division multiplexing. 376.501.291. 473 Fan. xi. 165 First order in the exponent.458 Feinstein. M. A. 516 Fermat’s last theorem.252. 4. 317. 317. 245-247 definitions. 49. 1. 473 Fair randomization. 473 twenty questions. Multiple access channel. 193 Extension of code. 486 Feedback: discrete memoryless channels.127.327.248.49.379 Expectation.317 football. 282. 247. 228. 445. 242 converse. 326.506 efficient. 487 General multiterminal network.318 Hi-lo. 479 Finitely refutable. 38-40. T. x. 344-346 Gauss-Markov process.. 11. 335. D. 450.496.377.349.448. 125-132. 241 power constraint. theoretic optimality. 164. P.223. Relay channel additive white Gaussian noise (AWGN).270 Gaussian channel.304. 182 Factorial. 154.383..494.132.39. xi.378 achievability. 4. see Spectrum estimation Estimator.455 IALDEX empirical.473 universal. 59. 297.446. 124. 143 Shannon guessing. 237. viii-x. 132 St.328-332. 138 stock market. 133-135.455.284 function. See also Broadcast channel.281. 12.332 Estimation. 9. 121 mutual information.523 Galois fields. 133 Gallager.193. 129. See also Shannon-Farm-Elias code Fano code. D. 516 Fourier transform.516 Freque cy.211 Error exponent. 204-206. 244-245 capacity.413-415. 152 Franaszek. G.347. 488. 256-264 networks. R. 107.457.13. 53 Flow of information. 248. 247-250 capacity.432.294. 338 rate distortion function.368. 239-265. 3. R. R.516 Forney. 25 Exponential distribution.362 Gaarder. 435.272 Fractal.. 330 Euclidean distance. Ky. 136-138. 274-277 Generalized normal distribution. 316. W.223.448 Flow of time. 16. 457.297.446. see Normal distribution Gaussian source: quantixation.266.357 relative entropy.250.509 Fano.139-141. 6. 296-298.124 Fano’s inequality.305-316. 263 red and black.516 Gadsby. 120.377. 239 band-limited. G. A. 482.256.212-214..94..95 Game.465 Gamma distribution. 212 Gambling.506 spectrum. L. 377 Football. 86.279.

49. 24. 165.39. 140. 124.179 Independence bound on entropy.92-110.509 Fano’s. P.409 Gradient search. 265.176. S. 267.124.329 Chebyshev’s. 154. 1. R.446.517 Holsinger. 1.457. 209.41. 57.514. 66-69 state transition. 263.457.90-92. 125-132.247. 410 Hassner. 498. E. 516 Grouping rule.517 Huffman code. A. 232. A.. 26. 10. 490-493.335.362 Innovations. 29. 162-164 Goode& K. see Neyman-Pearson lemma iff (if and only if).59.346. R.288.. 100.481. 467 Information.d.291 Hypothesis testing. 49. 29. 209-212. 459 Guiasu.509 Hocquenghem. 43 Growth rate optimal.441. 496-501. viii.516 Hadamard. 516 Godel’s incompleteness theorem. 496. W.238... 233. U. E.471..368. 419 Hekstra. 145. 141. 121-124. A.497 Inequalities. 49. 27.509 Cauchy-Schwarz. 110. 312-315 optimal.. 178. M.. 106 distortion measure.304-315 Bayesian. J. 229 Gilbert.81. 55.).339 Hamming distortion.464.181 Incompressible sequences. L.. 96. D. 157.185-190. 163 Infinitely often (i. 101 Koknogorov structure function. 119-123.287. 78. T.. 4. 342. 69-7 1 High probability sets. 143.d. 92. 41. 139 Historical notes.42. 106. 163.519 Grenander. 162-163 Hamming. xi Goldman. 156-158. 482. 18.INDEX 535 i.. 323 Inference.87. source. 516 Hadamard inequality.372.473 Huffman..524 Hidden Markov models.. 77. R.499 Ziv’s. 482-509 arithmetic mean geometric mean. 327.342. 508 Jensen’s.97. 7.484.288.483 Markov’s.. 501-508 entropy power. 341.30. 49.516 Hamming code. S. V.278. 25.. see Information capacity Information inequality.342.318. 10. 6 i. 258 Input alphabet.516 Gill.519 log sum. l&4. S.204-206.502 information.458. 449. 165 Goldberg.4. x. 223.400-402. M. 136 Graph: binary entropy function.510 HDTV. 153. 5. 124.509.458.246. xi Goldbach’s conjecture. 26.216 Induction.516.478 Minkowski. 517 Horse race. 152.474 Images. P. 457.267.505.188 Instantaneous code. 25.373.171.509 triangle. See also Prefix code Integrability. 169 Gray. 500. 124. 47.517 Hopcroft.265.114. 212. 57 determinant.349. 38-40. 233. 76 Gravestone. Mutual information. 299 Young’s. 505 subset.508 Information rate distortion function..447. 182.o. Y.498.510..119.466.i.82.468. 514 Gotham. 182.56 Histogram. 27..26.517 Hartley.i. viii. T.484. 206. 28 Indicator function. 78.77. M.97.47. 253 Information channel capacity.83-92. 339.497.82. x. 251. 492 Bnmn-Minkowaki.435. 155. 184 Input distribution.204.494.482. 62. 171. 191 Grammar. see Fisher information. 177 random walk on. 49 Gravitation. 193.502 Halting problem.413-415. 180.218 Gaussian channel. 151.351.482 Kraft. 455. xi Gopinath.31.323. 86 .85. 78. Self information Information capacity. 110-124. 15 cumulative distribution function.165. 187..213.501. B. V.48. 517 Horibe. (independent and identically distributed). 222. 209-212 Hamming distance.369 Han.517 Hash functions. 136 Kolmogorov complexity. J. 339 entropy of.107. 6. 241.. J.

179. 155 and universal probability.370-372 Joint source channel coding theorem.196-202. 443 distortion typical. J. 127. ix.437. 169-175 Kohnogorov minimal sufficient statistic. xi Kawabata..374.335.378. L.519 Leibler. 176 Kohnogorov structure function.519 Language.310.517 Kohnogorov.352. G.. 215 Itakura-Saito distance.436. 49. 352.362.367 Landau. 477 Lexicographic order.153. xi Law of large numbers: for incompressible sequences. M.519. L. A. 11 horse race.347.138-140 Laplace. R.155.468. B.366. 1. 78.518.27. D.218. 348. 10. R.306-308. 163 Lieb. W..162 Khinchin. 512 Likelihood. 179.80. 175-179.442. 376.292-318 Lavenberg.442. E.511 Langdon. 519 Lempel. xi.349. 10... 249.181 method of types.6.316 Lin. V.468. 24. T. N.457. 352-356 Gaussian. 49.454. A. 457..66.252.10.297.471-473 Irreducible Markov chain. 252.178. 371.458. 7.110-124.182.536 Interference.306 Likelihood ratio test.486 Large deviations.458 Interference channel.9. 161. T.458. Ya.181.. 133-136.385 Lehmann. 104-105. 42.144. 524 Jaynee.138.364.57.195. 382-383 Intersymbol interference. 182.312. 160. 182.83. S.255. J.. 4.508 of integers.191. viii.83-92.388. 523 Joint AEP. 49. 85.145. 517 KaiIath. K. M. l&46 Jointly typical.163. 53. 142 Letter. A.. the Virginian.461.466.514.273.173 Kuhn-Tucker condition& 141.139 Leung.518 Kemperman.196-202. G.49.276. xi. 161. 195.158.515.470-472 Kullback.. 320 Lempel-Ziv coding. 334.. 162.3.518 Kobayashi.279.335 Leningrad. J... J.26. 319.297. 107. S. 212.464. 482 Johnson. 288.359..384-388 Joint distribution.. 518 Kieffer.462-466. ix. 518 Kimber. 276 Levy’s martingale convergence theorem. H.382. 143. 215-218.384.126. 518 Kraft.432. 459-474 Investor. 124. 458. H. 291. 277. 169 Laplace distribution. 358. 358-361..390..517 Jefferson..9.361.383. T. 237.519 Kullback Leibler distance.371 h8te8en. E.518 Kendall.62.295. 406.... 519 .217. I.217. 244.351.519 Levinson algorithm.323.152. A. 518 Kaul.. R. E. S.198. 157. 143. 47.335. 340 Jacobs. 238. 15 Joint entropy.525 Lempel-Ziv algorithm.82.360 Liao. F. 61. G. C.. C. 124.520 Levin.468..182 Kiirner. 335.. 18.457.384-387. 3. 195. 154.. ix.514 Knuth.. 4.441. 41. 107. 137..519 Liar paradox. 26.292.517 Jensen’8 inequality. 465..133-135. 319-326.247. 144.232.417.124.278. D. J. 168. 175 Kohnogorov sufficient statistic. 518 Keyboard.510.418. 18. 245 strongly.122. 49. 279. J.514.274. 76 Intrinsic complexity.216 ISDN. A. 226. xi Kelly.218.373..4. entropy of.519 Landauer. 140 Jelinek.147.. K.458 Gaussian. 145 Investment.245. 124.203. L. 517 Karueh.245.144-182..484 Lagrange multipliers. s.180.518 Kotel’nikov. J. 450. 335. 177. 124. 29.444. A. R. 294.. 288. NDEX 11.. J.518 Kraft inequality. 286-288 strong. 147. 308. 481. H.474 weak. A.11. 49. 59. 231. 332.250.518 Kohnogorov complexity. E.. xi King.51. 125-132 stock market.216 Joint type. 50.287.76.

460 Marshall.254-262. 10. 61.430-432 Lloyd. See also Channel. 136.211 Linear predictive coding.. 176 Minimum description length (MDL). 222. 199.30.35. 124 Malone. base of. 30 Models of computation. 511.520 Multiparameter Fisher information.471. 72. 77. 477 Marton. 140 Mandelbrot set.387-407.. 121 state transition. 266.519 Marginal distribution. 12’7.274-278 property of normal.374. 35. 7.377.518. 36.466.78. 519 Martian. 124. 210 Minkowski.. H..166. 29. 520 Minkowski inequality. See also Shannon-McMillan-Breiman theorem McMillan’s inequality..519 Macroscopic. M.220 Maximum posterion’ decision rule. 126-142. 124 MDL (minimum description length)... D. S. 18.330 determinant inequalities.. 27 distribution. 210 Linear code.505 Maximal probability of error. x Median.49. G. P. 47. 137. 130. 210.. 59. 90-92. 3 14 Maxwell-Boltzmann distribution. 388 covariance..INDEX 537 Linde... 182 Mazo.. 184. 10. 266. 258..62.483 Longo. J.75.443 Markov process.318. 221. 330 Multiple access channel.58.457 . 519 Linear algebra. 266.307 Log-normal distribution.41. xi Morse code. 238 Medical testing. discrete memoryless Mercury.31.478 Markowitz. 72 parity check. C. E. 338. 119.267 process. 268 Midpoint. W. 382.273. 102 Minimal sufficient statistic.49. 330 Minimum weight. 59. 373. 382. 210 permutation. 76 Toeplitz. 266 -272.378 between convex sets. xi McDonald. S.478-481 Log sum inequality.452.519 Logarithm. 234.47.458. 76.519 Lucky.80.204. 364 relative entropy. R. 297 Minimum variance.. channels with. 169 Merges. 120. 152 Marcus. 457. 167. 140. 17.215.33-38.253 Memoryless. 32. 78. 117. 305 Memory.379.471. 124. 32 Markov lemma. 279-286. R. 10. See also Wealth Monkeys. B. 487 Log optimal. 266. 124. 6.358.342.490 Microprocessor. 66-77.435-437.367.271..268. H.218. 520 Mixture of probability distributions.504. 486 Log likelihood.41. A.27.458 Molecules.4X-418. 436.61.57.274-277.520 Matrix: channel transition. 184. 38.484 Markov field. 182 Minimum distance. 35. 75.520 McEliece.460 Money.. J. Y. 190. 34. 461-473. viii.487 Maxwell’s demon. C. 122 Merton.. X0-162.403.269 Magnetic recording.520. 514. 57. 223. 72 probability transition. 143. 210-212.48. 62. 270 Moments. 250 Modulation. 234 Maximum likelihood. K. 182 Measure theory. 505 Mirsky. A. 501-508 doubly stochastic. L. 49 Macrostate. 193 Maximum entropy. 146 Modem. 184 Method of types. 238.66. 255. 273 List decoding. F.516 Morrell.446-454. 441. L.61. 118 Martingale. 148 Microstates. 35. 520 McMillan.80 Moy.514 Lovasz. 13 Logistic distribution.. B. 3.. 181 Moore.267.262 conditional limit theorem. See also Gauss-Markov process Markov’s inequality. 210. 178.75. 520 Message. 304 Markov chain.241 Modulo 2 arithmetic. 269 discrete random variable. R. R.345.

165-167.349.520 Paradox. 390-392 with feedback. 82.10. 183. xi Noise.378-384..538 Multiple access channel (Continued) achievability. 36.230.384.. 72 Perpendicular bisector.. 59.. A.4. 247.. J. 487. S.238-265.162.. 388 duality with Slepian-Wolf coding. 487 Parity.445. K. 181 Non-singular code. 1933. 399 with correlated sources. 520 Optimal decoding.332 Physics. J.508 additive noise channel.231.. 406 Multiuser information theory.524 Oppenheim. 484 relative entropy.416 Nonlinear optimization.521 Perihelion. 497 generalized. 447. 249 Oscillate. 124.247.341-457. 18. xi Orey.458. 483 mutual information. 10.168. 199. D. 251-253 Parallel Gaussian source. 448 definitions.248.520 Pagels. H. ix. 450. 487 maximum entropy property. 210 Parsing. 3.90 Norm: INDEX entropy of. 272 Permutation.183-222..912. 215.473 Olkin. S. 416-418 examples. 11. 419 Orthonormal. R. R...323. SO.332.163 Parallel channels.. 266 9. 169 Periodic.320. 250 frequency division. 123. 209.40-49. 4-6. 308 Perturbation.388.521 Patterson. 459. 519 fl. I.190.238-264. L. 347-349 Pareto distribution. 13 Nearest neighbor. 142-143. xi Ornatein. 81. 230 Music.230. S.126-130..450. M. 234 multivariate. 520 Neyman-Pearson lemma. 191 Non-negativity: discrete entropy.299. See also Gaussian channels. 524 Nahamoo. 3. 264 Parallel Gaussian channels..520 Orlitsky. 7. 1353%326.484~508 chain rule. 520.391. 189.. 393 capacity region. L. H.248 Occam’s Razor. 484 Nonsense. 220 Noiseless channel. 124. 253. 145. 164. 162. 182. 184 Ozarow.370 matrix.161.. 497 Philosophy of science. 33.131.136. 501-506 Nyquist.396.211 Parity check code. xi Nats.524 Neighborhood. 161.444.319.58. 209 Parity check matrix.4. 64 Oslick.337 Neal. see Network information theory Multivariate distributions. 230.404-407. xi Output alphabet.125. vii. 274.374-458 Newtonian physics.306. 1.357. 22 Myers. H.6. 1. 407 time division.145. 305. 389 converse. 22 conditional.332 Nobel.457. 450 Gaussian.181 Omura. D.132.. 127 Optimal portfolio.130. G.520 Nyquist-Shannon sampling theorem.403-407 Multiplexing. 378-379. 225.376-378.335 Pasco.179. D.474 Oracle. S&225.448. Gaussian source .487 entropy power inequality. 80-82.374.467.268 Multivariate normal. W. 229.374.236.488 z& 498 Normal distribution.458 Network information theory. 247. 3. 165 Ordentlich..235.368.215.. 142.169 Odds. 520 Orthogonal. viii. 232-265.325.141. 248 Periodogram. M. 270.122.522 Pearson. E.369. E. A. A.. A.220. viii.49. 1. 292 Network. 450. 376..215 Mutual information. 3. 169 Neyman. 520 Perez..379 Optimal doubling rate.322. 4 Phrase. 59.

73.260. 249 Proportional gambling. 344 parallel Gaussian source.524 Ray-Chaudhuri.184. 138 Pythagorean theorem. 276.411.380-382. 81.430-434. 146. 349. 255. see Entropy rate source coding.336-373 achievability. 249. 418 Radium.97.. see Horse race Radio. 11. 6. 330.333 Prolate spheroidal functions. 121-124. 428 degraded.272 Precession. 34.367 Reza.216 Reed. 456 Rathie. R.286. 62. 487 Rebalanced portfolio. 139. 77.378-382.287..342-369 binary source.167. 369. 430 reversely degraded.4.408. 212.. 461-473.440-442 Rate distortion region. 432 Gaussian. 1..512 Rayleigh distribution. 147.265. S. 199-202. 441 Rate distortion function. 222. 127. 18.453. 519.332 Quantization. J. S.521 Pinsker.66. 247. ix. 428. 249.136. 238 Random coding. 3. 340. 448. 341 Rate distortion theory. 502 Prior: Bayesian. 228 Quantizer. 127.85. 162 Pixels.305 average. 10. N. E..458 achievability. L. F. 122. P.. 224. 126. K. 137. 99. 461 Receiver. 169 Prefix.475 Punctuation. 346 Rate distortion function with side information. 128. H. M. 280.501-507 Posner. 342 Rate region. J. 439.320 Prefix code. 130.172. 349 theorem. 472.ZNDEX 539 Rank. 194 entropy.448. 73 Polynomial number of types. 150 Redistribution. 430 converse. 521 Rice. 212 Reed-Solomon codes. S. 432 Renyi entropy. 471-474 log optimal. 519.61. 351-358 converse. 393. 239. 364 Relay channel.294.84. 461-473. 122. 98. 152 Pollak. 459.281. 521 Pinkston. 499 Renyi entropy power..460. 523 Polya urn model. 129.458 Power spectral density.346. 130. S.154.. 430 capacity. see Achievable rate channel coding.7.77. 55 Rate distortion code.457. 3. 124.81. 384. 183 Recurrence. 193.. 521 Race. 143.432 with feedback. 134 .. 127.430. 72. 124 Principal minor.272.7. 82. 18. 53. 143. 297.427. 102-104.460 Relative entropy.454. 241. 106.262. 5. 249.367.521 Pitfalls.82. 198. 276 Probability mass function. R.473 Random number generator.230.357. J. 341.. 168 Probability density. 13 Probability of error. 203. 288.. 143.403-407. 194.461.478-481 competitively optimal. M. 376378.416.334. 514 Power.. 337 Rabiner. 9-11 achievable. 34 Redundancy. 3 12-3 16 universal. 380 physically degraded.75 Recursion. 182.297. O.389 maximal. 73. 349 Gaussian source.209.92.351-370 with side information. 461 Positive definite matrices. 287.231 Relative entropy “distance”. C.368 Quantized random variable. 319.164. 342 convexity of. 499 Reverse water-filling.353.239-264.301.84. 521 Rissanen.514 Portfolio.92. 485. 432 Randomization. 193. 137.478-481 rebalanced.521 Rate. 121 Projection. 238. I. 10. 139. O. 210 Rae. 142. 133. 3.451. 430 definitions. 35. T.357.302 Pombra. 265. 240.407.336-338.367. D. 212 Reinvest.225.202 Probability transition matrix.

125. 12.3. 458.328 Second law of thermodynamics. 522 Sanov’s theorem. 149 Self-reference..409 converse. P.379.467-470. See also Shannon code.405 Simplex. 377.. 344 Source channel separation. A.463 Sine function. F. 59. 42. See also Statistical mechanic8 Self-information. 182.433. 337.. 10. 80.459 Shannon-Fan0 code.309.377. 474 Signal. 409-410.358 Squared error. E.255.522.517.96.407.494. G.223.. 240.369.124.198.273.. A.357. 243. M.523 Slepian-Wolf coding. 288 Source coding theorem.523. 457. 481 Sanov. 124. 318 Sardinas. 413-4 15 distributed source code. 457. J. 522 Score function. 189 Set sum. 2. 162 Shannon.144. 342 Sharpe. R. 248 Singular code. viii.523 . E...452. 498 agn function. A.372.59.49.520 Schwarx. P. C. 410-413 achievable rate region.525 Schnorr.107.388. 419 Spectral representation theorem.243. 523 Shuffles. N. 415 many sources.338. R.. 68. 182.130. 327. J.454 interpretation.. 96 SNR (signal to noise ratio). lOl-104..458.76 Run length.292. 10. J. Shannon-Fano-Eliaa code. 286. 415 Slice code.522 Satellite. 89.291-296.383.108. 136. 151. 108-110 Shakespeare.345. 272-278 Speech.340. 249. 215-218.471. H. J. P. 449.. 168.457 achievability. xi Sampling theorem.367-369 &am. 59. 255. 222. 238 Schafer. 376 Slepian. 250.475. 10 binary.374.523 Shim&.481. 12.. 481. 10. I.4. 11 and doubling rate.20 Self-punctuating.524 Source..480 and source coding.334.272.4.49. 122.405 Software.95.433-442. 149 Solomon. 248 Samuelson.. 408. M. 250. 408 duality with multiple access channels. xi Shore. 248.131. xi Rook.449.138-140. 23. 10.499. 448.407-418. W.124 Shannon-McMiIlan-Breiman theorem.48 Side information. 121. 36.113. C..454. 521 Schalkwijk.9.312. 77.477.95. 479.513 SaIx.. G.265.329.143. 3 Signal to noise ratio. J. 458 Sigma algebra..522 Schuitheise. 1. P.248.43.451.349 Spectrum.520-622 Sandwich argument. xi.457. 47 Saddlepoint. 274.124.509.509. 308.423 Sato.498 Sphere packing.275-278.313. 342 GaUSSian.340 Sphere. 373.474. 3. See also Shannon code Shannon-Fano-Elias code.. 162 Sender. 497.. 118.346. 416-418 examples. 460.134. W. 142-143 Salehi. 9. 212 Solomonoff.380.47. 79 universal. Petersburg paradox.449 Source code. 444. D. viii. Shannon-McMiIlan-Breiman theorem Shannon code.377. 474-478 Shannon’8 first theorem (source coding theorem). 263 St.274. M.117.249.308.457.522 Scaling of differential entropy. vii.451.357. A. 80. 3. vi.503-506 distortion.87. 182.33-36. 88 Shannon’8 second theorem (channel coding theorem).522. 198 Shannon’s third theorem (rate distortion theorem). 88 Spanish. 81 Sink.349 Spectrum estimation. M.380.135.215.INDEX Roche. 1..

523 Tangent.65.286.280. 287. 487 Trigram model.123 Summary. see Jointly typical strongly typical set.6.305. 458. 50. 18. 38 sufficient.141 Submatrix.373. 151.38. 129. 45 football. 11.318 gambling.. 81 Tie reversible. 93. A. 76 Stationary: ergodic process.331. 146. 391.469.80. 312. 13. 124.ZNDEX State diagram.384.359.160 Tus&dy. A. 37.383. 317. 183 Treasury bonds.226. 59. 133-136 Stochastic process.479.181. 524 Trace.370-372.504. 175 Suffix.49. 147 Turing machine.. xi Tseng. M.509 Subspace.. 34.. 169. 129-131.407. 10.407. 224 Surface. 375.471.220 Synchronization.118-120.269. 211 Szasz’s lemma.49.465. 469-47 1 Statistic. 210 Subtree. A.447.61-78. 407 Surface area.3.473.508 Strategy. 181.459481. x.319-326.273. 4. W.47.505 Tornay.3SS Stuart. 136 Stein’s lemma. 345 Thermodynamics.3. 69 Timesharing.106 Sufficient statistic.456. 460 Tree. 457 Type.331 Symbol. 487 Subfair odds. 490-493. 255 Markov chain. S.49. 9.33.65. viii-x.171-174 Triangle inequality. 136 Trott. 51-54. xi Turing.36. D. 155. 262. 125.436.272-278 stationary.228 jointly typical set. see Stationary Stock.453 Toeplitz matrix.480 Stock market..98.35.4. 503 &ego..275. 72 Time division multiplexing. 37 Statistical mechanics. 120.. G.367. A. G.309.218.. 207. 72-74.89 stock market. See also Second law of thermodynamics Thomas.424.178. 272-277. 7-9..450. D.191.122.280. 279. 4.442 Strongly typical set.437 Typical set. 12.93. 117.457 Support set. 11.141 Superposition coding.508 Superfair odds. xi.474-478 Gaussian channels. C.268 Ternary. 200 .406. 146.176.236. C.. 459.279.227. 182 Tanabe.60-78. 140. 154. 126. 63. 1. 518 Student-t distribution. 1. 255.391 Test channel.. J. 1.471.88.71. 250 Temperature. 254.487 Tallin. 358.56.333 Stirling’s approximation. 134. 332.255 Transmitter. 35.89. 370.82. flow of.396. 10. 63 Symmetric channel. L.418 Two-way channel. 50-56. 18. 190.159.59. 76 Systematic code. 40. 435. M.96. 284 Stochastic approximation to English. ix. x.84. 509. 486.223 Strongly typical. 371 Typical set decoding.310. 226 Typical sequence.147.36-38. 374. 422. 502 Subset inequalities. 460 Telephone.522 Tang. J.278 Statistics.395.476 ergodic.407 Time invariant Markov chain.459. 111-116. 228.450. 266. L.281-286.185. 9.466. 255.430-432.282. 36-38.490 Type class.61-74.304 of English.514 Tuttle.250 channel capacity. 516 Table of differential entropies. 386. see Ergodic Gaussian.208.485 process. 166 portfolio.97.406.279.276. 161.51. 465-471 Strong converse.367. 519 TV.490 Typewriter. 364-367.420. 36.299 Triangular distribution. 176 minimal sufficient. 178. 176. 130. 512 Time. 105. 379.220 Typical.280-318.104-106. 60. 513 Thomasian.47.41.333 Kohnogorov sufficient.. 11. 131. 247.460. 502.

M. 485. J.525 Zvonkin. 226. M.499 Zero-error capacity.511. 12. 21.. 326 Um. W.494 Uncertainty. 9.9. 247. 3. 167 Universal probability. N.357. H.. 407. 27.223 Zhang. 519 . xi Yao. 331. 182. 247.511 Young’s inequality.317. K. 300 Vase. 519. 58. M. 10. T. 459-480 Wealth relative. 166.. 107.287-291. Y. 124. xi. J. 327. 516. xi. 335. Z. 335.457. 28. J.525.457. A.487 discrete.340. 18-20. 225. 101.73 V’yugin. 90-92.72. 181 Venn diagram. 124. 120-124 Universal. J.523..383 Weakly typical. 35. J. 249. 523 White noise.Ullman.. R.80. 4. F. P. 525 Yamamoto. 79. See also Covariance matrix Variational distance. 144-148.41. 73 Uniform fair odds. 523 Wilcox.525 Ziv.234-264. 182.450. 280. 419 Volume. G. A. 5.6. x. 338 Wald.48. A. J..458.382. W. 264. 523 Van der Meulen. 238. 100 Universal source coding..174.458.. M. K. H..518 Yeung... See also Lempel-Ziv coding Ziv’s inequality. 523 Vocabulary. 36. 320. 498.. 517 Unbiased. 141 Uniform distribution: continuous..524..118..443. See also Gaussian channel Wiener. D. xi Waterfilling.117. 326.57.409 Weaver. x Viterbi.524 William of Occam. 14. 151.270.. 166.. 226-229.81. 320. J.. C. V. 523 WeibulI distribution. 512 Watanabe. 10. I.524 Variance. 223. 47.85. 243. see Typical Wealth. 44 Wozencraft. V.. A. A. 4. E. 160-162.326-329. 443.82. A. W. 108.320. 5. 498-501 Voronoi region. J. 129 Uniquely decodable. J. 323 Zurek.319. W.274.335.349 Waveform.524 Vitamins. 524..330. xi. 30.405. 335.458. ix. 170. 524 World Series. 34..515. C...22.450.45. 524 Willems. 458. 319. 524 Wyner. 107. 457.524 Wolf. 161 Witten. A. 524 Woodward. 457.524 Van Camper&out.326 UNIX..316. 72.178-182 Universal gambling. 222. D. 487 Welch..334. 126 Weather. 35. 256. 124. C.516... M. 229. 523 Wallmeier. H. 125-142. 454. H. 124.450 Unfair odds. 167. 155.332. 523 Wolfowitz.46 Verdugo Lazo.45. H.

Sign up to vote on this title

UsefulNot usefultelecommunication

telecommunication

- lec9 tans
- Survey Mutual-Information-Based Registration of Medical Images
- Decision Trees
- k nearest neighbor
- information theory
- Block Diagram of Communication System
- Error Correction Coding
- Lecture note of Wireless Communication (Charles Darwin University)
- Understanding the Impact of Interference on Collaborative Relays
- Digital Communication Notes
- A JOINT TIMING OFFSET AND CHANNEL ESTIMATION USING FRACTIONAL FOURIER TRANSFORM AND CAZAC SEQUENCES
- Fading
- Radio Procedures.pdf
- Ultrawide Band Wikipedia
- Ora000004 Cdma2000 Principle Issue4.0
- Call Proceessing of cdma
- 818-4829-1-PB
- 06 Keysight Agilent Hewlett Packard MIMO Performance and Condition Number in LTE Test 5990-4759EN
- A Tutorial on Convolutional Coding With Viterbi Decoding
- SMART ANTENNA IEEE
- 3G MOBILE COMPUTING
- 3g Mobile Computing
- 3G MOBILE COMPUTING [DownloadPPts.com]
- 25143515 Wireless Communication
- Rutgers Lib 40558 PDF 1
- Elements of Information