NORMAN ABRAMSON Associate Professor of Electrical Engineering ‘Stanford University INFORMATION mHoiLORY — comes, CopINc lll McGRAW-HILL ELECTRONIC SCIENCES SERIES EDITORIAL BOARD Ronald Bracewell Colin Cherry Willis W. Harman Edward W. Herold John G. Linvill ‘Simon Ramo John G. Truxal ABRAMSON Information Theory and Coding BREMER Superconductive Devices Nor Ye introduction ‘ory of Finite-state Machines in Francisco GILL Introduction to the Theory of Fi te 5 London PAPOULIS ‘The Fourier Integral and Tts Applications STEINBERG AND LEQUEUX (TRANSLATOR R. N. BRACEWELL) HUELSMAN Circuits, Matrices, and Linear Vector Spaces | Radio Astronomy McGRAW-HILL Book Company, IneCONTENTS Preface v Glossary of Symbols and Entropy Expressions INTRODUCTION 1 a 12 13 4 15 What Information Theory Is Not 1 What Information Theory Is 3 Encoding Information 3 ‘A Problem in Taformation ‘Transmission 6 Some Questions 9CHAPTER 2 INFORMATION AND SOURCES 24 The Definition of Information 11 2 The Zero-memory Information Source 18 24 Some Properties of Entropy. 1 Extensions of a Zero-memory Source 19 The Markov Information Source 22 The Adjoint Source 27 Extensions of a Markov Source 20 ‘The Structure of Language 83 CHAPTER 3 SOME PROPERTIES OF CODES BA Introduetion 45 2 Uniquely Decodable Codes 47 3 Instantaneous Codes 49 “€4 Construction of an Tnstantaneous Code §2 65, Tho Kraft h Disoussion 5% “66 The Kraft Inequality—Proot 57 £7 MeMillan’s Inequality 59 88 Some Examples 00 wality—Statement and CHAPTER 4 CODING INFORMATION SOURCES we Average Length of a Code 65 42° A Method of Encoding for Spec Sources. 68 £3 Shannon's First Theorem 44 Shannon's Firet Theorem for Markov Sources 74 445. Coding without Extensions 75 44 Finding Binary Compact Codes— Hutlman Codes 77 47 Completing the Proof 82 “€8 rary Compact Codes 83 £9 Code Bificiensy and Redundancy 85 n 6 6 CHAPTER 5 CHANNELS AND MUTUAL, INFORMATION 3 5A Introduction 98 \§2 Information Channels 94 5:8 Probsbility Relations in a Channel 98 54 A Priori and A Posteriori Entropies 100 55 A Generalization of Shannon's First ‘Theorem 101 5-6 Mutual Information 105 5-7 Properties of Mutual Information 107 58 Noiscless Channels and Deterministic Channels 111 5.9 Caseaded Channels 113, 5-10 Redwood Channels and Sufficient Reductions 118 ‘of Mutual Information 123, Information of Several Alphabets 127 443. Channel Capacity 131 5-14 Conditional Mutual Information 135 sun 2 CHAPTER 6 RELIABLE MESSAGES: THROUGH UNRELIABLE CHANNELS. 7 6-1 Introduction 147 62. Error Probability and Decision Rules 149 63 The Fano Bound 153 6-4 Reliable Messages and Unreliable Channels 155, ‘An Example of Coding to Correct Errors 158) 66 Hamming Distance 163 67. Shannon's Second Theorem for the BSC— The First Step 165, 68. Random Coding—The Second Step 170 69 Shannon's Second Theorem— Discussion 172 zGLOSSARY OF SYMBOLS AND ENTROPY EXPRESSIONS rrxiv GLOSSARY OF SYMBOLS AND ENTROPY EXPRESSIONS _[Tnformation A427 | channel e be Inputalphabet Output alphabet Fiovne G-2, An information channel. 6-2. General Symbols source alphabet ‘souree symbol @ —_-Bumber of source symbole S* nth extension of source alphabet ‘e, symbol from nth extension of source alphabet S adjoint of source 8 Pe probability of source symbol s X code alphabet code ymbol 7 umber of eode symbols (also, number of channel input symbols) Xe code word (s sequence of 2's) corresponding to s, 1, umber of eode eymbols in code word X; corresponiling to dy number of eode symbols in eode word corresponding toes L—average length of a code word for S L,—sverage length of a code word for S* A channel input alphabet fa channel input symbol + umber of channel input symbols (also, number of code symbols) [as nth extension of channel input alphabet ‘2. symbol from nth extension of channel input alphabet B_ channel output alphabet channel output symbol 4 umber of channel output symbols Be nth extension of channel output alphabet 8; symbol from nth extension of channel output alphabet m order of a Markov source Ps element of a channel matrix; the probability of receiving by if a, is sent Perror probability of « BSC (p = 1 — p) C channel oapscity Ps error probability ‘M . mumber of messages in a code GLOSSARY OF SYMBOLS AND ENTROPY EXPRESSIONS xv R information rate D_— bamming distance 40) decision rule 6.3. Entopy Exprenions feces eB eceer zy tals) =e as there tral ted when is reeeived after #, has been Feceived (Ersttorder Markov source) ms) = rem phy son the neers 8 ming = 3 Peat) bu pis fonultonal entropy of the firxt-order Markov source 3 1 HS) = prem +08 aray a) Togr He) = oop} + (1 - 0) log AUS) = ‘entropy function (Figure 2-3) 1 say Sree py soy of te sat ik HU) = J Peo M oe pigs endo exoy oA pein entropy) eae stare H(A/B) = yr bos; Ha, B = FP, ee 15 doint entropy of A and B f Pea, 1 B) = H(A) = H(A/B) motu information of A and B 1a; B) = ¥ P0/a) og 74°) conditional mutual informationxvi GLOSSARY OF SYMBOLS AND ENTROPY EXPRESSIONS, H(A, B/0) a PQ, 6 oe mula» J Pane iapgins ake 1A; B/C) = H(A/C) — H(A/B, ©) T(A;B; 0) = 1(A; B) = As 8/0) golie Fa, 57) cequivocation of A and B with respect we quivgaton of A with respect to B ‘mutual information of A and B, given C ‘mutual information of 4, B, and C 1 INTRODUCTION 1.1. What Information Theory Is Not Information theory is a remarkably appealing name to assign to f scientific discipline; when applied to the subject matter of this book, it is also somewhat deceptive. The origins of information theory date back to Claude E. Shannon's publication of a paper in the Bell System Technical Journal in 1948 (Shannon, 1948). Shannon, perhaps realizing the deceptive qualities of the word 1 References indicated in parentheses may be found in the lst of References ‘at the end of the book.2 INFORMATION THEORY AND CODING information, calted his paper A Mathematical Theory of Communica tion. In terms of the colloquia! meaning of information, Shannon's paper deals with the carriers of information—symbols—and not with information itself. It deals with communication and the means of ‘communication rather than that elusive end product of communi- cation—information ‘The distinction we wish to make here is an important one. Start- ing in Chapter 2, we shall derive a number of fundamental properties of symbols used to transmit information. We shall learn that the symbols must obey certain laws if they are to be eapable of trans- maitting information; we shall relate properties of the symbols to the amount of information they can convey. Whether a given symbol under consideration aetually does convey information, how- fever, will in general depend upon factors outside the scope of our theory. For example, “le soleil brille” will convey information to ‘only some of the readers of this book. A common language facili- fates the transmission of information. Slightly less obvious are psychological factors which ean affect information. The sentence "The sun is shining” may conceivably have more than meteoro- logical implications if heard by a psychotic. Semantic factors can cause the same set of words to convey various meanings to various listeners. Shannon (1948) has commented that “the semantic aspects of communication are irrelevant to the engineering prob- Jem.” Weaver (1949) has pointed out, however, that the reverse is not necessarily true—that the engineering (or technical) aspects ‘of communication may be relevant to the semantie, psychological, ‘and linguistie aspects. In Section 2-8 we illustrate the application fof the theory developed in this book to linguistics. Exeept for Section 2-8 and certain notes at the end of each chapter, we do not treat the more specialized applications of information theory to other fields We shall treat the central ideas of the theory of information, with particular stress on information measure and its interpretation. The reader may wish to investigate in greater detail the application of information theory to some other field. The possibilities here fare almost endless, ‘The material discussed in this book ean be related to the information provided by a statistical experiment (Lindley, 1956; Kullback, 1959; Grettenberg, 1962). We shall see that the concept of entropy, central to information theory as treated here, has at least a formal equivalence with the entropy INTRODUCTION 3 of thermodynamics (Brillouin, 1956; Jaynes, 1959). ‘The applica~ tions of information theory to psychology (Quasiler, 1956), art (Pierce, 1961, pp. 250-267), and semantics (Bar-Hillel and Carnap, 1952) have been explored. Finally, the reader is referred to a stimulating treatment of some theological aspects of information theory (Elias, 1958). 1-2. What Information Theory Is The first step in our study of information will be to define a measure of information and to investigate the properties of this measure. The properties we investigate will lend an air of plausi- bility to our measure, and will aid in relating the mathematical theory to the physical model motivating the theory. It is impor- tant to note, however, that the justification for our definition of an information measure cannot be obtained by relationships contained entirely within the framework of the definition, It is clear that wwe may set up a framework of information theory which, taken by itself, is self-consistent and quite reasonable. Yet such a frame- work without further justifcation would constitute merely a mathematical discipline, It is only in the relationship of the frame- work which we erect to quantities entirely apart from this frame- work that the justification for the theory can be found. ‘Thus wwe shall derive a definition of information and a set of relationships for our definition which taken by themsolves are quite reasonable. ‘The definition of information, however, will not be justified by the internal consisteney of the relationships, but by showing how these relationships apply to quantities not involved in the information theory framework. In order to emphasize this necessity for eor- espondence between our mathematical model and the physical world, we shall use this introductory chapter to ask several iimpor- tant questions which may be formulated in a manner completely independent of any particular information measure. In Chapters 2, 3, and 4 we shall see how our definition of information provides Quantitative and mathematically sound answers to these questions 1-3. Encoding Information Let us consider some examples of information transmission in order to introduce the basic ideas of information theory. We4 INFORMATION THEORY AND CODING restrict ourselves at first to consideration of a particularly simple but important type of information—binary information. Exam- ples of binary information are easy to cite. ‘The information stored in punched cards, messages transmitted by on-or-off teletype systems, and the information stored in bistable elements of elee- tronie computers are but a few such examples, Restricting the remainder of this chapter to information of this form will greatly simplify the points we wish to make, It is interesting to note that, contrary to popular belief, the binary representation of information is not relatively new, but has been known for quite a while, In fact, an early reference to the importance of binary information is given in Matthew 5:37: “But let your communication be Yea, yea; Nay, nay: for whatsoever is more than these cometh of evil.” ‘This point of view may be somewhat extreme, and we shall consider the theory of information in terms of both binary and nonbinary information, starting in Chapter 2. ‘A simple example of the representation of nonbinary information in terms of the binary digits 0 and 1 is given in Table 1-1, ‘Tamux 1-1, Divan Convo ov ‘im Dremtat. Diors Decimat Binary agit reprezentation ‘The correspondence of binary sequences to decimal digits given in Table 1-1 is a simple example of a code. (‘The 10 binary sequences of Table 1-} are called cade words, and the 10 decimal digits are called the message symbols, )In Section 3-1 we shall define a code INTRODUCTION 5 + and code words more carefully. For the present, however, we can be somewhat imprecise in our discussion, Clearly we may obtain sequence of binary digits for any sequence of decimal digits (message symbols) by the use of the code of Table 1-1, Conversely, from a sequence of binary digits arising from this code, we may work backward to obtain a unique sequence of decimal di The possibility of working backward from a string of binary code words to the corresponding message symbols is not always straightforward. For example, consider the code defined by Table 1-2. ‘Tanue 1-2. A Breany Coon Meswage Code ‘eunbols wards a ° a o & oo “ ut If wo are given a sequence of code words from this eode, we may not be able to work backward to a unique set of message symbols. ‘The binary sequence 111001 qt) might have arisen from ’ 848 1-2) orfrom ' cae ‘Now the reader may object here and point out that th of a comma (or space) is all that is needed. ‘This is, of eourse, true; the use of a comma (or space), however, eontradiets our assumption of a binary code. If we use a comma to separate code words, we ‘re in effect using three symbols for the code—zero, one, and comma, It is easy to find a eode which doos not suffer from the difficulties of the code of Table 1-2. If wo are given a sequence of code words Of the code of Table 1-3, we may work backward to determine a5 INFORMATION THEORY AND CODING unique set of message symbols, In this chapter we concern our- selves only with such codes, ‘Tamu 1-3. A Bran Cone Menage Code symbols swords a ° * 10 * 110 110 1a? A Problem in Information Transmission ‘To illustrate somo of the ideas of coding and their relationship to the measurement of information, let us consider the following problem. It is desired to sct up a communication system between San Francisco and New York. ‘The system is to transmit the state of the weather in San Francisco at given time intervals. ‘The ystem is required to use only on-off (ie. binary) equipment. To ‘simplify matters, it is desired to classify the state of weather in San Francisco in one of four possible conditions—sunny, cloudy, rainy, ot foggy. These four possible conditions may be considered to comprise the four message symbols as shown in Table 1-4. Also shown in Table 1-4 are the probabilities assumed for each of the States, We assume that the four states are equiprobable. ‘Tams 1-4. Tae Stare or tue Wearwe IN Sax FRANCISCO Menayes Probabilities Sunny 7 Cloudy i Rainy t orev + One possible method of encoding these messages into a sequence of binary symbols is to set up the following correspondence, called. code @ INTRODUCTION 7 Code @ Sunay. 0 Cloudy. oO Rainy 10 a Fossy n Thus, using code @, “sunny, foggy, foggy, cloudy” would be coded as “OO11I101." clear that code @ is an acceptable code for the transmission information in the sense that, given a sequence of eode v0 7 may work backward to reconstruct a unique sequence cof messages which gave rie to that particular sequence of code it is also clear that using code @, it is necessary to send 2 binary digits (binits)+ for each message. Furthermore, the reader may casily satisfy himself that it is not possible to find another accept- able code which allows one to use lose than 2 binits per message. & Now consider a similar problem presented to an engineer in Los Angeles. That is, it is desired to set up a similar binary com- munication system to transmit the state of the weather in Los ‘Angeles to New York. We know that there are important meteor- ‘logical differences between San Francisco and Los Angeles weather. One of these is accounted for by classifying the state of the Los Angeles weather as eunny, cloudy, rainy, or moggy. Now, although / the difference between fog and’ smog is not inconsiderable to a resident of one of these cities, it does not appear to be a factor in } the design of the communications system. Since the four states i are coded into binary sequences, the significanee or meaning of a particular sequence is irrelevant from a communications point of view. it Another meteorological difference can exist, however, which is | definitely not irrelevant to the communications problem. In all fairness to the climate of Los Angeles, we must assign different probabilities to the four possible states. These are shown in Table 1 {From this point on we shall make use of the contraction bint for binary digit. It is important to make a distinction between the binit (binary digit) normation which we shall define in Chapter 2). AS we information under certain circumstances,8 INFORMATION THEORY AND CODING ‘Tanue 1-5. Tux Sturn ov Tue Wearure ry Los Ananuas Mesoages Probie sunny + Cleuly ; Ray, i Smo + If we use code @ to transmit this information, we shall do as well as, but not better than, the communication system transmitting from San Francisco. That is, if we use code @, we send 2 binits per ‘message regardless of the state of the weather. Consider, however, the possibility of using the following code, called code 6, to transmit, Code Sunny Gia ws Using code @ to transmit the message “sunny, smoggy, smogey, cloudy," we would send “1000110.” Again, any binary sequence coming from this code gives rise to a unique sequence of messages. This is true since every binary sequence corresponding to a message ends in a 0 and thus the 0 may be thought of as signaling the end of a code word. ‘The aver- ‘age length L (in binits) of a code word using code ® may be caleu- lated as L = 2Pr (sunny) + 3 Pr (cloudy) + 4 Pr (rainy) + 1 Pr (smoggy) = 2G) +30) +4@) +16) 1f binits/message (1) ‘That ts, for our communication system from Los Angeles to New York we have found a method of transmitting information about the state of the weather, using an average of only 1} bi message, rather than 2 binits per message. The reader may verify that if code @ is used for transmitting the weather in San Francisco (Table 1-1), a value of Z = 24 binits per message will result. We have thus shown that it is possible to transmit the same type of INTRODUCTION 9 information from Los Angeles using about 6 per cent fewer binits per message, on the average. A reduction of 6 per cent in the number of binary digits to be transmitted in a practical eommuni- cation system is a gain of some importance. Furthermore, this gain has been achieved by the simple expedient of using different Inbels for the messages sent. 1-5, Some Quest ‘The example given in the previous section raices several questions of afundamental nature. First of all, achieving a gain of 6 per cent, in s0 simple a fashion has whetted our appetite for further improve- ment. Can we, then, obtain additional improvement of this sort by relabeling the messages in an even cleverer fashion? If further improvement is possible (and in this particular example itis), how far can we go? That is, what is the minimum number of binits per message which we are required to use in order to transmit this information? Once we have established the minimum value of L, the problem of actually constructing a code which achieves tl minimum value arises. What are some practical methods of syn- thesizing such codes? The last of the questions suggested by our simple example is “Why?” What differences are there in the situations described for Los Angeles and San Franciseo which allow us to use fewer binits in order to transmit the state of the weather for Los Angeles? This last question is certainly the most fundamental. In other terms, the question may be restated as “What is the nature of information?” ‘The fact that we need fewer binits for the weather in Los Angeles implies that, in some sense, providing the state of the ‘weather in Los Angoles gives less information than providing the state of the weather in San Francisco. We shall see that this vague notion of amount of information may be made precise by the proper definition of a measure of information. From the example given in Section 1-4, it is clear that the definition of information is con- cerned with the probability of occurrence of the various messages In the next three chapters we shall find answers to these questions by defining an information measure based on the message probabili- ties, That is, we shall obtain the minimum value for the average umber of binits per message which we must use; we shall derive10 INFORMATION THEORY AND CODING methods of constructing codes whieh will allow us to achieve this minimum, and, finally, we shall discuss the nature o itself NOTES Note 1. An easily read mathematician’s view of information theory can be {ound in an expository article by MeMillan (1953). . MeMillan alao presents an amusing description of « mathematician's view of a communication engi reer's view of information theory. Note 2. Some appreciation of the tremendous range of information theory (in the wide sense) may be obtained from an examination of approximately four thousand entries in Stumpers's bibliographies of information theory (1953, 1955, 1957, 1960) PROBLEM 4-1, In Section 1-4 we found two codes, code @ and code @, useful in trans- aitting the state of the weather in Los Angeles. The average length of code ‘@-was 2 binits per message, and the average length of code @ was 1} binits per message. In Chaptor 4 we shall show that the smallest possible average length of » code for the problem of Table 1-5 is 18 binits per message, Furs thermore, we shall find a method of generating such a code Without looking ahead into Chapter 4, try your band at finding « code which ‘achieves this minimum average length. "Remember that sequences of words of ‘your code must represent a unique sequence of messages 2 INFORMATION AND SOURCES 2-1. The Definition of Information In Chapter 1, we formulated soveral fundamental questions about the nature of information. In order to answer these questions, we now define a measure of information, and show that this measure hhas certain properties which we should demand of any reasonable information measure. Note, however, the fact that we are able to show the reasonableness and the internal consistency of our definition does not provide a justification for the definition. We shall justify the definition only by answering the questions posed "12 INFORMATION THEORY AND CODING. in Chapter 1—questions which did not depend upon any particular definition of information—in terms of our definition. Definition. Let E be some event which occurs with prob- ability P(E). If we are told that event # has occurred, then we say we have received i 1() = los pig 2. (B) = los pry ey ‘units of information, ‘The choice of base for the logarithm in the above definition amounts to a choice of the unit we use for information, since 1 Toga If wo-use a logarithm to the base 2, the resulting unit of information is called the bit (a contraction of binary unit). log. = + logs = 2) 1a = aeplyy Wi 30 If we use the natural logarithm, the resulting unit of information is ealled the nat (a contraction of natural unit) 1B) = 5 nats (2-30) If we use a logarithm to the base 10, then the unit of information is the Hartley. It was R. V. Hartley who first suggested the use of a logarithmic measure of information (Hartley, 1928). I(E) = logis Fan Hartleys (2-3e) In general, if we use a logarithm to the base r, ME) = low pegs Pay units (234) From (2-2), we see that 1 Hartley = 3.32 bits (2-40) I nat = 1.44 bits (2-40) INFORMATION AND SOURCES 13 We note, also, thatif P(E) = 3, then 1(E) = 1bit. Thats, one Dit is the amount of information we obtain when one of two possible equally likely alternatives és specified. Such a situation may occur ‘when one flips a coin or examines the output of a binary communica- tion system. ‘To obtain a feel for the amount of information transmitted in a modern communication system, consider a single television picture. Such a pieture may be thought of as an array of black, white, and gray dots, with roughly 500 rows and 600 columns. We shall assume that each of these 500 X 600 = 300,000 dots may take on any one of 10 distinguishable brightness levels, so that there are 10. different possible TV pictures. If each of these pictures is equally likely, the probability of a given picture is 1/108. and the amount of information provided by one such picture ist 1(B) = 300,000 log 10 ~ 10° bits ‘We may compare the information contained in one television pic- ture caleulated above with the information contained in 1,000 words from a radio announcer. Let us assume that the announcer has a voeabulary of 10,000 words and that he selects 1,000 words from this voeabulary in a completely random fashion. (It is possible to find radio announcers for whom this is a reasonable approxima- tion.) Then the probability of any one sequence of 1,000 words is 1/(10,000)'-%, and the amount of information provided is T(E) = 1,000 log 10,000 - = 13 X 10° bits ‘Thus, a (TV) picture is, indeed, worth 1,000 (radio) words, 2.2. The Zero-memory Information Source It will be useful to have available a mathematical description of fan information-generating mechanism, In this section, therefore, +t Henceforth, we shall write the logarithm to the base 2 of z simply as og z, omitting the subscript 2 on the “log.” ‘The natural logarithm of z will be written as In z, All other logarithms will be indieated by displaying the base 468 subsoript (et loge 2)14 INFORMATION THEORY AND CODING wwe define a model of a discrete information source as shown in Figure 2-1. We think of the source as emitting a sequence of symbols from a fixed finite source alphabet S = (81, #, ..., a}. Successive symbols are selected according to some fixed probability law. We shall occasionally refer to the source itself as S, but no confusion will be generated thereby. For the simplest kind of source, we rove 2-1. An information souree. assume that successive symbols emitted from the source are sta~ tistically independent. Such an information source is termed a zero-memory souree and is completely deseribed by the source alphabet S and the probabilities with which the symbols occur: Ps), Ps), « «4 Pla) We may calculate the average information provided by a zero- memory information source as follows: If symbols occurs, we obtain an amount of information equal to 1 1s) = 108 pry ‘The probability that this will happen is just P(s,), so that the average amount of information obtained per symbol from the source is YPEi(s) bits 5 ‘where the J’ indicates a summation over the g symbols of the source S. This quantity, the average amount of information per source symbol, is called the eniropy H(8) of the zero-memory source. HS) * pre log pia its (2-50) {The relation of the entropy of information theory to the entropy of sta- tistical thermodynamies is dissed in Brillouin (1050) INFORMATION AND SOURCES 15 Consider the source S = |x, sso] with Pla) =} and 4 Then H(S) = flog 2 + 4 log 4 +4 log 4 is If 1(«) is measured in r-ary units, then (S) is also measured in rary units, and we have 48),= J Pl)ing phy racy unite SH From Equation (2-2), we see that H(S) HAS) = oe Note that by the definition given in (2-1) we can interpret 1(s,) as the information needed to make the occurrence of % certain, H(S) then may be interpreted either as the average amount of information per symbol provided by the source or as the average amount of uncertainty which the ‘observer has before his inspec- tion of the output of the source. We shall make use of both these interpretations in the material which follows. First, however, we demonstrate some simple pro- erties of the entropy of a source. 2-3. Some Properties of Entropy In order to derive some prop- erties of entropy, we shall have to make use of a property of the logarithm. The natural loga- rithm of z is sketched in Figure 2-2. Also sketched on the same ‘set of axes is the line defined by the equation y = 2 — 1 ‘We may easily verify that the line y = x — 1 always lies above Frovay 2-2. The natural logarithm of sand 2 —16 INFORMATION THEORY AND CODING the curve y = In x. Thus we obtain the inequality InzS2-1 (28) with equality if, and only if, x= 1 ‘Multiplying (2-6) by — 1, we obtain the additional inequality eee en with equality if, and only if, z = 1. nally, we make use of (2-6) to obtain one further inequality. Let x, 2... , 4 and Yr, Yas» - » Y be any two sets of prob- abilities. "That is, x20 yO forallandj and ‘Then, using (2-2), we write $ sett pip Sint and applying the inequality (2-6) to each term in the sum, (2-8a) (2-86) with equality if, and only if, 2. = ye for all i. As pointed out above, one possible interpretation of the entropy of a soureo is the average information per source symbol out of that source. It is quite natural, therefore, to investigate the manner INFORMATION AND SOURCES 17 in which the entropy depends upon the probabilities of the various souree symbols, In particular, we should like to know how much information a zero-memory information source may give us. et a zero-memory information source with q symbols be defined by the source alphabet $ = {a}, i= 1,2, ... , q, and the probs bilities P(e) = Pi =1,2,.. +, ‘Then H(S) is given by ms) = § Poe eo Consider the quantity tog¢ - (8) = 3 Prog - . Prog p, = } Prlog oP = loge s Pyln Ps (210) ‘The last line is obtained using (2-2). Now, we apply the inequal- ity given in (2-7) to (2-10) and obtain 20 (241) ‘Thus, H(S) is always less than, or equal to, log. Furthermore, the inequality was introdueed at one point only, so that from the equality condition of (2-7) we see we shall have strict equality if, and only if, P; = 1/q for all ¢, ‘That is, we have shown that for a zero-memory information source with a g-symbol source alphabet, the mazimum vatue of the entropy is exactly log 9, and this maximum value is achieved if, and only if, all the source symbols are equiprobable. One particularly important example of a zero-memory informa- tion source is the zero-memory binary source. For such a source, the source alphabet $-is just {0, 1]. ‘The probability of a 0 is18° INFORMATION THEORY AND CODING. and the probability of a 1 is 1 — a, We define & «. From (2-5) we may calculate the entropy of such a source, 1 (8) = vlog! + cto} vin a ‘The function of w given in (2-12) occurs frequently in information- theory problems. Accordingly, it is customary to assign a special symbol to this funetion, We define H() = wtog + stg) e and we call H () the entropy function. Note the distinction between (2-12) and (2-13). H(S) gives the entropy of a particular source S, while H(e) is a function of the variable @ defined on (0, 1]. The ‘meaning of the symbol H{() is conveyed by its argument.’ Another point to note is that lim «loge = 0 a“ and so we define O log 0 = 0 We have plotted H(s) as a function of w for w in (0, 1] in Figure 23, Note that if the output of this binary source is certain (either © =0 or w= 1), then the source provides no information. ‘The maximum average amount of information provided by each symbol of a binary source is log 2, or 1 bit; this oceurs if, and only if, the O and 1 are equally likel ‘The outputs of a binary source are binary digits or binits. Thus, ‘we see that a sequence of binits from a zero-memory binary informa tion source with equiprobable Os and Is will provide 1 bit of informa- tion per binit. If the Os and 1s are not equally probable, then the amount of information provided by a given binit will be either less than or greater than 1 bit, depending upon its probability [ef. (2-1)]. ‘The average amount of information provided by a binit from such a binary source, however, will always be less than or equal to 1 bit per binit (Figure 2-3), It should be noted that the maximum amount of information Provided by a zero-memory information source with @ symbols Sa ty os * INFORMATION AND SOURCES 19 increases slowly as q is increased. In fact, the maximum amount of information provided by such a souree increases only as the loga- rithm of the number of source symbols, and in order to double the ‘maximum amount of information per symbol provided by a souree with q symbols, we must use a source with q? symbols 19 10 os] las aa} los | oz os| los $5 05 oa oa 03] 3 04] a2 ol oa ar az 03s 0s 807 Gao 10 Froune 2-3. H(a)—the entropy funetion. 2-4, Extensions of a Zero-memory Source In discussing the properties of information sources and informs- tion channels, in subsequent chapters we shall often find it useful to deal with blocks of symbols rather than individual symbols. For example, in the ease of the binary source treated in the previous section we may think of the binits from the source as being emitted in groups of two. The binary source considered in this manner20" INFORMATION THEORY AND CODING is clearly equivalent to a source with four possible symbols—00, 01, 10, and 11. It is possible to extend this idea, If we think of our original binary source as emitting binits in groups of three, then, since there are eight possible binary sequences of length 3, the binary source considered in this manner is equivalent to a source with @ source alphabet of eight symbols. In general, if we have a zero-memory source 8 with souree alpha- bot {21 «.. » fol we may consider the outputs of S taken n ata time. There are g* such sequences of outputs. We formalize this natural notion in the following definition. Definition. Let S be a zero-memory information source with source alphabet {s,, a, ..-, | and with the probability of s: equal to P,. Then the nth extension of S, S*, is a zero-memory source with g* symbols oy, ¢n.. +5 oe}. Each « corresponds to some sequence of nof thes. P(;), the probability of e, is just ‘the probability of the corresponding sequence of 4/3. That is, if o, corresponds to (6, sy +++» $s then Plo) = PaPa = > Pi Since one symbol from S*, the nth extension of the zero-memory source S, corresponds to n symbols from S, we expect the entropy per symbol of S* to be n times the entropy per symbol of S. This is not difficult to prove. Let ¢, be the symbol of S* corresponding to the sequence (8, s, - .- »%) of S. Then ($= J P(e) log pt eu) & ai) where the summation is over the q* symbols of S*, When dealing with a source and its extensions, we shall use the notation Y) to z ite a summation over all the symbols of the nth extension Such a summation over the q* symbols of S* is equivalent tom summations, each one over the ¢ symbols of S. ‘This is s0 since 2 = (Sy 8) » » » »8%)» For example, since P(o,) = PaPa-* > Pi, for a zero-memory source, + Note that, by our definition, the frat extension ofa source S is the souree tect, INFORMATION AND SOURCES VPC) =F Phas + Pa ~$ Efren efasre- 3 =1 ‘Equation (2-14) may be written as 1 HS) = pre 16 pp, P, = [Peon 7, + J Pte0 lon Fe ag +++ + SP) log z 16) If we take just the first term of the sum of summations given sbove, we see that we may perform n — I of the indicated summa- tions, Pro oe 7, = PPP, + Plog Sri Sn =} Pitog z = SP. tor p =) 17) and using (2-17) to evaluate the other terms of (2-16), we obtai the desired result: H(S*) = n1(8) (2.18) Bxample 2-2. We take the second extension of tho source of Example 2-1 Recall that this source had an alphabet S = [s,s] with P(e) = § and PQs) = P(os) =f Then 8" has nine symbols, as follows:22 INFORMATION THEORY AND CODING Symbols of Probability Pte)... 1 = Y re wm Elog 4 + 4X blog 8 +4 x ve log 16 rh £5. The Markov Information Source ‘The zero-memory source considered so far is too restrictive for some applications. A more general type of information source with 4 symbols than the zero-memory source already considered is one in which the occurrence of a source symbol may depend upon a finite number m of preceding symbols, Such a source (called an mth-order Markov source) is specified by giving the source alphabet 'S and the set of conditional probabilities} POR Bin 58) for i= 1,2, , 34p = 152, For an mth-order Markov source, the probability of emitting a given symbol is known if we know the m preceding symbols. At any one time, therefore, we shall call the m preceding symbols the slate of the mth-order Markov source at that time, Since there ae q possible symbols, an mth-order Markov source will have q* wsible states. “As symbols are emitted from the source, the state changes. A handy way to illustrate the behavior of a Markov source is through the use of a sae diagram. In a state diagram we represent each of the g* possible states of the source by a single Point, and the possible transitions from state to state by arrows. Bxample 2-8. Consider a second-order Markov source with the binary source alphabet $= (0,1). We assume the conditional symbol probabilities P0/00) = PAY) = 08 PA/0) = PO/L) = 0.2 PCO/) = PlO/0) = PU/o1) = PA/so) = 05 Because is equal to 2 and we have assumed a second-order Markov source, we have four states of the source—O0, 01, 10, 11. The state diagram for this 1 Tho time sequence of symbols implied by the conditional probability PSB Bn = + 8m) 18 By Ai» ‘That is, «follow INFORMATION AND SOURCES 23 source is shown in Figure 2-4. The possible state by the four dots. ‘The possible state transitions are indieated by arrows state to state, with the probsbility of a transition shown by a tumber associ ated with exch arrow. For example, if we are in state 00 we can goto either 1 or 00 but not to state 10 or 11.. The probability of remaining in state 00 is shown as 0.8, and the probability of going to state Of In dealing with mth-order Markov information sources we shall restrict ourselves to the consideration of what are called ergodic sources, For the mathematician and for the mathematical statis tician, the concept of ergodicity and the conditions under w source is ergodic are somewhat involved. For our purposes, how- ever, the concept of an ergodic souree is simplicity itself. An 03 1.0 Fiovar 2-4, State diagram of a sec- Fravar 2. State diagram of a non- ‘ond-order Markov source, ergodic second-order Markov source. ergodic source is merely a source which, if observed for a very long time, will (with probability 1) emit a sequence of source symbols which is “typical.” In fact, the idea of sources possessing the ergodic property is so natural that some readers may have difficulty in picturing a source which is not ergodic. We therefore give an example of a nonergodic information souree. Example 2-4. Consider n second-order Markor source with the binary source alphabet $ = (0, 1}. We assume the conditional symbol probabilities P(0/00) = Patt) = 1.0 PQ/O) = PO/1) = 0 PO/O) = PO/10) = PA/o) = PA/o) = 05 As in the previous example, wo have four etates—00, 01, 10, 11. The state ‘dingram for this source is shown in Figure 2-5. Note that, for this source, if24 INFORMATION THEORY AND CODING we ever arrive at cither state 00 or £1, we stay in that state forever. Further= moro, let us select one of the four possible states at random (i. exch state has ‘a probability of } of boing selected), Then, if we start in the state selected, ber of state transitions have taken place, we ity 0.5. ‘That is, ater «large number of sym i souree, the source will emit a 0 with probabi ity 0.5 and a 1 with probability 0.5. In any gieen sequence out of the source, however, if we waits long time, we will almost surely see either all zeros or all ‘ones. In other words (with probability 1) we will not see a typieal sequence {rom this souroe; i is not ergodic, The discussion given above points out the difficulty, from our point of view, with nonergodie sources. If we select the initial state of a Markov source (according to some set of initial probabili- ties over the states) and wait a large number of state transitions, we know that there will be some definite probability with which cach of the states will occur. Furthermore, as suggested by our use of the word (ypical in the previous paragraph, for an ergodic source the states which actually do appear in a very long sequence will (with probability 1) occur with these same probabilities. It is 1 further remarkable property of ergodic Markov sources (Feller, 1950) that the probability distribution over the set of states which appears after many state transitions (or, equivalently, the distribu- tion of states in a typical output sequence) does not depend upon the initial distribution with which the states are chosen. ‘There is a unique probability distribution over the set of states of an ‘ergodic Markov source, and the states in any long output sequence will (with probability 1) oceur with this distribution. ‘This unique probability distribution is called the sfationary distribution of the ergodic Markov process; since the stationary distribution docs not depend upon the initial distribution with which the slates are chosen, H can be ealeulated from the conditional aymbol probabilitics alone. For example, Tor the Markov source given in Figure 2-1 it is pos- sible to show that the stationary distribution is POO) = PUY) = PO) = P(10) = Fe GaN ‘When we specify the conditional symbol probabilities P(si/sj, $5, «+ ; 4) of an ergodic mth-order Markov process, we also impli itly specify the q* state probabilities P(e, 6... . , 8.) Upon combining these two probabilities, we obtain the probability of the INFORMATION AND SOURCES 25, joint event “the source is in the state specified by (&4, iy « fand s; occurs.” ‘This probability is just POG Bin = = Bia 8) = POR By Bip «+ + 5 Bin) PCBin Bin + = = 9 Bin) (221) Tt should be noted that we have not discussed the problem of actu- ally calculating the state probabilities of an ergodic Markov source, given the conditional source probabilities. ‘This is, in general, & difficult task, and the reader is referred to Feller (1950) or Bharucha- Reid (1960) fordetails. All we make use of is the fact that the state probabilities may be calculated from a knowledge of the conditional ‘symbol probabilities. We may calculate the average information provided by an mth-order Markov sourcet asfollows: If we are in thestatespecified iy iy Sin» « « » 8) (Le, the m previous symbols emitted were +5 4), then the conditional probability of receiving Pattee Phage ny, «. . o.). Fram (rl) the lfermation'ee obtain if s, occurs while we at woe th) is Hel tig Big os ‘Therefore, the average amount of information per symbol while tee are in slate (24, ty «+ » #,) is given in Equation (2-23): H(S/ 85 5 « «+ 5 a) = PP Big = 2 + 7 Bia) By oo 4 Big) (2-23) If we average this quantity over the g* possible states, we obtain the average amount of information, or the entropy of the mth-order Markov source HES) = Y Pt ty» « 4 WHC /t4y tn © « 42) (2-24) In writing (2-240), we have used the fact that the state (5:5 Si» +++ 5 84) is equivalent to a symbol from S=, Using (2-23) in Henceforth we shall omit the word ergodic when discussing such sources. ‘The remainder of this book will treat only the ergodie ease. ‘Strictly speaking, we have not yet defined S$», the mth extension of a Markee source 8. The use of $ in (2-24), however, is unambiguous. The ‘complete definition of an extension of a Markov source is given in Section 27.26° INFORMATION THEORY AND CODING (2-24a), we obtain HOS) = J Plt tin = 1 8a) DPC B= - 8) X18 Paracay «J Py Big - «1 BA)PCOLS%0 Bin + 5 Bu) X log pe = . Ply Bie 6 + 5 Bs 8) legge X08 Pa, where (2-21) was used in the last step. Note that if $ is zero-memory rather than Markov, PO By By - «1 Bi) = PCO) and (2-24b) reduces to (2-5a) Example 2-6. We consider the Markov source of Figure 2-4. The station sry distribution of this source is given in (2-20), The relevant probabilities are summarized in Table 2-1, ‘Tamun 2-1. Pronanraries von sux Manxor Sounce oF Fravee 2-4 tte — Pilea) Plann) Plan ana) see EES nerd ne 000 8 01 02 010 05 on 05 100 101 10 a he le he Se She he ho oe ‘Then the entropy is ca 11) = J Popa 0p wl, ey o 2 reloe gg +2 aeloe gh +41 ogg = 081 bitin INFORMATION AND SOURCES 27 The Adjoint Source Hf we are given an mth-order Markov souree, we may, in principle, calculate the stationary distribution the distribution of states of the Markov source. For a firstorder Markov souree, the set of states is identical with the set of source symbols, and the stationary distribution gives us the first-order (unconditional) probability istribution of source syznbols directly. Fora higher-order Markov source, the first-order probability distribution of source symbols ‘may be obtained from the stationary distribution with little addi- ional effort. The Markov source with stationary distribution given in (2-20), for example, may be shown to have the first-order symbol probabilities P(0) = P(1) = 1. Using the firt-order symbol probabilities of a Markov source, we may define another souree. Definition. Let S = {si 8)... , %) be the source alphabet of an mth-order Markov souree, and let. Py, Pa, . ..., Py be the first-order symbol probabilities of the source. The adjoint source to S, written 8, is the zcro-memory information source with source alphabet identical with that of S, and with symbol probabilities Py Ps... Pe For example, by the symmetry of the state diagram shown in Figure 2-4, 0 and 1 are equally probable. Thus, the adjoint of the source of Figure 2-4 is a zero-memory binary souree with equi- probable input symbols and H(S) Note that the adjoint of a ‘zero-memory source S is just S itself, We shall show that the entropy of the adjoint source S is never less than the entropy of &. ‘This fact has an important interpretation. ‘The two sources $ and Shave identical first-order symbol probabilities. They differ only in the fact that S has additional constraints expressed in terms of conditional symbol probabilities imposed on its output sequences. These constraints, therefore, serve to decrease the average amount of information flowing out of the source In order to simplify the notation, we shall prove that H(S) is greater than or equal to H(S) when S is a first-order Markov source. ‘The corresponding proof for the mth-order Markov source is Aireot extonsion. Let 8 be a first-order Markov source with symbols 6, 8, - « «28 INFORMATION THEORY AND CODING and conditional symbol probabilities P(s/s)), i, = 1,2)... , Let Ps, Ps, . . ., Pe be the first-order symbol probabilities of S, and let S be the adjoint source. If we define P(e, s) as the joint probability that the souree is in the state specified by s, and 4 oceurs, we may write [corresponding to (2-21)] Plas, 8) = Plsi/a)P; (2-25) ‘Now we examine the double summation 3 Pe, 8) tos pe (2-26) 5 From (2-Sa) we see that this summation is less than or equal to 0, with equality if and only if P(s, 8) = PsP; for all ¢andj 27) ron combining (226) and (2-26) and writing the inequality, we J Pe 9) oe iy <0 1 1 2 P(e, 8) log peazay S 2 Ploy 8) log p <3 Sreaoat 29 _ The summation over the index j of the double summation on the right may be performed immediately by noting that the logarithm is independent of j and that, 3 Pla, 0) = Pe 20 that we get J P6199 toe pavay $ J Pelee; H(S) < H(S) INFORMATION AND SOURCES 29 ‘The condition for equality [expressed in (2-27)] is simply that & and 5; are statistically independent, ic., that S is really a zero- memory source. We have already seen one example of the relation expressed in (2-29). Recall that, for the Markov source of Figure 24, H(S) = O81 bit while H(S) = 1 bit. 2.7. Extensions of a Markov Source In Section 2-4 we defined an extension of a zero-memory source. An extension of a Markov source may be defined in a similar manner by considering blocks of n symbols from the source as comprising a new symbol 6; Definition, Let S be an mth-order Markov information source with source alphabet (8, 8s, . . . , 8) and condi tional symbol probabilities P(6,/s;, 8 - - » 54.) ‘Then the nth extension of S, S*, is a uth-order Markov souree with 9* symbols, #1, ¢2, .- . ,¢e- Each a; corresponds to some sequence of n of the #,, and the conditional sym- bol probabilities of a are P(oi/as, 6iy - . » + 0},). The definitions of these probabilities and of 4 are given directly below. In order to describe completely the statistics of the nth extension of an mth-order Markov source, we must specify the conditional probabilities PCG%/ 85 Bin» + + B50) (2-30) where «; represents a symbol of the nth extension—a sequence of n ymbols 5. ‘The sequence (6 Si» « . « » 8.) is equivalent to some sequence of 64, S8Y oy 05 , & Where = [m/n], the smallest integer greater than or equal to m/n. ‘The conditional symbol probabilities of (2-30), therefore, ean be written as PL G5 Fi «+ + 5 Gin) (2-31) For example, the third extension of a fifth-order Markov source with ¢ symbols would be a second-order Markov souree with symbols, Note that by taking atleast m extensions of an mth-order Markov source we can always obtain a first-order Markov source. Finally, we produce an expression for P(oi/eiy ¢iy +» - 4) in terms30 INFORMATION THEORY AND CODING of the conditional symbol probabilities of the original source Let o¢ = (Si iy = 8). Then Peay Oy» «+ 84) = Ply By = =» 4 i/ Sin Bip PO /Biy Bap + + 5 Se) PCS/ tay Bn Pl) tina ‘The form of the last term in the product assumes that n> m, If nS m, the last term would be P(6is/shy Sian = + + 9 Bas ‘The entropy of the nth extension of a’ zero-memory source has bbeen shown to be n times the entropy of the original source. It is easy to show that this property also holds for Markov sourees. Again we carry out the proof for the case of a first-order Markov souree. Tet S be a first-order Markov source with source alphabet {81 5, . . . , sel, transition probabilities P(s./s,), and stationary distribution P, +, Py Let S* be the nth extension of 8 with symbols o, i= 1,2, ..., q*. S* is a firetorder Markov source (by our definition of x). HOS) = YY Po, 0) log 5 Fea (2-83a) The right side of (2-334), when viewed in terms of the source S* is a double summation with both i and j running from 1 to 4 Alternatively, we may think of these summations in terms of the 2nth extension of the original source S. In this case, HS») = YPC, 2) log rea (2-330) Rewriting (2-32) for the case m = 1, we find Plo/a)) = PCy 8y P(si,/8i.4) (2-34) and the right side of (2-336) may be separated into n summations: 1 H(S*) = pre #108 BG, 75) + 1 + ZF Mos rer) 238) INFORMATION AND SOURCES 31 Fach of these summations may be reduced. For example, by per- forming 2n — 2 of the indicated summations on the first term, 1 , 1 7a) ~ Pe 8) 16 Be 55 = HS) (2-36) J Peo odo 7 Hence H(S*) = nH(S) 37) Further interesting properties of the entropy of a Markov source may be obtained by considering S, the adjoint of S*. Let P(e), PG@2), . . . , P(oq) be the first-order symbol probabilities of the e, the symbols of the nth extension of the first-order Markov source considered above. Since g; corresponds to the sequence (8iy «%,), we sce that P(e.) may also be thought of as the nth-order joint probability of the s, H®) = J Pe) 08 5, & = prem aoe Bin a frstonder Markov source, bower PG 145) = Pl)PC/86) “Cult (289) Putting (29) into (2-38), we obtain BO) ~ Drone. -.n0 [eels +b! to blo ae 5] = HS) + wD) 20) H(S) = nHl(S) + (H(S) — H(S)] (2-41) Note that ‘the term in the brackets on the right side of (2-41) is nonnegative constant, independent of n. If $ is an mth-order (rather than just a first-order) Markoy source, then (2-41) must be Teplaced by HG) = nS) + (242)32 INFORMATION THEORY AND CODING where ¢a is a positive constant which (as long as n > m) depends only upon the statistics of S (Problem 2-1) If we divide both sides of (242) by m, we find HS) ny + 243) ‘We have previously seen (2-29) that H(S*) > H(S*) = n(S) (244) Equation (2-43), however, tells us that this inequality becomes Jess important as n becomes larger. More precisely, from (2-43), we have 1m 209 In other words, for large n, the Markov constraints on the symbols. from S* become less and less important. A word of caution may be appropriate at this point. The adjoint of the nth extension of S is not the same as the nth extension of the adjoint of S. H(S) (2-45) HS) # HS) (2-46) In fact, since § is a zero-memory source, H(S*) = nHl(S) (247) which may be contrasted with (2-44). Example 2-8. We may collect some results given in previous examples dealing with the source of Figure 2- to illustrate the preceding equations. We Ihave obtained H(S) = 081 bit HS) ~ 1.00 bit From (237), H(S) = 2H(S) = 1.62 bits ‘Wo can ealeulate BB) = YP) WE PGs & = 1.86 bits ‘Even more tedious ealeulations yiold HG) = 2.66 dite HGS!) = 3.47 bits INFORMATION AND SOURCES 33 HS) = 1.00 bit approaches M(S). 2-8. The Structure of Language In the previous sections of this chapter, we have defined a model of an information source, and brought out. some simple properties ‘of our model. It is of some interest. to investigate how closely such a model approximates the physical process of information generation. A particularly important ease of information genera- tion is the genoration of a message composed of English words. In this section we show how the generation of such a message may ‘be approximated by a sequence of successively more and more com- plicated information sources. ‘Let us restrict ourselves to a set of 27 symbols—the 26 English letters and a space. The simplest possible source using this alpha- bet is a zcro-memory source with all 27 symbols equally probable. ‘The entropy of this source is easily calculated. H(S) = log 27 = 4.75 bits/symbol (248) A typical sequence of symbols emitted by such a source is shown in Figure 2-6. We refer to this sequence as the zeroth approxima- tion to English. ZEWRTZYNSADXESYIRQY_WGECLJ_OBVERBQPOZD ‘YMBUAWVLBTQCNIKPMP__KMVUUGBSAXHLHSIE_M. Fiovne 2-0. Zeroth approximation to English, Note that there is no perceptible structure to this sequence and it is not possible to identify it as coming from any particular Tanguage using the same alphabet. A better approximation to English may be obtained by employing the actual probabilities 504434 INFORMATION THEORY AND CODING of the symbols used (Table 2-2). ‘The entropy of @ zoro-memory source with probabilities given in Table 2-2 is. H(S) = ¥ Potox 5 = 4.03 bite/symbol 49) ‘Tanux 2-2. Pnomanrusvies or Srunois 1 Exouin (Reza, 1961) Symbol Probabitity Symbol Probability 8 0.1859 0.0682 0.0127 0.0218 0.0317 0.1031 0.0208 0.0152 0.0467 0.0575 0.0008 00039 0.0821 0.0198 0.057% 0.0632 0.0152 0.0008 0.0488 0.0514 0.0706 0.0228 0.0083 0.0175 0.0013 0.0168 0.0005 N4KadcHoRovow A typical sequence of symbols emitted by this source is given in Figure 2-7. AL_NGAE___1TP_NNR_ASAEV_OIE_BAINTHA_HYR (00_POER_SETRYGAIETRWCO__EHDUARU_EU_C_F ‘T_NSREM_DIY_RESE__F_O_SRIS_R___UNNASHOR Fiovne 2-7. First approximation to English, ‘Though the sequence shown here cannot qualify as good English, it does exhibit some of the structure of the language. (Compare this with our zeroth approximation.) ‘The “words” of this approxi- mation are, for the most part, of reasonable length, and the pro- portion of vowels to consonants seems more realistic. We may improve upon the souree which generated the first approximation by using a first-order Markov source with the appropriate condi- INFORMATION AND SOURCES 35, tional symbol probabilities. ‘These probabilities are given by Pratt (1942). HS) = ¥ P65 08 pay 3.32 bits symbol (250) Itis possible to generate a typical sequence of symbols out of such a first-order Markov source by using the probabilities given in Prat. Shannon, however, has pointed out & much cleverer method. —Ordi- nary English text has the probabilities we seek contained in it. We therefore open a book and select a letter at random, say U. Next, we skip a few lines, read until we come to the first U and select the first letter after the U—in this case it was R. Again we skip a few lines, read until we come to an R, and select the next letter. Using this procedure, we construeted a second approxima- tion to English (Figure 2-8). URTESHETHING_AD_B_AT_FOULE_ITHALIORT_W ACT_D_STE_MINTSAN_OLINS_TWID_OULY_TE_T HIGHE_CO__YS_TH_HR_UPAVIDE_PAD_CTAVEI Fiavne 2-8, Second approximation to English ‘Note how the English flavor of the sequence makes itself felt in the second approximation. We would have little trouble in identifying this sequence as an approximation to English rather ‘than, say, French. ‘We can apply Shannon’s method to construct even better approxi ‘mations to English. By selecting letters from a book according to ‘the two preceding letters we ean construct a typical sequence from a second-order Markov source approximating English. IANKS_CAN_OU_ANG_RLER_THATTED_OF_T0_8 HOR_OF_TO_HAVEMEM_A_I_MAND_AND_BUT_. WHISSITABLY_THERVEREER FIGHTS TAKILLIS_TA Fiaunx 29, Thied approximation to English Shannon (1951) has estimated the entropy of the source corre sponding to the sequence shown in Figure 2-0 as 3.1 bits per symbol Using other methods, he has estimated that the entropy of English,36 INFORMATION THEORY AND CODING taking into account all the preceding text, is between 0.6 and 1.3 bits per symbol. Tt is possible to extend the process used above to generate typical sequences out of mth-order (m > 3) Markov sources hav- ing the symbol probabilities of English. The construction of such sequences, however, becomes impractical for m greater than 2. Instead Shannon jumped to a zero-memory information source with the English words rather than the English letters as the output symbols. The probabilities of oecurrenee of the various words are approximately the same as in English text. Shannon (1948) obtained the approximation shown in Figure 2-10, REPRESENTING AND SPEEDILY I8 AN GOOD APT CAN DIFFERENT NATURAL HERE HE, N CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MES- SAGE HAD BE Ficumx 2-10. Fourth approximation to English, An even more complicated approximation to English ean be constructed by letting the probability that a given word is selected depend upon one preceding word. ‘The souree corresponding to this approximation is a first-order Markov source with the Eng words as symbols. Shannon (1948) also constructed a typical sequence from this type of source (Figure 2-11). THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEX- PECTED Fioune 2.11, Fifth approximation to English Its interesting to note that this sequence is a reasonable approxi- mation to the sort of output one might expect from a very excited ‘and quite incoherent speaker. The faet that we can approximate (at least to some extent) as complicated an information source as an English speaker by the simple models of zero-memory and Markov sources is encouraging. Many INFORMATION AND SOURCES 37 in realistic communication problems are of a much simpler nature, ‘and we may expeet that our models provide better approximations to reality in those cases. A striking illustration of the differences in several Western Janguages may be obtained by constructing sequences using the statistics ofthese languages. We have done this or three languages, and the sequences obtained are given in Figures 2-12 0 2-14, As before, the first approximation corresponds to a sequence out of a zero-memory souree; the second approximation from a first-order ‘Markov source; and the third approximation from a second-order Markov souree. R_EPTTFVSIEOISETE_TTLGNSSSNLN_UNST_FSNST F_E_IONIOILECMPADINMEC_TCEREPTTFLLUMGLR ADBIUVDCMSFUAISRPMLGAVEAI_MILLUO (@) First approximation to French ITEPONT_JENE_IESEMANT_PAVEZ_L_BO_S_PAS E_1QU_SUIN_DOTI_CIs_NC_MOUROUNENT—FUI ‘T_JE_DABREZ_DAUIETOUNT_LAGAUVRSOUT_MY (©) Second approximation to French JOU_MOUPLAS_DE_MONNERNAISSAINS_DEME_U §_VREH_BRE_TU_DE_TOUCHEUR_DIMMERE_—LL ‘ES_MAR_ELAME_RE_A_VER_IL_DOUVENTS_50 (©) Third epproximation to French Fiovnr 2-12, A series of approximations to French, NNBNNDOETTNIMAD_TSI_ISLEENS_LRI_LDRRBNF- REMTDEETKE_U_HB¥_EVSN_BRGANWN_IENEEHM EN_RHN_LHD_SRG_EITAW_BESRNNCLG (@) First approximation to German AFERORERGERAUSCHTER_DEHABAR_ADENDERG. E_E_UBRNDANAGR_ETU_ZUBERKLIN_DIMASO N_DEU_UNGER_EIEIEMMLILCHER_WELT_WIERK @) Second approximation to German BET_EREINER_SOMMEIT_SINACH_GAN_TURHATT ER_AUM_WIE_BEST_ALLIENDER—TAUSSICHELLE, —LAUFURCHT_ER_BLEINDESEIT_UBER_KONN— (© Third approximstion to German Fiovns 2-18. A series of approximations to German.INFORMATION THEORY AND CODING VOALNAO_NEL_D_NIS_ETR_TEGATUEORC_S_ASU DU_ZELNNTSSCASOSED—T_I_R_EIS_TAMMO_TIL UOEDEO_UEI_EOSEELA_NMSLAANTE( (a) First approximation to Spanish CINDEUNECO_PE_CAL_PROS_E_LAS_LABITEIAS ‘TE_ONTOMECITRODRESIO_PAY EN SPUSEL_LA <_UTAJARETES_OLONDAMIVE_FSA_8_CLUS_ (&) Second approximation to Spanish RAMA_DE_LLA_EL_GUIA_IMO_SUS_CONDIAS_S 7 UERBALL ;_SE_SUS_SUPAROCEDA (6) Third approximation to Spanish Fiauxs 2-14, A series of approximations to Spanish. Asa final example in this series, we present a series of approxi tions (Figure 2-15) to another Western language and leave it to the reader to determine which language. SETIOSTT_NINN_TUFENMIUTIAUE_N_IREATSRI_M MATPSAC_SES_LN_ANEUSUNTINU —AR_TM_UMOECNU_RIREIAL_AEFIITP (@) First approximation to 7 (CT_QU_VENINLUM_UA_QUREO_ABIT_SAT_FIUMA GE_ICAM_MESTAM_M_QUM_CUTAT_PAM_NOND QUM_0_M_FIT_NISERIST_B_1._ONO_THOSEROCO () Second approximation to ? RCUM_SITECI_LIBEMUS_ACERELEN_TE MINUS_UTER! INQUENEQUE_IRA (© Third approximation to 7 Fiauns 2-15. A series of approximations to ? NOTES Note 1. The word entropy was coined by Clausius in 1876 from the Greek words tr and rrirue. Together these words imply a “turning in.” Clausins ised entropy to denote that part of the energy of a system which cannot be transforme:! into mechanical work without the transmission of heat to some ‘other body, or a change in volume. "Boltzmann, in 1896, was the first to show that the entropy of a system could be expressed in terms of the average value INFORMATION AND SOURCES 39 ‘of the logarithm of the probabilities of the states of the system. Shannon (2948) introduced the word into information theory. Entropy is certeinly the most important but not the only point of contact between information theory and statistical mechanies. Jaynes (1980) discussed the use of the portion function of statistical mechanics in information theory. ole 2, Throughout this book we assume that the symbel probabilities of the sources we treat are known. When these probabilities are unknown (and perhaps even the number of source symbols is unknown), we may wish to esti imate the entropy of « zero-memory source from an observation of outputs of that source. Miller and Madow (1054) find the maximum-ikelihood estimate of the entropy based on such an observation. sq httharin (1960 wee the experimental probate to form che natural imate 19) = poe He shows that (8) isa biased, consistent, asymptotically normal estimate of aon asymptotically timate of 1 sms ~ HU) ~ 15" ee +0) where q is the number of symbols of the source alphabet and Nis the size of the observation, Blyth (1958) investigates certain other est H(S) and, in addition, shows that there exists no unbiased estimate of H(S), Note 3." In Section 2-2 we define the entropy of w variable which ‘ean assume one of a finite number of values, (The distribution function of the random variable consists of a finite number of steps.) In order to treat a random variable » which ean sssume values in some continuum (eg, its distribution function is absolutely continuous), we might consider a sequence of finitestep distribution functions approximating the absolutely continuous distribution function. Let [s] denote the integer part of e. Define the new random variable nt sate ran te{s.=3} Jette saree ore t th random val be SA n nce, tke random varie tonite and wor Unfertoaay HS) dn ‘not remain finite! ‘enyi (1959) defined the dimension of the random variable ¢ a8 (a) = tim HS.) Tog tnd the d-dimensional entropy of # 8 HAS) =40 INFORMATION THEORY AND CODING ‘when these quantities exist. When the distribution function consists of « finite ‘number of steps, the dimension is clearly zero and Hy(S) reduces to H(S). enyi (1959) shows that, when the distribution funetion is absolutely continu ‘ous and HS) is finite, dl) = 1, and (letting ple) be the density function) a8) = [7 reo on sh an 7) if this integral exists. Csiszar (1961) provides « partial converse to these results fle 4. In addition to generating words of a langasge from an artifical souree as illustrated in Section 2-8, itis possible to generate musical eompo- sitions. Pinkerton (1056) has used this method of musical composition, -Pieree (Q961) devotes several pages to the generation of such musi; perhaps the ulti- ‘mate in artistic information theory is displayed in some passages from the “Tlliae Suite for String Quartet” roproduced in Pierce (1967, p. 260). Note 5. As we have mentioned, the use of the entropy of a source as a ‘measure ofthe amount of information provided by that source will be justified by Shannon's first theorem (Chapter 4). It is slso possible (Feinstein, 1958) to provide a justfieation based on the fact that entropy ie the only function of the symbel probabilities ofa souree satisfying certain properties. Define three z2ero-memory sources and their symbol probabilities as follows (@ = 1 = a): Si. & ae = Entropy is the unique function (except for 8 multiplieative constant) of the souree symbol probabilities satis ying (a) HS) = WIS) + PHS. ©) H(S,) in continuons fenetion of a (©) H(S) isa symmetric function of Py, Py, PROBLEMS: 2-1. Prove Equation (2-42). 2-2, The state diagram of « rstonler Markov source with source alphabet 'S = (0, 1,2] isshown in Figure P22. We define p = 1 ~ p. By symmetry, ‘we see that the stationary distribution is P(0) = P(l) = P@) = Fravm P22 (a) Find 108), (@) Find H(S). Does your answer check for p = 0, p= 1? 4(0) For which value of pis H(S) a maximum? {(d) Investigate the behavior of H(S) for p = « where « = 0. (@) Investigate the behavior of H(S) for p = 1 — 3, where 8 = 0. (f) Draw the state diagram for 5? () Find H(S*), HG. 2-8. Two zero-memory sources $,and Sshave q:and g:symbols, respectively. ‘ay¢ symbols of S, oceur with probebilities P,,é = 1,2, , av; the aymbols of S, occur with probabilities Qy, f= 1, 2, ‘and the entropies of Si fand 8; are Hand He, respectively. A new 2ero-memory source S(), called» Imizture of 8, and 8, is formed with 9) + gs 4ymbols. ‘The first qs aymbols of SO) have probabilities MP, i= 1, 2. + y gu and the last gy symbols of 'S(\) have probabilities XQ, = 1,2... ,¢n (= 1-2) (@ Show that ISO) = Alls + Kits + HO) fand provide an interpretation of this equati ‘#(W) Express Ns the value of X which maximizes 2(S(), in terms of Hf, and Hy, Vind UIS0.)). ites 2-4. Generalize part a of Problem 2-3 to the ease of n roro-memory sources, By By... 8. 2-6. You may wish to use the identities (for 0 110 then the only three-binit prefix still unused is 111 and we might set 44 1110 and son Tn the above code, note how starting the code by letting s; cor respond to 0 cut down the number of possible code words. Once wwe had taken this step we were restricted to code words starting with 1, We might expect, therefore, that if we were to select a 2-binit code word to represent s;, we would have more freedom of selection in the ensuing words, and we might not have to use two code words as long as the last two words in the code just constructed. To test this conjecture, we synthesize another binary instantaneous eode for our five symbols; this time we start by setting 1 00 ‘Then we may set a 01 and we still have two prefixes of length 2 which are unused. We may use these as shown on the next page. ‘SOME PROPERTIES OF CODES 53 410 5 110 4 11 ‘The question of which of the two codes constructed above better eannot be resolved on the basis of the information given. ‘Our two coding examples show that in the construetion of an instan- taneous code, the shorter we make the first few words of the eode, the longer we will have to make the later words. By using 0 as a ‘word in the first code, we restricted all later words to sequences starting with 1, In the second eode, we used 00 as the first word. In this case, we were able to form later words by starting with 1 and in addition we could use words starting with 01. 3-5. The Kraft Inequality—Statement and Discussion In Section 3-4 we discussed qualitatively certain constraints on the size of words of an instantaneous code. It is also possible to express the constraints illustrated in a quantitative fashion. ‘The rest of this chapter will be concerned with quantitative constraints ‘on the word lengths of a code. Consider an instantaneous code with source alphabet S= [2.5 ed and code alphabet X = (11, x2, ... ,2;]- Let the code words be Xy, Xs, . .., X, and define the length (ic., number af eode sym- bola) of word X:as 4. It is often desirable that the lengths of the ode words of our code be as small as possible, Necessary and sufficient conditions for the existence of an instantaneous code with Word lengths l,l, . .. , ly are provided by the Kraft inequality (Kraft, 1919). A necessary and sufficient condition for the existence of an instantaneous code with word Hengths fs, la, . . . , dy is that Srect where r is the number of different symbols in the code alphabet, @2)54 INFORMATION THEORY AND CODING For the binary case, the Kraft inequality tells us that the J must satisfy the equation Sorc 3) where the summation is overall the words ofthe bloc code, Before proving the Kraft inequality, it will be instructive to show how tne might use the inequality to determine whether a given sequence of Jie acceptable as the lengths of the eode words or an instantane- ous code Tet us take an information source with four possible symbols, 1 4 ty, and a. In Table 3-6 we have listed five possible eodes for encoding the symbols of this source into a binary alphabet. ‘Tapux 3-6, Five Brvar Cops Cole Code Cole © CodeD Code 0 ° 100 10 110 mM Now we calculate 2-4 for each of these codes For code @ we see that PEP peep et =a Hence, the word lengths of code @ are acceptable as the word lengths of an instantaneous eode. Note, however, that the Kraft inequality does not tell us that code @ is an instantaneous code. ‘The inequality is merely a condition on the word lengths of the code and not on the words themselves. In particular, for this example the Kraft inequality says that there does exist an instan- taneous binary code with four eode words, each of length 2. In ‘SOME PROPERTIES OF CODES 55 ‘this case it is clear not only that the word lengths of code @.satisfy ‘the Kraft inequality, but also that the words themselves do com- prise an instantancous code. For code 6 we caleulate Yrerrperporpas Again, we see that the word lengths of this code are acceptable as ‘word lengths of an instantaneous code. Furthermore, from inspee- ttion of the code words themselves, we see that code © satisfies condition (3-1) and is, therefore, actually instantaneous. Code € is identical with code @ except that we have dropped one binit from the second code word. We calculate Prrarperperpes 1 fand this set of word lengths again satisfies the Kraft inequality. In addition, upon checking, we again find that code © is instan- taneous, Code © is also obtained from code «@ by dropping one init (this time from the fourth word). We find that the word Tengths of code © satisfy the Kraft inequality. As we have men- tioned, this does not ensure that code 9 is instantaneous, and indeed ‘we see that the fourth code word is a prefix of the third code word. Condition (3-1) is thus violated, and code ® is not instantancous. Finally, for code & of Table 3-6, we calculate Yovarrporpospor -h For this code no further investigation is required. ‘The word Jengths do not satisiy the Kraft inequality, and therefore code & ‘could not possibly be an instantaneous block code. We consider one more example before proceeding to the proof ‘of the Kraft inequality. Suppose we wish to encode the outputs ‘of a decimal source, $= {0, 1, 2... , 9}, into a binary instan-5é INFORMATION THEORY AND CODING ecagueltgde Ruri ay edciceves ge tase ge seating tn oud Uo layaibceed Ge Aeeual ee ae tively short binary code words. This is a realistic requirement for a source which emits many more Os and Is than 2s, 3s, etc. If we were to encode 0s and Is from the source asf 140 os) then we might be interested in the question of how short it is pos- sible to make the remaining eight code words. If we require all these eight code words to be of the sume length, say I, the Kraft ‘Tamu 8-7. A Bivany Coon von Decneat, Droms Decimal Binary digits code 0 10 11000 11008 11010 aot 11100 i01 1110 nut inequality will provide us with a direct answer to the question, By the inequality we know we must have Yous 5) By assumption we have le = 1, ly = 2, and Putting these relations in (3-5), we find a+3+8@9 <1 or 125 6) 1 We ould not encode O—> 0 and 1+ 1 since this would exhaust al posible single-binit profes before we had encoded the remaining eight eource symbol ‘and we would not he able to construct an instantaneous code, ‘SOME PROPERTIES OF CODES 57 ‘This not possible, therefore, to find an instantaneous code meeting, ‘our requirements when <5. The Kraft inequality tells us that ‘such a code exists with J equal to 5, but it does not tell us how to ‘eonstruct this code, The code we seek, however, is not hard to find; it is given in Table 3-7. Kraft Inequality—Proof In the previous section we presented the Kraft inequality, er with some examples designed to help the reader obtai ‘understanding of the nature of the constraints imposed by this lity. Now we organize and rearrange these constraints into | proof of the inequality. First, we prove that the inequality is ‘sufficient for the existence of an instantaneous code by actually “constructing an instantaneous code satisfying yrvsi G7) “Assume that we are given word lengths fy, la, . . ly satisfying , and that we are required to construct an instantaneous code rith these word lengths. ‘The required word lengths, ly,» « sy ‘or may not be all distinct, We shall find it useful in our truction to consider all words of the same length at one time. tus, therefore, define n; to be the number of words in our cade Tength 1; n: to be the number of words of length 2; ete, If the tof the fis equal to I, we have We may use the n, to rewrite (3-7). The summation of ns m terms of the form rt, ns terms of the form r-*, ete. It may then be written as Yartsi 9) _ oF, on multiplying by x,58 INFORMATION THEORY AND CODING Upon writing out this equation and rearranging terms, we get me St nit—nght— ss mer ela) We may obtain an interesting sequence of inequalities from (8-11a) by dropping the term on the left and di ter St nght nyt mar (BNIB) (B1le) (11d) (le) This set of inequalities is the key to the construction of the eode we seek, We are required to form m, words of length 1. ‘There are r possible such words that we may form, using an r-symbol code alphabet. Since m 1, then 2* > nl if we take n large enough. Equation (3-19) holds for any integer m; so we must have Src (3-20) 3-8. Some Examples ‘We end this chapter dealing with the properties of codes with two ‘more illustrations of the Kraft inequality and an example of the eon- struction of an instantaneous code. First, assume we wish to encode a source with 10 source symbols into a trinary instantaneous code with word lengths 1, 2, 2, 2, 2, 2, 3, 3,3, 3. Applying the test of the Kraft inequality, we caleulate Yo = 4 +50) +40) -i>1 It is not possible, therefore, to find an instantaneous trinary code with these word lengths. For the second example, assume that we wish to encode symbols from a source with nine symbols into a trinary instantaneous code SOME PROPERTIES OF CODES 61 ‘with word lengths 1, 2,2, 2, 2, 2, 8,3, 3. ‘This time, upon applying ‘our test, we find $45) +304) =1 ‘Therefore, the code we seek is possible. rr) Note how the construction of the above code illustrates the cod- ‘ing method used in the proof of the Kraft inequality. We used one “prefix of length 1 (namely, 0), and we were then restricted to the ‘remaining two prefixes of length 1 for the other code words. ‘This eft us with two times three, or six, permissible code words of length 2, We used only five of these possibilites, saving the sixth (22) for use as a prefix in the three final words. ‘NOTES Note. A method of constructing » nonblock code which is uniquely decoda- ie in the colloquial senso (but not inthe sense defined in Section 3-2) is due to lias, To illustrate his method, consider a zero-memory source with two symbols A and B, occurring with probability 0.7and 0.3, respectively. Wemay ie epresent an arbitrary infinite-output sequence from such « source msm point in the interval (0, 1} as indicated above. Sequences starting with A are ‘signed to tho interval [0 0.7]; sequences starting with AZ are assigned to the BABAAB82 INFORMATION THEORY AND CODING interval (049, 070); te.) To encode a sequence from this source we merely rovide a binary expansion foreach point in (0, 1} a fellows: o oy oo yw a a [Note that itis not necessary to receive the complete binary sequence before we start decoding. For example, ifthe binary sequence starts with O11 «== » tre know that the point represented must lie between 0.375 and 0.50; hence the frat symbol from the source must be A. If the binary sequence starts with (110, the point represented mut lie between 0.375 and 0.4375; hence the frst three symbols from the source must be AAB. "This idea ia the basis of « novel proof of Shannon’ fist theorem (Section 4-3; Chapter 4, Note 1) due to Billingsley (1961). Billingsley expresses sequences from an information source a¢ point inthe unit interval and uses results from the theory of Hausdorf dimension to show that the natural r-adic expansion ‘provides the most efficient specifetion of this point. ‘Note 2, One of the most interesting applications of the ideas discussed in Chapter 3 is that of genetic coding (Golomb, 1961, 1962). Tt has boon deter- mined that the vast amount of information necessary to specify the structure ff a biological system is contained in the chromosomes of the parent system. ‘More precisely it is the deoxyribonucleic acid (DNA) contained in the chromo- somes which transmits the genetic information. In 1953 Crick and Watson Showed that DNA existed in the form of a double helix. ‘These helices may be thought of as being connected by sequences of four nucleotides wh the genetic message. The nucleotides, usually designated A, C, 6, (for adenine, eytosine, guanine, and thymine), correspond to the code symbols Aiscussed in Chapter 3. Experimental evidence therefore indieates that natare ‘operates with a four-symbol code alphabet. These code symbols are combined some manner to represent about twenty amino acids which must be mamu- factured. by the new biological system. The manner in which nucleotides (A, C, G, T) are coded to represent the different amino acids is the primary problem in genetic coding, PROBLEMS. 4-1. An instantaneous code has word lengths ly ly. «+ 4 which satisfy the strict inequality joa ‘The code alphabet is X = [21 2n « «- )#11+ Show that there exist sequences SOME PROPERTIES OF CODES 63, of code finboln 2, 2p 2p « «which ennnot be decoded into sequences af ode words source has six posible outputs with probabilities a shown in Table Boa. Coder e, 0c, 0, & and 9, an given inthe table, are conaerd {Gy Which of there codes are uniquely decodable? {@) Which are instantaneous? (Bind the average lath fr al the uniqly decodable codes. Tanux P32 : Output Pls) 8 e > 5 « ° ° o a a 10 10 a on 110 119, y om mo ht * om ite 10 . oui = nor a ee 38. (a) Which of the sets of word lengths shown in Table P2-8 are accept= ‘able for a uniquely decodable code when the code alphabet is X = (0, 1,2)? Tamun P33 Word length ts 12 3 4 6 Number of words of length (in each eode: Colle. 1 Code @. 1 Code @..: ° Code 2. ne £ (@) Construct an instantaneous code for each cot of acceptable lengths in the table. 3-4. A zero-memory source has the alphabet 8 = {a,b,c} with probabilities 06, 03, and 0.1, respectively. This source is encoded into a nonblock binary ode using the method described above in Note 1. ind ak many of the fist Dinite of the code word corresponding to acbeaad + + » x8 possible.ODING FORMATION |. The Average Length of a Code In Chapter 3 we have shown how to construct instantaneous odes which map symbols from a source alphabet into words com- d of symbols from some code alphabet. For a given souree habet and a given code alphabet, however, we can construct, ny instantaneous or uniquely decodable codes. of acceptable codes forces us to find a criterion by ose among the codes. Perhaps the natural criterion for this ion, although by no means the only possibility, is length.Sgt 65 INFORMATION THEORY AND CODING CODING INFORMATION SOURCES 67 ‘We have, in fact, already used this criterion explicitly in the exam- we temporarily limit our considerations to zero-memory s¢ ples of Chapter 1, and implicitly in several of our later coding fe ieagectcdwn visors thi rnioioas aed discussions. Consider a zero-memory source S, with symbols 1, Aeon no other considerations, from the standpoint of mere ei crees caodtatng 20 Pa | he spectively al economy of expresion and the resulting economy of communication block code encode these symbols into a code alphabet of r symbols, ‘equipment, we prefer a code with many short words to one with long ‘and let the length of the word corresponding to a be 4, Then the swords, - We therefore define the average length of a code. ety oh hi ervsaoaccy saat Definition. Leta block code transform the source syin- i wW H(S) = — ¥ Plog Pe (42) Let Qu Qs - - » » Qe beany g numbers such that Q, 2 0 forall ¢ cd s Q: = 1. By the inequality (2-8), we know that 1 y Pilog PB (43) swith equality if and only if P= Qfor alli, Hence average length as small as p Definition, !Consider a uniquely decodable code which H(S) S — ¥ Prlog & (44) aymbols from a source S into eode words com- at posed of symbols from an r-ary code alphabet. ‘This eode ‘with equality if and only if P; = Q;for all i. ‘will be called compact (for the source S) if its average Equation (4-4) is valid for any set of nonnegative numbers Qs Tength is less than or equal to the average length of all ‘hich sum to 1. We may, therefore, choose other uniquely decodable codes for the same source and the same code alphabet. (4-5) ‘Using these two definitions, we may formulate ‘the fundamental problem of coding information sources as that of finding compact fied obtain codes. Noite that both definitions refer only to the word Lengths: - of codes, and not to the words themselves. Because of this, we = oi ot wer ena 9 <~ $ rer + $ roe) taneous codes (Section 3-7). MeMillan’s inequality assures _us . frord lengths achievable Tira uniquely decodable Stogr } P+ toe (Sr) chievable in an instantaneous code. ie the minimum value of L that it is possible to < Llogr + log 3 4 (48) ps ‘vith an instantaneous cove de i ‘Our definition of L is valid for either zero-memory or Markov { _ lf we add the requirement that our code be instantaneous, the J {information sources. In order to simplify the discussion, however, Kraft inequality tells us that the argument of the second logarithm68 INFORMATION THEORY AND CODING on the right of (4-6) must be less than or equal to 1. The logarithm is, therefore, less than or equal to 0, and H(S) < Llogr (4-70) H(S) ‘Tog <% flea H(S) in (4-76) is measured in bits, Recall that Z is the average number of rary symbols we use to encode S. If we measure the entropy in rary units also, as in (2-5e), then (4-76) may be written HAS) 10 Peer As a check, we ealeulate Z direetly: CODING INFORMATION SOURCES 71 Example 4-2 As «final illustration of coding when the bound given in Equations (4-7) ean be achieved, consider the zero-memory source with seven feymbols shown in Table 43. ‘Assume that we wish to construct an instantancous tinary code for this ‘source. First, we compute the entropy of the source (using trinary units to ‘simplify arithmetic). Bee 1. ~ § rae = JF trinary units /symbol We know, therefore, that we cannot construct a trinary instantaneous code for this information source which uses fewer than 4) trinary symbols per souree faymbol on the average. We ean construct « eae which does achieve exactly this value for L, however, since the P, of this source are of the form (J), with ‘integer. Using (4-9) to find the required word lengths, we obtain the code Asa check, we compute L: 125, Pay = inary aymbola/eureeeymbol Eo72 INFORMATION THEORY AND CODING 48. Shannon's First Theorem ‘The coding problem for the zero-memory source with sym- bol probabilities of the form (1/r)* has been solved. We now turn our attention to the zero-memory source with arbitrary symbol probabilities Equation (4-06) tells us that if log, (1/P2) is an integer, we should choose the word length l, equal to this integer. If log, (1/P) is not, an integer, it might seem reasonable that a compact code could be found by selecting i as the first integer greater than this value. This tempting conjecture is, in fact, not valid, but we shall find that selecting I, in this manner ean Iead to some important results. We select l, therefore, as the unique integer satis log, py S b< low fe +1 (4-10) First, we check to see that the word lengths chosen in this manner ‘the Kraft inequality and are, therefore, acceptable as the ‘word lengths of an instantaneous code. Taking exponentials of the left inequality of (4-10) leads to or aan Perk Summing (4-11) over all i, we obtain a> Die Equation (4-10), therefore, defines an acceptable set of 4 for an instantaneous code. ‘If we multiply (4-10) by P, and sum over all i, we find HAS) SL < HAS) +1 (412) Before proceoding, we should like to point out an important difference between (4-12) and the lower bound for Z obtained in (4-7). Equations (4-7) express a bound for the average length 1, independent of any particular eoding scheme. ‘The bound requires only that our code be instantaneous. Equation (4-12), on the other SS ny CODING INFORMATION SOURCES 73, hand, was derived by assuming the coding method defined in (4-10). It provides both a lower and an upper bound on L valid when we use the given coding method (4-10). Since (4-12) is valid for any zero-memory source, we may apply it to the nth extension of our original source S. HAS) S$ In < HAS!) +1 (413) Z, in (4-13) represents the average length of the code words cor- responding to symbols from the nth extension of the source S. ‘That is, if Asis the length of the code word corresponding to symbol aq and P(a) is the probability of «i, then a 5 Pods (414) L,/n, therefore, is the average number of code symbolst used per single symbol from 8. From (2-16) we know that the entropy of ‘Sv isn times the entropy of 8. Equation (4-13) then may be written as In HAS) < < His) +} (4-150) and so it is possible to make Z4/n as close to H,(S) as we wish by coding the nth extension of S rather than S, (4-158) Equation (4-15a) is known as Shannon's first theorem or the noiseless coding theorem. It is one of the two major theorems of information theory. Equation (4-15a) tells us that we ean make the average number of r-ary code symbols per source symbol as small ‘as, but no smaller than, the entropy of the source measured in rary units. The price we pay for deereasing L,/n is the inereased Do not confuse the symbols b. number of code symbols used per cates that in order to obtain this blocks of n rather than singly. in and L. They both refer to the average gle source symbol. La/n, however, indie wverage, we coded the source symbols ia74 INFORMATION THEORY AND CODING coding complexity caused by the large number (g*) of source symbols with which we must deal 4-4, Shannon's First Theorem for Markov Sources The results of the previous three sections may be extended to include the ease of Markov sources. We provide the necessary proofs by applying our bounds on the average length to an appropri- ate adjoint source—a zero-memory source. Then we need only use the,properties of adjoint sourees as derived in Sections 2-6 and 2-7 to complete the proofs. ‘We define the first-order Markov source S, with sourve symbols 4, 8... 8 and conditional symbol probabilities P(«/s,). We also define S*, the nth extension of 8, with symbols 6, 03, - - . aq and conditional symbol probabilities P(oi/s,). We refer to the first-order (unconditional) symbol probabilities of S and S* as P, and P(e), respectively. "The process of encoding the symbols 4, 8». « » & into an instantaneous block code is identical for the source S and its adjoint source 8. If the length of the code word corresponding to ssl, the average length of the code is ((4-1)] L >» Pa (4-16) Furthermore, the average length is identical for S and 8 sinee P;, the first-order symbol probability of s, is the same for both these sources. S, however, is a zero-memory source, and we may apply (476) to obtain HAS) SL (417) This inequality may be augmented [(2-29)] to read. HAS) $ HAS) SL (4-18) An analogous inequality may be written for S* and ¥. HAS*) < HAS) S In (a9) L, is the average length of a code word corresponding to one of the «;, as defined in (4-14), ‘Again, as in Section 4-3, we remark that these inequalities are completely general in that they do not depend upon the coding CODING INFORMATION SOURCES 75 scheme we use. If we now select the /; according to (4-10), we may bound Z above and below (4-12), HS) P,> +--+ > P.. By regarding the last two symbols as 1 of S as combined into one symbol, we obtain a new source} from S H(S) = ¥ Polo p, containing only q~ 1 symbols. We refer to this new source as a a reduction of S. The symbols of this reduction of S may be reor- = 1.22 bits/souree symbol dered, and again we may combine the two least probable symbols Note that Lis indeed bounded by to form a reduction of this reduction of S. By proceeding in this . manner, we construct sequence of sources, ench containing one BS Tes ee I fewer symbol than the previous one, until we arrive at a source ‘This fact, however, is little consolation for a poor code. Tt with only two symbols is easy to find an instantaneous code for this source which is better RP Feet cyst shh a pmree olidkcarmton than code @. Such a code (code &) is given in the last column of BEaqiinivccckwccou nue ee eet ca ater eee J Wetak hia toe a zromenery ore forty ake femenine aber cis pase ce Since we are allowed to code only one symbol fom Sa aime, tdowe not make = 1.33 binits/souree symbol Uany diforenco whether Sin a neromemory or a Marlow ure. rnd sue7B INFORMATION THEORY AND CODING Original source Reduced sources Symbols Probabititiee SS BS Frovar 4-1. A souree and its reductions Construction of a sequence of reduced sources, as illustrated, is the first step in the construction of a compact instantaneous code for the original source S. ‘The second step is merely the recog- nition that a binary compact instantaneous code for the last reduced source (a source with only two symbols) is the trivial code with the two words 0 and 1. As the final step, we shall show that if we have a compact instantaneous eode for one of the sources: in the sequence of reduced sources, it is a simple matter to construct ‘4 compact instantaneous code for the source immediately preceding this reduced source. Using this fact, we start at the last reduced source and its trivial compact instantaneous code and work back- ward along the sequence of sources until we arrive at a compact instantaneous code for the original source. Let us assumo that we have found a compact instantaneous code for S,, one of the sources in a sequence of reduced sources. One of ‘the symbols of S;, say ss, is formed from two symbols of the pre- ceding source S,-1. We eall these two symbols gy and gy; Each of the other symbols of S, corresponds to one of the remaining sym- bols of Ss. Then the compact instantaneous code for Sis is formed from the code for S, as follows: We assign to each symbol of S,-1 (except suo and sa1) the code word used by the corresponding symbol of §;. ‘The eode words used by suo and ‘a1 are formed by adding a 0 and 1, respectively, +o tho code word used for ta. Cony It is easy to see that the code we form in this manner is instan- CODING INFORMATION SOURCES 79 taneous [condition (3-1)]. The proof that the code is compact is not so immediate, and we defer the proof of this fact until after we have illustrated the construction of a compact eode, Example 4-4. We illustrate the synthesis of a binary compact code for the source of Figure 4-1. Original source Reduced sources Syne Prabae ole bilities Code Ss Ss 8 on. “ OM 20.2 0104 p03 OF ono Jor on owo1 ‘roux 4-2, Synthesis of a compact code. We formed the compact code shown at the left in the three steps previously noted. First, we formed a sequence of reduced sources from the originalsource S. (Seealso Figure 4-1.) ‘Then we assigned the words 0 and 1 to the last source in the sequence (Sy in this case). Finally, we worked our way back from Ss to S through the reduced sources, As we passed from source to source, we “decomposed” ‘one code word each time in order to form two new code words. Several properties of compact codes are illustrated by this pro- cedure. ‘The multiplicity of compact eodes is especially important. Note that the method used to form tsro words from one in passing from reduced source to reduced souree is merely to add a binit to the fend of the word we decompose. It makes no difference which of the ‘two words formed is assigned to which of the source symbols. ‘This means that the assignment of the two code symbols 0 and 1 to tthe various words of the compact code we construct is arbitrary. fe may complement the jth digit of every word of the eode and ‘obtain another compact code. For example, if we complement the first and last digits of the code of Figure 4-2, we have the “new” ‘compact code: ‘1 The complement of O is 1; the complement of 1 is 0.80 INFORMATION THEORY AND CODING CODING INFORMATION SOURCES 81 0 If we choose the second or the third, the word lengths obtained 10 are m1 1, 2,3, 4, 5,5 1100 ee fight ‘The average lengths of the two codes obtained are identical: 1010 L = 1004) + 2(0.3) + 4(0.1) + 4(0.1) + 400.06) + 4(0.04) ‘This method ot pecdiucing « new oumpiiot cae, hawaret ints = 2.2 binits/symbol in just trivial differences in the two codes. ‘The new code is L = 0A) + 20.3) + 30.1) + 40.1) + 5(0.06) + 5(0.04) obtained from the old code by a relabeling. It is also possible to = 22 binits/symbol obtain two compact codes for the same source which are funda- A : mentally different. To see this, we synthesize a different code and we cannot construct an instantaneous code for this source with for the example given in Figure 4-2. a smaller average length. __ Another point made evident by the synthesis procedure deseribed is that it may sometimes be unnecessary to form a sequence of reductions of the original source all the way to a source with only I two symbols. ‘This is so since we need only form reductions until Original source we find the first reduction for which we have a compact eode, Once ee we have a compact code for any reduetion of a source, we may start Pe ee ‘working backward from this compact code, as described in rule . (4-24). ‘This point is illustrated in Figure 4-4. Example 4-8. We construet a different code for the source of Example 44 in Figure 43. not Reduced Original source pny Symbols Probebititien Code S Fiavme 4-3, Synthesis of a compact code, rs Ps > --~ > Py, the lengths of the words assigned to these symbols will be ordered so that, < lz S++ < ly This is not surprising, Tt is merely an expression of the fact that we assign the shortest words to the most probable symbols of our code. The second property is perhaps a bit less obvious. We have shown that the lengths of the last two words (in order of decreasing probability) of a compact code are identical: haha 27) If there are several symbols with probability P,, we may assign their subscripts so that the words assigned to the last two symbols differ only in their last digit. #8. r-ary Compact Codes Section 4-6 pointed out that the construction of a binary compact code proceeded in three steps. First, we formed a sequence of reduced sources from the original source. ‘Then we found a compact code for one of the sources in this sequence. Finally, we worked our way backward through the sequence, constructing new compact codes from the ones we had obtained, until we formed a compact ode for the original source S, In this section we shall see that the construction of a compact code consists of the same three steps when the code alphabet has r symbols. The last two of these steps, furthermore, will be changed in no important respect from the binary case. The formation of reduced sources preparatory to the synthesis, of a binary compact code proceeded by combining the ‘wo least Probable source symbols in order to form a single symbol. When wwe wish to form an r-ary compact code, we shall combine the source84 INFORMATION THEORY AND CODING ‘symbols r ata time in order to form one symbol in the reduced source. ‘One hitch appears in the r-ary case, however, which did not appear in the binary ease. In the binary case, each souree in the sequence of reduced sourees contained one fewer symbol than the souree immediately preceding. In the rary ease, we combine r symbols ‘to form one symbol, and thus each source in the sequence has r — 1 fewer symbols than the preceding source. We would like the last source in the sequence to have exactly r symbols. (This will allow ‘us to construct the trivial compact code for this source.) The last source will have r symbols if and only if the otiginal souree has r+ a(r—1) symbols, where a is an integer. Therefore, if the original source does not have r + a(r ~ 1) symbols, we add “dummy symbols” to the source until this number is reached. ‘The dummy symbols are assumed to have probability 0, and so they may be ignored after a code is formed Example 4-6. Consider the source S with 11 symbols shown in Figure 4-5. We wish to form a sequence of resiuced sources from ths suree before encoding the source in a quaternary code—a code using four code symbols. Ifthe last source in the sequence is to have four symbols, S must have 4 + 3a symbols, whore ais an integer. Since 11 isnot ofthe form 4 + Sa, we add two dummy {ym to to bring the total number of symbols to 18. Now, reducing the {ource four symbols at time produces a source with exactly four symbols, Original source Reduced eoureer Symbols Probabilities Ss BRSS5RSSER8 Ficuns 45. A source and its reductions CODING INFORMATION SOURCES 85 Having formed the reduetions shown in Figure 4-5, we proceed to synthesize a compact code in the same manner as in Section 4-6. We assign r code words, each of length 1, to the last reduction in order to form s compact code for this source. ‘This code is enlarged, just as in the binary ease, to form a compact code for the preceding reduced source. Each time we go from a reduced source to the preceding reduced source, we form r symbols from one symbol for a net gain of r — 1 symbols. The proof that if we start with a compact code we shall produce a compact code by this pro- cedure is entirely analagous to the proof provided in Section 4-7 (Problem 4-2), Example 4-7, To provide an illustration of the procedure deseribed above, wwe find quaternary compact code (Figure 4-6) for the souree of Figure 4-5. Original cource Reduced sources ‘Symbol Probability Cole word Fiounx 46, A quaternary compact code. 49. Code Efficiency and Redundancy Shannon’s first theorem shows that there exists a common yard- stick with which we may measure any information source. The valve of a symbol from an information source S may be measured in terms of an equivalent number of binary digits needed to repre~8 INFORMATION THEORY AND CODING sent one symbol from that source; the theorem says that the average value of a symbol from S is H(S). More generally, the average value of a symbol from S in terms of r-ary digits is 11,(S). ‘Let the average length of a uniquely decodable rary code for the source 8 be L. L cannot be less than H,(S). Accordingly, we define 9, the efficiency of the eode, by 1 HO (428) It is also possible to define the redundancy of a code. Redundancy = 1 — 9 L- HAS) tae) i (4-29) ple 4-8. Consider the zero-memory source $ = {41, 2], with P(s:) = Fiand Piss) = We ealulste HOS) = Plog 4 +4 oe § = Oat bit ‘A-compact code for this source is as follows: 8 Po) a t * + 1 ‘The average length of this code is 1 binit, and so the elficieney is = 08t1 ‘To improve the efcieney, we might code S, the second extension of S: Compact « Pod ‘ode te i. am ts 10 a ve m9 a CODING INFORMATION SOURCES ‘The average length of this codo i $f binite, The entropy of S* is 2H(8); 50 2x osu x16 a 7 = 0.961 the third and fourth extensions of S, we find efficiencies m= 0.985, and = 0.901 As we encore higher and higher extensions of the original sourse S, the ef cieney must approach 1, In this example, the approach is quite rapid, and little increase in efficiency can be obtained by going further than the second extension. Such behavior is typical of Huflman coding. ‘The previous example showed the increase in efficiency obtained by encoding higher and higher extensions of a source. It will be instructive to examine the efficiency as a function of r, the number ‘of symbols in the code alphabet. Example 4-9. We takes zero-memory source 8 with 13 symbols and symbol probabilities as shown in Table 4-5. In the same table we list the compact (Huffman) code for code alphabets comprised of 2 to 13 symbols. ‘Tamux 4-5. Cowract Cones ror Dirvenest Cope Aurnasets Compact codes for r = ~® «32 UO 987 6 5 4 2 + 00000000 000 w 4 ‘HEL Beat Bae Clee oar seat Jaan eres et eens Sena te 22 2 2 2 2 2 2 2 2 20 100 ve 33 3 8 3 3 3 3 3 2 201 100 * 4404 44 4 4 4 3 202 100 te 55 5 5 5 5 5 50 32 23 20 101 aa 6 6 6 6 6 6 6 51 33-30 21 100 tr 7 7 7 7 7 7 6 52 34 31 22 LOL od 8 8 8 8 80 71 62 53 40 32 220 1110 ac 9 9 9 9 81 72 63 54 41 330 221 111100 ee A A AO 91 82 73 64 550 42 331 2220 111101 a B BO Al 92 83 74 65 551 43 382 2221 111110 ve C BI AZ 93 84 75 66 552 44 333 2222 111111 Average keoghh.. 1 HP Ht HEH HH OW88 INFORMATION THEORY AND CODING ‘The entropy of the source of Table 4-5 is 8.125 bits per symbol. Using this fact and (4-28), we may plot the efficiency asa function ofr. 1.09 ose 04 ose} 080 oa os ose 22} 80, ode etic Ta. +. 2 Number of code symbol Frovne 4-7. Code efficiency versus number of eode symbols. From Figure 4-7 we se that the efficiency tends to limb as r decreases. The increase in eficiency, however, in not monotonic. Note the efficiency of the code for r= 2andr = 4, The symbol probabilities are all of the form 1/2 for 1/4%, where « isan integer. In these eases, we know (Section 4-2) that we fan find a compact code with average length equal to the entropy. NOTES, Note 1. Tn this chapter we have proved Shannon's fist theorem only for ergovlic Markov sources with a finite number of symbols (Le. states). A more ‘legant proof ofthis theorem, valid for any stationary, ergodic source, was given by MeMillan (1058) in & slightly different form, ealled the asymptotic equi- partition property (AEP). For a general source 8, let 1 Shs ) © 18 Fa, od ‘Shas the AEP if Is, « .- ,4,)/n converges in probability to H(S). The importance of the AEP Tica in the fact that long sequences of outputs from 8 source with this property’ can be divided into two classes "| A class such that each sequence in the elass has probebility roughly equal toon 2. A class composed of sequences which hardly ever occur ‘A simple combinatorial proof of the AEP has been provided by Thomasian, CODING INFORMATION SOURCES 89. (1960). A generalization to more complicated sourees is given by Peres (1958). Nole 2. In Seetion 4-6 we provided an example to show that two diferent (in their word lengths) binary codes might each be compact for a given source. Golomb has investigated the conditions under which this phenomenon ean ‘occur and the number of diferent possible nontrivially diferent compact codes for a given source. ‘We can describe the construction of a code by moans of a code tree (Fano, 1001). For example, eonsider the binary code and its corresponding tree. to uo mn ‘Then the question of how many different codes exist for a source with ¢ symbols ean be phrased in teria of code tres. For q = 2 there is only one possible tree corresponding to the word lengths. nat net ‘For q = 3 there is again only one possible tree corresponding to the word lengths. ar he? For q = 4 there are two posible trees.90 INFORMATION THEORY AND CODING For q = 5 there are three possible trees. For q = Gand 7 there are five and nine diferent trees, respectively. Golotab bas also found conditions on the symbol probabilities in order for ‘more than one compact code to exist. For example, with q = 4, we must clearly have P, = Py + Ps. Further analysis shows that $< Py < § if two ‘different compact codes exist. ‘Note 8. We have treated the problem of coding information sources under the assumption that the duration (or some other criterion of cost) isthe same {for each code symbol. When this is not the ease, certain modifiations of the results of Chapter 4 are required. Let the code alphabet be Xe eyes. ad tnd Iet the duration of code symbol 2; be 4 ‘Then if N(T) is the number of sequences of duration exaetly 7, N(Q) = NOt) + NT — 6) $206 NT -4) When this difference equation is solved, we find that N(P) grows as ART for large T, where A is some constant and ity isthe largest real root of the charac ‘teristic equation Pepee bee pee 1]0 ‘The asymptotic number of equivalent binits per unit time is then tnd this result ean be used to modify our form of Shannon's frst theorem. ‘The problem of finite time coding for such a code alphabet (equivalent to Huffman coding) has been treated by Karp (1961). CODING INFORMATION SOURCES 91 PROBLEMS 4:1, Prove Equation (1-22) for mth-order Markov sources. 4.2, Show that proceeding from redaced source to reduced source, asdesribed in Section 4-8, will produce an r-ary compact code, if one starts with an rary ‘compact code. 4-8. A sequence of symbols from S* is coded by the Huffman procedure into fe code alphabet X = [zs, 24. - 2). The result of this coding procedure nay be viewed as anew information source with the rource alphabet X. Show that the probability of each of the symbols z, of this new soureo approaches fr as m increases. 44. A zero-memory binary source has P(0) = 0.1 and P(1) = 09. (@) Find HS). (©) Find Z, the average word length of a compact cole for $ when X = 0, 11. (pind Lu for n= 2,3, 4and n+ ©, when > isenended int a sompast code, still with X= (0, 1). (@ Find the effcieney of your four codes. 46. In Problem 4-1 we encoded 5, S#, 5%, and S¢ into X, These codings ‘produced sequences of Os and 1s. We can view these sequences as being “mitted from some new source S,, a4 shown in Figure P 4-5. Find H(S4) when n= 1,2,3,4 pinay source So Frovme P45 4-6, Given the following table: Sofa fetal a] | Powel a la lala lala le (@) Find (8) and (5). (©) Find « compact code for (8) when X = (0,1) and X = (0,1, 2}. (©) Compute L for both the above codes. 441 Given the following table: Bees am | ajalajeda os 0.1 | 0.05 | 0.08 | 0.08 | 0.0592 INFORMATION THEORY AND CODING (a) Find a compset code for this source with X = (0, 1,2) (@) There may be more than one nontrivially different (Le, different sets of word lengths) compact code for this source and this alphabet.” List the sets of ‘word lengths forall such codes you ean find “8, Let « =} in Problem 2-14, A binary ende is found for S with L = H(S).. Find 17, the average length of a compact code for S'. 48, The source 8 has nine symbols; each occurs with probability §. (a) Find a compact code for S using the code alphabet X = (0, 1}. (©) Find a compsct code for 8 using the code alphabet X = (0, 1, 2}. (©) Find s compact eode for $ using the code alphabet X = {0, 1, 2,8) 410. A source 5 has six symbols with probabilities P, to Py, respectively. Assume that we have ordered the Po that P, > P, > ‘We wish to find a compact code for this source using the code al = 10,1,2,3). Find 2 set of word lengths for such a compact code if Pe = gy. 4-11. Find all possible different compact binary codes for the souree shown in the following tablo: Bessa [aleiealalala PO). -+1+++-|0.20 0.18 0.12 0.10 0.10 0.08 | 0.06 0.06 0.05[0.08, (Count as “different” se only those coves with different ats af word lengths & J-12. (a) Find the fve diferent code trees corresponding to q = 6 in Note 2 ©) Find the nine diferent cade trees coresponding to 9 = 7. 4-18, This problem deals wth a generalization of Noto. Find all posible dierent code’ trees corresponding to trinary compact codes for sources with 12 3,45,6,7,8 5 CHANNELS AND MUTUAL INFORMATION 5-1. Introduction In the first four chapters, we were concerned with properties of information sources and with transformations of sequences of source symbols into sequences of code symbols, We were able to relate our information measure to properties of information sources. Tn particular, the entropy of a source (expressed in suitable units) was shown to define a lower bound to the average number of code symbols needed to encode each source symbol, We used this bound in Seetion 4-9 to define the efficiency and redundancy of a cy94 INFORMATION THEORY AND CODING code. Indeed, in retrospect we note that a large portion of the first part of this book was devoted to providing a background for our definitions of efficiency and redundaney and to the synthesis of codes containing as little redundancy as possible. In view of this preoccupation with redundancy reduetion up to now, it may surprise the reader to learn that Chapters 5 and 6 will be concerned primarily with methods of putting redundancy back into codes! We shall see that it is not always desirable to use codes with little or no redundancy. In this chapter, our interest will shift from the information source to the information channel—that is, from information generation to information transmission, Our introduction of the concept of an information channel leads directly to the possibility of errors arising in the process of information transmission. We shall study the effect such errors have on our efforts to transmit information, This will, in turn, lead to the possibility of eoding in order to decrease the effect of errors caused by an information channel. The reader should not be surprised to learn that our information measure may be used to analyze this type of coding, as well as the type of coding already discussed. In fact, in spite of the considerable progress we have made so far, the central result of information theory and the most dramatic use of the concept of entropy are yet to come. This result—Shannon's remarkable second theorem—will use the entropy idea to describe how we may utilize an unreliable information ehan- nel to transmit reliable information! ‘information Channels Our primary concern in the rest of this book will be the informa- tion channel, Definition. An information channelt is described by giving an input alphabet A = {aj, i= 1, 2, Be an output alphabet B= [0], j= 1,2,..., s;anda 4 The channel defined above is sometimes called w zero-memory information channel. A more general definition, where the probability of « given output 1, may depend upon several preceding input symbols or even output aymabols, is also possible. Such channels are referred to as channels with memory. CHANNELS AND MUTUAL INFORMATION 95 set of conditional probabilities P(bj/a)) for all ¢ and P(b,/as) is just the probability that the output symbol b, will be received if the input symbol a, is sent. A particular channel of great theoretical and practical importance is the binary symmetric S ‘The channel diagram of the BSC is shown in Figure 5-2. As usual, we let) =1—p. This channel has two input symbols (a, = 0, a; = 1) and two output symbols (6; = 0, bs = 1). It is symmetric because the probability of receiving a 1 if a 0 is sent is _ 0 1 Fioune 5-1. An information channel, Fone 52. The binary sym- ‘motrie channel (BSC), equal to the probability of receiving a 0 if 1is sent; this probability, the probability that an error will oceur, is p A convenient way of describing an information channel is to arrange the conditional probabilities of its outputs as shown in Figure 5:3. Outpuse bs Pisa) Pnfa) ~~ Pola Plas) Pobvfas) +++ Plbv/ad POs/a) POrfar) Pebu/as) Frovas 5:3, Description of an information channel Note that each row of this array corresponds to a fixed input, and that the terms in this row are just the probabilities of obtaining the various b; at the output if that fixed input is sent, We shall see ‘this description of an information channel so often that it will be Useful to have a more streamlined notation for it. Accordingly, we define Py = Pooja) 641)9 INFORMATION THEORY AND CODING and the array of Figure 5-3 becomes the channel matrix P: Pa Pa c++ Pa Bie ene eae FaiPa ed a An information channel is completely described by giving its channel matriz. We therefore use P interchangeably to represent both the ‘Panel uated oud the dena. Tish sor of the hast wth rminpoae oS gat or channel, and each column corresponds to a channel output. Note a fundamental property of the channel matrix—the terms in any one row of the matrix must sum to 1.+ This follows since, if we send any input symbol a;, we must get some output symbol. We write this equation for future reference: Dye meee gee (63) ‘The channel mateix of the BSC is PP (4) ba wo Just as we did in the case of information sources, we may view the inputs and outputs of a channel in blocks of n symbols, rather than individually. Thus, we define the nth extension of a channel. Definition. Consider an information channel with input, alphabet A= {ai}, ¢= 1,2, - . . , rj output alphabet, B= (bi j= 1,2, and channel matrix Pu Pa Ps Ba. =- Pa Pa ‘Tho nth extension of this channel has input alphabet A= [ad,i= 1,2, . .. ,rsjoutput alphabet B* = [85], J=1,2,-.., 9%} and channel matrix Such matrices are called Markov matrices oF stochastic matrices. CHANNELS AND MUTUAL INFORMATION 97 Un Me +++ Th n-[t» Ha The Tmt Us Each of the inputs a consists of a sequence of n ele- mentary input symbols (a, a, . . + » di), and each of the outputs 8; consists of a sequence of n elementary output symbols (bj, bj, ... » ba). ‘The probability Tly = P(B,/a) consists of the product of the correspond- ing elementary symbol probabilities. Tust as was the case when we defined the extension of an informa- tion source, the extension of an information channel is not really 4 new concept, but just a new way of viewing an old concept. ‘Merely by looking at symbols of some channel in blocks of length n, ‘we obtain the nth extension of that channel. Example 8-1. The second extension of the BSC isa channel with four input symbols and four output symbols. ts channel matrix is shown in Figure 5-4 rrp pt p* pp Ppp ppp Fiouns 5-4, Channel matrix of the (BSC). Bt pp pp pt wa|fe > pp ‘We note that the channel matrix of the (BSC) may be written as a matrix of matrices. Let P, as before, be the channel matrix of the BSC. Then, the channel matrix of the (BSC)? can also be written _ [ee n-[% | The above matrix is known as the Kronecker square (Bellman, 1960) (or tensor square) of the matrix P. In the more general case the channel matrix of the nth extension of a channel is the nth Kronecker power of the original ehannel matrix. In the first part of the book, we used our information measure to measure the average amount of information produced by a source. The function of an information channel, however, is not to produce98 INFORMATION THEORY AND CODING information, but to transmit information from the input to the output. We expect, therefore, to use our information measure to measure the ability of a channel to transport information. This will indeed be the case; we now proceed to an investigation of ‘the amount of information a channel can transmit, 5-3. Probability Relations in a Channel Consider an information channel with r input symbols and s output symbols, We define the channel by the channel matrix P: Pu Pas Pu P; el Pa Pa Let us select input symbols according to the probabilities P(a;), P(a:), . . . , P(a,) for transmission through this channel. Then the output symbols will appear according to some other set of probabilities: P(b;), P(bs), . . . , P(b,). The relations between the probabilities of the various input symbols and the probabilities of the various output symbols are easily derived, For example, there are r ways in which we might receive output symbol bs. If ai is sent, b: will occur with probability Pir; if ae is sent, 6 will oceur with probability P21; ete. We therefore write P(a,)Pu + Pla)Pu + * ++ + P(@)Pa = PQ) (5-60) P(a:)Pur + Pla:)Pae + + ++ + P(@)Pra = P(bs) (5-65) P(a:)Pu + P(as)Pu + +++ + PC)Pn = Plb,) (6-60) Faquations (5-6) provide us with expressions for the probabilities of the various output symbols if we are given the input probabilities P(q,) and the channel matrix, the matrix of conditional probabilities P(b,/a). For the remainder of this chapter, we assume that we are given the P(q,) and the P(b;/a;), so that the P(,) may be eal- culated from (5-6). Note, however, that if we are given the output probabilities P(,) and P(,/a,), it may not be possible to invert the system of linear equations (5-6) in order to determine the P(a). For exaruple, ina BSC with p = J, any set of input probabilities CHANNELS AND MUTUAL INFORMATION 99 will lead to output symbols which are equiprobable, In general, there may be many input distributions which lead to the same output distribution. If we are given the input distribution, on the other hand, we may alWaye calculate a unique output distribu- tion with the aid of (5-6). In addition to the P(b,), there are two more sets of probabilities associated with an information channel which may be calculated from the P(a,) and the P(b,/a,). According to Bayes’ Jaw, the conditional probability of an input a, given that an output b; has been received, is Psa) P(a) Pease) = POmiried (5-70) or, using (6-6), Pea) = —PeolanPa) 6) 3 PeovayPeay The probabilities P(a,/b,) are sometimes referred to as backward Probabilities in order to distinguish them from the forward proba- bilities Po/a). The numerator of the right side of (5-7) is the probability of the joint event (0,4), Play b)) = P(b;/a)P(a) (5-8a) and this quantity may also be reeog- nized as Frovne 5.5, A noisy informa- P(e, bj) = P(ai/b,)P(b)) (5-86) tion channel ~ Example 6-2. Let us illustrate the calculation of the various probabilities ‘sssociated with an information channel. We take a binary channel; that is, A = (0, 1) and B = (0,1). We assume that the P(b,/a:) are as shown in the ae a »-[t d] We amociate the rors and columns ofthe above matrix with the aput and euipat symbols i the aural aver Theriore Pr [b= Ofe Soh 2 Frit ija~ 0) =f oa, Fialy, vo tone ta P(e 0) ob aah Br laci}= 4. The informalon given shove i nealy’ summariaed in Piste 3.100 INFORMATION THEORY AND CODING ‘The probabilities ofthe output symbols are obtained with the use of (6-6). Pr (b= 0} = @) + DG) =H (6-00) Pr (b= 1) = + DG) = 08) As a check, we note that Pr | = 0) +Pr (b= 1] = 1. The conditional input probabilities are obtained from (6-7) aw Pr ia =f =) - 2@ _ (1 te= 07 = 0) ~ SAP ae (6-100) Prfa=1/ a1) = 2® _ foo 10 91) = Sy ats (6-10) The other two backward probabilities may be similarly obtained. A sim- ler method, however, is to use the fact that Pr a = 0/b = 0} + Pr {a = 1/ b= 0} = Land Pr (a 0/6 =1) + Pr a= 1/b= 1) = 1, Thuy Pr (a= 1/0 = 0) = ay (6-106) Pr (a=0/f = 1) = yy (6-104) ‘The probabilities of various joint events are found by using (6-8). We ealeu- late just one of these: Pr {a = 0,b = 0] = Pr [a = 0/) = 0} Pr {b= 0} andy 61) = 5-4. A Priori and A Posteriori Entropies v ‘The various output symbols of our channel occur according to the set of probabilities P(,). Note that the probability a given output symbol will be b; is P(b,) if we do not know which input symbol is sent. On the other hand, if we do know that input symbol a: is sent, the probability that the corresponding output. will be bj changes from P(,) to P(b,/a,). Likewise, we recall that the input symbol a is chosen with probability P(a). If we observe the out- put symbol }j, however, we know the probability that a, is the corresponding input symbol is P(a/b,) ((5-7)). Let us focus our attention on this change induced on the probabilities of the various input symbols by the reception of a given output symbol b; We shall refer to the P(a,) as the a priori probabilities of the input symbols—the probabilities of the a; before the reception of an out put symbol. The P(a/b,) will be called the a posteriori proba- bilities of the input symbols—the probabilities after the reception CHANNELS AND MUTUAL INFORMATION 101 of}, From Section 2-2 we know that we can calculate the entropy of the set of input symbols with respect to both these sets of proba- bilities. The a priori entropy of A ist H(A) = J PC) ton B55 2) and the a posteriori entropy of A, when b; is received, is H(A/b) = J Plo/t) oe wag 6-18) ‘The interpretation of these two quantities follows directly from Shannon’s first theorem. H(A) is the average number of binits needed to represent a symbol from a source with the a priori proba- bilities Pa), i= 1, 2, ... , 7; H(A/b) is the average number of binits needed to represent a symbol : from a source with the posteriori prob- 5 —— ee abilities P(a/),f=1,2,... 57. Brample 6-3. We repest, es Figure 5-5, ‘the figure used in Examplo 6-2 for case of reference. ‘Thea priori entropy of the set of input symbols is H(A) = Pog + Hog 4 ~ 0811 bit 4) Pyeons 5-6, A noisy informa Af we receive the symbol 0 st the output of io channel, the channel, our a posteriori probabilities will he given by (5-104) and (5-108). ‘The a posteriori entropy is H(A/O) = ¥ log # + zr log 21 = 0.276 bit 9) It we reecive the symbol 1, on the other hand, the a posteriori entropy is HAM) = plow f+ $5 low $ = 0.998 bit 19) ‘Hence, if «0 is received, the entropy—our uncertainty about which iaput was sent—deereases, whereas if a1 is received, our uncertainty inereasos. wv 5-5 Generalization of Shannon's First Theorem Shannon's first theorem tells us that the entropy of an alphabet ‘may be interpreted as the average number of binits necessary to In the remainder of this book it will be convenient to omit the subseripts ‘om o; and b, when writing sume over all aymbols of the A or B alphabets,102 INFORMATION THEORY AND CODING. represent one symbol of that alphabet. Consider this interpreta- tion applied to the a priori and a posteriori entropies (Figure 5-7) Before reception of the output symbol of the channel, we associ- ate the a priori probabilities P(a) with the input alphabet A. ‘The average number of binits necessary to represent a symbol from this alphabet is H(A). If we receive « Crome 8 a given symbol, say b,, we associate = the a posteriori probabilities P(a/by) Ficene 5-7. An information with the input alphabet. The aver- ‘age number of binits necessary to rep- resent a symbol from the alphabet with these (a posteriori) statis- ties is H(A/b,). Since the output symbols occur with probabilities (i), we might expect that the average number of binits (now aver- ‘aged over 6, also) necessary to represent an input eymbol a, if we are given an output symbol, is the average a posteriori entropy, F POH’) (17) ‘This important result is, in fact, true. Tt does not follow from Shannon's first theorem, however. ‘That theorem deals only with coding for a source with a fixed set of source statisties and not with coding for a source which selects a new set of source statistics after each output symbol. We therefore generalize Shannon's first ‘theorem to cover this ease, ‘The question we ask in order to obtain such # generalization is ‘the same question we asked in order to obtain Shannon's first theorem, namely, “What is the most efficient method of coding from a source?” (In this ease, the source is A.) This time, how- ever, the statisties of the source we wish to eode change from symbol tosymbol. The indication of which set of source statistics we have is provided by the output of the channel bs, Since a compact eode for one set of source statisties will not, in general, be a compact code for another set of source statisties, we take advantage of our knowledge of b for each transmitted symbol to construct s binaryt ‘codes—one for each of the possible received symbols bj. When the output of our channel is 6, we use the jth binary code to encode the transmitted symbol a;. Let the word lengths of our s codes be as shown in Table 5-1. 1 The binary assumption is not necessary, but it serves to simplify the dis- cussion which follows CHANNELS AND MUTUAL INFORMATION ‘Tanus 5-1. Wonp Luxoris von # Coons Input Code Code Code 1 2 Ia tel fee If we require these codes to be instantancous, we may apply the first part of Shannon's first theorem (4-7) to each code separately and obtain H(A/) SX Pla/b ls * Ls G18) L; is the average length of the jth code. We employ the conditional probabilities P(a/t,) rather than the marginal probabilities P(a) to calculate L,, since the jth code is employed only when b, is the received symbol. The average number of bi used for each member of the A alphabet when we encode in this fashion is obtained by averaging with respect to the received symbols b;. Multiplying G-18) by P(b) and summing over all B yield J H(A/o)PO) < FY PCa, bly G19) Phare Z Ww'ths average number of binite per symbol from the A alphabet—averaged with respect to both input and output symbols. Note the similarity of (5-19) to (4-7). In order to show that the bound (5-19) can be achieved, we next describe a specific coding procedure. When b, is the output of our ‘channel, we select ly, the word length of the code word eorrespond- {ng to the input a, as the unique integer satisfying 1 Jon eae $< Noe pray +1 20) ‘Word lengths defined in this fashion satisfy the Kraft inequality for each j. The ly, therefore, define s sets of word lengths accepta ble as the word lengths of # instantaneous codes, Now multiply ‘This can be shown in exactly the same manner used in our proof of Shan- ‘non's fret theorem (Section 4-3).104 INFORMATION THEORY AND CODNG 6-20) by Pla, b) = Pla/b) PO), PO)PCa/b) I0K pers < WiP(OH b) 1 < Pl)Pla/d) toe pays + Plas») (621) and sum this equation over all members of the A and B alphabets: YPrOMam where p= 1p. Assumo that the probabilities of a 0 and a 1 being ies of 8 O and a 1 boing trans mitted are w and 6, respectively. We-write the mutual information in the form 1a: 8) = HH) ~ H@/8) = m1 ~ 3 70) r/o me as ~ 10 -¥ 0 (pin! + rie) = 10) ~ (rive! + pim3) ow ‘The probabilities that 6) = 0 and D, = 1 are easily calculated to be wp + ép Fioune: 5.10. Geometric interpretation of the mutual information of a BSC, and op + ap, reapectively. Hence 1A; 8) = [lp + on tn hse + or +59 06 =] ~ (p06 5 + pion os CHANNELS AND MUTUAL INFORMATION ITT We may write I(A; B) in terme of the entropy function (Figure 2-3) (A; B) = Hlep + H(p), and Figure 6-10 provides » ‘geometric proof of the nonnegativty of mutual information, Certain limiting conditions of interest may slso be seen from Figure 5-10. For example, for a fixed value of p, we may vary w and examine the behavior fof I(A; B). We see that I(4; B) achieves its maximum when « = }, and the ‘alue of this maximum is 1 ~ Hip). For » = O or = 1, on the other hand, ‘the mutual information is 0. 546, Noiseless Channels and Deterministic Channels In this section, we define two special types of channels and obtain simplified expressions for the mutual information of these channels. b bs bs abe bs ek Froume 5-11, A noveless chaanel. In the discussion that follows, we assume that each column of the channel matrix has at least one nonzero element, An output symbol corresponding to a column of zeros will oceur with probabil- ity 0 for any distribution over the input symbols. Tt is therefore ‘of no interest and may be ignored. Definition, A channel deseribed by a channel matrix with one, and only one, nonzero element in each column will be called s noiseless channel. Example 6-6. ‘The channel matrix of « noiseless channel is given a8 $400 00 polo 0 8 wy v0 ooo o O14 ‘The channel diagram of this channel is shown in Figure 5-11.112 INFORMATION THEORY AND CODING A.BSC with probability of error p equal to 0is a noiseless channel, Note, however, that a BSC with probability of error equal to 1 also noiseless channel! This is an expression of the fact that a channel which is consistently in error can be as useful as a channel which is consistently correct. Definition. A channel described by a channel matrix with one, and only one, nonzero element in each row will be called a deterministic channel ‘Example 6-6. The channel matrix of a deterministic channel is given as shown in Figure 512 en Fioune 5-12, A deterministic channel, ‘Since there is but one nonzero clement in each row of a determinis- tie channel matrix, and the sum of the elements in each row must, be 1, the elements of a deterministic channel matrix are all either orl The mutual information of the types of channels defined above can be easily ealeulated. First, consider a noiseless channel. In a noiseless channel, when we observe the output bj, we know with probability 1 which a; was transmitted—that is, the conditional probabilities P(a./6,) are all either 0 or 1. Now, we write the equiv- cation H(A/B) as HA/B) = FPO) Y POM eazy — 649) 7 2 CHANNELS AND MUTUAL INFORMATION. 113 and we note that all the terms in the inner summation (being either of the form 1 X log 1 or 0 X log j) are zero, Hence, for a noiseless channel, H(A/B) =0 6-50) ‘This conclusion is also evident in view of the generalization of Shannon's first theorem (Section 5-5). ‘The outputs of a noiseless ‘channel are sufficient by themselves to specify the inputs to the channel. Hence, the average number of binits necessary to specify the inputs when we know the outputs is zero, From (5-30) we see that, for a noiseless channel, 1(A; B) = H(A) (6-51) ‘The amount of information transmitted through such a channel is equal to the total uncertainty of the input alphabet. For deterministic channels, we may derive set of analogous results, In a deterministic channel, the input symbol a; is sufficient to determine the output symbol b, with probability 1. Hence, all the probabilities P(,/a) are either 0 or 1, and H(B/A) = Y PCed) Y Ps/0) toe BGuyas -o 52) ‘On using (5-37), we have, for a deterministic channel, 1(A;B) = HB) 63) BEF, Comoded Chennels Some interesting properties of entropy and mutual information are revealed by consideration of the cascade of two channels (Figure 5-13). [A detailed investigation of eascades of binary channels is provided by Silverman (1955).) A 2 ‘Chane © (Channa 2 Frou 5-13, The cascade of two channels, We assume that a channel with an r-symbol input alphabet 4 and an saymbol output alphabet B is eascaded with a second ‘channel, as indicated above. The input alphabet of the second114 INFORMATION THEORY AND CODING. ‘channel is identified with B, and its output alphabet, consisting of ¢ symbols, is denoted by C. ‘The fact that the alphabets are connected by the cascade of Figure 5-13 implies certain relationships among the symbol prob- abilities. When a, a symbol from A, is transmitted, the output of the first channel is some symbol from B, say b;. In turn, by produces an output ci from the second channel. ‘The symbol c ‘depends on the original input a, only through by. In act, if we know the intermediate symbol}, the probability of obtaining the terminal symbol e depends only upon 6,, and not upon the initial symbol 4a, which produced b,. This property of eascaded channels may be written as Pla/by, a) = Ples/b)) for all 4, j, & (6-54) Indeed, (5-54) may be taken as the definition of what we mean by the cascade of two channels. A direct application of Bayes’ rule to (5-54) yields a similar equation in the reverse direction: Plai/b,, +) = Pla/by (5-55) It should be emphasized at this point that (5-54) and (5-55) hold only in the special ease where A, B, and C are the alphabets of cascaded channels, as indicated in Figure 5-13. ‘As we transmit information through cascaded channels from A to B to C, it seems plausible that the equivocation should increase— that is, H(A/C) should be greater than H(A/B). Let us investi- gate this question. H(A/C) — H(A/B) = re: 0 lee raray - 2 r@p lor pa 1 P(a, b, €) 108 (a7) is Plaje) Pla, 6,0 lor pray y Pla/b) Be P(a/e) CHANNELS AND MUTUAL INFORMATION 115 ‘Now we use (5-55) in (5-56); Pla/b, ©) H(A/C) — H(A/B) = PCa, b, ©) low pares SFG: g Pla/b,e) og, 2 Pe, DY Poly oe Piajey 57) We may use the inequality (2-8a) to show that the summation over the A alphabet in (5-57) is nonnegative. Hence, H(A/C) ~ H(A/B) > 0 H(A/C) > H(A/B) An immediate consequence of (5-59) is (A; B) > (A; 0) (5-00) These useful inequalities were apparently first proved by Woodward (0955). ‘They show that information channels tend to “leak” information, ‘The information that finally comes out of a cascade of channels can be no greater than the information which would emerge from an intermediate point in the easeade if we eould tap such a point. ‘The condition for equality to hold in (5-59) and (5-60) is of some interest. Retracing the steps in our proof to (5-50), we see that the equality holds if, and only if, P(a/b, ©) = P(a/e) (5-610) for all a symbols and all } and c symbols, such that PG, ¢) #0. Equivalently, the condition may also be written as P(a/b) = P(a/e) (5-610) for all a symbols and all b and ¢ symbols, such that P(, ¢) 0. ‘The condition for equality deserves some comment. At first lance, it might appear that the equality would apply if, and only if, ‘the second channel in the easeade of Figure 5-13 were noiseless. If this channel is noiscless, itis not hard to check that our condition (6-616) will apply. ‘That condition, however, will also apply in ‘other circumstances, as the next example shows.116 INFORMATION THEORY AND CODING rte 61. Tah cmb ch Gad Loo Gil 044. A channel diagram of the cascade is shown in Figure 5-14, Fioune 5-14. Cascaded channels, In spite of the fact that nether channel ie noiseless, it may be ween thet (6-618) holds, and, hence, MA; B) = HAs) In this example, (6-618) holds for any assignment of probabilities over the input alphabet 4. Tes also possible to find eases where (6-616) hol only for some specific input disteibution. We shall go into thie point further in the next section, We can illustrate the loss of information as it flows through a cascade of channels by employing people as information channels. A message, originally written in English, is translated into another language, and then translated back into English by sceond translator who has not seen the original message. ‘The result of this process will be a corrupted version of the original meseage, and may be thought of as the result of passing a message through a noisy channel. In order to simulate cascade of channels, we repeat the process, this time, however, starting with the corrupted version of the message, CHANNELS AND MUTUAL INFORMATION 117 ‘The experiment just described was performed using a simple four-line poem, The Turtle, by Ozden Nash, ‘The poem was trans- lated from English to French to English to German to English to Spanish to English. No effort was made to retain the rhyme or meter of the original piece. ‘The turtle lives "twixt plated decks Which practically conceal its sex. it clever of the turtle ivabassionaniene, ‘The output of the English-French-English channel was ‘The turtle lives in a scaled carapace which in fact hides its sex. 1 find that it is clever for the turtle to be so fertile in such a tricky situation, ‘The output of the English-German-English channel was ‘The turtle lives in an enclosed shell under which, in reality, it hiidesitssex. 1 find that the turtle must be very clever, indeed, to be so fertile in such a tight situation. Finally, the output of the English-Spanish-English channel was ‘The turtle lives inside a closed shell, under which, really, it hridesitesex. I feel the turtle had to be certainly elever to be so fertile in a so tight situation. ‘The noisiness of the human communication channel and the resulting loss of information have been recognized for some time, ‘Thueydides, in Book I of ““The Peloponnesian War,” states: Of the events of war, Ihave not ventured tospeak from any chanee informa tion, nor according to any notion of my own [i.e., a priori probabilities); T have described nothing but what I saw myself, or learned from others ‘of whom I made the most careful and particular inquiry [i.e., noiseless channels}. The task was a laborious one, because eyewitnesses of the ‘same occurrence gave different accounts of them, as they remembered or were interested in the actions of one side or the other [i., noisy channels). As a final (but more quantitative) example of the loss of informa- tion in cascaded channels, we consider the ease of the cascade of two identical BSCs,118 INFORMATION THEORY AND CODING ‘Brample 6-8. Two TSC, each with channel matrix Us] fm) ‘The two possible inputs to the first BSC are chosen with equal probability. Hence, from (5-48) we have 1A; B) = 1 ~ Hp) (5-62) tis easy to show that the eascade of these BSCs is equivalent toa single BSC with probability of error 2pp, Hence, 1A; © = 1 — Hepp) (563) another identioal BSC is addled (with output alphabet D), we obtain 1A; D) = 1 — HGptp +9) (564) ‘These curves are plotted in Figure 5-15. are cascaded as follows i lo 02040608 id Single chanel ror probability 9 Frovws 6-15. Mutual information of the cascade of m BSCs. (Input symbols are assumed equally probable.) 5-10. Reduced Channels and Sufficient Reductions In many types of information channels encountered in real life, the set of channel outputs is far larger than the user would like. For example, scientific data relayed from a satellite via a binary ‘CHANNELS AND MUTUAL INFORMATION 119 telemetry channel often contains information irrelevant to the pri- mary phenomenon under investigation. The antenna on the earth in such a gystem might obtain a sequence of pulses of various amplitudes. The receiver would take each pulse and, if its ampli- ‘ude is greater than some threshold, interpret the pulse as 9 1"; if ‘the amplitude is less than the threshold, the receiver interprets the Pulse asa “0.” We may think of two different channels in the situation just described. First, there is the channel with binary inputs (sent from the satellite) and a large number of outputs (Corresponding to the number of distinguishable pulse amplitudes). Second, there is the channel with binary inputs and binary out- puts (corresponding to the outputs of our receiver). This second channel is clearly a simplification of the first channel; we eall the ‘second channel a reduction of the first, Definition. Consider a ehannel with r inputs and ¢ out- puts deseribed by a channe! matrix P. Pu Pass Pu Puss pa[Ps Pa oo: Py Pass - Pa Pas++ Pa Pr We define a new channel with r inputs and s ~ 1 outputs by adding together any two columns of P. We eall the channel matrix of the new ehannel P’. Pa Pa Put Pigs +> Pe Pa Pao- + Put Pais 07> Pu Pa Bae Pat Pri *- > Pr ‘The new channel P’ is called an elementary reduction of P. We may repeat this process a number of times, forming an elementary reduction of P’, ete. The end product of more than one elementary reduction will be called simply a reduction of the original channel P, Bxample 6-9, In Example 5-1 we formed the channel matrix of the (BSC)*: pp mp ot pale Poy pp pop pt pp pp ppp120 INFORMATION THEORY AND CODING CHANNELS AND MUTUAL INFORMATION 121 ‘An elementary reduction of P is formed by combining the first and second An channel?” In order to answer this question, we need only eonsider opp wt pa|? ot the ease of elementary reductions. ‘The question in the case of a general reduction may then be answered by indus ‘Lot us form an elementary partition of the channel PP pp > mp Pu Pu A reduction of Pis formed by combining the second and third elumns of P's pu|?s Px (5-67) bale Without loss of generality, we may assume that the elementary partition is formed by combining the first two columns of P. ‘This situation is described in the channel diagram shown in Figure 5-17. »r |? > of PP > A useful way of viewing a reduced channel is as shown in Figure 5-16, ‘The deterministic channel combines symbols of the B alpha- ‘anne! Fioone 5-16, A reduced channel bet into a smaller number of symbols of the C alphabet. Hence, the channel with input alphabet and output alphabet ( indicated by dashed lines in Figure 5-16 is a reduetion of channel P. This ‘method of constructing a reduced channel allows us to use the results of the previous section on channel cascades, In particular, we have (referring to Figure 5-16) é Ficums 8.17. Channel reduction by cascade. In Section 5-0 we found necessary and sufficient conditions that a ‘eascade not lose information. ‘These were {(5-610)] H(A/) > H(A/B 5-65) es pe Ce ore P(a/b) = Pla/e) (6-08) HA; $ KA;B) 6-60) forall ab, and e symbols, such that Forming a reduction of a channel decreases (or at best leaves P(b, 0) #0 unchanged) the mutual information of the input and output alphabets, This is the price we pay for simplification in the channel. ‘A most important question suggested by the above remarks is, “When can we simplify the channel without paying a penalty in reduced mutual information?” ‘That is, “When is the mutual information of a reduced channel equal to that of the original Since we are investigating an clementary reduction, this condition is satisfied trivially for all B symbols except the two symbols we have combined, b; and b;, Let ¢ be the C symbol formed from band bs. On applying (5-68) to b, and bs, we find that the neces- "sary and sulficient conditions are P(a/by) = P(a/e:) = P(a/bs) foralla (5-69)122 INFORMATION THEORY AND CODING This is equivalent tot P(a/bi) = P(a/bs) for alla 6-70) In other words, the two output symbols by and bs may be combined without loss of information if, and only if, the backward probabilities P(a/b) and P(a/bs) are identical for all a. This is an important result, both in terms of understanding information and from a practical point of view. It provides conditions under which a channel may be simplified without paying a penalty, The back- ward probabilities, however, depend upon the a priori probabilities P(aq); ie, they depend upon how we use our channel, It is of even more interest to determine when we may combine channel ‘outputs no matter how we use the channel, ie., for any a priori probabilities, ‘This may be done by using Bayes’ law to rewrite _Poy/a\P(@)_ __Plbs/a)P(a) yr »/a)P(a) ¥ P(bs/a)P(a) foralla (5-71) P(bi/a) _ PPovar@ Plbs/a) ~ ¥ P(b,/a)P(a) for all a (6-72) If (5-72) is to hold for all possible a priori probabilities P(a), we must have P(bs/a) = const X P(bs/a) for all a (5-73) Equation (5-73) is the condition we seek. If we have a channel matrix satisfying (5-73), we may combine two columns of the matrix, and the new channel matrix will be just as good as the original one. More precisely, for any set of probabilities over the input alphabet, the mutual information of the channel and the reduced channel will be identical. A reduced channel with this Property will be called a suficient reduction Example 6-10. The channel itd Gidl 4 The condition on P(a/e,) follows automatically from (5-70). may be reduced to CHANNELS AND MUTUAL INFORMATION 123 al ‘This channel is « sufisiont reduction of the original channel. ‘and finally to 5-11. Additivity of Mutual Information Another important property of mutual information is additivity, In this seetion we investigate additivity by considering the average amount of information provided about a set of input symbols by a succession of output is, we consider the ease where ‘we may gain information about the input by a number of observa- tions, instead of a single observation. An example of this situation ‘occurs when the input symbols of a noisy channel are repeated a number of times, rather than transmitted just once. Such a procedure might be used to improve the reliability of information transmitted through an unreliable channel. Another example is aan information channel where the response to a single input is a sequence of output symbols, rather than a single output symbol We investigate the additivity property of mutual information in the special case where the output for a single input symbol consists of two symbols. The more general case where the output consists of n symbols may then be treated by induction, Let us modify our model of the information channel, then, so that instead of a single output for each input, we receive two symbols, say b, and ci. The symbols b, and ¢, are from the output j=1,2,..., 8, 0nd C= fa], b= 1, 2, ae, e Without loss of generality, we may assume that the two output symbols are received in the order be. Then the a priori probabili- ties of the input symbols P(a,) change into the a posteriori prob- P(ai/>,) upon reception of the first output. symbol; upon Teception of the second output symbol, they change into the “even more a posteriori” probabilities P(a/b,, c.). If the two symbols b, and c, are received, the average uncertainty ‘or entropy of the set of input symbols changes from HA) = ¥ P00) 8 75 (6-740)124 INFORMATION THEORY AND CODING to the a posteriori entropy WA/b) = J Pla/t) toe rare and then to the “even more a posteriori” entropy’ H(A/by 04) = J Pla/by cs) on py (STAC) - PCy, &) As in Section 5-5, we average H(A/b,) over the b, to find the average a posteriori entropy, or the equivoeation of A with respect to B Y PO)H(A/») = W(A/B) (6-750) In the same manner, we may average H(A/by, ex) over all by and cx in order to find the equivocation of A with respect to B and & P(b, ©) H(A/b, ¢) = H(A/B, C) (5-750) ‘The results of our generalization of Shannon's first theorem (Sec- tion 5-5) apply direetly to H(A/B, C); H(A/B, C) is the average number of binits necessary to encode a symbol from the A alphabet alter we are given the corresponding B and C symbols, Equations (5-75a) and (5-75) suggest two different ways we might measure the amount of information B and C yield about A—the mutual information of (B,C) and A. First, we might define the mutual information of A and (B,C), just as we did when the channel output consisted of a single symbol. That is, 1(A; B, C) = H(A) — H(A/B, C) (6-76) Second, we might consider the amount of information provided about A by B alone, and then the amount of information about A provided by C after we have seen B, ‘These quantities are H(A) — W(A/B) 170) and H(A/B) ~ H(A/B, C) (6-70) ‘The first of these has already been defined as 1(A; B) = H(A) ~ H(A/B) (6-780) ‘CHANNELS AND MUTUAL INFORMATION 125 It is natural to define (5-77b) as 1(A; C/B) = H(A/B) ~ H(A/B, ©) 6-780) called the mutual information of A and C, given B. Upon ad (6-78a) and (5-78b), we find (Aj B) + 1(4; C/B) = H(A) — H(A/B, ©) (A; B, C) (6-79) Equation (5-79) expresses the additivity property of mutual information. It says that the average information provided by an ‘observation does not depend upon whether we consider the observa- tion in its entirety or broken into its component parts. This equation may be generalized immediately to I(A;B, O, . ..,D) = WA; B) + 1A5C/B) + + + 1(A; D/B,C,...) (6-80) where the term on the left is the average amount of information about A provided by an observation from the alphabets B,C, .. ., D. The first term on the right is the average amount of informa- tion about A provided by an observation from the alphabet B. ‘The second term on the right is the average amount of information about A provided by an observs from the alphabet C after an observation from the alphabet B, ete. ‘The particular order of the information we receive is, of course, irrelevant for (5-79) and (5-80). For example, we may write [cor- responding to (5-79)] I(A; B,C) = 1(A; C) + 1(A; B/C) (81) We may write the information quantities discussed above in several different forms. From (5-76) we have (As B, ©) = H(A) ~ H(A/B, €) J Peo) toe pigs — YS Plot oboe parE sy ae = y PC, b Nos peas = y PCa, b 2108 pari ae 7 J Pla, blog OY 2) ae (6-82a)126 INFORMATION THEORY AND CODING Another useful form is found by multiplying the numerator and denominator of the logarithm of the above equation by P(b, ¢) PCa, b, 6) Pore Fe WA; B,C) = y P(a, b, €) log ‘The reader should note the similarity of (5-82a) and (5-82h) to (5-312) and (5-31). We might have obtained (5-822) and (5-825) merely by replacing b in (5-B1a) and (5-318) by (6, ¢). This argu- ment suggests the definition HB, C/A) i PCa, b, €) log py It is easily verified that I(A; B, C) = H(B, ©) — H(B, C/A) (5-84) vj 8 . To illustrate the additivity of mutual information, we ee ce ppl ott where p 1 —p. This time, however, we sscume that the input symbol (Gither 0 oF 1) is repeated, so that the output of the channel consists of two binary symbols b,, es for each input symbol a. Furthermore, for simplicity ‘we amume that the two inputs are chosen with equal probabilities. There fore, eotting w= ) in (5-48), we find MA; B) = 1 ~ Hip) (6-85) To find 1(4; B, C), we use (6828). Table 5-2 gives the necessary probabilities. ‘Teoux $2, Paonannaries or 4 Revere BSC he Pha) Play byes) Plby od) woot ip op pop wept wo pep ip ip ieee CHANNELS AND MUTUAL INFORMATION 127 Using these probabilities im (6-828) yields 1; 2,0) = wg 2B P sotm[i-n(sts)] oso ‘The interpretation of (5-86) is clear. If we observe the outputs 10 or O1 from such a channel, its meaning is entirely ambiguous; the two possible inputs will still be equally probable, sad we gain no information from our observation. Ton the other hand, we obterve O0 oF 11, wo gain information about the input ‘equivalent to that gained by observing a single output from a BSC with error probability ee ae Pre ‘From (6-85) the information of such an observation is 1-855) ‘We observe cither 00 or LI with probability p* + p%; hence (5-80), ‘The arguments given above are easily generalized to the case of a BSC used with more than # single repetition. For example, if each input produces three binary outputs, we obtain 148.6.) = + P)[1 0 (Bem) | +3908 — Ho) sn Equations (585), (6-86), and (5-88) are plotted in Figure 5-18 5-12. Mutual Information of Several Alphabets In our investigation of the additivity of mutual information in Section 5-11, we were led to a consideration of the sequence of entropy quantities. H(A) H(A/B) H(A/B, ©)128 INFORMATION THEORY AND CODING ‘Mutual ntermaton 1A; 002 a8 “06 os Probability of ror p Frovns 6-18, Mutual information of a BSC with w repetitions. Each member of this sequence is no greater than the preceding member. We saw that the difference between two successive mem- bers could be interpreted as the average information about A provided by a new observation. (A; B) = H(A) ~ H(A/B) (6-900) 1(A; C/B) = H(A/B) ~ H(A/B, ©) (5-006) I(A; B) is the mutual information of A and B; I(A; C/B) is the mutual information of A and C after we are given B. Both these quantities, however, involve the mutual information of just two alphabets, It is also possible to define the mutual information of more than two alphabets (McGill, 1954). We define the mutual information of A, B, and C by 1(A; B; C) = 1(A; B) ~ 1(A; B/C) (6-910) Our definition of the mutual information of A, B, and C implies that 1(4; B; C) is symmetric in 4, B, and C. If this is true, CHANNELS AND MUTUAL INFORMATION 129 (6-912) can also be written (Aj B; C) = 1(B; C) ~ 1(B; C/A) (6-910) 4(C; A) — 1(C; A/B) (6-91) To prove the symmetry of I(4; B; C), we write (5-91a) as 1A: B:0) = J Pa) tox BS f P(a, b/c) = J, P68 9 6 pe/eP Or) P(a, b)P(o/e)P(o/0) Pe PCG, by 108 p@PO)P Ca, b/2) PCa, b)PLo, PO, & a PCa by €) 108 BES PEYPLE)PCG, b, 6) = H(A, B, ©) ~ H(A, B) ~ H(A, ©) ~ H(B, 6) + H(A) + HB) + HO) G92) In addition to exhibiting the symmetry we want, (5-92) is reminis- cent of the expression for the mutual information of two alphabets: 1(A; B) = H(A, B) — H(A) — H(B) (6-93) It is easy to generalize (5-92) and (5-03) to more than three alpha- bets. For example, the mutual information of A, B, C, and D is (A; B; 0; D) = 1(A; B; ©) — 1(A; BC/D) = [H(A, B, C, D)) = [(A, B,C) + H(A, B, D) + H(A, C, D) + H(B, C, D)] + (H(A, B) + H(A, ©) + H(A, D) + H(B, C) + H(B, D) + H(C, D)] ~ [H(A) + H(B) + H(C) + H(D)) (6-94) Blachman (1961) has suggested a generalization of Figure 5-9 to id in the interpretation of the above expressions. For three alphabets, we have the relationships shown in Figure 5-19.130 INFORMATION THEORY AND CODING Although Figure 5-19 is an important aid in remembering relationships among the quantities we have defined, it ean also be somewhat deceptive. The mutual information (4; B) was shown to be nonnegative; the mutual information 1(A; B; C), however, HALO, HA,B.O, eT Oe @ o ey , © @ Fravas 5-19, Some information relationships. ean be negative, ‘This means that the intersection of the three circles of Figure 5-194 ean be negative! ‘To show this we present an example, [Bxample 6-12. Consider the three binary alphabets A, B,C. Let a.and by be selected ax 0 or 1, each with probability } and cach independently of the ‘other. Finally, we assume that cris selected as 0 if ay equals by and as 1 if a CHANNELS AND MUTUAL INFORMATION 137 doce not equsl by. Some of the probabilities of these three random variables are given in Table 53. ‘Tanux 5:3. Pronanttsrins oF Tumse Raspoxt VARIARES abi Plan dyed) Player) Plaw bylea) Plas, b) Pe) Se 00 + 01 oo ou 100 10 0 um SUES Sa eee ‘Using this table, we ealeulate 1A; B) = 0 bits 1A; B/C) = bit MA; B; C) = HA; B) ~ (A; B/C) = ~1 bit This dear why we get such an answer, Since A and B are statistically indo- pendent, (4; B) = 0, and B provides no information about A. If we already Prow C, however, earning B tells us which A was chosen, and therefore pro- ‘vides us with one bit of information, 5413, Channel Capacity Consider an information channel with input alphabet , output alphabet B, and conditional probabilities P(},/a). In order to ‘calculate the mutual information 104; B) = YPC) tog PSP 95) it is necessary to know the input symbol probabilities P(a,). The ‘mutual information, therefore, depends not only upon the channel, Dut also upon how we use the channel—ie., the probabilities with which we choose the channel inputs. It is of some interest to exam- ine the variation of (A; B) as we change the input probabilities132 INFORMATION THEORY AND CODING Example 6-18. For the BSC with probability of error p, we found [(5-48)) 1A; B) = Hwp +p) — He) 6-00) where w is the probability of selecting a 0 at the input.and 2 = 1 — w, 1 =p. We may plot (6-06) as a function of w for a fixed 7 (se Figure 820), Honce, the mutual information of x BSC varies from 0 to 1 — Hip), The Tn these eases, the input is known with probability 1 st the output, even before an output symbol is ‘minimum value of 0 is achieved when w = 0 or 1 received. ‘The maximum value of 1 ~ H(p) ie achieved when w = 4 that ie, ‘hen both inputs are equally probable, belie Mutual information 1; 0 t 1 Probably of3°0” atiaputas Fravxs 5.20. Mutual information of « BSC. For a general information channel, we see that the mutual information can always be made 0 by choosing one of the input symbols with probability 1. Since the mutual information is non- negative, this is an easy answer to the question of what is the minimum value of /(4; 8). ‘The question of the maximum value of 1(A; B) for a general channel is not so easily answered, however. The maximum value of 1(4; B) as we vary the input symbol Probabilities is called C, the eapacity of the channel: ce ) Note that the capacity of an information channel is a funetion only of the conditional probabilities defining that channel. It does not depend upon the input probabilities—how we use the channel. ly From Figure 5-20 we see that the capacity of a BSC with error probability p is 1 — H(p) — CHANNELS AND MUTUAL INFORMATION 133 ‘The calculation of the eapacity of an information channel is, in mneral, quite involved (Muroga, 1953; Shannon, 1957b; Fano, 1961). Th eerain case, however, the calculation can be simplied. The 10 oa} 08 og pe 4} 4 ‘dl loa lo 0 02 00 08 08 18 Probability of erorp Fiouns 5-21. Capacity of a BSC. most important class of channels for which the calculation simplifies is the class of uniform channels. Definition. Consider a channel defined by the channel matrix Pu Pa Pw Pu Px Ps Fa Pa Pr, As before, Py = P(a/b,). This channel is said to be ‘uniform if the terms in every row and every column of the channel matrix consist of an arbitrary permutation of the terms in the first row. ample B14, We have aleady dst with one camps of wiorm infor imation shann-the BG. “The safure!genalnaon of he DSC, ey134 INFORMATION THEORY AND CODING symmetric channel (PSC), is uniform channel with r input and r output aym- Dols. The channel matrix of the rSC is shown in Figure 6-22. Fa Fai Fi Fioune 5-22. Channel matrix of the r8C. ‘As usual, p = 1 — p. The overall probability of error for this channel is p, Dut there are r ~ 1 possible incorrect output aymbols for each input symbol ‘We now calculate the eapacity of a general uniform channel ‘The capacity is the maximum of 1(A; B) as we vary the input distribution. I(A; B) = H(B) — H(B/A) = H@) ~ J Pe) S PO/a) ox pera, (98) The summation over B in the last term of (5-98) is a summation, for each a, of the terms in the ith row of the channel matrix, For uniform channel, however, this summation is independent of i Hence, 1A; B) = WB) ~ J PO/0) oe pera5 (6-99) 5 and the last term of (5-99) is independent of the input symbol distribution. ‘To find the maximum of the right side of (5-99), we need only find the maximum of H(B). Since the output alphabet ‘consists of r symbols, we know that /1(B) cannot exceed log r bits. H(B) will equal log r bits if and only if all the output symbols occur with equal probability. In general, it is not true that there exists a distribution over the input symbols such that the output symbols are equiprobable. For uniform ebannel, however, it is easy to check that equiprobable symbols at the input produce equiprobable symbols at the output. Therefore, the maximum value of (5-09), the capacity of the uniform channel, is CiUNHES AND MUTUAL MFORANON 18 @ = bogr — J Poe py = logr + Y P(b/a) log Pb/a) // (6-100) ‘Using (5-100), we ealeulate the capacity of the r8C: coker eeu perenne = log r ~ p log © — ) - Hip) 101) T 5:14, Coneitonal Mutvl Information ‘The channel capacity is the maximum value of 1A; B) = Y Plat) tog (6-102) the average of log [P(6/a)/P(O)] over both the input alphabet A and the output alphabet B. The mutual informstion may also be written : Pib/a) (A; B) = YP (b/a) log OE = > P(a)I(a; B) (5-103) where we have defined 119; B) = ¥ P/a) on (108) I(a; B) is called the conditional mutual information (conditional on a). The conditional mutual information is the average of log [P(b/a)/P(b)] with respect to the conditional probability P(b/a). In general, /(a; B) depends on the input symbol a. When the input symbols are selected according toa set of probabilities which achieve channel capacity, however, we shall show that J(a; B) does ‘not depend upon a for any input symbol with P(a) #0. When the input probabilities are selected so as to achieve capacity, (a; B) = C (6-105) for all a such that P(a) ¥ 0.136 INFORMATION THEORY AND CODING This fact is central to the calculation of the channel eapacity for channels more general than the uniform channel treated in the previous section (Fano, 1961). We shall use this fact when we prove Shannon’s second theorem in Section 6-10. ‘We prove (5-105) by contradiction. That is, we assume we have ® set of input probabilitiest P(a,), P(a),..., P(a,) which achieve channel eapacity, but for which (5-105) does not hold, and all the (a; B) are not equal to the capacity. Since the average of 1(a; B) does equal the capacity, there must be at least one 1(a; B) greater than C and at least one Z(a; B) less than C. With- out loss of generality, we assume 1a; B) > C (6-106a) Mas; B) <@ (6-106) Now we change the probabilities P(ax), Pas), Pla), . . . , Pla) (5-107a) to Play) +4, Pa) — 4, Pla), ... , Plas) (6-107) where A is some small positive number less than P(a;), and show that the value of the mutual information increases. Since the ori nal probabilities (5-107a) were assumed to achieve capacity, this is a contradiction; henee, our assumption that all the I(a; B) were not constant must have been in error, Let us denote the new probabilities in (5-107b) by Py(a,), Px(as), , Pi(a,). The corresponding output probabilities will be writ ten Pa(hs), Pulls), . . . , Pr(bs). ‘The P,(b) are given by Px) = ¥ P(@PO/a) = P(b) + A[P(b/a) — P(b/as)) (5-108) Let 1,(Aj B) be the value of the mutual information ealeulated using the probabilities P,(a); by assumption, the value of the mutual information using the original probabilities P(a) is C, the channel capacity. We calculate We assume that none of the P(a,) are equal to aero. If P(a)) = Owe may ‘consider a new channel derived from the old channel by deleting the input cy CHANNELS AND MUTUAL INFORMATION 137 1A; B) — © = J Pua) Y 0/2) og He ice = Fre) ¥ Pe/a) oe “64 -s[J P6b/a) log PO) ~ J Peve9 X log P(6/as)] + J Ps) low p55 ie J 70 Pe) 1%) Upon adding and subtracting [J PO /a oe pi; — YPO/a) lox pay] 6-110 on both sides of (5-109), we obtain (v/a 1(4;B) ~ C= 4[¥ P@/ay) log wn >(o/as P) = J Perey on His?) + FP) 1 Pry = a[ Mes; B) ~ Has; B)] £5 2.0) 0029 oy We wish to show that the right side of (5-111) is positive, in order to arrive ata contradiction. ‘The first term on the right of (5-111) is positive by (5-106). ‘The second term, however, must be negative by our often used inequality (2-80). Henee, superially, it appears ‘as if we cannot draw any conclusions about the sign of the right side of (S111). Such pessimism is not warranted, however, as we shall now demonstrate by examining the last term of (5-111) in more detail. J P.O) toe Foy = DAPO + AP O/a ~ PO/as} 1 X log alP 7a) = Pile P(b) 112)INFORMATION THEORY AND CODING For x small enough, we may approximate lo; For enough, we may approxi og (1/(1 + 2)] by ao 2. Using this fact in (5-112), we see that for 4 sufficiently. Po) _ 1 y Pi) oe py = md {PO + AlP@/a) — PO/a)}} x APO/a = PO/ed) fea 5 P/a0 ~ PO/a91 — gs J PO/e) — Posane Por =a" y [PO/a) ~ PO/ad mz) poe (113) since J P(/a;) = J P(d/as) = 1. Thus, the second term (a negative quantity) of (5-111) goes as a* for A sufficiently small anes heft tra ( & positive quantity goes as; by making A sufficiently small, the right side can be made positive, our contradiction. oe ee tee ‘The assumption that not all the conditic in tional mutual informations (a; B) equal the capacity must be incorre ier ty st be ime ‘et, and we have proved NoTES Mat oo ans ee La ee ee es ae ee and outputs defined in Section 5-1 ioe ea Poo P(-/a) on B for each ain A, Panes East eared cna Ta"

You might also like

• The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  

  From Everand

  The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  Mark Manson

  Rating: 4 out of 5 stars

  4/5 (5813)
• The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  

  From Everand

  The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  Brene Brown

  Rating: 4 out of 5 stars

  4/5 (1092)
• Never Split the Difference: Negotiating As If Your Life Depended On It

  

  From Everand

  Never Split the Difference: Negotiating As If Your Life Depended On It

  Chris Voss

  Rating: 4.5 out of 5 stars

  4.5/5 (844)
• • Magazines
  • Podcasts
  • Sheet music
• Principles: Life and Work

  

  From Everand

  Principles: Life and Work

  Ray Dalio

  Rating: 4 out of 5 stars

  4/5 (608)
• The Glass Castle: A Memoir

  

  From Everand

  The Glass Castle: A Memoir

  Jeannette Walls

  Rating: 4.5 out of 5 stars

  4.5/5 (1716)
• Grit: The Power of Passion and Perseverance

  

  From Everand

  Grit: The Power of Passion and Perseverance

  Angela Duckworth

  Rating: 4 out of 5 stars

  4/5 (590)
• Sing, Unburied, Sing: A Novel

  

  From Everand

  Sing, Unburied, Sing: A Novel

  Jesmyn Ward

  Rating: 4 out of 5 stars

  4/5 (1104)
• Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  

  From Everand

  Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  Margot Lee Shetterly

  Rating: 4 out of 5 stars

  4/5 (897)
• Shoe Dog: A Memoir by the Creator of Nike

  

  From Everand

  Shoe Dog: A Memoir by the Creator of Nike

  Phil Knight

  Rating: 4.5 out of 5 stars

  4.5/5 (540)
• The Perks of Being a Wallflower

  

  From Everand

  The Perks of Being a Wallflower

  Stephen Chbosky

  Rating: 4.5 out of 5 stars

  4.5/5 (2104)
• The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  

  From Everand

  The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  Ben Horowitz

  Rating: 4.5 out of 5 stars

  4.5/5 (348)
• Bad Feminist: Essays

  

  From Everand

  Bad Feminist: Essays

  Roxane Gay

  Rating: 4 out of 5 stars

  4/5 (1018)
• Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  

  From Everand

  Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  Ashlee Vance

  Rating: 4.5 out of 5 stars

  4.5/5 (474)
• Her Body and Other Parties: Stories

  

  From Everand

  Her Body and Other Parties: Stories

  Carmen Maria Machado

  Rating: 4 out of 5 stars

  4/5 (822)
• The Outsider: A Novel

  

  From Everand

  The Outsider: A Novel

  Stephen King

  Rating: 4 out of 5 stars

  4/5 (1850)
• The Emperor of All Maladies: A Biography of Cancer

  

  From Everand

  The Emperor of All Maladies: A Biography of Cancer

  Siddhartha Mukherjee

  Rating: 4.5 out of 5 stars

  4.5/5 (271)
• The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  

  From Everand

  The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  Viet Thanh Nguyen

  Rating: 4.5 out of 5 stars

  4.5/5 (122)
• Angela's Ashes: A Memoir

  

  From Everand

  Angela's Ashes: A Memoir

  Frank McCourt

  Rating: 4.5 out of 5 stars

  4.5/5 (441)
• Brooklyn: A Novel

  

  From Everand

  Brooklyn: A Novel

  Colm Toibin

  Rating: 3.5 out of 5 stars

  3.5/5 (1947)
• The Little Book of Hygge: Danish Secrets to Happy Living

  

  From Everand

  The Little Book of Hygge: Danish Secrets to Happy Living

  Meik Wiking

  Rating: 3.5 out of 5 stars

  3.5/5 (401)
• A Man Called Ove: A Novel

  

  From Everand

  A Man Called Ove: A Novel

  Fredrik Backman

  Rating: 4.5 out of 5 stars

  4.5/5 (4610)
• The Art of Racing in the Rain: A Novel

  

  From Everand

  The Art of Racing in the Rain: A Novel

  Garth Stein

  Rating: 4 out of 5 stars

  4/5 (4203)
• A Tree Grows in Brooklyn

  

  From Everand

  A Tree Grows in Brooklyn

  Betty Smith

  Rating: 4.5 out of 5 stars

  4.5/5 (1929)
• The World Is Flat 3.0: A Brief History of the Twenty-first Century

  

  From Everand

  The World Is Flat 3.0: A Brief History of the Twenty-first Century

  Thomas L. Friedman

  Rating: 3.5 out of 5 stars

  3.5/5 (2259)
• Steve Jobs

  

  From Everand

  Steve Jobs

  Walter Isaacson

  Rating: 4.5 out of 5 stars

  4.5/5 (807)
• The Yellow House: A Memoir (2019 National Book Award Winner)

  

  From Everand

  The Yellow House: A Memoir (2019 National Book Award Winner)

  Sarah M. Broom

  Rating: 4 out of 5 stars

  4/5 (98)
• Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  

  From Everand

  Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  Gilbert King

  Rating: 4.5 out of 5 stars

  4.5/5 (266)
• A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  

  From Everand

  A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  Dave Eggers

  Rating: 3.5 out of 5 stars

  3.5/5 (231)
• Team of Rivals: The Political Genius of Abraham Lincoln

  

  From Everand

  Team of Rivals: The Political Genius of Abraham Lincoln

  Doris Kearns Goodwin

  Rating: 4.5 out of 5 stars

  4.5/5 (234)
• Yes Please

  

  From Everand

  Yes Please

  Amy Poehler

  Rating: 4 out of 5 stars

  4/5 (1898)
• The Woman in Cabin 10

  

  From Everand

  The Woman in Cabin 10

  Ruth Ware

  Rating: 3.5 out of 5 stars

  3.5/5 (2521)
• Fear: Trump in the White House

  

  From Everand

  Fear: Trump in the White House

  Bob Woodward

  Rating: 3.5 out of 5 stars

  3.5/5 (738)
• Wolf Hall: A Novel

  

  From Everand

  Wolf Hall: A Novel

  Hilary Mantel

  Rating: 4 out of 5 stars

  4/5 (3811)
• John Adams

  

  From Everand

  John Adams

  David McCullough

  Rating: 4.5 out of 5 stars

  4.5/5 (2409)
• On Fire: The (Burning) Case for a Green New Deal

  

  From Everand

  On Fire: The (Burning) Case for a Green New Deal

  Naomi Klein

  Rating: 4 out of 5 stars

  4/5 (74)
• The Light Between Oceans: A Novel

  

  From Everand

  The Light Between Oceans: A Novel

  M.L. Stedman

  Rating: 4.5 out of 5 stars

  4.5/5 (789)
• Manhattan Beach: A Novel

  

  From Everand

  Manhattan Beach: A Novel

  Jennifer Egan

  Rating: 3.5 out of 5 stars

  3.5/5 (792)
• The Constant Gardener: A Novel

  

  From Everand

  The Constant Gardener: A Novel

  John le Carré

  Rating: 3.5 out of 5 stars

  3.5/5 (104)
• The Unwinding: An Inner History of the New America

  

  From Everand

  The Unwinding: An Inner History of the New America

  George Packer

  Rating: 4 out of 5 stars

  4/5 (45)
• Rise of ISIS: A Threat We Can't Ignore

  

  From Everand

  Rise of ISIS: A Threat We Can't Ignore

  Jay Sekulow

  Rating: 3.5 out of 5 stars

  3.5/5 (137)
• Little Women

  

  From Everand

  Little Women

  Louisa May Alcott

  Rating: 4 out of 5 stars

  4/5 (104)