THE MATHEMATICAL THEORY OF COMMUNICATION CLAUDE E, SHANNON WARREN WEAVER, Foreword by Richard E, Blahut and Bruce Hajole University f ins Press Urbana and ChicagoRECENT CONTRIBUTIONS TO THE MATHEMATICAL "HEORY OF COMMUNICATION By Warren Weaver1 Introductory Note on the General Setting of the Analytical Communication Studies’ 1.1. Communication ‘The word communication willbe ust here in 8 very broad senso to ialude all of the prosduree by which ope mind may afet nother, "Tis, of cous, involves not ouly written and ora fpeeeh, but also ruse, the pictorial ers, the theaze, the ballet, Tein fant al buman behavior, In some connections it ay be osreb to uee a till broader definition of eommunisstion, amely, ene which would include the procedures by means of ‘which ope meebenian (eay automatic equipment to trac an eplane snd to compute Ha probable futare positions) aos Shuther meebenitm (cay a guided mae chasing this airplane "The language of ie memeratdutn wil ellen appear to refer fo the special byt still very Dosd end Saportant, Held ofthe eom~ muniegtion of aptceh; but pracealy everything seid appli BERL ota maa oe emer ected oo REE EP Meet ic bea onsen Coat evom ute Pesos =)a 4 The Mathematica! Theory of Communication ‘ually well to musio of any srt and to sill or moving pictures, ‘a television, 112, Thee Levels of Communications Problems Relative tothe broad subject of communication, tore seem to be problems at thee levels. Thus it sms reasonable toa, serially Leven A. How accuretaly ean the eymbols of communication be tranwmitted? (The teshnical problem) Leven B. How precisely do the transite symbole convey the esired meaning? (The seme problem) Lares C. Mow olfectively doce the received meaning aot can duct ee dsied way? (The effectiveness problem.) ‘The technical problems are congeraed with the aceursey of ‘ronaference from sender to reeiver af sete of sytbole {wrth speech), oF of s continuously varying signal (aléphnie or radio teanumission of voioe or music, oF of «continuously varying tno lmessional pater (Ielevsion), ete, Mathematically, the frst involves transmission of finite set of dierte symbols, the second the tetnsmission of oe continuous faction of tine, and be hind the tranoision of many continuo function of time oof one continuous funtion of time and of two space oor Isistority cle approximation, inthe interpretation of mening by the resever, as compared with the intended meaning of the sender, This every deep and involved situation, even when one deals only with tho relatively simpler problems of communicating trough speed, ‘One extent! complication ia lueteated by the remark that if Mr. X is aupected not to understand what Mr. aay, thea fi ‘theoretically not posible, by having Mr. Y do nothing bt tale further with Me. X, completely to elaify this stuation in any finite time. If Mr, ¥ saye “Do you now understand met” and ‘Mr, X says “Cortinly, T do” thas ie nt necessarily 4 ert tion that understanding hae heen achieved. Tt may just be Ut Mr. X did ot wnderwtand the question. Il ths ounde ily, try Some Recent Contributions: Weaver 5 gna "Cay pat mnie rorunie?™ with the anewer “Hat Webiate gama” Fin tht ths ase diet yo at theres field of sesh ommend! to «tle she's (ner campy hae) by oan which (a) sve prosnebly bere ore tan approximations the ea beng exlnine, ut which (b) are andetandable since they re pared in language wich has previoely Been ude reusmaiy lar by operational ean, For eaope does aot take bmg to make the mba for "yes ny anges ope tally neta "The soma probe as wide sanctions If oo think of commuticoton in genera, Conte, or example th eating o {Hn of US nwa te ‘The efetocneprolons ae coer withthe sce with wich Ge meaning conveyed tothe recive lene othe deed Sonnet on is rust Tt many enn at fit lane aoenaly ner to ly thatthe prot of al communication son cece tho conf of the seuaver, Bt with any reasonably broad ettn of eon icles tat commenieaton eer ect ondvet ore wilout any crite aod probable fect st al "The problem ef efctivene involves aeteteconieatons inthe cao the fle art In the cof speech, wen roa, iChnvelveseonsderaton beh range all w way om te mare Inatnes af le, trough al the eveoloea! abd emotions topes of propaganda Deny hen Yale jdguete Which are teowny fe give wefulrnaning to the words “sme” and ned inte pein tere hk etn on elvtivenss “Te efeivencte prblen relly fred ith 2 rman pion, an oveape Ha rater vague war ad Sr a ecu ol erase Ee Lay, Sea ‘item rab n the elution of Mind” in Quarterly Review of Holy,° The Mathematical Theory of Communication there iin fat overlap between al ofthe suggested categories of problems So sated, one would be incline t thnk that Level Aisa rela tively supericial one, involving ony the engineering detsils of sve design of a ecmmunieation system; while B and Cssem to contin most i ot al ofthe phileeophical content ofthe general ‘problem of eottnusiestion, "The mathematical thory of the enginetring aspects of ea tmunleation, as developed chiy by Claude Shaan atthe Bel ‘elepone Laboratories, admittedly apples in the As ieiance only to problem A, namly, the technical problem of aecuracy of transference of various types of signal fom tender to recive, But te theory hs, T think, «doepsigaiicsace which proves that ‘tho preesding parageap ie seriusly inaccurate. Part of the sig nifleanoe ofthe new theory comes fom the fect that levels B and above, eta make use any of thove signal aceuscis which torn out to be possible when sualyzed at Level A. Thus any binita- tlons ascovered in the theory at Level A nesesarily apply to levels B and C. But a langer part ofthe sgnieance comes from the fact thatthe analyse at Level A dinelows that this evel over- laps the other levels more thn one could youuble naively sgpoc. ‘Thus the theory of Level Asst leat toa signieant degre, also theory of tevels and G1 hope thatthe mueweding paste of his memorandum wil tuminate and justify thse let ears, 2 Communication Problems at Level A 2.1, A Communication Systm ond Its Problems ‘Tho communication syetens condoned may be symbolically rep rented a follows: Some Racent Contributions: Weaver oO “Theinfrmation sone sets a desire mesoge oto st of pose rnnger (is partisary importer eat ch Fogle coorderab explanation Inter. The slcted oenage Iiay const of writen o paken wore oro ction mu, “Toe traenitr shang thie message into the signal which actully unt over the commanicton hanna frm oe tent torso recive, Tn the coe of tpn, the ehantel ie we theignal a Yerying seca! citent on is ‘wine the rane miter Ue set of devies (pone feapamiier te) eh Change the sound peste ofthe Vole ft the varying cesta arent In flegrphy, the trnumiter ede ween words ina sequence of ineruped erento varying eat (adh, Specs). In ral sped, the afonnation sures i the Bay CTanmlte the voce weshanim produng the varying 9D Presute {ibe sgnl) which franouted tough the ir (hn och. Inset anal smply space (or the ete, Sy one ail prefers hat entigite and mieading word), od ihe sia is the electromagosto wave wich etna The reaver aot fives reste, ching he ens stad tiga back into acme, and handing ie teeage tothe deinaton, Wher 1 tlt you ny btn the informa Alo our youth destination; sy wal estas ii rane nie, snd your ear and the saocoed cghth one it Oe Ta te proces of being ett i unfortunately shar: ceri cra thing re ed fo he signal neh were ot Ietende by the infomoion sure, ‘Thee envantel ads5 The Mathematical Theory of Communication nay be distortion of sound (ln talaphony, for example) or static (in radio), or ditortone in shape or shading of peture (tele slo), oF crore in uaaamision (telegraph of facile, Allo tose changes in te transmitted signal are allel nit "The Rind of questions which one aes to sak concerning such How doe one measure amount of sformation? 1. How does one measure the capucty of communiestion ehaanel?” The action of the teananitter in changing the message into the sana often involves s soding proves. Whst are the crac tarat of an ecient ending proece? And when the coding i te eficent ae pombe, at whet rate caa the ehaanel convey ‘2 What are the general charateritice of nove? How does rolie afent the accuracy of the mescage Soally resived at the Sestintion? How ean one minimize the undesirable ests of ‘ois, end to what extent can they be eliinsted? Hf Gh signal being transmitted is continuous (as in orl ‘speech oF mle) rather than being formed of aiacrete symbols (as in written spech, telegraph, ee}, how does this fact atect the problem? We will now stata, without any proofs and with «minimum of mothematialtrsnology, the main rests which Shannon bas 22. inermetion ‘The word snformation in this theory, is used ina special sense that mint not be eonftud with ie ordinary usage. Tn parila, information mst not be confused with meaning In Tac, tro rseeages, ona of which is heily lade with resning and the other of whish ie pure noasene, can be exactly ‘suivant, from the present vcwpont, na rgande ofortestion Tis this, undoubtedly, that Shannon mens wen he says that the semantic aspeets of communication are ielevant tothe ex- inssring sept” Bot thie sone not mean thet the enginoaring Sapecta are necezarily relevant to the semantic agpec "To be aur, thi word information in comminieston theory reste not much to What you do aay, ae to What You could ea ‘Soma Resent Contributions: Weaver ° ‘Tha information is « mssaute of onl freedom of eboie when ‘one scot iesage If one confront with every slomentary situation where he hs to eooce one of two alteroetive muesges, then iti scbitarly aid thet the information, asociated with ‘his situetion, unity. Note that ii isleadng (although often onveniet) to say hat one or the other message conveys unit, information. ‘The oonoeptof information spplies not to the indi- ‘vidual messages (as the concept of meaning would), but esther fo thesluation ss 8 whole, tbe wait information indeating that fn this situation one bas aa ativunt of freedom of ehoie, in electing a message, wich tis onvenient-o regard as standard | ‘The bo imessiges between which one must chose, in such & seleotion, tan bo anything ove kes, One tight be the tonto the {King Jones Veron of the Bible, end the other might be "os." ‘The tanamitter might ode thee two massages go thas “ero” is the signal forthe fst, and “ooe” the signal forthe second; or <0 that alowed sitet current owing) ix the signal forthe fit sud aa open eieut (no cures lowing) the signal fr Ube se ‘ond, Thus the two positon, sled and open, af simple relay, tit earespond tothe two message. ‘To be somewtst more dfuits, the amount of information is defined inthe simplest cases to bo moueured by the logaethm of the nurber of avilable choice. Tt being convenient 10 ute Tog fo the base 10, the inforoation, when there aze only two choles, 5 proportional to the logarithms of 2 to the base 2 But this is nly oo that a two-hole situation is characterized by informa: tion of unity, neh already been stated above. This unit of information ie elle "bi" this word, Rest suggstod by John W. Tukey, being s condensation of "binary dip.” When num bow ave ekpresed in the binary sytem there are only two digits, fnmely O aid 1; jue ae ten git, 0 to 9 inclusive, ace used in the decimal number system wbich employe 10 as a base. Zero Sd one may’ be ten symbolialy o represent any tn eho ‘a: noted above 2 that binary digit” oF "bit! natural 4 sso Glste with te tvorehoios situation which bas unit information, Hone he avalable sey 16 alterative messages among whieh0 ‘The Mathematica Theory of Communication bein sly chon thn se 10 = 2 ato 1 coo ae ht ths lal ip catered 4 ot Te double oa quer, whe one itt tat informa ton eed se lain a te ane ene ets the unoling of he try, themes ese aed a ce te lgaie mena net teeta Se Ar cones nt oe ton fe rl en eae ted shove ta oe se era lg he ee tim labo any, © ed I rpc, sasha at one ray ca aoe unl nfarnaten, how washes ed by aay te elya? Tt sana very rensble tease te Ute ts sel hart Gn at uh soe. hod tide the way oe oe on ese Iearitnl dein ifrmaton, For tre wea eae ot mapeding loot 8 caer nhs kenya ries at 0, 01,01, 00, 08 OTE ee ‘heh tie lags aoe, sn at of wha te lays at oe hod the logan foe tae Sar he thatthe lgrthie meen ngs cw of ean {oth caton, un evo woul wi Sniae cane e fainhle tie sure Se ner of ph ane Scab the lpi and ene elec wena a mond neta “ron tne far nse to arly sip stone terete intration sure oe ocho heen oe tral deote eamgn—ikes man picking os ae of eta Stndad bahay going \cegrane” Abe mite se et equ of sla we et of eects a teed sees then forming te meage Thea ore Pick ou ont word ater nthe tha nual ceed ee sae cing pw fo to mca eth pit sn imprint neraion which haben inthe beth sofay comasio the nto mabr enone Nanette mil fatty pe eae emage Horas esses sel neces ees Soe Recent Contrivctions: Weaver 4 eowroed by probsbltin; aod in ut by prbabitie which are Sat inependons, bt which, at any stage of the proses, depend pon the presting dhecee ‘Thum if wo are coneemed With nul sec, andthe Tat symbol chosen i they” then te [pobabity tal the next word bea aril of «verb form other Un vial very stall Thin probable infuenc etches ‘ver mote than tivo words fase Afar hese word tho vest" the probably for "hata he ext word i fry high, Sd for “lap” ax the next word is very fw "Tht thee ee probabilities whe exer eran degree of tootover the English languge slo become obviow i one tins, forexampl of the fast that i or language the ditonary con {ats no wow whatever in wich the nit eer followed toed fa ibs iraru,w; x or 2; tat the probability 1 ctaty nro that a iil | be fallowed by any of so teuera Simin, anyone would ages that Ue probably i low for such a sequence of word ar -Constantioople fshing nay pink) Ineidntally, iis low, But oot eo; fori ie peretly Fong to thin of «pana in wich ene cutee cle wi Gonataatnoplefabing” abd the next gine with "Nasty nk" ‘hed we might ebcrve in pasing that he unikelyfourmord euence une daeusion has oveured i # age good Baglsh fetes nimey the one above 1 Sam nih rosie ane of sys (ih a, teconding to certain probabilities is called a alochastic process, tot see of she proce i ih de Sea Muro di Ofte Mako proces wick mii ‘ity foportanc for eoumusieatin theory, tne Being WHat atid anodic pucses The aoalycal deal bore are com ‘Testa ad Uw founing co doo and owed tat ita aka me ofthe bet eerte of fhe but tathemetcane to rete the Sseviated theory; but the rg nate of an vedic pooes is {Sey tw understand. Teis one with producer a sequenco of sym Sols wich wold be a pol-takers dream, beens en reabonsbly Tene sample Lends to be sepientatie of the squeneo As & Siete "Suppose that two persons chaos samples in dierent” The Mathomatical Thoory of Commnication sways, and study what trnds their staitosl properties would show asthe ecnpes become larger. Ifthe situation is ergodic, then those tro persons, oworer they may have ehoten their samples, agro in toinextinates of the properties of the whole Ergodic systems, in other words, exhibit e percalrty safe and eerulortng sort ofsatstialrogulanty, "Now let us ztury to the iden of information, When we have sn information source which in producing & mestage by succes: sivuly selecting diverete symbole (lettre, words, musical nots spots of a corain tite, et), the probability of choice of the ‘ariour symbole atone stage of the proces being dependent on th previo eoies (Le «Markt proces), whit abst he "The quaatity which uniquely meets the natural roquirements that one sets up for “information” tarne out to be exact that ‘hich is Known in thermodynamics as entropy. Tt expressed in (orm of the varius probabilities involved those of enting to sertain tages in the prous of forming messages, ad the proba bilities that, when in those sage, certain symbol be chosen next ‘The formula, morever, Javaves the logarithm of probabil ities, so that is « natural generlattion of the lgastheic roacue spoken of above in connection wit simple cates ‘To those who ave studied the physical seienes, itis most ‘guifteant that an eattopy-lke expression appears in the theory 42'8 meature of miormaton. Lotedioed by Cinusiasnesly ont hundred years ago, eloy esneated with the name of Bolte mann, and given deep meaning by Gib in bis clase wore on statistical mechanls, entropy hse bacon 4 basic and porvacive ' neept thet Bddington emake “The law thst entropy les increases — the second law of thermodyamiee — holds I think, the supreme poiion among the Is of Natu.” Inthe physic] seienoes, the entropy associated with sta tion is a measur of the degree of randounues, or of “shud rece" I you wil in the eitustion; andthe tendency of physical syst to becom lees and les organized, to become more and ‘more perfectly shullad, is 20 basic that Edington acpues thet Its primary thie tendency ‘ich gives time its arrow — which rrould reveal tous, fr example, whether a movie ofthe physical world ie being ran forward of Bucward, | { Some Recent Contributions: Weaver ‘hur wn one nets concept entry in eramusiaton eet Tha formatin Se menue by entropy ae ly el re Seite le anne of flo of ele we aren‘, josh wel a jie» Gemodprate eatin orth nips) we” We il tant hit ‘eee of te ore gtr siaeaes of ti thoy Mise alse te etopy tor th fformmion, of he (aes iy of extsin hfomaton nue, ont cn > ra cmon tat te sure conte oempny te se Sut" to ttm te na, toy Z this roughly means thab this source tha erin source, 2 3, 2 {hints ccioe of eyeabol to form a measoge, about 80 er cont ty toe anit ould possibly be with these same symbols, One sine the tlative enuropy is ead the redundancy. Thi i he fraction of the steveture of the message which is determined not Tits fee see ofthe sendor; but rather by the aceplel ita! rales governing the uso ofthe eymbols in question. Tt teeny called redupdangy, fo he fraction ofthe messge i fn Tac ralundant in smnething close to the ordinary sense; that isto say, this faction of the tvesage La unnecessary (and hence vepestige or redundant} in the songe that f (were missing dhe Troseage would sil be esetilly expla, or at lexst could be Completed Tis mort intersting to nate thet the redundancy of Ents is just bout 60 por east ao that about half of the letters or vwords we choose in writing or epeaking are under our free choiee, Wid abou bal (altBough we ate oot ondinariy amare ofS) are Teall conelled by the eatinticl stricture of the language UREN aSh Ste ahs von a presale Haber2 —K_—= SST 4% The Mathematicat Pheory of Commzation Apart fom more serous implication, which again we wil post= one to our @nal dseusion, itis inlerestng to note Cat Fnnguage must have at least 50 per cent of real freolom for lative entropy) inthe cboive of Teter if one isto be able to coneruct satisfactory erossmant pusse. If tse complete fee dom, then every array of letters Isa eosswotd poze, Tt has ‘only 20 percent of freedom, Un i would be imponile to con struoterossword purses in sub complexity and number a wuld make the game popular Shannon has tatinatad that I the Eres language had only abou 30 percent redundency, then it would be posible to constrst thre-dimensionl erocomord pussies Before closing this sation an information, it sould be noted. that the real reason that Level A analysis deale witha conopt of information which characterizes the whole statsial nature of the informtion sure, adie not eonermed withthe indivcal message (and not at all dretly concerned with the rnaning of the individual messages) is that from the pot of view of eal nesting, a communieation systam must face the problem of handling sy message thatthe source ean produce. iti oo! Prsible oF penticable to design an system mich ean handle everything perfectly, then the system should be. designe to ‘handlo wel the jote it is most likely to be asked to do, and should resign its tobe ls feint forthe rae task, This aot of consideration leads ut once to the necaity nf sharaterneg ie statistical naturo of the whole ensemble of mexeager which 8 given kind of soure can and will praduee, And information, {6 wed in eomsmamiestion Uoory, dots fur tie Although i x aot at all the purpose of tie paper to be eon cerned with mathematica! details, nevertheless eeome een to have at good an understanding se possible ofthe entopy- lke xpreion whieh measures information. If ene i concerned, in a simple case, with a et of windepenent symbols ora mt of independent complet message for that tntier, whoce prea ties of eboiee are py py - = py then the actal expression for ‘he informations # Intog ps polog e+ +--+ pulog pel 2 pilog Some Recent Contributions: Weaver aa woul in math, that ese the pbs indie aw wt co ty tee le th ial ow, 9 ween etnies * ‘Peds compl; bt we pone fist lat we ate choosing only betwon two possible masse one pobre then for the ao a 2 reer te olier If oae reckons, for this ca, the mums sect Hit wt he ga ay aly the two messages are equally probable; that is to #8) cae hr oe nex ons romp ft Tos between the two messages. Just as goon a2 one mosage ste pbale tah (ne ODP ‘Sota Uoweme awn oe ace Pa A ee pmo: wep) the ake of i ey us ceo thi express a et toe where om poi i ty oe ens) ttn Hier (owe sn anf cee ne nero) ve His la sbilities are equal (i. it ssn te tne mien we a ae ay tee nubian in the Tec os when ae rn of coi it Eat sia han boy coies then Hi lrget when te prob spin of ea sascha 2 ple san er Te ct at you den toward sme vertain choices which have more than their share of probability. Span oth thet Fan st ene cnc at Foal ae ys anton a wich ne ew need se Te aot and ne bon te om of GUGM Arm such «eas don elle to have very smal vat he information (the freedom of choice, the uncertainty) value — wun the numberof ese Sed, we bave jst ate that hen cc os sean ined HeThe Mathematical Thoory of Commanicosion the information is the gresor, the more nexsly equal ate the probable ofthe varios eres Thee sane important way increasing namely by inreasing the numberof case, More sccurutely, if all choles are equally Ikely, the more choices there fare, th larger will be. There is more “information” 1 you tele freely out of e set of By standard message, thea if you select freely out of at of twenty-five 23, Copacty of « Communication Channel Alter the diseusson of the presotngsetion, one isnot surprised ‘thatthe capacity of channel tobe desrbed not interme of the numberof symbole it can transmit, but rahe i tert of the Saformation it transmits, Or better, since this lst phrase lends itself partieularly wall to misinterpetaton ofthe word infor tation, the capacity of a channa is to be deerbed im terma of its ability to transmit what is produced out of rouroe of given I the source i of «simple sort in which all symbols ate ofthe same time duration (hich ie the east, for example, with tele type) i the soure is such thet each symbol sboren represents 1 bits of Information (being fetly shovea fom among 2" sym tos) and if the channel een trap, aay maytbele per oeeond, thon tke capacity of C of the channel define to bene bite pet Ta rom gunsra ots, ona ht fo inks gocount ofthe varying lengths of th various sjmbole. Thor the general expresion fot capacity of @ chanaeliavolver the logarithm of the numbers of symbols of certain tine duration (whic introduces, of eaure, the idea of Information and corresponds to the factor # in the simple ease of the preceding paragraph); and also involves the oximber of such syinbols balled wich corresponds to the factor ofthe proeeding paregraph). ‘Thaw in th gonerl cst apacty measures not the nutber of symbols tment pet second, but rather the amount of information transmitted per ‘Seoond, using ita per second te uni. 24, Coding AL the outat it was pointed out thatthe tronuiter accepts the ‘message and turns i into something called the egnal, the later Dring what aetually pases over the channel ta the rectver. ‘Some Recent Contributions: Weaver w agree vil enn geo pal trie by ped ger mown tun swe oe eet ‘aol pr son. ”B The Mathematical Theory of Communication But the etatstial charactor ofthe sigal as actually teanemited by e ebannel, and hence the entry in the channel, is deter- mined both by what one attempt to feed into the channel and by the eapebiliie of the chanoel to handle diferent signal situations For expe, in telegraphy, thee hsve to be spaces between dots and dot, between dota and dashes, and between dashes and dashes, or the dots ad dashes would aot be recog” sinable ‘Now it tums out that when & channel does have oertsin oon strainle of thie aor, which limit complete signal from, there fre certain etaistionl signal charactrities whieh lad to signal entropy whichis larger tha i would befor any other eatistical signal strutare, and inthis important ene, the signal entropy {exactly equal tothe channel capacity. In terms of these ideas, itis now poasble presitey to cha sctorae tho most ficient kind of coding, The best transmitter, in fat ie that whioh codes the mesrage in sich @ way thet the signal has just Wow optimum etstiecal characteristics which ae best suited to the channel to be used — which in fact maxi- nse the signal (or axe may say, the channel) entropy and make equal othe capacity C of th channel ‘This kind of eoding lends, by the fundamental theorem above, to the maximum rate C/2f for the transmission of symbols, But for this gain in transmision rate, one pave a price, For rather perverely it happens that ae ony make the coding mote and ‘ore nearly ideal, one ie foreed to longet aad longst delays in ‘the process of coding. Part of this lemme i met by the fet that in electronic equipment “long” may mean an exsediagly anal faction of a second, and part by the fect that one makes compromise, balancing the gain in trasemieion rate gainet oes How doos noise affect information? Information is, we moet steadily remember, © meaereof one's freedom af choice in ler {ng a message. The groter thie freedom of choice, and bance the arettzr the information, the greater i the uncertainty thet the reasage actonly elected is come particular ooe. Thus greater Some Recent Contributions: Weaver ” trado of soit, greater uncetsioty, eter information 6 td in band Tr ne radu then the revd mage ents cers distortion, enain rors eras extraneous ate, er oaid ete Toed one tty thal he rocrved mesa ‘Sheen of the eects of tm note, an inreaeed wat tainty But ifthe unseat ip (remedy the iofoto i sree, tad ths sounds though te moze were Bench! Ses gecoaly tu tet when ther isn, the reve sign cebintl grote information op ete, the received signal it ‘Sted out of a one vased ot thn the ranamid gal {Fase abuntion which beau istrtes the semantic tap ‘Ren ebich on en fal ib does ot remember hat “infra Gt" ed ere with seca seaning that monsre fedon oP Soto ant hence unwise what nce hs tren made {tether posible forthe word information to have eter fool or bad comstations, Uncertainty Which ass by vireo edo of chic on the port of the soot is deiable uncer {Se Uncanny whieh eestor of eros or Beane of the ingens of nue in undesabie worn sane tar eve the ker in saying tbat De rcsved sig bs moe fortion, Some of ib nformadon is spurous SeMnderabie andes been stried via the mise. To gst {Ee canal atomntion ee seoved signal we must obs but thi spurious potion, ilar ean der up ths pint we ave to sop fOr 9 le dete, Suppone on ht vo eso ymbol, och ne the mesage Sintole generetad by the tforaton souee, avd the signal ‘bots ehh ae eoualysesvew The proeblitie of there ioc of symbole ae intereated, for early the probability St eecsving eortin eysbol depends upon what spmbol mas sto Wit no ror frm nove o mor cause, he recived Guar wold crcyond preity te ream symbols ent Shin the presen ponte er, te probable for rec ‘itols woul ebvouly be lade! Yeaiy on Une which cor~ ‘apd or lcly eerespond othe masse symbols sent ‘Rei she sitanton eve can cael what eae the enizopy af neat of symbols ration fo the ober. Lets oF Sample, comider te entropy of the menage rave to theThe Mathemation Theory of Communication Signal tis unfortanate that we cannot andertand the sense invoived hero witht gong ito sme deta. Sopp for the moment that on knows tht a certain nal syuibol hae atully been recived. Thon etch metiage syrael taken on erin probeblity relatively lage forth symbol dental with or te fymbole smiles tothe ene resived, and relative’ eal for ell ‘thers, Using his tof probabil, one calealtee a tetativ troy vale. This i the mowmage entry on the assumption ofa defaite known reeived or nal symbol. Under any good Conditions it value is low, since the probabilities ivalved ary fet spread around rather cre on the various ease, but ane Teaily loaded on one ot few cast value would be nro (seepage 1) in any cae where oie was comply abut, for thes the signal symbol beng own al mesa probabilities vould be tro exept fr one syebol early the one tceved), ‘hich would have a probebity of nity For each asspton to the signal symbol seid, one ean collate ono of then tentative manage entropies, Caleate all of them, and then average them, weighting each omen secodanse ‘rita he probaly ofthe signal ayo named in elolating ie Eattpie alle inthe may, won there ate two at ‘pmbols to conser, ae ale relative entre. "The petedat oe just eseribed eth entropy of the mestage restive to the Seu and Shannon hs aed he th eucestin ite sgnfesnce ia It measures the svcge sncsrtanty i the meng then the signal oso, If there were no nome; then there would be no tocelinty concerning the meaage ithe Spal i inown” Tt the ntorston sore has aay. tse) Sncrtaity after th signal ss kao, then thie must Be unde Stable nowrtnty dus to ise "The dincoson of tbo lat few paragrapbn otters around the quantity “the average uncertainty in as mesa outer wien te Teteived signal is known." Itcanegilly ell be phrase a terms Of the smlar quan “te average sarsnty onsen te ‘eveved signal wben the meseage sent is known.” Tin eter ner woul, of curs, ao be tro if ther wae no sn "hr tothe ntnaasoshp of tise quantities, 1s eany t wove tha Hea) — Hye) = HW) — Hats) Some Recent Contributions: Wonver 0 snore H (2) the entopy or information of the source of mete Ty iy) the entry or information of rexined Sass TES) aout rte wean teenage one {Puke Soa be Keowts Hay) the osertunty into rece ‘ule ft eget be Laon ore spares pat of Che SENed onl infomation which ede to noe The ih ide ZTPRSSLeltan ce unfit ne ned Spe of te bad effet of the oi. i.e pole exlin nat oon meas by the eapay ua ey chanel I'm fot dened to be eg to te Sacha ms (ibis person) st which wef inforation Teri antanty ros noe crag) can be tras pid over the shane My ies a ape, hr, of + “natu rio? What can cow ha, hat to ake the ate larger or salle? Tie anew. Smt one ca’ af: ths rte by cheng enuce howe titel character are utably rented tothe restraint SeUSei by theatre f the channel That ope en maxiie reat teaming well intra by sng pores cng (ee pase 1-17) “Lato, fly, let consider the fame seo for 0 Aue Shana Sepoce a he wl casa hay to ete {ule ucrbos 9 ope, eoppone Ue necting fom ab Iefematnn sores couasariad yan entopy of HG) tite err he entopy of the receive signals being H() Bis 57 ond th che epacey Ca equal toon re ta Tartan by aevng apron coo eptems, he ott 7G source can be tranaied over te ehuonel witha Hl ret se pou However wl x fueny of ee 798 rele” thee code which mets the demand. Bat if te {Buk capacty bfx than H(e), the etrpy of the sores ‘tom ybich acme espe, tei tps o deve {Shen which reduce the er raeney a Tow az one ay Bee. Howes clever one th the coding roe wil ays te tun tht afc Ue sgnal i teceved there retina sme t= SesRhi an) uceanty abst wha the rege may ad sta undliaie yoni this eqevoction — ill says Sevequal too gra tan (=) —C. Fuster, ee Meet intone code which capable of reducing hitTho Mathomatial Theory of Communication undesirable une, ceria he mass, doe ovale ‘ls ecnde Ce) by an ahtnady al noun hs mat apertures of cue ne te nna soi opin ue aad eed er, te miler bow coupled ot eppopens the cling roe Thi povel then ives» Pes and aos Sato tiople deneipon of the stone dependability oe ean et Chica ron sommuictn chnea abe spent in Ue proteome ‘One prc cosnuot, pointed ot by Samson, ld be sete. Snce Hap oat 80 pervs ceftnda i would Eeablet sve abet onal Une rary leggy ty. poe sootng roman ete en were ng totes tft C nln satu Whe set cho, how” ASM natal fh reludeoey. For he enaising es ancy np bo! thease Th very atte fo at Scere ofthe fet tins be redundancy of Bape gy one te for exami, il rm hentatien ab necting eer i toeling ht Ea rin ding Ganmiion Up tothe pit we have Den cence with mages fomed et of doe mtn ns words are fomed ef tae wenteness of ‘ord, rudy of ste, or hallo eae of fie Mtmber of rete spot Wit bappent oe then i one cetnders continue tteage euch te seeking wie ith connious variation of ph and ene? ‘ery rch ne ay ay thatthe ended thoy is sone ‘eh tore disk and complica intheatiely, bat nt Sevstliy diferest. Many ef the above watonets for he Grete cave Feque no moditeaton and others Tequle oly ‘nao change Ge tans wish aj» sn al the loin Ae 1 practi attr, On Aapy noted fo a cnauiny ‘signal which is built up of simple harmonic constituents. rae all Frey a of ei, ih ly wi ‘hand fr ter eget tsa, oinncy of een per tesond Thu alBongh the hs vlee dow cntin higher he. ome Recent Contributions: Weaver 2 graces, very satisfestory communication ean be achieved over (Mitepne channel that handles frequonsive only up t, say four thousand. With frequencies up to tan or twelve chouseod, high frlity radio tranceasion of eyraphoni suse i posible, ee ‘There ie very convenient mathematieal theorem which sates that continous signal, T seconds in duration and band-tiited a eequency to the ranks from Ota W, can he completly spec fod by stating 27W number. ‘Ths 3s really & remackable {eore, Ordinarily continuous uve can be only approxi~ IRotaly“haracterieed by stating any finite mumber of points through which it pases, and a inte nunber would in general be nied for complete iformasion about the curve, Buti the fhrve i bil up oat of simple harmonic constituents ofa timed uber of fequencey, a2 complex sound is Dull up aut of Timited number of pute toes, hen a finite number of parametars inal that in peoumery, This has the powerfl advantage of Feducing the chatecer ofthe eemauncation problem for con tinuous signals from complicated situation where one would have to dea with an abate rmber of variables to a consider dbly simpler situation where cae deals with « Gite (though Tange) notnber of variables. Te the theory for the continuous ease there ate developed fonulne which desribe the naximum capacity C of chanoe! ffequeney bandwidth W, when the average power used in trae sritne iP. the channel being subject to noise of power Ny This sabe being “wine thental nee” of « special king wach ‘Shannon defines, This white thera noite iv itself band tiited fn frequency, and the empitade of the varios frequmey oon ivunta are subject to 4 normal (Gaussian) probability dst {ition Under thee ieurstaos, Shanon obtains the theorem, guia relly quite remarkable in ita simplicity and ie ebepe, CEer ite puclle by the best coding, to transmit binary digits the Pate of vita per second and have an arbitrarily Jow frequeey of eror ‘Bat this rate eannotporsibly be exceeded, no matter bow elever fe coding, witht ging rite fo a dente frequency of ror. For the cise of arbitrary nole, rather an the special "white* ‘he Mathomatial Theory of Communication hema” mee asus above, Shanon doe notated Sa {sing on wit oma hel payne tana uper ad lower for shana sspsty ned aso derives Tite for chanel capasty when oor puiden not the average pone of te tami, Dt fe te Jastantaneous. power. * lly sold he sed Unt Shanon oan ee wish sre neomatly romana le ib vi are of hoey step and erepng cnc, wc o's gent of into meg og hatatre Ue Ait ofS eee ‘so sn of tt whch ce gas infomation, ste of union ad shanal ape a these being relative to certain fidelity re ieneci pate 3 ‘The Interrelationship of the Three Levels of Communication Problems 3.1. tnoductony Ts the fat action of thie paper i wae suggested hal ere ae thtee levels at which one may consider the general communication problem, Namely, one may ask aves A. How accurate ‘ransmited?| y can tho symbole of communication be Levan B, How precisely do th transmitted symbols convey the ddsred meaning? Leven. How effectively das the reosived meuning afet con dct inthe desired way? 1 was suggested thatthe mathematical theary of communica tion, as developed ty Shannon, Wiener, and other, and partes lacy the more deftly engineering theory teenie by Shenton, ldough ostensibly applicable only to Level A problems, acc lly is helpful and euggotve forthe level B and C probleme. Some Recent Contributions: Weaver 5 We then took look, in setion 2, at what this mathematical theory in what concept develop, what results it has obtained Ie in the purpos of this sonchiding setion to review the situs ion, end see to what extent and in what terms the orginal fection was usted in indicating that Ue progress made et Level [his eapoble of eontibuting to levels B and, was juried in indicating thatthe interrelation of the three evel is eo eon~ loorebe that one’s lial concTiion may be that the separation into the thee levels is realy ariiclal and undesirable, 132. Gonoraity ofthe Theory ot L2vel A ‘The vu Bet renal, and nde te ead cat cass he defor burden of he aruneat st tb matierateal theory Beefy ural in sie, fd in te probs 3 con and of cma amp and pore nh ea “isa theory wo genorl tnt oe dr ot sdf ay what iad of pmol ae bing cored winter ween ltrs Sr mondo mun oreo sien wor or eon a, Slur ‘The tour ae ome ratte fev iinrnintly apy f al thse nd to oe fos of SSromuicaen This nme, af coun, tat te hry i = ‘Sau tngiatively nner tits dealig ith te ‘eles th pemataien pole ithe se {ioctl cu ay tae isan cidenc of thi generality thatthe ory soils snort to nin fac nally he as hery of eos rap ee ef of edn. st {BE theory contutee to the poles of talon frm ae angnge To uote lhe te snp try hte chery Maun emettrtic of onaning st wel at of norman Enum, the ste doveoyed i tin work sonnet ly wit te poem afte open ergn oct ommptes ha IC ao eur that Shanon us st mien paper on te dhsgn of eampater whieh wosil be capable of Paying © Sti Geof thom. Ard i eo ahr Gest pies £0 the pont contention tt fhe pape slows with he ear Atac thr ne mun ay tot such» compar “hk” nero The Mathematical Pheory of Communication smut subeastilly roy the conventional implication of ‘verb “to think. Yara . ‘As a teond pin, tsorn shar tht an iporint conti on ot pb nr of emma {ine a fink svi o gum scommanicaon tts itis done a the eutuet ofthis theory: bt ths breakdown of Ue station tbe vey deeply sale ad aporprit one eoames convinced when he sees how smoothly and generally this viowpint Toads to cereal ies, ee alg certaniy tue that 2 conidertion of communion on level Band © wll eu stations tthe mote Sgr on egy bat as eaually kel Dat what ie reied tee addin aa no real revision, anise ‘Thus when one moves to lvls B and Ct ; td may prove tobe salto lke aon of the sail characte of the Aetination. One ean ining, av an edditon to the dlogran ‘other box able "Snare Receiver” tryed bine he fnginesingeeeiver (which changes sgn a tesage) tnd the destination ‘This sumatis veeiver sbjete the mesage to. second deodng, the demand op this one beng Unt ms ath the aia oman charset of tho meg to ie ainsi capaci fh tay of rece ot of that subset of reoeivers which constitute the audience on wishes to affect. Nea = Sina fone can imagite another box in the diagram hich, snide the information sure and the tanita, would be labeled “semantic the be "ale a 3 abl mie” the box previously labeled as Simply “ole” aw being lebled “engineering nice" From this toutes i imposed int tbe sgl the perturbations or distortions meaning which ar not Intended by the aout but which ieeapaly act he deviation. And te problem of seman decoding rust tk this sesati noise fate aecoune. It Te ao pessbe to thnk ofan adjortment of erga! resage so that ie Aisin tal nemo mean eden ‘Thirdly sea hight sggesive forthe poem a al eves that tor nd enfin ae ad iit deren whe, no mati how ood the eafing one trie to efowd tov ch ever a Some Recent Contributions: Weaver channel (Le, 17> C). Hore again o general theory af all levels ‘Tl surely have Go tae into aoeount not ony the eapacty of the Tanne! but alco (oven the words are ght!) the capacity ofthe [ftease. Tf one tee to overerond the eapecity of the audiener, se prabuy eur, by lveet analogy, that you do not, =o 8 ‘Spetk il the sudence up snd then waste only the remainder by Biting: More likely, and again by diret analogy, if you over owt the capacity of the audience you forse & general and fnescapable error and ealusion. “Fourhly, it shard to Believe hat levle B and C do ot have rmieh to learn from, and do ot bave the approash to their Plame teefully oriented by, the development in thi theery Ete entropies in relation to the conoept of information. Tho aonept of informotion developed im thie Uheory at fist secine dieappinting and bizarre — dirappointing because it bas eihing to do with motning, and birarre because i desle not vith sngle mesuge but rather with the statistial character of Tivole ensomble of message, bizarre also boestoe in thes foleal terme the two words Snfermation and ancertanty find theres 0 be pat Thin, however, that these sould be ony temporary reac tions; and thet one sould say, at tho cod that ths analy has to penteatingly cleared the air that ope i now, perhaps fF the fit time, reedy for a real theory of meaning. An engineering ‘emotion theory ie jst ke very proper and diserect git WRecping your tclogrem, She payt no attention to the meaning, tether it be sad, or joyous, or esbarrasing. But the must be ‘ropa to deal with all tat come to er desk. This ides that» ‘foumniation eysert ought to ty to deal with all possible ‘coagea, an tht the intligent way to try isto base design on the matical character of the source, i surely not without ‘onfleanee for commuization in general. Language mst, be ‘Gelgned (or developed) witha view tothe totality of things chat than may wah fo aay but nt being able to accomplish every {hing too should do as well oe poasbla as often ax posible "That i to sayy it oo sould dea! with its tsk statistically. Me eoneeplof the infrtaton to be ssocinted with » sure reads dot, a we have sen, to a study ofthe statisti! struc tare of language; and thir sbudy reveals about the Bois lan‘Phe Methemetical Theory of Communication frag aan xa, tration wh ae ty pen imo wry pn of nage tnd cman. Tan Saat he power tay of ery sniog Mask dens ene at pomians oa ui, St Sins i entety sda bn oof nm tate el tp of ating nly th cs tnt On th ue eit oration dee Say ove tat hinge § pl of cosa coos ‘erating ty, te ings fo er it ine n hag hfe er Or pape meting may ess bang ose of shat on wih epy emai sen Spend hopper fey i he hey, awa te sarin el enrages ang Stow nay ten ud seston bt oe tthe pase ne Stree pel Was ih ‘prt op and sg a a tbe yey my SS hse Snail Cee ts non at bay 2 mel ie na lag i a ee "ea iret Eddgon welt have om ling one the word mal slew erty tt ly a tapes be wo have en hed nh arth ey say aia te bogey mae al spe nea nea ‘THE MATHEMATICAL THEORY OF COMMUNICATION By Claude ShannonIntroduction ‘The reoent development of various soethods of modlaton eich Maen and PPS which exchange bandvidth for sigael-to-olee rao has intensified the interent m «gener theory of commun Tuten, A bes for euch hiory 32 cotained in the importest Supers of Nyquist! and Hurley" on thi sujest, Dn the present Peper we wll extend the thoory to inelade » number of new ane in parteuar the eet of noi in the channel, and the fantom posable due to the attisteal sruotre of the orignal Granegevaat duc to the nature of the Bnal destination of the information "The fundamental problem of communication i that of epro~ duct one pint ether exacly or ApDTORiDAtely w umemage ‘Ricca at anor poi, Frequent the messges have meanings that ia they refer to or ase eoerelatd astording to some eye teat Morn plyncelor conceptaal eatties. Those semantic we oF coumunication ate irlevant tothe enginering rob eS tue signifcant export ix tat the actal messege = oA weatad from a act of posible mossagss. The system must be th potibie elton, 208 jut the one Chis is uokowa at the me ‘exigned to operate for ec ‘whi wil actualy be chown since of dag + nui, Hy “Cain Furs Aflstig TS Sead” Bal Spi, men Tory LIE neni of inertien Bell Spor TechsThe Mathematical Thoory of Commsieaton the mumber of messages inthe otto ite then this number or aay monotone funtion of this nutiber een be regarded as 8 measure of the information produced when One message is chosen ftom the se, all choiews being eqully likely. Ax was pointed out by Hartly the most natural coe te the logarithmle function. Although this definition sust be generalised considerably when we consider the infuenee of the tlatistice of the mestage and ‘when we hive a continuous range of messages, we will in all ses uae an axetil logatithaie measure ‘The logaithmie measure Je more convenient for various 1. It is practically more wreful, Paratdtes of engineering importance auch at time, bandwidth, numberof relays, ee, tend to vary leary with th logarithm of the nusiber of possiblities. For example, adding one relay to group doubles the mumber of Pssible sates ofthe relays Tt adde Ito the base 2 Igacthmn of this number. Doubling the time roughly oquares the number of Drosible messiges, or doubles the logarithm, et, 2, It ie neater to our ntitvefoting a o te proper measure ‘This is elosely related to (1) since we inttiely. measure cation by near comparaon with ommon standard, One Tele for example, that two punched carda should have twice the ‘capacity of one for information storage, and to ideatial haa nels twice tbe eaptetty of one for uansmltng information, 3. This mathomatieally more stable, Meny of the lniting operations are simple interns of the logarithm but would require tlumey resttement interme ofthe numberof possibilities, ‘The cholee of logarithms bate corresponds tothe chee of unit for measuring information. If the base 2 is wed the resulting units may be called binary digits, or more bray bis sword suggested by J. W, Tukey. A device with two stable pos tions, suchas a olay of Mip-op cir, ean store one bt of information. such devices can sore 0 bits, since the total number of posible state x 2" end log. 2” = Nf the Bas 10 is ‘ued the unite mony be ealed decimal digits. Since ogy BC logs Mog 2 = 882 ote Me Irtroducton 4 deca ig is abost 3} it A git whet on desk om Pig thie a ton nab patra hry ae 8 orate capacity of oe dsial cig Ia atalytical work where Ince and ferentiation are involved the bee i some Ges uefa. ‘The reltig uno fornation wil be called metural units, Chae fom th Beata base 8 merely reauits ulipicaton by lo Bye emurionionaytem wo wil mean sytem ofthe type inetd edemateliy Fig. 1. Te conn of ewently fire pets 2, Arnot wh ope oh gy nse a coin» pal mabe for ttaniaon over te bane. Tn Uaephoy tis opreton sonics merely of eanging sand pre sine» proertone! lst ares I lege we bave Srencotingeprtin wih produce «sequence of 4 dashes Sad "ece ew the shane! corresponding the manage. E> 8 Saul PCM apeten the deren sasch faction must be pls epee, qoute sid ace a Really inte4 ‘The Mathematical Pheory af Communication L_} leaved properly to construct the signal. Vocoder systems, tle- Vision and frequency modulation are other examples of couples operations spplied to the message to cbisin te signal 3. The channel is merely the medium uted to transmit che signal from transmiter to recive, Tt muy be e pair of wit, 2 Seasialeabl, band of radi fraquencies, a beam of light eke ‘During tabstission, ar atone ofthe terminals, the signal may be perturbed by avis. This is indiated scheiatially fm Fig 1 by the note souree acting on the transmitted signal to pence the recived signal. 4. The receiver ordinatily performs the inverse operation of ‘tat done by the transmitter, neonatrusting the tegen tho signal ‘8. The destination is Uh person (or thing) for whom the me. sgo ie intended. We wish to consider ccriain general problems involving oom ‘munieation sysara, To do this tis first necessary to heros the various elements involved as mathematical entities, suhably ideatsed from their physiea! countenarts We may, roughte lassify communication eyotems ito three anal salegnien, diserte, continuous and mised. By a discrete system we wil ‘enn on in which botk the message and the signal ave soquonoe of disrots symbols. A typical care i telegraphy where the age is sequence of lettory and the sgnel a saquens of dots aches and spaces A continuous syste is one in whieh the Introduction Tal ofthe paper