INFORMATION THEORY AND ERROR CONTROL CODES Information and Entropy Important Points to Remember ‘© The information theory is used for mathematical modeling and analysis of the communications systems. ‘© Average information is represented by entropy. * The joint entropy represents entropy of joint occurrence of two or more events. ‘© When the entropy of individual event is evaluated from joint probabilities of the events, it is marginal entropy. Part A : Short Answered Questions Q.1 What is average is information ? What does it mean ? USP [INTU : April-16, Marks 4] OR Define entropy ? 0S [INTU : May-16, 17, Marks 2] ‘April16, May-11, Marks 4] ‘Ans, : Entropy is also called average information per ‘message. It is the ratio of total information to number of messages. i.e, Total information Number of messages = Pr 1082) 5 k (ee) Entropy, H = Q.2 When is the average information delivered by 1 source of alphabet size 2, maximum ? os [aNTU ‘Ans. : Average information is maximum, when. the two messages are equally likely ie. p; =p =} Then ec.-04] the maximum average information is given as, 1 1 Hmax * 31082 2 + 5 1og2 2= 1 bitfmessage @-) | | | | Q.3 State any four properties of entropy. ESP [INTU : May-13] Ans. : 1) For sure event or impossible event entropy is zero. 2) For M number of equally likely symbols, entropy is log, M. 3) Upper bound on entropy is Hyysx = log, M. 4) Entropy is lower bound on average number of bits per symbol. Q4 What is self information ? Derive self Information I(x,). State its properties. UEP [INTU : May-10, Marks 5, April-16, Marks 8] ‘Ans.:The amount of information transmitted through the message m, with probability pj is given as, Amount of Information : I, = ei 3] (Q41) } = 810 Vm) Unit of information : Togio2 Hh tha sors: equation top| Ke Normally the unit of information is ‘bit’ Q5 Calculate the amount of information if binary digits (binits) occur with equal likelihood in binary PCM. (GRE [INTU + April-6, Marks 4] Ans. : We know that in binary PCM, there are only two binary levels, ie. 1 or 0. Since they occur with equal likelihood, their probabilities of occurrence wil be, PyC(O level) = py (1! level) = 3pam. 0) sag dn uy - SNOWYONIEN TOINHORL alle STO Fx wy panord ard jo nonany © 22 14 30 201d y (rd) (4-1) (4) [SE lego +} 2] tsoyd = (Ettore +| 5) ord = ‘samonaq uonenbs anoge way] d-1 = td Md = 1d sanmqeqosd yy saBessour omy aaey a4i ax2p{ (#) Boy 2d As a cao, Me a es (74) 144 Re 1 z ‘aq q]LM uonenba aaoge saSessaut omy 10 (Swf on 5 ‘se uaa st Adosua yep MOU aN : “SEY Ue Suen T-195 ‘Zt-Kew “60-"M0N ¢ MUNC] a cumusprent 2q, 0} Adoxua 103 uoRIpucs ayy enoid pue 23835 UO le sew “t39s ‘6o-Sew + uN 9 “sobessout 2m Jo 4, Aaiqeqosd yo uoHouNs & se (1) Sdonue Jo woneHen 2 J01d “A1>%N] A|jonbe oe soBvesom ayy Tpoq way wnuIpREU =} AdonUa axp eM, anoLd “Sjeanaodsox (4 - 1) pus d yo sonmqeqord yum soBessou yuapuadepuy om syuisuen aomos y 3D “sq. N 2a Te aBesseur yee Aq pouzes uoqeunosut jo yunowe uot ‘saBessour Ajoqy Aqenbs z= Ware aru 31 (at “tps ty st tu pure tw ©} anp Ajamiquio> panies uoneuO;Ur yo yunoWe uawy Wu hq pour uonemop oy st 2] pue fu aSessour Aq poses uoneuxonn oy st 1] at (Ht o1az st pauie> uoneuopur Jo yunoure oxy ‘ponmusuen Sueq aBessour ayy smouy zoarDou JT (H -p4ow ose st parses uoneuoyut ‘eSessoum ap ynoqe Aureyeoun axour st ary I(t :suy ‘wopeuuosmy jo sonradosd yueoday 2365 2° suopsand paremsuy $u0T : Hed ~FapoD jon s0uty paw Rsooy woxivmsopy (9D) Term ‘se uonenba aaoge JO “SHY AIM we> am (9D) UoHENba wry 20 + y 80]= [gy] 80] aus i ) soo 2a s td | ‘J [eie}- | ea = ty uonnuyep Ag. (as T ‘st %u puv ut 04 anp Spaysod.s0> pouse> uonrunopY aoyru, “Wd 'd st afessour aysodwos jo Aymqeqoid ayy ‘uapuadapur are Zu pue hu safessou aoing “uu afessaut jo Ayiqeqosd st Td pue lu aSessau jo Amqegoid st ld aay td ld (190) (#4 ):201=%1 pue (*) FAO ID ‘ow tw | pu tu saGessou Aq pares squmouie enplarpur ayy. id eae A 7 ten ‘sr nmouTe Jo UORTEGap ayy, : "sy Te sven ‘or-andy + une] en p+ ly = 21 ‘sy Zu pue lw'o3 anp Ajazs0dwo> Pauses uopeunoyyy Jo jMoUR oy rH onod WOR Zur aBessow fq popes uoReuNO; amp 81 2] PUR Tau eBessour fq poures voneuoyyy oxp 51 1 51 9D “uomeuusopur Jo 11q | Soure WOd Areurg = (anng) 2181p Areurq Jo uoneoynuapr ysax109 ay sm, et eSpee aul Garay (rsd) TS ipa ale sa zor = %] pue z“Boq = YY (7 [shone me (fan = vaar8 0q | pare uoneuosUT Jo pOUTe aus we Suoqyeaqumuiod prea aDigital Communications 2-3 Information Theory and Error Control Codes 4 ee # 3 2 02 07245 A 0.9709 2.9708 7218 4 a Fig. Q.8.1 Plot of entropy ‘H’ with probability ‘p' for two messages. Th maxim fH occurs at p=, Le. when monnage ae equal ely Q.9 Write a short note on information rate. [NTU : May-16, Marks 3] Ans, : The information rate is represented by R and it is given as, | Taformation Rate: R=rH | = (Q91) Here Ris infomation vale His entropy or average information. and ris rate at which messages are generated. Information rate R is represented in average number of bits of information per second. It is calculated as follows R (in messages / second) * (# in Information bits / message) Information bits/second Q.10 An analog signal is bandlimited to B Hz and sampled at Nyquist rate. The samples are quantized into 4 levels. Each level represents one message. Thue there are 4 messages. The probabilities of occurrence of ind pp = pa = §- Find out Information rate of the source. ESF [INTU: Aprili6, Marks 8] these 4 levels (messages) are py = p, ‘Ans. : i) To calculate entropy (H) : We have four messages with probabilities p; = (or entropy) is given as, 1 (1) oo xP +7 ose }+7s82(Z2) Flog, 8+3 3825 Jt 382(5 3} +3 logy @) H = 18 bits/message + (Q10.1) FF recricaL PUBLICATIONS”. An up thus for knowledge ‘orcooeSS a -ajdurexs snp ur pessnostp awoyps uorssrusuen ay Aq pozqan Ajoyaqdwoo you st Wa AeuIg, jo Amnge Surdure> uonewonay axp yi SMOUS STII, (ZOT'D UoHENbs a2s) puosas sad uoHEUIO;UT JO SHq 9%¢ Sunmusuey aze aM JIB pouTEIgO axey am OLD UE Ing puadds ad vowmuuofs Jo sug q y Bursenuos Jo ojgedeo st auows Surpoo aaoge ayp ‘uoReuHO;N Jo 314 1 SuKeauo> jo ajqede> st yn eUO aug ‘pos/sinuig g # = 2es/seSessow g zx aBessour/eynng z = apex (Synmg) suStp Areurg ‘oq THEM SyStp Arewig Jo ayer uorSsnusuEN sua} “295/saSessaur g Z JO aIex aLp ye yu2s are safessout ye MOW aM ‘aBessour Yea pues 0} parmber axe (suIg) susIp LAeUIG om sm, ov" eigen VOD ages, wt uMoys se Wd Areuig Stsn papoo aq U9 S[9A3] sno} aSouy, “sfaaa] Moy axe axayR ‘eaoge passnosip am ajdurexe ayy Ul quewwi0s (ord) 329/11 9 9E = aBessouy/syg 1 x esjsoflessourg Z = ‘uonenbs aaoge wt Fy pur s yo sonqes SumMg He=¥ ‘se (6D) wonenbo Aq wont st ayer woReWZO}L] + (y) eves uopeunoyuy eyeIno}e9 O4 (IH -pasjsaSessourg Z = 4 ‘puoces sad saBessoyy ‘qeuSjs aBessour auo soyerausd ajdures Azone aouis ‘sasysojdures gz = ayer ysmbsn, “sq yeudts pomrumpueg zpig 20} ayer YSMbAN “ayer sMbAN ye porcures st yeuBrs aap 1H MowDy 9, 1 UW) eyes eBessow eqeindjeo o4 (1! ‘Sep0D 104310 404i pun hsorys womusofiay wT ‘sworsvoqunuio yeaa tsp. tg + Syotevonand WOOL pl neat