' Info Theo

ITC Shannon

Informotion mesi a
Ancalog Diseree te

comm sySHCm
Noi Noisy channe De codRT
Source encoder cdulator
nodudator
ett

Dertnatto
Param e t r S

Source alphabe ymbd rate

ophabe pmpbati ty occunren
probablirtic accurrence of smbols (dependane

Measures of ofoTmation
Ifo. is ansmitted oDm source hich i

analog o dligitad
LeF the disereekf Souurce i e n d i 9g Sm
m bok
DoK

* S'z weith
thprobabity P, PaPa
Out Q hirif we choose S iHh PP
whan the Pro bai kry of Occurence is Jo ie
i contauns infoo an o
high uncer tau'nty then
vice ver soA

bo thee meaare of info wa dej inea

uanu tavey
by Shannon o-s

wnit is
oepen danl-
P on +he baae e

3 Paperkraft
 a
qolde Tatio
most occuYI ng pna be F

t h e self-unfo 6 and bif the

bo
ince 0Pz SI, y 20
The amount of info depends on
only on its
Pmbabilty of Occurrenu rolther than 1ts
octual conten / Phyic a) interpretabio

ConsieY the ax Of toSSig a Pair con T

binay
inaYy Source is producing outp ur 0 0Y tai
1 foT head
PCO) = PC) = 0.5

Loc
(2) =
1 bit

lo base 2
bit
naat

decit or
H Cha1+luy)

For a dic cree memory lessSounc each meme

isindepent o p7evious message
tptal
probability , Pr P. P,.. P

r = - log,(..P)
otal into
-,) log,C)
R)
Paperkraft
log
 Prope7 ties

Info T
uncetuunty T

No nPo
P 1 +then
Total info is
Sum o log,
relymersageS then
then

W L be
M= 2 equauy messa9
messa9
ea
e ac
chh

info
carried by
by
amt. of N Ievels

N bits

t
transmiHed
are
o find the
inaryy Symbols
probabi uties
&R 4
corre.sponding selt info

bits
2 bits 1,= 0.415
A I, =

statemenE if the

Prove the foouoin9 being +raoJnmi tte

receiver knouws the message
Zero
the into is

as mesage is knosn
A
A: Asume pbabi|ity
log4
logCD
T O

Paperkraft
 Entrop averog infp

H(s) = Ltota log() o()

L
HCS)=E Pi log( uniysmbol

Q A Dm pduce 1, Z , 4
pro babiüties as 0.5 0.2, 0.2, 0.
Detemie ebopy Obtain into in 4 K g 2

A:HCD= o5 log(2)+ e
0.4 og(5) + 0. wa (1o)
=
L+0.928+ o.332

7 bits ymbol

4, 13(1o) + log,(5) +2l9a(2)

3.321f+ 2.321+ 2
1643 bits

Find the rel" bet H, natr and bits

blts

A: T leg,) J l o 9 2 ) bits
I oat
lo3e CE)
Paperkraft
 2. 302

B84 nads
H 3 32) bits
na .442 bits

nformation Kate

LeF us Suppose that he symbols ahe emi Hed by

8Ource at fixed ra te ssym/sec The avg

rate Rs bits/sec are

are deined ab the Pod

andthe
o avg info contained by pen symbo
mess aqe symbolrate
Rs HC)* bissec

Rs H Tir ime to send sin1e mesaq

T

tngvent has pasible outcome

P P P a ,P4 as Dakrmine
enhropy Lso obtan the ra of i o t f there
are 3 outomes per second

CAns on nex)

Paperkraft
 ADC

A HCs) = Llog,4t l09,2 + L by

2 4
o.5 +0.5 t o.15

75 Symt e c blls/symbol

R HO) x Ys

15 3
14 bitr / sec

Source encoding i n r e i thro é eiuey

Souru Channel don)
PAsdkate-
AnD
encodin9 Eno denj3

The outaut Consists04 150 sumbol 32 oF

which ocur with Po L and remaing
o4
Ocur ith em emit
a
o 23b
2000 ec a n tha rh ymb o
are chojen irdependunt find ovq ino nak

HCU) Ys KE 6941 2000

15, 3 bib'ec
Kx L x e6 + x log,23

3 +3.94 94
aperkrafl
 ouru Coding Occuren siuL
Take ehers Ato H
Parameter( length ef odt
Pixed code length
eter Code letter cod ett
A
D o

OO
e1O0
O

2 Vaviable length code CULC)

leters Code
leteT Code etter) code
A
olD ol
O0 C

Ex ABA DEAD R tG ABA DCA B

Fixed L 000 001 00D OIOI0 0o0 oo 2
VLC : o0 ol0 o0 l00 oi coplo 8

VL E ololdele
Reqoup VLC 2 in ou'kb on mann N
b) 2eqro n n 6inge q0up
VLC1 wuqu, iabhnto
no Prepx, r e d n r
AEABAAPP
4BAA 6APA
of other
be prefix t o have
*Godes shouldn8t
n t guen es
Paperkraft
 P4)
Avg Co&t le-g
wo r d
foa
those in h i h no cod
Prehix codes are

n code e. also callnd prejxfree cod

TeRix op
InStanteneo w codk As soon as a seqyuunu
bit car reaponcling to any o n e 0 he coc
worolJ detected, we can oledare that
umbol is decoded
There i ho decoding dela this

Consider the follausing hay beep encodeein binay

With 0 & ldentfy djerent code
Coole coou2 coded Codl4 Coda 5

00 00 00

QL 10

00 00 0DI
I- O0O

fired Cod 14 2
Variaye Code 2,4,5
' Inatontarcbus cods 2,4,6
*Ditinct Code 2,3, 4 5,6
Cwde 2,4,G
Uiqle Code 23,4,D,6

Paperkraft
 P(X2)= 0.125 PC14) 0. 125
PCX1) = 0.5 PCX)= 0.25

Ihas 2 code Compal their e Phicienc

Parameter (2 elbiceny
A

NOH Avg cod ung th is detinnd a wg. no.

bits e r Souru/ WH
o1 alphabe

No | Code ebbicceny is alebined

a
eno p
enroP
pon Qyq_Codu

HCs)
T

V= Code redundancy

. ont
2 2 Hcs)= 75
ka0. 97S
+3xLt 3*

0.75 H +Lt =
\ 1E
75

Paperkraft
Dptimum Code 15 it i instanten eous und h ons
minimum aug len for
qiven SDwcR t h fulhula
probalo kty asign men for the sou2LL
ymbs,
Enmpy Coda when VLC is
ts designed u d hati
sudn ha
avg
ave code word
length,L apoathua ks
nano
oy Dms
H6)i.e. enropy 1

A bimaru Sour
_emitina an indepenàan
eauence o Os and o Pp and P
Plot he enopy of
the
rupttid
30urcesY_
HCX) =
P lb + EP)lo9

Paperkraft
 bhannon- tano Coding

t js tech nique fo
Source
coding preyia cod
baced dumbols and their
prpbabiuHes.
ThisisSuboptina code as it doe'ntahiee f*
louo& possible cosde uWord langth
+produce fairly icient VLC
IE is used in 10mPLO DE compreSLon mehod blth
I S a part of zip file

Algomim

Amang the Souru

symbolsin oadeT s
decrea Sing pobauity The sym bols wh
eaua prmbcubicties
equcd can be listedin any
afbitanu Drder

Divide the et iato 2 such tho he Sun A4

tht prDabiities
seis Some o0
nearly ame

Astin 0 to the upper Set and

ond 1 tto the
lower se Repea Sps i l ea
ub eE Onkains
Paperkraft aing symbe
Ex1 A
Dms
0.4 0.t has 5 Sumbols
Find o.8, 0.),O.|S th their probabiiti
S-C ode
and
and
A caleulate
calculate
fren
P(xe)co0.4
Lcol 2. el
3Cod
Codr

O.18
O
0.187
o./
ID
0.1

L = 2
0.4+0:18+ 0.1 0.25
225

H (X) = 04 x lo92 (25) + o.18

xloqu()
+O17xlo +0.5 sio +0.lxlota)

HCX) = d4 bi+symbol

HC) 4 0.3%06 2 151 0.9561

A
2

Pperkn:ft
 A Dms ha
304,019, o.l6, OiC, o.
Conshu J-C code Find enhopy owg coL
and co de
icieny
Co d
A 0.4
O.19 0
O.16 O
2
O. 15
2

H(= 0.4x log, (25) +014 Kloq,(C

a
+0.16 xlo 0.15 xlg/o Loe
HCS)= 1
G48
49
bitl yObo

L= t 2/o.15 +3(02s
2.25

M 0.955//
 Huuman's Code
H
manm coding producc prefix code tha
always achteve low edt poss ible
word length average cod
t
iS a
optima code
(orloses dundanc Sithhigheu etiainy
A Knousn
pimun_ code (o) Minimnun Kudurda
code

lo hm
Lirt the SOurusymbol in order e4
decrearing pohability The dumbal uh
eua pmbabilike can be amaned.

Combine
he prooabilhies a the ymbcl
hawing the louwelt_probability. Ned enrdar
he resultant pobabilihes thi
reduchion 4 The Came proces _ rpealed

un there are exacH 2orcere Pzobob.-

The ind step i calRA

femain
Lt redachon

Paperkraft
Stavt tncoding ubith he lat reduho
Asign 0 othe fint dig in th cod
erdS-f all the souru
sDciakd Hh th lant radudio n
probob
then astig Ho the 2 pobob
probob.

Now 9o bac to the previsw

reduchon step 2 01o te
Sacon dig fo7th 2
prdbabi rie nd
7 Combin e od in this
duion SheP
Tetainina al
aSimentsmadu in 9
Aepeat4) i l the 1 column is reahe2

Constu humannod fo
O.4
| 04 | 04
25 0.360.4 Z1
O.137 0.25
0.15
. 0.25
.35+00 0.6
YA: 025401 o.43
o.18000
0.1 O01
0-101l 0.15O1O
Paperkraft
 2 114
HCS)= 144
bit/symbol
L=0.4 3 (o.6) 2-2

HS) = 9 4 2.1514 0.1779

22

Kra'sIneguality Condition
It i a
necess aTy and suicien condükon

Pmve the exisenu o a preiK

code eY
uniquy decodeble cod
n Symbolr 4?no.o bib
rep resent smb|
d to

Ex. HOuPgan fo1 SAHARA

A

Paperkra
 A Dms has 1Symbol
A

D.5 0.25 0.125 0.0625 o.03125 0.0ISt25

Cons t t Hubhmann 6ds ethiuenu4

A: V -
V V
/ V
41

Code 0
4
lo ItO il1o 10

SeP

Ru fmon pde opinal if t pmbabikh

are
on
inptoeii o&, thits becoz a the
ane inDderr
ae
cod LooT ak bit Levc. pr
.prtwmetit cwdim oeun h

Paperkraft
 Ariihmetic Coding

he probab t wpno be repregenking tha

be encoded by inhena of res na.
le to an

b/
Sucesive Sumbols in he meag reduu thir
inteTva in advanu wh Hhe pmbability h E
dymbe1

The mese likely symbel reduce the b

range b
thut add Pewev nO 0 bi to th
e and
mesag

Lpaithm

T ode Symbol 's hare

a no,Pom 1to n ymbol
Tep res ented
has pPob Pr (i)

Coobouund Pri J

high-bound Pr

Tane
=
h - lo
loo + rang * leobound)

+ rane * hijt - bun)

Paperkraft
 udith P= 0.5,0,2%,02j
B,c
Ex Considev 3 cymbo A cod
Arith mehie
Encode BACA Sin8
sing
C U.b0 9275
A 0.15 A 0.625 b25
aG054 815
.5937 rao.17125
0.601375oo12
0.e21 o.g62r

0. 0.5 0.593 75
0.543155
A,B,cD
4 symbol ith pobabiitis o2, 0.2,0.4, 0.21
4 0.2
B
0.01
20012
0.3 -012
-
-
0-2 0:048
0
0.04
6.D S6
Enuodu for AC

O+C0.2 o )* 0.2 =
0.04
0-24+Co.2-o) *0,2 0.og
O.03+ o.2 o) * 0.4 0.SI6
o.16+ Co.2- o) 0.f= 0.20
 foo
0.04+ (0.0a - 0 . 0 t ) * 0 . 2 = o.043

0.D4 + lo.08-0.04) * 0.2 = o.056

O-D56 + (o.030.04) * of = b oT2

0.D72 + (0.08-0.4) O2= 0. o8

o.0564 (0.012-0.056) * 0.2 = 0 0 - 3 D.0512

0.0542 + 0 . 0 1 6 x 0.2

O. 0f6 0 ,0, 44

o.ol6

Runlen gth_Coding
00 DO 00I|I
, ) , ( 01,0),1o0,1)

enn .
Lpal-
 empelZiv alsh LZw coding)
olietion a
ry based co duna

EX Encode using Lzw-" DRESSES

Sniginal di di ono
D
R
E
S 3

Snpue CurneF dee Eno de New Dit

shin Before olp Enny inder
D D
-Nil- - Nil
N
DR/4
DR E N
RE/5
DRE S N L2 ES 6
DRESS N
o,1,2,3 SS/3
DRE SSE N 0,1,2,33 SEl8
RESSES ES
ES
 HW Encode ABAAABD LZu
and DADA DAA DDA

Le u n i

ShaY input data Tva eat vnme

tet titi bw

Probabiity Related ShuPR

Ex1 Throw dige and observe the ns of dots
appeanng onthe top fau Conchroct S.
E b e event than odd no.ocurs, E2 even n8.
A S 1,2,3, 4, 5,6 E3 Pime ns
Find eent E4 thal even DY prm . o CCuns
E =,23,4,5.63 i. E, U
E+E2 E
Lis the ucomL fo1
fo an een Pm mi n0. o

EsE E,
vent hat do primu ns. E gE,
event tha 7 dors app i#3
eveH e n and o dd ocursmtonaue, 203