Unit-I: Information Theory & Source coding-

Information Theory
Evaluation of performance digital communication
system
Efficiency-to represent information of a given
source
Rate of transmission of information reliably-
over a noisy channel
Given an information source & a noisy channel,
information theory provides limits on ---
1) minimum number of bits per symbol required
to fully represent the source (Source Encoding)
2)the maximum rate at which reliable
communication can take place over the noisy
channel (Channel Encoding)
Uncertanity, Surprise, Information
Discrete Memoryless Source (DMS)
-output emitted per unit of time, successive outcomes are
independent & identically distributed.
Source output-modelled as a discrete r.v. S
S={ s
0
, s
1
, ---- s
k-1
}
With probabilities
p(S = s
k
)=P
k
; k=0,1,..K-1 &
Condition
Source output symbols are statistically independent

=

1
1
K
o k
k
P
When symbol s
k
is emitted
-message from the source comes out
- probability P
k
=1
- No surprise : certanity
-No information
When source symbols occur with different
probabilities, i.e. probability P
k
is low
-more surprise-:uncertanity
-more information
Amount of information = 1/(probability of occurance)
more Unexpected/Uncertain an event is, the more
information is obtained
Examples:
Information/ self information: Defination
K message symbols of DMS as m
1
, m
2
---m
K
with
probability P
1
,P
2
,----P
K
Amount of Information transmitted = I
K
Logarithmic base 2- unit bits
e- nats
10- decit/ Hartelys
bits P
P
I
K
K
k ), ( log )
1
( log
2 2
= =
Why logarithmic relation
K message symbols of DMS as m
1
,m
2
---m
K
Each symbol is equiprobable & statistically independent
Transmission of symbol-carries information (I)
All symbols are equiprobable- carries same information
I depends on K in some way as I = f(K), f is to be determined
Second symbol in succeeding interval, another quantity of information I
Information due to bothI + I = 2I
If there are K alternatives in one interval, there are K
2
pairs in both intervals
2I=f (K
2
)
In general for m intervals
m I =f (K
m
)
Simplest function to satisfy is log. f(K)=A log K , Aconstant of
proportionality
I = log K, All K symbols are equiprobable, P
K
=1/K, K=1/P
K
I=log ( 1/P
K
)
Properties of information
1)If P
k
=1,
-receiver being certain @ message being transmitted
-Information-ZERO
2) More uncertain/less probable message-carries more
information
3) I
1
-information by message m
1
with probability P
1
&
I
2
-information by message m
2
with probability P
2
and m
1
& m
2
statistically independent ;
I(m
1 ,
m
2
)= I(m
1
) + I(m
2
)
4) M=2
N
: equally likely & independent messages
-Information N bits
Proof: probability of each message= (1/M)=P
K
Information I
k
=
bits N N M
P
N
K
; ) 2 ( log ) 2 ( log ) ( log )
1
( log
2 2 2 2
= = = =
Entropy H -a measure of information
In communication system-all possible messages
are considered
DMS with source alphabet S having M different
messages / symbols
Described by avg. information per source symbol
Entropy H
--depends only on the probability of symbol
Proof:-M different messages/symbols- m
1
,m
2
,.m
M
with
probabilities P
1
,P
2
,..P
M
Sequence of L messages is transmitted & L>>M

=
=
M
k
k
K
symbol bits
P
P
1
2
/ );
1
( log
Properties of Entropy
1. H=0, if P
K
=0 or 1
2. For M equiprobable symbols, H=log
2
M
Proof: for equiprobable symbols P
1
=P
2
=---=P
M
=1/M
) ( log
)} ( log
1
{
) ( log
1
) ( log
1
) ( log
1
)
1
( log )
1
( log )
1
( log
)
1
( log
2
2
2 2 2
2
2
2 2
1
2 1
1
2
M H
M
M
times M
M
M
M
M
M
M
P
P
P
P
P
P
P
P H
M
M
M
k
k
K
=
=
+ + + =
+ + + =
=

=
3. Entropy is bounded with upper bound=log
2
(M)
i.e. 0 H log
2
(M)
Proof:
4 A source transmits two independent messages
with probabilities of P & (1-P). Prove that the entropy
is maximum when both the messages are equally
likelly
Proof:
Information Rate (R)
For a source rsymbol rate/message rate
Unit messages/sec
H=avg. number of bits per message
R = rH, bits/sec
Examples
Extension of a DMS/ Discrete zero memory
source
For a block of symbols-each consisting of n
successive source symbols
Such source extended source
Source alphabets S
n
Distinct blocks K
n
, k=number of distinct
symbols in alphabet
H(S
n
)=n H(S)
Example: consider the second order extended DMS
with the source alphabet S consisting of three
symbols s
o
, S
1
, S
2
with probability of occurance as
P
0
= , P
1
= , P
2
= . Calculate entropy of the
extended source.
H(S
n
)= H(S
2
)=2. H(s)---1)
H(s)=calculate= 3/2 bits ---2)
Putting in 1
H(S
n
)=3 bits
Source Coding
classification
Source encoder fundamental requirements
a) codewordsbinary forms
b) source codes uniquely decodable
Properties of source encoder
1) No additional information
2) Dosent destroys information content
3) reduces the fluctuations in the Information Rate
4) avoids symbol surges
Source Coding Theorem-Shannons First Theorem
Theorem
Given a DMS of Entropy H(S), the average
codeword length for any source encoding is
bounded as
H(S)fundamental limit on the avg. number of
bits/symbol ( ) for representation of DMS =
L
)" (S H L
L
min
L
L
S H
code
) (
=
Properties of source codes
A) Uniquely Decodable: Single possible meaning
B) A prefix code: no complete codeword is
the prefix of any other codeword
Source decoder starting from the sequence,
decoder decodes one codeword at a time
Uses decision tree- Initial state, terminal
state
Decode the received sequence 010110111
a) Fixed length code:- fixed codeword length
-- code 1 & 2
b) variable length code:-variable codeword length
-- code 3 to 6
c) Distinct code:- each codeword is distinguishable
from other
-- all codes except 1
d) Prefix free codes: complete codeword is not
the prefix of any other codeword
-- code 2, 4, 6
e) Uniquely decodable codes: code 2, 4, 6
f) Instanteneous codes: code 2, 4, 6
g) Optimum codes: Instanteneous, minimum
avg. length L
Kraft-McMillan Inequality Criteria
If DMS forms prefix code, source alphabet {S
0
,
S
1
,S
k-1
}, source statics {P
0
, P
1
,--P
k-1
)
Codeword for symbol S
k
-length l
k
Codeword lengths of all the code satisfy certain
Inequality Kraft McMillan Inequality &
If the codeword lengths of a code for a DMS
satisfy the Kraft-McMillan Inequality , then a prefix
code with these codeword length can be
constructed.

1
0
1 2
k
k
l
k
*Given a DMS of Entropy H(S), the average
codeword length of a prefix code is bounded
as H(S) < [H(S)+1]
* when =H(S)Prefix code matches to
DMS
L
L
Extended Prefix Code
To match an arbitrary DMS with prefix code
For n
th
extension of code, a source encoder
operates on block of n samples, rather than
individual samples
- -avg. codeword length of the extended prefix
code
For n,
n
L
]
1
) ( [ ) (
] 1 ) ( . [ ) (
] 1 ) ( [ ) (
n
s H
n
L
S H
S H n L S H n
S H L S H
n
n
n
n
n
+ <
+
+
<
<
) (
1
lim
S H L
n
n
n
=
>
making order n of a source-large enough
-DMS can be represented faithfully
The avg. codeword length of an extended prefix
code can be made as small as entropy of the
source provided the extended code has a high
enough order in accordance with Shannons
Source Coding Theorem
Huffman Code- Variable Length Source code
For Huffman code: fundamental limit
Huffman code Optimum Code
No other Uniquely decodable set of codewords
smaller avg. codeword length for the given DMS
Algorithm
1. List the given source symbols by the order of
decreasing probability.
- assign 0 & 1 for the last two source symbols
- Splitting stage
2. These last two source symbols in the sequence are
combined to form a new source symbol with
probability equal to the sum of the two original
probabilities.
) (S H L
-The probability of new symbol is placed in the list
in accordance with its value
-As list of source symbols is reduced by one
reduction stage
3. Repeat procedure until list contents a final set of
source statics of only two for which a 0 & a 1 are
assigned and it is an optimum code
4. starting from the last code, work backward to
form an optimum code for the given symbol
Example 1.Consider a DMS with source alphabet S
with symbols S
0
, S
1
, S
2
, S
3
, S
4
with probabilities of
0.4, 0.2, 0.2, 0.1, 0.1 respectively. Form the source
code using Huffman Algorithm & verify the
Shannons first theorem of source coding
Symbol
S
0
S
1
S
2
S
3
S
4
Probability
0.4
0.2
0.2
0.1
0.1
Step-I
Splitting Stage
Step-II
0.4
0.2
0.2
0.2
Reduction
Step-III
0.4
0.4
0.2
Step-IV
0.6
0.4
0.2
0.4
0.6
0
1
0
1
0
1
0
1
Ans:
Symbol Prob. Codeword codeword length l
k
S
0
0.4 00 2
S
1
0.2 10 2
S
2
0.2 11 2
S
3
0.1 010 3
S
4
0.1 011 3
l bits/symbo 2 . 2
3 1 . 0 3 1 . 0 2 2 . 0 2 2 . 0 2 4 . 0
4
0
=
+ + + + =
=

= K
K K
l P L
l bits/symbo 13193 . 2 )
1
( log ) (
4
0
2
= =

= k
K
k
P
p S H
verified is theorem satsified, ) ( As S H L
Huffman code- process not unique
Prob. of combined symbol=another probability
in the list
place the probability of the new symbol
A) as high as possible or
B) as Low as possible
Variance (
2
)
To measure variability in codeword lengths of a
source code
As low as possible

=
=
1
0
2 2
) (
k
k
k k
L l P
Shannon Fano Coding
Principle: The codeword length increases, as the symbol
probability decreases.
Algorithm:-involves succession of divide & conquer steps
1. Divide symbols into two groups
- such that the group probabilities are as nearly equal as
possible
2. Assign the digit 0 to each symbol in the first group & digit
1 to each symbol in the second group
3. For all subsequent steps, subdivide each group into
subgroup & again repeat step 2.
4. Whenever a group contents just one symbol, no further
subdivision is possible & the codeword for that symbol is
complete.
- when all groups are reduced to one symbol, the codewords
are given by the assigned digits, reading from left to right.