Information Theory and Coding

14EC803/C

Coding for Discrete memoryless sources

Prepared by
K.Kalpana
Assistant Professor
ECE DEPT,
BEC,BAPATLA
Source encoding:

Discrete sk bk
Source
Memoryless Binary
encoder
source sequence

What is the need for Source coding?

• For efficient representation of data generated by a discrete source. The process by which this
representation is accomplished is called Source encoding.
• The device that performs the representation is called a Source encoder.
K-1
 -
Let Lmin represents the minimum possible value of L then coding efficiency is defined as

η =Lmin/L
-
For an efficient encoder η=1.

According to source encoding theorem the condition for distortion less encoding is
-L≥H(S)
Lmin=H(S)->->-> η=H(S)/L -
• For efficient signal transmission ,the redundant information should be removed from the signal prior to
transmission, this operation with no loss of information is called data compaction.

• Data compaction is achieved by assigning short descriptions to most frequent outcomes and longer
descriptions to less frequent outcomes.

• Some of data compaction schemes are:

1.Prefix coding
2.Huffmann coding
1.Prefix Coding

• Let the code word assigned to source symbol Sk be denoted by mk1, mk2, …………., mkn where all are
0’s and 1’s and n is code word length.

• Let initial part be mk1,……..,mki for some i≤n, any sequence made up of initial part of the code word is
called a Prefix of the code word.

What is Prefix code?

• A prefix code is defined as a code in which no code word is the prefix of any other code word.
Illustration of Prefix code:
Source Symbol Probability Code-1 Code-2 Code-3
of
occurrence
S0 0.5 0 0 0
S1 0.25 1 10 01
S2 0.125 00 110 011
S3 0.125 11 111 011

• Code-1 is not a Prefix code since 0 of S0 is prefix to code 00 of S2 and 1 is also a prefix to 11 of S3.
• Code-3 is also not a prefix code since 0 of s0 is a prefix to every other code word.
• Code-2 is an example of prefix code.
 • To decode a sequence of code words generated from Prefix source code, a source decoder is used
which can be represented as a ‘Decision Tree’.

• The tree has an initial state and terminal states corresponding to source symbols, the decoder always
starts at the initial state.

S0

S1

S2

Initial state

Tree for Code-2

Decision Tree diagram 0 1

Symbol Code 1
A
0
A 0 1
0
B 111 1
E B
C 1011
0
1
D 1010

E 110 D C

All the prefix codes satisfy Kraft-McMillan inequality given by

K-1 (-lk)
Σ 2 ≤1
k=0
Conclusion:
The main aim of source coding is to encode the source symbols with minimum number of
bits, so that the bandwidth required for transmission is minimized and based on efficiency parameter
optimum coding technique is determined.

References:

1.Communication Systems, 3rd edition, Simon Haykin.

2.Prefix Codes – www.CS.Princeton.edu.com