Professional Documents
Culture Documents
ELEMENTS OF ENCODING
3.1 Purpose of encoding; Separable
binary codes
2
Huffman Coding
Huffman coding results in an optimal code. It is the code that has the highest
efficiency.
The Huffman coding procedure is as follows:
1. List the source symbols in order of decreasing probability.
2. Combine the probabilities of the two symbols having the lowest probabilities
and reorder the resultant probabilities, this step is called reduction 1. The same
procedure is repeated until there are two ordered probabilities remaining.
3. Start encoding with the last reduction, which consists of exactly two ordered
probabilities. Assign 0 as the first digit in the code word for all the source
symbols associated with the first probability; assign 1 to the second probability.
4. Now go back and assign 0 and 1 to the second digit for the two probabilities
that were combined in the previous reduction step, retaining all the source
symbols associated with the first probability; assign 1 to the second probability.
5. Keep regressing this way until the first column is reached.
6. The code word is obtained tracing back from right to left.
3
Huffman Encoding - Example
5
6
7
8
The source coding theorem
The source coding theorem states that for a DMS X,
with entropy H (X), the average code word length L
per symbol is bounded as L≥ H (X).
L can be made as close to H (X) as desired for some
suitable chosen code.
Thus, with 𝐿𝑚𝑖𝑛 = 𝐻 (𝑋)
The code efficiency can be rewritten as 𝜂 = 𝐻(𝑋)/𝐿
Re-written as:
M= bits per letter i.e M=2 for binary codes
9
Huffman as an optimal code
What is an optimal code?
1. The code efficiency is maximum
How and when?
2. The code gives the lowest possible average codeword
length for a given M which results in maximum efficiency and
minimum redundancy.
3. “Compression” should be maximum for an efficient source
encoder
Moreover, in an optimal code, symbols that occur more
frequently (have a higher probability of occurrence) will have
shorter codewords than symbols that occur less frequently.
10
Shannon Fano Vs Huffman
Coding
11
Shannon fano and Huffman Code
14
Practice problem
Take M=4
P (M=3) S. F H
code Code
0.2
0.2
0.15
0.15
0.1
0.1
0.05
0.05
15
Thank you