You are on page 1of 33

data compression basics

HUFFMAN CODING

1

Overview
 In this chapter, we describe a very popular coding algorithm called the Huffman coding algorithm  Present a procedure for building Huffman codes when the probability model for the source is known  A procedure for building codes when the source statistics are unknown  Describe a new technique for code design that are in some sense similar to the Huffman coding approach

2

1

data compression basics

Huffman Coding Algorithm

3

Huffman Coding Algorithm

4

2

data compression basics

Huffman Coding Algorithm

5

Huffman Coding Algorithm

6

3

data compression basics Huffman Coding Algorithm 7 Huffman Coding Algorithm 8 4 .

data compression basics Huffman Coding Algorithm 9 Huffman Coding Algorithm 10 5 .

data compression basics Minimum Variance Huffman Codes 11 Minimum Variance Huffman Codes 12 6 .

data compression basics Minimum Variance Huffman Codes 13 Minimum Variance Huffman Codes 14 7 .

data compression basics Minimum Variance Huffman Codes 15 Huffman Coding (using binary tree)  Algorithm in 5 steps: 1. Combine the smallest two by addition 4. generate code by alternating assignment of 0 and 1 16 8 . GOTO step 2. Find the grey-level probabilities for the image by finding the histogram 2. until only two probabilities are left 5. Order the input probabilities (histogram magnitudes) from smallest to largest 3. By working backward along the tree.

18 9 . In step 1 we find the histogram for the image. giving 4 possible gray levels. The image is 10 rows by 10 columns.data compression basics Huffman Coding (using binary tree)  Coding Procedures for an N-symbol source  Source reduction  List all probabilities in a descending order  Merge the two symbols with smallest probabilities into a new compound symbol  Repeat the above two steps for N-2 steps  Codeword assignment  Start from the smallest source and work back to the original source  Each merging point corresponds to a node in binary codeword tree 17 Example 1 We have an image with 2 bits/pixel.

20 10 . the probabilities are ordered. Step 1: Histogram 19 Example 1  Step 2.data compression basics Example 1  Converted into probabilities by normalizing to the total number of pixels     Gray level 0 has 20 pixels Gray level 1 has 30 pixels Gray level 2 has 10 pixels Gray level 3 has 40 pixels a.

data compression basics Example 1  Step 3. 21 Example 1  Step 4 repeats steps 2 and 3. combine the smallest two by addition. Step 4: Reorder and add until only two values remain. d. where reorder (if necessary) and add the two smallest probabilities. 22 11 .

4 branch 23 Example 1  The assigned 0 & 1 are brought back along the tree & wherever a branch occurs the code is put on both branches 24 12 .6 branch & 1 to 0. actual code assignment is made.  Start on the right-hand side of the tree and assign 0’s & 1’s  0 is assigned to 0.data compression basics Example 1  Step 5.

data compression basics Example 1  Assign the 0 & 1 to the branches labeled 0. appending to the existing code. 25 Example 1  Finally.1 & 0.3.2 branch) 26 13 . the codes are brought back one more level. & where the branch splits another assignment 0 & 1 occurs (at 0.

is the most likely to occur (40% of the time) & thus has least information in the information theoretic sense.data compression basics Example 1  Now we have Huffman code for this image  2 gray levels have 3 bits to represent & 1 gray level has 1 bit assigned  Gray level represented by 1 bit. 27 Exercise  Using the example 1. EE465: Introduction to Digital Image Processing 28 14 . g3. find a Huffman code using the minimum variance procedure.

5 (NEW) compound symbols 29 0.5 0.25 1 0.5 0.125 0.125 1 30 15 .25 0 0.5 0.25 0.5 codeword 0 0 1 10 110 111 0.5 0.data compression basics Example 2 Step 1: Source reduction symbol x S N E W p(x) 0.25 (EW) Example 2 Step 2: Codeword assignment symbol x NEW S 0 N 1 0 S EW 10 E N 1 0 W 111 110 W E 1 0 p(x) 0.5 0.25 0.5 0.125 0 0.125 0.25 0.

2 0.1 0.4 0.1 0. 31 Example 2 Step 1: Source reduction symbol x e a i o u p(x) 0. at each merging point (node).2 (ou) 0.4 0.data compression basics Example 2 0 1 NEW 0 1 0 S EW 10 N 1 0 110 W E 1 0 NEW 1 0 1 S or EW 01 N 1 0 000 001 W E The codeword assignment is not unique.6 (aiou) 0.2 0.4 0.4 (iou) 0. we can arbitrarily assign “0” and “1” to the two branches (average code length is the same). In fact.2 compound symbols 32 16 .4 0.2 0.2 0.

1 0.data compression basics Example 2 Step 2: Codeword assignment symbol x e a p(x) 0.2 0.6 0 (aiou) 0.1 0.4 0.4 0.4 0.4 (iou) 0.2 0.2 0.2 0.2 (ou) 000 0010 0011 compound symbols 33 Example 2 0 1 (aiou) e 00 01 (iou) a 000 001 (ou) i 0010 0011 o u binary codeword tree representation 34 17 .4 1 codeword 1 01 i o u 0.2 0.

4 0.122bps r l H ( X ) 0.2bps 5 H(X ) i 1 pi log 2 pi 2.878bps 35 Example 3 Step 1: Source reduction compound symbol 36 18 .122=0. we have to spend three bits per sample.1 4 2.1 4 0.1 codeword length 1 1 01 2 3 000 0010 4 0011 4 5 l i 1 pi li 0.4 1 0.2 0.078bps If we use fixed-length codes.2 0. which gives code redundancy of 3-2.2 2 0.2 3 0.1 0.data compression basics Example 2 symbol x e a i o u p(x) 0.

data compression basics Example 3 Step 2: Codeword assignment compound symbol 37 Adaptive Huffman Coding 38 19 .

data compression basics Adaptive Huffman Coding 39 Update Procedure 40 20 .

data compression basics Update Procedure 41 Update Procedure 42 21 .

data compression basics Update Procedure 43 Update Procedure 44 22 .

data compression basics Update Procedure 45 Dynamic Huffman Coding 46 23 .

t(1) 48 24 .t(1) * r represents the root * 0 represents the null node * t(1) denotes the occurrence of T with a frequency of 1 47 TE  Stage 2 (First occurrence of e) r / \ 1 t(1) / \ 0 e(1)  Order: 0.data compression basics T  Stage 1 (First occurrence of t ) r /\ 0 t(1)  Order: 0.1.e(1).

t(1).t(1) : Misfit 49 Reorder: TEN r / \ t(1) 2 / \ 1 e(1) / \ 0 n(1)  Order: 0.1.n(1).1.2.e(1).data compression basics TEN  Stage 3 (First occurrence of n ) r / \ 2 t(1) / \ 1 e(1) / \ 0 n(1)  Order: 0.n(1).e(1).2 50 25 .

data compression basics TENN  Stage 4 ( Repetition of n ) r / \ t(1) 3 / \ 2 e(1) / \ 0 n(2)  Order: 0.2.t(1).e(1).t(1).3 : Misfit 51 Reorder: TENN r / \ n(2) 2 / \ 1 e(1) / \ 0 t(1)  Order: 0.n(2).n(2) are swapped 52 26 .1.e(1).2  t(1).n(2).

1.n(2).t(1).n(2).2.3 53 TENNES  Stage 6 (First occurrence of s) r / \ n(2) 4 / \ 2 e(2) / \ 1 t(1) / \ 0 s(1)  Order: 0.4 54 27 .1.s(1).data compression basics TENNE  Stage 5 (Repetition of e ) r / \ n(2) 3 / \ 1 e(2) / \ 0 t(1)  Order: 0.e(2).t(1).e(2).

2.1.e(2).data compression basics TENNESS  Stage 7 (Repetition of s) r / \ n(2) 5 / \ 3 e(2) / \ 2 t(1) / \ 0 s(2)  Order: 0.t(1).3.s(2).3.n(2).t(1).n(2).5  s(2) and t(1) are swapped 56 28 .5 : Misfit 55 Reorder: TENNESS r / \ n(2) 5 / \ 3 e(2) / \ 1 s (2) / \ 0 t(1)  Order : 0.e(2).s(2).

1.5  N(2) and e(3) are swapped 58 29 .data compression basics TENNESSE  Stage 8 (Second repetition of e ) r / \ n(2) 6 / \ 3 e(3) / \ 1 s(2) / \ 0 t(1)  Order : 0.e(3).e(3).1.n(2).t(1).3.s(2).6 : Misfit 57 Reorder: TENNESSE r / \ e(3) 5 / \ 3 n(2) / \ 1 s(2) / \ 0 t(1)  Order : 1.s(2).t(1).n(2).3.

data compression basics TENNESSEE  Stage 9 (Second repetition of e ) r / \1 e(4) 5 0/ \1 3 n(2) 0/ \1 1 s(2) 0/ \1 0 t(1) 0  Order : 1.3.1.s(2).5 59 ENCODING The letters can be encoded as follows:     e:0 n : 11 s : 101 t : 1001 60 30 .e(4).t(1).n(2).

n frequency = { 1(4) + 2(2) + 3(2) + 1(4) } / (4+2+2+1) = 18 / 9 = 2 61 ENTROPY Entropy = i=1.( 0.11 ) = .22 + 0.(0.44 + 0.22 + 0.11 * log20.22 * log20.11) / log2 = 1.n (length*frequency)/ i=0.22 + 0.22 * log20.11 * log0.22 * log0.44 * log0.44 + 2(0.8367 62 31 .n (pi log2 pi) = .data compression basics Average Code Length Average code length = i=0.44 * log20.

The problem with static coding is that the tree has to be constructed in the transmitter and sent to the receiver. 64 32 . Considering this. the average code length will improve if the transmitted text is bigger. it need not be sent. actually the performance of dynamic coding is better. piece of code etc. Dynamic coding is better.in this exercise. Also. Since the tree in dynamic coding is constructed on the receiver as well. But.data compression basics Ordinary Huffman Coding  TENNESSE 9 0/ \1 5 e(4) 0/ \1 s(2) 3 0/ \1 t(1) n(2)      ENCODING E:1 S : 00 T : 010 N : 011 Average code length = (1*4 + 2*2 + 2*3 + 3*1) / 9 = 1.89 63 SUMMARY The average code length of ordinary Huffman coding seems to be better than the Dynamic version. The tree may change because the frequency distribution of the English letters may change in plain text technical paper.

data compression basics Summary of Huffman Coding Algorithm  Achieve minimal redundancy subject to the constraint that the source symbols be coded one at a time  Sorting symbols in descending probabilities is the key in the step of source reduction  The codeword assignment is not unique. it does not know where to start) 65 33 . Exchange the labeling of “0” and “1” at any node of binary codeword tree would produce another solution that equally works well  Only works for a source with finite number of symbols (otherwise.