TSBK01 Image Coding and Data Compression

Lecture 3: Source Coding Theory

Jörgen Ahlberg Div. of Sensor Technology Swedish Defence Research Agency (FOI)

Code trees and tree codes 3. Coding & codes 2. Optimal codes  The source coding theorem .Outline 1.

 Types of codes: – FIFO: Fixed input (i. # source symbols. fixed output. variable output. – FIVO: Fixed input. Should be comma-free. – VIVO: Variable input.e. . variable output.. – VIFO: Variable input.Part 1: Coding & codes  Coding: To each sorce symbol (or group of symbols) we assign a codeword from an extended binary alphabet. fixed output (i.e.  FIVO and VIVO are called variable length codes (VLC).. # code symbols).

a4} probabilities P(a1) = ½ P(a2) = ¼ P(a3) = P(a4) = 1/8. ….  a1 a2 a3 a4 FIFO: 00 01 10 11 FIVO: 0 01 110 111 .Example  Assume a memoryless source with alphabet A = {a1.

Four different classes Singular Non-singular Uniqely decodable Instantaneous a1 a2 a3 a4 0 0 0 0 0 010 01 10 10 0 Decoding probem: 010 could 00 10 mean a1a4 or a2 or a3a1. but the first symbol (a3 or a4) cannot be decoded until the third ’1’ arrives (Compare 11010 and 110010). 11 110 110 111 All codes Non-singular codes Decoding probem: 1100000000000000001… is uniqely decodable. Uniqely decodable Instantaneous .

. – Don’t care whether it’s a3 or a4.Data Compression  Efficient codes utilize the following properties: – Uneven symbol probability – Inter-symbol dependence (memory source) – Acceptable distortion  Examples: – The FIVO example – ”There’s always an a3 after a1”.

0 a1 0 1 1 1 a4 a2 0 a3  Tree codes are comma-free and instantaneous. …. our favourite example code {a1. a4} = {0. 10.Part 2: Code Trees and Tree Codes  Consider. 111}.  The codewords are the leaves in a code tree. 110. again. No codeword is a prefix of another! .

. it is possible to construct a code tree. if this is valid for a given set of codeword lengths.4)  Conversely.Kraft’s Inequality  For a uniqely decodable code with codeword lengths li we have (Proof: Sayood 2.

Kraft’s Inequality and Tree Codes  If KI is valid for a set of codeword lengths. – The tree then has 2lmax leaves. – Place the codewords. – Let’s illustrate. . there is a tree code with such lengths.  Proof: Create a maximal tree with the size from the longest codeword length lmax. cut the tree. and use KI to prove that there is enough leaves.

.. Possible whenever KI is valid. 3.. Place l2 in the tree.e.. i. 2lmax (1 –  2-l1) leaves remain. 2.and 2lmax – 2lmax – l1 = 2lmax (1 – 2-l1) leaves remain. Then 2lmax – l1 leaves disappear. 3} and {li} = {1. 2. .  2-l1 · 1. 3} ! Cannot be used: cut! Place l1 in the tree. . Then 2lmax (1 – 2-l1 – 2-l2) leaves remain. After placing N codeword lengths.lmax = 3 leads to this tree: l1 l2 l4 l3 Try with {li} = {1. 2.

Part 3: Optimal Codes Average codeword length [bits/codeword] Kraft’s Inequality Optimal codeword lengths ) the entropy limit is reached! But what about the integer constraints? li = – log pi is not always an integer! .

.The Source Coding Theorem  Assume that the source X is memory-free.  We have:  We can come arbitrarily close to the entropy! . blocks of n symbols. and create the tree code for the extended source.e. i.

e. 1952) to create an optimal tree code! . the Huffman algorithm (D. p2 = 0.25.75 e = 1  Instead..75 ) l1 = log 4 = 2 l2 = d – log 0. use.Huffman.g.In Practice  Two practical problems need to be solved: – Bit-assignment – The integer contraint  Theoretically: Chose li = – log pi – Rounding up not always the best! – Example: Binary source p1 = 0.

. Kraft’s Inequality.Summary  Coding: Assigning binary codewords to (blocks of) source symbols.  The Source Coding Theorem.  Tree code .  Variable-length codes (VLC) and fixed-length codes.  Instantaneous codes ½ Uniqely decodable codes ½ Non-singular codes ½ All codes  Tree codes are instantaneous.