Image and video compression:

Chapter V: Lossless compression methods

Wassim Hamidouche

Associate Professor
INSA Rennes, CNRS, IETR - UMR 6164, FRANCE
whamidou@insa-rennes.fr

Chapter V: Lossless compression methods, May 6th, 2022

Motivation of lossless coding
Motivation for lossless image compression

ˆ Lossless coding
ˆ Decode a signal is an exact replica of the original
ˆ There is no loss of information between the encoder and the
decoder
ˆ Exploit statistical redundancy to achieve a bit rate close to the
entropy of the source

W. Hamidouche — Motivation of lossless coding 1/68

Motivation for lossless image compression

ˆ Motivations of using lossless coding

ˆ Applications where image features are important and how
these would be affected by lossy image coding: medical
imaging, errors in a diagnosis or interpretation
ˆ Applications where it is essential that data is preserved intact:
for example text, or numerical information in a computer file
ˆ Lossless compression methods can provide an efficient means of
representing the sparse matrices that result from quantization
of transform coefficients in a lossy compression system

W. Hamidouche — Motivation of lossless coding 2/68

Loss coding methods

ˆ Approaches exploiting data statistics to provide lossless

compression
ˆ Huffman coding: coding individual symbols at a rate close to
the first order entropy
ˆ Arithmetic coding: achieving fractional bit rates for symbols

W. Hamidouche — Motivation of lossless coding 3/68

Loss coding methods

ˆ Approaches exploiting data statistics to provide lossless

compression
ˆ Predictive coding: Exploits data correlations, as opposed to
symbol redundancies, w/o quantization for lossless image
compression (preprocessing)
ˆ Dictionary-based methods: Lempel-Ziv (LZ) and
Lempel–Ziv–Welch (LZW) algorithms used for source data
with unknown statistics (text file).

Lossless compression methods, if used in isolation, are typically

restricted to ratios of around 4:1.

W. Hamidouche — Motivation of lossless coding 4/68

Symbol encoding
Symbol encoding

ˆ A generic model for lossless compression

ˆ Mapping input symbols from the source into codewords
ˆ Mapping must be conditioned by a representative probability
model (compression)

W. Hamidouche — Symbol encoding 5/68

Symbol encoding

ˆ A generic model for lossless compression

ˆ A probability model can be derived directly from the input
data being processed, from a priori assumptions
ˆ Compression efficiency depends mainly on the probability
model accuracy
ˆ Ideal symbol-to-code mapper will produce a codeword
requiring log2 (1/Pi ) bits

W. Hamidouche — Symbol encoding 6/68

Symbol encoding

ˆ Entropy, efficiency, and redundancy

¯ the
For a given encoder which codes a signal at an average rate l,
coding efficiency can be defined as

H
E= ¯ (1)
l
A coding scheme can also be characterized by its redundancy, R:

l¯− H
R= 100% (2)
H

W. Hamidouche — Symbol encoding 7/68

Symbol encoding

ˆ Prefix codes and unique decodability

ˆ No codeword is the prefix of any other

Alternative codes for a given probability distribution for the

four-letter alphabet {a1 , a2 , a3 , a4 }

Letter Probability C1 C2 C3
a1 0.5 0 0 0
a2 0.25 0 1 10
a3 0.125 1 00 110
a4 0.125 10 11 111
Average length 1.25 1.25 1.75

W. Hamidouche — Symbol encoding 8/68

Symbol encoding

ˆ Coding redundancy

Calculate the first order entropy of the alphabet and the

redundancy of the encoding:

Letter Probability Codeword

s0 0.06 0110
s1 0.23 10
s2 0.30 00
s3 0.15 010
s4 0.08 111
s5 0.06 0111
s6 0.06 1100
s7 0.06 1101
W. Hamidouche — Symbol encoding 9/68
Huffman coding
Huffman coding

ˆ The basic algorithm

ˆ Represents each symbol si with a variable-length binary
codeword ci
ˆ Length of ci is determined by rounding Hi up to the nearest
integer.
ˆ Produce prefix codes: no valid codeword is the prefix of any
other

W. Hamidouche — Huffman coding 10/68

Huffman coding

ˆ Huffman algorithm

Algorithm 1 Huffman tree formation

1: Create ranked list of symbols s0 , s1 , . . . , sN in deceasing proba-
bility order
2: repeat
3: Combine pair of symbols with lowest probabilities
4: Update probability of new node P(si′ ) = P(si ) + P(sj )
5: Sort intermediate nodes: order si′ in the ranked list
6: until A single root node is formed with P = 1
7: Label the upper path of each branch node with a binary 0 and
the lower path with a 1
8: Scan tree from root to leaves to extract Huffman codes for each
symbol

W. Hamidouche — Huffman coding 11/68

Huffman coding

ˆ Example of Huffman encoding

Consider a source which generates five symbols

S = {s1 , s2 , s3 , s4 , s5 } where P(s1 ) = 0.2, P(s2 ) = 0.4,
P(s3 ) = 0.2, P(s4 ) = 0.15, P(s5 ) = 0.05.
Compute the Huffman tree for this alphabet.

W. Hamidouche — Huffman coding 12/68

Huffman coding

ˆ Example of Huffman encoding

0
(0.4) (0.4) (0.4) (0.6)

(0.2) (0.4) 0 (0.4)

(0.2)
1
0
(0.2) (0.2) (0.2)
1
0
(0.15)
(0.2)
(0.05) 1
1
By tracing each path through the graph, we can see that the
Huffman codewords for this alphabet are s1 = 01, s2 = 1,
s3 = 000, s4 = 0010, s5 = 0011 and in average codeword length of
2.2 bits/symbol.
W. Hamidouche — Huffman coding 13/68
Huffman coding

ˆ Minimum variance Huffman coding

ˆ Minimizes the length of the longest codeword: minimizing
dynamic variations in transmitted bit rate
ˆ Forming a new ranked list of nodes, the newly formed node is
inserted in the list as high up as possible, without destroying
the ranking order

W. Hamidouche — Huffman coding 14/68

Huffman coding

ˆ Minimum variance Huffman coding

Algorithm 2 Minimum variance Huffman tree formation

1: Create ranked list of symbols s0 , s1 , . . . , sN in deceasing proba-
bility order
2: repeat
3: Combine pair of symbols with lowest probabilities
4: Update probability of new node P(si′ ) = P(si ) + P(sj )
5: Sort intermediate nodes: insert si′ as high up as possible in
the ranked list
6: until A single root node is formed with P = 1
7: Label the upper path of each branch node with a binary 0 and
the lower path with a 1
8: Scan tree from root to leaves to extract Huffman codes for each
symbol
W. Hamidouche — Huffman coding 15/68
Huffman coding

ˆ Minimum variance Huffman coding Example

Consider a source which generates five symbols

S = {s1 , s2 , s3 , s4 , s5 } where P(s1 ) = 0.2, P(s2 ) = 0.4,
P(s3 ) = 0.2, P(s4 ) = 0.15, P(s5 ) = 0.05.
Compute the Huffman tree for this alphabet.

W. Hamidouche — Huffman coding 16/68

Huffman coding

ˆ Minimum variance Huffman coding Example

0
(0.4) (0.4) (0.4) (0.6)

(0.2) (0.4) 0 (0.4)

(0.2) 1
0
(0.2) (0.2) (0.2)
1
0
(0.15)
(0.2)
(0.05) 1
1
By tracing each path through the graph, we can see that the
Huffman codewords for this alphabet are s1 = 10, s2 = 00,
s3 = 11, s4 = 010, s5 = 011 and in average codeword length of 2.2
bits/symbol and max of 3 rather than 4.
W. Hamidouche — Huffman coding 17/68
Huffman decoding

ˆ Huffman decoding algorithm

Huffman decoding is a simple process that browses the input bits

to switch between the upper and lower path in the graph (used for
encoding by the encoder) until a leaf is reached and the codeword
identified.

W. Hamidouche — Huffman coding 18/68

Huffman decoding i

ˆ Huffman decoding algorithm

Require: Bitstream representing a sequence of symbols

b[0], b[1], . . . , b[K − 1] and Huffman tree structure
1: i ← 0
2: repeat
3: repeat
4: Start at root node
5: if b[i] == 0 then
6: Select upper branch
7: else
8: Select lower branch
9: end if
W. Hamidouche — Huffman coding 19/68
Huffman decoding ii

10: i ←i +1
11: until Leaf node reached
Ensure: Symbol
12: until i == K

W. Hamidouche — Huffman coding 20/68

Huffman coding

ˆ Properties of Huffman codes: Kraft inequality

An alphabet with N symbols S = {si }, if encoded using a binary

prefix code and ranked according to their probability, Pi , we will
have l1 ≤ l2 ≤ · · · ≤ lN . The Kraft inequality states:

N
X
2−li ≤ 1. (3)
i=1

since li = log2 P1i

In the case of a dyadic source (all codewods lengths are integers)

N
X
l¯ = li Pi = H(S). (4)
i=1

W. Hamidouche — Huffman coding 21/68

Huffman coding

ˆ Properties of Huffman codes: Kraft inequality

The average length of a codeword in a Huffman encoded alphabet

will depend primarily on the distribution of probabilities in the
alphabet and on the number of symbols in the alphabet

N   N  
X 1 X 1
l¯ = log2 Pi < log2 + 1 Pi < H(S) + 1. (5)
Pi Pi
i=1 i=1

The Huffman code for an alphabet, S, therefore has an average

length l¯ bounded by

H(S) ≤ l¯ ≤ H(S) + 1. (6)

W. Hamidouche — Huffman coding 22/68

Huffman coding

ˆ Optimal code must satisfy the following conditions

ˆ For any two symbols si and sj in an alphabet S, if
P(si ) > P(sj ) then li ≤ lj
ˆ The two symbols with the lowest probabilities have equal and
maximum lengths
ˆ In a tree corresponding to the optimal code, each intermediate
node must have two branches
ˆ If we collapse several leaf nodes connected to a single
intermediate node into a single composite leaf node, then the
tree will remain optimal

W. Hamidouche — Huffman coding 23/68

Huffman coding

ˆ Limitations of Huffman coding :

ˆ Encoding one symbol at a time
ˆ Dependency on a precomputed encoding and decoding
structure
ˆ Huffman coding in practice: consider an 8-bit source with the
following probabilities:

Value (x) P(x) I (x) Code bits

0 0.20 2.232 10 2
+1 0.10 3.322 110 3
-1 0.10 3.322 1110 4
+2 0.05 4.322 11110 5
-2 0.05 4.322 11111 5
All other values 0.5 1 0+value 9

W. Hamidouche — Huffman coding 24/68

Huffman coding

ˆ Huffman adaptive coding:

W. Hamidouche — Huffman coding 25/68

Symbol formation and encoding
Symbol formation and encoding

ˆ Dealing with sparse matrices

ˆ Predictive and transform coding of block produce after
quantisation sparse matrix
ˆ Matrix contains a relatively small proportion of nonzero entries
with most of its energy compacted toward the lower
frequencies

W. Hamidouche — Symbol formation and encoding 26/68

Symbol formation and encoding

ˆ Dealing with sparse matrices

ˆ Run-length coding (RLC) can be used to efficiently represent
long strings of identical values by grouping them into a single
symbol which codes the value and the number of repetitions
ˆ Code runs of zeros in combination with the value of the next
nonzero entry
ˆ Transform the sparse block into a vector with zig-zag scan
then encode the vector in RLC

W. Hamidouche — Symbol formation and encoding 27/68

Symbol formation and encoding

ˆ Run-length encoding of sparse energy-compact matrices

Example
 
8 5 2 0
3 0 0 0
C Q =  

0 0 0 0
1 0 0 0

Perform zig-zag scanning to produce a 1D vector

cQ = 8 5 3 0 0 2 0 0 0 1 0 0 0 0 0 0
h i

Present this vector using run/value symbols:

c Q = (0/8) (0/5) (0/3) (2/2) (3/1) EOB

h i

W. Hamidouche — Symbol formation and encoding 28/68

Symbol formation and encoding

ˆ Symbol encoding in JPEG

ˆ Picture partitioning in 8 × 8 blocks followed by DCT and
quantisation
ˆ DC coefficients are encoded in DPCM from the previous block
ˆ AC coefficients are rearranged in a vector with zig-zag scan
then encoded in RLC
Input video
YcbCy DC
DCT-II Q DPCM
Entropy JPEG bitsteam
coder

RLC
AC

Quantiser Huffman
tables tables

Tables

W. Hamidouche — Symbol formation and encoding 29/68

Symbol formation and encoding

ˆ Symbol encoding in JPEG Example

Consider the following quantized luminance DCT matrix.

Assuming that the value of the DC coefficient in the preceding
block is 40, what is the JPEG symbol sequence for this matrix?
Using the default JPEG Huffman tables, compute its Huffman
codes and the resulting output bitstream

 
42 16 2 0 0 0 0 0
−21 −15 0 0
 
0 0 0 0 
CQ
 
=
 10 3 −3 0 0 0 0 0  
 −2 2 2 0 0 0 0 0 
 
0 −1 0 0 0 0 0 0

W. Hamidouche — Symbol formation and encoding 30/68

Symbol formation and encoding

ˆ Symbol encoding in JPEG Example

DC coefficient coding: The prediction residual for the current

DC coefficient is DCi (pred) = DCi − DCi−1 = 2. Size = 2 and
amplitude = 2
Code DC : 011 10

Category Code length Code word

0 2 00
1 3 010
2 3 011
3 3 100

W. Hamidouche — Symbol formation and encoding 31/68

Symbol formation and encoding

ˆ Symbol encoding in JPEG Example

AC coefficients coding : Zig-zag scan:

" #
16 −21 10 −15 0 0 0 3 −2 0 · · ·
CQ =
· · · 2 −3 0 0 0 0 0 2 −1 0 · · ·

W. Hamidouche — Symbol formation and encoding 32/68

Symbol formation and encoding

ˆ Symbol encoding in JPEG Example

Coding of the first and second AC coefficients (0/16)

size = 5, amplitude = 16 (0/-21) size = 5, amplitude = 21
Code1 : 11010 10000
Code2 : 11010 01010
Coeff run/size Huffman code Amplitude bits
16 0/5 11010 10000 10
-21 0/5 11010 01010 10
10 0/4 1011 1010 8
-15 0/4 1011 0000 8
3 3/2 111110111 11 11
-2 0/2 01 01 4
2 1/2 11011 10 7
-3 0/2 01 00 4
2 5/2 11111110111 10 13
-1 0/1 00 0 3
EOB 0/0 1010 - 4
Total ACs 82

W. Hamidouche — Symbol formation and encoding 33/68

Symbol formation and encoding

ˆ Symbol encoding in JPEG Example

Total bits of 82+5 = 87 instead of 64 × 8bits= 512,

CR:5.9:1 or 1.36 bpp

W. Hamidouche — Symbol formation and encoding 34/68

Golomb coding
Golomb coding

ˆ Golomb coding is a variable length coding:

ˆ Shorter codewords can be assigned to symbols with higher
probabilities and vice versa
ˆ Simply encoded and decoded algorithmically without the need
for look-up tables
ˆ Possible to find an encoding that is an optimal prefix code for
geometric distribution
ˆ Golomb family includes 1) unary, 2) Golomb, 3) Golomb–Rice,
4) Exponential Golomb code

W. Hamidouche — Golomb coding 35/68

Golomb coding

ˆ Unary codes:

Encodes a nonnegative integer, i, by i 0s followed by a single 1 (or

the other way around).

W. Hamidouche — Golomb coding 36/68

Golomb coding

ˆ Golomb codes:

Divides all index values i into equal-sized groups of size m. The

codeword is then constructed from a unary code that characterizes
each group, followed by a fixed-length code vi that specifies the
remainder of the index that has been encoded. The remainder vi is
computed as follows:
 
i
vi = i − m (7)
m

W. Hamidouche — Golomb coding 37/68

Golomb coding

ˆ Golomb–Rice codes:

The Golomb–Rice code is a special case of Golomb code where

m = 2k

W. Hamidouche — Golomb coding 38/68

Golomb coding

ˆ Exponential Golomb codes:

Divides the alphabet into exponentially increasing group sizes.

The Exp-Golomb code for a symbol index i comprises a unary code
(ζi zeros, followed by a single 1), concatenated with a fixed-length
code, vi : 
v = 1 + i − 2ζi
i
(8)
ζi = ⌊log (i + 1)⌋
2

W. Hamidouche — Golomb coding 39/68

Golomb coding

ˆ Example Golomb codes for index values:

i Unary Golomb-Rice (m = 4) Exp-Glomb

0 1 1|00 1|
1 01 1|01 01|0
2 001 1|10 01|1
3 0001 1|11 001|00
4 00001 01|00 001|01
5 000001 01|01 001|10
6 0000001 01|10 001|11
7 00000001 01|11 0001|000
8 000000001 001|00 0001|001
9 0000000001 001|01 0001|010

W. Hamidouche — Golomb coding 40/68

Golomb coding

ˆ Example Exp-Golomb encoding and decoding:

1) What is the Exp-Golomb code for the symbol index 22810 ?

2) Show how the Exp-Golomb codeword 00000100111 would be
decoded and compute the value of the corresponding symbol index.

W. Hamidouche — Golomb coding 41/68

Golomb coding

ˆ Example Exp-Golomb encoding and decoding:

1) ζi = ⌊log2 (i + 1)⌋ = 7
vi = 1 + i − 2ζi = 10110 = 11001012
The code is then : 00000001 1100101
2) Decode the code 000001 00111:
i = vi − 1 + 2ζi = 38 with ζi = 5 and vi = 001112 = 710

W. Hamidouche — Golomb coding 42/68

Arithmetic coding
Arithmetic coding

ˆ Principal of arithmetic coding:

ˆ Encode a group of symbols in one fractional number
ˆ Use of a cumulative probability distribution as a basis for
codeword generation for a sequence of symbols
ˆ Subdivide the half-open interval [0, 1) according to the symbol
cumulative probabilities

W. Hamidouche — Arithmetic coding 43/68

Arithmetic coding

ˆ Basic arithmetic encoding algorithm:

Let us assume that we have a set of M symbols that comprises our

source alphabet S = {s0 , s1 , s2 , . . . , sM−1 }
The source symbol probabilities are therefore defined as

P(m) = P(x = m), m = 0, . . . , M − 1 (9)

The cumulative probability distribution

m
X
Px (m) = P(x = k), m = 0, . . . , M − 1 (10)
k=0

W. Hamidouche — Arithmetic coding 44/68

Arithmetic coding

ˆ Basic arithmetic encoding algorithm:

Let us consider a sequence of N input symbols

X = xi , i = 1, · · · , N , and we define l(i) to be the lower limit at
stage i of the encoding and u(i) to be the corresponding upper
limit, then at the start of encoding:

l(0) = Px (−1) = 0,
u(0) = Px (M − 1) = 1

W. Hamidouche — Arithmetic coding 45/68

Arithmetic coding

ˆ Basic arithmetic encoding algorithm:

After the first symbol is encoded, i = 1,

l(1) = Px (x1 − 1)
u(1) = Px (x1 )
After the second symbol, i = 2, we have

l(2) = l(1) + (u(1) − l(1))Px (x2 − 1),

u(2) = l(1) + (u(1) − l(1))Px (x2 )

W. Hamidouche — Arithmetic coding 46/68

Arithmetic coding

ˆ Basic arithmetic encoding algorithm:

In general after the ith symbol, xi , we have

l(i) = l(i − 1) + (u(i − 1) − l(i − 1)) Px (xi − 1),

u(i) = l(i − 1) + (u(i − 1) − l(i − 1)) Px (xi )

W. Hamidouche — Arithmetic coding 47/68

Arithmetic coding

ˆ Example Arithmetic encoding

Consider a sequence of symbols {s2 , s2 , s3 } from an alphabet
{s1 , s2 , s3 } with probabilities {P(1), P(2), P(3)}.

(0)=1 (1) (2) (3)









(0)=0 (1) (2) (3)


W. Hamidouche — Arithmetic 
coding   48/68
Arithmetic coding

ˆ Example Arithmetic encoding - numerical example

Using the same symbols and sequence in the previous example with
P(1) = 0.5, P(2) = 0.25, P(3) = 0.25. Compute the arithmetic
code for the sequence {s2 , s2 , s3 }.

(0)=1 (1)=0.75 (2)=0.6875 (3)=0.6875








(0)=0 (1)=0.5 (2)=0.625 (3)=0.671875

   
W. Hamidouche — Arithmetic coding 49/68
Arithmetic coding

ˆ Arithmetic encoding algorithm

Algorithm 4 Arithmetic encoding

1: Initialize upper and lower limits: l(0) = 0 u(0) = 1
Require: Sequence of N symbols xi ; and cumulative probabilities
2: i ← 0
3: repeat
4: i ←i +1
5: Compute the lower interval limit: l(i) = l(i − 1) +
(u(i − 1) − l(i − 1)) Px (xi − 1)
6: Compute the higher interval limit: u(i) = l(i − 1) +
(u(i − 1) − l(i − 1)) Px (xi )
7: until i == N
Ensure: Arithmetic codeword v ∈ [l(N), u(N))

W. Hamidouche — Arithmetic coding 50/68

Arithmetic coding

ˆ The basic arithmetic decoding algorithm

ˆ The encoding algorithm creates an arithmetic codeword,
v̂ ∈ {l(N), u(N)}
ˆ The purpose of the decoder is to generate the same output
sequences {x1 , x2 , · · · , xN }, as that processed at the encoder,
where {xi = k} ← sk
ˆ Recursively updating the upper and lower limits of the
half-open interval

W. Hamidouche — Arithmetic coding 51/68

Arithmetic decoding

ˆ The basic arithmetic decoding algorithm

Initializing lower and upper limits

l(0) = Px (0) = 0, u(0) = Px (M − 1) = 1

The sequence of symbol indices xi is computed as:

x1 = {x : Px (x1 − 1) ≤ v̂ < Px (x1 )}

In general:

xi = {x : l(i) ≤ v̂ < u(i)}

W. Hamidouche — Arithmetic coding 52/68

Arithmetic decoding

ˆ The basic arithmetic decoding algorithm

We need to update the lower and upper limits

l(i) = l(i − 1) + (u(i − 1) − l(i − 1)) Px (xi − 1)

u(i) = l(i − 1) + (u(i − 1) − l(i − 1)) Px (xi )

W. Hamidouche — Arithmetic coding 53/68

Arithmetic decoding

ˆ Arithmetic decoding algorithm

Algorithm 5 Arithmetic decoding

1: Initialize upper and lower limits: l(0) = 0 u(0) = 1
Require: Arithmetic code tag v̂ and cumulative probabilities
2: i ← 0
3: repeat
4: i ←i +1
5: Compute symbol index i: xi = {x : l(i) ≤ v̂ < u(i)} where
l(i) = l(i − 1) + (u(i − 1) − l(i − 1)) Px (xi − 1)
and u(i) = l(i − 1) + (u(i − 1) − l(i − 1)) Px (xi )
Ensure: Symbol sxi
6: until i == N

W. Hamidouche — Arithmetic coding 54/68

Arithmetic coding

ˆ Example Arithmetic decoding - numerical example

Consider the decoding of arithmetic code 0.538, which represents
three symbols sequence from the alphabet {s1 , s2 , s3 , s4 } with
P(s1 ) = 0.6, P(s2 ) = 0.2, P(s3 ) = 0.1 and P(s4 ) = 0.1.

(0)=1 (1)=0.6 (2)=0.54 (3)=0.54






(0)=0 (1)=0 (2)=0.48 (3)=0.534

   
W. Hamidouche — Arithmetic coding 55/68
Advantages and properties of arithmetic coding

ˆ Benefits of arithmetic coding

ˆ Beneficial with highly skewed probability distributions.
ˆ Separates the modeling process from the coding process
ˆ Symbol statistics can be simply adapted to changing
conditions without major recomputation of coding tables.
ˆ For the case of binary arithmetic coding, only one probability
value is required

W. Hamidouche — Arithmetic coding 56/68

Advantages and properties of arithmetic coding

ˆ Length of arithmetic codes

Based on the Kraft inequality, we can have :

   
1 1
log2 Q ≤ lv ≤ log2 Q +1 (11)
P(k) P(k)

 
1
lv = log2 Q +1 (12)
P(k)

W. Hamidouche — Arithmetic coding 57/68

Binary implementations of arithmetic coding

ˆ Binary arithmetic coding

Consider a source with four symbols with probabilities P(s0 ) = 0.5,

P(s1 ) = 0.25, P(s2 ) = 0.125 and P(s3 ) = 0.125. Perform
arithmetic encoding of the sequence {s1 , s0 , s2 }.
Stage 1: s1 I1 = [0.5, 0.75)10 = [0.10, 0.11)2 . Send code 1.
Stage 2: s0 I2 = [0.5, 0.625)10 = [0.100, 0.110)2 . Send code 0.
Stage 3: s2 I3 = [0.59375, 0.609375)10 = [0.10011, 0.100111)2 .
Send code 011.
We can transmit any value in this range as our arithmetic code
ṽ ∈ [0.10011, 0.100111)2 . i.e. ṽ = 100110.

W. Hamidouche — Arithmetic coding 58/68

Binary implementations of arithmetic coding

ˆ Binary arithmetic coding

Consider a source with four symbols with probabilities P(s0 ) = 0.5,

P(s1 ) = 0.25, P(s2 ) = 0.125 and P(s3 ) = 0.125. Perform
arithmetic encoding of the sequence {s1 , s0 , s2 }.
(0)=1 (1)=0.10 (2)=0.101 (3)=0.100111



(0)=0 (1)=0.1 (2)=0.01 (3)=0.01011

   
W. Hamidouche — Arithmetic coding 59/68
Binary implementations of arithmetic coding

ˆ Binary arithmetic decoding

Rx bit Interval Symbol

1 [0.5, 1)10 = [0.1, 1.0)2 -
0 [0.5, 0.75)10 = [0.10, 0.11)2 s1
0 [0.5, 0.625)10 = [0.100, 0.110)2 s0
1 [0.5625, 0.625)10 = [0.1001, 0.1010)2 -
1 [0.59375, 0.625)10 = [0.10011, 0.10100)2 -
0 [0.59375, 0.609375)10 = [0.10011, 0.100111)2 s2

W. Hamidouche — Arithmetic coding 60/68

Context-based arithmetic coding

ˆ Context-based arithmetic coding

ˆ Exploit the Ability to separate modeling from coding
ˆ Assign a probability distribution to the symbols based on the
context created by previously processed symbols
ˆ For each symbol received, the appropriate probability model is
invoked

W. Hamidouche — Arithmetic coding 61/68

Context-based arithmetic coding

ˆ Context-based arithmetic coding

ˆ For each symbol received, the appropriate probability model is
invoked
ˆ The context can either be fixed, based on prior knowledge, or
modified in real-time, based on the frequency of symbols
actually processed
ˆ The decoder must be aware of the various models in use at the
encoder,
ˆ Adaptation at encoder and decoder can be synchronized

W. Hamidouche — Arithmetic coding 62/68

Context-based arithmetic coding

ˆ Context-based arithmetic coding Example

Consider a source with three symbols with the following

precomputed conditional probabilities. Perform arithmetic
encoding of the sequence

Initial Context 1 Context 2 Context 3

P(s0 ) = 0.5 P(s0 |s0 ) = 0.125 P(s0 |s1 ) = 0.25 P(s0 |s2 ) = 0.625
P(s1 ) = 0.375 P(s1 |s0 ) = 0.75 P(s1 |s1 ) = 0.25 P(s1 |s2 ) = 0.125
P(s2 ) = 0.125 P(s2 |s0 ) = 0.125 P(s2 |s1 ) = 0.5 P(s2 |s2 ) = 0.25

W. Hamidouche — Arithmetic coding 63/68

Context-based arithmetic coding

ˆ Context-based arithmetic coding Example

The encoding proceeds as follows

(0)=1 (1)=0.5 (2)=0.4375 (3)=0.4375

0.875 0.4375

0.5 0.28125 

0.125

(0)=0 (1)=0 (2)=0.125 (3)=0.28125

   

W. Hamidouche — Arithmetic coding 64/68

Conclusion
Conclusion

ˆ Lossless coding is important in many applications

ˆ Entropy coding reduces the statistical redundancy: leverage
symbols distribution and variable length code
ˆ Huffman reaches the entropy for dyadic source
ˆ Arithmetic coding enables fractional bitrate for symbols
(adopted in AVC, HEVC, and VVC standards)

W. Hamidouche — Conclusion 65/68