1
Arithmetic coding
Elias coding
Arithmetic coding as finiteprecision Elias
coding
Redundancy due to finite precision
Multiplicationfree arithmetic coding
Contextadaptive coding
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2
Elias coding
Entropy coding algorithm for sequences of
symbols x with general (conditional) probabilities
Representation of x by a subinterval of the unit
interval [0,1)
Width of the subinterval is approximately equal to
the probability f
X
(x)
Subinterval for x can be determined by recursive
subdivision algorithm
Represent x by shortest binary fraction in the
subinterval
Subinterval of width f
X
(x) is guaranteed to contain
one number that can be represented by L binary
digits, with
( )
f
X
x
0.1111
0.111
0.1101
0.11
0.1011
0.101
0.1001
0.1
0.0111
0.011
0.0101
0.01
0.0011
0.001
0.0001
0
0
1
( )
2
log L f ~ ÷
X
x
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3
Example: Elias coding of memoryless binary source
( ) ( )
1 3
0 1
4 4
X X
f f = =
0
1
1
4
n
c
n n
c a +
0
1
4
0
1
1
16
4
16
1
7
64
16
64
0
37
256
28
256
1
121
1024
148
1024
1 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4
Example: Elias coding of memoryless ternary source
Courtesy: T. Wiegand
f
X
a
( )
= f
X
b
( )
=
1
4
f
X
c
( )
=
1
2
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5
Elias coding: choose binary fraction in subinterval
Uniquely identify interval [c
n
,c
n
+a
n
) by a binary fraction
Length of bitstring
Bitstring
bits
0. [ , )
n
n n n
L
bbbbbb b c c a e +
L
n
= log
2
1
a
n
(
(
(
+1= h
X
0:n÷1
x
0:n÷1
( )
(
(
+1
ˆ 2 2 1
n n
L L
n n n
c c c
÷
(
= + >
¸ ¸
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6
Successive decoding
Elias bit string can be decoded symbol by symbol, starting with the
first symbol
For each symbol n=0,1,2, . . .
• Calculate the intervals [c
n
,c
n
+a
n
) corresponding to all possible x
n
• Determine the interval for which
• Emit the corresponding x
n
)
ˆ ,
n n n n
c c c a e +
¸
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7
Elias coding for memoryless source
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
÷
÷ +
Interval
width
Interval
lower limit
Start with unit
interval
nth symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1
where { , , ,..., }
i
X i X j
j
X K
F f o o
o o o o
÷
=
÷
=
=
¿
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8
Elias coding for changing probabilities
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n
n
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
÷
÷ +
Interval
width
Interval
lower limit
Start with unit
interval
nth symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1
where { , , ,..., }
n n
n
i
X i X j
j
X K
F f o o
o o o o
÷
=
÷
=
=
¿
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9
Elias coding for Markov random processes
Elias coding algorithm for Markovp sources
{ }
( )
( )
: 1
: 1
0 0
1  : 1
1  : 1
Initialize 0 and 1.
For each 0,1,...
Set min ,
Update ,
Update ,
n n p n
n n p n
n n X n n p n
n n n X n n p n
c a
n
p p n
a a f x
c c a F x
' ÷ ÷
' ÷ ÷
' + ÷ ÷
' + ÷ ÷
= =
=
'
=
÷
÷ +
X
X
x
x
Interval
width
Interval
lower limit
Start with unit
interval
nth symbol
to be
encoded
Cumulative conditional
distribution (excluding
current symbol)
Up to p
previous
symbols
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10
Arithmetic coding
Elias coding not practical for long symbol strings:
required arithmetic precision grows with string length
Finite precision implementations: “arithmetic coding”
Widely used in modern image and video compression algorithms:
JBIG, JPEG, JPEG2000, H.263, H.264/AVC
n
x
Arithmetic
coder
Arithmetic
decoder
n
x
( )
PMF
n
X n
f x
( )
PMF
n
X n
f x
bit stream c
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11
Finite precision for arithmetic coding
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
÷
÷ +
Nbit precision is
permissible, if
rounding down
Pbit approximation of
probabilities
At most N+P LSBs change,
except for a possible carry
affecting MSBs.
Each bit can be affected by
at most one carry.
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12
Represent by
state variable C
n
Represent by
state variable r
n
Send
Finite precision for arithmetic coding (cont.)
Output code string
Maximum value of r
n
can be limited by bit stuffing
MSBs that will 1 bits LSBs that
no longer change represent lower that might be
interval boundary for affected by a
further computation carry
. 01111111 1
n
n
r N P
c xxxxxxxxx xxx cccccccc c
+ +
=
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13
Inefficiency due to interval rounding
Recall: Subinterval of width f
X
(x) is guaranteed to contain one
number that can be represented by L
n
binary digits, with
Hence, rounding one interval value a
n+1
increases bit string by
L
n
~ ÷log
2
f
X
0:n÷1
x
0:n÷1
( )
log
2
a
n
f
X
x
n
( )
a
n+1
< log
2
2
N÷1
+1
2
N÷1
= log
2
1+ 2
1÷N
( )
~
1
ln2
2
1÷N
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14
( )
( )
( )
( ) ( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
2 2 2
log 1 log log 1
( )
1
ln 2
0
X X
X X
X X
X
X
X X
X X
x x
X X
f x f x
E f x E f x E
f x f x
H X
f x
E
f x
f x f x
E f x f x
f x f x
(
  (
    A A
(
÷ + = ÷ ÷ + (  (  
¸ ¸
 

( (
\ . \ .
\ . ¸ ¸
¸ ¸
=
( A
~
(
¸ ¸
( A A
= · = A =
(
¸ ¸
¿ ¿
Limited precision probabilities
Efficiency loss
provided that rounded
probabilities still add to 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15
Example: inefficiency for binary source
due to representing smaller probability by 2
P
log
2
p
( )
r
e
l
a
t
i
v
e
b
i
t

r
a
t
e
i
n
c
r
e
a
s
e
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16
Interval width is normalized to by binary shift after
each symbol
Approximation, with :
Corresponds to approximating of p(L) within a factor of 2:
Factor o can be lumped into external probabilities
Optimum o depends somewhat on data, typically
JBIG
Multiplicationfree binary arithmetic coding
2 3
or 0.708 or
3 4
o o o = = =
( )
( )
( ) ( )
( ) (1 ( )) ( )
Ap L p L p L
Ap M A p L A p L A p L
o
o
'
~ · =
'
= ÷ ~ ÷ · = ÷
1/ 2 1 A s <
p(L) s p(M)
( ) ( )
p L p L
A
o
~
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17
MPSLPS switch
Problem: interval width for symbol M can become negative
Solution: switch the role of M and L, if
Problem: interval for symbol M can be smaller than interval for L if
Solution: “Conditional exchange” of symbols, when problem occurs
Improves compression efficiency, implemented in JPEG and JBIG
( ) ( )
1
0 if ,1
2
A p L p L A

' '
÷ < > e

.
¸
( )
1
2
p L
'
>
( ) ( ) ( ) )
1 1
4 2
1
,
2
p L A p L p L A
' ' '
> ÷ · > e
¸
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18
Sufficiency of binary coders
Consider r.v. equivalent to a r.v.
No loss by conditionally encoding bits (in any order)
Supply arithmetic coder with symbol/probability pairs
Total number of (conditional) probabilities to be estimated per symbol x
. . . same as with Kary encoding of x
{ }
0,1, , 2 1
K
X e ÷
0
1
1 K
B
B
B
÷
 


=



\ .
B
MSB
LSB
( ) ( )
( ) ( ) ( )
0 1 0 1 0 1 2
  , ,
K K
H X H
H B H B B H B B B B
÷ ÷
=
= + + +
B
( ) ( ) ( ) ( )
0: 1
0 0 1 1 1 1  0: 1
, , , , , , with 0,
k k
K K k B k
b p b p b p p f
÷
÷ ÷ ÷
=
B
b
1
1 2 2 2 1
K K ÷
+ + + = ÷
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19
Consider stationary binary Markovk process with probabilities
Context labeling function, enumerating all possible context vectors
Backward probability estimation separately for each context
Adaptive probability estimation
} {
n
X
( ) ( )
0: 1 ( ): 1
 0: 1  ( ): 1
0, 0,
k k n n k n
X k X n k n
f f
÷ ÷ ÷
÷ ÷ ÷
=
X X
x x
“Context vector,”
2
k
possible
combinations
( ) { } { }
: 0,1 0,1, , 2 1
k
k
ì e ÷ ÷ x x
 
( )
( )
( )
( )
( )
0 ( ): 1
0
0 ( ): 1 1 ( ): 1
,
, ,
n k n
n k n n k n
C n
p n
C n C n
ì
ì ì
÷ ÷
÷ ÷ ÷ ÷
(
+ A
¸ ¸
=
( (
+ A + + A
¸ ¸ ¸ ¸
x
x x
Count of zeroes that
have previously
occured in that context
Probability
of symbol
x[n]=0
Bias for low counts,
typically A=1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20
Adaptive probability estimation (cont.)
Tracking changing statistics, compromise between
• Rapid adaptation (small past sample counts)
• Accurate estimates (large past sample counts)
Scaled count estimation algorithm
{ } { }
 
0 1
1 1
0 0
0 1 min 0 1 max
0 1
0 1
0
0
0 1
Initialize 0
For 0,1,
If 1
C 1
else
C 1
If min C , or max C ,
C ; C
2 2
Estimate
2
n
C C
n
x
C
C
C C C C
C C
C
p n
C C
= =
=
=
÷ +
÷ +
> >
( (
÷ ÷
( (
¸ ¸ ¸ ¸
+ A
=
+ + A
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21
Context adaptive coding
2d extension of Markov model
Markov conditional independence property
N
·
+ n
“causal halfplane”
 
x n
“Coding
context”
N+ n
f
X n
¸
(
¸
X
N
·
+n
X n
¸
(
¸
, X
N
·
+n
( )
= f
X n
¸
(
¸
X
N+n
X n
¸
(
¸
, X
N+n
( )
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 22
Context adaptive coding (cont.)
Coding context vector can be mapped into context label directly
without loss
• Feasible for binary images
• Example: JBIG uses 10 binary pixels as context label
For 8bit images, number of different contexts
• Context might have to be clustered
• Combine with prediction
256
N
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 23
Reading
W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B.
Arps, “An overview of the basic principles of the QCoder
adaptive binary arithmetic coder,” IBM J. Res. Develop., vol.
32, no. 6, November 1988.
Witten, Radford, Neal, Cleary, “Arithmetic Coding for Data
Compression” Communications of the ACM, vol. 30, no. 6,
pp. 520540, June 1987.
Moffat, Neal, Witten, “Arithmetic Coding Revisited,” ACM
Transactions on Information Systems, vol. 16, vo. 3, pp.
256–294, July 1998.