Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no.

1
Arithmetic coding
 Elias coding
 Arithmetic coding as finite-precision Elias
coding
 Redundancy due to finite precision
 Multiplication-free arithmetic coding
 Context-adaptive coding


Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 2
Elias coding
 Entropy coding algorithm for sequences of
symbols x with general (conditional) probabilities
 Representation of x by a subinterval of the unit
interval [0,1)
 Width of the subinterval is approximately equal to
the probability f
X
(x)
 Subinterval for x can be determined by recursive
subdivision algorithm
 Represent x by shortest binary fraction in the
subinterval
 Subinterval of width f
X
(x) is guaranteed to contain
one number that can be represented by L binary
digits, with

( )
f
X
x
0.1111
0.111
0.1101
0.11
0.1011
0.101
0.1001
0.1
0.0111
0.011
0.0101
0.01
0.0011
0.001
0.0001
0
0
1
( )
2
log L f ~ ÷
X
x
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 3
Example: Elias coding of memoryless binary source
( ) ( )
1 3
0 1
4 4
X X
f f = =
0
1
1
4
n
c
n n
c a +
0
1
4
0
1
1
16
4
16
1
7
64
16
64
0
37
256
28
256
1
121
1024
148
1024
1 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 4
Example: Elias coding of memoryless ternary source
Courtesy: T. Wiegand

f
X
a
( )
= f
X
b
( )
=
1
4
f
X
c
( )
=
1
2
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 5
Elias coding: choose binary fraction in subinterval
 Uniquely identify interval [c
n
,c
n
+a
n
) by a binary fraction


 Length of bit-string



 Bit-string




bits
0. [ , )
n
n n n
L
bbbbbb b c c a e +

L
n
= log
2
1
a
n

+1= h
X
0:n÷1
x
0:n÷1
( )

+1
ˆ 2 2 1
n n
L L
n n n
c c c
÷

= + >

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 6
Successive decoding
 Elias bit string can be decoded symbol by symbol, starting with the
first symbol
 For each symbol n=0,1,2, . . .
• Calculate the intervals [c
n
,c
n
+a
n
) corresponding to all possible x
n
• Determine the interval for which
• Emit the corresponding x
n

)
ˆ ,
n n n n
c c c a e +

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 7
Elias coding for memoryless source
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
÷
÷ +
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
i
X i X j
j
X K
F f o o
o o o o
÷
=
÷
=
=
¯
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 8
Elias coding for changing probabilities
Elias coding algorithm
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n
n
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
÷
÷ +
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative distribution
(excluding current symbol)
( ) ( )
1
0
0 1 2 1

where { , , ,..., }
n n
n
i
X i X j
j
X K
F f o o
o o o o
÷
=
÷
=
=
¯
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 9
Elias coding for Markov random processes
Elias coding algorithm for Markov-p sources
{ }
( )
( )
: 1
: 1
0 0
1 | : 1
1 | : 1
Initialize 0 and 1.
For each 0,1,...
Set min ,
Update ,
Update ,
n n p n
n n p n
n n X n n p n
n n n X n n p n
c a
n
p p n
a a f x
c c a F x
' ÷ ÷
' ÷ ÷
' + ÷ ÷
' + ÷ ÷
= =
=
'
=
÷
÷ +
X
X
x
x
Interval
width
Interval
lower limit
Start with unit
interval
n-th symbol
to be
encoded
Cumulative conditional
distribution (excluding
current symbol)
Up to p
previous
symbols
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 10
Arithmetic coding
 Elias coding not practical for long symbol strings:
required arithmetic precision grows with string length
 Finite precision implementations: “arithmetic coding”
 Widely used in modern image and video compression algorithms:
JBIG, JPEG, JPEG-2000, H.263, H.264/AVC

n
x
Arithmetic
coder
Arithmetic
decoder
n
x
( )
PMF
n
X n
f x
( )
PMF
n
X n
f x
bit stream c
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 11
Finite precision for arithmetic coding
( )
( )
0 0
1
1
Initialize 0 and 1.
For each 0,1,
Update
Update
n n X n
n n n X n
c a
n
a a f x
c c a F x
+
+
= =
=
÷
÷ +
N-bit precision is
permissible, if
rounding down
P-bit approximation of
probabilities
At most N+P LSBs change,
except for a possible carry
affecting MSBs.
Each bit can be affected by
at most one carry.

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 12




Represent by
state variable C
n




Represent by
state variable r
n




Send
Finite precision for arithmetic coding (cont.)
 Output code string








 Maximum value of r
n
can be limited by bit stuffing

MSBs that will 1 bits LSBs that
no longer change represent lower that might be
interval boundary for affected by a
further computation carry
. 01111111 1
n
n
r N P
c xxxxxxxxx xxx cccccccc c
+ +
=
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 13
Inefficiency due to interval rounding
 Recall: Subinterval of width f
X
(x) is guaranteed to contain one
number that can be represented by L
n
binary digits, with


 Hence, rounding one interval value a
n+1
increases bit string by






L
n
~ ÷log
2
f
X
0:n÷1
x
0:n÷1
( )

log
2
a
n
f
X
x
n
( )
a
n+1
< log
2
2
N÷1
+1
2
N÷1
= log
2
1+ 2
1÷N
( )
~
1
ln2
2
1÷N
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 14
( )
( )
( )
( ) ( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
2 2 2
log 1 log log 1
( )
1

ln 2
0
X X
X X
X X
X
X
X X
X X
x x
X X
f x f x
E f x E f x E
f x f x
H X
f x
E
f x
f x f x
E f x f x
f x f x

| |
| | | | A A

÷ + = ÷ ÷ + | | |

| |
|

\ . \ .
\ .

=
A
~


A A
= · = A =


¯ ¯
Limited precision probabilities
 Efficiency loss
provided that rounded
probabilities still add to 1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 15
Example: inefficiency for binary source
due to representing smaller probability by 2
-P

log
2
p
( )
r
e
l
a
t
i
v
e

b
i
t
-
r
a
t
e

i
n
c
r
e
a
s
e

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 16
 Interval width is normalized to by binary shift after
each symbol
 Approximation, with :



 Corresponds to approximating of p(L) within a factor of 2:


 Factor o can be lumped into external probabilities
 Optimum o depends somewhat on data, typically





JBIG
Multiplication-free binary arithmetic coding
2 3
or 0.708 or
3 4
o o o = = =
( )
( )
( ) ( )
( ) (1 ( )) ( )
Ap L p L p L
Ap M A p L A p L A p L
o
o
'
~ · =
'
= ÷ ~ ÷ · = ÷
1/ 2 1 A s <
p(L) s p(M)
( ) ( )
p L p L
A
o
~
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 17
MPS-LPS switch
 Problem: interval width for symbol M can become negative



 Solution: switch the role of M and L, if

 Problem: interval for symbol M can be smaller than interval for L if


 Solution: “Conditional exchange” of symbols, when problem occurs
 Improves compression efficiency, implemented in JPEG and JBIG
( ) ( )
1
0 if ,1
2
A p L p L A

|
' '
÷ < > e
|

.

( )
1
2
p L
'
>
( ) ( ) ( ) )
1 1
4 2
1
,
2
p L A p L p L A
' ' '
> ÷ · > e

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 18
Sufficiency of binary coders

 Consider r.v. equivalent to a r.v.

 No loss by conditionally encoding bits (in any order)



 Supply arithmetic coder with symbol/probability pairs


 Total number of (conditional) probabilities to be estimated per symbol x


. . . same as with K-ary encoding of x
{ }
0,1, , 2 1
K
X e ÷
0
1
1 K
B
B
B
÷
| |
|
|
=
|
|
|
\ .
B
MSB
LSB
( ) ( )
( ) ( ) ( )
0 1 0 1 0 1 2
| | , ,
K K
H X H
H B H B B H B B B B
÷ ÷
=
= + + +
B
( ) ( ) ( ) ( )
0: 1
0 0 1 1 1 1 | 0: 1
, , , , , , with 0,
k k
K K k B k
b p b p b p p f
÷
÷ ÷ ÷
=
B
b
1
1 2 2 2 1
K K ÷
+ + + = ÷
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 19
 Consider stationary binary Markov-k process with probabilities



 Context labeling function, enumerating all possible context vectors

 Backward probability estimation separately for each context








Adaptive probability estimation
} {
n
X
( ) ( )
0: 1 ( ): 1
| 0: 1 | ( ): 1
0, 0,
k k n n k n
X k X n k n
f f
÷ ÷ ÷
÷ ÷ ÷
=
X X
x x
“Context vector,”
2
k
possible
combinations
( ) { } { }
: 0,1 0,1, , 2 1
k
k
ì e ÷ ÷ x x
| |
( )
( )
( )
( )
( )
0 ( ): 1
0
0 ( ): 1 1 ( ): 1
,
, ,
n k n
n k n n k n
C n
p n
C n C n
ì
ì ì
÷ ÷
÷ ÷ ÷ ÷

+ A

=

+ A + + A

x
x x
Count of zeroes that
have previously
occured in that context
Probability
of symbol
x[n]=0
Bias for low counts,
typically A=1
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 20
Adaptive probability estimation (cont.)
 Tracking changing statistics, compromise between
• Rapid adaptation (small past sample counts)
• Accurate estimates (large past sample counts)
 Scaled count estimation algorithm

{ } { }
| |
0 1
1 1
0 0
0 1 min 0 1 max
0 1
0 1
0
0
0 1
Initialize 0
For 0,1,
If 1
C 1
else
C 1
If min C , or max C ,
C ; C
2 2
Estimate
2
n
C C
n
x
C
C
C C C C
C C
C
p n
C C
= =
=
=
÷ +
÷ +
> >

÷ ÷


+ A
=
+ + A
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 21
Context adaptive coding
 2-d extension of Markov model








 Markov conditional independence property

N
·
+ n
“causal half-plane”
| |
x n
“Coding
context”

N+ n

f
X n

X
N
·
+n
X n

, X
N
·
+n
( )
= f
X n

X
N+n
X n

, X
N+n
( )
Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 22
Context adaptive coding (cont.)
 Coding context vector can be mapped into context label directly
without loss
• Feasible for binary images
• Example: JBIG uses 10 binary pixels as context label
 For 8-bit images, number of different contexts
• Context might have to be clustered
• Combine with prediction


256
N

Bernd Girod: EE398A Image and Video Compression Arithmetic Coding no. 23
Reading
 W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr., R. B.
Arps, “An overview of the basic principles of the Q-Coder
adaptive binary arithmetic coder,” IBM J. Res. Develop., vol.
32, no. 6, November 1988.
 Witten, Radford, Neal, Cleary, “Arithmetic Coding for Data
Compression” Communications of the ACM, vol. 30, no. 6,
pp. 520-540, June 1987.
 Moffat, Neal, Witten, “Arithmetic Coding Revisited,” ACM
Transactions on Information Systems, vol. 16, vo. 3, pp.
256–294, July 1998.

Sign up to vote on this title
UsefulNot useful