Professional Documents
Culture Documents
Arithmetic Coding
Additional Material
Spring 2015
Introduction
Basic Encoding and Decoding
Scaling and Incremental Coding
Integer Implementation
Adaptive Arithmetic Coding
Binary Arithmetic Coding
Applications
JBIG, H.264, JPEG 2000
Arithmetic coding:
Can resolve all of these problems.
Code a sequence of symbols without having to generate codes for all
sequences of that length.
1
00
010 011
abcd…..
dcba….. 0 1
CMPT365 Multimedia Systems 4
0 1
Arithmetic Coding a b c
Disjoint and complete partition of the range [0, 1)
[0, 0.8), [0.8, 0.82), [0.82, 1)
Each interval corresponds to one symbol
Interval size is proportional to symbol probability
0 1
Observation: once the tag falls into an interval, it never gets out
of it
1 0.8
0 0.8 0.82 1.0
2 0.02 Map to real line range [0, 1)
Order does not matter
3 0.18 Decoder need to use the same order
1 2 3
Range 1
0 0.8 0.82 1.0
1 2 3
Range 0.8
0 0.64 0.656 0.8
1 2 3
Range 0.144
0.656 0.7712 0.77408 0.8
1 2 3
Range 0.00288
0.7712 0.773504 0.7735616 0.77408
Final range: [0.7712, 0.773504): Encode 0.7712
Difficulties: 1. Shrinking of interval requires high precision for long sequence.
2. No output is generated until the entire sequence has been processed.
CMPT365 Multimedia Systems 8
Cumulative Density Function (CDF)
For continuous distribution: Probability Mass Function
0.4
x
0.2 0.2 0.2
FX ( x) P( X x) p( x)dx
1 2 3 4 X
For discrete distribution:
i
1.0
FX (i ) P ( X i ) P( X k ) CDF 0.8
k 1
Properties: 0.4
Non-decreasing
Piece-wise constant
Each segment is closed at the lower end.
0.2
X
1 2 3 4
1 2 3
Decode 1
0 0.8 0.82 1.0
1 2 3
Decode 3
0 0.64 0.656 0.8
1 2 3
Decode 2
0.656 0.7712 0.77408 0.8
1 2 3
Decode 1
0.7712 0.773504 0.7735616 0.77408
Receive 0.7712 1 2 3
Decode 1
Low = 0; high = 1;
x = Encoded_number
While (x ≠ low) {
n = DecodeOneSymbol(x);
output symbol n;
x = (x - CDF(n-1)) / (CDF(n) - CDF(n-1));
};
Introduction
Basic Encoding and Decoding
Scaling and Incremental Coding
Integer Implementation
Adaptive Arithmetic Coding
Binary Arithmetic Coding
Applications
JBIG, H.264, JPEG 2000
Key Observation:
As the RANGE reduces, many MSB’s of LOW and HIGH become identical:
• Example: Binary form of 0.7712 and 0.773504:
0.1100010.., 0.1100011..,
We can output identical MSB’s and re-scale the rest:
Incremental encoding
This also allows us to achieve infinite precision with finite-precision integers.
0 0.5 1.0
Output 0, then shift left by 1 bit
[0, 0.5) [0, 1): E1(x) = 2 x
1 0.8
Input 1 2 0.02
0 0.8 1.0 3 0.18
Input 3
0 0.656 E2: Output 1
0.8 2(x – 0.5)
Input 2
0.312 0.5424 0.54816 0.6 E2: Output 1
0.0848 0.09632
E1: 2x, Output 0
0.1696 0.19264 E1: Output 0
1 0.8
encoding the next symbol
2 0.02
3 0.18
Next:
Integer Implementation
• E3 scaling
Adaptive Arithmetic Coding
Binary Arithmetic Coding
Applications
• JBIG, H.264, JPEG 2000
Review
Integer Implementation
Integer representation
E3 Scaling
Minimum word length
1 2 3
Range 0.144
0.656 0.7712 0.77408 0.8
1 2 3
Range 0.00288
0.7712 0.773504 0.7735616 0.77408
Input sequence: “1321”
0 0.5 1.0
Output 0, then shift left by 1 bit
[0, 0.5) [0, 1): E1(x) = 2 x
1 0.8
Input 1 2 0.02
0 0.8 1.0 3 0.18
Input 3
0 0.656 E2: Output 1
0.8 2(x – 0.5)
Input 2
0.312 0.5424 0.54816 0.6 E2: Output 1
0.0848 0.09632
E1: 2x, Output 0
0.1696 0.19264 E1: Output 0
1 0.8 0.8
2 0.02 0.82
EncodeSymbol(n) { 3 0.18 1
//Update variables
RANGE = HIGH - LOW;
HIGH = LOW + RANGE * CDF(n);
LOW = LOW + RANGE * CDF(n-1);
1 0.8 0.8
HIGH = LOW + RANGE * CDF(n); 2 0.02 0.82
LOW = LOW + RANGE * CDF(n-1);
3 0.18 1
//keep scaling before decoding next symbol
while LOW, HIGH in [0, 0.5) or [0.5, 1) {
scale LOW, HIGH by E1 or E2 rule
Left shift Tag and read one more bit to LSB
}
return n;
}
1 2 3
k P(k) nk Cum(k)
0 40 41 50 0 - - 0
i 1 0.8 40 40
n k
Cum(i )
2 0.02 1 41
CDF FX (i ) k 1
3 0.18 9 50
N N
CMPT365 Multimedia Systems 31
Integer Implementation
HIGH LOW + RANGE * CDF(n);
LOW LOW + RANGE * CDF(n-1);
RANGE HIGH LOW 1
HIGH LOW RANGE Cum(n) / N 1
LOW LOW RANGE Cum(n 1) / N
Why + 1 in RANGE and – 1 in HIGH?
HIGH should be less than the LOW of the next interval
The best integer value is HIGH = (next LOW) – 1
[HIGH, HIGH + 1) still belongs to the current interval,
although we could not represent it explicitly.
0 n-1 n
256 40 256 0
If n = 1: HIGH 0 1 203 LOW 0 0
50 50
256 41 256 40
If n = 2: HIGH 0 1 208 LOW 0 204
50 50
256 50 256 41
If n = 3: HIGH 0 1 255 LOW 0 209
50 50
1 2 3
HIGH 0 . 0 x x x x x x x 1 1 1 ...
LOW 0 . 0 x x x x x x x 0 0 0 ...
This also ensures the RANGE is doubled after scaling:
HIGH – LOW + 1 (2 x HIGH + 1 – 2 x LOW + 1) = 2(HIGH – LOW + 1)
0 . 1 x x x x x x x 1 1 1 ...
0 . 1 x x x x x x x 0 0 0 ...
Input 1
0 203 255
Input 3
0 167 203 E2: Output 1
78 151
LOW: 167 (10100111) HIGH: 203 (11001011)
Review
Integer Implementation
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
E1 E3 E2 E1
n n
E2 E3 E1 E2
n n
Ei E j
n
Apply n Ej scalings, followed by an Ei scaling.
Input 1
0 0.8 1.0
Input 3
0 0.656
0.8
Input 2
0.312 0.5424 0.54816 0.6 E2: Output 1
0.0848 0.09632
E1: Output 0
0.1696 0.19264
With E3:
0.312 0.6
E2
0 0.25 0.5 0.75 1
E3
0 0.25 0.5 0.75 1
Input 2
0.312 0.5424 0.54816 0.6 E2: Output 1
0.0848 0.09632
E1: Output 0
0.1696 0.19264
With E3:
0.312 0.6
0.1696 0.19264
With E3: Apply E3 whenever it is possible, nothing else is needed
Need E1 scaling
Need E2 scaling
Need E3 scaling
No scaling is required.
Ready to encode/decode
the next symbol.
Review
Integer Implementation
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
DecodeSymbol(Tag) {
RANGE = HIGH - LOW + 1;
n = 1;
While (Tag > LOW + RANGE * Cum(n) / N - 1) {
n++;
}
Round off to integer: HIGH of each interval
HIGH = LOW + RANGE * Cum(n) / N - 1;
LOW = LOW + RANGE * Cum(n-1) / N;
if (E3 is possible) {
LOW = (LOW - QUARTER) << 1;
HIGH = ((HIGH - QUARTER) << 1) + 1;
Tag = (Tag - QUARTER) << 1;
Tag = Tag | ReadBits(1);
}
}
}
Review
Integer Implementation
Integer representation
E3 Scaling
Complete Algorithm
Minimum word length
0 40 41 50
RANGE Cum(n)
HIGH LOW 1 k P(k) Nk Cum(k)
N
0 - - 0
RANGE Cum(n 1)
LOW LOW 1 0.8 40 40
N 2 0.02 1 41
3 0.18 9 50
Review
Integer Implementation
Binary Arithmetic Coding
Adaptive Arithmetic Coding
symbol 0 symbol 1
0 x 1
Only two intervals in [0, 1): [0, x), [x, 1)
0 0.36 0.6
0 0.6 1
Increasingly popular:
JBIG, JBIG2, JPEG2000, H.264
Review
Integer Implementation
Binary Arithmetic Coding
Adaptive Arithmetic Coding
Example
Initial counters for 0’s and 1’s:
C(0)=C(1)=1.
P(0)=P(1)=0.5
0 0.5 1
Decoding
Initial counters for 0’s and 1’s:
C(0)=C(1)=1 P(0)=P(1)=0.5
Decode 0
0 0.5 1
abcbcab abcbcab
1-D
cbabcba
Context 2-D
template Context
template
CMPT365 Multimedia Systems 66
Example 0110101
Binary sequence: 0, 1
Neighborhood (template) size: 3
2^3=8 possible combinations (contexts) of 3 neighbors
(x(n-1), x(n-2), x(n-3)).
Collect frequencies of 0’s and 1’s under each context
Huffman coding:
H(X) <= Average Bit Rate <= H(X) + 1/m
Impractical: need to generate codewords for all sequences of
length m.
Arithmetic coding:
Partition the range [0, 1) recursively according to symbol
probabilities.
Incremental Encoding and Decoding
E1, E2, E3 scaling