=
=
M
i
b
b
X i
X q
i
i
dx f y x
dx f x Q x
1
2
2
2
1
) (
+
Quantizer input
Quantization noise
Quantizer output
8/55
Quantized Bitrate with FLC
If the number of quantizer output is M, then the
bitrate of the quantizer output is R = log
2
M(
Eample: M = 8 R = 3
Quantizer design problem:
Given an input pdf f
X
(x) and the number of levels M in the
quantizer, find the decision boundaries {b
i
} and the
reconstruction levels {y
i
} so as to minimize the mean
squared quantization error
9/55
Quantized Bitrate with VLC
For VLC representation of quantization intervals, the
bitrate depends on decision boundary selection
Example: eightlevel quantizer:
=
=
(
=
=
M
i
b
b
X i
M
i
i i
i
i
dx x f l
y P l R
1
1
1
) (
) (
10/55
Optimization of Quantization
Rateoptimized quantization
Given: Distortion constraint
q
2
D*
Find: { b
i
}, { y
i
} binary codes
Such that: R is minimized
Distortionoptimized quantization
Given: Rate constraint R R*
Find: { b
i
}, { y
i
} binary codes
Such that:
q
2
is minimized
11/55
Uniform Quantizer
All intervals are of the same size
Boundaries are evenly spaced (step size:), except for out
most intervals
Reconstruction
Usually the midpoint is selected as the representing value
Quantizer types:
Midrise quantizer: zero is not an output level
Midtread quantizer: zero is an output level
12/55
Midrise vs. Midtread Quantizer
Midrise Midtread
13/55
Uniform Quantization of Uniform Source
If the source is uniformly distributed in [X
max
, X
max
],
the output is quantized by an Mlevel uniform
quantizer, then the quantization step size is
and the distortion is
,
2
max
M
X
=
.
12 2
1
2
1 2
2
2
2 /
1
) 1 (
max
2
2
=

\

M
i
i
i
q
dx
X
i
x
This is quite difficult to evaluate!
14/55
Alternative MSQE Derivation
We can also compute the power of quantization
error q = x Q(x), q [/2, /2] by:
.
12
1
2
2 /
2 /
2 2
=
dq q
q
15/55
The SNR of Quantization
For nbit uniform quantization of an uniform source of
[X
max
, X
max
], the SNR is 6.02n dB, where n = log
2
M:
. dB 02 . 6 2 log 20
log 10
2
12
12
) 2 (
log 10
12
12
) 2 (
log 10 log 10
10
2
10
2
max
2
max
10
2
2
max
10
2
2
10
n
M
M
X
X
X
n
q
s
= =
=





\


\

=


\

=


\

16/55
Example: Quantization of Sena
Darkening and contouring effects of quantization
8 bits / pixel
2 bits / pixel
1 bits / pixel
3 bits / pixel
17/55
Quantization of Nonuniform Sources
Given a nonuniform source, x [100, 100],
P(x[1, 1]) = 0.95, and we want to design an 8level
(3bit) quantizer.
A nave approach uses uniform quantizer ( = 25):
95% of sample values represented by only two numbers:
12.5 and 12.5, with a maximal quantization error of 12.5
and minimal error of 11.5
If we use = 0.3 (two endintervals would be huge)
Max error is now 98.95 (i.e. 100 1.05), however, 95% of
the time the error is less than 0.15
18/55
Optimal that minimizes MSQE
Given pdf f
X
(x) of the source, lets design an Mlevel
midrise uniform quantizer that minimizes MSQE:
. ) (
2
1
2
) (
2
1 2
2
1
2
2
1
1
) 1 (
2
2
2
\

\

+

\

=
M
X
i
i
i
X q
dx x f
M
x
dx x f
i
x
M
Granular error
Overload error
Overload error
Granular error
x Q(x)
19/55
Solving for Optimum Step Sizes
Given an f
X
(x) and M, we can solve for numerically:
Optimal uniform quantizer for different sources:
. 0 ) (
2
1
) 1 (
) (
2
1 2
) 1 2 (
1
2
1
1
) 1 (
2
2
\

=

\

\

M
X
i
i
i
X
q
dx x f
M
x M
dx x f
i
x i
d
d
M
20/55
Overload/Granular Regions
Selection of the step size must trade off between
overload noise and granular noise
21/55
Variance Mismatch Effects (1/2)
Effect of variance mismatch on the performance of a
4bit uniform quantizer
22/55
Variance Mismatch Effects (2/2)
The MSQE as a function of variance mismatch with a
4bit uniform quantizer.
23/55
Distribution Mismatch Effects
Given 3bit quantizer, the effect of distribution
mismatch for different sources (SNR errors in dB):
Form lefttoright, we assume that the sources are uniform,
Gaussian, Laplacian, and Gamma, and compute the
optimum MSQE step size for uniform quantizer
The resulting gets larger from lefttoright
if there is a mismatch, larger than optimum gives better performance
3bit quantizer is too coarse
24/55
Adaptive Quantization
We can adapt the quantizer to the statistics of the
input (mean, variance, pdf)
Forward adaptive (encoderside analysis)
Divide input source in blocks
Analyze block statistics
Set quantization scheme
Send the scheme to the decoder via side channel
Backward adaptive (decoderside analysis)
Adaptation based on quantizer output only
Adjust accordingly (encoderdecoder in sync)
No side channel necessary
25/55
Forward Adaptive Quantization (FAQ)
Choosing analysis block size is a major issue
Block size too large
Not enough resolution
Increased latency
Block size too small
More side channel information
Assuming a mean of zero, signal variance is
estimated by
.
1
1
0
2 2
=
+
=
N
i
i n q
x
N
26/55
Speech Quantization Example (1/2)
16bit speech samples 3bit fixed quantization
27/55
Speech Quantization Example (2/2)
16bit speech samples 3bit FAQ
Block = 128 samples
8bit variance quantization
28/55
FAQ Refinement
So far, we assumed uniform pdf over maximal ranges,
we can refine it by computing the range of distribution
adaptively for each block
Example: Sena image, 8x8 blocks, 2x8bit for range
per block, 3bit quantizer
Original 8 bits/pixel Quantized 3.25 bits/pixel
29/55
Backward Adaptive Quantization (BAQ)
Key idea: only encoder sees input source, if we do
not want to use side channel to tell the decoder how
to adapt the quantizer, we can only use quantized
output to adapt the quantizer
Possible solution:
Observe the number of output values that falls in outer levels
and inner levels
If they match the assumed pdf, is good
If too many values fall in outer levels, should be enlarged,
otherwise, should be reduced
Issue: estimation of pdf requires large observations?
30/55
Jayant Quantizer
N. S. Jayant showed in 1973 that adjustment based
on few observations still works fine:
If current input falls in the outer levels, expand step size
If current input falls in the inner levels, contract step size
The total product of expansions and contraction should be 1
Each decision interval k has a multiplier M
k
If input s
n1
falls in the k
th
interval, step size is multiplied by M
k
Innerlevel M
k
< 1, outerlevel M
k
> 1
Step size adaptation rule:
where l(n1) is the quantization interval at time n1.
,
1 ) 1 (
=
n n l n
M
31/55
Output Levels of 3bit Jayant Quantizer
The multipliers are symmetric:
M
0
= M
4
, M
1
= M
5
, M
2
= M
6
, M
3
= M
7
32/55
Example: Jayant Quantizer
M
0
= M
4
= 0.8, M
1
= M
5
= 0.9
M
2
= M
6
= 1.0, M
3
= M
7
= 1.2,
0
= 0.5
Input: 0.1, 0.2, 0.2, 0.1, 0.3, 0.1, 0.2, 0.5, 0.9, 1.5
33/55
Picking Jayant Multipliers
We must select
min
and
max
to prevent underflow
and overflow of step sizes
Selection of multipliers
Total production of expansion/contractions should be 1
Scaled to probability of events in each interval, we have
Pick > 1, and let M
k
=
l
k, we have
and l
k
are chosen, P
k
is known.
. 1
0
=
=
k
n
k
M
k
M
. , where , 1
0
0
=
=
= = =
M
k
k
k
k
P
k
M
k
n N
N
n
P M
k
, 0
0
=
=
M
k
k k
P l
34/55
Example: Ringing Problem
Use a 2bit Jayant quantizer to quantize a square
wave
P
0
= 0.8, P
1
= 0.2 pick l
0
= 1, l
1
= 4, ~ 1.
35/55
Jayant Quantizer Performance
To avoid overload errors, we should expand rapidly
and contracts moderately
Robustness over changing input statistics
Jayant is about 1dB lower
than fixed quantizer
36/55
Nonuniform Quantization
For uniform quantizer,
decision boundaries
are determined by a
single parameter .
We can certainly reduce
quantization errors
further if each decision
boundaries can be
selected freely
37/55
pdfoptimized Quantization
Given f
X
(x), we can try to minimize MSQE:
Set derivative of
q
2
w.r.t. y
j
to zero and solve for y
j
,
we have:
If y
j
are determined, the b
j
s can be selected as:
( ) . ) (
1
2
2
1
=
=
M
i
b
b
X i q
i
i
dx x f y x
.
) (
) (
1
1
=
j
j
j
j
b
b
X
b
b
X
j
dx x f
dx x xf
y
( ) . 2 /
1 j j j
y y b + =
+
y
j
is the center of mass of
f
X
in [b
j1
, b
j
)
38/55
LloydMax Algorithm (1/3)
LloydMax algorithm solves y
j
and b
j
iteratively until
an acceptable solution is found
Example:
For midrise quantizer,
b
0
= 0,
b
M/2
is the largest input,
we only have to find
{ b
1
, b
2
, , b
M/21
} and
{ y
1
, y
2
, , y
M/21
}.
39/55
LloydMax Algorithm (2/3)
Begin with j = 1, we want to find b
1
and y
1
by
Pick a value for y
1
(e.g. y
1
= 1), solve for b
1
and
compute y
2
by
y
2
= 2b
1
+ y
1
,
and b
2
by
Continue the process until all {b
j
} and {y
j
} are found
. ) ( ) (
1
0
1
0
1
=
b
b
X
b
b
X
dx x f dx x xf y
. ) ( ) (
2
1
2
1
2
=
b
b
X
b
b
X
dx x f dx x xf y
40/55
LloydMax Algorithm (3/3)
If the initial guess of y
1
does not fulfills the termination
condition:
where
we must pick a different y
1
and repeat the process.
,
2 / 2 /
M M
y y
, 2
1 2 / 1 2 / 2 /
+ =
M M M
y b y
. ) ( ) (
2 /
1 2 /
2 /
1 2 /
2 /
=
M
M
M
M
b
b
X
b
b
X M
dx x f dx x xf y
41/55
Example: pdfOptimized Quantizers
We can achieve gain over the uniform quantizer
(9.24) (7.05)
(12.18) (9.56)
(14.27) (11.39)
42/55
Mismatch Effects
Nonuniform quantizers also suffer mismatch effects.
To reduce the effect, one can use an adaptive non
uniform quantizer, or an adaptive uniform quantizer
plus companded quantization techniques
Variance mismatch on a 4bit Laplacian nonuniform quantizer.
43/55
Companded Quantization (CQ)
In companded quantization, we adjust (i.e. rescale)
the intervals so that the size of each interval is in
proportion to the probability of inputs in each interval
equivalent to a nonuniform quantizer
44/55
Example: CQ (1/2)
The compressor function:
The uniform quantizer:
step size = 1.0
.
1
1
1 1 if 2
) (
3
4
3
2
3
4
3
2
<
> +
=
x
x
x x
x c
x
x
45/55
Example: CQ (2/2)
The expandar function:
The equivalent nonuniform
quantizer
.
2 2
2 2
2 2 if
) (
2
3
2
3
2
1
< +
>
=
x
x
x
x c
x
x
x
46/55
Remarks on CQ
If the level of quantizer is large and the input is
bounded by x
max
, it is possible to choose a c(x) such
that the SNR of CQ is independent to input pdf:
SNR = 10 log
10
(3M
2
) 20log
10
,
where c(x) = x
max
/ ( x) and a is a constant.
Two popular CQ for telephones: law and Alaw
law compressor
Alaw compressor
). sgn(
) 1 ln(
) 1 ln(
) (
max
max
x x x c
x
x
+
+
=
.
1 ), sgn(
0 ), sgn(
) (
max
max
max
1
ln 1
ln 1
max
1
ln 1
=
+
+
+
x
x
A A
A x
x
A
x A
x x
x
x c
x
x A
47/55
Entropy Coding of Quantizer Outputs
The levels of the quantizer is the alphabet of entropy
coders, for Mlevel quantizer, FLC needs log
2
M bits
per output
Example of VLC coded output of minimum MSQE
quantizers:
Note: nonuniform quantizer has higher entropies since high
probability regions uses smaller step sizes
48/55
Vector Quantization
Vector quantization groups source data into vectors
At both the encoder and decoder, we have a set of vectors
called the codebook. Each codevector in the codebook is
assigned a binary index.
49/55
Why Vector Quantization (1/2)?
Correlated multidimensional data have limited valid
ranges
useless
useless
50/55
Why Vector Quantization (2/2)?
Looking at the data from a higher dimension allow us
to better fit the quantizer structure to the joint pdf
Example: quantize the Laplacian source data two at a time:
11.44 dB 11.73 dB
51/55
Vector Quantization Rule
Vector quantization (VQ) of X may be viewed as the
classification of the outcomes of X into a discrete
number of sets or cells in Nspace
Each cell is represented by a vector output Y
j
Given a distance measure d(x, y), we have
VQ output: Q(X) = Y
j
iff d(X, Y
j
) < d(X, Y
i
), i j.
Quantization region: V
j
= { X: d(X, Y
j
) < d(X, Y
i
), i j}.
V
j
Y
j
x
1
x
2
52/55
Codebook Design
The set of quantizer output points in VQ is called the codebook
of the quantizer, and the process of placing these output points
is often referred to as codebook design
The kmeans algorithm is often used to classify the outputs
Given a large set of output vectors from the source, known as the
training set, and an initial set of k representative patterns
Assign each element of the training set to the closest
representative pattern
After an element is assigned, the representative pattern is updated
by computing the centroid of the training set vectors assigned to it
When the assignment process is complete, we will have k groups of
vectors clustered around each of the output points
53/55
The LindeBuzoGray Algorithm
1. Start with an initial set of reconstruction values
{Y
i
(0)
}
i=1..M
and a set of training vectors {X
n
}
n=1..N
.
Set k = 0, D
(0)
= 0. Select threshold .
2. The quantization regions {V
i
(k)
}
i=1..M
are given by
V
j
(k)
= {X
n
: d(X
n
, Y
i
) < d(X
n
, Y
j
), j i}, i = 1, 2, , M.
3. Compute the average distortion D
(k)
between the
training vectors and the representative value
4. If (D
(k)
D
(k1)
)/D
(k)
< , stop; otherwise, continue
5. Let k = k + 1. Update {Y
i
(k)
}
i=1..M
with the average
value of each quantization region V
i
(k1)
. Go to step 2.
54/55
Example: Codebook Design
Initial state Final state
55/55
Impact of Training Set
The training sets used to construct the codebook
have significant impact on the performance of VQ
Images quantized at 0.5
bits/pixel, codebook size 256