Lossy compression

n Lower the bit-rate R by allowing some acceptable distortion

D of the signal.

Lossless coding
Rate R
D=0

Distortion D

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 1

Types of lossy compression problems

n Given maximum rate R, n Given distortion D, minimize

minimize distortion D rate R

R R

D D

Equivalent constrained optimization problems,

often unwieldy due to constraint.

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 2

1
 Lagrangian formulation of the R-D problem

n Instead of cost function D, with constrained R, or R, with

constrained D, use unconstrained Lagrangian cost function

J=D+ λ R
Rate R

lines of constant J=D+ λ R

slope −1/λ

Distortion D
n Requires convex R-D function

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 3

Topics in lossy compression

n Information theoretical bounds for lossy compression: rate

distortion theory
l R-D function for memoryless Gaussian sources with mean-squared
error distortion criterion
l R-D function for Gaussian sources with memory
l R-D for images
n Practical lossy compression techniques: quantization
l Scalar quantization and vector quantization
l Quantizer design for fixed length codes
l Quantizer design for variable length codes

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 4

2
 Rate distortion theory

n Rate distortion theory calculates the minimum transmission

bit-rate R for a required picture quality.

Video U V
Source Coder Decoder
Distortion
D

Bit-rate at least R
for distortion D Š D*

n Results of rate distortion theory are obtained without

consideration of a specific coding method.

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 5

Mutual information

n "Mutual information" is the information that symbols u and

symbols v convey about each other.
n Average mutual information:
I(U;V ) = H(V) − H (V | U ) = H(U) − H (U | V)
P(u,v)
= ∑u ∑ v P(u,v)log
P(u)P(v)
n Properties of mutual information:
0 ≤ I(U ;V ) = I(V;U )
I(U;V ) ≤ H (U )
I(V;U ) ≤ H(V)
n Channel coding: channel capacity C is maximum of
mutual information between transmitter and receiver.
Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 6

3
 Distortion

n Symbol u sent, v received

n Per-symbol distortion:
d(u, v) ≥ 0
d(u, v) = 0 for u = v

n Average distortion:

D(U, V ) = E{d(U, V )} = ∑ u ∑ v P(u, v)d(u, v)

n Distortion criterion:
D ≤ D∗
maximum average distortion
Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 7

Rate distortion function

n Definition:
R(D ∗ ) = min{I(U;V )}
∗
D≤D

n For a given maximum average distortion D*, the rate

distortion function R(D*) is the lower bound for the
transmission bit-rate.
n Typical R(D*) function: R

H(U)
R(0) Š H(U)
H(U ): entropy of the source
R( 0): rate with just not perceptible
loss of quality

D* D*
max

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 8

4
 Shannon Lower Bound

n It can be shown that H(U − V | V) = H(U | V)

n Thus
R(D∗ ) = min{H(U) − H(U | V)}
∗
D≤D

= H(U) − max{H(U | V)}

D≤ D ∗

= H(U) − max{H(U − V | V)}

D≤ D ∗

n Ideally, the source coder would produce distortions u-v that

are statistically independent from the reconstructed signal v
(not always possible!).
n Shannon lower bound:
R(D ∗ ) ≥ H(U) − max H(U − V)
D≤ D ∗

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 9

R(D*) function for a memoryless

Gaussian source and MSE distortion
Gaussian source, variance σ
2
n Rate distortion function
n Mean squared error R [bit]

D = E{(U − V )2 }
4
3
n Rule of thumb: 6 dB ≅ 1 bit
2
n The R(D*) for non-Gaussian 1
sources with the same
variance σ is always below
2 SNR [dB]
6 12 18 24

this Gaussian R(D*)-curve.

∗1 σ 2  σ 
2

R(D ) = log ∗  SNR = 10 log10  ∗  [dB]

2 D  D 

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 10

5
 R(D*) function for
Gaussian source with memory I
n Jointly Gaussian source with power spectrum Φ uu (ω)
n Mean squared error distortion D = E{(U − V )2 }
n Parametric formulation of the R(D*) function

1
D(θ) = ∫ min{θ,Φ uu (ω )}dω
2π ω
Φ (ω) 
max 0, log uu
1 1
R(θ) = ∫ dω
2π ω
 2 θ 

n R(D*) for non-Gaussian sources with the same power

spectral density is always lower.

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 11

R(D*) function for

Gaussian source with memory II
Φuu (ω)

preserved spectrum Φvv (ω)

reconstruction error
spectrum

ω
white noise θ

no signal
transmitted

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 12

6
 Rate distortion function
for images I
n Signal model: Gaussian source with acf
Ruu (∆x, ∆y) = exp −ω 0 ∆x 2 + ∆y 2 ( )
log(Φ uu (ω x ,ω y ))
n Power spectral density: 3
−
2π  ω x + ω y 
2 2
2
Φ uu (ω x ,ω y ) = 2  1 + 
6 dB
ω0  ω 02 
ω 0 = − ln(0.93)

46.15 ω
correlation between 0
adjacent pixels 42.19 ω
0 0
0

ωy −46.15 ω
0
−42.19 ω
0 ωx
Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 13

Rate distortion function

for images II
n Mean squared error criterion: D = E{(U − V )2 }
n After numerical integration:
4.0

3.5
memoryless increasing decreasing
3.0 Gaussian ω0
Rate [bits/sample]

2.5
~2.3 bit
2.0
Gaussian
1.5
with memory
1.0

0.5

0.0
0 5 10 15 20 25 30 35 40
SNR [dB]

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 14

7
 Summary: rate distortion theory

n Lagrangian cost function J=D+ λ R can be used instead of

solving constrained rate or distortion minimization.
n Rate-distortion theory: minimum transmission bit-rate for
given distortion
n Shannon Lower Bound assumes statistical independence
between distortion and reconstructed signal
n R(D*) for memoryless Gaussian source and MSE: 6 dB/bit
n R(D*) for Gaussian source with memory and MSE: encode
spectral components independently, introduce white noise,
suppress small spectral components
n Theoretical gain ~2.3 bits/sample by exploiting spatial
redundancy in the video signal

Bernd Girod: EE368b Image and Video Compression Rate Distortion Theory no. 15