Image Restoration: Govind Sharma

Restoration Model
Restoration in presence of noise only

Restoration in presence of blur and noise
Image Restoration
EE 604
Govind Sharma
Department of Electrical Engineering

Indian Institute of Technology, Kanpur
11/4/2019
G. Sharma Restoration
Restoration Model
Outline
Restoration Model
Restoration Model
Image Restoration
Image restoration may be viewed as an estimation process in which

operations are performed on an observed or measured image to
estimate the ideal image that would be observed if no degradation
were present in an imaging system.
Restoration Model
Restoration Model
Image Perfect Perfect

light imaging image fI [k, l]
distribution system digitizer
Physical Physical Image Image

imaging image restoration display
system digitizer system system
Restoration Model
Potential degradations
(a) Diffraction in optical system

(b) Sensor nonlinearities
(c) Optical system aberrations
(d) Atmospheric turbulence effects
(e) Image motion blur
(f) Camera motion blur
(g) Geometric distortions
(h) Noise due to electronics, film granularities etc.
Restoration Model
Restoration Goal
Usually, it will not be possible to restore perfectly the observed
image. We try to minimize an error criterion. Minimum mean
squared error criterion
h i
E = E |fI (x , y ) − fˆ(x , y )|2
where fI (x , y ) is output of perfect imaging system and fˆ(x , y ) is

display image of a practical system.
Since restoration process is to be performed digitally, we use the
criterion on the sampled images.
h i
E = E |f [k, l] − fˆ[k, l]|2
where f [k, l] is output of the perfect digitizer and fˆ[k, l] is the

output of the image restoration system.
Restoration Model
Simplified model of degradation
There are no general solutions for the restoration problem. We

take a simplified model
g[k, l] = H(f [k, l]) + n[k, l]
where H() is a general transformation, f [k, l] is the output of the

perfect digitizer, g[k, l] is degraded image and n[k, l] is noise.
First we assume that H() is an Identity operator. That is
degradation is due to noise only.
Restoration Model
Some noise models
We assume that noise is independent of spatial coordinates and

that it is uncorrelated to the image pixel values. Some important
noise probability densities are
(a) Gaussian (circuit and sensor noise)
(b) Rayleigh (in range imaging)
(c) Gamma (laser imaging)
(d) Exponential (laser imaging)
(e) Uniform (quantization)
(f) Impulse or salt-pepper (due to faulty switching)
Restoration Model
Test Pattern
Restoration Model
Effect of noise
Restoration Model
Effect of noise
Restoration Model
Spatial filtering
When only additive noise is present we usually use spatial filtering.

Some of the filters are
(a) Arithmetic mean filter:
1 X
fˆ[k, l] = g[i, j]
MN [i,j]∈S
kl
where Skl is a rectangular neighborhood of size M × N

centered at point [k, l]. This operation can be implemented by
a spatial filter whose all coefficients are 1/MN. Image is
blurred by this filter.
Restoration Model
Spatial filtering
(b) Geometric mean filter:

  1
MN
fˆ[k, l] = 
Y
g[i, j]
[i,j]∈Skl
It performs slightly better than arithmetic mean filter.

(c) Harmonic mean filter:
MN
fˆ[k, l] = P
1/g[i, j]
[i,j]∈Skl
It works well for salt noise, but fails for pepper noise.
Restoration Model
Image restoration example
Restoration Model
Spatial filtering
(d) Contraharmonic mean filter:
g[i, j]Q+1
P
[i,j]∈Skl
fˆ[k, l] =
g[i, j]Q
P
[i,j]∈Skl
where Q is called the order of the filter. It works well for salt
noise if Q < 0, and for pepper noise if Q > 0. It reduces to
arithmetic mean if Q = 0 and harmonic mean if Q = −1.
Restoration Model
Restoration example
Restoration Model
Spatial filtering
(e) Median filter: Median of the neighborhood

(f) Midpoint filter: average of minimum and maximum values in
the neighborhood
(g) Alpha trimmed mean: d/2 smallest and largest values in the
neighborhood are dropped and arithmetic mean of the
remaining values is taken.
(h) Adaptive filtering
Restoration Model
Restoration example
Restoration Model
Adaptive mean filter
σ 2
fˆ[k, l] = g[k, l] − n2 (g[k, l] − mL )
σL
where ML = local mean of the pixel values in Skl , σL2 = local
variance of the pixel values in Skl and σn2 = noise variance. If noise
variance is much smaller than local variance this filter returns the
estimated value close to observed value.
Restoration Model
Restoration example
Restoration Model
Adaptive median filter
We use the following definitions
zmax = maximum intensity value in Skl

zmin = minimum intensity value in Skl
zmed = median intensity value in Skl
zkl = observed intensity value g[k, l]
Smax = maximum size of neighborhood Skl
A1 = zmed − zmin
A2 = zmed − zmax
B1 = zkl − zmin
B2 = zkl − zmax
Restoration Model
Adaptive median filter
Stage A: If A1 > 0 and A2 < 0, go to Stage B

else increase the window size
If window size ≤ Smax repeat Stage A
else output zmed
Stage B: If B1 > 0 and B2 < 0, output zkl
else output zmed
Restoration Model
Restoration example
Restoration Model
Linear shift invariant degradation
Simplified degradation model we are using is
g[k, l] = H(f [k, l]) + n[k, l]
We further make assumption that H is a linear shift invariant

operator. So
g[k, l] = h[k, l] ∗ f [k, l] + n[k, l]

G(u, v ) = H(u, v )F (u, v ) + N(u, v )
h[k, l] is also known as point spread function (PSF). Restoration

method in this case is also known as deconvolution.
Restoration Model
Estimation of degradation function
(a) Estimation by image observation

(b) Estimation by experimentation
(c) Estimation by modeling
Restoration Model
PSF estimation by experimentation
Restoration Model
Example estimation by modeling

Suppose image f (x , y ) undergoes planar motion x0 (t), y0 (t) are
components of motion. If duration of exposure is T , then observed
image is
ZT
g(x , y ) = f (x − x0 (t), y − y0 (t))dt
0
Z∞ Z∞
G(u, v ) = g(x , y )e −j2π(ux +vy ) dxdy
−∞ −∞
Z∞ Z∞ ZT
= f (x − x0 (t), y − y0 (t))dte −j2π(ux +vy ) dxdy
−∞ −∞ 0
Restoration Model
ZT Z∞ Z∞
G(u, v ) = f (x − x0 (t), y − y0 (t))e −j2π(ux +vy ) dxdydt
0 −∞ −∞
ZT
= F (u, v )e −j2π(ux0 (t)+vy0 (t)) dt
0
ZT
= F (u, v ) e −j2π(ux0 (t)+vy0 (t)) dt
0
ZT
H(u, v ) = e −j2π(ux0 (t)+vy0 (t)) dt
0
Restoration Model
If image has uniform motion x0 (t) = at/T , y0 (t) = bt/T then
ZT
H(u, v ) = e −j2π(uat/T +vbt/T ) dt
0
T
= sin(π(ua + vb))e −jπ(ua+vb)
π(ua + vb)
Restoration Model
Motion Blur example
Restoration Model
Restoration methods
(a) Inverse filtering

(b) Wiener filtering
(c) Least squares and Constrained least squares
Restoration Model
Inverse filtering
1
Q(u, v ) = H(u,v )
F̂ (u, v ) = Q(u, v )G(u, v )
N(u,v )
= F (u, v ) + H(u,v )
If H(u, v ) has small value inverse filter may blow up. So it is

modified as
(
1
Q(u, v ) = H(u,v ) , if H(u, v ) > threshold
0, otherwise
Restoration Model
Wiener Filter
Let us first consider 1-D case. Let y [k] be observed sequence and
s[k] is a sequence we want to estimate. We assume that these two
processes are zero mean jointly wide sense stationery sequences
with known auto and cross correlation functions. Estimation
method is linear minimum mean squared error criterion. Let w [k]
represent the impulse response and W (u) its Fourier transform
then we want to minimize
E = E [|s[k] − ŝ[k]|2 ]
This can be done using orthogonality principle.
Restoration Model
Wiener filter
∞
X
E [(s[k] − w [l]y [k − l])y [m]] = 0, −∞ < m < ∞
l=−∞
Taking expectation inside summation sign we get
∞
X
E [s[k]y [m]] = w [l]E [y [k − l]y [m]]
l=−∞
X∞
Rsy [k − m] = w [l]Ryy [k − m − l]
l=−∞
X∞
Rsy [n] = w [l]Ryy [n − l] (k-m=n)
l=−∞
Rsy [n] = w [n] ∗ Ryy [n]
Ssy (u) = W (u)Syy (u)
Restoration Model
Wiener filter
Let
g[k] = h[k] ∗ f [k] + n[k]
Since noise n[k] is uncorrelated to f [k], we get Rgg [m] as sum of
two terms and so
Sgg (u) = |H(u)|2 Sff (u) + Snn (u)
where we use the relationship between power spectral densities

when a random process is passed through a linear system. Similarly
Sfg (u) = H ∗ (u)Sff (u)
Restoration Model
Wiener filter
Wiener filter is
H ∗ (u)Sff (u)
W (u) =
|H(u)|2 Sff (u) + Snn (u)
In 2-D case we will get
H ∗ (u, v )Sff (u, v )

W (u, v ) =
|H(u, v )|2 Sff (u, v ) + Snn (u, v )
H ∗ (u, v )
=
|H(u, v )|2 + SSnn (u,v )
ff (u,v )
Restoration Model
Motion blur restoration example
Restoration Model
Wiener filter noise and blur
Restoration Model
Least squares
Recall that linear shift invariant model
g[k, l] = h[k, l] ∗ f [k, l] + n[k, l]
Suppose image has size M × N and filter has size J × K then
blurred image will have size (M + J − 1) × (N + K − 1). We can
write this relation ship in vector space form
g = Bf + n
We want to find a matrix C so that fˆ = C g and
k g − B fˆk2
is minimized. The minimum norm least square solution is given by
fˆ = B − g
where pseudo inverse B − is obtained via singular value
decomposition.
Restoration Model
Least square restoration example
Restoration Model
Least square restoration example
Restoration Model
Constrained least squares
Least squares method is very sensitive to noise. To reduce noise

sensitivity a measure of smoothness like second derivative is
minimized. Thus criterion is
M−1
X N−1
X 2
ξ= ∇2 f [k, l]
k=0 l=0
subject to constraint
k g − B fˆk2 = knk2
Second derivative is calculated using Laplacian operator
Restoration Model
Laplacian operator is given by ∇2 f [k, l] = p[k, l] ∗ f [k, l] where

 
0 1 0
p[k, l] = 1 −4 1
 
0 1 0
Problem is solved by converting it into frequency domain and

Langrangian multiplier method. The solution is given by
Restoration Model
H ∗ (u, v )
F̂ (u, v ) = G(u, v )
|H(u, v )|2 + λ|P(u, v )|2
where λ is chosen to satisfy the constraint. If noise has variance σ 2
then knk2 = (M + J − 1) × (N + K − 1)σ 2 . For λ = 0 then we get
inverse filter. Problem is solved iteratively by trying various values
of λ.
Restoration Model
Constrained least square restoration example
Restoration Model
References
1. Introduction to Digital Image Processing, William K. Pratt,

CRC Press.
2. Digital Image Processing, Rafael C. Gonzalez and Richard E.
Woods, Prentice Hall.
3. Fundamentals of Digital Image Processing, Anil K. Jain,
Prentice Hall.

Image Restoration: Govind Sharma

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Image Restoration: Govind Sharma

Uploaded by

Copyright:

Available Formats

Restoration Model

Restoration in presence of noise only

Department of Electrical Engineering

Restoration in presence of noise only

Restoration in presence of blur and noise

Image restoration may be viewed as an estimation process in which

Image Perfect Perfect

Physical Physical Image Image

(a) Diffraction in optical system

where fI (x , y ) is output of perfect imaging system and fˆ(x , y ) is

where f [k, l] is output of the perfect digitizer and fˆ[k, l] is the

Simplified model of degradation

There are no general solutions for the restoration problem. We

g[k, l] = H(f [k, l]) + n[k, l]

where H() is a general transformation, f [k, l] is the output of the

Some noise models

We assume that noise is independent of spatial coordinates and

When only additive noise is present we usually use spatial filtering.

where Skl is a rectangular neighborhood of size M × N

(b) Geometric mean filter:

It performs slightly better than arithmetic mean filter.

Image restoration example

(d) Contraharmonic mean filter:

(e) Median filter: Median of the neighborhood

Adaptive mean filter

Adaptive median filter

We use the following definitions

zmax = maximum intensity value in Skl

Adaptive median filter

Stage A: If A1 > 0 and A2 < 0, go to Stage B

Linear shift invariant degradation

Simplified degradation model we are using is

g[k, l] = H(f [k, l]) + n[k, l]

We further make assumption that H is a linear shift invariant

g[k, l] = h[k, l] ∗ f [k, l] + n[k, l]

h[k, l] is also known as point spread function (PSF). Restoration

Estimation of degradation function

(a) Estimation by image observation

PSF estimation by experimentation

Example estimation by modeling

Example estimation by modeling

Example estimation by modeling

If image has uniform motion x0 (t) = at/T , y0 (t) = bt/T then

Motion Blur example

(a) Inverse filtering

If H(u, v ) has small value inverse filter may blow up. So it is

This can be done using orthogonality principle.

Sgg (u) = |H(u)|2 Sff (u) + Snn (u)

where we use the relationship between power spectral densities

Sfg (u) = H ∗ (u)Sff (u)

In 2-D case we will get

H ∗ (u, v )Sff (u, v )

Motion blur restoration example

Wiener filter noise and blur

Least square restoration example

Least square restoration example

Constrained least squares

Least squares method is very sensitive to noise. To reduce noise

Second derivative is calculated using Laplacian operator

Constrained least squares

Laplacian operator is given by ∇2 f [k, l] = p[k, l] ∗ f [k, l] where

Problem is solved by converting it into frequency domain and