► Difference between image enhancement

and restoration
Image Restoration
► Enhancement is the process of manipulating
an image so that the result is more suitable
than the original for a specific-application.

4/5/2020 2

Image Restoration Image restoration vs. image enhancement

► Image restoration: attempts to recover an image that has  Enhancement:

been degraded by using a prior knowledge of the ► largelya subjective process
degradation phenomenon. ► Priori
knowledge about the degradation is not a must
(sometimes no degradation is involved)
► Procedures are heuristic and take advantage of the
► Thus, restoration techniques are oriented toward modeling psychophysical aspects of human visual system
the degradation and applying the inverse process in order
 Restoration:
to recover the original image.
► more an objective process
► Images are degraded
► This approach usually involves formulating a criterion of ► Tries to recover the images by using the knowledge about the
goodness that will yield an optimal estimate of desired degradation
result.
4/5/2020 3 4/5/2020 4

1
 A Model of Image Degradation/Restoration A Model of Image Degradation/Restoration
Process Process

► Degradation If H is a linear, position-invariant process, then

 Degradation function H the degraded image is given in the spatial domain by
 Additive noise  ( x, y ) g ( x, y )  h ( x, y ) f ( x , y )   ( x, y )

4/5/2020 5 4/5/2020 6

Noise Sources Noise Models (1)

► The principal sources of noise in digital images arise during ► White noise
image acquisition and/or transmission  The Fourier spectrum of noise is constant

 Image acquisition ► With the exception of spatially periodic noise, we assume

e.g., light levels, sensor temperature, etc.  Noise is independent of spatial coordinates
 Noise is uncorrelated with respect to the image itself
 Transmission
e.g., lightning or other atmospheric disturbance in wireless
network

4/5/2020 8 4/5/2020 9

2
 Noise Models (2) Noise Models (3)

 Gaussian noise  Erlang (gamma) noise: Laser imaging

Electronic circuit noise, sensor noise due to poor illumination and/or
high temperature (This type of noise occurs in images mostly due to illumination problems)

 Rayleigh noise  Exponential noise: Laser imaging

Range imaging (This type of noise occurs in images mostly due to illumination problems)
This type of noise is mostly present in range image.
 Uniform noise: Least descriptive; Basis for numerous random
Range images are used in many remote sensing applications where the number generators
pixel value indicates the distance between the object and the camera
system.
 Impulse noise: quick transients,
such as faulty switching

4/5/2020 10 4/5/2020 11

Gaussian Noise (1) Gaussian Noise (2)

The PDF of Gaussian random variable, z, is given by The PDF of Gaussian random variable, z, is given by
1 1
p( z )  e  ( z  z ) /2 p( z )  e  ( z  z ) /2
2 2 2 2

2 2

where, z represents intensity  70% of its values will be in the range

z is the mean (average) value of z (   ), (   )

 is the standard deviation
 95% of its values will be in the range

(  2 ), (  2 )
4/5/2020 12 4/5/2020 13

3
 Rayleigh Noise Erlang (Gamma) Noise

The PDF of Rayleigh noise is given by The PDF of Erlang noise is given by
2  a b z b 1  az
 ( z  a )e
 ( z  a )2 / b
for z  a  e for z  0
p( z )   b p( z )   (b  1)!
 0 for z  a 0 for z  a


The mean and variance of this density are given by

The mean and variance of this density are given by
z  a  b / 4
z b/a
b(4   )
2   2  b / a2
4

4/5/2020 14 4/5/2020 15

Exponential Noise Uniform Noise

The PDF of uniform noise is given by

The PDF of exponential noise is given by
 1
 ae  az for z  0  for a  z  b
p( z )   p( z )   b  a
0 for z  a  0 otherwise

The mean and variance of this density are given by The mean and variance of this density are given by
z  1/ a z  (a  b) / 2
 2  1/ a 2  2  (b  a ) 2 /12

4/5/2020 16 4/5/2020 17

4
 Impulse (Salt-and-Pepper) Noise

The PDF of (bipolar) impulse noise is given by

 Pa for z  a

p( z )   Pb for z  b
0
 otherwise

if b  a, gray-level b will appear as a light dot,

while level a will appear like a dark dot.

If either Pa or Pb is zero, the impulse noise is called

unipolar
4/5/2020 18 4/5/2020 19

Examples of Noise: Noisy Images(1)

Examples of Noise: Original Image

4/5/2020 20 4/5/2020 21

5
 Examples of Noise: Noisy Images(2)
Linear, Position-Invariant Degradations

g ( x, y)  H  f ( x, y)  ( x, y)

4/5/2020 22 4/5/2020 23

Linear, Position-Invariant Degradations Linear, Position-Invariant Degradations

H is linear
H  af1 ( x, y )  bf 2 ( x, y )   aH  f1 ( x, y )   bH  f 2 ( x, y)  Assume for a moment that  ( x, y )  0
f1 and f 2 are any two input images. if H is a linear operator and position invariant,
H  ( x   , y   )   h( x   , y   )
An operator having the input-output relationship g ( x, y )  H  f ( x, y ) 
g ( x, y )  H  f ( x, y )  is said to be position invariant  
  f ( ,  ) H  ( x   , y   )  d d 
if   Convolution
  integral in 2-D
H  f ( x   , y   )  g ( x   , y   )  
 
f ( ,  )h( x   , y   )d d 
for any f ( x, y ) and any  and  .
4/5/2020 24 4/5/2020 26

6
 Estimation of Degradation Model
Linear, Position-Invariant Degradations Degradation model:
g ( x, y )  f ( x, y )  h ( x, y )   ( x, y )
or
In the presence of additive noise, G ( u , v )  F ( u , v ) H ( u , v )  N ( u, v )
if H is a linear operator and position invariant,
Purpose: to estimate h(x,y) or H(u,v)

  Why? If we know exactly h(x,y), regardless of noise, we can do

g ( x, y )    f ( ,  )h( x   , y   )d d    ( x, y ) deconvolution to get f(x,y) back from g(x,y).
 

 h ( x, y ) f ( x , y )   ( x, y )
Methods:
1. Estimation by Image Observation
G (u, v)  H (u, v) F (u, v)  N (u , v)
2. Estimation by Experiment

4/5/2020 27
3. Estimation by Modeling (Images from Rafael C. Gonzalez and Richard E.
Wood, Digital Image Processing, 2nd Edition.

Estimation by Image Observation Estimation by Experiment

Original image (unknown) Degraded image Used when we have the same equipment set up and can repeat the
experiment.
Response image from
f(x,y) f(x,y)*h(x,y) g(x,y) Input impulse image the system

System
Observation
H( )
DFT Subimage
Estimated Transfer Gs (u, v) g s ( x, y )
function A ( x, y ) g ( x, y )
Gs (u, v ) Restoration
H ( u, v )  H s ( u, v )  process by DFT DFT
Fˆs (u, v ) estimation
DFT Reconstructed DFT A ( x, y) A G ( u, v )
This case is used when we Fˆs ( u , v )
know only g(x,y) and cannot Subimage G ( u, v )
H ( u, v ) 
repeat the experiment! fˆs ( x, y) A (Images from Rafael C. Gonzalez and Richard E.
Wood, Digital Image Processing, 2nd Edition.

7
 Estimation by Modeling
3. Mathematical Modeling (1) Used when we know physical mechanism underlying the image
formation process that can be expressed mathematically.

Original image Severe turbulence Example:

► Environmental conditions cause degradation
Atmospheric
Turbulence model
A model about atmospheric turbulence
H (u, v)  ek ( u v
2 2 5/ 6
)

H (u , v)  e  k (u  v 2 )5 / 6
2

k = 0.0025
k : a constant that depends on Mild turbulence Low turbulence
the nature of the turbulence
if k is increase Turbulence also increase
if k is decrease Turbulence also decrease k = 0.001
4/5/2020 35 k = 0.00025
(Images from Rafael C. Gonzalez and Richard E.
Wood, Digital Image Processing, 2nd Edition.

Mathematical Modeling (2) Mathematical Modeling (4)

g ( x, y )   f  x  x0 (t ), y  y0 (t ) dt
T

► Derive a mathematical model from basic principles 0

 
G (u , v)    g ( x, y )e  j 2 (ux  vy ) dxdy
 
E.g., An image blurred by uniform linear motion between
   f  x  x (t ), y  y (t )dt  e
  T
the image and the sensor during image acquisition  0 0
 j 2 ( ux  vy )
dxdy
  0

     f  x  x0 (t ), y  y0 (t )  e  j 2 (ux  vy ) dxdy dt

T  

0 
   
T  j 2 ux0 ( t )  vy0 ( t )
  F (u , v)e dt
0
T  j 2 ux0 ( t )  vy0 ( t )
 F (u , v)  e dt
0
4/5/2020 37 4/5/2020 39

8
 Mathematical Modeling (4) Mathematical Modeling (5)

T  j 2 ux0 ( t )  vy0 ( t ) 
H (u , v )   e dt Suppose that the image undergoes uniform linear motion
0
in the x-direction and y-direction, at a rate given by
Suppose that the image undergoes uniform linear motion
x0 (t )  at / T and y0 (t )  bt / T
in the x-direction only, at a rate given by x0 (t )  at / T .
T  j 2 ux0 ( t )  vy0 ( t ) 
T H (u , v )   e dt
H (u, v)   e  j 2 ux0 (t ) dt 0
0
T
T   e  j 2 [ ua  vb ]t /T dt
  e  j 2 uat /T dt 0
0
T
T  sin  (ua  vb)  e  j (ua  vb )
 sin( ua)e  j ua  (ua  vb)
 ua
4/5/2020 40 4/5/2020 41

Blur

out-of-focus blur motion blur

Question 1: How do you know they are blurred?

I’ve not shown you the originals!
4/5/2020 42 2020/4/5 Question 2: How do I deblur an image?

9
 Linear Blur Model Linear Blur Model
• Spatial domain • Frequency (2D-DFT) domain

x(m,n) h(m,n) y(m,n) X(u,v) H(u,v) Y(u,v)

blurring filter blurring filter

2020/4/5
Gaussian blur motion blur 2020/4/5
Gaussian blur motion blur

Blurring Effect Image Restoration: Deblurring/Deconvolution

x(m,n) h(m,n) g(m,n) g(m,n) ^

x(m,n)
Gaussian blur blurring filter deblurring/
deconvolution
filter

• Non-blind deblurring/deconvolution
Given: observation g(m,n) and blurring function h(m,n)
Design: g(m,n), such that the distortion between x(m,n) and ^
x(m,n)
is minimized
motion blur

^
x(m,n)
2020/4/5 2020/4/5
From [Gonzalez & Woods]

10
 Inverse Filtering Inverse Filtering
An estimate of the transform of the original image
N (u, v)
G (u , v) F (u, v)  F (u, v) 
F (u , v)  H (u, v)
H (u , v)
1. We can't exactly recover the undegraded image
because N (u, v) is not known.
F (u, v) H (u, v)  N (u, v)
F (u , v)  2. If the degradation function has zero or very
H (u, v)
small values, then the ratio N (u, v) / H (u, v) could
N (u, v)
 F (u , v)  easily dominate the estimate F (u, v).
H (u, v)
4/5/2020 48 4/5/2020 49

► Advantage:
Inverse Filtering
1. It requires only the blur point spread function as a priori
knowledge
EXAMPLE
2. The inverse filter produces perfect reconstruction in the
absence of noise
The image in Fig. 5.25(b) was inverse filtered using the
► Disadvantage: exact inverse of the degradation function that generated
that image. That is, the degradation function is
1. It is not always possible to obtain inverse. For inverse to
exist, the matrix should be non-singular.
5/6
 k  u  M /2   ( v  N /2) 2 
2

H (u, v)  e  
, k  0.0025
2. It will not perform well in presence of noise. If the noise is
present in the image, inverse filter tend to amplify noise
which is undesirable.
4/5/2020 50 4/5/2020 51

11
 Inverse Filtering

One approach is to limit the filter frequencies to values near the origin.

EXAMPLE

The image in Fig. 5.25(b) was inverse filtered using the A Butterworth
lowpass
exact inverse of the degradation function that generated function of
order 10
that image. That is, the degradation function is
5/6
 k  u  M /2   ( v  N /2)2 
2

H (u, v)  e  

k  0.0025, M  N  480.
4/5/2020 52 4/5/2020 53

Deblurring: Inverse Filtering Deblurring: Pseudo-Inverse Filtering

x(m,n) h(m,n) y(m,n) g(m,n) ^

x(m,n) x(m,n) h(m,n) y(m,n) g(m,n) ^
x(m,n)
blurring filter inverse filter blurring filter deblur filter

X(u,v) H(u,v) = Y(u,v) 1 What if at some (u,v), H(u,v)

1 Inverse filter: G(u,v) =
G(u,v) = H(u,v) is 0 (or very close to 0) ?
H(u,v)
Y(u,v) 1
X(u,v) = = Y(u,v) Exact recovery!  1
H(u,v) H(u,v)  | H (u, v) | 
Pseudo-inverse filter: G (u, v)   H (u, v) small
 0 | H (u, v) |  threshold
2020/4/5 2020/4/5

12
 Inverse and Pseudo-Inverse Filtering More Realistic Distortion Model
1 additive white Gaussian noise
G(u, v)  w(m,n)
H (u, v)
x(m,n) h(m,n) + y(m,n) g(m,n) ^
x(m,n)
blurring filter deblur filter

Y(u,v) = X(u,v) H(u,v) + W(u,v)

• What happens when an inverse filter is applied?

blurred image
X (u, v) H (u, v)  W (u, v)
Xˆ (u, v)  Y (u, v)G (u, v) 
 1 H (u, v)
 | H (u, v) | 
G (u, v)   H (u , v)
W (u, v)
 0 | H (u, v) |   X (u, v)  close to zero at high frequencies
H (u, v)
2020/4/5 2020/4/5
Adapted from Prof. Xin Li  = 0.1

Radially Limited Inverse Filtering Radially Limited Inverse Filtering

 1
Radially limited  u 2  v2  R Image
inverse filter: G (u , v )   H (u, v) size:
 u 2  v2  R 480x480
 0

• Motivation Original Blurred Inverse filtered

– Energy of image signals is
concentrated at low frequencies
– Energy of noise uniformly is Radially
R limited
distributed over all frequencies
inverse
– Inverse filtering of image signal filtering:
dominated regions only

2020/4/5 2020/4/5
R = 40 R = 70 R = 85
From [Gonzalez & Woods]

13
 Minimum Mean Square Error (Wiener) Minimum Mean Square Error (Wiener)
Filtering Filtering
 N. Wiener (1942) The minimum of the error function is given in the frequency domain
 Objective by the expression
Find an estimate of the uncorrupted image such that the mean  H *(u , v) S f (u , v) 
square error between them is minimized F (u , v)    G (u, v)
 S f (u , v) | H (u , v) |  S (u, v) 
2

 H *(u , v) 

 
  G (u , v)
e2  E ( f  f )2  | H (u , v) |  S (u , v ) / S f (u , v) 
2

 1 | H (u , v) |2 
  G (u, v)
 H (u , v ) | H (u , v ) | 2
 S (u , v ) / S f (u , v ) 
4/5/2020 60 4/5/2020 61

Minimum Mean Square Error (Wiener) Minimum Mean Square Error (Wiener)
Filtering Filtering

 1 | H (u, v) |2 
F (u , v)    G (u , v)
 H (u , v) | H (u, v) |  S (u, v) / S f (u, v) 
2
 1 | H (u , v) |2 
F (u , v)  
(u , v) |2  K 
G (u , v)
 H (u , v ) | H
H (u , v) : degradation function
H *(u, v): complex conjugate of H (u , v) K is a specified constant. Generally, the value of K
| H (u, v) |  H *(u, v) H (u, v)
2 is chosen interactively to yield the best visual results.

S (u , v) | N (u , v) |2  power spectrum of the noise

S f (u , v) | F (u , v) |2  power spectrum of the undegraded image

4/5/2020 62 4/5/2020 63

14
 Minimum Mean Square Error (Wiener) Left:
Filtering degradated
image

Middle:
inverse
filtering

Right:
Wiener
filtering

4/5/2020 64 4/5/2020 65

Wiener (Least Square) Filtering Weiner Filtering

H * (u, v)
Wiener filter: G(u, v)  Blurred Inverse
| H (u, v) |2  K image filtering

 W2 noise power
K
 X2 signal power

• Optimal in the least MSE sense, i.e. Radially limited Weiner

G(u, v) is the best possible linear filter that minimizes inverse filtering filtering
R = 70

error energy  E  Xˆ (u , v)  X (u , v) 
2

 

• Have to estimate signal and noise power

2020/4/5 2020/4/5
From [Gonzalez & Woods]

15
 Inverse vs. Weiner Filtering Weiner Image Denoising
distorted inverse filtering Wiener filtering w(m,n)
motion blur
+ x(m,n) h(m,n) + y(m,n)
noise
• What if no blur, but only noise, i.e. h(m,n) is an impulse
or H(u, v) = 1 ?
less noise
H * (u, v)  W2
Wiener filter: G(u, v)  where K
| H (u, v) |2  K  X2

for H(u,v) = 1
less noise
Wiener 1 1  X2
denoising filter: G(u, v)   
1  K 1   W2 /  X2  X2   W2
2020/4/5 2020/4/5
From [Gonzalez & Woods] Typically applied locally in space

Weiner Image Denoising Summary of Linear Image Restoration Filters

1
Inverse filter: G(u,v) =
H(u,v)
adding noise
noise var = 400  1
 | H (u, v) | 
Pseudo-inverse G (u, v)   H (u , v)
filter:  0 | H (u, v) | 

 1
Radially limited  u 2  v2  R
G (u, v)   H (u, v)
inverse filter:  0
 u 2  v2  R

local Wiener H * (u, v)  W2

Wiener filter: G(u, v)  where K
denoising
| H (u, v) |2  K  X2
Wiener  X2
denoising filter: G(u, v) 
2020/4/5 2020/4/5  X2   W2

16
 Constrained Least Squares Filtering Constrained Least Squares Filtering
 H *(u, v) 
F (u, v)   2
G (u, v)
 | H (u, v) |  | P(u, v) | 
2

► In Wiener filter, the power spectra of the undegraded

image and noise must be known. Although a constant
estimate is sometimes useful, it is not always suitable. P(u, v) is the Fourier transform of the function
 0 1 0 
Constrained least squares filtering just requires the mean
p( x, y )   1 4 1  laplacian filter
►
and variance of the noise.
 0 1 0 
 is a parameter ,
to control the balance between noise artifacts and
4/5/2020 72 4/5/2020 cositency with observed data 73

Examples

► P(u,v) has large amplitude at high

frequencies where the noise tends to be
dominant.
► It modifies the denominator to reduce the
noise effect at high frequencies.
► Proper choice of P(u,v) and gamma can
minimize higher order derivatives

4/5/2020 74 4/5/2020 75

17