You are on page 1of 85

IMAGE RESTORATION

Outline
A model of the image degradation / restoration
process
Noise models
Restoration in the presence of noise only
spatial filtering
Periodic noise reduction by frequency domain
filtering
Linear, position-invariant degradations
Estimating the degradation function
Inverse filtering & Wiener filtering
Constrained Least square filtering
Geometric mean filter
Geometric and spatial transformation

What is Image Restoration?


Image restoration is to restore a
degraded image back to the original
image
Image enhancement is to manipulate
the image so that it is suitable for a
specific application.

Image Restoration
Image restoration attempts to restore
images that have been degraded
Identify the degradation process and
attempt to reverse it
Similar to image enhancement, but
more objective

A model of the image


degradation/restoration process

g(x,y)=f(x,y)*h(x,y)+(x,y) Spatial dom


G(u,v)=F(u,v)H(u,v)+N(u,v) Frequency

A model of the image


degradation/restoration process
Where,
f(x,y) - input image
f^(x,y) - estimated original image
g(x,y) - degraded image
h(x,y) - degradation function
(x,y) - additive noise term

Noise models
Sources of noise
Image acquisition (digitization) - Imaging
sensors can be affected by ambient conditions
Image transmission - Interference can be added
to an image during transmission

Spatial properties of noise


Statistical behavior of the gray-level values of
pixels
Noise parameters, correlation with the image

Frequency properties of noise


Fourier spectrum
Ex. white noise (a constant Fourier spectrum)

Noise models
Noise model is decided based on
understanding of
the physics of the sources of noise.
Gaussian: poor illumination
Rayleigh: range image
Gamma(Erlang),
Exponential:
laser
imaging
Impulse: faulty switch during imaging
Uniform: least used

Noise probability density


functions
Noise cannot be predicted but can be
approximately described in statistical way
using the probability density function (PDF)

Gaussian noise
Mathematical tractability in spatial
and frequency domains
Used frequently in practice
Electronic circuit noise and sensor
noisep ( z ) 1 e ( z ) 2 / 2 2

Note:

Intensit

p( z )dz 1

mean

variance

Gaussian noise PDF

70% in [(), ()]


95% in [(), ()]

Rayleigh noise
2
2
( z a )e ( z a ) / b
p( z ) b
0

for z a
for z a

The mean and variance of this density are given


by
b( 4 )
2
a b / 4 and
4

a and b can be obtained through mean and


variance

Rayleigh noise PDF

Erlang (Gamma) noise

p( z )

a b z b 1 az
e
(b 1)!

for z 0
for z 0

The mean and variance of this density are


given by b / a and 2 b2
a

a and b can be obtained through mean and


variance

Gamma noise (PDF)

Exponential noise
for z 0

for z 0

p( z )

ae az

The mean and variance of this density


are given by
1 / a and 2

1
a2

Exponential Noise PDF


Special case of Erlang PDF with b=1

Uniform noise
Less practical, used for random
number generator
1

if a z b
p( z ) b a
0
otherwise
ab
Mean:
2
2
(
b

a
)
2

Variance:
12

Uniform PDF

Impulse (salt-and-pepper)
nosie
Quick transients, such as faulty switching
during imaging

Pa for z a

p ( z ) Pb for z b
0 otherwise

If either Pa or Pb is zero, it is called unipolar.


Otherwise, it is called bipolar.

Impulse (salt-and-pepper)
nosie PDF

Image Degradation with Additive Noise


g ( x, y ) f ( x, y ) ( x, y )
Degraded images

Original image

Histogram

Image Degradation with Additive Noise (cont.)


Degraded images

Histogram

Periodic noise
Is an image that arises from electrical
and electromechanical interference
during image acquisition
It can be observed by visual
inspection both in the spatial domain
and frequency domain
can be reduced significantly via
frequency domain filtering

Periodic Noise

Periodic noise
looks like dots
In the frequency
domain

Estimation of noise
parameters
Periodic noise
Observe the frequency spectrum

Random noise with unknown PDFs


Case 1: imaging system is available
Capture images of flat environment

Case 2: noisy images available


Take a strip from constant area
Draw the histogram and observe it
Measure the mean and variance
09/02/16

Estimation of Noise
Image histogram cannot be
used to estimate noise PDF.

It is better to use the


histogram of one area of an
image that has constant
intensity to estimate noise
PDF.

09/02/16

Noise Removal Restoration


Methods
Mean filters

Arithmetic mean filter


Geometric mean filter
Harmonic mean filter
Contra-harmonic
mean filter

Order statistics filters

09/02/16

Median filter
Max and min filters
Mid-point filter
alpha-trimmed filters

Adaptive filters
Adaptive local noise
reduction filter
Adaptive median filter

Degradation model:

Mean Filters

g ( x, y ) f ( x, y ) h( x, y ) ( x, y )
To remove this part

Arithmetic mean filter or moving average filter

f ( x, y ) 1
g ( s, t )

mn ( s ,t )S xy
- is the simplest of the mean filters
- smooths local variations in an image
- noise is reduced as a result of blurring

09/02/16

mn = size of moving window

Mean Filters Contd...


Geometric mean filter

f ( x, y )

1
mn

g
(
s
,
t
)

( s ,t )S xy

- achieves smoothing as compared to arithmetic mean filter


- it tends to lose less image detail in the process

09/02/16

Mean Filter: Example

Original
image

Image
obtained
using a 3x3
arithmetic
mean filter

09/02/16

Image
corrupted
by AWGN

Image
obtained
using a 3x3
geometric
mean filter

Harmonic and Contraharmonic Filters


Harmonic mean filter

f ( x, y )

mn

( s ,t )S xy

1
g ( s, t )

Contraharmonic mean filter

f ( x, y )

mn = size of moving window

Q 1
g
(
s
,
t
)

( s ,t )S xy

g ( s, t )

Works well for salt noise


but fails for pepper noise

Positive Q is suitable for


eliminating pepper noise.
Negative Q is suitable for
eliminating salt noise.

( s ,t )S xy

Q = the filter order

09/02/16

For Q = 0, the filter reduces to an arithmetic mean filter.


For Q = -1, the filter reduces to a harmonic mean filter.

Contraharmonic Filters: Example


Image
corrupted
by pepper
noise with
prob. = 0.1

Image
corrupted
by salt
noise with
prob. = 0.1

Image
obtained
using a 3x3
contraharmonic
mean filter
With Q = 1.5

Image
obtained
using a 3x3
contraharmonic
mean filter
With Q=-1.5

09/02/16

Original image

Order-Statistic Filters
subimage

Statistic parameters
Mean, Median, Mode,
Min, Max, Etc.

Moving
window

09/02/16

Output image

Order-Statistics Filters
Median filter

f ( x, y ) median g ( s, t )
( s ,t )S xy

Max filter

f ( x, y ) max g ( s, t )
( s ,t )S xy

Reduce dark noise


(pepper noise)

Min filter

f ( x, y ) min g ( s, t )
( s ,t )S xy

Reduce bright noise


(salt noise)

Midpoint filter

09/02/16

f ( x, y ) 1 max g ( s, t ) min g ( s, t )
( s ,t )S xy
2 ( s ,t )S xy

Median Filter : How it works


A median filter is good for removing impulse, isolated noise
Salt noise

Pepper noise
Median

Degraded image

Moving
window

Salt noise
Pepper noise

Filter output
Normally, impulse noise has high magnitude
and is isolated. When we sort pixels in the
moving window, noise pixels are usually
at the ends of the array.

Therefore, its rare that the noise pixel will be a median value.
09/02/16

Sorted
array

Median Filter : Example


1

Image
corrupted
by salt-andpepper
noise with
pa=pb= 0.1

09/02/16

Images obtained using a 3x3 median filter

Max and Min Filters: Example


Image
corrupted
by pepper
noise with
prob. = 0.1

Image
corrupted
by salt
noise with
prob. = 0.1

Image
obtained
using a 3x3
max filter

Image
obtained
using a 3x3
min filter

09/02/16

Alpha-trimmed Mean Filter


This filter is useful in situations involving multiple types of noise
such as a combination of salt-and-pepper and Gaussian noise.

f ( x, y )

1
mn d

g ( s, t )

( s ,t )S xy

where,

gr(s,t) represent the remaining mn-d pixels after removing


the d/2 highest and d/2 lowest values of g(s,t).

09/02/16

Alpha-trimmed Mean Filter: Example


1
Image
corrupted
by additive
uniform
noise

Image 2
obtained
using a 5x5
arithmetic
mean filter

09/02/16

Image
additionally
corrupted
by additive
salt-andpepper
noise

Image 2
obtained
using a 5x5
geometric
mean filter

Alpha-trimmed Mean Filter: Example (cont.)


1
Image
corrupted
by additive
uniform
noise

Image 2
obtained
using a 5x5
median filter

09/02/16

Image
additionally
corrupted
by additive
salt-andpepper
noise

Image 2
obtained
using a 5x5
alphatrimmed
mean filter
with d = 5

Alpha-trimmed Mean Filter: Example (cont.)


Image
obtained
using a 5x5
arithmetic
mean filter

Image
obtained
using a 5x5
geometric
mean filter

Image
obtained
using a 5x5
median filter

Image
obtained
using a 5x5
alphatrimmed
mean filter
with d = 5

09/02/16

A model of the image degradation


/restoration process

g(x,y)=f(x,y)*h(x,y)+(x,y)
G(u,v)=F(u,v)H(u,v)+N(u,v)
09/02/16

Linear, position-invariant
degradation
Properties of the degradation function H

Linear system
H[af1(x,y)+bf2(x,y)]=aH[f1(x,y)]
+bH[f2(x,y)]

Position(space)-invariant system
H[f(x,y)]=g(x,y) is position invariant if
H[f(x-, y-)]=g(x-, y-)

09/02/16

Estimation of Degradation Function


Degradation model:
g ( x, y ) f ( x, y ) h ( x, y ) ( x, y )
or

G ( u, v ) F ( u , v ) H ( u , v ) N ( u , v )
If we know exactly h(x,y), regardless of noise, we can do
deconvolution to get f(x,y) back from g(x,y).
Methods:
1. Estimation by Image Observation
2. Estimation by Experiment
3. Estimation by Modeling
09/02/16

Estimation by Image Observation


Original image (unknown)

f(x,y)

Degraded image

f(x,y)*h(x,y)

g(x,y)

Observation

Estimated Transfer
function

DFT

Gs (u, v )

09/02/16

g s ( x, y )
Restoration
process by
estimation

G (u, v )
H (u, v ) H s (u, v ) s
Fs (u, v )
This case is used when we
know only g(x,y) and cannot
repeat the experiment!

Subimage

DFT

Fs (u, v )

Reconstructed
Subimage

fs ( x, y )

Estimation by Experiment
Used when we have the same equipment set up
Response image from
the system

Input impulse image

System
H( )

A ( x , y )

g ( x, y )

DFT

DFT

DFT A ( x, y ) A
09/02/16

G (u, v )
G (u, v )
H (u, v )
A

Estimation by Modeling
Used when we know physical mechanism underlying the image
formation process that can be expressed mathematically.
Original image

Severe turbulence

Example:
Atmospheric
Turbulence model

H (u, v ) e
k = 0.0025
Mild turbulence

Low turbulence

k = 0.001

k = 0.00025

09/02/16

k ( u 2 v 2 )5 / 6

Estimation by Modeling: Motion Blurring


Assume that camera velocity is

( x0 (t ), y0 (t ))

The blurred image is obtained by


T

g ( x, y ) f ( x x0 (t ), y y0 (t ))dt
0

where T = exposure time.

G (u, v )

j 2 ( ux vy )
g
(
x
,
y
)
e
dxdy


09/02/16

f ( x x0 (t ), y y0 (t ))dt e j 2 ( ux vy ) dxdy

f ( x x (t ), y y (t ))e
0

j 2 ( ux vy )

dxdy dt

Estimation by Modeling: Motion Blurring (cont.)

G (u, v )
0

f ( x x (t ), y y (t ))e
0

j 2 ( ux vy )

dxdy dt

F (u, v )e j 2 ( ux0 ( t ) vy0 ( t )) dt


0

F (u, v ) e j 2 ( ux0 ( t ) vy0 ( t )) dt


0

Then we get, the motion blurring transfer function:


T

H (u, v ) e j 2 ( ux0 ( t ) vy0 ( t )) dt


0

For constant motion


T

H (u, v ) e
09/02/16

( x0 (t ), y0 (t )) ( at , bt )
j 2 ( ua vb )

T
dt
sin( (ua vb))e j ( ua vb )
(ua vb)

Motion Blurring Example


For constant motion

H (u, v )

T
sin( (ua vb))e j ( ua vb )
(ua vb)

Original image

09/02/16

Motion blurred image


a = b = 0.1, T = 1

Inverse Filtering
From degradation model:

G ( u , v ) F ( u, v ) H ( u , v ) N ( u , v )
after we obtain H(u,v), we can estimate F(u,v) by the inverse filter:

G (u, v )
N (u, v )

F (u, v )
F (u, v )
H (u, v )
H (u, v )
Noise is enhanced
when H(u,v) is small.
To avoid the side effect of enhancing
noise, we can apply this formulation
to freq. component (u,v) with in a
radius D0 from the center of H(u,v).
In practical, the inverse filter is not popularly used.
09/02/16

Inverse Filtering
The simplest approach to restoration is direct inverse
filtering, where we compute an estimate, (u,v), of the
transform of the original image simply by dividing the
transform of the degraded image, G(u,v), by the
degraded function.
g(m,n) = f(m,n)*h(m,n)+(m,n)
g = Hf +
If the noise is zero the error function is denoted as
T() = ||g- ||2
T() = g2+H2-2H

=0

09/02/16

Inverse Filtering Contd...


To find Mean Square Error
0+2H2-2H =0
2H22H
H >>>>> H = (Inverse Filtering)
G(u,v) = H(u,v)(u,v)
(u,v) =
With addition of noise the observed degraded image in
frequency domain
G(u,v) = F(u,v)H(u,v)+N(u,v)
(1)

09/02/16

Inverse Filtering Contd...


Divide equation one by H(u,v)
=
.(2)

We know that (u,v) =


Substitute (u,v) in eqn (2)
(u,v) = F(u,v)+
If noise is zero the estimated image (u,v) is equal to
original image, but noise will not be properly removed
in inverse filtering.

Inverse Filtering: Example

Original image

Blurred image
Due to Turbulence
09/02/16

Result of applying
the full filter

Result of applying
the filter with D0=70

H (u, v ) e

0.0025( u 2 v 2 )5 / 6

Result of applying
the filter with D0=40

Result of applying
the filter with D0=85

Inverse Filtering

Limitations:
1. Even if the degradation function is known the
undegraded image cannot be recovered exactly
because N(u,v) is the random function which is not
known.
2. If the degradation function has 0 or small value the
ratio
easily dominates the estimate F(u,v,) one
approach to get ride of 0 (or) small value problem to
limits the filter frequency to the value near the origin.

WIENER FILTERING
Inverse filtering has no explicit provision for handling
noise but the wiener filtering it incorporates both
degradation function, statistical characteristics of noise
taken into the restoration process.
e2 = E[(f-]
Objective of the wiener filter is to find the estimate of
uncorrupted image f, such that the mean square error is
minimize the wiener filter is optimum filter
Diagram

Wiener Filtering Contd...


The error between the input signal and the estimated
signal is given by the mean square error.
e(x,y) = f(x,y) (x,y)
E[f(x,y) (x,y)2] =0

According to the principle of orthogonality the expected


value of
f(x,y) (x,y) totally orthogonal with g(x,y) is zero.
E[f(x,y) (x,y)g(x,y)] = 0
(x,y) = g(x,y)*r(x,y)
E[f(x,y)- (g(x,y)*r(x,y))g(x,y)] =0
E[f(x,y)g(x,y)] = E[(r(x,y)*g(x,y))g(x,y)]
=E{g(x,y)}

Wiener Filtering Contd...


fgfg(x,y) =
fg (x,y) = gg(k,l)

fg (x,y) = r(x,y)* gg(x,y)


On taking Fourier transform
Sfg(u,v) = R(u,v)Sgg(u,v)
Sfg and Sgg are the Power Spectral Density
R(u,v) =
W.k.t g(x,y) = f(x,y)*h(x,y)
Cross correlation between f(x,y) and g(x,y) is given as

Wiener Filtering Contd...


fg (x,y) = E [f(x+l1,y+l2)g*(x,y)]
= E [f(x+l1,y+l2)(f(x,y)*h(x,y))]
= E [f(x+l1,y+l2) ]
=
fg (x,y) = ff(l1+k1,l2+k2)
= h*(x,y)*ff(-x,-y)
On taking Fourier transform
Sfg(u,v) = H*(u,v)Sff(u,v)
Without the presence of noise g=h*f
gg (l1,l2) = hh (l1,l2)*ff(l1,l2)
gg (l1,l2) = [h(l1,l2)*h*(-l1,-l2)]*ff(l1,l2)

Wiener Filtering Contd...


Sgg(u,v) = H(u,v)H*(u,v). Sff(u,v)
= |H(u,v)|2 Sff(u,v)
R(u,v)= =
With presence of noise Sgg(u,v) = |H(u,v)|2 Sff(u,v)+ N(u,v)
= |N(u,v)|2
R(u,v)=
=
= R(u,v)G(u,v)
Multiply and divide by H(u,v) in R(u,v) and sub in
= G(u,v)

Wiener Filtering Contd...


= = G(u,v)
Wiener filter also know as minimum mean square filter
or least mean square filter.
Wiener filter does not have the same problem as the
inverse filter unless both H(u,v) and S(u,v) are zero for
the same value of u&v
H(u,v) = degradation function
H*(u,v) = complex conjugate of H(u,v)
|H(u,v)|2 = H*(u,v) H(u,v)
S(u,v) = |N(u,v)|2 = Power spectrum of the noise
Sf(u,v) = |F(u,v)|2 = Power spectrum of an undegraded
image.

Wiener Filtering

Consideration:
1. When a noise is zero
(x,y)=0, S(u,v) =0
=
It reduces to inverse filtering
2. IF H(u,v)=1
=

Wiener Filtering
Signal to Noise ratio
3. Signal to noise ratio is greater than 1
>>1
Then
= G(u,v) --- Here the wiener filter act as a all
pass filters.
ADVANTAGES:
1.The wiener filter does not have zero value problem untill
both H(u,v) and is equal to zero.
2.The result obtained by wiener filter is more closer to the
original image than inverse filter.

Approximation of Wiener Filter


Wiener Filter Formula:

H
(
u
,
v
)
1
F (u, v )
G (u, v )
2
H (u, v ) H (u, v ) S (u, v ) / S f (u, v )
2

Difficult to estimate
Approximated Formula:

H (u, v )
1
G (u, v )
2
H (u, v ) H (u, v ) K

F (u, v )

In Practice, K is chosen manually to obtain the best visual result!

09/02/16

Wiener Filter: Example

Original image

Blurred image
Due to Turbulence
09/02/16

Result of the
full inverse filter

Result of the inverse


filter with D0=70

Result of the
full Wiener filter

Wiener Filter: Example (cont.)

Original image

Blurred image
Due to Turbulence
09/02/16

Result of the inverse


filter with D0=70

Result of the
Wiener filter

Example: Wiener Filter and Motion Blurring


Image
degraded
by motion
blur +
AWGN

Result of the
inverse filter

2=650

2=325

09/02/16

2=130

Result of the
Wiener filter

Constrained Least Squares Filter


Degradation model:

In matrix form,

g ( x, y ) f ( x, y ) h( x, y ) ( x, y )

g Hf

Aims to find the minimum of a criterion function

M 1 N 1

f ( x, y )

x 0 y 0

Subject to the constraint


2

2
g Hf

where

w wT w

Constrained least square filter is given by,

F (u, v )

H * (u, v )
2

G ( u, v )

H (u, v ) P (u, v )

where
P(u,v) = Fourier transform of p(x,y) =
09/02/16

0 1 0
1 4 1

0 1 0

Constrained Least Squares Filter: Example


Constrained least square filter

F (u, v )

H * (u, v )
2

G (u, v )

H (u, v ) P (u, v )

is adaptively adjusted to achieve the best result.

Results from the previous slide obtained


from the constrained least square filter
09/02/16

Constrained Least Squares Filter: Example (cont.)


Image
degraded
by motion
blur +
AWGN

Result of the
Constrained
Least square
filter

2=650

2=325

09/02/16

2=130

Result of the
Wiener filter

Constrained Least Squares Filter:Adjusting


Define

r g Hf

( ) r r r
2

where a = accuracy factor

1.

Specify an initial value of

2.

Compute

3.

Stop if
is satisfied
Otherwise return step 2 after increasing if
1

r a

We want to adjust gamma so that

It can be shown that

r a

or decreasing if
Use the new value of to recompute

r a

F (u, v )
09/02/16

H * (u, v )
2

H (u, v ) P (u, v )

G ( u, v )

Constrained Least Squares Filter:Adjusting (cont.)

H * (u, v )

F (u, v )

G (u, v )

H (u, v ) P (u, v )

R (u, v ) G (u, v ) H (u, v ) F (u, v )


M 1 N 1

1
r
MN

2
r
( x, y )

x 0 y 0

1
m
MN

M 1 N 1

1
2

MN

M 1 N 1

( x, y )
x 0 y 0

( x, y ) m

x 0 y 0

MN 2 m
09/02/16

For computing

For computing

Constrained Least Squares Filter: Example


Original image

Use correct noise


parameters

Blurred image
Due to Turbulence

Use wrong noise


parameters

09/02/16

Correct parameters:
Initial = 10-5
Correction factor = 10-6
a = 0.25
2 = 10-5

Wrong noise parameter


2 = 10-2

Results obtained from constrained least square filters

Geometric Mean filter


This filter represents a family of filters combined into a
single expression

H (u, v )
H (u, v )

F (u, v )
2

S (u, v )
2
H (u, v )
H (u, v )

S
(
u
,
v
)

= 1 the inverse filter


= 0 the Parametric Wiener filter
= 0, = 1 the standard Wiener filter
= 1, < 0.5 More like the inverse filter
= 1, > 0.5 More like the Wiener filter
Another name: the spectrum equalization filter
09/02/16

G (u, v )

Geometric Transformation
These transformations are often called rubber-sheet transformations:
Printing an image on a rubber sheet and then stretch this sheet according
to some predefine set of rules.

A geometric transformation consists of 2 basic operations:


1. A spatial transformation :
Define how pixels are to be rearranged in the spatially
transformed image.
2. Gray level interpolation :
Assign gray level values to pixels in the spatially
transformed image.

09/02/16

Geometric Transformation : Algorithm


( x, y )

( x, y )

Image f to be
restored
1.
2.

Distorted image g

Select coordinate (x,y) in f to be restored


Compute

x r ( x, y )
y s ( x, y )
3. Go to pixel
( x, y )
in a distorted image g
09/02/16

4. get pixel value at


g ( x, y )
By gray level interpolation
5. store that value in pixel f(x,y)
5

Spatial Transformation
To map between pixel coordinate (x,y) of f and pixel coordinate
(x,y) of g

x r ( x, y )

y s( x, y )

For a bilinear transformation mapping between a pair of


Quadrilateral regions

x r ( x, y ) c1 x c2 y c3 xy c4
y s( x, y ) c5 x c6 y c7 xy c8
To obtain r(x,y) and s(x,y), we need
to know 4 pairs of coordinates
and its corresponding
( x, yare
) called tiepoints.
which

09/02/16

( x, y )

( x, y )

( x, y )

Gray Level Interpolation: Nearest Neighbor


Since ( x , y may
) not be at an integer coordinate, we need to
Interpolate the value of

g ( x, y )

Example interpolation methods that can be used:


1. Nearest neighbor selection
2. Bilinear interpolation
3. Bicubic interpolation

09/02/16

Geometric Distortion and Restoration Example


Original image and
tiepoints

Distorted image

Tiepoints of distorted
image

Restored image

Use nearest
neighbor
intepolation

09/02/16

Geometric Distortion and Restoration Example (cont.)


Original image and
tiepoints

Distorted image

Tiepoints of distorted
image

Restored image

Use bilinear
intepolation

09/02/16

Example: Geometric Restoration


Original image

Geometrically distorted
image
Use the same
Spatial Trans.
as in the previous
example

Restored image

Difference between
2 above images
09/02/16