You are on page 1of 85

# IMAGE RESTORATION

Outline
A model of the image degradation / restoration
process
Noise models
Restoration in the presence of noise only
spatial filtering
Periodic noise reduction by frequency domain
filtering
Inverse filtering & Wiener filtering
Constrained Least square filtering
Geometric mean filter
Geometric and spatial transformation

## What is Image Restoration?

Image restoration is to restore a
degraded image back to the original
image
Image enhancement is to manipulate
the image so that it is suitable for a
specific application.

Image Restoration
Image restoration attempts to restore
attempt to reverse it
Similar to image enhancement, but
more objective

## g(x,y)=f(x,y)*h(x,y)+(x,y) Spatial dom

G(u,v)=F(u,v)H(u,v)+N(u,v) Frequency

## A model of the image

Where,
f(x,y) - input image
f^(x,y) - estimated original image

Noise models
Sources of noise
Image acquisition (digitization) - Imaging
sensors can be affected by ambient conditions
Image transmission - Interference can be added
to an image during transmission

## Spatial properties of noise

Statistical behavior of the gray-level values of
pixels
Noise parameters, correlation with the image

## Frequency properties of noise

Fourier spectrum
Ex. white noise (a constant Fourier spectrum)

Noise models
Noise model is decided based on
understanding of
the physics of the sources of noise.
Gaussian: poor illumination
Rayleigh: range image
Gamma(Erlang),
Exponential:
laser
imaging
Impulse: faulty switch during imaging
Uniform: least used

## Noise probability density

functions
Noise cannot be predicted but can be
approximately described in statistical way
using the probability density function (PDF)

Gaussian noise
Mathematical tractability in spatial
and frequency domains
Used frequently in practice
Electronic circuit noise and sensor
noisep ( z ) 1 e ( z ) 2 / 2 2

Note:

Intensit

p( z )dz 1

mean

variance

## 70% in [(), ()]

95% in [(), ()]

Rayleigh noise
2
2
( z a )e ( z a ) / b
p( z ) b
0

for z a
for z a

by
b( 4 )
2
a b / 4 and
4

variance

p( z )

a b z b 1 az
e
(b 1)!

for z 0
for z 0

## The mean and variance of this density are

given by b / a and 2 b2
a

variance

## Gamma noise (PDF)

Exponential noise
for z 0

for z 0

p( z )

ae az

are given by
1 / a and 2

1
a2

## Exponential Noise PDF

Special case of Erlang PDF with b=1

Uniform noise
Less practical, used for random
number generator
1

if a z b
p( z ) b a
0
otherwise
ab
Mean:
2
2
(
b

a
)
2

Variance:
12

Uniform PDF

Impulse (salt-and-pepper)
nosie
Quick transients, such as faulty switching
during imaging

Pa for z a

p ( z ) Pb for z b
0 otherwise

## If either Pa or Pb is zero, it is called unipolar.

Otherwise, it is called bipolar.

Impulse (salt-and-pepper)
nosie PDF

g ( x, y ) f ( x, y ) ( x, y )

Original image

Histogram

Histogram

Periodic noise
Is an image that arises from electrical
and electromechanical interference
during image acquisition
It can be observed by visual
inspection both in the spatial domain
and frequency domain
can be reduced significantly via
frequency domain filtering

Periodic Noise

Periodic noise
looks like dots
In the frequency
domain

Estimation of noise
parameters
Periodic noise
Observe the frequency spectrum

## Random noise with unknown PDFs

Case 1: imaging system is available
Capture images of flat environment

## Case 2: noisy images available

Take a strip from constant area
Draw the histogram and observe it
Measure the mean and variance
09/02/16

Estimation of Noise
Image histogram cannot be
used to estimate noise PDF.

## It is better to use the

histogram of one area of an
image that has constant
intensity to estimate noise
PDF.

09/02/16

Methods
Mean filters

## Arithmetic mean filter

Geometric mean filter
Harmonic mean filter
Contra-harmonic
mean filter

## Order statistics filters

09/02/16

Median filter
Max and min filters
Mid-point filter
alpha-trimmed filters

reduction filter

Mean Filters

g ( x, y ) f ( x, y ) h( x, y ) ( x, y )
To remove this part

## Arithmetic mean filter or moving average filter

f ( x, y ) 1
g ( s, t )

mn ( s ,t )S xy
- is the simplest of the mean filters
- smooths local variations in an image
- noise is reduced as a result of blurring

09/02/16

## Mean Filters Contd...

Geometric mean filter

f ( x, y )

1
mn

g
(
s
,
t
)

( s ,t )S xy

## - achieves smoothing as compared to arithmetic mean filter

- it tends to lose less image detail in the process

09/02/16

Original
image

Image
obtained
using a 3x3
arithmetic
mean filter

09/02/16

Image
corrupted
by AWGN

Image
obtained
using a 3x3
geometric
mean filter

## Harmonic and Contraharmonic Filters

Harmonic mean filter

f ( x, y )

mn

( s ,t )S xy

1
g ( s, t )

f ( x, y )

Q 1
g
(
s
,
t
)

( s ,t )S xy

g ( s, t )

## Works well for salt noise

but fails for pepper noise

## Positive Q is suitable for

eliminating pepper noise.
Negative Q is suitable for
eliminating salt noise.

( s ,t )S xy

09/02/16

## For Q = 0, the filter reduces to an arithmetic mean filter.

For Q = -1, the filter reduces to a harmonic mean filter.

## Contraharmonic Filters: Example

Image
corrupted
by pepper
noise with
prob. = 0.1

Image
corrupted
by salt
noise with
prob. = 0.1

Image
obtained
using a 3x3
contraharmonic
mean filter
With Q = 1.5

Image
obtained
using a 3x3
contraharmonic
mean filter
With Q=-1.5

09/02/16

Original image

Order-Statistic Filters
subimage

Statistic parameters
Mean, Median, Mode,
Min, Max, Etc.

Moving
window

09/02/16

Output image

Order-Statistics Filters
Median filter

f ( x, y ) median g ( s, t )
( s ,t )S xy

Max filter

f ( x, y ) max g ( s, t )
( s ,t )S xy

## Reduce dark noise

(pepper noise)

Min filter

f ( x, y ) min g ( s, t )
( s ,t )S xy

## Reduce bright noise

(salt noise)

Midpoint filter

09/02/16

f ( x, y ) 1 max g ( s, t ) min g ( s, t )
( s ,t )S xy
2 ( s ,t )S xy

## Median Filter : How it works

A median filter is good for removing impulse, isolated noise
Salt noise

Pepper noise
Median

Moving
window

Salt noise
Pepper noise

Filter output
Normally, impulse noise has high magnitude
and is isolated. When we sort pixels in the
moving window, noise pixels are usually
at the ends of the array.

Therefore, its rare that the noise pixel will be a median value.
09/02/16

Sorted
array

## Median Filter : Example

1

Image
corrupted
by salt-andpepper
noise with
pa=pb= 0.1

09/02/16

Image
corrupted
by pepper
noise with
prob. = 0.1

Image
corrupted
by salt
noise with
prob. = 0.1

Image
obtained
using a 3x3
max filter

Image
obtained
using a 3x3
min filter

09/02/16

## Alpha-trimmed Mean Filter

This filter is useful in situations involving multiple types of noise
such as a combination of salt-and-pepper and Gaussian noise.

f ( x, y )

1
mn d

g ( s, t )

( s ,t )S xy

where,

## gr(s,t) represent the remaining mn-d pixels after removing

the d/2 highest and d/2 lowest values of g(s,t).

09/02/16

1
Image
corrupted
uniform
noise

Image 2
obtained
using a 5x5
arithmetic
mean filter

09/02/16

Image
corrupted
salt-andpepper
noise

Image 2
obtained
using a 5x5
geometric
mean filter

1
Image
corrupted
uniform
noise

Image 2
obtained
using a 5x5
median filter

09/02/16

Image
corrupted
salt-andpepper
noise

Image 2
obtained
using a 5x5
alphatrimmed
mean filter
with d = 5

Image
obtained
using a 5x5
arithmetic
mean filter

Image
obtained
using a 5x5
geometric
mean filter

Image
obtained
using a 5x5
median filter

Image
obtained
using a 5x5
alphatrimmed
mean filter
with d = 5

09/02/16

## A model of the image degradation

/restoration process

g(x,y)=f(x,y)*h(x,y)+(x,y)
G(u,v)=F(u,v)H(u,v)+N(u,v)
09/02/16

Linear, position-invariant
Properties of the degradation function H

Linear system
H[af1(x,y)+bf2(x,y)]=aH[f1(x,y)]
+bH[f2(x,y)]

Position(space)-invariant system
H[f(x,y)]=g(x,y) is position invariant if
H[f(x-, y-)]=g(x-, y-)

09/02/16

g ( x, y ) f ( x, y ) h ( x, y ) ( x, y )
or

G ( u, v ) F ( u , v ) H ( u , v ) N ( u , v )
If we know exactly h(x,y), regardless of noise, we can do
deconvolution to get f(x,y) back from g(x,y).
Methods:
1. Estimation by Image Observation
2. Estimation by Experiment
3. Estimation by Modeling
09/02/16

## Estimation by Image Observation

Original image (unknown)

f(x,y)

f(x,y)*h(x,y)

g(x,y)

Observation

Estimated Transfer
function

DFT

Gs (u, v )

09/02/16

g s ( x, y )
Restoration
process by
estimation

G (u, v )
H (u, v ) H s (u, v ) s
Fs (u, v )
This case is used when we
know only g(x,y) and cannot
repeat the experiment!

Subimage

DFT

Fs (u, v )

Reconstructed
Subimage

fs ( x, y )

Estimation by Experiment
Used when we have the same equipment set up
Response image from
the system

## Input impulse image

System
H( )

A ( x , y )

g ( x, y )

DFT

DFT

DFT A ( x, y ) A
09/02/16

G (u, v )
G (u, v )
H (u, v )
A

Estimation by Modeling
Used when we know physical mechanism underlying the image
formation process that can be expressed mathematically.
Original image

Severe turbulence

Example:
Atmospheric
Turbulence model

H (u, v ) e
k = 0.0025
Mild turbulence

Low turbulence

k = 0.001

k = 0.00025

09/02/16

k ( u 2 v 2 )5 / 6

## Estimation by Modeling: Motion Blurring

Assume that camera velocity is

( x0 (t ), y0 (t ))

## The blurred image is obtained by

T

g ( x, y ) f ( x x0 (t ), y y0 (t ))dt
0

## where T = exposure time.

G (u, v )

j 2 ( ux vy )
g
(
x
,
y
)
e
dxdy

09/02/16

f ( x x0 (t ), y y0 (t ))dt e j 2 ( ux vy ) dxdy

f ( x x (t ), y y (t ))e
0

j 2 ( ux vy )

dxdy dt

## Estimation by Modeling: Motion Blurring (cont.)

G (u, v )
0

f ( x x (t ), y y (t ))e
0

j 2 ( ux vy )

dxdy dt

0

0

T

0

## For constant motion

T

H (u, v ) e
09/02/16

( x0 (t ), y0 (t )) ( at , bt )
j 2 ( ua vb )

T
dt
sin( (ua vb))e j ( ua vb )
(ua vb)

## Motion Blurring Example

For constant motion

H (u, v )

T
sin( (ua vb))e j ( ua vb )
(ua vb)

Original image

09/02/16

## Motion blurred image

a = b = 0.1, T = 1

Inverse Filtering

G ( u , v ) F ( u, v ) H ( u , v ) N ( u , v )
after we obtain H(u,v), we can estimate F(u,v) by the inverse filter:

G (u, v )
N (u, v )

F (u, v )
F (u, v )
H (u, v )
H (u, v )
Noise is enhanced
when H(u,v) is small.
To avoid the side effect of enhancing
noise, we can apply this formulation
to freq. component (u,v) with in a
radius D0 from the center of H(u,v).
In practical, the inverse filter is not popularly used.
09/02/16

Inverse Filtering
The simplest approach to restoration is direct inverse
filtering, where we compute an estimate, (u,v), of the
transform of the original image simply by dividing the
transform of the degraded image, G(u,v), by the
g(m,n) = f(m,n)*h(m,n)+(m,n)
g = Hf +
If the noise is zero the error function is denoted as
T() = ||g- ||2
T() = g2+H2-2H

=0

09/02/16

## Inverse Filtering Contd...

To find Mean Square Error
0+2H2-2H =0
2H22H
H >>>>> H = (Inverse Filtering)
G(u,v) = H(u,v)(u,v)
(u,v) =
frequency domain
G(u,v) = F(u,v)H(u,v)+N(u,v)
(1)

09/02/16

## Inverse Filtering Contd...

Divide equation one by H(u,v)
=
.(2)

## We know that (u,v) =

Substitute (u,v) in eqn (2)
(u,v) = F(u,v)+
If noise is zero the estimated image (u,v) is equal to
original image, but noise will not be properly removed
in inverse filtering.

## Inverse Filtering: Example

Original image

Blurred image
Due to Turbulence
09/02/16

Result of applying
the full filter

Result of applying
the filter with D0=70

H (u, v ) e

0.0025( u 2 v 2 )5 / 6

Result of applying
the filter with D0=40

Result of applying
the filter with D0=85

Inverse Filtering

Limitations:
1. Even if the degradation function is known the
undegraded image cannot be recovered exactly
because N(u,v) is the random function which is not
known.
2. If the degradation function has 0 or small value the
ratio
easily dominates the estimate F(u,v,) one
approach to get ride of 0 (or) small value problem to
limits the filter frequency to the value near the origin.

WIENER FILTERING
Inverse filtering has no explicit provision for handling
noise but the wiener filtering it incorporates both
degradation function, statistical characteristics of noise
taken into the restoration process.
e2 = E[(f-]
Objective of the wiener filter is to find the estimate of
uncorrupted image f, such that the mean square error is
minimize the wiener filter is optimum filter
Diagram

## Wiener Filtering Contd...

The error between the input signal and the estimated
signal is given by the mean square error.
e(x,y) = f(x,y) (x,y)
E[f(x,y) (x,y)2] =0

## According to the principle of orthogonality the expected

value of
f(x,y) (x,y) totally orthogonal with g(x,y) is zero.
E[f(x,y) (x,y)g(x,y)] = 0
(x,y) = g(x,y)*r(x,y)
E[f(x,y)- (g(x,y)*r(x,y))g(x,y)] =0
E[f(x,y)g(x,y)] = E[(r(x,y)*g(x,y))g(x,y)]
=E{g(x,y)}

## Wiener Filtering Contd...

fgfg(x,y) =
fg (x,y) = gg(k,l)

## fg (x,y) = r(x,y)* gg(x,y)

On taking Fourier transform
Sfg(u,v) = R(u,v)Sgg(u,v)
Sfg and Sgg are the Power Spectral Density
R(u,v) =
W.k.t g(x,y) = f(x,y)*h(x,y)
Cross correlation between f(x,y) and g(x,y) is given as

## Wiener Filtering Contd...

fg (x,y) = E [f(x+l1,y+l2)g*(x,y)]
= E [f(x+l1,y+l2)(f(x,y)*h(x,y))]
= E [f(x+l1,y+l2) ]
=
fg (x,y) = ff(l1+k1,l2+k2)
= h*(x,y)*ff(-x,-y)
On taking Fourier transform
Sfg(u,v) = H*(u,v)Sff(u,v)
Without the presence of noise g=h*f
gg (l1,l2) = hh (l1,l2)*ff(l1,l2)
gg (l1,l2) = [h(l1,l2)*h*(-l1,-l2)]*ff(l1,l2)

## Wiener Filtering Contd...

Sgg(u,v) = H(u,v)H*(u,v). Sff(u,v)
= |H(u,v)|2 Sff(u,v)
R(u,v)= =
With presence of noise Sgg(u,v) = |H(u,v)|2 Sff(u,v)+ N(u,v)
= |N(u,v)|2
R(u,v)=
=
= R(u,v)G(u,v)
Multiply and divide by H(u,v) in R(u,v) and sub in
= G(u,v)

## Wiener Filtering Contd...

= = G(u,v)
Wiener filter also know as minimum mean square filter
or least mean square filter.
Wiener filter does not have the same problem as the
inverse filter unless both H(u,v) and S(u,v) are zero for
the same value of u&v
H*(u,v) = complex conjugate of H(u,v)
|H(u,v)|2 = H*(u,v) H(u,v)
S(u,v) = |N(u,v)|2 = Power spectrum of the noise
Sf(u,v) = |F(u,v)|2 = Power spectrum of an undegraded
image.

Wiener Filtering

Consideration:
1. When a noise is zero
(x,y)=0, S(u,v) =0
=
It reduces to inverse filtering
2. IF H(u,v)=1
=

Wiener Filtering
Signal to Noise ratio
3. Signal to noise ratio is greater than 1
>>1
Then
= G(u,v) --- Here the wiener filter act as a all
pass filters.
1.The wiener filter does not have zero value problem untill
both H(u,v) and is equal to zero.
2.The result obtained by wiener filter is more closer to the
original image than inverse filter.

## Approximation of Wiener Filter

Wiener Filter Formula:

H
(
u
,
v
)
1
F (u, v )
G (u, v )
2
H (u, v ) H (u, v ) S (u, v ) / S f (u, v )
2

Difficult to estimate
Approximated Formula:

H (u, v )
1
G (u, v )
2
H (u, v ) H (u, v ) K

F (u, v )

09/02/16

## Wiener Filter: Example

Original image

Blurred image
Due to Turbulence
09/02/16

Result of the
full inverse filter

## Result of the inverse

filter with D0=70

Result of the
full Wiener filter

## Wiener Filter: Example (cont.)

Original image

Blurred image
Due to Turbulence
09/02/16

## Result of the inverse

filter with D0=70

Result of the
Wiener filter

Image
by motion
blur +
AWGN

Result of the
inverse filter

2=650

2=325

09/02/16

2=130

Result of the
Wiener filter

## Constrained Least Squares Filter

In matrix form,

g ( x, y ) f ( x, y ) h( x, y ) ( x, y )

g Hf

M 1 N 1

f ( x, y )

x 0 y 0

2

2
g Hf

where

w wT w

## Constrained least square filter is given by,

F (u, v )

H * (u, v )
2

G ( u, v )

H (u, v ) P (u, v )

where
P(u,v) = Fourier transform of p(x,y) =
09/02/16

0 1 0
1 4 1

0 1 0

## Constrained Least Squares Filter: Example

Constrained least square filter

F (u, v )

H * (u, v )
2

G (u, v )

H (u, v ) P (u, v )

## Results from the previous slide obtained

from the constrained least square filter
09/02/16

Image
by motion
blur +
AWGN

Result of the
Constrained
Least square
filter

2=650

2=325

09/02/16

2=130

Result of the
Wiener filter

Define

r g Hf

( ) r r r
2

1.

## Specify an initial value of

2.

Compute

3.

Stop if
is satisfied
Otherwise return step 2 after increasing if
1

r a

## It can be shown that

r a

or decreasing if
Use the new value of to recompute

r a

F (u, v )
09/02/16

H * (u, v )
2

H (u, v ) P (u, v )

G ( u, v )

## Constrained Least Squares Filter:Adjusting (cont.)

H * (u, v )

F (u, v )

G (u, v )

H (u, v ) P (u, v )

M 1 N 1

1
r
MN

2
r
( x, y )

x 0 y 0

1
m
MN

M 1 N 1

1
2

MN

M 1 N 1

( x, y )
x 0 y 0

( x, y ) m

x 0 y 0

MN 2 m
09/02/16

For computing

For computing

Original image

## Use correct noise

parameters

Blurred image
Due to Turbulence

## Use wrong noise

parameters

09/02/16

Correct parameters:
Initial = 10-5
Correction factor = 10-6
a = 0.25
2 = 10-5

2 = 10-2

## Geometric Mean filter

This filter represents a family of filters combined into a
single expression

H (u, v )
H (u, v )

F (u, v )
2

S (u, v )
2
H (u, v )
H (u, v )

S
(
u
,
v
)

## = 1 the inverse filter

= 0 the Parametric Wiener filter
= 0, = 1 the standard Wiener filter
= 1, < 0.5 More like the inverse filter
= 1, > 0.5 More like the Wiener filter
Another name: the spectrum equalization filter
09/02/16

G (u, v )

Geometric Transformation
These transformations are often called rubber-sheet transformations:
Printing an image on a rubber sheet and then stretch this sheet according
to some predefine set of rules.

## A geometric transformation consists of 2 basic operations:

1. A spatial transformation :
Define how pixels are to be rearranged in the spatially
transformed image.
2. Gray level interpolation :
Assign gray level values to pixels in the spatially
transformed image.

09/02/16

## Geometric Transformation : Algorithm

( x, y )

( x, y )

Image f to be
restored
1.
2.

Distorted image g

## Select coordinate (x,y) in f to be restored

Compute

x r ( x, y )
y s ( x, y )
3. Go to pixel
( x, y )
in a distorted image g
09/02/16

## 4. get pixel value at

g ( x, y )
By gray level interpolation
5. store that value in pixel f(x,y)
5

Spatial Transformation
To map between pixel coordinate (x,y) of f and pixel coordinate
(x,y) of g

x r ( x, y )

y s( x, y )

## For a bilinear transformation mapping between a pair of

x r ( x, y ) c1 x c2 y c3 xy c4
y s( x, y ) c5 x c6 y c7 xy c8
To obtain r(x,y) and s(x,y), we need
to know 4 pairs of coordinates
and its corresponding
( x, yare
) called tiepoints.
which

09/02/16

( x, y )

( x, y )

( x, y )

## Gray Level Interpolation: Nearest Neighbor

Since ( x , y may
) not be at an integer coordinate, we need to
Interpolate the value of

g ( x, y )

## Example interpolation methods that can be used:

1. Nearest neighbor selection
2. Bilinear interpolation
3. Bicubic interpolation

09/02/16

## Geometric Distortion and Restoration Example

Original image and
tiepoints

Distorted image

Tiepoints of distorted
image

Restored image

Use nearest
neighbor
intepolation

09/02/16

## Geometric Distortion and Restoration Example (cont.)

Original image and
tiepoints

Distorted image

Tiepoints of distorted
image

Restored image

Use bilinear
intepolation

09/02/16

## Example: Geometric Restoration

Original image

Geometrically distorted
image
Use the same
Spatial Trans.
as in the previous
example

Restored image

Difference between
2 above images
09/02/16