You are on page 1of 123

Image Filtering

Goals
Low-level image processing operations

Smoothing and noise reduction


Feature enhancement!

Example: Remove Noise


(if you know how to model that noise specifically!)

Input image from DIP by Gonzalez+Woods, 2008

median filter

Outline
Linear filtering
Convolution operations
Smoothing
Low-pass and high-pass filters
Sharpen

Filtering in the Fourier domain


Rank filters
4

Linear Mappings
L1 is a linear operation if
L[I1 + I2]
= L[I1] + L[I2]
and
L[ Ii]
= L[Ii]

(additive)
(homogeneous)

Equivalently, L is linear if
L[I1 + I2] = L[I1] + L[I2]
(i.e. L preserves linear combinations)
5

Image == Heightfield
I

z = Is intensity

Filter: Let Intensity Through


Selectively

7
(Airsan Grease Filter, but pretend...)

13

12

16

15

23

10

Linear Operations: Weighted Sum

13

12

16

15

23

10

??

??

??

??

10

Linear Operations: Weighted Sum

K
I

13

12

16

1/3

1/3

1/3

??

??

15

23

10

Neighborhood: N(x) = x 1
Operation:
Average

??

??

11

Linear Operations: Weighted Sum

13

12

16

1/3

1/3

15

23

10

??

??

12

1/3

13.66

??

Linear Operations: Weighted Sum

13

12

16

15

1/3

1/3

23

10

??

13

1/3

13.66

??

??

Linear Operations: Weighted Sum

13

12

16

15

23

1/3

1/3

10

14

1/3

13.66 14.33

??

??

Linear Operations: Weighted Sum

13

12

16

15

23

10

1/3

1/3

1/3

13.66 14.33

18

??

15

Linear Operations: Weighted Sum

13

12

16

15

23

10

1/3

1/3

1/3

13.66 14.33

18

16

16

Correlation
Linear operation in 1D:

I ( x)

K (i) I ( x i)

i a

I is the input image; I is the output of the operation


K is the kernel of the operation; many choices!
N(x) is a neighbourhood of size (2a+1)
17

Correlation
Linear operation in 1D:

I ( x)

K (i) I ( x i)

i a

Linear operation in 2D:

I ( x, y )

K (i, j) I ( x i, y j)

i a j b

I is the input image; I is the output of the operation


K is the kernel of the operation; many choices!
N(x,y) is a neighbourhood of size (2a+1) x (2b+1)
18

Correlation

(Template-Matching Preview)

10 11 10 0 0 1
9 10 11 1 0 1
10 9 10 0 2 1
11 10 9 10 9 11
9 10 11 9 99 11
10 9 9 11 10 10
1/9

I
K
1

1
1

1
1

1
1

X X

X 10
X
X
X
X

X X

X
X
X
X
X X X X

1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) =


19
1/9.( 90) = 10
Slide credit: Christopher Rasmussen

Correlation

(Template-Matching Preview)

10 11 10 0 0 1
9 10 11 1 0 1
10 9 10 0 2 1

11 10 9 10 9 11
9 10 11 9 99 11
10 9 9 11 10 10

1/9
(again)

1
1

1
1

1
1

X X

X X

X 10
X
X
X
X

X
X
X
X
X X X X

K (i, j ) I ( x i, y j )

1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) =


I ( x, y )

i a j b

1/9.( 90) = 10

20

Slide credit: Christopher Rasmussen

Correlation with a Unit-Impulse Image


I

Correlation with a Unit-Impulse Image


I

_0

0
_

7
_

6
_

5
_

_0

0
_

_0

Correlation reveals copy of K, rotated by 180


Matlab:
filter2( [5 6 7], [0 0 0 1 0 0 0 0] )
> ans = 0 0 7 6 5 0 0 0

Would be nice to get


result that is NOT rotated!
(Ideas?)

Correlation
I ( x, y )

K (i, j) I ( x i, y j)

i a j b

Convolution
I ( x, y)

K (i, j) I ( x i, y j)

i a j b

K (i, j) I ( x i, y j)

i a j b

So if K(i,j) = K(-i, -j), then Correlation == Convolution

23

Correlation
I ( x, y )

K (i, j) I ( x i, y j)

i a j b

Convolution
I ( x, y)

K (i, j) I ( x i, y j)

i a j b

K (i, j) I ( x i, y j)

i a j b

I K I

Matlab:
conv2([0 0 0 1 0 0 0 0], [5 6 7], 'same' )
> ans = 0 0 5 6 7 0 0 0
24

Linear Filtering
(warm-up)
0 0 0
0 1 0
0 0 0

Original

25
Slide credit: D.A. Forsyth

Linear Filtering
(warm-up)
0 0 0
0 1 0
0 0 0
Original

Filtered
(no change)

26
Slide credit: D.A. Forsyth

Linear Filtering (Convolution)


I ( x, y)

K (i, j) I ( x i, y j)

i a j b

0 0 0
1 0 0
0 0 0

Original

27
Slide credit: D.A. Forsyth

Linear Filtering (Convolution)


I ( x, y)

K (i, j) I ( x i, y j)

i a j b

0 0 0
1 0 0
0 0 0
Original

Shifted left
By 1 pixel

28
Slide credit: D.A. Forsyth

Linear Filtering

1 1 1
1 1 1
1 1 1

Original

29
Slide credit: D.A. Forsyth

Linear Filtering

1 1 1
1 1 1
1 1 1
Original

Blur (with a
box filter)

30
Slide credit: D.A. Forsyth

Linear Filtering

0 0 0
0 2 0
0 0 0
Original

1 1 1
1 1 1
1 1 1

(Note that filter sums to 1)

31
Slide credit: D.A. Forsyth

Linear Filtering

0 0 0
0 2 0
0 0 0
Original

1 1 1
1 1 1
1 1 1

Sharpening filter
- Accentuates differences
with local average
- Also known as Laplacian
32
Slide credit: D.A. Forsyth

33
Slide credit: D.A. Forsyth

Example
K=ones(9,9);

I2=conv2(I,K);

34

Yucky details
When filter window falls off the edge of the
image?
need to extrapolate
methods:

2006 Steve
Marschner 35

clip filter (black)


wrap around
copy edge
reflect across edge
vary filter near edge

Smoothing Kernels
(Low-pass filters)
Mean filter:

1 1 1
1
1 1 1

9
1 1 1

1 1 1
Weighted
1

1
2
1
smoothing filters: 10

1 1 1

1 2 1
1
2 4 2

16
1 2 1
37

Gaussian Kernel
Weight contributions of neighboring pixels by
nearness
0.003
0.013
0.022
0.013
0.003

0.013
0.059
0.097
0.059
0.013

0.022
0.097
0.159
0.097
0.022

0.013
0.059
0.097
0.059
0.013

0.003
0.013
0.022
0.013
0.003

5 x 5, = 1

Constant factor at front makes volume sum to 1


38
Slide credit: Christopher Rasmussen

Review of
= sigma = standard deviation = (variance)
Data, d [d1

d2

Mean :

d 3 ... d n ]

n
2
Variance : (d i )
StdDeviation : Variance
Matlab: mean(d), var(d), std(d)

Figure by Petter Strandmark

39

Smoothing with a Gaussian

40
Slide credit: D.A. Forsyth

Smoothing with a box filter

41
Slide credit: D.A. Forsyth

>> [X,Y] = meshgrid(-0.75:0.25:0.75, -0.75:0.25:0.75)

X =
-0.7500
-0.7500
-0.7500
-0.7500
-0.7500
-0.7500
-0.7500

-0.5000
-0.5000
-0.5000
-0.5000
-0.5000
-0.5000
-0.5000

-0.2500
-0.2500
-0.2500
-0.2500
-0.2500
-0.2500
-0.2500

0
0
0
0
0
0
0

0.2500
0.2500
0.2500
0.2500
0.2500
0.2500
0.2500

0.5000
0.5000
0.5000
0.5000
0.5000
0.5000
0.5000

0.7500
0.7500
0.7500
0.7500
0.7500
0.7500
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

Y =

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

>> Z = (1/(2*pi*sigma.^2)) * exp(-(X.^2 + Y.^2)/(2*pi*sigma.^2));


>> surf(X,Y,Z)

42

Separable Kernels
Separable filters can be written
K(m,n) = f(m)g(n)
For a rectangular neighbourhood with size
(2M+1)x(2N+1), I (m, n ) f ( g I ( N (m, n )))
I ( m, n )

g ( j ) I ( m, n j )

j N

I ( m, n )

f (i )I (m i, n)

i M

43

Gaussian Smoothing Kernels

2
2

1
x y
g ( x, y)
exp

2
2
2
2

x2

exp 2
2
2
2
g ( x) g ( y )
1

y2
exp 2
2
2
2
1

Separable!
44

Gaussian Smoothing Kernels


Amount of smoothing depends on and
window size.
Width > 3
7x7; =1.

7x7; =9.

19x19; =1.

19x19; =9.

45

Scale Space
Convolution of a Gaussian with standard
deviation with itself is a Gaussian
standard deviation 2.
Repeated convolution by a Gaussian filter
produces the scale space of an image.

46

Scale Space Example


1

11x11; =3.

47

Gaussian Smoothing Kernel Top-5


1. Rotationally symmetric
2. Has a single lobe
- Neighbors influence decreases monotonically

3. Still one lobe in frequency domain


- No corruption from high frequencies

4. Simple relationship to
5. Easy to implement efficiently
48

Smoothing vs. Sharpening


Smoothing Integration, i.e.

z = Is
intensity

Summing
Sand-papering the height fields slope

Sharpening Differentation, i.e.


Subtracting
Accentuating the height fields slope
49

Sharpening: 1st and 2nd Derivative


f
f ( x 1) f ( x)
x

(Look ahead)

f
f ( x 1) f ( x 1) 2 f ( x)
x
2

(Look ahead and look back)


50

f ( x 1) f ( x)
f ( x 1) f ( x 1) 2 f ( x)

51
Figure from DIP by Gonzalez+Woods, 2008

Differentiation and convolution


Recall, for 2D
function, f(x,y):
f
f x , y f x, y
lim

This is linear and


shift invariant, so
the result of a
convolution

We could
approximate this as
f f xn1 , y f xn , y

x
x

(which is obviously
a convolution)
-1

1
52
Slide credit: D.A. Forsyth

Vertical gradients from finite


differences

53
Slide credit: D.A. Forsyth

Differential Filters
Prewitt operator:

1 0 1
1 0 1

1 0 1

Sobel operator:

1 0 1
2 0 2

1 0 1
54

1st Derivative: From 1D to 2D


f
x g x
f
f ( x 1) f ( x) in 1D 2D: f
f
x
g y
y

Gradient is anisotropic

mag(f )

g g
2
x

2
y

is isotropic

gy
Orientation of gradient: ( x, y) tan
gx
1

Imagine gx and gy as vectors that form two sides of a right triangle.


The result of vector-addition would point perpendicular to the
55
gradient, so we want the hypotenuse instead!

2nd Derivative: From 1D to 2D


2
2
2 f

f
2
f ( x 1) f ( x 1) 2 f ( x) in 1D 2D: f
2
2
x
x
y

2 f
f ( x 1, y) f ( x 1, y) 2 f ( x, y)
2
x
2 f
f ( x, y 1) f ( x, y 1) 2 f ( x, y)
2
y
1
1
1

1
-8
1

1
1
1

Incorporating diagonals,
isotropic in increments of 45

-1
-1
-1

0
1
0

-1
8
-1

1
-4
1

0
1
0

-1
-1
-1

Laplacian: use this (negative) version


of kernel for convenience

Image Sharpening
Also known as Enhancement
Increases the high frequency components to
enhance edges
I = I + (k*I), where k is a high-pass filter
kernel and is a scalar in [0,1]

57

Image Sharpening: Example 1


I ' I I I ( K * I )
2

-1
-1
-1

-1
8
-1

-1
-1
-1

Figure from NASA, obtained on DIP

58

Image Sharpening: Example 1


I ' I I I ( K * I )
2

Intensity range = [0, 254]

Intensity range = [-251, 283]

K*I

59

Image Sharpening: Example 1

Intensity range = [-225, 473]

60

Image Sharpening: Example 2


I ' I ( I K * I ) " Unsharp Mask"

Figure from NASA, obtained on DIP

61

Image Sharpening: Example 2


I ' 2 I K * I " Unsharp Mask"

Intensity range = [-45, 306]

62

63

I= Unsharp Mask
(from Example 2)

64

I=I + Laplacian
(from Example 1)

65

Filters are templates


Filter at some point can be
seen as taking a dotproduct between the
image and some vector
Filtering the image is a set
of dot products

- filters look like the effects


they are intended to find
- filters find effects they
look like

66
Slide credit: D.A. Forsyth

We need scaled
representations
Find correlations/convolutions at all scales
e.g., when finding hands or faces, we dont know what
size they will be in a particular image
Template size is constant, but image size changes
Efficient search for correspondence
look at coarse scales, then refine with finer scales
much less cost, but may miss best match
Examining all levels of detail
Find edges with different amounts of blur
Find textures with different spatial frequencies (levels
of detail)
67
Slide credit: D.A. Forsyth

Fourier Domain

68

Fourier Domain
The Fourier Domain represents the image
by the sum of weighted frequency
components
Many image processing operations are
simpler in the Fourier Domain
Convolutions are simpler in the Fourier
Domain
69

Jean Baptiste Joseph Fourier


(1768-1830)
Fourier Series:
Any periodic function
can be rewritten as a
weighted sum of sines
and cosines of different
frequencies.
70

Geometry Review

Matlab:
= atan2(y, x)

(unit-vector on x-axis)

x component C x cos

71

Geometry Review

y component C y sin
72

Geometry Review

C C x cos y sin

Now, use these polar


coordinates to
represent imaginary
numbers!

73

Geometry Review
C C cos j sin
Apply Euler's formula*

C
Imaginary
component

cos j sin

C C e j
( e j cos j sin )

Real component
*Feynman:
Now lets represent all points using this polar notation. If the
one points
of the most
remarkable,
almost astounding,
in all
mathematics
came
from a periodic
functionformulas
(of space
oroftime,
using74
variable t here), we could model them using a Fourier Series!

Fourier Transform
(Continuous, 1D)

F { f (t )} F ( )

f (t )e

j 2t

dt

The way to convert back from Fourier domain is the


Inverse Fourier Transform (more later):

j 2t

f (t )

F ( )e

d
75

Fourier Transform
(Continuous, 1D)

F { f (t )} F ( )

f (t )e

j 2t

dt

In Frequency domain, think of as cycles/meter or cycles/sec (Hz)


(according to units of t)

The way to convert back from Fourier domain is the


Inverse Fourier Transform (more later):

j 2t

f (t )

F ( )e

d
76

Frequency Spectra
Visualize the Fourier Transform
integral as summing the contribution
=
+
of signals with different frequencies
(and amplitudes) to reconstruct a
spatially or temporally (i.e. usual)
periodic signal
=

Figures from Alexei Efros

Frequency Spectra

Frequency Spectra

Frequency Spectra

Frequency Spectra

x2 Example

Discrete Fourier Transform


(1D)

Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x )e

j 2ux / M

u = 0, 1, 2, , M-1

x 0

Takes a discretized signal of length M (e.g. a row of an image,


indexed by x) and returns a new (complex) function of length M
that is indexed by u.
What is u? It represents frequency of sinusoidal component

Discrete Fourier Transform


(1D)

Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x )e

j 2ux / M

u = 0, 1, 2, , M-1

x 0

To understand this
apply Euler's formula*

e j cos j sin

Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x)cos(2ux / M ) j sin( 2ux / M )


x 0

Can think of this as a change of basis: we move from space in which signal is
represented as coefficients of delta functions to frequency where signal is
88
represented as coefficients of sinusoidal basis functions

Fourier Transform

(Derivation: Continuous 1D Discrete)


(again)

F ( )

f (t )e

j 2t

dt

Model the sampling of f(t) as multiplication


with a sampling function:

~
f (t )

f (t ) (t xt )

~
F ( )

What is (.) and


why does this give
us a comb-filter?

~
j 2t
f (t )e
dt

(continued)

~
F ( )

~
j 2t
f (t )e
dt

f (t ) (t xt )e

j 2t

dt

xt )e
dt
f (t ) (t (Integral
is only non - zero when t xt )
j 2t

f ( x )e

j 2xt

(continued)

90

~
F ( )

f ( x )e

j 2xt

(from last page)

1
freq
and we want M equally-spaced samples,
t
so sweep continuous frequency-space
u 1
with samples , where
M t
Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x )e

j 2ux / M

u = 0, 1, 2, , M-1

x 0

Inverse Discrete Fourier Transform (IDFT)

1
f ( x)
M

M 1

F (u )e
u 0

j 2ux / M

x = 0, 1, 2, , M-1
91

Discrete Fourier Transform


(2D)

F (u , v)

M 1 N 1

f ( x, y )e

ux vy
j 2 ( )
M N

x 0 y 0

1
f ( x, y )
MN

M 1 N 1

F (u, v)e
u 0 v 0

where:
u = 0, 1, 2, , M-1
v = 0, 1, 2, , N-1
j 2 (

ux vy
)
M N

where:
x = 0, 1, 2, , M-1
y = 0, 1, 2, ,92N-1

Fourier Basis Functions


0

3
94

2D Fourier Domain Example

Full image

First 1 basis fn

95

2D Fourier Domain Example

Full image

First 9 basis fns

First 1 basis fn

First 4 basis fns

First 16 basis fns

First 400 basis fns96

Fast Fourier Transform


Nave DFT is O(N4).

The FFT computes the DFT of an NxN image


in O(N2 log2N) time if N=2M
FFTW = Fastest FT in the West (http://www.fftw.org/)
97

Image Spectra
F is a 2D MxN array of complex numbers.
Common to plot |F| and Arg(F) as images
|F| is the magnitude or amplitude spectrum
Arg(F) is the phase spectrum
F is periodic: F(u,v) = F(u+M,v) = F(u,v+N) =
F(u+M,v+N).
F has conjugate symmetry so |F(u,v)|=|F(-u,-v)|.
98

Slide by Alexei Efros

This is the
magnitude
transform
of the
cheetah pic

Slide by Alexei Efros

This is the
phase
transform
of the
cheetah pic

Slide by Alexei Efros

Slide by Alexei Efros

This is the
magnitude
transform
of the zebra
pic

Slide by Alexei Efros

This is the
phase
transform
of the zebra
pic

Slide by Alexei Efros

Oriented Components

Curious things about FT on


images
The magnitude spectra of all natural images are quite similar
Heavy on low-frequencies, falling off in high frequences
Will any image be like that, or is it a property of the
world we live in?
Most information in the image is carried in the phase, not the
amplitude
Seems to be a fact of life
Not quite clear why

Slide by Alexei Efros

Reconstruction
with zebra
phase, cheetah
magnitude

Slide by Alexei Efros

Reconstruction
with cheetah
phase, zebra
magnitude

Slide by Alexei Efros

Filtering in the Fourier Domain


(This is the payoff for all that hard work!)

The Convolution Theorem states that


f g [ f ][ g ]
where denotes the Fourier Transform.
To convolve image f by kernel g:
Compute DFTs F and G of f and g
Multiply F by G
Compute the inverse DFT of FG.

110

function g = dftfilt(f, H)
%DFTFILT Performs frequency domain filtering.
%
From DIPUM by Gonzalez, Woods, and Eddins 2004
%
G = DFTFILT(F, H) filters F in the frequency domain using the
%
filter transfer function H. The output, G, is the filtered
%
image, which has the same size as F. DFTFILT automatically pads
%
F to be the same size as H. Function PADDEDSIZE can be used to
%
determine an appropriate size for H.
%
%
DFTFILT assumes that F is real and that H is a real, uncentered
%
circularly-symmetric filter function.
%
%
%

Copyright 2002-2004 R. C. Gonzalez, R. E. Woods, & S. L. Eddins


Digital Image Processing Using MATLAB, Prentice-Hall, 2004
$Revision: 1.5 $ $Date: 2003/08/25 14:28:22 $

% Obtain the FFT of the padded input.


F = fft2(f, size(H, 1), size(H, 2));
% Perform filtering.
g = real(ifft2(H.*F));
% Crop to original size.
g = g(1:size(f, 1), 1:size(f, 2));

(from Matlab Help)


See also: fftshift() and ifftshift()

111

2D convolution theorem example


|F(sx,sy)|

f(x,y)
*
g(x,y)

|G(sx,sy)|

h(x,y)

|H(sx,sy)|
Slide by Steve Seitz

Ideal Low-Pass Filter


0 u 2 v 2 r 2
K (u, v )
1 otherwise

113

Ideal Low-Pass Filter


0 u 2 v 2 r 2
K (u, v )
1 otherwise

114

Gaussian Low-Pass
Filter
u2 v2
K (u, v ) exp
2
2

115

Ideal High-Pass Filter


0 u 2 v 2 r 2
K (u, v )
1 otherwise

116

Gaussian High-Pass
Filter
u2 v2
K (u, v ) 1 exp
2
2

117

Band-Stop Filter
K (u, v )

1
1 r

2n
1
/
2
2
2
2

u v r0

118

Band-Pass Filter
K (u, v ) 1

1
1 r

2n
1
/
2

u 2 v 2 r02

119

X-pass filter example

(a) Lowpass filter


From Young, Gerbrands and van
Vliet; CVOnline.

(b) Bandpass filter


filter

(c) Highpass
120

Orientation Filtering

Removing Unwanted Periodic Component

Rank Filters
Simple non-linear filters based on the
ordering of the grey levels in the
neighbourhood.

123

Max Filter
I(m,n) = max{I(i,j) : (i,j)N(m,n)}

124

Min Filter
I(m,n) = min{I(i,j) : (i,j)N(m,n)}

125

Median Filter
I(m,n) = median{I(i,j) : (i,j)N(m,n)}

126

Range Filter
I(m,n) = max{I(i,j) : (i,j)N(m,n)}
- min{I(i,j) : (i,j)N(m,n)}

127

Summary
Linear filtering
Convolution in the image domain
Separable filters
Smoothing filters and Gaussian filters
Differential filters

Convolution in the Fourier domain


Rank filters
128