You are on page 1of 123

# Image Filtering

Goals
Low-level image processing operations

## Smoothing and noise reduction

Feature enhancement!

## Example: Remove Noise

(if you know how to model that noise specifically!)

## Input image from DIP by Gonzalez+Woods, 2008

median filter

Outline
Linear filtering
Convolution operations
Smoothing
Low-pass and high-pass filters
Sharpen

## Filtering in the Fourier domain

Rank filters
4

Linear Mappings
L1 is a linear operation if
L[I1 + I2]
= L[I1] + L[I2]
and
L[ Ii]
= L[Ii]

(additive)
(homogeneous)

Equivalently, L is linear if
L[I1 + I2] = L[I1] + L[I2]
(i.e. L preserves linear combinations)
5

Image == Heightfield
I

z = Is intensity

## Filter: Let Intensity Through

Selectively

7
(Airsan Grease Filter, but pretend...)

13

12

16

15

23

10

13

12

16

15

23

10

??

??

??

??

10

## Linear Operations: Weighted Sum

K
I

13

12

16

1/3

1/3

1/3

??

??

15

23

10

Neighborhood: N(x) = x 1
Operation:
Average

??

??

11

13

12

16

1/3

1/3

15

23

10

??

??

12

1/3

13.66

??

13

12

16

15

1/3

1/3

23

10

??

13

1/3

13.66

??

??

13

12

16

15

23

1/3

1/3

10

14

1/3

13.66 14.33

??

??

13

12

16

15

23

10

1/3

1/3

1/3

13.66 14.33

18

??

15

## Linear Operations: Weighted Sum

13

12

16

15

23

10

1/3

1/3

1/3

13.66 14.33

18

16

16

Correlation
Linear operation in 1D:

I ( x)

K (i) I ( x i)

i a

## I is the input image; I is the output of the operation

K is the kernel of the operation; many choices!
N(x) is a neighbourhood of size (2a+1)
17

Correlation
Linear operation in 1D:

I ( x)

K (i) I ( x i)

i a

## Linear operation in 2D:

I ( x, y )

K (i, j) I ( x i, y j)

i a j b

## I is the input image; I is the output of the operation

K is the kernel of the operation; many choices!
N(x,y) is a neighbourhood of size (2a+1) x (2b+1)
18

Correlation

(Template-Matching Preview)

10 11 10 0 0 1
9 10 11 1 0 1
10 9 10 0 2 1
11 10 9 10 9 11
9 10 11 9 99 11
10 9 9 11 10 10
1/9

I
K
1

1
1

1
1

1
1

X X

X 10
X
X
X
X

X X

X
X
X
X
X X X X

## 1/9.(10x1 + 11x1 + 10x1 + 9x1 + 10x1 + 11x1 + 10x1 + 9x1 + 10x1) =

19
1/9.( 90) = 10
Slide credit: Christopher Rasmussen

Correlation

(Template-Matching Preview)

10 11 10 0 0 1
9 10 11 1 0 1
10 9 10 0 2 1

11 10 9 10 9 11
9 10 11 9 99 11
10 9 9 11 10 10

1/9
(again)

1
1

1
1

1
1

X X

X X

X 10
X
X
X
X

X
X
X
X
X X X X

K (i, j ) I ( x i, y j )

I ( x, y )

i a j b

1/9.( 90) = 10

20

I

I

_0

0
_

7
_

6
_

5
_

_0

0
_

_0

## Correlation reveals copy of K, rotated by 180

Matlab:
filter2( [5 6 7], [0 0 0 1 0 0 0 0] )
> ans = 0 0 7 6 5 0 0 0

## Would be nice to get

result that is NOT rotated!
(Ideas?)

Correlation
I ( x, y )

K (i, j) I ( x i, y j)

i a j b

Convolution
I ( x, y)

K (i, j) I ( x i, y j)

i a j b

K (i, j) I ( x i, y j)

i a j b

## So if K(i,j) = K(-i, -j), then Correlation == Convolution

23

Correlation
I ( x, y )

K (i, j) I ( x i, y j)

i a j b

Convolution
I ( x, y)

K (i, j) I ( x i, y j)

i a j b

K (i, j) I ( x i, y j)

i a j b

I K I

Matlab:
conv2([0 0 0 1 0 0 0 0], [5 6 7], 'same' )
> ans = 0 0 5 6 7 0 0 0
24

Linear Filtering
(warm-up)
0 0 0
0 1 0
0 0 0

Original

25
Slide credit: D.A. Forsyth

Linear Filtering
(warm-up)
0 0 0
0 1 0
0 0 0
Original

Filtered
(no change)

26
Slide credit: D.A. Forsyth

## Linear Filtering (Convolution)

I ( x, y)

K (i, j) I ( x i, y j)

i a j b

0 0 0
1 0 0
0 0 0

Original

27
Slide credit: D.A. Forsyth

## Linear Filtering (Convolution)

I ( x, y)

K (i, j) I ( x i, y j)

i a j b

0 0 0
1 0 0
0 0 0
Original

Shifted left
By 1 pixel

28
Slide credit: D.A. Forsyth

Linear Filtering

1 1 1
1 1 1
1 1 1

Original

29
Slide credit: D.A. Forsyth

Linear Filtering

1 1 1
1 1 1
1 1 1
Original

Blur (with a
box filter)

30
Slide credit: D.A. Forsyth

Linear Filtering

0 0 0
0 2 0
0 0 0
Original

1 1 1
1 1 1
1 1 1

## (Note that filter sums to 1)

31
Slide credit: D.A. Forsyth

Linear Filtering

0 0 0
0 2 0
0 0 0
Original

1 1 1
1 1 1
1 1 1

Sharpening filter
- Accentuates differences
with local average
- Also known as Laplacian
32
Slide credit: D.A. Forsyth

33
Slide credit: D.A. Forsyth

Example
K=ones(9,9);

I2=conv2(I,K);

34

Yucky details
When filter window falls off the edge of the
image?
need to extrapolate
methods:

2006 Steve
Marschner 35

## clip filter (black)

wrap around
copy edge
reflect across edge
vary filter near edge

Smoothing Kernels
(Low-pass filters)
Mean filter:

1 1 1
1
1 1 1

9
1 1 1

1 1 1
Weighted
1

1
2
1
smoothing filters: 10

1 1 1

1 2 1
1
2 4 2

16
1 2 1
37

Gaussian Kernel
Weight contributions of neighboring pixels by
nearness
0.003
0.013
0.022
0.013
0.003

0.013
0.059
0.097
0.059
0.013

0.022
0.097
0.159
0.097
0.022

0.013
0.059
0.097
0.059
0.013

0.003
0.013
0.022
0.013
0.003

5 x 5, = 1

## Constant factor at front makes volume sum to 1

38
Slide credit: Christopher Rasmussen

Review of
= sigma = standard deviation = (variance)
Data, d [d1

d2

Mean :

d 3 ... d n ]

n
2
Variance : (d i )
StdDeviation : Variance
Matlab: mean(d), var(d), std(d)

39

## Smoothing with a Gaussian

40
Slide credit: D.A. Forsyth

## Smoothing with a box filter

41
Slide credit: D.A. Forsyth

X =
-0.7500
-0.7500
-0.7500
-0.7500
-0.7500
-0.7500
-0.7500

-0.5000
-0.5000
-0.5000
-0.5000
-0.5000
-0.5000
-0.5000

-0.2500
-0.2500
-0.2500
-0.2500
-0.2500
-0.2500
-0.2500

0
0
0
0
0
0
0

0.2500
0.2500
0.2500
0.2500
0.2500
0.2500
0.2500

0.5000
0.5000
0.5000
0.5000
0.5000
0.5000
0.5000

0.7500
0.7500
0.7500
0.7500
0.7500
0.7500
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

Y =

-0.7500
-0.5000
-0.2500
0
0.2500
0.5000
0.7500

## >> Z = (1/(2*pi*sigma.^2)) * exp(-(X.^2 + Y.^2)/(2*pi*sigma.^2));

>> surf(X,Y,Z)

42

Separable Kernels
Separable filters can be written
K(m,n) = f(m)g(n)
For a rectangular neighbourhood with size
(2M+1)x(2N+1), I (m, n ) f ( g I ( N (m, n )))
I ( m, n )

g ( j ) I ( m, n j )

j N

I ( m, n )

f (i )I (m i, n)

i M

43

2
2

1
x y
g ( x, y)
exp

2
2
2
2

x2

exp 2
2
2
2
g ( x) g ( y )
1

y2
exp 2
2
2
2
1

Separable!
44

## Gaussian Smoothing Kernels

Amount of smoothing depends on and
window size.
Width > 3
7x7; =1.

7x7; =9.

19x19; =1.

19x19; =9.

45

Scale Space
Convolution of a Gaussian with standard
deviation with itself is a Gaussian
standard deviation 2.
Repeated convolution by a Gaussian filter
produces the scale space of an image.

46

1

11x11; =3.

47

## Gaussian Smoothing Kernel Top-5

1. Rotationally symmetric
2. Has a single lobe
- Neighbors influence decreases monotonically

## 3. Still one lobe in frequency domain

- No corruption from high frequencies

4. Simple relationship to
5. Easy to implement efficiently
48

## Smoothing vs. Sharpening

Smoothing Integration, i.e.

z = Is
intensity

Summing
Sand-papering the height fields slope

## Sharpening Differentation, i.e.

Subtracting
Accentuating the height fields slope
49

## Sharpening: 1st and 2nd Derivative

f
f ( x 1) f ( x)
x

(Look ahead)

f
f ( x 1) f ( x 1) 2 f ( x)
x
2

## (Look ahead and look back)

50

f ( x 1) f ( x)
f ( x 1) f ( x 1) 2 f ( x)

51
Figure from DIP by Gonzalez+Woods, 2008

## Differentiation and convolution

Recall, for 2D
function, f(x,y):
f
f x , y f x, y
lim

## This is linear and

shift invariant, so
the result of a
convolution

We could
approximate this as
f f xn1 , y f xn , y

x
x

(which is obviously
a convolution)
-1

1
52
Slide credit: D.A. Forsyth

## Vertical gradients from finite

differences

53
Slide credit: D.A. Forsyth

Differential Filters
Prewitt operator:

1 0 1
1 0 1

1 0 1

Sobel operator:

1 0 1
2 0 2

1 0 1
54

## 1st Derivative: From 1D to 2D

f
x g x
f
f ( x 1) f ( x) in 1D 2D: f
f
x
g y
y

Gradient is anisotropic

mag(f )

g g
2
x

2
y

is isotropic

gy
Orientation of gradient: ( x, y) tan
gx
1

## Imagine gx and gy as vectors that form two sides of a right triangle.

The result of vector-addition would point perpendicular to the
55
gradient, so we want the hypotenuse instead!

## 2nd Derivative: From 1D to 2D

2
2
2 f

f
2
f ( x 1) f ( x 1) 2 f ( x) in 1D 2D: f
2
2
x
x
y

2 f
f ( x 1, y) f ( x 1, y) 2 f ( x, y)
2
x
2 f
f ( x, y 1) f ( x, y 1) 2 f ( x, y)
2
y
1
1
1

1
-8
1

1
1
1

Incorporating diagonals,
isotropic in increments of 45

-1
-1
-1

0
1
0

-1
8
-1

1
-4
1

0
1
0

-1
-1
-1

## Laplacian: use this (negative) version

of kernel for convenience

Image Sharpening
Also known as Enhancement
Increases the high frequency components to
enhance edges
I = I + (k*I), where k is a high-pass filter
kernel and is a scalar in [0,1]

57

## Image Sharpening: Example 1

I ' I I I ( K * I )
2

-1
-1
-1

-1
8
-1

-1
-1
-1

58

## Image Sharpening: Example 1

I ' I I I ( K * I )
2

K*I

59

60

## Image Sharpening: Example 2

I ' I ( I K * I ) " Unsharp Mask"

61

## Image Sharpening: Example 2

I ' 2 I K * I " Unsharp Mask"

62

63

I= Unsharp Mask
(from Example 2)

64

I=I + Laplacian
(from Example 1)

65

## Filters are templates

Filter at some point can be
seen as taking a dotproduct between the
image and some vector
Filtering the image is a set
of dot products

## - filters look like the effects

they are intended to find
- filters find effects they
look like

66
Slide credit: D.A. Forsyth

We need scaled
representations
Find correlations/convolutions at all scales
e.g., when finding hands or faces, we dont know what
size they will be in a particular image
Template size is constant, but image size changes
Efficient search for correspondence
look at coarse scales, then refine with finer scales
much less cost, but may miss best match
Examining all levels of detail
Find edges with different amounts of blur
Find textures with different spatial frequencies (levels
of detail)
67
Slide credit: D.A. Forsyth

Fourier Domain

68

Fourier Domain
The Fourier Domain represents the image
by the sum of weighted frequency
components
Many image processing operations are
simpler in the Fourier Domain
Convolutions are simpler in the Fourier
Domain
69

## Jean Baptiste Joseph Fourier

(1768-1830)
Fourier Series:
Any periodic function
can be rewritten as a
weighted sum of sines
and cosines of different
frequencies.
70

Geometry Review

Matlab:
= atan2(y, x)

(unit-vector on x-axis)

x component C x cos

71

Geometry Review

y component C y sin
72

Geometry Review

C C x cos y sin

## Now, use these polar

coordinates to
represent imaginary
numbers!

73

Geometry Review
C C cos j sin
Apply Euler's formula*

C
Imaginary
component

cos j sin

C C e j
( e j cos j sin )

Real component
*Feynman:
Now lets represent all points using this polar notation. If the
one points
of the most
remarkable,
almost astounding,
in all
mathematics
came
from a periodic
functionformulas
(of space
oroftime,
using74
variable t here), we could model them using a Fourier Series!

Fourier Transform
(Continuous, 1D)

F { f (t )} F ( )

f (t )e

j 2t

dt

## The way to convert back from Fourier domain is the

Inverse Fourier Transform (more later):

j 2t

f (t )

F ( )e

d
75

Fourier Transform
(Continuous, 1D)

F { f (t )} F ( )

f (t )e

j 2t

dt

## In Frequency domain, think of as cycles/meter or cycles/sec (Hz)

(according to units of t)

## The way to convert back from Fourier domain is the

Inverse Fourier Transform (more later):

j 2t

f (t )

F ( )e

d
76

Frequency Spectra
Visualize the Fourier Transform
integral as summing the contribution
=
+
of signals with different frequencies
(and amplitudes) to reconstruct a
spatially or temporally (i.e. usual)
periodic signal
=

## Figures from Alexei Efros

Frequency Spectra

Frequency Spectra

Frequency Spectra

Frequency Spectra

x2 Example

(1D)

## Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x )e

j 2ux / M

u = 0, 1, 2, , M-1

x 0

## Takes a discretized signal of length M (e.g. a row of an image,

indexed by x) and returns a new (complex) function of length M
that is indexed by u.
What is u? It represents frequency of sinusoidal component

(1D)

## Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x )e

j 2ux / M

u = 0, 1, 2, , M-1

x 0

To understand this
apply Euler's formula*

e j cos j sin

F (u )

M 1

## f ( x)cos(2ux / M ) j sin( 2ux / M )

x 0

Can think of this as a change of basis: we move from space in which signal is
represented as coefficients of delta functions to frequency where signal is
88
represented as coefficients of sinusoidal basis functions

Fourier Transform

(again)

F ( )

f (t )e

j 2t

dt

## Model the sampling of f(t) as multiplication

with a sampling function:

~
f (t )

f (t ) (t xt )

~
F ( )

## What is (.) and

why does this give
us a comb-filter?

~
j 2t
f (t )e
dt

(continued)

~
F ( )

~
j 2t
f (t )e
dt

f (t ) (t xt )e

j 2t

dt

xt )e
dt
f (t ) (t (Integral
is only non - zero when t xt )
j 2t

f ( x )e

j 2xt

(continued)

90

~
F ( )

f ( x )e

j 2xt

## (from last page)

1
freq
and we want M equally-spaced samples,
t
so sweep continuous frequency-space
u 1
with samples , where
M t
Discrete Fourier Transform (DFT)

F (u )

M 1

f ( x )e

j 2ux / M

u = 0, 1, 2, , M-1

x 0

## Inverse Discrete Fourier Transform (IDFT)

1
f ( x)
M

M 1

F (u )e
u 0

j 2ux / M

x = 0, 1, 2, , M-1
91

## Discrete Fourier Transform

(2D)

F (u , v)

M 1 N 1

f ( x, y )e

ux vy
j 2 ( )
M N

x 0 y 0

1
f ( x, y )
MN

M 1 N 1

F (u, v)e
u 0 v 0

where:
u = 0, 1, 2, , M-1
v = 0, 1, 2, , N-1
j 2 (

ux vy
)
M N

where:
x = 0, 1, 2, , M-1
y = 0, 1, 2, ,92N-1

0

3
94

Full image

First 1 basis fn

95

Full image

First 1 basis fn

## Fast Fourier Transform

Nave DFT is O(N4).

## The FFT computes the DFT of an NxN image

in O(N2 log2N) time if N=2M
FFTW = Fastest FT in the West (http://www.fftw.org/)
97

Image Spectra
F is a 2D MxN array of complex numbers.
Common to plot |F| and Arg(F) as images
|F| is the magnitude or amplitude spectrum
Arg(F) is the phase spectrum
F is periodic: F(u,v) = F(u+M,v) = F(u,v+N) =
F(u+M,v+N).
F has conjugate symmetry so |F(u,v)|=|F(-u,-v)|.
98

This is the
magnitude
transform
of the
cheetah pic

This is the
phase
transform
of the
cheetah pic

This is the
magnitude
transform
of the zebra
pic

This is the
phase
transform
of the zebra
pic

## Slide by Alexei Efros

Oriented Components

## Curious things about FT on

images
The magnitude spectra of all natural images are quite similar
Heavy on low-frequencies, falling off in high frequences
Will any image be like that, or is it a property of the
world we live in?
Most information in the image is carried in the phase, not the
amplitude
Seems to be a fact of life
Not quite clear why

Reconstruction
with zebra
phase, cheetah
magnitude

Reconstruction
with cheetah
phase, zebra
magnitude

## Filtering in the Fourier Domain

(This is the payoff for all that hard work!)

## The Convolution Theorem states that

f g [ f ][ g ]
where denotes the Fourier Transform.
To convolve image f by kernel g:
Compute DFTs F and G of f and g
Multiply F by G
Compute the inverse DFT of FG.

110

function g = dftfilt(f, H)
%DFTFILT Performs frequency domain filtering.
%
From DIPUM by Gonzalez, Woods, and Eddins 2004
%
G = DFTFILT(F, H) filters F in the frequency domain using the
%
filter transfer function H. The output, G, is the filtered
%
image, which has the same size as F. DFTFILT automatically pads
%
F to be the same size as H. Function PADDEDSIZE can be used to
%
determine an appropriate size for H.
%
%
DFTFILT assumes that F is real and that H is a real, uncentered
%
circularly-symmetric filter function.
%
%
%

## Copyright 2002-2004 R. C. Gonzalez, R. E. Woods, & S. L. Eddins

Digital Image Processing Using MATLAB, Prentice-Hall, 2004
\$Revision: 1.5 \$ \$Date: 2003/08/25 14:28:22 \$

## % Obtain the FFT of the padded input.

F = fft2(f, size(H, 1), size(H, 2));
% Perform filtering.
g = real(ifft2(H.*F));
% Crop to original size.
g = g(1:size(f, 1), 1:size(f, 2));

## (from Matlab Help)

See also: fftshift() and ifftshift()

111

## 2D convolution theorem example

|F(sx,sy)|

f(x,y)
*
g(x,y)

|G(sx,sy)|

h(x,y)

|H(sx,sy)|
Slide by Steve Seitz

0 u 2 v 2 r 2
K (u, v )
1 otherwise

113

## Ideal Low-Pass Filter

0 u 2 v 2 r 2
K (u, v )
1 otherwise

114

Gaussian Low-Pass
Filter
u2 v2
K (u, v ) exp
2
2

115

## Ideal High-Pass Filter

0 u 2 v 2 r 2
K (u, v )
1 otherwise

116

Gaussian High-Pass
Filter
u2 v2
K (u, v ) 1 exp
2
2

117

Band-Stop Filter
K (u, v )

1
1 r

2n
1
/
2
2
2
2

u v r0

118

Band-Pass Filter
K (u, v ) 1

1
1 r

2n
1
/
2

u 2 v 2 r02

119

## (a) Lowpass filter

From Young, Gerbrands and van
Vliet; CVOnline.

## (b) Bandpass filter

filter

(c) Highpass
120

Orientation Filtering

## Removing Unwanted Periodic Component

Rank Filters
Simple non-linear filters based on the
ordering of the grey levels in the
neighbourhood.

123

Max Filter
I(m,n) = max{I(i,j) : (i,j)N(m,n)}

124

Min Filter
I(m,n) = min{I(i,j) : (i,j)N(m,n)}

125

Median Filter
I(m,n) = median{I(i,j) : (i,j)N(m,n)}

126

Range Filter
I(m,n) = max{I(i,j) : (i,j)N(m,n)}
- min{I(i,j) : (i,j)N(m,n)}

127

Summary
Linear filtering
Convolution in the image domain
Separable filters
Smoothing filters and Gaussian filters
Differential filters

Rank filters
128