07/03/2021

Computer Vision Computer Vision

Chapter 3: Image Processing Chapter 3. Image Processing

1 2

Content Arithmetical/Logical Operations

• Rappel: digital image representation • AND operation
• Point Processing • OR operation
• Convolution and Linear filtering • Image subtraction
• More neighborhood operators • Image addition
‒ Median/max/min filters
‒ Arithmetical/Logical operations
‒ Binary image and morphological operations
• Image transforms

3 4

3 4

1
 07/03/2021

AND operation OR operation

Original And mask Output image Original OR mask Output image

5 6

5 6

Image Addition Average Images

• If f and g are two images, the • g(x,y) is the addition of f(x,y) and noise 𝜂(𝑥, 𝑦)
pixelwise addition R is defined as:
‒ R(x,y) = Min( f(x,y)+g(x,y) ; 255 )

• If we have several images {g(x,y)}, we can

• Image addition is used to compute the average one
‒ lower the noise in a serie of images
‒ increase the luminance by adding the
image to itself

Source : Eric Favier. L'analyse et le traitement des images. ENISE.

7 8

7 8

2
 07/03/2021

Average Images Image subtraction

• The pixelwise substraction
of two images f and g is:
S(x,y) = Max( f(x,y)-g(x,y) ; 0 )

• Image substraction is used

to
‒ detect defaults, detect
difference between images
‒ detect motion in images

Source : Eric Favier. L'analyse et le traitement des images. ENISE.

9 10

9 10

Image subtraction Image subtraction

After detection, we still have some noise, that

we can clean to keep only the object of interest

11 12

11 12

3
 07/03/2021

Image multiplication Operations on images

• The multiplication S of an image f by a ratio
(factor) is defined as:
‒ S(x,y) = Max( f(x,y)*ratio ; 255)
• Image multiplication can be used to increase the
contrast or the luminosity

Source : Eric Favier. L'analyse et le traitement des images. ENISE. Source : www.nte.montaigne.u-bordeaux.fr/SuppCours/5314/Dai/TraitImage01-02.ppt
13 14

13 14

Exercise Binary images

• Given two images as bellow

1. Transform images to negative ones

2. Process to have an image which has only the
“ball”

15

15 17

4
 07/03/2021

Binary images Thresholding

• Given a grayscale image or an intermediate matrix →
• Two pixel values: foreground (object, 1) and threshold to create a binary output.
background (0)
Example: edge detection
• Be used
‒ To mark region(s) of interest
‒ As results of thresholding method

Gradient magnitude fg_pix = find(gradient_mag > t);

Looking for pixels where gradient is strong.

18 19

Thresholding
• Given a grayscale image or an intermediate matrix →
threshold to create a binary output.
Example: background subtraction
Thresholding
• Given a grayscale image or an intermediate matrix →
threshold to create a binary output.
Example: intensity-based detection

- =

Looking for pixels that differ

significantly from the “empty” fg_pix = find(im < 65);
background.
Looking for dark pixels
fg_pix = find(diff > t);

20 21

5
 07/03/2021

Thresholding Issues
• Given a grayscale image or an intermediate matrix →
threshold to create a binary output. • What to do with “noisy” binary
outputs?
Example: color-based detection ‒ Holes
‒ Extra small fragments

• How to demarcate multiple regions

of interest?
‒ Count objects
‒ Compute further features per object
fg_pix = find(hue > t1 & hue < t2);

Looking for pixels within a certain hue range.

Slide credit: Kristen Grauman Slide credit: Kristen Grauman

22 23

Morphological operators Dilation

• Change the shape of the foreground regions via • Expands connected components
intersection/union operations between a scanning • Grow features
structuring element and binary image. • Fill holes
• Useful to clean up result from thresholding

• Main components
‒ Structuring element
‒ Operators:
• Basic operators: Dilation, Erosion
• Others: Opening, Closing, … Before dilation After dilation
Slide credit: Kristen Grauman

24 25

6
 07/03/2021

Erosion Structuring elements

• Erode connected components • Masks of varying shapes and sizes used to
• Shrink features perform morphology, for example:
• Remove bridges, branches, noise

• Scan mask (structuring element) over the

object (foreground) borders (inside and
outside) and transform the binary image
Before erosion After erosion

Slide credit: Kristen Grauman

26 27

Dilation Dilation
• Moving S on each pixel of A • As max filter
‒ check if the intersection (pixels belonging to object) is not empty
• Can be applied both on
‒ If yes, the center of B belongs to the result image
• If a pixel of S is onto object pixels (A), then the central pixel belongs to ‒ binary images
object ‒ or grayscale images
‒ Otherwise (i.e. all pixels of are background), set to background (no change)
𝑨 ⊕S

Mask /
Structuring element
Structuring element
S

28 29

7
 07/03/2021

Erosion Erosion
• We put the element S on each pixel x of A
• As min filter
‒ like convolution

• If all pixels of S are onto object pixels (A), then the central pixel belongs • Can be applied both on
to object
‒ Otherwise (i.e. a mask pixel is background), set to background
‒ binary images
‒ or grayscale images
𝑨 ⊝S

Structuring Mask /
element Structuring element

30 31

2D example Opening
• Erode, then dilate
• Remove small objects, keep original shape

Before opening After opening

Shapiro & Stockman

32 33

8
 07/03/2021

Closing Opening vs Closing

• Dilate, then erode
• Fill holes, but keep original shape Structuring element

Opening

Closing
Before closing After closing
demo: http://bigwww.epfl.ch/demo/jmorpho/start.php

34 35

Connected component labeling Connected component labeling

• We loop over all the image to give a unique number (label) for each region First loop over the image
• All pixels from the same region must have the same number (label) • For each pixel in a region, we set
• Objectifs: ‒ or the smallest label from its top or left neighbors
‒ Counting objects ‒ or a new label
‒ Separating objets
‒ Creating a mask for each object
‒ …

36 37

9
 07/03/2021

Connected component labeling Connected component labeling

First loop over the image First loop over the image
• For each pixel in a region, we set • For each pixel in a region, we set
‒ or the smallest label from its top or left neighbors ‒ or the smallest label from its top or left neighbors
‒ or a new label ‒ or a new label

38 39

Connected component labeling Connected component labeling

First loop over the image Second loop over the image
• For each pixel in a region, we set • For each pixel in a region, we set
‒ or the smallest label from its top or left neighbors ‒ the smallest from its own label and the labels
from its down and right neighbors
‒ or a new label

40 41

10
 07/03/2021

Connected component labeling Connected component labeling

Second loop over the image Second loop over the image
• For each pixel in a region, we set • For each pixel in a region, we set
‒ the smallest from its own label and the labels ‒ the smallest from its own label and the labels
from its down and right neighbors from its down and right neighbors

42 43

Connected component labeling CC labeling: how many neighbors?

• Two loops are enough? • Advice: Use different connexities for edges and regions
‒ example: spiral region ! ‒ 4-Connexity for regions
‒ 8-Connexity for edges
• Solutions
‒ We continue, go and back two ways, until
no new change in labels

‒ It is possible to do only one loop: manage

a table of equivalences when 2 different
labels are neighbors

44 45

11
 07/03/2021

CC labeling: how many neighbors? Content

• Regions labeling • Rappel: digital image representation
‒ We use 4-connexity • Point Processing
‒ Each loop, we compare 2 neighbors • Convolution and Linear filtering
• More neighborhood operators
• Image transforms
• Edge labeling ‒ Frequency domain
• Frequencies in images
‒ 8-connexity
• Fourrier transform
‒ Each loop, we compare 4 neighbors
• Frequential Processing (frequential filters)
‒ PCA (additional reading)

47

46 47

Frequencies in images Frequencies in images

• What are the (low/high) frequencies in an image?
‒ Frequency = intensity change
Intensity profil
‒ Slow changes (homogeneous /blur regions): low frequency
‒ fast/abrupt changes (egde, contour, noise): high frequency

High frequency Most of

energy
concentrated
in low
Low frequency
frequencies

48 49

48 49

12
 07/03/2021

Low frequencies High frequencies

50 51

50 51

Image spectral analysis Frequencies in a signal

• An image is a visual signal
‒ We can analyse the frequencies of the signal
• How?
‒ we will create a new « image » which will contains all
frequencies of the image
• Like a 2D frequency graphic
‒ The basic tool for it is the Fourier Transform

• We talk about the frequency domain, opposing to

the spatial domain (image)

52 Source : Gonzalez and Woods. Digital Image Processing. Prentice-Hall, 2002. 53

52 53

13
 07/03/2021

Fourier series Example

A bold idea (1807) - Jean t = [0,2], f = 1
Baptiste Joseph Fourier (1768-1830):
g(t) = sin(2πf t) + (1/3)sin(2π(3f) t)

Any univariate function can

be rewritten as a weighted
sum of sines and cosines of g = +
different frequencies.

Our building block: t

A sin(t ) + B cos(t )

Coefficient
Add enough of them to get
any signal g(t) you want!
Hays Slides: Efros

55

54 55

Fourier Transform Fourier Series

• Fourier transform is a mathematical transform
that
‒ Decomposes functions depending on space or time into
functions depending on spatial or temporal frequency ≈

We want to get this

function

f(x) |F(𝟂)|
signal Fourier transform
of signal Slide by Alexei A. Efros

56

56 57

14
 07/03/2021

Fourier Series Fourier Series

≈ + ≈ +

We want to get this We want to get this

function function

= =

Slide by Alexei A. Efros Slide by Alexei A. Efros

58 59

Fourier Series Fourier Series

≈ + ≈ +

We want to get this We want to get this

function function

= =

Slide by Alexei A. Efros Slide by Alexei A. Efros

60 61

15
 07/03/2021

Fourier Series Fourier Series

≈ + =

We want to get this We want to get this function We’ll get there in the limit
function

Slide by Alexei A. Efros Slide by Alexei A. Efros

62 63

The math Magnitude and phase

• Fourier transform stores the magnitude and phase at
each frequency
‒ Magnitude encodes how much signal there is at a particular
frequency
‒ Phase encodes spatial information (indirectly)
• Where are the sines and cosines?
‒ For mathematical convenience, this is often notated in terms
• The result is a complex function of real and complex numbers

• We’ve been showing only the amplitude A (spectre) so far: Amplitude: Phase:

I ( )
• Phase is also encoded:  = tan −1
A =  R( ) 2 + I ( ) 2 R( )
Slide by Steve Seitz Slide by Rober Pless

64 65

16
 07/03/2021

Discrete Fourier transform Basic Fourier Transform pairs

Source: Václav Hlavác - Fourier transform, in 1D and in 2D

67

66 67

Basic Fourier Transform pairs Basic Fourier Transform pairs

Source: Václav Hlavác - Fourier transform, in 1D and in 2D Source: Václav Hlavác - Fourier transform, in 1D and in 2D

68 69

68 69

17
 07/03/2021

Example: Music 2D FFT

• We think of music in terms of frequencies at different • Continuous FFT:
magnitudes

• Inverse FFT:

Slide: Hoiem

70 71

2D FFT - discrete Image Fourrier transform

Enhanced Spectra
Original image Spectra | F(u,v) |
log (1+ | F(u,v) |)

Frequency
domain
Spatial
domain

73

72 73

18
 07/03/2021

Image Fourrier transform Image Fourrier transform

Real part

Imaginary part
74 75

74 75

Inverse Fourrier transform Fourier analysis in images

Real part Intensity images (spatial domain)

Fourier images (spectral images – amtitude images)

Imaginary part http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering More: http://www.cs.unm.edu/~brayer/vision/fourier.html

76 77

76 77

19
 07/03/2021

Signals can be composed Fourier Transform of an image

Intensity images (spatial domain) Fourier decomposition
Natural image Frequency coefficients (amplitude)

+ =
f(x,y) |F(𝟂)|
What does it mean to be at pixel x,y?
What does it mean to be more or less bright in the Fourier decomposition image?
Fourier images (spectral images – amtitude images) Slide by Steve Seitz
http://sharp.bu.edu/~slehar/fourier/fourier.html#filtering
78

78 79

Fourier Bases Fourier Bases

Teases away ‘fast vs. slow’ changes in the image.

Green = sine
Blue = cosine

Hays
This change of basis is the Fourier Transform Hays
80 81

80 81

20
 07/03/2021

2D Fourier Transform Basis reconstruction

Slide by Steve Seitz

Danny Alexander
83

82 83

Image filtering in the frequential

2D Fourier transform domain
Image Fourier Spectrum
• We will be dealing only with functions (images) of finite
duration so we will be interested only in Fourier
Transform

Percentage of image power enclosed in circles (small to large) :

90, 95, 98, 99, 99.5, 99.9

Most of energy concentrated in low frequencies

84 85

84 85

21
 07/03/2021

Image filtering in frequential

Now we can edit frequencies!
domain
FFT

FFT

=
Inverse FFT

Slide by Derek Hoiem

87

86 87

Low-pass and high-pass filtering Low-pass filter

f(x,y) F(u,v)

spatial domain frequency domain

f(x,y) F(u,v)

G(u,v) = F(u,v) • H(u,v) •

filter
g(x,y) G(u,v)

Purpose: smoothing noise

g(x,y) H(u,v)
Slide by Alexei Efros

89

88 89

22
 07/03/2021

H(u,v) - Ideal low-pass filter Blurring - Ideal low-pass filters

1 D(u,v)  D0 D(u,v) =  u2 + v2
H(u,v) =
0 D(u,v) > D0 D0 = cut off frequency

H(u,v)
H(u,v)

v 1

u 0 D0 D(u,v)

99.7% 99.37% 98.65%

90 91

90 91

The ringing problem The ringing problem

Freq. domain Spatial domain

G(u,v) = F(u,v) • H(u,v)

Convolution Theorem

g(x,y) = f(x,y) * h(x,y)

IFFT sinc(x) 250

200
H(u,v) h(x,y)
150

100
 D0  Ringing radius +  blur
50

0
0 50 100 150 200 250

92 93

92 93

23
 07/03/2021

H(u,v) - Gaussian filter Blurring - Gaussain lowpass filter

H(u,v) H(u,v)
1

1/ e

v
u 0 D0 D(u,v)

-D2(u,v)/(2D20)
H(u,v) = e D(u,v) =  u2 + v2

Softer Blurring + no Ringing

99.11% 98.74% 96.44%

94 95

94 95

The Gaussian lowpass filter Blurring in the Spatial Domain

Freq. domain Spatial domain 111
Averaging = convolution with = point multiplication of the
111
transform with sinc:
1 2 1
Gaussian Averaging = convolution with 2 4 2
1 2 1
= point multiplication of the transform with a gaussian.
0.15

300 1

250
0.1 0.8

200 0.6

150 0.05 0.4

100 0.2

50 0 0
0 50 100 -50 0 50
0
0 50 100 150 200 250 300 Image Domain Frequency Domain

96 97

96 97

24
 07/03/2021

High-pass filter High-pass gaussian filter

H(u,v)
H(u,v) - Ideal Filter H(u,v)
1
0 D(u,v)  D0 D(u,v) =  u2 + v2 1− 1 / e
H(u,v) = v
1 D(u,v) > D0 D0 = cut off frequency
H(u,v) u 0 D0 D(u,v)
H(u,v)

v 1 -D2(u,v)/(2D20)
H(u,v) = 1 - e
u 0 D0 D(u,v)
D(u,v) =  u2 + v2

98 99

98 99

High-pass filtering High pass filtering

Original image Spectre with filters

−1 −1 −1
ℎ1 (3𝑥3) = −1 8 −1
−1 −1 −1

100 101

100 101

25
 07/03/2021

High pass filtering Band-pass filtering

0 D(u,v)  D0 - w D(u,v) =  u2 + v2
2
w
H(u,v) = 1 D0- w2  D(u,v)  D0 + 2
D0 = cut off frequency
w
0 D(u,v) > D0 + 2 w = band-width

H(u,v) H(u,v)

1
v
D(u,v)
0 D0- w D0 D0+w
2 2
u
Can be obtained by multiplying the filter functions of a low-pass and
of a high-pass in the frequency domain

102 103

102 103

Removing sinus noise Removing sinus noise

Brayer

104 105

26
 07/03/2021

High-pass filtering + orientation Hybrid Images

A. Oliva, A. Torralba, P.G. Schyns, SIGGRAPH 2006

106 107

Content
• Rappel: digital image representation
• Point Processing
• Convolution and Linear filtering
• More neighborhood operators
• Image transforms
‒ Frequency domain
‒ PCA (additional reading)
• PCA
• Example of using PCA for face recognition

108

108

27