Multimedia Retrieval
Ch 5 Image Processing
Ch 5 Image Processing
Anne Ylinen
Anne Ylinen
Agenda
Agenda
Types of image processing
Types of image processing
Application areas
Application areas
Image analysis
Image analysis
Image features
Image features
Types of Image Processing
Types of Image Processing
Image Acquisition
Image Acquisition
Camera
Camera
Scanners
Scanners
X
X


ray imagers
ray imagers
Computer tomography (CT)
Computer tomography (CT)
Magnetic resonance scanners (MR)
Magnetic resonance scanners (MR)
Ultra sound devices (US)
Ultra sound devices (US)
Types of Image Processing
Types of Image Processing
Image Restoration
Image Restoration
Geometric distortions
Geometric distortions
Noise
Noise
Unsharpness
Unsharpness
Types of Image Processing
Types of Image Processing
Image Reconstruction
Image Reconstruction
using models
using models
different viewpoint
different viewpoint
another imaging device
another imaging device
Types of Image Processing
Types of Image Processing
Image Enhancement
Image Enhancement
Contrast enhancement
Contrast enhancement
amplitude scaling amplitude scaling
contrast modification contrast modification
Histogram normalization
Histogram normalization
nonadaptive nonadaptive histogram modification histogram modification
adaptive histogram modification adaptive histogram modification
Edge enhancement
Edge enhancement
linear edge linear edge crispening crispening
statistical differencing statistical differencing
Types of Image Processing
Types of Image Processing
Image Registration
Image Registration
Rigid registration
Rigid registration
Non
Non


rigid registration
rigid registration
Used in medical applications, cartography, face
Used in medical applications, cartography, face
recognition, etc.
recognition, etc.
Types of Image Processing
Types of Image Processing
Image Compression, Storage and Transmission
Image Compression, Storage and Transmission
Lossless
Lossless
image can be exactly reconstructed image can be exactly reconstructed
Lossy
Lossy
approximate reconstruction approximate reconstruction
Types
Types
of
of
Image Processing
Image Processing
Image Analysis
Image Analysis
Image analysis aims to generate a description of the
Image analysis aims to generate a description of the
image or of objects present in the image.
image or of objects present in the image.
Application Areas
Application Areas
Medical Imaging
Medical Imaging
MR, CT, US MR, CT, US
Geo Information Systems, Satellite, Aerial photography
Geo Information Systems, Satellite, Aerial photography
and Cartography
and Cartography
Biometry
Biometry
Face and fingerprint recognition, Face and fingerprint recognition, handpalm handpalm recognition, recognition,
tracking people tracking people
feature feature based and holistic approaches based and holistic approaches
Optical Character Recognition
Optical Character Recognition
Industrial Vision
Industrial Vision
Multimedia and Image Databases
Multimedia and Image Databases
Image Analysis
Image Analysis
extract information from an image
extract information from an image
detection
detection
classification
classification
parameter estimation
parameter estimation
structural analysis
structural analysis
Image Analysis
Image Analysis
comparison
feature
extraction
model
selection
observed
data
database of
models
model features
observed
features
match criterion
selected
model
Image Analysis
Image Analysis
Image analysis task
Image analysis task
the selection of the features
the selection of the features
the representation of the models
the representation of the models
the matching criterion
the matching criterion
the selection strategy
the selection strategy
Image Features
Image Features
Image
Image
2
2


dimensional signal
dimensional signal
represented by a matrix F of pixels of N rows and M
represented by a matrix F of pixels of N rows and M
columns
columns
A pixel value
A pixel value
f(n,m
f(n,m
) is an intensity or a vector of 3
) is an intensity or a vector of 3
RGB components
RGB components
mathematical operations are possible e.g. derivative
mathematical operations are possible e.g. derivative
and Fourier transformation
and Fourier transformation
Image Features
Image Features
Pixel Features
Pixel Features
Neighborhood and Image filtering
Neighborhood and Image filtering
each pixel an individual feature each pixel an individual feature
neighboring pixels grouped together neighboring pixels grouped together
used to obtain higher level features used to obtain higher level features
Image Features
Image Features
Scale space and derivatives
Scale space and derivatives
scale at which objects are seen in an image depends on the scale at which objects are seen in an image depends on the
distance between object and camera distance between object and camera
scale space theory for handling image structures at scale space theory for handling image structures at
different scale different scale
derivatives important for edge detection, point feature derivatives important for edge detection, point feature
detection, and so on detection, and so on
Image Features
Image Features
Texture
Texture
small elementary pattern repeated periodically or quasi small elementary pattern repeated periodically or quasi 
periodically periodically
geometric or radiometric pattern geometric or radiometric pattern
important clues for segmenting the image important clues for segmenting the image
typified by typified by
the distance over which the patter is repeated the distance over which the patter is repeated
the direction in which the pattern is repeated the direction in which the pattern is repeated
the properties of the elementary pattern the properties of the elementary pattern
co co occurrence matrices occurrence matrices
Image Features
Image Features
Point Features
Point Features
Interest points
Interest points
corner points and spots corner points and spots
video tracking, stereo matching, object recognition video tracking, stereo matching, object recognition
Harris corner detector Harris corner detector
Image Features
Image Features
Harris corner detector Harris corner detector
image image I(x,y I(x,y) and sifted image ) and sifted image I(x+u I(x+u, , y+v y+v) )
Gaussian window function Gaussian window function w(x,y w(x,y) )
E(u,v E(u,v) should change fast for small sifts of ( ) should change fast for small sifts of (u,v u,v) )
[ ]
+ + =
y x
y x I v y u x I y x w v u E
,
2
) , ( ) , ( ) , ( ) , (
cuv bv au v u E 2 ) , (
2 2
+ +
Image Features
Image Features
2 1
2 1
2
2 1
,
2
2
det
) ( det
M of s eigenvalue ,
) , (
] , [ ) , (
+ =
=
=
traceM
M
traceM k M R
I I I
I I I
y x w M
where
v
u
M v u v u E
y x
y y x
y x x
Image Features
Image Features
2
Corner
Edge
Edge
Flat
R depends only on
eigenvalues of M
R is large for a corner
R is negative with large
magnitude for an edge
R is small for a flat
region
R > 0
R < 0
R < 0 R small
sourse(www.wisdo sourse(www.wisdom m.weiz .weizm mann.ac.il/~deniss/vision_sp ann.ac.il/~deniss/vision_spr ring04/files/Inva ing04/files/Invar riantFeatu iantFeatur res.ppt es.ppt )
Image Features
Image Features
Line elements Line elements
line segments have a width in the image equal to the scale of line segments have a width in the image equal to the scale of
the image, Gaussian like profile across the line the image, Gaussian like profile across the line
calculate the second derivative in the direction orthogonal to calculate the second derivative in the direction orthogonal to
the gradient vector the gradient vector
more stable result is obtained by approximating the more stable result is obtained by approximating the
neighborhood of each candidate line element by quadratic neighborhood of each candidate line element by quadratic
surface: surface:
( (n,m n,m) is the position of the candidate line element ) is the position of the candidate line element
ckl bl ak m n f l m k n f 2 ) , ( ) , (
2 2
+ + +
Image Features
Image Features
using Taylor expansion using Taylor expansion
1 1
, ,
2 2
are are eigenvalues eigenvalues of H of H
for true line element, one for true line element, one eigenvalue eigenvalue should be large and should be large and
the other small the other small
[ ]
+
yy xy
xy xx
f f
f f
H
where
l
k
H l k m n f l m k n f ) , ( ) , (
Image Features
Image Features
Edge elements
Edge elements
stepwise transition in intensities stepwise transition in intensities
neighboring edge elements linked to neighboring edge elements linked to gether gether form an edge form an edge
segment segment
gradient is large at the position of an edge gradient is large at the position of an edge
Gradient Gradient based methods based methods
Laplacian Laplacian based methods based methods
Canny Canny s s method method
Image Features
Image Features
Canny Canny s s method method
1. 1. Smooth the image with Gaussian filter Smooth the image with Gaussian filter
g(x,y g(x,y)= )=g g
c c
(x,y (x,y) )
* *f(x,y f(x,y) )
where where
where where represents the width of the Gaussian distribution represents the width of the Gaussian distribution
2. 2. Compute the second derivative in the gradient direction Compute the second derivative in the gradient direction
3. 3. Find zero crossings of the second derivative Find zero crossings of the second derivative


\
 +
=
2
2 2
2
exp
2
1
) , (
y x
y x g
c
2 2
2 2
2
2
2
y x
yy y xy y x xx x
g g
g g g g g g g
n
g
+
+ +
=
Image Features
Image Features
Pros: Pros:
One pixel wide edges One pixel wide edges
Edges are grouped together Edges are grouped together
(often good for (often good for
segmentation) segmentation)
Robust against noise! Robust against noise!
Cons: Cons:
Complicated to understand Complicated to understand
and implement and implement
Slow Slow
References
References
Blanken
Blanken
et al, Multimedia Retrieval, 2007, Springer
et al, Multimedia Retrieval, 2007, Springer
Pratt, W: Digital Image Processing, 2001, John Wiley
Pratt, W: Digital Image Processing, 2001, John Wiley
& Sons INC
& Sons INC
Bovik
Bovik
, A: Handbook of Image & Video Processing,
, A: Handbook of Image & Video Processing,
2000, Academic Press
2000, Academic Press
Castelman
Castelman
, K: Digital Image Processing, 1996, Prentice
, K: Digital Image Processing, 1996, Prentice
Hall
Hall
Harris, C: A Combined Corner and Edge Detector,
Harris, C: A Combined Corner and Edge Detector,
1988,
1988,