The Correspondence Problem

and "nterest Point Detection

Vclav Hlavc
Center for Machine Perception
Czech Technical University Prague
hlavac@fel.cvut.cz
Courtesy
Martin Urban (the talk is based on his
presentation) and Jana Kostkova, Jan Kybic,
Jiri Matas, Radim Sara
2
. The Correspondence Problem (CP) lies at the core of a number of
computer vision problems:
tracking (correspondence in consecutive frames
narrow-baseline stereo
wide-baseline stereo
egomotion (camera motion) estimation
motion segmentation in sequences with a moving camera
recognition, categorisation, .
2. Demo of typical applications
%e CP is commonIy soIved by robust matcing of "Interest
Points"
4 "Interest Points" are regions wit distinguising property,
aIIowing tere detection in a viewpoint and iIIumination invariant
manner
5 arris "Corner" (C) Detection AIgoritm
C are rotation and transIation invariant interest points
C are interest points most commonIy used in tracking and stereo
Lecture Overview
3
"IdeaI" soIution: a pixel to pixel mapping from A to B (B NULL)
What it is "The correspondence problem?
A B
Ex. result: x-shift
AppIications: D Reconstruction
D Reconstruction
Camera motion tracking ; image stabilization
original stabilized
original stabilized
Camera motion tracking ; D animation
Motion keying / segmentation
input sequence
Background sequence Foreground sequence
Motion Detection
nput:
Output:
Medical imaging image registration
from the atlas
test slice
deform. field
before registration after
11
"IdeaI" soIution: a pixel to pixel mapping from A to B
The correspondence problem
A B
How to do it?
12
The correspondence problem
A B
Intuitive approac:
For pxl A, find a pxl B
with the most similar neighbourhood.
ProbIems: - How to measure similarity of image patches?
- undistinguishable regions (e.g. texture-less)
- not surjective map (onto) due to occlusions
- not bijective map (one to one) due to scale changes
- huge data
; very ard / impossibIe to recover te mapping by direct
minimisation
13
The correspondence problem
ConcIusion:
t is very hard / impossible to recover dense pxl 2 pxl mapping between
two images.
SoIution:
. Recover the correspondence relation just between several well
distinguished image features (interest points / corners / Harris points).
2. Estimate multi-view transformation (e.g. epipolar geometry, camera
motion).
. Having epipolar geometry, try to find dense correspondences.
14
Corner Detection: ntroduction
Corner detector detects points with distinguished neighbourhood(*) well
suited for matching verification.
undistinguished patches:
distinguished patches:
15
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
16
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
17
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
18
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
19
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
20
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
21
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
22
-
Demo of a point + with well distinguished neighbourhood.
+
> 0
Corner Detection: ntroduction
23
Example of detected points
Corner Detection: ntroduction
24
Corner Detection: Basic principle
undistinguished patches: distinguished patches:
mage gradients of undist. patches are (0,0) or have only one
principle component.
mage gradients of dist. patches have two principle components.
; rank ( _

) = 2
25
Algorithm (C. Harris, )
filter the image by gaussian (2x D convolution), sigma_d
compute the intensity gradients (2x D conv.)
for each pixel and given neighbourhood, sigma_i:
- compute auto-correlation matrix
A = _

- and evaluate the response function #(A):

#(A) >> 0 for rank(A)=2, #(A) 0 for rank(A)<2
4 choose the best candidates (non-max suppression and
thresholding)
26
Corner Detection: Algorithm (R. Harris, )
Harris response function #(A):
#(A) = det (A) k*trace
2
(A) ,
[lamda,lambda2] = eig(A)
27
Corner Detection: Algorithm (R. Harris, )
AIgoritm properties:
+ "invariant to 2D image shift and rotation
+ invariant to shift in illumination
+ "invariant to small view point changes
+ low numerical complexity
- not invariant to larger scale changes
- not invariant to high contrast changes
- not invariant to bigger view point changes
28
Corner Detection: Algorithm (C. Harris, )
Exp.: Harris points and view point change
29
Corner Detection: Harris points versus sigma_d and sigma_i
Sigma_

Sigma_d
30
Corner Detection: Application
Algorithm:
. Corner detection
2. Tentative correspondences
- by comparing similarity of the corner neighb. in the searching window
(e.g. cross-correlation)
. Camera motion geometry estimation (e.g. by RANSAC)
- finds the motion geometry and consistent correspondences
4 D reconstruction
- triangulation, bundle adjustment
D camera motion tracking / D reconstruction
31
Corner Detection: Camera Tracking Application - Boujou
nput sequence
32
Corner Detection: Camera Tracking Application - Boujou
Harris points
33
Corner Detection: Camera Tracking Application
points consistent with D camera motion
34
Corner Detection: Camera Tracking Application
D points and D camera motion
35
Corner Detection: Camera Tracking Application
D animation
36
Corner Detection: Camera Tracking Application - Boujou
nput sequence
37
Corner Detection: Camera Tracking Application - Boujou
Harris points
38
Corner Detection: Camera Tracking Application - Boujou
points consistent with D camera motion
39
for the presentation copied almost done presntation
artin Urban
for used demo images and software:
Jana Kostkov, CMP - (slide : images, disparity map)
Radim Sra, CMP - (slide 5: images, D face reconstruction demo)
Jan Kybic, CMP - (slide 0: medical image registration)
d (slide -: Boujou Demo, slide 6: img. sequences)
ImagineerSystems (slide : MoKey Demo, slide : img. sequences)
%e PixeI Farm (slide -: img. sequence)
Acknowledgements