Corner and Interest

point detection
• Many applications require relating two or more images in order to extract
information from them.
• For example, if two successive frames in a video sequence taken from a moving
camera can be related, it is possible to extract information regarding the depth of
objects in the environment and the speed of the camera.
• The brute force method of comparing every pixel in the two images is
computationally prohibitive for the majority of applications.
• Intuitively, one can image relating two images by matching only locations in the
image that are in some way interesting.
• Such points are referred to as interest points and are located using an interest
point detector.
• Finding a relationship between images is then performed using only these points.
This drastically reduces the required computation time.
 Applications of Corner Detectors
The use of interest points (and thus corner detectors) to find
corresponding points across multiple images is a key step in many
image processing and computer vision applications. Some of the most
notable examples are:
•stereo matching
•image registration (of particular importance in medical imaging)
•stitching of panoramic photographs
•object detection/recognition
•motion tracking
•robot navigation
• Local invariant features
– Detection of interest points
• (Harris corner detection)
• Scale invariant blob detection: LoG
– Description of local patches
• SIFT pipeline for invariant local features
 Local invariant features
1) Detection: Identify the interest
points

2) Description: Extract vector feature

descriptor surrounding each interest
point. x1  [ x1(1) ,  , xd(1) ]

3) Matching: Determine
correspondence between descriptors in
two views
x 2  [ x1( 2 ) , , xd( 2 ) ]
 1. Identify a Specific Instance
• General objects
– Challenges: rotation, scale, occlusion,
localization
– Approaches
• Geometric configurations of keypoints
– Works well for planar, textured objects
 1. Identify a Specific Instance
• Faces
– Typical scenario: few examples per face,
identify or verify test example
– What’s hard: changes in expression, lighting,
age, occlusion, viewpoint
– Basic approaches (all nearest neighbor)
1. Project into a new subspace (or kernel space) (e.g.,
“Eigenfaces”=PCA)
2. Measure face features
3. Make 3d face model, compare shape+appearance (e.g., AAM)
2. Detect Instance of a Category
• Much harder than specific instance
recognition

• Challenges
– Everything in instance recognition
– Intraclass variation
– Representation becomes crucial
• Template or sliding window
• Works well when
– Object fits well into rectangular window
– Interior features are discriminative

Schneiderman Kanade 2000

• Parts-based
 3. Assign a label to a pixel or
region
• Stuff
– Materials, object regions, textures, etc.
– Approaches
• Label patches + CRF
• Segmentation + Label Regions
 General Process of Object
Recognition
Specify Object Model

Generate Hypotheses

Score Hypotheses

Resolution
 General Process of Object
Recognition
Example: Template Matching

Intensity Template, at x-y

Specify Object Model
Scanning window

Generate Hypotheses

Normalized X-Corr
Score Hypotheses

Resolution Threshold + Non-

max suppression
General Process of Object Recognition
Example: Keypoint-based
Instance Recognition

A1 B3
Specify Object Model A3
A2

Affine-variant B2
B1
point locations
Generate Hypotheses
Affine
Parameters

Score Hypotheses
# Inliers

Resolution Choose hypothesis with max

score above threshold
Overview of Keypoint Matching
1. Find a set of
distinctive key-
points
A1 B3
2. Define a region
around each
A2 A3 keypoint
B2
3. Extract and
B1
normalize the
region content
fA fB
4. Compute a local
descriptor from the
normalized region
d ( f A, fB )  T
5. Match local
descriptors
 Main challenges
• Change in position and scale
• Change in viewpoint
• Occlusion
• Articulation
 Goals for Keypoints

Detect points that are repeatable and distinctive

 Hessian Detector
• Hessian determinant
Ixx

 I xx I xy 
Hessian ( I )  
 I xy I yy 
Iyy
Ixy

Intuition: Search for strong

derivatives in two
orthogonal directions
 Hessian Detector
• Hessian determinant
Ixx

 I xx I xy 
Hessian ( I )  
 I xy I yy 
Iyy
Ixy

det( Hessian( I ))  I xx I yy  I xy2

In Matlab:
I xx .  I yy  ( I xy )^ 2
K. Grauman, B. Leibe
 Hessian Detector – Responses

Effect: Responses mainly

on corners and strongly
textured areas.
 Harris Detector
• Second moment matrix
(autocorrelation matrix)
 I x2 ( D ) I x I y ( D )
 ( I ,  D )  g ( I )   
I I
 x y D( ) I 2
y ( )
D  

Intuition: Search for local

neighborhoods where the
image content has two
main directions
(eigenvectors).
K. Grauman, B. Leibe
 Harris Detector [Harris88]

• Second moment
Ix Iy
matrix 1. Image
derivatives
(autocorrelation 2. Square of Ix 2 Iy 2 I x Iy
matrix) derivatives

3. Gaussian g(Ix2) g(Iy2) g(IxIy)

filter g(I)

4. Cornerness function – both eigenvalues are strong

g(IxIy)
har  det[  ( I , D)]   [trace( ( I , D))] 
g ( I x2 ) g ( I y2 )  [ g ( I x I y )]2   [ g ( I x2 )  g ( I y2 )]2

5. Non-maxima suppression har

27
Harris Detector – Responses

Effect: A very precise

corner detector.
What Is A Useful Signature Function?
• Laplacian-of-Gaussian = “blob” detector
 Laplacian-of-Gaussian (LoG)
• Local maxima in scale
 
space of Laplacian-of-
Gaussian


Lxx ( )  Lyy ( ) 



 List of
(x, y, s)

 Difference-of-Gaussian (DoG)
• Difference of Gaussians as approximation of the
Laplacian-of-Gaussian

- =
 DoG – Efficient Computation
• Computation in Gaussian scale pyramid

Sampling with
step =2

1 
Original image  2 4

 Maximally Stable Extremal Regions
• Based on Watershed segmentation algorithm
• Select regions that stay stable over a large
parameter range

K. Grauman, B. Leibe
 Example Results: MSER

35 K. Grauman, B. Leibe
 Local Descriptors
• The ideal descriptor should be
– Robust
– Distinctive
– Compact
– Efficient

• Most available descriptors focus on

edge/gradient information
– Capture texture information
– Color rarely used
K. Grauman, B. Leibe
Local Descriptors: SIFT
Descriptor

Histogram of oriented
gradients
• Captures important texture
information
• Robust to small translations /
affine deformations
K. Grauman, B. Leibe
Local Descriptors: Shape Context

Count the number of points

inside each bin, e.g.:

Count = 4

...
Count = 10

Log-polar binning: more

precision for nearby points,
more flexibility for farther
points.

K. Grauman, B. Leibe
Local Descriptors: Geometric Blur

Compute
edges at four
orientations
Extract a patch
in each channel

~ Apply spatially varying

blur and sub-sample
Example descriptor
(Idealized signal)

Berg & Malik, CVPR 2001

K. Grauman, B. Leibe
 Choosing a detector
• What do you want it for?
– Precise localization in x-y: Harris
– Good localization in scale: Difference of Gaussian
– Flexible region shape: MSER

• Best choice often application dependent

– Harris-/Hessian-Laplace/DoG work well for many natural
categories
– MSER works well for buildings and printed things

• Why choose?
– Get more points with more detectors
 The probability of zero values on the first-order diffe
rence map of textured regions
– another measurement of streaking artifacts

original median filtered first-order difference map

G. Cao, et al. , “Forensic detection of median filtering in digital images”, ICME 2010

41
Median filter based detection

original median filtered first-order difference map

42