2/10/11

Features  Points  Extrac0on  

IVÁN  F.  MONDRAÓN  B.  

Cmputer  Vision  Group  

Introduc0on  
Detec0ng   features   is   an   important   step   in   localizing   or  
recognizing  objects  in  the  image.    
Example  applica0ons:  
•  Mosaicing  –  alignment  of  overlapping  image  parts  
•  Stereo  image  processing  
•  Medical  imaging  –  superimpose  different  modali0es  
•  Mo0on   compensated   processing   of   video   sequences  
(stabiliza0on,  filtering,  compression)    
•  Recogni0on  –  comparison  with  template  
•  Visual  search  –  image  data  base  query  

VISUAL  FEATURES  
Visual Features can be defined as an
“interesting” part of an image, used as a starting
point for computer visual algorithms on images
sequences.

Visual Features should have a Repeatability

property (detect the same feature in two or more images
of the same scene taken under different viewing
conditions ).
1.  Invariance: unaffected to large deformations
2.  Robustness: noise, blur, discretizaciton, etc

1
 2/10/11

VISUAL  FEATURES  

• Other Desirable Properties:

• Distinctiveness/informativeness: intensity patterns
around point should show a lot of variation, so feature
can be distinguished and matched.
•  Quantity: The number of detected features should be
sufficiently large, such that a reasonable number of
features are detected even on small objects.

VISUAL  FEATURES  

• Other Desirable Properties:

• Accuracy: The detected features should be
accurately localized,
both in image location, as with respect to scale and
possibly shape.
• Efficiency: Preferably, the detection of features in a
new image should be as faster as possible

Kinds  of  Features  

•  There  are  a  large  variety  of  visual  features.  
They  can  classified  as:  
– Edges:  points  where  there  is  a  boundary  (or  an  
edge)  between  two  image  regions.  
– Corners  (Interest  Points):  point-­‐like  features  in  
an  image,  which  have  a  local  points  in  the  2D  
image  with  high  curvature.  These  do  not  necessarily  
correspond  to  projec0ons  of  3D  corners.  

2
 2/10/11

Kinds  of  Features  

•  There  are  a  large  variety  of  visual  features.  
They  can  classified  as:  
– Blobs   (Regions   of   interest):   generally  
correspond   to   a   point   (a   local   maximum)   with   an  
associated  descrip0on  of  the  surrounding  region.      
– Ridges:   Is   a   one   dimensional   curve   that  
represents   an   axis   of   symmetry   associated   to   a  
local  maximum  points.  

Classifica0on  
Feature  Detector   Edge   Corner   Blob  
Canny   X  
Sobel   X  
Harris   X   X  
SUSAN   X   X  
Shi  &  Tomasi   X  
Laplacian  of  Gaussian   X   X   X  
Difference  of  Gaussians   X   X  
SIFT   X   X  
SURF   X   X  

Gaussian  Smoothing  
•  Is  a  2D  convolu0on  operator  used  to  “blur”  images  
and  remove  noise.  The  kerned  is  made  in  order  to  
represents  the  shape  of  a  Gaussian.  
x 2 +y 2
1 −
G(x, y) = e 2σ 2
2πσ 2

•  At  a  defined  scale-­‐  σ,  the  Input  Images  I(x,y)  

convolved  with  the  Gaussian  kernel  G(x,y, σ) is  
I(x, y,σ ) = G(x, y,σ ) * I(x, y)

3
 2/10/11

Laplacian  and  LoG  

•  Laplacian:  Is  a  2D  isotropic  measure  of  the  
second  spa0al  derivate  of  an  image.  
∂ 2I ∂ 2I
L(x, y) = +
∂x 2 ∂y 2
∇ L = Lxx + Lyy
2

Laplacian of Gaussian (LoG): is a convolution of the Gaussian

smoothing filter with the Laplacian

x +y
2 2
1 ⎡ x 2 + y 2 ⎤ − 2σ 2
LoG(x, y) = − 1− e
πσ 4 ⎢⎣ 2σ 2 ⎥⎦
∇ 2σ L = ∇ 2Gσ * L

Edges  based  on  LoG  

One dimension
illustration of 2nd
derivative edge

Corners  
•  Detected  points  correspond  to  points  in  the  
2D  image  with  high  curvature.  
•   Image  Corners  do  not  necessarily  correspond  
to  projec0ons  of  3D  corners.    
•  Corners  are  found  at  various  types  of  
junc0ons,  on  highly  textured  surfaces,  at  
occlusion  boundaries,  etc  

4
 2/10/11

Harris  Corner  Detector  

•  Based  on  the    magnitude  of    eigenvalues  λ1    ,  λ2  
of  the  Second  moment  matrix  (auto-­‐correla7on  
matrix)      

Harris  Corner  Detector  

1.  Compu0ng  the  first-­‐order  deriva0ves  Ix  and  Iy

2.  Takes  the  product  of  these  gradient  images  
3.  Images  are  smoothed  with  a  Gaussian  kernel  
4.  Cornerness  measure  using  (Cx,Cy)=λ1 λ2 – k(λ1 +λ2 )

Harris  Corner  Detector  

1.  Invariant  to  brightness  offset:  I(x,y)≈I(x,y)+c  

2.  Invariant  to  shi>  and  rota7on  
3.  Not  invariant  to  scaling    

Edge Corner

5
 2/10/11

Scale  Invariance  

•Find local maxima of:

–Harris corner detector in space (image coordinates)
–For each Harris corner detect characteristic scale as maximum
scale-normalized Laplacian σ • ∇ 2 L in different scales (σ)

Hessian  Detector  
•  The   eigenvalues   of   the   second   moment   matrix  
intensity   func0on   I(x,y))     are   used   to   measure  
the  affine  shape  of  the  point  neighborhood  

Hessian  Detector  
•  Hessian   describe   how   the   normal   to   an   Isosurface  
changes.    
•  Two  kind  of  measures    based  on  the  Hessian:  
–  Determinant  of  Hessian:   Det(H) = Ixx Iyy − Ixy 2 = λ1λ 2
–  Trace  of  Hessian  (Laplacian):   Trace(H) = Ixx + Iyy = λ1 + λ 2
•  Blobs   are   detected   by   finding   local   maximums   at  
same  loca0on  on  Det(H)  and    Trace(H)  

6
 2/10/11

Hessian  Detector  

•  Compute  the  second-­‐order  Gaussian  smoothed  derivates  Ixx  ,  Iyy , Ixy

•  Compute  the  determinant  of  the  Hessian=Ixx  Iyy – Ixy2
•  Find  structures  with  local  maximum  of  the Det(H) and Trace(H)

SIFT  (Scale  Invariant  Feature  Transform)  

•  Generates  image  features,  “keypoints”  

– invariant  to  image  scaling  and  rota0on  
– par0ally  invariant  to  change  in  illumina0on  and  3D  
camera  viewpoint  
– many  can  be  extracted  from  typical  images  
– highly  dis0nc0ve  
.  

SIFT-­‐  LoG  approxima0on  

Laplacian  of  Gaussians  LoG  are  computa0onal  

expensive  to  calculate  

approximated with a Difference of Gaussians

DoG

7
 2/10/11

SIFT:  SIFT-­‐  LoG  approxima0on  

•  The   difference-­‐of-­‐Gaussian   filter   provides   an  

approxima0on   to   the   scale-­‐normalized  
Laplacian  of  Gaussian  σ2∇2G.  The  difference-­‐
of-­‐Gaussian   filter   is   in   effect   a   tunable  
bandpass  filter  

SIFT:  SIFT-­‐  LoG  approxima0on  

SIFT:  DoG  Detec0on  scheme  

2k2σ
2kσ
2kσ
2σ
2σ
kσ
kσ
σ
σ

An octave corresponds to doubling the value of σ.

8
 2/10/11

SIFT:  DoG  Detec0on  scheme  

•  Scale  Space   First

Octave

Images  
Second

Third
Fourth

•  Difference  of  
Gaussian  
Images  

SIFT:  DoG  Detec0on  scheme  

•  Pixel  marked  with  “X”  is  

compared  to  26  neighbors  
on  the  DoG  image,  in  a  
3x3x3  window  that  spans  
adjacent  pixels  and  scales  
•  If  Pixel  is  a  Maximum  or  a  
Minimum,  it  is  selected  

SIFT:  Descriptor  
Basic idea:
•  Take 16x16 square window around detected feature
•  Compute edge orientation (angle of the gradient - 90°) for each pixel
•  Throw out weak edges (threshold gradient magnitude)
•  Create histogram of surviving edge orientations

0 2π
angle histogram

9
 2/10/11

SIFT  (Scale  Invariant  Feature  Transform)  

Full version
•  Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below)
•  Compute an orientation histogram for each cell
•  16 cells * 8 orientations = 128 dimensional descriptor

SURF:  Speeded  Up  Robust  Features  

•  Fast  interest  point  detec>on  
•  Dis0nc0ve  interest  point  descrip>on  
•  Speeded-­‐up  descriptor  matching  
•  Invariant  to  common  image  transforma0ons:  
–  Image  rota0on,  Scale  changes,  Illumina0on  change  
And  Small  change  in  Viewpoint    
•  Based  on  Hessian  matrix  

Methodology  
•  Using  integral  images  for  major  speed  up  
–  Integral  Image  (summed  area  tables)    is  a  quickly  and  efficiently  
method  to  obtain  the  sum  of  values  in  a  rectangular  subset  of  a  grid.  

Integral Image a(x, y) = a(x, y − 1) + I(x, y) → a(x,-1) = Ii (−1, y)

computed in one-pass II (x, y) = II (x − 1, y) + a(x, y)

Sum o pixel
values within a
rectangular region

Cost four additions

operation only 30  

10
 2/10/11

SURF:  Hessian  Approxima0on  

•  Simple  approxima0on  of  Gaussian  deriva0ves  

Gaussian second order partial Approximations to Dyy and Dxy

derivatives in y-direction and xy- using box filters (mean/average
direction filter)

Det(H) ≈ Dxx Dyy − (0.9Dxy ) 2

SURF:  scale  invariance  

•  Scale  invariance  achieved  by  up-­‐scaling  of  the  
filter  at  constant  cost  Instead  of  itera0vely  
reducing  the  image  size.  

9 x 9, 15 x 15, 21 x 21, 27 x 27  39 x 39, 51 x 51 …

1st octave 2nd octave

SURF:Detec0on  
•  Non-­‐maximum  suppression  and  Interpola0on  
of  maximum  of  Det(H)  in  image  space  x,y  and  
scale  t

33  

11
 2/10/11

SURF:  Descriptor  
•  Orienta0on  Assignment  
Circular neighborhood of
radius 6s around the interest point
(s = the scale at which the point was detected)

x response y response

Side length = 4s
Cost 6 operation34  to
compute the response

SURF:  Descriptor  
•  Dominant  orienta0on  
–  The  Haar  wavelet  responses  are  represented  as  vectors  
–  Sum  all  responses  within  
a  sliding  orienta0on  
window  covering  an  angle    
of  60  degree  
–  The  longest  vector  is  the    
dominant  orienta0on  

35  

SURF:  Descriptor  
•  Split  the  interest  region  up  into  4  x  4  square  sub-­‐regions  with  
5  x  5  regularly  spaced  sample  points  inside  
•  Calculate  Haar  wavelet  response  dx  and  dy  
•  Weight  the  response  with  a  Gaussian  kernel  centered  at  the  
interest  point  
•  Sum  the  response  over  each  sub-­‐region  for  dx  and  dy  
separately    feature  vector  of  length  32    
•  In  order  to  bring  in  informa0on  about  the  polarity  of  the  
intensity  changes,  extract  the  sum  of  absolute  value  of  the  
responses    feature  vector  of  length  64  
•  Normalize  the  vector  into  unit  length  

36  

12
 2/10/11

SURF:  Descriptor  

37  

Feature  Matching  
Given  a  feature  in  I1,  how  to  find  the  best  match  
in  I2?  
1.  Define  distance  func0on  that  compares  two  
descriptors  
2.  Test  all  the  features  in  I2,  find  the  one  with  min  
distance  

Feature  Matching  
Simple  approach  is  SSD(f1,  f2)    
•  sum  of  square  differences  between  entries  of  the  two  descriptor  

f1 f2

I1 I2

13
 2/10/11

Feature  Matching  

ra0o  distance  =  SSD(f1,  f2)  /  SSD(f1,  f2’)  

•  f2  is  best  SSD  match  to  f1  in  I2  
•  f2’    is    2nd  best  SSD  match  to  f1  in  I2  
•  gives  small  values  for  ambiguous  matches  

f1
f2' f2

I1 I2

Examples  of  Features  

;Bug Workaround;

(a) Features obtained using Good Features to Track,

(b) Keypoints obtained using SIFT (the green arrows represents
the keypoints orientation and scale
(c) Descriptors obtained using SURF (red circles and line
represents the descriptor scale and angle)

Object  Detec0on  

14
 2/10/11

Features  Applica0ons  

SURF Matching SIFT Matching

Mosaic  Building  

Video  Stabiliza0on  

15
 2/10/11

References  
•  C.  Harris  and  M.  Stephens,  “A  combined  corner  and  edge  detector,”  
in  Alvey  
•  Vision  Conference,  pp.  147–151,  1988.  
•  D.  Lowe,  “Dis0nc0ve  image  features  from  scale-­‐invariant  keypoints,”  
Interna0onal.  Journal  of  Computer  Vision,  vol.  2,  no.  60,  pp.  91–110,  
2004.  
•  H.  Bay,  T.  Tuytelaars,  and  L.  Van  Gool,  “SURF:  Speeded  up  robust  
features,”  
•  in  Proceedings  of  the  European  Conference  on  Computer  Vision,  pp.  
404–417,  2006.  
•  Tinne  Tuytelaars  and  Krys0an  Mikolajczyk    "Local  Invariant  Feature  
Detectors:  A  Survey",  Founda0ons  and  Trends®  in  Computer  
Graphics  and  Vision:  Vol.  3:  No  3,  pp  177-­‐280.  2008    

16