9/9/2016

SIFT,track,OpenCV,ImageProcessing

EricYuan'sBlog
PerstandoetPraestando

Home

AboutMe

SIFT
ByERIC |Published:OCTOBER9,2013

SIFT(Scaleinvariantfeaturetransform)isoneofpopularfeaturematching
algorithms,itisgoodbecauseofitsseveralattributes.

Search
Tosearch,typeandhitenter

TagCloud

Algorithmalphabeta

activecontour

Therearekindsofprimitivewaystodoimagematching,forsomeimages,even
comparethegrayscalevaluepixelbypixelworkswell.However,whatiffeatures
inimagesareindifferentscales?Whatiffeaturesareindifferentorientations?
Whatiftheilluminationofeachimageisdifferentfromeachother?Whatifpart
ofthefeatureisshelteredbysomething?
Firstletslookatthisexample(picturestolenfromDavidG.Lowespaper):

ArmadilloBilateralFilterBlur

pruning

C++camShiftClusterCNNComputer
VisionConvolutionalNeuralNetworks

DeepLearningdeepnetwork
DynamicProgrammingGradient

DescentimageblendingImageProcessing

LeetCode

imagepyramid

MachineLearning
Matlab
MNISTneural
networksNLPOctaveOpenCV
meanShift

PanoramaPCAPoissonblendingPooling
pyramidblendingRANSACRecurrentNeural

NetworksRNNsseamcarvingSIFTsnake

SoftmaxSparseAutoencoder

UFLDLUnsupervisedLearning

track

ZCA

Pages
AboutMe

Categories
Algorithm
DeepLearning

Sayifwewannafindthetoytrainandfrog,asleftpictures,intheimageonthe
right,alloftheproblemsaboveappear,forexample,thefrog,morethanhalfof
itsbodyisshelteredbythatblackobject.

DynamicProgramming
Graphics
LeetCode
MachineLearning
MachineReadingComprehension

BySiftmethod,wegotthisresult:

Maths
NLP
OpenCV
Somethingelse
Twaddle
Uncategorized
Vision

Archives
July2016
August2015
June2015
April2015

http://ericyuan.me/sift/

1/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing
March2015
October2014
September2014
July2014
June2014
May2014
April2014
March2014
February2014
January2014
November2013
October2013
September2013

Reallyeffectivemethod,andOMG!Ididntnoticethetrainingreenrectangle.
Friends
Nowletstalkabouthowthismethodworks.

JekyCui'sblog
QiCai'sBlog

Ingeneral,themethodincludestheseparts:
Imagepyramid
Keypointsfinding
Refiningkeypoints
Keypointsorientationassignment
Localimagedescriptor
IMAGEPYRAMID
ThereisonearticleaboutImagePyramidinmyblog,soIllnotgivemuchdetails
aboutwhatisimagepyramidandhowtogenerateithere.
HereweareusingLaplacianpyramid,i.e.differencebetweendifferentlevelof
Gaussianpyramid,thatisbecause,inLaplacianpyramid,wecaneasilyfindhigh
frequencyinformationoftheimage,andfeaturesofanimage,aremostlyinthese
highfrequencypartsofimage.
Infact,DifferenceofGaussian(DoG)isanapproximationofLaplacianof
Gaussian(LoG).
Hereweactuallyneedtobuildseveralpyramids,fromdifferentscaleofimage,
andeachpyramid,wecallitanoctave:

http://ericyuan.me/sift/

2/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

Foreachoctaveofscalespace,theinitialimageisrepeatedlyconvolvedwith
Gaussianstoproducethesetofscalespaceimagesshownontheleft.Adjacent
GaussianimagesaresubtractedtoproducethedifferenceofGaussianimageson
theright.Aftereachoctave,theGaussianimageisdownsampledbyafactorof
2,andtheprocessrepeated.
Whyweneedtheseoctavepyramids?Thatisforscaleinvariant,bydoingthis,
wecanfindkeypointsindifferentscales.
Thenumberofoctavesandscalesdependonthesizeofyouroriginalimage
(Lowesuggestsmaking4octavespyramidsand5blurlevelsforeachpyramid).
Furthermore,Lowesuggeststhatwecandoublethesizeoftheoriginalimage,
thatisforproducingmorekeypoints.
KEYPOINTSFINDING
Thisstepiseasy,wejustgottheDoGpyramids,andnowwejustfind
maxima/minimaincurrentscale,andfindmaxima/minimainadjacentscales.
FirstineachlevelsinDoGpyramid,wefindpixelswhichisbigger(or
smaller)thantheeightadjacentpixels,andforallthesemaxima/minimapoints,
compareitsvaluewiththevalueofitsadjacentpixelsinlevelsaboveandbelowit.
Andthemaximaandminimapointswefinallyget,havegreaterorleastvalue
thanits9+8+9=26neighbors.

http://ericyuan.me/sift/

3/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

Forthefirstscaleandlastscaleofeachoctave,therearenoenoughadjacent
pixelstocalculate,sowesimplynotconsiderthesescaleswhenfindinglocal
maxima/minima.
REFININGKEYPOINTS
Afterfindingmaximaandminima,thenextstepistoperformadetailedfittothe
nearbydataforlocation,scale,andratioofprincipalcurvatures.This
informationallowspointstoberejectedthathavelowcontrast(andaretherefore
sensitivetonoise)orarepoorlylocalizedalonganedge.
FirstusingTaylorexpansionofscalespacefunction,bythisfunctionandits
derivatives,wecanfindlocationoftheextremum,andthefunctionvalueatthe
extremum,isusefulforrejectingunstableextremawithlowcontrast.
Secondly,wemusteliminateedgeresponses,becausefeaturesonedgesarenot
goodfeatures,whatweneediscorners,thisisaboutHarriscornerdetector,
andmaybeIllcoverthatinfuturearticles.
Infact,ifwewanttofindacorner,wedonothingjustcalculategradientsof
twodifferentorientationofanimage,ifbothofthesegradientsarebig,itisa
cornerelseifonlyonegradientisbigandtheotherissmall,thispointisonan
edgeelsethispointisinaflatregion.
KEYPOINTSORIENTATIONASSIGNMENT
Byassigningaconsistentorientationtoeachkeypointbasedonlocalimage
properties,thekeypointdescriptor(willbecoveredinnextstep)canbe
representedrelativetothisorientationandthereforeachieveinvariancetoimage
rotation.
Assumingwearegoingtoassignorientationtothebelowkeypoint:

Wechoosearegionwhichthekeypointisinthecenterofit,andthesizeofthis
regionisuptothescalewhichthatkeypointisin.

http://ericyuan.me/sift/

4/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

Byusingtheaboveformula,wecancalculatethegradientmagnitudemand
orientationthetaofeachpixelintheregionwechose:

andanorientationhistogramisformedwiththeseorientationsand
magnitudes.Theorientationhistogramhas36binscoveringthe360degree
rangeoforientations.Eachsampleaddedtothehistogramisweightedbyits
gradientmagnitudeandbyaGaussianweightedcircularwindowwithathatis
1.5timesthatofthescaleofthekeypoint.

Sayaboveisthehistogramwejustcreated,wecanseethepeakisat2029
degrees,andwesimplyassignthekeypointorientation3(thethirdbinof
histogram)furthermore,wecanseeanotherpeakwhichat300309degrees,
andthevalueofitisover80%ofthethirdbinvalue,underthiscircumstance,we
createanewkeypointwhichisassignedbyorientation31(whichmeansthe31st
binofhistogram).
LOCALIMAGEDESCRIPTOR
Thepreviousoperationshaveassignedanimagelocation,scale,andorientation
toeachkeypoint.Thenextstepistocomputeadescriptorforthelocalimage
regionthatishighlydistinctiveyetisasinvariantaspossibletoremaining
variations.
Fordoingthis,wefirstcreatea16*16pixelwindowwhichhavegradientvaluesof
pixelsaroundkeypointinit,andwebreakthiswindowintosixteen4*4pixel
windows,ofcourse,inordertoachieveorientationinvariance,thecoordinatesof
http://ericyuan.me/sift/

5/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

thedescriptorandthegradientorientationsarerotatedrelativetothekeypoint
orientation.

Sincewealreadyknowthegradientmagnitudesandorientationsofthesepixels,
justlikethepreviousstep,weputorientationsofthesepixelsintoa8bin
histogram(45degreesperbin).
Additionally,aGaussianweightingfunctionwithequaltoonehalfthewidthof
thedescriptorwindowisusedtoassignaweighttothemagnitudeofeachsample
point.ThepurposeofthisGaussianwindowistoavoidsuddenchangesinthe
descriptorwithsmallchangesinthepositionofthewindow,andtogiveless
emphasistogradientsthatarefarfromthecenterofthedescriptor,astheseare
mostaffectedbymisregistrationerrors.Thisprocessisshowedinthepicture
below,andinthepicture,assumingweuseda8*8pixelswindow,andthe
windowwasbrokenintofour4*4pixelwindows.

Bynow,wehavea4*4arrayofhistogramswith8orientationbinsineach,that
is,a4*4*8=128elementfeaturevectorforeachkeypoint.
Finally,thefeaturevectorismodifiedtoreducetheeffectsofilluminationchange
bynormalizingvectorslength.
Ifwedidallaboveprocesstotwoimages,saywegotk1keypointsinimage1,and
k2keypointsinimage2,sowehavek1*128vectorsinimage1,andk2*128vectors
inimage2,whatwedoissimplycompareeachkeypointsvectorsinimage1with
keypointsvectorsinimage2,andfindiftherearepairsofkeypoints.

http://ericyuan.me/sift/

6/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

OPENCVCODE
ItissupereasytodothisinOpenCV,theSIFTclassalreadydideverythingfor
you,sojustuseit!
01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

#include<iostream>
#include<opencv2/opencv.hpp>
#include<opencv2/nonfree/nonfree.hpp>
#include"opencv2/legacy/legacy.hpp"
usingnamespacecv;
usingnamespacestd;

intmain(intargc,char**argv)
{
//readimages
Matimg_1c=imread("img1.jpg");
Matimg_2c=imread("img2.jpg");
Matimg_1,img_2;
//transformimagesintograyscale
cvtColor(img_1c,img_1,CV_BGR2GRAY);
cvtColor(img_2c,img_2,CV_BGR2GRAY);

SIFTsift(50,5);
vector<KeyPoint>key_points_1,key_points_2;
Matdetector;
//dosift,findkeypoints
sift(img_1,Mat(),key_points_1,detector);
sift(img_2,Mat(),key_points_2,detector);

SiftDescriptorExtractorextractor;
Matdescriptors_1,descriptors_2;
//computedescriptors
extractor.compute(img_1,key_points_1,descriptors_1);
extractor.compute(img_2,key_points_2,descriptors_2);

//useburteforcemethodtomatchvectors
BruteForceMatcher<L2<float>>matcher;
vector<DMatch>matches;
matcher.match(descriptors_1,descriptors_2,matches);

//drawresults
Matimg_matches;
drawMatches(img_1c,key_points_1,img_2c,key_points_2,ma
tches,img_matches);
imshow("sift_Matches",img_matches);
waitKey(0);
return0;
}

YoucanalsochangevalueswhencreatingSIFT,by:
1 SIFT::SIFT(intnfeatures=0,intnOctaveLayers=3,doubleco
ntrastThreshold=0.04,doubleedgeThreshold=10,doublesigm
a=1.6);
TESTRESULT
(using50keypointsand5octavelayers)

http://ericyuan.me/sift/

7/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

ThisentrywaspostedinAlgorithm,OpenCVandtaggedImageProcessing,OpenCV,SIFT,
track.Bookmarkthepermalink.Postacommentorleaveatrackback:TrackbackURL.

BilateralFiltering

PoissonBlending

OneComment

Susannah
PostedFebruary4,2016at1:07pm|Permalink

Imalittleconfusedbythecodeinlines2529.IfImreadingthedocumentation
correctly,theSIFToperator(lines2223ofyourcode)alreadycomputesthedescriptors.
Soitappearstomethatyourcodeiscomputingthedescriptorstwiceisthatcorrect?
Reply

OneTrackback
BySURF,SIFT,OpenCV,ComputerVisiononNovember13,2013at11:30am

PostaComment
Youremailisneverpublishednorshared.Requiredfieldsaremarked*
Comment

http://ericyuan.me/sift/

8/9

9/9/2016

SIFT,track,OpenCV,ImageProcessing

YoumayusetheseHTMLtagsandattributes<ahref=""title=""><abbrtitle=""><acronym
title=""><b><blockquotecite=""><cite><code><deldatetime=""><em><i><qcite="">
<s><strike><strong>
Name*

Email*

Website

PostComment

PoweredbyWordPress.BuiltontheThematicThemeFramework.

http://ericyuan.me/sift/

9/9