3.2K views

Uploaded by pi194043

Overview and Implementation Details of OpenCV code for Spatio Temporal Feature extraction using harris 3d corner detector.You can see the demo videos at http://pi-virtualworld.blogspot.com/

save

You are on page 1of 10

Pi19404

February 27, 2013

Contents

Contents

Feature Detection - Harris Corner 3D Detector 3

0.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 0.2 Spatio-Temporal Features . . . . . . . . . . . . . . . 0.2.1 Model of Spatio-termporal Image Sequence . 0.3 Computation of matrix parameters . . . . . . . . . . 0.3.1 Matrix Paramter A,B,C . . . . . . . . . . . . . . . 0.3.2 Matrix parameters D,E,F . . . . . . . . . . . . 0.3.3 Implementation Details . . . . . . . . . . . . . . 0.4 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.5 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 0.6 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

3 3 4 6 6 7 8 8 9 9 10

2 | 10

0.1 Introduction

In this article we will look at the extension of harris corner detector called harris 3D detector used to detect robust spatio temporal corners . The harris corner detection is used to detect robust spatial corners which detects points with significant local variation of image intensities. The 3D harris corner detector is used to detect spatial as well as temporal corners that can be detected over a sequence of frames.It finds spatical corners with velocity change which exhibit changes in space as well as time.

The aim of spatial feature detection is to identify point in the image that correpond to object in motion that can be tracked in adjacent image frames robustly. The aim of spatio-temporal feature detection is to identify points in the image where significant spatial and temporal intensity changes are observed.These locations provide a sparse representation of motion in the image and local neighborhood of these points can be used to represent motion occuring in the image. It is required that the feature be as sparse as possible to reduce the computational requirement and enable us to achieve real time tracking.

3 | 10

Feature Detection - Harris Corner 3D Detector Potential candidates for Spatial interest points are edge pixels,points with significant changes in the neighborhood and junction of corners. Potential candidates for Spatio-Temporal interest points are points which exhibit large variation in spatial as well as temporal dimensions.

Consider a Image sequence represented by I (x; y; t) . In harris corner detector we considered a small window and analyzed the change in average intensity of as window is shifted by small ammount.If the change is large of all the shifts we considered the point as a interest corner. For spatio-temporal corner we will consider a 3D window about a point and analyze the average intensity change as window is shifted by small ammount in spatial as well as temporal dimension.If the change is large for all spatial as well as temporal shifts we would consider the point as spatio-temporal interest corner.

The point being considered is I (u; v; t) .x,t, are spatio-temporal shifts.Thus we accumulate frames at different time instants.We compute average change in intensity of frame at instant t with shifted windows at time instants t 1; t + 1.

E (x; y; ) =

jI (x + u; y + v; t + ) I (u; v; t)j

(1)

4 | 10

I (x + u; y + v; t + )

@I @I I (x + u; y + v; t + ) = I (u; v; t) + x @x + y @y + @I @t

W W W

@I @I jx @x + y @y + @I j @t

jxX + yY + T j

x2 X 2 + y2 Y 2 + T 2 + 2xyXY

W

+2

yY T + 2xXT

P

Y2+

2 T 2 + 2xy

XY

(2)

E (x; y; ) = Ax2 + By2 + 2Cxy + 2yD + 2xE + 2 F x A C E E (x; y; ) = x y RC B DS Ry S E D F The shifts considered are (0; 1; 0); (1; 1; 0); (0; 1; 0); (1; 1; 0); (0; 1; 1); (1; 1; 1) (0; 1; 1); (1; 1; 1); (0; 1; 1); (1; 1; 1); (0; 1; 1); (1; 1; 1).

These shifts correspond to spatial derivatives in x and y direction , spatial derivatives wrt termporal windows . In Harris 2D detector we used a 2D window ,similarily in 3D harris detector we use a 3D window. The block size represented the size of 2D window in harris corner detector. For 3D harris corner detector we would have to store the image frames corresponding to block size.Large the block size in temporal dimension longer is observation interval for which change in spatial interest points are observed. Again the eigen values of matrix are analysed and points which provide all three large eigen values indicates a corner points. A similar measure to 2D harris corner detector can be used.

QP Q

5 | 10

(3)

Thus we take maximal value of k and compute the response function for each point .If the response function is above a certian threshold we can classify it as a spatio-temporal corner.

If the point being being analyzed is I(u,v,t) then we require image blocks I (u bs; v bs) to I (u + bs; v + by ) for all image frames from t bst to t + bst ie a 3D block about the point u; v; t. To accomodate for the first order spatial and temporal shifts we require image frames in time interval t bst 1 to t + bst + 1 about the point u; v with a patch size of bs. Thus first step in the program is to accumulate frames frames for anaysis.

bst + 2

For harris corner detector we computed this as

A= B= C=

w w w

1 1

@I W @x

j j

@I W @y

j j

(4)

@I @I W @y @x

6 | 10

Feature Detection - Harris Corner 3D Detector y directions about a image patch. For 3D harris corner detector we compute this in a similar way where the average is performed for image frames in temporal direction as well. Thus for a stack of frames we compute the spatical derivatives along the x and y for each accumulated frame and take the average of all the frames.

A= B= C=

W T W T W T

1 1

@IT W @x

j j

@IT W @y

j j

(5)

@IT @IT W @x @y

First we will look at computation of matrix parameter

= @t

j @I j

D= E=

@I @I @t @y @I @I @t @x

(6)

The first order temporal derivate is given by temporal kernel 12 1 and expressed as Dt (u; v; t) = I (u; v; t 1) + 2I (u; v; t) I (u; v; t + 1). The above expression gives the termporal derivative about the point (u; v; t). This is computed as average for all points in the window and over all the frames of block

= W T

jDt(u; v; t)j

We already compute spatial derivatives for each image frame and as shown earlier we can compute temporal derivative about each point in the frame.

7 | 10

D=

W T

@I @I W @t @y

E=

W T

@I @I W @t @x

The first step is to accumulate a set of frames . The number of frames required are blocks izet ime + 2 in temporal dimension. We require a temporal filter which compute the temporal derivative at each point using a temporal kernel. Thus we will have a set of images .For each image we compute the termporal derivatives and the spatial derivative at each point in the image we compute the partial products for each point in the image for all @I @I @I the image frames j @I j2 ; jfrac@I@xj2 ; j @y j2 ; @I @x ; @I @y @t @t @t We sum the results of all the image frames. We apply a box filter to result of the sum. We scale the result by temporal block size. This gives a 6D vector image which contains the parameters of the matrix at each point of the image. dd Using this we compute 1 the response function with the maximal value of k = 27 .

0.4 Output

The test input and output files are placed in the repository https: //github.com/pi19404/m19404/tree/master/FEATURE_DETECTOR The test input file is i2.avi and output file is harris_3d_demo.ogv We can observe feature are detected in region where significant motion is present and these can help in describing the motion.

8 | 10

The Harris 3D corner detector is computationally expensive. The future work include optimizing the algorithm. Inorder to have SpatioTemporal corners at a point is required that significant spatial gradients being present about the point. Thus if we make the assumption that spatial corners remain constant we can keep the same spatial gradients for all the frame being computed. The spatial gradient are computed only for the first frame . Another method may be to detect the spatial corner points and compute the existance of spatio temporal corners only at these points These methods may enable us to reduce the computation and use the harris3d corner detector for real time applications. Feature extraction temporal features. and motion classification based on spatio-

0.6 Code

we define a main feature_detector base class containing methods and data common to all feature detector. the harris_3d class is derived class containing specific implementations of algorithms. The code OpenCV code can be found in code repository https://github.com/pi19404/m19404/tree/master/FEATURE_DETECTOR or https://code.google.com/p/m19404/source/browse/FEATURE_DETECTOR/

9 | 10

Bibliography

Bibliography

[1] Chris Harris and Mike Stephens. A combined corner and edge detector. In: In

Proc. of Fourth Alvey Vision Conference.

[2]

Ivan Laptev. On Space-Time Interest Points. In: Int. J. Comput. Vision 64.2-3 (Sept. 2005), pp. 107123.

7.

[3]

1994, pp. 593 600.

Jianbo Shi and C. Tomasi. Good features to track. In: Computer Vision and

Pattern Recognition, 1994. Proceedings CVPR '94., 1994 IEEE Computer Society Conference on.

doi: 10.1109/CVPR.1994.323794.

10 | 10

- OpenCL 2D Convolution Using Separable Filters -Box FilterUploaded bypi194043
- Fast 3D Extension of Fast Feature Detection for SpatioTemporal Corner DetectionUploaded bypi194043
- polynomial approximation of a 2D signalUploaded bypi194043
- Feature Detection - Overview of Harris Corner Feature DetectionUploaded bypi194043
- Polynomial Approximation of 1D signalUploaded bypi194043
- SubPixel Corner Localization AlgorithmUploaded bypi194043
- Overview and Implementation of Fast Corner Detection MethodUploaded bypi194043
- Gaussian Multivariate Distribution -Part 1Uploaded bypi194043
- Fast Asymmetric Learning for Cascade Face Detection Training/Testing UtilityUploaded bypi194043
- OpenCL Image Convolution Filter - Box FilterUploaded bypi194043
- A linear channel filterUploaded bypi194043
- Adaptive Skin Color DetectorUploaded bypi194043
- Fast 2D Separable Symmetric/Anti-Symmmetric ConvolutionUploaded bypi194043
- Normalized convolution for image interpolationUploaded bypi194043
- OpenCL Heterogenenous program for Image Processing - ColorSpace conversion BGR-HSV,HSV-BGR,BGR-GRAYUploaded bypi194043
- temporal filtersUploaded bypi194043
- Gesture Recognition using Hidden Markov ModeUploaded bypi194043
- Android OpenCV Face detectionUploaded bypi194043
- shape classification using Histogram of oriented gradientsUploaded bypi194043
- Modified Canny Edge DetectionUploaded bypi194043
- Noise Models in Image processingUploaded bypi194043
- Image enhancement using FusionUploaded bypi194043
- Uniform Color QuantizationUploaded bypi194043
- Single Passs Connected Component LabellingUploaded bypi194043
- opencv2refmanUploaded byjayneildalal
- Control Limited Adaptive Histogram Equalization for Image EnhancementUploaded bypi194043
- Symmetic Nearest Neighbour Anisotropic 2D image filterUploaded bypi194043
- OpenCL Heterogeneous parallel program for Gaussian FilterUploaded bypi194043
- Dense optical flow expansion based on polynomial basis approximationUploaded bypi194043
- A detailed descriptions and results for different color constancy algorithmsUploaded bypi194043

- Modified Canny Edge DetectionUploaded bypi194043
- Polynomial Approximation of 2D image patch -Part 2Uploaded bypi194043
- Compiling Native C/C++ library for AndroidUploaded bypi194043
- Multi Class Logistic Regression Training and TestingUploaded bypi194043
- Gaussian Multivariate Distribution -Part 1Uploaded bypi194043
- ARM Neon Optimization for image interleaving and deinterleavingUploaded bypi194043
- A linear channel filterUploaded bypi194043
- Adaptive Skin Color DetectorUploaded bypi194043
- OpenVision Library Gaussian Mixture Model ImplementationUploaded bypi194043
- Continuous Emission Hidden Markov Model for sequence classificationUploaded bypi194043
- Fast 2D Separable Symmetric/Anti-Symmmetric ConvolutionUploaded bypi194043
- Dense optical flow expansion based on polynomial basis approximationUploaded bypi194043
- Implementation of discrete hidden markov model for sequence classification in C++ using EigenUploaded bypi194043
- Markov chain implementation in C++ using EigenUploaded bypi194043
- C++ InheritanceUploaded bypi194043
- Local Binary PatternUploaded bypi194043
- Mean Shift AlgorithmUploaded bypi194043
- C++ Const,Volatile Type QualifiersUploaded bypi194043
- Random Ferns for Patch DescriptionUploaded bypi194043
- Integral Image for Computation of Mean And VarianceUploaded bypi194043
- Embedded Systems Programming with ARM on Linux - Blinking LEDUploaded bypi194043
- C++ virtual functions and abstract classUploaded bypi194043
- C++ static members and functionUploaded bypi194043
- Uniform Local Binary Pattern and Spatial Histogram ComputationUploaded bypi194043
- Tan and Triggs Illumination normalizationUploaded bypi194043
- Normalized convolution for image interpolationUploaded bypi194043

- settopbox_userguide.pdfUploaded byaravindmay25
- Proyecto Tesis Bravo Chuica ListoUploaded byRodrigoTuse
- Dossier Inf IUploaded byCaryaxo Rama
- Love is a Caged Rat Giving Itself OrgasmsUploaded byRuth Field
- Guia de Investigação Computacional Para WindowsUploaded byperionda99
- Algebra BooleanaUploaded byJulio Luis
- LA ORGANIZACIÓN COMO SISTEMAUploaded byMilixitaELF
- Armament oUploaded byPedraza Hernandez
- Manifiesto Colectivo Sobre Las Tecnologías de La Comunicación en La EducaciónUploaded byoscar gutierrez
- Agentes Físicos y Químicos AntimicrobianosUploaded byJoaquín Chavolla
- Flujo en Conductos CerradosUploaded byFabián Ruiz Paredes
- investigacion de operaciones unidad. 4Uploaded byErika Ramirez
- Emisión de Licencias Institucionales de Seguridad RadiológicaUploaded byHaVy Carrasco Tuston
- Bernulf Kanitscheider - Universo y RealidadUploaded byAlgoritzmi
- ElectrocardiologiaUploaded byAbyura
- Th 10994Uploaded bymeganebaka
- mcallister annotated bibliography revisedUploaded byapi-341851116
- Penic IliumUploaded byMauricio Rueda Rodriguez
- Network Virtualisation Project Using Mininet PptUploaded byArjun Krishna P R
- CISE 412 Mechatronics Lab ManualUploaded bykarthisan
- Pensamiento y RazonamientoUploaded byJorgeMartinez
- curso2009_RadionicaUploaded byMariaHelena Garozzi
- Lindfield Times February 2008Uploaded byLindfield Times
- Planeación de Matemáticas.Uploaded byChavoFelix
- Les effets des champs électromagnétiques sur l’équilibre énergétique par INERISUploaded byNeptunya
- Ili-wi09 Mfl Pigging Rev AUploaded byGuneesha Singh
- Las Pilas de CombustibleUploaded byjheremi
- CAPITULO 10Uploaded byroyer
- Understanding MPLS Label Operations on EX Series Switches - Technical Documentation - Support - Juniper NetworksUploaded byNelsonbohr
- Pm Force CutUploaded byMartín Sosa