Feature Detection - Harris Corner 3D Detector

Pi19404
February 27, 2013

Contents

Contents
Feature Detection - Harris Corner 3D Detector 3

0.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 0.2 Spatio-Temporal Features . . . . . . . . . . . . . . . 0.2.1 Model of Spatio-termporal Image Sequence . 0.3 Computation of matrix parameters . . . . . . . . . . 0.3.1 Matrix Paramter A,B,C . . . . . . . . . . . . . . . 0.3.2 Matrix parameters D,E,F . . . . . . . . . . . . 0.3.3 Implementation Details . . . . . . . . . . . . . . 0.4 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.5 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . 0.6 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

3 3 4 6 6 7 8 8 9 9 10

2 | 10

Feature Detection - Harris Corner 3D Detector

Feature Detection - Harris Corner 3D Detector

0.1 Introduction
In this article we will look at the extension of harris corner detector called harris 3D detector used to detect robust spatio temporal corners . The harris corner detection is used to detect robust spatial corners which detects points with significant local variation of image intensities. The 3D harris corner detector is used to detect spatial as well as temporal corners that can be detected over a sequence of frames.It finds spatical corners with velocity change which exhibit changes in space as well as time.

0.2 Spatio-Temporal Features

The aim of spatial feature detection is to identify point in the image that correpond to object in motion that can be tracked in adjacent image frames robustly. The aim of spatio-temporal feature detection is to identify points in the image where significant spatial and temporal intensity changes are observed.These locations provide a sparse representation of motion in the image and local neighborhood of these points can be used to represent motion occuring in the image. It is required that the feature be as sparse as possible to reduce the computational requirement and enable us to achieve real time tracking.

3 | 10

Feature Detection - Harris Corner 3D Detector Potential candidates for Spatial interest points are edge pixels,points with significant changes in the neighborhood and junction of corners. Potential candidates for Spatio-Temporal interest points are points which exhibit large variation in spatial as well as temporal dimensions.

0.2.1 Model of Spatio-termporal Image Sequence

Consider a Image sequence represented by I (x; y; t) . In harris corner detector we considered a small window and analyzed the change in average intensity of as window is shifted by small ammount.If the change is large of all the shifts we considered the point as a interest corner. For spatio-temporal corner we will consider a 3D window about a point and analyze the average intensity change as window is shifted by small ammount in spatial as well as temporal dimension.If the change is large for all spatial as well as temporal shifts we would consider the point as spatio-temporal interest corner.

The point being considered is I (u; v; t) .x,t, are spatio-temporal shifts.Thus we accumulate frames at different time instants.We compute average change in intensity of frame at instant t with shifted windows at time instants t 1; t + 1.

E (x; y;  ) =

jI (x + u; y + v; t +  ) I (u; v; t)j

(1)

4 | 10

Feature Detection - Harris Corner 3D Detector Consider First order approximation of

I (x + u; y + v; t +  )

@I @I I (x + u; y + v; t +  ) = I (u; v; t) + x @x + y @y +  @I @t

E (x; y;  ) = E (x; y;  ) = E (x; y;  ) =

W W W

@I @I jx @x + y @y +  @I j @t

jxX + yY + T j

x2 X 2 + y2 Y 2 + T 2 + 2xyXY
W

+2

yY T + 2xXT

E (x; y;  ) = x2 W X 2 + y2 +2y W Y T + 2x W XT

P

Y2+

 2 T 2 + 2xy

XY

(2)

E (x; y;  ) = Ax2 + By2 + 2Cxy + 2yD + 2xE +  2 F x A C E E (x; y;  ) = x y  RC B DS Ry S E D F  The shifts considered are (0; 1; 0); (1; 1; 0); (0; 1; 0); (1; 1; 0); (0; 1; 1); (1; 1; 1) (0; 1; 1); (1; 1; 1); (0; 1; 1); (1; 1; 1); (0; 1; 1); (1; 1; 1).
These shifts correspond to spatial derivatives in x and y direction , spatial derivatives wrt termporal windows . In Harris 2D detector we used a 2D window ,similarily in 3D harris detector we use a 3D window. The block size represented the size of 2D window in harris corner detector. For 3D harris corner detector we would have to store the image frames corresponding to block size.Large the block size in temporal dimension longer is observation interval for which change in spatial interest points are observed. Again the eigen values of matrix are analysed and points which provide all three large eigen values indicates a corner points. A similar measure to 2D harris corner detector can be used.
QP Q

5 | 10

Feature Detection - Harris Corner 3D Detector

R = Det(M ) kT r(M )2 = 1 2 3 + k(1 + 2 + 3 )3 2  = 1  = 3 1 H = 3 ( k(1 +  +  )3 ) 1 For H ! 0; we get k (1+ )3 + As k increases;  3 1

Positive local maxima of H corresponds to spatia-temporal corners 1 Maximal values of k is 27

(3)

Thus we take maximal value of k and compute the response function for each point .If the response function is above a certian threshold we can classify it as a spatio-temporal corner.

0.3 Computation of matrix parameters

If the point being being analyzed is I(u,v,t) then we require image blocks I (u bs; v bs) to I (u + bs; v + by ) for all image frames from t bst to t + bst ie a 3D block about the point u; v; t. To accomodate for the first order spatial and temporal shifts we require image frames in time interval t bst 1 to t + bst + 1 about the point u; v with a patch size of bs. Thus first step in the program is to accumulate frames frames for anaysis.

bst + 2

0.3.1 Matrix Paramter A,B,C

For harris corner detector we computed this as

A= B= C=

w w w
1 1

@I W @x

j j

@I W @y

j j

(4)

@I @I W @y @x

The average magnitude of derivative along the x direction and

6 | 10

Feature Detection - Harris Corner 3D Detector y directions about a image patch. For 3D harris corner detector we compute this in a similar way where the average is performed for image frames in temporal direction as well. Thus for a stack of frames we compute the spatical derivatives along the x and y for each accumulated frame and take the average of all the frames.

A= B= C=

W T W T W T
1 1

@IT W @x

j j

@IT W @y

j j

(5)

@IT @IT W @x @y

0.3.2 Matrix parameters D,E,F

First we will look at computation of matrix parameter

= @t

j @I j

D= E=

@I @I @t @y @I @I @t @x

(6)

The first order temporal derivate is given by temporal kernel 12 1 and expressed as Dt (u; v; t) = I (u; v; t 1) + 2I (u; v; t) I (u; v; t + 1). The above expression gives the termporal derivative about the point (u; v; t). This is computed as average for all points in the window and over all the frames of block

= W T

jDt(u; v; t)j

We already compute spatial derivatives for each image frame and as shown earlier we can compute temporal derivative about each point in the frame.

7 | 10

Feature Detection - Harris Corner 3D Detector

D=

W T

@I @I W @t @y

E=

W T

@I @I W @t @x

0.3.3 Implementation Details

The first step is to accumulate a set of frames . The number of frames required are blocks izet ime + 2 in temporal dimension. We require a temporal filter which compute the temporal derivative at each point using a temporal kernel. Thus we will have a set of images .For each image we compute the termporal derivatives and the spatial derivative at each point in the image we compute the partial products for each point in the image for all @I @I @I the image frames j @I j2 ; jfrac@I@xj2 ; j @y j2 ; @I @x ; @I @y @t @t @t We sum the results of all the image frames. We apply a box filter to result of the sum. We scale the result by temporal block size. This gives a 6D vector image which contains the parameters of the matrix at each point of the image. dd Using this we compute 1 the response function with the maximal value of k = 27 .

0.4 Output
The test input and output files are placed in the repository https: //github.com/pi19404/m19404/tree/master/FEATURE_DETECTOR The test input file is i2.avi and output file is harris_3d_demo.ogv We can observe feature are detected in region where significant motion is present and these can help in describing the motion.

8 | 10

Feature Detection - Harris Corner 3D Detector

0.5 Future Work

The Harris 3D corner detector is computationally expensive. The future work include optimizing the algorithm. Inorder to have SpatioTemporal corners at a point is required that significant spatial gradients being present about the point. Thus if we make the assumption that spatial corners remain constant we can keep the same spatial gradients for all the frame being computed. The spatial gradient are computed only for the first frame . Another method may be to detect the spatial corner points and compute the existance of spatio temporal corners only at these points These methods may enable us to reduce the computation and use the harris3d corner detector for real time applications. Feature extraction temporal features. and motion classification based on spatio-

0.6 Code
we define a main feature_detector base class containing methods and data common to all feature detector. the harris_3d class is derived class containing specific implementations of algorithms. The code OpenCV code can be found in code repository https://github.com/pi19404/m19404/tree/master/FEATURE_DETECTOR or https://code.google.com/p/m19404/source/browse/FEATURE_DETECTOR/

9 | 10

Bibliography

Bibliography
[1] Chris Harris and Mike Stephens.  A combined corner and edge detector. In: In
Proc. of Fourth Alvey Vision Conference.

1988, pp. 147151.

[2]

Ivan Laptev.  On Space-Time Interest Points. In: Int. J. Comput. Vision 64.2-3 (Sept. 2005), pp. 107123.

7.
[3]

issn: 0920-5691. doi: 10.1007/s11263-005-1838url: http://dx.doi.org/10.1007/s11263-005-1838-7.

1994, pp. 593 600.

Jianbo Shi and C. Tomasi.  Good features to track. In: Computer Vision and
Pattern Recognition, 1994. Proceedings CVPR '94., 1994 IEEE Computer Society Conference on.

doi: 10.1109/CVPR.1994.323794.

10 | 10