Lections on Image analysis, OpenCV

7. Working with camera. Background and motion analysis


www.uralvision.blogspot.com perevalovds@gmail.com

USU / IMM Fall 2010

Working with the camera

Getting frames from the camera
VideoCapture capture; // A class to work with cameras and videos capture.open (0); // Open the first chamber (numbered from 0) // But if you specify a string VideoCapture capture.open ("myfile.avi"); - will read the avi-file if (! capture.isOpened ()) { // If the camera does not complete the work cout <<"no camera" <<endl; return -1; }

Mat image, smoothed, edges; for (;;) { capture>> image; // get the frame

cvtColor (image, smoothed, CV_BGR2GRAY); GaussianBlur (smoothed, smoothed, Size (7,7), 1.5, 1.5); Canny (smoothed, edges, 0, 30, 3); imshow ("image", image); imshow ("edges", edges); if (waitKey (30)> = 0) break; // wait for keypress 30 ms, if pressed - exit } // Destructor capture himself off the camera return 0;

Getting frames from the camera

Camera calibration
Consider a simple calibration when a camera looks at the side of the plane. (There are more sophisticated calibration methods, which allow to eliminate the distortion at the edges of images from the camera connected with the inaccuracy of optics).

Original image

Image with the selected calibration points (Angles of the red 4-gon)

Calibration results: image is aligned

Camera Calibration
1. To calibrate the need to specify the coordinates of several points on the image and indicate to what point they should go.
Must be at least 4 points to search for perspective projection. In this case any point should not lie on one line.

Note: the coordinates are real numbers, which improves the accuracy of the result. You can use the sub-pixel methods that calculate the position of objects with an accuracy higher than pixel.
2. Then, using the getPerspectiveTransform calculate the optimal matrix of perspective transformation (a method of least squares type, which helps to use a lot of points taken from the error). 3. Finally, this matrix is passed to the functionwarpPerspective for calibrated images.

Camera Calibration
const int K = 4, // we use 4 points. cv:: Point2f src [K]; // coordinates of points on the image cv:: Point2f dst [K]; // coordinates of the resulting points
src [0] = cv:: Point2f (..., ...); // coordinates of the corners of the image src [1] = cv:: Point2f (..., ...); // should go clockwise from top left corner src [2] = cv:: Point2f (..., ...); // - in accordance with dst, see below src [3] = cv:: Point2f (..., ...); dst [0] = cv:: Point2f (0, 0) // the resulting point; w and h- The size of the result dst [1] = cv:: Point2f (w, 0); dst [2] = cv:: Point2f (w, h); dst [3] = cv:: Point2f (0, h); Mat transform = getPerspectiveTransform(Src, dst); // compute the matrix transforms. warpPerspective(Image, imageResult, transform, cv:: Size (w, h), INTER_LINEAR); // Get from the input image image result - imageResult

Methods for analysis of background

Methods for analysis of background
Problem - to calculate the pixels are the background and the background.

1. Learning the background
Idea: store the image background and then consider the difference between shots with the camera from the stored images. A more advanced method - "code book", he remembers the brightness of the background in each pixel for a while, that allows to deal with objects such as opening doors. There are also more complex algorithms.

Problems: - Rapid changes in lighting - should be specifically consider - The shadows of objects are often perceived as no background, which is usually undesirable, so also be specially taken into account (through the color analysis).

2. Adaptive learning
Simple method, where the background is averaging of the last N frames const float k = 0.01; back = (1-k) * back + k * image; a low pass filter A few seconds later the object came into the picture becomes the background. For some tasks it is good.

Plus - no need to specifically construct a situation where there is no one in the frame for storing the background.
There are more sophisticated algorithms automatically calculate the background with the help of several accounting staff.

3.Using data on the depth
1. Stereo camera 2. Camera with the calculation of flight time (based on a laser rangefinder, time of filght cameras) 3. Kinect Provide information on the distance to each pixel, allowing long-discarded objects by implementing a threshold depth of processing.

Motion Analysis

The difference between two frames the algorithm "Motion Detector"
1. Let image1, image2 - two consecutive frames from the camera, single-channel. 2. Building a module of their difference Mat diff; absdiff (image1, image2, diff); 3. Perform threshold processing Mat bin;// Here we will find the pixels that correspond to areas of motion threshold (diff, bin, 100 / * threshold * /, 255, CV_THRESH_BINARY);

The difference between two frames the algorithm "Motion Detector"
This method allows you to find the area to which there was a movement.
For the analysis directions movement of such objects - the notion of Optical Flow.

What is the optical flow
Optical Flow(Optical flow, optic flow) - is a vector field of apparent motion of objects, surfaces and edges in a visual scene, caused by relative motion between the observer (eye camera) and the stage.

Note: it seems that the point of the camera moves down, but maloteksturirovannoy of flow is not - as the local pixels in the neighborhood do not change


The main application of optical flow
1. To determine the direction in which the moving objects in the frame. 2. The production of films - for a smooth morphing between successive frames, or between the shots taken adjacent cells (the most characteristic is used in the movie "The Matrix"). 3. In principle can be applied to stereo vision - to determine the distance to an object by analyzing the optical flow of personnel coming from the two chambers.

Methods of calculating the optical flow
Input: two frames. Output - vector field (fx (x, y), fy (x, y)) - Vector of geometric shift pixels from the first frame on the second.

(I) Block ("Naive" methods)
For each point searched shift that minimizes the difference in the local window.

(II) Differential (Most used)
Estimation of the derivatives of x, y, t. 1. Lucas-Kanade - Very fast. 2.Farneback- Good enough, but slow 3.CLG - Very high quality but not yet implemented in OpenCV 4. Pyramidal Lucas-Kanade, calculated only on "points of interest" 5. Horn-Schunk - not very resistant to noise

(III) based on discrete optimization (Intensive)
The solution is constructed using the methods of min-cut, max-flow, linear programming and belief propagation.

Methods of calculating the optical flow
1. Lucas-Kanade
Local method, which uses the Taylor expansion in time t in the neighborhood of each pixel independently. Dignity: Quickly calculated. Shortcomings: "Aperture problem" - in areas with a uniform texture, works poorly due to the fact that it is local. Realized only in the C-version of OpenCV,cvCalcOpticalFlowLK.

Flow superimposed on the image. Note that the flow inside the palm is not found. Flow drawn in increments of 10 pixels, the function line. (Note: The values of the flux smoothed over the neighborhood for a more stable result.)

Methods of calculating the optical flow
2. Farneback
Local method used to calculate an approximation of the image by a polynomial function. Dignity: Significantly more resistant to the problems of the aperture. Shortcomings: Runs noticeably slower Lucas-Kanade.

Methods of calculating the optical flow
void calcOpticalFlowFarneback( const Mat & prevImg, // first frame, an 8-bit single channel image const Mat & nextImg, // second frame, the type and size as a prevImg Mat & flow, // resulting flow will be of the type CV_32FC2 double pyrScale, // <1, the scale of the construction of the pyramid.0.5 int levels, // Number of levels of the pyramid5 int winsize, // window averaging. The more - the result // More diffuse, but also more resistant to noise5 int iterations, // Number of iterations at each level of the pyramid 3 int polyN, // window size to calculate the approximation // Polynomial7 double polySigma, // parameter is Gaussian for smoothing derivatives // In the construction of approximation1.5 int flags // flags //OPTFLOW_USE_INITIAL_FLOW- Do not use!
//OPTFLOW_FARNEBACK_GAUSSIAN- Use a Gaussian for the averaging // Gives a more accurate result at the expense of quality.


Methods of calculating the optical flow
3. CLG "Combining Local and Global" The method of combining Lucas-Kanade and Horn-Schunk Now (2010), he and his modification is considered one of the best quality / speed.

Not yet implemented in OpenCV.
Difficult task (for exam): find a realization of CLG in C / C + +, compile, and run on a pair of test images.

Sign up to vote on this title
UsefulNot useful