Introduction to OpenCV

Vadim Pisarevsky
Senior Software Engineer
Intel Corporation, Software and Solutions Group

2/67
Agenda
General Information
Getting Started with OpenCV
Modules Description
Interaction with Intel IPP
Python Interface
Some Usage Examples
Summary

3/67
General Information

4/67
What is OpenCV?
OpenCV stands for Open Source Computer Vision Library
Includes over 500 functions implementing computer vision, image
processing and general-purpose numeric algorithms.
Portable and very efficient (implemented in C/C++)
Has BSD-like license (that is, absolutely free for academic and
commercial use)
Available at http://sourceforge.net/projects/opencvlibrary

5/67
Why OpenCV?
Computer Vision Market is large and continues to grow
There is no standard API (like OpenGL and DirectX in graphics, or
OpenSSL in cryptography), most of CV software is of 3 kinds:

Research code (slow, unstable, independent/incompatible data types for
every library/toolbox)

Very expensive commercial toolkits (like Halcon, MATLAB+Simulink, …)

Specialized solutions bundled with hardware (Video surveillance,
Manufacturing control systems, Medical equipment …)
Standard library would simplify development of new applications and
solutions much easier.
Special optimization for Intel Architectures:

Creates new usage models by achieving real-time performance for quite
“heavy” algorithms (like face detection)

Makes Intel platforms attractive for CV developers

6/67
The Project History
“CVL” project was started; the main
goals/features:
• Human-Computer interface is a
main target
• Real-Time Computer Vision Library
for using by UI Developers,
Videoconferences, Games
• Highly optimized for Intel Arch.
1999/01
First Open
Source Release:
OpenCV alpha 3
at CVPR’00
2000/06 2000/12
Continuous development and
various research projects
OpenCV beta 1
with Linux
support:
CVPR’01
2006/10
OpenCV 1.0 with
MacOSX
support

7/67
OpenCV Community
The library is actively used by a large number of companies (such as
Intel, IBM, Microsoft, SONY, Siemens, Google,…) and research centers
(Stanford, MIT, CMU, Cambridge, INRIA etc.)
>14000 members of the forum OpenCV@yahoogroups.com, with
average daily traffic ~10-20 messages.
Community contributes to the project: bug reports, patches, new
features (video acquisition, 3d tracking, textures, Python interface)

8/67
Supported Platforms
 (iMac G5, GCC,
native APIs)
N/A ? (not tested)  (w. IPP, GCC,
native APIs)
MacOSX
Reported to build
on UltraSparc,
Solaris
  
Others
(BSD,
Solaris…)

 (GCC, ICC)  (w. IPP; GCC,
ICC)
 (w. IPP; GCC,
ICC)
Linux
N/A Partial
Support
(w. IPP,
VS98+PSDK,
VS2005(OpenMP))
 (w. IPP;
VS98,VS2005(Op
enMP),ICC,GCC,
BCC)
Windows
Other (PPC,
Sparc)
IA64
(Itanium)
EM64T (x64) IA32 (x86)

9/67
Library Design
Initial goal: build high-level ready-to-use components, like gesture
recognition, camera calibration etc.
But … we have fundamental Computer Vision problem: can not create
CV system that works always and everywhere (because of diff. lighting
conditions, subject variety …) => need to analyze each problem.
“Analysis”: 1581, "resolution of anything complex into simple
elements"
We split a complex problem into elements/building blocks. Each block
may require further decomposition. Then for every block (compound
or primitive) we either use existing functions or create our own =>
we finally got multi-level library, which components may be re-used

10/67
Example: Gesture Recognition
How to find hand?
• By color
• Using motion detection
• Depth information (w. stereo camera)

How to extract the shape?
What are the lighting conditions?
Dynamic or static gestures?
Pre-defined set of gestures
or extendable by user?
+
?
Depth Map with
Stereo Camera
3D Clustering → 2D
Color Histograms
Statistical Classification
(Image moments as features)
One of
algorithms:
So for this algorithm we need
stereo,
3D clustering, color histograms,
image moments,
statistical classification
(or just Mahalanobis distance)
and the way to visualize all the
algorithm stages for debugging
no free libraries could do
all/most of this! => We had to
do it ourselves

11/67
Library Architecture
CV + CVAUX
High-level components
(face detection,
camera calibration),
image processing,
motion analysis …
HighGUI
Simple GUI,
Image and
Video Input/Output
MLL
Statistical Classifiers,
Clustering Algorithms
CXCORE
Basic structures and algorithms (matrix and math operations),
XML support, drawing functions
Samples/Demos
IPP
Optimized code for Intel CPUs

12/67
Getting Started with OpenCV

13/67
The First OpenCV Program
1. #include <cv.h>
2. #include <highgui.h>
3. #include <math.h>
4. int main( int argc, char** argv ) {
5. CvPoint center;
6. double scale=-3;
7. IplImage* image = argc==2 ? cvLoadImage(argv[1]) : 0;
8. if(!image) return -1;
9. center = cvPoint(image->width/2,image->height/2);
10. for(int i=0;i<image->height;i++)
11. for(int j=0;j<image->width;j++) {
12. double dx=(double)(j-center.x)/center.x;
13. double dy=(double)(i-center.y)/center.y;
14. double weight=exp((dx*dx+dy*dy)*scale);
15. uchar* ptr =
&CV_IMAGE_ELEM(image,uchar,i,j*3);
16. ptr[0] = cvRound(ptr[0]*weight);
17. ptr[1] = cvRound(ptr[1]*weight);
18. ptr[2] = cvRound(ptr[2]*weight); }
19. cvSaveImage( “copy.png”, image );
20. cvNamedWindow( "test", 1 );
21. cvShowImage( "test", image );
22. cvWaitKey();
23. return 0; }
Radial gradient

14/67
How to build and run the program?
Obtain the latest version of OpenCV:
a) Get the stable release (currently, 1.0) at http://
www.sourceforge.net/projects/opencvlibrary
b) Or get the latest snapshot from CVS:
:pserver:anonymous@opencvlibrary.cvs.sourceforge.net:/cvsroot/opencvlibrary
Install it:
Windows: run installer, open opencv.sln (opencv.dsw) with Visual Studio and
build the solution
Linux/MacOSX: use ./configure + make + make install (see the INSTALL
document)
Build the sample:
Windows:
– cl /I<opencv_inc> sample.cpp /link /libpath:<opencv_lib_path> cxcore.lib
cv.lib highgui.lib
– Create project for VS (see opencv/docs/faq.htm)
– or use opencv/samples/c/cvsample.vcproj (cvsample.dsp) as starting point
Linux:
g++ -o sample `pkg-config –cflags` sample.cpp `pkg-config –libs opencv`
Run it: sample lena.jpg

15/67
The Program Analysis Line-by-Line
1-23: The code is clear, short and
cross-platform, no need to use with
MFC/GTK/QT/…, cross-platform
1-2. Include OpenCV headers first
7. IplImage is OpenCV image type
7, 19. Use cvLoadImage and
cvSaveImage to load/save an image
in multiple formats: JPEG,
JPEG2000, BMP, PNG, TIFF, PPM.
9. Use cvPoint and other
“constructor” functions to initialize
simple structures on fly.
15. Use CV_IMAGE_ELEM macro to
access image pixels
16-18. Use cvRound for very fast
float->int conversion.
20. Use cvNamedWindow to create
self-updating highgui window
21. Use cvShowImage to display
image in window
22. Use cvWaitKey to get input from
user
1. #include <cv.h>
2. #include <highgui.h>
3. #include <math.h>
4. int main( int argc, char** argv ) {
5. CvPoint center;
6. double scale=-3;
7. IplImage* image = argc==2 ? cvLoadImage(argv[1]) : 0;
8. if(!image) return -1;
9. center = cvPoint(image->width/2,image->height/2);
10. for(int i=0;i<image->height;i++)
11. for(int j=0;j<image->width;j++) {
12. double dx=(double)(j-center.x)/center.x;
13. double dy=(double)(i-center.y)/center.y;
14. double weight=exp((dx*dx+dy*dy)*scale);
15. uchar* ptr =
&CV_IMAGE_ELEM(image,uchar,i,j*3);
16. ptr[0] = cvRound(ptr[0]*weight);
17. ptr[1] = cvRound(ptr[1]*weight);
18. ptr[2] = cvRound(ptr[2]*weight); }
19. cvSaveImage( “copy.png”, image );
20. cvNamedWindow( "test", 1 );
21. cvShowImage( "test", image );
22. cvWaitKey();
23. return 0; }

16/67
Modules Descriptions:
HighGUI

17/67
Functionality Overview
“Smart” windows
Image I/O, rendering
Processing keyboard and other events, timeouts
Trackbars
Mouse callbacks
Video I/O

18/67
Windows
cvNamedWindow(window_name, fixed_size_flag);
creates window accessed by its name. Window handles repaint, resize events.
Its position is remembered in registry:
cvNamedWindow(“ViewA”,1);
cvMoveWindow(“ViewA”,300,100);
cvDestroyWindow(“ViewA”);

cvShowImage(window_name, image);
copies the image to window buffer, then repaints it when necessary. {8u|16s|
32s|32f}{C1|3|4} are supported.
only the whole window contents can be modified. Dynamic updates of parts of the
window are done using operations on images, drawing functions etc.
On Windows native Win32 UI API is used
On Linux – GTK+ 2.x
On MacOSX – Carbon.

19/67
Image I/O
IplImage* cvLoadImage(filename, colorness_flag);
loads image from file, converts it to color or grayscale, if needed.
image format is determined by the file contents.
cvSaveImage(filename, image);
saves image to file, image format is defined by filename extension.
Supported formats: BMP, JPEG (libjpeg), JPEG 2000 (Jasper), PNG (libpng), TIFF
(libtiff), PPM/PGM.
// converting jpeg to png
IplImage* img = cvLoadImage(“picture.jpeg”,-1);
if( img ) cvSaveImage( “picture.png”, img );

20/67
Waiting for keys…
cvWaitKey(delay=0);
waits for key press event for <delay> ms or infinitely, if delay is 0.
To make program continue execution if no user actions is taken, use
cvWaitKey(<delay_ms!=0>); and check the return value
// opencv/samples/c/delaunay.c

for(…) {

int c = cvWaitKey(100);
if( c >= 0 )
// key_pressed
break;
}

21/67
Trackbars & Mouse Callbacks
// opencv/samples/c/morphology.c
int dilate_pos=0; // initial position value
void Dilate(int pos) {
… cvShowImage( “E&D”, dilate_result );
}
int main(){
cvNamedWindow(“E&D”,1);
cvCreateTrackbar(“Dilate”,”E&D”, &dilate_pos,10,Dilate);

cvWaitKey(0); // check for events & process them
}
// opencv/samples/c/lkdemo.c
void on_mouse(int event,int x,int y,int flags, void* param) { … }
int main(){
cvNamedWindow(“LkDemo”,1);
cvSetMouseCallback(“LkDemo”,on_mouse,0);

cvWaitKey(0); // check for events & process them
}

22/67
Video I/O API
CvCapture* cvCaptureFromCAM(camera_id=0);
initializes capturing from the specified camera
CvCapture* cvCaptureFromFile(videofile_path);
initializes capturing from the video file.
IplImage* cvQueryFrame(capture);
retrieves the next video frame (do not alter the result!), or NULL if there is
no more frames or an error occured.
cvGetCaptureProperty(capture, property_id);
cvSetCaptureProperty(capture, property_id, value);
retrieves/sets some capturing properties (camera resolution, position within
video file etc.)
cvReleaseCapture(&capture);
do not forget to release the resouces at the end!
Used interfaces:
• Windows: VFW, IEEE1394, MIL, DirectShow (in progress), Quicktime (in
progress)
• Linux: V4L2, IEEE1394, FFMPEG
• MacOSX: FFMPEG, Quicktime

23/67
Video I/O Sample
// opencv/samples/c/lkdemo.c
int main(…){

CvCapture* capture = <…> ?
cvCaptureFromCAM(camera_id) :
cvCaptureFromFile(path);
if( !capture ) return -1;
for(;;) {
IplImage* frame=cvQueryFrame(capture);
if(!frame) break;
// … copy and process image
cvShowImage( “LkDemo”, result );
c=cvWaitKey(30); // run at ~20-30fps speed
if(c >= 0) {
// process key
}}
cvReleaseCapture(&capture);}
opencv/samples/c/lkdemo.c:
(needs camera to run)

24/67
Modules Descriptions:
CXCORE

25/67
Functionality Overview
Matrix and Image Arithmetical and Logical Operations
Linear algebra, DFT/DCT, vector math
Dynamic structures
Drawing functions
XML I/O

26/67
Matrix (CvMat)
CvMat – fundamental 2D array type in
OpenCV:
typedef struct CvMat {
int type; // data type: CV_8UC1, CV_8UC3, …
// … CV_16UC1, …, CV_64FC1
int step;
union { // aliased pointers to the data
unsigned char* ptr;
float* fl; double* db;
} data;
int rows; // ~width
int cols; // ~height

} CvMat;
float A[3][3] = {{1, 0, 0}, {0, 1, 0}, {0, 0, 1}};
CvMat mA = cvMat( 3, 3, CV_32F, &A[0][0] ); // initialize 3x3 identity matrix
printf( “%g\n”, cvDet(&mA) ); // will print 1.0
]
]
]
]
]

1 0 0
0 1 0
0 0 1
// create large color image as a matrix
CvMat* image = cvCreateMat( 1024, 768, CV_8UC3 );
// fill the image with red color (blue=0, green=0, red = 255)
cvSet( image, cvScalar(0,0,255));
step
height
width
Pixels/Elements
Data pointer

27/67
Image (IplImage)
It came from the older Intel Image Processing Library (IPL).
typedef struct _IplImage {
int nChannels; // 1, 2, 3, 4 …
int depth; // 8, 16, 32, 64, signed/unsigned/floating-point
int width, height;
int widthStep; // step
char* imageData; // pointer to the data

} IplImage;
// create color image
IplImage* image = cvCreateImage( cvSize(1024,768), 8, 3 );
// fill it with random values (color noise)
CvRNG rng = cvRNG(0x12345);
cvRandArr( &rng, image, CV_RAND_UNI, cvScalar(0,0,0),
cvScalar(256,256,256));
Almost every OpenCV function that takes CvMat* accepts IplImage*,
and vice versa, so you may use any (but CvMat* is preferable)

28/67
Basic operations on Matrixes/Images
cvReleaseMat(&A); // release matrix
CvMat* B = cvCloneMatrix(A); // make a copy of matrix
cvCopy( A, B ); // copy contents of A to B
cvSet(A, cvScalar(a,b,c,d)); cvZero(A); // initialize matrix
// Manipulating part of a matrix (no data is copied)
CvMat B; cvGetSubArr(A,&B,cvRect(3,1,2,5));
// Split matrix into a separate channels,
merge channels together
CvMat* R = cvCreateMat(RGB->rows,RGB->cols, CV_8UC1),
*G = cvCloneMat(R), *B=cvCloneMat(G);
cvSplit(RGB, B, G, R); … cvMerge(B, G, R, RGB);
// Accessing individual matrix elements
for(i=0;i<B->rows;i++) for(j=0;j<B->cols;j++) // transpose A
CV_MAT_ELEM(*B,float,i,j)=CV_MAT_ELEM(*A,float,j,i);
height
width
Data pointer
step

29/67
Arithmetical and Logical Operations
CvMat *A, *B, *C;
CvScalar S=cvScalar(a,b,c,d);
cvAdd(A, B, C); // C = A + B
cvSub(A, B, C); // C = A – B
cvAddS(A, S, C); // C = A + S
cvSubRS(A, S, C); // C = S – A
cvMul(A, B, C, a); // C = a*A*B
cvDiv(A, B, C, a); // C = a*A/B
// C = a*A + b*B + c
cvAddWeighted( A, a, B, b, c, C);
cvCmp(A, B, C, op); // C = A op B,
// op is =, ≠, <, ≤, > or ≥
cvCmpS(A, a, C, op); // C = A op a
cvMin(A, B, C); // C = min(A, B)
cvMax(A, B, C); // C = max(A, B)
cvMinS(A, a, C); // C = min(A, a);
cvMaxS(A, a, C); // C = max(A, a);
cvAnd(A, B, C); // C = A & B
cvAndS(A, a, C); // C = A & a
… cvOr, cvOrS, cvXor, cvXorS,
cvNot(A, C); // C = ~A
cvConvertScale(A,C,a,b); // C = A*a + b (A and C may
have different types!)
S = cvSum(A); // S = ∑x,y A(x,y)
S = cvAvg(A); // S = mean(A)
cvAvgStdDev(A,&m,&sigma); // compute mean
// and standard deviation
a=cvNorm(A,B,norm); // a=||A-B||norm,
// norm is L1,L2,L∞
cvMinMaxLoc(A,&m,&M,&mpos,&Mpos); // find
// global minimum, maximum and their positions
And there is a lot more!
See opencv/docs/ref/opencvref_cxcore.htm

30/67
Linear Algebra, Math Functions
cvGEMM(A,B,a,C,b,D,flags); // generalized matrix product: D=a*A
op(A)
*B
op(B)
+
b*C
op(C)
cvSolve(A,b,x,method); // solve Ax=b (linear system or least squares problem)
cvSVD, cvSVDBkSb // singular value decomposition and back substitution
// some other functions to look at
cvDet, cvTrace, cvInvert, cvDotProduct, cvCrossProduct, cvEigenVV
cvDFT(A,B,flags); cvDCT(A,B,flags) // forward/inverse discrete Fourier and
Cosine transforms
// fast vector transcendental functions (5-20x faster than exp(),log() etc.!)
cvExp(A,B); cvLog(A,B); cvPow(A,B,a); /* B=pow(A,a); */
cvCartToPolar(X,Y,mag,phase); cvPolarToCart(mag,phase,X,Y);

31/67
Dynamic Structures
(when 2d array is not enough)
The sample task: collect the positions of all non-zero pixels in the image.
Where to store the coordinates?
Possible solutions:

Allocate array of maximum size (2*image_width*image_height – a huge value)

Use two-pass algorithm

Or use a list or similar data structure
// create a sequence of non-zero pixel positions
CvSeq* get_non_zeros( const IplImage* img, CvMemStorage* storage )
{
CvSeq* seq = cvCreateSeq( CV_32SC2, sizeof(CvSeq), sizeof(CvPoint), storage );
for( int i = 0; i < img->height; i++ ) for( int j = 0; j < img->width; j++ )
if( CV_IMAGE_ELEM(img,uchar,i,j) )
{ CvPoint pt={j,i}; cvSeqPush( seq, &pt ); }
return seq;
}

32/67
Memory Storages
Memory storage is a linked list of memory blocks.
It acts like stack:
new data is allocated always at the end
data is not deallocated until storage is deleted or
cleared.
Key functions:
cvCreateMemStorage(block_size),
cvReleaseMemStorage(storage);
cvClearMemStorage(storage);
cvMemStorageAlloc(storage,size);
• In video/signal processing pipelines cvClearMemStorage is usually called in top-level loop.
• Good for multi-threaded applications
• Example: one of audio processing pipelines became several times faster when new/delete
were replaced with CvMemStorage allocation functions.
• All OpenCV “dynamic structures” reside in memory storages

33/67
Sequences
CvSeq is a linked list of continuous blocks.
Large sequences may take multiple storage blocks, small sequences can be stored together in a single storage
block (e.g. contours).
Sequence is deque: adding/removing elements to/from either end is O(1) operation,
inserting/removing elements from the middle is O(N).
Accessing arbitrary element is ~O(1)
Sequences can be sequentially read/written using readers/writers (similar to STL’s iterators)
Sequences can not be released, only the whole storage can be deleted or cleared.
Key functions:
cvCreateSeq, cvClearSeq,
cvSeqPush, cvSeqPop, cvSetPushFront, cvSeqPopFront
cvGetSeqElem,
cvStartReadSeq, cvStartWriteSeq

34/67
Sets and Sparse Matrices
CvSet is unordered collection of elements, adding/removing elements is O(1)
operation.
Sparse matrix in OpenCV uses CvSet + hash table to storing the elements.
// Collecting the image colors
CvSparseMat* get_color_map( const IplImage* img ) {
int dims[] = { 256, 256, 256 };
CvSparseMat* cmap = cvCreateSparseMat(3, dims, CV_32SC1);
for( int i = 0; i < img->height; i++ ) for( int j = 0; j < img->width; j++ )
{ uchar* ptr=&CV_IMAGE_ELEM(img,uchar,i,j*3);
int idx[] = {ptr[0],ptr[1],ptr[2]};
((int*)cvPtrND(cmap,idx))[0]++; }
// print the map
CvSparseMatIterator it;
for(CvSparseNode *node = cvInitSparseMatIterator( mat, &iterator );
node != 0; node = cvGetNextSparseNode( &iterator )) {
int* idx = CV_NODE_IDX(cmap,node); int count=*(int*)CV_NODE_VAL(cmap,idx);
printf( “(b=%d,g=%d,r=%d): %d\n”, idx[0], idx[1], idx[2], count ); }
return cmap; }

35/67
Drawing Operations
It is possible to draw different shapes in an image (IplImage/CvMat):

Lines, Circles, Ellipses, Elliptic Arcs

Filled polygons or polygonal contours

Text (using one of embedded fonts)
Everything can be drawn with different colors, different line width, with or without antialiasing
Arbitrary images types are supported
All the rendering is done in memory, use cvShowimage to see the results immediately, use
cvSaveImage to see the results later using external image viewer
Using CXCORE + HighGUI you can visualize everything,
from object tracking and vector fields to color histograms etc.
opencv/samples/c/drawing.cpp:

36/67
XML I/O: save your data
Need a config file?
Want to save results of your program to pass it to another one, or look at it
another day?
Use cvSave, cvLoad and other XML I/O functions from cxcore.
// somewhere deep in your code… you get 5x5
// matrix that you’d want to save
{ CvMat A = cvMat( 5, 5, CV_32F, the_matrix_data );
cvSave( “my_matrix.xml”, &A );
}
// to load it then in some other program use …
CvMat* A1 = (CvMat*)cvLoad( “my_matrix.xml” );
<?xml version="1.0"?>
<opencv_storage>
<my_matrix type_id="opencv-matrix">
<rows>5</rows>
<cols>5</cols>
<dt>f</dt>
<data>
1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.
0. 1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.
0. 1.</data></my_matrix>
</opencv_storage>
my_matrix.xml

37/67
So what about config file?
CvFileStorage* fs = cvOpenFileStorage(“cfg.xml”,
0, CV_STORAGE_WRITE);
cvWriteInt( fs, “frame_count”, 10 );
cvWriteStartWriteStruct( fs, “frame_size”,
CV_NODE_SEQ);
cvWriteInt( fs, 0, 320 );
cvWriteint( fs, 0, 200 );
cvEndWriteStruct(fs);
cvWrite( fs, “color_cvt_matrix”, cmatrix );
cvReleaseFileStorage( &fs );
Writing config file
CvFileStorage* fs = cvOpenFileStorage(“cfg.xml”, 0, CV_STORAGE_READ);
int frame_count = cvReadIntByName( fs, 0, “frame_count”, 10 /* default value */ );
CvSeq* s = cvGetFileNodeByName(fs,0,”frame_size”)->data.seq;
int frame_width = cvReadInt( (CvFileNode*)cvGetSeqElem(s,0) );
int frame_height = cvReadInt( (CvFileNode*)cvGetSeqElem(s,1) );
CvMat* color_cvt_matrix = (CvMat*)cvRead(fs,0,”color_cvt_matrix”);
cvReleaseFileStorage( &fs );
Reading config file
<?xml version="1.0"?>
<opencv_storage>
<frame_count>10</frame_count>
<frame_size>320 200</frame_size>
<color_cvt_matrix type_id="opencv-
matrix">
<rows>3</rows> <cols>3</cols>
<dt>f</dt>
<data>…</data></color_cvt_matrix>
</opencv_storage>
cfg.xml

38/67
Modules Descriptions: CV
and CVAUX

39/67
Functionality Overview
Basic Image Processing: Image filters, Morphology, Pyramids, Color space
conversions, Geometrical transformations, Histograms.
Advanced Image Processing and Feature extraction: Corner detection, Canny
edge detector, Hough transform, Distance transform, Watershed, Inpainting,

Shape analysis and Computational geometry: Image moments, contours,
Delaunay triangulation and Voronoi Tesselation
Motion Analysis: Optical flow, Object tracking (Meanshift & CAMSHIFT),
Motion templates, Kalman filters
Camera calibration, Epipolar geometry
Object detection (Haar classifier)
Advanced Blob tracking (aka Video Surveillance module)
...

40/67
Basic Image Processing: Gaussian Pyramids
cvPyrDown(A,B); A – W×H, B – W/2×H/2
cvPyrUp(B,A1); B – W×H, A – 2W×2H

41/67
Basic Image Processing: Morphology
Opening: C=AoB= (AΘB)⊕B
Dilation: C=A⊕B
Erosion: C=AΘB Image A
Closing: C=A•B= (A⊕B)ΘB C=Grad(A)=(A⊕B)-(AΘB)
Compound operations: cvMorphologyEx(A,C,B,op);
Primitive operations: cvErode(A,C,B); cvDilate(A,C,B);
Erosion: C(x,y)=min
B(x’,y’)=1
A(x+x’,y+y’)
Dilation: C(x,y)=max
B(x’,y’)=1
A(x+x’,y+y’)

42/67
Basic Image Processing:
Some other important functions
cvCvtColor(bgr_image, grayscale_image, CV_BGR2GRAY);
cvCvtColor(lab_image, bgr_image, CV_Lab2BGR);
Color space conversion: cvCvtColor(A,B,color_conversion_code);
cvSmooth(image, image, CV_GAUSSIAN, 7, 7, 1., 1.); // inplace Gaussian smoothing
// with 7x7 kernel and σ=1.
Smoothing image using different methods: cvSmooth(A,B,method,parameters…);
cvSobel(A, Ax, 1, 0, 3); // compute dA/dx using 3x3 Sobel kernel:
Computing spatial image derivatives: cvSobel(A,B,dx,dy,kernel_size);
]
]
]
]
]




1 0 1
2 0 2
1 0 1
CvMat* kernel = cvCreateMat(11,11,CV_32F); cvSet(kernel,cvScalar(1./(kernel->rows*kernel->cols)));
cvFilter2D(A, B, kernel, cvPoint(kernel->cols/2,kernel->rows/2)); // slower alternative to
cvSmooth(…, CV_BLUR,…)
Image convolution with arbitrary kernel: cvFilter2D(A,B,kernel,anchor_point);
Computing image histogram: cvCalcArrHist(image_planes,hist);
int n = 256; CvHistogram* hist = cvCreateHist(1,&n,CV_HIST_ARRAY);
cvCalcArrHist((CvArr**)&grayscale_image, hist); // compute histogram for grayscale image

43/67
Feature Extraction: Canny Edge Detector
Computing binary edge map from grayscale image: cvCanny(A,B,t
1
,t
2
);
Assisting object segmentation Accelerating face detection

44/67
Feature Extraction: Hough Transform
Line Detection: cvHoughLines(image,
lines_buffer, params)
Circle Detection: cvHoughCircles(image,
centers&radiuses)
Vanishing points estimation

45/67
Advanced Image Processing:
Extended Distance Transform and Watershed
Extended distance transform
(O(N) Voronoi diagram):
cvDistTransform(A, distances,…, labels);
Watershed marker-based segmentation:
cvWatershed(A, markers);

46/67
Advanced Image Processing:
Inpainting
Reconstruction of marked areas using the neighbor pixels:
cvInpaint( image, mask );

47/67
Contours
Contour representation:
In OpenCV contours are extracted from black-
n-white images. Outer or internal boundaries of
white areas are stored as polygons:
Grayscale image:
180000 pixels
Contours:
<1800 points
Approximated Contours:
<180 points
cvCmpS(grayscale, T, bw, CV_CMP_GT);
cvFindContours(bw, storage, &contours);
contours=cvApproxPoly(contours,params);

48/67
Motion Analysis and Object Tracking
Feature tracking: cvCalcOpticalFlowPyrLK
Color object tracking: cvCamShift
Motion templates: cvUpdateMotionHistory, cvCalcMotionGradient,
cvCalcGlobalOrientation, cvSegmentMotion

49/67
Camera Calibration
cvFindChessboardCorners
[ ]
¹
¹
¹
'
¹
¹
¹
¹
'
¹
]
]
]
]
]

2 1 2 1
, , , ,
1 0 0
0
0
p p k k c f
c f
y y
x x
cvCalibrateCamera2
Camera matrix + lens distortion coefficients
cvUndistort2,
cvFindExtrinsicCameraParams2
See opencv/samples/c/calibration.cpp

50/67
Face/Object Detection
“objects” “non-objects”
opencv/apps/haartraining
cvLoad, cvDetectHaarObjects

51/67
Video Surveillance
Blob
Entering
Detection
Blob
Tracking
Trajectory
Generation
FG/BG
Detection
Video frames
Object ID’s and
trajectories
For details see opencv/docs/vidsurv,
opencv/samples/c/blobtrack.cpp and
opencv/cvaux/include/cvvidsurv.hpp

52/67
Modules Descrpitions: MLL

53/67
Functionality Overview
Statistical classification
Regression
Clustering
Algorithms: SVM, kNN, Normal
Bayesian classifier, Neural Nets
(MLP), Decision Trees, Random
Trees, Boosting, EM.

54/67
MLL Usage: SVM
// … prepare matrix of training feature vectors, and store them as rows of matrix “data”,
// and prepare the vector of responses.
CvMat* data = cvCreateMat( num_samples, num_features, CV_32F );
CvMat* responses = cvCreateMat( num_samples, 1, CV_32F );
for(i=0;i<num_samples;i++) {
for(j=0;j<num_features;j++) CV_MAT_ELEM(*data,float,i,j)=f
ij
;
CV_MAT_ELEM(*responses,float,i,0)=y
i
;
}
// initialize parameters
CvSVMParams params(CvSVM::C_SVC, CvSVM::RBF, 0, myGamma, 0, myC, 0, 0,
cvTermCriteria(CV_TERMCRIT_EPS+CV_TERMCRIT_ITERS, 1000, 1e-6));
// create and train the model
CvSVM svm(data, responses, 0, 0, params);
CvMat sample; cvGetRows(data,&sample,0,1);
float r = svm.predict(&sample); // predict the response for the first training vector
svm.save(“my_svm.xml”); // save the model to XML file

CvSVM svm1;
svm1.load(“my_svm.xml”); // load the model from XML file

55/67
Interaction with IPP

56/67
OpenCV and IPP
CV
Image Processing
and
Vision Algorithms
HighGUI
GUI,
Image and
Video I/O
MLL
Statistical Classifiers
and
Clustering Tools
CXCORE
basic structures and algoritms,
XML support, drawing functions
IPP is not installed

57/67
OpenCV and IPP
CV
Image Processing
and
Vision Algorithms
HighGUI
GUI,
Image and
Video I/O
MLL
Statistical Classifiers
and
Clustering Tools
CXCORE
basic structures and algoritms,
XML support, drawing functions
IPP is installed
IPP
Fast architecture-specific low-level functions

58/67
Performance Boost with IPP
3x-10x Vector math functions (exp, log, sin, cos …)
~1.5x-2x Face detection
~1.5x-3x DFT/FFT/DCT
~1.8x Corner detection
~1x-4x Image affine and perspective transformations
~1.5x-2x Distance transform
~1.5x-3x Image moments
~1x-3x Color Conversion (RGB to/from Grayscale,HSV,Luv)
~1.5x-4x Template Matching
~2x-8x Linear convolution
~2x-18x Median filter
~3x-7x Morphology
~3x Gaussian Pyramids
Speed increase (OpenCV+IPP
vs pure OpenCV)
Function Group

59/67
OpenCV + IPP recipe
Install OpenCV; make sure to add the bin subdirectory to the system
path
Install IPP; make sure to add the bin subdirectory to the system path
Enjoy faster OpenCV apps!
How to check whether IPP is used or not?
const char* plugin_info = 0;
cvGetModuleInfo(0,0,&plugin_info);
ipp_is_used = plugin_info != 0 && strstr(plugin_info,”ipp”)!=0;
How to turn off IPP? And turn on it again?
cvUseOptimized(0); … cvUseOptimized(1); // the function is slow!

60/67
Using OpenCV+IPP at Expert Level
#include <cv.h>
#include <highgui.h>
#include <ipp.h> // only for direct call to ippiMedianFilter_8u_C3R!
#include <stdio.h>
int main(int,char**){
const int M=3;
IppiSize msz={M,M}; IppiPoint ma={M/2,M/2};
IplImage* img=cvLoadImage(“lena.jpg”,1);
IplImage* med1=cvCreateImage(cvGetSize(img),8,3);
IplImage* med2=cvCloneImage(med1);
int64 t0 = cvGetTickCount(),t1,t2;
IppiSize sz = {img->width-M+1,img->height-M+1};
double isz=1./(img->width*img->height);
cvSmooth(img,med1,CV_MEDIAN,M); // use IPP via OpenCV interface
t0=cvGetTickCount()-t0;
cvUseOptimized(0); // unload IPP
t1 = cvGetTickCount();
cvSmooth(img,med1,CV_MEDIAN,M); // use C code
t1=cvGetTickCount()-t1;
t2=cvGetTickCount();
ippiMedianFilter_8u_C3R( // use IPP directly
&CV_IMAGE_ELEM(img,uchar,M/2,M/2*3), img->widthStep,
&CV_IMAGE_ELEM(med1,uchar,M/2,M/2*3), med1->widthStep, sz, msz, ma );
t2=cvGetTickCount()-t2;
printf(“t0=%.2f, t1=%.2f, t2=%.2f\n”, (double)t0*isz,(double)t1*isz,(double)t2*isz);
return 0; }
Triple 3x3 median filter

61/67
How does it works?
// cv/src/_cvipp.h

IPCVAPI_EX(CvStatus, icvFilterMedian_8u_C3R, “ippiFilterMedian_8u_C3R”,
CV_PLUGINS1(CV_PLUGIN_IPPI), (const void* src, int srcstep, void* dst, int dststep,
CvSize roi, CvSize ksize, CvPoint anchor ))

(Quite Similar to IPP dispatcher mechanism)
// cv/src/cvswitcher.cpp

#undef IPCVAPI_EX
#define IPCVAPI_EX() …
static CvPluginFuncInfo cv_ipp_tab[] = {
#undef _CV_IPP_H_
/* with redefined IPCVAPI_EX every function declaration turns to the table entry */
#include "_cvipp.h“
#undef _CV_IPP_H_
{0, 0, 0, 0, 0}
};
CvModule cv_module( &cv_info );

// cv/src/cvsmooth.cpp
icvFilterMedian_8u_C3R_t icvFilterMedian_8u_C3R_p = 0;
void cvSmooth() { if( icvFilterMedian_8u_C3R_p ) { /* use IPP */ } else { /* use C code */… }

62/67
OpenCV + Python

63/67
Python Interface
from opencv.cv import *
from opencv.highgui import *
cvNamedWindow("win", CV_WINDOW_AUTOSIZE)
cap=cvCreateFileCapture(“./Road.avi”)
img2 = temp = None
while 1:
img = cvQueryFrame(cap)
if not img: break
if not img2:
img2 = cvCloneImage(img)
temp = cvCloneImage(img)
cvFlip(img, img2, 0)
cvMorphologyEx(img2, img2, temp, None,
CV_MOP_GRADIENT, 2 );
cvShowImage("win", img2)
if cvWaitKey(30) >= 0: break
• Python is a great tool for fast
algorithm/demo prototyping (no
variable declaration, fully
automatic memory
management, no compile
phase)
• Wrappers are generated
automatically using SWIG
• Installs automatically with
OpenCV
• Covers full cxcore, cv, highgui
(no MLL yet)

64/67
Some Usage Examples

65/67
1. Improving IPP MPEG4 Encoder
Problem statement: MPEG-4 has capability to store global motion
transformation between frames => more compact representation of
motion vectors
Want to estimate rigid affine transformation, robust to “outliers” –
small objects moving in a different direction.
We just used cvCalcOpticalFlowPyrLK + RANSAC
With this feature IPP MPEG-4 video encoder (already very good one)
achieved 10% smaller bitrate with the same quality!
Now there is new function in OpenCV: cvEstimateRigidTransform
OpenCV Other projects

66/67
2. Extracting Text from Video
No ready function in OpenCV
What is text? How do we distinguish it from the other video content?

Each letter has a single color (e.g. white or yellow)

It has smooth shape

Letters are organized in words: horizontally-aligned groups.

All letters in the word (and all the subtitles in video frame) have the same color.

Letters & words preserve their shapes across several frames, they either stay at the same position
or move smoothly

Let’s use cvFindContours (we do not know the threshold level, so try different ones).

Sort the contours by y coordinates of bounding rectangle to identify clusters (words)

Use cvDrawContours + cvAvgStdDev to check the color.

Track feature points of letter candidates using cvCalcOpticalFlowPyrLK and check motion
smoothness and rigidity.

67/67
Summary
OpenCV is a long-term project and has established reputation because
of good quality
Together with IPP it is used in different areas and for different kinds of
applications, from small research projects to large commercial
products.
Intel is still standing behind of OpenCV
OpenCV will continue to evolve; the further grow will be application-
driven, (e.g. in video surveillance area)
Thank you!

Agenda
General Information Getting Started with OpenCV Modules Description Interaction with Intel IPP Python Interface Some Usage Examples Summary

2/67

General Information

3/67

Portable and very efficient (implemented in C/C++) Has BSD-like license (that is. absolutely free for academic and commercial use) Available at http://sourceforge.net/projects/opencvlibrary 4/67 .What is OpenCV? OpenCV stands for Open Source Computer Vision Library Includes over 500 functions implementing computer vision. image processing and general-purpose numeric algorithms.

or OpenSSL in cryptography).Why OpenCV? Computer Vision Market is large and continues to grow There is no standard API (like OpenGL and DirectX in graphics. Manufacturing control systems. independent/incompatible data types for every library/toolbox) Very expensive commercial toolkits (like Halcon. most of CV software is of 3 kinds: ● ● ● Research code (slow. …) Specialized solutions bundled with hardware (Video surveillance. MATLAB+Simulink. unstable. Medical equipment …) Standard library would simplify development of new applications and solutions much easier. Special optimization for Intel Architectures: ● ● Creates new usage models by achieving real-time performance for quite “heavy” algorithms (like face detection) Makes Intel platforms attractive for CV developers 5/67 .

Games • Highly optimized for Intel Arch. First Open Source Release: OpenCV alpha 3 at CVPR’00 OpenCV beta 1 with Linux support: CVPR’01 OpenCV 1.The Project History “CVL” project was started.0 with MacOSX support 1999/01 2000/06 2000/12 2006/10 Continuous development and various research projects 6/67 . the main goals/features: • Human-Computer interface is a main target • Real-Time Computer Vision Library for using by UI Developers. Videoconferences.

CMU. INRIA etc. Microsoft. MIT. with average daily traffic ~10-20 messages. IBM. textures. SONY.OpenCV Community The library is actively used by a large number of companies (such as Intel.com. 3d tracking. Community contributes to the project: bug reports. Cambridge.) >14000 members of the forum OpenCV@yahoogroups. Siemens. Python interface) 7/67 . Google.…) and research centers (Stanford. new features (video acquisition. patches.

GCC. ICC) ? (not tested)  (GCC. IPP. VS98. native APIs)  (w. BCC)  (w. GCC. ICC)   (iMac G5.Supported Platforms IA32 (x86) EM64T (x64) IA64 (Itanium) Partial Support Other (PPC. IPP. Solaris MacOSX N/A Others (BSD. Linux  (w. GCC. IPP.VS2005(Op VS98+PSDK. IPP. Sparc) N/A Windows (w.ICC. GCC. Solaris…)    8/67 . native APIs) Reported to build on UltraSparc. IPP.GCC. VS2005(OpenMP)) enMP). ICC)  (w.

lighting conditions. But … we have fundamental Computer Vision problem: can not create CV system that works always and everywhere (because of diff. like gesture recognition. “Analysis”: 1581. Then for every block (compound or primitive) we either use existing functions or create our own => we finally got multi-level library. which components may be re-used 9/67 . "resolution of anything complex into simple elements" We split a complex problem into elements/building blocks. Each block may require further decomposition. camera calibration etc.Library Design Initial goal: build high-level ready-to-use components. subject variety …) => need to analyze each problem.

statistical classification (or just Mahalanobis distance) and the way to visualize all the algorithm stages for debugging Depth Map with Stereo Camera 3D Clustering → 2D Color Histograms + Statistical Classification (Image moments as features) no free libraries could do all/most of this! => We had to do it ourselves 10/67 . stereo camera) One of algorithms: ? How to extract the shape? What are the lighting conditions? Dynamic or static gestures? Pre-defined set of gestures or extendable by user? So for this algorithm we need stereo.Example: Gesture Recognition How to find hand? • • • … By color Using motion detection Depth information (w. image moments. color histograms. 3D clustering.

drawing functions IPP Optimized code for Intel CPUs 11/67 . XML support. Clustering Algorithms HighGUI Simple GUI. camera calibration). motion analysis … MLL Statistical Classifiers. Image and Video Input/Output CXCORE Basic structures and algorithms (matrix and math operations). image processing.Library Architecture Samples/Demos CV + CVAUX High-level components (face detection.

Getting Started with OpenCV 12/67 .

14.i++) 11.x)/center. image ). return 0. ptr[2] = cvRound(ptr[2]*weight).h> 2.j*3). double dy=(double)(i-center. double scale=-3. CvPoint center. double weight=exp((dx*dx+dy*dy)*scale).h> 3. #include <highgui. } 13/67 Radial gradient . 16. char** argv ) { 5.png”. ptr[0] = cvRound(ptr[0]*weight).image->height/2).j<image->width. 15.uchar. #include <math.y. 17. 18. int main( int argc. cvSaveImage( “copy. 8. IplImage* image = argc==2 ? cvLoadImage(argv[1]) : 0. 1 ). ptr[1] = cvRound(ptr[1]*weight). uchar* ptr = &CV_IMAGE_ELEM(image. 10.j++) { 12. cvShowImage( "test". 20. image ). 13. center = cvPoint(image->width/2. cvNamedWindow( "test". double dx=(double)(j-center. for(int i=0. 7. } 19.i<image->height.y)/center. 9.i.The First OpenCV Program 1.h> 4. 23. for(int j=0. 6. 21.x. 22. if(!image) return -1. #include <cv. cvWaitKey().

lib – Create project for VS (see opencv/docs/faq.0) at http:// www.sourceforge.jpg 14/67 . 1.sln (opencv.sourceforge.cvs.htm) – or use opencv/samples/c/cvsample.cpp `pkg-config –libs opencv` Run it: sample lena.net/projects/opencvlibrary b) Or get the latest snapshot from CVS: :pserver:anonymous@opencvlibrary. open opencv.How to build and run the program? Obtain the latest version of OpenCV: a) Get the stable release (currently.lib cv.lib highgui./configure + make + make install (see the INSTALL document) Build the sample: Windows: – cl /I<opencv_inc> sample.net:/cvsroot/opencvlibrary Install it: Windows: run installer.cpp /link /libpath:<opencv_lib_path> cxcore.dsw) with Visual Studio and build the solution Linux/MacOSX: use .vcproj (cvsample.dsp) as starting point Linux: g++ -o sample `pkg-config –cflags` sample.

PPM. double dx=(double)(j-center. 20. 21. ptr[0] = cvRound(ptr[0]*weight).j*3).The Program Analysis Line-by-Line 1.h> 4.i++) 11. uchar* ptr = &CV_IMAGE_ELEM(image. for(int i=0.image->height/2). 9. return 0. 10. double scale=-3. double dy=(double)(i-center. TIFF. Use CV_IMAGE_ELEM macro to access image pixels 16-18. 15. Use cvPoint and other “constructor” functions to initialize simple structures on fly. 14. cvShowImage( "test". 19. 17. image ). 13. 22. #include <highgui. double weight=exp((dx*dx+dy*dy)*scale). 9. 15. ptr[1] = cvRound(ptr[1]*weight). cvSaveImage( “copy. 6. cvWaitKey(). 20. PNG.x. Use cvShowImage to display image in window 22. center = cvPoint(image->width/2. 16.i<image->height. cross-platform 1-2.h> 2. Include OpenCV headers first 7. 18. 1 ). IplImage is OpenCV image type 7. 23. ptr[2] = cvRound(ptr[2]*weight). } 19. Use cvNamedWindow to create self-updating highgui window 21. Use cvRound for very fast float->int conversion.y. } 15/67 1-23: The code is clear.j<image->width. #include <math. Use cvLoadImage and cvSaveImage to load/save an image in multiple formats: JPEG. BMP. for(int j=0. JPEG2000. 7. IplImage* image = argc==2 ? cvLoadImage(argv[1]) : 0.x)/center. short and cross-platform. image ). CvPoint center. int main( int argc. char** argv ) { 5. #include <cv.y)/center.j++) { 12. 8.uchar. if(!image) return -1. no need to use with MFC/GTK/QT/…. Use cvWaitKey to get input from user .png”. cvNamedWindow( "test".h> 3.i.

Modules Descriptions: HighGUI 16/67 .

rendering Processing keyboard and other events.Functionality Overview “Smart” windows Image I/O. timeouts Trackbars Mouse callbacks Video I/O 17/67 .

resize events. Dynamic updates of parts of the window are done using operations on images. cvMoveWindow(“ViewA”.Windows cvNamedWindow(window_name. then repaints it when necessary. Window handles repaint. cvDestroyWindow(“ViewA”). 18/67 .300. only the whole window contents can be modified.x On MacOSX – Carbon. creates window accessed by its name. On Windows native Win32 UI API is used On Linux – GTK+ 2. {8u|16s| 32s|32f}{C1|3|4} are supported. image). drawing functions etc.100).1). copies the image to window buffer. fixed_size_flag). Its position is remembered in registry: cvNamedWindow(“ViewA”. … cvShowImage(window_name.

jpeg”. image). converts it to color or grayscale. img ).-1). PNG (libpng). saves image to file. colorness_flag). PPM/PGM. image format is determined by the file contents. cvSaveImage(filename. Supported formats: BMP. if( img ) cvSaveImage( “picture. JPEG (libjpeg). loads image from file. // converting jpeg to png IplImage* img = cvLoadImage(“picture. if needed.png”. image format is defined by filename extension. TIFF (libtiff). JPEG 2000 (Jasper). 19/67 .Image I/O IplImage* cvLoadImage(filename.

To make program continue execution if no user actions is taken. if delay is 0. and check the return value // opencv/samples/c/delaunay. use cvWaitKey(<delay_ms!=0>). } 20/67 .c … for(…) { … int c = cvWaitKey(100). if( c >= 0 ) // key_pressed break.Waiting for keys… cvWaitKey(delay=0). waits for key press event for <delay> ms or infinitely.

10. &dilate_pos.c void on_mouse(int event. dilate_result ).Trackbars & Mouse Callbacks // opencv/samples/c/morphology.”E&D”. // check for events & process them } 21/67 . } int main(){ cvNamedWindow(“E&D”.1). void* param) { … } int main(){ cvNamedWindow(“LkDemo”. // check for events & process them } // opencv/samples/c/lkdemo.int flags. … cvWaitKey(0).0).c int dilate_pos=0.Dilate). … cvWaitKey(0). // initial position value void Dilate(int pos) { … cvShowImage( “E&D”. cvSetMouseCallback(“LkDemo”.1).int y.int x. cvCreateTrackbar(“Dilate”.on_mouse.

or NULL if there is no more frames or an error occured. initializes capturing from the specified camera CvCapture* cvCaptureFromFile(videofile_path). IplImage* cvQueryFrame(capture). position within video file etc.Video I/O API CvCapture* cvCaptureFromCAM(camera_id=0). value). IEEE1394. cvGetCaptureProperty(capture.) cvReleaseCapture(&capture). retrieves the next video frame (do not alter the result!). Quicktime (in progress) • Linux: V4L2. retrieves/sets some capturing properties (camera resolution. initializes capturing from the video file. FFMPEG • MacOSX: FFMPEG. property_id). do not forget to release the resouces at the end! Used interfaces: • Windows: VFW. Quicktime 22/67 . IEEE1394. cvSetCaptureProperty(capture. property_id. MIL. DirectShow (in progress).

// … copy and process image cvShowImage( “LkDemo”.c: (needs camera to run) 23/67 .Video I/O Sample // opencv/samples/c/lkdemo.) { IplImage* frame=cvQueryFrame(capture). if( !capture ) return -1. result ). // run at ~20-30fps speed if(c >= 0) { // process key }} cvReleaseCapture(&capture).c int main(…){ … CvCapture* capture = <…> ? cvCaptureFromCAM(camera_id) : cvCaptureFromFile(path). for(.. c=cvWaitKey(30).} opencv/samples/c/lkdemo. if(!frame) break.

Modules Descriptions: CXCORE 24/67 .

DFT/DCT.Functionality Overview Matrix and Image Arithmetical and Logical Operations Linear algebra. vector math Dynamic structures Drawing functions XML I/O 25/67 .

// will print 1. union { // aliased pointers to the data unsigned char* ptr. int rows. double* db. green=0. cvDet(&mA) ). 0. } data. … CvMat – fundamental 2D array type in // … CV_16UC1. …. CV_64FC1 OpenCV: int step. 1. CvMat mA = cvMat( 3.Matrix (CvMat) typedef struct CvMat { int type. 26/67  1 0 0  0 1 0    0 0 1   . &A[0][0] ). 1}}. 0}. 768. {0. // ~width int cols. {0. red = 255) cvSet( image. // fill the image with red color (blue=0. Data pointer width height Pixels/Elements step float A[3][3] = {{1. 3.255)). // initialize 3x3 identity matrix printf( “%g\n”. CV_8UC3. 0}.0 // create large color image as a matrix CvMat* image = cvCreateMat( 1024. float* fl. CV_32F. // ~height … } CvMat.0. // data type: CV_8UC1. 0. cvScalar(0. CV_8UC3 ).

2. cvScalar(0. signed/unsigned/floating-point int width.256. // pointer to the data … } IplImage.Image (IplImage) It came from the older Intel Image Processing Library (IPL). cvRandArr( &rng. 32. Almost every OpenCV function that takes CvMat* accepts IplImage*. height. 64. image.256)). cvScalar(256.0. 4 … int depth. 3 ). so you may use any (but CvMat* is preferable) 27/67 . and vice versa.768). CV_RAND_UNI. typedef struct _IplImage { int nChannels. 3. // 1. 16. // fill it with random values (color noise) CvRNG rng = cvRNG(0x12345). // step char* imageData. 8. // create color image IplImage* image = cvCreateImage( cvSize(1024. // 8. int widthStep.0).

… cvMerge(B. *B=cvCloneMat(G). G. cvGetSubArr(A. // release matrix CvMat* B = cvCloneMatrix(A). cvZero(A).Basic operations on Matrixes/Images cvReleaseMat(&A). R). B.5)). // make a copy of matrix cvCopy( A.c.i++) for(j=0.cvRect(3. RGB). // initialize matrix // Manipulating part of a matrix (no data is copied) CvMat B.i<B->rows.float. CV_8UC1). G. // Accessing individual matrix elements for(i=0.d)).j<B->cols.1.float. cvSplit(RGB.2.i).j)=CV_MAT_ELEM(*A. B ). Data pointer width height step 28/67 .i.j. *G = cvCloneMat(R). // Split matrix into a separate channels.&B. cvScalar(a. R. merge channels together CvMat* R = cvCreateMat(RGB->rows.RGB->cols. // copy contents of A to B cvSet(A.j++) // transpose A CV_MAT_ELEM(*B.b.

// C = A op a cvMin(A.&m.&sigma). // C = a*A*B cvDiv(A.L∞ cvMinMaxLoc(A. B. a. C). // norm is L1. C). b. C).c. // C = A – B cvAddS(A. C). cvMaxS(A. C). C). // op is =. C). B. a). C). > or ≥ cvCmpS(A. a). // C = ~A cvConvertScale(A. C). B) cvMinS(A. C). // S = ∑x.norm).htm 29/67 . // compute mean // and standard deviation a=cvNorm(A. // C = A + S cvSubRS(A. B. B. C. // C = S – A cvMul(A. C).C.&M.b. // C = min(A.Arithmetical and Logical Operations CvMat *A. // C = min(A. a). S = cvSum(A). CvScalar S=cvScalar(a. // S = mean(A) cvAvgStdDev(A. // C = A & a … cvOr. *B. cvXor. C. a. B. B. // C = A op B. a). a. B. cvXorS. B) cvMax(A.y A(x. // C = A + B cvSub(A.&mpos.&Mpos). B. cvAdd(A. c. // a=||A-B||norm. cvCmp(A. cvNot(A. // C = max(A. cvOrS.y) S = cvAvg(A).B. a. C. S. ≤. B. S. ≠.b).a. // C = max(A. maximum and their positions cvAnd(A. // find // global minimum. // C = A & B cvAndS(A. <. // C = A*a + b (A and C may have different types!) And there is a lot more! See opencv/docs/ref/opencvref_cxcore. op). op).d).L2. *C. // C = a*A/B // C = a*A + b*B + c cvAddWeighted( A. C).&m. C. a.

cvSVDBkSb // singular value decomposition and back substitution // some other functions to look at cvDet.!) cvExp(A.B. cvCrossProduct.Y.phase. cvPolarToCart(mag.a).Linear Algebra.B. 30/67 . cvDotProduct. Math Functions cvGEMM(A. cvEigenVV cvDFT(A.x.B).log() etc.b.flags).Y).method).phase).a). */ cvCartToPolar(X. cvInvert.B.D.B). /* B=pow(A. cvDCT(A.C.mag. // generalized matrix product: D=a*Aop(A) *Bop(B) + b*Cop(C) cvSolve(A.X.b. cvLog(A. // solve Ax=b (linear system or least squares problem) cvSVD.a. cvTrace.flags) // forward/inverse discrete Fourier and Cosine transforms // fast vector transcendental functions (5-20x faster than exp(). cvPow(A.B.flags).

sizeof(CvPoint). sizeof(CvSeq). i < img->height. &pt ).Dynamic Structures (when 2d array is not enough) The sample task: collect the positions of all non-zero pixels in the image. Where to store the coordinates? Possible solutions: ● ● ● Allocate array of maximum size (2*image_width*image_height – a huge value) Use two-pass algorithm Or use a list or similar data structure // create a sequence of non-zero pixel positions CvSeq* get_non_zeros( const IplImage* img.i. j++ ) if( CV_IMAGE_ELEM(img. cvSeqPush( seq.uchar. } 31/67 .j) ) { CvPoint pt={j. for( int i = 0. CvMemStorage* storage ) { CvSeq* seq = cvCreateSeq( CV_32SC2. } return seq. j < img->width. i++ ) for( int j = 0. storage ).i}.

• Good for multi-threaded applications • Example: one of audio processing pipelines became several times faster when new/delete were replaced with CvMemStorage allocation functions. Key functions: cvCreateMemStorage(block_size). cvMemStorageAlloc(storage.size). • All OpenCV “dynamic structures” reside in memory storages 32/67 . It acts like stack: new data is allocated always at the end data is not deallocated until storage is deleted or cleared. • In video/signal processing pipelines cvClearMemStorage is usually called in top-level loop. cvReleaseMemStorage(storage).Memory Storages Memory storage is a linked list of memory blocks. cvClearMemStorage(storage).

cvStartWriteSeq 33/67 .Sequences CvSeq is a linked list of continuous blocks. cvClearSeq. Sequence is deque: adding/removing elements to/from either end is O(1) operation. small sequences can be stored together in a single storage block (e. Large sequences may take multiple storage blocks.g. cvSetPushFront. Key functions: cvCreateSeq. Accessing arbitrary element is ~O(1) Sequences can be sequentially read/written using readers/writers (similar to STL’s iterators) Sequences can not be released. cvSeqPopFront cvGetSeqElem. inserting/removing elements from the middle is O(N). cvSeqPop. cvStartReadSeq. cvSeqPush. contours). only the whole storage can be deleted or cleared.

256. } return cmap.r=%d): %d\n”. CV_32SC1). printf( “(b=%d. for(CvSparseNode *node = cvInitSparseMatIterator( mat. idx[1]. int idx[] = {ptr[0]. count ). Sparse matrix in OpenCV uses CvSet + hash table to storing the elements. idx[2]. node != 0. j++ ) { uchar* ptr=&CV_IMAGE_ELEM(img. int count=*(int*)CV_NODE_VAL(cmap. dims. 256 }. } 34/67 . ((int*)cvPtrND(cmap.idx))[0]++. // Collecting the image colors CvSparseMat* get_color_map( const IplImage* img ) { int dims[] = { 256.node). node = cvGetNextSparseNode( &iterator )) { int* idx = CV_NODE_IDX(cmap.uchar.i.Sets and Sparse Matrices CvSet is unordered collection of elements. i < img->height. adding/removing elements is O(1) operation. idx[0].j*3). CvSparseMat* cmap = cvCreateSparseMat(3.ptr[1].ptr[2]}. for( int i = 0.idx). &iterator ). i++ ) for( int j = 0. j < img->width.g=%d. } // print the map CvSparseMatIterator it.

35/67 . Elliptic Arcs ● Filled polygons or polygonal contours ● Text (using one of embedded fonts) Everything can be drawn with different colors. different line width. from object tracking and vector fields to color histograms etc. with or without antialiasing Arbitrary images types are supported All the rendering is done in memory. use cvSaveImage to see the results later using external image viewer opencv/samples/c/drawing.Drawing Operations It is possible to draw different shapes in an image (IplImage/CvMat): ● Lines.cpp: Using CXCORE + HighGUI you can visualize everything. use cvShowimage to see the results immediately. Ellipses. Circles.

xml” ). 0. 0.</data></my_matrix> </opencv_storage> 36/67 . 0. 0. 0. 0. 0. 0. 0. cvSave( “my_matrix. &A ). } // to load it then in some other program use … CvMat* A1 = (CvMat*)cvLoad( “my_matrix. <?xml version="1. 1. 1. 0. cvLoad and other XML I/O functions from cxcore. 0. 0.XML I/O: save your data Need a config file? Want to save results of your program to pass it to another one. 1. 0. 1. 0. 0.xml // somewhere deep in your code… you get 5x5 // matrix that you’d want to save { CvMat A = cvMat( 5. 0. 0. or look at it another day? Use cvSave. 5.0"?> <opencv_storage> <my_matrix type_id="opencv-matrix"> <rows>5</rows> <cols>5</cols> <dt>f</dt> <data> 1.xml”. 0. 0. 0. the_matrix_data ). my_matrix. CV_32F.

320 ). cvEndWriteStruct(fs). “frame_count”. CV_STORAGE_WRITE). 200 ). CV_NODE_SEQ). 0. int frame_count = cvReadIntByName( fs. 37/67 . 10 /* default value */ ). cvWriteStartWriteStruct( fs.So what about config file? Writing config file CvFileStorage* fs = cvOpenFileStorage(“cfg. 0.xml”.xml”. CvSeq* s = cvGetFileNodeByName(fs. int frame_height = cvReadInt( (CvFileNode*)cvGetSeqElem(s. cvWriteInt( fs. cmatrix ). “frame_count”.0.”color_cvt_matrix”). 0. cvWriteInt( fs. CvMat* color_cvt_matrix = (CvMat*)cvRead(fs. cvReleaseFileStorage( &fs ).xml <?xml version="1.”frame_size”)->data.seq.0) ).0. 10 ).0"?> <opencv_storage> <frame_count>10</frame_count> <frame_size>320 200</frame_size> <color_cvt_matrix type_id="opencvmatrix"> <rows>3</rows> <cols>3</cols> <dt>f</dt> <data>…</data></color_cvt_matrix> </opencv_storage> Reading config file CvFileStorage* fs = cvOpenFileStorage(“cfg. 0.1) ). “color_cvt_matrix”. int frame_width = cvReadInt( (CvFileNode*)cvGetSeqElem(s. cvWriteint( fs. “frame_size”. 0. CV_STORAGE_READ). cvWrite( fs. cfg. cvReleaseFileStorage( &fs ).

Modules Descriptions: CV and CVAUX 38/67 .

Watershed.. Hough transform. Canny edge detector. Histograms. contours. Advanced Image Processing and Feature extraction: Corner detection. Inpainting. Motion templates. Geometrical transformations. Epipolar geometry Object detection (Haar classifier) Advanced Blob tracking (aka Video Surveillance module) . Pyramids. Distance transform. Kalman filters Camera calibration. Color space conversions. Morphology. … Shape analysis and Computational geometry: Image moments. Object tracking (Meanshift & CAMSHIFT).Functionality Overview Basic Image Processing: Image filters. Delaunay triangulation and Voronoi Tesselation Motion Analysis: Optical flow. 39/67 ..

B – W×H. A – 2W×2H 40/67 . A – W×H.B).A1).Basic Image Processing: Gaussian Pyramids cvPyrDown(A. B – W/2×H/2 cvPyrUp(B.

C.y)=maxB(x’.y+y’) Dilation: C(x.B. Opening: C=AoB= (AΘB)⊕B Closing: C=A•B= (A⊕B)ΘB C=Grad(A)=(A⊕B)-(AΘB) 41/67 .y’)=1A(x+x’.C. cvDilate(A.y’)=1A(x+x’.C.op).B).y)=minB(x’.B).Basic Image Processing: Morphology Primitive operations: cvErode(A. Erosion: C(x.y+y’) Image A Erosion: C=AΘB Dilation: C=A⊕B Compound operations: cvMorphologyEx(A.

cvCvtColor(lab_image.dx.kernel->rows/2)). CV_GAUSSIAN. Smoothing image using different methods: cvSmooth(A.B. // compute dA/dx using 3x3 Sobel kernel:  − 1 0 1  − 2 0 2    − 1 0 1   Image convolution with arbitrary kernel: cvFilter2D(A.11. cvSmooth(image. cvCvtColor(bgr_image.B.parameters…).Basic Image Processing: Some other important functions Color space conversion: cvCvtColor(A..color_conversion_code).CV_HIST_ARRAY)./(kernel->rows*kernel->cols))).cvScalar(1. image. 0. kernel. cvCalcArrHist((CvArr**)&grayscale_image. 3). // inplace Gaussian smoothing // with 7x7 kernel and σ=1. int n = 256. Computing spatial image derivatives: cvSobel(A.method. // slower alternative to cvSmooth(…. cvSet(kernel.dy. cvSobel(A. // compute histogram for grayscale image 42/67 . CV_BGR2GRAY).hist).CV_32F). 1. cvPoint(kernel->cols/2. cvFilter2D(A. CV_Lab2BGR).…) Computing image histogram: cvCalcArrHist(image_planes.B. bgr_image. grayscale_image. 1.&n. CvMat* kernel = cvCreateMat(11.anchor_point).). 7. CvHistogram* hist = cvCreateHist(1. 7. Ax. B.kernel_size).kernel.B. CV_BLUR. 1. hist).

B.t2). Assisting object segmentation Accelerating face detection 43/67 .t1.Feature Extraction: Canny Edge Detector Computing binary edge map from grayscale image: cvCanny(A.

params) Vanishing points estimation Circle Detection: cvHoughCircles(image. centers&radiuses) 44/67 . lines_buffer.Feature Extraction: Hough Transform Line Detection: cvHoughLines(image.

45/67 . distances. markers).….Advanced Image Processing: Extended Distance Transform and Watershed Extended distance transform (O(N) Voronoi diagram): cvDistTransform(A. Watershed marker-based segmentation: cvWatershed(A. labels).

mask ).Advanced Image Processing: Inpainting Reconstruction of marked areas using the neighbor pixels: cvInpaint( image. 46/67 .

cvFindContours(bw.Contours In OpenCV contours are extracted from blackn-white images. Grayscale image: 180000 pixels 47/67 Contours: <1800 points Approximated Contours: <180 points . Outer or internal boundaries of white areas are stored as polygons: cvCmpS(grayscale. Contour representation: contours=cvApproxPoly(contours.params). T. CV_CMP_GT). &contours). bw. storage.

cvCalcGlobalOrientation. cvCalcMotionGradient.Motion Analysis and Object Tracking Feature tracking: cvCalcOpticalFlowPyrLK Color object tracking: cvCamShift Motion templates: cvUpdateMotionHistory. cvSegmentMotion 48/67 .

 1   [ k1. cvFindExtrinsicCameraParams2 cvCalibrateCamera2   fx   0  0  0 fy 0 cx  cy  . p1.cpp 49/67 . p2 ]     Camera matrix + lens distortion coefficients See opencv/samples/c/calibration.Camera Calibration cvFindChessboardCorners cvUndistort2. k2 .

cvDetectHaarObjects 50/67 .Face/Object Detection “objects” “non-objects” opencv/apps/haartraining cvLoad.

opencv/samples/c/blobtrack.cpp and opencv/cvaux/include/cvvidsurv.Video Surveillance Video frames FG/BG Detection Blob Entering Detection Blob Tracking Trajectory Object ID’s and Generation trajectories For details see opencv/docs/vidsurv.hpp 51/67 .

Modules Descrpitions: MLL 52/67 .

Functionality Overview Statistical classification Regression Clustering Algorithms: SVM. EM. Decision Trees. Boosting. Neural Nets (MLP). Random Trees. kNN. Normal Bayesian classifier. 53/67 .

MLL Usage: SVM
// … prepare matrix of training feature vectors, and store them as rows of matrix “data”, // and prepare the vector of responses. CvMat* data = cvCreateMat( num_samples, num_features, CV_32F ); CvMat* responses = cvCreateMat( num_samples, 1, CV_32F ); for(i=0;i<num_samples;i++) { for(j=0;j<num_features;j++) CV_MAT_ELEM(*data,float,i,j)=fij; CV_MAT_ELEM(*responses,float,i,0)=yi; } // initialize parameters CvSVMParams params(CvSVM::C_SVC, CvSVM::RBF, 0, myGamma, 0, myC, 0, 0, cvTermCriteria(CV_TERMCRIT_EPS+CV_TERMCRIT_ITERS, 1000, 1e-6)); // create and train the model CvSVM svm(data, responses, 0, 0, params); CvMat sample; cvGetRows(data,&sample,0,1); float r = svm.predict(&sample); // predict the response for the first training vector svm.save(“my_svm.xml”); // save the model to XML file … CvSVM svm1; svm1.load(“my_svm.xml”); // load the model from XML file

54/67

Interaction with IPP

55/67

OpenCV and IPP
CV Image Processing and Vision Algorithms MLL Statistical Classifiers and Clustering Tools

IPP is not installed
HighGUI GUI, Image and Video I/O

CXCORE basic structures and algoritms, XML support, drawing functions

56/67

OpenCV and IPP CV Image Processing and Vision Algorithms MLL Statistical Classifiers and Clustering Tools IPP is installed HighGUI GUI. XML support. drawing functions IPP Fast architecture-specific low-level functions 57/67 . Image and Video I/O CXCORE basic structures and algoritms.

5x-4x ~1x-3x ~1.5x-3x 3x-10x ~1.5x-3x ~1.5x-2x .8x ~1. sin.HSV.5x-2x ~1x-4x ~1.Performance Boost with IPP Function Group Gaussian Pyramids Morphology Median filter Linear convolution Template Matching Color Conversion (RGB to/from Grayscale. cos …) Face detection 58/67 Speed increase (OpenCV+IPP vs pure OpenCV) ~3x ~3x-7x ~2x-18x ~2x-8x ~1. log.Luv) Image moments Distance transform Image affine and perspective transformations Corner detection DFT/FFT/DCT Vector math functions (exp.

cvGetModuleInfo(0. make sure to add the bin subdirectory to the system path Install IPP.”ipp”)!=0.&plugin_info).OpenCV + IPP recipe Install OpenCV. … cvUseOptimized(1).0. make sure to add the bin subdirectory to the system path Enjoy faster OpenCV apps! How to check whether IPP is used or not? const char* plugin_info = 0. // the function is slow! 59/67 . How to turn off IPP? And turn on it again? cvUseOptimized(0). ipp_is_used = plugin_info != 0 && strstr(plugin_info.

msz. (double)t0*isz. img->widthStep.CV_MEDIAN. t2=cvGetTickCount()-t2. ma ). IplImage* img=cvLoadImage(“lena.M/2. cvSmooth(img. } 60/67 . // use IPP via OpenCV interface t0=cvGetTickCount()-t0. cvUseOptimized(0). printf(“t0=%. // unload IPP t1 = cvGetTickCount().M}.M).1).med1.h> int main(int.t1. t2=%. sz.h> #include <ipp. IplImage* med2=cvCloneImage(med1).char**){ const int M=3.M). IppiSize sz = {img->width-M+1.3).CV_MEDIAN.2f./(img->width*img->height).uchar.M/2*3).8.M/2. ippiMedianFilter_8u_C3R( // use IPP directly &CV_IMAGE_ELEM(img. double isz=1.jpg”. IppiPoint ma={M/2.(double)t1*isz. IplImage* med1=cvCreateImage(cvGetSize(img).h> // only for direct call to ippiMedianFilter_8u_C3R! #include <stdio. t1=%. &CV_IMAGE_ELEM(med1.img->height-M+1}. IppiSize msz={M. int64 t0 = cvGetTickCount().2f.h> #include <highgui. cvSmooth(img. med1->widthStep.Using OpenCV+IPP at Expert Level Triple 3x3 median filter #include <cv. return 0.M/2}.M/2*3).uchar.(double)t2*isz).2f\n”. t2=cvGetTickCount().t2.med1. // use C code t1=cvGetTickCount()-t1.

CvSize roi. 0. icvFilterMedian_8u_C3R. int dststep. int srcstep. void cvSmooth() { if( icvFilterMedian_8u_C3R_p ) { /* use IPP */ } else { /* use C code */… } 61/67 .cpp icvFilterMedian_8u_C3R_t icvFilterMedian_8u_C3R_p = 0. 0. CV_PLUGINS1(CV_PLUGIN_IPPI). 0} }. CvPoint anchor )) … // cv/src/cvswitcher. void* dst. (const void* src. CvModule cv_module( &cv_info ). … // cv/src/cvsmooth.h … IPCVAPI_EX(CvStatus.h“ #undef _CV_IPP_H_ {0. 0. CvSize ksize.cpp … #undef IPCVAPI_EX #define IPCVAPI_EX() … static CvPluginFuncInfo cv_ipp_tab[] = { #undef _CV_IPP_H_ /* with redefined IPCVAPI_EX every function declaration turns to the table entry */ #include "_cvipp.How does it works? (Quite Similar to IPP dispatcher mechanism) // cv/src/_cvipp. “ippiFilterMedian_8u_C3R”.

OpenCV + Python 62/67 .

Python Interface • Python is a great tool for fast algorithm/demo prototyping (no variable declaration. highgui (no MLL yet) from opencv. no compile phase) • Wrappers are generated automatically using SWIG • Installs automatically with OpenCV • Covers full cxcore. CV_MOP_GRADIENT. img2. 0) cvMorphologyEx(img2. img2) if cvWaitKey(30) >= 0: break 63/67 . img2. fully automatic memory management. None. cv.avi”) img2 = temp = None while 1: img = cvQueryFrame(cap) if not img: break if not img2: img2 = cvCloneImage(img) temp = cvCloneImage(img) cvFlip(img.highgui import * cvNamedWindow("win"./Road. 2 ).cv import * from opencv. temp. cvShowImage("win". CV_WINDOW_AUTOSIZE) cap=cvCreateFileCapture(“.

Some Usage Examples 64/67 .

1. robust to “outliers” – small objects moving in a different direction. We just used cvCalcOpticalFlowPyrLK + RANSAC With this feature IPP MPEG-4 video encoder (already very good one) achieved 10% smaller bitrate with the same quality! Now there is new function in OpenCV: cvEstimateRigidTransform OpenCV Other projects 65/67 . Improving IPP MPEG4 Encoder Problem statement: MPEG-4 has capability to store global motion transformation between frames => more compact representation of motion vectors Want to estimate rigid affine transformation.

so try different ones). All letters in the word (and all the subtitles in video frame) have the same color. white or yellow) It has smooth shape Letters are organized in words: horizontally-aligned groups.g. Sort the contours by y coordinates of bounding rectangle to identify clusters (words) Use cvDrawContours + cvAvgStdDev to check the color. they either stay at the same position or move smoothly ● ● ● ● Let’s use cvFindContours (we do not know the threshold level.2. Track feature points of letter candidates using cvCalcOpticalFlowPyrLK and check motion smoothness and rigidity. 66/67 . Extracting Text from Video No ready function in OpenCV What is text? How do we distinguish it from the other video content? ● ● ● ● ● Each letter has a single color (e. Letters & words preserve their shapes across several frames.

the further grow will be applicationdriven. Intel is still standing behind of OpenCV OpenCV will continue to evolve.g. (e. from small research projects to large commercial products.Summary OpenCV is a long-term project and has established reputation because of good quality Together with IPP it is used in different areas and for different kinds of applications. in video surveillance area) 67/67 .

Thank you! .

Sign up to vote on this title
UsefulNot useful