Processing Qing Chen (David) E-mail: qchen@discover.uottawa.ca DiscoverLab School of Information Technology & Engineering University of Ottawa J anuary 15, 2007 2 Outline 1. Introduction 2. Image data structure in OpenCV 3. Basic operations for images 4. Working with videos 5. References and resources 3 1. Introduction General description Open source computer vision library in C/C++. Optimized and intended for real-time applications. OS/hardware/window-manager independent. Generic image/video loading, saving, and acquisition. Both low and high level API. Provides interface to Intel's Integrated Performance Primitives (IPP) with processor specific optimization (Intel processors). 4 1. Introduction Features: Image data manipulation (allocation, release, copying, setting, conversion). Image and video I/O (file and camera based input, image/video file output). Matrix and vector manipulation and linear algebra routines. Various dynamic data structures (lists, queues, sets, trees, graphs). Basic image processing (filtering, edge detection, corner detection, sampling and interpolation, color conversion, morphological operations, histograms, image pyramids). Structural analysis (connected components, contour processing, distance transform, various moments, template matching, Hough transform, polygonal approximation, line fitting, ellipse fitting, Delaunay triangulation). Camera calibration (finding and tracking calibration patterns, calibration, fundamental matrix estimation, homography estimation, stereo correspondence). Motion analysis (optical flow, motion segmentation, tracking). Object recognition (eigen-methods, HMM). Basic GUI (display image/video, keyboard and mouse handling, scroll-bars). Image labeling (line, conic, polygon, text drawing). 5 1. Introduction OpenCV modules: cv - Main OpenCV functions. cvaux - Auxiliary (experimental) OpenCV functions. cxcore - Data structures and linear algebra support. highgui - GUI functions. 6 2. Image data structure in OpenCV Load and display an image in OpenCV: #i ncl ude " cv. h" / / mai n OpenCV f unct i ons #i ncl ude " hi ghgui . h" / / OpenCV GUI f unct i onsi ncl ude <st di o. h> i nt mai n( ) { / * decl ar e a new I pl I mage poi nt er , t he basi c i mage dat a st r uct ur e i n OpenCV */ IplImage* newImg; / * l oad an i mage named " appl e. bmp" , 1 means t hi s i s a col or i mage */ newImg = cvLoadImage("apple.bmp",1); / / cr eat e a new wi ndow cvNamedWi ndow( " Wi ndow" , 1) ; / / di spl ay t he i mage i n t he wi ndow cvShowI mage( " Wi ndow" , newI mg) ; / / wai t f or key t o cl ose t he wi ndow cvWai t Key( 0) ; cvDest r oyWi ndow( " Wi ndow" ) ; / / dest r oy t he wi ndow cvRel easeI mage( &newI mg ) ; / / r el ease t he memor y f or t he i mage r et ur n 0; } 7 2. Image data structure in OpenCV IplImage is the basic image data structure in OpenCV 8 3. Basic operations for images Threshold #i ncl ude "cv. h" #i ncl ude "hi ghgui . h" #i ncl ude "mat h. h" i nt mai n( ) { I pl I mage* sr c; I pl I mage* col or Thr esh; I pl I mage* gr ay; I pl I mage* gr ayThr esh; i nt t hr eshol d = 120, maxVal ue = 255; i nt t hr eshol dType = CV_THRESH_BI NARY; sr c = cvLoadI mage( "appl e. bmp", 1) ; col or Thr esh = cvCl oneI mage( sr c ) ; gr ay = cvCr eat eI mage( cvSi ze( sr c- >wi dt h, sr c- >hei ght ) , I PL_DEPTH_8U, 1 ) ; cvCvt Col or ( sr c, gr ay, CV_BGR2GRAY ) ; gr ayThr esh = cvCl oneI mage( gr ay ) ; cvNamedWi ndow( " sr c", 1 ) ; cvShowI mage( "sr c", sr c ) ; cvNamedWi ndow( " gr ay", 1 ) ; cvShowI mage( "gr ay", gr ay ) ; cvThreshold(src, colorThresh, threshold, maxValue, thresholdType); cvThreshold(gray, grayThresh, threshold, maxValue, thresholdType); cvNamedWi ndow( " col or Thr esh" , 1 ) ; cvShowI mage( " col or Thr esh" , col or Thr esh ) ; cvNamedWi ndow( " gr ayThr esh", 1 ) ; cvShowI mage( "gr ayThr esh", gr ayThr esh ) ; cvWai t Key( 0) ; cvDest r oyWi ndow( "sr c" ) ; cvDest r oyWi ndow( "col or Thr esh" ) ; cvDest r oyWi ndow( "gr ay" ) ; cvDest r oyWi ndow( "gr ayThr esh" ) ; cvRel easeI mage( &sr c ) ; cvRel easeI mage( &col or Thr esh ) ; cvRel easeI mage( &gr ay ) ; cvRel easeI mage( &gr ayThr esh ) ; r et ur n 0; } 9 3. Basic operations for images Canny edge detection #i ncl ude "cv. h" #i ncl ude "hi ghgui . h" i nt mai n( ) { I pl I mage* newI mg; / / or i gi nal i mage I pl I mage* gr ayI mg; / / gr ay i mage f or t he conver si on of t he or i gi nal i mage I pl I mage* cannyI mg; / / gr ay i mage f or t he canny edge det ect i on / / l oad or i gi nal i mage newI mg = cvLoadI mage( " appl e. bmp", 1) ; / / cr eat e a si ngl e channel 1 byt e i mage ( i . e. gr ay- l evel i mage) gr ayI mg = cvCr eat eI mage( cvSi ze( newI mg- >wi dt h, newI mg- >hei ght ) , I PL_DEPTH_8U, 1 ) ; / / conver t or i gi nal col or i mage ( 3 channel r gb col or i mage) t o gr ay- l evel i mage cvCvt Col or ( newI mg, gr ayI mg, CV_BGR2GRAY ) ; cannyI mg = cvCr eat eI mage( cvGet Si ze( newI mg) , I PL_DEPTH_8U, 1) ; / / canny edge det ect i on cvCanny(grayImg, cannyImg, 50, 150, 3); cvNamedWi ndow( "sr c" , 1) ; cvNamedWi ndow( "canny", 1) ; cvShowI mage( " sr c", newI mg ) ; cvShowI mage( " canny" , cannyI mg ) ; cvWai t Key( 0) ; cvDest r oyWi ndow( "sr c" ) ; cvDest r oyWi ndow( "canny" ) ; cvRel easeI mage( &newI mg ) ; cvRel easeI mage( &gr ayI mg ) ; cvRel easeI mage( &cannyI mg ) ; r et ur n 0; } 10 3. Basic operations for images Contour detection #i ncl ude "cv. h" #i ncl ude "cxcor e. h" #i ncl ude "hi ghgui . h" i nt mai n( ) { I pl I mage* newI mg = NULL; I pl I mage* gr ayI mg = NULL; I pl I mage* cont our I mg = NULL; / / par amet er s f or t he cont our det ect i on CvMemSt or age * st or age = cvCr eat eMemSt or age( 0) ; CvSeq * cont our = 0; i nt mode = CV_RETR_EXTERNAL; mode = CV_RETR_CCOMP; / / det ect bot h out si de and i nsi de cont our cvNamedWi ndow( "sr c", 1) ; cvNamedWi ndow( "cont our ", 1) ; / / l oad or i gi nal i mage newI mg = cvLoadI mage( "appl ebw. bmp", 1) ; / / cr eat e a si ngl e channel 1 byt e i mage ( i . e. gr ay- l evel i mage) gr ayI mg = cvCr eat eI mage( cvSi ze( newI mg- >wi dt h, newI mg- >hei ght ) , I PL_DEPTH_8U, 1 ) ; / / conver t or i gi nal col or i mage ( 3 channel r gb col or i mage) t o gr ay- l evel i mage cvCvt Col or ( newI mg, gr ayI mg, CV_BGR2GRAY ) ; cvShowI mage( "sr c" , newI mg ) ; / / make a copy of t he or i gi nal i mage t o dr aw t he det ect ed cont our cont our I mg = cvCr eat eI mage( cvGet Si ze( newI mg) , I PL_DEPTH_8U, 3) ; cont our I mg=cvCl oneI mage( newI mg ) ; / / f i nd t he cont our cvFindContours(grayImg, storage, &contour, sizeof(CvContour), mode, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0)); / / dr aw t he cont our cvDrawContours(contourImg, contour, CV_RGB(0, 255, 0), CV_RGB(255, 0, 0), 2, 2, 8); cvShowI mage( "cont our ", cont our I mg ) ; cvWai t Key( 0) ; cvDest r oyWi ndow( " sr c" ) ; cvDest r oyWi ndow( "cont our " ) ; cvRel easeI mage( &newI mg ) ; cvRel easeI mage( &gr ayI mg ) ; cvRel easeI mage( &cont our I mg ) ; cvRel easeMemSt or age( &st or age) ; r et ur n 0; } 11 3. Basic operations for images Dilate/Erode #i ncl ude "cv. h" #i ncl ude "cxcor e. h" #i ncl ude "hi ghgui . h" i nt mai n( ) { I pl I mage* newI mg = NULL; I pl I mage* di l at eI mg = NULL; I pl I mage* er odeI mg = NULL; cvNamedWi ndow( "sr c", 1) ; cvNamedWi ndow( "di l at e", 1) ; cvNamedWi ndow( "er ode", 1) ; / / l oad or i gi nal i mage newI mg = cvLoadI mage( "appl e. bmp", 1) ; cvShowI mage( "sr c" , newI mg ) ; / / make a copy of t he or i gi nal i mage di l at eI mg=cvCl oneI mage( newI mg ) ; er odeI mg=cvCl oneI mage( newI mg ) ; / / di l at e i mage cvDilate(newImg,dilateImg,NULL,4); / / er ode i mage cvErode(newImg,erodeImg,NULL,4); cvShowI mage( "di l at e", di l at eI mg ) ; cvShowI mage( "er ode", er odeI mg ) ; cvWai t Key( 0) ; cvDest r oyWi ndow( " sr c" ) ; cvDest r oyWi ndow( "di l at e" ) ; cvDest r oyWi ndow( "er ode" ) ; cvRel easeI mage( &newI mg ) ; cvRel easeI mage( &di l at eI mg ) ; cvRel easeI mage( &er odeI mg ) ; r et ur n 0; } 12 3. Basic operations for images Flood and Fill #i ncl ude "cv. h" #i ncl ude "cxcor e. h" #i ncl ude "hi ghgui . h" i nt mai n( ) { I pl I mage* newI mg = NULL; I pl I mage* f f I mg = NULL; / / f l ood and f i l l par amet er s i nt l o_di f f , up_di f f ; / / t he l ow and up f l ood r andge whi ch can be adj ust ed CvConnect edComp comp; CvPoi nt f l oodSeed; / / t he or i gi nal pi xel wher e t he f l ood begi ns CvScal ar f l oodCol or ; l o_di f f =8; up_di f f =8; f l oodCol or = CV_RGB( 255, 0, 0 ) ; / / set t he f l ood col or t o r ed cvNamedWi ndow( "sr c", 1) ; cvNamedWi ndow( "f l ood&f i l l ", 1) ; / / l oad or i gi nal i mage newI mg = cvLoadI mage( "appl e. bmp", 1) ; cvShowI mage( "sr c" , newI mg ) ; / / make a copy of t he or i gi nal i mage f f I mg=cvCl oneI mage( newI mg ) ; f l oodSeed=cvPoi nt ( 60, 60) ; / / f l oodi ng st ar t f r ompi xel ( 60, 60) / / Fl ood and Fi l l f r ompi xel ( 60, 60) wi t h col or r ed and t he f l ood r ange of ( - 8, +8) cvFloodFill( ffImg, floodSeed, floodColor, CV_RGB( lo_diff, lo_diff, lo_diff ), CV_RGB( up_diff, up_diff, up_diff ), &comp, 8, NULL); cvShowI mage( "f l ood&f i l l ", f f I mg ) ; cvWai t Key( 0) ; cvDest r oyWi ndow( " sr c" ) ; cvDest r oyWi ndow( "f l ood&f i l l " ) ; cvRel easeI mage( &newI mg ) ; cvRel easeI mage( &f f I mg ) ; r et ur n 0; } 13 3. Basic operations for images Rotate and Scale #i ncl ude "cv. h" #i ncl ude "hi ghgui . h" #i ncl ude "mat h. h" i nt mai n( ) { I pl I mage* sr c; I pl I mage* dst ; i nt del t a; i nt angl e; sr c = cvLoadI mage( "appl e. bmp", 1) ; dst = cvCl oneI mage( sr c ) ; del t a = 1; angl e = 0; cvNamedWi ndow( "sr c", 1 ) ; cvShowI mage( "sr c" , sr c ) ; f or ( ; ; ) { f l oat m[ 6] ; doubl e f act or = ( cos( angl e*CV_PI / 180. ) + 1. 1) *3; CvMat M = cvMat ( 2, 3, CV_32F, m) ; i nt w = sr c- >wi dt h; i nt h = sr c- >hei ght ; m[ 0] = ( f l oat ) ( f act or *cos( - angl e*2*CV_PI / 180. ) ) ; m[ 1] = ( f l oat ) ( f act or *si n( - angl e*2*CV_PI / 180. ) ) ; m[ 2] = w*0. 5f ; m[ 3] = - m[ 1] ; m[ 4] = m[ 0] ; m[ 5] = h*0. 5f ; cvGetQuadrangleSubPix( src, dst, &M, 1, cvScalarAll(0)); cvNamedWi ndow( "dst " , 1 ) ; cvShowI mage( "dst ", dst ) ; i f ( cvWai t Key( 5) == 27 ) br eak; angl e = ( angl e + del t a) %360; } r et ur n 0; } 14 4. Working with videos Video capture from a file: CvCapture* cvCaptureFromFile( const char* filename ); Video capture from a camera: CvCapture* cvCaptureFromCAM( int index ); example: / / capt ur e f r omvi deo devi ce #0 CvCapt ur e* capt ur e = cvCapt ur eFr omCAM( 0) ; 15 4. Working with videos Grab a frame: cvGrabFrame( CvCapture* capture ); Get the image grabbed with cvGrabFrame: cvRetrieveFrame( CvCapture* capture ); example: I pl I mage* i mg = 0; i f ( ! cvGr abFr ame( capt ur e) ) { / / capt ur e a f r ame pr i nt f ( "Coul d not gr ab a f r ame\ n\ 7") ; exi t ( 0) ; } / / r et r i eve t he capt ur ed f r ame i mg=cvRet r i eveFr ame( capt ur e) ; Release the capture source: cvReleaseCapture(&capture); For a better understanding of video processing with OpenCV, refer to the face detection example under the dir: C:\Program Files\OpenCV\samples\c\facedetect.c 16 5. References and resources http://www.intel.com/technology/computing/opencv/index.htm OpenCV official webpage. http://opencvlibrary.sourceforge.net/ OpenCV documentation and FAQs. http://tech.groups.yahoo.com/group/OpenCV/ OpenCV forum at Yahoo Groups. http://www.site.uottawa.ca/~laganier/tutorial/opencv+directshow/cvis ion.htm This is a good walkthrough for OpenCV and the Microsoft DirectShow technology by Prof. Robert Laganire of university of Ottawa. The configuration of OpenCV for MS .Net is also included. http://ai.stanford.edu/~dstavens/cs223b/stavens_opencv_optical_flo w.pdf This is another OpenCV introduction focused on Optical Flow, the installation of OpenCV is also included. 17 About myself I am a Ph.D. candidate at the DiscoverLab, School of Information Technology and Engineering, University of Ottawa. My general research interests include image processing and computer vision. My current research topic is focused on real-time vision-based hand gesture recognition for Human Computer Interface (HCI). For more information about me, please refer to my webpage: http://www.discover.uottawa.ca/~qchen