Machine Vision and Image Processing Algorithm –

Fall 2009
Mario

1
` Introducion to OpenCV
` OpenCV structure and classes
` OpenCV functions
` Image acquisition using webcam

2
` OpenCV stands for Open Source Computer
Vision Libraryy
` Being developed at Intel since 1999
` Written in C/C++; Contains over 500 functions.
` Available on Windows, Linux and MacOSX.
` So far is extensively used in many companies
and research centers
` Home page:
www intel com/technology/computing/opencv/
www.intel.com/technology/computing/opencv/

3
` Computer Vision Market is large and continues to grow

` There is no standard API (like OpenGL and DirectX in graphics, or

OpenSSL in cryptography), most of CV software is of 3 kinds:
◦ Research code (slow, unstable, independent/incompatible data types for every
library/toolbox)
b a y/too bo )
◦ Very expensive commercial toolkits (like Halcon, MATLAB+Simulink, …)
◦ Specialized solutions bundled with hardware (Video surveillance, Manufacturing
control systems, Medical equipment …)

` Standard library would simplify development of new applications and

solutions much easier.

` Special optimization for Intel Architectures:

◦ Creates new usage models by achieving real-time performance for quite “heavy”
algorithms (like face detection)
◦ Makes Intel platforms attractive for CV developers

4
` The library is actively used by a large number of
companies (such as Intel, IBM, Microsoft, SONY,
Siemens, Google,…) and research centers
(Stanford MIT
(Stanford, MIT, CMU
CMU, Cambridge
Cambridge, INRIA etc
etc.))
` >14000 members of the forum
OpenCV@yahoogroups com with average daily
OpenCV@yahoogroups.com,
traffic ~10-20 messages.
` Communityy contributes to the project:
p j bug
g reports,
p
patches, new features (video acquisition, 3d
tracking, textures, Python interface)

5
` Extract from license:
[Copyright Clause]
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistribution's
ed st but o s o of sou
source
ce code must
ust retain
eta tthe
e abo
above
e copy
copyright
g t notice,
ot ce,
this list of conditions and the following disclaimer.

* Redistribution's in binary form must reproduce the above copyright notice,

this list of conditions and the following
g disclaimer in the documentation
and/or other materials provided with the distribution.

* The name of Intel Corporation may not be used to endorse or promote products
derived from this software without specific
p prior
p written permission.
p
[Disclaimer]

I brief:
In b i f Sure!
S !

6
7
 CV
Image processing
and vision algorithms HighGUI
g and Video I/O
GUI, Image

CXCORE
basic structures and algoritms,
g ,
XML support, drawing functions We will mostly
focus on these two
in this presentation

8
z IplImage* cvLoadImage(image_path, colorness_flag);
g from file,, converts to color or g
loads image y , if need,, and returns it ((or
grayscle,
returns NULL).
image format is determined by the file contents.

z cvSaveImage(image_path,
S I (i th image);
i )
saves image to file, image format is determined from extension.

z BMP, JPEG (via

BMP ( i libjpeg),
libj ) PNG (via
( i libpng),
lib ) TIFF (via
( i libtiff),
libtiff) PPM/PGM fformats
t
are supported.

IplImage* img =
cvLoadImage(“picture.jpeg”,-1);
if( img ) cvSaveImage( “picture.png”, img
);

9
 Scanline()…
Windows GUI
OpenCV-based
Image
Timage Processing
Image
(Bitmap) (Conversion,
Processing
I lI
IplImageToTBitmap()
T TBit () Transform etc
(Conversion
Transform,
(Conversion,etc.))
transform, etc.)

Image1->Picture->LoadFromFile() IplImage *
cvLoadImage()

Image File
Other image
(bmp, jpg, cvSaveImage()
Source
png))
(e.g. Camera)

10
typedef struct CvMat { Data pointer
int type; // data type: CV_8UC1, width
CV_8UC3,, …
// … CV_16UC1, …, CV_64FC1
int step;
union { // aliased pointers to the
data
unsigned char* ptr;
float* fl; double* db; Pixel/element
} data; height
int rows; // ~width
int cols; // ~height
…
} CvMat;

Example: // create large color image as a matrix

CvMat* image = cvCreateMat( 1024, 768, CV_8UC3 );
// fill the image with red color (blue=0, green=0, red =
255)
cvSet( image, cvScalar(0,0,255));

11
` cvGEMM(A,B,a,C,b,D,flags); // generalized matrix product:
D=a*Aop(A) *Bop(B) + b*Cop(C)
` cvSolve(A,b,x,method); // solve Ax=b (linear system or least
squares problem)
` cvSVD, cvSVDBkSb // singular value decomposition and back
substitution

// some other functions to look at

` cvDet, cvTrace, cvInvert, cvDotProduct, cvCrossProduct, cvEigenVV
` cvDFT(A,B,flags); cvDCT(A,B,flags) // forward/inverse discrete Fourier and
` Cosine transforms

// fast vector transcendental functions (5-20x faster than exp(),log() etc.!)

` cvExp(A,B); cvLog(A,B); cvPow(A,B,a); /* B=pow(A,a); */
` cvCartToPolar(X,Y,mag,phase); cvPolarToCart(mag,phase,X,Y);

12
` The sample task: collect the positions of all non-zero pixels in the image.
` Where to store the coordinates?
` Possible solutions:
◦ Allocate array of maximum size (2*image_width*image_height – a huge value)
◦ Use two-pass algorithm
◦ Or use a list or similar data structure

// create a sequence of non-zero pixel positions

CvSeq*q g get_non_zeros( ( const IplImage*
p g img,
g, CvMemStorage*
g storage
g )
{
CvSeq* seq = cvCreateSeq( CV_32SC2, sizeof(CvSeq),
sizeof(CvPoint), storage );

for( int i = 0; i < img->height; i++ )

for( int j = 0; j < img->width; j++ )
if( CV_IMAGE_ELEM(img,uchar,i,j) )
{
CvPoint pt={j,i}; cvSeqPush( seq, &pt ); }
return seq;
}
13
` Memory storage is a linked list
off memory blocks.
bl k
` It acts like stack:
◦ new data is allocated always
y at
the end
◦ data is not deallocated until
g is deleted or cleared.
storage
` Key functions:
◦ cvCreateMemStorage(block_size),
◦ cvReleaseMemStorage(storage);
◦ cvClearMemStorage(storage);
◦ cvMemStorageAlloc(storage,size);

14
 ` Need a config file?
` Want to save results of your program to pass it to
another one, or look at it another day?
` Use cvSave, cvLoad and other XML I/O functions
from cxcore.
// somewhere deep in your code… you get 5x5 <?xml version="1.0"?>
// matrix that you’d want to save <opencv_storage>
{ CvMat A = cvMat( 5, 5, CV_32F, the_matrix_data
);
<my_matrix type_id="opencv-matrix">
cvSave( “my matrix xml” &A );
“my_matrix.xml”, <rows>5</rows>
rows 5 /rows
} <cols>5</cols>
// to load it then in some other program use … <dt>f</dt>
CvMat* A1 = (CvMat*)cvLoad( “my_matrix.xml” ); <data>
1. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.
0 1.
0. 1 0.
0 0.
0 0.
0 0.
0 0.
0 1.
1 00. 0.
0 0.
0 00.
0. 1.</data></my_matrix>
</opencv_storage>
15
16
` Create an IplImage object:
◦ IplImage * srcImage;
` Load Image
◦ srcImage =
cvLoadImage(FileName, -
1);
);
` Show Image
◦ bitmap = new
Graphics::TBitmap();
◦ IplImageToTBitmap(srcIma
p g p(
ge, bitmap);
◦ Image1->Picture->Bitmap
= bitmap;
Or
◦ cvNamedWindow(“image”);
◦ cvShowImage(“image”,
srcImage);

17
 src

` Color conversion can

RGB Color space
be performed
f using
cvCvtColor()
function
function.
` Void CV_RGB_HSV,
cvCvtColor(const dst
CvArr* src, const
CvArr* dst, int code);
◦ Src: source image
g HSV Color space
◦ Dst: destination image
◦ Code: color conversion operation
x E.g. CV_RGB_XYZ, CV_RGB_HSV,
CV_RGB_YCrCb,

18
19
` We can set ROI in an
image
image
i by
b using
i
cvSetImageROI function.
` void x, y width
cvSetImageROI(IplImage*
image, CvRect rect); height ROI
◦ Image: the source image
◦ Rect : the roi rectangle
` ROI
O iss defined
de ed by oneo e point
po t
(x,y), width and the height.
CvRect r = cvRect(int x, int y, int width, int
height);

20
 wsrc
` We can use cvCopy to copy
se ected e
selected e e ts from
elements o input
put ROI
array to output array. hsrc src
` If the passed input array is
IplImage, then its ROI fields is
used
used.
` Both array must have the
same type.
` void cvCopy( const CvArr*
src, CvArr*
C A * dst,
d t constt CvArr*
C A * wdst
mask=NULL );
◦ src The source array.
◦ dst The destination array.
hdst dst
◦ maskkO i mask,
Operation k 8 bi
8-bit
single channel array; specifies
elements of destination array to
be changed.
If there is ROI
ROI, wdst = wroi and hdst = hroi
Else wdst = wsrc and hdst = hsrc

21
ROI

22
 j
img
` The simplest way to
access image pixel is i
by using a CvScalar
structure
t t and
d
cvGet2D() function cvGet2D
S[0]
[ ]
` Example:
CvScalar s; s
s=cvGet2D(img,i,j); S[1]

S[2]

S[3]

23
` Example of source code for reading an image:

image = cvLoadImage(filenames.at(k).c_str(), 1);

for(int i=0;i<image->height;i++)
{
for(int j=0;j<image->width;j++)
{
CvScalar n;
n = cvGet2D(image, i, j);
intensity[j+roiX*i] = n.val[0];
}
}

` The value of each pixel is stored in array

intensity[width*height]

24
` We can use
cvSampleLine() to img
sample one line in the
image. pt1
` int cvSampleLine(
IplImage* img,
CvPoint pt1, CvPoint
pt2, void* buffer );
◦ img Image. pt2
◦ pt1 Starting line point.
◦ pt2 Ending line point.
◦ buffer Buffer to store the cvSampleLine
line points; must have enough
size to store MAX(|pt2.x -
pt1.x| + 1,|pt2.y - pt1.y|+1)
points. buffer
…

25
 Start point End point
cvSampleLine(image, cvPoint(0, 120), cvPoint(319, 120), lineProfile->data.ptr);

for(int i=0;i<lineProfile->cols;i++)
{
int height = lineProfile->data.ptr[i];
int posX = i;

cvCircle(procImage, cvPoint(posX, height), 1, whiteLine);

}

cvShowImage("Processed
Sh I ("P d I
Image",
" procImage);
I )

26
` We can make a binary image by applying
threshold to an image.
` In
I OpenCV,
O CV we can use threshold
th h ld
cvThreshold()
` Types of threshold:
◦ Binary (CV_THRESH_BINARY)
x if src(x,y)>threshold Æ
d (
dst(x,y)=max,
) else
l 0
0;
◦ Inverted Binary
(CV_THRESH_BINARY_INV)
x if src(x,y)>threshold Æ
dst(x,y)=0, else max;
◦ Truncated (CV_THRESH_TRUNC)
x if src(x,y)>threshold Æ dst(x,y)=
threshold, else 0;
◦ Threshold to zero
(CV_THRESH_TOZERO)
x if src(x,y)>threshold Æ dst(x,y)=
( ,y), else 0;
src(x,y), ;
◦ Inverted Threshold to zero
(CV_THRESH_TOZERO_INV)
x if src(x,y)>threshold Æ
dst(x,y)=0, else src(x,y);

27
IplImage *thresImage = cvCreateImage(cvSize(image-
>width, image->height), image->depth, 1);

thresImage->origin = 1;
cvCvtColor(image, thresImage, CV_BGR2GRAY);
cvThreshold(thresImage, thresImage, 125, 255,
CV_THRESH_BINARY_INV);

cvShowImage("Threshold",
Sh I ("Th h ld" thresImage);
th I )
cvReleaseImage(&thresImage);

28
` There are several image filter that CV_BLUR
p
is available in OpenCV such as
gaussian filter, median filter,
bilateral filter, and blur.
` We can use this function:
` void cvSmooth( const CV_GAUSSIAN,
CvArr* src, CvArr* dst,
int
smoothtype=CV_GAUSSIAN,
int param1=3, int
param2=0);
◦ Src: The source image.
◦ Dst: The destination image.
◦ Smoothtype:
CV_BLUR_NO_SCALE, CV_BLUR,
CV_GAUSSIAN, CV_MEDIAN,
CV_BILATERAL
◦ Param1 and param2 are size of the
kernel

29
` Draw a box (100,100)
◦ cvRectangle(img,
g g, cvPoint(100,100),
, ,
cvPoint(200,200), cvScalar(255,0,0), 1);
` Draw a circle
◦ cvCircle(img, cvPoint(100,100), 20,
cvScalar(0,255,0), 1); (200,200)
( , )
` Draw a line segment 20
◦ cvLine(img, cvPoint(100,100), (100,100)
cvPoint(200,200), cvScalar(0,255,0), 1);
` Draw a set of p
polylines
y
◦ cvPolyLine(img,curveArr,nCurvePts,nCurve
s,isCurveClosed,cvScalar(0,255,255),line
Width);
(100,100)
` Draw a set of filled polygons
◦ cvFillPoly(img,curveArr,nCurvePts,nCurve
s,cvScalar(0,255,255)); (200,200)

30
Rectangle Circle

Line 31
` Opencv has a capability of acquiring images from
a camera such as webcam.
` This is implemented in cvcam class, in the newer
version
i off the
th library,
lib it iis iincluded
l d d iin highgui.
hi h i
` Therefore, we can implement and try our
developed algorithm in real time by writing our
code in the callback function of the image
acquisition.
q

32
` A function that will be
automatically called camera

by a generated event.
` For example in a
camera, a callback
Callback()
function is called
every time a new
image is captured by
the camera Algorithm

33
` Include files:
◦ #include <cvcam.h>
◦ #include <cv.h>
◦ #include <highgui.h>
` Check the number of camera in the system
◦ // R
Returns
t th
the number
b of
f available
il bl cameras i
in th
the
system.
◦ Int ncams = cvcamGetCamerasCount();
◦ // Exit the program when the video camera is not
a ailable
available.
◦ if( ncams < 1 )
◦ {
◦ Application->MessageBox("Can not find a video
!" "
camera!", " MB
"error", OK)
MB_OK);
◦ ShowMessage("Can not find a video camera!");
◦ exit(1);
◦ }

34
` Set the property of the camera
◦ cvcamSetProperty(cameraSelected,
cvcamSetProperty(cameraSelected
CVCAM_RNDWIDTH , &width);
◦ cvcamSetProperty(cameraSelected,
CVCAM_RNDHEIGHT , &height);
◦ cvcamSetProperty(cameraSelected,
CVCAM_PROP_ENABLE, &t);
◦ cvcamSetProperty(cameraSelected,
CVCAM PROP RENDER &t);
CVCAM_PROP_RENDER,
◦ cvcamSetProperty(cameraSelected,
CVCAM_PROP_CALLBACK, myCallBack);
` Start the camera
◦ cvcamInit();
◦ cvcamStart();

35
36
37
` Install the OpenCV Library into your computer.
` Create a new application with Image, OpenDialog and
two Button components.
` Create an IplImage object;
◦ IplImage * image;
` When the Button is pressed, initialize the IplImage to
load the image from file.
◦ Image = cvLoadImage(…); //read from the
reference
` When the second button is pressed, call the function to
save the image into another file
◦ if( img ) cvSaveImage( “picture.png”,
img );
38
` Using the same project file, create another button.
` When this button is pressed, convert the image from
Color to Grayscale using this function:
◦ cvCvtColor(bgr_image,
cvCvtColor(bgr image, grayscale_image,
grayscale image,
CV_BGR2GRAY);
` Add another button to perform filtering of an image. Use
this function:
◦ cvSmooth(image, image, CV_GAUSSIAN, 7,
7, 1., 1.); // inplace Gaussian
smoothing// with 7x7 kernel and σ=1.

39
` Create Histogram of the Image by accessing each
pixel of the image.
◦ Red
◦ Green
◦ Blue
◦ Grayscale

40
` Create a program to extract ROI from an image
` Use cvSetImageROI(IplImage *image, CvRect
rect);

Image x,y width

height ROI

41
` Create two IplImage objects.
` Create an ROI from the first image
` Copy the image into the second image

copy
ROI ROI

Image2
Image1

42
` Create an Application with
◦ One Timage
◦ One Tchart
◦ TButtons and TEdit
start end
` Define these variable:
◦ unsigned char * buffer;
◦ CvPoint start
start, end;
` Use this function:
◦ cvSampleLine( IplImage*
img
img, CvPoint pt1
pt1, CvPoint
pt2, void* buffer );
` Assign the value to the Tchart

43
` Draw a rectangle in an image
◦ First click sets the first point (use CvPoint)
◦ Second click sets the second point (use CvPoint)
◦ Then draw the rectangle from first point to second point (use
cvRectangle())
` Draw a circle
◦ First click sets the center of the image (use CvPoint)
◦ Second click sets the radius of the circle (use CvPoint and calculate the
radius)
◦ Then draw the circle using cvCircle(
Ci l ( … )
` Draw a line
◦ First click sets the first point (use CvPoint)
◦ Second click sets the second point (use CvPoint)
◦ Then draw the rectangle from first point to second point (use cvLine( …
))
` Next, use the mouse to draw on the image, draw rectangle, line
and circle

44
` Use the sample code that can be obtained from
this url:
http://140.124.201.17/opencv/
` The fil
Th filename: W
WebcamOpenCV.zip
b O CV i
` Open the project using your C++ Builder.
` Implement these function:
◦ color conversion
◦ Thresholding
g
◦ Drawing histogram of the image

45
` Getting Started with OpenCV by Vadim
Pisarevsky (vadim.pisarevsky@intel.com)
` http://www710.univ-lyon1.fr/~bouakaz/OpenCV-
0 9 5/docs/ref/OpenCVRef ImageProcessing htm
0.9.5/docs/ref/OpenCVRef_ImageProcessing.htm
` Introduction to programming with OpenCV, Gady
Agam,
g , http://www.cs.iit.edu/~agam/cs512/lect-
p g
notes/opencv-intro/opencv-intro.html

46
t6669036@ntut.edu.tw

47