Object Detection using Python OpenCV

Saturday, March 30, 2019 8:33 PM

Clipped from: https://circuitdigest.com/tutorial/object-detection-using-

python-opencv

Object Detection using Python & OpenCV

We started with learning basics of OpenCV and then done some basic image
processing and manipulations on images followed by Image segmentations
and many other operations using OpenCV and python language. Here, in this
section, we will perform some simple object detection techniques using
template matching. We will find an object in an image and then we will
describe its features. Features are the common attributes of the image such
as corners, edges etc. We will also take a look at some common and popular
object detection algorithms such as SIFT, SURF, FAST, BREIF & ORB.

As told in the previous tutorials, OpenCV is Open Source Commuter

Vision Library which has C++, Python and Java interfaces and supports
Windows, Linux, Mac OS, iOS and Android. So it can be easily installed
in Raspberry Pi with Python and Linux environment. And Raspberry Pi with

Food Recognizer Page 1

in Raspberry Pi with Python and Linux environment. And Raspberry Pi with
OpenCV and attached camera can be used to create many real-time image
processing applications like Face detection, face lock, object tracking, car
number plate detection, Home security system etc.

Object detection and recognition form the most important use case for
computer vision, they are used to do powerful things such as

• Labelling scenes
• Robot Navigation
• Self-driving cars
• Body recognition (Microsoft Kinect)
• Disease and cancer detection
• Facial recognition
• Handwriting recognition
• Identifying objects in satellite images

Object Detection VS Recognition

Object recognition is the second level of object detection in which computer

is able to recognize an object from multiple objects in an image and may be
able to identify it.

Now, we will perform some image processing functions to find an object

from an image.

Finding an Object from an Image

Here we will use template matching for finding character/object in an image,

use OpenCV’s cv2.matchTemplate() function for finding that object

import cv2
import numpy as np

Load input image and convert it into gray

image=cv2.imread('WaldoBeach.jpg')
cv2.imshow('people',image)
cv2.waitKey(0)
gray=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)

Load the template image

template=cv2.imread('waldo.jpg',0)
#result of template matching of object over an image

Food Recognizer Page 2

#result of template matching of object over an image
result=cv2.matchTemplate(gray,template,cv2.TM_CCOEFF)
sin_val, max_val, min_loc, max_loc=cv2.minMaxLoc(result)

Create bounding box

top_left=max_loc
#increasing the size of bounding rectangle by 50 pixels
bottom_right=(top_left[0]+50,top_left[1]+50)
cv2.rectangle(image, top_left, bottom_right, (0,255,0),5)

cv2.imshow('object found',image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Food Recognizer Page 3

In cv2.matchTemplate(gray,template,cv2.TM_CCOEFF), input the gray-scale
image to find the object and template. Then apply the template matching
method for finding the objects from the image, here cv2.TM_CCOEFF is
used.

The whole function returns an array which is inputted in result, which is the
result of the template matching procedure.

And then we use cv2.minMaxLoc(result), which gives the coordinates or the

bounding box where the object was found in an image, and when we get
those coordinates draw a rectangle over it, and stretch a little dimensions of
the box so the object can easily fit inside the rectangle.

There are variety of methods to perform template matching and in this case
we are using cv2.TM_CCOEFF which stands for correlation coefficient.

cv2.matchTemplate takes a “sliding window” of the object and slides it over

the image from left to right and top to bottom, one pixel at a time. Then for
each location, we compute the correlation coefficient to determine how
“good” or “bad” the match is.

Regions with sufficiently high correlation can be considered as matches, from

there all we need is to call to cv2.minMaxLoc to find where the good
matches are in template matching.

Food Recognizer Page 4

matches are in template matching.

Feature Description Theory

In template matching we slide a template image across a source image until

a match is found. But it is not the best method for object recognition, as it
has severe limitations. This method isn’t very resilient.

The following factors make template matching a bad choice for object
detection.

• Rotation renders this method ineffective.

• Size (known as scaling) affects this as well.
• Photometric changes (e.g. brightness, contrast, hue etc.)
• Distortion form view point changes (Affine).
The one solution for this problem is image features

Image features are interesting areas of an image that are somewhat

unique to that specific image. They are also called key point features or
interest points.

The sky is an uninteresting feature, whereas as certain keypoints (marked in

red circles) can be used for the detection of the above image (interesting
Features). The image shown above clearly shows the difference between the
interesting feature and uninteresting feature.

Importance of feature detection

Food Recognizer Page 5

Features are important as they can be used to analyze, describe and match
images. They have extensive use in:

• Image alignment – e.g panorma stiching (finding corresponding

matches so we can stitch images together)
• 3D reconstruction
• Robot navigation
• Object recognition
• Motion tracking
• And more!

What defines the interest points?

Interesting areas carry a lot of distinct information and unique information of

Food Recognizer Page 6
Interesting areas carry a lot of distinct information and unique information of
an area. Typically, they are areas of high change of intensity, corners or
edges and more. But always be careful as noise can appear “informative”
when it is not! So try to blur so as to reduce noise.

Characteristic of Good or Interesting Features

Repeatable – They can be found in multiple pictures of the same scene.

Distinctive – Each feature is somewhat unique and different to other

features of the same scene.

Compactness/Efficiency – Significantly less features than pixels in the

image.

Locality – Feature occupies a small area of the image and is robust to

clutter and occlusion.

Food Recognizer Page 7

Corners as features

Corners are identified when shifting a window in any direction over that
point gives a large change in intensity.

Corners are not the best cases for identifying the images, but yes they have
certainly good use cases of them which make them handy to use.

So to identify corners in your image, imagine the green window we are

looking at and the black one is the image we want to find corners in, and
now when we move the window only inside the black box we see there is no
change in intensity and hence the image is flat i.e. no corners identified.

Now when we move the window in one direction we see that there is change
of intensity in one direction only, hence it’s an edge not a corner.

When we move the window in the corner, and no matter in what direction

Food Recognizer Page 8

When we move the window in the corner, and no matter in what direction
we move the window now there is a change in intensity, and this is identified
as a corner.

So let’s identify corner with the help of Harris Corner Detection algorithm,
developed in 1998 for corner detection and works fairly well.

The following OpenCV function is used for the detection of the corners.

cv2.cornerHarris(input image, block size, ksize, k)

Input image - Should be grayscale and float32 type.

blockSize - The size of neighborhood considered for corner detection

ksize - Aperture parameter of Sobel derivative used.

k - Harris detector free parameter in the equation

Output – array of corner locations (x,y)

Also an important thing to note is that Harris corner detection algorithm

requires a float 32 array datatype of image, i.e. image should be gray
image of float 32 type.

import cv2
import numpy as np

Load image then grayscale

image = cv2.imread('chess.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

The cornerHarris function requires the array datatype to be float32

gray = np.float32(gray)
harris_corners = cv2.cornerHarris(gray, 3, 3, 0.05)

We use dilation of the corner points to enlarge them

kernel = np.ones((7,7),np.uint8)
harris_corners = cv2.dilate(harris_corners, kernel, iterations = 2)

Threshold for an optimal value, it may vary depending on the image

Food Recognizer Page 9

Threshold for an optimal value, it may vary depending on the image

image[harris_corners > 0.025 * harris_corners.max() ] = [255, 127,

127]

cv2.imshow('Harris Corners', image)

cv2.waitKey(0)
cv2.destroyAllWindows()

Corner Harris returns the location of the corners, so as to visualize these

tiny locations we use dilation so as to add pixels to the edges of the corners.
So to enlarge the corner we run the dilation twice. And then we again do
some thresholding to change the colors of the corners.

The following function is used for the same with the below mentioned
parameters

cv2.goodFeaturesToTrack(input image, maxCorners, qualityLevel,

minDistance)

Input Image - 8-bit or floating-point 32-bit, single-channel image.

Food Recognizer Page 10

 • Input Image - 8-bit or floating-point 32-bit, single-channel image.
• maxCorners – Maximum number of corners to return. If there are
more number of corners than the total numbers of corners which are
actually found, then the strongest one of them is returned.
• qualityLevel – Parameter characterizing the minimal accepted quality
of image corners. The parameter value is multiplied by the best corner
quality measure (smallest eigenvalue). The corners with the quality
measure less than the product are rejected. For example, if the best
corner has the quality measure = 1500, and the qualityLevel=0.01 ,
then all the corners with the quality measured less than 15 are rejected.
• minDistance – Minimum possible Euclidean distance between the
returned corners.

import cv2
import numpy as np

img = cv2.imread('chess.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

We specify the top 50 corners

corners = cv2.goodFeaturesToTrack(gray, 100, 0.01, 15)

for corner in corners:

x, y = corner[0]
x = int(x)
y = int(y)
cv2.rectangle(img,(x-10,y-10),(x+10,y+10),(0,255,0), 2)

cv2.imshow("Corners Found", img)

cv2.waitKey()
cv2.destroyAllWindows()

It also returns the array of location of the corners like previous method, so
we iterate through each of the corner position and plot a rectangle over it.

Food Recognizer Page 11

Problems with corners as features
Corner matching in images is tolerant of or corner detection don’t
have any problem with image detection when the image is
• Rotated
• Translated (i.e. shifts in image)
• Slight photometric changes e.g. brightness
or affine intensity

However, it is intolerant of:

• Large changes in intensity or photometric
changes)
• Scaling (i.e. enlarging or shrinking)

Food Recognizer Page 12

SIFT, SURF, FAST, BRIEF & ORB Algorithms

Scale Invariant Feature Transform (SIFT)

The corner detectors like Harris corner detection algorithm are rotation
invariant, which means even if the image is rotated we could still get the
same corners. It is also obvious as corners remain corners in rotated image
also. But when we scale the image, a corner may not be the corner as
shown in the above image.

SIFT is used to detect interesting keypoints in an image using the difference

of Gaussian method, these are the areas of the image where variation
exceeds a certain threshold and are better than edge descriptor.

Then we create a vector descriptor for these interesting areas. And the scale
Invariance is achieved via the following process:
i. Interesting points are scanned at several different scales.
ii. The scale at which we meet a specific stability criteria, is then
selected and encoded by the vector descriptor. Therefore, regardless of the
initial size, the more stable scale is found which allows us to be scale
invariant.

Rotation invariance is achieved by obtaining the Orientation

Assignment of the key point using image gradient magnitudes. Once we
know the 2D direction, we can normalize this direction.

Food Recognizer Page 13

know the 2D direction, we can normalize this direction.

A full paper on SIFT can be read here:

http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf.

And you can also find a tutorial on the official OpenCV link.

Food Recognizer Page 14

Speeded Up Robust Features (SURF)

SURF is the speeded up version of SIFT, as the SIFT is quite computational

expensive

SURF was developed to improve the speed of a scale invariant feature

detector. Instead of using the Difference of Gaussian approach, SURF uses
Hessian matrix approximation to detect interesting points and uses the sum
of Haar wavelet responses for orientation assignment.

A full paper on SIFT can be read here: http://www.vision.ee.ethz.ch/

~surf/eccv06.pdf

Alternatives of SIFT and SURF

As the SIFT and SURF are patented they are not freely available for
commercial use however there are alternatives to these algorithms which
are explained in brief here

Features from Accelerated Segment Test (FAST)

• Key point detection only (no descriptor, we can use SIFT or SURF to
compute that)
• Used in real time applications

Here you can find the papers on FAST

https://www.edwardrosten.com/work/rosten_2006_machine.pdf

Binary Robust Independent Elementary Features (BRIEF)

• Computers descriptors quickly (instead of using SIFT or SURF)

• it is quite fast.

Here you can find the paper on BRIEF

http://cvlabwww.epfl.ch/~lepetit/papers/calonder_pami11.pdf

Oriented FAST and Rotated BRIEF (ORB)

• Developed out of OpenCV Labs (not patented so free to use!)

• Combines both Fast and Brief
Here you can find the paper on ORB

http://www.willowgarage.com/sites/default/files/orb_final.pdf

Food Recognizer Page 15

Using SIFT, SURF, FAST, BRIEF & ORB in OpenCV

Flow process for SIFT, SURF, FAST, BRIEF & ORB

Feature Detection implementation

The SIFT & SURF algorithms are patented by their respective creators, and
while they are free to use in academic and research settings, you should
technically be obtaining a license/permission from the creators if you are
using them in a commercial (i.e. for-profit) application.

Below we are explaining programming examples of all the

algorithms mentioned above.

SIFT

import cv2
import numpy as np

image = cv2.imread('paris.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Food Recognizer Page 16

Create SIFT Feature Detector object
sift = cv2.xfeatures2d.SIFT_create()

#Detect key points

keypoints = sift.detect(gray, None)
print("Number of keypoints Detected: ", len(keypoints))

Draw rich key points on input image

image = cv2.drawKeypoints(image, keypoints, None,

flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)

cv2.imshow('Feature Method - SIFT', image)

cv2.waitKey(0)
cv2.destroyAllWindows()

Console Output:

Number of keypoints Detected: 1893

Here the keypoints are (X,Y) coordinates extracted using sift detector and
drawn over the image using cv2 draw keypoint function.

Food Recognizer Page 17

SURF

import cv2
import numpy as np

image = cv2.imread('paris.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Create SURF Feature Detector object, here we set hessian threshold

to 500

# Only features, whose hessian is larger than hessianThreshold are retained

by the detector

#you can increase the value of hessian threshold to decrease the keypoints

surf = cv2.xfeatures2d.SURF_create(500)

keypoints, descriptors = surf.detectAndCompute(gray, None)

print ("Number of keypoints Detected: ", len(keypoints))

Draw rich key points on input image

image = cv2.drawKeypoints(image, keypoints, None,

flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)

cv2.imshow('Feature Method - SURF', image)

cv2.waitKey()
cv2.destroyAllWindows()

Console Output:

Number of keypoints Detected: 1548

Food Recognizer Page 18

FAST

import cv2
import numpy as np

image = cv2.imread('paris.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Create FAST Detector object

fast = cv2.FastFeatureDetector_create()
# Obtain Key points, by default non max suppression is On
# to turn off set fast.setBool('nonmaxSuppression', False)
keypoints = fast.detect(gray, None)
print ("Number of keypoints Detected: ", len(keypoints))

Draw rich keypoints on input image

image = cv2.drawKeypoints(image, keypoints, None,

flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)

cv2.imshow('Feature Method - FAST', image)

cv2.waitKey()
cv2.destroyAllWindows()

Console Output:

Food Recognizer Page 19

Console Output:

Number of keypoints Detected: 8960

BRIEF

import cv2
import numpy as np

image = cv2.imread('paris.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Create FAST detector object

brief = cv2.xfeatures2d.BriefDescriptorExtractor_create()

Create BRIEF extractor object

#brief = cv2.DescriptorExtractor_create("BRIEF")

# Determine key points

keypoints = fast.detect(gray, None)

Obtain descriptors and new final keypoints using BRIEF

keypoints, descriptors = brief.compute(gray, keypoints)

print ("Number of keypoints Detected: ", len(keypoints))
Food Recognizer Page 20
print ("Number of keypoints Detected: ", len(keypoints))

Draw rich keypoints on input image

image = cv2.drawKeypoints(image, keypoints, None,

flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)

cv2.imshow('Feature Method - BRIEF', image)

cv2.waitKey()
cv2.destroyAllWindows()

Console Output:

Number of keypoints Detected: 8735

ORB

import cv2
import numpy as np

image = cv2.imread('paris.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

Create ORB object, we can specify the number of key points we

desire

orb = cv2.ORB_create()

Food Recognizer Page 21

orb = cv2.ORB_create()
# Determine key points
keypoints = orb.detect(gray, None)

Obtain the descriptors

keypoints, descriptors = orb.compute(gray, keypoints)

print("Number of keypoints Detected: ", len(keypoints))

Draw rich keypoints on input image

image = cv2.drawKeypoints(image, keypoints, None,

flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)

cv2.imshow('Feature Method - ORB', image)

cv2.waitKey()
cv2.destroyAllWindows()

Console Output:

Number of keypoints Detected: 500

We can specify the number of keypoints which has maximum limit of 5000,
however the default value is 500, i.e. ORB automatically would detect best
500 keypoints if not specified for any value of keypoints.

So this is how object detection takes place in OpenCV, the same programs

Food Recognizer Page 22

So this is how object detection takes place in OpenCV, the same programs
can also be run in OpenCV installed Raspberry Pi and can be used as a
portable device like Smartphones having Google Lens.

Food Recognizer Page 23