Object Detection

Object Detection
TA : Young-geun Kim
Biostatistics Lab.,
Seoul National University
March-June, 2018
Seoul National University Deep Learning March-June, 2018 1 / 57

Index
1 Introduction
2 R-CNN
3 YOLO
4 Evaluation

Introduction
Introduction

Introduction
In this session, we will learn about...
The Object Detection problem.

Regions with CNN features (R-CNN), a region proposal-based
approach model.
You-Only-Look-Once (YOLO), an unified approach model.
Evaluation metrics for detection models.

Introduction
What is Object Detection?
Object Detection is a task finding where and what objects are

(Location + Classification).
An integral part of various vision application such as Automated
Driving System, Face Detection and Object Counting.
Figure: from https://youtu.be/MPU2HistivI (YOLO v3 clip).

Introduction
What is Object Detection? (Conti.)

For given image, task-taker should answer the predicted region and
class confidence.
A region is expressed as rectangular called bounding box.
The number of objects is not provided.
Figure: from Ren et al., 2015..

Introduction
Exact region (or bounding box) of each object is called Ground-Truth

(GT) box, the minimal rectangular containing whole part of the
object.
A region is parameterized by (x, y , w , h) where (x, y ) is the
coordinate of top-left (or center) point, w is the width, and h is the
height of the bounding box.
Intersection over Union (IoU), the ratio of intersection area to union
area, between predicted region and GT box presents the accuracy
about location.

Introduction
There are various types of objects.

For example, VOC challenge requires detecting following 20 kinds of
object classes.
Middle Level Low Level
Person person
Animal bird, cat, cow, dog, horse, sheep
aeroplane, bicycle, boat, bus, car,
Vehicle
motorbike, train
bottle, chair, diningtable, potted plant,
Indoor
sofa, tv/monitor
mean Average Precision (mAP), an estimator of the area under the

precision-recall curve (AUCPR), usually presents the accuracy about
classification.

Introduction
An object is considered detected if task-taker selects any region with

predicted label satisfying following conditions.
Condition 1 : Highly overlapped with GT box of the object.
Condition 2 : Correctly classified.

Introduction
Challenges
Infinitely Imbalanced Structure : Background (BG) is the majority

class. There are few positive regions (GT) and infinitely many
negative regions (BG).
Predicted
N P Total
N TN FP # of BG
True
P FN TP # of GT
Table: The confusion matrix of object detection.
In this structure, accuracy about positive class is severely suffered.

This means that finding an object position as it is difficult.

Introduction
Challenges (Conti.)
Dynamic Scale : Shape of objects is various. Some are tiny/huge

and some are horizontally/vertically long.
Figure: from VOC2012.

This means that our model should recognize various scale of regions.

Introduction
Challenges (Conti.)
Multi-task : Finding object position (Location) and classifying the

object (Classification) each is difficult. Object Detection requires
performing both tasks simultaneously.
In practice, the test time of detection model should be short, but due
to the high level of difficulty, it is challenging.

Introduction
Approaches
Pre-deep learning approaches (Do not cover. See 50 years of object

recognition: Directions forward).
Regions with CNN features (R-CNN), a region proposal-based
approach model.
You-Only-Look-Once (YOLO), an unified approach model.

R-CNN
R-CNN

R-CNN
Regions with CNN features
Regions with CNN features (R-CNN) is a region proposal-based

approach model (Girshick et el., 2014).
R-CNN selects regions using Selective Search (Uijlinga et al., 2013),
warps them as to the same scale and extracts features to learn
class-specific SVMs.
Figure: from Girshick et el., 2014.

R-CNN
Selective Search
Selective Search (SS) is an hierarchical grouping algorithm whose

domain is a set of region.
For given set of regions, SS greedily merges regions.
The distance measure is a partial summation of similarity about
colour, texture, size, and fill.
Figure: from Uijlinga et al., 2013.

R-CNN
Selective Search (Conti.)
Considering various features from fine-level region, SS distinguishes

objects and captures their hierarchical structure.
Initialization is based on a graph-based segmentation algorithm
(Felzenszwalb and Huttenlocher, 2004.) whose time complexity is
nearly linear in the number of pixels.

R-CNN
Detection Network
Proposed regions pass through CNNs which consists of classifier and

bounding box (bbox) regressor.
AlexNet (Krizhevsky et al., 2012.) is applied with replaced FC layer
for corresponding number of class including BG.
After tuning AlexNet, class-specific linear SVMs and bbox regressor
(Felzenszwalb et al., 2010.) are learned by using extracted feature.
bbox regressor predicts (x, y , w , h) of GT and use it to adjust
proposed regions.

R-CNN
Limitation of R-CNN
R-CNN requires fine-tuning CNN, learning multiple SVMs and bbox

regressor (multi-stage pipeline).
Training SVMs and bbox regressor requires feedforwarding all regions
in all images and saving all extracted features.
Because of the same reason in training, test time is too long. It takes
47 second to perform detection for a single image.

R-CNN
Spatial Pyramid Pooling Network

Feedforwarding all proposed regions is time-consuming approach.
Spatial Pyramid Pooling (SPP; He et al., 2014.) models the spatial
connectivity and makes various regions into the fixed size.
For usual CNNs, warp ◦ conv 6= conv ◦ warp since there is no spatial
connectivity between raw image and extracted feature.
Figure: from He et al., 2014.

R-CNN
Spatial Pyramid Pooling Network (Conti.)

SPP Network learns the spatial connectivity, still capturing semantic
content.
SPP reduces the computation cost, but SPP Network is still
multi-stage pipeline.
Figure: from He et al., 2014.

R-CNN
Fast R-CNN
Fast R-CNN (Girshick and Ross, 2015.) is a variation of R-CNN

applying Region of Interest (RoI) pooling, a kind of SPP.
Training is single-stage by using multi-task loss. Multi-task loss
enables us to update all weights simultaneously.
Figure: from Girshick and Ross, 2015.

R-CNN
Region of Interest Pooling
RoI pooling connects the raw image and the final extracted feature
before FC layers.
RoI feature vector passes two sibling FC layer.
Figure: from Girshick and Ross, 2015.

R-CNN
Region of Interest Pooling (Conti.)
In contrast to usual max-pooling, RoI pooling has dynamic filter size.

Back propagation through RoI pooling requires activated positions for
each region.

R-CNN
Multi-task Loss
For given region (xr , yr , wr , hr ) in an image, Fast R-CNN calculates p

and t k = (txk , tyk , twk , thk ), the predicted probability vector and location
for class k parameterized by following.
txk = (x k − xr )/wr
tyk = (y k − yr )/hr
twk = log(w k /wr )
thk = log(hk /hr )
Let u, v be the true class and location of corresponding GT box for

given region. v is parameterized by substituting (x, y , w , h) of the GT.

R-CNN
Multi-task Loss (Conti.)
To train multi-task model, the loss function is designed as
L(p, u, t u , v ) = Lcls (p, u) + λ[u ≥ 1]Lloc (t u , v )
where Lcls (p, u) = − log pu is log loss for true class u and
X
Lloc (t u , v ) = huber (tiu − vi ).
i∈{x,y ,w ,h}
The hyper-parameter λ controls balance between classification loss

and regression loss.
For u = 0, background region, Lloc doesn’t have any role.
Lloc is a function of |tiu − vi |, so L is invariant to translation, flipping
and rescaling.

R-CNN
Limitation of Fast R-CNN (Conti.)
Compared to R-CNN, Fast R-CNN achieves slightly higher accuracy

with nearly 100 times short test time, but the test time is still long.
In VOC 2007 test task, Fast R-CNN takes 1830ms per image.
Region proposal task, SS is a huge piece consuming 1510ms per
image.

R-CNN
Faster R-CNN
Faster R-CNN (Ren et al., 2015) is a variation of Fast R-CNN using

Region Proposal Network (RPN).
Roughly speaking, Faster R-CNN = RPN + Fast R-CNN.
Contrast to SS, RPN has learnable weight for multi-task loss.

R-CNN
Region Proposal Network
For given point in an image, RPN classifies objectness of several

regions centered on that point and regresses exact location.
Pre-determined points in each image are called anchors.
Figure: adjusted from VOC2012.

R-CNN
Region Proposal Network (Conti.)

1. For selected anchor, view small region nearby the anchor in the
level of extracted feature.
2. Determine the objectness of k regions centered on the
corresponding anchor in the raw image map.
Figure: from Ren et al., 2015.

R-CNN
Region Proposal Network (Conti.)
3. For all regions classified to be positive, adjust them using reg layer.
Figure: from Ren et al., 2015.

R-CNN
Multi-task loss for RPN
RPN uses multi-task loss similar to Fast R-CNN. Exact formula is

1 X 1 X ∗
L({pi }, {ti }) = Lcls (pi , pi∗ ) + λ pi Lreg (ti , ti∗ )
Ncls Nreg
i i
where i is the index of an anchor.

Here, pi is the predicted probability of anchor i being an object. pi∗ is
the ground-truth label. t is the same to Fast R-CNN.
(Opinion) This data has multi-label structure. Note that input domain
of loss is an anchor box, not anchor boxes sharing center. This
design relaxes the issue about class correlation between anchor boxes.

R-CNN
Training faster R-CNN
RPN and fast R-CNN share feature extractor part. This shared
structure reduces test-time, the origin of its name ”Faster R-CNN”.
Sharing structure is implemented by following sequence.
Phase Feature Extractor Region Proposal
Initialized from
1. Train RPN -
ImageNet model
Initialized from RPN from
2. Train fast R-CNN
ImageNet model phase 1.
Frozen from
3. Tune RPN -
phase 2.
Frozen from RPN from
4. Tune fast R-CNN
phase 2. phase 3.

R-CNN
Summary of R-CNN variations

All the models use bbox regressor to adjust proposed region. R-CNN
uses SVM and others use softmax classifier.
Region Proposal Method Region Scaling Method
R-CNN SS Warping
Fast R-CNN SS RoI pooling
Faster R-CNN RPN RoI pooling
Table: Key methodologies.
mAP (%) test time (ms/image)

R-CNN 66.0 > 104
Fast R-CNN 66.9 1830
Faster R-CNN 69.9 198
Table: Evaluation on VOC 2007 test set, adjusted from Girshick and Ross,
2015. and Ren et al., 2015.

YOLO
YOLO

YOLO
You-Only-Look-Once
You-Only-Look-Once (YOLO; Redmon et al., 2016.) is an unified

approach model. YOLO has one CNNs solving both location and
classification problem.
In the introduction of paper: ‘Humans glance at an image and
instantly know what objects are in the image, where they are, and
how they interact.‘’
For given image, YOLO feedforwards only one time, remarkably
reducing test time.
All the figures, tables, and equations in this section are come from
Redmon et al., 2016.

YOLO
Terminology
An image is divided by S × S grid cells.

If the center of an object falls into a grid cell, that grid cell is
responsible for detecting that object.

YOLO
Terminology (Conti.)
Each grid cell predicts B bounding boxes and corresponding

objectness confidence.
Each bounding box is parametrized by (x, y , w , h), the same to
R-CNN.
truth .
The objectness confidence is Pr (Object) ∗ IOUpred

YOLO
All bounding boxes sharing grid cell have the same conditional class
probability, formally Pr (Classi |Object).
truth is
At test time, the class-specific confidence, Pr (Classi ) ∗ IoUpred
predicted by multiplying predicted conditional class confidence and
objectness confidence.

YOLO

YOLO
Architecture
For given image, YOLO predicts (x, y , w , h) and objectness

confidence for all bounding boxes and conditional class probability for
all grid cells.
Considering its spatial meaning, we can view the output as
S × S × (B ∗ 5 + C ) box.
In VOC competition, S = 7, B = 2, and C = 20.

YOLO
Architecture (Conti.)
Following figure describes the architecture of YOLO.

For given image, convolution layers extract features and final FC layer
predicts bounding box parameters, objectness confidence, and
conditional class probability.

YOLO
Loss
Following is the loss function of YOLO. The first two terms are about
bbox regression. Next two terms are about objectness classification
and the last term is about the class classification.
Here, 1i and 1ij are indicators about responsibility of ith grid cell and
its jth bounding box, respectively.
YOLO
Performance
YOLO is the first deep-learning model in the context of real-time

detection with the state-of-the-art accuracy.
Real-Time : 30 frames per second or better. When the speed of car is
60km/h, car moves 0.55m between detections.

YOLO
Performance (Conti.)
Compared with fast R-CNN, YOLO has high location error and low
background error.
Correct: correct class and IoU >.5, Loc: correct class, .1<IoU<.5,
Sim: class is similar, IoU>.1, Other: class is wrong, IoU>.1,
Background: IoU<.1 for any object.

Evaluation
Evaluation

Evaluation
Non Maximum Suppression
Some of predicted regions severely overlap.

In object detection, multiple detection for single GT is penalized.
Figure: from
https://kr.mathworks.com/help/vision/ref/selectstrongestbbox.html

Evaluation
Non Maximum Suppression (Conti.)

Non Maximum Suppression (NMS) is a pre-work for evaluation,
removing overlapped regions using confidence.
Choose the most confident bounding box and remove all other boxes
with high IoU with the box. Repeat until there is no more box.
NMS is applied to both RPN and detection network.
Figure: from
https://kr.mathworks.com/help/vision/ref/selectstrongestbbox.html.
Evaluation
Evaluation measures
In Infinitely Imbalance Structure, performance measures using TN

may unsuitable.
Predicted
N P Total
N TN FP # of BG
True
P FN TP # of GT
Table: The confusion matrix of object detection.
Detecting all objects as it is easy. Just classify all regions to all object.
What would be the value of TN? If the model is reasonable, TN
should be ∞.

Evaluation
Evaluation measures (Conti.)
Main evaluation measures in object detection are based on Precision

and Recall.
Precision : the proportion of TP among positive labeled,
TP/(TP+FP).
Recall : the proportion of TP among positive, TP/(TP+FN).
F1 score : the harmonic mean of precision and recall.
AUCPR : the area under the precision-recall curve. Commonly used
estimator for AUCPR in object detection is Average Precision (AP).

Evaluation
Average Precision
Let c be the threshold of confidence. Than AUCPR can be expressed

as Z ∞
AUCPR = Precision(c)dRecall(c)
−∞
where Precision(c) and Recall(c) are the precision and recall at

threshold level c, respectively.
\
By plugging in the empirical precision and recall, Precision(c) and
\
Recall(c), we get an estimator of AUCPR,
Z ∞
\ =
AUCPR \
Precision(c)d \
Recall(c).
−∞

Evaluation
Average Precision (Conti.)
Here, by the definition of Riemann−Stieltjes integral,

Z ∞
\ =
AUCPR \
Precision(c)d \
Recall(c)
−∞
X
\ # of P have conf . equal to c
= Precision(c) .
# of P
c∈{confi |i∈P}
This is the Average Precision (AP), an weighted average of precisions

at each confidence level of GT box.

Evaluation
Average Precision (Conti.)
Considering various kinds of class, the mean of AP is used. This is

called mean Average Precision (mAP).
In practice, Interpolated AP is used due to the wiggles in the
precision-recall curve. Unlike the ROC curve, it may not hold
monotonicity.

Evaluation
References
Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014). Rich
feature hierarchies for accurate object detection and semantic
segmentation. In Proceedings of the IEEE conference on computer
vision and pattern recognition (pp. 580-587).
Uijlings, Jasper RR, et al. ”Selective search for object recognition.”
International journal of computer vision 104.2 (2013): 154-171.
Felzenszwalb, Pedro F., and Daniel P. Huttenlocher. ”Efficient
graph-based image segmentation.” International journal of computer
vision 59.2 (2004): 167-181
Felzenszwalb, Pedro F., et al. ”Object detection with discriminatively
trained part-based models.” IEEE transactions on pattern analysis and
machine intelligence 32.9 (2010): 1627-1645.

Evaluation
References (Conti.)
A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification

with deep convolutional neural networks. In NIPS, 2012.
He, Kaiming, et al. ”Spatial pyramid pooling in deep convolutional
networks for visual recognition.” european conference on computer
vision. Springer, Cham, 2014.
Girshick, Ross. ”Fast r-cnn.” arXiv preprint arXiv:1504.08083 (2015).
Simonyan, Karen, and Andrew Zisserman. ”Very deep convolutional
networks for large-scale image recognition.” arXiv preprint
arXiv:1409.1556 (2014).

Evaluation
References (Conti.)
Ren, Shaoqing, et al. ”Faster r-cnn: Towards real-time object

detection with region proposal networks.” Advances in neural
information processing systems. 2015.
Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You
only look once: Unified, real-time object detection. In Proceedings of
the IEEE conference on computer vision and pattern recognition (pp.
779-788).
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and
Zisserman, A. (2010). The pascal visual object classes (voc)
challenge. International journal of computer vision, 88(2), 303-338.

Evaluation
References (Conti.)
Boyd, Kendrick, Kevin H. Eng, and C. David Page. ”Area under the
precision-recall curve: Point estimates and confidence intervals.” Joint
European Conference on Machine Learning and Knowledge Discovery
in Databases. Springer, Berlin, Heidelberg, 2013.
Introduction to modern information retrieval

Object Detection

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Object Detection

Uploaded by

Copyright:

Available Formats

Object Detection

Seoul National University Deep Learning March-June, 2018 1 / 57

Seoul National University Deep Learning March-June, 2018 2 / 57

Seoul National University Deep Learning March-June, 2018 3 / 57

In this session, we will learn about...

The Object Detection problem.

Seoul National University Deep Learning March-June, 2018 4 / 57

What is Object Detection?

Object Detection is a task finding where and what objects are

Figure: from https://youtu.be/MPU2HistivI (YOLO v3 clip).

Seoul National University Deep Learning March-June, 2018 5 / 57

What is Object Detection? (Conti.)

Figure: from Ren et al., 2015..

What is Object Detection? (Conti.)

Exact region (or bounding box) of each object is called Ground-Truth

Seoul National University Deep Learning March-June, 2018 7 / 57

What is Object Detection? (Conti.)

There are various types of objects.

mean Average Precision (mAP), an estimator of the area under the

Seoul National University Deep Learning March-June, 2018 8 / 57

What is Object Detection? (Conti.)

An object is considered detected if task-taker selects any region with

Seoul National University Deep Learning March-June, 2018 9 / 57

Infinitely Imbalanced Structure : Background (BG) is the majority

In this structure, accuracy about positive class is severely suffered.

Seoul National University Deep Learning March-June, 2018 10 / 57

Dynamic Scale : Shape of objects is various. Some are tiny/huge

Figure: from VOC2012.

Seoul National University Deep Learning March-June, 2018 11 / 57

Multi-task : Finding object position (Location) and classifying the

Seoul National University Deep Learning March-June, 2018 12 / 57

Pre-deep learning approaches (Do not cover. See 50 years of object

Seoul National University Deep Learning March-June, 2018 13 / 57

Seoul National University Deep Learning March-June, 2018 14 / 57

Regions with CNN features

Regions with CNN features (R-CNN) is a region proposal-based

Figure: from Girshick et el., 2014.

Seoul National University Deep Learning March-June, 2018 15 / 57

Selective Search (SS) is an hierarchical grouping algorithm whose

Figure: from Uijlinga et al., 2013.

Seoul National University Deep Learning March-June, 2018 16 / 57

Selective Search (Conti.)

Considering various features from fine-level region, SS distinguishes

Seoul National University Deep Learning March-June, 2018 17 / 57

Proposed regions pass through CNNs which consists of classifier and

Seoul National University Deep Learning March-June, 2018 18 / 57

R-CNN requires fine-tuning CNN, learning multiple SVMs and bbox

Seoul National University Deep Learning March-June, 2018 19 / 57

Spatial Pyramid Pooling Network

Figure: from He et al., 2014.

Spatial Pyramid Pooling Network (Conti.)

Figure: from He et al., 2014.

Fast R-CNN (Girshick and Ross, 2015.) is a variation of R-CNN

Figure: from Girshick and Ross, 2015.

Seoul National University Deep Learning March-June, 2018 22 / 57

Region of Interest Pooling

Figure: from Girshick and Ross, 2015.

Seoul National University Deep Learning March-June, 2018 23 / 57

Region of Interest Pooling (Conti.)

In contrast to usual max-pooling, RoI pooling has dynamic filter size.

Seoul National University Deep Learning March-June, 2018 24 / 57

For given region (xr , yr , wr , hr ) in an image, Fast R-CNN calculates p

Let u, v be the true class and location of corresponding GT box for