Professional Documents
Culture Documents
articolDalalTriggs PDF
articolDalalTriggs PDF
1
Normalize Weighted vote Contrast normalize Collect HOGs Person /
Input gamma &
Compute Linear
gradients into spatial & over overlapping over detection nonperson
image SVM classification
colour orientation cells spatial blocks window
Figure 1. An overview of our feature extraction and object detection chain. The detector window is tiled with a grid of overlapping blocks
in which Histogram of Oriented Gradient feature vectors are extracted. The combined vectors are fed to a linear SVM for object/non-object
classification. The detection window is scanned across the image at all positions and scales, and conventional non-maximum suppression
is run on the output pyramid to detect object instances, but this paper concentrates on the feature extraction process.
edge directions, even without precise knowledge of the cor- coarse spatial sampling, fine orientation sampling and strong
responding gradient or edge positions. In practice this is im- local photometric normalization turns out to be the best strat-
plemented by dividing the image window into small spatial egy, presumably because it permits limbs and body segments
regions (cells), for each cell accumulating a local 1-D his- to change appearance and move from side to side quite a lot
togram of gradient directions or edge orientations over the provided that they maintain a roughly upright orientation.
pixels of the cell. The combined histogram entries form the
representation. For better invariance to illumination, shad- 4 Data Sets and Methodology
owing, etc., it is also useful to contrast-normalize the local
responses before using them. This can be done by accumu- Datasets. We tested our detector on two different data sets.
lating a measure of local histogram energy over somewhat The first is the well-established MIT pedestrian database
larger spatial regions (blocks) and using the results to nor- [18], containing 509 training and 200 test images of pedestri-
malize all of the cells in the block. We will refer to the nor- ans in city scenes (plus left-right reflections of these). It con-
malized descriptor blocks as Histogram of Oriented Gradi- tains only front or back views with a relatively limited range
ent (HOG) descriptors. Tiling the detection window with of poses. Our best detectors give essentially perfect results
a dense (in fact, overlapping) grid of HOG descriptors and on this data set, so we produced a new and significantly more
using the combined feature vector in a conventional SVM challenging data set, INRIA, containing 1805 64128 im-
based window classifier gives our human detection chain ages of humans cropped from a varied set of personal pho-
(see fig. 1). tos. Fig. 2 shows some samples. The people are usually
The use of orientation histograms has many precursors standing, but appear in any orientation and against a wide
[13,4,5], but it only reached maturity when combined with variety of background image including crowds. Many are
local spatial histogramming and normalization in Lowes bystanders taken from the image backgrounds, so there is no
Scale Invariant Feature Transformation (SIFT) approach to particular bias on their pose. The database is available from
wide baseline image matching [12], in which it provides http://lear.inrialpes.fr/data for research purposes.
the underlying image patch descriptor for matching scale- Methodology. We selected 1239 of the images as positive
invariant keypoints. SIFT-style approaches perform remark- training examples, together with their left-right reflections
ably well in this application [12,14]. The Shape Context (2478 images in all). A fixed set of 12180 patches sampled
work [1] studied alternative cell and block shapes, albeit ini- randomly from 1218 person-free training photos provided
tially using only edge pixel counts without the orientation the initial negative set. For each detector and parameter com-
histogramming that makes the representation so effective. bination a preliminary detector is trained and the 1218 nega-
The success of these sparse feature based representations has tive training photos are searched exhaustively for false posi-
somewhat overshadowed the power and simplicity of HOGs tives (hard examples). The method is then re-trained using
as dense image descriptors. We hope that our study will help this augmented set (initial 12180 + hard examples) to pro-
to rectify this. In particular, our informal experiments sug- duce the final detector. The set of hard examples is subsam-
gest that even the best current keypoint based approaches are pled if necessary, so that the descriptors of the final training
likely to have false positive rates at least 12 orders of mag- set fit into 1.7 Gb of RAM for SVM training. This retrain-
nitude higher than our dense grid approach for human detec- ing process significantly improves the performance of each
tion, mainly because none of the keypoint detectors that we detector (by 5% at 104 False Positives Per Window tested
are aware of detect human body structures reliably. (FPPW) for our default detector), but additional rounds of
The HOG/SIFT representation has several advantages. It retraining make little difference so we do not use them.
captures edge or gradient structure that is very characteristic To quantify detector performance we plot Detection Er-
of local shape, and it does so in a local representation with ror Tradeoff (DET) curves on a log-log scale, i.e. miss rate
an easily controllable degree of invariance to local geometric ( 1Recall or TruePos+FalseNeg
FalseNeg
) versus FPPW. Lower val-
and photometric transformations: translations or rotations ues are better. DET plots are used extensively in speech and
make little difference if they are much smaller that the local in NIST evaluations. They present the same information as
spatial or orientation bin size. For human detection, rather Receiver Operating Characteristics (ROCs) but allow small
Figure 2. Some sample images from our new human detection database. The subjects are always upright, but with some partial occlusions
and a wide range of variations in pose, appearance, clothing, illumination and background.
probabilities to be distinguished more easily. We will often strength and edge-presence based voting were tested, with
use miss rate at 104 FPPW as a reference point for results. the edge threshold chosen automatically to maximize detec-
This is arbitrary but no more so than, e.g. Area Under ROC. tion performance (the values selected were somewhat vari-
In a multiscale detector it corresponds to a raw error rate of able, in the region of 2050 graylevels).
about 0.8 false positives per 640480 image tested. (The full Results. Fig. 3 shows the performance of the various detec-
detector has an even lower false positive rate owing to non- tors on the MIT and INRIA data sets. The HOG-based de-
maximum suppression). Our DET curves are usually quite tectors greatly outperform the wavelet, PCA-SIFT and Shape
shallow so even very small improvements in miss rate are Context ones, giving near-perfect separation on the MIT test
equivalent to large gains in FPPW at constant miss rate. For set and at least an order of magnitude reduction in FPPW
example, for our default detector at 1e-4 FPPW, every 1% on the INRIA one. Our Haar-like wavelets outperform MIT
absolute (9% relative) reduction in miss rate is equivalent to wavelets because we also use 2nd order derivatives and con-
reducing the FPPW at constant miss rate by a factor of 1.57. trast normalize the output vector. Fig. 3(a) also shows MITs
best parts based and monolithic detectors (the points are in-
5 Overview of Results
terpolated from [17]), however beware that an exact compar-
Before presenting our detailed implementation and per- ison is not possible as we do not know how the database in
formance analysis, we compare the overall performance of [17] was divided into training and test parts and the nega-
our final HOG detectors with that of some other existing tive images used are not available. The performances of the
methods. Detectors based on rectangular (R-HOG) or cir- final rectangular (R-HOG) and circular (C-HOG) detectors
cular log-polar (C-HOG) blocks and linear or kernel SVM are very similar, with C-HOG having the slight edge. Aug-
are compared with our implementations of the Haar wavelet, menting R-HOG with primitive bar detectors (oriented 2nd
PCA-SIFT, and shape context approaches. Briefly, these ap- derivatives R2-HOG) doubles the feature dimension but
proaches are as follows: further improves the performance (by 2% at 104 FPPW).
Generalized Haar Wavelets. This is an extended set of ori- Replacing the linear SVM with a Gaussian kernel one im-
ented Haar-like wavelets similar to (but better than) that used proves performance by about 3% at 104 FPPW, at the cost
in [17]. The features are rectified responses from 99 and of much higher run times1 . Using binary edge voting (EC-
1212 oriented 1st and 2nd derivative box filters at 45 inter- HOG) instead of gradient magnitude weighted voting (C-
vals and the corresponding 2nd derivative xy filter. HOG) decreases performance by 5% at 104 FPPW, while
omitting orientation information decreases it by much more,
PCA-SIFT. These descriptors are based on projecting gradi- even if additional spatial or radial bins are added (by 33% at
ent images onto a basis learned from training images using 104 FPPW, for both edges (E-ShapeC) and gradients (G-
PCA [11]. Ke & Sukthankar found that they outperformed ShapeC)). PCA-SIFT also performs poorly. One reason is
SIFT for key point based matching, but this is controversial that, in comparison to [11], many more (80 of 512) principal
[14]. Our implementation uses 1616 blocks with the same vectors have to be retained to capture the same proportion of
derivative scale, overlap, etc., settings as our HOG descrip- the variance. This may be because the spatial registration is
tors. The PCA basis is calculated using positive training im- weaker when there is no keypoint detector.
ages.
Shape Contexts. The original Shape Contexts [1] used bi-
6 Implementation and Performance Study
nary edge-presence voting into log-polar spaced bins, irre- We now give details of our HOG implementations and
spective of edge orientation. We simulate this using our C- systematically study the effects of the various choices on de-
HOG descriptor (see below) with just 1 orientation bin. 16 1 We use the hard examples generated by linear R-HOG to train the ker-
angular and 3 radial intervals with inner radius 2 pixels and nel R-HOG detector, as kernel R-HOG generates so few false positives that
outer radius 8 pixels gave the best results. Both gradient- its hard example set is too sparse to improve the generalization significantly.
DET different descriptors on MIT database DET different descriptors on INRIA database
0.2 0.5
Lin. RHOG
Lin. CHOG
Lin. ECHOG
Wavelet
PCASIFT 0.2
Lin. GShaceC
Lin. EShaceC
MIT best (part) 0.1
miss rate
miss rate
MIT baseline
0.1
Ker. RHOG
0.05 Lin. R2HOG
Lin. RHOG
Lin. CHOG
0.05 Lin. ECHOG
Wavelet
0.02
PCASIFT
0.02 Lin. GShapeC
0.01 Lin. EShapeC
6 5 4 3 2 1
0.01 6 5 4 3 2 1
10 10 10 10 10 10 10 10 10 10 10 10
false positives per window (FPPW) false positives per window (FPPW)
Figure 3. The performance of selected detectors on (left) MIT and (right) INRIA data sets. See the text for details.
tector performance. Throughout this section we refer results tive masks. Several smoothing scales were tested includ-
to our default detector which has the following properties, ing =0 (none). Masks tested included various 1-D point
described below: RGB colour space with no gamma cor- derivatives (uncentred [1, 1], centred [1, 0, 1] and cubic-
rection; [1, 0, 1] gradient filter with no smoothing; linear corrected [1, 8, 0, 8, 1]) as well as 33 Sobel masks and
gradient voting into 9 orientation bins in 0 180 ; 1616 22 diagonal ones 1 0 1
0 , 0 1 (the most compact cen-
1 0
pixel blocks of four 88 pixel cells; Gaussian spatial win- tred 2-D derivative masks). Simple 1-D [1, 0, 1] masks at
dow with = 8 pixel; L2-Hys (Lowe-style clipped L2 norm) =0 work best. Using larger masks always seems to de-
block normalization; block spacing stride of 8 pixels (hence crease performance, and smoothing damages it significantly:
4-fold coverage of each cell); 64128 detection window; for Gaussian derivatives, moving from =0 to =2 reduces
linear SVM classifier. the recall rate from 89% to 80% at 104 FPPW. At =0,
Fig. 4 summarizes the effects of the various HOG param- cubic corrected 1-D width 5 filters are about 1% worse than
eters on overall detection performance. These will be exam- [1, 0, 1] at 104 FPPW, while the 22 diagonal masks are
ined in detail below. The main conclusions are that for good 1.5% worse. Using uncentred [1, 1] derivative masks also
performance, one should use fine scale derivatives (essen- decreases performance (by 1.5% at 104 FPPW), presum-
tially no smoothing), many orientation bins, and moderately ably because orientation estimation suffers as a result of the
sized, strongly normalized, overlapping descriptor blocks. x and y filters being based at different centres.
6.1 Gamma/Colour Normalization For colour images, we calculate separate gradients for
We evaluated several input pixel representations includ- each colour channel, and take the one with the largest norm
ing grayscale, RGB and LAB colour spaces optionally with as the pixels gradient vector.
power law (gamma) equalization. These normalizations have 6.3 Spatial / Orientation Binning
only a modest effect on performance, perhaps because the The next step is the fundamental nonlinearity of the de-
subsequent descriptor normalization achieves similar results. scriptor. Each pixel calculates a weighted vote for an edge
We do use colour information when available. RGB and orientation histogram channel based on the orientation of the
LAB colour spaces give comparable results, but restricting gradient element centred on it, and the votes are accumu-
to grayscale reduces performance by 1.5% at 104 FPPW. lated into orientation bins over local spatial regions that we
Square root gamma compression of each colour channel im- call cells. Cells can be either rectangular or radial (log-polar
proves performance at low FPPW (by 1% at 104 FPPW) sectors). The orientation bins are evenly spaced over 0
but log compression is too strong and worsens it by 2% at 180 (unsigned gradient) or 0 360 (signed gradient).
104 FPPW. To reduce aliasing, votes are interpolated bilinearly between
6.2 Gradient Computation the neighbouring bin centres in both orientation and posi-
Detector performance is sensitive to the way in which tion. The vote is a function of the gradient magnitude at the
gradients are computed, but the simplest scheme turns out pixel, either the magnitude itself, its square, its square root,
to be the best. We tested gradients computed using Gaus- or a clipped form of the magnitude representing soft pres-
sian smoothing followed by one of several discrete deriva- ence/absence of an edge at the pixel. In practice, using the
DET effect of gradient scale DET effect of number of orientation bins DET effect of normalization methods
0.5 0.5 0.2
0.2 0.2
0.1
0.1 0.1
miss rate
miss rate
miss rate
bin= 9 (0180)
=0 bin= 6 (0180)
0.05 0.05 0.05
=0.5 bin= 4 (0180) L2Hys
bin= 3 (0180) L2norm
=1
bin=18 (0360) L1Sqrt
0.02 =2 0.02 bin=12 (0360) L1norm
=3 bin= 8 (0360) No norm
=0, ccor bin= 6 (0360) Window norm
0.01 6 5 4 3 2 1
0.01 6 5 4 3 2 1
0.02 5 4 3 2
10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10
false positives per window (FPPW) false positives per window (FPPW) false positives per window (FPPW)
(a) (b) (c)
DET effect of window size DET effect of kernel width,, on kernel SVM
DET effect of overlap (cell size=8, num cell = 2x2, wt=0)
0.5 0.5
0.5
miss rate
0.1
miss rate
Linear
0.02 64x128 0.02 =8e3
0.02 overlap = 3/4, stride = 4
overlap = 1/2, stride = 8 56x120 =3e2
overlap = 0, stride =16 48x112 =7e2
0.01 6 5 4 3 2 1
0.01 6 5 4 3 2 1
0.01 6 5 4 3 2 1
10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10
false positives per window (FPPW) false positives per window (FPPW) false positives per window (FPPW)
magnitude itself gives the best results. Taking the square root
reduces performance slightly, while using binary edge pres-
ence voting decreases it significantly (by 5% at 104 FPPW).
Fine orientation coding turns out to be essential for good 20
15
rather coarse. As fig. 4(b) shows, increasing the number
of orientation bins improves performance significantly up to 10
would help to improve the detection results in more general [10] T. Joachims. Making large-scale svm learning practical. In
situations. B. Schlkopf, C. Burges, and A. Smola, editors, Advances in
Kernel Methods - Support Vector Learning. The MIT Press,
Acknowledgments. This work was supported by the Euro-
Cambridge, MA, USA, 1999.
pean Union research projects ACE M EDIA and PASCAL. We
thanks Cordelia Schmid for many useful comments. SVM- [11] Y. Ke and R. Sukthankar. Pca-sift: A more distinctive rep-
resentation for local image descriptors. CVPR, Washington,
Light [10] provided reliable training of large-scale SVMs.
DC, USA, pages 6675, 2004.
References [12] D. G. Lowe. Distinctive image features from scale-invariant
keypoints. IJCV, 60(2):91110, 2004.
[1] S. Belongie, J. Malik, and J. Puzicha. Matching shapes. The
8th ICCV, Vancouver, Canada, pages 454461, 2001. [13] R. K. McConnell. Method of and apparatus for pattern recog-
[2] V. de Poortere, J. Cant, B. Van den Bosch, J. de nition, January 1986. U.S. Patent No. 4,567,610.
Prins, F. Fransens, and L. Van Gool. Efficient pedes- [14] K. Mikolajczyk and C. Schmid. A performance evaluation of
trian detection: a test case for svm based categorization. local descriptors. PAMI, 2004. Accepted.
Workshop on Cognitive Vision, 2002. Available online: [15] K. Mikolajczyk and C. Schmid. Scale and affine invariant
http://www.vision.ethz.ch/cogvis02/. interest point detectors. IJCV, 60(1):6386, 2004.
[3] P. Felzenszwalb and D. Huttenlocher. Efficient matching of [16] K. Mikolajczyk, C. Schmid, and A. Zisserman. Human detec-
pictorial structures. CVPR, Hilton Head Island, South Car- tion based on a probabilistic assembly of robust part detectors.
olina, USA, pages 6675, 2000. The 8th ECCV, Prague, Czech Republic, volume I, pages 69
[4] W. T. Freeman and M. Roth. Orientation histograms for 81, 2004.
hand gesture recognition. Intl. Workshop on Automatic Face- [17] A. Mohan, C. Papageorgiou, and T. Poggio. Example-based
and Gesture- Recognition, IEEE Computer Society, Zurich, object detection in images by components. PAMI, 23(4):349
Switzerland, pages 296301, June 1995. 361, April 2001.
[5] W. T. Freeman, K. Tanaka, J. Ohta, and K. Kyuma. Com- [18] C. Papageorgiou and T. Poggio. A trainable system for object
puter vision for computer games. 2nd International Confer- detection. IJCV, 38(1):1533, 2000.
ence on Automatic Face and Gesture Recognition, Killington,
VT, USA, pages 100105, October 1996. [19] R. Ronfard, C. Schmid, and B. Triggs. Learning to parse pic-
tures of people. The 7th ECCV, Copenhagen, Denmark, vol-
[6] D. M. Gavrila. The visual analysis of human movement: A ume IV, pages 700714, 2002.
survey. CVIU, 73(1):8298, 1999.
[20] Henry Schneiderman and Takeo Kanade. Object detection
[7] D. M. Gavrila, J. Giebel, and S. Munder. Vision-based pedes- using the statistics of parts. IJCV, 56(3):151177, 2004.
trian detection: the protector+ system. Proc. of the IEEE In-
telligent Vehicles Symposium, Parma, Italy, 2004. [21] Eric L. Schwartz. Spatial mapping in the primate sensory pro-
jection: analytic structure and relevance to perception. Bio-
[8] D. M. Gavrila and V. Philomin. Real-time object detection for logical Cybernetics, 25(4):181194, 1977.
smart vehicles. CVPR, Fort Collins, Colorado, USA, pages
8793, 1999. [22] P. Viola, M. J. Jones, and D. Snow. Detecting pedestrians
using patterns of motion and appearance. The 9th ICCV, Nice,
[9] S. Ioffe and D. A. Forsyth. Probabilistic methods for finding France, volume 1, pages 734741, 2003.
people. IJCV, 43(1):4568, 2001.