2010 8th IEEE International Conference on
1
Control and Automation
Xiamen, China, June 9-11, 2010
Region Covariance based Object Tracking Using Monte Carlo Method
Xiaofeng Ding, Chengrong Huang, Fengchen Huang,
Lizhong Xu, Member, IEEE, and Xiao-fang Li, Member, IEEE
Abstract— Covariance features enabled efficient fusion of
different type of image features have low dimensions and
covariance-based object tracking has been proved robust,
versatile for a modest computational cost. In this paper, a
method combined Monte Carlo method and covariance features
is proposed. Monte Carlo method is used to determine the scope
of the search target at the region level. Covariance features are
used to model the objects appearance at the object level. An
improved object matching and occlusion handling strategies
are given, which are followed by an appearance model update
method. Experiments show our approach is robust and effective
for tracking the object with irregular movement and partial
occlusions.
I. INTRODUCTION
Object tracking is always an important task in computer
vision, it has various application in visual surveillance,
human-computer interaction, vehicle navigation and so on.
Object tracking can be classified into point tracking, kernel
tracking and silhouette tracking [1]. For differen tracking
task different tracking method is used. Point tracking use
point features which are usually detected by Harris, KTL
and SIFT detectors to describe object. It is implemented by
find the points’ relationship in single and consecutive frames.
Kernel tracking is typically performed by computing the
motion of the kernel from one frame to the next. Silhouette
tracking is performed by find the object frame by means of an
object model generated using the previous frames, wherein
the object model is in the form of a color histogram, object
edges or the object contours. Details of these three object
tracking categories are referred to Ref. [1].
Generally, a suitable feature will be selected to describe
the target in kernel tracking and silhouette tracking. So
how to select a discriminate, robust and easy computing
feature is very critical to the performance of object tracking.
Firstly, the raw pixel values of several image statics such
as color, gradient and filter responses are used for image
features. These features are easy to change in the presence
of illumination changes and non-rigid motions. Then, his-
togram, a nature extension of raw pixel values, is used.
However, histogram described features are exponential with
its numbers. Recently, the covariance region descriptor has
This work is supported by the National High Technology Research and
Development Program of China (No. 2007AA11Z227) and the Natural
Science Foundation of Jiangsu Province of China (No. BK2009352).
Xiaofeng Ding, Fengchen Huang, Lizhong Xu and Xiao-fang
Li are with the College of Computer and Information, Hohai
University, Nanjing, 210098, China. dingxfeng@gmail.com,
lzhxu@hhu.edu.cn.
Chengrong Huang is with the School of Computer Engineering, Nanjing
Institute of Technology, Nanjing, 211167, China.
978-1-4244-5196-8/10/$26.00 ©2010 IEEE 1802
FrB7.
been proposed in [2]. Covariance matrix extracted from a
region is used as region descriptor. The covariance matrix
fuses multiple features which might be correlated such as
intensity, color, gradients, filter responses, etc. It is robust,
discriminative and the computing based on integral image
make the computational cost is independent of the size of
the region.
Covariance-based object tracking proposed in [2], [3],
[4] has been proved robust and versatile for a modest
computational cost. Tuzel et al. [2] proposed a global search
algorithm to find the best matching between consecutive
video frames, wherein covariance features are proved faster
and robuster than color histograms. Porikli et al. [3] proposed
a simple and elegant algorithm to track non-rigid objects
using a covariance based object description and an update
mechanism based on means on Riemannian manifolds. Wu et
al. [4] and Palaio, Batista [5] proposed the covariance-based
particle filter for single and multi objects tracking, separately.
However, when particle filters are applied to visual tracking,
the particle degeneracy and how to establish the target’s
motion model are two of the main problems.
In this paper, we propose a Monte Carlo method [6]
combined a covariance based object description to track non-
rigid objets with irregular movement and partial occlusions.
In order to improve the performance of occlusion handling,
we give a new object matching and model update strategy.
The paper is organized as follows: Section II describes
the region covariance descriptor; in section III, we give the
Monte Carlo method; then the object matching and model
update strategy are introduced in section IV; experiment
results are provided in section V, which is followed by
conclusions in section VI.
II. R EGION C OVARIANCE D ESCRIPTOR
The region covariance descriptor proposed in [2] by Tuzel
et al. fuses multiple features into a covariance matrix which
is low-dimension and discriminative.
Let I be a three dimensional color image. Let F be the
W × H × d dimensional feature image extracted from I
F (x, y) = φ(I, x, y) (1)
where the function φ can be any mapping such as intensity,
color, gradients, filter responses, etc. For a given rectangular
region R in image I, let {fk }k=1..n be the d-dimensional
feature points inside R. The region covariance can be ex-
pressed as
n
1 X
CR = (fk − µ)(fk − µ)T , (2)
n−1
k=1

where CR is a d × d positive matrix, µ is the mean of all
the points in R.
Differen composite of different sub-feature extracted from
the image represents differen property. To have a correct
object representation we need to choose what kind of features
to extract from the image I. In this paper, we define φ as

φ = x y R(x, y) G(x, y) B(x, y)
 (3)
I(x, y) Ix (x, y) Iy (x, y) ,
where x and y are the pixel location; R(x, y), G(x, y) and
B(x, y) are the red, green and blue color values; I is the
intensity; Ix (x, y) and Iy (x, y) are the image derivatives
calculated through the filter [−1 0 1] in x, y direction.
A. Covariance Computation and Dissimilarity Metric
During the covariance based object tracking process, a
fast covariance matrix computation method of a given region
is necessary. After getting the regions’ covariance matrixes,
how to measure the dissimilarity of the matrixes is the basis
of the good tracking performance. To over come these two
problems, a integral image based computation method [2]
and a Riemannian Manifold [3] are used.
The integral image based covariance matrix computation
method proposed in [2] will be used in our paper. When this
method is used in covariance computation, the computation
cost is independent of the size of the region. Details of the
computation process are referred to Ref. [2].
Although a covariance metric has been proposed in [3]
which is an invariant metric for the tangent space for
symmetric positive definite matrices. For the completeness
of this paper and the convenience of the next section IV of
the object matching and model update, we provide here a
necessary introduction.
A covariance metric proposed in [3] is given by
1 1

hy, ziX = tr X − 2 yX −1 zX − 2 , (4)
where hy, ziX is the metric defined by a collection of inner
products on the tangent space, y and z are arbitrary points
of the manifold; tr(.) denotes the matrix’s trace; the capital
letters denote the points on manifold and the small letters
corresponds to vectors on the tangent space.
The distance on the manifold, i.e. the dissimilarity between
two regions’ covariance matrixes, is defined in terms of
minimum length curves between points on the manifold. The
curve with minimum length is called the geodesic and the
length of the curve is the intrinsic distance. Let y ∈ TX M ,
where TX M is the tangent space at point X ∈ M . There is
a unique geodesic starting at X with tangent vector y. The
exponential map, exp X: TX M → M , maps the vector y to
a point Y belonging to the previous geodesic. We denote by
logX its inverse. The exponential map is given by
1 1 1
1 object’s scale is also Gaussian distribution, and the mean is
expX (y) = X 2 exp X − 2 yX − 2 X 2 . (5)
The logarithm map is given by
1 1 1
1
logX (Y ) = X 2 log X − 2 Y X − 2 X 2 . (6)
1803
FrB7.1
Fig. 1: The object is presented with a rectangle, and its
corresponding 100 sampling rectangles.
The distance between X and Y is given by d(X, Y ) =
k logX (Y )kX . Combining (6) and (4), we have
d2 (X, Y ) = hlogX (Y ), logX (Y )iX
1 1
(7)
= tr log2 (X − 2 Y X − 2 ) .
III. M ONTE C ARLO M ETHOD
Tracking can be considered as estimation of the state
given all the measurements up to the moment, or equiva-
lently constructing the probability density function of object
location [3]. Generally, a filtering with prediction approach
is used. The most common filter is particle filter. However,
the particle degeneracy and how to establish the target’s
motion model are critical to the tracking performance. Our
approach is based on the Monte Carlo method [6], which is a
special case of particle filter and do not consider the particle
degeneracy and the object’s motion model.
During the Monte Carlo method, if we get the object’s
location in the current frame, then using Monte Carlo method
to sample the object’s possible locations and scales in the
next frame. After getting these samples, the best matching,
i.e. the object’s location, can be got by (11).
Let the distribution of the object’s motion between con-
tinuous frames is two-dimensional Gaussian. Using the two-
dimensional Gaussian distribution to produce n sampling
rectangles. The center of these rectangles subjects to two-
dimensional Gaussian distribution, the mean is the center of
the rectangle in the former frame, the variance reflects the
speed of the object’s movement. The sampling rectangles’
centers are produced by the Box-Muller [7] method. Let the
random variable x and y is the uniform distribution in [0, 1],
then u and v expressed as
( √
u = ( −2 ln x cos 2πy) × σ + µ
√ , (8)
v = ( −2 ln x sin 2πy) × σ + µ
which are the two-dimensional Gaussian distribution. Let the
the scale of the object in the former frame, the variance
reflects the rate of its change. Then using Monte Carlo
method, we can get the object’s possible location and scale
in the next frame, see Fig. 1.

(a) Object (b) C 1 (c) C 2 (d) C 3 (e) C 4 (f) C 5
Fig. 2: Object representation. Construction of the five covari-
ance matrixes from overlapping regions of an object feature
image.
IV. O BJECT M ATCHING AND M ODEL U PDATE
To obtain the most similar region to the target, we can
use (7) to measure the distance between the covariance
matrixes corresponding to the target object window and the
candidate regions. Theoretically a single covariance matrix
extracted from a region is usually enough to matching the
region in different views and poses. So we can find the best
object matching by finding the minimum distance with (7).
However, during object tracking, occlusions often happen.
Tuzel et al. [2] and Palaio, Batista [5] represented an object
with five covariance matrixes of the image features computed
inside the object region, as show in Fig. 2. Based on this
representation, two different dissimilarities of the two objects
were given in Ref. [2], [5]. The dissimilarity in Ref. [5] is
given by
X5
ρ(O1 , O2 ) = d2 (COi
1
i
, CO 2
), (9)
i=1
i i
where CO and CO (i = 1, ..., 5) are the five covariance
1 2
matrixes of object 1 and object 2, respectively, which is better
than the dissimilarity in Ref. [2]. Equation (9) integrates all
the distances of the five regions of a object. It is robust when
the object is with partial occlusions.
Combining the Monte Carlo method, an improved object
matching strategy based on (9) is given. In the following
statement, k = 1, ..., n and i = 1, ..., 5 expect a special
note. Among all the sampling regions, let the k-th sampling
region’s location and scale be xk . dk is the distance of the k-
th sampling region (or candidate region) and the target object
region (or reference object). Let d1k be the weight of the k-
th sampling region. Then, the predictive object location and
scale is Pn xk
k=1 dk
x̂ = Pn 1 .
k=1 dk
Let the predictive object location and scale be the (n + 1)-th
sampling region.
Among all the sampling regions, for the k-th sampling
region, we can get its five covariance matrixes representation
Cki . For a given i, by min1≤k≤n d2 (Cki , CTi ) we can get
five sampling regions k1 , k2 , k3 , k4 , k5 based on the spatial
relationships between the original object and the subregions
FrB7.1
(a) Origin (b) Occlusion-1 (c) Occlusion-2
(d) Occlusion-3 (e) Occlusion-4 (f) Occlusion-5
Fig. 3: The original object and different occlusions. a: the
original object. b-f: the object under different types of partial
occlusions.
in Fig. 2. By (9), the best and robust object matching is
5
X
arg min d2j (Cji , CTi ), (11)
j={k1 ,k2 ,k3 ,k4 ,k5 ,n+1}
i=1
where Cji and CTi are the j-th sampling’s and the target
object’s five covariance matrixes, respectively.
Solutions for the covariance matrixes update were pro-
posed in [3] and [5], wherein the method in [3] considered
the previous T covariance matrixes by a gradient descent
approach and the method in [5] just considered the current
and the last covariance matrixes by a middle mean approach.
In this paper, we propose a novel solution for the covariance
matrix update based on a variable mean of the the current and
the last covariance matrixes. Let X and Y are the last and
current covariance matrixes. d(X, Y ) is the distance between
X and Y . Then our model update is
1 1 1 1
C = X 2 exp(X − 2 (αd(X, Y ))X − 2 )X 2 , (12)
where α is the rate of model update, and when d(X, Y ) is
more than a given threshold the model will de updated.
V. E XPERIMENTS AND R ESULTS
To demonstrate the performance of the proposed approach,
experiments were carried on several video sequences. The
experiments show that our approach is effective, feasible and
robust, when the object’s motion is under one or several
of these situations: (1) irregular movement; (2) target scale
(10) variations; (3) partial occlusions; (4) illumination changes.
A. Experiment 1
An experiment was implemented to prove that our object
matching strategy is robust and exactly. In Table I, the
values in the final column do not differ much although the
occlusions are different; and from column 2 to 6, in each
column the minim value is corresponding to the maxim
visual area of the target object.
1804
 FrB7.1

Occlusion d2 (CO1 , C1)

j d2 (CO2 , C2)
j d2 (CO3 , C3)
j d2 (CO4 , C4)
j d2 (CO5 , C5)
j ρ(O, Oj )
1 2.0875 1.5445 2.5791 0.6510 3.3970 10.2591
2 1.8770 1.6951 2.7369 0.2546 2.8200 9.3836
3 1.9158 0.4171 2.9884 0.3370 2.8523 8.5106
4 2.3987 0.3247 3.8712 2.6385 2.6440 11.8771
5 2.1392 0.6884 3.3502 1.9920 2.8185 10.9883

TABLE I: The dissimilarities of the object without and with different partial occlusions. These dissimilarities are calculated
i
under the best matching selected manually. CO is the i-th covariance matrix (Fig. 2) of the object in Fig. 3(a). Cji is the i-th
covariance matrix of the object with occlusion-j (Fig. 3) with the i-th covariance representation.

(a) frame 1 (b) frame 3 (c) frame 4 (d) frame 10

(a) frame 10 (b) frame 181 (c) frame 332

(e) frame 1 (f) frame 3 (g) frame 4 (h) frame 10

Fig. 4: A head is tracked against a irregular movement.

(d) frame 392 (e) frame 416

B. Experiment 2
Figure 4 shows the tracking results on a continuous
sequence of a head’s irregular movement and illustrate how
to use the covariance of the object’s motion σ1 and the
covariance of the rate of scale variation σ2 . In Fig. 4, the top
row outlines the tracking result with σ1 = 5, the rectangles
in frame 4 and 10 are not well overlapping the head; the
tracking results in the second row are well done while σ1
increases to 8, where σ1 is in pixel units.
C. Experiment 3
In this experiment, a pedestrian with gray overcoat is
tracked. The pedestrian’s scale is increasing as his walking
from far to near and the pedestrian also stops for a while or
turns around during the walking. In Fig. 5, from frame 181
to 332, the pedestrian turns around and changes his walking
direction; in frame 392, the pedestrian stops for a while to
have a look; in frame 416, another pedestrian passes in front
of him which causes the partial occlusion. Results show in
Fig. 5 illustrate that our approach can successfully tracking
the pedestrian.
D. Other Experiments
Experiments for outdoors tracking of pedestrians or rid-
ers are also implemented. Although the outdoors video
sequences are easy to be degraded by some noises or
camera vibration, the tracking results show our proposed
approach effective. Due to the space limitation, we omit these
experiments’ results.
VI. C ONCLUSIONS
A Monte Carlo method is introduced to the covariance
based object tracking. The proposed approach can track the
object with partial occlusions and irregular movements for
(f) frame 497 (g) frame 647 (h) frame 684
Fig. 5: Tracking result of the pedestrian walking sequence.
the above experiments. The minimum cover ideal is used
to reduce the computing area of the integral image when
the sampling regions’ covariance matrixes are calculated.
Compared to particle filter, the particle degeneracy and how
to model the object’s motion are not needed to consider with
our approach’s performance is similarly with it for the above
experiments. The model update strategy can give a good
prediction while the object is under occlusions.
R EFERENCES
[1] Y. Alper, J. Omar and S. Mubarak, “Object tracking: A survey,” ACM
Computing Surveys, 2006, 38(4).
[2] O. Tuzel, F. Porikli and P. Meer, “Region Covariance: A Fast De-
scriptor for Detection and Classification,” In Computer Vision - ECCV
2006, 2006, pp.589-600.
[3] F. Porikli, O. Tuzel and P. Meer, “Covariance Tracking using Model
Update Based on Means on Riemannian Manifolds,” Computer Vision
and Pattern Recognition, 2006 IEEE Computer Society Conference on,
2006.
[4] Y. Wu, B. Wu, et al. “Probabilistic tracking on Riemannian mani-
folds,” In Pattern Recognition, 2008. ICPR 2008. 19th International
Conference on, 2008, pp. 1-4.
[5] H. Palaio, J. Batista. A region covariance embedded in a particle filter
for multi-object tracking. In workshop on Visual Surveillance, ECCV
2008, 2008.
[6] Haiqin Zhang, Houqiang Li, Object Tracking with Monte Carlo
Methods. Journal of Image and Graphics, vol. 13, No. 5, pp. 937-
943, 2008.
[7] T. Shinzato. Box Muller Method. http://www.sp.dis.
titech.ac.jp/˜shinzato/boxmuller.pdf

1805