Car Re-identification from Large Scale Images

Using Semantic Attributes

Qi Zheng, Chao Liang, Wenhua Fang, Da Xiang, Xin Zhao, Chengping Ren, Jun Chen
National Engineering Research Center for Multimedia Software
School of Computer Science, Wuhan University, Wuhan, China
Email: zhengq@whu.edu.cn, cliang@whu.edu.cn

Abstract—Car re-identification, searching a specific car object
from a large-scale car image database, is investigated in this
paper. Previous work mainly focuses on fixed pose and overlooks
the special appearance. However, avoiding matching other poses
would lead to coarse results of the car retrieval. And some special
attributes like individual paintings which are greatly helpful
for car retrieval have not drawn enough attention. This paper
addresses these problems through multi-poses matching and re-
ranking based on special attributes. Our core idea lies in query
expansion method that can capture weighted attributes to build
the retrieval model, which allows us to estimate invisible attributes
by the visible ones to construct complete attributes vectors to
car retrieval in any poses. Furthermore, we divide all attributes
into two groups, special attributes and common attributes.
Here special attributes represent the abnormal appearance like
individual paintings or car damage while common attributes
denote the intrinsic appearance of car. Using special attributes
to re-rank results turns out to be beneficial to improve the
retrieval performance. In the end, the experiments demonstrate
the effectiveness of our approach on the car datasets.
the utility of example-based retrieval with color correlograms
avoid the limitations of strict color classification. Ming-Kuang
Tsai et al. [4] used active shape model (ASM) to fit 3D vehicle
models to a 2D image, then obtain those parts to rectify
vehicles from disparate views into the same reference view.
These methods are mainly searching in low dimensional, so
they avoid finding other features in the objects, leading to the
coarse results of the car retrieval. Another influential approach
considers sensors as add-ons [5] [6]. Rogerio Feris et al. [5]
searched for vehicles in surveillance videos based on attributes
(such as color, direction of travel, speed, length, height, etc.).
They focus on surveillance surroundings that some of the
characteristics are obtained by the kinetic sensors, which
limits their applications. Besides, the last two methods [4] [5]
mentioned above are restricted to the limited condition of
common-pose car retrieval.
Query Image Gallery Images
……

I. I NTRODUCTION
Attributes

The car re-identification problem is the task of instance in front-left

1 0 0 1 0 1…
Attributes in Attributes in Attributes in
search on specific car, which can obtain several similar car front-left
1 0 0 1 0 1…
right
0 1 0 0 0 1…
back-right
1 0 0 0 0 1…

images in a gallery. It has attracted a lot of attention in 1 0 1 1 0 1…

1 1 0 1 0 1…
1 0 0 1 0 0…

1 0 1 1 0 1…
1 0 0 1 0 0…

1 1 0 0 0 1…
……
multimedia retrieval system with the greatly increasing number Attributes in Attributes Attributes in
……
0 0 0 1 0 1… 0 0 0 1 0 1… 0 0 0 1 0 1…
front-right in right back-right …… …… ……
of cars. Unfortunately, the traditional technology using license 1 0 0 1 0 1… 0 0 0 1 0 1… 0 1 0 1 0 1…

plate recognition seems impossible to be utilized in practical

situations, due to massive cases of invisible or vague license ×
plates. Under some special conditions, e.g., plates are replaced
by fake ones or new cars always have no plates, car plate ×
recognition methods are totally useless. Thus plate recognition ×
is only useful in constrained conditions like HD bayonet
monitoring system. However, many car makers produce dif- × Distance
: Measure
Ranking

ferent types of cars, and provide variety of unique car suit

or painting for individuals, which makes car retrieval based
on appearance greatly important. Nonetheless, compared with
the traditional car categorization [1], the car re-identification
is still a more challenging work due to the requirement of
finding out the specific target from a group of cars of the same
class. Considering the practical conditions, it is even worse
due to appearances of cars change with the pose variation and
appearance change.
Most of previous retrieval approaches typically exploit low-
level features [2] [3] [4], such as color, texture, SIFT feature
, because they can be relatively easily and reliably measured.
Shin-Yu Chen et al. [2] used the multiple-instance learning
(MIL) and eigen color method to learn specific visual proper-
ties of vehicles from query images. Lisa M. Brown [3] studied
……
Fig. 1. The framework of our retrieval system. This is the core part of our
system. When it comes a query image with a car of front-left pose, we firstly
flip it to front-right pose due to the same attributes in this two poses, more
details are in section II. Then the attributes in other 4 poses would be estimated
by the original pose (here is front-right pose). The red cells is estimated value
which is combined with other attributes in different pose to construct a new
attributes vector. Next, we measure all of the attributes vector pairs distance
between query image and gallery image in each pose. To the end, the retrieval
result, shown in the bottom of figure, is obtained.
Noted that there are strong correlations between different
poses, e.g. a car in front-right direction with ”head-bumper”
indicate that there must be ”back-bumper” in that car though

978-1-4673-7478-1/15/$31.00
c 2015 IEEE
 TABLE I. B INARY ATTRIBUTES SPACE WITH 27 ATTRIBUTES .
we could not see it. Besides, some special appearance of the
car, like individual painting, can locate the same car in gallery head-bumper back-bumper roof-antenna container
roof-rack front-plate back-plate low-underpan
images more accurately. Thus we propose a novel method middle-underpan high-underpan sivler-wheel spare-wheel
in Fig.1. It can tackle with the car retrieval problem under tail-fin vertical-taillight four-headlight horizontal-grille
multi-poses circumstances. Our work is inspired by query coldCair-intake minivan sportyCar jeep
ordinary-car ragtop-open wheel-visible new
expansion based on term co-occurrence and term similarity simple-texture damage roof-Cargo
which has been widely investigated in the text retrieval works
with varying degrees of success [7]. So we bring the query
expansion idea to our system to deal with the multi-poses to the car statue, and the left attributes are the type of the car.
problem and take advantage of the strong correlations to Fig.3 shows an example of some attributes.
evaluate the expansive attributes in another pose. Here using
invisible attributes as expansive attributes to deal with pose
change problem is an important part in our work. Then we
Jeep
adopt distance measurements trained by 5 car poses (these
poses are seen in Fig.5) to find the optimal result. Based on head-bumper spare-wheel
the visible attributes combined with expansive attributes for
each car pose, we focus on the sematic attributes distance
between the query image and each gallery image. In addition,
we divide all attributes into two groups, special attributes high-underpan
back-bumper
including ”damage”, ”complex-texture” and common attributes
including others. If any special attributes detected in the query
image, the SIFT features would be utilized for re-ranking the
sivler-wheel
retrieval result. In our experiments, It is demonstrated that our ragtop-open

methods can handle those conditions effectively.

Above all, there are two primary contributions in this paper:
(1) we propose a novel method for multi-poses car retrieval low-underpan horizontal-grille

with the estimation in visible attributes, as outlined in Fig.2.

wheel-visible
(2) we introduce the special attributes different from common
attributes to improve our re-identification performance.
Front-left attributes Back-right attributes
Fig. 3. A typical example of attributes.Note that some attributes are not
head-bumper back-bumper
visible in one pose.
jeep spare-wheel

simple-texture jeep

N-roof-antenna simple-texture
Car Pose Based on the statistics of car datasets, we
N-roof-rack N-roof-antenna
divide the car pose into 8 directions illustrate in Fig.4.
high-underpan N-roof-rack

… …
… …
Color Histogram The main color of a car is not the
estimate

back-bumper
Invisible attributes :
spare-wheel
Fig. 2. Attributes estimation. We estimate the invisible attributes to measure
the images of other poses. Some invisible attributes like spare-wheel in front-
left can be easily estimated by other visible attributes. Thus a query expansion
is used to measure attributes vector in other pose.
II. C AR R EPRESENTATION
Introduction of some relevant concepts in car retrieval is
presented in this section. In addition, we show the training of
detector for each attribute in Section 2.2.
A. Appearances
The appearances of cars are usually changing with different
poses which bring the variation of visibility about attributes.
We introduce the invisible attributes estimation based on
car pose to bridge the gaps among the different car poses.
Meanwhile, the color histogram and sift feature are significant
to our preprocessing and result re-ranking respectively.
Binary Semantic Attributes According to the intrinsic
property of cars, we define the following space of Na = 27
binary attributes for our study (summarized in Tab. I ). Seven-
teen of those attributes are car components, and six are related
binary variable, but it hardly changes by the car pose. Thus as
a preprocessing step in our framework, the color is used for
clustering the images of similar color to obtain a subset of the
original datasets as a coarse retrieval result. Experiment shows
that it improves our performance substantially. Standard color
histogram is used here because they are robust to occlusions
and lighting and view changes. And the pioneering work of
Swain [8] has shown the effectiveness of color histograms to
distinguish the large number of objects.
Sift Feature The SIFT [9] algorithm has been suc-
cessfully used for describing images local features. The SIFT
features show the invariant to image scaling, translation, and
rotation, even partially invariant to affine distortion, illumi-
nation changes, noise addition and partial occlusion. Because
of its remarkable performance on feature matching, the SIFT
has been widely applied to the retrieval work [10] [11]. In this
paper, any special attributes detected will lead the sift matching
here to ranking the retrieval result.
B. Attribute Detection
As described in section 2.1, we classify the pose of car
into 8 directions. Additionally, we notice that the symmetry
direction (except front-back pair ) owe the same attributes.
Thus we decrease the number of directions to 5 by horizontal

Fig. 4. Typical poses. Eight canonical poses of cars noted by red arrows.
flipping processing without losing the accuracy, as shown in
Fig.5.
Fig. 5. Symmetry characteristic of the 8 poses. The 8 pose could be flipped
into 5 poses without decreasing the accuracy but bring down the complexity
of the computation.
To detect semantic attributes, we first cluster the car images
by 5 directions. In each cluster we extract mult-dimensional
low-level color and texture feature vector denoted attributes
from each car image with the method in [12]. It consists
of 507-dimensional feature vectors extracted from m*n equal
sized blocks from the image. Both of the number m and n are
related to the car direction and the attribute position tightly.
They are obtained experimentally, and set to (3,2) or (2,2) here.
We derived 8 color channels (RGB, HSV and YCbCr) and
20 texture filters (Gabor, Schmid, LBP) from the luminance
channel. We use the same parameter choices for γ, λ , θ and
σ 2 as [13] for Gabor filter extraction, and for τ and σ for
Schmid extraction, similar to [13] . Finally, we use a bin size
of 16 to describe each channel except LBP luminance channel
which is set to 59 bins.
However, some attributes will be occluded from different
poses. For example, if a car is facing to the camera, the
attribute like back-bumper can not be seen in the image. But
the back-bumper often exists with the emergence of front-
bumper or the car type of Jeep( illustrated in Fig.2). So it
inspires us that occluded attributes could be estimated by other
visible attributes. Here we use SVM to train the weight vector
wv from each viewpoint:
1
min θ(wv ) = max min( ||wv ||2
wv ,b αi ≥0 wv ,b 2
X n (1)
− αi (yvi (wvT avi + b) − 1))
i=1
Where yvi denotes the self-occluded attribute vector from the
v direction, avi is the the non-occluded attributes vector at the
same direction, and αi , b, n represents the Lagrange multiplier,
intercept term, n samples, respectively . Then to the test data,
we estimate the self-occluded attributes Ovm by:
Ovm = wv Av + b (2)
Note that the output value of the classifiers for each attribute
is from minus infinity to plus infinity. For better distance
measure, we transform the original value to (0,1) by sigmoid
function [14] as:
0 1
Ovm = −Ovm (3)
e +1
III. S IMILARITY M EASUREMENT
This section describes the similarity measurement ap-
proaches to match the car instance from different images. Each
measurement is implemented in one direction and we exploit
occluded attributes detector to bridge those different directions.
In the case of color histograms the dissimilarity between
a query image Q and a gallery images G in the datasets is
measured using the weighted Mahalanobis distance follow the
work of [15]
Dh (Q, G) = [h(Q) − h(G)]t A[h(Q) − h(G)] (4)
where h(Q) and h(G) denote the color histogram of query
image Q and gallery images G respectively. A is a weight
matrix, whose elements aij corresponding to the similarity of
colors. In order to make the final result more reliable, the
distance Dh (Q, G) is set a higher value than a threshold value.
If satisfied results are calculated, we leave this decision for
attributes measurement.
In order to apply our attributes to re-identification, we in-
vestigate how attributes can be fused to enhance performance.
Because each attribute is not held the same important position
due to variability in how reliably they are calculated, how
prevalent they are in the imbalanced data, and how informative
they are about identity, we need to decide which attributes are
more important from the full set and how to obtain the weights.
Besides, the attributes weights will be changed with the variant
direction. For better measuring the attribute vector similarity,
metric learning is conducted. Given a pair of samples xi and
xj for each car pose, the Mahalanobis distance between them
is:
d2M (xi , xj ) = (xi − xj )T M (xi − xj ) (5)
where M >= 0 is a positive semi-definite matrix. We adopt the
Mahalanobis distance matrix M definition by Martin Kostinger
et.al. [13] as
X−1 X−1
M= − (6)
yij =1 yij =0
Where
X X
= (xi − xj )(xi − xj )T (7)
yij =1
yij =1
X X
T
= (xi − xj )(xi − xj ) (8)
yij =0
yij =0
here yij = 1 and yij = 0 denote similar pairs and dissimilar
pairs respectively. More details can be found in [16]. The
learned matrix M is used for computing the distance between
the attribute vectors.
The SIFT features matching stage is only working on the
condition of any special attributes are found in the previous
stage. The SIFT algorithm is utilized to extracted features from
each image in gallery as well as query image. The best can-
didate match for each feature point is found by identifying its
nearest neighbor in the database of feature points. The feature
point with minimum Euclidean distance for the invariant SIFT
descriptor vector is used for defining the nearest neighbor. And
a match is accepted only if its distance is less than a ratio times
the distance of the second closest one, and the ratio is usually
obtained experimentally.
The number of matching points is considered as a criterion
for re-ranking the car retrieval result. Supposing Mi is the
match amount between query image and the number i retrieval
image. We reorder the retrieval result by its value of the
corresponding Mi due to the high accuracy of the SIFT match.
The first image in the final results is considerable meaningful
for the related work.
IV. E XPERIMENTS
In the following, we conduct several experiments on a
challenging dataset to investigate the performance of our
attributes-based re-identification approach.
We collect 4556 images from cars datasets supported by
Jonathan Krause [17] and NetCarShow.com for training. The
Krause Cars dataset contains 16,185 images of 196 classes of
cars. Classes are typically at the level of Make, Model, Year.
The data is split into 8,144 training images and 8,041 testing
images. We chose 2856 images from that dataset for training
because some cars are not the same one though they are belong
to the same class. The NetCarShow website contains the most
of cars in the world, and it supports car images with many
poses. We capture 1700 images from NetCarShow website.
All of our training imgaes attributes are labeled manually due
to those datasets do not contain complete attributes.
Our train data is classified into two sets, Set-I for invisible
attributes estimation and Set-II for attributes detectors. In Set-I
we chose 500 images for training. Meanwhile, the left 4028
images in Set-II are organized into 2014 similar pairs and 2014
dissimilar pairs. Each image contains the 27 attributes. The
ground truth of attributes in our dataset obtained manually.
Note that we only train the visible attributes (less than 27 at-
tributes) weight for each direction cluster. For a input image to
query, we estimate the invisible attributes by Eq.1 to complete
the other direction attributes. It supports us to retrieval the car
image from other 4 directions. The Fig.6 indicates that not all
attributes could perform well.
1
0.9
0.8
0.7
attribute accuracy
0.6
0.5
0.4
0.3
0.2
0.1
0
head−bumper back−bumper front−plate back−plate simple−texture ragtop−open low−underpan sivler−wheel spare−wheel
Fig. 6. Attribute accuracy. It demonstrates the fluctuation of the detection
accuracy of attributes. The ”sivler-wheel” shows the lowest value here may be
caused by the rapid rotation of wheel. Meanwhile, the ”ragtop-open” attribute
take the best performance with 0.8025.
We compared our method (AOR+OAE+SAP) with tradi-
tional attributes object retrieval method (AOR) without oc-
cluded attributes estimation (OAE) and special attributes pro-
cessing (SAP). For evaluation, we use the average cumulative
match characteristic (CMC) curves [18]
PN
n=1 f (pi , r)
CM C@r = × 100% (9)
N
to show the ranked matching rates. We randomly choose
200 pairs from the test set and split the cars images into
the probe set and gallery set. We repeated 10 times and
compute the average matching rate. Fig.7 shows CMC curves
for our approach and AOR.The AOR approach take the worst
performance since it miss the images of irrelevant car poses
and serve as baseline. The best performance is reached by the
AOR+OAE+SAP approach, especially in rank5 to rank20. The
AOR+OAE approach perform slightly worse but much better
than traditional approach.
 CMC
0.8
AOR
0.7 AOR+OAE
AOR+OAE+SAP
0.6
0.5
Matches
0.4
0.3
0.2
0.1
0 retrieval with local context analysis.” Acm Transactions on Information
10 20 30 40
Rank
Fig. 7. Cumulative Match Characteristic curve for AOR, AOR+OAE and
AOR+OAE+SAP methods.
V. C ONCLUSIONS
We have proposed a feasible method of car re-identification
based on semantic attributes. We have also proposed a new
measurement approach with the invisible attributes estimation
and introduced special attributes to re-rank our retrieval result.
In the future work, we will focus on the two points: (1)
refining the features representation to improve the performance
of attributes detection, (2) deep analysis of the correlation
among the different car poses.
ACKNOWLEDGMENT
The research was supported by National Nature Science
Foundation of China (No. 61231015). National High Technol-
ogy Research and Development Program of China (863 Pro-
gram, No. 2015AA016306). National Natural Science Foun-
dation of China (6 1172173, 61303114). Technology Research
Program of Ministry of Public Security (No. 2014JSYJA016).
The EU FP7 QUICK project under Grant Agreement (No.
PIRSES-GA-2013-612652). National Nature Science Founda-
tion of China (No. 61170023). Major Science and Technol-
ogy Innovation Plan of Hubei Province (No. 2013AAA020).
Internet of Things Development Funding Project of Ministry
of industry in 2013 (No. 25). China Postdoctoral Science
Foundation funded project (2013M530350, 2014M562058).
Specialized Research Fund for the Doctoral Program of High-
er Education (No. 20130141120024). Nature Science Foun-
dation of Hubei Province (2014CFB712). The Fundamental
Research Funds for the Central Universities (2042014kf0025).
The Fundamental Research Funds for the Central Univer-
sities (2042014kf0250, 2014211020203). Scientific Research
Foundation for the Returned Overseas Chinese Scholars, State
Education Ministry ([2014]1685).
R EFERENCES
[1] Z. Sun and E. Technol, “On-road vehicle detection: a review,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, pp. 694–
711, 2006.
[2] S. Chen, J. Hsieh, and J. Wu, “Vehicle retrieval using eigen color
and multiple instance learning,” IEEE International Conference on
Intelligent Information Hiding and Multimedia Signal Processing, pp.
657–660, 2009.
[3] L. Brown, “Example-based color vehicle retrieval for surveillance,”
IEEE International Conference on Advanced Video and Signal Based
Surveillance, pp. 91–96, 2010.
[4] M. Tsai, Y. Lin, and W. Hsu, “Content-based vehicle retrieval using 3d
model and part information,” IEEE Transactions on Acoustics, Speech
and Signal Processing, pp. 1025–1028, 2012.
[5] R. Feris, B. Siddiquie, and Y. Zhai, “Attribute-based vehicle search
in crowded surveillance videos,” ACM International Conference on
Multimedia Retrieval, 2011.
[6] K. Tanaka, Y. Kishino, T. Terada, and S. Nishio, “A destination pre-
diction method using driving contexts and trajectory for car navigation
systems,” pp. 190–195, 2009.
[7] J. Xu and W. B. Croft, “Improving the effectiveness of information
50 Systems, vol. 18, no. 1, pp. 79–112, 2000.
[8] M. Swain and D. Ballard, “Color indexing,” International Journal of
Computer Vision, vol. 7, no. 1, pp. 11–32, 1991.
[9] D. Lowe, “Distinctive image features from scale-invariant keypoints,”
International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110,
2004.
[10] X. Wu, Y. Tang, and Z. Zhang, “Video object matching based on sift
algorithm,” Proc. Int. Conf. Neural Netw. Signal Process, pp. 412–415,
2008.
[11] A. Anjulan and N. Canagarajah, “Object based video retrieval with local
region tracking,” Signal Process.:Image Commun., vol. 22, pp. 607–621,
2007.
[12] B. Prosser, W. Zheng, and S. Gong, “Person re-identification by support
vector ranking,” British Machine Vision Conference, vol. 42, no. 7, pp.
25–32, 2010.
[13] D. Gray and H. Tao, “Viewpoint invariant pedestrian recognition with
an ensemble of localized features,” European Conference on Computer
Vision, vol. 5302, pp. 262–275, 2008.
[14] T. Mitchell, “Machine learning,” WCB McGraw-Hill, 1997.
[15] J. Hafner, H. Sawhney, and W. Equitz, “Efficient color histogram
indexing for quadratic form distance functions,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 17, no. 7, pp. 729–736,
1995.
[16] M. Kostinger, M. Hirzer, and P. Wohlhart, “Large scale metric learning
from equivalence constraints,” IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, pp. 2288–2295, 2012.
[17] J. Krause, M. Stark, and D. Jia, “3d object representations for fine-
grained categorization,” IEEE International Conference on Computer
Vision Workshops, pp. 554–561, 2013.
[18] H. Moon and P. Phillips, “Computational and performance aspects of
pca-based face-recognition algorithms,” Perception, vol. 30, no. 3, pp.
303–321, 2001.