IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO.
11, NOVEMBER 2021 7279
Multi-Stream Siamese and Faster Region-Based
Neural Network for Real-Time Object Tracking
Yi Liu, Liming Zhang , Member, IEEE, Zhihui Chen, Yan Yan , Member, IEEE,
and Hanzi Wang , Senior Member, IEEE
Abstract— Object tracking is a challenging task in computer
vision based intelligent transportation systems. Recently, Siamese
based object tracking methods have attracted significant attention
due to their highly efficient performance. These tracking methods
usually train a Siamese network to match the initial target patch
of the first frame with candidates in a new frame. In these
methods, the offline training of the deep neural network and
the online instance searching are effectively combined. However,
these methods usually do not include template update or object
re-identification, which easily results in the drift problem. In
this paper, we propose a novel real-time object tracking method
to overcome the above problems by effectively combining a
multi-stream Siamese network and a region-based convolutional
neural network. Specifically, a novel multi-stream Siamese net-
work is proposed to search the target and update the instance
template in a new frame. In addition, a faster region-based
convolutional neural network detector is used to perform object
re-identification in order to improve the tracking performance
by making full use of the object category information. These two
networks are tightly coupled to ensure that the proposed tracking
method has high efficiency and strong discriminative capability.
Experimental results on several object tracking benchmarks
show that our tracking method can effectively track vehicles
and pedestrians in video sequences by exploiting the object
category information. The proposed tracking method achieves
real-time operations and outperforms several other state-of-the-
art methods.
Index Terms— Object tracking, instance search, multi-stream
Siamese network, Faster R-CNN.
I. I NTRODUCTION
O BJECT tracking is one of the fundamental prob-
lems in computer vision. It has a variety of applica-
tions, such as video surveillance, human-computer interaction,
behavior analysis, and intelligent transportation systems. For
Manuscript received November 26, 2019; revised April 23, 2020; accepted
June 3, 2020. Date of publication July 28, 2020; date of current version
November 1, 2021. This work was supported in part by the National Natural
Science Foundation of China under Grant U1605252 and Grant 61872307,
in part by the National Key Research and Development Program of China
under Grant 2017YFB1302400, in part by the Science and Technology
Development Fund of Macao SAR under Grant FDCT 079/2016/A2, and in
part by the University of Macau Research Grant under Grant MYRG2018-
00111-FST. The Associate Editor for this article was W. Lin. (Corresponding
author: Hanzi Wang.)
Yi Liu, Zhihui Chen, Yan Yan, and Hanzi Wang are with the Fujian Key
Laboratory of Sensing and Computing for Smart City, School of Informatics,
Xiamen University, Xiamen 361005, China (e-mail: liuyi_xmu@163.com;
zhihui-chen@foxmail.com; yanyan@xmu.edu.cn; wang.hanzi@gmail.com).
Liming Zhang is with the Department of Computer and Information Science,
Faculty of Science and Technology, University of Macau, Taipa, Macau
(e-mail: lmzhang@um.edu.mo).
Digital Object Identifier 10.1109/TITS.2020.3006927
1558-0016 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
autonomous driving applications, a car needs to track the
objects in the scene and estimate where and how they will
move. Although there are numerous tracking methods pro-
posed in recent years, this task is still challenging since the
scenarios can be highly complex and the appearance of the
object may significantly change due to unpredictable factors
(e.g., fast motion, occlusion, and in/out-of-plane rotation)
during tracking.
To deal with target appearance changes, traditional tracking
methods, including MIL [1], PST [2], TLD [3], CREST [4],
DSST [5], Struck [6], and QKCF [7], usually learn the
hand-crafted feature representations of a tracked object during
the online tracking process and use the learned features to track
the target. However, these tracking methods are online trained
without taking advantage of a large number of video sequences
available offline. Note that the offline video sequences involve
various scenarios and they can be used for learning robust
feature representations of objects to handle complex challenges
in the object tracking task.
Recently, convolutional neural networks (CNNs) have been
adopted to learn robust feature representations of objects from
large-scale datasets [8] and they have achieved significant
progress on various computer vision tasks, including face
recognition [9], [10], image annotation [11], [12], object clas-
sification [13], [14] and object detection [15], [16]. Motivated
by the great success of CNNs, several recent works [17], [18]
formulate the object tracking task as a binary classification
problem, which aims to separate the tracked object from the
background using CNNs. These methods use pre-trained CNN
models and exploit the robust feature representation capability
of CNNs for the classification task. However, these pre-trained
CNN models may not be optimal for the object tracking task
because object tracking is not only a binary classification
problem but also an instance-level problem.
Different from the previous classification based tracking
methods, some methods (e.g., [19], [20]) treat object track-
ing as an instance search problem. These methods learn a
fully-convolutional deep Siamese network for object tracking.
However, the existing Siamese instance search based tracking
methods have the following limitations. Firstly, the single
Siamese instance search network does not consider online
template update. As a result, these tracking methods may
easily drift when the object appearance significantly changes,
especially for visual surveillance under traffic scenes, since
only the object state in the first frame is used as the template
 7280 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021
Fig. 1. Tracking results obtained by SiamFC and our method on three
challenging videos.
in the Siamese network [20]. Secondly, the search region in
these tracking methods is too small, which may result in low
accuracy. However, the computational complexity will increase
if the search region is enlarged. It is difficult to achieve a trade-
off between running speed and tracking accuracy. Therefore,
there are two main issues to be addressed for these Siamese
instance search based tracking methods, namely, how to update
the template in an effective way, and how to perform object
re-identification in a larger search region.
To this end, we propose a novel real-time object tracking
method that effectively combines a multi-stream Siamese
network and a region-based convolutional neural network,
namely MSRT. The proposed method can implement both
the object template update and object re-identification tasks.
Firstly, we propose a multi-stream Siamese network to ensure
highly efficient search and update instance templates during
the tracking process. Secondly, we train a metric learning
based model (consisting of several distance functions) to
determine whether the tracked target is lost or not. When the
tracked target is lost, a detector built on a Faster Region-based
Convolutional Neural Network (Faster R-CNN) is triggered to
perform object re-identification by using the object category
information, which leads to significant performance improve-
ments.
Since SiamFC [19] is the baseline of our proposed MSRT,
we illustrate some tracking results obtained by SiamFC [19]
and our method on different videos in Fig. 1. The results
show that learning the multi-stream deep Siamese network
with the R-CNN detector, instead of using a single Siamese
search network, can achieve superior accuracy and real-time
performance at the same time.
The main contributions of this paper can be summarized as
follows:
• A new multi-stream deep Siamese network for real-time
object tracking is proposed. Unlike previous Siamese
based tracking methods, which cannot effectively deal
with significant target appearance changes, the proposed
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
tracking method can effectively alleviate this problem by
introducing the online update of the instance template into
a multi-stream Siamese network.
• An object re-identification algorithm, based on both the
metric learning based model and the Faster R-CNN detec-
tor, is also proposed. As a result, the proposed tracking
method can easily re-locate the lost target by using the
object category information (e.g., vehicles or pedestrians).
Experimental results demonstrate that our tracking method
outperforms several state-of-the-art tracking methods on three
public object tracking benchmarks. In particular, it can effec-
tively deal with the challenges of out-of-view and occlusion.
The rest of this paper is organized as follows. Related work for
visual tracking is reviewed in Section II. The implementation
details of the proposed method are introduced in Section III.
Extensive experiments and the analysis are presented in
Section IV, followed by the conclusion in Section V.
II. R ELATED W ORK
Deep learning [11], [13], [21], [22] has recently been used
in the object tracking task [24], [26], [27]. In general, most
deep learning based tracking methods are based on deep clas-
sification, and their running speeds are slow. In this section,
we briefly review the tracking methods closely related to our
work, including deep classification based tracking methods,
Siamese instance search based tracking methods and real-time
tracking methods.
A. Deep Classification Based Tracking Methods
A number of deep classification based tracking methods
have been developed in the literature [4], [17], [24]–[26],
[28], [29]. These methods aim to extract the target feature
representations that are robust to target appearance changes
by classifying the target candidates in each frame. HCF [26]
uses the convolutional feature maps generated by a CNN
(e.g., AlexNet [13] or VGG-Net [22]) to encode the target
appearance variations. CREST [4] integrates feature extrac-
tion, response map generation and model update into the
neural network in an end-to-end training manner. The methods
in [23]–[25], [28], [29] also focus on the deep features,
which are utilized to represent the target appearance. The
disadvantage of these methods is that they suffer from heavy
computational costs because the neural networks used to
evaluate a large number of candidate objects are complex. For
example, the multi-domain tracking method (MDNet) [17],
which is one of the most accurate tracking methods, can only
be implemented at 1 fps on a high-end GPU. SANet [18] is
also a highly accurate tracking method based on a recurrent
neural network, and it is much slower than MDNet. These
tracking methods are difficult to use in real-time applications.
B. Siamese Instance Search Based Tracking Methods
The Siamese instance search based tracking methods
(e.g., [19], [20], [30]–[32]) select the target from a search
region by using a Siamese network trained on a large-scale
dataset offline. The major part of a Siamese network is a
 LIU et al.: MULTI-STREAM SIAMESE AND FASTER REGION-BASED NEURAL NETWORK
Fig. 2. Pipeline of the proposed tracking method. The proposed tracking method consists of two modules: one is a multi-stream instance search module,
and the other is a re-identification and location module. These two modules are separately trained.
two-branch CNN. Two branches share the same parameters
and measure the similarity between the target object and the
target candidates cropped from a search region in each frame.
SiamFC [19] uses a fully convolutional network to perform
offline training and online object searching together, which
can be implemented in real time. SINT [20] uses a Siamese
network that contains a fully connected layer to learn a generic
matching function for tracking. However, SINT is much slower
(about 2 fps) than SiamFC (about 80 fps), because the fully
connected layer in SINT contains many parameters (more
than 4000). Based on [19] and [20], some effective tracking
methods (such as [30], [31], and [32]) have been proposed.
However, they only work in restricted environments due to
the lack of online update and re-identification. In this paper,
we propose to learn a multi-stream Siamese network by intro-
ducing an online update branch and an adaptive metric-based
re-identification detector to the Siamese network. Based on
these improvements, our method outperforms several other
state-of-the-art tracking methods on challenging videos.
C. Real-Time Tracking Methods
The running speed of a tracking method is critical to
many practical applications, such as intelligent transporta-
tion systems, video surveillance, motion analysis, and so
on [33], [34]. The recently proposed real-time tracking meth-
ods can be divided into two categories. Most of the track-
ing methods are in the first category, which is based on
correlation filters. This category of tracking methods can be
efficiently operated in the frequency domain. For example,
Bolme et al. [35] propose a new correlation filter based
tracking method via learning the minimum output sum of
squared error (MOSSE) filter, and this method can achieve
high running speed. Henriques et al. [36] introduce the ker-
nel function into the correlation filter and they propose the
kernelized correlation filter (KCF) tracking method. Later,
SRDCF [37] and Staple [38] are respectively proposed to
improve the performance of the correlation filters by solving
the boundary effects and extracting more effective features.
More efforts on improving correlation filter based tracking
methods can be found in [40]–[43]. Moreover, there are a few
real-time tracking methods proposed in the literature based
on the deep features (e.g., Re3 [44] and GOTURN [45]).
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
7281
However, these tracking methods lack online adaptability. In
this paper, we propose a deep Siamese instance search based
tracking method with online update and re-identification to
achieve a good tradeoff between running speed and tracking
accuracy.
III. T HE P ROPOSED M ETHOD
In this section, we present the proposed real-time object
tracking method, which combines a multi-stream Siamese
network and a region-based convolutional neural network
(MSRT). We firstly introduce the pipeline of the pro-
posed MSRT (see Fig. 2 for an illustration). Then we
describe the details of the multi-stream Siamese network and
the region-based convolutional neural network, respectively.
Finally, we show how to use the MSRT to achieve highly
accurate and real-time online tracking in detail.
A. Pipeline of the Proposed MSRT
MSRT contains two modules: a multi-stream instance search
module (termed MS) and a re-identification and location
module (termed RL). MS is a multi-stream Siamese network.
The inputs of MS consist of two templates and one search
region. One template I1 is cropped from the bounding box of
the target in the first frame. Meanwhile, the other template
I2 is cropped from a confident bounding box in the frame U.
Especially, the frame U is selected from the frames between
the first frame and the current frame with the highest confident
value based on the Mahalanobis distance. In each frame,
MS aims to search the target in the search region χ by using
two templates. Finally, MS outputs two response maps, and it
generates a large number of target candidates according to the
response maps.
RL consists of a metric learning (ML) based model and a
Faster R-CNN detector. It is employed to re-identify and locate
the target via the following steps. At first, the metric learning
based model in RL uses the target candidates generated by MS
as the inputs, and it outputs the estimated tracking state in the
current frame. The outputs contain two types of tracking states:
the target is tracked Ttrack or the target is lost Tlost . When the
target is tracked, RL locates the target by using the response
maps generated by MS, and the template I2 is not updated.
When the target is lost, the pre-trained Faster R-CNN detector
 7282 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021
Fig. 3. An illustration of the proposed multi-stream Siamese network. The
proposed network includes three streams sharing the same parameters with
one another.
re-identifies the target in the current frame by exploiting the
object category information. Meanwhile, the template I2 is
updated by RL after re-identification. Especially, the category
information in the first frame is identified by using the
pre-trained Faster R-CNN detector. The details of MS and RL
are described in the next two subsections, respectively.
B. MS: Multi-Stream Instance Search Module
To track objects with fast speed, we design MS as
a fully-convolutional neural network that includes several
streams sharing the same parameters with one another. As
shown in Fig 2, the inputs of MS include two templates (i.e.,
I1 , and I2 ) and a search region ( χ), and the outputs of MS are
two response maps and a large number of target candidates.
To avoid confusion, the input of the first stream is the target
template I1 , which is marked in the first frame. The second
stream takes the search region χ in the current frame as its
input. The input of the third stream is the second template I2 ,
which is selected by RL from the target candidates based on
the Mahalanobis distance. MS extracts the appearance features
from the three inputs and it uses these features to generate two
response maps as shown in Fig 2. MS aims to find the target
in the search region χ in each frame by using two templates
(i.e., I1 , and I2 ). The extracted appearance features are denoted
as ϕ(·). The two generated response maps Rm (·, ·) can be
written as follows:
Rm (I1 , χ) = corr (ϕ(I1), ϕ(χ)),
Rm (I2 , χ) = corr (ϕ(I2), ϕ(χ)),
where corr (·) is the correlation operation. Finally, MS outputs
a large number of target candidates based on the two response
maps in the frame. These target candidates are cropped from
the regions with high response scores in the two response maps
during tracking.
In this paper, we train MS using the following steps. At
first, we train the first stream and the second stream with
a discriminative loss, as shown in Fig 3. Different from
SiamFC [19] (which uses the logistic loss), we adopt the
contrastive loss [46] for MS because the contrastive loss is
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
Fig. 4. Examples of training pairs extracted from three video sequences. The
first row represents the instance templates and the second row represents the
search regions from the same video sequences.
more effective for the tracking task. The contrastive loss is
defined as follow:
ł(y[i ], ρ[i ]) = y[i ] · [max(Smargin − ρ[i ], 0)]2
+ (1 − y[i ]) · ρ[i ]2 , (2)
where Smargin is a distance threshold, and ρ[i ] =
xI [i ] − xC [i ] is the Euclidean distance between two feature
vectors (xI [i ] and xC [i ]) extracted from a template-candidate
pair, which comprises a template and a larger search region.
y[i ] ∈ {1, 0} is the ground-truth label provided in the training
data. After training these two streams, the third stream uses
the same parameters of the first stream.
We train MS by using a set of template-region pairs. The
template-region pairs are cropped from the annotated video
sequences in the ImageNet [8] dataset. To ensure that all
the search regions have the same size, the missing areas
are filled with the mean RGB value of the image using the
strategy of [19]. By randomly selecting target candidates in the
search region, a template-region pair will effectively generate
N template-candidate pairs. Fig. 4 illustrates examples of
training the template-region pairs extracted from three video
sequences. Finally, we define the contrastive loss as
1 
N
Ł(y, ρ) = ł(y[i ], ρ[i ]), (3)
N
i=1
The proposed MS is a fully-convolutional network without
any fully-connected layers, so it has fewer parameters than
the deep networks with fully-connected layers. Thus, MS runs
faster than the deep networks with fully connected layers and
it is able to achieve real-time performance.
(1)
C. RL: Re-Identification and Location Module
The RL module is designed for re-identification, and it
is an important module of the tracking method, especially
for long-term tracking. The RL module consists of a metric
learning based model and a Faster R-CNN detector. In this
paper, RL is employed to re-identify and locate the target,
and it formulates the re-identification and location problem as
a Markov Decision Process (MDP). At first, the inputs of RL
are two response maps and a large number of target candidates
generated by MS. Then, the metric learning based model in
RL uses the target candidates to estimate the tracking state
 LIU et al.: MULTI-STREAM SIAMESE AND FASTER REGION-BASED NEURAL NETWORK
(i.e., Ttrack or Tlost ) of MSRT in the current frame. Finally,
RL outputs the location of the tracked target.
Next, we will introduce the details of the main components
used in the proposed MSRT.
1) Metric Learning Based Model: To estimate the tracking
state of MSRT, we propose a model based on metric learning,
which consists of two Mahalanobis distance functions, namely
MLM. The inputs of MLM are the candidates generated by
MS, and the output is the tracking state of MSRT. MLM
estimates the tracking state by calculating the dissimilarity
scores between the target candidates and the two templates. If
the sum of the dissimilarity scores is higher than a threshold,
MLM outputs the tracking state as Tlost (i.e., the target is lost)
and sends the re-identification request to the Faster R-CNN
detector. Otherwise, MLM outputs the tracking state Ttrack
(i.e., the target is tracked) and locates the target in the current
frame through the two response maps generated by MS.
The dissimilarity function ψ(Ci , C j ) between two cropped
images Ci and C j measures the distance between two feature
vectors xCi and xC j extracted from Ci and C j , respectively.
And it is defined as follows:
ψ(Ci , C j ) = (xCi , xC j )
= (xCi − xC j )T M(xCi − xC j ), (4)
where (·) yields a dissimilarity score between the two
vectors xCi and xC j . M is a metric matrix, and it is usually
learned by two steps: dimensionality reduction (e.g., by using
principle component analysis (PCA)) is firstly applied to xCi
and xC j , and then metric learning (e.g., the KISS metric [47])
is performed.
Köstinger et al. [47] propose an effective and efficient
method to learn a Mahalanobis distance. We follow this
method to learn the metric matrix M. Given a pair of cropped
images Ci and C j , we define the pairwise differences as
xi j = xCi − xC j . Note that we partition the training data into
xi j + and xi j − . xi j + represents that Ci and C j are the cropped
images containing the same object, while xi j − represents that
Ci and C j are the cropped images containing different objects.
The metric matrix M is defined as follow:
 
M = ( (xi j + )(xi j + )T )−1 − ( (xi j − )(xi j − )T )−1 . (5)
In this paper, we use two metric matrices M A and M B ,
where M A is used for calculating the dissimilarity score
between one target candidate generated by MS and the tem-
plate I1 , and M B is used for calculating the dissimilarity
score between one target candidate generated by the first
response map and the other target candidate generated by
the second response map. These two metric matrices are
separately calculated. We train the two metric matrices M A
and M B with a large number of pairs of cropped images via
Eq. (5). We finally define our dissimilarity function (·) as
 (xCi , xC j ) = (xCi − xC j )T M A (xCi − xC j )
+ λ(xCi − xC j )T M B (xCi − xC j ), (6)
where λ is a constant coefficient.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
7283
2) Faster R-CNN Based Re-Identification Algorithm: For
the recently proposed tracking methods, the object appearance
model is usually learned without considering the object cate-
gory information, even though the object category information
is important information for object tracking, especially for
long-term object tracking. To effectively exploit the object
category information, we propose an object re-identification
algorithm, namely Faster RCNN based re-identification algo-
rithm, by using the Faster R-CNN detector.
The Faster R-CNN based re-identification algorithm consists
of two steps, including the initialization of the Faster R-CNN
detector and the re-identification of the target in the frames
when the target is lost. For the first frame, the bounding box
of the specified target is used as the initial input of the Faster
R-CNN detector for the initialization of the Faster R-CNN
detector. For the following frames, the Faster R-CNN detector
is used to re-identify the lost target by using the category
information of the target object when the target is lost. Firstly,
the cropped templates and the search region in the current
frame are used as the inputs of MS. MS generates two response
maps and a number of target candidates, which are further used
as the inputs of MLM. Secondly, MLM outputs the tracking
state (i.e., Ttrack or Tlost ). When the tracking state is Tlost ,
the Faster R-CNN detector will be activated and it will be
used to re-identify the target by using the object category
information. When the tracking state is Ttrack , RL locates
the target by using the two response maps generated by MS.
Instead of re-identifying the object in each frame, we re-
identify the object in the frames of the lost target, which can
reduce the computational costs.
In particular, we use the same network architecture as
in [48] to estimate the output by using the softmax func-
tion. The Max-pooling layers and ReLU layers are employed
after the convolutional layers. The training data of the net-
work include 20 object categories, which cover most object
categories (e.g., vehicles and pedestrians) in the tracking
dataset.
3) High-Confidence Template Update: Most recent tracking
methods [50]–[54] update the templates at each frame without
considering the tracking state. In fact, incorrect tracking results
(due to the target being severely occluded or completely lost)
may result in inaccurate template updates, which can further
lead to tracking failures.
For the frame that the estimated tracking state of MSRT is
Ttrack , both the shape of the response maps and the score of the
softmax function in the Faster R-CNN detector can be used to
indicate the estimated confidence degree of the tracking state
to some extent. The ideal response maps should have only one
sharp peak and they should be smooth in all other areas when
the detected target is matched to the templates. We only use
the most confident target candidate (which is selected based
on MLM) to update the template I2 .
We update the template I2 when the response maps have
sharper peaks and less noises. We transform the response maps
Rm (I1 , χ) and Rm (I2 , χ) into matrices S1 and S2 , respectively.
And then, we define the maximum response scores of two
response maps (which correspond to the first frame and the
U-th frame) as Smax1 and Smax2, respectively. We measure the
 7284 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021

distance of two response maps σ as follows: Algorithm 1 The Proposed MSRT

n∗m 1: Input:
|S1i − S2i |2
σ = i=1 , (7) 2: Image Ft .
n∗m
3: Previous target location lt −1 .
where S1i and S2i denote the elements of the matrices S1 and 4: Output:
S2 , respectively, and n ∗m is the size of the matrices. When the 5: The estimated target location lt .
estimated tracking state is Ttrack , and the response maps have /* Initialization of MSRT */
sharp peaks (which means that the Smax1 and Smax2 values of 6: if t = 1 then
the response maps are much higher than the average values of 7: Crop the bounding box and initialize the first template
the response maps) and low noise (i.e., they have low σ value), I1 .
the template I2 is updated. By using the above template update 8: Initialize the multi-stream Siamese network.
strategy, we update the template under certain conditions rather 9: Initialize the Faster R-CNN detector.
than updating it every frame. This way significantly reduces 10: end if
the computational costs. 11: if t = 2 then
12: Crop the search region χ2 .
D. Online Tracking 13: Generate the target candidates by using the multi-stream
We first train the multi-stream Siamese network, the Faster Siamese network.
R-CNN detector and the metric-learning based model offline. 14: Estimate the target location lt using Eq. (1).
Then, we implement the proposed MSRT for online tracking. 15: Initialize the second template I2 .
The whole algorithm of the proposed MSRT is summarized 16: end if
in Algorithm 1. /* Online tracking */
1) Initialization: At first, we crop a template I1 from the 17: if t = 3, . . . , T then
first frame, which is the labeled bounding box centered on the 18: Crop the search region χt .
target. The template I1 is used as the input of the proposed 19: Generate the target candidates by using the multi-stream
MSRT to initialize the multi-stream Siamese network and the Siamese network.
Faster R-CNN detector. Meanwhile, a search region χ2 that 20: Estimate the tracking state obtained by MSRT using
is centered on the target in the second frame, is cropped. Eq. (6).
Secondly, the template I1 and the search region χ2 are used as 21: if the target is tracked then
the input of the first two streams of MS. After that, MS outputs 22: Estimate the target location lt using Eq. (1).
a number of target candidates. RL takes these target candidates 23: else if the target is lost then
as its input, and it estimates the target location by selecting the 24: Re-identify the target and by using estimate the target
most similar target candidate (which has the highest response location lt by using the Faster R-CNN detector.
value). We crop the tracked target in the second frame as the 25: Update the template I2 .
template I2 . Finally, the proposed MSRT is well initialized 26: end if
after these steps. 27: end if
2) Tracking: As shown in Algorithm 1, the template I1 and
the template I2 are initialized by using the first two frames.
At time step t (t  3), we crop a search region χt from the method and several state-of-the-art tracking methods on the
target region predicted in the previous frame. The templates OTB-2013, OTB-2015 and LaSOT datasets are presented for
I1 , I2 and the search region χt are used as the inputs of MS to the performance evaluation.
generate two response maps and target candidates. Then these
target candidates are input to MLM to estimate the tracking
state of MSRT in the current frame. If the target is tracked, A. Experimental Settings
the proposed MSRT locates the target by search the region
1) Evaluation Metrics: In the tracking task, the most com-
with the maximum response value in the two response maps.
mon evaluation method is to initialize a tracking method with
Otherwise, RL is activated to re-identify the target by using the
the bounding box in the first frame and report the average
Faster R-CNN detector and then locates the target. During the
precision or success of all the results. This straightforward
process of online tracking, the proposed MSRT generates two
method is referred to as a one-pass evaluation (OPE). We
response scores (i.e, Smax1 and Smax2) for the two response
adopt the precision plot and the success plot of OPE as the
maps and estimates the distance σ between the two response
evaluation metrics [55], [56]. The precision plot depicts the
maps after predicting the target location. And then, MSRT
average center location error (CLE), which is the Euclidean
updates the template I2 with the most confident tracking result
distance between the center of the ground-truth target location
according to these three values (i.e, Smax1, Smax2 , and σ ).
and the center of the estimated target region at each frame.
The success plot shows the average overlap score (OS), which
IV. E XPERIMENTS is the area under the curve (AUC) between the ground-truth
In this section, the experimental settings are introduced first. target bounding box and the tracked target bounding box of
Then the comparison results between the proposed tracking each frame.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
 LIU et al.: MULTI-STREAM SIAMESE AND FASTER REGION-BASED NEURAL NETWORK 7285

2) Competing Methods: In our experiments, we compare

our method with other eight state-of-the-art tracking methods,
which can be roughly divided into two categories. The first cat-
egory includes two base-line Siamese based tracking methods
(i.e., SiamFC [19] and SINT [20]). The second one consists of
several recently proposed tracking methods that can run faster
than 5fps (including BACF [40], Staple [38], GOTURN [45],
Re3 [44], SRDCF [37] and KCF [36]).
3) Implementation Details: Each branch of the multi-stream
Siamese network uses the AlexNet-like [13] network, which Fig. 5. The precision and success plots of OPE on the OTB-2013 dataset.
consists of five convolutional layers. Max pooling is only The solid lines in different colors represent 9 different tracking methods. The
employed after the first two convolutional layers. In addition, center location errors at a threshold of 20 pixels for precision plots and the
average overlap scores for success plots are respectively shown in the legend.
the batch normalization [59] is adopted after each convolu- (a) Precision plots of OPE. (b) Success plots of OPE.
tional layer. We use the ReLU layer as the non-linear function
of the hidden layer. The Faster R-CNN detector uses the same
network structure as in [48]. The number of candidates output
by the MS module affects the performance of MSRT. In the
experiments, when this number is set as 500, MSRT achieves
the highest accuracy, while maintaining real-time performance.
In addition, the proposed MSRT uses k different scales to track
the target. The value of k denotes the number of layer in the
image pyramid structure, and it affects the tracking accuracy
and speed of MSRT. When k is large (such as 5), the accuracy
improvement of the proposed method is limited, but its speed Fig. 6. The speed and accuracy plots obtained by the state-of-the-art tracking
methods on the OTB-2015 [55] dataset. The proposed MSRT achieves the best
is much slower. In this paper, in order to maintain real-time accuracy among the real-time tracking methods.
tracking, k is set as 3 to achieve a good tradeoff between the
running speed and tracking accuracy.
4) Training: We adopt the 3-step training algorithm for
MSRT. Firstly, we train the proposed MS by using the
ILSVRCDET dataset [8]. The parameters of the embedding
function of MS are estimated by minimizing Eq. (2) with the
Stochastic Gradient Descent (SGD) using MatConvNet [60],
and the threshold score Smargin is experimentally set to 0.5.
The training process is performed over 100 epochs, and we
set the size of mini-batches as 8 in each iteration. Secondly,
we train the Faster R-CNN, which consists of two components:
the RPN and the Fast R-CNN. We use the PASCAL VOC Fig. 7. The precision and success plots of OPE on the OTB-2015 dataset.
The solid lines in different colors represent 9 different tracking methods. The
2010 dataset [49] (which includes 20 types of objects) for center location errors at a threshold of 20 pixels for precision plots and the
training, since the category information in it is sufficient to average overlap scores for success plots are respectively shown in the legend.
cover most categories in the tracking dataset. More specifi- (a) Precision plots of OPE. (b) Success plots of OPE.
cally, we first train the RPN, and then we use the proposals
generated by the RPN to train the Fast R-CNN. We train the
the whole image. The proposed MSRT is implemented using
Fast R-CNN using 500 proposals. Thirdly, we train the metric
MATLAB on a computer equipped with a single NVIDIA
learning based model MLM of RL with ILSVRCDET [8] and
GTX 1080 GPU and an Intel 6700K 4.0 GHz CPU. The
set the threshold of the score of the dissimilarity function 
proposed MSRT runs at 27 frames per second.
in MLM to 0.4.
5) Tracking: We set the size of the templates as 127 × 127,
and set the size of the search region as 255 × 255, which is B. Experiments on the OTB-2013 Dataset
four times as large as the size of the templates. We conduct We compare the proposed MSRT with 8 other tracking
some experiments to analyze the size of the search region. This methods (including SINT, KCF, SiamFC, BACF, SRDCF,
size of the search region is generally sufficient to cover the Staple, Re3 and GOTURN), on the OTB-2013 dataset [56].
target position in the next frame in practice. And when the size The evaluation is based on the precision and success plots
of the search region is large enough (i.e., over four times of of one-pass evaluation on the 50 video sequences of the
the template), the increasing of the size has less influence on OTB-2013 dataset. As shown in Fig. 5, the proposed MSRT
the accuracy, while the computational costs are significantly achieves the second best performance in the precision plots of
increased. However, when target is lost, the proposed tracking OPE and the best performance in the success plots of OPE.
method needs lager search region. This is the reason why The proposed MSRT performs significantly better than the
we use a fast R-CNN detector to re-identify the target in real-time tracking methods based on deep networks [44], [45].

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
 7286 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021

Fig. 8. Qualitative evaluation of 7 tracking methods on 7 challenging sequences (from top to bottom: Car Scale, Biker1, Basketball, Singer2, T iger,
Coke, Bird1) from the OTB-2015 dataset. The proposed MSRT performs favorably against the 6 state-of-the-art competing tracking methods, including
SINT [20], KCF [36], SiamFC [19], BACF [40], SRDCF [37] and Staple [38].

Among the 9 competing tracking methods in Fig. 5, the pro-

posed MSRT (66.0%/87.8%) shows better performance than
the KCF [36] (51.3%/74.1%), SiamFC [19] (60.6%/80.1%),
BACF [40] (61.9%/80.7%), SRDCF [37] (62.6%/83.8%),
Staple [38] (60.0%/79.3%), Re3 [44] (31.6%/46.0%) and
GOTURN [45] (44.4%/62.0%), and it shows comparable
results to the SINT [20] (65.5%/88.2%). Although SINT
obtains the best performance in the precision plots of OPE,
it runs at only 4 fps and achieves a lower average OS than Fig. 9. The precision and success plots of OPE on the LaSOT dataset. The
solid lines in different colors represent the results obtained by 9 different
MSRT with regard to the success plots of OPE. These results tracking methods. The center location errors at a threshold of 20 pixels for
show the superior effectiveness and efficiency of the proposed precision plots and the average overlap scores for success plots are respectively
tracking method. shown in the legend. (a) Precision plots of OPE. (b) Success plots of OPE.

C. Experiments on the OTB-2015 Dataset with the 8 competing tracking methods on all 100 videos
The OTB-2015 [55] dataset is an extension of the of the OTB-2015 dataset. Fig. 6 shows the speed and accu-
OTB-2013 dataset and it is more challenging than the racy plots obtained by the 9 competing tracking methods
OTB-2013 dataset. We also compare the proposed MSRT on the OTB-2015 dataset. The proposed MSRT method

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
 LIU et al.: MULTI-STREAM SIAMESE AND FASTER REGION-BASED NEURAL NETWORK 7287

Fig. 10. Attribute based evaluation. The experimental results on 11 different challenging factors (including scale variation, out-of-plane rotation, in-plane
rotation, occlusion, deformation, fast motion, illumination variation, background clutter, motion blur, out of view and low resolution.) are presented. The
number of video sequences for each attribute challenging factors is shown in the parenthesis.

achieves the best accuracy among the 9 competing track-
ing methods, and it maintains real-time tracking speed.
The detailed precision and success plots of OPE obtained
by each tracking method are also reported in Fig. 7.
MSRT (64.3%/86.2%) shows better performance than the
other 8 tracking methods, including KCF (47.5%/69.2%),
SiamFC (58.1%/76.9%), BACF (60.1%/79.3%), SRDCF
(59.8%/78.9%), Staple (58.2%/78.4%), Re3 (28.9%/40.6%),
GOTURN (42.7%/62.0%) and SINT (59.2%/78.6%).
D. Experiments on the LaSOT Dataset
The Large-scale Single Object Tracking (LaSOT)
dataset [57] is a large-scale dataset, which consists
of 1,400 sequences with more than 3.52 million frames.
The average length of the videos in LaSOT is more than
2,500 frames. Some sequences in LaSOT can be used for
long-term tracking, and each sequence in this dataset is
very challenging. We compare the proposed MSRT with the
8 competing tracking methods (including SiamFC, SINT,
BACF, Staple, SRDCF, GOTURN, KCF and LCT [58]) on the
LaSOT dataset. Fig. 9 shows the precision and success plots
obtained by the 9 competing tracking methods on the LaSOT
dataset. As shown in Fig. 9, the proposed MSRT method
achieves the best accuracy among the 9 competing tracking
methods. The detailed precision and success plots of OPE
obtained by each tracking method are also reported in Fig. 9.
MSRT (39.9%/35.7%) achieves better performance than the
other 8 tracking methods: SiamFC (35.8%/34.1%), SINT
(33.9%/29.9%), BACF (27.7%/23.9%), Staple (26.6%/23.1%),
SRDCF (27.1%/22.7%), GOTURN (22.5%/17.9%), KCF
(21.1%/18.4%) and LCT [58] (24.6%/19.3%). The propsed
MSRT is more effective than the 8 competing tracking
methods for long-term tracking. Although MSRT achieves
only 1% better performance than the baseline SiamFC in
terms of the precision, it obtains over 4% improvement in
terms of the success. Compared with another Siamese based
tracking method (i.e., SINT), the MSRT achieves over 5%
performance improvement in terms of the precision on the
LaSOT dataset. These results show the superior effectiveness
and efficiency of the proposed tracking method. In addtion,
LCT is a well-known long-term tracking method. The
proposed MSRT achieves better performance than the LCT
method on the LaSOT dataset. Because the proposed MSRT
takes the advantages of both the template update and object
re-identification.
E. Qualitative Evaluation and Attribute Analysis
Fig. 8 summarizes the qualitative comparisons of MSRT
with 6 competing tracking methods (SINT, KCF, SiamFC,
BACF, SRDCF, and Staple) on 7 challenging sequences
from OTB-2015. These challenging sequences contain most
of the challenging attributes. Overall, the proposed MSRT
achieves better results than the other 6 competing methods.
For the Car Scale sequence, the scale of the target changes
significantly and the proposed MSRT achieves the highest
accuracy. For the Bi ker 1 sequence, it includes occlusion and
fast motion, Staple, KCF and SRDCF lose the target in frame
75. For the T iger sequence, it includes motion blur and fast
motion, all the tracking methods lose the target in frame 150,
and the proposed MSRT re-identifies the target after a few

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
 7288 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021

Fig. 11. The overlap score (OS) plots obtained by the 9 tracking methods over 6 challenging video sequences. The solid lines in different colors represent
the results obtained by different tracking methods. The larger OS values indicate the better performance (best viewed in color).

frames. For the Coke sequence, all the tracking methods fail
to track the target when the coke is in the situation of in-plane
rotation, except for the proposed MSRT. For the Si ngle2
sequence, the appearance of the target changes significantly
due to shadows. SiamFC gradually drifts away from the target
when the singer turns around. For the Basketball sequence,
some tracking methods fail to reliably track the target (i.e.,
the human) because of the illumination variations. The BACF
and the proposed MSRT are able to track the target accurately
for this sequence. For the Bir d1 sequence, it includes full
occlusion and fast motion, which make the tracking task
challenging. Only the proposed MSRT performs accurately
across the whole sequence.
In Fig. 10, we further analyze the tracking method perfor-
mance on all 11 video attributes (e.g., illumination variation
and out-of-plane rotation) annotated in the OTB-2015 dataset.
The results show that the proposed MSRT achieves the best
results on the videos annotated with 10 out of 11 attributes.
Moreover, we find that MSRT is more effective at handling
the out-of-view (OV) and occlusion (OC) attributes because
it can re-identify the lost target by using the metric learning
based model (MLM) and the Faster R-CNN detector. When
the target is occluded or lost, the two base-line Siamese
based tracking methods (i.e., SiamFC and SINT) can not
accurately locate the target. This is because both methods do
not include the template update and target re-identification
procedures. Fig. 11 shows the frame-by-frame OS plots
obtained by all 9 competing tracking methods over six
challenging video sequences. The proposed MSRT achieves
the best performance among all the competing tracking
methods.
However, for the low-resolution (LR) videos, we also note
that the performance of the proposed MSRT is worse than that
of SiamFC. This is because Siamese based tracking methods
track the target by using one template. When the template is a
low-resolution template, these tracking methods will lose the
target. In contrast, the proposed MSRT tracks the target by
using two low-resolution templates, which makes MSRT lose
the target frequently.
F. Tracking on Traffic Video Sequences
The OTB-2015 dataset contains 14 traffic video sequences,
which include vehicles and pedestrians. Fig. 12 shows the
frame-by-frame center location error plots obtained by the
9 competing tracking methods over 9 challenging video
sequences. The solid lines in different colors represent the
results obtained by different tracking methods. From Fig. 12,
we can see that the proposed MSRT achieves lower center
location errors than the other competing methods, which
demonstrates the robustness and the effectiveness of the pro-
posed MSRT. Table I shows the performance comparison on
the 13 traffic video sequences in terms of the average overlap
score. From Table I, we can see that the vehicle in the Car 1,
Car 2, Car 24, Car 4, Car Dar k sequences is successfully
tracked by 4 tracking methods (i.e., SINT, Staple, BACF, and
MSRT), among which SINT, BACF, and MSRT track the target
more accurately on these vehicle sequences. In addition, for
the human2, human4, human5, human6, human7, human8,
human9 video sequences, BACF, Staple and MSRT perform
well on the sequences. In particular, the human3 sequence is
very challenging because it contains five different challenging
factors, including scale variation, occlusion, deformation, out-
of-plane rotation and background clutter. The proposed MSRT
achieves better performance than these competing tracking
methods on this sequence. In short, the proposed MSRT
achieves the best overall performance among these competing
tracking methods on the traffic video sequences.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
 LIU et al.: MULTI-STREAM SIAMESE AND FASTER REGION-BASED NEURAL NETWORK 7289

Fig. 12. The center location error (CLE) plots obtained by the 9 tracking methods over 9 challenging video sequences. The solid lines in different colors
represent the results obtained by different tracking methods. The smaller CLE values indicate the better performance (best viewed in color).

TABLE I
T HE AVERAGE C ENTER L OCATION E RRORS AT A T HRESHOLD OF 20 P IXELS ON 13 T RAFFIC V IDEO S EQUENCES OF THE OTB-2015 D ATASET O BTAINED
BY THE P ROPOSED MSRT AND THE O THER 8 T RACKING M ETHODS . T HE B EST VALUE I S H IGHLIGHTED BY B OLD

G. Ablation Study
In this section, we evaluate two variants of the proposed
MSRT. The first one is the multi-stream Siamese network
based tracking method (MST), which is the proposed MSRT
that does not use the Faster R-CNN detector. The second one
is the single Siamese and faster region-based convolutional
neural network based tracking method (SRT), which elimi-
nates the third stream of the multi-stream Siamese network
from the proposed MSRT. We report the average center
location errors at a threshold of 20 pixels for the preci-
sion (ACLS20), the average overlap score (AOS) and the
speed on two different datasets (e.g., OTB-2015 and LaSOT)
in Table II.
1) MSRT vs. MST: Table II shows the tracking results of
MST on the OTB-2015 dataset. MST is much faster than
MSRT, but it achieves a lower ACLS20 and AOS than MSRT.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
 7290 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021
TABLE II
T HE ACLS20, THE AOS AND THE S PEED O BTAINED BY THE P ROPOSED
MSRT, MST AND SRT ON THE OTB-2015 AND THE LaSOT D ATASETS
In MSRT, the Faster R-CNN detector aims to re-identify the
lost target for long-term tracking. If the tracking method loses
the target object, the Faster R-CNN detector can re-identify
the target again. Even if the target appearance has significantly
changed, the Faster R-CNN detector can still locate it by using
the object category information. A reliable Faster R-CNN
detector performs more accurately on the out-of-view and
occlusion attributes.
2) MSRT vs. SRT: Table II also shows the tracking results
of SRT on the OTB-2015 dataset. SRT achieves the lowest
ACLS20 and AOS among the three tracking methods, and it
is slightly faster than the others. The proposed MSRT uses
two template streams to improve the accuracy of the Siamese
based tracking method. Different from the existing Siamese
based tracking methods, the proposed MSRT uses one stream
to update the template online. We test SRT without using the
third stream, which is used to update the template. As shown
in Table II, SRT obtains faster running speed because it does
not need to consume time for template updating. However,
removing the third stream also leads to much less accurate
performance, especially for AOS. Note that the AOS of SRT is
even worse than the AOS of SiamFC. These results shows that,
a Siamese based tracking method achieves better performance
in terms of ACLS20 by using the Faster R-CNN detector,
but it also leads to worse performance in terms of AOS.
The proposed MSRT combines the advantages of both the
Siamese based tracking method and the Faster R-CNN detector
to achieve better performance.
V. C ONCLUSION
In this paper, we propose a new Siamese search based
tracking method, which effectively combines a multi-stream
Siamese network and a region-based convolutional neural
network, namely MSRT. The proposed MSRT method is able
to provide the reliable online tracking accuracy, while maintain
real-time tracking speed. More specifically, we treat object
tracking as two sub-tasks: Siamese search based tracking
and re-identification. Unlike previous Siamese search based
tracking methods, which cannot effectively deal with the sig-
nificant appearance changes of a target, the proposed tracking
method can achieve robust tracking results by updating the
template in the multi-stream Siamese network. Moreover,
a re-identification algorithm based on metric-learning is also
proposed. Therefore, the proposed method can re-locate a lost
target during tracking. Extensive experiments show that the
proposed MSRT can effectively track objects in challenging
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
video sequences, and it can also effectively track vehicles and
pedestrians in traffic video sequences. Compared with several
state-of-the-art tracking methods, the proposed MSRT achieves
better tracking accuracy. In future work, the proposed MSRT
can be extended to multiple object tracking by using multiple
templates and networks, since it is an instance search based
tracking-by-detection method for object tracking.
R EFERENCES
[1] B. Babenko, M.-H. Yang, and S. Belongie, “Robust object tracking with
online multiple instance learning,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 33, no. 8, pp. 1619–1632, Aug. 2011.
[2] S. Zhang, Y. Qi, F. Jiang, X. Lan, P. C. Yuen, and H. Zhou, “Point-to-
set distance metric learning on deep representations for visual tracking,”
IEEE Trans. Intell. Transport. Syst., vol. 19, no. 1, pp. 187–198,
Jan. 2018.
[3] Z. Kalal, K. Mikolajczyk, and J. Matas, “Tracking-learning-detection,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1409–1422,
Jul. 2012.
[4] Y. Song, C. Ma, L. Gong, J. Zhang, R. W. H. Lau, and M.-H. Yang,
“CREST: Convolutional residual learning for visual tracking,” in Proc.
IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 2555–2564.
[5] M. Danelljan, G. Hager, F. S. Khan, and M. Felsberg, “Discriminative
scale space tracking,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39,
no. 8, pp. 1561–1575, Aug. 2017.
[6] S. Hare et al., “Struck: Structured output tracking with kernels,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 38, no. 10, pp. 2096–2109,
Oct. 2016.
[7] G. Ding, W. Chen, S. Zhao, J. Han, and Q. Liu, “Real-time scalable
visual tracking via quadrangle kernelized correlation filters,” IEEE
Trans. Intell. Transport. Syst., vol. 19, no. 1, pp. 140–150, Jan. 2018.
[8] O. Russakovsky et al., “ImageNet large scale visual recognition chal-
lenge,” Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, Dec. 2015.
[9] N. Wang, D. Tao, X. Gao, X. Li, and J. Li, “Transductive face sketch-
photo synthesis,” IEEE Trans. Neural Netw. Learn. Syst., vol. 24, no. 9,
pp. 1364–1376, Sep. 2013.
[10] N. Wang, X. Gao, L. Sun, and J. Li, “Bayesian face sketch synthesis,”
IEEE Trans. Image Process., vol. 26, no. 3, pp. 1264–1274, Mar. 2017.
[11] X. Ke, J. Zou, and Y. Niu, “End-to-end automatic image annotation
based on deep CNN and multi-label data augmentation,” IEEE Trans.
Multimedia, vol. 21, no. 8, pp. 2093–2106, Aug. 2019.
[12] Y. Niu, J. Chen, and W. Guo, “Meta-metric for saliency detection
evaluation metrics based on application preference,” Multimedia Tools
Appl., vol. 77, no. 20, pp. 26351–26369, Mar. 2018.
[13] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
Process. Syst., Dec. 2012, pp. 1097–1105.
[14] Y. Niu, H. Zhang, W. Guo, and R. Ji, “Image quality assessment for color
correction based on color contrast similarity and color value difference,”
IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 4, pp. 849–862,
Apr. 2018.
[15] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierar-
chies for accurate object detection and semantic segmentation,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014, pp. 580–587.
[16] R. Girshick, “Fast R-CNN,” in Proc. IEEE Int. Conf. Comput. Vis.
(ICCV), Dec. 2015, pp. 1440–1448.
[17] H. Nam and B. Han, “Learning multi-domain convolutional neural
networks for visual tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2016, pp. 4293–4302.
[18] H. Fan and H. Ling, “SANet: Structure-aware network for visual
tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops
(CVPRW), Jul. 2017, pp. 2217–2224.
[19] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr,
“Fully-convolutional siamese networks for object tracking,” in Proc. Eur.
Conf. Comput. Vis. Workshops, Oct. 2016, pp. 850–865.
[20] R. Tao, E. Gavves, and A. W. M. Smeulders, “Siamese instance search
for tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2016, pp. 1420–1429.
[21] I. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv. Neural
Inf. Process. Syst., Dec. 2014, pp. 2672–2680.
[22] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” in Proc. Int. Conf. Learn. Repres.,
May 2015, pp. 1–14.
 LIU et al.: MULTI-STREAM SIAMESE AND FASTER REGION-BASED NEURAL NETWORK
[23] Y. Niu, L. Lin, Y. Chen, and L. Ke, “Machine learning-based framework
for saliency detection in distorted images,” Multimedia Tools Appl.,
vol. 76, no. 24, pp. 26329–26353, Dec. 2017.
[24] J. Valmadre, L. Bertinetto, J. Henriques, A. Vedaldi, and P. H. S. Torr,
“End-to-end representation learning for correlation filter based tracking,”
in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017,
pp. 5000–5008.
[25] T. Liu, X. Cao, and J. Jiang, “Visual object tracking with partition
loss schemes,” IEEE Trans. Intell. Transport. Syst., vol. 18, no. 3,
pp. 633–642, Mar. 2017.
[26] C. Ma, J.-B. Huang, X. Yang, and M.-H. Yang, “Hierarchical convolu-
tional features for visual tracking,” in Proc. IEEE Int. Conf. Comput.
Vis. (ICCV), Dec. 2015, pp. 3074–3082.
[27] M. Gao, L. Jin, Y. Jiang, and B. Guo, “Manifold siamese network:
A novel visual tracking convnet for autonomous vehicles,” IEEE Trans.
Intell. Transport. Syst., vol. 21, no. 4, pp. 1612–1623, Apr. 2020.
[28] S. Pu, Y. Song, C. Ma, H. Zhang, and M.-H. Yang, “Deep attentive
tracking via reciprocative learning,” in Proc. Adv. Neural Inf. Process.
Syst., Dec. 2018, pp. 1935–1945.
[29] G. Bhat, J. Johnander, M. Danelljan, F. S. Khan, and M. Felsberg,
“Unveiling the power of deep tracking,” in Proc. Eur. Conf. Comput.
Vis., Sep. 2018, pp. 493–509.
[30] Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, and W. Hu, “Distractor-
aware Siamese networks for visual object tracking,” in Proc. Eur. Conf.
Comput. Vis. (ECCV), Oct. 2018, pp. 103–119.
[31] X. Wang, C. Li, B. Luo, and J. Tang, “SINT++: Robust visual tracking
via adversarial positive instance generation,” in Proc. IEEE/CVF Conf.
Comput. Vis. Pattern Recognit., Jun. 2018, pp. 4864–4873.
[32] A. He, C. Luo, X. Tian, and W. Zeng, “A twofold siamese network
for real-time object tracking,” in Proc. IEEE/CVF Conf. Comput. Vis.
Pattern Recognit., Jun. 2018, pp. 4834–4843.
[33] Y. Niu, W. Lin, X. Ke, and L. Ke, “Fitting-based optimisation for
image visual salient object detection,” IET Comput. Vis., vol. 11, no. 2,
pp. 161–172, Mar. 2017.
[34] Y. Niu, W. Lin, and X. Ke, “CF-based optimisation for saliency
detection,” IET Comput. Vis., vol. 12, no. 4, pp. 365–376, Jun. 2018.
[35] D. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object
tracking using adaptive correlation filters,” in Proc. IEEE Comput. Soc.
Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 2544–2550.
[36] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed
tracking with kernelized correlation filters,” IEEE Trans. Pattern Anal.
Mach. Intell., vol. 37, no. 3, pp. 583–596, Mar. 2015.
[37] M. Danelljan, G. Hager, F. S. Khan, and M. Felsberg, “Learning spatially
regularized correlation filters for visual tracking,” in Proc. IEEE Int.
Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 4310–4318.
[38] L. Bertinetto, J. Valmadre, S. Golodetz, O. Miksik, and P. H. S. Torr,
“Staple: Complementary learners for real-time tracking,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016,
pp. 1401–1409.
[39] X. Ke, M. Zhou, Y. Niu, and W. Guo, “Data equilibrium based automatic
image annotation by fusing deep model and semantic propagation,”
Pattern Recognit., vol. 71, pp. 60–77, Nov. 2017.
[40] H. K. Galoogahi, A. Fagg, and S. Lucey, “Learning background-aware
correlation filters for visual tracking,” in Proc. IEEE Int. Conf. Comput.
Vis. (ICCV), Oct. 2017, pp. 1144–1152.
[41] M. Wang, Y. Liu, and Z. Huang, “Large margin object tracking with
circulant feature maps,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jul. 2017, pp. 4800–4808.
[42] Y. Yao, X. Wu, L. Zhang, S. Shan, and W. Zuo, “Joint representation
and truncated inference learning for correlation filter based tracking,” in
Proc. Eur. Conf. Comput. Vis., Sep. 2018, pp. 560–575.
[43] M. Zhang et al., “Visual tracking via spatially aligned correlation filters
network,” in Proc. Eur. Conf. Comput. Vis., Sep. 2018, pp. 484–500.
[44] D. Gorden, A. Farhadi, and D. Fox, “Re3: Real-time recurrent regression
networks for object tracking,” IEEE Robot. Autom. Mag., vol. 3, no. 2,
pp. 788–795, Apr. 2018.
[45] D. Held, S. Thrun, and S. Savaresei, “Learning to track at 100 fps with
deep regression networks,” in Proc. Eur. Conf. Comput. Vis., Oct. 2016,
pp. 749–765.
[46] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by
learning an invariant mapping,” in Proc. IEEE Comput. Soc. Conf.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2006, pp. 1735–1742.
[47] M. Kostinger, M. Hirzer, P. Wohlhart, P. M. Roth, and H. Bischof, “Large
scale metric learning from equivalence constraints,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., Jun. 2012, pp. 2288–2295.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.
7291
[48] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-
time object detection with region proposal networks,” in Proc. Adv.
Neural Inf. Process. Syst., Dec. 2015, pp. 91–99.
[49] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and
A. Zisserman, “The Pascal visual object classes (VOC) challenge,” Int.
J. Comput. Vis., vol. 88, no. 2, pp. 303–338, Jun. 2010.
[50] D. Comaniciu, V. Ramesh, and P. Meer, “Kernel-based object tracking,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 5, pp. 564–577,
May 2003.
[51] Z. Hong, Z. Chen, C. Wang, X. Mei, D. Prokhorov, and D. Tao, “MUlti-
store tracker (MUSTer): A cognitive psychology inspired approach to
object tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2015, pp. 749–758.
[52] H. Nam, M. Baek, and B. Han, “Modeling and propagating CNNs in
a tree structure for visual tracking,” 2016, arXiv:1608.07242. [Online].
Available: http://arxiv.org/abs/1608.07242
[53] J. Ning, J. Yang, S. Jiang, L. Zhang, and M.-H. Yang, “Object
tracking via dual linear structured SVM and explicit feature map,” in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016,
pp. 4266–4274.
[54] S. Zhang, S. Zhao, Y. Sui, and L. Zhang, “Single object tracking
with fuzzy least squares support vector machine,” IEEE Trans. Image
Process., vol. 24, no. 12, pp. 5723–5738, Dec. 2015.
[55] Y. Wu, J. Lim, and M. H. Yang, “Object tracking benchmark,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 37, no. 9, pp. 1834–1848,
Sep. 2015.
[56] Y. Wu, J. Lim, and M.-H. Yang, “Online object tracking: A bench-
mark,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2013,
pp. 2411–2418.
[57] H. Fan et al., “LaSOT: A high-quality benchmark for large-scale
single object tracking,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2019, pp. 5369–5378.
[58] C. Ma, X. Yang, C. Zhang, and M.-H. Yang, “Long-term correlation
tracking,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jun. 2015, pp. 5388–5396.
[59] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in Proc. Int. Conf.
Mach. Learn., Jul. 2015, pp. 448–456.
[60] A. Vedaldi and K. Lenc, “MatConvNet: Convolutional neural net-
works for MATLAB,” in Proc. ACM Int. Conf. Multim., Oct. 2015,
pp. 448–456.
Yi Liu received the B.S. degree from the School
of Automation, Beijing Institute of Technology,
in 2011, and the M.S. degree from the School
of Aeronautic Science and Engineering, Beihang
University, in 2015. He is currently pursuing the
Ph.D. degree with the School of Informatics, Xiamen
University, China. His current research interests
include pattern recognition, metric learning, and
visual tracking.
Liming Zhang (Member, IEEE) received the B.S.
degree in computer software from Nankai Univer-
sity, China, the M.S. degree in signal processing
from the Nanjing University of Science and Technol-
ogy, China, and the Ph.D. degree in image process-
ing from the University of New England, Australia.
She is currently an Assistant Professor with the
Faculty of Science and Technology, University of
Macau. Her research interests include signal process-
ing, image processing, computer vision, and IT in
education.
 7292 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 22, NO. 11, NOVEMBER 2021

Zhihui Chen received the B.S. degree from the
School of Informatics, Xiamen University, China,
in 2011, where he is currently pursuing the Ph.D.
degree with the School of Informatics. His current
research interests include pattern recognition, com-
puter vision, and machine learning.
Hanzi Wang (Senior Member, IEEE) received the
Ph.D. degree in computer vision from Monash
University, Australia. He is currently a Distin-
guished Professor of “Minjiang Scholars” in Fujian
province and the Founding Director of the Cen-
ter for Pattern Analysis and Machine Intelligence
(CPAMI), Xiamen University, China. His research
interests include computer vision and pattern recog-
nition, including visual tracking, robust statistics,
object detection, video segmentation, model fitting,
3D structure from motion, image segmentation and
its related fields.

Yan Yan (Member, IEEE) received the Ph.D. degree

in information and communication engineering from
Tsinghua University, China, in 2009. From 2009 to
2010, he worked as a Research Engineer with the
Research and Development Center, Nokia Japan,
and a Project Leader with Panasonic Singapore Lab
in 2011. He is currently an Associate Professor
with the School of Informatics, Xiamen University,
China. He has published over 60 papers in interna-
tional journals and conferences. His research inter-
ests include computer vision and pattern recognition.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY ROORKEE. Downloaded on January 30,2023 at 17:51:38 UTC from IEEE Xplore. Restrictions apply.