2020 Eighth International Conference on Advanced Cloud and Big Data (CBD)
Video Object Detection based on Non-local Prior of Spatiotemporal Context*
2020 Eighth International Conference on Advanced Cloud and Big Data (CBD) | 978-1-6654-2313-7/20/$31.00 ©2020 IEEE | DOI: 10.1109/CBD51900.2020.00040
Wei Lu1, Wei Xu2, Zebin Wu1, Yang Xu1, Zhihui Wei1
1
School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing 210094, China,
2
China Railway Shanghai Group Co., Ltd., Nanjing Power Supply Section, NangJing 210011, China
Abstract—The appearances of objects in video sequence
affected by complex background, motion blur and partial
occlusion, which make the object detection in video sequence a
hard work. Due to these problems, traditional image object
detection methods cannot perform well in video sequence
image. A fast and effective video object detection method is
necessary to improve the detection efficiency. In this paper, we
propose a non-local prior based spatiotemporal attention
model based for video object detection. Unlike existing
attention models, the proposed model can make full use of the
spatiotemporal contextual information extracted from video
sequence images. We apply our models in common object
detection framework and evaluate it on Overhead Contact
System (OCS) driving recorder dataset and OTB50 dataset.
The proposed model achieves a greater increase in mAP value
which proves our model can gains good performance in
various complex video sequences.
Keywords—Non-local prior; video object detection; Yolov2;
convolutional neural network
I. INTRODUCTION
Object detection in images has significant achieved
success in recent years due to the emergence of deep
convolutional neural networks (CNNs). The deep neural
network mainly use convolution kernel to extract image
features and preserve the input’s neighborhood relations in
their higher-level feature representations. The convolution
layer has multiple kernels to extract multiple features of the
image. The kernels of a layer use the same parameters, so
the convolutional neural network has less calculation but
more effective. Compared with traditional methods, CNNs
can extract features from massive data more effectively. The
application of deep convolutional neural networks in object
detection mainly has two categories, one is two-stage
detection algorithms such as [8][9][10], and the other is one-
stage detection frameworks like [11][12][13].
The two-stage detection algorithms divide the target
detection into several steps. First, the interest of regions(ROI)
are generated from image to find the areas that are most
likely to contain objects. If the region proposal algorithm is
slow or the areas proposed are too much, the detection speed
*This work was supported in part by the National Natural Science
Foundation of China under Grant 61772274, Grant 61701238, Grant
61471199, Grant 11431015, and Grant 61671243, in part by the Jiangsu
Provincial Natural Science Foundation of China under Grant BK20180018
and Grant BK20170858, in part by the Fundamental Research Funds for
the Central Universities under Grant 30919011103, Grant 30917015104,
and Grant 30919011234, and in part by the China Postdoctoral Science
Foundation under Grant 2017M611814. (Corresponding author: Zebin Wu.
978-1-6654-2313-7/20/$31.00 ©2020 IEEE
DOI 10.1109/CBD51900.2020.00040
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 04:55:31 UTC from IEEE Xplore. Restrictions apply.
will be slow. Moreover, the accuracy of the region proposal
algorithm has a great impact on the detection accuracy. If the
region proposal algorithm has a poor performance, the
detection accuracy will also be worse.
The one-stage detection algorithms adopt anchor idea for
region proposals, and then directly returns the target's
category probability and position offset to get the final result.
The parameters of the anchor are determined by the label
clustering of the training set. Since the position of the
anchors are determined in advance, the speed is greatly
accelerated compared to the two-stage detection algorithms,
and the accuracy is also ensured.
Video object detection is an important component of
image object detection. With the rapid growth of video data,
video object detection has attracted more attention.
Compared with still images, video frames provide richer
temporal information, which can be employed to improve the
accuracy of object detection, but also have much more
difficulties. Due to the problems of complex background,
blurry images, partial occlusion, etc., video object detection
is still a challenging work. How to keep the temporal and
spatial consistency of the target during the detection process
and ensure the target detected in the intermediate frame are
the main difficulties of video object detection.
The human visual system can find the object in current
frame with the features of previous frame. For example, if an
object is detected in previous frame but not in current frame,
we can recover the missing object by previous frame’s
features. Therefore, we can use pervious frame’s features to
enhance current frame’s feature map. Since video sequence
images have more temporal information than still images, the
key to improve the accuracy of video object detection is
making full use of the temporal information.
An intuitive idea to effectively utilize the temporal
information is to combine multiple frames on the temporal
channel and extract features for detection by 3D
convolutions [1] in spatiotemporal dimension. Compared
with 2D CNN, 3D CNN can extract much more temporal
information in video frames. The 3D filters can be extended
from pre-trained 2D filters [2] [3]. Another approach is to
combine the object tracking algorithm into the object
detection framework to improve the accuracy, such as the T-
CNN [4] [5]. T-CNN can improve temporal consistency by
propagate detection results to neighbor frames to reduce
sudden changes of detection results. In addition, optical flow
information, such as the target motion information is also
used to extract the temporal relationship between frames [6].
Optical flow can show target motion information clearly to
help CNN extract temporal information, but also needs huge
177
 of calculation. In order to improve the speed of video target
detection, Google has proposed a method combining fast and
slow network for video detection [7]. However, some of
these approaches cannot make effective use of temporal
information, some of these approaches have huge calculation,
leading to slow speed and low precision.
When we try to find object in an image, we will focus on
regions which have the object and pay more attention on it.
Meanwhile, we will reduce attention on the other regions.
Attention mechanism is inspired by this. Attention
mechanism can distinguish the importance of each region
and enhance their features. In general, attention mechanism
is mainly divided into two categories: soft attention and hard
attention. The hard attention will only focus on important
areas and dislodge unimportant parts of this image. This way
can find main feature rapidly, but will loss some features.
The soft attention will use weights which learned by
network to weight the image and strengthen significant
regions without loss any features. So soft attention
mechanism is more suitable for deep learning.
Self-attention is a soft attention mechanism relating
different positions of a single sequence in order to compute a
representation of the sequence. In [17], self-attention is
mainly used for machine translation, but it can also be
applied in image and video problems in computer vision with
some improvements.
Inspired by the non-local mean algorithm and the self-
attention mechanism, Wang et al. [15] proposed the concept
of non-local neural networks, using non-local blocks to find
sample layers and convolutional layers to expand the
receptive field of the convolution kernel, but multi-layer
convolution will cause a lot of waste of computing power.
In the traditional image processing algorithm, Buades et
al. [14] proposed a non-local mean filtering algorithm, which
uses the global image information to improve the denoising
effect. This algorithm takes advantage of the characteristics
of multiple image blocks with similar structure in natural
images to find the center points of the most similar image
blocks for weighted averaging. Compared with the mean
value filtering algorithm that uses the local information of
the image, it can remove the image noise more effectively
and retain the edge features of the image better.
Inspired by the non-local mean filtering algorithm, Wang
et al. proposed the concept of non-local neural networks. The
core of this algorithm lies in non-local blocks. The non-local
blocks directly calculate the similarity between any two
points of the image to get the long range dependency, and
obtains a mask to enhance the feature map. Compared with
multi-layer convolution, the global image information can be
effectively used without losing image feature information,
and the calculation amount is smaller. Meanwhile, these
points can come from different frames, so the non-local
blocks can also extract the temporal contextual information
of the video sequence images to enhance the image features
and improve the video object detection accuracy.
1
 yi =
C( x)
¦ f (x , x )g(x ) 
i j j
long-range, non-local dependency in frames. The non-local
blocks obtain global information by directly calculating the
similarity between any two locations. In the video, these two
locations may be spatial or temporal, so the relationship can
be spatial or temporal. By adding non-local blocks, the
convolutional neural network can effectively use the
spatiotemporal contextual information of the video, greatly
improving the detection accuracy.
In this paper, we propose a video object detection method
based on non-local prior. After trying several basic detection
networks, we adopt the YOLO model as the basic detection
model. Thus we use the same loss function and prediction
method with YOLOv2. We design Spatiotemporal non-local
models to capture the long-range dependencies in time and
space. Section Ċ introduces the non-local models. Section
ċ shows the details of the proposed method. In section Č,
we will introduce our datasets and compare our framework
with classical or state-of-art models on them. Experiments in
high-speed railway Overhead Contact System (OCS) driving
recorder dataset and OTB50 prove our framework achieves
good performance in various complex scenarios.
II. NON-LOCAL MODULE
Due to the limitations of the convolution kernel, the
receptive field is only a small rectangular region calculated
by kernel. This architecture forced the network to pay
attention to the local image information but ignored the
global information. For this problem, the traditional
convolutional neural network uses a combination of down-
178
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 04:55:31 UTC from IEEE Xplore. Restrictions apply.

j
Equation (1) describes the calculation process of the non-
local model. Where represents the value at position of
the feature map and represents the value of all positions of
the feature map. Function is mapping function learned by
network. By equation (1) we can calculate all positions’ non-
local dependencies value to obtain the non-local mask of
the feature map.
As is summarized in equation (2), each position in the
image has been calculated by similarity metric function to
find the long range non-local dependency, while in the
traditional convolution operation, it can only cover the
adjacent points of the position. The fully connected layer can
also find the long range non-local dependency by calculating
all positions, but compared with the fully connected layer,
there are the following differences:
(1) In the non-local blocks, the similarity is obtained by
the designed formula and the trained weights. But the fully
connected layer only uses learned weights. Compared with
the fully connected layer, the non-local blocks can measure
the similarity between image positions better.
(2) The non-local blocks support variable sizes of the
input and maintains the same size of the output, so it can be
used in any position of the network. While the fully
connected layer can only accept input of fixed size and
generate output of fixed size, so it mainly be used as the last
layer of the network. Because the input is stretched into one
column, the location information of feature maps will lost in
the fully connected layer. Compared with the fully connected
 layer, non-local blocks are more flexible to use and can frame only is only to provide temporal contextual
maintain the position information of feature maps. information for current frame prediction. The purpose of this
(3) Non-local blocks can be used anywhere in the design is to make full use of the spatial contextual
network and are very flexible to use, while fully connected information of the video sequence image, while effectively
layers are usually used at the last layer of the network. use the temporal contextual information of the image.
Compared with the fully connected layer, non-local modules
can build a network with a richer hierarchical structure.
Due these advantages, we use non-local blocks to
enhance the features of the video frames. The non-local
blocks mainly contains the following calculation.

T ( xiT )I ( x j )
 f ( xi , x j ) e  

Equation (2) describes the similarity metric function ,

referring to the non-local mean algorithm and the bilateral
filtering algorithm, using the Gaussian function as the metric
function, and further adding two convolutional layers to
extract feature map information more effectively.

 C ( x) ¦ f (x , x ) 
i j

j

Equation (3) describes the normalization function ( ). Fig 1. the structure of video object detection method based on non-local
priors
 Zi h( yi )  xi  
Equation (4) describes the method to get the final result
of the non-local blocks.
The non-local mask obtained by equation (1) will be
calculated by the mapping function ℎ and then added to the
original feature map using the residual structure method to
obtain the final result of the non-local blocks. It will enhance
the feature map of current stage to improve the accuracy of
detection. Finally, we get as the result of the non-local
blocks.
III. VIDEO OBJECT DETECTION BASED ON NON-LOCAL PRIOR
We use YOLOv2 as the basic network to extract video
sequence image features. Due to the problems of complex
background, blurry images, and partial occlusion in video
sequence images, we propose non-local blocks for extracting
spatiotemporal contextual information to improve the feature
extraction capability of the method. Fig. 1 illustrates the
structure of video object detection method based on non-
local priors.
In video sequence images, due to temporal continuity
between frames, the front and back frames of the video are
similar in space, so how to use the features of the previous
frame to enhance the features of the next frame is the key
point of video object detection. In this method, we use 2
continuous frames of the video sequence image for detection.
The previous frame only uses spatial non-local blocks to
provide temporal, and the current frame uses both the spatial
and temporal non-local blocks. The result of previous frame
and current frame in spatial non-local blocks will be used as
the input of the temporal non-local blocks. The previous
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 04:55:31 UTC from IEEE Xplore. Restrictions apply.

In the spatial non-local block, the input is the feature map
of the previous frame at a specific layer, and the output is the
feature map enhanced by the spatial non-local mask, which
is used as the input of the next layer of the previous frame
network. In the temporal non-local block, the input is the
respective feature map of the previous frame and the next
frame at a specific layer, and the output is the feature map
enhanced by the non-local mask of time, which is used as the
input of the next layer of the next frame network. Fig. 2 and
Fig. 3 illustrates the calculation process of spatial non-local
block and temporal non-local block.
As shown in Fig. 2 and Fig.3, the non-local model of our
method includes spatial non-local block and temporal non-
local module, contains four convolution layers and an active
layer. To make the non-local blocks more flexible, the input
feature map and output feature map are the same size. The
non-local model will be used repeatedly in our framework.
Fig. 2 illustrates the calculation process of spatial non-
local block to obtain the result feature map. Where
represents the feature map of the current frame and
represents the feature map of the current frame which is
enhanced by the spatial non-local module. After introducing
the non-local modules in the spatial dimension, the feature
map through the three convolutional layers, , and ,which
have 1×1×1 convolution kernel learned by network. The
channels of resulted feature map are half of , and will be
the same as by the last convolutional layer. This follows
the bottleneck design of and reduces the computation of a
block by about a half. The resulted feature map of layer T
and layer will be calculated by matrix multiplication and
normalized by SoftMax function, then calculated with the
resulted feature map of layer by matrix multiplication.
179
 After that the resulted feature map is fed through a
convolutional layer which has 1×1×1 convolution kernel
learned by network and the spatial non-local mask of current
frame is obtained. Finally, we combine the mask and the
input feature map of the spatial non-local block by
element-wise adding to obtain the resulted feature map
which enhanced by the spatial non-local block. The resulted
feature map will be used in next layer of the network.
Fig. 2. spatial non-local block
Fig. 3. temporal non-local block
180
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 04:55:31 UTC from IEEE Xplore. Restrictions apply.
Fig. 3 illustrates the calculation process of temporal non-
local block. Where represents the feature map of the
current frame, represents the feature map of the
previous frame, and represents the feature map of the
current frame which is enhanced by the spatial non-local
module. The calculation process of the temporal non-local
block is similar to the spatial non-local block, but the input
of the layer and layer is changed from to , to
measure the similarity between the current frame and the
previous frame.
We also do experiments in FasterRcnn to verify the
validity of our model. FasterRcnn use Resnet-50 as feature
extraction network, so we add non-local blocks after each
residual blocks. The non-local blocks’ architecture are the
same as the structure described above in Yolov2.
IV. EXPERIMENTS
A. Hardware environment
We trained our model on GTX1080ti (11GB). The other
configuration of hardware environment are shown below.
RAM: 32GB
CPU: Intel(R) Core(TM) i7-8700 (3.20GHz)
GPU: GTX1080ti (11GB)
Operating System: ubuntu 18.04
B. Dataset
We conduct experiments on two datasets, Overhead
Contact System driving recorder dataset and OTB50 dataset.
The OCS driving recorder dataset used in this article was
captured by high definition driving recorder device which is
installed in the cab of the train. The frame rate ranged from
24 frames per second to 60 frames per second, and the size
of the captured images ranged from 1920*1080 to
2880*2160. We selected 156 video clips containing foreign
objects of the bird's nest. The dataset contains 2597 labeled
images. These OCS images are captured from different train
lines and various imaging conditions, such as motion blur
and partial occlusion.
The category of the objects is the birdnest, so we
randomly selected 80% of video clips as the training set, and
the remaining 20% are used as the test set.
The OTB50 is a video dataset proposed by Yi Wu[16] et
al. This dataset contains 49 video clips. Among them, 48
video clips contained 1 object, and 1 video clip contained 2
objects, for a total of 49 categories. We set every object
belongs to one category, and set the clips’ name as the
category. A total of 26499 images were contained in OTB50
and were all labeled. Fig. 4 shows part of the first frame of
video where the target object is initialized with a bounding
box.
Because our model needs continuous frames to extract
the temporal relationship between frames, we selected the
first 80% of video clips’ frames as the training set (21181
images), and the remaining 20% are used as the test set
(5318 images).
 Fig. 6. blurry video image captured by driving recorder.

TABLE I. THE MAP ON OTB50 OF DIFFERENT NETWORK

Network Resolution dataset mAP

Fig. 4. Part of the first frame of videos in OTB40 dynamic

FasterRcnn OTB50 76.83
size
C. Performance Evaluation
Our work used batch-GD with a mini-batch size of 64.
We use a weight decay of 0.0005 with a momentum of 0.9 Yolov2 544*544 OTB50 83.43
and set the initial learning rate to 0.0001.
Following protocols widely adopted in object detection, Ours (based
dynamic
we evaluate our method on the test set and use mean average on
size
OTB50 79.70
precision (mAP) as the evaluation metric. FasterRcnn)
During the training and testing process, 2 frames in the
Ours (based
video are read in video sequence each time. The previous 544*544 OTB50 84.31
on Yolov2)
frame only uses spatial non-local blocks, and the current
frame uses both the spatial and temporal non-local blocks, As is shown in Table.1, we conduct experiments on
making full use of the spatiotemporal contextual information OTB50 with our method and classical or state-of-the-art
of video sequence images to improve the detection accuracy. models. In OTB50, the mAP achieved by our network
Fig. 5 shows the video image captured in poor lighting improves to 84.31% while the baseline Yolov2 achieves
conditions, and the abnormal object is difficult to detect. Fig. 83.43%, and the mAP achieved by our network improves to
6 shows the blurry video image captured by driving recorder. 79.70% while the baseline FasterRcnn achieves 76.83%.
It proves that our model can detect the target in complex Experiments shows that Darknet-19 performs better than
background. Resnet-50 in video images, and our models are proved to be
effective.
After trying several basic detection networks, we choose
Darknet-19 as our feature extraction network finally.
Therefore, we mainly compare our approach with the
original Yolov2.

TABLE II. THE MAP ON OUR DATASET OF DIFFERENT NETWORK

Network Resolution dataset mAP

OCS driving
Yolov2 544*544 recorder 54.78
dataset

OCS driving
Ours (based
544*544 recorder 58.31
on Yolov2)
dataset
As is shown in Table.2, in OCS driving recorder dataset,
Fig. 5. video image captured in poor lighting conditions combing Spatiotemporal non-local model with the prediction
results of original image improves to 58.31% measure by
mAP while the baseline Yolov2 achieves 54.78%.
Experiments show that the method designed in this paper can

181

Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 04:55:31 UTC from IEEE Xplore. Restrictions apply.
 maintain good detection accuracy in various complex OCS
video sequence images.
CONCLUSION
In this work, we propose a non-local model which can
effectively use the Spatiotemporal contextual information of
video sequence images to improve the detection accuracy.
The non-local model proposed in this paper does not require
additional supervision information or modification of the
loss function, and can be flexibly added to any model based
on convolutional neural network. It can capture the long-
range non-local dependency between frames in spatial and
temporal to enhance the features of current stage to
effectively improve the accuracy of detention.
The result of the experiment shows that the video object
detection method based on non-local prior of spatiotemporal
context designed in this paper has higher detection accuracy
than Yolov2, indicating that this method can better deal with
various complex video sequence images and has higher
application value. In the future, we will conduct the research
on designing an end-to-end framework to complete pillar
number plate detection and recognition at the same time.
REFERENCES
[1] Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani,
Manohar Paluri: Learning Spatiotemporal Features with 3D
Convolutional Networks. IEEE International Conference on
Computer Vision (ICCV). pp.4489-4497 (2015)
[2] João Carreira, Andrew Zisserman, Quo Vadis: Action Recognition? A
New Model and the Kinetics Dataset. Computer Vision and Pattern
Recognition (CVPR). pp.4724-4733 (2017)
[3] Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes:
Spatiotemporal Residual Networks for Video Action Recognition.
Neural Information Processing Systems (NIPS). pp.3468-3476 (2016)
[4] Kai Kang, Wanli Ouyang, Hongsheng Li, Xiaogang Wang: Object
Detection from Video Tubelets with Convolutional Neural Networks.
Computer Vision and Pattern Recognition (CVPR). pp.817-825 (2016)
[5] Kai Kang, Hongsheng Li, Junjie Yan, Xingyu Zeng, Bin Yang, Tong
Xiao, et al.: T-CNN: Tubelets With Convolutional Neural Networks
182
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 04:55:31 UTC from IEEE Xplore. Restrictions apply.
for Object Detection From Videos. IEEE Transactions on Circuits &
Systems for Video Technology (2018)
[6] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, Yichen Wei: Flow-
Guided Feature Aggregation for Video Object Detection. IEEE
International Conference on Computer Vision (ICCV). pp.408-417
(2017)
[7] Mason Liu, Menglong Zhu, Marie White, Yinxiao Li, Dmitry
Kalenichenko: Looking Fast and Slow: Memory-Guided Mobile
Video Object Detection. arXiv preprint arXiv: 1903.10172 (2019)
[8] Ross B. Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik: Rich
Feature Hierarchies for Accurate Object Detection and Semantic
Segmentation. Computer Vision and Pattern Recognition (CVPR).
pp.580-587 (2014)
[9] Ross B. Girshick: Fast R-CNN. IEEE International Conference on
Computer Vision (ICCV). pp.1440-1448 (2015)
[10] Shaoqing Ren, Kaiming He, Ross B. Girshick, Jian Sun: Faster R-
CNN: Towards Real-Time Object Detection with Region Proposal
Networks. Neural Information Processing Systems (NIPS). pp.91-99
(2015)
[11] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, Ali
Farhadi: You Only Look Once: Unified, Real-Time Object Detection.
Computer Vision and Pattern Recognition (CVPR). pp.779-788
(2016)
[12] Joseph Redmon, Ali Farhadi: YOLO9000: Better, Faster, Stronger.
Computer Vision and Pattern Recognition (CVPR). pp.6517-6525
(2017)
[13] Joseph Redmon, Ali Farhadi: YOLOv3: An Incremental
Improvement. arXiv preprint arXiv: 1804.02767 (2018)
[14] Antoni Buades, Bartomeu Coll, Jean-Michel Morel: A Review of
Image Denoising Algorithms, with a New One. Multiscale Modeling
and Simulation. 4(2). pp.490-530 (2005)
[15] Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, Kaiming He:
Non-Local Neural Networks. Computer Vision and Pattern
Recognition (CVPR). pp.7794-7803 (2018)
[16] Yi Wu, Jongwoo Lim, Ming-Hsuan Yang: Online Object Tracking: A
Benchmark. Computer Vision and Pattern Recognition (CVPR)濁
pp.2411-2418 (2013)
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion
Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin: Attention is
All you Need. Neural Information Processing Systems (NIPS).
pp.5998-6008 (2017)