Journal of Ambient Intelligence and Humanized Computing

https://doi.org/10.1007/s12652-020-01679-8

ORIGINAL RESEARCH

Fast key‑frame image retrieval of intelligent city security video based

on deep feature coding in high concurrent network environment
Chuhong Li1 · Bo Zhou1

Received: 27 June 2019 / Accepted: 3 January 2020

© Springer-Verlag GmbH Germany, part of Springer Nature 2020

Abstract
Aiming at the problems faced by the connection management module in single packet processing in intelligent city security
video retrieval, we firstly propose the traffic locality quantization index based on the traffic characteristics of the backbone
link to quantitatively analyze the traffic locality characteristics in the backbone link. And then, a key-frame abstraction and
retrieval of videos based on deep learning is proposed to improve the efficiency and accuracy of video retrieval, where an
adaptive key-frame selection algorithm is designed and the existing convolutional neural network framework is used to extract
the features of key-frames, and unsupervised, semi-supervised and supervised retraining models are designed to improve the
effectiveness of the feature extraction of the convolutional neural network and the accuracy of the video retrieval. Experi-
mental results based on the public video datasets show that our proposed key-frame image retrieval model realizes a good
precision for key-frame representation, and achieves high accuracy and efficiency for video retrieval.

Keywords  Intelligent city · Security video · Deep feature · Deep learning · Key-frame selection · Retraining model

1 Introduction Literature (Wu et al. 2018) puts forward three controlling

With the popularity of video information on network, a
large number of video information is generated every day
and the traditional information retrieval technology based
on key words cannot keep up this development (Jakub et al.
2018). As a result, content-based video retrieval has become
a research “hotspot” in recent years. Currently, most search
engines generally use the technology of cluster load bal-
ance, which makes the entrance of system easy to become
network’s bottleneck while high concurrent request of video
search happens on network (Song et al. 2018). For the goal
of improving network performance of video retrieval plat-
form, some scholars focus on the strategy of data concurrent
access and the strategy of data cache, which are the key
modules affecting service performance seriously (Chen et al.
2017; Song et al. 2017).
* Bo Zhou
zhoubo028@scu.edu.cn
Chuhong Li
2016151476001@stu.scu.edu.cn
1
College of Architecture and Environment,
Sichuan University, Chengdu 610000, Sichuan, China
strategies of data concurrent access: the strategy of service
quality as high priority; the strategy of concurrency number as
high priority; and the adaptive strategy of balance. For the first
strategy, the system allocates all the resources for the current
task, which ensures the shortest response time for single task
while concurrency is still low. For the second strategy, the sys-
tem executes multi-task of retrieval simultaneously in the way
of threaded concurrency. Although the executive performance
of single task cannot be improved in this strategy, the whole
operating efficiency can be increased in the condition of high
concurrency. With the adaptive strategy of balance, system will
switch between the first strategy and the second strategy above
smoothly according to the number of concurrent accesses and
the special needs of video retrieval, aiming at improve the adap-
tive capacity of system farthest (Okabe et al. 2018).
Although the pressure of video feature retrieval will
be alleviated and the whole system performance will be
improved in high concurrent network environment, it still
consumes a lot of resources and bandwidth (Feng et al.
2019). How to process and analyze video quickly and
effectively has become an important problem to be solved
urgently. Video key-frame extraction is an effective method
for fast browsing and retrieval of massive video data. It is
also the basis of video summarization (Dong et al. 2018).

13
Vol.:(0123456789)

Commonly used video key-frame extraction methods
include shot boundary method, image feature based method,
clustering based method and motion information analysis
method (Li 2017; Wang et al. 2014). These four methods
do not make enough use of visual saliency, which results
in the low degree of representation and redundancy of the
extracted key-frames, and low consistency with the key-
frames extracted by users. The purpose of some researches
is to extract key-frames which are more consistent with
human vision system and highly able to generalize the orig-
inal video content. A video key-frame extraction method
based on video saliency and two-step hierarchical clustering
algorithm is proposed in literature (Cappallo et al. 2018).
The specific contents are as follows: The saliency detection
algorithm is applied to the original video sequence. Com-
bined with the underlying features and motion information
of the salient region, a fusion feature vector is constructed.
Then the redundant information is removed by calculating
the similarity between the feature vectors (Lin et al. 2017).
Secondly, a two hierarchical clustering algorithm based on
mutual information is proposed. One time hierarchical clus-
tering adaptively determines the clustering threshold, which
is real time. Two-time clustering artificial sets clustering
threshold to meet the needs of different users (Roy et al.
2017). Finally, the frame with the maximum mutual infor-
mation between the other frames in each cluster is selected
as the key-frame. Compared with the current mainstream
video key-frame extraction algorithm, the effectiveness and
applicability of the proposed algorithm is verified, and the
selected key-frames are more suitable for human vision
system.
A key-frame extraction method based on PCA and tem-
poral K-means clustering is proposed in literature (Yoo et al.
2017). The main research is how to use fast convergent clus-
tering algorithm to cluster similar features on the basis of
ideal feature extraction. The specific content of the work is
as follows: Firstly, Principal component analysis (PCA) is
used to extract the principal component of the fusion feature
vector. The number of principal components of a video in
time coordinates is determined by cumulative contribution
rate, and considers the number of principal components as
the number of key-frames (Deng and Yu 2019). Secondly,
the similarity between video frames is calculated, and the
distance time distribution curve is generated, and the initial
boundary of clustering is determined according to the peak
value of the curve. Thirdly, the initial cluster is optimized
by the time series K-means clustering algorithm. Finally, the
nearest frame of the distance cluster center is selected from
each cluster as the key-frame.
The experimental results show that the key-frame gener-
ated by literature (Rameshnath and Bora 2019) has a higher
matching degree to the user’s summary. And the content
of the original video is highly generalized, key-frames are
13
C. Li, B. Zhou
more suitable for human vision system. However, the num-
ber of key-frames is not determined flexibly according to
the type and length of the video, which does not provide an
effective basis for the user to extract the appropriate length
of key-frames. With the rapid development of information
technology, varieties of video technologies are widely used
in the Internet and wireless sensor networks. In order to
solve the problems of traditional video coding, such as high
computational complexity and poor error resilience capabil-
ity, distributed compressive video sensing has become a hot
research topic both at home and abroad in recent years and is
the most competitive video application solution (Chenchen
et al. 2018). The technical problems to be solved urgently are
that the existing video coding and reconstruction methods
still need to be further optimized, the reconstruction rate is
increased, and the complexity of the algorithm is reduced.
When the video frame is reconstructed, a multi-hypothe-
sis bidirectional best match strategy is proposed in literature
(Haijun et al. 2018). The idea of motion alignment estima-
tion is introduced, and the best matching patch is selected
from the reference frame by using the patch structural simi-
larity of the measured domain (Ying and Daiyin 2018). The
optimal video generation matrix is extracted pixel by pixel
in the search window centered on the best matching block,
and the residual compensation is applied to reconstruct the
non-key-frame. Simulation results show that at the same
sampling rate, the proposed algorithm can improve the qual-
ity of reconstructed video frames, and the reconstruction
time does not increase significantly, and the subjective visual
effect is better.
Nevertheless, the existing prediction reconstruction algo-
rithm is not clear enough to reconstruct the video in the
vigorous scene of the sport, so the best matching interpola-
tion reconstruction method in distributed compressive video
sensing is proposed in literature (Liu et al. 2003). The best
matching interpolation dictionary is formed by the inter-
polation method in the searching window centered on the
best matching patch, and the interpolation method makes
the residual signal sparser. Simulation results show that this
method can obtain higher reconstruction quality.
In conclusion, both bidirectional best matching strategy
and the best matching interpolation method are applied in
the distributed compressive video sensing. The key-frame
is reconstructed by the best matching interpolation method,
and the nonkey-frame is reconstructed by the bidirectional
best matching method. Simulation results show that the
proposed algorithm significantly improves the quality of
reconstruction, and the reconstruction time does not increase
significantly.
Aiming at the problems faced by the connection manage-
ment module in single packet processing, we firstly propose
the traffic locality quantization index based on the traffic
characteristics of the backbone link to quantitatively analyze
Fast key-frame image retrieval of intelligent city security video based on deep feature coding…
the traffic locality characteristics in the backbone link. Sec-
ondly, the auxiliary space of constant overhead based on
the traditional hash structure is added, and a fast flow table
lookup method (FFTL) is designed to reduce the flow-
table access overhead. And then, a key-frame abstraction
and retrieval of videos based on deep learning is proposed
to improve the efficiency and accuracy of video retrieval,
where an adaptive key-frame selection algorithm is designed
and the existing convolutional neural network framework
is used to extract the features of key-frames, and unsuper-
vised, semi-supervised and supervised retraining models are
designed to improve the effectiveness of the feature extrac-
tion of the convolutional neural network and the accuracy of
the video retrieval. Experimental results based on the pub-
lic video datasets show that our proposed key-frame image
retrieval model realizes a good precision for key-frame rep-
resentation, and achieves high accuracy and efficiency for
video retrieval.
2 Fast flow table lookup for high
concurrency network
The existing flow table lookup operations are classified into
three types: hash table, Bloom filter and content addressable
memory (Burtnyk and Wein 1976, 1971). For flow tables
with hash table structure, one hash table lookup includes two
steps: hash value calculation and conflict chain comparison.
The worst case performance of using the linked list method
to deal with hash conflict is very poor. All N keywords are
inserted into the same position, where a linked list with the
length of N is generated. Therefore, the worst lookup length
is O(N).
Many research works have been done to try to balance the
length of the collision chain on each slot as much as possible
to ensure that the average lookup length is as close as pos-
sible to the best case of O(1 + a), where a is the load factor.
A good hash algorithm is needed to achieve this function.
The literature (Liu and Fan 2005) pointed out that although
the complex cryptographic hashing method, such as MD5,
SHA-1, can be used to achieve all conflict chain length dis-
tribution equilibrium in the hash table, a good hash function
usually consumes a large number of CPU resource. Even
the hardware implementation of the hash unit requires 64
clock cycles.
The multi-hash effect is better than the single-hash
mentioned above. The Bloom filter is a multiple hash
that is often combined with high-speed hardware such
as TCAM, FPGA and so on, to provide efficient content
lookup. The literature (Radhakrishnan 2001) pointed out
that the Bloom filter is commonly used to represent a set
to provide member query operations, and the query result
has an error probability. And the query result of the Bloom
filter has only two states, and does not satisfy the connec-
tion state maintenance requirement in the traffic processing
system. The literature (Stippick 2004) proposed a modified
Fingerprint Compressed Filter(FCF). Each Hash slot of
FCF contains d fixed-size sub-tables and d hash functions
which are mutually independent. A connection will cal-
culate the d-times hash value, and finally insert into the
shortest sub-table, but this makes it necessary to find d
conflict chains for each search in the backbone with dense
number of packets. The network will bring a large search
overhead, and it uses a single bit for timeout processing,
which is a slot-level processing method. Although the FCF
will compress the FID where it can get the fast access of
the SRAM, but its scalability is also limited by the capac-
ity of the SRAM.
In the aspect of optimizing the search operation by means
of network locality, the literature (Mademlis et al. 2018)
indicates that the backbone network has high concurrency
characteristics, and the number of concurrent connections
reaches two million. The newly emerging connection per
second is only about 20,000, accounting for 1% of the total
concurrency. The backbone network is slow to update and
has a certain degree of locality. Therefore, it uses FPGA
and SRAM to realize the high-speed Cache of the flow table
to speed up the access speed, which is limited by the cir-
cuit complexity of the FPGA and the capacity limitation
of the SRAM. Mainly for the first few bytes of data packet
processing, specifically for the protocol identification sys-
tem, the model can not solve the full packet load processing
problem such as DPI system. In addition, TCAM (Ternary
Content Addressable Memory) (Gong et al. 2013) is also
used to accelerate the search operation. However, TCAM
is too expensive and consumes a lot of power, and the scale
of the flow table is also limited by the storage capacity. In
the high concurrent network environment, a large number
of active connections will be affected by traffic fluctuation
and burst traffic and be forced to replace, leading to system
missed detection.
In order to improve flow table lookup speed, flow char-
acteristic of the video surveillance backbone links was
explored. It proves that the backbone traffic not only has
high concurrency and high arrival rate, but also has good
network locality characteristic (Huang et al. 2018) in an
appropriate cached window. Based on these characteristic
and the principle of locality, a fast flow table lookup method
was implemented by using a naive Hash table structure with
constant increase of auxiliary space. The theoretical analy-
sis and experiments on real-life data traces show that the
proposed method can reduce the length and the time of flow
table lookup by 20.2% and 17.1% compared with the exist-
ing method, respectively. Therefore, it will be adopted to
achieve the fast key-frame image retrieval of security videos
in intelligent city.
13
 C. Li, B. Zhou

3 Video retrieval model based
on convolutional neural network
After extracting the key-frames of each shot, the relevant
video sequences can be retrieved efficiently and accurately
through content-based retrieval of video frames (Aote and
Potnurwar 2018; Ponni et al. 2018). At present, the deep
learning architecture used in image and video retrieval sys-
tem adopts a lot of parameters, which makes the structure
of deep learning complex, and difficult to implement. It
takes too long to calculate and train the network parameters
(Naveed et al. 2018).
In this paper, the external structure of deep neural net-
work is improved. An improved model is designed to retrain
parameters so as to make full use of some prior information
to generate low-dimensional image and video representation.
They can reduce the computation time and memory require-
ments of retrieval.
In this paper, BVLC Reference CaffeNet model is used
as a deep neural network model, which is the result that
AlexNet model is trained in ImageNet Large Scale Visual
Recognition Challenge, ILSVRC. The model consists of
eight trained neural network layers, the first five are convolu-
tion layers, and the last three are full connection layers. And
the first, second and fifth convolution layers are maximum
pooling layers. ReLU (Rectified Linear Unit) non-linear acti-
vation function is used for five convolution layers and the
first two full connection layers. The output of the third full
connection layer is a distributed result with 1000 ImageNet
classes.
Softmax loss is used for model training. The overall archi-
tecture of CaffeNet is as shown in Fig. 1.
In this paper, CaffeNet has been modified for the follow-
ing reasons:
1. Because the activation function is a spatially arranged
structure, the convolution layer retains spatial informa-
tion, while the full connection layers connect with all
input neurons, so the full connection layers do not con-
tain spatial information.
2. The model parameters required for the full connection
layer are much more than those for the convolution layer.
3. The full connection layer needs to predict the size of the
input image, while the convolution layer does not need
to predict this information.
Therefore, the full connection layer is not responsible
for feature extraction, but the convolution layer is respon-
sible for feature extraction.
The modified scheme in this paper uses the fifth con-
volution layer, expressed as CONV5, or the fourth con-
volution layer, expressed as CONV4. The dimension of
CONV5 is 13 × 13 × 384. Therefore, the MAC (Maximum
Activations of Convolutions) layer generates a 256-dimen-
sional or 384-dimensional feature for each video frame.
The flow chart of this retraining program is shown in Fig. 2.
3.1 Unsupervised retraining
Unsupervised retraining retrains the pre-trained convolu-
tional neural network model based on a given data set to
maximize the cosine similarity between each video key-
frame and its nearest “N-key-frame”.
{ }
Assuming that I = x{i , i = 1, … , N represents
} the set
of N key-frames, X = xi , i = 1, … , N represents the
corresponding features of N key-frames, 𝜇 represents the
nearest “N-key-frame” average vector of xi , and 𝜇i can be
defined as follows:
1∑ i
n
𝜇i = x (1)
n l=1 l
The new objective function of I can be determined by
solving the following optimization problem:
∑
N
xiT 𝜇i
max J = max
‖xi ‖‖𝜇i ‖ (2)
xi ∈X xi ∈X
i=1 ‖ ‖‖ ‖
Equation (2) is solved by using gradient descent method.
And the first order gradient of objective function J is:

Videos

Video feature Convolution layer 1 Convolution layer 2 Convolution layer 3

227×227×3 55×55×96 27×27×256 13×13×384

Fully Connected Layer8 Fully Connected Layer 7 Fully Connected Layer 6 Convolution layer 5 Convolution layer 4
1000 4096 4096 13×13×256 13×13×384

Fig. 1  Eight-layer framework of CaffeNet deep neural network

13
Fast key-frame image retrieval of intelligent city security video based on deep feature coding…

Fig. 2  The flow diagram of the Error back propagation

Loss
retraining program based on
improved deep learning
Videos Convolution layer 1 MAC1 Vectors

Convolution layer 2 MAC2 Vectors

Convolution layer 3 MAC3 Vectors

Retrain
Vectors with 384 objection
Convolution layer 4 MAC4
dimensional function

Vectors with 256

Convolution layer 5 MAC5
dimensional

Non-supervision

Semi-supervision
Class label

Supervision
Sematic information

𝜕J 𝜇i xiT 𝜇i tagged descriptor, where xi is the key-frame descriptor and yi

= − x
‖xi ‖‖𝜇i ‖ ‖x ‖3 ‖𝜇i ‖ i (3)
𝜕xi ‖ ‖‖ ‖ ‖ i ‖ ‖ ‖ is the tag corresponding to the key-frame. This scheme regu-
lates the convolutional neural layer of CNN mode (Zheng et al.
The updating rule for the vth iteration of each image is
2018). Its goal is to maximize the cosine similarity between
as follows:
xi and the latest “m-correlation descriptor” while minimiz-
⎛ ing the cosine similarity between xi and the latest “l-correla-
𝜇i xi(v)T 𝜇i (v) ⎞
xi(v+1) = xi(v) + 𝜂 ⎜ − x ⎟ (4) tion descriptor”. key-frames belonging to the same class are
⎜��x (v) �� i �
� 𝜇
� (v) �� i � i ⎟
�x � 𝜇 defined as related frames, and key-frames belonging to differ-
⎝ � i �� � � i �� � ⎠
ent classes are defined as
{ uncorrelated frames.
}
In order to better control the learning rate of the neural net- Assuming that I = Ii , i = 1, … , N represents N image
work, the updating rule is normalized and defined as follows: sets with relevant information, x = FL (I) represents the L-level
output result
{ of the pre-trained
} CNN model for the input image
⎛ ⎞ set I, X = xi , i = 1, … , N represents the corresponding fea-
� (v) �� i �⎜ 𝜇i xi(v)T 𝜇i (v) ⎟
{ }
xi(v+1) xi(v) + 𝜂 �xi ��𝜇 �⎜ tures of the image, and Ri = rk , k = 1, … , K i represents
� �� � �x(v) ��𝜇i � � (v) �3 � i � i ⎟
= − x
⎜ � i �� � �x � �𝜇 � ⎟ the set of relevant descriptors for the i-th key-frame. The new
⎝� � � i � ⎠
descriptor of key-frame set I is determined by solving the fol-
(5)
lowing two optimization problems:
Using the above features as the targets in the inter-
est layer, a regression task is established for the neural ∑
N
xiT 𝜇+i
network. The weight of CafeNet is initialized, and the
+
max J = max
‖x ‖‖𝜇i ‖ (6)
xi ∈X xi ∈X
i=1 ‖ i ‖‖ + ‖
backward propagation method is used to train the neu-
ral network based on the target data set. Euclidean loss
(Euclidean distance loss function) is used in the training ∑
N
xiT 𝜇−i
min J − = min
‖xi ‖‖𝜇i ‖ (7)
process of the regression task. xi ∈X xi ∈X
i=1 ‖ ‖‖ − ‖
where 𝜇−i is the average vector of the nearest “ l -descriptor”
3.2 Semi‑supervised retraining of xi , and 𝜇+i is the average vector of the nearest “ m-descrip-
tor” of xi.
Semi-supervised retraining improves the performance of Con- The normalized update rule of the v- th iteration is rewrit-
volutional Neural Network (CNN) descriptors by using infor- ten as follows:
mation derived from class labels. Given ( xi , yi ) to represent a

13
 C. Li, B. Zhou

⎛ ⎞ The normalized updating rule of the v-th iteration can

� �� � 𝜇+i xi(v)T 𝜇+i
xi(v+1) = xi(v) + 𝜍1 �xi(v) ��𝜇+i �⎜ − xi(v) ⎟ be defined as the following two equations:
� �� �⎜ �x(v) ��𝜇i � �x(v) ��𝜇i � ⎟
⎝� � � �
� i �� + � � i �� + � ⎠ ⎛ ⎞
(8) � (v) �� k �⎜ qk xi(v)T qk (v) ⎟
xi(v+1) xi(v) + a�xi ��q �⎜
� �� � �x(v) ��qk � � (v) �3 � k � i ⎟
= − x
⎛ 𝜇−i xi(v)T 𝜇−i ⎞ ⎜ � i �� � �x � �q � ⎟
� (v) �� i �⎜ ⎝� � � i � ⎠
xi(v+1) (v)
= xi −𝛽1 �xi ��𝜇− � − xi(v) ⎟
� �� �⎜ �x(v) ��𝜇i � �x(v) ��𝜇i � ⎟ (13)
⎝� � � �
� i �� − � � i �� − � ⎠
(9) ⎛ ⎞
� (v) �� k �⎜ xj(v)T qk
Fuse the normalized updating rules, that is, add Eq. (8) and
qk (v) ⎟
xj(v+1) xj(v) − a�xj ��q �⎜
� �� � �x(v) ��qk � � (v) �3 � k � j ⎟
= − x
Eq. (9) together: ⎜ � j �� � �x � �q � ⎟
⎝� � � j � ⎠
⎛ 𝜇+i xi(v)T 𝜇+i ⎞ (14)
� (v) �� i �⎜
xi(v+1) = xi(v) + 𝜍1 �xi ��𝜇+ � − xi(v) ⎟+ The above key-frame descriptor is used as the target of
� �� �⎜ �x(v) ��𝜇i � �x(v) ��𝜇i � ⎟
⎝� � � �
� i �� + � � i �� + � ⎠ the interest layer, and the deep neural network is retrained.
⎛ 𝜇−i xi(v)T 𝜇−i ⎞
� �� �
𝛽1 �xi(v) ��𝜇−i �⎜ − xi(v) ⎟
� �� �⎜ �x(v) ��𝜇i � �x(v) ��𝜇i � ⎟ 4 Experimental results and analysis
⎝� � � �
� i �� − � � i �� − � ⎠
(10)
4.1 Evaluation criteria
Using the above object descriptor, the neural network
can be retrained by using the key-frame related information In this paper, the performance of video retrieval algorithm
through backward propagation technology. is evaluated by precision, recall rate and average preci-
sion. The higher the recall rate and accuracy, the better the
3.3 Supervised retraining performance of the retrieval system (Naruse et al. 2014).
Recall refers to the ratio of the number of relevant video
The idea of supervised training comes from the theory of frames found by the user and the number of all clips in
relevant feedback. Supervised retraining considers feedback the video library related to the target clip during a query.
from different users, which consists of relevant key-frames for Precision is the ratio of the number of relevant key-frames
querying objective. The goal of this scheme is to modify the found by the user to the number of all frames found during
model parameters to maximize the cosine similarity between a query. Precision and recall rate are defined as follows:
the specified query and its related key-frames, and to mini-
mize the cosine similarity between the specified query and its Ncorrect
Recall = (15)
unrelated key-frames. { } Ncorrect + Nmiss
Assuming
{ that Q = Q} k , k = 1, … , K represents the query
set, I+k = Ii , i = 1, … , Z represents
{ the relevant
} key-frame Ncorrect
of the specified query, I−k = Ij , i = 1, … , O represents the Precision = (16)
Ncorrect + Nfalse
irrelevant key-frame of the specified query, x = FL (I) repre-
sents the L-level output of the pre-trained CNN model for the Where,Ncorrect , Nfalse and Nmiss are the number of key-
key-frame I, and q = FL (Q) represents the L-level output of frames for correct detection, error detection, and missed
the neural network for the query Q. detection, respectively. Precision is equal to the number of
By solving the following optimization problems, we can relevant retrieval images/the number of retrieval images,
obtain new object functions of relevant and irrelevant key- while Recall rate is equal to the number of relevant retrieval
frames for query, respectively: images/the number of relevant images (Zhu et al. 2013).
The average precision is defined as the average query pre-
∑
Z
xiT qk
maxk J + = maxk (11) cision of all query experiments, and the precision QP of i-th
‖xi ‖‖qk ‖ query experiment is defined as follows:
xi ∈X+ xi ∈X+ i=1 ‖ ‖‖ ‖

1 ∑ Ri i
N n
∑
0 xjT qk APi = t (17)
mink J − = mink Qi n=1 n n
‖ ‖‖ k ‖ (12)
xi ∈X− xi ∈X−
j=1 ‖xj ‖‖q ‖
‖ ‖ where Qi is the number of related images of i-th query; N
is the total number of search images; Rni is the number of n

13
Fast key-frame image retrieval of intelligent city security video based on deep feature coding…

optimal related retrieval images; tni is the indicator function.
If the n-th retrieval image is related to i-th query, tni = 1 or
tni = 0.
4.2 Experimental environment and parameter
setting
We evaluate the proposed approach on five public data-
sets: the UKbench, Corel-5 K, Holidays, UQ_VIDEO and
San Francisco Landmarks (SFL) (Liu et al. 2016). In the
UKbench and Holidays, relevant images are near-duplicates
or the same objects/scenes to the query, while the Corel-5 K
involves category-level relevant images without any near-
duplicate ones. SFL is a realistic large-scale dataset with
a variable number of relevant images for different queries.
We employ the performance measures from the original
papers of these datasets and demonstrate the query specific
fusion improves considerably for all these diverse datasets;
UQ_VIDEO is used as video data set of benchmark, which
is a real video set collected from YouTube aiming at similar
shot detection experiments. The data set contains 169, 952
videos, and 3, 305, 525 key-frames are extracted from the
video set (Song et al. 2013).
The algorithm is implemented based on Caffe Deep
Learning framework, and features are extracted from UQ_
VIDEO dataset by using CONV5 and CONV4. The retrain-
ing module is used to replace the ReLU layer of CaffeNet
framework.
4.3 The experimental result of deep neural network
Firstly, the learning effect of the proposed deep neural net-
work is evaluated. The improved retraining model in this
paper is tested on UQ_VIDEO dataset, as shown in Table 1.
In Table  1, FU denotes unsupervised retraining, RRI
denotes semi-supervised retraining, and FRRI denotes fully
supervised retraining. It can be seen from the Table 1 that
the performance of CONV5 is better than that of CONV4.
In addition, Scheme RRI and FU significantly improve the
performance of the original CaffeNet framework. The per-
formance of RRI is better than retraining scheme FU. The
reason is that the model training process of scheme FU does
not utilize user information and relevant feedback informa-
tion. Therefore, retraining scheme “FRRI + CONV5” is
adopted as the feature representation scheme of video in the
following experiments.
4.4 Experimental results of video retrieval
In order to comprehensively evaluate the video retrieval
effect of our proposed model, the existing three retrieval
models with better performance are selected and compared
with our model. The comparison models are as follows: ITQ,
QBH and KMH, where ITQ and KMH are both HASH-
based retrieval schemes, while QBH improves the HASH
effectively and achieves obvious results. The UQ_VIDEO
video dataset is searched by 24 query statements in literature
(Atmojo et al. 2015), and they are shown in Fig. 3.
It can be seen from the graph that the algorithm achieves
the optimal result. The recall rate of this algorithm is better
than that of KMH. When the precision of this algorithm is
0.1, the recall rate is greater than 0.9. When the precision
is less than 0.4, the recall rate performance of ITQ is poor.
When the precision is greater than 0.3, the recall rate of
QBH is higher than that of the other two schemes, but still
lower than that of this scheme.
Figure 4 is the relationship between key-frame and preci-
sion for four retrieval algorithms. It can be seen from Fig. 4
that the average absolute accuracy first increases and then
decreases as the value of the key-frame increases, while the
number of key-frames k keeps increasing, and the cover-
age and accuracy of key-frames also increase. When the
-frame value is greater than a certain value, the key-frame
redundancy rate increases, and the average absolute accuracy

Table 1  The representation results of three retraining algorithms

Feature representation Feature dimensions Average accuracy
of key-frames

CONV4 384 0.45891

FU + CONV4 384 0.61082
RRI + CONV4 384 0.81004
FRRI + CONV4 384 0.92074
CONV5 256 0.53612
FU + CONV5 256 0.68325
RRI + CONV5 256 0.88743
Fig. 3  The relationship between recall rate and precision for four
FRRI + CONV5 256 0.95278
retrieval algorithms

13

Fig. 4  The relationship between key-frame and precision for four
retrieval algorithms
decreases. When the average absolute accuracy takes the
maximum value, the accuracy of the key-frame is high, the
redundancy rate is low, and the effect is good. In summary,
for the content change is richer and longer, the suitable range
of the key-frame value is [10, 20], and the content range is
not obvious, the length is short, and the suitable range of the
key-frame value is [20,30].
4.5 Calculation time of our proposed algorithm
To improve the QoS of video retrieval, this paper designs a
kind of technology of data cache. In the method of retrieval
result’s data cache, by the construction of rapid lookup table
of video content feature data, the pressure of video feature
retrieval will be alleviated and the whole system performance
will be improved. While in the method of retrieval task’s data
cache, by the construction of retrieval task table, the repetitive
process because of the same tasks will be avoided, and then
the pressure to the system from the retrieval of hotspot video
will be decreased largely. This paper establishes testing envi-
ronments and finishes the simulative experiments, which veri-
fies the efficiency to improve the service performance of the
video retrieval platform resulting from the strategies above.
The experimental environment is: Intel Core i7 7700; the
main frequency is 3.6 GHz and the memory is 16 GB. In this
paper, the data sets of different scales (obtained by partition-
ing UQ_VIDEO data set) are experimented, and the retrieval
time of those several retrieval algorithms is counted and the
result is shown in Table 2.
As can be seen from the Table 2, both QBH and KMH are
retrieval algorithms based on HASH operation. The HASH
operation of these two algorithms requires more time, and the
processing time increases with the increase of data set size.
This algorithm is based on deep learning. It takes a certain
amount of time in the training phase, but it can be processed
offline in the training phase. In the online retrieval phase of
13
C. Li, B. Zhou
Table 2  The retrieving time for comparison retrieval algorithms
Video size QBH KMH ITQ Proposed
200 M 0.034 0.033 0.041 0.036
400 M 0.042 0.039 0.046 0.046
600 M 0.051 0.049 0.052 0.047
800 M 0.061 0.062 0.067 0.051
1000 M 0.079 0.078 0.081 0.061
this algorithm, the processing time of small-scale data sets is
close to that of QBH and KMH, but this algorithm has obvi-
ous advantages in the retrieval time of large-scale data sets.
Based on these characteristic and the principle of locality, a
fast flow table lookup method was implemented by using a
naive Hash table structure with constant increase of auxiliary
space. The theoretical analysis and experiments on real-life
data traces show that the proposed method can reduce the
length and the time of flow table lookup by 20.2% and 17.1%
compared with the existing method, respectively. Therefore,
our proposed algorithm can achieve the fast key-frame image
retrieval of security videos in intelligent city.
5 Conclusion
Aiming at the problems faced by the connection management
module in single packet processing for intelligent city security
video retrieval, we firstly propose the traffic locality quantiza-
tion index based on the traffic characteristics of the backbone
link to quantitatively analyze the traffic locality characteris-
tics in the backbone link. And then, a key-frame abstraction
and retrieval of videos based on deep learning is proposed to
improve the efficiency and accuracy of video retrieval, where
an adaptive key-frame selection algorithm is designed and the
existing convolutional neural network framework is used to
extract the features of key-frames, and unsupervised, semi-
supervised and supervised retraining models are designed to
improve the effect of the feature extraction of the convolu-
tional neural network and the accuracy of the video retrieval.
Experimental results in the public video datasets show that
our proposed key-frame image retrieval model realizes a good
precision for key-frame representation, and achieves an accu-
racy and high efficiency for video retrieval.
References
Aote SS, Potnurwar A (2018) An automatic video annotation frame-
work based on two level keyframe extraction mechanism [J]. Mul-
timed Tools Appl 24(12):78–92
Atmojo UD et al (2015) System-level approach to the design of ambi-
ent intelligence systems based on wireless sensor and actuator
networks. J Ambient Intell Humaniz Comput 6(2):153–169
Fast key-frame image retrieval of intelligent city security video based on deep feature coding…
Burtnyk N, Wein M (1971) Computer-generated key-frame
animation[J]. J SMPTE 80(3):149–153
Burtnyk N, Wein M (1976) Interactive skeleton techniques for enhanc-
ing motion dynamics in key-frame animation[J]. Commun ACM
19(10):564–569
Cappallo S, Svetlichnaya S, Garrigues P et al (2018) The new modality:
emoji challenges in prediction, anticipation, and retrieval[J]. IEEE
Trans Multimed 24(7):191–202
Chen W, Qing Z, Yeting Z et al (2017) A NoSQL-SQL hybrid organi-
zation and management approach for real-time geospatial data: a
case study of public security video surveillance[J]. ISPRS Int J
Geo-Inform 6(1):21–43
Chenchen J, Zhen D, Mingtao P et al (2018) Heterogeneous hash-
ing network for face retrieval across image and video domains[J].
IEEE Trans Multimed 24(23):1–12
Deng Y, Yu Y (2019) Self-feedback image retrieval algorithm based
on annular color moments[J]. EURASIP J Image Video Process
20(1):36–45
Dong J, Li X, Snoek CGM (2018) Predicting visual features from text
for image and video caption retrieval[J]. IEEE Trans Multimed
24(4):91–107
Feng Y, Zhou P, Xu J et al (2019) Video big data retrieval over media
cloud: a context-aware online learning approach[J]. IEEE Trans
Multimed 24(7):12–31
Gong Y, Lazebnik S, Gordo A et al (2013) Iterative quantization:
a procrustean approach to learning binary codes for large-
scale image retrieval[J]. IEEE Trans Pattern Anal Mach Intell
35(12):2916–2929
Haijun Z, Yuzhu J, Wang H et al (2018) Sitcom-star-based clothing
retrieval for video advertising: a deep learning framework[J]. Neu-
ral Comput Appl 44(23):208–231
Huang S, Mao C, Tao J et al (2018) A novel chinese sign language
recognition method based on keyframe-centered clips[J]. IEEE
Signal Process Lett 25(3):442–446
Jakub L, Werner B, Klaus S et al (2018) On influential trends in inter-
active video retrieval [J]. IEEE Trans Multimed 24(21):13–21
Li X (2017) Tag relevance fusion for social image retrieval[J]. Mul-
timed Syst 23(1):29–40
Lin J, Duan LY, Wang S et al (2017) HNIP: compact deep invariant
representations for video matching, localization, and retrieval[J].
IEEE Trans Multimed 19(9):1968–1983
Liu L, Fan G (2005) Combined key-frame extraction and object-based
video segmentation[J]. IEEE Trans Circuits Syst Video Technol
15(7):869–884
Liu T, Zhang HJ, Qi F (2003) A novel video key-frame-extraction algo-
rithm based on perceived motion energy model[J]. IEEE Trans
Circuits Syst Video Technol 13(10):1006–1013
Liu W, Wen Y, Yu Z et al (2016) Large-margin softmax loss for convo-
lutional neural networks[J]. Comput Sci 2:507–516
Mademlis I, Tefas A, Pitas I (2018) A salient dictionary learning frame-
work for activity video summarization via key-frame extraction[J].
Inf Sci 43(2):319–331
Naruse N, Tenmyo O, Tomita K et al (2014) 1-HKUST: object detec-
tion in ILS VRC 2014[J]. Comput Sci 42(6):837–845
Naveed E, Wook BS, Hammad M et al (2018) Multi-scale contrast and
relative motion-based key-frame extraction[J]. EURASIP J Image
Video Process 2018(1):40–48
Okabe M, Dobashi Y, Anjyo K (2018) Animating pictures of water
scenes using video retrieval[J]. Vis Comput 34(3):347–358
Ponni ASS, Ramakrishnan S (2018) Fibonacci Based key-frame Selec-
tion and Scrambling for Video Watermarking in DWT–SVD
Domain[J]. Wirel Personal Commun 69(25):28–35
Radhakrishnan R (2001) Video summarization using descriptors of
motion activity: a motion activity based approach to key-frame
extraction from video shots[J]. J Electron Imaging 10(4):90–99
Rameshnath S, Bora PK (2019) Perceptual video hashing based on
temporal wavelet transform and random projections with applica-
tion to indexing and retrieval of near-identical videos. Multimed
Tools Appl 78:18055–18075
Roy PP, Bhunia AK, Pal U (2017) Date-field retrieval in scene image
and video frames using text enhancement and shape coding[J].
Neurocomputing. 274:S0925231217306689
Song J, Yang Y, Huang Z et al (2013) Effective multiple feature hash-
ing for large-scale near-duplicate video retrieval[J]. IEEE Trans
Multimed 15(8):1997–2008
Song J, Gao L, Liu L et al (2017) Quantization-based hashing: a gen-
eral framework for scalable image and video retrieval[J]. Pattern
Recognit 75:S0031320317301322
Song J, Gao L, Liu L et al (2018) Quantization-based hashing: a gen-
eral framework for scalable image and video retrieval[J]. Pattern
Recogn 75:175–187
Stippick J (2004) Advanced tray support system using orthogonal
grillage[J]. J Comput Aided Des Comput Graph 24(22):632–642
Wang Q, Si L, Zhang D (2014) Learning to hash with partial tags:
exploring correlation between tags and hashing bits for large scale
image retrieval[C]//European Conference on Computer Vision.
Springer, NY, pp 378–392
Wu S, Song H, Cheng G et al (2018) Civil engineering supervision
video retrieval method optimization based on spectral clustering
and R-tree[J]. Neural Comput Appl 21(2):12–23
Ying Z, Daiyin Z (2018) Height retrieval in postprocessing-based Vide-
oSAR image sequence using shadow information[J]. IEEE Sens
J 44(13):8–21
Yoo G, Kim H, Hong S et al (2017) Implementation of convergence
P2P information retrieval system from captured video frames[J].
Peer-to-Peer Netw Appl 11:1–11
Zheng F, Tang H, Liu YH (2018) Odometry-vision-based ground vehi-
cle motion estimation with SE(2)-constrained SE(3) poses[J].
IEEE Trans Cybern 127(99):1–12
Zhu X, Huang Z, Shen HT et al (2013) Linear cross-modal hashing for
efficient multimedia search[J]. Comput Sci 2013:143–152
Publisher’s Note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.
13