IEEE ITOEC(ISSN:2693-289X)
Real-Time Object Detection in UAV Vision
based on Neural Processing Units
Ming Liu1 , Linbo Tang1* , Zongya Li1
1. Beijing Key Laboratory of Embedded Real-time Information Processing Technology, Beijing 100081, China;
249676161@qq.com, tanglinbo@bit.edu.cn, 842782996@qq.com
Corresponding Author: Linbo Tang Email: tanglinbo@bit.edu.cn
Abstract—With the reduction of Unmanned Aerial
2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC) | 978-1-6654-3185-9/22/$31.00 ©2022 IEEE | DOI: 10.1109/ITOEC53115.2022.9734340
Vehicle (UAV) hardware cost and the development of deep
learning algorithm, the real-time object detection algorithm
applied in UAV vision has great advantages in many fields.
However, due to the limited energy consumption and
computing power of embedded devices used in the drones
and the variable object scales and complex backgrounds in
the UAV vision restrict the applications in object detection
based on the drones. In this paper, we optimized the
generation of anchor boxes, introduced a new module to
increase the receptive field to improve the detection of small
targets, and used adaptively spatial feature fusion in the
feature pyramid to increase feature fusion of multi-scale
features. At last we pruned the model to make it lighter and
faster, and got the Average Precision (AP) of 89.7% for
UAV car aerial images and the speed of 35.7 FPS by
running on Neural Processing Units (NPUs), which proves
the feasibility of the intelligent object detection algorithm’s
efficient processing in hardware resource limited
environment.
Keywords—UAV aerial images; car detection; embedded
hardware; real-time processing
I. INTRODUCTION
In recent years, UAV has played an irreplaceable role
in many fields including military and civilian fields
relying on the virtue of their lightweight, Maneuverability,
flexible movement and low energy consumption. Real-
Time Object Detection is an important task in applications
of UAV, and has been well-explored. With the rapid
development of computer vision and artificial intelligence
technology, among many target detection algorithms, the
method based on deep learning has been widely used
because it does not need the Feature Engineering, and has
higher precision.
The object detection algorithms based on deep
learning are mainly divided into two stage methods and
one stage methods. The typical algorithms of the former
are R-CNN, as well as the improved f Fast R-CNN and
Faster R-CNN. The detection speed of these algorithms is
too slow to meet the requirements of real-time. One stage
algorithm does not need to extract candidate regions first,
and the speed is greatly improved. Typical algorithms
include Yolo series and SSD.
However, the usual object detection methods based on
deep learning need a large amount of calculation and large
978-1-6654-3185-9/22/$31.00 ©2022 IEEE
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 05:15:40 UTC from IEEE Xplore. Restrictions apply.
storage spaces for storing intermediate results. Due to the
limited load capacity of UAV, for example, the maximum
load weight of Inspire 2 professional UAV produced by
Dajiang company is 810 g. The UAV platform for car
aerial images object detection can only use the embedded
system driven by battery and limited computing resources.
Therefore, the limitation of hardware cost and
transmission delay is still the primary problem for object
detection tasks in UAV vision.
To sum up, the difficulties of deploying deep neural
network algorithm to achieve effective object detection
based on UAV focus on two aspects [6]:
(1) UAV vision has special mission scenarios and
object characteristics. Different from the mainstream
objection detection, most of these task scenes adopt the
conventional horizontal angle of view, and the object
scale is larger and the characteristics are more obvious.
The content of the drone shots is complex, the size of the
target to be inspected is small, and the features are
ambiguous, which are not the task type targeted by the
mainstream algorithm network design.
(2) The computing power resources of UAV platform
are limited, which is difficult to meet the need of real-time
detection.
II. PROPOSED OBJECT DETECTION ALGORITHM
A. The framework of the YOLOv3 algorithm
We use YOLOv3 as a reference model due to its high
precision and easy to deploy to embedded platforms such
as NPU.
YOLOv3 is an algorithm proposed in recent years [1].
YOLOv3 uses a more powerful feature extraction network
Darknet-53. As shown in Figure 1, due to the introduction
of a residual structure, Darknet-53 deepens the network to
53 layers, and the feature extraction capability is further
improved. In addition, YOLOv3 draws on the anchor
mechanism in Faster R-CNN, and introduces the feature
pyramid network (FPN) structure [2], which performs
detection on three feature scales of 13×13, 26×26, and
52×52. Greatly improved the detection effect of YOLO
series algorithms on small targets.
1951

Fig. 1. YOLOv3 network structure
However, because the network structure of YOLOv3
is still too complicated, the detection speed of YOLOv3 is
further reduced, and it cannot run in real time on edge
devices with limited computing power such as drones.
B. Optimize the parameters of the anchor
Figure 2 is the sample distribution obtained based on
the sample width and height of the general data set COCO
and VOC data set. Figure 3 is the sample distribution
obtained based on the sample width and height of the
UAV aerial image data set used in this paper. We can see
clearly that the sample size of the aerial data set changes
greatly, and most of the targets are with smaller scales, so
it is necessary to re-select the appropriate anchor
parameters to make better predictions on the aerial data
set.
Fig. 2. Bounding and anchor distribution of general data set (left) and
UAV aerial image data set (right)
The usual method to re-select the appropriate anchor
parameters is clustering the training set by K-means
method, which is executed by a separate program. In our
method, we have embed this function into the training
progress, and adaptively calculate the best anchor box
value in different training sets during each training.
In the task of object detection, the purpose of
clustering is to make the IOU value of the anchor box and
the ground truth as large as possible, and Avg IOU is
calculated by the following formula.
gtbox
Avg IOU = IOU centroid
Among them, box is the target box of the sample label,
and centroid is the cluster center.
Table 1 is the comparison of Avg IOU of the
traditional K-means clustering algorithm on the data sets
of this paper.
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 05:15:40 UTC from IEEE Xplore. Restrictions apply.
TABLE I. THE COMPARISON OF DIFFERENT ANCHOR GENERATION
METHODS
Method Clusters Avg IoU
K-means 9 0.73
Improved methods 9 0.75
The test results show that the improved method for
optimizing anchor frame parameters is better on the aerial
data set used in this article. Table 1 is the comparison of
Avg IOU of the traditional K-means clustering algorithm
on the data set of this article.
C. Context augmentation module
The object detection requires context information,
especially in terms of small objects, which are the main
components of the aerial data sets. Therefore, we propose
an improved receptive field module (IRFM), as shown in
Figure 4, which merges multi-scale dilated convolution
[3], and set the trainable parameter W to perform
weighted fusion with the shortcut layer outputs.
Fig. 3. The structure of IRFM
The IRFM is a module that increases the receptive
field, and improves the detection accuracy of small targets
almost without increasing computational consumption.
D. Adaptively Multi-scale Spatial Feature Fusion
YOLOv3 uses the feature pyramid (FPN) for
contextual information fusion, which merges adjacent
(1) feature maps with different grains from top to bottom,
which greatly improved the multi-scale expression ability
of the network. However the concatenation-based fusion
method between different feature maps cannot make full
use of the features of different scales. So we try to use the
adaptively spatial feature fusion (ASFF) method to
enhance the multi-scale expression ability of the network
[4], which is crucial to target detection tasks in UAV
vision.
1952

Fig. 4. Overall network structure
As shown in Figure 1 and Figure 5, {C3, C4, C5}
represent different levels after input image being down-
sample by {8, 16, 32} times. {L1, L2, L3} are denoted the
feature levels generated by IRFM and FPN, {P1, P2, P3}
are the features levels generated by adaptively spatial
feature fusion module (ASFFM). X i is defined as the
input of the mth（m  {1,2,3}）layer of ASFFM. X
is defined as
the result of resizing from the
nth（n  {1,2,3}）layer to the mth layer at the position (x,
y). Yi ,mj is defined as the
mth（m  {1,2,3}）layer of ASFFM at the position (x, y).
So, the outputs of the ASFFM are:
Yi ,mj  aim, j  X i(,1j,m )  bim, j  X i(,2j,m )  cim, j  X i(,3j,m )
In the above formula, the adaptive parameters
a m b m and c m are respectively obtained by 1×1
convolution of the resized {L1, L2, L3} feature maps.
And the parameters a, b and c are passed through concat
through softmax so that their range is within [0,1] and the
sum is 1, as follows:
[ a m , b m , c m ]  Soft max( F )
Take a m as an example, a m is calculated by the
following formula, b m c m follow the same calculation
principle.
maij
m e
a i, j  maij bmij
e e e
In this way, the features of all layers of FPN are fully
fused together under the guidance of adaptive weights,
and {P1, P2, P3} is used as the final output of the entire
network.
E. Model compression
The CNN network is still complicated and each
iteration calculation will generate tens of millions of
calculation weights. In the meantime, not all
convolutional layer channels will be put into the training
process, so there are many redundant convolutional layer
channels in the network that have a very small impact on
the prediction results, resulting in a decrease in
computational efficiency. In this paper we choose a
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 05:15:40 UTC from IEEE Xplore. Restrictions apply.
method to perform channel pruning [5] and layer pruning
simultaneously to compress the width and depth of the
model respectively. The model compression process and
specific steps are as follows in Figure 6.
Fig. 5. Model compression process
 Perform sparse training on the network model to
determine the importance of the clipped channel.
 Determine the pruned channel according to the
( n ,m ) calculated judgment standard, and start pruning.
i, j
 Evaluate the previous CBL of each shortcut layer,
sort the highest gamma value of each layer, and
take the smallest layer for pruning.
output of the  Calculate the global threshold, and use this
threshold to control the pruning process.
 Introduce a local safety threshold to determine
whether pruning is excessive, and set its value in
(2) the algorithm to the percentile of all values in a
specific pruning layer. When the size of the
trainable scale factor is less than the threshold, stop
pruning, prevent excessive iterative pruning from
causing permanent damage to the model.
 Fine-tune the parameters of the pruning model to
ensure complete pruning.
(3)
III. EXPERIMENT AND ANALYSIS
A. Introduction of NPU hardware platform and
algorithm embedded implementation
UAV platforms face the problem of limited hardware
resources. Among the existing multiple smart computing
front-end application solutions: Embedded GPU projects
mcij
(4) have a short implementation cycle, but the chip power
consumption is high, and at the same time it is difficult to
control independently; FPGA can be customized, but the
development cost is high; high-performance NPU has the
benefits of low cost and low power consumption.
In this paper, we decide to use T710 development
board to build an embedded airborne target recognition
system. T710 was released in 2019 by ZiGuangZhanRui
and has an AI computing power of 3.2 TOPS and a
comprehensive computing power of 4.2 TOPS. Compared
with the traditional GPU as a large-scale chip solution of
the AI computing unit, it has an energy efficiency of more
than 2.5 TOPS/W. T710 is also equipped with an eight-
core high-performance processor which adopts a four-core
cortex-A75 and four small-core cortex-A55 architecture,
and with a dominant frequency of 2.0 GHz and super
1953
 computing performance. We choose this board for its low
power consumption and accelerated computational
efficiency of convolutional networks and be in support of
the localization development of chip. The figure 7 is the
overall idea of the target detection system.
Fig. 6. The integral system design
Here are the specific steps:
 Train and test at a high-performance computer to
verify the algorithm and get a high-performance
model based on the data set.
 Replace the operator that is not supported by NPU.
In this case, the nearest neighbor interpolation
upsampling operator is replaced by deconvolution
operator, and the coordinate decoding process of
the network is removed too which is not supported
by NPU.
 Use the model quantization and transformation
tool. In this case, uds710 imaging AI NDK is
offered by ZiGuangZhanRui company which is
installed in Ubuntu 16.04 environment, the model
is quantized into 8-bit width and transformed into
an executable file that can be recognized by NPU.
 The processor of T710 is equipped with Ubuntu
system, in which we call its API to accelerate the
network computing process, and uses C++
programming to supplement the coordinate
decoding process and Non-Maximum Suppression
(NMS) process, so as to complete the end-to-end
computing output.
B. Data set
This article focuses on the car targets in aerial images.
In order to verify the performance of the research results
in this article on aerial perspective images, we constructs
an aerial vehicle data set that includes multiple aerial
perspectives, multiple vehicle types, and multiple ground
object backgrounds, including an aerial vehicle data set of
12840 images. Among them, 3456 photos were taken by
myself, and the rest were a mixture of unlabeled public
data sets and Visdrone 2019. This paper randomly selects
70% and 30% from the entire data set as the training set
and the test set, respectively. All models in this paper are
trained based on this training set and evaluated on the test
set.
1954
Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 05:15:40 UTC from IEEE Xplore. Restrictions apply.
Fig. 7. UAV aerial images of our data set
The width and height distribution of the samples in the
data set are as follows:
Fig. 8. The distribution of objects
From the Figure 9, we can get information about the
objects in the data set. Most of the targets have a width of
50 pixels and a length of 20 ~ 30 pixels, while the aspect
ratio is concentrated between 0.4 ~ 0.6, indicating that
most of the targets are small and have an overall long bar
shape, because the height is smaller than the width.
C. Evaluation measurement
The effect of target detection is determined by the
classification accuracy and positioning accuracy of the
prediction frame. Therefore, the target detection problem
is both a classification problem and a regression problem.
The comprehensive evaluation index of the target
detection algorithm is usually the average accuracy rate.
Average Precision (AP) AP is defined as the area
under the PR curve, which is used to measure the average
classification accuracy of a class in the data set. The
calculation formula is as follows:
1
AP   P( R)dt (5)
0
D. Experimental result
This article designs three sets of experiments to train
and test the three improvement points, and finally
combines the three improvement points to design a
comprehensive experiment and conduct training and
testing:
Experiment 1. Train and test the optimized anchor
boxes YOLOv3. There are mostly small targets in the data
set. This paper performs clustering and adaptive anchor
box calculations for the data set. Three anchor boxes are
 still allocated for each scale. The calculation results are as Table above shows that our proposed object detection
follows: algorithm based on YOLOv3 in this paper improves AP
by 7.4% compared with YOLOv3 on this data set, and the
TABLE II. ANCHOR BOXES ON DIFFERENT FEATURE MAP AP after compression is still about 6.2% higher than
Feature Anchor boxes Anchor boxes
YOLOv3.
map (K-means) (improved method) Comprehensive experiment Compare several groups
16x16 (64,91) (102,59) (153,128) (64,86) (105,58) (151,127) of algorithms in the same data set, table 5 shows the
32x32 (38,34) (40,58) (66,36) (43,22) (41,49) (68,35) comparison results of YOLOv3-ours, YOLOv3,
YOLOv3-spp and yolov4 in accuracy, model size and
64x64 (15,36) (40,21) (23,44) (11,9) (21,16) (26,30) inference time. We can see that the size of the model is
reduced by 86.5% compared to YOLOv3. And the
Yolov3-ours model has great advantages in detection
TABLE III. AP CALCULATED BY DIFFERENT METHODS
accuracy and model size.
Anchor generation method AP@0.5 (car)
K-means 0.835 TABLE VI. AP CALCULATED BY DIFFERENT SETS OF EXPERIENCES

Ours 0.846 methods Input AP@0.5 Time/ms Size/MB

Size (car) (NPUs)
Experiment 2. Train and test the YOLOv3 which YOLOv3 512 0.835 150 246.3
introduces IRFM. The function of the IRFM structure is to YOLOv3-SPP 512 0.841 153 250.6
increase the receptive field of the feature map. Generally
YOLOv4 512 0.877 not supported 256.7
speaking, the receptive field of shallow features is smaller, by NPU
and the receptive field of deep features is larger. YOLOv3 YOLOv3-ours 512 0.897 28 36.9
fuses deep features with shallow features through
upsampling. This article first inputs shallow features into
IRFM, and then merges them with deep features. The IV. CONCLUSIONS
following table analyzes the effect of IRFM on feature In this paper, we propose a new object detection
maps of different scales. algorithm in UAV vision based NPU, which realizes the
real-time object detection in resource constrained
TABLE IV. AP CALCULATED BY DIFFERENT LOCATIONS OF IRFM embedded environment. Based on YOLOv3, the
IRFM Location AP@0.5 (car) algorithm introduces a new IRFM to enhanced receptive
field to obtain richer context information, we also used
After C3 0.847 adaptively spatial feature fusion in the feature pyramid to
After C4 0.863 increase feature fusion of multi-scale features. In addition,
After C5 0.865 we optimized the generation of anchor boxes, which
proves to achieve better performance in the UAV car
After C4 and C5 0.871
aerial image sets. At last, we pruned the model to make it
lighter and faster. Final experiments on our UAV data set
It can be found from the table above that compared show that the new algorithm can run successfully on the
with YOLOv3, when IRFM is inserted after C3, AP domestic NPU hardware platform, and gains the decent
almost does not increase, and when IRFM is inserted after AP at the same time, the detection speed fully meets the
C4 and C5, AP increases 2.5%. this experiment needs of real-time detection.
determines the appropriate location of IRFM and
effectiveness of IRFM.
Experiment 3. Train and test the improved multi- REFERENCES
scale feature fusion YOLOv3 on the data set. Combing [1] Redmon J, Farhadi A, “YOLOv3: An Incremental Improvement,”
experiment1 and experiment 2 to get our best model result. arXiv e-prints, 2018.
The test results are shown in the table below: [2] Lin T Y, Dollar P, Girshick R, et al, “Feature Pyramid Networks
for Object Detection,” IEEE Computer Society, 2017.
TABLE V. AP CALCULATED BY DIFFERENT SETS OF EXPERIENCES
[3] Liu S, Di H, Wang Y, “Receptive Field Block Net for Accurate
methods AP@0.5 (car) and Fast Object Detection,” 2017.
Yolov3 0.835 [4] Liu S, Huang D, Wang Y, “Learning Spatial Fusion for Single-
Shot Object Detection,” 2019.
YOLOv3 with optimized anchors 0.846 [5] Zhuang L, Li J, Shen Z, et al, “Learning Efficient Convolutional
YOLOv3 with IRFM after C4 and C5 0.871 Networks through Network Slimming,” IEEE, 2017.
[6] Zhang Wei, Zhang Xingtao, Wang Xueli, Chen Yunfang, Li
YOLOv3 with ASSM 0.884
Yanchao, “DS-YOLO: A real-time small target detection
YOLOv3-ours (with all improvements) 0.909 algorithm deployed on UAV terminal,” Journal of Nanjing
University of Posts and Telecommunications (natural science
YOLOv3-ours (after compression) 0.897 edition), 2021,41(01): 86-98. DOI: 10.14132/j.cnki.1673-
5439.2021.01.011.

1955

Authorized licensed use limited to: ULAKBIM UASL - Uluslararasi Kibris Universitesi. Downloaded on March 22,2023 at 05:15:40 UTC from IEEE Xplore. Restrictions apply.