Visual-based Tracking and Control Algorithm Design for Quadcopter

UAV
Xiangdong Qin, Tingting Wang
School of Mechanical and Electrical Engineering, Hohai University, Changzhou 213022, China
E-mail: 1. 956143376@qq.com

Abstract: The problems of tracking moving target of quad-rotor UAV involve target tracking in image sequence and flight
control of UAV. Because Camshift algorithm is easily disturbed by background with similar color and has poor robustness
to occlusion interference when tracking objects in image sequence. A new tracking algorithm based on Kalman filter and
Camshift algorithm with multi-feature fusion is proposed. On this basis, the horizontal displacement between the target and
the UAV is calculated as the control input, and a position-attitude outer-inner loop controller is designed to ensure the target
near the center of view of the camera in order to track the target efficiently. Experimental and simulation results verify the
performance of the algorithm.
Key Words: quad-rotor UAV; target tracking; Camshift algorithm; outer-inner loop control

1 Introduction algorithm is proposed to realize UAV target tracking, but

the tracking failure caused by the target leaving the field
Over the past few decades, UAVs have been applied in
of view is not considered. In the literature[6-7], control
many fields, such as making models, aerial photography,
laws and filters for UAV target tracking are designed
real-time monitoring and so on. Because of the
respectively, but specific target tracking algorithm is not
advantages of simple structure, easy operation, low cost,
proposed.
remote control and vertical take-off and landing, UAV is
Vision-based tracking and control technology not only
an ideal platform for complex operations, such as tracking
has high theoretical significance, but also has wide
ground target.
application value. From the current research situation, the
Ground target tracking is the tracking of specific
ground target tracking of quad-rotor UAV is still in the
ground targets by UAV in the air. The technology has
development stage.
broad application prospects. In the military field, fighters
This paper is organized as follows. Firstly, the design
can target ground targets to track attacks, and in the civil
of target tracking algorithm is described in Section II.
field, they can provide security services for cities.
Secondly, Section III details the design of target tracking
A lot of research has been carried out on the tracking of
controller. Then, simulation and real-time experimental
moving targets by UAV and a variety of tracking
results are presented in Section IV. Finally, some
algorithms have been proposed, including Meanshift
conclusion remarks are included in Section V.
algorithm, Camshift algorithm, optical flow tracking
algorithm, HOG tracking algorithm, TLD tracking 2 Design of Target Tracking Algorithm
algorithm and so on. In the literature[1-3], the optical
2.1 Camshift Algorithm
flow tracking algorithm is used to effectively track target,
but the calculation amount is too large, which affects the Camshift algorithm uses the color histogram model of
real-time performance of the algorithm. In the the target to transform the image into a color probability
literature[4], TLD algorithm is used to realize character distribution map, initializes the size and position of a
recognition and tracking. In the literature[5], a search window, and adaptively adjusts the position and
background subtraction combined with Kalman filter size of the search window according to the results of the

978-1-7281-0106-4/19/$31.00 2019
c IEEE 3564
previous frame, so as to locate the central position of the
target in the current image[8].
2.2 Kalman Filter Method
Kalman filter is a common method to solve optimal
state estimation in system state estimation theory. It
recursively realizes the optimal estimation of the state of
the current moment by estimating the state of the previous
moment and observing the state of the current moment.
Kalman filter is applied to target tracking. The motion
state parameters of the target in the image sequence are
obtained by prediction and updating, so that the position
of the target in image sequence can be predicted.
2.3 A Modified Camshift Algorithm
A good tracking template is the premise of reliable
tracking in target tracking process. CamShift template
uses the traditional color probability distribution map, that
is, only the H component in HSV color model is counted
and used as a template, which leads to a single tracking
feature in the process of target tracking. When the
background is close to the H value of the target, the
tracking accuracy of the target decreases. Considering the
strong stability of texture feature and edge feature, they
are introduced into the tracking template to establish
three-dimensional histogram of color feature, LBP texture
feature and edge direction feature. This adaptive weighted
fusion multi-feature algorithm improves the problem that
the original CamShift algorithm has a single template
information and is susceptible to the interference of
similar color background.
A prediction mechanism can predict the next frame
position of the target. At present, Kalman filter algorithm
and particle filtering algorithm are commonly used.
Because Kalman filter has the characteristics of small
computation and high real-time performance, and the time
of shooting two adjacent frames of video is very short, the
target can be approximately considered as moving at a
uniform speed. In order to solve the problem of tracking
failure caused by obstacle occlusion and considering the
real-time requirement of target tracking, Kalman filter is
used to predict target position in this paper. Therefore,
Kalman filter combining with Camshift algorithm for
adaptive weighted multi-feature fusion is proposed in this
paper.
The 31th Chinese Control and Decision Conference (2019 CCDC)
The combined algorithm uses Kalman filter to predict
the center position of the search window in Camshift
algorithm. The specific implementation steps are as
follows.
Table 1: Combinational Algorithm Flow
Algorithm
Step 1: Select the tracking target with the mouse
and determine the search window.
Step 2: Initialize of Kalman filter by using search
window centroid position.
Step 3: Predict centroid position of target by using
Kalman filter.
Step 4: Calculate centroid position of target by
using the modified Camshift algorithm.
Step 5: If the target is occluded, the predicted
value of Kalman filter is used as the observation
value to correct Kalman filter.
Else, the output value of Camshift
algorithm is used as the observation value to
update Kalman filter.
Step 6: Go to Step 3.
In the algorithm flow, the Bhattacharyya distance d Bha
between the current frame target image and the hue
histogram of the previous frame target image is calculated
to determine when the target is occluded. A threshold Tb
can be set. If d Bha ≥ Tb is satisfied, the target begins to
be occluded. The value of Tb is 0.35 in this paper.
3 Design of Target Tracking Controller
The target tracking algorithm is designed to get the
pixel position of the target in the image sequence. Then
the horizontal displacement between the UAV and the
target is calculated and the tracking controller of the
quadrotor is designed to ensure that the target is always
near the center of the field of vision obtained by the
airborne camera.
3.1 Dynamics Model of Quadrotor
In order to establish the mathematical model of a
quad-rotor UAV, it is assumed that: 1) The aircraft is a
cross-symmetrical rigid body structure. 2) The origin of
the body coordinate system is located at the geometric
3565
 center of the body. 3) The influence of drag on aircraft is In the formula, l is the distance from the center of the
not considered. 4) The lift generated by the motor is
propeller to the center of the quadcopter. J xx , J yy and
proportional to the square of the rotational speed.
In this paper, the flight mode of UAV is chosen as cross J zz are the moment of inertia and they are all constant.
mode which is shown as Figure 1. m is the quality of UAV.
The motion of quadrotor consists of three axial linear U1 , U 2 , U 3 and U 4 are four control input
motions and angular motions around three axes. In order variables.
to describe the motion state of a quadcopter conveniently
­U1 = F1 + F2 + F3 + F4
and accurately, inertial coordinate frame °U = F − F
° 2
I = {xi , yi , z i } and quadcopter body coordinate frame
4 2
® (2)
U = F − F
B = {xb ,y b ,z b } are established. ° 3 3 1
°¯U 4 = F1 − F2 + F3 − F4

In the formula, Fi =1......4 is the lift generated by four

motors.
  ;L

2L 3.2 The Design of Controller

;E
The camera installed on the platform below the center
2E <L
<E of the UAV captures the image in real time and calculates
=L
=E
the horizontal displacement of UAV and the target
  through the pixel position of the target. From the
knowledge of photographic geometry, the coordinate
transformation relationship between the pixel position of
Fig. 1: Quadcopter’s notation
the target in the image plane and the horizontal
displacement of the target and UAV can be expressed
The displacement of B coordinate frame relative to
as[9]:
I coordinate frame in three directions is expressed by X,
Y and Z respectively. ij , θ and ψ represent the ­ z cos ȕ cos Į
°x = ui
rotation angles of the body coordinate frame B around ° f sin (ȕ + Į)
° z cos Į
the X, Y and Z axes of the inertial coordinate frame I ®y = (3)
° sin (ȕ + Į)
respectively. ij is the roll angle, θ is the pitch angle vi − v 0
°
and ψ is the yaw angle. °Į = arctan f
¯
Based on Newton-Mechanics equation and ignoring the
In the above formula, z is the flight altitude of UAV,
influence of air drag and gyroscopic effect, the dynamic
f is the focal length of camera, β is the pitch angle of
equation of UAV is established.
camera, α is the angle between the object-image
­ .. 1 two-point connection and the optical axis of the camera.
° x = (sin ij sin ȥ + sin ș cos ij cos ȥ ) m U 1
° .. (u 0 ,v0 ) is the coordinate of the pixel of the image center
° y = (sin ș cos ij sin ȥ − sin ij cos ȥ ) 1 U 1 point.
° m
° .. 1 In this paper, in order to ensure the UAV tracking target
° z = (cos ș cos ij) U 1 − g effectively, an outer-inner loop position-attitude controller
° m
® .. l J yy− J zz . . is designed. For outer-loop position subsystem,
°ij = U2 + șȥ (1)
Compensation-based PD control method is adopted.
° J xx J xx
° .. l J zz − J xx . .
°ș = J U 3 + J ijȥ According to the reference values [xd ,yd ,z d ]T , the
° yy yy
° .. 1 J xx− J yy . . expected angles ijd and ș d of the inner-loop attitude
°ȥ = U 4 + șij
J zz J zz subsystem are obtained by the outer-loop position
¯

3566 The 31th Chinese Control and Decision Conference (2019 CCDC)
subsystem. The controlled variable U is obtained by the So, the horizontal displacement ( xd ,y d ) of the target
inner-loop attitude subsystem from the desired angle. and UAV in the world coordinate frame is obtained from
ec as the input of the tracking controller by Formual(3).
Therefore, virtual control variable ȝ = [ ȝ x ,ȝ y ,ȝ z ]T is
The overall structure of the ground target tracking
introduced. controller for UAV is shown in Fig.2.

­μ x = U 1 (cos φ d sin θ d cosψ + sin φ d sinψ )

°
®μ y = U 1 (cos φ d sin θ d sinψ − sin φ d cosψ )
Image sequence tracking
(4)
°μ = U (cos φ cos θ ) Pixel displacement
¯ z 1 d d
zd
ψd coordinate transformation
The control variable U1 and the desired angles of
xd yd

inner-loop attitude can be obtained directly from the Position controller

outer-loop position controller. ϕd θd

­U 1 = μ z (cos φ d cos θ d ) Attitude controller

°
[
®θ d = arctan (cosψ ⋅ μ x + sinψ ⋅ μ y ) μ z ] U2 U3 U4 U1

[
°φ = arctan cos φ (sinψ ⋅ μ − cosψ ⋅ μ ) μ
¯ d d x y z ] Attitude feedback
Dynamics model of quadrotor

Position feedback
(5)
So, for outer-loop position subsystem, PD control Fig. 2: Overall architecture of tracking controller
method is designed as follows.
­ . ..
4 Experiment Results and Analysis
°μ x = − k px ⋅ xe − k dx ⋅ x e + m ⋅ x d
. ..
°° 4.1 Simulation Verification of Controller
®μ y = −k py ⋅ ye − k dy ⋅ y e + m ⋅ y d (6)
° . .. Performance
°μ z = −k pz ⋅ z e − k dz ⋅ z e + m ⋅ z d + mg
°¯
In order to verify the performance of the designed
In the above formula, xe = x − xd , ye = y − y d and tracking controller, the reference input of the UAV is
ze = z − zd . given as follows: xd = 4 , yd = 4 , z d = 3 , ψ d = 1 rad.
Backstepping control algorithm is adopted for The simulation results of position and attitude
inner-loop attitude control. Take the expression of U 2 as controller are shown in Fig. 3 and Fig. 4.
an example. 
1 ª ⋅⋅ ⋅ ⋅ º
U2 = «φ d + (c12 + 1) z1 − a1 θ ψ − (c1 + c2 ) z 2 » (7) 6
b2 ¬ ¼ 4
⋅ ⋅
where z1 = φ − φd , z 2 = φ − φ d + c1 z1 , a1 = ( I y − I z ) I x , 2 xd
x1
x2
0
0 1 2 3 4 5 6 7 8 9 10
b1 = l I x . Time(s)
6
In order to ensure that the UAV follows the ground
4
target, it is necessary to ensure that the target is always
y

2 yd

near the center of the image sequence obtained by the y1

y2
0
airborne camera. The tracking error of the target in the 0 1 2 3 4 5
Time(s)
6 7 8 9 10

image plane is 4

ec = c*p − cp 2
z

(8) zd
1 z1
z2
0
0 1 2 3 4 5 6 7 8 9 10
where c p * and c p are the coordinates of the image Time(s)

Fig. 3: Response curve of position controller

central pixel and the target central pixel, respectively.

The 31th Chinese Control and Decision Conference (2019 CCDC) 3567
 1

-1
1 (degree)

2 (degree)
-2
0 1 2 3 4 5 6 7 8 9 10
Time(s)
2
1
0
-1 1
(degree)

2
(degree)
-2
0 1 2 3 4 5 6 7 8 9 10
Time(s)
Fig. 5: Quadcopter UAV platform
1.5
1 2 3 4 5 6 7 8 9 10
Time(s)
1
As is shown in Figure 6, this work process is explained
(degree)
0.5 d

1
(degree) with a schematic diagram, where airborne camera takes
(degree)
0
2
pictures, and the images are transferred to Intel NUC
0 1 2 3 4 5 6 7 8 9 10
Time(s)  board to calculated the horizontal distance between target
and camera centre. Eventually, results will be sent to N3
Fig. 4: Response curve of attitude controller
flight controller.

From Figure 3 to Figure 4, [ x1,y1,z1,ij1,ș1,ȥ1 ] Intel NUC

Images Visual tracking

represents position and attitude response of double-loop transportation algorithm

PID algorithm, which is proposed in literature [10],

[ x2 ,y2 ,z2 ,ij2 ,ș2 ,ȥ 2 ] represents position and attitude N3 Flight controller Onboard-SDK
Quadrotor

response of the control algorithm proposed in this paper.

From the response curve of the system, it can be seen that Fig. 6: Tracking process frame chart
the designed UAV tracking controller can track the
designated position quickly and stably, and in this process, According to Figure 7, in the experiment, a toy car
the inner-loop attitude controller can stabilize the three model is chosen as the moving target to be tracked, and a
attitude angles of UAV. Compared with the PID algorithm, chair with the same color as the interference.
the algorithm proposed in this paper improves the
performance of maintaining stability time and reaching
the expected time to some extend.

4.2 Tracking Experiment

In this section, experiments are implement to verify the

performance of tracking algorithm.
Figure 5 shows that the DJI UAV is chosen as the
experimental platform. Also, a high-performance CPU is
on board to execute program, whose version is
Intel@Core i5-8350U CPU @1.70GHz.
Fig. 7: Experimental process

3568 The 31th Chinese Control and Decision Conference (2019 CCDC)
 International Conference on Robotics and Biomimetics,
2399-2405, 2012.

[3] H. W. Mao, C. H. Yan, Glen P A, Jennie S, Automated multiple

target detection and tracking in UAV videos. Proceeding of the
Airborne Intelligence, Surveillance, Reconnaissance, 568-577,
2010.
[4] H. S. Ou, J. F. Sun, J. L. Yu, An improved TLD target tracking
algorithm based on the variable-scale, compressing tracking,
International Conference on Communication Software and
Networks, 1223-1228, 2017.
[5] W. H. Huang, Q. L. Zeng, M. Chen, Motion characteristics
estimation of animals in video surveillance. Advanced Information
Fig. 8: Tracking results in image sequences Technology, Electronic and Automation Control Conference
1098-1102, 2017.

The above experimental results show that the proposed [6] Z. Y. Li, Y. F. Xu, G. L. Yang, Simulation of UAV flight control for

tracking algorithm can deal with short-time background tracking ground target, Science Technology and Engineering,

color interference and obstacle occlusion. According to Vol.8, No.5, 1374-1378, 2008.

many flight tracking experimentsˈ compared with single [7] J. Z. Yu, H. X. Liu, C. Y. Liu, Application of AMAUKF on the

Camshift algorithm or Kalman filter algorithmˈ the condition of target re-tracked by UAV, Computer Measurement &

improved tracking algorithm in this paper has been Control, Vol.20, No.2, 516-519, 2012.

greatly improved in accuracy and reliability. [8] L. Wang, Z. J. Li, Z. P. Chen, X. M. Ma, R. Zhang, R. You, The
Application of Improved Camshift Algorithm in Hand Tracking,
5 Conclusion and Futurework
International Conference on Industrial Informatics-Computing
In this paper, the ground moving target tracking Technology, Intelligent Technology, Industrial Information
problem of Four-rotor UAV is studied, and a tracking Integration, 87-90, 2017.
algorithm based on a modified Camshift algorithm [9] L. Ma, C. X. Mu, X. Chao, W. K. Yang., H. P. He, Design of
combining with Kalman filter is proposed, which can deal Target Tracking System for Quad-rotor UAV, Control Engineering
with the problem of target tracking failure caused by of China, Vol.22, No.6, 1076-1081, 2015.
background color interference and obstacle occlusion for [10] J. F. Lin, Study of Flight Control Algorithm for Quad-rotor Aircraft,
a short time. In order to ensure that the target is always Nanchang University, 2015.
near the central position of the image field obtained by the
airborne wireless camera, a position-attitude outer-inner
loop controller is designed, which can track the target
effectively while ensuring flight stability. The simulation
and experiment results show that the designed tracking
system has good performance. In the next research, the
more efficient target tracking algorithm and flying control
system will be combined to implement exploration in
more complex environment.

Reference
[1] Z. Wang, X. J. Yang, Moving Target Detection and Tracking Based
on Pyramid Lucas-Kanade Optical Flow, International Conference
on Image, Vision and Computing, 66-69, 2018..

[2] Menna S, Ramy E, Mohamed E, On-board multiple target

detection and tracking on camera-equipped aerial vehicles, IEEE

The 31th Chinese Control and Decision Conference (2019 CCDC) 3569