See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/348368287

Vehicle Position Estimation with Aerial Imagery from Unmanned Aerial Vehicles

Conference Paper · October 2020

DOI: 10.1109/IV47402.2020.9304794

CITATIONS READS
14 56

4 authors, including:

Friedrich Kruber Eduardo Sanchez Morales

Technische Hochschule Ingolstadt Technische Hochschule Ingolstadt
12 PUBLICATIONS 106 CITATIONS 17 PUBLICATIONS 70 CITATIONS

SEE PROFILE SEE PROFILE

Michael Botsch
Technische Hochschule Ingolstadt
79 PUBLICATIONS 431 CITATIONS

SEE PROFILE

All content following this page was uploaded by Eduardo Sanchez Morales on 22 February 2022.

The user has requested enhancement of the downloaded file.

 Vehicle Position Estimation with Aerial Imagery from
Unmanned Aerial Vehicles
Friedrich Kruber1 , Eduardo Sánchez Morales1 ,
Samarjit Chakraborty2 , Michael Botsch1

Abstract— The availability of real-world data is a key element

for novel developments in the fields of automotive and traffic
research. Aerial imagery has the major advantage of recording
arXiv:2004.08206v2 [cs.CV] 13 May 2020

multiple objects simultaneously and overcomes limitations such

as occlusions. However, there are only few data sets available.
This work describes a process to estimate a precise vehicle
position from aerial imagery. A robust object detection is
crucial for reliable results, hence the state-of-the-art deep
neural network Mask-RCNN is applied for that purpose. Two
training data sets are employed: The first one is optimized
for detecting the test vehicle, while the second one consists of
randomly selected images recorded on public roads. To reduce
errors, several aspects are accounted for, such as the drone
movement and the perspective projection from a photograph.
The estimated position is comapared with a reference system
installed in the test vehicle. It is shown, that a mean accuracy
of 20 cm can be achieved with flight altitudes up to 100 m,
Full-HD resolution and a frame-by-frame detection. A reliable
position estimation is the basis for further data processing,
such as obtaining additional vehicle state variables. The source
code, training weights, labeled data and example videos are
made publicly available. This supports researchers to create
new traffic data sets with specific local conditions.

I. INTRODUCTION
Real-world data is essential for automotive research and
traffic analysis. The publicly available data sets can be
mainly split in two groups: The first group provides data from
the vehicle perspective as provided in the KITTI [1], Waymo
[2] or Audi A2D2 [3] data sets. These kind of data sets
boost research in terms of in-vehicle functionality, mainly
computer vision tasks. In order to analyze traffic with an
overall view on the situation, other approaches are prefer-
able. Commonly used infrastructure sensing technologies like
inductive loops provide accurate accumulated traffic data,
while not being capable of providing individual trajectories
of traffic participants. Aerial imagery from Unmanned Aerial
Vehicles (UAV), usually drones, overcome this limitation.
Yet, only few birds-eye-view data sets are available, while
interest is growing due to technological progress. Recently
published work and data sets are dicussed in Section II. The
behavior of traffic participants and infrastructural conditions
differ throughout the world. This underlines the need for data
according to the local specifics. The present work describes
a process to generate reliable position data. Figure 1 depicts
an example from the experiments of Section IV.
1 Technische Hochschule Ingolstadt, Research Center CARISSMA,
Esplanade 10, 85049 Ingolstadt, Germany, {firstname.lastname}@thi.de
2 University of North Carolina at Chapel Hill (UNC), USA,
{firstname}@cs.unc.edu
Fig. 1: Bird’s eye view on an orthorectified map of the test track
at 50 m flight altitude. Depicted is the trajectory of the reference in
black and drone in red, respectively. The car was driven in spirals
to capture different vehicle poses and positions in the image frame.
The blue rectangle and cross indicate the image frame and center.
For estimation purposes, a true-to-scale T-junction is painted white
on the test track.
For further details on how to obtain additional vehicle state
variables from UAV imagery, given the position estimation,
the reader is referred to [4].
Aerial remote sensing measurements have various ad-
vantages, such as dozens of objects can be captured in
parallel with one sensor. UAVs are versatile by means of
locations and covered area on ground. Also, the hovering
position can be chosen to reduce oclusion. The generated
data suits for both, research on individual traffic participant
behavior and its predictions [5], [6], as well as accumulated
traffic flow analysis [7], [8]. Batteries are the bottleneck, but
this disadvantage can nowadays be compensated by tethered
systems, which allow flight durations of several hours. Wind
and water resistance, alongside with low-light capabilities of
cameras, are constantly improving, but remain a weak point.
Before generating such birds-eye-view data, several as-
pects need to be taken into account: Drones are often
equipped with non-metric cameras, so that the distortion has
to be removed. The videos are affected by some movement
and rotation of the hovering drone. Estimating the location
of a vehicle within its environment requires a fixed frame.
This property can be achieved algorithmically using image
registration techniques. Photograhps yield a perspective pro-
jection, so that the detected objects are displaced compared
to the ground truth. Section III details how these aspects can
be addressed.
Own contributions
The three main contributions of the present work can be
stated as follows:
First, a framework to obtain precise vehicle positions from
UAV imagery based on instance segmentation and image
registration is provided. To the knowledge of the authors,
no other comparable open sourced framework is available.
Second, it is shown, how the accuracy can be optimized,
compared to related work. Reducing the error is meaningful,
for example, to associate a vehicle to its actual lane. Fur-
thermore, a small error is necessary to detect lane changes
at the right time instance and to compute criticality measures
in a general sense. Accurate data acquisition is also essential
to understand the locally characterized driving behavior and
to develop algorithms based on it. A precise localization
and representation of the vehicle’s shape allows the use
of simple trackers such as [9], which associates detections
across frames in a video sequence.
Third, the method’s capabilities and limitations are evalu-
ated with an industrial grade reference sytem. This work is a
step towards large-scale colletion of traffic data using UAVs
and discusses its feasibility and challenges.
II. RELATED WORK
Object detection and tracking via UAV gained attention
over the past years. DroNet [10] investigates the real time
capability of vehicle detection with small onboard hardware.
DroNet is a lean implementation of the YOLO network
[11], where the number of filters in each layer is reduced.
DroNet outputs several frames-per-second (fps) with onboard
hardware, but at the cost of lower detection performance
and image resolution. The network struggles with variations
in flight height and vehicle sizes. It outputs horizontal
bounding boxes, which are not suitable for estimating certain
variables such as orientation. The R3 network [12] enables
the detection of rotated bounding boxes. R3 is a bounding
box detector, while Mask R-CNN [13] yields instance seg-
mentation. Reference [14] approaches vehicle detection via
instance segmentation. One goal of [14] is to obtain a high
detection rate at higher altitudes, while the present work
pursues a precise position estimation at lower altitudes up
to 100 m. Also, the experimental results are compared to a
reference system this paper.
The methodology of [15] to assess aeriel remote sensing
performance is comparable to this work. A test vehicle was
equipped with a GPS logger to receive positions and speed.
The images were geo-referenced to obtain a fixed frame. The
main differences to the present work can be stated as follows:
The detection algorithm compares the differences between
two frames, hence identifying moving objects by localizing
altered pixel values. This type of detector is prone to errors,
e.g., during vehicle standstills, changing light conditions or
due to the movement of vegetation, as stated by the authors.
The output is a non-rotated bounding box, which fails to
estimate the vehicles shape and thereby worsens the position
estimation. The reference sensor1 in [15] provides accuracy
of 20 cm at best, assuming the Diffenrential GPS version.
In the present work, the reference sensor’s accuracy is 1 cm,
which is necessary to compare at pixel level, see Section III-
A. The images in [15] were processed with a Gaussian blur
filter, which is claimed to eliminate high frequency noise.
Applying such a filter blurs the edges and is contraproductive
when applying a neural network detector. Finally, relief
displacement was not taken into account, which causes an
increasing error with growing distance to the principal point,
see Section III-D.2. The authors state a normalized root mean
square error of 0.55 m at a flight altitude of 60 m. By the
same measure, the error obtained in the present work is much
lower with 0.18 m at a flight altitude of 75 m and identical
image resolution.
Except for DroNet, a missing publicly available implemen-
tation is common to all the above mentioned publications.
Recently, the highD [16], inD [17] and INTERACTION
[18] data sets were published. They provide processed
traffic data obtained with drone and static camera images.
While [16] provides German highway data, [18] offers urban
sceneries like crossings and roundabouts and [17] includes
furthermore pedestrians and cyclists. The Stanford campus
data set mainly captures pedestrians and bicycles on a cam-
pus, the publication focuses on human trajectory prediction
[19]. While [16], [17], [18] provide traffic scenario data sets,
this paper describes the procedure to obtain vehicle positions
and compares the results to a widely accepted reference.
Finally, the code is open sourced for further improvements
and to faciliate the generation of new data sets.
III. METHOD
Generating traffic data with UAVs is appealing, but certain
challenges have to be mastered. First of all, the images
are recorded with a flying object, i. e. a fixed frame has
to be established. Second, photographs yield a perspective
projection. The top of objects are displaced from their
bases in vertical recorded photgraphs, leading to a false
interpretation, when directly computing positions from their
bounding boxes. Obtaining bounding boxes in a sequence of
many images, on the other hand, requires matured detection
techniques, which are limited by the accuracy and amount of
labeled training data. Finally, when performing a benchmark,
the mapping and synchronisation of both data sources have
to be considered.
In the following, the main steps are described as depicted
in Figure 2. Beforehand, the coordinate systems used in this
work are explained.
1 Video VBOX Pro
 Data generation Pre-Processing
Video and DGPS Image registration
Fig. 2: The overall process: From data recording to relief displacement correction.
The vehicle moves on the Local Tangent Plane (LTP),
where xL points east, yL north and zL upwards, with an
arbitrary origin oL on the ground of the earth. The Local
Car Plane (LCP) is defined according to the ISO 8855 norm,
where xC points to the hood, yC to the left seat, zC upwards,
with the origin oC at the center of gravity of the vehicle.
For simplification, it is assumed that 1) the xC yC -plane is
parallel to the xL yL -plane, 2) the centre of mass is identical
to the geometrical centre, and 3) the sensor in the vehicle
measures in the LCP. The Pixel Coordinate Frame (PCF)
is a vertical image projection of the LTP, where xP and yP
represent the axes, with the origin oP in one corner of the
image. Quantities expressed in PCF are given in pixels (px).
Throughout this work, vectors are represented in boldface
and matrices in boldface, capital letters.
A. Data Recording
The data set was recorded on a test track. This gives
degrees of freedom regarding arbitrary trajectories within
the image frame. Experiments can be repeated with the same
setup. On the other side, recording on public roads challenges
the detector. Since test vehicles are a limited resource, a
different approach is chosen to validate the detector to some
extend. A second training set, recorded on public roads, is
used to validate wheter the test vehicle is detected in a robust
manner. See Table II in Section IV for further details. The
results suggest, that given a suitable large training data set,
the detection on public roads also performs well.
Next, the recording process is detailed. Table I depicts the
flight altitudes and Ground Sampling Distance (GSD) for
the drone2 in use. The GSD is also known as photo scale or
spatial resolution, see Section III-D.1 for details about the
computation.
Flight altitude 50 m 75 m 100 m
Number of frames 14 532 15 217 24 106
GSD [cm/px] 3.5 5.2 6.9
TABLE I: Total count of frames and GSD per altitude. At higher
altitudes a larger area on ground has to be covered, thus increasing
the number of video frames.
Generally, for a vertical photograph, the GSD S is a
function of the focal length f of the camera and flight altitude
H above ground:
f a matching point on two images was found. The matches
S=
. (1)
H
Varying altitudes brings flexibility in the trade-off between
GSD and captured area on the ground. The videos were
recorded with 50 fps and 4K resolution (3840 px x 2160 px)
2 DJI Phantom 4 Pro V2
Object Detection Post-Processing
Rotated Bounding Box Mapping, Relief D.
Fig. 3: Registered image (left) and the raw drone image (right) of
a Ground Control Point (GCP) from 1.5 m height. The left image
borders are clipped due to translation and rotation. The red box
depicts the object location from the first video frame, which was
shot around 30 s beforehand.
and afterwards compressed to Full-HD (1920 px x 1080 px)
by applying a bicubic interpolation. For each altitude five
videos were recorded. The camera exposure time was kept
1
constant at 400 s for all recordings.
A spiral template trajectory was driven to obtain different
vehicle poses and to cover a large are of the image. The test
vehicle was then equipped with a driving robot and a Satellite
Navigation system3 that receives RTK correction data. This
ensures an identical reproduction of the trajectory for all
experiments, and a centimeter-accurate vehicle localization.
B. Pre-Processing
The pre-processing consists generally of two parts: The
camera calibration and the image registration. According
to the manufacturer of the drone, the camera is shipped
calibrated, so this step is skipped. The image registration
is performed to overlay the sequential frames over the first
frame to ensure a fixed frame. The registration applied in
this work is composed of a correction of the orientation,
translation, and scaling of the image. Figure 3 depicts an
example of the registration result. This process involves three
steps in order to find correspondences between two images: a
feature detector, a descriptor and finally the feature matching.
The goal of the detector is to find identical points of interest
under varying viewing conditions. The descriptor is a feature
vector, which describes the local area around the point of
interest. For this work, the Speeded Up Robust Features
algorithm [20] is used as a detector and descriptor. To match
the points between two images, the distances between the
feature vectors are computed. If the distance fulfills a certain
criterion, e. g., a nearest neighbor ratio matching strategy,
are then fed into the MLESAC algorithm [21] to eliminate
outliers. Lastly, a randomly selected subset of the remaining
matching points are used for the image scaling, rotation and
translation.
3 GeneSys ADMA-G-PRO+
C. Object detection
The pre-processed images are fed into the Mask-RCNN L
implementation of [22], which is pre-trained on the Common f
Objects in Context (COCO) data set [23]. Mask R-CNN
extends Faster R-CNN [24] by adding a parallel, Fully Con- PCF
volutional Network [25] branch for instance segmentation, oc
H
next to the classification and bounding box regression from a a´
d r
Faster R-CNN. The network is a so called two stage detector:
In the first stage, feature maps generated by a backbone
network are fed into a Region Proposal Network, which
outputs Regions of Interest (RoI). In the second stage, the
predicitions are performed within the RoIs. Additionally, a LTP
Feature Pyramid Network is included for detecting objects Oc
A
at different scales [26].
h
Transfer learning has been applied with two training sets:
A´ R
The first one contains 196 randomly selected and manually A´´
labeled images with the test vehicle being present on all D

images. The second set contains 133 randomly selected

images with 1 987 manually labeled vehicles, recorded on
public roads. This is done to examine the performance under
general conditions. In the following, the two training sets Fig. 4: Geometry of the relief displacement, adapted from [28].
The red bar depicts an object of height h. Due to the perspective
are named ”specialized” and ”general”. Details about the projection and R > 0, the top of the bar A is displaced on the
training procedure can be obtained from the code. Finally, the photo compared to the bottom A0 . The relief displacement d is the
binary mask output is used to compute the smallest rectangle distance between the corresponding points a and a0 in the PCF.
containing all mask pixels using an OpenCV [27] library.
 T
D. Post-Processing steps defined in PCF as bi = xb,i,P yb,i,P , and the bounding
box is defined in PCF as
To complete the process, two more steps are performed.  
First, the output from the detector, given in PCF, has to be BP = b1 b2 b3 b4 . (5)
mapped on the LTP. Finally, the disturbing relief displace-
The corners of the bounding box are mapped to the LTP as
ment is handled.
shown in Eq. (4) to obtain BPL . The geometric centre of the
1) PCF Mapping: A comparison to the reference requires vehicle oveh is calculated by
the mapping of the PCF on the LTP. For this, GCPs are
placed on the xL yL -plane, in such a way that they are  
max(BPL 1,i )+min(BPL 1,i )
visible on
 the PCF. T The i-th GCP is defined  in LTP as 2
gi,L = xi,L yi,L , and in PCF as gi,P = xi,P yi,P .
T oveh = 
max(BPL 2,i )+min(BPL 2,i )
, (6)
The GSD S is calculated from two GCPs by 2

for i = 1, . . . , 4. The dimensions of the detected vehicle are

|gi+1,L − gi,L |
S= . (2) calculated next. Let
|gi+1,P − gi,P |
||b2 − b1 || < ||b3 − b1 || < ||b4 − b1 ||, (7)
 canTthen be expressed in meters by g̃i =
The i-th GCP
gi,P · S = x̃i ỹi . The orientation offset δ from the then ŵ = S · ||b2 − b1 || and ˆl = S · ||b3 − b1 || are the
LTP to the PCF is calculated as δ = θi − θi,L , with θi = estimated width ŵ and length ˆl of the vehicle in meters.
atan2(ỹi+1 − ỹi , x̃i+1 − x̃i ). θi,L is calculated by analogy. 2) Relief displacement: Photographs yield a perspective
The GCP g̃i is then rotated as follows projection. A variation in the elevation of an object results in
T
a different scale and a displacement of the object. An incrase
ĝi = R (δ) g̃i , (3) in the elevation of an object causes the position of the feature
where R (·) is a 2D rotation matrix. The linear offsets from to be displaced radially outwards from the principal point Oc
the LTP to the PCF are calculated by ∆ = ĝi − gi,L . Finally, [28].
 T Assuming a vertical camera angle, the displacement can
a pixel pP = xP yP on the PCF can be mapped to the
be computed from the similar triangles LOc A00 and AA0 A00 ,
LTP by 
T
 according to Figure 4:
pLP = R (δ) (pP · S) − ∆. (4)
D R d r
= , = , (8)
The next stage is to semantically define the four bounding h H h H
box corners. It is assumed that the box covers the complete where the second equation is expressed in GSD, with d
shape of the vehicle. The i-th corner of the bounding box is defining the relief displacement and r the radial distance
between oc and the displaced point a in PCF. D defining the
1.2
equivalent distance of d, projected on ground, R the radial 1.004
distance from Oc , H flight altitude and h being the object
1
height in LPT. L is the camera lense exposure station, where 1.002
light rays from the object intersect before being imaged at

the cameras’ sensor. The relief displacement decreases with
an increasing hovering altitude and is zero at Oc .
According to Eq. 8, the bounding box has to be shifted
radially. Two approaches are described: The first one requires
knowledge of the vehicle sizes, and the second one is an
approximation for unknown vehicle dimensions. Since the
training is performed to detect the complete vehicle body, the
corner closest to Oc can be usually identified as the bottom of
the vehicle. So the height of this point is equal to the ground
clearance. Knowing the height of this corner, its displacement
is corrected as described in the following.
Defining the horizontal and vertical resolution of the image
as rx and ry , the coordinates in PCF of bi w.r.t. the image
center are given by

xb,i,img
 
xb,i,P − r2x
 overall error is reduced compared to the initial situation of
= r . (9)
yb,i,img yb,i,P − 2y
The shift ∆x,P along the xP axis is calculated on the PCF
by
xb,i,img · hb,i,L
∆x,P = , (10)
H will be presented.
where hb,i,L is the height of the i-th corner on the LTP. The
shift for ∆y,P is computed by analogy along the yP axis. The
approximated coordinates bi,shift of bi are then given by
 T
bi,shift = bi − ∆x,P ∆y,P . (11)
Let w be the width and l the known length of the vehicle
and b1 be the closest corner to the image centre. Then, b1
is used for scaling bl and bw as follows
w 
bw,scaled = · (bw − b1 ) + b1 , and (12)
ŵ
 
l
bl,scaled = · (bl − b1 ) + b1 , (13)
ˆl
where w is the element of BP associated with ||b2 − b1 || and
l associated with ||b3 − b1 ||, respectively. The shifted centre
of the vehicle can then be calculated by
bw,scaled + bl,scaled
oveh,shift = . (14)
2 • Rotation and GSD with GCPs,
When gathering data on public roads, the vehicle dimensions
are unknown and can not be estimated with a mono camera.
An approximation for the displacement can be performed
by assuming that two corners of the bounding box, which
form ˆl and one of the corners is the closest to oc , the
ground clearance is usually visible. The height of the ground
clearance can be approximated with 15 cm for passenger cars
[29]. The remaining two corners can usually be referred to as
the vehicle body shoulders, which protrude further than the
roof of the vehicle. The shoulders height is roughly half of
the vehicle height and can be approximated with 75 cm for
passenger cars. Then all four corners can be shifted following
rotation [deg]
scaling factor
0.8
1.000
0.6
0.998 0.4
rotation 0.2
0.996 scaling
0
0 10 20 30 40 50 60
time [s]
Fig. 5: Image registration: Scaling and rotation for a period of 60 s:
Rotation in till 30 s. The scaling factor in with a drop
at around 35 s, caused by an altitude drop.
Eq. (10). Although this is only a coarse approximation, the
neglecting the displacement.
IV. EXPERIMENTS
In this section the main potential sources of errors are
being discussed. Later, the results of estimating the position
A. Sources of errors
The overall process involves several steps, of which all
affect the accuracy of the position estimation. Two groups
of potential errors can be distinguished: The first group de-
scribes general issues appearing from aerial imagery captured
by an UAV. Here, the registration and relief displacement
have the most significant influence. The second group is
only of concern, when comparing to a reference system.
In this group, maximizing the distance between the GCPs
and localizing them precisely on the PCF is essential. The
following enumeration lists the main potential sources of
errors:
• Camera calibration and image registration,
• Image compression,
• Camera exposure time,
• Training data generation and object shape detection,
• Location of the shape boundary on the vehicle body,
• Sensor synchronisation.
Wide angle lenses have the preferable focal length to cap-
ture a large area on ground. They are usually affected by
barrel distortion, which decreases the GSD with increasing
distance from the optical axis Oc . According to the drone
manufacturer, the camera in use performs the corrections
automatically. Every pixel deviation in the feature detection
and matching during the image registration process inevitably
leads to a deviation in rotation and GSD. Changing light
conditions and the slight movements of the hovering drone
affect the perception and thus influence the matching. Figure
5 depicts a typical example of rotation and scaling, recorded
over a period of one minute at 100 m altitude. While achiev-
ing robust results, some potential outliers with a magnitude
of approximately 0.1 % can be observed for the scaling
parameter, which translates into a deviation of up to 12 cm
at 100 m altitude. Filtering these variables was omitted to
examine the robustness of the image registration algorithms.
The images are compressed in two ways. First, the res-
olution is reduced by half for both axes. Second, storing
the images as JPEGs leads to lossy image compression. For
example, smooth transitions can be found, which reduce
the sharpness of edges. Another component is the camera
exposure time. With exposure time, motion blur is induced,
which can stretch the vehicle on the image or blurs the edges.
Therefore, short exposure times are preferable, but at the cost
of less light exposure.
A robust semantic segmentation is crucial for achieving
reliable results. Even though Mask-RCNN provides excellent
results, see Section IV-B, minor deviations of at least one
pixel can not be avoided. The deviations stem mainly from
the manual image labeling and limited training data.
With an radially increasing distance from Oc , the relief
displacement has a major influence on the results. The
problem in correcting the displacement of vehicles is rooted
in the complex shapes and varying heights. It is difficult to
determine which part of the car has been detected exactly,
even when inspected by a human: Assuming a flight altitude
of 100 m and a vehicle detected close to the image border,
e. g., a distance of 60 m to Oc , the displacement increases
with 0.6 cm per cm change in object height. Detecting a
feature of the vehicle at 30 cm height, instead of the body
bottom (Section III-D.2), yields an error of 9 cm.
The last two sources of error are only of concern when
comparing the results with a reference system. The mapping
of the PCF to the LTP is based on localizing the GCPs,
see Section III. Due to the limited resolution and image
compression, a mis-locatization of typically one pixel per
GCP in the PCF can be induced. Hence, all variables
concerning the mapping, namely the GSD S, orientation
offset δ and linear offset ∆, see Section III-D.1, are affected.
The resulting error depends on the distance between two
GCPs, hence |gi+1,L − gi,L | should be maximized.
Synchronisation between the two sensors is attained via
UTC time stamps. Since UTC time stamps can not be
associated to a certain frame for the drone in use, a LED
signal, triggered by the Pulse-per-second signal (PPS) from
a satellite navigation receiver, was recorded. This appears
to be the best solution, since the circuit delay within the
receiver and the LED rising time can be neglected. The first
video frame showing the illuminated LED is associated with
the corresponding UTC time stamp. Hence, the maximum
1
synchronisation error is limited here to fps = 20 ms. During
the experiments, the maximum speed was around 30 km/h
leading to a worst case error of 17 cm due to synchronisation.
B. Detection performance
A set of 50 images has been labeled for evaluation. Table
II depicts the Average Precision (AP) results according to
Weights Specialized General
Metric AP@0.5 AP@[0.5, 0.95] AP@0.5 AP@[0.5, 0.95]
50 m 1.00 0.89 1.00 0.84
75 m 1.00 0.89 1.00 0.82
100 m 1.00 0.91 1.00 0.86
TABLE II: AP evaluation for all altitudes and both training sets.
the PASCAL VOC (AP@IoU = 0.5) [30] and COCO
(AP@IoU[0.5:0.05:0.95]) [23] definitions, where the Inter-
section over Union threshold is abbreviated as IoU. The
detection is robust and the AP is similar for all three flight
heights. An AP@IoU = 0.5 of 1 exhibits a detection rate of
100 % for the evaluation images. That holds for all images
(see. Table I) detected with the specialized training weights.
For the general weights, the detection rate is 99.97 % w.r.t
to the images listed in Table I. This is reasonable, since the
environment of the test track does not exhibit structures to be
confused with a top-down view of a vehicle shape. It should
be noted, that the detection performance does not directly
reflect the accuracy of the position estimation, since the
bounding box is computed according to the most outer pixels
of the shape. Assuming the most outer pixels are detected
and they do match to the actual vehicle body border, the
position could still be computed correctly on a pixel level,
although the IoU is less than one.
C. Position estimation
This section is concluded with the experimental results.
Figure 6 depicts the graph for each flight altitude, both
training weights and the three main processing steps, where
depicts results for non registered images, for reg-
istered images, and for registered images with corrected
relief displacement.
Image registration is the key to obtain reasonable results.
The correction of the relief displacement improves the results
by 0.8 px on a weighted average4 . Note, that the impact of
the displacement is dependent on the distance R. Hence, data
sets recording vehicles at the image border benefit more.
As mentioned before, the relief displacement is reduced
with higher hovering altitudes. Remembering the issue re-
garding which vehicle part is actually corresponding to the
outer pixel detected (Section III), explains the best results in
pixel measure for an altitude of 100 m. However, expressing
the error in the metric units, the error is lowest at 50 m.
The experiments with the general training weights, which
are based on images recorded on public roads, perform at
weighted average only 0.3 px worse. This underlines the
suitability of the framework for applications on public roads.
Table III depicts a detailed comparison.
The mean error is 20 cm and 14 cm for a flight altitude
of 100 m and 50 m, respectively. Regardless of the training
weights, 90 % of all frames have an error of 7 px or less.
Summarizing this section and the experimental results
leads to the following conclusions: 1) A robust image reg-
istration is crucial for a good performance. Omitting the
4 weighted by the number of frames per height, see Section III
 Altitude 100 m 75 m 50 m
Weights Specialized General Specialized General Specialized General
Corrections reg reg+shift reg reg+shift reg reg+shift reg reg+shift reg reg+shift reg reg+shift
Median [px] 3.27 2.41 3.33 2.71 3.26 2.70 3.67 3.08 4.47 3.93 4.14 4.05
Mean [px] 3.87 2.95 3.96 3.27 3.75 2.99 4.09 3.34 4.53 3.98 4.47 4.26
90% [px] 7.39 6.01 7.52 6.50 6.98 5.28 7.42 5.80 7.04 6.08 7.75 7.12
99% [px] 11.19 8.88 11.33 9.58 9.67 8.08 9.60 8.27 9.97 8.10 10.81 10.05
99.9% [px] 11.85 9.72 12.23 12.07 11.15 9.63 11.07 10.04 10.83 8.95 12.89 13.27
Mean [m] 0.27 0.20 0.27 0.23 0.20 0.16 0.21 0.17 0.16 0.14 0.16 0.15

TABLE III: Accumulated frequency of the error: Depicted for all three altitudes, both training weights and corrections.

Specialized training weights General Training weights

4) Regarding real-world applications, the vehicle can at least
7 14 21 28 35 42 7 14 21 28 35 42
1
50m
1
50m
be associated to its lane as exemplified in Figure 1. Note,
that to some extend, the error values reported in Figure 6 and
Cumulative frequency

0.8 0.8
Table III can be assigned to the synchronisation and mapping
0.6 0.6
uncertainty, which is only of concern when benchmarking
0.4 0.4 two data sources. 5) The vehicle to lane association also
holds for the general training set, so that one can expect
0.2 0.2
similar results for public roads, given an approriate training
0
2 4 6 8 10 12
0
2 4 6 8 10 12 data set.
10 20 31 41 52 62 10 20 31 41 52 62
1
75m
1
75m V. CONCLUSIONS
Cumulative frequency

0.8 0.8 Vehicle detection by means of UAV imagery is an attrac-

0.6 0.6
tive option to generate data sets with relatively low effort.
This paper describes an approach based on deep neural
0.4 0.4
network object detection and automated image registration
0.2 0.2 techniques with state-of-the-art algorithms. A procedure to
reduce the impact of relief displacement originated by the
0 0
2 4 6 8 10 12 2 4 6 8 10 12 perspective projection of vertical images is described as well.
14 28 41 55 69 83 14 28 41 55 69 83
1
100m
1
100m
Additionally, an overview of the potential sources of errors,
and how to minimize their impact, is given.
Cumulative frequency

0.8
0.6
0.4
0.2
0
2 4 6 8 10 12
Error in [cm] and [px]
reg+shift
Fig. 6: Cumulative frequency diagrams: The left column depicts
the results for the specialized training weights, the right column for
the general training weights. The error above each plot is depicted
in [cm], and in [px] below each plot.
effect of the relief displacement yields larger errors, when
hovering above the region of interest is not feasable and
objects are detected throughout the complete image frame.
2) Considering the pixelwise results, similar performance
can be observed for all three altitudes, which proofs that
data can be obtained from different flight heights by a single
Mask-RCNN network. This advantage can also be helpful for
different object sizes. 3) The best results in metric values are
retrieved at lower altitudes. Alternatively, in order to capture
a larger surface area, one can record at higher altitudes,
increase the resolution and crop the image if necessary.
0.8
The estimated vehicle position is compared to a reference
0.6
system. Recording the data on a test track with consistent
0.4 conditions ensures meaningful results. It is shown, that with-
out applying any time-smoothing techniques, the position
0.2
can be estimated in a reliable manner. The mean error is
0
2 4 6 8 10 12 20 cm and 14 cm for a flight altitude of 100 m and 50 m,
Error in [cm] and [px] respectively. Furthermore, 90 % of the 53 855 independently
reg raw evaluated frames have an error of 7 px or less. To highlight
the generalization capabilities, the experiments were anal-
ysed for two training data sets. One is a specialized data set,
while the second being recorded on public roads. Both sets
perform at a similar level. This allows the framework to be
used for a wide range of applications. Interested readers are
referred to the repository [31], where the code, label data
and exmaple videos are made publicy available.
ACKNOWLEDGMENT
The authors acknowledge the financial support by the
Federal Ministry of Education and Research of Germany
(BMBF) in the framework of FH-Impuls (project number
03FH7I02IA). The authors thank the AUDI AG department
for Testing Total Vehicle for supporting this work.
 R EFERENCES [21] P. Torr and A. Zisserman, “MLESAC: A New Robust Estimator with
Application to Estimating Image Geometry,” Computer Vision and
[1] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets Robotics: Image Understanding, June 2000.
The KITTI Dataset,” International Journal of Robotics Research [22] Waleed Abdulla, “Mask R-CNN for object detection and instance
(IJRR), 2013. segmentation on Keras and TensorFlow,” 2017. [Online]. Available:
[2] “Waymo Open Dataset: An autonomous driving dataset,” 2019. https://github.com/matterport/Mask RCNN
[Online]. Available: https://www.waymo.com/open [23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in
[3] J. Geyer, Y. Kassahun, M. Mahmudi, X. Ricou, R. Durgesh, A. S.
Context,” in Computer Vision – ECCV, 2014.
Chung, L. Hauswald, V. H. Pham, M. Mhlegg, S. Dorn, T. Fernandez,
[24] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-
M. Jnicke, S. Mirashi, C. Savani, M. Sturm, O. Vorobiov, and
Time Object Detection with Region Proposal Networks,” in Advances
P. Schuberth, “A2D2: AEV Autonomous Driving Dataset,” 2019.
in Neural Information Processing Systems 28, 2015.
[Online]. Available: http://www.a2d2.audi
[25] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
[4] E. Sánchez Morales, F. Kruber, M. Botsch, B. Huber, and A. Garcı́a
for semantic segmentation,” in IEEE Conference on Computer Vision
Higuera, “Accuracy Characterization of the Vehicle State Estimation
and Pattern Recognition (CVPR), 2015.
from Aerial Imagery,” in IEEE Intelligent Vehicles Symposium (IV),
[26] T. Lin, P. Dollr, R. Girshick, K. He, B. Hariharan, and S. Belongie,
2020.
“Feature Pyramid Networks for Object Detection,” in IEEE Conference
[5] E. Sánchez Morales, R. Membarth, A. Gaull, P. Slusallek, T. Dirn- on Computer Vision and Pattern Recognition (CVPR), 2017.
dorfer, A. Kammenhuber, C. Lauer, and M. Botsch, “Parallel Multi- [27] G. Bradski, “The OpenCV Library,” Dr. Dobb’s Journal of Software
Hypothesis Algorithm for Criticality Estimation in Traffic and Colli- Tools, 2000.
sion Avoidance,” in IEEE Intelligent Vehicles Symposium (IV), 2019. [28] Lillesand, Kiefer, and Chipman, Remote Sensing and Image Interpre-
[6] P. Nadarajan, M. Botsch, and S. Sardina, “Machine Learning Archi- tation, 2003, vol. 5.
tectures for the Estimation of Predicted Occupancy Grids in Road [29] Verband der TV e.V., “Merkblatt 751,” 2008.
Traffic,” Journal of Advances in Information Technology, 2018. [30] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and
[7] L. Neubert, “Statistische Analyse von Verkehrsdaten und die Model- A. Zisserman, “The Pascal Visual Object Classes (VOC) Challenge,”
lierung von Verkehrsfluss mittels zellularer Automaten,” Ph.D. disser- International Journal of Computer Vision, 2010.
tation, Universität Duisburg, 2000. [31] F. Kruber and E. Sánchez Morales, “Vehicle Detection and
[8] F. Kruber, J. Wurst, S. Chakraborty, and M. Botsch, “Highway traffic State Estimation with Aerial Imagery.” [Online]. Available:
data: macroscopic, microscopic and criticality analysis for capturing https://github.com/fkthi
relevant traffic scenarios and traffic modeling based on the highD
data set,” 2019. [Online]. Available: https://arxiv.org/abs/1903.04249
[9] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online
and realtime tracking,” in IEEE International Conference on Image
Processing (ICIP), 2016.
[10] C. Kyrkou, G. Plastiras, T. Theocharides, S. I. Venieris, and C. Bouga-
nis, “DroNet: Efficient convolutional neural network detector for
real-time UAV applications,” in Design, Automation Test in Europe
Conference Exhibition (DATE), 2018.
[11] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look
Once: Unified, Real-Time Object Detection,” in IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2016.
[12] Q. Li, L. Mou, Q. Xu, Y. Zhang, and X. X. Zhu, “R3 -
Net: A Deep Network for Multi-oriented Vehicle Detection in
Aerial Images and Videos,” CoRR, 2018. [Online]. Available:
http://arxiv.org/abs/1808.05560
[13] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,”
in Proceedings of the International Conference on Computer Vision
(ICCV), 2017.
[14] L. Mou and X. X. Zhu, “Vehicle Instance Segmentation From Aerial
Image and Video Using a Multitask Learning Residual Fully Con-
volutional Network,” IEEE Transactions on Geoscience and Remote
Sensing, 2018.
[15] G. Guido, V. Gallelli, D. Rogano, and A. Vitale, “Evaluating the
accuracy of vehicle tracking data obtained from Unmanned Aerial
Vehicles,” International Journal of Transportation Science and Tech-
nology, 2016.
[16] R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The highD
Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on Ger-
man Highways for Validation of Highly Automated Driving Systems,”
in 21st International Conference on Intelligent Transportation Systems
(ITSC), 2018.
[17] J. Bock, R. Krajewski, T. Moers, L. Vater, S. Runde, and L. Eckstein,
“The inD Dataset: A Drone Dataset of Naturalistic Vehicle
Trajectories at German Intersections,” 2019. [Online]. Available:
https://arxiv.org/abs/1911.07602
[18] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann,
J. Kümmerle, H. Königshof, C. Stiller, A. de La Fortelle, and
M. Tomizuka, “INTERACTION Dataset: An INTERnational, Ad-
versarial and Cooperative moTION Dataset in Interactive Driving
Scenarios with Semantic Maps,” arXiv:1910.03088, 2019.
[19] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, “Human
Trajectory Prediction In Crowded Scenes,” in European Conference
on Computer Vision (ECCV), 2016.
[20] H. Bay, T. Tuytelaars, and L. Van Gool, “SURF: Speeded Up Robust
Features,” in Computer Vision – ECCV, 2019.

View publication stats