Cloud Detection System for UAV Sense and Avoid:
Cloud Distance Estimation using Triangulation
1st Adrian Dudek
Institute of Flight Systems
University of the Bundeswehr Munich
Neubiberg, Germany
adrian.dudek@unibw.de
Abstract—Sense and Avoid is gaining importance in the
integration of unmanned aerial vehicles (UAVs) into civil
airspace. In order to be seen by other air traffic participants
operating under visual flight rules (VFR), own visibility must be
granted. This includes the observance of legal minimum cloud
distances. An airborne electro-optical cloud detection and
avoidance system could cope with this task on unmanned
platforms. In this paper we propose an approach for cloud
distance estimation using computer vision algorithms. In
addition to the functional architecture, results from a simulation
environment are shown and the demands for future
developments are addressed.
Keywords—UAV Sense and Avoid, cloud distance estimation
I. INTRODUCTION
The integration of unmanned aircraft into civil airspace
remains a challenge. While systems such as the Traffic
Collision Avoidance System (TCAS) are used to avoid
collisions in controlled airspaces in which mainly air traffic
participants operate according to instrument flight rules (IFR),
the pilots flying in uncontrolled airspaces according to visual
flight rules (VFR) are responsible for avoiding other aircrafts
and ensuring their own visibility. Depending on the airspace
used, certain minimum cloud distances must therefore be
respected. With regard to our application of unmanned air
cargo transportation, it is expected that it will take place
primarily in the lower airspace. Thus, a Sense & Avoid system
that is able to detect and avoid clouds is essential as on-board
equipment. Apart from ensuring own visibility, there are
further reasons not to enter clouds. On the one hand, there is
an increased risk of turbulence in the vicinity of clouds caused
by convection. On the other hand, at low temperatures,
especially near clouds, icing can occur and influence the
aerodynamic structure.
The focus of our previous investigations, described in [1],
was on segmentation of clouds in a simulation environment.
As a second important step the results of cloud segmentation
are used to perform an approach of cloud distance estimation,
which is examined in the following paper. First, state of the
art cloud detection research projects are presented followed by
an explanation of the 3D reconstruction method used. The
procedure for estimating cloud distance is explained step by
step beginning with cloud segmentation methods, shown in
[1]. Moreover, results are shown and compared with ground
truth data from the simulation. Finally, a possibility is shown
how cloud ground truth data can be collected in real flight
images to validate the determined cloud distance.
The KoKo II project was funded by the German federal aviation research
programme (LuFo V, 3rd call).
978-1-7281-9825-5/20/$31.00 ©2020 IEEE
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 28,2020 at 12:28:08 UTC from IEEE Xplore. Restrictions apply.
2nd Peter Stütz
Institute of Flight Systems
University of the Bundeswehr Munich
Neubiberg, Germany
peter.stuetz@unibw.de
II. STATE OF THE ART
A. Cloud detection on-board UAV
Cloud detection as part of UAV Sense & Avoid is an
upcoming topic in the integration of unmanned systems in
European airspace. However, research on cloud detection has
mainly focused on ground-based or space-based applications
for current weather determination in meteorology. As shown
in the previous chapter, an on-board cloud distance
estimation system is indispensable in airspaces with traffic
under visual flight rules.
An early publication dealing with UAV-based cloud
detection has already shown that it is possible to determine
the distance of clouds in flight altitude using electro-optical
systems. In their work the authors use feature point ranging
in combination with an image-based classifier that classifies
local image patches as ground, cloud or sky [2]. In
comparison to [2] we try to separate cloud objects from the
background using colour-based segmentation first and
subsequently determine the distance in these areas with the
presented method.
Previous research work at our Institute of Flight Systems
has already focused on determining the degree of coverage
and estimating vertical cloud distance using a two-view
triangulation approach in order to avoid dangers during
ascent and descent due to icing and turbulence. However, in
our recent project the preservation of visibility is of highest
priority. Therefore, we propose to investigate whether this
method using 3D reconstruction from two views is also
suitable for applications with perspective in flight direction
for a short-term avoidance of clouds in flight altitude.
B. Monocular cloud distance estimation using two-view
triangulation
The advantage of using monocular approaches compared
to stereoscopic ones is that the baseline necessary for
triangulation is less limited by the aircrafts design. By using
the monocular two-view triangulation, only one camera is
needed. The collection of images from different perspectives
is done at different times. However, in case of rapid changes
of the cloud structure or shifting due to wind, the estimation
result will be negatively affected. Therefore, the time interval
between the images needs to be chosen according to the flight
speed [3].
The purpose of cloud detection as part of a Sense & Avoid
system requires the application of the approach for cloud
distance estimation in perspective of flight direction, which
brings along the challenge of optical flow divergence besides
varying backgrounds (sky, ground and horizon). When
observing clouds in upward or downward perspective, the
optical flow vectors maintain one direction throughout the
image and there are barely any differences in the flow vector
length depending on the image position for objects at equal
distances. In contrast, when an object is approached in
direction of flight (Fig. 1, left) the optical flow vectors start at
a point called focus of expansion and the flow vectors of
objects of equal distance become larger the further away they
are from this point in the image (Fig. 1, right) [4]. It is thus
expected that the accuracy of the distance determination will
depend on the cloud position in the image.
Fig. 1. Apparent growth of a circle when approaching a sphere (left)
and corresponding radial optical flow vectors (right) [4]
III. PROPOSED CLOUD DISTANCE ESTIMATION APPROACH
The unaccelerated horizontal flight without wind
influence is examined in the following. First a sensor image is
taken from a simulation environment based on the Presagis
toolkit including aircraft state data and camera orientation
(Fig. 2). Before starting with cloud distance estimation, cloud
segmentation methods are applied, such as described in [1].
There we have shown that colour-based segmentation
methods are highly dependent on lighting conditions,
visibility and ground texture, so false detections are probable.
Image areas that were classified as clouds serve as regions of
interest while others are not considered. As a next step a Shi-
Thomasi feature detector is applied on these regions [5]. It is
expected that more features will be recognized in the cloud
edge areas, because the cloud structure is usually striking in
the cloud edge area, while the middle of the cloud can have
rather homogeneous structures [2]. After a certain flight
distance the second input image is taken and the aircraft state
data and camera orientation and position is known. On the one
hand, the distance lying between the two images should not be
too small, otherwise the optical flow vectors are too small for
robust triangulation. On the other hand, the time between the
images should not be too long, otherwise tracking will be more
difficult if the characteristics of a cloud change substantially
[6].
In the second input image features detected in the first
image are tracked using the Lucas-Kanade algorithm [7]. Here
features that were detected in the first image but could not be
tracked in the second image are eliminated. In order to reduce
the number of erroneously tracked features, the Random
Sample Consensus (RANSAC) algorithm is used. This
procedure is able to eliminate large errors from a data set, even
if the proportion of errors is relatively large [8]. In this use
case, tracked features that do not correspond to the main
direction of movement can be removed. Subsequently,
triangulation is performed using the camera matrix P (1) as
input.
fx s cx
 P = K [R | t] = [ 0 fy cy ] [R | t]
0 0 1
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 28,2020 at 12:28:08 UTC from IEEE Xplore. Restrictions apply.
K is the intrinsic matrix containing camera characteristics
such as the focal length ( 𝑓𝑥 , 𝑓𝑦 ) resulting from image
resolution and field of view. It also takes into account the
optical centre (𝑐𝑥 , 𝑐𝑦 ) and the axis skew (s). The extrinsic
matrix [𝑅|𝑡] contains rotation and translation parameters of
the camera [9]. The result of the triangulation are 3D points
that include the required depth information. With this distance
information, statements can be made about the cloud structure
on the front and the averaged oblique distance to the cloud can
be given if several features within a cloud contour could be
detected, tracked and triangulated. As a last step the 3D points
are transformed into the WGS84 coordinate system. Thus,
features with a very low altitude can be eliminated with a
known terrain height, if structures on the ground have been
classified as potential clouds by the cloud segmentation.
1st input
image
Cloud
segmentation
Feature
detection
2nd input Feature Remove
image tracking untracked
RANSAC Remove
algorithm outliers
Triangulation
Absolute
feature
positions
Cloud
distance
Fig. 2. Cloud distance estimation approach
IV. FIRST RESULTS AND VALIDATION
The following discussion presents the results obtained on
simple, single cloud objects to obtain basic performance
measures. These results form the basis for later investigations
with more complex cloud formations. First, the distance
determination in unaccelerated horizontal straight flight
without a visual simulation of atmospheric effects and terrain
is considered. Due to the optical flow divergence, it is
expected that the determination of distances in clouds near the
focus of expansion will provide less accurate values due to the
small change in feature position. In the following, clouds are
systematically placed in a plane frontal to the UAV using
Presagis simulation environment (Fig. 3). According to the
proposed procedure, cloud segmentation is applied first. In
this case, a brightness filter with an adaptive threshold allows
the clouds to be separated from the black background very
easily (Fig. 4).

 Fig. 3. Systematically placed Fig. 4. Segmentation result using
clouds without simulated terrain brightness threshold
and atmospheric effects
Then features are detected in these segmented areas (Fig.
5, green) and tracked in the second input image (Fig. 5, blue).
This shows the expected different change of feature positions
depending on the position in the image. After eliminating the
untracked features and the features that do not correspond to
the flow direction, triangulation is applied to estimate the
distance. All feature distance values within a cloud contour are
averaged and displayed in the cloud object (Fig. 6, orange).
The actual distance to the cloud centre is projected in green
into the corresponding cloud object (Fig. 6, green).
Since the detected features are located on the cloud
surface, but according to the current state of the simulation
environment only the distance to the cloud centre is known,
the two distance values can be compared only conditionally,
because the cloud surface is closer than the cloud centre
position from a UAV point of view. However, comparing the
distance values allows to determine if the magnitude is
correct. During the flight towards the cloud formation shown
in Fig. 3, the distance values of the outer clouds are constantly
within the range of the actual distance with a relatively small
deviation (Fig. 6). For the cloud in the centre of the image,
distances are estimated that oscillate more around the value of
the actual distance (Fig. 6). However, the estimation of
distances becomes better as the cloud occupies increasing
areas of the image and the vectors of the optical flow become
longer.
Fig. 5. Detected (green) and tracked (blue) features
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 28,2020 at 12:28:08 UTC from IEEE Xplore. Restrictions apply.
Fig. 6. Averaged cloud distance estimation results (orange) and actual
distance to the cloud center (green)
The fluctuation of the distance estimation of cloud objects
near the focus of expansion is visualized in the following
example. As shown in Fig. 2 as the last step, absolute positions
are calculated after triangulation of the features. These
absolute positions can then be plotted on a map. For this task
we use ArcGIS Runtime SDK. If three clouds are placed in a
plane of the same distance in flight altitude and are detected
by the proposed method (Fig. 7), the feature positions shown
in Fig. 8 are obtained. The true cloud centre and the bounding
boxes of the clouds are shown in black. The cloud features
whose positions could be determined are shown as a "+"
symbol. Depending on their estimated altitude, they are
colour-coded.
The absolute altitude of a feature can be specified by using
the previously estimated slant distance and the feature position
in the image. When operating under visual flight rules, a
minimum horizontal distance of 1.5km and a minimum
vertical distance of 1000ft apply in most airspaces. Therefore,
features whose determined altitude falls below the minimum
vertical distance of 1000ft are displayed in red, while features
which are more than 1000ft lower than the UAV are displayed
in green and those which are more than 1000ft above the UAV
flight altitude are displayed in blue. The current UAV position
is shown in orange (Fig. 8).
Fig. 7. Clouds placed in flight altitude with detected (green) and
tracked (blue) features
 symbol in three colours depending on altitude. The flight
altitude of the UAV is 1000m in this scenario.

Fig. 11. Second input image with detected (green) and tracked (blue)
features

Fig. 8. Visualization of estimated cloud feature positions: cloud

ground truth (black), features below (green), above (blue) and within
(red) legal vertical cloud distance of 1000 ft

The clouds shown in Fig. 7 have their centre at flight

altitude and have a maximum vertical extent of 1000m, so
most features would have to be drawn in red, but some also in
green and blue. It is noticeable that the features of the outer
clouds are too far out compared to the ground truth. The cloud
features of the cloud in the middle scatter very much (Fig. 8),
which explains the fluctuating average distance of this cloud.
When approaching this cloud, the features are more
concentrated in the area of the bounding box and the features
of the outer clouds move towards the actual cloud centre. As
expected, the closer the UAV gets to a cloud object, the better
the distance estimation. The fact that a cloud approached
head-on by this method provides less accurate distance values
compared to clouds in other parts of the image is problematic, Fig. 12. Averaged cloud distance estimation result (orange), actual
distance to cloud center (green) and segmented contour (black)
because the cloud in the centre of the image has the highest
priority when aiming to avoid clouds.
The next scenario shows a cloud formation with a terrain
model from our simulation environment (Fig. 9). The result of
the colour-based segmentation approach introduced in [1] is
shown in Fig. 10.

Fig. 9. Cloud scene in simulation Fig. 10. Segmentation result

environment

Fig. 11 shows the second input image containing detected Fig. 13. Visualization of estimated cloud feature positions: cloud ground
features (green) and tracked features (blue). Fig. 12 illustrates truth (black), features below (green), above (blue) and within (red) legal
vertical cloud distance of 1000 ft
the estimated averaged cloud distance (orange) and the actual
distance to the cloud centre (green) of each segmented cloud
The cloud centres of the two upper clouds are located at an
object. The 2D map display (Fig. 13) shows the true bounding
altitude of 3000m and have a maximum vertical extension of
boxes and centre positions of the clouds in black, the UAV
1000m. The estimated distance result of these clouds is close
position in orange, and the cloud feature positions as a "+"
to the actual distance from the cloud centre (Fig. 12). In Fig.13

Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 28,2020 at 12:28:08 UTC from IEEE Xplore. Restrictions apply.
it can be seen that some absolute positions of the cloud
features are within the range of the bounding box, even if
some positions were estimated too far out. Both clouds are
placed well above the minimum vertical distance, so all
associated features should be marked blue, which is the case
(Fig. 13). The two lower and front clouds are estimated too far
away (Fig. 12). The distance values should tend to be slightly
less than the distance to the cloud centre, since the features
whose distance is determined lie on the cloud surface.
However, the cloud on the lower left is estimated to be much
too far away. If the feature positions of this cloud are analysed,
some features are scattered upwards along a line (Fig. 13). The
features that are very far away from the cloud bounding box
are all marked green, meaning they are estimated to be below
about 700m. Some tests have shown that this behaviour only
occurs in clouds with the ground as background. The more
sensitive the cloud segmentation was chosen, the more the
distance estimation was worsened towards too large values. It
turns out that although clouds have prominent structures in the
edge areas, which are more likely to be detected by feature
detectors than more homogeneous areas in the middle of the
cloud [2] (Fig. 11), these edge areas also tend to be more
transparent, so that features on the ground are also detected in
this area, which means that the determined distance for the
entire cloud object is increased. This behaviour can be seen in
the lower left cloud. Some of the triangulated features do not
belong to the cloud, but to the ground behind the cloud. Since
the height of the features is also determined, ground features
could be eliminated if the terrain height at this position is
known. An alternative would be to shrink segmented cloud
objects artificially, for example by using the morphological
operation erosion or by rescaling the segmented cloud
contours, so that transparent cloud edge areas are not
segmented. The cloud centre of the cloud on the lower left is
placed at 600m height and has a vertical extension of 300m.
So most of the features should be marked green and the rest
red. The cloud centre of the cloud on the lower right is placed
in flight altitude with a vertical extension of 1000m that means
features would have to be classified in green, red and blue.
Both clouds show the expected colour distribution of the
features.
V. CONCLUSION AND FUTURE WORK
The presented results have shown that an estimation of
cloud distance in perspective of the flight direction for short-
term cloud avoidance is possible with the proposed approach.
Issues with distance estimation arise near the focus of
expansion and in transparent cloud areas when the distance to
features on the ground is determined. The distance
information obtained is valuable in several aspects. On the one
hand, avoidance manoeuvres can be initiated when
approaching a cloud object. On the other hand, the determined
dimension of the cloud can be used to classify the cloud type
and draw conclusions about convective activity in the cloud
area. However, the reliability of the distance estimation
approach strongly depends on the background respectively the
terrain characteristics. In addition, more complex cloud
situations and also hazy visibility conditions lead to increasing
challenge of separation between individual cloud objects, for
example by colour segmentation. In these cases, it is suggested
to triangulate features in the large cloud areas and then cluster
them into groups of similar distance. It should be mentioned
that flying under or over a cloud layer is preferable to avoiding
individual clouds in high cloud cover. Based on the gained
knowledge, following improvements are proposed:
Authorized licensed use limited to: UNIVERSIDAD DE OVIEDO. Downloaded on December 28,2020 at 12:28:08 UTC from IEEE Xplore. Restrictions apply.
 Elimination of erroneously detected ground objects
by known terrain height and estimated altitude of the
features or adjustment of cloud segmentation
 Investigations to improve the determination of
distances in the area of focus of expansion
 Clustering of cloud features in groups of similar
distance for separation of clouds at high coverage.
 Utilization of multiple perspectives, meaning more
than two input images, for distance estimation.
Apart from improving the presented method we want to
evaluate different depth determination methods of image
processing in order to apply them in the future in a context
sensitive adaptive system during flight. This adaptive system
will be extended by further cloud segmentation methods based
on deep learning, which are described in [10] and [11], for
example. In addition, investigation of real-time cloud
detection algorithms in real flight plays a central role in our
future research. Compared to simulation, the collection of
ground truth data in the atmosphere is more complicated. One
possibility of determining the true cloud positions is briefly
explained in the following. The satellite data portal
Copernicus Open Access Hub of the European Space Agency
(ESA) provides images taken by the satellites Sentinel-2A and
Sentinel-2B. Every five days one of the satellites flies over a
large area south of Munich and records this area. Exactly at
this moment we try to record airborne clouds and determine
the distance to them. Afterwards the actual cloud position
provided by satellite data can be compared with the cloud
position determined by computer vision methods. For these
flights we mount an action camera, which records the GPS
position, at the wing of a general aviation aircraft.
REFERENCES
[1] A. Dudek, F. Funk, M. Russ, and P. Stuetz, “Cloud Detection System
for UAV Sense and Avoid : First Results of Cloud Segmentation in a
Simulation Environment,” in IEEE International Workshop on
Metrology for AeroSpace, 2019.
[2] H. Nguyen et al., “EO/IR Due Regard Capability for UAS Based on
Intelligent Cloud Detection and Avoidance,” in AIAA
Infotech@Aerospace, 2010
[3] F. Funk and P. Stütz, “A Passive Cloud Detection System for UAV :
Concept and First Results,” in International Symposium on Enhanced
Solutions for Aircraft and Vehicle Surveillance Applications (ESAVS),
2016.
[4] D. A. Forsyth and J. Ponce, Computer Vision - A Modern Approach,
2nd ed. Edinburgh: Pearson Education, p. 345, 2012.
[5] J. S. J. Shi and C. Thomasi, “Good features to track,” in IEEE
Computer Society Conference on Computer Vision and Pattern
Recognition, 1994.
[6] F. Funk and P. Stutz, “A passive cloud detection system for UAV:
Analysis of issues, impacts and solutions,” 5th IEEE Int. Work. Metrol.
AeroSpace, Metroaerosp. 2018 - Proc., pp. 236–241, 2018.
[7] J.-Y. Bouguet, “Pyramidal Implementation of the Lucas Kanade
Feature Tracker - Description of the algorithm,” Intel Corporation -
Microprocessor Research Labs. 1999.
[8] M. A. Fischler and R. C. Bolles, “Random sample consensus,”
Commun. ACM., vol. 24, no. 6, pp. 381–395, 1981
[9] R. Szeliski, “Computer Vision: Algorithms and Applications.”
Springer-Verlag, London, pp. 46–47, 2011.
[10] M. Shi, F. Xie, Y. Zi and J. Yin, "Cloud detection of remote sensing
images by deep learning," 2016 IEEE International Geoscience and
Remote Sensing Symposium (IGARSS), pp. 701-704, 2016.
[11] S. Dev, A. Nautiyal, Y. H. Lee and S. Winkler, "CloudSegNet: A Deep
Network for Nychthemeron Cloud Image Segmentation," in IEEE
Geoscience and Remote Sensing Letters, vol. 16, no. 12, pp. 1814-
1818, 2019.