Professional Documents
Culture Documents
Ares(2017)529646 - 31/01/2017
Author: Yoko Watanabe, Balint Vanek, Akos Zarandy, Antal Hiba, Shinji Suzuki,
Ryota Mori
Publish date: 31/01/2017
1
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Document Information Sheet
General information
Deliverable no. 4.3
EU EU-H2020 GA-690811
Grant no.
Japan NEDO GA- 062800
WP no. 4.2
Revision 8
Approvals
Authors EU Yoko Watanabe (ONERA), Balint Vanek (SZTAKI), Akos Zarandy
(SZTAKI), Antal Hiba (SZTAKI)
WP leader UTOKYO
2
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
History
Revision Date Modifications Authors
1 13/12/2016 Creation Yoko Watanabe
2 21/12/2016 SZTAKI 1st input Antal Hiba
1st update on the vision-based navigation Yoko Watanabe
3 22/12/2016
approach (ONERA)
2nd update on the vision-based navigation Yoko Watanabe
approach (ONERA)
4 09/01/2017
Integration of UTOKYO’s input on the
trahectory optimization
5 16/01/2017 SZTAKI 2nd input Antal Hiba
3rd update on the vision-based navigation Yoko Watanabe
6 24/01/2017 approach (ONERA)
3
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Table of Contents
Introduction ............................................................................................................................... 6
1. VISION project overview ............................................................................................... 6
2. WP4.2 overview............................................................................................................. 7
3. Objective of the document............................................................................................ 8
Preliminary design of vision-based navigation ........................................................................ 9
1. Sensor failure scenarios................................................................................................. 9
a. GNSS-based approach .................................................................................................... 9
b. ILS-based approach ...................................................................................................... 10
2. Reference coordinate frames and transformations................................................... 11
a. Inertial frame: FI........................................................................................................... 11
b. ECEF coordinate frame: FECEF ..................................................................................... 11
c. Geodetic coordinate frame: FGEO ............................................................................... 11
d. Local NED and ENU frames: FNED, FENU..................................................................... 12
e. Runway fixed frame: FRWY .......................................................................................... 13
f. Vehicle carried frame: FV ............................................................................................. 13
g. Aircraft body frame: FB ................................................................................................ 13
h. Camera frame and pixel-coordinates on an image: FC, Fi ........................................... 14
3. The aircraft kinematic model ...................................................................................... 15
4. Navigation sensors ...................................................................................................... 16
a. AHRS (IMU + Magnetometer) ...................................................................................... 16
b. GNSS/SBAS ................................................................................................................... 16
c. Baro-altimeter .............................................................................................................. 17
d. Inclinometer ................................................................................................................. 17
e. Vision systems .............................................................................................................. 17
f. ILS ................................................................................................................................. 19
a. Navigation performance criteria .................................................................................. 20
b. Integrity monitoring ..................................................................................................... 21
6. Preliminary design of vision-aided navigation system .............................................. 27
a. Estimator design........................................................................................................... 27
b. Fault detection and estimation (FDE) .......................................................................... 29
7. Validation plan............................................................................................................. 30
a. Validation of the image processing algorithms............................................................ 30
b. Validation of the estimation algorithms ...................................................................... 30
c. Plan for 1st flight test .................................................................................................... 30
d. Preliminary plan for further flight tests ....................................................................... 31
8. Others........................................................................................................................... 31
Preliminary design of vision-based guidance ......................................................................... 32
1. Trajectory optimization for small deviation from the nominal glide path ............... 32
a. Trajectory optimization................................................................................................ 32
b. Decision making ........................................................................................................... 32
c. Validation plan ............................................................................................................. 32
2. Image-based obstacle avoidance approach ............................................................... 33
a. Obstacle detection and possible avoidance maneuvers.............................................. 33
b. Detection and tracking ................................................................................................. 33
4
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
c. Decision based on tracks.............................................................................................. 33
Terminology ............................................................................................................................. 35
References................................................................................................................................ 36
5
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Introduction
1. VISION project overview
To contribute towards the global goal of the aircraft accident rate reduction, this EU-Japan
collaborative research project called VISION (Validation of Integrated Safety-enhanced Intelligent
flight cONtrol) has the objectives of investigating, developing, and above all, validating advanced
aircraft Guidance, Navigation and Control (GN&C) solutions that can automatically detect and
overcome some critical flight situations. The focus of this project is on the final approach phase
where more than half of the fatal accidents have occurred in the last decade. The VISION project
tackles the following two different types of fault scenarios, which cover a dominant part of the causal
factors of the fatal aircraft accidents in the world.
• Flight control performance recovery from
o Actuator faults/failures (jamming, authority deterioration, etc.)
o Sensor failures (lack of airspeed etc.)
• Navigation and guidance performance recovery from
o Sensor failures (lack of SBAS, lack of ILS, etc.)
o Proximity of unexpected obstacles
These two fault types have completely different natures and so require different approaches. While
the first recovery scenario set will be achieved by applying fault detection and fault tolerant control
techniques, the second one will be achieved by proposing precision navigation and approach
guidance systems using new onboard sensing technologies. Flight validations of the developed
techniques and methods will be performed on JAXA’s MuPAL-alpha experimental aircraft in Japan for
the first scenario and on USOL K-50 UAV platform in Europe for the second scenario.
Figure 1 illustrates how the European and Japanese partners from academia, national research
institutes and industries will make synergetic and complementary efforts to achieve these ambitious
scientific objectives within this VISION project, in order to improve global aviation safety.
6
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
2. WP4.2 overview
WP4 of VISION project is dedicated to the second type of failure scenario, “Navigation and Guidance
performance recovery.” WP4.2 aims at developing navigation and guidance algorithms which use
vision information, provided by the systems developed in WP 4.1, for the purpose of maintaining
flight safety and navigability in the fault scenarios defined in WP 2.2. The objectives of this work-
package are the following:
• To specify the vision information and its performance criteria to be fed back into the aircraft
navigation and guidance system for each fault scenario,
• To develop vision-based navigation algorithms which estimate the aircraft flight path relative
to a runway, from the specified vision information in the navigation data degradation/failure
scenario,
• To develop vision-based guidance algorithms which modify the aircraft flight path in order to
ensure flight safety upon the detection of unexpected obstacles on the current planned path,
and
• To adapt and improve the performance of the developed systems through analysing results
of simulations and intermediate flight tests.
7
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
3. Objective of the document
WP4.2 has been active since M4 (06/2016) and continues until M30 (08/2018). The first six month of
this work-package 4.2 was devoted to
• Investigation in existing approaches for vision-based navigation and guidance for aircraft final
approach and landing,
• Study in navigation performance criteria and integrity monitoring algorithms, especially for
GNSS-based approach, and
• Preliminary design of vision-based navigation and guidance for specific fault scenarios
defined in WP2.2.
This document provides current results of the above listed tasks, focusing on descriptions of
preliminary design of vision-based navigation (in case of sensor failure) and guidance (in case of
obstacle detection) systems, which are to be evaluated with data recorded during the first flight test
campaign.
8
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Preliminary design of vision-based navigation
1. Sensor failure scenarios
This section recalls the navigation sensor failure scenarios, defined in WP2.2 and provided in D2.1
“Fault scenario descriptions,” for which we propose a vision-based navigation system to mitigate a
critical situation. Two different approach procedures are considered in this VISION project; the GNSS-
based LPV approach (APV-I or SBAS CAT-I) and the classical ILS approach. The interest of using
onboard vision in case of failure for both scenarios is to allow the pilot to continue the approach until
a decision height corresponding to APV-I or SBAS CAT-I instead of triggering a go-around (or missed
approach) procedure. VISION system only addresses the final segment approach, supposing that an
accurate relative position with respect to a runway is available on the initial segment for initializing
the visual tracker.
a. GNSS-based approach
In the first set of the sensor failure scenarios, it is supposed that the aircraft applies the GNSS-based
LPV approach procedure. In this procedure, horizontal and vertical approach guidance uses the GNSS
positioning based on GPS signals and the SBAS augmentation [dgac-2011].
The following four GNSS failures were listed in D2.1.
1) Degradation of GPS signals due to a reduced number of visible satellites with SBAS
augmentation
2) Lack of SBAS signals with GPS signals fully available
3) Degradation of GPS signals due to ionosphere interference with SBAS augmentation
4) Lack of SBAS signals and GPS signals due to jamming
We consider that the GNSS/SBAS functions correctly at the beginning of the final approach phase and
then the failure occurs before the aircraft reaches at DA. GNSS-based system causes positioning
errors with various reasons, such as ionosphere interference, lower atmosphere, multi-path effect,
and the satellite constellation. SBAS system provides the augmentation information which improves
the positioning accuracy.
9
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
During the last WP4 technical meeting (held in Madrid in 10/2016, in conjunction with PM1), the
WP4 partners agreed on prioritizing the scenario 2 (lack of SBAS with GPS) then 4 (lack of SBAS and
GPS) over the other two. It was also re-confirmed the need of a supervision loop which provides
continuously a Horizontal and Vertical Protection Levels (HPL/VPL) corresponding to an estimated
error of position with a given probability of missed detection.
Figure 3 illustrates a vision-aided navigation system, proposed in this project, with degraded GNSS
positioning.
b. ILS-based approach
The second set of the sensor failure scenario supposes the classical ILS approach procedure. In this
procedure, the ILS provides lateral and vertical deviations from a 3-degree final glide path with
precision.
In VISION project, we consider the following two ILS failures.
As in the GNSS-based navigation scenario, we suppose that the ILS functions at the beginning of the
final approach phase and then the failure occurs before the aircraft reaches at DA. During flight
validation, as ONERA flight test site is not equipped with ILS, ILS sensor will be emulated onboard by
using real-time data and a-priori knowledge on the environment (runway location, elevation and
size).
The WP4 partners agreed on that scenario 5, which coincide with Scenario 2 in terms of available
sensor set, is prioritized over 6.
Figure 4 shows a vision-aided ILS-like approach guidance system in case of ILS failure.
10
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
2. Reference coordinate frames and transformations
Before introducing the navigation systems design, this section overviews different reference
coordinate frames.
a. Inertial frame: FI
An inertial reference frame is a fixed frame, in which Newton’s second law is valid.
(1)
where Ra=6,378,137.0 m is the length of the Earth semi-major axis and e=0.08181919 is the Earth
eccentricity (values according to WGS84 [WGS-1984]).
From (1), it is straightforward that the longitude λ can be obtained by
(2)
Finding (ϕ, h) from the position in FECEF has some difficulty. The Bowring’s method [Bowring-1976],
which uses a center of curvature of the earth surface point corresponding to the point of interest
(h=0 in the equation (1)), is widely used to obtain a good approximate solution without iteration. This
method calculates the latitude ϕ by
(3)
11
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Then, the MSL altitude can be obtained by using the resulting latitude ϕ.
(4)
Our scenarios of VISION project consider a final approach phase of an aircraft, which is a near-ground
operation. Hence, the altitude h remains small and so does the approximation error.
Let pECEF0 be the origin of the NED (or ENU) frame expressed in FECEF, and (ϕ0, λ0, h0) be its geodetic
coordinates. Then, a point given by pECEF in FECEF can be expressed in the NED (or ENU) frame as
follows.
(5)
Figure 5 shows the three Earth-fixed coordinate frames; FECEF, FGEO and FNED. Those frames are
fixed to the rotating earth and hence not an inertial frame (i.e., Newton’s second law does not apply).
However, in many airplane dynamics problems, the Earth’s rotation ωE relative to FI can be
neglected and those Earth-fixed frames can be used as an inertial frame [Etkin-1972]. In our
navigation system design, the NED frame is used as an inertial reference.
12
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
e. Runway fixed frame: FRWY
In our application of the local relative navigation for the aircraft
final approach, we define a frame locally fixed to the runway on
which the aircraft is approaching. Let pECEFthd be a runway
reference point, which locates at an intersection of the runway
threshold and center line, expressed in the ECEF frame. We set this
point as an origin of the runway fixed frame. Assume a flat runway
with an approach direction ϕRWY from the North and a slight slope
of θRWY from the horizontal plane. Then, as shown in Figure 6, we
define the runway frame orientation so that a rotation matrix from
FNED to FRWY becomes RY(θRWY)Rz(ψRWY). Figure 6 Runway Frame
Hereinafter, we define the NED frame at pECEFthd. Then the
transformation from FNED to FRWY is simply given by the rotation.
(5)
(6)
In the navigation filter design, instead of the Euler angles, the quaternion q is often used to represent
the attitude in order to avoid the singularity occurring at ±90° pitch angle. A relation between the
Euler angles and the quaternion q = [q0 q1 q2 q3] is given by
(7)
13
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
h. Camera frame and pixel-coordinates on an image: FC, Fi
A camera coordinate frame is fixed to a camera with its origin at the center-of-projection (COP) of
the camera, its Z-axis aligned with the camera optical axis (which is perpendicular to the image plane),
X- and Y-axes aligned with the horizontal (rightward in image width) and vertical (downward in image
height) axes of the image plane.
Let pNEDc be the camera’s COP and (φc, θc, ψc) be the Euler angles representing the camera attitude
with respect to FNED. Then, the position of a point in FC is give by
Let pBc and RCB be a position and an orientation (represented by a rotation matrix from FB to FC) of
the camera expressed in the aircraft body frame. Then, pC and RC can be written as
(8)
In VISION project, we consider cameras rigidly fixed to the aircraft body. Then pBc and RCB become
known constant, and so pC and RC can be determined by pB and RB, i.e. the aircraft position and
attitude, as seen in the equation (8). For Stereo-vision system, the left camera’s frame is usually used
as a reference frame.
Figure 7 Different coordinate frames (NED, Body, Camera and Pixel-coordinate frames)
2D pixel-coordinates are defined by a frame fixed on an image with its origin at the image left-top, x-
axis pointing to the right (in an image width direction) and y-axis pointing down (in an image height
direction), as seen in Figure 7. Assuming a pin-hole camera model, a transformation from the 3D
camera frame to 2D pixel-coordinates is determined by the camera’s intrinsic parameters; axis-skew
s, focal length (fx, fy) and offset of the image center (principle point) (x0, y0).
Let pi be a 2D pixel coordinate of the point pC expressed in the camera frame. Then, pi is given by
(9)
14
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
where KC is called the camera intrinsic matrix, which is determined by a camera calibration process,
for example, by using a Matlab calibration toolbox from Caltech1.
ZC is the image depth and αC is its inverse. αC is called inverse depth and is widely used in computer
vision in order to exploit the linearity property in the measurement equation (9) [Matthies-
1989][Civera-2008]. The same concept of inverse range is also introduced in bearing-only target
tracking problems [Aidala-1983].
The VISION project treats the aircraft local navigation problem during the final approach. By
neglecting the earth’s rotation and curvature, the aircraft kinematic equations are given by
(10)
where the rotation matrices RNED and RB are given by (5) and (7) respectively, and the 4x4 matrix Ω
is defined as below.
1
https://www.vision.caltech.edu/bouguetj/calib_doc/
15
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
4. Navigation sensors
This section summarizes navigation sensors available onboard an aircraft, and their measurement
models.
(11)
(12)
(13)
where b denote the measurement bias, and ν the measurement noise modelled as a zero-mean
Gaussian noise. g =[0 0 g] is the earth gravity vector defined in the NED frame. As we consider the
near-ground operation, the normal gravity magnitude g at the latitude ϕ can be approximated by
that at the earth ellipsoid surface:
where e is the Earth eccentricity, ge = 9.7803253359 m/s2 and gp = 9.8321849378 m/s2 are the
normal gravity at the equator and at the pole, respectively [WGS-1984]. For more simplicity, we can
even ignore the variation of the gravity with latitude and approximate it by constant g = g(ϕ0) at the
origin of the local reference frame.
b. GNSS/SBAS
GNSS (Global Navigation Satellite System) provides pseudo-range measurements from visible
satellites to a receiver. The pseudo-range is obtained by multiplying the signal travel time from the
satellite to the receiver by the speed of light c. As the satellite clock and the receiver clock are not
synchronized, the pseudo-range measurement includes a bias due to their offset. Then the pseudo-
range measurement of i-th visible satellite is modelled by
(14)
where pECEFsati and pECEFrec are the position in FECEF, τI and τ are the clock bias of the i-th satellite
and the receiver, respectively. The receiver position in FECEF can be obtained from the aircraft
position pECEF and the fixed receiver position in FB. Here we use an estimate of the satellite clock
bias and its estimation error is included in the noise (that’s why there is a “hat” on τi). νρI is a zero-
mean Gaussian measurement noise which includes errors from different sources such as the satellite
clock and ephemeris errors, compensation errors in ionosphereic and tropospheric signal delays,
multi-path effects, etc [Faurie-2011]. Standard deviation of this measurement noise, denoted by
σUERE (UERE stands for User Equivalent Range Error), is calculated and provided by the receiver. The
receiver clock-bias can be modelled as a 2nd order random process, such as,
16
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
(15)
c. Baro-altimeter
Barometric altimeter measures air pressure to obtain altitude information. A relation between the air
pressure pair and the MSL altitude h is given by
where g is the gravity acceleration, R = 287.058 J·kg·K−1 is the specific gas constant, L = 0.0065 K/m
is the decrease rate of the atmosphere temperature in height, and T0 = 15°C = 288.15 K is the
standard temperature [icao-1993]. This gives a = 2.2557x10−4 and b = 5.2557. P0 is the pressure
adjusted to the standard atmosphere at the Sea-Level, and this information is given as the QNH
pressure. Then by using it, the barometer pressure measurement can be converted to the altitude
measurement which is modelled as
(16)
d. Inclinometer
Two inclinometers are used to measure roll (φ) and pitch (θ) angles of the aircraft.
e. Vision systems
In VISION project, two types of vision systems are proposed to use in order to increase an accuracy
and integrity of the aircraft relative navigation to the runway in case of sensor failure. Detailed
description of preliminary designs of these two vision systems is given in D2.1 “Preliminary
description of vision sensor for navigation and guidance recovery scenario”. Vision system 1 provides
a relative position and orientation information of cameras with respect to the runway, while Vision
system 2 provides also the relative motion (linear and angular velocity) information.
The vision system 1 and 2 will be designed to output the 3D relative position Dp of the camera to the
threshold point expressed in the runway frame FRWY and the relative orientation angles
(Dφ, Dθ, Dψ) of the camera (FC) to the runway (FRWY), giving the rotation matrix RRWY/C.
17
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
The error model of Dp is obtained by an error model of pCthd, which is obtained from the pixel-
coordinate measurement pi and the scaling factor measurement αC in equation (9). By assuming un-
biased Gaussian error model for pi and αC , the inverse transformation of (9) gives the error model
of Dp. The error model of the relative attitude (Dφ, Dθ, Dψ) can be modelled as zero-mean Gaussian
with changing sigma’s. The details will be provided in the deliverable D4.2 “Preliminary description of
vision-based system for navigation and guidance recovery scenario” to be submitted in M11.
Those measurements can be expressed in function of the aircraft pose (pECEF, qB), the known
camera pose parameters (pBc, RCB) in the aircraft body, and the runway pose parameters (pECEFthd,
θRWY, ψRWY) as follows.
Or the following.
Based on simulation results, the output error can be modeled with experimentally determined set of
Gaussian distributions, where sigma is the function of Dp. Again, the details will be provided in the
deliverable D4.2 “Preliminary description of vision-based system for navigation and guidance
recovery scenario” to be submitted in M11.
18
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
f. ILS
ILS is composed of the lateral localizer (LOC)
and the vertical glide slope (GS), which
provides final approach guidance
information, that is, the deviations of the
aircraft flight path from the desired glide
course (normally 3° glide path). Each of the
LOC and GS antennas emits two lobes of
signals modulated at 90Hz and 150Hz
centered at the desired glide path in
horizontal and vertical plan respectively
(Figure 8). The deviations from the desired
glide path can be measured by taking a
difference in amplitude of the two signals,
called DDM (Difference in depth of
modulation).
where DM is the percentage modulation depth of 90 /150 Hz signals. DDM becomes zero when the
aircraft is on the desired glide course.
DDM is linear with respect to a deviation angle (denoted ∆θLOC and ∆θGS in Figure 8) from the front
course line (DDM=0).
where KLOC and KGS are called the angular displacement sensitivity. For the localizer, the nominal
displacement sensitivity is defined in distance (but not in angle) and it is 0.00145 DDM/m at the ILS
reference datum (= runway threshold) within the half course sector. For the glide slope, KGS = 0.0875
is defined [icao-2006].
19
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
5. Navigation performance requirement and Integrity monitoring
This section provides the current specifications of GNSS and ILS for being used in the aircraft final
approach procedure defined by ICAO.
The navigation performance required for the GNSS-based approach procedure is given in [icao-2006]
and summarized in the table below (Table 1).
ILS
The navigation performance required to ILS is also given in [icao-2006] and summarized in Table 2.
Table 2 Navigation performance requirement for ILS-based approach [icao-2006]
ILS- Accuracy Integrity Continuity Availability
LOC DDM Modula-tion Alert limit of Alert limit of Max Continuity False signal
(95%) tolerance course displace- duration of risk risk
alignment ment signal
error sensitivity absence
-6 -9
CAT 0.031 to ±1.0% 3m ±10% 2s 2x10 / 30s 0.5x10 /
III 0.005 (10ft) approach
-6 -9
CAT 0.031 to ±1.5% 7.5m ±17% 5s 2x10 / 15s 0.5x10 /
II 0.005 (25ft) approach
-6 -7
CAT 0.031 to ±2.5% 10.5m ±17% 10s 4x10 / 15s 1.0x10 /
I 0.015 (35ft) approach
0.015DDM
20
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
ILS- Accuracy Integrity Continuity Availability
GS DDM Modul Course Alert Alert Alert Max Continu False signal
(95%) a-tion alignm limit of limit of limit of duratio ity risk risk
toleran ent course displace DDM= n of
ce alignmen ment 0.0875 signal
t error sensitivit displace absence
y error ment
-6 -9
CAT 0.035 ±1.0% +0.04θ -0.075θ ±25% 0.7475θ 2s 2x10 / 0.5x10 /
III to +0.10θ 15s approach
0.023
-6 -9
CAT 0.035 ±1.5% +0.075 -0.075θ ±25% 0.7475θ 2s 2x10 / 0.5x10 /
II to θ +0.10θ 15s approach
0.023
-6 -7
CAT 0.035 ±2.5% +0.075 -0.075θ ±25% 0.7475θ 6s 4x10 / 1.0x10 /
I θ +0.10θ 15s approach
b. Integrity monitoring
Integrity monitoring function is the must for the aircraft navigation system in order to avoid a pilot
(or auto-pilot), by providing a warning, to continue using erroneous information in guidance and
control.
• LS-RAIM
Let ρ = [ρ1 ρ2 … ρN]T be a GNSS measurement vector consisting of a set of pseudo-range
measurements of N visible satellites. Then, it can be written as a nonlinear function of the receiver
position pECEFrec and its clock bias τ as given in (14):
(17)
Let be a predicted value of the receiver position and clock bias before taken the
measurements. Usually, it coincides with the estimate at the previous time step. Then the GNSS
measurement vector (17) can be linearized about this prediction as follows.
(18)
(19)
21
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
A new estimate for the receiver position and clock bias can be obtained as a least-square solution.
(20)
The associated estimation error covariance will be PGNSS = GρRρGρ T where Rρ is the covariance
matrix of the measurement noise νρ.
The LS-RAIM (Least Square-error RAIM) algorithm uses a residual:
(21)
(22)
The sum of the square of the residual (error) is defined as SSE = ∆yρ T Rρ -1∆yρ , and the statistical test
is applied on the following value.
(23)
Under the hypothesis H0, SSE results in a Chi-squared distribution with N-4 degree of freedom,
. For the fault detection, the decision test value Td can be derived by a maximum
allowable false detection probability pfd such that the following equation satisfies (See also Figure 9).
(24)
The failure case H1 is detected when T > Td, otherwise no failure case H0 is assumed.
22
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Figure 9 Chi-squared distribution and Decision test value
• Protection levels
The protection levels are defined by the smallest detectable position error by means of the test
statistics T > Td, while satisfying the missed detection requirement. Consider the case of failure on
the i-th satellite (but not on any other satellites). Then the measurement residual ∆yρ in (22)
incorporates an bias b on its i-th element, and the SSE value now follows the non-central Chi-
squared distribution with N-4 degree of freedom, denoted as where λ is a non-
centrality parameter (Figure 9). The minimum value for λ which satisfies the missed detection
probability (pmd) can be identified by
(25)
for a given pmd and the decision value Td defined in (24). From the relation between the non-
centrality parameter λ and the pseudo-measurement bias b, the smallest detectable bias of the i-th
satellite measurement ρi can be obtained by
(26)
where σρi is the standard deviation of the measurement noise νρi, and Dρii is the i-th diagonal
element of the matrix Dρ defined in (24). Finally, the position (as well as the receiver clock bias)
estimation bias induced by this pseudo-range bias is derived from the equation (20).
where Gρ(:,i) is the i-th column of the matrix Gρ defined in (20). From this, the horizontal and vertical
protection levels (HPL, VPL) are defined by
23
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
(27)
These correspond to the smallest detectable horizontal/vertical position error which satisfies the
miss detection probability requirement. In other words, these are the largest horizontal/vertical
position error bias possible when not detecting the occurring failure.
• Fault exclusion
Once the failure is detected by means of T > Td, the next step is to identify the error source (satellite)
and exclude it. In order to do so, a bank of N estimators {Ei | i=1,2,…N}, each of which uses (N-1)
pseudo-range measurements excluding one from the i-th satellite, is used. The estimated receiver
position and clock bias as well as the test value Ti are calculated in the same manner as in (20) and
(23) for each estimator Ei with a different set of (N-1) pseudo-range measurements. Under the
assumption H1, only one estimator out of N does not use the erroneous measurement and will
provide a correct estimation result. If the failure occurs on the i-th satellite measurement, the
estimator Ei is such an estimator. We define the decision value for the data exclusion Te in the same
manner as in (24) but with the requirement of the maximum false decision of exclusion pfde instead
of pfd. Then, if only one estimator, say Ej, out of the N estimators {Ei | i=1,2,…N} gives T ≤ Te, the
failure can be identified on the j-th satellite and its pseudo-range measurement will be excluded.
Like in RAIM, consider the full-set estimator E0 which uses all the N pseudo-range measurements,
and a bank of N subset estimators {Ei | i=1,2,…N}, each of which uses (N-1) pseudo-range
measurements excluding one from the i-th satellite. Then the separation ∆xi is defined for each
subset estimator Ei as a difference in the estimated states between this estimator Ei and the full-set
one E0. The separation ∆xi and its covariance matrix ∆Pi are given by
(28)
(29)
where Pi is the error covariance of the estimation state of Ei , and hρiT is the Jacobian matrix (a row
vector) of the measurement ρi with respect to the estimation state x. The horizontal and vertical
positions and their associated covariance matrices are given by
24
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
(30)
Eigenvalue decomposition can be applied to the horizontal separation covariance matrix ∆Phi and the
separation ∆pHi can be projected on the eigenvectors:
(31)
Then the test value of the horizontal separations is defined and is often approximated as follows.
(32)
In case of no failure, these test values follow a zero-mean Gaussian distribution with a standard
deviation of ∆σΗi1. In case of presence of failure on one of the pseudo-range measurements used in
Ei, they follow a non-null mean Gaussian distribution. Based on the maximum allowable false
detection probability criterion pfd, the decision value TdHi is calculated as follows.
(33)
where erfc() is the complementary error function. The failure is detected when THi > TdHi. The same
statistical test is performed also on the vertical separation, by using TVi = |∆pVi| and ∆σVi = √∆PVi.
These two tests on the horizontal and vertical separations are independently performed for each of
the N subset estimators. That is why the minimum false detection probability pfd is divided by 2N, a
total number of the statistical tests.
Similarly to the RAIM algorithms, the horizontal and vertical protection level (HPL and VPL) are to
be defined based on the missed detection probability criterion (pmd). HPL is defined as the maximum
position error possible without detecting any failure when satisfying this criterion:
(34)
By the triangle inequality, the HPL can be approximated by the following in a conservative manner.
(35)
Consider the case of failure of the j-th satellite, where the j-th subset estimator Ej functions correctly
and its estimation error follows a zero-mean Gaussian distribution with a known covariance matrix
PHi. Then, the HPL for this case of the j-th satellite failure, denoted as HPLj, is obtained by solving
(36)
25
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Through the similar operation as (31) by using Eigenvalue decomposition,
(37)
where σΗi1 is the largest standard deviation of the covariance PHi. HPLj can be obtained for all
j=1,2,…N. Then the HPL is defined as its maximum value.
(38)
In the same way, the vertical protection level can be obtained as follows.
(39)
26
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
6. Preliminary design of vision-aided navigation system
This section provides description of the preliminary design of vision-aided navigation systems for
SBAS or ILS failure scenarios, defined in D2.1 and recalled in Section 1. Figure 10 illustrates an
overview of the aircraft navigation system, which is composed of the estimator and the integrity
monitoring system for fault detection and exclusion (FDE).
a. Estimator design
In this preliminary design, we suppose that the aircraft attitude qB and angular velocity ωB are
estimated separately from the position and linear velocity by a fusion of AHRS and inclinometer
measurements. This attitude estimator is assumed to give the un-biased estimated attitude and
angular velocity as well as their error covariance at the INS frequency.
Then, the objective of this estimator is to reconstitute the aircraft position and translational velocity
from available onboard sensors. As shown in Figure 10, the estimator is based on INS integration and
correction of its drift by the other sensors (GNSS, vision, barometer) through non-linear/linear
Kalman filtering techniques. The estimator runs at a frequency of INS (AHRS), and after each INS
integration, each of newly arrived sensor measurements will be examined for its validity and used for
the estimation correction if it is valid.
27
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
(40)
where pNED and vNED are the the aircraft position and linear velocity expressed in the local NED
frame fixed at the known runway threshold position pECEFthd, (τ, vτ) are the GNSS receiver clock bias
and drift, and ba is the accelerometer bias.
INS integration
By taking a time-derivative of the estimation sate vector (40), the process model is given by
(41)
By discretizing this process model, and the INS integration will be given by
(42)
(43)
where the last term is due to the attitude estimation error and
Measurement update
At each time step, newly arrived measurements will be used to make a correction on the predicted
state (42). The sensor set includes GNSS, baro-altimeter, vision systems and ILS. The integrity
monitoring algorithm, to be presented in the next section, is applied first so that the erroneous
information source will be removed. Then a Kalman filter-like update process (EKF, UKF, etc.) is
applied on the sensor models of only the selected (non-excluded by the FDE function) sensor
measurements. The estimation error on the attitude and the angular velocity would be also
considered when propagating the error covariance.
28
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
The GNSS pseudo-range measurements (instead of the resulting receiver position and clock bias) are
directly used in the estimation filter, which results in a tight fusion of GNSS with INS and other
sensors. For the purpose of integrity monitoring, the sensor measurement sets should be
independent each other, and so it is preferable to use row sensor data (especially for vision system).
GNSS-RAIM
As stated before, GNSS sensor normally provides a set of redundant measurements, while the other
sensors do not. Therefore, for GNSS, the RAIM algorithm introduced in Section 5.b can be already
applied before the sensor fusion. In the RAIM, the initial receiver position can be obtained from the
predicted state from the INS integration (42). So the erroneous pseudo-range measurement is
supposed to be removed already before the measurement update.
In the preliminary design, the occurrence of failure is assumed to be at the current time step but not
before. Extension of the work by assuming the failure occurrence sometime in the past could be
considered later in the project, by keeping the bank of estimation filters from the previous time step
for the FDE.
29
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
7. Validation plan
This section overviews the validation plan of the vision-based navigation systems for the sensor
failure scenarios.
30
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
• Recording: Record video data with other sensor data during final approach and landing.
Different approach paths (and orientations) are needed for better evaluation of image
processing algorithms.
• Reliability test: The system has to operate during the experiments.
8. Others
Different modules use different frames (local NED/ENU, body, runway fixed, camera, etc.) introduced
in Section 2. These differences require special attention during the project.
31
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Preliminary design of vision-based guidance
1. Trajectory optimization for small deviation from the nominal glide path
a. Trajectory optimization
UTOKYO works on online trajectory optimization. The scenario is defined as follows; when a plane is
approaching the runway, a flying obstacle on the glide path is detected, or a moving obstacle is
detected on the runway. The optimal flight path which can avoid the obstacle is computed based on
the detected obstacle’s position. Research on potential trajectory optimization algorithms is ongoing.
The optimal trajectory should avoid the obstacle with sufficient reliability based on the image sensor
information and the onboard-information (aircraft position and attitude). The image sensor can
detect the relative distance to the obstacle, but it is difficult to detect the exact position of the
obstacle due to the presence of noise in the data. Besides, the possible detection range and the
camera angle are also limited, and the optimal trajectory should keep the constraints associated with
the aircraft flight envelope and ATC. Therefore, the position of the obstacle should be estimated first,
and then the optimal trajectory can be generated to avoid the obstacle. The constraints mentioned
above are considered in the optimization, and the objective function will consist of the distance from
the obstacle, aircraft acceleration, etc.
b. Decision making
In the optimal computation, the fundamental decision must be determined whether a landing
approach continues or go-around is carried out. Decision making algorithms are investigated based
on situations. Present operations and aeronautics laws are checked at the same time.
c. Validation plan
Since it is difficult to conduct a flight experiment of the whole scenario for safety reasons, it will be
validated in a simulation with the required data obtained offline. The offline data will be obtained in
advance with two drones. A drone (drone A) will act as an approaching aircraft with an image sensor
and GPS/INS installed, while another drone (drone B) will act as a flying obstacle with GPS. Drone A
will fly straight on a nominal path, and drone B will intersect the nominal path at certain timing.
Drone A will detect the drone B by image sensor, and the relative distance to the drone B will be
obtained. Since Drone A will have a GPS/INS installed, its own position and attitude can also be
obtained. Drone B has a GPS, so the accuracy of the relative distance between drone A and drone B
will also be investigated. This data acquisition experiment is expected to be conducted several times
to simulate various cases. Finally, using the obtained data (image sensor data and aircraft
position/attitude data), a whole scenario including trajectory optimization will be numerically
simulated. In numerical simulations, a fixed-wing aircraft model of UAV will be considered.
32
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
2. Image-based obstacle avoidance approach
Our approach is based on recent image-based obstacle avoidance results achieved at SZTAKI [Bauer-
2016, Hiba-2016, Zsedrovits-2016]. We have reached the proof of concept level real-time on-board
avoidance in the case of sky-background detections. There are many interesting contributions on this
topic [Fasano-2008, Fasano-2015, Ramasamy-2014], most of them utilize RADAR/LIDAR and IR
sensors beyond the visible light spectrum video. IR is very useful in bad weather conditions and in the
detection of jet engines. Our results with visible light spectrum images can be applied to small- and
micro-UAVs and could enhance the reliability of sensor fusion approaches that lead to obstacle
avoidance products for mid- and large-sized UAVs and aircrafts.
Our decision making approach assumes that the intruder and the own aircraft follow a straight flight
path with constant speed. We model the intruder as a disk with radius r in the horizontal plane. With
these assumptions we could estimate the closest point of approach (CPA) as a multiplicand of 2r (the
size of the intruder is not known), and also the time to closest point of approach (TTCPA). It is easy to
give an appropriate threshold on CPA and TTCPA. For instance, we initiate an avoidance maneuver if
the intruder approaces us closer than 10 times its size, and we want to make this decision 20 seconds
before this CPA realizes. For details see [Bauer-2016].
33
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Figure 11 Basic properties of detected aircraft on the image plane
34
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
Terminology
AAIM : Aircraft Autonomous Integrity Monitoring
ABAS : Aircraft-Based Augmentation System
AGL : Above Ground Level
AHRS : Attitude and Heading Reference System
ATC : Air Traffic Control
CAT : CATegory
COP : Center of Projection
CPA : Closest Point of Approach
DA(H) : Decision Altitude (Height)
DDM : Difference in Depth of Modulation
DGAC : Direction Générale de l’Aviation Civile (French Civil Aviation Authority)
DoF : Degrees of Freedom
ECEF : Earth-Centered Earth-Fixed
EKF : Extended Kalman Filter
ENU : East-North-Up
FAA : Federal Aviation Administration (U.S.A.)
FDE : Fault Detection and Exclusion
GCS : Ground Control Station
GN&C : Guidance, Navigation and Control
GNSS : Global Navigation Satellite System
GPS : Global Positioning System
GS : Glide Slope
HAL : Horizontal Alert Level
HITL : Hardware-In-The-Loop
HPL : Horizontal Protection Level
ICAO : International Civil Aviation Organization
ILS : Instrument Landing System
IMU : Inertial Measurement Unit
INS : Inertial Navigation System
LOC : LOCalizer
LPV : Localizer Performance with Vertical guidance
NED : North-East-Up
PL : Payload
RAIM : Receiver Autonomous Integrity Monitoring
RNAV : aRea NAVigation
RTK : Real-Time Kinematic
SBAS : Satellite-Based Augmentation System
SSE : Sum of Squared Errors
TCH : Threshold Crossing Height
TTCPA : Time to Closest Point of Approach
UAV : Unmanned Aerial Vehicle
UKF : Unscented Kalman Filter
VAL : Vertical Alert Level
VPL : Vertical Protection Level
WP : Work Package
35
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
References
[ec-2012] “RNAV Approaches”, Eurocontrol, 2012.
[dgac-2011] “Technical Guidelines O1 – PBN Guidelines for RNP APCH operations also known as
RNAV(GNSS) Edition No.2”, DGAC, 2011.
[faa-2015] “Aeronautical Information Manual – Official Guide to Basic Flight Information and
ATC Procedures”, US Department of Transportation FAA, 2015.
[casa] “Go-arounds”, Australian Government Civil Aviation Safety Authority,
https://www.casa.gov.au/standard-page/go-arounds
[WGS-1984] “World Geodetic System 1984 – Its Definition and Relationships with Local Geodetic
Systems”, National Imagery and Mapping Agency, 1984.
[Etkin-1972] B. Etkin, “Dynamics of Atomospheric Flight”, John Wiley & Sons, 1972.
[Bowring- B.R. Bowring, “Transformation from Spatial to Geographical Coofdinates”, Survey
1976] Review 23, 1976.
[Cai-2011] G. Cai, B.M. Chen and T.H. Lee, “Unmanned Rotorcraft Systems”, Chapter:
Coordinate Systems and Transformations, Springer, 2011.
[Matthies- L. Matthies and T. Kanade, “Kalman Filter-based Algorithms for Estimating Depth
1989] from Image Sequences”, International Journal of Computer Vision, 1989.
[Civera-2008] J. Civera, A.J. Davison and J.M. Martinez Montiel, “Inverse Depth Parametrization for
Monocular SLAM”, IEEE Transactions on Robotics, vol.24, no.5, 2008.
[Aidala-1983] V.J. Aidala and S.E. Hammel, “Utilization of Modified Polar Coordinates for Bearings-
Only Tracking”, IEEE Transactions on Automatic Control, vol.28, no.3, 1983.
[Faurie-2011] F. Faurie, “Algorithmes de contrôle d’intégrité pour la navigation hybride GNSS et
systèmes de navigation inertielle en présence de multiples mesures satellitaires
défaillantes,” Ph.D. thesis, University of Bordeaux, 2011.
[icao-1993] “Manual of the ICAO Standard Atmosphere”, ICAO Doc 7488/3, 1993.
[icao-2005] “Global Navigation Satellite System (GNSS) Manual”, ICAO Doc 9849/457, 2005.
[icao-2006] “Aeronautical Telecommunications: Volume 1 Radio Navigation Aid”, Annex 10 to
the Convention on International Civil Aviation, ICAO, 2006.
[Brown-1992] R. Grover Brown, “A baseline of GPS RAIM Scheme and a Note on the Equivalence of
Three RAIM methods”, Journal of the Institute of Navigation, vol.39, no.3, 1992.
[Lee-1996] Y. Lee, K. Van Dyke, B. Decleene, J. Studenny and M. Beckmann, “Summary of RTCA
SC-159 GPS Integrity Working Group Activities”, Journal of the Institute of
Navigation, vol.43, no.3, 1996.
[Brenner- M.A. Brenner, “Navigation system with solution separation apparatus for detecting
1998] accuracy failures”, US Patent 5760737 A, 1998.
[Bauer-2016] P. Bauer, A. Hiba, B. Vanek, A. Zarandy, and J. Bokor. Monocular image-based time
to collision and closest point of approach estimation. Mediterranean Conference on
Control and Automation, 2016.
[Hiba-2016] A. Hiba, T. Zsedrovits, P. Bauer, and A/ Zarandy. Fast horizon detection for airborne
visual systems. International Conference on Unmanned Aircraft Systems, 2016.
[Zsedrovits- T. Zsedrovits, P. Bauer, B. J. M. Pencz, A. Hiba, I. Gozse, M. Kisantal, M. Nemeth, Z.
2016] Nagy, B. Vanek, A. Zarandy, and J. Bokor. Onboard visual sense and avoid system for
small aircraft. IEEE Aerospace and Electronic Systems Magazine, 2016.
[Fasano-2008] G. Fasano, D. Accardo, A. Moccia, C. Carbone, U. Ciniglio, F. Corraro, and S. Luongo.
Multi-sensor-based fully autonomous non-cooperative collision avoidance system
for unmanned air vehicles. Journal of aerospace computing, information, and
communication, 5(10):338_360, 2008.
[Fasano-2015] G. Fasano, D. Accardo, A. Elena Tirri, A. Moccia, and E. De Lellis. Sky region obstacle
detection and tracking for vision-based UAS sense and avoid. Journal of Intelligent &
Robotic Systems, pages 1_24, 2015.
36
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)
[Ramasamy- S. Ramasamy, R. Sabatini, and A. Gardi. Avionics sensor fusion for small size
2014] unmanned aircraft sense-and-avoid. In Metrology for Aerospace (MetroAeroSpace),
2014 IEEE, pages 271_276. IEEE, 2014.
37
EU-H2020 GA-690811 NEDO GA-062800
VISION: Validation of Integrated Safety-enhanced Intelligent flight cONtrol (2016-2019)