electronics

Technical Note
Dynamic Performance Evaluation of Various GNSS
Receivers and Positioning Modes with Only One
Flight Test
Cheolsoon Lim 1 , Hyojung Yoon 1 , Am Cho 2 , Chang-Sun Yoo 2 and Byungwoon Park 1, *
1 School of Aerospace Engineering, Sejong University, 209 Neungdong-ro, Gwangjin-gu, Seoul 05006, Korea;
csleem@sju.ac.kr (C.L.); hyovov@sju.ac.kr (H.Y.)
2 Future Aircraft Research Division, Korea Aerospace Research Institute, Daejeon 34133, Korea;
cho1838@kari.re.kr (A.C.); csyoo@kari.re.kr (C.-S.Y.)
* Correspondence: byungwoon@sejong.ac.kr; Tel.: +82-02-3408-4385




Received: 20 November 2019; Accepted: 7 December 2019; Published: 11 December 2019


Abstract: The performance of global navigation satellite system (GNSS) receivers in dynamic modes
is mostly assessed using results obtained from independent maneuvering of vehicles along similar
trajectories at different times due to limitations of receivers, payload, space, and power of moving
vehicles. However, such assessments do not ensure valid evaluation because the same GNSS signal
environment cannot be ensured in a different test session irrespective of how accurately it mimics
the original session. In this study, we propose a valid methodology that can evaluate the dynamic
performance of multiple GNSS receivers in various positioning modes with only one dynamic test. We
used the record-and-replay function of RACELOGIC’s LabSat3 Wideband and developed a software
that can log and re-broadcast Radio Technical Commission for Maritime Services (RTCM) messages
for the augmented systems. A preliminary static test and a drone test were performed to verify proper
operation of the system. The results show that the system could efficiently evaluate the performances
of stand-alone, differential GNSS, and real time kinematics positioning for three GNSS receivers in
two different positioning modes by repeatedly re-radiating the recorded signals acquired through
only one flight. Our proposed system is expected to be useful in evaluating dynamic navigation
performance accurately and conveniently in a valid manner.

Keywords: GNSS; GNSS augmentation system; GNSS signal recording and replaying; RTCM; GNSS
performance evaluation

1. Introduction
GNSS signals transmitted by medium Earth orbit (MEO) satellites [1] are easily affected by noise
and errors during propagation through atmospheric layers [2]. In addition to errors and noise, the
GNSS positioning performance completely depends on various factors, such as the positioning mode,
receiver location, signal reception time, satellite geometry, and constellation [3]. Because of these
vulnerabilities and dependency to signal environments, field testing with real signals before applying
the high-accurate applications is very important to analyze whether their navigation and position will
work as planned in the real world.
The performance of GNSS is generally assessed through long-term static positioning error analysis
for several hours or even days [4]. In such analyses, although all the signal environments are not strictly
controlled, the effects of uncontrolled environments in a statistical sense are not so overwhelming that
the long-term processing and analysis distort the original performance of GNSS receivers. Static test
environments allow sufficient power, time, and space for the comparison of the performance of various

Electronics 2019, 8, 1518; doi:10.3390/electronics8121518 www.mdpi.com/journal/electronics

Electronics 2019, 8, 1518 2 of 21

GNSS receivers simultaneously under the same conditions (e.g., satellite signal environment). Static
tests also enable direct analysis of errors in a position domain because they are generally performed
based on precisely surveyed known positions.
On the other hand, performance assessments during dynamic tests are very difficult and
non-standardized. Unlike the static test, the space and power are very limited in dynamic vehicles,
and thus, simultaneous test sessions or exactly same trajectories cannot be guaranteed. In particular,
the maximum operating time of an unmanned aerial vehicle (UAV) is very short (less than 30 min),
and its space and payload are not sufficient to carry multiple GNSS receivers. Satellite visibility,
ionospheric [5,6] and tropospheric effects, and noise levels generally change with time. Even if the time
correlation of GNSS signals between multiple test sessions is high, different trajectories or attitudes
might induce large differences in GNSS signal quality. Post-processing RINEX (receiver independent
exchange format) data from various receivers can be one of the good options for evaluating the
characteristics of receiver measurements in a dynamic environment [7,8]. This method is suitable for
evaluating the measurement quality of each receiver and for analyzing the performance of known
or newly developed algorithms, but it has limitations in evaluating the performance of commercial
receivers whose detailed positioning algorithms are not disclosed.
In order to analyze the GNSS navigation performances of dynamic systems such as UAV
applications, it is very important to ensure validity of evaluation by providing the exact same
dynamic environments for all the receivers or modes. To overcome the limitations of the dynamic test,
several radio frequency (RF) record-and-replay approaches have been suggested and applied to a test
driving [9,10]. This testing approach represents an effective and efficient method of testing stand-alone
hardware receivers in a dynamic mode, but it cannot be applied for high-performance positioning
modes, which require additional correction information to the GNSS RF signals.
In this study, we propose a practical methodology that provides the same signal environments to
different receivers and various augmented navigation modes to meet emerging and increasingly high
expectations of users. For system implementation, we used RACELOGIC LabSat 3 Wideband [11],
recording and replaying GNSS signals (e.g., GPS L1, L2, and L5) as suggested by previous research. In
addition, our proposed “RTCM Logger and Broadcaster” software stores Radio Technical Commission
for Maritime Services (RTCM) [12,13] messages and feeds them to multiple GNSS receivers to
evaluate the differential GNSS (DGNSS) [14,15] or real-time kinematic (RTK) [16] performance as
well as stand-alone GNSS performance. We also analyze whether pseudorange and carrier phase
measurements from live and replayed signals are both equivalent, which can explain the repeatability
of the position solutions.
The remainder of this paper is structured as follows. In Section 2, the suggested methodology and
the developed system are introduced. Section 3 shows the results of preliminary static tests to verify
the GNSS performance evaluation system. The evaluation system was used for a drone flight test, and
its performance assessments for multiple GNSS receivers and various positioning modes are described
in Section 4. The discussions and conclusions are given in Section 5.

2. Method and System Implementation

2.1. GNSS Signal Recording and Replaying

Because repeatable and controllable signals cannot be ensured for every test session in dynamic
testing under live-sky conditions, the ability to exhaustively field-test products in real-world conditions
is necessary for developers of location-aware chipsets, devices, and systems before launching them.
When budgets and time are both under pressure, they inevitably select simulators, but simulators cannot
anticipate the full richness and variety of real-world conditions. In this sense, the record-and-replay
method, which is derived from software-defined radio (SDR) paradigm [17], is one of the easy and
efficient means of characterizing new high-performing product performance in the real world by
providing a combination of realism and repeatability, which is ideal for performance verification.
Electronics 2019, 8, 1518 3 of 21

In this study, we used RACELOGIC’s LabSat 3 Wideband, which can record and replay GNSS
signals. LabSat 3 supports multi-frequency GNSS signals, such as GPS L1/L2/L5, GLONASS G1/G2/G3,
Galileo E1/E5/E6, BeiDou B1/B2/B3, QZSS L1/L2/L5, and SBAS L1/L5 anywhere and anytime without
the need for connecting to a desktop or laptop computer [11]. This device is suitable for the purpose
of our verification methodology, i.e., providing the same signal environment to all GNSS receivers
for which the performance is to be evaluated. The simulation or validation test is based on the
replay functionality of similar devices, but most are for representing the resilience of their algorithms
or devices to increasing GNSS vulnerabilities, such as jamming, spoofing [18–20], multipath [21],
and interferences.

2.2. Software for Logging and Broadcasting RTCM Messages

Most GNSS receivers support various types of positioning modes, such as precise point positioning
(PPP) [22], differential GNSS (DGNSS) [14], satellite-based augmentation system (SBAS) based on
wide-area DGPS (WADGPS) [23], real-time kinematic (RTK) [16], and network RTK [24–26] in addition
to the stand-alone mode. Among these positioning modes, the stand-alone and SBAS modes do not
require an additional communication channel for receiving correction data. Therefore, these modes are
easy and simple to configure in UAVs, and recorded GNSS signals are easily provided using LabSat 3.
For other modes, correction data generated from ground reference station facilities must be fed to the
GNSS receivers through additional communication channels. Even though a pair of two high-end
signal recorders enables the reproduction of the working process of an RTK base-rover system [27],
GNSS RF signals should be logged at one as a rover and the other as a base station, and they should
be synchronized to replay the RTK process. In particular, since various services that provide GNSS
correction messages via the internet without any physical base station have recently become common,
applications using the pair of two devices are inevitably becoming very limited. Therefore, it is not
only complicated to configure a real-time system, but also impossible to perform the test using a single
device. To solve this problem practically, we propose a method that enables a rover GNSS receiver to
operate in DGNSS, RTK, or network RTK modes using the RF signal replayed at only one device. We
developed a software that logs the correction data of the augmentation system during the recording of
real-time GNSS RF signals. The software can also re-broadcast the logged corrections to the GNSS
receivers when replaying the recorded signals. Using the software, it is possible to evaluate and
compare the performances of various positioning modes for multiple GNSS receivers.
The developed software consists of ‘RTCM Logger’ and ‘Broadcaster’ parts. The RTCM Logger
writes RTCM version 2 and 3 messages in a file and the RTCM Broadcaster re-broadcasts the logged
RTCM messages to each GNSS receiver. While LabSat 3 is recording GNSS RF signals, the RTCM
Logger can receive RTCM messages through the Network Transport of RTCM via Internet Protocol
(NTRIP) [28], Transmission Control Protocol/Internet Protocol (TCP/IP) or a serial port. Then, the
received RTCM messages are converted and written in a format starting with a header ($RTCMV2
or $RTCMV3) to facilitate the distinguishing of the augmentation mode. Subsequently, the RTCM
messages are written in the binary format with its coordinated universal time (UTC) [29], as shown in
Figure 1.2019, 8, x FOR PEER REVIEW
Electronics 4 of 22

Scheme of logging Radio Technical Commission for Maritime Services (RTCM) messages
Figure 1. Scheme
(RTCM Logger).
Electronics 2019, 8, x FOR PEER REVIEW 4 of 22

Electronics 2019, 8, 1518 4 of 21

During signal replay, the RTCM Broadcaster can obtain the UTC time of the replayed signal
through the National Marine Electronics Association (NMEA) [30] GGA sentence from the GNSS
rover receiver. The RTCM Broadcaster searches for the UTC time from the logged file, and then feeds
the timely corresponding message(s) to the receivers, as shown in Figure 2. The correction age of
the NEMA GGA sentence represents the duration since the correction was fed to the receiver, which
is typically 1 s for normal DGPS and RTK conditions. We can ensure that the correction has been
Figure 1. Scheme of logging Radio Technical Commission for Maritime Services (RTCM) messages
delivered to the receiver on time by monitoring the correction age, and this methodology also allows
(RTCM Logger).
us to simulate a long-latency case due to a bad connection to the correction server.

Figure 2. Scheme of re-broadcasting RTCM messages (RTCM Broadcaster).

Figure 2. Scheme of re-broadcasting RTCM messages (RTCM Broadcaster).
2.3. Performance Evaluation Method of Various GNSS Receivers
2.3. Performance Evaluation
Figure 3 illustrates theMethod
flow ofofthe
Various GNSS Receivers
methodology for evaluating and analyzing the performance
of GNSS receivers
Figure for a moving
3. illustrates the flowvehicle. First, the received
of the methodology live GNSS
for evaluating and signals are recorded
analyzing while a
the performance
vehicle with LabSat 3 is moving along a specific trajectory. At the same time,
of GNSS receivers for a moving vehicle. First, the received live GNSS signals are recorded while RTCM messages fora
DGPS
vehicleorwith
RTKLabSat
are also written
3 is moving byalong
the RTCM Logger.
a specific Then, the
trajectory. At therecorded signals
same time, are re-radiated
RTCM messages forto
GNSS receivers, and the RTCM message(s) corresponding to the UTC time
DGPS or RTK are also written by the RTCM Logger. Then, the recorded signals are re-radiated to for the replayed signal
is/are
GNSSfed to the receivers
receivers, and the by the RTCM
RTCM Broadcaster.
message(s) Through
corresponding to these procedures,
the UTC time for thethetrajectory
replayed of the
signal
drone flight for each positioning mode (e.g., stand-alone, DGPS, RTK, SBAS)
is/are fed to the receivers by the RTCM Broadcaster. Through these procedures, the trajectory of theof the receivers can be
obtained. We processed receiver independent exchange format (RINEX) [31] files
drone flight for each positioning mode (e.g., stand-alone, DGPS, RTK, SBAS) of the receivers can be (one for the trajectory
of the vehicle,
obtained. Weand another receiver
processed from reference stations exchange
independent located near the rover)
format (RINEX)using[31]
GNSS post-processing
files (one for the
software
trajectory of the vehicle, and another from reference stations located near the rover) using the
Trimble Business Center (TBC) [32] to obtain 5 mm accurate true trajectories of GNSSmoving
post-
vehicle. Finally, we evaluated the performance of the receivers by comparing the
processing software Trimble Business Center (TBC) [32] to obtain 5 mm accurate true trajectories of calculated trajectory
from multiplevehicle.
the moving receiversFinally,
with the wetrue trajectorythe
evaluated obtained using TBC.
performance of the receivers by comparing the
calculated trajectory from multiple receivers with the true trajectory obtained using TBC.
Electronics 2019, 8, 1518 5 of 21
Electronics 2019, 8, x FOR PEER REVIEW 5 of 22

Electronics 2019, 8, x FOR PEER REVIEW 5 of 22

Figure3.3.Scheme
Figure Schemeofoffeeding
feedingRTCM
RTCMmessages
messages(RTCM
(RTCMBroadcaster).
Broadcaster).

3.3.Verification
Verificationofofthe
thePerformance
Performance Evaluation
Figure 3. SchemeEvaluation
System
System
of feeding RTCM messages (RTCM Broadcaster).

3.1. Static Test Configuration

3.1. 3. Verification
Static of the Performance Evaluation System
Test Configuration
To verify the suitability of our developed system, we conducted static tests using LabSat 3,
ToStatic
3.1. verify theConfiguration
Test suitability of our developed system, we conducted static tests using LabSat 3, and
and the RTCM message logging/broadcasting software. First, we performed range-domain and
the RTCM message logging/broadcasting
To verifyanalyses
the suitability of our developed
software.
system,
First,conducted
we performed range-domain and position-
position-domain to examine whether the use ofweLabSat static tests
3 degraded theusing LabSat 3, and
performance. For this
domain analyses
the RTCM to examine
message whether the use
logging/broadcasting of LabSat
software. First,3 degraded therange-domain
performance.and Forposition-
this analysis,
analysis, we collected GNSS satellite signals received by we performed
a GNSS antenna (NovAtel’s GPS-703-GGG
we collected GNSStosatellite
domain analyses examinesignals
whetherreceived
the use ofby a GNSS
LabSat antenna (NovAtel’s
performance.GPS-703-GGG
For this analysis,Triple-
Triple-Frequency Pinwheel GNSS Antenna) installed on3 the
degraded
roof ofthethe Chung-moo building at Sejong
Frequency Pinwheel
we collected GNSS GNSS
satelliteAntenna) installed
signals received by aon the antenna
GNSS roof of (NovAtel’s
the Chung-moo building
GPS-703-GGG at Sejong
Triple-
University in South Korea, as shown in Figure 4. The signals were processed using NovAtel FlexPak6
Frequency
University Pinwheel
in South GNSS
Korea, Antenna)
as shown in installed
Figure 4.on thesignals
The roof ofwere
the Chung-moo
processed usingbuilding at Sejong
NovAtel FlexPak6
(high-end receiver),
University
u-blox
in Southu-blox
M8P
Korea, M8P
(low-cost
as shown
receiver),
in Figure
and LabSat
4. The signals
3, whichusing
were processed
are connected to a signal
(high-end receiver), (low-cost receiver), and LabSat 3, which areNovAtel
connectedFlexPak6
to a signal
splitter. After recording
(high-end the live signals on receiver),
LabSat 3,andtheLabSat
recorded signals
are were re-radiated to both
splitter. Afterreceiver),
recording u-blox M8P
the live (low-cost
signals on LabSat 3, the recorded 3, which
signals connected to a signal
were re-radiated to both
FlexPak6 and M8P receivers. We obtained the RINEX file for each receiver, for use
splitter. After recording the live signals on LabSat 3, the recorded signals were re-radiated to both in the range domain
FlexPak6 and M8P receivers. We obtained the RINEX file for each receiver, for use in the range
analysis. Position
FlexPak6 and results were also
M8P receivers. Weobtained
obtained from both receivers,
the RINEX file for eachas receiver,
described forinuse
Section
in the2,range
and then
domain analysis. Position results were also obtained from both receivers, as described in Section 2,
domain analysis.
performance analysisPosition results were also obtained
in the position-domain from both receivers, as described in Section 2,
was performed.
andandthenthen
performance analysis in the position-domain was performed.
performance analysis in the position-domain was performed.

Figure
Figure 4. Test
4. Test configuration
configuration forforverifying
verifyingperformance
performance degradation
degradationdue to the
due useuse
to the of LabSat 3. 3.
of LabSat

Figure 4. Test configuration for verifying performance degradation due to the use of LabSat 3.
 The subscript r denotes the source of signals (real time (RT) or LabSat 3 (LS)), and the subscripts
s and i denote the s-th satellite and type of carrier frequency, respectively. In addition,
𝜌, 𝐵, 𝑏, 𝑇, 𝐼, 𝑀, 𝑁, 𝜆, and 𝜀 denote the geometric range from satellite to ground receiver, receiver clock
bias, satellite clock bias, tropospheric delay, ionospheric delay, multipath error, integer ambiguity,
carrier wavelength,
Electronics 2019, 8, 1518 and measurement noise, respectively. 6 of 21
As direct one-by-one comparisons of the measurements cannot be performed in a statistical
manner (Figure 5) due to different 𝐵 and 𝑁, we calculated DD measurements for the s-th and t-th
3.2. Measurement-Domain Test
satellites between RT and LS. DD (𝛥𝛻) between measurements of RT and LS can be expressed as
To compare the real time measurements with those from the recorded RF signals, we analyzed
follows.
double difference (DD) measurements using code and phase measurements. Even in a static test, code
𝛥𝛻𝑃 𝛥𝛻𝜌 𝛥𝛻𝐵 𝛥𝛻𝑏 𝛥𝛻𝑇 𝛥𝛻𝐼 𝛥𝛻𝑀 𝛥𝛻𝜀 (3)
(P) and phase (Φ) measurements obtained from both signals are mathematically different to each other
due to their own𝛥𝛻Φ receiver𝛥𝛻𝜌clock 𝛥𝛻𝐵 ) and cycle
bias (B𝛥𝛻𝑏 𝛥𝛻𝑇 ambiguity
𝛥𝛻𝐼 𝛥𝛻𝑀 (N ) terms,𝛥𝛻𝑁 as described
𝜆 𝛥𝛻𝜀 in Equations (4) (1)
and (2) [22].
In the above equations, P the
s geometric
s range
s (𝜌),
s multipath
s errors (M), and ionospheric (I) and
r,i = ρr + Br − b + Tr + Ir,i + MPr,i + εPr,i (1)
s
tropospheric (T) delay are also canceled out because both signals are from a zero-baseline
configuration. Receiver (𝐵) Φsr,iand s
= ρsatellite bs +clock
r + Br −(𝑏 Trs −biases
s
Ir,i +M s s
r,i λi + εΦby
+ Nremoved
Φsr,ialso
are s (2)
r,i the differences between
satellites
The and receivers,
subscript respectively.
r denotes If the code
the source measurements
of signals (real time from RT or
(RT) andLabSat
LS are 3exactly
(LS)),the
and same,
the
only noise (𝛥𝛻𝜀 should remain in DD of the code measurements (𝛥𝛻𝑃
subscripts s and i denote the s-th satellite and type of carrier frequency, respectively. In addition, . In the same manner, the
phase DD (𝛥𝛻Φ would not contain all the terms other than the ambiguity
ρ, B, b, T, I, M, N, λ, and ε denote the geometric range from satellite to ground receiver, receiver (𝛥𝛻𝑁 𝜆 and noise
(𝛥𝛻𝜀 bias,. satellite
clock Assuming that
clock anytropospheric
bias, cycle slip ofdelay,
both measurements
ionospheric delay, doesmultipath
not occurerror,
in Equation (4), DD of
integer ambiguity,
𝛥𝛻Φ
carrier
integerwavelength,
ambiguity (𝛥𝛻𝑁 and measurement
can be estimated noise,and
respectively.
removed by the mean integer value of 𝜆 from
As direct one-by-one comparisons of the measurements cannot
Equation (4). Figure 6a,b show the DD residual errors of carrier phase measurements obtained frombe performed in a statistical
manner
real-time(Figure 5) due
and LabSat 3 signals for allBthe
to different and N, wesatellites,
visible calculated DD measurements
respectively. The maximum for theand
s-th and
root t-th
mean
satellites between RT and LS. DD (∆∇) between measurements of RT and
square (RMS) values of DD residuals of both receivers are less than 0.02 m, as shown in Table 1. LS can be expressed as follows.

Direct comparison
Figure 5. Direct comparisonof
ofGPS
GPSL1
L1pseudorange
pseudorangemeasurements
measurementsfrom
fromreal
realtime
time(RT)
(RT)and
andLabSat
LabSat3
3(LS).
(LS).

∆∇Pi = ∆∇ρ + ∆∇B − ∆∇b + ∆∇T + ∆∇Ii + ∆∇M∆∇Pi + ∆∇ε∆∇Pi (3)

∆∇Φi = ∆∇ρ + ∆∇B − ∆∇b + ∆∇T + ∆∇Ii + ∆∇M∆∇Φi + ∆∇Ni λi + ∆∇ε∆∇Φi (4)

In the above equations, the geometric range (ρ), multipath error (M), and ionospheric (I) and
tropospheric (T) delay are also canceled out because both signals are from a zero-baseline configuration.
Receiver (B) and satellite (b) clock biases are also removed by the differences between satellites and
receivers, respectively. If the code measurements from RT and LS are exactly the same, only noise
(∆∇ε∆∇Pi ) should remain in DD of the code measurements (∆∇Pi ). In the same manner, the phase DD
( would not contain all the terms other than the ambiguity (∆∇Ni λi ) and noise (∆∇ε∆∇Φi ). Assuming
that any cycle slip of both measurements does not occur in Equation (4), DD of integer ambiguity
∆∇Φ
(∆∇Ni ) can be estimated and removed by the mean integer value of λ i from Equation (4). Figure 6a,b
i
show the DD residual errors of carrier phase measurements obtained from real-time and LabSat 3
signals for all the visible satellites, respectively. The maximum and root mean square (RMS) values of
DD residuals of both receivers are less than 0.02 m, as shown in Table 1.
Electronics 2019, 8, 1518 7 of 21
Electronics 2019, 8, x FOR PEER REVIEW 7 of 22

(a) (b)
Figure6.6. Double
Figure Doubledifference
difference(DD)
(DD)residuals
residualsof
ofcarrier
carrierphase
phasemeasurements;
measurements;(a)
(a)NovAtel,
NovAtel,(b)
(b)u-blox.
u-blox.

Table 1. Root mean square (RMS) and maximum of double difference residuals of carrier
Table 1. Root mean square (RMS) and maximum of double difference residuals of carrier phase
phase measurements.
measurements.
GNSS Receiver RMS Max.
GNSS Receiver RMS Max.
NovAtel
NovAtel FlexPak6
FlexPak6 0.0011m
0.0011 m 0.0089
0.0089
mm
u-blox M8P 0.0014 m 0.0157 m
u-blox M8P 0.0014 m 0.0157 m

3.3.
3.3.Position-Domain
Position-DomainTest
Test
We
Wealso
alsoexamined
examinedwhether
whether performance
performance degradation
degradationoccurred
occurred duedue
to the use use
to the of LabSat 3 and3 our
of LabSat and
developed
our developed software through
software position
through domain
position analysis.
domain We conducted
analysis. We conducted real-time and semi-real-time
real-time and semi-real-
static
time tests
staticusing the live
tests using theand
liverecorded signals.
and recorded As shown
signals. in Figure
As shown 7, 3 receivers
in Figure and LabSat
7, 3 receivers 3 were3
and LabSat
connected to the same antenna to receive the same live satellite signals.
were connected to the same antenna to receive the same live satellite signals. RTCM v2 and v3RTCM v2 and v3 messages
generated
messages by Trimbleby
generated NetR9 wereNetR9
Trimble respectively fed to M8Pfed
were respectively for to
RTK M8Ppositioning and EVK-6and
for RTK positioning for DGPS
EVK-6
positioning. These messages were also logged by our developed RTCM logger.
for DGPS positioning. These messages were also logged by our developed RTCM logger. After After the real-time static
the
test, we performed the semi-real-time static test by re-radiating the recorded signals
real-time static test, we performed the semi-real-time static test by re-radiating the recorded signals to both receivers.
In
toaddition, for DGPS
both receivers. and RTK
In addition, forpositioning,
DGPS and RTK RTCM v2 and v3RTCM
positioning, messages at the
v2 and v3 time corresponding
messages at the time
to
corresponding to the UTC time in the NMEA GGA sentence obtained for each receiver receiver,
the UTC time in the NMEA GGA sentence obtained for each receiver were fed to each were fedas to
shown in Figure
eachElectronics
receiver, as8.
2019, xFinally,
8,shown in we
FOR PEER obtained
REVIEW
Figure the NMEA
8. Finally, files (GGA)
we obtained from each
the NMEA files receiver.
(GGA) from each 8 of 22
receiver.

Figure 7. Real-time
Figure static
7. Real-time testtest
static configuration
configurationfor
for verifying performance
verifying performance degradation
degradation duedue touse
to the theofuse of
our developed software.
our developed software.
 Figure 7. Real-time static test configuration for verifying performance degradation due to the use of
our developed software.
Electronics 2019, 8, 1518 8 of 21

Figure 8.8. Semi-real-time

Figure Semi-real-timestatic
statictest
testconfiguration
configuration
forfor verifying
verifying performance
performance degradation
degradation duedue to use
to the the
use
of of developed
our our developed software.
software.

Figures
Figures 99 and
and 10 10 show
show the
the differences
differencesbetween
between east-north-up
east-north-up (ENU)(ENU) errors
errors of of the
the real-time
real-time and and
semi-real
semi-real timetime tests
tests in
in DGPS
DGPS andand RTK
RTKmodes.
modes. In In case
case of
of the
the DGPS
DGPS mode,
mode, therethere were
were discrepancies
discrepancies
due
due toto noise
noise differences
differences between
between real-time
real-time andand recorded
recorded RF RF signals,
signals, but biases were not observed.
Furthermore,
Furthermore,the theposition
positionfix
fixtype
typeininthe
the NMEA
NMEA GGA GGAsentence
sentence turned
turned to to 22 (DGPS
(DGPS mode)
mode) at at the
the exact
exact
same
same time
time that
that the
the real-time
real-time process
process turned
turned to to the
the DGPS
DGPS mode,mode, and
and thethe fix
fix type
type stayed
stayed at at 22 for
for the
the
remainder
remainderof ofthe
thetest.
test.For
Forthe
theRTKRTKcase,
case,although
although thethe
discrepancies
discrepancies increased
increased to about
to about0.6 m0.6during
m during the
initial convergence
the initial convergence period of RTK
period (float(float
of RTK mode, fix type
mode, = 5), =the
fix type 5),RMS valuevalue
the RMS of theofdiscrepancies
the discrepancies was
only 0.0010.001
was only m during the RTK
m during fixedfixed
the RTK modemode (fix type = 4). =Except
(fix type for the
4). Except for initial convergence
the initial convergence period, the
period,
RTK fix type
the RTK (Fixed
fix type or Float)
(Fixed was the
or Float) wassame
the overall. These These
same overall. resultsresults
imply that
imply thethat
usethe
of recorded signals
use of recorded
Electronics 2019, 8, x FOR PEER REVIEW 9 of 22
and ourand
signals developed softwaresoftware
our developed does notdoesleadnotto position performance
lead to position degradation.
performance degradation.

(a) (b)
Figure9.9.(a)
Figure (a)Discrepancies
Discrepanciesinin ENU
ENU errors
errors between
between both
both signals
signals in the
in the DGPS
DGPS mode;
mode; (b) NMEA
(b) NMEA GGAGGA
fix
fix type
type inDGPS
in the the DGPS mode.
mode.
 (a) (b)
Figure 9. (a) Discrepancies in ENU errors between both signals in the DGPS mode; (b) NMEA GGA
Electronics type8,in1518
fix2019, the DGPS mode. 9 of 21

(a) (b)
Figure10.
Figure 10.(a)
(a)Discrepancies
Discrepanciesinin ENU
ENU errors
errors between
between both
both signals
signals in the
in the RTKRTK mode;
mode; (b) NMEA
(b) NMEA GGAGGA
fix
fix type
type in theinRTK
the RTK
mode.mode.

4.4.Flight
FlightTest
TestPerformance
PerformanceEvaluation
Evaluation
4.1. Drone Flight Test Configurations
4.1. Drone Flight Test Configurations
For the dynamic test, we selected a drone capable of mounting the LabSat 3 antenna and configured
For the dynamic test, we selected a drone capable of mounting the LabSat 3 antenna and
a flight scenario for the GNSS signal acquisition. We used DJI’s Phantom 3 Professional. Table 2 shows
configured a flight scenario for the GNSS signal acquisition. We used DJI’s Phantom 3 Professional.
the specifications of Phantom 3.
Table 2 shows the specifications of Phantom 3.
Table 2. Specifications of DJI’s Phantom 3 Professional.

Diagonal Size 350 mm

Weight 1.28 kg
Max Ascent Speed 5 m/s
Max Descent Speed 3 m/s
Max Speed 57.6 km/h
Max Flight Time Approx. 23 min
Max Tilt Angle 35◦
Max Transmission
3.5–5 km
Distance

We divided the drone’s total flight time of 10 min into 3 min of remaining stationary on the
ground and 7 min of dynamic maneuvering. The drone flight scenario is as follows. First, the drone
was allowed to remain stationary on the ground for 3 min after switching it on. Then the drone was
flown along a square trajectory, as shown in Figure 11, after taking off. There are four flight path
segments, and each segment includes 1 min of hovering at its waypoint and 30 s of maneuvering to
another waypoint.
We carried out the drone flight test on the rooftop of Gwanggaeto building in Sejong University
on November 6, 2018, as shown in Figure 12. GNSS signals collected by the drone were recorded in
LabSat 3, and RTCM messages generated by Trimble NetR9 (baseline length <1 km) were logged at the
same time, as shown in Figure 13. The developed RTCM logger stored the real-time streams of RTCM
v 2.3 message to feed the receivers Type (MT) 1 for differential corrections and MT 3 for reference
station parameters. It also logged RTCM v3.1 messages of MT 1004 and 1012 with 1005 to re-broadcast
L1 and L2 GPS and GLONASS RTK observables in order to replay GPS/GLONASS RTK.
 We divided the drone’s total flight time of 10 min into 3 min of remaining stationary on the
ground and 7 min of dynamic maneuvering. The drone flight scenario is as follows. First, the drone
was allowed to remain stationary on the ground for 3 min after switching it on. Then the drone was
flown along a square trajectory, as shown in Figure 11, after taking off. There are four flight path
Electronics 2019, 8, 1518 10 of 21
segments, and each segment includes 1 min of hovering at its waypoint and 30 s of maneuvering to
another waypoint.

Electronics 2019, 8, x FOR PEER REVIEW 11 of 22

Electronics 2019, 8, x FOR PEER REVIEW 11 of 22
Figure 11. Drone flight scenario.
Figure 11. Drone flight scenario.

We carried out the drone flight test on the rooftop of Gwanggaeto building in Sejong University
on November 6, 2018, as shown in Figure 12. GNSS signals collected by the drone were recorded in
LabSat 3, and RTCM messages generated by Trimble NetR9 (baseline length <1 km) were logged at
the same time, as shown in Figure 13. The developed RTCM logger stored the real-time streams of
RTCM v 2.3 message to feed the receivers Type (MT) 1 for differential corrections and MT 3 for
reference station parameters. It also logged RTCM v3.1 messages of MT 1004 and 1012 with 1005 to
re-broadcast L1 and L2 GPS and GLONASS RTK observables in order to replay GPS/GLONASS RTK.
After recording the real time signals through the drone flight test, we performed a total of three
re-radiation tests to evaluate the performance of the stand-alone, DGPS, and RTK modes. The
recorded signals and RTCM messages were processed by three types of GNSS receivers: Novatel
flexpak6, SwiftNav’s Piksi Multi, u-blox M8P, and u-blox EVK-6 (Figure 14). Flexpak6 and Piksi Multi
receivers among the them support Beidou and Galileo as well as GPS and GLONASS, however DGPS
and RTK of GPS and GLONASS have been analyzed because the correction messages for only GPS
and GLONASS have been stored. Even though Phantom 3 does not provide enough payload and
Figure 12. Drone
space to carry three receivers, the performances offlight testreceivers
all the location. with various positioning modes
Figure 12. Drone
Figure12. Drone flight
flight test location.
test location.
could be evaluated with only one flight test.

Figure 13.
Figure 13. Drone
Drone flight
flight test
testconfigurations
configurations(real-time).
(real-time).
Figure 13. Drone flight test configurations (real-time).
Electronics 2019, 8, 1518 11 of 21

After recording the real time signals through the drone flight test, we performed a total of three
re-radiation tests to evaluate the performance of the stand-alone, DGPS, and RTK modes. The recorded
signals and RTCM messages were processed by three types of GNSS receivers: Novatel flexpak6,
SwiftNav’s Piksi Multi, u-blox M8P, and u-blox EVK-6 (Figure 14). Flexpak6 and Piksi Multi receivers
among the them support Beidou and Galileo as well as GPS and GLONASS, however DGPS and
RTK of GPS and GLONASS have been analyzed because the correction messages for only GPS and
GLONASS have been stored. Even though Phantom 3 does not provide enough payload and space to
carry three receivers, the performances of all the receivers with various positioning modes could be
evaluated
Electronicswith
2019, only one
8, x FOR flight
PEER test.
REVIEW 12 of 22

Figure 14.14.Drone
Figure Droneflight
flighttest
testconfigurations
configurations (re-radiation).
(re-radiation).

4.2. Stand-Alone Mode

In the first test (stand-alone mode), only the recorded signals were input to three receivers, and
4.2. Stand-Alone Mode
the performance was analyzed using NMEA files obtained from each receiver. Figures 15 and 16 show
the estimated trajectories
In the first of the drone
test (stand-alone andonly
mode), ENUthe
errors in the
recorded stand-alone
signals mode,
were input respectively.
to three receivers,Table
and 3
the performance was analyzed using NMEA files obtained from each receiver. Figures
shows RMS and maximum values of errors for each direction. Our proposed system allows intuitive 15 and 16
show the estimated
interpretation and easytrajectories
comparisonof the drone
of the and ENU errors
performance of theinthree
the stand-alone mode, respectively.
different receivers with a single
Table
flight test.3 shows RMS and maximum values of errors for each direction. Our proposed system allows
intuitive interpretation and easy comparison of the performance of the three different receivers with
a single flight test.
Table 3. RMS and maximum of ENU errors in the stand-alone mode.

SwiftNav u-blox u-blox

Piksi Multi EVK-6 M8P
East 0.8947 m 1.0870 m 0.8400 m
RMS North 0.7828 m 1.0204 m 0.7163 m
Up 4.3090 m 1.6958 m 2.4837 m
East 2.5083 m 4.4078 m 1.1333 m
Max. North 2.6784 m 5.7743 m 1.6973 m
Up 9.0660 m 5.3500 m 3.3260 m

Figure 15. Horizontal and vertical flight trajectories of the drone in the stand-alone mode.
 In the first test (stand-alone mode), only the recorded signals were input to three receivers, and
the performance was analyzed using NMEA files obtained from each receiver. Figures 15 and 16
show the estimated trajectories of the drone and ENU errors in the stand-alone mode, respectively.
Table 3 shows RMS and maximum values of errors for each direction. Our proposed system allows
intuitive2019,
Electronics interpretation
8, 1518 and easy comparison of the performance of the three different receivers12with
of 21
a single flight test.

Electronics 2019, 8, x FOR PEER REVIEW 13 of 22

Figure 15. Horizontal and vertical flight trajectories of the drone in the stand-alone mode.
Figure 15. Horizontal and vertical flight trajectories of the drone in the stand-alone mode.

Figure 16. ENU errors in the stand-alone mode.

Figure 16. ENU errors in the stand-alone mode.
4.3. DGPS Mode
Table 3. RMS and maximum of ENU errors in the stand-alone mode.
Because only u-blox EVK-6 has the DGPS functionality among the three types of receivers, the
recorded signals and RTCM v2 messages were SwiftNav u-blox
input to u-blox EVK-6u-blox
for the second test (DGPS mode).
The performance was analyzed using the NMEAPiksi Multi EVK-6
file obtained M8P
from this receiver, referenced to the TBC
East 0.8947 m 1.0870 m 0.8400 m and ENU errors in the
true trajectory. Figures 17 and 18 show the estimated trajectories of the drone
DGPS mode, respectively.RMS Table 4North 0.7828
shows that m and
the RMS 1.0204 m 0.7163
maximum m of errors are about 1–2 m
values
and 3–5 m for each direction. Up 4.3090 m 1.6958 m 2.4837 m
East 2.5083 m 4.4078 m 1.1333 m
Max. North 2.6784 m 5.7743 m 1.6973 m
Up 9.0660 m 5.3500 m 3.3260 m

4.3. DGPS Mode

Because only u-blox EVK-6 has the DGPS functionality among the three types of receivers, the
recorded signals and RTCM v2 messages were input to u-blox EVK-6 for the second test (DGPS
mode). The performance was analyzed using the NMEA file obtained from this receiver, referenced
to the TBC true trajectory. Figures 17 and 18 show the estimated trajectories of the drone and ENU
errors in the DGPS mode, respectively. Table 4 shows that the RMS and maximum values of errors
are about 1–2 m and 3–5 m for each direction.
Electronics 2019, 8, x FOR PEER REVIEW 14 of 22

Electronics 2019,8,8,1518
Electronics2019, x FOR PEER REVIEW 13 of 21
14 22

Figure 17. Horizontal and vertical flight trajectories of the drone in the DGPS mode.
Figure 17. Horizontal and vertical flight trajectories of the drone in the DGPS mode.
Figure 17. Horizontal and vertical flight trajectories of the drone in the DGPS mode.

Figure 18.
Figure ENU errors
18. ENU errors in
in DGPS
DGPS mode.
mode.

Table 4. RMS and maximum of ENU errors in the DGPS mode.

Table 4. RMSFigure 18. ENU errors
and maximum of ENU inerrors
DGPSinmode.
the DGPS mode.
u-blox
Table 4. RMS and maximum of ENUu-bloxerrors in the DGPS mode.
EVK-6
EVK-6
East 0.8139 m
u-blox
RMS East
North 0.8139 m m
0.8039
RMS North EVK-6
Up 0.8039 m m
0.9259
East
Up 0.8139m
0.9259 m
East 4.8190 m
RMS North
East 0.8039 m
Max. North 4.8190 m
5.2008 m
Up
Max. North 0.9259
Up 5.2008 mm
3.7500
m
East
Up 4.8190m
3.7500 m
Max. North 5.2008 m
4.4. RTK Mode
4.4. RTK Mode Up 3.7500 m
4.4.1. Scenario I: RTK Performance of Different Receivers
4.4. RTK
4.4.1. Mode I: RTK Performance of Different Receivers
Scenario
In the third test (RTK mode), the recorded signals and RTCM v3 messages were input to u-blox
M8P and
4.4.1.In theSwiftNav
thirdI: test
Scenario RTKPiksi
(RTK Multi,
mode),and
Performance thethe
of performance
recorded
Different was
signals
Receiversandanalyzed
RTCM v3 using NMEA
messages filesinput
were obtained from
to u-blox
both receivers. Figures 19 and 20 show the estimated trajectories of the drone
M8P and SwiftNav Piksi Multi, and the performance was analyzed using NMEA files obtained fromand ENU errors with
the RTK
both In the third
fix type
receivers. test (RTK
in the 19
Figures RTK mode),
mode,
and the recorded
respectively.
20 show signals and RTCM
Tabletrajectories
the estimated 5 shows RMS v3 messages
anddrone
of the were
maximum input
values
and ENU to u-blox
of errors
errors with
M8P and SwiftNav Piksi Multi, and the performance was analyzed using NMEA files obtained from
both receivers. Figures 19 and 20 show the estimated trajectories of the drone and ENU errors with
Electronics 2019, 8, x FOR PEER REVIEW 15 of 22
Electronics 2019, 8, 1518
x FOR PEER REVIEW 15
14 of
of 22
21
the RTK fix type in the RTK mode, respectively. Table 5 shows RMS and maximum values of errors
the RTK direction
for each fix type inasthe RTK
well mode, respectively.
as Time-to-first-fix Table
(TTFF) and5 shows
RTK fixRMS
rate.and maximum
According values
to these of errors
results, the
for
for each direction
each direction
positioning as well as
as well of
performance Time-to-first-fix
as the
Time-to-first-fix (TTFF)
two receivers(TTFF) and RTK
and RTK
in the RTK fix
modefix rate.
arerate. According
According
similar to
at the 1tocmthese
these results,
results,
level, the
the
and Piksi
positioning
Multi usingperformance
positioning performance
dual frequency of
of the
the two receivers
two is
signals superiorin
receivers intothe
the RTK
RTK
M8P inmode
TTFF are
mode andsimilar
are similar
RTK fixat the
the 11 cm
atrate. level, and
cm level, and Piksi
Piksi
Multi using
Multi using dual
dual frequency
frequency signals
signals is
is superior
superior to
to M8P
M8P in
in TTFF
TTFF and
and RTK
RTK fix
fix rate.
rate.

Figure 19.
Figure 19. Horizontal
Horizontal and
and vertical
vertical flight
flight trajectories of the
trajectories of the drone
drone in
in the
the RTK
RTK mode.
mode.
Figure 19. Horizontal and vertical flight trajectories of the drone in the RTK mode.

Figure
Figure 20. ENU errors
20. ENU errors and
and RTK
RTK fix
fix type
type in
in RTK
RTK mode.
mode.
Figure 20. ENU errors and RTK fix type in RTK mode.
Table 5. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the RTK mode.
Table 5. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the RTK mode.
Table 5. RMS and maximum of ENU errors, and SwiftNav
TTFF and RTK fix rates in the RTK mode.
u-blox
SwiftNav
Piksi Multi u-blox
M8P
SwiftNav
Piksi Multi u-blox
M8P
East 0.0046 m 0.0047 m
RMS East
North Piksi Multi
0.0046
0.0042 mm M8P m
0.0047
0.0034 m
RMS East
North
Up 0.0046
0.0042 m
0.0566 m 0.0047
0.0479mm
0.0034
RMS North
Up
East 0.0042
0.0140m
0.0566 m 0.0034
0.0122mm
0.0479
Max. Up
East
North 0.0566
0.0140m
0.0140 m 0.0479
0.0112mm
0.0122
Max. Up
East
North 0.0900m
0.0140 m 0.0730mm
0.0122
0.0112
RTK Max.
Fix Rate North
Up 0.0140
0.0900 m
>99.99% 0.0112
0.0730 m
98.35%
RTK Fix Rate
TTFF Up 0.0900
>99.99% m
10 s 0.0730
98.35% m
14 s
RTKTTFF
Fix Rate >99.99%
10 s 98.35%
14 s
TTFF 10 s 14 s
Electronics 2019, 8, x FOR PEER REVIEW 16 of 22
Electronics 2019, 8, 1518 15 of 21
4.4.2. Scenario
Electronics 2019, 8,II: RTKPEER
x FOR Performance
REVIEW of Different Position Modes 16 of 22

4.4.2.The u-bloxII: M8P

Scenario receiver usedof for
RTK Performance the previous
Different Position RTK
Modes scenario supports different dynamic
4.4.2. Scenario II: RTK Performance of Different Position Modes
platform modes, such as portable, automotive, and airborne modes, and it adjusts the navigation
The u-blox M8P receiver used for the previous RTK scenario supports different dynamic platform
Theengine
filtering u-bloxadaptively
M8P receiver
to theused for the
expected previousenvironment.
application RTK scenario Tosupports
investigate different
the effect dynamic
of the
modes, such as portable, automotive, and airborne modes, and it adjusts the navigation filtering engine
platformposition
different modes, modes
such asonportable,
the RTK automotive,
accuracy, weand airborne
compared themodes, and results
positioning it adjusts theautomotive
of the navigation
adaptively to the expected application environment. To investigate the effect of the different position
filtering
and engine
airborne adaptively
modes using theto the expected
recorded application
signal environment.
and logged RTCM message,To investigate
althoughthe theeffect
data of the
were
modes on the RTK accuracy, we compared the positioning results of the automotive and airborne
different position modes on
acquired from an airborne system. the RTK accuracy, we compared the positioning results of the automotive
modes using the recorded signal and logged RTCM message, although the data were acquired from an
and Even
airborne modes
though bothusing the recorded
positioning modessignal and logged
calculated RTCM using
RTK positions message, although
the same GNSS thesignals
data were
and
airborne system.
acquired from
correction an airborne
messages obtainedsystem.
from the flight test, the wrong dynamic model of automotive mode
Even though both positioning modes calculated RTK positions using the same GNSS signals and
cannot Even though
provide both positioning
position modes calculated
results as accurately RTK
as the right positions
model using the
of airborne modesameas GNSS
shownsignals and
in Figures
correction messages obtained from the flight test, the wrong dynamic model of automotive mode
correction
21 and 22 and messages
Table 6.obtained
Since thefrom the flight
automotive test,assumes
mode the wrongthedynamic model of to
vehicle dynamics automotive
be plane-like,modeit
cannot provide position results as accurately as the right model of airborne mode as shown in Figures 21
cannotnot
could provide position
have solved theresults
RTK fixas accurately as the
solutions with right
the model
errors of airborne
of several metersmode
untilasthe
shown
drone inflew
Figures
up
and 22 and Table 6. Since the automotive mode assumes the vehicle dynamics to be plane-like, it
21the
to andaltitude
22 and of
Table 6. Since
116 m, whilethetheautomotive
airborne modemode assumesitsthe
acquired RTKvehicle dynamics
fix positions in tothebemiddle
plane-like,
of theit
could not have solved the RTK fix solutions with the errors of several meters until the drone flew up
could not have
ascending phase.solved the RTK fix solutions with the errors of several meters until the drone flew up
to the altitude of 116 m, while the airborne mode acquired its RTK fix positions in the middle of the
to the altitude of 116 m, while the airborne mode acquired its RTK fix positions in the middle of the
ascending phase.
ascending phase.

Figure 21. Horizontal and vertical flight trajectories of the drone in the automotive and airborne
modes.
Figure 21. Horizontal and vertical flight trajectories of the drone in the automotive and airborne modes.
Figure 21. Horizontal and vertical flight trajectories of the drone in the automotive and airborne
modes.

Figure 22.
Figure ENU errors
22. ENU errors and
and RTK
RTK fix
fix type
type in
in the
the automotive
automotive and
and airborne
airborne modes.
modes.

Figure 22. ENU errors and RTK fix type in the automotive and airborne modes.
Electronics 2019, 8, 1518 16 of 21
Electronics 2019, 8, x FOR PEER REVIEW 17 of 22

Table RMS and

Table 6.6. RMS and maximum of ENU
maximum of ENU errors,
errors, and
and TTFF
TTFF and
and RTK
RTK fix
fix rates
rates in
in the
the automotive
automotive and
and
airborne modes.
airborne modes.
Automotive
Automotive Airborne
Airborne
East
East 0.0052mm
0.0052 0.0047m
0.0047 m
RMS
RMS North
North 0.0042 mm
0.0042 0.0035
0.0035 m m
Up 0.0458 m 0.0473 m
Up 0.0458 m 0.0473 m
East
East 0.0153mm
0.0153 0.0130m
0.0130 m
Max. North 0.0132 m 0.0110 m
Max. North
Up
0.0132 m
0.0780 m
0.0110 m
0.0730 m
Up 0.0780 m 0.0730 m
RTK Fix Rate 93.32% 98.49%
RTK Fix Rate 93.32% 98.49%
TTFF
TTFF 4040s s 13 s
13 s

The results of this scenario show that the valid dynamic model enables the solution to follow the
trajectory, which
true trajectory, which implies
implies this
this method
method is
is helpful
helpful for
for evaluating
evaluating if the platform or model has been
appropriately for
designed appropriately for each
each kinematic
kinematic system.
system.

4.4.3. Scenario III: RTK Performance of Different

Different Constellations
Constellations with
with Tropospheric
TroposphericEffects
Effects
This methodology can easily compare the RTK performance of GPS-only with that of
GPS/GLONASS using using the
theRFRFsignal
signalwith
withthethe correction
correction dataset
dataset acquired
acquired fromfrom the drone
the drone flightflight
test.
test.
Figure 23 and Table 7 show that multi-constellation of GPS and GLONASS contributes to reducing
Figure 23 and Table 7 show that multi-constellation of GPS and GLONASS contributes to
the initialization time and improving the accuracy of the RTK RTK fixed
fixed solutions.
solutions. The Novatel flexpak6
receiver can also
also be
beconfigured
configuredtotoapply
applythe
thetropospheric
tropospheric correction to to
correction thethe
RTKRTK positioning.
positioning. Since
Since the
the altitude
altitude difference
difference between
between thethe reference
reference station
station and
and theUAV’s
the UAV’smaneuver
maneuverwas was only
only 33 m, the
positioning
positioning results
results difference
difference is
is not
not noticeable
noticeable while the tropospheric correction obviously shortened
the initialization time for the ambiguity fix. However, the proposed method makes it easy to see how
the receiver’s RTK performance of the dynamicdynamic vehicle
vehicle varies
varies according
according toto each
each configuration.
configuration.

ENU errors
Figure 23. ENU errors and
and RTK fix type of the multi GNSS and Tropospheric
Tropospheric correction
correction test.
test.
Electronics 2019,
Electronics 2019,8,
8,1518
x FOR PEER REVIEW 18
17 of 22
of 21

Table 7. RMS and maximum of ENU errors, and TTFF and RTK fix rates of the multi GNSS and
Table 7. RMScorrection
Tropospheric and maximumtest. of ENU errors, and TTFF and RTK fix rates of the multi GNSS and
Tropospheric correction test.
GPS Only GPS Only GPS+GLO GPS+GLO
(Tropo.GPSon) Only (Tropo.
GPS off)
Only (Tropo.
GPS+GLO on) (Tropo. off)
GPS+GLO
East (Tropo.
0.0053 m on) 0.0056 m off)
(Tropo. (Tropo.
0.0065 m on) (Tropo. off) m
0.0065
RMS North East0.0040 0.0053
m m 0.0040 m m
0.0056 0.0038
0.0065mm 0.0037
0.0065 m m
RMSUp North
0.0599 0.0040
m m 0.0040
0.0611 m m 0.0038m
0.0594 m 0.0037 m m
0.0581
Up 0.0599 m 0.0611 m 0.0594 m 0.0581 m
East 0.0191 m 0.0135 m 0.0189 m 0.0139 m
Max. North East0.0116 0.0191
m m 0.0135
0.0123 m m 0.0189m
0.0110 m 0.0139 m m
0.0129
Max. North 0.0116 m 0.0123 m 0.0110 m 0.0129 m
Up 0.0970 m 0.0890 m 0.1110 m 0.0850 m
Up 0.0970 m 0.0890 m 0.1110 m 0.0850 m
RTK Fix
RTK Fix Rate 98.61%98.61% 96.40%
96.40% 99.54%
99.54% 97.80%
97.80%
Rate
TTFF TTFF 13 s 13 s 32 s 32 s 55ss 20 s20 s

4.4.4.
4.4.4. Scenario IV: Performance
Performance of
of RTK
RTK Fed
Fed with
with Old
Old Correction
CorrectionMessage
Message
We
We also
also simulated aa correction
correction latency
latencycasecaseby byintentionally
intentionallyallowing
allowingthethe RTCM
RTCM Broadcaster
Broadcaster to
to feed
feed oldold RTCMmessages
RTCM messagesinto intothe
thereceiver.
receiver.FiveFiveandandten
ten second
second latencies
latencies were
were assigned for this
this
simulation
simulation andand the results
results were
were compared
compared with with the
the normal
normal RTK
RTK results,
results, whose
whose correction
correction ages
ages were
were
typically
typically 1 or 2 s old. Even though the ages of correction in the NMEA sentences were printed outout
old. Even though the ages of correction in the NMEA sentences were printed as
as 6 and
6 and 11 s11
assweas designed,
we designed, the errors
the errors were were less3than
less than mm 3inmmRMSinforRMS
eachfor each direction
direction once theonce the
position
position
had beenhad beentoturned
turned to RTK
RTK fixed fixed
mode asmode
shown asin shown in Figures
Figures 24–26
24, 25, and 26 and
and Table
Table 8.
8. The
The accuracy
accuracy
degradation
degradationofofthe theRTK
RTKfixed positions
fixed due due
positions to thetolatency under an
the latency open-sky
under were negligible,
an open-sky however,
were negligible,
the generalthe
however, performance during the during
general performance ambiguity the resolution
ambiguityprocess wasprocess
resolution affectedwas
by the latency.
affected byThe
the
maximum error of the RTK float solution fed by 10 s old correction message was
latency. The maximum error of the RTK float solution fed by 10 s old correction message was larger larger by 3 cm than
that
by 3without
cm thanlatency, and it took
that without 75 s and
latency, for the receiver
it took 75 swith old receiver
for the messageswith
to find
oldthe solutiontowhile
messages find the
the
general
solutionreceiver
while needed only 14
the general s. Other
receiver ambiguity
needed onlyfixing
14 s.performance, for examples
Other ambiguity TTFF and overall
fixing performance, for
ambiguity fixing rate, were also degraded due to the latency.
examples TTFF and overall ambiguity fixing rate, were also degraded due to the latency.

Figure 24. Horizontal

Figure 24. Horizontal and
and vertical
verticalflight
flighttrajectories
trajectoriesofofthe
thedrone
droneininthe
thecase
caseofofRTCM
RTCMmessages
messageswith
witha
delay of 5 and 10 s.
a delay of 5 and 10 s.
Electronics 2019, 8, 1518 18 of 21
Electronics 2019, 8, x FOR PEER REVIEW 19 of 22

Figure 25. ENU errors and RTK fix type in the case of RTCM messages with a delay of 5 and 10 s.

Table 8. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the case of RTCM messages
with a delay of 5 and 10 s.

No Latency 5s 10 s
East 0.0049 m 0.0049 m 0.0073 m
RMS North 0.0035 m 0.0065 m 0.0100 m
Up 0.0468 m 0.0467 m 0.0484 m
East 0.0130 m 0.0154 m 0.0217 m
Max. North 0.0110 m 0.0223 m 0.0302 m
Up 0.0730 m 0.0840 m 0.1090 m
RTK Fix Rate 98.49% 96.47% 80.98%
Figure 25. ENU errors and RTK fix type in the case
TTFF 13 sof RTCM messages
29 s with
75a sdelay of 5 and 10 s.
Figure 25. ENU errors and RTK fix type in the case of RTCM messages with a delay of 5 and 10 s.

Table 8. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the case of RTCM messages
with a delay of 5 and 10 s.

No Latency 5s 10 s
East 0.0049 m 0.0049 m 0.0073 m
RMS North 0.0035 m 0.0065 m 0.0100 m
Up 0.0468 m 0.0467 m 0.0484 m
East 0.0130 m 0.0154 m 0.0217 m
Max. North 0.0110 m 0.0223 m 0.0302 m
Up 0.0730 m 0.0840 m 0.1090 m
RTK Fix Rate 98.49% 96.47% 80.98%
TTFF 13 s 29 s 75 s

Figure 26. Time latency versus RMS values of horizontal and vertical errors in RTK mode.
Figure 26. Time latency versus RMS values of horizontal and vertical errors in RTK mode.
Table 8. RMS and maximum of ENU errors, and TTFF and RTK fix rates in the case of RTCM messages
with a delay of 5 and 10 s.

No Latency 5s 10 s
East 0.0049 m 0.0049 m 0.0073 m
RMS North 0.0035 m 0.0065 m 0.0100 m
Up 0.0468 m 0.0467 m 0.0484 m
East 0.0130 m 0.0154 m 0.0217 m
Max. North 0.0110 m 0.0223 m 0.0302 m
Up 0.0730 m 0.0840 m 0.1090 m
RTK Fix Rate 98.49% 96.47% 80.98%
TTFF 13 s 29 s 75 s
Figure 26. Time latency versus RMS values of horizontal and vertical errors in RTK mode.
Electronics 2019, 8, 1518 19 of 21

5. Conclusions
With more applications of GNSS and more users relying on position, the evaluation of positioning
performance is becoming increasingly more important. When evaluating the navigation performance
of various GNSS receivers, it is very important to provide the same signal environment to all the
receivers. However, it is difficult to perform the performance evaluation by mounting all receivers at
once because of the limitation of payload capacity of vehicles such as UAVs. Therefore, the performance
of GNSS receivers in dynamic modes has been assessed by comparing positioning results obtained
from each receiver loaded on vehicles flying along similar trajectories independently at different times.
However, such assessments do not ensure validness of evaluation for all receivers. To ensure evaluation
validity of the dynamic test, exhaustively numerous field-tests are essential in real-world conditions,
which impose a huge pressure on budgets and time.
In this study, we proposed a methodology to guarantee the validity of the dynamic performance
evaluation of GNSS receivers with only one flight test. We used RACELOGIC’s LabSat 3 Wideband
with the ability to record and replay the GNSS signals to provide the same signal environment to
multiple GNSS receivers. Furthermore, we developed a software that can store RTCM messages and
re-broadcast them to GNSS receivers using Python. In order to verify the suitability of the system, we
performed preliminary static tests and investigated whether performance degradation occurred due to
the use of LabSat 3 or the RTCM Logger and Broadcaster. The results of range and position domain
tests showed that performance degradation does not occur. After system verification, we performed a
drone flight test using DJI’s Phantom 3 Professional. During the dynamic test, GNSS live signals were
recorded in LabSat 3, and RTCM messages were also stored simultaneously by our software. Then,
we performed three re-radiation tests by applying the recorded signals and RTCM messages to three
different GNSS receivers (SwiftNav Piksi Multi, u-blox M8P, and u-blox EVK-6). In this manner, we
obtained trajectory estimates of the drone in different positioning modes (stand-alone, DGPS, SBAS,
and RTK) of all the GNSS receivers. Based on a total of six trajectories, we compared and analyzed the
performance of the GNSS receivers in various positioning modes.
The most significant advantage of our system is that performance analyses of unlimited GNSS
receivers and modes are possible with only one flight test. It enables positioning performance evaluation
of multiple receivers that cannot be loaded on a small vehicle. Moreover, it can save the cost and
time of multiple flight testing as only one dynamic test is required for evaluation. We expect our
suggested system to be useful as an accurate and convenient tool for evaluating the dynamic navigation
performance in the near future.

Author Contributions: All the authors have contributed to the presented work. C.L. designed the evaluation
system and implemented the RTCM Logger/Broadcaster software. H.Y. carried out static/drone flight tests and
data analysis. A.C. and C.-S.Y. provided valuable comments and critical feedback regarding the algorithm and
applications. B.P. suggested the original concept of the system and supervised its development and the direction of
the research. All authors participated in formulating the idea and in discussing the proposed approach and results.
Funding: This research was supported by the MSIT (Ministry of Science and ICT) of Korea under the ITRC
(Information Technology Research Center) support program (IITP-2019-2018-0-01423) supervised by the IITP
(Institute for Information and Communications Technology Promotion). This research was also supported by
the Unmanned Vehicles Advanced Core Technology Research and Development Program through the National
Research Foundation of Korea (NRF), Unmanned Vehicle Advanced Research Center (UVARC) funded by the
Ministry of Science and ICT, the Republic of Korea (2017M1B3A2A01049995).
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Hofmann-Wellenhof, H.; Lichtenegger, H.; Wasle, E. GNSS—Global Navigation Satellite Systems; Springer:
Vienna, Austria, 2008; pp. 55–85.
2. Misra, P.; Enge, P. Global Positioning System: Signals, Measurements, and Performance, 2nd ed.; Ganga-Jamuna
Press: Lincoln, MA, USA, 2006; pp. 147–185.
Electronics 2019, 8, 1518 20 of 21

3. Kaplan, E.D. Understanding GPS: Principles and Applications; Artech House: Boston, MA, USA, 1996;
pp. 301–375.
4. UNAVCO. UNAVCO Campaign GPS/GNSS Handbook; UNAVCO: Boulder, CO, USA, 2018; pp. 1–20.
5. Kim, M.; Seo, J.; Lee, J. A Comprehensive Method for GNSS Data Quality Determination to Improve
Ionospheric Data Analysis. Sensors 2014, 14, 14971–14993. [CrossRef] [PubMed]
6. Seo, J.; Walter, T.; Chiou, T.-Y.; Blanch, J.; Enge, P. Evaluation of Deep Signal Fading Effects Due to Ionospheric
Scintillation on GPS Aviation Receivers. In Proceedings of the Institute of Navigation GNSS, Savannah, GA,
USA, 16–19 September 2008.
7. Vinande, E.; Weinstein, B.; Chu, T.; Akos, D. GNSS receiver evaluation record-and-playback test methods.
GPS World 2010, 21, 28–34.
8. Cristodaro, C.; Ruotsalainen, L.; Dovis, F. Benefits and Limitations of the Record and Replay Approach for
GNSS Receiver Performance Assessment in Harsh Scenarios. Sensors 2018, 18, 2189. [CrossRef] [PubMed]
9. Hadas, T.; Kazmierski, K.; Sośnica, K. Performance of Galileo-only dual-frequency absolute positioning
using the fully serviceable Galileo constellation. GPS Solut. 2019, 23, 108. [CrossRef]
10. Kazmierski, K. Performance of Absolute Real-Time Multi-GNSS Kinematic Positioning. Artif. Satell. 2018,
53, 75–88. [CrossRef]
11. RACELOGIC. LabSat 3 Wideband. Available online: https://www.labsat.co.uk/index.php/en/products/labsat-
3-wideband (accessed on 10 December 2019).
12. RTCM Special Committee No. 104. RTCM Standard 10403.1. Radio Technical Commission for Maritime Services;
RTCM Special Committee: Arlington, VA, USA, 2011; pp. 378–383.
13. RTCM Special Committee No. 104. RTCM Standard 10402.3. Radio Technical Commission for Maritime Services;
RTCM Special Committee: Arlington, VA, USA, 2010.
14. Parkinson, B.W.; Enge, P. Differential GPS. In Global Positioning System: Theory and Applications; Parkinson, B.W.,
Spilker, J.J., Axelrad, P., Enge, P., Eds.; AIAA: Washington, DC, USA, 1996; pp. 3–50.
15. Park, B.; Kim, J.; Kee, C.; Cleveland, A.; Parsons, M.; Wolfe, D.; Kalafus, R. RRC Unnecessary for DGPS
Messages. IEEE Trans. Aerosp. Electron. Syst. 2006, 42, 1149–1160. [CrossRef]
16. Park, B. A Study on Reducing Temporal and Spatial Decorrelation Effect in GNSS Augmentation System:
Consideration of the Correction Message Standardization. Ph.D. Thesis, Seoul National University, Seoul,
Korea, 2008.
17. Lo Presti, L.; Falletti, E.; Nicola, M.; Troglia Gamba, M. Software Defined Radio Technology for GNSS
Receivers. In Proceedings of the IEEE Metrology for Aerospace (MetroAeroSpace), Benevento, Italy, 29–30
May 2014; pp. 314–319.
18. Dimc, F.; Bažec, M.; Borio, D.; Gioia, C.; Baldini, G.; Basso, M. An Experimental Evaluation of Low-Cost
GNSS Jamming Sensors. J. Inst. Navig. 2017, 64, 93–109. [CrossRef]
19. Borio, D.; Gioia, C. A Sum-of-Squares Approach to GNSS Spoofing Detection. IEEE Trans. Aerosp. Electron.
Syst. 2016, 52, 1756–1768. [CrossRef]
20. Borio, D.; Gioia, C.; Baldini, G.; Giuliani, R. A Real-time SDR Sanitizer for GNSS Interference Mitigation
in Road Transportation. In Proceedings of the 24th International Electrotechnical and Computer Science
Conference ERK 2015, Bernardin, Portoroz, Slovenia, 21–23 September 2015.
21. Hünerbein, K.; Lange, W.R. Testing Acquisition of GPS/GNSS Location and Velocity to Improve Safety in
Autonomous Driving. In Proceedings of the European Test and Telemetry Conference, Nürnberg, Germany,
26–28 June 2018.
22. Kouba, J.; Héroux, P. Precise Point Positioning Using IGS Orbit and Clock Products. GPS Solut. 2011, 5,
12–28. [CrossRef]
23. Kee, C.; Parkinson, B.W.; Axelrad, P. Wide Area Differential GPS. In Proceedings of the 3rd International
Technical Meeting of the Satellite Division of the Institute of Navigation, Colorado Spring, CO, USA, 19–21
September 1990; pp. 587–598.
24. Park, B.; Kee, C. The Compact Network RTK Method: An Effective Solution to Reduce GNSS Temporal and
Spatial Decorrelation Error. J. Navig. 2010, 63, 343–362. [CrossRef]
25. Kim, J.; Song, J.; No, H.; Han, D.; Kim, D.; Park, B.; Kee, C. Accuracy Improvement of DGPS for Low-Cost
Single-Frequency Receiver Using Modified Flächen Korrektur Parameter Correction. ISPRS Int. J. Geo Inf.
2017, 6, 222. [CrossRef]
Electronics 2019, 8, 1518 21 of 21

26. Song, J.; Park, B.; Kee, C. Comparative Analysis of Height-Related Multiple Correction Interpolation Methods
with Constraints for Network RTK in Mountainous Areas. J. Navig. 2016, 69, 991–1010. [CrossRef]
27. NTRIP—Networked Transport of RTCM via Internet Protocol. Available online: https://igs.bkg.bund.de/
ntrip/about (accessed on 13 October 2019).
28. Spirent. GNSS Record and Playback: Four Test Needs that Demand Richer Real World Testing, Spirent
eBook. Available online: https://www.spirent.com/assets/eb/eb_four-test-needs-that-demand-richer-real-
world-testing?submited=Y (accessed on 13 October 2019).
29. European Space Agency. Atomic Time. Available online: https://gssc.esa.int/navipedia/index.php/Atomic_
Time (accessed on 13 October 2019).
30. NMEA Data. Available online: https://www.gpsinformation.org/dale/nmea.htm (accessed on 24 May 2019).
31. RINEX. The Receiver Independent Exchange Format. Available online: ftp://igs.org/pub/data/format/
rinex301.pdf (accessed on 13 October 2019).
32. Trimble Geospatial. Trimble Business Center. Available online: https://geospatial.trimble.com/products-and-
solutions/trimble-business-center (accessed on 13 October 2019).

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).