Professional Documents
Culture Documents
Abstract Developing and testing algorithms for autonomous vehicles in real world
is an expensive and time consuming process. Also, in order to utilize recent advances
in machine intelligence and deep learning we need to collect a large amount of
annotated training data in a variety of conditions and environments. We present
a new simulator built on Unreal Engine that offers physically and visually realistic
simulations for both of these goals. Our simulator includes a physics engine that can
operate at a high frequency for real-time hardware-in-the-loop (HITL) simulations
with support for popular protocols (e.g. MavLink). The simulator is designed from
the ground up to be extensible to accommodate new types of vehicles, hardware
platforms and software protocols. In addition, the modular design enables various
components to be easily usable independently in other projects. We demonstrate
the simulator by first implementing a quadrotor as an autonomous vehicle and then
experimentally comparing the software components with real-world flights.
1 Introduction
1
2 Shah, Dey, Lovett, and Kapoor
Fig. 1 A snapshot from AirSim shows an aerial vehicle flying in an urban environment. The inset
shows depth, object segmentation and front camera streams generated in real time.
2 Related Work
While an exhaustive review of currently used simulators is beyond the scope of this
paper, we mention a few notable recent works that are closest to our setting and has
deeply influenced this work.
Gazebo [13] has been one the most popular simulation platforms for the research
work. It has a modular design that allows to use different physics engines, sensor
models and create 3D worlds. Gazebo goes beyond monolithic rigid body vehicles
and can be used to simulate more general robots with links-and-joints architecture
such as complex manipulator arms or biped robots. While Gazebo is fairly feature
rich it has been difficult to create large scale complex visually rich environments
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles 3
that are closer to the real world and it has lagged behind various advancements in
rendering techniques made by platforms such as Unreal engine or Unity.
Other notable efforts includes Hector [17] that primarily focuses on tight integra-
tion with popular middleware ROS and Gazebo. It offers wind tunnel tuned flight
dynamics, sensor models that includes bias drift using Gaussian Markov process and
software-in-loop using Orocos toolchain. However, Hector lacks support for popu-
lar hardware platforms such as Pixhawk and protocols such as MavLink. Because
of its tight dependency on ROS and Gazebo, it’s limited by richness of simulated
environments as noted previously.
Similarly, RotorS [7] provides a modular framework to design Micro Aerial Ve-
hicles, and build algorithms for control and state estimation that can be tested in
simulator. It is possible to setup RotorS for HITL with Pixhawk. RotorS also uses
Gazebo as its platform, consequently limiting its perception related capabilities.
Finally, jMavSim [1] is easy to use simulator that was designed with a goal of
testing PX4 firmware and devices. It is therefore tightly coupled with PX4 simula-
tion APIs, uses albeit simpler sensor models and utilizes simple rendering engine
without any objects in the environment.
Apart from these, there have been many games like simulators and training ap-
plications, however, these are mostly commercial closed-source software with little
or no public information on models, accuracy of simulation or development APIs
for autonomous applications.
3 Architecture
Fig. 2 The architecture of the system that depicts the core components and their interactions.
as inputs by the physics engine to compute the next kinematic state of bodies in the
simulated world. This kinematic state of bodies along with the environment models
for gravity, air density, air pressure, magnetic field and geographic location (GPS
coordinates) provides the ground truth for the simulated sensor models.
The desired state input to the flight controller can be set by human operator using
remote control or by a companion computer in the autonomous setting. The com-
panion computer may perform expensive higher level computations such as deter-
mining next desired waypoint, performing simultaneous localization and mapping
(SLAM), computing desired trajectory etc. The companion computer may have to
process large amount of data generated by the sensors such as vision cameras and
lidars which in turn requires that simulated environments have reasonable details.
This has been one of the challenging areas where we leverage recent advances in
rendering technologies implemented by platforms such as Unreal engine [11]. In
addition, we also utilize the underlying pipeline in the Unreal engine to detect col-
lisions. The companion computer interacts with the simulator via a set of APIs that
allows it to observe the sensor streams, vehicle state and send commands. These
APIs are designed such that it shields the companion computer from being aware
of whether its being run under simulation or in the real world. This is particularly
important so that one can develop and test algorithms in simulator and deploy to
real vehicle without having to make additional changes.
The AirSim code base is implemented as a plugin for the Unreal engine that can
be dropped in to any Unreal project. The Unreal engine platform offers an elaborate
marketplace with hundreds of pre-made detailed environments, many created using
photogrammetry techniques [18] to generate reasonably faithful reconstruction of
real-world scenes.
Next, we provide more details on the individual components of the simulator.
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles 5
Fig. 3 Vehicle model for the quadrotor. The four blue vertices experience the controls u1 , ..u4 ,
which in turn results in the forces F1 , .., F4 and the torques τ1 , .., τ4 .
AirSim provides an interface to define vehicle as a rigid body that may have arbitrary
number of actuators generating forces and torques. The vehicle model includes pa-
rameters such as mass, inertia, coefficients for linear and angular drag, coefficients
of friction and restitution which is used by the physics engine to compute rigid body
dynamics.
Formally, a vehicle is defined as a collection of K vertices placed at positions
{r1 , .., rk } and normals {n1 , .., nk }, each of which experience a unitless vehicle spe-
cific scaler control input {u1 , .., uk }. The forces and torques from these vertices are
assumed to be generated in the direction of their normals. However note that the
positions as well as normals are allowed to change during the simulation.
Figure 3 shows how a quadrotor can be depicted as a collection of four vertices.
The control input ui drives the rotational speed of the propellers located at the four
vertices. We compute the forces and torques produced by propellers using [4]:
2 1
Fi = CT ρωmax D4 ui and τi = 2
C pow ρωmax D5 ui .
2π
Here CT and C pow are the thrust and the power coefficients respectively and are
based on the physical characteristics of the propeller, ρ is the air density, D is the
propeller’s diameter and ωmax is the max angular velocity in revolutions per minute.
By allowing the movements of these vertices during the flight it is possible to sim-
ulate the vehicles with capabilities such as Vertical Take-Off and Landing (VTOL)
and other recent quadrotors that change their configuration in flight.
The vehicle model abstract interface also provides a way to specify the cross
sectional area in body frame that in turn can be used by physics engine to compute
the linear and angular drag on the body.
6 Shah, Dey, Lovett, and Kapoor
3.2 Environment
3.2.1 Gravity
While many models use a constant number to model the gravity, it varies in a com-
plex manner as demonstrated by models such as GRACE [23]. For most ground
based or low altitude vehicles these variations may not be important; however, it
is fairly inexpensive to incorporate a more accurate model. Formally, we approxi-
mate the gravitational acceleration g at height h by applying binomial theorem on
Newton’s law of gravity and neglecting the higher powers:
R2e
h
g = g0 · ≈ g0 · 1 − 2 .
(Re + h)2 Re
Here Re is Earth’s radius and g0 is the gravitational constant measured at the surface.
Accurately modeling the magnetic field of a complex body such as Earth is a com-
putationally expensive task. The World Magnetic Model (WMM) model [6] by Na-
tional Oceanic and Atmospheric Administration (NOAA) is one of the best known
magnetic models of Earth. Unfortunately, the most recent model WMM2015 is
fairly complex and computationally expensive for real-time applications.
We implemented the tilted dipole model where we assume Earth as a perfect
dipole sphere [14, pp 27-30]. This ignores all but the first order terms to derive mag-
netic field estimate using the spherical geometry. This model allows us to simulate
variation of the magnetic field as we move in space as well as areas that are often
problematic such as polar regions. Given a geographic latitude θ , longitude φ and
altitude h (from surface of the earth), we first compute the magnetic co-latitude θm
using:
cos θm = cos θ cos θ 0 + sin θ sin θ 0 cos(φ − φ 0 ).
Where θ 0 and φ 0 denote the latitude and longitude of the true magnetic north pole.
Then, the total magnetic intensity |B| is computed as:
Re 3 p
|B| = B0 ( ) 1 + 3 cos2 θm
Re + h
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles 7
Here B0 is the mean value of the magnetic field at the magnetic equator on the
Earth’s surface, θm is the magnetic co-latitude and Re is the mean radius of the
Earth. Next, we determine the inclination α and declination β angles using:
θ0
(
sin(φ − φ 0 ) cos
sin θ , if cos θm > sin θ 0 sin θ
tan α = 2 cot θm and sin β = m
0
cos(φ − φ 0 ) cos θ
sin θm , otherwise.
Finally, we can compute the horizontal field intensity (H), the latitudinal (X), the
longitudinal (Y ) and the vertical field (Z) components of the magnetic field vector
as follows:
The relationship between the altitude and the pressure of the Earth’s atmosphere is
complicated due to the presence of many distinct layers, each with its own individual
properties. First we compute Standard Temperature T and Standard Pressure P using
1976 U.S. Standard Atmosphere model [22, eq 1.16, 1.17] for altitude below 51
kilometers and switch to the model in [3, Table 4] beyond that up to 86 km. Then,
P
the air density is ρ = R·T (where R is the specific gas constant.)
The kinematic state of the body is expressed using 6 quantities: position, orientation,
linear velocity, linear acceleration, angular velocity and angular acceleration. The
goal of the physics engine is to compute the next kinematic state for each body given
the forces and torques acting on it. We strive for an efficient physics engine that can
run its update loop at high frequency (1000 Hz) which is desirable for enabling real-
time simulation scenarios such as high speed quadrotor control. Consequently, we
implement a physics engine that avoids the extra complexities of a generic engine
allowing us to tightly control the performance and make trade-offs that best meet
our requirements.
Since the vehicle moves in the presence of air, the linear and the angular drag has a
significant effect on the dynamics of the body. The simulator computes the magni-
tude |Fd | of the linear drag force on the body according to the drag equation [24]:
1
|Fd | = ρ|v|2Clin A.
2
8 Shah, Dey, Lovett, and Kapoor
Here Clin is the linear air drag coefficient, A is the vehicle cross-section and ρ is the
air density. This drag force acts in the direction opposite to the velocity vector v
Computing the angular drag for arbitrary shape remains complex and compu-
tationally intensive task. Many existing physics engines use a small but often an
arbitrary damping constant as a substitute for computing actual angular drag. We
provide simple but better approximations to model the angular drag.
Consider an infinitesimal surface area ds in the extremity of the body experienc-
ing the angular velocity ω. As the linear velocity dv experienced by ds is given by
rds × ω, we can now use the linear drag equation for ds [19, pp 160-161]:
1
|dF| = ρ|rds × ω|2Clin ds, where direction of dF is −rds × ω.
2
R
Now, the drag torque is computed by integrating over the entire surface: τd = S rds ×
dF. To simplify the implementation, we approximate the body of the vehicle as set
of connected faces which further can be approximated as a rectangular box for the
purpose of evaluating the integral.
3.3.2 Accelerations
In addition to the drag forces and torques, we also need to consider the forces Fi and
the torques τi present on the vehicle at the vertex located at ri relative to center of
gravity (see section 3.1). We thus compute the net force and torque as:
We obtain the linear acceleration by applying Newton’s second law and then adding
gravity vector to compute the net acceleration, a = Fnet /m + g. The angular acceler-
ation in body frame is given by Euler’s rotation equation: α = I −1 · (τnet − (ω × (I ·
ω))), where, I is the inertia tensor and ω is angular velocity, both in body frame.
3.3.3 Integration
We update the position pk+1 of the body at time k + 1 by integrating the velocity and
the initial position p0 . The first order integration algorithms such as Euler method
diverges quickly with unbounded error although very simple to implement. In our
implementation we use Velocity Verlet algorithm instead of Runge Kutta for its
computationally inexpensiveness and stability while still being second order method
[9]. Formally,
ak + ak+1 1
vk+1 = vk + · dt pk+1 = pk + vk · dt + · ak · dt 2
2 2
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles 9
The angular velocity is updated in similar manner as linear velocity however updat-
ing orientation isn’t straight forward. One of the approach is to maintains the ori-
entation as a rotation matrix that is updated every time step. However this causes a
slow drift which must be corrected by orthonormalization at regular intervals which
is expensive. Alternative approach is to maintain rotations as much more efficient
quaternions which are also numerically stable and trivially normalizable. One of
the problem, however, is that the orientation quaternion is maintained in the world
frame while the angular velocity is maintained in the body frame in our framework.
To update the orientation, we first compute the angle-axis pair (αdt , u) where αdt is
the angle traversed around unit vector u. We can compute the angle αdt = |ω| · dt
and axis by u = ω/|ω|. This allows us to compute equivalent change in quaternion
qdt representing the change in orientation in time dt. As noted before, qdt is in body
frame while qk in world reference frame. The problem now remains that of adding
qdt to qk to obtain qk+1 which can be proven to given by relationship qk+1 = qk · qdt .
3.3.4 Collisions
Unreal engine offers a rich collision detection system optimized for different classes
of collision meshes and we directly use this feature for our needs. We receive the im-
pact position, impact normal and penetration depth for each collision that occurred
during the render interval. Our physics engine uses this data to compute the collision
response with Coulomb friction to modify both linear and angular kinematics.[8]
3.4 Sensors
3.4.1 Barometer
To simulate barometer, we compute ground truth pressure using the detailed model
of atmosphere (sec 3.2.3) and model the drift in the pressure measurement over
time using Gaussian Markov process [20] for more realistic behavior in long flights.
Formally, if we denote the current bias factor as bk then the drift is modeled as:
dt
bk+1 = w · bk + (1 − w) · η, where: w = e− τ and η ∼ N(0, s2 ).
Here τ, is the time constant for the process and set to 1 hour in our model. η is
a zero mean Gaussian noise with standard deviation that can be selected using the
10 Shah, Dey, Lovett, and Kapoor
data available in [5]. This pressure p is then added with white noise drawn from
zero mean Gaussian distribution with standard deviation set from datasheet of the
sensor (such as MEAS MS56112). Finally we convert the pressure to altitude using
barometric formula used by the sensor’s driver:
" a·R #
T0 p −( g )
h= −1 ,
a p0
here T0 is the reference temperature (15 deg C), a = −6.5 × 10−3 is the temperature
gradient, g and R are the gravity and the specific gas constants, p0 is the current sea
level pressure and p is the measurement.
Gyroscope and accelerometers constitute the core of the inertial measurement unit
(IMU) [26]. We model these by adding white noise and bias drift over time to the
ground truth. For gyroscope, given the true angular velocity in body frame ω, we
compute the measurement ω out as,
Here parameters ra , bias b0 and the time constant for bias drift ta can either be ob-
tained from Allan variance plots or from datasheets. Accelerometer output is com-
puted in the similar manner except that we must first subtract gravity from the true
linear acceleration in the world frame and then convert the result to the body frame
before we add bias drift and noise.
3.4.3 Magnetometer
We use the tilted dipole model for Earth’s magnetic field 3.2.2, given the geographic
coordinates to compute the components of the ground truth magnetic field in body
frame and add the white noise as specified in the datasheet.
Our GPS model simulates latency (typically 200ms), slower update rates (typically
50 Hz) and horizontal and vertical position error estimate decay rates to simulate
gaining fix over time. The decay rate is modeled using first order low pass filter
individually parameterized for horizontal and vertical fix.
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles 11
Since advanced rendering and detailed environments have been a key requirement
for AirSim we chose Unreal Engine 4 (UE4) [11] as our rendering platform. UE4
offers several features that made it an attractive choice including it being an open
source and available on Linux, Windows as well as OSX. UE4 brings some of
the cutting edge graphics features such as physically based materials, photomet-
ric lights, planar reflections, ray traced distance field shadows, lit translucency etc.
Figure 1 shows a screen-shot from AirSim which highlight near photo-realistic ren-
dering capabilities. Further, Unreal’s large online Marketplace has various pre-made
elaborate environments, many of which are created using photogrammetry tech-
niques.
4 Experiments
15 6
10 5 Simulated X (m)
5 4 Real X (m)
0 3
Simulated X (m) 2
5 1
10 Real X (m) 0
150 10 20 30 40 50 10 5 10 15 20 25 30 35 40
15 6
10 5 Simulated Y (m)
5 4 Real Y (m)
0 3
Simulated Y (m) 2
5 1
10 Real Y (m) 0
150 10 20 30 40 50 10 5 10 15 20 25 30 35 40
Time (seconds) time (seconds)
(c) Space-Time Plot for Circle (d) Space-Time Plot for Square
Fig. 4 Evaluating the differences between the simulated and the real-world flight. In top figures,
the purple and the red lines depict the track from simulation and the real-world flights respectively.
factors such as integration errors, vehicle model approximations and mild random
winds.
Sensor Models: Besides evaluating the entire simulation pipeline we also inves-
tigated individual component models, namely the barometer (MEAS MS5611-
01BA), the magnetometer (Honeywell HMC5883) and the IMU (InvenSense MPU
6000). Note that the simulated GPS model is currently simplistic, thus, we only fo-
cus on the three more complex sensor models. For each of the above sensors we use
the manufacture specified datasheets to set the parameters in the sensor models.
• IMU: We measured readings from the accelerometers and gyroscope as the
vehicle was stationary and flying. We observed that while the characteristics
were similar when the vehicle was stationary (gyro: simulated variance 2.47e−7
rad2 /s2 , real-world variance 6.71e−7 rad2 /s2 , accel.: simulated variance 1.78e−4
m2 /s4 , real-world variance 1.93e−4 m2 /s4 ), the observed variance for an in-
flight vehicle was much higher than the simulated one (accel.: simulated 1.75e−3
m2 /s4 vs. real-world 9.46 m2 /s4 ). This is likely in real-world the airframe vi-
brates when the motors are running and that phenomenon is not yet modeled in
AirSim.
• Barometer: We raised the sensor periodically between two fixed heights: ground
level and then elevated to 178 cm (both in simulation and real-world). Figure 5(a)
shows both the measurements (green is simulated, blue is real-world) and we ob-
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles 13
1013.0 0.3
0.2 Real x-axis (Gauss)
Simulated x-axis (Gauss)
1012.8 0.1
0.0
0.1
1012.6
0.2
0.30
1012.4 10 20 30 40 50 60 70 80
0.3
0.2 Real y-axis (Gauss)
1012.2 0.1 Simulated y-axis (Gauss)
0.0
1012.0 Real Pressure (mbar) 0.1
Simulated Pressure (mbar) 0.2
1011.80 50 100 150 200
0.30 10 20 30 40 50 60 70 80
Time (seconds) Time (seconds)
Fig. 5 Figure 5(a) and 5(b) show that barometer and the magnetometer characteristics in simula-
tion closely resemble that of the real world.
serve that the signals have similar characteristics. Note that the offset between the
simulated and the real-world pressure is due the difference in absolute pressure
in the real-world and the one in the simulation. There is also a small increase in
the middle due to a temperature increase, which wasn’t simulated. Overall, the
characteristics of the simulated sensor matches well to the real sensor.
• Magnetometer: We placed the vehicle on the ground and then rotated it by 90◦
four times. Figure 5(b) shows the real-world and the simulated measurements
and highlight that they are very similar in characteristic.
AirSim offers hi-fidelity physical and visual simulation that allows to generate large
quantity of training data cheaply for building machine learning models. AirSim API
design allows developing algorithms against simulator and then deploy them with-
out change on real vehicles. The core components of AirSim including physics en-
gine, vehicle models, environment models and sensor models are designed to be
independently usable with minimal dependencies outside of AirSim and are eas-
ily extensible. AirSim is inspired by the goal of developing reinforcement learning
algorithms for the autonomous agents that can operate in the real world.
The task of mimicking the real-world in real-time simulation is a challenging
endeavor. There are a number of things that can be improved. Currently we do not
simulate richer collision response or advanced ground interaction models which
may be possible in future by implementing our physics engine interface for NVIDIA
PhysX and utilizing features such as physics sub-stepping. Also we do not simulate
various oddities in camera sensors except those directly available in Unreal engine.
We plan to add advanced noise models and lens models in future. The degradation
of GPS signal due to obstacles is not simulated yet which we plan to add using
ray tracing methods. We also plan to add more advanced wind effects and thermal
14 Shah, Dey, Lovett, and Kapoor
simulations for fixed wing vehicles. Our extensibility APIs have been designed with
above future work in mind and can also be used to realize other vehicle types.
References