International Conference on Cyberworlds 2008
Design of UAV Telepresence and Simulation Platform based on VR
Qiao Zhi-hua , Li Yi-bo, Kang Shao-peng, Zhu Qiong
Department of Auto-control, Shenyang Institute of Aeronautical Engineering, China
helloqiao2002@163.com
Abstract
A “man-in-the-onboard-loop” solving scheme is
proposed in order to overcome the disadvantages of
UAV control method and solve the problems on UAV
pilot training. This scheme is an integration platform
for UAV telepresence and simulation. It uses
“telepresence” virtual reality technology to merge the
dynamic images from airborne camera and the
geography data information generated by computer on
the ground so as to build a virtual “true” flight
environment. It gives the ground-operator much enough
sense display and interaction feedback to increase his
sense of immersion to control the UAV in a long
distance, thus enables the UAV to be a “man-control”
aircraft. Besides that, it provides a platform for
training UAV pilot in such applications as training and
simulation. We present the whole and detailed design
for the platform in both structure and function, and
discuss the key technology in it. Finally, a prototype
system is adopted to validate the feasibility of this
scheme.
1. Introduction
Unmanned Aerial Vehicle (UAV), also called
pilotless aircraft, means such a class of planes that no
human driver is required on them but to aviate
automatically under the control of some programs, or
be remotely controlled by operators who are on the
ground or on master plane. The UAVs can make long
distance navigation with the help of automatic pilot,
program control system, remote control and
measurement system, navigation system, etc. Compared
with their manned counterparts, the UAVs have the
advantages of small weight and space-occupancy, low
cost [1], good concealment, thus is ad hoc excellent in
executing the dull, dirty or danger (Tri-D) tasks.
At present, the UAVs are operated and controlled by
two principal modes, namely, the wireless telecontrol
and the program control that is realized by the computer
on the vehicles [2]. The former one, being the wider-
used mode, works through remote control by the
operators, who take the terminal of wireless controller
in hand to signal the wireless communication apparatus
on the UAVs. The mode of wireless telecontrol has
such disadvantages that the operation is quite difficult,
or even impossible in case the vehicle is out of the
range of visible distance. Furthermore, it needs
978-0-7695-3381-0/08 $25.00 © 2008 IEEE
DOI 10.1109/CW.2008.113
tremendous time-cost to train the operators.
Alternatively, the mode of program control that realized
by the computer on the vehicles means to install an
flight control computer on the UAV, where the
program-oriented control enable the UAV to aviate
autonomously and to fulfill specified flight missions.
This mode is a trend of the development of the UAVs
in the future with regard to operation and control.
However, there are also problems in this domain, since
the current mode of program control is merely effective
to special types of aircraft and specified tasks, with
unsatisfied stability and reliability during the operation
and control for the UAVs to carry out other errands.
In practice, the operation and control system for the
UAVs adopts the manner of combining vehicle-loaded
program control with human participating, with the
manner of manual operation being involved in most
cases. Accordingly, the pilot on the earth manipulates
the UAV in the way of remote control [3], which
requires the pilot, who has certain deal of operational
experience and favorable mental quality, to make full
aware of the pneumatic characteristics of the UAV.
Besides, such pilots confront more strict requirements.
As the ground control station accepts the “remote
surveyed” data from the UAV in real-time, the pilot
should conduct and control the UAV according to these
data, as well as other ones, such as those for task
scheme, navigation, information, etc. In the whole
procedure, there has not an intuitional and real flight
environment, so misoperation is an inevitable incident
for the pilot. In addition, the operation and control
system on UAVs can only implement the assignments,
but the function of simulative training is not provided.
Consequently, training the pilots is commonly
conducted on the actual UAVs, which can be destroyed
completely even if a very small misoperation happens
to the pilot, not to mention the life-span of the UAV
itself.
Aiming at the disadvantages of the foregoing
operation and control modes for the UAVs, this paper
proposes a solving scheme, which is realized on a
“man-in-onboard-loop” telepresence operation and
stimulation platform on the basis of the technology of
virtual reality (VR). On one side, the proposed scheme
adopts the VR technology to build up a circumstance of
“real” aircraft cabin in the control station. This cabin is
equipped with both control furnishings, e.g. accelerator
lever and control stick, and display instruments that
520
indicate the height, the speed, the gesture, the of the platform takes the usual PC network as the
parameters of the atmosphere, the states of the devices, configuration with the service mode of client/server
etc. By employing the “telepresence” technology, the where the whole platform is managed through
real-time images of the environment obtained by the supervising and examining the host human-computer
UAV are presented to the pilot without delay. As such, interaction PC. The system output the huge scenes by
operation in the virtual aircraft cabin is felt as steering multi-channel, while adopts flinging gigantic screen to
the UAV in the real cab. On the other side, the function enable the pilot to feel himself in the scenes, resulting
of flight simulation can be realized through loading the in the increase the feeling of immersion. Moreover, the
dynamic model of the aviation, which provides a prospect-generating system for the telepresence
simulation platform for the pilot training of the UAVs. operation on UAVs can hang the visions together in
The design of the structure, the functions, and the key virtue of the register technology for strengthened reality.
technologies in the system will be discussed in detail in
the following sections. 3. System structure and realization
platform
2. Telepresence Technology The telepresence and simulation platform for UAV
employs the Distributed Vega module in the virtual
The Virtual Reality (VR) Technique is a novel scenes driving software, the Vega. The application
integrated information technology that firstly appeared program runs in many low-layer systems to realize the
in the last years of 20th century and grew up from then function of multi-image hardware manage system,
on. It is a branch of senior human-computer interaction which takes ordinary PC network as its system structure,
technology that can imitate the behaviors of human’s as shown in Fig. 1. The solid and dashed lines with
various apperceive systems in entironment by means of arrows indicate the directions of real-time and un-real-
jointly utilizing such technologies as digital image time communications among all computers nodes.
processing (DIP), computer graphics (CG), multimedia, The system is designed to adopt multi-node
sensor, computer simulation (CS), etc. The VR can help Master/slave mode, where the node of Master acts as
the users to perceive and research the variation rules of the Master Node, and the other nodes are slave ones,
the objective world since they are put into the with each node being a whole PC. The nodes are
surroundings where they feel as to be personally on the connected to each other through 10/100MBps Ethernet
scene. The VR technique has three basic characteristics, card and Exchangers. In the whole system configuration,
namely, Immersion, Interaction and Imagination [4], of the network communication modules for distributed
which the most intrinsical one is the user’s immersing interactive simulation are the kernel for the operation of
the whole system.
to the virtual scenes. Depending on the different
manners of the users participating in VR and the degree
of immersion, the VR technique can be categorized into UAV load Solution Sound
four styles that are supported by desktop, immersion, calculation calculation simulation
strengthening and network distribution, respectively. Computer Computer Computer
Telepresence is a class of important virtual
environment that stands for a special reality Virtual Flight control
instruments Master computer
strengthening technique, by which the users, though Computer
being far away from a certain real site, can bind the
virtual objects generated by the computer to the three-
dimensional images of the real physical entities from Flight console
Scene Scene Scene including
the remote site. By this way, the users obtain adequate computer 03
computer 01 computer 02 operation stick
feeling on the real target surroundings where they feel and so on
themselves as stepping into it. Provided with interactive
feedback, the users can take part in the operation and Fig. 1 Structure of telepresence operation system for
control on the remote site. The telepresence operation The telepresence and simulation platform for UAV
and simulation platform for UAVs investigated in this illustrated in Fig.2 consists of a lot of subsystems such
paper make combined use of the network distribution, as UAV load subsystem, ground
immersion and strengthening technique of VR. Based communication/control subsystem, real-time simulation
on the technology of distributing VR, the general frame model subsystem, scenes generating subsystem, aerial

521
and electrical instruments simulation subsystem, semi-
reality simulation subsystem for flight control,
acoustics subsystem, multi-channel projection
subsystem and managing, examining and human-
computer interaction subsystem.
Fig. 2 General composing of the system
The system runs under the circumstance of Win2000
operating system, with the programme software of
VC++6.0. The Vega of MultiGen-Paradigm Company
is selected as the scene driving software, which is
combined with OpenGL to develop the interface design.
Multigen Creator is adopted in large territory modeling,
GL Studio of DISTI Company is used for virtual
instruments modeling and the interactive
communication is programmed by using WinSocket. In
addition, the 3DS max and Photoshop are used for the
work of assistant modeling and image processing.
4. Functions of the subsystems
As explained in Fig. 2, the managing, examining and
human-computer interactive module plays an decision-
making role in the whole platform. First of all, it judge
the system’s working state based on human-computer
interaction since UAV load subsystem, ground
communication/control subsystem and real-time
simulation model subsystem expel each other in
practice. For instance, when the UAV is conducted the
teleprecence operation, the real-time simulation model
is not working. Similarly, the UAV system would not
continue to run in the process of training simulation. As
a result, the UAV telepresence operational platform and
the simulation platform can be integrated successfully.
The UAV load subsystem, ground
communication/control subsystem are comprised of
load calculation computer, digital transmission station,
the ground information processing and other
components. These two subsystems have the functions
of transferring the flight parameters of the UAV, as
well as the information from the imagine sensor on it,
to ground in real-time, and then classifying and
retransmitting them to management system and scenes
generating system.
522
The module that carries the real-time simulation
model establishes the kinetic and dynamic model for
the UAV so that carries out simulative calculation
according to the control instructions input by the user.
The results concerning to the data that indicating the
real-time gesture of the virtual aircraft can serve as a
substitute of the UAV system, where the objective of
pilot training is met with this help.
In the simulation platform, the scenes generating
system provides the pilot with virtual prospects of
targeted surroundings, such as the airport, the runway,
the urban area, the rural area, the river, the mountain
and the condition of weather, all appearing the same as
the real ones. In the telepresence operation platform, the
scenes generating system will piece the real picrtures
sent from the image sensor on UAV together with the
virtual landform, in other words, combine the real
signals in sky with virtual scenes on ground. Given
such information, the pilot would feel as onboard
personally when operating the UAV remotely. During
system design for the data station, we take full account
of the bandwidth of the communication links, which are
mainly used to ensure the reliable transmission of the
data of control and assignments, with the delivery of
scenes data being in the second place. In case the
scenes cannot be transmitted, it is the VR technique, not
the telepresnece, will be adopted by the system to
realize the “man-in-onboard-loop” operation merely
based on the data of non-video sensors on UAV and the
weather status of the region where the vehicle concerns.
However, at the stages of fly-off and landfall, the scene
transmission should be ensured prior to any other tasks.
Featuring no human onboard, the UAV can achieve
strong-forced flight, producing high acceleration and so
on. Hence, the real aircraft cabin for human operation is
abandoned, for which there is no movable cabin system
herein and the semi-reality simulation system for flight
control merely includes the aviation operational lever,
the aerial and electrical instruments and the FPD.
Among them, the aviation operational lever should be
prepared with the real systems of force feedback and
flight parameters, which are realized by the aerial and
electrical instruments simulation subsystem through
computer simulation.
The acoustic system imitates a variety of sounds that
exist during navigation so as to enhance the feeling of
immersion. It should be noticed that the scenes
generating system can simulate living climate situations
such as wind, rain, snow and midst, in superb
accordance with the weather at the place the aircraft
passing through.
5. Key techniques and realization
DEM Database Terrain Model

In the whole design process about this platform, from 3D

the establishment of the model database to the display Data Preprocessing geographic
of three-dimensional visual, and to the UAV environme
ntal model
telepresence, many technologies are applied. The main Extract culture
techniques includes dynamic environment modeling, Texture Database character
visual real-time solution, Web-based remote control
technology, limited bandwidth data communications Construct 3D Model
link video transmission technology, predict Satellite picture of characteristic
substances
transmission delay to the effect of remote control and
predict control technology and multi-sensor images Special Effect
mosaic technology. Now how to establish the visual Special Effect Model
database and transform the coordinate in three-
Meteorological
dimensional visual display will be discussed. UAV Flight Visual Database
Condition

5.1. Establishing visual database Fig. 3 Modeling process of visual database

The UAV flight range is quite great, so large-scale dimensional scene, the different perspective of the
terrain simulation must be adopted in the UAV flight scenes View 4 (c), (d) below.
simulation [5]. In this system, the real-time 3D viewing
terrain model database by using the digital terrain 5.2. Transforming Coordinate inThree
elevation data and satellite remote sensing images was Dimensional Scene Display
established, and the terrain database management
module of the Vega is used for the system to manage Real UAV usually has Inertial Navigation System
the whole terrain. and GPS System in the process of UAV ’ s
In terrain simulation,Liaoning Province terrain was
used. In the flight environment simulation, two kinds of
light effects ,day and night , and rain, snow, frog,
cloudy day and fine etc weather effects were simulated.
The process of flight take-off and landing can be
simulated and the given location and angel information
also can be received and corresponding view was
switched real-time. The real views even can be (a) UAV model effect in (b) virtual instrument
Creator
provided in this system, including runway, building,
cloud, landform and physiognomy etc., which can make
the “pilot” who uses this system has the feeling with
self participations.
The modeling process of its view environment
database is showed in Figure 3.
In the modeling simulation of the whole system, In
addition to scenes modeling, but also UAV motion (c) Viewpoint placed in the font side of UAV
body model was established, the effect of UAV model
shown in Figure 4 (a), In addition, the system also
requires the real aircraft cockpit simulation, to display
flight parameters and status. Although the real UAV
cockpit and instrument was not exited, but the visual
simulation systems by means of ground simulation of
the cockpit and instrument provided UAV navigator
more sense of immersion, the effect of virtual (d) Viewpoint bound in UAV
instruments was shown in Figure 4 (b), in three- Fig. 4 Effect of modeling the UAV

523
telepresence and control by airborne computer sends
latitude and longitude coordinates of aircraft to control
station immediately. But Three-dimensional Display
System based on Vega by the rules are Right Hand
Cartesian coordinate system. In order to correct
locate 、 display and describe trajectory in three-
dimensional scene , the system must converse the
latitude _longitude information which received from the
informations of aircraft’s position to Cartesian
coordinate. The same problems exist with flight
simulation.
Conversion of latitude _longitude to Azimuth
distance usually use Vincenty Formula 、 Bowring
Formula 、 TDSL Formula and so on. But a higher
accuracy method—Space Precision Location ( SPL)
was adopted in this system. It is a practial algorithms
which based on different space conversion. It avoids
errors above TDSL Formula, which are caused by
solving some of some questions throught projection
method. The Conversion Formula as follows [ 6 ] :
Definition: the space different Coordinate conversion
matrix is:
⎡ − sin j cos j 0 ⎤
⎢
D ( j, w) = ⎢sinw cosj - sinw sin j cosw⎥⎥ (1)
⎢⎣cosw cosj cosw sin j sinw ⎥⎦
⎡ cosj cosj ⎤
F ( j, w) = ⎢⎢ cosw sin j ⎥
⎥ (2)
⎢⎣ sinw⎥⎦
parameter declaration： j represents longitude；w
represents latitude ； h represents elevation ； f
represents polar radius ； R (w) representse earth
longitude radius.
⎡ R ( w) + h ⎤
⎢ ⎥
T ( j, w, h) = F ( j , w) • ⎢ R( w) + h ⎥
⎢ R( w) • (1 − f ) + h ⎥ 2
⎣ ⎦
represents geocentric coordinates。
When the two places latitude _longitude position
coordinates and elevation, that was A( j 1, w1 , h1 ) and
B ( j 2 , w2 , h2 ) ,were given, how to calculate cartesian
coordinate of B( x, y, z) relative to A. The Formula was
given:
⎡x ⎤
B ⎢⎢ y ⎥⎥ = D( j 1 , w1 ) • [T ( j 2 , w 2 , h 2 ) − T ( j 1 , w1 , h1 ) ]
⎢⎣ z ⎥⎦
System selects flight initial position through the main
control computer input and the aircraft take the point of
takeoff as the origin of database(the origin of database
524
can be fixed ,but also can be controlled by
programs),for example, Place A （ East
longitude 123 ° ， north latitude 41 ° ， hight 100
meters）, All the targets, topography of the longitude
and latitude location information can reference point A
to conversion.
6. Conclusion
In this paper, the “man-in-the-onboard-loop” UAV
telepresence and simulation platform based on virtual
reality technology proposed is validated feasible in
prototype system, although some key technologies such
as image mosaic and delay predict are still in studying.
On August 8 2005, the America Ministry of National
Defence published the latest edition of UAV roadmap,
that is, Unmanned Aerial Vehicle System Roadmap
2005~2030, and the edition put forward clearly that
realizing “man-in-the-onboard-loop” UAV telepresence
by use of “telepresence” virtual reality technology is a
trend. On the one hand, the research on platform in this
paper can improve the stability and reliability of UAV
telepresence, reduce the difficulties of controlling and
realize controlling UAV on a long distance, on the other
hand, provide the training of UAV pilot with effective
support.
Acknowledgment
This project is supported by the ShenYang Talent Fund
Project (Grant No.06syrc03)
References
[1] Lamont, G.B.; Slear, J.N.; Melendez, K. UAV Swarm
Mission Planning and Routing using Multi-Objective
Evolutionary Algorithms[J]. Computational Intelligence in
Multicriteria Decision Making, IEEE Symposium on 1-5
(3) April 2007 Page(s):10 - 20
[2] Myeong-Wuk Jang; Agha, G.Scalable agent distribution
mechanisms for large-scale UAV simulations[J]. Integration
of Knowledge Intensive Multi-Agent Systems, 2005.
International Conference on April 18-21, 2005 Page(s):85 -
90
[3] Rasmussen, S.J.; Chandler, P.R. MultiUAV: a multiple
UAV simulation for investigation of cooperative control[J].
Simulation Conference, 2002. Proceedings of the Winter
Volume 1, 8-11 Dec. 2002 Page(s):869 - 877 vol.1
[4] Cummings, M.L.; Mitchell, P.J. Predicting Controller
Capacity in Supervisory Control of Multiple UAVs[J].
Systems, Man and Cybernetics, Part A, IEEE Transactions on
( 4 ) Volume 38, Issue 2, March 2008 Page(s):451 - 460
[5] Nigam, Nikhil; Kroo, Ilan. Persistent Surveillance Using
Multiple Unmanned Air Vehicles[J]. Aerospace Conference,
2008 IEEE 1-8 March 2008 Page(s):1 - 14
[6] Wu Han-bao,Zhou Yong-feng and Sun Wei-min. Three
Dimensional Problem Spaces Alignment Algorithm Research
[J]. Ship Electronic Engineering, 2003, (5): 28-31.