Applied Soft Computing 110 (2021) 107602

Contents lists available at ScienceDirect

Applied Soft Computing

journal homepage: www.elsevier.com/locate/asoc

3D robotic navigation using a vision-based deep reinforcement

learning model
∗
P. Zieliński , U. Markowska-Kaczmar
Department of Computational Intelligence, Wroclaw University of Science and Technology, 27 Wybrzeże Wyspiańskiego st., 50-370 Wrocław, Poland

graphical abstract

article info a b s t r a c t

Article history: In this paper, we address a problem of vision-based 3D robotic navigation using deep reinforcement
Received 29 July 2020 learning for an Autonomous Underwater Vehicle (AUV). Our research offers conclusions from the
Received in revised form 13 April 2021 experimental study based on one of the RoboSub 2018 competition tasks. However, it can be
Accepted 5 June 2021
generalized to any navigation task consisting of movement from a starting point to the front of the
Available online 15 June 2021
next station. The presented reinforcement learning-based model predicts the robot’s steering settings
MSC: using the data acquired from the robot’s sensors. Its Vision Module may be based on a built-in
65D19 convolutional network or a pre-trained TinyYOLO network so that a comparison of various levels of
97R40 features’ complexity is possible. To enable evaluation of the proposed solution, we prepared a test
environment imitating the real conditions. It provides the ability to steer the agent simulating the
Keywords:
AUV and calculate values of rewards, used for training the model by evaluating its decisions. We study
Deep reinforcement learning
A2C the solution in terms of the reward function form, the model’s hyperparameters and the exploited
PPO camera images processing method, and provide an analysis of the correctness and speed of the model’s
Vision-based navigation functioning. As a result, we obtain a valid model able to steer the robot from the starting point to the
YOLO destination based on visual cues and inputs from other sensors.
CNN © 2021 Elsevier B.V. All rights reserved.

1. Introduction in an underwater environment independently, constitute with

Autonomous Underwater Vehicles (AUVs), i.e. unmanned ve-
hicles and robots, able to navigate and perform various tasks
∗ Corresponding author.
E-mail address: p.zielinski@pwr.edu.pl (P. Zieliński).
their remotely operated counterparts (ROVs) a group, universally
referred to as Unmanned Underwater Vehicles (UUVs) and are
widely used both in industry and research applications, espe-
cially in particularly difficult or dangerous tasks. To operate fully
autonomously, the robot has to move between each key point,
performing various tasks [1]. Its controller should plan the robot’s

https://doi.org/10.1016/j.asoc.2021.107602
1568-4946/© 2021 Elsevier B.V. All rights reserved.
P. Zieliński and U. Markowska-Kaczmar
movement efficiently, avoiding existing obstacles and optimizing
the path to minimize its length and required time. Therefore, the
problem of underwater robot navigation is a crucial part of an
autonomous controller, on which the ability to perform further
tasks depends most.
Underwater robots are often designed and applied for school
and academic competitions, where they have to perform tasks
similar to those of professional vehicles. However, their environ-
ment of operation is often limited to smaller areas, such as lakes,
basins, or special facilities. One of these competitions, RoboSub,
is organized yearly in San Diego, on a TRANSDEC testing facility;
AUVs prepared for this competition perform a series of tasks,
imitating current research conducted in the local US Navy unit.
The main aim of this research is to explore the feasibility of
applying deep reinforcement learning for training a vision-based
navigation controller of an autonomous underwater vehicle. The
research was inspired by the RoboSub competition, attended by
one of the authors, where the ability to navigate between each
task is essential. Navigation can be defined as the ability to
determine the robot’s position and to plan a suitable and safe
path to the goal location [2]. A crucial aspect of robotic navigation
is the ability to sense objects around the robot and to know its
relative pose. The ability to navigate an agent in a 3D environ-
ment from a starting point towards the target object without
specifying the expected path, using only visual observations and
sensors readings, seems attractive not only for AUVs, but also for
similar tasks performed by unmanned aerial vehicles (UAVs), or
intelligent agents in 3D games.
Depending on the nature of the environment, various strate-
gies might be used for path planning. With an accurate model
of the world and tractable action space, classical optimization
methods perform robustly, for example, Dijkstra algorithm [3], A*
algorithm [4], Artificial potential field method [5]. In case of more
complex environments or large action spaces, the performance of
these methods degrades rapidly. Therefore, some studies present
evolutionary-based approaches to solve this problem, such as
multiobjective evolutionary algorithm [6], a combination of an
evolutionary algorithm and game theory [7] or hybrid Particle
Swarm (PSO) and Modified Bat algorithms [8].
However, evolutionary algorithms are often difficult to tune,
which is why currently reinforcement learning (RL) is used in
many fields of robotics. Thanks to deep neural networks’ capacity
for processing high-dimensional data, deep reinforcement learn-
ing (DRL) models can use raw sensor data without manual feature
engineering. Among many of the methods, the most frequently
tested one in the context of AUV navigation is Deep Deterministic
Policy Gradients (DDPG [9]): [10–14]. You et al. [15] leverage
DDPG for 2D target tracking in an unmanned combat air vehicle.
Authors in [16,17] and [18] applied Deep Double-Q Network
(D3QN [19]) for path planning, and in [17] hierarchical Deep Q-
Network (h-DQN [20]) was used for the same task. In [21], target
search tasks were solved using Asynchronous Advantage Actor–
Critic (A3C [22]) model. Sporyshev et al. [23] solve a problem
of search and surveillance mission by multiple cooperative AUVs
using Proximal Policy Optimization (PPO [24]). This DRL method
is also used in [25] for mapping from the images produced by the
cameras to the velocity of AUVs and in [26] for a motion planning
system, which avoids obstacles and can reach multiple target
points. In [27], authors present how to train an agent to tackle
a hybrid objective of 3D path-following and collision avoidance
for an AUV.
The aforementioned DRL methods utilize inputs from ad-
vanced sensors, such as sonars, and did not rely on visual cues.
Lately, a combination of DRL and deep learning vision methods
become more and more popular in the robot navigation domain.
For example, [28] used such an approach for robotic arm motion
2
Applied Soft Computing 110 (2021) 107602
control, [29] for obstacle avoidance, [30] for navigating a robot to
a target given by an image and [15] for 2D target tracking. These
papers focus only on two-dimensional problems; the problem
of obstacle avoidance in a 3D environment was recently tackled
by [31].
To sum up, there are several brand new survey papers. [32]
discusses in detail state of the art deep reinforcement learning,
imitation learning, and transfer learning in robot control. [33] de-
scribes the fundamental background behind sim-to-real transfer
in deep reinforcement learning and reviews the main methods
being utilized. [34] overviews commonly perceived challenges in
deep RL and how they can be addressed.
In this research, we concentrate on an analysis of a vision-
based approach to the problem of 3D robotic navigation, where
we compare the proposed deep reinforcement learning-based
model’s effectiveness depending on the visual feature embed-
dings used by the controller. Furthermore, we present a study
of several reward functions and their influence on the process
of training and achieved results. The solution presented in the
paper has been investigated in terms of the efficiency of the
training process and the correctness of functioning, with varying
environmental conditions and for various hyperparameters’ set-
tings. The research is based on a specifically designed simulation
environment, in which training and evaluation of models were
conducted.
We summarize our contributions as follows:
• We present a novel, vision-based approach for 3D naviga-
tion, which utilizes data from the robot’s sensors and an
object detection model for target object location. The pro-
posed solution is trained as a deep reinforcement learning
model and can steer the robot to navigate from the starting
point to the target object.
• We analyze the reward function used for training the model.
We propose four relevant components and prove that each
one is necessary to achieve the correct behavior of the
controller.
• We test various image processing methods and show how
the level of visual features’ complexity processed by the
model influences its performance.
• We evaluate the solution regarding its success rate and
achieved reward value, as well as the model’s prediction
speed.
• We provide a simulation environment for similar experi-
ments.
The rest of the paper is organized as follows. Section 2 states
the problem of AUV navigation. The proposed solution is shown
in Section 3, while Section 4 presents the experiments con-
ducted during the research and discusses their outcomes. Finally,
Section 5 shows our conclusions.
2. Problem description and formulation
In the research we consider one of the tasks from the RoboSub
2018 competition, which consists of navigating from the start-
ing area (dock) towards the gate, indicating the entrance to the
competition task sequence. Fig. 1 illustrates point configuration
for this task; point P0 refers to the starting point, whereas P∗ is
the target position. It is assumed that initially, the target object
is not in the robot’s field of vision and is located several meters
away from the starting point. The navigation problem is defined
as the movement from the dock towards the gate, stopping at a
point, where the gate fits the sight of the camera and the robot is
situated centered, in front of it. This scenario provides necessary
robustness, since a situation where the robot ends its navigation
P. Zieliński and U. Markowska-Kaczmar
Fig. 1. Plan view of an example configuration of a starting point P0 and a
destination point P∗ for the examined task.
in front of the next task’s station is common for other competition
tasks from RoboSub.
The goal of the robot’s controller is to choose the velocity set-
tings so that the robot, starting from point P0 , reaches destination
point P∗ in the shortest possible time. Point P0 is defined as the
place where the robot ended the previous task, or from where it
starts the challenge, whereas point P∗ indicates the target, which
is the place of executing the next task; in the case of the entrance
gate, it is any point which allows the robot to go under the gate by
moving forward. Both points are given as a four-element vector
(Eq. (1)), relative to the center of the competition facility.
P = x, y, z , ϕy : x, y, z , ϕy ∈ R
{ }
where:
x, y, z – position on axes X , Y , Z ,
ϕy – heading angle (angle about vertical axis – we ignore the
remaining angles since the AUV was designed not to be able
to rotate around them).
To solve this navigation problem, the robot’s controller has to
find the best move to the target using the information delivered
from sensors. Formally, we can say that the main task of the
controller model is to estimate the settings vector Y based on the
input data X received from sensors, as it is expressed by Eq. (2).
model
X−
−−→ Y
The AUV was equipped with three high-definition color cam-
eras. Each one generates an image tensor Ximg of shape 1280 ×
720 × 3, containing integers in the range [0, 255] (Eq. (3)).
Ximg = x(i,j,k) ∈ [0, 255] = Z 1280×720×3
{ }
Our model utilizes one of the cameras, mounted on the front of
the robot and facing forward.
Apart from the cameras, one of the crucial sensors of the
AUV is an AHRS (Attitude and Heading Reference System), an in-
tegrated sensor composed of a gyroscope, an accelerometer and
a compass, which allows accurate orientation and both linear
and angular acceleration measurement. The controller analyses
following three-element vectors (with real values per each axis):
linear acceleration vector Xa (Eq. (4)), angular velocity vector Xω
(Eq. (5)) and rotation vector Xϕ (Eq. (6)) with values in the range
(−180, 180].
[m]
Xa = ax , ay , az : ax , ay , az ∈ R
{ }
s2
Applied Soft Computing 110 (2021) 107602
Fig. 2. Settings vector components: longitudinal velocity vz , lateral velocity vx ,
vertical velocity vy and yaw velocity ωy .
[ ]
rad
Xω = ωx , ωy , ωz : ωx , ωy , ωz ∈ R
{ }
2
(5)
s
Xϕ = ϕx , ϕy , ϕz : ϕx , ϕy , ϕz ∈ R [◦ ]
{ }
(6)
Finally, using a depth sensor of the robot, we utilize the
current depth value Xd , measured in meters (Eq. (7)).
Xd = d : d ∈ R, d ≥ 0 [m] (7)
The settings vector Y estimated by the model consists of four
real values in the range [−1, 1] (according to Eq. (8)). All these
values are illustrated in Fig. 2:
• longitudinal velocity setting vz (vz = 1 stands for maximal
forward velocity),
• lateral velocity setting vx (vx = 1 stands for maximal velocity
(1)
to the right),
• vertical velocity setting vy (vy = 1 stands for maximal
emergence speed),
• yaw velocity setting ωy (ωy = 1 stands for maximum clock-
wise angular velocity).
Y = vz , vx , vy , ωy : vz , vx , vy , ωy ∈ R
{ }
(8)
In the original AUV, these settings are treated as target values
of velocities in the respective directions; they were used by the
robot’s internal controller to calculate the control vector of the
thrusters. In this research we assume, that this internal controller
is able to set the actuation of the thrusters using the given
velocity setpoint; this design demands from the model to learn
(2)
to handle the inaccuracies of the internal controller, as well as
the delay needed to reach the set velocity.
3. The method
(3)
The high-level structure of the controller is presented in Fig. 3.
It is based on a deep reinforcement learning model. It receives
an input vector X that contains information from all sensors and
returns a settings vector Y.
The image from the front camera is processed by the Vision
Module first. This module is responsible for feature extraction
from an image and dimensionality reduction. Its output is con-
catenated with data from other sensors (i.e. linear acceleration,
angular velocity, current rotation and depth) in the next module
that also performs further processing. A recurrent network is
applied as the final module of the controller (Time-Series Analysis
Module) to discover the temporal relationship between streams
of data (images and other sensors inputs). The output of this
(4) module is the settings vector Y.
3
P. Zieliński and U. Markowska-Kaczmar
Fig. 3. High-level controller structure, showing its main modules; X and Y are the controller’s input and output (Eq. (2)), Ximg is the image tensor (Eq. (3)), Xa
refers to linear acceleration vector (Eq. (4)), Xω is the angular velocity vector (Eq. (5)), Xϕ is the orientation vector (Eq. (6)) and Xd refers to the current depth value
(Eq. (7)).
One of the crucial characteristics of the problem is its high
complexity. The controller has to process multidimensional in-
put data: a stream of images and numerical values. The most
straightforward approach is to train the entire model at once
using raw data from the environment, expecting it to discover the
existing relationships between particular input values and correct
settings, leading to the target point. On the other hand, parts of
the model could be extracted and trained separately, which could
simplify the training task. In our solution we investigate both of
these approaches.
The controller’s model was trained in the actor–critic strat-
egy [35], based on the A2C (Advantage Actor–Critic) framework
[22]. This approach utilizes two models; the actor (policy gradient
method) is used to control the agent behavior, while the critic
(action-value function method) evaluates the actor’s decision.
Both models are trained in parallel: the actor policy πθ (s, a) is
trained to maximize the value of rewards received for actions
taken by the agent and the critic function Qω∗ (s, a) is optimized
to minimize reward function approximation error. Here, s refers
to the state vector (in our case, equivalent to the input vector X
containing the information from the environment: visual obser-
vations Ximg and sensors readings Xa , Xω , Xϕ , and Xd ), and a is
the action vector (which refers to the output settings vector Y).
With the A2C, the model is trained using multiple agents, which
interact with individual environments, using their copies of the
global network. Agents train their individual network’s weights
independently, and at the end of each set of episode, the global
network is updated synchronously.
The original implementation of the A2C uses a vanilla policy
gradient method for controlling the agent. In our research we de-
cided to replace it with PPO (Proximal Policy Optimization) [24].
A classic gradient ascent method is here enhanced using trust
regions, which limit the maximal update step size. The size of
the trust region is set using the clipped surrogate objective func-
tion. This approach efficiently restricts policy weights changes,
improving the stability of the training process.
Next, the solution’s modules are described in detail.
3.1. Vision module
The Vision Module is presented in detail in Fig. 4. As an input,
it receives the data from the camera mounted on the front of the
robot. First the original input Ximg of shape 1280 × 720 × 3 is
rescaled to shape 416 × 416 × 3 of integers in the range [0, 255].
Because the AUV’s camera resolution is constant, we decided not
to crop the image additionally, as the deformation resulting from
resizing is invariable.
As it is shown in Fig. 4, the Vision Module offers two methods
of image processing. They can be applied in various configura-
tions (using both of them or separately); the element assigned
as Method is responsible for the appropriate choice of the op-
tion. The first module is a convolutional network (CNN), based
on an architecture from [36], which is trained end-to-end with
the whole model. In the second one, YOLO [37] architecture is
4
Applied Soft Computing 110 (2021) 107602
Fig. 4. The Vision Module elements; Method block refers to a particular vision
processing method chosen in the configuration (this defines which of the tensors
are passed over); Fully connected layers are optional, trainable layers, which
process the output of preceding blocks; the module receives a vector Xv is , which
is the image tensor Ximg scaled to shape 416 × 416 × 3, and outputs a vector
Yv is .
applied. It is an object detection model architecture, which fea-
tures high processing speed and satisfactory accuracy. It performs
image analysis in a single-shot manner. As an output, it estimates
the coordinates of the target object’s bounding box on the image.
We used TinyYOLO — a small version of the architecture (Fig. 5).
In our vision module, YOLO delivers three different levels of
output information: basic features, raw features, or a bounding
box; it is trained in advance, separately from the controller’s
model, and reused as a feature extractor. Further data processing
is available using additional, trainable fully-connected layers. The
Vision Module returns a vector Yv is . Below, both methods are
described in detail.
CNN. As mentioned before, the CNN used in our solution has an
architecture based on [36]. It consists of the following layers, each
followed by a ReLU activation function:
• convolutional layer (32 filters 8 × 8 with stride 4 × 4),
• convolutional layer (64 filters 4 × 4 with stride 2 × 2),
• convolutional layer (64 filters 3 × 3 with stride 1 × 1),
• flattening layer,
• fully-connected layer (512 neurons).
The output of the CNN is a tensor of shape 48 × 48 × 64. It is then
flattened and can be processed using fully connected layers. In a
configuration which uses the CNN, the whole controller model is
trained end-to-end.
YOLO. Its architecture is presented in Fig. 5. The model consists
of several convolutional layers; each of the first six layers is fol-
lowed by a max-pooling layer. In each convolutional layer Batch
Normalization is applied. The network uses the ReLU activation
function. The final output of the network is the bounding box of
the detected object. To provide a deeper analysis of the model’s
predictions, we added the ability to process its hidden feature
maps (in Fig. 5: basic features and raw features, referring to the
level of features’ complexity). Due to their shape, these feature
maps need to be flattened and can be further analyzed using
trainable layers of the controller’s model. In theory, this can yield
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Fig. 5. Object detection model architecture: Conv refers to a convolutional layer; Max-pool is a max-pooling layer; batch-norm indicates batch normalization; ReLU
and Sigmoid are activation function types; input vector Xv is is the image tensor Ximg scaled to shape 416 × 416 × 3.

a better understanding of the image content, as compared to plain λcoord – position component relevance coefficient,
bounding box parameters. S – number of rows and columns in YOLO grid,
obj obj
The YOLO network is an object detection model, i.e. it esti- 1i – step function (1i = 1 if cell i contains the
mates the position and the size of each object on the image. It is obj
detected object; 1i = 0 if it does not),
based on a grid of size 13 × 13: the input image is divided into
(x∗i , y∗i ) – target bounding box position,
169 cells and for each of them the network predicts the location
(xi , yi ) – predicted bounding box coordinates,
of one object (its bounding box’s position and size). Because of
the grid shape, the input image size should be set to 416 × 416 • Lsize – bounding box size error (comparing predicted and true
(for each cell to be of shape 32 × 32 pixels). width and height of the bounding box) – Eq. (14):
Each of the convolutional layers perform feature extraction, fi-
nally returning an intermediate tensor Ibbox of shape 13 × 13 × 5, S 2 [( ]
i.e. a prediction of a bounding box for each cell in the grid, in a
∑ obj
√ ∗ √ )2 (√ ∗ √ )2
Lsize = λcoord 1i wi − wi + hi − hi
form of five-element vector [C , x, y, w, h], where C is the detec-
i=0
tion confidence (Eq. (10)), x and y are coordinates of the bounding
box center and w and h are its width and height (all scaled to (14)
range [0, 1]). Finally, the model’s output Ybbox is estimated by
(wi∗ , h∗i ) – real width and height of the bounding box,
choosing the cell with the highest confidence value, according to
Eq. (9). (wi , hi ) – predicted size of the detection,
[ ]
Ybbox = arg max(Ibbox ) = Ĉ , x̂, ŷ, ŵ, ĥ (9) • Lobj , Lnoobj – positive and negative detection error, based on
C comparing the confidence of the detection Ci with the
ground truth Ci∗ (Ci∗ = 0 for negative detection; Ci∗ = 1
C = p (obj) × IoU true
pred (10) for positive) – Eqs. (15) and (16):
S2
∑ obj
)2
Ci∗ − Ci
(
|pred ∩ true| Lobj = 1i (15)
IoU true
pred = (11) i=0
|pred ∪ true|
where:
S 2

p (obj) – probability, that the bounding box contains the

∑ noobj
)2
Lnoobj = λnoobj Ci∗ − Ci
(
1i (16)
object,
i=0
IoU true
pred – Intersection over Union between ground truth and
estimated bounding box. λnoobj – negative detection loss relevance factor,
noobj noobj
Our YOLO model applies a modified loss function. It was 1i – step function (1i = 1 if cell i does not
noobj
designed to detect a single type of objects, therefore classification contain the detected object; 1i = 0 if it does).
error component has been removed. The assumed loss function is
Considering the task, which should be fulfilled by the con-
described in Eq. (12):
troller, the YOLO model has to recognize a single object in the
L = Lpos + Lsize + Lobj + Lnoobj (12) field of view (the gate), therefore we always use only the detec-
where: tion, which has the highest confidence C . It is worth noticing that
in case of no object in the field of view (which will happen often
• Lpos – detected object position error (based on the difference during the AUV operation, especially in the initial part of the run),
between the target and predicted bounding box position) –
the YOLO model will return a bounding box prediction with a low
Eq. (13):
confidence value (close to 0). We assume that in such a case, the
S2
∑ [( )2 )2 ] controller model will be able to learn to take appropriate actions
obj
Lpos = λcoord x∗i − xi + y∗i − yi
(
1i (13) to discover the target object location (for example, rotate to look
i=0 for the target object in other parts of the environment).
5
P. Zieliński and U. Markowska-Kaczmar
3.2. Data processing module
The data collected from the robot’s sensors (AHRS and depth
sensor) are concatenated with the Vision Module’s output, cre-
ating the input for the Data Processing Module. To predict the
correct values of velocity settings, the model needs to learn the
relation between the sensors’ readings, the Vision Module output,
and the correct settings of the AUV’s velocity. Therefore, we add
a sequence of trainable, fully-connected layers, which process the
concatenated input of the Module, trained jointly with the entire
DRL model. If we increase the number of layers and neurons in
this sequence, the model should be able to learn more complex
relations between the input parameters, but on the other hand,
it can make the training process more difficult.
3.3. Time-series analysis module
Due to a strong and long-term time dependency occurring in
the considered problem, actions taken by the controller should
be based not only on the current but also previous observations
of the environment. We decided to use a recurrent network —
an LSTM layer, which is trained together with the controller’s
model. It performs analysis of the output of the previous module,
conditioning output on current observations and hidden state (of
length 256), passed to consecutive LSTM cells in subsequent time
steps. The size of the hidden state in the Time-Series Analysis
Module was based on our preliminary studies, which are not
described here.
The output of the module is the settings vector Y, used to
control the robot’s movement.
3.4. Reward function
One of the key aspects of reinforcement learning is the reward
function used for training models [35]. It should accurately eval-
uate agent actions, promoting those leading to the success and
discouraging the model from making incorrect decisions. Ideally,
the optimal reward function should enable the model to learn the
correct behavior in the environment. In our research this would
mean reaching the target point using an optimal path, without
unnecessary deviations, preserving natural orientation and ap-
propriate speed. Therefore, we propose four rewards and examine
how they influence the model’s training and performance:
• position reward Rp , based on the distance of the agent from
the target point (e.g. the center of the gate), calculated
individually for each dimension according to Eq. (17) (see
Fig. 6a); this component should help the model learn to
minimize the distance to the target point (position reward
increases as the agent approaches the target): as the agent
approaches the target position, the function will rise to 1,
reaching kmin,d at a distance equal to lmin,d - this allows
explicit shaping of this reward function component in each
direction;
( )
⏐ ⏐ ln kmin,d
Rp,d = exp ⏐dtarget − dagent ⏐ ∗ (17)
lmin,d
where:
d – the axis along which the distance is measured (X ,
Y is formally described by Eq. (22). The fact of successful com-
⏐ or Z ),
⏐
⏐dtarget − dagent ⏐ – absolute distance between agent and
target position,
kmin,d ∈ (0, 1) – function value achieved at distance
lmin,d ; during research, kmin,d and lmin,d were set empir-
ically;
6
Applied Soft Computing 110 (2021) 107602
• rotation reward Rr , assessing agent’s angular position around
yaw axis in relation to the normal of target object front
plane (illustrated in Fig. 6b and formally described by Eq.
(18)); rotation component’s aim is to align the agent with
the target object, so that it can approach it by going forward;
Rr = cos γtarget − γagent
( )
(18)
where:
γ – normal angular position around yaw axis;
• velocity reward Rv , evaluating agent’s movement direction
and speed, described by Eq. (19); it should make the model
learn to move towards the gate as fast as possible;
⏐ ⏐
⏐vagent ⏐
Rv = cos ∢ vagent , Ttarget
[ ( )]
∗ (19)
vmax
where:
∢ vagent , Ttarget – angle between agent velocity vector
( )
and a vector connecting agent’s center of mass and
target ⏐point (see Fig. 6c),
⏐
⏐vagent ⏐ – length of the velocity vector, normalized to
maximal agent’s speed vmax ;
• angular velocity reward Rav , determining the correctness of
current agent’s velocity in relation to its rotation, given in
Eq. (20); it rewards angular movement towards the target
angular position around yaw axis (i.e. parallel to target
object’s normal), discouraging the model from turning away
from the target: the difference of tanh allows comparing
the current angle and angular velocity so that the rotation
towards the target angle returns values close to 0, and dif-
ferent in other cases; we then apply cos on the difference to
reward zeroing (i.e. adjusting the angular velocity to the tar-
get angle); the normalization of the difference with tanh(1)
is added to prevent the reward function from returning −1
for all misaligned angles and angular velocities;
tanh (∆γ ) − tanh ωY
[ ( )]
Rav = cos π ∗ (20)
tanh 1
where:
∆γ = γtarget − γagent – difference between agent and
target normals’ angular positions around the yaw axis
(same as in Eq. (18)), normalized to the range [−1, 1],
ωY – normalized yaw angular velocity.
To prevent such behavior when the agent concentrates on
collecting small rewards for a long time, we introduced two re-
strictions. The first one limits the length of training episodes and
resets the environment after exceeding the maximum number of
steps Lepisode . The second one relies on discounting positive re-
wards using function Dexp , described in Eq. (21), which decreases
exponentially from the first step, reaching value 0.1 at the half of
the episode length.
( )
ln 0.1
Dexp = exp step ∗ (21)
1
L
2 episode
The agent receives several distinct rewards and penalties.
When reaching the target, it is awarded a high reward, which
pletion of the task is determined using a box surrounding the
target object with some margin; agent hitting the box indicates
successful finish and, therefore, causes resetting the environment
and starting a new training episode.
Rsuccess = 50 ∗ Dlin ∗ Rr ∗ wr + Rp,av g ∗ wp
( )
(22)
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Fig. 6. Parameters used to calculate each reward component; (a) position reward (evaluates the distance from the target position along each axis: ∆x, ∆y and ∆z)
(Eq. (17)); (b) rotation reward (calculated using the difference between the yaw angular orientations of the agent’s normal (Nagent ) and the target’s normal (Ntarget ):
γagent and γtarget ) (Eq. (18)); (c) velocity reward (assesses agent’s linear velocity vagent (its direction and magnitude), comparing it to a vector connecting the agent
and the target Ttarget ) (Eq. (19)); (d) angular velocity reward (evaluates the agent’s angular velocity around the yaw axis (ωY ) to encourage aligning with the target’s
normal) (Eq. (20)).

its hyperparameters on the process of training, the efficiency of

1( ) operating and the achieved results. In this section, we describe
Rp,av g = Rp,x + Rp,y + Rp,z
3
kmin,lin − 1
Dlin = ∗ step + 1
lmin,lin
where:
wr , wp – weights for rotation and position components
(inferred experimentally in preliminary studies using grid
search method),
Rp,av g – average position reward for each axis,
Dlin – linear discount factor, taking the value kmin,lin at step =
lmin,lin (promotes reaching the target in a low number of
steps).
The agent is punished when hitting an obstacle (Rcol_penalty =
−1) or when making a fatal mistake, such as untimely emergence
or missing the target (Rfatal = −10). A fatal mistake causes im-
mediate resetting of the environment and starting a new training
episode. We did not consider collisions to be a fatal mistake, since
during our research we noticed multiple runs, in which the agent
would collide with the target object. Treating such an event as a
fatal error increased the instability of training and did not allow
the model to learn the correct behavior.
4. Results and discussion
The controller model has been tested to verify its capability of
solving the aforementioned problem and analyze the influence of
(23) the platform used for the research, experimental procedure and
experiments conducted to evaluate the proposed solution. The
code for training and evaluating the object detection and con-
(24) troller models is available on demand from the corresponding
author.
4.1. Simulation environment
To review the applicability of the proposed solution it was
necessary to prepare a simulation environment, resembling real
conditions on the test facility accurately. The application needs
to provide tools for controlling agents inserted into the environ-
ment, as well as simulating the physics of their movement and
the readouts of sensors and cameras.
It is necessary to mention some details on the competition
course, which should be simulated in the environment. The fa-
cility is divided into four quarters, each containing the same set
of objects. The region in which a team performs their run is
drawn randomly, and the robot’s starting orientation is drawn
with a coin flip (see Fig. 7). To address the possibility of varying
conditions on the facility, the simulation has to provide necessary
randomization of environment parameters (e.g. water hue and
opacity, sun position and brightness etc.). It would ensure our
model preserves robustness to applying in different conditions.
Since we assume that the robot is equipped with an internal
controller, which controls the speed of the robot’s thrusters ac-
cording to the velocity setpoint provided by the DRL model, we
used a simplified dynamic model of the AUV. In the simulation
7
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Algorithm 1 Dataset generation

Input: env ironment, number of collected images, target object, agent
object
Output: dataset of annotated images
1: for k = 1, 2 to n do
2: Reset(env ironment);
3: xr , yr , zr ← PutAgent(agent , target);
4: AddNoiseObjects();
5: αr , βr , γr ← AdjustRotation(agent, target);
6: bbox ← CalculateBoundingBox(agent, target);
7: image ← CollectImage();
8: dataset ← Add(image, bbox);
9: end for
10: return dataset
Fig. 7. Test facility layout; each region is marked with letters A-D; robot’s
starting orientations are indicated with green and blue arrows.
Table 1
Object detection model hyperparameters.
Hyperparameter Value
Mini-batch size 30
Learning rate 1 × 10−4
Model saving frequency 10 000 steps
Position component relevance coefficient (λcoord ) 5
Negative detection loss relevance factor (λnoobj ) 0.5
Number of rows and columns in YOLO grid (S) 13

Fig. 8. Example images of the conditions in the environment: the real image
(on the left) and an example from the simulation (on the right).

we consider its inertia and drag; their parameters were chosen

empirically to roughly resemble the real robot’s behavior.
During the research we have used Unity software, basing
the environment on RoboSub facility model made available by
Coleman University team. One of its crucial features is Unity ML-
Agents toolkit [38], providing tools for inserting agents, that can
be controlled using custom DRL models. The environment might
be used as a versatile base for other AUV-related projects. It is one
of the significant contributions of this research; we provide access
to the simulation environment on GitHub.1 Example images of Fig. 9. Plots of loss function and IoU for object detection model training.
the conditions in the real and simulated environment are shown
in Fig. 8.
(x, y, w, h) (center coordinates x and y, width w and height h),
4.2. Object detection model
enclosing the gate on the image.
The image generation method is presented in Algorithm 1.
Our detection model, designed with TensorFlow [39], was
Using this approach, we collected a dataset of 2000 positive and
trained to detect a gate, which is used as the target object
2000 negative training examples. Half of the images have been
in the first competition task. Our research does not focus on
noised with other objects. Similarly, we generated a separate set
YOLO hyperparameters’ analysis; we use default configuration
of 400 images as our test dataset.
and present observations of the training process. To evaluate the
model we used Intersection over Union (Eq. (11)) between the Training and evaluation. To train the object detection model we
object’s ground truth bounding box and the model’s prediction, used an online image augmentation method, applying various
and loss function value [37] (Eq. (12)). randomized operations on an image (horizontal flipping, random
crop, gaussian blur and noise, contrast normalization and random
Dataset to train YOLO. We used the simulation environment to
affine transformation). For drawing mini-batches (each of size
generate a set of images presenting the gate. The image gener-
30), we used a subset of 1000 images (a buffer, drawn randomly
ation function resets the environment and then, after drawing
every time the model is saved). The use of a buffer was required
the facility’s quarter and objects’ positions, it adjusts the agent’s
to enable online image augmentation in the limited memory of
orientation, so that the gate is visible on the camera image,
the test platform. Then, for each of the following 10 000 training
adding small, random position deviations. To make the dataset
steps, a 30-image mini-batch was drawn from the buffer for
noisy, the function can fill the object’s surroundings with various
training the model. After this amount of steps, the buffer was re-
objects used in other competition tasks. Furthermore, the condi-
drawn and the cycle was repeated. The model has been trained
tions (such as water hue and opacity, sun position and brightness,
using default YOLO hyperparameters (Table 1).
etc.) were modified on-line during the dataset generation.
Our training strategy was based on learning steps, followed by
Our simulation enables automatic annotation calculation. By
evaluation, model weights storing and redrawing of the training
comparing relative position of the agent and the target ob-
subset. Fig. 9 presents the learning curve. According to the re-
ject, it generates the coordinates of a bounding box as a vector
sults, the model achieves the best level of performance between
6th and 7th model save (that is where the IoU metric reaches
1 https://github.com/piotlinski/TransdecEnvironment. its maximum value on both train and test dataset). Empirical
8
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Table 2
Test platform configuration.
CPU AMD Ryzen 7 2700
RAM 32 GB
GPU Nvidia RTX 2070
OS Ubuntu Linux 18.04

Table 3
Default parameters used in experiments.
Hyperpa- Value Description
rameter
Ttraining 24 [h] Model training period (hours)
Fig. 10. Histogram of IoU for the test dataset using the selected object detection
nenv s 12 Number of agents
model. It is visible, that the selected model returns accurate predictions for the
ntrain_steps 50 000 Number of steps between each
training dataset, with only a few examples having a low (<0.3) IoU metric value.
evaluation and model saving
nev al_steps 200 Number of evaluation steps
idec 10 Interval between each model’s
comparison of saved models’ performance confirms, that the 70k- decision
Lepisode 4000 Maximum number of steps per
step model yields best results on a test video generated with our episode
simulation, which is why this model has been used for further |Ct | 256 LSTM cell state vector length
research. Fig. 10 shows a histogram of intersection over union val- Lminibatch 30 Number of steps in one minibatch
ues for the test dataset using the selected object detection model. nminibatches 4 Number of minibatches used for
model training
Additionally, we present some examples of model predictions lr 0.00025 Constant learning rate value
compared to the ground truth in Fig. 11. εPPO 0.2 PPO clipping threshold

kmin,x 0.3
4.3. Navigation controller
kmin,y 0.3
Expected position reward value
kmin,z 0.3
kmin achieved at distance lmin
To study the proposed solution we used an implementation lmin,x 0.8
for each dimension (Eq. (17))
of the deep reinforcement learning model from Stable Baselines lmin,y 0.5
lmin,z 2.0
library [40], which is a fork of OpenAI’s Baselines [41]. Training
and evaluation were conducted using the simulation prepared for wr 0.2 Rotation component weight in
the project. We measure the performance using 4 metrics: success reward (Eq. (22))
wp 0.1 Position component weight in
• mean reward gathered by the agent R̄, averaged for each success reward (Eq. (22))
agent and environment instance, as in Eq. (25): kmin,lin 0.5 Linear discount factor
lmin,lin Lepisode parameters (Eq. (24))
Ne T
1 ∑∑
R̄ = ri (t) (25)
Ne
i t =0
and missing the target qM ), together with the fraction of all
where: of runs, during which a collision occurred (qC ):
Ne – total number of agents, |S | |E | |M | |C |
T – learning episode length, qS = ; qE = ; qM = ; qC = (27)
|T | |T | |T | |T |
ri (t) – value of a reward received by the agent in ith
environment, in tth time step; where:

• average episode length L̄, i.e. number of steps taken by the
agent before reaching the target or resetting the environ-
ment; good results are indicated by large reward achieved
in small number of steps; small reward received in short
episodes can be caused by incorrect agent behavior, such as
untimely emergence or missing the target, both of which are
considered fatal mistakes in the competition; on the other
hand, a large number of steps taken in one episode may sug-
gest, that the agent is wandering around the environment;
• speed of predictions it¯ s , calculated as an average number of
steps per second, as shown in Eq. (26):
Nsteps
it¯ s =
Tev al
where:
Nsteps – number of steps taken by the agent during
evaluation,
Tev al – total evaluation time in seconds;
• episode statistics (Eq. (27)), i.e. total fraction of successful
approaches (ended by reaching the target) qS and failed
episodes (distinguishing between untimely emergence qE
|S | – number of episodes which ended successfully,
|E | – number of episodes failed by emerging,
|M | – number of episodes where the agent missed the
target,
|T | – total number of episodes conducted in the eval-
uation; |T | = |S | + |E | + |M |,
|C | – number of episodes during which a collision
occurred.
We applied a unified training procedure to compare various
model designs directly. Using actor–critic architecture and A2C
method, we conducted a parallel training of 12 agents in a con-
stant period of 24 h on our test platform described by Table 2. Our
(26) training strategy assumes a cyclic model evaluation every 50 000
training steps. Testing was conducted for each agent by perform-
ing 200 steps according to model prediction in each of the 12
environments and registering average reward and episode length.
The evaluation sequence, used for assessing the best model from
each training procedure (i.e. the one achieving the highest av-
erage reward per test episode during training) was repeated 5
times, with forced reset of each simulation instance. All metrics
were averaged over each evaluation episode (or summed, in case
of episode statistics).
9
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Fig. 11. Example images from the dataset used to train the YOLO model, with drawn ground-truth boxes (red) and predictions (green). The two images on the left
contain added ‘noise’ objects, whereas the images on the right were not noised.

Table 4 Table 5
Hyperparameters examined in the research. The controller model configuration used for reward function evaluation.
Hyperparame- Values Description Hyperparameter Values
ter Visual feature embeddings bounding box
Visual feature {bounding box, It defines the visual features Vision FC layers None
embeddings raw features, embedding method in the Vision Data Processing FC layers [64, 64, 64, 64]
basic features, Module (Fig. 4)
built-in conv}

Vision FC [l1 , l2 , l3 , . . .] They assign the number of

layers neurons in fully connected layers
(32)
in the Vision Module; None The complex form of reward function is a weighted sum of
refers to no additional hidden
layers in the Vision Module
elementary reward functions. The weights have been chosen em-
Data [l1 , l2 , l3 , . . .] They define the number of pirically, based on the agent’s behavior preliminary tests, which
Processing FC neurons in subsequent FC layers are not described here. Their values are as follows:
layers in the Data Processing Module
(Fig. 4) • wv = 0.8 (velocity reward weight),
• wav = 0.2 (angular velocity reward weight),
• wp = 0.1 (position reward weight),
• wr = 0.2 (rotation reward weight),
The experiments are focused on studying the influence of
• b = 0.2 (velocity reward bias).
visual feature embeddings choice and reward function form on
the model’s performance, speed and the process of training. Each The reward function form from Eq. (32) was proposed based
experiment utilizes a model trained with default parameters from on the following insights. Conditioning the reward on the velocity
Table 3, whereas model hyperparameters examined in our re- component Rv should guarantee that the agent moves towards
search and the possible settings are shown in Table 4. the target, whereas additional bias b should enable agent’s move-
During research, we examined the configuration of the Data ment evaluation even in case of zero velocity (which occurs often
Processing Module’s hidden fully-connected layers to choose the in the starting phase of the training; lack of bias led to random
optimal settings. The process of tuning these hyperparameters is wandering and collecting small rewards). Since the lengthwise
described in Appendix. position component Rp,Z is intuitively the most important one,
it was used to condition rewards calculated along the other two
4.3.1. Experiment 1: The influence of reward function components axes (Eq. (31)). Additionally, we scale the function to the range
on model training [−1, 1] to stabilize the training process.
The main aim of this experiment is to examine the importance The experiment consists of two phases. First, to examine the
of the four aforementioned reward function components. We process of training, we trained a model using each form of the
researched four elementary forms of reward function: reward function. We assumed a constant configuration of the
model (see Table 5). Then, the best models for each reward
• analyzing only agent’s position Rpos (Eq. (28)), function form were evaluated using an averaged reward function,
given in Eq. (33).
1 ( )
Rpos = ∗ Dexp ∗ Rp,x + Rp,y + Rp,z + Rr (28) 1(
4
)
Rev = Rp,X + Rp,Y + Rp,Z + Rr + Rv + Rav (33)
6
• assessing only agent’s velocity Rvel (Eq. (29)),
The results of the training are shown in Fig. 12. We use the
1 following denominations: position-only refers to reward func-
Rv el = ∗ Dexp ∗ (Rv + Rav ) (29)
2 tion from Eq. (28), velocity-only is the form defined in Eq. (29),
noangvel refers to complex form without angular velocity com-
• complex form without angular velocity Rno_ang vel (Eq. (30)), ponent from Eq. (30) and complex is the full form from Eq. (32).
Dexp ∗ (Rv + b) wv + wp ∗ Rp,agg + wr ∗ Rr Here, the rewards were calculated according to the form used for
Rno_ang v el = ∗ training the model (e.g. position-only form for the position-only
1+b wv + wp + wr model).
(30) Using the model with the highest value of average reward
achieved in test sequences, we conducted the evaluation proce-
dure. All the relevant metrics are shown in Fig. 13: (a) shows
(
Rp,agg = Rp,Z ∗ Rp,X + Rp,Y
)
(31) average reward (Eq. (33)) achieved by the model, (b) presents av-
erage episode length during evaluation and (c) presents episode
• complex form, using all components Rcomplex (Eq. (32)). statistics, i.e. stacked fractions of episodes, which ended with
emerging, missing the target or success, with a separate column
Dexp ∗ (Rv + b) wv + wav ∗ Rav + wp ∗ Rp,agg + wr ∗ Rr showing a total fraction of runs with collision. Since we assumed
Rcomplex = ∗
1+b wv + wav + wp + wr a constant model architecture, model’s speed was not analyzed.
10
P. Zieliński and U. Markowska-Kaczmar
Fig. 12. Training results for each of the reward function forms — average reward achieved in testing (performed every ntrain_steps of training, according to the training
procedure); smoothing was not applied.
Fig. 13. Performance evaluation results for each of the reward function forms:
(a) mean reward achieved during evaluation; (b) average episode length; (c)
episode statistics: here, the main column presents stacked fractions of episodes
when agent emerged, missed the target, or achieved success (all of them summing
up to 100%), while a separate column shows a total fraction of runs when a
collision occurred.
Analysis. The difference between models trained using complex
(Eqs. (30) and (32)) and simple (Eqs. (28) and (29)) forms of
11
Applied Soft Computing 110 (2021) 107602
reward function is easily visible; position- and velocity-based
reward functions do not lead to successful training of the model,
achieving low values of average reward both during training
and evaluation phase. On the other hand, the complex reward
function forms allow the model to reach high rewards, which
indicates learning how to act in the environment.
This conclusion is confirmed by the values of the other metrics
and observations of the agent’s movement. Models trained using
simple forms of reward function do not reach the target. Instead,
they miss it or emerge above the surface. The model trained
using the complex reward function without the angular velocity
component performs significantly better, but it still misses the
target in most runs. Observations show, that, due to the lack
of the angular velocity reward compound (Eq. (20)), this agent
has problems with aligning to the target normal. The complex
form, which includes all of the proposed reward components,
allows training a model, which achieves the highest fraction of
successful runs and collects the biggest reward during evaluation,
with almost no untimely emergence. However, here one should
notice that almost 10% of runs included colliding with another
object (most probably the gate).
Analysis of average episode lengths is consistent with the
other metrics. Best models achieve high reward value in relatively
short episodes, which means they manage to operate fairly well
in the environment. Position- and velocity-based reward forms
yield models reaching low reward value in short episodes, which
indicates their poor performance.
4.3.2. Experiment 2: The influence of visual features embedding
choice on the model performance
During our research we compared the process of training
and the performance of the controller, depending on the used
image processing method in the Vision Module. Here, we used
the complex reward function form (Eq. (32)), with default model
parameters from Table 3 and researched configurations listed in
Table 6.
We denominate them as follows. First term refers to the visual
feature embedding: default is the best bounding box-based model
(as concluded in Appendix), conv refers to applying the built-in
CNN, while raw and basic signify models utilizing feature maps
extracted from the YOLO network.
For each method based on complex feature maps (conv, raw
and basic) we analyzed first a model without Vision FC layers;
in this case we used more neurons in Data Processing FC layers:
[256, 64, 64, 64] for CNN-based models and [1024, 256, 64, 64]
for those processing YOLO feature maps, where the bigger size is
required by a significantly longer value vector. Such plain models
P. Zieliński and U. Markowska-Kaczmar
Table 6
All configurations of the model investigated in the experiment; each row shows a set of hyperparameters of a separate examined model, according to Table 4;
column name is the denomination used further in the experiment (for example, model named conv-bb-fc uses built-in CNN output and YOLO network bounding
box prediction as visual feature embeddings, it has one FC layer with 256 neurons in the Vision Module and four FC layers with 64 neurons in the Data Processing
Module)
Name Visual feature embeddings
default bounding box
conv built-in conv
conv-fc built-in conv
conv-bb (built-in conv + bounding box)
conv-bb-fc (built-in conv + bounding box)
raw raw features
raw-fc raw features
raw-bb (raw features + bounding box)
raw-bb-fc (raw features + bounding box)
basic basic features
basic-fc basic features
basic-bb (basic features + bounding box)
basic-bb-fc (basic features + bounding box)
are denominated with no additional suffix (i.e. conv, raw and
basic).
Apart from that, we used models with additional Vision FC
layers and default number of neurons in Data Processing FC
layers ([64, 64, 64, 64]); then, for built-in convolutional layers we
applied an additional hidden layer with 256 neurons, and for
YOLO-based feature processing, two layers of [1024, 256] neurons
were used. These models are marked with a suffix -fc.
Moreover, for each of the complex feature maps-based models,
we examined a version using a combination of the feature maps’
data and the bounding box prediction from the YOLO model. We
denominate these models with a suffix -bb (abbr. bounding box).
The training process’ results for each visual feature embedding
method (conv, raw and basic) are shown in Fig. 14. After that, we
evaluated the models achieving the highest testing performance
and gathered the metrics in Fig. 15.
Analysis. When looking at models without Vision FC layers it is
obvious that the only successful alternative to the default bound-
ing box-based configuration uses the basic features from the
YOLO network. The ‘basic’ model achieved over 40% of successful
runs. Raw features- and built-in convolution-based models are
not able to reach correct behavior in the environment and os-
cillate around 0.0 reward throughout the entire process of train-
ing. Including additional, built-in convolutional layers greatly in-
creases the complexity of the model, which makes its training
significantly more difficult. On the other hand, the ‘raw’ features
extracted from the YOLO network are probably too specific for the
detection task, which is why the model is unable to use them for
navigation. This shows that data stored in the penultimate layer
cannot be used in the proposed solution. It is worth noticing, that
concatenating bounding box prediction with no Vision FC layers
did not improve the model’s performance at all. This is probably
caused by the huge size of features tensor, leading the model to
ignore the appended 5-element bounding box vector.
Although adding Vision FC layers did not improve the per-
formance of raw features- and built-in convolution-based mod-
els significantly, additional dimensionality reduction allowed the
models with bounding box vector concatenation to achieve much
better results. Especially, the ‘conv-bb-fc’ model reached 25% of
successful runs, with average reward close to 10.00. However,
additional Vision FC layers cause a small worsening of basic
features-based models’ performance.
What draws attention is the visible difference in prediction
speed between vision processing methods. Compared to the end-
to-end approach in models with built-in convolutional layers,
the utilization of the external YOLO-based network suffers from
a noticeable decline in prediction speed: all models based on
12
Applied Soft Computing 110 (2021) 107602
Vision FC layers Data Processing FC layers
None [64, 64, 64, 64]
None [256, 64, 64, 64]
[256] [64, 64, 64, 64]
None [256, 64, 64, 64]
[256] [64, 64, 64, 64]
None [1024, 256, 64, 64]
[1024, 256] [64, 64, 64, 64]
None [1024, 256, 64, 64]
[1024, 256] [64, 64, 64, 64]
None [1024, 256, 64, 64]
[1024, 256] [64, 64, 64, 64]
None [1024, 256, 64, 64]
[1024, 256] [64, 64, 64, 64]
feature maps extracted from the YOLO network are over 35%
slower than the default approach and the basic ‘conv’ model.
Apart from the bigger size of the Data Processing Module FC
layers or the addition of Vision Module FC layers, the delay is
likely to be caused by the extraction of a high-dimensional tensor
of feature maps from the YOLO network in each step.
During our experiments, we observed a small decrease in the
number of iterations per second when using bounding box pre-
dictions from the YOLO network. The difference ranges between
5 and 10%. However, it is worth mentioning, that the effect of
adding Vision Module FC layers on the prediction speed is rather
small, with the slowdown below 5%.
To discuss the behavior of the controller model, which
achieved the best results, we present three example attempts
to the discussed navigation task. We plotted the robot’s position
during each run together with linear velocity settings output by
the controller (Eq. (8)), averaged throughout the fragment be-
tween each plotted vector. We distinguished three general types
of acquired paths: successful runs with satisfactory (Fig. 16a) or
distorted path (Fig. 16b) and failed attempts (Fig. 16c).
Subjectively satisfactory paths, which constitute roughly half
of all successful attempts, could be observed when the robot was
able to notice the target object during the initial part of the run.
This allowed it to keep the target in the field of vision throughout
the entire run and navigate the robot on a smooth path with
steady velocity settings.
The inability to observe the target object at the beginning
often resulted in a chaotic movement of the robot in the initial
phase, which can be interpreted as searching for the target. After
finding the target object, the controller usually started to navigate
the robot towards it, however, the resulting path was not as
smooth as in the previous group of runs.
Among unsuccessful attempts, we can identify two main
causes. First, the model could continue to search for the target
object for too long without finding it, ultimately exceeding the
fixed episode length without reaching the target. The other cause
results from the lack of knowledge about the orientation of the
target object (the detection model only provides a bounding box
of the detection); the controller could attempt to navigate to the
target object from a wrong position (i.e. from the side) and hence
pass by the target object.
5. Conclusions
In this paper we presented the research on the proposed
autonomous underwater vehicle navigation controller, featuring
deep reinforcement learning. We analyze several methods of
visual feature embeddings extraction and show their influence
P. Zieliński and U. Markowska-Kaczmar
Fig. 14. Training results for each of the visual features embedding method — average reward achieved in testing (performed every ntrain_steps of training, according
to the training procedure); smoothing was not applied; (a) methods based on features extracted using built-in CNN; (b) methods based on raw features extracted
from the YOLO network; (c) methods based on basic features extracted from the YOLO network.
on the model’s training and functioning. The final model can
predict the robot’s steering settings based on the data from sen-
sors, successfully navigating between two predefined points in a
3D environment. This shows, that deep reinforcement learning
with vision-based approach, which was previously used in 2D
navigation problems [29,30], might be applied in an autonomous
underwater vehicle controller.
Among all of the achievements of the research, we would
like to emphasize the analysis of the visual features extraction
approaches applied in our model. As opposed to previous ap-
proaches [28–30], we show how various levels of feature com-
plexity influence the feasibility of training the DRL model suc-
cessfully; the utilization of the bounding box prediction from
the YOLO network, trained in a supervised manner on data from
the simulation, allows the model to achieve significantly better
results than an end-to-end solution, which uses solely built-
in, trainable layers. Furthermore, we compare various levels of
feature maps extracted from the YOLO network and prove the
advantage of ‘basic’ features over the ‘raw’ maps extracted from
the penultimate layer of the network.
13
Applied Soft Computing 110 (2021) 107602
Our paper shows a comparison of various model architectures
(especially in the context of the configuration of hidden, fully-
connected layers) and their influence on the performance and the
process of training. On that basis, we chose a combination that
provides the best results, with respect to the assumed metrics.
Furthermore, we analyzed several reward function forms and
four aspects assessed during the agent’s operation, leading us to
the final form, which allows the model to learn fairly correct
behavior.
The research can be extended in many directions. First of all,
the reward function could be enhanced. Currently, the robot does
not always move stably and sometimes rotates throughout navi-
gation. Further analysis of the reward function form might as well
help to reduce the number of collisions, which occur even with
the most successful solutions. Additionally, the final condition
could be specified better, to guarantee a correct orientation of
the robot in the target position and enable it to stop at the target
point. It is important to mention the instability of training DRL
models, which is easily visible on learning curves in Figs. 12 and
14. This could be further analyzed by repeating the runs multi-
ple times and analyzing the statistics for each configuration. An
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Fig. 16. Robot paths and velocities acquired by using a trained ‘default’ model;
‘velocity’ refers to controller settings aligned as in Fig. 2; (a) successful run with
satisfactory path; (b) successful run with distorted path; (c) failed run; position
axes are measured in meters.

this should improve the model’s performance, especially the abil-

Fig. 15. Performance evaluation results: various methods of visual feature
embeddings extraction; default refers to models without Vision FC layers, bb
indicates combining embeddings with bounding box prediction, whereas fc
signify including Vision FC layers; (a) mean reward achieved during evaluation;
(b) average episode length; (c) number of model iterations per second; (d)
episode statistics, as in Fig. 13.
ity to avoid collisions and find the target object quickly. An
interesting direction would be applying the solution to the prob-
lem of navigation of other types of robots, such as small un-
manned ground vehicles or unmanned aerial vehicles, which
would however require a different reward function form.

Abbreviations

interesting aspect to consider is transfer learning, i.e. fine-tuning The following abbreviations are used in this manuscript:
the YOLO network during the controller training, which could im-
A2C Advanced Actor–Critic
prove the performance of the best solutions. Additionally, positive
AUV Autonomous Underwater Vehicle
results of ‘basic’ features-based models suggest, that utilizing a
CNN Convolutional Neural Network
pre-trained feature extractor, such as VGG [42] or ResNet [43]
DL Deep Learning
in the vision module could lead to training a successful model.
DRL Deep Reinforcement Learning
Another direction of the research could be applying the solution
FC Fully-Connected
to other competition tasks; an important test for our solution
IoU Intersection over Union
would be to use the model in a real AUV’s navigation problem,
which would verify its ability to steer a true robot. LSTM Long Short-Term Memory
Our approach has big development potential. Especially, our PPO Proximal Policy Optimization
model could be extended to process data from additional sensors RL Reinforcement Learning
(such as other robot’s cameras, proximity sensors or sonars); YOLO You Only Look Once

14
P. Zieliński and U. Markowska-Kaczmar Applied Soft Computing 110 (2021) 107602

Table 7
Data processing module hyperparameters’ tuning — the model configuration.
Hyperparameter Values
Visual feature embeddings bounding box
Vision FC layers None
{[16, 16], [64, 64], [256, 256]
Data Processing FC layers
[64, 64], [64, 64, 64], [64, 64, 64, 64]}

Fig. 17. Training results: Data Processing Module with 2 hidden FC layers with
varying number of neurons; plot shows average reward achieved in testing
(performed every ntrain_steps of training, according to the training procedure);
smoothing was not applied.

CRediT authorship contribution statement

P. Zieliński: Conceptualization, Methodology, Software, Vali-

dation, Formal analysis, Investigation, Resources, Data curation,
Writing - original draft, Writing - review & editing, Visualiza-
tion. U. Markowska-Kaczmar: Conceptualization, Methodology,
Validation, Formal analysis, Resources, Writing - original draft,
Writing - review & editing, Supervision, Project administration,
Funding acquisition.

Declaration of competing interest

The authors declare that they have no known competing finan-

cial interests or personal relationships that could have appeared
to influence the work reported in this paper.

Acknowledgments

We thank members of KN Robocik for the design and con-

struction of the robot and help with designing the environment.
Acknowledgments are directed also to Nick Cantrell, a former
RoboSub competitor, who shared a model of the Transdec facility.
We thank the RoboNation – AUVSI Foundation for designing and
conducting the RoboSub competition.
Fig. 18. Performance evaluation results: Data Processing Module with two
hidden FC layers; (a) mean reward achieved during evaluation; (b) average
episode length; (c) number of model iterations per second; (d) episode statistics,
as in Fig. 13.

Appendix. Data processing module hyperparameters’ tuning

To choose the optimal Data Processing Module structure, espe-
cially the number of FC layers and the number of neurons in each
layer, we investigated how they influence the process of training
and the model’s performance. Here, we utilized the complex
reward function form from Eq. (32). Model’s hyperparameters
were set as in Table 7, with default ones as in Table 3.
The examination was conducted in two stages. First, we com-
pared models with 2 hidden layers in the Data Processing Module,
with varying number of neurons. We researched models with
2 × 16, 2 × 64 and 2 × 256 neurons. The results of the training
are shown in Fig. 17. It is visible, that each model is able to learn
how to act in the environment and achieve high values of mean
reward in the training phase.
15
The best of the models saved during the training were then
evaluated. The results of the evaluation are shown in Fig. 18.
One can notice, that all models achieve similar results in the
evaluation phase as well. However, the ‘2x64’ model reaches
the best performance: achieves highest value of average reward
both during training and evaluation, with the highest fraction of
successes and mean episode length.
The second stage relied on examining various numbers of
hidden layers in the Data Processing Module, each containing the
same number of neurons. The previous part shows that model
with two 64-neuron layers performs best, that is why we decided
to test configurations of 2, 3 and 4 layers with 64 neurons in each
P. Zieliński and U. Markowska-Kaczmar
Fig. 19. Training results: Data Processing Module with 2, 3 and 4 hidden FC
layers with 64 neurons each; plot shows average reward achieved in testing
(performed every ntrain_steps of training, according to the training procedure);
smoothing was not applied.
Fig. 20. Performance evaluation results Data Processing Module with 2, 3 and
4 hidden FC layers with 64 neurons each; (a) mean reward achieved during
evaluation; (b) average episode length; (c) number of model iterations per
second; (d) episode statistics, as in Fig. 13.
16
Applied Soft Computing 110 (2021) 107602
of them. The results of training were gathered in Fig. 19. As be-
fore, the difference between each model is not easily noticeable;
each one achieves high value of average reward during training.
After training, the models which achieved the highest average
reward value were evaluated. The evaluation metrics are shown
in Fig. 20. The results do not show significant differences between
the models again. They achieve comparable values of rewards and
fractions of successes, however the ‘4x64’ model shows slightly
better performance than the other ones.
Modifications of Data Processing FC layers’ configuration does
not have a radical effect on the model’s training process. Each
of the analyzed settings provides the feasibility for the model to
learn how to act in the environment. However, it is worth notic-
ing, that configuration using 4 layers with 64 neurons achieves
the best results, both during training and evaluation phase.
Similarly, changes in these hyperparameters do not alter the
model’s performance during the evaluation process significantly,
although the ‘4x64’ neuron model performs best again, achieving
over 50% of successful runs.
It is visible, that the number of hidden layers in the Data
Processing Module does not influence the speed of the model.
Possibly, the differences are much smaller than the delays oc-
curring in one of the other modules or during data transmission
between the model and the environment.
Due to the best performance of a model with 4 hidden FC
layers in the Data Processing Module, each with 64 neurons, we
based our experiments on this Data Processing FC layers settings.
References
[1] C.W. Warren, A technique for autonomous underwater vehicle route
planning, in: Proceedings of the Symposium on Autonomous Underwater
Vehicle Technology, Vol. 15, 1990, pp. 201–205, http://dx.doi.org/10.1109/
48.107148, (3).
[2] H. Choset, K.M. Lynch, S. Hutchinson, G. Kantor, W. Burgard, L.
Kavraki, S. Thrun, Principles of Robot Motion Theory, Algorithms, and
Implementations, MIT, 2005.
[3] H.I. Kang, B. Lee, K. Kim, Path planning algorithm using the particle swarm
optimization and the improved dijkstra algorithm, in: 2008 IEEE Pacific-
Asia Workshop on Computational Intelligence and Industrial Application,
Vol. 2, 2008, pp. 1002–1004, http://dx.doi.org/10.1109/PACIIA.2008.376.
[4] A.K. Guruji, H. Agarwal, D. Parsediya, Time-efficient A* algorithm for
robot path planning, Proc. Technol. 23 (2016) 144–149, http://dx.doi.org/
10.1016/j.protcy.2016.03.010, 3rd International Conference on Innovations
in Automation and Mechatronics Engineering 2016, ICIAME 2016 05-
06 February, 2016. URL http://www.sciencedirect.com/science/article/pii/
S2212017316300111.
[5] F.A. Cosío, M.P. Castañeda, Autonomous robot navigation using adaptive
potential fields, Math. Comput. Modelling 40 (9) (2004) 1141–1156, http://
dx.doi.org/10.1016/j.mcm.2004.05.001, URL http://www.sciencedirect.com/
science/article/pii/S0895717704003097.
[6] H. Li, M. Liu, K. Liu, Bio-inspired geomagnetic navigation method for
autonomous underwater vehicle, J. Syst. Eng. Electron. 28 (6) (2017)
1203–1209, http://dx.doi.org/10.21629/JSEE.2017.06.18.
[7] B.J. Dzienkowski, C. Strode, U. Markowska-Kaczmar, Employing game
theory and computational intelligence to find the optimal strategy of an
autonomous underwater vehicle against a submarine, in: FedCSIS, 2016,
pp. 31–40.
[8] I.K. Ibraheem, F.H. Ajeil, Multi-objective path planning of an autonomous
mobile robot in static and dynamic environments using a hybrid PSO-
MFB optimisation algorithm, 2018, CoRR . arXiv:1805.00224. URL http:
//arxiv.org/abs/1805.00224.
[9] T.P. Lillicrap, J.J. Hunt, A.e. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver,
D. Wierstra, Continuous control with deep reinforcement learning, 2015,
arXiv e-prints arXiv:1509.02971.
[10] K. Wu, M.A. Esfahani, S. Yuan, H. Wang, Depth-based obstacle avoidance
through deep reinforcement learning, in: Proceedings of the 5th Interna-
tional Conference on Mechatronics and Robotics Engineering, in: ICMRE’19,
ACM, New York, NY, USA, 2019, pp. 102–106, http://dx.doi.org/10.1145/
3314493.3314495, URL http://doi.acm.org/10.1145/3314493.3314495.
[11] I. Carlucho, M. De Paula, S. Wang, B.V. Menna, Y. Petillot, G.G. Acosta,
AUV position tracking control using end-to-end deep reinforcement learn-
ing, OCEANS (2019) http://dx.doi.org/10.1109/OCEANS.2018.8604791, 2018
MTS/IEEE Charleston [8604791].
P. Zieliński and U. Markowska-Kaczmar
[12] R. Yu, Z. Shi, C. Huang, T. Li, Q. Ma, Deep reinforcement learning based
optimal trajectory tracking control of autonomous underwater vehicle,
in: 2017 36th Chinese Control Conference (CCC), 2017, pp. 4958–4965,
http://dx.doi.org/10.23919/ChiCC.2017.8028138.
[13] Y. Huo, Y. Li, X. Feng, Model-free recurrent reinforcement learning for
AUV horizontal control, IOP Conf. Ser.: Mater. Sci. Eng. 428 (2018) 012063,
http://dx.doi.org/10.1088/1757-899x/428/1/012063.
[14] C. Wang, L. Wei, Z. Wang, M. Song, N. Mahmoudian, Reinforcement
learning-based multi-AUV adaptive trajectory planning for under-ice field
estimation, Sensors 18 (11:3859) (2019).
[15] S. You, M. Diao, L. Gao, F. Zhang, H. Wang, Target tracking strat-
egy using deep deterministic policy gradient, Appl. Soft Comput. 95
(2020) 106490, http://dx.doi.org/10.1016/j.asoc.2020.106490, URL http://
www.sciencedirect.com/science/article/pii/S1568494620304294.
[16] C. Yan, X. Xiang, C. Wang, Towards real-time path planning through deep
reinforcement learning for a UAV in dynamic environments, J. Intell. Robot.
Syst. (2019) http://dx.doi.org/10.1007/s10846-019-01073-3.
[17] Y. Sun, X. Ran, G. Zhang, H. Xu, X. Wang, AUV 3D path planning based on
the improved hierarchical deep Q network, J. Mar. Sci. Eng. 8 (2) (2020)
http://dx.doi.org/10.3390/jmse8020145, URL https://www.mdpi.com/2077-
1312/8/2/145.
[18] X. Zhou, Y. Gao, L. Guan, Towards goal-directed navigation through
combining learning based global and local planners, Sensors (Basel)
(2019) http://dx.doi.org/10.3390/s19010176, URL https://www.ncbi.nlm.
nih.gov/pmc/articles/PMC6339171/.
[19] Z. Wang, N. de Freitas, M. Lanctot, Dueling network architectures for deep
reinforcement learning, 2015, CoRR abs/1511.06581. arXiv:1511.06581.
URL http://arxiv.org/abs/1511.06581.
[20] T.D. Kulkarni, K.R. Narasimhan, A. Saeedi, J.B. Tenenbaum, Hierarchical
deep reinforcement learning: Integrating temporal abstraction and intrinsic
motivation, in: Proceedings of the 30th International Conference on Neural
Information Processing Systems, in: NIPS’16, Curran Associates Inc., Red
Hook, NY, USA, 2016, pp. 3682–3690.
[21] X. Cao, C. Sun, M. Yan, Target search control of AUV in underwater
environment with deep reinforcement learning, IEEE Access 7 (2019)
96549–96559, http://dx.doi.org/10.1109/ACCESS.2019.2929120.
[22] V. Mnih, A. Puigdomènech Badia, M. Mirza, A. Graves, T.P. Lillicrap,
T. Harley, D. Silver, K. Kavukcuoglu, Asynchronous methods for deep
reinforcement learning, 2016, arXiv e-prints arXiv:1602.01783.
[23] M. Sporyshev, A. Scherbatyuk, Reinforcement learning approach for co-
operative AUVs in underwater surveillance operations, in: 2019 IEEE
Underwater Technology (UT), 2019, pp. 1–4, http://dx.doi.org/10.1109/UT.
2019.8734293.
[24] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy
optimization algorithms, 2017, arXiv e-prints arXiv:1707.06347.
[25] Y. Liu, F. Wang, Z. Lv, K. Cao, Y. Lin, Pixel-to-action policy for underwater
pipeline following via deep reinforcement learning, in: 2018 IEEE Inter-
national Conference of Intelligent Robotic and Control Engineering (IRCE),
2018, pp. 135–139, http://dx.doi.org/10.1109/IRCE.2018.8492943.
[26] Y. Sun, J. Cheng, G. Zhang, H. Xu, Mapless motion planning system
for an autonomous underwater vehicle using policy gradient-based deep
reinforcement learning, J. Intell. Robot. Syst. (2019) http://dx.doi.org/10.
1007/s10846-019-01004-2.
[27] S.T. Havenstrøm, A. Rasheed, O. San, Deep reinforcement learning con-
troller for 3D path following and collision avoidance by autonomous
underwater vehicles, Front. Robot. AI 7 (2021) 211, http://dx.doi.org/10.
3389/frobt.2020.566037, URL https://www.frontiersin.org/article/10.3389/
frobt.2020.566037.
[28] F. Zhang, J. Leitner, M. Milford, B. Upcroft, P.I. Corke, Towards vision-
based deep reinforcement learning for robotic motion control, 2015, ArXiv
abs/1511.03791.
17
Applied Soft Computing 110 (2021) 107602
[29] L. Xie, S. Wang, A. Markham, N. Trigoni, Towards monocular vision based
obstacle avoidance through deep reinforcement learning, in: Robotics:
Science and Systems Workshop 2017: New Frontiers for Deep Learning
in Robotics ; Conference Date: 15-07-2017 through 15-07-2017, 2017.
[30] J. Kulhanek, E. Derner, T. de Bruin, R. Babuska, Vision-based navigation
using deep reinforcement learning, in: 2019 European Conference on
Mobile Robots (ECMR), IEEE, 2019, http://dx.doi.org/10.1109/ecmr.2019.
8870964.
[31] J. Roghair, K. Ko, A.E.N. Asli, A. Jannesari, A vision based deep reinforcement
learning algorithm for UAV obstacle avoidance, 2021, arXiv:2103.06403.
[32] J. Hua, L. Zeng, G. Li, Z. Ju, Learning for a robot: Deep reinforcement
learning, imitation learning, transfer learning, Sensors 21 (4) (2021) http://
dx.doi.org/10.3390/s21041278, URL https://www.mdpi.com/1424-8220/21/
4/1278.
[33] W. Zhao, J.P. Queralta, T. Westerlund, Sim-to-real transfer in deep rein-
forcement learning for robotics: a survey, in: 2020 IEEE Symposium Series
on Computational Intelligence (SSCI), 2020.
[34] J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, S. Levine, How to train
your robot with deep reinforcement learning: lessons we have learned,
Int. J. Robot. Res. (2021) http://dx.doi.org/10.1177/0278364920987859.
[35] R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, second
ed., The MIT Press, 2018, URL http://incompleteideas.net/book/the-book-
2nd.html.
[36] V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A.
Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A.
Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, D. Hassabis,
Human-level control through deep reinforcement learning, Nature 518
(2015) 529–533, http://dx.doi.org/10.1038/nature14236.
[37] J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You only look once: Unified,
real-time object detection, in: 2016 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), 2016, pp. 779–788, http://dx.doi.org/10.
1109/CVPR.2016.91.
[38] A. Juliani, V.-P. Berges, E. Vckay, Y. Gao, H. Henry, M. Mattar, D. Lange,
Unity: A general platform for intelligent agents, 2018, ArXiv abs/1809.
02627.
[39] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado,
A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving,
M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I.
Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O.
Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow:
Large-scale machine learning on heterogeneous systems, 2015, Software
available from tensorflow.org. URL http://tensorflow.org/.
[40] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P.
Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J.
Schulman, S. Sidor, Y. Wu, Stable baselines, 2018, https://github.com/hill-
a/stable-baselines.
[41] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J.
Schulman, S. Sidor, Y. Wu, P. Zhokhov, OpenAI baselines, 2017, https:
//github.com/openai/baselines.
[42] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-
scale image recognition, in: Y. Bengio, Y. LeCun (Eds.), 3rd International
Conference on Learning Representations, ICLR 2015, San Diego, CA, USA,
May 7-9, 2015, Conference Track Proceedings, 2015, URL http://arxiv.org/
abs/1409.1556.
[43] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition,
in: 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 2016, pp. 770–778, http://dx.doi.org/10.1109/CVPR.2016.90.