Expert Systems With Applications 137 (2019) 292–307

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Double Q-PID algorithm for mobile robot control

Ignacio Carlucho, Mariano De Paula∗, Gerardo G. Acosta
INTELYMEC, Centro de Investigaciones en Física e Ingeniería del Centro CIFICEN-UNICEN-CICpBA-CONICET, Olavarría 7400, Argentina

a r t i c l e i n f o a b s t r a c t

Article history: Many expert systems have been developed for self-adaptive PID controllers of mobile robots. However,
Received 1 November 2018 the high computational requirements of the expert systems layers, developed for the tuning of the PID
Revised 26 June 2019
controllers, still require previous expert knowledge and high efficiency in algorithmic and software exe-
Accepted 26 June 2019
cution for real-time applications. To address these problems, in this paper we propose an expert agent-
Available online 27 June 2019
based system, based on a reinforcement learning agent, for self-adapting multiple low-level PID con-
Keywords: trollers in mobile robots. For the formulation of the artificial expert agent, we develop an incremental
Reinforcement learning model-free algorithm version of the double Q-Learning algorithm for fast on-line adaptation of multi-
Double Q-learning ple low-level PID controllers. Fast learning and high on-line adaptability of the artificial expert agent
Incremental learning is achieved by means of a proposed incremental active-learning exploration-exploitation procedure, for
Double Q-PID a non-uniform state space exploration, along with an experience replay mechanism for multiple value
Mobile robots
functions updates in the double Q-learning algorithm. A comprehensive comparative simulation study
Multi-platforms
and experiments in a real mobile robot demonstrate the high performance of the proposed algorithm for
a real-time simultaneous tuning of multiple adaptive low-level PID controllers of mobile robots in real
world conditions.
© 2019 Elsevier Ltd. All rights reserved.

1. Introduction the operative conditions, as could happen for instance in a robo-

Mobile robots have a prevailing role in the development of sev-
eral technological systems and currently they are used in a vast
number of industrial applications, such as agro, process, aerospace,
off-shore and others. In addition, in the new era of the fourth in-
dustrial revolution, also called Industry 4.0, there is an increas-
ing participation of these mobile robots in industrial applications
and they are essential parts of their manufacturing execution sys-
tems too. However, nowadays it is of paramount importance that
these types of robots perform their tasks in an autonomous way
and, usually, they must operate in multiple and uncertain environ-
ments, which pose a big challenge for autonomous robot control
strategies.
From an expert system point of view, many proposals were
made for controlling mobile robots for direct applications in in-
dustrial environments. However, the real time adaptability needed
for this type of applications is still an open issue for the major-
ity of these proposals. Under these conditions, the expert system
layer has to be able to adapt to changes in the environment and
∗
Corresponding author.
E-mail addresses: ignacio.carlucho@fio.unicen.edu.ar (I. Carlucho), mariano.
depaula@fio.unicen.edu.ar (M. De Paula), ggacosta@fio.unicen.edu.ar (G.G. Acosta).
tized warehouse application.
The simplicity and fast computation of the Proportional Inte-
gral Derivative (PID) controller made it the most popular low-level
control strategy and it is currently implemented in several mobile
robots’ control systems (Åström & Hägglund, 2006). A PID con-
trol strategy requires the tuning of its parameters according to
the controlled system, usually done for a certain operating point.
Classical control theory has contributed with many tuning meth-
ods (Åström & Hägglund, 2004; Ziegler & Nichols, 1993). However,
in autonomous robots with complex dynamics, often coupled and
highly non-linear, and with multiple degrees of freedom, the PID
tuning techniques tend to have a poor performance requiring an
exhaustive expert knowledge about the system behavior and, of-
ten, about its operative environments too.
The main motivation of this work lies in having a fast intelli-
gent expert system for the simultaneous on-line tuning of multiple
low-level PIDs controllers in real-time. Particularly, mobile robots
exhibit highly non-linear, uncertain and coupled behavior and they
are very difficult, even impossible, to be analytically modeled with
enough accuracy. In addition when multiple PID controllers are im-
plemented they suffer a considerable performance degradation in
the face of variable and uncertain conditions. For these reasons,
adaptive PID control techniques have arisen as a way to set the
controller gains according to the desired system behavior.

https://doi.org/10.1016/j.eswa.2019.06.066
0957-4174/© 2019 Elsevier Ltd. All rights reserved.
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
Classical PID tuning methods have been extended with artificial
intelligence techniques to achieve better performance and higher
adaptability of the PID controllers (Tandan & Swarnkar, 2015).
Adaptability implies the capability of an entity (i.e. intelligent ex-
pert agent) to self-adapt, driving and improving its behavior to
carry out an specific task in a partially (or fully) unknown and un-
certain environment to ultimately achieve a certain goal. In this
sense, reinforcement learning (RL) is one of the most powerful
learning paradigms (Sutton & Barto, 1998), for adaptive control for-
mulations, in which an artificial expert agent learns to solve a con-
trol task only by means of its experience. In the context of adap-
tive PID control for mobile robots, an RL expert agent must learn
to adapt the gains of the PID controllers in an on-line way, whilst
the real-time interactions, between the robot and its environment
occur.
In this paper, we propose an RL agent-based expert system
for adapting on-line the gains of the PID controllers, one of the
most widely used low-level control techniques in autonomous mo-
bile robots. Our present formulation is based on some ideas out-
lined in an our previous work (Carlucho, De Paula, Villar, & Acosta,
2017). However, unlike the previous formulation, in this work a
higher efficient algorithm for a RL agent-based expert system for
on-line autonomous adaptation of low-level PID controllers of mo-
bile robots is presented. The new formulation combines a new and
improved incremental active learning exploration-exploitation pro-
cedure for the double Q-learning algorithm. As other novelty, we
have also incorporated the definition of support data dictionary for
the incorporation of the experience replay mechanism which uses
this data set for fast learning of a low-level control policy. In addi-
tion, a temporal memory is defined to manage the proposed incre-
mental active learning mechanism for a non-uniform state space
exploration while the experience replay mechanism is used for the
double Q-learning updates of function values. The proposed ap-
proach was initially tested in different mobile robots, i.e. terres-
trial, aerial and underwater robots, for which different benchmark
simulation platforms were used. Also, a comprehensive compara-
tive simulation study was made demonstrating the higher perfor-
mance of the expert Double Q-PID Algorithm. Finally, experiments
in a real-time mobile robot demonstrates the high performance of
the proposed algorithm for a real-time simultaneous on-line tun-
ing of multiples adaptive low-level PID controllers.
The article is structured as follows. Section 2 provides a
overview of related works about adaptive PID formulations for dif-
ferent mobile robots. Section 3 provides the necessary background.
In Section 4 the proposed Double Q-PID algorithm is explained fol-
lowed by the obtained results in Section 5. Finally in Section 6 the
final remarks are presented.
2. Related work
Since the original works of Ziegler and Nichols (1942, 1993),
many techniques were developed for the tuning of PID controllers
(Bansal, Sharma, & Shreeraman, 2012; Cohen & Coon, 1953; Åström
& Hägglund, 2004; Wang, Fung, & Zhang, 1999) and most of them
optimize merit functions such as integral absolute error and in-
tegral time squared error (Ogata, 2010; Pessen, 1994; Zhuang &
Atherton, 1993). Usually mobile robots, used in several industrial
applications, are complex systems with changing operative condi-
tions and they have further requirements for tuning, mostly due
to errors in the models and changes in the operative conditions.
For this reason, the attention has been turned towards the devel-
opment of controllers able to auto-tune in order to compensate
for those issues. In this sense, a number of adaptive control for-
mulations were proposed to self adapt the PID gains, using differ-
ent strategies, for a variety of mobile robots. To name some rele-
vant works, Hägglund and Åström (1991) proposed a model-based
293
adaptive formulation using frequency response analysis by means
of tracking a point on a Nyquist curve to initialize the adaptive
algorithm to build up PID gain schedules. Myungsoo, Jun and Sa-
fonov (1999) use unfalsified control techniques (Safonov & Tung-
Ching Tsao, 1997) to propose an iterative procedure to adjust the
PID gains. In this formulation, an initial discretized set of candidate
gains and an operative specification must be supplied, and then if
the algorithm cannot find a set of gains, it must be run again, re-
laxing the performance specification and/or changing the initial set
of possible gains. Chang and Jin (2013) proposed a PID trajectory
tracking controller based on a non-linear kinematic model of the
mobile robots, used to predict the robot trajectory, however only
simulation results were reported. Antonelli (2007); Antonelli, Cac-
cavale, Chiaverini, and Fusco (2003) formulated different adaptive
versions based on PID control laws with an adaptive compensa-
tion of the dynamics for controlling autonomous underwater vehi-
cles (AUVs) in six degrees-of-freedom. However, in such proposals
the control gains must be adjusted manually, first in simulation
and then with the real system during its operation (Antonelli, Chi-
averini, Sarkar, & West, 2001). In the work of Barbalata, De Car-
olis, Dunnigan, Petillot, and Lane (2015) an adaptive on-line tun-
ing method for a coupled two-close-loop proportional controller of
four degrees-of-freedom of an AUV is presented where the gains
of each controller must be determined on-line according to the er-
ror signals. Rout and Subudhi (2017) developed an adaptive tuning
method for a PID controller using an inverse optimal control tech-
nique based on a NARMAX model (Chen & Billings, 1989) for the
representations of the non-linear dynamics for path following con-
trol design for an AUV.
Notice that the aforementioned methods require prior knowl-
edge about the models of the systems and, often, an intensive sim-
ulation phase is needed to set the initial algorithms conditions. The
ideas of auto-tuning methods have inspired the researchers of the
artificial intelligence community to formulate proposals of adaptive
PID controllers (Tandan & Swarnkar, 2015). Mainly, these proposals
were based on the idea of incorporating an extra layer of decision
making for on-line tuning of the PID gains, allowing the controllers
to be able to solve more challenging tasks.
Fuzzy logic theory (Zadeh, 1973) has been combined with PID
control techniques in such a way as to dynamically adjust the
PID gains. In the work of Acosta, Mayosky, and Catalfo (1994) a
fuzzy system was implemented to manage a fuzzy conjunc-
tion originated by an heuristic method for the system parame-
ters estimation, based on experimental measurements of closed-
loop performance. Also, robot motion control systems based on
PID (Sood, 2011) and PI (Sood & Kumar, 2014) control architec-
tures with embedded fuzzy rules were developed to tune the
controllers gains aiming to decrease rise time, removing steady
state error quickly and avoiding overshoot. Similarly, Li, He, and
Yin (2012) proposed a trajectory tracking control for a four-
wheeled differential mobile robot using a closed loop fuzzy PID
control architecture with path and heading error feedback. Simi-
larly, Sheikhlar, Fakharian, and Adhami-Mirhosseini (2013) imple-
mented a fuzzy PI controller for a four wheeled omni-directional
soccer robot using the tracking error and its derivative as in-
puts for the fuzzy inference rules. Analogously, Salhi and Al-
imi (2013) proposed a hybrid fuzzy PID controller, feed with
the outputs of a Kinet sensor, for trajectory planning and obsta-
cle avoidance of a terrestrial mobile robot. On the other hand,
Geder, Palmisano, Ramamurti, Sandberg, and Ratna (2008) have
presented and discussed two different approaches of a fuzzy
logic PID controller for a simulated unmanned underwater ve-
hicle, one using weighted gait combination and the other using
a modification of mean bulk angle bias. Recently, for AUV con-
trol, Khodayari and Balochian (2015) have proposed a self-adaptive
fuzzy PID controller for the attitude control of the robot, based on
294 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
a dynamic model obtained from mechanical principles. Zhao, Yi,
Peng, and Peng (2017) have proposed a self-adaptive fuzzy-PID
controller for a simulated AUV which obtains the required ad-
justment of the PID gains according to the error and the change
of error, in both distance and yaw. Similarly, a number of works
with fuzzy PID formulations have been proposed for unmanned
aerial vehicles (Abu Rmilah, Hassan, & Bin Mardi, 2016; Ahmed,
Kushsairy, Bakar, Hazry, & Joyo, 2015; Cohen, Bossert, & Col, 2002;
Kuantama, Vesselenyi, Dzitac, & Tarca, 2017; Wei & Cohen, 2015).
Also, fuzzy fractional calculus (Takači, Takači, & Takači, 2014) has
been used for fractional fuzzy PID control formulations in mobile
robots applications (Liu, Pan, & Xue, 2015; Shah & Agashe, 2016;
Sharma, Rana, & Kumar, 2014).
However, the design of fuzzy self-tuning PID control strategies
requires some prior knowledge of the system behavior. This is a
big disadvantage since it means that an expert’s prior knowledge
of the technological system is still required in order to setup the
controller, or at least to define the inference rules as well as the
initial controller gains. This is not a trivial task since the range
of the PID gains is crucial for achieving an optimal performance.
Moreover, these problems are aggravated in several mobile robots,
such as AUVs, that have multiply degrees of freedom and highly
coupled dynamics.
Analogously to fuzzy PID formulations, artificial neural net-
works (ANN) (Haykin, 1998) have been combined with PID control
methods to give rise to adaptive PID formulations for controlling
mobile robots. Mostly, the ANNs have been used to learn the dy-
namic models of the systems based on collected data coming from
real or simulated interactions. Calvo-Rolle, Casteleiro-Roca, Quin-
tián, and Del Carmen Meizoso-Lopez (2013) have proposed an in-
telligent hybrid system which combines a rule based system with
artificial neural networks and support vector machines for tun-
ing a PID controller in an open loop way according with the op-
eration point of the system. Rossomando and Soria (2015) pro-
posed a neural PID formulation to control a mobile robot based
on the implementation of a dual control strategy through a kine-
matic controller and a discrete-time adaptive neural PID controller,
which compensates the dynamic nonlinearities of the robot. Also,
adaptive PID approaches have been developed based on the anal-
ogy with feed forward neural networks resulting in a continuous
auto-tuning PID method and the effectiveness of this approach
was only demonstrated for velocity and orientation tracking con-
trol of simulated nonholonomic terrestrial mobiles robots (Singh,
Bisht, & Padhy, 2013; Ye, 2008). For underwater vehicles (Dyda &
Os’kin, 2004), neural networks based self-tuning PID control for-
mulations have been proposed as in the work of Dong, Guo, Lin,
Li, and Wang (2012) where a self-tuning PID controller, for an AUV
with spherical body was presented. The proposed approach con-
sists in a neural network identifier and a self-tuning PID controller
which is realized by a two-layer neural network. The weights of
the two-layer neural network represent the PID gains and its out-
put represents the control commands. However, only simulation
results were presented where only the forward speed of the AUV
was controlled, therefore it is unclear if the method is suitable for
more complex systems with higher degrees of freedom and cou-
pled dynamics. A similar proposal has been made in (Hernández-
Alvarado, García-Valdovinos, Salgado-Jiménez, Gómez-Espinosa, &
Fonseca-Navarro, 2016) where a neural network on-line adjusts the
PID gains according to the position tracking error of a mini AUV. In
spite of the obtained results, it is worth to note that the range of
available PID constants is limited by the neural network activation
function of the output layer. Moreover, the initial values of the PID
constants are determined by the randomly initialized weights of
the neural network. In Pan, Lai, Yang, and Wu (2013) a comparison
of a neural network controller against a PID as a tracking control
strategy for an autonomous surface vehicle with unknown dynam-
ics is made, showing that the neural network controller was able
to perform with an overall lower error. This fact reveals the need
for autonomous expert systems layers that are capable of on-line
adapting the PIDs’ constants. Also, for unmanned aerial vehicles an
implementation of the PID structure as an artificial neural network
was made in Fatan, Sefidgari, and Barenji (2013) for controlling the
altitude of a quadcopter robot and in (Önder Efe, 2011) a fractional
PID formulation (PIλ Dμ ) with neural network aided finite impulse
response approximator to control an unmanned aerial quadrotor
vehicle was formulated. Overall, although the proposed neural PID
controllers achieve a better performance that the standard PID for-
mulation, the estimation range of the PID parameters is quite lim-
ited, due mostly by the activation function of the neural network
output layer.
Real adaptive control strategies require a continuous learning
capacity which is proper of the reinforcement learning paradigm
(Sutton & Barto, 1998). In a RL framework an artificial agent learns
an optimal control policy based on the successive interactions said
agent has with the environment. In this sense, low-level PID con-
trollers of mobile robots could be continuously adjusted by an op-
timal control policy, which maps robot states to optimal PID pa-
rameters according to the current specified operative conditions
(Carlucho et al., 2017). When the low-level control is accomplished
by two or more PID controllers, say for example, one for each
degree of freedom of a mobile robot, the simultaneous tuning is
a challenging task such that a few proposals have been formu-
lated using RL approach for the adjustment of multiple PID con-
trollers of mobile robots. Gloye, Göktekin, Egorova, Tenchio, and
Rojas (2005) proposed three low-level PID controllers (one for the
forward direction, one for the sidewards direction and other for
the angle of rotation) for an autonomous mobile robot, where the
parameters of the PID controllers are selected by a policy gra-
dient based RL method which learns from the observed expe-
rience, summarized in the collected rewards/punishments. How-
ever, the presented formulation was only tested in simulation in
a robot with uncoupled dynamics. Moreover, in order to achieve
a successful performance, this formulation requires a human ex-
pert with knowledge to set the initial PIDs constants that are
then further adapted by the algorithm. Similarly, in the work of
el Hakim, Hindersah, and Rijanto (2013) the standard Q-learning
method (Watkins & Dayan, 1992) was used to self-tune the PID
controllers of a soccer robot. In this particular case, the entire pos-
sible finite state transitions were given in advance, something not
practical for real robotic applications. Moreover, the action domain
was discretized a priori, within a narrow range, requiring human
knowledge for tuning what can be considered as hyper parame-
ters of the presented algorithm. Carlucho et al. (2017) proposed an
incremental version of Q-learning algorithm for adaptive low-level
PID control of mobile robots. In this work, only terrestrial robots
were taken into account for the experiments.
Particularly, Q-learning algorithm (Watkins & Dayan, 1992) is
probably one of the most widespread RL algorithms and it has
been used in many RL formulations. However, the new developed
double Q-learning algorithm (Hasselt, 2010) has significantly im-
proved the learning performance reducing the over estimation typ-
ical of the original Q-learning. Double Q-learning was first de-
veloped by Hasselt (2010) and has since been used in different
branches of RL as in the work of Zongzhang Zhang (2017) who for-
mulated the weighted double Q-learning showing empirical sim-
ulation results. Also, the double Q-learning algorithm has been
adopted for recent developments in the emergent field of deep
learning (LeCun et al., 2015; Mnih et al., 2015) and also these
formulations have been enhanced with experience replay mech-
anisms using past experience (Pan, Zaheer, White, Patterson, &
White, 2018b; van Seijen & Sutton, 2015; Zhang & Sutton, 2017),
demonstrating better performance by smoothing over the data and
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
Fig. 1. Low level PID control module.
eliminating correlation on the samples. Van Hasselt, Guez, and
Silver (2016) popularized the double Q-learning algorithm in the
deep reinforcement learning field successfully solving the ATARI
domain problems. Also, recent advances in fields like macroeco-
nomics have been reported as in Goumagias, Hristu-Varsakelis, and
Assael (2018) who used double deep RL to determine the tax eva-
sion behavior of taxpayer entities.
3. Background
In this section we briefly outline the basic elements necessary
for our double Q-PID formulation which will be exposed in the
next section.
3.1. RL for adaptive PID
The proposed control architecture is based on a classical PID
controller, where its control law is (Kuo, 1995):
u ( kT ) = u ( kT − T ) + k 1 e ( kT ) + k 2 e ( kT − T ) + k 3 e ( kT − 2 T ) (1)
where:
 
k k
k1 = k p 1+ d (2)
T where rt+1 is the instantaneous reward obtained by the system
   
kd T Then, when the optimal state-action function is learned by inter-
k2 = −k p 1 + 2 − (3)
T ki
 
kd
k3 = k p (4)
ki
being kp , ki and kd the parameters (gains) of the PID controller,
T the sampling time and e(kT) the error term used to form the
derivative, proportional and integral terms that in turn yield the
control action u(kT) Eq. (1) at time instant kT, with k = 0, 1, 2, . . ..
Commonly, mobile robots whether terrestrial, aerial, aquatic or
underwater have, at least, two or more degrees of freedom to be
controlled. In this sense, a control module as the depicted in Fig. 1
is used for low-level control. Notice that in the low level control
module there could be as many PID controllers as manipulated
variables. Thus, each PID should determine the magnitude of each
low-level control action (uti ) to be executed on the robot according
295
to the PID control law of Eq. (1). On the other hand, we can think
the parameters of the implemented PID controllers as arranged in
an action vector k = (k1 , k2 , . . . , kD )T , with D being the number of
PID controllers, such that an RL agent needs to select the action
vector seeking to achieve a certain operative goal. Thus, we can
use a two-layer control architecture as in Fig. 2 where the low-
est layer is the PID control module and the highest layer is com-
posed by an artificial agent. The RL agent is continuously learn-
ing by interacting with the environment and fixing the PIDs gains
k1 = (k1p , k1i , k1d ), . . . , kD = (kDp , kD
i
, kD
d
) through the learned control
policy π .
Into the RL framework, this decision problem can be formulated
as a Markov Decision Process (MDP) (Monahan, 1982), where the
aim of the RL agent is to find an optimal policy π ∗ that defines
the optimal controller parameters (kt ) for different system states
(xt ) such that the total expected reward is maximized:
J ∗ = max Jπ = max Eπ {Rt |xt = x} (5)
Then, we can define the optimal state-action value function
(Q( · , · )) as (Sutton & Barto, 1998):
Q ∗ (xt , kt ) = E {rt+1 + γ max Q ∗ (xt+1 , kt+1 )|xt , kt } (6)
transition from the state xt to xt+1 and γ is the discount rate.
actions, the optimal policy can be directly obtained as:
π ∗ = arg max Q ∗ (xt , k ) (7)
k
One of the most widespread RL algorithm, to find the optimal
policy π ∗ , is the well-known off-policy model-free Q-learning algo-
rithm (Watkins & Dayan, 1992). Its simplest one-step update rule
is:
Q (xt , kt ) ← Q (xt , kt ) + α [rt+1 + γ max Q (xt+1 , kt+1 ) − Q (xt , kt )]
k
(8)
where α ∈ (0, 1] is the learning rate and γ ∈ (0, 1] is the discount
factor. According to Eq. (8), the state-action function Q, directly ap-
proximates Q∗ , regardless of the policy being followed. Thus, the
optimal policy can then be easily obtained as described in Eq. (7),
296 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
Fig. 2. Structure of the QPID approach.
with probability of convergence of 1, regardless of the policy fol-
lowed (Bhatnagar, Sutton, Ghavamzadeh, & Lee, 2009; Sutton &
Barto, 1998).
3.2. Double Q-learning
Double Q-learning was first developed by Hasselt (2010) and
has since been applied successfully to solve a number of problems
(Goumagias et al., 2018; Pan, Wang, Cheng, & Yu, 2018a; Van Has-
selt et al., 2016; Zongzhang Zhang, 2017).
Double Q-learning was developed in order to reduce the over-
estimation present in the original Q-learning algorithm, and this is
achieved by a double estimator for the Q function, whereby two
functions QA and QB are used. Each Q function must be updated
with a value from the other, i.e., if the policy uses the function QA
to chose the action according to Eq. (7), then the updating rule of
QA , as in Eq. (8), is:
QA (xt , kt ) ← QA (xt , kt ) + α [rt+1 + γ max QB (xt+1 , kt+1 ) − QA (xt , kt )]
k
(9)
Alternatively, when the action selection according to Eq. (7) is
made using QB , directly the value function updating rule is:
QB (xt , kt ) ← QB (xt , kt ) + α [rt+1 + γ max QA (xt+1 , kt+1 ) − QB (xt , kt )]
k incorporated to X, i.e. X = {X ∪ x }. Note, that the discretization
(10)
Since the update of QA and QB is done with its opposite, each
update takes into consideration a different subset of the data set,
thus removing the bias in the value estimation. Due to the charac-
teristics of the update, the computational burden is not increased,
maintaining the design characteristics of the original Q-learning. In
this sense, the RL proposals using the Q-learning algorithm could
be easily enhanced using double Q-learning to reduce overestima-
tion.
3.3. Incremental discretization of the states and actions spaces
Normally, in the standard Q-learning application a state-action
discretization is made in advance. In this case if the discretiza-
tion is coarse the results could be poor, on the other hand, if it
is too thin the Q-learning algorithm could become intractable. For
this reason, many proposals incorporating function approximators
were made (Buçoniu, Babuska, Schutter, & Ernst, 2010; Timmer &
Riedmiller, 2007) but the action selection step introduces an op-
timization bias which is often problematic and can cause an un-
stable learning process (Neumann & Peters, 2008) or require a lot
of computational load (LeCun et al., 2015; Mnih et al., 2015) not
suitable for robotics systems operating in real time. For this reason
we include a mechanism for incremental discretization of the state
and action spaces. Particularly, in this section we only present the
main ideas of incremental discretization of the state space and ac-
tion space developed in Carlucho et al. (2017). These concepts are
the basis of our current proposal. However here we just describe
the original concepts, while later, in Section 4, we will adapt and
formalize these ideas according to our current proposal.
Let X be a data set for the state space representation; η( · , · ) a
membership function; ρ a scalar that represent a threshold value
and xt the current system state, at time t. So, as the system evolves
over time, step by step, a state transition occurs from one state
xt to another xt+1 . We drop the time subindex t to simplify the
notation, yielding x = xt and x = xt+1 . Then, the incremental con-
formation process of set X is as follows: each time that a transi-
tion state occurs, we must verify if x provides relevant informa-
tion regarding X. For this, we use the membership function, such
that if η (x, x ) > ρ ∀ x ∈ X, then x contributes with relevant in-
formation for the state space representation. Therefore, x must be
could be done until a maximum level Z, being ρ z the discretiza-
tion step for each discretization level z of the state space, i.e. we
have  = (ρ0 , ρ1 , . . . , ρZ ).
As the interactions progress the learning process does too. So,
in our formulation, an agent must fix the PIDs gains according to
the performance requirements. In a Q-learning algorithm, at each
decision step, an action k must be chosen from a given set K of
possible actions by means of a control policy π . In the proposed
approach, we initially have a fairly coarse discretization of the ac-
tion space summarized in the set K. Then, as the agent interac-
tions progresses, the degree of discretization of the set K is prop-
erly augmented in those regions where the best actions lie. This
idea is schematically represented in the simplest form in Fig. 3.
The represented process in this figure is as follows.
First, suppose that the system at a time t is in a state xt and
the agent chooses the action kt (represented by shadow cell in
Fig. 3) from a set of possible actions K (represented by all cells
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
Fig. 3. Incremental action discretization process.
of a row in Fig. 3). Then, suppose that after the selected action
is applied on the system, it evolves to a state xt+1 = xt , i.e., this
means that system remains invariant. It is expected that the agent
will in the next decision step, by means of the policy in Eq. (7),
choose the same action if K remains unchanged. Then, if after N
time steps the agent selects the same action from K and con-
sequently the system evolves towards the same state configura-
tion, the set K will be changed to offer a higher discretization.
This is called the incremental discretization process of K, car-
ried out as is indicated in Fig. 3. That is, the shadow cell, in the
top row, will be discretized with a step δ j , as is shown in the
second row in Fig. 3 and the new configuration of the set K is
obtained.
Again, suppose that the agent selects the same action during N
times and the system remains invariant, again the splitting process
is repeated, and the new K is given as in Fig. 3. Consequently, this
incremental procedure must be systematically repeated each time
that the system remains invariant and as the learning progresses
the action space will be discretized as it is depicted in Fig. 3. Anal-
ogously, as in the state space discretization, the discretization pro-
cess of the action space could be done until a maximum level J,
being δ j the discretization step for each discretization level, i.e.
= δ1 , . . . , δJ , is achieved.
4. Double Q-PID algorithm
4.1. Incremental active learning
In this section we present the incremental active learning
mechanism for simultaneous exploration of the state and action
space.
On that subject, the exploration-exploitation dilemma is a cru-
cial one for RL agents. One potential problem encountered is that
a large amount of robot-environment interactions may be neces-
sary to learn a successful policy. In that sense, the RL agent has to
exploit its current knowledge of the environment and at the same
time it needs to explore the environment sufficiently to learn more
about its reward-relevant structure. To tackle these issues we de-
veloped an incremental active learning procedure which involves
the ideas presented in the previous section.
297
As we have exposed, the interaction between the artificial agent
and its environment starts from a coarse discretization of the state
and action space. From this point on, what could happen is that
the agent chooses an action (Eq. (7)) and the system evolves re-
peatedly to a same successor state. Under these circumstances, we
previously stated that the system remains invariant and a finner
discretization of the action space is required to improve the con-
trol policy.
Let Mt be a temporal memory defined as a tuple Mt = (xt , kt )
containing information about the system state at time t and the
action taken at that moment. Then, to determine if the system
stays or not invariant at a time t, we compare the last N mem-
ories and if these memories are equals it means that the system
has not experienced a change from the perspective of the agent,
i.e., Mt = Mt−1 = . . . = Mt−N . If this is the case, the agent needs
to explore new possible actions. Therefore a new subset of possi-
ble actions is generated through an incremental action discretiza-
tion (explained above) of the action space around kt (see Fig. 3).
For the next decision step, the agent will have to choose its ac-
tion from this recently created subset of actions. In addition, it is
worth noting that while the incremental discretization of the ac-
tion space is being made, a discretization of the state space must
also be done (as described in the previous section). In summary,
both procedures, the incremental discretization of the action space
and of the state space, must be made each time that the system
remains invariant, i.e., each time that Mt = Mt−1 = . . . = Mt−N .
The incremental active learning mechanism, by means of the
incremental discretization of both the state and action space, fur-
thers the exploration into a promissory region of the feasible so-
lution space. In this way, the agent avoids the unnecessary explo-
ration of large regions, concentrating in a more relevant space, go-
ing from a coarse discretization to a more refined and focused one.
This idea can be seen in Fig. 4 where it can be seen a promis-
ing initial region defined by the surrounding regions around the
shaded reachable states. Within this region there are reachable
states, all of them representing sub-regions of the state space.
Thus, the set of all reachable states defines what we call a reach-
able region of the state space. In this way, working with large data
sets is avoided and, as a consequence, the computational cost is
lowered, a desirable goal when working with autonomous mobile
robots.
298 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307

Fig. 4. Schematic representation for the incremental discretization of the state space.

4.2. Stochastic experience replay
Typical of deep RL formulations, stochastic experience replay
mechanisms have been developed for faster off-line updates of the
value functions approximators in continuous domains (Mnih et al.,
2015; Zhang & Sutton, 2017). This is a simple but efficient mech-
anism which repeatedly samples from a buffer R of recent experi-
ence, recorded from real transitions τi = (xt , ut , rt , xt+1 ).
0 , K, η (·, · ), n, m, N, ,
shows the main steps of our proposed approach, which we will ex-
plain in detail in the current subsection. In addition, to clarify the
computational procedure and facilitate the reproducibility of the
Double Q-PID algorithm, we make the code available in an open
repository1 .
Algorithm 1 Pseudo-code for the Double Q-PID algorithm.
1: Inputs: α , γ , r (·, · ),

This strategy behaves like model-based RL paradigms which

2: x0 ← get sensor readings
simulate one-step transitions repeatedly to update the model and
3: Initialize Replay Buffer R
action-value functions. However in our proposal, unlike model-
4: Initialize X
based formulations, no model is being constructed. We simply ran-
5: Initialize Q A , Q B
domly select, in each algorithm iteration, a number Γ of transi-
6: loop
tions selected from the replay buffer R to update the state-action
7: - greedy action selection → kt
value functions. The replay buffer must be bounded and it can
8: Set the current temporal memory as Mt = (xt , kt )
only store a maximum number m of states transitions, i.e. R =
9: Interact with the system, observe the transition state xt+1
(τ1 , τ2 , . . . , τi , . . . τm ). When R reaches its maximum size, the oldest
and the obtained reward rt+1
information will be removed in order to store the new experienced
10: if the system is variant then
transitions.
11: Store the current transition τ = (xt , kt , xt+1 , rt+1 ) in the
When the experience replay mechanism is being used, it is ben-
replay buffer R
eficial to initialize the state-action values functions, QA and QB ,
12: Identify the nearest state xi ∈ X to xt+1
randomly. In practice, the random initialization of value functions
13: if xt+1 is in the vicinity of xi then
in RL formulations is not a trivial task, however it has a dou-
14: Double Q-learning updating → Update Q A and Q B ac-
ble benefit. On the one hand, this is desirable since it contributes
cording to Eqs. (9) and (10)
with exploration, crucial in RL. On the other hand, more impor-
15: else
tant is still the fact that the possibility of bias in the value func-
16: Incorporate xt+1 in X
tions decreases. As an additional advantage we can say that ran-
17: Set up Q A and Q B for incorporating the new point xt+1
domly initializing the functions generates one less input parameter
for the algorithm. This represents a significant advantage since the
18: end if
smaller the number of hyper parameters, the lower the influence
19: else
of the designer on the behavior of the algorithm. This simplifies
20: Incremental active learning
re-implementation of the algorithm and the ability to easily switch
21: end if
platforms.
22: Stochastic experience replay
23: Set the system state for the new execution loop
4.3. Algorithmic statement 24: end loop

In Algorithm 1 we outline the pseudo-code of the proposed

Double Q-PID algorithm in a compact form. This pseudo-code 1
https://github.com/IgnacioCarlucho/Double_QPID.
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307 299

In Line 1 the input parameters of the algorithm are indicated.

Some of these parameters are common in Q-learning applications,
such as the learning rate α , the discount rate γ , the reward func-
tion r( · , · ) and the initial exploration probability
0 . However
some are specific of the proposed algorithm such as the initial
coarse set of possible actions K, the membership function η( · , · )
for incremental state discretization, the number n of initial inter-
actions for the replay buffer initialization and the maximum ad-
missible size of the data set X. The parameter N is the number
of temporal memories (Mt ) that must be compared for determin-
ing the invariability of the system. Finally,  and
contain the
thresholds for the different levels of incremental discretization of
the state and action spaces respectively.
From Line 2 to 5 the algorithm components are initialized. In
robotic applications the robot initial state, x0 , commonly, can not
be fixed in advance. So, the initial state is composed by sensor
measurements as is indicated in Line 2.
In Line 3 the replay buffer (R) is initialized. Following, in Line Fig. 5. Platforms used for testing the algorithm.
4, the data set X is initialized with the n + 1 states experienced
during the initial interactions, i.e. X = {x0 , x1 , . . . , xn+1 }. Once we
have the initial representations of the state and action spaces, X In Line 22 the stochastic experience replay mechanism (detailed
and K, in Line 5 the state-action values QA and QB are randomly in Section 4.2) is carried out for faster updating of the value func-
initialized. tions QA and QB using the replay buffer R. Finally, in Line 23 the
The Double Q-PID algorithm can run indefinitely, if it were nec- system state for the algorithm must be updated for the next loop
essary, executing the loop between Line 6 and Line 24 continu- execution. So, if the system varied (condition in Line 10 was true)
ously. In our formulation we propose to use an
-greedy scheme the new current system state will be the xi identified in Line 12
for action selection (Line 7) based on QA and QB . and if the system remained invariant, directly the new system state
First it is necessary to choose, e.g. randomly, between updating will be the transition state observed during the agent-robot inter-
the (QA or QB ) value function according to Eq. (9) or Eq. (10) as action in Line 9.
appropriate. Secondly, with probability 1 −
t the greedy action is
obtained from on QA , if updating A was chosen, or based on QB , 5. Results
if updating B was chosen. With a probability
t a non-greedy ac-
tion is selected. Then, in Line 8 with the current state (xt ) and the In this section we present different experiments to test our
current selected action (kt ), the temporal memory is updated as proposed algorithm and discuss the obtained results. First, we
Mt = (xt , kt ). Once the action kt was determined, the gains of the outline the experimental setup where we present the platforms
low-level PID controllers (in PID module of Fig. 2) are set. Thus, used to test the performance of our proposed approach and also
this low-level controller interacts with the robot (Fig. 1) and, af- we present the setup for the reinforcement learning experiments.
ter one step ahead interaction, we observe the transition state xt+1 Then, simulation results are shown followed by a performance
and we compute the reward signal rt+1 (Line 9) using the reward comparison made between the proposed approach and the origi-
function. nal adaptive incremental Q-learning method. Finally, real-time re-
After the interaction has occurred, we must decide if the sys- sults using the Pioneer 3-AT robot are presented.
tem remains invariant or not, by comparing the last N the tempo-
ral memories (Mt , Mt−1 , . . . , Mt−N ) according to the procedure ex- 5.1. Experimental setup
plained in Section 4.1. Then, this process is carried out along the
conditional statement from Line 10 to Line 18. When the system 5.1.1. Experimental platforms
“is variant” this mean that the last N temporal memories are not The proposed approach was first tested in different simulated
equal. Therefore, we must incorporate the experienced transition τ environments using three different robotics platforms depicted in
in the replay buffer R (Line 11). Also, as was previously developed Fig. 5. Then experiments using a real robot were made. The plat-
in Section 4.1, we have to identify the nearest2 state xi ∈ X to the forms used for the simulated experiments were the Pioneer 3-
current system state xt+1 . If xt+1 is inside the vicinity of any sate AT (Fig. 5a), a Quadrotor (Fig. 5b) and the ICTIOBOT (Fig. 5c), an
xi ∈ X (Section 3.3), we have to update the state-action value func- autonomous underwater vehicle (AUV). For the real-time experi-
tions Q A (xi , kt ) and Q B (xi , kt ) according to the double Q-learning ments an implementation on a Pioneer 3-AT, available in our labo-
rules (Section 3.2). In the case that xt+1 gives relevant information ratories, was carried out.
with respect to X, i.e. η (xt+1 , xi ) > ρt ∀xi ∈ X, xt+1 has to be in- Notice that all used platforms are completely different in term
corporated in X (Line 16). The incorporation of xt+1 , means also of dynamic behavior and also have distinctive characteristics which
the augmentation of QA and QB with initial (random) state-actions serve well as to test the adaptive performance of the proposed
values for all (xt+1 × K) (Line 17). On the other hand, when the algorithm in different conditions. In this way, we aim to demon-
system “is not variant” (Line 19) the incremental active learning strate the adaptive ability of the proposed approach for adaptive
procedure (Section 4.1) must be carried out as is indicated in Line. low-level PID control of mobile robots operating in different envi-
ronments.
The first robot used is the Pioneer 3-AT, a mobile robot with
2
four differential drive wheels, extensively used in several research
Note that to simplify the reading, we say “nearest” because we use the eu-
clidean distance as membership function η (xt+1 , xi ) in our applications (Section 5),
works as an experimental platform. The independent turning ac-
but keep in mind that η is an algorithm input and it can be defined in the most tion of the wheels allows it to simultaneously control both the
convenient way. forward and the turning speed. In addition, the manufacturer pro-
300 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
vides an open source model suitable for implementation in Gazebo
with ROS. The other simulation platform used is the Hector-
quadrotor (Alejo, Cobano, Heredia, & Ollero, 2016; Arokiasami,
Vadakkepa, Tan, & Srinivasan, 2016) which is a small flying robot.
This unmanned aerial vehicle (UAV) has also available a simula-
tion platform for Gazebo and ROS. Finally, the proposed algorithm
is tested for the adaptive low-level PID control of an underwater
robot, the ICTIOBOT. It is an autonomous underwater vehicle (AUV)
equipped with four thrusters that allow for a control of 4 degrees
of freedom (DOF), namely: surge, sway, pitch and yaw. A partic-
ularity of the AUVs is their coupled and high nonlinear dynamics
and therefore the tuning of low-level PID controllers is not a sim-
ple task. For the ICTIOBOT simulation, a dynamic model was de-
veloped by Petit, Paulo, Carlucho, Menna, and Paula (2016) and a
simulated environment with ROS is used to test the proposed al-
gorithm.
All the simulation environments used for the experiments are
available to the public and we provide a tutorial on how to set
up the environments in the corresponding github repository where
the code of the algorithm is hosted too3 .
-greedy ex-
5.1.2. RL setup
Controller setup
t that represents the probability of the agent of taking an action
In the series of experiments conducted, the actions taken by the
incremetal Q-learning agent are the gains of the PID controllers.
We employed one PID controller for each degree of freedom, thus:
t =
0 +
1 e−t (13)
kt = ( k1p , k1i , k1d , k2p , k2i , k2d , . . . , kDp , kDi , kDd ) (11)
0 = 0.05 and
1 = 0.3 for all ex-
where D varies with the application and the amount of degrees of
freedom that are being controlled.
For both the Pioneer 3-AT and the Hector-quadrotor, we control
the movements on the horizontal plane. So, here we control the
linear and angular speed only, therefore we only have two degrees
of freedom, i.e. D = 2. In theses cases, kt is a gains vector while the
control action vector to be executed on the robot is ut = (u1 , u2 ).
Here, u1 is the control command for linear instantaneous speed
and u2 is the control command for the rotational instantaneous
speed with respect to the vertical z axis. In the case of the IC-
TIOBOT, we only control the movements in three degrees of free-
dom using four thrusters, therefore the composition of the manip-
ulated control variable is ut = (u1 , u2 , u3 , u4 ). Note that while the
ICTIOBOT has four degree of freedom (surge, heave, yaw, pitch),
we consider the case where the inmersion manuever is made with
zero pitch. Therefore the control must be made on the horizontal
plane, resulting in a control problem in three degrees of freedom,
i.e. surge, heave and yaw are controlled.
It is important to note that the initial values of kt are taken
from a uniform random distribution such that kimin ≤ ki ≤ kimax . By
randomly initializing the values of kt we encourage the algorithm
to explore, while at the same time, we reduce the number of
hyper-parameters that the user needs to supply. Therefore, sim-
plifying the user requirements when working with real-time plat-
forms.
Reward function: The reward function is defined based on the
operative requirements of the system. So, in our experiments these
operative requirements are velocities setpoints. In this sense, we
use a Gaussian function to shape the reward of the form:
 
1
xt − xreq
2
rt (xt , xreq ) = exp −
2π σ 2σ 2
with σ being a free parameter that determines the width of the
Gaussian function and the double bar operator (|| · ||) denotes the
3
https://github.com/IgnacioCarlucho/Double_QPID.
euclidean distance between the current state xt and the setpoint
xreq . It is worth noting that both the current state, xt , and the re-
quested state, xreq , are vectors with dimensions varying depend-
ing on the degrees of freedom, while the reward is a scalar value.
However, notice that the Gaussian function always gives a high
positive reward signal when the current system state (xt ) is close
to its fixed target (xreq ) and a lower reward is obtained when the
distance between xt and xreq is higher.
Exploration/Exploitation setup: Another important aspect of the
RL formulation is how the exploration-exploitation of the agent
knowledge is implemented. If the learned representation is ex-
ploited too early, the agent performance may reach a local max-
imum failing to see better policies due to the lack of exploration
of the state-space. On the other hand, if the exploration is too
high, the learned policy is never implemented since the agents
randomizes its behavior. A common practice is a middle ground,
with a high exploration policy during the first stages of learning,
when the agent has little knowledge of the problem, and over time
change the behavior to that of exploiting the learned represen-
tation, something that commonly happens on later stages of the
learning processes. Thus, to this end we utilize an
ploration scheme with an exponential decay rate of the parameter
based on the current policy, or rather take an exploratory action
based on a random policy. Then, the decay rate is defined as:
where, unless otherwise stated,
periments.
5.2. Simulated experiments
Following, the simulation experiments and the obtained results
are presented. To organize the presentation, we will show the re-
sults in different subsections, one for each simulated robot plat-
form.
In all cases, unless otherwise stated, we defined the remain-
ing inputs of the algorithm as follow: the learning rate is fixed as
α = 0.20; the discount factor is γ = 0.95; the shape parameter of
the reward function is σ = 1.0; to initialize the replay buffer we fix
the number of initial interactions in n = 8; for temporal memories
comparison we use N = 7 memories; for incremental discretiza-
tion of the state space we use  = (ρ0 , ρ1 , . . . , ρmax ) with ρ =
−((ρmax − ρmin )/max ) + ρmax being ρmax = 0.25, ρmin = 0.005
and max = 20; and for the incremental discretization of the action
space we set δ0 = 0.25 and δl = δl−1 /2 with lmax = 8 for each ac-
tion dimension.
Also, it is worth noting that during all preliminary trials, the
Double Q-PID algorithm had quickly learned to cancel the deriva-
tive terms, i.e. the derivative terms were almost always 0, or very
close to 0 and, in practice, they can be ignored. Consequently, in
order to show the results as clearly as possible, we have decided
to exclude the derivative terms since they are practically 0, or very
close to 0, so they can be ignored. Following, only for practical rea-
sons and unless stated otherwise, we do not include the deriva-
tive terms, in consequence, the vector action of the RL agent is
kt = (k1p , k1i , . . . kDp , kD ).
i
(12)
5.2.1. Mobile terrestrial robot
As we have stated, to test the proposed algorithm in a ter-
restrial robot, simulations experiments were performed using the
Gazebo simulator for the Pioneer 3-AT. As stated earlier, there are
two manipulated variables ut = (u1 , u2 ) and the system state is
defined as xt = (vx , wz ) being vx and wz the controlled variables,
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307 301

Fig. 6. Results of Test 1. Simulated Pioneer 3-AT with vxre f = 0.4 m/s and wzre f = −0.3 rad/s.

Fig. 7. Results of Test 2. Simulated Pioneer 3-AT with vxre f = −0.3 m/s and wzre f = −0.4 rad/s.

Fig. 8. Results of Test 3. Simulated Hector-quadcopter with vre f = −0.5 m/s and wre f = −0.2 rad/s.

i.e. the linear and angular velocity respectively4 . Therefore, in the
low level PID module (Fig. 2) there are two low-level controllers
(D = 2). Unless clarified, for all the tests carried out in the remain-
der of this work, the initial values for the gains (kdj ) in the vector
kt are randomly selected, such that 0 ≤ kdj ≤ 2.
Fig. 6 shows the results obtained for the experiment Test 1.
In this trial, the reference for the controlled variables was set as
vxre f = 0.4m/s and wzre f = −0.3m/s. In Fig. 6a the obtained con-
trolled velocities are shown. Here, we can see how in less than a
few seconds the desired velocity setpoint is achieved. Fig. 6b shows
a schedule of the selected controller gains. As can be seen, after
about 100 seconds, the agent finds a suitable combination of con-
troller gains, therefore it continues to seek a solution around these
values. This fact can also be seen Fig. 6c where the evolution of
the incremental action discretization level is shown. Here we can
see how the incremental discretization of the action space evolves
over time, reaching finally the fixed maximum level lmax = 8. It can
be seen that after about 100 seconds the agent is starting to find
4
To simplify the notation we omit the subindex t for vx and wz
a set of gains to successful drive the robot according to its op-
erative desired setpoint. This means that the action selection has
stabilized allowing the incremental discretization of the state and
action spaces to occur by means of the incremental active learning
procedure developed in Section 4.1. In this sense, we can think that
the agent successfully adapts to the required conditions and, there-
fore, specializes in the search for actions (controller gains) within a
certain region. It is worth noting that in under 300s the algorithm
is able to reach the maximum discretization level for the action
space, obtaining the optimal set of controller gains.
A second independent trial, Test 2, was also carried out using
this platform and the obtained results are shown in Fig. 7. Analo-
gously to the previous trial, Fig. 7a shows the robot instantaneous
velocities, Fig. 7b shows the controller gains and Fig. 7c the evo-
lution of the incremental action discretization level. In this case
we use a completely different setpoint with vxre f = −0.4 m/s and
wzre f = −0.3 rad/s, meaning that the vehicle must perform a left
turn while moving backward. The objective of this is to test the
algorithm for different operating conditions, since the dynamic be-
havior of the Pioneer 3-AT is different accordingly to the direction
302 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307

Fig. 9. Simulation experiment of Hector-quadcopter with vre f = 0.3 m/s and wre f = 0.2 rad/s under wind disturbance.

Fig. 10. Results of Test 4. Simulated ICTIOBOT with vxre f = −0.2 m/s, vzre f = 0.0 m/s and wzre f = 0.2 rad/s.

Fig. 11. Results of Test 5. Simulated ICTIOBOT with vxre f = 0.15 m/s, vzre f = 0.05 m/s and wzre f = −0.25 rad/s.

of motion. So, the different velocity requirements for Test 2 with
respect to Test 1 show the adaptive performance of the algorithm.
It can be seen that the performance of the algorithm is similar to
the previous case and, again, the required velocities are obtained
almost immediately, reaching the maximum discretization level in
under 300s.
5.2.2. Quadracopter
Similar to the previous experiments, following we present sim-
ulation results for the quadracopter robot.
Fig. 8 shows the result for the trial Test 3, where we aimed
to control the flying robot in a horizontal plane. In this way, the
controlled magnitudes are the forward and turning speed, v and w
respectively. Therefore, for this the control task, once the robot is
hovering, our proposed algorithm starts to run with the following
velocity requests vxre f = −0.5 m/s and wzre f = −0.2 rad/s. As can be
seen in Fig. 12a the linear and turning velocities achieve their ref-
erence values, both after an early overshoot (around the first five
seconds), however the Double Q-PID algorithm rapidly stabilizes
the controlled magnitudes. Fig. 12b shows the gains schedule for
the low-level controllers. Similar to the trials in Section 5.2.1, in
Fig. 12b it is clear that as the learning process progresses, the agent
focuses on improving its policy and only actions in a bounded re-
gion of the actions space are executed, that is, a higher discretiza-
tion level (Fig. 12c) of the state and action spaces is achieved by
the incremental active learning mechanism (Section 4.1).
Aiming to demonstrate the robustness an adaptability of the
Double Q-PID algorithm, in Fig. 9 the results for a more demand-
ing trial are shown. In this case the quadcopter is facing medium
intensity wind force with the following velocity requests vxre f =
0.3 m/s and wzre f = 0.2 rad/s. As can be seen, the proposed algo-
rithm successful manages the robot achieving their reference re-
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307 303

Fig. 12. Simulation experiment of ICTIOBOT with vxre f = 0.20 m/s, vzre f = 0.0 m/s and wzre f = 0.20 rad/s under current.

Fig. 13. Simulation experiment using QPID to control the ICTIOBOT with vxre f = −0.2 m/s, vzre f = 0.0 m/s and wzre f = 0.2 rad/s.

Fig. 14. Heat maps of Q-values for the experiments of the trial Test 4 using DQPID and QPID.

quests in a short time period. As can be seen in Fig. 9a, there is a Table 1
Performance indexes summary for the comparative study between
negligible swing in the velocities due to the effect of the variable
DQPID and QPID.
medium wind force.
Comparative summary

Algorithm MAHA ED MSE ST

5.2.3. Underwater vehicle
In this subsection we present the results obtained when con- Test 1 DQPID 0.2271 0.00282 0.01438 5.07
QPID 0.4904 0.00351 0.01813 7.51
trolling the underwater vehicle ICTIOBOT. Unlike previous cases,
Test 2 DQPID 0.2141 0.00276 0.01497 5.04
we are facing a control problem of 3DOF due that we will control QPID 0.4752 0.00337 0.01864 7.03
the surge (vx ), sway (vz ) and yaw (wz ) velocities. For this reason, Test 3 DQPID 0.1501 0.00240 0.01690 4.81
we use a control module with three low-level controllers for this QPID 0.1797 0.00274 0.01906 5.07
3DOF control task. Moreover, the highly coupled dynamics of the Test 4 DQPID 0.2561 0.00409 0.01674 10.12
QPID 1.0858 0.02813 0.02441 –
vehicle entails a more challenging control task for the control sys-
Test 5 DQPID 0.3957 0.00368 0.01171 18.09
tem. QPID 0.7357 0.01395 0.01844 40.16
Fig. 10 shows the results for a trial, Test 4, with vxre f = −0.2 m/s,
vzre f = 0.0 m/s and wzre f = 0.2 rad/s. As it can be seen the proposed
algorithm successfully controls the system, with only a little initial son only the initial coarse discretized action space was available
overshoot in the controlled magnitudes. However the AUV achieves for the selection of controllers gains. However, as it can be seen
the requested setpoint in almost 10s (Fig. 10a). Notice that during the proposed algorithm successfully controls the system with only
the first 100 seconds the system was notably varying and conse- a little initial overshoot in the controlled magnitudes and the AUV
quently the system memories (Mt ) were varing too. For this rea- achieves the requested setpoint in almost 10 seconds (Fig. 10a).
304 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
Fig. 15. DQPID for a real-time implementation in Pioneer 3-AT with vxre f = −0.2 m/s and wzre f = 0.1 rad/s.
Fig. 16. Results of the controlled variables for different real-time implementations of DQPID in Pioneer 3-AT.
Another trial was made with different operative references:
Test 5. In this case, the fixed setpoint was vxre f = 0.15 m/s, vzre f =
0.05 m/s and wzre f = −0.25 rad/s. With this operative reference, we
aim to test the algorithm in a more complex control task, that im-
plies moving forward while turning and submerging (z axis points
downwards). Moreover, the dynamic behavior of the vehicle is dif-
ferent from that required in Test 4 for multiple reasons, for exam-
ple, the behavior of the thrusters is not the same when moving
forwards than backwards, the robot geometry is not symmetrical
and therefore the drag coefficients are not the same. The results of
Test 5 are shown in Fig. 11 and, despite the facts mentioned, we
can see that the algorithm successfully solves this control task. We
can see in Fig. 11a that again a small overshoot is present, none
the less, the velocities are achieved quite fast (around 12 s). Simi-
lar to Test 4, in Fig. 11b and c it can be seen that during the first
moments due to the variability of the system memories only the
initial coarse discretized action space was available for the selec-
tion of the controllers gains.
In addition, analogously as it was made for the aerial robot,
here we also outline a more demanding simulation experiment
for the ICTIOBOT regarding medium intensity ocean currents, as
it is explained in the outstanding book of Fossen (2002) for ma-
rine control systems. Fig. 12 shows the obtained results for this
trial where we can see that the Double Q-PID algorithm success-
fully controls the requested variables. Also, in the Fig. 12a it can be
seen that the AUV reaches its specification in a short time with an
insignificant variation around the setpoint.
5.3. Performance comparison
In order to make a performance comparison with other com-
parable methodology, in this section, we present a comparison be-
tween the proposed Double Q-learning (DQPID) and the original
Incremental Q-learning (QPID) strategy (Carlucho et al., 2017). For
this matter, we repeat all trials of the previous section but using
the QPID algorithm.
In order to perform a comparative evaluation, different indexes
were calculated to compare the performance of the algorithms.
For this comparative, we use as performance metrics: the Maha-
lanobis distance (MAHA), the euclidean distance (ED), the mean
squared error (MSE) and the setting time (ST) given in seconds. In
Table 1 there is a summary of the different performance metrics
computed for all the performed trials. As can be seen the DQPID
algorithm always outperforms to the QPID algorithm.
Notice in Table 1 that when the QPID algorithm is used to con-
trol the underwater vehicle, in Test 4, there is not setting time re-
ported. This means that the QPID algorithm did not achieve the
required operative conditions, while DQPID managed to reach the
setting time in over ten seconds. In Fig. 13 we show the results
obtained for Test 4 when the QPID algorithm is used. Compar-
ing these results against those obtained when the DQPID algo-
rithm was used, Fig. 10, brings out the far superior performance of
the DQPID algorithm. In addition, when analyzing the velocity re-
sponses, Fig. 13a, we found out that the incremental QPID was un-
able to stabilize these controlled variables. This poor performance
contrasts with the successful results obtained when the DQPID al-
gorithm is used, shown in Fig. 10a, highlighting even more the im-
provement of the proposed DQPID algorithm with respect to the
original QPID.
In Fig. 14a representation of the state-action function Q in a 2-
dimensional plot is shown. The horizontal axis represents the for-
ward velocity vx , the vertical axis represents the turning velocity
wz and a heat map represents the Q-values. It can be seen that
the highest Q values are in the proximities of the required set-
point. However, if we compare the heat map shown in Fig. 14a and
that shown in Fig. 14b, it is clear that when the DQPID is used the
Q values are strongly marked up. This could be attributed to the
fact that the DQPID algorithm implements a stochastic experience
replay mechanism that takes advantage of the past experience,
seen by the agent, to continuously update the state-action function
what allows agent a faster improvement of its performance.
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
Fig. 17. DQPID for a real-time implementation in Pioneer 3-AT with setpoint changes.
Fig. 18. Heat map.
5.4. Real-time implementation
In this subsection we validate the proposed algorithm in a real
mobile robot. Following, we show the results obtained when im-
plementing the proposed algorithm in a real-time platform, the Pi-
oneer 3-AT. In this case, the input parameters for the DQPID al-
gorithm are the same that those used in the previous trials. Out-
door trials were perform in a field, with uneven ground, making
the control problem a more difficult task.
In Fig. 15 the obtained results for a trial with a fixed setpoint
vxre f = −0.2 m/s and wzre f = 0.1 rad/s are shown. As it can be seen
the DQPID algorithm is able to successfully reach the desired ref-
erence conditions in a short time period while successfully sta-
bilizing the PID constants. Following we present, in Fig. 16, re-
sults for different trials which have been made for other opera-
tive references. In all those cases it can be seen that the DQPID
algorithm is consistently successful in reaching the requested
setpoints.
Finally, in order to further demand the adaptive ability of the
algorithm we set a test where the reference is suddenly changed a
number of times. Initially we set the setpoint as vxre f = 0.2, wzre f =
0.1 after 200 s we change the request to vxre f = −0.4, wzre f = 0.3
to then finally go back to the initial request. Fig. 17 shows the
obtained results for this trial. As it can be seen the algorithm
is able to reach all the desired velocities in a fast way achiev-
ing a smooth behavior of the robot. In addition, it is noteworthy
that there are no significant overshooting and unstable behaviors
at the times of the reference changes, while also the algorithm
successfully stabilizes the PID constants. Fig. 18 shows the result-
ing heat map of Q values for this trial where we can clearly see
that the hotter regions correspond with that where the setpoints
laid.
305
6. Concluding remarks
In this work we have proposed a model-free algorithm for a
RL agent-based expert system for adaptive low-level PID control of
mobile robotic systems, namely the Double Q-PID algorithm, which
can: (i) adaptively drive the parameter selection of the low-level
PID controllers of mobile robots, thus improving the adaptability
and the real-time performance;(ii) make a controlled exploration
through the most promising regions of the state and action spaces
by means of an sparse incremental exploration/exploitation pro-
cess; (iii) while an incremental discretization of the state and ac-
tion spaces is carried out, by means of the proposed incremental
active learning mechanism, thus obtaining more updated informa-
tion and data efficiency and consequently improving the accuracy
of the environmental information for the action decision-making.
The proposed Double Q-PID algorithm demonstrated a success-
ful performance obtaining satisfactory results for multiple trials,
carried out in different (simulated and real) mobile robotic plat-
forms, as it was shown in the results section. Furthermore, com-
parative studies with a different algorithm were provided, showing
how our proposal surpasses the performance in multiple domains
and operative conditions. The extended amount of results is indica-
tive of the adaptive capabilities of our proposal. Moreover, it is im-
portant to note that the optimal PIDs parameters were obtained
in an extremely short amount of time, within seconds, making the
implementation of the algorithm feasible for real-time robotic ap-
plications.
The proposed methodology works without the need of a model,
unlike most classical control methods, and with a low requirement
of transitional data, a problem faced by most of the modern rein-
forcement learning algorithms being researched. In this sense, our
proposed approach takes the best of both worlds, model-free ap-
proaches and data efficient algorithms, achieving an successful per-
formance. The data efficiency could be appreciated in terms of the
learning speed achieved in the performed trials. It can be quan-
tified by the number of interactions needed to get an acceptable
performance. Clearly, this is an advantage for real world applica-
tions for example with respect to other recent proposals like the
deep reinforcement learning approach.
Another issue no less important is that the proposed method-
ology is robust in terms of hyper-parameter tuning, which lowers
the requirements for human operators in deployment on different
robotic platforms. That is, in addition to the common setup param-
eters of any RL application, perhaps the most demanding point is
to provide an approximate initial coarse discretization of the ac-
tion space. Instead, in our work we use a random and wide initial-
ization for the initial PIDs gains, since our algorithm has proved
to be robust to the initialization, as can be seen in the results.
The algorithm robustness was demonstrated in the implementa-
tions for the different platforms. It highlights that there is no need
306 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
to fine tune the initialization of the hyper-parameters for the dif-
ferent controlled systems. In real time robotic applications this is
a paramount advantage since no intensive user knowledge and in-
tervention is needed in the setup phase.
A potential improvement may be to extend this proposal to
work also in higher layers of decision-making systems for con-
trolling mobile robots. For example, the Double Q-PID algorithm
could be used as part of autonomous trajectory tracking control
systems to solve end-to-end control problems typical of mobile
robots. As future works the authors suggest to extend the method-
ology by means of exploring the possibility of extending the ac-
tion discretization to a continuous space effectively breaking loose
of the constrains of space discretization. Briefly, this discretization
could be done sparsely, and complemented by a continuous space
representation by means of a combination of different techniques.
In addition, the estimation of the parameters of the PIDs could be
extended by another type of continuous RL techniques that use,
for example, neural networks as policy approximators such as pol-
icy gradient methods. Furthermore, it should also be possible to
extend the proposal to solve the navigation problem in mobile
robots, by providing the self-tuning of a series of cascade PID con-
trollers. Nonetheless, the proposed agent-based expert system for-
mulation is general, in such a way that could be implemented in
another real platforms, such as autonomous underwater vehicles,
quadcopter or drones and others. Also, the proposal could be used
in problems from different domains, to name a few, such as the de-
cision making system for maximum power point tracking of photo
voltaic sources and also for wind energy conversion systems, elec-
tric engine control systems and others.
Finally, we have to mention that the stability of our proposed
Double Q-PID algorithm was demonstrated through an exhaustive
experimentation. However, the stability analysis in a theoretical
and analytic way (like in classical control formulations, for ex-
ample, made by the Routh-Hurwitz stability criterion or Lyapunov
analysis) of the model-free RL-based algorithms (like our proposal)
is yet an open issue (Busoniu, de Bruin, Toli, Kober, & Palunko,
2018). Therefore the stability of the overall Double Q-PID algorithm
poses an even greater challenge for future research works.
Disclosure of conflicts of interest
The authors have no conflicts of interest to declare.
Credit authorship contribution statement
Ignacio Carlucho: Conceptualization, Data curation, Formal
analysis, Writing - original draft. Mariano De Paula: Conceptu-
alization, Data curation, Formal analysis, Writing - original draft.
Gerardo G. Acosta: Conceptualization, Formal analysis, Writing -
original draft.
Acknowledgements
We gratefully acknowledge the support of NVIDIA Corporation
with the donation of the Titan X Pascal GPU used for this research.
Particularly, we would like to thanks the National Research and
Technology Council of Argentina (CONICET) for the economic sup-
port of Eng. Ignacio Carlucho with a Ph.D. fellowship.
References
Abu Rmilah, M. H. Y., Hassan, M. A., & Bin Mardi, N. A. (2016). A PC-based simu-
lation platform for a quadcopter system with self-tuning fuzzy PID controllers.
Computer Applications in Engineering Education, 24(6), 934–950. doi:10.1002/cae.
21769.
Acosta, G. G., Mayosky, M. A., & Catalfo, J. M. (1994). An expert PID controller uses
refined Ziegler and Nichols rules and fuzzy logic ideas. Applied Intelligence, 4(1),
53–66. doi:10.10 07/BF0 0872055.
Ahmed, S. F., Kushsairy, K., Bakar, M. I. A., Hazry, D., & Joyo, M. K. (2015). Atti-
tude stabilization of quad-rotor (UAV) system using fuzzy PID controller (an ex-
perimental test). In Proceedings of the second international conference on com-
puting technology and information management (ICCTIM) (pp. 99–104). IEEE.
doi:10.1109/ICCTIM.2015.7224600.
Alejo, D., Cobano, J. A., Heredia, G., & Ollero, A. (2016). A reactive method for col-
lision avoidance in industrial environments. Journal of Intelligent & Robotic Sys-
tems, 84(1), 745–758. doi:10.1007/s10846- 016- 0359- 7.
Antonelli, G. (2007). On the use of adaptive/integral actions for six-degrees-of-
freedom control of autonomous underwater vehicles. IEEE Journal of Oceanic En-
gineering, 32(2), 300–312. doi:10.1109/JOE.2007.893685.
Antonelli, G., Caccavale, F., Chiaverini, S., & Fusco, G. (2003). A novel adaptive con-
trol law for underwater vehicles. IEEE Transactions on Control Systems Technology,
11(2), 221–232. doi:10.1109/TCST.2003.809244.
Antonelli, G., Chiaverini, S., Sarkar, N., & West, M. (2001). Adaptive control of an au-
tonomous underwater vehicle: experimental results on ODIN. IEEE Transactions
on Control Systems Technology, 9(5), 756–765. doi:10.1109/87.944470.
Arokiasami, W. A., Vadakkepa, P., Tan, K. C., & Srinivasan, D. (2016). Vector di-
rected path generation and tracking for autonomous unmanned aerial/ ground
vehicles. In Proceedings of the ieee congress on evolutionary computation (CEC)
(pp. 1375–1381). doi:10.1109/CEC.2016.7743949.
Bansal, H. O., Sharma, R., & Shreeraman, P. R. (2012). PID Controller tuning tech-
niques: A Review. Journal of Control Engineering and Technology, 2(4), 168–176.
Barbalata, C., De Carolis, V., Dunnigan, M. W., Petillot, Y., & Lane, D. (2015).
An adaptive controller for autonomous underwater vehicles. In Proceedings of
the IEEE/RSJ international conference on intelligent robots and systems (IROS)
(pp. 1658–1663). Hamburg: IEEE. doi:10.1109/IROS.2015.7353590.
Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M., & Lee, M. (2009). Natural actor-
critic algorithms. Automatica, 45(11), 2471–2482. doi:10.1016/j.automatica.2009.
07.008.
Buçoniu, L., Babuska, R., Schutter, B. D., & Ernst, D. (2010). Reinforcement learning
and dynamic programming using function approximators (1st). CRC Press.
Busoniu, L., de Bruin, T., Toli, D., Kober, J., & Palunko, I. (2018). Reinforcement learn-
ing for control: Performance, stability, and deep approximators. Annual Reviews
in Control, 46, 8–28. doi:10.1016/j.arcontrol.2018.09.005.
Calvo-Rolle, J. L., Casteleiro-Roca, J. L., Quintián, H., & Del Carmen Meizoso-
Lopez, M. (2013). A hybrid intelligent system for PID controller using in a steel
rolling process. Expert Systems with Applications, 40(13), 5188–5196. doi:10.1016/
j.eswa.2013.03.013.
Carlucho, I., De Paula, M., Villar, S. A., & Acosta, G. G. (2017). Incremental Q-learning
strategy for adaptive PID control of mobile robots. Expert Systems with Applica-
tions, 80, 183–199. doi:10.1016/J.ESWA.2017.03.002.
Chang, H., & Jin, T. (2013). Adaptive tracking controller based on the PID for mo-
bile robot path tracking. In J. Lee, M. C. Lee, H. Liu, & J.-H. Ryu (Eds.), Intelligent
robotics and applications (pp. 540–549). Berlin, Heidelberg: Springer Berlin Hei-
delberg.
Chen, S., & Billings, S. A. (1989). Representations of non-linear systems: the NAR-
MAX model. International Journal of Control, 49(3), 1013–1032. doi:10.1080/
00207178908559683.
Cohen, G., & Coon, G. (1953). Theoretical consideration of retarded control. Transac-
tions of ASME, 75(1), 827–834.
Cohen, K., Bossert, D. E., & Col, L. (2002). PID and fuzzy logic pitch attitude hold
systems for a fighter jet AIAA-2002-4646 PID and fuzzy logic pitch attitude hold
systems for a fighter jet. In Proceedings of the AIAA guidance, navigation, and
control conference and exhibit. doi:10.2514/6.2002-4646. Monterey, California
Dong, E., Guo, S., Lin, X., Li, X., & Wang, Y. (2012). A neural network-based self-
tuning PID controller of an autonomous underwater vehicle. In Proceedings of
the IEEE international conference on mechatronics and automation (pp. 898–903).
IEEE. doi:10.1109/ICMA.2012.6283262.
Dyda, A. A., & Os’kin, D. A. (2004). Neural network control system for underwater
robots. IFAC Proceedings Volumes, 37(10), 427–432. doi:10.1016/S1474-6670(17)
31769-X.
Fatan, M., Sefidgari, B. L., & Barenji, A. V. (2013). An adaptive neuro PID for con-
trolling the altitude of quadcopter robot. In Proceedings of the 18th international
conference on methods & models in automation & robotics (MMAR) (pp. 662–665).
IEEE. doi:10.1109/MMAR.2013.6669989.
Fossen, T. I. (2002). Marine control systems : guidance, navigation and control of ships,
rigs and underwater vehicles. Marine Cybernetics.
Geder, J. D., Palmisano, J., Ramamurti, R., Sandberg, W. C., & Ratna, B. (2008). Fuzzy
logic PID based control design and performance for a pectoral fin propelled
unmanned underwater vehicle. In Proceedings of the international conference
on control, automation and systems (pp. 40–46). IEEE. doi:10.1109/ICCAS.2008.
4694526.
Gloye, A., Göktekin, C., Egorova, A., Tenchio, O., & Rojas, R. (2005). Learning to
Drive and Simulate Autonomous Mobile Robots. In Proceedings of the Robocup
(pp. 160–171). Springer, Berlin, Heidelberg. Lecture notes in computer science
Goumagias, N. D., Hristu-Varsakelis, D., & Assael, Y. M. (2018). Using deep Q-learning
to understand the tax evasion behavior of risk-averse firms. Expert Systems with
Applications, 101, 258–270. doi:10.1016/J.ESWA.2018.01.039.
Hägglund, T., & Åström, K. J. (1991). Industrial adaptive controllers based
on frequency response techniques. Automatica, 27(4), 599–609. doi:10.1016/
0 0 05-1098(91)90 052-4.
el Hakim, A., Hindersah, H., & Rijanto, E. (2013). Application of reinforcement learn-
ing on self-tuning PID controller for soccer robot multi-agent system. In Pro-
ceedings of the joint international conference on rural information & communica-
 I. Carlucho, M. De Paula and G.G. Acosta / Expert Systems With Applications 137 (2019) 292–307
tion technology and electric-vehicle technology (RICT & ICEV-T) (pp. 1–6). IEEE.
doi:10.1109/rICT-ICeVT.2013.6741546.
Hasselt, H. v. (2010). Double q-learning. In Proceedings of the 23rd international
conference on neural information processing systems - volume 2. In NIPS’10
(pp. 2613–2621). USA: Curran Associates Inc.
Haykin, S. (1998). Neural networks: A comprehensive foundation (2nd). Prentice Hall.
Hernández-Alvarado, R., García-Valdovinos, L. G., Salgado-Jiménez, T., Gómez-
Espinosa, A., & Fonseca-Navarro, F. (2016). Neural network-based self-Tuning PID
control for underwater vehicles. Sensors (Basel, Switzerland), 16(9). doi:10.3390/
s16091429.
Khodayari, M. H., & Balochian, S. (2015). Modeling and control of autonomous un-
derwater vehicle (AUV) in heading and depth attitude via self-adaptive fuzzy
PID controller. Journal of Marine Science and Technology (Japan), 20(3), 559–578.
doi:10.10 07/s0 0773- 015- 0312-7.
Kuantama, E., Vesselenyi, T., Dzitac, S., & Tarca, R. (2017). PID And fuzzy-PID control
model for quadcopter attitude with disturbance parameter. International Jour-
nal of Computers Communications & Control, 12(4), 519. doi:10.15837/ijccc.2017.4.
2962.
Kuo, B. C. (1995). Automatic control systems. Prentice Hall.
LeCun, Y., Bengio, Y., Hinton, G., Y. , L., Y. , B., & G. , H. (2015). Deep learning. Nature,
521(7553), 436–444. doi:10.1038/nature14539. arXiv: 1312.6184v5.
Li, D. L., He, Z. S., & Yin, Y. F. (2012). Trajectory tracking control of mo-
bile robot based on self-Adaptive PID combined with preview and fuzzy
logic. Advanced Materials Research, 590, 268–271. 10.4028/www.scientific.net/
AMR.590.268
Liu, L., Pan, F., & Xue, D. (2015). Variable-order fuzzy fractional PID controller. ISA
Transactions, 55, 227–233. doi:10.1016/j.isatra.2014.09.012.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. a., Veness, J., Bellemare, M. G.,
et al. (2015). Human-level control through deep reinforcement learning. Nature,
518(7540), 529–533. doi:10.1038/nature14236. arXiv: 1312.5602.
Monahan, G. E. (1982). State of the art—A survey of partially observable Markov
decision processes: theory, models, and algorithms. Management Science, 28(1),
1–16. doi:10.1287/mnsc.28.1.1.
MyungsooJun, & Safonov, M. (1999). Automatic PID tuning: an application of un-
falsified control. In Proceedings of the IEEE international symposium on computer
aided control system design (pp. 328–333). IEEE. doi:10.1109/CACSD.1999.808669.
Neumann, G., & Peters, J. R. (2008). Fitted Q-iteration by advantage weighted regres-
sion. In Advances in neural information processing systems 21 (pp. 1177–1184).
Ogata, K. (2010). Modern control engineering. Prentice-Hall.
Önder Efe, M. (2011). Neural network assisted computationally simple PI d control
of a quadrotor UAV. IEEE Transactions on Industrial Informatics, 7(2). doi:10.1109/
TII.2011.2123906.
Pan, C.-Z., Lai, X.-Z., Yang, S. X., & Wu, M. (2013). An efficient neural network ap-
proach to tracking control of an autonomous surface vehicle with unknown dy-
namics. Expert Systems with Applications, 40(5), 1629–1635. doi:10.1016/J.ESWA.
2012.09.008.
Pan, J., Wang, X., Cheng, Y., & Yu, Q. (2018). Multisource transfer double DQN based
on actor learning. IEEE Transactions on Neural Networks and Learning Systems,
29(6), 2227–2238. doi:10.1109/TNNLS.2018.2806087.
Pan, Y., Zaheer, M., White, A., Patterson, A., & White, M. (2018). Organizing experi-
ence: A deeper look at replay mechanisms for sample-based planning in con-
tinuous state domains. Technical Report. arXiv: 1806.04624v1.
Pessen, D. W. (1994). A new look at PID-controller tuning. Journal of Dynamic Sys-
tems, Measurement, and Control, 116(3), 553. doi:10.1115/1.2899252.
Petit, A., Paulo, C., Carlucho, I., Menna, B., & Paula, M. D. (2016). Prediction of the
hydrodynamic coefficients of an autonomous underwater vehicle. In Proceedings
of the 3rd IEEE/OES south american international symposium on oceanic engineer-
ing (SAISOE) (pp. 1–6). doi:10.1109/SAISOE.2016.7922474.
Åström, K. J., & Hägglund, T. (2004). Revisiting the Ziegler–Nichols step response
method for PID control. Journal of Process Control, 14(6), 635–650. doi:10.1016/j.
jprocont.20 04.01.0 02.
Åström, K. J., & Hägglund, T. (2006). Advanced PID control. ISA - The Instrumentation,
Systems and Automation Society.
Rossomando, F. G., & Soria, C. M. (2015). Identification and control of nonlinear dy-
namics of a mobile robot in discrete time using an adaptive technique based
on neural PID. Neural Computing and Applications, 26(5), 1179–1191. doi:10.1007/
s00521- 014- 1805- 8.
Rout, R., & Subudhi, B. (2017). Inverse optimal self-tuning PID control design for an
autonomous underwater vehicle. International Journal of Systems Science, 48(2),
367–375. doi:10.1080/00207721.2016.1186238.
307
Safonov, M., & Tung-Ching Tsao (1997). The unfalsified control concept and learning.
IEEE Transactions on Automatic Control, 42(6), 843–847. doi:10.1109/9.587340.
Salhi, K., & Alimi, A. M. (2013). Fuzzy-PID hybrid controller for mobile robot using
point cloud and low cost depth sensor. In Proceedings of the international con-
ference on individual and collective behaviors in robotics (ICBR) (pp. 92–97). IEEE.
doi:10.1109/ICBR.2013.6729280.
van Seijen, H., & Sutton, R. (2015). A deeper look at planning as learning from re-
play. In Proceedings of the 2nd multidisciplinary conference on reinforcement learn-
ing and decision making.
Shah, P., & Agashe, S. (2016). Review of fractional PID controller. Mechatronics, 38,
29–41. doi:10.1016/j.mechatronics.2016.06.005.
Sharma, R., Rana, K., & Kumar, V. (2014). Performance analysis of fractional order
fuzzy PID controllers applied to a robotic manipulator. Expert Systems with Ap-
plications, 41(9), 4274–4289. doi:10.1016/j.eswa.2013.12.030.
Sheikhlar, A., Fakharian, A., & Adhami-Mirhosseini, A. (2013). Fuzzy adaptive PI con-
trol of omni-directional mobile robot. In Proceedings of the 13th Iranian confer-
ence on fuzzy systems (IFSC) (pp. 1–4). IEEE. doi:10.1109/IFSC.2013.6675667.
Singh, A., Bisht, G., & Padhy, P. K. (2013). Neural network based adaptive non linear
PID controller for non-holonomic mobile robot. In Proceedings of the interna-
tional conference on control, automation, robotics and embedded systems (CARE)
(pp. 1–6). IEEE. doi:10.1109/CARE.2013.6733732.
Sood, V. (2011). Autonomous robot motion control using fuzzy PID controller. In
High performance architecture and grid computing (pp. 385–390). Springer Berlin
Heidelberg.
Sood, V., & Kumar, P. (2014). Adaptive fuzzy controller for controlling the dynamics
of robot motion. International Journal of Fuzzy Computation and Modelling, 1(1),
37. doi:10.1504/IJFCM.2014.064230.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning : An introduction. MIT
Press.
Takači, D., Takači, A., & Takači, A. (2014). On the operational solutions of fuzzy
fractional differential equations. Fractional Calculus and Applied Analysis, 17(4).
doi:10.2478/s13540- 014- 0216- y.
Tandan, N., & Swarnkar, K. K. (2015). PID controller optimization by soft computing
techniques–A review. International Journal of Hybrid Information Technology, 8(7),
357–362.
Timmer, S., & Riedmiller, M. (2007). Fitted Q iteration with CMACs. In Proceedings
of the IEEE international symposium on approximate dynamic programming and
reinforcement learning (pp. 1–8). IEEE. doi:10.1109/ADPRL.2007.368162.
Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with dou-
ble Q-Learning. In Proceedings of the 30th AAAI conference on artificial intelligence,
AAAI (pp. 2094–2100). arXiv: 1712.01275.
Wang, Q.-G., Fung, H.-W., & Zhang, Y. (1999). PID Tuning with exact gain and phase
margins. ISA Transactions, 38(3), 243–249. doi:10.1016/S0019- 0578(99)00020- 8.
Watkins, C. J., & Dayan, P. (1992). Technical note: Q-Learning. Machine Learning,
8(3/4), 279–292. doi:10.1023/A:1022676722315.
Wei, W., & Cohen, K. (2015). Development of a model based fuzzy-PID controller
for the AeroQuad cyclone quad-copter. In Proceedings of the AIAA infotech @
aerospace. Reston, Virginia: American Institute of Aeronautics and Astronautics.
doi:10.2514/6.2015-2029.
Ye, J. (2008). Adaptive control of nonlinear PID-based analog neural networks for
a nonholonomic mobile robot. Neurocomputing, 71(7), 1561–1565. doi:10.1016/j.
neucom.2007.04.014.
Zadeh, L. A. (1973). Outline of a new approach to the analysis of complex systems
and decision processes. IEEE Transactions on Systems, Man, and Cybernetics, SMC-
3(1), 28–44. doi:10.1109/TSMC.1973.5408575.
Zhang, S., & Sutton, R. S. (2017). A deeper look at experience replay. In Proceedings
of the NIPS deep reinforcement learning symposium.
Zhao, J., Yi, W., Peng, Y., & Peng, X. (2017). Design and simulation of a Self-adaptive
Fuzzy-PID Controller for an autonomous underwater vehicle (pp. 867–878). Cham:
Springer.
Zhuang, M., & Atherton, D. (1993). Automatic tuning of optimum PID controllers.
IEE Proceedings D Control Theory and Applications, 140(3), 216. doi:10.1049/ip-d.
1993.0030.
Ziegler, J. G., & Nichols, N. B. (1942). Optimum settings for automatic controllers.
Transactions of the ASME, 64, 759–768.
Ziegler, J. G., & Nichols, N. B. (1993). Optimum settings for automatic controllers.
Journal of Dynamic Systems, Measurement, and Control, 115(2B), 220. doi:10.1115/
1.2899060.
Zhang, Z., Zhiyuan, P., & Kochenderfer, M. J. (2017). Weighted double Q-learning. In
Proceedings of the twenty-sixth international joint conference on artificial intelli-
gence, IJCAI (pp. 3455–3461). doi:10.24963/ijcai.2017/483.