See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/340181129

Augmenting MPC schemes with active learning: Intuitive tuning and guaranteed
performance

Article · March 2020

DOI: 10.1109/LCSYS.2020.2983384

CITATIONS READS
0 66

3 authors:

Raffaele Soloperto Johannes Koehler

Universität Stuttgart Universität Stuttgart
17 PUBLICATIONS   76 CITATIONS    34 PUBLICATIONS   108 CITATIONS   

SEE PROFILE SEE PROFILE

Frank Allgöwer
Universität Stuttgart
700 PUBLICATIONS   13,654 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Dynamic operation with nonlinear model predictive control View project

Alternative Scanning Methods for High-speed Atomic Force Microscopy View project

All content following this page was uploaded by Raffaele Soloperto on 01 April 2020.

The user has requested enhancement of the downloaded file.

 Augmenting MPC schemes with active learning:
Intuitive tuning and guaranteed performance
Raffaele Soloperto, Johannes Köhler, Frank Allgöwer
Abstract—A framework to augment an existing model predic-
tive control (MPC) design/implementation with active learning is
proposed. Active learning is achieved by employing a user-defined
learning cost function (e.g. enforcing persistence of excitation
or using exploration terms from reinforcement learning), with
the aim to improve model knowledge and reduce uncertainty
through model adaptation. The framework is applicable to a
general class of nonlinear MPC design procedures and ensures
desired performance bounds for the resulting closed-loop, which
can be intuitively tuned compared to the initial MPC design. The
performance bounds are obtained by coupling the active learning
objective with performance bounds of the primary MPC, using
tools from multi-objective MPC and average constraints from
economic MPC. The overall framework can be easily imple-
mented1 and it is intuitive to tune. The resulting computational
demand typically is comparable to the original MPC scheme. We
demonstrate the practicality of the proposed framework using a
numerical example involving a nonlinear uncertain model, and
active learning.
I. I NTRODUCTION
Model predictive control (MPC) [1] is an optimization based
control method that can deal with general nonlinear systems
subject to hard state and input constraints. The performance
of MPC schemes depends largely on the accuracy of the
prediction model. Thus, online model refinement in MPC,
e.g., using machine learning approaches [2], [3], or (robust)
adaptive methods [4], [5], [6], [7], is an active research topic.
Passive learning, i.e. without any active system excitation, is
often not sufficient to achieve satisfactory performance [8]. In
this paper, we present a framework to augment an existing
MPC implementation with a user defined active learning cost,
while retaining the original properties regarding closed-loop
constraint satisfaction and performance.
Related works: In Reinforcement Learning approaches [9],
a random excitation input is applied in order to explore un-
known dynamics. Since this might lead to constraint violation,
in [10] a predictive safety filter is proposed with the aim to
ensure safety, without considering performance.
Optimizing for performance while ensuring safety is the
goal of many applications. The ideal performance under model
uncertainty is obtained using dual control [11], which is in
This work was supported by the German Research Foundation under Grants
GRK 2198/1 - 277536708, AL 316/12-2, and MU 3929/1-2 - 279734922. The
authors thank the International Max Planck Research School for Intelligent
Systems (IMPRS-IS) for supporting Raffaele Soloperto.
Raffaele Soloperto, Johannes Köhler, and Frank Allgöwer are with
the ”Institute for Systems Theory and Automatic Control”, Univer-
sity of Stuttgart, 70550 Stuttgart, Germany. (email:raffaele.soloperto, jo-
hannes.koehler, frank.allgower@ist.uni-stuttgart.de).
1 An exemplary implementation can be found at: https://www.ist.
uni-stuttgart.de/dokumente/public/Soloperto 20.m
general computationally intractable. As a result, dual MPC
schemes [12], [13], [14], [15] try to approximately solve the
dual control problem. However, these approaches (i) fail to
provide theoretical guarantees regarding closed-loop safety
and/or performance [13], [14], [15], or (ii) are limited to
linear system dynamics [12], [15]. To overcome the drawbacks
associated with existing dual MPC schemes, we focus on
the problem of augmenting an existing MPC implementa-
tion to enhance learning/system excitation, while preserving
the safety and performance guarantees of the original MPC
scheme.
Contributions: We present a framework to augment an
existing nonlinear MPC scheme with a learning cost function
to incentivize active learning. The framework is applicable
to a wide variety of nonlinear MPC designs: general eco-
nomic MPC schemes [16], MPC schemes without terminal
constraints [17], robust MPC schemes [18], and combinations
thereof [19], [20], [21]. For this general setup, we derive
closed-loop performance guarantees depending on intuitively
tunable constants, which allow for a simple trade-off between
potential performance relaxation and freedom to explore the
system. These performance bounds are enabled using tech-
niques from averaged constraint MPC [22], which extend
existing approaches for multi-objective MPC [23], [24].
In addition, the proposed framework poses no restrictions
on the predicted learning cost function, allowing for the
incorporation of most existing formulations, e.g., persistence
of excitation cost [5], covariance predictions using recursive
least square (RLS) [12], variance maximization/exploration
in Gaussian processes (GPs) [25], and general reinforcement
learning based functions [10].
To summarize, we propose a comprehensive framework to
extend existing MPC implementations in order to incorporate
a learning cost function. The desired performance and learning
can be easily tuned with intuitive scalar constants. In addition,
the computational demand of the proposed learning augmented
MPC framework is typically only moderately increased com-
pared to the baseline MPC implementation. We showcase the
practicality of the proposed framework with a nonlinear uncer-
tain system, involving reference tracking and active learning.
Outline: Section II presents the basic setup and discusses
preliminaries regarding the existing MPC framework. The
proposed framework is presented in Section III, including
a theoretical derivation of performance bounds. Section IV
demonstrates the results with a numerical example and Sec-
tion V concludes the paper.
 II. P ROBLEM F ORMULATION
A. Problem setup
We consider a nonlinear, time-invariant, perturbed discrete-
time system
xt+1 = fw (xt , ut , dt ),
with state x ∈ Rn , control input u ∈ Rm , disturbance d ∈
D ⊂ Rq , time t ∈ N, and perturbed (unknown) system fw .
We impose point-wise in time state and input constraints
(xt , ut ) ∈ Z, ∀t ≥ 0,
where Z ⊆ Rn+m is a constraint set.
We consider the case where the model can be learned
online to ensure an improvement in the performance and
therefore to reduce conservatism. This gives rise to a time-
varying nominal model ft (x, u) and some uncertainty bounds
Wt (x, u), satisfying the following condition for all t ≥ 0
fw (x, u, d) − ft (x, u) ∈ Wt (x, u).
Such a condition is standard in robust MPC [26, Ass. 1],
robust adaptive MPC using set-membership estimation [4],
[5], [6], [7] and is enforced in recent non-parametric machine
learning approaches [27]. To ensure constraint satisfaction
despite the uncertainty Wt , tube-based robust MPC approaches
predict sets Xk|t ⊂ Rn that contain the uncertain predicted
trajectories. For linear models, the sets Xk|t can often be
obtained using vertex enumeration [6], while computationally
more efficient approaches parametrize Xk|t with a nominal
trajectory xk|t and some scaled set, e.g. using fixed robust
positive invariant (RPI) sets [18], [21], polytopic tubes [4],
[5] or incremental Lyapunov functions [7], [10].
To reduce the conservatism of this tube propagation, a
parametrized control law πk|t : Rn → Rm [6], [7] is typically
included in the prediction, e.g., πk|t (x) = Kx + uk|t with
a nominal input uk|t and a stabilizing feedback K ∈ Rm×n ,
compare [4], [5]. In the special case of nominal MPC (i.e., with
no uncertainty), we have ft (x, u) := fw (x, u, d), Wt (x, u) :=
{0} and the predictions reduce to standard state and input
trajectories πk|t = uk|t , Xk|t = xk|t .
B. Existing MPC framework
In this paper, we consider the case where an existing
MPC scheme is already implemented. Such a scheme ensures
constraint satisfaction and an acceptable level of closed-
loop performance, according to some user-defined criteria. In
Section III, we show how this scheme can be augmented in
order to incentivize learning of system (1), and thus reduce
the uncertainty Wt .
Primary cost function: Based on the definition of the sets
X·|t and the parametrized control law π·|t , we define a general
cost function as follows
N
X −1
E(X·|t , π·|t ) := `(Xk|t , πk|t ) + Vf (XN |t ),
k=0
where N ∈ N is the MPC prediction horizon, ` is the bounded
stage cost, and Vf is the bounded terminal cost. By `min ∈ R
we denote the lower bound on ` satisfying
`min ≤ `(X, π), (4)
for all X, π satisfying (x, π(x)) ∈ Z for all x ∈ X.
(1) In most MPC implementations, the cost function (3) is
simply based on some nominal trajectory `(xk|t , uk|t ) [4], [5],
[6], [7], [20]. In a robust tube framework, one can also use
a worst-case cost satisfying `(X, π) ≥ `(x, π(x)), ∀x ∈ X,
which provides stronger performance guarantees [18], [21].
(2) Existing MPC scheme: Given the nominal model ft (x, u),
the description of the uncertainty Wt , the primary cost func-
tion E, and possibly a suitable terminal set Xf ⊂ Rn , the
resulting optimization problem is given by:
Et∗ := min E(X·|t , π·|t ) (5a)
X·|t ,π·|t
s.t. ft (xk|t , πk|t (xk|t )) + wk|t ∈ Xk+1|t , (5b)
(xk|t , πk|t (xk|t )) ∈ Z, (5c)
∀wk|t ∈ Wt (xk|t , πk|t (xk|t )), ∀xk|t ∈ Xk|t , (5d)
{xt } ∈ X0|t , XN |t ⊆ Xf , (5e)
k = 0, . . . , N − 1.
The solution of (5) are the optimal cost Et∗ , sets X∗·|t
∗
and control laws π·|t . A discussion on various tractable for-
mulations for the sets X·|t , control laws π·|t and the tube
propagation (5b) can be found in [7], [26, Rk. 1]. The resulting
closed-loop system is given by
xt+1 = fw (xt , ut , dt )∈ X∗1|t , ∗
ut = π0|t (xt ). (6)
The following assumption characterizes the fact that the
existing MPC scheme (5) is properly designed, i.e., ensures
recursively feasibility and provides a suitable performance
bound on the primary objective `.
Assumption 1. Problem (5) is feasible for all t ≥ 0 and
the constraints (2) are satisfied for the resulting closed-loop
system (6). Furthermore, there exist constants c ≥ `min ,
α ∈ (0, 1], such that for any trajectory (X·|t , π·|t ) satisfying
the constraints in (5), and for any xt+1 ∈ X1|t , there exists
a candidate trajectory (X·|t+1 , π·|t+1 ), which satisfies the
constraints in (5) and satisfies the following performance
bound
E(X·|t+1 , π·|t+1 ) ≤ E(X·|t , π·|t ) − α`(X0|t , π0|t ) + c. (7)
The following remark summarizes different MPC design
procedures that satisfy Assumption 1 and the corresponding
constants α, c.
Remark 1. Recursive feasibility and constraint satisfaction
are standard properties of MPC schemes, which can be
guaranteed with suitable conditions on the tube propagation
(X·|t , (5b)), the model update (ft , Wt ) and the terminal set Xf ,
compare [7, Thm. 1] for general conditions. In the following,
we discuss how α and c vary according to the considered
scenario.
(3) • MPC with terminal ingredients (Vf , Xf ): The standard
MPC design [1] uses a (robust) positive invariant terminal set
Xf combined with a control Lyapunov function Vf , which is
used as a terminal cost. In the nominal case, such a design
directly implies satisfaction of Assumption 1 with α = 1 and
c := `(xs , us ) = min `(x, u)
(x,u)∈Z
s.t. f (x, u) = x,
compare [16]. Furthermore, in case of tracking MPC (`
positive definite), (8) holds with c = `min = 0.
In the robust case with a nominal stage cost `(xk|t , πk|t ), the
value of c increases by a factor depending on the magnitude of
the model mismatch, compare e.g. [5, Thm. 7], [26, Thm. 1]. In
case a worst-case stage cost ` is used, we have c = `(Ω, πf ),
with some RPI set Ω, compare [18].
• MPC without terminal ingredients2 (Vf := 0, Xf :=
n
R ): In (nominal) tracking MPC without terminal ingredients,
inequality (7) is satisfied with some sub-optimality index α ∈
(0, 1] and c = 0 for a sufficiently large prediction horizon N
[17]. In the economic context, the bound (7) holds with α = 1,
c = `(xs , us ) + N with some N > 0 [19]. In the robust case,
similar bounds hold with a larger constant c, compare [20],
[21].
• instability.
These issues are mainly due to finding an appropriate trade-
off between the two costs is, in general, not intuitive, and
(8) therefore tuning is not only expensive in terms of time and
resources, but might also generate unsafe behaviours during
experiments. Note that even an arbitrarily small λ allows the
learning cost to have dominant effects when the primary cost
is close to zero.
To avoid the above mentioned shortcomings resulting from
such a naive augmentation, we propose a framework for learn-
ing MPC that is easy to implement, allows for intuitive tuning,
and gives guaranteed performance bounds on the primary cost.
B. Proposed MPC scheme
In the following, we introduce the proposed MPC frame-
work which is based on Problem (5) augmented with active
learning. For the latter, we consider a general learning cost
function Ht (X·|t , π·|t ). Frequent choices for such a function
are discussed in Section III-D. The proposed MPC scheme is
defined as follows
min Ht (X·|t , π·|t ) (9a)
X·|t ,π·|t ,∆E
t

III. P ROPOSED MPC F RAMEWORK s.t. E(X·|t , π·|t ) = EtB + ∆E

t , (9b)
This section contains the proposed learning MPC framework ∆E
t ≤ β̄ max{Et+ , 0} + γ̄ + Yt−1 , (9c)
for nonlinear uncertain systems. A motivating scenario is ∆E ≤ β max max{Et+ , 0} + γ max , (9d)
t
described in Section III-A, a general description of the learning
ft (xk|t , πk|t (xk|t )) + wk|t ∈ Xk+1|t , (9e)
cost function together with the proposed MPC framework is
presented in Section III-B. In Section III-C, we show the (xk|t , πk|t (xk|t )) ∈ Z, (9f)
main theorem, an interpretation of the theoretical result, and ∀wk|t ∈ Wt (xk|t , πk|t (xk|t )), ∀xk|t ∈ Xk|t , (9g)
the related proof. Section III-D discusses different existing {xt } ∈ X0|t , XN |t ⊆ Xf , (9h)
learning formulations that can be incorporated in the proposed
k = 0, . . . , N − 1.
MPC framework.
The solution of Problem (9) are the optimal sets X̂·|t , control
A. Motivating scenario laws π̂·|t , the relaxation factor ∆ ˆEt , and the primary cost
Consider the case where system (1) is controlled with an function Êt := E(X̂·|t , π̂·|t ). The resulting closed-loop system
MPC scheme defined by Problem (5). Moreover, assume that is given by
the MPC scheme (5) provides an acceptable, but unfortunately
xt+1 = fw (xt , ut , dt ) ∈ X̂1|t , ut = π̂0|t (xt ). (10)
not ideal, level of performance, according to some user-defined
criteria. In particular, due to the presence of uncertainty, Compared to Problem (5), the proposed scheme (9) mini-
a further improvement of the performance would require a mizes the learning cost Ht , while the primary cost E only
significant effort from the user, if at all possible. appears in the additional constraints (9b)–(9d), which are
In such a setup, a simple and appealing way to reduce explained in the following.
the uncertainty, and therefore to increase performance, can be The value EtB used in (9b) represents the desired primary
obtained by augumenting the primary cost E with a learning cost, satisfying Et∗ ≤ EtB by definition. In the following, we
cost Ht , which has the goal to induce excitation in (1). In this discuss three different design choices for EtB , which all satisfy
case, the cost E in Problem (9) is replace by E + λHt , where B
Et+1 ≤ Êt − α`(X̂0|t , π̂0|t ) + c, with α, c ∈ R from bound (7)
λ ≥ 0 is a tuning parameter, compare e.g. [5], [12]. However, in Assumption 1.
combining two potentially conflicting cost functions, might 1) A simple choice for the desired cost EtB is given by
lead to one or more of the following shortcomings: EtB := Êt−1 −α`(X̂0|t−1 , π̂0|t−1 )+c, which requires the
• arbitrary deterioration of the primary performance, explicit knowledge of the constants α ∈ (0, 1] and c ∈ R
• permanent steady-state error, for the implementation. Remark 1 and the references
• generation of undesired oscillatory behaviours, therein discuss in detail how valid constants can be
2 In MPC without terminal constraints, the performance bound (7) and even
computed offline for different design procedures.
recursive feasibility might only hold if E(X·|t , π·|t ) ≤ C holds with some 2) Since offline computed constants α, c may be con-
constant C, which then needs to be recursively ensured. servative, an intuitive alternative is to use the feasible
 candidate solution (X·|t , π·|t ) from Assumption 1, which
satisfies
(7)
EtB := E(X·|t , π·|t ) ≤ Êt−1 − α`(X̂0|t−1 , π̂0|t−1 ) + c.
This approach is particularly appealing in case terminal
ingredients are employed (due to the simple construction
of the candidate solution).
3) The desired performance bound EtB can also be defined
by additionally solving the original MPC scheme (5)
using:
(7)
EtB := Et∗ ≤ Êt−1 − α`(X̂0|t−1 , π̂0|t−1 ) + c.
The variable ∆E t in (9b), represents how much we allow the
primary cost function Êt to deviate from the desired bound
EtB . The proposed scheme will allow for a performance degra-
dation/relaxation ∆E t if this helps to minimize the learning cost
Ht . However, as stated above, deteriorating the performance
might lead to unsafe or undesired behaviour of the closed-loop
system (10). For this reason, the term ∆E t is upper-bounded
using (9c) and (9d).
In particular, (9c) can be seen as an average constraint with
storage Yt , as employed in economic MPC schemes, compare
[22]. The parameter γ̄ ≥ 0 represents the average absolute
performance relaxation we allow for active learning (i.e.
minimization of Ht ), while β̄ ∈ [0, 1) represents a percentage
of the relative relaxation in the performance requirements. The
variable Et+ , used in (9c)–(9d), is defined as follows
1),2),3)
Et+ := Êt−1 − EtB ≥ α`(X̂0|t−1 , π̂0|t−1 ) − c.
The operation max{Et+ , 0} in (9c)-(9d) is used to ensure that
we allow for a performance relaxation only if it is possible
to guarantee a potential decrease of the primary cost function,
i.e., Et+ ≥ 0. Finally, the storage Yt is updated as follows
Yt := Yt−1 + β̄ max{Et+ , 0} + γ̄ − ∆E
t ,
with Y0 ≥ 0. The update rule in (12) is needed to enforce av-
erage bounds on ∆E t , similar to average constraint MPC [22].
The parameters β max ≥ 0 and γ max ≥ 0 in (9d) have an
analogous meaning to β̄ and γ̄, respectively, but are used to
impose hard (non-averaged) bounds on ∆E t .
Similar tools are used in Multi-objective MPC to obtain
desired bounds [23], [24], which are less flexible than the
proposed framework due to the lack of tuning variables γ̄, β̄,
γ max , β max . Thus, the proposed framework can also be applied
to general multi-objective MPC, where Ht is not necessarily
related to active learning.
C. Theoretical Analysis and Interpretation
Theorem 1. Let Assumption 1 hold. Then Problem (9) is
feasible for all t ≥ 0 and the following performance bound
holds for all T ≥ 0:
ÊT − Ê0 + YT − Y0
T
X −1
≤T (γ̄ + c − β̄α`min ) − ᾱ`(X̂0|t , π̂0|t ),
t=0
where ᾱ := (1 − β̄)α ≥ 0. Furthermore, if β max ∈ [0, 1), then
the following performance bound holds for all T ≥ 0
ÊT − Ê0 (14)
T
X −1
≤T (γ max + c − β max α`min ) − αmax `(X̂0|t , π̂0|t ),
t=0
max max
where α := (1 − β )α > 0.
The proof of Theorem 1 is detailed at the end of this section.
Interpretation of Theorem 1 and special cases: In the
following, we show how (13) and (14) can be interpreted in
economic and tracking MPC schemes.
In an economic MPC setup (α = 1), where infinite average
performance are typically of interest, by choosing β̄ = 0, we
get the following bound
PT −1
`(X̂0|t , π̂0|t )
lim sup t=0 ≤ γ̄ + c. (15)
T →∞ T
Hence, one can choose the parameter γ̄ to easily tune the
primary performance, which corresponds to an additional
average suboptimality in the cost. Moreover, by setting γ̄ = 0,
the bound (15) is equivalent to the average performance bound
usually obtained based on the economic MPC scheme (5) with
Assumption 1, compare [16], [18], [21]. In addition, one can
PT a time-varying relaxation γ̄t ≥ 0, satisfying
also consider
limT →∞ t=0 γ̄t /T ≤ γ̄, which leads to the same bound in
(15), while additionally giving the freedom to distribute active
learning over time.
On the other side, in a tracking MPC case, we have c ≥ 0,
(11)
Vf (X) ≥ 0, and `(X, π) ≥ `min = 0 for all X, π satisfying
(x, π(x)) ∈ Z, ∀x ∈ X. In such a case, the transient behaviour
and convergence are crucial. Based on (13), the following non-
averaged performance bound holds
T
X −1
ᾱ`(X̂0|t , π̂0|t ) ≤ Ê0 + Y0 + T (c + γ̄), ∀T ≥ 0. (16)
(12)
t=0
Note that for c = γ = Y0 = 0, the bound (16) reflects
the usual suboptimality estimate in MPC without terminal
constraints [17], with the relaxed suboptimality index α ≤ α.
Moreover, based on (14), the following holds
Êt+1 − Êt ≤ −αmax `(X̂0|t , π̂0|t ) + c + γ max . (17)
Hence, if `(X, π) ≥ αl (kxk), Ê0 ≤ αu (kxk) + p, with some
αl , αu ∈ K∞ 3 , p ≥ 0, then (17) implies practical asymptotic
stability (w.r.t c + γ max , p) of the closed-loop system with a
practical Lyapunov function Êt .
Proof of Theorem 1
Recursive feasibility: We show feasibility of Problem (9)
for all t ≥ 0, using the recursive feasibility conditions
for Problem (5) in Assumption 1. Firstly, every trajectory
(X·|t , π·|t ) satisfying the constraints in Problem (9) trivially
satisfies the constraints in Problem (5). In the following,
we show that the candidate trajectory (X·|t+1 , π·|t+1 ) from
(13) 3 By K
∞ we denote the class of functions α : R≥0 → R≥0 , which are
continuous, strictly increasing, unbounded and satisfy α(0) = 0.
Assumption 1 satisfies the constraints in Problem (9). First, distribution θ ∈ N (θ̄t , Σt ), where the covariance is predicted
B
the different bounds Et+1 are such that the candidate solution using
(9b)
satisfies ∆E B
t+1 = E(X·|t+1 , π·|t+1 ) − Et+1 ≤ 0. Σ−1 −1 −2
G(xk|t , uk|t )G> (xk|t , uk|t ),
k|t = Σk−1|t + σ σ > 0.
Given Y0 ≥ 0, one can recursively show
(9c)
Correspondingly, the learning cost Ht can be chosen to
(12)
Yt = Yt−1 + β̄ max{Et+ , 0} + γ̄ − ∆E
t ≥ 0, ∀t ≥ 1. minimize tr(Σk|t ) or the expected cost, as done in [12].
Persistence of excitation (PE) can be incentivised by choos-
This implies that ∆E t+1 ≤ 0 satisfies constraint (9c). Satis- ing Ht = −β, and including
faction of (9d) is trivially ensured since β max , γ max ≥ 0 by N −1
construction.
X
G(xk|t , πk|t (xk|t ))G> (xk|t , πk|t (xk|t ))  βI, ∀xk|t ∈ Xk|t .
Satisfaction of (13): Let us consider the following k=0

B
Êt+1 − Êt = Et+1
(9b)
+ ∆E + E (11) as an additional constraint in Problem (9), which allows for
t+1 − Êt = −Et+1 + ∆t+1
(12)
convex relaxations in the linear polytopic case, compare [5].
+ +
= − Et+1 + Yt + β̄ max{Et+1 , 0} + γ̄ − Yt+1 . (18) Similar to [10], the learning cost can also be defined based
+ on some external exciting input uL t (e.g. resulting from a
If Et+1 ≥ 0, we have
Reinforcement Learning algorithm) using Ht := kut − uL 2
t k .
(18) +
Êt+1 − Êt = −(1 − β̄)Et+1 + Yt + γ̄ − Yt+1 . (19) Compared to a predictive safety filter [10], the proposed
approach also provides suitable bounds on the transient and
+
Similarly, if Et+1 < 0, the following holds average performance, which is crucial in many applications.
+(18) Moreover, one can define Ht := E(x̃·|t , π·|t ), where x̃ is
Êt+1 − Êt = −Et+1 + Yt + γ̄ − Yt+1 . (20) based on a different online adapted/learned model fˆt , e.g.
Let us now define the set τ as follows using least-mean square parameter estimation [4], [7] or policy
gradients from reinforcement learning [3].
τ (T ) := {t ∈ N[0,T ] |Et+ < 0}, (21)
where N[0,T ] is the set of all the integers between 0 and T . IV. N UMERICAL E XAMPLE
Combining both the cases, we have for all T ≥ 0 In the following example, we illustrate how the tuning
variables β̄ and γ̄ can be used to intuitively tune the primary
ÊT − Ê0 + YT − Y0
performance and active learning. We consider a mass-spring-
T −1
(19),(20),(21) X
+
X
+ damper system subject to additive and parametric uncertainty
= − (1 − β̄)Et+1 + T γ̄ − β̄Et+1
t=0 t∈τ (T ) mẍ1 = −k0 exp(−x1 )x1 − dẋ1 + u + w, (22)
T −1
(4),(11)
≤ −
X
+
(1 − β̄)Et+1 + T γ̄ + T β̄c − T β̄α`min with mass m = 1, uncertain damping constant d = 1, spring
t=0
constant k0 = 0.33, disturbance w, |w| ≤ 0.01 and, constraint
T −1 set Z = [−0.1, 1.1] × [−5, 5] × [−5, 5]. The model (22) is
(11) X
≤ − (1 − β̄)α`(X̂0|t , π̂0|t ) + T (γ̄ + c − β̄α`min ). discretized with a Euler discretization with a sampling time
t=0 of Ts = 0.1s. At time t = 0, we start with a nominal
dˆ0 = 1.1 which is then learned through a Least Mean Square
Similarly, condition (14) can be shown using an analogous
(LMS) algorithm. We make use of the robust MPC approach
reasoning to the proof of (13), by using (9d) in (18) instead
proposed in [26], and we employ a standard quadratic track-
of (12) to bound ∆Et . Satisfaction of (16) can be proved based
ing stage cost, with weight matrices Q = diag(10, 1), and
on (13) and on the non-negativity of c, `, and YT .
R = 1. The variable EtB is defined according to case 3),
i.e., EtB = Et∗ . The goal is to induce excitation in system
D. Learning formulations xref ref
(22) while steering it toward the desired PN reference t , ut .
−1 L
A standard approach in learning MPC schemes is to aug- We consider the learning cost Ht = k=0 kuk|t − ut+k k2 ,
ment the primary cost function with a term that incentives where uL ref
t = ut + 3 sin(0.2t) is an external probing signal
learning [5], [12]. Any learning cost Ht can be directly incor- that excites system (22), similar to [10]. In Fig. 1, we show
porated in the proposed framework, with the main difference four different scenarios. In particular, in all four scenarios, we
that we guarantee suitable performance bounds. The different set Y0 = β max = 0, and γ max = ∞, so that we focus on how
learning cost functions Ht typically depend on the considered the variables β̄ and γ̄ influence the closed-loop behaviour.
model description (ft ,Wt ) and the corresponding update rule. First, we consider the two extremum cases, i.e., the case of
In particular, if the model (ft ,Wt ) is obtained through a pure tracking (passive learning) and of pure (active) learning.
Gaussian Process (GP),PNthen similar to [25], the learning cost In the case of pure tracking (β̄ = 0, γ̄ = 0), the solution
−1
Ht can be chosen as k −Σt (Xk|t , πk|t ) to seek operations of Problem (9) is equivalent to the one from Problem (5).
with maximum uncertainty/covariance Σt . In the case of pure learning (β̄ = 0, γ̄ = ∞), the proposed
In case of nonlinear models linear in the uncertain parame- MPC scheme reduces to the safety filter shown in [10], i.e.,
ters, i.e., a term G(x, u)θ appearing in the model, a Recursive maximal learning while guaranteeing constraint satisfaction.
Least Square (RLS) filter can be used to obtain a Gaussian In this case, we see that the external input uL t generates high

1
0.8
0.6
0.4
0.2
0 vol. 2, pp. 253–279, 2019.
0 5 10 15 20
Fig. 1: Comparison of closed-loop trajectories. The value e20
indicates the accuracy of the estimated dˆ20 at time 20s, and
ˆ
is defined as et := |dtd−d| . Note that e0 = 10%.
oscillations in closed-loop, leading to a high tracking error.
Such a behaviour can be considered as safe in the sense that
the constraints are satisfied, but due to the large tracking error
it may not be desirable in practical applications.
In order to generate the desired trade-off between learning
and tracking, we consider the case where β̄ = 0.8 (80%) and
γ̄ = 0, where we see that the system shows some probing
only during the transient phase, and then it converges to the
desired reference point, as discussed in Section III-B. In the
case where β̄ = 0 and γ̄ = 5.0, we see that the controller
induces a constantly bounded amount of active probing, which
is mainly visible when the system is close to the steady-state.
This implies a negligible relaxation on the convergence rate,
and a visible probing when the system is close to the desired
steady state. To conclude, the results shown in Fig. 1 confirm
the intuitive meaning of the parameters β̄ and γ̄, how they
influence the closed-loop and therefore the accuracy of the
ˆ economic model predictive control without terminal conditions,” in Proc.
estimated parameter d.
V. C ONCLUSION
We proposed a framework for enhancing an existing MPC
design with active learning to improve model quality and
thus performance. We proved suitable transient and average
performance bounds on the resulting MPC scheme with in-
tuitively tunable constants. The practicality of the proposed
approach is demonstrated using an example involving tracking
and active learning. Designing active learning cost functions
that guarantee closed-loop reduction of uncertainty is part of
future work.
R EFERENCES
[1] J. B. Rawlings and D. Q. Mayne, Model predictive control: Theory and
design. Nob Hill Pub., 2009.
[2] L. Hewing, J. Kabzan, and M. N. Zeilinger, “Cautious model predic-
tive control using gaussian process regression,” IEEE Transactions on
Control Systems Technology, 2019.
[3] M. Zanon and S. Gros, “Safe reinforcement learning using robust MPC,”
arXiv preprint arXiv:1906.04005, 2019.
[4] M. Lorenzen, M. Cannon, and F. Allgöwer, “Robust MPC with recursive
model update,” Automatica, vol. 103, pp. 461–471, 2019.
[5] X. Lu, M. Cannon, and D. Koksal-Rivet, “Robust adaptive model pre-
dictive control: Performance and parameter estimation,” arXiv preprint
arXiv:1911.00865, 2019.
[6] M. Bujarbaruah, X. Zhang, M. Tanaskovic, and F. Borrelli, “Adaptive
MPC under time varying uncertainty: Robust and stochastic,” arXiv
preprint arXiv:1909.13473, 2019.
View publication stats
[7] J. Köhler, P. Kötting, R. Soloperto, F. Allgöwer, and M. A. Müller,
“A robust adaptive model predictive control framework for nonlinear
uncertain systems,” arXiv preprint arXiv:1911.02899, 2019.
[8] A. Mesbah, “Stochastic model predictive control with active uncertainty
learning: A survey on dual control,” Annual Reviews in Control, vol. 45,
pp. 107–117, 2018.
[9] B. Recht, “A tour of reinforcement learning: The view from continuous
control,” Annual Review of Control, Robotics, and Autonomous Systems,
[10] K. P. Wabersich and M. N. Zeilinger, “Safe exploration of nonlinear
dynamical systems: A predictive safety filter for reinforcement learning,”
arXiv preprint arXiv:1812.05506, 2018.
[11] A. A. Feldbaum, “Dual control theory. i,” Automation and Remote
Control, vol. 21, no. 9, pp. 1240–1249, 1960.
[12] R. Soloperto, J. Köhler, M. A. Müller, and F. Allgöwer, “Dual adaptive
MPC for output tracking of linear systems,” in Proc. 58th IEEE Conf.
Decision and Control (CDC), 2019, pp. 1377–1382.
[13] S. Thangavel, S. Lucia, R. Paulen, and S. Engell, “Dual robust nonlinear
model predictive control: A multi-stage approach,” Journal of Process
Control, vol. 72, pp. 39–51, 2018.
[14] E. Arcari, L. Hewing, and M. N. Zeilinger, “An approximate dynamic
programming approach for dual stochastic model predictive control,”
arXiv preprint arXiv:1911.03728, 2019.
[15] A. Iannelli, M. Khosravi, and R. S. Smith, “Structured exploration in
the finite horizon linear quadratic dual control problem,” arXiv preprint
arXiv:1910.14492, 2019.
[16] D. Angeli, R. Amrit, and J. B. Rawlings, “On average performance and
stability of economic model predictive control,” IEEE transactions on
automatic control, vol. 57, pp. 1615–1626, 2012.
[17] A. Boccia, L. Grüne, and K. Worthmann, “Stability and feasibility of
state constrained MPC without stabilizing terminal constraints,” Systems
& control letters, vol. 72, pp. 14–21, 2014.
[18] F. A. Bayer, M. A. Müller, and F. Allgöwer, “Min-max economic model
predictive control approaches with guaranteed performance,” in Proc.
55th IEEE Conf. Decision and Control (CDC), 2016, pp. 3210–3215.
[19] L. Grüne and M. Stieler, “Asymptotic stability and transient optimality
of economic MPC without terminal conditions,” Journal of Process
Control, vol. 24, no. 8, pp. 1187–1196, 2014.
[20] J. Köhler, M. A. Müller, and F. Allgöwer, “A novel constraint tighten-
ing approach for nonlinear robust model predictive control,” in Proc.
American Control Conf. (ACC). IEEE, 2018, pp. 728–734.
[21] L. Schwenkel, J. Köhler, M. A. Müller, and F. Allgöwer, “Robust
21st IFAC World Congress, 2020, accepted.
[22] M. A. Müller, D. Angeli, F. Allgöwer, R. Amrit, and J. B. Rawlings,
“Convergence in economic model predictive control with average con-
straints,” Automatica, vol. 50, pp. 3100–3111, 2014.
[23] D. He, L. Wang, and J. Sun, “On stability of multiobjective NMPC with
objective prioritization,” Automatica, vol. 57, pp. 189–198, 2015.
[24] L. Grüne and M. Stieler, “Multiobjective model predictive control for
stabilizing cost criteria,” Discrete & Continuous Dynamical Systems-B,
pp. 2823–2830, 2019.
[25] N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger, “Gaussian process
optimization in the bandit setting: No regret and experimental design,”
arXiv preprint arXiv:0912.3995, 2009.
[26] J. Köhler, R. Soloperto, M. A. Müller, and F. Allgöwer, “A computa-
tionally efficient robust model predictive control framework for uncertain
nonlinear systems,” IEEE Transactions on Automatic Control, 2020, to
appear.
[27] E. Maddalena and C. Jones, “Learning non-parametric models with
guarantees: A smooth lipschitz interpolation approach,” p. 6, 2019.
[Online]. Available: http://infoscience.epfl.ch/record/265764