Optimal Control Formulation

We are concerned with numerical solution procedures for optimal control

Optimal Control problems that comply with the following general formulation:
Lectures 19-20: Direct Solution Methods
ˆ 0 , tf ]nu and v ∈ IRnv that
Determine: u ∈ C[t
Benoı̂t Chachuat <benoit@mcmaster.ca> minimize: φ(x(tf ), v)
subject to: ẋ(t) = f (t, x(t), u(t), v); x(t0 ) = h(v)
i
ψ (x(tf ), v) ≤ 0
ψ e (x(tf ), v) = 0
Department of Chemical Engineering
κi (t, x(t), u(t), v) ≤ 0, t0 ≤ t ≤ tf
Spring 2009 κe (t, x(t), u(t), v) = 0, t0 ≤ t ≤ tf
u(t) ∈ [uL , uU ], v ∈ [vL , vU ]

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 1 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 2 / 32

Direct Methods of Optimal Control Control Parameterization

Principle 1 Subdivide the optimization horizon [t0 , tf ] into ns ≥ 1 control stages,
Discretize the control problem, then apply NLP techniques to the resulting t0 < t1 < t2 < · · · < tns = tf
finite-dimensional optimization problem
2 In each subinterval [tk−1 , tk ], approximate u(t) = U k (t, ω k )

Studied extensively over the last 30 years Lagrange Interpolating Polynomials (Degree M)
Proved successful for many complex applications M  
∆ (M) t−tk−1
X
Take advantage of the power of state-of-the-art NLP solvers uj (t) = Ujk (t, ω k ) = ω ki,j φi tk −tk−1 , tk−1 ≤ t ≤ tk
i =0 uj (t)
Can be applied to ODE, DAE and even PDAE models Ujk (t, ω k )

k
ωM,j
Three Main Variants 
 1, if M = 0 k
1 Direct simultaneous approach
 ω1,j
M

k
τ − τq
 Y
(M) ∆ ω0,j
◮ or orthogonal collocation, or full discretization φi (τ ) = , if M ≥ 1
τ i − τq

 q=0

t

2 Direct sequential approach q6=i tk−1 tk
◮ or single-shooting or control vector parameterization (CVP) τ
0 τ0 τ1 τM 1
3 Direct multiple-shooting approach with collocation points 0 ≤ τ0 ≤ τ1 < · · · < τM−1 < τM ≤ 1
Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 3 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 4 / 32
Control Parameterization Direct Simultaneous Method
piecewise constant piecewise linear without continuity
u(t) piecewise linear with continuity piecewise cubic with continuity Principle
Transcription into a finite-dimensional NLP through discretization of both
control and state variables

State Collocation
x(t) = X (t, ξ k ), tk−1 ≤ t ≤ tk , k = 1, . . . , ns

Lagrange Polynomial Representation (Degree N)

t
t0 t1 tk tns −1 tns = tf N  
∆ (N) t−tk−1
X
Xj (t, ξ kj ) = ξ ki,j φi , tk−1 ≤ t ≤ tk
Control bounds can be enforced by bounding the control coefficients tk −tk−1
i =0
ωk in each subinterval
Control continuity (or higher-order continuity) at stage times tk can (N)
with: φi (τq ) = δi ,q , q = 0, . . . , N
be enforced via linear inequality constraints between ω k and ωk+1
The choice of the collocation points τi has no effect on the solution
Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 5 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 6 / 32

Direct Simultaneous Method Direct Simultaneous Method (cont’d)

Principle
Transcription into a finite-dimensional NLP through discretization of both Original Optimal Control Problem
control and state variables
ˆ 0 , tf ]nu and v ∈ IRnv that
Determine: u ∈ C[t
State Collocation minimize: φ(x(tf ), v)
k
x(t) = X (t, ξ ), tk−1 ≤ t ≤ tk , k = 1, . . . , ns subject to: ẋ(t) = f (t, x(t), u(t), v); x(t0 ) = h(v)
i
ψ (x(tf ), v) ≤ 0
Monomial Basis Representation (Degree N)
ψ e (x(tf ), v) = 0
∆
N
X (N)

t−tk−1
 κi (t, x(t), u(t), v) ≤ 0, t0 ≤ t ≤ tf
Xj (t, ξ kj ) = ξ k0,j + (tk − tk−1 ) ξ ki,j Ωi , tk−1 ≤ t ≤ tk
tk −tk−1 κe (t, x(t), u(t), v) = 0, t0 ≤ t ≤ tf
i =1
u(t) ∈ [uL , uU ], v ∈ [vL , vU ]
(N)
with: Ωi (0) = 0
(N)
Ω̇i (τq ) = δi ,q , q = 1, . . . , N

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 6 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 7 / 32
Direct Simultaneous Method (cont’d) Direct Simultaneous Method (cont’d)
Class Exercise: Discretize the following optimal control problem into an
Fully-Discretized Optimization Problem NLP via the full discretization approach:
Z 1
1
Determine: ω 1 , . . . , ω ns ∈ IRnu M , ξ 1 , . . . , ξ ns ∈ IRnx N and v ∈ IRnv that min [u(t)]2 dt
u(t) 0 2
minimize: φ(X ns (tns , ξ ns ), v) s.t. ẋ(t) = u(t) − x(t); x(0) = 1
x(1) = 0
subject to: X kt (tk,q , ξ k ) = f (tk,q , X k (tk,q , ξ k ), U k (tk,q , ω k ), v), ∀k, q
X 1 (t0 , ξ 1 ) = h(v); X k+1 (tk , ξ k+1 ) = X k (tk , ξ k ), ∀k Consider a single control stage, ns = 1
ψ i
(X ns (t , ξ ns ), v) ≤0 Approximate the state and control profiles using affine functions,
ns
e M=N=1
ψ (X ns (tns , ξ ns ), v) = 0
κi (tk,q , X k (tk,q , ξ k ), U k (tk,q , ω k ), v) ≤ 0, ∀k, q
κe (t k,q , X k (t k,q , ξ k ), U k (t
k,q , ω k ), v) = 0, ∀k, q
L U
ξk ∈ [ξ , ξ ], ω∈ [ω , ω U ],
L v∈ [vL , vU ]
∆
with tk,q = tk−1 + τq (tk − tk−1 ), k = 1, . . . , ns , q = 1, . . . , N

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 8 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 9 / 32

Direct Simultaneous Method (cont’d) Direct Sequential Method

Principle
Pros and Cons
Transcription into a finite-dimensional NLP through discretization of the
very large-scale NLP problems in the variables
T T control variables only, while the ODEs are embedded in the NLP problem
pT = (ξ 1 , . . . , ξ ns T , ω 1 , . . . , ω ns T , vT )
The numbers of time stages and collocation points as well as the Original Optimal Control Problem
position of the collocation points must be chosen a priori
ˆ 0 , tf ]nu and v ∈ IRnv that
Determine: u ∈ C[t
The stage times can be optimized as part of the decision vector p too
Infeasible path method: minimize: φ(x(tf ), v)
◮ The ODEs are satisfied at the converged solution of the NLP only
subject to: ẋ(t) = f (t, x(t), u(t), v); x(t0 ) = h(v)
◮ But, this saves computational effort and allows unstable systems
i
ψ (x(tf ), v) ≤ 0
Path constraints are easily accommodated by enforcing inequality
constraint at the collocation points ψ e (x(tf ), v) = 0
κi (t, x(t), u(t), v) ≤ 0, t0 ≤ t ≤ tf
Class Exercise: Give sufficient conditions for existence of an optimal κe (t, x(t), u(t), v) = 0, t0 ≤ t ≤ tf
solution to the fully-discretized optimization problem u(t) ∈ [uL , uU ], v ∈ [vL , vU ]

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 10 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 11 / 32
Direct Sequential Method (cont’d) Direct Sequential Method (cont’d)
Class Exercise: Discretize the following optimal control problem into an
NLP via the sequential approach:
Partially-Discretized Optimization Problem Z 1
1
min [u(t)]2 dt
u(t) 0 2
Determine: ω1 , . . . , ω ns ∈ IRnu M and v ∈ IRnv that
s.t. ẋ(t) = u(t) − x(t); x(0) = 1
minimize: φ(x(tns ), v) x(1) = 0
subject to: ẋ(t) = f (t, x(t), U k (t, ω k ), v); x(t0 ) = h(v) Consider a single control stage, ns = 1
i
ψ (x(tns ), v) ≤ 0 Approximate the control profile using affine functions, M = 1
ψ e (x(tns ), v) = 0
κi (t, x(t), U k (t, ω k ), v) ≤ 0, tk−1 ≤ t ≤ tk , ∀k
κe (t, x(t), U k (t, ω k ), v) = 0, tk−1 ≤ t ≤ tk , ∀k
ω(t) ∈ [ω L , ω U ], v ∈ [vL , vU ]

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 12 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 13 / 32

Direct Sequential Method (cont’d) Direct Sequential Method (cont’d)

Issue 2: How to handle path constraints?

Gradients Cost &
Constraints

Reformulation as integral constraints:

ns Z tk  2
Sensitivity Variables OR Adjoint Variables State Variables
X
xp (t) λ(t) x(t) Kej (p) := κij (t, x(t), U k (t, ωk ), v) dt
NLP Solver k=1 tk−1

ns Z
X tk n o2
t t t Kij (p) := max 0; κij (t, x(t), U k (t, ω k ), v) dt
k=1 tk−1
Control Variables
U(t, ω) But, relaxation needed to ensure regular constraints:
Numerical
Integration Kj (p) ≤ ǫ
t
with ǫ > 0 a small nonnegative constant
Issue 1: How to calculate the cost and constraint values and derivatives?

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 14 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 15 / 32
Direct Sequential Method (cont’d) Direct Sequential Method (cont’d)

Issue 2: How to handle path constraints? Pros and Cons

Relatively small-scale NLP problems in the variables
T
Discretization as interior point constraints: pT = (ω 1 , . . . , ω ns T , vT )
Stage times can be optimized as part of the decision vector p too
 
κij tk,q , x(tk,q ), U k (tk,q , ω k ), v ≤ 0
  The accuracy of the state variables is enforced via the error-control
κej tk,q , x(tk,q ), U k (tk,q , ω k ), v = 0 mechanism of state-of-the-art numerical solvers
Feasible path method:
at a given set of points tk,q ∈ [tk−1 , tk ] in each stage k = 1, . . . , ns ◮ The ODEs are satisfied at each iteration of the NLP algorithm
◮ But, this is computationally demanding and handles mildly unstable
The combination of reformulation and discretization approaches is recom- systems only
mended!
Class Exercise: Give sufficient conditions for existence of an optimal
solution to the partially-discretized optimization problem

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 15 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 16 / 32

Scalar Dynamic Optimization Example Scalar Dynamic Optimization Example (cont’d)

Consider the dynamic optimization problem to:
R1 Optimal Results for 100 Stages Optimal Results for 100 Stages
minimize: J(u) := 0 [x1 (t)]2 + [x2 (t)]2 dt


25 1.5
u(t)
subject to: ẋ1 (t) = x2 (t); x1 (0) = 0 20
1

ẋ2 (t) = −x2 (t) + u(t); x2 (0) = −1 15

0.5

x2 (t) + 0.5 − 8[t − 0.5]2 ≤ 0, ∀t

x2 (t)
u(t) 10

0
−20 ≤ u(t) ≤ 20, ∀t 5

-0.5
0 x2 (t)
path constraint
Solution Approach Used -5
0 0.2 0.4
t
0.6 0.8 1
-1
0 0.2 0.4
t
0.6 0.8 1

Direct sequential approach with piecewise constant controls

Optimal Cost
Nonlinear program solved with SQP methods
ns 10 20 40 100
Inequality path constraint reformulated as: J(u ⋆ ) 1.13080 × 10−1 0.97320 × 10−1 0.96942 × 10−1 0.96893 × 10−1
Z 1
max{0; x2 (t) + 0.5 − 8[t − 0.5]2 }2 dt ≤ ǫ, with ǫ = 10−6
0

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 17 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 18 / 32
Multiple-Shooting Method Multiple-Shooting Method (cont’d)

Principle
Transcription into a finite-dimensional NLP through discretization of the Original Optimal Control Problem
control variables only, with state discontinuities allowed at stage times ˆ 0 , tf ]nu and v ∈ IRnv that
Determine: u ∈ C[t

Extra Decision Variables and Constraints minimize: φ(x(tns ), v)

New decision variables ξk0 ∈ IRnx , k = 2, . . . , ns , such that subject to: ẋ(t) = f (t, x(t), u(t), v); x(t0 ) = h(v)
i
ψ (x(tns ), v) ≤ 0
ẋk (t) = f (t, xk (t), U k (t, ω k ), v), tk−1 ≤ t ≤ tk
ψ e (x(tns ), v) = 0

k h(v) if k = 1, κi (t, x(t), u(t), v) ≤ 0, t0 ≤ t ≤ tf
with: x (tk−1 ) =
ξ k0 otherwise. κe (t, x(t), u(t), v) = 0, t0 ≤ t ≤ tf
New (nonlinear) equality constraints: u(t) ∈ [uL , uU ], v ∈ [vL , vU ]

xk (tk ) − ξ k+1
0 = 0, k = 1, . . . , ns − 1

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 19 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 20 / 32

Multiple-Shooting Method (cont’d) Multiple-Shooting Method (cont’d)

Partially-Discretized Optimization Problem Gradients Cost &

Constraints

Determine: ω 1 , . . . , ω ns ∈ IRnu M , ξ 10 , . . . , ξ n0s ∈ IRnx and v ∈ IRnv that

minimize: φ(x(tf ), v) Sensitivity Variables OR Adjoint Variables State Variables
xkp (t) λk (t) xk (t)
subject to: ẋk (t) = f (t, xk (t), U k (t, ω k ), v), tk−1 ≤ t ≤ tk , ∀k NLP Solver

x1 (t0 ) = h(v); xk (tk−1 ) = ξ k0 , k = 2, . . . , ns

t t t
ψ i (x(tf ), v) ≤ 0
Control Variables
ψ e (x(tf ), v) = 0 U(t, ω)
κi (t, x(t), U k (t, ω k ), v) ≤ 0, tk−1 ≤ t ≤ tk , ∀k Numerical
Integration
κe (t, x(t), U k (t, ω k ), v) = 0, tk−1 ≤ t ≤ tk , ∀k
t
ξ k0 ∈ [ξ L0 , ξ U
0 ], ω(t) ∈ [ω L , ω U ], v ∈ [vL , vU ]
Exact same procedure as the sequential approach, except that multiple
independent sets of differential equations
Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 21 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 22 / 32
Multiple-Shooting Method Handling Functionals with State Variables Participating
Consider the functional in Mayer form:
Pros and Cons ∆
F(p) = φ(x(tf ), p)
Lies between the sequential and simultaneous approaches: sequential
infeasible-path method subject to the parametric initial value problem:
Shares many attractive features of the sequential and simultaneous
approaches: ẋ(t) = f (t, x(t), p), t0 ≤ t ≤ tf ; x(t0 ) = h(p)
◮ accurate solution of the differential equations
◮ ability to deal with unstable systems safely Key Issues in Sequential Approaches:
SQP solvers exploiting the special structure of the resulting NLP Given that a unique solution x(t; p̄) to the ODEs exists for given p̄ ∈ P,
problem can/must be devised
1 Calculate the functional value, F(p̄)
Multiple shooting lends itself naturally to parallelization
◮ Use state-of-the-art methods for differential equations

Class Exercise: Give sufficient conditions for existence of an optimal 2 Calculate the functional derivatives, Fpj (p̄), w.r.t. p1 , . . . , pnp
solution to the partially-discretized optimization problem ◮ 3 possible procedures: finite differences, forward sensitivity analysis,
adjoint sensitivity analysis

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 23 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 24 / 32

Functional Derivatives Functional Derivatives (cont’d)

Finite Differences Approach: State Sensitivities: Under which conditions is x(t; ·) continuously
differentiable w.r.t. p1 , . . . , pnp , at a point p̄ ∈ P, for t0 ≤ t ≤ tf ?
φ(x(tf ), p1 , . . . , pi + δpi , . . . , pnp ) − φ(x(tf ), p) 1 A unique solution x(t; p̄) to the ODEs exists on [t , t ]
0 f
Fpi (p) ≈
δpi 2 f is continuously differentiable w.r.t. x and p, and piecewise

Procedure for Finite Differences Approach continuous w.r.t. t

3 h is continuously differentiable w.r.t. p
Initial Step:
Solve ODEs for the actual parameter values p;
Sensitivity Equations:
Calculate the value of F(p)
∆ ∂x(t)
Loop: j = 1, . . . , np The state sensitivities, xpj (t) = ∂pj , satisfy the ODEs:
Set p̃i := p i , i 6= j; p̃j := p j + δpj ;
ẋpj (t) = f x (t, x(t), p) xpj (t) + f pj (t, x(t), p)
Solve ODEs for the perturbed parameter values p̃
F(p̃) − F(p)
Calculate the value of F(p̃), and Fpj (p) ≈ with the initial conditions xpj (t0 ) = hpj (p)
δpj
End Loop
One sensitivity equation for each parameter p1 , . . . , pnp !
Trade-off: Choose δpj small enough, but not too small! Functional derivative: Fpj (p) = φx (x(tf ), p)xpj (tf ) + φpj (x(tf ), p)
Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 25 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 26 / 32
Functional Derivatives (cont’d) Functional Derivatives (cont’d)
Forward Sensitivity Approach:
Procedure for Forward Sensitivity Approach
State and Sensitivity Numerical Integration: t0 → tf Fpi (p) = φx (x(tf ), p) xpi (tf ) + φpi (x(tf ), p)

ẋ(t) = f (t, x(t), p); x(t0 ) = h(p) with: ẋpi (t) = f x (t, x(t), p) xpi (t) + f pi (t, x(t), p); xpi (t0 ) = hpi (p)
ẋp1 (t) = f x (t, x(t), p) xp1 (t) + f p1 (t, x(t), p); xp1 (t0 ) = hp1 (p)
Number of state & state sensitivity equations: (nx + 1) × np
.. ..
. . But, independent of the number of functionals, nF
ẋpnp (t) = f x (t, x(t), p) xpnp (t) + f pnp (t, x(t), p); xpnp (t0 ) = hpnp (p) Adjoint (Reverse) Sensitivity Approach:

Function and Gradient Evaluation:

Z tf
T
Fpi (p) = φpi (x(tf ), p) + λ(t0 ) hpi (p) + f pi (t, x(t), p)T λ(t)dt
F(p) = φ(x(tf ), p) t0

Fp1 (p) = φx (x(tf ), p)xp1 (tf ) + φp1 (x(tf ), p) with: λ̇(t) = −f x (t, x(t), p)T λ(t); λ(tf ) = φx (x(tf ), p)
..
.
Number of state & adjoint equations: 2nx × nF
Fpnp (p) = φx (x(tf ), p)xpnp (tf ) + φpnp (x(tf ), p)
But, independent of the number of parameters
Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 27 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 28 / 32

Functional Derivatives (cont’d) Functional Derivatives (cont’d)

Procedure for Adjoint Sensitivity Approach Procedure for Adjoint Sensitivity Approach (cont’d)
State Numerical Integration: t0 → tf Function and Gradient Evaluation:
ẋ(t) = f (t, x(t), p); x(t0 ) = h(p); F(p) = φ(x(tf ), p)
Store state values x(t) at mesh points, t0 < t1 < · · · < tM = tf Fp1 (p) = φp1 (x(tf ), p) + λ(t0 )T hp1 (p) + q1 (t0 )
Adjoint Numerical Integration: tf → t0 ..
.
λ̇(t) = − f x (t, x(t), p)T λ(t); λ(tf ) = φx (x(tf ), p)T Fpnp (p) = φpnp (x(tf ), p) + λ(t0 )T hpnp (p) + qnp (t0 )
q̇1 (t) = − f p1 (t, x(t), p)T λ(t); q1 (tf ) = 0
..
.
q̇np (t) = − f pnp (t, x(t), p)T λ(t); qnp (tf ) = 0;

Need to interpolate state values x(t) between mesh points

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 29 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 30 / 32
Functional Derivatives (cont’d) Existence of an Optimal Solution (Finite Dimension)

Class Exercise: Consider the functional

F(p1 , p2 ) = x(1)
with: ẋ(t) = −x(t) + p1 ;
Recalls: Weierstrass’ Theorem
Let P ⊂ IRn be a nonempty, compact set, and let φ : P → IR be
continuous on P. Then, the problem min{φ(p) : p ∈ P} attains its
minimum, that is, there exists a minimizing solution to this problem.
x(0) = p2

1 Calculate the value of F by first solving the ODE analytically; then,

Why closedness of P? Why continuity of φ? Why boundedness of P?
calculate the derivatives Fp1 and Fp2
2 Derive the sensitivity equations and their respective initial conditions,
then find a solution to these equations; apply the forward sensitivity φ φ
formula to calculate Fp1 and Fp2 φ
3 Derive the adjoint equation and its terminal condition, then find a φ(c)

solution to this equation; apply the adjoint sensitivity formula to

calculate Fp1 and Fp2
a b a c b a +∞
(a) (b) (c)

Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 31 / 32 Benoı̂t Chachuat (McMaster University) Direct Methods Optimal Control 32 / 32