You are on page 1of 27

# Optimal Control With Discounting

## Elfego López Torres

OPTIMIZACIÓN DINÁMICA
elphego14@gmail.com

12 de enero de 2018

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 1 / 27

Current Value Hamiltonian Function

## We have noted that a major feature of the control problem is to maximise

the objective function F (x, u). However, for many economic problems
F (x, u) would represent such things as profits or net benefits. The
economist would not simply maximise such an income stream without first
discounting it to the present. Thus, if δ were the rate of discount then the
aim of the control would be to
Z T
máx J = e−δt F (x, u) dt (1.1)
{u(t)} 0

## subject to various conditions which are unaffected by the discounting.

Thus, the typical continuous time maximization principle problem with
discounting is the control problem

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 2 / 27

Current Value Hamiltonian Function

Z T
máx J = e−δt F (x, u) dt (1.2)
{u(t)} 0
ẋ = g(x, u)
x(0) = x0
x(T ) = xT

## Consider now discounting under a continuous time model. Consider the

control problem (1.2). The Lagrangian is
Z T n o
L= e−δt F (x, u) + λ[g(x, u) − ẋ] dt (1.3)
0

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 3 / 27

Current Value Hamiltonian Function

## Hc (x, u) = F (x, u) + µg(x, u) (1.5)

then

Hc = Heδt or H = Hc e−δt
µ = λeδt or λ = µe−δt

Now reconsider our five optimality conditions. Since eδt is a constant for a
change in the control variable, then condition (i) is simply ∂Hc /∂u = 0.
The second condition is less straightforward. We have

∂H ∂Hc −δt
λ̇ = − =− e
∂x ∂x

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 4 / 27

Current Value Hamiltonian Function

From λ = µe−δt

λ̇ = µ̇e−δt − δµe−δt
Equating these we have

∂Hc −δt
− e = µ̇e−δt − δµe−δt
∂x
∂Hc
or µ̇ = − + δµ
∂x
Condition (iii) is

## ∂H ∂Hc −δt ∂Hc

ẋ = = e = = g(x, u)
∂λ ∂λ ∂µ
while condition (iv) becomes

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 5 / 27

Current Value Hamiltonian Function

## unchanged. To summarise, define

the current value Hamiltonian and ∂Hc
µ̇ = − + δµ 0≤t≤T
current value Lagrangian multiplier, ∂x
i.e. 3

∂Hc
δt ẋ = = g(x, u)
Hc (x, u) = H(x, u)e ∂µ
= F (x, u) + µg(x, u) 4

when λ = µe−δt .
Then the x(0) = x0
optimality conditions are:
5
1

∂Hc
=0 0≤t≤T µ(T )e−δt = 0 (or x(T ) = xT )
∂u

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 6 / 27

Current Value Hamiltonian Function

## Current Value Formulation

These optimality conditions allow us to eliminate the control variable u
using condition (i) and to obtain two differential equations:one for the
state variable, x, and the other for the current value costate variable, µ.
Many control problems in economics literarture have the following
structure:

Z t1
máx F (t, x, u)e−δt dt
u∈U ⊆R t0

ẋ = g(t, x, u)
x(t0 ) = x0 (1.6)

(a) x(t1 ) = x1

(b) x(t1 ) ≥ x1

(c) x(t1 ) free

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 7 / 27

Current Value Hamiltonian Function

The new feature is the explicit appearance of the discount e−δt . For such
problems it often convenient to formulate the maximum principle in a
slightly different form. The ordinary Hamiltonian is
H = p0 F (t, x, u)e−δt + pg(t, x, u). Multiply it by eδt to obtain the current
value Hamiltonian Hc = Heδt = p0 F (t, x, u) + eδt pg(x, t, u). Introducing
µ = eδt p as the current value shadow price for the problem, one can
write Hc in the form (where we put p0 = µ0 )

## Hc (t, x, u, µ) = µ0 F (t, x, u) + µg(t, x, u) (1.7)

Note that if µ = eδt p, then µ̇ = δeδt p + eδt ṗ and so ṗ = e−δt (µ̇ − δµ).
Also, Hc = Heδt implies that ∂Hc /∂x = eδt (∂H/∂x). So ṗ = −∂H/∂x
takes the form µ̇ − δµ = −∂Hc /∂c. In fact, one can prove the following:

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 8 / 27

Current Value Hamiltonian Function

Theorem (1)

## The Maximum principle Current Value Formulation.

Suppose that the admissible pair (x∗ (t), u∗ (t)) solves problem (1.6) and let Hc
be the current value Hamiltonian (1.7). Then there exists a continuous function
µ(t) and a number µ0 , either 0 or 1, such for all t ∈ [t0 , t1 ] we have (µ0 , µ(t))
6= (0, 0) and:

## u = u∗ (t) maximizes Hc (t, x∗ (t), u, µ(t)) for u ∈ U

∂Hc (t, x∗ (t), u∗ (t), µ(t))
µ̇(t) − δµ(t) = −
∂x
Finally, the transversality conditions are:
µ(t1 ) no conditions
µ(t1 ) ≥ 0 (µ(t1 ) = 0 if x∗ (t1 ) > x1 )
µ(t1 ) = 0

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 9 / 27

Current Value Hamiltonian Function

## The Mangasarian and Arrow sufficiency, have immediate extensions to

problem (1.6). The conditions in Theorem (1) are sufficient for optimality
if µ0 = 1 and

## Hc (t, x, u, µ(t)) is concave in (x, u) (Mangasarian)

or (more generally)

H
b c (t, x, µ(t)) = máx Hc (t, x, u, µ(t)) is concave in x (Arrow)
u∈U

## RONALD SHONE. Economic Dynamics Phase Diagrams and Their

Economic Application, Second Edition, Cambridge University Press, New
York 2002.
Knut Sydsæter, Peter Hammond, Atle Seierstad and Arne
Strøm. Further Mathematics for Economic Analysis. Pearson Education
Limited 2005.
Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 10 / 27
Exercises

Z 10
x − u2 e−t dt

1 opt
0
ẋ = x + u
(2.1)
1
x(0) =
2
x(10) = 0

## The current value Hamiltonian is

Hc = x − u2 + µ(x + u)

## with optimality conditions

∂Hc
1
∂u = −2u + µ = 0
2 µ̇ = −1
3 ẋ = x + u
Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 11 / 27
Exercises

## From (1) we have u = µ/2, which when substituting (2.4) in (2.2)

substituted into (3) gives ẋ = x + µ/2.
Thus we have two differential equations 10 − t
ẋ = x + (2.5)
2
µ solving (2.5)
ẋ = x + (2.2)
2
t 9
µ̇ = −1 (2.3) x(t) = − + k2 et (2.6)
2 2
solve (2.3) applying condition in (2.6)

µ(t) = −t + k1 t 9
x∗ (t) = − + 5et
2 2
transversality condition:
and
µ∗ (t) = 10 − t (2.4) u∗ (t) = 5 − 0.5t

Exercises

## From (1) we have u = µ, which when

Z 2 substituted into (3) gives ẋ = 3x + 2µ.
x2 − u2 e−2t dt

2 opt Thus we have two differential equations
0
ẋ = 3x + 2u (2.7)
x(0) = 3
ẋ = 3x + 2µ (2.8)
µ(2) = 0 µ̇ = −2x − µ (2.9)
The current value Hamiltonian is
Hc = x2 − u2 + µ(3x + 2u) from (2.8): µ = ẋ/2 − 3x/2, we make
with optimality conditions
∂Hc
1
∂u = −2u + 2µ = 0  
d ẋ 3x
2 µ̇ = −2x − µ µ= −
dt 2 2
3 ẋ = 3x + 2u ẍ 3ẋ
µ̇ = − (2.10)
2 2

Exercises

## substituting (2.10) in (2.9) The roots are: repeated eigenvalues,

r1 = 1 = r2 = 1, so the general
ẍ 3ẋ ẋ 3x solution
− = −2x − −
2 2 2 2
simplifying: x(t) = k1 tet + k2 et (2.12)
ẍ − 2ẋ + x = 0 (2.11) from (2.12)
The equation (2.11) is a homogeneous
differential equation of second order. ẋ(t) = k1 et (t + 1) + k2 et .
We can solve the characteristic We know that
equation
ẋ 3x ẋ(t) 3x(t)
r2 − 2r + 1 = 0 µ= − ∴ µ(t) = −
2 2 2 2

so

k1 et (t + 1) + k2 et 3 k1 tet + k2 et

µ(t) = −
2 2
Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 14 / 27
Exercises

## simplifying then, the general solution

1
µ(t) = −k1 tet + k1 et − k2 et (2.13)
2
applying the conditions: x(0) = 3, µ(2) = 0 in (2.12) and (2.13), so we
have

k1 = −2 k2 = 3
Finally

## x∗ (t) = 3et − 2tet

µ∗ (t) = 2tet − 4et

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 15 / 27

Exercises

5
u2 x2
Z 
3 opt − + xu − e−3t dt
0 2 2
ẋ = u (2.14)
x(0) = 1
µ(5) = 0

## The current value Hamiltonian is From (1) we have u = x + µ, which

u 2
x 2 when substituted into (2) and (3),
Hc = − + xu − + µu gives ẋ = x + µ and µ̇ = 2µ. Thus we
2 2
have two differential equations
with optimality conditions
∂Hc
1
∂u = −u + x + µ = 0 ẋ = x + µ (2.15)
2 µ̇ = −u + x + 3µ µ̇ = 2µ (2.16)
3 ẋ = u

Exercises

## µ(t) = k1 e2t x(t) = k2 et with x(0) = 1

transversality condition
x∗ (t) = et

µ (t) = 0
then from (2.15) for u

ẋ − x = 0 u∗ (t) = et

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 17 / 27

Exercises

Z2 " 2 #
1
4 opt x− + u2 e−3t dt
2
0
(2.17)
ẋ = 3x − 2u
x(0) = 2
µ(2) = 0
The current value Hamiltonian is
 2
1
Hc = x − + u2 + µ(3x − 2u)
2
with optimality conditions From (1) we have u = µ, which when
substituted into (3) gives ẋ = 3x + 2µ.
∂Hc Thus we have two differential equations
1 = 2u − 2µ = 0
∂u
2 µ̇ = 1 − 2x
ẋ = 3x − 2µ (2.18)
3 ẋ = 3x − 2u µ̇ = 1 − 2x (2.19)

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 18 / 27

Exercises

0
from (2.18): µ = 3x x
2 − 2 : factoring:
3x x0
 
d
µ= − (r − 1)(r + 4) = 0
dt 2 2

3x0 x00 .
µ0 = − (2.20) The roots r1 = −1 and r2 = 4, are real
2 2
substituting (2.20) in (2.19) and distinct, thus our particular
solution is −2/ − 4 = 1/2, so the
3x0 x00 general solution
− = 1 − 2x
2 2
normalizing
1
x(t) = k1 e−t + k2 e4t + (2.22)
2
x00 − 3x0 − 4x = −2 (2.21)
We can solve the characteristic 3x x0
we know that µ = 2 − 2 ∴
equation
3x(t) x0 (t)
r2 − 3r − 4 = 0 µ= −
2 2

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 19 / 27

Exercises

that is to say
 
3 1 1
k1 e−t + k2 e4t + −k1 e−t + 4k2 e4t

µ(t) = −
2 2 2
simplifying

k2 e4t 3
µ(t) = 2k1 e−t − +
2 4
applying the conditions, we have

## 3(e8 − 1)e2 3(e2 + 4)

k1 = k2 =
2(e10 + 4) 2(e10 + 4)
finally

" #
e2 + 4 e4t + e10 − e2 e−t
 
∗ 3 1
x (t) = +
2 e10 + 4 3
" #
3 −e2 − 4 e4t + 4 e10 − e2 e−t
 
∗ ∗
u (t) = µ (t) = +1
4 e10 + 4

Exercises

## Solving (2.17) through matricial methods.

Matricial form (2.18) and (2.19)
      
ẋ 3 −2 x 0
= +
µ̇ −2 0 µ 1
Define:


3 −2
A= .
−2 0
The characteristic equation of A is

 
A − δI = 3 − δ −2

−2 −δ
= δ(δ − 3) − 4
= δ 2 − 3δ − 4 = 0
= (δ + 1)(δ − 4) = 0,

Exercises

## the two eigenvalues of A are

δ1 = 4 δ2 = −1.
The eigenvector for δ1 :

   
−1 −2 a 0
A − δ1 v1 = 0 = = ,
−2 −4 b 0

then:
   
−1 −2 0 Gauss/Jordan 1 2 0
−−−−−−−−−→ .
−2 −4 0 0 0 0
The eigenvector of δ1 is:
 
−2
v1 = .
1

Exercises

## The eigenvector for δ2 :

    
4 −2 c 0
A − δ 2 v2 = 0 = = ,
−2 1 d 0

then:

− 12
   
4 −2 0 Gauss/Jordan 1 0
−−−−−−−−−→ .
−2 1 0 0 0 0
The eigenvector of δ2 is:
 
1
v2 = .
2
The homogeneous solution is:

     
x(t) −2 4t 1 −t
= k1 e + k2 e .
µ(t) h 1 2

Exercises

## The non-homogeneous solution is:

   
ẋ 0
= ,
µ̇ nh 0
then:

3x − 2µ = 0 (2.23)
− 2x + 1 = 0 (2.24)

from (2.24):
1
x= ,
2
substituting the value of x in (2.23):
3
µ= .
4

Exercises

## " # " # " # " #

1
x(t) −2 4t 1 −t
= k1 e + k2 e + 23 ,
µ(t) 1 2 4

## " # " # " # "  #" # " #

x∗ (t) −3 e2 + 4 −2 4t 3 e8 − 1 e2 1 −t 1
2
= e + e + 3 .
µ∗ (t) 4 (e10 + 4) 1 2 (e10 + 4) 2 4

Finally

" #
e2 + 4 e4t + e10 − e2 e−t
 
∗ 3 1
x (t) = +
2 e10 + 4 3
" #
3 −e2 − 4 e4t + 4 e10 − e2 e−t
 

µ (t) = +1 .
4 e10 + 4

## Elfego López Torres (IPN) Escuela Superior de Economı́a 12 de enero de 2018 25 / 27

Exercises

Z20
4K − u2 e−0.25t dt

5 máx
u≥0
0
(2.25)
K̇ = −0.25K + u
K(0) = K0
K(20) is free

Economic interpretation: K(t) is the value of machine, u(t) is the repair effort,
4K − u2 is the instantaneous net profit at time t, and e−0.25t is the discount
factor
Solution: The current value Hamiltonian is: Hc = 4K − u2 + µ(−0.25K + u)
(with µ0 = 1), and so ∂H ∂Hc
∂u = −2u + µ and ∂K = 4
c
− 0.25µ. Assuming that
∂Hc∗
u (t) > 0 (we thy assumption in the following), ∂u = 0, so u∗ (t) = 0.5µ(t).

## The adjoint function µ satisfie

∂Hc∗
µ̇ − 0.25µ = − = −4 + 0.25µ, µ(20) = 0
∂K

Exercises

It follows that

## µ(t) = 8 1 − e0.5t−10 u∗ (t) = 0.5µ = 4 1 − e0.5t−10

 
and
Note that u∗ (t) > 0 in [0, 20). The time path of K ∗(t) is found from
K̇ ∗ = −0.25K ∗ + u∗ = −0.25K ∗ + 4 1 − e0.5t−10 . Solving this linear
differential equation with k ∗ (0) = K0 , we get
 
∗ 16 −10 −0.25t 16
K (t) = K0 − 16 + e e + 16 − e0.5t−10
3 3
2

Here Hc = 4K − u + µ(−0.25K + u) is concave in (K, u), so we have found
the complete solution.