Optimization methods (MFE)

Lecture 04

Elena Perazzi

EPFL

Fall 2019

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 1 / 27

Today’s topics

Introduction to dynamic optimization.

Finite horixon problems. Solving by backward induction.

Example: optimal growth model.

Infinite horizon problem. Example: optimal growth model.

I Euler equations: analytic solution in special cases.
I Dynamic programming: the Bellman equation. Solutions by value
function iteration.
I Solutions by inearization of the Euler equation around steady states.

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 2 / 27

 An example: optimal growth
model

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 3 / 27

The optimal growth model (Cass-Koopmans model)
Agents’ objective is to maximize multi-period utility

ΣT t
t=0 β U(ct )

Infinite-horizon problem is obtained for T → ∞

Technology:
yt = f (kt )
Capital accumulation:

kt+1 = kt (1 − δ) + it δ → capital depreciation

Resource constraint:

ct + it = yt
or ct + (kt+1 − kt (1 − δ)) = f (kt )

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 4 / 27

The optimal growth model (Cass-Koopmans model)
We set δ = 1 (because the solution is simpler!). Problem:
max{kt+1 ,ct }T ΣT t
t=0 β U(ct )
0

s.t. ct + kt+1 = f (kt )

kt+1 ≥ 0;
k0
The solution is a sequence {c0 , k1 }, {c1 , k2 }, ..., {cT , kT +1 }.
We could write the Lagrangean, but in this case we substitute the
constraint in the utility function
max{kt+1 }T ΣT t
t=0 β U(f (kt ) − kt+1 )
0

Last period, no savings:

kT +1 = 0
FOC
−U 0 (f (kt ) − kt+1 ) + βf 0 (kt+1 )U 0 (f (kt+1 ) − kt+2 ) = 0 t < T
Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 5 / 27
Optimal growth: a special case
Let’s set
U(c) = ln(c)
f (k) = k α , 0<α<1
FOC
1 α−1 1
= αβkt+1
ktα − kt+1 α −k
kt+1 t+2
define the saving rate zt
kt+1
zt ≡
ktα
FOC becomes
1 1 1 1 α−1
= αβ kt+1
ktα 1 − zt α 1−z
kt+1 t+1
zt 1
= αβ
1 − zt 1 − zt+1
Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 6 / 27
A recursive solution
Start from the boundary condition zT = 0 and solve backward
zT = 0
αβ
zT −1 =
1 + αβ
αβ(1 + αβ)
zT −2 =
1 + αβ(1 + αβ)
αβ(1 + αβ + (αβ)2 + ... + (αβ)n−1 )
zT −n =
1 + αβ + (αβ)2 + .... + (αβ)n
So
limn→∞ zT −n = αβ
This is the solution when there is a terminal period, which is infinitely
far away. Not guaranteed to be also the solution for the
infinite-horizon problem since in general
max limT →∞ ΣT T
t=0 U(ct ) 6= limT →∞ maxΣt=0 U(ct )

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 7 / 27

Infinite-horizon problem: recursive methods
Idea of dynamic programming to break down the infinite-horizon
problem into 2-period subproblems: ’

V (k0 ) = maxk1 {U(f (k0 ) − k1 ) + βV (k1 )}

V (k) : Value function

This is the Bellman equation. We can regard this as an equation
where the argument is the function v : Functional equation.
It involves two types of variables: state variables, that summarize the
state of the world at the beginning of each period; choice variables. In
this formulation (having substituted ct = f (kt ) − kt+1 the choice
variable at t is kt+1 , which coincides with the state variable at t + 1.
FOC wrt the choice variable kt+1 :

U 0 (f (kt ) − kt+1 ) = βV 0 (kt+1 )

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 8 / 27

Assume that the solution exists, and that the solution is such that
kt+1 = g (kt ). Then we can write the Bellman equation as

V (kt ) = U(f (kt ) − g (kt )) + βV (g (kt ))

totally diferentiating wrt kt :

V 0 (kt ) = U 0 (f (kt )−g (kt ))f 0 (kt )−(U 0 (f (kt )−g (kt ))−βV 0 (g (kt )))g 0 (kt )

The FOC says that the second term is 0, so

V 0 (kt ) = U 0 (f (kt ) − g (kt ))f 0 (kt )

(this is called the envelope condition) Hence the FOC becomes

U 0 (f (kt ) − kt+1 ) = βU 0 (f (kt+1 ) − kt+2 )f 0 (kt+1 )

This is the Euler equation

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 9 / 27

In our special case U(c) = ln(c), f (k) = k α the Euler equation is
kt+1 kα
= αβ t+1
ct ct+1
Substituting the FOC into the resource constraint ktα = ct + kt+1 we get
ktα kα
= 1 + αβ t+1
ct ct+1
So
ktα kα
= 1 + αβ(1 + αβ t+2 )
ct ct+2
Iterating this procedure we finally obtain
ktα 1
=
ct 1 − αβ
so the policy function is
ct = (1 − αβ)ktα
Note that this is the answer we guessed before, based on the finite-horizon
problem.
Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 10 / 27
The policy function can be written as kt+1 = αβktα
0.3

0.25

0.2
Kt+1

0.15

0.1

0.05

0
0 0.05 0.1 0.15 0.2 0.25 0.3
Kt

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 11 / 27

 Value function iteration

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 12 / 27

Value function iteration
Another way to obtain a solution (of the optimal growth problem) is based
on iteration of the value function. Algorithm:
Start with a guess for the value function for some period T + 1; e.g.
V0 (kT +1 ) = 0
In the previous period clearly
V1 (kT ) = U(f (kT ))
(consume everything at T !)
In each of the previous periods solve a problem of the form
.Vn (k) = maxk + ((U(f (k) − k + ) + βVn−1 (k + ))
which is a regular maximization problem.
Eventually, V (k) converges to its true value.
This method (implemented numerically) is very good for more
complex problems.
How do we know that it converges?
Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 13 / 27
A general recursive problem

General deterministic problem with ingredients:

I Infinite horizon
I Discrete time
I State variable xt in some set X
I Constraint correspondence xt+1 ∈ Γ(xt ).
Given sequential problem
V (x0 ) ≡ maxx1 ,x2 ,... Σ∞ t
t=0 β F (xt , xt+1 )
s.t. xt+1 ∈ γ(xt ), t = 0, 1, ...
If the problem has a bounded solution for every x0 ∈ X , then the
problem can be equivalently formulated as a Functional Equation
V (xt ) = maxy ∈Γ(x) F (xt , xt+1 ) + βV (xt+1 )
The optimal growth model can be written in the form above, where
we define F (kt , kt+1 ) ≡ U(f (kt ) − kt+1 )
x in general l−dimensional vectors.
Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 14 / 27
The functional equation
Functional equation (FE)

V (xt ) = maxxt+1 ∈Γ(xt ) F (xt , xt+1 ) + βV (xt+1 )

The RHS of this equation can be viewed as a map

T : C (X ) → C (X )

C (X ): set of bounded, continuous functions, on which we can impose

the sup norm

||f (x) − g (x)|| = supx∈X |f (x) − g (x)|

f is a fixed point of T ⇔ f solves FE

How do we know that a fixed point exists/ is unique?
How do we find it?
Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 15 / 27
Contraction Mapping Theorem

The map T : C (X ) → C (X ) is a contraction if, for every f , g ∈ C (X )

||T (f ) − T (g )|| ≤ β||f (x) − g (x)||

with 0 ≤ β < 1
If a contraction T has a fixed point, the fixed point is unique.
Suppose two fixed points f and f 0 . Then
||f − f 0 || = ||T (f ) − T (f 0 )|| ≤ β||f − f 0 || . So it must be that f and
f 0 coincide.
If X is a complete metric space, a contraction T has a unique fixed
point.

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 16 / 27

Blackwell sufficient conditions

How to make sure that T is a contraction? with the Blackwell sufficient

conditions! Assume:
T is monotonic, i.e. f (x) ≥ g (x) for all x ⇒ Tf (x) ≥ Tg (x) for all x
T satisfies the ”discounting property” : there is a δ ∈ (0, 1) s.t. for
any a > 0 and any g (x) ∈ C (X ) the function f (x) = g (x) + a satisfies

Tf (x) − Tg (x) ≤ δa

If the two above conditions are satisfied, then T is a contraction.

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 17 / 27

Blackwell sufficient conditions
We now prove that the Blackwell conditions are satisfied by the operator
T of the Bellman equation.
Monotonicity: Suppose V1 (x) ≥ V2 (x). Then for each x

(TV1 )(x) = maxy G1 (y ; x)

(TV2 )(x) = maxy G2 (y ; x)

where G1 (y ; x) = F (x, y ) + βV1 (y ), G2 (y ; x) = F (x, y ) + βV2 (y ).

Clearly, for every x,

(V1 (y ) ≥ V2 (y )) ⇒ G1 (y ; x) ≥ G2 (y ; x)
⇒ maxy G1 (y ; x) ≥ maxy G2 (y ; x).
⇒ (TV1 )(x) ≥ (TV2 )(x).

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 18 / 27

Blackwell sufficient conditions

Discounting: Suppose V1 (x) = V2 (x) + a. Then for each x

(TV1 )(x) = maxy F (x, y ) + βV1 (y )

= maxy F (x, y ) + β(V2 (y ) + a)
= maxy (F (x, y ) + βV2 (y )) + βa
= maxy (TV2 )(x) + βa

This shows that discounting applies (with δ = β).

Monotonicity and discounting property satisfied by the operator T of
the Bellman equation → a fixed point (=solution) exists and is
unique.

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 19 / 27

Euler equations and dynamics
around stady states

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 20 / 27

Euler equations

In the optimal growth problem we have seen that the solution must
satisfy the Euler equation

U 0 (f (k0 ) − k1 ) = βf 0 (k1 )U 0 (f (k1 ) − k2 )

In the formulation

V (x) = maxy F (x, y ) + βV (y )

the Euler equation reads

F2 (xt , xt+1 ) + βF1 (xt+1 , xt+2 ) = 0

Fi , i = 1, 2 → gradient wrt i − th argument.

Remember x is in general an n− dimensional vector.

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 21 / 27

Consider a problem with F (x, y ) (x an n-dimensional vector), concave,
strictly increasing in x. Suppose that the state xt is a non-negative vector.
Then sufficient conditions for a sequence {xt∗ , xt+1
∗ , ....} to be an optimum

are
The sequence {xt∗ , xt+1
∗ , ....} satisfies a Euler equation

F2 (xt∗ , xt+1
∗ ∗
) + βF1 (xt+1 ∗
, xt+2 )=0

for all t, and

The transversality condition is satisfied:

limt→∞ β t F2 (xt∗ , xt+1

∗ ∗
)xt+1 =0

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 22 / 27

Steady states

A steady state x ∗ satisfies

F2 (x ∗ , x ∗ ) + βF1 (x ∗ , x ∗ ) = 0

If the initial state is the steady state, the future state will always be
the steady state.
The steady state is locally stable if the initial state is in a region
around the steady state, the solution is a seqence {xt } that converges
to the steady state for t → ∞.
Many economic problems have solution of this type. Also the optimal
growth model!

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 23 / 27

Dynamics around the steady state
Suppose the initial state x0 is not too far from steady state. The
solution of the problem is a sequence {xt∗ } satisfying for all t’s

F2 (xt∗ , xt+1
∗ ∗
) + βF1 (xt+1 ∗
, xt+2 )=0

with x0∗ = x0 . We can linearize the Euler equation around steady state

F1 (xt∗ , xt+1
∗
) = F1 (x ∗ , x ∗ ) + F11 × (xt∗ − x ∗ ) + F12 × (xt+1
∗
− x ∗)
F2 (xt∗ , xt+1
∗
) = F2 (x ∗ , x ∗ ) + F21 × (xt∗ − x ∗ ) + F22 × (xt+1
∗
− x ∗)

where Fij , i, j = 1, 2 denote Fij (x ∗ , x ∗ ). Hence

F2 (xt∗ , xt+1
∗
) + βF1 (xt∗ , xt+1
∗
) = (F2 (x ∗ , x ∗ ) + βF1 (x ∗ , x ∗ )) +
F21 × (xt∗ − x ∗ ) + (F22 + βF11 ) × (xt+1
∗
− x ∗ ) + βF12 × (xt+2
∗
− x ∗)

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 24 / 27

Dynamics around the steady state

Defining zt ≡ xt∗ − x ∗ , the linearized Euler equation is

F21 zt + (F22 + βF11 )zt+1 + βF12 zt+2 = 0

This is a 2nd order difference equation, that can be written as

  −1 −1 −1
β F12 (F22 + βF11 ), β −1 F12
  
zt+2 F21 zt+1
=
zt+1 I, 0 zt

We are looking for a solution

 that
 tends
  asymptotically to steady
j z1 0
state, i.e. such that M →
z0 0
If we find such a solution, it is the unique solution. If we don’t find it,
then there is no optimal path starting from x0 and ending in x ∗ .

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 25 / 27

Steady state stability in the optimal growth model

0.3

0.25

0.2
Kt+1

0.15

0.1

0.05

0
0 0.05 0.1 0.15 0.2 0.25 0.3
Kt

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 26 / 27

Dynamics around the steady state

Blanchard-Kahn conditions
If M has l eigenvalues whose norm is smaller than 1, the unique
solution is converging to steady state.

Proof & intuition: next class!

Elena Perazzi (EPFL) Optimization methods (MFE) Lecture 04 Fall 2019 27 / 27