Dynamic Programming and Optimal

Control
Final Recitation

Dario Brescianini, Robin Ritz | 17.12.2014 |

Outline

Overview
Dynamic Programming Algorithm (DPA)
Deterministic Systems and the Shortest Path (SP)
Infinite Horizon Problems, Stochastic SP
Deterministic Continuous-Time Optimal Control

Dario Brescianini, Robin Ritz | 17.12.2014 |

Overview

Dario Brescianini, Robin Ritz | 17.12.2014 |

Dynamic Programming Algorithm (DPA)

Basic Problem

Alternative Problem Formulation

Reformulations
Time lag, correlated disturbances, forecasts,

Dario Brescianini, Robin Ritz | 17.12.2014 |

Dynamic Programming Algorithm (DPA)

Basic idea: Principle of Optimality
Algorithm:

Minimizing the recursion equation for each

us the optimal policy:

and

gives

Dario Brescianini, Robin Ritz | 17.12.2014 |

Deterministic Systems and the Shortest Path

Consider now problems where

is a finite set,
No disturbance
.

Convert DP to SP (and vice versa)

DP:

SP:

Viterbi Algorithm
Dario Brescianini, Robin Ritz | 17.12.2014 |

Deterministic Systems and the Shortest Path

DP finds all optimal paths to end node. Sometimes not
needed.
Exploit structure of these problems to come up with
efficient algorithms for solving shortest path problems:
Label Correcting Algorithm
Step 1: Remove a node i from OPEN and for each child j of i,
execute step 2.
Step 2: If di + aij < min{dj,UPPER}, set dj = di + aij and set i to be the
parent of j. In addition, if jt, place j in OPEN if it is not already in
OPEN, while if j=t, set UPPER to the new value di+ait of dt.

Step 3: If OPEN is empty, terminate; else go to step 1.

Dario Brescianini, Robin Ritz | 17.12.2014 |

Infinite Horizon Problems

Consider time-invariant system with infinite horizon:

Optimal policy is stationary:

Optimal cost solves Bellmans equation:

Dario Brescianini, Robin Ritz | 17.12.2014 |

Infinite Horizon Problems: Stochastic Shortest Path

Stochastic Shortest Path problems:

Cost-free termination state :

a policy and an integer
such that:

Dario Brescianini, Robin Ritz | 17.12.2014 |

Infinite Horizon Problems: Stochastic Shortest Path

Value iteration:
Step 1: Choose an initial guess
.
Step 2: Update cost values with the value iteration formula:

Step 3: If

converged for all , terminate. Else go to step 2.

Dario Brescianini, Robin Ritz | 17.12.2014 | 10

Infinite Horizon Problems: Stochastic Shortest Path

Policy iteration:
Step 1: Choose an initial stationary policy .
Step 2: Policy evaluation (compute cost of current policy):
(lin. sys. of eq.)

Step 3: Policy improvement (find a better policy):

Step 4: If

for all , terminate. Else go to step 2.

Dario Brescianini, Robin Ritz | 17.12.2014 | 11

Infinite Horizon Problems: Stochastic Shortest Path

Linear programming:
Optimal cost

For each admissible pair

solves the following linear program:

we get one linear constraint

Dario Brescianini, Robin Ritz | 17.12.2014 | 12

Infinite Horizon Problems: Discounted Problems

Discounted problems:

Discounted cost:

Dario Brescianini, Robin Ritz | 17.12.2014 | 13

Deterministic Continuous-Time Optimal Control

Basic Problem

No noise: deterministic.
Goal: Find an admissible control trajectory
,
,
and corresponding state trajectory
which minimize
the cost.
Solution is found by HJB or Minimum Principle.

Dario Brescianini, Robin Ritz | 17.12.2014 | 14

Deterministic Continuous-Time Optimal Control

Hamilton-Jacobi-Bellman Equation (cont.-time analog to DPA)

Derived by discretizing and taking limits of DPA.

Partial differential equation. Very hard to solve!
Usually guess a solution and proof that is satisfies HJB.

Sufficient condition.
Optimal policy:

that minimize RHS of HJB.

Dario Brescianini, Robin Ritz | 17.12.2014 | 15

Deterministic Continuous-Time Optimal Control

Minimum Principle (Only finds optimal solution for a specific initial condition
Define Hamiltonian:
Then:

Only necessary conditions.

Various extensions (e.g. fixed terminal state, ).
Dario Brescianini, Robin Ritz | 17.12.2014 | 16