Lecture 3.b - Dynamic Programming

University of Zagreb, Croatia
Lecture 3.b
Heuristic Optimization Methods:
Dynamic Programming
Slides prepared by Nina Skorin-Kapov, Lea Skorin-
Kapov
Academic year 2020/2021

Graduate Studies Programme
Dynamic Programming
 A technique to make a sequence of related

decisions and systematically find the optimal
solution
 Generalization of recursion
 Difficult to recognize when DP can be applied
A problem can be solved with DP if its solution can be
constructed from solutions to its subproblems
 There is no standard problem formulation
needed as in LP and IP-based methods
Zavod za telekomunikacije 2
Dynamic Programming
 Main idea: start with a small part of the problem and find
the optimum. Then systematically extend the problem→
“bottom-up approach”
 Opposite of B&B
 OPTIMALITY PRINCIPLE: any subsequence of the
optimal sequence of decisions is the optimal solution to
the corresponding subproblem
 KEY: in a given state, subsequent decisions do not depend on
previous ones but just the current state
e.g., shortest path:

dn
d1 di
dj
Example 1: shortest path
3
6
1 A B 2
S 4 1 2 4 6 1 1
E S C A B D E
2 3 1
C D 1 2
directed acyclic graph linearization
shortest path from S dist ( D) = min{dist ( B) + 1, dist (C ) + 3}

to D is via B or C
If we calculate from left to right, when we get to

any node v, we have all the information needed
to calculate dist(v)
Example 1: shortest path
V: set of nodes
E: set of edges
l(u,v): the weight of edge (u,v)
initialize all dist(∙) values to ∞

dist(s)= 0
for each v ϵ V\{s}, in linearized order:
dist(v)=min(u,v)ϵE{dist(u)+l(u,v)}
the algorithm solves subproblems, {dist(u): u ϵ V}. We start with the

smallest subproblem and iteratively solve larger subproblems
in realistic cases, such an acyclic graph is implicitly given: nodes

represent subproblems to be solved, and edges represent their
interdependencies (e.g., to solve subproblem B we first have to solve
subproblem A).
Example 2: find min. number of coins that add to sum S
 Given: N coins, value (V1,...,Vn)

 Find the minimum number of coins the sum of which is S
(we can use as many coins of one type as we want)
 Sub-solution (state): a solution for sum i<=S
 to find the sub-solution for i we use the sub-solution for j (where
j<i).
 for each coin j, Vj<=i, look at the minimum number of coins
found for the (i-Vj) sum (we have already found it previously).
Let this be m. If m+1 is less than the minimum number of coins
already found for current sum i, then we write the new result for
it.
source: https://www.topcoder.com/community/data-science/data-science-tutorials/dynamic-programming-from-novice-to-advanced/
 Coin values: 1, 3, 5
 Sum S = 11
Pseudokod:
Set Min[i] equal to Infinity for all of i

Min[0]=0
For i = 1 to S
For j = 0 to N - 1
If (Vj<=i AND Min[i-Vj]+1<Min[i])
Then Min[i]=Min[i-Vj]+1
Output Min[S]
Sum Number Coin value added

of coins to a smaller sum
to obtain this sum
(it is displayed in
brackets)
0 0 -
1 1 1(0)
2 2 1(1)
3 1 3(0)
4 2 1(3)
5 1 5(0)
calculate the 6 2 1(5)
remaining rows
...
...
...
on your own ☺
Example 3: Knapsack problem
n=total number of objects

V[1,...,n]: value of objects 1 to n global
W[1,…,n]: weight of objects 1 do n variables
C= knapsack capacity
Define the procedure K(c, i): returns the maximum value that can
be obtained with capacity c and considering only objects from i to
n.
Example:
V=[5,2,4,6,12,3] We would like to
W=[4,1,3,7,8, 2] find: K(13,1)
C=13, n=6
1st object chosen, reduce
1st object, not chosen continue by remaining available capacity by
considering the 2.nd object K(13,1) the weight of the 1st object and
continue
2nd object not K(13,2) 2nd object K(9,2)

chosen chosen
K(13,3) K(12,3) K(9,3) K(8,3)
K(13,4) K(10,4) K(12,4) K(9,4) K(9,4) K(6,4) K(8,4) K(5,4)
- we notice that procedure calls are repeated→ once a sub-solution

is found, we do not need to solve the given subproblem again !
- in each iteration we solve a new and independent subproblem of
the same type as a previous subproblem
Pseudocode:
K(c,i){
if i>n return 0
if c<W[i]
return K(c, i+1)
else
return max{K(c, i+1), V[i]+K(c-W[i], i+1)}
Additionally: once a subproblem has been solved, the solution is stored and can be looked
up if we again come to that subproblem
Example 4: Stagecoach problem
D 2
7
B G
6 4 3
1 6
4
A E 3 I
2 3 5
2
C 3 H
4
1
F
Decision 1 Decision 2 Decision 3 Decision 4

Possible transitions between stages
B C
Decision 1
A 1 2
D E F
B 7 6 4 Decision 2
C 3 2 4
G H
D 2 4
E 6 3
Decision 3
F 3 1
I
G 3
Decision 4
H 5
Formulating the problem
 We have 4 decisions (steps, phases)
 Let xn, n=1…4, be the destination in the nth decision

 A path is described by A→x1 →x2 →x3 →x4=I
Example: for path A→C→F→H→I: x1 = C; x2 = F; x3 = H; x4 = I
 Let fn(s,xn) represent the shortest path from a node s to

the end node (I) if the path traverses node xn (assuming
we arrived to s in the (n-1)st step), i.e. if our nth decision
is to go to node xn
f n ( s, xn ) = cost of link ( s, xn ) in graph + shortest path xn to I =
d ( s, xn ) + min f n +1 ( xn , xn +1 )
xn+1
Formulating the problem
 Let fn*(s) and xn* represent:

f n ( s) = min f n ( s, xn ) = f n ( s, xn )
xn
 then:
f n ( s, xn ) = cost of link ( s, xn ) in graph + shortest path xn to I =

d ( s, xn ) + min f n +1 ( xn , xn +1 ) = d ( s, xn ) + f n +1 ( xn )
xn+1
 We have a total of 4 decisions, i.e. x4=I: f5(I)=0
 Objective: find f1*(A) and the corresponding path

 DP work “bottom up”: starts from the smallest sub-
problem and then expands:
f4*(s) → f3*(s) → f2*(s) → f1*(s)
Example
D 2
7
B G
6 4 3
1 6
4
A E 3 I
2 3 5
2
C 3 H
4
1
F
Decision 4
Example solution procedure
 If we are at the 4th decision, the path is uniquely

defined by the state (node) s we are in since
we know that we will go from s→I, i.e. x4=I
f 4 ( s) = min f 4 (s, x4 ) = f 4 ( s, I ) = d ( s, I ) + f 5 ( I ) = d ( s, I )
x4
0
G
Decision n=4 3
s f4*(s) x4*
G 3 I
I
H 5 I
5
Decision 4
H
Example
D 2
7
B G
6 4 3
1 6
4
A E 3 I
2 3 5
2
C 3 H
4
1
F
Decision 3
 Decision n=3, s=D

D 2 f 4 (G ) = 3

f 3 ( D, x3 ) = d ( D, x3 ) + f 4 ( x3 ) G
x3 can be G or H : 4

f 3 ( D, G ) = d ( D, G ) + f 4 (G ) = 2 + 3 = 5

f 3 ( D, H ) = d ( D, H ) + f 4 ( H ) = 4 + 5 = 9
f 4 ( H ) = 5
f 3 ( D ) = min f 3 ( D, x3 ) = min( 5, 9) = 5 H
x3
x3 = G
Part of Decision 3
Decision n=3
f3(s,x3)=d(s,x3)+f4*(x3)
s f3*(s) x3*
f3(s, x3=G) f3(s, x3=H)
x3=G x3=H
D 5 9 5 G
 Decision n=3, s=E

f 4 (G ) = 3

f 3 ( E , x3 ) = d ( E , x3 ) + f 4 ( x3 ) G
x3 can be G or H : 6
E

f 3 ( E , G ) = d ( E , G ) + f 4 (G ) = 6 + 3 = 9 3

f3 ( E, H ) = d ( E, H ) + f 4 (H ) = 3 + 5 = 8
f 4 ( H ) = 5
f 3 ( E ) = min f 3 ( E , x3 ) = min( 9, 8) = 8 H
x3
x3 = H
Part of Decision 3
Decision n=3
f3(s,x3)=d(s,x3)+f4*(x3)
s f3*(s) x3*
f3(s, x3=G) f3(s, x3=H)
x3=G x3=H
D 5 9 5 G
E 9 8 8 H
 Decision n=3, s=F

f 4 (G ) = 3

f 3 ( F , x3 ) = d ( F , x3 ) + f 4 ( x3 ) G
x3 can be G or H :

f 3 ( F , G ) = d ( F , G ) + f 4 (G ) = 3 + 3 = 6

f3 ( F , H ) = d ( F , H ) + f 4 (H ) = 1 + 5 = 6
f 4 ( H ) = 5
3
f 3 ( F ) = min f 3 ( F , x3 ) = min( 6, 6) = 6 H
x3
1
F
x3 = G or H
Part of Decision 3
Decision n=3
f3(s,x3)=d(s,x3)+f4*(x3)
s f3*(s) x3*
f3(s, x3=G) f3(s, x3=H)
x3=G x3=H
D 5 9 5 G
E 9 8 8 H
F 6 6 6 G or H
D 2
7
B G
6 4 3
1 6
4
A E 3 I
2 3 5
2
C 3 H
4
1
F
Decision 2
 Decision n=2, s=B f 3 ( D) = 5


f 2 ( B , x2 ) = d ( B , x2 ) + f 3 ( x2 ) D
7
x2 can be D, E or F : B
6
 f 3 ( E ) = 8
f 2 ( B, D ) = d ( B, D) + f 3 ( D) = 7 + 5 = 12 4

f 2 ( B, E ) = d ( B, E ) + f 3 ( E ) = 6 + 8 = 14 E

f 2 ( B, F ) = d ( B, F ) + f 3 ( F ) = 4 + 6 = 10
 f 3 ( F ) = 6
f ( B ) = min f 2 ( B, x2 ) = min(12, 14, 10) = 10
2
x2
x2 = F F
Part of Decision 2
Decision n=2
f2(s,x2)=d(s,x2)+f3*(x2)
s f2*(s) x2*
f2(s, x2=D) f2(s, x2=E) f2(s, x2=F)
x2=D x2=E x2=F
B 12 14 10 10 F
University of Zagreb, Croatia
f 3 ( D) = 5
 Decision n=2, s=C D

f 2 (C , x2 ) = d (C , x2 ) + f 3 ( x2 )
x2 can be D, E or F :
f 3 ( E ) = 8

f 2 (C , D) = d (C , D ) + f 3 ( D) = 3 + 5 = 8 E

f 2 (C , E ) = d (C , E ) + f 3 ( E ) = 2 + 8 = 10 3
 2
f 2 (C , F ) = d (C , F ) + f 3 ( F ) = 4 + 6 = 10
C
4 f 3 ( F ) = 6
f 2 (C ) = min f 2 (C , x2 ) = min( 8, 10, 10) = 8 F
x2
x2 = D
Part of Decision 2
Decision n=2
f2(s,x2)=d(s,x2)+f3*(x2)
s f2*(s) x2*
f2(s, x2=D) f2(s, x2=E) f2(s, x2=F)
x2=D x2=E x2=F
B 12 14 10 10 F
C 8 10 10 8 D
D 2
7
B G
6 4 3
1 6
4
A E 3 I
2 3 5
2
C 3 H
4
1
F
Decision 1
 Decision n=1, s=A f 2 ( B ) = 10


f1 ( A, x1 ) = d ( A, x1 ) + f 2 ( x1 ) B
x1 can be B or C : 1

f1 ( A, B) = d ( A, B) + f 2 ( B) = 1 + 10 = 11 A

f1 ( A, C ) = d ( A, C ) + f 2 (C ) = 2 + 8 = 10 2 f 2 (C ) = 8
C
f1 ( A) = min f1 ( A, x1 ) = min(11, 10) = 10
x1
x1 = C
Decision 1
Decision n=1
f1(s,x1)=d(s,x1)+f2*(x1)
s f1*(s) x1*
f1(s, x1=B) f1(s, x1=C)
x1=B x1=C
A 11 10 10 C
OPTIMAL SOLUTION PATH LENGTH
Reading the solution
Decision n=1 Decision n=2
s x2*
s x1*
B F
A C C D
Decision n=3
Decision n=4
s x3* s x4*
D G G I
E H H I
F G or H
OPTIMAL SOLUTION: path A→C→D→G→I:
Zavod za telekomunikacije x1 = C; x2 = D; x3 = G; x4 = I 33
DP problems: characteristics
 The problem can be divided into phases or

stages where a decision must be made, i.e. a
sequence of interrelated decisions
 Each stage has a certain number of states
 Optimality principle must hold: in a given state,
the optimal solution of subsequent decisions
depends only on the current state (independent
of previous stages)
 A recursive relationship can be defined which
calculates the optimal solution in stage n given
the optimal solution for stage n+1
DP characteristics
 Finds the OPTIMAL solution (exact)
 The procedure is constructive – moves

backwards through stages until the optimal
solution for all stages is found
Example: Floyd’s All-Pairs Shortest Path algorithm
 Given: directed weighted graph G=(V,E)

 Find: all-pairs shortest paths
 Complexity O(n3)
 Notation:
 N – number of nodes
 c(i,j) – weight of link (i,j)
 D – Distance matrix NxN
 Algorithm
1. For the graph construct NxN matrix D0 = {dij 0}, where dij is the
weight of the direct directed link from i to j. If i and j are
unconnected then dij = ∞. Set k = 0.
2. k = k + 1calculate new matrix Dk according to:
dij k = min{dij k-1, dik k -1+dkj k-1}
1. If k = N then STOP, otherwise go to step (2)
 Notation:
 D0 – distance matrix of direct links
 D1 – shortest path distance matrix of direct links or path via node 1
 D2 – shortest path distance matrix of direct links or path via node 1
or 2
 …
 Dn – shortest path distance matrix of direct links or path via node 1,
2…, n
 After (k-1)st step of the algorithm we have Dk-1= {dijk-1}
which represents the shortest path from i to j which does
not pass through nodes k, k+1, …, n (i.e. can only pass
through nodes 1,2,…, k-1 while i and j can be ≥ k)
 In the kth step, we have Dk-1 and now check whether node
k is on the optimal path from i to j for every i to j in N
 Main idea: each step checks whether the current path

from node i to j is shorter than a path via node k, i.e.,
the sum of the shortest path from i to k and the shortest
path from k to j.
L(i,k)
k
i
L(k,j)
L(i,j) j
L(i,k)+L(k,j) < L(i,j) ??

Pseudocode
Floyd (D0={ dij0}, N)

for k = 1 to N
for i = 1 to N
for j = 1 to N
dijk = min { dijk-1 , dikk-1 + dkjk-1}
end for
end for
end for
return Dn
Example
Example of graph and corresponding matrix D0 = {dij 0}, where

dij is the weight of the direct directed link from i to j. If i and j
are unconnected then dij = ∞
1
4
2 −   5
4 −   
2
5
3 D0 = 
2  − 
2  
 3 2 −
3 4
Example
4 Initial distance matrix D0

1 2 −   5
4 −   
2
5
3 D0 = 
2  − 
 
2  3 2 −
3 4
Distance matrix after 1st iteration (k=1)
dijk = min { dijk-1 , dikk-1 + dkjk-1} −   5
4 −  
9 Updated
D1 = 
Example calculations of dij1
d131 = min{d130 , d110 + d130 } = d130=∞
2  − 7
d241 = min{d240 , d210 + d140 } = min{∞, 4+5} = 9  
 3 2 −
Example
Distance matrix after 1st iteration (k=1)

4
1 2 −   5
4 −  
9
2
5
3 D1 = 
2  − 7
 
3
2
4  3 2 −
Distance matrix after 2nd iteration (k=2)
dijk = min { dijk-1 , dikk-1 + dkjk-1} −   5
4 −  9
D2 =  
d232 = min{d231, d221+d231} = d23 1 2  − 7
d41 2 = min{d41 1, d42 1 +d21 1} = min{∞, 3+4} = 7  
7 3 2 −
Example
Distance matrix after 2nd iteration (k=2)

1
4
2 −   5
4 −  
9
5 D2 = 
2 3 2  − 7
 
2 7 3 2 −
3 4
Distance matrix after 3rd iteration (k=3)
−   5
4 −  9
D3
=  
3 2 2 2
d23 = min{d23 , d23 +d33 } = d23 2
2  − 7
d41 3 = min{d41 2, d43 2 +d31 2} = min{7, 2+2} = 4  
4 3 2 −
Example
4 Distance matrix after 3rd iteration (k=3)

1 2
−   5
4 −  9
5
2 3 D3 = 
2  − 7
 
2 4 3 2 −
3 4
Distance matrix after 4th iteration (k=4)
− 8 7 5
Example calculations of dij4 4 − 11 9 
D4 = 
d13 4 = min{d13 3, d14 3 +d43 3} = min{∞, 5+2} = 7 2 10 − 7
d32 4 = min{d32 3, d34 3 +d42 3} = min{∞, 7+3} = 10  
4 3 2 −
Example
4
1 2
Optimal solution to all-pairs shortest
5 paths
2 3
3
2
4 − 8 7 5
4 − 11 
9
D =
4
2 10 − 7
 
4 3 2 −
References
 F.S. Hillier, G.J. Lieberman, “Introduction to Operations

Research, 7th edition”, Mc-Graw Hill, NY, 2001
 S. Dasgupta, C. Papadimitriou, U. Vazirani , “Algorithms”,
McGraw Hill, 2008

Lecture 3.b - Dynamic Programming

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 3.b - Dynamic Programming

Uploaded by

Copyright:

Available Formats

University of Zagreb, Croatia

Academic year 2020/2021

 A technique to make a sequence of related

e.g., shortest path:

directed acyclic graph linearization

shortest path from S dist ( D) = min{dist ( B) + 1, dist (C ) + 3}

If we calculate from left to right, when we get to

initialize all dist(∙) values to ∞

the algorithm solves subproblems, {dist(u): u ϵ V}. We start with the

in realistic cases, such an acyclic graph is implicitly given: nodes

 Given: N coins, value (V1,...,Vn)

Set Min[i] equal to Infinity for all of i

Sum Number Coin value added

n=total number of objects

2nd object not K(13,2) 2nd object K(9,2)

K(13,4) K(10,4) K(12,4) K(9,4) K(9,4) K(6,4) K(8,4) K(5,4)

- we notice that procedure calls are repeated→ once a sub-solution

Decision 1 Decision 2 Decision 3 Decision 4

 We have 4 decisions (steps, phases)

 Let xn, n=1…4, be the destination in the nth decision

Example: for path A→C→F→H→I: x1 = C; x2 = F; x3 = H; x4 = I

 Let fn(s,xn) represent the shortest path from a node s to

 Let fn*(s) and xn* represent:

 We have a total of 4 decisions, i.e. x4=I: f5(I)=0

 Objective: find f1*(A) and the corresponding path

 If we are at the 4th decision, the path is uniquely

 Decision n=3, s=D

 Decision n=3, s=E

 Decision n=3, s=F

 Decision n=2, s=B f 3 ( D) = 5

 Decision n=1, s=A f 2 ( B ) = 10

OPTIMAL SOLUTION PATH LENGTH

Decision n=1 Decision n=2

 The problem can be divided into phases or

 Finds the OPTIMAL solution (exact)

 The procedure is constructive – moves

 Given: directed weighted graph G=(V,E)

 Main idea: each step checks whether the current path

L(i,k)+L(k,j) < L(i,j) ??

Floyd (D0={ dij0}, N)

dijk = min { dijk-1 , dikk-1 + dkjk-1}

Example of graph and corresponding matrix D0 = {dij 0}, where

4 Initial distance matrix D0

Distance matrix after 1st iteration (k=1)

Distance matrix after 2nd iteration (k=2)

4 Distance matrix after 3rd iteration (k=3)

 F.S. Hillier, G.J. Lieberman, “Introduction to Operations

You might also like

 Let fn(s) and xn represent: