Dynamic Programming

Irham Fadlika ST MT
Introduction
• Proposed by Dr. Richard Bellman and associates (late
1950s)
• Useful techniques in solving variety of problems
• In scheduling power generation systems, Dynamic
Programming (DP) have been developed for :
➢ The economic dispatch of thermal systems
➢ The solution of hydrothermal economic-scheduling
problems
➢ The practical solution of the unit commitment problem
• Cost of transporting
unit shipment from
node A to node N
• Values on the arcs
represent the costs
• The problem is to find
the minimum cost
route from A to N
• Starting at A, the minimum cost path to N is ACEILN.
• Starting at C, the least cost path to N is CEILN.
• Starting at E, the least cost path to N is EILN
• Starting at I, the least cost path to N is ILN.
• Starting at L, the least cost path to N is LN.

• The choice of route is made in sequence

• The optimum sequence is called the optimal policy; any
subsequence is a subpolicy
• The stages I, II, III, IV, V
• Set of choices of nodes
𝑋𝑖
• 𝑉𝑎 𝑋𝑖 , 𝑋𝑖 + 1 represent
the cost of traversing
stage
• 𝑓𝑎 𝑋𝑖 minimum cost for
stages I through a to
arrive at some particular
node 𝑋𝑖
 𝑋0 ∶ 𝐴
𝑋1 ∶ 𝐵, 𝐶, 𝐷
𝑋2 ∶ 𝐸, 𝐹, 𝐺
𝑋3 ∶ 𝐻, 𝐼, 𝐽, 𝐾
𝑋4 ∶ 𝐿, 𝑀
𝑋5 ∶ 𝑁

𝑓𝐼 (𝑋1 ): Minimum cost for the first stage

𝑓1 𝐵 = 𝑉1 𝐴, 𝐵 = 5
𝑓1 𝐶 = 𝑉1 𝐴, 𝐶 = 2
𝑓1 𝐷 = 𝑉1 𝐴, 𝐷 = 3
𝑓𝐼𝐼 (𝑋2 ): Minimum cost for the stage I and II as a function of 𝑋2
𝑓𝐼𝐼 𝐸 = min 𝑓𝐼 𝑋1 + 𝑉𝐼𝐼 𝑋𝐼 , 𝐸
𝑋1
= 𝑚𝑖𝑛 5 + 11, 2 + 8, 3 + ∞ = 10
𝑋𝐼 = 𝐷 =𝐶 = 𝐷 𝑋𝐼 = 𝐶
The cost is infinite for node D since there is no path from D to E:
𝑓𝐼𝐼 𝑓 = min 𝑓𝐼 𝑋1 + 𝑉𝐼𝐼 𝑋𝐼 , 𝑓 = min ∞, 6,9 = 6, 𝑋𝐼 = 𝐶
𝑋1
𝑓𝐼𝐼 𝐺 = min 𝑓𝐼 𝑋1 + 𝑉𝐼𝐼 𝑋𝐼 , 𝐺 = min ∞, 11,9 = 9, 𝑋𝐼 = 𝐷
𝑋1
𝑓𝐼𝐼𝐼 (𝑋3 ): Minimum cost for the stage I, II, and III as a function of 𝑋3
𝑓𝐼𝐼𝐼 𝐻 = min 𝑓𝐼𝐼 𝑋2 + 𝑉𝐼𝐼𝐼 𝑋2 , 𝐻 = min 13, 14, ∞ , 𝑋2 = 𝐸
𝑋2
In general,
𝑓𝐼𝐼𝐼 𝑋3 = min 𝑓𝐼𝐼 𝑋2 + 𝑉𝐼𝐼𝐼 𝑋2 , 𝑋3
𝑋2
𝑓𝐼𝑉 (𝑋4 ): Minimum cost for the stage I through IV as a function of
𝑋4
𝑓𝐼𝑉 𝑋4 = min 𝑓𝐼𝐼𝐼 𝑋3 + 𝑉𝐼𝑉 𝑋3 , 𝑉4
𝑋3

𝑓𝐼𝑉 𝐿 = min 13 + 9, 12 + 3, 11 + 7, 13 + ∞ = 15, 𝑋3 = 𝐼

𝑋3 = 𝐻 = 𝐼 = 𝐽 = 𝐾
𝑓𝐼𝑉 𝑀 = min 13 + ∞, 12 + 6, 11 + 8, 13 + 5 = 18, 𝑋3 = 𝐼 𝑜𝑟 𝐾
𝑋3 = 𝐻 = 𝐼 = 𝐽 = 𝐾
𝑓𝑉 (𝑋5 ): Minimum cost for the stage I through V as a function of 𝑋5
𝑓𝑉 𝑁 = min 𝑓𝐼𝑉 𝑋4 + 𝑉𝑉 𝑋4 , 𝑋5
𝑋4
= min 15 + 4, 18 + 3 = 19, 𝑋4 = 𝐿
𝑋4 = 𝐿 = 𝑀
Tracing back, the path of minimum cost is found as follows:
Allocation Problem Table 3.4 lists the profits to
be made in each of four
ventures as a function of the
investment in the particular
venture. Given a limited
amount of money to allocate,
the problem is to find the
optimal investment allocation.
The only restriction is that
investments must be made in
integer amounts. For
instance, if one had 10 units
to invest and the policy were
to put 3 in I, 1 in II, 5 in
III, and 1 in IV, then ….
 𝑃𝑟𝑜𝑓𝑖𝑡 = 0.65 + 0.25 + 0.65 + 0.20 = 1.75
The problem is to find an allocation policy that yields the
maximum profit. Let
𝑋1 , 𝑋2 , 𝑋3 , 𝑋4 be investments in I through IV
𝑉 𝑋1 , 𝑉 𝑋2 , 𝑉 𝑋3 , 𝑉 𝑋4 be profits
𝑋1 + 𝑋2 + 𝑋3 + 𝑋4 = 10 is the constraint; that is
10 units must be invested
To transform this into a multistage problem, let the stages be
𝑋1 , 𝑈1 , 𝑈2 , 𝐴
where
𝑈1 = 𝑋1 + 𝑋2 𝑈1 ≤ 𝐴 𝑈2 ≤ 𝐴
𝑈2 = 𝑈1 + 𝑋3 𝐴 = 0, 1, 2, 3, … , 10
𝐴 = 𝑈2 + 𝑋4
The total profit is
𝑓 𝑋1 , 𝑋2 , 𝑋3 , 𝑋4 = 𝑉1 𝑋1 + 𝑉2 𝑋2 + 𝑉3 𝑋3 + 𝑉4 𝑋4
which can be written
𝑓 𝑋1 , 𝑈1 , 𝑈2 , 𝐴 = 𝑉1 𝑋1 + 𝑉2 𝑈1 − 𝑋1 + 𝑉3 𝑈2 − 𝑈1 + 𝑉4 𝐴 − 𝑈2
At the second stage, we can compute
𝑓2 𝑈1 = max 𝑉1 𝑋1 + 𝑉2 𝑈1 − 𝑋1
𝑋1 =0,1,…,𝑈1
Next, at the third stage,
𝑓3 𝑈2 = max 𝑓2 𝑈1 + 𝑉3 𝑈2 − 𝑈1
𝑋1 =0,1,…,𝑈1
finally, the last stage is
𝑓4 𝐴 = max 𝑓3 𝑈2 + 𝑉4 𝐴 − 𝑈2
𝑈2