80-680-04 Dynamic Optimization in Management

Mid-term exam - Fall 2011 - solution

Problem 1 (20 points)
A reservoir stores water for three competing uses: electricity production,
farmland irrigation, and recreation. At the beginning of each month, a central
management agency decides on the amount or water released from the reservoir
for the production of electricity and for irrigation purposes. The revenue from
selling one unit of water for electricity production is /, the revenue from selling
one unit of water for farmland irrigation is ). The revenue from recreation is
constant if the level of the reservoir is suciently high. Write a dynamic program
maximizing the total expected annual revenues from the reservoir, taking into
account the following information.
1. The basic demand for electricity must be met, otherwise the agency has
to pay a large penalty 1. A quantity of
|
units of water is required to
satisfy the basic demand in month t.
2. The amount of water stored in the reservoir cannot be larger than ' or
smaller than : at any time.
3. The hydroplant cannot process more than Q units of water per month.
4. At least G units of water must be provided each year for farmland irriga-
tion.
5. The water contained in the reservoir is renewed by natural inows. Inows
are uncertain, but can be forecasted with accuracy one month in advance.
The probability density of receiving an inow of . units of water during
month
|
is independent of what happened in the previous months and given
by 0
|
(.).
6. When the amount of water stored in the reservoir is smaller than 1, there
is a loss of revenue from recreational activities. This loss is equal to
r(1r), where r denotes the amount of water stored in the reservoir.
Denote:
r : amount of water stored in the reservoir at the beginning of the month
(state variable)
j : total amount of water already provided for farmland irrigation (state
variable)
. : inow forecast for the coming month (state variable)
n : amount of water released for the production of electricity (decision vari-
able)
n : amount of water released for farmland irrigation (decision variable)
t : month index (stage variable)
q
|
(r, j, .) : maximum total revenue from the operation of the reservoir from
month t until the end of the year if at the beginning of month t there are r
units of water in the reservoir, j units of water already provided in the previous
Page 1 of 5.
80-680-04 Dynamic Optimization in Management
Mid-term exam - Fall 2011 - solution
months for farmland irrigation, and the forecast for the coming month is that
the inow will be ..
q
|
(r, j, .) = max
(u,u)1(r,,:)
{/n + )n 11
u<j

r(1r)
+
+
R

0
q
|+1
(r + . n n, j + n, /)0
|
(/)d/

q
13
(r, j, .) =

if j < G
cr otherwise
where c is a suitably chosen constant reecting the value of the water stored
in the reservoir for future uses, and where
1(r, j, .) = {(n, n) : 0 n Q, n 0, : r + . n n '} .
Problem 2 (30 points)
Consider a machine that can fail with a probability j
n
, where : is the number
of units produced by the machine since the last maintenance operation. The cost
of a maintenance operation is c. A total of units have to be produced using
this machine; accordingly, if the machine fails before the units are produced,
it has to be replaced at a cost r c. Assume that j
0
= 0 and that j
n
j
n+1
for 0 : 1.
Denote:
: : the number of units produced by the machine since the last maintenance
or replacement.
/ : the number of units that remain to be produced to attain

|
(:) : the minimum expected cost of maintenance and replacement if there
are / units to produce and : units where produced since the last maintenance
or replacement.
A dynamic programming model nding the optimal maintenance schedule
of this machine over an innite horizon is given by:

|
(:) = min{c +
|1
(1); j
n
(r +
|1
(1)) + (1 j
n
)
|1
(: + 1)} , : 1

0
(:) = 0, : 0.
1. Complete the above model by computing
|
(0) and explain your result.

|
(0) = min{c +
|1
(1);
|1
(1)} =
|1
(1). Since a new machine can-
not fail before producing at least one unit, it is never optimal to do a
maintenance operation on a new machine, and the total cost of a new
machine is equivalent to the total cost of the same machine after the rst
unit has been produced.
2. Show that
|
(:) is non-decreasing in / for all : 0.
At / = 1 :
1
(:) = min{c; j
n
r} 0 =
0
(:).
We now assume that
|
(:)
|1
(:) and show that this implies
|+1
(:)

|
(:).

|+1
(:) = min{c +
|
(1); j
n
(r +
|
(1)) + (1 j
n
)
|
(: + 1)}
min{c +
|1
(1); j
n
(r +
|1
(1)) + (1 j
n
)
|1
(: + 1)} =
|
(:). By
induction, the proposition is true for all / 0.
Page 2 of 5.
80-680-04 Dynamic Optimization in Management
Mid-term exam - Fall 2011 - solution
3. Show that
|
(:) is non-decreasing in : for all / 0. What can be con-
cluded about the form of an optimal strategy?
At / = 0 :
0
(:) is a constant function, therefore non-decreasing in :
We now assume that
|1
(: + 1)
|1
(:) and show that this implies

|
(:)
|
(: 1).

|
(:) = min{c +
|1
(1); j
n
(r +
|1
(1)) + (1 j
n
)
|1
(: + 1)}
where
j
n
(r +
|1
(1))+(1 j
n
)
|1
(:+1) = j
n
(r +
|1
(1)
|1
(: + 1))+

|1
(: + 1)
and (r +
|1
(1)
|1
(: + 1))
(r +
|1
(1) c
|2
(1))
= (r c +
|1
(1)
|2
(1)) 0
As a consequence,
j
n
(r +
|1
(1)
|1
(: + 1)) +
|1
(: + 1)
j
n1
(r +
|1
(1)
|1
(: + 1)) +
|1
(: + 1)
= j
n1
(r +
|1
(1)) + (1 j
n1
)
|1
(: + 1)
and
|
(:) = min{c +
|1
(1); j
n
(r +
|1
(1)) + (1 j
n
)
|1
(: + 1)}
min{c +
|1
(1); j
n1
(r +
|1
(1)) + (1 j
n1
)
|1
(: + 1)}
min{c +
|1
(1); j
n1
(r +
|1
(1)) + (1 j
n1
)
|1
(:)} =
|
(: 1)
and the proposition is true for all / 0.
From these results, we know that if at stage /,

|
(j) = c +
|1
(1), then for all : j, we have
c +
|1
(1)
|
(:)
|
(j) = c +
|1
(1) and it is better to do a
maintenance operation for all : j.
4. Suggest an algorithm to solve this problem numerically and evaluate the
number of operations required to obtain the solution.
The solution can be obtained by backward induction from
0
. At each
step / = 1, ..., compute
|
(:) for : = 1, ..., , stopping as soon as the
threshold is obtained. There is a maximum of
2
operations.
Problem 3 (20 points)
You want to share a total amount between persons having the same log-
arithmic utility n(r) = lnr for receiving an amount r. Show, using a Dynamic
Programming argument, that the best way to distribute the amount among
these persons in order to maximize the total utility of the group is to give each
one an equal share.
This is a resource allocation problem. Denote
Page 3 of 5.
80-680-04 Dynamic Optimization in Management
Mid-term exam - Fall 2011 - solution
: : the number of persons remaining
a : the amount not yet allocated
q
n
(a) : the maximal total utility of a group of : persons when the total
amount to allocate between them is a.
q
n
(a) = max
0ro
{lnr + q
n1
(a r)}
q
1
(a) = lna
We want to show that q
n
(a) = :ln(
o
n
).
At : = 1, q
1
(a) = lna = ln(
o
1
)
Assume that q
n
(a) = :ln(
o
n
), we will show that this implies q
n+1
(a) =
(: + 1) ln(
o
n+1
)
q
n+1
(a) = max
0ro

lnr + :ln(
or
n
)

Dierentiating with respect to r yields

J
Jr

lnr + :ln(
or
n
)

=
1
r(or)
(r a + :r), r(a r) 6= 0
J
Jr

1
r(or)
(r a + :r)

=
1
r
2
(or)
2

:r
2
+ (a r)
2

< 0. This is a
concave function.
(r a + :r) = 0, Solution is: r

=
o
n+1
, which is interior if a 6= 0.
Replacing, we nd q
n+1
(a) = ln
o
n+1
+:ln(
o
n+1
) = (: + 1) ln(
o
n+1
), and the
proposition is true for all :, provided a 6= 0.
Problem 4 (30 points)
Consider a rm selling a product on a market where the selling price during
period / is given by
j
|
= cj
|1
+ -
|
and where c is a positive constant and the -
|
are discrete independent random
variables of mean 0 and known stationnary distribution
1
, 1 [-
|
= 0
I
] = r
I
, i =
1, ...:.
Denote:
: quantity produced during the current period
: : quantity in stock at the beginning of the current period
n : quantity in stock at the end of the current period
j : price during the current period
, : periodic discount factor
(:, j) : maximal total expected discounted prot over an innite horizon if
the quantity in stock is : and the current price is j.
Assume that the production cost in a given period is given by the function
c() and the holding cost by the function /(n), where both c and / are increasing
concave functions.
1. Write a dynamic program to maximize the total expected discounted prot
of the rm over an innite horizon, by deciding in each period how much
to produce, to store, and to sell. Assume innite demand, that is, any
quantity put on the market by the rm is sold at price j.
1
There was an error in the question, the values taken by

should not have been 1 to ,

but rather

where goes from 1 to .

Page 4 of 5.
80-680-04 Dynamic Optimization in Management
Mid-term exam - Fall 2011 - solution
(:, j) = max
(j,u)1(s)
{j (: + n) c() /(n) + ,
P
n
I=1
(n, cj + 0
I
)r
I
}
1(:) = {(, n) : 0, n : + }
2. Assuming an interior solution, show that the value function can be de-
composed as
(:, j) = j: + n(j) (1)
and write down a functional equation characterizing the function n(j).
(:, j) = j:+max
j,u1(s)
{j jn c() /(n) + ,
P
n
I=1
(n, cj + 0
I
)r
I
}
Assume (:, j) = j: + n(j), we then have
(:, j) = j: + max
j,u1(s)
{j jn c() /(n)
+,
P
n
I=1
(n(cj + 0
I
) + n(cj + 0
I
)) r
I
}
=j:+max
j,u1(s)
{j ( n) c() /(n) + ,ncj
P
n
I=1
r
I
+ ,n
P
n
I=1
0
I
r
I
+,
P
n
I=1
n(cj + 0
I
)r
I
}
=j:+max
j,u1(s)
{j ( n) c() /(n) + ,ncj + ,
P
n
I=1
n(cj + 0
I
)r
I
}
= j: + n(j)
where
n(j) = max
j,u1(s)
{j ( n) c() /(n) + ,ncj + ,
P
n
I=1
n(cj + 0
I
)r
I
}
3. Write the rst order conditions characterizing an optimal strategy of the
rm.
Assuming an interior solution, since c and / are concave functions,the
optimal strategy for the rm is independent of : and satises:
c
0
() = j
/
0
(n) = j (c, 1). For the inventory to be interesting, you need c, 1.
4. Give an interpretation of the function n(j) and of the decomposition per-
formed in Equation (1). Discuss the implication of Expression (1) in terms
of computational burden.
The value function is decomposed into two parts, the rst part is the value
of the stock at the current selling price, and the function n(j) represents
the value of the production and storage installation, which depends on
the price level. Expression (1) allows to reduce the dimensionality of
the problem, since optimal strategies are independent of : and since it is
sucient to compute the function n(j) to characterize the value function.
Page 5 of 5.