FURTHER MATHEMATICS FOR ECONOMIC ANALYSIS SECOND EDITION KNUT SYDSATER PETER HAMMOND ATLE SEIERSTAD ARNE STROMCONTROL THEORY WITH MANY. VARIABLES To be sure, mathematics can be extended to any branch of knowledge, including economics, provided the concepts are so clearly defined as to permit accurate symbolic representation. That is only another way of saying that in some branches of discourse it is desirable to know what you are talking about. —D. MacDouglas (1956) his chapter begins by extending the optimal control theory developed in the previous chapter +o problems with several state and control variables. in Section 10.1 the main emphasis is on appropriate generalizations of results from Chapter 9. The explanations are brief because the essential motivation was given in Chapter 9. However, we give a proof of the Arrow sufficiency theorem in the case of several state and contro! variables. Section 10.2 contains examples illustrating the theory. Section 10.3 extends the infinite horizon theory of Section 9.11. In fact, the majority of the control models that appear in economics literature assume an infinite horizon. A good treatment of infinite horizon control theory with a large number of economic applications can be found in Weitzman (2003). if neither the Mangasarian nor the Arrow concavity conditions are satisfied, then we need some assurance that the control problem has a solution. Even if the maximum principle gener- ates a unique candidate for an optimal solution, there might still be no optimal solution. This difficulty is discussed in Section 10.4. The Filippov-Cesari theorem gives sufficient conditions {or existence of an optimal control in a rather general control problem. The section ends with a formulation of precise sensitivity results in control theory. These are seldom spelled out except in specialized literature. Section 10.5 offers a heuristic proof of the maximum principle, which, at least in the case of a free end, is close to a proper proof. Proving the maximum principle is much harder when ‘there are terminal constraints on the state variables. Section 10.6 has a short discussion of control problems with mixed constraints of the tyoe h(t, x, u) = 0, and Section 10.7 considers pure state constraints that can be written in the vector form h(t, x) > 0. In fact, many of the control problems that economists have considered involve mixed and/or pure constraints. Finally, Section 10.8 briefly discusses some generalizations.360 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES 10.1 Several Control and State Variables Chapter 9 studied control problems with only one state and one control variable. In this section most of the results from Chapter 9 are generalized to control problems with an arbitrary number of state and control variables. The standard end-constrained problem with n state variables and r controls is to find a pair of vector functions (x(t), u(t) = (x1 (0), +4 %n(t), ur (t), «+5 tr(A)) defined on the fixed time interval [to, ty] in order to maximize 1 [ f(t, x(f), u(t) dt qa) 'o subject to the dynamic constraints dnt 29 _ aie.xt, 400) a < or a(t, (9, w(t) Q) xn(t) Fy Belts x(0), 00) the initial conditions x(t) =, i cn (x° = (x9,...,29) isa given pointinR") 3) the terminal conditions @ x(n) =x}, () xh) 2x), o (©) x(t) free, and the control variable restrictions u(t) =(u(),-..,ur) EU CR’ (Uisagiven set in R’) (5) In (2) the system of differential equations is also written as a vector differential equation, where = (dx /dt, dxy/dt, ...,d%n/dt), and g = (g1, 825+» Bn) i8 a vector function. These n equations describe the rate of growth of each state variable. In (4), the numbers fy and x} are fixed, The pair (x(0), u(¢)) is admissible if u(t), ..., y(t) ate all piecewise continuous, u(t) takes values in U and x(t) = (x1(t),... Xn(6)) is the corresponding continuous and piecewise differentiable vector function that satisfies (2), (3), and (4). The functions f and fis--+» Se and their partial derivatives w.rt. the variables xj are assumed to be continuous in all the n +r +1 variables. By analogy with the single variable problem in Section 9.4, associate an adjoint func- tion pj(t) with each state variable x;(¢). The Hamiltonian H = H(t,x,u,p), with p = (Diy ++ Pa)eis then defined by H(t,X,0, p) = pof(t,¥,w) + P-REXw) = pofhxw) + pigixw 6) islSECTION 10.1 / SEVERAL CONTROL AND STATE VARIABLES 361 The Maximum Principle ‘The maximum principle for the problem gives necessary conditions for optimality, but the conditions are far from sufficient. For a proof see Fleming and Rishel (1975). SSHEOREM 10.771 (THE MAXIMUM PRINCIPLE? STANDARD END CONSTRAINTS) Suppose that (x*(¢), u*(#)) is an optimal pair for the standard end-constrained problem (1)-(5). Then there exist a constant po, with po = 0 or po = 1, and a continuous and piecewise differentiable adjoint vector function p(t) = (pilt),-«.5 pa (@)), such that forall tin ff, 1], one has (po, p(t)) # (0, 0), and: (A) The control function u*(t) maximizes the Hamiltonian H(t, x(t), u, p()) foru € U, ie. H(t,x"(0),u, p) < (t,x, 0"), pO) forall win U a (B) Wherever u*(1) is continuous, the adjoint functions satisfy GH x), uO, PO) Ox; 7 BO =— sont (8) (© Corresponding to the terminal conditions (a), (b), and (c) in (4), one has the respective transversality conditions (a’) pi(t)) no condition, () pil) = 0 (pilts) = (c’) pil) =0, if x7(q) > x), o) NOTE 1 Forany inequality in (4)(b) that is reversed, the corresponding inequality in (9)(b’) is reversed as well. NOTE 2 One can show the following additional properties (a) The Hamiltonian H(t, x*(0),u*(¢), p(t) _ is continuous for all ¢ (10) (b) If the partial derivatives af/91 and ag;/¢, i = 1, ...,n, exist and are continuous, then aH), uO, PO) or ay u(t) continuous for allt (12)362 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES NOTE 3. Suppose that the terminal condition is that all state variables x(t) are free, for i =1,....1. In this case, (9)(c') yields p(t) = 0, and then po = 1 NOTE 4 The adjoint variables in Theorem 10.1.1 can be given price interpretations corres- ponding to those in Section 9.6 for the case n = r = 1. Indeed, let wls(al,....x}) and define the value function V associated with the standard problem as f Vea, x!,t0,n1) =max [J f@,x(O.ue)at = (xt), we) admissible} (13) to ‘Then, under precise assumptions stated in Section 10.4, for i = 1, 2,..-.M ave av " ave. agp PO Pad —Piltt), am (to), ORT (14) where H™ denotes the Hamiltonian evaluated along the optimal path. Sufficient Conditions The simplest general sufficiency theorem is the following: THEOREM 10.1.2 (MANGASARIAN) Consider the standard end-constrained problem (1)-(5) with U convex, and sup- pose thatthe partial derivatives af /8u and 8g; /8u; all exist and are continuous. Ifthe pair (x*(t), u*(0)), together with a continuous and piecewise differentiable adjoint function p(t), satisfies all the conditions in Theorem 10.1.1 with po = 1, and if H(t,x, u, p() is concave in (x, u) for all in [0,4] (as) then (x*(1), u*(0) solves the problem. If the function H(t, x, u, p(t)) is strictly concave in (x, w), then (x"(t), u*(#)) is the unique solution to the problem. NOTE 5 Because a sum of concave functions is concave, the concavity condition (15) is satisfied if f and pygi,...5 Pngn are all concave in (x, U). At this point the reader might want to study Example 10.2.1 and then do Problem 10.2.1 The proof of ‘Theorem 10.1.2 very similar to the proof of Theorem 9.7.1, so we skip it. Instead, we take a closer look at a generalization of Mangasarian’s theorem due to Arrow (see Arrow and Kurz (1970)). Define the maximized Hamiltonian as AG, x, p) = max H(t, x, WP) (16) assuming that the maximum value is attained. Then the appropriate generalization of Theorem 9.7.2 is this:SECTION 10.1 / SEVERAL CONTROL AND STATE VARIABLES 363 THEOREM 10.1.3 (ARROW) Suppose that (x*(), u*(¢)) is an admissible pair in the standard end-constrained problem (1)~(5) that, together with the continuous and piecewise differentiable adjoint (vector) function p(t), satisfies all the conditions in Theorem 10.1.1 with po = 1. Suppose further that A(t, x, p(s) is concave in x for all t in [9,41] 0) Then (x*(t), u*(t)) solves the problem. Proof: Let (x,u) = (x(t), u(t)) be an arbitrary admissible pair. We must show that Dy = J F0.x°O,u'(O)dt — ff! F,x(e),u()) dt > 0. Let us simplify the notation by letting f* Mnote F(.x°(0), ur), F denote f(t,x(0),w(d), H* denote Hit, x*@),u"(0), pO), ete. As in the proof of Theorem 9.7.1, its easy to see that o.= fur macs [99 -e@-verar @ 7 5 Integration by parts yields a 1 [vo GO) - FO) dt = [00-00 -x ear * po) (w(t) = x*(0) — 7 ty Gi) 2- | p0-w erat To explain the last inequality, note first that because x(iq) = x*(fo) we get PE) &@) = 8°) = Pl) + (A) — (4) = YL rrentsy — Ht) ii) 0 ial ‘We claim that this sum is > 0, which will imply the inequality in (fi). In fact, for i = 1,2,..-54, we have x;(t1) = x}(t) = x}, so the corresponding terms are 0. Also, for i = m+ 1,....7, the corresponding terms in the sum in (iii) are 0 because by (9)(c'), pi(th) = 0.Ifi =I + 1,...,m and x3(q) > x}, the corresponding terms are 0 because by (9)(b'), p(t) = 0. Finally, if x7 (r then x(t;) — xf(fy) > O and, by (9)(b'), pi(ti) = 0, so the corresponding terms are > 0. Allin all, this proves that the sum in Gi) is=0. To proceed, note that by the definition of 7, =s. w= ad Hel @v) It follows from (i){iv) that Dy [a = A-pe)- a -xoylat o ‘0 But (8) implies that —(¢) is the (partial) gradient vector VxH", which must equal VA" by the envelope theorem (Theorem 3.10.2). Because H is concave wr. x, it follows from Theorem 2.4.1 that H-H <-pO@G)-x'O), of AY-H>pOEO-*O) ‘This implies that the integral on the right-hand side of (v) is nonnegative for all in [fo,f1], 80 Dy 2 0 ‘as required. .364 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES NOTE 6 The result in Problem 3 shows that condition (15) implies (17). Thus Theorem 10.1.3 generalizes Theorem 10.1.2. NOTE 7 Suppose that in the standard end-constrained problem (1)~(5) one requires that x(t) € A() forall r, where A(f) for each t is a given convex set in R". Suppose also that x" (1) is an interior point of A(t) for each t. The conclusion in Theorem 10.1.3 is then valid, and x + A(t, x, p(2)) need only be concave in the set A(?). Variable Final Time Consider problem (1)-(5) with variable final time t. The problem is among all control functions u(t) that during the time interval [to, 11] steer the system from x° along a time path satisfying (2) to a point where the boundary conditions in (4) are satisfied, to find one which maximizes the integral in (1). The time fy at which the process stops is not fixed, as the different admissible control functions can be defined on different time intervals. Theorem 9.8.1 has then the following immediate generalization: FUEOREM 10/104 THE MAXIMUM PRINCIPLE WITH A’ VARIABLE FINAL TIME) Suppose that (x*(f), u*(0)) is an admissible pair defined on [fo, tf] that solves problem (1)=(5) with ty free (t; € (fo, 00)). Then all the conditions in the max- imum principle (Theorem 10.1.1) are satisfied on [fo ff] and, in addition, HG. x(t), w'(@), pe) = 0 (18) For a proof, see Hestenes (1966). Neither the Mangasarian nor the Arrow theorem applies to variable final time problems. For sufficiency results, see Seierstad and Sydseeter (1987). Current Value Formulations with Scrap Values The theorems in Section 9.9 on current value formulations of optimal control problems with scrap value functions can easily be generalized to the following problem involving several state and control variables: max { SGX, wear + sexae"" J, X() =e(,x,0), x) =x? (19) acven’ UJ, @ x)= i (b) x(n) > x}, I+1,... (20) (©) x(t) free, = i=m+l,....0 Here r denotes a discount factor. The current value Hamiltonian is by definition HO(t,x,u, A) = Aof(t.x,u) +4- g(x, u) (21)SECTION 10.1 / SEVERAL CONTROL AND STATE VARIABLES 365 [THEOREM 100195 ° (THE MAXIMUM PRINCIPLE) Suppose that (x*(¢), u*(f)) is an optimal pair for the problem (19)~(20). Then } there exist a continuous and piecewise continuously differentiable vector function A(t) = (Ar(t), ..., An(t)) and a constant Ag, with Ao = 0 or Ao = 1, such that (20, A(0)) # (0,0) for all in [, #1], and such that: (A) For allt in [t, ty), u=u"(t) maximizes H“(t,x*(t),u, (0) for ue U (22) (B) Wherever u(t) is continuous, OH(t, x*(t), u*(t), A@)) ele, 23) Ox; " ey Ai) - ri) = — : (C) Finally, corresponding to the terminal conditions (20)(a), (b), and (c), one has the respective transversality conditions: (a’) Ai(ti) no condition, i=,..,0 (b) A(t) 2195) (with =ifxt(q) >a), i=l41..4m 4) ©) Ai) = 29 SY) femtl.n Ox THEOREM 10.1.6 (SUFFICIENT CONDITIONS: ARROW) ~ The conditions in Theorem 10.1.5 are sufficient (with Ao = 1) if AP (,x,4(O) = max H(t, x, u, AO) is concave in x (25) and S(x) is concave in x. (26) The problems for this section are of a theoretical nature. Non-theoretical exercises are found at the end of the next section. “PROBLEMS FOR'SE 1, Consider the variational problem with an integral constraint max [" Fea, dt, x(t) =x", x4) = [cesna=« In A ‘Transform the problem to a control problem with one control variable (w = 2%) and two state variables x = x(¢) and y() = ff, G(t, x(t), X(t) dr.366 © CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES 2. Prove (11) assuming that u°(t) is differentiable and u*(1) belongs to the interior of U. (Hint: Differentiate H(t,x"(¢), u(t), p(®) totally watt ¢.) @® 3. Let S and U be convex sets in R" and R’, respectively, and let F(x, u) be a real-valued concave 10.2 EXAMPLE 1 function of (x,u), x € S,u € U. Define F(x) = max F(x, w) (#) where we assume that the maximum value exists for each x € S. Prove that f is concave in S. (Hint: Let x1, ¥) € S, A € [0,1] and choose uy, up in U such that f(x;) = F(x1,u), (2) = F%, w).) Let B be a convex set in R" x RY and define the set Uy = {u : (x,) € B). Prove that (8) = maxyeu, F(x, w) is concave. 4. Rewrite the following problem as one of the type (1)-(5): ‘1 "1 ax f f,x,u)dt, F= (tau), xl) =x", we, ['voswaax n Here fp, f, x°, and K are given numbers, f, g, and h are given functions, and U is a subset of FR. Some Examples In this section the theory from the previous section is used to solve some multidimensional control problems. The first is intended to be simple enough for you to be able to make a real effort to solve it before looking at the suggested solution. Solve the problem EQ =yO, x0) =0, x(7)is free T — tu? aes [ (x(@) + y() — ju?) at, j)=utty, y(0)=0, y(PYis free wet Verify that the last equality in (10.1.14) is satisfied. Solution: Suppose that (x*(t), y*(t), u*(f) solves the problem. With the two adjoint vari. ables p and pp, the Hamiltonian is H = x+y —}u?-+ piy-+ pout, which is clearly concave in (x, y,u). (Because x(T) and y(T') are free, Note 10.1.3 implies po = 1.) We see that HL =1,H) =1+ pi,and Hy = —u + po. The differential equations for pj and p2 are y(t) = —1 with py(T) = 0, and pr(t) = =1 = pi(t) with po(T) = 0. It follows that py(¢) = T — 1. Hence, pr(t) = -1+4—T and therefore pr(t) = -1 + 412 — Tt + A. The requirement p2(7) = 0 implies that A= 47? +T. Thus pwQ=T-1 pat)=y-1 +T-1EXAMPLE 2 SECTION 10.2 / SOME EXAMPLES 367 Now H is concave in w and u € R, so H has its maximum when Hj, = 0. This gives u'() = pr) = f(T 1) + T= 1. Since j*() = ut) = F(T — 1)? + T 1, we find by integration that y*(t) = —}(7 —1)3 +71 — 41? + B. The initial condition y*(0) = 0 gives B= 379. From i*(t) = y"(0) we get x°() = (T — 8+ 57? — §P + ET +C. The requirement x*(0) = 0 gives C = — 3,7. Hence the optimal choices for x* and y* are (=n +hrP peter t—gr, y= hr - $T1-5P +57? Mangasarian’s theorem shows that we have found the optimal solution, The value function is V(T) = fg (x*(0) + y*(@) — 4u"(0) de, and a rather tedious computation (using Leibniz’s formula) yields V'(T) = $7? + 373 + 37°. On the other hand, H*(T) = x*(T) + y*(T) — $(u*(T))? + pu(T)y*(Z) + p2(T)u* (7) is easily seen to equal $77 + 37? + 57%, confirming (10.1.14). ' (Iwo-sector model) (This is related to a model of Mahalanobis.) Consider an economy that is divided into two sectors, Sector 1 produces investment goods, while sector 2 produces consumption goods. Define 4 () = output in sector i per unit of time, i = 1,2 u(t) = the fraction of output in sector | that is invested in sector 1 ‘Assume that each unit of investment in either sector increases output in that sector by a units. It follows that 4, = aux, and i = a(1 —u)x1, where a is a positive constant. By definition, one has 0 < u(t) < 1. Finally, if the planning period starts at time ¢ = 0, then x1(0) and x2(0) are historically given. We consider the problem of maximizing total consumption in a given planning period [0, T]. The problem is then, with a, T, x?, and x9 as positive constants: max T (ee =au(nO, (0) = x2, x4 (T)is free walt) dt, wet. Jy i) =a —u())x(), 720) = 29, x27) is free ‘The Hamiltonian is H = x2 + piaux; + pra( — u)x, where py and pz are the adjoint variables associated with the two differential equations. (Because both terminal stocks are free, Note 10.1.3 implies po = 1.) Suppose that (x#(s), x(0)) and u*(t) solve the problem. According to Theorem 10.1.1 there exists a continuous vector function (p:(t), p2(t)) such that for all ¢ in (0, 7), u*(¢) is the value of w in [0, 1] which maximizes x3(¢) + py (A)aux7(¢) + pa(t)a(l — w)xz(t). Collecting the terms in H which depend on u, note that u*(r) must be chosen as that value of win [0, 1] which maximizes a(ps(t) — p2(t))x7 (t)u. Now, x](0) = x} > 0, and because (0) = au*(#)x{ it follows that x{(0) > 0 for all ¢. The maximum condition therefore implies that u*(t) should be chosen as w(t) { 1 if pil) > pratt) @ 0 if i) < pratt)368 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES The function p2(t) satisfies p2(t) = —9H*/8x. = —1 with p(T) = 0. Hence, po(t)=T-t The function pi(t) satisfies p(t) = —9H*/Ax1 = —pr(tau*(t) — pr(t)a(l — u"(0)), with p)(T) = 0. Because p(T) = p2(T) = 0, one has pi(T) = 0. From po(t) = —1, it follows that pi(t) < p2(t) in an interval immediately to the left of T. (See Fig. 1.) If we let 1* = inf(t € (0,7) : pi(t) < pr(t) fort in (¢, 7)}, then (¢*, 7) is the largest such interval. (Possibly, r* = 0.) Using (i) it follows that u*(r) = 0 in (¢*, 7). Hence, pit) = —ap2(t) = —a(T — 1) in (¢*, 7). Integration yields py(t) = —aTt + fart. But p;(T) = 0, so Cy = 3aT? and hence pit) =—aTt + har? that? = har -1°, te [t",T] Pus P2 Figure 1 The behaviour of py and pr. Unless pi(t) < p(t) for all ¢ in [0, 7), the number r* is determined by the equation pilt*) = po(t*). Using the expressions found for py (¢) and p2(t), it follows that tf =T 2a if T >2/a, otherwise * = Consider the case when T > 2/a, sot* > 0. How does py (t) behave in the interval (0, 1°)? Note first that . —api(t) if pul) > polt) A= - ~apr(t) fp) po(t), then —ap;(t) < —ap2(t). Whatever is the relationship between p) (1) and pr(t), we always have Bilt) $ —apr(t) = a(t T) In particular, ifr < *, then p(t) < a(t ~T) < a(t* —T) = —2. Because po(t) = I for allt and p1(t*) = po(t*), we conclude that py(t) > p(t) fort < 1%. Hence, u*(¢) = 1 for tin (0, r*]. The maximum principle therefore yields the following candidate for an optimal control, in the case when T > 2/a: 1 ifr € (0,7 -2/al f fal > 2/0) Gi) 0 ifte(T-2/a,T] For t in (0, T — 2/a), we have u*(t) = 1 and so pi(t) = —apy(0), ie. pi) = Cem“. Because pi(t*) = pa(t*) = T — t* = 2/a, this yields pit) = Qfaye***7, te (0, T — 2/a)SECTION 10.2 / SOME EXAMPLES 369 Itis easy to find explicit expressions for x7(t) and x3(t). (See Problem 2.) In the other case, when T < 2/a, one has t* = 0, so the candidate for an optimal control is u*()=0 forallrin(0,7] (T <2/a) In this example the maximum principle yields only one candidate for an optimal control (in each of the cases T > 2/a and T < 2/a). The Hamiltonian is not concave in (x1, x2, u) (because of the product 1x1). Thus the Mangasarian theorem does not apply. For x; > 0 and x) > 0, however, the maximized Hamiltonian H defined in (10.1.16) is xotapin if p> pr H(t, X1,.%2, Pty Pr) xatapox if pi < po For each ¢ in (0, 7], the function ff is linear in (x1, x2). It is therefore concave in the set A = {(x1,%2) : x1 = 0, x2 = O}. According to Theorem 10.1.3 and Note 10.1.7, the solution to the two-sector problem has been found. 1 BLEMS FOR SECTION 10.2 1. Solve the problem 4 #() = = i vmx [oon +a. AD =nO, nO)=2, x) isfiee wenn do HO =u, nO =4, m(d)isfree @® 2. In Example 2, for the case when T > 2/a, find the functions x}(t) and x3(¢) corresponding to the control function given in (i). @3. (a) Solve the problem Hu), m=O, Sn(1) =un(t), x2(0) = 0, x2(T) is free xi(T) is free 7 max [ (dx + be a — un) dt, { lo with 0 < uy(t) < 1,0 T —2/b > 0. (Compared with Example 2, an extra flow of income amounting to one unit (say 1 billion per year) can be divided between extra capital investment in either the investment or consumption goods sectors, or consumed directly.)370 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES 10.3 5. Solve the problem 7 i =x, nist max f ee are me oie at eee ata Jo fhebx, (0) =), x (1) is free where T, b,c, and u° are positive constants with beT > 2 and 2be < 1. (Economic interpret- ation: Oil is produced at the rate of u° per unit of time. The proceeds can be used to increase the capacity x1 in the sector producing consumption goods. By adjusting the physical units, ut. The production of consumption goods is proportional to x; and, by adjusting the constant of proportionality is chosen as 1. The production of consumption goods increases the stock of pollution, xp, at a constant rate per unit, This subtracts cx from utility per unit of time.) 6. Consider the problem RO = f(KO, uO) —e), KO) 2) = -ul), x(0) ‘0, K(T) = Kr : avs vianrers | oe where u(t) > 0,c(t) = 0. Here K (1) denotes capital stock, x(t) isthe ‘stock ofa natural resource, c(t) is consumption, and u(t) isthe rate of extraction. Moreover, U is a utility function and f is the production function. The constants T, Ko, Kr, and 29 are positive. Assume that U' > 0, U" <0, fi > 0, fi > 0, and that f(K, u) is concave in (K,, u). This problem has two state variables (K’ and x) and two control variables (u and c), (a) Write down the conditions in Theorem 10.1.1, assuming that u(t) > 0 and c(t) > 0 at the optimum. (b) Derive from these conditions that Era fe(K.w) din z . qk, 4) ic (K 1) F(R w) where & is the elasticity of the marginal utility. See Section 8.4. @®7. Solve the problem x(2) is free 2 — tu)dr, aay [0-4 Oe vel0.1) Infinite Horizon Infinite horizon control problems were introduced in Section 9.11. This section extends the analysis in several directions. Consider as a point of departure the problem > im xO) = max [ F(t, x(0), uO) dt, (0) = gl, xO, uO), x(to) 0) My u@ev where x! is a fixed vector in R". Suppose the integral converges whenever (x(r), u(t)) satisfies the differential equation and x(+) tends to the limit x! as ¢ tends to oo. For thisSECTION 10.3 / INFINITE HORIZON 371 problem the maximum principle holds. If we replace the condition lim, .. x(t) = x! with Tim, <0 X(t) = x! or limy-o0 X(t) free, then the maximum principle again holds, except for the transversality conditions. When the integral in (1) does not converge for all admissible pairs, what is a reasonable optimality criterion? Suppose (x(¢), u(¢)) is an arbitrary admissible pair, and (x*(t), u*(#)) is a pair we wish to test for optimality. Define : : pwn= [ rex@wenar-[ peexe,uenae ) ‘There are several optimality criteria in economics literature which differ in how Dy(t) behaves for large values of f. The simplest of these criteria is OVERTAKING OPTIMAL ‘The pair (x*(t), u(t) is OT optimal if for each admissible pair (x(t), w(t)) 8) there exists a number Ty such that Dy(r) > 0 for all t > Ty. More important than overtaking optimality is the next criterion: CATCHING-UP OPTIMAL The pair (x*(t), u*(t)) is CU optimal if for each admissible pair (x(t), u(t) and @ every 6 > O there exists a number Ty,- such that Dy(t) > —e whenever > Ty.e- NOTE 1 In general, let f(s) be a function defined for all t > fp. Following the discussion of upper and lower limits in Sections A.3 and A.4, define F(t) = inf { f(t): >t}. Then F(f) is an nondecreasing function of 1, and we define the lower limit of f(t) as t tends to infinity as lim f(t) = jim F(t) = Jim (inf {f(@) ir 2 }) () 00 Here we allow lim,.oo F(¢) = 00. The following characterization is useful and quite straightforward to prove. For each ¢ > 0 there exists a 1’ a such that f(f) > a—e forall t >" fQza = { With this definition the requirement in (4) can be formulated as: (x*(), u*(A) is CU optimal <=> lim Dy(t) = 0 for all admissible pairs (x(#), w(t) Soo372 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES ‘We tum next to the behaviour of x(t) as ¢ approaches infinity. The requirement that x(t) tends to a limit as ¢ approaches infinity is often too restrictive, So is the alternative require- ment that lim;-so0 x(t) 2 x! because it excludes paths where x(t) oscillates indefinitely. Among many possible terminal conditions consider the following: Tim;-+0o %i(f) exists and is equal to x!, i=Ll (7a) Tim,_,., %i(t) > x}, isl (7) ‘no conditions imposed on x;(t) as t > 00, i=m+l,....n (Te) One can show the following theorem (Halkin (1974): THEOREM 10.3.1 (THE MAXIMUM PRINCIPLE: INFINITE HORIZON) Suppose the pair (x*(1), u*(t)) satisfies the differential equation in (1), the initial condition x(fq) = x°, and the terminal conditions (7). If this pair is OT or CU optimal, then it must satisfy all the conditions in Theorem 10.1.1 except the transversality conditions. The problem with this theorem is that when I < n it gives too many solution candidates because it includes no transversality condition. Here is a result that gives sufficient conditions for CU optimality: THEOREM 10.3.2. (SUFFICIENT CONDITIONS: INFINITE HORIZON) Consider problem (1) with the terminal conditions (7). Suppose that the pair (x*(0), u"(O), together with the continuous and piecewise differentiable adjoint function p(t), satisfy the conditions (A) and (B) of Theorem 10.1.1, with py = 1, for all r > tp. Suppose too that U is convex, that H(t,x,u, p(t) is concave in (x, u) (8) and lim [p(t) - (x(t) — x*(t))] = 0 for all admissible x(t) (9) 00 Then the pair (x*(t), u*(r)) is CU optimal. Proof: Applying the arguments in the proof of Theorem 9.7.1 and putting t; = 1, we obtain Da(t) = p(O)-(x(1)—x"(1)). Taking lim on both sides, it follows that im, , Da(t) > 0. The following conditions are sufficient for (9) to hold (see Seierstad and Sydseeter (1987), Section 3.7, Note 16). For all admissible x(¢): Jim [pi(@)(x} ~ xP(O)] = 0 i=1..,m (10a)SECTION 10.4 / EXISTENCE THEOREMS AND SENSITIVITY 373 There exists a constant M such that Ipi()| tp ™ an Either there exists a number t’ > f such that pi(t) = 0 for all t > ¢’, or there exists a ym (10c) number P such that |x)(¢)| < P for all t > f9 and lim p(t) = 0 fm There exists a number Q such that |xj(#)| < Q for all t = t, and jim, pilt)=0 (10a) NOTE 2 (Malinvaud’s transversality conditions) If the terminal conditions 7(a)-(c) are replaced by the conditions x;(t) > x} for all f > to and all i = 1,...,n, then the inequalities p(t) = 0 for all t > to and 10(a) are sufficient for (9) to hold. EBROBLEMS FOR SECTION 10: @ 1. Given € (0, 1), solve the problem max [ (r-we"'d, F=ue', xQ)=%m>0, we [0,1] lo @® 2. (a) Solve the following problem when r > a > 0: conge dy =aum, x1(0) =a} > 0 max foxe'dt, i wel.) Jo d=a(l—-wm, (0) =4$=0 (b) Show that the problem has no solution when r 0 such that (x(t)|| . Then Theorem 10.4.1 is stil valid if the requirements are satisfied for all tin [t, To].EXAMPLE 1 “EXAMPLE 2 SECTION 10.4 / EXISTENCE THEOREMS AND SENSITIVITY 375 ‘A technical problem with the Filippov-Cesari existence theorem is that, in order to ensure the existence of an optimal control, the class of admissible control functions must be en- larged to include “measurable” functions. These can be “much more discontinuous” than piecewise continuous functions. But as long as they are bounded, they will still yield in- tegrable functions of r along the optimal path. (For a brief survey, see Lee and Markus (1967), pp. 55-56.) In almost all control problems encountered in applications one can assume that if there is a measurable control that solves the problem, then there exists a piecewise continuous control that solves the problem. Consider the problem max fi! x2d1, # = 1—u?, x(0) = x(I) =4, w € [-1,2] The Hamiltonian H = x? + p(1 — uv?) is not concave in (x, u) and Arrow’s sufficiency condition also fails. In Problem 3 you are asked to find a unique solution candidate by using the maximum principle. Use Theorem 10.4.1 to prove that this candidate is optimal. Solution: Note first that (x(t), w(t), 4, 1) is an admissible pair. Also, N@,x) ={@? +y,1-w): y SO, we [-1, 2) which does not depend on 1. As u varies in [—1, 2], the second coordinate takes all values between 1 and —3, For fixed x, the first coordinate takes all values less than or equal tox. The set N(t, x) is therefore as illustrated in Fig. 1. 1 N(x) Jt 2 3 Figure 1. The set N(¢, x) in Example 1 is convex. Obviously, N(¢,.r) is convex as an “infinite rectangle”, so (a) is satisfied. The set U = {-1, 2] is compact, Since |(¢)| = [1 — u?(t)| < 3 for all admissible w(t), any admissible x(F) satisfies 1 < x(¢) <7 for all rin (0, 1], which takes care of (c). We conclude that the ‘unique pair satisfying the conditions in the maximum principle is optimal. ' Show the existence of an optimal control for Example 10.2.2. (Hint: Use Note 2.) Solution: Clearly, w(t) = 0 gives an admissible solution, and the set U = (0, 1] is compact. The set N = N(¢, x) is here N(jxm2) ={Q2ty, aun, a(l-w)m) ty $0, ue (0,0) ‘This is the set of points (£1, >, &3) in R? with & < x9 and (2, 63) lying on the line segment that joins (0, ax1) to (ax;, 0) in R?. Hence N is convex. ‘The inequality in (2) is also satisfied because lg(t.x1, x2, 4) = [aun a(1 — wn) || = V(@um)? + @ = wx)? = alm|V2u? = Qu +1 < alxi| = ay/x} < afta ali(.x2)Il376 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES using the fact that 2u? ~ 2u + 1 = 2u(u — 1) +1 < 1 forall w in (0, 1]. The existence of an optimal control follows from Theorem 10.4.1 t Precise Sensitivity Results Here we shall briefly present precise conditions forthe sensitivity resultsin (10.114) tohold. Consider the standard end-constrained problem (10.1.1)-(10.1.5) and assume that admissible pairs exist. Let x! = (x},...,x4) and define V@,x!,.n) = surf f° r0.xtn,wonar (x(), w()) admissible} 8 » (Ifm = 0, the right end point is free and V will not have x! as an argument.) The function V is called the (optimal) value function of the problem. It is defined only for those (x, x', fp 1) for which admissible pairs exist. If for a given (x°, x!, fo, t) an optimal pair exists, then V is finite and equal to the integral in (10.1.1) evaluated along the optimal pair. (This was the case studied in Section 9.6.) If the set in (3) is not bounded above, then V = co. Suppose that (x*(r), u*(r)) solves problem (10.1.1)-(10.1.5) with x° = %°, x! = 1, fp ty = fi for pp = 1, with corresponding adjoint function p(t). The next theorem gives sufficient conditions for V to be defined in a neighbouthood of (3°, X*, ip, f1), and for V to be differentiable at (&°, x! fo, 71) with the following partial derivatives: aV@.R eA) - pt = pitfo), 4 a pilto) a avi fo, fy Y OX fH) _ _ pay, vat © a} AVR, 81, fof SA ee Slo) __a1G,x° Ge), uC), Po) © 3 iol) ae ein) wy ih = HG, xG),wG), PA) o THEOREM 10.4.2: ~ ~ Consider the standard end-constrained problem (10.1.1)~(10.1.5) with a compact control region U. Suppose that (a) (x*(1), u*(2)) is a unique optimal solution. (b) p(t) is uniquely determined by the necessary conditions given x*(1), u*(t), and po=l () There exist continuous functions a() and b(t) such that IF(,x,w)l| 0, x(0) free. b ef.) @ 3. Find the unique solution candidate in Example 1 using the maximum principle. (Hint: Argue why w*(t) can only take the values 0 and 2, and why any admissible x(t) is > 0 in (0, 1].) 10.5 A Heuristic Proof of the Maximum Principle ‘A full proof of the general maximum principle is quite demanding and draws on several advanced results in the theory of differential equations which are not in the toolkit of most economists. The heuristic arguments for the main results given below, although not precise, give a good indication of why the maximum principle is correct. We restrict our attention to problems with one state and one control variable. Consider the following control problem with two alternative terminal conditions: " (tr) free a [ F(,x,u)dt, = g(t,x,u), xt) = Xo, {e = @ ‘Think of x = x(£) as a firm’s capital stock and "f(t, x, u) dt as the total profit over the planning period ['o, J, in line with our general economic interpretation in Section 9.6. Define the value function by " ; x(t) free V(t, x) = max [ieso.nonas 405) = als. x06), UG), =n | @ we [Jr x(t) =x Thus V(t, x) is the maximum profit obtainable if we start at time with the capital stock x. Suppose the problem in (i) has a unique solution, which we denote by i(s; t, x), #(5; t,x), foro <1 Ss Sn. ‘Then, by definition, #(¢; t, x) = x.378 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES Figure 1. The case of x(n) free. Consider any starting point (1',x’) that lies on the optimal path I" defined by the original solution i(s; t,x). If there were a better path I’ starting at (1, x’) it would have been optimal forthe solution starting at (t, x) to follow I’ over the time interval {r', n].! (See Fig. 1, which deals with the case when x(t) is free.) For this reason, an optimal solution starting at (t, x) is automatically an optimal solution from ((’, x") as well: the “ail” of an optimal solution must be optimal. Using the uniqueness of (&(s; t,x), #(s; t,x) forall (t,x), this implies the relations X(s;1,x) Hsyt',x))=Hs:t.), Hojt',x)= whenever t’ € [f, s] and x’ = ¥(1';t, x). Hence, vestita= [ 16. Hs; t,x), (s; t,x) ds Differentiate this equation wert. t’ att’ = t. Because di(t'; 1, x)/dt’ = g(t’, 80°51.) (05 1,2), we have Vf (00) + VEU wg (ts x, UE; #3) fle. altst,9) (ii) Hence, if we define Blt. x) = Vu(t,x) and introduce the Hamiltonian function H(t, x, w, p) = f(t, x,u) + p g(t, x, 1), then equation (iii) can be written in the form Vit, x) + H(t, x, (3 t,x), BU x) =0 (iv) Starting at the point (f, x), consider an alternative control which is a constant » on an interval [t,t + At] and optimal thereafter. Let the corresponding state variable be x"(s) for s in [t,t + Ar). Then +e von f £(5,x(8), v)ds + Vir + At x'(t + AD) a sar Vor + At, x" + AN) — Vx) +/ F(s,x"(s), v)ds <0 W) Dividing this inequality by Af and letting At ~> 0*, we get £V(r, x") + f(G,x, v) <0. Now, 4.V(t,x) = Vi(e.x(0) + Vil, x)”. Since Vi(t,x) = (t,x) and 2°) = g(t,x, v), we must have Vit, x) + BU, x)g(t, x, v) + f(t,x,v) 0 ' “Better path” I’ is intuitive language. It means that there exists an admissible pair (x(s), «(s)) (with corresponding path T") that gives a higher value to the integral of ff over (t11] when (x(s), 0(5)) is inserted, as compared with the value resulting from (E(s; f, x), (6; 4, x))SECTION 10.5 / A HEURISTIC PROOF OF THE MAXIMUM PRINCIPLE 379. Thus forall vin U, Wit, x) + H(t, x,v, B(t,x)) $0 Because of (iv), this implies that the optimal control i(¢; 1, x) must maximize H(t, x, w, p(t, x)) wrt. w € U. In addition, Vee x) + max HOG, x,u, Vy(t,2)) =0 (wi) This is called the Hamilton-Jacobi-Bellman equation. Next, define x*(t) = E(t; to, xo) and u*(r) ii(t; to, xo). These functions give the optimal solution to the original problem. Also, let p(t) = p(t,x*(0)). Then i(¢;#,x°(2)) = a"), and therefore = u(t) maximizes H(t,x°(,u, p@) wat we U (wit) Finally, differentiating (iv) wart. x and using the envelope theorem (see Section 3.10), we get Vat H+ Hyp, =0 Because Vi = g, this yields p} + Pie(t, x, (0; t,x) = —Hy. If we letx = x°() and use u(t) = a(t; t,x), then p = Bi + pli = Bl + Bett. x.u"(), 80 Bt) = - Hy. 0°), @), PO) (viii) By definition of V, if x(t) is free, then V(t, x) = 0 for all x. Thus f(t, x) = 0, and so we have the transversality condition Pin) =0 (ix) Conditions (vii) to (ix) are the necessary conditions in the maximum principle (with 1 fixed and x(t) free). If x(t) is fixed, condition (ix) is not valid (and not needed). ‘We have shown that -H"(to), Vig = Plto) ®) In fact, the first equality follows from (iv) and the second one from the definitions of the functions p and p. These are two of the formulas in (10.1.14). Reversing time gives the other two formulas: vy vy Hn), Vy, =—plh) (xi) Variable Final Time Problems Consider problem (i) witht; free. Suppose (x*(1), u*(¢)) is an optimal solution defined on [t, #7 ‘Then conditions (vi)-(viii) must be valid on the interval [tf], because (x*(1), u°(t)) must be an optimal pair forthe corresponding fixed time problem with t; = ¢}. Moreover, atthe terminal time 1 the value function's derivative wt. fy must be 0, (As a function of fy it has a maximum a 7.) Because of (xi), this implies that HG) = HG), E), PED) eh er) 0 (xii) ‘This equation gives an extra condition for determining f, and is precisely condition (9.8.2). NOTE 1 Inthe above heuristic “proof” of the maximum principle, dfferentiability of the function V was assumed without proof.380 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES 10.6 EXAMPLE 1 Mixed Constraints This section describes control problems where the admissible pairs (x, u) are required to satisfy additional constraints involving the state variable of the form h(t, x, u) 2 0. Such restrictions often occur in economic models. If the control variable u as well as the state vector x appear in the function h, the restriction is often referred to asa “mixed constraint”, while restrictions of the type h(t, x) 2 0 are called “pure state constraints”. Whether or not mixed constraints are present in a given control problem is partly a question of the form in which the problem is stated. Consider the following problem. Consider the growth problem T max [ U((l-u) f(K)) dr, =u, K(0) = Ko, K(T)=Kr, u20, f(K)-u20 «So Here there are two constraints for each t—namely, the simple constraint hy(t, K,u) =u 20 and the mixed constraint hz(t, K, u) = f(K)—u > 0. However, if we specify a new control variable v so that K = uf(K), then the simple constraint 0 < v < 1 replaces the mixed constraints, (If we require f(K) — u > k > 0, this trick does not work.) I ‘We consider the general mixed constraints problem max f fGxuddt, x=glt,x,u), xo) = @ a Sig h(t,x,u) 20 forall ¢ @ with the terminal conditions (a) x(n) accel (o) x(4)2a}, fHltl.ym @) (c) x(t) free, =m+1,....7 {As usual, x is n-dimensional and w is r-dimensional, while h is an s-dimensional vector function, so that the inequality h(¢, x, u) > 0 represents the s inequalities Ay(t,x(),w(N) =0, @) [All the restrictions on u(t) are assumed to have been incorporated into (2). Thus, no ad- ditional requirement of the form u € U is imposed. In addition to the usual requirements on f and g, itis assumed that h is a C! function in (t,x, u). The pair (x(#), w(t) is ad- missible if us(¢), ..., up (0) are all piecewise continuous, and x(t) = (x1(f),---+ #1) is the corresponding continuous and piecewise differentiable vector function that satisfies X = g(t, ¥, W), X(fo) = x9, (2), and (3). The theorem below gives sufficient conditions for the solution of the mixed constraints problem (1)-(3). To economists, it will come as noSECTION 10.6 / MIXED CONSTRAINTS 381 surprise that we associate multipliers q(t), ...,qe(t) with the constraints (2) and define the Lagrangian function, with q = (g1, ..., 4s), a8 L(x, 0, pg) = HG,X, Up) + gel (t.x, w) © ia with the Hamiltonian H(t. x, u, p) = f (t,x, a) + D7) pigi(t, x, uw) (with po = 1). In the following theorem 9.* /du; and 4.£*/8x; in (6) and (8) are partial derivatives of £ evaluated at (¢,x*(0), u*(0), p(t), a). HEOREM 10.6.1 (SUFFICIENT CONDITIONS) — Suppose (x*(t), u*(r)) is an admissible pair in the mixed constraints problem (1)+(3). Suppose further that there exist functions p(t) = (pi(f), .--» Pn(t)) and q(t) (qi(t),..-.gs(t)), where p(?) is continuous, while p(t) and q(t) are piecewise continuous, such that the following requirements are all satisfied: bL* ou (6) q(t) 20, and q(t) =0 if Ax(t, x*()), u*(t)) > 0, kK=1,....8 (7) Bt) = . at all continuity points of u*(t), G=slwn (8) No conditions on p;(ti), (a) ph) > 0, and pi(t)) =0 if a(n) > x}, “sm (9b) pi(ti) =0, n (8c) H(t,x, u, p(s) is concave in (x, u) (10) hy(t, X, w) is quasiconcave in (x, u), as ay Then (x*(t), u*(t)) solves the problem. ‘A proof of this theorem is given in Seierstad and Sydsaeter (1987), Section 4.3, which also discusses necessary conditions, generalizations, and examples, and has further references to other literature. A simpler treatment can be found in Léonard and Long (1992), Chapter 6. Note that as in nonlinear programming a constraint qualification is often necessary for the existence of a pair (x(t), u(®)) of the type occurring in the theorem. The constraint qualification, more or less, requires that the control w appears in each constraint. “EXAMPLE 2” Solve the mixed constraints problem Iy(t,x,u) =u-c20 1 ina f udt, %=ar—u, x(0)=29, X(T) free, b In(t, xu) =ax—u>0 Here x is the capital stock, w is consumption, and c is a subsistence level. The constants 7, a,c, and x° are positive, with T > 1/a and ax® > c.382 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES Solution: ‘The Hamiltonian and the Lagrangian are H=utpex—u, L=H+qiu-d+q(ax-w) Here H as well as hy and hp are linear and hence concave in (x, «). The following conditions from Theorem 10.6.1 are therefore sufficient for optimality: aet an 1 p(t) +q@) - g(t) =0 @ a(t) 20, and qi(t)=0 if u(t) >¢ ii) qo(t) > 0, and g(t) =0 if ax") > w(t) Gi) att BO=-F- = ap(t)—agr(t), p(T) =0 (iv) wee, ax*()—u"(N 20 «) Because x*(0) = x° > 0 and &*(t) = ax*(t) — u*(0) > 0 for all , one has x*(¢) 2 x° for all t. Ifw*(f) =, then ax*() — u(t) = ax*(t) c= ax® ~c > 0, and then from (ii), qo(t) = 0. On the other hand, if u*(t) > c, then (ji) implies q; (1) = 0. Hence, for all r in {0, T], at least one of qi (t) and qo(t) is 0. Because p(T) = 0, it follows that p(t) < 1 for ¢ sufficiently close to (and less than) 7. ‘Equation (i) shows that for such t we have qo(t) = 1 — p(t) +qi(t) = 1 — p() > 0. Then gi(t) must be 0, so qa(t) = 1 — p(t) and (iv) gives p(t) = —a whenever p(t) < 1 We see from Problem 5.4.8 that the solution of (iv) is p(t) = g agy(t)e"" dr, which is clearly > Oforr < T. Hence, p(t) = —ap(t) - aga(t) < -ap(t) < 0 fort 1/a and condition (jv) says that p(T) = 0. It follows that we must have p(0) > 1, Since p is strictly decreasing, there is a unique point 1* in (0, T') such that p(¢*) = 1. Fort in (¢*, T] we have p(t) = —a, so p(t) = a(T ~ 1). By continuity, a(T ~t*) = p(t") =1,s0t* =T -I/a For t in [0,¢*) we have p(t) > 1, so (i) implies that q1(7) > qz(t) and, because at least one of qi(t) and qp(t) is 0, in fact ga(t) = 0. Then from (ii), u*(t) = ¢ so that 4*(0) = ax"(0) — ¢, with x*(0) = x°. Solving this linear differential equation yields x*() = (x° — c/a)e” + c/a. The differential equation for p(¢) is 6 = —ap because qn = 0. Hence, p(t) = Ae~™ with p(e*) = 1, so p(t) = et Since x*(f) is continuous also at r*, and #* = 0 in (¢*, T], x*(t) has the constant value (x9 -c/a)e* + c/a fort in (*, T]. We have found the following candidate for an optimal solution, with r* = T — 1/a: wo x) |p ao a | relor) | © | @®—elae™+eja | eae) | eat —4 ° I re(*,T) | ax*() (29 —cjae™ e/a a(T —1) 0 l=a(T-1) ‘Theorem 10.6.1 implies that this candidate is optimal. Note that in this example the multi- pliers gy (#) and qo(t) are continuous. 'SECTION 10.7 / PURE STATE CONSTRAINTS 383 PROBLEMS FOR SECTION 10.6 @ 1. (a) Write down the conditions in Theorem 10.6.1 for the problem max b (b) Solve the problem. (Hint: Guess that u*(¢) = x*(1) on some interval (0, 1*], and w*(0) < x*(P) on (¢*, 2). Then g(t**) = 0, and w"(it~) = x*(@) = ut), We can use the following argument? to show that q(t*~) = 0: From a.£*/u = 0 we get q(t) = —p(t) — u*(O. In particular, g(t*~) = —p(e*) — ut(t*) < —plt") =u" (t**) = ga") = 0.) =x)dt, $=-u, x)= @® 2. Solve the problem x(2)free, x20 2 ax f (e—pw)dr, é x(0) =1, (Hint: Guess that u*(¢) = x*(f) on some interval (0, 1"), and u*(t) < x*(#) on (¢*, 2]. As in Problem 1, g(r*~) = 0.) ® 3. Solve the following variant of Example 2: 1 ana f udt, t=ax—u, x0) =x°>0, x(T)2ar, c 0,¢ > 0, T > 1/a,ax® > c, and x° < xp < (x° - c/a)e%” + c/a. (This model can be interpreted as a simple growth model with a subsistence level c.) @® 4, Solve the problem 1 In(t,x,u) = 1-420 ina f xdt, k=x+u, x0)=0, x(1) free, | ho(t,x,u)=1+u20 ° hs(t,x,u) =2—x-u>0 (Hint: See the solution to Example 9.4.1. Try with u*(t) = 1, x*(0) 1 in the beginning. 10.7 Pure State Constraints ‘This section briefly discusses a result giving sufficient conditions for a pure state-constrained problem, It gives an indication of the type of results that can be proved, but we refer to the control theory literature for proofs, examples, and generalizations. Consider the following pure state-constrained problem, (t,x, 0), x(to) wneUCR (1) max f Flxudt, & h(,x) 20 forall Q) 2 The same argument is useful in other problems also, for example in Problem 2.384 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES with the terminal conditions @) xh (b) xh) 2x}, EST +1... m (©) x(t) free, =m+i, @) Note that, in contrast to the mixed constraints case, we now allow a restriction of the form. u € U. The vector function h is s-dimensional, and the pure state constraint (2) can be written Ay(t,x())20, k=1,....8 @ ‘The sufficient conditions given in the next theorem are somewhat more complicated than those in Theorem 10.6.1. In particular, the adjoint functions may have jumps at the terminal time. The Lagrangian associated with this problem is £E,X, 0 p,q) = HX. p) +) gehie(t, x) i with H(t, x, u, p) as the usual Hamiltonian (with po = 1). THEOREM 10°71 (SUFFICIENT CONDITIONS) — Suppose (x*(#), u*(t)) is admissible in problem (1)-(3), and that there exist vector functions p(t) and q(f), where p(7) is continuous and p(t) and q(t) are piecewise continuous in (to, 11), and numbers fy, k = 1,...,5, such that the following conditions are satisfied with po = 1: u=u*() maximizes H(t, x*(1), u, p() for u in U get) 20, and ge(t) =0 if helt, x*() > 0, Bt) at all continuity points of u*(t), xi Atty, pi(0) can have a jump discontinuity, in which case S Ohg (ty, x" pale) ~ pale) = FB OD, ‘kel XE Be =O, with 6 =0 if he(t,x*(n)) > 0, No conditions on pi(ti), pi(t) =O, and pi(th) =0 if xf(4) > x}, pilti) =0, At,x, p(0)) = maxyev H(t, x, u, p(t) is concave in x g(t, x) is quasiconcave in x, ‘Then (x*(t), u*(#)) solves the problem. a (8) o (10) (1a) (1b) (Ile) (12) (13)SECTION 10.7 / PURE STATE CONSTRAINTS 385 Here p(t) = (pi(t),.-+. a(t) and q(t) = (q(t), .... s(t), while 3.£*/8.x; denotes the value of 8.£/33; at (1, x*(9), u*(#), pO), a), NOTE 1. The conditions in this theorem are somewhat restrictive. In particular, sometimes the solution requires p(t) to have discontinuities at interior points of {fo, 1]. For details and a proof, see Seierstad and Sydsceter (1987), Chapter 5. Solve the problem 4 max f= 2%, d=ueR, x(0)=0, x(4) free, x() <1 Solution: The Lagrangian is £ = H +q(1—x) =x -(u-2)? + putq(1—2). Here H is concave in (x, u) and h(t, x) = 1 — x is quasiconcave. The conditions (i)-(iv) below are therefore sufficient for optimality. Equation (i) below results from the observation that His concave in u and u € R, so condition (6) is equivalent to the condition 3H*/au = 0. ur(t) = bp) +2 (i) q(t)>0, and q(t) =0 if x*() <1 Gi) Le bO=-F>= qt). p(4)=0 ii) p(4-) — p(4) =-B <0, and B=0 if x*(4) <1 @v) ‘We can guess the form of the solution as long as we eventually verify that all the conditions in the theorem are satisfied. Accordingly, we guess that x°(t) < 1 in an interval (0, r*) and that x*(t) = Lin (t*, 4]. Then in (¢*, 4), w(t) = *(t) = 0, and from (i), p(t) = —4. But then from (iii) and (iv), B = p(4) ~ On (0, 1*), from (ii) and (iii), P() = 1. Since p(t) is continuous at", pe) = 4 Hence, p(t) = —4+ (e* — 1), and from (i), u(t) = 37-0. ieee) =H 30° —1) yields x°(¢ Lr —1)?+C on ([0, t*) Since x*(t*) = I, we get x*(1*) = C = 1. But x*(Q) = 0, so t* = 2. Our suggestion is therefore: In0, 2): w(t) = 1 - fr, 2*(@) = 1 — 42-1), p@) = 1-2, and q(t) =0. In 2, 4]: uw") = 0,27(1) = 1, pO) = —4 (except p4) = 0), and g(t) = 1 with 6 = 4. ‘You should now verify that all the conditions (i)~(iv) are satisfied. Note that p(t) has a jump from —4 to 0. 1 @® 1. Solve the problem ain [39d gau-t, xO x(5) free, x20, u20 (Hint; It seems a good idea to keep x(t) as low as possible all the time.)386 CHAPTER 10 / CONTROL THEORY WITH MANY VARIABLES @® 2. Solve the problem 2 max [i =xdt, & 0 (0) xQ) free, x20, we[-11] (Hint: Start by reducing x(t) as quickly as possible until x(t) = 0.) @® 3. Solve the problem x(10) free, x20, uER od, @ 4, Consider the problem €10,2], x) =1, x@)= max fo —1udr, 0 (a) Solve the problem when the constraint t + 1 — x > 0 is not imposed. (b) Solve problem (#). 10.8 Generalizations In Chapter 9 and the previous sections of this chapter we have discussed how to solve the most commonly encountered problems in optimal control theory. Nevertheless, many important economic problems require methods that go beyond those presented so far. More General Terminal Conditions In some dynamical optimization problems the standard terminal conditions are replaced by the requirement that x(t) at time 1; hits a target defined as a certain curve or surface in I". The optimal path in such a problem must end at some point x! and therefore, in particular, will solve the corresponding control problem where all the admissible paths end at x. The conditions in Theorem 10.1.1 are therefore still valid, except the transversality conditions, which must be modified. See e.g. Seierstad and Sydseter (1987), Chapter 3. Markov Controls The optimal solutions we have been looking for have been functions of time, u*(t) and x*(t). Such control functions are called open-loop controls. Faced with the problem of steering an economic system optimally, such open-loop controls are often inadequate. The problem is that “disturbances” of many types will almost always occur, which will divert theSECTION 10.8 / GENERALIZATIONS 387 system from the optimal path initially computed. If one still uses the “old” control u*(t), the resulting development of the economy may be far from optimal, and it may end up at an undesirable final state. This problem is partly resolved if we are able to “synthesize” the optimal control, in the sense of expressing the optimal control as a function of the present time s and the present state y. In this case, for each time s and each point y in the state space, we specify the optimal control {i = fi.y) to use. Such controls are called closed-loop or Markov controls. We can find such Markov controls by solving the control problem with an arbitrary start point (5,y), 5 € [to, 11). The controls u*(t) obtained will depend on the starting point (s, y), u*(t) = ut, (0). Of course, at time s, the control f(s, y) = uj y(s) is used. Then O(s, y) is the required Markov control. But these Markov controls are only conditionally optimal. They tell us which control to use after a disturbance has occurred, but they are optimal only in the absence of further disturbances. If we stipulate the probability of future disturbances and then want to optimize the expected value of the objective functional, this gives a stochastic control problem, in which optimal Markov controls are determined by a different set of necessary conditions. Discrete time versions of stochastic control problems are discussed extensively in Chapter 12 of this book. Jumps in State Variables So far we have assumed that the control functions are piecewise continuous and that the state variables are continuous. In certain applications (e.g. in the theory of investment), the optimum may requite sudden jumps in the state variables. For example, somebody who buys an apartment worth, say, $200,000 will experience a corresponding downward jump in their bank balance. See e.g. Seierstad and Sydszter (1987), Chapter 3.