OPTIMAL CONTROL Second Edition FRANK L. LEWIS Dept. of Electrical Engineering University of Texas at Arlington Arlington, TX VASSILIS L. SYRMOS Dept. of Electrical Engineering University of Hawaii at Manoa Honolulu, HI A Wiley-Interscience Publication JOHN WILEY & SONS, INC. New York + Chichester + Brisbane + ‘Toronto + Singapore“This text is printed on acid-free paper Copyright © 1995 by John Wiley & Sons, Ine Al sights reserved. Published simultaneously in Canad, Reproduction or translation of any part of this work beyond that permitted by Section 107 or 108 of the 1976 United States Copyright Act without the permission of the copyright, fovwner is unlawful. Requests for permission of further information should be addressed wo the Permissions Department John Wiley & Sons, Ine, 605 Third Avenue, New York, NY. ro1se-0012, This publication is designed to provide acurate and suthortative informacion in regard wo the subject matte covered. Iti sold with the understanding that the publisher isnot engaged in endering legal, accounting, for other professional services. I Tegal advice oe other ‘expert assistance is required, the services of a competent Profesional person should be sought. Library of Congress Cataloging in Publication Data: Lewis, Frank L. Optimal contro! J Frank L. Lewis, Vasilis L, Syrmos: Poth Includes index. ISBN 0.471-03378.2 (cloth : alk. papery Contol theory. 2. Mathemateal optimization, 1. Symon Vassilis Loh Tile (adn 31487 1905 629.8312 —de20 9.15649 Printed in United States of Amesica wWoR76s 4324To Theresa and Christopher, who have opened my eves Frank Lewis To my father, my first teacher Vassilis SyrmosPREFACE This book is intended for use in a second graduate course in modern control theory. A background in the state-variable representation of systems is as- sumed, Matrix manipulations are the basic mathematical vehicte, and for those whose memory needs refreshing, Appendix A provides a short review. The book is also intended as a reference. Numerous tables make it easy to find the equations needed to implement optimal controllers for practical ap- plications. ur interactions with nature can be divided into two categories: observation and action. While observing, we process data from an essentially uncooper- ative universe to obtain knowledge. Based on this knowledge, we act to achieve our goals. This book emphasizes the control of systems assuming perfect and complete knowledge. The dual problem of estimating the state of ur surroundings is briefly studied at the end of the hook. A rigorous course in optimal estimation is required to conscientiously complete the picture be- gun in this text. Our intention was to present optimal control theory in a clear and direct fashion. This goal naturally obscures the more subtle points and unanswered questions scattered throughout the field of modem system theory. What ap- pears here as a completed picture is in actuality a growing body of knowledge that can be interpreted from several points of view, and that takes on different personalities as new research is completed. We have tried to show with many examples that computer simulations of optimal controllers are easy to implement and are an essential part of gaining, an intuitive feel for the equations. Students should be able to write simple programs as they progress through the book, to convince themselves that they hhave confidence in the theory and understand its practical implications. viiPREFACE Relationships to classical control theory have been pointed out, and a root locus approach to steady-state controller design is included. A chapter on optimal control of polynomial systems is included to provide a background. for further study in the field of adaptive control. Two chapters on robust control are also included to expose the reader to this rapidly growing area of interest. The first author wants to thank his teachers: J. B. Pearson, who gave him the initial excitement and passion for the field; E. W. Kamen, who tried to teach him persistence and attention to detail; B. L. Stevens, who forced him to consider applications to real situations; R. W. Newcomb, who gave him self-confidence; and A. H. Haddad, who showed him the big picture and the humor behind it all. We also want to thank our students, who forced us to take the work seriously and become a part of it Frank L. Lewis Fort Worth, Texas January 1995 ssitis L. SyRMOS. Honolulu, Hawait January 1995CONTENTS LLL Optimization without Constraints, 1.2 Optimization with Equality Constraints, 5 1.3 Numerical Solution Methods, 21 Problems, 22 2 Optimal Control of Discrete-Time Systems 27 2.1 Solution of the General Discrete Optimization Problem, 27 2.2 Discrete-Time Linear Quadratic Regulator, 41 2.3 Digital Control of Continuous-Time Systems, 66 2.4 Steady-State Closed-Loop Control and Suboptimal Feedback, 79 2.5 Frequency-Domain Results, 114 Problems, 119 3 Optimal Control of Continuous-Time Systems 129 3.1 The Calculus of Variations, 129 3.2 Solution of the General Continuous Optimization Problem, 131 3.3 Continuous-Time Linear Quadratic Regulator, 161 3.4 Steady-State Closed-Loop Control and Suboptimal Feedback, 185 3.5 Frequency-Domain Results, 199 Problems, 203x CONTENTS 4° The Tracking Problem and Other LOR Extensions 215 4.1 ‘The Tracking Problem, 215 4.2. Regulator with Function of Final State Fixed, 222 4.3 Second-Order Variations in the Performance Index, 224 4.4 The Discrete-Time Tracking Problem, 229 4.5 Discrete Regulator with Function of Final State Fixed, 244 4.6 Discrete Second-Order Variations in the Performance Index, 251 Problems, 256 5. Final-Time-Free and Constrained Input Control 259 5.1 Final-Time-Free Problems, 259 5.2 Constrained Input Problems, 281 Problems, 312 6 Dynamic Programming 31s 6.1 Bellman’s Principle of Optimality, 315 6.2 Discrete-Time Systems, 319 6.3 Continuous-Time Systems, 328 Problems, 341 7 Optimal Control for Polynomial Systems 347 7.1 Discrete Linear Quadratic Regulator, 347 7.2 Digital Control for Continuous-Time Systems, 352 Problems, 356 8 Output Feedback and Structured Control 359 8.1 Linear Quadratic Regulator with Output Feedback, 359 8.2 Tracking a Reference Input, 376 83 Tracking by Regulator Redesign, 391 844 Command-Generator Tracker, 396 85 Explicit Model-Following Design, 403, 8.6 Output Feedback in Game Theory and Decentralized Control, 409 Problems, 417 9 Robustness and Multivariable Frequency-Domain Techniques 421 9.1 Introduction, 421 9.2 Multivariable Frequency-Domain Analysis, 423 9.3 Robust Output-Feedback Design, 447 9.4 Observers and the Kalman Filter, 450 9.5 LQG/Loop-Transfer Recovery, 478CONTENTS 9.6 H, Design, 500 Problems, 506 Appendix A. Appendix B References Index 521 529 5351 STATIC OPTIMIZATION In this chapter we discuss optimization when time is not a parameter, The discussion is preparatory to dealing with time-varying systems in subsequent chapters. A reference that provides an excellent treatment of this material is Bryson and Ho (1975), and we shall sometimes follow their point of view. ‘Appendix A should be reviewed, particularly the section that discusses matrix calculus, OPTIMIZATION WITHOUT CONSTRAINTS A scalar performance index L(u) is given that is a function of a control or decision vector u © R*. We want to select the value of w that results in a ‘minimum value of L(u), To solve this optimization problem, write the Taylor series expansion for an increment in Las dL = Lidu + $ du L,du + 0(3), ay Where 0(3) represents terms of order three. The gradient of L with respect to wis the column m vector (12 and the hessian matrix isSTATIC OPTIMIZATION a3 Le = 5 ow Ly is called the curvature matrix. For more discussion on these quantities, see Appendix A. Note that the gradient is defined throughout the book as a column vector, which is at variance with some authors, who define it as a row vector. A critical or stationary point occurs when the increment dL. is zero to first order for all increments du in the contro}. Hence L, 0 (tay for a critical point. ‘Suppose that we are at a critical point, so L, = 0 in (J.1-1), In order for the critical point to be a local minimum, we require that dL = $ du Lgdu + 0G) (4s) be positive for all increments du. This is guaranteed if the curvature matrix La, is positive definite, a0: (11-6) If L,, is negative definite, the critical point is a local maximum; and if L,, is indefinite, the critical point is a saddle point. If L,, is semidefinite, then higher terms of the expansion (1.1-1) must be examined to determine the type of critical point. Recall that L,,, is positive (negative) definite if all its eigenvalues are pos- itive (negative), and indefinite if it has both positive and negative eigenvalues, all nonzero. It is semidefinite if it has some zero eigenvalues. Hence if IL.4| = 0, the second-order term does not completely specify the type of critical point. Example 1.1- Let u © Rand I Quadratic Surfaces Lu) = wth [40 G2 bye sy sha $u'Qu + Su 2 ‘The critical point is given by 0 8Li OPTIMIZATION WITHOUT CONSTRAINTS 3 w= O'S, “ where u® denotes the optimizing control ‘The type of critical point is determined by examining the hessian La = @. co it u* is a minimum if L,, > 0, oF (Appendix A) gy, > 0. guides ~ ahs > 0. Its a maximum if fy, < 0. oF gy < 0. giz > OIE (Q| <0, then w* isa saddle point. If|Q) = 0.then is singular poinr and we cannot determine whether itis a minimum oF a maximum from Ly By substituting (4) into (2) we find the extremal value of the pecformance index to be 48'9'90"'S - s'9's © ree (ee wf} jew ne ® [2 fey ofa ef -E) ® isa minimum, since fa, > 0. Using (6), wesc that the minimum value of Lis L¥ = ‘Then "The contours ofthe 0) in (7) are drawn in Fig. 1-1, whore w = ba wl, The arrows represent the padiem nooes- [He | ” Note that the gradient is always perpendicular t the contours and pointing in the direction of increasing L(w), . We shall use an asterisk to denote optimal values of w and L when we ‘want to be explicit. Usually, however, the asterisk will be omitted Example 1.1-2: Optimization by Scalar Manipulations We have discussed optimization in terms of vectors and the gradient. As an alternative approach, we could deal entirely in terms of scalar quantities. ‘To demonstrate, let, Ly, us) = Sah + ays +1 * ts o4 STATIC OPTIMIZATION FIGURE 1.1-1 Contours and the gradient vector. Where u, and u, are sealars. A critical point occurs where the derivatives of with respect to all argumenis are equal to 2er0: 2h ay, 6 2m +1=0 ES so a critical point is (1, 1).1.2. OPTIMIZATION WITH EQUALITY CONSTRAINTS 5 Note that (1) is an expanded version of (7) in Example 1.1-1, so we have just derived the same answer by another means. ‘Vector notation simplifies the bookkeeping involved in dealing with multidimen. sional quantities, and for that reason it is very attractive for our purposes. . 1.2. OPTIMIZATION WITH EQUALITY CONSTRAI Now let the scalar performance index by L(x, u), a function of the control vector « R" and an auxiliary (state) vector x € R’. The problem is to select uv to minimize L(x, w) and simultaneously satisfy the constraint equation fox, w) = 0. a2 The auxiliary vector x is determined for a given u by the relation (1.2-1), so that fis a set of n scalar equations, f © R” To find necessary and sufficient conditions for a Jocal minimum also sat. isfying f(x. u) = 0, we shall proceed exactly as we did in the previous section, first expanding dL. in a Taylor series and then examining the first- and second- order terms. Let us first gain some insight into the problem, however, by considering it from three points of view (Bryson and Ho 1975, Athans and Falb 1966). Lagrange Multipliers and the Hamiltonian Ata stationary point, dl is equal to zero to first order for all increments du when df is zero. Thus we require that dl. = Lidu + Lidx = 0 (122 and df = fdu + fade = 0. (1.23) ince (J.2-1) determines x for a given 1, the increment dx is determined by (1.2-3) for a given control increment du. Thus, the Jacobian matrix f, is nonsingular and de = ~f.'fidue (2-4) ‘Substituting this into (1.2-2) yields aL = LE ~ LEY.) du. 25) The derivative of L with respect to w holding f constant is therefore given by6 STATIC OPTIMIZATION =P Uff wo FUP "Ly (1.2.6) where f-7 means (f;')". Note that aL au (1.2.7) In order that dL equal zero to first order for arbitrary increments du when df = 0, we must have Ly ~ FIf AL, = 0. (12-8) This is a necessary condition for a minimum. Before we deriv. condition, let us develop some more insight and a very valuable tool by examining two more ways to obtain (I.2-8) Write ( ) and (1.2-3) as dt] _ [ut Lr ]far df fe fy Sd, equations defines a stationary point, and it must have a solution [dx du"]". The only way this can occur is if the (n + 1) x (n + m) coefficient matrix has rank less than m + 1. That is, its rows must be linearly dependent so there exists an n vector A such that (12-9) way [# | =0. (2:10) Then Ly (2-11) Li + Mf, = 0. (12-12) Solving (1.2-11) for A gives AN = -Lif3', (12-13) and substituting in (1.2-12) again yields the condition (1.2-8) for a stationary point. It is worth noting that the left-hand side of (1.2-8) is the transpose of the ‘Schur complement of LT in the coefficient matrix of (1.2-9) (Appendix A)12. OPTIMIZATION WITH EQUALITY CONSTRAINTS. The vector A € R" is called a Lagrange multiplier, and it will turn out to be an extremely useful tool for us. To give it some additional meaning now, Jet du = 0 in (1.2-2), (1.2-3) and eliminate dx to get a2) Therefore ay. Paw so that —A is the partial of L with respect to the constraint holding the control ‘u constant. It shows the effect on the performance index of holding the control constant when the constraints are changed. ‘As a third method of obtaining (1.2-8), let us develop the approach we shall use for our analysis in subsequent chapters. Adjoin the constraints to the performance index 0 define the Hamiltonian function =A, 2-15) H(xy uy A) = L(x, w) + AYO, (12-16) where A&R" is an as yet undetermined Lagrange multiplier. To choose x, 1 and A to yield a stationary point, proceed as follows, Increments in H depend on increments in x, u, and A according to dH = Hi! dx + HS du + HY da (1217) Note that aH Hy =< = fx, uw), - = Sy =f (12-18) so suppose we choose some value of v and demand that H,=0. 2-19) ‘Then x is determined for the given u by f(x, «) = 0, which is the constraint relation. In this situation the Hamiltonian equals the performance index: Apa = (12 Recall that if f= 0, then de is given in terms of du by (1.2-4), We should rather not take into account this coupling between du and dr, so it is con- venient to choose A so that8 STATIC OPTIMIZATION =0. 221 ‘Then, by (1.2-17), inerements dx do not contribute to dH. Note that this yields a value for A given by Mensano (1.2.22) or (1.2-13), If (1.2-19) and (1.2-21) hold, then dL = dH = Hi du. (1.2.23) since H = L in this situation. To achieve a stationary point, we must therefore finally impose the stationarity condition 0 (12.24 In summary, necessary conditions for a minimum point of L(x. «) that also satisfies the constraint fix, u) = 0 are an HH pao, 1.2.25 a f=% (1.2-25a) iH a L, + fTa=0, (1.2-25b) a 1, +f1a=0, (1,2-25e) with HG. 1, A) defined by (12-16). The way we shall often use them, these three equations serve to determine x, A, and u in that respective order. Com- pare the last two of these equations with (1.2-11) and (1.2-12). In most applications we are not interested in the value of A, but we must find its value since it is an intermediate variable that allows us to determine the quantities of interest, u, x, and the minimum value of L. ‘The usefulness of the Lagrange-multiplier approach can be summarized as follows. In reality de and du are not independent increments, because of (1.2- 4). By introducing an undetermined multiplier 4, however, we obtain an exira degree of freedom, and A can be selected to make dx and due behave as if they were independent increments. Setting independently to zero the partials of H with respect to all arguments as in (1.2-25) therefore yields 4 stationary point. (Compare this with Example 1.1-2.) By introducing Lagrange multi- pliers, we have thus been able to replace the problem of minimizing L(x, «) ee1.2. OPTIMIZATION WITH EQUALITY CONSTRAINTS 9 subject to the constraint f(x, u) = 0 with the problem of minimizing the Hamiltonian H(x, u, A) without constraints Conditions (1.2-25) determine a stationary point. We are now ready to derive a test that guarantees that this point is a minimum. We shall proceed as we did in Section 1.1 Write Taylor series expansions for increments in L and f as at =U aa] | + sae ave raed cae (12-26) a=, al] + Has" aenlf fo] +08). where f au ax and so on. (What are the dimensions of f,.?) “To introduce the Hamiltonian, use these equations to see that vf] ue maf] 0 and BOB = 0, we can invert R + BYOB and so the optimal control is (R + BYOBY* BTQe «ay Compare this with (11) in Example 1.22, Using (12) in (4) and (9) gives the optimal-state and multiplier values of x= “= BIR + BYOB)" BQ) c, a3) (2 — OBR + BQBY' BQ) c a4) By the matrix inversion lemma (Appendix A) A= (O14 BRIBE € «sy if [Ql * 0. These reduce to the results of Example 1.2-2 in the scalar case, To verify that control (12) results in @ minimum, use (1.2-31) to determine that the ‘constrained curvature matrix is6 STATIC OPTIMIZATION LL, = R + BYOB. 06) which is positive definite by our restrictions on R and @. ‘Using (12) and (13) in (1) yields the optimal value LY = Je'lQ ~ OBR + BOB BQ} c, a Lr = 4TH, is) so that wey wa ae Example 1.2-4: | Multiple Constraints Find the shortest distance between the parabola yeattbetd w tnd the straight line +e a See Fig. 1.23 In this problem there are two constraints, flay) = 95 ~ ad ~ bx, d= 0 @ and “ Filey Ya) = Ya ‘here (x, ¥,) is @ point on the parabola and (x,y) is a point on the line. Let us use a the performance index one-half of the distance between these points squared: Lexy. Yea) = Hy EP + HO, = 9 6 To solve this proble we could proceed by defining ae ee constraint makes this a little messy. Instead, we shall use scalars Introducing one Lagrange multiplier for each constraint, the Hamiltonian is1.2. OPTIMIZATION WITH EQUALITY CONSTRAINTS 17, year+ bx td bey) FIGURE 12-3. Mu iple constraints, WH, F + Hy — 9? +A), = acd ~ bd) + Ad o whence for a stationary point we require Hy = ~~ ahve, ~ ba, = 0. ® fone a Ay =O, ° co) an 0, a2) 3) Solving (12) for y, yields18 STATIC OPTIMIZATION y= art tbe +, as From (9) and (11), we get Aegon eyn as) and using (14) and y, = x, + ¢ from (13) results in ayo tart tbn td 6) whence x= Had + (b+ Nx td-o. an According to (10) and (11), A, so by (15) and then (17) ay 2a on AL= Mart + (b= Dnt d 0) «gy Finally, note that (8) is just (Gar, + (b= IV, = 0 a9 Rar, +b - Ward + ~ Dy td-0= 0. 20) ‘The cubic equation (20) must be solved for the optimal x, for given a, by dy ¢. If the line crosses the parabola, the intersection(s) will be the optimal solutions. (Then A, = A; = 0. Why?) Otherwise there will be a single closest pair of distinct points Gy, yO. Ge ¥2). Once x; is Known, x5, yy, and yp are determined by (17), (14), and (5), respectively. Substitating these optimal valucs in (S) allows us to find the mini- mum distance V2L*. . Effect of Changes in Constraints Equation (1.2-28) expresses the increment dL in terms of df. de, and du. In the discussion following that equation we let df = 0, found dx in terms of du, and expressed dL in terms of du. That gave us conditions for a stationary point (H, = 0 and H, = 0) and led to the second-order coefficient matrix L,, in (1.2-31), which provided a test for the stationary point to be a con- strained minimum, In this subsection we are interested in dL as a function of an increment df in the constraint. We want to see how the performance index L changes in response to changes in the constraint f if we remain at a stationary point. We are therefore trying to find stationary points near a given stationary point12 OPTIMIZATION WITH EQUALITY CONSTRAINTS 19 See Fig. 1.2-4, which shows how the stationary point moves with changes ing. {AL the stationary point (u, x)* defined by flx, u) = 0, the conditions H, = 0, H, = 0, and H, = O are satisfied. If the constraint changes by an inerement so that f(x, u) = df, then the stationary point moves to (w+ du, x + dx). The Partials in'(1.2-25) change by dH, = df =f, dx +f, du, (1.2.32a) dH, = H,, de +H, du +f da, (1.2-32b) dH, = Hy. dx + Hy, du +f dd. (1.2-320) In order that we remain at a stationary point, the increments dH, and dH, should be zero. This requirement imposes certain relations between the changes dx, du, and df, which we shall use in (1.2-28) to determine dL. as a function of df To find dr and du as functions of df with the requirement that we remain at an optimal solution, use (1.2-32a) to find de =f.) df — ff, duu (1.233) and set (1.2-32b) to zero to find FIGURE 1.2-4 Locus of stationary points as the constraint varies,20 STATIC OPTIMIZATION dn ATH, de + H,, do) (1.234) Now use these relations in (1.2-32e) to obtain dH, = Hy ~ Hake fa ~ FEE He + PP AS fe) ee + Hy ~ FF THA P= 0 so that du = Why Hye ~ FFA) fe! af = —C df. (12-35) Using (1.2-35) in (1.2-33) yields de = + FF ME) Has ~ FFM Sc af SHU + S.C) of. (1.2:36) Equations (1.2-35) and (.2-36) are the required expressions for the in- crements in the stationary values of control and state as functions of df. If iL1,| + 0, then dx and di can be determined in terms of df, and the existence of neighboring optimal solutions as f varies is guaranteed. To determine the inerement dL. in the optimal performance index as a function of df substitute (1.2-35) and (1.2-36) into (1.2-28), using 11, = 0, H, = O since we began at a stationary point (u, x)*, The result is found after some work to be dL = -X" df + $f", 1 CTLL,C) df + 03). (1.237) From this we see that the first and second partial derivatives of L*(x, u) with respect to f(x, u) under the restrictions dH, = 0, dH, = 0 are au F (12-38) = fH fe! — CULL. (1.239) Equation (1.2-38) allows a further interpretation ofthe Lagrange multiplier: it indicates the rate of change of the optimal value of the performance index with respect to the constraint13 NUMERICAL SOLUTION METHODS — 21 1.3 NUMERICAL SOLUTION M HODS Analytic solutions for the stationary point (u, x)" and minimal value L* of the performance index cannot be found except for simple functions L(x, «) and fix, «In most practical cases, numerical optimization methods must be used. Many methods exist, but steepest descent or gradient (Luenberger 1969, Bryson and Ho 1975) methods are probably the simplest ‘The steps in constrained minimization by the method of steepest descent are (Bryson and Ho 1975): Select an initial value for u Determine x from fix, u) = 0. Determine A from A = ~f,"L. Determine the gradient vector H, = L, + fA. Update the control vector by Au = —KH,, where K is a positive scalar constant (o find a maximum use Au = KH,) 6. Determine the predicted change in the value of L, AL = HY Aw = KH1H,, If AL is sufficiently small, stop. Otherwise go to step 2, There are many variations to this procedure. If the step-size constant K is too large, then the algorithm may overshoot the stationary point (w, x)* and convergence may not occur. The step size should usually be reduced as (a, x)® is approached, and several of the existing variations differ in the approach to adapting K. ‘There are many software routines available for unconstrained optimization. Example 1.3-1 shows how simple it is 10 use these to solve the constrained minimization problem. Example 1.3-1: Use of MATLAB Optimization Toolbox Routines “The numerical solution of the constrained optimization problem of minimizing L(x. u) suibject to f(x, u) = 0 can be obtained using the MATLAB function consizn available under the Optimization Toolbox. This function takes in the user-defined subroutine Jfunctan, which computes the value of the funetion, the constraints, and the initial ‘conditions, We illustrate the use of the function by numerically solving the Examples 12-2 and 12-4 ‘a. For Example 1.2-2, let a = 3, b= 2,m= le = 1. Here, Xa, XQ) =u, Acconding to equations (1) and (2) in Example 1.2222 STATIC OPTIMIZATION we, XQ) FX) = + G(X) = XA) + XQ) = 1 = 0. ‘The subroutine funci.m is shown in Fig. 13-2. b, For Example 1.2-4, let a= 1. b= Ive= and d = 2. Here XA) =, xia) According (0 equations (1), (2), and (3) in Example 12-4 xa)? X(3) = XY? Seem G00 = XG) — XL? = XU) = G00) o () ~ XQ) + 1 = 0. ‘The subroutine functim is shown in Fig. 1.33 ‘The program to obtain the solution is given in Fig. 1.3-1. The initial guess XO is chosen to satisfy the constraints, The variable oprions(13) is changed to indicate the number of equality constraints, The optimum solution is obtained as x* = 0.6923, u* = 0.3077. For Example 1.2-4, the optimum solution is (xf, yf) = (0, 2), Gxt. y¥) = (15, 0). . PROBLEMS Sec L.1-1: Find the critical points w (classify them) and the value of L(w*) in Example 1.1-1 if ag wo=| 5} st=10 U1 Sketch the contours of L and find the gradient L,, 1-2: \d the minimum value of Lo, 2) fay td + 3x, a Find the curvature matrix at the minimum. Sketch the contours. showing the gradient at several pointsPROBLEMS — 23 8 Constrained optimization using MATLAB function constr (.) % for Example 1.2-2 and 1.2-4 XO=(0,2]; & Initial quess options (13)=1; § Indicate that there is one x=consty(’funct ', X0,options| FIGURE 1.31 MATLAB program for numerical solution for Examples 1.2-2 and 12-4. juality constraint % Subroutine for use with MATLAB function constr(.) & for Example 1.2-2 § FO0 is the value of the function and gix) is the constraint function [£,¢]=Eunet (x) ¢ aye 8 + x@ye 708 (CO 4KQ)-1; FIGURE 13-2 MATLAB subroutine for Example 1.2-2. % Subroutine for use with MATLAB funetSon conste(.} & for Example 1.2-4 @ £00) is the value of the function and g(a the constraint function [f,g]=Eunct 1X) 7 £ aa a2 (x) -K2N)2 + CRE -RIAN"2D 725 (3) KG 2-K(1)-2; (ayox(2ii: FIGURE 13-3 MATLAB subroutine for Example 1.2.4, 1.1-3: Failure of Test for Mi ‘minimum at the origin. ‘a. Verify that the origin is a critical point. b, Show that the curvature matrix is singular at the origin. ‘e. Prove that the critical point is indeed a minimum, jimality. The function f(x, y) = 2° + y* has a24 STATIC OPTIMIZATION Closest Point of Approach. A ship is moving at 10 miles per hour on a course of 30° (measured clockwise from north, which is 0°). Find its closest point of approach to an island that at time 1 = 0 is 20 miles east and 30 miles north of it. Find the distance to the island at this point, Find the time of closest approach. Shortest Distance Between Two Points. Let P, = (xy. ) and Py Gx y,) be two given points. Find the third point P, = (x, yy) such that dy 4, is minimized, where d, is the distance from P, 10 P, and dy isthe distance from P, to Ps. 1.2-3: Meteor Closest Point of Approach. A meteor is in a hyperbolic orbit described with respect to the earth at the origin by Pe Find its closest point of approach to a satellite that is in such an orbit that it hhas a constant position of (x, y\). Verify that the solution indeed yields a minimum, 1.2-4: Shortest Distance Between a Parabola and a Poi ‘moving along the path 1. A meteor is yar 3y- ay A space station is at the point (x, ») = (2, 2). ‘a, Use Lagrange multipliers to find a cubic equation for x at the closest point of approach. bb. Find the closest point of approach (x,y), and the distance from this point to 2, 2), 1.2.5: Rectangles with Maximum Area, Minimum Perimeter. 4. Find the rectangle of maximum area with perimeter p. That is, maximize Lx, y) = ay “ subject to fis, y) = 2x + yp =O. % b. Find the rectangle of minimum perimeter with area a?, That is, minimize Loy y) = 2x + 2y @)PROBLEMS — 25 subject 10 foxy) = ay — @ = 0. @ ¢. In each case, sketch the contours of L(x, y) and the constraint, Optimization problems related like these two are said t0 be dual 1.2-6: Linear Quadratic Case. Minimize Find x, w, a*, L* 1.2-7: Linear Quadratic Case. In the LQ problem define the Kalman gain K 2 BOB + Ry BO wo a Express u®, A 0 and L* in terms of K. b. Let * Q ~ QBB'QB + ky BQ 2 so that L* = e'S,e/2. Show that QU — BK) = (I~ BR)" QU ~ BK) + K'RK. @ Hence, factor L* as a perfect square. (Let VQ and VR be the square roots of Q and R.) . Show that = (Q' + BRB) cy 1.28: Geometric Mean Less Than or Equal to Arithmetic Mean, a, Show that the minimum value of °y*c* on the sphere x? + y? + © is (713). . Show that the maximum value of 2° + (PB) is P. + 2 on the surtace -°y26 STATIC OPTIMIZATION ©. Generalize part a or b and so deduce that, for a, > 0, (ayay--- a)" (a, +a, toe + ain Note: The problems in parts a and b are dual (Fulks 1967). 1,2-9: Find the point nearest the origin on the line 3x + 2y + dy 3 = 4 1.2:10; Rectangle Inside Ellipse. 8, Find the rectangle of maximum perimeter that can be inscribed inside an ellipse. That is, maximize 4(x + y) subject to constraint <°/a? + y°/b° 1 b. Find the rectangle of maximum area 4xy that can be inscribed inside an ellipse.2 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS We are now ready to extend the methods of Chapter 1 to the optimization of a performance index associated with a system developing dynamically through time. Its important to realize that we shall be making a fairly subtle change of emphasis. In Chapter 1, the focus of our attention was initially on the performance index, and we introduced the notion of constraints as the discussion proceeded. In this and subsequent chapters we are forced to begin with the constraint equations, since these turn out to be exactly the dynamical equations describing the system. These relations are fixed by the physics of the problem. The performance index is selected by the engineer to make the system behave in a desired fashion. In Section 2.1 we derive the general solution of the optimization problem for discrete-time systems. In Section 2.2 we discuss the very important special case of linear systems with u quadratie performance index, dealing with the two situations of fixed final state, which yields an open-loop control, and free final state, which yields a closed-loop contol, In Section 2.3 we show how to apply these results to the digital control of continuous-time systems. Some connections with classical root-locus design are given in Section 2.5. 2.1 SOLUTION OF THE GENERAL. OPTIMIZATION PROBLEM. SCRETE Problem Formulation Suppose the plant is described by the very general nonlinear discrete-time dynamical equation co28 OPTIMAL CONTROL OF DISCRETE-TIME Keer =f Quay with initial condition xq, The superscript on function f means that it can in general be time-varying. Let the state x, be an n vector and the control input ty be an m vector. Since (2.1-1) determines the state at time k + 1 given the control and state at time &, itis our constraint relation, Clearly. f © Suppose there is an associated scalar performance index given in the gen eral form J, = GN, xy) + SD LO wade (21-2) where [i, NJ isthe time interval over which we are interested in the behavior of the system. dV, x,) is a function of the final time NV and the state at the final time, and LY, 4) is an in-general time-varying function of the state and control input at each intermediate time & in |i, N}. The optimal control problem isto find the control uf on the interval {i, V] that drives the system Q.1-1) along a trajectory xf such that performance index (2.1-2) is mini- mized Example 21-1: Some Useful Performance Indices To clarify the problem formulation, itis worthwhile to discuss some common petfor- mance indices that we can select for the given system (2.1-1). 4. Minimum-Time Problems Suppose we want to find the control u, to drive the system from the given initial state 4%, 10 a desired final state x R° in minimum time. Then we could select the perfor ‘mance index. gene w and specify the boundary condition wee @ In this case = Nand £ = 0, or equivalently & = 0 and L = 1 b. Minimum: Fuel Problems To find the scalar control u, to drive the system from x, oa desired final state x at a fixed time N using minimum fuel, we could use J= D lu. e)2.1 SOLUTION OF THE GENERAL DISCRETE OPTIMIZATION PROBLEM 29 since the fuel burned is proportional to the magnitude of the control vector. Then = and L! = ly The boundary condition xy =x would again apply. 6 Minimun-Energy Problems Suppose we want 10 find 1, 10 minimize the energy of the final state and all inter- ‘mediate states, and also of the control used to achieve this. Let the final time N again be fixed. Then we could use ae sey + $5 ite, + rat, ® where g. rand s are scalar weighting factors. Then ¢ = Ssrlxy amd L = Mgafay + rf.) are quadratic functions. “Minimizing the encrey corresponds in some sense t0 Keeping the state and the contol elose to zero. If is more important to us tha the intermediate ate be small, then we should choose q large to weight it heavily in J, which we are uying ainimize. IF itis more important that the control energy be small. then we should solect a large value of r I we are more interested in a small final state, then s should be large, For more generality, we could select weighting marices Q. RS instead of scalars. . At this point, several things should be clearly understood. First, the system dynamics (2.1-1) are given by the physics of the problem, while the perfor- mance index (2.1-2) is what we choose to achieve the desired system re- sponse. Second, to achieve different control objectives, different types of per- formance indices J are selected. Finally, the optimal control problem is, characterized by compromises and trade-offs, with different weighting factors in J resulting in different balances between conformability with performance objectives and magnitude of the required optimal controls, In practice, it is usually necessary to do a control design with a trial per- formance index J, compute the optimal control uf, and then run a computer mulation to see how the system responds to this u?. If the response is not acceptable, the entire process is repeated using another J with different state and control weightings. After several repetitions have been done to find an acceptable 1 this final version of uz is applied to the actual system, Problem Solution Let us now solve the optimal control problem for the general nonlinear system Q.1-1) with associated performance index (2.1-2). ‘To determine the optimal control sequence uu... uf, minimizing 4, we shall proceed basically as we did in Chapter 1, using the powerful Lagrange-multiplier approach. Since there is @ constraint function f*(t, ) specified at each time & in the interval of interest i, NJ, we shall also require30 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS Lagrange multiplier at each time. Each constraint has an associated La- grange multiplier ‘Thus, let A, € R’, and append the constraint (2.1-1) to the performance index (2.1-2) to define an augmented performance index J’ by FO = BN ay) + D Ay, wd + ABA CP MO = ee Ik 2-3) Note that we have associated with f* the multiplier A,,,, not A,. This is done with the benefit of hindsight, as it makes the solution neater. Defining the Hamiltonian function as FBC, Uy) = LAs Wy) + AT fies ds (2.14) we can write HIN, xy) — Any + HG, w) + SD Ue uy) — AE 21-5) where some minor manipulations with indices have been performed. Note that the Hamiltonian is defined slightly differently than in Chapter 1, since wwe did not include x,., in HY. Furthermore, a Hamiltonian is defined at each time k ‘We now want to examine the increment in J" due to increments in all the variables x,, A. and u,. We assume the final time N is fixed. According to the Lagrange-multiplier theory, at a constrained minimum this increment dJ’ should be zero. Therefore, write (buy — Aud dey + CHL)" ds, + H,)* du, + Ue, = AT de, + HB)" dual + Di a day 21-6) where ae «ax and so on, Necessary conditions for a constrained minimum are thus given by2.1 SOLUTION OF THE GENERAL DIS RETE OPTIMIZATION PROBLEM 31 aH ao Sica 21a wa 1-70) — Wek em) be 1 ere) which arise from the terms inside the summations and the coefficient of du, and (22-4) ds =0. 2.1-8a) (2) a-0 2.18) (In fact, examining (2.1-3) and (2.1-4) shows that A, does not occur in J’. We hhave defined it in such a manner that the lower index in (2.1-7b) can be taken as i, instead of i + 1, solely as a matter of neatness.) These conditions are certainly not intuitively obvious, so we should discuss them a litle to see what they mean, First, compare (2.1-7) with the conditions for a static minimum (12-25). hey are very similar, except that our new conditions must hold at each time in the interval of interest [i, N— I], since x,, uj, and A, are now sequences. (2.1-7e) is called the stationarity condition. Writing (2.1-7) explicitly in terms of L* and f* using (2.14) yields the formulation in Table 2.1-1. Equality (2.1-9a) is just the constraint equation, fr system equation. It is a recursion for the state x, that develops forward in time. Evidently, (2.1-96) is a recursion for A, that develops hackward in time! ‘The (fictitious) Lagrange multiplier is thus a variable that is determined by its own dynamical equation, It is called the costate of the system, and (2.1 9b) is called the adjoint system. The system (2.1-9a) and the adjoint system. (2.1-9b) are coupled difference equations. They define a nvo-point boundary value problem. since the boundary conditions required for solution are the initial state x, and the final costate Ay. These problems are in general ex- tremely difficult to solve. We consider some examples later. ‘The stationarity condition (2.1-9c) allows the optimal control 1, to be ex pressed in terms of the costate. We therefore have a rather curious situation’ we do not really care what A, is, but this method of solution requires us to find A, as an intermediate step in finding the optimal control. We have not yet discussed (2.1-8). The first of these equations holds only at final time & = 1N, whereas the second holds only at initial time & = i. They are not dynamical recursions like (2.1-7a) and (2.1-7b). These two equations32 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS TABLE 21-1 Discrete Nonlinear Optimal Controller ‘Shstem model: Ben SS) REE Performance index: J.= 6.) +S Lo) Honitnian heap Optimal controle: State equation: aH _ py 2. ee FH) (2.1-9a) Costate equation: noth (E eth ara Stationarity contion -m-(L) +8 (2.1-9) Boundary conditions: ( (Syne ax, * ax specify the split boundary conditions needed to solve the recursions (2.1-9). ‘Two possibilities exist for each of these equations. If the initial state «, is fixed, then ds, = 0, so that (2.1-8b) holds regardless of the value of #7. On the other hand, in the case of free initial stare, dx, is not zero, so (2.1-8b) demands that an 0. (2.1-10) In our applications the system starts at a known initial state x. Thus, the first2.1 SOLUTION OF THE GENERAL DISCREI © OPTIMIZATION PROBLEM 33 ‘ease holds, dt, = 0, and there is no constraint on the value of #,. We shall therefore ignore (2.1-8b) and use as the initial condition the given value of We do need to deal with two possibilities for the final state x. In the case of a fixed final state we shall use the desired value of xy as the terminal condition, Then 2, is not free to be varied in determining the optimal solution and dx ~ 0, so that (2.1-8a) holds. On the other hand, if we are not interested in a particular value for the final state, then xy can be varied in determining the optimal solution. In this case dry is not zero. For this free-final-state situation, (2.1-8a) demands that oe 5 2.1 os Or ‘Then, the terminal condition is the value (2.1-11) of the final eostate Ay. In summary, the initial condition for the two-point boundary-value problem (2.1-9) is the known value of x,. The final condition is either a desired value of x, oF the value (2,1-11) of Ay. These comments will become clearer as we proceed. ‘An Example In order to develop some feel for the theory we have just derived, let us consider an example. We shall see that the solution of the optimal control problem is not straightforward even in the simplest cases, because of the two- point nature of the state and costate equations, but that once the solution is obtained it imparts a great deal of intuition about the control of the system, ‘We also show how to run software simulations to test our optimal control designs. Example 2.1-2: Optimal Control for Sealar Linear System Consider the simple linear dynamical system Bir = at, + Pit, wo where lowercase a and b are used t© emphasize that we are dealing with the scalar ‘case. Let the given initial condition be x. Suppase the interval oF interest is [0, N] and that we are concerned with minimizing control energy so that for some scalar weighting factor r Let us discuss two eases, corresponding to two ways in which we might want the system to behave,34 OPTIMAL CONTROL OF DI -RETE-TIME SYSTEMS 4. Fixed Final Stare First, suppose we want to make the system end up at time k = N in exactly the particular (reference) state ry yah 8 To find the optimal control sequence 4, u.---, uy-» (note that xy does not depend fon us) that drives (1) from the given x to the desired x, =r, while minimizing (2), we can use Table 2.1-1. First, let us compute (2.1.9), ‘The Hamiltonian is Walt ant pa + Alar, + buy), @ so the conditions (2.1-9) are 6) © ant Oa TE + hse o Solving the stationarity condition (7) for u, in terms of the costate yields Bas cl If we can find the optimal A,, we can therefore use (8) to find the optimal contro To find A, eliminate u, in (5) using (8). Then Bay = aK 8 — Yes o a9) is the ratio of “control effect” to control weighting. Equation (6) is a simple homogeneous difference equation, with solution given by A= ah tay ap2.1. SOLUTION OF THE GENERAL DISCRETE OPTIMIZATION PROBLEM 35 This an all well and good, but we do not know Ay. To find it, proceed as follows. Use (9) to pet ay = a — a" Ay ay ‘This can be viewed as a difference equation with a forcing function of —yAya" * ', ‘0 the solution in terms of, is ays ates — Dah aa) = aby ~ yaaa? Sa ™ 3) Using the formula for the sum of a geometric series we have cate, — yy gue Ee) Heo aby — yaya? oxy — yaya DED any — wages «4 ‘The state at time & is thus a linear combination of the known initial state and the as yet unknown final costate. To find Ay, We need to make use of the boundary conditions 2.18). Since 4, is fixed, de, = 0 and (2.1-8b) is satisfied. Since we are also demanding the fixed final state (3), we have di, = 0 so that (2.1-8a) is satisfied, In words, the algorithm cannot vary either x, or xy in determining the constrained minimum for this, problem. According to (14), the final state is expressed in terms of the unknown Ay as (as) where ea daa) lay 16) (Defining auxiliary variables is a good trick for making our results look neater than they actually are!) Solving for Ay in terms of the given x, and the known desired iy = ry yields 1 ln ~ an [Note that ay isthe final state of the plant (1) in the ease of zero input. The final36 OPTIMAL CONTROL OF DISCRETE-TIME SYSTEMS ccostate Ay is thus proportional 10 the desired final state ry minus the final state ax, which the system would reach by itself with no control; it makes sense that the control required to drive x, 10 xy = ry should depend on this difference! ‘AL this point, we cam determine the costate 10 be, using (11). A= EU aon as) and the optimal control to be, using (8), agua a9) ‘This is the solution to our problem, and uf, for & = 0, 1...,N~ 1 will drive x, 10 ty = Fy while minimizing the control energy 2). I is worthwhile to examine wa ftle bit, Note that (19) can be written ae at Ey a a 20) so that in the case of fixed final state the optimal control is independent of the control weighting r, It should also be stressed that uf given by (19) is an open-loop control. It depends on the inital and final states, but not on intermediate values x, of the state, ‘This is discussed further in Section 2.2 For completeness, let us determine the optimal state ajectory x? and performance index Jj under the influence of uf. ‘Substituting (20) in (1) yields uaa) aa) dry — ate en ‘The first observation worthy of note is that the optimal state trajectory af is inde- pendent of both r and b! Equation (21) is a dynamical system with forcing function given by the second term, 50 its solution is le ats 3 arta oy Using the formula for the sum of a geometric series and simplifying yields =e) ws Bt = aby + (ry — aed oe en or2. SOLUTION OF THE GENERAL DISCRETE OPTIMIZATION PROBLEM 37 (= a2 May + = a ry ae) 4) Note that xf xy and xf = ny as required. Infact, xf is a time-varying linear com- bination of x, and r, containing proportionately less of xy and more of ry as & increases from 0 t0 N. The optimal performance index is found by using (20) in (2) 5) Using the formula for the sum of a geometric series and simplifying results in hey an 26) ‘Thus the farther the plant zero the larger the cost J put response «x, is from the desired final state ry 1b, Free Final State Now suppose that we still desire the system to end up in state vy al time xy, but that swe decide to choose quite a different method for ensuring this. Now, we do not need ‘ny t0 be exactly equal to ry, but we only need x 10 be clase to ro Let us therefore make the difference xy ~ ry small by including it in the performance index, so that (2) becomes, Gy = rr 5S a en Now the optimal control will attempt to make [xy ~ ra smatl while also using low ‘control energy. (As we shall see, this will not guarantee that, will exaetly equal ry.) In this case b= May es) but and L! are not changed. The Hamiltonian is still given by (4), and conditions (2.1-9) are still (5)-(7), This means that all of our work in part a up through (15) is tunchanged by adding to J the final-state weighting ferm, ‘The only change is in the ‘boundary conditions (2.1-8), ‘Since x is not constrained to take on an exact value, it can be varied in determining the optimal control, Hence dr, # 0, and so according to (2.1-8a) we must have ab ye es 29) “The final costate is now related to the final state xy und the desired final state ry by