When Is a Linear Control System Optimal?

The purpose of this paper is to formulate, study, and (in certain cases) resolve the
R. E. K A L M A N
Research Institute for Advanced Studies
(RIAS), Baltimore, M d .
Inverse Problem of Optimal Control Theory, which is the following: Given a control
law, find all performance indices for which this control law is optimal.
Under the assumptions of (a) linear constant plant, (b) linear constant control law,
(c) measurable state variables, (d) quadratic loss functions with constant coefficients,
(e) single control variable, we give a complete analysis of this problem and obtain various
explicit conditions for the optimality of a given control law. An interestingfeature of the
analysis is the central role of frequency-domain concepts, which have been ignored in
optimal control theory until very recently. The discussion is presented in rigorous math-
ematical form.

Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019

The central conclusion is the following (Theorem 6): A stable control law is optimal
if and only if the absolute value of the corresponding return difference is at least equal
to one at all frequencies. This provides a beautifully simple connecting link between
modern control theory and the classical point of view which regards feedback as a means
of reducing component variations.

Introduction (c) All state variables can be directly measured.

T H E G R E A T interest in optimal control theory in
recent years has brought with it a great deal of criticism as well.
There are nagging doubts-—particularly strongly felt b y engi-
neers—that the theory is overidealized, hence impractical. I t is
argued that the choice of the performance index to be optimized
is arbitrary and subjective, perhaps only a matter of taste. If so,
then it is pointless to devote too much effort to finding a control
law which is the best in some narrow, individualistic sense-—it
would be more sensible to see approximate control laws which
are not rigidly tied to a single performance index.
These are naive objections and cannot be accepted from the
scientific point of view { l } . 1 They do contain, however, an im-
portant element of truth and suggest the following problem:
Instead of asking for a control law corresponding to a given per-
formance criterion, one might seek to determine all performance
criteria (if any) for which a given control law is optimal. W e might
thereby discover general properties shared b y all optimal control
laws. W e might be able to separate control laws which are opti-
mal in some sense from those which are not optimal in any sense.
If this plan of attack is successful, then the preceding objections
will be irrelevant since the most important aspects of optimality
will hold independently of the specific choice of a performance
index { 2 } . This is what we shall do in the present paper.
The problem outlined in the foregoing has long been known in
the calculus of variations, the "parent" of optimal control
theory. It is the inverse problem of the calculus of variations [1,
chapter 10, §1]: 2 Find all variational problems (if any) whose
Euler equations are satisfied b y a given family of curves. This is
a difficult problem which has been the subject of numerous in-
vestigations { 3 } .
B y analogy, we shall speak of the inverse problem of optimal
control theory. Very little is known today about this problem.
W e shall not attempt here t o give a complete analysis, but shall
consider only a special (though large) class of problems which is
of great interest in control engineering. Our main assumptions
will be the following:
(а) T h e plant is governed b y a linear differential equation with
constant coefficients.
(б) T h e control law is linear and constant.
1 Numbers in braces refer to special comments collected at the end
of the paper.
2 Numbers in brackets designate References at end of paper
C o n t r i b u t e d b y t h e A u t o m a t i c Control Division of THE AMERICAN
SOCIETY OP MECHANICAL ENGINEERS a n d p r e s e n t e d at the
Automatic Control Division, Minneapolis, Minn., June 19-21, 1963.
Manuscript received at ASME Headquarters, March 13, 1963.
(d) The loss functions are quadratic forms with constant co-
efficients in the state and control variables.
(e) There is only one control variable.
The first four of these assumptions are made in order to obtain
explicit results. The last assumption can be removed, but only at
the cost of a more refined analysis [2].
It is hoped that this paper will reduce the " g a p " which exists
today between optimal control theory and conventional control
engineering practice.
There are many interesting and unexpected results. Let us
emphasize some of these already here:
1 Every control law, stable or unstable, is optimal in some
sense. T o avoid such a trivial conclusion, the class of performance
indices must be restricted. If—in addition to (d)—we make the
usual assumption that the loss function does not contain cross
products of the state variables with the control variable [3-8], it
turns out that a feedback system is optimal if and only if the abso-
lute value of the return difference is at least 1 at all frequencies
(Theorem 6). This is a severe requirement.
The preceding criterion is well known in classical feedback
theory: it means that sensitivity to component variations in the
forward loop is diminished (not accentuated) b y feedback. Thus
a single criterion assures insensitivity with respect to component
variations as well as optimal performance in the dynamical sense.
This is a highly important result in the pure theory of control.
In addition, it provides theoretical support for certain intuitive
design procedures long used in control engineering.
2 Conditions of optimality can be expressed most con-
veniently b y means of frequency-domain formulas (Theorem 5).
This is a new trend in optimal control theory, heretofore rigidly
confined to the time domain { 4 } . I t is hoped that renewed in-
terest in frequency-domain techniques will tend to break down
the artificial dividing lines between conventional and modern
control theory.
3 T o obtain a "tight" control loop, the term representing con-
trol error in the definition of the loss function must be taken to be
large relative to the term representing control energy. This leads
to optimal control with high loop gains; the closed-loop poles of
the system are then either near the open-loop zeros or near a
Butterworth configuration (Theorem 11). Such a system has the
attributes usually demanded from good control system design
(moderate overshoot in response to a step input, pronounced
low-pass frequency characteristic).
4 There have been many attempts [9-10] to prescribe uni-
Joint
versal closed-loop pole configurations for optimal single-loop con-
trol systems independently of the open-loop transfer function.

Journal of Basie Engineering Copyright © 1964 by ASME

MARCH 1964 / 51
These prescriptions are usually obtained from experimental matrices, lower-case letters vectors, and lower-case Greek letters
analog-computer studies. They are rigorously confirmed by the scalars.)
present investigation only in the case of high loop gains, see (3) The control law is given by
above. When a high loop gain is not possible or not desired, the
optimal control law will always depend to some extent on the H(t) = -(k, x(t)),' (2)
plant which is being controlled and no universal prescriptions can
where k is a real, constant «-vector of feedback coefficients, and
be made. In such cases, it is strongly recommended that the
(k, x) is the inner product. B y abuse of language, it will be often
calculation of the control law proceed b y the standard methods of
convenient to talk of a "control law k" instead of the "control law
optimal control theory, which take into account the properties of
-(k, x>."
the plant in a quantitative way { 5 } .
Substituting the control law (2) into (1) we obtain the free
differential system representing the plant under closed-loop
Acknowledgments control:
The research, of which this paper forms a part began as a result dx/dt = (F - gk')x = Fkx. (3)

Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019

of a conference with Dr. E. B. Stear of the Aeronautical Systems
Division, Air Force Systems Command, Mr. P. A. Reynolds, of
the Cornell Aeronautical Laboratories, and their associates. The
idea of using frequency-domain methods arose during conversa-
tions with Professor S. Lefschetz. Many improvements in the
paper were suggested by the author's colleagues at RIAS, par-
ticularly Dr. A. A. Frederickson. The work was supported in
part by the U. S. Air Force under Contracts AF 49(638)-1026
and AF 33(657)-855 and by the National Aeronautical and Space
Administration under Contract NASr-103.
Subdivision of the Problem
Implementing the plan outlined in the introduction will require
a rather length}' analysis. But the details of the arguments must
not be allowed to obscure the ideas involved. For this reason, it
is convenient to break up the discussion into six separate prob-
lems. The study of these problems will lead to the solution of the
Inverse Problem of Optimal Control Theory under certain clearly
defined conditions.
Problem A. What optimization problems lead to a constant,
linear control law?
Problem B. How is this optimal control law explicitly com-
puted?
Problem C. When is the optimal control law stable?
Problem D . What algebraic conditions are necessary and
sufficient for the existence of a constant, linear, stable (optimal)
control law?
Problem E. What is the most convenient form of the preceding
conditions?
Problem F. What can be said about the pole-zero pattern of
the closed-loop transfer function of an optimal system?
These problems are intentionally vaguely phrased; merely a
guide to, and not the main object of, the following analysis.
Mathematical Description of the Plant
It is assumed that the plant (or control object) is a finite-dimen-
Bional, continuous-time, constant, linear dynamical system [11],
It is also assumed that only a single input to the plant is available
for control purposes, and that all control variables can be meas-
ured directly { 6 } . (Assumptions (a-c) in the Introduction.)
The purpose of control is to return the state of the plant to the
origin after it has been displaced from there by some extremal
disturbance: we are concerned with a regulator system.
The control law is constant and linear. (Assumption (d) in the
Introduction.)
The precise mathematical form of the preceding assumptions is:
The behavior of the plant is represented by the differential
system
dx/dt = Fx + gn(l), (1)
where x is a real n-vector, the stale of the plant; n(t) is a con-
tinuous, real-valued function of time, the control function; F is a
real constant n X n matrix; and g is a real constant n X 1 matrix,
i.e., an n-vector. (Similar notations will be used throughout the
paper without special comment. In general, capitals will denote
Here the prime denotes the transpose. (The inner product
(k, x) may be written also as k'x.) The abbreviation 1<\ = F —
gk', where k is any ?i-vector, will be used frequently.
The resolvent of the matrix F will be frequently used. This
matrix-valued function of the complex variable s is defined by
$ ( s ) = (si - F)"> (I = unit matrix). (4)
Clearly every element of $ ( s ) is a rational function of s. 5>(s) may
be regarded also as the formal Laplace transform of the matrix
function exp Ft {7}.
With the aid of (4), it is easy to express transfer functions in
vector-matrix notation. For instance, let t] be the scalar k'x.
Then the transfer function from n to ?7 is given by
7i(s)/ii(s) = k'$(s)g. (5)
We shall repeatedly use the notation ip(s) = del (si — F) for
the characteristic polynomial of the matrix F.
In analogy with Fk = F — gk' we have then also the notations
$,.(«) = (si - Fk)~l and Ms) = det(sl - Fk).
It is frequently necessary to manipulate these frequency-
domain expressions in the same way as one manipulates transfer
functions in elementary control theory. T o illustrate the algebraic
steps involved, let us note some common formulas:
tk(s) = det(sl - F + gk') = det[(sl - F)-(I + $(s)gk')]
= \p(s) det(I + $(s)gk') = ^(<0(1 + k'$(s)g),
where the last step follows by a well-known matrix identity.
Thus we have an explicit expression for the rational function
1 + k'<t>(s)g = tk(s)/f(s). (6)
The quantity Tk(s) = 1 + k'<l'(s)g is the so-called return dif-
ference of classical feedback theory [20],
T o express closed-loop transfer functions in terms of open-loop
ones, we proceed as follows:
k'$k(s)g = k'(sl -F + gk')~1g
= k'[(sl - F)(I + $(s)gk'))->g
= k'[I - $(s)gk'/( 1 + *'$(s)<7)]$(«)</
= k'^(s)g-H(s)/^)l
It should be noted that transfer functions—which express in-
put-output relations—are independent of the choice of the (in-
ternal) state variables.
For instance, suppose the (numerical) state vector x is replaced
by
= Tx,
where T is a nonsingular constant matrix. Then the matrices F,
g, k in (1) and (2) must be replaced by
p = TFT~\ $ = Tg, and ic = (T-^'k.
3 T h e minus sign in (2) is a notational convenience and a reminder
that we are usually dealing with negative feedback.

52 / MARCH 1964 Transactions of the AS ME

The transfer function (5) is invariant under this change of coordi-
nates because
k'$(s)§ = k'T~\sl - TFT~1)~1Tg
= k'T~'[T(sI - F)T~l]~lTg
= k'(sl - = k'<f>(s)g.
Remark {4} is of direct relevance here.
For future purposes, it is very important to have a strict
correspondence between time-domain and frequency-domain
methods of system description. The frequency-domain form of
( I ) and (2) is the transfer function k''l'(s)g. However, knowledge
of this transfer function will not uniquely identify the matrices F,
g, and k appearing in (1) and (2) (even disregarding the lack of
uniqueness due to the arbitrary choice of the basis for x—see R e -
mark { 4 } ) unless certain conditions are satisfied. These are
[II]: (a) the plant is completely controllable and (b) the control
law k is completely observable.
The first condition means that all state variables can be affected
by some suitable choice of the control function p(t). The second
condition means that the control function p(t) given by (2) can be
identically zero only if the state is identically zero; in other
words, the control law k is picked in such a way that any change
from zero in the state variables is counteracted b y some control
action.
There is no loss of generality (as far as the problem discussed in
this paper is concerned) in assuming that both of these con-
ditions hold { 8 } .
In practically all the analysis which follows, we must assume
that
The plant is completely controllable.
This condition can be expressed in concrete form b y [11]
rank \g, Fg F"-'g] = n. (A-.1)
On the other hand, the assumption that k is a completely ob-
servable control law, i.e., that [11]
rank [k, F'k, . . ., (F'^-'k] = the minimum of ( 9 ) and the corresponding optimal control law.
is not needed until much later (see Theorem 6).
Mathematical Formulation of the Optimization Problem
Our first objective ought to be to find all optimization problems
which result in a constant, linear control law (Problem A). How-
ever, the general solution of this problem is not known at present.
Therefore we shall merely give some sufficient conditions. In the
next section we will see that these conditions imply that the control
law is constant and linear.
Let us recall the mathematical definition of the dynamic op-
timization problem in control theory j 9 } . W e denote by x<>)
the unique solution or motion of (1) corresponding to some fixed,
continuous control function p.(t) and the initial state x0 — xM(0;
Xo). The loss function (or Lagrangian) L is an arbitrary smooth
function of t, x, and p. Suppose xo is a fixed initial state and <1 > 0
a fixed value of the time. Then the integral
V(xo, tx; p) = f W, x„(t; x 0 ), p(t))dt
Jo
is a continuous functional of the control function p(t). As usual,
V is called the performance index of the control system (1-2).
We ask: For what continuous control function p*(t) (if any) is
V(x0, t\\ p.) a minimum?
The minimum value of V(xo, h; p) will be denoted by V*(xo, h).
The Principle of Optimality of the calculus of variations shows
[3-4] that the function p*(t) (if it exists at all) may always be
generated by a control law of the type
P*(t) = -X(t, x(t)).
Thus the precise formulation of Problem A is the following:
Journal of Basie Engineering
What conditions must be imposed on h and L to assure thai x
constant (not explicitly dependent on t) and linear in xi
Note that constancy and linearity of x a r e entirely different
properties.
If either h < °° or L depends explicitly on t, we have a non-
stationary situation and in general x will not be constant. W e
assume therefore that L ^ L(t) {10} and that h = .
The latter assumption must be made precise. Let 0 denote the
set of all continuous functions p(t) defined on the interval [0, °°)
for which the functional
P(.r„, <*>; p) = lim f L(Xfl(t; x0), p(t))dt (9)
Jo
has a finite value. (Of course, 0 may be the empty set.)
Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019
W e ask: For what control function p*(t) in Q (if any) is V(xo, 00;
p.) a minimum1
The minimum of F(aro, 0 3 ; p) is written as V*(xo,
It seems to be very difficult to give explicit conditions on L
which are equivalent to the linearity of x- F ° r this reason, we
shall base all further considerations on a simple and well-known
sufficient condition: L is a quadratic form in the n + 1 variables
(x, p) { l l } . Since we have required L L(t), the coefficients
must be constants. In other words, the most general form of L
that we shall study is given by
2L(x, p) = x'Qx + 2(r'x)p + ap* (Q, r, a constants). (10)
This is Assumption (d) in the Introduction.
With this preliminary analysis, we arrive at the following pre-
cise definition of the central problem of the paper:
Inverse Problem of Linear Optimal Control Theory. Given a com-
(vli)
pletely controllable constant linear plant (1) and constant linear
control law (2). Determine all loss functions L of the fcrm (10)
such that the control law minimizes the performance index (9).
Review of the Optimization Problem
N o w we turn to Problem B. W e seek an explicit expression for
As the general theory of minimizing (7) is quite well known [3-4],
we shall stress those aspects of the problem which result from
letting ti->-o. This problem was solved in [3].
The loss function L cannot be completely arbitrary. W e want
to investigate what restrictions must be imposed on L for purely
mathematical reasons. W e shall also examine the physical sig-
nificance of L.
The first restriction on L is a consequence of Pontryagin's
"Maximum Principle" [4, 12], According to this principle, the
variational problem (7) has a solution only if the so-called pre-
Hamiltonian function [4]
H(t, x,p,p) = L(t, x, p) + (p, Fx + gp)
(where p is an «-vector, the costate of ( 1 ) ) has an absolute
minimum with respect to p. for every fixed value of (t, x, p).
Substituting (10) into H, it follows that II will have a minimum
only if cr & 0. If cr > 0, then H has a unique minimum for all t,
x, p. This is the so-called regular case. If 0 = 0, then H can have
a minimum if and only if r'x = 0 and p'g = 0, in which case H is
(7) independent of p. This is the singular case. In the latter event,
the Maximum Principle furnishes very little information about
the proper choice of p { 1 2 } .
T o avoid complications which are of little interest in this paper,
we shall consider only the regular case. (The same assumption is
made in most of the classical literature on the calculus of varia-
tions.) Hence cr must be positive. W e may set cr = 1 without
loss of generality. Then (10) takes the form:
2L(x, p) = x'(Q - rr')x + (p + r'x)* (Q = Q')-
A second restriction is required to assure that the minimum
(8) value of (7) does not diverge to — «> as h approaches some
finite value. (In other words, we want to rule out the possibility of
MARCH 1 964 / 53
"conjugate points.") The standard way to avoid trouble is co
assume that Q — rr' is nonnegative definite. It is well known that
a symmetric matrix Q — rr' is nonnegative if and only if there is a
p X n matrix H (where p = rank Q — rr') such that Q — rr' =
H'H. Consequently we shall write L in the form
2L(x, p.) = + (p + r'x)'.
This limit is readily evaluated numerically by computing the
solution of (12) which starts at P ( 0 ) = 0 and letting t —* — <*>.
(ii) The control law (13), with II(<; h, 0) replaced b y P „ ,
generates functions /i°°(<) (dependent on Xo) which are always in
fl [3, Theorem 6.7], Thus if x°°(t) is the solution of (1) corre-
sponding to pF(t), then
(11)

Let us pause to examine the physical significance of (11). V(xo, pT) - (1/2)||zo||V„
Clearly L is nonnegative. If L is positive, we incur a loss. The
object of optimal control is to minimize this loss. The loss is zero = lim f L(x°(t), if(t))dt < co.
if and only if both terms in L vanish separately. Therefore H is
chosen in such a way that the state of the plant is "satisfactory"
Since L is nonnegative, the matrix P a is nonnegative definite.
if and only if Hx — 0 { 1 3 } . The choice of r in L fixes the desired
(iii) Of course, ixw(t) is not necessarily optimal and therefore
level p. = — r'x of the control variable.

Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019

in general
In the rather intricate calculations which are to follow, it would
be unpleasant to drag along the parameter r. Actually, r may be V*(X*, co) g F(so, co; pT).
eliminated without loss of generality.
But in reality the strict inequality sign cannot hold for any So [3,
Let us change the control variable p. to
Theorem 6.7]. In other words, p," = n*.
p = n + (r, x). N o w we have obtained an explicit expression for the minimum
of the performance index ( 9 )
Then (11) becomes
2V*(x„, » ) = |W|aP„; (15)
2L(x, p) = + p• (11a)
and the corresponding optimal control law is given b y
and (1) must be changed to
H*(t) = - ( P „ f f , x(0)
dx/dt = Fx + gP, (F = F - gr' = Fr), (la) (16)
= —(k*, x(t)).
while k in (2) is to be replaced b y
The preceding results may be summarized as
k = k — r.
THEOREM 1. (Solution of Problem B.) Consider a completely con-
I t is then clear that the problems ( F , g, k, L) and (F, g, k, L) trollable plant and the associated optimization problem (9), where L
are equivalent. is given by (11a). This problem always has a solution, which is
From now on, unless explicit mention is made to the contrary, all obtained by solving the differential equation (IS) and then evaluating
considerations will be restricted to the case r — 0, and the adjectivethe limit (H). The minimum value of the performance index is given
"optimal" will always refer to minimizing (9) with respect to by (16). The optimal control law is given by (16).
the loss function (11a). For the sake of simplicity, the bar on F, k,
and L will be dropped.
Returning to the problem of minimizing (7), let us recall the Stability of the Optimal Control Law
results of the analysis given in [3-4]. Optimality does not imply stability! In fact, let x be any control
Let II(<; <i, 0) = P(t) be the unique symmetric solution of the law. Define L so that L = 0 when p. = ~x(t, x) a n d positive for
Riccati-type matrix differential equation any other value of p. Then the loss is identically zero if and only
if the control law x is used. Since L is nonnegative, V*(xo, co) is
-dP/dt = PF + F'P - Pgg'P + H'H (P = P') (12)
also nonnegative for any Xo. I t follows that the given control
which satisfies the initial condition law x—which is entirely arbitrary— is optimal and even unique.
T o understand the difficulty raised by this example, let us ex-
n « , ; tu 0) = P(h) = 0. amine more closely the physical significance of the condition
L = 0. Evidently this can happen only when the control law is
Then the minimum of ( 7 ) is given explicitly by
p. = 0. Then all solutions of ( 1 ) and (2) are of the form x(t) =
2V*(xo, h) = M2n(0; (., 0),4 eFtxo. Consider now the linear subspace Xi of the state space
X = R" defined b y
and the corresponding unique optimal control law is
Xy = {x; \\HeF'x\\ = 0|.
Mt) = - < n ( t ; h, 0)g, x(t)). (13)
If the state x<, belongs to Xi, then the vector x(t) = eF% also
Up to this point, the analysis holds for every finite <i > 0.
belongs to Xi for any t, since e A ( ' + T ) = eAieAr for any matrix A
Additional arguments are needed to solve also the problem of
and any scalars t, r. Thus Xi is invariant under the control law
minimizing (9).
p — 0; in fact, X ! is the largest subspace of X such that L = 0
(i) If the plant is completely controllable, it may be shown [3,
when Xo is in Xi. Hence 0 is an optimal control law (conceivably
Theorem 6.6] that
not the only one) with respect to states in X,. Actually, the
lim 11(0; h, 0 ) = P„ optimal control law is unique b y Theorem 1. Thus the optimal
11—* CO control law throughout the subspace Xi consists in setting p. = 0.
exists and is unique. In view of the constancy of the plant and of Going back for a moment to the case r ^ 0, it is clear that in
L, the choice of the origin of time in (11) is immaterial: general Xi is given b y

11(0; 0 ) = I I ( - < i ; 0, 0) for all h > 0. X! = {x; \\HeF^x\\ = 0} (Fr = F — gr').

Therefore P „ may be defined equivalently as As far as initial states in X i are concerned, the outcome of the
optimization problem is determined a priori b y the choice of r.
lim H « ; 0, 0) = P „ . (14)
(l—eo I t is clear from these observations that there is no loss of
generality in requiring
4 We use the notation || x where A is any symmetric matrix, for
the quadratic form x'Ax. dimension Xi = 0. (Aj)

54 / MARCH 1964 Transactions of the AS ME

 This abstract mathematical condition is equivalent to the con-
trol-theoretical statement:
The -pair [F, H] is completely observable. (At')
In turn, this is equivalent to the concrete mathematical condition
rank [H', F', H', . . ., (F')»-W] = n. (At")
These matters are discussed (with proofs) in [11].
In conventional language, (At') means the following: Suppose
the p-vector y = IIx denotes the outputs of the plant with re-
spect to which the performance of the control system is to be
optimized. Then y must not vanish identically along any free
motion of the plant unless the initial state x<> = 0. (The outputs
y used for defining the performance index must be carefully dif-
ferentiated from those outputs which are directly measured [13].
One might say that (At) assures the nondegeneracy of the state-
ment of the optimal control problem.
If (A2) holds, then L cannot be identically zero along any
motion of (1). Then V* is positive definite (not merely non-
negative definite) and its derivative V* along optimal motions is
— L. In other words, (At) implies that F* = (1/2)||i||2P„ is a
Lyapunov function. B y a theorem well known in the Lyapunov
stability theory [14, Corollary 1.3; 15, Theorem 8] the desired
result of this section follows immediately.
THEOREM 2. If Assumptions (,i4i- 2 ) are satisfied, then Pm is
positive definite and the optimal control law is (asymptotically) stable
(i.e., all eigenvalues of F — gk*' have negative real parts).
This may be regarded as the central theorem of linear optimal
control theory. Under slight additional hypotheses it is valid also
for the nonconsistant case [3, Theorem 6.10].
Our preceding analysis implies also a much stronger result:
THEOREM 3. (Solution of Problem C.) Under Assumption (A{), a
necessary and sufficient condition for the stability of the optimal con-of Fk then follows immediately from the Lyapunov stability
trol law is that all eigenvalues of F restricted to Xi have negative real
parts.
Algebraic Criterion for Optimality
Now we seek an explicit necessary and sufficient condition for a
given control law k to be optimal (Problem D ) . Unfortunately,
conditions (14) and (16) provided by Theorem 1 are inadequate for
this purpose because P „ is defined only indirectly by the limit
(14).
Let us observe, however, that by (14) P „ is an equilibrium
Btate of the Riccati differential equation (12). Accordingly P„
satisfies the algebraic equation
-PF - F'P = H'H - Pgg'P, (17)
which is obtained by setting dP/dt = 0 in (12).
This does not characterize P„ completely because (17)—being
a system of quadratic algebraic equations—does not have a unique
solution in general. Even if P is required to be nonnegative
definite, (17) may fail to have a unique solution { 1 4 } .
Fortunately, the difficulty may be removed with the help of
Assumption (/t 2 ). This provides the solution of Problem D :
THEOREM 4. (Algebraic Characterization of Optimality.) Consider and k, the matrix P serving as a connecting link. W e want to
a completely controllable plant and the associated variational prob-
lem with H satisfying (Ai). Let kbe afixedcontrol law. Then a (i)
necessary and (ii) sufficient condition for k to be a stable optimal con-k can be found at all. I t is truly astounding that this relation
trol law is that there exist a matrix P which satisfies the algebraic re-
lations
P = P' is positive definite, (18)
Pg = k, (19)
-PFk - Fk'P = H'H + kk'. (20)
Proof of Theorem 4. (i) Suppose that k — k* is a stable optimal
control law corresponding to some L which satisfies (At). By
Theorem 1 there exists a symmetric matrix Pa which satisfies
Journal of Basie Engineering
(14) and (16), and therefore also (17) and (19). Replacing Pg b y
k in (17), it follows that (17) and (19) together imply (20). B y
Theorem 2, P„ is positive definite. Hence optimality implies the
existence of matrix P which satisfies (18)-(20).
(ii) Conversely, let k be a fixed control law. Suppose that we
are given a matrix H which satisfies (At) and a matrix P which
satisfies (18)-(20). Then P also satisfies (17). W e want to show
that P is equal to P„ of Theorems 1 - 2 , for then the optimality of k
follows from (19), and stability follows from (18) and (20). Con-
sequently it is sufficient to prove
LEMMA 1. If H satisfies (At), then (17) has a unique positive
definite solution.
Proof of Lemma 1. Let P be a positive definite solution of ( 1 7 ) .
Let k = Pg. Then the matrix Fk is stable. This is proved b y in-
troducing the scalar function
Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019
V(x) = IMI'p,
which is positive definite by hypothesis. Its derivative along the
solutions of the differential equation
dx/dt = F^c (21)
is given by
V = x'[PFk + Fk'P]x
= -||Hx\\' - \q'Px\*,
since P satisfies (17). Clearly V is nonpositive and is zero only if
g'Px = 0, which implies that k'x = 0. It follows that V is
identically zero along a solution eFKTXO of (17) only if
\\HeFLTXO\\ = |[ffeF'xo|| = 0.
B y (At) this can hold only if x0 = 0. Hence V can vanish
identically only along the trivial solution x(t) = 0. The stability
theory [14, Corollary 1.3].
N o w suppose that (17) has two positive definite solutions P,-
(i = 1, 2). Let k( = P{g. Then P<FH + Fki'P( = -H'H by
(17), and
PiFh + F'PX - (PtF„ + F'P,)'
= (Pi - Pt)Fh + Fk,' (Pj - P,)' = 0.
It is well known [16, chapter V I I I ] that the matrix equation XA
+ BX = 0 has a unique solution, namely X = 0, whenever
X,[A] + X k [B] ^ 0 for any pair j, k. (22)
Since both Fi and Ft are stable matrices, condition (22) is obviously
satisfied. Hence Pi — P 2 = 0.
This completes the proof of Lemma 1.
Finally, by the Lyapunov stability theory, equation (20)
together with (18) implies that A; is a stable control law, i.e., that
F i is a stable matrix. Theorem 4 is proved.
Frequency-Domain Characterization of Optimality
Since F and g are usually known, Assumption (At) and the rela-
tions (18) may be regarded as constraints on the parameters H
eliminate P . This is Problem E.
I t is quite remarkable that a simple relation connecting H and
must be stated in the frequency-domain if it is to be reasonably
simple. This is the main result of the paper from which every-
thing else will follow { 1 5 } .
The solution of Problem E is given by
THEOREM 5. (Frequency-Domain Characterization of Optimality.)
Consider a completely controllable plant and the associated
variational problem, with L satisfying (At). Let k be a fixed control
law. Then a (i) necessary and (ii) sufficient condition for k to be
an optimal control law is that kbe a stable control law and that the
condition
MARCH 1 964 / 55
 |l + k'$(io>)g\2 = 1 + ||#<I>(2co)</,!» (I) and F, g have the form {26), then the vector q has the representation

hold for all real co. 7i

Proof, (i) The assumptions being the same as in Theorem 4,
there is a unique matrix P satisfying (18)-(20). W e add and Q =
(28)
subtract sP from the left-hand side of (17) and obtain
L Y„ J
P(sl - F) + (-si - F')P = H'H - Pgg'P. (23)
In other words, if F and g have the canonical form (26), we can
Then we multiply the right-hand side by 3?(s)g and the left-hand identify the components of the vector q with the numerator co-
side b y g ' $ ' ( — « ) efficients of the rational function (27).
Proof of Lemma 2. The canonical form (26) follows immedi-
g'$'(-s)Pg + g'P$(s)g = g'<i>'(-s)[H'H - Pgg'P]<t>(s)g. (24) ately from the definition of controllability [17]. Formula (27) can
then be verified readily by elementary algebraic manipulations;
Recalling that Pg = k by (19), we get
for instance, by signal-flow-graph methods. For the significance

Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019

[1 + £'$(-%] [1 + k'$(s)g] = 1 + g'<f>'(-s)H'H$(s)g. and proof of this lemma see [11].
Another fact which we need here concerns the factorization of
Setting s = ico gives (I). 6 nonnegative polynomials. This is the main step in the method of
Finally, by Theorem 4, k = Pg is necessarily a stable control spectral factorization [18, 19].
law if P satisfies (17)-(20). LEMMA 3. Let 9(co 2 ) be a polynomial in co2 with real coefficients
(ii) Suppose A; is a given stable control law. Theorem 2 implies that is nonnegative for all real co. Then there exist q polynomials
that given some H satisfying (^2) equation (17) has a unique vk(iu>) (1 g j g 11) with real coefficients such that
positive definite solution Pm and there is an optimal control law
k* = Pmg. Using Condition (I), we will prove that the matrix
(29)
P r o so determined satisfies (19), in other words, that k = k*. 6(«>2) = £ -«») = £
Then (20) will be also satisfied and it will follow by Theorem 4 k=\ k=1
that Condition ( I ) is sufficient for optimality. Moreover, we can always let q = 1 and choose Vi(s) so that all its
W e consider again (24), which is a consequence of (17), and let zeros have nonpositive real parts. With these two restrictions we
s = ico. W e replace j|//$(!co)ff||5 by its value given b y (I). In- have the unique "factorization" 0(co 2 ) = |j<I(?CO)|2.
troducing the abbreviations Proof of Lemma 3. Everything follows easily b y factoring
0(co 2 ) according to classical algebra.
TT(KO) = g'PJS>{iw)g,
Returning to the proof of the theorem, we recall equation (6),
K(iw) = k'$(io))g,
1 + k'$(s)g = tM/t(s).
and simplifying the resulting expressions we obtain
Hence (25) implies
|l + ir|2 = |l + K\2.
\Ui<»)\2 = |<M«o)| 2 = Q(co2).
Since the optimal control law corresponding to P„ is k* = Pag it
In other words, the nonnegative polynomial 0(co 2 ) of Lemma 3 has
is clear that
two different factorizations. B y assumption, the zeros of ^t(s)
it — K*. are in the left-half plane. B y optimality and Theorem 2, the same
is true concerning the zeros of ^ ^ ( s ) . Hence
In other words, the proof has been reduced to showing that
k'$(io})g = k*'$(ib>)g. (30)
|l + &'$(tw)ff|2 = |l + &*$(wo)£f|2 (25)
B y Lemma 2, we take F and g in the representation (26). Then
implies k = k*.
(27) shows immediately that k = k*.
W e recall the hypotheses that (a) the pair [F, g] is completely
This completes the proof of the theorem { 1 6 } .
controllable and that (6) A; is a stable control law. W e need to
Theorem 5 is not quite convenient as stated, because Assump-
know also
tion (A2) is expressed in time-domain language. It is easy to see
LEMMA 2. Suppose [F, g] is completely controllable. Then one can
that Assumptions (.A 1-2) are equivalent to the frequency-domain
always find a basis in the state space X such that F and g have the
condition:
representation
Rational functions H$(s)g have no common cancelable factors (B1-2)
0 1 ~o
The equivalence of this condition with (AI~ 2 }—in other words,
with the complete controllability of [F, g] and the complete
F = g = observability of [F, H\-—is established in [11],
0 1 0 Theorem 3 holds if and only if all zeros of the common cancela-
-Cin-1 ble factors of H$>(s)g have negative real parts.

(26)
Implications of Optimality
where det (si — F) = \J/(s) = s" + a „ s " - 1 + . . . + ai. From Theorem 5 we can deduce a number of interesting rela-
Moreover, if tions between optimality and frequency-domain concepts. In
this way we obtain a fully satisfactory solution of the Inverse
Y^t'co)"-1 + . . . + 7i
q'$(iw)g = — (27) Problem of Linear Optimal Control Theory.
(wo)" + a„(i'co)" _ I + . . . + «,' From Condition ( I ) it is clear that a stable control law may be
optimal only if the return difference Tk(iu) satisfies the condition
6 The quadratic form || x for x complex and A real is defined, as
usual, by |n(ico)| = |1 + k'$(s)g\2 > 1. (II)
IA =2
XiaijXj, This condition may well fail to hold { 1 7 } . If it does hold, then
H may be obtained according to Lemma 3 b y factoring the non-
where Xi is the complex conjugate of Xi. negative polynomial T(co 3 ):

56 / MARCH 1964 Transactions of the AS ME

 r(c0 2 )/|^(26)) 2 | = |l + fc'$(fco)ff|2 -
Many different matrices H will satisfy (31) because there are
many different factorizations of I\co2). However, not all H's
obtained in this way define optimization problems which yield
the control law k. Since k was assumed to be stable, the situa-
tion described in Theorem 3 must hold. Therefore (see remark
following (fii-2)) the rational functions H$(s)g must not possess
any common cancelable factor which has a zero with nonnegative
real part. H's which do not satisfy this requirement must be
discarded.
A proper choice of H is always possible. Using the unique
factorization described at the end of Lemma 3, and Lemma 2, we
1 the control error (as represented by the term ||ffx||2) and deviating
(31) from the given control law (as represented by the term (p + r'x)2).
Combining these terms additively to form the loss function (11), it
follows that the control law k is "better" than the control law r if and
only if the return differences satisfy the condition
\Tk(iu)/Tr(io>)\ > 1 for all real 0). (116)
Proof. Condition ( I I ) for the case r =t= 0 is given by (34). But
|7v«o)l = \tkm/t(io>)\. (35)
The same holds for Tr(ico) and ( l i b ) follows immediately.
Formula (33) implies also a useful arithmetic condition for
optimality, which we state only for the case r = 0:

Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019

THEOREM 9. A control law k is optimal in the sense of Theorem 5
can write
if and only if
|l+fc'$(?co) ff | 2 - 1 = lA'$(fco)sr|2, (32)
(a) tpk(s) satisfies the Routh-Huruiitz conditions;
where all zeros of the transfer function h'$(s)g have nonpositive (b) *(co 2 ) = |^t(jco)|2 — \\J/(iu)\2 is a nonnegative polynomial
real parts. in co2.
We assume now (without loss of generality, see Section 3) that k
In other words, it is possible to characterize optimality solely
is a completely observable control law. This is equivalent to the
in terms of the open and closed-loop characteristic equations!
fact that the polynomials ipM and \f/(s) are relatively prime.
This is related to the fact that the numerical values of k depend
Since
on the coordinate system chosen to describe the state variables,
|l + k ' $ ( i u ) g Y - 1 = |^(w)/f(i<o)|J - 1 whereas \f/ and \pk are independent of this arbitrary choice!
(33) Condition (b) in Theorem 9 is equivalent to

(b') SE^co2) has no real, positive root of odd multiplicity.

the transfer function h'$(s)g has then no cancelable factors, con-
dition (B1-2) is satisfied, and therefore also (/1 2 ). The matrix T o express ( b ' ) in the form of inequalities on the coefficients of
H = h' so constructed satisfies the requirements of Theorem 5. * ( c o 2 ) is a classical problem in the theory of equations. B y tech-
Hence we have: niques based on Sturm's theorem, it is possible in principle to
derive such inequalities (and to prove the Routh-Hurwitz con-
THEOREM 6 . (Solution of the Inverse Problem of Linear Optimal Con-
ditions, see [16, Chapter X V ] ) . This is a very difficult exercise in
trol Theory.) Consider a completely controllable plant with a algebra; the general inequalities are not known today.
stable, completely observable control law k. Then k is a nonde-
W e shall give explicit inequalities corresponding to (a)-(b) only
generate optimal control law if and only if Condition (II) is satisfied.
in the special cases n = 2 and n = 3. The derivations are ele-
In general, k will be optimal with respect to several H's, which are
mentary and therefore omitted.
determined from (31) and must satisfy in addition the noncancel-
Using the notations
lation conditions mentioned above {18}.
The requirement Ir^'co)! > 1 is a celebrated result of classical <J/(s) = det (si - F) = sn + a „ s " - 1 + . . . + «,,
feedback theory [20]. It assures that the sensitivity of the sys-
tem to component variations in the forward loop is diminished by ^(s) = det (si - F + gk') = s n + fts""1 + . . . + ft,
the addition of feedback. The larger the value of |rt(i'w)|, the we have: Necessary and sufficient conditions for optimality of k
greater the effect of feedback in reducing sensitivity. B y Theorem are: n = 2:®
6, the same condition also assures dynamic optimality. W e have
thus a beautifully simple relationship between the "classical" (ft > 1 (36)
argument in favor of feedback and the "modern" concept of (ft >0
dynamic optimality.
It is well worth recording also the consequences of Theorems 5 (ft2 - «i2 ^ 0
j f t 2 - a22 - 2 ( f t - a , ) ^ 0 (37)
and 6 when r ^ O .
(both equal signs cannot hold simultaneously
THEOREM 7. Given any stable control law k, there exists a loss
function (11) with H nonsingular for which k is optimal. n = 3:
Proof. Of course, this result hinges on the possibility of choosing
r in a convenient way. Recalling the discussion of equivalent ft > 0 (38)
problems in Section 5, it follows that Condition ( I I ) may be ft >0
generalized to
ftft - ft > 0
|l + (k' - r')$r(w)?|2 > 1. (Ila)
ft2 - ft2 S 0
We must show that r can be chosen in such a way that the strict ft2 - a,2 - 2(ft - a,) S 0, (39)
inequality sign holds. N o w Either ft2 - a22 - 2(ftft - ot,,at3) & 0 or
ft2 - a2s - 2(ftft - a.tt,) =
|l + (k' - r')$ r («o)ffl a = |l + [^(ico) - fc(«»)]/*,<iw)|»
- V(ft2 - «.2)[ft2 - «3 2 - 2(ft - a,)].'
\pk(iw)
> 1, (34)
\pr(iu) Asymptotic Properties of Optimal Control Laws
from which it is obvious that the required r exists. Q.E.D. N o w we turn to Problem F : If L is given by

THEOREM 8. Consider a completely controllable plant with a given (40)

2L(x, p) = p|M|2 + m2,
control law r. Suppose ive wish to find a "better" control law k. In 6 These conditions were first presented by the writer in an oral dis-

determining k an optimal compromise must be made between reducing cussion at the 1962 Joint Automatic Control Conference.

Journal of Basie Engineering MARCH 1964 / 57

where H is fixed and p is variable parameter, how do the optimal l
control laws behave as p —>• m? W e assume of course that p > 0. w2(n-m) p
The solution of this problem is greatly simplified by a simple
has an increasingly sharp low-pass characteristic as (n — m) —<• co t
observation based on Condition ( I ) of Theorem 5: The same con-
the bandwidth being p 1 / 2 ( "~"' ) . (A high return difference requires
trol law is obtained if H is replaced by the matrix hh', where h is de-
large bandwidth.)
termined by
Moderate overshoot and low-pass frequency-response have
||F$(ico)ff||« = |V$(i»)f|«. (41) always been regarded as typical of good servomechanism design.
Since the left-hand side of (41) is a nonnegative polynomial in co2, This fact is now " p r o v e d " with a rigorous theoretical analysis
the existence of h follows from Lemmas 2-3. The vector h is not and brought within the framework of optimal control theory
uniquely determined by (41); it is again convenient to fix it b y {20}.
using the unique factorization mentioned in Lemma 3 { 1 9 } . I t must be emphasized, however, that the influence of the plant
Replacing H by h' means passing from the loss function defined parameters may be significant for moderate values of p. In such
by (41) to that defined by cases the machinery of optimal control theory (e.g., (14)) may
lead to a control law which is not easily obtainable by the usual

Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019

2L(x, p) = pv* + P2, (42)
intuitive engineering design methods.
where The essential idea of Theorem 11 is attributed to Chang [18—
19], who called it the "root^square-locus" method {21}.
V = {K x) = Yh h'x<- (43) I t is of some interest to give explicit formulas for the closed-
t= l loop characteristic equation of a system which is optimal with
If we choose the special coordinates described in Lemma 2, then respect to the loss function (42). We shall do this only when
n — 2. In this case the characteristic polynomial is written in
Xi = ( d / d t y - l X i (i = 2, . . ., n),
the usual form
and every component of h t is nonnegative since all zeros of
n s 2 + frs + 0 , = s 2 + 2f*co*s + cot»,
his'-1
must have negative real parts. Hence we have where f* is the (closed-loop) damping ratio and co* is the (closed-
f=i loop) undamped natural frequency.
THEOREM 10. Consider a completely controllable plant with a single We want an explicit relationship between p, f*, and co*, given
control variable. Without affecting the corresponding control law, a fixed transfer function h'$>(s)g. These relations follow from
every quadratic loss function (11a) may be replaced by (48), in (36)-(37) as well as from (44), which we can write also as
which -q is a linear combination with nonnegative coefficients of a
- c o 2 + 2f*co*?'co -I- co*2 2 , , . „ ,
certain state variable X\ and its derivatives. - 1 + p\h'$(ia))g\*, (47)
Condition ( I ) is now of the form (ico — Si)(t'co — S2)

|l + &'$(KO)S|2 = 1 + p|A'$(ico)S|2. (44) where Si and S2 are the open-loop poles.

It was first noticed by Chang [18, 19] that this equation ad- Case 1. Si = s 2 = 0. Then co* > 0 and f* ^ 1 / V 2 -
mits a revealing interpretation in terms of root loci. Writing
Subcase. If h'$(s)g = 1/s 2 , then f* = l / \ / 2 and p = co*4.
f(ico) to denote the numerator of the transfer function h'$(s)g(a
polynomial with real coefficients of degree m < n), it is clear that

4
1
(44) is equivalent to Case 2 . Si = 0 , S2 = 1. Then co* > 0 and f* &
2 + 4co*2
&(»)**< ~s) = t(s)t(-s) + pf(s)f(-s). (45) Subcase. If h'$(s)g = l / s ( s — 1) then f* attains its lower
As p —• <» precisely 2m zeros of (45) tend to the zeros of bound, while p = co*4.
f ( s ) f ( — s )- The remaining 2(n — m) zeros tend to co and are
asymptotic to the zeros of the equation Case 3. Si = s 2 = —1. Then co* g 1 and f* S: \ l — I -
1 2 ^ 2co*2
s2(n-m) = p (46)
but both equal signs cannot hold simultaneously.
Since all zeros of the polynomial i/'*(s) must be in the lefWialf Subcase (a). If /t'($)(s)(7 = l / ( s 2 + 1), then p = co*4 - 1,
plane, it follows that m zeros of 1pk(s) tend to the corresponding co* > 1, and f* attains its lower bound.
zeros of f ( s ) (which are in the left-half plane, or on the imaginary
Subcase (b). If h'$(s)g = s/(s + l) 2 , then p = f* 2 - 1, f* > 1,
axis by the definition of h) while the remaining n — m zeros of
and co* = 1.
\pk{s) tend asymptotically to a Butterworth pattern [9, 10, 19] of
radius p 1 / 2 ( n ~ m ) . Case 4. si = i and s2 = — i. Then co* ^ 1 and f* §
These observations may be summarized in the following form:
THEOREM I I . (Solution of Problem F). Consider a completely
trollable plant and the optimization problem (9) corresponding to thetaneously.
con-
{ * — ——, but again both equal signs cannot hold simul-

loss function (43). Asp—* co, m closcd-loop poles of the optimal

Subcase (a). If /i'$(s)</ = l / ( s 2 + 1), then p = co*4 - 1, and f*
control system tend to the m zero of h'$>(s)g, while the remainder tend
attains its lower bound.
to a Butterworth configuration of order n —TO and radius p1/2("-m>.
Subcase (b). If h'$(s)g = s/(s 2 + 1), then p 2 = 2(2f* 2 - 1)
The optimal closed-loop poles are asymptotically independent of
the (open-loop) poles of the plant. f * > 0, and co* = 1.
This is a highly important result. Only in Case 4 is it possible to have a damping ratio less than
Note that the number pUHn-mt j s closely related to the loop l/\/2 This is to be expected. If p is small, the performance
gain. At large values of p the return difference |r*(?'co)| becomes index defined by (46) will be optimized and stability is achieved
very large so that the system becomes insensitive with respect to by introducing a small amount of damping. A large amount of
plant variations. At the same time, the dynamical behavior of damping would require too much control energy.
the closed-loop system becomes independent of the uncontrolled Thus in usual cases a second-order system can be optimal only
dynamics of the plant. if it has a damping ratio at least as high as l / \ / 2 . This con-
The step response of a transfer function G(s) with no zeros and firms a well-known "rule-of-thumb" used in designing instrument
a Butterworth pattern of poles has an overshoot from 0 to about servomechanisms, which calls for f* = 0.7 — 0.8.
23 percent as (n —TO)increases from 1 to co [10], Moreover, the It is interesting that the optimality conditions (39) do not put
frequency response function an upper limit on f*

58 / MARCH 1964 Transactions of the AS ME

References
1 P. Funk, "Variationsrechnung und ihre Anwendung in
Physik und Technik," Springer, Berlin, 1962 (Grundlehren Series 94).
2 R. E. Kalman, " O n the Inverse Problem of Optimal Control
T h e o r y " (to be published).
3 R . E. Kalman, "Contributions to the Theory of Optimal Con-
trol," Bol. Soc. Mat. Mexicana, 1960, pp. 102-119.
4 R. E. Kalman, " T h e Theory of Optimal Control and the
Calculus of Variations," Proc R A N D — U n i v . Calif. Symp. on
Optimization Theory, 1960; Univ. of Calif. Press, 1963, chap. 16.
5 R . E. Kalman, T . S. Englar, and R . S. Bucy, "Fundamental
Study of Adaptive Control Systems," A S D Technical Report 61-27.
6 A. E. Bryson and W . F. Denham, "Multivariable Terminal
Control for Minimum Mean Square Deviation From a Nominal
Path," Proceedings of the Symposium on Vehicle Systems Optimi-
zation, Institute of the Aerospace Sciences, November, 1961.
7 E. B. Lee, "Design of Optimum Multivariable Control Sys-
tems," J O U R N A L OP B A S I C ENGINEERING, TRANS. A S M E , S e r i e s D,
vol. 83, 1961, pp. 85-90.
8 C. W . Merriam, III, " A Class of Optimum Control Systems,"
Journal of the Franklin Institute, vol. 267, 1959, pp. 267-281.
9 D . Graham and R. C. Lathrop, " T h e Synthesis of 'Optimum'
Transient Response: Criteria and Standard Forms," Trans. AIEE,
vol. 72, part II, 1953, pp. 278-288.
10 D . T . McRuer, D . Graham, et al„ "Performance Criteria for
Linear Constant Coefficient Systems With Deterministic Inputs,"
Technical Report ASD-Technical Report 61-501, February, 1962.
11 R. E. Kalman, "Mathematical Description of Linear Dynami-
cal Systems," S I A M J. Control, 1963.
12 L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and
E. F. Mischenko, " T h e Mathematical Theory o{ Optimal Processes,"
Interscience Publishers, New York, N. Y . , 1962.
13 K . J. Astrom, J. E. Bertram, et al., "Current Status of
Linear Optimal Control T h e o r y , " SIAM J. Control. 1963.
14 R . E. Kalman and J. E. Bertram, "Control System Analysis
and Design Via the 'Second Method' of Lyapunov," JOURNAL OF
BASIC ENGINEERING, T R A N S . A S M E , Series D , v o l . 82, 1960, p. 371.
15 J. P. LaSalle and S. Lefschetz, "Stability by Lyapunov's
Direct M e t h o d , " Academic Press, New York, N. Y., 1961.
16 F. R . Gantmakher. " T h e Theory cf Matrices," Chelsea, 1959.
17 R . E. Kalman, "Lyapunov Functions for the Problem of
Lur'e in Automatic Control," Proc. Nat. Acad. Sci. USA, vol. 49,
1963, pp. 201-205.
18 S. S. L. Chang, " R o o t Square Locus Plot—a Geometrical
Method for the Synthesis of Optimal Servosystems," I R E Conven-
tion Record. 1960.
19 S. S. L. Chang, "Synthesis of Optimum Control Systems,"
McGraw-Hill Book Co., Inc., New York, N. Y „ 1961.
20 H. W . Bode, "Network Analysis and Feedback Amplifier De-
sign," Van Nostrand and Co., Inc., New York, N . Y., 1945.
21 D . A. S. Fraser, "Nonparametric Methods in Statistics," John
Wiley & Sons, Inc., New York, N. Y „ 1957.
22 O. Bolza, "Lectures on the Calculus of Variations," D o v e r
Press, New York, N. Y „ 1960, pp. 31-32.
23 J. Douglas. "Solution of the Inverse Problem of the Calculus
of Variations," Trans. Am. Math. Soc., vol. 50, 1941, pp 71-128.
24 F. B. Hildebrand, "Methods of Applied Mathematics,"
Prenticc-Hall, Inc., Englewood Cliffs, N. J., 1952.
25 A. A. Frederickson, to be published.
26 P. D . Joseph and J. Tou, "On Linear Control T h e o r y , "
Trans. AIEE, vol. 80, part II, 1961, pp. 193-195.
27 R. E. Kalman, "Canonical Structure of Linear Dynamical
Systems," Proc. Nat. Acad Sci. USA, vol. 48, 1962, pp. 596-600.
28 E. G. Gilbert, "Controllability and Observability in Multi-
variabl • Control Systems," SIAM J. Control, 1963.
29 G. A. Bliss, "Lectures on the Ca cuius of Variations," Chicago
Univ. Press, Chicago, 111., 1946.
30 L. Berkovitz. "Variational Methods in Problems of Control
and Programming," J. Math. Anal. Appl., vol. 3, 1961, pp. 145-169.
31 V. M . Popov, "Absolute Stability of Nonlinear Systems of
Automatic Control," Avt. i Telemekh., vol. 22, 1961, pp. 961-979.
APPENDIX
{ l } Optimal control theory must be regarded as a mathemati-
cal tool. A rigid definition of the performance index cannot and
should not be avoided, because the power of mathematical
reasoning is available only when problems are precisely stated.
(If we seek approximations, then it must be precisely defined what
constitutes a good or bad approximation.) It is true that the
optimal control theory of today can solve only a few problems
(mostly in the realm of linear mathematics), and that it is not ap-
plicable to many of the more complex and admittedly more im-
portant questions that interest the engineer. But this is not
Journal of Basie Engineering
a defect in the conceptual foundations of the theory; it is
merely an argument for more basic research to extend the ap-
plicability of the existing methods.
{2} The issue involved here is quite similar to the objections
raised against parametric statistical procedures. T o assume that
a given statistical model has certain well-defined but unknown
parameters (for instance, a Gaussian distribution with un-
known mean and variance) is highly restrictive, and the resulting
conclusions must not be interpreted too widely. The remedy is to
use nonparametric statistics [21]. Then one can claim more
general conclusions, but they will be necessarily less explicit than
in the parametric case.
{3} The precise definition of the inverse problem of the calcu-
lus of variations is the following: Given a family of curves which
Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019
satisfy the vector differential equation
(*) £ = F(t, x, x) (a;, F = n-vectors, •= d/dt).
Determine all scalar functions L{t, x, x) such that every solution of
(*) is a regular extremal of the ordinary variational problem
fL{t, x, x)dt;
in other words, determine all L such that (*) is identical with the
Euler eqitations
and L ; ; is positive definite along every solution of (*).
A solution of this problem always exists when n = 1 [22]. The
case n — 2 has been investigated and settled by Douglas [23].
His results are much too complicated to be stated here. The
problem is reduced to the solution of a system of first^order par-
tial differential equations for the elements of the matrix L „ , from
which L is determined b y quadratures. N o t every family ( * ) is
a regular extremal. For instance, there is no L (with L positive
definite) for which the curves generated by
21 = Xi' + Xi', Xl = x,
are extremal.
See also the interesting heuristic treatment of Hildebrand [24,
Sect. 2.14-2.16],
{4} It seems that frequency-domain concepts are unavoidable
in system theory when explicit conditions are desired. A classical
example is of course Nyquist's stability criterion. While equiva-
lent criteria can be stated in the time domain (all characteristic
roots must have negative real parts) or even in terms of the sys-
tem constants (the Routh-Hurwitz conditions), the frequency-
domain Nyquist criterion is very elegant, simple to state, and
often more useful than the others.
The deeper significance of the frequency-domain formulas is
that they provide "coordinate-free" system description: since
transfer functions refer to input-output relations, they are inde-
pendent of the choice of the coordinates in state space. General
results of optimal control theory must be of course independent of
the arbitrary choice of coordinates.
A full discussion of the relations between the transfer-function
and state-variable methods of describing linear dynamical sys-
tems may be found in [11],
{5} The practical significance of optimality conditions is ex-
plored in detail by Frederickson [25]. The reader should consult
his paper also for a theoretical analysis of the so-called I T E 2
(integral of time X error squared) and I T 2 E (integral of time
squared X error squared) performance criteria [10].
{6} These assumptions are of course highly restrictive. One
obtains a hierarchy of problems depending on the number of con-
trol variables and the number of state variables which can be
measured directly.
If all state variables can be measured, the optimal controller does
not contain dynamical elements because the best control action
at any instant depends only on the values of the state variables at
that instant. But if some control variables cannot be measured
directly-—which happens very often in practical problems—opti-
MARCH 1964 / 59
mal control theory requires that the missing state variables be
estimated from the known ones using Wiener filtering techniques
[13, 26]. The Wiener filter will contain dynamical elements which
are to be regarded as a part of the controller.
For example, if only one state variable can be measured then
any optimal control loop can be specified by two transfer func-
tions: the transfer function of the plant from control input to
measured output and the transfer function of the controller from
plant output to control input. Given the first transfer function,
by the methods of this paper it is possible to obtain implicit
conditions which must be satisfied by the second transfer func-
tion if it is to be optimal. No explicit form of these conditions
is known (at present).
{7} The reader is reminded that in most of control theory (and
also in this paper) there is no need to introduce the mathematical
concept of a Laplace transform. All one requires are certain
functions of a complex variable s. Only in rare cases is it useful
to know that transfer functions can be interpreted as Laplace
transforms, i.e., as an integral. In that case special restrictions
must be imposed on s. No such restriction is needed or implied in
the definition (4).
{8} It follows from the author's canonical decomposition
theorem [27] (see also [11, 28]) that only one " p a r t " of the plant
is included in the control loop, namely that which is completely
controllable and completely observable. Only this part is of in-
terest in studying the optimality of the control law.
{9} In the classical calculus of variations, this problem is called
the Problem of Lagrange or the Problem of Mayer or the Problem
of Bolza. All three are equivalent [29, §69.]
{10J This assumption is not necessary. For instance, Fred-
erickson [25] shows that every loss function of the form
L(t, x, p) = nHI'e, + M2 (& positive integer,
Qi nonnegative definite)
is strictly equivalent (yields the same minimum for (9) and the
same optimal control law) to a constant loss function of the type
L{x, p) = ||Z||2Q2 + yu2 (Q2 nonnegative definite).
{ l l } This is a very special assumption. For example, if L is
a homogeneous polynomial of degree 2k in (x, p), then under
certain additional conditions % will again be linear in x. The ex- condition [F, 11} = completely observable (i.e., (^1 2 ')) is not
plicit form and significance of these additional conditions on L
are not known at present, however.
{12} The part of Pontryagin's maximum principle which is
used here is equivalent to the classical necessary condition of
Weierstrass. Historically, this condition was first stated for the
ordinary problem (no constraints) of the calculus of variations
[29, §9]. The extension of the Weierstrass condition to the
problem of Lagrange [29, §78] has been shown [30] to be equiva-
lent to Pontryagin's requirement that the pre-Hamiltonian H
possess an absolute minimum with respect to p for every (I, x, p).
The Weierstrass, Pontryagin theory is needed here only to
establish that cr > 0; the rest of the analysis proceeds in the
spirit of the Hamilton-Jacobi-Carathdodory theory.
{13} Note that this requirement does not suffice to determine
H uniquely. There is an essential arbitrariness involved here;
in fact, the lack of a nice physical interpretation of the term
||//a:||2 has been a serious difficulty in the applications of optimal
control theoiy. This difficulty is removed to a large extent in
the present paper. See especially Section 10.
{14} Consider the matrices
[1 - 1 ] , r = 0.
It is easily verified that the two sj'mmetric matrices
1 - V 2 + 1
, r V2-
l ~ L - V 2 - + 1 V 2
= |~3 + V 2 1 + V 2 l
+ V2 1 + V2 J
60 / MARCH 1964
are both solutions of (17). Pi is nonnegative definite and P2 is
positive definite. Pi is the one which corresponds to the limit
(14). Condition ( A j ) fails to hold, because
[ cosh t
sinh t
sinh fI
cosh tj
and if .r0 = £ j J , then \\HeF'xa\\ == 0.
{15} The idea of using frequency-domain concepts in the
present context is due to V. M . Popov [31]. B y an extension of
Popov's ideas, the writer has succeeded in obtaining a solution of
the celebrated problem of Lur'e [17]. In fact, Theorem 4 is a
variant of the Main Lemma used in [17], which has also other
Downloaded from https://asmedigitalcollection.asme.org/fluidsengineering/article-pdf/86/1/51/5763755/51_1.pdf by Istanbul Teknik Universitesi user on 27 November 2019
important applications in system theoiy.
{16} The assumption that "k is a stable control law" is an im-
portant part of Theorem 5. To see why, let us write
|l + fc'$(«o)0|s = |l + K»'w)|2,
using Lemma 3. Suppose the rational function v(iw) k'$>(io})g;
in other words, some zeros of v(s) have positive real parts. By
Lemma 2 there exists a vector q such that
v(ico) - q'<$(iw)g (q ^ k),
and we may replace k by q without any effect on (I). Thus ( I ) is
satisfied b y many vectors. But only one of these corresponds to
the optical control law, namely that which is determined by the
special factorization mentioned in Lemma 3.
{17} Plotting the frequency-response function k'<b{iw)g in the
complex plane, it is easy to see that ( I I ) can be satisfied only if
lim |!C0&'$(?CI>)(7| > 0. Using the canonical coordinate system
£1)—>-co
it follows that the latter condition is satisfied if and only if k; ^ 0
for i = l , . . . , n, which means that every single state variable
must be fed back! This is a reasonable consequence of the
theory: since all state variables are assumed to be measurable,
some information would be discarded if a coefficient k( were zero.
Thus optimality requires using all available information.
{18} If H ^ h' is a matrix consisting of more than one row, the
sufficient to guarantee that the optimal control law is completely
observable.
For instance, if IJ = diag [/in, . . ., h„„,], hti > 0, and F, g have
the canonical form (26), then
||//$(IU))!7||2 = [/i„ + ZteCO2 + + h„„o)*
By proper choice of the /i (i the numerator and denominator will
have common factors. Then if
||ff$(w)ff||2 = |A'$(«w)(/|2,
where h' is the result of the special factorization mentioned in
Lemma 3, li'Q?(i)g will also possess cancelable factors. Hence
[F, h'] and consequently [F, k'\ will not be completely observable.
{19} And then it follows further that P „ ( Q ) - PJ.hh') is
nonnegative definite for all Q. See [2],
{20} That frequency-domain methods are applicable just as
easily to low or high-order s3'stems has always been claimed as a
major advantage for these methods. The present analj'sis shows
that, qualitatively speaking, similar claims can be made also for
time-domain methods. When it is a question of accurately com-
puting an optimal control law, however, serious difficulties of a
numerical nature may arise in dealing with high-order systems.
If we had general design methods of the frequency-domain type—•
none exists today which is really general—it, too, would be
subject to the same numerical difficulties.
{21} Chang 's results were restricted to the optimization of
the step response of the system. This inessential restriction
arose solely because Chang used the older (frequency-domain)
version of optimal control theory.
Transactions of the AS ME