LINEAR AND NONLINEAR PROGRAMMING Second Edition David G. Luenberger Stanford University w ADDISON-WESLEY PUBLISHING COMPANY Reading, Massachusetts Menlo Park, California - London - Amsterdam - Don Mills, Ontario - SydneyPREFACE This book is intended as a text covering the central concepts of practical optimization techniques. It is designed for either self-study by professionals or classroom work at the undergraduate or graduate level for students who have a technical background in engincering, mathematics, or science. Like the field of optimization itself, which involves many classical disciplines, the book should be useful to system analysts, operations researchers, nu- merical analysts, management scientists, and other specialists from the host of disciplines from which practical optimization applications aré drawn. The prerequisites for convenient use of the book are relatively modest; the prime requirement being some familiarity with introductory elements of linear al- gebra. Certain sections and developments do assume some knowledge of more advanced concepts of linear algebra, such as eigenvector analysis, or some background in sets of real numbers, but the text is structured so that the mainstream of the development can be faithfully pursued without reliance on this more advanced background material. Although the book covers primarily material that is row fairly standard, it is intended to reflect modern theoretical insights. These provide suructure to what might otherwise be simply a collection of techniques and results, and this is valuable both as a means for learning existing material and for developing new results. One major insight of this type is the connection between the purely analytical character of an optimization problem, ex- pressed perhaps by properties of the necessary conditions, and the behavior of algorithms used to solve a problem. This was a major theme of the first edition of this book and the second edition expands and further illustrates this relationship. As in the first edition, the material in this book is organized into three separate parts. Part I is a self-contained introduction to linear programming, a key component of optimization theory. The presentation in this part is fairly conventional, covering the main elements of the underlying theoryvill Preface of linear programming, many of the most effective numerical algorithms, arid many of its important special applications. Part I, which is independent of Part I, covers the theory of unconstrained optimization, including both derivations of the appropriate optimality conditions and an introduction to basic algorithms. This part of the book explores the general properties of algorithms and defines various notions of convergence. Part II] extends the concepts developed in the second part to constrained optimization problems. Except for a few isolated sections, this part is also independent of Part I. ible to go directly into Parts II and IT omitting Part I and, in fact, the book has been used in this way in many universities. Each part of the book contains enough material to form the basis of a one-quarter course. In either classroom use or for self-study, it is important not to overlook the suggested exercises at the end of each chapter. The selections generally include exercises of a computational variety designed to test one’s under- standing of a particular algorithm, a theoretical variety designed to test one’s understanding of a given theoretical development, or of the variety that extends the presentation of the chapter to new applications or theoretical aréas. One should attempt at least four or five exercises from each chapter, In progressing through (he book it would be unusual (o read straight through from cover to cover. Generally, one will wish to skip around. In order to facilitate this mode, I have indicated sections of a specialized or digressive nature with an asterisk*. There are several principal objectives of the revision comprising the second edition. First, Part I, the linear programming portion, has been ex- panded to include more explanation, many more application exercises, and a new chapter on transportation and network flow problems. This portion can now serve by itself as a basic text on linear programming. A second objective has been to add more applications and examples to the nonlinear programming parts. These serve primarily to illustrate problem formulation and application of the theory. Finally, a third objective has been to incor- porate many of the important developments that have occurred in the field of nonlinear programming in the past ten years. Several new sections in various chapters and a new chapter on Lagrange methods are included. It is reassuring and satisfying to find that the principles of analysis exposited in the first edition are indeed directly applicable to the methods developed in the intervening years. The new methods thus serve as further examples of those general principles. Tam indebted to several people for their unselfish help in the devel- ‘opment and preparation of this second edition. Shmuel Oren contributed several new exercises, some new developments, and a great deal of helpful advice. Daniel Gabay spent a good part of one summer at Stanford helping with the revision of the nonlinear programming portion of the text. I also wish to thank Edison Tse, Darrell Duffie, and Shao-Hong Wan for help and contributions. I was fortunate to have several very conscientious and con-Preface x structive reviewers—Dimitri Bertsekas, Leon Lasdon, Ronald Rardin, and Paul Rubin—-whose suggestions greatly improved the revision. 1 wish to thank Nancy Florence for her excellent secretarial help. In addition, over the years many Stanford students and professars at other universities have expressed comments, suggestions, and corrections to me regarding the first edition. They, too, have contributed significantly to this second edition. Stanford, California January 1984 D.G.L.CONTENTS Chapter 1 Introduction Optimization Fe 1 2 Type POben 5 ee 2 13. Size of Problems. . oes 5 Iterative Algorithms and Convergence 6 PART I Linear Programming Chapter 2 Basic Properties of Linear Programs 21 Introduction . ei 2.2 Examples of Linear Programming Problems...) . 14 2.3. Basic Solutions eee? 2.4 The Fundamental Theorem of Linear Programming. | | 18 2.5 Relations toConverity 2... 2... 2... 20 gt FAhiapter 3 The Simplex Method 31 Pivots. . Se 0 wa3.2- Adjacent Extreme Points | | | ta 6 =33 Determining a Minimum Feasible Solution... 2. . 40 ‘3-4 Computational Procedure—Simplex Method . . . . . 44 5 Artificial Variables... 8 226 Variables with Upper Bounds | 1s) 1. 8B “3.7 Matrix Form of the Simplex Method . 2 2. 2... 88 N38 The Revised Simplex Method. : : 59 *3,9 The Simplex Method and LU Decomposition - >. |. 65 3.10: Decomposition 229g 68 3.11 Summary 2 2. ee OS B12 Exercises 2 2 8PART IL Chapter 6 6.1 62 63 64 65 66 67 68 69 Chapter 7 a 72 73 14 a 16 1 18 Duality J Dual Linear Programs. 2 2. 0 ee The Duality Theorem Fe Relations to the Simplex Procedure Sensitivity and Complementary Slackness : The Dual Simplex Method... The Primal-Dual Algorithm . Reduction of Linear Inequalities a. Exercises 2 2 2 ee ‘Transportation and Network Flow Problems The Transportation Problem . Finding a Basic Feasible Solution Basis Triangularity : Simplex Method for Transportation Problems. oo The Assignment Problem... 2) ee Basic Network Concepts. . Se Minimum Cost Flow 2... 6 ee Maximal Flow... te Primal-Dual Transportation Algorithm ‘Summary es Exercises. 2 ee Unconstrained Problems Basic Properties of Solutions and Algorithms First-Order Necessary Conditions... 2... Examples of Unconstrained Problems 2... . Second-Order Conditions . Convex and Concave Functions : Minimization and Maximization of Convex Functions . Global Convergence of Descent Algorithms Speed of Convergence . Summary Pere . Beenie gece Basic Descent Methods Fibonacci and Golden Section Search... 1 ws Line Search by Curve Fitting... oe Global Convergence of Curve Fitting . Closedness of Line Search Algorithms Inaccurate Line Search Fs ‘The Method of Steepest Descent Applications of the Theory Newton's Method ae 889 90¢ ose 78 998 104 10 17 121 123, 126 133 134 7 141 149 157 158 168 170 174 176 180 182 189 193, 194 197 207 2 214 220 22519 7.10 7 712 Chapter 8 81 8.2 83 84 85 8.6 87 88 Chapter 9 91 9.2 93 94 95 9.6 o7 98 99 9.10 PART IL Chapter 10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9 10.10 Chapter 11 et Coordinate Descent Methods Spacer Steps . Summary... Exercises Conjugate Direction Methods Conjugate Dire ns, Contents Descent Properties of the Conjugate Direction Method . The Conjugate Gradient Method ‘The Conjugate Gradient Method as an Optimal Process . The Partial Conjugate Gradient Method . Extension to Nonquadratic Problems . Parallel Tangents Exercises Quasi-Newton Methods Modified Newton Method Construction of the Inverse Davidon-Fletcher-Powell Method . The Broyden Family Convergence Properties Scaling . Memoryless Quasi-Newton Methods | Combination of Steepest Descent and Newton's Method . Summary Exercises, Constrained Minimization Constrained Minimization Conditions Constraints Tangent Plane First-Order Necessary Conditions (Eauaty Constraints) « Examples Second-Order Conditions . Eigenvalues in Tangent Subspace Sensitivity... . . Inequality Constraints . Summary... . Exel Primal Methods Advantage of Active Set Methods imal Methods Feasible Direction Methods . 27 230 21 232 238 241 243 246 248 252 254 287 261 263 265 268 2m 25 279 282 287 288 295 297 300 301 306 312 314 318 318 322 323, 326xiv Contents 4 WS 11.6 17 11.8 11.9 11.10 Chapter 12 124 12.2 12.3 12.4 25 12.6 12.7 12.8 12.9 12.10 Chapter 13 B41 13.2 13.3 13.4 13.5 13.6 13.7 13.8 13.9 Chapter 14 141 14.2 14.3 14.4 14.5 14.6 14.7 14.8 14.9 ‘Appendix A Al A2 AB ‘The Gradient Projection Method... Convergence Rate of the Gradient Projection Method - The Reduced Gradient Method . . Convergence Rate of the Reduced Gradient Method Variations © 6. ee Summary Exercises Penalty and Barrier Methods ‘Penalty Methods . Barrier Methods . of Penalty and Bartier Functions » Penalty Functions ts and Penalty Methods Normalization of Penalty Functions : Penalty Functions and Gradient Projection . Exact Penalty Functions Summary Exercises Dual and Cutting Plane Methods Local Duality. 2. 0 ee Dual Canonical Convergence Rate... . Separable Problems . Augmented Lagrangians The Dual Viewpoint Cutting Plane Methods Kelley's Convex Cutting Plane Algorithm Modifications . tae Exercises Lagrange Methods Quadratic Programming Direct Methods Relation to Quadratic Programming Modified Newton Methods Descent Properties . Rate of Convergence Quasi-Newton Methods Summary Exercises Mathematical Review Sets oe Matrix Notation... Spaces 330 337 345 350 357 359 360 366 369 311 378 380 382 384 387 391 392 397 402 403 4 416 48 420 421 423 aay 433, 435 39 444 446 449 450 455 456 457Appendix C Bibliography Index Eigenvalues and Quadratic Forms . Topological Concepts . . . . . - Functions fae Convex Sets Basic Definitions Hyperplanes and Polytopes Separating and Supporting Hyperlanes Gaussian Elimination Contents xv 458 459 476 487Chapter 1 INTRODUCTION 1.1 OPTIMIZATION The concept of optimization is now well rooted as a principle underlying the analysis of many complex decision or allocation problems. It offers a certain degree of philosophical elegance that is hard to dispute, and it often offers an indispensable degree of operational simplicity. Using this optimization philosophy, onc approaches a complex decision problem, involving the sc- lection of values for a number of interrelated variables, by focussing attention on a single objective designed to quantify performance and measure the quality of the decision. This one objective is maximized (or minimized, de- pending on the formulation) subject to the constraints that may limit the selection of decision variable values. If a suitable single aspect of a problem can be isolated and characterized by an objective, be it profit or loss in a business setting, speed or distance in a physical problem, expected return in the environment of risky investments, or social welfare in the context of government planning, optimization may provide a suitable framework for analysis. It is, of course, a rare situation in which it is possible to fully represent all the complexities of variable interactions, constraints, and appropriate objectives when faced with a complex decision problem. Thus, as with all quantitative techniques of analysis, a particular optimization formulation should be regarded only as an approximation. Skill in modelling, to capture the essential elements of a problem, and good judgment in the interpretation of results are required to obtain meaningful conclusions. Optimization, then, should be regarded as a tool of conceptualization and analysis rather than as a principle yielding the philosophically correct solution. Skill and good judgment, with respect to problem formulation and it terpretation of results, is enhanced through concrete practical experience and a thorough understanding of relevant theory. Problem formulation itself2 Chapter 1 introduction always involves a tradeoff between the conflicting objectives of building a mathematical model sufficiently complex to accurately capture the problem description and building a model that is tractable. The expert model builder is facile with both aspects of this tradeoff. One aspiring to become such an expert must learn to identify and capture the important issues of a problem mainly through example and experience; one must learn to distinguish trac- table models from nontractable ones through a study of available technique and theory and by nurturing the capability to extend existing theory to new situations. This book is centered around a certain optimization structure—that characteristic of linear and nonlinear programming. Examples of situations leading to this structure are sprinkled throughout the book, and these ex- amples should help to indicate how practical problems can be often fruitfully structured in this form. The book mainly, however, is concerned with the development, analysis, and comparison of algorithms for solving general subclasses of optimization problems. This is valuable not only for the al- gorithms themselves, which enable one to solve given problems, but also because identification of the collection of structures they most effectively solve can enhance one’s ability to formulate problems. 1.2 TYPES OF PROBLEMS The content of this book is divided into three major parts: Linear Program- ming, Unconstrained Problems, and Constrained Problems. The last two Parts together comprise the subject of nonlinear programming. Linear Programming Linear programming is without doubt the most natural mechanism for for- mulating a vast array of problems with modest effort. A linear programming problem is characterized, as the name implies. by linear functions of the unknowns; the objective is linear in the unknowns, and the constraints are linear equalities or linear inequalities in the unknowns. One familiar with other branches of linear mathematics might suspect, initially, that linear programming formulations are popular because the mathematics is nicer, the theory is richer, and the computation simplér for linear problems than for nonlinear ones. But, in fact, these are not the primary reasons. In terms of mathematical and computational properties, there are much broader classes of optimization problems than linear programming problems that have elegant and potent theories and for which effective algorithms are avail- able. It seems that the popularity of linear programming lies primarily the formulation phase of analysis rather than the solution phase—and for good cause. For one thing, a great number of constraints and objectives that arise in practice are indisputably linear. Thus, for example. if one formulates a problem.with a hudaet constraint restrictino the total amount of money ta1.2 Types of Problems 3 be allocated among two different commodities, the budget constraint takes the form x, + x2 SR, where x,, i = 1, 2, is the amount allocated to activity i, and B is the budget. Similarly, if the objective is, for example, maximum weight, then it can be expressed as wix1 + w2x2, where w;, i = 1, 2, is the unit weight of the commodity i. The overall problem would be expressed as maximize wyxy + W2x2 subject to x1 +2 0. In more compact vector notation,t this standard problem becomes minimize 7x Q) subjectto Ax =b and x>0. Here x is an n-dimensional column vector, ¢” is an n-dimensional row vector, A is an m X n matrix, and b is an m-dimensional column vector. The vector inequality x > 0 means that each component of x is nonnegative. + See Appendix A for a description of the vector notation used throughout this book. Wes12 Chapter 2B Before giving some examples of areas in which linear programming prob- tems arise naturally, we indicate how various other forms of linear programs. can be converted to the standard form. Example 1 (Slack variables). Consider the problem minimize cix) + ¢2k2 + 7 + Cay subject to aiyx; + a@i2X2 + + dinky S by yx, + Ant, + + Anke © by Xi + Omake + 2 + Ainnkn = Dm and x 20,42 = 0,. 6.5% = 0. In this case the constraint set is determined entirely by linear inequalities. The problem may be alternatively expressed as minimize cx) + 2x2 + °° + CnXn subject to aux) + aX. + + dintn + Yt by Gy1X1 + 2X2 + + GanXn + = be Gi X1 + Amar +o + Gann + Ym = bm and 120,52 20,0... %n 20, and Vi =O. 20... Ym 20. The new positive variables y, introduced to convert the inequalities to equal- ities are called slack variables (or more loosely, slacks). By considering the problem as one having n + m unknowns x1, X2,... 5 Xm, Ys Yao - = 9 Ye the problem takes the standard’ form. The m x (n +m) matrix that now describes the linear equality constraints is of the special form [A,I] (that is, its columns can be partitioned into two sets; the first n columns make up the original A matrix and the last m columns make up an m x m identity matrix); Example 2 (Surplus variables). If the linear inequalities of Example 1 are reversed so that a typical inequality is Ginx, + GaXx2 t+ + + Ginkn = bis it is clear that this is equivalent to ant, + apts + + Oink — yr
0 is not present and hence x, is free to take on either positive or negative values. We then write uM =u 1, @) where we require u, > Qand v, > 0. If we substitute u, — 1 for x, everywhere in (1), the linearity of the constraints is preserved and all variables are now required to be nonnegative. The problem is then exptessed in terms of the n + I variables uy, v1, X25 X35. + « » Xne There is obviously a certain degrée of redundancy introduced by this technique, however, since a constant added to 1, and v, does not change x1 (that is, the representation of a given value x, is not unique). Nevertheless, this does not hinder the simplex method of solution. Example 4 (Free variables—second method). A second approach for con- verting to standard form when x, is unconstrained in sign is to eliminate x, together with one of the constraint equations, Take any one of the m equi tions in (1) which has a nonzero coefficient for x,. Say, for example, GaXy + Aakr to + dinka = bi, (4) where az, # 0. Then x; can be expressed as a linear combination of the other variables plus a constant. If this expression is substituted for x, everywhere in (1), we are led to a new problem of exactly the same form but expressed in terms of the variables x2, x3, .. . x, only. Furthermore, the ith equation, used to determine x1, is now identically zero and it too can be eliminated. This substitution scheme is valid since any combination of nonnegative var- iables 2,45, ... X» leads to a feasible x, from (4), since the sign of x1 is unrestricted. As a result of this simplification, we obtain a standard linear program having n — 1 variahles and m — 1 constraint equations. The value of the variable x, can be determined after solution through (4). Example 5 (Specific case). As a specific instance of the above technique consider the problem minimize =x, + 3x2 + 4x3 subject to x1 + 2x, +3 = 5 2n + 3x +43 = 6 eee14 Chapter 2 Basic Properties of Lit ar Programs Since x, is free, we solve for it from the first constraint, obtaining a= 5 — 2 = a3. (6) Substituting this into the objective and the second constraint, we obtain the equivalent problem (subtracting five from the objective) minimize x2 + 3x3 jectta x +x, = 4 $220, 13 = 0, which is a problem in standard form. After the smaller problem is solved (the answer is x2 = 4, x; = 0) the value for x; (x; = —3) can be found from (5). 2.2 EXAMPLES OF LINEAR PROGRAMMING PROBLEMS Linear programming has long proved its merit as a significant model of nu- merous allocation problems and economic phenomena. The continuously expanding literature of applications repeatedly demonstrates the importance of linear programming as a general framework for problem formulation. In this section we present some classic examples of situations that have natural formulations. Example 1 (The diet problem). How can we determine the most economical diet that satisfies the basic minimum nutritional requirements for good health? Such a problem might, for example, be faced by the dietician of a large army. We assume that there are available at the market n different foods and that the ith food sells at a price c; per unit, In addition there are m basic nutritional ingredients and, to achieve a balanced diet, each indi- vidual must receive at least b; units of the jth nutrient per day. Finally, we assume that each unit of food i contains ay; units of the jth nutrient. If we denote by x; the number of units of food iin the diet, the problem then is to select the x;’s to minimize the total cost cats + Cak2 to + Cake subject to the nutritional constraints ayxy + aiax. + + dinky > by GaXy + dakz to + Aankn = bz GmsX1 FH AmrXz + + AinnXn = bm2.2 Examples of Line 18 and the nonnegativity constraints x 20,2 20,000,420 on the food quantities. This problem can be converted to standard form by subtracting a nonnegative surplus variable from the left side of each of the m linear inequalities. The diet problem is discussed further in Chapter 4 Example 2 (The transportation problem). Quantities a1, a2, ... 5 diy, Te- spectively, of a certain product are to be shipped from each of m locations and received in amounts 61, b2, .. . » bn» Tespectively, at each of n desti- nations. Associated with the shipping of a unit of product from origin i to destination j is a unit shipping cost cy. It is desired to determine the amounts xy to be shipped between each origin—destination pair i = 1,2, ...'m j= 1,2,..., 3 80 as to satisfy the shipping requirements and minimize the total cost of transportation. To formulate this problem as a linear programming problem, we set up the array shown below: by by bn The ith row in this array defines the variables associated with the ith origin, while the jth column in this array defines the variables associated with the jth destination. The problem is to place nonnegative variables xy in this array so that the sum across the ith row is a;, the sum down the jth column is by, and the weighted sum >j-1 D7, cyxy, representing the transportation cost, is minimized, Thus, we have the linear programming problem: minimize S cyxy subject to > xy ‘ 7m © Day= hy for f=1,2,...50 o G=1,2,...,m5 Tae nt In order that the constraints (6), (7) be consistent, we must, of course, assume that 41 a; = Df-1 6; which corresponds to assuming that the total amount shipped is equal to the total amount received.18 Chapter 2. Basic Properties of Linear Programs The transportation problem is now clearly seen to be a linear program- ming problem in mn variables. The equations (6), (7) can be combined and expressed in matrix form in the usual manner and this results in an (m + n) x (mn) coefficient matrix consisting of zeros and ones only. Example 3 (Manufacturing problem). Suppose we own a facility that is capable of engaging in n different production activities, each of which pro- duces various amounts of m commodities. Each activity can be operated at any level x, > 0 bs ¢; dollars and yields aj units of the jth commodity. Assuming linearity of the production facility, if we are given a set of m numbers bi, b2,.. . » Om describing the output requirements of the m commodities. and we wish to produce these at minimum cost, ours is the linear program (1). Example 4 (A warehousing problem). Consider the problem of operating a warchousc, by buying and selling the stock of a certain commodity, in order to maximize profit over a certain length of time. The warehouse has a fixed capacity C, and there is a cost r per unit for holding stock for one period. The price of the commodity is known to fluctuate over a number of time periods—say months. In any period the same price holds for both purchase or sale. The warehouse is originally empty and is required to be empty at the end of the last period. To formulate this problem, variables are introduced for each time period. In particular, let x; denote the level of stock in the warehouse at the beginning of period i. Let u, denote the amount bought during period i, and let s, denote the amount sold during period i. If there are n periods, the problem is maximize (pis; — rx) subjectto xj.) =x) tu — G=1,2,...,2-1 0 Xn + Un ~ Sn wtuec i=2,...,0 m1 =0, 420, uw 20, 920, 720. If the constraints are written out explicitly for the case n = 3, they take the form Meas +x 0 a2 — un + Sp +43 =0 x + =C —x3 — dy + 55 =0 a ta |=C2.3 Basic Solutions 17 Note that the coefficient matrix can be partitioned into blocks corresponding to the variables of the different time periods. The only blocks that have nonzero entries are the diagonal ones and the ones immediately above the diagonal. This structure is typical of problems involving time. 2.3 BASIC SOLUTIONS Consider the system of equalities Ax =b, ® where x is an n-vector, b an m-vector, and A is an m x n matrix. Suppose that from the n columns of A we select a set of m linearly independent columns (such a set exists if the rank of A is m). For notational simplicity assume that we select the first m columns of A and denote the m x m matrix determined by these columns by B. The matrix B is then nonsingular and we may uniquely solve the equation Brn = b ) for the m-vector x». By putting x = (xp, 0) (that is, setting the first m components of x equal to those of xm and the remaining components equal to zero), we obtain a solution to Ax = b. This leads to the following defi- nition. Definition. Given the set of m simultaneous linear equations in n un- knowns (8), let B be any nonsingular m x m submatrix made up of columns of A. Then, if all n — m components of x not associated with columns of B are set equal to zero, the solution to the resulting set of equations is said to be a basic solution to (8) with respect to the basis B. The components of x associated with columns of B are called basic variables. In the above definition we refer to B as a basis, since B consists of m linearly independent columns that can be regarded as a basis for the space E”, The basic solution corresponds to an expression for the vector b as a linear combination of these basis vectors. This interpretation is discussed further in the next section. In general, of course, Eq. (8) may have no basic solutions. However, we may avoid trivialities and difficulties of a nonessential nature by making certain elementary assumptions regarding the structure of the matrix A. First, we usually assume that n > m, that is, the number of variables x, exceeds the number of equality constraints. Second, we usually assume that the rows of A are linearly independent, corresponding to linear independence of the m equations. A linear dependency among the rows of A would lead either to contradictory constraints and hence no solutions to (8), or to a redundancy that could be eliminated. Formally, we explicitly make the fol- lowing assumption in our development, unless noted otherwise.18 Chapter 2, Basic Properties of Linear Programs Full rank assumption. The mx n matrix A has m 0}. For this value of e the solution given by (15) is feasible and has at most p — | positive variables. Repeating this process if necessary, we can elimi- nate positive variables until we have a feasible solution with corresponding columns that are linearly independent. At that point Case | applies. Proof of (ii). Let x = (x1, x2, . . . , %,) bean optimal feasible solution and, as in the proof of (i) above, suppose there are exactly p positive variables Xi, 2). ++ 4 %p- Again there are two cases; and Case 1, corresponding to linear independence, is exactly the same as before. Case 2 also goes exactly the same as before, but it must be shown that for any e the solution (15) is optimal. To show this, note that the value of the solution x — ey is ° cx — ec’y. (16) For e sufficiently small in magnitude, x - ey isa feasible solution for positive ‘or negative values of e. Thus we conclude that ey = 0. For, if ey # 0, an e of small magnitude and proper sign could be determined so as to render (16) smaller than e7x while maintaining feasibility. This would violate the assumption of optimality of x and hence we must have'e7y = 0. Having established that the new feasible solution with fewer positive components is also optimal, the remainder of the proof may be completed exactly as in part (i). This theorem reduces the task of solving a linear programming problem to that of searching over basic feasible solutions. Since for a problem having, n variables and m constraints there are at most a m}) ~ min — m)! basic solutions (corresponding to the number of ways of selecting m of n columns), there are only a finite number of possibilities. Thus the funda- mental theorem yields an obvious, but terribly inefficient, finite search tech- nique. By expanding upon the technique of proof as well as the statement of the fundamental theorem, the efficient simplex procedure is derived. It should be noted that the proof of the fundamental theorem given above is of a simple algebraic character. In the neXt section the geometric inter- pretation of this theorem is explored in terms of the general theory of convex sets, Although the geomctric interpretation is acsthctically pleasing and25 Relations to Convexity 21 theoretically important, the reader should bear in mind, lest one be diverted by the somewhat more advanced arguments employed, the underlying elementary level of the fundamental theorem. 2.5 RELATIONS TO CONVEXITY Our development to this point, including the above proof of the fundamental equations. These results, however, have interesting interpretations in terms of the theory of convex sets that can lead not only to an alternative derivation of the fundamental theorem, but also to a clearer geometric understanding of the result. The 1 main ink between the algebraic and geometric th standard form and a points ¢ spondence as follows. The reader is referred to Appendix B for a more complete summary of concepts related to convexity, but the definition of an extreme point is stated here. Definition. A point x in a convex set C is said to be an extreme point of C if there are no two distinct points x, and x2 in C such that x = ax; + (1 — a)x, for some a, 00. an A vector x is an extreme point of K if and only if x is a basic feasible solution to (17). Proof. Suppose first that x = (1, 42)... 5 Xm, 0,0,... , 0) is a basic feasible solution to (17). Then MyM) + 2282 + + + Xml = b, where a, a2,... , am, the first m columns of A, are linearly independent. Suppose that x could be expressed as a convex combination of two other points in K; say, x = ay +(I — a)2,00,/ = 1,2,...,. To show that x is a basic feasible solution it must be shown that the vectors a1, a2, . . . , a are linearly independent. We dv this by contradiction. Suppose a), a2, . . . , a are linearly dependent. Then there is a nontrivial linear combination that is zero: yids + yoda tov + yale = 0. Define the n-vector y (yi, Yar ss ey Yay 0, 0, 1 0, x+ey20, x ey=0. We then have x = K(x + ey) + K(x — ey) which expresses x as a convex combination of two distinct vectors in K. This cannot occur, since x is an extreme point of K. Thus a, a2,... , ay are linearly independent and x is a basic feasible solution. (Although if k < m, it is a degenerate basic feasible solution.) This correspondence between extreme points and basic feasible solu- tions enables us to prove certain geometric properties of the convex polytope K defining the constraint set of a linear programming problem. Corollary 1. If the convex set K corresponding to (17) is nonempty, it has at least one extreme point. Proof. This follows from the first part of the Fundamental Theorem and the Equivalence Theorem above. 9] Corollary 2. If there is a finite optimal solution to a linear programming problem, there is a finite optimal solution which is an extreme point of the constraint set. Carallary 3. The constraint set K corresponding to (17) possesses at most a finite number of extreme points. Proof. There are obviously only a finite number of basic solutions obtained by selecting m basis vectors from the n columns of A. The extreme points of K are a subset of these basic solutions. f Finally, we come to the special case which occurs most frequently in practice and which in some sense is characteristic of well-formulated linear