his book provides a unified, insightful, and n Rune eee ee toe ecto anc eee eet ECE Ress eee Wa et eR RL Pee een een ere Peon Ge eRe ee et Pee ed ; Den nha nome koe acne eeu g John N. Tsitsiklis Contains a detaited treatment of formulations and algorithms Ore ee eee Pee ene ee tere ee a p ee once eter eee irmitris Bertsimas and John N. Tsitsiklis Perea tots See eka Saye Ocoee ate fae mena Sa eetea eral) ieee eae a eer ee an as10. ATHENA SCIENTIFIC OPTIMIZATION AND COMPUTATION SERIES Convex Analysis and Optimization, by Dimitri P, Bertsekns with Am golia Neuic aud Asuman B. Ozdaglar, 2003, ISBN 1-886520-45-0, 560 ages, Introduction to Probability, by Dimitri P. Bertsekas and John N. Tsitsiklis, 2002, ISBN 1-886529-40-X, 430 pages. Dynamir Programing and Optimal Control, 2nd Fdition, Vols. 1 an UL, by Dimitsi P. Bertsekass, 2001 ISBN 1-86520-08-6, 840 pager Nonlinear Programming, Second Edition, by Dimitri P, Bortsokns, 1999, ISBN 1-S86529-00-0, 7S0 pages. Network Optimization: Continnow snd Discrete Models, by Dimitri P. Rertselais, 1998, ISBN 1-886529-02-7, 608 pages: Network Flows and Mouotropic Optimization, by R. Tyrrell Rock. afellar, 1998, ISBN 1-884529-06-X, 634 pages. Iutroduction to Linea Optinization, by Dimitris Bertsinses: awl Jalan N. Taitsillis, 1997, ISIN’ 1-886529-19-1, GOS pages. Pavallel and Distributed Computation: Nuynerieal Methods, by Dine atri P. Bevtsekas and Jol N, ‘Tsitsiklis, 1997, ISBN 1-886520-01-4, TIB pages. Newo-Dynamic Programming, by Dimitri P. Bertsekas and Joha N. ‘Dsilsiblis, 1996, ISBN 1-886529-10-8, 512 pages. Constrained Optindzation snd Lageeuge Multiplier Methods, by Disn- it P. Bortsekas, 1995, ISBN 1-886529.04-3, 410 pages. Stochastic Optimal Control: The Disercte-Tizne Case, by Dimityi P, Bertackas and Steven E, Shrove, 1996, ISBN 1-886529-03-4, 330 pages Introduction to Linear Optimization Dimitris Bertsimas John N. Tsitsiklis Massachusetts Institute of Technology t Athena Scientific, Belmont, MassachusettsContents Preface Be 1. Introduction. weed 1.1. Variants of the linear programming problem 2 1.2, Examples of linear prograraniing problems 6 LX. Pionewiso linear cates olyjective functions 15 14. Graphical representation and solution at 1.5, Linear algebra backyronnd and notation 26 LG. Meociths and operation comnts 32 LT. Bxercises 34 18. History, notes, and sources 38 2 The geometry of linear programming. ce sal Polyhedra and convex sets 12 Extceme points, verticns, and basic feasible solutions 46 Polyhedsa in standard form Degencracy Bxistence of extreme points Optimality of extreme points Representation of bounded polyhedra Projections of polyhedra: Fouriet-Motzkin elimination” Summary ercises 2.11, Notes and sources 3. The simplex method 6. we BL 3.1. Optimality conditions 82 3.2. Development of the simplox method 87 3.3. Implementations of the simplex methor sa37, 38 3.9, 3.10. a2. 63 ba 65, 68, 67. 08. Antieyeling: lexicography and Bland’s rule Finding an initial basie feasible solution Column geometry and the simplex method Computational efficieney of the simplex method Summary Exercises Notes and sonrees Duality theory. Motivation The dual problem ‘The duality thooreut Optimal dual variables as mansinal costs Standard form problenis and the dusl sisuplex method Farkas” Jesnma and linear inequalities Fron separating hyperplanes to duality* Cones aud extreme rays Represeuitation of polyhedra General linear programming duality* Smunmiary Exercises Notes and sources Sensitivity analysis... . cae Local sensitivity analysis Global dependence cn Ue right-hand side vector 1¢ sot of all dual optianal solutions* Global depeadene? on the cost vector Parametric progromming Exercises Notes and sounves Large scale optimization. 2. 2... Delayed column generation The cutting stock problema Cutting plane methods Dantzig: Wolfe decorapesition Stochastic programming, and Benders decomposition Snanmary Exercises Notes ant sounees Contents ws. ML 19. 14 ws 129 va 139 140 142 116 155 356 105, 169 v4 iy vs3 186 Conte Sa 82. 88 a 2 93 94. 95, 9.6. 97, 98 10. 10.1. 10.2 19.3, 10.4 105. ants Network flow probleas, Graphs Pormnulation of the network flow problem ‘The network simplex algoritiim ‘The negative cost eydle algorithm, he maximum flow problew Duality in network flow problems Dual ascent methods ‘The assignment problem ae Use auction algoritho The shortest path problem ‘The thinirsvam spasining tree problem Summary Bxercises Notes and sources . Complexity of linear programming and the ellipsoid method. 2s See Efficient algorithms and computational complexity ‘Phe key geometric result bebiind the ellipsnidt met (Phe ellipsoid anethiod for the feasibility problem Phe ellipsoid method for optimization Problems with exponentially many constraints? Summary Exerc Notes and sources Interior point methods... "The affine sealing algorithm Convergence of affine sealing* “The potential reduction algorithen ‘The primal path following algorithy The primak-dual path following algoritlm An overview . Notes and sources Integer programming formulations Modicling teckmiques Guidelines for strong formulations Moving with exponentially many constraines Summary Bxoreises 393 395 sot 09 419 431 was 440 us ast 452 G1 4165 am 472106. a. 1d 112. na, 1a. 115. 116, IL? 1s, 119) 11.10. ay 12. a 12.2 123. 124, 125, 126, 12.7 Bs. Contents Notes and sources air Integer programming methods... sao Cutting plane methods 480 Branch and bound soll 485 Dynamic programming a 490 Integer prossemming duality | 494 Approximation algorithins 507 Local seareh . Su Simalated annealing | | 512 Complexity theory 5d Summary coll co sae Bxercises : 523 Notes md sonroes 580 The art ft linear optimization... ss a3 Modeling languages for Hinear optimization 534 Linear optimization libraries wad general obsorvations 535 The fleet assignment problem 537 ‘The air traffic flow management problem. Bad ‘The job shop scheduling problom : 551 Summary... 502 Exercises 563 Notes and sources 567 References 860 Index ee 8D Preface The purpose of this book is to provide a unified, insightful, and modern ‘treatment of linear optimization, that is, linear progeamming, network flow problems, and discrete linear optimization. We discuss both classical top- ics, as well as the stato of the art. We give special attention to theory, but also cover applications and present case stncies. Our main objective is to help the reader become a sophisticated practitioner of (linear) optimiza tion, ot a researcher. More specifically, we wish to develop the abilit: formulate fairly complex optimization problems, provide an appreciation of the main classes of problems that ase practically solvable, describe the available solution methods, and build an understanding of the qualitative properties of the zatutions they provide. Our general philosophy is that insight matters most, For the sub- ject matter of this book, this necessarily reqwiees a geometric view. On the other hand, problems are solved by algorithms, and these can only be described algebraically. Hence, our focus is on the beantiful interplay between algebra and geometry. We build understanding using figures and geometric arguments, and then translate ideas into algebraic formulas and algorithies, Given enough time, we expect that the reader will develop the ability to pass from one domain to the other without much effort. “Another of our objectives is to be comprehensive, but economical. We have made an effort to cover and highlight all of the prineipal ideas in this field. Howover, we have not tried t0 be encyclopedic, or to diseuss every possible detail relevant to a particular algorithm. Our premise is that once mature understanding of the basie principles is in place, further details can bbe acquired by the reader with little additional effort Our last objective is to bring the reader up to date with respect to the state of the art. This is especially true in our treatment of interior point methods, large seale optimization, and the presentation of case studies that stretch the lisnits of currently available algorithms andi computers, ‘The success of any optimization methodology hinges on its ability ta deal with large and important problems. In that senso, the last chapter (on the art of linear optimization, is a critical part of this book. Tt will, we bhope, convince the reader that progress on challenging problems requires doth problem specific insight, as well as a deeper understanding of the underlying theory.xii Preface In any book dealing with linear programming, there are some impor- tard choices to be made regarding the treatment of the simplex method. ‘Traditionally, the simplex method is developed in terms of the full simplex tableau, which tends to become the central topie, We have found that the full simplex tableau is a useful device for working out numerical examples. Sut other than that, we have tried not to overemphasize its importance. Let us also mention another deporture from many other textbooks. Introductory treatments often forus on standard form problems, whieh is sufficient for the purposes of the simplex method. On the other hand, this approach often Jeaves the reader wondering, whether certain properties are generally tre, and can hinder the deeper understanding of the subject. We depart from this tradition: we consider the geneval form of linear progran- ming problems and define key concepts (e.g., extreme points) within this context. (Of course, when it comes to algorithms, we often have to special ize to the standard forma.) In the same spirit, we separate the structural understanding of linear programming from the particxlars of the simplex method. For example, we include a derivation of duality theory that does not rely on the simplex method Finally, this Beak contains a treatment of several important topics that are not commonly covered. ‘These inchude @ discussion of the ool- umn geometry and of the insights it provides into the efficiency of the sirsplex method, the connection between duality and the pricing of finam- cial assets, 2 unified view of delayed column generation and cutting plane methods, stochastic programming and Benders decomposition, the auction algorithm for the assignment problem, certain theoretical implications of the ellipsoid algorithm, a thorough treatment of interior point’ methods, ant a whole chapter on the practice of linear optimization. There are also several noteworthy topies that ate cavered in the exercises, such as Leontief systems, strict complementarity, options pricing, von Neumann's algorithm, submodular function minjtnization, and bounds for a mumber of intoger programming problems. Here is a chapter by chapter description of the book. Chapter 1: Introduces the linear programming problem, together with ‘a number of examples, and provides some background material on linear algebra, Chapter 2: Deals with the basie geometric properties of polybedra, focus- ing on the definition and the existence of extreme points, and emphasizing the interplay betwen the geometric and the algebraic viewpoints. Chapter 3: Contains more or les the classical material associated with the simplex method, as well as a discussion of the cohima geometry, Ut starts with a high-level and gcometrically motivated derivation of the simplex thiod, It then introduces the revised simplex method, and concludes ith the simplex tzbieatt, The usual topies of Phase I and anticycling are Preface also covered. Chapter 4: It is a comprehensive treatment, of linear programming, du ality. ‘The duality theorem is first obtained as a corollary of the simplex method. & more abstract derivation is also provided, based on the separat- ing hyperplane theorem, which is developed from first principles. Tt ends with a deeper look into the geometry of polyhedra. Chapter 5: Discusses sensitivity analysis, that is, the dependence of s0- Intions and the optimal cost on the problem data, including parametric programming. It also develops a characterization of dual optimal solutions as subgradients of a suitably defined optimal cost funetion. Chapter 6: Presents the complementary ideas of delayed column gen- eration and cutting planes. These methods are first developed at a high level, and are then made concrete by discussing the cutting stock prob- em, Dantzig-Wolfe decomposition, stochastic programming, and Benders decompasition. Chapter 7: Provides a comprehensive review of the principal results and methods for the different variants of the network fow pratlem. It contains representatives from all major types of algorithms: primal descent (the simplex method), dual escent (the primal-dual method), and approximate ‘ual ascent (the auction algorithm). The focus is on the major algorithmic ideas, rather than on the refinements that can Yead to better complexity estimates. Chapter 8: Includes a discussion of complexity, a development of the el lipsoid method, and a proof wf tlve polynomiality of linear programming. It also discusses the equivalence of separation and optimisation, and provides ‘examples where the ellipsoid algorithm can be used to derive polynomial time results for problems involving an exponential number of constraints, Chapter 9: Contains an overview of all major classes of interior point methods, including affine scaling, potential reduction, and path following (both primal and primal-dual) methods. It includes a discussion of the ‘waderlying geometric ideas and computational issues, 28 well as convergence proofs and complexity analysis Chapter 10: Introduces integer programming formulations of disexete optimization problems. It provides a number of examples, as well as some intuition as to what constitutes a “strong” formulation, Chapter 11: Covers the major classes of integer programming algorithms, including exact methods (branch and bousd, cutting planes, dynamic pro- gramming), approximation algorithms, and heuristie methods (local search and sisnuleted annealing). It also introduces a duality theory for integer programming Chapter 12: Deals with the art in linear optimization, i.c., the processxiv Preface cof modeling, exploiting problem structure, and fine tuning of optisization algorithms. We discuss the relative performance of Interior point metlt- ods and different variants of the simplex method, in a realistic large scale setting. We also give some indication of the size of problems that can be eurrenily solved. An important theme that mins through several chapters is the model ‘ng, ecmplexity, and algorithms for problems with an exponential number constraints. We discuss modeling in Section 10.3, complexity in Section 8.5, algorithmic approaches in Chapter 6 ead 8.5, and we conclude wth case study in Section 125. ‘There is a fait number of exervises that are given at the end of each chapter. Most of them are intended to deepen the understanding of the subject, or to explore extensions of the thoory in the text, as opposed to routine drills, However, several numerical exercises are also included. Starrod exereises are supposed to be fairly hacd. A solutions yamal for ‘qualifed instructors can be obtained from the authors. We have made a special effort to keep the text as modular as possible, allowing the reader to omit certain topics without loss of continuity. For example, much of the material in Chapters 9 and 6 is rarely used in the rest of the book, Furthermore, in Chapter 7 (on network flow problems), a rearlor who las gonte tarot the problem formatation (Sections 7.1-7.2) ean immediately move to any later section in that chapter. Also, the interior point algorithms of Chapter 9 are not used later, with the exception of some of the applications in Chapter 12. Bvon within the core chapters (Chapters 1-4), there aro many sections that can be skipped ditring a first ending. Some sections have been marked with a star indicating that they contain somewhat more advariced material that js not «ually covered in an introductory course. ‘The book was developed while we took tus teaching a first-year graduate course at M.LT., for students in engineering and operations re- search. ‘The only prerequisite is a working knowledge of linear algebra. In fact, it is only a small snbset of linear algebra that, is needed (e.g.. the cancepts of subspaces, Hmmear Independence, and the sank of a matrix}. Howover, these elementary tools are sometimes used in subtle ways, and sonie mathematical maturity on the part of the vader can lect to a better appreciation of the subject. The book can be used to teach several different types of courses. The first two suggestions below ase one.semester variants that we have tried at ALT, but there are also other meaningful alternatives, depending on the students’ background and the course's objectives, (a) Cover most of Chapters 1-7, and if time permits, cover a small number ‘of iopics from Chapters 9-42. (b) An alternative could be tho same as above, except that interior point Proface ~ algorithms (Chapter 9) are fully covered, replacing network flow prob- leas (Chapter 7) () A broad overview course cant be constructed by concentrating on the casier material in most of the chapters, The core of such a course could consist of Chapter 1, Sections 2.1-2.4, 31-35, 41-43, 5.1, 71+ 7.3, 9.1, 10.1, some of the easier material in Chapter 11, and an application from Chopter 12, Finally, the book is also suitable for a half-course on integer pro- gramming, based on parts of Chapters 1 and 8, as well as Chapters 10-12, ‘Phere is a truly large literature on linear optimization, and we make no attempt to provide a comprehensive bibliography. ‘To a gveat extent, the sources that we cite axe either original references of historical interest, oF recent texts where additional information can be found. For those topics, however, that touch upon current research, we also provide pointers to recent journal articles. We would like to express our thanks to a number of individuals. We are grateful to our colleagues Dimitri Bertse¥as and Rob Freund, Tor nnany discussions on the subjects in this book, as well as for reading parts of the viamusetipt. Several of our stuclents, colleagues, and feiends have com tributed by reading parts of the manuscript, providing critical comments, and working on the exercises: Jim Christodouteas, ‘Phalia Chryssikou, Austin Frakt, David Gamarnik, Leon Hsu, Spyros Kontogiorgis, Peter Mar~ bach, Gina Mourtzinou, Yannis Paschalidis, Georgia Porakis, Lakis Poly- maenokos, Jay Seturaman, Sarah Stock, Paul ‘Tseng, and Ben Van Roy. But mostly, we are grateful vo our families for their patience, love, and support in the eaurse of this long, project. (@ Dimitris Bertsimas John N. Tsitsiktis Cambridge, January 1997Chapter 1 Introduction —— 1.1. Variants of the linear programming problem 1.2. Examples of linear programming problems 1.3. Pieoowise linear convex objective functions 1.4. Graphical representation and solution 1.5. Linear algebra background and notation 1.6. Algorithms and operation counts 1.7. Exercises 1.8. History, notes, and sources Contents2 Chap. 1 Introduction In this chapter, we introduce finear programming, the problem of mini- rnjzing a linear cost function subject to linear equality and inequality con- straints. We consider @ few equivaleut forms and then present a number of examples to illustrate the applicability of linear programming to a wide variety of contexts, We also soive a few simple examples and obtain some basic geometric intuition on the nature of the problem. "The chapter ends with a review of linear algebra and of the conventions used in describing the computational requirements {operation count) of algorithms. 1.1 Variants of the linear programming problem In this section, we pose the linear programming problem, discuss a few special forms that it takes, and establish some standard notation that we will be using. Rather than starting abstractly, we frst state a conceate eximple, which is meant to facilitate understanding of the formal definition that will follow, ‘The example we give is devoid of any interpretation. Later on, in Section 1.2, we will have ample opportunity te develop examples that arise in practical settings. Example 1.1. The following is @ linear programming problem: minimize Qe — ra + subject to 3) + te <2 32 ~ =5 rtm 23 a zo oy <0, Here, 9523, nd 2 are variables whose vasa to he dose to minimize the Hear cost fiction 22y + Ag, sujet to nt of linear equality and Srequniny contrainte Some of tse conse, such as 21 > 0 ad 23 0, tmount 0 simple rotitions ot the ign of certain variables ‘The remaining Constrnints are of the fm a’ ©. a's bara’ b whore a= (aso to) ie ven vector, = (aia za) tthe yetr of acon vale, ae thei inner product Sol, ar and b age eae. For example, inthe fst Consteists we tave a Us 10s1) and b= 2 ‘We now generalize. In a gener! linear programming problem, we are given a cast weelar © = (€i,.+-5¢q) aad we scek to minisnize a linear cost funetion e'¢ = 7, ei ower all n-dimensional vectors x = (£1,---y.y)s Tg Gauavad farther i Seton 1.5, al vectors ze asuped tobe column vectors, ad ao treats nth in atrbeveetor pradicts. Tow vectors ae nated a transpose OF (coinmn) sectors Hoover, Shenert we cee to a velar ise the text we ge te mare eeonomical notation x ~ (21.4) even though ian column vector ‘ha who wine with or tain riny wih ena Seen 3 ne See. 1.1 Variants of the linear prograsnming problem 3 subject to a set of linear equality and inequality constraints. In particulary let My, Mp, Mg be some finite index sets, and suppose that for every # in any one of these sets, we are given an n-dimensional vector aj and 0 scaler },, that will be used to form the ith constraim.. Let abso Ny end Na be subsets of {I,...,10} that indicate which variables 2; are constrained to be honmtegative or nompesitive, respectively; We then cossider the problem minimize ¢'x subject to alx Sb, tM. axe, GEM, ax ab, iC Ms, av m0 7eMn 2 7e Ne ‘The variables 21,...,a% are called decision variables, and a vector x sat= isiving all of the consteaints is called a feasible solution or feasible vector ‘The sct of all feasible solutions is called the feasible set or feasible region. Ifj is in ncither NY, nor Na, there are no restrictions on the sign of 2), in which case we say that 2; 1s a free or wnrestricted variable. ‘Che function nis called the objective function or cost function. A feasible solution x* ‘hat minimizes the objective funetion (that is, cx" < ex, forall feasible x) is called an optimal feasible solution or, simply, an optima solution. ‘Yhe value of ex" is then called the optimal cost. On the other hand, if for every real number K wo cun find a feasible. solution x whose cost is less than K., we say that the optimal cost is —oo or that the cost is unbounded telow. (Sometimes, we will abuse terminology and say thet the problem is untounded) We finally note that, there is no need to study maximization problems separately, because maximizing €'x is equivalent to minimizing the linear cost function —e'% "An equality constraint a(x = 6; is equivalent to the two constraints alx <>, and alx > ty. [a addition, any constraint of the form a/c < by can, be rewritten as (~a,)'x > ~by. Finally, constraints of the form #) > @ ar 1) <0 are special cases of constraints of the form ax 2 by, where ay is @ ‘unit vector and 6 = @. We conclude that the feasible set in a general linear programming problem can be expressed exclusively in terms of inequality constraints of the form a/x > b. Suppose that there is a total of m such constraints, indexed by i = 1,... mt, Wet b= (hip. sBm)> and tet A be the sm x n matrix whose rows are the Tow vectors a... jas Chat js, A- ‘Then, the constraints alx > b,, i= 1,..-.m, ean be expressed compactly Me eee Awe > te acl tha Hnear nenarammine problem can be written,4 Chap. 1 Introduction as minimize e'x subject to Ax > b, (1.2) Inequolities such as Ax > b will always be interpreted eomponentwise; that is, for every ¢, the ith component of the vector Ax, which is ax, is greator ‘than or equal to the ith componest &, of the vector b. Example 1.2 The linear programming problem in Example 1.1 can be rewrite minimize 22, ~ 2 + 4s subject tor, — 2 eee Ser ~ 29 ~3r + 25 Whiel is of the same form as the problem oo 0 3 1 0 0-3 1 0 A o 0 tooL 10 0 0 0 0-1 0 and B= (-2,5,-6,3,0,0) Standard form problems A linear programming problem of the form subject to Ax = b (13) x20, Is said to be in standard fore, We provide an interpretation of problems in standard form, Suppose that x has dimension 1s aul let Ai,..., A, be the columns of A. Then, the constraint Ax —b can be written in the form Intuitively, there are n available resource vectors A1,...,An, and a target vector b, We wish to “synthesize” the target vector b by using a non hogative amour =; of each resouree vector Ay, while minimizing the cost 1 Gti, where c; is the unit cost of the ith resource. The following is a more concrete examnle See. I.1 Variants of the linear programming problem 5 Example 1.3 (The diet problem) Suppose that there ate n different foods ‘and m different nutrients, and that we are given the following table with the nutritional content of a unit of each food food 1 + food n cuiziont 1 [an an Let A be the m xn matrix with enisies a,,. Note that the jth column A, of this matrix represents the nutritional content of the jth food. Jet b be a vector with the requirements of an ideal diet or, equivalently, a specification of the nutritional contents of an “ideal food.” We then interpret the standaed form problem as the problem of mixing nonnegative quantities 2, of the available foods, to synthesize the idea) food at minimal cost. In a variant of this problem, the vector b specifies the minimal requirements of an avlequate diet; in that case, the constraints Ax = b are replaced by Ax > ', and the problem is not in standard form. Reduction to standard form AAs argued earlier, any linear programming problem, including the standard form problem (1.3), i8 a special case of the general form (1.1). We now ‘argue that the converse is also true and that. a general linear programming problem can be transformed into an equivalent problem in standard form, Here, when we say that the two problems axe equivalent, we mean that. given 1 feasible solution to one problem, wo can construct a feasitie solution to the other, with the same cost. In particular. the two problems have the same optimal cost and given an optimal solution to one problem, wo can construct. an optintal solution to the other. ‘The problem transformation we have in mind involves two steps: (a) Elimination of free variables: Given an unrestricted variable 2, in a problem in general form, we replace it by x} —2,, where e and 2 fare new variables on which we impose the sign constraints 7 > 0 and 2, > 0. The underlying idea is that any real number ean be written as the difference of two nonnegative numbers, (b) Elimination of inequality constraints: Given an inequality constraint of the form. Yaya Shs6 Chap. 1 Introduction Wwe introduce a new variable s; and the standard form constraints Sages ta, = by 520. Such a variable », is called a slack variable. Similatly, an inequality constraint $77) aij) > b, can be put in standard form by intro ducing a surplus variable sj and the constraints 27 a2) ~ 5) — by 20. We conclude that a general problem can be brought into standard form and, therefore, we only need to develop methods that are capable of solving standard form problems, Example 1.4 The problem minimize 2ey dere subject tom + m2 > 3 Beh Ory = Md nm 20, is equivalent to the standard fostn problems minimize 261 + def — any subject tot + ay ~ ap ay 8 Bn, + taf — 205 =u sea} ap ay 2 0. For example, given the feasible solution (a) 22) = (6, ~2) to the original prob: Jem, we obtain the feastble solution (14,2,25 25) = (6,9,21) to the standard form problem, which has the same cost. Conversely, given the fessible solstion (v1.22, ,23) = (8,1,6,0) to the standard form problem, we obtain the feasible solution (x1, 12} = (8, 5) to the original problem with tie same cost In the sequel, we will often use the general form Ax > b to develop the theory of linear programming, However, when it comes to algorithms, and especially the simplex and interior point methods, we will be focusing on the standard form Ax = b, x > 0, which is computationally more convenient. 1.2 Examples of linear programming problems In this seetion, we discuss a number of examples of linear programming, problems. One of our purposes is to indicate the vast range of sitiations to which linear programming can be applied. Another purpose is to develop some familiarity with the art of constructing mathematical formulations of lonsely defined ontimization problems, Sec. 1.2 Examples of linear programming problems 7 A production problem ‘A firm produces n different goods using m different raw materials. Let b,, i= 1,...m, be the available amovnt of the ith raw material. ‘The jth good, 7 — 1,-..,7, reqntives aj, units of the ith material and results in a revenue of ¢; per umit produced. ‘The firm faces the problem of deciding how much of each good to produce in order to maximize its total revenne. 1 this example, the choice of the decision variables is simple. Let 2), J =1,.-yn, be the amount of the jth good. Then, the problem facing the firm can be forumfated as follows: maximize ery to + Gata subject to agar t+ + inte Sb t= Dees, 520, j=h Production planning by a computer manufacturer ‘The example that we consider here is a problem that Digital Equipment Corporation (DEC) had faced in the fourth quarter of 1988. It illustrates tne complexities and uncertainties of real world applications, as well as the usefulness of mathematical modeling for making important strategie decisions. mn the second quarter of 1988, DEC introcued a new family of (single CPU) computer systems and workstations: GP-1, GP-2, ama GP-3, which are general purpose computer systems with different memory, disk storage, ‘and expansion capabilities, as well as WS-1 and WS-2, which are work- stations. In Table LL, we list the models, the list prices, the average disk usage per system, and the memory usage. For example, GP-1 uses four 256K memory boards, and 3 out of every 10 units are produced with a disk ative, System | Price | # disk drives | # 260K boards GP 860,000 | 03 4 @p-2 | $10,000 ur 2 aps | so] 0 2 WS-1 830,000 | 14 2 WS-2 | $15,000 oO 1 ‘Table 1.1: Features of the five different DEC systems. Shipments of this new family of products statted in the third quarter and ramped slowly during the fourth quarter. The following difficulties aura antininatad far tha newt anarter:8 Chap. 1 Introduction (2), The in-house supplier of CPUs could provide at most 7,000 units, due to debugging problems, (b) ‘The supply of disk drives was uncertain and was estimated by the manufacturer to be in the range of 3,000 to 7,000 units {©) The supply of 256K memory boards was also limited in the range of 8,000 to 16,000 units ‘On the deman side, the marketing department established that the ‘maximum demand for the fist quarter of 1989 would be 1,800 for GP-1 systoms, 300 for GP- systems, 3,800 systems for the whole GP family, and 3.200 systems for the WS family. Included in these projections were 500 orders for GP-2, 500 orders for WS-1, and 400 orders for WS-2 that had already been received and had to be fulfilled in the next. quarter. In the previous quarters, in order to address the disk drive shortage, DEC had produced GP-1, GP-3, and WS-2 with no disk drive (although 3 out of 10 customers for GP-1 systems wanted a disk drive), and GP-2, WS-1 with one disk drive. We refer to this way of configuring the systems as the constrained miode of production. In addition, DEC could address the shortage of 256K memory boards by using to alternative boards, instead of four 256K memory boards, in ‘the GP- system, DEC could provide 4,000 alternative boards for the next quarter. It was clear to the manufacturing staff that the problem had become complex, as revenue, profitability, and customer satisfaction were at isk, ‘The following decisions needed to be made: (a) The production plan for the first quarter of 1989. ib) Concerning disk drive usage, should DEC continue to manufacture products in the constrained mode, or should it plan to satisly eus- ‘tomer preferences? (©) Concerning memory poards, showld DEC use alternative memory boards for its GP-1 systems? (4) A final decision that had to be made was related to tradeofis be- ‘tween shortages of disk drives and of 256K memory boards. The manufacturing staff would like to concentrate their efforts on either decreasing the shortage of disks or decreasing the shortage of 256K memory boards. Hence, they would like to know which alternative would have a larger effect on revenue. In order to model the problem that DEC faced, we introduce variables iy 22, 74,14, 25, that represent the number (in thousands) of GP-21, GP- 2, GP-3, WS-1, and WS-2 systems, respectively, to be produced in the next quarter. Strictly speaking, since 1000. stands for number of units it ‘mist be an integer. ‘This eas be accomplished by truncating each 2 alter ‘the third decimal point: given the 0A (min demand for WS-2) yy 22,84, F4,F5 > O. Notice that the objective function is in miions of dollars, ts vame respects, this is a pessimistic forntulation, because the 256K memory and disk drive availability were set to 8 and 3, respectively, which is the lowest ‘alo in the range that wes estimated, Ut is actually of interest to determine the solution to this problem as the 256K memory availability ranges from 8 to 16, and the disk drive availability ranges from 3 to 7, because this provides valuable information on the sensitivity of the optimal solution ot fvailability. In another respect, the formulation is optimistie because, Tor example, it assumes that the revenue from GP-I systems is 6021 for any {ty < 18, even though a demand for 1,800 GP-I systems is not guaranteed. In order to accommiodate the other three choices that DEC had, sore: of the problem constraints have to be modified, as follows. If we use the tunconstrained inode of production for disk drives, the constraint 22-+24 <3 is replaced by 0.3m + 1.72 + 1dr <3 Furthermore, if we wish to use alternative memory boards in GP-1 systems, wwe replace the constraint 4; + 219 + 2ry + 2rq tats < 8 by the two10 Chap. 1 Introduction constraints Qn <4, Pap + 20y 42g tay <8. ‘The four combinations of dhoiecs lead to four different linear programming problems, each of which needs to be solved for a variety of parameter values because, as discussed earlier, the right-hand side of some of the constraints is only known to lie within @ certain range. Methods for solving linear programming problems, when certain parameters are allowed to vary, will be studied in Chapter 5, where this ease study is revisited, Multiperiod planning of electric power capacity A state wants to plan its electricity eapacity for the next T years. ‘The stale has a forecast of dy megawatts, presumed accurate, of the demand for electricity during year t= 1,....7°. The existing capacity, which is in oil-fired plants, that will not be retired aud will be available during year £, is ex. There are two alternatives for expanding electric capacity: eonl- fed or nuclear power plants. ‘There is a capital cost of e per megawatt of coal-fired capacity that becomes operational at the beginning of year t ‘The corresponding capital cost for nuclear power plants i n,. Por various political and safety reasons, it has been decided that no more than 20% of the total capacity should ever be muclear. Coal plants last for 20 years, while mielear plants last for 15 years. A least cost capacity expansion plan is desired “The first step in formulating this probler as a linear progresring problem is to define the decision variables. Let xy and ye he the amount of coal (respectively, nuclear) eapacity brought on line at the beginning of year f. Let we and 2 be the total coal (respectively, nuclear) capacity available in year t. ‘The cost of a capacity expansion pian is therelore, tore + new): Since coal-fired! plants Inst for 20 years, we have ue Mom ¢t max(ict—19} Similacly, for nuctear power plants, yw a~mati 1-14) See, 1.2. Examples of linear programming problems ut Since the available capacity must meet the forecasted demand, we require wba be ed, CMT Finally, since no more than 20% of the total capacity should ever be nuclear, wwe have £02, wrnte = which can be written as 0.82; — 0.2u, < 0.2c¢ ‘Summarizing, the eapacity expansion problem is as follows: minimize (evr, + ney) subject to aw SD a Dey ee amenit—2a) mete de ee baLeTs 08%, — 0.20, < O.2e, Pade Te Zee Yes ths 2 > 0, We note that this formulation is not entirely realistic, because it disregards certain economies of scale that may favor larger plants. However, it can provide a ballpark estinate of the true cost, A scheduling problem in the previous examples, the choice of the decision variables wos faiely straightforward, We now discuss an example where this choice is less obv ‘A hospital wants to make a weekly night shift (12pm-8am) schedule for its nurses. ‘The demand for nurses for the night shift on day j is an integer dj, j = 1,.-.47. Every nurse works § days in a row on the night shift. ‘The problem is to find the minimal number of nurses the hespital neetis to hire. ‘One could try using a decision variable y; equal to the number of iurses that work on day j. With this definition, however, we would not be able to capture the constraint that every nurse works 5 days in a row. For this reason, we choose the decision variables differently, and define x as2 Chap. 1 Introduction the number of nurses starting their week on day j. (For example, a nurse whose wook starts on day § will work days 5,6,7,1,2.) We then have the following problem formulation: minitsive 2 + xy + ryt ae subject tom 4m ae nba + a5 n+ ap by + ay ay bm bay by ta +25 + 24 ty ta 2; integer. 2 20, This would be a lincar programming problem, except for the constraint that each rj must be an integer, and we actually have a linear integer program- ming problem. One way of dealing with this issue is to Ignore (“relax”) the integrality constraints and obtain the so-called linear programming re- lazation of the original problem. Because the linear programming, problem has fewer constrainis, and therefore more options, the optimal cost will be Jess than or equal to the optimal cost of the original problem. If the optimal solution to the linear programming relaxation happens to be integer, then {is also an optimal solution to the original problem, Ifit is not integer, we ‘can round each 2; upwards, thus obtaining a feasible, but not necessarily ‘optimal, solution to the original problem. It turns out that for this partic- ular problem, an optimal solution can be found without too much effort However, this is the exception rathor than the rule: finding optimal sola- tions to general integer programming problems is typically difficult; some methods will be discussed in Chapter 11 Choosing paths in a communication network Consider a communication network consisting of m nodes. Nodes are con- nected by communication Tinks. A link allowing one-way transmission from. node i to node j is described by an ordered pair (i,j). Let A be the set of all links. We assume that each link (i,j) € A can carry up to tis bits vet second, There is a positive charge c,) per bit transmitted along that link, Bach node k generates data, at the rate of D* bits per second, that have to be transmitted to node &, either through a direct link (k.#) or by tracing a sequence of links. The problem is to choose paths along which all data reach their inteuded destinations. while minimizing the total cost. We allow the data with the same origin aud destination to be split: and be transmitted along different paths In order to formulate this problem as a linear programming problem, \we introduce variables 2 indicating the amount of data with origin fe and See, 1.2 Examples of lineas programming peoblems 13 destination ¢ that traverse link (i,j). Let o, Wink, Bad aM, if ime 0, otherwise. network, of data with ‘Thus, Df is the net inflow at node 4, from outside th ing formulation: origin & and destination £, We then have the follo minnie So Seuslf wines Bet at subject to SD alfa GbE Theat wudaien idea Yoel sms, ned all > 0, WEA BE= Teen “The frst constraint is a flow conservation constraint at node i for data with origin K and destination € ‘The expression Ls (Gea) represents the amount of data with origin and destination & and ¢, respec- tively, that leave node # along some link, ‘The expression ya (ise represents the amount of data with the same origin and destination that center node i through some link. Finally, is the net amount of such dlata that enter uode i from outside the network, The second constraint lexpresses the requirement that the total traffic through a link (jj) cannot ‘exoced the link’s capacity. This problem is known as the multicommodity flaw problem, with the teaffic corresponding to each origin-destination pair viewed as a different commodity. A mathematically similar problem arises when we consier a. transportation company that wishes to transport several commodities from their origins to their destinations through a network, ‘There is a version of this problem, ksown as the minimum cost network flow problem, in Which we do not distinguish between different, commodities. Instead, we fre given the amount b, of external supply or demand at each node #, and the abjective is to transport material from the supply nodes to the demand nodes, at minimym cost, ‘The network Sow problem, which is the subject taf Chapter 7, contains as special cases some important problems such as the shortest path problem, the maxinmum flow problem, and the assignment neaterrT Chap. 1 Introduction Pattern classification ‘Wo are given m examples of objects and for each one, say the ith one, a description of its features in terms of an n-dimensional vector a;. Objects belong to one of two classes, and for each example we are told the class that it belongs to. More concretely, suppose that each object is an image of an apple for an orange (these are our two classes). In this context, we can use a threo-dimensional feature vector a, to summarize the contents of the ith image. The three components of a, (the features) cauld be the ellipticity of the object, the length of its stem, and its color, as measured in some seale, We are interested in designing a classifier which, given a new object (other than the originally available examples), will figure out whether itis an image of an apple or of an cranes. A linear classifier is defined in terms of an n-dimensional vector ¥ and a scalar ry1, and operates as follows. Given a new object with feature vector a, the classifier declares it to be an object of the first class if ‘and of the second class if aK < tyes In words, @ linear classifier makes decisions on the basis ofa linear combin&- tion of the different features. Our objective is to use the available examples in order to design a “good” linear classifier ‘There axe many ways of approsehing this problem, but a reasonable starting point could be the requirement that the classifier must give tho correct answer for each one of the available examples. Let $ be the set of examples of the first class. We are then looking for some x and zrug1 that satisfy the constraints Bix > eps 5 aix ana ies, ax AfOx) + (1 = NI). Note that if x and y are veetors in 3" and if A ranges in (0,1), hen points of the form dx + (1 — Aly belong to the line segment joining x and y. The definition of a convex fanction refers to the valves of f. 3s its argument traces this segment. If were linear, the inequality in port (@) of the definition would hold with equality. ‘The inequality therefore theans tat when we restrict attention to such a segment, the graph of the fimetion lies no higher than the graph of a corresponding linear function; see Figure 1.1) Xe is ensily scen that a funetion fis convex if and only if the funtion =f is concave. Note that a funetion of the form f(x) = ay + D2 where o,..-,dy ate scalars, called an affine function, is both convex and concave. (It turns out that affine functions are the only functions that are both convex and concave.) Convex (as well as concave) functions play a central role in optimization. We say that a veetar x is local minismarta of f if fix) < fy) for ally in the vicinity of x. We also say that x is « global minimum if f(x) < F9) for ally. A convex function cannot have local minima that fail to be global niinima (see Figure 1-1), and this property is of great help in designing ficient optimization algorithms. Let ¢1,-.+,€, be vectors in R", let di, the function f 2" = R defined by Flos) =, pox, (6x + a) ody, be scalars, and consider16 Chap. 1 Introduetion Ags) + ~ anfty) ee ) x Axt (l= Ay y A (a) © Figure 1.1: (a) Mlustration of the definition of «eaux function (b) A concave function. (c) A function that is neither convex nor concave; note that A i @ local, but not globsl, mivinsut, [seo Figure 1.2(a)]- Such a function is convex, as a consequence of the following resuit. ‘Theorem 1.1 Let f,, fm : R" + K be convex functions. Then, the function f defined by f() = maxset,..m fi) is algo convex. Proof, Let x,y € R" and let Ac 0,4}. We have Ox +(1-A)y) Lapa, (0% + (1 Aly) Sj anax, (MiG) + (1 A)K(9)) Spex Af) + max, (1 N49) = AG) + (1-0. u A fonction of the form max;1,,...n(€)-+d;) is called a piecewise linear ver function. A sixnple example isthe absolute value fnetion defined by S(z) = 2) = max{2,—r}. As illustrated ia Figure 1.2(b), a piecewise linear ‘convex function can be used to approximate a gencral convex function. We now consider a generalization of linear programming, where the objective function is piecewise linear and convex rather than linear: rwinimize , max, (€(x + di) subject to Ax>b, Sec. 1 Piecewise linear convex objective functions wr en dy it wz e © 0) Fignre 1.2: (a) A piecowise linear convex funtion of a single variable. (b) An appreximation of a convex function by a piecewise linear convex function: Note that maxi, .m(€f + di) is equal to the smallest number = that satisfies « 2 efx + dj for all i. For this reason, the optimization problem under consideration is equivalent to the linear programming problem subject to 2 2 eed, f= bev, Ax >b where the decision variables are 2 and x. ‘Tosummarize, linear programming can be used to solve problems with piecewise linear convex cost functions, aud the latter class of functions ean be used as an approximation of more general convex cost functions, On the other hand, such a piecewise linear approximation is not always a good idea because it ean turn a smooth function into a nousmooth one (piecewise Tinear fimctions have discontinuous derivatives). We finally note that if we are given a constraint of the form f(x) such a constraint ean be rewritten as fixtach i= ‘and linear programrming is again applicable, Problems involving absolute values ‘Consider a problem of the form rninimize See subject © Ax >b,18 Chap. 1 Introduction where X = (t1s...42m), and where the cost coefficients o; are aswamed to be nonnegative. The cost criterion, being the sum of the piecewise Jimear convex functions ©;[zs| is easily shown to be piecewise linear and convex (Exercise 1.2). Wowever, expressing this cost criterion in the form max;(e)x + dj) is a bit involved, and more ditect ronte is preferable. We observe that |2y| is the smallest number 2, that satisties 2; <2 and “1; € x, and we obtain the linear programming formulation subject to Ax > b ee SR ISL An alternative method for dealing with absolute values isto introduce new variables 1, 2; , constrained to be nonnegative, and let 2 = 2 — (Our intention i wo have 2; = x! or 2, = 2; , depending on whether 2, 8 positive or negative.) We then replace every occurrence of |2,| with ay? +25 ‘and obtain the alternative formulation minimize Sete +27) St subject to Ax! — Ax” > b xix 20 where x = (ef y..-025f) and x” = (py. rm) ‘The relations xi = xf ~ 27, f 20, 2; % 0, are not enough to guarantee that |z,| =x +27, and the validity of this reformulation may not be entirely obvious. Let’ us assume for simplicity that ¢ > 0 for all i. At an optimal solution to the reformulated problem, and for each i, we must have either 2? = or 27 = 0, because otherwise we could reduce both 2 and 2; by the same amount and preserve feasibility, while reducing the cost, in contradiction of optimality. Having guaranteed that either 2° or x, = 0, the desired relation |r) =r +4, now follows ‘The formal vomeetuess of the two reformulations that have been pre- sented here, and in a somewhat more general setting, is the subject of Exercise 1.5. We also note that the nonnegativity assurmption on the cost coefficients cis erucial because, otherwise, the cost eriterion is nonconvex. Example 1.5 Consider the problem minimize 2jz|-+22 subject to ary +02 > 4 See. 1.3 Piecewise linear convex objective functions 19 (Our frst reformulation yields minionine 250 429 subject to tz > 4 te the second yields minimize ef + 2ey 4 ae subject tof — a) + ae We now continue with some applications involving piecewise linear convex objective functions, Data fitting We are given m data points of the form (ai,bs), # = 1,.-.,m, where a ©" and by € 8, and wish to build a model that predicts the value of the variable from knowledge of the vector a. In such a situation, one often uses a linear model of the jorm b = a'x, where x is a parameter vector to be determined. Given a particular parameter vector x, the residual, or prediction error, at ‘the ith dia point is defined as |b, ~afx|. Given a choice between alternative models, one should choose a model that “explains” the available data as best as possible, ic., a model that results in small residuals, One possibility is to minimize the largest residual. This is the problem of minimizing max |b; ~ a), with respect to x, subject to no constraints. Note that we are dealing hove with a piecewise linear convex cost criterion. The following is an equivalent liuear programming formulation: subject to aly <2, i sbetaleSs FSA. m, the decision variables being 2 and x. In an alternative formulation, we could adopt the cost esiterion Sale cot20 Chap. 1 Introduction Since [b; — afx! is the smallest number 2 that satisfies by — alx < 2 and ~b, a(x < zy, we obtain the formulation miniroiae ay Hon + Fay subject 0 yale Sa, FS 1... btajx Sa, t= 1....m, Tn practice, one may wish to use the quadratic cost criterion 2 (by a{x)2, in order to obtain a “least squares fit.” This is a problema which is casier than Kinear programming; it can be solved using calculus methods, but its diseussion is outside the seape of this hook, Optimal control of linear systems Consider a dynamical system that evolves according to a model of the form x(-+1) = Ax(t) + Buc), we) = ext Here x(t) is the state of the system at time f, y(t) is the system output, assumed sealar, and u(t) is a control veetor that we are free to choose subject to linear constraints of the form Du(é) < d (these might include saturation constraints, ie., hard hounds on the magnitude of each com- ponent of u(t)). Te mention some possible applications, this could be a model of an airplane, an engine, an elecitical ccuit, a mechanical system, ‘a mamifaeturing system, or even a model of economic growth. We are also fziven the initial state x(0). In one possible problem, we are to choose the values of the controt variables w(0),..., u(T—1) to drive the state x(T) t0 4 target state, assumed for simplicity to be zero, In addition to zeroing the state, it is often desirable to keep the magnitude of the output sinall at all intermediate times, and we may wish to minimize wax v(t) We then obtain the following linear programming problem: minimize = subject to —2 D. ‘The feasible set is the shaded region in Figure 1.3, in ord to find an optimal solution, we proceed as follows. For any given scalar 2, we consider the set of all points whose cost e'x is equal to z; this is the line described by the equation 1, ~1 = z Note that this line is perpendicular to the vector ¢ = (-1,—1) Different values of = lead to different ines, all of them parallel to each other. fn22 Chap. 1 Introduction anty <3) igure 1.8: Graphical solution of the problem in Example 16 particular, increasing 2 corresponds to moving the line = = —x1 — 2 along the direction of the vector ©. Siuce we ate interested in minimizing =, we would Tike to move the line as much as possible in the direction of ~e, as long as we do not leave the Feasible region, ‘The best 180 can. do is 2 = —2 (see Figure 1.8), and the vector x = (1,1) i8 an optimal solution. Note that vhis is a corner of the feasible set. (The concept of a “earner” will be defined formally in Chapter 2.) For a problem in three dimensions, the same approach can be sed except that the set of points with the same value of e'x iv plane, instead of a line. This plane is again perpendicular to the veetor ¢, aitd the objective is to slide this plane as much as possible in the direction of ~€, a8 long as ‘we do not leave the feasible sot Example 1.7 Suppose that the feasible set is the unit eube, described by the constraints ()< 2) $1, #=1,2,, and that € = (-1,-1,-1), ‘Then, the vector X= (1,1,1) isan optimal solution. Once more, the optimal solution happens to be a corner of the feasible set (Figure 1). Sec. 1.4 Graphical representation and solution 23 * Figure 1.4: The three-dimensional linear prograniming problem in Bxample 1.7 In both of the preceding examples, the feasible set is bounded (does not extend to infinity), and the problem has @ unique optimal solution This is not always the case and we have some additional possibilities that aye illustrated by the example that follows. Example 1.8 Consider the feasible set in 98 defined by the constraints -ntm si mn 20 2 > 0, whieh is shown in Figure 1.5, (a) Por the cost veetor © = (1,1), it is clear that x = (0,0) is the wnigne opti (b) For the cast vector e = (1,0), there are multiple optimal solutions, namely. every vector x of the form <= (0,72), with 0 < xy <1, is optimal, Note Unat the set of optimal solutions is bounded. 1 solution, (0) For the cost vector e = (0,1), there are multiple optimal solutions, namely, every veetor x of the form x = (21,0), With 1 > 0, is optimal. In this cas the set of optimal solutions is unbounded (contains veetors of arbitrarily large magnitude’. (4) Consicor tho evst vector € ~ (—1, -1). For any feasible solution (21,22), we can always produce another feasible solution with loss cost, by increasing the value of x1. Therefore, no feasible solution is optimal. Furthermore, by considering vectors (11,222) with ever ineroasivg vallos of 1 and a, we ‘can obtain a sequence of feasible solutions whose cost comverges to 00 ‘We thorefore say that the optimal cost. is ~os. () If we impose an additional constraint of the form 21 4ira <—2, its evident that no feasible solution exists4 Chap. 1 Introduetion Figure 1.5: The feasible set in Example 1.8. For each choice of «, an optimal solution is obtained by moving &s much as possible in the ditection of ~e ‘To summarize the insights obtained from Example 1.8, we have the following possibilities: (a) There exists a unique optimal solution (b) There exist multiple optimal solutions; in this case, the set of optimal © solutions can be either bounded or unbounded: (©) The optimal cost is —o0, and no feasible solution is optimal (€) The feasible set is empty. In principle, there is an additional possibility: an optimal solution does nat exist even though the problem is feasible and the optimal cost is hot —90; this is the ease, for example, in the problem of minitnizing 1/2 subject to 1 > 0 (for every feasible solution, there exists another with less cost, but the optimal cost is not —o0). We will see later in this book that ty never arises in linear programming. this possibil In the examples that we have considered, if the problem has at least (one optimal solution, then an optimal solution can be found among the ‘comers of the feasible set. In Chapter 2, we will show that this is a general feature of linear programming problems, as long as the feasible set has at least. one corner. Sec. 1.4 Graphical representation and solution 25 Visualizing standard form problems. We now discuss a method that allows us to visualize standard form problems even if the dimension n of the vector x is greater than three. ‘The reason for wishing to do so is that when n < 3, the feasible set of a standard form problem does not have much variety and does not provide vawugh insight into the general case. (In contrast, if the feasible set is described by constraints of the form Ax > b, enough variety is obtained even if x has dimension three.) Suppose that we have a standard form problem, and that the matrix A has dimensions m xn. In particular, the decision veetor x is of dimension n and we have m equality constraints. We assume that m 0 [Figure 1.6(a)], and note that n= 3 and m = 1. If we stand on the plane defined by the constraint 41+ 22 +43 1, then the feasible set has the appearance of a triangle in ‘two-dimensional space. Furthermore, each edge of the triangle corresponds to one of the constraints 21,.23,23 > 0; see Figure 1.6(b) ible set in %* defined by 2a ey La () (b) 2 Figure 1.6: (a) An n-dimensional view of the feasible set. (b) Au (11 — m)-cimensional view of the same set,26 Chap. 1 Introduction 1.5 Linear algebra background and notation ‘This section provides a summary of the sain notational conventions that ‘ve will be employing. It also contains a brief review of those results from linear algebra that are used in the sequel. Set theoretic notation If S is a set and x is an clement of S, we write x € S.A set can be specified in the form § = { | x satisfies P}, as the sct of all clements having property P. The cardinality of a finite set $ is denoted by |S. The union of two sets $ and T is denote! by SUT, and their intersection by SAT. We use $\F to denote the set of all elements of $ that do not, belong toT. The notation $ CT means tha; $ is a subset of T, ic., every element of $ is also an element of T in particular, $ could be equal to 7. If in addition $ #7, we say that Sis a proper subset of T. We use 0 to denote the empty set. ‘Uhe symbols 3 and W have the meanings “there exists” and “for all,” respectively. We use ® to denote the set of real numbers, For any real mumbers @ and 6, we define the closed and open intervals (a,6] and (a,5), respectively, by fet] = (re Rlacr sd and (a,) = {we Rlace 0 for every vector x, with oquality holding if and only if x = 0. The expression vx’ is the Euclidean norm of x and is denoted by ||x|. The Schwarz inequality asserts ‘that for any two vectors of the same dimension, we have bx’y! < [xy with equality holding if and only if one of the two vectors is a scalar multiple of the other. If A is an m xm matrix, we use A, to denote its jth column, that is, Ay = (013,255.03). (This is our only exception to the rule of using lower case characters to represent vectors.) We also use a; to denote the wetor formed by the entries of the ih row, that is, a= (01042. -- thn). Il i ~ a Ay Ao An |= tot |