Preface

To make decisions optimally is one of the basic desires of a human being. In (relatively rare!)
situations where one can quantify both candidate decisions (i.e., to represent them as points x
varying in certain well-defined set X) and the criteria we are interested in (that is, to represent
these criteria as real-valued function of x ∈ X), the problem of optimal decision making falls in
the scope of Mathematics, specifically, the area called Optimization. A part of Optimization is
Mathematical Programming interested in the optimization problems of the form

max {f0 (x) : x ∈ X} (∗)

x

where the feasible set X is a subset of the space Rn of real n-dimensional vectors x, and the
objective f is a real-valued function. A typical description of the feasible set X of a Mathematical
Programming (MP) problem is by a finite system of constraints fi (x) ≤ gi :

X = {x ∈ Rn : fi (x) ≤ bi , 1 ≤ i ≤ m},

where fi (x) are given real-valued functions of n variables x, and bi are given reals. Thus, a
generic MP problem (or MP program) is of the form

max {f (x) : fi (x) ≤ bi , i = 1, ..., m} (!)

x

In other words, Mathematical Programming deals with the case when our candidate decisions
can be encoded by vectors x of certain dimension n and we are interested in finitely many criteria
represented by real-valued functions f (x), f1 (x), ..., fm (x). Since in general it is impossible to
optimize simultaneously more then one criteria (usually, you cannot simultaneously be as wealthy
as possible and as healthy as possible), (!) treats the criteria “asymmetrically:” we impose
bounds on all but one of the criteria and optimize the remaining criterion (optimize health, not
allowing wealth to become too low, so to speak).
Linear Optimization (called also Linear Programming) is part of Optimization Theory han-
dling Linear Optimization problems, those where the objective f (x) and the constraints fi (x)
are linear functions of x:
∑
n ∑
n
f (x) = cT x = cj xj , fi (x) = aTi x = aij xj .
j=1 j=1

LO is the simplest and the most frequently used in applications part of Mathematical Program-
ming. Some of the reasons are:
• reasonable “expressive power” of LO — while the world we live in is mainly nonlinear,
linear dependencies in many situations can be considered as quite satisfactory approxima-
tions of actual nonlinear dependencies. At the same time, a linear dependence is easy to
specify, which makes it realistic to specify data of Linear Optimization models with many
variables and constraints;

• existence of extremely elegant, rich and essentially complete mathematical theory of LO;

• last, but by far not least, existence of extremely powerful solution algorithms capable to
solve to optimality in reasonable time LO problems with tens and hundreds of thousands
of variables and constraints.
viii CONTENTS

In our course, we will focus primarily on “LO machinery” (LO Theory and Algorithms), leaving
beyond our scope practical applications of LO which are by far too numerous and diverse to be
even outlined in a single course. The brief outline of the contents is as follows:

1. LO Modeling, including instructive examples of LO models and “calculus” of LO models

– collection of tools allowing to recognize the possibility to pose an optimization problem
as an LO program;

2. LO Theory – geometry of LO programs, existence and characterization of optimal solu-

tions, theory of systems of linear inequalities and duality;

3. LO Algorithms, including Simplex-type and Interior Point ones, and the associated com-
plexity issues.

I apologize in advance for possible typos and will be extremely grateful to those bringing the
typos, or whatever other shortcomings of the Notes, to my attention.
Arkadi Nemirovski,
December 1, 2011.
CONTENTS 1

Main Notational Conventions

Vectors and matrices. By default, all vectors in this course are column vectors. Usually,
we utilize “MATLAB notation:” a vector with coordinates x1 , ..., xn is written down as x =
[x1 ; ...; xn ]. More general, if A1 , ..., Am are matrices with the same number of columns, we write
[A1 ; ...; Am ] to denote the matrix which is obtained when writing A2 below A1 , A3 below A2 ,
and so on. If A1 , ..., Am are matrices with the same number of rows, then [A1 , ..., Am ] stands for
the matrix which is obtained when writing A2 to the right of A1 , A3 to the right of A2 , and so
on. For example,
[ ]
1 2 5 6
[1, 2, 3, 4] = [1; 2; 3; 4]T , [[1, 2; 3, 4], [5, 6; 7, 8]] = [1, 2, 5, 6; 3, 4, 7, 8] = .
3 4 7 8

Diag{A1 , A2 , ..., Am } denotes the block-diagonal matrix with the diagonal blocks A1 , ..., Am . For
example,  
1 2
 3 
Diag{[1, 2], 3, [4, 5; 6, 7]} = 


4 5 
6 7
where blank spaces are filled with zeros.
Sometimes we write Colj [A] to refer to j-th column of a matrix A, and write Rowi [A] to
refer to the transpose of i-th row of a matrix A. Thus, so that both Col and Row are column
vectors, and Rowi [A] = Coli [AT ].
For a square nonsingular matrix A, A−T means [A−1 ]T .
The zero vectors and matrices are, as always, denoted by 0; if we have reasons to point
out the sizes of a zero vector/matrix, we write something like 03×4 . The unit m × m matrix is
denoted by I or Im .
We write A ≽ B (or, which is the same, B ≼ A) to express the fact that A, B are symmetric
matrices of the same size such that A − B is positive semidefinite; A ≻ B (or, which is the same,
B ≺ A) means that A, B are symmetric matrices of the same size such that A − B is positive
definite.
2 CONTENTS
Lecture 1

Introduction to LO: Examples of LO

Models

In this lecture, we define the main entity we are interested in our course – a Linear Optimiza-
tion problem, provide a number of instructive examples and address the question of when an
optimization problem can be posed as an LO one.

1.1 LO Program: Definition and Examples

1.1.1 Definitions and Terminology
An LO program
A Linear Optimization problem, or program (called also Linear Programming problem/program,
abbreviations LO and LP) is the problem of optimizing a linear function cT x of an n-dimensional
vector x under finitely many linear equality and nonstrict inequality constraints. For example,
the following Mathematical Programming problems
  
  x1 + x2 ≤ 20 
min x1 : x1 − x2 = 5 (1.1.1)
x   
x 1 , x2 ≥ 0
and   

 
 2x1 ≥ 20 − x2 

  
x1 − x2 = 5
max x1 + x2 : (1.1.2)
x 
 
 x1 ≥ 0 

  
x2 ≤ 0
are LO programs. In contrast to this, the optimization problems
  
  x1 + x2 ≤ 20 
min exp{x1 } : x1 − x2 = 5 (1.1.3)
x   
x 1 , x2 ≥ 0
and   

 
 ix1 ≥ 20 − x2 , i = 2, 3, ... 

  
x1 − x2 = 5
max x1 + x2 : (1.1.4)
x 
 
 x1 ≥ 0 

  
x2 ≤ 0

3
4 LECTURE 1. INTRODUCTION TO LO: EXAMPLES OF LO MODELS

are not LO programs: (1.1.3) has a nonlinear objective, and (1.1.4) has infinitely many con-
straints.
A careful reader could say that (1.1.2) is “the same” as (1.1.1) (since the exponent
is monotone, it is the same what to minimize, x1 or exp{x1 }). Similarly, (1.1.4) is
“the same” as (1.1.2), since for x1 ≥ 0, the infinitely many constraints ix1 + x2 ≥ 20,
i = 2, 3, ... are equivalent to the single constraint 2x1 + x2 ≥ 20. Note, however,
that we classify optimization problems according to how they are presented, and not
according to what they can be equivalent/reduced to.
Now, we can somehow “standardize” the format in which an LO program is presented. Specifi-
cally,
• every linear equality/inequality
∑ can be equivalently rewritten in the form where the left
hand side is a weighted sum nj=1 aj xj of variables xj with coefficients, and the right hand
side is a real constant; e.g., 2x1 ≥ 20 − x2 is equivalent to 2x1 + x2 ≥ 20;

• the
∑ sign of a nonstrict linear∑inequality always can be made ”≤”, since the inequality
j aj xj ≥ b is equivalent to j [−aj ]xj ≤ [−b];
∑
• a linear equality constraint
∑ j aj x
∑ j = b can be represented equivalently by the pair of
opposite inequalities j aj xj ≤ b, j [−aj ]xj ≤ [−b];
∑
• ∑
to minimize a linear function j cj xj is exactly the same to maximize the linear function
j [−cj ]xj .

Canonical form of an LO program. In view of the above observations, every LO program

can be equivalently written down as a problem of maximizing a linear objective under finitely
many nonstrict linear inequality constraints of the ” ≤ ”-type, i.e., in the canonical form
{∑ ∑ }
n
Opt = max j=1 cj xj : nj=1 aij xj ≤ bi , 1 ≤ i ≤ m [“term-wise” notation]
x { }
⇔ Opt = max c x : ai x ≤ bi , 1 ≤ i ≤ m
T T [“constraint-wise” notation]
x { }
⇔ Opt = max c x : Ax ≤ b
T [“matrix-vector” notation]
x
(1.1.5)
where c = [c1 ; ...; cn ], ai = [ai1 ; ...; ain ], A = [aT1 ; aT2 ; ...; aTm ], b = [b1 ; ...; bm ].

Standard form of an LO program. An LO program in the standard form reads

{ ∑n }
∑n = bi , 1 ≤ i ≤ m
j=1 aij xj
Opt = max cj xj : [“term-wise” notation]
x
{
j=1 xj ≥ 0, j = 1, ...,
}n
a Tx = b , 1 ≤ i ≤ m
i
⇔ Opt = max cT x : i [“constraint-wise” notation]
x
{ x j ≥ 0, 1 ≤ j} ≤ n
⇔ Opt = max cT x : Ax = b, x ≥ 0 [“matrix-vector” notation]
x
(1.1.6)
where c = [c1 ; ...; cn ], ai = [ai1 ; ...; ain ], A = [aT1 ; aT2 ; ...; aTm ],
b = [b1 ; ...; bm ]. As compared with
(1.1.5), in the standard form of an LO all “general” linear constraints are equalities, and the
inequality constraints are sign constraints, specifically, the restrictions of nonnegativity, imposed
on all variables. The standard form is as “universal” as the canonical one:
1.1. LO PROGRAM: DEFINITION AND EXAMPLES 5

Observation 1.1.1 Every LO program can be straightforwardly converted into an equivalent

program in the standard form.

Proof. We lose nothing by assuming that the original form of the program is the canonical one,
specifically, { }
max eT y : P y ≤ p (!)
y

(note change in the notation). Now, to say that P y ≤ p is exactly the same as to say that
P y + u = p for certain nonnegative vector u; in addition, every real vector y can be represented
as the difference of two nonnegative vectors: y = v − w. It follows that (!) is equivalent to the
problem { }
max cT x := dT [v − w] : Ax := B(v − w) + u = b := p, x ≥ 0
x=[u;v;w]

which is an LO in the standard form. 2

In the sequel, when investigating the “geometry” of LO, it will be more convenient to use
the canonical form of an LO program; the standard form is preferable when presenting LO
algorithms.

1.1.2 LO Terminology
We are about to present the most basic “vocabulary” of LO. For the sake of definiteness, in our
presentation we refer to the canonical format of an LO program (1.1.5), leaving the “translation”
to the case of a program in the standard form to the reader. The vocabulary is as follows:

• The variable vector x in (1.1.5) is called the decision vector of the program; its entries xj
are called decision variables. The linear function cT x is called the objective function (or
just objective) of the problem, and the inequalities aTi x ≤ bi are called the constraints.
Sometimes, with slight abuse of wording, we refer to the vector c itself as to the objective;

• The structure of (1.1.5), given the way we are writing the problem down, reduces to the
sizes m (number of constraints) and n (number of variables). The data of an LO program is
the collection of numerical values of the coefficients in the cost vector (or simply objective)
c, in the right hand side vector b and in the constraint matrix A;

• A solution to (1.1.5) is an arbitrary value of the decision vector. A solution x is called

feasible if it satisfies the constraints: Ax ≤ b. The set of all feasible solutions is called the
feasible set of the program; the program is called feasible, if the feasible set is nonempty,
and is called infeasible otherwise;

• Given a program (1.1.5), there are three possibilities:

– the program is infeasible. In this case, its optimal value Opt, by definition, is −∞
(this convention is logical, since in the case in question one cannot point out a feasible
solution with the value of the objective > −∞);
– the program is feasible, and the objective is not bounded from above on the feasible
set, meaning that for every real a one can point out a feasible solution x such that
cT x > a. In this case, the program is called unbounded, and its optimal value Opt
is, by definition, +∞.