Linear Programming:

Theory and Applications

Linear Programming (LP) Problem

The problem of optimizing (minimize or
maximize) a linear objective function
subject to linear equality and inequality
constraints
min
f(x)
s.t.

ci(x) = 0, i = 1, , p
ci(x) 0, i = p+1, , m

General Form LP
min cT x
s.t.
aiT x bi ,

i {1,2,..., p}

aiT x bi ,

i { p 1, p 2,..., m}

x is the vector of decision variables:

x = [x1 x2 xn]T

ais and c are vectors of dimension n

bis are real numbers

Standard Form LP
A standard form LP is in the following format:
T

min c x
s.t.
Ax b
x0
where

mn

,b , c
m

Standard Form LP
In a standard form LP, the coefficient
matrix A is assumed to have full row rank
That is, the rows of A are linearly independent
You cannot find a linear combination of rows of A
that is equal to zero vector

If A does not have full row rank, one can

easily reduce it to a full row rank matrix by
elementary row operations

Standard Form LP
Used while implementing Simplex Method to
solve an LP
A general form LP can be converted to a
standard form LP
How?
Objective function? (max -> min)
Multiply by -1

Inequalities?
Add slack or excess (surplus) variables

Variables unrestricted in sign? (xi - free)

xi xi xi
xi , xi 0

Example
max 3 x1 5 x2
s.t
x1 4
2 x1 x2 12
3 x1 2 x2 18
x1 0, x2 free

Convert this LP to a standard form LP

Example

min 3 x1 5 x 5 x

s.t
x1 s1 4
2 x1 x2 x2 12
3 x1 2 x2 2 x2 e1 18

x1 , x , x , s1 , e1 0

Some Definitions
An LP is feasible if the feasible region of the
problem is not empty, or equivalently, if there is a
vector x which satisfies all the constraints
x is called a feasible solution if it satisfies all the
constraints
x is an optimal solution if it is a feasible solution and
minimizes the objective function value among all
feasible solutions

An LP is called unbounded if there exists a

sequence of feasible solutions {xk} such that

c x
T

Some Definitions
If an LP is feasible but not unbounded, then it
has an optimal solution
In summary, there are three alternatives for an
LP
It can be feasible and has an optimal solution
It can be unbounded
It can be infeasible
Feasible region is empty

Duality
Associated with every (primal) LP, there is
another LP called dual LP
Main idea behind duality:
To find a bound on the optimal solution of the
primal problem

Dual of a dual LP is the primal LP

Duality
Consider the following LP

max 4 x1 x2 3 x3
s.t.
x1 4 x2 1
3 x1 x2 x3 3
x1 , x2 , x3 0

Duality
Any feasible solution to this LP provides a
lower bound on the optimal objective
function value
(x1,x2,x3) = (1,0,0)
Optimal objective function value 4

(x1,x2,x3) = (0,0,3)
Optimal objective function value 9
How good is this bound?
Is it close to the optimal value?

Duality
To answer these questions, we have to
find an upper bound
In the given example
Multiply the first constraint by 2 and multiply
the second constraint by 3 and sum those two
inequalities

Duality
x1 4 x2 1

(multiply by 2)

3x1 x2 x3 3

(multiply by 3)

11x1 5 x2 3x3 11

4 x1 x2 3x3 11x1 5x2 3x3 11

Optimal objective function value is not greater
than 11

Duality
What is the best upper bound that can be found
using this method?

x1 4 x2 1
3x1 x2 x3 3

(multiply by y1 )
(multiply by y2 )

( y1 3 y2 ) x1 (4 y1 y2 ) x2 ( y2 ) x3 y1 3 y2
To use this inequality for finding an upper bound,
coefficient of xis should be at least as large as
the corresponding coefficient in the objective
function

Duality
And, we want to minimize the right hand side of
the inequality to find the best upper bound

min y1 3 y 2
s.t.
y1 3 y 2 4
4 y1 y 2 1
y2 3
y1 , y 2 0

Dual LP

Dual of a Standard Form LP

T

min c x
s.t.

max b y
s.t.

Ax b
x0

A yc
y free

Primal LP

Dual LP

Primal and Dual Summary

Duality
We can interpret the optimal dual prices yi
as the marginal cost of changing the right
hand side of the ith equation
Why?

Economic Interpretation
The dual prices, or shadow prices allow us to put
a value on resources
Alternatively, they allow us to consider the
sensitivity of the optimal solution value to
changes in the input
By examining the dual variable for each
constraint, we can determine the value of an
extra unit of the corresponding resource
We can then determine the maximum amount
we would be willing to pay to have a unit of that
resource

Weak Duality Theorem

Let x be any feasible solution to the primal
LP and y be any feasible solution to the
dual LP. Then,
cTx bTy

Proof?
x 0 and c - ATy 0
Check the inner product of these two vectors

cTx bTy is called the duality gap

Consequences of Weak Duality

If the primal LP is unbounded, then the
dual LP must be infeasible
If the dual LP is unbounded, then the
primal LP must be infeasible
If x is feasible for the primal LP, y is
feasible for the dual LP, and cTx = bTy,
then x must be optimal for the primal LP
and y must be optimal for the dual LP

Strong Duality Theorem

If both the primal LP and the dual LP have
feasible solutions, then they both have
optimal solutions and for any primal
optimal solution x and dual optimal
solution y, we have cTx = bTy

Strong Duality Theorem: Optimality

Conditions
x Rn is an optimal solution for the primal
LP if and only if
x is primal feasible: Ax = b, x 0
and there exists y Rm such that y is dual
feasible: ATy c
and there is no duality gap: cTx = bTy

Complementary Slackness
x Rn is an optimal solution for the primal
LP if and only if
x is primal feasible: Ax = b, x 0
and there exists y Rm such that y is dual
feasible: s:= c - ATy 0
and we have xisi = 0 for each i {1,2,,n}
(This is equivalent to the last condition of the
optimality conditions due to strong duality)

Example
Consider the following LP

You are given the information that x2 and x3 are

positive in the optimal solution. Use the
complementary slackness conditions to find the
optimal dual solution.

Solving an LP
Simplex method
Moves on the boundary of the feasible region
from one corner (extreme) point to another
one

Interior point methods

Moves along a path in the interior of the
feasible region

Some More Definitions

A polyhedron is a set of the form

{x n Bx d }
for some matrix B, and vector d
Let a be a vector of dimension n and b is a real
number, then {x n aT x b} is called a
halfspace
A polyhedron is an intersection of halfspaces

Some More Definitions

Let P a polyhedron in Rn. A vector x P is
an extreme point if it cannot written as a
convex combination of two other points in
P
That is, there does not exist y, z P and
(0,1) such that x = y+(1- )z

Simplex Method Basics

Consider a standard form LP
T

min c x
s.t.
Ax b
x0
where A is a matrix of dimension mn, b is a
vector of dimension m, and c is a vector of
dimension n
Recall that A has full row rank, i.e., m n

Simplex Method Basics

Let S denote the feasible region of the standard
form LP

S {x Ax b, x 0}
n

Note that S is also a polyhedron

Partition the columns of matrix A into two sets of
B and N such that B is of dimension mm and
has full rank (i.e., B-1 exists)
A=[B N]

B is called a basis matrix

Simplex Methods Basics

Partition the decision vector x into two sets xB
and xN such that the variables in xB (or xN)
correspond to the columns in B (or N)
That is,

xB
x
xN

Then, we have B xB + N xN = b
Setting xN= 0, we have xB = B-1b

xB B 1b
x
is called basic solution
x N 0

Simplex Method Basics

xB is called basic variables, and xN is
called nonbasic variables
If a basic solution is feasible, i.e., B-1b 0,
then it is called a basic feasible solution

Simplex Method Basics

x S is an extreme point of S if and only if x is a
basic feasible solution
Two extreme points are adjacent if they differ in
only one basic variable
While moving from one extreme point to another
one, the new basic variable is called the entering
variable, and the new nonbasic variable is called
the leaving variable

Basic Theorem of LP
Basic Theorem of LP: Consider a
standard form LP. If the feasible
region (S) has at least one extreme
point and there exists an optimal
solution, then there exists an optimal
solution that is an extreme point

Simplex Method Basics

Simplex method starts from an extreme
point and moves to an adjacent extreme
point if the objective function value
improves
Since there are only a finite number of
extreme points, the method terminates at
an optimal solution after a finite number of
iterations or detects that the problem is
infeasible or unbounded

Simplex Method Illustration

Simplex Algorithm Basics

At an extreme point, all the nonbasic
variables are zero and the basic variables
are greater than or equal to zero.
To move from one extreme point to
another one, we increase the value of a
nonbasic variable (this is the entering
variable) and look at the changes in the
values of the basic variables to determine
the leaving variable

Simplex Algorithm Basics

Let B be a basis, then

A B N
xB
x
xN
z cT x cBT xB cTN xN
Ax Bx B Nx N b xB B 1b B 1 Nx N
z cBT ( B 1b B1 Nx N ) cTN xN
cBT B 1b (cTN cBT B 1 N ) xN

Simplex Algorithm Basics

Let B be a basis, then

A B N
xB
x
xN
z cT x cBT xB cTN xN
Ax Bx B Nx N b xB B 1b B 1 Nx N
z cBT ( B 1b B1 Nx N ) cTN xN
cBT B 1b (cTN cBT B 1 N ) xN

Simplex Algorithm Basics

Let B be a basis, then

A B N
xB
x
xN
z cT x cBT xB cTN xN
Ax Bx B Nx N b xB B 1b B 1 Nx N
z cBT ( B 1b B1 Nx N ) cTN xN
cBT B 1b (cTN cBT B 1 N ) xN

Simplex Algorithm Basics

Let B be a basis, then

A B N
xB
x
xN
z cT x cBT xB cTN xN
Ax Bx B Nx N b xB B 1b B 1 Nx N
z cBT ( B 1b B1 Nx N ) cTN xN
cBT B 1b (cTN cBT B 1 N ) xN

Simplex Method Basics

Lets assume that this basic solution is feasible,
i.e., B-1b 0, then
1

xB B b B Nx N
1

z c B b (c c B N ) x N
T
B

T
N

T
B

Simplex Method Basics

What happens when we increase the
value of a nonbasic variable, say xj?
Let aj be the column of A that corresponds
to xj
Lets look at the changes in the values of
the basic variables and the objective
function

Simplex Method Basics

1

xB B b B a x j
1

z c B b (c j c B a ) x j
T
B

T
B

T 1 j
c

c
If j
B B a 0 , then increasing the value of

xj improves the objective function value.

Otherwise, there is no point of increasing the
value of xj
T 1 j
c

c
Observation: Choose a variable with j
BB a 0
Also called the entering variable

c j cBT B 1a j is called the reduced cost

Simplex Method Basics

If the ith element of B-1aj is positive, i.e.,
[B-1aj ]i > 0, then the value of basic variable xi
decreases as we increase the value of xj
If B-1aj 0 (assuming c j cBT B 1a j 0 ), then the
LP is unbounded
WHY?

Among the basic variables that decrease as we

increase the value of the entering variable, the
one that reaches to zero first (also called leaving
variable) leaves the basis and becomes a
nonbasic variable
Determined based on the ratio test

Simplex Method Outline

Start with a basic feasible solution
Determine the entering variable (a
nonbasic variable with a negative reduced
cost)
A popular way is to choose the nonbasic
variable with the most negative reduced cost
If there is no entering variable, then the
current solution is optimal

Determine the leaving variable

If no leaving variable, problem is unbounded

Form the new basis, and go to Step 2.

Simplex Method in Tableau Format

Simplex Method: Example

Consider the following LP

min 7 x1 2 x2
s.t
x1 2 x2 4
5 x1 x2 20
2 x1 2 x2 7
x1 , x2 0

Simplex Method: Example

First, convert it to a standard form LP

min 7 x1 2 x2
s.t
x1 2 x2 x3 4
5 x1 x2 x4 20
2 x1 2 x2 x5 7
x1 , x2 , x3 , x4 , x5 0

Simplex Method: Example

The initial tableau is as follows:

Simplex Method: Example

Choose x1, x3 and x4 as the basic variables in
the initial basis B
The corresponding (basic feasible) solution is
as follows:

Simplex Method: Example

The basic feasible solution is
7
x1 2
x 0
2

x x3 15
2
x4 5
x5 2

0

Simplex Method: Example

Is this solution optimal?
No, we can improve the objective function
value by increasing the value of x5
The reduced cost of x5 is -7/2

Choose x5 as the entering variable

Perform the ratio test to determine the
leaving variable:
There is only one candidate (x4 leaves the
basis)

Simplex Method: Example

The new basic variables are x1, x3 and x5
And the new tableau is

Simplex Method: Example

Is this solution optimal?
No, we can improve the objective function value
by increasing the value of x2
The reduced cost of x2 is -5/3

Choose x2 as the entering variable

Perform the ratio test to determine the leaving
variable:

8
4
40
min{
,
}
x3 leaves the basis
11 / 5 1 / 5 11

Simplex Method: Example

The new basic variables are x1, x2 and x5
And the new tableau is

Simplex Method: Example

Is this solution optimal?
Yes

The optimal solution is as follows:

x1=36/11, x2=40/11, x3=0, x4=0, x5=75/11

Optimal objective function value is -332/11

Note that the objective function value improved
at each iteration

Graphical Illustration

Optimization Software

Excel Solver
CPLEX
GAMS
Matlab

Example using Excel Solver

Consider the following standard form LP

min 7 x1 2 x2
s.t
x1 2 x2 x3 4
5 x1 x2 x4 20
2 x1 2 x2 x5 7
x1 , x2 , x3 , x4 , x5 0
Solve it using Excel Solver

Another Example
A bond portfolio manager has $100,000 to allocate to two
different bonds; one corporate and one government bond.
The corporate bond has a yield of 4%, a maturity of 3 years
and an A rating from Moody's that is translated into a
numerical rating of 2 for computational purposes. In contrast,
the government bond has a yield of 3%, a maturity of 4 years
and rating of Aaa with the corresponding numerical rating of
1 (lower numerical ratings correspond to higher quality
bonds). The portfolio manager would like to allocate her
funds so that the average rating for the portfolio is no worse
than Aa (numerical equivalent 1.5) and average maturity of
the portfolio is at most 3.6 years. Any amount not invested in
the two bonds will be kept in a cash account that is assumed
to earn no interest for simplicity and does not contribute to
the average rating or maturity computations. How should the
manager allocate her funds between these two bonds to
achieve her objective of maximizing the yield from this
investment?

Another Example: Formulation

Let x1 and x2 denote the amount of funds (in
thousands of dollars) allocated to the corporate
and government bonds, respectively
max 4 x1 3 x2
s.t
x1 x2 100
2 x1 x2
1.5
x1 x2
3 x1 4 x2
3 .6
x1 x2
x1 , x2 0

Linear?

Another Example: Formulation

max 4 x1 3 x2
s.t
x1 x2 100
0.5 x1 0.5 x2 0
0.4 x2 0.6 x1 0
x1 , x2 0

Another Example: Solution

Exercise: Solve the model in the previous
slide using the simplex method
Solution found by Excel Solver
x1 = 50, x2 = 50, z = 350