Linear Programming

Linear Programming
Linear programming Simplex method LP duality

What is it?
n

Quintessential tool for optimal allocation of scarce resources, among a number of competing activities. (e.g., see ORF 307) Powerful and general problem-solving method. – shortest path, max flow, min cost flow, generalized flow, multicommodity flow, MST, matching, 2-person zero sum games

n

Why significant?
n

Fast commercial solvers: CPLEX, OSL. Powerful modeling languages: AMPL, GAMS. Ranked among most important scientific advances of 20th century. Also a general tool for attacking NP-hard optimization problems. Dominates world of industry. – ex: Delta claims saving $100 million per year using LP

n

n

n

n

Reference: The Allocation of Resources by Linear Programming, Scientific American, by Bob Bland.

Princeton University • COS 226 • Algorithms and Data Structures • Spring 2004 • Kevin Wayne •

http://www.Princeton.EDU/~cos226

2

Applications
Agriculture. Diet problem. Computer science. Compiler register allocation, data mining. Electrical engineering. VLSI design, optimal clocking. Energy. Blending petroleum products. Economics. Equilibrium theory, two-person zero-sum games. Environment. Water quality management. Finance. Portfolio optimization. Logistics. Supply-chain management, Berlin airlift. Management. Hotel yield management. Marketing. Direct mail advertising. Manufacturing. Production line balancing, cutting stock. Medicine. Radioactive seed placement in cancer treatment. Operations research. Airline crew assignment, vehicle routing. Physics. Ground states of 3-D Ising spin glasses. Plasma physics. Optimal stellarator design. Telecommunication. Network design, Internet routing. Sports. Scheduling ACC basketball, handicapping horse races.

Brewery Problem: A Toy LP Example
Small brewery produces ale and beer.
n

Production limited by scarce resources: corn, hops, barley malt. Recipes for ale and beer require different proportions of resources.
Beverage Ale Beer Quantity Corn (pounds) 5 15 480 Hops (ounces) 4 4 160 Malt (pounds) 35 20 1190 Profit ($) 13 23

n

How can brewer maximize profits?
n

Devote all resources to ale: 34 barrels of ale 7.5 barrels of ale, 29.5 barrels of beer 12 barrels of ale, 28 barrels of beer

Þ $442. Þ $776. Þ $800.
4

n

Devote all resources to beer: 32 barrels of beer Þ $736.

n

n

3

Brewery Problem

Brewery Problem: Feasible Region

Hops 4A + 4B £ 160 Ale s. t. Beer Profit Corn Hops Malt (0, 32) 160 0

Malt 35A + 20B £ 1190

max 13A + 23B 5A + 15B £ 480 4A + 4B £ 35A + 20B £ 1190

A

,

B ³

(12, 28) Corn 5A + 15B £ 480

(26, 14)

Beer
(0, 0)
5

Ale

(34, 0)
6

Brewery Problem: Objective Function

Brewery Problem: Geometry
Brewery problem observation. Regardless of objective function coefficients, an optimal solution occurs at an extreme point.

Extreme points

(0, 32) (12, 28)

Pr of it

(0, 32) (12, 28) 13A + 23B = $1600

(26, 14)

(26, 14) 13A + 23B = $800

Beer
(0, 0)

Beer
(0, 0)
7

Ale

(34, 0)

13A + 23B = $442

Ale

(34, 0)
8

Standard Form LP
"Standard form" LP.
n

Brewery Problem: Converting to Standard Form
Original input.
max 13A + 23B s. t. 5A + 15B £ 480 4A + 4B £ 160 0 35A + 20B £ 1190

Input: real numbers cj, bi, aij . Output: real numbers xj. n = # nonnegative variables, m = # constraints. Maximize linear objective function subject to linear inequalities.
n

n

n

n

A

,

B ³

(P) max s. t.

j =1

å aij x j

j =1 n

åcj x j
= bi 1 £ i £ m

(P) max c T x s. t.

Ax = b x ³ 0

Standard form.
n

Add slack variable for each inequality. Now a 5-dimensional problem.
max 13A + 23B

n

xj ³ 0 1 £ j £ n

Linear. No x2, xy, arccos(x), etc. Programming. Planning (term predates computer programming).

s. t.

4A +

5A + 15B + SC 4B

= 480 + SH , SH = , SM ³ 160 0
10

35A + 20B

A
9

,

B

, SC

+ SM = 1190

Geometry
Geometry.
n

Geometry
Extreme point property. If there exists an optimal solution to (P), then there exists one that is an extreme point.
n

Inequalities : halfplanes (2D), hyperplanes. Bounded feasible region: convex polygon (2D), (convex) polytope.

n

Only need to consider finitely many possible solutions.

Challenge. Number of extreme points can be exponential!
n

Convex: if a and b are feasible solutions, then so is (a +b) / 2. Extreme point: feasible solution x that can't be written as (a + b) / 2 for any two distinct feasible solutions a and b.
extreme point a a b b

Consider n-dimensional hypercube.

Greed. Local optima are global optima.
n

Extreme point is optimal if no neighboring extreme point is better.

Convex

Not convex

11

12

Simplex Algorithm
Simplex algorithm. (George Dantzig, 1947)
n

Simplex Algorithm: Basis
Basis. Subset of m of the n variables. Basic feasible solution (BFS). Set n - m nonbasic variables to 0, solve for remaining m variables.
n

Developed shortly after WWII in response to logistical problems. Used for 1948 Berlin airlift.
never decrease objective function

n

Generic algorithm.
n

Solve m equations in m unknowns. If unique and feasible solution Þ BFS. BFS corresponds to extreme point! Simplex only considers BFS.
{B, SH, SM } (0, 32) Basis {A, B, SM } (12, 28)

Start at some extreme point. Pivot from one extreme point to a neighboring one. Repeat until optimal.

n

n

n

n

n

How to implement? Linear algebra.
max 13A + 23B s. t. 4A + 5A + 15B + SC 4B = 480 + SH , SH = , SM ³ 160 0 + SM = 1190

Infeasible {A, B, SH } (19.41, 25.53) {A, B, SC } (26, 14)

Beer

35A + 20B

A

,

B

, SC

13

{SH, SM, SC } (0, 0)

Ale

{A, SH, SC } (34, 0)
14

Simplex Algorithm: Pivot 1

Simplex Algorithm: Pivot 1

max Z subject to 13A + 23B 5A 4A + 15B + SC 4B + SH , SH -

Z =
=

0 160 0

= 480 + SM , SM = 1190 ³

35A + 20B

A

,

B

, SC

Basis = {SC, SH, SM} A=B=0 Z=0 SC = 480 SH = 160 SM = 1190

max Z subject to 13A + 23B 5A 4A + 15B + SC 4B + SH , SH -

Z =
=

0 160 0

= 480 + SM , SM = 1190 ³

35A + 20B

A

,

B

, SC

Basis = {SC, SH, SM} A=B=0 Z=0 SC = 480 SH = 160 SM = 1190

Substitute: B = 1/15 (480 – 5A – SC)
max Z subject to
16 3 1 3 8 3 85 3

Why pivot on column 2?
n

Each unit increase in B increases objective value by $23. Pivoting on column 1 also OK.

A A A A
, B

,

A + B +

23 15 1 15 4 15 4 3

SC SC SC SC +
,

-

Z = - 736
= 32 32 550 0 =

SH SH

SC

+ SM , SM

= ³

Basis = {B, SH, SM} A = SC = 0 Z = 736 B = 32 SH = 32 SM = 550

n

Why pivot on row 2?
n

Preserves feasibility by ensuring RHS ³ 0. Minimum ratio rule: min { 480/15, 160/4, 1190/20 }.
17

n

16

Simplex Algorithm: Pivot 2
max Z subject to
16 3 1 3 8 3 85 3

Simplex Algorithm: Optimality
When to stop pivoting?

A A A A
, B

,

A + B +

23 15 1 15 4 15 4 3

SC SC SC SC +
,

-

Z = - 736
= 32 32 550 0 =

SH SH

SC

+ SM , SM

= ³

Basis = {B, SH, SM} A = SC = 0 Z = 736 B = 32 SH = 32 SM = 550

n

If all coefficients in top row are non-positive.

Why is resulting solution optimal?
n

Any feasible solution satisfies system of equations in tableaux. – in particular: Z = 800 – SC – 2 SH Thus, optimal objective value Z* £ 800 since SC, SH ³ 0. Current BFS has value 800 Þ optimal.

n

n

Substitute: A = 3/8 (32 + 4/15 SC – SH)
max Z subject to -

B + A A , B
,

1 10 1 10 25 6

SC

SC

SC

SC

SC

- 2 SH + + ,
1 8 3 8 85 8

-

Z = - 800
= = = ³ 28 12 110 0

SH

SH + SM
, SM

SH

SH

Basis = {A, B, SM} SC = SH = 0 Z = 800 B = 28 A = 12 SM = 110

max Z subject to -

B + A A , B
18

,

1 10 1 10 25 6

SC

SC

SC

SC

SC

- 2 SH + + ,
1 8 3 8 85 8

-

Z = - 800
= = = ³ 28 12 110 0

SH

SH + SM
, SM

SH

SH

Basis = {A, B, SM} SC = SH = 0 Z = 800 B = 28 A = 12 SM = 110

19

Simplex Algorithm: Issues
Remarkable property. In practice, simplex algorithm typically terminates in at most 2(m+n) pivots.
n

LP Duality: Economic Interpretation
Brewer's problem: find optimal mix of beer and ale to maximize profits.
(P) max 13A + 23B s. t. 5A + 15B £ 480 4A + 4B £ 160 0 35A + 20B £ 1190

No polynomial pivot rule known. Most pivot rules known to be exponential in worst-case.

n

A* = 12 B* = 28 OPT = 800

Issues. Which neighboring extreme point? Degeneracy. New basis, same extreme point.
n

A

,

B ³

"Stalling" is common in practice.

Entrepreneur's problem: Buy individual resources from brewer at minimum cost.
n

Cycling. Get stuck by cycling through different bases that all correspond to same extreme point.
n

C, H, M = unit price for corn, hops, malt. Brewer won't agree to sell resources if 5C + 4H + 35M < 13.
(D) min 480C s. t. 5C 15C + 160H + 1190M + + , 4H + 4H + 35M ³ 13 20M ³ 23

n

Doesn't occur in the wild. Bland's least index rule Þ finite # of pivots.

n

C
20

H

,

M ³

0

C* = 1 H* = 2 M* = 0 OPT = 800
21

LP Duality
Primal and dual LPs. Given real numbers aij, bi, cj, find real numbers xi, yj that optimize (P) and (D).
n

LP Duality: Economic Interpretation
Sensitivity analysis.
n

Q. How much should brewer be willing to pay (marginal price) for additional supplies of scarce resources? A. corn $1, hops $2, malt $0. Q. New product "light beer" is proposed. It requires 2 corn, 5 hops, 24 malt. How much profit must be obtained from light beer to justify diverting resources from production of beer and ale? A. Breakeven: 2 ($1) + 5 ($2) + 24 (0$) = $12 / barrel.

(P) max s. t.

j =1

å aij x j £ bi 1 £ i £ m
xj ³ 0 1 £ j £ n

j =1 n

åcj x j

(D) min s. t.

i =1

å aij yi ³ c j 1 £ j £ n
yi ³ 0 1 £ i £ m

i =1 m

å bi yi
n

m

n

n

Duality Theorem (Gale-Kuhn-Tucker 1951, Dantzig-von Neumann 1947). If (P) and (D) have feasible solutions, then max = min.
n

Special case: max-flow min-cut theorem. Sensitivity analysis.

How do I compute marginal prices (dual variables)?
n

Simplex solves primal and dual simultaneously. Top row of final simplex tableaux provides optimal dual solution!

n

n

22

23

History
1939. Production, planning. (Kantorovich, USSR)
n

History
1939. Production, planning. (Kantorovich) 1947. Simplex algorithm. (Dantzig)

Propaganda to make paper more palatable to communist censors.
"I want to emphasize again that the greater part of the problems of which I shall speak, relating to the organization and planning of production, are connected specifically with the Soviet system of economy and in the majority of cases do not arise in the economy of a capitalist society." "the majority of enterprises work at half capacity. There the choice of output is determined not by the plan but by the interests and profits of individual capitalists."

USSR

USA

n

Kantorovich awarded 1975 Nobel prize in Economics for contributions to the theory of optimum allocation of resources. Staple in MBA curriculum. Used by most large companies and other profit maximizers.
25 26

n

n

History
1939. Production, planning. (Kantorovich) 1947. Simplex algorithm. (Dantzig) 1950. Applications in many fields.
n

History
1939. 1947. 1950. 1979.
n n

Military logistics. Operations research. Control theory. Filter design. Structural optimization.

Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian)

n

Geometric divide-and-conquer. Solvable in polynomial time: O(n4 L) bit operations. – n = # variables – L = # bits in input Theoretical tour de force, not remotely practical.

n

n

n

n

27

28

History
1939. 1947. 1950. 1979. 1984.
n n

History
1939. 1947. 1950. 1979. 1984. 1990.
n n

Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar)

O(n3.5 L). Efficient implementations possible.

Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar) Interior point methods.

O(n3 L) and practical. Extends to even more general problems.

29

30

History
1939. 1947. 1950. 1979. 1984. 1990.
n n

History
1939. 1947. 1950. 1979. 1984. 1990. Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar) Interior point methods.

Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar) Interior point methods.

Interior point faster when polyhedron smooth like disco ball. Simplex faster when polyhedron spiky like quartz crystal.

Current research.
n

Approximation algorithms. Software for large scale optimization. Interior point variants.

n

n

31

32

Ultimate Problem Solving Model
Ultimate problem-solving model?
n

Perspective
LP is near the deep waters of NP-completeness.
n

Shortest path. Maximum flow. Min cost flow. Generalized multicommodity flow. Linear programming. Semidefinite programming. ... TSP (or any NP-complete problem)
intractable (conjectured) tractable

Solvable in polynomial time. Known for less than 25 years.

n

n

n

n

Integer linear programming.
n

n

LP with integrality requirement. NP-hard.

n

n

n

n

Does P = NP? No universal problem-solving model exists unless P = NP.

An unsuspecting MBA student transitions from tractable LP to intractable ILP in a single mouse click.
33 34

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.