Are you sure?
This action might not be possible to undo. Are you sure you want to continue?
Linear Programming
Linear programming Simplex method LP duality
What is it?
n
Quintessential tool for optimal allocation of scarce resources, among a number of competing activities. (e.g., see ORF 307) Powerful and general problemsolving method. – shortest path, max flow, min cost flow, generalized flow, multicommodity flow, MST, matching, 2person zero sum games
n
Why significant?
n
Fast commercial solvers: CPLEX, OSL. Powerful modeling languages: AMPL, GAMS. Ranked among most important scientific advances of 20th century. Also a general tool for attacking NPhard optimization problems. Dominates world of industry. – ex: Delta claims saving $100 million per year using LP
n
n
n
n
Reference: The Allocation of Resources by Linear Programming, Scientific American, by Bob Bland.
Princeton University • COS 226 • Algorithms and Data Structures • Spring 2004 • Kevin Wayne •
http://www.Princeton.EDU/~cos226
2
Applications
Agriculture. Diet problem. Computer science. Compiler register allocation, data mining. Electrical engineering. VLSI design, optimal clocking. Energy. Blending petroleum products. Economics. Equilibrium theory, twoperson zerosum games. Environment. Water quality management. Finance. Portfolio optimization. Logistics. Supplychain management, Berlin airlift. Management. Hotel yield management. Marketing. Direct mail advertising. Manufacturing. Production line balancing, cutting stock. Medicine. Radioactive seed placement in cancer treatment. Operations research. Airline crew assignment, vehicle routing. Physics. Ground states of 3D Ising spin glasses. Plasma physics. Optimal stellarator design. Telecommunication. Network design, Internet routing. Sports. Scheduling ACC basketball, handicapping horse races.
Brewery Problem: A Toy LP Example
Small brewery produces ale and beer.
n
Production limited by scarce resources: corn, hops, barley malt. Recipes for ale and beer require different proportions of resources.
Beverage Ale Beer Quantity Corn (pounds) 5 15 480 Hops (ounces) 4 4 160 Malt (pounds) 35 20 1190 Profit ($) 13 23
n
How can brewer maximize profits?
n
Devote all resources to ale: 34 barrels of ale 7.5 barrels of ale, 29.5 barrels of beer 12 barrels of ale, 28 barrels of beer
Þ $442. Þ $776. Þ $800.
4
n
Devote all resources to beer: 32 barrels of beer Þ $736.
n
n
3
Brewery Problem
Brewery Problem: Feasible Region
Hops 4A + 4B £ 160 Ale s. t. Beer Profit Corn Hops Malt (0, 32) 160 0
Malt 35A + 20B £ 1190
max 13A + 23B 5A + 15B £ 480 4A + 4B £ 35A + 20B £ 1190
A
,
B ³
(12, 28) Corn 5A + 15B £ 480
(26, 14)
Beer
(0, 0)
5
Ale
(34, 0)
6
Brewery Problem: Objective Function
Brewery Problem: Geometry
Brewery problem observation. Regardless of objective function coefficients, an optimal solution occurs at an extreme point.
Extreme points
(0, 32) (12, 28)
Pr of it
(0, 32) (12, 28) 13A + 23B = $1600
(26, 14)
(26, 14) 13A + 23B = $800
Beer
(0, 0)
Beer
(0, 0)
7
Ale
(34, 0)
13A + 23B = $442
Ale
(34, 0)
8
Standard Form LP
"Standard form" LP.
n
Brewery Problem: Converting to Standard Form
Original input.
max 13A + 23B s. t. 5A + 15B £ 480 4A + 4B £ 160 0 35A + 20B £ 1190
Input: real numbers cj, bi, aij . Output: real numbers xj. n = # nonnegative variables, m = # constraints. Maximize linear objective function subject to linear inequalities.
n
n
n
n
A
,
B ³
(P) max s. t.
j =1
å aij x j
j =1 n
åcj x j
= bi 1 £ i £ m
(P) max c T x s. t.
Ax = b x ³ 0
Standard form.
n
Add slack variable for each inequality. Now a 5dimensional problem.
max 13A + 23B
n
xj ³ 0 1 £ j £ n
Linear. No x2, xy, arccos(x), etc. Programming. Planning (term predates computer programming).
s. t.
4A +
5A + 15B + SC 4B
= 480 + SH , SH = , SM ³ 160 0
10
35A + 20B
A
9
,
B
, SC
+ SM = 1190
Geometry
Geometry.
n
Geometry
Extreme point property. If there exists an optimal solution to (P), then there exists one that is an extreme point.
n
Inequalities : halfplanes (2D), hyperplanes. Bounded feasible region: convex polygon (2D), (convex) polytope.
n
Only need to consider finitely many possible solutions.
Challenge. Number of extreme points can be exponential!
n
Convex: if a and b are feasible solutions, then so is (a +b) / 2. Extreme point: feasible solution x that can't be written as (a + b) / 2 for any two distinct feasible solutions a and b.
extreme point a a b b
Consider ndimensional hypercube.
Greed. Local optima are global optima.
n
Extreme point is optimal if no neighboring extreme point is better.
Convex
Not convex
11
12
Simplex Algorithm
Simplex algorithm. (George Dantzig, 1947)
n
Simplex Algorithm: Basis
Basis. Subset of m of the n variables. Basic feasible solution (BFS). Set n  m nonbasic variables to 0, solve for remaining m variables.
n
Developed shortly after WWII in response to logistical problems. Used for 1948 Berlin airlift.
never decrease objective function
n
Generic algorithm.
n
Solve m equations in m unknowns. If unique and feasible solution Þ BFS. BFS corresponds to extreme point! Simplex only considers BFS.
{B, SH, SM } (0, 32) Basis {A, B, SM } (12, 28)
Start at some extreme point. Pivot from one extreme point to a neighboring one. Repeat until optimal.
n
n
n
n
n
How to implement? Linear algebra.
max 13A + 23B s. t. 4A + 5A + 15B + SC 4B = 480 + SH , SH = , SM ³ 160 0 + SM = 1190
Infeasible {A, B, SH } (19.41, 25.53) {A, B, SC } (26, 14)
Beer
35A + 20B
A
,
B
, SC
13
{SH, SM, SC } (0, 0)
Ale
{A, SH, SC } (34, 0)
14
Simplex Algorithm: Pivot 1
Simplex Algorithm: Pivot 1
max Z subject to 13A + 23B 5A 4A + 15B + SC 4B + SH , SH 
Z =
=
0 160 0
= 480 + SM , SM = 1190 ³
35A + 20B
A
,
B
, SC
Basis = {SC, SH, SM} A=B=0 Z=0 SC = 480 SH = 160 SM = 1190
max Z subject to 13A + 23B 5A 4A + 15B + SC 4B + SH , SH 
Z =
=
0 160 0
= 480 + SM , SM = 1190 ³
35A + 20B
A
,
B
, SC
Basis = {SC, SH, SM} A=B=0 Z=0 SC = 480 SH = 160 SM = 1190
Substitute: B = 1/15 (480 – 5A – SC)
max Z subject to
16 3 1 3 8 3 85 3
Why pivot on column 2?
n
Each unit increase in B increases objective value by $23. Pivoting on column 1 also OK.
A A A A
, B
,
A + B +
23 15 1 15 4 15 4 3
SC SC SC SC +
,

Z =  736
= 32 32 550 0 =
SH SH
SC
+ SM , SM
= ³
Basis = {B, SH, SM} A = SC = 0 Z = 736 B = 32 SH = 32 SM = 550
n
Why pivot on row 2?
n
Preserves feasibility by ensuring RHS ³ 0. Minimum ratio rule: min { 480/15, 160/4, 1190/20 }.
17
n
16
Simplex Algorithm: Pivot 2
max Z subject to
16 3 1 3 8 3 85 3
Simplex Algorithm: Optimality
When to stop pivoting?
A A A A
, B
,
A + B +
23 15 1 15 4 15 4 3
SC SC SC SC +
,

Z =  736
= 32 32 550 0 =
SH SH
SC
+ SM , SM
= ³
Basis = {B, SH, SM} A = SC = 0 Z = 736 B = 32 SH = 32 SM = 550
n
If all coefficients in top row are nonpositive.
Why is resulting solution optimal?
n
Any feasible solution satisfies system of equations in tableaux. – in particular: Z = 800 – SC – 2 SH Thus, optimal objective value Z* £ 800 since SC, SH ³ 0. Current BFS has value 800 Þ optimal.
n
n
Substitute: A = 3/8 (32 + 4/15 SC – SH)
max Z subject to 
B + A A , B
,
1 10 1 10 25 6
SC
SC
SC
SC
SC
 2 SH + + ,
1 8 3 8 85 8

Z =  800
= = = ³ 28 12 110 0
SH
SH + SM
, SM
SH
SH
Basis = {A, B, SM} SC = SH = 0 Z = 800 B = 28 A = 12 SM = 110
max Z subject to 
B + A A , B
18
,
1 10 1 10 25 6
SC
SC
SC
SC
SC
 2 SH + + ,
1 8 3 8 85 8

Z =  800
= = = ³ 28 12 110 0
SH
SH + SM
, SM
SH
SH
Basis = {A, B, SM} SC = SH = 0 Z = 800 B = 28 A = 12 SM = 110
19
Simplex Algorithm: Issues
Remarkable property. In practice, simplex algorithm typically terminates in at most 2(m+n) pivots.
n
LP Duality: Economic Interpretation
Brewer's problem: find optimal mix of beer and ale to maximize profits.
(P) max 13A + 23B s. t. 5A + 15B £ 480 4A + 4B £ 160 0 35A + 20B £ 1190
No polynomial pivot rule known. Most pivot rules known to be exponential in worstcase.
n
A* = 12 B* = 28 OPT = 800
Issues. Which neighboring extreme point? Degeneracy. New basis, same extreme point.
n
A
,
B ³
"Stalling" is common in practice.
Entrepreneur's problem: Buy individual resources from brewer at minimum cost.
n
Cycling. Get stuck by cycling through different bases that all correspond to same extreme point.
n
C, H, M = unit price for corn, hops, malt. Brewer won't agree to sell resources if 5C + 4H + 35M < 13.
(D) min 480C s. t. 5C 15C + 160H + 1190M + + , 4H + 4H + 35M ³ 13 20M ³ 23
n
Doesn't occur in the wild. Bland's least index rule Þ finite # of pivots.
n
C
20
H
,
M ³
0
C* = 1 H* = 2 M* = 0 OPT = 800
21
LP Duality
Primal and dual LPs. Given real numbers aij, bi, cj, find real numbers xi, yj that optimize (P) and (D).
n
LP Duality: Economic Interpretation
Sensitivity analysis.
n
Q. How much should brewer be willing to pay (marginal price) for additional supplies of scarce resources? A. corn $1, hops $2, malt $0. Q. New product "light beer" is proposed. It requires 2 corn, 5 hops, 24 malt. How much profit must be obtained from light beer to justify diverting resources from production of beer and ale? A. Breakeven: 2 ($1) + 5 ($2) + 24 (0$) = $12 / barrel.
(P) max s. t.
j =1
å aij x j £ bi 1 £ i £ m
xj ³ 0 1 £ j £ n
j =1 n
åcj x j
(D) min s. t.
i =1
å aij yi ³ c j 1 £ j £ n
yi ³ 0 1 £ i £ m
i =1 m
å bi yi
n
m
n
n
Duality Theorem (GaleKuhnTucker 1951, Dantzigvon Neumann 1947). If (P) and (D) have feasible solutions, then max = min.
n
Special case: maxflow mincut theorem. Sensitivity analysis.
How do I compute marginal prices (dual variables)?
n
Simplex solves primal and dual simultaneously. Top row of final simplex tableaux provides optimal dual solution!
n
n
22
23
History
1939. Production, planning. (Kantorovich, USSR)
n
History
1939. Production, planning. (Kantorovich) 1947. Simplex algorithm. (Dantzig)
Propaganda to make paper more palatable to communist censors.
"I want to emphasize again that the greater part of the problems of which I shall speak, relating to the organization and planning of production, are connected specifically with the Soviet system of economy and in the majority of cases do not arise in the economy of a capitalist society." "the majority of enterprises work at half capacity. There the choice of output is determined not by the plan but by the interests and profits of individual capitalists."
USSR
USA
n
Kantorovich awarded 1975 Nobel prize in Economics for contributions to the theory of optimum allocation of resources. Staple in MBA curriculum. Used by most large companies and other profit maximizers.
25 26
n
n
History
1939. Production, planning. (Kantorovich) 1947. Simplex algorithm. (Dantzig) 1950. Applications in many fields.
n
History
1939. 1947. 1950. 1979.
n n
Military logistics. Operations research. Control theory. Filter design. Structural optimization.
Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian)
n
Geometric divideandconquer. Solvable in polynomial time: O(n4 L) bit operations. – n = # variables – L = # bits in input Theoretical tour de force, not remotely practical.
n
n
n
n
27
28
History
1939. 1947. 1950. 1979. 1984.
n n
History
1939. 1947. 1950. 1979. 1984. 1990.
n n
Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar)
O(n3.5 L). Efficient implementations possible.
Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar) Interior point methods.
O(n3 L) and practical. Extends to even more general problems.
29
30
History
1939. 1947. 1950. 1979. 1984. 1990.
n n
History
1939. 1947. 1950. 1979. 1984. 1990. Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar) Interior point methods.
Production, planning. (Kantorovich) Simplex algorithm. (Dantzig) Applications in many fields. Ellipsoid algorithm. (Khachian) Projective scaling algorithm. (Karmarkar) Interior point methods.
Interior point faster when polyhedron smooth like disco ball. Simplex faster when polyhedron spiky like quartz crystal.
Current research.
n
Approximation algorithms. Software for large scale optimization. Interior point variants.
n
n
31
32
Ultimate Problem Solving Model
Ultimate problemsolving model?
n
Perspective
LP is near the deep waters of NPcompleteness.
n
Shortest path. Maximum flow. Min cost flow. Generalized multicommodity flow. Linear programming. Semidefinite programming. ... TSP (or any NPcomplete problem)
intractable (conjectured) tractable
Solvable in polynomial time. Known for less than 25 years.
n
n
n
n
Integer linear programming.
n
n
LP with integrality requirement. NPhard.
n
n
n
n
Does P = NP? No universal problemsolving model exists unless P = NP.
An unsuspecting MBA student transitions from tractable LP to intractable ILP in a single mouse click.
33 34
This action might not be possible to undo. Are you sure you want to continue?
Use one of your book credits to continue reading from where you left off, or restart the preview.