Lecture 3.a - Exact Methods

University of Zagreb, Croatia
Lectures 3.a
Heuristic Optimization Methods:
Overview of exact methods
Slides prepared by Nina Skorin-Kapov
Academic year 2020/2021

Graduate Studies Programme
Outline
 Concept behind LP solvers (simplex)
 Why is IP harder than LP?
 Branch and bound
Zavod za telekomunikacije 2
Some exact methods:
 Simplex and interior point methods for LP

problems –variables are continuous
 Branch and Bound and A*: based on implicit
enumeration of all solutions;
 ‘divide and conquer’ concept
Outline
 Some concepts behind LP solvers
Linear programming (LP)
Standard form of a In matrix form: min c T x 

LP problem subject to :
Ax = b
min c1 x1 + ... + cn xn  x  0
Vector of
subject to : continuous
decision variables xT = ( x1 ,..., xn ),
a11 x1 + ... + a1n xn = b1 cT = (c1 ,..., cn ),
Constant
... vectors b T = (b1 ,..., bm ),
am1 x1 + ... + amn xn = bm  a11


... a1n 

Constant A =  ... ... ... 
x1  0,..., xn  0 matrix a ... amn 
 m1
LP Example
 A company makes 2 types of ice cream: A and

B. To make ice cream A, they need 1 bag of
sugar, one bottle of milk and 1 bag of flavoring
per kg of ice cream. To make ice cream B, they
need 2 bottles of milk and 1 bag of flavoring per
kg (it is sugar-free). The company buys 7 bags
of sugar, 10 bottles of milk and 8 bags of
flavoring daily. The profit per kg of ice cream A
is 2.25 Euros and for ice cream B is 4 Euros.
How much of each type should the company
produce daily to maximize their profit?
LP Example – LP formulation
Objective
Max 2.25x1 + 4x2
Function
s.t. x1 < 7
Problem specific
x1 + 2x2 < 10
constraints
x1 + x 2 < 8
x1 > 0 and x2 > 0 Non-negativity

Constraints
LP Example: Graphical Solution
 1st constraint: x1 < 7 (and non-negativity)

x2
8
7 x1 = 7
6
Feasible points
5 for first constraint
4
3
2
(7, 0)
1
x1
1 2 3 4 5 6 7 8 9 10
 2nd constraint: x1 + 2x2 < 10 (and non-negativity)

x2
8
7
(0, 5)
6
5 x1 + 2x2 = 10
4
3 Feasible points
2 for second constraint
(10, 0)
1
x1
1 2 3 4 5 6 7 8 9 10
 3rd constraint: x1 + x2 < 8 (and non-negativity)

x2
(0, 8)
8
7
6 x1 + x2 = 8
5
4
Feasible points
3
for third constraint
2
1 (8, 0)
x1
1 2 3 4 5 6 7 8 9 10
LP Example 1: Graphical Solution
 Feasible region (solution space)

x2
8 x1 + x2 = 8
7
6 x1 = 7
5
4
3
Feasible x1 + 2x2 = 10
2
Region
1
x1
1 2 3 4 5 6 7 8 9 10
 Objective Function: 2.25x1 + 4x2

x2
8
7
6 (0, 4.5) Possible objective Function line
5 2.25x1 + 4x2 = 18
4
3
2 Feasible
1 Region (8, 0)
x1
1 2 3 4 5 6 7 8 9 10

x2
Example Objective
obj. func. 8 Objective Function functions
lines 2.25x1 + 4x2 = 16 lines are
7
parallel
6 Objective Function
5 2.25x1 + 4x2 = 18
4 Objective Function
3 2.25x1 + 4x2 = 20
2 Feasible
1 Region
x1
1 2 3 4 5 6 7 8 9 10

x2
8 Objective
7 function grows
6
5 Obj. function =
20
4
18
3 16
2 Feasible
1 Region
x1
1 2 3 4 5 6 7 8 9 10
 Optimal solution
x2
Maximum
8
Objective Function Line
7 2.25x1 + 4x2 = 21.5
6
5 Optimal Solution
4
(x1 = 6, x2 = 2)
3
2
Feasible
1
Region
x1
1 2 3 4 5 6 7 8 9 10
Solving linear programs
 Efficient exact algorithms have been developed

to solve LPs
 Simplex method
 Interior point methods
 This is possible because the feasible region

(solution space) is a convex set and the
objective function is a convex function
 Thus the global optimum is necessarily a node of the
polytope representing the feasible region
 A local optimum is also a global optimum
 ‘fairly easy’; No need for heuristics

Convex Sets and Convex Hull
 A set is convex if for

every pair of points in
the set, the line
segment connecting
them is also in the convex concave
set.
 The convex hull for a

set of points X is the
minimal convex set
containing X.
Simplex
 Initially developed by G. Danzig in 1947
 Main idea: The search jumps from one feasible

corner point solution to the next adjacent one
until an optimal one is found
 Optimality test: if a feasible corner point solution has
no adjacent feasible corner point solution that is
better, then it is an optimal solution (convex hull;
local optimum is a global optimum)
Interior point methods
 Initially developed: 1984, N. Karmarker, AT&T Bell Labs

 Potential to solve huge LPs that are too much for
Simplex
 Main idea:
 Iteratively checks feasible solutions, but unlike simplex it
traverses interior feasible solutions towards the optimal; always
moves to a better solution
 The search moves in the direction of the fastest improvement
(perpendicular to objective function) using the projected gradient
and step size alpha
Outline
 Concept behind LP solvers
Integer linear programming (ILP or IP)
 BUT, many real-world problems have discrete

variables!!
 ILP: Discrete decision variables, linear
objective function and constraints
 Mixed ILP (MILP): discrete and continuous
decision variables
 Harder to solve than LP
 Exacttechniques: Branch and bound,
 Approximate: relaxations, decomposition
approaches, cutting plane algorithms
Why is IP more difficult than LP: Example
max 3 x1 + 2 x2
subject to : x2
3 x1 + x2  9 -x1 + x2 < 1
x1 + 3 x2  7 5
3x1 + x2 < 9
− x1 + x2  1
4
x1 , x2  0, x1 , x2 integer Max 3x1 + 2x2
3
Optimum LP (2.5,1.5)
2
x1 + 3x2 < 7
Solution space = 1
feasible solutions x1
are the integer 1 2 3 4 5 6 7
points in the
shaded region
max 3 x1 + 2 x2 Round the LP optimum?

subject to : “upwards”: (3,2) not feasible point
3 x1 + x2  9
“downwards” : (2,1) feasible, obj. func. = 8
x1 + 3 x2  7 x2
− x1 + x2  1
x1 , x2  0, x1 , x2 integer 5
4 (3,2) not feasible

3 The LP optimum (2.5,1.5)
Is (2,1) the optimum solution? (2,1) feasible, obj.func. = 8
2
NO!!!!
Integer solutions closest to the 1
LP solution are not neccessarily x1
optimal for the IP!!! 1 2 3 4 5 6 7
max 3 x1 + 2 x2 We solve the problem with brute-force here

subject to :
3 x1 + x2  9 NOTE:The optimal
x1 + 3 x2  7 x2
solution doesn’t have
− x1 + x2  1 to be a corner point!!!!
x1 , x2  0, x1 , x2 integer 5 That makes it harder
to solve!
4
x1 x2 obj.func.
1. 0 0 0 3 Optimum LP (2.5,1.5)
2. 1 0 3 (2,1) feasible, obj.func.=8
3. 2 0 6 2
4. 3 0 9 (3,0) feasible, obj.func.=9,
5. 0 1 2 1 OPTIMAL IP
6. 1 1 5 x1
7. 2 1 8 1 2 3 4 5 6 7
8. 1 2 7
LP-relaxation
 LP-relaxation: relax the integral constraints

 A solution to the LP-relaxation is an upper bound
(maximization problem) or a lower bound (minimization
problem) on the solution to the IP
 The only time IP is as easy as LP is if the LP relaxation
has an integral optimal solution
 There are some special types of IP problems which guarantee
integral solution of the LP-relaxation by their structure:
transport, transshipment, max flow…
 A method to solve IP problems: LP-relaxation+rounding
but doesn’t always work well (see example)
 Obtained solution may not even be feasible
 Doesn’t guarantee optimality
Outline
 Concept behind LP solvers
Branch and Bound
 Used to help solve IP and BIP problems exactly
 “divide and conquer” principle
 Main idea: divide the original problem into

subproblems and try to eliminate those
subproblems which surely do not contain the
optimal solution
 Implicit enumeration of possible solutions –

build a solution tree
Initialization
1. Solve the LP-relaxation of the original problem

 If the LP solution is integral→ problem solved!
 OPT(LP-relaxation) >= OPT(IP) for maximization
 If not, we have an upper bound on the optimal
solution
2. Find a feasible integer solution
 We can round the LP solution, but check feasibility!!!
 Trial and error
…
Example – IP formulation & LP-relaxation
Initialization:
Max Z=2x1 + 3x2 1. Solve the LP-
s.t. 195x1 +273x2 < 1365 relaxation
4x1 + 40x2 < 140
x1 < 4
Solution to the LP-
Drop the x1 > 0 and x2 > 0 relaxation:
integrality x1, x2 integer Z=14.66
constraints (x1, x2)=(2.44, 3.26)
IP!
Root node
Solution to the LP- Upper Bound to the IP:

relaxation: UB(IP)=OPT(LP)= 14.66
Z=14.66
(x1, x2)=(2.44, 3.26)
Since the coefficients in Z are
Branch-and-Bound integral and the variables must be
tree root node integral for the IP, an Upper
Bound is UB(IP)=14
1 If Z doesn’t
Upper bound on IP solution have integral
UB=14 coefficients, use
(x1,x2)= the OPT(LP)
value
(2.44, 3.26) Values of the variables for the
optimal LP solution
Find a feasible solution
Initialization:
2. Find a feasible solution
 E.g., rounding:
rounding
(x1, x2)=(2.44, 3.26) (x1, x2)=(2, 3)
Z = 13
Check feasibility:
195x1 +273x2 < 1365
4x1 + 40x2 < 140 Incumbent
x1 < 4 solution
Z*=13
x1 > 0 and x2 > 0
Branching
 Branching: choose a branching variable xi

which is non-integral in the LP solution (xi(LP))
of that node
 Ifthere are multiple non-integral variables, choose
any or make a problem specific or general rule,
e.g. the variable farthest from integrality
 Divide the problem into 2 subproblems each
with an added constraint:
xi  xi (LP ) xi  xi (LP )
Branching
Choose 1 Original
branching UB=14 problem 1
variable (x1,x2)=
x1=2.44 (2.44,3.26)
x1  2 x1  3
Subproblem 3
Subproblem 2
2 3
Subproblems
Subproblem 2 (SP2) Subproblem 3 (SP3)
Max Z=2x1 + 3x2 Max Z=2x1 + 3x2
s.t. 195x1 +273x2 < 1365 s.t. 195x1 +273x2 < 1365
4x1 + 40x2 < 140 4x1 + 40x2 < 140
x1 < 4 x1 < 4
x1 < 2 x1 ≥ 3
x1 > 0 and x2 > 0 x1 > 0 and x2 > 0
x1, x2 integer x1, x2 integer
New
constraints
Subproblems
Solve LP-relaxation of
Subproblem 2:
Z=13.9
Subproblem 2 (SP2) (x1, x2)=(2, 3.3)
Max Z=2x1 + 3x2
Since the coefficients in Z are
s.t. 195x1 +273x2 < 1365 integral→ Upper bound on SP2
4x1 + 40x2 < 140 UB(SP2)=13
x1 < 4
x1 < 2
2
x1 > 0 and x2 > 0
UB=13
x1, x2 integer (x1, x2)=
(2, 3.3)
Subproblems
Subproblem 3:
Subproblem 3 (SP3) Z=14.52
(x1, x2)=(3, 2.86)
Max Z=2x1 + 3x2
s.t. 195x1 +273x2 < 1365 Since the coefficients in Z are

integral→ Upper bound on SP3
4x1 + 40x2 < 140
UB(SP2)=14
x1 < 4
x1 ≥ 3
x1 > 0 and x2 > 0 3
x1, x2 integer UB=14
(x1, x2)=
(3, 2.86)
1st Branching Iteration
1
UB(IP)=14 Z*=13
(x1,x2)=
(2.44,3.26)
x1  2 x1  3
2 3
UB=13 UB=14
(x1, x2)= (x1, x2)=
(2, 3.3) (3, 2.86)
Fathoming tests
Fathoming tests are used to ‘conquer’ nodes

(subproblems)–one or more must hold
1. The LP relaxation has no solution
 The additional constraint has made it infeasible
2. The Upper bound of a node ≤ Z*
3. The value of the variables of the optimal
solution of the LP-relaxation is integral, i.e. the
optimal solution to the subproblem has been
found
1. If it is better than Z*, update Z*
2. If not, the node was conquered by test 2
Node 2 – fathoming tests
FATHOMING TESTS
1. LP-relaxation infeasible Z*=13
2. UB(node)of a node ≤ Z*
3. OPT(node) found
2
1. LP solution feasible UB=13
2. UB(SP2)=13 ≤ Z*=13 (x1, x2)=
(2, 3.3)
Conquered by test 2 :
The best solution to subproblem
2 is 13, and we already found CONQUERED
such a solution so it’s a waste of
time to look for another
FATHOMING TESTS
3. OPT(node) found
3
UB=14
1. LP solution feasible (x1, x2)=
2. UB(SP3)=14 ≤ Z*=13 (3, 2.86)
3. LP solution is not integral
Node 3 is not conquered – keep branching!
2nd Branching Iteration
1 Z*=13
UB(IP)=14
(x1,x2)=
(2.44,3.26)
Choose
branching
2 3 variable.
UB=13 UB=14 x1 is integral,
(x1, x2)= (x1, x2)= so choose
(2, 3.3) (3, 2.86) x2=2.86
x2  2 x2  3
4 5
New Subproblems
Subproblem 4 (SP4) Subproblem 5 (SP5)
Max Z=2x1 + 3x2 Max Z=2x1 + 3x2
s.t. 195x1 +273x2 < 1365 s.t. 195x1 +273x2 < 1365
4x1 + 40x2 < 140 4x1 + 40x2 < 140
x1 < 4 x1 < 4
x1 ≥ 3 x1 ≥ 3
x2 < 2 x2 ≥ 3
x1 > 0 and x2 > 0 x1 > 0 and x2 > 0
x1, x2 integer x1, x2 integer
Constraints New
from SP3 constraints
New Subproblems
Subproblem 4 (SP4) Subproblem 4:
Z=14
Max Z=2x1 + 3x2 (x1, x2)=(4,2)
s.t. 195x1 +273x2 < 1365

4x1 + 40x2 < 140 Upper bound on SP4
UB(SP4)=14
x1 < 4
x1 ≥ 3
x2 < 2 4
x1 > 0 and x2 > 0 UB=14
x1, x2 integer (x1, x2)=
(4,2)
New Subproblems
Subproblem 5 (SP5)
Max Z=2x1 + 3x2 Subproblem 5→ No solution!
s.t. 195x1 +273x2 < 1365
4x1 + 40x2 < 140
x1 < 4 5
x1 ≥ 3 No solution,
x2 ≥ 3 LP-relax
x1 > 0 and x2 > 0 infeasible
x1, x2 integer INFEASIBLE!!!
2nd Branching Iteration
1
UB(IP)=14 Z*=13
(x1,x2)=
(2.44,3.26)
2 3
UB=13 UB=14
(x1, x2)= (x1, x2)=
(2, 3.3) (3, 2.86)
x2  2 x2  3
4 5
UB=14 No solution,
(x1, x2)= LP-relax
(4,2) infeasible
FATHOMING TESTS
1. LP-relaxation infeasible
Z*=13
3. OPT(node) found
4
UB=14
1. LP solution feasible (x1, x2)=
2. UB(SP4)=14 ≤ Z*=13 (4,2)
3. LP solution is integral
Conquered by test 3
Solution to the LP-relaxation = CONQUERED
14 is better than incumbent Z*=14
solution Z* =13 → new Z*=14
FATHOMING TESTS
3. OPT(node) found
5
No solution,
1. LP solution infeasible LP-relax
infeasible
Conquered by test 1
CONQUERED
2nd Fathoming Iteration
1
UB(IP)=14
Z*=14 When all nodes
(x1,x2)=
(2.44,3.26)
are conquered→
STOP.
Z* is the optimal
2 3
UB=13 UB=14
solution to the
(x1, x2)= (x1, x2)= original IP
(2, 3.3) (3, 2.86)
x2  2 x2  3
4 5
UB=14 No solution,
(x1, x2)= LP-relax
(4,2) infeasible
Branch and Bound algorithm
1. Initialization
1. Solve LP-relaxation of the original problem; get UB on IP problem
2. Find any feasible itegral solution (e.g. rounding) → Z*
2. Branching iterations
1. Branching
1. Choose an unconquered node n (e.g. the last created or with max UB)
2. Choose a branching variable xi (e.g. furthest from integer at node n)
3. Make 2 new nodes (subproblems) with constraints:
xi  xi (LP ) xi  xi (LP )
2. Solve LP-relaxation of new subproblems and find their UBs
3. Perform fathoming tests to try to conquer the new nodes
3. Optimality check
1. If there are unconquered nodes, go to Step 2. Otherwise, STOP and
Z* is the optimal solution
Branch and Cut
 Improved technique for solving pure Binary IP

problems
 Combines branch and bound with cutting
planes
 Cutting plane (CP): a CP for an IP problem
is a new constraint that reduces the feasible
region of the LP-relaxation without
eliminating any IP feasible solutions

Lecture 3.a - Exact Methods

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lecture 3.a - Exact Methods

Uploaded by

Copyright:

Available Formats

University of Zagreb, Croatia

Academic year 2020/2021

 Concept behind LP solvers (simplex)

 Why is IP harder than LP?

 Branch and bound

 Simplex and interior point methods for LP

 Some concepts behind LP solvers

 Why is IP harder than LP?

 Branch and bound

Standard form of a In matrix form: min c T x 

am1 x1 + ... + amn xn = bm  a11

 A company makes 2 types of ice cream: A and

x1 > 0 and x2 > 0 Non-negativity

 1st constraint: x1 < 7 (and non-negativity)

 2nd constraint: x1 + 2x2 < 10 (and non-negativity)

 3rd constraint: x1 + x2 < 8 (and non-negativity)

 Feasible region (solution space)

 Objective Function: 2.25x1 + 4x2

 Objective Function: 2.25x1 + 4x2

 Objective Function: 2.25x1 + 4x2

 Efficient exact algorithms have been developed

 This is possible because the feasible region

 ‘fairly easy’; No need for heuristics

 A set is convex if for

 The convex hull for a

 Initially developed by G. Danzig in 1947

 Main idea: The search jumps from one feasible

 Initially developed: 1984, N. Karmarker, AT&T Bell Labs

 Concept behind LP solvers

 Why is IP harder than LP?

 Branch and bound

 BUT, many real-world problems have discrete

max 3 x1 + 2 x2 Round the LP optimum?

4 (3,2) not feasible

max 3 x1 + 2 x2 We solve the problem with brute-force here

 LP-relaxation: relax the integral constraints

 Concept behind LP solvers

 Why is IP harder than LP?

 Branch and bound

 Used to help solve IP and BIP problems exactly

 “divide and conquer” principle

 Main idea: divide the original problem into

 Implicit enumeration of possible solutions –

1. Solve the LP-relaxation of the original problem

Solution to the LP- Upper Bound to the IP:

 Branching: choose a branching variable xi

xi  xi (LP ) xi  xi (LP )

Subproblem 2 (SP2) Subproblem 3 (SP3)

Max Z=2x1 + 3x2 Max Z=2x1 + 3x2

s.t. 195x1 +273x2 < 1365 Since the coefficients in Z are

Fathoming tests are used to ‘conquer’ nodes

Node 3 is not conquered – keep branching!

Subproblem 4 (SP4) Subproblem 5 (SP5)

Max Z=2x1 + 3x2 Max Z=2x1 + 3x2

s.t. 195x1 +273x2 < 1365

 Improved technique for solving pure Binary IP

You might also like