Column

Generation

Jacques Desrosiers
Ecole des HEC & GERAD
 Contents

• The Cutting Stock Problem  Alternative Formulations to

the Cutting Stock Problem
 Basic Observations
 IP Column Generation
 LP Column Generation
 Branch-and- ...
 Dantzig-Wolfe Decomposition
 Acceleration Techniques
 Dantzig-Wolfe decomposition
vs Lagrangian Relaxation  Concluding Remarks

 Equivalencies

COLUMN GENERATION 2
 A Classical Paper :
The Cutting Stock Problem
P.C. Gilmore & I : set of items
R.E. Gomory ni : number of times
item i is requested
A Linear Programming li : length of item i
Approach to the Cutting L : length of a standard roll
Stock Problem. J : set of cutting patterns
aij : number of times item i
Oper. Res. 9, 849-859. is cut in pattern j
(1960) Y j : number of times
pattern j is used

COLUMN GENERATION 3
 The Cutting Stock
Problem ...

P : min  Y j Set J can be huge.

jJ
Solution of LP,
 aijY j  ni ,
jJ
iI the linear relaxation of P,
by column generation.
Y j  0, jJ LP : min  Y j
jJ
Yj integer, jJ
a Y
jJ
ij j  ni , iI
Minimize the number
of standard rolls used Y j  0, jJ
COLUMN GENERATION 4
 The Cutting Stock
Problem ...
Given a subset J '  J
and the dual multipliers
i   i ( J ' ), i  I ,
LP : min  Y j the reduced cost of
j J any new patterns

a Y
j J
ij j  ni , i I
must satisfy:
min 1   aij i  0 ;
jJ \ J '
iI
Y j  0, jJ otherwise, LP is optimal.

COLUMN GENERATION 5
 The Cutting Stock
Problem ...
Reduced costs for j  J ' ai is a decision variable:
are non negative, hence: the number of times
item i is selected in a
min 1    i aij new pattern.
jJ \ J '
iI

 min 1    i aij The Column Generator

jJ
iI
is a Knapsack Problem.
 min 1    i ai  max   i ai
iI
iI

l a
iI
i i  L
l a i i L
iI
ai  0 integer, i  I
ai  0 integer, i  I
COLUMN GENERATION 6
 Basic
Observations
Keep At an inferior level,
the coupling constraints solve the
at a superior level, Column Generator,
in a Master Problem; which is often separable
in several independent
this with the goal sub-problems;
of obtaining a
Column Generator which use a specialized
is rather easy to solve. algorithm that exploits
its particular structure.

COLUMN GENERATION 7
 LP
Column Generation
MASTER PROBLEM

Columns Dual Multipliers

COLUMN GENERATOR
(Sub-problems)

Optimality Conditions: primal feasibility

complementary slackness
dual feasibility

COLUMN GENERATION 8
 Historical
Perspective
G.B. Dantzig & Authors give credit to:
P. Wolfe L.R. Ford & D.R. Fulkerson

Decomposition Principle A Suggested Computation

for Linear Programs. for Multi-commodity flows.

Oper. Res. 8, 101-111. Man. Sc. 5, 97-101.

(1960) (1958)

COLUMN GENERATION 9
 Historical Perspective :
a Dual Approach
DUAL MASTER PROBLEM

Rows Dual Multipliers

ROW GENERATOR
(Sub-problems)

J.E. Kelly The Cutting Plane Method

for Solving Convex Programs.
SIAM 8, 703-712.
(1960)
COLUMN GENERATION 10
Dantzig-Wolfe Decomposition :
the Principle

P : min cx x  Conv ( X ) rewritten as :

Ax  b x x  p p  d  r r

x X p ( ) r (  )


p ( )
p 1
On conv( X ), define sets
() : extreme points  p  0, p  ()
() : extreme rays  r  0, r  ()

COLUMN GENERATION 11
Dantzig-Wolfe Decomposition :
Substitution

P : min cx P : min c( 
p ( )
x p p  d  )
r (  )
r r

Ax  b A(  x p p  d  )  br r
p ( ) r (  )
x X  p 1
p ( )

 p  0, p  ()
 r  0, r  ()

COLUMN GENERATION 12
 Dantzig-Wolfe Decomposition :
The Master Problem
The Master Problem
P : min c( 
p ( )
x p p  d  )
r (  )
r r P : min 
p ()
(cx p ) p   (cd )
r (  )
r r

A( 
p ( )
x p p  d  )  b
r (  )
r r 
p ()
( Ax p ) p   ( Ad )
r (  )
r r b


p ( )
p 1 
p ()
p 1

 p  0, p  ()  p  0, p  ()
 r  0, r  ()  r  0, r  ()

COLUMN GENERATION 13
 Dantzig-Wolfe Decomposition :
The Column Generator
Given the current min cx   ( Ax)   0
dual multipliers x  conv( X )
for a subset of columns :
coupling constraints ( ) extreme rays if

convexity constraint o (c   A) d r  0,
for some r  ()
otherwise
generate (if possible)
extreme points if
new columns
cx p   ( Ax p )   0  0,
with negative
reduced cost : for some p  ()

COLUMN GENERATION 14
 Remark

extreme rays if extreme rays if

(c   A) d r  0,
for some r  ()
otherwise
extreme points if
cx p   ( Ax p )   0  0,
for some p  ()
(c   A) d r  0,
for some r  ()
otherwise
extreme points if
cx p   ( Ax p  b)  b   0 ,
for some p  ()

COLUMN GENERATION 15
Dantzig-Wolfe Decomposition :
Block Angular Structure

P : min  c x k k

kK

A x
kK
k k
b

x X , kK
k k Exploits the structure of
many sub-problems.

Similar developments
& results.

COLUMN GENERATION 16
Dantzig-Wolfe Decomposition :
Algorithm
MASTER PROBLEM

Columns Dual Multipliers

COLUMN GENERATOR
(Sub-problems)

Optimality Conditions: primal feasibility

complementary slackness
dual feasibility

COLUMN GENERATION 17
Dantzig-Wolfe Decomposition :
a Lower Bound
Given the current Current solution value
dual multipliers + minimum
( ) (coupling constraints) reduced cost column
o (convexity constraint),
(b   0 )  min cx   ( Ax)   0
a lower bound can x X
be computed
at each iteration,
 min cx   ( Ax  b)
as follows:
x X

COLUMN GENERATION 18
Lagrangian Relaxation Computes
the Same Lower Bound
P : min cx L( )  min cx   ( Ax  b)
x  conv( X )
Ax  b
provides a lower bound on P :
x X
  if (c   A)d r  0, for some r  () ;
 cx p   ( Ax p  b) for some p  (), otherwise.
The best lower bound is obtained using max L( ).

COLUMN GENERATION 19
Dantzig-Wolfe vs Lagrangian
Decomposition Relaxation
Essentially utilized Essentially utilized
for Linear Programs for Integer Programs

Relatively difficult to Easy to implement with

implement subgradient adjustment
for multipliers 

Slow convergence No stopping rule !

Rarely implemented  6% of OR papers

COLUMN GENERATION 20
 Equivalencies
Dantzig-Wolfe In both methods, coupling
Decomposition & or complicating constraints
go into a
Lagrangian Relaxation DUAL MULTIPLIERS
ADJUSTMENT PROBLEM :
in DW : a
LP Master Problem
if both have the same in Lagrangian Relaxation :
sub-problems max L( )

COLUMN GENERATION 21
 Equivalencies ...

Column Generation corresponds to the solution

process used in Dantzig-Wolfe decomposition.

This approach can also be used directly by

formulating a Master Problem and sub-problems
rather than obtaining them by decomposing a
Global formulation of the problem. However ...

COLUMN GENERATION 22
 Equivalencies ...
… for any Column
Generation scheme, The definition of the
there exits Global Formulation
a Global Formulation is not unique.
that can be decomposed
by using a A nice example:
generalized Dantzig-
Wolfe decomposition
The Cutting Stock
which results in the
Problem
same Master and sub-
problems.

COLUMN GENERATION 23
 The Cutting Stock Problem :
Kantorovich (1960/1939)
K: set of min  Y k
available rolls kK

Y k
: binary variable,  i  ni ,
X
kK
k
iI
1 if roll k is cut,
0 otherwise i i
l
iI
X k
 L Y k
, kK
k
X i : number of times Y  0 or 1, k  K
k
item i is cut
on roll k X  0 integer, i  I , k  K .
i
k

COLUMN GENERATION 24
 The Cutting Stock Problem :
Kantorovich ...
Kantorovich’s LP |K| integer Knapsack
lower bound is weak: Problems, (which

l n
iI
i i / L. provide extreme point
columns).
However, Dantzig-Wolfe Aggregation of
decomposition provides identical columns
the same bound as the in the Master Problem.
Gilmore-Gomory LP
bound if sub-problems Branch & Bound
performed on k
are solved as ... X .
i

COLUMN GENERATION 25
 The Cutting Stock Problem :
Valerio de Carvalhó (1996)
Network type formulation on graph ( N , A)
N  {0, 1, 2,..., L  1, L}
A  {(u , v) : 0  u  v  L and v  u  li , i  I } {(u , u  1), u  0,1,..., L  1}

Example with L  5, and l1  2, l2  3

COLUMN GENERATION 26
 The Cutting Stock Problem :
Valerio de Carvalhó ...
min Z
X
( u ,u  li ) A
u , u  li  ni , iI

X
( 0 ,v )
0v  z,

X
( u ,v )
uv  X
( v ,u )
vu  0, v  {1, 2 ,  , L  1}

X
(u , L )
uL  z,

X uv  0 integer, (u , v )  A
COLUMN GENERATION 27
 The Cutting Stock Problem :
Valerio de Carvalhó ...
The sub-problem is a The Master Problem
shortest path problem appears without the
on a acyclic network. convexity constraint.

This Column Generator The correspondence

only brings back with Gilmore-Gomory
extreme ray columns, formulation is obvious.
the single extreme point
being the null vector. Branch & Bound
performed on X uv .
COLUMN GENERATION 28
 The Cutting Stock Problem :
Desaulniers et al. (1998)
It can also be viewed as a Column Generation tools
Vehicle Routing Problem developed for Routing
on a acyclic network Problems can be used.
(multi-commodity flows): Columns correspond to
paths visiting items
Vehicles Rolls the requested number of
Customers Items times.
Demands li  ni Branch & Bound
performed on
Capacity L X . k
ij

COLUMN GENERATION 29
 IP
Column Generation
IP : min 
p ( )
(cx p ) p   (cd )
r (  )
r r
IP : min cx
Ax  b 
p ( )
( Ax p ) p   ( Ad )
r (  )
r r b
x X
 p 1
x integer p ( )

 p  0, p  ()
 r  0, r  ()
x x 
p ( )
p p  d 
r (  )
r r

x integer
COLUMN GENERATION 30
 Integrality
Property
The sub-problem In this case,
satisfies the
Integrality Property max L( )  v( LP )
otherwise
if it has an integer
optimal solution for v( LP )  max L( )  v( IP)
any choice of linear
objective function,
even if the integrality i.e., the solution process
restrictions on the partially explores the
variables are relaxed. integrality gap.
COLUMN GENERATION 31
 Integrality
Property ...
In most cases, the … some overlooked
Integrality Property is a formulations become
undesirable property! very good when a
Dantzig-Wolfe
decomposition process
Exploiting the non trivial
is applied to them.
integer structure reveals
that ...
The Cutting Stock Problem
Localization Problems
Vehicle Routing Problems
...

COLUMN GENERATION 32
 IP Column Generation :
Branch-and-...
Branch-and-Bound : Branch-and-Cut :

branching decisions cutting planes defined

on a combination of the on a combination of the
original (fractional) variables original variables;

of a Global Formulation on  at the Master level, as

which Dantzig-Wolfe coupling constraints;
Decomposition is applied.  in the sub-problem,
as local constraints.

COLUMN GENERATION 33
 IP Column Generation :
Branch-and-...

IP : min c x
Ax  b
x X
{ Branching &
Cutting decisions }
x integer Dantzig-Wolfe decomposition
applied at all decision nodes

COLUMN GENERATION 34
 IP Column Generation:
Branch-and-...
Branch-and-Price : For alternative methods,
see the work of
a nice name
S. Holm & J. Tind

which hides
C. Barnhart, E. Johnson,
a well known G. Nemhauser, P.
solution process Vance, M.
relatively easy to apply. Savelsbergh, ...

F. Vanderbeck & L. Wolsey

COLUMN GENERATION 35
 Application to Vehicle Routing and
Crew Scheduling Problems (1981 - …)

 J. Desrosiers, Y. Dumas, F. Global Formulation :

Soumis & M. Solomon Non-Linear Integer
Time Constrained Routing and
Scheduling
Multi-Commodity Flows
Handbooks in OR & MS, 8
(1995) Master Problem :
 G. Desaulniers et al. Covering & Other
A
Unified Framework for Linking Constraints
Deterministic Vehicle Routing
and Crew Scheduling Problems Column Generator :
Fleet
T. Crainic & G. Laporte (eds) Resource Constrained
Management & Logistics (1998) Shortest Paths

COLUMN GENERATION 36
 Resource Constrained
Shortest Path Problem on G=(N,A)

Min  c x    T ij ij i i
r

( i , j ) A iN rR

P(N, A) : x ij  x
j:( j ,i ) A
ij  1, i  o;  1, i  d ; 0, i  o, d
j:( i , j ) A

xij ( f ij (Ti )  T jr )  0, (i, j )  A, r  R

(  ij i i  (
x ) a r

j:( i , j ) A
T r
 ij i , i  N , r  R
x ) b r

j:( i , j ) A

xij binary, (i, j )  A

COLUMN GENERATION 37
 Integer Multi-Commodity
Network Flow Structure
Min 
kK
(  ij xij 
c k k
 i Ti )
k kr

( i , j ) A k iN k rR


kK
(  ij )  bi ,
x k
i  N k
j:( i , j ) Ak kK


kK
(  m,ij ij  m,iTi )  bm , m  M
a k
x k
 a kr kr

( i , j )A k iN k

( x k , T k )  P( N k , Ak ), k  K

COLUMN GENERATION 38
 Vehicle Routing and Crew
Scheduling Problems ...
Sub-Problem
is strongly NP-hard Master Problem results in
Set Partitioning/Covering
It does not posses the type Problems
Integrality Property
Paths  Extreme points

Branching and Cutting decisions are

taken on the original network flow,
resource and supplementary variables

COLUMN GENERATION 39
 IP Column Generation :
Acceleration Techniques
Exploit all the Structures
on the Column Generator With Fast Heuristics

Master Problem Re-Optimizers

Global Formulation Pre-Processors

To get Primal
& Dual Solutions

COLUMN GENERATION 40
 IP Column Generation :
Acceleration Techniques ...
Link all the Structures Be Innovative !
Multiple Columns : Primal Perturbation &
selected subset close to Dual Restriction :
expected optimal solution to avoid degeneracy and
convergence difficulties
Partial Pricing in case of
many Sub-Problems : Branching & Cutting :
as in the Simplex Method on integer variables !

Early & Multiple Branching Branch-first, Cut-second

& Cutting : quickly gets Approach :
local optima exploit solution structures
COLUMN GENERATION 41
 Stabilized Column Generation
min cx
min cx
Ax  y1  y2  b max b max b
Ax  b
x0 A  c A  c
x0
0  y1   1 ,0  y2   2 d1    d 2

Perturbed Primal Restricted Dual

min cx  d1 y1  d 2 y 2
Ax  y1  y 2  b
x0
0  y1   1 , 0  y2   2
Stabilized Problem
COLUMN GENERATION 42
 Concluding Remarks
 DW Decomposition is … the Analytic Center
an intuitive framework Cutting Plane Method
that requires all tools
(Vial, Goffin, du Merle,
discussed to become Gondzio, Haurie, et al.)
applicable
 “easier” for IP which exploits recent
 very effective in
developments in interior
point methods,
several applications
Imagine what could be and is also compatible
done with theoretically with Column Generation.
better methods such as
COLUMN GENERATION 43
“Bridging Continents and Cultures”
F. Soumis M. Desrochers
M. Solomon Y. Dumas
Canada, USA, Italy,
G. Desaulniers Denmark, Sweden, M. Gamache
P. Hansen Norway, Ile Maurice, D. Villeneuve
France, Iran, Congo,
J.-L. Goffin K. Ziarati
New Zealand, Brazil,
O. Marcotte Australia, Germany, I. Ioachim
G. Savard Romania, Switzerland, M. Stojkovic
Belgium, Tunisia,
O. du Merle G. Stojkovic
Mauritania, Portugal,
O. Madsen China, The Netherlands, N. Kohl
P.O. Lindberg ... A. Nöu
B. Jaumard … et al.

COLUMN GENERATION 44