CO 250: Introduction to Optimization

Module 4: Duality Theory (Weak Duality)

University
c of Waterloo 1 / 20
 Recap: Shortest Path LP

Solutions to a shortest path instance

a b c
G = (V, E), s, t ∈ V , ce ≥ 0 for all
e ∈ E, correspond to feasible d
h
0, 1-solutions for the LP s t

X

min ce xe : e ∈ E j i g
X

s.t. xe : e ∈ δ(U ) ≥ 1
(U ⊆ V, s ∈ U, t ∈
/ U) where
x≥0 • b = 1;
• A has a row for every s, t-cut
This LP is of the form: δ(U ), and a column for every
edge e; and
min{cT x : Ax ≥ b, x ≥ 0} • AU e = 1 if e ∈ δ(U ) and
AU e = 0 otherwise.

University
c of Waterloo 2 / 20
 Recap: Shortest Path Dual

min{cT x : Ax ≥ b, x ≥ 0} (P) a b c

d
The dual of (P) is given by h
s t
T T
max{b y : A y ≤ c, y ≥ 0} (D)
j i g
If (P) is a shortest path LP, then we
can rewrite (D) as

X Theorem
max (yU : s ∈ U, t 6∈ U )
X If x̄ is feasible for (P) and ȳ is
s.t. (yU : e ∈ δ(U )) ≤ ce feasible for (D), then bT ȳ ≤ cT x̄.
(e ∈ E) Equivalent: y feasible widths and P
y≥0 an s, t-path −→ 1T y ≤ c(P )

University
c of Waterloo 3 / 20
 This Lecture

Question: Can we find max cT x

lower-bounds on the optimal
s.t. Ax ? b
value of a general LP?
x?0
In the LP on the right,
Recall: in the primal-dual pair
Ax ? b
min{cT x : Ax ≥ b, x ≥ 0} (P)
stands for a system of
inequalities whose sign is one of max{bT y : AT y ≤ c, y ≥ 0} (D)
≤, = or ≥, and

x?0 • each non-negative variable, xe , in

indicates that variables are either
non-negative, non-positive, or
free.
University
c of Waterloo
(P) corresponds to a ‘≤‘-constraint
in (D), and
• each ‘≥‘-constraint in (P)
corresponds to a non-negative
variable yU in (D).
4 / 20
 Weak Duality in General

Consider the primal LP Its dual LP is given by

max cT x min bT y
s.t. Ax ? b s.t. AT y ? c
x?0 y?0

Question: What are the question

marks?
A: As before:

primal variables ≡ dual constraints

primal constraints ≡ dual variables

University
c of Waterloo 5 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Example 1: Its dual LP:

max (1, 0, 2)x (P) min (3, 4)y (D)

       
3 −1 0 ≤ 3 3 1 1
s.t. x
1 0 1 = 4 s.t. −1 0 y ? 0
x1 , x2 ≥ 0, x3 free 0 1 2
y?0
University
c of Waterloo 6 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Example 1: Its dual LP:

max (1, 0, 2)x (P) min (3, 4)y (D)

       
3 −1 0 ≤ 3 3 1 ≥ 1
s.t. x
1 0 1 = 4 s.t. −1 0 y ≥ 0
x1 , x2 ≥ 0, x3 free 0 1 = 2
y1 ≥ 0, y2 free
University
c of Waterloo 7 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Example 2: To compute dual LP, check

right-hand side of table:
min dT y (P)
T
s.t. W y ≥ e max eT x (D)
y≥0 s.t. W x ? d
x?0

University
c of Waterloo 8 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Example 2: To compute dual LP, check

right-hand side of table:
min dT y (P)
T
s.t. W y ≥ e max eT x (D)
y≥0 s.t. W x ≤ d
x≥0

University
c of Waterloo 9 / 20
 Primal-Dual Pairs

Example 2: To compute dual LP, check

right-hand side of table:
min dT y (P)
T
s.t. W y ≥ e
max eT x (D)
y≥0
s.t. W x ≤ d
x≥0
Substitute:
• d −→ c
• e −→ b
• y −→ x
• W T −→ A
• x −→ y

University
c of Waterloo 10 / 20
 Primal-Dual Pairs

Example 2: To compute dual LP, check

right-hand side of table:
min cT x (P)
s.t. Ax ≥ b
max bT x (D)
x≥0
s.t. AT y ≤ c
y≥0
Substitute:
• d −→ c
• e −→ b This is consistent with the earlier
discussion we had!
• y −→ x
• W T −→ A
• x −→ y

University
c of Waterloo 11 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Example 3: Its dual LP:

max (12, 26, 20)x (P) min (−2, 2, 13)y (D)

       
1 2 1 ≥ −2 1 4 2 12
s.t. 4 6 5 x ≤  2  s.t. 2 6 −1 y ? 26
2 −1 −3 = 13 1 5 −3 20
x1 ≥ 0, x2 free, x3 ≥ 0 y?0
University
c of Waterloo 12 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Example 3: Its dual LP:

max (12, 26, 20)x (P) min (−2, 2, 13)y (D)

       
1 2 1 ≥ −2 1 4 2 ≥ 12
s.t. 4 6 5 x ≤  2  s.t. 2 6 −1 y = 26
2 −1 −3 = 13 1 5 −3 ≥ 20
x1 ≥ 0, x2 free, x3 ≥ 0 y1 ≤ 0, y2 ≥ 0, y3 free
University
c of Waterloo 13 / 20
 Primal-Dual Pairs
The following table shows how constraints and variables in primal and
dual LPs correspond:

Theorem
Let (Pmax ) and (Pmin ) represent the above. If x̄ and ȳ are feasible for
the two LPs, then
cT x̄ ≤ bT ȳ
If cT x̄ = bT ȳ, then x̄ is optimal for (Pmax ), and ȳ is optimal for (Pmin ).

University
c of Waterloo 14 / 20
 Primal-Dual Pairs

Theorem
Let (Pmax ) and (Pmin ) represent the above. If x̄ and ȳ are feasible for
the two LPs, then
cT x̄ ≤ bT ȳ
If cT x̄ = bT ȳ, then x̄ is optimal for (Pmax ), and ȳ is optimal for (Pmin ).

Example 3 (continued): Its dual LP:

max (12, 26, 20)x (P) min (−2, 2, 13)y (D)

       
1 2 1 ≥ −2 1 4 2 ≥ 12
s.t. 4 6 5 x ≤  2  s.t. 2 6 −1 y = 26
2 −1 −3 = 13 1 5 −3 ≥ 20
x1 ≥ 0, x2 free, x3 ≥ 0 y1 ≤ 0, y2 ≥ 0, y3 free

Feasible solutions: x̄ = (5, −3, 0)T and ȳ = (0, 4, −2)T .

Since (12, 26, 20)x̄ = (−2, 2, 13)ȳ = −18 −→ both are optimal!

University
c of Waterloo 15 / 20
 Proving the General Weak Duality Theorem
General Primal LP:
max cT x
s.t. rowi (A)x ≤ bi (i ∈ R1 )
rowi (A)x ≥ bi (i ∈ R2 )
rowi (A)x = bi (i ∈ R3 )
xj ≥ 0 (j ∈ C1 )
xj ≤ 0 (j ∈ C2 )
xj free (j ∈ C3 )
University
c of Waterloo
Its dual according to the table:
min bT y
s.t. colj (A)T y ≥ cj (j ∈ C1 )
colj (A)T y ≤ cj (j ∈ C2 )
colj (A)T y = cj (j ∈ C3 )
yi ≥ 0 (i ∈ R1 )
yi ≤ 0 (i ∈ R2 )
yi free (i ∈ R3 )
16 / 20
 Proving the General Weak Duality Theorem

General Primal LP: Its dual according to the table:

max cT x
s.t. rowi (A)x ≤ bi (i ∈ R1 )
rowi (A)x ≥ bi (i ∈ R2 )
rowi (A)x = bi (i ∈ R3 )
xj ≥ 0 (j ∈ C1 )
xj ≤ 0 (j ∈ C2 )
xj free (j ∈ C3 )
min bT y
s.t. colj (A)T y ≥ cj (j ∈ C1 )
colj (A)T y ≤ cj (j ∈ C2 )
colj (A)T y = cj (j ∈ C3 )
yi ≥ 0 (i ∈ R1 )
yi ≤ 0 (i ∈ R2 )
yi free (i ∈ R3 )

We can rewrite the above LPs using slack variables!

University
c of Waterloo 17 / 20
 Proving the General Weak Duality Theorem

General Primal LP: Its dual according to the table:

max cT x min bT y
s.t. Ax + s = b s.t. AT y + w = c (?)
si ≥ 0 (i ∈ R1 ) wj ≤ 0 (j ∈ C1 )
si ≤ 0 (i ∈ R2 ) wj ≥ 0 (j ∈ C2 )
si = 0 (i ∈ R3 ) wj = 0 (j ∈ C3 )
xj ≥ 0 (j ∈ C1 ) yi ≥ 0 (i ∈ R1 )
xj ≤ 0 (j ∈ C2 ) yi ≤ 0 (i ∈ R2 )
xj free (j ∈ C3 ) yi free (i ∈ R3 )
Suppose x̄ and ȳ are feasible for the original primal and dual LPs
Let s̄ = b − Ax̄ and w̄ = c − AT ȳ. It follows that
(?)
ȳ T b = ȳ T (Ax̄ + s̄) = (ȳ T A)x̄ + ȳ T s̄ = (c − w̄)T x̄ + ȳ T s̄ = cT x̄ − w̄T x̄ + ȳ T s̄.

We can show that w̄T x̄ ≤ 0 and ȳ T s̄ ≥ 0 −→ ȳ T b ≥ cT x̄

University
c of Waterloo 18 / 20
 Consequences of Weak Duality

Theorem
Let (Pmax ) and (Pmin ) represent (i) (Pmax ) is unbounded −→
the above table. If x̄ and ȳ are (Pmin ) infeasible
feasible for the two LPs, then
(ii) (Pmin ) is unbounded −→
T T
c x̄ ≤ b ȳ (Pmax ) infeasible

(iii) (Pmax ) and (Pmin ) feasible

If cT x̄ = bT ȳ, then x̄ is optimal for
−→ both must have optimal
(Pmax ), and ȳ is optimal for
solutions
(Pmin ).
Proof: (i) Suppose, for a contradiction, that ȳ is feasible for (Pmin ).
By weak duality −→ cT x̄ ≤ bT ȳ for all x̄ feasible for (Pmax ), and hence
the latter is bounded.
(ii) Similar to (i)
(iii) weak duality −→ both (Pmax ) and (Pmin ) bounded
Fundamental Theorem of LP −→ Both LPs must have an optimal solution!

University
c of Waterloo 19 / 20
 Recap
• We can use the above table to compute duals of general LPs
• Weak duality theorem: if x̄ and ȳ are feasible for (Pmax ) and
(Pmin ), then:
cT x̄ ≤ bT ȳ
Both are optimal if equality holds!

University
c of Waterloo 20 / 20