Professional Documents
Culture Documents
minn c> x
x2R
s.t.Ax = b, (P)
x 0.
79
Obtaining a lower bound on the cost function
80
Finding best possible lower bound
maxb> y
y2Rm
s.t.A> y c. (D)
81
Properties
Theorem 7
For the primal-dual pair of optimization problems stated above, the following
are true.
1. If (P) is infeasible, and (D) is feasible, then (D) is unbounded.
2. If (P) is unbounded, then (D) is infeasible.
3. Weak Duality: For any feasible solution x̄ and ȳ of the respective
problems, we always have c> x̄ b> ȳ.
4. Strong Duality: Show that for the respective optimal solutions x⇤
and y ⇤ , we must have c> x⇤ = b> y ⇤ .
HW: Give an example of (P) and (D) where both are infeasible.
82
Proof
83
Farkas’ Lemma
To prove the strong duality theorem, we will make use of an alternative form of
Farka’s lemma.
Lemma 2 (Farkas’ Lemma). Let A 2 Rm⇥n and b 2 Rm . Then, exactly one
of the following sets must be empty:
1. {x 2 Rn | Ax = b, x 0}
2. {y 2 Rm | A> y 0, b> y > 0}.
Lemma 3 (Alternative form of Farkas’ Lemma). Let A 2 Rm⇥n and b 2 Rm .
Then, exactly one of the following sets must be empty:
1. {x 2 Rn |Ax b}
2. {y 2 Rm |y 0, y > A = 0, y > b < 0}.
84
Lagrangian Function
min f (x)
x2Rn
s.t. gi (x) 0, i 2 [m] := {1, 2, . . . , m},
hj (x) = 0, j 2 [p].
where
85
Lagrangian Dual
where ⇥ X X ⇤
d( , µ) := inf f (x) + i gi (x) + µi hj (x) .
x
i2[m] j2[p]
86
Lagrangian Dual Optimization Problem
max d( , µ)
2Rm ,µ2Rp
s.t. 0,
( , µ) 2 dom(d).
87
Example 1: Lagrangian Dual of LP
minn c> x
x2R
s.t. Ax = b, x 0.
88
Example 2: Least Norm Solution
Find L and d.
89
Weak and Strong Duality
90
Example 3
min x2
x2R
s.t. x 1 0, x 0.
Find the optimal value of the above problem, derive the dual and determine
whether strong duality holds.
91
Example 4
Find the optimal value of the above problem, derive the dual and determine
whether strong duality holds.
92
KKT Optimality Conditions
For the above primal and dual optimization problems, x̄, ¯ and µ̄ are said to
satisfy KKT optimality conditions if the following holds:
Primal Feasibility: gi (x̄) 0, i 2 [m], hj (x̄) = 0, j 2 [p].
Dual Feasibility: ¯ 0.
Complementary Slackness: ¯ i gi (x̄) = 0 for all i 2 [m].
P P
Stationarity: rx f (x̄) + i2[m] ¯ i rx gi (x̄) + j2[p] µ̄i rx hj (x̄) = 0.
93
Sufficient Condition for Optimality
Let x̄, ¯ and µ̄ satisfy KKT conditions stated above. From primal and dual
feasibility we have
⇥ X X ⇤
¯
d( , µ̄) = inf f (x) + ¯ i gi (x) + µ̄i hj (x)
x
i2[m] j2[p]
X X
f (x̄) + ¯ i gi (x̄) + µ̄i hj (x̄) f (x̄).
i2[m] j2[p]
94
Necessary and Sufficient Condition for Optimality
Theorem 8
Suppose the primal optimization problem is convex which satisfies Slater’s
constraint qualification condition: there exists x̄ 2 int(D) in the domain of
the optimization problem for which gi (x̄) < 0 for all i 2 [m] and hi (x̄) = 0
for all i 2 [p].
95
Convex Theorem of the Alternative
min f (x)
x2Rn
s.t. gi (x) 0, i 2 [m] := {1, 2, . . . , m},
Proof: Blackboard.
96
Strong Duality Theorem
min f (x)
x2Rn
s.t. gi (x) 0, i 2 [m] := {1, 2, . . . , m},
Proof: Blackboard.
97
Other notions of constraint qualification
If all the constraint functions gi (x) and hj (x) are affine, then constraint
qualification holds.
98