Professional Documents
Culture Documents
A special case of the NLP arises when the objective functional f is quadratic
and the constraints h, g are linear in x ∈ lRn . Such an NLP is called a Quadratic
Programming (QP) problem. Its general form is
1 T
minimize f (x) := x Bx − xT b (3.1a)
2
over x ∈ lRn
subject to A1 x = c (3.1b)
A2 x ≤ d , (3.1c)
where A ∈ lRm×n , m ≤ n. For the time being we assume that A has full row
rank m.
The KKT conditions for the solution x∗ ∈ lRn of the QP (3.2a),(3.2b) give rise
to the following linear system
µ ¶µ ∗ ¶ µ ¶
B AT x b
∗ = , (3.3)
A 0 λ c
| {z }
=: K
pT Bx∗ = pT (b − AT λ∗ ) = pT b − (Ap)T λ∗ ,
| {z }
= 0
whence
1 T
f (x) = p Bp + f (x∗ ) .
2
In view of p ∈ Ker A, we can write p = Zu , u ∈ lRn−m , and hence,
1 T T
f (x) = u Z BZu + f (x∗ ) .
2
Since Z T BZ is positive definite, we deduce f (x) > f (x∗ ). Consequently, x∗ is
the unique global minimizer of the QP (3.2a),(3.2b).
•
Optimization I; Chapter 3 58
P T KP = LDLT , (3.4)
AB −1 AT λ∗ = AB −1 b − c (3.6)
x∗ = Y wY + ZwZ , (3.7)
Ax∗ = AY wY + |{z}
AZ wZ = c , (3.8)
= 0
BY wY + BZwZ + AT λ∗ = b .
Z T BZwZ = Z T b − Z T BY wY . (3.9)
The reduced KKT system (3.9) can be solved by a Cholesky factorization of the
reduced Hessian Z T BZ ∈ lR(n−m)×(n−m) . Once wY and wZ have been computed
as the solutions of (3.8) and (3.9), x∗ is obtained according to (3.7).
Finally, the Lagrange multiplier turns out to be the solution of the linear system
arising from the multiplication of the first equation in (3.7) by Y T :
S̃ := − AB̃ −1 AT . (3.13)
We set
We rearrange (3.18) by exchanging the second and third rows and columns
∗
B11 AT11 | B12 AT21 x1 b1
A11 0 | A12 0 λ∗1 c1
−− −− | −− −− − = − . (3.19)
B21 AT12 | B22 AT22 x∗2 b2
A21 0 | A22 0 λ∗2 c2
T
Observing B12 = B21 , in block form (3.19) can be written as follows
µ ¶µ ∗ ¶ µ ¶
A BT ψ1 α1
= , (3.20)
B D ψ2∗ α2
| {z } | {z } | {z }
=: K =: ψ ∗ =: α
S = D − BA−1 B T , (3.22)
Optimization I; Chapter 3 62
where
µ ¶
−1 0 A−1
11
A = . (3.23)
A−T
11 −A−T
11 B −1
11 A11
We assume that Ã11 ∈ lRm1 ×m1 is an easily invertible approximation of A11 and
define
µ ¶
B11 ÃT11
à = . (3.24)
Ã11 0
S̃ := D − B Ã−1 B T . (3.27)
(0) (0)
Given a start iterate ψ (0) = (ψ1 , ψ2 )T , we solve the KKT system (3.20) by
the transforming null-space iterations
we find
1 T
f (x) = f (x(ν) − p) = p Bp − (b(ν) )T p + g ,
2
where g := 12 (x(ν) )T Bx(ν) − bT x(ν) .
The equality constrained QP problem to be solved at the (ν+1)-st iteration
step is then:
1 T
minimize p Bp − (b(ν) )T p (3.34a)
2
over p ∈ lRn
subject to Cp = 0 (3.34b)
aTi p = 0 , i ∈ Iac (x(ν) ) , (3.34c)
We denote the solution of (3.34a)-(3.34c) by p(ν) . The new iterate x(ν+1) is then
obtained according to
aTi x(ν) − di
aTi x(ν+1) = aTi x(ν) − αν aTi p(ν) ≤ di ⇐⇒ αν ≤ .
aTi p(ν)
aTi x(ν) − di
αν := min (1, min ). (3.36)
/ ax (x(ν) )
i∈I aTi p(ν)
aT
i p
(ν) <0
We define
aiT x(ν)−di
Ib` (p(ν) ) := {i ∈
/ Iac (x(ν) ) | aTi p(ν) < 0 , min T (ν)
< 1} . (3.37)
/ ax (x(ν) ) ai p
i∈I
Clearly,
¡ ¢
aTi x(ν+1) = aTi x(ν) − αν p(ν) = di , i ∈ Ib` (x(ν) ) .
If we set
(ν)
µi := 0 , i ∈ {1, ..., p} \ Iac (x(ν) ) ,
Optimization I; Chapter 3 65
it is clear that x(ν) , λ(ν) , and µ(ν) satisfy the first KKT condition with respect
to the original QP problem (3.29a)-(3.29c).
Since x(ν) is feasible, the second and third KKT conditions also hold true.
We check the fourth KKT condition in terms of the sign of the multiplier µ(ν) :
If
(ν)
µi ≥ 0 , i ∈ Iac (x(ν) ) ,
the fourth KKT condition holds true. Consequently, x(ν) is a strict local mini-
mum, since B is symmetric positive definite.
On the other hand, if there exists j ∈ Iac (x(ν) ) such that
(ν)
µj < 0,
we remove that constraint from the active set. We show that this strategy
produces a direction p in the subsequent iteration step that is feasible with
respect to the dropped constraint:
Theorem 3.4 Feasible and descent direction for active set strategy
We assume that x̂ ∈ lRn satisfies the KKT conditions for the QP problem
(3.34a)-(3.34c), i.e., in particular (3.39) holds true. Further, assume that the
constraint gradients ci , 1 ≤ i ≤ m, ai , i ∈ Iac (x̂) are linearly independent. Fi-
nally, suppose there is j ∈ Iac (x̂) such that µ̂j < 0.
Let p is the solution of the QP problem
1 T
minimize p Bp − b̂T p (3.40a)
2
over p ∈ lRn
subject to Cp = 0 (3.40b)
aTi p = 0 , i ∈ Iac (x̂) \ {j} . (3.40c)
where b̂ := B x̂ − b.
Then, p is a feasible direction for the constraint j, i.e.,
aTj p ≤ 0 .
We consider a primal-dual active set strategy which does not require feasibility
of the iterates. It is based on a Moreau-Yosida type approximation of the
indicator function of the convex set of inequality constraints
K := {v ∈ lRp | vi ≤ 0 , 1 ≤ i ≤ p} . (3.48)
The indicator function IK : K → lR of K is given by
½
0 , v∈K
IK (v) := . (3.49)
+∞ , v ∈ /K
Optimization I; Chapter 3 69
aTi x∗ − di ≤ 0 , µ∗i ≥ 0 ,
µ∗i (aTi x∗ − di ) = 0 , 1 ≤ i ≤ p
problem
1 T
minimize Q(x) := x Bx − xT b (3.58a)
2
over x ∈ lRn
subject to cTi x = ci , 1 ≤ i ≤ m , (3.58b)
aTi x = di , i ∈ Iac (x ) . (ν)
(3.58c)
Since feasibility is not required, any startiterate (x(0) , λ(0) , µ(0) ) can be chosen.
The following results show that we can expect convergence of the primal-dual
active set strategy, provided the matrix B is symmetric positive definite.
Theorem 3.3 Reduction of the objective functional
Assume B ∈ lRn×n to be symmetric, positive definite and refer to k · kE :=
((·)T B(·))1/2 as the associated energy norm. Let x(ν) , ν ≥ 0, be the iterates
generated by the primal-dual active set strategy. Then, for Q(x) := 12 xT Bx−xT b
and ν ≥ 1 there holds
Q(x(ν) ) − Q(x(ν−1) ) =
1
= − kx(ν) − x(ν−1) k2E + (x(ν) − x(ν−1) )T (Bx(ν) − b) =
2
1 X (ν)
= − kx(ν) − x(ν−1) k2E − µi aTi (x(ν) − x(ν−1) ) .
2 (ν) i∈Iac (x )
For i ∈ Iac (x(ν) ) we have aTi x(ν) = di , whereas aTi x(ν−1) = di for i ∈ Iac (x(ν−1) ).
we thus get
Q(x(ν) ) − Q(x(ν−1) ) =
1 X (ν)
= − kx(ν) − x(ν−1) k2E − µi (di − aTi x(ν−1) ) .
2
i∈Iac (x(ν) )
/ ac (x(ν−1) )
i∈I
(ν)
But µi ≥ 0, i ∈ Iac (x(ν) ) and aTi x(ν−1) ≤ di , i ∈
/ Iac (x(ν−1) ) which gives the
assertion.
•
Corollary 3.4 Convergence of a subsequence to a local minimum
Optimization I; Chapter 3 71
Let x(ν) , ν ≥ 0, be the iterates generated by the primal-dual active set strategy.
Then, there exist a subsequence lN0 ⊂ lN and x∗ ∈ lRn such that x(ν) → x∗ , ν →
∞, ν ∈ lN0 . Moreover, x∗ is a local minimizer of (3.29a)-(3.29c).
Proof. The sequence (x(ν) )lN is bounded, since otherwise we would have
Q(x(ν) ) → +∞ as ν → ∞ in contrast to the result of the previous theo-
rem. Consequently, there exist a subsequence lN0 ⊂ lN and x∗ ∈ lRn such
that x(ν) → x∗ , ν → ∞, ν ∈ lN0 . Passing to the limit in the KKT system for
(3.58a)-(3.58c) shows that x∗ is a local minimizer.
•
The following result gives an a priori estimate in the energy norm.
Theorem 3.5 A priori error estimate in the energy norm
Let x(ν) , ν ≥ 0, be the iterates generated by the primal-dual active set strategy
and lN0 ⊂ lN such that x(ν) → x∗ , ν → ∞, ν ∈ lN0 . Then, there holds
X
kx(ν) − x∗ k2E ≤ 2 µ∗i (di − aTi x(ν) ) . (3.60)
i∈Iac (x∗ )
/ ac (x(ν) )
i∈I
1 T
minimize Q(x) := x Bx − xT b (3.61a)
2
over x ∈ lRn
subject to Ax ≤ d, (3.61b)
Bx∗ + AT µ∗ − b = 0, (3.62a)
Ax∗ − d ≤ 0, (3.62b)
µ∗i (Ax∗ − d)i = 0, 1≤i≤p, (3.62c)
µ∗i ≥ 0, 1≤i≤p. (3.62d)
Optimization I; Chapter 3 72
Bx∗ + AT µ∗ − b = 0, (3.63a)
Ax∗ + z ∗ − d = 0, (3.63b)
µ∗i zi = 0, 1≤i≤p, (3.63c)
zi∗ , µ∗i ≥ 0, 1≤i≤p. (3.63d)
rb := Bx + AT µ − b , rd := Ax + z − d .
We will have a closer look at the relationship between a minimizer of B (β) (x)
and a point (x, µ) satisfying the KKT conditions for (3.62a)-(3.62b).
If x(β) is a minimizer of B (β) (x), we obviously have
p
X β
(β)
∇x B (x(β)) = ∇x Q(x(β)) + ai = 0 . (3.72)
i=1
di − aTi x(β)
(β) β
zi := , 1≤i≤p (3.73)
di − aTi x(β)
We have to compare (3.74) with the first of the KKT conditions for (3.62a)-
(3.62b) which is given by
p
X
∇x L(x, µ) = ∇x Q(x) + µi ai = 0 . (3.75)
i=1
aTi x − di ≤ 0 , 1 ≤ i ≤ p , (3.76a)
µi ≥ 0 , 1 ≤ i ≤ p , (3.76b)
µi (aTi x − di ) = 0 , 1 ≤ i ≤ p . (3.76c)
On the other hand, as β → 0 a minimizer x(β) and the associated z (β) come
closer and closer to satisfying (3.76c). This is reason why z (β) is called perturbed
(approximate) complementarity.
Optimization I; Chapter 3 76
z (β) → µ∗ as β → 0 . (3.78)
(iii) For sufficiently small β > 0, the Hessian ∇xx B (β) (x) is positive definite.
Proof. We refer to M.H. Wright; Interior methods for constrained optimiza-
tion. Acta Numerica, 341-407, 1992.
•