You are on page 1of 32

# Convex Optimization Boyd & Vandenberghe

## inequality constrained minimization

logarithmic barrier function and central path
barrier method
feasibility and phase I methods
complexity analysis via self-concordance
generalized inequalities

121
Inequality constrained minimization

minimize f0(x)
subject to fi(x) 0, i = 1, . . . , m (1)
Ax = b

## fi convex, twice continuously differentiable

A Rpn with rank A = p
we assume p is finite and attained
we assume problem is strictly feasible: there exists x with

Examples

## entropy maximization with linear inequality constraints

Pn
minimize i=1 xi log xi
subject to F x  g
Ax = b

## differentiability may require reformulating the problem, e.g.,

piecewise-linear minimization or -norm approximation via LP

## SDPs and SOCPs are better handled as problems with generalized

inequalities (see later)

## Interior-point methods 123

Logarithmic barrier
reformulation of (1) via indicator function:
Pm
minimize f0(x) + i=1 I(fi(x))
subject to Ax = b

## where I(u) = 0 if u 0, I(u) = otherwise (indicator function of R)

approximation via logarithmic barrier
Pm
minimize f0(x) (1/t) i=1 log(fi (x))
subject to Ax = b
10
an equality constrained problem
5
for t > 0, (1/t) log(u) is a
smooth approximation of I 0

approximation improves as t
5
3 2 1 0 1
u
Interior-point methods 124
logarithmic barrier function

m
X
(x) = log(fi(x)), dom = {x | f1(x) < 0, . . . , fm(x) < 0}
i=1

## convex (follows from composition rules)

twice continuously differentiable, with derivatives

m
X 1
(x) = fi(x)
i=1
f i (x)
m m
X 1 X 1
2(x) = 2
f i (x)f i (x) T
+ 2
fi(x)
f (x)
i=1 i i=1
fi(x)

Central path

## minimize tf0(x) + (x)

subject to Ax = b

(for now, assume x(t) exists and is unique for each t > 0)
central path is {x(t) | t > 0}

## example: central path for an LP c

minimize cT x
subject to aTi x bi, i = 1, . . . , 6
x(10)
x
T T
hyperplane c x = c x (t) is tangent to
level curve of through x(t)

## Interior-point methods 126

Dual points on central path
x = x(t) if there exists a w such that
m
X 1
tf0(x) + fi(x) + AT w = 0, Ax = b
i=1
fi(x)

## therefore, x(t) minimizes the Lagrangian

m
X
L(x, (t), (t)) = f0(x) + i(t)fi(x) + (t)T (Ax b)
i=1

## where we define i(t) = 1/(tfi(x(t)) and (t) = w/t

this confirms the intuitive idea that f0(x(t)) p if t :

p g((t), (t))
= L(x(t), (t), (t))
= f0(x(t)) m/t

## Interior-point methods 127

Interpretation via KKT conditions

## 1. primal constraints: fi(x) 0, i = 1, . . . , m, Ax = b

2. dual constraints:  0
3. approximate complementary slackness: ifi(x) = 1/t, i = 1, . . . , m
4. gradient of Lagrangian with respect to x vanishes:

m
X
f0(x) + ifi(x) + AT = 0
i=1

## Interior-point methods 128

Force field interpretation

## centering problem (for problem with no equality constraints)

Pm
minimize tf0(x) i=1 log(fi (x))

## tf0(x) is potential of force field F0(x) = tf0(x)

log(fi(x)) is potential of force field Fi(x) = (1/fi(x))fi(x)

## the forces balance at x(t):

m
X
F0(x(t)) + Fi(x(t)) = 0
i=1

## Interior-point methods 129

example
minimize cT x
subject to aTi x bi, i = 1, . . . , m

## objective force field is constant: F0(x) = tc

constraint force field decays as inverse distance to constraint hyperplane:
ai 1
Fi(x) = T
, kFi(x)k2 =
bi a i x dist(x, Hi)

3c
t=1 t=3

Barrier method

## given strictly feasible x, t := t(0) > 0, > 1, tolerance > 0.

repeat
1. Centering step. Compute x(t) by minimizing tf0 + , subject to Ax = b.
2. Update. x := x(t).
3. Stopping criterion. quit if m/t < .
4. Increase t. t := t.

## terminates with f0(x) p (stopping criterion follows from

f0(x(t)) p m/t)
centering usually done using Newtons method, starting at current x
choice of involves a trade-off: large means fewer outer iterations,
more inner (Newton) iterations; typical values: = 1020
several heuristics for choice of t(0)

## Interior-point methods 1211

Convergence analysis

log(m/(t(0)))
 

log

## plus the initial centering step (to compute x(t(0)))

centering problem

## see convergence analysis of Newtons method

tf0 + must have closed sublevel sets for t t(0)
classical analysis requires strong convexity, Lipschitz condition
analysis via self-concordance requires self-concordance of tf0 +

Examples

## inequality form LP (m = 100 inequalities, n = 50 variables)

102 140

Newton iterations
120
0
10
duality gap

100

102 80
60
4
10
40
106 = 50 = 150 =2 20
0
0 20 40 60 80 0 40 80 120 160 200
Newton iterations

## starts with x on central path (t(0) = 1, duality gap 100)

terminates when t = 108 (gap 106)
centering uses Newtons method with backtracking
total number of Newton iterations not very sensitive for 10

## Interior-point methods 1213

geometric program (m = 100 inequalities and n = 50 variables)
P 
5 T
minimize log k=1 exp(a0k x + b0k )
P 
5 T
subject to log k=1 exp(a ik x + bik ) 0, i = 1, . . . , m

102

100
duality gap

102

104

106 = 150 = 50 =2

0 20 40 60 80 100 120
Newton iterations

## Interior-point methods 1214

family of standard LPs (A Rm2m)

minimize cT x
subject to Ax = b, x0

## m = 10, . . . , 1000; for each m, solve 100 randomly generated instances

35
Newton iterations

30

25

20

15 1
10 102 103
m

## Interior-point methods 1215

Feasibility and phase I methods

## feasibility problem: find x such that

fi(x) 0, i = 1, . . . , m, Ax = b (2)

## phase I: computes strictly feasible starting point for barrier method

basic phase I method

minimize (over x, s) s
subject to fi(x) s, i = 1, . . . , m (3)
Ax = b

## if x, s feasible, with s < 0, then x is strictly feasible for (2)

if optimal value p of (3) is positive, then problem (2) is infeasible
if p = 0 and attained, then problem (2) is feasible (but not strictly);
if p = 0 and not attained, then problem (2) is infeasible

## Interior-point methods 1216

sum of infeasibilities phase I method

minimize 1T s
subject to s  0, fi(x) si, i = 1, . . . , m
Ax = b

## for infeasible problems, produces a solution that satisfies many more

inequalities than basic phase I method

## example (infeasible set of 100 linear inequalities in 50 variables)

60 60

40 40
number

number
20 20

0 0
1 0.5 0 0.5 1 1.5 1 0.5 0 0.5 1 1.5
bi aTi xmax bi aTi xsum

## left: basic phase I solution; satisfies 39 inequalities

right: sum of infeasibilities phase I solution; satisfies 79 inequalities

## Interior-point methods 1217

example: family of linear inequalities Ax  b + b
data chosen to be strictly feasible for > 0, infeasible for 0
use basic phase I, terminate when s < 0 or dual objective is positive

Newton iterations
100
80 Infeasible Feasible
60
40
20
0
1 0.5 0 0.5 1

Newton iterations

Newton iterations
100 100
80 80
60 60
40 40
20 20
0 0
100 102 104 106 106 104 2
100
10
number of iterations roughly proportional to log(1/||)

## Interior-point methods 1218

Complexity analysis via self-concordance

## sublevel sets (of f0, on the feasible set) are bounded

tf0 + is self-concordant with closed sublevel sets

second condition

## holds for LP, QP, QCQP

may require reformulating the problem, e.g.,
Pn Pn
minimize i=1 xi log xi minimize i=1 xi log xi
subject to F x  g subject to F x  g, x  0

## needed for complexity analysis; barrier method works even when

self-concordance assumption does not apply

## Interior-point methods 1219

Newton iterations per centering step: from self-concordance theory

## tf0(x) + (x) tf0(x+) (x+)

#Newton iterations +c

## bound on effort of computing x+ = x(t) starting at x = x(t)

, c are constants (depend only on Newton algorithm parameters)
from duality (with = (t), = (t)):

## tf0(x) + (x) tf0(x+) (x+)

Xm
= tf0(x) tf0(x+) + log(tifi(x+)) m log
i=1
m
X
tf0(x) tf0(x+) t ifi(x+) m m log
i=1
tf0(x) tg(, ) m m log
= m( 1 log )

## Interior-point methods 1220

total number of Newton iterations (excluding first centering step)

(0)
  
log(m/(t )) m( 1 log )
#Newton iterations N = +c
log

5 104

4 104
figure shows N for typical values of , c,
3 104
m
N

4 m = 100, = 105
2 10 t(0)
1 104

0
1 1.1 1.2

## confirms trade-off in choice of

in practice, #iterations is in the tens; not very sensitive for 10

## Interior-point methods 1221

polynomial-time complexity of barrier method

for = 1 + 1/ m:

m/t(0)
  
N =O m log

number of Newton iterations for fixed gap reduction is O( m)

## multiply with cost of one Newton iteration (a polynomial function of

problem dimensions), to get bound on number of flops

## this choice of optimizes worst-case complexity; in practice we choose

fixed ( = 10, . . . , 20)

## Interior-point methods 1222

Generalized inequalities

minimize f0(x)
subject to fi(x) Ki 0, i = 1, . . . , m
Ax = b

## f0 convex, fi : Rn Rki , i = 1, . . . , m, convex with respect to proper

cones Ki Rki
fi twice continuously differentiable
A Rpn with rank A = p
we assume p is finite and attained
we assume problem is strictly feasible; hence strong duality holds and
dual optimum is attained

## Interior-point methods 1223

Generalized logarithm for proper cone

## dom = int K and 2(y) 0 for y K 0

(sy) = (y) + log s for y K 0, s > 0 ( is the degree of )

examples
Pn
nonnegative orthant K = Rn+: (y) = i=1 log yi , with degree = n
positive semidefinite cone K = Sn+:

(Y ) = log det Y ( = n)

## second-order cone K = {y Rn+1 | (y12 + + yn2 )1/2 yn+1}:

2
(y) = log(yn+1 y12 yn2 ) ( = 2)

## Interior-point methods 1224

properties (without proof): for y K 0,

(y) K 0, y T (y) =

Pn
nonnegative orthant Rn+: (y) = i=1 log yi

## positive semidefinite cone Sn+: (Y ) = log det Y

(Y ) = Y 1, tr(Y (Y )) = n

y1
2 .. T
(y) = 2 2
, y (y) = 2
2
yn+1 y1 yn y n

yn+1

## Interior-point methods 1225

Logarithmic barrier and central path

## logarithmic barrier for f1(x) K1 0, . . . , fm(x) Km 0:

m
X
(x) = i(fi(x)), dom = {x | fi(x) Ki 0, i = 1, . . . , m}
i=1

## i is generalized logarithm for Ki, with degree i

is convex, twice continuously differentiable

## minimize tf0(x) + (x)

subject to Ax = b

## Interior-point methods 1226

Dual points on central path
x = x(t) if there exists w Rp,
m
X
tf0(x) + Dfi(x)T i(fi(x)) + AT w = 0
i=1

## therefore, x(t) minimizes Lagrangian L(x, (t), (t)), where

1 w
i(t) = i(fi(x(t))),
(t) =
t t

## from properties of i: i(t) Ki 0, with duality gap

m
X
f0(x(t)) g((t), (t)) = (1/t) i
i=1

## Interior-point methods 1227

example: semidefinite programming (with Fi Sp)

minimize cT x Pn
subject to F (x) = i=1 xiFi + G  0

## logarithmic barrier: (x) = log det(F (x)1)

central path: x(t) minimizes tcT x log det(F (x)); hence

## dual point on central path: Z (t) = (1/t)F (x(t))1 is feasible for

maximize tr(GZ)
subject to tr(FiZ) + ci = 0, i = 1, . . . , n
Z0

Barrier method

## given strictly feasible x, t := t(0) > 0, > 1, tolerance > 0.

repeat
1. Centering step. Compute x(t) by minimizing tf0 + , subject to Ax = b.
2. Update. x := x(t).
P
3. Stopping criterion. quit if ( i i)/t < .
4. Increase t. t := t.

P
only difference is duality gap m/t on central path is replaced by i i /t

& '
(0)
P
log(( i i)/(t ))
log

Examples

## second-order cone program (50 variables, 50 SOC constraints in R6)

102

Newton iterations
120
duality gap

0
10

102 80

104 40
106 = 50 = 200 =2
0
0 20 40 60 80 20 60 100 140 180
Newton iterations

## semidefinite program (100 variables, LMI constraint in S100)

140
2
10
Newton iterations
duality gap

100 100

102
60
4
10

106 = 150 = 50 =2 20
0 20 40 60 80 100 0 20 40 60 80 100 120
Newton iterations
Interior-point methods 1230
family of SDPs (A Sn, x Rn)

minimize 1T x
subject to A + diag(x)  0

## n = 10, . . . , 1000, for each n solve 100 randomly generated instances

35
Newton iterations

30

25

20

15
101 102 103
n

## Interior-point methods 1231

Primal-dual interior-point methods

## update primal and dual variables at each iteration; no distinction

between inner and outer iterations
often exhibit superlinear asymptotic convergence
search directions can be interpreted as Newton directions for modified
KKT conditions
can start at infeasible points
cost per iteration same as barrier method