Professional Documents
Culture Documents
= inff
0
(x) [ f
i
(x) 0, i = 1, . . . , m, h
i
(x) = 0, i = 1, . . . , p
p
; X
opt
is the set of optimal points
x is locally optimal if there is an R > 0 such that x is optimal for
minimize (over z) f
0
(z)
subject to f
i
(z) 0, i = 1, . . . , m, h
i
(z) = 0, i = 1, . . . , p
|z x|
2
R
examples (with n = 1, m = p = 0)
f
0
(x) = 1/x, domf
0
= R
++
: p
= 0, no optimal point
f
0
(x) = log x, domf
0
= R
++
: p
=
f
0
(x) = xlog x, domf
0
= R
++
: p
= , local optimum at x = 1
Convex optimization problems 43
Implicit constraints
the standard form optimization problem has an implicit constraint
x T =
m
i=0
domf
i
p
i=1
domh
i
,
we call T the domain of the problem
the constraints f
i
(x) 0, h
i
(x) = 0 are the explicit constraints
a problem is unconstrained if it has no explicit constraints (m = p = 0)
example:
minimize f
0
(x) =
k
i=1
log(b
i
a
T
i
x)
is an unconstrained problem with implicit constraints a
T
i
x < b
i
Convex optimization problems 44
Feasibility problem
nd x
subject to f
i
(x) 0, i = 1, . . . , m
h
i
(x) = 0, i = 1, . . . , p
can be considered a special case of the general problem with f
0
(x) = 0:
minimize 0
subject to f
i
(x) 0, i = 1, . . . , m
h
i
(x) = 0, i = 1, . . . , p
p
t
(x) 0, f
i
(x) 0, i = 1, . . . , m, Ax = b (1)
for xed t, a convex feasibility problem in x
if feasible, we can conclude that t p
; if infeasible, t p
, u p
, tolerance > 0.
repeat
1. t := (l + u)/2.
2. Solve the convex feasibility problem (1).
3. if (1) is feasible, u := t; else l := t.
until u l .
requires exactly log
2
((u l)/) iterations (where u, l are initial values)
Convex optimization problems 416
Linear program (LP)
minimize c
T
x + d
subject to Gx _ h
Ax = b
convex problem with ane objective and constraint functions
feasible set is a polyhedron
P
x
c
Convex optimization problems 417
Examples
diet problem: choose quantities x
1
, . . . , x
n
of n foods
one unit of food j costs c
j
, contains amount a
ij
of nutrient i
healthy diet requires nutrient i in quantity at least b
i
to nd cheapest healthy diet,
minimize c
T
x
subject to Ax _ b, x _ 0
piecewise-linear minimization
minimize max
i=1,...,m
(a
T
i
x + b
i
)
equivalent to an LP
minimize t
subject to a
T
i
x + b
i
t, i = 1, . . . , m
Convex optimization problems 418
Chebyshev center of a polyhedron
Chebyshev center of
T = x [ a
T
i
x b
i
, i = 1, . . . , m
is center of largest inscribed ball
B = x
c
+ u [ |u|
2
r
x
cheb
x
cheb
a
T
i
x b
i
for all x B if and only if
supa
T
i
(x
c
+ u) [ |u|
2
r = a
T
i
x
c
+ r|a
i
|
2
b
i
hence, x
c
, r can be determined by solving the LP
maximize r
subject to a
T
i
x
c
+ r|a
i
|
2
b
i
, i = 1, . . . , m
Convex optimization problems 419
(Generalized) linear-fractional program
minimize f
0
(x)
subject to Gx _ h
Ax = b
linear-fractional program
f
0
(x) =
c
T
x + d
e
T
x + f
, domf
0
(x) = x [ e
T
x + f > 0
a quasiconvex optimization problem; can be solved by bisection
also equivalent to the LP (variables y, z)
minimize c
T
y + dz
subject to Gy _ hz
Ay = bz
e
T
y + fz = 1
z 0
Convex optimization problems 420
generalized linear-fractional program
f
0
(x) = max
i=1,...,r
c
T
i
x + d
i
e
T
i
x + f
i
, domf
0
(x) = x [ e
T
i
x+f
i
> 0, i = 1, . . . , r
a quasiconvex optimization problem; can be solved by bisection
example: Von Neumann model of a growing economy
maximize (over x, x
+
) min
i=1,...,n
x
+
i
/x
i
subject to x
+
_ 0, Bx
+
_ Ax
x, x
+
R
n
: activity levels of n sectors, in current and next period
(Ax)
i
, (Bx
+
)
i
: produced, resp. consumed, amounts of good i
x
+
i
/x
i
: growth rate of sector i
allocate activity to maximize growth rate of slowest growing sector
Convex optimization problems 421
Quadratic program (QP)
minimize (1/2)x
T
Px + q
T
x + r
subject to Gx _ h
Ax = b
P S
n
+
, so objective is convex quadratic
minimize a convex quadratic function over a polyhedron
P
x
f
0
(x
)
Convex optimization problems 422
Examples
least-squares
minimize |Ax b|
2
2
analytical solution x
= A
b (A
is pseudo-inverse)
can add linear constraints, e.g., l _ x _ u
linear program with random cost
minimize c
T
x + x
T
x = Ec
T
x + var(c
T
x)
subject to Gx _ h, Ax = b
c is random vector with mean c and covariance
hence, c
T
x is random variable with mean c
T
x and variance x
T
x
> 0 is risk aversion parameter; controls the trade-o between
expected cost and variance (risk)
Convex optimization problems 423
Quadratically constrained quadratic program (QCQP)
minimize (1/2)x
T
P
0
x + q
T
0
x + r
0
subject to (1/2)x
T
P
i
x + q
T
i
x + r
i
0, i = 1, . . . , m
Ax = b
P
i
S
n
+
; objective and constraints are convex quadratic
if P
1
, . . . , P
m
S
n
++
, feasible region is intersection of m ellipsoids and
an ane set
Convex optimization problems 424
Second-order cone programming
minimize f
T
x
subject to |A
i
x + b
i
|
2
c
T
i
x + d
i
, i = 1, . . . , m
Fx = g
(A
i
R
n
i
n
, F R
pn
)
inequalities are called second-order cone (SOC) constraints:
(A
i
x + b
i
, c
T
i
x + d
i
) second-order cone in R
n
i
+1
for n
i
= 0, reduces to an LP; if c
i
= 0, reduces to a QCQP
more general than QCQP and LP
Convex optimization problems 425
Robust linear programming
the parameters in optimization problems are often uncertain, e.g., in an LP
minimize c
T
x
subject to a
T
i
x b
i
, i = 1, . . . , m,
there can be uncertainty in c, a
i
, b
i
two common approaches to handling uncertainty (in a
i
, for simplicity)
deterministic model: constraints must hold for all a
i
c
i
minimize c
T
x
subject to a
T
i
x b
i
for all a
i
c
i
, i = 1, . . . , m,
stochastic model: a
i
is random variable; constraints must hold with
probability
minimize c
T
x
subject to prob(a
T
i
x b
i
) , i = 1, . . . , m
Convex optimization problems 426
deterministic approach via SOCP
choose an ellipsoid as c
i
:
c
i
= a
i
+ P
i
u [ |u|
2
1 ( a
i
R
n
, P
i
R
nn
)
center is a
i
, semi-axes determined by singular values/vectors of P
i
robust LP
minimize c
T
x
subject to a
T
i
x b
i
a
i
c
i
, i = 1, . . . , m
is equivalent to the SOCP
minimize c
T
x
subject to a
T
i
x +|P
T
i
x|
2
b
i
, i = 1, . . . , m
(follows from sup
u
2
1
( a
i
+ P
i
u)
T
x = a
T
i
x +|P
T
i
x|
2
)
Convex optimization problems 427
stochastic approach via SOCP
assume a
i
is Gaussian with mean a
i
, covariance
i
(a
i
^( a
i
,
i
))
a
T
i
x is Gaussian r.v. with mean a
T
i
x, variance x
T
i
x; hence
prob(a
T
i
x b
i
) =
_
b
i
a
T
i
x
|
1/2
i
x|
2
_
where (x) = (1/
2)
_
x
e
t
2
/2
dt is CDF of ^(0, 1)
robust LP
minimize c
T
x
subject to prob(a
T
i
x b
i
) , i = 1, . . . , m,
with 1/2, is equivalent to the SOCP
minimize c
T
x
subject to a
T
i
x +
1
()|
1/2
i
x|
2
b
i
, i = 1, . . . , m
Convex optimization problems 428
Geometric programming
monomial function
f(x) = cx
a
1
1
x
a
2
2
x
a
n
n
, domf = R
n
++
with c > 0; exponent
i
can be any real number
posynomial function: sum of monomials
f(x) =
K
k=1
c
k
x
a
1k
1
x
a
2k
2
x
a
nk
n
, domf = R
n
++
geometric program (GP)
minimize f
0
(x)
subject to f
i
(x) 1, i = 1, . . . , m
h
i
(x) = 1, i = 1, . . . , p
with f
i
posynomial, h
i
monomial
Convex optimization problems 429
Geometric program in convex form
change variables to y
i
= log x
i
, and take logarithm of cost, constraints
monomial f(x) = cx
a
1
1
x
a
n
n
transforms to
log f(e
y
1
, . . . , e
y
n
) = a
T
y + b (b = log c)
posynomial f(x) =
K
k=1
c
k
x
a
1k
1
x
a
2k
2
x
a
nk
n
transforms to
log f(e
y
1
, . . . , e
y
n
) = log
_
K
k=1
e
a
T
k
y+b
k
_
(b
k
= log c
k
)
geometric program transforms to convex problem
minimize log
_
K
k=1
exp(a
T
0k
y + b
0k
)
_
subject to log
_
K
k=1
exp(a
T
ik
y + b
ik
)
_
0, i = 1, . . . , m
Gy + d = 0
Convex optimization problems 430
Design of cantilever beam
F
segment 4 segment 3 segment 2 segment 1
N segments with unit lengths, rectangular cross-sections of size w
i
h
i
given vertical force F applied at the right end
design problem
minimize total weight
subject to upper & lower bounds on w
i
, h
i
upper bound & lower bounds on aspect ratios h
i
/w
i
upper bound on stress in each segment
upper bound on vertical deection at the end of the beam
variables: w
i
, h
i
for i = 1, . . . , N
Convex optimization problems 431
objective and constraint functions
total weight w
1
h
1
+ + w
N
h
N
is posynomial
aspect ratio h
i
/w
i
and inverse aspect ratio w
i
/h
i
are monomials
maximum stress in segment i is given by 6iF/(w
i
h
2
i
), a monomial
the vertical deection y
i
and slope v
i
of central axis at the right end of
segment i are dened recursively as
v
i
= 12(i 1/2)
F
Ew
i
h
3
i
+ v
i+1
y
i
= 6(i 1/3)
F
Ew
i
h
3
i
+ v
i+1
+ y
i+1
for i = N, N 1, . . . , 1, with v
N+1
= y
N+1
= 0 (E is Youngs modulus)
v
i
and y
i
are posynomial functions of w, h
Convex optimization problems 432
formulation as a GP
minimize w
1
h
1
+ + w
N
h
N
subject to w
1
max
w
i
1, w
min
w
1
i
1, i = 1, . . . , N
h
1
max
h
i
1, h
min
h
1
i
1, i = 1, . . . , N
S
1
max
w
1
i
h
i
1, S
min
w
i
h
1
i
1, i = 1, . . . , N
6iF
1
max
w
1
i
h
2
i
1, i = 1, . . . , N
y
1
max
y
1
1
note
we write w
min
w
i
w
max
and h
min
h
i
h
max
w
min
/w
i
1, w
i
/w
max
1, h
min
/h
i
1, h
i
/h
max
1
we write S
min
h
i
/w
i
S
max
as
S
min
w
i
/h
i
1, h
i
/(w
i
S
max
) 1
Convex optimization problems 433
Minimizing spectral radius of nonnegative matrix
Perron-Frobenius eigenvalue
pf
(A)
exists for (elementwise) positive A R
nn
a real, positive eigenvalue of A, equal to spectral radius max
i
[
i
(A)[
determines asymptotic growth (decay) rate of A
k
: A
k
k
pf
as k
alternative characterization:
pf
(A) = inf [ Av _ v for some v 0
minimizing spectral radius of matrix of posynomials
minimize
pf
(A(x)), where the elements A(x)
ij
are posynomials of x
equivalent geometric program:
minimize
subject to
n
j=1
A(x)
ij
v
j
/(v
i
) 1, i = 1, . . . , n
variables , v, x
Convex optimization problems 434
Generalized inequality constraints
convex problem with generalized inequality constraints
minimize f
0
(x)
subject to f
i
(x) _
K
i
0, i = 1, . . . , m
Ax = b
f
0
: R
n
R convex; f
i
: R
n
R
k
i
K
i
-convex w.r.t. proper cone K
i
same properties as standard convex problem (convex feasible set, local
optimum is global, etc.)
conic form problem: special case with ane objective and constraints
minimize c
T
x
subject to Fx + g _
K
0
Ax = b
extends linear programming (K = R
m
+
) to nonpolyhedral cones
Convex optimization problems 435
Semidenite program (SDP)
minimize c
T
x
subject to x
1
F
1
+ x
2
F
2
+ + x
n
F
n
+ G _ 0
Ax = b
with F
i
, G S
k
inequality constraint is called linear matrix inequality (LMI)
includes problems with multiple LMI constraints: for example,
x
1
F
1
+ + x
n
F
n
+
G _ 0, x
1
F
1
+ + x
n
F
n
+
G _ 0
is equivalent to single LMI
x
1
_
F
1
0
0
F
1
_
+x
2
_
F
2
0
0
F
2
_
+ +x
n
_
F
n
0
0
F
n
_
+
_
G 0
0
G
_
_ 0
Convex optimization problems 436
LP and SOCP as SDP
LP and equivalent SDP
LP: minimize c
T
x
subject to Ax _ b
SDP: minimize c
T
x
subject to diag(Ax b) _ 0
(note dierent interpretation of generalized inequality _)
SOCP and equivalent SDP
SOCP: minimize f
T
x
subject to |A
i
x + b
i
|
2
c
T
i
x + d
i
, i = 1, . . . , m
SDP: minimize f
T
x
subject to
_
(c
T
i
x + d
i
)I A
i
x + b
i
(A
i
x + b
i
)
T
c
T
i
x + d
i
_
_ 0, i = 1, . . . , m
Convex optimization problems 437
Eigenvalue minimization
minimize
max
(A(x))
where A(x) = A
0
+ x
1
A
1
+ + x
n
A
n
(with given A
i
S
k
)
equivalent SDP
minimize t
subject to A(x) _ tI
variables x R
n
, t R
follows from
max
(A) t A _ tI
Convex optimization problems 438
Matrix norm minimization
minimize |A(x)|
2
=
_
max
(A(x)
T
A(x))
_
1/2
where A(x) = A
0
+ x
1
A
1
+ + x
n
A
n
(with given A
i
R
pq
)
equivalent SDP
minimize t
subject to
_
tI A(x)
A(x)
T
tI
_
_ 0
variables x R
n
, t R
constraint follows from
|A|
2
t A
T
A _ t
2
I, t 0
_
tI A
A
T
tI
_
_ 0
Convex optimization problems 439
Vector optimization
general vector optimization problem
minimize (w.r.t. K) f
0
(x)
subject to f
i
(x) 0, i = 1, . . . , m
h
i
(x) 0, i = 1, . . . , p
vector objective f
0
: R
n
R
q
, minimized w.r.t. proper cone K R
q
convex vector optimization problem
minimize (w.r.t. K) f
0
(x)
subject to f
i
(x) 0, i = 1, . . . , m
Ax = b
with f
0
K-convex, f
1
, . . . , f
m
convex
Convex optimization problems 440
Optimal and Pareto optimal points
set of achievable objective values
O = f
0
(x) [ x feasible
feasible x is optimal if f
0
(x) is a minimum value of O
feasible x is Pareto optimal if f
0
(x) is a minimal value of O
O
f
0
(x
)
x
is optimal
O
f
0
(x
po
)
x
po
is Pareto optimal
Convex optimization problems 441
Multicriterion optimization
vector optimization problem with K = R
q
+
f
0
(x) = (F
1
(x), . . . , F
q
(x))
q dierent objectives F
i
; roughly speaking we want all F
i
s to be small
feasible x
is optimal if
y feasible = f
0
(x
) _ f
0
(y)
if there exists an optimal point, the objectives are noncompeting
feasible x
po
is Pareto optimal if
y feasible, f
0
(y) _ f
0
(x
po
) = f
0
(x
po
) = f
0
(y)
if there are multiple Pareto optimal values, there is a trade-o between
the objectives
Convex optimization problems 442
Regularized least-squares
minimize (w.r.t. R
2
+
) (|Ax b|
2
2
, |x|
2
2
)
0 10 20 30 40 50
0
5
10
15
20
25
F
1
(x) = |Ax b|
2
2
F
2
(
x
)
=
|
x
|
22O
example for A R
10010
; heavy line is formed by Pareto optimal points
Convex optimization problems 443
Risk return trade-o in portfolio optimization
minimize (w.r.t. R
2
+
) ( p
T
x, x
T
x)
subject to 1
T
x = 1, x _ 0
x R
n
is investment portfolio; x
i
is fraction invested in asset i
p R
n
is vector of relative asset price changes; modeled as a random
variable with mean p, covariance
p
T
x = Er is expected return; x
T
x = var r is return variance
example
m
e
a
n
r
e
t
u
r
n
standard deviation of return
0% 10% 20%
0%
5%
10%
15%
standard deviation of return
a
l
l
o
c
a
t
i
o
n
x
x(1)
x(2) x(3) x(4)
0% 10% 20%
0
0.5
1
Convex optimization problems 444
Scalarization
to nd Pareto optimal points: choose
K
1
f
0
(x
2
)
2
f
0
(x
3
)
for convex vector optimization problems, can nd (almost) all Pareto
optimal points by varying
K
0
Convex optimization problems 445
Scalarization for multicriterion problems
to nd Pareto optimal points, minimize positive weighted sum
T
f
0
(x) =
1
F
1
(x) + +
q
F
q
(x)
examples
regularized least-squares problem of page 143
take = (1, ) with > 0
minimize |Ax b|
2
2
+ |x|
2
2
for xed , a LS problem
0 5 10 15 20
0
5
10
15
20
|Ax b|
2
2
|
x
|
22
= 1
Convex optimization problems 446
risk-return trade-o of page 144
minimize p
T
x + x
T
x
subject to 1
T
x = 1, x _ 0
for xed > 0, a quadratic program
Convex optimization problems 447