Professional Documents
Culture Documents
Convex functions
Quasiconvexity
(y, f (y))
(x, f (x))
▶ f is concave if −f is convex
▶ f is strictly convex if dom f is convex and for x, y ∈ dom f , x ≠ y, 0 < 𝜃 < 1,
concave functions:
▶ affine: ax + b on R, for any a, b ∈ R
▶ powers: x 𝛼 on R++ , for 0 ≤ 𝛼 ≤ 1
▶ logarithm: log x on R++
▶ entropy: −x log x on R++
▶ negative part: min{0, x}
convex functions:
▶ affine functions: f (x) = aT x + b
▶ any norm, e.g., the ℓp norms
– ∥x∥ p = (|x1 | p + · · · + |xn | p ) 1/p for p ≥ 1
– ∥x∥ ∞ = max{|x1 |, . . . , |xn |}
▶ sum of squares: ∥x∥ 22 = x12 + · · · + xn2
▶ max function: max(x) = max{x1 , x2 , . . . , xn }
▶ softmax or log-sum-exp function: log(exp x1 + · · · + exp xn )
f (x) x ∈ dom f
f̃ (x) =
∞ x ∉ dom f
f (x) + ∇f (x) T (y − x)
(x, f (x))
𝜕 2 f (x)
∇2 f (x)ij = , i, j = 1, . . . , n,
𝜕xi 𝜕xj
f (x, y)
1
T
y y
2
∇2 f (x, y) = 3 ⪰0
y −x −x 0
2 2
y 0 −2 x
Convex Optimization Boyd and Vandenberghe 3.11
More examples
Ín
▶ log-sum-exp: f (x) = log k=1 exp xk is convex
1 1
∇2 f (x) = diag(z) − T 2 zzT (zk = exp xk )
1 z
T (1 z)
( k zk v2k ) ( k zk ) − ( k vk zk ) 2
Í Í Í
T 2
v ∇ f (x)v = ≥0
( k zk ) 2
Í
În
▶ geometric mean: f (x) = ( k=1 xk )
1/n on Rn++ is concave (similar proof as above)
epi f
f (E z) ≤ E f (z)
prob(z = x) = 𝜃, prob(z = y) = 1 − 𝜃
▶ suppose X ∼ N (𝜇, 𝜎 2 )
▶ with f (u) = exp u, Y = f (X) is log-normal
▶ we have E f (X) = exp(𝜇 + 𝜎 2 /2)
▶ Jensen’s inequality is
p(f (X))
E f (X)
f (E X)
p(X)
EX
Convex functions
Quasiconvexity
3. show that f is obtained from simple convex functions by operations that preserve convexity
– nonnegative weighted sum
– composition with affine function
– pointwise maximum and supremum
– composition
– minimization
– perspective
examples
▶ log barrier for linear inequalities
m
log(bi − aTi x), dom f = {x | aTi x < bi , i = 1, . . . , m}
∑︁
f (x) = −
i=1
examples
▶ piecewise-linear function: f (x) = maxi=1,...,m (aTi x + bi )
▶ sum of r largest components of x ∈ Rn :
examples
▶ distance to farthest point in a set C: f (x) = supy∈C ∥x − y∥
▶ maximum eigenvalue of symmetric matrix: for X ∈ Sn , 𝜆max (X) = sup ∥y∥ 2 =1 yT Xy is convex
▶ support function of a set C: SC (x) = supy∈C yT x is convex
▶ the function g(x) = inf y∈C f (x, y) is called the partial minimization of f (w.r.t. y)
▶ if f (x, y) is convex in (x, y) and C is a convex set, then partial minimization g is convex
examples
▶ f (x, y) = xT Ax + 2xT By + yT Cy with
A B
⪰ 0, C≻0
BT C
examples
▶ f (x) = exp g(x) is convex if g is convex
▶ f (x) = 1/g(x) is convex if g is concave and positive
▶ you will use this composition rule constantly throughout this course
▶ you need to commit this rule to memory
Convex functions
Quasiconvexity
the function
(x − y) 2
f (x, y) = , x < 1, y<1
1 − max(x, y)
is convex
constructive analysis:
▶ (leaves) x, y, and 1 are affine
▶ max(x, y) is convex; x − y is affine
▶ 1 − max(x, y) is concave
▶ function u2 /v is convex, monotone decreasing in v for v > 0
▶ f is composition of u2 /v with u = x − y, v = 1 − max(x, y) , hence convex
in disciplined convex programming (DCP) users construct convex and concave functions as
expressions using constructive convex analysis
(x − y) 2
, x < 1, y<1
1 − max(x, y)
import cvxpy as cp
x = cp.Variable()
y = cp.Variable()
expr = cp.quad_over_lin(x - y, 1 - cp.maximum(x, y))
expr.curvature # Convex
expr.sign # Positive
expr.is_dcp() # True
√
▶ consider convex function f (x) = 1 + x2
▶ expression f1 = cp.sqrt(1+cp.square(x)) is not DCP
▶ expression f2 = cp.norm2([1,x]) is DCP
▶ CVXPY will not recognize f1 as convex, even though it represents a convex function
Convex functions
Quasiconvexity
▶ g is convex if f is convex
examples
▶ f (x) = xT x is convex; so g(x, t) = xT x/t is convex for t > 0
▶ f (x) = − log x is convex; so relative entropy g(x, t) = t log t − t log x is convex on R2++
(0, −f ∗ (y))
y<0
−1 − log(−y)
f (y) = sup (xy + log x) =
∗
x>0 ∞ otherwise
1 T −1
f ∗ (y) = sup (yT x − (1/2)xT Qx) = y Q y
x 2
Convex functions
Quasiconvexity
S 𝛼 = {x ∈ dom f | f (x) ≤ 𝛼}
a b c
▶ f is quasiconcave if −f is quasiconvex
▶ f is quasilinear if it is quasiconvex and quasiconcave
Convex Optimization Boyd and Vandenberghe 3.39
Examples
|x| is quasiconvex on R
√︁
▶
▶ ceil(x) = inf{z ∈ Z | z ≥ x} is quasilinear
aT x + b
f (x) = , dom f = {x | cT x + d > 0}
cT x + d
is quasilinear
∇f (x)
x