SYS 6003: Optimization Fall 2015

Lecture 6
Instructor: Quanquan Gu Date: Sep 12th

The following theorem provides a sufficient and necessary condition to verify a function
is convex.

Theorem 1 (First order condition for convex functions) Suppose f : Rd → R is a

continuously differentiable function over its convex domain domf , then f (x) is convex if
and only if

f (y) ≥ f (x) + ∇f (x)> (y − x) (1)

for all x, y ∈ domf .

Proof: We first prove the forward direction “⇒”

Suppose that f (x) is convex, then for any x, y ∈ domf and any α ∈ [0, 1], we have f (αy +
(1 − α)x) ≤ αf (y) + (1 − α)f (x). Rearranging this inequality leads to

f (αy + (1 − α)x) − (1 − α)f (x) f (αy + (1 − α)x) − f (x)

f (y) ≥ = f (x) + . (2)
α α
By Taylor expansion, we have

f (x + α(y − x)) = f (x) + α∇f (x)> (y − x) + o(α).

where o(α) means limt→0 o(α)/α = 0. Also note that x + α(y − x) = αy + (1 − α)x. Thus,
it follows from (2) that

α∇f (x)> (y − x) + o(α) o(α)

f (y) ≥ f (x) + = f (x) + ∇f (x)> (y − x) + ,
α α
which immediately leads to (1) by taking α → 0.
Now we prove the backward direction “⇐”:
We want to show that, for any x, y ∈ domf and any α ∈ [0, 1],

f (αx + (1 − α)y) ≤ αf (x) + (1 − α)f (y).

Let z ≡ αx+(1−α)y, since domf is a convex set, we have z ∈ domf . Since x, y, z ∈ domf ,
we have

f (x) ≥ f (z) + ∇f (z)> (x − z) (3)

>
f (y) ≥ f (z) + ∇f (z) (y − z). (4)

Now multiply inequality (3) by α and inequality (4) by (1 − α) to obtain:

αf (x) + (1 − α)f (y) ≥ αf (z) + α∇f (z)> (x − z) + (1 − α)f (z) + (1 − α)∇f (z)> (y − z)

1
And we are left with the right hand side equal to:

f (z) + ∇f (z)> (αx + αz + (1 + α)(y − z)) = f (z) + ∇f (z)> (αx + (1 − α)y − z)

= f (z) + ∇f (z)> (z − z)
= f (z).

And since we know f (z) = f (αx+(1−α)y), we can conclude this is indeed a convex function.

Figure 1: Illustrating the 1st Order Condition for Convex Functions

In order to prove that a function is convex, we can use the definition. But sometimes
that can be tedious. In the following, we will introduce second order sufficient and necessary
condition for convex functions, which provides an easy way to prove a function is convex.

Theorem 2 (Second order condition for convex functions) Suppose f : Rd → R is

twice continuously differentiable over its convex domain domf , then f is convex if and only
if,

∇2 f (x)  0 for all x ∈ domf

Proof: By the mean value theorem, we have:

1
f (y) = f (x) + ∇f (x)> (y − x) + (y − x)> ∇2 f (z)(y − x), (5)
2
where z = αx + (1 − α)y, α ∈ [0, 1]. Note that since domf is convex, we have z ∈ domf .
We first prove the forward direction “⇒”:
Since f is convex, by the first order condition, we have for any x, y ∈ domf ,

f (y) ≥ f (x) + ∇f (x)> (y − x). (6)

Therefore, by combining (5) and (6), we have

1
(y − x)> ∇2 f (z)(y − x) ≥ 0.
2

2
Let y → x, then z → x. By the continuity of ∇2 f (x), we then have

(y − x)> ∇2 f (x)(y − x) ≥ 0.

Due to the arbitrariness of y and x, it follows that ∇2 f (x)  0.

We now prove the backward direction “⇐”:
Consider any x, y ∈ domf where α ∈ [0, 1]. Let z = αx + (1 − α)y, since domf is a convex
set, we have z ∈ domf . Since ∇2 f (x)  0 for all x ∈ domf , we have ∇2 f (z)  0. For any
y ∈ domf , by the definition of positive semidefinite, we then know

(y − x)> ∇2 f (z)(y − x) ≥ 0. (7)

Therefore, by combining (5) and (7), we have f (y) ≥ f (x) + ∇f (x)> (y − x) . By the
first-order condition for convex functions, f is convex.

Now we will illustrate the application of second-order condition for convex functions with
several examples.

Example 1 (Quadratic Function) f (x) = 12 x> Px+q> x+r, where P ∈ Rd×d , P  0, q ∈

Rd , r ∈ R, x ∈ Rd .
f (x) is convex, since

∇2 f (x) = P  0.

Example 2 (Loss function of Linear Regression)

1
f (x) = kAx − bk22 ,
2
where A ∈ Rn×d , x ∈ Rd , b ∈ Rn . f (x) is convex, since

∇2 f (x) = A> A  0,