Professional Documents
Culture Documents
1
4.1
Defini2on:
Convex
func2on
x θ x + (1 − θ )y y 2
Func2ons
taxonomy
Convex Non-‐Convex
3
Local
minimum
is
global
minimum
Theorem 4.1A
For a convex function f (x), if x* is a local minimum
in ! n , it is also a global minimum in ! n .
Proof:
1) if x* is not a global minimum, then there exists a y, such that
f (y) < f (x* ).
2) For any point in the line {α y + (1 − α )x*; 0 < α < 1}, we have
by convexity of f (x) that
f (α y + (1 − α )x* ) ≤ α f (y) + (1− α ) f (x* )
!## #"### $ !###"###$
T1 T2
4
3) Since f (y) < f (x* ) by assumption, T2 < f (x* ). This implies
T1 = f (α y + (1 − α )x* ) < f (x* ).
5
4.2
Deriva2ve
based
characteriza2on
Theorem
4.2A
A funtion f (x) is convex in ! n if and only if for
every pair of vectors x and y in ! n , we have
f (y) ≥ f (x) + (y − x) ∇f (x)
T
6
f (x)
(y, f (y))
f3 (x1 , x2 ) = x / x2
2
1
10
Convexity
of
norms
Proof of convexity:
f (θ x + (1− θ )y) ≤ f (θ x) + f ((1− θ )y) (Triangle inequality)
≤ θ f (x) + (1− θ ) f (y) (Homogeneity)
11
Convexity
of
Max
func2on
12
4.4
The
quadra2c
func2on
in
standard
form
4.4.1 Defini2on
f (x) = 0.5xT Qx − bT x + c
Exercise
Compute
the
gradient
and
Hessian
of
the
following
func2ons:
T
⎡ 4 2 ⎤ ⎡ 1 ⎤
f (x) = x ⎢ T
⎥x − ⎢ ⎥ x+8
⎣ 2 3 ⎦ ⎣ 2 ⎦
T
⎡ 4 1 ⎤ ⎡ 1 ⎤
f (x) = x ⎢ T
⎥x − ⎢ ⎥ x+8
⎣ 3 3 ⎦ ⎣ 2 ⎦
16
4.5.1
Gradient
of
the
general
quadra2c
form
Direct
approach
∂ T
1) show that
∂xl
( x Qx ) = ql T x + ql x
∂ ∂
2) Then
∂xl
( 0.5x T
Qx ) −
∂xl
( b T
x ) = 0.5 ( q l x + q l x ) − bi
T
17
Alterna2ve
approach
f (x) = 0.5x Qx − b x + c
T T
g[x,d ] (α ) = f (x + α d)
=0.5xT Qx + 0.5α 2 dT Qd + α dT Qx
− bT x − α bT d + c
We know that
′ ] (α ) = dT ∇f (x + α d)
g[x,d
′ ] (0) = dT ∇f (x)..........................(E1)
g[x,d
18
′ ] (0) directly:
We can also compute g[x,d
g[x,d ] (α )=0.5xT Qx + 0.5α 2 dT Qd + α dT Qx
− bT x − α bT d + c
′ ] (α )=α dT Qd + dT Qx − bT d
g[x,d
′ ]
g[x,d (0)=d T
Qx − b T
d.....................(E2)
19
4.5.2
Hessian
of
the
quadra2c
form
Direct
approach
{F(x)}i, j =
∂2
∂xi ∂x j
f (x) =
∂
∂xi
({∇f (x)} )j
n
From (E1) we get {∇f (x)} j = ∑ Q j,k xk − b j
k=1
Hence {F(x)}i, j =
∂2
∂xi
({∇f (x)} ) = Qj j,i
⇒ F(x) = Q 20
Alterna2ve
approach
f (x) = 0.5xT Qx − bT x + c
g[x,d ] (α ) = f (x + α d)
=0.5x Qx + 0.5α d Qd + α d Qx
T 2 T T
− bT x − α bT d + c
′′ ] (α ) = dT Qd.......................(A)
g[x,d
We know from chapter 2 that
′′ ]
g[x,d (α ) = d T
H(x + α d)d...........(B)
(A) and (B) imply
H(x) = Q.
21
Summary:
Gradient
and
Hessian
of
quadra2c
form
f (x) = 0.5x Qx − b x + c
T T
Gradient:
∇ ( f (x)) = Qx − b
Hessian:
F(x) = Q
22
4.6
Minimiza2on
of
quadra2c
forms
(1) FONC
∇ ( f (x* )) = Qx* − b = 0 is solvable.
⇒ Qx* = b is solvable.
(2a) SONC
the Hessian, F(x) = Q is postive semi-definite.
(2b) SOSC-SMin
Q is postive definite.
23
Write
the
following
func2on
in
quadra2c
form
and
comment
on
the
existence
and
uniqueness
of
minima.
f1 (x1 , x2 ) = x + x 1
2 2
2
f2 (x1 , x2 ) = x + x + x1 x2
2
1
2
2
f3 (x1 , x2 ) = x − x 2
1
2
2
f4 (x1 , x2 ) = x1 x2
f6 (x1 , x2 ) = x + x + 2x1 x2 + x1 + x2
2
1
2
2
f7 (x1 , x2 ) = x + x + 2x1 x2 + x1 − x2
2
1
2
2 24
n n
f1 (x) = ∑ ∑xx j k
j=1 k= j+1
n n
f2 (x) = ∑ ∑ x j xk
j=1 k=1
n n n
f3 (x) = ∑ ∑ x j xk + ∑ xk
j=1 k=1 k=1
n n n
f4 (x) = ∑ ∑ x j xk + ∑ (−1) xk n
25
Least squares solution of Ax = b
The least square solution is defined as the
minimizer of
2
f (x) = Ax − b
= ( Ax − b ) ( Ax − b )
T
=xT A T Ax − 2bT Ax + bT b
The minimum is given by ∇f (x* ) = 2A T Ax* − 2A T b = 0
⇒ A T Ax* = A T b, for which the solution always
exists (why ?)
The Hessian F(x) = A T A, which is always postive
semi-definite (why ?)
26
Suppose A has full column rank. Then A T A is invertible.
Then x* = (A T A)−1 A T b.
! leaves it unchanged.
(1) Applying this approximation on b
i.e., P2 = P, where P = A(A T A)−1 A T .
8) P′ = I − P (verify)
10) Combine the relations in (6), (8) and (9) by using the
relationship between B and A.
29
Least squares solution of Ax = b
The regularized least square solution is defined as the
2
minimizer of f (x) = Ax − b + λ x
2
= ( Ax − b ) ( Ax − b ) + λ x
T 2
=x T ( A T A + λ I ) x − 2bT Ax + bT b
The minimum is given by
∇f (x ) = 2 ( A A + λ I ) x − 2A b = 0
* T * T
30
definite.
Exercise:
Polynomial
approxima2on
i=1
is minimized.
31
Let a = [a1 an ]T h = [h(s1 )h(sN )]T g = [g(s1 )g(sN )]T
Then
h = Sa,
where {S}i, j = (si ) j−1
Then f (a1 ,,an ) = h − g = Sa − g = ( Sa − g ) ( Sa − g ) .
2 2 T
32
Exercise:
Polynomial
approxima2on
II
i=1 i=1
is minimized.
34
Exercise:
Polynomial
approxima2on
IV
0 ⎝ dx ⎠
i=1
is minimized.
35