FABIZ Lecture Notes 31

2.8. Extreme points

2.8.1. De…nition (Extreme points and extreme values concepts). Consider f ( ) : A Rn ! R and
x0 2 A.
(1) The point x0 2 A is a (global) minimum point for f ( ) if the following condition holds:
f x0 f (x) ; 8x 2 A:
The value f (x0 ) is the (global) minimum value of f ( ) on A.
(2) The point x0 2 A is a (global) maximum point for f ( ) if the following condition holds:
f x0 f (x) ; 8x 2 A:
The value f (x0 ) is the (global) maximum value of f ( ) on A.
(3) The point x0 2 A is a local minimum point for f ( ) if the following condition holds:
9V 2 N x0 ; f x0 f (x) ; 8x 2 A \ V:
The value f (x0 ) is a local minimum value of f ( ) on A.
(4) The point x0 2 A is a local maximum point for f ( ) if the following condition holds:
9V 2 N x0 ; f x0 f (x) ; 8x 2 A \ V:
The value f (x0 ) is a local maximum value of f ( ) on A.
(5) The point x0 2 A is an unconstrained local minimum point for f ( ) if x0 is an interior point of A
and a local minimum point for f ( ).
(6) The point x0 2 A is an unconstrained local maximum point for f ( ) if x0 is an interior point of A
and a local maximum point for f ( ).
(7) The least upper bound of the function f ( ) on A is denoted by inf f (x) and it is the least upper
x2A
bound (possibly in…nite) for the set f (A) ( inf f (x) = inf f (A)).
x2A
(8) The greatest lower bound of the function f ( ) on A is denoted by sup f (x) and it is the greatest
x2A
lower bound (possibly in…nite) for the set f (A) (sup f (x) = sup f (A)).
x2A

inf f (x) = ()
x2A
1. f (x), 8x 2 A
2. 8 > , 9x 2 A, f (x ) < .
2.8.2. Theorem (Thm. 2.1, [8]). Consider f ( ) : A Rn ! R, A closed (A = A).
Assume there is 2 R such that the set A = fx 2 A; f (x) g is nonempty and bounded. [i.e. the
–level set is relatively compact]
Then there is at least a point x0 2 A such that f (x0 ) = inf f (x) (the function attains its minimum).
x2A

Proof. Consider = inf f (x) and observe that f (x) 8x 2 A.

x2A
If < then Ab would be empty, a contradiction with the hypotheses, so it has to be that .
If = , since the set A is nonempty, there are points with the required quality and the proof is
…nished for this this case.
Consider > .
If is not an accumulation point for the set f (A), then should be an isolated point for f (A). Then
2 f (A) so there is a point x0 2 A such that = f (x0 ) (in this case, x0 is also an isolated point for A).
32 Cristian Necul¼
aescu

If is an accumulation point for the set f (A), then consider a sequence (xn )n2N such that xn 2 A
8n 2 N and f (xn ) ! . Because > , and the codomain is separated (as a topological space), the
sequence (f (xn ))n2N will eventually satisfy f (xn ) (i.e.: 9n0 2 N such that 8n n0 , f (xn ) )
so the sequence (xn )n2N may be considered such that xn 2 A, 8n 2 N, f (xn ) ! and f (xn ) (i.e.
xn 2 A , 8n 2 N). Since A is bounded, the sequence (xn )n2N is also bounded and has a subsequence
(xnk )k2N convergent to a value x0 2 A (because the sequence is from the set A which is included in the
closed set A). Since the function f ( ) is continuous, f (x0 ) = lim f (xnk ) = , so the function f ( ) attains
k!1
its minimum value at the point x0 .
2.8.3. Theorem (Thm. 2.2, [8]). Consider f ( ) : A Rn ! R and = inf f (x). Assume there is
x2A
b > such that 8 < b the set A is compact. Then there is x 2 A such that f (x0 ) = (f ( ) attains its
0

minimum value). [Observe that the previous continuity requirement for f ( ) is replaced by the compactity
requirement not for just one level set A , but for all level sets A ]
Proof. For ; 1 2T( ; b] with < 1 the sets A and A
T 1 are nonempty, compact0and A A 1 , so
that the intersection A is nonempty. Then for x0 2 A it happens that f (x ) , 8 2 ( ; b]
2( ;b] 2( ;b]
which means f (x0 ) = inf f (x), so that in fact f (x0 ) = .
x2A

A consequence of the previous results is usually used:

2.8.4. Theorem (Thm. 2.3, [8]). Consider f ( ) : A Rn ! R. If f ( ) is continuous and A is
compact then there are points x , x 2 A such that f (x) 2 [f (x0 ) ; f (x1 )], 8x 2 A. [When the function
0 1

is continuous and the domain is compact both the minimum and the maximum values of the function are
attained]
2.8.5. Theorem (Thm. 2.5, [8]). Consider f ( ) : A Rn ! R such that the function is continuous
in a neighborhood of the compact set A0 A. If f (x) > 8x 2 A0 then the inequality also holds for a
neighborhood of A0 .
Proof by contradiction

2.9. Unconstrained Local Optimization

Consider f ( ) : A Rn ! R and x0 2 int (A).
The derivative of f ( ) at x0 in the direction h is the linear form f 0 (x0 ; ) : Rn ! R, f 0 (x; h) =
P @f 0
n @
(x ) hi = f 0 (x) h is the derivative of the function g (t) = f (x0 + th) at t = 0 ( (g (x0 + th)) ).
i=1 @xi @t t=0
Pn P n @2f
The second derivative of f ( ) in the direction h is the quadratic form f 00 (x; h) = (x0 ) hi hj =
i=1 j=1 @xi @xj

t @2f 0 t 00 00 @2f
h (x ) h = h f (x) h, where f (x) = (x0 ) is the Hessian of f ( ) at x0 .
@xi @xj i;j=1;n @xi @xj i;j=1;n
@2
The quadratic form f 00 (x; h) is the second derivative of g (t) = f (x0 + th) at t = 0 ( 2 (g (x0 + th)) ).
@t t=0

2.9.1. Theorem (Thm. 3.1, [8]). Consider a class C 2 function f ( ) : A Rn ! R and x0 2 int (A).
If x0 is an unconstrained local minimum, then f 0 (x0 ) = 0 and f 00 (x0 ) is nonnegative.
 FABIZ Lecture Notes 33

2.9.2. Theorem (Thm. 3.2, [8]). Consider a class C 2 function f ( ) : A Rn ! R and x0 2 int (A).
If f 0 (x0 ) = 0 and f 00 (x) is positive, then 9 ; m > 0 such that f (x) f (x0 ) + m kx x0 k, 8x 2 A and
kx x0 k < .

A solution x0 of the system f 0 (x) = 0 is called critical (stationary) point, and the corresponding
value of the function f (x0 ) is called critical value. A critical point x0 is called nondegenerate if f 00 (x0 ) is
nonsingular (the determinant is nonzero).

2.9.3. Theorem (Thm. 3.5, [8]). Consider a class C 2 function f ( ) : A Rn ! R and x0 2 int (A) a
nondegenerate critical point. The point x0 is a local minimum if and only if f 00 (x0 ) is nonnegative. The
point x0 is a local maximum if and only if f 00 (x0 ) is nonpositive.
34 Cristian Necul¼
aescu

Procedure for 2 variable function:

fx0 (x; y) = 0
1. Solve (F OC) :
fy0 (x; y) = 0
2. For each solution of (F OC) system,
fx002 (x; y) fxy
00
(x; y)
2.a. Calculate the Hessian: Hf (x; y) =
fyx (x; y) fy002 (x; y)
00

2.b. Decide:
det Hf (x; y) fx002 (x; y) (x; y)
>0 >0 maximum
>0 <0 minimum
<0 saddle (no extreme)
=0 the method doesn’t decide

For the general case: for each stationary point consider the quadratic function:
P
n
@2f
'( ) = @xi @yj
(a) i j .
i;j=1
If:
1. ' ( ) is negative then a is an extreme point, namely a local interior maximum point for f ( ).
2. ' ( ) is positive then a is an extreme point, namely a local interior minimum point for f ( ).
3. ' ( ) is not de…ned, then a is not a local interior extreme point (it is a saddle point)
4. ' ( ) is nonnegative or nonpositive, then the procedure cannot decide.
The Hessian Hf ( ) is the matrix of the quadratic function:
0 1
@2f @2f
B @x1 @x1 (a) @xn @x1
(a) C
@2f B .. .. C
Hf (a) = (a) =B
B . . C
C
@xi @xj @ @2f A
i;j=1;n
@2f
(a) (a)
@x1 @xn @xn @xn
The Hessian (and hence the quadratic form ' ( )) is positive if an only if all the determinants are
positive:
@2f @2f
(a) (a)
@x1 @x1 @xj @x1
8j = 1; n, j = .. .. >0
. .
2 2
@ f @ f
(a) (a)
@x1 @xj @xj @xj
and negative if and only if
8j = 1; n, ( 1)j j > 0.
 FABIZ Lecture Notes 37

2.10. Constrained Optimization

Procedure for 2 variables functions:
[opt] f (x; y) constrained by g (x; y) = 0
1. Form the Lagrange function:
L (x; y; ) = f (x; y) + g (x; y)
2. Apply (F8OC) for L ( ; ; ) to obtain:
< L0x (x; y) = 0
(CF OC) : L0y (x; y) = 0
:
g (x; y) = 0
For each solution (x0 ; y0 ; 0 ) of (CF OC):
3. Di¤erentiate the constraint: gx0 (x0 ; y0 ) dx + gy0 (x0 ; y0 ) dy = 0
4. Establish the signature of the quadratic form:
(dx; dy) 7! L00x2 (x0 ; y0 ; 0 ) dx2 + 2L00xy (x0 ; y0 ; 0 ) dxdy + L00y2 (x0 ; y0 ; 0 ) dy
2
gx0 (x0 ;y0 )dx+gy0 (x0 ;y0 )dy=0

Alternative
0 method: The bordered Hessian 1
@g @g
0 (x0 ; y0 ) (x0 ; y0 )
B @x @y C
B @g 2
@ L 2
@ L C
B C
B (x0 ; y0 ) (x0 ; y0 ; 0 ) (x0 ; y0 ; 0 ) C
B @x @x 2 @x@y C
@ @g @2L @2L A
(x0 ; y0 ) (x0 ; y0 ; 0 ) (x 0 ; y 0 ; 0 )
@y @y@x @y 2
Example:
f (x; y) = xy subject to x + y = 1
L (x; y; ) = xy + (x + y 1)
@L
(x; y; ) = y +
@x
@L
(x; y; ) = x +
@y
) 8 8
< y+ =0 >
< y=
x = 1 1 1
(CF OC) : x+ =0 ) ) solution ; ;
: x+y =1 >
: 2 =1) = 1 2 2 2
2
(CSOC) :
Alternative 1: di¤erentiate the constraint
x + y 1 = 0 ) dx + dy = 0j(x;y)=(1=2;1=2) ) dx + dy = 0 ) dy = dx.
[Calculate the second–order di¤erential for the function (x; y) 7! L (x; y; 0 )]

d2(x;y) L (x; y; ) = L00x2 (x; y; ) dx2 + 2L00xy (x; y; ) dxdy + L00y2 (x; y; ) dy 2 =
= 2dxdy

1 1 1 1 1
d2(x;y) L ; ; = 2dxdyjdy= dx = 2dx2 < 0 (negative form) so the point ; is a conditional
2 2 2 2 2
local interior maximum.
38 Cristian Necul¼
aescu

Alternative
0 2: bordered Hessian 1
@g @g
0 (x0 ; y0 ) (x0 ; y0 )
B @x @y C
B @g 2
@ L 2
@ L C
B C
B (x0 ; y0 ) (x0 ; y0 ; 0 ) (x0 ; y0 ; 0 ) C =
B @x @x 2 @x@y C
@ @g @2L @2L A
(x0 ; y0 ) (x0 ; y0 ; 0 ) (x 0 ; y 0 ; 0 )
0@y 1@y@x @y 2
0 1 1
= @ 1 0 1 A
1 1 0
the determiant is:
0 1 1
1 1
3 = 1 0 1 = 2 > 0 so the point ; is a conditional local interior maximum.
1 1 0 2 2