Lecture 2.

Mathematics Preliminaries

Sets and some related concepts

Lines in Euclidean space Rn
Linear combination
Convex set
Multivariable functions
Convex and concave functions
Unconstrainted optimization problems
Solving optimization problems using geometrical approach

1 / 21
Sets and related concepts

Sphere in Rn :
Open sphere: S(a, r ) = {x ∈ Rn | ∥ x − a ∥< r , r > 0}.
Closed sphere: S(a, r ) = {x ∈ Rn | ∥ x − a ∥≤ r , r > 0}.
Neighborhood of x0 ∈ Rn :

U(x0 ) := {x ∈ Rn | ∥x − x0 ∥ ≤ ϵ, ϵ > 0}.

Let D ⊂ Rn .
x0 ∈ D is called an interior point of D if there exists a neighborhood
U(x0 ) ⊂ D.
x0 ∈ Rn is called a boundary point of D if every neighborhood of x0
contains at least one point in D and at least one point not in D.

2 / 21
Sets and related concepts

D ⊂ Rn is called open ⇐⇒ every point of D is interior point.

D ⊂ Rn is called closed ⇐⇒ D contains all its boundary points.
Remark:
; and Rn are both closed and open.
Intersection of finite number of open sets is open. Union of any family
of open sets is open.
Intersection of any family of closed sets is closed. Union of finite
number of closed sets is closed
Let D ⊂ Rn , the complement of D in Rn : D c := Rn \ D.
D is open if D c is closed. D is closed if D c is open.
D is bounded ⇐⇒ ∃S(0, r ) : D ⊂ S(0, r ).
3 / 21
Lines in Euclidean space Rn

A line passing through x0 ∈ Rn with direction vector z ∈ Rn \ {0} is:

y = x0 + λz, λ ∈ R.
0 ≤ λ ≤ 1 then y = x0 + λz is the equation of the segment connecting x0
and x0 + z.
λ ≥ 0 then y = x0 + λz is the equation of the half-line starting from x0
in direction z.
λ ≤ 0 then y = x0 + λz is the equation of the half-line starting from x0
in opposite direction of z.
The line passing through two points a, b ∈ Rn is y = λa + (1 − λ)b,
λ ∈ R.
When λ ∈ [0, 1], y = λa + (1 − λ)b is the equation of the line
segment connecting a and b.

4 / 21
Linear combination

The linear combination of k vectors {xi ∈ Rn | i = 1, . . . , k } is:

k
αi xi ,
X
i =1
αi (i = 1, . . . , k) are constant.
A vector x can be represented by a linear combination of
{xi ∈ Rn | i = 1, . . . , k } if and only if there exist αi ∈ R, i = 1, . . . , k such
that:
k
αi xi .
X
x=
i =1
A set of vector {x1 , . . . , xk } is said to be linearly independent if the
equality
k
αi xi = 0
X
i =1
imply that all coefficients αi , i = 1, . . . , k , are equal zero.
5 / 21
Convex set

A set D ⊂ Rn is convex if for every x1 , x2 ∈ D and for any α ∈ [0, 1]

one has αx1 + (1 − α)x2 ∈ D.
The intersection of a family of convex sets is convex. However, the
union of two convex sets are not necessarily convex.
If A and B are convex then A + B and cA are convex for all c ∈ R.

6 / 21
Convex set

Convex hull of a set D ⊂ Rn is the intersection of all convex sets

containing D, denoted by coD.
Convex combination of k vectors {xi ∈ Rn | i = 1, . . . , k } is:
k
αi xi ,
X
i =1
Pk
while αi ≥ 0, i =1 αi = 1.
The convex hull of a set D ∈ Rn consists of all convex combinations of
its elements.

7 / 21
Polytopes and Polyhedra

Let C be a convex set, y ∈ C is a boundary point of C . A hyperplane

passing through y is called hyperplane of support (supporting
hyperplane) of the set C if entire set C lies completely in one of two
half-space into which this hyperplane divides the space Rn .
A convex polytope is the intersection of a finite number of
half-spaces.
A polyhedron is a nonempty bounded polytope.

8 / 21
Multivariable functions

Let f be defined on D ⊂ Rn to R, f is a function of n real variables

x1 , x2 , . . . , xn and is called a multivariable function.
The graph of f (x) is the set:

Graph(f ) = {(x, f (x)) : x ∈ D } ⊂ Rn+1 .

The set L(α, f ) = {x ∈ D : f (x) = α}, α ∈ R is called a level set of f .

For each α ∈ R, Lα (f ) := {x ∈ D : f (x) É α} is called a sublevel set of f .
For each α ∈ R, Lα (f ) := {x ∈ D : f (x) Ê α} is called a superlevel set of
f.

9 / 21
Multivariable functions

Let f (x) be defined on D ⊂ Rn , x0 ∈ D. For h ∈ R sufficiently small such

that (x10 , . . . , xi0−1 , xi0 + h, xi0+1 , . . . , xn0 ) ∈ D, we consider

f (x10 , . . . , xi0−1 , xi0 + h, xi0+1 , . . . , xn0 ) − f (x 0 )

lim .
h→0 h
If this limit exists then it is called the partial derivative of f (x) with
∂f (x0 )
respect to xi , denoted by or fxi (x0 ).
∂xi
∂f
If the partial derivative with respect to xj of at x0 exists then it is
∂xi
called the second-order partial derivative of f (x) with respect to xi
∂2 f (x0 )
and xj at x0 , and is denoted by or fxi xj (x0 ).
∂xi ∂xj

10 / 21
Multivariable functions

The vector
¸T
∂f 0 ∂f ∂f
·
0 0 0
∇f (x ) := (x ), (x ), . . . , (x )
∂x1 ∂x2 ∂xn

is called the gradient of f at x0 .

The Hessian matrix of f (x) at x0 is defined by
 
fx1 x1 (x0 ) fx1 x2 (x0 ) . . . fx1 xn (x0 )
 
f 0 fx2 x2 (x0 ) . . . fx1 xn (x0 )
0  x x (x )
H(x ) =  2 1


 ... ... ... ... 
 
fxn x1 (x0 ) fxn x2 (x0 ) . . . fxn xn (x0 )

Example. Find the gradient, and Hessian of

f (x1 , x2 ) = 5x1 + 8x2 + x1 x2 − x12 − x22 at x0 = (0;0).
11 / 21
Multivariable functions

Let f (x) be a function defined on D, d ∈ Rn , |d| ̸= 0, x0 ∈ intD. Then

the directional derivative with respect to the direction d of f (x) at
x0 :
f (x0 + λd) − f (x0 )
∂f (x0 , d) = lim+
λ→0 λ
exists (finite or infinite).
∂f (x0 , d) < 0 then d is a descent direction of f (x) at x0 .
∂f (x0 , d) > 0 then d is a ascent direction of f (x) at x 0 .
d = ∇f (x0 ) is the steepest ascent direction of f (x) at x0 .
d = −∇f (x0 ) is the steepest descent direction of f (x) at x0 .
A vector d ∈ Rn is a feasible direction at x ∈ D if exits α0 > 0 such
that x + αd ∈ D for all α ∈ [0, α0 ].

12 / 21
Multivariable functions

Example. Consider the problem

1 9
minimize x12 + x22 + 3x2 +
2 2
subject to x1 , x2 ≥ 0

Are all feasible directions at x0 = (1, 3) ascent directions?

Are all feasible directions at x1 = (0, 3) ascent directions?
Are all feasible directions at x2 = (1, 0) ascent directions?
Are all feasible directions at x3 = (0;0) ascent directions?

13 / 21
Multivariable functions

Let f (x) be defined on an open set D ⊂ Rn , f (x) is differentiable at

x0 ∈ D if all the partial derivatives f (x) with respect to every variable
at x0 and for every feasible direction one has
D E
f (x0 + d) = f (x0 ) + ∇f (x0 ), d + o(∥ d ∥)

where o(∥ d ∥) is infinitesimal with higher degree than ∥ d ∥ i.e.

o(∥ d ∥)
lim =0
∥d∥→0 ∥d∥

f (x0 + d) − f (x0 ) − ∇f (x0 ), x − x0

­ ®
⇔ lim =0
∥d∥→0 ∥d∥
f (x) is called differentiable on D if f (x) is differentiable at every
x ∈ D.
14 / 21
Multivariable functions

Let f (x1 , x2 , . . . , xn ) be second-order continuously differentiable

function on an open set containing a closed set D ⊂ Rn , x0 ∈ D, the
Taylor expansion to the second order of f (x) at x0 :
D E
f (x) = f (x0 ) + ∇f (x0 ), (x − x0 )
1
+ (x − x0 )T H(x0 )(x − x0 ) + o(∥x − x0 ∥2 )
2
Exercise: Write the Taylor expansion to the first and second order of
f (x1 , x2 ) = e x1 +x2 at x0 = (0;0).

15 / 21
Convex and concave functions

Let f (x) be defined on a convex set D ⊂ Rn

f (x) is a convex function ⇔ ∀x1 ̸= x2 ∈ D , λ ∈ [0, 1] one has

f (λx1 + (1 − λ)x2 ) ≤ λf (x1 ) + (1 − λ)f (x2 )

f (x) a concave function ⇔ ∀x1 ̸= x2 ∈ D , λ ∈ [0, 1] one has

f (λx1 + (1 − λ)x2 ) ≥ λf (x1 ) + (1 − λ)f (x2 )

When the two inequalities are strict then we have the concepts of
strictly convex and strictly concave.

16 / 21
Convex and concave functions

f (x) defined on D is convex if it epigraph

Epi(f ) = {(x, ξ) ∈ D × R : f (x) É ξ} ⊂ R n+1

is convex.
f (x) defined on D is concave if its hypograph

Hypo(f ) = {(x, ξ) ∈ D × R : f (x) ≥ ξ} ⊂ R n+1

is convex.
Remark: If f (x) is convex (strictly convex) on D then −f (x) is concave
(strictly concave) on D.

17 / 21
Convex and concave functions

Let f1 (x),f2 (x) be convex and defined on the convex sets D1 , D2 ,

respectively.
(f1 + f2 )(x) := f1 (x) + f2 (x) is convex on D1 ∩ D2 .
(f1 + f2 )(x) is strictly convex on D1 ∩ D2 is at least one of two
f1 (x), f2 (x) is strictly convex.
max{f1 , f2 }(x) = max{f1 (x), f2 (x)} is convex D1 ∩ D2 .
(αf1 + βf2 )(x) := αf1 (x) + βf2 (x) is convex for all x ∈ D1 ∩ D2 , and real
numbers α, β > 0.

18 / 21
Convex and concave functions

1 If f (x) is defined on a convex set D ⊂ Rn is convex then all the

sublevel sets Lα (f ) := {x ∈ D : f (x) ≤ α} are convex for all α ∈ R.
2 If f (x) is defined on a convex set D ⊂ Rn is concave then all the
superlevel sets Lα (f ) := {x ∈ D : f (x) ≥ α} is convex for all α ∈ R.

The above propositions are only necessary but not sufficient.

Example: f (x) = x 3 then Lα (f ), Lα (f ) are convex but f (x) is neither
convex nor concave.
Functions satisfying Proposition 1 is called quasi-convex functions,
and which satisfying Proposition 2 is called quasi-concave functions.

19 / 21
Convex and concave functions

Let f (x) be differentiable on a convex set D ⊂ Rn then

f (x) is convex on D if and only if
f (x2 ) − f (x1 ) ≥ 〈∇f (x1 ), x2 − x1 〉 for all x1 , x2 ∈ D .
f (x) is concave on D if and only if
f (x2 ) − f (x1 ) ≤ 〈∇f (x1 ), x2 − x1 〉 for all x1 , x2 ∈ D .
Let f (x) be differentiable to the second-order on a convex set D ⊂ Rn then
f (x) is convex on D if and only if the Hessian matrix H(x) is positive
semidefinite on D. f (x) is strictly convex on D if the Hessian matrix
is positive definite.
f (x) is concave on D if and only if the Hessian matrix H(x) of f (x) is
negative semidefinite on D.

20 / 21
Convex and convex function

Let f (x) be convex on D. Then f (x) attains its global minimum at x∗

is and only if ∇f (x∗ ) = 0.
Let f (x) be concave on D. Then f (x) attains its global maximum at
x∗ if and only if ∇f (x∗ ) = 0.

21 / 21