Fundamentals of Optimization

Lecture 2

Xavier Cabezas

FCNM-ESPOL
December 21, 2020
 Outline

Convex sets

Topological properties of convex sets

Separation and support of convex sets

Convex sets 2
 Definition of convex set

Let C be a subset of Rn . C is said a convex set if

λx1 + (1 − λ)x2 ∈ C, ∀x1 , x2 ∈ C, ∀λ ∈ [0, 1]

.

x2
x2

x1
x1

(a) A convex set. (b) A nonconvex set.

Convex sets 3
 Convex combination
A convex combination is any point x ∈ Rn of the form
x = λ 1 x1 + λ 2 x2 + . . . + λ k xk ,
with λ1 + λ2 + . . . + λk = 1 and λi ≥ 0, ∀i = 1, 2, . . . , k,

x1 , x2 , . . . , xk ∈ Rn and λ1 , λ2 , . . . , λk ∈ R.

The set of all convex combinations of points of a subset S of Rn is called

convex hull and is denoted by conv S.

x4 x3

x5

x1 x2
(a) S = { x1 , x2 , x3 , x4 , x5 } (b) S is uncountable.
Convex sets 4
 Examples of convex sets

An hyperplane H is a set of the form { x | a0 x = b }, where a ∈ Rn ,

a 6= 0 and b ∈ R.

• If x ∈ H, then a0 x = b. Hence a0 x − a0 x = b − b for all point in

x ∈ H. That is, H can be written in reference to a fixed point,
a0 (x − x) = 0.

a0 x = b
Figure: A hyperplane H.

Convex sets 5
 Examples of convex sets

An halfspace (H − , H + ) associated with a hyperplane, is a set of the

form { x | a0 x { ≤, ≥ } b }, where a ∈ Rn , a 6= 0 and b ∈ R.
• Similar to H, H − (H + ) can be written in reference to a fixed point,
a0 (x − x) ≤ 0 (a0 (x − x) ≥ 0).

a0 x ≤ b

Figure: A negative halfspace H − .

Convex sets 6
 Examples of convex sets

A polyhedral set is a nonempty subset P of Rn of the form

P = { x | a0i x ≤ bi , i = 1, . . . , r } ,
where ai are vectors and bi are scalars.

• A polyhedral set is also called polyhedron.

• The feasible region of a linear programming problem is a polyhedron.

Convex sets 7
 Examples of convex sets

A cone is a subset C of Rn such that ax ∈ C, for all a ∈ R and x ∈ C.

• A cone (nonnegative) combination is any point x ∈ Rn of the

form
x = λ1 x1 + λ2 x2 + . . . + λk xk ,
with λi ≥ 0, ∀i = 1, 2, . . . , k,

x1 , x2 , . . . , xk ∈ Rn and λ1 , λ2 , . . . , λk ∈ R.

• A convex cone (finitely generated), is a set of the form

k
X
C = {x | x = λi xi , λi ≥ 0, i = 1, . . . , k }
i=1
.

Convex sets 8
 Intersection and sum of vectors of a convex set

Theorem
Let C1 and C2 be convex sets in Rn . Then
1. C1 ∩ C2 is convex.
2. C1 + C2 = { x1 + x2 | x1 ∈ C1 , x2 ∈ C2 } is convex.
3. C1 − C2 = { x1 − x2 | x1 ∈ C1 , x2 ∈ C2 } is convex.

Proof.
1. Taking x1 and x2 of C1 ∩ C2 , it is easy to note that
λx1 + (1 − λ)x2 is also in .C1 ∩ C2 , since that x1 , x2 ∈ C1 and
x1 , x2 ∈ C2 , and both sets are convex.
2. Let x11 + x12 and x21 + x22 be elements of C1 + C2 . The
expression λ(x11 + x12 ) + (1 − λ)(x21 + x22 ) can be written as
(λx11 + (1 − λ)x21 ) + (λx12 + (1 − λ)x22 ).
3. Similar to 2.

Convex sets 9
 Caratheodory theorem

Theorem
Let S be a subset of Rn . Every element of conv S can be represented as
a convex combination of no more than n + 1 elements of S.
• The theorem is equivalent to:
(n+1 n+1
)
X X
conv S = λ i xi | λi = 1, λi ≥ 0, xi ∈ S, i = 1, . . . , n + 1
i=1 i=1

Proof.
In the lecture.

Convex sets 10
 Outline

Convex sets

Topological properties of convex sets

Separation and support of convex sets

Topological properties of convex sets 11

 Closure and interior of a convex sets

Theorem
Let C be a convex set in Rn with a nonempty interior. Let x1 ∈ cl C and
x2 ∈ int C. Then λx1 + (1 − λ)x2 ∈ int C for each λ ∈ (0, 1).

Proof.
In the lecture, but we need the following figure:

C x2
y z2

x1 z
z1

Topological properties of convex sets 12

 Closure and interior of a convex sets

Corollary
If C is a convex set, then int C is convex.

Corollary
If C is a convex set with int C 6= ∅, then cl C is convex.

Corollary
If C is a convex set with int C 6= ∅, then cl int C = cl C.

Corollary
If C is a convex set with int C 6= ∅, then int cl C = int C.

Topological properties of convex sets 13

 Outline

Convex sets

Topological properties of convex sets

Separation and support of convex sets

Separation and support of convex sets 14

 Projection theorem

Theorem
Let C be a closed convex in Rn and y ∈ / C. Then there exist a unique
point x̄ ∈ S with minimum distance from y. Furthermore, x̄ is such a
point if and only if (y − x̄)0 (x − x̄) ≤ 0 for all x ∈ C.

The vector x̄ is called the projection of y on C and is sometimes denoted

by [x]+ , i.e.,
x̄ = [x]+ = argmin ky − xk
x∈C

Proof.
In the lecture

Separation and support of convex sets 15

 Projection theorem

C
x̄ x

(y − x̄)0 (x − x̄) ≤ 0

Separation and support of convex sets 16

 Hyperplanes of separation

Let C1 and C2 be nonempty sets in Rn and Let H = { x | a0 x = b }, then

H is said to separate C1 and C2 if a0 x ≥ b for each x ∈ C1 and a0 x ≤ b
for each x ∈ C2 .

• If in addition C1 ∪ C2 6⊂ H, then H properly separates C1 and C2 .

• If a0 x > b for each x ∈ C1 and a0 x < b for each x ∈ C2 , then H

strictly separates C1 and C2

• If a0 x ≥ b + ε for each x ∈ C1 and a0 x ≤ b for each x ∈ C2 , where

ε > 0, then H strongly separates C1 and C2 .

Separation and support of convex sets 17

 Hyperplanes of separation

C1

C2

H
C1
C2
H

(a) Improper separation. (b) Proper separation.

C1 C1

H H

C2 C2

(c) Strict separation. (d) Strong separation.

Separation and support of convex sets 18

 Fundamental separation theorem

Theorem
Let C be a nonempty closed convex set in Rn and y ∈ / C. Then there
exists a nonzero vector a and a scalar b such that a0 y > b and a0 x ≤ b for
each x ∈ C.

Proof.
• By the projection theorem there exists an unique minimizing point
x̄ ∈ C such that (y − x̄)0 (x − x̄) ≤ 0 for each x ∈ C. Let
a = y − x̄ 6= 0 and b = a0 x̄, we obtain a0 x ≤ b.
• ky − x̄k2 = (y − x̄)0 (y − x̄) = a0 (y − x̄) > 0, then a0 y > b.

Corollary
Let C be a nonempty closed convex set in Rn . Then C is the intersection
of all halfspaces containing C.

Separation and support of convex sets 19

 Theorem of Farkas

Theorem
Let A ∈ Rm×n and c ∈ Rn . Then exactly one of the following systems
has a solution:
1. Ax ≤ 0 and c0 x > 0.
2. A0 y = c and y ≥ 0.

Proof.
If system 2 has a solution, A0 y = c for some y > 0. If there is an x such
that Ax ≤ 0, then c0 x = (A0 y)0 x = y 0 Ax ≤ 0, i.e., the system 1 has no
solution.
The second part uses the fundamental separation theorem. That will be a
homework (easy to follow from the literature) as it is relevant particularly
in linear programming.

Separation and support of convex sets 20

 Supporting hyperplane theorem

Let S be a nonempty subset of Rn and x̄ ∈ ∂S.

H = { x | a0 (x − x̄) = 0 } is called a supporting hyperplane of S at x̄ if
either S ⊂ H + or S ⊂ H − .
Theorem
If C ∈ Rn is a nonempty convex set and x̄ is a point that does not
belong to int C, there exists a nonzero vector a, such that a0 (x − x̄) ≤ 0
for all x ∈ C.
a H

C
x̄ x

Separation and support of convex sets 21

 Proving the supporting hyperplane theorem

Proof.
If x̄ ∈
/ cl C, then the fundamental separation theorem can be applied.

Therefore, we assume that x̄ ∈ ∂C. Then there is a sequence

{ yk } ∈
/ cl C such that yk → x̄.

Corresponding to each yk there is a ak = pk /kpk k such that a0k yk > a0k x

for each x ∈ cl C (fundamental separation theorem).

Since kak k = 1, The sequence { ak } is bounded and, as we already

know, it has a convergent subsequence { ak }K with limit a.

Consider a0k yk > a0k x for all ak belonging to this subsequence and take
the limit as k → ∞, we obtain a0 x ≤ a0 x̄, i.e., a0 (x − x̄) ≤ 0.

Separation and support of convex sets 22

 Separating hyperplane theorem

Theorem
If C1 ∈ Rn and C2 ∈ Rn are two nonempty and disjoint convex sets,
there exists a hyperplane that separates them, i.e., a nonzero vector a
such that a0 x1 ≤ a0 x2 for all x1 ∈ C1 and x2 ∈ C2 .

C1
H

C2

Proof.
Let C = { x | x = x1 − x2 , x1 ∈ C1 , x2 ∈ C2 }. Note that 0 ∈/ C, since
C1 ∩ C2 = ∅. Since the supporting hyperplane theorem there is a vector
a 6= 0 such that a0 (x − 0) ≤ 0 for all x ∈ C. Then a0 x1 ≤ a0 x2 .
Separation and support of convex sets 23