The University of Texas at Austin

Department of Electrical and Computer Engineering

EE381K: Convex Optimization — Fall 2020

Lecture 3

Aryan Mokhtari Wednesday, September 2, 2020

Goal: In this lecture, we first show that under what conditions a polyhedron has at least one
extreme point. Then, we talk about the connection between extreme points and optimal solutions.
Next, we introduce the notions of vertex and basic feasible solution for a polyhedron. Finally, we
state the Theorem of Alternatives (ToA) and provide its proof.

1 When does a polyhedron have extreme points?

As we mentioned earlier, not all polyhedrons have extreme points. For, instance a half space does
not have any extreme point. In fact, if a polyhedron contains a line it doesn’t have any extreme
point. Only pointed polyhedrons have extreme points. We prove this in the following theorem.

Theorem 1. For a polyhedron P = {x | Ax ≤ b, Dx = e}, the following statements are equivalent:

• (a) P has at least one extreme point.

• (b) P does not contain a line.

 
A
• (c) P is a pointed polyhedron. (rank = n)
D

Proof. We prove these statements are equivalent by showing (a) ⇒ (c) ⇒ (b) ⇒ (a)

(a) ⇒ (c) : Note that  then P has at 

if (a) holds least
one extreme
 point 
which we denote by x̂.
AJ(x̂) A AJ(x̂)
Hence, we have rank = n. Since rank ≥ rank then we obtain that
D D D
 
A
rank = n and therefore P is a pointed polyhedron.
D
 
A
(c) ⇒ (b) : If P is a pointed polyhedron, then n rows of are linearly independent. Let’s
D
 
A
denote {â1 , . . . , âk , d̂k+1 , . . . , d̂n } as the set of n linearly independent rows of . Suppose the
D
polyhedron P contains a line of the form x0 + λv0 , where x0 ∈ P, v0 6= 0, and λ ∈ R. Then, for
any λ ∈ R we have
â>
i (x0 + λv0 ) ≤ b̂i for i = 1, . . . , k

This condition only holds if â> >

i x0 ≤ b̂i and âi v0 = 0 for i = 1, . . . , k. Further, for any λ ∈ R we
have
d̂>
i (x0 + λv0 ) = êi for i = k + 1, . . . , n

1
This condition only holds if d̂> >
i x0 = êi and d̂i v0 = 0 for i = 1, . . . , k. Note that vectors
{â1 , . . . , âk , d̂k+1 , . . . , d̂n } are linearly independent (create a basis for Rn ), but their inner product
with vector v0 is zero. This implies that vector v0 should be 0 which is a contradiction.

(b) ⇒ (a) : Assume that P does not contain a line. Pick a point x0 in the polyhedron. Define
J(x0 ) as the indices of active constraints at x0 , i.e., J(x0 ) = {i | a> i x0 = bi }. If span({ai | i ∈
J(x )} ∪ {d , . . . , d }) = R n then the proof is complete, since the span of a subset of rows of matrix
 0 1 p
A n
is R , hence, it has n linearly independent rows and hence its rank is n and therefore x0 is an
D
extreme point. Now suppose span({ai | i ∈ J(x0 )} ∪ {d1 , . . . , dp }) 6= Rn , then there exists a vector
v 6= 0 such that its inner product with all of the vectors {ai | i ∈ J(x0 )} and {d1 , . . . , dp } is zero.
Now consider the following line: {y | y = x0 + λv, for any λ ∈ R}. Note that for the points on this
line we have
a>i y = bi , for i ∈ J(x0 ), d>
j y = ej , for j = 1, . . . , p.

Hence, all active constraints remain active for any. point in this line, but since polyhedron P does
not contain any line, there should be a λ∗ such that x0 + λ∗ v hits a new constraint a> k x ≤ bk
> ∗
meaning ak (x0 + λ v) = bk , where k ∈ / J(x0 ). It can be shown that ak does not belong to
span({ai | i ∈ J(x0 )} ∪ {d1 , . . . , dp }), since if it belongs to it then we have a> k v = 0 which
>
implies ak x0 = bk and it contradicts with the fact that k ∈ / J(x0 ). Hence, ak does not belong to
span({ai | i ∈ J(x0 )} ∪ {d1 , . . . , dp }). Therefore, if we define a new point in the polyhedron x1 as
x1 = x0 + λ∗ v, we can conclude that

dim (span({ai | i ∈ J(x0 )} ∪ {d1 , . . . , dp })) < dim (span({ai | i ∈ J(x1 )} ∪ {d1 , . . . , dp }))

If we do this process at most n times then we reach a point x̂ such that dim (span({ai | i ∈ J(x0 )})) =
n and therefore x̂ is an extreme point.

Remark 1. According to the above theorem for any polyhedron P = {x | Ax ≤ b, Dx  =  e}, we

A
can easily check if it has extreme points or not by following the rank test of matrix . If the
D
rank of this matrix is n, then it has at least one extreme point. If the rank is smaller than n, then
it does not have any extreme point.

2
2 Extreme Points and Optimality
First, remember that for each LP we have three possible outcomes:
1. LP is infeasible and p∗ = +∞
2. LP is unbounded below and p∗ = −∞
3. LP has an optimal solution and p∗ is finite
In this section, we prove the following theorem which shows the connection between extreme points
and optimality for the case that p∗ is finite.
Theorem 2. Suppose the following LP has an optimal solution
minimize c> x
subject to x ∈ P, (1)
and polyhedron P has at least one extreme point. Then, there exists an extreme point of P that is
an optimal solution.
Proof. If LP has an optimal solution, then we can consider σ as the optimal value, i.e, c> x∗ = σ,
where x∗ is an optimal solution. Now define the polytope Q as
Q = {x ∈ P | c> x = σ}.
Indeed, Q ⊆ P. Note that P does not contain a line, hence Q also does not contain a line.
Therefore, Q has at least one extreme point. Note that any point that belongs to Q is an optimal
solution. Now it remains to show that an extreme point of Q is also an extreme point of P.
Consider x̂ as an extreme point of Q. Now suppose it is not an extreme point of P. This means
that x̂ can be written as x̂ = λy + (1 − λ)z where y, z ∈ P and λ ∈ (0, 1). Note that
c> y ≤ σ, c> z ≤ σ, c> x̂ = λc> y + (1 − λ)c> z = σ
These conditions are satisfied if and only if
c> y = c> z = σ,
Therefore, we obtain that y, z ∈ Q which is a contradiction as it shows that x̂ is not an extreme
point of Q. Hence, we can conclude that an extreme point of Q is also an extreme point of P. The
proof is complete.
Exercise. Provide an example where the LP has an optimal solution, but the set of optimal
solutions does not have any extreme point.

2.1 Back to assignment example

n X
X n
minimize cij xij
i=1 j=1
Xn
subject to xij = 1 for i = 1, . . . , n (2)
j=1
n
X
xij = 1 for j = 1, . . . , n
i=1
0 ≤ xij ≤ 1 for i = 1, . . . , n j = 1, . . . , n

3
Remember that in the previous lecture we showed that all EPs of this polyhedron are permutation
matrices and vice versa. From this lecture, we know that at least one of the optimal solutions of
the above problem is an EP. Hence, if we solve this problem using simplex method and find the
optimal solution that is an EP, then the obtained solution is also a permutation matrix and is a
feasible and optimal solution for the original problem.

3 Extreme point, vertex, and basic feasible solution

Definition 1. A point x ∈ P is a vertex if there exists some vector c ∈ Rn such that c> x < c> y
for all y ∈ P and y 6= x.

Definition 2. A point x ∈ P is a basic feasible solution of P = {x | Ax ≤ b, Dx = e} if the

concatenation of AJ(x) and D has n linearly independent columns.

We already have shown that a basic feasible solution is equivalent to an extreme point. Next, we
complete the relations between these two and vertex.

Theorem 3. The following statements are equivalent:

• x̂ is a vertex of P

• x̂ is an extreme point of P

• x̂ is a basic feasible solution of P

Proof. Homework.

The concept of basic feasible solution is useful for implementing the Simplex method.

4 Theorem of Alternatives for Linear Inequalities

We prove ToA as we will use in the lecture to prove Farkas’ Lemma. Later, we will use the result
of Farkas’ Lemma to prove strong duality for LPs.

Theorem 4 (Theorem of Alternatives). For any given matrix A ∈ Rm×n and vector b ∈ Rm
exactly one of the following statements holds:
(i) There exists an x ∈ Rn that satisfies Ax ≤ b.
(ii) There exists a z ∈ Rm that satisfies z ≥ 0, A> z = 0, b> z < 0.

Proof. We divide the proof into two steps:

Step 1: First we show that both of these two statements can not both hold at the same time. We
show that if both hold then it would lead to a condition.

Ax − b ≤ 0, z≥0 ⇒ z> (Ax − b) ≤ 0 ⇒ z> b ≥ 0

which is in contrast with b> z < 0. Hence, both (i) and (ii) cannot hold both at the same time.

Step 2: Next, we prove that it is impossible that both of these statements could be false. To prove
this claim, we show that if (i) is false, then (ii) is true. We show this by induction on the number
of columns of A:

4
Induction base: A has 1 column. We first show that if A has only 1 column, then when (i) is
false (ii) is true.
In this case, x is a scalar and the conditions in (i) and (ii) simplify as
(i) There exists an x ∈ R that satisfies a1 x ≤ b1 , . . . , am x ≤ bm .
(ii) There exists a z ∈ Rm that satisfies z ≥ 0, m
P Pm
i=1 ai zi = 0, i=1 bi zi < 0.

If (i) is false then it means this system of inequalities a1 x ≤ b1 , . . . , am x ≤ bm is not solvable. This
system is not solvable if and only if one of the following holds:
1) ∃i such that ai = 0 and bi < 0.
b
2) abii < ajj for some ai > 0 and aj < 0.
Now we show that under both cases (ii) is true.
If we are in case (1), then define z = ei where ei is a vector that all its components are zero except
the ith one which is 1. It can be easily checked that for z = ei the conditions in (ii) hold since
m
X m
X
z = ei ≥ 0, ai zi = 0, bi zi = bi < 0.
i=1 i=1

If we are in case (2), then we define z as a vector that all its components are zero except the ith
one which is 1/ai and jth one which is −1/aj . In this case we can also easily checked that for z
the conditions in (ii) hold since
m m
X X bi bj
z ≥ 0, ai zi = 0, bi zi = − < 0.
ai aj
i=1 i=1

Hence, the induction step is complete, and if A has only one column the Theorem of Alternatives
holds.

Induction step: Suppose ToA holds for the case that A has n − 1 columns and we aim to show
that it also holds for the case that A has n columns.
Consider an inequality Ax ≤ b where A ∈ Rm×n . Suppose it is infeasible. Then, we will show its
alternate holds. This system can be written as
n
X
Aik xk ≤ bi for i = 1, . . . , m
k=1

We can divide the inequalities Ax ≤ b in three groups:

I+ := {i | Ain > 0}, I0 := {i | Ain = 0}, I− := {i | Ain < 0}.

Define Cik as
   
Aik bi

 Ain if i ∈ I+ (Ain > 0) 
 
 Ain if i ∈ I+ (Ain > 0) 

Cik = di = (3)
 − Aik

if i ∈ I− (Ain

< 0)   − bi

if i ∈ I− (Ain < 0) 

Ain Ain

5
Using these definitions, we can scale the inequalities with Ain 6= 0 to get an equivalent system
n−1
X
Cik xk + xn ≤ di for i ∈ I+
k=1
n−1
X
Cik xk − xn ≤ di for i ∈ I−
k=1
n−1
X
Aik xk ≤ bi for i ∈ I0
k=1

The first two systems are feasible if and only if

n−1 n−1
! !
X X
max Cik xk − di ≤ min di − Cik xk
i∈I− i∈I+
k=1 k=1

Hence, Ax ≤ b is solvable if and only if there exists (x1 , . . . , xn−1 ) such that
n−1
X
(Cik + Cjk )xk ≤ di + dj for i ∈ I− , j ∈ I+
k=1
n−1
X
Aik xk ≤ bi for i ∈ I0
k=1

This is a system of inequalities with n − 1 variables.

If this system in infeasible, then according to the hypothesis of the induction its alternate holds
and there exist uij , vi such that

uij ≥ 0 for i ∈ I− , j ∈ I+ , vi ≥ 0 for i ∈ I0

X X X
(Cik + Cjk )uij + vi Aik = 0, k = 1, . . . , n − 1.
i∈I− j∈I+ i∈I0
X X X
(di + dj )uij + bi vi < 0
i∈I− j∈I+ i∈I0

Now if we define the vector z ∈ Rm as

1 X
zi = uij , for i ∈ I−
−Ain
j∈I+
1 X
zj = uij , for j ∈ I+
Ajn
i∈I−

zi = vi , for i ∈ I0 ,

then it can be verified that z ≥ 0, A> z = 0, b> z < 0 hold, and the proof is complete.