MATHEMATICAL REVIEW

Optimización

Javier Parra Peña1

1 Doctorado en Ingenierı́a

January 13, 2017

Javier Parra Peña Optimización

 MATHEMATICAL REVIEW

Content

1 MATHEMATICAL REVIEW
Vectors
Matrices
Vectorial spaces

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Content

1 MATHEMATICAL REVIEW
Vectors
Matrices
Vectorial spaces

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Definition List, length

Suppose n is a nonnegative integer. A list of length n is an ordered collection
of n elements (which might be numbers, other lists, or more abstract entities)
separated by commas and surrounded by parentheses. A list of length n looks
like this: (x1 , . . . , xn ).
Two lists are equal if and only if they have the same length and the same
elements in the same order.
Lists differ from sets in two ways: in list, order matters and repetitions have
meaning; in sets, order and repetitions are irrelevant.

Definition: Rn
Rn is the set of all lists of length n of elements of R:
Rn = {(x1 , . . . , xn ) : xj ∈ R for j = 1, . . . , n}.
In this case xj is the j th coordinate of (x1 , . . . , xn ).

Definition: addition in Rn
Addition in Rn is defined by adding corresponding coordinates:
(x1 , . . . , xn ) + (y1 , . . . , yn ) = (x1 + y1 , . . . , xn + yn )

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Vectores

A vector x is...
... an array of n scalars xl , x2 , ..., xn . Here xi is called the i-th component
of the vector x.
The notation x represents a column vector, whereas the notation xT
represents the transposed row vector.
 
a1
 a2 
a =  . 
 
 .. 
an
T
 
a = a1 a2 ... an

Vectors are denoted by lowercase boldface letters, such as a, b, c, x and y.

The collection of all n-vectors forms the n-dimensional Euclidean space,
which is denoted by Rn .
Javier Parra Peña Optimización
 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Vectors
Special Vectors
The Zero vector, denoted by 0, is a vector consisting entirely of
zeros.
The sum vector is denoted by 1 or e and has each component equal
to 1.
The i-th coordinate vector, also referred to as the i-th unit vector,
is denoted by ei , and consists of zeros except for a 1 at the i-th
position.

Vector Addition and Multiplication by a Scalar

Let x and y be two n-vectors. The sum of x and y is written as the
vector x + y. The j-th component of the vector x + y is xj + yj .
The product of a vector x and a scalar a is denoted by ax and is obtained
by multiplying each element of x by a.
Javier Parra Peña Optimización
 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Vectors

Line segment
The line segment connecting two vectors x and y is denoted [x, y] and
consists of all vectors of the form αx + (1 − α)y with 0 ≤ α ≤ 1.

Linear and Affine Independence

APcollection of vectors x1 , . . . , xk ∈ Rn is considered linearly independent
k
if j=1 λj xj = 0 implies that λj = 0 ∀ j = 1, . . . , k.
A collection of vectors x0 , x1 , . . . , xk ∈ Rn is considered to be affinely
independent if (x1 − x0 ), . . . , (xk − x0 ) are linearly independent.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Vectors
Linear, Affine, and Convex Combinations and Hulls
A vector y ∈ Rn is said to be a linear combination
Pk of the vectors
x1 , . . . , xk ∈ Rn if y can be written as y = j=1 λj xj for some scalars
λ1 , . . . , λk . Pk
If, in addition, j=1 λj = 1 then y is said to be an affine combination of
x1 , . . . , xk .
Furthermore, if λ1 , . . . , λk are restricted to be nonnegative, then this is
known as a convex combination of x1 , . . . , xk .
The linear, affine, or convex hull of a set S ⊆ Rn is, respectively, the set
of all linear, affine, or convex combinations of points within S.

Spanning Vectors
A collection of vectors x1 , . . . , xk ∈ Rn , where k ≥ n, is said to span Rn if
any vector in Rn can be represented as a linear combination of x1 , . . . , xk .
The cone spanned by a collection of vectors x1 , . . . , xk , for any k ≥ 1, is
the set of nonnegative linear combinations of these vectors.
Javier Parra Peña Optimización
 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Vectors
Basis
A collection of vectors x1 , . . . , xk ∈ Rn is called a basis of Rn if it spans
Rn and if the deletion of any of the vectors prevents the remaining vectors
from spanning Rn . It can be shown that x1 , . . . , xk form a basis of Rn if
and only if x1 , . . . , xk are linearly independent and if, in addition, k = n.

Inner Product or Scalar Product

n
The inner product
Pn of two vectors x and y in R is defined by
T T
x y = y x = j=1 xj yj . If the inner product of two vectors is equal to
zero, then the two vectors are said to be orthogonal.

Norm of a Vector
The norm of a vector x in Rn is denoted by ||x|| and defined by
1 Pn 1
||x|| = (xT x) 2 = ( j=1 xj2 ) 2 This is also referred to as the `2 norm, or
Euclidean norm.
Javier Parra Peña Optimización
 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Vectors

Cauchy-Schwartz Inequality
Let x and y be two vectors ∈ Rn , and let |xT y| denote the absolute value
of xT y. Then the following inequality, referred to as the
Cauchy-Schwartz inequality, holds true: |xT y| ≤ ||x|| · ||y||.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Content

1 MATHEMATICAL REVIEW
Vectors
Matrices
Vectorial spaces

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Matrices

Definition
A matrix is a rectangular array of numbers. If the matrix has m rows and n
columns, it is called an m × n matrix. Matrices are denoted by boldface capital
letters, such as A, B and C. The entry in row i and column j of a matrix A is
denoted by ai,j , its i-th row is denoted by Ai , and its j-th column Aj .
 
a1,1 a1,2 . . . a1,n
 a2,1 a2,2 . . . a2,n 
A= .
 
.. .. .. 
 .. . . . 
am,1 am,2 ... am,n

Addition of Matrices and Scalar Multiplication of a Matrix

Let A and B be two m × n matrices. The sum of A and B, denoted by A + B,
is the matrix whose (i, j)-th entry is ai,j + bi,j . The product of a matrix A by a
scalar c, cA is the matrix whose (i, j)-th entry is cai,j .

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Matrix Multiplication
Let A be an m × n matrix and B be an n × p matrix. Then the product AB is
definedPto be the m × p matrix C whose (i, j)-th entry ci,j is given by
ci,j = nk=1 ai,k bk,j ∀i = 1, . . . , m j= 1, . . . , p.

Transposition
Let A be an m × n matrix. The transpose of A, denoted by AT is the n × m
matrix whose (i, j)-th entry is equal to aj,i . A square matrix A is said to be
symmetric if A = AT . It is said to be skew symmetric if AT = −A.
 
a1,1 a2,1 . . . am,1
 a1,2 a2,2 . . . am,2 
AT =  .
 
.. .. .. 
 .. . . . 
a1,n a2,n . . . am,n

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Special Matrices
Zero matrix, denoted by 0, is an m × n matrix whose elements are all
equal to zero.
Identity matrix, denoted by I or In , is a square n × n matrix if ai,j = 0 for
all i 6= j and ai,j = 1 for all i = j.
Permutation matrix, denoted by P is a matrix that has the same rows of
I, but which are permuted in some order.
Orthogonal matrix, denoted by Q, is a matrix of m × n which satisfy:
QT Q = In or QQT = Im . In particular, if Q is square Q−1 = QT . Note
that a permutation matrix P is an orthogonal square matrix.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Partitioned Matrices
A matrix can be partitioned into submatrices. For example, the m × n matrix A
could be partitioned as follows:
 
A1,1 A1,2
A=
A2,1 A2,2

Where A1,1 is m1 × n1 , A1,2 is m1 × n2 , A2,1 is m2 × n1 and A2,2 is m2 × n2 ,

m = m1 + m2 , y n = n1 + n2 .

Determinant of a Matrix
Let A be an n × n matrix. The P determinant of A, denoted by det[A], is defined
iteratively as follows: det[A] = ni=1 ai1 det[Ai1 ] where Ai1 is the cofactor of
ai1 , defined as (−1)i+1 times the submatrix of A formed by deleting the i-th
row and the first column, and the determinant of any scalar is the scalar itself.
Similar to the use of the first column above, the determinant can be expressed
in terms of any row or column.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Inverse of a Matrix
A square matrix A is said to be nonsingular if there is a matrix A−1 , called the
inverse matrix, such that AA−1 = A−1 A = I. The inverse of a square matrix, if
it exists, is unique. A square matrix has an inverse if its determinant is not 0.

Rank of a Matrix
Let A be a m × n matrix. The rank of A, denoted R(A), is the maximum
number of linearly independent rows (columns) of the matrix A.
It is the set of all vectors in Rn that can be written as linear combinations of
the columns of A, i.e., R(A) = {Ax|x ∈ Rn }
If the rank of A is equal to min{m, n}, A is said to have full rank.
The nullspace (or kernel) of A, denoted N(A), is the set of all vectors x mapped
into zero by A: N(A) = {x|Ax = 0}. The nullspace is a subspace of Rn .
The rank of a matrix A is invariant under the following operations:
1 Multiplication of the columns of A by nonzero scalars,
2 Interchange of the columns,
3 Addition to a given column a linear combination of other columns

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Norm of a Matrix
Let A be an n × n matrix. Most commonly, the norm of A, denoted by ||A||, is
defined by ||A|| = max||x=1|| ||Ax||
where ||Ax|| and ||x|| are the usual Euclidean (`2 ) norms of the corresponding
vectors. Hence, for any vector ||z||, ||Az|| ≤ ||A|| ||z||. A similar use of an `p
norm || · || induces a corresponding matrix norm ||A||p . In particular, the above
matrix norm, sometimes denoted ||A||2 , is equal to the [maximum eigenvalue of
AT A]1/2 .
hP P i1/2
n n 2
Also, the Frobenius norm of A is given by ||A||F = i=1 j=1 |a ij | and is
simply the `2 norm of the vector whose elements are all the elements of A.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Eigenvalues and Eigenvectors

Let A be an n × n matrix. A scalar λ and a nonzero vector x satisfying the
equation Ax = λx are called, an eigenvalue and an eigenvector of A.
To compute the eigenvalues of A, the equation det[A − λI] = 0. The
polynomial equation in λ can be solved for the eigenvalues of A.
If A is symmetric, then it has n (possibly nondistinct) eigenvalues. The
eigenvectors associated with distinct eigenvalues are necessarily orthogonal, and
for any collection of some p coincident eigenvalues, there exists a collection of
p orthogonal eigenvectors. Hence, given a symmetric matrix A, we can
construct an orthogonal basis B for Rn , that is, a basis having orthogonal
column vectors, each representing an eigenvector of A.
Furthermore, let us assume that each column of B has been normalized to have
a unit norm. Hence, BT B = I, so that B−1 = BT . Such a matrix is said to be
an orthogonal matrix or an orthonormal matrix.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Eigenvalues and Eigenvectors, continuation

Now, consider the (pure) quadratic form xT Ax, where A is an n × n symmetric
matrix. Let λ1 , . . . , λn be the eigenvalues of A, let Λ = diag {λ1 , . . . , λn } be a
diagonal matrix comprised of diagonal elements λ1 , . . . , λn and zeros elsewhere,
and let B be the orthogonal eigenvector matrix comprised of the orthogonal,
normalized eigenvectors b1 , . . . , bn as its columns. Define the linear
transformation x = By that writes any vector x in terms of the eigenvectors of
A. Under this transformation, the given quadratic form becomes
xT Ax = yT BT ABy = yT BT ΛBy = yT Λy = ni=1 λi yi2 . This is called a
P
diagonalization process. Observe also that P we have AB = BΛ, so that because
B is orthogonal, we get A = BΛBT = ni=1 λi bi bTi .
This representation is called the spectral decomposition of A. For an m × n
matrix A, a related factorization A = UΣVT , where U is a m × m orthogonal
matrix, V is a n × n orthogonal matrix, and Σ is a m × n matrix having
elements Σij = 0 for i 6= j, and Σij ≥ 0 for i = j, is known as a singular-value
decomposition (SVD) of A. Here, the columns of U and V are normalized
eigenvectors of AAT and AT A, respectively. The Σij values are the (absolute)
square roots of the eigenvalues of AAT if m ≤ n or of AT A if m ≥ n. The
number of nonzero Σij values equals the rank of A.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Definite and Semidefinite Matrices

Let A be an n × n matrix.
Positive definite: A is said to be positive definite if xT Ax > 0 for all nonzero
x ∈ Rn
Positive semidefinite: A is said to be positive semidefinite if xT Ax ≥ 0 for all
x ∈ Rn .
Negative definite: A is called negative definite if xT Ax < 0 for all nonzero
x ∈ Rn
Negative semidefinite: A is called negative semidefinite if xT Ax ≤ 0 for all
x ∈ Rn
Indefinite: A matrix that is neither positive semidefinite nor negative
semidefinite is called indefinite.
By the diagonalization process, the matrix A is positive definite, positive
semidefinite, negative definite, and negative semidefinite if and only if its
eigenvalues are positive, nonnegative, negative, and nonpositive, respectively.
Also, by the definition of A and B above, if A is positive definite, then its
square root A1/2 is the matrix satisfying A1/2 A1/2 = A and is given by
A1/2 = BΛ1/2 BT .
Javier Parra Peña Optimización
 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Content

1 MATHEMATICAL REVIEW
Vectors
Matrices
Vectorial spaces

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Espacios vectoriales

Definition
A vector space is a set V along with an addition on V and a scalar
multiplication on V such that the following properties hold:
commutativity: u + v = v + u ∀ u, v ∈ V;
associativity: (u + v) + w = u + (v + w) and
(ab)v = a(bv) ∀ u, v, w ∈ V and ∀ a, b ∈ R;
additive identity: there exists an element 0 ∈ V such that
0 + v = v ∀ v ∈ V;
additive inverse: for every v ∈ V , there exists w ∈ V such that v + w = 0;
multiplicative identity 1v = v ∀ v ∈ V;
distributive properties a(u + v) = au + av and (a + b)u = au + bu
∀ a, b ∈ R and ∀u, v ∈ V.

Javier Parra Peña Optimización

 Vectors
MATHEMATICAL REVIEW Matrices
Vectorial spaces

Espacios vectoriales

Properties
The definition of a vector space requires that it have an additive identity.
A vector space has a unique additive identity. Each element v in a vector
space has an additive inverse, an element w in the vector space such that
v + w = 0. Because additive inverses are unique, we can let −v denote
the additive inverse of a vector v. We define w − v to mean w + (−v).
0v = 0 for every v ∈ V.
a0 = 0 for every a ∈ R.
(−1)v = −v for every v ∈ V.

Javier Parra Peña Optimización