CS/ECE/ISyE 524 Introduction to Optimization Spring 2017–18

11. Quadratic forms and ellipsoids

ˆ Quadratic forms

ˆ Orthogonal decomposition

ˆ Positive definite matrices

ˆ Ellipsoids

Laurent Lessard (www.laurentlessard.com)

Quadratic forms
ˆ Linear functions: sum of terms of the form ci xi where the
ci are parameters and xi are variables. General form:

c1 x 1 + · · · + cn x n = c T x

ˆ Quadratic functions: sum of terms of the form qij xi xj

where qij are parameters and xi are variables. General form:

q11 x12 + q12 x1 x2 + · · · + qnn xn2 (n2 terms)

 T   
x1 q11 . . . q1n x1
 ..   .. . . . . T
=.  . . ..   ..  = x Qx
 
xn qn1 . . . qnn xn

11-2
Quadratic forms
Example: 4x 2 + 6xy − 2yz + y 2 − z 2
 T   
x 4 6 0 x
y  0 1 0  y
z 0 −2 −1 z
 T    
x 4 p2 q2 x  p1 + p2 = 6
In general: y  p1 1 r2  y  q1 + q2 = 0
z q1 r1 −1 z r1 + r2 = −2


 T   
x 4 3 0 x
Symmetric: y  3 1 −1 y 
z 0 −1 −1 z

11-3
Quadratic forms

Any quadratic function f (x1 , . . . , xn ) can be written in the

form x T Qx where Q is a symmetric matrix (Q = Q T ).

Proof: Suppose f (x1 , . . . , xn ) = x T Rx where R is not

symmetric. Since it is a scalar, we can take the transpose:

T
x T Rx = x T Rx = x TR Tx

Therefore:

x T Rx = 1
x T Rx + x T R T x = x T 12 (R + R T )x


2

So we’re done, because 12 (R + R T ) is symmetric!

11-4
Orthogonal decomposition

Theorem. Every real symmetric matrix Q = Q T ∈ Rn×n

can be decomposed into a product:

Q = UΛU T

where Λ = diag(λ1 , . . . , λn ) is a real diagonal matrix, and

U ∈ Rn×n is an orthogonal matrix. i.e. it satisfies U T U = I .

This is a useful decomposition because orthogonal matrices

have very nice properties...

11-5
Orthogonal matrices

A matrix U is orthogonal if U T U = I .

ˆ If the columns are U =

 
u1 u2 · · · um , then we have:
   
u1T u1 · · · u1T um 1 ··· 0
U T U =  ... .. ..  =  .. . . .. 

. .  . . .
T T
um u1 · · · um um 0 ··· 1

Columns of U are mutually orthogonal: uiT uj = 0 if i 6= j.

ˆ If U is square, U −1 = U T , and U T is also orthogonal.

11-6
Orthogonal matrices

ˆ columns can be rearranged and the factorization stays valid.

   T
  λ 1 0 0 u1
u1 u2 u3  0 λ2 0   u2T 
0 0 λ3 u3T

= λ1 u1 u1T + λ2 u2 u2T + λ3 u3 u3T

   T
  λ1 0 0 u1
= u1 u3 u2  0 λ3 0  u3T 
0 0 λ2 u2T

11-7
Orthogonal matrices
ˆ Orthogonal matrices preserve angle and (2-norm) distance:

(Ux)T (Uy ) = x T (U T U)y = x T y

In particular, we have kUzk = kzk for any z.

ˆ If Q = UΛU T , then multiply by ui :
 T   
u1T λ1 · · · 0 u1T
 ..   .. . . . . 
Qui =  .   . . ..   ..  ui = λi ui
 
unT 0 · · · λn unT

So multiplication by Q simply scales each ui by λi . In other

words: (λi , ui ) are the eigenvalue-eigenvector pairs of Q.

11-8
Orthogonal matrix example
Rotation matrices are orthgonal:
 
cos θ sin θ
Rθ =
− sin θ cos θ
We can verify this:
  
T cos θ − sin θ cos θ sin θ
Rθ Rθ =
sin θ cos θ − sin θ cos θ
2
 
cos2 θ + sin θ cos θ sin θ − sin θ cos θ
=
sin θ cos θ − cos θ sin θ sin2 θ + cos2 θ
 
1 0
=
0 1

Note: RθT = R−θ . This holds for 3D rotation matrices also...

11-9
Eigenvalues and eigenvectors
If A ∈ Rn×n and there is a vector v and scalar λ such that
Av = λv

Then v is an eigenvector of A and λ is the corresponding

eigenvalue. Some facts:
ˆ Any square matrix has n eigenvalues.
ˆ Each eigenvalue has at least one corresponding eigenvector.
ˆ In general, eigenvalues & eigenvectors can be complex.
ˆ In general, eigenvectors aren’t orthogonal,
 and may not
even be linearly independent. i.e. V = v1 · · · vn may
not be invertible. If it is, we say that A is diagonalizable
and then A = V ΛV −1 . Otherwise, Jordan Canonical Form.
ˆ Symmetric matrices are much simpler!

11-10
Recap: symmetric matrices
ˆ Every symmetric Q = Q T ∈ Rn×n has n real eigenvalues λi .
ˆ There exist n mutually orthogonal eigenvectors u1 , . . . , un :

Qui = λi ui for all i = 1, . . . , n

(
1 if i = j
uiT uj =
0 if i 6= j

ˆ If we define U = u1 · · · un then U T U = I and

 

 
λ1 · · · 0
Q = U  ... . . . ...  U T
 
0 · · · λn

11-11
Eigenvalue example
Consider the quadratic: 7x 2 + 4xy + 6y 2 + 4yz + 5z 2 .
A simple question: are there values that make this negative?
 T   
x 7 2 0 x
equivalent to: y  2 6 2 y 
z 0 2 5 z

Orthogonal decomposition:
  1 2 T
2 2
  
−3 3 0 0 − 13 2

7 2 0 3 3 3 3
2 6 2 =   32 − 13 2 
0 6 0  32 − 13 2

3  3
0 2 5 − 32 − 23 1
0 0 9 − 23 − 23 1
3 3

Eigenvalues are {3, 6, 9}.

11-12
Eigenvalue example
Eigenvalue decomposition:
  1 2 T
2 2
  
−3 3 0 0 − 13 2

7 2 0 3 3 3 3
2 6 2 =  2 − 1
 2 
0
 2
6 0  3 − 13 2
3 3 3  3
0 2 5 − 32 − 23 1
0 0 9 − 23 − 23 1
3 3

Define new coordinates:

   1 2 T
2

−3
 
p 3 3 x
q  =   23 − 13 2
3
y 
r − 23 − 23 1 z
3
Then we can write:
 T      T   
x 7 2 0 x p 3 0 0 p
y  2 6 2 y  = q  0 6 0 q 
z 0 2 5 z r 0 0 9 r
11-13
Eigenvalue example

After some manipulations, we discovered that

7x 2 + 4xy + 6y 2 + 4yz + 5z 2 = 3p 2 + 6q 2 + 9r 2

where:
p = − 31 x + 23 y − 23 z
2
q= 3
x − 31 y − 23 z
2
r= 3
x + 23 y + 31 z

Conclusion: the quadratic can never be negative.

11-14
Recap
Question: Is x T Qx ever negative?

Answer: Look at the orthogonal decomposition of Q:

ˆ Q = UΛU T
ˆ Define new coordinates z = U T x.
ˆ x T Qx = λ1 z12 + · · · + λn zn2

If all λi ≥ 0, then x T Qx ≥ 0 for any x.

If some λk < 0, set zk = 1 and all other zi = 0. Then

find corresponding x using x = Uz, and x T Qx < 0.

11-15
Positive definite matrices

For a matrix Q = Q T , the following are equivalent:

1. x T Qx ≥ 0 for all x ∈ Rn
2. all eigenvalues of Q satisfy λi ≥ 0

A matrix with this property is called positive semidefinite

(PSD). The notation is Q  0.

Note: When we talk about PSD matrices, we always assume

we’re talking about a symmetric matrix.

11-16
Positive definite matrices

Name Definition Notation

Positive semidefinite all λi ≥ 0 Q0
Positive definite all λi > 0 Q0
Negative semidefinite all λi ≤ 0 Q0
Negative definite all λi < 0 Q≺0
Indefinite everything else (none)

Some properties:
ˆ If P  0 then −P  0
ˆ If P  0 and α > 0 then αP  0
ˆ If P  0 and Q  0 then P + Q  0
ˆ Every R = R T can be written as R = P − Q for some
appropriate choice of matrices P  0 and Q  0.
11-17
Ellipsoids

ˆ For linear constraints, the set of x satisfying c T x = b is a

hyperplane and the set c T x ≤ b is a halfspace.

ˆ For quadratic constraints:

If Q  0, the set x T Qx ≤ b is an ellipsoid.

11-18
Ellipsoids

ˆ By orthogonal decomposition, we can write x T Qx = z T Λz

where we defined the new coordinates z = U T x.
ˆ The set of x satisfying x T Qx ≤ 1 corresponds to the set of
z satisfying λ1 z12 + · · · + λn zn2 ≤ 1.
ˆ If Q  0, then λi > 0. In the z coordinates, this is a
stretched sphere (ellipsoid). In the zi direction, it is
stretched by √1λi .
ˆ Since x = Uz, and this transformation preserves angles and
distances (think of it as a rotation), then in the xi
coordinates, it is a rotated ellipsoid.
ˆ The principal axes (the zi directions) map to the ui
directions after the rotation.

11-19
Ellipsoids
Plot of the region

3p 2 + 6q 2 + 9r 2 ≤ 1

Ellipse axes are in the

directions e1 , e2 , e3

Plot of the region

7x 2 + 4xy + 6y 2 + 4yz + 5z 2 ≤ 1

Ellipse axes are in the

directions u1 , u2 , u3
11-20
Norm representation
If Q  0 we can define the matrix square root:
1. Let Q = UΛU T be an orthogonal decomposition
√ √
2. Let Λ1/2 = diag( λ1 , . . . , λn )
3. Define Q 1/2 = UΛ1/2 U T .

We have the property that Q 1/2 is symmetric and

Q 1/2 Q 1/2 = Q. Also:
2
x T Qx = (Q 1/2 x)T (Q 1/2 x) = Q 1/2 x

1/2 2
Therefore: x T Qx ≤ b ⇐⇒ Q x ≤ b

11-21