L.

Vandenberghe ECE133A (Fall 2019)

5. Orthogonal matrices

• matrices with orthonormal columns

• orthogonal matrices

• tall matrices with orthonormal columns

• complex matrices with orthonormal columns

5.1
 Orthonormal vectors

a collection of real m-vectors a1, a2, . . . , an is orthonormal if

• the vectors have unit norm: kai k = 1

• they are mutually orthogonal: aiT a j = 0 if i , j

Example

 0   1   1 
 0 , 1   1 
√  1 ,
  
√  −1 
2 0  2 0 
 
 −1 
     

Orthogonal matrices 5.2

 Matrix with orthonormal columns

A ∈ Rm×n has orthonormal columns if its Gram matrix is the identity matrix:

T
T 
A A = a1 a2 · · · an a1 a2 · · · an
 

 aT a1 aT1 a2 · · · aT1 an
 T1

 a a1 aT2 a2 · · · aT2 an

 2.

=  . .. ... .. 

 aT a1 aTn a2 · · · aTn an
 
 n


 1 0 ··· 0 
0 1 ··· 0
 
 
= 
 .. .. . . . .. 

0 0 ··· 1
 
 
 

there is no standard short name for “matrix with orthonormal columns”

Orthogonal matrices 5.3

 Matrix-vector product

if A ∈ Rm×n has orthonormal columns, then the linear function f (x) = Ax

• preserves inner products:

(Ax)T (Ay) = xT AT Ay = xT y

• preserves norms:
 1/2
= (xT x)1/2 = k xk

T
k Axk = (Ax) (Ax)

• preserves distances: k Ax − Ayk = k x − yk

• preserves angles:
 T
(Ax) (Ay)
 
xT y

∠(Ax, Ay) = arccos = arccos = ∠(x, y)
k Axk k Ayk k xkk yk

Orthogonal matrices 5.4

 Left-invertibility

if A ∈ Rm×n has orthonormal columns, then

• A is left-invertible with left inverse AT : by definition

AT A = I

• A has linearly independent columns (from page 4.24 or page 5.2):

Ax = 0 =⇒ AT Ax = x = 0

• A is tall or square: m ≥ n (see page 4.13)

Orthogonal matrices 5.5

 Outline

• matrices with orthonormal columns

• orthogonal matrices

• tall matrices with orthonormal columns

• complex matrices with orthonormal columns

 Orthogonal matrix

Orthogonal matrix

a square real matrix with orthonormal columns is called orthogonal

Nonsingularity (from equivalences on page 4.14): if A is orthogonal, then

• A is invertible, with inverse AT :

AT A =I

=⇒ AAT = I
A is square

• AT is also an orthogonal matrix

• rows of A are orthonormal (have norm one and are mutually orthogonal)

Note: if A ∈ Rm×n has orthonormal columns and m > n, then AAT , I

Orthogonal matrices 5.6

 Permutation matrix

• let π = (π1, π2, . . . , πn) be a permutation (reordering) of (1, 2, . . . , n)

• we associate with π the n × n permutation matrix A

Aiπi = 1, Ai j = 0 if j , πi

• Ax is a permutation of the elements of x : Ax = (xπ1, xπ2, . . . , xπn )

• A has exactly one element equal to 1 in each row and each column

Orthogonality: permutation matrices are orthogonal

• AT A = I because A has exactly one element equal to one in each row

n
1 i=j

(AT A)i j = Aki Ak j =
X
k=1
0 otherwise

• AT = A−1 is the inverse permutation matrix

Orthogonal matrices 5.7

 Example

• permutation on {1, 2, 3, 4}

(π1, π2, π3, π4) = (2, 4, 1, 3)

• corresponding permutation matrix and its inverse

 0 1 0 0   0 0 1 0 
0 0 0 1 1 0 0 0
   
T
A =  A−1 = A = 
   
,
1 0 0 0 0 0 0 1

 
0 0 1 0 0 1 0 0
   
   
   

• AT is permutation matrix associated with the permutation

(π̃1, π̃2, π̃3, π̃4) = (3, 1, 4, 2)

Orthogonal matrices 5.8

 Plane rotation

Rotation in a plane

Ax
cos θ − sin θ
 
A=
sin θ cos θ x
θ

Rotation in a coordinate plane in Rn: for example,

 cos θ 0 − sin θ 
A =  0 1 0


 sin θ 0 cos θ 



describes a rotation in the (x1, x3) plane in R3

Orthogonal matrices 5.9

 Reflector

Reflector: a matrix of the form

A = I − 2aaT

with a a unit-norm vector ( kak = 1)

Properties

• a reflector matrix is symmetric

• a reflector matrix is orthogonal

AT A = (I − 2aaT )(I − 2aaT ) = I − 4aaT + 4aaT aaT = I

Orthogonal matrices 5.10

 Geometrical interpretation of reflector

0
H y = (I − aaT )x

line through a and origin

z = Ax = (I − 2aaT )x

• H = {u | aT u = 0} is the (hyper-)plane of vectors orthogonal to a

• if kak = 1, the projection of x on H is given by

y = x − (aT x)a = x − a(aT x) = (I − aaT )x

(see next page)

• reflection of x through the hyperplane is given by product with reflector:

z = y + (y − x) = (I − 2aaT )x

Orthogonal matrices 5.11

 Exercise

suppose kak = 1; show that the projection of x on H = {u | aT u = 0} is

y = x − (aT x)a

• we verify that y ∈ H :

aT y = aT (x − a(aT x)) = aT x − (aT a)(aT x) = aT x − aT x = 0

• now consider any z ∈ H with z , y and show that k x − zk > k x − yk :

k x − zk 2 = k x − y + y − zk 2
= k x − yk 2 + 2(x − y)T (y − z) + k y − zk 2
= k x − yk 2 + 2(aT x)aT (y − z) + k y − zk 2
= k x − yk 2 + k y − zk 2 (because aT y = aT z = 0)
> k x − yk 2

Orthogonal matrices 5.12

 Product of orthogonal matrices

if A1, . . . , Ak are orthogonal matrices and of equal size, then the product

A = A1 A2 · · · Ak

is orthogonal:

AT A = (A1 A2 · · · Ak )T (A1 A2 · · · Ak )
= ATk · · · AT2 AT1 A1 A2 · · · Ak
= I

Orthogonal matrices 5.13

 Linear equation with orthogonal matrix

linear equation with orthogonal coefficient matrix A of size n × n

Ax = b

solution is
x = A−1 b = AT b

• can be computed in 2n2 flops by matrix-vector multiplication

• cost is less than order n2 if A has special properties; for example,

permutation matrix: 0 flops

reflector (given a): order n flops
plane rotation: order 1 flops

Orthogonal matrices 5.14

 Outline

• matrices with orthonormal columns

• orthogonal matrices

• tall matrices with orthonormal columns

• complex matrices with orthonormal columns

 Tall matrix with orthonormal columns

suppose A ∈ Rm×n is tall (m > n) and has orthonormal columns

• AT is a left inverse of A:
AT A = I

• A has no right inverse; in particular

AAT , I

on the next pages, we give a geometric interpretation to the matrix AAT

Orthogonal matrices 5.15

 Range

• the span of a collection of vectors is the set of all their linear combinations:

span(a1, a2, . . . , an) = {x1 a1 + x2 a2 + · · · + xn an | x ∈ Rn }

• the range of a matrix A ∈ Rm×n is the span of its column vectors:

range(A) = { Ax | x ∈ Rn }

Example
 1 0  
  x1  

range(  1 2  ) =  x1 + 2x2

 

 | x1, x2 ∈ R
 
 0 −1 
 
  −x2
  

    

Orthogonal matrices 5.16

 Projection on range of matrix with orthonormal columns

suppose A ∈ Rm×n has orthonormal columns; we show that the vector

AAT b

is the orthogonal projection of an m-vector b on range(A)

AAT b

range(A)

• x̂ = AT b satisfies k A x̂ − bk < k Ax − bk for all x , x̂

• this extends the result on page 2.12 (where A = (1/kak)a)

Orthogonal matrices 5.17

 Proof

the squared distance of b to an arbitrary point Ax in range(A) is

k Ax − bk 2 = k A(x − x̂) + A x̂ − bk 2 (where x̂ = AT b)

= k A(x − x̂)k 2 + k A x̂ − bk 2 + 2(x − x̂)T AT (A x̂ − b)
= k A(x − x̂)k 2 + k A x̂ − bk 2
= k x − x̂k 2 + k A x̂ − bk 2
≥ k A x̂ − bk 2

with equality only if x = x̂

• line 3 follows because AT (A x̂ − b) = x̂ − AT b = 0

• line 4 follows from AT A = I

Orthogonal matrices 5.18

 Outline

• matrices with orthonormal columns

• orthogonal matrices

• tall matrices with orthonormal columns

• complex matrices with orthonormal columns

 Gram matrix

A ∈ Cm×n has orthonormal columns if its Gram matrix is the identity matrix:

H
H 
A A = a1 a2 · · · an a1 a2 · · · an
 

 a H a1 a1H a2 · · · a1H an
 1H

 a a1 a2H a2 · · · a2H an

 2.

=  . .. .. 

 a H a1 anH a2 · · · anH an
 
 n


 1 0 ··· 0 
0 1 ··· 0
 
 
= 
 .. .. . . . .. 

0 0 ··· 1
 
 
 

• columns have unit norm: kai k 2 = aiH ai = 1

• columns are mutually orthogonal: aiH a j = 0 for i , j

Orthogonal matrices 5.19

 Unitary matrix

Unitary matrix

a square complex matrix with orthonormal columns is called unitary

Inverse

AH A =I

=⇒ AAH = I
A is square

• a unitary matrix is nonsingular with inverse AH

• if A is unitary, then AH is unitary

Orthogonal matrices 5.20

 Discrete Fourier transform matrix

√
recall definition from page 3.37 (with ω = e2πj/n and j = −1)

 1 1 1 ··· 1 
1 ω−1 ω−2 ω−(n−1)
 
 ··· 
W =  1 ω−2 ω−4 ω−2(n−1)
 
··· 
 .. .. .. .. 

1 ω−(n−1) ω−2(n−1) ω−(n−1)(n−1)
 

 ··· 


√
the matrix (1/ n)W is unitary (proof on next page):

1 H 1
W W = WW H = I
n n

• inverse of W is W −1 = (1/n)W H
• inverse discrete Fourier transform of n-vector x is W −1 x = (1/n)W H x

Orthogonal matrices 5.21

 Gram matrix of DFT matrix

we show that W H W = nI

• conjugate transpose of W is

 1 1 1 ··· 1 
1 ω2 ωn−1
 
 ω ··· 
WH 1 ω2 ω4 ω2(n−1)
 
=  ··· 
 .. .. .. .. 

1 ωn−1 ω2(n−1) ω(n−1)(n−1)
 

 ··· 


• i, j element of Gram matrix is

(W H W)i j = 1 + ωi− j + ω2(i− j) + · · · + ω(n−1)(i− j)

H H ωn(i− j) − 1
(W W)ii = n, (W W)i j = i− j = 0 if i , j
ω −1
(last step follows from ωn = 1)

Orthogonal matrices 5.22