L2 - Slides PDF

Matrix Computations: Theory and Numerical Methods
Projectors and Factorization
John Leth
jjl@es.aau.dk
Department of Electronic Systems,
Aalborg University,
Denmark
Agenda
Matrix Computations:
Theory and Numerical
Methods
John Leth
Projectors
Example Projectors
Complementary projectors Example
Complementary projectors
Orthogonal projectors Orthogonal projectors
The SVD and EVD of an orthogonal projectors The SVD and EVD of an
orthogonal projectors
Construction of an orthogonal projector
Construction of an
orthogonal projector
QR Factorization
QR Factorization
The full QR factorization
The full QR factorization Example of how to use the
Example of how to use the QR factorization QR factorization
Gram-Schmidt
Orthogonalization
Gram-Schmidt Orthogonalization Modified Gram-Schmidt
Modified Gram-Schmidt Operation count

A remark
Operation count
Cholesky Factorization
A remark
Existence and uniqueness
Dept. of Electronic Systems,

Aalborg University,
18 Denmark
Projectors
Definition
Methods
John Leth
Definition 3 Projectors
A projector (or projection matrix) is an idempotent matrix P, that is, Example
P2 = P
Orthogonal projectors
(hence P is square) The SVD and EVD of an
Construction of an
QR Factorization
Example of how to use the
QR factorization
Gram-Schmidt
Orthogonalization
Modified Gram-Schmidt
Operation count
A remark
(a) An oblique projection (b) An orthogonal projection

Aalborg University,
18 Denmark
Projectors
Example
Methods
John Leth
Projectors
4 Example
Example Orthogonal projectors

The SVD and EVD of an
Construction of an
" # " # orthogonal projector
1 0 0 1 0 0
QR Factorization
P= 0 1 0 =⇒ P 2 = 0 1 0 idempotent The full QR factorization
0 0 0 0 0 0 Example of how to use the
QR factorization
" # " #
0 1 0 1 0 0 Gram-Schmidt
P= 1 0 0 =⇒ P 2 = 0 1 0 not idempotent Orthogonalization
0 0 0 0 0 0 Operation count
A remark

Aalborg University,
18 Denmark
Projectors
Methods
John Leth
Let P be a projector then Projectors

Example
(I − P)2 = I + P 2 − 2P = I − P 5 Complementary projectors
hence I − P is also a projector, called the complementary projector. The SVD and EVD of an
The word “complementary” comes from the facts that Construction of an
1 range(P) = null(I − P) QR Factorization
2 range(I − P) = null(P) The full QR factorization
3 null(P) ∩ range(P) = range(I − P) ∩ range(P) = {0} QR factorization
Gram-Schmidt
Which implies that Orthogonalization
Any vector x can be written uniquely as: Operation count
x = x1 + x2 with x1 ∈ range(P) and x2 ∈ range(I − P) A remark
(in fact x1 = Px and x2 = (I − P)x ) Cholesky Factorization

By definition this implies that the subspaces range(I − P) and range(P) are
complementary

Aalborg University,
18 Denmark
Projectors
Methods
John Leth
Projectors
Example
6 Orthogonal projectors
Definition The SVD and EVD of an
A projector P which is also hermitian (P = P∗) is called an orthogonal orthogonal projectors
Construction of an
projector. orthogonal projector
The word “orthogonal” comes from the fact that range(I − P) and range(P) QR Factorization
are orthogonal:
QR factorization
∗ def ∗ ∗ ∗ 2
u v = ((I − P)x ) Pz = x (I − P)Pz = x (P − P )z = 0 Gram-Schmidt
Orthogonalization
That is, if we write x = x1 + x2 with x1 ∈ range(P) and x2 ∈ range(I − P), Operation count
then x1 ⊥x2 . A remark

Aalborg University,
18 Denmark
Projectors
The SVD and EVD of an orthogonal projectors
Methods
Let P be an m × m projector, and assume that range(I − P) and range(P) are
John Leth
orthogonal.
Let {q1 , . . . , qm } be any orthonormal basis for Cm such that Projectors
Example
span{q1 , . . . , qn } = range(P) Orthogonal projectors
span{qn+1 , . . . , qm } = range(I − P) = null(P) 7 The SVD and EVD of an

Construction of an
Then with the unitary matrix Q = [q1 · · · qm ] we obtain orthogonal projector
QR Factorization
h i
I 0 def The full QR factorization
PQ = [q1 · · · qn 0 · · · 0] ⇒ Q PQ = n
∗
= Σ. Example of how to use the
0 0 QR factorization
Gram-Schmidt
That is we have constructed an SVD (and EVD) of P: Orthogonalization
h i
I 0 Operation count
P = QΣQ , ∗
Σ= n , Q = [q1 · · · qm ]. A remark
0 0
Which also show that P is an orthogonal projector, and that it has a particular
nice reduced SVD
P = Q̂ Q̂ ∗ , Q̂ = [q1 · · · qn ].

Aalborg University,
18 Denmark
Projectors
Let {a1 , . . . , an } be a set of linear independent vectors in Cm . Theory and Numerical
Methods
John Leth
Construct an orthogonal projector P onto the subspace span{a1 , . . . , an }.
Projectors
Let A = [a1 · · · an ] then the above may be stated equivalently as Example
Construct an orthogonal projector P onto the subspace range(A). Orthogonal projectors
Let b be any vector and y its orthogonal projection onto range(A) (that is orthogonal projectors
8 Construction of an
y = Pb), then orthogonal projector
QR Factorization
ai∗ (b − y ) = 0 for i = 1, . . . , n The full QR factorization
QR factorization
Gram-Schmidt
Orthogonalization
Operation count
A remark
Figure: Orthogonal projection

Aalborg University,
18 Denmark
Projectors
Let {a1 , . . . , an } be a set of linear independent vectors in Cm . Methods
John Leth
Construct an orthogonal projector P onto the subspace span{a1 , . . . , an }.
Projectors
Let A = [a1 · · · an ] then the above may be stated equivalently as Example
Construct an orthogonal projector P onto the subspace range(A). Orthogonal projectors
Let b be any vector and y its orthogonal projection onto range(A) (that is 8 Construction of an
y = Pb), then orthogonal projector
QR Factorization
ai∗ (b − y ) = 0 for i = 1, . . . , n The full QR factorization
QR factorization
or compactly written
Gram-Schmidt
Orthogonalization
∗ def ∗ ∗ ∗
0 = A (b − y ) = A (b − Ax ) = A b − A Ax . Modified Gram-Schmidt
Operation count
A remark
Hence
A∗ Ax = A∗ b ⇒ x = (A∗ A)−1 A∗ b ⇒ y = Ax = A(A∗ A)−1 A∗ b Existence and uniqueness
That is, the (orthogonal) projection P is given by
P = A(A∗ A)−1 A∗ . (or = AA∗ ) if {a1 , . . . , an } is orthonormal.

Aalborg University,
18 Denmark
QR Factorization
Introduction and the reduced QR factorization
Let A = [a1 · · · an ] be an m × n matrix (m ≥ n) with full rank n, and consider Methods
the subspaces John Leth
Projectors
span{a1 } ⊂ span{a1 , a2 } ⊂ · · · span{a1 , . . . , an }
Example
The idea of QR factorization is the construction of a sequence of orthonormal Orthogonal projectors
vectors q1 , q2 , . . . , qn such that The SVD and EVD of an
Construction of an
span{q1 , . . . , qi } = span{a1 , . . . , ai } for all i = 1, . . . , n. orthogonal projector
9 QR Factorization
This yields the following relation (the reduced QR factorization of A) The full QR factorization
QR factorization
A = Q̂ R̂
Gram-Schmidt
Orthogonalization
with Q̂ an m × n matrix with orthonormal columns and R̂ and n × n Modified Gram-Schmidt
Operation count
upper-triangular matrix;
A remark
  Cholesky Factorization
r11 r12 ··· r1n
 .. .. 
r22 . . 
Q̂ = [q1 · · · qn ], R̂ =  rii ̸= 0

.. .. 
.
 
.
rnn
Aalborg University,
18 Denmark
QR Factorization
Methods
Definition John Leth
A QR factorization of A ∈ Cm×n (m ≥ n) is a factorization Projectors

Example
A = QR Complementary projectors
where orthogonal projectors
Construction of an
▶ Q ∈ Cm×m is unitary. orthogonal projector
▶ R ∈ Cm×n is upper triangular. QR Factorization

10 The full QR factorization
The existence and uniqueness of a QR factorization is answered by: QR factorization
Gram-Schmidt
Theorem Orthogonalization
Let A ∈ Cm×n (m ≥ n). Modified Gram-Schmidt
Operation count
▶ A has a QR factorization A remark
▶ If A has full rank then it has a unique reduced QR factorization, with R Cholesky Factorization
having positive diagonal entries. Existence and uniqueness
Similar definition and result holds for m ≤ n

(By duality)

Aalborg University,
18 Denmark
QR Factorization
Example of how to use the QR factorization
Methods
John Leth
Projectors
Example
For given A ∈ Cm×n and b ∈ Cm solve for x : Complementary projectors
b = Ax . The SVD and EVD of an
Construction of an
Note that Q ∗ b = Rx with A = QR. orthogonal projector
▶ Compute a QR factorization A = QR QR Factorization

▶ Compute Q∗b 11 Example of how to use the
QR factorization
▶ Solve Q ∗ b = Rx .
Gram-Schmidt
Orthogonalization
We still need to know how to compute a QR factorization: One way is given
by the Operation count
A remark
Gram-Schmidt Orthogonalization procedure.

Aalborg University,
18 Denmark
QR Factorization
Example of how to use the QR factorization
Methods
John Leth
Projectors
The QR Algorithm is one of the jewels of numerical analysis Example
Construction of an
QR Factorization
12 Example of how to use the
QR factorization
Gram-Schmidt
Orthogonalization
Operation count
A remark
Under suitable conditions (see Lecture 28) the QR Algorithm will converge to Existence and uniqueness
a Schur form (upper triangular) for the matrix A

Aalborg University,
18 Denmark
Gram-Schmidt Orthogonalization
Classical Gram-Schmidt
For a given m × n matrix A = [a1 . . . an ] with QR factorization A = QR, one Methods
John Leth
may find the columns qi of Q = [q1 · · · qn ] and the entries rij of R = [rij ] by
means of the classical Gram-Schmidt iteration Projectors
Example
Construction of Q = [q1 · · · qn ] orthogonal projectors
Construction of an
For i =1 to n
QR Factorization
∗
vi = ai − (q1∗ ai )q1 − · · · − (qi−1 ai )qi−1 The full QR factorization
= ai − (q1 q1∗ )ai − · · · − (qi−1 qi−1
∗
)ai QR factorization
13 Gram-Schmidt
qi = vi /||vi ||2 Orthogonalization
Construction of R = [rij ] Operation count
A remark

qi∗ aj i≠ j, Cholesky Factorization
rij = Existence and uniqueness
||vi ||2 i = j.
The classical Gram-Schmidt iteration is unstable (it is sensitive to the effect of

rounding errors on a computer)
Aalborg University,
18 Denmark
Methods
John Leth
Projectors
The classical Gram-Schmidt iteration can be written on matrix form as: Example
q1 = P1 a1 /||P1 a1 ||, q2 = P2 a2 /||P2 a2 ||, · · · , qn = Pn an /||Pn an ||, Orthogonal projectors

where Construction of an
P1 = I, QR Factorization
∗
Pi = I − Q̂i−1 Q̂i−1 , Q̂i−1 = [q1 · · · , qi−1 ], i ≥ 2. Example of how to use the
QR factorization
Gram-Schmidt
The modified Gram-Schmidt iteration computes each Pi by a sequence of Orthogonalization
i − 1 projections of rank m − 1: 14 Modified Gram-Schmidt
Operation count
∗
Pi = (I − qi−1 qi−1 ) · · · (I − q1 q1∗ ) = P⊥qi−1 · · · P⊥q1 , A remark
with (I − qq ∗ ) the orthogonal projection onto the subspace orthogonal to q. Existence and uniqueness

Aalborg University,
18 Denmark
Operation count
Methods
John Leth
Projectors
Example
Construction of an
Figure: Algorithm for reduced QR factorization
QR Factorization
Theorem QR factorization
The Gram-Schmidt algorithm require ∼ 2mn2 flops to compute a QR Gram-Schmidt

Orthogonalization
factorization of an m × n matrix Modified Gram-Schmidt
15 Operation count
▶ Each addition, subtraction, multiplication, division, or square root counts A remark
as one flop. Cholesky Factorization

▶ The symbol ∼ means Existence and uniqueness
lim (number of flops)/2mn2 = 1.

m,n→∞
Pn Pn Pn
Proof: i=1 j=i+1
4m ∼ i=1
(i)4m ∼ 2mn2
Aalborg University,
18 Denmark
Householder Triangulation
A remark
Methods
John Leth
The Gram-Schmidt (GS) algorithm (giving a reduced QR factorization) can be Projectors

view as successive application of elementary triangular matrices Rk on the Example
right of A ∈ Cm×n Complementary projectors

AR1 R2 · · · Rn = Q̂ triangular orthogonalization orthogonal projectors
Construction of an
with R̂ −1 = R1 R2 · · · Rn .
QR Factorization
Another way of computing the (full) QR factorization is by means of
Householder Triangulation (HT) which can be view as successive application Example of how to use the
of elementary unitary matrices Rk on the left of A QR factorization
Gram-Schmidt
Orthogonalization
Qn · · · Q2 Q1 A = R orthogonal triangulation
Operation count
with Q ∗ = Qn · · · Q2 Q1 . 16 A remark
▶ HT is more numerical stable than GS Cholesky Factorization

▶ The HT algorithm (Alg. 10.1) require ∼ 2mn2 − 2/3n3 flops to compute
a QR factorization of an m × n matrix

Aalborg University,
18 Denmark
Definitions
A hermitian matrix A ∈ Cm×m is positive definite if Matrix Computations:

Methods
x ∗ Ax > 0, for all x ̸= 0. John Leth
Projectors
Note that since A is hermitian Example
y ∗ Ax = x ∗ Ay , Orthogonal projectors
hence x ∗ Ax ∈ R. Construction of an
Definition QR Factorization
A Cholesky factorization of a positive definite A is a factorization The full QR factorization
A = R ∗ R,
QR factorization
Gram-Schmidt
Orthogonalization
where R is upper-triangular with positive diagonal. Modified Gram-Schmidt
Operation count
▶ Cholesky factorization involves only half as many operations as Gaussian A remark
elimination (LU factorization). 17 Cholesky Factorization

▶ R = [rij ] can be computed by Existence and uniqueness
q Pi−1 Pi−1
rii = aii − r2 , rij = (aij − rkj rki )/rii , i ≤ j.
k=1 ki k=1
Hence the entries of R can be computed recursively as

r11 , r12 , · · · , r1m , r22 , r23 , · · · Aalborg University,
18 Denmark
Methods
John Leth
Projectors
Theorem Example
Every positive definite A has a unique Cholesky factorization. Complementary projectors
The factor R in the Cholesky factorization RR ∗ of A are sometimes referred to Construction of an
as the (matrix) square root of A and denoted A1/2 . The notion of a (matrix)
QR Factorization
square root can be generalized to positive semi-definite matrices
QR factorization
Theorem (Horn and Johnson, Theorem 7.2.6 on page 405.) Gram-Schmidt

Orthogonalization
For every positive semi-definite matrix A and every k ∈ N there exists a unique Modified Gram-Schmidt
positive semi-definite matrix B such that A = B k . Operation count

A remark
Hence with k = 2 we get A = BB = BB ∗ , so we call B = A1/2 the (matrix)

18 Existence and uniqueness
square root of A.

Aalborg University,
18 Denmark

L2 - Slides PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

L2 - Slides PDF

Uploaded by

Copyright:

Available Formats

Matrix Computations: Theory and Numerical Methods

Projectors and Factorization

Modified Gram-Schmidt Operation count

Dept. of Electronic Systems,

(a) An oblique projection (b) An orthogonal projection

Dept. of Electronic Systems,

Example Orthogonal projectors

Dept. of Electronic Systems,

Let P be a projector then Projectors

(in fact x1 = Px and x2 = (I − P)x ) Cholesky Factorization

Dept. of Electronic Systems,

Dept. of Electronic Systems,

span{qn+1 , . . . , qm } = range(I − P) = null(P) 7 The SVD and EVD of an

Dept. of Electronic Systems,

Figure: Orthogonal projection

A∗ Ax = A∗ b ⇒ x = (A∗ A)−1 A∗ b ⇒ y = Ax = A(A∗ A)−1 A∗ b Existence and uniqueness

That is, the (orthogonal) projection P is given by

P = A(A∗ A)−1 A∗ . (or = AA∗ ) if {a1 , . . . , an } is orthonormal.

Definition John Leth

A QR factorization of A ∈ Cm×n (m ≥ n) is a factorization Projectors

▶ R ∈ Cm×n is upper triangular. QR Factorization

Similar definition and result holds for m ≤ n

Dept. of Electronic Systems,

▶ Compute a QR factorization A = QR QR Factorization

Dept. of Electronic Systems,

a Schur form (upper triangular) for the matrix A

Dept. of Electronic Systems,

The classical Gram-Schmidt iteration is unstable (it is sensitive to the effect of

q1 = P1 a1 /||P1 a1 ||, q2 = P2 a2 /||P2 a2 ||, · · · , qn = Pn an /||Pn an ||, Orthogonal projectors

Dept. of Electronic Systems,

The Gram-Schmidt algorithm require ∼ 2mn2 flops to compute a QR Gram-Schmidt

as one flop. Cholesky Factorization

lim (number of flops)/2mn2 = 1.

The Gram-Schmidt (GS) algorithm (giving a reduced QR factorization) can be Projectors

right of A ∈ Cm×n Complementary projectors

▶ HT is more numerical stable than GS Cholesky Factorization

Dept. of Electronic Systems,

A hermitian matrix A ∈ Cm×m is positive definite if Matrix Computations:

elimination (LU factorization). 17 Cholesky Factorization

Hence the entries of R can be computed recursively as

Theorem (Horn and Johnson, Theorem 7.2.6 on page 405.) Gram-Schmidt

positive semi-definite matrix B such that A = B k . Operation count

Hence with k = 2 we get A = BB = BB ∗ , so we call B = A1/2 the (matrix)

Dept. of Electronic Systems,

You might also like