B.

Linear Operators – Matrices

B.1 Definition

Definition. A Linear Operator is a function A : S → S′ between Vector Spaces S and S′ , such

that for all x, y ∈ S and α ∈ C (or R) the following holds,

Additivity : A(x + y) = Ax + Ay (B.1)

Scalability : A(αx) = αAx. (B.2)

In this tutorial, we only consider Linear Operators between finite-dimensional Complex Vector
Spaces CN and CM with α ∈ C. Consequently, a Linear Operator A is represented by a Matrix
A : CN → CM .

235
Derivation.
′ M
Assume {un }N 1
n=1 and {um }m=1 is an Orthonormal Basis of an abstract N -dimensional vector space
S and the abstract M -dimensional vector space S , respectively, and A : S → S′ is a Linear Operator
′

with y = Ax for all x ∈ S and y ∈ S′ .

Then, with x = [x1 , . . . , xN ]T and y = [y1 , . . . , yM ]T representing the coordinates of x and y with respect
′ M
to the basis {un }N
n=1 and {um }m=1 , respectively, we obtain the Matrix A with y = Ax as follows:

* N
+ N
X X
ym = hy, u′m i = hAx, u′m i = A xn un , u′m = hAun , u′m ixn (B.3)
n=1 n=1

Consequently, the elements of matrix A are obtained by

△ △
[A]m,n = hAun , u′m i and [AT ]n,m = hAun , u′m i. (B.4)

1 For instance the vector space of bandlimited periodic signals.

236
 A : S → S′ , u1 7→ Au1 = A1,1 u′1 + A2,1 u′2
△ △
A1,1 = hAu1 , u′1 i, A2,1 = hAu1 , u′2 i

u1
A1,1 u′1
S S′

Au1
u2 A2,1
u′2

237
B.2 Elementwise Perspective on Matrices

A ∈ CM ×N with [A]m,n = am,n = Re {am,n } + j Im {am,n } ∈ C, (B.5)

and
  
a1,1 a1,2 ··· a1,N x1

 a2,1 a2,2 ··· a2,N 
 x2 

Ax =  .. .. ..  .. 
 . . .  . 
aM,1 aM,2 · · · aM,N xN
 
a1,1 x1 + a1,2 x2 + · · · + a1,N xN

 a2,1 x1 + a2,2 x2 + · · · + a2,N xN 

= .. 
 . 
aM,1 x1 + aM,2 x2 + · · · + aM,N xN
 PN 
n=1 a 1,n x n
 PN a x 
 n=1 2,n n 
= .. .
 . 
PN
n=1 aM,n xn

238
Definition. Matrix, Transpose Matrix, Adjoint Matrix
 
a1,1 a1,2 · · · a1,N
 a2,1 a2,2 · · · a2,N 
 
A =  .. .. .. ,
 . . . 
aM,1 aM,2 · · · aM,N

 
a1,1 a2,1 ··· aM,1
T

 a1,2 a2,2 ··· aM,2 

A = .. .. .. ,
 . . . 
a1,N a2,N · · · aM,N

 
a∗1,1 a∗2,1 ··· a∗M,1
 a∗1,2 a∗2,2 ··· a∗M,2 
AH = 
 
.. .. .. .
 . . . 
a∗1,N a∗2,N · · · a∗M,N

239
The Adjoint Matrix AH ∈ CN ×M of a matrix A ∈ CM ×N has the constituting property


H
hAx, yi = x, AH y ⇔ y H Ax = AH y x, ∀x ∈ CN , y ∈ CM . (B.6)

Matrix Columns, Matrix Rows

 
a1,n

 a2,n 

A = [a1 a2 · · · aN ] with an =  .. 
 . 
aM,n
 
aT1
T
 aT2 
with aTn = [a1,n a2,n · · · aM,n ]
 
A = .. 
 . 
aTN
 
aH 1
 aH 2
  
AH =  with aTn = a∗1,n a∗2,n · · · a∗M,n ,
 
.. 
 . 
aH
N

T
H
with AT = A and AH = A.

240
Definition. Identity Matrix, Zero Matrix, Inverse Matrix, Selection Vector
 
1 0 ··· 0
 0 1
 ··· 0  
I =  .. . . . ..  = diag [1 1 · · · 1]
 . . 
0 0 ··· 1
 
0 0 ··· 0

 0 0 ··· 0 

0= .. .. .. 
 . . . 
0 0 ··· 0

A−1 A = I and A + (−A) = 0

· · 0} 1 0| ·{z
ei = [ 0| ·{z · · 0} ]T ∈ CN with
i−1 zeros N −i zeros
 
  0  
a1,1 a1,2 ··· a1,N . a1,i
  ..   a2,i
 

 a2,1 a2,2 ··· a2,N   


Aei =  .. .. ..   1  =  ..  = ai .
 . . .  .   .
 .. 

aM,1 aM,2 · · · aM,N aM,i
0

241
B.3 Fundamental Subspace Perspective on Matrices

Any Linear Operator A : CN → CM is related to Four Fundamental Subspaces, viz.


Range Space of A : range [A] = Ax| x ∈ CN (B.7)
 
Null Space of A : null [A] = x ∈ CN  Ax = 0 (B.8)
 
Orthogonal Complement of range [A] : range [A]⊥ = null A H
(B.9)
 
Orthogonal Complement of null [A] : null [A]⊥ = range AH . (B.10)

CN
range [A]

null [A] 0
0
CM

242
B.4 Eigenvectors and Eigenvalues

Definition. An Eigenvector of a matrix A ∈ CM ×M is a nonzero vector v ∈ CN such that

Av = λv for some λ ∈ C. (B.11)

The constant λ is the Eigenvalue to the Eigenvector v. The v and its corresponding λ form a
so-called Eigenpair of the matrix A.
For finite dimensional matrices the Set of Eigenvalues is discrete, the maximum number of different
eigenvalues is M .

The Eigenvalues of Self-Adjoint Matrices AH = A are real-valued and the Eigenvectors

form an ONB.
Proof.

(1) λv H v = v H (λv) = v H Av = v H AH v = (Av)H v = (λv)H v = λ∗ v H v ⇒ λ = λ∗

H H H H ∗ H
(2) λi v H H H
j v i = v j (λi v i ) = v j Av i = v j A v i = (Av j ) v i = (λj v j ) v i = λj v j v i ⇒ vH
j vi = 0

since λi and λj are real-valued (1st part) and different for i 6= j by assumption.

243
B.5 Orthogonal Projectors

Best Approximation and Orthogonal Projection

The Projection Operator P : CM → CM is related to the Best Approximation problem, i.e.,

find x̂ ∈ U ⊂ CM that is closest to a given x ∈ CM . As a result, we obtain x̂ = P x ⊥ x − P x.

x ∈ CM

P : CM → U
U⊥

U
x̂ = P x

244
According to the Closest Point Theorem2 we obtain

Existence : ∃x̂ ∈ U such that kx − x̂k ≤ kx − sk for every s ∈ U (B.12)

Orthogonality : x − x̂ ⊥ U (B.13)
Linearity : x̂ = P x and P only depends on U (B.14)
Indempotency : P 2 = P (B.15)
Self-Adjointness : P H = P . (B.16)

Note.
Projection Operators are always Indempotent.
Orthogonal Projectors must be Indempotent and Self-Adjoint.
Oblique Projectors (non-orthogonal projectors) are only indempotent, i.e., P 2 = P , but Self-
Adjointness does not hold.
Oblique Projectors are not considered in this tutorial.

2 The Closest Point Theorem applies for the category of Convex Sets which Linear Subspaces are belonging to.

245
Orthogonal Projection on Subspaces.
Given a pair of matrices A : CN → CM and B : CM → CN and A is a Left-Inverse of B, i.e.,
AB = I, then

BA is a Projector onto range [B] ⊂ CN , (B.17)

since (BA)2 = BABA = B(AB)A = BA.

If BA is also self-adjoint, then

BA is an Orthogonal Projector onto range [B] ⊂ CN , (B.18)

since (BA)2 = BABA = B(AB)A = BA and (BA)H = BA.

Example. Assume the column vectors of U form an ONB of U, then P = U U H is the Orthogonal
Projector onto U. For a proof consider the plugins

A = UH and B = U . (B.19)

246
Orthogonal Projection via Pseudo-Inverse.
If (AH A)−1 exists, then A+ H −1 H
left = (A A) A is a Left-Inverse of A, i.e.,

−1 H
(1) AA+ H
left = A(A A) A is the Orthogonal Projector onto range [A] (B.20)

with so-called Pseudo-Inverse A+

left .

Otherwise, if (AAH )−1 exists, then A+ H H −1

right = A (AA ) is a Right-Inverse of A, i.e.,

H −1
 H
(2) A+ H
right A = A (AA ) A is the Orthogonal Projector onto range A (B.21)

with the Pseudo-Inverse A+

right .

247
Weighted Sum of Disjoint Orthogonal Projectors

Assume rank-R matrices A : CM → CM that can be represented by means of a Weighted Sum of

rank-Ri Disjoint Orthogonal Projectors P i : CM → CM ,
P
X
A= wi P i , (B.22)
i=1

P
with rank R = Pi=1 Ri and range [P i ] ⊥ range [P j ] for all i 6= j, i = 1, . . . , P , i.e., all projectors are
mutually orthogonal to each other.

A weighted sum of disjoint orthogonal projectors A : CM → CM referring to (B.22) is a Normal Matrix

with

AAH = AH A. (B.23)

The respective projectors and weights depend on A.

Note. The weights take Complex Values, i.e., wi ∈ C .

248
B.6 Special Matrices

Definition. A : CM → CM referring to (B.22) is a Self-Adjoint Matrix when

AH = A. (B.24)

Self-Adjoint Matrices are also Normal Matrices.

Note. The weights strictly take Real Values, i.e., wi ∈ R .

Definition. A : CM → CM referring to (B.22) is a Positive Definite Matrix when it is Self-

Adjoint3 and

xH Ax ≥ 0, for all x ∈ CM . (B.25)

Note. The weights strictly take Nonnegative Values, i.e., wi ≥ 0 .

3 There exist more general definitions of Positive Definite Matrices.

249
Eigenpairs of Normal Matrices

Normal Matrices A are Linear Combinations of Orthogonal Projectors, i.e., A =

P P
i=1 wi P i .

Consequently, any uj ∈ range [P j ] and its corresponding weight wj form an Eigenpair of A, since for
every element of range [P j ] we obtain P j uj = uj by definition and P i uj = 0 for i 6= j due to the
orthogonality of projectors, and thus

P
!
X
Auj = w i P i uj (B.26)
i=1
P
X
= wi (P i uj ) = wj uj . (B.27)
i=1

Obviously, each wj and uj form an Eigenpair consisting of an Eigenvector and an Eigenvalue of

the matrix:

Auj = wj uj , for all j = 1, . . . , P. (B.28)

250
Spectral Theorem of Normal Matrices

Since (1) Normal Matrices A are Linear Combinations of Orthogonal Projectors P i and
any uj ∈ range [P j ] and its corresponding weight wj form an Eigenpair of A, and
(2) since any P i can be represented as a Linear Combination of rank-1 projectors ui,j uH
i,j with respect
Ri
to elements of an ONB {ui,j }j=1 , with

range [P i ] = range [U i ] = span [ui,1 · · · ui,Ri ] , (B.29)

with U i = [ui,1 , . . . , ui,Ri ] and Ri is the dimension of range [P i ], we obtain the main result of the
Spectral Theorem of Normal Matrices,

P
X
A= wi P i (B.30)
i=1
P
X Ri
P X
X
= wi U i U H
i = wi ui,j uH
i,j (B.31)
i=1 i=1 j=1
  H 
w1 I R1 ×R1 · · · 0 U1
  .. . .. .
.. . 
  ..  .
 
= U1 · · · UP  . (B.32)
0 · · · wP I RP ×RP UHP

251
B.7 Singular Value Decomposition

Derivation and Properties of the SVD

Any Compact Linear Operator A can be decomposed by means of the Singular Value Decom-
position (SVD) and any Finite Dimensional Matrix A : CN → CM is a compact linear operator.
(1) Since the product of matrices AH A—the so-called Gramian Matrix—is Normal, Self-Adjoint
and Positive Definite,4 the Spectral Theorem can be applied such that
P
X R
X
H
A A= λi P i = λi v i v H
i , (B.33)
i=1 i=1

with rank R = R1 + · · · + RP ≤ min(M, N ) and at least P different weights λi > 0.

 H
The vectors {v i }R
i=1 form an ONB of range A .

H
4 AH A = AH A and v H AH Av = v H λv = λv H v = λ kvk2 ≥ 0.

252
(2) Given the Orthonormal Vectors {v i }R R
i=1 , the vectors {ui }i=1 , with

△ −1
ui = λi 2 Av i , (B.34)

again constitute an ONB:

−1 −1
uH H
j ui = (λj Av j ) λi Av i
2 2

−1 −1 H
= λj 2 λi 2 v H
j A Av i
−1 −1
= λj 2 λi 2 λi v H
j vi

1 ; i=j
= .
0 ; otherwise

Definition.
H
The elements of an ONB {v i }R R
i=1 with Eigenvalues {λi }i=1 of a Gramian Matrix A A are called
Right Singular Vectors of A.
√
The elements of the ONB {ui = 1/ λi Av i }R R
i=1 with Eigenvalues {λi }i=1 of a Gramian Matrix
H
A A are called corresponding Left Singular Vectors of A.

253
The Singular Value Decomposition (SVD) of an arbitrary matrix A : CN → CM is constituted
by

R
X
A= σi ui v H
i , with matrix rank R (B.35)
i=1
uH
j ui = δi,j and v H
j v i = δi,j and σi > 0 i, j = 1, . . . , Q. (B.36)

with δi,j = 1 when i = j and zero otherwise.

The set of Left Singular Vectors {ui }R i=1 of nonzero Singular Values σi > 0 provides an ONB
for the Image Space (range [A] ∈ CM ) of the matrix A, i.e.,

range [A] = span [u1 u2 · · · uR ] . (B.37)

Correspondingly, the Right Singular Vectors

 {v i }R
i=1 of nonzero Singular Values σi > 0 provides
an ONB for the Image Space (range A ∈ C ) of the matrix AH , which is equal to the Complement
H N
 
of the Null Space (null [A]⊥ ≡ range AH ∈ CN ) of the matrix A, i.e.,
 
range AH = null [A]⊥ = span [v 1 v 2 · · · v R ] . (B.38)

254
Derivation.
Applying the matrix A to a vector x can be equivalently expressed by

Ax = A(x− + x⊥ ) = Ax− ,
 
with x− ∈ range AH ≡ null [A]⊥ and x⊥ ∈ null [A].
 H
Since {v i }R
i=1 forms an ONB for range A ≡ null [A]⊥ , we obtain

Ax = Ax−
R
X
=A vivH
i x−
i=1
R
X
= Av i v H
i x−
i=1
XR p
= λi ui v H
i x−
i=1
R p
X
= λi ui v H
i x,
i=1
√
with σi = λi .

255
Sorted Matrix Representation

By sorting the Singular Values according to

σ1 ≥ σ2 ≥ · · · ≥ σR with R = rank [A] ,

we obtain the Sorted Matrix Representation of the SVD,

A = U ΣV H , with (B.39)

U = [u1 u2 · · · uR ] ∈ CM ×R , (B.40)
 
σ1

Σ= ... 
 = diag [σ1 σ2 · · · σR ] ∈ C
R×R
(B.41)
σR

 
vH 1
VH =  ...  ∈ CR×N .
 
(B.42)
vH R

256
Extended Sorted Matrix Representation. By accomplishing the ONB of the Left Singular Vec-
tors (column vectors of U ) and Right Singular Vectors (column vectors of V ) by their Orthog-
onal Complements, and taking into account that Q = min(M, N ),

U⊥ ⊥
R<Q = [uR+1 uR+2 · · · uQ ] and U Q<M = [uQ+1 uQ+2 · · · uM ] (B.43)
V⊥ ⊥
R<Q = [v R+1 v R+2 · · · v Q ] and V Q<N = [v Q+1 v Q+2 · · · v N ] (B.44)
   ⊥ 
range [U ] ⊕ range U ⊥ R<Q ⊕ range U Q<M = C
M
and (B.45)
 ⊥   ⊥  N
range [V ] ⊕ range V R<Q ⊕ range V Q<N = C , (B.46)

we obtain the Extended Sorted Matrix Representation of the SVD of matrices with maxi-
mum rank (R = Q)

  
 ⊥
 Σ


 U , U Q<M VH ; M ≥ N (Tall)

 0
H
A=

U Σ  VH  ; M = N (Square) (B.47)

   V

 U Σ 0 ⊥,H ; M ≤ N (Wide).
V Q<N

257
SVD of Matrices without maximum Rank.
For Tall Matrices AM ≥N with linear dependent columns or Wide Matrices AM ≤N with linear
dependent rows, i.e., with

rank [A] < Q = min{M, N }, (B.48)

we obtain the cases

   
Σ R×R 0R×N −R  
  VH
AM >N = U, U⊥ ⊥
R<Q , U Q<M
 0N −R×R 0N −R×N −R  (B.49)
  V ⊥,H
R<Q
0M −N ×R 0M −N ×N −R
  
  Σ R×R 0R×N −R VH
AM =N = U, U⊥
R<Q ⊥,H (B.50)
0N −R×R 0N −R×N −R V R<Q
 
      VH
  Σ R×R 0R×M −R 0R×N −M  V ⊥,H
AM <N = U, U⊥
R<Q R<Q
. (B.51)
0M −R×R 0M −R×M −R 0M −R×N −M ⊥,H
V Q<N

258
Fundamental Subspaces (continued)

Any Linear Operator A : CN → CM is related to Four Fundamental Subspaces, viz.


Image Space of A : range [A] = Ax| x ∈ CN (B.52)
 
Null Space of A : null [A] = x ∈ CN  Ax = 0 (B.53)
⊥  H

Orthogonal Complement of range [A] : range [A] = null A (B.54)
 
Orthogonal Complement of null [A] : null [A]⊥ = range AH . (B.55)

CN
range [A]

null [A] 0
0
CM

259
ONB.
     
The column vectors of U ext = U , U ⊥ with U ⊥ = U ⊥ R<Q , U⊥
Q<M and V ext = V , V ⊥ with
 
V⊥ = V⊥ R<Q , V ⊥
Q<N from the Extended SVD of a rank-R matrix A ∈ CM ×N constitute the
Orthonormal Bases for the Four Fundamental Subspaces range [A], range [A]⊥ , null [A], and
null [A]⊥ , viz.

range [A] = span [U ext ei | i = 1, . . . , R] (B.56)

⊥
range [A] = span [U ext ei | i = R + 1, . . . , Q] ⊕ span [U ext ei | i = Q + 1, . . . , M ] (B.57)
null [A] = span [V ext ei | i = R + 1, . . . , Q] ⊕ span [V ext ei | i = Q + 1, . . . , N ] (B.58)
null [A]⊥ = span [V ext ei | i = 1, . . . , R] . (B.59)

The Four Fundamental Subspaces of the matrices A : CN → CM and AH : CM → CN form

Orthogonal Complements of the N - and M -dimensional Vector Spaces:
 
CN = null [A] ⊕ range AH , (B.60)
 
CM = range [A] ⊕ null AH . (B.61)

260
References

ˆ A. Papoulis and S. U. Pillai. Probability, Random Variables and Stochastic Processes, 4th edition, Mc
Graw Hill, 2002.
ˆ H. Stark and J. W. Woods. Probability, Random Processes, and Estimation Theory for Engineers, 2nd
edition, Prentice Hall, 1994.
ˆ W. Utschick. Stochastische Signale, Aktuelles Manuskript zur gleichnamigen Vorlesung an der Technis-
chen Universität München, 2012.
ˆ G. Strang. Linear Algebra and Its Applications, Harcourt Brace Jovanovich, Publishers, 1988.
ˆ L. N. Trefethen and D. Bau III. Numerical Linear Algebra, Society for Industrial and Applied Mathematic,
1997.
ˆ W. Utschick. Mathematische Methoden der Signalverarbeitung, Aktuelles Manuskript zur gleichnamigen
Vorlesung an der Technischen Universität München, 2012.

265