LINEAR AND ROBUST CONTROL SYSTEMS

ELEC 9731

1 - Matrix Methods Review

c Prof. Victor Solo

School of Electrical Engineering and
Telecommunications
UNSW
Sydney

Slide 1
 Topics

1 Matrix and Vector - Basic Properties

2 Linear Dependence and Rank
3 Eigenvalues and Eigenvectors:
Eigenvalue Decomposition (EVD) and
Singular Value Decomposition (SVD)
4 Functions of Matrices
5 Partitioned Matrix Inversion
6 Generalised Inverses
7 Examples

References
G. Strang. Applied Linear Algebra.
T. Kailath. Linear Systems. Appendices.

Slide 2
 (A) Matrix and Vector - Basic Properties
Matrix (Rectangular)
A = Am×n = [aij ]; m rows, n columns
Column vector: x = xn×1 = [xi ]
Adjoint: AT = ATn×m = [aji ]
Conformable product
P
C = AB ↔ cij = k
aik bkj
Inner product
T
Pn
a b= 1
ai bi
Outer product (dyadic product)
cm×1 dpT×1 = [ci dj ]m×n
Positive Definite Matrix
xT Ax > 0, x 6= 0
Positive Semi-Definite Matrix
xT Ax ≥ 0
Symmetric Matrix
A = AT
Hermetian transpose = complex conjugate transpose
AH = (A∗ )T

Slide 3
(A) Matrix and Vector - Basic Properties -
Continued
Norms
v
u n
√ uX
kxk = k x k2 = xT x = t x2i
1
n
X p p1
k x kp = ( |xi | )
1

Angle between vectors

xT y
cos θx,y = kxkkyk

Orthogonal vectors
Π
θx,y = 2 or xT y = 0

Slide 4
 (B) Linear Dependence and Rank
[B1] Linear dependence and vector spaces
A set of m-vectors c1 , · · · , cn is said to be linearly
independent if there is no linear combination
Pn
1 civi that is zero.
The vector space spanned by c1 , · · · , cn is the set
of vectors obtainable from c1 , · · · , cn by addition
and scalar multiplication.
Collect c1 , · · · , cn into a matrix An×m . Then note
that we can express A in terms of its column
vectors or row vectors
 
T
r1
 
 : 
Am×n = [c1 , · · · , cn ] = 
 

 : 
 
T
rm

Slide 5
 (B) Linear Dependence and Rank -
Continued
Forming linear combinations of columns of A is
equivalent to multiplying A into a vector since
 
v1
  n
 :  X
Av = [c1 , · · · , cn ]  = ci vi
 
 : 
  1

vn

Similarly vector pre multiplication generates

linear conbinations of row vectors
 
r1T
  m
 :  X
wT A = [w1 , · · · , wm ]  = wj rjT
 
 : 
  1
T
rm

Slide 6
 (B) Linear Dependence and Rank -
Continued II
[B2] The four fundamental sub spaces
(i) Column space
The column (vector) space of A is the vector space
spanned by the columns of A denoted cs(A). It is also
called the range of A and denoted R(A).
(ii) Row space
The row space is the space spanned by the rows.
Since (wT A)T = AT w it is also the column space of
AT and so denoted cs(AT ) or R(AT ).
(iii) Right Null space
or kernel of A is the space spanned by the null vectors
vr such that Avr = 0. It is denoted N (A) or ker(A).
(iv) Left Null space

or left kernel is the space spanned by the null vectors

wk such that wkT A = 0. Since (wkT A)T = AT wk this is
also the right null space of AT and denoted N (AT ).

Slide 7
 (B) Linear Dependence and Rank -
Continued III
[B3] Rank & Nullity

r = (column) rank ofA

= dim. column space ofA
= dim.cs(A) = dim.R(A)
ν = (column) nullity ofA
= dim. right null space ofA
= dim.N (A)

Clearly r = # linearly independent columns

ν = # dependent columns
So r + ν = n
Similarly
rT = row rank of A = dim R(AT )
νT = row nullity of A = dim N (AT )
and rT + νT = m

Slide 8
 (B) Linear Dependence and Rank -
Continued IV
[B4] Fundamental Theorem of Matrix Algebra
We show every n-vector is in N (A) or R(AT )
If this holds then ν + rT = n but
ν + r = n ⇒ r = rT i.e. column rank=row rank
and this is the fundamental result.
The result is obvious once we write Av = 0 as
 
r1T
 :  v = 0 ↔ riT v = 0
 
 
T
rm
So every null vector is ⊥ to every row vector (and
linear combinations thereof) and any vector ⊥ to
every row vector is automatically a null vector.
Thus the rank of a matrix is its common row and
column rank.
If A is a square matrix we say it has full rank of
rank = order. It is then called non-singular.
Otherwise it is singular.

Slide 9
 - Continued V
[B5] Determinants.
A has a determinant if f it is square. If a linear
combination of columns is added to another
column the determinant is unchanged. So if A is
singular we can make one of its columns all 0′ s.
Then it has zero determinant. So det A = |A| = 0
iff A is singular.
[B6] Inverse
A has a left inverse AL if AL A = I. A has a right
inverse AR if AAR = I
If A is square it has an inverse iff it has full rank.
[Suppose A−1 A = I but for some c 6= 0 ,
Ac = 0 ⇒ A−1 0 = c = 0 a contradiction ].
N.B. A−1 =adjoint(A)/detA
(ABC)−1 = C −1 B −1 A−1

Slide 10
 (C) Eigenvalues and Eigenvectors
[C1] If An×n is a square matrix, a vector p
satisfying Ap = λp for some scalar λ is called a
right eigenvector of A and λ is called an
eigenvalue (eval).
[C2] Similarly a left eigenvector satisfies
′ ′
q T A = λ q T ↔ AT q = λ q
′
and λ is another eigenvalue.
[C3] Left and right eigenvectors are orthogonal for
distinct eigenvalues.
Proof: Let q be a left eigenvector with eval λ.
′
Let p be a right evec with eval λ

⇒ q T Ap = λq T p
′
= λ qT p
′
T
⇒q p = 0 ifλ 6= λ

Slide 11
 (C) Eigenvalues and Eigenvectors -
Continued
[C4] Finding eigenvalues
If A is square we must solve (A − λI)p = 0 to find
λ. This means A − λI is singular and this implies
|A − λI| = 0
Now this equation is an nth degree polynomial in
λ and so has n roots which are the eigenvalues.
Thus there may be repeated eigenvalues. This
polynomial is the characteristic polynomial of A.
[C5] (i) Neither p nor q need be unit vectors when
pT q = 0

(ii) If A is symmetric the evals are real since

Ap = λp ⇒ Ap∗ = λ∗ p∗
⇒ pH Ap = λpH p
also pH Ap = λ∗ pH p
⇒λ = λ∗

Slide 12
 (C) Eigenvalues and Eigenvectors -
Continued II
[C6] Eigenvalue Decomposition (EVD)
(also called spectral decomposition).
Suppose the evals are distinct : λ1 , · · · , λn
Let P = (p1 , · · · , pn ), Q = (q1 , · · · , qn ) be the
corresponding right and left evecs. These are
mutually orthogonal so
QT P = [qiT pj ] = [δij ] = I
⇒ QT = P −1
(The pi are linearly independent so p has full rank
and so has an inverse). Continuing
⇒ P QT = I
 
q1T
 
⇒ (p1 , · · · , pn )  : 

 = I
qnT
n
X
⇒ pi qiT = I
1

Slide 13
 (C) Eigenvalues and Eigenvectors -
Continued III
From this we can get the EVD.

Api = λi pi , i = 1, · · · , n
⇒ Api qiT = λi pi qiT , i = 1, · · · , n
n
X Xn
⇒A pi qiT = λi pi qiT
1 1
n
X
⇒ A = P ΛQT = λi pi qiT
1

Slide 14
 (C) Eigenvalues and Eigenvectors -
Continued IV
[C7] Cayley Hamilton Theorem
From the EVD
A2 = P ΛQT P ΛQT = P Λ2 QT
induction ⇒ Am = P Λm QT
Pn
Now suppose |A − λI| = 1 au λu
Pn
Cayley Hamilton Theorem: Then 1 au Au = 0

Slide 15
 (C) Eigenvalues and Eigenvectors -
Continued V
Proof:
n
X n
X
au Au = au P Λu QT
1 1
n
X
= P( au Λu )QT
1
n
X n
X
= P diag( au λu1 , · · · , au λun )QT
1 1

= P diag(0, · · · , 0)QT
= 0

NB
(i) If A is symmetric then left evecs=right evecs
so Q = P .
(ii) Again note very carefully that in the EVD
even if pi have unit length qi will not (and vice
versa). We have only pTi qi = 1.

Slide 16
 (C) Eigenvalues and Eigenvectors - SVD
[C8] Singular Value Decomposition (SVD)
Am×n is rectangular m > n
So rank (A) ≤ n
Then AATm×m , AT An×n are square and AAT has
rank ≤ n , so it has at least m − n null vectors
⇒ m − n zero evals. Next
λI − AAT = |λI − AT A|
so the remaining n evals of AAT are the same as
those of AT A.
Suppose these n evals are distinct. Since AT A is
positive semidefinite these are non negative so we
denote them ρ21 , · · · , ρ2n .

Slide 17
 (C) Eigenvalues and Eigenvectors - SVD
Continued II
Let u1 , · · · , un be the corresponding evecs. They
are orthogonal (we make them unit length),

AT Aui = ρ2i ui , i = 1, · · · , n (0.1)

let vi = Aui /ρi , (soviT vi = 1) (0.2)
⇒ AT vi = AT Aui /ρi = ρi ui (0.3)

Then consider that further

AAT vi = Aui ρi = ρ2i vi (0.4)

So vi are evecs of AAT with evals ρ2i and they are

mutually orthogonal.
We can now establish the
singular value decomposition of A
n
X
SVD: A = ρi vi uTi = V ΛU T (0.5)
1

(not to be confused with the EV D : they agree

when A is symmetric.)

Slide 18
 (C) Eigenvalues and Eigenvectors - SVD
Continued III
Consider the n × n matrix
n
X n
X
(A − ρi vi uTi )T (A − ρi vi uTi )
1 1
n
X n
X
= (AT − ρi ui viT )(A − ρi vi uTi )
1 1
Xn
= AT A − ρ2i ui uTi
1
n
X n
X
− ρ2i ui uTi + ρ2i ui uTi
1 1
n
X
= AT A − ρ2i ui uTi
1
= 0 by EVD

This establishes the result.

Slide 19
 (C) Eigenvalues and Eigenvectors - SVD
Continued IV
To summarise
(i) (0.2) ⇒ Aui = ρi vi ↔ AT Aui = ρ2i ui
(ii) (0.4) ⇒ AT vi = ρi ui ↔ AAT vi = ρ2i vi
compare to EVD
T
Pn
Api = λi pi , A qi = λi qi , A = 1 λi pi qiT
when A is symmetric vi = ui and ρi = λi

Slide 20
 (C) Eigenvalues and Eigenvectors - SVD
Continued V
(iii) SVD tells about near singularity. Suppose
one singular value ρ0 is small. Then consider the
relation:

Au0 = ρ0 v0
 
u01
 
↔ (c1 , · · · , cn ) 
 : 
 = ρ0 v0
uon
n
X
⇒ ci uoi = ρ0 v0
1

Now v0 is a unit vector so this equation gives a

linear combination of columns of A that is nearly
0.
Similarly the equation AT v0 = ρ0 u0 gives a linear
combination of rows of A that is nearly zero.

Slide 21
 D. Matrix Functions
We already defined f (A) = P f (Λ)QT
[D1] Matrix exponential can then be defined in
several ways. With distinct eigenvalues

eAt = P eΛt QT = P diag(eλi t )QT

∞
X (λi t)r T
= P diag( )Q
0
r!
∞ r
X t
= P diag(λri )QT
0
r!
∞ r
X t r
= A
0
r!
 
0 1
e.g. A =   ⇒ A2 = 0 ⇒ Am = 0, m ≥ 2
0 0
 
At
1 t
⇒e = 
0 1
If A has repeated ergenvalues Am = P J m QT etc..
J is the Jordan form.

Slide 22
 [D2] Matrix Calculus
(a) Matrices
If A(t) = [aij (t)] can then easily show
d
dt A(t)B(t) = A(t) dB dA
dt + B(t) dt
d −1
dt A = −A−1 dA
dt A−1

Similarly
Z ∞
L(A(t)) = A(s) = A(t)e−st dt
0
Z ∞
= [ aij (t)e−st dt]
0
= [aij (s)]

e.g. L(eAt ) = (sI − A)−1

Slide 23
 [D2] Matrix Calculus - Continued
(b) Vectors
If φ = f (x) = f (x1 , · · · , xn )
 T
dφ ∂f ∂f
Then dx = ∂x1 , · · · , ∂xn
If φ=f (x) , then
 
∂φ1 ∂φ2 ∂φm
∂x1 ∂x1 ··· ∂x1
 

 : 

dφ T  
dx =
 : 

 

 : 

∂φ1 ∂φn
∂xn ··· ··· ∂xn

Then can show

d T
c x = c
dx
d
(Ax)T = AT
dx
d T
x Ax = Ax + AT x = 2Ax(Asymmetric)
dx

Slide 24
 [D2] Matrix Calculus - Continued II
[D3] Matrix Integration
RT
0
A(s)ds = [sT0 aij (s)ds]
[D4] Signal Norms
Z ∞
1
k s k2 = ( s2 (t)dt) 2
−∞
Z ∞
p
k s kp = ( |s(t)| dt)1/p ↑k s k∞ = sup |s(t)|
−∞ t

Slide 25
 [E] MATRIX INVERSION
[E1] Partitioned Matrix Inversion
If A,B are square matrices and the indicated
inverses exist, then
 −1
A D
  =
B C
 
A−1 + A−1 D∆c−1 BA−1 −A−1 D∆c−1
 
−1 −1 −1
−∆c BA ∆c

where ∆c = C − BA−1 D
The result is easily checked by direct
multiplication. In a similar way show
 −1
A D
  =
B C
 
−1
∆A −∆−1A DC −1
 
−C −1 B∆−1 A C −1
+ C −1
B∆ −1 −1
A C

where ∆A = A − DC −1 B

Slide 26
 [E] MATRIX INVERSION - Continued
[E2] Matrix Inversion Lemma
Equating the two inverses in section E1 gives
(A − DC −1 B)−1 = A−1 + A−1 D∆c−1 BA−1
or in a more usual notation (change −C −1 with C
and interchange B and D)
(A + BCD)−1 = A−1 − A−1 B∆−1 DA−1
∆ = C −1 + DA−1 B

Slide 27
 [E] MATRIX INVERSION - Continued II
[E3] Determinant Identities
Expand in rows to see that
   
A D A 0
det = det A det C = det
0 C B C

Now use the identity

    
−1
A D A 0 I A D
=
B C B I 0 ∆c

to find that (with ∆c as given in section E1)

 
A D
det = det A det ∆c
B C

Similarly show that this also is

 
A D
det = det C det ∆A
B C

Now set A = I m , C = I n to see from the last two identities

that


det I n − BD = det(I m − DB)

In particular, if b, d are vectors,

det(A + bdT ) = det[(A)(I + A−1 bdT )]

= det(A) det(I + A−1 bdT )
= (1 + dT A−1 b) det A

Slide 28
 F. Generalised Inverses
[Fi)] We say G−m×p is a generalised inverse
(g-inverse) of Gp×m if GG− G = G
T − baT
Example G = ab , then G = aT abT b

Proof:
− T baT T
GG G = ab aT abT b ab = abT = G
[F(ii)] solving Gx = b when G is singular.
Suppose G has rank r and let φ1 , · · · , φk be
linearly independent null vectors i.e.
Gφt = 0, t = 1, · · · , k
If G− is any g-inverse of G, then a general
solution to Gx = b is (assuming there is one)
−
Pk
x = G b + 1 αr φr
where αr are free parameters.

Slide 29
 F. Generalised Inverses - Continued
Proof. Let x0 be a solution so that Gx0 = b , then
Gx = GG− b + 0 = GG− Gx0 = Gx0 = b
Let x1 be another solution so that Gx1 = b, then

G(x1 − x0 ) = b − b = 0
k
X ′
⇒ x1 − x0 = θr φr for some θr s
1
k
X
⇒ x1 = x0 + θr φ r
1
k
X
= G− b + αr φr
1

recovering the general form.

Slide 30
 F. Generalised Inverses - Continued II
[F(iii)] Moore - Penrose g-inverse
Let G have SVD
G U Λ VT
=
p×m p×n n×n n×m

n =rankG ≤ min(p, m). Note that

U T U = In = V T V
Then a g-inverse is
G+ V Λ−1 UT
=
mx×p m×n n×n n×p

It is called the Moore-Penrose g-inverse

Proof

GG+ G = U ΛV T (V Λ−1 U T )U ΛV T
= U ΛΛ−1 ΛV T
= U ΛV T = G

Slide 31
 F. Generalised Inverses - Continued III
[F(iv)] Moore-Penrose Properties

G+ G = V Λ−1 U T U ΛV T
= V Λ−1 ΛV T
T
= V Vm×m (6= Im )
GG+ = U ΛV T V Λ−1 U T
= U ΛΛ−1 U T
T
= U Up×p (6= Ip )

[F(v)] Projection g-inverse

Suppose p > m; if G has rank m then a g-inverse
is

G− = (GT G)−1 GT

Proof: GG− G = G(GT G)−1 GT G = G

Slide 32
 F. Generalised Inverses - Continued IV
[F(vi)] Projection Matrices
Suppose Xn×r has rank r(< n). Find
X⊥ (n − r) × n of rank n − r , orthogonal to X
T
i.e. X⊥ X=0
i.e. all columns of X⊥ are orthogonal to all
columns of X and W = [X, X⊥ ] has full rank.
Projection Lemma:
In×n = X(X T X)−1 X T + X⊥ (X⊥
T
X⊥ )−1 X⊥
T

Proof: Let
 
(X T X)−1 XT
M =  
T
(X⊥ X⊥ )−1 T
X⊥
 
(X T X)−1 X T
⇒ MW =   (X, X⊥ )
T
(X⊥ X⊥ )−1 X⊥
T

 
Ir×r 0
=  
0 I(n−r)×(n−r)

Slide 33
 F. Generalised Inverses - Continued V

= In×n
⇒W = M −1
⇒ WM = I
 
(X T X)−1 X T
= (X, X⊥ )  
T
(X⊥ X⊥ )−1 X⊥
T

= X(X T X)−1 X T + X⊥ (X⊥

T
X ⊥ )− X ⊥
T

as required
N.B. PX = X(X T X)−1 X T is a projection matrix
e.g. if

ξ = Xa + X⊥ b
⇒ PX ξ = X(X T X)−1 X T a + 0
= Xa
∈ column space of X

Slide 34
 F. Generalised Inverses - Continued VI
[F(vii)] Miscellaneous
(a) Column interchange is represented by post
multiplication by a permutation matrix
    
a1 b1 0 1 b1 a1
e.g.     =  
a2 b2 1 0 b2 a2
(ii) If the cols of Y are linearly dependent on the
cols of X then Y = XA for some matrix A
Proof. If v is linearly dependent on ξ1 , · · · , ξr ,
Pr
the columns of X then v = 1 αr ξr , for some
coefficients αr
 
α1
 
⇒ v = (ξ1 , · · · , ξr ) 
 :  = Xα


αr
So there are vectors a1 , · · · , am with
Y = (y1 , · · · , ym )
= (Xa1 , · · · , Xam )
= X(a1 , · · · , .am ) = XA

Slide 35
 G. Examples
[G(i)] Linear Combination of Columns
    
3 1 2 1 0
    
A =  2 2 1  1  =  0 
    

1 3 2 −2 0
     
3 1 2
     
=1× 2 +1× 2 −2×
   
 1


1 3 2

[G(ii)] 2x2 Matrix Inverse

 −1  
a b 1  d −b
  = 
c d ∆ −c a
∆ = det = ad − bc

[G(iii)] Evals and Evecs

 
2 2
A= 
3 1

Slide 36
 G. EXAMPLES - Continued
For evals solve

λ−2 −2
|λI − A| =
−3 λ−1
= (λ − 2)(λ − 1) − 6
= λ2 − 3λ + 2 − 6
= λ2 − 3λ − 4
= (λ − 4)(λ + 1)
⇒ Roots are λ1 , λ2 = 4, −1
For right evecs solve: Api = λi pi i = 1, 2
For λ1
    
(1) (1)
2 2 p1 p1
   = 4 
(1) (1)
3 1 p2 p2
(1) (1) (1)
⇒ 2p1 + 2p2 = 4p1
(1) (1) (1)
3p1 + p2 = 4p2
(1) (1) (1) (1)
⇒ 2p2 = 2p1 ⇒ p2 = p1
Second equation agrees.

Slide 37
 G. EXAMPLES - Continued II
We may take
   
(1)
p1 1
 = 
(1)
p2 1

For λ2
    
(2) (2)
2 2 p1 p1
   = −1 ×  
(2) (2)
3 1 p2 p2
(2) (2) (2)
⇒ 2p1 + 2p2 = −p1
(2) (2) (2)
⇒ 3p1 + p2 = −p2
(2) (2)
⇒ 2p2 = −3p1
(2) 3 (2)
⇒ p2 = − p1
2
and second equation agrees.
   
(2)
p1 2
We may take 
(2)
 =  
p2 −3

Slide 38
 G. EXAMPLES - Continued III
So
 
1 2
P =  
1 −3
⇒ QT = P −1
 
1  −3 −2 
=
−5 −1 1
 
3 2
5 5
=  
1
5 − 51
 
q1T
=  
q2T

which gives left evecs

Slide 39
 G. EXAMPLES IV
 
2 2
Use the same matrix as above A =  
3 1
and find its SVD. We have
  
T 2 3 2 2
A A =    
2 1 3 1
 
8 8
=  
8 10

Repeating the type of calaulations above gives

roots
√ √
ρ1 = 9 + 2 5; ρ2 = 9 − 5 etc...

Slide 40
 Symbolic Computation in Matlab
   
a b 1
e.g. given A =   ,x =  .
0 c −d
Find Ax
Matlab code is

>> syms a
>> syms b
>> syms c
>> syms d

A*b
ans =
[ a-b*d, -c*d]

Slide 41