You are on page 1of 32

# EE263 Autumn 2007-08

Stephen Boyd

Lecture 15
Symmetric matrices, quadratic forms, matrix
norm, and SVD
eigenvectors of symmetric matrices
quadratic forms
inequalities for quadratic forms
positive semidefinite matrices
norm of a matrix
singular value decomposition
151

## Eigenvalues of symmetric matrices

suppose A Rnn is symmetric, i.e., A = AT
fact: the eigenvalues of A are real
to see this, suppose Av = v, v 6= 0, v Cn
then
v T Av = v T (Av) = v T v =

n
X
i=1

but also
T

v T Av = (Av) v = (v) v =

|vi|2

n
X
i=1

|vi|2

## so we have = , i.e., R (hence, can assume v Rn)

Symmetric matrices, quadratic forms, matrix norm, and SVD

152

## Eigenvectors of symmetric matrices

fact: there is a set of orthonormal eigenvectors of A, i.e., q1, . . . , qn s.t.
Aqi = iqi, qiT qj = ij
in matrix form: there is an orthogonal Q s.t.
Q1AQ = QT AQ =

A = QQT =

n
X

iqiqiT

i=1

## in particular, qi are both left and right eigenvectors

Symmetric matrices, quadratic forms, matrix norm, and SVD

153

Interpretations

replacements
A = QQT
x

QT

QT x

QT x

Ax

## linear mapping y = Ax can be decomposed as

resolve into qi coordinates
scale coordinates by i
reconstitute with basis qi
Symmetric matrices, quadratic forms, matrix norm, and SVD

154

or, geometrically,
rotate by QT
diagonal real scale (dilation) by
rotate back by Q

decomposition
A=

n
X

iqiqiT

i=1

155

example:


1/2
3/2
3/2 1/2


 
T


1
1
1
0
1
1
1
1

=
0 2
2 1 1
2 1 1

A =

q2q2T x
x

q1

q1q1T x

1q1q1T x
q2

2q2q2T x
Ax
156

## proof (case of i distinct)

suppose v1, . . . , vn is a set of linearly independent eigenvectors of A:
Avi = ivi,

kvik = 1

then we have
viT (Avj ) = j viT vj = (Avi)T vj = iviT vj
so (i j )viT vj = 0
for i 6= j, i 6= j , hence viT vj = 0
in this case we can say: eigenvectors are orthogonal
in general case (i not distinct) we must say: eigenvectors can be
chosen to be orthogonal
Symmetric matrices, quadratic forms, matrix norm, and SVD

157

Example: RC circuit
i1
v1

c1

in
vn

resistive circuit

cn

ck v k = ik ,

i = Gv

## G = GT Rnn is conductance matrix of resistive circuit

thus v = C 1Gv where C = diag(c1, . . . , cn)
note C 1G is not symmetric
Symmetric matrices, quadratic forms, matrix norm, and SVD

158

use state xi =

civi, so
x = C 1/2v = C 1/2GC 1/2x

where C

1/2

## = diag( c1, . . . , cn)

we conclude:
eigenvalues 1, . . . , n of C 1/2GC 1/2 (hence, C 1G) are real
eigenvectors qi (in xi coordinates) can be chosen orthogonal
eigenvectors in voltage coordinates, si = C 1/2qi, satisfy
C 1Gsi = isi,

## Symmetric matrices, quadratic forms, matrix norm, and SVD

sTi Csi = ij

159

Quadratic forms
a function f : Rn R of the form
f (x) = xT Ax =

n
X

Aij xixj

i,j=1

## is called a quadratic form

in a quadratic form we may as well assume A = AT since
xT Ax = xT ((A + AT )/2)x
((A + AT )/2 is called the symmetric part of A)
uniqueness: if xT Ax = xT Bx for all x Rn and A = AT , B = B T , then
A=B
Symmetric matrices, quadratic forms, matrix norm, and SVD

1510

Examples
kBxk2 = xT B T Bx

Pn1
i=1

(xi+1 xi)2

kF xk2 kGxk2
sets defined by quadratic forms:
{ x | f (x) = a } is called a quadratic surface
{ x | f (x) a } is called a quadratic region

1511

## Inequalities for quadratic forms

suppose A = AT , A = QQT with eigenvalues sorted so 1 n

xT Ax = xT QQT x
= (QT x)T (QT x)
n
X
i(qiT x)2
=
i=1

n
X

(qiT x)2

i=1

= 1kxk2
i.e., we have xT Ax 1xT x
Symmetric matrices, quadratic forms, matrix norm, and SVD

1512

## similar argument shows xT Ax nkxk2, so we have

nxT x xT Ax 1xT x

## note also that

q1T Aq1 = 1kq1k2,

1513

## Positive semidefinite and positive definite matrices

suppose A = AT Rnn
we say A is positive semidefinite if xT Ax 0 for all x
denoted A 0 (and sometimes A  0)
A 0 if and only if min(A) 0, i.e., all eigenvalues are nonnegative
not the same as Aij 0 for all i, j
we say A is positive definite if xT Ax > 0 for all x 6= 0
denoted A > 0
A > 0 if and only if min(A) > 0, i.e., all eigenvalues are positive
Symmetric matrices, quadratic forms, matrix norm, and SVD

1514

Matrix inequalities
we say A is negative semidefinite if A 0
we say A is negative definite if A > 0
otherwise, we say A is indefinite
matrix inequality: if B = B T Rn we say A B if A B 0, A < B
if B A > 0, etc.
for example:
A 0 means A is positive semidefinite
A > B means xT Ax > xT Bx for all x 6= 0
Symmetric matrices, quadratic forms, matrix norm, and SVD

1515

## many properties that youd guess hold actually do, e.g.,

if A B and C D, then A + C B + D
if B 0 then A + B A
if A 0 and 0, then A 0
A2 0

A 6 B,

B 6 A

1516

Ellipsoids

## if A = AT > 0, the set

E = { x | xT Ax 1 }
is an ellipsoid in Rn, centered at 0

s1

s2

1517

1/2

qi, i.e.:

## eigenvectors determine directions of semiaxes

eigenvalues determine lengths of semiaxes
note:
in direction q1, xT Ax is large, hence ellipsoid is thin in direction q1
in direction qn, xT Ax is small, hence ellipsoid is fat in direction qn

## if E = { x | xT Bx 1 }, where B > 0, then E E A B

Symmetric matrices, quadratic forms, matrix norm, and SVD

1518

## Gain of a matrix in a direction

suppose A Rmn (not necessarily square or symmetric)
for x Rn, kAxk/kxk gives the amplification factor or gain of A in the
direction x
obviously, gain varies with direction of input x
questions:
what is maximum gain of A
(and corresponding maximum gain direction)?
what is minimum gain of A
(and corresponding minimum gain direction)?
how does gain of A vary with direction?
Symmetric matrices, quadratic forms, matrix norm, and SVD

1519

Matrix norm
the maximum gain

kAxk
x6=0 kxk
is called the matrix norm or spectral norm of A and is denoted kAk
max

kAxk2
xT AT Ax
T
max
=
max
=

(A
A)
max
2
2
x6=0 kxk
x6=0
kxk
p
so we have kAk = max(AT A)

min kAxk/kxk =
x6=0

## Symmetric matrices, quadratic forms, matrix norm, and SVD

min(AT A)

1520

note that
AT A Rnn is symmetric and AT A 0 so min, max 0
max gain input direction is x = q1, eigenvector of AT A associated
with max
min gain input direction is x = qn, eigenvector of AT A associated with
min

## Symmetric matrices, quadratic forms, matrix norm, and SVD

1521

35 44
44 56

1 2
example: A = 3 4
5 6

AT A =

then kAk =

0.620
0.785
0.785 0.620



90.7
0
0 0.265



0.620
0.785
0.785 0.620

T

max(AT A) = 9.53:



0.620

0.785 = 1,




2.18

0.620
A
= 4.99 = 9.53

0.785
7.78

## Symmetric matrices, quadratic forms, matrix norm, and SVD

1522

min gain is

min(AT A) = 0.514:




0.785

0.620 = 1,



A


0.46

0.785
0.14
=

= 0.514
0.620
0.18

kAxk
9.53
0.514
kxk

1523

n1
consistent
with
vector
norm:
matrix
norm
of
a

R
is

p
max(aT a) = aT a

## for any x, kAxk kAkkxk

scaling: kaAk = |a|kAk
triangle inequality: kA + Bk kAk + kBk
definiteness: kAk = 0 A = 0
norm of product: kABk kAkkBk

1524

## more complete picture of gain properties of A given by singular value

decomposition (SVD) of A:
A = U V T
where
A Rmn, Rank(A) = r
U Rmr , U T U = I
V Rnr , V T V = I
= diag(1, . . . , r ), where 1 r > 0
Symmetric matrices, quadratic forms, matrix norm, and SVD

1525

A = U V T =

r
X

iuiviT

i=1

## i are the (nonzero) singular values of A

vi are the right or input singular vectors of A
ui are the left or output singular vectors of A

## Symmetric matrices, quadratic forms, matrix norm, and SVD

1526

AT A = (U V T )T (U V T ) = V 2V T

hence:
vi are eigenvectors of AT A (corresponding to nonzero eigenvalues)
i =

kAk = 1

## Symmetric matrices, quadratic forms, matrix norm, and SVD

1527

similarly,
AAT = (U V T )(U V T )T = U 2U T

hence:
ui are eigenvectors of AAT (corresponding to nonzero eigenvalues)
i =

## u1, . . . ur are orthonormal basis for range(A)

v1, . . . vr are orthonormal basis for N (A)

1528

Interpretations
A = U V T =

r
X

iuiviT

i=1

V Tx

V T x

Ax

## linear mapping y = Ax can be decomposed as

compute coefficients of x along input directions v1, . . . , vr
scale coefficients by i

## reconstitute along output directions u1, . . . , ur

difference with eigenvalue decomposition for symmetric A: input and
output directions are different
Symmetric matrices, quadratic forms, matrix norm, and SVD

1529

## v1 is most sensitive (highest gain) input direction

u1 is highest gain output direction
Av1 = 1u1

1530

## SVD gives clearer picture of gain as function of input/output directions

example: consider A R44 with = diag(10, 7, 0.1, 0.05)
input components along directions v1 and v2 are amplified (by about
10) and come out mostly along plane spanned by u1, u2
input components along directions v3 and v4 are attenuated (by about
10)
kAxk/kxk can range between 10 and 0.05
A is nonsingular
for some applications you might say A is effectively rank 2

1531

## example: A R22, with 1 = 1, 2 = 0.5

resolve x along v1, v2: v1T x = 0.5, v2T x = 0.6, i.e., x = 0.5v1 + 0.6v2
now form Ax = (v1T x)1u1 + (v2T x)2u2 = (0.5)(1)u1 + (0.6)(0.5)u2
v2
u1
x

v1

Ax

u2

1532