23 views

Uploaded by ygormesquita

- B.sc. Computer Science
- Ponnapalli PhD Thesis
- eigenvalues and eigenvectors - gilbert strang
- Eigenvalue and Eigen Vectors
- Hopf Bif Phenomena Bajaj
- odeanalysis (1)
- Quantitative Methods for Management Sciences
- Solids and Fluids Turbulent Flow Turbulence Modelling
- Arabic Text Summarization Based on Latent Semantic Analysis to Enhance Arabic Documents Clustering
- ICA Overview
- grassmann.pdf
- Imex
- Properties of MIMO LTI Systems
- differential equation and scientist C O N T E N T S.docx
- Lecture Robust Zhou
- Solns2
- Exploring Time-Dependent Symptom Outcomes
- Whitening Rot
- Drineas CMU 2012
- 2000-amplitude-dependency-of-damping-in-buildings-and-critical-tip-drift-ratio.pdf

You are on page 1of 9

Gregor Gregorcic

Department of Electrical Engineering

University College Cork

IRELAND

e-mail: gregorg@rennes.ucc.ie

Last modied: November 28, 2001

1 Singular Value Decomposition SVD

The singular value decomposition is the appropriate tool for analyzing a mapping from one

vector space into another vector space, possibly with a dierent dimension. Most systems of

simultaneous linear equations fall into this second category.

Any m by n matrix A can be factored into:

A = USV

T

(1)

where U is orthogonal

1

m by m matrix and the columns of the U are the eigenvectors of AA

T

.

Likewise, V is orthogonal n by n matrix and the columns of the V are the eigenvectors

2

of

A

T

A. The matrix S is diagonal and it is the same size as A. Its diagonal entries, also called

sigma,

1

, . . . ,

r

, are the square roots of the nonzero eigenvalues of both AA

T

and A

T

A.

They are the singular values of matrix A and they ll the rst r places on the main diagonal

of S. r is the rank of A.

The connections with AA

T

and A

T

A must hold if the equation 1 is correct. It can be

seen:

AA

T

=

_

USV

T

__

VS

T

U

T

_

= USS

T

U

T

(2)

and similarly

A

T

A = VS

T

SV

T

(3)

From eq. 2, U must be the eigenvector matrix for AA

T

. The eigenvalue matrix in the middle

is SS

T

which is m by m with the eigenvalues

1

=

2

1

, . . . ,

r

=

2

r

on the diagonal. From

eq. 3, V must be the eigenvector matrix for A

T

A. The diagonal matrix S

T

S has the same

1

=

2

1

, . . . ,

r

=

2

r

, but it is n by n.

1

An orthogonal matrix is a square matrix with columns built out of the orthonormal vectors. The vectors

are orthonormal when their lengths are all 1 and their dot products are zero. If the matrix Q is orthogonal

then Q

T

Q = I and Q

T

= Q

1

, the transpose is the inverse.

If the matrix Q is not square and Q

T

Q = I then Q is called orthonormal matrix.

If the columns of the Q are orthogonal vectors, (heir dot products are zero, but their lengths are not all 1)

then Q

T

Q is a diagonal matrix, not the identity matrix.

2

The number is an eigenvalue of matrix M if and only if: det (MI) = 0. This is the characteristic

equation, and each solution has a corresponding eigenvector x: (MI) x = 0 or Mx = x.

1

Example 1: Find the singular value decomposition of A =

_

2 2

1 1

_

.

Solution: Compute AA

T

, nd its eigenvalues (it is generally preferred to put them into

decreasing order) and then nd corresponding unit eigenvectors:

AA

T

=

_

8 0

0 2

_

det

_

AA

T

I

_

= det

_

8 0

0 1

_

= 0

(8 ) (2 ) = 0

1

= 8,

2

= 2

Their corresponding unit eigenvectors are:

AA

T

u

1

=

1

u

1

_

8 0

0 2

_ _

u

11

u

12

_

= 8

_

u

11

u

12

_

8u

11

= 8u

11

u

11

= 1

2u

12

= 8u

12

u

12

= 0

u

1

=

_

1

0

_

AA

T

u

2

=

2

u

2

_

8 0

0 2

_ _

u

21

u

22

_

= 2

_

u

21

u

22

_

8u

21

= 2u

21

u

21

= 0

2u

22

= 2u

22

u

22

= 1

u

2

=

_

0

1

_

The matrix U is then:

U = [ u

1

u

2

] =

_

1 0

0 1

_

.

The eigenvalues of the A

T

A are the same as the eigenvalues of the AA

T

. The eigenvectors

of the A

T

A =

_

5 3

3 5

_

are:

A

T

Av

1

=

1

v

1

_

5 3

3 5

_ _

v

11

v

12

_

= 8

_

v

11

v

12

_

5v

11

+ 3v

12

= 8v

11

v

11

= v

12

3v

11

+ 5v

12

= 8v

12

v

12

= v

11

Choice of v

11

will dene v

12

and vice versa. In general v

11

and v

12

can be any numbers, but

since vector v

1

should have length of 1, the v

11

and v

12

are chosen as follows:

v

1

= 1

_

v

2

11

+v

2

12

= 1 v

11

= v

12

=

1

2

v

1

=

_

1

2

1

2

_

.

Unit eigenvector v

2

is:

A

T

Av

2

=

2

v

2

_

5 3

3 5

_ _

v

21

v

22

_

= 2

_

v

21

v

22

_

5v

21

+ 3v

22

= 2v

21

v

21

= 0

3v

21

+ 5v

22

= 2v

22

v

22

= 1

v

21

= v

22

v

2

=

_

1

2

1

2

_

.

The matrix V is then:

V =

_

v

1

v

2

=

_

1

2

1

2

1

2

1

2

_

and V

T

=

_

1

2

1

2

1

2

1

2

_

.

The matrix S is:

S =

_

1

0

0

2

_

=

_

1

0

0

2

_

=

_

2

2 0

0

2

_

.

2

Finally the SVD of the A is:

A = USV

T

=

_

1 0

0 1

_ _

2

2 0

0

2

_

_

1

2

1

2

1

2

1

2

_

T

A =

_

1 0

0 1

_ _

2

2 0

0

2

_

_

1

2

1

2

1

2

1

2

_

.

Example 2: Find the singular value decomposition of A =

_

2 2

1 1

_

.

Solution: The rank r = 1!

AA

T

=

_

8 4

4 2

_

1

= 10,

2

= 0 u

1

=

_

1

5

2

5

_

.

For the second eigenvalue

1

= 0, equation AA

T

u

2

=

2

u

2

is no use. Since the matrix U

is usually square, another column (vector u

2

in this case) is needed. Eigenvectors u

1

and u

1

must be orthogonal, their dot product is zero (eq. 4), and their length must be 1 (eq. 5).

u

11

u

12

+u

21

u

22

= 0 (4)

u

2

= 1

_

u

2

12

+u

2

22

= 1 (5)

The eigenvector u

1

is:

1

5

u

12

+

2

5

u

22

= 0 u

22

= 2u

12

_

u

2

12

+u

2

22

= 1

_

u

2

12

+ (2u

12

)

2

u

12

=

1

5

u

2

=

_

1

5

2

5

_

and matrix U is:

U = [ u

1

u

2

] =

_

1

5

2

5

1

5

2

5

_

.

A

T

A =

_

5 5

5 5

_

A

T

Av

1

=

1

v

1

_

5 5

5 5

_ _

v

11

v

12

_

= 10

_

v

11

v

12

_

v

1

=

_

1

2

1

2

_

The vector v

2

can be found using eq. 4 and eq. 5:

v

2

=

_

1

2

1

2

_

, the matrix V =

_

1

2

1

2

1

2

1

2

_

and V

T

=

_

1

2

1

2

1

2

1

2

_

.

Matrix S is:

S =

_

1

0

0

2

_

=

_

1

0

0

2

_

=

_

10 0

0 0

_

and SVD of A can be written as:

A = USV

T

=

_

1

5

2

5

1

5

2

5

_

_

10 0

0 0

_

_

1

2

1

2

1

2

1

2

_

.

3

The Matlab function svd() can be used to nd the singular value decomposition. If the ma-

trix A has many more rows than columns, the resulting U can be quite large, but most of

its columns are multiplied by zeros in A. In this situation, the economy sized decomposition

saves both time and storage by producing an m by n U, an n by n S and the same V.

Example 3: For the matrix A

A =

2 0

0 -3

0 0

the full singular value decomposition is

[U,S,V]=svd(A)

U =

0 1 0

-1 0 0

0 0 1

S =

3 0

0 2

0 0

V =

0 1

1 0

For this small problem, the economy size decomposition is only slightly smaller:

[U,S,V]=svd(A,0)

U =

0 1

-1 0

0 0

S =

3 0

0 2

V =

0 1

1 0

4

2 Application of the SVD

2.1 The eective rank

Rank of the matrix is the number of independent rows or equivalently the number of indepen-

dent columns. In computations this can be hard to decide. Counting the pivots [2] is correct

in exact arithmetic, but in real arithmetic the roundo error makes life more complicated.

More stable measure of the rank is to compute A

T

A or AA

T

, which are symmetric but share

the same rank as A. Based on the accuracy of the data, we can set the tolerance, 10

6

for

example, and count the singular values of A

T

A or AA

T

above it. The number of singular

values above the specied tolerance is the eective rank.

2.2 Least Squares and the Pseudoinverse

Vector x is called a least squares solution of linear system

Ax = b (6)

when it minimizes

Ax b . (7)

For any linear system Ax = b, the associate normal system

A

T

Ax = A

T

b (8)

is consistent, and all solutions of the normal system are least squares solutions of Ax = b. If

A has linearly independent columns and if A

T

A is invertible then the system Ax = b has a

unique least squares solution:

x =

_

A

T

A

_

1

A

T

b (9)

If A does not have full rank, the least squares problem still has a solution, but it is no longer

unique. There are many vectors x that minimize Ax b. The optimal solution of the

Ax = b is the vector x, that has the minimum length

3

. This optimal solution is called

x

+

and the matrix which produces x

+

from b is called the pseudoinverse of A. Pseudoinverse

of A is denoted by A

+

, so we have:

x

+

= A

+

b (10)

If the singular value decomposition of A is A = USV

T

(eq.1), the pseudoinverse of A is then:

A

+

= VS

+

U

T

(11)

The singular values

1

, . . . ,

r

are on the diagonal of m by n matrix S, and the reciprocals of

the singular values

1

1

, . . . ,

1

r

are on the diagonal of n by m matrix S

+

. The minimum length

solution to Ax = b is x

+

= A

+

b= VS

+

U

T

b. The pseudoinverse of A

+

is A

++

= A. If A

1

exists, then A

+

= A

1

and the system 6 has a unique least squares solution 9.

Example 4: Find the pseudoinverse of A =

_

1 2 2

.

Solution: The SVD of A is:

A =

_

1 2 2

=

_

1

_

3 0 0

_

1

3

2

3

2

3

2

3

1

3

2

3

2

3

2

3

1

3

_

_

3

Also known as the Minimal Norm Least Squares solution.

5

The pseudoinverse of A is:

A

+

=

_

1 2 2

+

=

_

1

3

2

3

2

3

2

3

1

3

2

3

2

3

2

3

1

3

_

_

_

_

1

3

0

0

_

_[1] =

_

1

9

2

9

2

9

_

_

The Matlab function pinv() can be used to nd the pseudoinverse:

A=pinv([-1 2 2])

A =

-0.1111

0.2222

0.2222

2.3 Illconditioned Least Squares Problem

The Least Squares procedure will fail, when A is rank decient. The best we can do is to

nd x

+

. When A is nearly rank decient, small changes

4

in the vector b will produce wildly

dierent solution vectors. x. In this case the pseudoinverse gives more stable solution.

Example 5: Consider system given in eq. 6, where

A =

_

_

1 2

2 4

3 6

4 8.0001

_

_

and b =

_

_

2

4

6

8

_

_

. (12)

The second column of A is almost rst column multiplied by 2, A is close to being singular.

The Least Squares solution is:

x =

_

A

T

A

_

1

A

T

b =

_

2

0

_

(13)

If the third element in vector b changes from 6 to 5.9999 and 6.0001, Least Squares gives the

following solutions:

x =

_

0.2857

0.8571

_

for b(3) = 5.9999 and x =

_

3.7143

0.8571

_

for b(3) = 6.0001.

It can be seen that solution vector changes from x =

_

0.2857

0.8571

_

to x =

_

3.7143

0.8571

_

given

this very small change in the vector b. The pseudoinverse

5

provides the following solution of

the system given in eq. 12:

x

+

= A

+

b =

_

0.4000

0.8000

_

. (14)

Changes in the b would no longer change the result signicantly:

x =

_

0.4000

0.8000

_

for b(3) = 5.9999 and x =

_

0.4000

0.8000

_

for b(3) = 6.0001

This example illustrates that the use of a pseudoinverse can enhance the stability of our

calculations.

4

Vector b is the data vector and it is usuary corrupted by the noise.

5

Matlab command pinv(A)*[2 4 6 8] will not give the same result as 14. See section 2.3.1.

6

2.3.1 Truncation, Precision and Tolerance

To examine the problem of precision and tolerance it is necessary to look at more than four

digits after the decimal point.

The SVD of A in example 5 is:

A = USV

T

where:

A =

_

_

1 2

2 4

3 6

4 8.0001

_

_

U =

_

_

0.18257321210769 0.19518092543763 0.48445018521904 0.83299426565087

0.36514642421537 0.39036185084826 0.64472592855133 0.54645602873302

0.54771963632306 0.58554277626589 0.59130068078336 0.08663926392867

0.73030015136897 0.68312640770977 0.00000000000709 0.00000000000729

_

_

S =

_

1

0

0

2

0 0

0 0

_

_

=

_

_

12.24751403383832 0

0 0.00003055034170

0 0

0 0

_

_

and

V =

_

0.44721121036078 0.89442838356553

0.89442838356553 0.44721121036078

_

.

In practice, the singular values in S which are close to zero are usually set to zero

6

. The

singular value

2

= 0.00003055034170 is pretty close to zero. There are two possible decisions:

I.) If

2

is not set to zero, the following truncation

7

of the matrices U, S and V can be

made and the Minimal Norm Least Squares solution of the system 12 can be found:

x

+

= A

+

b = VS

+

U

T

b =

_

2.00000000006548

0.00000000002910

_

where:

V =

_

0.44721121036078 0.89442838356553

0.89442838356553 0.44721121036078

_

S

+

=

_

0.08164922262895 0

0 3.273285810744304e + 004

_

U

T

=

_

_

0.18257321210769 0.19518092543763

0.36514642421537 0.39036185084826

0.54771963632306 0.58554277626589

0.73030015136897 0.68312640770977

_

_

T

and

b =

_

_

2

4

6

8

_

_

.

6

Answer to the question what is close to 0 can be dicult task.

7

svd(A,0) can be used in Matlab. It produces the economy size decomposition.

7

This is the same solution as would be obtained from the Matlab function pinv(), which uses

the SVD and sets all singular values that are within machine precision to zero. As can be

seen, this solution is close to The Least Squares solution 13. It was shown in example 5, that

solution vector x found this way, is very sensitive to the noise in the vector b.

II.) If

2

is set to zero, the Minimal Norm Least Squares solution

8

of the system 12 is:

x

+

= A

+

b = VS

+

U

T

b =

_

0.39999573334471

0.79999679999076

_

(15)

where:

V =

_

0.44721121036078

0.89442838356553

_

S

+

= [0.08164922262895]

U

T

=

_

_

0.18257321210769

0.36514642421537

0.54771963632306

0.73030015136897

_

_

T

and

b =

_

_

2

4

6

8

_

_

.

Tolerance in the pinv() can be specied manually

9

. If the tolerance is loose enough

10

, the

pinv() will give the same result as 15.

x=pinv(A,0.0001)*b

x =

0.39999573334471

0.79999679999076

It was shown in example 5, that this solution is less sensitive to the noise in the vector b.

8

The Matlab function svds(A,k) can be used here. k is the number of k largest singular values of A. See

help svds.

9

See help pinv

10

Same problem as remark 6.

8

References

[1] Gilbert Strang. Linear Algebra and Its Applications. Harcourt College Publishers, 3

rd

edition, 1988.

[2] Gilbert Strang. Introduction to Linear Algebra. WellesleyCambridge Press, Box 812060,

Wellesley MA 02181 USA, June 1993.

[3] Pavlina Mizori-Oblak. Matematika za studente tehnike in naravoslovja. Univerza v Ljubl-

jani, Fakulteta za strojnistvo, 4

th

edition, 1991.

[4] The MathWorks, Inc., 24 Prime Park Way, Natick, MA 01760-1500. Using MATLAB,

January 1999.

[5] Howard Anton. Elementary Linear Algebra. John Wiley & Sons, 7

th

edition, December

1993.

[6] Barry M. Wise and Neal B. Gallagher. An Introduction to Linear Algebra. Eigenvector

Research, Inc., 830 Wapato Lake Road, Manson, WA 98831, USA.

9

- B.sc. Computer ScienceUploaded byAakash Santhanam
- Ponnapalli PhD ThesisUploaded bySri Priya Ponnapalli
- eigenvalues and eigenvectors - gilbert strangUploaded byapi-301025469
- Eigenvalue and Eigen VectorsUploaded bymm
- Hopf Bif Phenomena BajajUploaded byDávid Borbély
- odeanalysis (1)Uploaded byperescintra4314
- Quantitative Methods for Management SciencesUploaded bySumudu Adikari
- Solids and Fluids Turbulent Flow Turbulence ModellingUploaded byRenato Al
- Arabic Text Summarization Based on Latent Semantic Analysis to Enhance Arabic Documents ClusteringUploaded byLewis Torres
- ICA OverviewUploaded byArunesh Mittal
- grassmann.pdfUploaded byMathSpace
- ImexUploaded byanandnick3212232
- Properties of MIMO LTI SystemsUploaded byTom Kigezi
- differential equation and scientist C O N T E N T S.docxUploaded byPhcorner Itachi
- Lecture Robust ZhouUploaded bySangeetha Veera
- Solns2Uploaded bythermopolis3012
- Exploring Time-Dependent Symptom OutcomesUploaded byIoana Bălăban
- Whitening RotUploaded byVanidevi Mani
- Drineas CMU 2012Uploaded byjuan perez arrikitaun
- 2000-amplitude-dependency-of-damping-in-buildings-and-critical-tip-drift-ratio.pdfUploaded byDemçe Florjan
- LucariniLSquares2001Uploaded byhankillote
- Topic Decomposition and SummarizationUploaded bymouhabalde
- Amt-142 (Matrices) Unit 1Uploaded byVishal Gupta
- On The Gravity Recommendation SystemUploaded bygravityrd
- 09e69c1ecc724a74d770f496b759a21cf071Uploaded byFlorent Munier
- RafaelUploaded byhemanta57jena
- Fem Odal Updating BookUploaded bysholran
- Lecture 011Uploaded byMahmoud El-Mahdy
- Miscellaneous Algebra Prelim Solutions: University of MinnesotaUploaded bydmorawski
- Pulliam T.H. Solution Methods in CFDUploaded bySanthosh Mamidala

- A Level - Maths - List of FormulaeUploaded bysherlyn may lol
- Fourier SeriesUploaded byAakash Verma
- Fourier TransformUploaded bytiktiktak
- Maths and PhysicsUploaded bySaiSpan
- MMAE 501 Book Matrices.pdfUploaded byAshley Bill
- Learning Polynomials With Neural NetworksUploaded byjoscribd
- Optimality Boundary Conditions for Stochastic Elliptic System Involving Higher Order OperatorUploaded byIJARP Publications
- Practce Question on AodUploaded byJohn Wick
- Alg Complete AssignmentsUploaded bytanzeil
- AGII.pdfUploaded byIgor Bogdanoff
- 08-Kecs-TomaUploaded byelectrotehnica
- AP3114 Lecture WK12 Fourier-Analysis ReviewUploaded byGuoXuanChan
- Methods of Numbers formula sheetUploaded byramiro505
- mathgen-2049188854.pdfUploaded byDiego Koz
- Fundamental Theorem of AlgebraUploaded byja_mufc_scribd
- Wessam Math 119.DocUploaded byDalizanolo Hulu
- Complex 5Uploaded byParul Priya
- Chap-01-Maths Top 10.pdfUploaded byDinesh Kumar
- (SpringerBriefs in Mathematics) Alain Haraux-Nonlinear Vibrations and the Wave Equation-Springer International Publishing (2018)Uploaded byMARCUS.VARANIS8748
- integrals(Raju).pdfUploaded byvinithk
- trigonometry-part-1.pptUploaded byMaria Mae
- CH7m.pdfUploaded byQa Sim
- Legendre Equation ProblemsUploaded byShahbaz Ahmed
- Laplace transform homotopy perturbation method for the approximation of variational problems.pdfUploaded byTanveer Sajid
- 2016=A simple proof of the closed graph theorem=1ppUploaded byDumitru D. DRAGHIA
- (Textbooks in Mathematics) James R. Kirkwood, Bessie H. Kirkwood-Elementary Linear Algebra-Chapman and Hall_CRC_CRC Press (2018)Uploaded byJaime Echeverria
- Math 55 a Honors Abstract AlgebraUploaded byccclll
- Section 6 IntegrationUploaded byAkshay Mehra
- Historical Overview of Kronecker Function RingsUploaded byΧάρης Φραντζικινάκης
- Group Theory(MIlne).pdfUploaded byGuillermo Aleman