ENGINEERING COMPUTATION Lecture 4

Stephen Roberts
Michaelmas Term

Computation of matrix eigenvalues and eigenvectors

Topics covered in this lecture:

1. Iterative Power method for approximating the dominant eigenvalue

2. The Rayleigh Quotient method
3. Deflation techniques
4. Awareness of other methods for approximating eigenvalues

Engineering Computation ECL4-1

Motivation
Many problems can be cast as eigenvalue - eigenvector problems. e.g.

Vibrating mechanical systems and resonances. You know that the

eigenvalues give 2 - (resonant frequencies) and the eigenvectors
give the natural modes of vibration.
Electrical coupled - resonant circuits.
Stress and strain calculations.
Geometry transforms using matrices.
Stability of non-linear systems (eg control engineering)

How can we use computers to find eigenvalues and eigenvectors

efficiently?

Engineering Computation ECL4-2

1
 Notes before we proceed.

1. The numerical methods discussed in this lecture come into

their own when dealing with large matrices. We will use
them on small matrices (~ 33) for demonstration
purposes, even though the eigenvalues and eigenvectors
could be found directly!
2. In practice use the routines found in packages (e.g.
function eig in MATLAB). They are optimised and will
avoid nasties.
3. Well keep to symmetric matrices - general non-
symmetric matrices are much harder!

Engineering Computation ECL4-3

Revision on eigenvalues and eigenvectors

The eigenvalues or characteristic roots of an NN matrix A are the N real or

complex number i such that the equation Ax = x has non-trivial solutions 1,
2, 3, ...,N .

If A is symmetric then the eigenvalues are real.

Rewriting Ax = x as (A I )x = 0 , the s are the roots of the polynomial

determinant equation (A I ) = 0 .
The eigenvectors or characteristic vectors of A are the set of N-vectors x = u i
(some books use qi ) which are the non-trivial solutions of Ax = i x . i.e.
Au i = i u i for each i.

Recall that, if none of the i are repeated, normalised ( u 2 = 1 ) eigenvectors

form an orthonormal set. i.e. u iT u i = 1 , but u iT u j = 0 for all i j .

Any vector x can be expressed in terms of that orthonormal set:

x = a1u1 + a2u 2 + L + a N u N .

Engineering Computation ECL4-4

2
 Power Method for eigenvalues and eigenvectors

Engineering Computation ECL4-5

Power Method for eigenvalues and eigenvectors

Assume that for a matrix A there is a unique (ie only one) largest eigenvector 1,
say, where 1 = max j , j = 1,K N .
j

Then we can find 1 by the Power method as described below:

Consider the n+1th iteration x n+1 = Ax n .

Lets start with an initial guess x 0 = a1u1 + a2u 2 + L + aN u N expressed in terms

of the eigenvectors of A.

Then x1 = Ax 0 = a1Au1 + a2 Au 2 + L + aN Au N = a11u1 + a22u 2 + L + a N N u N

Do it again x 2 = Ax1 = a112u1 + a2 22u 2 + L + aN 2N u N

After n iterations x n = Ax n1 = a11nu1 + a2n2u 2 + L + aN nN u N

Engineering Computation ECL4-6

3
After n iterations x n = Ax n 1 = a11n u1 + a 2 n2 u 2 + L + a N nN u N

Since we have defined 1 as the largest eigenvalue, eventually the term a11n u1 will
dominate, provided a1 0 , and 1 > 1. In this case the vector x n = a11n u1 will be
parallel to the eigenvector u1 corresponding to the largest eigenvalue.

To ensure that the limit remains finite and nonzero we modify the above slightly to
take the ratio
x n +1 Ax n a11n +1u1 a n +1 u1
= = 1 1n = 1 as n
x n
x n n
a11 u1 a11 u1

The sign of 1 is positive if the sign of the corresponding terms in x n +1

are the same as in x n , and negative if they alternate.
As before, x n +1 will be parallel to the largest eigenvector so we have found both 1
and u1

Engineering Computation ECL4-7

Example
7 4 1
Lets try this on the matrix 4 4 4 which has eigenvalues 12, 6 and 0 and
1 4 7
1 1 1
1 1 1
normalised eigenvectors 1 , 0 and 2 .
3 2 6
1 1 1

Engineering Computation ECL4-8

4
Write a small MATLAB function:

function [x,lambda]=powermat1(A,x0,nit)
% calculates the largest eigenvalue and corresponding eigenvector of
% matrix A by the power method using x0 as the starting vector and
% carrying out nit interactions.
%

x = x0;

for n = 1:nit
xnew = A*x;
lambda = norm(xnew,inf)/norm(x,inf);
fprintf('n = %4d lambda = %g x = %g %g %g \n', n, lambda, x');
x=xnew;
end

x = x/norm(x); %normalise x
fprintf('n = %4d normalised x = %g %g %g\n', n, x');

%end

Engineering Computation ECL4-9

Run it with initial guess x0 = [1 2 3]T :

powermat1(A,[1 2 3]',10)
n = 1 lambda = 10 x = 1 2 3
n = 2 lambda = 10.8 x = 18 24 30
n = 3 lambda = 11.3333 x = 252 288 324
n = 4 lambda = 11.6471 x = 3240 3456 3672
n = 5 lambda = 11.8182 x = 40176 41472 42768
n = 6 lambda = 11.9077 x = 489888 497664 505440
n = 7 lambda = 11.9535 x = 5.92531e+006 5.97197e+006 6.01862e+006
n = 8 lambda = 11.9767 x = 7.13837e+007 7.16636e+007 7.19436e+007
n = 9 lambda = 11.9883 x = 8.58284e+008 8.59963e+008 8.61643e+008
n = 10 lambda = 11.9941 x = 1.03095e+010 1.03196e+010 1.03296e+010
n = 10 normalised x = 0.577068 0.57735 0.57763

You can see that x slowly turns towards the eigenvector and gets very close to
1 = 12 after 10 iterations.

Engineering Computation ECL4-10

5
Now try with starting vector x0 = [0 1 -1]T :

powermat1(A,[0 1 -1]',10)
n = 1 lambda = 3 x = 0 1 -1
n = 2 lambda = 6 x = 3 0 -3
n = 3 lambda = 6 x = 18 0 -18
n = 4 lambda = 6 x = 108 0 -108
n = 5 lambda = 6 x = 648 0 -648
n = 6 lambda = 6 x = 3888 0 -3888
n = 7 lambda = 6 x = 23328 0 -23328
n = 8 lambda = 6 x = 139968 0 -139968
n = 9 lambda = 6 x = 839808 0 -839808
n = 10 lambda = 6 x = 5.03885e+006 0 -5.03885e+006
n = 10 normalised x = 0.707107 0 -0.707107

Note that it does not work in this case. We have found the second largest
eigenvalue instead. The problem is that x 0 is perpendicular to u1 and so a1 = 0
0
in x = a1u1 + a2u 2 + L + a N u N . So we must always have a component
0
of x parallel to u1. A solution is to always try a couple of non parallel starting
vectors.

Engineering Computation ECL4-11

Convergence Properties

If we consider only the two largest eigenvalues, then

n +1

n +1
a1u1 + a2 2 u2
x a11n +1u1 + a2n2 +1u 2 1
= 1
xn a11nu1 + a2n2u 2
n
a1u1 + a2 2 u 2
1
n
2
The proportional error thus decays with successive iterations as .

1
2
Since the reduction per iteration is , this is ; linear convergence.

1

Confirm this by plotting the error in our example: using powermat2.m

Engineering Computation ECL4-12

6
 1
Power method error
10

0
10

-1
10

-2
10
Error

-3
10

-4
10

-5
10

-6
10
0 2 4 6 8 10 12 14 16 18 20
Iteration number

Engineering Computation ECL4-13

Summary:
1. The Power method can be used to find the dominant eigenvalue of a symmetric
matrix.

2. The method has linear convergence.

3. The method requires an initial guess and it is not obvious how this can be chosen
in practice.

4. The method does work if the dominant eigenvalue has multiplicity r.

The estimated eigenvector will then be a linear combination of the r eigenvectors.

Engineering Computation ECL4-14

7
 Rayleigh Quotient method

Engineering Computation ECL4-15

The Rayleigh quotient method .

Modify the power method by calculating the Rayleigh Quotient at each iteration:

(x n )T Ax n
( )
r xn =
(x n )T x n

This can be done with an extra line of code:

rayleigh = (x'*xnew)/(x'*x);

Running this with rayleigh1 gives a far more rapid rate of convergence,
and this shows in the error plot from rayleigh2

Engineering Computation ECL4-16

8
 rayleigh1(A,[1 2 3]',10)
n = 1 lambda = 10 rayleigh = 11.1429
n = 2 lambda = 10.8 rayleigh = 11.76
n = 3 lambda = 11.3333 rayleigh = 11.9381
n = 4 lambda = 11.6471 rayleigh = 11.9844
n = 5 lambda = 11.8182 rayleigh = 11.9961
n = 6 lambda = 11.9077 rayleigh = 11.999
n = 7 lambda = 11.9535 rayleigh = 11.9998
n = 8 lambda = 11.9767 rayleigh = 11.9999
n = 9 lambda = 11.9883 rayleigh = 12
n = 10 lambda = 11.9941 rayleigh = 12
n = 10 normalised x = 0.577068 0.57735 0.577632

Engineering Computation ECL4-17

Comparison
2
of Rayleigh coefficient & Power method error
10

0
10

-2
10

-4
10
Error

-6
Power Method
10

-8
10

-10
10

-12 Rayleigh Method

10
0 2 4 6 8 10 12 14 16 18 20
Iteration number

Engineering Computation ECL4-18

9
Convergence Properties

Following the previous error analysis, considering only the two largest
eigenvalues,

( )
r xn =
( x n ) T Ax n

(a u + a u ) (a u
n
1 1 1 2
n
2 2
T n +1
1 1 1 + a 2 n2 +1u 2 )
(x n ) T x n (a u + a u ) (a u
n
1 1 1 2
n
2 2
T n
1 1 1
n
+ a2 u 2
2 )
2 n +1

1 + a 2 2
2

a12 12 n +1 + a 22 22 n +1 2
a1 1
= = 1
a12 12 n + a 22 22 n a2 2
2 2n

1 + 2
a1 1
In this case the proportional error thus decays with successive iterations as
2n
2

.
1
2

Since the reduction per iteration is 2 , this is ; quadratic or second order
1
convergence and is much faster.
Engineering Computation ECL4-19

Deflation techniques

Engineering Computation ECL4-20

10
Deflation techniques
So far we have considered how to find the largest eigenvalue. Suppose
we have done this, then how do we find the rest?

As a first attempt, note that if we can calculate the inverse matrix A-1, we
can calculate the smallest eigenvalue, because if we pre-multiply the
eigenvalue-eigenvector equation Aui = iui by A-1 and reverse the
equation, we get A 1ui = i 1u i , and so the eigenvalues of the inverse of a
matrix are the inverses of the eigenvalues. Unfortunately, finding A-1 for a
large matrix can pose problems.

A more general way forward, is, after finding the largest eigenvalue 1, to
make it into the smallest by deflation and then go on to find the new
largest one, say 2 .

Engineering Computation ECL4-21

Hotellings deflation:

Consider (A u u )u
1 1
T
1 j ( )
= Au j 1u 1u 1T u j = j u j 1u 1 u 1T u j .

If j = 1, then (A u u )u
1 1
T
1 1 ( )
= 1u 1 1u 1 u 1T u 1 = 0u 1 .
If j 1, then (A u u )u
1 1
T
1 j = j u j 1u 1 (0 ) = j u j .

( )
thus, A 1u 1u 1T has the same eigenvectors as A, and the same eigenvalues
as A except that the largest one has been replaced by 0. thus we can use the
power method with Rayleighs coefficient to find the next biggest and so on.

Engineering Computation ECL4-22

11
Example
7 4 1
For our matrix 4 4 4 which has a largest eigenvalue 1 = 12 , corresponding
1 4 7
1 1 1 1
1 1
to the eigenvector u1 1 , u u T
= 1 1 1 .
3 3
1 1

1 1 1 1

7 4 1 1 1 1 3 0 3
Replace A by B = 4 4 4 1 1 1 = 0 0 0 and apply Rayleigh:
12
3
1 4 7 1 1 1 3 0 3

Engineering Computation ECL4-23

rayleigh1(B,[1 2 3]',4)
n = 1 lambda = 2 raleigh = 0.857143
n = 2 lambda = 6 raleigh = 6
n = 3 lambda = 6 raleigh = 6
n = 4 normalised x = -0.707107 0 0.707107

we get the second eigenvalue immediately. But we might have trouble with the
3rd (why?).

Warning. In practice, Hotellings deflation suffers from rounding errors for large
matrices and has been replaced by superior, but similar, methods such as
Wielandts deflation (look it up in Kreyzig 7th ed., p. 1016) (Burden and
Faires).

Packages such as MATLAB use more sophisticated and robust methods.

Engineering Computation ECL4-24

12
Try MATLAB function eig to get eigenvalues and eigenvectors:

[eigenvectors,eigenvalues] = eig(A)

eigenvectors =
4.0825e-001 -7.0711e-001 5.7735e-001
-8.1650e-001 1.8041e-016 5.7735e-001
4.0825e-001 7.0711e-001 5.7735e-001

eigenvalues =
-3.8257e-015 0 0
0 6.0000e+000 0
0 0 1.2000e+001

Note the rounding errors The 0 values are ~ 10-15, not exactly 0.

Engineering Computation ECL4-25

Illustrative Example

Four unit masses are

arranged in a square (see
diagram) and are
interconnected by springs.
the horizontal and vertical
springs have unit
stiffness, and the diagonal
springs have two times
unit stiffness.

Engineering Computation ECL4-26

13
 x1
1 -1x1

x1
2 x1
2
2 2
x1
2
2

Consider the effect of moving mass 1 a distance x1 to the right.

The spring 12 will exert a force kx1 = 1x1 on mass 1,

and a force 1x1 on mass 2.
x
The movement of spring 13 is only 1 and,
2
resolving horizontally gives a force 1x1 on mass 1 in the x1 direction.

Consideration of the horizontal forces on mass 1 due to movements

of all the masses gives (assuming sinusoidal motion):

m&x&1 = 2 x1 = 2 x1 + 1y1 + 1x 2 + 0 y 2 + 1x3 1y 3 + 0 x 4 + 0 y 4

Engineering Computation ECL4-27

Writing out the similar equations of motion of all four masses,

and reversing the signs gives a matrix eigenvalue/eigenvector equation

Kx = 2 x = x ,

2 1 1 0 1 1 0 0 x1
1 2 0 0 1 1 0 1 y
1
1 0 2 1 0 0 1 1 x2

0 0 1 2 0 1 1 1 and co-ordinate vector x = y 2
where K =
1 1 0 0 2 1 1 0 x3

1 1 0 1 1 2 0 0 y3
0 0 1 1 1 0 2 1 x
4
0 1 1 1 0 0 1 2 y 4

Engineering Computation ECL4-28

14
Run this through MATLAB:

[eigenvectors,eigenvalues] = eig(K)

eigenvectors =

-0.4300 0.3722 -0.5836 -0.1487 -0.2271 -0.1111 -0.3536 0.3536

-0.4451 -0.3342 -0.1527 0.1806 -0.2553 -0.5649 0.3536 -0.3536
0.4300 -0.3722 -0.5836 -0.1487 0.2271 -0.1111 -0.3536 -0.3536
-0.2341 0.3410 0.0683 0.6068 0.4516 -0.0457 -0.3536 -0.3536
0.2493 0.3654 -0.3626 0.2776 -0.4234 0.4080 0.3536 -0.3536
0.2341 -0.3410 0.0683 0.6068 -0.4516 -0.0457 -0.3536 0.3536
-0.2493 -0.3654 -0.3626 0.2776 0.4234 0.4080 0.3536 0.3536
0.4451 0.3342 -0.1527 0.1806 0.2553 -0.5649 0.3536 0.3536

eigenvalues =

2.0000 0 0 0 0 0 0 0
0 2.0000 0 0 0 0 0 0
0 0 -0.0000 0 0 0 0 0
0 0 0 -0.0000 0 0 0 0
0 0 0 0 2.0000 0 0 0
0 0 0 0 0 0.0000 0 0
0 0 0 0 0 0 4.0000 0
0 0 0 0 0 0 0 6.0000

Engineering Computation ECL4-29

Why are the eigenvalues 2.0000 repeated?

what do 0.0000 eigenvalues represent?

What is the maximum frequency?

Can you identify some of the modes of vibration?

Engineering Computation ECL4-30

15
Advanced Methods
This has only been an introductory to some of the methods of finding
eignvalues and eigenvectors. We have focused on methods for finding the
dominant eigenvalue/eigenvector and briefly mentioned deflation methods
which can be used to find the other eigenvalues .
In practice you may come across other methods. In particular,

1. The Inverse Power Method a variant on the power method that is

faster than the latter and finds the closest eigenvalue to the initial guess.
2. Matrix reduction methods such as the QR algorithm (Faires and Burden p
573) for finding all the eigenvalues which in practice are better than
deflation-based methods which suffer from round-off errors.
So why dont we use matrix reduction methods all the time?
Matrix methods are not efficient in some cases, such as for the analysis of
very large systems (eg complex finite element models) which may have
millions of variables and you may only be interested in finding a subset of
the eigenvalues (100s). In this case using the Power or Rayleigh method
is more efficient.

These methods do not form part of the official course syllabus but you should
be aware of their existence and usefulness in practice.

Engineering Computation ECL4-31

16