Jim Lambers

MAT 610
Summer Session 2009-10
Lecture 14 Notes
These notes correspond to Sections 7.3 and 8.2 in the text.
The Eigenvalue Problem: Power Iterations
The Unsymmetric Eigenvalue Problem
We now consider the problem of computing eigenvalues of an n n matrix . For simplicity, we
assume that has eigenvalues `
1
, . . . , `

such that
`
1
`
2
`
3
`

.
We also assume that is diagonalizable, meaning that it has n linearly independent eigenvectors
x
1
, . . . , x

such that x

= `

for i = 1, . . . , n.
Suppose that we continually multiply a given vector x
(0)
by , generating a sequence of vectors
x
(1)
, x
(2)
, . . . dened by
x
()
= x
(1)
=

x
(0)
, / = 1, 2, . . . .
Because is diagonalizable, any vector in

is a linear combination of the eigenvectors, and

therefore we can write
x
(0)
= c
1
x
1
+c
2
x
2
+ +c

.
We then have
x
()
=

x
(0)
=

=1
c

=1
c

= `

1
[
c
1
x
1
+

=2
c

(
`

`
1
)

]
.
Because `
1
`

for i = 2, . . . , n, it follows that the coecients of x

, for i = 2, . . . , n, converge
to zero as / . Therefore, the direction of x
()
converges to that of x
1
. This leads to the most
basic method of computing an eigenvalue and eigenvector, the Power Method:
1
Choose an initial vector q
0
such that q
0

2
= 1
for / = 1, 2, . . . do
z

= q
1
q

= z

,z

2
end
This algorithm continues until q

converges to within some tolerance. If it converges, it con-

verges to a unit vector that is a scalar multiple of x
1
, an eigenvector corresponding to the largest
eigenvalue, `
1
. The rate of convergence is `
1
,`
2
, meaning that the distance between q

and a
vector parallel to x
1
decreases by roughly this factor from iteration to iteration.
It follows that convergence can be slow if `
2
is almost as large as `
1
, and in fact, the power
method fails to converge if `
2
= `
1
, but `
2
= `
1
(for example, if they have opposite signs). It
is worth noting the implementation detail that if `
1
is negative, for example, it may appear that
q

is not converging, as it ip-ops between two vectors. This is remedied by normalizing q

so
that it is not only a unit vector, but also a positive number.
Once the normalized eigenvector x
1
is found, the corresponding eigenvalue `
1
can be computed
using a Rayleigh quotient. Then, deation can be carried out by constructing a Householder
reection 1
1
so that 1
1
x
1
= e
1
, as discussed previously, and then 1
1
1
1
is a matrix with block
upper-triangular structure. This decouples the problem of computing the eigenvalues of into the
(solved) problem of computing `
1
, and then computing the remaining eigenvalues by focusing on
the lower right (n 1) (n 1) submatrix.
This method can be impractical, however, due to the contamination of smaller eigenvalues by
roundo error from computing the larger ones and then deating. An alternative is to compute
several eigenvalues at once by using a block version of the Power Method, called Orthogonal
Iteration. In this method, is multiplied by an n: matrix, with : 1, and then the normalization
of the vector computed by the power method is generalized to the orthogonalization of the block,
through the Q1 factorization. The method is as follows:
Choose an n : matrix Q
0
such that Q

0
Q
0
= 1

for / = 1, 2, . . . do
7

= Q
1
7

= Q

(QR Factorization)
end
Generally, this method computes a convergent sequence {Q

}, as long as Q
0
is not decient in
the directions of certain eigenvectors of

, and `

`
+1
. From the relationship
1

= Q

= Q

Q
1
,
we see that if Q

converges to a matrix Q, then Q

Q = 1 is upper-triangular, and because

Q = Q1, the columns of Q span an invariant subspace.
2
Furthermore, if Q

is a matrix whose columns span (range(Q))

, then
[
Q

(Q

[
Q Q

]
=
[
Q

Q Q

(Q

Q (Q

]
=
[
1 Q

0 (Q

]
.
That is, `() = `(1) `((Q

), and because 1 is upper-triangular, the eigenvalues of 1 are

its diagonal elements. We conclude that Orthogonal Iteration, when it converges, yields the largest
: eigenvalues of .
If we let : = n, then, if the eigenvalues of are separated in magnitude, then generally
Orthogonal Iteration converges, yielding the Schur Decomposition of , = QTQ

. However,
this convergence is generally quite slow. Before determining how convergence can be accelerated,
we examine this instance of Orthogonal Iteration more closely.
For each integer /, we dene T

= Q

. Then, from the algorithm for Orthogonal Iteration,

we have
T
1
= Q

1
Q
1
= Q

1
7

= (Q

1
Q

)1

,
and
T

= Q

= Q

Q
1
Q

1
Q

= Q

1
Q

= Q

(Q

1
Q

)
= 1

(Q

1
Q

).
That is, T

is obtained from T
1
by computing the Q1 factorization of T
1
, and then multi-
plying the factors in reverse order. Equivalently, T

is obtained by applying a unitary similarity

transformation to T
1
, as
T

= 1

(Q

1
Q

) = (Q

1
Q

T
1
(Q

1
Q

) = l

T
1
l

.
If Orthogonal Iteration converges, then T

converges to an upper-triangular matrix T = Q

Q
whose diagonal elements are the eigenvalues of . This simple process of repeatedly computing
the Q1 factorization and multiplying the factors in reverse order is called the Q1 Iteration, which
proceeds as follows:
Choose Q
0
so that Q

0
Q
0
= 1

T
0
= Q

0
Q
0
for / = 1, 2, . . . do T
1
= l

(Q1 factorization)
T

= 1

end
3
It is this version of Orthogonal Iteration that serves as the cornerstone of an ecient algorithm
for computing all of the eigenvalues of a matrix. As described, Q1 iteration is prohibitively expen-
sive, because O(n
3
) operations are required in each iteration to compute the Q1 factorization of
T
1
, and typically, many iterations are needed to obtain convergence. However, we will see that
with a judicious choice of Q
0
, the amount of computational eort can be drastically reduced.
It should be noted that if is a real matrix with complex eigenvalues, then Orthogonal Iteration
or the Q1 Iteration will not converge, due to distinct eigenvalues having equal magnitude. However,
the structure of the matrix T

in Q1 Iteration generally will converge to quasi-upper-triangular

form, with 1 1 or 2 2 diagonal blocks corresponding to real eigenvalues or complex-conjugate
pairs of eigenvalues, respectively. It is this type of convergence that we will seek in our continued
development of the Q1 Iteration.
The Symmetric Eigenvalue Problem
The Power Method, when applied to a symmetric matrix to obtain its largest eigenvalue, is more
eective than for a general matrix: its rate of convergence `
2
,`
1

2
, meaning that it generally
converges twice as rapidly.
Let be an nn symmetric matrix. Even more rapid convergence can be obtained if we consider
a variation of the Power Method. Inverse Iteration is the Power Method applied to ( j1)
1
.
The algorithm is as follows:
Choose x
0
so that x
0

2
= 1
for / = 0, 1, 2, . . . do
Solve (j1)z

= x

for z

x
+1
= z

,z

2
end
Let have eigenvalues `
1
, . . . , `

. Then, the eigenvalues of ( j1)

1
matrix are 1,(`

j),
for i 1, 2, . . . , n. Therefore, this method nds the eigenvalue that is closest to j.
Now, suppose that we vary j from iteration to iteration, by setting it equal to the Rayleigh
quotient
:(x) =
x

x
x

x
,
of which the eigenvalues of are constrained extrema. We then obtain Rayleigh Quotient Iteration:
Choose a vector x
0
, x
0

2
= 1
for / = 0, 1, 2, . . . do
j

= x

Solve (j

1)z

= x

for z

x
+1
= z

,z

2
end
4
When this method converges, it converges cubically to an eigenvalue-eigenvector pair. To see
this, consider the diagonal 2 2 matrix
=
[
`
1
0
0 `
2
]
, `
1
`
2
.
This matrix has eigenvalues `
1
and `
2
, with eigenvectors e
1
and e
2
. Suppose that x

=
[
c

,
where c
2

+:
2

= 1. Then we have
j

= :(x

) =
[
c

]
[
`
1
0
0 `
2
] [
c

]
= `
1
c
2

+`
2
:
2

.
From
j

1 =
[
`
1
(`
1
c
2

+`
2
:
2

) 0
0 `
2
(`
1
c
2

+`
2
:
2

)
]
= (`
1
`
2
)
[
:
2

0
0 c
2

]
,
we obtain
z

=
1
`
1
`
2
[
c

,:
2

,c
2

]
=
1
c
2

:
2

(`
1
`
2
[
c
3

:
3

]
.
Normalizing yields
x
+1
=
1

c
6

+:
6

[
c
3

:
3

]
,
which indicates cubic convergence to a vector that is parallel to e
1
or e
2
, provided c

= :

.
It should be noted that Inverse Iteration is also useful for a general (unsymmetric) matrix , for
nding selected eigenvectors after computing the Schur decomposition = QTQ

, which reveals
the eigenvalues of , but not the eigenvectors. Then, a computed eigenvalue can be used as the
shift j, causing rapid convergence to a corresponding eigenvector. In fact, in practice a single
iteration is sucient. However, when no such information about eigenvalues is available, Inverse
Iteration is far more practical for a symmetric matrix than an unsymmetric matrix, due to the
superior convergence of the Power Method in the symmetric case.
5