CHAPTER 6 EIGENVALUES AND EIGENVECTORS II This chapter has two distinct objectives. One is to explain why Francis’s algorithm works, and the other is to introduce some methods for solving large, sparse eigenvalue problems. These two projects, which might at first seem to be completely different, have several common elements. The first section introduces the important notions of eigenspace and invariant subspace. The latter concept is particularly important for a deeper understanding of eigenvalue problems and algorithms. At the end of Section 6.1 we discuss the computation of invariant subspaces by eigenvalue swapping. In Section 6.2 we introduce subspace iteration, a natural multidimensional gen- eralization of the power method, and simultaneous iteration, a practical way of implementing nested subspace iterations that was once an important tool for solving large, sparse eigenvalue problems. We then introduce simultaneous iteration with a change of coordinate system at each step, which brings us close to an explanation of Francis’s algorithm. The one additional concept that is needed for an understanding of Francis’s al- gorithm is that of Krylov subspaces. Once we have this, we are able to show in Section 6.3 that Francis’s algorithm is indeed simultaneous iteration with changes of coordinate system. This essentially concludes our explanation of Francis’s algorithm. Fundamentals of Matrix Computations, Third Edition. By David S. Watkins 409 Copyright © 2010 John Wiley & Sons, Inc.410 EIGENVALUES AND EIGENVECTORS I However, we also prove a duality theorem that links Francis’s algorithm to inverse iteration and Rayleigh quotient iteration. At the end of the section we make the connection between Francis’s iteration and the QR decomposition, which explains why the Francis algorithm sometimes (well, almost always) goes by the name of QR algorithm. In Section 6.4 we begin our discussion of large, sparse eigenvalue problems, beginning with simultaneous iteration and the shift-and-invert strategy. We then move on to the Arnoldi process, a variant of the Gram-Schmidt process that generates orthogonal bases of Krylov subspaces. The algorithm then seeks approximations to eigenvectors in the Krylov subspaces that are generated. Thus we see that the concept of Krylov subspace is important to both objectives of this chapter. Section 6.5 builds on Section 6.4 by showing how to do implicit restarts of the Arnoldi process. The implicitly-restarted Arnoldi (IRA) process and its variant, the Krylov-Schur algorithm, are currently the most successful methods for solving large, sparse eigenvalue problems. They are often used in conjunction with the shift-and-invert strategy. In situations where methods of Section 6.5 are inadequate, for example when the matrix is too large for the shift-and-invert strategy, an alternative to consider is the class of Jacobi-Davidson methods. These are discussed in the final section of the chapter. 6.1 EIGENSPACES AND INVARIANT SUBSPACES Let A € C"*”, let A be any complex number, and consider the set of vectors Sy = {v EC" | Av = dv}. You can easily check that Sy is a subspace of C”. Exercise 6.1.1 (a) Show that if v1, v2 € S), then vj + ve € Sy. (b) Show that if v € Sy and a € C, then av € S). Thus S) is a subspace of C”. o Since the equation Av = Av can be rewritten as (AJ — A)v = 0, we see that S) =N(AI — A), the null space of AJ — A. The space S) always contains the zero vector. It contains other vectors if and only if ) is an eigenvalue of A, in which case we call S) the eigenspace of A associated with . The notion of an invariant subspace is a useful generalization of eigenspace. Since we will want to talk about both real and complex invariant subspaces, we will begin by introducing some notation to make the presentation more flexible. Let F denote either the real or complex number field. Thus F” and F”*” will denote either R” and R"*" or C" and C"*", depending on the choice of F. Let A € F"*". A subspace S of F” is said to be invariant under A if Ar € S whenever x € S, in other words, AS C S. We will also sometimes bend the language slightly and say that S is an invariant subspace of A. The spaces {0} and F” are both invariant subspaces of A. The next exercise gives some nontrivial examples of invariant subspaces.EIGENSPACES AND INVARIANT SUBSPACES 411 First we recall some notation from Chapter 3. Given 21, r2,..., ZR € F”, span{x),...,2,} denotes the set of all linear combinations of 7), ..., <;. That is, k span{z1,...,24} = {des lq eFi= heh i=1 Notice that there is now some ambiguity in the meaning of span{z,,..., 7x}. The coefficients c; are taken from F. They can be either real or complex, depending on the context. There was no such ambiguity in Chapter 3, because there we always had F=R. Exercise 6.1.2 (a) Let v € F” be any eigenvector of A. Show that span{v} is invariant under A. (b) Let Sy be any eigenspace of A (of dimension 1 or greater). Show that S, is invariant under A. (c) Let v1, ..., ve € F” be any k eigenvectors of A associated with eigenvalues Ai, -++5 Ax. Show that the space S = span{v;,..., vg} is invariant under A. Tt can be shown that if A is semisimple, every invariant subspace has this form. (d) Let Find the eigenspaces of A, and find a space S that is invariant under A and is, not spanned by eigenvectors of A. oa If we think of A asa linear operator mapping F” into F”, then an invariant subspace can be used to reduce the operator in the following sense: suppose S is an invariant subspace under A that is neither {0} nor F", and let A denote the restricted operator that acts on the subspace S, that is, A=A |s. Then A maps S into S. It is clear that every eigenvector of A is an eigenvector of A, and so is every eigenvalue. Thus we can obtain information about the eigensystem of A by studying A, an operator that is simpler than A in the sense that it acts on a lower-dimensional space. The following sequence of theorems and exercises shows how this reduction can be carried out in practice. We begin by characterizing invariant subspaces in the language of matrices. Theorem 6.1.3 Let S be a subspace of F” with a basis a, ..., xy. Thus S = span{x1,...,ap}. LetX =[ 2 +++ aye | © F™**. Then S is invariant under A € F™* if and only if there exists B € F*** such that AX = XB.