Rank and Null Space Calculations Using Matrix Decomposition without Column Interchanges Laslie V. Foster Department of Mathematics and Computer Science San Jose State University San Jose, California 95192 Submitted by James H. Wilkinson ABSTRACT ‘The most widely used stable methods for numerical determination of the rank of matrix A are the sngulas valve decomposition and the QR algonthm with coun lnterchanges. Here two algorithms are presented which determine rank and aallity in 4 numerically table manner without using column interchanges: One algoritan makes, tse of the condition estimator of Cline, Mole, Stewart and Wilkinson and relative to alternative stable algorithms i particularly efficient for sparse matrices, The second algorithm is important ia the case that one wishes to test for rank and mulity while sequentially adding columns to a matrix. 1, INTRODUCTION ‘The most widely used stable methods for numerical determination of the rank or nullity of an MN matrix A are the singular value decomposition (SVD) and the QR algorithm with eolvaan interchanges (see [5], [17]). The singular value decomposition is considered more precise whereas the QR. algorithm with column interchanges is more efficient and in practice usually is sufficiently precise [5]. However in certain applications both algorithms are inefficient. For example with a sparse matrix A the SVD does not take sufficient adwastage of the sparsity of A and the QR algorithm with cohumn interchanges forces an ordering of the columns based on numerical not sparsity considerations. ‘We will present two algorithms which determine rank and aullity in a sumerically stable manner without using coluina interchanges in the sense that although columns axe dropped the columns are not reordered. One LINEAR ALGEBRA AND ITS APPLICATIONS 74:47-71 (1988) a (© Elevier Scien Publishing Co, Ine, 1986 2 Vandert ave. New York, NY 10017 24.0795 /86 43.500 LESLIE V. FOSTER Tnwonem 4.2. Ifs! is the smallest singular value of A,, and if e, and Si, as in step 2 in the above algorithn are calculated on a computer with relative machine precision 1, then (3.8) is true with y=¥N. Furthermore when e, , and so we stop. Thus N(A)=1 is caleulated, and w is in the calculated null space with |} Aw;ii/(iwylf = a2/V10,a LESLIE V, FOSTER If we apply our implementation of Algorithm 2, for k~ 1 we obtain Ry = (column 1 of A), W,=(1), ¢,=1> 0% for k=2, Ry = (columns 1 and 2 of A), %=(-LDF, y= 4/12 > a, and for k=3, Ry = (the first 3 columns of A), %=(—L,1a)"/a, e,=a%/V2 6, and finally, for k= 5 Ws =(0,-2,1), It|/Ifty|| =a > a. Thus N(A)=1 is calculated, and w, = (= 1.1.4.0)" is in the calculated null space with {jAAw,|[/Iw,|| = a?/V2 We can now illustrate some properties of other potential snethods for ull space determination. One scheme would be to accept a column of A as jearly dependent if the elements on a diagonal of the triangular portion of some QR factorization of A are smaller than some specified tolerance rot. For this example such a scheme would select a null space of dimension 0 or 3, both incorrect. An alternative to this idea has been suggested by Heath (15 i R, in A=OR, has a small diagonal entry, one drops the column of A corresponding to the first small entry. The result is retriangularized to form Rg, which is examined for sina diagonal entries, repeating until some Ry has no small entries. H this algorithm is performed on our A with a tolerance ToL =a, no dependences are detected. If Tou is increased the calculated ‘ally will be 1 only for Tot. in the narrow interval a < ToL. < V5a and then the aull vector calculated by assuming the small diagonal entry is zero is 1,~1.0,0)", satisfies ||Awi[/ijwj= a /V2 ~ a, and is not an a? null vector. A thisd potential scheme would be to use our Algorithm 2 except that if an entering column provokes ¢, < Tot, then consider the last entered columa the culprit and drop it rather than Some prior column. Applying this scheme to our example, we obtain e)=1, eg 4/¥2, ¢,~a*/¥2 1, it follows that’ £/wil-<¢/IWul-+ anf Al = ey. + agHllAl, where a, = a,+ a,/W. Finally, due to this inequality, by (A.10), and since by construction E,- I, since jjry||> 1, and by our» LESLIE V, FOSTER bound on |f~é, then A‘vv4l//\sa] <[IEall + anil All + |e1/|\Wll < éy +(a,+a5)ql]All Finally, since }A— A’) N it is highly probable that no column of an MXN matrix A will be exactly linearly Gependent on the other cohimns of A and no vector x will exactly satisfy Ax-= 0, Therefore for computer calculations we need to define the numerical rank and nullity of A. We wil do so with respect to a tolerance ¢ and the Euclidean matrix norm jj Although no single definition of numerical rank is best Sor all applications, rank determination based on singular values is very widely used and provides a point of comparison for other methods of rankRANK AND NULL SPACE CALCULATIONS “6 determination (see [11]). As we see, the following definition will lead to singular values: Deriwrrion 21. The numerical ¢ rank of an Mx N matrix A is defined by RCA) = min (rank(B):|/A ~ Bil diagonal matrix, and U and V are, respectively, Mx M and NN orthogonal matrices. When M > N we will call the diagonal entries s, < sy <--- < Sy the singular values of A. For later convenience we choose to order the singular values so that the first singular value is the sinallest. When M e, (26) RYA)=N-N(A). en50 LESLIE V, FOSTER Proof. The results (24) and (2.6) aro well known [11]. Although the author has not seen (2.3) or (2.5) elsewhere in the forms presented here, these results and (2.7) follow easily from standard results in linear algebra. In the interest of space, we will not present the straightforward proofs here. . ‘The results (2.3) and (2.5) are geometrical —(2.3) stating that © nullity is the largest dimension of a space that is approximately annihilated by A, and (2.5) stating that e rank is the smallest dimension of a space that approxiruates ‘well vectors in the range of A. A subspace $ of dimension N{.A) that satisfies (2.3) will be called an e null space of A. The results (2.4) and (2.6) suggest tone method of calculating the numerical rank of A—via the SVD. For many of our results we will find (2.3) and (2.7) more useful Some other well-known results that will be of use to ws fallow. Ix © R%, then Ly. If52 LESLIE V. FOSTER conditions (3.1) and (3.3) are satisfied, then p < Ly. If (3.1), (32), (3.3) are true and A has a gap in its singular values with no singular values in Passes, (35) then p= NA). Proof. To prove (3.1) and (3.2) imply p > Ly, note that €~ L, follows immediately. To show (3.1) and (3.3) fnply p < Ly, let S be the subspace spanned by Wy eMye Then for w= Wy eS Awl < D Awilivsd< max liAwilllyth, soma oly using (3.1) and propesties of matrix norms. Furthermore cond,(W) = WWijymin, . offal, /||W2l,) is a property of condition numbers and so CS |Wiijllyl|, Iwi. Therefore for any w © S, ||Aw||/|}wl| < eda. gllaDlythiC/(IW lilly) <4. Since § is a subspace of dimen- sion p, by the Courant-Fischer theorem the singular value 3, of A satisfies 5, < Ce. Therefore A has at least p singular values less than or equal to Ce, and p< Ly follows If (3.1)-(.3) and (3.5) are true, then Ly {ollows immediately. a and p= L, 2= MA) . ‘The importance of the result isthe following: If our rank determniration is based on approximate singular values and if (3.1) is trae, then itis natural to let the estimated «nullity determined by the algorithm be p. If (3.1)~(3.3) fare true and & has been chosen to lie in a gap in the singular value of A as in (9.5), then this calculated nullity equals the e mulity based on singular values. When ¢ is close to a singular value, so that (3.5) is uot true, then the calculated « nullity may not agree with the e nullity determixed by singular values. However, we do not feel that this is a serious limitation in or algorithm, since if is not contained in a reasonable sized gap in the singular values of A, then even when singular values are used the physical interpre- tation of numerical ¢ rank is unclear (22). Also, f (3.5) is not true, then Theorem 3.4 sill provides hounds on the ealeulated nullityRANK AND NULL SPACE CALCULATIONS ss We would like to develop algorithms that are simple and that insure that (3.2) and (3) are true with y and C not large. In order to fully develop these algorithms we will deseribe several modifications related to the condition (3.2). The first is introduced to allow inclusion of error analysis in onr development: We will assume that ¢,, cur kth approximate singular valve satisfies p+ (3.6) eS 78, t+ Oay rather than (3.2). Here A, = anil), where » is the relative machine precision and a is assumed to depend only on the dimension of A and is not large. Furthermore, for e, <¢ we assume that teal (1) en 8as instead of the stronger condition &, ~ Awill/{Iwil} Of course, fA, = 0 then (8.6) and (3.7) are equivalent to our earlier conditions. and if 4,0 then (3.6) and (3.7) lead to a theorem similar to Theorem 3.4, as we wilh see. ‘Another modification related to (3.2) is now introduced, since we will find it easiest to estimate an interior singular value s, of A by first estimating the smallest singular value s* of certain submatrices A, of A. Note the use of the superscript to distinguish s* from s,. In our algorithm we will assume that ey ‘our estimate of $4 is related to s* by epee dy Reh p tl (38) with A, as earlier. In addition let us assume that when ¢, < ¢ our algorithm calculates an approximate null vector w, of the submatrix A, such that (3.9) We will see shortly that (3.8) and (3.9) lead directly to (3.6), (2.7), and a theorem like Theorem 3.4. Inthe algorithm below we will assume that ¢, and ‘%, satisfying (2.8) and (3.9) are available. The details of calculating. ¢, and wy will be diseussed in the next section. To describe our algorithms we assume that a parameter ¢is given and thay we wish to determine the ¢ nullity p of an MXN matrix A and ane rullspace S of A. If desired, (2.7) can then be used to obtain a computed34 LESLIE. FOSTER N. Our algorithms 2 Caleulate e, 3. Mey <<, let p+ p +1, determine s satisfying (3.9), and let w, be an Nvector formed by expanding w, puting zeros in elements correspond- ing to eohimmns of A dropped in forming Ay. Let Ay, be Ay with a column dropped corresponding to a largest element in absolute value of Wy (ties may be broken arbitanly), Let k= k-+1 and go to 2 4. Ife, > «, stop, Let the current p be p, the calculated mullity of A, and S= span (wy...) Aucontrunt 2 1 Let Ay=ay, k= 2 Caleulate ¢, 3. Me, eand n eand n=N, stop. Let the current p be p. the calculated nullity, and § = span{wy.....¥,} nelle As we will discuss in more detail later, Algorithm 1 has the advantage that the quantity e, must be determined only p +1 times. On the other hand, the matrices A, of Algorithm 2 will generally be smaller than those of Algo- rithm 1, We now have Twponem 3.10. Let L, be the number of singular values of A tess than cor equal to (€~ 6,,)/7, and let Ly be the number of singular values of A less than or equal to Cle ,). If the conditions (3.1), (3.8), and (3:9) are true, then p > Ly. If the conditions (3.1) and (8.3) are true, then p< Lo. If (3.1), (6.3), (3.8), and (3.9) are true and A has no singular values in Me ecole a) (a1) then p= NA).RANK AND NULL SPACE CALCULATION Proof. By our constructions {ili = [Sl] and 1A |= Aw, Hare trv {ally tre, so that (3.9) implies (3.7). By successive application of the interlace theorem of Section 2, s* < sy, so that (3.6) follows immediately from (3.8). Now the proof of Theorem 3 is valid with minor modifications in some of the inequalities to reflect introduction of 8 . It fs important for the succoss of our algorithms that neither C nor y be large. We will discuss the size of y in the next section. We will now discuss C, we assume without Joss of generality that the calculated mull vectors w,, i= Ls, are normalized 50 that [illo = ‘Vurones 3.11. If the null vectors are normalized so that jl. — then C satisfies Ca ND. (3.12) Proof. Since in each algorithm we dropped columns of A corresponding to a largest component in absolute value of wy, it follows easily that there exists a row permutation matric P, and a column permutation matrix P, such that W’ = PWP, is lower trapozoid with ones on the diagonal and whose elements have magnitude one or less. Let Z be the first p rows of W’. Then for x = R? clearly |Wxi|, > |Z. To obtain a lower bound for |IZx\], we will let y = Zx and assume without loss of generality that lyjl..= 1. Then if Z has components z,,, for i> 1 we have x,=y,—Eiciz, x, and so [x,|<1+ fault ++ +12, I Since |= Iy;| <1, by induetion itis easly proven that Jxi<2!"1 and consequently that ||, <2"—1<2". Now for y= Zx, Weil /Ath > IZ beth = l/s > 1/2. Furthermore, since lw. =1, it follows by properties of matrix noms that Wl]p= max, iw lh 3 the author has constructed matrices A with p=N/2, C= 2, and no singular Values in 0 << 2°~4e/y/p, and yet with NA) p 1(A)+1. Such matrices appear to be exceedingly rare in practice 4. IMPLEMENTATIONS AND ERROR ANALYSIS ‘The results of Section 3 were to some extent general, and they are applicable to a variety of implementations of either algorithm. In this section we wish to provide specific natural implementations, complete the partial error analysis of Section 3, and discuss the size of 7. Implementations of Aigorithea I can be based on subroutines in Linpac Oo, in the case of sparse matrices, other published approaches. To deseribe such implementations the following additions are required to our previous description of Algorithm 1: InpLesenratio’ oF ALcontTus 1 1. Let Ay=A, k=1, p=0, Use the Linpack subroutine sonpe (with or ‘without the pivoting option)—or, in the case of a sparse matrix A, the algprithm of (10)—to determine. R in a QR faconzaton of Ay. Let R= 2. As outlined below, use the LinPac subroutine srxco—or, in the case of sparse matrices, its sparse matrix modification [14]—to determine an approcimate null veetor to Ry fand Ay). Determine e, as outined below.RANK AND NULL SPACE CALCULATIONS 37 3. Ie, <6,let po +1 and let w, be an N-veetor formed by expanding ‘% putting zeros in elements corresponding to columns of A dropped in forming Ay. Using the subroutine sctex—or, for sparse matrices, a suitable modification as described below—drop the column. of Ry (and ‘A,) conesponding to the component of w, with largest magnitude, and retriangularize the result using Givens transformations to form Ry.,. Let kek +1 and go to 2. 4. Wo eg>& stop. Let the current p be p, the calculated nullity. Let $= span(w...-. The details of the axpacx subroutines are available in (5) and will not be presented here except to discuss ¢ and wy. The Lnpack routine srmco and its sparse variation [14] determine a vector y such that the vector x in Ryx = y is large relative to y. The trai wll vector Ww, deseribed for step 2 will be the x vector of srrco. Also, our approximation e, will be the calculated ratio llyi/Ixil=lyi/Ii¥yf. ‘This calculation requires a simple change in TACO, since srnco caleulates yi /\bsl, not lly i/lIx We should note that the sparse matrix algorithm [10] of step 1 uses Givens transformations, takes strong advantage of the sparsity of A, allows use of convenient fixed data structures, and is numerically stable. For sparse (or dense) matrices in step 3 a Hessenberg matrix results after a column of Ry is azopped, and as is standard {5], this matrix can be retriangularized by applying a sequence of Givens transformation making use of any sparsity in R, [15]. For sparse matrices this updating can be done within the fired data structure of (10) To analyze the conditions (3.8) and (3.9) we note that although the author knows of no published theoretical bonds for the condition estimators of Cline, Moler, Stewart, and Wilkinson used in the tivpack subroutine srxco, extensive numerical tests have been performed (4, 19]. If e, = lyi/ixl| as above and if s(A, +E) represents the smallest singular value of a matrix A, + E, then the tests and analysis of [4] and {19} indicate that a ratio similar to e,/s(A, +E), but involving the one norm, is typically 2 and almost always 10 or less, and & is a matrix such that |[E||/||ull @ modest multiple of machine precision. Furthermore, since these LixeaCx condition estimators, when modified appropriately, appear to work well for norms other than | 19}, we expect that ¢, /s(Ay + €) <7 with y= 10 oF less. Since s( A, +E) <3(Ay)+||Eh, (3.2) follows with A= ani Agi) < ania, where @ a modest sized factor. To examine (3.9) we note that the algorithms sciex, srRco, and sone and the use of the Givens transformations described above are all stable procedures [5, 24]. It follows that for any k, a unitary matrix Q, and error matrices E, and Ej exist with Q,(A, + B,)= Ry, (Ry + Es) =, (Eyl <s LESLIE V. FOSTER ayy Auliy, and Ei) < ai|lAylin, where vi, is the calculated veetor defined above, and where a, and aj, are of modest size and depend only on the size of A. These results imply that [Aye lL 7 a aly] and m e and n=N, stop. Let the current p be p, the calculated nullity, Pan(W and § Pethaps the motivation of this implementation is not immediately clear. However, # A=OR and if no columns of A are dropped (50 €4> 6 R= Lycs.M)s then it follows easily that in exact arithmetic |l//jt the two norm of column k of R°, the psuedoinverse of R. Therefore, by the constructions of the algorithm, ey would be the reciprocal of the maximam two norm of a column of R*. Since here the smallest singular value of A is, 1 /||H° ily we conclude by (2.9) that in the case that no eolumns are dropped the tests in steps 3 and 4 involving ¢, would be a precise way to examine rank deficieney. That this property persists when columns are dropped and in inexact arithmetic is the content of the following theorem, whose proof is in the appendix