4 Systems of Linear Equations

In this chapter direct methods for solving systems of linear equations

   
a11 · · · a1n b1
 · ·   · 
   
Ax = b, A= · · , b= · 
   
· · ·
an1 · · · ann bn
will be presented. Here A is a given n × n matrix, and b is a given vector.
We assume in addition that A and b are real, although this restriction is
inessential in most of the methods. In contrast to the iterative methods
(Chapter 8), the direct methods discussed here produce the solution in
finitely many steps, assuming computations without roundoff errors.
This problem is closely related to that of computing the inverse A−1 of
the matrix A provided this inverse exists. For if A−1 is known, the solution
x of Ax = b can be obtained by matrix vector multiplication, x = A−1 b.
Conversely, the ith column āi of A−1 = [ā1 , . . . , ān ] is the solution of the
linear system Ax = ei , where ei = (0, . . . , 0, 1, 0, . . . 0)T is the ith unit
vector.
A general introduction to numerical linear algebra is given in Golub
and van Loan (1983) and Stewart (1973). algol programs are found in
Wilkinson and Reinsch (1971), fortran programs in Dongarra, Bunch,
Moler, and Stewart (1979) (linpack), and in Andersen et al. (1992) (la-
pack).

4.1 Gaussian Elimination. The Triangular

Decomposition of a Matrix

In the method of Gaussian elimination for solving a system of linear equa-

tions

(4.1.1) Ax = b ,
 4.1 Gaussian Elimination. The Triangular Decomposition of a Matrix 191

where A is an n × n matrix and b ∈ IRn , the given system (4.1.1) is trans-

formed in steps by appropriate rearrangements and linear combinations of
equations into a system of the form
 
r11 . . . r1n
.
Rx = c, R= ..
. .. 
0 rnn

which has the same solution as (4.1.1). R is an upper triangular matrix, so

that Rx = c can easily be solved by “back substitution” (so lang as rii = 0,
i = 1, . . . , n):
 n
 :
xi := ci − rik xk rii for i = n, n − 1, . . . , 1 .
k=i+1

In the first step of the algorithm an appropriate multiple of the first

equation is subtracted from all of the other equations in such a way that
the coefficients of x1 vanish in these equations; hence, x1 remains only in
the first equation. This is possible only if a11 = 0, of course, which can be
achieved by rearranging the equations if necessary, as long as at least one
ai1 = 0. Instead of working with the equations themselves, the operations
are carried out on the matrix
 
a11 . . . a1n b1
. . .
(A, b) =  .. .. .. 
an1 . . . ann bn

which corresponds to the full system given in (4.1.1). The first step of the
Gaussian elimination process leads to a matrix (A
, b
) of the form
 a
a
12 ... a
1n b
1 
11
 0 a
22 ... a
2n b
2 


 
(4.1.2) (A , b ) =  .. .. .. ..  ,
 . . . . 
0 a
n2 . . . a
nn b
n
and this step can be described formally as follows:
(4.1.3)
(a) Determine an element ar1 = 0 and proceed with (b); if no such r exists,
A is singular; set (A
, b
) := (A, b); stop.
(b) Interchange rows r and 1 of (A, b). The result is the matrix (Ā, b̄).
(c) For i = 2, 3, . . . , n, subtract the multiple

li1 := āi1 /ā11

192 4 Systems of Linear Equations

of row 1 from row i of the matrix (Ā, b̄). The desired matrix (A
, b
) is
obtained as the result.
The transition (A, b) → (Ā, b̄) → (A
, b
) can be described by using
matrix multiplications:

(4.1.4) (Ā, b̄) = P1 (A, b), (A
, b
) = G1 (Ā, b̄) = G1 P1 (A, b),

where P1 is a permutation matrix, and G1 is a lower triangular matrix:

(4.1.5) 
0 1 0
 1 
   
 ..  1 0
 . 
   −l21 1 
 1 
P1 :=   G1 := 
 ... .. .

1 0  ← r, .
 
 1  −ln1 0 ··· 1
 .. 
 . 
0 1

Matrices such as G1 , which differ in at most one column from an iden-

tity matrix, are called Frobenius matrices. Both matrices P1 and G1 are
nonsingular; in fact
 
1 0
 l21 1 
P1−1 = P1 , G−1  .
1 =  .. .. 
. .
ln1 0 ··· 1

For this reason, the equation systems Ax = b and A
x = b
have the

same solution: Ax = b implies A
x = G1 P1 Ax = G1 P1 b = b
, and A
x = b

implies Ax = P1−1 G−1
−1 −1

1 A x = P1 G1 b = b.
The element ar1 = ā11 which is determined in (a) is called the pivot
element (or simply the pivot), and step (a) itself is called pivot selection
(or pivoting). In the pivot selection one can, in theory, choose any ar1 = 0
as the pivot element. For reasons of numerical stability [see Section 4.5]
it is not recommended that an arbitrary ar1 = 0 be chosen. Usually the
choice
   
ar1  = max ai1 
i

is made; that is, among all candidate elements the one of largest absolute
value is selected. (It is assumed in making this choice however — see Sec-
tion 4.5 — that the matrix A is “equilibrated”, that is, that the orders of
magnitudes of the elements of A are “roughly equal”.) This sort of pivot
selection is called partial pivot selection (or partial pivoting), in contrast
to complete pivot selection (or complete pivoting), in which the search for
 4.1 Gaussian Elimination. The Triangular Decomposition of a Matrix 193

a pivot is not restricted to the first column; that is, (a) and (b) in (4.1.3)
are replaced by (a
) and (b
):
(a
) Determine r and s so that
   
ars  = max aij 
i,j


and continue with (b ) if ars = 0. Otherwise A is singular; set
(A
, b
) := (A, b); stop.
(b
) Interchange rows 1 and r of (A, b), as well as columns 1 and s. Let the
resulting matrix be (Ā, b̄).
After the first elimination step, the resulting matrix has the form (4.1.2):
 
a
11 a
T b
1
(A
, b
) =  
0 Ã b̃

with an (n − 1)-row matrix Ã. The next elimination step consists simply
of applying the process described in (4.1.3) for (A, b) to the smaller matrix
(Ã, b̃). Carrying on in this fashion, a sequence of matrices
(A, b) := (A(0) , b(0) ) → (A(1) , b(1) ) → · · · → (A(n−1) , b(n−1) ) =: (R, c),
is obtained which begins with the given matrix (A, b) (4.1.1) and ends with
the desired matrix (R, c). In this sequence the jth intermediate matrix
(A(j) , b(j) ) has the form
(4.1.6)
 
∗ ··· ∗ ∗ ··· ∗ ∗
 . . .. .. .. .. 
 . . . . .
   
0 ∗ ∗ ··· ∗ ∗
  A
(j)
A
(j)
b
(j)
   11 12 1

(A(j) , b(j) ) =  =
0 ··· 0 ∗ ··· ∗ ∗
(j) (j)
  0 A22 b2
. .. 
. .. .. .. 
. . . . .
0 ··· 0 ∗ ··· ∗ ∗
(j)
with a j-row upper triangular matrix A11 . (A(j) , b(j) ) is obtained from
(A(j−1) , b(j−1) ) by applying (4.1.3) on the (n − j + 1) × (n − j + 2) matrix
(j−1) (j−1) (j) (j) (j)
(A22 , b2 ). The elements of A11 , A12 , b1 do not change from this
step on; hence they agree with the corresponding elements of (R, c). As in
the first step, (4.1.4) and (4.1.5), the ensuing steps can be described using
matrix multiplication. As can be readily seen

(A(j) , b(j) ) = Gj Pj (A(j−1) , b(j−1) ),

(4.1.7)
(R, c) = Gn−1 Pn−1 Gn−2 Pn−2 · · · G1 P1 (A, b) ,
194 4 Systems of Linear Equations

with permutation matrices Pj and nonsingular Frobenius matrices Gj of

the form
 1 0 
 .. 
 . 
 1 
 
(4.1.8) Gj =  −lj+1,j 1 .
 
 .. .. . . 
 . . . 
0 −ln,j 0 1

In the jth elimination step (A(j−1) , b(j−1) ) → (A(j) , b(j) ) the elements below
the diagonal in the jth column are annihilated. In the implementation of
this algorithm on a computer, the locations which were occupied by these
elements can now be used for the storage of the important quantities lij ,
i ≥ j + 1, of Gj ; that is, we work with a matrix of the form

 
r11 r12 ··· r1j r1,j+1 ··· r1n c1
 
 .. .. .. 
 λ21 r22 ··· r2j 
 . . . 
 
 .. .. .. .. 
 λ31 λ32 . . . . 
 
 . .. 
T (j) =
 .
. . rjj rj,j+1 ··· rj,n cj 

 . 
 . .. (j) (j) (j) 
 . . λj+1,j aj+1,j+1 ··· aj+1,n bj+1 
 
 . 
 . .. .. .. .. .. 
 . . . . . . 
 
(j) (j) (j)
λn1 λn2 ··· λn1 an,j+1 ··· an,n bn

Here the subdiagonal elements λk+1,k , λk+2,k , . . ., λnk of the kth column
are a certain permutation of the elements lk+1,k , . . ., ln,k in Gk (4.1.8).
Based on this arrangement, the jth step T (j−1) → T (j) , j = 1, 2, . . . ,
n − 1, can be described as follows (for simplicity the elements of T (j−1) are
denoted by tik , and those of T (j) by t
ik , 1 ≤ i ≤ n, 1 ≤ k ≤ n + 1):
(a) Partial pivot selection: Determine r so that
   
trj  = maxtij .
i≥j

If trj = 0, set T (j) := T (j−1) ; A is singular: stop. Otherwise carry on

with (b).
(b) Interchange rows r and j of T (j−1) , and denote the result by T̄ = (t̄ik ).
(c) Replace
 4.1 Gaussian Elimination. The Triangular Decomposition of a Matrix 195

t
ij := lij := t̄ij /t̄jj for i = j + 1, j + 2, . . . , n,

t
ik := t̄ik − lij t̄jk for i = j + 1, . . . , n and k = j + 1, . . . , n + 1,
t
ik = t̄ik otherwise.

We note that in (c) the important elements lj+1,j , . . ., lnj of Gj are

stored in their natural order as t
j+1,j,..., t
nj . This order may, however, be
changed in the subsequent elimination steps T (k) → T (k+1) , k ≥ j, because
in (b) the rows of the entire matrix T (k) are rearranged. This has the
following effect: The lower triangular matrix L and the upper triangular
matrix R,
   
1 0 t11 · · · · · · t1n
 t21 1   .. .. 
   . . 
L :=  .. .. ..  , R :=  . ..  ,
. . .  . . . 
tn1 · · · tn,n−1 1 0 tnn

which are contained in the final matrix T (n−1) = (tik ), provide a triangular
decomposition of the matrix P A:

(4.1.9) LR = P A.

In this decomposition P is the product of all of the permutations appearing

in (4.1.7):
P = Pn−1 Pn−2 · · · P1 .
We will only show here that a triangular decomposition is produced
if no row interchanges are necessary during the course of the elimination
process, i.e., if P1 = · · · = Pn−1 = P = I. In this case,
 
1 0
 l21 1 
L=
 ... .. .. ,

. .
ln1 · · · ln,n−1 1

since in all of the minor steps (b) nothing is interchanged. Now, because of
(4.1.7),
R = Gn−1 · · · G1 A,
therefore

(4.1.10) G−1 −1
1 · · · Gn−1 R = A.

Since
196 4 Systems of Linear Equations
 
1 0
 .. 
 . 
 
 1 
G−1
j = 
 lj+1,j 1 
 .. 
 .
..
. 
0 ln,j 1
it is easily verified that
 
1 0
 l21 1 
G−1 −1 −1   = L.
1 G2 · · · Gn−1 =  .. .. .. 
. . .
ln1 · · · ln,n−1 1

Then the assertion follows from (4.1.10).

Example.     
3 1 6 x1 2
2 1 3 x2 = 7 ,
1 1 1 x3 4
     
3∗ 1 6 2 3 1 6 2 3 1 6 2
     
   2 1 17   1 2 10 
2 1 3 7→ 3 3
−1 3 → 3 3
−1 3 .
     
1 2∗ 10 2 1 ∗
1 1 1 4 3 3
−1 3 3 2
− 12 4

The pivot elements are marked. The triangular equation system is

    
3 1 6 x1 2
0 2
3
−1   x2  =  10
3
.
1
0 0 −2 x3 4

Its solution is
x3 = −8,
x2 = 32 ( 10
3
+ x3 ) = −7,
x1 = 13 (2 − x2 − 6x3 ) = 19.
Further    
1 0 0 3 1 6
P = 0 0 1 , PA = 1 1 1 ,
0 1 0 2 1 3
and the matrix P A has the triangular decomposition P A = LR with
   
1 0 0 3 1 6
L= 1
3
1 0, R = 0 2
3
−1  .
2 1
3 2
1 0 0 − 12
 4.1 Gaussian Elimination. The Triangular Decomposition of a Matrix 197

Triangular decompositions (4.1.9) are of great practical importance in

solving systems of linear equations. If the decomposition (4.1.9) is known
for a matrix A (that is, the matrices L, R, P are known), then the equation
system Ax = b can be solved immediately with any right-hand side b ; for
it follows that
P Ax = LRx = P b,
from which x can be found by solving both of the triangular systems

Lu = P b, Rx = u

(provided all rii = 0).

Thus, with the help of the Gaussian elimination algorithm, it can be
shown constructively that each square nonsingular matrix A has a trian-
gular decomposition of the form (4.1.9). However, not every such matrix A
has a triangular decomposition in the more narrow sense A = LR, as the
example  
0 1
A=
1 0
shows. In general, the rows of A must be permuted appropriately at the
outset.
The triangular decomposition (4.1.9) can be obtained directly without
forming the intermediate matrices T (j) . For simplicity, we will show this
under the assumption that the rows of A do not have to be permuted in
order for a triangular decomposition A = LR to exist. The equations A =
LR are regarded as n2 defining equations for the n2 unknown quantities

rik , i ≤ k,
lik , i≥k (lii = 1);
that is, for all i, k = 1, 2, . . . , n


min(i,k)
(4.1.11) aik = lij rjk (lii = 1).
j=1

The order in which the lij , rjk are to be computed remains open. The
following versions are common:
In the Crout method the n × n matrix A = LR is partiotioned as
follows:
1
3
5
7
2 4 6 8 9
198 4 Systems of Linear Equations

and the equations A = LR are solved for L and R in an order indicated by

this partitioning:
"1
1. a1i = l1j rji , r1i := a1i , i = 1, 2, . . . , n,
j=1

"
1
2. ai1 = lij rj1 , li1 := ai1 /r11 , i = 2, 3, . . . , n,
j=1

"
2
3. a2i = l2j rji , r2i := a2i − l21 r1i , i = 2, 3, . . . , n, etc.
j=1

And in general, for i = 1, 2, . . . , n,

i−1

rik := aik − lij rjk , k = i, i + 1, . . . , n,
j=1
(4.1.12)  :
i−1

lki := aki − lkj rji rii , k = i + 1, i + 2, . . . , n.
j=1

In all of the steps above lii = 1 for i = 1, 2. . . . , n.

In the Banachiewicz method, the partitioning

1
2 3
4 5
6 7
8 9

is used; that is, L and R are computed by rows.

The formulas above are valid only if no pivot selections is carried out.
Triangular decomposition by the methods of Crout or Banachiewicz with
pivot selection leads to more complicated algorithms; see Wilkinson (1965).
Gaussian elimination and direct triangular decomposition differ only in
the ordering of operations. Both algorithms are, theoretically and numeri-
cally, entirely equivalent. Indeed, the jth partial sums
j

(j)
(4.1.13) aik := aik − lis rsk
s=1

of (4.1.12) produce precisely the elements of the matrix A(j) in (4.1.6), as

can easily be verified. In Gaussian elimination, therefore, the scalar prod-
ucts (4.1.12) are formed only in pieces, with temporary storing of the inter-
mediate results; direct triangular decomposition, on the other hand, forms
 4.1 Gaussian Elimination. The Triangular Decomposition of a Matrix 199

each scalar product as a whole. For these organizational reasons, direct tri-
angular decomposition must be preferred if one chooses to accumulate the
scalar products in double-precision arithmetic in order to reduce round-
off errors (without storing double-precision intermediate results). Further,
these methods of triangular decomposition require about n3 /3 operations
(1 operation = 1 multiplication + 1 addition). Thus, they also offer a simple
way of evaluating the determinant of a matrix A: From (4.1.9) it follows,
since det(P ) = ±1, det(L) = 1, that

det(P A) = ± det(A) = det(R) = r11 r22 · · · rnn .

Up to its sign, det(A) is exactly the product of the pivot elements. (It
should be noted that the direct evaluation of the formula
n

det(A) = sign(µ1 , . . . , µn )a1µ1 a2µ2 · · · anµn
µ1 ,...,µn =1
µi
=µk für i
=k

requires n!  n3 /3 operations.)
In the case that P = I, the pivot elements rii are representable as
quotients of the determinants of the principal submatrices of A. If, in the
representation LR = A, the matrices are partitioned as follows:
    
L11 0 R11 R12 A11 A21
= ,
L21 L22 0 R22 A12 A22

it is found that L11 R11 = A11 ; hence det(R11 ) = det(A11 ), or

r11 · · · rii = det(A11 ),

if A11 is an i × i matrix. In general, if Ai denotes the ith leading principal

submatrix of A, then

rii = det(Ai )/ det(Ai−1 ), i ≥ 2,

r11 = det(A1 ).

A further practical and important property of the method of triangular

decomposition is that, for band matrices with bandwidth m,

 
∗ ··· ∗ 0
 ... . . . .. 
 . 
 .. .. 
∗ . . 
A=
 .. ..
 , aij = 0 for |i − j| ≥ m,

 . . ∗ 1
 .. .. .. 
 . . . m
0 ∗ ··· ∗
200 4 Systems of Linear Equations

 
m

the matrices L and R of the decomposition LR = P A of A are not full: R

is a band matric with bandwidth 2m − 1,
 
∗ ··· ∗ 0 ··· 0
 .
. .. 
.. .. ..
 . . 
 .. .. 
 . . 0
R=
 ..
,

 . ∗ 
 .. .
 . ..  2m−1

0 ∗

and in each column of L there are at most m elements different from zero. In
contrast, the inverses A−1 of band matrices are usually filled with nonzero
entries.
Thus, if m  n, using the triangular decomposition of A to solve
Ax = b results in a considerable saving in computation and storage over
using A−1 . Additional savings are possible by making use of the symmetry
of A if A is a positive definite matrix (see Sections 4.3 and 4.A).

4.2 The Gauss-Jordan Algorithm

In practice, the inverse A−1 of a nonsingular n×n matrix A is not frequently

needed. Should a particular situation call for an inverse, however, it may be
readily calculated using the triangular decomposition described in Section
4.1 or using the Gauss-Jordan algorithm, which will be described below.
Both methods require the same amount of work.
If the triangular decomposition P A = LR of (4.1.9) is available, then
the ith column āi of A−1 is obtained as the solution of the system

(4.2.1) LRāi = P ei ,

where ei is the ith coordinate vector. If the simple structure of the right-
hand side of (4.2.1), P ei , is taken into account, then the n equation systems
(4.2.1) (i = 1, . . . , n) can be solved in about 23 n3 operations. Adding
the cost of producing the decomposition gives a total of n3 operations
to determaine A−1 . The Gauss-Jordan method requires this amount of
work, too, and offers advantages only of an organizational nature. The
Gauss-Jordan method is obtained if one attempts to invert the mapping
x → Ax = y, x ∈ IRn , y ∈ IRn , determined by A in a systematic manner.
Consider the system Ax = y: