Chapter 10

LU Decomposition and Matrix

Inversion

1
 Introduction

• Gauss elimination solves [A] {x} ={B}

• It becomes insufficient when solving these

equations for different values of {B}

• LU decomposition works on the matrix [A] and

the vector {B} separately.

• LU decomposition is very useful when the

vector of variables {x} is estimated for different
parameter vectors {B} since the forward
elimination process is not performed on {B}.
2
 LU Decomposition

If:
L: lower triangular matrix
U: upper triangular matrix
Then,
[A]{X}={B} can be decomposed into two
matrices [L] and [U] such that:
1. [L][U] = [A]  ([L][U]){X} = {B}

3
 LU Decomposition

Consider:
[U]{X} = {D}
So, [L]{D} = {B}
2. [L]{D} = {B} is used to generate an intermediate vector {D}
by forward substitution.
3. Then, [U]{X}={D} is used to get {X} by back substitution.

4
Summary of LU Decomposition

5
 LU Decomposition

As in Gauss elimination, LU decomposition must

employ pivoting to avoid division by zero and to

minimize round off errors. The pivoting is done

immediately after computing each column.

6
 LU Decomposition
System of linear equations [A]{x}={B}
a12 a31
 a11 a12 a13   x1  b1  l 21  ; l 31 
a    
a23   x2   b2 
a11 a11
 21 a 22

a31 a32 a33   x3  b3  a32\

l 32  \
a22
Step 1: Forward elimination [ L][U ]  [ A]
a11 a12 a13  1 0 0
[U ]   0 a22/ a23/  
[L ]  l 21 1 0
 0 0 a33 
//
 l 31 l 32 1 
7
 LU Decomposition
Step 2: Generate an intermediate vector {D} by forward
substitution
 1 0 0  d1  b1 
l  d   b 
 21 1 0  2   2 
l31 l32 1 d 3  b3 
Step 3: Get {X} by back substitution.

a11 a12 a13   x1   d1 

 0 a'     
 22 a'23   x2   d 2 
 0 0 a' '33   x3  d 3 

8
 LU Decomposition-Example

 3  0.1  0.2   3 0.1 0.2 

A  0.1 7  0.3
0.3  0.2 10 
 0 7.003 0.293
 
  0 0.19 10.02 
0.1 0.3
l 21   0.03333; l 31   0.1000
3 3
a32\ 0.19
l 32  \   0.02713
a22 7.003

3  0.1  0.2   1 0 0

U   0 7.003  0.293 [L ]  0.03333 1 0
0   0.1000 .02713 1 
 0 10 .012 
9
 LU Decomposition-Example (cont’d)

Use previous L and U matrices to solve the system:

 3  0.1  0.2   x1   7.85 
0.1     
 7  0.31  x2    19.3
0.3  0.2 10   x3   71.4 

Step 2: Find the intermediate vector {D} by forward substitution

 1 0 0  d1   7.85  d1   7.85 

0.0333  d    19.3  d    19.5617
 1 0  2     2  
0.1000  0.02713 1 d 3   71.4  d 3   70.0843 

10
 LU Decomposition-Example
(cont’d)

Step 3: Get {X} by back substitution.

3  0.1  0.2   x1   7.85   x1   3 

0 7.0033  0.2933  x    19.5617   x     2.5 
  2     2  
0 0 10.012   x3   70.0843   x3  7.00003

11
 Matrix Inverse Using the LU
Decomposition

• LU decomposition can be used to obtain the inverse of

the original coefficient matrix [A].
• Each column j of the inverse is determined by using a
unit vector (with 1 in the jth raw ).

15
 Matrix Inverse: LU Decomposition

[A] [A]-1 = [A]-1[A] = I

Ax1  {b}1 Ax2  {b}2 Ax3  {b}3

1 0 0
     
Ax1  0 Ax2  1 Ax3  0
0 0 1
     

1st column 2nd column 3rd column

of [A]-1 of [A]-1 of [A]-1

A 1
 {x}1 {x}2 {x}3 
16
 Matrix inverse using LU decomposition
Example
 3  0.1  0.2   1 0 0 3  0.1  0.2 
A  0.1 7  0.3 [L ]  0.03333 1 0 U   0 7.003  0.293
0.3  0.2 10   0.1000 .02713 1  0 
   0 10 .012 

1A. [L]{d}1 = {b}1

 1 0 0  d1  1 d1   1 
0.0333  d   0  d    0.03333
 1 0  2     2   
0.1000  0.02713 1 d 3  0 d 3    0.1009  1st column
1B. Then, [U]{X}1={d}1 of [A]-1

3  0.1  0.2   x1   1   x1   0.33249 

0 7.0033  0.2933  x    0.03333   x    0.00518
  2     2  
0 0 10.012   x3    0.1009   x3   0.01008
17
 Matrix inverse using LU decomposition
Example (cont’d)

2A. [L]{d}2 = {b}2

 1 0 0  d1  0 d1   0 
0.0333         
 1 0 d 2   1  d 2    1 
0.1000  0.02713 1 d 3  0 d 3  0.02713
2nd column
2B. Then, [U]{X}2={d}2 of [A]-1

3  0.1  0.2   x1   0   x1  0.004944

0 7.0033  0.2933  x    1    x   0.142903
  2     2  
0 0 10.012   x3  0.02713  x3   0.00271 
18
 Matrix inverse using LU decomposition
Example (cont’d)

3A. [L]{d}3 = {b}3

 1 0 0  d1  0 d1  0

0.0333  d   0  d   0
 1 0  2     2   
0.1000  0.02713 1 d 3  1 d 3  1

3B. Then, [U]{X}3={d}3 3rd column

of [A]-1

3  0.1  0.2   x1  0  x1  0.006798

0 7.0033  0.2933  x   0   x   0.004183
  2     2   
0 0 10.012   x3  1  x3   0.09988 

19
Matrix inverse using LU decomposition
Example (cont’d)

 0.33249 0.004944 0.006798

[ A]1   0.00518 0.142903 0.004183
 0.01008 0.00271 0.09988 

20
 LU decomposition (Operation Counts)

Basic Approach

Consider [A]{x} = {b}

a) Gauss-type "decomposition" of [A] into [L][U] n3/3

flops

[A]{x} = {b} becomes [L] [U]{x} = {b}; let [U]{x}  {d}

b) First solve [L] {d} = {b} for {d} by forward subst. n2/2
flops

c) Then solve [U]{x} = {d} for {x} by back substitution n2/2

flops
21
 LU decomposition

LU Decomposition Variations
Doolittle [L1][U]  from Gaussian elim, General [A]
Crout [L][U1] General [A]
Cholesky [L][L] T Pos. Def. Symmetric [A]

Cholesky works only for Positive Definite Symmetric

matrices

Doolittle versus Crout:

• Doolittle just stores Gaussian elimination factors where Crout uses a
different series of calculations (see C&C 10.1.4).
• Both decompose [A] into [L] and [U] in n3/3 FLOPS
• Different location of diagonal of 1's
• Crout uses each element of [A] only once so the same array can be
used for [A] and [L\U] saving computer memory!

22
 LU decomposition

Matrix Inversion
Definition of a matrix inverse:

[A] [A]-1 = [ I ]

==> [A] {x} = {b}

[A]-1 {b} = {x}

First Rule: Don’t do it.

(numerically unstable calculation)

23
 LU decomposition
Matrix Inversion
If you really must --
1) Gaussian elimination: [A | I ] –> [U | B'] ==> A-1
2) Gauss-Jordan: [A | I ] ==> [I | A-1 ]

Inversion will take

n3 + O(n2)
flops if one is careful about where zeros are (taking advantage of the
sparseness of the matrix)

Naive applications (without optimization) take 4n3/3 + O(n2) flops.

For example, LU decomposition requires n3/3 + O(n2) flops.

Back solving twice with n unit vectors ei:
2 n (n2/2) = n3 flops.
Altogether: n3/3 + n3 = 4n3/3 + O(n2) flops
24
FLOP Counts for Linear Algebraic Equations
Summary
FLOP Counts for Linear Algebraic Equations, [A]{x} =
{b}
Gaussian Elimination (1 r.h.s) n3/3 + O (n2)
Gauss-Jordan (1 r.h.s) n3/2 + O (n2)
LU decomposition n3/3 + O (n2)
Each extra LU right-hand-side n2
Cholesky decomposition (symmetric A) n3/6 + O (n2)
Inversion (naive Gauss-Jordan) 4n3/3 +O (n2)
Inversion (optimal Gauss-Jordan) n3 + O (n2)
Solution by Cramer's Rule n!
25
 Vector and Matrix Norms

Norm is a real-valued function that provides a measure of

size or “length” (magnitude) of vectors and matrices.
Norms are useful in studying the error behavior of
algorithms.
A simple example is a vector in three - dimensiona l Euclidean space
that can be represented as
F   a b c 
where a, b, and c are the distances along x, y, and z axes, repectively.

26
Vector and Matrix Norms (cont’d)

F e  a 2  b2  c 2
Length or Euclidean norm of [F]

27
 Vector Norms
Vector Norms: Scalar measure of the magnitude of a vector
Here are some vector norms for n x 1 1/ p
 n 
vectors {x} with typical elements xi.
Each is in the general form of a p norm
x p



x
i 1
i
p



defined by the general relationship:
 n 
1. Sum of the magnitudes: x 1  


 i 1
xi 


2. Magnitude of largest element: x 
 max x i
i
(infinity norm)
1/ 2
 n 
3. Length or Euclidean norm: x 2



x
i 1
i
2



28
 Matrix Norms

Matrix Norms: Scalar measure of the magnitude of a matrix.

Matrix norms corresponding to vector norms above are
defined by the general relationship:

A p
 max  A x p
x 1
n

a
1. Largest column sum:
A 1 max ij
(column sum norm) j
i 1

2. Largest row sum: n

(row sum norm) A 
 max
i
a
j1
ij
(infinity norm)
n n

3. Eucledian or Frobenius norm: Ae  ij

A 2

i 1 j 1 29
 Vector and Matrix Norms (cont’d)

• Matrix Condition umber Defined as:

Cond A  A  A1

• For a matrix [A], this number will be greater than or

equal to 1.
• k = cond [A]  ||A-1|| ||A||  1
If k  1 or k is small, the system is well-conditioned
If k >> 1, system is ill conditioned.
30
 Chapter 11

Special Matrices and

Gauss-Seidel

37
 Introduction

• Certain matrices have particular structures that can be

exploited to develop efficient solution schemes.

• A banded matrix is a square matrix that has all

elements equal to zero, with the exception of a band
centered on the main diagonal. These matrices typically
occur in solution of differential equations.

• The dimensions of a banded system can be quantified

by two parameters: the band width BW and half-
bandwidth HBW. These two values are related by
BW=2HBW+1.

38
Banded matrix

39
 Tridiagonal Systems
• A tridiagonal system has a bandwidth of 3:
 f1 g1   x1   r1 
e   x  r 
 2 f2 g2   2    2 
 e3 f3 g 3   x3  r3 
    
 e4 f 4   x4  r4 

• An efficient LU decomposition method, called

Thomas algorithm, can be used to solve such an
equation. The algorithm consists of three steps:
decomposition, forward and back substitution, and
has all the advantages of LU decomposition. (See
Example 11.1)
40
Fig 11.2

41
 Cholesky Decomposition
 This method is suitable for only symmetric systems
where:
 l11 0 0
a ij  a ji and A A T [L ]  l 21 l 22 0
 
 l 31 l 32 l 33 

A  L * LT

 a11 a12 a13   l11 0 0  l11 l 21 l 31 

a a a   l l 0  *  0 l 22 l 32 
 21 22 23   21 22   
a31 a32 a33   l 31 l 32 l 33   0 0 l 33 

4242
 Cholesky Decomposition

i 1
aki   l iji 1l kj
l ki  akij 1  l ij  l kjfor i  1, 2, , k  1
j 1
l ki  l ii for i  1, 2, , k  1
k 1 l ii
l kk  akk   l kj2k 1
l kk  akkj 1  l kj2
j 1

See Example 11.2

43
 Pseudocode for Cholesky’s LU
Decomposition algorithm (cont’d)

44
 Iterative Solution Methods

• Iterative or approximate methods provide an alternative

to the elimination methods. The Gauss-Seidel method
is the most commonly used iterative method.
• The system [A]{X}={B} is reshaped by solving the first
equation for x1, the second equation for x2, and the
third for x3, …and nth equation for xn. We will limit
ourselves to a 3x3 set of equations.

45
 Iterative Solution Methods

Impetus for Iterative Schemes:

1. May be more rapid if coefficient matrix is "sparse"
2. May be more economical with respect to memory
3. May also be applied to solve nonlinear systems
Disadvantages:
1. May not converge or may converge slowly
2. Not appropriate for all systems

46
 Iterative Solution Methods

Basic Mechanics:
Starting with:
a11x1 + a12x2 + a13x3 + ... + a1nxn = b1
a21x1 + a22x2 + a23x3 + ... + a2nxn = b2
a31x1 + a32x2 + a33x3 + ... + a3nxn = b3
: :
an1x1 + an2x2 + an3x3 + ... + annxn = bn
Solve each equation for one variable:
x1 = [b1 – (a12x2 + a13x3 + ... + a1nxn )} / a11
x2 = [b2 – (a21x1 + a23x3 + ... + a2nxn )} / a22
x3 = [b3 – (a31x1 + a32x2 + ... + a3nxn )} / a33
:
xn = [bn – (an1x2 + an2x3 + ... + an,n-1xn-1 )} / ann
47
 Gauss-Seidel

a11 x1  a12 x2  a13 x3  b1 b1  a12 x2  a13 x3

x1 
a11

a21 x1  a22 x2  a23 x3  b2  x2 

b2  a21 x1  a23 x3
a22
b3  a31 x1  a32 x2
x1 
a31 x1  a32 x2  a33 x3  b3 a33

Now we can start the solution process by choosing

guesses for the x’s. A simple way to obtain initial
guesses is to assume that they are zero. These
zeros can be substituted into x1 equation to
calculate a new x1=b1/a11.
48
 Gauss-Seidel

• New x1 is substituted to calculate x2 and x3.

The procedure is repeated until the
convergence criterion is satisfied:

new
x x old
 a ,i  i
new
i
100%   s
x i

49
 Iterative Solution Methods -- Gauss-Seidel

 
x 2j1   b 2  a 21 x1j1  a 23 x 3j  
 a 2n x nj  a 22

x 3j1   b3  a x1j1  a 32 x 2j1  
 a 3n x nj  a 33
 31 

 
x nj1   b n  a n1 x1j1  a n2 x 2j1  
 a n,n 1 x nj11  a nn


The newest values within the right-hand side equations

will provide better estimates of the next value.

50
 Jacobi iteration Method

An alternative approach, called Jacobi iteration,

utilizes a somewhat different technique. This
technique includes computing a set of new x’s on
the basis of a set of old x’s. Thus, as the new
values are generated, they are not immediately
used but are retained for the next iteration.

51
Iterative Solution Methods – the Jacobi
Method

 
x1j1   b1  a12 x 2j  a13 x 3j  
 a1n x nj  a11

x 2j1   b 2  a x1j  a 23 x 3j   a 2n x nj  a 22
 21

 
x nj1   b n  a n1 x1j  a n2 x 2j  
 a n,n 1 x nj 1  a nn


52
 Gauss-Siedel

The Gauss-Seidel method The Jacobi iteration method

53
 Convergence Criterion for Gauss-Seidel
Method
• The Gauss-Seidel method is similar to the
technique of fixed-point iteration.
• The Gauss-Seidel method has two fundamental
problems as any iterative method:
1. It is sometimes non-convergent, and
2. If it converges, converges very slowly.
• Sufficient conditions for convergence of two linear
equations, u(x,y) and v(x,y) are:
u u
 1
x y
v v
 1
x y
54
 Convergence Criterion for Gauss-
Seidel Method (cont’d)
• Similarly, in case of two simultaneous equations,
the Gauss-Seidel algorithm can be expressed as:
b1 a12
u (x 1 , x 2 )   x2
a11 a11
b 2 a21
v (x 1 , x 2 )   x1
a22 a22
u u a12
0 
x 1 x 2 a11
v a21 v
 0
x 1 a22 x 2
55
 Convergence Criterion for Gauss-
Seidel Method (cont’d)

Substitution into convergence criterion of two linear

equations yield: a12 a21
 1, 1
a11 a22
In other words, the absolute values of the slopes
must be less than unity for convergence:
a11  a12 For n equations
n

a22  a21
aii  a
j 1
i, j

j i

That is, the diagonal element must be greater than

the off-diagonal element for each row.

56
 Gauss-Siedel Method- Example 1

7.85  0.1x2  0.2 x3

 3  0.1 0.2   x1   7.85  x1  3
0.1       19.3  0.1x1  0.3 x3
 7  0.3  x2    19.3  x2 
7
0.3  0.2 10   x3   71.4  x3  71.4  0.3 x1  0.2 x2
10
• Guess x1, x2, x3= zero for the first guess
Iter. x1 x2 x3 |a,1|(%) |a,2| (%) |a,3| (%)
0 0 0 0 - - -
1 2.6167 -2.7945 7.005610 100 100 100
2 2.990557 -2.499625 7.000291 12.5 11.8 0.076

57
 Improvement of Convergence Using
Relaxation
x i
new
  x i
new
 1     x old
i

• Where  is a weighting factor that is assigned a

value between [0, 2]
• If  = 1 the method is unmodified.
• If  is between 0 and 1 (under-relaxation)
this is employed to make a non convergent
system to converge.
• If  is between 1 and 2 (over-relaxation) this
is employed to accelerate the convergence.

58
 Gauss-Seidel Method- Example 2

 8 x1  x2  2 x3  20
 3x1  x2  7 x3  34
2 x1  6 x2  x3  38 8x 1  x 2  2x 3  20
Rearrange so that
the equations are 2x 1  6x 2  x 3  38
diagonally dominant
3x 1  x 2  7 x 3  34
 20  x2  2 x3  38  2 x1  x3  34  3x1  x2
x1  x2  x3 
8 6 7

59
 Gauss-Seidel Method- Example 2

iteration unknown value a maximum a

0 x1 0
x2 0
x3 0
1 x1 2.5 100.00%
x2 7.166667 100.00%
x3 -2.7619 100.00% 100.00%
2 x1 4.08631 38.82%
x2 8.155754 12.13%
x3 -1.94076 42.31% 42.31%
3 x1 4.004659 2.04%
x2 7.99168 2.05%
x3 -1.99919 2.92% 2.92%

60
 Gauss-Seidel Method- Example 2

The same computation can be developed with relaxation

where  = 1.2
First iteration:
 20  x 2  2 x3  20  0  2(0)
x1    2.5
8 8
Relaxation yields: x1  1.2(2.5)  0.2(0)  3
38  2x 1  x 3 38  2(3)  0
x2    7.333333
6 6
Relaxation yields: x 2  1.2(7.333333)  0.2(0)  8.8
 34  3x1  x 2  34  3(3)  8.8
x3    2.3142857
7 7
Relaxation yields: x  1.2(2.3142857)  0.2(0)  2.7771429
3
61
 Gauss-Seidel Method- Example 2

Iter. unknown value relaxation a maximum a

1 x1 2.5 3 100.00%
x2 7.3333333 8.8 100.00%
x3 -2.314286 -2.777143 100.00% 100.000%
2 x1 4.2942857 4.5531429 34.11%
x2 8.3139048 8.2166857 7.10%
x3 -1.731984 -1.522952 82.35% 82.353%
3 x1 3.9078237 3.7787598 20.49%
x2 7.8467453 7.7727572 5.71%
x3 -2.12728 -2.248146 32.26% 32.257%
4 x1 4.0336312 4.0846055 7.49%
x2 8.0695595 8.12892 4.38%
x3 -1.945323 -1.884759 19.28% 19.280%

62
Iterative Solution Methods - Comparasion

If either method is going to converge,

Gauss-Seidel will converge
faster than Jacobi.

Why use Jacobi at all?

Because you can separate the n-equations into n

independent tasks, it is very well suited computers with
parallel processors.

63
 Convergence of Iterative Solution Methods
Rewrite given system: [A]{x} = { [B] + [E] } {x} = {b}
where [B] is diagonal, or triangular so we can solve
[B]{y} = {g} quickly. Thus,
[B] {x}j+1= {b}– [E] {x}j
which is effectively: {x}j+1 = [B]-1 ({b} – [E] {x}j )
True solution {x}c satisfies: {x}c = [B]-1 ({b} – [E] {x}c)
Subtracting yields: {x}c – {x}j+1= – [B]-1 [E] [{x}c – {x}j]
So ||{x}c – {x}j+1 ||  ||[B]-1 [E]|| ||{x}c – {x}j ||
Iterations converge linearly if || [B]-1 [E] || < 1
=> || ([D] + [Lo])-1 [Uo] || < 1 For Gauss-Seidel
=> || [D] -1 ([Lo] + [Uo]) || < 1 For Jacobi
64
Convergence of Iterative Solution Methods

Iterative methods will not converge for all systems of

equations, nor for all possible rearrangements.

If the system is diagonally dominant,

i.e., | aii | > | aij | where i  j then

bi a i1 a a in
xi   x1  i2 x 2   xn
a ii a ii a ii a ii

a ij
with all < 1.0, i.e., small slopes.
a ii
65
Convergence of Iterative Solution Methods

A sufficient condition for convergence exists:

n
a ii  a
j1
ij

ji

Notes:
1. If the above does not hold, still may
converge.
2. This looks similar to infinity norm of [A]

66
 Why Iterative Solutions?

We often need to solve [A]{x} = {b} where n = 1000's

• Description of a building or airframe,
• Finite-Difference approximations to PDE's. Most of A's
elements will be zero; a finite-difference approximation to
Laplace's equation will have five aij0 in each row of A.
Direct method (Gaussian elimination)
• Requires n3/3 flops (say n = 5000; n3/3 = 4 x 1010
flops)
• Fills in many of n2-5n zero elements of A
Iterative methods (Jacobi or Gauss-Seidel)
• Never store [A] (say n = 5000; [A] would need 4n2 = 100
Mb)
• Only need to compute [A-B] {x}; and to solve [B]{xt+1} = {b}
67
 Why Iterative Solutions?

• Effort:

Suppose [B] is diagonal,

solving [B] {v} = {b} n flops
Computing [A-B] x 4n flops
For m iterations 5mn flops
For n = m = 5000, 5mn = 1.25x108
At worst O(n2).

68
 Software: Excel

The Ctrl and Shift keys are held down while the Enter key is 69
depressed to perform the function.
Software: Matlab (MATrix LABoratory) / Octave

70
71