Decomposition by Gaussian Elimination

MAT307 Numerical Analysis Doolittle and Crout Decompositions

Cholesky Decomposition

MAT307 Numerical Analysis

Decomposition Methods

Dr. Mustafa Ağgül

mustafaaggul@hacettepe.edu.tr

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Why do we care about matrix decomposition?

Consider the simple linear equation, 18x = 486. To solve this equation
for x, our intuition leads us to cancel multiples of 2 or 3 from both sides
instead of 18 directly. The idea is to factorize the number 18 as 2 · 3 · 3
first and then perform easier divisions.
For an analog of this idea in the system of linear equations, consider
Ax = b. Assume A = LU such that L is a lower triangular and U is an
upper triangular matrix1 . Hence,

Ax = b ⇒ LU x = b ⇒ Ly = b and U x = y.

This way, the equation Ax = b is replaced by two easier equations:

Ly = b and U x = y. One can employ forward substitution to solve for y
first and then backward substitutions to solve for x.
Several approaches exist to find such lower and upper triangular matrices:
Decomposition by Gaussian Elimination, Doolittle’s and Crout’s methods,
and Cholesky decomposition.
1
LU decomposition is not the only option. See, e.g., eigenvalue
decomposition, singular value decomposition, etc.
Dr. Mustafa Ağgül Hacettepe University
 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

LU decomposition by Gaussian elimination is obtained by expressing this

elimination with elementary matrices. A matrix A is converted into an
upper triangular matrix U by the Gaussian elimination method. This way,
the matrix U is naturally selected. Elementary matrices will be used
specifically for the selection of the matrix L. For example, let a matrix A
be converted to a matrix U by left-multiplying the elementary matrices
L1 , L2 , . . . , Lk respectively. Then,

Lk . . . L2 L1 A = U.
Multiply this equality from the left by L−1 −1 −1
k , . . . , L2 , L1 in the given
order to get

A = L−1 −1 −1
1 L2 . . . Lk U.

Choosing2 L = L−1 −1 −1
1 L2 . . . Lk , we obtain L and U matrices.

2
Li ’s are always lower triangular matrices. Inverses of such matrices and
their products with one another are always lower triangular matrices.
Dr. Mustafa Ağgül Hacettepe University
 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
Multiply given A from the left by L1 , L2 and L3 in the given order.
    
1 0 0 0 3 −7 −2 2 3 −7 −2 2
 1 1 0 0  −3 5 1 0 = 0 −2 −1 2 
  −9
 −2 0 1 0
 6 −4 0 −5 0 10 4

3 0 0 1 −9 5 −5 12 0 −16 −11 18
| {z }| {z }
L1 A  
  
1 0 0 0 3 −7 −2 2 3 −7 −2 2
 0 1 0 0  0 −2 −1 2 = 0 −2 −1 2 
  −1
 0 5 1 0
 0 10 4 −9 0 0 1

0 −8 0 1 0 −16 −11 18 0 0 −3 2
| {z }| {z }
L2 L1 A
    
1 0 0 0 3 −7 −2 2 3 −7 −2 2
 0 1 0 0  0 −2 −1 2 = 0 −2 −1 2 .
−1 −1
 0 0 1 0
 0 0 1
  0 0 1

0 0 −3 1 0 0 −3 2 0 0 0 −1
| {z }| {z } | {z }
L3 L2 L1 A U

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
Therefore, L3 L2 L1 A = U . Then the matrix L for which A = LU should
be L = (L3 L2 L1 )−1 . Start with finding the matrix L3 L2 L1 :
   
1 0 0 0 1 0 0 0 1 0 0 0
 1 1 0 0  0 1 0 0  0 1 0 0 
L3 L2 L1 =
−2
 0 1 0
 0 5 1 0
 0 0 1 0

3 0 0 1 0 −8 0 1 0 0 −3 1

 
1 0 0 0
 1 1 0 0 .
=
−2
 5 1 0

3 −8 −3 1
 
1 0 0 0
−1 1 0 0
By the theorem, L = (L3 L2 L1 )
−1
=
  can be found.
−5
 2 1 0

−3 8 3 1

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
" #
1 2 −1
Consider the matrix A = 2 0 1 .
0 1 2
" #" # " # " #
1 0 0 1 2 −1 1 2 −1 1 0 0
−2 1 0 2 0 1 = 0 −4 3 , 2 1 0 .
0 0 1 0 1 2 0 1 2 0 0 1
| {z }
=L−1
1
" #" # " # " #
1 0 0 1 2 −1 1 2 −1 1 0 0
0 1 0 0 −4 3 = 0 −4 3 , 0 1 0 .
0 1 1 0 1 2 11 1
4 0 0 4
0 −4 1
| {z } | {z }
=U =L−1
2
" # " #" #
1 2 −1 1 0 0 1 2 −1
Thus, A = 2 0 1 = 2 1 0 0 −4 3 .
0 1 2 0 1
−4 1 0 0 11
4
| {z }| {z }
=L=L−1 −1
1 L2
=U

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Definition
Doolittle’s and Crout’s methods are also LU decompositions. Doolittle
decomposition assumes 1’s on the diagonal of the matrix L, while Crout’s
approach requires the same for the matrix U .
    
5 4 1 1 0 0 5 4 1
    
A =  10 4 = 2 2  → Doolittle
    
9 1 0  0 1
    
10 13 15 2 5 1 0 0 3

    
4 1
5 4 1 5 0 0 1 5 5
    
A =  10 4  =  10 2  → Crout
    
9 1 0  0 1
    
10 13 15 10 5 3 0 0 1

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
Consider the following matrix for Doolittle Decomposition:
    
a11 a12 ... a1n 1 0 ... 0 u11 u12 ... u1n
 a21 a22 ... a2n   `21 1 ... 0  0 u22 ... u2n 
 . . . . = . . . .  . . . . 
. . .. . . . .. . . . . .
.
 . . .
  . . .
 . . .

an1 an2 ... ann `n1 `n2 ... 1 0 0 ... unn

I Multiplying the first row of L with columns of U , we obtain

u11 = a11 , u12 = a12 , . . . , u1n = a1n .

I Multiplying L with the first column of U , we find
a21 a31 an1
`21 = , `31 = , . . . , `n1 = .
u11 u11 u11

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Algorithm
Consider the following matrix for Doolittle Decomposition:
    
a11 a12 ... a1n 1 0 ... 0 u11 u12 ... u1n
 a21 a22 ... a2n   `21 1 ... 0  0 u22 ... u2n 
 . . . . = . . . .  . . . . 
. . . . . . . . . . . .
. . .
 . . .
  . . .
 . . .

an1 an2 ... ann `n1 `n2 ... 1 0 0 ... unn

I Following the same fashion for i = 2, . . . , n and j = 2, . . . , n, one

can get j−1
X
i−1
aij − `ik ukj
X k=1
uij = aij − `ik ukj and `ij =
ujj
k=1

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
5 4 1 1 0 0 u11 u12 u13
    
 =  `21 u22 u23 
 10 9 4   1 0   0
  
10 13 15 `31 `32 1 0 0 u33

u11 = 5, u12 = 4 and u13 = 1.

    
5 4 1 1 0 0 5 4 1
    
 =  `21 u22 u23 
 10 9 4  1 0 
 0
 
 
10 13 15 `31 `32 1 0 0 u33

`21 = 2 and `31 = 2.

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
5 4 1 1 0 0 5 4 1
    
= 2 u22 u23 
 10 9 4  1 0 
 0
 
 
10 13 15 2 `32 1 0 0 u33

u22 = 1 and u23 = 2.

    
5 4 1 1 0 0 5 4 1
    
= 2
 10 9 4  1 0 
 0 1 2
  
 
10 13 15 2 `32 1 0 0 u33

`32 = 5.

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
5 4 1 1 0 0 5 4 1
    
= 2
 10 9 4  1 0 
 0 1 2
  
 
10 13 15 2 5 1 0 0 u33

u33 = 3.
    
5 4 1 1 0 0 5 4 1
    
= 2
 10 9 4  1 0 
 0 1 2 
 
 
10 13 15 2 5 1 0 0 3

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
Consider the following matrix for Crout Decomposition:
    
a11 a12 ... a1n `11 0 ... 0 1 u12 ... u1n
 a21 a22 ... a2n   `21 `22 ... 0  0 1 ... u2n 
 . . . . = . . . .  . . . . 
 . . .. .   . . .. .  . . .. . 
. . . . . . . . .
an1 an2 ... ann `n1 `n2 ... `nn 0 0 ... 1

I Multiplying L with the first column of U , we obtain

`11 = a11 , `21 = a21 , . . . , `n1 = an1 .

I Multiplying the first row of L with columns of U , we find
a12 a13 a1n
u12 = , u13 = , . . . , u1n = .
`11 `11 `11

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Algorithm
Consider the following matrix for Crout Decomposition:
    
a11 a12 ... a1n 1 0 ... 0 u11 u12 ... u1n
 a21 a22 ... a2n   `21 1 ... 0  0 u22 ... u2n 
 . . . . = . . . .  . . . . 
. . . . . . . . . . . .
. . .
 . . .
  . . .
 . . .

an1 an2 ... ann `n1 `n2 ... 1 0 0 ... unn

I Following the same fashion for i = 2, . . . , n and j = 2, . . . , n, one

can get i−1
X
j−1
aij − `ik ukj
X k=1
`ij = aij − `ik ukj and uij =
`jj
k=1

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
5 4 1 `11 0 0 1 u12 u13
    
 =  `21 `22 u23 
 10 9 4   0  0 1
  
10 13 15 `31 `32 `33 0 0 1

`11 = 5, `21 = 10 and `31 = 10.

    
5 4 1 5 0 0 1 u12 u13
    
 =  10 `22 u23 
 10 9 4   0  0 1
  
10 13 15 10 `32 `33 0 0 1
4 1
u12 = and u13 = .
5 5

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
4 1
5 4 1 5 0 0 1 5 5
    
 =  10 `22 u23 
 10 9 4   0  0 1
  
10 13 15 10 `32 `33 0 0 1

`22 = 1 and `32 = 5.

    
4 1
5 4 1 5 0 0 1 5 5
    
 =  10 u23 
 10 9 4   1 0  0 1
  
10 13 15 10 5 `33 0 0 1

u23 = 2.

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
4 1
5 4 1 5 0 0 1 5 5
    
 =  10
 10 9 4   1 0  0 1 2 
  
10 13 15 10 5 `33 0 0 1

`33 = 3.
    
4 1
5 4 1 5 0 0 1 5 5
    
 =  10
 10 9 4  1 0 
 0 1 2 
 
 
10 13 15 10 5 3 0 0 1

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Definition
An n × n symmetric matrix A is positive-definite if and only if xT Ax > 0
for all ~0 6= x ∈ Rn .

Definition
The Cholesky decomposition of a positive-definite matrix A is a
decomposition of the form A = LLT , where L is upper triangular with
positive diagonal entries.

Algorithm
Consider the following matrix for Cholesky Decomposition:
    
a11 a21 ... an1 `11 0 ... 0 `11 `21 ... `n1
 a21 a22 ... an2   `21 `22 ... 0  0 `22 ... `n2 
 . . . . = . . . .  . . . . 
 . . .. .   . . .. .  . . .. . 
. . . . . . . . .
an1 an2 ... ann `n1 `n2 ... `nn 0 0 ... `nn

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Algorithm
    
a11 a21 ... an1 `11 0 ... 0 `11 `21 ... `n1
 a21 a22 ... an2   `21 `22 ... 0  0 `22 ... `n2 
 . . . . = . . . .  . . . . 
 . . .. .   . . .. .  . . .. . 
. . . . . . . . .
an1 an2 ... ann `n1 `n2 ... `nn 0 0 ... `nn

I Diagonal entries are I Off-diagonal entries are

√ 1
`11 = a11 `21 = a21
`11
1
p
`22 = a22 − `221 `31 = a31
p `11
`33 = a33 − (`231 + `232 ) 1
`32 = (a32 − `31 `21 )
.. `22
. ..
.
q k−1
`kk = akk − (`2k1 + `2k2 + · · · + `2kk−1 ) 1 X
`ik = (aik − `ij `kj ).
`kk
j=1

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
25 15 −5 `11 0 0 `11 `21 `31
    
 =  `21 `22 `22 `32 
 15 18 0   0   0
  
−5 0 11 `31 `32 `33 0 0 `33

`11 = 5.

    
25 15 −5 5 0 0 5 `21 `31
    
 =  `21 `22 `22 `32 
 15 18 0  0 
 0
 
 
−5 0 11 `31 `32 `33 0 0 `33

`21 = 3 ve `31 = −1.

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
25 15 −5 5 0 0 5 3 −1
    
= 3 `22 `22 `32 
 15 18 0  0 
 0
 
 
−5 0 11 −1 `32 `33 0 0 `33

`22 = 3.

    
25 15 −5 5 0 0 5 3 −1
    
= 3 `32 
 15 18 0  3 0 
 0 3
 
 
−5 0 11 −1 `32 `33 0 0 `33

`32 = 1.

Dr. Mustafa Ağgül Hacettepe University

 Decomposition by Gaussian Elimination
MAT307 Numerical Analysis Doolittle and Crout Decompositions
Cholesky Decomposition

Example
    
25 15 −5 5 0 0 5 3 −1
    
= 3
 15 18 0  3 0 
 0 3 1 
 
 
−5 0 11 −1 1 `33 0 0 `33

`33 = 3.

    
25 15 −5 5 0 0 5 3 −1
    
= 3
 15 18 0  3 0 
 0 3 1 
 
 
−5 0 11 −1 1 3 0 0 3

Dr. Mustafa Ağgül Hacettepe University