MA423 Matrix Computations

Lectures 6 & 7: System of Linear Equations-I

Rafikul Alam
Department of Mathematics
IIT Guwahati

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Outline

• Solution of triangular system

• Gaussian elimination
• LU decomposition

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Linear system
Let A ∈ Rn×n be nonsingular and b ∈ Rn .

Problem: Solve Ax = b for x ∈ Rn .

Idea: For a nonsingular M, the solution of MAx = Mb is given by

x = (MA)−1 Mb = A−1 M −1 Mb = A−1 b.

So, the strategy is to choose M so that the system

MAx = Mb

is easy to solve. Gaussian elimination provides such an M for which MA

is upper triangular.

The matlab command

>> x= A\b
solves the system Ax = b using Gaussian elimination.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Lower triangular linear system

Consider the lower triangular linear system of equations

    
`11 x1 b1
`21 `22   x2   b 2 
  ..  =  ..  .
    
 .. .. . .
 . . .  .   . 
`n1 `n2 · · · `nn xn bn

By forward substitution, we have

x1 = b1 /`11 
Pi−1
xi = bi − j=1 `ij xj /`jj , i = 2 : n.

Cost: n2 flops
Pn Rn
Indeed, i=1 2i = 0 2xdx+ lower order terms ' n2 .

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Column-oriented forward substitution

Writing Lx = b as

L(:, 1)x(1) + · · · + L(:, n)x(n) = b

we obtain column-oriented forward substitution.

x = zeros(n,1);
for j=1:n-1
x(j) =b(j)/L(j,j);
b(j+1:n) = b(j+1:n)-L(j+1:n,j)*x(j);
end
x(n) = b(n)/L(n,n);

• Solving a lower triangular system costs n2 flops.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Upper triangular linear system
Consider the upper triangular system
    
u11 u12 ··· u1n x1 b1
 u22 ··· u2n  x2  b2 
..   ..  =  ..  .
    
 ..
 . .  .   . 
unn xn bn

If u11 , . . . , unn are nonzero, then by back substitution, we have a unique

solution
xn = bn /unn 
Pn
xi = bi − j=i+1 uij xj /ujj , i = n − 1, . . . , 1.

Cost: An upper triangular system is solved by back substitution and costs

n2 flops.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Gaussian elimination
Strategy: Transform a given linear system Ax = b to an equivalent
triangular linear system Ax
b =b b. Consider the system
 
x −y −z =2 1 −1 −1 2
3x − 3y + 2z = 16 ⇐⇒  3 −3 2 16 
2x − y + z = 9 2 −1 1 9
| {z }
augmented matrix

Use first equation to eliminating x from 2nd and 3rd equation

 
x −y −z =2 1 −1 −1 2
5z = 10 ⇐⇒  0 0 5 10  .
y + 3z = 5 0 1 3 5

Now interchange 2nd and 3rd equations

     
x −y −z =2 1 −1 −1 2 x 3
y + 3z = 5 ⇐⇒  0 1 3 5  ⇒ y  = −1 .
5z = 10 0 0 5 10 z 2

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Gaussian elimination (GE)

Gaussian elimination can be rewritten as a method that factorizes a

matrix. We consider three variants of GE. These variants yield three
matrix factorizations, namely,

• LU factorization: A = LU
• Row permuted LU factorization: PA = LU
• Row and column permuted LU factorization: PAQ = LU

Here P and Q are permutation matrices. An n × n permutation matrix is

obtained by permuting rows of the identity matrix In .

The matrix L is unit lower triangular and U is upper triangular. A lower

triangular matrix L is called unit lower triangular if the diagonal entries of
L are 1, that is, `jj = 1 for j = 1 : n.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

LU Decomposition
Definition: An LU decomposition of a matrix A ∈ Rn×n is a factorization
of the form A = LU, where L is unit lower triangular and U is upper
triangular. Thus
    
a11 · · · a1n 1 u11 · · · u1n
A =  ... .. ..  =  .. .. .. ..  = LU.
 
. .   . .  . . 
an1 ··· ann `n1 ··· 1 unn

We wish to construct a nonsingular M such that MA is upper triangular.

We expect M to have the following properties:
• M = L−1 −1 −1
n−1 Ln−2 · · · L1
• Each Lj is unit lower triangular
• The product L := L1 L2 · · · Ln−1 requires NO computation

Then
MA = U =⇒ L−1 −1 −1
n−1 Ln−2 · · · L1 A = U =⇒ A = LU,

where L is unit lower-triangular and U is upper-triangular.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
LU factorization
Suppose A is 4 × 4 matrix. Then schematically
   
× × × × × × × ×
× × × ×  × × ×
  −→  
× × × ×   × × ×
× × × × × × ×
| {z } | {z }
A L−1
1 A
   
× × × × × × × ×
 × × ×  × × ×
  −→  
 × × ×  × ×
× × × × ×
| {z } | {z }
L−1
1 A L−1 −1
2 L1 A
   
× × × × × × × ×
 × × ×  × × ×
  −→  
 × ×  × ×
× × ×
| {z } | {z }
L−1 −1
2 L1 A L−1 −1 −1
3 L2 L1 A

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Example
   
1 2 3 1 0 0
Let A := 4 5 6 . Consider L1 := 4 1 0 . Then
7 8 0 7 0 1
   
1 0 0 1 2 3
L−1
1 = −4 1 0 and L−1
1 A = 0 −3 −6  .
−7 0 1 0 −6 −21
   
1 0 0 1 0 0
Now consider L2 := 0 1 0 . Then L−1
2 = 0 1 0 ,
0 2 1 0 −2 1
   
1 2 3 1 0 0
U := L−1
2 L −1
1 A = 0 −3 −6 and L := L1 L2 = 4 1 0 .
0 0 −9 7 2 1

Thus we obtain A = LU.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Elimination matrix
 
0
 .. 
 . 
 
 0 
Define `k := 
  and Lk := I + `k e > for k = 1 : (n − 1).
k
`k+1,k 

 .. 
 . 
`nk
Then
 
Lk = I + 0 · · · 0 `k 0 · · · 0
 
1
 .. 

 . 

 1 
=  

 `k+1,k 1 

 .. . . 
 . . 
`nk 1

is unit lower triangular.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Product of elimination matrices
Consider Lk = I + `k ek> . By construction ek> `k = 0. Consequently

(I + `k ek> )(I − `k ek> ) = I + `k ek> − `k ek> − `k ek> `k ek> = I .

| {z }
Lk

This shows that L−1 >

k = I − `k ek . Next observe that

Lk Lk+1 = (I + `k ek> )(I + `k+1 ek+1

>
) = I + `k ek> + `k+1 ek+1
>
.

Consequently

L = L1 L2 · · · Ln−1 = I + `1 e1> + `2 e2> + · · · + `n−1 en−1

>
 
1
`21 1 
  
= I + `1 `2 · · · `n−1 0 = `31 `32 1

.
 
 .. .. .. .. 
 . . . . 
`n1 `n2 · · · `n,n−1 1

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Creating zeros via elimination matrix
Applying L−1
k to the k-th column of an n × n matrix A, we have
       
a1k a1k a1k 0
 ..   ..   ..   .. 
 .   .   .   . 
       
−1  akk  >  akk 
     akk   0 
Lk   = (I − `k ek ) ak+1,k  = ak+1,k  − `k+1,k  akk
   
ak+1,k       
 ..   ..   ..   .. 
 .   .   .   . 
ank ank ank `nk
 
ak1
 .. 
 . 
 
akk 
= 
 0  when `ik = aik /akk , i = k + 1 : n.

 
 .. 
 . 
0

This shows that if akk 6= 0 then Lk can be used to create zeros in the
k-th column of A below akk .
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Multiplying a matrix by an elimination matrix
Let A ∈ Rn×n and Lk := I + `k ek> ∈ Rn×n . Then

L−1 > >

 
k A = (I − `k ek )A = A − `k ek A = A − `k ak1 ··· akn

   
a11 ··· akk ··· a1n 0
 .. .. .. ..   .. 

 . . . ··· .   . 
  
 ak1 ··· akk ··· akn   0  
= −
 `k+1,k  ak1
 ··· akn .
 ak+1,n
 ··· ak+1,k ··· ak+1,n   
 .. .. .. ..   .. 
 . ··· . . .   . 
an1 ··· ank ··· ann `nk

The outer product shows that the first k rows of A remain unchanged
when L−1 −1
k is multiplied to the left of A. Let B := Lk A. In matlab, B
can be written compactly as a rank-1 update (outer product from)

B = A(k + 1 : n, :) − `(k + 1 : n) ∗ A(k, :)

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Gaussian elimination = LU decomposition
For L1 := I + `1 e1> , with `i1 := ai1 /a11 , i = 2 : n, we have

a11 a12 ··· a1n

 
(1) (1)
 0 a22 ··· a2n  (1)
 where aij = aij − `i1 a1j .
L−1 A = ,

1 .. .. ..
 Cost: 2(n − 1)2 flops.

 . . .
(1) (1)
0 an2 ··· ann

The elimination is possible only when a11 6= 0.

The vector `1 can be stored in the first column of L−1

1 A in place of zeros.
This will reduce the storage requirement.

Thus overwriting A, in matlab notation, we have

A(2 : n, 1) = A(2 : n, 1)/A(1, 1); % multipliers

A(2 : n, 2 : n) = A(2 : n, 2 : n) − A(2 : n, 1) ∗ A(1, 2 : n);

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Gaussian elimination = LU decomposition
(1) (1)
For L2 := I + `2 e2> with `i2 := ai2 /a22 , i = 3 : n, we have
 
a11 a12 a13 ··· a1n
(1) (1) (1)
 0 a22 a23 ··· a2n 
 where a(2) = a(1) − `i2 a(1) .
 
(2) (2)
L−1 −1 ···

2 L1 A =  0 0 a33 a3n , ij ij 2j
 Cost: 2(n − 2)2 flops.
.. .. .. ..

 
 . . . . 
(2) (2)
0 0 an3 ··· ann

(1)
The elimination is possible only when a22 6= 0. The vector `2 can be
stored in the second column of L−1 −1
2 L1 A in place of zeros.

Again, overwriting A, in matlab notation, we have

A(3 : n, 2) = A(3 : n, 2)/A(2, 2); % multipliers

A(3 : n, 3 : n) = A(3 : n, 3 : n) − A(3 : n, 2) ∗ A(2, 3 : n);

Hence we have L−1 −1

n−1 · · · L1 A = U ⇒ A = L1 L2 · · · Ln−1 U = LU.
Cost: 2(n − 1)2 + 2(n − 2)2 + · · · + 2 ' 2n3 /3 flops.
R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC
Example
 
2 4 −2
Consider A :=  4 9 −3 . Then
−2 −3 7
  
0 2 4 −2
L1 = I +  2  e1> , L−1
1 A= 0 1
 1 ,
−1 0 1 5
   
0 2 4 −2
L2 = I + 0 e2> , L−1 −1
2 L1 A = 0 1
 1 .
1 0 0 4

This gives
    
2 4 −2 1 0 0 2 4 −2
A = L1 L2 0 1 1 = 2 1 0 0 1 1 .
0 0 4 −1 1 1 0 0 4

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Gaussian Elimination with No Pivoting (GENP)
function [L, U] = GENP(A);
% [L U] = GENP(A) produces a unit
% lower triangular matrix L and an upper
% triangular matrix U so that A= LU.

[n, n] = size(A);
for k = 1:n-1
% compute multipliers for k-th step
A(k+1:n,k) = A(k+1:n,k)/A(k,k);
% update A(k+1:n,k+1:n)
j = k+1:n;
A(j,j) = A(j,j)-A(j,k)*A(k,j);
end
% strict lower triangle of A, plus I
L = eye(n,n)+ tril(A,-1);
U = triu(A); % upper triangle of A

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Solution of Ax = b by LU factorization
An n × n linear system Ax = b can be solved in three steps:
2n3
• Compute LU factorization A = LU. Cost: flops.
3
• Solve Ly = b for y . Cost: n2 flops.
• Solve Ux = y for x. Cost: n2 flops.

Thus the cost for solving system Ax = b is 2n3 /3 flops.

Question: What will be the complexity if the system Ax = b is solved as
x = A−1 ∗ b? Answer: 2n4 /3 flops.

Question: Does LU decomposition of A exist when A is nonsingular?

 
0 1
Answer: Not always. LU decomposition of A = does not exist.
1 1

Theorem: Let A be nonsingular. Then A admits a unique LU

factorization ⇔ all leading principal submatrices of A are nonsingular,
that is, A(1 : j, 1 : j) is nonsingular for j = 1 : n.

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC

Existence of LU factorization
Proof: Suppose that A = LU exists and unique. Then writing
    
A11 A12 L 0 U11 U12
= 11 ,
A21 A22 L21 L22 0 U22

we have det(A11 ) = det(L11 ) det(U11 ) = det(U11 ) 6= 0. (Why?)

Conversely, suppose that all leading principal submatrices of A are

nonsingular. We prove the result by induction. Suppose the result is true
for n − 1.

Let  = A(1 : n − 1, 1 : n − 1) and  = L̂Û be unique LU factorization.

Then writing
      
 b L̂ 0 Û u L̂Û L̂u
A= = = ,
c ann ` 1 0 d `Û `u + d

we have L̂u = b, `Û = c and d = ann − `u which give unique `, u, d.

Finally, 0 6= det(A) = det(Û)d ⇒ d 6= 0. This completes the proof. 

R. Alam, IIT Guwahati (July-Nov 2023) MA423 MC