NUMERICAL ANALYSIS

R. Owusu
February 6, 2023

1 OVERVIEW: Matrix Algebra

What is a matrix?
A matrix is a rectangular array of numbers arranged in m horizontal rows
and n vertical columns.
 
a11 a12 a13 · · · a1n
 a21 a22 a23 · · · a2n 
 
A =  a31 a32 a33 · · · a3n  (1)
 
 .. .. .. . . .. 
 . . . . . 
am1 am2 am3 · · · amn
We say that A is an m × n matrix. aij in the ith row and jth column of
A is called the (i, j)th element of A and we write the matrix

A = (aij )m×n

If m = n, then A is called a square matrix of order n.

aii , i = 1, 2, ..., n are called the diagonal elements of the square matrix.

1.1 Properties of Matrices

Let A, B, C be three matrices of order m × n. Then

1. A · (B · C) = (A.B).C

2. A · (B + C) = A · B + A · C

1
 3. (A + B) · C = A · C + B · C, then n × n diagonal matrix, where aij = I
for all i is called the identity matrix of order n and is denoted by In .
For example,
 
  1 0 0
1 0
I2 = and I3 =  0 1 0 
0 1
0 0 1
4. If A is n × n matrix then A · In = In = A

5. If A is a square matrix of order n then we define Ap = A · A, · · · , A (p

factor ) and A0 = I , then we have

Ap Aq = Ap+q

for non-negative integers P, q

1.2 Some Special Matrices

1.2.1 SYMMETRIC MATRICES
A matrix A = [aij ] is called symmetric if A = AT . That is A is symmetric
if and only if aij = aji , for 1 ≤ i ≤ m, 1 ≤ j ≤ n.

1.2.2 UNITARY MATRICES

A square matrix A with the property that A−1 = A is said to be unitary.
 
1 0 ··· 0
 0 1 ··· 0 
 .. .. . . . ..  (2)
 
 . . . 
0 0 ··· 1

1.2.3 DIAGONAL MATRIX

A square matrix A = (aij )n×n such that aij = 0 for i 6= j is called a diagonal
matrix.
 
a11 0 0 ··· 0
 0 a22 0 · · · 0 
 
 0 0 a33 · · · 0  (3)
 .. .. .. . . .. 
 
 . . . . . 
0 0 0 · · · ann

2
1.2.4 TRIDIAGONAL MATRIX
A square matrix A = (aij )n×n in the form
 
a11 a12 0 0 0
 a21 a22 a23 0 0 
..
 
A =  0 a32 a33
 . 0

 (4)

 0 . .. . ..

0 an−1,n 
0 0 0 an,n−1 ann

is called a tridiagonal matrix.

1.2.5 INVERSE MATRICES

DEFINITION:
If A is a square matrix and we can find another matrix of the same size, say
B, such that
AB = BA = I
then we call A invertible and we say that B is an inverse of the matrix
A. If we can’t find such a matrix B, we call A a singular matrix and
(det(A) = 0).

THEOREM

Suppose that A is invertible and that both B and C are inverses of A,

then B = C and we will denote the inverse as A−1 .

AA−1 = A−1 A = I
THEOREM

Suppose that A and B are invertible matrices of the same size. Then,

1. AB is invertible and (AB)−1 = B −1 A

2. A−1 is invertible and (A−1 )−1 = A

3. For n = 0, 1, 2, ..., An is ivertible and (An ) = A−n = (A−1 )n .

4. If c is a non - zero scalar cA is invertible and (cA)−1 = 1c A−1 .

5. AT is invertible and (AT )−1 = (A−1 )T

3
 If A is an n × n matrix then the following statements are equivalent

1. A is invertible

2. The only solution to the system Ax = 0 is the trivial solution.

3. A is row equivalent to In .

4. Ax = b has exactly one solution for every n × 1 matrix b.

5. Ax = b is consistent for every n × 1 matrix b

1.2.6 Upper and Lower Triangular Matrix

A square matrix A = (aij )n×n is said to be an lower triangular matrix if
aij = 0, for all i < j. If aij = 0, for all i > j, then A is an upper triangular
matrix.
   
l11 0 0 · · · 0 u11 u12 u13 · · · u1n
 l21 l22 0 · · · 0   0 u22 u23 · · · u2n 
   
L =  l31 l32 l33 · · · 0  ; U =  0 0 u33 · · · u3n 
  
 .. .. .. . . ..   .. .. .. . . .. 

 . . . . .   . . . . . 
ln1 ln2 ln3 · · · lnn 0 0 0 · · · unn

2 SOLUTION OF SYSTEM OF LINEAR EQUA-

TIONS
We will need the following general system of n equations and m unknowns.

a11 x1 + a12 x2 + · · · + a1m xm = b1

a21 x1 + a22 x2 + · · · + a2m xm = b2
.. .. .. .. .. .
.+. . . . = ..
an1 x1 + an2 x2 + · · · + anm xm = bn

In this system aij , bj ∈ R and xi ’s are the unknowns.

2.1 THEOREM
Given a system of n equations and m unknowns, there will be one of three
possibilities for solutions to the system.

1. There will be no solution.

4
 2. There will be exactly one solution.

3. There will be infinitely many solutions.

NB: If there is at least one solution to the system, we call it consistent
it is called inconsistent system.

2.2 SOLUTION OF SYSTEM OF LINEAR EQUATIONS

The above system can be written as;
Ax = b
     
a11 a12 a13 ··· a1n x1 b1

 a21 a22 a23 ··· a2n 

 x2 
 
 b2 
 
A=
 a31 a32 a33 ··· a3n  , x =  x 3  ; b =  b3 
    
.. .. .. .. ..  ..   .. 
. . . . .  .  .
 
 
am1 am2 am3 · · · amn xm bn
If A = D is a square-diagonal matrix, then
b1 b2 b3 bn
x1 = , x2 = , x3 = , · · · , xn = ,
d11 d22 d dnn
If A is the lower triangular matrix
 
l11 0 0 · · · 0
 l21 l22 0 · · · 0 
 
L =  l31 l32 l33 · · · 0 
 
 .. .. .. . . . 
 . . . . .. 
ln1 ln2 ln3 · · · lnn
Lx = b has the solution (Forward Substitution);
b1 1 1
x1 = , x2 = (b2 − l21 x1 ) , x3 = (b3 − l31 x1 − l32 x2 )
l11 l22 l33
Thus
1
xn = (bn − ln1 x1 − ln2 x2 − · · · − ln,n−1 xn−1 )
lnn
If A is the lower triangular matrix
 
u11 u12 u13 · · · u1n
 0 u22 u23 · · · u2n 
 
U =
 0 0 u 33 · · · u 3n

(5)
 .. .. .. . . .. 

 . . . . . 
0 0 0 · · · unn

5
 U x = b has the solution; (Backward Substitution)
bn 1
xn = , xn−1 = (bn − un−1,n xn ) , . . .
unn un−1,n
1
x3 = (b3 − −u34 x4 − . . . − u3n xn − u3n−1 xn−1 )
u3n
Then finally
1
x1 = (b1 − u12 x2 − u13 x3 − · · · − u1,n−1 xn−1 − u1,n xn )
u11
We may not all the time have simple systems of linear equations where A is
a (1) Diagonal matrix, (2) Upper triangular matrix or Lower triangular
matrix.

What do we do, if A is not in the forms above? Or what if A is not

invertible? Can we reduce A in the forms above?
If YES, then it becomes simple to solve the system of equations.

To do this, we find some general purpose algorithms to solve Ax = b.

1. Direct method
(a) Gaussian Elimination Method
(b) Gauss-Jordan Method
(c) Decomposition method (LU and Cholesky)
2. Iterative method
(a) Gauss-Jacobi method
(b) Gauss-Seidel method

3 Gaussian Elimination
How is a set of equations solved numerically? One of the most popular tech-
niques for solving simultaneous linear equations is the Gaussian elimination
method. The approach is designed to solve a general set of n equations and
n unknowns
Gaussian elimination consists of two steps

• Forward Elimination of Unknowns

• Back Substitution

6
3.1 Gaussian elimination: Without Pivoting - Naive
Assume an n equations and n unknowns as below

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + · · · + a2n xn = b2
.
a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = ..
an1 x1 + an2 x2 + an3 x3 + · · · + ann xn = bn

3.2 Forward Elimination of Unknowns

Below are the steps involved:

• STEP 1:

The first unknown, x1 is eliminated.

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + · · · + a2n xn = b2
.. .. .. ..
. . . .
an1 x1 + an2 x2 + an3 x3 + · · · + ann xn = bn
By dividing Equation 1 by a11 and multiplying by a21

 
a21
(a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1 )
a11
a21 a21 a21
a12 x2 + · · · +
a21 x1 + a1n xn = b1
a11 a11 a11
Subtract this equation from the first equation to give
   
a21 a2 a21
a22 − a12 x2 + · · · + a2n − a1n xn = b2 − b1
a11 a11 a11
or
0 0 0
a22 x2 + · · · + a2n xn = b2

Repeat this process for the third equation to the nth equation to produce

7
 a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1
0 0 0 0
a22 x2 + a23 x3 + · · · + a2n xn = b2
0 0 0 0
a32 x2 + a33 x3 + · · · + a3n xn = b3
.. .. ..
. . .
0 0 0 0
an2 x2 + an3 x3 + · · · + ann xn = bn

• STEP 2: Repeat this process again to eliminate x2

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

0 0 0 0
a22 x2 + a23 x3 + · · · + a2n xn = b2
0 0 0 0
a32 x2 + a33 x3 + · · · + a3n xn = b3
.. .. ..
. . .
0 0 0 0
an2 x2 + an3 x3 + · · · + ann xn = bn

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

0 0 0 0
a22 x2 + a23 x3 + · · · + a2n xn = b2
00 00 00 00
a32 x2 + a33 x3 + · · · + a3n xn = b3
.. .. ..
. . .
00 00 00 00
an2 x2 + an3 x3 + · · · + ann xn = bn

At the end of the (n-1) forward eliminations we should get

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

0 0 0 0
a22 x2 + a23 x3 + · · · + a2n xn = b2
00 00 00
+ a33 x3 + · · · + a3n xn = b3
.. ..
. .
(n−1) (n−1)
ann x n = bn

END OF FORWARD ELIMINATION: Matrix Form at End of Forward

Elimination
    
a11 a12 a13 · · · a1n x1 b1
 0 a012 a013 · · · 0
a1n  b2
0
  x2  
   
00 00 00
 
 0 0 a13 · · · a1n 
  x3  = 
   b3 
 .. .. .. .. .   ..
 
 . . . ··· .   ..   .
  

(n−1) (n−1)
0 0 0 0 ann xn bn

8
3.3 Back Substitution
From the last step in the Forward Elimination process, proceed by solving
from the last equation. Thus;
(n−1)
bn
xn = (n−1)
ann
(i−1) (i−1) (i−1) (i−1)
bi − ai,i+1 xi+1 − ai,i+2 xi+2 − · · · − ai,n xn
xi = (i−1)
aii
for i = n-1, · · · ,1 or
n
(i−1)
bi−1
P
i − ai,j xj
j=i+1
xi = (i−1)
for i = n − 1, n − 2, · · · , 1
aii

3.4 Pitfalls of Naive Gaussian Elimination

1. Division by Zero

2. Round-off Error

Example     
25 5 1 a1 106.8
 64 8 1   a2  =  177.2 
144 12 1 a3 279.2

Find the values of a1 , a2 and a3 using the Naive Gaussian Elimination.

Solution

• First Step Divide Row 1 by 25 and multiply by 64

 
64
[ 25 5 1 106.8 ] ∗
25

to get
[ 64 12.8 2.56 273.408 ]
subtract result from Row 2
[64 8 1 177.2]
− [64 12.8 2.56 273.408]
0 −4.8 −1.56 −96.208

9
 Expected Result
    
25 5 1 a1 106.8
 0 −4.8 −1.56   a2  =  −96.208 
144 12 1 a3 279.2

Divide Row 1 by 25 and multiply by 144

 
144
[ 25 5 1 106.8 ] ∗
25
to get
[ 144 28.8 5.76 615.168 ]
subtract result from Row 3
[144 12 1 279.2]
− [144 28.8 5.76 615.168]
0 −16.8 −4.76 −335.968

Expected Result
    
25 5 1 a1 106.8
 0 −4.8 −1.56   a2  =  −96.208 
0 −16.8 −4.76 a3 −335.968

• Second Step Divide Row 2 by -4.8 and multiply by -16.8

 
−16.8
[ 0 −4.8 −1.56 −96.208 ] ∗
−4.8
to get
[ 0 −16.8 −5.46 −336.728 ]

subtract result from Row 3

[0 −16.8 −4.76 −335.968]
− [0 −16.8 −5.46 −336.728]
0 0 0.7 0.76

Expected Result
    
25 5 1 a1 106.8
 0 −4.8 −1.56   a2  =  −96.208 
0 0 0.7 a3 0.76

10
 • Back Substitution From third equation
0.7a3 = 0.76
0.76
a3 = 0.7
= 1.08571
Put value of a3 into second equation
−4.8a2 − 1.56a3 = −96.208

The solution is
−96.208+1.56∗a3
a2 = −4.8
−96.208+1.56∗1.08571
= −4.8
= 19.6905
Put value of a2 and a3 into first equation
25a1 + 5a2 + a3 = 106.8
106.8−5∗a2 −a3
a1 = 25
106.8−5∗19.6905−1.08571
= 25
= 0.290472
Hence the solution vector is
   
a1 0.290472
 a2  =  29.6905 
a3 1.08571

3.5 Gaussian Elimination: With Partial Pivoting

What are some techniques for improving the Naive Gaussian elimination
method?
To avoid division by zero as well as reduce round-off error, Gaussian elimi-
nation with partial pivoting is the method of choice.
How does Gaussian elimination with partial pivoting differ from Naive
Gaussian elimination?

The two methods are the same, except in the beginning of each step of
forward elimination, a row switching is done based on the following criterion.
• STEP 1
Take the maximum of
|akk |, |ak−1,k |, · · · · · · , |ank |
at the beginning of each k th step of the forward elimination

11
 • STEP 2
Assume |apk | is the maximum in the pth row, swap that row with the
k th row.

• END OF GAUSSIAN ELIMINATION with PARTIAL PIVOTING.

Example: Given the system below solve for the solution

20x1 + 15x2 + 10x3 = 45

−3x1 − 2.248x2 + 7x3 = 1.751
5x1 + x2 + 3x2 = 9

Solution: In matrix notation

    
20 15 10 x1 45
 −3 −2.249 7   x2  =  1.751 
5 1 3 x3 9

• Forward Elimination of Unknowns

The largest in absolute value in the first column

|20|, | − 3|, |5|

is identified and row swapped with the first row

20 is the largest and at the same time in first row hence:

    
20 15 10 x1 45
 −3 −2.249 7   x2  =  1.751 
5 1 3 x3 9

• STEP 1
Divide row 1 by 20 and multiply by -3 thus
−3
20
= −0.15

( [20 15 10] [45] ∗ −0.15 )

This gives
[−3 −2.25 −1.5] [−6.75]
Subtract result from Row 2
[−3 −2.2249 7] [1.751]
− [−3 −2.25 −1.5] [−6.75]
0 0.001 8.5 8.501

12
 Expected Result
    
20 15 10 x1 45
 0 0.001 8.5   x2  =  8.501 
5 1 3 x3 9

• Divide row 1 by 20 and multiply by 5 thus 5/20 = 0.25

( [20 15 10] [45] ∗ 0.25 )

This gives
[5 3.75 2.5] [11.25]
Subtract result from Row 3
[5 1 3] [9]
− [5 3.75 2.5] [11.25]
0 −2.75 0.5 −2.25

Expected Result
    
20 15 10 x1 45
 0 0.001 8.5   x2  =  8.501 
0 −2.75 0.5 x3 −2.25

• STEP 2
The largest in absolute value in the second column

|0.001|, | − 2.75|

is identified and row swapped with the second row −2.75 is the largest
thus row 3 hence swapped with row 2
    
20 15 10 x1 45
 0 −2.75 0.5   x2  =  −2.25 
0 0.001 8.5 x3 8.501

• Divide row 2 by -2.75 and multiply by 0.001 thus

0.001/ − 2.75 = −0.00036363

13
 ( [0 −2.75 0.5] [−2.25] ∗ −0.0003633 )
This gives

[0 0.00099998 −0.00018182] [0.00081816]

Subtract result from Row 3

[0 0.001 8.5] [8.501]

− [0 0.00099998 −0.00018182] [0.00081816]
0 0 8.50018182 8.50018184
Expected Result
    
20 15 10 x1 45
 0 −2.75 0.5   x2  =  −2.25 
0 0 8.5001 x3 8.5001

• Back Substitution

8.5001x3 = 8.5001
8.5001
x3 =
8.5001
= 1

Put the value of x3 in Row 2

−2.75x2 + 0.5x3 = −2.25

−2.25 − 0.5x3
x2 =
−2.75
−2.25 − 0.5 ∗ 1
=
−2.75
−2.25 − 0.5
=
−2.75
−2.75
=
−2.75
= 1

14
 Put x3 and x2 into Row 1

20x1 + 15x2 + 10x3 = 45

45 + 15x2 − 10x3
x1 =
20
45 − 15 ∗ 1 − 10 ∗ 1
=
20
30 − 10
=
20
= 20/20 = 1

So the Solution is
  
x1 1
[X] =  x2 = 1 
 
x3 1
Can we use Naive Gaussian elimination methods to find the determinant
of a square matrix?
One of the more efficient ways to find the determinant of a square matrix
is by taking advantage of the following two theorems on a determinant of
matrices.
Theorem
Let [A] be a n ∗ n matrix,if [B] is a n ∗ n matrix that results from adding or
subtracting a multiple of one row to another row then
det(A) = det(B)
Theorem
Let [A] be an n ∗ n matrix that is either an upper or lower triangular or
diagonal then
det(A) = a11 ∗ a22 ∗ · · · ∗ aii ∗ · · · ∗ ann
Yn
= aii
i=1

What if we cannot find the determinant of the matrix using the Naive
Gaussian elimination method because of division by zero problems during
the Naive Gaussian elimination method?
Then Gaussian elimination with partial pivoting can be applied. However,
the determinant of the resulting upper triangular matrix may differ by a
sign.

15
 The following theorem applies in addition to the previous two to find the
determinant of a square matrix.

Theorem
Let [A] be an n ∗ n matrix, if [B] is a matrix resulting from switching one
row with the another row then

det(B) = −det(A)

Key Note

If [A] is an n ∗ n matrix and det(A) 6= 0 then

1. [A] is invertible

2. [A]−1 exists

3. [A][X] = [C] has unique solution

4. [A][X] = 0 solution is [X] = [0]

5. [A][A]−1 = [I] = [A]−1 [A]

16