Chapter 3

System of Equations
3.1 Direct methods for system of linear equations (SLE)
3.1.1 Gaussian method
3.1.2 Gaussian method with partial pivoting
3.1.3 Jordan's method
3.1.4 Jordan’s method for matrix inversion
3.1.5 Matrix decomposition

3.2 Indirect methods for SLE

3.2.1 Gauss Jacobi method
3.2.2 Gauss Seidel method

3.3 Systems of non-linear equations using Newton's method

3.1 Direct methods for system of
linear equations (SLE)

 Gaussian elimination method

 Gaussian method with partial pivoting
 Jordan's method
 Jordan’s method for matrix inversion
 Matrix decomposition

2
 Gaussian Elimination Method
A method to solve simultaneous linear
equations of the form AX=C

Two steps
1. Forward Elimination
2. Back Substitution
 Forward Elimination
The goal of forward elimination is to transform the
coefficient matrix into an upper triangular matrix

 25 5 1  x1  106.8 
 64 8 1  x   177.2 
   2  
144 12 1  x3  279.2

25 5 1   x1   106.8 
 0  4.8  1.56  x    96.21
   2  
 0 0 0.7   x3   0.735 
 Cont…
A set of n equations and n unknowns
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
. .
. .
. .
an1 x1  an 2 x2  an3 x3  ...  ann xn  bn

(n-1) steps of forward elimination

 Cont...
Step 1
For Equation 2, divide Equation 1 by a11 and
multiply by a21 .

 a21 
 a (a11 x1  a12 x2  a13 x3  ...  a1n xn  b1 )
 11 
a21 a21 a21
a21 x1  a12 x2  ...  a1n xn  b1
a11 a11 a11
 Cont...
Subtract the result from Equation 2.
a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
a21 a21 a21
− a21 x1  a a12 x2  ...  a a1n xn  a b1
_________________________________________________
11 11 11

 a21   a21  a21

 a22  a12  x2  ...   a2 n  a1n  xn  b2  b1
 a11   a11  a11

or a x  ...  a x  b
'
22 2
'
2n n
'
2
 Cont...
Repeat this procedure for the remaining
equations to reduce the set of equations as
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
'
a32 x2  a33
'
x3  ...  a3' n xn  b3'
. . .
. . .
. . .

an' 2 x2  an' 3 x3  ...  ann

'
xn  bn'

End of Step 1
 Cont...
Step 2
Repeat the same procedure for the 3rd term of
Equation 3.
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  a3" n xn  b3"
. .
. .
. .

an" 3 x3  ...  ann

"
xn  bn"
End of Step 2
 Cont...
At the end of (n-1) Forward Elimination steps, the
system of equations will look like
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  a3" n xn  b3"
. .
. .
. .

n 1 n 1 
ann xn  bn

End of Step (n-1)

 Cont...

Matrix Form at the end of Forward Elimination

a11 a12 a13  a1n   x1   b1 

 0 a' '
a23 '
 a 2 n x2     b2' 
 22    
0 0 "
a33  a3n   x3    b3 
" "

    
           
 0 0 0 (n 1 )
0 ann   xn  bn(n-1 ) 
 Back Substitution
Solve each equation starting from the last equation

25 5 1   x1   106.8 
 0  4.8  1.56  x    96.21
   2  
 0 0 0.7   x3   0.735 

Example of a system of 3 equations

Back Substitution Starting Eqns
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  an" xn  b3"
. .
. .
. .

n 1 n 1 
ann xn  bn
 Cont…
Start with the last equation because it has only one unknown
( n 1)
b
xn  n
( n 1)
a nn
 Cont…
( n 1)
b
xn  n
( n 1)
a nn

bii 1  ai,ii11 xi 1  ai,ii12 xi  2  ... ai,in1 xn

xi  i 1 for i  n  1,...,1
aii
i 1
  aiji 1 x j
n
bi
j i 1
xi  i 1 for i  n  1,...,1
a ii
 Example 1
Solve the following set of equations
using Gaussian elimination.

 25 5 1  a1  106.8 
 64   
8 1 a 2  177.2 
     
144 12 1  a3  279.2
 Cont…

 25 5 1  a1  106.8   25 5 1  106.8 
 64 8 1 a 2   177.2    64 8 1  177.2 
       
144 12 1  a3  279.2 144 12 1  279.2

1. Forward Elimination
2. Back Substitution
Forward Elimination
 Number of Steps of Forward
Elimination
Number of steps of forward elimination is
(n1)(31)2
 Forward Elimination: Step 1
 25 5 1  106.8  Divide Equation 1 by 25 and
 64 8 1  177.2 
  64
144 12 1  279.2 multiply it by 64,  2.56 .
25

25 5 1  106.8 2.56  64 12.8 2.56  273.408

. 64 8 1  177.2
Subtract the result from  64 12.8 2.56  273.408
Equation 2
0  4.8  1.56   96.208

 25 5 1  106.8 
Substitute new equation for  0  4.8  1.56   96.208
Equation 2  
144 12 1  279.2 
 Cont...
 25 5 1  106.8  Divide Equation 1 by 25 and
 0  4.8  1.56   96.208
 
144 12 1  279.2  multiply it by 144, 144  5.76 .
25

25 5 1  106.8 5.76  144 28.8 5.76  615.168

144 279.2
.
12 1 
Subtract the result from  144 28.8 5.76  615.168
Equation 3
0  16.8  4.76   335.968

25 5 1  106.8 
Substitute new equation for  0  4.8  1.56   96.208 
Equation 3  
 0  16.8  4.76   335.968
 Forward Elimination: Step 2
25 5 1  106.8  Divide Equation 2 by −4.8
 0  4.8  1.56   96.208 
  and multiply it by −16.8,
 0  16.8  4.76   335.968  16.8
. 3.5
 4.8
0  4.8  1.56   96.208 3.5  0  16.8  5.46   336.728
0  16.8  4.76  335.968
Subtract the result from
Equation 3  0  16.8  5.46   336.728
0 0 0.7  0.76

25 5 1  106.8 
Substitute new equation for  0  4.8  1.56   96.208
Equation 3  
 0 0 0.7  0.76 
 Back Substitution
25 5 1  106.8  25 5 1   a1   106.8 
 0  4.8  1.56   96.2   0  4.8  1.56 a    96.208
     2  
 0 0 0.7  0.7   0 0 0.7   a3   0.76 

Solving for a3
0.7 a3  0.76
0.76
a3 
0.7
a3  1.08571
 Cont...
25 5 1   a1   106.8 
 0  4.8  1.56 a    96.208
   2  
 0 0 0.7   a3   0.76 

Solving for a2
 4.8a2  1.56a3  96.208
 96.208  1.56a3
a2 
 4.8
 96.208  1.56 1.08571
a2 
 4.8
a2  19.6905
 Cont...
25 5 1   a1   106.8 
 0  4.8  1.56 a    96.2
   2  
 0 0 0.7   a3   0.76 

Solving for a1
25a1  5a2  a3  106.8
106.8  5a2  a3
a1 
25
106.8  5 19.6905  1.08571

25
 0.290472
Gaussian Elimination Solution
 25 5 1  a1  106.8 
 64 8 1 a2   177.2 
     
144 12 1  a3  279.2

 a1  0.290472
a    19.6905 
 2  
 a3   1.08571 
 Cont…

Solution  a1  0.290472
a    19.6905 
The solution vector is
 2  
 a3   1.08571 

The polynomial that passes through the three data points is then:
difficulty#1. Division by zero
10 x2  7 x3  3
6 x1  2 x2  3x3  11
5x1  x2  5x3  9

0 10  7  x1   3 
6 2 3   x2   11
    
5  1 5   x3   9 
Is division by zero an issue here?
12 x1  10 x2  7 x3  15
6 x1  5x2  3x3  14
5x1  x2  5x3  9

12 10  7  x1  15
6 5 3   x2   14
    
 5  1 5   x3   9 
 Is division by zero an issue here? YES
12 x1  10 x2  7 x3  15
6 x1  5x2  3x3  14
24 x1  x2  5x3  28

12 10  7  x1  15 12 10  7  x1   15 

6 5 3   x2   14  0 0 6.5  x2   6.5
         
24  1 5   x3  28 12  21 19   x3   2

Division by zero is a possibility at any step

of forward elimination
difficulty#2. Large Round-off Errors

 20 15 10  x1   45 
 3  2.249 7   x   1.751
   2  
 5  x  1 1
1
3   x3   9 
  
x  1
 2  
 x3  1
Exact Solution
 Cont...
 20 15 10  x1   45 
 3  2.249 7   x   1.751
   2  
 5 1 3   x3   9 

Solve it on a computer using 6 significant digits with chopping

 x1   0.9625 
 x    1.05 
 2  
 x3  0.999995
 Cont...
 20 15 10  x1   45 
 3  2.249 7   x   1.751
   2  
 5 1 3   x3   9 

Solve it on a computer using 5 significant digits with chopping

 x1   0.625 
 x    1.5 
 2  
 x3  0.99995

Is there a way to reduce the round off error?

Gaussian Elimination with Partial Pivoting
• Avoids division by zero
• Reduces round off error
Gaussian Elimination with
Partial Pivoting
 Row pivoting
• Most commonly used partial pivoting procedure

• Search the pivotal column

• Find the largest element in magnitude

• Then switch this row with the pivotal row

36
 Row pivoting

a11(1) a12(1) a13(1)  a1(i1)  a1(1j)  a1(1n) 

 ( 2) ( 2) 
 0 a22 a23  a2( 2i )  a2( 2j)  a2( 2n) 
 0 ( 3) ( 3)  Interchange
0 a33  a3(3i )  a3( 3j)  a3n
  these rows
          
 0 0 0  aii(i )  aij(i )  ain( i ) 
 
          
 0 0 0  a (jii )  a (jji )  a (jni ) 
 
          
 (i ) 
 0 0 0  ani(i )  anj(i )  ann 
Largest in magnitude 37
What is Different About Partial Pivoting?

At the beginning of the k th step of forward elimination,

find the maximum of

akk , ak 1,k ,................,ank

a pk
If the maximum of the values is
in the p th row, k  p  n, then switch rows p and k.
 Partial Pivoting
What does it Mean?
Gaussian Elimination with Partial Pivoting ensures that
each step of Forward Elimination is performed with the
pivoting element |akk| having the largest absolute value.
Matrix Form at Beginning of 2nd Step of
Forward Elimination

a11 a12 a13  a1n   x1   b1 

 0 a' '
a23  ' 
a 2 n x2   ' 
b2
 22    
 0 a32' '
a33  a3n   x3    b3 
' '

    
           
 0 a'n 2 an' 3 an' 4 ann'
  xn  bn' 
 Cont...
6 14 5.1 3.7 6   x1   5 
0  7 6 1 2   x 2    6
    
0 4 12 1 11  x3    8 
    
0 9 23 6 8   x4   9 
0  17 12 11 43  x5   3 

Which two rows would you switch?

 Cont...

6 14 5.1 3.7 6   x1   5 
0  17 12 11 43  x   3 
  2   
0 4 12 1 11  x3    8 
    
0 9 23 6 8   x4   9 
0  7 6 1 2   x5   6

Switched Rows
 Gaussian Elimination
with Partial Pivoting

A method to solve simultaneous linear

equations of the form AX=C

Two steps
1. Forward Elimination
2. Back Substitution
Matrix Form at the Beginning of 2nd Step
of Forward Elimination

a11 a12 a13  a1n   x1   b1 

 0 a' '
a23  ' 
a 2 n x2   ' 
b2
 22    
 0 a32' '
a33  a3n   x3    b3 
' '

    
          
 0 a'n 2 ' '
an 3 an 4 '
   
ann   xn  bn 
'
 Matrix Form at the End of Forward
Elimination

a11 a12 a13  a1n   x1   b1 

 0 a' a'23  a'2 n   x2   b2' 
 22    
0 0 "
a33  a3n   x3    b3 
" "

    
           
 0 0 0 (n 1 )
   
0 ann   xn  bn 
(n-1 )
Back Substitution Starting Eqns
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  an" xn  b3"
. .
. .
. .

n 1 n 1 
ann xn  bn
 Back Substitution
( n 1)
b
xn  n
( n 1)
a nn

i 1 n
i 1
bi   aij xj
j i 1
xi  i 1 for i  n  1,...,1
a ii
 Example 2
Solve the following set of equations
by Gaussian elimination with partial
pivoting

 25 5 1  a1  106.8 
 64   
8 1 a 2  177.2 
     
144 12 1  a3  279.2
 Cont…

 25 5 1  a1  106.8   25 5 1  106.8 
 64 8 1 a   177.2    64 8 1  177.2 
   2    
144 12 1  a3  279.2 144 12 1  279.2

1. Forward Elimination
2. Back Substitution
 Forward Elimination: Step 1
• Examine absolute values of first column, first row
and below.
25 , 64 , 144

• Largest absolute value is 144 and it exists in row 3.

• Switch row 1 and row 3.

 25 5 1  106.8  144 12 1  279.2

 64 8 1  177.2    64 8 1  177.2 
   
144 12 1  279.2  25 5 1  106.8 
 Cont...
144 12 1  279.2 Divide Equation 1 by 144 and
 64 8 1  177.2  64
  multiply it by 64,  0.4444 .
 25 5 1  106.8  144

144 12 1  279.2 0.4444  63.99 5.333 0.4444  124.1

.
64 8 1  177.2
Subtract the result from  63.99 5.333 0.4444  124.1
Equation 2 0 2.667 0.5556  53.10

144 12 1  279.2
Substitute new equation for  0 2.667 0.5556  53.10
Equation 2  
 25 5 1  106.8 
 Cont...
144 12 1  279.2 Divide Equation 1 by 144 and
 0 2.667 0.5556  53.10 25
  multiply it by 25,  0.1736.
 25 5 1  106.8  144

144 12 1  279.2 0.1736  25.00 2.083 0.1736  48.47

25 1  106.8
.
5
Subtract the result from  25 2.083 0.1736  48.47
Equation 3
0 2.917 0.8264  58.33

144 12 1  279.2
Substitute new equation for  0 2.667 0.5556  53.10
Equation 3  
 0 2.917 0.8264  58.33
 Step 2
• Examine absolute values of second column, second row
and below.
2.667 , 2.917

• Largest absolute value is 2.917 and it exists in row 3.

• Switch row 2 and row 3.

144 12 1  279.2 144 12 1  279.2

 0 2.667 0.5556  53.10   0 2.917 0.8264  58.33
   
 0 2.917 0.8264  58.33  0 2.667 0.5556  53.10
 Cont...
144 12 1  279.2 Divide Equation 2 by 2.917 and
 0 2.917 0.8264  58.33
  multiply it by 2.667, 2.667  0.9143.
 0 2.667 0.5556  53.10 2.917

0 2.917 0.8264  58.33 0.9143  0 2.667 0.7556  53.33

.
0 2.667 0.5556  53.10
Subtract the result from  0 2.667 0.7556  53.33
Equation 3 0 0  0.2   0.23

144 12 1  279.2 
Substitute new equation for  0 2.917 0.8264  58.33 
 
Equation 3  0 0  0.2   0.23
 Back Substitution

144 12 1  279.2  144 12 1   a1   279.2 

 0 2.917 0.8264  58.33    0 2.917 0.8264 a    58.33 
     2  
 0 0  0.2   0.23  0 0  0.2   a3   0.23

Solving for a3
 0.2a3  0.23
 0.23
a3 
 0.2
 1.15
 Cont...
144 12 1   a1   279.2 
 0 2.917 0.8264 a    58.33 
   2  
 0 0  0.2   a3   0.23

Solving for a2
2.917 a2  0.8264 a3  58.33
58.33  0.8264 a3
a2 
2.917
58.33  0.8264  1.15

2.917
 19.67
 Cont...
144 12 1   a1   279.2 
 0 2.917 0.8264 a    58.33 
   2  
 0 0  0.2   a3   0.23

Solving for a1
144a1  12a2  a3  279.2
279.2  12a2  a3
a1 
144
279.2  12 19.67  1.15

144
 0.2917
Gaussian Elimination with Partial Pivoting
Solution

 25 5 1  a1  106.8 
 64 8 1 a   177.2 
   2  
144 12 1  a3  279.2

 a1  0.2917
a    19.67 
 2  
 a3   1.15 
Gaussian Elimination Solution
 25 5 1  a1  106.8 
 64 8 1 a2   177.2 
     
144 12 1  a3  279.2

 a1  0.290472
a    19.6905 
 2  
 a3   1.08571 
 Partial Pivoting: Example 3
Consider the system of equations

10 x1  7 x2  7
 3x1  2.099 x2  6 x3  3.901
5 x1  x2  5 x3  6
In matrix form

 10  7 0  x1   7 
 3 2.099 6  x  3.901
   2 =  
 5  1 5  x 3   6 

Solve using Gaussian Elimination with Partial Pivoting using five

significant digits
 Partial Pivoting: Example
Forward Elimination: Step 1
Examining the values of the first column
|10|, |-3|, and |5| or 10, 3, and 5
The largest absolute value is 10, which means, to follow the
rules of Partial Pivoting, we switch row1 with row1.

Performing Forward Elimination

10 7 0  x1   7 

 10  7 0  x1   7 
 3 2.099 6  x   3.901  0  0.001 6  x   6.001
  2      2   
 5  1 5  x3   6   0 2.5 5  x3   2.5 
 Cont...
Forward Elimination: Step 2
Examining the values of the first column
|-0.001| and |2.5| or 0.0001 and 2.5
The largest absolute value is 2.5, so row 2 is switched with
row 3

Performing the row swap

10 7 0  x1   7  10 7 0  x1   7 


 0  0.001 6  x   6.001 0 2.5 5   x    2.5 
  2      2   
 0 2.5 5  x3   2.5   0  0.001 6  x3  6.001
 Cont...
Forward Elimination: Step 2

Performing the Forward Elimination results in:

10  7 0   x1   7 
 0 2.5 5   x    2.5 
  2   
 0 0 6.002  x3  6.002
 Cont...
Back Substitution
Solving the equations through back substitution

10  7 0   x1   7 
x3 
6.002
1
 0 2.5 5   x    2.5  6.002
  2   
 0 0 6.002  x3  6.002
2.5  5 x3
x2   1
2.5
7  7 x2  0 x3
x1  0
10
 Cont...
Compare the calculated and exact solution
The fact that they are equal is coincidence, but it does
illustrate the advantage of Partial Pivoting

 x1   0   x1   0 
X  calculated   x2    1 X  exact   x 2    1
 x3   1   x3   1 
 Gauss-Jordan Method
• The Gauss-Jordan Method is similar to the
Gaussian Elimination.

• The method requires almost 50% more

operations.
 Gauss-Jordan Method

The Gauss-Jordan method changes the matrix into the

identity matrix.
 Gauss-Jordan Method
There are one phases to the solving technique
• Elimination --- use row operations to convert
the matrix into an identity matrix.
• The new b vector is the solution to the x
values.
 Gauss-Jordan Algorithm
[A]{x} ={b}

• Augment the n x n coefficient matrix with the vector of right

hand sides to form a n x (n+1)
• Interchange rows if necessary to make the value a11 with the
largest magnitude of any coefficient in the first row
• Create zero in 2nd through nth row in first row by subtracting
ai1 / a11 times first row from ith row
 Gauss-Jordan Elimination Algorithm
• Repeat (2) & (3) for first through the nth rows, putting the
largest magnitude coefficient in the diagonal by interchanging
rows (consider only row j to n ) and then subtract times the jth
row from the ith row so as to create zeros in all positions of jth
column and the diagonal becomes all ones
• Solve for all of the equations, xi = ai,n+1
 Gauss-Jordan Elimination
• Solve:
2 x1  3x2  7
4 x1  5 x2  13

• Only care about numbers – form “tableau” or

“augmented matrix”:

2 3 7
 
4 5 13
 Gauss-Jordan Elimination
• Given:
2 3 7
 
4 5 13
• Goal: reduce this to trivial system

1 0 ?
 
0 1 ?
and read off answer from right column
 Gauss-Jordan Elimination

2 3 7
 
4 5 13

• Basic operation 1: replace any row by

linear combination with any other row
• Here, replace row1 with 1/2 * row1
1 3
2
7 2
 
4 5 13
 Gauss-Jordan Elimination

1 3
2
7 2
 
4 5 13

• Replace row2 with row2 – 4 * row1

1 3
2
7 2
 
0 1  1
• Negate row2
1 3
2
7  2
 
0 1 1 
 Gauss-Jordan Elimination

1 3
2
7 2
 
0 1 1 

• Replace row1 with row1 – 3/2 * row2

1 0 2
 
0 1 1
• Read off solution: x1 = 2, x2 = 1
 Gauss-Jordan Elimination
• For each row i:
– Multiply row i by 1/aii
– For each other row j:
• Add –aji times row i to row j
• At the end, left part of matrix is identity,
answer in right part
• Can solve any number of R.H.S.
simultaneously
 Example
• Solve the system of equations using Gauss-
Jordan Method
 R1  R2  R2
x  y  2z  0 1 1 2 0 
1 1 2 0
x  2 y  3 z  1  
1 2 3 1 1 2 3 1
2 x  2 y  z  3  2 2 1 3 0 1 1 1
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
 R1  R2  R2
x  y  2z  0 1 1 2 0 
  1 1 2 0
x  2 y  3 z  1  0 1 1 1 1 2 3 1
2 x  2 y  z  3  2 2 1 3 0 1 1 1
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
2 R1  R3  R3
x  y  2z  0 1 1 2 0 
  2 2 4 0
x  2 y  3 z  1  0 1 1 1 2 2 1 3
2 x  2 y  z  3  2 2 1 3 0 0 3 3
 Cont…
• Solve the system of equations using Gauss-
Jordan Method

x  y  2z  0 1 1 2 0  2 R1  R3  R3
  2 2 4 0
x  2 y  3 z  1 0 1 1 1 2 2 1 3
2 x  2 y  z  3 0 0 3 3 0 0 3 3
 Cont…
• Solve the system of equations using Gauss-
Jordan Method

x  y  2z  0 1 1 2 0  1R2  R2
  0 1 1 1
x  2 y  3 z  1 0 1 1 1
2 x  2 y  z  3 0 0 3 3
 Cont…
• Solve the system of equations using Gauss-
Jordan Method

x  y  2z  0 1 1 2 0  1R2  R2
  0 1 1 1
x  2 y  3 z  1 0 1 1 1 
2 x  2 y  z  3 0 0 3 3
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
R2  R1 
x  y  2z  0 1 1 2 0  R1
  0 1 1 1
x  2 y  3 z  1 0 1 1 1  1 1 2 0
2 x  2 y  z  3 0 0 3 3 1 0 1 1
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
R2  R1 
x  y  2z  0 1 0 1 1  R1
  0 1 1 1
x  2 y  3 z  1 0 1 1 1  1 1 2 0
2 x  2 y  z  3 0 0 3 3 1 0 1 1
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
1
x  y  2z  0 1 0 1 1   R3  R3
3
 
x  2 y  3 z  1 0 1 1 1 
0 0 1 1

2 x  2 y  z  3 0 0 3 3
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
1
x  y  2z  0 1 0 1 1  R3  R3
3
 
x  2 y  3 z  1 0 1 11
0 0 1 1

2 x  2 y  z  3 0 0 1 1
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
R3  R2 
x  y  2z  0 1 0 1 1 R2
  0 0 1 1
x  2 y  3 z  1 0 1 11 0 1 1 1
2 x  2 y  z  3 0 0 1 1 0 1 0 2
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
R3  R2 
x  y  2z  0 1 0 1 1  R2
  0 0 1 1
x  2 y  3 z  1 0 1 0 2  0 1 1 1
2 x  2 y  z  3 0 0 1 1  0 1 0 2
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
 R3  R1  R1
x  y  2z  0 1 0 1 1 
  0 0 1 1
x  2 y  3 z  1 0 1 0 2  1 0 1 1
2 x  2 y  z  3 0 0 1 1  1 0 0 0
 Cont…
• Solve the system of equations using Gauss-
Jordan Method
 R3  R1  R1
x  y  2z  0 1 0 0 0 
  0 0 1 1
x  2 y  3 z  1 0 1 0 2  1 0 1 1
2 x  2 y  z  3 0 0 1 1  1 0 0 0
 Cont…
• Solve the system of equations using Gauss-
Jordan Method

x  y  2z  0 1 0 0 0 
  (0, 2, 1)
x  2 y  3 z  1 0 1 0 2 
2 x  2 y  z  3 0 0 1 1 
 LU Decomposition Method
LU Decomposition is another method to solve a set of
simultaneous linear equations.
For most non-singular matrix [A] that one could conduct
Gauss Elimination forward elimination steps, one can
always write it as
[A] = [L][U]
where
[L] = lower triangular matrix
[U] = upper triangular matrix
How does LU Decomposition work?
If solving a set of linear [A][X] = [C]
equations [L][U][X] = [C]
If [A] = [L][U] then [L]-1
Multiply by [L]-1[L][U][X] = [L]-1[C]
Which gives [I][U][X] = [L]-1[C]
Remember [L]-1[L] = [I] which [U][X] = [L]-1[C]
leads to
[L]-1[C]=[Z]
Now, if [I][U] = [U] then
[L][Z] = [C] (1)
Now, let
[U][X] = [Z] (2)
Which ends with
and
 LU Decomposition
How can this be used?

Given [A][X] = [C]

1. Decompose [A] into [L] and [U]
2. Solve [L][Z] = [C] for [Z]
3. Solve [U][X] = [Z] for [X]

94
 Decompose [A] to [L] and [U]

1 0 0 u11 u12 u13 

A  LU    21 1 0  0 u 22 u 23 
 31  32 1  0 0 u 33 

[U] is the same as the coefficient matrix at the end of the

forward elimination step.
[L] is obtained using the multipliers that were used in the
forward elimination process
 Finding the [U] matrix
Using the Forward Elimination Procedure of Gauss Elimination

 25 5 1
 64 8 1
 
144 12 1
 25 5 1 
 2.56; Row2  Row12.56   0  4.8  1.56
64
Step 1:
25
144 12 1 

25 5 1 
 5.76; Row3  Row15.76   0  4.8  1.56 
144
25
 0  16.8  4.76
 Finding the [U] Matrix
25 5 1 
Matrix after Step 1:  0  4.8  1.56 
 
 0  16.8  4.76

25 5 1 
 16.8
Step 2:  3.5; Row3  Row23.5   0  4.8  1.56
 4.8
 0 0 0.7 

25 5 1 
U    0  4.8  1.56
 0 0 0.7 
 Finding the [L] matrix
1 0 0
 1 0
 21
 31  32 1

Using the multipliers used during the Forward Elimination Procedure

a 21 64
 25 5 1  21    2.56
From the first  64 8 1 a11 25
step of forward   a31 144
elimination 144 12 1  31    5.76
a11 25
 Finding the [L] Matrix

From the second 25 5 1 

 0  4.8  1.56   32  a32   16.8  3.5
step of forward   a 22  4.8
elimination  0  16.8  4.76

 1 0 0
L  2.56 1 0
5.76 3.5 1
 Does [L][U] = [A]?

 1 0 0 25 5 1 
LU   2.56 1 0  0  4.8  1.56 
5.76 3.5 1  0 0 0.7 
?
3.2 Indirect Methods for Solving SLE
 Gauss Jacobi method
 Gauss Seidel method
 Iterative Techniques
• The method of solving simultaneous linear algebraic
equations using Gaussian Elimination and the Gauss-
Jordan Method. These techniques are known as
direct methods.
• Problems can arise from round-off errors and zero on
the diagonal.
• One means of obtaining an approximate solution to
the equations is to use an “educated guess”.
 Iterative Methods

• We will look at two iterative methods:

– Gauss-Jacobi Method
– Gauss-Seidel Method
 Introduction
• For systems that have coefficient
matrices with the appropriate structure –
especially large, sparse systems (many
coefficients whose value is zero) –
iterative techniques may be preferable.
 4 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0   u1  0.08
 1 4  1 0 0  1 0 0 0 0  
 0 0 0 0 0 0   u 2  0.16
 0 1 4 1 0 0 1 0 0 0 0 0 0 0 0 0   u3  0.36
     
 0 0 1 4 0 0 0 1 0 0 0 0 0 0 0 0   u 4  1.64 
 1 0 0 0 4  1 0 0  1 0 0 0 0 0 0 0   u5  0.16
     
 0 1 0 0 1 4 1 0 0 1 0 0 0 0 0 0   u 6   0.0 
 0 0 1 0 0 1 4 1 0 0  1 0 0 0 0 0   u 7   0.0 
     
 0 0 0 1 0 0 1 4 0 0 0  1 0 0 0 0   u8   1.0 
     
 0 0 0 0 1 0 0 0 4 1 0 0  1 0 0 0   u9  0.36
 0 0 0 0 0 1 0 0 1 4  1 0 0  1 0 0  u10   0 
     
 0 0 0 0 0 0 1 0 0 1 4  1 0 0  1 0  u11   0 
 0 0 0 0 0 0 0 1 0 0  1 4 0 0 0  1 u12   1.0 
     
 0 0 0 0 0 0 0 0 1 0 0 0 4  1 0 0  u13  1.64 
 0 0 0 0 0 0 0 0 0 1 0 0  1 4  1 0  u14   1.0 
   
0 0 0 0 0 0 0 0 0 0  1 0 0  1 4  1 u15   1.0 
     
0 0 0 0 0 0 0 0 0 0 0  1 0 0  1 4  u16   2.0 
 Basic Idea
• Convert the system Ax  B
into the equivalent system x  Cx  d
a12 a13 b1
x1   x2  x3 
a11 a11 a11
a11 x1  a12 x2  a13 x3  b1
a21 a23 b2
a21 x1  a22 x1  a23 x3  b2 x2   x1  x1 
a22 a22 a22
a31 x1  a32 x1  a33 x3  b3
a31 a b3
x3   x1  32 x2 
a33 a33 a33

• Generate a sequence of approximation x (1) , x (2) ,...

where ,
x(k )  Cx(k 1)  d
 Jacobi Method
1
• Consider the two-by-two x   y3
2
system 1
2x  y  6 y   x3
x  2y  6 2

• Start with x  y  1/ 2
(1) (1)

• Simultaneous updating 1 (1) 1 11

– New values of the variables x ( 2)
  y 3  3
2 4 4
are not used until a new 1 1 11
iteration step is begun y ( 2)   x (1)  3    3 
2 4 4
 Jacobi Method

• Con’t
1 ( 2) 11 13
x (3)
  y 3   3 
2 8 8
1 ( 2) 11 13
y (3)
  x 3   3 
2 8 8
 Jacobi Method
• Consider the three-by-three system
2 x1  x2  x3  1
x1  2 x2  x3  6
x1  x2  2 x3  3
x1  0.5x2  0.5x3  0.5
x2  0.5 x1  0.5x3  3.0
x3  0.5 x1  0.5 x2  1.5

 x1( k )   0.0 0.5  0.5  x1( k 1)   0.5

 (k )     x ( k 1)    3.0 
 2  
x   0.5 0.0 0.5   2   
 x   0.5 0.5 0.0   x
( k ) ( k 1)    1.5 
 3     3   

Start with x (0)  (0,0,0)

 Jacobi Iteration
• If the diagonal is dominant, the matrix can be
rewritten in the following form
 Jacobi Iteration
 The technique can be rewritten in a shorthand fashion,
where D is the diagonal, A” is the matrix without the
diagonal and c is the right-hand side of the equations.
 Jacobi Iteration
• The technique solves for the entire set of x
values for each iteration.

• The problem does not update the values until

an iteration is completed.
 Gauss-Seidel Method

• Iterative or approximate methods provide

an alternative to the elimination methods.

• The Gauss-Seidel method is the most

commonly used iterative method.
 b1  a12 x2  a13 x3
x1 
a11
b2  a21 x1  a23 x3
x2 
a22
b3  a31 x1  a32 x2
x1 
a33

•Now we can start the solution process by

choosing guesses for the x’s. A simple way to
obtain initial guesses is to assume that they are
zero. These zeros can be substituted into
x1equation to calculate a new x1=b1/a11.
• New x1 is substituted to calculate x2 and x3. The
procedure is repeated until the convergence criterion
is satisfied:

xij  xij 1
 a ,i  j < s
100% 
xi

For all i, where j and j-1 are the present and

previous iterations.
Convergence Criterion for Gauss-Seidel Method
• The Gauss-Seidel method has two fundamental
problems as any iterative method:
– It is sometimes nonconvergent, and
– If it converges, converges very slowly.
• Recalling that sufficient conditions for convergence
of two linear equations, u(x,y) and v(x,y) are
u u
 1
<
x y
v v
 1
<
x y
• Similarly, in case of two simultaneous equations, the
Gauss-Seidel algorithm can be expressed as

b1 a12
u ( x1 , x2 )   x2
a11 a11
b2 a21
v( x1 , x2 )   x1
a22 a22
u u a12
0 
x1 x2 a11
v a21 v
 0
x1 a22 x2
• Substitution into convergence criterion of two linear
equations yield:
a12 a21
<1
 <1

a11 a22
• In other words, the absolute values of the
slopes must be less than unity for convergence:
a11 > a12
a22 
> a21
For n equations :
n
aii >  ai , j
j 1
j i
 Gauss-Seidel Method
Basic Procedure:
-Algebraically solve each linear equation for xi
-Assume an initial guess solution array
-Solve for each xi and repeat
-Use absolute relative approximate error after each iteration to
check if error is within a pre-specified tolerance.
 Gauss-Seidel Method
Why?
If the physics of the problem are understood, a close initial guess can
be made, decreasing the number of iterations needed.
 Gauss-Seidel Method
Algorithm
A set of n equations and n unknowns:
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
a21 x1  a22 x2  a23 x3  ...  a2n xn  b2
. .
. .
. .

an1 x1  an 2 x2  an3 x3  ...  ann xn  bn

 Gauss-Seidel Method
Algorithm
Rewriting each equation
c  a12 x 2  a13 x3   a1n x n From Equation 1
x1  1
a11

c2  a21 x1  a23 x3   a2 n xn
x2  From equation 2
a22
  
cn 1  an 1,1 x1  an 1, 2 x2   an 1,n  2 xn  2  an 1,n xn From equation n-1
xn 1 
an 1,n 1
cn  an1 x1  an 2 x2    an ,n 1 xn 1 From equation n
xn 
ann
 Gauss-Seidel Method
Algorithm
General Form of each equation
n

a
n
c1   a1 j x j cn 1  n 1, j xj
j 1 j 1
j  n 1
x1 
j 1 xn 1 
a11 an 1,n 1
n
cn   a nj x j
n
c2   a2 j x j
j 1 j 1
j n
x2 
j 2
xn 
a 22 a nn
Gauss-Seidel Method
Algorithm
General Form for any row ‘i’
n
ci   aij x j
j 1
j i
xi  , i  1,2,, n.
aii
 Gauss-Seidel Method
Solve for the unknowns
Assume an initial guess for [X]
Use rewritten equations to solve for
 x1  each value of xi.

x  Important: Remember to use the most

 2 recent value of xi.

  
 
 xn-1 
 xn 
 Gauss-Seidel Method
Calculate the Absolute Relative Approximate Error
x inew  x iold
a i  new
100
xi
So when has the answer been found?

The iterations are stopped when the absolute relative

approximate error is less than a prespecified tolerance for all
unknowns.
 Gauss-Seidel Method: Example
Given the system of equations The coefficient matrix is:
12x 1  3x 2 - 5x 3  1
x 1  5x 2  3x 3  28 12 3  5

3x1  7x2  13x3  76

A   1 5 3 
 3 7 13 
With an initial guess of
Will the solution converge using the
 x1  1
Gauss-Siedel method?
 x   0
 2  
 x3  1
 Gauss-Seidel Method: Example
Checking if the coefficient matrix is diagonally dominant
a11  12  12  a12  a13  3   5  8
12 3  5
A   1 5 3  a22  5  5  a21  a23  1  3  4
 3 7 13 
a33  13  13  a31  a32  3  7  10

The inequalities are all true and at least one row is strictly greater than:
Therefore: The solution should converge using the Gauss-Siedel Method
 Gauss-Seidel Method: Example
12 3  5  a1   1   x1  1
 1 5 3  a   28 With an initial guess of  x   0
   2    2  
 3 7 13   a3  76  x3  1

Rewriting each equation

1  3 x 2  5 x3 1  30   51
x1  x1   0.50000
12 12
28  x1  3 x3 28  0.5  31
x2  x2   4.9000
5 5
76  3 x1  7 x 2
x3  76  30.50000   74.9000 
13 x3   3.0923
13
Gauss-Seidel Method: Example
The absolute relative approximate error
0.50000  1.0000
a 1  100  67.662%
0.50000

4.9000  0
a 2
 100  100.00%
4.9000

3.0923  1.0000
a 3
 100  67.662%
3.0923

The maximum absolute relative error after the first iteration is 100%
 Gauss-Seidel Method: Example
After Iteration #1
 x1  0.5000 
 x   4.9000
 2  
 x3  3.0923
After Iteration #2
Substituting the x values into the equations
1  34.9000   53.0923  x1  0.14679
x1   0.14679  x    3.7153 
12  2  
 x3   3.8118 
28  0.14679   33.0923
x2   3.7153
5

76  30.14679   74.900 
x3   3.8118
13
Gauss-Seidel Method: Example
Iteration #2 absolute relative approximate error
0.14679  0.50000
a 1  100  240.62%
0.14679
3.7153  4.9000
a 2
 100  31.887%
3.7153
3.8118  3.0923
a 3  100  18.876%
3.8118
The maximum absolute relative error after the second iteration is 240.62%
 Gauss-Seidel Method: Example
Repeating more iterations, the following values are obtained
Iteration a1
a 1 a2
a 2
a3
a 3

1 0.50000 67.662 4.900 100.00 3.0923 67.662

2 0.14679 240.62 3.7153 31.887 3.8118 18.876
3 0.74275 80.23 3.1644 17.409 3.9708 4.0042
4 0.94675 21.547 3.0281 4.5012 3.9971 0.65798
5 0.99177 4.5394 3.0034 0.82240 4.0001 0.07499
6 0.99919 0.74260 3.0001 0.11000 4.0001 0.00000

 x1  0.99919
The solution obtained   
x
 2   3.0001 
 x3   4.0001 
 x1  1
is close to the exact solution of  x2   3
 x3  4
 3.3 Newton’s Method for
System of non-linear Equations
• Consider a non linear system of equations
f1 ( x1 , x2 ,..., xn )  0
f 2 ( x1 , x2 ,..., xn )  0

f n ( x1 , x2 ,..., xn )  0
 Cont...
• The totality of the arguments x1 , x2 ,...,xn
May be considered as an n-dimenstional vector.
 x1   f1 
   
 x2   f2 
x  f  
 
   
x  f 
 n  n
Therefore, the above system f ( x)  0
 Cont...
• System f ( x)  0 is solved by the method of Successive
approximations.
• Suppose the kth approximation
x  ( x , x ,..., x )
k k
1
k
2
k T
n

• Then the exact root of this equation can be

represented in the form x  x ( k )  e( k )
Where e k  (e1k , e2k ,...,enk error
) of the root .
 Cont...
• On the assumption that the function f ( x) is0
continuously differentiable in the interval
containing x and x k
• Using Taylor series we expand f (x)about x k

f ( x k  e k )  f ( x k )  f ' ( x k )e k  0
 Cont...
Where  f1 f1 f 
 1
 x x2 xn
 1 
f ( x )  
' k
  W (xk )
 
 f n f n f n 

 x1 x2 xn 

f ( x k )  w( x k )e k  0

then using Gaussian Elimination we solve for ek

 Cont...
• Then k 1
x  x e
k k

 x1k 1   x1k  e1k 

 k 1   k   k 
 x2    x2   e2 
     
     
 xn   xnk  enk 
k 1
Example:Using Newton’s approximation
the solution of

x12  x22  x32  1

2 x12  x22  4 x3  0
3x12  4 x2  x32  0
given x10  x20  x30  0.5
 Cont...
 0.25 2 x1 2 x2 2 x3  1 1 1 
f ( x 0 )   1.25  w( x)  4 x1 2 x2  4  w( x 0 )  2 1  4
 1.00  6 x1  4 2 x3  3  4 1 

Then w( x 0 )(e 0 )   f ( x 0 )
1 1 1  e1  0.25
0

2 1  4 e 0   1.25 
  2   

3  4 1  e3  1.00 
  
0
 Cont...
• Then using Gaussian Elimination Method we solve

e10  0.375, e20  0.000, e30  0.125

• Then we compute the next approximation

x1  x 0  e 0
 x11  0.875
 1  
 2 
x  0.500 
 x1  0.375
 3  