Gaussian Elimination

Major: All Engineering Majors

Author(s): Autar Kaw

http://numericalmethods.eng.usf.edu
Transforming Numerical Methods Education for STEM
Undergraduates
 Naïve Gauss Elimination

http://numericalmethods.eng.usf.edu
 Naïve Gaussian Elimination
A method to solve simultaneous linear
equations of the form [A][X]=[C]

Two steps
1. Forward Elimination
2. Back Substitution
 Forward Elimination
The goal of forward elimination is to transform the
coefficient matrix into an upper triangular matrix

 25 5 1  x1  106.8 
 64 8 1  x   177.2 
   2  
144 12 1  x3  279.2

25 5 1   x1   106.8 
 0  4.8  1.56  x     96.21
   2  
 0 0 0.7   x3   0.735 
 Forward Elimination
A set of n equations and n unknowns
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
. .
. .
. .
an1 x1  an 2 x2  an 3 x3  ...  ann xn  bn

(n-1) steps of forward elimination

 Forward Elimination
Step 1
For Equation 2, divide Equation 1 by a11 and
multiply by a21 .

 a21 
 a ( a11 x1  a12 x2  a13 x3  ...  a1n xn  b1 )
 11 
a21 a21 a21
a21x1  a12 x2  ...  a1n xn  b1
a11 a11 a11
 Forward Elimination
Subtract the result from Equation 2.
a21 x1  a22 x2  a23 x3  ...  a2 n xn  b2
a21 a21 a21
− a21 x1  a a12 x2  ...  a a1n xn  a b1
11 11 11
_________________________________________________
 a21   a21  a21
 a22  a12  x2  ...   a2 n  a1n  xn  b2  b1
 a11   a11  a11

or a x  ...  a x  b
'
22 2
'
2n n
'
2
 Forward Elimination
Repeat this procedure for the remaining
equations to reduce the set of equations as
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
'
a32 x2  a33
'
x3  ...  a3' n xn  b3'
. . .
. . .
. . .

an' 2 x2  an' 3 x3  ...  ann

'
xn  bn'

End of Step 1
 Forward Elimination
Step 2
Repeat the same procedure for the 3rd term of
Equation 3.
a11 x1  a12 x2  a13 x3  ...  a1n xn  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  a3" n xn  b3"
. .
. .
. .

an" 3 x3  ...  ann

"
xn  bn"
End of Step 2
 Forward Elimination
At the end of (n-1) Forward Elimination steps, the
system of equations will look like
a11 x1  a12 x 2  a13 x3  ...  a1n x n  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
a x  ...  a x  b
"
33 3
"
3n n
"
3
. .
. .
. .

 n 1  n 1 
ann xn  bn

End of Step (n-1)

Matrix Form at End of Forward
Elimination
a11 a12 a13  a1n   x1   b1 
0 '
a22 '
a23  '
a 2 n x2    b2' 
    
0 0 "
a
33  a3n   x3    b3 
" "

    
          
 0 0 0 0 ann   xn  bn 
(n 1 )
    (n-1 )
 Back Substitution
Solve each equation starting from the last equation

25 5 1   x1   106.8 
 0  4.8  1.56  x    96.21
   2  
 0 0 0.7   x3   0.735 

Example of a system of 3 equations

Back Substitution Starting Eqns
a11 x1  a12 x 2  a13 x3  ...  a1n x n  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  an" xn  b3"
. .
. .
. .

 n 1  n 1 
ann xn  bn
 Back Substitution
Start with the last equation because it has only one unknown
( n 1)
b
xn  n
( n 1)
a nn
 Back Substitution
( n 1)
b
xn  n
( n 1)
a nn

bi i 1  ai ,ii11 xi 1  ai,ii12 xi  2  ...  ai ,in1 xn

xi   i 1 for i  n  1,...,1
aii
n
 i 1
bi   aij i 1 x j
j i 1
xi   i 1 for i  n  1,...,1
a ii
 THE END

http://numericalmethods.eng.usf.edu
 Additional Resources
For all resources on this topic such as digital audiovisual
lectures, primers, textbook chapters, multiple-choice tests,
worksheets in MATLAB, MATHEMATICA, MathCad and
MAPLE, blogs, related physical problems, please visit

http://numericalmethods.eng.usf.edu/topics/gaussian_elimi
nation.html
 Naïve Gauss Elimination
Example

http://numericalmethods.eng.usf.edu
 Example 1
The upward velocity of a rocket is given at three
different times

Table 1 Velocity vs. time data.

Time, t  s  Velocity, v  m/s 

5 106.8
8 177.2
12 279.2

The velocity data is approximated by a polynomial as:

v t   a1t 2  a 2 t  a3 , 5  t  12.
Find the velocity at t=6 seconds .
 Example 1 Cont.
Assume
v t   a1t 2  a2t  a3 , 5  t  12.

Results in a matrix template of the form:

t12 t1 1  a1   v1 
 2   a   v 
t 2 t2 1  2  2
t32 t3 1 
 a3 
  v3 

 

Using data from Table 1, the matrix becomes:

 25 5 1  a1  106.8 
 64 8 1 a   177.2 
   2  
144 12 1  a3  279.2
 Example 1 Cont.
 25 5 1  a1  106.8   25 5 1  106.8 
 64 8 1 a 2   177.2    64 8 1  177.2 
       
144 12 1  a 3  279.2 144 12 1  279.2

1. Forward Elimination
2. Back Substitution
Forward Elimination
 Number of Steps of Forward
Elimination
Number of steps of forward elimination is
(n1)(31)2
 Forward Elimination: Step 1
 25 5 1  106.8  Divide Equation 1 by 25 and
 64 8 1  177.2 
  64
multiply it by 64,  2.56 .
144 12 1  279.2 25
 25 5 1  106.8  2.56   64 12.8 2.56  273.408

.  64 8 1  177.2
Subtract the result from   64 12.8 2.56  273.408
Equation 2
0  4.8  1.56   96.208

 25 5 1  106.8 
Substitute new equation for  0  4.8  1.56   96.208
Equation 2  
144 12 1  279.2 
Forward Elimination: Step 1 (cont.)
 25 5 1  106.8 
Divide Equation 1 by 25 and
 0  4.8  1.56   96.208
  144
144 12 1  279.2  multiply it by 144, 25  5.76 .

 25 5 1  106.8  5.76  144 28.8 5.76  615.168

Subtract the result from 144 279.2

.
12 1 
Equation 3  144 28.8 5.76  615.168
 0  16.8  4.76   335.968
25 5 1  106.8 
Substitute new equation for 
Equation 3 0  4.8  1.56   96.208 
 0  16.8  4.76   335.968
 Forward Elimination: Step 2
25 5 1  106.8  Divide Equation 2 by −4.8
 0  4.8  1.56   96.208 
  and multiply it by −16.8,
 0  16.8  4.76   335.968  16.8
 3.5
 4. 8 .
0  4.8  1.56   96.208  3.5   0  16.8  5.46   336.728
0  16.8  4.76  335.968
Subtract the result from
Equation 3  0  16.8  5.46   336.728
0 0 0 .7  0.76

25 5 1  106.8 
Substitute new equation for  0  4.8  1.56   96.208
Equation 3  
 0 0 0.7  0.76 
Back Substitution
 Back Substitution
25 5 1  106.8  25 5 1   a1   106.8 
 0  4.8  1.56   96.2   0  4.8  1.56 a    96.208
     2  
 0 0 0.7  0.7   0 0 0.7   a3   0.76 

Solving for a3
0.7 a3  0.76
0.76
a3 
0 .7
a3  1.08571
 Back Substitution (cont.)
25 5 1   a1   106.8 
 0  4.8  1.56 a    96.208
   2  
 0 0 0.7   a3   0.76 

Solving for a2
 4.8a2  1.56a3  96.208
 96.208  1.56a3
a2 
 4.8
 96.208  1.56 1.08571
a2 
 4.8
a2  19.6905
 Back Substitution (cont.)
25 5 1   a1   106.8 
 0  4.8  1.56 a    96.2
   2  
 0 0 0.7   a3   0.76 

Solving for a1
25a1  5a2  a3  106.8
106.8  5a2  a3
a1 
25
106.8  5  19.6905  1.08571

25
 0.290472
Naïve Gaussian Elimination Solution
 25 5 1  a1  106.8 
 64 8 1 a2   177.2 
     
144 12 1  a3  279.2

 a1  0.290472
a    19.6905 
 2  
 a3   1.08571 
 Example 1 Cont.
Solution  a1  0.290472
a    19.6905 
The solution vector is
 2  
 a3   1.08571 

The polynomial that passes through the three data points is then:

v t   a1t 2  a2t  a3
 0.290472t 2  19.6905t  1.08571, 5  t  12

v 6  0.290472 6  19.6905 6  1.08571

2

 129.686 m/s.
 THE END

http://numericalmethods.eng.usf.edu
Naïve Gauss Elimination
Pitfalls

http://numericalmethods.eng.usf.edu
Pitfall#1. Division by zero
10 x2  7 x3  3
6 x1  2 x2  3x3  11
5 x1  x2  5 x3  9

0 10  7  x1   3 
6 2 3   x2   11
    
5  1 5   x3   9 
Is division by zero an issue here?
12 x1  10 x2  7 x3  15
6 x1  5 x2  3x3  14
5 x1  x2  5 x3  9

12 10  7  x1  15
6 5 3   x2   14
    
 5  1 5   x3   9 
 Is division by zero an issue here?
YES
12 x1  10 x2  7 x3  15
6 x1  5 x2  3x3  14
24 x1  x2  5 x3  28

12 10  7  x1  15 12 10  7  x1   15 

6 5 3   x2   14  0
     0 6.5  x2    6.5
    
24  1 5   x3  28 12  21 19   x3    2

Division by zero is a possibility at any step of

forward elimination
Pitfall#2. Large Round-off Errors
 20 15 10  x1   45 
  3  2.249 7   x   1.751
   2  
 5 1 3   x3   9 

Exact Solution

 x1  1
 x   1
 2  
 x3  1
Pitfall#2. Large Round-off Errors
 20 15 10  x1   45 
  3  2.249 7   x   1.751
   2  
 5 1 3   x3   9 

Solve it on a computer using 6 significant digits with chopping

 x1   0.9625 
 x    1.05 
 2  
 x3  0.999995
Pitfall#2. Large Round-off Errors
 20 15 10  x1   45 
  3  2.249 7   x   1.751
   2  
 5 1 3   x3   9 

Solve it on a computer using 5 significant digits with chopping

 x1   0.625 
 x    1.5 
 2  
 x3  0.99995

Is there a way to reduce the round off error?

 Avoiding Pitfalls
Increase the number of significant digits
• Decreases round-off error
• Does not avoid division by zero
 Avoiding Pitfalls

Gaussian Elimination with Partial Pivoting

• Avoids division by zero
• Reduces round off error
 THE END

http://numericalmethods.eng.usf.edu
 Gauss Elimination with
Partial Pivoting

http://numericalmethods.eng.usf.edu
 Pitfalls of Naïve Gauss Elimination
• Possible division by zero
• Large round-off errors
 Avoiding Pitfalls
Increase the number of significant digits
• Decreases round-off error
• Does not avoid division by zero
 Avoiding Pitfalls
Gaussian Elimination with Partial Pivoting
• Avoids division by zero
• Reduces round off error
 What is Different About Partial
Pivoting?
At the beginning of the kth step of forward elimination,
find the maximum of

akk , ak 1,k ,................, ank

If the maximum of the values is a pk
in the p th
row, k  p  n, then switch rows p and
k.
Matrix Form at Beginning of 2nd
Step of Forward Elimination
a11 a12 a13  a1n   x1   b1 
0 '
a22 '
a23  ' 
a 2 n x2   ' 
b2
    
0 a '
32 a '
33  a3n   x3    b3 
' '

    
           
 0 '
an 2 ' '
an 3 an 4 ann   xn  bn 
'
    '
 Example (2nd step of FE)

6 14 5.1 3.7 6   x1   5 
0  7 6 1 2   x2    6
    
0 4 12 1 11   x3    8 
    
 0 9 23 6 8   x 4  9
 
0  17 12 11 43  x5   3 

Which two rows would you switch?

Example (2nd step of FE)

6 14 5.1 3.7 6   x1   5 
0  17 12 11 43  x   3 
  2   
0 4 12 1 11   x3    8 
    
 0 9 23 6 8 x
   
4 9
0  7 6 1 2   x5    6

Switched Rows
 Gaussian Elimination
with Partial Pivoting
A method to solve simultaneous linear
equations of the form [A][X]=[C]

Two steps
1. Forward Elimination
2. Back Substitution
 Forward Elimination
Same as naïve Gauss elimination method
except that we switch rows before each
of the (n-1) steps of forward elimination.
Example: Matrix Form at Beginning
of 2nd Step of Forward Elimination
a11 a12 a13  a1n   x1   b1 
0 a '
a '
 ' 
a 2 n x2   ' 
b2
 22 23    
0 a '
32 a '
33  a3n   x3    b3 
' '

    
          
 0 a '
n2 a '
n3 a '
n4 ann   xn  bn 
'
    '
Matrix Form at End of Forward
Elimination
a11 a12 a13  a1n   x1   b1 
0 '
a22 '
a23  '
a 2 n x2    b2' 
    
0 0 "
a
33  a3n   x3    b3 
" "

    
         
 0 0 0 0 ann   xn  bn 
(n 1 )
    (n-1 )
Back Substitution Starting Eqns
a11 x1  a12 x 2  a13 x3  ...  a1n x n  b1
'
a22 x2  a23
'
x3  ...  a2' n xn  b2'
"
a33 x3  ...  an" xn  b3"
. .
. .
. .

 n 1  n 1 
ann xn  bn
 Back Substitution
( n 1)
b
xn  n
( n 1)
a nn

n
 i 1  i 1
bi   aij x j
j i 1
xi   i 1 for i  n  1,...,1
a ii
 THE END

http://numericalmethods.eng.usf.edu
 Gauss Elimination with
Partial Pivoting
Example

http://numericalmethods.eng.usf.edu
 Example 2
Solve the following set of equations
by Gaussian elimination with partial
pivoting

 25 5 1  a1  106.8 
 64   
8 1 a 2  177.2  
     
144 12 1  a 3  279.2
 Example 2 Cont.
 25 5 1  a1  106.8   25 5 1  106.8 
 64 8 1 a   177.2    
    2   64 8 1  177. 2 
144 12 1  a 3  279.2 144 12 1  279.2

1. Forward Elimination
2. Back Substitution
Forward Elimination
 Number of Steps of Forward
Elimination

Number of steps of forward elimination is

(n1)=(31)=2
 Forward Elimination: Step 1
• Examine absolute values of first column, first row
and below.
25 , 64 , 144
• Largest absolute value is 144 and exists in row 3.
• Switch row 1 and row 3.

 25 5 1  106.8  144 12 1  279.2

 64 8 1  177.2    64 8 1  177.2 
   
144 12 1  279.2  25 5 1  106.8 
Forward Elimination: Step 1 (cont.)
144 12 1  279.2 Divide Equation 1 by 144 and
 64 8 1  177.2  64
  multiply it by 64,  0.4444 .
 25 5 1  106.8  144

144 12 1  279.2  0.4444   63.99 5.333 0.4444  124.1

.
Subtract the result from  64 8 1  177.2
Equation 2   63.99 5.333 0.4444  124.1
0 2.667 0.5556  53.10

Substitute new equation for 144 12 1  279.2

Equation 2  0 2.667 0.5556  53.10
 
 25 5 1  106.8 
Forward Elimination: Step 1 (cont.)
144 12 1  279.2 Divide Equation 1 by 144 and
 0 2.667 0.5556  53.10 25
  multiply it by 25,  0.1736 .
 25 5 1  106.8  144

144 12 1  279.2  0.1736   25.00 2.083 0.1736  48.47

.
 25 5 1  106.8
Subtract the result from
Equation 3   25 2.083 0.1736  48.47
0 2.917 0.8264  58.33

Substitute new equation for 144 12 1  279.2

Equation 3  0 2.667 0.5556  53.10
 
 0 2.917 0.8264  58.33
 Forward Elimination: Step 2
• Examine absolute values of second column, second row
and below.
2.667 , 2.917
• Largest absolute value is 2.917 and exists in row 3.
• Switch row 2 and row 3.

144 12 1  279.2 144 12 1  279.2

 0 2.667 0.5556  53.10   0 2.917 0.8264  58.33
   
 0 2.917 0.8264  58.33  0 2.667 0.5556  53.10
 Forward Elimination: Step 2 (cont.)
Divide Equation 2 by 2.917 and
144 12 1  279.2
 0 2.917 0.8264  58.33 multiply it by 2.667,
  2.667
 0 2.667 0.5556  53.10  0.9143.
2.917
0 2.917 0.8264  58.33  0.9143   0 2.667 0.7556  53.33

.
0 2.667 0.5556  53.10
Subtract the result from  0 2.667 0.7556  53.33
Equation 3
0 0  0.2   0.23

144 12 1  279.2 
Substitute new equation for  0 2.917 0.8264  58.33 
Equation 3  
 0 0  0.2   0.23
Back Substitution
 Back Substitution
144 12 1  279.2  144 12 1   a1   279.2 
 0 2.917 0.8264  58.33    0 2.917 0.8264 a    58.33 
     2  
 0 0  0.2   0.23  0 0  0.2   a3   0.23

Solving for a3
 0.2a3  0.23
 0.23
a3 
 0.2
 1.15
 Back Substitution (cont.)
144 12 1   a1   279.2 
 0 2.917 0.8264 a    58.33 
   2  
 0 0  0.2   a3    0.23

Solving for a2
2.917a2  0.8264a3  58.33
58.33  0.8264a3
a2 
2.917
58.33  0.8264 1.15

2.917
 19.67
 Back Substitution (cont.)
144 12 1   a1   279.2 
 0 2.917 0.8264 a    58.33 
   2  
 0 0  0.2   a3    0.23

Solving for a1
144a1  12a2  a3  279.2
279.2  12a2  a3
a1 
144
279.2  12 19.67  1.15

144
 0.2917
Gaussian Elimination with Partial
Pivoting Solution

 25 5 1  a1  106.8 
 64 8 1 a   177.2 
   2  
144 12 1  a3  279.2

 a1  0.2917 
a    19.67 
 2  
 a3   1.15 
 Gauss Elimination with
Partial Pivoting
Another Example

http://numericalmethods.eng.usf.edu
 Partial Pivoting: Example
Consider the system of equations
10 x1  7 x2  7
 3 x1  2.099 x2  6 x3  3.901
5 x1  x2  5 x3  6

In matrix form
 10  7 0  x1   7 
 3 2.099 6  x  3.901
   2 =  
 5  1 5  x3   6 

Solve using Gaussian Elimination with Partial Pivoting using five

significant digits with chopping
 Partial Pivoting: Example
Forward Elimination: Step 1
Examining the values of the first column
|10|, |-3|, and |5| or 10, 3, and 5
The largest absolute value is 10, which means, to
follow the rules of Partial Pivoting, we switch
row1 with row1.

Performing Forward Elimination

 10  7 0  x1   7  10 7 0  x1   7 
 3 2.099 6  x   3.901

 5
 2   
 1 5  x3   6 
  0  0.001 6  x   6.001

 0 2.5
 2   
5  x3   2.5 
 Partial Pivoting: Example
Forward Elimination: Step 2
Examining the values of the first column
|-0.001| and |2.5| or 0.0001 and 2.5
The largest absolute value is 2.5, so row 2 is
switched with row 3

Performing the row swap

10 7 0  x1   7  10 7 0  x1   7 
 0  0.001 6  x   6.001

 0 2.5
 2   
5  x3   2.5 
 0
 2.5 5   x    2.5 
 2   
 0  0.001 6  x3  6.001
 Partial Pivoting: Example
Forward Elimination: Step 2

Performing the Forward Elimination results in:

10  7 0   x1   7 
 0 2.5 5   x    2. 5 
  2   
 0 0 6.002  x3  6.002
 Partial Pivoting: Example
Back Substitution
Solving the equations through back substitution

10  7
6.002
0   x1   7  x3  1
 0 2 .5 5   x    2.5  6.002
  2   
 0 0 6.002  x3  6.002 2.5  5 x3
x2   1
2.5

7  7 x 2  0 x3
x1  0
10
 Partial Pivoting: Example
Compare the calculated and exact solution
The fact that they are equal is coincidence, but it
does illustrate the advantage of Partial Pivoting

 x1   0   x1   0 
 X  calculated   x2    1  X  exact   x 2    1
 x3   1   x3   1 
 THE END

http://numericalmethods.eng.usf.edu
Determinant of a Square Matrix
Using Naïve Gauss Elimination
Example

http://numericalmethods.eng.usf.edu
 Theorem of Determinants

If a multiple of one row of [A]nxn is added or

subtracted to another row of [A]nxn to result in
[B]nxn then det(A)=det(B)
 Theorem of Determinants
The determinant of an upper triangular matrix [A]nxn is given by

det  A   a11  a22  ...  aii  ...  ann

n
  aii
i 1
 Forward Elimination of a
Square Matrix
Using forward elimination to transform [A]nxn to an upper triangular matrix,
[U]nxn.

 A nn  U  nn

det  A  det U 
 Example
Using naïve Gaussian elimination find the
determinant of the following square
matrix.
 25 5 1
 64 8 1
 
144 12 1
Forward Elimination
 Forward Elimination: Step 1
 25 5 1
 64 8 1 Divide Equation 1 by 25 and
  64
144 12 1 multiply it by 64,  2.56 .
25
 25 5 1  2.56   64 12.8 2.56
.  64 8 1
Subtract the result from   64 12.8 2.56
Equation 2
0  4.8  1.56
 25 5 1 
Substitute new equation for  0  4.8  1.56
Equation 2  
144 12 1 
 Forward Elimination: Step 1 (cont.)
 25 5 1 
 0  4.8  1.56 Divide Equation 1 by 25 and
  multiply it by 144, 144  5.76 .
144 12 1  25
 25 5 1  5.76  144 28.8 5.76
.
144 12 1
Subtract the result from  144 28.8 5.76
Equation 3
 0  16.8  4.76

25 5 1 
Substitute new equation for  0  4.8  1.56 
Equation 3  
 0  16.8  4.76
 Forward Elimination: Step 2
Divide Equation 2 by −4.8
25 5 1 
 0  4.8  1.56  and multiply it by −16.8,
   16.8
 0  16.8  4.76  3. 5
 4. 8 .
0  4.8  1.56   3.5   0  16.8  5.46
.
0  16.8  4.76
Subtract the result from
Equation 3  0  16.8  5.46
0 0 0.7

Substitute new equation for

25 5 1 
 0  4.8  1.56
Equation 3  
 0 0 0.7 
 Finding the Determinant
After forward elimination

 25 5 1 25 5 1 
 64 8 1   0  4.8  1.56
   
144 12 1  0 0 0.7 
.

det  A   u11  u 22  u33

 25    4.8  0.7
 84.00
 Summary

-Forward Elimination
-Back Substitution
-Pitfalls
-Improvements
-Partial Pivoting
-Determinant of a Matrix
 Additional Resources
For all resources on this topic such as digital audiovisual
lectures, primers, textbook chapters, multiple-choice tests,
worksheets in MATLAB, MATHEMATICA, MathCad and
MAPLE, blogs, related physical problems, please visit

http://numericalmethods.eng.usf.edu/topics/gaussian_elimi
nation.html
 THE END

http://numericalmethods.eng.usf.edu