MA2213: Numerical Analysis I

Lecture 5: Gaussian elimination

Zhenning Cai

September 9, 2019

Cramer’s rule provides a nice and concise presentation for the solution to general linear
systems of equations. However, a more practical way to compute the solution is Gaussian
elimination. We first review how this technique is used in solving linear equations.

Example 1. Consider the linear system of equations

E1 : x1 + x2 + 3x4 = 4,
E2 : 2x1 + x2 − x3 + x4 = 1,
(1)
E3 : 3x1 − x2 − x3 + 2x4 = −3,
E4 : −x1 + 2x2 + 3x3 − x4 = 4,

where E1 , E2 , E3 , E4 are simply labels of the equations. To solve this system, we first eliminate
the unknown x1 by the following operations:

E2 ← E2 − 2 × E1 , E3 ← E3 − 3 × E1 , E4 ← E4 + E1 . (2)

The notation means to apply the operations on the right-hand side of “←” to both sides of
the equations, and then assign the resulting equation to the symbol on the left-hand side of
“←”. The result of (2) is

E2 : −x2 − x3 − 5x4 = −7,

E3 : −4x2 − x3 − 7x4 = −15, (3)
E4 : 3x2 + 3x3 + 2x4 = 8.

Here E1 is not written since it is unchanged. The second step is to eliminate x2 :

E3 ← E3 − 4 × E2 , E4 ← E4 + 3 × E2 . (4)

The result is
E3 : 3x3 + 13x4 = 13,
(5)
E4 : − 13x4 = −13.
For this particular example, there is no need to eliminate x3 by another step. We can start to
find the solution by backward substitution. By E4 , it is clear that x4 = 1. Inserting this result
to E3 , we get x3 = 0. By substituting these two values into (3), one finds that x2 = 2. Then
the final step is to find the value of x1 by E1 , and the result is x1 = −1.

In the process of Gaussian elimination, it is actually unnecessary to write down the com-
plete equations like (3) and (5). It is sufficient to just write down all the coefficients, and this
is can be well organized if the augmented matrix is utilized.

1
Example 2. The augmented matrix for the linear system (1) is
 
1 1 0 3 4
 2 1 −1 1 1
 .
 3 −1 −1 2 −3
−1 2 3 −1 4

Based on such a representation, we can still perform the operations (2). For example, “E2 ←
E2 − 2 × E1 ” means “to multiply the first row of the augmented matrix by −2, and then add
the result to the second row”. After performing (2), the matrix becomes
 
1 1 0 3 4
0 −1 −1 −5 −7
 
0 −4 −1 −7 −15 .
0 3 3 2 8

Note that in the above result, the lower-triangular part of the first column is zero, which means
that in the last three equations, x1 has been eliminated. Similarly, the aim of the second step
(4) is to make the lower-triangular part of the second column zero. The result is
 
1 1 0 3 4
0 −1 −1 −5 −7
 . (6)
0 0 3 13 13
0 0 0 −13 −13

In general, a third elimination step, which multiplies the third row by a number and adds the
result to the fourth row, is required to make the matrix upper-triangular. In this case, the
last entry in the third row happens to be zero. By now, we have obtained a triangular linear
system, which can be solved by backward substitution. The process is the same as in Example
1.

Based on this method, we are going to study the algorithm in this lecture.

1 Algorithm for Gaussian elimination

It is straightforward to write down the pseudocode for Gaussian elimination, which is given
in Algorithm 1. The first eight lines are elimination steps, and line 9–16 are backward substi-
tution. We are going to count the number of operations in this algorithm:

• Divisions occur in line 3, 9 and 15. Line 3 is executed (n−1)+(n−2)+· · ·+1 = n(n−1)/2
times. Therefore the total number of divisions is n(n − 1)/2 + n = n(n + 1)/2.
• Multiplications occur in line 5 and 13. Line 5 is executed n(n − 1) + (n − 1)(n − 2) +
· · · + 2 × 1 = (n − 1)n(n + 1)/3 times, and line 13 is executed n(n − 1)/2 times. Therefore
the total number of multiplications is n(n − 1)(2n + 5)/6.
• Subtractions occur in line 5 and 13. The number of subtractions equals the number of
multiplications, which is n(n − 1)(2n + 5)/6.

By the above calculation, the time complexity of Gaussian elimination is O(n3 ), and when
n is large, the total number of operations is dominated by 2n3 /3. By comparison with the
method using Cramer’s rule, this method has less computational time.
Below we are going to test the accuracy of Gaussian elimination by the equations we solved
for Cramer’s rule.

2
Algorithm 1 Solve the linear system with augmented matrix à = (aij )n×(n+1)
1: for j = 1, · · · , n − 1 do
2: for i = j + 1, · · · , n do
3: mij ← aij /ajj
4: for k = j + 1, · · · , n + 1 do
5: aik ← aik − mij ajk
6: end for
7: end for
8: end for
9: xn ← an,n+1 /ann
10: for i = n − 1, · · · , 1 do
11: xi ← ai,n+1
12: for j = i + 1, · · · , n do
13: xi ← xi − aij xj
14: end for
15: xi ← xi /aii
16: end for
17: return (x1 , · · · , xn )T

Example 3. Consider the equations:

0.373x + 0.296y = 0.521,

0.326x + 0.260y = 0.456,

whose exact solution of this system is x = 1 and y = 0.5. For the 2 × 2 system, we can code
the Gaussian elimination method without using loops:
# include < stdio .h >

int main ()
{
float A [2][2] = {{0.373 , 0.296} , {0.326 , 0.260}} , b [2] = {0.521 , 0.456};
float factor = A [1][0] / A [0][0];
float x [2];
x [1] = b [1] - b [0] * factor ;
x [1] /= ( A [1][1] - A [0][1] * factor ) ;
x [0] = b [0] - x [1] * A [0][1];
x [0] /= A [0][0];

printf ( " Solution : (%.8 f , %.8 f ) \ nResidual : (% e , % e ) \ n " , x [0] , x [1] ,

A [0][0]* x [0]+ A [0][1]* x [1] - b [0] , A [1][0]* x [0]+ A [1][1]* x [1] - b [1]) ;
return 0;
}

The output is
Solution : (1.00000918 , 0.49998853)
Residual : (0.000000 e +00 , 0.000000 e +00)

This result shows that Gaussian elimination gives exact solution to the finite-digit version of
the original equation.

3
2 Dealing with zero pivot elements
However, Gaussian elimination does not always work well as “division by zero” may occur in
the third line of Algorithm 1. In this case, we need to swap equations to keep the process
moving. A simple example is given below.

Example 4. Consider the following linear system:

E1 : x1 − x2 + 2x3 − x4 = −8,
E2 : 2x1 − 2x2 + 3x3 − 3x4 = −20,
(7)
E3 : x1 + x2 + x3 = −2,
E4 : x1 − 2x2 + 4x3 + 3x4 = 2.

We first eliminate x1 by performing the following operations:

E2 ← E2 − 2 × E1 , E3 ← E3 − E1 , E4 ← E4 − E1 , (8)

and then the last three equations become

E2 : − x3 − x4 = −4,
E3 : 2x2 − x3 + x4 = 6,
E4 : −x2 + 2x3 + 4x4 = 10.

But now we cannot use E2 to eliminate the variable x2 since the coefficient of x2 in E2 happens
to be zero. But we can use either E3 or E4 to do the job. In order to make the process more
systematic, we introduce the “swapping” operation: E2 ↔ E3 , whose result is

E2 : 2x2 − x3 + x4 = 6,
E3 : − x3 − x4 = −4,
E4 : −x2 + 2x3 + 4x4 = 10.

The Gaussian elimination process can resume after the swapping operation.
The above process can also be represented by operations on the augmented matrix:
   
1 −1 2 −1 −8 1 −1 2 −1 8
2 −1 3 −3 −20  0 −1 −1 −4
  −→ 0  (Eliminate x1 )
1 1 1 0 −2   0 2 −1 1 6
1 −2 4 3 2 0 −1 3 4 10
 
1 −1 2 −1 8
0 2 −1 1 6
−→ 0
 (Swap the 2nd and 3rd rows)
0 −1 −1 −4
0 −1 3 4 10
−→ · · ·

The remaining steps are left as an exercise.

When eliminating the variable xi , the ith diagonal entry of the augmented matrix is called
the pivot element. As we see in the previous example, the pivot element can be zero. In this
case, we need to swap rows to resume the elimination process. If there are no possible rows
to swap, it means that the matrix is singular.
When implementing this method, exchanging values of two complete rows can be expensive.
This can be avoided by introducing an additional array ri . Its function can be demonstrated
by the following example.

4
Example 5. Consider again the linear system (7). In the beginning, we set ri = i for
i = 1, 2, 3, 4. Thus there is no difference between ri and i, and we can rewrite (8) as

Er2 ← Er2 − 2 × Er1 , Er3 ← Er3 − Er1 , Er4 ← Er4 − Er1 .

After this step, we meet a zero pivot element. Now, instead of swapping E2 and E3 , we swap
the values of r2 and r3 , after which we have

r1 = 1, r2 = 3, r3 = 2, r4 = 4.

We now see that we can use Er2 to eliminate x2 :

1
Er3 ← Er3 − 0 × Er2 , Er4 ← Er4 + × Er2 .
2
Swapping two integers is much cheaper than swapping the rows of a matrix.

Now we can write down the pseudocode of the above algorithm:

Algorithm 2 Solve the linear system with augmented matrix à = (aij )n×(n+1)
1: for i = 1, · · · , n do
2: ri ← i
3: end for
4: for j = 1, · · · , n − 1 do
5: if arj j = 0 then
6: k ←j+1
7: while k ⩽ n and ark j = 0 do
8: k ←k+1
9: end while
10: if k = n + 1 then
11: return “Error: matrix is singular”
12: else
13: Swap rj and rk
14: end if
15: end if
16: for i = j + 1, · · · , n do
17: mij ← ari j /arj j
18: for k = j + 1, · · · , n + 1 do
19: ari k ← ari k − mij arj k
20: end for
21: end for
22: end for
23: xn ← arn ,n+1 /arn n
24: for i = n − 1, · · · , 1 do
25: xi ← ari ,n+1
26: for j = i + 1, · · · , n do
27: xi ← xi − ari j xj
28: end for
29: xi ← xi /ari i
30: end for
31: return (x1 , · · · , xn )T