ME 6185

April 2022
Advanced Numerical Analysis

Lecture – 2
Solution of System of Equations

Sheikh Mohammad Shavik, PhD

Assistant Professor
Dept. of Mechanical Engineering, BUET
Office: Room no. ME421, ME Building
Email: shavik@me.buet.ac.bd

1 ME6185 Department of Mechanical Engineering, BUET

Difference between Systems of Linear and
Nonlinear Equations

Linear system Nonlinear system

2 ME6185 Department of Mechanical Engineering, BUET

System of Linear Equations
f1(x1, x2, ..., xn) = 0
f2(x1, x2, ..., xn) = 0
A general set of equations. n equations, n unknowns.
...
fn(x1, x2, ..., xn) = 0

Linear Algebraic Equation: An equation of the form f(x)=0 where, f

is a polynomial with linear terms.

a11x1 + a12x2 + ... + a1nxn = b1

a21x1 + a22x2 + ... + a2nxn = b2 A general set of linear algebraic equations.
... ... ... ... n equations, n unknowns.
an1x1 + an2x2 + ... + annxn = bn

Matrix Form: [A] {x} = {b}

[A] nxn Coefficient matrix
{x} nx1 Unknown vector
{b} nx1 Right-Hand-Side (RHS) vector
3 ME6185 Department of Mechanical Engineering, BUET
Numerical Solution of Systems of Linear
Algebraic Equations

4 ME6185 Department of Mechanical Engineering, BUET

(Naive) Gauss Elimination Method
§ Consider the following system of n equations.
a11x1 + a12x2 + ... + a1nxn = b1 (1)
a21x1 + a22x2 + ... + a2nxn = b2 (2)
………
………
an1x1 + an2x2 + ... + annxn = bn (n)

Step 0 (optional): Form the augmented matrix of [A|B]. Johann Carl Friedrich Gauss
(1777–1855)

Step 1 Forward Elimination: Reduce the system to an upper triangular system.

(1.1) First eliminate x1 from 2nd to nth equations.
- Multiply the 1st eqn. by a21/a11 & subtract it from the 2nd equation. This is the new 2nd eqn.
- Multiply the 1st eqn. by a31/a11 & subtract it from the 3rd equation. This is the new 3rd eqn.
...............
- Multiply the 1st eqn. by an1/a11 & subtract it from the nth equation. This is the new nth eqn.

Important: In these steps the 1st eqn is the pivot equation and a11 is the pivot element. Note that a
division by zero may occur if the pivot element is zero. Naive-Gauss Elimination does not check for
this.

5 ME6185 Department of Mechanical Engineering, BUET

Step 0: Form Augmented matrix
b1
b2

b3

bn b1
b2

bn

6 ME6185 Department of Mechanical Engineering, BUET

Step 1.1: eliminate x1 from 2nd to nth
equations

7 ME6185 Department of Mechanical Engineering, BUET

Step 1.1: eliminate x1 from 2nd to nth
equations

8 ME6185 Department of Mechanical Engineering, BUET

9 ME6185 Department of Mechanical Engineering, BUET
Step 1.1: eliminate x1 from 2nd to nth equations

10 ME6185 Department of Mechanical Engineering, BUET

Step 1.2: eliminate x2 from 3rd to nth equations

11 ME6185 Department of Mechanical Engineering, BUET

Step 1.3: eliminate x2 from 4th to nth equations

12 ME6185 Department of Mechanical Engineering, BUET

…….Step 1.n-1: eliminate xn-1 from nth equation

13 ME6185 Department of Mechanical Engineering, BUET

Forward Elimination

14 ME6185 Department of Mechanical Engineering, BUET

(Naive) Gauss Elimination Method
Step 2 Back substitution: Find the unknowns starting from
the last equation.

(2.1) Last equation involves only xn. Solve for it.

(2.1) Use this xn in the (n-1)th equation and solve for xn-1.
.......
……..
(2.n) Use all previously calculated x values in the 1st eqn and
solve for x1.

15 ME6185 Department of Mechanical Engineering, BUET

Back Substitution

16 ME6185 Department of Mechanical Engineering, BUET

Gauss Elimination: Recipe in a Flow Chart

17 ME6185 Department of Mechanical Engineering, BUET

Gauss Elimination: Recipe in a Flow Chart

18 ME6185 Department of Mechanical Engineering, BUET

Gauss Elimination: Recipe in a Flow Chart

19 ME6185 Department of Mechanical Engineering, BUET

MATLAB
Code

20 ME6185 Department of Mechanical Engineering, BUET

Using MATLAB code

21 ME6185 Department of Mechanical Engineering, BUET

Example: Naive Gauss Elimination Method

§ Example: Solve the following system using Naive Gauss

Elimination
6x1 – 2x2 + 2x3 + 4x4 = 16
12x1 – 8x2 + 6x3 + 10x4 = 26
3x1 – 13x2 + 9x3 + 3x4 = -19
-6x1 + 4x2 + x3 - 18x4 = -34

Step 0: Form the augmented matrix

6 –2 2 4 | 16
12 –8 6 10 | 26
3 –13 9 3 | -19
-6 4 1 -18 | -34

22 ME6185 Department of Mechanical Engineering, BUET

Example: Naive Gauss Elimination Method
6 –2 2 4 | 16
Augmented matrix: 12 –8 6 10 | 26
3 –13 9 3 | -19
-6 4 1 -18 | -34

Step 1: Forward elimination

6 –2 2 4 | 16 (Does not change. Pivot is 6)
(1.1) Eliminate x1 0 –4 2 2 | -6 row2 - row1*12/6
0 –12 8 1 | -27 row3 - row1*3/6
0 2 3 -14 | -18 row4 - row1*(-6)/6

(1.2) Eliminate x2 6 –2 2 4 | 16 (Does not change.)

0 –4 2 2 | -6 (Does not change. Pivot is -4)
0 0 2 -5 | -9 row3 – row2*(-12)/(-4)
0 0 4 -13 | -21 row4 – row2*2/(-4)

(1.3) Eliminate x3 6 –2 2 4 | 16 (Does not change.)

0 –4 2 2 | -6 (Does not change.)
0 0 2 -5 | -9 (Does not change. Pivot is 2)
0 0 0 -3 | -3 row4 – row3*4/2

23 ME6185 Department of Mechanical Engineering, BUET

Example: Naive Gauss Elimination Method

Step 2: Back substitution

(2.1) Find x4 x4 = (-3)/(-3) = 1

(2.2) Find x3 x3 = (-9+5*1)/2 = -2
(2.3) Find x2 x2 = (-6-2*(-2)-2*1)/(-4) = 1
(2.4) Find x1 x1 = (16+2*1-2*(-2)-4*1)/6 = 3

24 ME6185 Department of Mechanical Engineering, BUET

Pitfalls of Elimination Method
§ These information is relevant for not only for Gauss elimination method
but also for other elimination techniques
§ Division by Zero
§ The primary reason that the foregoing technique is called “naive” is
that during both the elimination and the back-substitution phases, it is
possible that a division by zero can occur
§ For example,

§ The normalization of the first row would involve division by a11 = 0

§ Problems also can arise when a coefficient is very close to zero
§ The technique of pivoting has been developed to partially avoid these
problems which will be discussed later
25 ME6185 Department of Mechanical Engineering, BUET
Pitfalls of Elimination Method
§ Round-Off Errors
§ Because computers can carry only a limited number of significant figures,
round-off errors can occur and must be considered when evaluating the
results
§ The problem of round-off error can become particularly important when
large numbers of equations are to be solved
§ This is due to the fact that every result is dependent on previous results
§ Consequently, an error in the early steps will tend to propagate—that is,
it will cause errors in subsequent steps
§ Specifying the system size where round-off error becomes significant is
complicated by the fact that the type of computer and the properties of
the equations are determining factors
§ A rough rule of thumb is that round-off error may be important when
dealing with 100 or more equations
§ In any event, we can always substitute the answers back into the original
equations to check whether a substantial error has occurred or, not
26 ME6185 Department of Mechanical Engineering, BUET
Techniques for Improving Solutions
§ Pivoting
§ In Naive Gauss Elimination, a division by zero occurs if the pivot element is
zero. Note that zero pivot elements may be created during the elimination
step even if they are not present in the original matrix.

§ Pivoting is used to avoid this problem. We interchange rows and columns

at each step to put the coefficient with the largest magnitude on the
diagonal.

§ In addition to avoiding the division by zero problem, pivoting reduces the

round-off errors. It makes the solution of ill-conditioned systems easier.

§ Complete pivoting uses both row and column interchanges. It is rarely

used because switching columns changes the order of the x’s and,
consequently, adds significant and usually unjustified complexity to the
computer program

§ Partial pivoting uses only row interchanges. This is used most of the times.
We will use this.
27 ME6185 Department of Mechanical Engineering, BUET
Partial Pivoting Example
Example: Solve the following system using Gauss Elimination with pivoting.
2x2 + + x4 = 0
2x1 + 2x2 + 3x3 + 2x4 = -2
4x1 – 3x2 + x4 = -7
6x1 + x2 - 6x3 - 5x4 = 6

Step 0: Form the augmented matrix

0 2 0 1 | 0
2 2 3 2 | -2
4 –3 0 1 | -7
6 1 -6 -5 | 6
Step 1: Forward Elimination
(1.1) Eliminate x1 : But the pivot element is 0. We have to interchange the 1st row
with one of the rows below it. Interchange it with the 4th row because 6 is the
largest possible pivot.
6 1 -6 -5 | 6 6 1 -6 -5 | 6
2 2 3 2 | -2 Now eliminate x1 0 1.6667 5 3.6667 | -4
4 –3 0 1 | -7 0 -3.6667 4 4.3333 | -11
0 2 0 1 | 0 0 2 0 1 | 0

28 ME6185 Department of Mechanical Engineering, BUET

Partial Pivoting Example (cont’d)
(1.2) Eliminate x2 : from the 3rd and 4th eqns. Pivot element is 1.6667.
There is no division by zero problem. Still, we will perform pivoting to
reduce round-off errors. Interchange the 2nd and 3rd rows.
6 1 -6 -5 | 6 6 1 -6 -5 | 6
0 -3.6667 4 4.3333 | -11 Eliminate x2 0 -3.6667 4 4.3333 | -11
0 1.6667 5 3.6667 | -4 0 0 6.8182 5.6364 | -9.0001
0 2 0 1 | 0 0 0 2.1818 3.3636 | -5.9999

(1.3) Eliminate x3 . abs(6.8182) > abs(2.1818), therefore no pivoting is necessary.

6 1 -6 -5 | 6
0 -3.6667 4 4.3333 | -11
0 0 6.8182 5.6364 | -9.0001
0 0 0 1.5600 | -3.1199

Step 2: Back substitution

x4 = -3.1199 / 1.5600 = -1.9999
x3 = [-9.0001 – 5.6364*(-1.9999)] / 6.8182 = 0.33325
x2 = [-11 – 4.3333*(-1.9999) – 4*0.33325] / -3.6667 = 1.0000
x1 = [6 – (-5)*(-1.9999) – (-6)*0.33325 – 1*1.0000] / 6 = -0.50000
Exact solution is x = [-2 1/3 1 -0.5] T. Use more than 5 significant
figures to reduce round-off errors.
29 ME6185 Department of Mechanical Engineering, BUET
Other Uses of Gauss Elimination
§ Gives the LU decomposition of A such that [L][U]=[A]. LU decomposition
is useful if we are solving many systems with the same coefficient matrix
A but different right-hand-side vectors
§ It can be used to calculate the determinant of a matrix. At the end of the Forward
Elimination step we get an upper triangular matrix. For this matrix, the determinant is
just the multiplication of diagonal elements. If we interchanged rows m times, then the
deteminant can be calculated as
|A| = (-1)m * a11 * a22 * … ann
§ Recall the example we used to describe pivoting. The A matrix was
0 2 0 1
2 2 3 2
4 –3 0 1
6 1 -6 -5
§ After the Forward Elimination step we found the following upper triangular matrix.
6 1 -6 -5
0 -3.6667 4 4.3333
0 0 6.8182 5.6364
0 0 0 1.5600
To get this, we used pivoting and interchanged rows twice. Therefore, the determinant
of A is:
|A| = (-1)2 * 6 * (-3.6667) * 6.8182 * 1.56 = -234.0028
30 ME6185 Department of Mechanical Engineering, BUET
Gauss-Jordan Method
§ This is another elimination technique. It is a variation of Gauss Elimination.
§ The difference is, when an unknown is eliminated, it is eliminated from all other
equations, not just the subsequent ones. At the same time all rows are normalized by
dividing them to their pivot element.
§ At the end of the forward elimination step Gauss-Jordan method yields an identity
matrix, not an upper triangular one.
§ There is no need for back substitution. Right-hand-side vector has the results.

§ Example : Solve the following 4x4 system using Gauss-Jordan method with
pivoting. Note that this is the same system that we used to demonstrate Gauss
Elimination with pivoting. 0 2 0 1 | 0
2 2 3 2 | -2
4 –3 0 1 | -7
6 1 -6 -5 | 6
• Interchange rows 1 and 4, divide the new first row by 6 and eliminate x1 from the 2nd,
3rd and 4th equations.
1 0.1667 -1 -0.8333 | 1
0 1.6667 5 3.6667 | -4
0 -3.6667 4 4.3333 | -11
0 2 0 1 | 0

31 ME6185 Department of Mechanical Engineering, BUET

Gauss-Jordan Method (cont’d)
• Interchange rows 2 and 3, divide the new second row by -3.6667 and
eliminate x2 from the 1st, 3rd and 4th equations.
1 0 -0.8182 -0.6364 | 0.5
0 1 -1.0909 -1.1818 | 3
0 0 6.8182 5.6364 | -9
0 0 2.1818 3.3636 | -6
• No pivoting is required. Divide the third row by 6.8182 and eliminate x3 from the 1st,
2nd and 4th equations.
1 0 0 0.04 | 0.58
0 1 0 -0.280 | 1.56
0 0 1 0 | -1.32
0 0 0 1.5599 | -3.12
• Divide the last row by 1.5599 and eliminate x4 from the 1st, 2nd and 3rd equations.
1 0 0 0 | -0.5
0 1 0 0 | 1.0001
0 0 1 0 | 0.3333
0 0 0 1 | -2
• Right-hand-side vector is the solution. No back substitution is required.
• Gauss-Jordan requires approximately 50% more operations than Gauss Elimination

32 ME6185 Department of Mechanical Engineering, BUET

Calculating the Inverse of a Matrix with
Gauss-Jordan
§ Calculating the inverse of a matrix is important if we want to solve many systems
with the same
§ coefficient matrix, but different right-hand-side vectors.
§ To take the inverse of A, augment it with an identity matrix and apply the Gauss-
Jordan method.
1 -1 2
§ Example: Find the inverse of the following matrix. A = 3 0 1
• Augment A with a 3x3 identity matrix 1 0 2
1 -1 2 | 1 0 0
3 0 1 | 0 1 0
1 0 2 | 0 0 1
• Apply the Gauss-Jordan method to this system to get
1 0 0 | 0 0.4 –0.2
0 1 0 | -1 0 1
0 0 1 | 0 –0.2 0.6
•3x3 matrix at the right is the inverse of A. This inverse can now be used to solve
a system with the coefficient matrix A and different right-hand-side vectors as {x}
= [A]-1 {B}
• LU decomposition can be used for the same purpose and it is more efficient
33 ME6185 Department of Mechanical Engineering, BUET
LU Decomposition
§ LU decomposition techniques: a class of elimination methods
§ The primary appeal of LU decomposition is that the time-consuming
elimination step can be formulated so that it involves only
operations on the matrix of coefficients, [A]
§ Thus, it is well suited for those situations where many right-hand-
side vectors {B} must be evaluated for a single value of [A]
§ Although there are a variety of ways in which this is done, we will
focus on showing how the Gauss elimination method can be
implemented as an LU decomposition
§ LU decomposition provides an efficient means to compute the
matrix inverse

34 ME6185 Department of Mechanical Engineering, BUET

LU Decomposition
§ General system of linear eqn.
§ Suppose that above Eqn. could be expressed as an upper triangular system
(consider, set of three simultaneous equations):

§ Recognize that this is similar to the manipulation that occurs in the first step of
Gauss elimination (Forward elimination)

§ Assume that there is a lower diagonal matrix , for which,

§ If this equation holds, it follows from the rules for matrix multiplication that,

and

35 ME6185 Department of Mechanical Engineering, BUET

LU Decomposition
A two-step strategy for obtaining solutions :
1. LU decomposition step:
[A] is factored or “decomposed” into
lower [L] and upper [U] triangular
matrices
2. Substitution step:
[L] and [U] are used to determine a
solution {X} for a right-hand side {B}
This step itself consists of two steps:
First, 𝐿 𝐷 = {𝐵} is used to generate an Figure: The steps in LU decomposition.
intermediate vector {D} by forward
substitution
Then, the result is substituted into

which can be solved by back substitution for {X}.

36 ME6185 Department of Mechanical Engineering, BUET

Finding the [L] and [U] matrix
3×3 system
Note, we are not using pivoting for
the examples. In reality, pivoting is
always used in LU decomposition
method similar to Gauss elimination

Factor, eliminate a21

Factor, eliminate a31

Factor, eliminate a31

37 ME6185 Department of Mechanical Engineering, BUET

Forward-substitution and Back-substitution

Forward substitution:

Back substitution:

38 ME6185 Department of Mechanical Engineering, BUET

Example: LU decomposition

Forward elimination:

Forward substitution:

Back substitution:

39 ME6185 Department of Mechanical Engineering, BUET

Crout Decomposition

§ For the LU decomposition implementation of Gauss

elimination, the [L] matrix has 1’s on the diagonal. This is
formally referred to as a Doolittle decomposition, or
factorization.
§ An alternative approach involves a [U] matrix with 1’s on
the diagonal. This is called Crout decomposition.
§ Although there are some differences between the
approaches (Atkinson, 1978; Ralston and Rabinowitz,
1978), their performance is comparable
40 ME6185 Department of Mechanical Engineering, BUET
Matrix Inversion using LU Decomposition method
§ The inverse can be computed in a column-by-column fashion by generating
solutions with unit vectors as the right-hand-side constants.
§ For example, if the right-hand-side constant has a 1 in the first position and
zeros elsewhere,

the resulting solution will be the first

column of the matrix inverse

Example:

41 ME6185 Department of Mechanical Engineering, BUET

Matrix Inverse Method

42 ME6185 Department of Mechanical Engineering, BUET

Matrix Inverse & Linear Solver in MATLAB

43 ME6185 Department of Mechanical Engineering, BUET