ME 310

Numerical Methods
3. 1. Solving Systems of Linear Algebraic Equations
Iterative Methods
Mechanical Engineering Department
Middle East Technical University
Ankara, Turkey
Ali Karakus (akarakus@metu.edu.tr)

These presentations were adapted from lecture notes of Dr. Cüneyt Sert, Dr. Sezer Özerinç and Altuğ
Özçelikkale.
They cannot be used and/or modified without the permission of the authors.
1


 Iterative Methods: Gauss-Siedel Method

• Gauss Elimination and its variants are called direct methods. They are not preferred for large systems
because round-off errors start to dominate.

• Iterative methods start with an initial guess vector and iteratively converge to the true solution.

• They stop when the pre-specified tolerance is reached, therefore round-off errors are not an issue.

• Given the system [A]{x} = {B} and initial guesses {x}0,

a11x1 + a12x2 + ... + a1nxn = b1

a21x1 + a22x2 + ... + a2nxn = b2
... ... ... ... ...
an1x1 + an2x2 + ... + annxn = bn
• Gauss-Seidel uses the first equation to solve for x1, second for x2, etc.

x1 = (b1 – a12x2 – a13x3 – . . . . . – a1nxn) / a11

x2 = (b2 – a21x1 – a23x3 – . . . . . – a2nxn) / a22
......
xn = (bn – an1x1 – an2x2 – . . . . . – a(n-1)(n-1)xn-1) / ann

 Iterative Methods: Gauss-Siedel Method

x1 = (b1 – a12x2 – a13x3 – . . . . . – a1nxn) / a11

x2 = (b2 – a21x1 – a23x3 – . . . . . – a2nxn) / a22

......
xn = (bn – an1x1 – an2x2 – . . . . . – a(n-1)(n-1)xn-1) / ann

• After the first iteration {x}1 is obtained. {x}1 is used to start a new iteration and iterations are repeated
until the tolerance is satisfied as:

xik − xik −1
ε a ,i = < εs
xik

for all the unknowns (i = 1, …, n), where k and k-1 indicate the present and previous iterations.
• Iterative methods are also advantageous for solving systems with sparse coefficient matrices, because
there is no need to store the zero-valued elements.

• Each double precision number requires 8 bytes. How much RAM do you need for a 100K x 100K matrix?

 Iterative Methods: Gauss-Siedel Method

Solve the following system using the Gauss-Seidel Method, starting with x10 = x20 = x30 = 0.0.
6x1 – 2x2 + x3 = 11
-2x1 + 7x2 + 2x3 = 5 Exact solution is {x}T = {2 1 1}T
x1 + 2x2 – 5x3 = -1
Rearrange the equations
x1 = (11 + 2x2 - x 3) / 6
x2 = ( 5 + 2x1 - 2x3) / 7
x3 = ( 1 + x1 + 2x2) / 5
First iteration
x11 = (11 + 2x20 - x 30) / 6 = (11 + 0 - 0)/6 = 1.833
x21 = ( 5 + 2x11 - 2x30) / 7 = (5 + 2*1.8333 - 0)/7 = 1.238
x 31 = ( 1 + x11 + 2x21) / 5 = (1 + 1.8333 + 2*1.2381)/5 = 1.062
Second iteration
x12 = (11 + 2x21 - x 31) / 6 = (11 + 2*1.238 – 1.062)/6 = 2.069
x22 = ( 5 + 2x12 - 2x31) / 7 = (5 + 2*2.069 – 2*1.062)/7 = 1.002
x 32 = ( 1 + x12 + 2x22) / 5 = (1 + 2.069 + 2*1.002)/5 = 1.015

and so on.

 Iterative Methods: Jacobi Method

• Gauss- Seidel always uses the newest available x values. Jacobi Method uses x values from the previous
iteration.
Repeat the previous example using the Jacobi method.

Rearrange the equations

x1 = (11 + 2x2 - x 3) / 6

x2 = ( 5 + 2x1 - 2x3) / 7

x3 = ( 1 + x1 + 2x2) / 5

First iteration (use x0 values)

x11 = (11 + 2x20 - x 30) / 6 = (11 + 0 - 0) / 6 = 1.833

x21 = ( 5 + 2x10 - 2x30) / 7 = ( 5 + 0 - 0) / 7 = 0.714

x 31 = ( 1 + x10 + 2x20) / 5 = ( 1 + 0 + 0) / 5 = 0.200

Second iteration (use x1 values)

x12 = (11 + 2x21 - x 31) / 6 = (11 + 2*0.714 – 0.200) / 6 = 2.038

x22 = ( 5 + 2x11 - 2x31) / 7 = (5 + 2*1.833 – 2*0.200) / 7 = 1.181

x 32 = ( 1 + x11 + 2x21) / 5 = (1 + 1.833 + 2*0.714) / 5 = 0.852

 Iterative Methods: Jacobi Method

Gauss-Seidel Method

Iter # 0 1 2 3 4 5

x1 0.000 1.833 2.069 1.998 1.999 2.000

x2 0.000 1.238 1.002 0.995 1.000 1.000

x3 0.000 1.062 1.015 0.998 1.000 1.000

Jacobi Method (always uses the xi values from the previous iteration)

Iter # 0 1 2 3 4 5 8
x1 0.000 1.833 2.038 2.085 2.004 1.994 2.000
x2 0.000 0.714 1.181 1.053 1.001 0.990 1.000
x3 0.000 0.200 0.852 1.080 1.038 1.001 1.000

• Usually Gauss-Seidel performs better than Jacobi method.

 Iterative Methods: Convergence
• These methods can be seen as a multi-variable application 11x1 + 13x2 = 286
of Simple One-Point Iteration Method. 11x1 − 9 x2 = 99

• Therefore they may diverge. A convergence criterion can be

derived based on that of the Simple One-Point Iteration
method (see Section 11.2.1 in Chapra and Canale 2010).

• If the magnitude of each diagonal element of the coefficient

matrix is larger than the sum of the magnitudes of the other
11x1 − 9 x2 = 99
elements in that row, convergence is guaranteed for any
11x1 + 13x2 = 286
initial set of guesses.
n
For all i = 1,...,n aii > ∑ aij
j =1
j ≠i
• Such systems are called diagonally dominant. This is a
sufficient but not necessary criterion for convergence.

• Many engineering problems satisfy this requirement.

 Gauss-Siedel with Relaxation

• Each new value of x is modified by a weighted average of the new and the previous value.

xinew = λ xinew + (1 − λ ) xiold 0 < λ < 2 : weighting factor

• If λ = 1 → no relaxation : original Gauss-Seidel

• If 0 < λ < 1 → under-relaxation : helps the convergence of a diverging system

• If 1 < λ < 2 → over-relaxation : speeds up the convergence of a converging system

(called successive over relaxation (SOR))

• Choice of λ is problem specific.

Exercise: The previous example has a diagonally dominant coefficient matrix. Interchange its 2nd and 3rd
equations. Is the system still diagonally dominant? Solve the system using both iterative methods we
learned. Compare their convergence rates with that of the solved example. Do you observe any
divergence?

Exercise: Solve the system we used in the Gauss-Seidel example with different weighting factors. Do you
notice any difference in the convergence rate?

 Gauss-Siedel with Relaxation

LOOP k from 1 to maxIter
LOOP i from 1 to n
xi=Bi
LOOP j from 1 to n
IF (i ≠ j) x i = x i - A ij x j
ENDLOOP Exercise: Improve this pseudocode with
x i = x i / A ii relaxation.
ENDLOOP
Exercise: Modify this pseudocode for the
CONVERGED = TRUE Jacobi method.
LOOP i from 1 to n
OUTPUT x i
εa = | (x i – x iold) / x i | * 100
IF (εa > tolerance) CONVERGED = FALSE
ENDLOOP
IF (CONVERGED = TRUE) STOP
ENDLOOP

Revisiting Newton-Raphson for System of Nonlinear Equations

Consider the following general form of a two-equation system. u ( x, y ) = 0

v ( x, y ) = 0
∂ ui ∂ ui
ui +1 = ui + ( xi +1 − xi ) + ( yi +1 − yi )
∂x ∂y
Write 1st order Taylor Series Expansion for these equations. ∂ vi ∂ vi
vi +1 = vi + ( xi +1 − xi ) + ( yi +1 − yi )
∂x ∂y

∂ ui ∂ ui ∂ ui ∂ ui
xi +1 + yi +1 = −ui + xi + yi
∂x ∂y ∂x ∂y
To find the solution set u i+1 = 0 and v i+1 = 0. Rearrange ∂ vi ∂ vi ∂ vi ∂ vi
xi +1 + yi +1 = − vi + xi + yi
∂x ∂y ∂x ∂y

Write in matrix form:

⎧ ∂ ui ∂ ui ⎫
⎡ ∂ ui ∂ ui ⎤ ⎪ − ui + xi + yi ⎪
⎢∂x x
⎧ i +1 ⎫ ∂ x ∂ y
∂ y ⎥⎪ ⎪ ⎪⎪ ⎪⎪ The 2 x 2 coefficient
⎢ ⎥⎨ ⎬=⎨ ⎬
⎢ ∂ vi ∂ vi ⎥ ⎪ ⎪ ⎪ ⎪ matrix is called the
⎢⎣ ∂ x ⎩ y ⎭ ∂ v ∂ v Jacobian matrix.
∂ y ⎥⎦ i +1
⎪ − vi + i
xi + i
yi ⎪
⎪⎩ ∂x ∂ y ⎪⎭

The method can be generalized for n simultaneous equations.

 Revisiting Newton-Raphson for System of Nonlinear Equations

• Given the following n nonlinear equations with n unknowns, form the Jacobian matrix:

f1 (x1, x2, ..., xn) = 0 k

⎡ ∂ f1 ∂ f1 ∂ f1 ⎤
⎢∂ x !
∂ x2 ∂ xn ⎥
⎢ 1 ⎥
f2 (x1, x2, ..., xn) = 0 ⎢ ∂ f2 ∂ f2 ∂ f2 ⎥
!
[ Z ] k = ⎢ ∂ x1 ∂ x2 ∂ xn ⎥
⎢ ⎥
..... ⎢ ! ! ! ! ⎥
⎢∂ f ∂ fn ∂ fn ⎥
⎢ n ! ⎥
fn (x1, x2, ..., xn) = 0 ⎢⎣ ∂ x1 ∂ x2 ∂ xn ⎥⎦

• Solve the following system using any of the techniques we learned in this chapter.

[Z]k {x}k+1 = -{F}k + [Z]k {x}k where Fik = fi ( x1k, x2k, …, xnk )

• k is the iteration counter. [Z]k is [Z] calculated at xi = xik . Therefore [Z] and {F} should be

recalculated for each iteration.

• [Z] might be difficult to calculate – finite-difference approximation solutions are preferred.
• Excellent initial guesses are required for convergence – approaching the problem from an
optimization point of view might provide better results.

 12