Part 4 - Linear Systems - Iterative Methods

GNE 333 - Engineering Analysis 1
Part 4 – Linear Systems
Iterative Methods
Dr. Yves Mansour

yves.mansour@lau.edu.lb
03/27/2024 GNE 333 - Engineering Analysis 1 – Part 4 – Linear Systems - Direct Methods 1
Objectives
 Understanding the meaning of matrix and vector norms and how they are
computed.
 Knowing how to use norms to compute the matrix condition number.
 Understanding how the magnitude of the condition number can be used to
estimate the precision of solutions of linear algebraic equations.
 Understanding the difference between the Gauss-Seidel and Jacobi methods.
 Knowing how to assess diagonal dominance and knowing what it means.
 Recognizing how relaxation can be used to improve the convergence of
iterative methods.
 Understanding how to solve systems of nonlinear equations with successive
substitution and Newton-Raphson.
Vector and Matrix Norms
For vector in three-dimensional Euclidean space:
The Euclidean norm of [F] is:
Similarly, for an n-dimensional vector
The concept can be extended further to a matrix [A],

Frobenius norm is defined by:
As with the other vector norms, it provides a single value to quantify the “size” of [A].
Vector and Matrix Norms (cont.)
There are alternatives to Euclidian called p-norms that can be represented generally by:
(Euclidean norm is identical to the 2-norm, )
Other important examples are: - For p = 1:

- For (p = ∞):
Using a similar approach, norms can be developed for matrices:
- Column-sum norm:
- Row-sum norm:
- Spectral norm: , where is the largest eigenvalue of .
(Ortega, 1972)
Matrix Condition Number
Matrix condition number is defined by: 1
For a linear system , it can be shown (Ralston & Rabinowitz, 1978; Gerald & Wheatley,
1989) that:
So, the relative error of the computed solution can be as large as the relative error of the
coefficients of multiplied by the condition number.
For example, if the coefficients of are known to -digit precision (i.e., error ) and if ,
solution will be valid to only digits (error ).
The extent of the ill-conditioning can be quantified by calculating
Example 11.3
Example 11.3
=1.833x408=747.86
Norms and Condition Number in MATLAB
MATLAB has built-in functions to compute both norms and condition numbers:
>> norm (X,p)

and
>> cond (X,p)
where is the vector or matrix and designates the type of norm or condition number
(, , , or ).
Note that the “cond” function is equivalent for a matrix to:
>> norm (X,p) * norm (inv(X),p)
Also, note that if is omitted, it is automatically set to 2.
Example 11.4
Example 11.4
=2.231x164.976=368
Jacobi Iterative Method
It is the simplest iterative method to solve a linear system. Consider a 33 system:
{
𝑎11 𝑥1 + 𝑎12 𝑥 2 +𝑎 13 𝑥 3=𝑏 1
𝑎 2 1 𝑥 1+ 𝑎 2 2 𝑥 2+ 𝑎 23 𝑥 3=𝑏 2
𝑎 3 1 𝑥 1+ 𝑎 3 2 𝑥 2+ 𝑎 3 3 𝑥 3= 𝑏3
We need an initial solution guess , we can start with
Solution at iteration is obtained, as in the fixed-point method, from:

If an , rows are permuted as in Gauss
elimination method (partial pivoting) so that
all .
Iterations are repeated until convergence.
Jacobi Iterative Method (cont.)
In general for a system , solution at iteration is obtained from:
𝑖− 1 𝑛
𝑏𝑖 − ∑ 𝑎 𝑖𝑗 𝑥( 𝑗𝑘 −1 ) − ∑ 𝑎 𝑖𝑗 𝑥( 𝑗𝑘 −1 )
𝑥𝑖( 𝑘 ) = 𝑗 =1 𝑗 =𝑖 +1
𝑎𝑖𝑖
The linear system residual at iteration is:
Iterations are repeated until:
or
For the vectors norm, we can choose = (least restrictive), , or (most restrictive)
Example 12.1
Example 12.1
Gauss-Seidel Iterative Method
In the expression of in Jacobi method, are known at iteration for = to .
𝑖− 1 𝑛
𝑏𝑖 − ∑ 𝑎 𝑖𝑗 𝑥 ( 𝑘 −1 )
𝑗 − ∑ 𝑎 𝑖𝑗 𝑥( 𝑗𝑘 −1 )
𝑥𝑖( 𝑘 ) = 𝑗 =1 𝑗 =𝑖 +1
(Jacobi)
𝑎𝑖𝑖
In Gauss-Seidel method, (for = to ) are used at iteration in the expression of , so:
𝑖− 1 𝑛
𝑏𝑖 − ∑ 𝑎 𝑖𝑗 𝑥( 𝑗𝑘) − ∑ 𝑎 𝑖𝑗 𝑥( 𝑗𝑘 −1)
𝑥𝑖( 𝑘 ) = 𝑗 =1 𝑗 =𝑖 +1
(Gauss-Seidel)
𝑎 𝑖𝑖
In other terms, in Gauss-Seidel method, we used the most recent value of . Except for
this difference, Gauss-Seidel method is identical to Jacobi’s.
Example 12.1
Check “Jacobi.m” and “GaussSeidel.m”

𝑖− 1 𝑛
𝑏𝑖 − ∑ 𝑎 𝑖𝑗 𝑥 ( 𝑘)
𝑗 − ∑ 𝑎 𝑖𝑗 𝑥( 𝑗𝑘 − 1)
Example 12.1
𝑥
(𝑘 )
𝑖 =
𝑗 =1
𝑎 𝑖𝑖
𝑗 =𝑖 +1
Convergence and Diagonal Dominance
It can be shown that a sufficient (but not necessary) condition for the convergence of
Gauss-Seidel and Jacobi methods is:
( is said diagonally dominant)
When diagonal dominance increases, Gauss-Seidel and Jacobi methods converge faster.
Diagonal dominance can be increased by rows permutation.
Gauss-Seidel method usually converges faster than Jacobi method, but both methods have
linear convergence, i.e., the required number of iteration is proportional to the system size.
Other methods converge much faster, such as conjugate gradient methods (CG, CGS,
BiCG, BiCGStab), quasi-minimal residual method (QMR), generalized minimal residual
method (GMRES), …
Relaxation
Relaxation represents a slight modification of the Gauss-Seidel method that is designed
to enhance convergence. At each iteration:
where λ is a weighting factor that is assigned a value between 0 and 2.

• For 0 < λ < 1, the modification is called underrelaxation, it is employed to converge
a non-convergent system or to improve the convergence by dampening out
• oscillations.
For λ > 1, the approach is called overrelaxation, it can accelerate the converge of a
slowly convergent system, when the successive values of are moving in the correct
direction toward the true solution but at too slow a rate.
This approach (λ > 1) is called successive overrelaxation, or SOR method.
The choice of λ is highly problem-specific and is often determined empirically. For a
single solution of a set of equations it is often unnecessary to know the optimal value of λ.
Example 12.2
Check “SOR.m”
Example 12.2
λ 1.2
ε 0.1
Iteration
X1 X2 ε (X1) ε (X2)
0 0 0
1 -3.5964 0.096864 100% 100%
2 -2.41264 0.361594 49% 73%
3 -1.37996 0.556492 75% 35%
4 -0.65192 0.692241 112% 20%
5 -0.14658 0.786372 345% 12%
6 0.203728 0.85162 172% 8%
7 0.446544 0.896846 54% 5%
8 0.614849 0.928195 27% 3%
9 0.731509 0.949923 16% 2%
10 0.81237 0.964984 10% 2%
11 0.868418 0.975423 6% 1%
12 0.907267 0.982659 4% 1%
13 0.934195 0.987675 3% 1%
14 0.95286 0.991151 2% 0%
15 0.965797 0.993561 1% 0%
16 0.974764 0.995231 1% 0%
17 0.98098 0.996389 1% 0%
Non-Linear Equations Systems
Consider 2 nonlinear equations
with 2 unknowns:
In general, we can have non-

linear equation of unknowns:
Non-Linear Systems - Successive Substitution
A simple approach is to use the same strategy that was employed for fixed-point iteration
and the Gauss-Seidel method.
(as in the fixed-point

method)
As in Gauss-Seidel method, we start with an initial guess
Solution and residual at iteration are obtained from:
[ ]
𝑓 1 ( 𝑥1 , 𝑥 2 ,… , 𝑥 𝑛 )
(𝑘) (𝑘 ) (𝑘)
Stopping criterion:
𝑟
(𝑘)
= 𝑓 2 ( 𝑥(𝑘)
1 , 𝑥2 , … , 𝑥𝑛 )
(𝑘) (𝑘)
⋮
or
𝑓 𝑛(𝑥 )
(𝑘) (𝑘) (𝑘)
1 ,𝑥 2 ,…, 𝑥 𝑛
Example 12.3
Example 12.3
Non-Linear Systems - Newton-Raphson Method
The previous example illustrates the most serious shortcoming of successive substitution:
Convergence often depends on the manner in which the equations are formulated.
The problem is solved by Newton-Raphson approach. Recall that for single non-linear
equation: 𝑓 (𝑥 )
(𝑘 )
𝑓 ( 𝑥 ) =0 ⇒ 𝑥
( 𝑘+ 1) (𝑘 )
=𝑥 − = 𝐹 ( 𝑥
( 𝑘)
)
𝑓 ′ (𝑥 )
( 𝑘)
For a system of non-linear equations, we are dealing with multivariable functions, so
[ ]
we define the Jacobian matrix J :
𝜕𝑓1 𝜕𝑓1 𝜕𝑓1
…
𝜕𝑥 1 𝜕𝑥 2 𝜕𝑥 𝑛
𝜕𝑓𝑛 𝜕𝑓𝑛
𝐽= 𝜕 𝑓 2 𝜕𝑓2 𝜕𝑓 2 ¿ …¿ ¿
… 𝜕𝑥 2 𝜕𝑥 𝑛
𝜕𝑥 1 𝜕𝑥 2 𝜕𝑥 𝑛
⋮ ⋮ ¿ ¿
Non-Linear Systems - Newton-Raphson Method
By analogy to a single variable function, for a system of non-linear equations, Newton-
Raphson method is written as:
is the inverse of Jacobian matrix evaluated from the solution at iteration .
[ ]
𝑓 ( 𝑥 ,𝑥 ,…,𝑥 ) 𝑓 ( 𝑥 ,𝑥 ,…,𝑥 ) … 𝑓 ( 𝑥 ,𝑥 ,…,𝑥 )
′ (𝑘) (𝑘) (𝑘) ′ (𝑘) (𝑘) (𝑘) ′ (𝑘) (𝑘) (𝑘)
1,𝑥1 1 2 𝑛 1,𝑥 2 1 2 𝑛 1,𝑥 𝑛 1 2 𝑛
[ 𝐽(𝑘) ]= 𝑓 ′2,𝑥 ( 𝑥(𝑘)1 ,𝑥(𝑘)2 ,…,𝑥(𝑘)𝑛 ) 𝑓 2,𝑥 ( 𝑥1 ,𝑥 2 ,…,𝑥 𝑛 )
′ (𝑘) (𝑘) (𝑘)
… 𝑓 2,𝑥 ( 𝑥1 ,𝑥 2 ,…,𝑥 𝑛 ) 𝑛,𝑥 ( 1 2
′ (𝑘) (𝑘)
2
𝑛 ) ¿ ¿ 𝑛,𝑥 ( 1 2
𝑛
𝑛 )¿
(𝑘) 𝑓 ′ 𝑥(𝑘) ,𝑥 (𝑘) ,…,𝑥(𝑘) … 𝑓 ′ 𝑥 (𝑘) ,𝑥(𝑘) ,…,𝑥(𝑘)
1 2 𝑛
⋮ ⋮ ¿ ¿
Example 12.4
Example 12.4
Example 12.4

Part 4 - Linear Systems - Iterative Methods

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Part 4 - Linear Systems - Iterative Methods

Uploaded by

Copyright:

Available Formats

GNE 333 - Engineering Analysis 1

Part 4 – Linear Systems

Dr. Yves Mansour

Similarly, for an n-dimensional vector

The concept can be extended further to a matrix [A],

Other important examples are: - For p = 1:

>> norm (X,p)

Solution at iteration is obtained, as in the fixed-point method, from:

Iterations are repeated until convergence.

Iterations are repeated until:

Check “Jacobi.m” and “GaussSeidel.m”

where λ is a weighting factor that is assigned a value between 0 and 2.

In general, we can have non-

(as in the fixed-point

As in Gauss-Seidel method, we start with an initial guess

Solution and residual at iteration are obtained from:

For a system of non-linear equations, we are dealing with multivariable functions, so

is the inverse of Jacobian matrix evaluated from the solution at iteration .

You might also like