Chapter 1: Iterative Methods: Abdelkrim E Professeur of Applied Mathematics - AI

Chapter 1: Iterative Methods
Abdelkrim E L MOUATASIM
AI
Professeur of Applied Mathematics -
https://sites.google.com/a/uiz.ac.ma/elmouatasim/
FPO - MME - S2
March 8, 2022
Plan
Plan
1 Iterative Methods I
Iterative Methods
Jacobi's Method
Convergence
2 Summary
Summary
3 Iterative Methods II
Iterative Methods
Gauss-Seidel Method
Convergence Analysis
A. El MOUATASIM Computing Science

Iterative Methods I Iterative Methods
Summary Jacobi's Method
Iterative Methods II Convergence
Outline
Iterative Methods
Jacobi's Method
Convergence
2 Summary
Summary
Iterative Methods
Gauss-Seidel Method

Iterative Methods
We (still) want to solve the linear system
Ax b.
All direct methods require O pn 3 q operations. For large matrices this
takes time and each operation introduces error that may accumulate.
Other methods may be preferable.
Iterative methods build a sequence of approximate solutions. Start from

guess x p0q and compute approximations x p1q , x p2q , . . . , x pN q which
converge to the exact solution.
Sequence can be truncated to save time or when solution found to

sucient accuracy.
Key elements of an iterative method:
1 A rapid convergence to the correct solution,
2 A fast algorithm for computing each successive approximation.

Iterative Methods
Ax b.


sucient accuracy.

Iterative Methods
Ax b.


sucient accuracy.

A general framework
Standard notation is to split the matrix as
P,AN
where N and P are (as yet unknown) n n matrices. The linear system
becomes
Nx Px b.
Then assume that we have an approximate solution, or guess, x p0q , and

dene the sequence
N x pi q P x pi q1
b, i 1, 2, . . .
Obviously consistent with the original equation. The algorithm will only
be useful if
1 N is nonsingular, and
2 The linear system Ny z is easy to solve.
A general framework
P,AN
becomes
Nx Px b.

dene the sequence
N x pi q P x pi q1
b, i 1, 2, . . .
be useful if
A general framework
P,AN
becomes
Nx Px b.

dene the sequence
N x pi q P x pi q1
b, i 1, 2, . . .
be useful if
General assumptions
In what follows we shall always assume that all diagonal entries are 1.
If this is not true, we can always arrange it to be so by dividing each row

by its diagonal element.
If the diagonal element of any row is zero, permute the rows. It must be
possible to arrange a nonsingular matrix such that every diagonal element
is non-zero just by row operations.

General assumptions


General assumptions


Jacobi's Method
One key requirement on the split
AN P
was that the Ny z should be easy to solve. The simplest possible
problem would be N I.
This means (as A is 1 on the diagonal) that
P AL AU
where AL and AU are the triangular parts of A (note sign!). The

iteration scheme is
x pi q pAL AU qx pi 1q b, i 1, 2, . . .
and the convergence matrix is M P.
Jacobi's Method
One key requirement on the split
AN P
was that the Ny z should be easy to solve. The simplest possible
problem would be N I.
This means (as A is 1 on the diagonal) that
P AL AU
where AL and AU are the triangular parts of A (note sign!). The

iteration scheme is
x pi q pAL AU qx pi 1q b, i 1, 2, . . .
and the convergence matrix is M P.
Example of Jacobi's Method
We look at the solution of the simple linear system

3
1
1
3
x1
x2
5
7
using Jacobi's Method. Ensure the diagonal elements are all 1:
1

5
A b
1
3 3
1 , 7 .
1
3 3
Check convergence by computing
P 0 0
1
3
M 1
3
which implies, as %pM q 1{3 1, that the method will converge.


3
1
1
3
x1
x2
5
7
1

5
A b
1
3 3
1 , 7 .
1
3 3
P 0 0
1
3
M 1
3


3
1
1
3
x1
x2
5
7
1

5
A b
1
3 3
1 , 7 .
1
3 3
P 0 0
1
3
M 1
3

Example of Jacobi's Method 2
The sequence has entries

i x1 x2
0 0 0
1 {
5 3 7 3 {
2 0.888889 1.777778
5 1.008230 2.004115
10 0.999983 1.999966
100 1.000000 2.000000
A starting guess of 0 appears to converge to x p1, 2qT . The

x
convergence is exponential with a slope 1.

Convergence
To work, the sequence

x pi q N 1 P x pi 1q b , i 1, 2, . . .
must have a limit. Intuitively clear that the matrix M N P is central
1
to the convergence or otherwise of this sequence.
Theorem: The iterative method described converges if and only if

M N P exists and
1
%pM q max |λi | 1.

i

Convergence

x pi q N 1 P x pi 1q b , i 1, 2, . . .
1

M N P exists and
1
%pM q max |λi | 1.

i
View x pi q as a vector in n-dimensional space: ensures that repeatedly

multiplying by M does not cause result to diverge.

Convergence

x pi q N 1 P x pi 1q b , i 1, 2, . . .
1

M N P exists and
1
%pM q max |λi | 1.

i
The p q can be hard to compute. Often use the weaker

spectral radius % M
condition
}M } 1 ñ Method converges.

Iterative Methods I
Summary Summary
Iterative Methods II
Outline
Iterative Methods
Jacobi's Method
Convergence
2 Summary
Summary
Iterative Methods
Gauss-Seidel Method

Iterative Methods I
Summary Summary
Iterative Methods II
Summary
Iterative methods are often based on the split
AN P.
Convergence depends on M N P.
1
Jacobi chooses N I and converges slowly.

Summary Gauss-Seidel Method
Iterative Methods II Convergence Analysis
Outline
Iterative Methods
Jacobi's Method
Convergence
2 Summary
Summary
Iterative Methods
Gauss-Seidel Method

Iterative Methods
Ax b.
AN
P, ùñ N x pi q P x pi q 1
b
and for Jacobi use N I .
Jacobi converges slowly: error drops by a constant amount each step.

Iterative Methods
Ax b.
AN
b

Iterative Methods
Ax b.
AN
b

Gauss-Seidel Method
In the Gauss-Seidel method split the coecient matrix as
AI pAL AU q
as before. Choose
N I AL ,
P AU .
This makes the iteration scheme
x pi q AL x pi q AU x pi 1q b, i 1, 2, . . .
Written in this form as AL strictly lower triangular. Consider each

component of the iteration scheme from the top down: each coecient
of x pi q is computed on the left before it is used on the right.

Gauss-Seidel Method
In the Gauss-Seidel method split the coecient matrix as
AI pAL AU q
as before. Choose
N I AL ,
P AU .
This makes the iteration scheme
x pi q AL x pi q AU x pi 1q b, i 1, 2, . . .
Written in this form as AL strictly lower triangular. Consider each

component of the iteration scheme from the top down: each coecient
of x pi q is computed on the left before it is used on the right.

Example of Gauss-Seidel Method
As above we look at

1

5
3
1
1
3
x1
x2
5
7
Ñ 1
1
3
1
x1
x2
3
7 .
3 3
We have

AL 1{30 0
, AU 0 1{3
0 0 0
and hence the iteration algorithm becomes

pi q pi 1q

x2 x pi q 1
pi q 3 3
x1 1 5 1 5
x pi q pi q
2
.
x2 1
7 7x 1
If read row-by-row from the top we have always computed the x pi q entries
before they are used.

Example of Gauss-Seidel Method 2
The sequence has entries

i x1 x2
0 0 0
1 {
5 3 16 9 {
2 1.074074 1.975309
5 1.000102 1.999966
10 1.000000 2.000000
100 1.000000 2.000000
A starting guess of 0 appears to converge to x p1, 2qT . The

x
convergence is exponential with a slope 2, much faster than Jacobi's
Method.

Convergence analysis
Two theorems about when these methods will work:
Theorem 1: If the coecient matrix A is strictly diagonally dominant

then both the Jacobi method and the Gauss-Seidel method will converge.
Theorem 2: If the coecient matrix A is symmetric and positive denite

then the Gauss-Seidel method will converge.

Chapter 1: Iterative Methods: Abdelkrim E Professeur of Applied Mathematics - AI

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 1: Iterative Methods: Abdelkrim E Professeur of Applied Mathematics - AI

Uploaded by

Copyright:

Available Formats

Chapter 1: Iterative Methods

A. El MOUATASIM Computing Science

A. El MOUATASIM Computing Science

We (still) want to solve the linear system

Iterative methods build a sequence of approximate solutions. Start from

Sequence can be truncated to save time or when solution found to

Key elements of an iterative method:

1 A rapid convergence to the correct solution,

2 A fast algorithm for computing each successive approximation.

A. El MOUATASIM Computing Science

We (still) want to solve the linear system

Iterative methods build a sequence of approximate solutions. Start from

Sequence can be truncated to save time or when solution found to

Key elements of an iterative method:

1 A rapid convergence to the correct solution,

2 A fast algorithm for computing each successive approximation.

A. El MOUATASIM Computing Science

We (still) want to solve the linear system

Iterative methods build a sequence of approximate solutions. Start from

Sequence can be truncated to save time or when solution found to

Key elements of an iterative method:

1 A rapid convergence to the correct solution,

2 A fast algorithm for computing each successive approximation.

A. El MOUATASIM Computing Science

Standard notation is to split the matrix as

Then assume that we have an approximate solution, or guess, x p0q , and

Standard notation is to split the matrix as

Then assume that we have an approximate solution, or guess, x p0q , and

Standard notation is to split the matrix as

Then assume that we have an approximate solution, or guess, x p0q , and

If this is not true, we can always arrange it to be so by dividing each row

A. El MOUATASIM Computing Science

If this is not true, we can always arrange it to be so by dividing each row

A. El MOUATASIM Computing Science

If this is not true, we can always arrange it to be so by dividing each row

A. El MOUATASIM Computing Science

One key requirement on the split

where AL and AU are the triangular parts of A (note sign!). The

One key requirement on the split

where AL and AU are the triangular parts of A (note sign!). The

Example of Jacobi's Method

We look at the solution of the simple linear system

using Jacobi's Method. Ensure the diagonal elements are all 1:

Check convergence by computing

which implies, as %pM q  1{3 1, that the method will converge.

A. El MOUATASIM Computing Science

Example of Jacobi's Method

We look at the solution of the simple linear system

using Jacobi's Method. Ensure the diagonal elements are all 1:

Check convergence by computing

which implies, as %pM q  1{3 1, that the method will converge.

A. El MOUATASIM Computing Science

Example of Jacobi's Method

We look at the solution of the simple linear system

using Jacobi's Method. Ensure the diagonal elements are all 1:

Check convergence by computing

which implies, as %pM q  1{3 1, that the method will converge.

A. El MOUATASIM Computing Science

Example of Jacobi's Method 2

The sequence has entries

A starting guess of  0 appears to converge to x  p1, 2qT . The

A. El MOUATASIM Computing Science

To work, the sequence

Standard notation is to split the matrix as

Standard notation is to split the matrix as

Standard notation is to split the matrix as

where AL and AU are the triangular parts of A (note sign!). The

where AL and AU are the triangular parts of A (note sign!). The

which implies, as %pM q 1{3 1, that the method will converge.

which implies, as %pM q 1{3 1, that the method will converge.

which implies, as %pM q 1{3 1, that the method will converge.

A starting guess of 0 appears to converge to x p1, 2qT . The

%pM q max |λi | 1.

%pM q max |λi | 1.

%pM q max |λi | 1.

Jacobi chooses N I and converges slowly.

Standard notation is to split the matrix as

Standard notation is to split the matrix as

Standard notation is to split the matrix as

In the Gauss-Seidel method split the coecient matrix as

In the Gauss-Seidel method split the coecient matrix as

A starting guess of 0 appears to converge to x p1, 2qT . The

Theorem 1: If the coecient matrix A is strictly diagonally dominant

Theorem 2: If the coecient matrix A is symmetric and positive denite