Iterative methods for linear

systems

Seidel iteration, Successive over-relaxation

1/11
Seidel iteration
Ax = b −→ A=L+D+U

lower diagonal upper

Rewrite
Dx + (L + U)x = b

2/11
Seidel’s and Jacobi iteration

▶ Jacobi iteration: Dx(n+1) + (L + U)x(n) = b

▶ Seidel iteration: Dx(n+1) + Lx(n+1) + Ux(n) = b

3/11
Seidel’s iteration in components
 
a11 a12 a13 · · · a1m
a21 a22 a23 · · · a2m 

A= 
a31 a32 a33 · · · a3m 
···

(n+1) (n) (n)

a11 x1 + a12 x2 + a13 x3 + · · · + a1m xm
(n)
= b1
(n+1) (n+1) (n)
a22 x2 + a21 x1 + a23 x3 + · · · + a2m x(n)
m = b2

···
At each iteration, sweep down the system of equations; at each step
(n+1)
use x(n+1) -s from previous steps: for xi use
(n+1) (n+1) (n) (n)
x1 · · · xi−1 (Jacobi iteration uses x1 · · · xi−1 )

4/11
Seidel iterations: Sufficient conditions for convergence

Write the original system as

x = Bx + c

with B = −D−1 (L + U)

Theorem S1
Let ∥B∥1 < 1 or ∥B∥∞ < 1.
Then, Seidel’s iterations converge for any initial x(0) with the rate of
a geometric series,

b∥ ⩽ q n ∥x(0) − x
∥x(n) − x b∥,

with q < ∥B∥

5/11
A weaker convergence condition

De ne BL = −D−1 L and BU = −D−1 U.

Theorem S2
Let ∥BU ∥ + ∥BU ∥ < 1. Then, Seidel’s iterations converge for any
initial x(0) , and
b∥ ⩽ q n ∥x(0) − x
∥x(n) − x b∥,
with
∥BU ∥
q=
1 − ∥BL ∥

6/11
Symmetric positive de nite matrices

In practice, we often have symmetric positive de nite A-s

Theorem S3
Let A is symmetric positive de nite. Then for any initial x(0) ,
Seidel’s iteration converges with the rate of a geometric series.

Note: No conditions on the norm of A or B

7/11
Necessary and sufficient conditions for convergence

Rewrite Seidel’s algorithm as a simple iteration:

e (n) + e
x(n+1) = Bx c
with
e = (D + L)−1 U
B

Theorem S4
Seidel’s iterations converge for any initial x(0) if and only if
e <1
ρ(B)

8/11
Jacobi vs Seidel

Loosely speaking,
▶ Jacobi method is suitable for A ≈ diagonal.
▶ Seidel’s method is suitable for A ≈ triangular (or symmetric).

9/11
Successive over-relaxation method (SOR)

At step j of iteration n + 1,
(n+1)
ej
1. Do a Seidel’s iteration step: compute x
(n+1) (n+1) (n)
2. Shift xi = ωe
xi + (1 − ω)xi
Here ω is an arbitrary parameter ω ∈ [0, 2]. Tune ω to speed up
convergence.

In the matrix form

Seidel iteration: x(n+1) = −D−1 Lx(n+1) − D−1 Ux(n) + c

SOR iteration: x(n+1) = (1 − ω)x(n)

( )
+ ω −D−1 Lx(n+1) − D−1 Ux(n) + c

10/11
Further variations of SOR

▶ Use j-dependent ωj
▶ Symmetric successive over-relaxation
▶ ...

11/11