You are on page 1of 18

Lecture 2:
Iterative methods for system of linear equations: The
Computer Tutorial 3: Implementation

## Reference: J. R. Shewchuk, An introduction to the

conjugate gradient method without the agonizing
pain, (1994). available at:
http://www.cs.cmu.edu/~jrs/jrspapers.html

## Objective: Given a Hermitian matrix A, and a

vector b, solve the linear system
Ax = b
Some linear algebra:

## A: Hermitian matrix and symmetric, positive

definite
zTAz > 0 for all nonzero vectors, z, with real elements.
positive definite example:

1 0 z0 1 0 z0
A , z z0 z1 z02 z12 0.
0 1 z1 0 1 z1
non-positive definite example:

0 1 1 0 1 1
A , z 1 1 2 0.
1 0 1 1 0 1
Some linear algebra:

## Inner product (dot, scalar) of two vectors, x, y:

n
T T
x, y x y y x xi yi
i 1
Orthogonal vectors: <x,y> = 0.
Transpose of matrix multiplication: (AB)T = BTAT
Inverse of matrix multiplication: (AB)-1 = B-1A-1

1 T T
f ( x) x Ax b x c
2
3 2 2
Example: A , b , c 0.
2 6 8
Plot of f(x):

figure?
T
f f f
f ( x) f ( x)
x1 x2 xn

## For every x, the gradient points in the direction of

steepest increase of f(x), and is orthogonal to the
contour lines

## Inner product (dot, scalar) of two vectors, x, y:

1 T
f ( x) x Ax bT x c
2
1 T 1 A is symmetric:
f ( x) A x Ax b Ax b AT = A
2 2

## Setting f(x) = 0 gives the minimization problem for

f(x). Hence, Ax = b can be solved by finding x that
minimizes f(x).
Method of Steepest Descent
Find residual vector: r(i) = b Ax(i)
This indicates how far we are from the correct value of b.
Note that r(i) = -f (x(i))
Also, if e(i) = x(i) x is the (error) vector indicating how far
we are from the solution, then r(i) = -Ae(i)
Determine the direction for the next step: move in the
direction in which f(x) decreases most quickly, i.e.
opposite f (x), that is, r(i) .
How big a step should be taken? x(1) = x(0) + r(0)
Determine by the condition that it should minimize f:
d T d
f ( x(1) ) f ( x(1) ) x(1) f ( x(1) )T r( 0) 0
d d
Method of Steepest Descent

## Note that f (x(1)) = -r(1)

T
r(1) r( 0 ) 0
(b Ax(1) )T r( 0 ) 0
(b A( x( 0 ) r( 0 ) ))T r( 0 ) 0
(b Ax( 0) )T r( 0 ) ( Ar( 0 ) )T r( 0 ) 0
(b Ax( 0 ) )T r( 0 ) ( Ar( 0 ) )T r( 0 )
r(T0 ) r( 0 ) r(T0 ) ( Ar( 0 ) )
r(T0 ) r( 0 )
r(T0 ) Ar( 0 )
Method of Steepest Descent
r(i ) b Ax(i )
T
r(i ) r(i )
(i ) T
r Ar(i )
(i )
x(i 1) x(i ) r
(i ) (i )
Premultiplying last equation by A

## r(i 1) r(i ) (i ) Ar(i )

Use this for i > 0. CAUTION: Since the feedback from x(i) is not
present here, use the form above periodically to prevent
misconvergence
Method of Steepest Descent was constructing
steps with successive residual vectors being
orthogonal:
T
r r
(1) ( 0 ) 0
Conjugate gradient method employs vectors that
are A-orthogonal (or conjugate)
T
(i ) 0
Details of the derivation of the method are
omitted

d (0) b Ax( 0 )
r(Ti) r(i )
(i )
d (Ti ) Ad (i )
x(i 1) x(i ) (i ) d (i )

r(i 1) r(i ) ( i ) Ad ( i )

r(Ti 1) r(i 1)
( i 1)
r(Ti) r(i )
r(i 1) r(i 1) ( i 1) d (i )

## If the matrix A is ill conditioned, the CG method may suffer

from numerical errors (rounding, overflow, underflow).
2 1 x 3
x 1501.5, y 3000
2 1.001 y 0
2 1 x 3
x 751.5, y 1500
2 1.002 y 0
1 1, ill conditioned
Matrix condition number: cond A A A
1, well conditioned
n
Matrix norm: A max Aij
1 i n
j 1

## For this example cond(A) = 5001 >>1

Suppose that M is a
symmetric positive definite
r( 0 ) b Ax( 0 )
matrix that approximates A, d (0) M 1r( 0 )
but easier to invert (well T 1
conditioned). Then we can r( i ) M r( i )
solve instead: M-1 Ax = M-1x (i )
d (Ti ) Ad ( i )
x(i 1) x( i ) (i ) d (i )

r(i 1) r(i ) ( i ) Ad ( i )
T 1
r( i 1) M r( i 1)
( i 1)
r(Ti) M 1r( i )
1
d (i 1) M r( i 1) ( i 1) d (i )

## Jacobi preconditioner: Aii i j

M ij
0 i j
Symmetric successive overrelaxation
preconditioner:
A L D LT
where L is the strictly lower part of A and D is
diagonal of A.
D 1 D
M ( L) D ( LT )
2
in the interval ]0,2[ is the relaxation parameter to
be chosen.
CG Method: sample code for Matlab
CG Method: sample problem

Sample problem:
4 1 1 x1 12
1 4 2 x2 1
1 2 4 x3 5

exact solution: x1 = 3, x2 = x3 = 1.