Method of Conjugate

Gradients
Dragica Vasileska

CG Method Conjugate Gradient (CG) Method

M. R. Hestenes & E. Stiefel, 1952
BiCG Method Biconjugate Gradient (BiCG) Method
C. Lanczos, 1952
D. A. H. Jacobs, 1981
C. F. Smith et al., 1990
R. Barret et al., 1994
CGS Method Conjugate Gradient Squared (CGS) Method (MATLAB Function)
P. Sonneveld, 1989
GMRES Method Generalized Minimal Residual (GMRES) Method
Y. Saad & M. H. Schultz, 1986
R. Barret et al., 1994
Y. Saad, 1996
QMR Method QuasiMinimalResidual (QMR) Method
R. Freund & N. Nachtigal, 1990
N. Nachtigal, 1991
R. Barret et al., 1994
Y. Saad, 1996

Conjugate Gradient Method

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Introduction, Notation and Basic Terms

Eigenvalues and Eigenvectors
The Method of Steepest Descent
Convergence Analysis of the Method of Steepest Descent
The Method of Conjugate Directions
The Method of Conjugate Gradients
Convergence Analysis of Conjugate Gradient Method
Complexity of the Conjugate Gradient Method
Preconditioning Techniques
Conjugate Gradient Type Algorithms for NonSymmetric Matrices (CGS, Bi-CGSTAB Method)

1. Introduction, Notation and Basic Terms

The CG is one of the most popular iterative methods for solving
large systems of linear equations
Ax=b
which arise in many important settings, such as finite difference and
finite element methods for solving partial differential equa-tions,
circuit analysis etc.
It is suited for use with sparse matrices. If A is dense, the best
choice is to factor A and solve the equation by backsubstitution.
There is a fundamental underlying structure for almost all the
descent algorithms: (1) one starts with an initial point; (2) determines according to a fixed rule a direction of movement; (3) mo-ves
in that direction to a relative minimum of the objective function; (4)
at the new point, a new direction is determined and the process is
repeated. The difference between different algorithms depends upon
the rule by which successive directions of movement are selected.

1. Introduction, Notation and Basic Terms (Contd)

A matrix is a rectangular array of numbers, called elements.
The transpose of an mn matrix A is the nm matrix AT with
elements
aijT = aji
Two square nn matrices A and B are similar if there is a nonsingular matrix S such that
B=S-1AS
Matrices having a single row are called row vectors; matrices
having a single column are called column vectors.
Row vector:
a = [a1, a2, , an]
Column vector:
a = (a1, a2, , an)
The inner product of two vectors is written as
T

x y x i yi
i 1

1. Introduction, Notation and Basic Terms (Contd)

A matrix is called symmetric if aij = aji .
A matrix is positive definite if, for every nonzero vector x
xTAx > 0
Basic matrix identities: (AB)T=BTAT and (AB)-1=B-1A-1
A quadratic form is a scalar, quadratic function of a vector with the
form
1 T
f(x) x Ax b T x c
2
The gradient of a quadratic form is
f(x)
x 1

1 T
1

A x Ax b
f ' ( x)

2
2
f(x)

x n

1. Introduction, Notation and Basic Terms (Contd)

Various quadratic forms for an arbitrary 22 matrix:
Positive-definite
matrix

Singular
positive-indefinite
matrix

Negative-definite
matrix

Indefinite
matrix

1. Introduction, Notation and Basic Terms (Contd)

Specific example of a 22 symmetric positive definite matrix:
3 2
2
2

A
, b
, c0 x
2 6
8
2
Contours of the
quadratic form

Gradient of
the quadratic form

Graph of a quadratic form f(x)

2. Eigenvalues and Eigenvectors

For any nn matrix B, a scalar and a nonzero vector v that satisfy
the equation
Bv=v
are said to be the eigenvalue and the eigenvector of B.
If the matrix is symmetric, then the following properties hold:
(a) the eigenvalues of B are real
(b) eigenvectors associated with distinct eigenvalues are
orthogonal
The matrix B is positive definite (or positive semidefinite) if and
only if all eigenvalues of B are positive (or nonnegative).
Why should we care about the eigenvalues? Iterative methods often
depend on applying B to a vector over and over again:
(a) If ||<1, then Biv=iv vanishes as i approaches infinity
(b) If ||>1, then Biv=iv will grow to infinity.

2. Eigenvalues and Eigenvectors (Contd)

Examples for ||<1 and || >1
=-0.5

The spectral radius of a matrix is: (B)= max|i|

=0.7
=-2

2. Eigenvalues and Eigenvectors (Contd)

Example: The Jacobi Method for the solution of Ax=b
- split the matrix A such that A=D+E
- Instead of solving the system Ax=b, one solves the modified system x=Bx+z, where B=-D-1E and z=D-1b
- The iterative method is: x(i+1)= Bx(i)+z, or
it terms of the error vector: e(i+1)= Be(i)
- For our particular example, we have that the eigenvalues
and the eigenvectors of the matrix A are:
1=7, v1=[1, 2]T
2=2, v2=[-2, 1]T
- The eigenvalues and eigenvectors of the matrix B are:
1=-2/3=-0.47,
v1=[2, 1]T
2= 2/3=0.47,
v2=[-2, 1]T

2. Eigenvalues and Eigenvectors (Contd)

Graphical representation of the convergence of the Jacobi method
Eigenvectors of B
together with their
eigenvalues

Convergence of the
Jacobi method which
starts at [-2,-2]T and
converges to [2, -2]T

The error vector e(0)

The error vector e(1)

The error vector e(2)

3. The Method of Steepest Descent

In the method of steepest descent, one starts with an arbitrary point
x(0) and takes a series of steps x(1), x(2), until we are satis-fied that
we are close enough to the solution.
When taking the step, one chooses the direction in which f
decreases most quickly, i.e.
f ' (x (i) ) b Ax (i)
Definitions:
error vector: e(i)=x(i)-x
residual:
r(i)=b-Ax(i)
From Ax=b, it follows that
r(i)=-Ae(i)=-f(x(i))
The residual is actually the direction of steepest descent

3. The Method of Steepest Descent (Contd)

Start with some vector x(0). The next vector falls along the solid line
x (1) x ( 0 ) r( 0 )
The magnitude of the step is determined with a line search procedure that minimizes f along the line:

d
T dx (1)
f(x (1) ) f ' (x (1) )
0
d
d
r(T1)r( 0 ) 0

b Ax (1) T r(0) 0
b A x (0) r(0) T r(0) 0
r(T0 ) r( 0 ) Ar( 0 ) T r( 0 ) 0

r(T0 ) r( 0 )
r(T0 ) Ar( 0 )

3. The Method of Steepest Descent (Contd)

Geometrical representation
(e)

(f)

3. The Method of Steepest Descent (Contd)

The algorithm

r(i) b Ax (i)
(i)

T
r(i)
r(i)
T
r(i)
Ar(i)

Two matrix-vector
multiplications are
required.

x (i 1) x (i) (i) r(i) e (i 1) e (i) (i) r(i)

To avoid one matrix-vector multiplication, one uses
r(i 1) r(i) (i) Ar(i)

The disadvantage of using this recurrence is that the residual sequence is

determined without any feedback from the value of x(i), so that round-off
errors may cause x(i) to converge to some point near x.

4. Convergence Analysis of the Method of Steepest Descent

The error vector e(i) is equal to the
eigenvector vj of A
r(i) Ae (i) Av j j v j
r T r(i)
1
(i)
(i)
, e (i 1) e (i) (i) r(i) 0
r T Ar(i) j
(i)

The error vector e(i) is a linear combination of the eigenvectors of A, and all
eigenvalues are the same
n

e (i) j v j
j1

r(i) Ae (i) j j v j j v j
T

j1

r r(i)
(i)

(i)

r T Ar(i)
(i)

j1

1
, e (i 1) e (i) (i) r(i) 0

3. The Method of Steepest Descent

In the method of steepest descent, one starts with an arbitrary point
x(0) and takes a series of steps x(1), x(2), until we are satis-fied that
we are close enough to the solution.
When taking the step, one chooses the direction in which f
decreases most quickly, i.e.
f ' (x (i) ) b Ax (i)
Definitions:
error vector: e(i)=x(i)-x
residual:
r(i)=b-Ax(i)
From Ax=b, it follows that
r(i)=-Ae(i)=-f(x(i))
The residual is actually the direction of steepest descent

3. The Method of Steepest Descent (Contd)

Start with some vector x(0). The next vector falls along the solid line
x (1) x ( 0 ) r( 0 )
The magnitude of the step is determined with a line search procedure that minimizes f along the line:

d
T dx (1)
f(x (1) ) f ' (x (1) )
0
d
d
r(T1)r( 0 ) 0

b Ax (1) T r(0) 0
b A x (0) r(0) T r(0) 0
r(T0 ) r( 0 ) Ar( 0 ) T r( 0 ) 0

r(T0 ) r( 0 )
r(T0 ) Ar( 0 )

3. The Method of Steepest Descent (Contd)

Geometrical representation
(e)

(f)

3. The Method of Steepest Descent (Contd)

Summary of the algorithm

r(i) b Ax (i)
T

(i)

r (i) r(i)
T

r (i) Ar(i)

x (i 1) x (i) (i)r(i)
To avoid one matrix-vector
multiplication, one uses instead
r(i 1) r(i) (i) Ar(i)

Efficient implementation of the

method of steepest descent
i0
r b Ax
r Tr
While i i max and 2 0
q Ar

T
r q
x x r
If i is divisib le by 50
r b - Ax
else
r r - q
endif
r Tr
i i 1

4. Convergence Analysis of the Method of Steepest Descent

The error vector e(i) is equal to the
eigenvector vj of A
r(i) Ae (i) Av j j v j
r T r(i)
1
(i)
(i)
, e (i 1) e (i) (i) r(i) 0
r T Ar(i) j
(i)

The error vector e(i) is a linear combination of the eigenvectors of A, and all
eigenvalues are the same
n

e (i) j v j
j1

r(i) Ae (i) j j v j j v j
T

j1

r r(i)
(i)

(i)

r T Ar(i)
(i)

j1

1
, e (i 1) e (i) (i) r(i) 0

4. Convergence Analysis of the Method of Steepest Descent

(Contd)
General convergence of the method can be proven by calculating
the energy norm
2

e(i 1) e T(i 1) Ae (i 1)
A

e(i )

2
A

, 1

2 2 2
j j j
2 3
2
j j j j j j

For n=2, one has that

2 1

( 2 2 )2
( 2 )( 3 2 )

max / min , 2 / 1
Conditioning number

1
1

4. Convergence Analysis of the Method of Steepest Descent

(Contd)

Worst-case scenario: =

The influence of the spectral

condition number on the convergence

5. The Method of Conjugate Directions

Basic idea:
Mathematical formulation:
1. Pick a set of orthogonal
1. For each step we choose a
search directions d(0), d(1), ,
point
d(n-1)
x(i+1)=x(i)+ (i) d(i)
2. Take exactly one step in
2. To find (i), we use the fact
each search direction to line up
with x
that e(i+1) is orthogonal to d(i)

d T( i ) e ( i 1) 0
d T( i ) e ( i ) ( i ) d ( i ) 0
d T( i ) e ( i ) ( i ) d T( i ) d ( i ) 0
(i)

d T( i ) e ( i )
d T( i ) d ( i )

5. The Method of Conjugate Directions (Contd)

To solve the problem of not knowing e(i), one makes the search
directions to be A-orthogonal rather then orthogonal to each other,
i.e.:
d T( i ) Ad ( j) 0

5. The Method of Conjugate Directions (Contd)

The new requirement is now that e(i+1) is A-orthogonal to d(i)
d
T dx (i 1)
f(x (i 1) ) f ' (x (i 1) )
0
d
d
r(iT1)d (i) 0

d T(i ) Ae (i 1) 0
d T(i ) A e (i ) (i )d (i ) 0
(i )

d T(i) r(i)
d T(i) Ad (i)

If the search vectors were the residuals, this

formula would be identical to the method of
steepest descent.

5. The Method of Conjugate Directions (Contd)

Proof that this process computes x in n-steps
- express the error terms as a linear combination of the search
directions
n 1

e ( 0) ( j) d ( j)
j 0

- use the fact that the search directions are A-orthogonal

5. The Method of Conjugate Directions (Contd)

Calculation of the A-ortogonal search directions by a conjugate
Gram-Schmidth process
1. Take a set of linearly independent vectors u0, u1, , un-1
2. Assume that d(0)=u0
3. For i>0, take an ui and subtracts all the components from it
that are not A-orthogonal to the previous search directions
i 1

u T(i) Ad ( j)

j 0

d T(j) Ad ( j)

d (i ) u (i ) ijd ( j) , ij

5. The Method of Conjugate Directions (Contd)

The method of Conjugate Directions using the axial unit vectors as
basis vectors

5. The Method of Conjugate Directions (Contd)

Important observations:

1. The error vector is A-orthogonal to all previous search directions

2. The residual vector is orthogonal to all previous search directions
3. The residual vector is also orthogonal to all previous basis vectors.
d iT Ae j d iT rj u iT rj 0 for i j

6. The Method of Conjugate Gradients

The method of Conjugate Gradients is simply the method of
conjugate directions where the search directions are constructed by
conjugation of the residuals, i.e. ui=r(i)
This allows us to simplify the calculation of the new search
direction because
1
r(Ti )r(i )
r(Ti )r(i )
T

ij (i 1) d T Ad
r(i 1)r(i 1)
( i 1)
( i 1)

i j1
i j1

The new search direction is determined as a linear combination of

the previous search direction and the new residual
d (i 1) r(i 1) id (i)

6. The Method of Conjugate Gradients (Contd)

Efficient implementation
i0
r b Ax
dr
new r T r
0 new
While i i max and new 2 0
q Ad

new
dTq
x x d
If i is d ivisible b y 50
r b - Ax
else
r r - q
endif
old new
new r T r

new
old
d r d
i i 1

The Landscape of Ax=b

Solvers
Direct
A = LU
Nonsymmetric
Symmetric
positive
definite

Iterative
y = Ay

Pivoting
LU

GMRES,
BiCGSTAB,

Cholesky

Conjugate
gradient

More Robust

More General

More Robust
Less Storage (if sparse)

Conjugate gradient iteration

x0 = 0, r0 = b, d0 = r0
for k = 1, 2, 3, . . .
k = (rTk-1rk-1) / (dTk-1Adk-1)
xk = xk-1 + k dk-1
rk = rk-1 k Adk-1
k = (rTk rk) / (rTk-1rk-1)
dk = rk + k dk-1

step length
approx solution
residual
improvement
search direction

One matrix-vector multiplication per iteration

Two vector dot products per iteration
Four n-vectors of working storage

7. Convergence Analysis of the CG Method

If the algorithm is performed in exact arithmetic, the exact solution
is obtained in at most n-steps.
When finite precision arithmetic is used, rounding errors lead to
gradual loss of orthogonality among the residuals, and the finite
termination property of the method is lost.
If the matrix A has only m distinct eigenvalues, then the CG will
converge in at most m iterations
An error bound on the CG method can be obtained in terms of the
A-norm, and after k-iterations:

x xk

2 x x0

( A ) 1
A ( A ) 1

8. Complexity of the CG Method

Dominant operations of Steepest Descent or Conjugate Gradient
method are the matrix vector products. Thus, both methods have
spatial complexity O(m).
Suppose that after I-iterations, one requires that ||e(i)||A||e(0)||A
(a) Steepest Descent: i 12 ln 1

(b) Conjugate Gradient: i 1 ln 2

The time complexity of these two methods is:

(a) Steepest Descent: O ( m )
(b) Conjugate Gradient: O ( m )
Stopping criterion: ||r(i)||||r(0)||

9. Preconditioning Techniques
References:
J.A. Meijerink and H.A. van der Vorst, An iterative solution method for
linear systems of which the coefficient matrix is a symmetric M-matrix,
Mathematics of Computation, Vol. 31, No. 137, pp. 148-162 (1977).
J.A. Meijerink and H.A. van der Vorst, Guidelines for the usage of
incomplete decompositions in solving sets of linear equations as they
occur in practical problems, Journal of Computational Physics, Vol. 44,
pp. 134-155 (1981).
D.S. Kershaw, The incomplete Cholesky-Conjugate gradient method for
the iterative solution of systems of linear equations, Journal of
Computational Physics, Vol. 26, pp. 43-65 (1978).
H.A. van der Vorst, High performance preconditioning, SIAM Journal
of Scientific Statistical Computations, Vol. 10, No. 6, pp. 1174-1185
(1989).

9. Preconditioning Techniques (Contd)

Preconditioning is a technique for improving the condition number
of a matrix. Suppose that M is a symmetric, positive-definite matrix
that approximates A, but is easier to invert. Then, rather than
solving the original system
Ax=b
one solves the modified system
M-1Ax= M-1b
The effectiveness of the preconditioner depends upon the condition
number of M-1A.
Intuitively, the preconditioning is an attempt to stretch the quadratic
form to appear more spherical, so that the eigenvalues become
closer to each other.
Derivation:
M EET , E 1AE T ~
x E 1b, ~
x ET x

9. Preconditioning Techniques (Contd)

Transformed Preconditioned CG Untransformed PCG Algorithm
~
~
(PCG) Algorithm
r E 1r , d ET d
~
i 0, ~
r E 1b E 1AE T ~
x, d ~
r

~
rT ~
r,
new

new

While i imax and new 2 0

~
~ E 1AE T d
q

~new
~
dT q
~
~
x~
x d
~
~
r E 1b - E 1AE T ~
x or ~
r ~
r - q
old new
new ~r T ~r

new
old
~
~
d~
r d
i i 1

i 0, r b Ax, d M 1r

new rT M -1r , 0 new

While i imax and new 2 0
q Ad

new
dT q
x x d
r b - Ax or r r - q
old new
new rT M -1r

new
old
d M 1r d
i i 1

9. Preconditioning Techniques (Contd)

There are sevaral types of preconditioners that can be used:
(a) Preconditioners based on splitting of the matrix A, i.e.
A=M-N
(b) Complete or incomplete factorization of A, e.g.
A = LLT + E
(c) Approximation of M=A-1
(d) Reordering of the equations and/or unknowns, e.g.
Domain Decomposition

9. Preconditioning Techniques (Contd)

(a) Preconditioning based on splittings of A
Diagonal Preconditioning:
M=D=diag{a11, a22, ann}
3 2
2
2
A
, b , c 0 x
2 6
8
2

Tridiagonal Preconditioning
Block Diagonal Preconditioning

9. Preconditioning Techniques (Contd)

(b) Preconditioning based on factorization of A
Cholesky Factorization (for symmetric and positive definite matrix)
as a direct method:
A = LLT
x = (L-T)(L-1b)
where:
Lii

i 1

Aii Lik
k 1
i 1

A ji L jk Lik
L ji

k 1

, j i 1, i 2, , n

Lii
Modified Cholesky factorization: A = LDLT

9. Preconditioning Techniques (Contd)

(b) Preconditioning based on factorization of A
Incomplete Cholesky Factorization:
A = LLT + E or A = LDLT + E , where Lij=0 if (i,j)P
Simplest choice is: P={(i,j)| aij=0; i,j,=1,2,,n} => ICCG(0)

~
~
ai bi ci
i bi c~i
di

LT
D

~
~ ~
~
~
a~ d 1 a b 2 d c~ 2 d
, b b , c~ c , i 1,2, , n
i

i 1 i 1

i m i m
1

i
T

Modified ICCG(0): M ( L D )D ( D L )
bi21 ci2 m

diag ( A ) diag ( M ), d i ai
d i 1 d i m

9. Preconditioning Techniques (Contd)

(c) Preconditioning based on approximation of A
If (J)<1, then the inverse of I-J is
(I J )

J k I J J 2 J3
k 0

Write matrix A as:

A=D+L+U=D(I+D-1(L+U)) => J= -D-1(L+U))
The inverse of A can be expressed as:
A 1 ( D(I J )) 1 (I J ) 1 D 1

( 1) D (L U ) D 1
k 0

For k=1, one has that: M 1 D 1 D 1 (L U ) D 1

For k=m, one usually uses:

m
k 1
1
k 1
M m k ( 1) D (L U ) D
k 0

9. Preconditioning Techniques (Contd)

(d) Domain Decomposition Preconditioning
Domain I
Domain II

A
1
Ax b, 0
BT
1

0
A2
BT

B1 x1 d1

B2 x2 d2

Q z f

M -1
0
0
M

0
0
1
1

1
T
M L 0
M
0 L , where L 0 M 2 0
2

B T BT S

1
0
0
S
1

0
B
1
1

M 0 M 2 B 2 , S* S BT M -1B1 BT M -1B 2
1
1
2
2
BT BT S*
1

10. Conjugate Gradient Type Algorithms for

Nonsymmetric Matrices
T
The Bi-Conjugate Gradient i 0, r b Ax, r r* 0
T *
*
*

p
r,
p
r
r
r , 0 new
new
(BCG) Algorithm was propoWhile i imax and new 2 0
sed by Lanczos in 1954.
q Ap
It solves not only the original
q* AT p*
new
system Ax=b but also the dual

linear system ATx*=b*.

p*T q
x x p
Each step of this algorithm
r r - q
requires a matrix-by-vector
r * r * - q*
product with both A and AT.
old new
*T
*

r
r
new
The search direction pj does
new

not contribute to the solution

old
directly.
p r p

p* r * p*
i i 1

10. Conjugate Gradient Type Algorithms for

Nonsymmetric Matrices (Contd)
The Conjugate Gradient
Squared (CGS) Algorithm
was developed by Sonneveld in
1984.
Main idea of the algorithm:
r j j ( A )r0
p j j ( A )r0
r*j j ( AT )r0*
p*j j ( AT )r*0

i 0, r0 b Ax 0 , r0* is arbitrary
p r0 , u r0
new r*0T r , 0 new
While i imax and new 2 0

*Tnew
r0 Ap
q u Ap
x x (u q )
r r - A (u q )
old new
new r0*T r

new
old
u r q
p u (q p )
i i 1

10. Conjugate Gradient Type Algorithms for

Nonsymmetric Matrices (Contd)
The Bi-Conjugate Gradient
Stabilized (Bi-CGSTAB)
Algorithm was developed by
van der Vorst in 1992.
Rather than producing iterates
whose residual vectors are of
the form
r j 2 ( A )r0
j

it produces iterates with

residual vectors of the form
r j j ( A ) j ( A )r0
p j j ( A ) j ( A )r0

i 0, r0 b Ax 0 , r0* is arbitrary
p r0
new r0*T r , 0 new
While i imax and new 2 0

*Tnew
r0 Ap
s r Ap
sT As

As T As
x x p s
r s - As
old new
new r0*T r

new
old
p r (p Ap)
i i 1

Complexity of linear solvers

Time to solve
model problem
(Poissons
equation) on
regular mesh

n1/2

n1/3

2D

3D

Sparse Cholesky:

O(n1.5 )

O(n2 )

CG, exact
arithmetic:

O(n2 )

O(n2 )

CG, no precond:

O(n1.5 )

O(n1.33 )

CG, modified IC:

O(n1.25 )

O(n1.17 )

CG, support trees:

Multigrid:

O(n1.20 ) -> O(n1+ ) O(n1.75 ) -> O(n1.31 )

O(n)

O(n)