2-Linear System PDF

Chap 2: Solving sets of Equations
• Metrics and vectors

• Elimination methods
• Inverse of matrix
• Ill-conditioned systems
• Iterative methods
Numerical Method, Chap 2 2022 1 Jung Hong Chuang, CS Dept., NCTU

Matrices and vectors
• Matrix notation
• Operations on matrices
• Linear system in matrix form
• Inner product, outer product
• Unit vector, zero vector
• Diagonal matrix
• Identity matrix
• Transposition matrix
• Permutation matrix

• Transposition matrix
– Two rows of an identity matrix are interchanged
1 0 0 0 9 6 2 13
0 0 0 1 4 2 8 1 
P1   , A
0 0 1 0 0 7 1 9
   
0 1 0 0 3 2 6 8
9 6 2 13 9 13 2 6
3 2 6 8  4 8 8 2
P1 A   AP1   
0 7 1 9 0 9 1 7 
   
4 2 8 11  3 11 6 2 
Row interchanged Column interchanged

• Permutation matrix
– Multiplication of several transposition matrices
• Symmetric matrix: aij=aji
• Transpose of A matrix
– Writing the rows as columns
If A is symmetric, then A  AT
For any matrix, ( AT )T  A and ( AB)T  BT AT
• Trace of a square matrix

tr( A)  sum of diagonal elements
tr( A)  tr ( AT )
• Lower triangular matrix: aij=0, for j>i
• Upper triangular matrix: aij=0, for i>j
• Tridiagonal matrix
– Has nonzero elements only on the diagonal and
in the position adjacent to the diagonal
– Can be stored as a matrix of nx3

• Determinant of a square matrix: det(A)
– A 3x3 matrix: spaghetti rule
– General rule:
• To expand in terms of the minors of some row or
column
– Triangularize a matrix first before computing
the determinant
• Determinant of a triangular matrix
– Product of the diagonal elements

• Characteristic polynomial of a matrix
 is an eigenvalue of the Engenvecto r corresponds to an
natrix A if there is a nonzero eigenvalue  : is the nonzero v
vector x such that such that
Ax  x Av  v.
( A  I ) x  0.
The characteristic polynomial : Trace of a matrix A
 sum of eigenvalue s of A.
PA ( )  det ( A  I )
If a matrix is triangula r, its eigenvalue s
Eigenvalue s are the roots of
are equal to the diagonal elements.
PA ( )  0.

Elimination methods
Solving a linear system Ax=b
• If A is a upper-triangular matrix, the
system can be solved by back substitution
4 x1  2 x2  x3  15
 10 x2  19 x3  77
 72 x3  216

Elimination methods
• Reduce the coefficient matrix to a upper-
triangular matrix
– Elimination method based on elementary row
operations
4 x1  2 x2  x3  15 4 x1  2 x2  x3  15 4 x1  2 x2  x3  15
 3x1  x2  4 x3  8  10 x2  19 x3  77  10 x2  19 x3  77
x1  x2  3x3  13  x2  11x3  37  72 x3  216

Elimination methods
• Elementary row operations
– Multiply any row of the augmented coefficient
matrix by a constant
– Add the multiple of one row to a multiple of any
other row
– Interchange the order of any two rows if
necessary
• Need to guard against zero multipliers by row
interchange
Yield an equivalent linear system, why?
But may have effect on the accuracy of the computed solution!
Elimination methods
Matrix form: work on augmented coefficient matrix A|b
 4  2 1  x1  15
  3  1 4  x    8 
  2   
 1  1 3  x3  13
 4  2 1 15 3 R  4 R 4  2 1 15 
 3  1 4 8   1 2
0  10 19 77 
  ( 1) R1  4 R3  
 1  1 2 13 0  2 11 37 
4  2 1 15 
 0  10 19 77 
2 R2 10 R3
0 0  72  216

Elimination methods
• During triangulation, if a zero is encountered on
the diagonal, we cannot use that row to eliminate
coefficients below that zero element
– Need to do row interchange
• If there is a zero on the diagonal after
triangulation, the back-substitution fails and there
is no solution!

Gaussian elimination
• Avoids the large coefficients resulting from
elimination by subtracting aij/ajj times the
first equation from the ith equation
– Increase precision
• Apply pivoting to avoid zero multiplier and
increase precision
– Complete pivoting
• May require row and column interchange
• Not frequently used
– Partial pivoting
• Require only row interchange
• Order vector can be used to keep track of the order of
rows when a row interchange is done
• pivoting increases precision. Why?
1. In reduction :
a
row j  ji row i
aii
Elements in row i has propogated errors.
a ji
[aik   ik ] has less error when aii is large.
aii
2. In back substituti on :
  xi 1   i 1  
1    
xi   ci  [ui ,i 1, ...., ui ,n , ]


uii  
  xn   n  
has less error when uii is large.
 4  2 1  x1  15
  3  1 4  x    8 
  2   
 1  1 3  x3  13
 4  2 1 15 R ( 3 / 4 ) R 4  2 1 15 
 3  1 4 8  2  1 0  2.5 4.75 19.25
  R3 (1/ 4 ) R1  
 1  1 2 13 0  0.5 2.75 9.25 
4  2 1 15 
 0  2.5 4.75 19.25
R3  ( 0.5 /  2.5 ) R2
0 0.0 1.80 5.40 
No pivoting is required here!

LU decomposition
• The multipliers can be stored in place of
zero
– Form a lower-triangular matrix, called L .
 4 2 1 15 
(0.75)  2.5 4.75 19.25

 (0.25) (0.20) 1.80 5.40 
 1 0 0 4  2 1 
L   0.75 1 0 U  0  2.5 4.75
 0.25 0.20 1 0 0 1.80 
We can easily var ify that
A  LU (when no pivoting is done) WHY??
LU decomposition
• If row interchange is performed,
A’ =LU, where
A’ is a permutation of the rows of A due to
row interchange from pivoting

LU decomposition


LU decomposition
Row interchange can be expensive.
Order vector: keeps track the order of rows. When a row interchanges is
indicated, we only change the corresponding elements in the order vector
0 2 0 1   x1   0  1   4   4 
2 2 3 2   x    2  2  2  3
  2     order vector :        
4  3 0 1   x3   7  3  3  2
          
6 1  6  5  x4   6   4  1   1 
0 2 0 1 0 6 1  6  5 6
2 2 3 2  2  row 1 row 4 2 2 3 2  2
   
4  3 0 1  7 4  3 0 1  7
   
 6 1  6  5 6  0 2 0 1 0
6 1 6 5 6  6 1 6 5 6 
G . E . 0 1 .6667 5 3 .6667  4  row 2 row 30  3.6667 4 4.3333  11
    
0  3.6667 4 4.3333  11 0 1.6667 5 3.6667  4 
   
 0 2 0 1 0  0 2 0 1 0 
LU decomposition
6 1 6 5 6  6 1 6 5 6 

G.E . 0  3.6667 4 4.3333  11  
G.E . 0  3. 6667 4 4.3333  11 
     
0 0 6.8182 5.6364  9.0001 0 0 6.8182 5.6364  9.0001
   
0 0 2.1818 3.3636  5.9999  0 0 0 1.5600  3.1199 
Back-substituti on order :1, 2, 3, 4
 1 0 0 0 6 1 6 5 
0.66667 1 0 
0  0  3. 6667 4 4.3333 
L U  
0.33333  0.45454 1 0 0 0 6.8182 5.6364 
   
 0.0  0.54545 0.32 1 0 0 0 1.5600 
6 1  6  5
4  3 0 1 
LU  A'   det( A)  (1) 2  6  3.6667  6.8182 1.5600  234.0028 ~ 234(exact)
2 2 3 2
 
0 2 0 1

Operational count
• For augmented matrix [A b]:
To reduce the elements below Total :

the diagonal in column 1 : Divisions
n 1
Divisions  n  1,   n  i  n 2 / 2  n / 2,
i 1
multiplica tions  n(n  1)
multiplica tions
Substracti ons  n(n  1) n 1
  (n  i  1)(n  i )  n 3 / 3  n / 3
i 1
For column i : Substracti ons
Divisions  n  i, n 1
  (n  i  1)(n  i )  n 3 / 3  n / 3
multiplica tions  (n  i  1)(n  i ) i 1
Substracti ons  (n  i  1)(n  i )

Total : 2n 3 / 3  n 2 / 2  7n / 6.
Gauss-Jordan scheme
• Variants to the Gaussian elimination
– Back-substitution can be performed by
eliminating elements above the diagonal after
the triangulation, using elementary row
operation upward from the last row
• The diagonal elements may all be made one as the first
step
• Gauss-Jordan scheme
– The elements above and below diagonal are
made zero at the same time
– Diagonal elements are made ones at the same
time, resulting in the identity matrix
– The column of right-hand side is the solution
Gauss-Jordan method
0 2 0 1 0  1 0 0 0  0.5 
2 2 3 2  2  0 1 0 0 1.0001
  
4  3 0 1  7 0 0 1 0 0.3333
   
6 1  6  5 6  0 0 0 1 2 

Gauss-Jordan method
0 2 0 1 0 1 0.1667  1  0.8333 1 
2 2 3 2  2  row 1 row 4 0 1.6667 5 3.3667  4 
   
4  3 0 
1  7 divide first row by 6 0  3.6667 4 4.3334  11
  Reduce the 1st column  
6 1  6  5 6  0 2 0 1 0 
1 0  0.8182  0.6364 0.5 Divide 3rd row by 6.8182 1 1 0 0.04  0.58

row 2  row 3 0  and reduce other elements
1  1.0909  1.1818 3  3rd column
in 0 1 0  0.280 1.56 
 

Divide new 2nd row by -3.6667 0 0 6.8182 5.6364  9

0 0 1 0  1.32 
Reduce 2nd column below and    
above the diagonal
0 0 2.1818 3.3636  6 0 0 0 1.5599  3.12
Divide 4th row by 1.5599 1 1 0 0  0.5 

and reduce other elements
in 4th column 0 1 0 0 1.0001
 
0 0 1 0 0.3333 
 
0 0 0 1 2 

Gauss-Jordan method
• The solution computed by Gauss-Jordan
method differs slightly from that obtained
with G.E.
– Round-off errors have been entered in a
different way
• Gauss-Jordan method requires almost 50%
more operations than G.E.
– (n^2-n)/2 divisions
– (n^3-n)/2 multiplications
– (n^3-n)/2 subtractions
– Total: n^3 + n^2 - 2n ~ O(n^3), compared to
2n^3/3 + n^2/2 - 7n/6 ~ O(2n^3/3) for G.E.
Scaled partial pivoting
• Partial pivoting without scaling
– When some rows have coefficients that are very
large in comparison to those in other rows,
partial pivoting may not give a correct solution
• Quantities of variables maybe in widely different units
• Scaled partial pivoting
– Scale each row by its coefficient of largest
magnitude first then solve it using G.E.
– A better way – uses the original eqs.,
eliminating the rounding off that may occur in
scaling eaus.
• Use scaling vector whose elements are elements in
each row of largest magnitude
3 2 100  x1  105
 1  3 100  x   102 Scaled partial pivoting :
  2   
 1 2  1   x3   2  Scale each row first with the
with correct solution largest magnitude
x  1.00, 1.00, 1.00 .  0.03 0.01 1.00 1.05
T
 0.01 0.03 1.00 1.02

 
No need for pivoting.  0.5 1.00  0.50 1.00
Triangulat e with 3 digits of precision : Now we need interchang e
3 2 100 105  row 1 with row 3.
0 3.67 133 135 
  * May introduce round-off errors
0 0  82.4  82.6
during scaling.
x  0.939, 1.09, 1.00 (is erroneous).
T

Using scaling vector Reducing the 1st column :
1 2  1 2 
3 2 100  x1  105
0 5 99 104 Do G.E. on original matrix,
 1  3 100  x   102  
w/o scaling!
  2    0  4 103 99 
 1 2  1   x3   2 
We divide each element in 2nd column
Scaling vector S  100, 100, 2 .
T
by corresponding element in S to get
Divide each element in first column R  1, 0.0500, - 0.0400 and no
T
by corresponding element in S to get row intercgang is needed.

R  0.0300,  0.0100, 0.500
T
Reducing the 2nd column :
and row 1 and 3 should be intercgang ed. 1 2  1 2 
0 5 99 104
Interchange elements of S to get  
0 0 182 182
S  2, 100, 100
T
Solution x  1.00, 1.00, 1.00

Use R to check pivoting!

Using order vector
For column 2 :
4  3 0  7
2 2 3  2
A(1,2) should be next pivot.
  Exchange elements in O
6 1  6 6 
to get O  3, 1, 2 ; row 1 as
T
Order vect or O  1, 2, 3 .

T
the next pivot row.

For column 1 : A(3,1) should be
Reducing column 2 :
the pivot; exchange elements in O
(0.6667)  3.667 4  11
to get O  3, 2, 1 . In reducing
T
(0.3333) (0.4545) 6.8182  9 
column 1, we use row 3 as pivot  
 6 1 6 6 
row to get
Back-substituti on in the order given by
(0.6667)  3.667 4  11
 (0.3333) 1.667 5 4
the final order vector [3, 1, 2] :
  first 2, then 1, finally 3
 6 1  6 6 
Multiple right-hand sides
• When all the right-hand sides are known,
we can augment A with those right-hand
sides and do the G.E: [A|b1,b2,…,bm]
• When the right-hand sides are not known
in advance
– Suppose we have solved Ax=b by G.E.
• We have A=LU ready
– For a new right-hand side b
LUx=b <-> Ly=b and Ux=y
• Forward substitution, followed by backward
substitution, requiring 2*n^2 operation count.

Tridiagonal systems
• Only those elements on the diagonal and
adjacent to the diagonal are nonzero
 4 2 0 0
 6 3 1 0

0 7 2 5
 
0 0 8 1
• In GE, only the nonzero elements are used.

– There is no need to store the zeros
– Coefficient matrix can be compressed into an
array of 3 columns
• The right-hand side into a 4th column
– Arithmetic count is reduced significantly
Inverse of matrices
• Division by a matrix is not defined
– The equivalent is to find the inverse of a matrix
• If A*B=I, B is said to be the inverse of A (and A is the
inverse of B)
• Finding the inverse of a matrix
– Using determinant
• Not efficient
– Use Gauss-Jordan form on [A, I]
• More expensive than using GE
– Use Gaussian elimination
• Apply GE on [A,I], or
• Apply A=LU to solving AX=I, i.e., Ax=ei, i=1, 2, …,n

Pathological systems
• Questions can be asked
– Does every square matrix have an inverse?
– Is there unique solution to the set of equation?
• So far we know that if there is zero on the
diagonal after elimination, then no unique
solution can be found for that system
It means that we can not solve the
 1 2 3 
A  2  1
system and can not find the inverse
4
 
 1  14 11  of A. In this case, we said the matrix
LU is A is singular.
 2.0 4.0  1.0 When A is singular, A does not have
( 0.5)  12.0 10.5  an inverse and Ax  b may have no
 
 ( 0 .5) ( 0 .333) 0  solution or infinitely many solutions,
depending on b.
Pathological systems
• Can we see if a matrix A is singular without
trying to triangulate it?
– A singular matrix has a determinant 0
• The matrix A on last slide has a zero on U, so det(A)=0
– The rank of the matrix is less than n, the
number of rows
– A singular matrix has rows that are linearly
dependent vectors (redundant system)
• Ex. For matrix A: -3 row1 + 2 row2 + row3=[0,0,0]
– A singular matrix has columns that are linearly
dependent vectors (redundant system)
• Ex. For matrix A: -10 col1 + 7 col2 + 8 col3 =[0,0,0]
– Ax=b has no unique solution (no sol. or inf. many sol)
Redundant and inconsistent
systems
• Even though a coefficient matrix A is
singular, Ax=b may have no solution or
infinitely many solutions for some b.
 1 2 3  5  Setting x3  1, we have [3,1 / 2,1].
A   2 4  1, b  7 We actually have infinitely many
 1  14 11  1
solutions! !
Apply GE to [ A,b], we have
Let x3  c, we can represent x1 and
 2.0 4.0  1.0 7.0
(0.5)  12.0 10.5 4.5 x2 as functiona of c, so the solution
 
 (0.5) (0.333) 0 0  space will be of dimension 1.
We find that x3 can be any The system is redundant! i.e., any one
value.Setting x3  0, we have equation can be a linear combinatio n
[17 / 4,3 / 8,0]. of the other two.
Redundant and inconsistent
systems
• Redundant system (for b=[5, 7, 1])
– The system has an infinitely many solution
– Dimension of the solution space of AX=b is 1
• Any one equation is a linear combination of other
equations, so there is a free variable x3
3[1,  2, 3, 5]  2[2, 4,  1, 7]  1[1,  14, 11, 1]
• Inconsistent system (for b=[5, 7, 2])

– No solution satisfies the equations.
• U(3, 3)=0, but b’(3)=-0.3333  2.0 4.0  1.0 7.0 
( 0.5)  12.0 10.5 5.5 
 
 (0.5) (0.333) 0  0.3333
Singular vs. nonsingular matrices
• Singular matrix A • Nonsingular matrix A
– It has no inverse – It has an inverse
– Its determinant is zero – Its determinant is
– There is no unique nonzero
solution to the system – There is a unique
Ax=b solution to the system
– Gaussian elimination Ax=b
cannot avoid a zero on – Gaussian elimination
the diagonal does not encounter a
– The rank is less than n zero on the diagonal
– Rows/columns are – The rank equals n
linearly dependent – Rows/columns are
linearly independent
Ill-conditioned systems
• Nearly singular coefficient matrix A
– It is nonsingular, but its U matrix has near-zero
elements on diagonal, and its determinant is close
to 0
• A system whose coefficient matrix is nearly
singular is called ill-conditioned systems
– Solutions to Ax=b is sensitive to the changes in
the elements of b and/or A
• That is, for small changes in the input (i.e., elements of b
or A), we get large changes in the solution
– This phenomenon shows up even more pointedly
in large systems
• Even the 2x2 system shows the effect of near singularity!
U has close to 0 diagonal and
 3.02  1.05 2.53 
inv(A) has very large elements
A   4.33 0.56  1.78 compared to A, so A is nonsingula r
 0.83  0.54 1.47  but is almost singular!
Consider b  [1.61, 7.23,  3.38]T ,
 4.33 0.56  1.78  exact solution to Ax  b is
LU   (0.6975)  1.4406 3.7715 
x  [1.0000, 2.0000,  1.0000]T .
(0.1917 ) (0.3003)  0.0039 
See very small element in U [3,3].
Let' s make a small change in just
 5.6611  7.2732  18.5503  the 1st element of b :
A1  200.5046  268.2570  669.9143 b1  [ 1.60, 7.23,  3.38]T ,
 76.8511  102.6500  255.8846 
Solution to Ax  b1 is
has elements very large in comparison to A.
x1  [1.0566, 4.0051,  0.2315]T .
What a difference !
Let' s consider another small change to b Note that we cannot test for the
b2  [1.61, 7.22,  3.38]T , accuracy of the computed solution
Solution x2  [1.07271, 4.6826, 0.0265]T , merely by substituti ng it into the
which differs much from the true solution! equation to see if the right -hand
sides are reproduced.
When A[1,1] is changed from 3.02 to 3.00,
Solution t o Ax  b is Ax  b
x  [1.1277, 6.5221, 0.7333]T . Ax1  b1  b
The system is also sensitive to the error in Ax2  b2  b
coeeficien t matrix. Check!
Why? See pages on condition number.

• Even the systems of two equation suffer
from this problem….
1.01 0.99  x  2.00 For b2  [1.98, 2.02]T , solution is

0.99 1.01  y   2.00
     x2  [0, 2]T !!
Exact solution : x  [1.00, 1.00]T .
For bˇ  [2.00, 2.00]T , solution is
For b1  [2.02, 1.98]T , solution is x2  [1, 1]T !!
x1  [2, 0]T .

Effect of precision
• What we can do for ill-conditioned systems?
– Do the calculations in higher precision.
 3.02  1.05 2.53  1.61 Use only three digits ：
[ A, b]   4.33 0.56  1.78 7.23  1 0  .073 0
 
 0.83  0.54 1.47  3.38 0 1  2.62 0
is ill -conditione d.
 
0 0 0 1
Using Maple with 10 digits of precision, we get Maple says that the matrix is singular!
x  [1.000000037 , 2.000001339 ,  0.9999994882 ] Actually t he system is inconsiste nt!
which is pretty close to the exact solution
x  [1, 2,  1]. Use only four digits ：
With precision of 20, we get a more accurate

1 0 0 0.9824 
0 1 0 1.346 
solution but it is still not exact.  
0 0 1  1.250
We get a poor solution!
Condition numbers and norms
• The degree of ill-conditioning of a matrix is
measured by its condition number, which is
defined in terms of its norms
– Norm is a measure of the magnitude of the
matrix
• To measure a single number a, we use its distance from
zero: |a|
• For any norm, the following properties are
essential: 1. A  0 and A  0 iff A  0.
2. kA  k A .
3. A  B  A  B . (Triangle inequality )
4. AB  A B .
Vector norms
• Norms of a vector
– Euclidean norm or 2 norm
• Length of the vector
1/ 2
 n
2
x 2    xi 
 i 1 
– 1-norm
• Sum of the absolute values of the elements
n
x 1   xi
i 1
– Maximum norm (or infinite norm)

• The maximum value of the elements
x   max xi
1i  n
Matrix norms
n
A 1  max  aij
1 j  n
i 1
n
A   max  aij
1i  n
j 1
Note that :
A 2  A1
A2 A
Errors in the solution
• How large is the errors in the solution?
– Substituting solution to the equation…
• Not good for ill-conditioned systems – can not tell how
large the error is
– Use norms to define “condition number”
Let x be the computed solution. Hence

Define the residual r  b  Ax e  A1r.
Let e be the error in x , e  x  x. Taking norms, we have
Because Ax  b, we have e  A1r  A1 r .
r  b  Ax  Ax  Ax
 A( x  x )  Ae
Errors in the solution
e  A1r  A1 r . Applying the same reasoning to

Ax  b and x  A1b, we have
Since r  Ae, we have r  A e .
b
So 3/28  x  A1 b .
A
r
 e  A1 r . All together, we have
A
1 r e 1 r
1
  A A .
A A b x b
Condition number
Condition number of a matrix

• For residual
– When the condition number is large, the
residual gives little information about the
accuracy of the solution
– When the condition number is nearly unity, the
residual is a good measure of the accuracy of
the solution


• Errors in the coefficient matrix
So x  x  A1Ex , and we have
Let A be the true coefficien t matrix,
x  x  A1Ex  A1 E x
E be the errors. Let A  A  E.
Let x be the computed solution t o E
 A1 A x,
A x  ( A  E ) x  b, A
and x be the solution of Ax  b, and
we have xx E E
 A1 A  Condition no.
x  A1b  A1 ( A x ) x A A
 A1 ( A  A  A) x
Relative error of the solution relative
 
 I  A1 ( A  A) x
to the computed solution
 x  A1 ( A  A) x can be as large as the relative error
 x  A1 Ex. in A multiplied by the condition number.

• Errors in right-handed side vector
Let b be the perturbed vector of b. So

If x and x satisfy Ax  b and Ax  b. xx 1
bb
 A A
x  x  A1b  A1 b  A1 b  b   x b
bb
1 1
bb  condition no.
 A bb  A Ax b
b
bb Relative error of the solution
1
 A A x relative to the true solution
b can be as large as the relative
error in b multiplied by the
condition number.

Iterative improvement
Residual correction method
Let x be the computed solution

Procedure :
of Ax  b.
1. Decompose A to LU , and
Define e  x  x and r  b  Ax.
apply LU to solve Ax  b.
Since
2. Compute r  b  Ax and
Ae  r ,
solve Ae  r using A  LU .
we can solve this equation for e, and 3. Get an improved estimate
apply the computed solution e as a of x from x  e .
correction to x , i.e., x  x  e . 4. Repeat step 2, 3 until desired
Repeat this correction until desired
accuracy is reached.
accuracy is achieved.
Solution of Ae  r is also subject to Note :

round-off error as the solution of The computatio n of r
Ax  b. must be as accurate as
Even so, unless the system is so possible. So use double -
ill -conditione d that e is not a precision arithmetic .
reasonable approximat e to e,
we will get an improved estimate
of x from x  e .

 4.23  1.06 2.11  Compute r  b  Ax using double
A   2.53 6.77 0.98 , precision :
 1.85  2.11  2.32 Ax  [5.24511, 5.22246,  2.59032]T .
b  5.28 5.22  2.58 r  [0.0349,  0.00246, 0.0103]T .
T
Exact solution :
x  [1.000, 1.000, 1.000]T . We solve Ae  r and get
e  [0.00822, 0.00300,  0.00000757 ]T .
Using 3-digit precision Finally,
Computed solution : x  e  [0.999, 1.000, 1.000]T
x  [0.991, 0.997, 1.000]T . gives almost exactly the correct solution.

Pivoting and precision
• Pivoting can
– Avoids zero diagonal elements for nonsingular
matrix
– Reduces the errors due to round off
• Only if the problem is mildly ill-conditioned
  x  By  C Solving for y, we have


 Dx  Ey  F F  CD /  CD
with  a very small number. y  if  is very small.
E  DB/ DB
C  B(C / B) C  C
x   0 !!
Without pivoting, we get  
 x  By  C showing that x  0 for any value of C
( E  DB/ ) y  F  CD /  and F if  is small enough.
Example
Suppose F  D  E and C    B.
With pivoting, we reduce
 Dx  Ey  D  E So that

  x  By    B ( BD  E ) / D
y  1,
to ( BD  E ) / D
 Dx  Ey  D  E DEE
( B  ( / D) E ) y x 1
 D
   B  ( / D)( D  E )

 D  BD  D  E  BD  E
 D D

Example
• For severely ill-conditioned problems,
pivoting alone cannot save the accuracy
– The best way to remedy is to increase precision
of computation

Iterative methods
• Direct methods • Iterative methods
– Gaussian – Jacobi method
elimination – Gauss-Seidel method
– Gaussian-Jordan – Preferred over G.E. for large
– Decomposition sparse coefficient matrix
methods – Will converge for any
• LU, QR, SVD starting values if the
coefficient matrix is
diagonally dominant, i.e.,
n
aii   aij , for each i  1,2,..., n.
j 1
j i
Iterative methods
Jacobi method
6 x1  2 x2  x3  11 Starting from some initial approximation,

 x1  2 x2  5 x3  1 we iterate based on
 2 x  7 x  2 x  5
 1 2 3
x1( n 1)  1.8333  0.3333 x (2n )  0.1667 x (3n )
Solution : x1  2, x2  x3  1.
Rewrite the system :
x (2n 1)  0.7143  0.2857 x1( n )  0.2857 x (3n )
6 x1  2 x2  x3  11 x (3n 1)  0.2000  0.2000 x1( n )  0.4000 x (2n )

 2 x1  7 x2  2 x3  5
 x  2 x  5 x  1
 1 2 3 Starting with x ( 0)  (0, 0, 0), at 9th
Rearrange the equations :
iteration we get x (9)  (2.00, 1.00, 1.00).
x1  1.8333  0.3333 x2  0.1667 x3
x2  0.7143  0.2857 x1  0.2857 x3
x3  0.2000  0.2000 x1  0.4000 x2 . diagonally dominant
Iterative methods
Jacobi method
General form for rearrangement : Iterative form :

x ( n 1)   D 1 ( L  U ) x ( n )  D 1b
n a
bi
xi    ij x j , i  1,2,..., n.
aii j 1 aii
j i  b' Bx ( n )
Matrix representation : x ( n 1)  G ( x ( n ) )  b' Bx ( n )

Let A  L  D  U .
Ax  ( L  D  U ) x  b
The same as fixed -point iteration!
Dx  ( L  U ) x  b
x   D 1 ( L  U ) x  D 1b

Iterative methods
Jacobi method
Ax  b, Iterative form :
 6 2 1   x1  11  x ( n 1)   D 1 ( L  U ) x ( n )  D 1b

 2 7 2   x2    5   G ( x ( n ) )  b' Bx ( n )

 1 2  5  x2   1
1.8333 
0 0 0 6 0 0 
b'  D 1b  0.7143
L   2
0 0, D  0 7 0 ,
0.2000 
 1
2 0 0 0  5
0  2 1 
B  D 1 ( L  U )
U  0 0 2
 0  0.3333 0.1667 
0 0 0
  0.2857 0 0.2857 
 0.2000  0.4000 0 
Iterative methods
Gauss-Seidel method
• Similar to Jacobi method, but use always
the most recent approximations of the
other variables
• The rate of convergence is more rapid than
for the Jacobi method
Starting from some initial approximat ion,
Starting with x ( 0)  (0, 0, 0), at 6th
we iterate based on
( k 1) iteration we get x ( 6)  (2.00, 1.00, 1.00).
x1  1.8333  0.3333 x 2  0.1667 x 3
(k ) (k )
x (2k 1)  0.7143  0.2857 x1( k 1)  0.2857 x (3k )

x (3k 1)  0.2000  0.2000 x1( k 1)  0.4000 x (2k 1)

Iterative methods
Gauss-Seidel method
• Matrix
representation
Matrix representation :
Let A  L  D  U .
Ax  ( L  D  U ) x  b
( L  D) x  Ux  b
x  ( L  D) 1Ux  ( L  D) 1 b
Iterative form :
x ( k 1)  ( L  D) 1Ux ( k )  ( L  D) 1 b

Iterative methods
Convergence issues
• If the coefficient matrix is diagonally
dominant, Jacobi and Gauss-Seidel
converge for any initial values
• When both methods converge, Gauss-
Seidel converges faster
• Without diagonal dominance, neither
Jacobi nor Gauss-Seidel is sure to converge
– There are some instances where the coefficient
matrix does not have diagonal dominance but
still both Jacobi and Gauss-Seidel do converge
– It can be shown that, if the coefficient matrix A
is symmetric and positive definite, the Gauss-
Seidel method will converge from any starting
values
Iterative methods
Convergence issues
• Most coefficient matrices are neither
diagonally dominant nor symmetric and
positive definite, it is suggested to solve as
many equations for the variable having the
largest coefficient!

Iterative methods
Convergence proof
Assume that A is diagonally dominant. Then the error of
Let x  ( x1 , x2 ,.., xn ) be the exact 1  
bi   aij x j 
( k 1)
xi  (k )
solution t o Ax  b. Then aii  j i 

satisfies
1  
xi  bi   aij x j ,  ik 1  xi  xi( k 1)
aii  j i 
1  
for i  1, 2,.., n. 
aii  j i

 aij x j  x j 
(k )


Let the error of the i-th component
1  
  aij  j 
k
of x ( k ) be
aii  j i 
 ik  xi  xi( k ) , for i  1, 2,.., n.

Iterative methods
Convergence issues
So, if  i
k
max
denotes the largest where
 kj for j  i, then   aij 
 j i 
  max  .
1
 ij j
a  k
i 1, 2 ,.., n
 aii 
 ik 1 
aii
 ij j 
a  k
j i
j i
aii
 
This implies that  ik 1 is smaller
a ij
than  i by a factor of at least  .
j i k
  i max    i max
k k
max
aii
The convergenc e will therefore be
ensured if   1,
i.e., if A is diagonally dominant.

Iterative methods
Accelerating convergence
• Convergence in the Gauss-Seidel method is still
slow. It can be speeded if we do overrelaxation
Gauss -Seidel iteration : Overrelaxation can be applied to
1  i 1 n  Gauss -Seidel method if we add
bi   aij x j   aij x j 
( k 1) ( k 1)
xi  (k )
to some multiple of the second
aii  j 1 j i 1 
term：
Equivalent ly, xi( k 1)  xi( k )
xi( k 1)  xi( k )  w  i 1 n 
( k 1)
 (k )
1  i 1 n  aii  j 1 j i 
( k 1) (k )
aii  j 1 j i 
w : the overrelaxation factor,
1  w  2.
Iterative methods
Accelerating convergence
 4 1 1 1 1
 1 4 1 1  1
 x 
1 1 4 1  1
  
 1 1 1  4  1
The exact solution is (-1, -1, -1, -1)
When w=1.3 the number of required iterations is minimum!

Iterative methods
Iteration is minimizing
• Getting successive improvements that
converges to the solution to Ax=b can be
considered to be minimizing the residual
error r(x)=b-Ax
• If A is symmetric and positive definite,
conjugate gradient method gives extremely
rapidly convergence
– It always converges in n tries with system of n
equations
– Each iteration of the conjugate gradient is more
expensive than Jacobi or Gauss-Seidel

Iterative methods
Iteration is minimizing
 4  3  1  x1   7  Start with x0  [0,0,0]T ,
 3 5 2 x    2  Gauss -Seidel converges in 20
  2   
  1 2 3   x2   3 iterations , Jacobi fails to converge.
Exact solution :
[3.9167, 3.5833,  2.0833]T If conjugate gradient is applied
with x0 , it converges in three tries :
The coefficien t matrix is x0  [0,0,0]T ,
symmetric and positive x1  [2.4520, 0.7006,  1.0508]T
definite. x2  [4.0670, 3.4771,  1.6197 ]T
x3  [3.9167, 3.5833,  2.0833]T

2-Linear System PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2-Linear System PDF

Uploaded by

Copyright:

Available Formats

Chap 2: Solving sets of Equations

• Metrics and vectors

Numerical Method, Chap 2 2022 1 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 2 Jung Hong Chuang, CS Dept., NCTU

Row interchanged Column interchanged

Numerical Method, Chap 2 2022 3 Jung Hong Chuang, CS Dept., NCTU

• Trace of a square matrix

Numerical Method, Chap 2 2022 5 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 6 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 7 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 8 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 9 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 11 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 12 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 15 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 17 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 18 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 19 Jung Hong Chuang, CS Dept., NCTU

Back-substituti on order :1, 2, 3, 4

Numerical Method, Chap 2 2022 21 Jung Hong Chuang, CS Dept., NCTU

To reduce the elements below Total :

Substracti ons  (n  i  1)(n  i )

Numerical Method, Chap 2 2022 24 Jung Hong Chuang, CS Dept., NCTU

1 0  0.8182  0.6364 0.5 Divide 3rd row by 6.8182 1 1 0 0.04  0.58

Divide 4th row by 1.5599 1 1 0 0  0.5 

Numerical Method, Chap 2 2022 25 Jung Hong Chuang, CS Dept., NCTU

 0.01 0.03 1.00 1.02

Numerical Method, Chap 2 2022 28 Jung Hong Chuang, CS Dept., NCTU

by corresponding element in S to get row intercgang is needed.

Solution x  1.00, 1.00, 1.00

Numerical Method, Chap 2 2022 29 Jung Hong Chuang, CS Dept., NCTU

Order vect or O  1, 2, 3 .

the next pivot row.

Numerical Method, Chap 2 2022 31 Jung Hong Chuang, CS Dept., NCTU

• In GE, only the nonzero elements are used.

Numerical Method, Chap 2 2022 33 Jung Hong Chuang, CS Dept., NCTU

3[1,  2, 3, 5]  2[2, 4,  1, 7]  1[1,  14, 11, 1]

• Inconsistent system (for b=[5, 7, 2])

Numerical Method, Chap 2 2022 41 Jung Hong Chuang, CS Dept., NCTU

1.01 0.99  x  2.00 For b2  [1.98, 2.02]T , solution is

Numerical Method, Chap 2 2022 42 Jung Hong Chuang, CS Dept., NCTU

With precision of 20, we get a more accurate

– Maximum norm (or infinite norm)

Let x be the computed solution. Hence

e  A1r  A1 r . Applying the same reasoning to

Numerical Method, Chap 2 2022 49 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 50 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 51 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 52 Jung Hong Chuang, CS Dept., NCTU

Let b be the perturbed vector of b. So

Numerical Method, Chap 2 2022 53 Jung Hong Chuang, CS Dept., NCTU

Let x be the computed solution

Solution of Ae  r is also subject to Note :

Numerical Method, Chap 2 2022 55 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 56 Jung Hong Chuang, CS Dept., NCTU

  x  By  C Solving for y, we have

Numerical Method, Chap 2 2022 58 Jung Hong Chuang, CS Dept., NCTU

Numerical Method, Chap 2 2022 59 Jung Hong Chuang, CS Dept., NCTU

General form for rearrangement : Iterative form :

Matrix representation : x ( n 1)  G ( x ( n ) )  b' Bx ( n )

Numerical Method, Chap 2 2022 62 Jung Hong Chuang, CS Dept., NCTU

 6 2 1   x1  11  x ( n 1)   D 1 ( L  U ) x ( n )  D 1b

x (2k 1)  0.7143  0.2857 x1( k 1)  0.2857 x (3k )