Solution of Nonlinear System of Equations

• Now, we are searching for the values x1, x2,…,xn that

simultaneously satisfy n number of non-linear equations;
• 𝑓 𝑥 ,𝑥 ,...,𝑥 =0
• 𝑓 𝑥 ,𝑥 ,...,𝑥 =0
• .
• .
• .
• 𝑓 𝑥 ,𝑥 ,...,𝑥 =0
• One way of solving this system is based on a multi-dimensional
version of the Newton-Raphson method.

1
Solution of Nonlinear System of Equations
• Recall we have obtained formulation of Newton-Raphson method
in “lesson 3”:
•

2
Solution of Nonlinear System of Equations
• This formulation of Newton-Raphson method can also be derived
by the first order Taylor’s series expansion of f around x=xi:
•

• Setting x=xi+1 in the equation above we

have:
• 𝑓 𝑥 ≅𝑓 𝑥 +𝑓 𝑥 𝑥 −𝑥 =0
• Solving for :
• 𝑥 =𝑥 −
• So we have arrived at the same result!
3
Nonlinear System of Equations
• Recall that the Taylor’s series expansion of the function f(x,y) about x0 and
y0 is:
, ,
•
• Similarly, the Taylor series expansion of k’th equation considering 1st order
terms only, gives:
, , ,
• , , , , , , , ,
• Where
• , , , , and , , , ,
• The expansion is at ( , , , ,…, , )
• And we are estimating function value at ( , , , ,…, , )

4
Nonlinear System of Equations, cont.
• First subscript (k) represents the equation and the second subscript represents
whether the function f is at present value (i) or at the next value (i+1).
• Setting all , terms equal to zero as it was done in one dimensional Newton-
Raphson method, we have:
, , ,
• 𝑥 , +𝑥 , +. . . +𝑥 , =
𝜕𝑓 , 𝜕𝑓 , 𝜕𝑓 ,
−𝑓 +𝑥 , +𝑥 +. . . +𝑥
 f1,i f1,i f1,i 
, , ,
𝜕𝑥 𝜕𝑥 𝜕𝑥
 x 
• This equation for k=1,2,…,n can be written in matrix
 1 x2 xn 
notation:  f2,i f2,i f2,i 

 Z    x1 
• 𝑍 𝑋 =− 𝐹 + 𝑍 𝑋
x2 xn 
• where,
• 𝑋 = 𝑥 , ,𝑥 , ,...,𝑥 , 𝑋 = 𝑥 ,𝑥 ,...,𝑥 and
     
, , , ,  
• 𝐹 = 𝑓 , ,𝑓 , ,...,𝑓 ,  fn ,i fn ,i

fn ,i 
 x x2 xn 
 1
5
Example 1
• Solve the following system of nonlinear equations with 5 significant
figures:
• root
•
• Solution
• Let’s find the [Z] matrix:

 f1,i f1,i 
 x x2  2x1,i  2x2,i 2x1,i  5
 Z    f 1 
f2,i   2 3x2,2 i 
2,i
 
 x1 x2 
6
Example 1, cont.

 Z  Xi 1  Fi    Z  Xi    Xi 1   Z  Fi    Xi 

1

• Since [Z] is a 2x2 matrix, we can find its inverse easily (For higher
order systems, Gauss elimination is used).

 x1,i 1  1 3 x2,2 i 5  2 x1,i   x1,2 i  2 x1,i x2,i  5 x2,i  3  x1,i 

  3     
x
 2,i 1  10  4 x1  6 x x 2
1 ,i 2 , i  6 x 2 , i  2 2 x1,i  2 x2,i   2 x1,i  x23,i  1   x2,i 

7
Nonlinear System of Equations, cont.

• For 5 significant figures, we need εs=0.5x10-3%

• Starting with x1,0=x2,0=0, we calculate:

8
LU Decomposition and Matrix Inversion
• Gauss elimination solves the linear algebraic equation system: [A]{X}={B}.
We need to solve the equation system over and over every time {B} changes.
• e.g., finite element discretization of a continuous structure results in a linear algebraic
equation system [A]{X}={B} for linear static analyses. [A] is called the stiffness matrix,
{X} is the nodal displacements and {B} is the force vector.
• For a given structure [A] will be constant.
• Every time the force vector changes (External point, surface and/or inertia loads) we
need to solve the system to obtain {X}.
• LU decomposition involves operations on the coefficient matrix [A],
therefore it is suitable where many right-hand vectors {B} need to be
evaluated.
• LU decomposition also provides an efficient means to compute matrix
inverse.
9
LU Decomposition
• Let’s consider;
•
• This equation can be written as:
•
• We can apply forward elimination to this system to have:
•
• For a 3x3 system

• ,

10
LU Decomposition, cont.
• Now consider a lower triangular matrix with 1’s on the main diagonal:
• , lii=1 and lij=0 if i<j

• which has the following property:

•
• For this equation to hold, we need:
•

• First, [A] is decomposed into [L] and [U].

• By using the equation [L]{D}={B} we determine {D}. Then, we solve {X} from
[U]{X}={D}.
11
LU Decomposition by Gauss Elimination
• Gauss elimination can be used to obtain the LU decomposition of a
matrix [A]. One can obtain [L] and [U] matrices during the forward
elimination process of Gauss elimination. Let’s consider a 3x3 system:

• Multiply first row by and then subtract from the

second and third rows, respectively.

12
LU Decomposition by Gauss Elimination, cont.
• Then multiply second row by and subtract from the third
row to obtain the [U] matrix:

• ,

• and the [L] matrix is simply:

13
LU Decomposition by Gauss Elimination, cont.
• We can prove [A]=[L][U] for the 3x3 system:

1 0 0 𝑎 𝑎 𝑎
• 𝐿 𝑈 = 𝑓 1 0 0 𝑎 𝑎
𝑓 𝑓 1 0 0 𝑎

𝑎 𝑎 𝑎
• = 𝑎 𝑓 𝑎 𝑓 +𝑎 𝑎 𝑓 +𝑎
𝑎 𝑓 𝑎 𝑓 +𝑎 𝑓 𝑎 𝑓 +𝑎 𝑓 +𝑎

• Using the modified coefficients obtained from the forward elimination:

• ,
• ,
14
LU Decomposition by Gauss Elimination, cont.
• We have:

• So we have proved that [A]=[L][U] for a 3x3 system. One can extend
this proof for an nxn system.

15
Example 10.1
• Obtain the LU decomposition of the coefficient matrix: 3 −0.1 −0.2
3 −0.1 −0.2 • 𝑈 = 0 7.00333 −0.293333
• 0.1 7 −0.3 0 0 10.0120
0.3 −0.2 10 • and
1 0 0
• 𝐿 = 0.0333333 1 0
• Solution 0.100000 −0.0271300 1
• Forward elimination: • We can verify the calculations:
. .
• 𝑓 = = 0.0333333,𝑓 = = 0.100000 • 𝐿 𝑈 =
3 −0.1 −0.2
• 0 7.00333 −0.293333 1 0 0 3 −0.1 −0.2
0 −0.190000 10.0200 • 0.0333333 1 0 0 7.00333 −0.293333
0.100000 −0.0271300 1 0 0 10.0120
• Forward elimination:
.
• 𝑓 =− = −0.0271300 3 −0.1 −0.2
.
• = 0.0999999 7 −0.3
3 −0.1 −0.2
0.3 −0.2 9.99996
• 0 7.00333 −0.293333
0 0 10.0120
• Omitting the round-off errors we have verified that [A]=[L][U].

• Finally, we have: 16
Example 10.2
• Solve • Forward substitution:
3 −0.1 −0.2 𝑥 7.85 1 0 0 𝑑 7.85
• 0.1 7 −0.3 𝑥 = −19.3 • 0.0333333 1 0 𝑑 = −19.3
0.3 −0.2 10 𝑥 71.4 0.100000 −0.0271300 1 𝑑 71.4
• By the LU decomposition method.
• 𝑑 = 7.85
• Solution • 𝑑 = −19.3 − 0.0333333 7.85 = −19.5617
• In LU decomposition we have: • 𝑑 = 71.4 − 0.100000 7.85 −
(−0.0271300) −19.5617 = 70.0843
• [L]{D}={B} (1)
• [U]{X}={D} (2) • 𝐷 = 7.85 −19.5617 70.0843

• [L] and [U] were calculated in previous

example. First we solve for {D} from Eq.(1) and
then we use {D} to obtain {X} from the second
equation.

17
Example 10.2, cont.
• Back substitution:

•
•
.
•
•

18
LU Decomposition by Gauss Elimination, cont.
• For an nxn system (after calculating the [L] and [U] matrices):

• Forward substitution: , i=1,2,…,n ([L]{D}={B})

• Back substitution: , i=n,n-1,…,1 ([U]{X}={D})

19
LU Decomposition by Gauss Elimination
with Partial Pivoting
• If the pivoting term is zero, we get division by zero error during the forward elimination.
To avoid this, we will consider LU decomposition with partial pivoting.
• Consider the following system:
1 5 2 𝑥 1
• 2 3 5 𝑥 = 2
5 0 4 𝑥 0
• Let’s change rows 1 and 3 to have the largest term as the pivoting element:
5 0 4 𝑥 0
• 2 3 5 𝑥 = 2
1 5 2 𝑥 1
• This operation is identical with multiplying both sides with the following permutation
matrix (obtained with changing rows 1 and 3 of the identity matrix):
0 0 1 1 5 2 𝑥 0 0 1 1 5 0 4 𝑥 0
• 0 1 0 2 3 5 𝑥 = 0 1 0 2 → 2 3 5 𝑥 = 2 or 𝑃 𝐴 𝑋 = 𝑃 𝐵
1 0 0 5 0 4 𝑥 1 0 0 0 1 5 2 𝑥 1
20
LU Decomposition by Gauss Elimination
with Partial Pivoting, cont.
 5 0 4   0 0 1  1 5 2 
2 3 5    0 1 0  2 3 5 
    
1 5 2  1 0 0   5 0 4 
 
P 1  A
• Let’s continue with the forward elimination:

 1 0 0  5 0 4  5 0 4   1 0 0   0 0 1  1 5 2 
 0.4 1 0  2 3 5    0 3 3.4    0.4 1 0  0 1 0  2 3 5 
        
 0.2 0 1  1 5 2  0 5 1.2   0.2 0 1  1 0 0  5 0 4 
   
L1 P 1  A

Forward elimination of x1 from the 2nd and 3rd equations is identical with multiplication with the
lower triangular matrix [L]1.
21
LU Decomposition by Gauss Elimination
with Partial Pivoting, cont.
• Let’s swap second and third rows (partial pivoting):
 1 0 0   5 0 4   5 0 4  1 0 0   1 0 0   0 0 1  1 5 2 
 0 0 1  0 3 3.4   0 5 1.2   0 0 1   0.4 1 0  0 1 0  2 3 5 
         
 0 1 0  0 5 1.2  0 3 3.4  0 1 0   0.2 0 1  1 0 0  5 0 4 
    
P 2 L1 P 1  A

• Eliminate x2 from the third equation:

 5 0 4  1 0 0 5 0 4  1 0 0  1 0 0   1 0 0   0 0 1  1 5 2 
 0 5 1.2  0 1 0   0 5 1.2   0 1 0  0 0 1   0.4 1 0   0 1 0  2 3 5 
          
 0 3 3.4  0 0.6 1  0 0 2.68   0 0.6 1   0 1 0   0.2 0 1  1 0 0  5 0 4 
         
U  L2 P 2 L1 P 1  A

• So we have obtained the [U] matrix.

22
 LU Decomposition by Gauss Elimination
with Partial Pivoting, cont.
•

• where,
• and

23
 LU Decomposition by Gauss Elimination
with Partial Pivoting, cont.
• In general, for an nxn system, we have:
• 𝐿 = 𝑃 𝐿 𝑃 ... 𝐿 𝑃 = 𝑃 𝑃 𝐿 ... 𝑃 𝐿
• and
• 𝑃 = 𝑃 𝑃 ... 𝑃
• The [L] and [P] matrices have the following property:
𝑖≤𝑗 𝑙
• If 𝐿 = 𝐿 then 𝑙 =
𝑖>𝑗 −𝑙

• and 𝑃 = 𝑃

• As an example: and

24
Matrix Inverse by the LU Decomposition
• The matrix inverse is defined as:
•
• Where [A] is an nxn square matrix and [I] is the nxn identity matrix.
• From LU decomposition we have:
•
• Multiply each side by [A]-1:
•
• Say the elements of [A]-1=xij, then, we can calculate the columns of [A]-1 as
follows:
• , ,…,

25
Example 10.3
• Employ LU decomposition to calculate the inverse of:
•

• Solution
• We have calculated [L] and [U] in previous example.
• and

26
Example 10.3, cont.
• Let’s calculate the first column of [A]-1:
𝑥 1
• 𝐿 𝑈 𝑥 = 0 • Back substitution:
𝑥 0
3 −0.1 −0.2 𝑥
• [L]{D}={B} (1) • 0 7.00333 −0.293333 𝑥 =
• [U]{X}={D} (2) 0 0 10.0120 𝑥
1
• Forward substitution: −0.0333333
−0.100904
1 0 0 𝑑 1 𝑥 0.332489
• 0.0333333 1 0 𝑑 = 0 • 𝑥 = −0.00518176
0.100000 −0.0271300 1 𝑑 0 𝑥 −0.0100783

• 𝐷 = 1 −0.0333333 −0.100904

27
Example 10.3, cont.
• Let’s calculate the second column of [A]-1:
𝑥 0 0
• 𝐿 𝑈 𝑥 = 1 , 𝐿 𝐷 = 1 → 𝐷 = 0 1 0.0271300
𝑥 0 0
𝑥 𝑥 0.00494407
• 𝑈 𝑥 = 𝐷 → 𝑥 = 0.142903
𝑥 𝑥 0.00270975
• Let’s calculate the third column of [A]-1:
𝑥 0 0
• 𝐿 𝑈 𝑥 = 0, 𝐿 𝐷 = 0 → 𝐷 = 0 0 1
𝑥 1 1
𝑥 𝑥 0.00679812
• 𝑈 𝑥 = 𝐷 → 𝑥 = 0.00418346
𝑥 𝑥 0.0998801
28
Matrix Inverse by the LU Decomposition,
cont.
• Combining the calculated columns of [A]-1, we have:
0.332489 0.00494407 0.00679812
• 𝐴 = −0.00518176 0.142903 0.00418346
−0.0100783 0.00270975 0.0998801

• We can verify if [A][A]-1=[I] holds to check if the calculated inverse is correct.

• For a given right-hand vector {B} we can solve the vector of unknowns {X} as follows:
• 𝐴 𝑋 = 𝐵 → 𝑋 = 𝐴 𝐵
• Calculating [A]-1 is expensive, however, if we are to solve {X} for many different {B}’s with
the coefficient matrix [A] fixed, it is feasible.
• Matrix inverse gives information on the condition of the system, i.e., if the system is well-
conditioned, ill-conditioned or singular. If the system is singular, the inverse does not
exist. If the scaled [A] has an inverse with elements several order magnitude greater than
1, it is an indication that the system is ill-conditioned.
• We can also check if [A][A]-1=[I] to see if the system is ill-conditioned or not.
29
Ill-Conditioned vs. Well Conditioned
• İll-conditioned system: Small changes in
the coefficient matrix results in large  400 201  x1   200 
variation of results.  800 401   x   200   x1  100, x2  200
  2  
• Well-conditioned system: Small changes
in the coefficient matrix results in small  401 201  x1   200 
variation of results.  800 401   x   200   x1  40000, x2  79800
  2  
• If the determinant of the coefficient
matrix is zero, the system is singular
• A small determinant of a scaled 1 201 / 400
coefficient matrix indicates ill-condition.  1 / 400  0.0025
1 401 / 800
• The matrix condition number accurately
describes the condition of a matrix.

30
Mathematical Background
Vector and Matrix Norms
• A norm is a real valued measure of size or length
of multi-component entities such as vectors and
matrices.
• The Euclidian norm of a vector is its length,
which can be calculated as:
•

• The Euclidian norm of a vector can be extended

to a matrix, which is called the Frobenius norm:

31
Vector and Matrix Norms, cont.
• There are alternatives to these norms such as P-norm for vectors:
⁄
•
• For p=2, P-norm is identical with the Euclidian norm.
• Maximum magnitude norm for vectors:
•
• For P-norm and maximum norm are identical.
• Maximum magnitude norm for matrices:
• (Column-sum norm)

• (Row-sum norm)

32
Matrix Condition Number
• The matrix condition number Cond[A] is obtained as:
•
• It can be shown that:
‖ ‖ ‖ ‖
•
‖ ‖ ‖ ‖
• Which means that the relative error of the norm of computed
solution can be as large as the relative error of the norm of coefficient
matrix [A] multiplied with the condition number Cond[A].
• If the coefficients of [A] are known to t digits precision (that is the
rounding errors are of magnitude 10-t) and Cond[A]=10c, then the
solution vector {X} may be valid to only (t-c) digits.

33
Example 10.4
• The Hilbert matrix that is notoriously ill-conditioned can be presented
in the general form:

• or

• Use the row-sum norm to estimate the matrix condition number for
the 3x3 Hilbert matrix.

34
Example 10.4, cont.
• Solution • Row-sum norm:
•
•
• Summing the absolute value of
• Let’s normalize [A] so that the each row element gives:
maximum element in a row is 1 • 1st row: 1+1/2+1/3=1.833
(this is not a requirement but we • 2nd row: 1+2/3+1/2=2.167
want to compare with the • 3rd row: 1+3/4+3/5=2.35
inverse): • The maximum of these values is
the row-sum norm:
• •

35
Example 10.4, cont.
• The inverse of the scaled matrix can be obtained by the LU decomposition as:
9 −18 10
• 𝐴 = −36 96 −60
30 −90 60
• Notice that the elements of [A]-1 are much larger than the elements of scaled [A],
indicating ill-condition.
• The row-sum norm of [A]-1 is:
• 1st row: 9+18+10=37
• 2nd row: 36+96+60=192
• 3rd row: 30+90+60=180
• The maximum of these values is the row-sum norm:
• ‖𝐴 ‖ = 192
• 𝐶𝑜𝑛𝑑 𝐴 = ‖𝐴‖ ‖𝐴 ‖ = 2.35 × 192 = 451.2 ≫ 1 (ill-conditioned, much larger than 1!)
• 10 = 451.2 → 𝑐 = 2.65
• If [A] is calculated/represented with t significant digits, the solution to [A]{x}={B} may be
valid to t-c digits.
36
Example 2
• Calculate the matrix condition number of
•

• using the row-sum norm.

• Normalize [A] so that the maximum element in a row is 1:
• →

• →
• (ill-conditioned)
•

37
Cholesky Decomposition
• We can use this method instead of LU decomposition for the symmetric matrices
since it requires less operations. Symmetric matrices can be decomposed as:
• 𝐴 = 𝐿 𝐿
• where
• 𝑙 = for k=1,2,…,n and i=1,2,…,k-1 (k=1 gives no result since i becomes 0)
• and

• 𝑙 = 𝑎 − 𝑙 for k=1,2,…,n

• The algorithm can result in execution error if the square root of a negative
number is taken in the calculation of lkk. This will never occur if the coefficient
matrix is positive definite i.e. for all nonzero {X} vectors.
• Finite element method always gives positive definite coefficient matrices.
38
Example 11.2
• Apply Cholesky decomposition to •
the symmetric matrix to decompose .
it into LU form: •
•
• For the third row (k=3):
• Solution •
.
• For the first row (k=1, we skip the 1st •
. .
equation): .
•
•
• For the second row (k=2):

39
Cholesky Decomposition, cont.
• Thus, we have:
•

• and

• We can verify that [A]=[L][U] (or [A]=[L][L]T) to check the correctness

of computation.

40
Gauss-Seidel Method
• This is the most commonly used iterative method for the solution of linear
algebraic equation systems.
• Let’s consider a 3x3 equation system:

• If the diagonal elements are all non-zero, we can write:

• (1)
• (2)
• (3)

41
Gauss-Seidel Method, cont.
• The solution starts with an initial guess for the solution vector {X}. A simple
way is to assume they are all equal to zero.
• This initial guess is substituted in Eq. (1) to obtain a new estimate for x1.
• This new value of x1 and initial values for remaining variables are
substituted in Eq. (2) to compute a new x2.
• These two values (and initial values of other variables if any) are used in Eq.
(3) to calculate the new x3. Iterations are continued until:
• , for all i

• In this equation, j and j-1 represent the present and previous iterations
respectively.

42
Example 11.3
• Solve the equation system by the • Let’s start computation assuming
Gauss-Seidel method: x2=x3=0:
. . .
•
•
• With this value of x1 and x3=0:
• Solution •
. . . .
. . .
•
. . .
• • With the calculated x1 and x2:
. . . . . . . .
• •
• First Iteration

43
Example 11.3, cont.
• Second Iteration
. . . . .
• ,
. .
• , .
. . . . .
• ,
. .
• , .
. . . . .
• ,
. .
• , .
• Max relative percent approximation error of second iteration is:

• The iteration is continued until convergence is achieved.

44
Example 11.3, cont.
• Gauss-Seidel method is similar to the simple fixed-point iteration.
Therefore, sometimes the method might not converge to a set of
roots {X}.
• Relaxation may be applied to enhance convergence:
•
• Where λ is the weighting factor with 0<λ≤2
• The choice of λ is problem specific and empirical.

45