7 Nla2022

Numerical Linear Algebra
Ramaz Botchorishvili
Kutaisi International University
November 2, 2022
Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 1 / 34

Numerical Linear Algebra
Ramaz Botchorishvili
Kutaisi International University
November 2, 2022

Well posed problem, ill conditioned problem, condition
Number
▶ Recap of Previous Lecture

▶ Round-of errors
▶ How to avoid cancellation and recursion errors
▶ Computational template for numerical linear algebra
▶ Thomas algorithm
▶ Q&A

Recap of Previous Lecture
▶ Perturbations in right hand side and coefficients

▶ Error sources
▶ Number systems
▶ Floating point

Round-off errors, 1
Rounding and Chopping

Round-off errors, 1
▶ Not all real numbers are machine representable

Round-off errors, 1
▶ Finite number of real numbers are only representable in computers

Round-off errors, 1
▶ Consider 0.d−1 ...d−m d−m−1

Round-off errors, 1
▶ Chopping:
in m-digit arithmetics d−m−1 and all other further digits are thrown
away

Round-off errors, 1
▶ Chopping:
away
▶ Rounding:
in m-digit arithmetics d−m is rounded up or down, d−m−1 and all other
further digits are thrown away

Round-off errors, 1
▶ Chopping:
away
▶ Rounding:
▶ Rounding down: d−m−1 < β/2

Round-off errors, 1
▶ Chopping:
away
▶ Rounding:
▶ Rounding up: d−m−1 ≥ β/2
Example 7.1
π = 3.1415926

Round-off errors, 1
▶ Chopping:
away
▶ Rounding:
Example 7.1
π = 3.1415926
▶ Two-digit arithmetic fl(π) = 0.31 · 10−2

Round-off errors, 1
▶ Chopping:
away
▶ Rounding:
Example 7.1
π = 3.1415926
▶ Three-digit arithmetic fl(π) = 0.314 · 10−2

Round-off errors, 1
▶ Chopping:
away
▶ Rounding:
Example 7.1
π = 3.1415926
▶ Three-digit arithmetic fl(π) = 0.314 · 10−2
▶ Four-digit arithmetic fl(π) = 0.3142 · 10−2
Round-off errors, 2
Machine precision, significant numbers
Definition 7.2
Machine precision µ is smallest positive number such that
fl(1 + µ) > 1

Round-off errors, 2
Definition 7.2
fl(1 + µ) > 1
Definition 7.3
Suppose x is real number and x̃ is its approximation. x̃ approximates x to s
significant digits if s is largest nonnegative integer for which relative error
satisfies the inequality:
|x − x̃ |
< 5 · 10−s
|x |

Round-off errors, 2
Definition 7.2
fl(1 + µ) > 1
Definition 7.3
Suppose x is real number and x̃ is its approximation. x̃ approximates x to s
significant digits if s is largest nonnegative integer for which relative error
satisfies the inequality:
|x − x̃ |
< 5 · 10−s
|x |
Example 7.4
▶ x = 1.31, x̃ = 1.3, |x − x̃ | = 0.01, |x|x−x̃| | = 0.007635
▶ 7.635 · 10−3 < 5 · 10−2 , agree up to two significant digits
Round-off errors, 3
▶ Round-off Error in Representation of a Real Number
Theorem 7.5
(
|x −fl(x )| 0.5β 1−m for rounding
|x | ≤µ=
β 1−m for chopping

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
▶ Proof for round-off case
Proof.
▶ ▶ Consider x = (0.d−1 ...d−m d−m−1 ..)β e , d−1 ̸= 0, 0 ≤ d−i < β

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
Proof.
▶ fl(x )rounded down = (0.d−1 ...d−m )β e

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
Proof.
▶ fl(x )rounded up = (0.d−1 ...d−m + β −m )β e

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
Proof.
▶ x ∈ (fl(x )rounded down , fl(x )rounded up )

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
Proof.
▶ |x − fl(x )rounded down | ≤ 0.5|fl(x )rounded down − fl(x )rounded up | = β e−m

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
Proof.
▶ |x − fl(x )rounded up | ≤ 0.5|fl(x )rounded down − fl(x )rounded up | = β e−m

Round-off errors, 3
Theorem 7.5
(
|x | ≤µ=
Proof.
▶ |x − fl(x )rounded up | ≤ 0.5|fl(x )rounded down − fl(x )rounded up | = β e−m
|x −fl(x )| 0.5β −m 0.5β −m
▶ |x | ≤ 0.d−1 ...d−m d−m−1 .. ≤ β −1 = 0.5β 1−m
Round-off errors, 4
Corollary 7.6
(
|x | ≤µ= ⇒ fl(x ) = x (1 + δ), |δ| ≤ µ

Round-off errors, 4
Corollary 7.6
(
|x | ≤µ= ⇒ fl(x ) = x (1 + δ), |δ| ≤ µ
Proof.
x −fl(x )
▶ ▶ We set: δ ≡ x

Round-off errors, 4
Corollary 7.6
(
|x | ≤µ= ⇒ fl(x ) = x (1 + δ), |δ| ≤ µ
Proof.
x −fl(x )
▶ ⇒ |δ| ≤ µ

Round-off errors, 4
Corollary 7.6
(
|x | ≤µ= ⇒ fl(x ) = x (1 + δ), |δ| ≤ µ
Proof.
x −fl(x )
▶ ⇒ |δ| ≤ µ
▶ ⇒ fl(x ) − x = x δ

Round-off errors, 4
Corollary 7.6
(
|x | ≤µ= ⇒ fl(x ) = x (1 + δ), |δ| ≤ µ
Proof.
x −fl(x )
▶ ⇒ |δ| ≤ µ
▶ ⇒ fl(x ) − x = x δ
▶ ⇒ fl(x ) = x (1 + δ)

Round-off errors, 5
▶ Round-off in floating point addition
Example 7.7
▶ ▶ Floating point system β = 10, m = 2, emin = −3, emax = 3

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100
▶ x1 + x2 = 9.9 + 0.11 = 10.01 = 0.1001 · 102

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100
▶ x1 + x2 = 9.9 + 0.11 = 10.01 = 0.1001 · 102
▶ fl(x1 + x2 ) = 0.10 · 102

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100
▶ x1 + x2 = 9.9 + 0.11 = 10.01 = 0.1001 · 102
▶ fl(x1 + x2 ) = 0.10 · 102
▶ (x1 + x2 ) − fl(x1 + x2 ) = −0.0001 · 102 = 0.1 · 10−1

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100
▶ x1 + x2 = 9.9 + 0.11 = 10.01 = 0.1001 · 102
▶ fl(x1 + x2 ) = 0.10 · 102
▶ (x1 + x2 ) − fl(x1 + x2 ) = −0.0001 · 102 = 0.1 · 10−1
(x1 +x2 )−fl(x1 +x2 )
▶ δ= x1 +x2 ≈ −0.999 · 10−3

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100
▶ x1 + x2 = 9.9 + 0.11 = 10.01 = 0.1001 · 102
▶ fl(x1 + x2 ) = 0.10 · 102
▶ (x1 + x2 ) − fl(x1 + x2 ) = −0.0001 · 102 = 0.1 · 10−1
(x1 +x2 )−fl(x1 +x2 )
▶ δ= x1 +x2 ≈ −0.999 · 10−3
▶ δ =≈ 0.999000 · 10−3 < 0.5 · 10−2
▶ Other arithmetic operations: ×, :, −

Round-off errors, 5
Example 7.7
▶ x1 = 0.99 · 101 , x2 = 0.11 · 100
▶ x1 + x2 = 9.9 + 0.11 = 10.01 = 0.1001 · 102
▶ fl(x1 + x2 ) = 0.10 · 102
▶ (x1 + x2 ) − fl(x1 + x2 ) = −0.0001 · 102 = 0.1 · 10−1
(x1 +x2 )−fl(x1 +x2 )
▶ δ= x1 +x2 ≈ −0.999 · 10−3
▶ δ =≈ 0.999000 · 10−3 < 0.5 · 10−2
▶ Other arithmetic operations: ×, :, −

▶ See other examples in the textbook p.34

Round-off errors, 6
▶ Guard digits are extra digits in arithmetic register

Round-off errors, 6
▶ Can be used for reducing round-off errors

Round-off errors, 6
▶ In multi-step calculations intermediate results are not rounded

Round-off errors, 6
▶ For computers with guard digits fl(x1 + x2 ) = (x1 + x2 )(1 + δ), |δ| ≤ µ

Round-off errors, 6
▶ For computers without guard digits
fl(x1 + x2 ) = x1 (1 + δ1 ) + x2 (1 + δ2 ), |δ1 | ≤ µ, |δ2 | ≤ µ

Round-off errors, 6
fl(x1 + x2 ) = x1 (1 + δ1 ) + x2 (1 + δ2 ), |δ1 | ≤ µ, |δ2 | ≤ µ
▶ IEEE standard:
Theorem 7.8
Laws of floating point arithmetic:
▶ fl(x1 + x2 ) = (x1 + x2 )(1 + δ), |δ| ≤ µ

Round-off errors, 6
fl(x1 + x2 ) = x1 (1 + δ1 ) + x2 (1 + δ2 ), |δ1 | ≤ µ, |δ2 | ≤ µ
▶ IEEE standard:
Theorem 7.8
▶ fl(x1 + x2 ) = (x1 + x2 )(1 + δ), |δ| ≤ µ
▶ fl(x1 × x2 ) = (x1 × x2 )(1 + δ), |δ| ≤ µ

Round-off errors, 6
fl(x1 + x2 ) = x1 (1 + δ1 ) + x2 (1 + δ2 ), |δ1 | ≤ µ, |δ2 | ≤ µ
▶ IEEE standard:
Theorem 7.8
▶ fl(x1 + x2 ) = (x1 + x2 )(1 + δ), |δ| ≤ µ
▶ fl(x1 × x2 ) = (x1 × x2 )(1 + δ), |δ| ≤ µ
▶ fl(x1 /x2 ) = (x1 /x2 )(1 + δ), |δ| ≤ µ

Round-off errors, 6
fl(x1 + x2 ) = x1 (1 + δ1 ) + x2 (1 + δ2 ), |δ1 | ≤ µ, |δ2 | ≤ µ
▶ IEEE standard:
Theorem 7.8
▶ fl(x1 + x2 ) = (x1 + x2 )(1 + δ), |δ| ≤ µ
▶ fl(x1 × x2 ) = (x1 × x2 )(1 + δ), |δ| ≤ µ
▶ fl(x1 /x2 ) = (x1 /x2 )(1 + δ), |δ| ≤ µ
▶ What if n operands are involved in computation?

Round-off errors, 7
Theorem 7.9
Round-off error in floating point addition

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n
▶
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

(x1 + x2 )(δ2 + δ3 + ... + δn ) + x3 (δ3 + δ4 + ... + δn )+... + xn δn

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n
▶
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

▶ |δi | ≤ µ, 2 ≤ i ≤ n

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n
▶
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

▶ |δi | ≤ µ, 2 ≤ i ≤ n
Proof.
▶ s = x1 + x2 + ... + xn

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n
▶
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

▶ |δi | ≤ µ, 2 ≤ i ≤ n
Proof.
▶ s = x1 + x2 + ... + xn
▶ s2 = fl(x1 + x2 ) = (x1 + x2 )(1 + δ2 ), |δ2 | ≤ µ

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n
▶
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

▶ |δi | ≤ µ, 2 ≤ i ≤ n
Proof.
▶ s = x1 + x2 + ... + xn
▶ s2 = fl(x1 + x2 ) = (x1 + x2 )(1 + δ2 ), |δ2 | ≤ µ
▶ si = fl(si−1 + xi ) = (si−1 + xi )(1 + δi ), |δi | ≤ µ, 2 ≤ i ≤ n

Round-off errors, 7
Theorem 7.9
▶ xi ∈ R, 2 ≤ i ≤ n
▶
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

▶ |δi | ≤ µ, 2 ≤ i ≤ n
Proof.
▶ s = x1 + x2 + ... + xn
▶ s2 = fl(x1 + x2 ) = (x1 + x2 )(1 + δ2 ), |δ2 | ≤ µ
▶ si = fl(si−1 + xi ) = (si−1 + xi )(1 + δi ), |δi | ≤ µ, 2 ≤ i ≤ n
▶ s3 = fl(s2 + x3 ) = (s2 + x3 )(1 + δ3 ) =
((xBotchorishvili
Ramaz 1 + x2 )(1(KIU)
+ δ2 ) + x3 )(1 + δLectures
3 ) = in(x1 + x2 )(1 + δ2 )(1 + δ3Fall
NLA ) +Term
x3 (1
2022+11
δ3/)34
Round-off errors, 8
Round-off error in floating point addition, cont.
Proof. cont.
▶ s3 = (x1 + x2 )(1 + δ2 + δ3 ) + x3 (1 + δ3 ) + O(µ2 )

Round-off errors, 8
Proof. cont.
▶ s3 = (x1 + x2 )(1 + δ2 + δ3 ) + x3 (1 + δ3 ) + O(µ2 )
▶ s3 − (x1 + x2 + x3 ) = (x1 + x2 )(δ2 + δ3 ) + x3 δ3 + O(µ2 ) ≈
(x1 + x2 )(δ2 + δ3 ) + x3 δ3 = (x1 + x2 )δ2 + (x1 + x2 + x3 )δ3

Round-off errors, 8
Proof. cont.
▶ s3 = (x1 + x2 )(1 + δ2 + δ3 ) + x3 (1 + δ3 ) + O(µ2 )
▶ s3 − (x1 + x2 + x3 ) = (x1 + x2 )(δ2 + δ3 ) + x3 δ3 + O(µ2 ) ≈
(x1 + x2 )(δ2 + δ3 ) + x3 δ3 = (x1 + x2 )δ2 + (x1 + x2 + x3 )δ3
▶ si − (x1 + x2 + ... + xi ) ≈
(x1 + x2 )δ2 + (x1 + x2 + x3 )δ3 + ... + (x1 + x2 + ... + xi )δi

Round-off errors, 8
Proof. cont.
▶ s3 = (x1 + x2 )(1 + δ2 + δ3 ) + x3 (1 + δ3 ) + O(µ2 )
▶ s3 − (x1 + x2 + x3 ) = (x1 + x2 )(δ2 + δ3 ) + x3 δ3 + O(µ2 ) ≈
(x1 + x2 )(δ2 + δ3 ) + x3 δ3 = (x1 + x2 )δ2 + (x1 + x2 + x3 )δ3
▶ si − (x1 + x2 + ... + xi ) ≈
▶ (x1 + x2 )δ2 + (x1 + x2 + x3 )δ3 + ... + (x1 + x2 + ... + xi )δi
si+1 − (x1 + x2 + ... + xi+1 ) = fl(si + xi+1 ) − (x1 + x2 + ... + xi+1 ) =

(si + xi+1 )(1 + δi+1 ) − (x1 + x2 + ... + xi+1 ) =
(si − (x1 + x2 + ... + xi )) + si δi+1 + xi+1 δi+1 =
≈ (x1 + x2 )(δ2 + δ3 + ... + δi ) + ... + xi δi + si δi+1 + xi+1 δi+1 =
(x1 + x2 )(δ2 + δ3 + ... + δi ) + ... + xi δi +
(x1 + x2 + ... + xi )δi+1 + O(µ2 ) + +xi+1 δi+1 =
≈ (x1 + x2 )(δ2 + δ3 + ... + δi ) + ... + xi δi +
(x1 + x2 + ... + xi + xi+1 )δi+1
Round-off errors, 9
Laws of floating point arithmetic
fl(x ⊙ y ) = (x ⊙ y )(1 + δ)
⊙ = +, −, ∗, /
|δ| ≤ µ

Round-off errors, 9
Laws of floating point arithmetic
fl(x ⊙ y ) = (x ⊙ y )(1 + δ)
⊙ = +, −, ∗, /
|δ| ≤ µ
Example 7.10
Biswa Nath Datta, 2010
fl(x (y + z)) = [xfl(y + z)](1 + δ1 =)

= x (y + z)(1 + δ1 )(1 + δ2 ) =
= x (y + z)(1 + δ1 + δ2 + δ1 δ2 ) =
≈ x (y + z)(1 + δ1 + δ2 )
Round-off errors, 10
Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n
3. fl(cA) = cA + E , |E | ≤ µ|cA|

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n
3. fl(cA) = cA + E , |E | ≤ µ|cA|
4. fl(A + B) = (A + B) + E , |E | ≤ µ|A + B|

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n
3. fl(cA) = cA + E , |E | ≤ µ|cA|
4. fl(A + B) = (A + B) + E , |E | ≤ µ|A + B|
5. fl(AB) = AB + E , |E | ≤ nµ|A| |B| + O(µ2 )

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n
3. fl(cA) = cA + E , |E | ≤ µ|cA|
4. fl(A + B) = (A + B) + E , |E | ≤ µ|A + B|
5. fl(AB) = AB + E , |E | ≤ nµ|A| |B| + O(µ2 )
Floating point errors and matrix operations

1. ∥fl(AB) − AB∥1 ≤ nµ∥A∥1 ∥B∥1 + O(µ2 ), A, B ∈ Rn×n

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n
3. fl(cA) = cA + E , |E | ≤ µ|cA|
4. fl(A + B) = (A + B) + E , |E | ≤ µ|A + B|
5. fl(AB) = AB + E , |E | ≤ nµ|A| |B| + O(µ2 )

2. ∥fl(Ab) − Ab∥1 ≤ nµ∥A∥1 ∥b∥1 , A ∈ Rn×n , b ∈ Rn

Theorem 7.11
Wilkinson, 1965
1. |M| = (|mij |)
2. A, B ∈ Rn×n
3. fl(cA) = cA + E , |E | ≤ µ|cA|
4. fl(A + B) = (A + B) + E , |E | ≤ µ|A + B|
5. fl(AB) = AB + E , |E | ≤ nµ|A| |B| + O(µ2 )

2. ∥fl(Ab) − Ab∥1 ≤ nµ∥A∥1 ∥b∥1 , A ∈ Rn×n , b ∈ Rn
3. ∥fl(QB) − QB∥F ≤ nµ∥A∥F , A, Q ∈ Rn×n , Q T Q = I

Avoid round-off errors due to recursion and cancellation, 1
Theorem 7.12
Round-off error in floating point addition: xi ∈ R, 2 ≤ i ≤ n
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈
|δi | ≤ µ, 2 ≤ i ≤ n

Theorem 7.12
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈
|δi | ≤ µ, 2 ≤ i ≤ n
Example 7.13
Pn 1 P1 1
S1 = i=1 i , S2 = i=n i
Figure: which sum is more accurate, S1 or S2 ?

Theorem 7.14
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈
|δi | ≤ µ, 2 ≤ i ≤ n

Theorem 7.14
fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈
|δi | ≤ µ, 2 ≤ i ≤ n
Example 7.15
▶ S1 = ni=1 1i , S2 = 1i=n 1i
P P
▶ Recommendation:
summation from smallest to largest terms is more accurate

Example 7.16
1−cos(x )
▶ Cancellation error, compute function for small x : f (x ) = x2

Example 7.16
1−cos(x )
Figure: Cancellation errors: left=wrong approach, right=correct approach

Example 7.16
1−cos(x )
Figure: Cancellation errors: left=wrong approach, right=correct approach
▶ Remedy: rewrite formula equivalently for avoiding cancellation

sin2 (x ) sin2 (x )
1 − cos(x ) = 1+cos(x ) , f (x ) = (1+cos(x ))x 2

Example 7.17
Figure: Same expressions written differently give different results

Example 7.18
1. Exact input data:
1.1 x = 0.54617, y = 0.54601

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2. 4-digit arithmetic with rounding

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002
2.3 Large relative error |d− d̃|
|d| = 0.25

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002
|d| = 0.25
3. Catastrophic cancellation: two numbers of approximately same size
substracted

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002
|d| = 0.25
substracted
4. Notice: substraction reveals errors of previous computations

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002
|d| = 0.25
substracted
Round-off errors due to recursion

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002
|d| = 0.25
substracted
▶ p.43, Biswa Nath Datta, 2010,
En = 1 − nEn−1 , n = 2, 3.., En = o1 x n e x −1 dx
R

Example 7.18
1.1 x = 0.54617, y = 0.54601
1.2 d = x − y = 0.00016
2.1 x̃ = 0.5462, ỹ = 0.5460
2.2 d̃ = x̃ − ỹ = 0.0002
|d| = 0.25
substracted
▶ p.43, Biswa Nath Datta, 2010,
En = 1 − nEn−1 , n = 2, 3.., En = o1 x n e x −1 dx
R
▶ Recommendation: rearrange recursion

Computational template of Linear Algebra, 1
1. Transform the problem to ”easier to solve” form, e.g. matrices of the

problem into matrices with special structure


Example 7.19
   
a11 a12 ... a1n a11 a12 ... a1n
a
 21 a22 ... a2n   0 a ... a2n 
22
 −→ 
  
 .. .. ... ..   .. .. ... .. 
 
an1 an2 ... ann 0 0 ... ann


Example 7.19
   
a11 a12 ... a1n a11 a12 ... a1n
a
 21 a22 ... a2n   0 a ... a2n 
22
 −→ 
  
 .. .. ... ..   .. .. ... .. 
 
an1 an2 ... ann 0 0 ... ann
2. Exploit special structure of associated matrices and solve transformed

problem


Example 7.19
   
a11 a12 ... a1n a11 a12 ... a1n
a
 21 a22 ... a2n   0 a ... a2n 
22
 −→ 
  
 .. .. ... ..   .. .. ... .. 
 
an1 an2 ... ann 0 0 ... ann
2. Exploit special structure of associated matrices and solve transformed

problem
3. From the solution of transformed problem recover solution of the
original problem

Definition 7.20
LU Decomposition, LU Factorisation
A = LU
A, L, U-square matrices, L-lower triangular, U - upper triangular

Definition 7.20
A = LU
Example 7.21
    
a11 a12 ... a1n l11 0 ... 0 u11 u12 ... u1n
a
 21 a22 ... a2n  l21 l22 ... 0   0 u22 ... u2n 
   
=

 .. .. ... ..   .. .. ... ..   .. .. ... .. 
  
an1 an2 ... ann ln1 ln2 ... lnn 0 0 ... unn

Definition 7.20
A = LU
Example 7.21
    
a11 a12 ... a1n l11 0 ... 0 u11 u12 ... u1n
a
 21 a22 ... a2n  l21 l22 ... 0   0 u22 ... u2n 
   
=

 .. .. ... ..   .. .. ... ..   .. .. ... .. 
  
an1 an2 ... ann ln1 ln2 ... lnn 0 0 ... unn
Example 7.22
LU factoriozation for solving linear systems
Ax = b ⇒ A = LU, LUx = b⇒⇒L(Ux ) = b, Ly = b ⇒ Ux = y

Thomas algorithm, 1
▶ LU Decompositionm of square tridiagonal matrices

Thomas algorithm, 1
▶ Efficient for solving linear systems for tridiagonal matrices:

Thomas algorithm, 1
▶ Cramer’s rule O(n!) flops

Thomas algorithm, 1
▶ Gaussian elimination O(n3 ) flops

Thomas algorithm, 1
▶ Thomas algorithm O(n) flops

Thomas algorithm, 1
▶ Fastest Computer today in development = hexapoint(1018 ) operations
per second

Thomas algorithm, 1
per second
▶ How fast can you solve linear system with 103 , 106 , 109 unknowns?

Thomas algorithm, 1
per second
Example 7.23
 
a11 a12 . . . . 0
a21 a22 a23 . . . . 
 
 . . . . . . . 
 
 
 . . aii−1 aii aii+1 . . 
 
 . . . . . . . 
 
 . . . . . . . 
 
0 . . . . ann−1 ann

Thomas algorithm, 1
per second
Example 7.23
 
a11 a12 . . . . 0
a21 a22 a23 . . . . 
 
 . . . . . . . 
 
 
 . . aii−1 aii aii+1 . . 
 
 . . . . . . . 
 
 . . . . . . . 
 
0 . . . . ann−1 ann
▶ Better storage scheme: store three ”diagonals” ONLY

Thomas algorithm, 2
▶ Tgridiagonal system
Example 7.24
    
a1 c1 . . . . 0 x1 f1
b2 a2 c2 . . . .  x2  f2 
    
. . . . . . . .   . 
    
   

. . bi ai ci . .  xi  =  fi 
   

. . . . . . .  .   .
   

. . . . . . .  .   . 
    
0 . . . . b n an xn fn
▶ Storage scheme: vectors a, b, c, f and x
Theorem 7.25
If LU decomposition of a tridiagonal matrix exists then matrices L and U
are bi-diagonal

Thomas algorithm, 3
▶ LU factorization of tgridiagonal
 system
a1 c1 . . . . 0
b2 a2 c2 . . . .
 
. . . . . . .
 
 
. . bi ai ci . .
 =
. . . . . . .
 
. . . . . . .
 
0 . . . . bn an
  
1 . . . . . 0 α1 c1 . . . . 0
β2 1 . . . . .   . α2 c2 . . . . 
  
. . . . . . .  . . . . . . . 
  

 
. . βi 1 . . .   .   . . αi ci . . 
 
. . . . . . .  . . . . . . . 
  
. . . . . . . . . . . . . . 
  
 
0 . . . . βn 1 0 . . . . . αn
▶ Storage scheme: vectors a, b, c and α, β
Thomas algorithm, 4
LU Decomposition
  
1 . . . . . 0 α1 c1 . . . . 0
β2 1 . . . . .   . α2 c2 . . . . 
  
. . . . . . .  . . . . . . . 
  

 
. . βi 1 . . .  . . . α ci . . 
i =

 
. . . . . . .  . . . . . . . 
  
. . . . . . .  . . . . . . . 
  
0 . . . . βn 1 0 . . . . . αn


α 1 = a1
  
a1 c1 . . . . 0 α1 β2 = b2 ⇒ β2 = b2 /α1





b2 a2 c2 . . . .
  
β2 c1 + α2 = a2 ⇒ α2 = a2 − β2 c1



. . . . . . .
  

  ..

⇒
. . b i ai ci . .

. . . . . . . 
αi−1 βi = bi ⇒ βi = bi /αi−1
  
βi ci−1 + αi = ai ⇒

. . . . . . .
  




0 . . . . b n an αi = ai − βi ci−1




2≤i ≤n

Thomas algorithm, 5
LU
 Decomposition, solving
 linear system  
1 . . . . . 0 α1 c1 . . . . 0 x1
β2 1 . . . . .   . α2 c2 . . . .  x2 
   
. . . . . . .  . . . . . . .  . 
   
  

. . βi 1 . . .   . . . αi ci . .   xi  =
   

. . . . . . .  . . . . . . .  . 
 
  
. . . . . . .  . . . . . . .  . 
   
0 . . . . βn 1 0 . . . . . αn xn
 
f1 
 f2 
  


Ax = b

. LUx = f
  


 
 fi  ⇒ L(Ux ) = f
  
. 
  

 Ux = y
.
  

Ly = f

fn

Thomas algorithm, 6
solving linear system, forward substituition


     y1 = f1

1 . . . . . 0 y1 f1 

β2 y1 + y2 = f2


β2 1 . . . . .  y2  f2 
     



. . . . . . .  .   .  y2 = f2 − β2 y1
     

    

  yi  =  fi  ⇒ ...
. . βi 1 . . .    

. . . . . . .   .  .
    
 

 βi yi−1 + yi = fi
. . . . . . .  .   . 
     

yi = fi − βi yi−1



0 . . . . βn 1 yn fn 

2 ≤ i ≤ n


Thomas algorithm, 7
Solving linear system, backward substituition

    
α1 c1 . . . . 0 x1 y1 
 . α2 c2 .

. . .  x2  y2 
    


αn xn = yn

 . . . . . . .  .   .  xn = yn /αn
     

    

 .
 . . α i ci . .   xi  =  yi  ⇒ αi xi + ci yi+1 = fi
    
 . 
 . . . . . .  .   . 
    


 xi = (fi − ci yi+1 )/αi
 . . . . . . .  .   . 
     

i = n − 1, n − 2, ..., 1

0 . . . . . αn xn yn

Thomas algorithm, 8
1. LU Decomposition
α1 = a1 , βi = bi /αi−1 , αi = ai − βi ci−1 , i = 2, 3, ..., n
2. Forward substitution

Thomas algorithm, 8
1. LU Decomposition
y1 = f1 , yi = fi − βi yi−1 , i = 2, 3, ..., n
3. Backward substitution

Thomas algorithm, 8
1. LU Decomposition
y1 = f1 , yi = fi − βi yi−1 , i = 2, 3, ..., n
xn = yn /αn , xi = (fi − ci yi+1 )/αi i = n − 1, n − 2, ..., 1

Thomas algorithm, 8
1. LU Decomposition
y1 = f1 , yi = fi − βi yi−1 , i = 2, 3, ..., n
Solving linear system, number of arithmetic operations

1. LU Decomposition 3(n − 1) operations

Thomas algorithm, 8
1. LU Decomposition
y1 = f1 , yi = fi − βi yi−1 , i = 2, 3, ..., n

2. Forward substitution 2(n − 1) operations

Thomas algorithm, 8
1. LU Decomposition
y1 = f1 , yi = fi − βi yi−1 , i = 2, 3, ..., n

3. Backward substitution 3(n − 1) + 1 operations

Thomas algorithm, 8
1. LU Decomposition
y1 = f1 , yi = fi − βi yi−1 , i = 2, 3, ..., n

3. Backward substitution 3(n − 1) + 1 operations
4. Total 8(n − 1) ≈ O(n) operations
Thomas algorithm, 10
▶ Q: When is LU Decomposition very useful?


▶ A: If solving the same system with many different right hand sides is
needed


needed
Example 7.26
▶ ▶ Finding inverse of a matrix


needed
Example 7.26
▶ AB = I or BA = I ⇒ B = A−1


needed
Example 7.26
▶ AB = I or BA = I ⇒ B = A−1
       
1 0 ... 0 1 0 0
0 1 ... 0 0 1 0
▶ I=  I =   I =   ...I =   ⇒ I = (I1 I2 ... In )
 ... 0 1   2   n  
0 0 ... 1 0 0 1


needed
Example 7.26
▶ AB = I or BA = I ⇒ B = A−1
       
1 0 ... 0 1 0 0
0 1 ... 0 0 1 0
▶ I=  I =   I =   ...I =   ⇒ I = (I1 I2 ... In )
 ... 0 1   2   n  
0 0 ... 1 0 0 1
▶ AB = I ⇒ Abi = Ii , i = 1, 2, ..n ⇒ B = (b1 b2 ...bn ) = A−1

▶ Q: Is Thomas algorithm always good to apply?

▶ A: No:

▶ A: No:
▶ if matrix is degenerate

▶ A: No:
▶ if stability conditions are not satisfied

▶ A: No:
|a1 | > |c1 | > 0,

|ai | ≥ |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.

▶ A: No:
|a1 | > |c1 | > 0,

|ai | ≥ |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.
Example 7.27
Figure: Stability condition satisfied

Example 7.28
Figure: Stability condition is not satisfied
Example 7.29
Figure: Stability condition is not satisfied

Stability
▶ strictly diagonally dominant by rows

Stability
▶
|a1 | > |c1 | > 0,

|ai | > |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.

Stability
▶
|a1 | > |c1 | > 0,

|ai | > |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.
▶ Assume perturbations in RHS Ax = f , Ax̃ = f˜

Stability
▶
|a1 | > |c1 | > 0,

|ai | > |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.

▶ A(x − x̃ ) = f − f˜

Stability
▶
|a1 | > |c1 | > 0,

|ai | > |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.

▶ A(x − x̃ ) = f − f˜
▶ bi (xi−1 − x̃i−1 ) + ai (xi − x̃i ) + ci (xi+1 − x̃i+1 ) = fi − f˜i

Stability
▶
|a1 | > |c1 | > 0,

|ai | > |bi | + |ci |, i = 2, 3, ..., n − 1,
|an | > |cn | > 0.

▶ A(x − x̃ ) = f − f˜
▶ bi (xi−1 − x̃i−1 ) + ai (xi − x̃i ) + ci (xi+1 − x̃i+1 ) = fi − f˜i
⇓
▶ ∥x − x̃ ∥∞ ≤ |a |−|b1 |−|c | ∥f − f˜∥∞
i i i

Q&A

7 Nla2022

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

7 Nla2022

Uploaded by

Copyright:

Available Formats

Numerical Linear Algebra

Kutaisi International University

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 1 / 34

Kutaisi International University

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 2 / 34

▶ Recap of Previous Lecture

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 3 / 34

▶ Perturbations in right hand side and coefficients

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 4 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 5 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 6 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 6 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 7 / 34

▶ Round-off Error in Representation of a Real Number

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 8 / 34

▶ Round-off Error in Representation of a Real Number

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 8 / 34

▶ Round-off Error in Representation of a Real Number

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 8 / 34

▶ Round-off Error in Representation of a Real Number

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 8 / 34

▶ Round-off Error in Representation of a Real Number

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 8 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

▶ Other arithmetic operations: ×, :, −

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

▶ Other arithmetic operations: ×, :, −

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 9 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 10 / 34

▶ What if n operands are involved in computation?

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 11 / 34

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 11 / 34

fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈

Ramaz Botchorishvili (KIU) Lectures in NLA Fall Term 2022 11 / 34

fl(x1 + x2 + ... + xn ) − (x1 + x2 + ... + xn ) ≈