Inm 1

Introduction to
Numerical Methods
Fiorella Sgallari
University of Bologna, Italy
Faculty of Engineering
Department of Mathematics - CIRAM
Numerical Methods
Numerical Methods:
Algorithms that are used to obtain numerical
solutions of a mathematical problem.
Why do we need them?
1. No analytical solution exists,
2. An analytical solution is difficult to obtain
or not practical.
Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 2

What do we need
Basic Needs in the Numerical Methods:

Practical:
can be computed in a reasonable amount of time.
Accurate:
• Good approximation to the true value
• Information about the approximation error (bounds,
error order,… )

Number Representation and accurcy
Number Representation
Normalized Floating Point Representation
Rounding and Chopping
Well-posedness and Condition Number of a Problem
Stability of Numerical Methods
Error sources
1. Simplified mathematical model
2. Errors in data
1. Systematic errors due to instruments
2. Random errors
3. Rounding in data and computation
4. Discretization error
These errors are introduced when an infinite procedure is
approximated by a finite procedure.
• Derivatives by finite differences
• Integrals by quadrature formulas
Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 5 5

Representing Real Numbers
You are familiar with the decimal system
312 .45 = 3 × 102 + 1 × 101 + 2 × 100 + 4 × 10−1 + 5 × 10−2
Decimal System Base =10 , Digits(0,1,…9)

Standard Representation
± 3 1 2 . 4 5
sign integral fractional
part part

Normalized Floating Point Representation
32.213 0.32213 x 102 normalized
0.003221x104 no
x = ± 0 . a 1 a 2 .... × β p
0 ≤ a i ≤ β − 1, a1 ≠ 0
p 1
x = ± m × β , ≤ m < 1
β
Sign exponent p mantissa m

Binary System
Binary System Base=2, Digits{0,1}
± 0. 1 b2 b3 ... × 2 p
sign mantissa exponent
b1 ≠ 0 ⇒ b1 = 1
(0.101) 2 = (1× 2 −1 + 0 × 2 −2 + 1× 2 −3 )10 = (0.625)10

Fact
Number that have finite expansion in one numbering

system may have an infinite expansion in another
numbering system
(0.1)10 = (0.000110011001100...) 2
You can never represent 0.1 exactly in any computer

Machine Numbers
F(β
β, t, L, U) =
{x ∈ ℜ = ± ( a 1β
−1
+ a2β −2
+ ... a t β −t
)β p
}∪ {0}
0 ≤ a i ≤ β − 1, a1 ≠ 0
L ≤ p ≤ U, L < 0, U > 0 Generally L=-U

Machine Numbers
Card(F(β
β, t, L, U) ) = 2 (β β t-1 (U-L+1) +1
β-1)β
xmin = βL−1 ≤ |x| ≤ βU(1 − β−t) = xmax
The floating-point numbers are not equally spaced along the real line
They get denser close to the smallest representable number
Ex: Card{1,2,3,4}=4
F(2,3,-1,2)

Positive numbers
F(2, 3, -1, 2) 33 elements

Machine Numbers
I. Exponent L ≤ p ≤ U, L < 0, U > 0
If p ∉ [L, U ] p>U overflow

p<L underflow
II. Mantissa , t digits available
If the number of digits in mantissa ≥ t: rounding and chopping

Rounding and Chopping
x x

Examples
β=10 t=4
X=0.372145 chopping x = fl ( x ) = 0.3721
X=0.3798165 x = fl ( x ) = 0.3798
rounding
y=0.1265873
y = fl ( y ) = 0.1266

Absolute and Relative Error
p p
Let x = mβ , x = fl ( x ) = m β
Absolute Error
x − fl (x)
Relative Error
x − fl ( x)
x
Sia x = mβ p , x = fl ( x ) = m β p
Chopping Rounding
Absolute Error
p− t 1 p−t
x − fl ( x ) < β x − fl ( x ) ≤ β
2
Relative Error
x − fl ( x ) x − fl ( x ) 1 1− t
≤ β 1− t ≤ β
x x 2
Machine precision-roundoff unit
 β 1− t chopping
x − fl ( x ) 
≤ eps eps =  1 1− t
x  β rounding
2
fl(1+eps)>1
We say the estimate is correct to n decimal digits if
Rel. Error ≤ 10− n
We say the estimate is correct to n decimal digits rounded if
1
Rel. Error ≤ × 10− n
2

Example: β=10, t=5
x=1 y=0.5 10-4 eps

fl(x+y)=fl(1.00005)=0.10001 x 101
x=1 y=0.4 10-4

fl(x+y)= fl(1.00004)=0.10000 x 101

IEEE-754 Format
Single precision 32 bit base 2
1 8 23
Exponent p* Mantissa m
p* = p + 127, 0 ≤ p* ≤ 255
Segno (0/1) ‘m has 24 bit , the first 1 is implicit’
Double precision 64 bit base 2

1 11 52
Exponent p* Mantissa m
p* = p + 1023, 0 ≤ p* ≤ 2047
Segno (0/1)
IEEE-754 Format Single Precision
Single precision 32 bit base 2
1− 24
1 1−t 2
eps = 2 = = 5.9605e - 008
2 2
Double precision 64 bit base 2
1 1−t 21−53
eps = 2 = = 1.1102e - 016
2 2
Matlab uses IEEE arithmetic with double precision

Special Exponents and Numbers
zero
0/1 0 0
NaN=not a number
0/1 111.....1111 ≠0
Infinite (overflow)
0/1 111.....1111 0

Machine Floating-point Operations
a = fl(a), b = fl(b)
a op b = fl(a opb ) = (a opb )(1+ ε )
op : +, −,*,/, with ε ≤ eps
It is important to notice that, together with properties of standard arithmetic
that are preserved when passing to floating-point arithmetic (like, for instance,
the commutativity of the sum of two addends, or the product of two factors),
other properties are lost.
An example is given by the associativity of sum:
x + (y + z) = (x + y) + z

Numerical cancellation
Subtracting two numbers almost equal is ill conditioned.

This fact leads to the cancellation of significant digits.
Ex: X1=0.147554326 X2=0.147251742, t=6, β=10

fl(X1)=0.147554 fl(X2)=0.147252
fl(fl(X1)-fl(X2))=0.302000x10-3
while the true difference is:
X1-X2=0.302584x10-3
The last digits of mantissa are changed!
This is due to the fact that after the difference fl(x1)-fl(x2)=0.000302 and
floating point representation we have three zeros at the end of mantissa.
Relative error ~ 10-3

Example
x2-6.433 x + 0.009474 = 0
6.433 − (6.433) 2 − 4(0.009474)

β=10 t=4 x2 =
2
fl(x)=0.2000 x 10-2 x=0.0014731 esatto
x − fl ( x ) 36%
≅ 0.357
x

Example
x2-6.433 x + 0.009474 = 0 β=10 t=4
b − ∆ = b − ( 6.433 ) 2 − 4( 0.009474 ) = 0.6433 × 10 1 − 0.6429 × 10 1
Even if our data have a small relative error
6.433 − 6.433 6.4300538 − 6.429

= 0. ≈ 0.16 × 10− 3
6.433 6.4300538
Error in b Error in ∆
Example
a2+b x + c = 0
2
−b± b − 4 ac
x1, 2 =
2a
Numerical canc .error in x1 if ∆ ≈ b , or in x2 if ∆ ≈ −b
To avoid this:
c c
x1 ⋅ x2 = → x2 =
a x1
If b 2 ≈ 4ac it is useful to use great precision
Condition Number of a Problem –
x f y Numerical algorithm: ψ
Perturbed data:
~
x = x + δx
x f ( x) = y
Propagation of errors? ~
x f (~x) = ~y
ψ ( ~x ) = ψ~

Condition Number of a Problem
A Problem is well posed if it admits a unique solution x

which depends with continuity on the data.
~
f ( x) − f ( x ) ~
x−x
≤K
f ( x) x
A Problem is called ill-conditioned if K(d) is “big” for any admissible
datum d (the precise meaning of “small” and “big” is going to change
depending on the considered problem).

We shall suppose the problem to be well posed.
The precise definition of stability depends on the context, but it is

related to the accuracy of the algorithm.
Numerical stability: the property that small perturbations

on the data yield perturbations of the same order on the
solution.

Example: unstable algorithm
1
We have to compute En = ∫ x n e x −1dx n = 1, 2, ...
0
integration by parts:
1
∫x
0
n x −1
e dx = [n x −1 1
x e 0
] 1
− ∫ nx
0
n −1 x −1
e dx
Recursive formula
1
En = 1 − nEn−1 n = 2, 3, ... E1 =
e

β=10 t=6 En = 1 − nEn −1 n = 2, 3, ...
while x9ex-1>0
in (0,1)

Rounding error in the computation of E1
1
E1 = , E1 = E1 + δ , δ ≈ 4.412 ⋅ 10−7
e
When computing E2 such error is multiplied by –2,
E 2 = 1 − 2 E1 = 1 − 2 E1 − 2δ = E 2 − 2δ
When computing E3 by –3, and so on.
At the end E9 by (-2)(-3)…(-9). So
n −1
E n = 1 − nE n −1 = E n + ( −1) n !δ
9!* 4.412 ⋅ 10 −7 ≅ 1.1601
1 − En
E n −1 = n = ...., 3, 2
n
To find an initial value:

1
1 1
x  1 n +1
∫x dx ≤ ∫ x = 
n x −1 n
En = e  =
0 0  n + 1 0 n +1
En → 0 approximate E20 = 0
n →∞

1 − En
E n −1 = n = ...., 3, 2 E 20 = 0
n

1
E20 = 0 Initial Error ≈
21
In the computation of E19 this error is multiplied by 1/20,
1
1 − E20 1 − E20 − 21
E19 = = =
20 20
1 − E20 1 1 1
= − ≈ ≅ 0.0024
20 20 ⋅ 21 20 21
True value E9 (3 cifre) ≅ 0.0916

Example: Well Conditioned Problem
(Wilkinson 1963)
Compute the roots of :

p(x)=(x-1)(x-2)…(x-19)(x-20)
=x20-210 x19+….
Computer β=2 t=30

Roots: 1, 2, 3, …,19, 20
To store our coefficients we round them at 30th binary digit.
Perturbation on coefficient of x19
-210 -210 + 2-23

(Wilkinson 1963)
The polinomial is : p(x)+2-23 x19=0

Are the roots changed ?
β=2 t=90

(Wilkinson 1963)
α)=x20-α
p(x,α α x19+….=0 α parameter
∂ p( x , α ) ∂ x ∂ p( x , α )
+ =0
∂x ∂α ∂α
19
∂x ∂p( x, α ) / ∂α x
=− = 20
∂α ∂p( x,α ) / ∂x
∑∏
20
j =1 ( x − j)
j ≠i
19 i =1
∂x i
= 20
i =1, 2 ,...,19 , 20
∂α x =i ∏ j =1 (i − j )
j ≠i

(Wilkinson 1963)
The computed numbers

give us a measure of
the sensitivity of each
root to coefficient α

Good algorithm
1. General and robust

2. Simple hypotheses
3. Numerical stability
4. Required resources
• Number of floating point operations
• Memory

Computational complexity
Computational complexity of an algorithm=

number of floating point operations
Measure unity
FLOP (floating point operation)
1 flop = one operation (+,-,*,/)

Example
Linear Systems: Ax=b n unknowns
Hypothesis:
Execution time for 1 multiplication about 10-6 seconds (old computer)
Cramer's Rule ~(n+1)!

(solution as quotient of n-order determinants)
Gauss Algorithm ~n3
n CRAMER GAUSS
13 24 hours 9.0x10-4sec
14 15 days 1.1x10-3sec
20 1.7x106 years 3.1x10-3sec

Inm 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Inm 1

Uploaded by

Copyright:

Available Formats

Introduction to

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 2

Basic Needs in the Numerical Methods:

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 3

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 5 5

You are familiar with the decimal system

312 .45 = 3 × 102 + 1 × 101 + 2 × 100 + 4 × 10−1 + 5 × 10−2

Decimal System Base =10 , Digits(0,1,…9)

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 6

32.213 0.32213 x 102 normalized

Sign exponent p mantissa m

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 7

Binary System Base=2, Digits{0,1}

(0.101) 2 = (1× 2 −1 + 0 × 2 −2 + 1× 2 −3 )10 = (0.625)10

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 8

Number that have finite expansion in one numbering

You can never represent 0.1 exactly in any computer

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 9

L ≤ p ≤ U, L < 0, U > 0 Generally L=-U

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 10

xmin = βL−1 ≤ |x| ≤ βU(1 − β−t) = xmax

They get denser close to the smallest representable number

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 12

F(2, 3, -1, 2) 33 elements

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 13

I. Exponent L ≤ p ≤ U, L < 0, U > 0

If p ∉ [L, U ] p>U overflow

II. Mantissa , t digits available

If the number of digits in mantissa ≥ t: rounding and chopping

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 14

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 15

X=0.372145 chopping x = fl ( x ) = 0.3721

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 16

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 19

x=1 y=0.5 10-4 eps

x=1 y=0.4 10-4

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 20

Single precision 32 bit base 2

Double precision 64 bit base 2

Single precision 32 bit base 2

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 22

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 23

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 24

Subtracting two numbers almost equal is ill conditioned.

Ex: X1=0.147554326 X2=0.147251742, t=6, β=10

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 25

6.433 − (6.433) 2 − 4(0.009474)

fl(x)=0.2000 x 10-2 x=0.0014731 esatto

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 26

x2-6.433 x + 0.009474 = 0 β=10 t=4

b − ∆ = b − ( 6.433 ) 2 − 4( 0.009474 ) = 0.6433 × 10 1 − 0.6429 × 10 1

Even if our data have a small relative error

6.433 − 6.433 6.4300538 − 6.429

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 29

A Problem is well posed if it admits a unique solution x

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 30

We shall suppose the problem to be well posed.

The precise definition of stability depends on the context, but it is

Numerical stability: the property that small perturbations

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 31

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 32

β=10 t=6 En = 1 − nEn −1 n = 2, 3, ...

Numerical Methods F. Sgallari Dept. Mathematics-CIRAM, Universityof Bologna 33