(4.

1) - Numerical Differentiation

1. Difference formulas derived using Taylor Theorem:

′
a. Difference formulas for f and their approximation errors:
Recall:
′ fx  h − fx
f x  lim .
h→0 h
Consider h  0 small. Numerical Difference Formulas:
′ fx  h − fx
f x ≈ - forward difference formula - two-points formula
h
′ fx − fx − h
f x ≈ - backward difference formula - two-points formula
h
′ fx  h − fx fx − fx − h fx  h − fx − h
f x ≈ 1   - central difference formula
2 h h 2h
- three-points formula

Approximation error: Consider the Taylor polynomials and remainder of fx  h and fx − h;
′ ′′
f x f c 2
fx  h  fx  h h where c is in x, x  h ;
1! 2!
′ ′′
f x f c 2
fx − h  fx − h h where c is in x − h, x .
1! 2!
Then
′′
′ fx  h − fx f c fx  h − fx
f x  − h  Oh
h 2! h
′′
′ fx − fx − h f c fx − fx − h
f x   h  Oh
h 2! h
Approximation errors for the forward difference and the backward difference formulas are
′′
′ fx  h − fx f c
R fd  f x −  − h , where c is in x, x  h .
h 2!
′′
′ fx − fx − h f c
R bd  f x −  h , where c is in x − h, x .
h 2!
Now consider
′ ′′ ′′′
f x f x 2 f c 1  3
fx  h  fx  h h  h
1! 2! 3!
′ ′′ ′′′
f x f x 2 f c 2  3
fx − h  fx − h h − h
1! 2! 3!
′′′ ′′′
′ f  1 x f c 2 
fx  h − fx − h  2hf x   h3
3! 3!

1
 ′′′ ′′′ ′′′
′ fx  h − fx − h f c 1  f c 2  fx  h − fx − h 2f c
f x  − 1  h 
3
− 1 h2
2h 2h 3! 3! 2h 2 3!
′′′
fx  h − fx − h f c 2 fx  h − fx − h
 − h   Oh 2 
2h 3! 2h
Approximation errors for the central difference formula is
′′′
′ fx  h − fx − h f c 2
R cd  f x −  − h where c is in x − h, x  h .
2h 6
′′
b. Difference formula for f and its approximation error:
′′
Derive a difference formula for f x :
′ ′′ ′′′
f x f x 2 f x 3 f 4 c 1  4
fx  h  fx  h h  h  h
1! 2! 3! 4!
′ ′′ ′′′
f x f x 2 f x 3 f 4 c 2  4
fx − h  fx − h h − h  h
1! 2! 3! 4!
′′
fx  h  fx − h  2fx  f xh 2  1 f 4 c 1   f 4 c 2 h 4
4!
′′ fx  h − 2fx  fx − h 2
f x  2
− h f 4 c 1   f 4 c 2 , c 1 is in x, x  h , c 2 is in x − h, x 
h 4!
fx  h − 2fx  fx − h
 − 2 f 4 ch 2
h2 4!
(f 4 c  1 f 4 c 1   f 4 c 2 , cx is in x − h, x  h )
2
fx  h − 2fx  fx − h
 − 1 f 4 cxh 2
h2 12

y 12

10

4
Graphically,
2

-1 1 2 3
-2
x

-4

red f forward , green f backward , blue f central

2
 ′
c. Difference formulas for f x :
#
Formula approx c
of
error
pts

′′
1 fx  h − fx f c
2-1 − h x, x  h
h 2!
′′
1 fx − fx − h f c
2-2 − h x − h, x
h 2!
′′′
1 1 fx  h − 1 fx − h f c 2
3-1 − h x − h, x  h
h 2 2 6
′′′
1 − 3 fx  2fx  h − 1 fx  2h f c 2
3-2 h x, x  2h
h 2 2 3
′′′
1 1 fx − 2h − 2fx − h  3 fx f c 2
3-3 h x − 2h, x
h 2 2 3
5
1 1 fx − 2h − 8 fx − h  8 fx  h − 1 fx  2h f c 4
5-1 h x − 2h, x  2h
h 12 12 12 12 30
5
1 − 25 fx  48 fx  h − 36 fx  2h  16 fx  3h − 3 fx  4h f c 4
5-2 h x, x  4h
h 12 12 12 12 12 5
′′
Difference formula for f x :
# of pts Formula approx error c
1 fx  h − 2fx  fx − h f 4 c 2
3-1 − h x − h, x  h
h2 12

3
1. Example Let fx  sine x − 1.
′
i. Approximate f 0 by all three difference formulas with h  0. 1, and 0. 01, and compute an upper
bound for each approximation error and compute the true error.
′′
ii. Approximate f 0 with h  0. 1 and 0. 01, and find an upper bound for the approximation error and
compare it with the true error.
′ ′
i. f x  e x cose x − 1, f 0  1
fx 0  h − fx 0  sine h − 1 − sine 0 − 1 sine h − 1
 
h h h
fx 0  − fx 0 − h 0 − sine −h − 1 − sine −h − 1
 
h h h
fx 0  h − fx 0 − h sine h − 1 − sine −h − 1

2h 2h
′′ ′′
f x  e x cose x − 1 − e 2x sine x − 1, f 0  1
′′′
f x  e x cose x − 1 − 3e 2x sine x − 1 − e 3x cose x − 1

y
1.00 y 0.6

0.99 0.4

0.98 0.2

-0.1 0.0 0.1 -0.1 0.0 0.1

x x
′′ ′′′
y  f x , x in −0. 1, 0. 1 y  f x , x in −0. 1, 0. 1
′′ ′′
R fd  h f c ≤ h f 0 ≤ h , c is in 0, h,
2 2 2
′′ ′′
R bd  h f c ≤ h f 0 ≤ h , c is in −h, 0,
2 2 2
2 2 ′′′ 2
R cd  h |f ′′′ c| ≤ h f h ≤ h e h − e 3h  cose h − 1 − 3e 2h sine h − 1
6 6 6
Forward difference:
sine h − 1 sine h − 1
h Approximation true error 1 − approx. error ≤ h
h h 2
sine 0.1 − 1 sine 0.1 − 1 0. 1  0. 05
0. 1  1. 049 8 1−  4. 977 1  10 −2
0. 1 0. 1 2
sine 0.01 − 1 sine 0.01 − 1 0. 01  0. 005
0. 01  1. 005 0 1−  4. 999 8  10 −3
0. 01 0. 01 2
Backward difference:

4
 − sine −h − 1 sine −h − 1
h Approximation true error 1  approx. error ≤ h
h h 2
− sine −0.1 − 1 sine −0.1 − 1 0. 1  0. 05
0. 1  0. 950 19 1  4. 981 0  10 −2
0. 1 0. 1 2
− sine −0.01 − 1 sine −0.01 − 1 0. 01  0. 005
0. 01  0. 995 1  4. 999 8  10 −3
0. 01 0. 01 2
Central difference:
0. 1 2 0.1
h  0. 1, R cd  |e − e 30.1  cose 0.1 − 1 − 3e 20.1 sine 0.1 − 1|  1. 046 656 699  10 −3
6
0. 01 2 0.01
h  0. 01, R cd  |e − e 30.01  cose 0.01 − 1 − 3e 20.01 sine 0.01 − 1|  8. 527 066 727  10 −7
6

approx. error
Approximation true error 2
h sine h − 1 − sine −h − 1 | h e h − e 3h  cose h − 1
sine h − 1 − sine −h − 1 1− 6
2h 2
2h − h 3e 2h sine h − 1|
6
sine 0.1 − 1 − sine −0.1 − 1 sine 0.1 − 1 − sine −0.1 − 1
1−
0. 1 20. 1 20. 1 1. 046 656 699  10 −3
 0. 999 98  1. 919 851 521  10 −5

sine 0.01 − 1 − sine −0.01 − 1 sine 0.01 − 1 − sine −0.01 − 1

1−
0. 01 20. 01 20. 01 8. 527 066 727  10 −7
 0. 999 999 998 1  1. 916 698 61  10 −9
′′ ′′
ii. f x  e x cose x − 1 − e 2x sine x − 1, f 0  1
′′′
f x  e x − e 3x  cose x − 1 − 3e 2x sine x − 1
f 4 x  e x − 6e 3x  cose x − 1  e 4x − 7e 2x  sine x − 1

-0.1 0.0 0.1

x

y  |f 4 x|, x in −0. 1, 0. 1

′′ f h − 2f0  f−h sine h − 1  sine −h − 1
f 0 ≈ 
h2 h2
2 2 2
error  |R|  h |−f 4 c|  h |−f 4 h|  h |e h − 6e 3h  cose h − 1  e 4h − 7e 2h  sine h − 1|
12 12 12
5
 2
h  0. 1, |R| ≤ 0. 1 |e 0.1 − 6e 30.1  cose 0.1 − 1  e 40.1 − 7e 20.1  sine 0.1 − 1|  6. 413 554 783  10 −3
12
2
h  0. 01, |R| ≤ 0. 01 |e 0.01 − 6e 30.01  cose 0.01 − 1  e 40.01 − 7e 20.01  sine 0.01 − 1|  4. 361 439 006  10 −5
12
Approximation true error
h sine h −1sine −h −1
approx. error
sine h −1sine −h −1
1− h2
h2

sine 0.1 −1sine −0.1 −1 sine 0.1 −1sine −0.1 −1
1−
0. 1 0.1 2
0.1 2 6. 413 554 783  10 −3
 0. 995 812 780 8  4. 187 219 244  10 −3
e 0.01 sine 0.01 −1e −0.01 sine −0.01 −1 sine 0.01 −1sine −0.01 −1
 1−
0. 01 0.01 2
0.01 2 4. 361 439 006  10 −5
 0. 999 999 998 1  4. 166 872 222  10 −5

Example Verify that the difference formula:

−3fx 0   4fx 0  h − fx 0  2h
′
f x 0  ≈ .
2h
provides the exact value of the derivative, regardless of h, for the functions: fx  1, fx  x and
fx  x 2 but not for fx  x 3 .
′
Let fx  1. Then f x  0 and
−3fx 0   4fx 0  h − fx 0  2h −31  41 − 1
  0.
2h 2h
′
Let fx  x. Then f x  1 and
−3fx 0   4fx 0  h − fx 0  2h −3x 0  4x 0  h − x 0  2h
  4h − 2h  1.
2h 2h 2h
′
Let fx  x 2 . Then f x  2x and
−3fx 0   4fx 0  h − fx 0  2h −3x 20  4x 0  h 2 − x 0  2h 2

2h 2h
−3x 0  4x 0  2x 0 h  h 2  − x 20  4x 0 h  4h 2 
2 2

2h
 4x 0 h  2x 0 .
2h
′
Let fx  x 3 . Then f x  3x 2 and
−3fx 0   4fx 0  h − fx 0  2h −3x 30  4x 0  h 3 − x 0  2h 3

2h 2h
−3x 0  4x 0  3x 20 h  3x 0 h 2  h 3  − x 30  6x 20 h  12x 0 h 2  8h 3 
2 3

2h
6x 20 h − 4h 3
  3x 20 − 2h 2 ≠ 3x 20 .
2h
Note that this example shows that a three-point difference formula can also be derived by determining coefficients
of
Fx 0   afx 0   bfx 0  h  cfx 0  2h
′
so that Fx 0   f x 0  for fx  1, x, x 2 . The disadvantage of using this method to derive a difference formula is
6
no information about the approximation error. Now let us use Taylor expansions of fx 0  h and fx 0  2h to
derive the approximation error.
′′
f x 0  2 f ′′′ c 1  3
4fx 0  h  4 fx 0   f ′ x 0 h  h  h
2! 3!
′′
f x 0  f ′′′ c2
− fx 0  2h  − fx 0   f ′ x 0 2h  2h 2  2h 3
2! 3!
2 ′′ 3
− 3fx 0   4fx 0  h − fx 0  2h  −3  4 − 1fx 0   h2f ′ x 0   h 4 − 4f x 0   h f ′′′ c 1  − 8f ′′′ c 2 
2 3!
3
 2hf ′ x 0  − h 7f ′′′ c , c is in x 0 , x 0  2h
3!
′ −3fx 0   4fx 0  h − fx 0  2h 2 ′′′
f x 0    7h f c
2h 12

2. Difference Formulas Derived Using Lagrangian Interpolating Polynomials:

Let x 0  x 1 . . . ,  x n and f n1 x be continuous for x in x 0 , x n . Let P n x be the
Lagrange interpolating polynomial which agrees with fx at data: x i , fx i  , i  0, 1, 2, . . . , n :
n n n
x − x k 
P n x  ∑ fx i L n,i x  ∑ fx i   x i − x k 
i0 i0 k0, k≠i

When fx ≈ P n x, the approximation error is

n
f n1 cx
R n x 
n  1!
x − x i .
i0
′
Consider to use P n ′ x to approximate f x:
n n n
x − x k 
′
P n x  ∑ fx i L ′n,i x  ∑ fx i  d
dx
 x i − x k 
i0 i0 k0, k≠i

The approximation error is

n
f n1 cx
R
′
n x  d
dx n  1!
x − x i 
i0
n n
 1 d f n1 cx x − x i   f n1 cx d x − x i 
n  1! dx i0
dx i0
n n n
 1 d f n1 cx x − x i   f n1 cx ∑  x − x i 
n  1! dx i0 k0 i0, i≠k

Note that
n
R
′
n x m   1
n  1!
f n1 cx  x m − x i .
i0, i≠m

Difference Formulas:
n  2, 3 points: x 0 , x 1 , x 2 . Let h  0 and x 1  x 0  h, and x 2  x 1  h  x 0  2h.

7
 P 2 x  fx 0 L 2,0 x  fx 1 L 2,1 x  fx 2 L 2,2 x,
P ′2 x  fx 0 L ′2,0 x  fx 1 L ′2,1 x  fx 3 L ′2,2 x.
x − x 1 x − x 2 
L ′2,0 x  d  1 x − x 2   x − x 1 
dx x 0 − x 1 x 0 − x 2  x 0 − x 1 x 0 − x 2 
L ′2,1 x  1 x − x 0   x − x 2 
x 1 − x 0 x 1 − x 2 
L ′2,2 x  1 x − x 0   x − x 1 
x 2 − x 0 x 2 − x 1 
x − x 2   x − x 1  x − x 0   x − x 2  x − x 0   x − x 2 
P ′2 x  fx 0   fx 1   fx 2 
x 0 − x 1 x 0 − x 2  x 1 − x 0 x 1 − x 2  x 2 − x 0 x 2 − x 1 
fx 0  fx 1  fx 2 
 x − x 2   x − x 1  − 2
x − x 0   x − x 2   x − x 0   x − x 2 
h2 h h2
fx 0  fx 1  fx 2 
P ′2 x 0   2
−2h  −h − 2
−2h  −h
2h h 2h 2
 1 − 3 fx 0   2fx 0  h − 1 fx 0  2h
h 2 2
′′′ ′′′
R ′ x 0   1 f cx−h−2h  1 h 2 f cx
3! 3
f x 0  h  1 −1 fx 0   1 fx 0  2h − 1 h 2 f cx
′ ′′′
h 2 2 3
′ ′′′
f x 0  2h  1 1 fx 0  − 2fx 1   3 fx 0  2h  1 h 2 f cx
h 2 2 3
′′′
Approximation errors are all Oh 2  if f x is bounded.

Exercises:
′ fx 0  − fx 0 − h
1. (1) Use the backward difference formula: f x 0  ≈ to approximate the derivative
2h
fx  sinx at x 0   using h  0. 1, h  0. 01 and h  0. 001.
(2) Compute the true error and an upper bound on the approximation error for each h.

2. Suppose we know that

(1) x 1 2 3 4 5 ;
fx 2. 4142 2. 6734 2. 8974 3. 0976 3. 2804
′ ′′ ′′′ 4 5
(2) for x in 1, 5, f x ≤ 2, f x ≤ 3, f x ≤ 4, f x ≤ 12, f x ≤ 23.
′
a. Approximate f 3 by
(1) an approximation using two points;
(2) an approximation using three points; and
(3) an approximation using five points..
b. Which formula you used in a. is the best? Provide fully explanation to support your answer.

3. (1) Derive the difference approximation for the first derivative:

8
 ′ fx 0  2h − fx 0 − h
f x 0  ≈ .
3h
(2) Derive the error term associated with this formula.
(3) Consider fx  lnx and x 0  2. Approximate f ′ 2 numerically by (1) with h  0. 1 and h  0. 01 and
estimate the approximation error using the error term obtained in (2) for each h.

4. (1) Derive the difference approximation for the second derivative:

fx 0  − 2fx 0  h  fx 0  2h
f ′′ x 0  ≈ .
h2
(2) Derive the error term associated with this formula.
′′
(3) Consider fx  e x and x 0  0. Approximate f 0 numerically by (1) with h  0. 1 and h  0. 01 and
estimate the approximation error using the error term obtained in (2) for each h.

5. (1) Derive the difference approximation for the second derivative:

fx 0 − 2h − 2fx 0 − h  fx 0 
f ′′ x 0  ≈ .
h2
(2) Derive the error term associated with this formula.
′′
(3) Consider fx  lnx and x 0  2. Approximate f 2 numerically by (1) with h  0. 1 and h  0. 01 and
estimate the approximation error using the error term obtained in (2) for each h.

6. Verify that each of the following difference approximations for the first derivative provides the exact value of
the derivative, regardless of h, for functions fx  1, fx  x, and fx  x 2 but not for the function
fx  x 3 .
′ −3fx 0   4fx 0  h − fx 0  2h
(1) f x 0  ≈
2h
′ 3fx 0  − 4fx 0 − h  fx 0 − 2h
(2) f x 0  ≈
2h
′ fx 0  h − fx 0 − h
(3) f x 0  ≈
2h