Chapter 8

Approximation Theory

Per-Olof Persson
persson@berkeley.edu

Department of Mathematics
University of California, Berkeley

Math 128B Numerical Analysis

Least Squares Approximation

Consider the approximation of the data {(xi , yi )} for i = 1, . . . , m

by a polynomial

Pn (x) = an xn + xn1 xn1 + + a1 x + a0 .

The coefficients a = (a0 , . . . , an ) are given by the solution to the

Normal Equations At Aa = At b, where

1 x1 x21 xn1

y1
1 x2 x2 xn y2
2 2
A = . b= .

.. .. ..
.. . . . ..
1 xm x2m xnm ym

The solution a is unique if all xi are unique.

Least Squares Approximation of Functions

approximating f C[a, b] by a polynomial

Consider P
Pn (x) = nk=0 ak xk which minimizes the error

n
!2
Z b X
E E(a0 , a1 , . . . , an ) = f (x) ak xk dx.
a k=0

Setting E/aj = 0 for j = 0, 1, . . . , n gives the normal equations

n
X Z b Z b
j+k
ak x dx = xj f (x) dx.
k=0 a a
Linearly independent polynomials
Definition
The set {0 , . . . , n } is said to be linearly independent on [a, b] if
c0 0 + c1 1 (x) + + cn n (x) = 0, for all x [a, b],
implies c0 = c1 = = cn = 0, otherwise it is linearly dependent.

Theorem
Suppose j (x) is a polynomial of degree j, for j = 0, . . . , n. Then
{0 , . . . , n } is linearly independent for any interval [a, b].

Definition
Q
The set of all polynomials of degree at most n is denoted n .

Theorem
Q
Suppose {0 , . . . , n } are
Q linearly independent polynomials in n.
Then any polynomial in n can be written uniquely as a linear
combination of 0 (x), . . . , n (x).
Orthogonal Functions
Definition
An integrable function w is a weight function on the interval I if
w(x) 0 for x I, but w(x) 6= 0 on any subinterval of I.

Definition
{0 , . . . , n } is said to be an orthogonal set of functions for the
interval [a, b] with respect to the weight
( function w if
Z b
0, when j 6= k,
w(x)k (x)j (x) dx =
a j > 0, when j = k.
If also j = 1 for each j = 0, . . . , n, the set is orthonormal.

Theorem
If {0 , . . . , n } is orthogonal on [a, b], then
P the least squares
approximation to f on [a, b] is P (x) = nj=0 aj j (x) where
Rb Z b
w(x)j (x)f (x) dx 1
aj = aR b = w(x)j (x)f (x) dx.
w(x)[j (x)]2 dx j a
a
Three-term recursions
Theorem
{0 , . . . , n } defined as below are orthogonal on [a, b] w.r.t. w:
Rb
xw(x)[0 (x)]2 dx
0 (x) 1, 1 (x) = x B1 , B1 = Ra b
2
a w(x)[0 (x)] dx
and when k 2,
k (x) = (x Bk )k1 (x) Ck k2 (x),
where
Rb 2
Rb
a xw(x)[k1 (x)] dx xw(x)k1 (x)k2 (x) dx
Bk = R b , Ck = a R b
2 2
a w(x)[k1 (x)] dx a w(x)[k2 (x)] dx

Corollary
{0 , . . . , n } above is linearly independent on [a, b] and for any
polynomial Qk (x) of degree k < n,
Z b
w(x)n (x)Qk (x) dx = 0
a
Legendre Polynomials

The set of Legendre polynomials {Pn (x)} is orthogonal on [1, 1]

w.r.t. the weight function w(x) = 1.

P0 (x) = 1, 0 = 2
P1 (x) = x, 1 = 2/3
1
P2 (x) = x2 , 2 = 8/45
3
3
P3 (x) = x3 x, 3 = 8/175
5
6 3
P4 (x) = x x2 + ,
4
4 = 128/11025
7 35
Chebyshev Polynomials

The Chebyshev polynomials are orthogonal on (1, 1) w.r.t.

w(x) = (1 x2 )1/2 , and are defined by:
Tn (x) = cos[n arccos x], n0
Three-term recurrence:
T0 (x) = 1, T1 (x) = x, Tn+1 (x) = 2xTn (x) Tn1 (x)

Theorem
Tn (x) has n simple zeros in [1, 1] at
 
2k 1
xk = cos , k = 1, 2, . . . , n
2n
and its extrema at
 
0 k
xk = cos with Tn (x0k ) = (1)k , k = 0, 1, . . . , n
n
Monic Chebyshev Polynomials

Define the monic Chebyshev polynomials Tn (x) by dividing Tn (x)

by its leading coefficient 2n1 :
1
T0 (x) = 1, and Tn (x) = Tn (x), n1
2n1
Three-term recurrence: T0 (x) = 1, T1 (x) = x, T2 (x) = x2 1/2,
1
Tn+1 (x) = xTn (x) Tn1 (x), n2
4

Clearly, the zeros of Tn (x) are the same as for Tn (x):

 
2k 1
xk = cos
2n

and the extreme values are at

(1)k
 
0 k
xk = cos , with Tn (x0k ) = n1 , k = 0, 1, . . . , n
n 2
Minimization property
Theorem
Q
Let e n denote the set of all monic polynomials of degree n. The
polynomials Tn (x), when n 1 have the property:
1 Y
= max |T (x)| max |P (x)|, for all P (x)
f
n1 n n n
2 x[1,1] x[1,1] n

Corollary
Suppose P (x) is the interpolating polynomial of degree n with
nodes at the zeros of Tn+1 (x). Then
1
max |f (x) P (x)| n max |f (n+1) (x)|,
x[1,1] 2 (n + 1)! x[1,1]
for f C n+1 [1, 1].

Extend to general intervals [a, b] by the change of variables

1
x = [(b a)x + a + b]
2
Reducing the degree of Approximating Polynomials

Consider the approximation of an nth degree polynomial

Pn (x) = an xn + an1 xn1 + + a1 x + a0

on [1, 1] by a polynomial Pn1 of degree n 1, in the sense

that it minimizes

max |Pn (x) Pn1 (x)|

x[1,1]

This occus when

Pn1 (x) = Pn (x) an Tn (x)

giving a minimum max-error of

|an |
max |Pn (x) Pn1 (x)| =
x[1,1] 2n1
Pade Approximation

Find approximation to a function f on an interval I by a

rational function r of degree N = n + m:
p(x) p0 + p1 x + + pn xn
r(x) = =
q(x) q0 + q1 x + + qm x m
Normalize by setting q0 = 1 = N + 1 parameters
Determine coefficients as for Taylor polynomials, i.e., by
imposing the condition
f (k) (0) = r(k) (0), for each k = 0, 1, . . . , N
n = N , m = 0 gives the N th Maclaurin polynomial
If f has the Maclaurin series expansion f (x) = i
P
i=0 ai x , the
coefficients are found by solving the linear system
Xk
ai qki = pk , k = 0, 1, . . . , N
i=0
for the N + 1 unknowns q1 , . . . , qm , p0 , . . . , pn
Chebyshev Rational Function Approximation

Use Chebyshev polynomials for more uniform accuracy

Find N th degree rational function r of the form
Pn
pk Tk (x)
r(x) = Pk=0m , where N = n + m and q0 = 1
k=0 qk Tk (x)

If f has the Chebyshev expansion f (x) =

P
k=0 ak Tk (x), the
coefficients are found by matching terms so that

(a0 T0 (x) + a1 T1 (x) + )(T0 (x) + q1 T1 (x) + + qm Tm (x))

(p0 T0 (x) + p1 T1 (x) + + pn Tn (x))

has no terms of degree N

Calculate the products of Chebyshev polynomials using
1 
Ti (x)Tj (x) = Ti+j (x) + T|ij| (x)
2
Chebyshev Expansion Coefficients

The Chebyshev coefficients,

1 1 f (x) 2 1 f (x)Tk (x)

Z Z
a0 = dx and ak = dx
1 1 x2 1 1 x2
are usually hard to calculate analytically
Change variable to = arccos x:

1 2
Z Z
a0 = f (cos ) d, ak = f (cos ) cos k d
0 0

and use numerical quadrature

Orthogonal Trigonometric Polynomials

For each integer n > 0, the set {0 , 1 , . . . , 2n1 }, where

0 (x) = 1/2
k (x) = cos kx, for each k = 1, 2 . . . , n,
n+k (x) = sin kx, for each k = 1, 2, . . . , n 1.
is orthogonal on [, ] with respect to w(x) = 1.

Trigonometric polynomials: Tn = span({0 , 1 , . . . , 2n1 }).

Least square approximation of f C[, ] by functions in TN :
n1
a0 X
Sn (x) = + an cos nx + (ak cos kx + bk sin kx)
2
k=1
1
Z
ak = f (x) cos kx dx, k = 0, 1, . . . , n

Z
1
bk = f (x) sin kx dx, k = 1, . . . , n 1

Sn when n is the Fourier series of f .
Discrete Trigonometric Approximation

Consider the 2m points {(xj , yj )}2m1

j=0 , with

j
xj = + , j = 0, . . . , 2m 1
m
Find the trigonometric polynomial Sn Tn that minimizes
2m1
X
E(Sn ) = [yj Sn (xj )]2
j=0

Simplified by discrete orthogonality:

2m1
X
k (xj )l (xj ) = 0
j=0
Discrete Trigonometric Approximation

Lemma
If the integer r is not a multiple of 2m:
2m1
X 2m1
X
cos rxj = 0 and sin rxj = 0
j=0 j=0

If the integer r is not a multiple of m:

2m1
X 2m1
X
(cos rxj )2 = m and (sin rxj )2 = m
j=0 j=0
Discrete Trigonometric Approximation

Theorem
The trigonometric polynomial
2m1
( " n1
#)2
X a0 X
Sn (x) = yj + an cos nxj + (ak cos kxj + bk sin kxj )
j=0
2
k=1

that minimizes the least squares sum

2m1
X
E(a0 , . . . , an , b1 , . . . , bn1 ) = (yj Sn (xj ))2
j=0
are
2m1
1 X
ak = yj cos kxj , k = 0, 1, . . . , n
m
j=0
2m1
1 X
bk = yj sin kxj , k = 1, . . . , n 1
m
j=0
Interpolatory Trigonometric Polynomials

The interpolatory trigonometric polynomial in Tm on the 2m

2m1
points {(xj , yj )}j=0 ,
j
xj = + , j = 0, . . . , 2m 1
m
is
m1
a0 + am cos mx X
Sm (x) = + (ak cos kx + bk sin kx)
2
k=1
where
2m1
1 X
ak = yj cos kxj , for k = 0, . . . , m,
m
j=0
2m1
1 X
bk = yj sin kxj , for k = 1, . . . , m 1.
m
j=0
Complex Form of the DFT

Consider the complex coefficients ck in

2m1
1 X
ck eikx
m
k=0

where
2m1
X
ck = yj eikj/m , for k = 0, . . . , 2m 1
j=0

ak and bk can be recovered from ck using Eulers formula

eiz = cos z + i sin z

The Fast Fourier Transform

Split the DFT into even and odd indices:

2m1
X m1
X m1
X
ck = yj eikj/m = y2j eik2j/m + y2j+1 eik(2j+1)/m
j=0 j=0 j=0
m1
X m1
X
= y2j eikj/(m/2) + eik/m y2j+1 eikj/(m/2)
j=0 j=0
ik/m
= Ek + e Ok

where Ek is the DFT of the even index inputs x2j and Ok is

the DFT of the odd index inputs x2j+1 .
But

ei(k+m)/m = ei eik/m = eik/m

The Fast Fourier Transform

The FFT algorithm can then be written:

(
Ek + eik/m Ok if k < m,
ck = ik/m
Ekm e Okm if k m

Use recursion, O(m log m) total work