Advanced Computational Fluid Dynamics

AA215A Lecture 1
Vector and Function Spaces

Antony Jameson

Winter Quarter, 2016, Stanford, CA

Last revised on January 7, 2016
Contents

1 Vector and Function Spaces 2

1.1 Basic concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Convex and concave functions: Jensen’s theorem . . . . . . . . . . . . . . . . . . . . 2
1.3 The generalized AM-GM inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Hölder’s inequality and the Cauchy-Schwarz inequality . . . . . . . . . . . . . . . . . 3
1.5 Vector Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.5.1 Minkowski’s inequality - triangle inequality for the p norm . . . . . . . . . . . 5
1.5.2 Equivalence of Norms for Finite Dimensional Vectors . . . . . . . . . . . . . . 5
1.6 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6.1 Infinity Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6.2 1 Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6.3 2 Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7 Norms of Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7.1 Function Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.7.2 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.7.3 Non-equivalence of Norms for Functions . . . . . . . . . . . . . . . . . . . . . 9
1.8 Inner Products of Functions: Orthogonality . . . . . . . . . . . . . . . . . . . . . . . 10
1.8.1 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8.2 Triangle Inequality for the ∞ and 1 Norms . . . . . . . . . . . . . . . . . . . 10
1.8.3 Cauchy Schwarz Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8.4 Triangle Inequality for the Euclidean Norm . . . . . . . . . . . . . . . . . . . 11
1.8.5 Pythagorean Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8.6 Linear Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8.7 Weierstrass Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.9 Portraits of Courant and Hilbert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

1
Lecture 1

Vector and Function Spaces

1.1 Basic concepts

The well known concepts of Euclidean space can be generalized to n dimensional vectors and also
to functions. These concepts are extremely useful in the analysis of numerical methods and are
briefly reviewed in this appendix. The idea of function space was one of the great achievements of
nineteenth century mathematics, particularly due to Banach and Hilbert. Hilbert was the first to
introduce spaces with abstract inner product. As a preliminary, it is useful to derive some basic
inequalities due to Jensen, Hölder, Cauchy and Schwarz.

1.2 Convex and concave functions: Jensen’s theorem

A function of f (x) of a real variable x is convex if

f (tx1 + (1 − t)x2 ) ≤ tf (x1 ) + (1 − t)f (x2 ), (1.1)

and concave if
f (tx1 + (1 − t)x2 ) ≥ tf (x1 ) + (1 − t)f (x2 ), (1.2)
whenever 0 < t < 1. It is strictly convex or concave if equality in these expressions implies x1 = x2 .
If f is twice differentiable, it is convex if f 00 ≥ 0 and concave if f 00 ≤ 0, and strictly convex or
concave if f 00 > 0 or f 00 < 0.
Jensen’s theorem:
If f (x) is a concave function, Ã n !
Xn X
ti f (xi ) ≤ f ti xi , (1.3)
i=1 i=1
Pn
whenever t1 , ..., tn ∈ (0, 1) and i=1 ti = 1.
This may be proved by induction. It is true for n = 2 by the definition of a concave function.
Suppose n ≥ 3 and the assertion holds for smaller values of n. Then, for i = 2, ..., n, set t0i =

2
LECTURE 1. VECTOR AND FUNCTION SPACES 3

Pn 0
ti /(1 − t1 ), so that i=2 ti = 1. Then,
n
X n
X
ti f (x1 ) = t1 f (x1 ) + (1 − t1 ) t0i f (xi )
i=1 i=2
à n !
X
≤ t1 f (x1 ) + (1 − t1 )f t0i xi
i=2
à n
!
X
0
≤f t1 x1 + (1 − t1 ) ti xi
i=2
à n
!
X
=f ti xi
i=1

1.3 The generalized AM-GM inequality

The geometric mean (GM) of n real numbers does not exceed the arithmetic mean (AM). This
result follows from Jensen’s
P theorem applied to the function log x, which is strictly concave Thus
if p1 , ..., pn > 0 and ni=1 pi = 1
n
à n !
X X
pi log ai ≤ log pi ai (1.4)
i=1 i=1

and hence
n
Y n
X
api i ≤ pi a i (1.5)
i=1 i=1
1
This is known as the generalized AM-GM inequality. Setting pi = n, recovers the basic AM-GM
inequality
à n !1 n
Y n
1X
ai ≤ ai (1.6)
n
i=1 i=1

1.4 Hölder’s inequality and the Cauchy-Schwarz inequality

1 1
Suppose p, q > 1 and p + q = 1. Then
¯ ¯ Ã !1 Ã n !1
¯Xn ¯ n
X p X q
¯ ¯ p q
¯ ak bk ¯ ≤ |ak | |bk | (1.7)
¯ ¯
k=1 k=1 k=1

Set x1 = ap , x2 = bq , p1 = p1 , p2 = 1q . By the generalized GM-AM theorem

ap bq
ab = xp11 xp22 ≤ p1 x1 + p2 x2 = + (1.8)
p q
Take vectors such that
n
X n
X
|ak |p = |bk |q = 1 (1.9)
k=1 k=1
LECTURE 1. VECTOR AND FUNCTION SPACES 4

Then ¯ n ¯
¯X ¯ X n n µ
X ¶
¯ ¯ |ak |p |bk |q 1 1
¯ a b
k k¯ ≤ |a b
k k | ≤ + = + =1 (1.10)
¯ ¯ p q p q
k=1 k=1 k=1

When p = 2 this reduces to the Cauchy-Schwartz inequality:

¯ n ¯ Ã n !1 Ã n !1
¯X ¯ X 2 X 2
¯ ¯
¯ ak bk ¯ ≤ a2k b2k (1.11)
¯ ¯
k=1 k=1 k=1

This may be proved directly by noting that for any value of ρ

n
X
(ak + ρbk )(ak + ρbk ) ≥ 0 (1.12)
k=1

and consequently ¯ n ¯
n
X ¯X ¯ n
X
2 ¯ ¯ 2
ak + 2ρ ¯ ak bk ¯ + ρ b2k ≥ 0 (1.13)
¯ ¯
k=1 k=1 k=1

Then setting P
| nk=1 ak bk |
ρ = − Pn 2 (1.14)
k=1 bk
Pn 2
and multiplying by k=1 bk à !à ! ¯ n ¯
n
X n
X ¯X ¯
¯ ¯
a2k b2k ≥¯ ak bk ¯ (1.15)
¯ ¯
k=1 k=1 k=1

1.5 Vector Norms

The size of an n-dimensional vector can be conveniently represented by a variety of measures. Such
measures are called norms, which are required to satisfy the following axioms:
To each vector x assign a number ||x|| where

1. ||x|| ≥ 0

2. ||αx|| = |α|||x|| for scalar α

3. ||x + y|| ≤ ||x|| + ||y|| (triangle inequality)

4. ||x|| = 0 if and only if x = 0

Some widely used norms are

P 1
• ||x||p = ( |xi |p ) p for p = 1, 2, 3, ...∞
P
• ||x||1 = |xi |
¡P ¢1
• ||x||2 = |xi |2 2

• ||x||∞ = max |xi |

i
LECTURE 1. VECTOR AND FUNCTION SPACES 5

Conditions (1) and (2) are evident. For (3)

P P
• ||x + y||1 = |xi + yi | ≤ (|xi | + |yi |) ≤ ||x||1 + ||y||1

• ||x + y||∞ = max |xi + yi | ≤ max |xi | + max |yi | ≤ ||x||∞ + ||y||∞
i i i

To verify (3) for ||x||2 use Cauchy-Schwartz inequality

|xT y| ≤ ||x|| ||y|| (1.16)

Then

¡ ¢1
||x + y||2 = (x + y)T (x + y) 2
¡ T T T
¢ 12 ³ T T 1
T 1
T
´1
2
≤ x x + 2|y x| + y y ≤ x x + 2(x x) (y y) + y y
2 2

= ||x||2 + ||y||2

1.5.1 Minkowski’s inequality - triangle inequality for the p norm

||x + y||p ≤ ||x||p + ||y||p (1.17)

Proof:
X X X
|xk + yk |p ≤ |xk + yk |p−1 |xk | + |xk + yk |p−1 |yk | (1.18)
Then by Hölder’s inequality
X ³X ´ 1 µ³X ´1 ³X ´1 ¶
p (p−1)q q p p p p
|xk + yk | ≤ |xk + yk | |xk | + |yk |
³X ´ 1 µ³X ´1 ³X ´1 ¶
p q p p
= |xk + yk | |xk |p + |yk |p (1.19)

since p − 1 = pq . Divide both sides by

³X ´1
q
|xk + yk |p (1.20)

to obtain the triangle inequality.

1.5.2 Equivalence of Norms for Finite Dimensional Vectors

For finite dimensional vectors all norms are equivalent in the sense that for 2 norms M (x) and
N (x) there exist constants c1 , c2 such that for all x

c1 M (x) ≤ N (x) ≤ c2 M (x) (1.21)

1 1
N (x) ≤ M (x) ≤ N (x) (1.22)
c2 c1
LECTURE 1. VECTOR AND FUNCTION SPACES 6

This need only be proved for M (x) = N∞ (x) since if M and N are both equivalent to N∞ then
they are also equivalent to each other. Suppose that

c1 N∞ ≤ M ≤ c2 N∞ (1.23)

and
d1 N∞ ≤ N ≤ d2 N∞ (1.24)
Then
c1 N ≤ d2 M (1.25)
and
d1 M ≤ c2 N (1.26)
where
c1 c2
N ≤M ≤ N (1.27)
d2 d1
Consider the unit cell S of vectors for which Ns (x) = 1. Let x0 and x1 be elements such that

N (x0 ) = min N (x), N (x1 ) = max N (x) (1.28)

x∈S x∈S

(These exist since N (x) is a continuous function of the elements of x.)

y
Now for any y, is in S, so
N∞ (y)
µ ¶
y
N (x0 ) ≤ N ≤ N (x1 ) (1.29)
N∞ (y)
or
N (x0 )N∞ (y) ≤ N (y) ≤ N (x1 )N∞ (y) (1.30)
This is the desired result with
c1 = N (x0 ), c2 = N (x1 ) (1.31)

1.6 Matrix Norms

||Ax||
The induced norm of a matrix is defined as ||A|| = sup .
x6=0 ||x||
||Ax|| is a continuous function of ||x|| so with ||x|| = 1

||A|| = sup ||Ax|| (1.32)

where the max is attained.

Let x be such that with ||x|| = 1

||A + B|| = ||(A + B)x|| ≤ ||Ax|| + ||Bx|| ≤ ||A|| + ||B|| (1.33)

so triangle inequality is satisfied. Also note that

||ABx|| ≤ ||A|| ||Bx|| ≤ ||A|| ||B|| ||x|| (1.34)

so that
||AB|| ≤ ||A|| ||B|| (1.35)
LECTURE 1. VECTOR AND FUNCTION SPACES 7

1.6.1 Infinity Norm

Corresponding to ||x||∞ we have
X
||A||∞ = max |aij | (max absolute row sum) (1.36)
i
j

since ¯ ¯ ¯ ¯ ¯ ¯
¯X ¯ ¯X ¯ ¯X ¯
||Ax||∞ ¯
= max ¯ ¯ ¯
aij xj ¯ ≤ max ¯ ¯ ¯
aij ¯ max |xj | ≤ max ¯ aij ¯¯ ||x||∞ (1.37)
i i j i
j j j

where the max is attained with xj = sgn(aij ).

1.6.2 1 Norm
Corresponding to ||x||1 we have
X
||A||1 = max |aij | (max absolute column sum) (1.38)
j
i

since
à !
X ¯¯ X ¯
¯ XX X X
||Ax||1 = ¯ aij xj ¯¯ ≤ |aij ||xj | = |aij | |xj |
¯
i j i j j i
à ! à !
X X X
≤ max |aij | |xj | = max |aij | ||x||1 (1.39)
j j
i j i

with the max attained when x = ej , where j is the index of the max sum.

1.6.3 2 Norm
Corresponding to ||x||2 we have
||Ax||22 xH AH Ax
= (1.40)
||x||22 xH x
where AH is the Hermitian transpose of A. P
Let ui be an eigenvector of AH A with real eigenvalue σ 2 . Set x = αi ui , then
P 2 2
xH AH Ax α σ
H
= P i 2 i ≤ σ12 (1.41)
x x αi

where σ12 is the largest eigenvalue of AH A. The decomposition is possible because ui are indepen-
dent.

1.7 Norms of Functions

1.7.1 Function Space
For considering errors we need to measure differences between functions. It is convenient to re-
gard functions as vectors in an infinite dimensional space. It turns out that many of the familiar
LECTURE 1. VECTOR AND FUNCTION SPACES 8

concepts of three dimensional Euclidean geometry carry over to function spaces. In particular we
can introduce norms to measure the size of a function, and an abstract definition of a generalized
inner product of two functions, corresponding to the scalar product of two vectors. This leads to
the concept of orthogonal functions. The introduction of axiomatic definition leads to proofs which
hold for a variety of different norms and inner products. In order to enable standard arguments of
real analysis to be carried over, function spaces are required to satisfy an axiom of completeness
that the limit of a sequence of elements in a given space is contained in the space. Spaces with an
inner product are known as Hilbert spaces in honor of their inventor.

1.7.2 Norms
Norms will be a measure of the size of a function, regarded as a vector. For example
µZ 1 ¶ 12
2
||f || = f dx (1.42)
0

is a generalization of
à 3
! 12
X
||f || = fi2 dx (1.43)
1
for Euclidean space.
To be more precise we require the norm of a function to satisfy the following axioms:

1. ||f || ≥ 0

2. ||αf || = |α|||f || for scalar α

3. ||f + g|| ≤ ||f || + ||g||

4. ||f || = 0 if and only if f = 0

If a function is given by a table

fi = f (xi ) for i = 0, 1, 2, ..., n (1.44)

it may be convenient to use as a measure

à n
!1
X 2

||f || = fi2 dx (1.45)

0

Then we can have

||f || = 0 for f 6= 0 (1.46)
This measure satisfies the first 3 axioms only. Such a measure is called a semi-norm.
Examples of norms are:

||f ||∞ = max |f (x)| Maximum norm (1.47)

x∈[a,b]

µZ b ¶ 21
2
||f ||2 = f (x)dx Euclidean norm (1.48)
a
LECTURE 1. VECTOR AND FUNCTION SPACES 9

Z b
||f ||1 = |f (x)|dx (1.49)
a
These are special cases of
µZ b ¶ p1
p
||f ||p = |f (x)| dx for p ≥ 1 (1.50)
a

Note that, for example, we can regard ||f ||2 as the limit of
à n
!1
1X 2 2

||f || = fi dx (1.51)
n
0

where fi is the table of values representing f (x).

We can also introduce weighted norms such as the weighted Euclidean norm
µZ b ¶ 12
||f || = f 2 (x)w(x)dx (1.52)
a

where w(x) is a non-negative weight function.

All these norms can be shown to satisfy the axioms.

1.7.3 Non-equivalence of Norms for Functions

For functions norms are not equivalent. If p > q then

||f ||p → 0 implies ||f ||q → 0 (1.53)

but not the other way round.

For example consider ½ 1 1
1 if − 2n ≤x≤ 2n
fn = (1.54)
0 otherwise
Then Z 1
1
fn2 dx = (1.55)
−1 n
and
1
||fn ||2 = √ → 0 as n → ∞ (1.56)
n
while
||fn ||∞ = 1 (1.57)
On the other hand if
||f ||∞ = ² (1.58)
then Z b
f 2 dx ≤ ²2 (b − a) (1.59)
a
so √
||f ||2 ≤ ² b − a → 0 as ² → 0 (1.60)
LECTURE 1. VECTOR AND FUNCTION SPACES 10

1.8 Inner Products of Functions: Orthogonality

1.8.1 Orthogonality
It is also convenient to introduce the idea of orthogonality in function space. Define the inner
product as
Z b
(f, g) = f (x) g(x) w(x) dx in the continuous case
a
n
X
= fi gi wi in the discrete case (1.61)
i=0

Then if (f, g) vanishes the functions are said to be orthogonal.

We have
(f, g) = (g, f ) commutative
(c1 f + c2 g, Φ) = c1 (f, Φ) + c2 (g, Φ) linearity (1.62)
(f, f ) ≥ 0 positivity
These can be taken as axioms for an abstract inner product.

1.8.2 Triangle Inequality for the ∞ and 1 Norms

We have
max |f + g| ≤ max(|f | + |g|) ≤ max |f | + max |g| (1.63)
and Z Z
|f + g|dx ≤ (|f | + |g|)dx (1.64)

Thus the ∞ and 1 norms satisfy the triangle inequality. They also already satisfy axioms (1), (2)
and (4).

1.8.3 Cauchy Schwarz Inequality

(f, g)2 ≤ (f, f )(g, g) (1.65)

Proof:

0 ≤ (f + pg, f + pg) ≤ (f, f ) + 2p|(f, g)| + p2 (g, g) (1.66)

But since this is a quadratic function of p it has imaginary roots or coincident real roots. The
discriminant yields the desired inequality.
Or set
|(f, g)|
p=− (1.67)
(g, g)
and multiply by (g, g) to get

0 ≤ (f, f ) (g, g) − 2|(f, g)|2 + |(f, g)|2 (1.68)

LECTURE 1. VECTOR AND FUNCTION SPACES 11

1.8.4 Triangle Inequality for the Euclidean Norm

1
||f + g||2 = (f + g, f + g) 2
1
≤ ((f, f ) + 2|(f, g)| + (g, g)) 2
³ 1 1
´1 (1.69)
2
≤ (f, f ) + 2(f, f ) 2 (g, g) 2 + (g, g)
= ||f || + ||g||
using the Schwartz inequality.

1.8.5 Pythagorean Theorem

In the Euclidean norm, if (f, g) = 0 then

||f + g||2 = ||f ||2 + ||g||2 (1.70)

Proof:

||f + g||2 = (f + g, f + g) = (f, f ) + (f, g) + (g, f ) + (g, g) (1.71)

1.8.6 Linear Independence

A set of functions φ0 , φ1 , ..., φn is said to be linearly independent if
¯¯X ¯¯
¯¯ ¯¯
¯¯ ci φi ¯¯ = 0 (1.72)

implies
ci = 0 for i = 0, 1, ..., n (1.73)
Orthogonal functions are independent since then from the Pythagorean theorem
¯¯ n ¯¯2
¯¯X ¯¯ n
X
¯¯ ¯¯
¯¯ ci φi ¯¯ = c2i ||φi ||2 (1.74)
¯¯ ¯¯
i=0 i=0

1.8.7 Weierstrass Theorem

Let f (x) be given in interval [a, b]. Let the lower bound of the error in the maximum norm for all
polynomials of order n be
En (f ) = min ||f − pn (x)||∞ (1.75)
pn (x)

Then if f is continuous
lim En (f ) = 0 (1.76)
n→∞
That is, a continuous function can be arbitrarily well approximated in a closed interval by a
polynomial of sufficiently high order.
The proof is by construction of the required pn (x) (Isaacson and Keller, p183).

1.9 Portraits of Courant and Hilbert

LECTURE 1. VECTOR AND FUNCTION SPACES 12

Figure 1.1: Richard Courant (1888-1972)

Figure 1.2: David Hilbert (1862-1943)

Advanced Computational Fluid Dynamics
AA215A Lecture 2
Approximation Theory

Antony Jameson

Winter Quarter, 2016, Stanford, CA

Last revised on January 7, 2016
Contents

2 Approximation Theory 2
2.1 Least Squares Approximation and Projection . . . . . . . . . . . . . . . . . . . . . . 2
2.2 Alternative Proof of Least Squares Solution . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 Bessel’s Inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.4 Orthonormal Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.5 Construction of Orthogonal Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.6 Orthogonal Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.7 Zeros of Orthogonal Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.8 Legendre Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.9 Chebyshev Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.10 Best Approximation Polynomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.11 Best Interpolating Polynomial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.12 Jacobi Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.13 Fourier Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.14 Error Estimate for Fourier Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.15 Orthogonality Relations for Discrete Fourier Series . . . . . . . . . . . . . . . . . . . 11
2.16 Trigonometric Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.17 Error Estimate for Trigonometric Interpolation . . . . . . . . . . . . . . . . . . . . . 13
2.18 Fourier Cosine Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.19 Cosine Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.20 Chebyshev Expansions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.21 Chebyshev Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.22 Portraits of Fourier, Legendre and Chebyshev . . . . . . . . . . . . . . . . . . . . . . 17

1
Lecture 2

Approximation Theory

2.1 Least Squares Approximation and Projection

Suppose that we wish to approximate a given function f by a linear combination of independent
basis functions φj , j = 0, .., n. In general we wish to choose coefficients cj to minimize the error
n
X
||f − cj φj || (2.1)
j=0

in some norm. Ideally we might use the infinity norm, but it turns out it is computationally very
expensive to do this, requiring an iterative process. The best known method for finding the best
approximation in the infinity norm is the exchange algorithm (M.J.D.Powell) On the other hand
it is very easy to calculate the best approximation in the Euclidean norm as follows. Choose cj to
minimize the least squares integral
 2
Z Xn
J(c0 , c1 , c2 , ...cn ) =  cj φj − f  dx (2.2)
j=0

equivalent to minimizing the Euclidean norm. Then we require

 
Z Xn
∂J
0= = 2 cj φj − f  φi dx (2.3)
∂ci
j=0

Thus the best approximation is

n
X
f∗ = c∗j φj (2.4)
j=0

Where  
n
X
 c∗j φj − f, φi  = (f ∗ − f, φi ) = 0, for i = 0, 1, 2, ..., n (2.5)
j=0

or
n
X
aij c∗j = bi (2.6)
j=0

2
LECTURE 2. APPROXIMATION THEORY 3

where
aij = (φi , φj ), bi = (f, φi ) (2.7)
Notice that since f∗ is a linear combination of the φj these equations state that
(f − f ∗ , f ∗ ) = 0 (2.8)
Thus the least squares approximation f ∗ is orthogonal to the error f − f ∗ . This means that f ∗ is
the projection of f onto the space spanned by the basis functions φj , in the same way that a three
dimensional vector might be projected onto a plane.
If the φj form an orthonormal set,
aij = δij (2.9)
where δij = 1 if i = j and 0 if i 6= j, and we have directly
c∗i = (f, φi ) (2.10)

2.2 Alternative Proof of Least Squares Solution

Let X
f∗ = c∗j φj (2.11)
be the best approximation and consider another approximation
X
cj φj (2.12)

Then if f − f ∗ is orthogonal to all φj

¯¯X ¯¯2 ¯¯X ¯¯2
¯¯ ¯¯ ¯¯ ¯¯
¯¯ cj φj − f ¯¯ = ¯¯ (cj − c∗j )φj − (f − f ∗ )¯¯ (2.13)
¯¯X ¯¯2
¯¯ ¯¯
= ¯¯ (cj − c∗j )φj ¯¯ + ||(f − f ∗ )||2 (2.14)
≥ ||(f − f ∗ )||2 (2.15)

Recall that in the Lecture Notes || · || = || · ||2 = (·, ·)1/2

2.3 Bessel’s Inequality

Since f ∗ is orthogonal to f − f ∗ the Pythagorean theorem gives
||f ||2 = ||f ∗ + f − f ∗ ||2 = ||f ∗ ||2 + ||f − f ∗ ||2 (2.16)
Also if the φj are orthonormal, X
||f ∗ ||2 = c∗j 2 (2.17)
so we have Bessel’s inequality X
c∗j 2 ≤ ||f ||2 (2.18)
P
Thus the series c2j is convergent and if ||f ∗ − f || → 0 as n → ∞ we have Parseval’s formula
X 2
c∗j = ||f ||2 (2.19)

This is the case for orthogonal polynomials because of the Weiesstrass theorem.
LECTURE 2. APPROXIMATION THEORY 4

2.4 Orthonormal Systems

The functions φi form an orthonormal set if they are orthogonal and ||φi || = 1. We can write

(φi , φj ) = δij (2.20)

where δij = 1 if i = j and 0 if i 6= j.

An example of an orthogonal set is

φj = cos(jx), for j = 0, 1, ..., n (2.21)

on the inerval [0, π]. Then if j 6= k

Z π Z π
1
(φj , φk ) = cos(jx) cos(kx) dx = (cos(j − k)x + cos(j + k)x) dx = 0 (2.22)
0 0 2
Also Z π Z π
2 1 π
(φj , φj ) = cos (jx)dx = (1 + cos(2jx)) dx = (2.23)
0 0 2 2

2.5 Construction of Orthogonal Functions

This can be done by Gram Schmidt orthogonalization. Define the inner product
Z b
(f, g) = f (x)g(x)dx (2.24)
a

Functions gi are called independent if

n
X
αi gi = 0 implies αi = 0 for i = 0, ..., n (2.25)
i=0

Let gi (x), i = 0, 1, ..., n, be any set of independent functions. Then set

f0 (x) = d0 g0 (x) (2.26)

f1 (x) = d1 (g1 (x) − c01 f0 (x)) (2.27)
..
.
fn (x) = dn (gn (x) − c0n f0 (x)... − cn−1,n fn−1 (x)) (2.28)

We require
(fi , fj ) = δij (2.29)
This gives
1
d0 = p (2.30)
(g0 , g0 )
Then 0 = (f0 , f1 ) = d1 ((f0 , g1 ) − c01 ) and in general

cjk = (fj , gk ) (2.31)

while the dk are easily obtained from

(fk , fk ) = 1 (2.32)
LECTURE 2. APPROXIMATION THEORY 5

2.6 Orthogonal Polynomials

A similar approach can be used to construct orthogonal polynomials. Given orthogonal polynomials,
φj (x), j = 0, 1, . . . , n, we can generate a polynomial of degree n + 1 by multiplying φn (x) by x,
and we can make it orthogonal to the previous φj (x) by subtracting a linear combination of these
polynomials. Thus, we set
n
X
φn+1 (x) = αn xφn (x) − cni φi (x) (2.33)
i=0
Now,
n
X
αn (xφn , φj ) − cni (φi , φj ) = 0 (2.34)
i=0
But (φi , φj ) = 0 for i 6= j so

cnj ||φj ||2 = αn (xφn , φj ) = αn (φn , xφj ) (2.35)

Since xφj is a polynomial of order j + 1 this vanishes except for j = n − 1, n.

Thus
φn+1 (x) = αxφn (x) − cnn φn (x) − cn,n−1 φn−1 (x) (2.36)
where

αn (φn , xφn ) αn (φn , xφn−1 )

cnn = , cn,n−1 = (2.37)
||φn ||2 ||φn−1 ||2
and α is arbitrary.

2.7 Zeros of Orthogonal Polynomials

An orthogonal polynomial Pn (x) for the interval [a, b] has n distinct zeros in [a, b]. Suppose it had
only k < n zeros x1 , x2 , ..., xk at which Pn (x) changes sign. Then consider

Qk (x) = (x − x1 ) ... (x − xk ) (2.38)

where for k = 0 we define Q0 (x) = 1. Then Qk (x) changes sign at the same points as Pn (x), and
Z b
Pn (x)Qk (x)w(x)dx 6= 0 (2.39)
a

But this is impossible since Pn (x) is orthogonal to all polynomials of degree < n.

2.8 Legendre Polynomials

These are orthogonal in the interval [−1, 1] with a constant weight function w(x) = 1. They can
be generated from Rodrigues’ formula

L0 (x) = 1 (2.40)
1 dn £ ¤
Ln (x) = (x2 − 1) n
, for n = 1, 2, ... (2.41)
2n n! dxn
LECTURE 2. APPROXIMATION THEORY 6

The first few are

L0 = 1
L1 (x) = x
1
L2 (x) = (3x2 − 1)
2
1 (2.42)
L3 (x) = (5x3 − 3x)
2
1
L4 (x) = (35x4 − 30x2 + 3)
8
1
L5 (x) = (63x5 − 70x3 + 15x)
8
where they can be successively generated for n ≥ 2 by the recurrence formula
2n + 1 n
Ln+1 = xLn (x) − Ln−1 (x) (2.43)
n+1 n+1
They are alternately odd and even and are normalized so that

Ln (1) = 1 , Ln (−1) = (−1)n (2.44)

while for |x| < 1

|Ln (x)| < 1 (2.45)
Also Z ½
1
0 if n 6= j
(Ln , Lj ) = Ln (x)Lj (x)dx = 2 (2.46)
−1 2n+1 if n = j
Writing the recurrence relation as
2n − 1 n−1
Ln (x) = xLn−1 (x) − Ln−2 (x) (2.47)
n n
it can be seen that the leading coefficient of Ln (x) is multiplied by (2n − 1)/n when Ln (x) is
generated. Hence the leading coefficient is

1 · 3 · 5 · ... · (2n − 1) 1 2n!

an = = n (2.48)
1 · 2 · 3 · ... · n 2 (n!)2

Ln (x) also satisfies the differential equation

· ¸
d ¡ ¢
2 d
1−x Ln (x) + n(n + 1)Ln (x) = 0 (2.49)
dx dx

2.9 Chebyshev Polynomials

Let
x = cos φ, 0 ≤ φ ≤ π (2.50)
and define
Tn (x) = cos(nφ) (2.51)
LECTURE 2. APPROXIMATION THEORY 7

in the interval [-1,1]. Since

cos(n + 1)φ + cos(n − 1)φ = 2 cos φ cos nφ, n ≥ 1 (2.52)

we have

T0 (x) = 1 (2.53)
T1 (x) = x (2.54)
..
.
Tn+1 (x) = 2xTn (x) − Tn−1 (x) (2.55)

These are the Chebyshev polynomials.

Property 1:
Leading coefficient is 2n−1 for n ≥ 1
Property 2:

Tn (−x) = (−1)n Tn (x) (2.56)

Property 3:
Tn (x) has n zeros at µ ¶
2k − 1 π
xk = cos for k = 1, ..., n (2.57)
n 2
and n + 1 extrema in [−1, 1] with the values (−1)k , at

kπ
x0k = cos for k = 0, ..., n (2.58)
n
Property 4 (Continuous orthogonality):
They are orthogonal in the inner product
Z 1
dx
(f, g) = f (x)g(x) √ (2.59)
−1 1 − x2
since 
Z π  0 if r 6= s
π
(Tr , Ts ) = cos rθ cos sθ dθ = 2 if r = s 6= 0 (2.60)
0 
π if r = s = 0
Property 5 (Discrete orthogonality):
They are orthogonal in the discrete inner product
n
X
(f, g) = f (xj )g(xj ) (2.61)
j=0

where xj are the zeros of Tn+1 (x) and

2j + 1 π
arccos xj = θj = , j = 0, 1, ..., n (2.62)
2n + 1 2
LECTURE 2. APPROXIMATION THEORY 8

Then 
n
X  0 if r 6= s
n+1
(Tr , Ts ) = cos rθj cos sθj = 2 if r = s 6= 0 (2.63)

j=0 n + 1 if r = s = 0
Property
¡ n−1 ¢ 6 (Minimax):
1/2 Tn (x) has smallest maximum norm in [−1, 1] of all polynomials with leading coefficient
unity.
Proof:
Suppose ||pn (x)||∞ were smaller. Now at the n + 1 extrema x0 , ..., xn of Tn (x), which all have the
same magnitude of unity,
1
pn (xn ) < Tn (xn ) (2.64)
2n−1
1
pn (xn−1 ) > Tn (xn−1 ) (2.65)
2n−1
..
.
¡ ¢
Thus pn (x) − 1/2n−1 Tn (x) changes sign n times in [−1, 1], but this is impossible since it is of
degree n − 1 and has n − 1 roots.

2.10 Best Approximation Polynomial

Let f (x) − Pn (x) have maximum deviations, +e, −e, alternately at n + 2 points x0 , ..., xn+1 in [a, b].
Then Pn (x) minimizes ||f (x) − Pn (x)||∞
Proof: Suppose ||f (x) − Qn (x)||∞ < |e|, then we have at x0 , ..., xn+1

Qn (x) − Pn (x) = f (x) − Pn (x) − (f (x) − Qn (x)) (2.66)

has same sign as f (x) − Pn (x). Thus it has opposite sign at n + 2 points, giving n + 1 sign changes.
But this is impossible since it has only n roots.

2.11 Best Interpolating Polynomial

The remainder for the Lagrange interpolation polynomial is

f n+1 (ξ)
f (x) − Pn (x) = wn (x) (2.67)
(n + 1)!

where
wn (x) = (x − x0 )(x − x1 ) ... (x − xn ) (2.68)
Thus we can minimize ||wn (x)||∞ in [−1, 1] by making xk the zeros of Tn+1 (x). Then
µ ¶n
1
wn (x) = Tn+1 (x) (2.69)
2
µ ¶n
1
||wn (x)||∞ = (2.70)
2
LECTURE 2. APPROXIMATION THEORY 9

2.12 Jacobi Polynomials

(α,β)
Jacobi polynomials Pn (x) are orthogonal in the interval [−1, 1] for the weight function

w(x) = (1 − x)α (1 + x)β (2.71)

Thus the family of Jacobi polynomials includes both the Legendre polynomials

Ln (x) = Pn(0,0) (x) (2.72)

and the Chebyshev polynomials

(− 1 ,− 1 )
Tn (x) = Pn 2 2 (x) (2.73)

2.13 Fourier Series

Let f (θ) be periodic with period 2π and square integrable on [0, 2π]. Consider the approximation
of f (θ)
Xn
1
Sn (θ) = a0 + (ar cos rθ + br sin rθ) (2.74)
2
r=1

Let us minimize ½Z ¾1
2π ¡ ¢2 2
||f − Sn ||2 = f (θ) − Sn (θ) dθ (2.75)
0
Let
J(a0 , a1 , ..., an , b1 , ..., bn ) = ||f − Sn ||2 (2.76)
The trigonometric functions satisfy the orthogonality relations
Z 2π ½
0 if r 6= s
cos(rθ) cos(sθ)dθ = (2.77)
0 π if r = s 6= 0
Z 2π ½
0 if r =
6 s
sin(rθ) sin(sθ)dθ = (2.78)
0 π if r = s 6= 0
Z 2π
sin(rθ) cos(sθ)dθ = 0 (2.79)
0
At the minimum allowing for these
Z 2π Z 2π
∂J
0= = −2 f (θ) cos(rθ)dθ + 2ar cos2 (rθ)dθ (2.80)
∂ar 0 0

whence Z 2π
1
ar = f (θ) cos(rθ)dθ (2.81)
π 0
and similarly Z 2π
1
br = f (θ) sin(rθ)dθ (2.82)
π 0
LECTURE 2. APPROXIMATION THEORY 10

Also since ||f − Sn ||2 ≥ 0 we obtain Bessel’s inequality

n Z
1 2 X¡ 2 ¢ 1 2π
a0 + ar + b2r ≤ f 2 (θ)dθ (2.83)
2 π 0
r=1

and since the right hand side is independent of n the sum converges and

lim |ar | = 0, lim |br | = 0, (2.84)

r→∞ r→∞

Also
lim ||f − Sn ||2 = 0 (2.85)
n→∞

If not, it would violate Weierstrass’s theorem translated to trigonometric functions (Isaacson and
Keller, p 230, p 198).

2.14 Error Estimate for Fourier Series

Let f (θ) have K continuous derivatives. Then the Fourier coefficients decay as
µ ¶ µ ¶
1 1
|ar | = O K
, |br | = O (2.86)
r rK
and µ ¶
1
|f (θ) − Sn (θ)| = O (2.87)
nK−1
Integrating repeatedly by parts
Z 2π
1
ar = f (θ) cos(rθ)dθ
π 0
Z 2π
1
=− f 0 (θ) sin(rθ)dθ
rπ 0 (2.88)
Z 2π
1
= 2 f 00 (θ) cos(rθ)dθ
r π 0
..
.

Let M = sup |f (K) (θ)|. Then since | cos(rθ)| ≤ 1 and | sin(rθ)| ≤ 1

2M
|ar | ≤ (2.89)
rK
and similarly
2M
|br | ≤ (2.90)
rK
LECTURE 2. APPROXIMATION THEORY 11

Also ¯ m ¯
¯ X ¯
¯ ¯
|Sm (θ) − Sn (θ)| ≤ ¯ (ar cos rθ + br sin rθ)¯
¯ ¯
r=n+1
Xm
4M
≤
rK (2.91)
r=n+1
Z ∞
dξ
≤ 4M
n ξK
4M 1
=
K − 1 nK−1
where the integral contains the sum of the rectangles shown in the sketch.

Thus the partial sums Sn (θ) form a Cauchy sequence and converge uniformly to f (θ), and

4M 1
|f (θ) − Sn (θ)| ≤ K−1
(2.92)
K −1n

2.15 Orthogonality Relations for Discrete Fourier Series

Consider
2N
X −1 2N
X −1
irθj
S= e = ω rj (2.93)
j=0 j=0

where
π
ω = ei N , ω 2N = 1 (2.94)
Then ½
1 − ω 2N r 0 if ω r 6= 1
S= = (2.95)
1 − ωr 2N if ω r = 1
LECTURE 2. APPROXIMATION THEORY 12

Therefore
2N
X −1 ½
irθj −isθj 0 if |r − s| 6= 0, 2N, 4N
e e = (2.96)
2N if |r − s| = 0, 2N, 4N
j=0

Taking the real part

2N
X −1
cos(r − s)θj = 0 if |r − s| 6= 0, 2N, 4N (2.97)
j=0
2N
X −1
cos(r + s)θj = 0 if |r + s| 6= 0, 2N, 4N (2.98)
j=0
so

2N
X −1
1
2N
X −1
¡ ¢  0 if r 6= s
cos rθj cos sθj = cos(r − s)θj + cos(r + s)θj = N if r = s 6= 0, N (2.99)
2 
j=0 j=0 2N if r = s = 0, N
P −1 rj
Note that 2Nj=0 w is the sum of 2N equally spaced unit vectors as sketched, except in the case
when ω r = 1.

2.16 Trigonometric Interpolation

Instead of finding a least square fit we can calculate the coefficients of the sum so that the sum
exactly fits the function at equal intervals around the circle. Let Ar and Br be chosen so that
X n−1
1 1
Un (θ) = A0 + (Ar cos rθ + Br sin rθ) + An cos nθ = f (θ) (2.100)
2 2
r=1

at the 2n + 1 points
j
θj = 2π (2.101)
2n
It is easy to verify that

2n−1
X  0 if r 6= s
cos rθj cos sθj = n if r = s 6= 0, n (2.102)

j=0 2n if r = s = 0, n

2n−1
X  0 if r 6= s
sin rθj sin sθj = n if r = s 6= 0, n (2.103)

j=0 0 if r = s = 0, n
LECTURE 2. APPROXIMATION THEORY 13

2n−1
X
sin rθj cos sθj = 0 (2.104)
j=0
It follows on multiplying through by cos rθj or sin rθj and summing over θj that
2n−1
1 X
Ar = f (θj ) cos rθj (2.105)
n
j=0

2n−1
1 X
Br = f (θj ) sin rθj (2.106)
n
j=0
for r = 1, ..., n. Note that writing
1
c0 = A0 (2.107)
2
1
cr = (Ar − iBr ) (2.108)
2
1
c−r = (Ar + iBr ) (2.109)
2
the sum can be expressed as
n−1
X n
X
Un (θ) = cr eirθ = ’c eirθ (2.110)
r
r=−n r=−n

where
2n−1
1 X
cr = f (θj )e−irθj (2.111)
2n
j=0
P
and ’ is defined as
n
X n
X
’a = 1
r ar − (an + a−n ) (2.112)
r=−n r=−n
2

2.17 Error Estimate for Trigonometric Interpolation

In order to estimate the error of the interpolation we compare the interpolation coefficients Ar and
Br with the Fourier coefficients ar and br . The total error can then be estimated as the sum of the
error in the truncated Fourier terms and the additional error introduced by replacing the Fourier
coefficients by the interpolation coefficients. The interpolation coefficients can be expressed in
terms of the Fourier coefficients by substituting the infinite Fourier series for f (θ) into the formula
for the interpolation coefficients. For this purpose it is convenient to use the complex form. Thus
2n−1
1 X
Cr = f (θj )e−irθj (2.113)
2n
j=0
2n−1 ∞
1 X −irθj X
= e ck eikθj (2.114)
2n
j=0 k=−∞
2n−1 ∞
1 X X
= ck e−i(r−k)θj (2.115)
2n
j=0 k=−∞
LECTURE 2. APPROXIMATION THEORY 14

Here
X ½
i(r−k)θj 2n if r − k = 2qn where q = 0, 1, ...
e = (2.116)
0 if r − k 6= 2qn
Hence
∞
X
Cr = cr + (c2kn+r + c2kn−r ) (2.117)
k=1
and similarly
∞
X
C−r = c−r + (c2kn+r + c2kn−r ) (2.118)
k=1
Accordingly
∞
X
Ar = Cr + C−r = ar + (a2kn+r + a2knr−r ) (2.119)
k=1
and similarly
∞
X
Br = i(Cr − C−r ) = br + (b2kn+r − b2kn−r ) (2.120)
k=1
Thus the difference between the interpolation and Fourier coefficients can be seen to be an aliasing
error in which all the higher harmonics are lumped into the base modes.
Now we can estimate |Ar − ar |, |Br − br | using
2M 2M
|ar | ≤ K
, |br | ≤ K . (2.121)
r r
We get
∞
X¯ ¯
¯ 1 1 ¯
|Ar − ar | ≤ ¯
2M ¯ + ¯
(2kn − r)K (2kn + r)K ¯
k=1
¯ ¯ (2.122)
2M X 1 ¯¯ ¯
∞
1 1 ¯
= ¯ ¡ ¢ + ¡ ¢ ¯
(2n)K kK ¯ 1 − r K 1 + r K¯
k=1 2kn 2kn
Since r ≤ n this is bounded by
∞ µ Z ∞ ¶ µ ¶
2M X 1 k 2(2K + 1)M dξ 2(2K + 1)M 1
(2 + 1) ≤ 1+ ≤ 1+ (2.123)
(2n)K kK 2K nK 1 ξK 2K nK K −1
k=1

Using a similar estimate for Br we get

5M 5M
|Ar − ar | < , |Br − br | < K (2.124)
nK n
for K ≥ 2. Finally we can estimate the error of the interpolation sum un (θ) as

|f (θ) − un (θ)| ≤ |f (θ) − Sn (θ)| + |Sn (θ) − un (θ)| (2.125)

Xn ½ ¾
4M 1
< + |Ar − ar | + |Br − br | (2.126)
K − 1 nK−1
r=0
4M 1 10M (n + 1)
< + (2.127)
K − 1 nK−1 nK
LECTURE 2. APPROXIMATION THEORY 15

2.18 Fourier Cosine Series

For a function defined for 0 ≤ θ ≤ π the Fourier cosine series is
∞
X
Sc (θ) = ar cos rθ (2.128)
r=0

where Z π
1
a0 = f (θ)dθ (2.129)
π 0
Z π
2
ar = f (θ) cos(rθ)dθ, r > 0 (2.130)
π 0
d
Hence since dθ cos(rθ) = 0 at θ = 0, π the function will not be well approximated at 0, π unless
0 0
f (θ) = f (π) = 0. The cosine series in fact implies an even continuation at θ = 0, π.
Now if all derivatives of f (θ) of order 0, 1, ..., K −1, are continuous, and f (p) (0) = f (p) (π) = 0 for
all odd p < K and f (K) (θ) is integrable, then integration by parts from the coefficients of f (K) (θ)
gives
M
|ar | ≤ k (2.131)
r

2.19 Cosine Interpolation

Let f (θ) be approximated by
n
X
Un (θ) = ar cos rθ (2.132)
0

and let
Un (θj ) = f (θj ) for j = 0, 1, ...n (2.133)
where
2j + 1 π
θj = (2.134)
n+1 2
Then

n
X  0 if 0 ≤ r =
6 s≤n
n+1
cos rθj cos sθj = 2 if 0 ≤ r = s ≤ n (2.135)

j=0 n + 1 if 0 = r = s
Now if we multiply the first equation by cos sθj and sum, we find that
n
2 X
ar = f (θj ) cos rθj for 0 < r < n (2.136)
n+1
j=0

n
1 X
a0 = f (θj ) (2.137)
n+1
j=0
LECTURE 2. APPROXIMATION THEORY 16

2.20 Chebyshev Expansions

Let
∞
X
g(x) = ar Tr (x) (2.138)
r=0

approximate f (x) in the interval [−1, 1]. Then G(θ) = g(cos θ) is the Fourier cosine series for
F (θ) = f (cos θ) for 0 ≤ θ ≤ π, since
Tr (cos θ) = cos rθ (2.139)
∞
X
G(θ) = g(cos θ) = ar cos rθ (2.140)
r=0

Thus Z Z
π 1
1 1 dx
a0 = f (cos θ)dθ = f (x) √ (2.141)
π 0 π −1 1 − x2
and Z Z
π 1
2 2 dx
ar = f (cos θ) cos rθdθ = f (x)Tr (x) √ (2.142)
π 0 π −1 1 − x2
Now from the theory of Fourier cosine series if f (p) (x) is continuous for |x| ≤ 1 and p = 0, 1, ..., K −1,
and f (K) (x) is integrable
M
|ar | ≤ K (2.143)
r
¡ 1 ¢
Since |Tr (x)| ≤ 1 for |x| ≤ 1 the remainder after n terms is O nK−1 . Now

F 0 (θ) = −f 0 (cos θ) sin θ (2.144)

F 00 (θ) = f 00 (cos θ) sin2 θ − f 0 (cos θ) cos θ (2.145)

F 000 (θ) = −f 000 (cos θ) sin3 θ + 3f 00 (cos θ) sin θ cos θ + f 0 (cos θ) sin θ (2.146)
and in general, if F (p) is bounded,

F (p) (0) = F (p) (π) = 0 (2.147)

when p is odd, since then F (p) contains the term sin θ. As a result, the favorable error estimate
applies, provided that derivatives of f exist up to the required order.

2.21 Chebyshev Interpolation

We can transform cosine interpolation to interpolation with Chebyshev polynomials by setting
n
X
Gn (x) = ar Tr (x) (2.148)
r=0

and choosing the coefficients so that

Gn (xj ) = f (xj ) for j = 0, 1, ...n (2.149)

LECTURE 2. APPROXIMATION THEORY 17

where µ ¶
2j + 1 π
xj = cos θj = cos (2.150)
n+1 2
These are the zeros of Tn+1 (x). Now for any k < n we can find the discrete least squares ap-
proximation. When k = n, because of the above equation, the error is zero, so the least squares
approximation is the interpolation polynomial. Moreover, since Gn (x) is a polynomial of degree n,
this is the Lagrange interpolation polynomial at the zeros of Tn+1 (x).

2.22 Portraits of Fourier, Legendre and Chebyshev

Figure 2.1: Joseph Fourier (1768-1830)

LECTURE 2. APPROXIMATION THEORY 18

Figure 2.2: Adrien-Marie Legendre (1752-1833)

Figure 2.3: Pafnuty Lvovich Chebyshev (1821-1894)