Linear Algebra and Differential Equations

Lecture 12
Inner Product Spaces

July 7, 2016

Linear Algebra and Differential EquationsLecture 12 Inner Product

July
Spaces
7, 2016

1 / 19

Dot (Scalar) product in Rn

Up to now we havent discussed the length of a vector. To do this we will
use Pythagoras theorem . If the coordinates of u in the standard basis are
u = [v1 vn ]T

kuk2 = u12 + + un2

now if we define the dot product by

u u = kuk2
then
(u + v) (u + v) = ku + vk2 = u u + 2u v + v v
but
ku + vk2 = (u1 + v1 )2 + + (un + vn )2
= (u12 + + un2 ) + 2(u1 v1 + + un vn ) + (v12 + + vn2 )
Thus
u v = u1 v1 + + un vn
Linear Algebra and Differential EquationsLecture 12 Inner Product
July
Spaces
7, 2016

2 / 19

Angles
Consider the triangle shown below

ku vk2 = (u v) (u v) = kuk2 + kvk2 2u v

But the law of cosines for the triangle is
ku vk2 = kuk2 + kvk2 2kukkvk cos
This implies
u v = kukkvk cos
Two vectors are orthogonal if u v = 0. Geometrically this means the two
vectors are perpendicular.
Linear Algebra and Differential EquationsLecture 12 Inner Product
July
Spaces
7, 2016

3 / 19

Projection
A useful application of the scalar product is to calculate the projection of
one vector on another. The vector u can be written as the sum of a vector
parallel to v, uk (called the projection of u onto v) and a vector
orthogonal to v, u
u = uk + u = av + u
Taking the scalar product with v gives
u v = av v
Thus
uk =

u v
vv

u = u uk

Linear Algebra and Differential EquationsLecture 12 Inner Product

July
Spaces
7, 2016

4 / 19

Orthonormal vectors
A set of vectors {v1 , v2 , , vn } that are mutually orthogonal also have to
be linearly independent. If
1 v1 + 2 v2 + + n vn = 0
Taking the scalar product of this with vi
0 = vi (1 v1 + 2 v2 + + n vn ) = i |vi |2
and so all the s are zero.
If the vectors are also unit vectors (length = 1), the set is called
orthonormal. If we have an orthonormal basis, {b1 , b2 , , bn } a general
vector can be expressed in the form
u = (u b1 )b1 + (u b2 )b2 + , (u bn )bn
this is called the Fourier expansion of u.
Linear Algebra and Differential EquationsLecture 12 Inner Product
July
Spaces
7, 2016

5 / 19

Gram-Schmidt Orthogonalization
Since orthonormal bases have such nice properties, it makes sense to work
with them. We can use the projection idea to construct an orthonormal
basis from a given basis. This procedure is called Gram-Schmidt
orthogonalization.
For example let us find an orthonormal basis for the span({v1 , v2 , v3 })
where

1
1
4
1

v2 = 4 v3 = 2
v1 =
1
4
2
1
1
0
Our first basis vector is b1 = v1 then
b2 = v2

v2 b1
b1
b1 b1

Linear Algebra and Differential EquationsLecture 12 Inner Product

July
Spaces
7, 2016

6 / 19

1
4 8

b2 =
4 4
1
Now
b3 = v3
and

4
2 4

b3 =
2 4
0

1
3

1 2
=
1 2
1
1

v3 b1
v3 b2
b1
b2
b1 b1
b2 b2

1
12
1
18
1

2
= 1
2 3
1

3
5

7
5

To obtain an orthonormal basis just divide the basis vectors by their

magnitudes.

Linear Algebra and Differential EquationsLecture 12 Inner Product

July
Spaces
7, 2016

7 / 19

Generalizing the dot product

The scalar product is so useful in Rn that it makes sense to extend the
idea to general vector spaces. We need to take two vectors and generate a
number that has similar properties to the dot product. the idea is to
combine < u|, a bra, and |v >, a ket, to form a number, a bracket
< u|v >. The notation is due to Dirac.
In Rn , |v > is a column vector and < u| = |u >T is a row vector.
u v = uT v = v T u
and
u (Av) = uT Av = vT AT u = v (AT u)

Linear Algebra and Differential EquationsLecture 12 Inner Product

July
Spaces
7, 2016

8 / 19

Inner Product Space

An inner product space V is a vector space with a product
< | >: V V R which satisfies
1

< (u)|v >= < u|v >

< u|v + w >=< u|v > + < u|w >

< u|v >=< v|u >

If v 6= 0 then < v|v > 0

Using the fourth property we can define a quantity analogous to length in

Rn called the norm
kvk2 =< v|v >
and two vectors are orthogonal if
< u|v >= 0

Linear Algebra and Differential EquationsLecture 12 Inner Product

July
Spaces
7, 2016

9 / 19

An Example
Consider C[-1,1], the set of continuous functions defined on [-1,1]. Define
an inner product by
Z 1
< f |g >=
f (t)g (t) dt
1

< f |g >=

g (t)f (t) dt =< g |f >

f (t)g (t) dt =
1

< af |g >= a

f (t)g (t) dt = a < f |g >

1

< f |g + h >=

f (t)(g (t) + h(t)) dt =< f |g > + < f |h >

1

< f |f >=

(f (t))2 dt > 0

so it is an inner product.
Linear Algebra and Differential EquationsLecture 12 Inner Product
JulySpaces
7, 2016

10 / 19

Fourier series
Using the inner product in C[-1,1] used in the previous example an
orthonormal basis is given by {1, cos t, sin t, cos 2t, sin 2t, }. Using
this basis any continuous function can be expressed as a linear
combination of the basis vectors in the form

a0 X
+
(an cos nt + bn sin nt)
f (t) =
2
n=1

This is called the Fourier series for f (t). The coefficients can be obtained
by taking the inner product of f (t) with the basis vectors.
Z 1
Z 1
a0 =
f (t) dt
an =
f (t) cos nt dt
1

bn =

f (t) sin nt dt
1

Linear Algebra and Differential EquationsLecture 12 Inner Product

JulySpaces
7, 2016

11 / 19

An example of a Fourier series

Let f (t) = t then
Z

a0 =

Z
t dt = 9

an =

t cos nt dt = 0
1

bn =

t sin nt dt
1

1

 
Z 1
1
1
cos nt

cos nt dt
= t
n
n
1
1
2
=
cos n
n
 2
n odd
n
=
2
n
n even
Linear Algebra and Differential EquationsLecture 12 Inner Product
JulySpaces
7, 2016

12 / 19

Thus the Fourier series for f (t) = tis



2
1
1
t=
sin t + sin 3t + sin 4t +

3
4
Putting t = 1/2


1
2
1 1 1
=
1 + +
2

3 5 7

1 1 1
= 1 + +
4
3 5 7

A graph of the first few partial sums are shown below

Linear Algebra and Differential EquationsLecture 12 Inner Product

JulySpaces
7, 2016

13 / 19

Cauchy-Schwarz inequality
In an inner product space
| < u|v > | kuk kvk
with equality only if u and v are proportional
We can show this by
0 << u + tv|u + tv >= kuk2 + 2t < u|v > +t 2 kvk2
the expression can never be negative so the discriminant of the quadratic
expression must be negative (i.e. complex roots)
(2 < u|v >)2 4|u|2 |v|2 0
Taking square roots gives the inequality.
Linear Algebra and Differential EquationsLecture 12 Inner Product
JulySpaces
7, 2016

14 / 19

The Triangle Inequality

In an inner product space
ku + vk ku| + |vk
|u + v|2 = < u + v|u + v >
= kuk2 + 2 < u|v > +kvk2
kuk2 + 2kuk kvk + kvk2
= (kuk + kvk)2
where we have used the Cauchy-Schwarz inequality to go from the second
line to the third line above. Taking square roots gives the inequality

Linear Algebra and Differential EquationsLecture 12 Inner Product

JulySpaces
7, 2016

15 / 19

Orthogonal Complement
Let V be a subspace of an inner product space U then the orthogonal
complement of V, V , is the set of vectors that are orthogonal to every
vector in V
1

V V = U

V V = 0

(V ) = V

Every vector u in U can be expressed uniquely as

u = uk + u
with uk V and u V .

Linear Algebra and Differential EquationsLecture 12 Inner Product

JulySpaces
7, 2016

16 / 19

An Example
The null space of a matrix consists of the solutions to

Ax = 0

3 2 5 2 4

1 1 1 1 1

1 0 3 4 6

2 3 0 3 1

x1
x2
x3
x4
x5

0
=
0

Gaussian elimination reduces this to

1
0

0
0

1 1 1 1

1 2 1 1

0 0 1 1

0 0 0 0

x1
x2
x3
x4
x5

0
=
0

Linear Algebra and Differential EquationsLecture 12 Inner Product

JulySpaces
7, 2016

17 / 19

The row space is spanned by

1
1

1
1

0
1
2
1
1

0
0
0
1
1

and the null space consists of the vectors of the form

3
2
2
2

x = x3 1 + x5
0
0
1
0
1
which are orthogonal to vectors in the row space i.e.
Null space of a matrix = (Row space of the matrix)
Linear Algebra and Differential EquationsLecture 12 Inner Product
JulySpaces
7, 2016

18 / 19

If A is a m n matrix. It maps from Rn to Rm . The row space of A is the

column space for AT . The null space of A is the orthogonal complement
of the row space in Rn .
Suppose y is in the row space, that is it is in the range of AT . So there is
some x Rm with
y = AT x
then if z is in null(A)
y z = (AT x) z = x (Az) = x 0 = 0
Similarly the the column space for A is the orthogonal complement of the
null space of AT in Rm .

Linear Algebra and Differential EquationsLecture 12 Inner Product

JulySpaces
7, 2016

19 / 19