Lecture 23: 6.

1
Inner Products

Wei-Ta Chu

2008/12/17
Definition

 An inner product on a real vector space V is a function

that associates a real number  u, vwith each pair of
vectors u and v in V in such a way that the following
axioms are satisfied for all vectors u, v, and w in V and all
scalars k.
 
u, v= v, u
 
u + v, w= u, w+  v, w
 
ku, v= k u, v
 
u, u0 and  u, u= 0 if and only if u = 0
A real vector space with an inner product is called a real
inner product space.

2008/12/17 Elementary Linear Algebra 2

Euclidean Inner Product on Rn

 If u = (u1, u2,…, un) and v = (v1, v2, …,

vn) are vectors in
Rn, then the formula
v, u= u · v = u1v1 + u2v2 +… +unvn
defines v, uto be the Euclidean product on Rn.
 The four inner product axioms hold by Theorem 4.1.2.

2008/12/17 Elementary Linear Algebra 3

Properties of Euclidean Inner Product

 Theorem 4.1.2
 If u, v and w are vectors in Rn and k is any scalar, then
 u·v=v·u
 (u + v) · w = u · w + v · w
 (k u) · v = k (u · v)
 v · v ≥0; Further, v · v = 0 if and only if v = 0
 Example
 (3u + 2v) · (4u + v)
= (3u) · (4u + v) + (2v) · (4u + v )
= (3u) · (4u) + (3u) · v + (2v) · (4u) + (2v) · v
=12(u · u) + 11(u · v) + 2(v · v)

2008/12/17 Elementary Linear Algebra 4

Euclidean Inner Product vs. Inner
Product
 The Euclidean inner product is the most important inner
product on Rn. However, there are various applications in
which it is desirable to modify the Euclidean inner
product by weighting its terms differently.
 More precisely, if w1,w2, …, wn are positive real numbers,
and if u=(u1,u2, …, un) and v=(v1,v2,
…, vn) are vectors in
Rn, then it can be shown

which is called weighted Euclidean inner product with

weights w1,w2, …,wn …

2008/12/17 Elementary Linear Algebra 5

Example

 Suppose that some physical experiment can produce any

of n possible numerical values x1,x2, …, xn.
 We perform m repetitions of the experiment and yield
these values with various frequencies; that is, x1 occurs f1
times, x2 occurs f2 times, and so forth. f1+f2+…+fn=m.
 Thus, the arithmetic average, or mean, is

2008/12/17 Elementary Linear Algebra 6

Example

 If we let f=(f1,f2,
…,fn), x=(x1,x2,
…,xn), w1=w2=…=wn=1/m
 Then this equation can be expressed as the weighted inner
product

2008/12/17 Elementary Linear Algebra 7

Weighted Euclidean Product

 Let u = (u1, u2) and v = (v1, v2) be vectors in R2. Verify that the
weighted Euclidean inner product  u, v= 3u1v1 + 2u2v2
satisfies the four product axioms.

 Solution:
 Note first that if u and v are interchanged in this equation, the right side
remains the same. Therefore,  u, v= v, u.
 If w = (w1, w2), then

u + v, w= 3(u1+v1)w1 + 2(u2+v2)w2 =

(3u1w1 + 2u2w2) + (3v1w1 + 2v2w2)=  u, w+ v, w
which establishes the second axiom.

2008/12/17 Elementary Linear Algebra 8

Weighted Euclidean Product

 ku, v=3(ku1)v1 + 2(ku2)v2 = k(3u1v1 + 2u2v2) = k  u, v

which establishes the third axiom.
 v, v= 3v1v1+2v2v2 = 3v12 + 2v22 .Obviously ,  v, v= 3v12 +
2v22 ≥0.Fur t
hermor e ,v, v= 3v12 + 2v22 = 0 if and only if
v1 = v2 = 0, That is , if and only if v = (v1,v2)=0. Thus, the
fourth axiom is satisfied.

2008/12/17 Elementary Linear Algebra 9

Definition

 If V is an inner product space, then the norm (or length)

of a vector u in V is denoted by ||u|| and is defined by
||u|| =  ½
u, u
 The distance between two points (vectors) u and v is
denoted by d(u,v) and is defined by
d(u, v) = ||u –v||

 If a vector has norm 1, then we say that it is a unit vector.

2008/12/17 Elementary Linear Algebra 10

Norm and Distance in Rn

 If u = (u1, u2,
…, un) and v = (v1, v2,
…, vn) are vectors in
Rn with the Euclidean inner product, then

d (u, v)  u v u v, u v 1/ 2  (u v ) 

1/ 2
(u v ) 
 (u1 v1 ) 2 (u2 v2 ) 2 ... (un vn ) 2

2008/12/17 Elementary Linear Algebra 11

Weighted Euclidean Inner Product

 The norm and distance depend on the inner product used.

 If the inner product is changed, then the norms and distances
between vectors also change.
 For example, for the vectors u = (1,0) and v = (0,1) in R2 with the
Euclidean inner product, we have
u 1 and d (u, v ) u v (1,1)  12 (1) 2  2
 However, if we change to the weighted Euclidean inner product

u, v= 3u1v1 + 2u2v2 , then we obtain
1/ 2
u  u, u (3 
11 2 
00)1/ 2  3
1/ 2
d (u, v ) u v  (1,1), (1,1) [3 
11 2 
(1) 
(1)]1/ 2  5

2008/12/17 Elementary Linear Algebra 12

Unit Circles and Spheres in Inner Product
Space
 If V is an inner product space, then the set of points in V
that satisfy
||u|| = 1
is called the unite sphere or sometimes the unit circle in
V. In R2 and R3 these are the points that lie 1 unit away
form the origin.

2008/12/17 Elementary Linear Algebra 13

Unit Circles in R2

 Sketch the unit circle in an xy-coordinate system in R2 using the

Euclidean inner product  u, v= u1v1 + u2v2
 Sketch the unit circle in an xy-coordinate system in R2 using the
Euclidean inner product  u, v= 1/9u1v1 + 1/4u2v2
 Solution
 If u = (x,y), then ||u|| = 
u, u ½ = (x2 + y2)½, so the equation of the unit

circle is x2 + y2 = 1.
 If u = (x,y), then ||u|| = 
u, u ½ = (1/9x2 + 1/4y2)½, so the equation of the

unit circle is x2 /9 + y2/4 = 1.

2008/12/17 Elementary Linear Algebra 14

Inner Products Generated by Matrices
u1  v1 
u  v 
 Let u 2  and v = 2 be vectors in Rn (expressed as n1
:  : 
 

n 
u  
vn 

matrices), and let A be an invertible nn matrix.
 If u · v is the Euclidean inner product on Rn, then the formula
u, v= Au · Av
defines an inner product; it is called the inner product on Rn
generated by A.
 Recalling that the Euclidean inner product u · v can be written
as the matrix product vTu, the above formula can be written in
the alternative form  u, v= (Av) TAu, or equivalently,
u, v= vTATAu

2008/12/17 Elementary Linear Algebra 15

Inner Product Generated by the Identity
Matrix
 The inner product on Rn generated by the nn identity matrix
is the Euclidean inner product: Let A = I, we have 
u, v= Iu ·
Iv = u · v
 The weighted Euclidean inner product  u, v= 3u1v1 + 2u2v2 is
the inner product on R2 generated by
3 0 
A  

0 2

since
3 0 
3 0 
u1  3 0u1 
u , v 
v1 v2 
 
 
 v v
 1 2  
 3u1v1 2u2 v2
0 2
0 2u
2  0
 2 u
2 

2008/12/17 Elementary Linear Algebra 16

Inner Product Generated by the Identity
Matrix
 In general, the weighted Euclidean inner product 
u,
v= w1u1v1 + w2u2v2 + … +wnunvn is the inner
product on Rn generated by

2008/12/17 Elementary Linear Algebra 17

An Inner Product on M22

If U 1 u u2  v1 v2 
  and V 
u3
 u4 


v3 v4 

are any two 22 matrices, then
U, V= tr(UTV) = tr(VTU) = u1v1 + u2v2 + u3v3 + u4v4
defines an inner product on M22
 For example, if 
1 2 1 0 
U   and V 
3 4
 3 2

then 
U, V= 1(-1)+2(0) + 3(3) + 4(2) = 16
 The norm of a matrix U relative to this inner product is
U U , U 1/ 2  u12 u22 u32 u42
and the unit sphere in this space consists of all 22 matrices U
whose entries satisfy the equation ||U|| = 1, which on squaring yields
u12 + u22 + u32 + u42 = 1

2008/12/17 Elementary Linear Algebra 18

An Inner Product on P2

 If p = a0 + a1x + a2x2 and q = b0 + b1x + b2x2 are any two

vectors in P2, then the following formula defines an inner
product on P2:
p, q= a0b0 + a1b1 + a2b2
 The norm of the polynomial p relative to this inner
product is
p p, p   a0 a1 a2
1/ 2 2 2 2

and the unit sphere in this space consists of all

polynomials p in P2 whose coefficients satisfy the
equation || p || = 1, which on squaring yields
a02+ a12 + a22 =1

2008/12/17 Elementary Linear Algebra 19

Theorem 6.1.1 (Properties of Inner
Products)
 If u, v, and w are vectors in a real inner product space,
and k is any scalar, then:
 
0, v= v, 0= 0
 
u, v + w=  u, v+ 
u, w
 
u, kv=k  u, v
 
u –v, w=  u, w–v, w
 
u, v –w=  u, v–u, w

2008/12/17 Elementary Linear Algebra 20

Example

 u –2v, 3u + 4v
= u, 3u + 4v–2v, 3u+4v
= u, 3u+  u, 4v– 2v, 3u– 2v, 4v
=3 u, u+ 4  u, v–6  v, u–8 v, v
= 3 || u ||2 + 4 
u, v–6  u, v–8 || v ||2
= 3 || u ||2 –2 u, v–8 || v ||2

2008/12/17 Elementary Linear Algebra 21

Example

 We are guaranteed without any further proof that the five

properties given in Theorem 6.1.1 are true for the inner
product on Rn generated by any matrix A.
 u, v + w= (v+w)TATAu
=(vT+wT)ATAu [Property of transpose]
=(vTATAu) + (wTATAu) [Property of matrix multiplication]
=u, v+ u, w

2008/12/17 Elementary Linear Algebra 22

Lecture 23: 6.2
Angle and Orthogonality

Wei-Ta Chu

2008/12/17
Theorem 6.2.1

 Theorem 6.2.1 (Cauchy-Schwarz Inequality)

 If u and v are vectors in a real inner product space, then
|u, v | ||u|| ||v||

 The inequality can be written in the following two forms

 The Cauchy-Schwarz inequality for Rn (Theorem 4.1.3)

follows as a special case of this theorem by taking 
u, vto
be the Euclidean inner product u‧v.

2008/12/17 Elementary Linear Algebra 24

Theorem 6.2.2

 Theorem 6.2.2 (Properties of Length)

 If u and v are vectors in an inner product space V, and if k is
any scalar, then :
 || u || 0
 || u || = 0 if and only if u = 0
 || ku || = | k | || u ||
 || u + v || || u || + || v || (Triangle inequality)
 Proof of (d)
(Property of absolute value)
(Theorem 6.2.1)

2008/12/17 Elementary Linear Algebra 25

Theorem 6.2.3

 Theorem 6.2.3 (Properties of Distance)

 If u, v, and w are vectors in an inner product space V, and if
k is any scalar, then:
 d(u, v) 0
 d(u,, v)) = 0 if and only if u = v
d(
 d(u, v) = d(v, u)
 d(u, v) d(u, w) + d(w, v) (Triangle inequality)

2008/12/17 Elementary Linear Algebra 26

Angle Between Vectors

 Cauchy-Schwarz inequality can be used to define angles

in general inner product cases.

 We define to be the angle between u and v.

2008/12/17 Elementary Linear Algebra 27

Example

 Let R4 have the Euclidean inner product. Find the

cosine of the angle between the vectors u = (4, 3, 1,
-2) and v = (-2, 1, 2, 3).

2008/12/17 Elementary Linear Algebra 28

Orthogonality

 Definition
 Two vectors u and v in an inner product space are called
orthogonal if 
u, v= 0.

 Example (
U, V= tr(UTV) = tr(VTU) = u1v1 + u2v2 + u3v3 + u4v4)
 If M22 has the inner project defined previously, then the matrices

1 0 
0 2
U  and V  
1 1
 0 0

are orthogonal, since 
U, V= 1(0) + 0(2) + 1(0) + 1(0) = 0.

2008/12/17 Elementary Linear Algebra 29

Orthogonal Vectors in P2
1
 Let P2 have the inner product p, q  p ( x)q ( x)dx and let p = x and
q = x2. 1

 Then 1/ 2 1/ 2
1  1 2  2
p p, p  
1/ 2
xxdx   x dx  
1   1  3
1/ 2 1/ 2
1 2 2  1 4  2
q q, q  
1/ 2
x x dx   x dx  
1   1  5
1 1
p, q 
xx dx 
2
x 3 dx 0
1 1

because  p, q= 0, the vectors p = x and q = x 2 are orthogonal

relative to the given inner product.

2008/12/17 Elementary Linear Algebra 30

Theorem 6.2.4 (Generalized Theorem of
Pythagoras)
 If u and v are orthogonal vectors in an inner product
space, then
|| u + v ||2 = || u ||2 + || v ||2
 Proof

2008/12/17 Elementary Linear Algebra 31

Example

 Since p =1 x and q = x 2 are orthogonal relative to the inner product

p, q  p ( x)q ( x)dx on P2.
1
 It follows from the Theorem of Pythagoras that
|| p + q ||2 = || p ||2 + || q ||2
 Thus, from the previous example:
2 2 2 2 2 2 16
p +q ( ) ( ) 2   
3 5 3 5 15
 We can check this result by direct integration:
1
p +q p +q, p +q 
2
( x x 2 )( x x 2 )dx
1
1 1 1
2 2 16
 x dx 2 
2
x dx 
3
x 4 dx  0  
1 1 1
3 5 15

2008/12/17 Elementary Linear Algebra 32

Orthogonality

 Definition
 Let W be a subspace of an inner product space V. A vector u
in V is said to be orthogonal to W if it is orthogonal to every
vector in W, and the set of all vectors in V that are
orthogonal to W is called the orthogonal complement (正交
補餘)) of W.
 If V is a plane through the origin of R3 with Euclidean inner
product, then the set of all vectors that are orthogonal to
every vector in V forms the line L through the origin that is
perpendicular to V. L

2008/12/17 Elementary Linear Algebra 33

Theorem 6.2.5

 Theorem 6.2.5 (Properties of Orthogonal Complements)

 If W is a subspace of a finite-dimensional inner product
space V, then:
 Wis a subspace of V.(
read“ W perp”)
 The only vector common to W and Wis 0;; that is ,W W
W= 0..
 The orthogonal complement of Wis W; that is , (W)= W.

2008/12/17 Elementary Linear Algebra 34

Proof of Theorem 6.2.5(a)

 Note first that 

0, w =0 for every vector w in W, so W 
contains at least the zero vector.
 We want to show that the sum of two vectors in W is
orthogonal to every vector in W (closed under addition)
and that any scalar multiple of a vector in W is
orthogonal to every vector in W (closed under scalar
multiplication).

2008/12/17 Elementary Linear Algebra 35

Proof of Theorem 6.2.5(a)

 Let u and v be any vector in W , let k be any scalar, and

let w be any vector in W. Then from the definition of W ,
we have  u, w=0 and  v, w =0.
 Using the basic properties of the inner product, we have
u+v, w=  u, w+  v, w= 0 + 0 = 0
ku, w= k u, w= k(0) = 0
 Which proves that u+v and ku are in W .

2008/12/17 Elementary Linear Algebra 36

Proof of Theorem 6.2.5(b)
The only vector common to W and Wis 0; that is ,W W= 0.
 If v is common to W and W , then v, v=0, which
implies that v=0 by Axiom 4 for inner products.

2008/12/17 Elementary Linear Algebra 37

Theorem 6.2.6

 Theorem 6.2.6
 If A is an mn matrix, then:
 The nullspace of A and the row space of A are orthogonal
complements in Rn with respect to the Euclidean inner
product.
 The nullspace of AT and the column space of A are
orthogonal complements in Rm with respect to the
Euclidean inner product.

2008/12/17 Elementary Linear Algebra 38

Proof of Theorem 6.2.6(a)

 We must show that if a vector v is orthogonal to every

vector in the row space, then Av=0, and conversely, that if
Av=0, then v is orthogonal to every vector in the row
space.
 Assume that v is orthogonal to every vector in the row
space of A. Then in particular, v is orthogonal to the row
vectors r1, r2,…, rm of A; that is

2008/12/17 Elementary Linear Algebra 39

Proof of Theorem 6.2.6(a)

 The linear system Ax=0 can be expressed in dot product

notation as

 And it follows that v is a solution of this system and

hence lies in the nullspace of A.

2008/12/17 Elementary Linear Algebra 40

Proof of Theorem 6.2.6(a)

 Conversely, assume that v is a vector in the nullspace of

A, so Av=0. It follows that
 But if r is any vector in the row space of A, then r is
expressible as a linear combination of the row vectors of
A, say
 Thus

 Which proves that v is orthogonal to every vector in the

row space of A.

2008/12/17 Elementary Linear Algebra 41

Example (Basis for an Orthogonal
Complement)
 Let W be the subspace of R5 spanned  By Theorem 6.2.6, the nullspace of A
by the vectors w1=(2, 2, -1, 0, 1), is the orthogonal complement of W.
w2=(-1, -1, 2, -3, 1), w3=(1, 1, -2, 0, -  In Example 4 of Section 5.5 we
1), w4=(0, 0, 1, 1, 1). Find a basis for showed that
the orthogonal complement of W.
 Solution 1 1
1  0 
 The space W spanned by w1, w2, w3,    
and w4 is the same as the row space of
the matrix
v1 0  and v 2  1
   
0
  0 
2 2 1 0 1 
  
0  
1 

1 1 2 3 1
A   form a basis for this nullspace.
1 1 2 0 1
   Thus, vectors v1 = (-1, 1, 0, 0, 0)
0 0 1 1 1  and v2 = (-1, 0, -1, 0, 1) form a
basis for the orthogonal
complement of W.

2008/12/17 Elementary Linear Algebra 42

Remarks

 In any inner product space V, the zero space {0} and the
entire space V are orthogonal complements.
 If A is an matrix, to say that Ax=0 has only the
trivial solution is equivalent to saying that the orthogonal
complement of the nullspace of A is all of Rn, or
equivalently, that the row space of A is all of Rn.

2008/12/17 Elementary Linear Algebra 43

Theorem 6.2.7 (Equivalent
Statements)
 If A is an mn matrix, and if TA : Rn  Rn is multiplication by A, then the following are
equivalent:
 A is invertible.
 Ax = 0 has only the trivial solution.
 The reduced row-echelon form of A is In.
 A is expressible as a product of elementary matrices.
 Ax = b is consistent for every n1 matrix b.
 Ax = b has exactly one solution for every n1 matrix b.
 det(A) ≠0.
 The range of TA is Rn.
 TA is one-to-one.
 The column vectors of A are linearly independent.
 The row vectors of A are linearly independent.
 The column vectors of A span Rn.
 The row vectors of A span Rn.
 The column vectors of A form a basis for Rn.
 The row vectors of A form a basis for Rn.
 A has rank n.
 A has nullity 0.
 The orthogonal complement of the nullspace of A is Rn.
 The orthogonal complement of the row of A is {0}.

2008/12/17 Elementary Linear Algebra 44