Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

MATH 4A - Linear Algebra with Applications

Lecture 26: Orthogonal projections

3 June 2019

Reading: §6.3-6.4
Recommended problems from §6.3: 1-13 odd, 19, 21, 22, 23
Announcement: please fill out online evaluations!
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Lecture plan

1 Finishing up w/ orthogonal sets

2 Orthogonal projections onto lines

3 General orthogonal projections

Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Recall: life is better with orthogonal bases

If B = {b1 , b2 , . . . , bn } is an orthogonal basis of Rn , then there are

explicit formulas for writing v as a linear combination of B. More
generally, we have the following:
Theorem
Let {u1 , u2 , . . . , up } be an orthogonal basis for a subspace W of
Rn . For each y in W , the weights in the linear combination

y = c1 u1 + · · · + cp up

are given by
y · uj
cj =
uj · uj
for each j = 1, . . . , p.
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

iClicker
Recall from Friday that the set of vectors B consisting of
     
3 −1 −1/2
u1 = 1 u2 =  2  u3 =  −2 
1 1 7/2
is an orthogonal basis of R3 . What is the u2 coordinate of
 
1
v = 2

1
in the B-basis?
(a) uu22·u·v2
(b) v·u
v·v
2

(c) uv·u
2 ·u2
1

v·u2
(d) u2 ·u2
(e) uv·u
3 ·u3
2
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Orthonormal bases

There is one more thing we can impose on an orthogonal set to

make it even nicer: require each of the vectors to have unit length.

A set {u1 , . . . , up } is an orthonormal set if it is an orthogonal set

of unit vectors. If W is spanned by such a set, we call the set
{u1 , . . . , up } an orthonormal basis of W .
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Important properties of orthonormal sets

1 An m × n matrix U has orthonormal columns if and only if

UT U = I .
2 (Ux) · (Uy) = x · y for any two vectors x and y in Rn . In
particular:
(i) kUxk = kxk for any vector x in Rn .
(ii) (Ux) · (Uy) = 0 if and only if x · y = 0
Intuition: when m = n, we call a matrix U with orthonormal
columns an orthogonal matrix (although a better name would be
orthonormal matrix...). The transformation x 7→ Ux is a rotation
of Rn .
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Examples

The standard basis of Rn is an orthonormal basis:

InT In = In In = In .

The set consisting of the vectors

 √   √   √ 
3/√11 −1/√ 6 −1/√66
v1 = 1/√11 v2 =  2/√6  v3 = −4/√ 66
1/ 11 1/ 6 7/ 66

is orthonormal. It forms an orthonormal basis of R3 .

Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Motivation: component analysis

It can be very useful to “resolve” a vector into a sum of orthogonal

components, e.g. when trying to figure out how fast a box slides
down a ramp. See board.
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

The mathematical problem

Given a nonzero vector u in Rn , we want to decompose another

vector y into the sum of two vectors, one a multiple of u and the
other orthogonal to u. That is, we want to write

y = ŷ + z,

where ŷ = αu for some α and z is orthogonal to u.

Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Deriving the solution

“z is orthogonal to u” means z · u = 0. But

z = y − ŷ = y − αu,

so
(y − αu) · u = y · u − αu · u = 0,
and we conclude
y·u
. α=
u·u
(Note: we assumed u 6= 0, so we’re not dividing by 0 here.)
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

The answer: orthogonal projection onto a line

Given a nonzero vector u and another vector y, we call
y·u
ŷ = u
u·u
the orthogonal projection of y onto u, and
z = y − ŷ
the component of y orthogonal to u.

Remark: the formula for ŷ depends on u, but the answer is the

same for any nonzero scalar multiple of u:
y · (ku) k2 y · u
(ku) = 2 u = ŷ.
(ku) · (ku) k u·u
In other words, ŷ only depends on the line L spanned by u. For
this reason, we sometimes call ŷ the orthogonal projection of y
onto u, and use the equivalent notation projL y = ŷ.
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

iClicker 1
   
9 −1
Let u= and y = . What is the projection of y onto
−2 4
the line spanned by u?
 
85 9
(a) −17 −2
 
−17 −1
(b) 85 4
 
−17 9
(c) 85 −2
 
85 −1
(d) −17 4
 
68/85
(e)
306/85
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

iClicker 2
  
9 −1
Let u = and y = . What is the component of y
−2 4
orthogonal to u? (Hint: use the answer to the previous question.)
 
918/85
(a)
−204/85
 
−17 −1
(b) 85 4
 
−17 9
(c) 85 −2
 
85 −1
(d) −17 4
 
68/85
(e)
306/85
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Example: orthognal projection onto subspace

Let {u1 , . . . , u4 } be any orthogonal basis of R4 and let

y = c1 u1 + · · · + c4 u4 .

Consider the subspace W = Span{u2 , u4 }. Write y as a sum of a

vector z1 in W , and a vector z2 in W ⊥ .

See board.
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Orthogonal decomposition theorem

Theorem
Let W be a subspace of Rn . Then each y in Rn can be written
uniquely in the form
y = ŷ + z
where ŷ is in W and z is in W ⊥ . In fact, if {u1 , . . . , up } is any
orthogonal basis of W , then
y · u1 y · up
ŷ = u1 + · · · + up
u1 · u1 up · up

and z = y − ŷ.

We call ŷ the orthogonal projection of y onto W , and sometimes

write it projW y.
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Why we care

It requires a bit of work to prove, but here is why the previous

theorem is so important:
projW y is the point in W that is closest to y. (See Theorem 9 in
the textbook for a precise formulation.)

This fact means orthogonal projection shows up in many real world

applications, since experiments and data have errors.
Finishing up w/ orthogonal sets Orthogonal projections onto lines General orthogonal projections

Example

Let      
−2 −1 3
u1 =  0  u2 =  1  y = 2 .
1 2 1
Let’s find the orthogonal decomposition of y with respect to the
subspace W = Span{u1 , u2 }.