Chapter 5 (5.1 - 5.2) Orthogonality

Chapter 5 Orthogonality
(5.1~5.2)
( 感謝中華大學資訊工程系李建興老師免費提供講義母本 )
5.1 The Scalar Product in R n
• The scalar product of two n×1 matrices x and y is

the 1×1 matrix xTy, or simply regarded as a real
number. That is, if x = (x1, x2, …, xn)T and y = (y1, y2,
…, yn)T, then
 y1 
y 
xT y  x1 x2  xn   2   x1 y1  x2 y 2  . . .  xn y n
 
y 
 n
2
Example 1
 3  4
If x   2 and y  3, then xT y  3  4  (-2)  3  1 2  8
   
 1 2
3
The Scalar Product in R and R 2 3
• Vectors in R2 and R3 can be represented by directed

line segments.
• The Euclidean length of a vector x in either R2 or R3
can be defined in terms of the scalar product:
 x 2
 x 2
if x  R 2
x  (xT x)1/2   12 22
x
 1  x 2  x3
2
if x  R 3
4
• The angle between two vectors is defined as the angle
 between the line segments.
• The distance between the vectors is measured by the
length of the vector joining the terminal point of x
and the terminal point of y
5
Figure 5.1.1
6
Definition
Let x and y be vectors in either R2 or R3. The distance
between x and y is defined to be the number ||x – y||.
7
Example 2
 3  1
• If x    and y    , then the distance between x
 4  7
and y is given by
 1 3  4
yx       
 7  4  3
|| y  x ||  (1  3) 2  (7  4) 2  5
8
Theorem 5.1.1
If x and y are two nonzero vectors in either R2 or R3 and
 is the angle between them, then
xTy = ||x|| ||y|| cos 
Figure 5.1.1
9
Theorem 5.1.1 proof
• By the law of cosines,
||y – x||2 = ||x||2 + ||y||2 – 2 ||x||||y|| cos 
or ||x||||y|| cos 
= ½(||x||2 + ||y||2 – ||y – x||2)
= ½(||x||2 + ||y||2 – (y – x)T( y – x))
= ½(||x||2 + ||y||2 – (yTy – yTx – xTy + xTx))
= ½(2 xTy)
= xTy
10
• If x is a nonzero vector, then we can form the unit
vector u of x as u  1 x
x
• If x and y are two nonzero vectors, u and v are the

unit vectors of x and y:
1 1
u x and v  y
x y
then the angle  between x and y is
xT y
cos   uT v
x y
11
Example 3
 3  1
• If x    and y   , then the directions of these two
 4  7
vectors are given by the unit vectors:
1  53  1  5 -12 
u x   4  and v  y 7 
x 5 y 5 2 
3 1 4 7 25 1
cos  u v  
T
      /4
5 5 2 5 5 2 25 2 2
12
Corollary 5.1.2
(Cauchy-Schwarz Inequality)
If x and y are vectors in either R2 or R3, then
|xTy|  ||x|| ||y||
with equality holding if and only if one of the vectors is
0 or one vector is a multiple of the other.
• Proof: hint: xTy = ||x|| ||y|| cos 
13
Definition
The vectors x and y are in R2 (or R3) are said to be
orthogonal if xTy = 0.
14
Example 4
(a) The vector 0 is orthogonal to every vector in R2.
(b) The vectors (3, 2)T and (-4, 6)T are orthogonal in R2.
(c) The vectors (2, -3, 1)T and (1, 1, 1)T are orthogonal
in R3.
15
Scalar and Vector Projections
• Let x and y be in either R2 or R3, then x can be
represented as p+z, where p is in the direction of y
and z is orthogonal to p.
16
• Let u = (1/||y||)y, thus u is a unit vector (length 1) in
the direction of y. We wish to find  such that p = u
and is orthogonal to z = x – u. Thus
x y cosθ xT y
α  x cosθ   ,
y y
 is called the scalar projection of x onto y and p is

called the vector projection of x onto y:
y xT y y xT y xT y
p  αu  α   2y T y
y y y y y y
17
Example 5
• Find the point Q on the line y = 1/3 x that is closest to
the point (1, 4).
18
Example 5
• Sol:
3 1  vT w 7 3  2.1
Set w   , v   , then Q  2
w   
1  4 w 10 1 0.7 
 Q  2.1, 0.7  is the closest point
19
Notation
If P1 and P2 are two points in 3-space, we will denote
the vector from P1 to P2 by P1 P2
20
• If N is a nonzero vector and P0 is a fixed point, the set
of points P such that P1 P2 is orthogonal to N forms a
plane  in 3-space that passes through P0. The vector
N and the plane  are said to be normal to each other.
A point P = (x, y, z) will lie on  if and only if
T
( P0 P ) N  0
If N = (a, b, c)T and P0 = (x0, y0, z0), the above

equation can be written as
a(x – x0) + b(y – y0) + c(z – z0) = 0
21
Example 6
• Find the equation of the plane passing through the
point (2, -1, 3) and normal to the vector N = (2, 3, 4)T.
• Sol:
P0 P  ( x  2, y  1, z  3)T
The equation is ( P0 P )T N  0
So, 2( x  2)  3( y  1)  4( z  3)  0
22
Orthogonality in R n
• If x  Rn then the Euclidean length of x is defined

by
x  (xT x)1/2  ( x12  x22    xn2 )1/2
• If x and y are two vectors in Rn, then the distance
between x and y is ||y – x||
• The Cauchy-Schwarz inequality holds in Rn (will be
shown in Section 4):
xT y
1  1
x y
23
• The angle  between two vectors x and y in Rn is
given by xT y
cosθ  ,0θπ
x y
• If u and v are the unit vectors of x and y:
1 1
u x and v  y
x y
then the angle  between u and v is the same as the
angle between x and y:
xT y
cosθ   uT v
x y
24
• The cosine can be computed by simply taking the
scalar product of the two unit vectors.
• Two vectors x and y in Rn are said to be orthogonal
if xTy = 0 and often the symbol “” is used to indicate
orthogonality.
• If x and y are orthogonal, we will write x  y
25
• If x and y are vectors in Rn, then
2
xy
 ( x  y )T ( x  y )
 ( xT  y T ) ( x  y )
 xT x  xT y  y T x  y T y
2 2
 x  2xT y  y
26
• If x and y are orthogonal, the above equation
becomes the Pythagorean Law:
2 2 2
xy  x  y
27
5.2 Orthogonal Subspaces
• Let A be an m×n matrix and let x  N(A), the null
space of A.
• Ax = 0, i.e.,
 a11 a12  a1n   x1  0
a a22  a 2 n   x2   0 
 21      
        
a am 2  amn   xn  0
 m1
ai1 x1  ai 2 x2  ...  ain xn  0, for i  1,, m
28
• Note that x is orthogonal to the ith column vector of
AT for i = 1, 2, …, m
• Therefore, x is orthogonal to any linear combination
of the column vector of AT
• So if y is any vector in the column space of AT, then
xTy = 0
• Conclusion: Each vector in N(A) is orthogonal to
every vector in the column space of AT
29
Definition
Two subspaces X and Y of Rn are said to be orthogonal
if xTy = 0 for every xX and every yY. If X and Y are
orthogonal, we write X  Y.
30
Example 1
• Let X be the subspace of R3 spanned by e1 and Y be
the subspace of R3 spanned by e2, if xX and yY,
then
1   α  0  0 
x  αe1  α 0   0  and y  βe 2  β 1  β 
       
0  0  0 0
0 
 xT y   α 0 0  β   0  X  Y
 
0
31
• The concept of orthogonal subspaces does not always
agree with our intuitive idea of perpendicularity.
– For example, the floor and wall of the classroom “look”
orthogonal, but the xy-plane and the yz-plane are not
orthogonal subspaces
– Think of the vectors x1 = (1, 1, 0)T and x2 = (0, 1, 1)T lying
in the xy-plane and the yz-plane, respectively, then
0 
x1T x 2   1 1 0  1  1
 
1
– These two subspaces are not orthogonal!
32
Example 2
• Let X be the subspace of R3 spanned by e1 and e2, and
let Y be the subspace spanned by e3, if xX and yY,
then 1  0   α  0  0 
x  αe1  βe 2  α 0  β 1   β  and y  γe 3  γ 0  0
         
0 0  0  1  γ 
0 
 x T y   α β 0  0   0  X  Y
 
 γ 
33
• Furthermore, if z = (z1, z2, z3)T is any vector in R3 that
is orthogonal to every vector in Y, then z  e3, and
hence 0 
z T e3  z1 z2 z 3  0   z 3  0
 
1
34
• If z3 = 0, then z  X
• X is the set of all vectors in R3 that is orthogonal to
every vectors in Y
Figure 5.2.1
35
Definition
Let Y be a subspace of Rn. The set of all vectors in Rn
that are orthogonal to every vector in Y will be denoted
Y. Thus
Y = {xRn | xTy = 0 for every yY}
The set Y is called the orthogonal complement of Y.
36
Note
• The subspaces X = Span(e1) and Y = Span(e2) of R3
are orthogonal, but they are not orthogonal
complement. Indeed,
X = Span(e2, e3) and Y = Span(e1, e3)
37
Remarks
1. If X and Y are orthogonal subspaces of Rn, then
X∩Y={0}.
2. If Y is a subspace of Rn, then Y is also a subspace of
Rn.
Pf:
1. If x  X∩Y and XY, then ||x||2 = xTx = 0  x = 0.
38
• Pf:
2. If xY and  is a scalar, then for any y  Y,
(x)Ty = (xTy) = 0 = 0
 xY
If x1 and x2 are elements of Y, then
(x1 + x2)Ty = x1Ty + x2Ty = 0 + 0 = 0
 (x1 + x2)Y
Therefore, Y is a subspace of Rn.
39
Fundamental Subspaces
• Let A be an mn matrix, a vector b  Rm is in the
column space of A if and only if b = Ax for some x 
Rn.
• If we think of A as a linear transformation mapping
Rn into Rm, then the column space of A is the same as
the range of A. Let A be an mn matrix and R(A)
denote the range of A. Thus
R(A) = {bRm| b = Ax for some x  Rn}
= the column space of A
The column space of AT, R(AT), is a subspace of Rn:
R(AT) = {yRn| y = ATx for some x  Rm}
= the column space of AT
40
• The column space of AT is essentially the same as the
row space of A except that it consists of vectors in Rn
(n1 matrices) rather than n-tuples
• Thus, y  R(AT) if and only if yT is in the row space
of A
• We have seen that each vector in N(A) is orthogonal
to every vector in the column space of AT (i.e., R(AT)),
R(AT)  N(A). [ let xN(A) (i.e., Ax = 0)
and yR(AT) (i.e., y = ATz)
 xTy = xT(ATz) = (Ax)Tz = 0Tz = 0 ]
41
Theorem 5.2.1
(Fundamental Subspace Theorem)
If A is an mn matrix, then N(A) = R(AT) and N(AT) =
R(A).
• (1) We have seen N(A)  R(AT)  N(A)  R(AT)

From the definition, if xR(AT), then xTy = 0 for
every y R(AT)  xT(ATz) = 0 for every zRm
 (Ax)T = 0  Ax = 0  xN(A)
Hence, N(A) = R(AT)
(2) Let B = AT, then
N(AT) = N(B) = R(BT) = R(A)
42
Example 3
1 0 
• Let A   
 2 0
(1) The column space of A consists of all vectors of the

form:
1  0   α  1 
α   β       α 
 2  0   2α   2
If x is any vector in R2, and b = Ax, then
1 0  x1   x1  1 
b        x1  
2 0  x2  2 x1   2
which is the column space of A.
43
(2) The null space of AT: N(AT) is of the form:
1 2  x1  0  x1   2 β    2
0 0  x   0   x    β   β  1
   2    2    
The range of A: R(A) = the column space of A:
 α  2 β   α
 2 β β     0      ,
 2α   β  2α 
 N ( AT )  R( A) 
44
Theorem 5.2.2
If S is a subspace of Rn, then dim S + dim S = n.
Furthermore, if {x1, ..., xr} is a basis for S and {xr+1, ...,
xn} is a basis for S, then {x1, ..., xr, xr+1, ..., xn} is a basis
for Rn.
45
Definition
If U and V are subspaces of a vector space W and each
wW can be written uniquely as a sum u+v, where
uU and vV, then we say W is a direct sum of U and
V and we write W = U  V.
46
Theorem 5.2.3
If S is a subspace of Rn, then Rn = S  S
47
Theorem 5.2.4
If S is a subspace of Rn, then (S) = S
48
• Therefore, if T is the orthogonal complement of a
subspace S, then S is the orthogonal complement of T.
• From Theorem 5.2.1, N(A) and R(AT) are orthogonal
complements of each other and N(AT) and R(A) are
orthogonal complements, we can write
N(A) = R(AT) and N(AT) = R(A)
49
Recall
• The system Ax = b is consistent if and only if b
R(A). Since R(A) = N(AT), we have the following
result.
50
Corollary 5.2.5
If A is an mn matrix and bRm, then either there is a
vector xRn such that Ax = b or there is vector yRm
such that ATy = 0 and yTb≠0
51
Note
(1) b R(A)
(2) b R(A)  b  N(AT)
Figure 5.2.2
52
Example 4
1 1 2 
• Let A  0 1 1,
1 3 4
find the bases for N(A), R(AT), N(AT), R(A).
• Sol:
1 1 2   1 1 2  1 0 1 
A  0 1 1   0 1 1   0 1 1 
     
1 3 4 0 2 2 0 0 0
 1 0 1  1 0 1  1 0 1 
AT  1 1 3  0 1 2  0 1 2
     
2 1 4 0 1 2 0 0 0
53
(1) N(A): x1 + x3 = 0
x2 + x3 = 0
set x3 = , then x1 = - and x2 = -.
 x1   α   1  1
  x2    α   α  1  N ( A)  Span(  1 )
       
 x3   α   1  1
54
(2) R(AT): Since (1, 0, 1) and (0, 1, 1) form a basis for
the row space of A
1  0 
 0 and 1 form a basis for the column space of AT
   
1 1
1   0 
 R ( AT )  Span(0, 1 )
   
1 1
55
(3) N(AT): x1 + x3 = 0
x2 + 2x3 = 0
set x3 = , then x1 = - and x2 = -2.
 x1    α    1   1
  x2    2α   α  2  N ( AT )  Span(  2 )
       
 x3   α   1  1
56
(4) R(A):
1  0 
R ( A)  Span( 0 , 1  )
   
1 2
57

Chapter 5 (5.1 - 5.2) Orthogonality

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 5 (5.1 - 5.2) Orthogonality

Uploaded by

Copyright:

Available Formats

Chapter 5 Orthogonality

• The scalar product of two n×1 matrices x and y is

• Vectors in R2 and R3 can be represented by directed

• If x and y are two nonzero vectors, u and v are the

 is called the scalar projection of x onto y and p is

If N = (a, b, c)T and P0 = (x0, y0, z0), the above

• If x  Rn then the Euclidean length of x is defined

• (1) We have seen N(A)  R(AT)  N(A)  R(AT)

(1) The column space of A consists of all vectors of the

You might also like