# 1

Useful mathematics, notation and deﬁnitions

Fact 1.1. If we multiply a column vector x by a matrix Amxn , this deﬁnes a function: Am⇥n : Rn ! Rm

Notation 1. A = [a(1)...a(n)], ai 2 Rm refers to columns 1 through n of matrix A. 2 3 a 1· 6 . 7 n Notation 2. A = 4 . . 5, ai 2 R refers to the rows of A. a m1· Proposition 1.1. D = AB and B is nonsingular =) rank (D) = rank (A) Proposition 1.2. Let Am⇥n , then, Rank (A)  {m, n} Proposition 1.3. D = AB =) Rank (D)  min {Rank (A), Rank (B )} Proposition 1.4. Let Am⇥n =) rank (A) = rank (AA0 ) = rank (A0 A).

Deﬁnition 1.1. C (A) = ha(1), ..., a(n)i refers to the space spanned by the columns of A.

1.1

Some Vector Calculus
k ⇥1

For S ( ) = min(Y-X )’(Y-x ) where Yn⇥1 , Xn⇥k and Fact 1.2. Derivative w.r.t. a column vector 0 @S( @S( ) B =B @ @
@ )
1

. . .

1 C C A
S( ) , @@ k

@S( ) @ k

Fact 1.3. Derivative w.r.t. a row vector @ S ( ) ⇣ @S( ) = @ 1 , ... @ 0 Fact 1.4. Let mapping, then @ ( ) B =B @ @ 0 0 ( ) : Rk ! Rm , then

B B ( )=B @
@ @
1( 0

0

m(

1 ) C 2( ) C C is a vector valued . . A .
1(

)

)

. . .

1 C C A
@ ( ) @ 0

@

@

m( 0

)

where each element is a row vector as in fact 1.3 and
1 This

is a m ⇥ k matrix1 .

symbol

0

means transpose.

1

.2 and matrix. . then is an m ⇥ n matrix. .8 holds =) we can always ﬁnd a decomposition A of B s. .. A. 0 1 ↵1 B . . @ y (↵ ) @y @x @ x (↵ ) = =A 0 0 0 @↵ @ x @↵ @↵0 the resulting matrix is m ⇥ r. then: 1 0 1 1 (x Bx + x0 B 0 x)) = [x0 (B + B 0 ) x] = (2x0 Ax) = y 0 2 2 2 Proposition 1. .5. then ( ) : Rk ! Rm . Proposition 1.8. @ x2 @↵1 @ x2 @↵2 . to see this : 0 . Let mapping.6. If Fact 1. .7.5. A @ x @↵0 is not well deﬁned. @ x3 @↵1 @ x3 @↵2 . 0 ( ) @ @y @ x0 is an k ⇥ m @y @ x0 = A where @y @y 0 Fact 1. A . . 1( ) @ 0 ( ) @ = ✓ @ ) @ . y = x0 Ax =) @y @x = 2x0 A = 2Ax 2 . Fact 1. ··· @ xn @↵1 @ xn @↵2 . (↵ ) Note 1. @ m( ) @ ◆ @ where each element is a column vector as in fact 1. since y is a scalar. . where ↵ = @ .. . Proof. Let A = y= 1 2 (B + B 0 ) be a symmetric matrix. . 1 C C C A r ⇥n @ xn @↵r Fact 1.Fact 1. Suppose y = Ax (↵). but Am⇥n ! B @ x0 B = B 0 @↵ @ 0 @ x1 @↵1 @ x1 @↵2 . then @ x = A where @ x is an n ⇥ m matrix.6.. .t A is symmetric matrix. C Proposition 1. y = x0 Bx = (x0 Bx)0 = x0 B 0 x = y 0 .7. Let y = Ax where A does not depend on x. B B ( ) = B @ 1( 0 m( 1 ) C 2( ) C C . then if A does ↵r not depend on ↵. Let y = Ax where A does not depend on x. @ x1 @↵r @ x2 @↵r @ x3 @↵r ··· ··· .. be a vector valued .

+ xk ˆk implies that y lives in the linear space generated by hx1 .. P =P0 3.y y-Xß=e x2 Xß 0 x1 Figure 1: Geometry of Least Squares. We call P a projection matrix if: 1. 2. . xk i ⇢ Rn .9. we account for all aspects of y that do no live in hx1 . . P2 = P Fact 1.. Deﬁnition 1. P : Rn ! Rn .... In our particular case P = X(X0 X) 1 X 3 . 1... however when we add "..2.2 Geometry of Least Squares y = x1 ˆ1 + . xk i.