Professional Documents
Culture Documents
Quadratic Forms
Systems
Sums
Applications
Code
Matrix fundamentals
A=
as aij .
The matrix A can also be written as A = (aij ).
GARTH TARR
2011
Fundamentals
Quadratic Forms
Systems
Sums
Applications
a
a
a
A = 11 12 13 ;
a21 a22 a23
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Matrix multiplication
b
b
b
B = 11 12 13
b21 b22 b23
A=
A B = (aij ) (bij ).
d11 d12
D = d21 d22
d31 d32
( r c ) ( c p ) = ( r p )
( r c ) ( r c ) = ( r c )
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Matrix multiplication
d11
d12
d21
d22
d31
d32
Systems
Sums
Applications
Code
D : 32
a1
12
d2
d3
a1
Quadratic Forms
Determinant
d1
Fundamentals
j=1
a11
a12
a13
a21
a22
a23
A : 23
Fundamentals
Quadratic Forms
Systems
c11
c12
c21
c22
Applications
2 2 determinant
cij Cij .
j=1
C = AD : 2 2
Sums
n
X
More
The trick is to pick a row (or column) with a lot of zeros (or
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
3 3 determinant
c
c
Apply the general formula to a 2 2 matrix: C = 11 12 .
c21 c22
Keep the first row fixed, i.e. set i = 1.
2
X
General formula when i = 1 and n = 2: det(C) =
c1j C1j
j=1
i = 1 and n = 3:
det(B) =
3
X
j=1
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Fundamentals
Quadratic Forms
A=
b11 b12 b13
det(B) = b21 b22 b23
b31 b32 b33
b21 b22
b21 b23
b22 b23
+ b13
b12
= b11
b31 b32
b31 b33
b32 b33
!
= b11 b22 b33 + b12 b23 b31 + b13 b21 b32
!
b13 b22 b31 + b11 b23 b32 + b12 b21 b33
Fundamentals
b12
b13
b11
b12
b21
b22
b23
b21
b22
b31
b32
b33
b31
b32
Quadratic Forms
Sums
Applications
Identity matrix
Applications
x 1 y1
a
=
b
x 2 y2
y1
a
x2
x1
the matrix.
1
The oriented area is the same as the usual area, except that it is negative
when the vertices are listed in clockwise order.
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Identity matrix
An identity matrix is the matrix analogue of the number 1.
identity matrix the result will be the same matrix (or vector).
everywhere else:
1
0
I = .
..
0
0
0 0
1 0
0 1
..
..
.
.
0 0
0 0
0 0
0 0
0 0
..
.
1 0
0 1
identity matrix.
If the size of I is not specified, it is assumed to be
Code
Systems
Sums
Determinant as an area
b11
Systems
Example (2 2)
a11 a12 1 0
AI =
a21 a22 0 1
a11 1 + a12 0 a11 0 + a12 1
=
a21 1 + a22 0 a21 0 + a22 1
a11 a12
=
= A.
a21 a22
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Inverse
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Vectors
Vectors are matrices with only one row or column. For example,
the column vector:
x1
x2
x= .
..
Definition (Inverse)
Requires a square matrix i.e. dimensions: r r
a11 a12
For a 2 2 matrix, A =
,
a21 a22
A1
1
=
det(A)
a22 a12
a21 a11
xn
denoted
A1 ,
i.e. AA
xx=
= I.
n
X
x2i
i=1
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Transpose
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Symmetry
a11
a21
A = (aij ) = .
..
..
.
a1n
a2n
..
.
A = (aji )
The subscripts gets swapped.
A is a n m matrix: the columns in A are the rows in A .
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Idempotent
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Order of operations
Matrix multiplication is non-commutative, i.e. the order of
Definition (Idempotent)
Commutativity
multiplication is important: AB 6= BA.
Matrix multiplication is associative, i.e. as long as the order
Associativity
stays the same, (AB)C = A(BC).
A(B + C) = AB + AC
(A + B)C = AC + BC
Example
Let A be a k k matrix and x and c be k 1 vectors:
Ax = c
A1 Ax = A1 c
Ix = A1 c
x = A1 c
Note: A1 c 6= cA1
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Matrix Differentiation
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Matrix Differentiation
= a.
A
= 2A.
Proof.
!
a =
(1 a1 + 2 a2 + . . . + k ak )
1 (1 a1 + 2 a2 + . . . + k ak )
2 (1 a1 + 2 a2 + . . . + k ak )
=
..
(1 a1 + 2 a2 + . . . + k ak )
=a
Matrix Calculus
Proof.
1
a11 a12
By means of proof, say =
and A =
, then
2
a12 a22
!
! 2
A =
1 a11 + 2a12 1 2 + 22 a22
"
! 2
#
2a
a
+
2a
11
12
1
2
22
2
= 1 ! 12
2a
a
+
2a
11
12
1
2
22
1
2
2
21 a11 + 2a12 2
=
21 a12 + 2a22 2
= 2A
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Matrix Differentiation
Quadratic Forms
Systems
Sums
Applications
Code
Rank
A
= A.
Proof.
a11 a12
1
, then
and A =
By means of proof, say =
a21 a22
2
a11 1 + a12 2
(A) =
a21 1 + a22 2
i
h
(a
+
a
)
11
1
12
2
1 2
= h
(a
+
a
)
21 1
22 2
1
2
"
#
independent
= A.
Fundamentals
Fundamentals
Quadratic Forms
v1
v2
v3
v4
dependent
Systems
Sums
Applications
Rank
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Trace
The maximum rank of an m n matrix is min(m, n).
Definition
A full rank matrix is one that has the largest possible rank,
Properties
tr(AB) = tr(BA)
More generally, for conformable matrices:
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Eigenvalues
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Eigenvalues
A is defined as
2 1
. We can find the eigenvaules of A by solving
Say A =
1 2
Ax = x.
Since the eigenvector x is different from the zero vector (i.e.
det
det(A I) = 0
2 1
1 0
=0
1 2
0 1
2
1
=0
1
2
(2 )(2 ) 1 1 = 0
2 4 + 3 = 0
( 1)( 3) = 0
The eigenvalues are the roots of this quadratic: = 1 and = 3.
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Sums
Applications
(AB) = B A
det(A) = det(A )
1
det(A)
AI = A and xI = x
det(A1 ) =
eigenvectors: det(A) = 1 2 n .
Systems
tr(A) = 1 + 2 + . . . + n .
Quadratic Forms
Useful rules
1 , 2 , . . . , n are positive.
Fundamentals
Determinant
a
=a
A
=A
A
= 2A
If A is a k k symmetric matrix,
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Quadratic forms
Fundamentals
Systems
Sums
Applications
Code
Quadratic forms
Q(x1 , . . . , xn ) =
aij xi xj .
ij
R2
Q(x) = x Ax
a11 x21
a22 x22 .
E.g. in
we have Q(x1 , x2 ) =
+ a12 x1 x2 +
Quadratic forms can be represented by a symmetric matrix A
such that:
Q(x) = x Ax
Q(x) =
x1 x2
a11
1
2 a21
1
2 a12
a22
x1
x2
1
= a11 x21 + (a12 + a21 )x1 x2 + a22 x22
2
x1 x2
1
2 a12
a22
1
2 a23
1
2 a13
1
2 a23
a33
x1
x2
x3
Systems
a11
1
x3
2 a12
1
2 a13
Quadratic Forms
Fundamentals
Quadratic Forms
Sums
Applications
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Positive definite
1 0
then Q1 (x) = x Ax = x21 + x22 .
If A =
0 1
Q1 is greater than zero at x 6= 0 i.e. (x1 , x2 ) 6= (0, 0).
The point x = 0 is a global minimum.
Q1 is called positive definite.
when x 6= 0.
We want to know if x = 0 is a max, min or neither.
200
Q1(x1,x2)
ax21 ,
150
100
50
0
10
5
0
5
x2
10
10
10
x1
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Negative definite
100
50
50
Q3(x1,x2)
Q2(x1,x2)
Sums
Applications
Code
150
0
50
100
10
200
10
5
0
5
10
x2
10
Quadratic Forms
10
0
5
Systems
Sums
Applications
10
10
x1
Code
Code
Positive semidefinite
Fundamentals
Quadratic Forms
Systems
Code
Sums
Applications
Code
Negative semidefinite
10
x2
x1
Fundamentals
Systems
1 0
then Q3 (x) = x Ax = x21 x22 .
0 1
Q3 can be take both positive and negative values.
E.g. Q3 (1, 0) = +1 and Q3 (0, 1) = 1.
Q3 is called indefinite.
If A =
100
1 1
then Q5 (x) = x Ax = (x1 + x2 )2 .
If A =
1 1
Q4 is always 0 but does equal zero at some x 6= 0
E.g. Q5 (10, 10) = 0
Q5 is called negative semidefinite.
1 1
then Q4 (x) = x Ax = x21 + 2x1 x2 + x22 .
If A =
1 1
Q4 is always 0 but does equal zero at some x 6= 0.
E.g. Q4 (10, 10) = 0.
Q4 is called positive semidefinite.
400
100
Q5(x1,x2)
300
Q4(x1,x2)
Quadratic Forms
Indefinite
1 0
then Q2 (x) = x Ax = x21 x22 .
0 1
Q2 is less than zero at x 6= 0 i.e. (x1 , x2 ) 6= (0, 0).
The point x = 0 is a global maximum.
Q2 is called negative definite.
If A =
Fundamentals
200
100
0
10
200
300
400
10
0
x2
0
10
10
10
x2
x1
Code
10
10
10
x1
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Positive Definite
Definition
..
.
M itself.
Negative Definite
Fundamentals
Quadratic Forms
Systems
Sums
Example
Eigenvalues
Applications
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
f (x) = 4x
f (x) = 0 = x = 0
f (x) = 4 > 0 = minimum at x = 0.
f (x) = 2x2
f (x)
1
x
1
Code
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Fundamentals
Quadratic Forms
Application
y1
x11
y2 x21
.. = ..
. .
yn
Quadratic Forms
Systems
Sums
Applications
Code
y1
y2
y = . ;
..
yn
for i = 1, 2, . . . , n and
x11
x21
X= .
..
b1
b2
b = . ;
..
bk
x12
x22
..
.
xn1 xn2
Quadratic Forms
Systems
Sums
Applications
We can write y = Xb as
x1
y1
y2 x
2
.. = .. b.
. .
xi1
xi2
xi = .
..
xik
yn
xn
. . . x1k
x1
. . . x2k x2
.. = .. .
. .
Fundamentals
. . . x1k
b1
. . . x2k b2
.. .. .
. .
. . . xnk
bk
x12
x22
..
.
Code
Fundamentals
Applications
A system of equations:
Sums
X X
Systems
y1 = x1 b
y2 = x2 b
..
.
xn
yn = xn b
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Fundamentals
X u =
x11 x21
x12 x22
x u + x21 u2 + x31 u3
= 11 1
x12 u1 + x22 u2 + x32 u3
= x 1 u1 + x 2 u2 + x 3 u3
=
3
X
Sums
Applications
Code
3
X
xi xi .
i=1
x
x12
x11 x21 x31 11
x21 x22
XX=
x12 x22 x32
x31 x32
x1
= x 1 x 2 x 3 x 2
x3
We can write,
u1
x31
u2
x32
u3
Systems
u1
x11 x12
x1
x
X = x21 x22 = x2 ; xi = i1 ; and u = u2
xi2
x3
u3
x31 x32
Quadratic Forms
= x1 x1 + x2 x2 + x3 x3
3
X
xi xi
i=1
x i ui
i=1
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
E(Y) = = 0. Then,
!
cov(Y, Y) = var(Y) = E [Y ][Y ]
Y1
Y 2
= E . Y 1 Y 2 Y n
.
.
Yn
2
Y1
Y1 Y2 Y1 Yn
Y2 Y1 Y 2 Y2 Yn
2
= E .
..
..
.
.
.
.
Yn Y1 Yn Y2
Yn2
var(Y1 )
cov(Y1 , Y2 ) cov(Y1 , Yn )
cov(Y2 , Y1 )
var(Y2 )
cov(Y2 , Yn )
var(Y) =
.
..
..
..
.
.
.
cov(Yn , Y1 ) cov(Yn , Y2 )
var(Yn )
constants, a?
!
var(a Y) = E [a Y a ][a Y a ]
!
= E a [Y ](a [Y ])
!
= E a [Y ][Y ] a
!
= a E [Y ][Y ] a
= a var(Y)a
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Quadratic Forms
Systems
u u =
Pn
2
i=1 ui
S() =
a var(Y)a
where u = y X or ui = yi xi .
n
X
(yi xi )2 = (y X) (y X)
S()
= 2X y + 2X X = 0 = X X = X y.
invertible) we get,
Sums
Applications
= (X X)1 X y.
Code
2 S()
Fundamentals
Operation
Systems
Sums
Applications
Matlab
A=matrix(c(5,7,10,2),
ncol=2,byrow=T)
A = [5,7;10,2]
det(A)
det(A)
det(A)
A1
solve(A)
inv(A)
A+B
A + B
A + B
AB
A %*% B
A * B
t(A)
A=
= 2X X.
q=vv=
vi2 ,
where v = Xc.
Quadratic Forms
Matrix Operations
= y y 2y X + X X.
var(a Y) = a var(Y)a
var(Y1 )
cov(Y1 , Y2 ) a1
= a1 a2
a2
cov(Y1 , Y2 )
var(Y2 )
Systems
5 7
10 2
i=1
Code
i=1
Quadratic Forms
Applications
Now, var(a1 Y1 + a2 Y2 ) =
=
where
var(Y1 )
cov(Y1 , Y2 )
,
var(Y) =
cov(Y1 , Y2 )
var(Y2 )
Fundamentals
Sums
= a1 Y1 + a2 Y2
a Y = a1 a2
Y2
var(a Y)
Fundamentals
Code
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Matrix Operations
Fundamentals
Quadratic Forms
Systems
Sums
Matlab
Figure 1
eigenvalues &
eigenvectors
eigen(A)
[V,E] = eig(A)
covariance
matrix of X
var(X) or cov(X)
cov(X)
estimate of
rank(A)
[x,y] = meshgrid(-10:0.75:10,-10:0.75:10);
surfc(x,y,x.^2 + y.^2)
ylabel(x_2)
xlabel(x_1)
zlabel(Q_1(x_1,x_2))
qr(A)$rank
rank(A)
r r identity
matrix, Ir
diag(1,r)
eye(r)
Figure 1
Figure 2
Quadratic Forms
Code
Matlab Code
Operation
Fundamentals
Applications
Systems
Sums
[x,y] = meshgrid(-10:0.75:10,-10:0.75:10);
surfc(x,y,-x.^2 - y.^2)
ylabel(x_2)
xlabel(x_1)
zlabel(Q_2(x_1,x_2))
Applications
Code
Matlab Code
Figure 3
[x,y] = meshgrid(-10:0.75:10,-10:0.75:10);
surfc(x,y,x.^2 + 2.*x.*y + y.^2)
ylabel(x_2)
xlabel(x_1)
zlabel(Q_4(x_1,x_2))
Fundamentals
Quadratic Forms
Systems
Sums
Applications
Code
Matlab Code
Figure 3
[x,y] = meshgrid(-10:0.75:10,-10:0.75:10);
surfc(x,y,x.^2 - y.^2)
ylabel(x_2)
xlabel(x_1)
zlabel(Q_3(x_1,x_2))
Figure 4
Figure 2
Figure 5
[x,y] = meshgrid(-10:0.75:10,-10:0.75:10);
surfc(x,y,-(x+y).^2)
ylabel(x_2)
xlabel(x_1)
zlabel(Q_5(x_1,x_2))
Figure 4
Figure 5