This action might not be possible to undo. Are you sure you want to continue?
1) Vector is defined as a = (a
1
,a
2
,,,a
n
). That is a point in n dimensional space
2) Distance between two vectors (x and y) is defined as
p
n
i
p
j i p
y x d
/ 1
1
) ) ( (
∑
·
− ·
When p=2 then subscript is usually dropped. It is a standard Eucilidian distance used
in everyday life.
3) Scalar product of two vectors
∑
·
·
n
i
i i
y x y x
1
) , (
Norm is defined as:
∑
·
·
n
i
p p
i p
x x
1
/ 1
) (  
If p=2 then subscript is usually dropped and:
2 / 1
) , (   x x x ·
3) Matrix. nxk matrix is defined:
,
`
.

·
nk n n
k
k
a a a
a a a
a a a
... ...
... ... ... ... ...
... ... ... ... ...
... ...
... ...
2 1
2 22 21
1 12 11
A
If n=k then it is called a square matrix.
Transpose of a matrix is defined:
ji ij
a · ) (
T
A
If a
ij
=a
ji
for all i and j pairs then square matrix is called symmetric. Obviously in this
case  . A A
T
·
Identity matrix is defined as a nxn matrix that has only diagonal elements:
,
`
.

·
1 ... 0 0
... ... ... ...
0 ... 1 0
0 ... 0 1
n
I
Usually subscript is dropped.
Multiplication matrix by vector is defined as:
∑
·
·
k
j
j ij i
b a
1
) (Ab
Obviously b is a k dimensional vector and Ab is an n dimensional vector.
If a matrix is square then bilinear form is defined as:
∑∑
· ·
· ·
n
i
n
j
j i ij
y x a
1 1
) , ( Ay x y Ax
T
When x=y it is called a quadratic form. Matrix is called positive semidefinite if for
all x the condition 0 ≥ Ax x
T
holds. If for all nonzero vectors above inequality is
strict then the matrix is positive definite. Negative semidefinite and definite matrices
are defined similarly.
Multiplication of nxk matrix by kxm matrix is defined as (the result is nxm) matrix):
∑
·
·
k
l
lj il ij
b a
1
) (AB
The following relations are true:
) ( ) ( ) det( ) 3
) ( ) 2
) ( ) 1
B det A det AB
A B C ABC
A B AB
·
·
·
T T T
T T T
Scalar product of two vectors can also be written
y x y x
T
· ) , (
This notation means that multiplication of a column vector with a row vector. It is a
scalar. However multiplication of a row vector by a colum vector is nxn matrix.
,
`
.

·
n n n n
n
n
y x y x y x
y x y x y x
y x y x y x
...
... ... ... ...
...
...
2 1
2 2 2 1 2
1 2 1 1 1
T
xy
7) Orthogonal matrix is defined as a matrix for which row vectors are orthogonal to
each other. It can be defined as:
I A A A A
T T 1
· ⇔ ·
−
In general matrix is called orthogonal if
I AA
T
·
where I is a diagonal matrix with min(n,k) elements equal to 1. All other elements are
equal to 0. In effect it means that vector composed of rows of this matrix are
orthogonal for different rows. Similarly column orthogonality can be defined.
If multiplication of a matrix by itself is equal to matrix itself then it is call idempotent:
A AA ·
Psuedoinversion (
−
A
) of a matrix is defined as:
A A AA ·
−
Rank of matrix: Rank of the matrix is maximum number linearly independent row
vectors or column vectors. It is also defined maximum square submatrix with non
zero determinant.
9) Trace of a quadratic matrix is defined as:
∑
·
·
n
i
ii
a tr
1
) (A
sum of the diagonal terms of the matrix. Trace has the following properties (they can
be easily verified):
∑
·
·
·
·
·
·
·
n
i
i
tr
tr tr
tr tr
tr s s tr
tr tr
n tr
1
) ( ) 6
) ( ) ( ) 5
) ( ) ( ) 4
) ( ) ( ) 3
) ( ) ( ) 2
) ( ) 1
λ A
BCA ABC
BA AB
A A
A A
I
T
n
where s is a scalar λ s are eigenvalues of the matrix.
10) Eigenvalue decompositions:
Eigenvalues of a matrix called a scalar value λ for which there is a nonzero solution
to the equation:
x Ax λ ·
Corresponding solution vecto is called eigenvector. Symmetric nxn matrix has n
eigenvectors. Some of the eigenvalues can be equal to each other. In this case they are
called degenerate. If a matrix has n eigenvalues and we build a matrix from this
eigenvectors so that each row is one of the eigenvectors of the matrix:
,
`
.

·
e
e
e
E
...
2
1
For symmetric matrix eigenvectors are orthonormal i.e.:
¹
'
¹
≠
·
·
·
j i
j i
ij j i
if 0
if 1
: satisfy that symbols Kronecker are where
ij
ij
δ
δ
δ e e
It implies that the matrix E is orthogonal.
We define:
,
`
.

· · Λ
n
n
diag
λ
λ
λ
λ λ λ
... 0 0
0 ... ... ...
0 ... 0
0 ... 0
) , , , , (
2
1
2 1
A symmetric matrix can be decomposed by its eigenvalues and eigenvectors:
E E A Λ ·
T
Then matrix inversion can be written:
E E A
1 1 − −
Λ ·
T
11) Singular value decomposition. There are several version of this. One of them is.
Any matrix of nxk can be decomposed as:
UDV A ·
Where U is nxn orthogonal matrix, V is kxk orthogonal matrix. D is a diagonal matrix
(with min(n,k) diagonal elements and remaining element 0).
Another version uses the fact about a matrix rank. If rank of a matrix is r then matrix
can be decomposed using:
T
UDV A ·
. Now U is nxr orthogonal matrix, V is kxr orthogonal matrix and D is rxr diagonal
matrix of positive elements. In the first version diagonal terms of D may have 0
elements. In the second version all elements are non0.
Some useful matrices for data analysis
Some facts about derivative of function
w.r.t. vectors
If a function depends on a vector x of length n then its derivative (if exists) w.r.t.
vector is defined as:
) , , , , (
1 n
x
f
x
f f
∂
∂
∂
∂
·
∂
∂
x
Note that derivative of a function w.r.t vector elements is a vector. The second
derivative is defined similarly:
,
`
.

∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
∂ ∂
∂
·
∂ ∂
∂
n n n n
n
n
x x
f
x x
f
x x
f
x x
f
x x
f
x x
f
x x
f
x x
f
x x
f
f
2
2
2
1
2
2
2
2 2
2
1 2
2
1
2
2 1
2
1 1
2
2
...
... ... ... ...
...
....
T
x x
Derivative of the scalar product:
y
x
y x
·
∂
∂ ) (
T
And derivative of the square of the vector norm:
n T
I
x x
x x
x
x
x x
·
∂ ∂
∂
·
∂
∂
) (
2
) (
2 T
T
3) Derivative of a quadratic form
T
A A
x x
Ax x
x A A
x
Ax x
+ ·
∂ ∂
∂
+ ·
∂
∂
T
T
T
T
) (
) (
) (
2
Derivatives of bilinear forms:
y A
x
Ax) (y
Ay
x
Ay) (x
T
T
T
·
∂
∂
·
∂
∂
Some facts about derivative of function w.r.t.
matrix
Definition: If a function depends on a matrix then its derivative w.r.t. matrix is
defined as a matrix composed of derivative of the function w.r.t. elements o the
matrix:
,
`
.

∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
∂
·
∂
∂
nk n n
k
k
a
f
a
f
a
f
a
f
a
f
a
f
a
f
a
f
a
f
f
...
... ... ... ...
...
...
) (
2 1
2 22 21
1 12 11
A
A
Similraly the second derivatives are defined. This time it is an array of 4
th
order.
2) Derivative of a trace is the identity matrix
I
A
A
·
∂
∂ ) ( (tr
Derivativ of determinant:
¹
'
¹
−
·
∂
∂
− −
−
symmetric is matrix if )) ( 2 )( det(
symmetric not is matrix if ) det( ) (det(
1 1
1
A A A
A A
A
A
diag
Derivative of a bilinear form
¹
'
¹
− +
·
∂
∂
symmetric is matrix if
symmetric not is matrix if
y) diag(x x y y x
y x
A
Ay) (x
T T T
T T
f
diag is defined as a matrix consisting of diagonal elements only.
If we put y=x we can get derivative of a quadratic form w.r.t matrix.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.