2
f
x1x1
2
f
x1xn
.
.
.
.
.
.
2
f
xnx1
2
f
xnxn
_
_
_
_
(4)
Note: The derivative of a real valued function with respect to a matrix, i.e., the derivative of a
real valued function of a matrix is dened to be the matrix of derivatives of corresponding elements.
1.4. Some results on matrix derivatives. Let A = (a
ij
) be an m x n matrix.
Let a = (a
1
, a
2
, a
3
, . . . a
n
) and a =
_
_
a
1
a
2
a
3
.
.
.
a
n
_
_
.
Date: August 20, 2004.
1
2 MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA
Let x = (x
1
, x
2
, x
3
, . . . x
n
) and z = (z
1
, z
2
, z
3
, . . . z
n
) so that
x =
_
_
x
1
x
2
x
3
.
.
.
x
n
_
_
z =
_
_
z
1
z
2
z
3
.
.
.
z
n
_
_
.
1.4.1. Derivative of a scalar with respect to a vector.
d(a
x)
dx
= a
(5)
1.4.2. Derivative of y = Ax.
y
x
= A (6)
1.4.3. Derivative of y = z
Ax.
y
z
= x
,
y
x
= z
A (7)
1.4.4. Derivative of y = x
Ax (A is nxn).
y
x
= x
(A
+ A) (8)
1.4.5. Derivative of y = x
Ax with A symmetric.
d(x
Ax)
dx
= 2x
= 2x
A
d(x
Ax)
dx
= 2Ax
(9)
1.4.6. Second derivative of y = x
Ax.
2
(x
Ax)
xx
= A
+ A (10)
1.4.7. Second derivative of y = x
Ax with symmetric A.
2
(x
Ax)
xx
= 2A = 2A
(11)
1.4.8. Trace with A nxn.
tr (A)
A
= I (12)
1.4.9. Determinant of A.
A
A
= A A
1
= A
+
, if A = 0,
= 0 if A = 0
(13)
1.4.10. Logarithm of det A.
ln A
A
= A
1
(14)
1.5. Some examples.
MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA 3
1.5.1. Example 1.
a: y = f(x) = 3x
1
+ 4x
2
b: a
= [3, 4]
c: x
= [x
1
, x
2
]
(a
x)
x
=
_
f
x
1
,
f
x
2
_
= [3, 4]
(ax)
x
=
_
f
x1
f
x2
_
=
_
3
4
_
(15)
1.5.2. Example 2.
a: y = x
Ax
b: x
= [x
1
, x
2
]
A =
_
3 5
5 7
_
y = [x
1
, x
2
]
_
3 5
5 7
_ _
x
1
x
2
_
= [x
1
, x
2
]
_
3x
1
+ 5x
2
5x
1
+ 7x
2
_
= 3x
2
1
+ 5x
2
x
1
+ 5x
1
x
2
+ 7x
2
2
= 3x
2
1
+ 10x
1
x
2
+ 7x
2
2
y
x
=
_
y
x
1
,
y
x
2
_
= [6x
1
+ 10x
2
, 10x
1
+ 14x
2
]
(16)
Now if we use the dierentiation formulas directly we obtain
y
x
= 2x
A
= 2 [x
1
, x
2
]
_
3 5
5 7
_
= 2 [3x
1
+ 5x
2
, 5x
1
+ 7x
2
]
= [6x
1
+ 10x
2
, 10x
1
+ 14x
2
]
(17)
and
4 MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA
y
x
= 2Ax
= 2
_
3 5
5 7
_ _
x
1
x
2
_
= 2
_
3x
1
+ 5x
2
5x
1
+ 7x
2
_
=
_
6x
1
+ 10x
2
10x
1
+ 14x
2
_
(18)
2. Kronecker Products of Matrices
2.1. Denition of a Kronecker Product. Let A be mxn and B be pxq. Then the Kronecker
product of A and B is denoted by A B and is dened by
A B =
_
_
a
11
B a
12
B a
1n
B
a
21
B a
22
B a
2n
B
a
m1
B a
m2
B a
mn
B
_
_
A B is of dimension (mp)x(nq)
2.2. Properties of Kronecker Products.
2.2.1. 1. If D
1
= A
1
B and D
2
= A
2
B then D
1
+ D
2
= (A
1
+ A
2
) B. Similarly, if E
1
=
A B
1
and E
2
= A B
2
then E
1
+ E
2
= A (B
1
+ B
2
).
2.2.2. (A B) = (A) B = A (B).
2.2.3. (A C)(B D) = (AB) (CD).
2.2.4. (P Q)
1
= P
1
Q
1
.
2.2.5. M N)
= M
.
2.2.6. A B = A
n
B
m
where A is m x m and B is n x n.
2.2.7. tr(A B) = tr(A)tr(B) where A and B are square.
3. Vectorization of Matrices
3.1. Denition of Vectorization. Let A be an n x m matrix; the notation vec(A) will mean
the nmelement column vector whose rst n elements are the rst column of A, a.
1
; the second n
elements, the second column of A, a.
2
, and so on. Thus
(A) = (a
.1
, a
.2
, ..., a
.m
)
=
_
_
a.
1
a.
2
.
.
.
a.
m
_
_
(19)
MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA 5
3.2. Properties of the vec operator.
3.2.1. vec(AB). Let A, B be n x m, m x q respectively. Then vec(AB) = (B
I) vec(A) = (I A)
vec(B).
3.2.2. Let A
1
, A
2
, A
3
be suitably dimensioned matrices. Then
(A
1
A
2
A
3
) = (I A
1
A
2
)
(A
3
)
= (A
3
A
1
)
(A
2
)
= (A
3
A
2
I)
(A
1
)
3.2.3. Let A
1
, A
2
, A
3
, A
4
be suitably dimensioned matrices. Then
(A
1
A
2
A
3
A
4
) = (I A
1
A
2
A
3
)
(A
4
)
= (A
4
A
1
A
2
)
(A
3
)
= (A
4
A
3
A
1
)
(A
2
)
= (A
4
A
3
A
2
I)
(A
1
)
3.2.4. Let A, B be m x n. Then vec(A + B) = vec(A) + vec(B).
3.2.5. Let A, B, C, D be suitably dimensioned matrices. Then
vec[(A + B)(C + D)] =[(I A) + (I B)][vec(C) + vec(D)]
=[(C
I) + (D
I)][vec(A) + vec(B)]
3.2.6. Let A, B be suitably dimensioned matrices. Then
tr(AB) =
(A
(B)
=vec(B
(A).
4. Partitioned Matrices
4.1. Denition of a Partitioned Matrix. Using the concept of submatrices, we can group the
elements of a matrix into submatrices. For example consider the matrix A below.
A =
_
_
3 4 7
2 5 2
1 0 4
_
_
(20)
We can partition it into 4 submatrices. The 2x2 matrix in the upper left hand corner can be
denoted A
11
A
11
= A({1, 2}, {1, 2}) =
_
3 4
2 5
_
(21)
Similarly the 2x1 matrix to the right of this matrix is given by
A
12
= A({1, 2}, {3}) =
_
7
2
_
(22)
The 2x1 matrix in the lower left is given by
6 MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA
A
21
= A({3}, {1, 2}) =
_
1 0
_
(23)
And nally the matrix in the lower right is given by
A
22
= A({3}, {3}) =
_
4
_
(24)
We can then write A as
A =
_
_
3 4 7
2 5 2
1 0 4
_
_
=
_
A
11
A
12
A
21
A
22
_
(25)
A partitioned matrix is called block diagonal if the submatrices not on the main diagonal are zero
matrices. For example the following matrix is block diagonal.
B =
_
B
11
0
0 B
22
_
(26)
4.2. Addition and multiplication. Consider two partitioned matrices, A and B, that are con
formably partitioned. Then we have the following relation for addition.
A + B =
_
A
11
+ B
11
A
12
+ B
12
A
21
+ B
21
A
22
+ B
22
_
(27)
For multiplication we obtain
AB =
_
A
11
A
12
A
21
A
22
_ _
B
11
B
12
B
21
B
22
_
=
_
A
11
B
11
+ A
12
B
21
A
11
B
12
+ A
12
B
22
A
21
B
11
+ A
22
B
21
A
21
B
12
+ A
22
B
22
_
(28)
4.3. Determinants of partitioned matrices. The determinant of a block diagonal matrix is easy
to compute as follows
 B =
_
B
11
0
0 B
22
_
=  B
11
  B
22
 (29)
In general the determinant of a partitioned matrix is given by
 A =
_
A
11
A
12
A
21
A
22
_
=  A
22
  A
11
A
12
A
1
22
A
21

=  A
11
  A
22
A
21
A
1
11
A
12

(30)
MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA 7
4.4. Inverses of partitioned matrices. The inverse of a block diagonal matrix is computeed as
follows
B
1
=
_
B
11
0
0 B
22
_
1
=
_
B
1
11
0
0 B
1
22
_
(31)
In general the inverse of a partitioned matrix is given by
A
1
=
_
A
11
A
12
A
21
A
22
_
1
=
_
A
1
11
_
I + A
12
F
2
A
21
A
1
11
_
A
1
11
A
12
F
2
F
2
A
21
A
1
11
F
2
_
F
2
=
_
A
22
A
21
A
1
11
A
12
_
1
(32)
To see how this formula is derived rst premultiply A by a matrix which will zero out lower left
hand matrix as follows.
_
I 0
A
21
A
1
11
I
_ _
A
11
A
12
A
21
A
22
_
=
_
A
11
A
12
0 A
12
A
1
11
A
12
+ A
22
_
(33)
Then postmultiply by a matrix to eliminate the upper right triangle creating a block diagonal
matrix.
_
I 0
A
21
A
1
11
I
__
A
11
A
12
A
21
A
22
__
I A
1
11
A
12
0 I
_
=
_
A
11
A
12
0 A
12
A
1
11
A
12
+ A
22
__
I A
1
11
A
12
0 I
_
=
_
A
11
0
0 A
12
A
1
11
A
12
+ A
22
_
(34)
This expression in useful because the inverse of a block diagonal matrix is just the inverse of each
block diagonal element. Now take the inverse of both sides of equation 34.
_
I A
1
11
A
12
0 I
_
1
_
A
11
A
12
A
21
A
22
_
1
_
I 0
A
21
A
1
11
I
_
1
=
_
A
11
0
0 A
12
A
1
11
A
12
+ A
22
_
1
(35)
Now premultiply both sides by the inverse of the rst matrix on the left of equation 35to obtain
_
A
11
A
12
A
21
A
22
_
1
_
I 0
A
21
A
1
11
I
_
1
=
_
I A
1
11
A
12
0 I
_ _
A
11
0
0 A
12
A
1
11
A
12
+ A
22
_
1
(36)
Now postmultiply both sides by the inverse of the second matrix on the left of equation 36 to
obtain
8 MISCELLANEOUS CONCEPTS OF MATRIX ALGEBRA
_
A
11
A
12
A
21
A
22
_
1
=
_
I A
1
11
A
12
0 I
_ _
A
11
0
0 A
12
A
1
11
A
12
+ A
22
_
1
_
I 0
A
21
A
1
11
I
_
=
_
I A
1
11
A
12
0 I
_
_
A
1
11
0
0
_
A
22
A
12
A
1
11
A
12
_
1
_
_
I 0
A
21
A
1
11
I
_
(37)
Now simply multiply out equation 37 as follows
_
A
11
A
12
A
21
A
22
_
1
=
_
I A
1
11
A
12
0 I
_
_
A
1
11
0
0
_
A
22
A
12
A
1
11
A
12
_
1
_
_
I 0
A
21
A
1
11
I
_
=
_
A
1
11
A
1
11
A
12
_
A
22
A
12
A
1
11
A
12
_
1
0
_
A
22
A
12
A
1
11
A
12
_
1
_
_
I 0
A
21
A
1
11
I
_
=
_
A
1
11
+ A
1
11
A
12
_
A
22
A
12
A
1
11
A
12
_
1
A
21
A
1
11
A
1
11
A
12
_
A
22
A
12
A
1
11
A
12
_
_
A
22
A
12
A
1
11
A
12
_
1
A
21
A
1
11
_
A
22
A
12
A
1
11
A
12
_
1
(38)
Now let
F
2
=
_
A
22
A
21
A
1
11
A
12
_
1
and rewrite equation 38 as
A
1
=
_
A
11
A
12
A
21
A
22
_
1
=
_
A
1
11
+ A
1
11
A
12
F
2
A
21
A
1
11
A
1
11
A
12
F
2
F
2
A
21
A
1
11
F
2
_
(39)
Much more than documents.
Discover everything Scribd has to offer, including books and audiobooks from major publishers.
Cancel anytime.