Summary of Matrix Algebra

D.1 Basic Definitions

A matrix is a rectangular array of numbers. More precisely, an m x n matrix has m rows and n
columns. The positive integer m is called the row dimension, and n is called the column
dimension.

We use uppercase boldface letters to denote matrices. We can write an m x n matrix generically
as

where aij represents the element in the ith row and the jth column. For example, a25 stands for the
number in the second row and the fifth column of A. A specific example of a 2 X 3 matrix is

A square matrix has the same number of rows and columns. The dimension of a square matrix is
its number of rows and columns.

A 1 X m matrix is called a row vector (of dimension m) and can be written as x = [x1,x2,…, xm]

An n X 1 matrix is called a column vector and can be written as

A square matrix A is a diagonal matrix when all of its non-main diagonal elements are zero, that
is, aij = 0 for all i≠j. We can always write a diagonal matrix as

1
The n x n identity matrix, denoted I, or sometimes I n to emphasize its dimension, is the diagonal
matrix with unity (one) in each diagonal position, and zero elsewhere:

The m x n zero matrix, denoted 0, is the m x n matrix with zero for all entries. This need not be a
square matrix

0=

D.2 Matrix Operations

Matrix Addition: Two matrices A and B, each having dimension m x n, can be added
element by element.

Matrices of different dimensions cannot be added

2
Scalar Multiplication: Given any real number γ (often called a scalar), scalar
multiplication is defined as γA= [γ aij], or

Matrix Multiplication: To multiply matrix A by matrix B to form the product AB, the
column dimension of A must equal the row dimension of B. Therefore, let A be an m x n matrix
and let B be an n xp matrix. Then matrix multiplication is defined as

In other words, the (i,j)th element of the new matrix AB is obtained by multiplying each element
in the ith row of A by the corresponding element in the jth column of B and adding these n
products together. A schematic may help make this process more transparent:

3
We can also multiply a matrix and a vector. If A is an n x m matrix and y is an m x 1 vector, then
Ay is an n x 1 vector. If x is a 1 x n vector, then xA is a 1 x m vector.

Properties of Matrix operation

In the following list of properties, A, B, and C are matrices with appropriate dimensions for
applying each operation, and α and β are real numbers.

1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

4
If A is m x n and B is n x m, then AB and BA are both defined, but they are not usually the
same; in fact, they have different dimensions, unless A and B are both square matrices. Even
when A and B are both square, AB ≠ BA, except under special circumstances.

Transpose

Properties of transpose

1.
2.
3.
4.

5.
6.

A square matrix A is a symmetric matrix if and only if A’=A. If X is any n x k matrix, then X’X
is always defined and is a symmetric matrix.

5
Partitioned Matrix Multiplication

Trace
The trace of a matrix is a very simple operation defined only for square matrices.

Properties of trace

1.
2.

6
 3.
4.
5.

Determinant of a matrix
The determinant is a value that can be computed from the elements of a square matrix. The
determinant of a matrix A is denoted det(A), det A, or |A|. Geometrically, it can be viewed as the
scaling factor of the linear transformation described by the matrix.

In the case of a 2 × 2 matrix the determinant may be defined as:

Similarly, for a 3 × 3 matrix A, its determinant is:

Each determinant of a 2 × 2 matrix in this equation is called a "minor" of the matrix A. This
procedure can be extended to give a recursive definition for the determinant of an n × n matrix,
the minor expansion formula.

A matrix is invertible if and only if its determinant is non-zero, and correspondingly the matrix is
singular (non-invertible) if and only if its determinant is zero.

Properties of Determinants
The determinant of a product of matrices is equal to the product of determinants.
Special types of matrices have special determinants; for example, the determinant of an
orthogonal matrix is always plus or minus one, and the determinant of a complex
Hermitian matrix? is always real.

Orthogonal matrix:

7
A real square matrix A, where AA’=A’A=I, then matrix A is orthogonal. An orthogonal matrix is
always invertible, and A-1=A’

Example:

Properties of orthogonal matrix

1. If A is orthogonal then A-1 is also orthogonal.

2. If A is orthogonal then A’ is also orthogonal
3. If A is orthogonal then A-1=A’
4. If A is orthogonal then |A|= 1
5. If A and B are orthogonal with order n then AB and BA are orthogonal

Inverse

Finding inverse

Example given a square matrix find

Step 1: find the determinant

Step 2: Find the cofactor of A

8
Step 3: Find the determinants of the minor in the cofactor

Step 4: Correct the entries with appropriate signs

Step 5: Transpose the cofactor

Step 6: multiply the cofactor by to get A-1

Properties of Inverse:

1. If an inverse exists, it is unique

2.
3.
4.

9
D.3 Linear Independence: Rank of a Matrix

Linear Independence
For a set of vectors having the same dimension, it is important to know whether one vector can
be expressed as a linear combination of the remaining vectors.

 0 is a vector of 0’s

 0 is a scalar 0

If holds for a set of scalars that are not all zero, then {x1,x2, …,
xr} is linearly dependent. The statement that {x1,x2, …, xr} is linearly dependent is equivalent to
saying that at least one vector in this set can be written as a linear combination of the others.

Example: are the following three vectors linearly independent?

Solution:

rewriting

10
Rank of a Matrix

Simply the rank of a matrix is the number of linearly independent column of the matrix.

The matrix has a full column rank if its determent is different from zero. If its determinant is zero
it has less than the number of column.

Rank of a matrix is found by two steps

1. Find determinant of the matrix. If determinant of the matrix is non-zero, its rank is the
number of the columns. If determinant is zero then go to step 2
2. By deleting one raw and column find determinant of all possible minors. If determinant
of one of the minor is non-zero then rank is equal with number of the column of the
minor. If all determinants of the column is zero, find the minor of the minor and
continues.

Example: Given the following 3x3 matrix find its rank

First find determinant |A|

|A|=1(-4)-2(-4)+(-1)(6-2)=0 so the matrix has less than 3 rank. So we have to find the rank of
2x2 matrix by deleting one column and one row and finding the determinant of all possible 2x2
matrices.

11
So, since at least one of the 2x2 matrix determinant is non-zero, the rank of the matrix is 2.

Properties of rank of a matrix

1.
2.
3.

D.4 Quadratic Forms and Positive Definite Matrices

Given A is a 2x2 matrix and x is a 2x1 vector

12
Properties of Positive Definite and Positive Semi-Definite Matrices:
1. A positive definite matrix has diagonal elements that are strictly positive, while a p.s.d.
matrix has nonnegative diagonal elements;
2. If A is p.d., then A-1 exists and is p.d.;
3. If X is n x k, then X’X and XX’ are p.s.d.;
4. If X is n x k and rank(X) = k, then X’X is p.d. (and therefore nonsingular).

D.5. Idempotent Matrix

It can also written as A2=A

Example 1: the matrix which is square and symmetric is given below. Find AA

Example 2: Idempotent matrix should not necessarily be symmetric

13
Properties of idempotent matrix
1. Let A be an n x n idempotent matrix. rank(A) = tr(A)
2. A is positive semi-definite

We can construct idempotent matrices very generally. Let X be an n x k matrix with rank(X) = k.
Define

D.6 Differentiation of Linear and Quadratic Forms

14
D.7 Moments and Distributions of Random Vectors

In order to derive the expected value and variance of the OLS estimators using matrices, we need
to define the expected value and variance of a random vector. As its name suggests, a random
vector is simply a vector of random variables. We also need to define the multivariate normal
distribution. These concepts are simply extensions of those covered in Appendix B.

Expected Value

Or

Properties of Expected Values

15
Variance-Covariance Matrix

Assuming

Properties of Variance:

Multivariate Normal Distribution

16
It is important to know that a multivariate normal random vector is completely characterized by
its mean and its variance-covariance matrix. Therefore, if y is an n x 1 multivariate normal
random vector with meanμ and variance-covariance matrix Σ, we write

We now state several useful properties of the multivariate normal distribution.

Properties of the Multivariate Normal Distribution

17
Chi-Square Distribution
In Appendix B, we defined a chi-square random variable as the sum of squared independent
standard normal random variables. In vector notation, if u ~ Normal(0,In), then u’u ~ χn2.

Properties of the Chi-Square Distribution:

t Distribution

F Distribution
An F random variable is obtained by taking two independent chi-square random variables and
finding the ratio of each standardized by degrees of freedom.

Properties of F distribution

18