Matrix Definition

A matrix is a rectangular array of numbers, symbols, or expressions, arranged

in rows and columns.

a11 a12 ... a1n

 
 a21 a22 ... a2n 
A= . .. .. 
 
 .. ..
. . . 
am1 am2 ... amn

1 / 27
Matrix Definition and Operations

A matrix is a rectangular array of numbers arranged in rows and columns.

Exercise (Easy): Given matrix
 
2 3
A=
4 5

Compute its transpose AT .

Exercise (Harder): Compute the determinant of matrix A.

2 / 27
Matrix Definition and Operations - Solutions

Solution (Easy): Transpose of matrix A is:

 
2 4
AT =
3 5

Solution (Harder): Determinant of A is:

det(A) = 2(5) − 3(4) = 10 − 12 = −2

3 / 27
Matrix Multiplication

For two matrices A and B to be multipliable, the number of columns in A

must equal the number of rows in B.
Exercise (Easy): Multiply matrices
 
1 2
A=
3 4

and  
2 0
B=
1 3
Exercise (Harder): What is the inverse of the matrix A?

4 / 27
Matrix Multiplication - Solutions

Solution (Easy): The product AB is:

   
1(2) + 2(1) 1(0) + 2(3) 4 6
=
3(2) + 4(1) 3(0) + 4(3) 10 12

Solution (Harder): For a 2x2 matrix, the inverse is given by:

 
1 d −b
A−1 =
det(A) −c a

where:  
a b
A=
c d
So for matrix A:
det(A) = (1 × 4) − (2 × 3) = −2
Using the formula for the inverse of a 2x2 matrix:
 
1 4 −2
A−1 =
−2 −3 1
 
−2 1
A−1 =
1.5 −0.5

5 / 27
Eigenvalues and Eigenvectors (Part 1)

For a square matrix A, λ is an eigenvalue of A and v is its corresponding

eigenvector if:

Av = λv
Exercise (Easy): Find the eigenvalue of:
 
2 0
0 3

Exercise (Harder): Find the eigenvector corresponding to the eigenvalue

found above.

6 / 27
Eigenvalues and Eigenvectors - Solutions (Part 1)

Solution (Easy): 1. Set up the characteristic equation |A − λI| = 0. 2. For

matrix:  
2 0
0 3
The characteristic equation becomes:
2−λ 0
=0
0 3−λ
Solving, we get two eigenvalues: λ1 = 2 and λ2 = 3.
Solution (Harder): 1. Use the eigenvalue λ1 = 2 in Av = λv. 2. For matrix:
 
2 0
0 3
We get:  
0 0
v=0
0 1
Eigenvector corresponding to λ1 = 2 is any scalar multiple of:
 
1
0
Repeat for λ2 = 3 to get:  
0
1
7 / 27
Eigenvalues and Eigenvectors (Part 2)

For a square matrix A, an eigenvalue λ and its corresponding eigenvector v

satisfy:

Av = λv
Exercise (Easy): Find the eigenvalue for:
 
3 1
1 3

Exercise (Harder): Find the eigenvector corresponding to the eigenvalue

found above.

8 / 27
Eigenvalues and Eigenvectors - Solutions (Part 2)

Solution (Easy): 1. Set up the characteristic equation |A − λI| = 0. 2. For

matrix:  
3 1
1 3
Characteristic equation is:

3−λ 1
=0
1 3−λ

Solving, we get eigenvalues: λ1 = 4 and λ2 = 2.

Solution (Harder): For λ1 = 4, using Av = λv yields:
 
−1 1
v=0
1 −1

Eigenvector for λ1 = 4 is:  

1
1

9 / 27
Vector Space

A vector space is a collection of vectors which is closed under addition and

scalar multiplication and satisfies certain axioms.
Exercise (Easy): Is the set of all 2 × 2 matrices a vector space under matrix
addition and scalar multiplication?
Exercise (Harder): Is the set of all symmetric 2 × 2 matrices a vector space
under the same operations?

10 / 27
Vector Space - Solutions

Solution (Easy): 1. Matrix addition and scalar multiplication are defined for
2 × 2 matrices. 2. These matrices satisfy all the axioms of a vector space.
Conclusion: Yes, the set of all 2 × 2 matrices forms a vector space.
Solution (Harder): 1. Consider two symmetric 2 × 2 matrices A and B. Their
sum A + B is also symmetric. 2. For any scalar c, the matrix cA is symmetric.
Conclusion: Yes, the set of all symmetric 2 × 2 matrices forms a vector space.

11 / 27
Projection: Introduction

Projection refers to the ”shadow” or ”reflection” of one vector onto another.

Geometrically, it represents how much of one vector lies in the direction of
another.
u

12 / 27
Projection Formula

For projecting vector u onto vector v:

u·v
projv (u) = v
v·v
u

projv (u)

13 / 27
Orthogonality

Vectors are orthogonal if they are perpendicular to each other. This means
their dot product is zero.
u·v =0
u

14 / 27
Projection Orthogonality

When projecting u onto v, the component of u that’s orthogonal to v is

given by:
u − projv (u)
u

u − projv (u)

15 / 27
Matrix Rank

Definition: The rank of a matrix A is the dimension of the column space (or
equivalently, row space) of A. It gives the number of linearly independent
columns (or rows).
Properties:
▶ rank(A) ≤ min(m, n) where m and n are the number of rows and
columns of A, respectively.
▶ rank(AB) ≤ min(rank(A), rank(B))
▶ If A is invertible, then rank(A) = n (full rank).

16 / 27
Matrix Trace

Definition: The trace of a square matrix A is the sum of its diagonal elements.
n
X
tr(A) = aii
i=1

Properties:
▶ tr(A + B) = tr(A) + tr(B)
▶ tr(kA) = ktr(A) for any scalar k.
▶ tr(AB) = tr(BA)
▶ The trace is the sum of the eigenvalues of the matrix.

17 / 27
Derivatives Related to Vectors

The gradient is a vector containing all the first order partial derivatives of a
function.
 ∂f 
 ∂x1
∂f 
 ∂x 
 2
∇f =  . 
 . 
 . 
∂f
∂xn

Exercise (Easy): Find the gradient of f (x, y) = x2 + y 2 .

Exercise (Harder): Find the direction in which f (x, y) = x2 + y 2 increases
most rapidly at the point (1,1).

18 / 27
Derivatives Related to Vectors - Solutions

Solution (Easy): The gradient of f (x, y) is:

 
2x
∇f =
2y

Solution (Harder): 1. Evaluate the gradient at (1,1).

 
2
∇f (1, 1) =
2

2. This gradient vector points in the direction of steepest ascent. Conclusion:

f (x, y) = x2 + y 2 increases most rapidly in the direction of the vector [2, 2] at
the point (1,1).

19 / 27
Introduction to Vector Derivatives

Definition:
For a scalar-valued function f (v) with v ∈ Rn , the gradient is a vector of all
its first-order partial derivatives. Given by:
 T
∂f ∂f ∂f
∇f (v) = , ,...,
∂v1 ∂v2 ∂vn

20 / 27
Linearity of Derivatives

Property 1:
Given scalar-valued functions f (v) and g(v), the derivative of their sum is the
sum of their derivatives:

∇(f + g) = ∇f + ∇g

Proof:
Using the linearity of differentiation:

∂(f + g) ∂f ∂g
= +
∂vi ∂vi ∂vi

21 / 27
Derivative of a Dot Product

Property 2:
For vectors v and w in Rn , the derivative of their dot product with respect to
v is:
∂(vT w)
=w
∂v
Proof:
Given h(v) = vT w

∂h ∂
= (v1 w1 + v2 w2 + · · · + vi wi + . . . )
∂vi ∂vi
∂vi wi
=
∂vi
= wi

∂(vT w)
Thus, ∂v
= w.

22 / 27
Derivative of a Vector Norm

Property 3:
The derivative of the squared norm of vector v is:

∂(vT v)
= 2v
∂v
Proof:
Expanding the squared norm:

vT v = v12 + v22 + . . . + vn
2

Differentiating with respect to vi :

∂(v12 + v22 + . . .)
= 2vi
∂vi

23 / 27
Derivatives Related to Matrices

The derivative of a function with respect to a matrix, often referred to as the

matrix derivative, is a higher-order tensor.
Exercise (Easy): Given f (A) = tr(AT A), where A is a m × n matrix and tr is
∂f
the trace of a matrix, compute ∂A .
Exercise (Harder): For the matrix function f (A) = tr(AB), where B is a
∂f
constant matrix, compute ∂A .

24 / 27
Derivatives Related to Matrices - Solutions
∂tr(AT A) ∂f
Solution (Easy): 1. Use the property: ∂A
= 2A. Thus, ∂A = 2A.
∂tr(AB) T . Hence, ∂f
Solution (Harder): 1. Use the property: ∂A
= B ∂A
= BT .

25 / 27
Given matrix  
2 1
A=
1 3

1. Compute A2 .
2. Determine if A is diagonalizable.
3. Find the eigenvalues and eigenvectors of A.
4. Compute the determinant of A.
5. For the function f (v) = Av, compute the derivative with respect to v.
6. Define the vector space spanned by the eigenvectors of A.

26 / 27
Solutions for Question 1
     
2 1 2 1 5 5
1. A2 = × = .
1 3 1 3 5 10
2. Eigenvalues of A are λ1 = 1 and λ2 = 4, since both are distinct, A is
diagonalizable.
 
1
3. Eigenvalue λ1 = 1 has eigenvector v1 = and eigenvalue λ2 = 4 has
−1
 
1
eigenvector v2 = .
1
4. det(A) = 2 × 3 − 1 × 1 = 5.
∂f
5. ∂v
= A.
6. Vector space spanned by v1 and v2 is R2 .

27 / 27