Lecture Notes on Computational Mathematics

Dr. K. Manjunatha Prasad

Professor of Mathematics,
Department of Data Science, PSPH
Manipal Academy of Higher Education, Manipal, Karnataka-576 104
kmprasad63@gmail.com, km.prasad@manipal.edu
 Lecture Notes on Computational Mathematics

Dr. K. Manjunatha Prasad

Department of Data Science, PSPH
Manipal Academy of Higher Education, Manipal, India

M.Sc Data Science/Biostatistics/Digital Epidemiology (I sem), Batch 2021-2022

Contents

1 Introduction to Matrix theory and Linear Algebra 4

1.1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 System of Linear Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Elementary Operations and Elementary Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Definition and Basics of Vector Spaces and Subspaces . . . . . . . . . . . . . . . . . . . . . . . 22
1.5 Linearly Independent Vectors, Basis of a Vector Space . . . . . . . . . . . . . . . . . . . . . . 24
1.6 Linear Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.7 Rank-Nullity Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.8 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
1.9 Inner Product Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

2 Decomposition of Matrices, Generalized inverses 39

2.1 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.2 Eigenvalues, Positive Definite Matrices and Decompositions . . . . . . . . . . . . . . . . . . . 44
2.3 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.3.1 Characteristics of a diagonalizable matrix . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.4 Positive Definite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.4.1 Properties of PD and PSD matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.5 Spectral Decomposition Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.6 Singular Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.7 The Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.8 LU Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.9 Cholesky Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.10 Generalized Inverses and Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

1
 Contents

2.11 Construction of generalized inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

2.12 Minimum Norm, Least Squares g-inverse and Moore-Penrose inverse . . . . . . . . . . . . . 59
2.12.1 Construction of Moore-Penrose inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

3 Graphs and algorithms 64

3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2 Incidence matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.2.1 Rank of Incidence matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2.2 Minor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3 Adjacency Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.3.2 Eigenvalues of Some Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.4 Laplacian Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.4.1 Properties of Laplacian Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.5 Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.5.1 Some properties of Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.5.2 Spanning Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.6 Graph theoretic algorithms for shortest spanning tree . . . . . . . . . . . . . . . . . . . . . . 75
3.6.1 Kruskal’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.6.2 Prim’s algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.7 Dijkstra’s Shortest Path algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4 Differential Calculus 81
4.1 Continuity and Differentiability of Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.2 Rolle’s Theorem, Mean Value Theorems, Taylor’s Theorem . . . . . . . . . . . . . . . . . . . . 83
4.3 L’Hospital’s Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3.1 L’Hospital’s Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.4 Taylor’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.5 Differentiation of vector valued functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

5 Numerical Methods 92
5.1 Solution of Algebraic and Transcendental Equations . . . . . . . . . . . . . . . . . . . . . . . 92
5.2 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.3 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.4 Newton-Raphson Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.5 Numerical Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.5.1 Finite differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.6 Newton-Cotes quadrature formula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.6.1 Trapezoidal Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.6.2 Simpson’s one-third rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Lecture Notes 2
 Contents

5.6.3 Simpson’s three-eighth rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

Lecture Notes 3
 Introduction to Matrix theory and Linear Algebra

1. Introduction to Matrix theory and Linear Algebra

1.1 Matrices
By m × n matrix, we mean an array of elements with m rows and n columns. The size or order of a
matrix is said to be of m × n if the matrix is with m rows and n columns. If A is an m × n matrix, the
element of A in the ith row and jth column is denoted by A(i, j) or A i j .
 
a 11 a 12 ··· a 1n
 
 a 21 a 22 ··· a 2n 
 
A=
 .. .. .. ..  .
 (1.1)
 . . . . 
 
a m1 a m2 ··· a mn

Henceforth in our discussion, we assume that the matrices are with scalars as elements unless indicated
otherwise.
Row and column Matrices: A matrix with just one row is known as row matrix and the matrix
with one column is known as column matrix.
Null or Zero Matrix: A null matrix or a zero matrix is the matrix with all entries equals to zero.
Submatrix of a Matrix: A submatrix of a given matrix is a matrix obtained by selecting entries
from columns and rows of given matrix, preserving the order. In other words, a submatrix obtained
by sub index set { i 1 , i 2 , · · · , i k } of {1, 2, · · · , m} in the increasing order and sub index set { j 1 , j 2 , · · · , j p } of
{1, 2, · · · , n} in the increasing order, the associated submatrix B is
 
a i 1 j1 a i 1 j2 · · · a i 1 j p
 
a i 2 j1 a i 2 j2 · · · a i 2 j p 
 
B= .

.. .. ..  .
 (1.2)
 .. . . . 
 
a i k j1 a i k j2 · · · a i k j p

Block Matrices: A block matrix is a matrix whose entries are the matrices, called blocks, of suitable
size so that by ignoring the block structure we obtain
 a
structure of a matrix with entries considered
A B
for the blocks. Consider a 2 × 2 block matrix T =   with blocks A, B, C & D; in which case, the
C D
number of columns of A and C are same, number of rows of A and B are same. Also, the number of
rows of D is same as that of C and the number of columns of D is same as the number of columns of
B. If A is a matrix of size m × n and D is a matrix of size k × p, then the 2 × 2 block matrix T is of size

Lecture Notes 4
 1.1 Matrices

(m + k) × (n + p) and
 
a 11 a 12 ··· a 1n b 11 b 12 ··· b1 p
 
a a 22 ··· a 2n b 21 b 22 ··· b2 p 
 21 
 .. .. .. .. .. .. .. .. 
 
 . . . . . . . . 
   
A B a a m2 a mn b m1 b m2 b mn 
 
··· ···
T =  =  m1 . (1.3)
C D  c 11 c 12 ··· c 1n d 11 d 12 ··· d1 p 
 
 
 c 21 c 22 ··· c 2n d 21 d 22 ··· d2 p 
 
 .. .. .. .. .. .. 
 
.. ..
 . . . . . . . . 
 
c k1 c k2 ··· c kn d k1 d k2 ··· dk p

Transpose and Conjugate transposes: Given an m × n matrix A, transpose of the matrix is an

n × m matrix denoted by A T , where (A T ) ji = A i j . In fact,
 
a 11 a 21 ··· a m1
 
 a 12 a 22 ··· a m2 
 
T
A =
 .. .. .. .. 
 . (1.4)
 . . . . 
 
a 1n a 2n ··· a mn
n× m

Whenever we consider matrices over a scalar field like complex numbers, we define conjugate transpose
of given matrix by  
a 11 a 21 ··· a m1
 
 a 12 a 22 ··· a m2 
 
A∗ = 
 .. .. .. .. 
 (1.5)
 . . . . 
 
a 1n a 2n ··· a mn
n× m
where, z = x − i y is the conjugate of the complex z = x + i y.
 
2 8 5 1  
  3+ i 2 4 − 2i
Example 1. Let A =  3 5 1 4 and B =
   be the given matrices. Then the trans-
6 7 − i 2 − 3i
0 6 3 7
 
2 3 0  
  3 −i 6
8 5 6
pose of A is A T =   and the conjugate transpose of B is B∗ = 
 
 2 7+ i .
 
5 1 3
4 + 2i 2 + 3i
 
1 4 7

Matrix Addition: For any two matrices A, B over the real field R (or complex field C), we define
the matrix addition if they are of same size. If A, B are matrices of size m × n, the matrix addition is

Lecture Notes 5
 1.1 Matrices

defined by (A + B) i j = A i j + B i j , i.e.,
   
a 11 a 12 ··· a 1n b 11 b 12 ··· b 1n
   
 a 21 a 22 ··· a 2n   b 21 b 22 ··· b 2n 
   
A+B =
 .. .. .. ..  +  ..
 
.. .. .. 
 (1.6)
 . . . .  .
  . . . 

 
a m1 a m2 ··· a mn b m1 b m2 ··· b mn
 
a + b 11 a 12 + b 12 ··· a 1n + b 1n
 11 
 a 21 + b 21 a 22 + b 22 ··· a 2n + b 2n
 

= .. .. .. ..
 (1.7)
.
 

 . . . 

a m1 + b m1 a m2 + b m2 ··· a mn + a mn

Matrix Multiplication: For any two matrices A, B over the real field R (or complex field C), we
define the matrix multiplication if the number of columns of the preceding is same as the number of
rows of the latter. Further, if A, B are matrices of size m × n and n × p respectively, then the matrix
n
P
multiplication AB = C is defined by C i j = A ik B k j .
k=1
  
a 11 a 12 ··· a 1n b 11 b 12 ··· b 1n
  
 a 21 a 22 · · · a 2n   b 21 b 22 · · · b 2n 
  
AB = .

.. .. ..   ..
 
.. .. .. 
 (1.8)
 .. . . .  . . . . 
  
a m1 a m2 · · · a mn b m1 b m2 · · · b mn
 n n n 
P P P
a 1 k b k1 a 1 k b k2 · · · a 1k b k p
 k=1 k=1 k=1 
 n n n 
 P P P
a b a b a 2k b k p 

···
 k=1 2k k1 k=1 2k k2

= k=1 
(1.9)
.. .. .. ..

.
 

 . . . 

P n Pn Pn 
a mk b k1 a mk b k2 · · · a mk b k p
k=1 k=1 k=1

Clearly, the product matrix in the above is of size m × p, where the number of rows is same as that of
preceding matrix in the product and the number of columns is same as that of the latter.
A good understanding of the definition of matrix multiplication is quite useful. We note some simple
facts which are often required. We assume that all products occurring here are defined in the sense that
the orders of the matrices make them compatible for multiplication.

(i) The j-th column of AB is the same as A multiplied by the j-th column of B.

(ii) The i-th row of AB is the same as the i-th row of A multiplied by B.

(iii) The (i, j)-th entry of ABC is obtained as


y1
 . 
h i  
x1 x2 ··· x p B  .. 
 
yq

Lecture Notes 6
 1.1 Matrices

 
y1
 . 
h i  
where x1 x2 ··· x p is the i-th row of A and  ..  is the j-th column of C.
 
yq
 
b 1T
 . 
h i  
(iv) If A = a 1 a 2 · · · a n and B =  ..  where a i (1 ≤ i ≤ n) denote columns of A and b Tj (1 ≤ j ≤ n)
 
bT n
denote rows of B, then
AB = a 1 b 1T + · · · + a n b T
n.
   
A 11 A 12 B11 B12
Block multiplication: Let A =  , B =  , be two matrices where each A i j
A 21 A 22 B21 B22
and B i j is itself a matrix. If compatibility for matrix multiplication is assumed throughout then we can
write  
A 11 B11 + A 12 B21 A 11 B12 + A 12 B22
AB =  .
A 21 B11 + A 22 B21 A 21 B12 + A 22 B22

Scalar Multiplication: If A is an m × n matrix over R (or C) and α is a scalar, then B = α A is an

m × n matrix such that B i j = α A i j for all i, j.
Square Matrix: A matrix is said to be a square matrix when the number of rows is same as that of
columns.
Diagonal Matrix: Diagonal matrix is a square matrix where only the possible nonzero entries are
seen on the diagonal and off diagonal entries are zeros.
Identity Matrix: Identity matrix of order n is the square matrix of size n × n, with diagonal entries
equal to 1 and the rest are zeros.
Identity matrices are in general denoted by I without any regard to its size and the size is assumed
conveniently depending on its position in the product or sum or in the equation. If I m , I n and A are the
matrices of order m, n and m × n respectively, then by the definition of matrix multiplication, it is easily
observed that I m A = A I n = A.
Lower triangular and upper triangular matrix: A square matrix is called lower triangular
matrix if a i j = 0, i < j.
A square matrix is called upper triangular matrix if a i j = 0, i > j. The transpose of one type of triangular
matrix is the triangular matrix of other type.
Inverse of a Matrix: Given an m × n matrix A, an n × m matrix B is said to be a right inverse of A
if AB = I m . Similarly, B is a left inverse of A if BA = I n .
1 −1
Right inverse and left inverse of a matrix A, whenever exists, are in general denoted by A −
r and A l
respectively. A square matrix A of size n × n is said to have inverse (A is invertible) if there exists a
matrix B of size n × n such that AB = BA = I n . Inverse of A, whenever it exists, is denoted by A −1 .
 
4 1 5
Example 2. Consider the matrix A =  . This matrix is right invertible but it has no left inverse
3 1 4

Lecture Notes 7
 1.1 Matrices

 
−1 −2
1 
and A −
 
r = −5 3.
2 1
 
−3 −4
 
Example 3. Consider the matrix A = 
4 6. This matrix is left invertible but it has no right inverse
1 1
 
11 10 16
−
1  9
−9 9 
and A −
l
= .
7 8
9 9 − 11
9
 
1 11 3
 
Example 4. Consider the matrix A =  1 19 5. This matrix has both left inverse and right inverse,


0 3 1
 
2 −1 −1
is equal to A −1 and is given by A −1 =  1 1
 
− − 1 .
 2 2 
3 3
2 − 2 4

Exercise 1.1. Prove that a matrix with one or more zero rows has no right inverse.

Solution. Without loss of generality, we may assume that k-th row of A m×n is zero, i.e.,
 
a 11 a 12 ··· a 1n
.. .. .. ..
 
.
 

 . . . 

a k−1 1 a k−1 2 a k−1
 
··· n
 
A= 0 0 ··· 0 .
 
 
a k+1 1 a k+1 2 ··· a k+1
 
n
 .. .. ..
 
.. 
 . . . . 
 
a m1 a m2 ··· a mn
 
b 11 b 12 ··· b 1m
 .. .. .. ..
 
.

 . . . 
 
 b k−1 1 b k−1 2 b k−1 m 
 
···
 
Suppose B =  b k1 b k2 ··· b km  is the right inverse of A. Then AB = I.
 
 
 b k+1 1 b k+1 2 ··· b k+1 m 
 
 .. .. .. 
 
..
 . . . . 
 
b n1 b n2 ··· b nm

Lecture Notes 8
 1.1 Matrices

But
  
a 11 a 12 ··· a 1n b 11 b 12 ··· b 1m
.. .. .. .. .. .. .. ..
  
. .
  

 . . . 
 . . . 

a k−1 1 a k−1 2 a k−1 b b k−1 2 b k−1 m 
  
··· n ···
   k−1 1 
AB =  0 0 ··· 0   b k1 b k2 ··· b km 
  
  
a k+1 1 a k+1 2 ··· a k+1 n   b k+1 1 b k+1 2 ··· b k+1 m 
  
 .. .. ..   .. .. .. 
  
.. ..
 . . . .  . . . . 
  
a m1 a m2 ··· a mn b n1 b n2 ··· b nm

 P n n
P n
P 
a b a 1l b l 2 ··· a 1l b l p
 l =1 1l l 1 l =1 l =1 

.. .. .. .. 
. . . .
 
 
 n n n

P P P 
 a k−1 l b l 1 a k−1 l b l 2 ··· a k−1 l b l p 
l =1 l =1 l =1
 

= 0 0 ··· 0  ̸= I
 
 n n n

P P P 
 a k+1 l b l 1 a k+1 l b l 2 ··· a k+1 l b l p 
l =1 l =1 l =1

.. .. ..
 
 .. 

 . . . . 

 P n n
P Pn 
a ml b l 1 a ml b l 2 ··· a ml b l p
l =1 l =1 l =1

Hence, the proof.

Exercise 1.2. Prove that a matrix with one or more zero columns has no left inverse.

Trace of a Square Matrix: Trace of a square matrix A = (a i j ) of order n is the sum of its n diagonal
entries. In other words,
n
X
T race(A) = a ii (1.10)
i =1
 
4 0 3 2
 
5 7 3 8
Example 5. Let A =   be the given matrix. Here, T race(A) = 4 + 7 + 3 + 5 = 19.
 
1 4 3 9
 
0 2 9 5

The following are very basic properties of matrix addition and multiplication.

(i) Matrix addition is commutative i.e., A + B = B + A.

(ii) Matrix addition is associative i.e., A + (B + C) = (A + B) + C.

(iii) Matrix multiplication is associative i.e., A(BC) = (AB)C.

(iv) Distribution property of matrix multiplication over addition i.e., A(B + C) = AB + AC.

(v) (A + B)T = A T + B T , whenever the addition is well defined.

Lecture Notes 9
 1.2 System of Linear Equations

(vi) (AB)T = B T A T , whenever the multiplication is well defined.

(vii) (A T )T = A.

(viii) Given a matrix A of size m × n and the matrix B, both the products AB and BA are well defined
if and only if B is a matrix of size n × m.

(ix) When the addition A + B is well defined for square matrices A and B, then

T race(A + B) = T race(A) + T race(B).

(x) When both products AB and BA are well defined, then T race(AB) = T race(BA).

(xi) For a square matrix A and a scalar α, T race(α A) = αT race(A).

(xii) Matrix multiplication need not be commutative. Many times, BA may not have been well defined,
even if AB exists, because of their size. In the following example AB and BA are different in spite
of the products are well defined:
         
1 1 1 0 2 0 1 0 1 1 1 1
  = ;  = . (1.11)
0 0 1 0 0 0 1 0 0 0 1 1

Exercise 1.3. Verify the properties (i) − (xi) above for matrix addition, multiplication, transpose and the
trace.
 
1 0 0
 
Exercise 1.4. For A = 
0 0 0, verify if there exist matrices X and Y such that X Y − Y X = A.


0 0 0
Exercise 1.5. Inverse of a matrix, whenever it exists, is unique.

Exercise 1.6. If an m × n matrix has both left inverse and right inverse, then the given matrix is square
and invertible.

Exercise 1.7. For any matrix A, show that A = 0 if and only if Trace A T A = 0.

Exercise 1.8. Let A be an n × n matrix such that trace AB = 0 for every n × n matrix B. Can we conclude
that A must be the zero matrix?

1.2 System of Linear Equations

In several practical situations, we come across system of ‘m’ linear equations with ‘n’ unknown variables
x j (1 ≤ j ≤ n) associated with coefficients a i j (1 ≤ i ≤ m; 1 ≤ j ≤ n) and the constants b i (1 ≤ i ≤ m) from the
scalar field R.

a 11 x1 + a 12 x2 + · · · + a 1n xn = b1

a 21 x1 + a 22 x2 + · · · + a 2n xn = b2

··· ··· ··· ··· ··· ··· ··· (1.12)

a m1 x1 + a m2 x2 + · · · + a mn xn = bm

Lecture Notes 10
 1.2 System of Linear Equations

The set of equations as above in equation (1.12) is called system of m linear equations with n unknowns.
In fact, a linear equation is an equation in which the degree of the variables involved is one. We also
write the above system of equations in the form of matrix equation as
    
a 11 a 12 · · · a 1n x b
  1  1 
 a 21 a 22 · · · a 2n   x2   b 2 
    
 ..

.. .. ..   ..  =  .. 
    (1.13)
 . . . . 
 .   . 
   

a m1 a m2 ··· a mn xn bm

or simply
Ax = b

where

• A is a matrix of size m × n (A ∈ Rm×n ) with (i, j)-th entry equals to constant coefficient,
h iT
• x is a variable column matrix x1 x2 · · · xn of size n × 1 (x ∈ Rn ) and
h iT
• b is a constant column matrix b 1 b2 ··· bm of size m × 1 (b ∈ Rm ).

Clearly, a solution for system of linear equation (1.12) provides a solution for matrix equation (1.13) and
vice versa.
Solution of System of Linear Equations: Any n-tuple (x1 , x2 , · · · , xn ) of elements of scalars which
satisfy each of the equations in (1.12) is called a solution of that system of linear equations.
Consistent System of Linear Equations: A system of linear equations

a 11 x1 + a 12 x2 + · · · + a 1n xn = b 1

a 21 x1 + a 22 x2 + · · · + a 2n xn = b 2

... ... ... ... ... ... ...

a m1 x1 + a m2 x2 + · · · + a mn xn = b m

is said to be consistent if the system has a solution. Otherwise, the system is said to be inconsistent.
Homogeneous System of Linear Equations: A system of linear equations Ax = b is said to be
homogeneous if b 1 , b 2 , · · · , b m , the constant scalar on the right-hand side, are zero’s.

Exercise 1.9. For m × n matrices A and B, A = B if and only if Ax = Bx for every column matrix x of
size n × 1.

Exercise 1.10. For an m × n matrix A having left inverse, the only solution for Ax = 0 is trivial.

Exercise 1.11. For an n × n matrix A having inverse, the solution for Ax = b is unique for every choice
of b.

Lecture Notes 11
 1.3 Elementary Operations and Elementary Matrices

1.3 Elementary Operations and Elementary Matrices

Whenever we consider the system of linear equations as in the following example
x1 − x3 = 2
x2 + 2x3 = 1
a solution is easily obtained by taking x3 = 0; x1 = 2 and x2 = 1. In this section, we develop a technique
that produces a linear system as in the above example which is equivalent to the given linear system.
Elementary Row Operations: The following are basic elementary row operations on a given
matrix:

(i) Interchanging rows of given matrix.

(ii) Multiplying a row by a nonzero constant.

(iii) Replace a row by adding that row by a constant time the other row.

Performing an elementary row operation on m × n matrix is same that of pre-multiplying an elementary

matrix obtained by performing the same row operation on an identity matrix I m .
In the following we have elementary matrices for row operations:
 
0 0 1
 
0 1 0
  (Interchanging row 1 and row 3)
1 0 0
 
1 0 0
 
0 c 0 (Multiplying a nonzero constant c to row 2)
 
0 0 1
 
1 0 0
 
0 1 c 
  (Add c times row 3 to row 2)
0 0 1
The inverse of a row operation is the one to be performed on the matrix obtained after performing the
row operation to revert back to the original matrix. In the following, we note that inverse of each of the
elementary row operations is again an elementary row operation. Clearly, from the above, elementary

Elementary Row Operation Inverse Operation

Interchange of i-th and j-th rows (i ̸= j) Interchange of i-th and j-th rows (i ̸= j) (Same)
Multiply i-th row by a nonzero scalar c Multiply i-th row by a nonzero scalar c−1
Replace i-th row by adding i-th row by a constant c time the j-th row (i ̸= j) Replace i-th row by adding i-th row by a constant − c time the j-th row (i ̸= j)

matrix correspond to each of the elementary row operations is invertible.

The techniques of elementary column operations and results correspond to elementary column ma-
trices are just an analogue.
Row Equivalent Matrices: Given matrices A and B of size m × n, B is said to be row equivalent
to the matrix A if B can obtained by finite sequence of elementary row operations on A.
Note that

Lecture Notes 12
 1.3 Elementary Operations and Elementary Matrices

(i) If B is row equivalent to the matrix A and obtained by performing finite sequence of elementary
row operations, we can get A from B by performing inverse operations on B in the reverse order.

(ii) If B is row equivalent to the matrix A and obtained by performing finite sequence of elementary
row operations, B can be obtained by pre-multiplying the elementary matrices correspond to those
elementary row operations in the same order. i.e.,

B = E k E k−1 . . . E 1 A

where E 1 , E 2 , . . . , E k are the elementary matrices correspond to those elementary row operations.
Since, each of E i in the above are invertible, we have that E k E k−1 . . . E 1 = M is invertible and

B = M A; A = M −1 B

Theorem 1.1. If the matrices A and B of size m × n are row equivalent, then the homogeneous system of
linear equations Ax = 0 and Bx = 0 have same solutions.

Row Reduced Matrix: An m × n matrix A said to be in the row reduced form if

(i) the leading nonzero entry, if exists, in any row is equal to one

(ii) each column that contain leading nonzero entry of any row has all other entries as zero.

Example 6. The following matrices are the examples for the matrices in row reduced form:
   
0 1 3 0     1 0 0
  1 0 −1 0 0 1  
1 0 0 0 ,  ,  , 0 1 0
0 1 3 0 1 0
   
0 0 0 1 0 0 1

Example 7. The following are not among the examples for the matrices in row reduced form:
   
2 0 0 0 0 1 0 1  
    1 0 2
0 1 3 4 , 1 2 0 0 ,  
0 0 1
   
0 0 0 0 0 0 1 0
Row Reduced Echelon Form: An m × n matrix A is said to be in the row reduced echelon form if

(i) A is in the row reduced form

(ii) every row below a null row is also null
(iii) if k 1 , k 2 , · · · , k r are the position of leading nonzero entries in the rows 1, 2, · · · , r respectively, then
k1 < k2 < · · · < k r .

Example 8. Find the row equivalent matrix of A in the row reduced echelon form:
 
1 3 1 −2 −3
 
1 4 3 −1 −4
A= .
 
2 3 −4 −7 −3
 
3 8 1 −7 −8

Lecture Notes 13
 1.3 Elementary Operations and Elementary Matrices

Solution. Perform row operations on the given matrix to get the row equivalent matrix in the row
reduced echelon form:
   
1 3 1 −2 −3 1 3 1 −2 −3
   
1 4 3 −1 −4 ≃ 0 1 2 −1 1
 
R 2 → R 2 + (−1)R 1 
 
 
2 3 −4 −7 −3 2 3 −4 −7 −3
   
3 8 1 −7 −8 3 8 1 −7 −8
 
1 3 1 −2 −3
 
≃ 0 1 2 1
−1
R 3 → R 3 + (−2)R 1 
 

0 −3 −6 −3 3 
 
3 8 1 −7 −8
 
1 3 1 −2 −3
 
≃ 0 1 2 1 −1

R 4 → R 4 + (−3)R 1 


0 −3 −6 −3 3 
 
0 −1 −2 −1 1
 
1 3 1 −2 −3
 
≃ 0 1 2 1 −1
R 3 → R 3 + (3)R 2 
 

0 0 0 0 0
 
0 −1 −2 −1 1
 
1 3 1 −2 −3
 
≃ 0 1 2 1 −1
R4 → R4 + R2 
 

0 0 0 0 0 
 
0 0 0 0 0
 
1 0 −5 −5 0
 
≃ 0 1 2 1 −1
R 1 → R 1 + (−3)R 2 
 

0 0 0 0 0 
 
0 0 0 0 0
 
1 0 −5 −5 0
 
0 1 2 1 −1
The row reduced echelon form of the given matrix A is  .
 
0 0 0 0 0
 
0 0 0 0 0
Rank of Matrix: Rank of given matrix A is the number of nonzero rows in the matrix B in the row
reduced echelon form which is row equivalent to A.

Theorem 1.2. For an n × n matrix A, the following statements are equivalent.

(i) A has inverse (invertible).

(ii) Homogeneous system of linear equations Ax = 0 has only a trivial solution.

(iii) A is row equivalent to an identity matrix I n .

Lecture Notes 14
 1.3 Elementary Operations and Elementary Matrices

(iv) A is a product of elementary matrices.

Given a linear system Ax = b, we have the following row reduction method to verify if the linear
system is consistent and give a solution whenever exists.
Solving Linear System by Row Reduction Method: Construct an augmented matrix T of size
m × (n + 1) taking A and b side by side
h i
T= A b

Perform sequence of elementary row operations to obtain a row equivalent matrix R in the row reduced
echelon form. If R is in the form
h i
R= B d

where B is the matrix in row reduced echelon form in the block correspond to A and d is the column
matrix in the block correspond to d.

• Conclude that the matrix equation Ax = b is inconsistent if the number of nonzero rows in B is
not same as that of R.

• Conclude that the matrix equation Ax = b is consistent if the number of nonzero rows in B is the
same as that of R and solution can be obtained by considering the linear system given by Bx = d.

Example 9. Determine the values of α, β so that the following system of linear equations has:

(i) no solution

(ii) more than one solution

(iii) a unique solution

2x + 3y + 5z = 9, 7x + 3y − 2z = 8, 2x + 3y + α z = β

Solution. The matrix form of the given system of equations is A X = b

          
2 3 5 x 9 2 3 5 x 9
          
i.e., 7 3 −2  y =  8  , where A = 7 3 −2 , X =  y , b =  8 
          

2 3 α z β 2 3 α z β

Consider the augmented matrix

 
2 3 5 9
 
[A b] = 
7 3 −2 8 


2 3 α β

Lecture Notes 15
 1.3 Elementary Operations and Elementary Matrices

Transform [A b] to row reduced echelon form by performing elementary row operations

   
2 3 5 9 2 3 5 9
  ≃  
[A b] = 
7 3 − 2 8  R 3 → R 3 + (−1)R 1 7 3 −2
  8 

2 3 α β 0 0 α−5 β−9

1 32 5 9
 
3 5 9

≃ 2 2 1 2 2 2
1   ≃
 R 2 → R 2 + (−7)R 1 0 −15

−39 −47 

R1 → R2  7 3 − 2 8 2 2 2
2    
0 0 α−5 β−9 0 0 α−5 β−9

−7 −1
 
−7 −1

≃ 1 0 5 5 ≃ 1 0 5 5
1  −47  R → −2 R 
0 −215 −39 39 47 
  
R1 → R1 + R2  2 2 2 2 0 1 15 15
5   15 
0 0 α−5 β−9 0 0 α−5 β−9

(a) No solution: If α = 5 and β ̸= 9, the matrix obtained in row reduced echelon form will be

−7 −1
 
−7 −1

1 0 5 5 ≃ 1 0 5 5
39 47 R 4 → R 4 0 1 39 47 
   
0 1
 15 15  β−9  15 15 
0 0 0 β−9 0 0 0 1

−7

≃ 1 0 5 0
1 
39 47 

R 1 → R 3 + R 1 0
 1 15 15 
5
0 0 0 1

1 0 −57 0

≃
47
0 1 39
 
R2 → − R3 + R2  15 0
15  
0 0 0 1

Since the number of nonzero rows in the row reduced echelon form of A is not same as that of
[A b], we must have rank [A b] ̸= rank A. Hence the given system of linear equations Ax = b is
inconsistent, i.e., the system has no solution.

(b) More than one solution: If α = 5 and β = 9, the matrix obtained in row reduced echelon form will
be 
−7 −1

1 0 5 5
39 47 
 
0 1
 15 15 
0 0 0 0
Since the number of nonzero rows in the row reduced echelon form of A is same as that of [A b], we
must have rank [A b] = rank A. Hence the given system of linear equations Ax = b is consistent.
Since rank A < the number of unknowns, therefore the system has more than one solution.

(c) Unique solution: If α ̸= 5 and β be any real, the matrix obtained in row reduced echelon form will

Lecture Notes 16
 1.3 Elementary Operations and Elementary Matrices

be

−7 −1
 
−7 −1

1 0 5 5 ≃ 1 0 5 5

0 1 39 47

R → R 3 
0 1 39 47 

 15 15  3 α−5  15 15 
β−9
0 0 α−5 β−9 0 0 1 α−5
7(β−9)

−1

≃ 1 0 0 5 + 5(α−5)
7 
39 47

R1 → R3 + R1 
0 1 15 15

5 
β−9
0 0 1 α−5
7(β−9)

−1

≃ 1 0 0 5 + 5(α−5)
39 
47 39(β−9) 

R2 → − R3 + R2 
0 1 0 15 − 15(α−5) 
15
β−9
0 0 1 α−5

Since the number of nonzero rows in the row reduced echelon form of A is same as that of [A b],
we must have rank [A b] = rank A. Hence the given system of equations Ax = b is consistent.
Since rank A = the number of unknowns, therefore the system has unique solution.

Finding a basis for null space of a matrix by Row Reduction Method: Let A be an m × n
matrix and let B be its row reduced echelon form and k i , 1 ≤ i ≤ r < m be the positions of columns with
leading nonzero entries of any row. Define S = { k i , 1 ≤ i ≤ r }. Then a basis of null space of B contains
exactly n − r vectors given by
{v j : j ∉ S },

where v j = (yp )n×1 , 1 ≤ p ≤ n is given by





1, if p = j
yp = 0, if p ̸= j and p ∉ S


− b i j , if p = k i ∈ S.


Note that null space is the set of all vectors x satisfying the system of homogeneous equation Ax = 0.

Example 10. Find a basis for null space of the matrix:

 
1 3 1 −2 −3
 
1 4 3
−1 −4
A= .
 
2 3 −4 −7 −3
 
3 8 1 −7 −8

Solution. Perform row operations on the given matrix to get the row equivalent matrix in the row

Lecture Notes 17
 1.3 Elementary Operations and Elementary Matrices

reduced echelon form:

   
1 3 1 −2 −3 1 3 1 −2 −3
   
1 4 3 −1 −4 ≃ 0 1 2−1 1
 
R 2 → R 2 + (−1)R 1 
 
 
2 3 −4 −7 −3 2 3 −4 −7 −3
   
3 8 1 −7 −8 3 8 1 −7 −8
 
1 3 1 −2 −3
 
≃ 0 1 −12 1

R 3 → R 3 + (−2)R 1 


0 −3 −6 −3 3 
 
3 8 1 −7 −8
 
1 3 1 −2 −3
 
≃ 0 1 2 1 −1
R 4 → R 4 + (−3)R 1 
 

0 −3 −6 −3 3 
 
0 −1 −2 −1 1
 
1 3 1 −2 −3
 
≃ 0 1 2 1 −1

R 3 → R 3 + (3)R 2 


0 0 0 0 0
 
0 −1 −2 −1 1
 
1 3 1 −2 −3
 
≃ 0 1 2 1 −1
R4 → R4 + R2 
 

0 0 0 0 0
 
0 0 0 0 0
 
1 0 −5 −5 0
 
≃ 0 1 2 1 −1
R 1 → R 1 + (−3)R 2 
 

0 0 0 0 0
 
0 0 0 0 0
 
1 0 −5 −5 0
 
0 1 2 1 −1
The row reduced echelon form of the given matrix A is B =  .
 
0 0 0 0 0
 
0 0 0 0 0
Note that here S = {1, 2} ⊆ {1, 2, 3, 4, 5}. A basis for null space of A is given by

B = {v j : j ∉ S } = {v3 , v4 , v5 },

where v j = (yp )n×1 , 1 ≤ p ≤ n is given by





1, if p = j
yp = 0, if p ̸= j and p ∉ S


− b i j , if p = k i ∈ S.


Lecture Notes 18
 1.3 Elementary Operations and Elementary Matrices

So,
     
5 5 0
     
−2 −1 1
     
v3 =  4 
v5 = 
     
1
 , v =  0 ,

0

     
0 1 0
     
0 0 1
Thus      


 5 5 0 

       
−2


   −1 1 
     
   
B= 
 
 1 ,
  0  , 0
    
      
 0 


    1  0 
     

 
0 0 1
 

Finding Inverse of a Matrix by Row Reduction Method: Given an n × n matrix A, construct

an augmented matrix T of size n × 2n by taking A and identity matrix I side by side
h i
T= A I

Reduce T in to row reduced echelon form

h i
R= B C

by performing a sequence of elementary row operations.

• Conclude that A is not invertible if the matrix B in the row reduced matrix has any zero row.

• Conclude that A is invertible if the matrix B in the row reduced matrix is I. Then the correspond-
ing C is the inverse of A.
 
1 1 0
 
Example 11. Find the inverse of the matrix A = 
1 −1 1, if exists.


1 −1 2

Solution. Consider the augmented matrix T of size 3 × 6 which is obtained by adjoining the columns of
identity matrix to A.
 
1 1 0 1 0 0
 
T = [A I] = 
1 −1 1 0 1 0


1 −1 2 0 0 1
Transform T to row reduced echelon form [B C] by performing a sequence of elementary row operations.
   
1 1 0 1 0 0 1 1 0 1 0 0
  ≃  
T = [A I] = 
1 −1 1 0 1 0 R 2 → R 2 + (−1)R 1 0 −2 1 −1 1 0
  

1 −1 2 0 0 1 1 −1 2 0 0 1

Lecture Notes 19
 1.3 Elementary Operations and Elementary Matrices

   
1 1 0 1 0 0 ≃ 1 1 0 1 0 0
≃   −1 
−1 1 −1

0 −2 1 −1 1 0 R 2 → 2 R 2 0 1
R 3 → R 3 + (−1)R 1   
2 2 2 0

0 −2 2 −1 0 1 0 −2 2 −1 0 1

   
1 1 0 1 0 0 ≃ 1 1 0 1 0 0
≃ 
−1 1 −1
 1  1 1

R 3 → R 3 + (2)R 2 0 1

2 2 2 0R 2 → R 2 + R 3 0 1
  0 2 −1 2
2
0 0 1 0 −1 1 0 0 1 0 −1 1

1 −1

1 0 0 2 1 2
≃
1 1 
 
R 1 → R 1 + (−1)R 2 
0 1 0 2 −1 2 
0 0 1 0 −1 1

The matrix in row reduced echelon form obtained above can be interpreted as [B C] where B = I 3 and

1 −1

1
2 2
1 1 .

C=2 −1 2 
0 −1 1

1
1 −21

2
Therefore, the inverse of A is A −1 =  1 1 
 
 2 −1 2  .
0 −1 1
 
2 2 3
 
1 0 −1, if exists.
Exercise 1.12. Find the inverse of the matrix A =  

0 1 1
 
1 2 1
 
Exercise 1.13. Find the inverse of the matrix B = 
0 1 0, if exists.


7 4 7
 
3 −6 −6
 
Exercise 1.14. Find the null space of the matrix P = 
−1 2 2.
3 −6 −6

Exercise 1.15. Find row equivalent matrix in the row reduced echelon form for each of the following
   
2 −1 3 2 1 2 −1
   
matrices: 
1 4 0 −1  , 2 1
 1.
2 6 −1 5 0 5 1

Exercise 1.16. Find row equivalent matrix in the row reduced echelon form for each of the following
matrices.Also find the rank of the matrices.
 
1 3 1 −2 −3
 
1 4 3 −1 −4
(i) 
 

2 3 −4 −7 −3
 
3 8 1 −7 −8

Lecture Notes 20
 1.3 Elementary Operations and Elementary Matrices

 
1 2 −3
 
2 1 0
(ii) 
 

−2 −1 3 
 
−1 4 −2
 
1 3
 
 0 −2
(iii) 
 

 5 −1
 
−2 3
 
1 2 0 −1
 
(iv) 
2 6 − 3 −3

3 10 −6 −5

Exercise 1.17. Verify the consistency of each of the system of linear equations given below and find all
solutions whenever the system is consistent.

(i) 4x − 5y + z = −3
2x + 3y − z = 3
3x − y + 2z = 5
x + 2y − 5z = −9

(ii) 2x − 2y + 4z = 9
x − y + 2z = 6
2x − 2y + z = 2
x − y + 0z = 2

(iii) x1 + x2 + x3 + x4 = 2
2x1 − x2 + 2x3 − x4 = −5
3x1 + 2x2 + 3x3 + 4x4 = 7
x1 − 2x2 − 3x3 + 2x4 = 2

(iv) 2x1 − 2x2 + 4x3 + 3x4 = 9

x1 − x2 + 2x3 + 2x4 = 6
2x1 − 2x2 + x3 + 2x4 = 3
x1 − x2 + x4 = 2

(v) 5x + y + z + t = 4
x + 7y + z + t = 12
x + y + 6z + t = −5
x + y + z + 4t = −6

Exercise 1.18. Determine the values of α and β so that the following system of linear equations has:

Lecture Notes 21
 1.4 Definition and Basics of Vector Spaces and Subspaces

(a) no solution.

(b) more than one solution.

(c) a unique solution.

(i) x + y - z = 1
2x + 3y + αz = 3
x + αy + 3z = 2

(ii) x + y + z = 6
x + 2y + 3z = 10
x + 2y + αz = β

1.4 Denition and Basics of Vector Spaces and Subspaces

A set V is called a vector space over the scalar field F, if there exist a binary operation of vector addition
(+) defined by

u + v ∈ V f or all u, v ∈ V (V ector Addition)

and for every α ∈ F, the unary operation (·) defined by α · v ∈ V for all v ∈ V (Scalar Multiplication)
satisfying the following properties:

(i) u + v = v + u for all u, v ∈ V (Commutative)

(ii) u + (v + w) = (u + v) + w for all u, v, w ∈ V (Associative)

(iii) There exists a null vector 0 such that 0 + v = v for all v ∈ V (Existence of zero vector)

(iv) For every v ∈ V there exists w ∈ V such that v + w = 0 (Existence of additive inverse)

(v) α · (β · v) = (αβ) · v for all α, β ∈ F and v ∈ V

(vi) (α + β) · v = α · v + β · v for all α, β ∈ F and v ∈ V

(vii) α · (u + v) = α · u + α · v for all α ∈ F and u, v ∈ V

(viii) 1 · v = v for all v ∈ V .

Objects of the vector space are called vectors. The additive inverse of vector v is unique and written as
−v. For our convenience, we write scalar product α · v as simply αv.

Example 12. For each positive integer k, consider a Cartesian product of k copies of real line R (denoted
by Rk ) that consists of set of all ordered k−tuples x = (x1 , x2 , . . . , xk ) of real numbers. The real number
x i in the i-th position of ordered k−tuples is called i-th coordinates of the points or vector x of Rk . If

Lecture Notes 22
 1.4 Definition and Basics of Vector Spaces and Subspaces

y = (y1 , y2 , . . . , yk ) and α is a real number (also known as scalar in this context) we define the following
operations:
x + y = (x1 + y1 , x2 + y2 , . . . , xk + yk ) (Vector addition)

α x = (α x1 , α x2 , . . . , α xk ) (Scalar multiplication)

satisfy Commutative, Associative and Distributive properties, making Rk a vector space over the field of
real numbers with null vector or zero vector is the k−tuples with zeros as entries.

Exercise 1.19. Prove that the set of all m × n matrices over a field F(F(m×n) ) is with reference to matrix
addition and scalar multiplication is a vector space.

Exercise 1.20. Prove that the set of all functions defined on a set S into field F with f + g defined by
( f + g)(s) = f (s) + g(s) and for c ∈ F, c f defined by (c f )(s) = c f (s), is a vector space.

Exercise 1.21. Prove that the set of all polynomials with single variable x having degree at most n, over
a field F is a vector space.

Subspace A subspace of vector space is a subset of vector space that inherits all the properties of
vector addition and scalar multiplication.

Theorem 1.3. For a nonempty subset W of a vector space V over a field F, the following statements are
equivalent:

(i) W is a subspace of V .

(ii) αw1 + βw2 ∈ W for every w1 , w2 ∈ W and α, β ∈ F.

Exercise 1.22. Prove that {0}, the subset containing only a zero vector is a subspace of given vector space.

Exercise 1.23. Let A be an m × n matrix over R and

N (A)(or nullspace of A) = { x ∈ Rn : Ax = 0}.

Prove that N (A) (or nullspace of A) is a subspace of Rn .

Exercise 1.24. Let A be an m × n matrix over R and

Range(A)(Range space of A) = { Ax ∈ Rm : x ∈ Rn }.

Prove that Range(A) is a subspace of Rm .

Exercise 1.25. Given subspaces W1 and W2 of vector space V over a field F, prove that W1 ∩ W2 is a
subspace.

Exercise 1.26. Given collection of subspaces {Wα } of vector space V over a field F, the intersection of
subspaces ∩Wα is also a subspace.

Lecture Notes 23
 1.5 Linearly Independent Vectors, Basis of a Vector Space

Exercise 1.27. Give an example of subspaces W1 and W2 of vector space V over a field F, such that
W1 ∪ W2 need not be a subspace.

Exercise 1.28. Is complement of a subspace of vector space, a subspace?

Exercise 1.29. Which of the following sets are vector spaces (with the natural operations of addition
and scalar multiplication)?

(i) Vectors (a, b, c, d) such that a + 2b = c − d.

(ii) n × n matrices A such that A 2 = I.

(iii) 3 × 3 matrices A such that a 11 + a 13 = a 22 + a 31 .

Linear Combination and Span of Vectors: A linear combination of vectors v1 , v2 , . . . , vm of vector

space V with reference to scalars α1 , α2 , . . . , αm is the vector

α1 · v1 + α2 · v2 + · · · + αm · vm ∈ V .

Subspace Spanned by Vectors: Given a collection of vectors S = {v1 , v2 , . . . , vm } from a vector space
V , subspace spanned by S = {v1 , v2 , . . . , vm } is the intersection W of all subspaces of V which contain S.

Exercise 1.30. Prove that a set of all linear combinations of vectors v1 , v2 , . . . , vm from vector space V for
all possible choices of scalars α1 , α2 , . . . , αm is span {v1 , v2 , . . . , vm }.

Addition of Subspaces: Given subspaces W1 and W2 of vector space V , the sum W1 + W2 is the set
of vectors
W1 + W2 = { x + y : x ∈ W1 , y ∈ W2 }

Given subspaces W1 and W2 of vector space V , the sum of subspaces W1 + W2 is said to be direct sum
and denoted by W1 ⊕ W2 if W1 ∩ W2 = {0}.

Exercise 1.31. Given subspaces W1 and W2 of vector space V , the sum W1 + W2 is a subspace.

Exercise 1.32. Let W1 and W2 be subspaces of a vector space V such that the set-theoretic union of W1
and W2 is also a subspace. Prove that one of the spaces Wi is contained in the other.

1.5 Linearly Independent Vectors, Basis of a Vector Space

A set of vectors {v1 , v2 , . . . , vm } from vector space V is said to be linearly independent if a linear combi-
nation
α1 · v1 + α2 · v2 + · · · + αm · vm = 0

only if α1 = α2 = · · · = αm = 0. If set of vectors in a vector space is not linearly independent, then it is

called linearly dependent.
Basis: A set of linearly independent vectors that span entire vector space is called basis of the
vector space. If a basis contains only a finitely many vectors, then the vector space is said to be of finite
dimension.

Lecture Notes 24
 1.5 Linearly Independent Vectors, Basis of a Vector Space

Example 13. For the vector space R3 , the vectors v1 = (1, 0, 0), v2 = (2, 1, 0) and v3 = (0, 1, 1) in it, we have
the following:

(i) Span of vectors v1 and v2 is the plane of all points with third coordinate equals to zero i.e., (α, β, 0).
So, these vectors do not make a basis for R3 .

(ii) The set of vectors {v1 , v2 , v3 } is independent and also span the vector space R3 [note that (α, β, γ) =
(α − 2β + 2γ)v1 + (β − γ)v2 + γv3 ]. Hence the set {v1 , v2 , v3 } is a basis for R3 .

Example 14. For the vector space Fn , the vectors

e1 = (1, 0, 0, . . . , 0)

e2 = (0, 1, 0, . . . , 0)

... .........

en = (0, 0, 0, . . . , 1)

form a basis. This basis is called standard basis for Fn .

Exercise 1.33. For R3 , prove that the vectors v1 = (1, 0, 0), v2 = (2, 1, 0) and v3 = (0, 1, 1) forms a basis.

Exercise 1.34. For the vector space R3 , the vectors v1 = (1, 0, 0), v2 = (2, 1, 0) span the plane of all points
with third coordinate equals to zero i.e., (α, β, 0).

Theorem 1.4. Let V be a vector space which is spanned by a finite set of vectors {v1 , v2 , . . . , vm }. Then
any linearly independent set of vectors in V is finite and contains no more than m vectors.

Theorem 1.5. For a vector space, the cardinality of basis remains same for every choice of basis.

Dimension of a Vector Space: The dimension of a vector space is the cardinality of any of its
basis.

Exercise 1.35. Prove Theorem 1.5.

Exercise 1.36. Prove that any subset of linearly independent set of vectors is linearly independent.

Exercise 1.37. Prove that any set that contain a linearly dependent set of vectors is linearly dependent.

Exercise 1.38. A set of vectors {v1 , v2 , . . . , vm } is linearly dependent if and only if there exists a vector
among {v1 , v2 , . . . , vm } which can be written as linear combination of the remaining vectors.

Exercise 1.39. Given a set of linearly independent vectors {v1 , v2 , . . . , vm } of an n dimensional vector
space V can be extended {v1 , v2 , . . . , vm , vm+1 , . . . , vn } to form a basis for V .

Exercise 1.40. Every finite dimensional vector space has a basis.

Exercise 1.41. If S is a subspace of a finite dimensional vector space V, then every linearly independent
subset of S is finite and is a part of basis for S.

Lecture Notes 25
 1.5 Linearly Independent Vectors, Basis of a Vector Space

Exercise 1.42. If S is a proper subspace of a n dimensional vector space V, then dim(S) < dim(V ).

Exercise 1.43. If S is a subspace of a finite dimensional vector space V, then there exists a subspace T
such that S ⊕ T = V .

Exercise 1.44. Verify that each of the following sets is a vector space and find its dimension:

(i) Vectors (a, b, c, d) such that a + b = c + d.

(ii) The set of solutions (x, y, z) to the system 2x − y = 0, 2y + 3z = 0.

Exercise 1.45. If x, y, z is a basis for R3 , which of the following are also bases for R3 ?

(i) x + 2y, y + 3z, x + 2z.

(ii) x + y − 2z, x − 2y + z, −2x + y + z.

(iii) x, y, x + y + z.

Exercise 1.46. If { x1 , x2 } and { y1 , y2 } are both bases of R2 , show that at least one of the following state-
ments is true:

(i) { x1 , y2 }, { x2 , y1 } are both bases of R2 .

(ii) { x1 , y1 },{ x2 , y2 }are both bases of R2 .

n
Exercise 1.47. Consider the set of all vectors x in Rn such that
P
x i = 0. Show that the set is a vector
i =1
space and find a basis for the space.

Exercise 1.48. Determine the dimension of the vector space of all n × n matrices A such that trace A = 0.

Exercise 1.49. If W1 and W2 are finite dimensional subspaces of a vector space V , then prove that
W1 + W2 is finite dimensional and

dim(W1 + W2 ) = dim(W1 ) + dim(W2 ) − dim(W1 ∩ W2 ).

Exercise 1.50. Write v = (2, −5, 3) in R3 as a linear combination of the vectors v1 = (1, −3, 2), v2 =
(2, −4, −1), and v3 = (1, −5, 7), if possible.

Exercise 1.51. Write v = (1, −2, 5) in R3 as a linear combination of the vectors v1 = (1, 1, 1), v2 = (1, 2, 3),
and v3 = (2, 1, −1).
 
3 1
Exercise 1.52. Can the matrix E =   be written as a linear combination of the matrices m 1 =
1 −1
       
1 1 0 0 0 2 0 1
  , m2 =   , m3 =   and m 4 =  .
1 0 1 1 0 −1 1 0

Exercise 1.53. Determine whether or not u, v, and w given below are linearly dependent:

Lecture Notes 26
 1.5 Linearly Independent Vectors, Basis of a Vector Space

(i) u = (1, 1, −1); v = (2, −3, 1); and w = (8, −7, 1).

(ii) u = (1, −2, −3); v = (2, 3, −1); and w = (3, 2, 1).

(iii) u = (1, 2, −3); v = (1, −3, 2); and w = (2, −1, 5).

Exercise 1.54. Show that P1 (x) = x + 2x2 , P2 (x) = 1 + 2x + x2 and P3 (x) = 2 + x form a linearly independent
set in V3 (x), vector space of all polynomials with degree ≤ 2.

Exercise 1.55. With valid reasons, explain which of the following vectors form a basis for R3 ?

(i) (1, 1, 1); (1, −1, 5).

(ii) (1, 2, 3); (1, 0, −1); (3, −1, 0); and (2, 1, −2).

(iii) (1, 1, 1); (1, 2, 3); and (2, −1, 2).

Exercise 1.56. Which of the following sets are subspaces?

(i) W ⊆ R3 consists of all vectors of the form (a, b, 1) where a and b are any real numbers.

(ii) W ⊆ R4 contains all vectors of the form (a, b, a − b, a + 2b) for a, b ∈ R

Exercise 1.57. For which value of k will the vector u = (1, −2, k) in R3 be a linear combination of the
vectors v = (3, 0, 2) and w = (2, −1, −5)?

Exercise 1.58. Determine whether or not x = (3, 9, −4, −2) in R4 is a linear combination of u = (1, −2, 0, 3), v =
(2, 3, 0, −1), w = (2, −1, 2, 1).

Exercise 1.59. Find the conditions on a, b and c so that (a, b, c) ∈ R3 belongs to the space spanned by
u = (2, 1, 0), v = (1, −1, 2) and w = (0, 3, −4).

Exercise 1.60. Show that u = (1, 2, 5), v = (1, 3, 7), w = (1, −1, −1) do not span R3 .

Exercise 1.61. If W = S pan {(1, 2, 0); (0, 1, 0)}, then {(1, 2, 0); (0, 1, 0)} is a basis for W.

Exercise 1.62. Let U and W be subspaces of R3 defined by U = {(a, b, c) : a = b = c} and W = {(0, b, c)}.
Show that R3 = U ⊕ W

Exercise 1.63. Let W be the subspace of R4 generated by the vectors (1, −2, 5, −3); (2, 3, 1, −4) and
(3, 8, −3, −5). Find a basis and the dimension of W.

Exercise 1.64. Let W be the subspace of R4 generated by the vectors (1, 4, −1, 3), (2, 1, −3, −1) and
(0, 2, 1, −5). Find a basis and the dimension of W.

Exercise 1.65. Let { x1 , . . . , x p }{ y1 , . . . , yq } be linearly independent sets in Rn , where p < q ≤ n. Show that
there exists i ∈ {1, . . . , q} such that { x1 , . . . , x p , yi } is linearly independent.

Lecture Notes 27
 1.6 Linear Transformations

1.6 Linear Transformations

Let V and W be two vector spaces over a field F. A linear transformation from V into W is a function

T :V →W

satisfying the following properties:

(i) T(x + y) = T(x) + T(y) for all x, y ∈ V (preserving addition);

(ii) T(α x) = αT(x) for all x ∈ V and α ∈ F (preserving scalar multiplication).

A linear transformation T : V → W is one-one if T(x) = T(y) implies x = y.

A linear transformation T : V → W is onto if for every w ∈ W, there exists v ∈ V such that T(v) = w.

A linear transformation T : V → W is said to be an isomorphism if it is one-one and onto (bijective).

Example 15. A mapping T : V → V defined by T(v) = v for all v ∈ V is a linear transformation.

Example 16. T : V → W defined by T(v) = 0 for every v ∈ V is also a linear transformation.

Example 17. Consider a vector space V of all polynomials in a variable x over a field F.T : V → V defined
d (P ( x))
by T(P(x)) = dx is a linear transformation.

Exercise 1.66. The conditions (i) and (ii) above in the definition of linear transformation is equivalent
to that T(α x + β y) = αT(x) + βT(y) for all x, y ∈ V and α, β ∈ F.

Exercise 1.67. For a linear transformation T : V → W, prove that T(0) = 0.

Exercise 1.68. For an m × n matrix A , prove that the mapping T : Fn → Fm defined by T(x) = Ax for
every x ∈ Fn , is a linear mapping.

Exercise 1.69. Which of the following functions T from R2 into R2 are linear transformations?

(i) T(x1 , x2 ) = (x2 , x1 )

(ii) T(x1 , x2 ) = (1 + x1 , x2 )

(iii) T(x1 , x2 ) = (x12 , x2 )

(iv) T(x1 , x2 ) = (x1 − x2 , 0).

Theorem 1.6. Consider a linear transformation T : V → W and w1 , w2 , . . . , wk are set of linearly inde-
pendent vectors in W in the range of T. If v1 , v2 , . . . , vk are vectors from V such that T(v1 ) = w1 , T(v2 ) =
w2 , . . . , T(vk ) = wk , then v1 , v2 , . . . , vk are linearly independent.

Theorem 1.7. If the linear transformation T : V → W is one-one, then linear independence of vectors
v1 , v2 , . . . , vk in V imply the linear independence of T(v1 ), T(v2 ), . . . , T(vk ) in W.

Lecture Notes 28
 1.7 Rank-Nullity Theorem

Theorem 1.8. Let V be an n-dimensional vector space over a field F and W be a vector space over the
same scalar field F. If v1 , v2 , . . . , vn is a basis for V and w1 , w2 , . . . , wn are any vectors from W, then there
exists exactly one linear transformation T : V → W such that T(v i ) = w i , 1 ≤ i ≤ n.

Range Space, Null Space and Rank of a Transformation: For a linear transformation T : V →
W, where V ,W are vector spaces over a field F, The ‘null space’ of linear transformation T is the set
K er(T) = { x ∈ V : T(x) = 0}. The ‘range of transformation T’ (Range(T)) is the set {T(x) ∈ W : x ∈ V }.

Example 18. In the first example given in section (1.6) the subspace {0} of V is the null space, in the
second entire vector space V itself the null space and in the third example the set of ‘constant polynomials’
form the null space.

Exercise 1.70. For a linear transformation T : V → W, prove that

(i) Ker(T) is a subspace of V .

(ii) Range(T) is a subspace of W.

(iii) T is one-one if and only if K er(T) = {0}.

(iv) If T is one-one, then dim(Range(T)) = dim(V ).

Exercise 1.71. Given linear transformation T : V → W and scalar α, mapping αT : V → W defined by

(αT)(v) = αT(v) f or all v ∈ V

is a linear transformation.

Exercise 1.72. Given linear transformations T : V → W and S : V → W, the mapping T + S : V → W

defined by
(T + S)(v) = T(v) + S(v) f or all v ∈ V

is a linear transformation.

Exercise 1.73. Given linear transformations T : X → Y and S : Y → Z, the composition of mapping

S ◦ T : X → Z defined by
S ◦ T(x) = S(T(x)) f or all x ∈ X

is a linear transformation.

1.7 Rank-Nullity Theorem

Rank: Given a linear transformation T : V → W, the rank of linear transformation is defined by
dim(T(V )). In other words, the rank of linear transformation is the dimension of range space.
Nullity: The dimension of null space is known as nullity of the transformation.

Lecture Notes 29
 1.7 Rank-Nullity Theorem

Theorem 1.9 (Rank-Nullity Theorem). Given finite dimensional vector spaces V and W, and a linear
transformation T from V into W, we have

Rank(T) = dim(V ) − N ull it y(T)

Proof. Let N be the null space of T : V → W and is of dimension k. Consider a basis { u 1 , u 2 , . . . , u k } of

N . Since N is a subspace of finite dimensional space V , we can extend the basis { u 1 , u 2 , . . . , u k } of N to

{ u 1 , u 2 , . . . , u k , v1 , v2 , . . . , v l }

form a basis for V , where k + l = n the dimension of V . Now we shall prove that

{T(v1 ), T(v2 ), . . . , T(vl )}

form a basis for the range of T.

Span of {T(v1 ), T(v2 ), . . . , T(vl )} is range of T: Any vector in the range of T is of the form T(v) for
some v of V . Since { u 1 , u 2 , . . . , u k , v1 , v2 , . . . , vl } is a basis for V , we have

v = α1 v1 + α2 v2 + · · · + αl vl + β1 u 1 + β2 u 2 + · · · + βk u k

for some scalars. Therefore

T(v) = α1 T(v1 ) + α2 T(v2 ) + · · · + αl T(vl ) + β1 T(u 1 ) + β2 T(u 2 ) + · · · + βk T(u k )

Since u 1 , u 2 , . . . , u k are from the null space N , β1 T(u 1 ) + β2 T(u 2 ) + · · · + βk T(u k ) = 0 and hence

T(v) = α1 T(v1 ) + α2 T(v2 ) + · · · + αl T(vl )

a linear combination of {T(v1 ), T(v2 ), . . . , T(vl )}. This proves that Range of T, T(V ) ⊆ Span of {T(v1 ), T(v2 ), . . . , T(vl )}
The reverse inclusion is trivial.
The set of vectors {T(v1 ), T(v2 ), . . . , T(vl )} in W are linearly independent: Consider a linear combina-
tion α1 T(v1 ) + α2 T(v2 ) + · · · + αl T(vl ) = 0. By linear property, we have that

T(α1 v1 + α2 v2 + · · · + αl vl ) = 0

which implies α1 v1 + α2 v2 + · · · + αl vl is in the null space N and a linear combination of u 1 , u 2 , . . . , u k .

Now the linear independence of { u 1 , u 2 , . . . , u k , v1 , v2 , . . . , vl } implies that the scalars α′i s are zero. This
proves the linear independence of {T(v1 ), T(v2 ), . . . , T(vl )}. Hence {T(v1 ), T(v2 ), . . . , T(vl )} is a basis for
Range of T and the dimension is l. In other words, Rank(T) = l.
Since k + l = n, we get l = n − k or

Rank(T) = dim(V ) − N ull it y(T)

Consider a linear transformation T : Fn → Fm and standard bases {v1 , v2 , . . . , vn } and {w1 , w2 , . . . , wm }

of Fn and Fm respectively. Without loss of generality, let v i of Fn be a column matrix of size n, with unit
(1’s) in the i-th position and the zero’s in the rest of the positions. Similarly, w j of Fm is also represented

Lecture Notes 30
 1.7 Rank-Nullity Theorem

by a column matrix of size m. From the definition of basis, an arbitrary vector v from Fn can be written
uniquely as α1 v1 + α2
v2 +· · · + αn vn , a linear combination of basis vectors and we have column matrix
α
 1
 α2 
 
representation as v = 
 .. . If we have,

 . 
 
αn
     
α11 α12 α1n
     
 α21   α22   α2n 
     
T(v1 ) =  .  , T(v2 ) =  .  , · · · , T(vn ) =  . 
    
 ..   ..   .. 

     
α m1 α m2 αmn

by linear property of transformation, it is easily verified that

  
a a 12 ··· a 1n α1
 11  
 a 21 a 22 ··· a 2n   α2 
  
T(v) = T(α1 v1 + α2 v2 + · · · + αn vn ) = 
 .. .. .. ..   .. 
 
 . . . . 
 . 
 

a m1 a m2 ··· a mn αn

Hence
 the matrix representation
 of linear transformation T by
a a 12 · · · a 1n
 11 
 a 21 a 22 · · · a 2n 
 
 .. .. .. ..  . Conversely, from the distributive properties of matrix multiplication over
 
 . . . . 
 
a m1 a m2 · · · a mn
the addition, and scalar multiplication, given any m × n matrix, it satisfies linear property and is a
linear transformation on Fn into Fm .

Exercise 1.74. Find the range, rank, null space, and nullity for the zero transformation and the identity
transformation on a finite-dimensional space V .

Exercise 1.75. Is there a linear transformation T from R3 into R2 such that T(1, −1, 1) = (1, 0) and
T(1, 1, 1) = (0, 1)?

Exercise 1.76. Prove that every n-dimensional vector space V over a field F is isomorphic to Fn .

Result : Given vector spaces V and W of dimension n and m respectively over a scalar
field F, Hom(V ,W) - the set of all homomorphism/transformations from V into W is a vector
space over F and isomorphic to Fm×n .

Exercise 1.77. Describe explicitly a linear transformation from R3 into R3 which has its range as the
subspace spanned by (1, 0, −1) and (1, 2, 2).

Exercise 1.78. Prove that the linear transformation T : V → W is one-one if and only if there exists a
linear transformation S : W → V such that S ◦ T : V → V is an identity transformation on V .

Lecture Notes 31
 1.8 Rank

Exercise 1.79. Prove that the linear transformation T : V → W is onto if and only if there exists a linear
transformation S : W → V such that T ◦ S : W → W is also an identity transformation on W.

Exercise 1.80. An m × n matrix A has left inverse if and only if A : Fn → Fm is one-one.

Exercise 1.81. An m × n matrix A has right inverse if and only if A : Fn → Fm is onto.

Exercise 1.82. An n × n matrix A has inverse if and only if A : Fn → Fn is an isomorphism.

Exercise 1.83. If the linear transformation T : V → W is one-one and onto (isomorphism), prove that
there exists a linear transformation S : W → V such that S ◦ T : V → V is an identity transformation on
V . Further, T ◦ S : W → W is also an identity transformation on W.

Exercise 1.84. If T is a linear map T : R2 → R such that T(1, 1) = 3 and T(0, 1) = −2, find

(i) T(8, 2)

(ii) T(−4, 6)

(iii) T −1 (6)

Exercise 1.85. If T is a linear map T : R2 → R such that T(3, 1) = (2, −4) and T(1, 1) = (0, 2),find

(i) T(7, 4)

(ii) T −1 (5, −3)

1.8 Rank
Column space of a matrix: Given m × n matrix A, each column of A is a vector from Rm and the
subspace spanned by those columns is known as ‘column space’ of matrix A (denoted by C (A)).
The dimension of column space of A is known as ‘column rank’ of matrix A.

Row space of a matrix: Given m × n matrix A, each row of A is a vector from Rn and the subspace
spanned by those rows is known as ‘row space’ of matrix A.
The dimension of row space of A is known as ‘row rank’ of matrix A.
Note: For any matrix A, we have

Row Rank(A) = Column Rank(A)

Definition 1.1 (Rank). Rank of an m × n matrix A is the Column Rank(A) which is same as the Row
Rank(A).

Theorem 1.10. Let A, B be matrices such that AB is defined. Then

rank(AB) ≤ min{rank(A), rank(B)}

Lecture Notes 32
 1.8 Rank

Proof. A vector in C (AB) is of the form ABx for some vector x, and therefore it belongs to C (A). There-
fore C (AB) ⊆ C (A) and hence rank(AB) ≤ rank A. Similarly, we observe that R (AB) ⊆ R (B) and there-
fore rank(AB) ≤ rank B.

Theorem 1.11. Let A be an m × n matrix of rank r, r ̸= 0. There exist matrices B, C of order m × r, r × n

respectively such that rank B = rank C = r and A = BC. This decomposition of matrix A is called
rank factorization of A

Proof. Consider a basis for the column space of A, say b 1 , b 2 , . . . , b r . Construct an m × r matrix B =
³ ´
b 1 · · · b r . Since each column of A is a linear combination of the columns of B, there exists an r × n
matrix C such that A = BC From the definition of B, it is trivial that rank B = r. Since r = rank A ≤
rank C and C is of size r × n, we obtain rank C = r.

Exercise 1.86. Let A be an m × n matrix. Then N (A) = (C (A T ))⊥ .

Exercise 1.87. Let A be an n × n matrix. Then the following conditions are equivalent.

(i) A is nonsingular, i.e., rankA = n.

(ii) For any b ∈ Rn , Ax = b has unique solution.

(iii) There exists unique matrix B such that AB = BA = I.

Theorem 1.12. Let A, B be m × n matrices. Then rank (A + B) ≤ rank A + rank B.

 
³ ´ Y
Proof. Let A = X Y , B = UV be rank factorizations of A, B. Then A + B = X Y + UV = X U  . So,
V
³ ´ ³ ´
rank (A + B) ≤ rank X U . Clearly, dim(C X U ) ≤ dim(C (X )) + dim(C (U)) = rank A + rank B. This
proves the theorem.

Exercise 1.88. Let A be an m × n matrix and M and N are the invertible matrices of size m × m and
n × n, respectively. Then prove that

(i) rank(M A) = rank A

(ii) rank(AN) = rank A.

Theorem 1.13. Given an m × n matrix  A of rank r, there exists invertible matrices M, N of order m × m,
Ir 0
n × n respectively such that M AN =  .
0 0

Example 19. Obtain the canonical form of the following matrix and give two different rank factoriza-
tions:  
3 6 6
A= .
1 2 2

Lecture Notes 33
 1.8 Rank

Solution. The canonical form of A is given by

 
Ir 0
A=P Q
0 0

where P and Q are invertible matrices and r is the rank of matrix A.

Consider the augmented matrix [I : A] to find the invertible matrix P. Now
     
h i 1 0 : 3 6 6 ≃ 0 1 : 1 2 2 ≃ 0 1 : 1 2 2
I : A =  R1 ↔ R2   R 2 → R 2 + (−3)R 1  
0 1 : 1 2 2 1 0 : 3 6 6 1 −3 : 0 0 0
     
0 1 1 2 2 3 1
Take M =  , then M A =   and P = M −1 =  .
1 −3 0 0 0 1 0
 
MA
Consider the augmented matrix   to find the invertible matrix Q. Now
I3
     
1 2 2 1 0 2 1 0 0
     
  0 0 0 0 0 0 0 0 0
MA
     
  ≃   ≃  
  = 1 0 0 C 2 → C 2 + (−2)C 1 

1 −2 0 C 3 → C 3 + (−2)C 1 1 −2 −2
 
I3
 
     
0 1 0 0 1 0 0 1 0
 
   
0 0 1 0 0 1 0 0 1
   
1 −2 −2   1 2 2
1 0 0
 and Q = N −1 = 0
   
Take N = 
0 1 0 , then M AN =
 1 0 .
0 0 0

0 0 1 0 0 1
 
       1 2 2
1 0 0 I 0 3 1 1 0 0 
Therefore A = M −1   N −1 = P  1

Q =    0 1 0.
0 0 0 0 0 1 0 0 0 0
 
0 0 1
 
3 h i
A rank factorization of A is A =   1 2 2 . Now
1
   
3 h i 3 h i h i−1 h i
A =   I1 1 2 2 =   2 2 1 2 2
1 1
 
3 h ih ih i
=   2 2−1 1 2 2
1
 
6 h i
=   12 1 1 ,
2

is another rank factorization.

Exercise 1.89. Obtain the canonical form of the following matrices and give two different rank factor-
izations:        
2 1 −2 1 1 −1 1 1 1 −1
 ,  ,  ,  .
1 0 −1 −1 1 1 2 2 1 1

Lecture Notes 34
 1.9 Inner Product Space

Exercise 1.90. Let A be an n × n matrix of rank r. Then there exists an n × n matrix Z of rank n − r such
that A + Z is nonsingular.

Theorem 1.14 (Frobenius Inequality). Let A, B be n × n matrices. Then

rank(AB) ≥ rank A + rank B − n.

   
I r 0 0 0
Proof. Let rank of A be r. If A = M −1   N −1 , then for Z = M −1   N −1 we get A + Z =
0 0 0 I n− r
−1 −1
M N is an invertible matrix. Further

rank B = rank((A + Z)B) = rank(AB + ZB) ≤ rank(AB) + rank(ZB)

≤ rank(AB) + rank(Z) = rank(AB) + n − r = rank(AB) + n − rank A.

This proves the Frobenius inequality.

1.9 Inner Product Space

Definition 1.2 (Inner Product). Given a vector space V over the real scalar field, a function which
assigns a real number 〈v, w〉 to every pair of vectors v, w in V is said to be an inner product if it satisfies
the following condition:

(i) 〈v, w〉 = 〈w, v〉 for all v, w ∈ V (Symmetric Property)

(ii) 〈v, v〉 ≥ 0 for all v ∈ V and equality hold only for v = 0 (Positive Definite Property)

(iii) 〈αv, w〉 = α〈v, w〉 for all v, w ∈ V and α ∈ R

(iv) 〈 u + v, w〉=〈 u, w〉 + 〈v, w〉 for all u, v, w ∈ V .

If we consider vector space over complex scalar field, then the property (i) is replaced by (i)′ 〈v, w〉 =
〈w, v〉 for all v, w ∈ V (Hermitian Property). (iii) and (iv) given above are linear property of inner
product.
An inner product space is a vectorspace together with an inner product.
n
Example 20. In Rn , 〈 x, y〉 = xT y =
P
x i yi is an inner product.
i =1

For a vector x in an inner product space, the positive square root of the inner product 〈 x, x〉 is called
the norm of x, denoted by || x||. Vectors x, y are said to be orthogonal or perpendicular if 〈 x, y〉 = 0, in
which case we write x ⊥ y.
Note: If v1 , v2 , . . . , vm are pairwise orthogonal non-zero vectors then they are linearly independent.

Definition 1.3 (Orthonormal Basis). A set of vectors v1 , v2 , . . . , vm is said to be an orthonormal basis for
the vector space V if the set is a basis for V and furthermore

 0 i f i ̸= j
〈v i , v j 〉 = (1.14)
 1 if i = j

Lecture Notes 35
 1.9 Inner Product Space

Lemma 1.1. Given a set of vectors { x1 , x2 , . . . , xm }, there exists an orthonormal basis for the span of
{ x1 , x2 , . . . , xm }.

Proof. Without loss of generality, let x1 ̸= 0. Define y1 = x1 and z1 = || y11 || y1 . For k ≥ 2, define yk =
kP
−1
xk − 〈 xk , z i 〉 z i . Put z k = 0 if yk = 0, otherwise write z k = || y1k || yk . From the construction, it is clear that
i =1
z1 , z2 , . . . , z k are in the linear span of x1 , x2 , . . . , xk and vice versa. By induction, it is easily proved that
yk (similarly z k ) is orthogonal to z i for all i < k.
For k = 2,
〈 y2 , z1 〉 = 〈(x2 − 〈 x2 , z1 〉 z1 ), z1 〉 = 〈 x2 , z1 〉 − 〈 x2 , z1 〉〈 z1 , z1 〉 = 0.

By induction if 〈 z i , z j 〉 = 0 for all i < j < k, then for p < k we have

kX
−1
〈 yk , z p 〉 = 〈(xk − 〈 xk , z i 〉 z i ), z p 〉 = 〈 xk , z p 〉 − 〈 xk , z p 〉〈 z p , z p 〉 = 0.
i =1

So, nonzero z′k s together form an orthonormal basis for span of { x1 , x2 , · · · , xm }.

Gram-Schmidt’s Process. Given a set of vectors { x1 , x2 , . . . , xm }, Gram-Schmidt’s process explains

a method of finding an orthonormal basis for span of { x1 , x2 , . . . , xm }.

Step 1. Set k = 1. Define y1 = x1 and go to Step 3.

Step 2. If k > m, go to step 6. Otherwise go to Step 4.

1
Step 3. Construct z k = 0 if yk = 0, otherwise write z k = || yk || yk . Proceed to Step 5.

kP
−1
Step 4. Define yk = xk − 〈 xk , z i 〉 z i and go to Step 3.
i =1

Step 5. Set k ≡ k + 1 and go to Step 2.

Step 6. Obtain orthonormal basis for span of { x1 , x2 , . . . , xm } by listing nonzero vectors from z1 , z2 , . . . , z m
and STOP.

Example 21. The following vectors form a basis for R3 . Use the Gram-Schmidt procedure to convert it
into an orthonormal basis, x = (2, 3, −1), y = (3, 1, 0), z = (4, −1, 2).

Solution. Given that x = (2, 3, −1), y = (3, 1, 0) and z = (4, −1, 2).
Let y1 = x.

p 1
Let y1 = (2, 3, −1), then || y1 || = 14, ∴ z1 = p (2, 3, −1)
14
Now,

1 1 1
y2 = (3, 1, 0) − 〈(3, 1, 0), p (2, 3, −1)〉 p (2, 3, −1) = (24, −13, 9)
14 14 14
p
826 p 1 (24, −13, 9).
Then || y2 || = 14 , therefore z2 =
826

Lecture Notes 36
 1.9 Inner Product Space

Now,

1 1
y3 = (4, −1, 2) − 〈(4, −1, 2), p (2, 3, −1)〉 p (2, 3, −1)−
14 14
1 1
〈(4, −1, 2), p (24, −13, 9)〉 p (24, −13, 9)
826 826
1
= (−7, 21, 49)
59
Then || y3 || = p7 , p1 (−1, 3, 7).
therefore z3 =
59 59
Thus { p1 (2, 3, −1), p 1 (24, −13, 9), p1 (−1, 3, 7)} is an orthonormal basis for the basis of R3
14 826 59
Orthogonal Space. Let W be a subset of an inner product space V . We define

W ⊥ = {v ∈ V : 〈v, w〉 = 0 for all w ∈ W }

Exercise 1.91. W ⊥ is a subspace of V .

Exercise 1.92. Prove that W ∩ W ⊥ = (0).

Exercise 1.93. Let A be a square matrix. Prove that the following conditions are equivalent:

(i) A = A T .

(ii) A 2 = A A T .

(iii) Trace A 2 = Trace A A T .

(iv) A 2 = A T A.

(v) Trace A 2 = Trace A T A.

Exercise 1.94. Let A be a square matrix with all row sums equal to 1. If A A T = A T A, then show that
the column sums of A are also equal to 1.

Exercise 1.95. Prove that W ⊕ W ⊥ = V . Further dim W + dim W ⊥ = dim V .

Exercise 1.96. If W1 ⊆ W2 ⊆ V are vector spaces, then (i) W2⊥ ⊆ W1⊥ . (ii) (W1⊥ )⊥ = W1 .

Exercise 1.97. Let A be an m × n matrix. Then N (A) = (C (A T ))⊥ .

Exercise 1.98. Find an orthonormal basis for the subspace spanned by

{(2, 1, 3); (1, 2, 1); (3, 3, 4)}.

Exercise 1.99. Find an orthonormal basis for the subspace spanned by

{(1, 2, −1); (0, 1, −1); (3, −7, 1)}.

Exercise 1.100. Which of the following functions define an inner product on R3 ?

(i) f (x, y) = x1 y1 + x2 y2 + x3 y3 + 1

(ii) f (x, y) = 2x1 y1 + 3x2 y2 + x3 y3 − x1 y2 − x2 y1

Lecture Notes 37
 1.9 Inner Product Space

(iii) f (x, y) = x1 y1 + 2x2 y2 + x3 y3 + 2x1 y2 + 2x2 y1

(iv) f (x, y) = x1 y1 + x2 y2

(v) f (x, y) = x13 y13 + x23 y23 + x33 y33 .

Lecture Notes 38