Lecture-1: Syllabus for MA 106 : Linear Algebra

1 We shall study the Gauss elimination method and determinants to solve

systems of linear equations.
2 We shall also discuss basics of vector spaces and inner product spaces.
3 Matrices are simplified using eigenvalues and eigenvectors. We shall learn
algorithms to diagonalize matrices of special types such as symmetric,
Hermitian, orthogonal, unitary and more generally normal matrices.
4 We shall learn to use eigenvalues to solve differential equations.
5 We will use diagonalization of matrices for identification of curves and
surfaces represented by quadratic equations.

Tutorial Dates Topics No. of hrs of lectures

22.01.2020 Matrices and Linear Equations 4
29.01.2020 Determinant and Vector Spaces 5
05.02.2020 Linear Transformations 3
12.02.2020 Inner Product Spaces 3
19.02.2020 Diagonalization of Matrices 5

1 / 61
References and course policies
Textbooks and References
(1) Serge Lang, Introduction to Linear Algebra, 2nd Ed. Springer, India.
(2) Gilbert Strang, Linear Algebra and its applications, 4th Ed. Cengage.
(3) M.K. Srinivasan and J.K. Verma, Notes on Linear Algebra, 2014.
(4) B.V. Limaye, Linear Algebra, Slides for a course at IIT Dharwad, 2019.
Attendance Policy: If attendance is less than 80 % , a DX grade will be awarded.

Two extra classes

Division Date Slot Time Venue
D1, D2 17, 24 Jan 7B 8.30 LA 001, LA 002
D3, D4 22, 29 Jan 7A 8.30 LA 001, LA 002

Evaluation scheme
Mode Time Marks
5 quizzes tutorial hours 9 marks, 3 best scores
Test 5 Feb, 8.30-9.25 11
Final exam TBA 30
Total 50
2 / 61
Notation

Notation:

1 N := {1, 2, 3, . . .}
2 Z := {. . . , −3, −2, −1, 0, 1, 2, 3, . . .}
3 R := the set of all real numbers
4 C := the set of all complex numbers
5 For n ∈ N, let us consider the Euclidean space
Rn := {(x1 , . . . , xn ) : xj ∈ R for j = 1, . . . , n}.
6 We let 0 := (0, . . . , 0). Also, for x := (x1 , . . . , xn ) and y := (y1 , . . . , yn ) in
Rn , and for α ∈ R, we define

(sum) x + y := (x1 + y1 , . . . , xn + yn ) ∈ Rn ,
(scalar multiple) α x := (α x1 , . . . , α xn ) ∈ Rn ,
(scalar product) x · y := x1 y1 + · · · + xn yn ∈ R.

3 / 61
Matrices
1 Let m, n ∈ N. An m × n matrix A with real entries is a rectangular array of
real numbers arranged in m rows and n columns, written as follows:

a11 · · · a1k · · · a1n

 
 .. .. .. 
 . . . 
· · · · · ·
 
 a
A :=  j1 a jk a  = [ajk ],
jn 
 . . .
. . .

 . . . 
am1 ··· amk ··· amn

where ajk ∈ R is called the (j, k)th entry of A for j = 1, . . . , m and

k = 1, . . . , n.

2 Let Rm×n denote the set of all m × n matrices with real entries.
3 If A := [ajk ] and B := [bjk ] are in Rm×n , then we say A = B ⇐⇒ ajk = bjk
for all j = 1, . . . , m and k = 1, . . . , n.
4 Let 0 ≤ r < m, 0 ≤ s < n. By deleting r rows and s columns from A, we
obtain an (m − r )×(n − s) submatrix of A.
4 / 61
Matrices
1 An n × n matrix, that is, an element of Rn×n , is called a square matrix of
size n.
2 A square matrix A = [ajk ] is called symmetric if ajk = akj for all j, k.
3 A square matrix A = [ajk ] is called skew-symmetric if ajk = −akj for all j, k.
4 A square matrix A = [ajk ] is called a diagonal matrix if ajk = 0 for all j 6= k.
5 A diagonal matrix A = [ajk ] is called a scalar matrix if all diagonal entries of
A are equal.
6 Two important scalar matrices are the identity matrix I in which all diagonal
elements are equal to 1, and the zero matrix O in which all diagonal
elements are equal to 0.
7 A square matrix A = [ajk ] is called upper triangular if ajk = 0 for all j > k,
and lower triangular if ajk = 0 for all j < k.
8 Note: A matrix A is upper triangular as well as lower triangular if and only if
A is a diagonal matrix.
5 / 61
Matrices: Examples
   
1 2 3 0 2 3
1 The matrix 2 4 5 is symmetric, while the matrix −2 0 5 is
3 5 6 −3 −5 0
skew-symmetric.
2 Note: Every diagonal entry of a skew-symmetric matrix is 0 since
ajj = −ajj =⇒ ajj = 0 for j = 1, . . . , n.
   
2 0 0 2 0 0
3 The matrix 0 3 0 is diagonal, while 0 2 0 is a scalar matrix.
0 0 4 0 0 2
   
2 1 −1 2 0 0
4 The matrix 0 3 1  is upper triangular, while the matrix  1 3 0
0 0 4 −1 1 4
is lower triangular.

6 / 61
Matrices: Row and Column vectors

1 A row vector a of length n is a matrix with only one row consisting of n real
numbers; it is written as
 
a = a1 · · · ak · · · an ,

where ak ∈ R for k = 1, . . . , n. Here a ∈ R1×n .

2 A column vector b of length n is a matrix with only one column consisting
of n real numbers; it is written as
 
b1
 .. 
.
 
bk  ,
b= 
 .. 
.
bn

where bk ∈ R for k = 1, . . . , n. Here b ∈ Rn×1 .

7 / 61
Operations on Matrices

1 Let m, n ∈ N, and let A := [ajk ] and B := [bjk ] be m × n matrices.

2 Then the m × n matrix A + B := [ajk + bjk ] is called the sum of A and B. If
α ∈ R, then the m × n matrix αA := [αajk ] is called the scalar multiple of
A by α.
3 These operations follow the usual rules:
i. A + B = B + A
ii. (A + B) + C = A + (B + C), which we write as A + B + C
iii. α(A + B) = αA + αB
iv. (α + β)A = αA + βA
v. α(βA) = (αβ)A, which we write as αβA

4 We write (−1)A as −A, and A + (−B) as A − B.

8 / 61
Operations on Matrices

1 The transpose of an m × n matrix A := [ajk ] is the n × m matrix AT := [bjk ]

(in which the rows and the columns of A are interchanged) where bjk = akj .
2 Clearly, (AT )T = A, (A + B)T = AT + BT and (αA)T = αAT .
3 Note:
i. A square matrix A is symmetric ⇐⇒ AT = A.
ii. A square matrix A is skew-symmetric ⇐⇒ AT = −A.

4 In particular, the preceding operations can be performed on row vectors, and

also on column vectors since they are particular types of matrices.
5 Also, note that the transpose of a row vector is a column vector, and vice
versa.

9 / 61
Operations on Matrices
 
b1
 .. 
.
  n×1
bk  ∈ R
We shall often write a column vector b :=  as
1 
 .. 
.
bn
 T
b1 · · · bk · · · bn in order to save space.
2 Let m, n ∈ N. Let α1 , . . . , αm ∈ R.
3 If a1 , . . . , am ∈ R1×n , then
α1 a1 + · · · + αm am ∈ R1×n
is called a (finite) linear combination of a1 , . . . , am .
4 Similarly, if b1 , . . . , bm ∈ Rn×1 , then
α1 b1 + · · · + αm bm ∈ Rn×1
is a (finite) linear combination of b1 , . . . , bm .
10 / 61
Operations on Matrices
1 Now, for k = 1, . . . , n, consider the column vector
 T
ek := 0 · · · 1 · · · 0 ∈ Rn×1 , where the kth entry is 1 and all other
entries are 0.
 T
2 If b = b1 · · · bk · · · bn is any column vector of length n, then it
follows that b = b1 e1 + · · · + bk ek + · · · + bn en , which is a linear
combination of e1 , . . . , en .
3 The vectors e1 , . . . , en are known as the basic column vectors in Rn×1 .
4 Let A := [ajk ] ∈ Rm×n . Then
ajn ∈ R1×n is called the jth row vector of A for
 
i. aj := aj1 ···
 
a1
 .. 
j = 1, . . . , m, and we write A =  .  and
am
 T
ii. ck := a1k · · · a mk is called the kth column vector of A for k = 1, . . . , n,
and we write A = c1 · · · cn .

11 / 61
Operations on Matrices: Examples
   
2 1 −1 1 0 2
Let A := and B := .
0 3 1 −1 4 1

   
3 1 1 10 5 −5
1 Then A + B := and 5A = .
−1 7 2 0 15 5
   
2 The row vectors of A are 2 1 −1 and 0 3 1 .

     
2 1 −1
3 The column vectors of A are and and .
0 3 1

 
2 0
4 Also, AT =  1 3.
−1 1

12 / 61
Matrix Multiplication
1 Before we discuss when and how a product AB of A and B may be defined,
we define the product of a row vector a and a column vector b of the same
length.
2 Since every matrix can be written in terms of row vectors as well as in terms
of column vectors, we shall then consider the product AB.
   T
3 Let n ∈ N, a := a1 · · · an ∈ R1×n and b := b1 · · · bn ∈ Rn×1 .
4 Define the product of a row vector a and a column vector b as follows:
 
b1
  . 
ab = a1 · · · an  ..  := a1 b1 + · · · + an bn ∈ R.
bn

a1
Let m ∈ N and A ∈ Rm×n . Then A =  ... , where a1 , . . . , am ∈ R1×n .
5
 

am
13 / 61
Matrix Multiplication
1 Recalling that b ∈ Rn×1 , we define
   
a1 a1 b
Ab =  ... b :=  ...  ∈ Rm×1 .
   

am am b

2 Finally, let p ∈ N and B ∈ Rn×p . Then B = [b1 · · · bp ], where

b1 , . . . , bp ∈ Rn×1 .
3 Noting that Ab1 , . . . , Abp belong to Rm×1, we define

AB = A[b1 · · · bp ] := [Ab1 · · · Abp ] ∈ Rm×p .

4 Thus    
a1 a1 b1 ··· a1 bp
 ..   .. ..  ∈ Rm×p .
AB =  .  [b1 · · · bp ] =  . . 
am am b1 ··· am bp
14 / 61
Matrix Multiplication
1 So if m, n, p ∈ N, A := [ajk ] ∈ Rm×n and B := [bjk ] ∈ Rn×p , then
AB ∈ Rm×p , and for j = 1, . . . , m; k = 1 . . . , p,
n
X
AB = [cjk ], where cjk := aj bk = aj` b`k .
`=1

2 Note that the (j, k)th entry of AB is a product of the jth row vector of A
with the kth column vector of B as shown below:
 
  b1k
 .. 



 . 

aj1 · · · aj` · · · ajn  
  b`k .
 
   .. 
 . 
bnk
3 Clearly, the product AB is defined only when the number of columns of A is
equal to the number of rows of B.
4 Note: AI = A, IA = A, AO = O and OA = O whenever these products are
defined.
15 / 61
Matrix Multiplication: Examples
 
  1 6 0 2
2 1 −1
1 Let A := , B := 2 −1 1 −2 .
0 3 1 2×3 2 0 −1 1 3×4
 
2 11 2 1
Then AB = .
8 −3 2 −5 2×4

2 Both products AB and BA are defined ⇐⇒ the number of columns of A is

equal to the number of rows of B and the number columns of B is equal to
the number of rows of A, that is, when A ∈ Rm×n and B ∈ Rn×m .
   
1 0 0 1
3 In general, AB 6= BA. For example, if A := and B := , then
0 0 0 0
   
0 1 0 0
AB := , while BA := .
0 0 0 0
4 Note that BA = O, while AB = B 6= O. Since A 6= I, we see that the
so-called cancellation law does not hold.

16 / 61
Matrix Multiplication: Important Remark

1 Let A := [ajk ] ∈ Rm×n , and let e1 , . . . 

, en be
 the basic column vectors in
a1k
 .. 
 . 
n×1
 
R . Then for k = 1, . . . , n, A ek =   ajk , which is the kth column of A.

 .. 
 . 
amk
This follows from our definition of matrix multiplication.
2 It follows that if A, B ∈ Rm×n , then
A = B if and only if Aek = Bek for each k = 1, . . . , n.

17 / 61
Properties of Matrix Multiplication

1 Consider matrices A, B, C and α ∈ R.

2 Then it is easy to see that (A + B)C = AC + BC, C(A + B) = CA + CB
and (αA)B = αAB = A(αB), if sums & products are well-defined.
3 Matrix multiplication also satisfies the associative law:

Proposition
Let m, n, p, q ∈ N. If A ∈ Rm×n , B ∈ Rn×p and C ∈ Rp×q , then A(BC) = (AB)C
(which we shall write as ABC).

Proof. Let A := [ajk ], B := [bjk ] and C := [cjk ]. Also, let (AB)C := [αjk ] and
A(BC) := [βjk ].
   
Pp Pn Pn Pp
Then αjk = i=1 `=1 aj` b`i cik = `=1 aj` i=1 b`i cik = βjk .

18 / 61
Properties of Matrix Multiplication continues...

Proposition
Let m, n, p ∈ N. If A ∈ Rm×n and B ∈ Rn×p , then (AB)T = BT AT .

Proof. Let A := [ajk ], B := [bjk ] and AB := [cjk ].

0 0
Also, let AT := [ajk ], BT := [bjk ] and (AB)T := [cjk0 ]. Then
n
X n
X
cjk = aj` b`k and so cjk0 = ckj = ak` b`j
`=1 `=1

for j = 1, . . . , m; k = 1, . . . , p. Suppose BT AT := [djk ]. Then

n
X n
X n
X
0 0
djk = bj` a`k = b`j ak` = ak` b`j = cjk0
`=1 `=1 `=1

for j = 1, . . . , m; k = 1, . . . , p. Hence the result.

19 / 61
Linear System
1 Let m, n ∈ N. A linear system of m equations in n unknowns x1 , ..., xn is
given by
a11 x1 + a12 x2 + · · · + a1n xn = b1 (1)
a21 x1 + a22 x2 + · · · + a2n xn = b2 (2)
.. .. .. .. ..
. . . . .
am1 x1 + am2 x2 + · · · + amn xn = bm (m)
where ajk ∈ R for j = 1, . . . , m; k = 1, . . . , n and also bj ∈ R for j = 1, . . . , m
are given.
 T
2 Let A := [ajk ] ∈ Rm×n , x := x1 · · · xn ∈ Rn×1 and
 T
b := b1 · · · bm ∈ Rm×1 .
3 Using matrix multiplication, we write the linear system as
Ax = b.

4 The m × n matrix A is known as the coefficient matrix of the linear system.

20 / 61
Linear System
1 A column vector x0 ∈ Rn×1 is called a solution of the above linear system if
it satisfies Ax0 = b.
2 Case (i) Homogeneous Linear System: b := 0, that is, b1 = · · · = bm = 0.
3 A homogeneous linear system always has a solution, namely the zero solution
 T
0 := 0 · · · 0 since A0 = 0.
4 Also, if r ∈ N and x1 , . . . , xr are solutions of such a system, then so is their
linear combination α1 x1 + · · · + αr xr , since
A(α1 x1 + · · · + αr xr ) = α1 Ax1 + · · · + αr Axr = 0 + · · · + 0 = 0.
5 Case (ii) General Linear System: b ∈ Rm×1 is arbitrary.
6 A nonhomogeneous linear system, that is, where b 6= 0, may not have a
solution, may have only one solution or may have (infinitely) many solutions.
7 Examples:
i. The linear system x1 + x2 = 1, 2x1 + 2x2 = 1 does not have a solution.
ii. The linear system x1 + x2 = 1, x1 − x2 = 0 has a unique solution, namely
x1 = 1/2 = x2 .
iii. The linear system x1 + x2 = 1, 2x1 + 2x2 = 2 has (infinitely) many solutions,
namely x1 = α, x2 = 1 − α, α ∈ R.
21 / 61
Linear System: An Important Note

1 Let S denote the set of all solutions of a homogeneous linear system Ax = 0.

If x0 is a particular solution of the general system Ax = b, then the set of all

solutions of the general system Ax = b is given by {x0 + s : s ∈ S} since

s ∈ S =⇒ A(x0 + s) = Ax0 + As = b + 0 = b.

Also, if y0 is another solution of the general system Ax = b, i.e., if Ay0 = b,

then A(y0 − x0 ) = Ay0 − Ax0 = b − b = 0 and hence (y0 − x0 ) ∈ S and we
can write y0 = x0 + s where s = (y0 − x0 ) ∈ S.
2 We shall, therefore, address the problem of finding all solutions of a
homogeneous linear system Ax = 0, and one particular solution of the
corresponding general system Ax = b.

22 / 61
Linear System: A Special Case

1 Suppose the coefficient matrix A is upper triangular and its diagonal

elements are nonzero.
2 Then the linear system is

a11 x1 + a12 x2 + · · · + · · · + · · · + · · · + a1n xn = b1 (1)

a22 x2 + a23 x3 + · · · + · · · + · · · + a2n xn = b2 (2)
.. .. .. ..
. . . .
a(n−1)(n−1) xn−1 + a(n−1)n xn = bn−1 (n − 1)
ann xn = bn (n)

which can be solved by back substitution as follows.

23 / 61
Linear System: A Special Case

xn = bn /ann
xn−1 = (bn−1 − a(n−1)n xn )/a(n−1)(n−1), where xn = bn /ann
.. .. ..
. . .
x2 = (b2 − a2n xn − · · · − a23 x3 )/a22 , where xn = · · ·, x3 = · · ·
x1 = (b1 − a1n xn − · · · − · · · − a12 x2 )/a11 , where xn = · · ·, x2 = · · ·

1 Here the homogeneous system Ax = 0 has only the zero solution and the
general system Ax = b has a unique solution.
2 Taking a cue from this special case of an upper triangular matrix, we shall
attempt to transform any m × n matrix to an upper triangular form.

24 / 61
Linear System continues...
1 In this process, we successively attempt to eliminate the unknown x1 from
the equations (m), . . . , (2), the unknown x2 from the equations (m), . . . , (3),
and so on.
2 Example: Consider the linear system
x1 − x2 + x3 = 0
−x1 + x2 − x3 = 0
10x2 + 25x3 = 90
20x1 + 10x2 = 80.

3 Eliminating x1 from the 4th, 3rd and 2nd equations,

x1 − x2 + x3 = 0
0 = 0
10x2 + 25x3 = 90
30x2 − 20x3 = 80.

25 / 61
Linear System: Example continues...
1 Interchanging the 2nd and the 3rd equations,
x1 − x2 + x3 = 0
10x2 + 25x3 = 90
0 = 0
30x2 − 20x3 = 80.

2 Eliminating x2 from the 4th equation, and then interchanging the 3rd and the
4th equations,
x1 − x2 + x3 = 0
10x2 + 25x3 = 90
−95x3 = −190
0 = 0.

3 Now the back substitution gives x3 = 2, x2 = (90 − 25x3 )/10 = 4 and

 T
x1 = −x3 + x2 = 2, i.e., x = 2 4 2 is the solution of the linear system.
26 / 61
Linear System: Example continues...

1 The above process can be carried out without writing down the entire linear
system by considering the augmented matrix

a11 a12 · · · a1n  b1
 
 a12 a22 · · · a2n  b2 
[A|b] :=  .. .. .. .. ..  .
  
 . . . .

 . 

am1 am2 · · · amn  bm

2 This m × (n + 1) matrix completely describes the linear system.

3 In the above example,
  
1 −1 1  0

−1 1 −1  0
[A|b] = 
0  90 .
 
10 25 
20 10 0  80

27 / 61
Linear System: Example continues...
1 Subtracting 20 times the
first row from the 4th row, and addingthe first row to the
 second
 row, we obtain
1 −1 1  0

R4 −20R1 , R2 +R1 , 0 0 0  0
−−−−−−−−−−→  0 10
 .
25  90
0 30 −20  80

2 Interchanging the 2nd and the 3rd rows, we obtain

  
1 −1 1  0

R2 ←→R3  0 10 25  90
−−−−−→  0 0  0 .
 
0 
0 30 −20  80
3 Finally, subtracting 3 times the 2nd row from the 4th row and interchanging
the 3rd and the 4th rows, we arrive
 at 
1 −1 1 
 0
R4 −3R2 , R3 ←→R4 0 10 25  90 
−−−−−−−−−−−→  0 0 −95  −190 .


0 0 0  0
28 / 61
Row Echelon Form

1 The upper triangular nature of the 3 × 3 matrix on the top left enables back
substitution.
2 We shall now consider a general form of a matrix for which the method of
back substitution works.
3 Let A ∈ Rm×n , that is, A is an m × n matrix with real entries.
4 A row of A is said to be zero if all its entries are zero.
5 If a row is not zero, then its first nonzero entry (from the left) is called the
pivot.
6 Thus all entries to the left of a pivot equal 0.
7 Suppose A has r nonzero rows and m − r zero rows.
8 Then 0 ≤ r ≤ m. Clearly, r = 0 ⇐⇒ A = O.

29 / 61
Row Echelon Form
 
0 1 4
1 If A =  0 0 0, then m = n = 3 and r = 2.
5 6 7
2 The pivot in the 1st row is 1 and the pivot in the 3rd row is 5.
3 A matrix A is said to be in a row echelon form (REF) if the following
conditions are satisfied.
i. The nonzero rows of A precede the zero rows of A.
ii. If A has r nonzero rows, where r ∈ N, and the pivot in row 1 appears in the
column k1 , the pivot in row 2 appears in the column k2 , and so on the pivot in
row r appears in the column kr , then k1 < k2 < · · · < kr .

     
0 1 4 0 1 4 0 1 4
4 The matrices  0 0 0,  5 6 7, 0 5 7 are not in REF.
5 6 7 0 0 0 0 0 0

30 / 61
Row Echelon Form: Examples and Pivotal Columns
 
5 6 7
1 The matrix  0 1 4 is in a REF.
0 0 0
2 Pivotal Columns:
i. Suppose a matrix A ∈ Rm×n is in a REF. If A has exactly r nonzero rows, then
there are exactly r pivots. A column of A containing a pivot, is called a pivotal
column. Thus there are exactly r pivotal columns, and so 0 ≤ r ≤ n. (We
have already noted that 0 ≤ r ≤ m.)
ii. In a pivotal column, all entries below the pivot equal 0.
3 Here is a typical example of how back substitution works when a matrix A is
in a REF.
4 Let
0 a12 a13 a14 a15 a16
   
b1
0 0 0 a24 a25 a26  b2 
A := 
0
 and b :=   .
b3 
0 0 0 a35 a36 
0 0 0 0 0 0 b4
where a12 , a24 , a35 are nonzero. They are the pivots.
31 / 61
Row Echelon Form: Example continues...
1 Here m = 4, n = 6, r = 3, pivotal columns: 2 , 4 and 5 .
 T
2 Suppose there is c := c1 · · · c6 ∈ R6×1 such that Ac = b. Then
0c1 + · · · + 0c6 = b4 , that is, b4 must be equal to 0.
3 Next, the linear system Ax = b =⇒ a35 x5 + a36 x6 = b3 , that is,
x5 = (b3 − a36 x6 )/a35 , where we can assign an arbitrary value to the
unknown x6 .
4 Next, Ax = b =⇒ a24 x4 + a25 x5 + a26 x6 = b2 , that is,
x4 = (b2 − a25 x5 − a26 x6 )/a24 , where we back substitute the values of x5 and
x6 .
5 Finally, Ax = b =⇒ a12 x2 + a13 x3 + a14 x4 + a15 x5 + a16 x6 = b1 , that is,
x2 = (b1 − a13 x3 − a14 x4 − a15 x5 − a16 x6 )/a12 , where we can assign an
arbitrary value to the variable x3 , and back substitute the values of x4 , x5
and x6 .
6 Also, we can assign an arbitrary value to the variable x1 .
7 The variables x1 , x3 and x6 to which we can assign arbitrary values
correspond to the nonpivotal columns 1 , 3 and 6 .
32 / 61
Linear System: Important observations
1 Suppose an m×n matrix A is in a REF, and there are r nonzero rows.

Let the r pivots be in the columns k1 , . . . , kr with k1 < · · · < kr , and let the
columns `1 , . . . , `n−r be nonpivotal.

Then xk1 , . . . , xkr are called the pivotal variables and x`1 , . . . , x`n−r are
called the free variables.
2 Let b := [b1 · · · br br +1 · · · bm ]T , and consider the linear system Ax = b.
3 Important Observations
i. The linear system has a solution ⇐⇒ br +1 = · · · = bm = 0. This is known as the
consistency condition.
ii. Let the consistency condition br +1 = · · · = bm = 0 be satisfied. Then we obtain
 T
a particular solution x0 := x1 · · · xn of the linear system by letting
xk := 0 if k ∈ {`1 , . . . , `n−r }, and then by determining the pivotal
variables/coordinates xk1 , . . . , xkr of x by back substitution.
iii. We obtain n − r basic solutions of the homogeneous linear system Ax = 0 as
follows.
33 / 61
Linear System: Important observations
Fix ` ∈ {`1 , . . . , `n−r }.
 T
Define s` := x1 · · · xn by xk := 1 if k = `, while xk := 0 if
k ∈ {`1 , . . . , `n−r } but k 6= `. Then determine the pivotal
variables/coordinates xk1 , . . . , xkr of s` by back substitution in Ax = 0.

 T
iv. Let s := r1 · · · rn ∈ Rn×1 be any solution of the homogeneous system,
that is, As = 0.
Then s is a linear combination of the n − r basic solutions s`1 , . . . , s`n−r .
To see this, let y := s − r`1 s`1 − · · · − r`n−r s`n−r .
Then Ay = As − r`1 As`1 − · · · − r`n−r As`n−r = 0, and moreover, the kth
entry of y is 0 for each k ∈ {`1 , . . . , `n−r } (why?).
It then follows that y = 0 (why?), that is, s = r`1 s`1 + · · · + r`n−r s`n−r .
Thus we find that the general solution of the homogeneous system Ax = 0 is
given by s = α1 s`1 + · · · + αn−r s`n−r , where α1 , . . . , αn−r ∈ R .

34 / 61
Linear System: Important observations
v. The general solution of Ax = b is given by

x = x0 + α1 s`1 + · · · + αn−r s`n−r , where α1 , . . . , αn−r ∈ R,

provided the consistency condition is satisfied.

1 Example:
   
0 2 1 0 2 5 0
0 0 0 3 5 0 1
Let A :=  and b :=  .
0 0 0 0 1 2  2
0 0 0 0 0 0 0

As we have seen earlier, here m = 4, n = 6, r = 3, pivotal columns: 2 , 4

and 5 , and nonpivotal columns: 1 , 3 , 6 .

Since b4 = 0, the linear system Ax = b is consistent.

35 / 61
Linear System: Example continues...

For a particular solution of Ax = b, let x1 = x3 = x6 = 0.

Then x5 + 2x6 = 2 =⇒ x5 = 2,
3x4 + 5x5 + 0x6 = 1 =⇒ x4 = −3,
2x2 + x3 + 0x4 + 2x5 + 5x6 = 0 =⇒ x2 = −2.
 T
Thus x0 := 0 −2 0 −3 2 0 is a particular solution.

Basic solutions of Ax = 0:  T
x1 = 1, x3 = x6 = 0 gives s1 := 1 0 0 0 0 0 ,
 T
x3 = 1, x1 = x6 = 0 gives s3 := 0 −1/2 1 0 0 0 ,
 T
x6 = 1, x1 = x3 = 0 gives s6 := 0 −1/2 0 10/3 −2 1 .

The general solution of Ax = b is given by

x = x0 + α1 s1 + α3 s3 + α6 s6 , that is,
x1 = α1 , x2 = −2 − (α3 + α6 )/2, x3 = α3 , x4 = −3 + (10/3)α6 ,
x5 = 2(1 − α6 ), x6 = α6 , where α1 , α3 , α6 ∈ R.

36 / 61
Linear System: Conclusion

1 Suppose an m × n matrix A is in a REF, and let r be the number of nonzero

rows of A. If b ∈ Rm×1 , then the linear system Ax = b has

i. no solution if one of br +1 , . . . , bm is nonzero.

ii. a unique solution if br +1 = · · · = bm = 0 and r = n (why?).

iii. infinitely many solutions if br +1 = · · · = bm = 0 and r < n. In this case,

there are n − r free variables which give n − r degrees of freedom .

2 Considering the case b = 0 ∈ Rm×1 and recalling that r ≤ m, we obtain the

following important results.

Proposition
Let A ∈ Rm×n be in REF with r nonzero rows. Then the linear system Ax = 0
has only the zero solution if and only if r = n. In particular, if m < n, then
Ax = 0 has a nonzero solution.

37 / 61
Gauss Elimination Method (GEM)

1 We have seen how to solve the linear system Ax = b when the matrix A is in
a row echelon form (REF).
2 We now explain the Gauss Elimination Method (GEM) by which we can
transform any A ∈ Rm×n to a REF.
3 This involves the following two elementary row operations (EROs):

Type I: Interchange of two rows

Type II: Addition of a scalar multiple of a row to another row

4 We shall later consider one more elementary row operation which is the
following:

Type III: Multiplication of a row by a nonzero scalar

38 / 61
GEM: Remark

1 First we remark that if the augmented matrix [A|b] is transformed to a matrix

[A0 |b0 ] by any of the EROs, then Ax = b ⇐⇒ A0 x = b0 for x ∈ Rn×1 , that
is, the linear systems Ax = b and A0 x = b0 have the same solutions.
2 This follows by noting that an interchange of two equations does not change
the solutions, neither does an addition of an equation to another, nor does a
multiplication of an equation by a nonzero scalar, since these operations can
be undone by similar operations, namely, interchange of the equations in the
reverse order, subtraction of an equation from another, and division of an
equation by a nonzero scalar.
3 Consequently, we are justified in performing EROs on the augmented matrix
[A|b] in order to obtain all solutions of the given linear system.

39 / 61
Transformation to REF
1 Let A ∈ Rm×n , that is, let A be an m × n matrix with entries in R. If A = O,
the zero matrix, then it is already in REF.
2 Suppose A 6= O.

i. Let column k1 be the first nonzero column of A, and let some nonzero entry
p1 in this column occur in the jth row of A. Interchange row j and row 1.
Then A is transformed to
0 · · · 0 p1 ∗ · · · ∗
 
0 · · · 0 ∗ ∗ · · · ∗
A0 :=  . ,
 
 .. .
.. .
.. .
.. .. .. .. 
. . .
0 ··· 0 ∗ ∗ ··· ∗

where ∗ denotes a real number. Note: p1 becomes the chosen pivot in row 1.
(This choice may not be unique.)

ii. Since p1 6= 0, add suitable scalar multiples of row 1 of A0 to rows 2 to m of

A0 , so that all entries in column k1 below the pivot p1 are equal to 0.
40 / 61
Transformation to REF
Then A0 is transformed to
 
0 ··· 0 p1 ∗ ··· ∗
0 ··· 0 0 ∗ ··· ∗
 
A00 := 

 .. .. .. .. .. .. .
.. 
. . . . . . .
0 ··· 0 0 ∗ ··· ∗

iii. Keep row 1 of A00 intact, and repeat the above process for the remaining
(m − 1) × n submatrix of A00 to obtain

0 · · · 0 p1 ∗ ∗ ∗ ∗ ··· ∗
 
0 · · · 0 0 ··· 0 p2 ∗ ··· ∗
 
000
 
A :=  0 · · · 0 0 ··· 0 0 ∗ ··· ∗ ,
 .. .. .. .. .. .. .. .. .. .. 
. . . . . . . . . .
0 ··· 0 0 ··· 0 0 ∗ ··· ∗

where p2 6= 0 and occurs in column k2 of A000 , where k1 < k2 .

41 / 61
Transformation to REF

Note: p2 becomes the chosen pivot in row 2. (Again, this choice may not be
unique.)

iv. Keep rows 1 and 2 of A000 intact, and repeat the above process till the
remaining submatrix has no nonzero row.
1 The resulting m × n matrix is in REF with pivots p1 , . . . , pr in columns
k1 , . . . , kr , and the last m − r rows are zero rows, where 1 ≤ r ≤ m.
Transformation to REF: Notation
2 Ri ←→ Rj will denote the interchange of the ith row Ri and the jth row Rj
for 1 ≤ i, j ≤ m with i 6= j.
3 Ri + αRj will denote the addition of α times the jth row Rj to the ith row Ri
for 1 ≤ i, j ≤ m with i 6= j.
4 αRj will denote the multiplication of the jth row Rj by the nonzero scalar α
for 1 ≤ j ≤ m.

42 / 61
Transformation to REF
1 Remark: A matrix A may be transformed to different REFs by EROs.
   
1 3 1 3
2 For example, we can transform A := by EROs to as well as
  2 0 0 −6
2 0
to , both of which are REFs.
0 3
   
0 1 1
3 Example (i). Let A := and b := . Then R1 ←→ R2 gives
1 1 2
 T
x := 1 1 as the solution of the linear system Ax = b.
4 Example (ii). Consider the linear system

3x1 + 2x2 + x3 = 3
2x1 + x2 + x3 = 0
6x1 + 2x2 + 4x3 = 6.

43 / 61
Transformation to REF: Examples
1 We can check that
 
3 2 1 3
 
3 2 1 3
R2 −(2/3)R1 
 2 1 1 0  −−−−−−−→  0 −1/3 1/3 −2 

R3 −2R1
6 2 4 6 0 −2 2 0
 
3 2 1 3
R3 −6R2 
−− −−→  0 −1/3 1/3 −2 .


0 0 0 12

2 In this case m = 3 = n, r = 2 and br0 +1 = b30 = 12 6= 0. Hence the given

linear system has no solution.
3 Example (iii). Consider the linear system
3.0x1 + 2.0x2 + 2.0x3 − 5.0x4 = 8.0
0.6x1 + 1.5x2 + 1.5x3 − 5.4x4 = 2.7
1.2x1 − 0.3x2 − 0.3x3 + 2.4x4 = 2.1.

44 / 61
Transformation to REF: Examples
1 We can check that
 
3 2 2 −5 8
[A|b] =  0.6 1.5 1.5 −5.4 2.7 
1.2 −0.3 −0.3 2.4 2.1

2
 
3 2 2 −5 8
EROs  0 0
−−−−−→ 0 1.1 1.1 −4.4 1.1  =: [A |b ].
0 0 0 0 0

3 Here m = 3, n = 4, r = 2, pivotal columns: 1 , 2 , nonpivotal columns: 3 ,

4.
4 Since br0 +1 = b30 = 0, the linear system A0 x = b0 has a solution.
5 For a particular solution of A0 x = b0 , let x3 = x4 = 0. Then
1.1 x2 = 1.1 =⇒ x2 = 1, 3x1 + 2x2 = 8 =⇒ x1 = 2.

45 / 61
Transformation to REF: Examples

 T
1 Thus x0 := 2 1 0 0 is a particular solution.
2 Since r = 2 < 4 = n, the linear system has (infinitely) many solutions.
3 For basic solutions of A0 x = 00 , where 00 = 0,
 T
let x3 = 1, x4 = 0, so that s3 := 0 −1 1 0 , and x4 = 1, x3 = 0, so
 T
that s4 := −1 4 0 1 .
4 The general solution of A0 x = b0 is given by x = x0 + α3 s3 + α4 s4 ,

that is, x1 = 2 − α4 , x2 = 1 − α3 + 4α4 , x3 = α3 , x4 = α4 , where α3 , α4 are

arbitrary real numbers.
5 These are precisely the solutions of the given linear system Ax = b.

46 / 61
REF: Applications to the Linear System

Proposition
Let A ∈ Rm×n . Then the linear system Ax = 0 has only the zero solution if and
only if any REF of A has n nonzero rows. In particular, if m < n, then Ax = 0 has
a nonzero solution.

Proof: We saw that these results hold if A itself is in REF. Since every A ∈ Rm×n
can be transformed to a REF A0 by EROs, and since the solutions of the linear
system Ax = 0 and the transformed system A0 x = 00 , where 00 = 0, are the same,
the desired results hold.

Note: Suppose an m × n matrix A is transformed by EROs to different REFs A0

and A00 . Suppose A0 has r 0 nonzero rows and A00 has r 00 nonzero rows and
0 ≤ r 0 , r 00 ≤ min{m, n}.Then the above result implies that r 0 = n ⇐⇒ r 00 = n.

We shall later see that r 0 = r 00 always.

A Challenge Problem: Let A ∈ R9×4 and B ∈ R7×3 . Is there X ∈ R4×7 such that
X 6= O but AXB = O?
47 / 61
Inverse of a Square Matrix
1 We now introduce a special kind of square matrices.
2 Let A be a square matrix of size n ∈ N, that is, let A ∈ Rn×n .
3 We say that A is invertible if there is B ∈ Rn×n such that AB = I = BA,
and in this case, B is called an inverse of A.
4 Examples    
1 0 0 1
The matrix A := is invertible. To see this, let B := , and
0 2 0
1/2 
1 0
check AB = I = BA. On the other hand, the nonzero matrix A := is
0 0
not invertible.
5 If A is invertible, then it has a unique inverse.
6 In fact, if AC = I = BA, then C = IC = (BA)C = B(AC) = BI = B by the
associativity of the matrix multiplication.
7 If A is invertible, its inverse will be denoted by A−1 , and so
A−1 A = I = AA−1 . Clearly, (A−1 )−1 = A.
48 / 61
Inverse of a Square Matrix

1 If A is invertible and if one can guess its inverse, then it is easy to verify that
it is in fact the inverse of A. Here are two easy examples.

Proposition
Let A be a square matrix. Then A is invertible if and only if AT is invertible. In
this case, (AT )−1 = (A−1 )T .

2 Proof. Suppose A is invertible and B is its inverse. Then AB = I = BA, and

so BT AT = IT = AT BT . Since IT = I, we see that AT is invertible and
(AT )−1 = BT = (A−1 )T .
Next, if AT is invertible, then A = (AT )T is invertible.

49 / 61
Inverse of a Square Matrix

Proposition
Let A and B be square matrices. If A and B are invertible, then the product AB
is invertible and then (AB)−1 = B−1 A−1 .

1 Proof. Let A and B be invertible. Using the associativity of matrix

multiplication, we easily see that

(B−1 A−1 )(AB) = B−1 (A−1 A)B = B−1 B = I

and
(AB)(B−1 A−1 ) = A(BB−1 )A−1 ) = AA−1 = I.

50 / 61
Elementary Matrices
1 Definition: An elementary matrix (of a particular type) is a matrix obtained
from the identity matrix by a single elementary row operation (of that type).
2 For example,
 
1
 .. 

 . 


 0 1 

Ti,j =
 . ..  = I + eij + eji − eii − ejj

 

 1 0 

 .. 
 . 
1
is an elementary matrix of type I, where eij is the square matrix of same size
for which the (i, j)th entry is 1 and all other entries are 0.
3 Note that
Ti,j · A is the matrix produced by exchanging row i and row j of A.
−1
Ti,j = Ti,j .
51 / 61
Elementary Matrices

1 The matrix
 
1
 .. 

 . 


 1 

Ti,j (α) = 
 ..  = I + αeij

 . 

 α 1 

 .. 
 . 
1

is an elementary matrix of type II.

2 Note that
Ti,j (α) · A is the matrix produced from A by adding α times row j to row i.
Ti,j (α)−1 = Ti,j (−α).

52 / 61
Elementary Matrices
1 The matrix
1
 
 .. 
 . 
 

 1 

Ti (α) =  α  = I + (α − 1)eii , α 6= 0
 

 1 

 .. 
 . 
1
is an elementary matrix of type III.
2 Note that
Ti (α) · A is the matrix produced from A by multiplying row i by α.
Ti (α)−1 = Ti (1/α).
3 We have proved the following result:
Lemma
An elementary matrix is invertible and the inverse is also an elementary matrix (of
the same type).
53 / 61
Row Canonical Form (RCF)

1 As we have seen, a matrix A may not have a unique REF. However, a special
REF of A turns out to be unique.
2 An m × n matrix A is said to be in a row canonical form (RCF) or a
reduced row echelon form (RREF) if
i. it is in a row echelon form (REF),
ii. all pivots are equal to 1 and
iii. in each pivotal column, all entries above the pivot are (also) equal to 0.

3 For example, the matrix

 
0 1 ∗ 0 0 ∗
0 0 0 1 0 ∗
A :=  
0 0 0 0 1 ∗
0 0 0 0 0 0

is in a RCF, where ∗ denotes any real number.

54 / 61
Row Canonical Form (RCF)

1 Suppose an m × n matrix A is in RCF and has r nonzero rows. If r = n, then

it has n pivotal columns,
  that is, all its columns are pivotal, and so A = I if
I
m = n, and A =   if m > n, where I is the n × n identity matrix and O is
O
the (m − n) × n zero matrix.
2 To transform an m × n matrix to a RCF, we first transform it to a REF by
elementary row operations of type I and II.
3 Then we multiply a row containing a pivot p by 1/p (which is an elementary
row operation of type III), and then we add a suitable nonzero multiple of
this row to each preceding row.
4 Every matrix has a unique RCF (Exercise: Show it by using induction on n,
the number of columns matrix has).

55 / 61
Row Canonical Form (RCF)

1 Example
 
−2 1 3 −2 0 2 0
 
1 3 0 2 0
2 6 −5 −2 4 −3EROs  0 0 −1 −2 0 −3
0 −−−−→ ,
0 5 10 0 15   0 0 0 0 0 6
2 6 0 8 4 16 0 0 0 0 0 0

which is in REF

   
1 3 0 4 2 6 1 3 0 4 2 0
EROs  0 0 1 2 0 3−EROs
 0 0 1 2 0 0
−−−−→ −−−→ 
 0 0 0 0 0 6   0 0 0 0 0 1
0 0 0 0 0 0 0 0 0 0 0 0

which is in RCF.

56 / 61
An Important Criterion for invertible Matrices
Theorem
Let A be a square matrix. Then the following are equivalent:
(a). A can be reduced to I by a sequence of elementary row operations.
(b). A is a product of elementary matrices.
(c). A is invertible.
(d). The system Ax = 0 has only the trivial solution x = 0.

1 Proof. (a) ⇒ (b). Let E1 , . . . , Ek be elementary matrices so that

Ek . . . E1 A = I . Thus A = E1−1 . . . Ek−1 .
2 (b) ⇒ (c). Elementary matrices are invertible.
3 (c) ⇒ (d). Suppose A is invertible. Then AX = 0 =⇒ A−1 (AX ) = X = 0.
4 (d) ⇒ (a). First observe that a square matrix in RCF is either the identity
matrix or its bottom row is zero.
If A can’t be reduced to I by elementary row operations then A0 = the RCF of
A has a zero row at the bottom. Hence A0 x = 0 has atmost n − 1 nontrivial
equations which have a nontrivial solution. This contradicts (d).
57 / 61
Finding the Inverse of an invertible Matrix
Corollary
Let A ∈ Rn×n . If there is B ∈ Rn×n such that either BA = I or AB = I, then A is
invertible, and A−1 = B. Also, if AB is invertible, then A and B both are
invertible.

1 Proof. Exercise.
2 Note: The above result is a definite improvement over requiring the existence
of a matrix B satisfying both BA = I and AB = I for the invertibility of a
square matrix A.
3 Remark: Suppose an n × n square matrix A is invertible. In order to solve the
linear system Ax = b for a given b ∈ Rn×1 , we may transform the augmented
matrix [A|b] to [I | c] by EROs.
4 Now Ax = b ⇐⇒ Ix = c for x ∈ Rn×1 . Hence Ac = b.
5 Thus c is the unique solution of Ax = b.
6 This observation is the basis of an important method to find the inverse of a
square matrix.
58 / 61
Gauss-Jordan Method for Finding the Inverse of a Matrix
1 Gauss-Jordan Method for Finding the Inverse of a Matrix:
2 Let A ∈ Rn×n be an invertible matrix.
3 Consider the basic column vectors e1 , . . . , en ∈ Rn×1 .
 
4 Then e1 · · · en = I.
5 Let c1 , . . . , cn be the unique elements of R n×1 such that

Ac1 = e1 , . . . , Acn = en , and define C := c1 · · · cn .
6 Then
     
AC = A c1 ··· cn = Ac1 ··· Acn = e1 ··· en = I.

7 By the last corollary it follows that C = A−1 .

8 Hence to find A−1 , we may solve the n linear systems
Ax1 = e1 , . . . , Axn = en simultaneously by considering the n × 2n augmented
matrix
[A|e1 · · · en ] = [A | I]
and transform A to its RCF, namely to I, by EROs.
59 / 61
Finding the inverse of an invertible Matrix

1 Thus if [A | I] is transformed to [I | C], then C is the inverse of A.

2 Remark: To carry out the above process, we need not know beforehand that
the matrix A is invertible.
3 This follows by noting that A can be transformed to the identity matrix by
EROs if and only if A is invertible.
4 Hence the process itself reveals whether A is invertible or not.
5 Example: Let  
−1 1 2
A :=  3 −1 1 .
−1 3 4

6 We use EROs to transform [A | I] to [I | C], where C ∈ R3×3 .

60 / 61
Gauss-Jordan Method: Example
     
−1 1 2 
 1 0 0 −1 1 2 
 1 0 0
3 −1 1 
 0 1 0 −→  0 2 7 
 3 1 0 −→
−1 3 4  0 0 1 0 2 2  −1 0 1

  
−1 1 2 1 0 0
  

 1 −1 −2 
 −1 0 0
 0 2 7  3 1 0−→ 0 1 3.5 1.5 0.5 0 
   

0 0 1 0.8 0.2 −0.2

0 0 −5  −4 −1 1


     
1 −1 0 
 0.6 0.4 −0.4 1 0 0 
 −0.7 0.2 0.3
−→ 0 1 0 
 −1.3 −0.2 0.7  −→ 0 1 0 
 −1.3 −0.2 0.7  .
0 0 1  0.8 0.2 −0.2 0 0 1  0.8 0.2 −0.2

Thus A is invertible and

 
−7 2 3
−1 1 
A =C= −13 −2 7 .
10
8 2 −2
61 / 61