Applications of Inner Product Spaces

Dr. Juhi Kesarwani Dr. Ashish Kumar Kesarwany

Mathematics Division, School of Advanced Sciences and Languages

VIT Bhopal University, Bhopal-Indore Highway, Kothrikalan, Sehore, Madhya Pradesh, 466114, India

August 9, 2024

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 1 / 46
 Table of Contents

1 QR factorization

2 Relations of fundamental subspaces

3 Least square solutions

4 Singular value decomposition

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 2 / 46
 Orthogonal Matrices

Definition (Orthogonal Matrices)

Let A be a square matrix of size n × n then A is said to be orthogonal if

AAT = AT A = In

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 3 / 46
 Orthogonal Matrices

Definition (Orthogonal Matrices)

Let A be a square matrix of size n × n then A is said to be orthogonal if

AAT = AT A = In

Example
 
cos θ sin θ
Let A = then
− sin θ cos θ
    
cos θ sin θ cos θ − sin θ 1 0
AAT = =
− sin θ cos θ sin θ cos θ 0 1

Therefore A is an orthogonal matrix.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 3 / 46
 QR factorization

QR factorization
The QR decomposition (also called the QR factorization) of a matrix A is the
decomposition of the matrix A into an orthogonal matrix and a upper triangular
matrix, i.e., there is a unique orthogonal matrix Q and an upper triangular matrix
R with only positive numbers on its main diagonal and

A = QR

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 4 / 46
 Example
 
1 2 0
Let A = 0 1 1 then A has the QR factorization given below
1 0 1
 √ √
√1 √1 − √16 √1
 
2 2
2 3 √ 2
√1 √2   0 3
A= 0 · q0 
 
3 6
√1 − √13 √1 0 0 3
2 6 2

√ √
√1 √1 − √16 √1
  
2 2
2 3 √ 2
√1 √2  0 3
Here Q =  0  is an orthogonal matrix and R =  q0 
  
3 6
√1 − √13 √1 0 0 3
2 6 2
is an upper triangular matrix.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 5 / 46
 Existence of QR factorization

The following result talks about the existence of QR factorization of a matrix A.

Result
Let A be a square matrix of size n ×n such that its columns are linearly independent
then its QR factorization exists.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 6 / 46
 How to find the QR factorization?
There are several methods for computing QR factorization of a given matrix. One
of such method is Gram-Schmidt orthogonalization process.
Consider the Gram-Schmidt orthogonalization process with the vectors to be
considered in the process as the columns of matrix A. That is, if
A = [a1 a2 · · · an ]
then
u1 = a1
⟨a2 , u1 ⟩
u2 = a2 − u1
∥u1 ∥2
⟨a3 , u1 ⟩ ⟨a3 , u2 ⟩
u3 = a3 − 2
u1 − u2
∥u1 ∥ ∥u2 ∥2
.. .. ..
. . .
⟨an , u1 ⟩ ⟨an , u2 ⟩ ⟨an , un−1 ⟩
un = an − u1 − u2 − · · · − un−1
∥u1 ∥2 ∥u2 ∥2 ∥un−1 ∥2
Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 7 / 46
 Cont. . .
ui
Then {u1 , u2 , . . . , un } is an orthogonal set of vectors. Set vi = then
∥ui ∥
{v1 , v2 , . . . , vn } is an orthonormal set of vectors. The resulting QR factorization is
given by
 
⟨v1 , a1 ⟩ ⟨v1 , a2 ⟩ · · · ⟨v1 , an ⟩
⟨v2 , a1 ⟩ ⟨v2 , a2 ⟩ · · · ⟨v2 , an ⟩
A = [a1 a2 · · · an ] = [v1 v2 · · · vn ]  .
 
.. .. 
 .. . ··· . 
⟨vn , a1 ⟩ ⟨vn , a2 ⟩ · · · ⟨vn , an ⟩

where Q = [v1 v2 · · · vn ] is an orthogonal matrix and

 
⟨v1 , a1 ⟩ ⟨v1 , a2 ⟩ · · · ⟨v1 , an ⟩
 0 ⟨v2 , a2 ⟩ · · · ⟨v2 , an ⟩
R= .
 
.. .. 
 .. . ··· . 
0 0 · · · ⟨vn , an ⟩

is an upper triangular matrix.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 8 / 46
 Example
 
1 1 0
Find the QR factorization of the matrix A = 1 0 1.
0 1 1

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 9 / 46
 Solution
     
1 1 0
Set a1 = 1, a2 = 0 and a3 = 1. We’ll use Gram-Schmidt orthogonalization
0 1 1
process to find an orthogonal set of vectors.
 
1
u1 = a1 = 1
0
 1 
⟨a2 , u1 ⟩ 2
u2 = a2 − 2
u1 = − 12 
∥u1 ∥
1
 2
−3
⟨a3 , u1 ⟩ ⟨a3 , u2 ⟩  2 
u3 = a3 − u 1 − u2 = 3
∥u1 ∥2 ∥u2 ∥2 2
3

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 10 / 46
 Solution cont. . .

Thus,
 √1   √   1 
6 −√
u1 2
 √1  , u2 6√ u3  √1 3 
v1 = = v2 = = − 66  , v3 = = 3 
 
∥u1 ∥ 2 ∥u2 ∥ √
6
∥u3∥ √1
0 3 3

and hence  √ 
√1 6
− √13
6√
 12 6

Q = [v1 v2 v3 ] = √ − √1 
 2 √6 3 
6 √1
0 3 3

and   √2 √1 √1
 
⟨v1 , a1 ⟩ ⟨v1 , a2 ⟩ ⟨v1 , a3 ⟩
 2 √3
2 2
√1 
R= 0 ⟨v2 , a2 ⟩ ⟨v2 , a3 ⟩ =  0 6 6
.
0 0 ⟨v3 , a3 ⟩ 0 0 √2
3

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 11 / 46
 Exercise 1:  
1 2 1
Find the QR factorization of the matrix A = 0 1 −4.
0 3 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 12 / 46
 Relations of fundamental subspaces

✿ In Module-2, we discussed about four subspaces namely null space of A (N (A)),

row space of A (R(A)), column space of A (C(A)) and null space of AT
(N (AT )) of an m × n matrix A.

✿ One of the most important applications of the orthogonal projection of vectors

onto a subspace is to study the relations or structures of the four fundamental
subspaces N (A), R(A), C(A) and N (AT ) of an m × n matrix A.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 13 / 46
 Relations of fundamental subspaces

✿ In Module-2, we discussed about four subspaces namely null space of A (N (A)),

row space of A (R(A)), column space of A (C(A)) and null space of AT
(N (AT )) of an m × n matrix A.

✿ One of the most important applications of the orthogonal projection of vectors

onto a subspace is to study the relations or structures of the four fundamental
subspaces N (A), R(A), C(A) and N (AT ) of an m × n matrix A.

Result
For any m×n matrix A, the null space N (A) and the row space R(A) are orthogonal
in Rn . Similarly, the null space N (AT ) of AT and the column space C(A) = R(AT )
are orthogonal in R⋗ .

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 13 / 46
 Result
For any m × n matrix A

➀ N (A) ⊕ R(A) = Rn .

➁ N (AT ) ⊕ C(A) = Rm

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 14 / 46
 Example
   
1 0
2 −1
Given two vectors   and  
   , find all vectors in R4 that are perpendicular to
1 −1
2 1
them.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 15 / 46
 Solution
     
x 1 0
y  4
2 −1
Suppose   ∈ R that is perpendicular to   and  
     , i.e.,
z 1 −1
t 2 1
       
* x 1 + * x 0 +
y  2 y  −1
 ,  = 0 and  ,  = 0
z  1 z  −1
t 2 t 1

=⇒ x + 2y + z + 2t = 0 and −y −z +t =0
=⇒ y = t − z & x = −4t + z

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 16 / 46
 The solution set of above homogeneous system of linear equations is
  

 x 

y
  
=   : x = −4t + z & y = t − z
 

 z 

t
 
  

 −4t + z 

t − z
  
=    : x = −4t + z & y = t − z

 z  

t
 
    

 −4t z 


t  −z  
=    +   : t, z ∈ R
 

 0  z 

t 0
 
     

 −4 1 

  
1 −1 
= t  + z   : t, z ∈ R


 0   1  

1 0
 

The above set is the collection of all the vectors that are perpendicular to given
two vectors.
Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 17 / 46
 Exercise 2:
Find a basis for the orthogonal complement of the row space of A where
 
1 2 8
(1) A = 2 −1 1
3 0 6
 
0 0 1
(2) A = 0 0 1
1 1 1

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 18 / 46
 Least square solutions

Recall the following result

Result
The system Ax = b has atleast one solution if and only if b belongs to the column
space C(A) of A. In this case, such a solution is unique if and only if the null space
N (A) of A is trivial.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 19 / 46
 Least square solutions

Recall the following result

Result
The system Ax = b has atleast one solution if and only if b belongs to the column
space C(A) of A. In this case, such a solution is unique if and only if the null space
N (A) of A is trivial.

Least square solution

/ C(A) then Ax = b is inconsistent. Note that for any x ∈ Rn , Ax ∈ C(A).
If b ∈
Thus, the best we can do is to find a vector x0 ∈ Rn such that Ax0 is closest to
the given vector b ∈ Rm , i.e., ∥Ax0 − b∥ is as small as possible. Such a solution
vector x0 gives the best approximation Ax0 to b, and is called least square solution
to Ax = b.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 19 / 46
 Note
However, since we have the orthogonal decomposition Rm = C(A) ⊕ N (AT ), we
know that for any b ∈ Rm , ProjC(A) (b) = bc ∈ C(A) is the closest vector to b
among the vectors in C(A). Therefore, a least square solution x0 ∈ Rn satisfies the
following:
Ax0 = bc = ProjC(A) (b)

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 20 / 46
 Note
However, since we have the orthogonal decomposition Rm = C(A) ⊕ N (AT ), we
know that for any b ∈ Rm , ProjC(A) (b) = bc ∈ C(A) is the closest vector to b
among the vectors in C(A). Therefore, a least square solution x0 ∈ Rn satisfies the
following:
Ax0 = bc = ProjC(A) (b)

Definition(Normal Equation)
Let Ax = b be system of m linear equations in n unknowns. Then the equation
AT Ax = AT b is called normal equation.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 20 / 46
 Note
However, since we have the orthogonal decomposition Rm = C(A) ⊕ N (AT ), we
know that for any b ∈ Rm , ProjC(A) (b) = bc ∈ C(A) is the closest vector to b
among the vectors in C(A). Therefore, a least square solution x0 ∈ Rn satisfies the
following:
Ax0 = bc = ProjC(A) (b)

Definition(Normal Equation)
Let Ax = b be system of m linear equations in n unknowns. Then the equation
AT Ax = AT b is called normal equation.

Result
Let A be m × n matrix and let b ∈ Rm be any vector. Then a vector x0 ∈ Rn is
a least square solution of Ax = b if and only if x0 is the solution of the normal
equation AT Ax = AT b.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 20 / 46
 Example
Find all the least square solutions to Ax = b and then determine the orthogonal
projection bc of b into the column space C(A) of A, where
   
1 −2 1 1
 2 −3 −1 0
A= −1 1
, b= 
2 1
3 −5 0 0

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 21 / 46
 Solution
 
  1 −2 1  
1 2 −1 3  15 −24 −3
2 −3 −1 
AT A = −2

−3 1 −5 
−1 1 = −24 39 3
2
1 −1 2 0 −3 3 6
3 −5 0
 
  1  
1 2 −1 3   0
0  
AT b = −2 −3 1 −5 
1 = −1
1 −1 2 0 3
0

A least square solution of Ax = b is a solution of AT Ax = AT b (Normal Equation),

i.e.,     
15 −24 −3 x1 0
−24 39 3  x2  = −1
−3 3 6 x3 3

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 22 / 46
 The augmented matrix of above system of linear equations is
    R1
15 −24 −3 | 0 −3 3 3 | 3 R1 → −3
−24 39 3 | −1 R1 ↔ R3 −24 39 3 | R
−1 R2 → 32
−3 3 3 | 3 15 −24 −3 | 0 R3 → R33
   
1 −1 −2 | −1 1 −1 −2 | −1 R2
−8 13 1 | −1  R2 → R2 + 8R1 0 5 −15 | −25  R2 → R5
3 R3 → R3 − 5R1 3 R3 → −33
5 −8 −1 | 0 0 −3 9 | 5
   
1 −1 −2 | −1 1 −1 −2 | −1
0 1 −3 | −5  R3 → R3 − R − 2 0 1 −3 | −5 
3 3
0 1 −3 | −5 3
0 0 0 | 0
which is in row echelon form. Here rank of coefficient matrix =rank of augmented
matrix=2 < 3(No. of variables). Hence this system has infinitely many solutions.
The system of linear equations corresponding to above row echelon form is
8
x1 − x2 − 2x3 = −1 =⇒ x1 = 5x3 −
3
−5 5
x2 − 3x3 = =⇒ x2 = 3x3 −
3 3

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 23 / 46
 Thus, the solution set is
   
  5x3 − 83
 
 x1 8 5 
= x2  : x1 = 5x3 − &x2 = 3x3 − = 3x3 − 53  : x3 ∈ R
3 3 
x3 x3
 
 8      8    
 −3 5x3   −3 5 
= − 53  + 3x3  : x3 ∈ R = − 53  + x3 3 : x3 ∈ R
0 x3 0 1
   

The above set contains all the least

 8square
 solution
  to Ax =8  b. Any least square
−3 5 5x3 − 3
solution of Ax = b will be x0 = − 53  + x3 3 = 3x3 − 53  and therefore
1
  0 x3
1
0
orthogonal projection of b = 1 onto C(A) is


0
 
1 −2 1  8
  
 2 −3 −1 5x3 − 35 1
2
ProjC(A) (b) = bc = Ax0 =   3x3 −  = −1
−1 1 2 3 3
x3 1
3 −5 0
Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 24 / 46
 Exercise 3:
Find all least square solutions x in R3 of Ax = b. Also, find the projection of b
onto C(A), where   
1 0 2 3
0 2 2 −3
 , b= 
−1 1 −1 0
−1 2 0 −3

Exercise 4:
Find
(a) a least-squares sollution of Ax = b
 
2
5
(b) the orthogonal projection of b = 6,


6
 
1 1 0
 1 0 −1
where A =  .
0 1 1
−1 1 −1
Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 25 / 46
 Exercise 5:
Find a least square solution to the following system of linear equations
   
3 −6   −1
4 −8 x =  7 
y
0 1 2

Exercise 6:
Find a least square solution to the following system of linear equations
   
1 0 1 6
1 1 −1 , b = 0
1 2 −3 0

and hence find orthogonal projection of b onto C(A).

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 26 / 46
 Exercise 7:
Find the least square solutions of Ax = b, where
   
2 0 1
A = −1 1 , b= 0 
0 2 −1

Exercise 8:
Find the least square solutions of Ax = b, where
   
0 1 6
A = 1 1 , b = 0
2 1 0

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 27 / 46
 Singular value decomposition

Recall
✿ All eigenvalues of a real symmetric matrix are real.

✿ Eigenvectors of a real symmetric matrix associated with distinct eigenvalues

are orthogonal.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 28 / 46
 Singular value decomposition

Recall
✿ All eigenvalues of a real symmetric matrix are real.

✿ Eigenvectors of a real symmetric matrix associated with distinct eigenvalues

are orthogonal.

Remark
Let A be an m × n matrix. Consider the matrix AT A. Then it is a symmetric matrix
of size n × n, so its eigenvalues are real.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 28 / 46
 Result
Let A be an m × n matrix. If λ is an eigenvalue of AT A, then λ ≥ 0.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 29 / 46
 Result
Let A be an m × n matrix. If λ is an eigenvalue of AT A, then λ ≥ 0.

Proof
Let v be an eigenvector of AT A corresponding to the eigenvalue λ, i.e., Av = λv .
We compute that

∥Av ∥2 = ⟨Av , Av ⟩ = (Av )T Av = v T AT Av = v T λv = λv T v = λ⟨v , v ⟩ = λ∥v ∥2

Since ∥Av ∥2 ≥ 0, it follows from the above equation that λ∥v ∥2 ≥ 0. Since
∥v ∥2 ≥ 0 (as our convention is that eigenvectors are non-zero), we deduce that
λ ≥ 0.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 29 / 46
 Definition(Singular values)
T
Let A be an m × n matrix. Let λ1 , λ2 , . . . , λn denote the eigenvalues of
√ A A, with
repetitions. Order these so that λ1 ≥ λ2 ≥ · · · ≥ λn ≥ 0. Let σi = λi , so that
σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0. The numbers σ1 ≥ σ2 ≥ . . . ≥ σn ≥ 0 are called the
singular values of A.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 30 / 46
 Definition(Singular values)
T
Let A be an m × n matrix. Let λ1 , λ2 , . . . , λn denote the eigenvalues of
√ A A, with
repetitions. Order these so that λ1 ≥ λ2 ≥ · · · ≥ λn ≥ 0. Let σi = λi , so that
σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0. The numbers σ1 ≥ σ2 ≥ . . . ≥ σn ≥ 0 are called the
singular values of A.

Result
The number of non-zero singular values of A equals the rank of A.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 30 / 46
 Example
 
3 2
Find the singular values of the matrix A = 2 3 .
2 −2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 31 / 46
 Example
 
3 2
Find the singular values of the matrix A = 2 3 .
2 −2

Solution
Here rank(A) is 2 and hence A has two singular values. Now
 
17 8
AT A =
8 17

The characteristics polynomial of AT A is

17 − λ 8
pAT A (λ) = det(AT A−λI2 ) = = λ2 −34λ+255 = (λ−25)(λ−9)
8 17 − λ

Thus,√the eigenvalues of√AT A are λ1 = 25 and λ2 = 9. So the singular are

σ1 = 25 = 5 and σ2 = 9 = 3.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 31 / 46
 Definition(Singular value decomposition)
Let A be an m × n matrix. A singular value decomposition of A is a factorization

A = UΣV T

where:
❁ U is an m × m orthogonal matrix.

❁ V is an n × n orthogonal matrix.

❁ Σ is an m × n matrix whose ith diagonal entry equals the ith singular value i
for i = 1, 2, . . . , r , where r is the rank of A. All other entries of are zero.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 32 / 46
 Definition(Singular value decomposition)
Let A be an m × n matrix. A singular value decomposition of A is a factorization

A = UΣV T

where:
❁ U is an m × m orthogonal matrix.

❁ V is an n × n orthogonal matrix.

❁ Σ is an m × n matrix whose ith diagonal entry equals the ith singular value i
for i = 1, 2, . . . , r , where r is the rank of A. All other entries of are zero.

Remark
The columns of matrix V are called right singular vectors and the columns of matrix
U are called left singular vectors.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 32 / 46
 Result
Let A be an m × n matrix and σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0 be singular values of A. Let
vi be an orthonormal eigenvector of AT A corresponding to eigenvalue σi 2 . Then
➀ ∥Avi ∥ = σi .

➁ If i ̸= j then Avi and Avj are orthogonal.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 33 / 46
 Result
Let A be an m × n matrix and σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0 be singular values of A. Let
vi be an orthonormal eigenvector of AT A corresponding to eigenvalue σi 2 . Then
➀ ∥Avi ∥ = σi .

➁ If i ̸= j then Avi and Avj are orthogonal.

Result
Let A be an m × n matrix. Then A has a (not unique) singular value decomposition
A = UΣV T ,where U and V are as follows:
❁ The columns of V are orthonormal eigenvectors v1 , . . . vn of AT A, where
AT Avi = σi 2 vi .

❁ If i ≤ r , so that σi ̸= 0, then the ith column ui of U is σi −1 Avi . By above

result, these columns are orthonormal,and the remaining columns of U are
obtained by arbitrarily extending to an orthonormal basis for Rm .

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 33 / 46
 Remark
We know that the vi ’s are eigenvectors of AT A corresponding to eigenvalues σi 2 ,i.e.,

AT Avi = σi 2 vi
=⇒ (AAT )Avi = σi 2 Avi
=⇒ AAT (σi ui ) = σi 2 σi ui
AAT ui = σi 2 ui

This shows that ui is an eigenvector of AAT corresponding to eigenvalue σi 2 . Thus,

columns ui of U are orthonormal eigenvectors of AAT corresponding to eigenvalue
σi 2 and columns of V are orthonormal eigenvectors of V corresponding to eigenvalue
σi 2 .

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 34 / 46
 Steps for Calculation of singular value decomposition

❁ Step-1: Compute AT A of a real m × n matrix A of rank r .

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 35 / 46
 Steps for Calculation of singular value decomposition

❁ Step-1: Compute AT A of a real m × n matrix A of rank r .

❁ Step-2: Compute the eigenvalues of AT A and hence compute singular values

of AT A.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 35 / 46
 Steps for Calculation of singular value decomposition

❁ Step-1: Compute AT A of a real m × n matrix A of rank r .

❁ Step-2: Compute the eigenvalues of AT A and hence compute singular values

of AT A.

❁ Step-3: Sort the singular values, possibly renaming them, so that

σ1 ≥ σ2 ≥ · · · ≥ σr .

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 35 / 46
 Steps for Calculation of singular value decomposition

❁ Step-1: Compute AT A of a real m × n matrix A of rank r .

❁ Step-2: Compute the eigenvalues of AT A and hence compute singular values

of AT A.

❁ Step-3: Sort the singular values, possibly renaming them, so that

σ1 ≥ σ2 ≥ · · · ≥ σr .

❁ Step-4: Construct the matrix of size m × n such that Σii = 1 for i = 1, . . . , r

and Σij = 0 when i ̸= j.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 35 / 46
 Steps for Calculation of singular value decomposition

❁ Step-1: Compute AT A of a real m × n matrix A of rank r .

❁ Step-2: Compute the eigenvalues of AT A and hence compute singular values

of AT A.

❁ Step-3: Sort the singular values, possibly renaming them, so that

σ1 ≥ σ2 ≥ · · · ≥ σr .

❁ Step-4: Construct the matrix of size m × n such that Σii = 1 for i = 1, . . . , r

and Σij = 0 when i ̸= j.

❁ Step-5: Compute the eigenvectors of AT A.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 35 / 46
 ❁ Step-6: Compute the (right singular) vectors v1 , . . . , vr by normalizing each
eigenvector xi by dividing it by ∥xi ∥. That is, let
xi
vi = , i = 1, . . . , r
∥xi ∥

If n > r , the additional n − r vectors vr +1 , . . . , vn need to be chosen as an

orthonormal basis in Null(A).

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 36 / 46
 ❁ Step-6: Compute the (right singular) vectors v1 , . . . , vr by normalizing each
eigenvector xi by dividing it by ∥xi ∥. That is, let
xi
vi = , i = 1, . . . , r
∥xi ∥

If n > r , the additional n − r vectors vr +1 , . . . , vn need to be chosen as an

orthonormal basis in Null(A).

❁ Step-7: Construct the orthogonal matrix V = [v1 v2 · · · vn ].

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 36 / 46
 ❁ Step-6: Compute the (right singular) vectors v1 , . . . , vr by normalizing each
eigenvector xi by dividing it by ∥xi ∥. That is, let
xi
vi = , i = 1, . . . , r
∥xi ∥

If n > r , the additional n − r vectors vr +1 , . . . , vn need to be chosen as an

orthonormal basis in Null(A).

❁ Step-7: Construct the orthogonal matrix V = [v1 v2 · · · vn ].

❁ Step-8: Compute the (left singular) vectors u1 , . . . , ur as

Avi
Avi = σi ui =⇒ ui = = σi −1 Avi , i = 1, . . . , r
σi
If m > r , the additional m − r vectors ur +1 , . . . , um need to be chosen as an
orthonormal basis in Null(AT ).

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 36 / 46
 ❁ Step-6: Compute the (right singular) vectors v1 , . . . , vr by normalizing each
eigenvector xi by dividing it by ∥xi ∥. That is, let
xi
vi = , i = 1, . . . , r
∥xi ∥

If n > r , the additional n − r vectors vr +1 , . . . , vn need to be chosen as an

orthonormal basis in Null(A).

❁ Step-7: Construct the orthogonal matrix V = [v1 v2 · · · vn ].

❁ Step-8: Compute the (left singular) vectors u1 , . . . , ur as

Avi
Avi = σi ui =⇒ ui = = σi −1 Avi , i = 1, . . . , r
σi
If m > r , the additional m − r vectors ur +1 , . . . , um need to be chosen as an
orthonormal basis in Null(AT ).

❁ Step-9: Construct the orthogonal matrix U = [u1 u2 · · · um ].

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 36 / 46
 Example
 
3 0
Find the singular value decomposition of A = .
4 5

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 37 / 46
 Example
 
3 0
Find the singular value decomposition of A = .
4 5

Solution
Here rank(A) = 2, so A has two singular values. Now
    
3 4 3 0 25 20
AT A = =
0 5 4 5 20 25

The characteristic polynomial of matrix AT A is

25 − λ 20 λ = 45
pAT A (λ) = det(AT A − λI2 ) = = (λ − 45)(λ − 5) =⇒ 1
20 25 − λ λ2 = 5
√ √
The singular values of A are σ1 = 45 and σ2 = 5. This implies
√ 
45 √0
Σ=
0 5
Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 37 / 46
 Eigenvector of AT A corresponding to eigenvalue λ1 = 45 : Consider
 
T −20 20
(A A − λ1 I2 )X = 0 =⇒ X =0
20 −20
which gives −20x1 + 20x
 2 = 0, i.e., x1 = x2 . Thus, eigenvector corresponding to
1
eigenvalue λ1 is s1 = . Therefore
1
" #
s1 √1
v1 = = √12
∥s1 ∥ 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 38 / 46
 Eigenvector of AT A corresponding to eigenvalue λ1 = 45 : Consider
 
T −20 20
(A A − λ1 I2 )X = 0 =⇒ X =0
20 −20
which gives −20x1 + 20x
 2 = 0, i.e., x1 = x2 . Thus, eigenvector corresponding to
1
eigenvalue λ1 is s1 = . Therefore
1
" #
s1 √1
v1 = = √12
∥s1 ∥ 2

Eigenvector of AT A corresponding to eigenvalue λ2 = 5 : Consider

 
20 20
(AT A − λ2 I2 )X = 0 =⇒ X =0
20 20
which gives 20x1 + 20x 2 = 0, i.e., x1 = −x2 . Thus, eigenvector corresponding to
−1
eigenvalue λ2 is s2 = . Therefore
1
" #
s2 − √12
v2 = = √1
∥s2 ∥ 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 38 / 46
 Therefore " #
√1 − √12
v = [v1 v2 ] = 2
√1 √1
2 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 39 / 46
 Therefore " #
√1 − √12
v = [v1 v2 ] = 2
√1 √1
2 2

Now compute u1 and u2 as following

  " 1 # " √1 #
−1 1 3 0 √2
u1 = σ1 Av1 = √ 1 = √310
45 4 5 √2 10
 " √1 # " √3 #
1 3 0 − 2 −

u2 = σ2 −1 Av2 = √ 1 = √1 10
5 4 5 √
2 10

Therefore " #
√1 √1
U = [u1 u2 ] = 10 10
√1 √1
10 10

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 39 / 46
 Therefore " #
√1 − √12
v = [v1 v2 ] = 2
√1 √1
2 2

Now compute u1 and u2 as following

  " 1 # " √1 #
−1 1 3 0 √2
u1 = σ1 Av1 = √ 1 = √310
45 4 5 √2 10
 " √1 # " √3 #
1 3 0 − 2 −

u2 = σ2 −1 Av2 = √ 1 = √1 10
5 4 5 √
2 10

Therefore " #
√1 √1
U = [u1 u2 ] = 10 10
√1 √1
10 10

Thus " # √  " √1 #

√1 √1 45 √1
A = UΣV T
= 10 10 √0 2 2
√1 √1 0 5 − √12 √1
10 10 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 39 / 46
 Note
Notice that when A is a symmetric matrix, AT A = AAT , so U = V . Less work is
required.

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 40 / 46
 Example
 
1 1
Find the singular value decomposition of A = .
1 0

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 41 / 46
 Example
 
1 1
Find the singular value decomposition of A = .
1 0

Solution
Note that A is a symmetric matrix so U = V . Now
 
2 1
AT A =
1 1

Here rank(A) = 2, so A has two singular values. The characteristics polynomial of

AT A is
√
3+ 5
2−λ 1 λ =
T
pAT A (λ) = det(A A − λI2 ) = = λ2 − 3λ + 1 =⇒ 1 2√
1 1−λ λ2 = 3− 5
2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 41 / 46
 Solution cont. . .

The singular values of A are

√ s√
p 3+ 5 1+ 5
σ1 = λ1 = =
2 2
s √ √
p 3− 5 1− 5
σ2 = λ1 = =
2 2

This implies " √ #

1+ 5
2 0√
Σ= 1− 5
0 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 42 / 46
 √
3+ 5
Eigenvector of AT A corresponding to eigenvalue λ1 = 2 : Consider
" √ #
1− 5
1√
(AT A − λ1 I2 )X = 0 =⇒ 2 X =0
1 − 1+2 5

The augmented matrix of this system of linear equation is

" √ # " √ #  √
1− 5 1− 5

1√ | 0 1 | 0 1− 5
2
√
1− 5 2
√ R2 → R2 − R1 2 1 | 0
R2 → 2 R2
1+ 5 1− 5 0 0 | 0
1 − 2 | 0 2 1 | 0

The system corresponding to above REF is

√ ! √ !
1− 5 1− 5
x1 + x2 = 0 =⇒ x2 = − x1
2 2
 
1√
Thus eigenvector corresponding to eigenvalue λ1 is s1 = 5 . Thus
− 1−2
 

1√
 √ 2 √
s1 2  10−2 5
v1 = =p √ 1− 5 = √ 1√
∥s1 ∥ 10 − 2 5 − 2 2 5

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 43 / 46
 √
3− 5
Eigenvector of AT A corresponding to eigenvalue λ2 = 2 : Consider
" √ #
1+ 5
1√
(AT A − λ2 I2 )X = 0 =⇒ 2
−1+ 5
X =0
1 2

The augmented matrix of this system of linear equation is

" √ # " √ #  √
1+ 5 1+ 5

1√ | 0 1 | 0 1+ 5
2
√
1+ 5 2
√ R2 → R2 − R1 2 1 | 0
R2 → 2 R2
−1+ 5 1+ 5 0 0 | 0
1 2 | 0 2 1 | 0

The system corresponding to above REF is

√ ! √ !
1+ 5 1+ 5
x1 + x2 = 0 =⇒ x2 = − x1
2 2
 
1√
Thus eigenvector corresponding to eigenvalue λ2 is s2 = . Thus
− 1+2 5

 

1√
 √ 2 √
s2 2  10+2 5
v2 = =p √ 1+ 5 =
∥s2 ∥ 10 + 2 5 − 2 − √ 1√
2 5

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 44 / 46
 Therefore  
√ 2 √ 2
√ √
10−2 5 10+2 5 
V = [v1 v2 ] =  √√1 1 =U
−√ √
2 5 2 5

Thus
 " √ # T
√ 2 √ 2 √ 2 √ 2
√ √ 1+ 5 √ √
T 10−2 5 10+2 5  2 0√ 10−2 5 10+2 5 
A = UΣV = 1 1 1
√√ −√ √ √√ − √ 1√

1− 5
2 5 2 5
0 2 2 5 2 5

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 45 / 46
 Exercise 9:  
2 2
Find the singular value decomposition of A = .
−1 1

Exercise 10:  
4 11 14
Find the singular value decomposition of A = .
8 7 −2

Exercise 11:  
1 −1
Find the singular value decomposition of A = 0 1 .
1 0

Exercise 12:  
1 1
Find the singular value decomposition A = √0 0 .
√
2 − 2

Dr. Juhi Kesarwani & Dr. Ashish Kesarwany (VITB) Applications of Inner Product Spaces August 9, 2024 46 / 46