Example:

Find the QR decomposition of the following matrix

Solution:
Since 𝐴 has independent column, then by QR decomposition, we can decompose 𝐴 into product
of two matrices,
𝐴 = 𝑄𝑅,
where, 𝑄 is orthonormal matrix (orthonormal basis for the column space of 𝐴),
𝑅 is upper triangular matrix with positive entries of the diagonal.
Let us calculate the matrix 𝑄.
Before, let us calculate orthogonal basis of the column space of 𝐴.
By using Gram-Schmidt orthogonalization process:
{𝑢1 , 𝑢2 } is the orthogonal basis of the column space of 𝐴.
𝑥2 ⋅𝑢1
𝑢1 = 𝑥1 , 𝑢2 = 𝑥2 − 𝑢1
𝑢1 ⋅𝑢1

5 9
𝑥1 = 1 , 𝑥2 = 7
−3 −5
1 5

5
𝑢1 = 𝑥1 = 1
−3
1

9 5
𝑥2 . 𝑢1 = 7 ⋅ 1 = 45 + 7 + 15 + 5 = 72
−5 −3
5 1
 5 5
𝑢1 ⋅ 𝑢1 = 1 ⋅ 1 = 25 + 1 + 9 + 1 = 36
−3 −3
1 1

9 5 9 5 −1
7 72 1 = 7 −2 1 = 5
𝑢2 = −
−5 36 −3 −5 −3 1
5 1 5 1 3

5 −1
Orthogonal basis= 1 , 5
−3 1
1 3
𝑢1 𝑢2
Orthonormal vectors are 𝑣1 = , 𝑣2 =
𝑢1 ‖𝑢2 ‖

𝑢1 = 25 + 1 + 9 + 1 = 6

𝑢2 = 1 + 25 + 1 + 9 = 6
 5 −1
1 1 1 5
𝑣1 = , 𝑣2 =
6 −3 6 1
1 3

{𝑣1 , 𝑣2 } forms an orthonormal basis.

5 1
−
6 6
1 5
Therefore, 𝑄 = 6 6 .
3 1
−
6 6
1 3
6 6

Since, we have 𝐴 = 𝑄𝑅,

Multiplying 𝑄𝑇 on both sides, we have
𝑄𝑇 𝐴 = 𝑄𝑇 𝑄𝑅 ⇒ 𝑅 = 𝑄𝑇 𝐴
 𝑅 = 𝑄𝑇 𝐴

5 1 3 1 5 9
−
𝑅= 6 6 6 6 1 7 = 6 12
1 5 1 3 −3 −5 0 6
− 1 5
6 6 6 6

5 1
−
6 6
5 9 1 5
Therefore, 1 7 = 6 6 6 12
.
−3 −5 3 1 0 6
1 5 −
6 6
1 3
6 6
Example:

Solution:
Home work. 𝑢1 = 𝑥1

Hint: 𝑥2 ⋅ 𝑢1
𝑢1 , 𝑢2 , 𝑢3 is the orthogonal basis for the column 𝑢2 = 𝑥2 − 𝑢1
𝑢1 ⋅ 𝑢1
space of the matrix 𝐴.
𝑥3 ⋅ 𝑢1 𝑥3 ⋅ 𝑢2
𝐴 = [𝑥1 𝑥2 𝑥3 ] 𝑢3 = 𝑥3 − 𝑢 − 𝑢
𝑢1 ⋅ 𝑢1 1 𝑢2 ⋅ 𝑢2 2
1 0 0 do the rest ……………
𝑥1 = 1 , 𝑥2 = 1 , 𝑥3 = 0
1 1 1
1 1 1
Note:
1) To have solution for the system 𝐴𝐱 = 𝐛, 𝒃 must be in the column space of 𝐴.

𝑎1 𝑎2 𝑎3 𝑏1 𝑥1
Proof: For example consider 𝐴 = 𝑎4 𝑎5 𝑎6 , 𝑏 = 𝑏2 , and 𝐱 = 𝑥2 is the solution of the system.𝐴𝐱 =
𝑎7 𝑎8 𝑎9 𝑏3 𝑥3
𝐛
𝑎1 𝑎2 𝑎3 𝑥1 𝑏1
⇒ 𝑎4 𝑎5 𝑎6 𝑥2 = 𝑏2
𝑎7 𝑎8 𝑎9 𝑥3 𝑏3
It can be written as follows
𝑎1 𝑎2 𝑎3 𝑏1
𝑎4 𝑥1 + 𝑎5 𝑥2 + 𝑎6 𝑥3 = 𝑏2
𝑎7 𝑎8 𝑎9 𝑏3

⇒ 𝑐1 𝑥1 + 𝑐2 𝑥2 + 𝑐3 𝑥3 = 𝑏, where 𝑐1 , 𝑐2 , 𝑐3 are the columns of the matrix 𝐴.

From the expression it is clear that the matrix 𝒃 expressed as the linear combination of the columns of the
matrix A. That is, we can say that 𝒃 is in the column space of 𝐴.
Least Squares solution:

Definition:

If 𝐴 is 𝑚 × 𝑛 and 𝑏 is in 𝑅𝑚 , a least-squares solution of 𝐴𝑥 = 𝑏 is an 𝑥ො in 𝑅𝑛 such that 𝑏 − 𝐴𝑥ො ≤

||𝑏 − 𝐴𝑥|| for all 𝑥 in 𝑅𝑛 .
Least Squares: Four Ways:
The Least Squares solution to 𝐴𝑥 = 𝑏 is 𝑥 + = 𝐴+ 𝑏

+ + 2
1) 𝑥 = 𝑥 = 𝐴 𝑏 makes 𝑏 − 𝐴𝑥 as small as possible. Least Squares solution
2) If another 𝑥ො achieves that minimum then 𝑥 + < ||𝑥||.
ො Minimum norm solution

𝑥 + is the minimum norm least square solution.

Example:

Also, determine the least-squared error in the least square solution of the given system.
Solution:

Clearly is it can be verified that 𝒃 is not in the column space of 𝐴.

Therefore, 𝐴𝐱 = 𝐛 is inconsistent.

We can find the least square solution for the given system.

෡
First we will take the projection of 𝒃 on to the column space of 𝐴. We denote that with 𝒃.

Let 𝑐1 , 𝑐2 are the columns of the matrix 𝐴.

𝑏෠ can be written as 𝑏෠ = 𝑃𝑟𝑜𝑗𝐴 𝑏.

𝑏 ⋅ 𝑐1 𝑏 ⋅ 𝑐2
𝑏෠ = 𝑐1 + 𝑐2
𝑐1 ⋅ 𝑐1 𝑐2 ⋅ 𝑐2

−1 1 −6
𝑏 = 2 , 𝑐1 = 1 , 𝑐2 = −2
1 1 1
6 1 7

−1 1 1 1
𝑏 ⋅ 𝑐1 = 2 ⋅ 1 = −1 + 2 + 1 + 6 = 8; 𝑐1 ⋅ 𝑐1 = 1 ⋅ 1 = 1 + 1 + 1 + 1 = 4
1 1 1 1
6 1 1 1

−1 −6 −6 −6
𝑏 ⋅ 𝑐2 = 2 ⋅ −2 = 6 − 4 + 1 + 42 = 45; c2 ⋅ 𝑐2 = −2 ⋅ −2 = 36 + 4 + 1 + 49 = 90
1 1 1 1
6 7 7 7
 1 −6 1 −6 −10
8 1 90 −2 1 +2 −2 = −2
෠
𝑏= + =2
4 1 45 1 1 1 4
1 7 1 7 16

෡
Now, solve 𝐴𝐱 = 𝒃
1 −6 −10
1 −2 𝑥1 = −2
1 1 𝑥2 4
1 7 16

1 −6 −10
Augmented matrix 1 −2 −2
1 1 4
1 7 16

By Gauss elimination method we need to make the elements below the principal diagonals to zeros.

𝑅2 → 𝑅2 − 𝑅1 ; 𝑅3 → 𝑅3 − 𝑅1 ; 𝑅4 → 𝑅4 − 𝑅1
 1 −6 −10
0 4 8 ⇒ 𝐼𝑡 𝑖𝑠 𝑐𝑙𝑒𝑎𝑟 𝑡ℎ𝑎𝑡 𝑅 , 𝑅 , 𝑅 are dependent.
0 7 14 2 3 4

0 13 26

Therefore, we have only two L.I. equations.

Which are, 𝑥1 − 6𝑥2 = −10,

4𝑥2 = 8 ⇒ 𝑥2 = 2
7𝑥2 = 14 ⇒ 𝑥2 = 2
13𝑥2 = 26 ⇒ 𝑥2 = 2
𝑥1 = −10 + 6 2 = 2
𝑥1 2
Therefore, the least square solution is 𝑥 =
2 2

−1 −10 9
The Least square error is 𝑏 − 𝐴𝑥ො = 2 − −2 = 4 = 49 + 16 + 9 + 100 = 174.
1 4 −3
6 16 −10
using QR decomposition
Example:

using QR factorization.

Also, determine the least-squared error in the least square solution of the given system.
Solution:

Given 𝐴𝐱 = 𝐛 is a inconsistent system.

Hint:

Since 𝐴 has independent columns. Therefore, we can decompose 𝐴 into

𝐴 = 𝑄𝑅
Where, 𝑄 is orthonormal matrix (columns of 𝑄 forms an orthonormal basis of the column space of
𝐴) and 𝑅 is the upper triangular matrix with positive diagonal elements.
We can find the columns of 𝑄 by Grams-Schmidt orthogonalization process.
After finding 𝑄, we can find 𝑅 by 𝑅 = 𝑄𝑇 𝐴
Given 𝐴𝐱 = 𝐛 ⇒ QR 𝐱 = 𝐛 ⇒ Rx = QT 𝐛.
By back substitution, we can find the x.
Normal Equations:

Let 𝐴𝐱 = 𝐛 be the given system then 𝐴𝑇 𝐴x = AT 𝐛 is called the normal equations.

Example:

Solution:
From the normal equations 𝐴𝑇 𝐴x = AT 𝒃
Example:

using normal equations.

Also, determine the least-squared error in the least square solution of the given system.
Theorem:

Let 𝐴 be an 𝑚 × 𝑛 matrix. The following statements are logically equivalent:

(a) The equation 𝐴𝑥 = 𝑏 has a unique least-squares solution for each 𝑏 in ℝ𝑚 .

(b)The columns of 𝐴 are linearly independent.
(c) The matrix 𝐴𝑇 𝐴 is invertible.
When these statements are true, the least-squares solution xො is given by
xො = 𝐴𝑇 𝐴 −1 𝐴𝑇 𝒃