# Math for CS Tutorial 2-3 1

• Solution of Systems of Linear Equations
• Gaussian Elimination
• LU Decomposition
• Gram-Schmidt vector orthogonalization

Tutorial 2-3. Contents
Math for CS Tutorial 2-3 2
Linear Systems in Matrix Form
n n nn n n
n n
n n
b x a x a x a
b x a x a x a
b x a x a x a
= + + +
= + + +
= + + +

2 2 1 1
2 2 2 22 1 21
1 1 2 12 1 11
(
(
(
(
¸
(

¸

=
(
(
(
(
¸
(

¸

(
(
(
(
¸
(

¸

n n nn n n
n
n
b
b
b
x
x
x
a a a
a a a
a a a
 

   

2
1
2
1
2 1
2 22 21
1 12 11
¬
Linear equation means that every variable is in the power 1 only.
If the variables in the equation are ordered x
1
, … x
n
and the missing variables
x
i
are written as … + 0·x
i
+ … , then the linear equation can be written in the
matrix notation, according to the matrix multiplication rules. Solution of linear
equations was one of the reasons, for matrix notation invention.
(1)
Math for CS Tutorial 2-3 3
An inconsistent example:
Geometric interpretation

(
¸
(

¸

=
(
¸
(

¸

(
¸
(

¸

5
4
4 2
2 1
2
1
x
x
(
¸
(

¸

0 0
2 1
Rank{A}=1
Rank{A|b}=2≠ Rank{A}
ERO:Multiply the first row with
-2 and add to the second row
(
¸
(

¸

÷3
4
0
2
0
1
Math for CS Tutorial 2-3 4
Full-rank systems
• If Rank{A}=n
Det{A} = 0 ¬ A
-1
exists ¬ Unique solution
(
¸
(

¸

=
(
¸
(

¸

(
¸
(

¸

÷ 2
4
1 1
2 1
2
1
x
x
Math for CS Tutorial 2-3 5
First step of elimination
(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
¸
(

¸

(
(
(
(
(
(
¸
(

¸

=
=
=
) 2 (
) 2 (
3
) 2 (
2
) 1 (
1
3
2
1
) 2 ( ) 2 (
3
) 2 (
2
) 2 (
3
) 2 (
33
) 2 (
32
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
) 1 (
11
) 1 (
1 1 ,
) 1 (
11
) 1 (
31 1 , 3
) 1 (
11
) 1 (
21 1 , 2
0
0
0
/
/
/
n
n
nn n n
n
n
n
n n
b
b
b
b
x
x
x
x
a a a
a a a
a a a
a a a a
a a m
a a m
a a m

    

(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
¸
(

¸

(
(
(
(
(
(
¸
(

¸

) 1 (
) 1 (
3
) 1 (
2
) 1 (
1
3
2
1
) 1 ( ) 1 (
3
) 1 (
2
) 1 (
1
) 1 (
3
) 1 (
33
) 1 (
32
) 1 (
31
) 1 (
2
) 1 (
23
) 1 (
22
) 1 (
21
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
n
n
nn n n n
n
n
n
b
b
b
b
x
x
x
x
a a a a
a a a a
a a a a
a a a a

    

Pivotal element
Math for CS Tutorial 2-3 6
Second step of elimination
(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
¸
(

¸

(
(
(
(
(
(
¸
(

¸

=
=
) 3 (
) 3 (
3
) 2 (
2
) 1 (
1
3
2
1
) 3 ( ) 3 (
3
) 3 (
3
) 3 (
33
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
) 2 (
22
) 2 (
2 2 ,
) 2 (
22
) 2 (
32 2 , 3
0 0
0 0
0
/
/
n
n
nn n
n
n
n
n n
b
b
b
b
x
x
x
x
a a
a a
a a a
a a a a
a a m
a a m

    

(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
¸
(

¸

(
(
(
(
(
(
¸
(

¸

) 2 (
) 2 (
3
) 2 (
2
) 1 (
1
3
2
1
) 2 ( ) 2 (
3
) 2 (
2
) 2 (
3
) 2 (
33
) 2 (
32
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
0
0
0
n
n
nn n n
n
n
n
b
b
b
b
x
x
x
x
a a a
a a a
a a a
a a a a

    

Pivotal element
Math for CS Tutorial 2-3 7
Back substitution algorithm
(
(
(
(
(
(
(
(
¸
(

¸

=
(
(
(
(
(
(
(
(
¸
(

¸

(
(
(
(
(
(
(
(
¸
(

¸

÷
÷
÷
÷ ÷ ÷
) (
) 1 (
1
) 3 (
3
) 2 (
2
) 1 (
1
1
3
2
1
) (
) (
1
) (
1 1
) 3 (
3
) 3 (
33
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
0 0 0 0
0 0 0
0 0
0
n
n
n
n
n
n
n
nn
n
n n
n
n n
n
n
n
b
b
b
b
b
x
x
x
x
x
a
a a
a a
a a a
a a a a

    

| |
1 , , 2 , 1
1
1
1
) ( ) (
) (
1
1
) 1 (
1
) 1 (
1 1
1
) (
) (
 ÷ ÷ =
(
¸
(

¸

÷ =
÷ = =
¿
+ =
÷
÷
÷
÷
÷
÷ ÷
÷
n n i x a b
a
x
x a b
a
x
a
b
x
n
i k
k
i
ik
i
i
i
ii
i
n
n
n n
n
n
n
n n
n
n
nn
n
n
n
Math for CS Tutorial 2-3 8
LU Decomposition
A=LU
Ax=b ¬LUx=b
Define Ux=y
Ly=b Solve y by forward substitution
ERO’s must be performed on b as well as A
The information about the ERO’s are stored in L
Indeed y is obtained by applying ERO’s to b vector
Ux=y Solve x by backward substitution
Math for CS Tutorial 2-3 9
LU Decomposition by Gaussian
elimination
(
(
(
(
(
(
(
(
¸
(

¸

(
(
(
(
(
(
(
(
¸
(

¸

=
÷ ÷ ÷
÷ ÷ ÷
) (
) (
1
) (
1 1
) 3 (
3
) 3 (
33
) 2 (
2
) 2 (
23
) 2 (
22
) 1 (
1
) 1 (
13
) 1 (
12
) 1 (
11
4 , 3 , 2 , 1 ,
3 , 1 2 , 1 1 , 1
2 , 3 1 , 3
1 , 2
0 0 0 0
0 0 0
0 0
0
1
1
0
0 0 1
0 0 0 1
0 0 0 0 1
n
nn
n
n n
n
n n
n
n
n
n n n n
n n n
a
a a
a a
a a a
a a a a
m m m m
m m m
m m
m
A
    

 
    

Compact storage: The diagonal entries of L matrix are all 1’s,
they don’t need to be stored. LU is stored in a single matrix.
There are infinitely many different ways to decompose A.
Most popular one: U=Gaussian eliminated matrix
L=Multipliers used for elimination
Math for CS Tutorial 2-3 10
An example: Linear System
(
(
(
¸
(

¸

=
(
(
(
¸
(

¸

(
(
(
¸
(

¸

29
20
14
6 4 3
4 3 2
3 2 1
3
2
1
x
x
x
Rank{A}=n ¬ A
-1
exists ¬ Unique solution
Math for CS Tutorial 2-3 11
Gaussian Elimination
(
(
(
¸
(

¸

÷ ÷ ÷ ¬
(
(
(
¸
(

¸

÷
÷
÷ ÷
÷ ÷ ¬
(
(
(
¸
(

¸

3
8
14
1 0 0
2 1 0
3 2 1
13
8
14
3 2 0
2 1 0
3 2 1
29
20
14
6 4 3
4 3 2
3 2 1
x
3
=3;
-x
2
-2x
3
=-8;
x
2
=2;
x
1
+2*2+3*3=14;
x
1
=14-4-9=1.

Math for CS Tutorial 2-3 12
LU Factorization
b L x U
b L x A L
LU A
b x A
1
1 1
÷
÷ ÷
=
=
=
=
(
(
(
¸
(

¸

÷ ÷ ÷
(
(
(
¸
(

¸

÷
÷ ¬
(
(
(
¸
(

¸

÷
÷
÷ ÷
÷ ÷
(
(
(
¸
(

¸

÷
÷ ¬
(
(
(
¸
(

¸

(
(
(
¸
(

¸

3
8
14
1 0 0
2 1 0
3 2 1
1 2 1
0 1 2
0 0 1
13
8
14
3 2 0
2 1 0
3 2 1
1 0 3
0 1 2
0 0 1
29
20
14
6 4 3
4 3 2
3 2 1
1 0 0
0 1 0
0 0 1
L
-1
U=L
-1
A
(
(
(
¸
(

¸

÷ ÷ =
(
(
(
¸
(

¸

(
(
(
¸
(

¸

÷
÷
1 0 0
2 1 0
3 2 1
6 4 3
4 3 2
3 2 1
1 2 1
0 1 2
0 0 1
Check that there were no mistakes:
L
-1
A

U

Math for CS Tutorial 2-3 13
LU Factorization
(
(
(
¸
(

¸

÷ ÷ =
(
(
(
¸
(

¸

(
(
(
¸
(

¸

÷
÷
1 0 0
2 1 0
3 2 1
6 4 3
4 3 2
3 2 1
1 2 1
0 1 2
0 0 1
Check that there were no mistakes:
L
-1
A

U

(
(
(
¸
(

¸

÷ =
(
(
(
¸
(

¸

(
(
(
¸
(

¸

÷
÷
3
8
14
29
20
14
1 2 1
0 1 2
0 0 1
L
-1
b b

L
-1

x
3
=3;
-x
2
-2x
3
=-8;
x
2
=2;
x
1
+2*2+3*3=14;
x
1
=14-4-9=1.

Math for CS Tutorial 2-3 14
the Gram-Schmidt procedure
for vector orthogonalization

• The purpose
we can construct a set of orthonormal
vectors u
i
from a set of n-dimensional
vectors v
i
. 1≤i≤m

• And v
i
can be represented by the linear
combination of u
i

Math for CS Tutorial 2-3 15
G-S procedure (cont.)
1. Select a vector from vi arbitrarily, say v1

2. By normalizing its length, we obtain the first vector, say

3. Select v2 and subtract the projection of v2 onto u1, we
get

w
2
=v
2
– (v
2
• u
1
)u
1
v
v
u
1
1
1
=
Math for CS Tutorial 2-3 16
G-S procedure (cont.)
4. And normalizing

5. Now we can check, that

0 , , , ,
2 1 2 1 1 1 , 2 2 1 2 1
= ÷ = ÷ = v u v u u u v v u u u
w
w
u
2
2
2
=
Math for CS Tutorial 2-3 17
G-S procedure (cont.)
6. The procedure continues by selecting v
3
and subtract
its projection on u
1
and u
2
, we have
w
3
=v
3
– (v
3
• u
1
)u
1
– (v
3
• u
2
)u
2
7. Then, the orthonormal vector u
3
is

8. By continuing this procedure, we shall construct the
set of orthonormal vectors u
i

w
w
u
3
3
3
=
Math for CS Tutorial 2-3 18
G-S procedure: Example
Consider v
1
=(0,1,1); v
2
=(1,-1,0); v
3
=(1,0,1)
They are not orthogonal: (v
i
,v
j
)≠0.

Following G-S:
) 1 , 1 , 0 (
2
1
1
= u
)
2
1
,
2
1
, 1 ( ) 1 , 1 , 0 (
2
1
) 0 , 1 , 1 ( ) 1 , 1 , 0 (
2
1
2
1
) 0 , 1 , 1 (
1 2 1 2 2
÷ = + ÷ = ·
÷
÷ ÷ = · ÷ = u v u v w
Math for CS Tutorial 2-3 19
Example (cont.)
)
2
1
,
2
1
, 1 (
3
2
)
2
1
,
2
1
, 1 (
2
2
2
÷ = = ÷ =
w
w
u
0 )
2
1
2
1
0 (
3
1
2 1
= + ÷ = · u u
We can check, that:
2 2 3 1 1 3 3 3
u u v u u v v w · ÷ · ÷ =
Math for CS Tutorial 2-3 20
Example (cont.)
)
2
1
,
2
1
, 1 (
3
2
)
2
1
0 1 (
3
2
) 1 , 1 , 0 (
2
1
2
1
) 1 , 0 , 1 (
3
÷ ·
|
|
.
|

\
|
+ + ÷ · ÷ = w
)
2
1
,
2
1
, 1 ( )
2
1
,
2
1
, 0 ( ) 1 , 0 , 1 ( )
2
1
,
2
1
, 1 (
2
3
3
2
) 1 , 1 , 0 (
2
1
) 1 , 0 , 1 (
3
÷ ÷ ÷ = ÷ · ÷ ÷ = w
) 0 , 0 , 0 (
3
= w
Why?