Linear Systems

Irwan Ary Dharmawan

Linear Systems
a11 x1 a12 x2 a13 x3 b1
a21 x1 a22 x2 a23 x3 b2
a31 x1 a32 x2 a33 x3 b3

a11

a21
a
31

a12

a13

a22

a23

a32

a33

x1 b1

x2 b2
x b
3 3

Linear Systems
Solve Ax=b, where A is an nn matrix and
b is an n1 column vector
Can also talk about non-square systems
where
A is mn, b is m1, and x is n1
Overdetermined if m>n:
more equations than unknowns

Underdetermined if n>m:
more unknowns than equations
Can look for best solution using least squares

Singular Systems
A is singular if some row is
linear combination of other rows
Singular systems can be
underdetermined:
2 x1 3 x2 5
4 x1 6 x2 10

or inconsistent:2 x1 3x2 5
4 x1 6 x2 11

Inverting a Matrix
Usually not a good idea to compute x=A -1b
Inefficient
Prone to roundoff error

In fact, compute inverse using linear solver

Solve Axi=bi where bi are columns of identity,
xi are columns of inverse

Many solvers can solve several R.H.S. at once

Gauss-Jordan Elimination
Fundamental operations:
1. Replace one equation with linear combination
of other equations
2. Interchange two equations

3. Re-label two variables

Combine to reduce to trivial system

Simplest variant only uses #1 operations,
but get better stability by adding
#2 (partial pivoting) or #2 and #3 (full
pivoting)

Gauss-Jordan Elimination
Solve:
2 x1 3 x2 7
4 x1 5 x2 13

Only care about numbers form

tableau or augmented matrix:
2

3
5

13

Gauss-Jordan Elimination
Given:
2

3
5

13

Goal: reduce this to trivial system

1

0
1

and read off answer from right column

Gauss-Jordan Elimination
2

3
5

13

Basic operation 1: replace any row by

linear combination with any other row
Here, replace row1 with 1/2 * row1 + 0 *
row2
7
1 32
2

13

Gauss-Jordan Elimination
1

13

Replace row2 with row2 4 * row1

1

Negate row2

1
7

Gauss-Jordan Elimination
1

Replace row1 with row1 3/2 * row2

1

0
1

Read off solution: x1 = 2, x2 = 1

Gauss-Jordan Elimination
For each row i:
Multiply row i by 1/aii
For each other row j:
Add aji times row i to row j

At the end, left part of matrix is identity,

answer in right part
Can solve any number of R.H.S.
simultaneously

Pivoting
Consider this system:
0

1
3

Immediately run into problem:

algorithm wants us to divide by zero!
More subtle version:
0.001 1

3
2

Pivoting
Conclusion: small diagonal elements
bad
Remedy: swap in larger element from
somewhere else

Partial Pivoting
0

1
3

Swap rows 1 and 2:

2

3
1

Now continue:
1

0
1

Full Pivoting
0

1
3

Swap largest element onto diagonal by

swapping rows 1 and 2 and columns 1 and
2:
3

Critical: when swapping columns, must

remember to swap results!

Full Pivoting
3

1
1

2
0
2

3
2

0
1

Swap results
1 and 2

3
2

Full pivoting more stable, but only slightly

Operation Count
For one R.H.S., how many operations?
For each of n rows:
Do n times:
For each of n+1 columns:
One add, one multiply

Total = n3+n2 multiplies, same # of adds

Asymptotic behavior: when n is large,
dominated by n3

Faster Algorithms
Our goal is an algorithm that does this
in
1/ n3 operations, and does not require
3
all R.H.S. to be known at beginning
Before we see that, lets look at a few
special cases that are even faster

Tridiagonal Systems
Common special case:
a11

a21

0
0

a12

a22

a23

a32

a33

a34

a43

a44

b1

b2

b3
b4

Only main diagonal + 1 above and 1 below

Solving Tridiagonal Systems

When solving using Gauss-Jordan:
Constant # of multiplies/adds in each row
Each row only affects 2 others
a11

a21

0
0

a12

a22

a23

a32

a33

a34

a43

a44

b1

b2

b3
b4

Running Time
2n loops, 4 multiply/adds per loop
(assuming correct bookkeeping)
This running time has a fundamentally
different dependence on n: linear
instead of cubic
Can say that tridiagonal algorithm is O(n)
while
Gauss-Jordan is O(n3)

Big-O Notation
Informally, O(n3) means that the dominant
term for large n is cubic
More precisely, there exist a c and n0 such
that
running time c n3
if
n > n0
This type of asymptotic analysis is often
used
to characterize different algorithms

Triangular Systems
Another special case: A is lowertriangular
a11

a21

a31
a
41

a22

a32

a33

a42

a43

a44

b1

b2

b3
b4

Triangular Systems
Solve by forward substitution
a11

a21

a31
a
41

a22

a32

a33

a42

a43

a44

b1
x1
a11

b1

b2

b3
b4

Triangular Systems
Solve by forward substitution
a11

a21

a31
a
41

a22

a32

a33

a42

a43

a44

b2 a21 x1
x2
a22

b1

b2

b3
b4

Triangular Systems
Solve by forward substitution
a11

a21

a31
a
41

a22

a32

a33

a42

a43

a44

b3 a31 x1 a32 x2
x3
a33

b1

b2

b3
b4

Triangular Systems
If A is upper triangular, solve by
backsubstitution
a11

0
0

a12

a13

a14

a15

a22

a23

a24

a25

a33

a34

a35

a44

a45

a55

b5
x5
a55

b1

b2

b3
b4

b5

Triangular Systems
If A is upper triangular, solve by
backsubstitution
a11

0
0

a12

a13

a14

a15

a22

a23

a24

a25

a33

a34

a35

a44

a45

a55

x4

b4 a45 x5
a44

b1

b2

b3
b4

b5

Triangular Systems
Both of these special cases can be solved in
O(n2) time
This motivates a factorization approach to
solving arbitrary systems:
Find a way of writing A as LU, where L and U are
both triangular

Ax=b LUx=b Ly=b Ux=y

Time for factoring matrix dominates
computation

Cholesky Decomposition
For symmetric matrices, choose U=LT
Perform decomposition
a11

a12
a
13

Ax=b
LTx=y

a12
a22
a23

a13
l11

a23 l21
l
a33
31

LLTx=b

0
l22
l32

0
l33

l11

0
0

l31

l32
l33

Ly=b

l21
l22

Cholesky Decomposition
a11

a12
a
13

a13
l11

a23 l21
l
a33
31

a12
a22
a23

0
l22
l32

0
l33

l11

0
0

l21
l22
0

l11 2 a11 l11 a11

a12
l11
a13
l31
l11

l11l21 a12 l21

l11l31 a13
2

l21 2 l22 2 a22 l22 a22 l21 2

l21l31 l22l32 a23

a23 l21l31
l32
l22

l31

l32
l33

Cholesky Decomposition
a11

a12
a
13

a12
a22
a23

a13
l11

a23 l21
l
a33
31

0
l22
l32

0
l33

i 1

lii aii lik2

k 1

i 1

l ji

aij lik l jk
k 1

lii

l11

0
0

l21
l22
0

l31

l32
l33

Cholesky Decomposition
This fails if it requires taking square root
of a negative number
Need another condition on A: positive
definite
For any v, vT A v > 0
(Equivalently, all positive eigenvalues)

Cholesky Decomposition
Running time turns out to be 1/6n3
Still cubic, but much lower constant

Result: this is preferred method for

solving
symmetric positive definite systems

LU Decomposition
Again, factor A into LU, where
L is lower triangular and U is upper
triangular
Ax=b
LUx=b
Ly=b
Ux=y

Last 2 steps in O(n2) time, so total time

dominated by decomposition

Crouts Method
a11

a21
a
31

a12
a22
a32

a13
l11

a23 l21
l
a33
31

0
l22
l32

0
l33

u11

0
0

u12
u22
0

More unknowns than equations!

Let all lii=1

u13

u23
u33

Crouts Method
a11

a21
a
31

a12
a22
a32

a13
1

a23 l21
l
a33
31
u11 a11

0
1
l32

0
1

u11

0
0

u12
u22
0

a21
u11
a
l31 31
u11

l21u11 a21 l21

l31u11 a31

u12 a12
l21u12 u22 a22 u22 a22 l21u12
a l u
l31u12 l32u22 a32 l32 32 31 12
u22

u13

u23
u33

Crouts Method
a11

a21
a
31

a12
a22
a32

a13
1

a23 l21
l
a33
31

0
1
l32

0
1

For i = 1..n
For j = 1..i

j 1

u ji a ji l jk uki
k 1

For j = i+1..n
l ji

i 1

a ji l jk uki
k 1

uii

u11

0
0

u12
u22
0

u13

u23
u33

Crouts Method
Interesting note: # of outputs = # of inputs,
algorithm only refers to elements not output
yet
Can do this in-place!
Algorithm replaces A with matrix
of l and u values, 1s are implied

u11

l21
l
31

u12

u22
l32

u13

u 23
u33

Resulting matrix must be interpreted in a special

way: not a regular matrix

Can rewrite forward/backsubstitution routines to

use this packed l-u matrix

LU Decomposition
Running time is 1/3n3
Only a factor of 2 slower than symmetric
case

This is the preferred general method for

solving linear equations

Pivoting very important

Partial pivoting is sufficient, and widely
implemented